Skip to contentSkip to conversation

Add Secure TSC support for SNP guests

PATCH Superseded Nikunj A Dadhania 5 participants 96 messages Dec 3, 2024 – Jan 6, 2025

Subsystems (5)
x86 KVM KVM x86 x86 MM +1 more subsystems

v15 of 16 Superseded Latest is v16 →

lore
Share & tools
lore
#0 lore raw

v15 cover Add Secure TSC support for SNP guests

This patchset is also available at:

https://github.com/AMDESE/linux-kvm/tree/sectsc-guest-latest

and is based on tip/master

Overview
--------

Secure TSC allows guests to securely use RDTSC/RDTSCP instructions as the parameters being used cannot be changed by hypervisor once the guest is launched. More details in the AMD64 APM Vol 2, Section "Secure TSC".

In order to enable secure TSC, SEV-SNP guests need to send a TSC_INFO guest message before the APs are booted. Details from the TSC_INFO response will then be used to program the VMSA before the APs are brought up. See "SEV Secure Nested Paging Firmware ABI Specification" document (currently at
https://www.amd.com/system/files/TechDocs/56860.pdf) section "TSC Info"

SEV-guest driver has the implementation for guest and AMD Security Processor communication. As the TSC_INFO needs to be initialized during early boot before APs are started, move the guest messaging code from sev-guest driver to sev/core.c and provide well defined APIs to the sev-guest driver.

Patches:
01-02: Patches moving SNP guest messaging code from SEV guest driver to
SEV common code
03-09: SecureTSC enablement patches
10-11: Generic TSC/kvmclock improvements
12-13: SecureTSC enablement patches.

Testing SecureTSC
-----------------

SecureTSC hypervisor patches based on top of SEV-SNP Guest MEMFD series:
https://github.com/AMDESE/linux-kvm/tree/sectsc-host-latest

QEMU changes:
https://github.com/nikunjad/qemu/tree/snp-securetsc-latest

QEMU commandline SEV-SNP with SecureTSC:

  qemu-system-x86_64 -cpu EPYC-Milan-v2 -smp 4 \
    -object memory-backend-memfd,id=ram1,size=1G,share=true,prealloc=false,reserve=false \
    -object sev-snp-guest,id=sev0,cbitpos=51,reduced-phys-bits=1,secure-tsc=on \
    -machine q35,confidential-guest-support=sev0,memory-backend=ram1 \
    ...

Changelog:
----------
v15:
* Rebased on latest tip/master
* Update commits/comments (Boris)
* Add snp_msg_free() to free allocated buffers (Boris)
* Dynamically allocate buffers for sending TSC_INFO (Boris)
* Fix the build issue at patch#1 (Boris)
* Carve out tsc handling in __vc_handle_msr_tsc() (Boris)

v14:
* Make zero_key as static const(Christophe JAILLET)
* As req structure does not have sensitive data remove memzero_explicit(). (Christophe JAILLET)
* Document the assumption that MSR_IA32_TSC is intercepted for Secure TSC (Xiaoyao Li)
* Update commit message for RDTSC/RTDSCP (Xiaoyao Li)
* Check against sev_status directly similar to DEBUG_SWAP, in case the handler gets called from early code. (Tom)
* Make enable_native_sched_clock() static for non-paravirt case (kernel test robot)
* Add a WARN_ON when kvmclock is selected in the guest so there is a splat in the guest as well. (Tom)
* Update commit message. (Tom)

Nikunj A Dadhania (13):
x86/sev: Carve out and export SNP guest messaging init routines x86/sev: Relocate SNP guest messaging routines to common code x86/sev: Add Secure TSC support for SNP guests
x86/sev: Change TSC MSR behavior for Secure TSC enabled guests x86/sev: Prevent RDTSC/RDTSCP interception for Secure TSC enabled
guests
x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC
enabled guests
x86/sev: Mark Secure TSC as reliable clocksource x86/cpu/amd: Do not print FW_BUG for Secure TSC
tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency
tsc: Upgrade TSC clocksource rating
tsc: Switch to native sched clock
x86/kvmclock: Abort SecureTSC enabled guest when kvmclock is selected x86/sev: Allow Secure TSC feature for SNP guests

 arch/x86/include/asm/msr-index.h        |   1 +
 arch/x86/include/asm/sev-common.h       |   2 +
 arch/x86/include/asm/sev.h              |  62 ++-
 arch/x86/include/asm/svm.h              |   6 +-
 include/linux/cc_platform.h             |   8 +
 arch/x86/boot/compressed/sev.c          |   3 +-
 arch/x86/coco/core.c                    |   3 +
 arch/x86/coco/sev/core.c                | 646 +++++++++++++++++++++++-
 arch/x86/coco/sev/shared.c              |  17 +-
 arch/x86/kernel/cpu/amd.c               |   3 +-
 arch/x86/kernel/kvmclock.c              |   9 +
 arch/x86/kernel/tsc.c                   |  41 ++
 arch/x86/mm/mem_encrypt.c               |   2 +
 arch/x86/mm/mem_encrypt_amd.c           |   4 +
 drivers/virt/coco/sev-guest/sev-guest.c | 485 +-----------------
 arch/x86/Kconfig                        |   1 +
 drivers/virt/coco/sev-guest/Kconfig     |   1 -
 17 files changed, 803 insertions(+), 491 deletions(-)

base-commit: 14fd319494cbd6ae74499a56758415c5dc67691c

Signature · 1 line
2.34.1
#1 lore raw

PATCH v15 1/13 x86/sev: Carve out and export SNP guest messaging init routines

Currently, the sev-guest driver is the only user of SNP guest messaging. All routines for initializing SNP guest messaging are implemented within the sev-guest driver and are not available during early boot. In prepratation for adding Secure TSC guest support, carve out APIs to allocate and initialize guest messaging descriptor context and make it part of coco/sev/core.c. As there is no user of sev_guest_platform_data anymore, remove the structure.

---
 arch/x86/include/asm/sev.h              |  24 ++-
 arch/x86/coco/sev/core.c                | 183 +++++++++++++++++++++-
 drivers/virt/coco/sev-guest/sev-guest.c | 197 +++---------------------
 arch/x86/Kconfig                        |   1 +
 drivers/virt/coco/sev-guest/Kconfig     |   1 -
 5 files changed, 220 insertions(+), 186 deletions(-)
arch/x86/include/asm/sev.h +20 -4
index 91f08af31078..f78c94e29c74 100644
@@ -14,6 +14,7 @@
 #include <asm/insn.h>
 #include <asm/sev-common.h>
 #include <asm/coco.h>
+#include <asm/set_memory.h>
 
 #define GHCB_PROTOCOL_MIN	1ULL
 #define GHCB_PROTOCOL_MAX	2ULL
@@ -170,10 +171,6 @@ struct snp_guest_msg {
 	u8 payload[PAGE_SIZE - sizeof(struct snp_guest_msg_hdr)];
 } __packed;
 
-struct sev_guest_platform_data {
-	u64 secrets_gpa;
-};
-
 struct snp_guest_req {
 	void *req_buf;
 	size_t req_sz;
@@ -253,6 +250,7 @@ struct snp_msg_desc {
 
 	u32 *os_area_msg_seqno;
 	u8 *vmpck;
+	int vmpck_id;
 };
 
 /*
@@ -458,6 +456,20 @@ void set_pte_enc_mask(pte_t *kpte, unsigned long pfn, pgprot_t new_prot);
 void snp_kexec_finish(void);
 void snp_kexec_begin(void);
 
+static inline bool snp_is_vmpck_empty(struct snp_msg_desc *mdesc)
+{
+	static const char zero_key[VMPCK_KEY_LEN] = {0};
+
+	if (mdesc->vmpck)
+		return !memcmp(mdesc->vmpck, zero_key, VMPCK_KEY_LEN);
+
+	return true;
+}
+
+int snp_msg_init(struct snp_msg_desc *mdesc, int vmpck_id);
+struct snp_msg_desc *snp_msg_alloc(void);
+void snp_msg_free(struct snp_msg_desc *mdesc);
+
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
 #define snp_vmpl 0
@@ -498,6 +510,10 @@ static inline int prepare_pte_enc(struct pte_enc_desc *d) { return 0; }
 static inline void set_pte_enc_mask(pte_t *kpte, unsigned long pfn, pgprot_t new_prot) { }
 static inline void snp_kexec_finish(void) { }
 static inline void snp_kexec_begin(void) { }
+static inline bool snp_is_vmpck_empty(struct snp_msg_desc *mdesc) { return false; }
+static inline int snp_msg_init(struct snp_msg_desc *mdesc, int vmpck_id) { return -1; }
+static inline struct snp_msg_desc *snp_msg_alloc(void) { return NULL; }
+static inline void snp_msg_free(struct snp_msg_desc *mdesc) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/coco/sev/core.c +177 -6

201 lines not on this page · raw message

drivers/virt/coco/sev-guest/sev-guest.c +22 -175

300 lines not on this page · raw message

arch/x86/Kconfig +1 -0
index 9d7bd0ae48c4..0f7e3acf37e3 100644
@@ -1559,6 +1559,7 @@ config AMD_MEM_ENCRYPT
 	select ARCH_HAS_CC_PLATFORM
 	select X86_MEM_ENCRYPT
 	select UNACCEPTED_MEMORY
+	select CRYPTO_LIB_AESGCM
 	help
 	  Say yes to enable support for the encryption of system memory.
 	  This requires an AMD processor that supports Secure Memory
drivers/virt/coco/sev-guest/Kconfig +0 -1
index 0b772bd921d8..a6405ab6c2c3 100644
@@ -2,7 +2,6 @@ config SEV_GUEST
 	tristate "AMD SEV Guest driver"
 	default m
 	depends on AMD_MEM_ENCRYPT
-	select CRYPTO_LIB_AESGCM
 	select TSM_REPORTS
 	help
 	  SEV-SNP firmware provides the guest a mechanism to communicate with
Signature · 1 line
2.34.1
maintainer
#2 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 4 lines · Nikunj A Dadhania

Currently, the sev-guest driver is the only user of SNP guest messaging. All routines for initializing SNP guest messaging are implemented within the sev-guest driver and are not available during early boot. In prepratation for adding Secure TSC guest support, carve out APIs to

Unknown word [prepratation] in commit message.
Suggestions: ['preparation', 'preparations', 'reparation', 'perpetration', 'reputation', 'perpetuation', 'peroration', 'presentation', 'repatriation', 'propagation', "preparation's"]

Please introduce a spellchecker into your patch creation workflow.

Quoted 48 lines

allocate and initialize guest messaging descriptor context and make it part of coco/sev/core.c. As there is no user of sev_guest_platform_data anymore, remove the structure.

---
 arch/x86/include/asm/sev.h              |  24 ++-
 arch/x86/coco/sev/core.c                | 183 +++++++++++++++++++++-
 drivers/virt/coco/sev-guest/sev-guest.c | 197 +++---------------------
 arch/x86/Kconfig                        |   1 +
 drivers/virt/coco/sev-guest/Kconfig     |   1 -
 5 files changed, 220 insertions(+), 186 deletions(-)
arch/x86/include/asm/sev.h +2 -4
index 91f08af31078..f78c94e29c74 100644
@@ -14,6 +14,7 @@
 #include <asm/insn.h>
 #include <asm/sev-common.h>
 #include <asm/coco.h>
+#include <asm/set_memory.h>
 
 #define GHCB_PROTOCOL_MIN	1ULL
 #define GHCB_PROTOCOL_MAX	2ULL
@@ -170,10 +171,6 @@ struct snp_guest_msg {
 	u8 payload[PAGE_SIZE - sizeof(struct snp_guest_msg_hdr)];
 } __packed;
 
-struct sev_guest_platform_data {
-	u64 secrets_gpa;
-};
-
 struct snp_guest_req {
 	void *req_buf;
 	size_t req_sz;
@@ -253,6 +250,7 @@ struct snp_msg_desc {
 
 	u32 *os_area_msg_seqno;
 	u8 *vmpck;
+	int vmpck_id;
 };
 
 /*
@@ -458,6 +456,20 @@ void set_pte_enc_mask(pte_t *kpte, unsigned long pfn, pgprot_t new_prot);
 void snp_kexec_finish(void);
 void snp_kexec_begin(void);

arch/x86/include/asm/sev.h void set_pte_enc_mask(pte_t *kpte, unsigned long pfn, pgprot_t new_prot); · line 467

Quoted 10 lines
arch/x86/include/asm/sev.h +9 -0
 
+static inline bool snp_is_vmpck_empty(struct snp_msg_desc *mdesc)
+{
+	static const char zero_key[VMPCK_KEY_LEN] = {0};
+
+	if (mdesc->vmpck)
+		return !memcmp(mdesc->vmpck, zero_key, VMPCK_KEY_LEN);
+
+	return true;
+}

This function looks silly in a header with that array allocation.

I think you should simply do:

	if (memchr_inv(mdesc->vmpck, 0, VMPCK_KEY_LEN))

at the call sites and not have this helper at all.

But please do verify whether what I'm saying actually makes sense and if it does, this can be a cleanup pre-patch.

Quoted 53 lines
arch/x86/include/asm/sev.h +9 -0
+
+int snp_msg_init(struct snp_msg_desc *mdesc, int vmpck_id);
+struct snp_msg_desc *snp_msg_alloc(void);
+void snp_msg_free(struct snp_msg_desc *mdesc);
+
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
 #define snp_vmpl 0
@@ -498,6 +510,10 @@ static inline int prepare_pte_enc(struct pte_enc_desc *d) { return 0; }
 static inline void set_pte_enc_mask(pte_t *kpte, unsigned long pfn, pgprot_t new_prot) { }
 static inline void snp_kexec_finish(void) { }
 static inline void snp_kexec_begin(void) { }
+static inline bool snp_is_vmpck_empty(struct snp_msg_desc *mdesc) { return false; }
+static inline int snp_msg_init(struct snp_msg_desc *mdesc, int vmpck_id) { return -1; }
+static inline struct snp_msg_desc *snp_msg_alloc(void) { return NULL; }
+static inline void snp_msg_free(struct snp_msg_desc *mdesc) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/coco/sev/core.c +3 -6
index c5b0148b8c0a..3cc741eefd06 100644
@@ -25,6 +25,7 @@
 #include <linux/psp-sev.h>
 #include <linux/dmi.h>
 #include <uapi/linux/sev-guest.h>
+#include <crypto/gcm.h>
 
 #include <asm/init.h>
 #include <asm/cpu_entry_area.h>
@@ -2580,15 +2581,9 @@ static struct platform_device sev_guest_device = {
 
 static int __init snp_init_platform_device(void)
 {
-	struct sev_guest_platform_data data;
-
 	if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP))
 		return -ENODEV;
 
-	data.secrets_gpa = secrets_pa;
-	if (platform_device_add_data(&sev_guest_device, &data, sizeof(data)))
-		return -ENODEV;
-
 	if (platform_device_register(&sev_guest_device))
 		return -ENODEV;
 
@@ -2667,3 +2662,179 @@ static int __init sev_sysfs_init(void)
 }
 arch_initcall(sev_sysfs_init);
 #endif // CONFIG_SYSFS
+
+static void free_shared_pages(void *buf, size_t sz)

arch/x86/coco/sev/core.c static int __init sev_sysfs_init(void) · line 2676

Quoted 10 lines
arch/x86/coco/sev/core.c +10 -0
+{
+	unsigned int npages = PAGE_ALIGN(sz) >> PAGE_SHIFT;
+	int ret;
+
+	if (!buf)
+		return;
+
+	ret = set_memory_encrypted((unsigned long)buf, npages);
+	if (ret) {
+		WARN_ONCE(ret, "failed to restore encryption mask (leak it)\n");

Looking at where this lands:

set_memory_encrypted
|-> __set_memory_enc_dec

and that doing now:

        if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) {
                if (!down_read_trylock(&mem_enc_lock))
                        return -EBUSY;

after

859e63b789d6 ("x86/tdx: Convert shared memory back to private on kexec")

we probably should pay attention to this here firing and maybe turning that _trylock() into a normal down_read*

Anyway, just something to pay attention to in the future.

Quoted 5 lines
arch/x86/coco/sev/core.c +5 -0
+		return;
+	}
+
+	__free_pages(virt_to_page(buf), get_order(sz));
+}

...

arch/x86/coco/sev/core.c static int __init sev_sysfs_init(void) · line 2783

Quoted 8 lines
arch/x86/coco/sev/core.c +8 -0
+struct snp_msg_desc *snp_msg_alloc(void)
+{
+	struct snp_msg_desc *mdesc;
+	void __iomem *mem;
+
+	BUILD_BUG_ON(sizeof(struct snp_guest_msg) > PAGE_SIZE);
+
+	mdesc = kzalloc(sizeof(struct snp_msg_desc), GFP_KERNEL);

The above ones use GFP_KERNEL_ACCOUNT. What's the difference?

Quoted 39 lines
arch/x86/coco/sev/core.c +39 -0
+	if (!mdesc)
+		return ERR_PTR(-ENOMEM);
+
+	mem = ioremap_encrypted(secrets_pa, PAGE_SIZE);
+	if (!mem)
+		goto e_free_mdesc;
+
+	mdesc->secrets = (__force struct snp_secrets_page *)mem;
+
+	/* Allocate the shared page used for the request and response message. */
+	mdesc->request = alloc_shared_pages(sizeof(struct snp_guest_msg));
+	if (!mdesc->request)
+		goto e_unmap;
+
+	mdesc->response = alloc_shared_pages(sizeof(struct snp_guest_msg));
+	if (!mdesc->response)
+		goto e_free_request;
+
+	mdesc->certs_data = alloc_shared_pages(SEV_FW_BLOB_MAX_SIZE);
+	if (!mdesc->certs_data)
+		goto e_free_response;
+
+	/* initial the input address for guest request */
+	mdesc->input.req_gpa = __pa(mdesc->request);
+	mdesc->input.resp_gpa = __pa(mdesc->response);
+	mdesc->input.data_gpa = __pa(mdesc->certs_data);
+
+	return mdesc;
+
+e_free_response:
+	free_shared_pages(mdesc->response, sizeof(struct snp_guest_msg));
+e_free_request:
+	free_shared_pages(mdesc->request, sizeof(struct snp_guest_msg));
+e_unmap:
+	iounmap(mem);
+e_free_mdesc:
+	kfree(mdesc);
+
+	return ERR_PTR(-ENOMEM);

arch/x86/coco/sev/core.c static int __init sev_sysfs_init(void) · line 2832

Quoted 10 lines
arch/x86/coco/sev/core.c +10 -0
+}
+EXPORT_SYMBOL_GPL(snp_msg_alloc);
+
+void snp_msg_free(struct snp_msg_desc *mdesc)
+{
+	if (!mdesc)
+		return;
+
+	mdesc->vmpck = NULL;
+	mdesc->os_area_msg_seqno = NULL;
	memset(mdesc, ...);

at the end instead of those assignments.

Quoted 14 lines
arch/x86/coco/sev/core.c +8 -0
+	kfree(mdesc->ctx);
+
+	free_shared_pages(mdesc->response, sizeof(struct snp_guest_msg));
+	free_shared_pages(mdesc->request, sizeof(struct snp_guest_msg));
+	iounmap((__force void __iomem *)mdesc->secrets);
 
 
+	kfree(mdesc);
+}
+EXPORT_SYMBOL_GPL(snp_msg_free);
drivers/virt/coco/sev-guest/sev-guest.c +0 -0
index b699771be029..5268511bc9b8 100644

...

Quoted 36 lines
drivers/virt/coco/sev-guest/sev-guest.c +3 -26
@@ -993,115 +898,57 @@ static int __init sev_guest_probe(struct platform_device *pdev)
 	if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP))
 		return -ENODEV;
 
-	if (!dev->platform_data)
-		return -ENODEV;
-
-	data = (struct sev_guest_platform_data *)dev->platform_data;
-	mapping = ioremap_encrypted(data->secrets_gpa, PAGE_SIZE);
-	if (!mapping)
-		return -ENODEV;
-
-	secrets = (__force void *)mapping;
-
-	ret = -ENOMEM;
 	snp_dev = devm_kzalloc(&pdev->dev, sizeof(struct snp_guest_dev), GFP_KERNEL);
 	if (!snp_dev)
-		goto e_unmap;
-
-	mdesc = devm_kzalloc(&pdev->dev, sizeof(struct snp_msg_desc), GFP_KERNEL);
-	if (!mdesc)
-		goto e_unmap;
-
-	/* Adjust the default VMPCK key based on the executing VMPL level */
-	if (vmpck_id == -1)
-		vmpck_id = snp_vmpl;
+		return -ENOMEM;
 
-	ret = -EINVAL;
-	mdesc->vmpck = get_vmpck(vmpck_id, secrets, &mdesc->os_area_msg_seqno);
-	if (!mdesc->vmpck) {
-		dev_err(dev, "Invalid VMPCK%d communication key\n", vmpck_id);
-		goto e_unmap;
-	}
+	mdesc = snp_msg_alloc();
+	if (IS_ERR_OR_NULL(mdesc))

drivers/virt/coco/sev-guest/sev-guest.c static int __init sev_guest_probe(struct platform_device *pdev) · line 911

Quoted 10 lines
drivers/virt/coco/sev-guest/sev-guest.c +4 -5
+		return -ENOMEM;
 
-	/* Verify that VMPCK is not zero. */
-	if (is_vmpck_empty(mdesc)) {
-		dev_err(dev, "Empty VMPCK%d communication key\n", vmpck_id);
-		goto e_unmap;
-	}
+	ret = snp_msg_init(mdesc, vmpck_id);
+	if (ret)
+		return -EIO;

You just leaked mdesc here.

Audit all your error paths.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#3 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 10 lines · Borislav Petkov
Quoted 4 lines · Nikunj A Dadhania

Currently, the sev-guest driver is the only user of SNP guest messaging. All routines for initializing SNP guest messaging are implemented within the sev-guest driver and are not available during early boot. In prepratation for adding Secure TSC guest support, carve out APIs to

Unknown word [prepratation] in commit message.
Suggestions: ['preparation', 'preparations', 'reparation', 'perpetration', 'reputation', 'perpetuation', 'peroration', 'presentation', 'repatriation', 'propagation', "preparation's"]

Please introduce a spellchecker into your patch creation workflow.

This is what I use with checkpatch, that didnt catch the wrong spelling. Do you suggest using something else ?

./scripts/checkpatch.pl --codespell < sectsc_v15/v15-0001-x86-sev-Carve-out-and-export-SNP-guest-messaging.patch
total: 0 errors, 0 warnings, 569 lines checked

"[PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging" has no obvious style problems and is ready for submission.

Regards
Nikunj

maintainer
#4 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 1 line · Nikunj A. Dadhania

This is what I use with checkpatch, that didnt catch the wrong spelling.

Not surprised.

Quoted 1 line

Do you suggest using something else ?

You can enable spellchecking in your editor with which you write the commit messages. For example:

https://www.linux.com/training-tutorials/using-spell-checking-vim/

Or, you can use my tool:

https://git.kernel.org/pub/scm/linux/kernel/git/bp/bp.git/log/?h=vp

You'd need to fish it out of the repo.

It doesn't completely replace checkpatch yet but I am extending it with features as I go. But it does spellcheck:

$ ~/dev/vp/.tip/bin/vp.py ~/tmp/review/new
prepratation for adding Secure TSC guest support, carve out APIs to
Unknown word [prepratation] in commit message.
Suggestions: ['preparation', 'preparations', 'reparation', 'perpetration', 'reputation', 'perpetuation', 'peroration', 'presentation', 'repatriation', 'propagation', "preparation's"]

Class patch:
original subject: [[PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines]
subject: [x86/sev: Carve out and export SNP guest messaging init routines]
sender: [Nikunj A Dadhania <[email protected]>]
author: [Nikunj A Dadhania <[email protected]>]
version: [15]
number: [1]
name: [x86-sev-carve_out_and_export_snp_guest_messaging_init_routines]
date: [Tue, 03 Dec 2024 14:30:33 +0530]
message-id: [[email protected]]

I'm sure there are gazillion other ways to automate it, ofc.

HTH.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#5 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 10 lines · Borislav Petkov
Quoted 1 line · Nikunj A. Dadhania

This is what I use with checkpatch, that didnt catch the wrong spelling.

Not surprised.

Quoted 1 line

Do you suggest using something else ?

You can enable spellchecking in your editor with which you write the commit messages. For example:

Quoted 7 lines

https://www.linux.com/training-tutorials/using-spell-checking-vim/

Or, you can use my tool:

https://git.kernel.org/pub/scm/linux/kernel/git/bp/bp.git/log/?h=vp

You'd need to fish it out of the repo.

Sure will give it a try.

Regards
Nikunj

#6 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

In reply to Borislav Petkov
Quoted 17 lines · Borislav Petkov

Review void set_pte_enc_mask(pte_t *kpte, unsigned long pfn, pgprot_t new_prot); · line 467

Quoted 13 lines · Nikunj A Dadhania
(quoted diff) +9 -0
@@ -458,6 +456,20 @@ void set_pte_enc_mask(pte_t *kpte, unsigned long pfn, pgprot_t new_prot);
 void snp_kexec_finish(void);
 void snp_kexec_begin(void);
 
+static inline bool snp_is_vmpck_empty(struct snp_msg_desc *mdesc)
+{
+	static const char zero_key[VMPCK_KEY_LEN] = {0};
+
+	if (mdesc->vmpck)
+		return !memcmp(mdesc->vmpck, zero_key, VMPCK_KEY_LEN);
+
+	return true;
+}

This function looks silly in a header with that array allocation.

Quoted 3 lines

I think you should simply do:

	if (memchr_inv(mdesc->vmpck, 0, VMPCK_KEY_LEN))

Just a minor nit, it will need a negation:

if (!memchr_inv(mdesc->vmpck, 0, VMPCK_KEY_LEN))

Quoted 4 lines

at the call sites and not have this helper at all.

But please do verify whether what I'm saying actually makes sense and if it does, this can be a cleanup pre-patch.

Yes, it makes sense. I have verified the code and below is the cleanup pre-patch.

From 4d249f393aeba7bed7fb99778b8ee8a24a33b5b7 Mon Sep 17 00:00:00 2001
From: Nikunj A Dadhania <[email protected]>
Date: Tue, 3 Dec 2024 20:48:28 +0530
Subject: [PATCH] virt: sev-guest: Remove is_vmpck_empty() helper

Remove the is_vmpck_empty() helper function, which uses a local array allocation to check if the VMPCK is empty. Replace it with memchr_inv() to directly determine if the VMPCK is empty without additional memory allocation.

---
 drivers/virt/coco/sev-guest/sev-guest.c | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)
drivers/virt/coco/sev-guest/sev-guest.c +2 -12
index b699771be029..62328d0b2cb6 100644
@@ -63,16 +63,6 @@ MODULE_PARM_DESC(vmpck_id, "The VMPCK ID to use when communicating with the PSP.
 /* Mutex to serialize the shared buffer access and command handling. */
 static DEFINE_MUTEX(snp_cmd_mutex);
 
-static bool is_vmpck_empty(struct snp_msg_desc *mdesc)
-{
-	char zero_key[VMPCK_KEY_LEN] = {0};
-
-	if (mdesc->vmpck)
-		return !memcmp(mdesc->vmpck, zero_key, VMPCK_KEY_LEN);
-
-	return true;
-}
-
 /*
  * If an error is received from the host or AMD Secure Processor (ASP) there
  * are two options. Either retry the exact same encrypted request or discontinue
@@ -335,7 +325,7 @@ static int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_r
 	guard(mutex)(&snp_cmd_mutex);
 
 	/* Check if the VMPCK is not empty */
-	if (is_vmpck_empty(mdesc)) {
+	if (!mdesc->vmpck || !memchr_inv(mdesc->vmpck, 0, VMPCK_KEY_LEN)) {
 		pr_err_ratelimited("VMPCK is disabled\n");
 		return -ENOTTY;
 	}
@@ -1024,7 +1014,7 @@ static int __init sev_guest_probe(struct platform_device *pdev)
 	}
 
 	/* Verify that VMPCK is not zero. */
-	if (is_vmpck_empty(mdesc)) {
+	if (!memchr_inv(mdesc->vmpck, 0, VMPCK_KEY_LEN)) {
 		dev_err(dev, "Empty VMPCK%d communication key\n", vmpck_id);
 		goto e_unmap;
 	}
Signature · 1 line
2.34.1
#7 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

In reply to Borislav Petkov
Quoted 31 lines · Borislav Petkov
Quoted 6 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +2 -0
@@ -2667,3 +2662,179 @@ static int __init sev_sysfs_init(void)
 }
 arch_initcall(sev_sysfs_init);
 #endif // CONFIG_SYSFS
+
+static void free_shared_pages(void *buf, size_t sz)

arch/x86/coco/sev/core.c static int __init sev_sysfs_init(void) · line 2676

Quoted 10 lines
arch/x86/coco/sev/core.c +10 -0
+{
+	unsigned int npages = PAGE_ALIGN(sz) >> PAGE_SHIFT;
+	int ret;
+
+	if (!buf)
+		return;
+
+	ret = set_memory_encrypted((unsigned long)buf, npages);
+	if (ret) {
+		WARN_ONCE(ret, "failed to restore encryption mask (leak it)\n");

Looking at where this lands:

set_memory_encrypted
|-> __set_memory_enc_dec

and that doing now:

        if (cc_platform_has(CC_ATTR_MEM_ENCRYPT)) {
                if (!down_read_trylock(&mem_enc_lock))
                        return -EBUSY;
Quoted 8 lines

after

859e63b789d6 ("x86/tdx: Convert shared memory back to private on kexec")

we probably should pay attention to this here firing and maybe turning that _trylock() into a normal down_read*

Anyway, just something to pay attention to in the future.

Yes, will keep an eye.

Quoted 9 lines
Quoted 6 lines
arch/x86/coco/sev/core.c +5 -0
+		return;
+	}
+
+	__free_pages(virt_to_page(buf), get_order(sz));
+}

...

Quoted 10 lines

arch/x86/coco/sev/core.c static int __init sev_sysfs_init(void) · line 2783

Quoted 8 lines
arch/x86/coco/sev/core.c +8 -0
+struct snp_msg_desc *snp_msg_alloc(void)
+{
+	struct snp_msg_desc *mdesc;
+	void __iomem *mem;
+
+	BUILD_BUG_ON(sizeof(struct snp_guest_msg) > PAGE_SIZE);
+
+	mdesc = kzalloc(sizeof(struct snp_msg_desc), GFP_KERNEL);

The above ones use GFP_KERNEL_ACCOUNT. What's the difference?

The above ones I have retained old code.

GFP_KERNEL_ACCOUNT allocation are accounted in kmemcg and the below note from[1]
----------------------------------------------------------------------------
Untrusted allocations triggered from userspace should be a subject of kmem accounting and must have __GFP_ACCOUNT bit set. There is the handy GFP_KERNEL_ACCOUNT shortcut for GFP_KERNEL allocations that should be accounted.
----------------------------------------------------------------------------

For mdesc, I had kept it similar to snp_dev allocation, that is why it is having GFP_KERNEL.

        snp_dev = devm_kzalloc(&pdev->dev, sizeof(struct snp_guest_dev), GFP_KERNEL);
        if (!snp_dev)

- goto e_unmap;
-
- mdesc = devm_kzalloc(&pdev->dev, sizeof(struct snp_msg_desc), GFP_KERNEL);

Let me know if mdesc allocation need to be GFP_KERNEL_ACCOUNT.

Quoted 11 lines

arch/x86/coco/sev/core.c static int __init sev_sysfs_init(void) · line 2832

Quoted 7 lines
arch/x86/coco/sev/core.c +7 -0
+void snp_msg_free(struct snp_msg_desc *mdesc)
+{
+	if (!mdesc)
+		return;
+
+	mdesc->vmpck = NULL;
+	mdesc->os_area_msg_seqno = NULL;
	memset(mdesc, ...);

at the end instead of those assignments.

Sure.

Quoted 56 lines
Quoted 15 lines
arch/x86/coco/sev/core.c +8 -0
+	kfree(mdesc->ctx);
+
+	free_shared_pages(mdesc->response, sizeof(struct snp_guest_msg));
+	free_shared_pages(mdesc->request, sizeof(struct snp_guest_msg));
+	iounmap((__force void __iomem *)mdesc->secrets);
 
 
+	kfree(mdesc);
+}
+EXPORT_SYMBOL_GPL(snp_msg_free);
drivers/virt/coco/sev-guest/sev-guest.c +0 -0
index b699771be029..5268511bc9b8 100644

...

Quoted 38 lines
drivers/virt/coco/sev-guest/sev-guest.c +4 -26
@@ -993,115 +898,57 @@ static int __init sev_guest_probe(struct platform_device *pdev)
 	if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP))
 		return -ENODEV;
 
-	if (!dev->platform_data)
-		return -ENODEV;
-
-	data = (struct sev_guest_platform_data *)dev->platform_data;
-	mapping = ioremap_encrypted(data->secrets_gpa, PAGE_SIZE);
-	if (!mapping)
-		return -ENODEV;
-
-	secrets = (__force void *)mapping;
-
-	ret = -ENOMEM;
 	snp_dev = devm_kzalloc(&pdev->dev, sizeof(struct snp_guest_dev), GFP_KERNEL);
 	if (!snp_dev)
-		goto e_unmap;
-
-	mdesc = devm_kzalloc(&pdev->dev, sizeof(struct snp_msg_desc), GFP_KERNEL);
-	if (!mdesc)
-		goto e_unmap;
-
-	/* Adjust the default VMPCK key based on the executing VMPL level */
-	if (vmpck_id == -1)
-		vmpck_id = snp_vmpl;
+		return -ENOMEM;
 
-	ret = -EINVAL;
-	mdesc->vmpck = get_vmpck(vmpck_id, secrets, &mdesc->os_area_msg_seqno);
-	if (!mdesc->vmpck) {
-		dev_err(dev, "Invalid VMPCK%d communication key\n", vmpck_id);
-		goto e_unmap;
-	}
+	mdesc = snp_msg_alloc();
+	if (IS_ERR_OR_NULL(mdesc))
+		return -ENOMEM;
 
Quoted 10 lines

drivers/virt/coco/sev-guest/sev-guest.c static int __init sev_guest_probe(struct platform_device *pdev) · line 911

Quoted 8 lines
drivers/virt/coco/sev-guest/sev-guest.c +3 -5
-	/* Verify that VMPCK is not zero. */
-	if (is_vmpck_empty(mdesc)) {
-		dev_err(dev, "Empty VMPCK%d communication key\n", vmpck_id);
-		goto e_unmap;
-	}
+	ret = snp_msg_init(mdesc, vmpck_id);
+	if (ret)
+		return -EIO;

You just leaked mdesc here.

Right

Quoted 1 line

Audit all your error paths.

Sure I will audit and send updated patch.

Regards
Nikunj

1) https://www.kernel.org/doc/html/v6.12/core-api/memory-allocation.html

maintainer
#8 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 1 line · Nikunj A. Dadhania

The above ones I have retained old code.

Right.

Quoted 6 lines

GFP_KERNEL_ACCOUNT allocation are accounted in kmemcg and the below note from[1]
----------------------------------------------------------------------------
Untrusted allocations triggered from userspace should be a subject of kmem accounting and must have __GFP_ACCOUNT bit set. There is the handy GFP_KERNEL_ACCOUNT shortcut for GFP_KERNEL allocations that should be accounted.
----------------------------------------------------------------------------

Interesting.

Quoted 10 lines

For mdesc, I had kept it similar to snp_dev allocation, that is why it is having GFP_KERNEL.

        snp_dev = devm_kzalloc(&pdev->dev, sizeof(struct snp_guest_dev), GFP_KERNEL);
        if (!snp_dev)
drivers/virt/coco/sev-guest/sev-guest.c +0 -3
-               goto e_unmap;
-
-       mdesc = devm_kzalloc(&pdev->dev, sizeof(struct snp_msg_desc), GFP_KERNEL);

Let me know if mdesc allocation need to be GFP_KERNEL_ACCOUNT.

Let's audit that thing:

* snp_init_crypto - not really untrusted allocation. It is on the driver probe path.

* get_report - I don't think so:

        /*      
         * The intermediate response buffer is used while decrypting the
         * response payload. Make sure that it has enough space to cover the
         * authtag.
         */
        resp_len = sizeof(report_resp->data) + mdesc->ctx->authsize;
        report_resp = kzalloc(resp_len, GFP_KERNEL_ACCOUNT);

That resp_len is limited and that's on the guest_ioctl path which cannot happen concurrently?

* get_ext_report - ditto

* alloc_shared_pages - all the allocations are limited but I guess that could remain _ACCOUNT as a measure for future robustness.

And that was it.

So AFAICT, only one use case is semi-valid.

So maybe we should convert those remaining ones to boring GFP_KERNEL...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#9 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 40 lines · Borislav Petkov
Quoted 1 line · Nikunj A. Dadhania

The above ones I have retained old code.

Right.

Quoted 6 lines

GFP_KERNEL_ACCOUNT allocation are accounted in kmemcg and the below note from[1]
----------------------------------------------------------------------------
Untrusted allocations triggered from userspace should be a subject of kmem accounting and must have __GFP_ACCOUNT bit set. There is the handy GFP_KERNEL_ACCOUNT shortcut for GFP_KERNEL allocations that should be accounted.
----------------------------------------------------------------------------

Interesting.

Quoted 10 lines

For mdesc, I had kept it similar to snp_dev allocation, that is why it is having GFP_KERNEL.

        snp_dev = devm_kzalloc(&pdev->dev, sizeof(struct snp_guest_dev), GFP_KERNEL);
        if (!snp_dev)
drivers/virt/coco/sev-guest/sev-guest.c +0 -3
-               goto e_unmap;
-
-       mdesc = devm_kzalloc(&pdev->dev, sizeof(struct snp_msg_desc), GFP_KERNEL);

Let me know if mdesc allocation need to be GFP_KERNEL_ACCOUNT.

Let's audit that thing:

* snp_init_crypto - not really untrusted allocation. It is on the driver probe path.

* get_report - I don't think so:

        /*      
         * The intermediate response buffer is used while decrypting the
         * response payload. Make sure that it has enough space to cover the
         * authtag.
         */
        resp_len = sizeof(report_resp->data) + mdesc->ctx->authsize;
        report_resp = kzalloc(resp_len, GFP_KERNEL_ACCOUNT);
Quoted 2 lines

That resp_len is limited and that's on the guest_ioctl path which cannot happen concurrently?

It is a trusted allocation, but should it be accounted as it is part of the userspace ioctl path ?

Quoted 5 lines

* get_ext_report - ditto

* alloc_shared_pages - all the allocations are limited but I guess that could remain _ACCOUNT as a measure for future robustness.

Ok.

Quoted 6 lines

And that was it.

So AFAICT, only one use case is semi-valid.

So maybe we should convert those remaining ones to boring GFP_KERNEL...

Sure, let me add this as a pre-patch.

Regards,
Nikunj

maintainer
#10 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 10 lines · Nikunj A. Dadhania
Quoted 10 lines

* get_report - I don't think so:

        /*      
         * The intermediate response buffer is used while decrypting the
         * response payload. Make sure that it has enough space to cover the
         * authtag.
         */
        resp_len = sizeof(report_resp->data) + mdesc->ctx->authsize;
        report_resp = kzalloc(resp_len, GFP_KERNEL_ACCOUNT);
Quoted 5 lines
Quoted 2 lines

That resp_len is limited and that's on the guest_ioctl path which cannot happen concurrently?

It is a trusted allocation, but should it be accounted as it is part of the userspace ioctl path ?

Well, it is unlocked_ioctl() and snp_guest_ioctl() is not taking any locks. What's stopping anyone from writing a nasty little program which hammers the sev-guest on the ioctl interface until the OOM killer activates?

IOW, this should probably remain _ACCOUNT AFAICT.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#11 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 5 lines

Well, it is unlocked_ioctl() and snp_guest_ioctl() is not taking any locks. What's stopping anyone from writing a nasty little program which hammers the sev-guest on the ioctl interface until the OOM killer activates?

Given sev-guest requires heightened privileges, can we not assume a reasonable user space? I thought that was an organizing principle.

Quoted 8 lines

IOW, this should probably remain _ACCOUNT AFAICT.

--
Regards/Gruss,
Boris.

https://people.kernel.org/tglx/notes-about-netiquette

Signature · 1 line
-Dionna Glaze, PhD, CISSP, CCSP (she/her)
maintainer
#12 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 2 lines · Dionna Amalie Glaze

Given sev-guest requires heightened privileges, can we not assume a reasonable user space?

And that sev-guest driver runs in the guest so the worst that can happen is, the guest gets killed for misbehaving. Oh well...

I guess that's ok.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#13 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

In reply to Borislav Petkov
Quoted 14 lines · Borislav Petkov
Quoted 13 lines · Nikunj A. Dadhania
Quoted 12 lines

* get_report - I don't think so:

        /*      
         * The intermediate response buffer is used while decrypting the
         * response payload. Make sure that it has enough space to cover the
         * authtag.
         */
        resp_len = sizeof(report_resp->data) + mdesc->ctx->authsize;
        report_resp = kzalloc(resp_len, GFP_KERNEL_ACCOUNT);

That resp_len is limited and that's on the guest_ioctl path which cannot happen concurrently?

Quoted 8 lines
Quoted 2 lines

It is a trusted allocation, but should it be accounted as it is part of the userspace ioctl path ?

Well, it is unlocked_ioctl() and snp_guest_ioctl() is not taking any locks. What's stopping anyone from writing a nasty little program which hammers the sev-guest on the ioctl interface until the OOM killer activates?

IOW, this should probably remain _ACCOUNT AFAICT.

Both get_report()/get_ext_report() are in the unlocked_ioctl(), we will retain the _ACCOUNT

That leaves us with only one site: snp_init_crypto(), should I fold this change in current patch ?
Regards
Nikunj

maintainer
#14 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 2 lines · Nikunj A. Dadhania

That leaves us with only one site: snp_init_crypto(), should I fold this change in current patch ?

Nah, a pre-patch pls.

Along with an explanation summing up our discussion in the commit message. This patch is already doing enough.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#15 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Nikunj A. Dadhania

That leaves us with only one site: snp_init_crypto(), should I fold this change in current patch ?

Nah, a pre-patch pls.

Along with an explanation summing up our discussion in the commit message. This patch is already doing enough.

From: Nikunj A Dadhania <[email protected]>
Date: Thu, 5 Dec 2024 12:00:56 +0530
Subject: [PATCH] virt: sev-guest: Replace GFP_KERNEL_ACCOUNT with GFP_KERNEL

Replace GFP_KERNEL_ACCOUNT with GFP_KERNEL in the sev-guest driver code. GFP_KERNEL_ACCOUNT is typically used for accounting untrusted userspace allocations. After auditing the sev-guest code, the following changes are necessary:

* snp_init_crypto(): Use GFP_KERNEL as this is a trusted device probe path.

Retain GFP_KERNEL_ACCOUNT in the following cases for robustness and specific path requirements:

* alloc_shared_pages(): Although all allocations are limited, retain GFP_KERNEL_ACCOUNT for future robustness.

* get_report() and get_ext_report(): These functions are on the unlocked ioctl path and should continue using GFP_KERNEL_ACCOUNT.

---
 drivers/virt/coco/sev-guest/sev-guest.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
(quoted diff) +0 -0

b/drivers/virt/coco/sev-guest/sev-guest.c
index 62328d0b2cb6..250ce92d816b 100644

drivers/virt/coco/sev-guest/sev-guest.c +0 -0
@@ -141,7 +141,7 @@ static struct aesgcm_ctx *snp_init_crypto(u8 *key, size_t

keylen)
{
struct aesgcm_ctx *ctx;

- ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT);
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
if (!ctx)
return NULL;

Signature · 1 line
2.34.1
#16 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

In reply to Nikunj A Dadhania
Quoted 4 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +0 -0
index c5b0148b8c0a..3cc741eefd06 100644

...

Quoted 13 lines
arch/x86/coco/sev/core.c +10 -0
+void snp_msg_free(struct snp_msg_desc *mdesc)
+{
+	if (!mdesc)
+		return;
+
+	mdesc->vmpck = NULL;
+	mdesc->os_area_msg_seqno = NULL;
+	kfree(mdesc->ctx);
+
+	free_shared_pages(mdesc->response, sizeof(struct

snp_guest_msg));

arch/x86/coco/sev/core.c +1 -0
+	free_shared_pages(mdesc->request, sizeof(struct

snp_guest_msg));

arch/x86/coco/sev/core.c static int __init sev_sysfs_init(void) · line 2838

Quoted 2 lines
arch/x86/coco/sev/core.c +2 -0
+	iounmap((__force void __iomem *)mdesc->secrets);
+	kfree(mdesc);

This is leaking mdesc->certs_data.

#17 lore raw

Re: [PATCH v15 01/13] x86/sev: Carve out and export SNP guest messaging init routines

Quoted 16 lines · Francesco Lavra
Quoted 4 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +0 -0
index c5b0148b8c0a..3cc741eefd06 100644

...

Quoted 9 lines
arch/x86/coco/sev/core.c +9 -0
+void snp_msg_free(struct snp_msg_desc *mdesc)
+{
+	if (!mdesc)
+		return;
+
+	mdesc->vmpck = NULL;
+	mdesc->os_area_msg_seqno = NULL;
+	kfree(mdesc->ctx);
+
Quoted 8 lines

arch/x86/coco/sev/core.c static int __init sev_sysfs_init(void) · line 2838

Quoted 6 lines
arch/x86/coco/sev/core.c +1 -0
+	free_shared_pages(mdesc->response, sizeof(struct

snp_guest_msg));

arch/x86/coco/sev/core.c +1 -0
+	free_shared_pages(mdesc->request, sizeof(struct

snp_guest_msg));

arch/x86/coco/sev/core.c +2 -0
+	iounmap((__force void __iomem *)mdesc->secrets);
+	kfree(mdesc);

This is leaking mdesc->certs_data.

Ack, will fix in the next version.

Regards
Nikunj

#18 lore raw

PATCH v15 2/13 x86/sev: Relocate SNP guest messaging routines to common code

In reply to Nikunj A Dadhania

At present, the SEV guest driver exclusively handles SNP guest messaging. All routines for sending guest messages are embedded within the guest driver. To support Secure TSC, SEV-SNP guests must communicate with the AMD Security Processor during early boot. However, these guest messaging functions are not accessible during early boot since they are currently part of the guest driver.

Hence, relocate the core SNP guest messaging functions to SEV common code and provide an API for sending SNP guest messages.

No functional change, but just an export symbol added for snp_send_guest_request() and dropped the export symbol on snp_issue_guest_request() and made it static.

---
 arch/x86/include/asm/sev.h              |  14 +-
 arch/x86/coco/sev/core.c                | 295 +++++++++++++++++++++++-
 drivers/virt/coco/sev-guest/sev-guest.c | 292 -----------------------
 3 files changed, 299 insertions(+), 302 deletions(-)
arch/x86/include/asm/sev.h +7 -7
index f78c94e29c74..53f3048f484e 100644
@@ -125,6 +125,9 @@ struct snp_req_data {
 #define AAD_LEN			48
 #define MSG_HDR_VER		1
 
+#define SNP_REQ_MAX_RETRY_DURATION      (60*HZ)
+#define SNP_REQ_RETRY_DELAY             (2*HZ)
+
 /* See SNP spec SNP_GUEST_REQUEST section for the structure */
 enum msg_type {
 	SNP_MSG_TYPE_INVALID = 0,
@@ -443,8 +446,6 @@ void snp_set_wakeup_secondary_cpu(void);
 bool snp_init(struct boot_params *bp);
 void __noreturn snp_abort(void);
 void snp_dmi_setup(void);
-int snp_issue_guest_request(struct snp_guest_req *req, struct snp_req_data *input,
-			    struct snp_guest_request_ioctl *rio);
 int snp_issue_svsm_attest_req(u64 call_id, struct svsm_call *call, struct svsm_attest_call *input);
 void snp_accept_memory(phys_addr_t start, phys_addr_t end);
 u64 snp_get_unsupported_features(u64 status);
@@ -469,6 +470,8 @@ static inline bool snp_is_vmpck_empty(struct snp_msg_desc *mdesc)
 int snp_msg_init(struct snp_msg_desc *mdesc, int vmpck_id);
 struct snp_msg_desc *snp_msg_alloc(void);
 void snp_msg_free(struct snp_msg_desc *mdesc);
+int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
+			   struct snp_guest_request_ioctl *rio);
 
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
@@ -492,11 +495,6 @@ static inline void snp_set_wakeup_secondary_cpu(void) { }
 static inline bool snp_init(struct boot_params *bp) { return false; }
 static inline void snp_abort(void) { }
 static inline void snp_dmi_setup(void) { }
-static inline int snp_issue_guest_request(struct snp_guest_req *req, struct snp_req_data *input,
-					  struct snp_guest_request_ioctl *rio)
-{
-	return -ENOTTY;
-}
 static inline int snp_issue_svsm_attest_req(u64 call_id, struct svsm_call *call, struct svsm_attest_call *input)
 {
 	return -ENOTTY;
@@ -514,6 +512,8 @@ static inline bool snp_is_vmpck_empty(struct snp_msg_desc *mdesc) { return false
 static inline int snp_msg_init(struct snp_msg_desc *mdesc, int vmpck_id) { return -1; }
 static inline struct snp_msg_desc *snp_msg_alloc(void) { return NULL; }
 static inline void snp_msg_free(struct snp_msg_desc *mdesc) { }
+static inline int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
+					 struct snp_guest_request_ioctl *rio) { return -ENODEV; }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/coco/sev/core.c +292 -3

310 lines not on this page · raw message

drivers/virt/coco/sev-guest/sev-guest.c +0 -292

310 lines not on this page · raw message

Signature · 1 line
2.34.1
maintainer
#19 lore raw

Re: [PATCH v15 02/13] x86/sev: Relocate SNP guest messaging routines to common code

arch/x86/coco/sev/core.c void snp_msg_free(struct snp_msg_desc *mdesc) · line 3129

Quoted 11 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +11 -0
+	rc = verify_and_dec_payload(mdesc, req);
+	if (rc) {
+		pr_alert("Detected unexpected decode failure from ASP. rc: %d\n", rc);
+		snp_disable_vmpck(mdesc);
+		return rc;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(snp_send_guest_request);
+

Applying: x86/sev: Relocate SNP guest messaging routines to common code .git/rebase-apply/patch:376: new blank line at EOF.
+
warning: 1 line adds whitespace errors.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#20 lore raw

Re: [PATCH v15 02/13] x86/sev: Relocate SNP guest messaging routines to common code

Quoted 18 lines · Borislav Petkov

arch/x86/coco/sev/core.c void snp_msg_free(struct snp_msg_desc *mdesc) · line 3129

Quoted 11 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +11 -0
+	rc = verify_and_dec_payload(mdesc, req);
+	if (rc) {
+		pr_alert("Detected unexpected decode failure from ASP. rc: %d\n", rc);
+		snp_disable_vmpck(mdesc);
+		return rc;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(snp_send_guest_request);
+

Applying: x86/sev: Relocate SNP guest messaging routines to common code .git/rebase-apply/patch:376: new blank line at EOF.
+
warning: 1 line adds whitespace errors.

Sure will fix it.

Regards
Nikunj

#21 lore raw

PATCH v15 3/13 x86/sev: Add Secure TSC support for SNP guests

In reply to Nikunj A Dadhania

Add support for Secure TSC in SNP-enabled guests. Secure TSC allows guests to securely use RDTSC/RDTSCP instructions, ensuring that the parameters used cannot be altered by the hypervisor once the guest is launched.

Secure TSC-enabled guests need to query TSC information from the AMD Security Processor. This communication channel is encrypted between the AMD Security Processor and the guest, with the hypervisor acting merely as a conduit to deliver the guest messages to the AMD Security Processor. Each message is protected with AEAD (AES-256 GCM).

---
 arch/x86/include/asm/sev-common.h |   1 +
 arch/x86/include/asm/sev.h        |  22 ++++++
 arch/x86/include/asm/svm.h        |   6 +-
 include/linux/cc_platform.h       |   8 +++
 arch/x86/coco/core.c              |   3 +
 arch/x86/coco/sev/core.c          | 116 ++++++++++++++++++++++++++++++
 arch/x86/mm/mem_encrypt.c         |   2 +
 7 files changed, 156 insertions(+), 2 deletions(-)
arch/x86/include/asm/sev-common.h +1 -0
index 50f5666938c0..6ef92432a5ce 100644
@@ -206,6 +206,7 @@ struct snp_psc_desc {
 #define GHCB_TERM_NO_SVSM		7	/* SVSM is not advertised in the secrets page */
 #define GHCB_TERM_SVSM_VMPL0		8	/* SVSM is present but has set VMPL to 0 */
 #define GHCB_TERM_SVSM_CAA		9	/* SVSM is present but CAA is not page aligned */
+#define GHCB_TERM_SECURE_TSC		10	/* Secure TSC initialization failed */
 
 #define GHCB_RESP_CODE(v)		((v) & GHCB_MSR_INFO_MASK)
 
arch/x86/include/asm/sev.h +22 -0
index 53f3048f484e..9fd02efef08e 100644
@@ -146,6 +146,9 @@ enum msg_type {
 	SNP_MSG_VMRK_REQ,
 	SNP_MSG_VMRK_RSP,
 
+	SNP_MSG_TSC_INFO_REQ = 17,
+	SNP_MSG_TSC_INFO_RSP,
+
 	SNP_MSG_TYPE_MAX
 };
 
@@ -174,6 +177,22 @@ struct snp_guest_msg {
 	u8 payload[PAGE_SIZE - sizeof(struct snp_guest_msg_hdr)];
 } __packed;
 
+#define SNP_TSC_INFO_REQ_SZ	128
+#define SNP_TSC_INFO_RESP_SZ	128
+
+struct snp_tsc_info_req {
+	u8 rsvd[SNP_TSC_INFO_REQ_SZ];
+} __packed;
+
+struct snp_tsc_info_resp {
+	u32 status;
+	u32 rsvd1;
+	u64 tsc_scale;
+	u64 tsc_offset;
+	u32 tsc_factor;
+	u8 rsvd2[100];
+} __packed;
+
 struct snp_guest_req {
 	void *req_buf;
 	size_t req_sz;
@@ -473,6 +492,8 @@ void snp_msg_free(struct snp_msg_desc *mdesc);
 int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
 			   struct snp_guest_request_ioctl *rio);
 
+void __init snp_secure_tsc_prepare(void);
+
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
 #define snp_vmpl 0
@@ -514,6 +535,7 @@ static inline struct snp_msg_desc *snp_msg_alloc(void) { return NULL; }
 static inline void snp_msg_free(struct snp_msg_desc *mdesc) { }
 static inline int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
 					 struct snp_guest_request_ioctl *rio) { return -ENODEV; }
+static inline void __init snp_secure_tsc_prepare(void) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/include/asm/svm.h +4 -2
index 2b59b9951c90..92e18798f197 100644
@@ -417,7 +417,9 @@ struct sev_es_save_area {
 	u8 reserved_0x298[80];
 	u32 pkru;
 	u32 tsc_aux;
-	u8 reserved_0x2f0[24];
+	u64 tsc_scale;
+	u64 tsc_offset;
+	u8 reserved_0x300[8];
 	u64 rcx;
 	u64 rdx;
 	u64 rbx;
@@ -564,7 +566,7 @@ static inline void __unused_size_checks(void)
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x1c0);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x248);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x298);
-	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x2f0);
+	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x300);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x320);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x380);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x3f0);
include/linux/cc_platform.h +8 -0
index caa4b4430634..cb7103dc124f 100644
@@ -88,6 +88,14 @@ enum cc_attr {
 	 * enabled to run SEV-SNP guests.
 	 */
 	CC_ATTR_HOST_SEV_SNP,
+
+	/**
+	 * @CC_ATTR_GUEST_SNP_SECURE_TSC: SNP Secure TSC is active.
+	 *
+	 * The platform/OS is running as a guest/virtual machine and actively
+	 * using AMD SEV-SNP Secure TSC feature.
+	 */
+	CC_ATTR_GUEST_SNP_SECURE_TSC,
 };
 
 #ifdef CONFIG_ARCH_HAS_CC_PLATFORM
arch/x86/coco/core.c +3 -0
index 0f81f70aca82..5b9a358a3254 100644
@@ -100,6 +100,9 @@ static bool noinstr amd_cc_platform_has(enum cc_attr attr)
 	case CC_ATTR_HOST_SEV_SNP:
 		return cc_flags.host_sev_snp;
 
+	case CC_ATTR_GUEST_SNP_SECURE_TSC:
+		return sev_status & MSR_AMD64_SNP_SECURE_TSC;
+
 	default:
 		return false;
 	}
arch/x86/coco/sev/core.c +116 -0

131 lines not on this page · raw message

arch/x86/mm/mem_encrypt.c +2 -0
index 0a120d85d7bb..95bae74fdab2 100644
@@ -94,6 +94,8 @@ void __init mem_encrypt_init(void)
 	/* Call into SWIOTLB to update the SWIOTLB DMA buffers */
 	swiotlb_update_mem_attributes();
 
+	snp_secure_tsc_prepare();
+
 	print_mem_encrypt_feature_info();
 }
 
Signature · 1 line
2.34.1
maintainer
#22 lore raw

Re: [PATCH v15 03/13] x86/sev: Add Secure TSC support for SNP guests

Quoted 10 lines · Nikunj A Dadhania

Add support for Secure TSC in SNP-enabled guests. Secure TSC allows guests to securely use RDTSC/RDTSCP instructions, ensuring that the parameters used cannot be altered by the hypervisor once the guest is launched.

Secure TSC-enabled guests need to query TSC information from the AMD Security Processor. This communication channel is encrypted between the AMD Security Processor and the guest, with the hypervisor acting merely as a conduit to deliver the guest messages to the AMD Security Processor. Each message is protected with AEAD (AES-256 GCM).

Quoted 5 lines

This patch changed somewhat from last time. When did Peter test it again and Tom review it again?

arch/x86/coco/sev/core.c static u64 sev_hv_features __ro_after_init; · line 100

Quoted 10 lines
arch/x86/coco/sev/core.c +2 -0
index a61898c7f114..39683101b526 100644
@@ -96,6 +96,14 @@ static u64 sev_hv_features __ro_after_init;
 /* Secrets page physical address from the CC blob */
 static u64 secrets_pa __ro_after_init;
 
+/*
+ * For Secure TSC guests, the BP fetches TSC_INFO using SNP guest messaging and

s/BP/BSP/

Quoted 9 lines
arch/x86/coco/sev/core.c +6 -0
+ * initializes snp_tsc_scale and snp_tsc_offset. These values are replicated
+ * across the APs VMSA fields (TSC_SCALE and TSC_OFFSET).
+ */
+static u64 snp_tsc_scale __ro_after_init;
+static u64 snp_tsc_offset __ro_after_init;
+
 /* #VC handler runtime per-CPU data */
 struct sev_es_runtime_data {
 	struct ghcb ghcb_page;

...

arch/x86/coco/sev/core.c int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req · line 3205

Quoted 6 lines
arch/x86/coco/sev/core.c +6 -0
+	memcpy(tsc_resp, buf, sizeof(*tsc_resp));
+	pr_debug("%s: response status 0x%x scale 0x%llx offset 0x%llx factor 0x%x\n",
+		 __func__, tsc_resp->status, tsc_resp->tsc_scale, tsc_resp->tsc_offset,
+		 tsc_resp->tsc_factor);
+
+	if (tsc_resp->status == 0) {

Like the last time:

	if (!tsc_resp->status)
Quoted 26 lines
arch/x86/coco/sev/core.c +26 -0
+		snp_tsc_scale = tsc_resp->tsc_scale;
+		snp_tsc_offset = tsc_resp->tsc_offset;
+	} else {
+		pr_err("Failed to get TSC info, response status 0x%x\n", tsc_resp->status);
+		rc = -EIO;
+	}
+
+e_request:
+	/* The response buffer contains sensitive data, explicitly clear it. */
+	memzero_explicit(buf, sizeof(buf));
+	memzero_explicit(tsc_resp, sizeof(*tsc_resp));
+e_free_mdesc:
+	snp_msg_free(mdesc);
+e_free_buf:
+	kfree(buf);
+e_free_rio:
+	kfree(rio);
+e_free_req:
+	kfree(req);
+ e_free_tsc_resp:
+	kfree(tsc_resp);
+e_free_tsc_req:
+	kfree(tsc_req);
+
+	return rc;
+}
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#23 lore raw

Re: [PATCH v15 03/13] x86/sev: Add Secure TSC support for SNP guests

Quoted 11 lines · Borislav Petkov
Quoted 10 lines · Nikunj A Dadhania

Add support for Secure TSC in SNP-enabled guests. Secure TSC allows guests to securely use RDTSC/RDTSCP instructions, ensuring that the parameters used cannot be altered by the hypervisor once the guest is launched.

Secure TSC-enabled guests need to query TSC information from the AMD Security Processor. This communication channel is encrypted between the AMD Security Processor and the guest, with the hypervisor acting merely as a conduit to deliver the guest messages to the AMD Security Processor. Each message is protected with AEAD (AES-256 GCM).

Quoted 7 lines
Quoted 5 lines

This patch changed somewhat from last time.

Yes, most of the change was dynamic allocation in snp_get_tsc_info().

Quoted 2 lines

When did Peter test it again and
Tom review it again?

It makes sense to drop both.

Quoted 35 lines

arch/x86/coco/sev/core.c static u64 sev_hv_features __ro_after_init; · line 100

Quoted 11 lines
arch/x86/coco/sev/core.c +2 -0
index a61898c7f114..39683101b526 100644
@@ -96,6 +96,14 @@ static u64 sev_hv_features __ro_after_init;
 /* Secrets page physical address from the CC blob */
 static u64 secrets_pa __ro_after_init;
 
+/*
+ * For Secure TSC guests, the BP fetches TSC_INFO using SNP guest messaging and

s/BP/BSP/

Quoted 9 lines
arch/x86/coco/sev/core.c +6 -0
+ * initializes snp_tsc_scale and snp_tsc_offset. These values are replicated
+ * across the APs VMSA fields (TSC_SCALE and TSC_OFFSET).
+ */
+static u64 snp_tsc_scale __ro_after_init;
+static u64 snp_tsc_offset __ro_after_init;
+
 /* #VC handler runtime per-CPU data */
 struct sev_es_runtime_data {
 	struct ghcb ghcb_page;

...

arch/x86/coco/sev/core.c int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req · line 3205

Quoted 6 lines
arch/x86/coco/sev/core.c +6 -0
+	memcpy(tsc_resp, buf, sizeof(*tsc_resp));
+	pr_debug("%s: response status 0x%x scale 0x%llx offset 0x%llx factor 0x%x\n",
+		 __func__, tsc_resp->status, tsc_resp->tsc_scale, tsc_resp->tsc_offset,
+		 tsc_resp->tsc_factor);
+
+	if (tsc_resp->status == 0) {

Like the last time:

Quoted 1 line
	if (!tsc_resp->status)

Ack.

Regards
Nikunj

#24 lore raw

Re: [PATCH v15 03/13] x86/sev: Add Secure TSC support for SNP guests

In reply to Nikunj A Dadhania
Quoted 10 lines · Nikunj A Dadhania

Add support for Secure TSC in SNP-enabled guests. Secure TSC allows guests to securely use RDTSC/RDTSCP instructions, ensuring that the parameters used cannot be altered by the hypervisor once the guest is launched.

Secure TSC-enabled guests need to query TSC information from the AMD Security Processor. This communication channel is encrypted between the AMD Security Processor and the guest, with the hypervisor acting merely as a conduit to deliver the guest messages to the AMD Security Processor. Each message is protected with AEAD (AES-256 GCM).

Quoted 3 lines

Just some minor nits if you have to respin...

Quoted 106 lines
---
 arch/x86/include/asm/sev-common.h |   1 +
 arch/x86/include/asm/sev.h        |  22 ++++++
 arch/x86/include/asm/svm.h        |   6 +-
 include/linux/cc_platform.h       |   8 +++
 arch/x86/coco/core.c              |   3 +
 arch/x86/coco/sev/core.c          | 116 ++++++++++++++++++++++++++++++
 arch/x86/mm/mem_encrypt.c         |   2 +
 7 files changed, 156 insertions(+), 2 deletions(-)
arch/x86/include/asm/sev-common.h +1 -0
index 50f5666938c0..6ef92432a5ce 100644
@@ -206,6 +206,7 @@ struct snp_psc_desc {
 #define GHCB_TERM_NO_SVSM		7	/* SVSM is not advertised in the secrets page */
 #define GHCB_TERM_SVSM_VMPL0		8	/* SVSM is present but has set VMPL to 0 */
 #define GHCB_TERM_SVSM_CAA		9	/* SVSM is present but CAA is not page aligned */
+#define GHCB_TERM_SECURE_TSC		10	/* Secure TSC initialization failed */
 
 #define GHCB_RESP_CODE(v)		((v) & GHCB_MSR_INFO_MASK)
 
arch/x86/include/asm/sev.h +22 -0
index 53f3048f484e..9fd02efef08e 100644
@@ -146,6 +146,9 @@ enum msg_type {
 	SNP_MSG_VMRK_REQ,
 	SNP_MSG_VMRK_RSP,
 
+	SNP_MSG_TSC_INFO_REQ = 17,
+	SNP_MSG_TSC_INFO_RSP,
+
 	SNP_MSG_TYPE_MAX
 };
 
@@ -174,6 +177,22 @@ struct snp_guest_msg {
 	u8 payload[PAGE_SIZE - sizeof(struct snp_guest_msg_hdr)];
 } __packed;
 
+#define SNP_TSC_INFO_REQ_SZ	128
+#define SNP_TSC_INFO_RESP_SZ	128
+
+struct snp_tsc_info_req {
+	u8 rsvd[SNP_TSC_INFO_REQ_SZ];
+} __packed;
+
+struct snp_tsc_info_resp {
+	u32 status;
+	u32 rsvd1;
+	u64 tsc_scale;
+	u64 tsc_offset;
+	u32 tsc_factor;
+	u8 rsvd2[100];
+} __packed;
+
 struct snp_guest_req {
 	void *req_buf;
 	size_t req_sz;
@@ -473,6 +492,8 @@ void snp_msg_free(struct snp_msg_desc *mdesc);
 int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
 			   struct snp_guest_request_ioctl *rio);
 
+void __init snp_secure_tsc_prepare(void);
+
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
 #define snp_vmpl 0
@@ -514,6 +535,7 @@ static inline struct snp_msg_desc *snp_msg_alloc(void) { return NULL; }
 static inline void snp_msg_free(struct snp_msg_desc *mdesc) { }
 static inline int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
 					 struct snp_guest_request_ioctl *rio) { return -ENODEV; }
+static inline void __init snp_secure_tsc_prepare(void) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/include/asm/svm.h +4 -2
index 2b59b9951c90..92e18798f197 100644
@@ -417,7 +417,9 @@ struct sev_es_save_area {
 	u8 reserved_0x298[80];
 	u32 pkru;
 	u32 tsc_aux;
-	u8 reserved_0x2f0[24];
+	u64 tsc_scale;
+	u64 tsc_offset;
+	u8 reserved_0x300[8];
 	u64 rcx;
 	u64 rdx;
 	u64 rbx;
@@ -564,7 +566,7 @@ static inline void __unused_size_checks(void)
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x1c0);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x248);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x298);
-	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x2f0);
+	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x300);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x320);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x380);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x3f0);
include/linux/cc_platform.h +0 -0
index caa4b4430634..cb7103dc124f 100644
@@ -88,6 +88,14 @@ enum cc_attr {
 	 * enabled to run SEV-SNP guests.

include/linux/cc_platform.h enum cc_attr { · line 98

Quoted 10 lines
include/linux/cc_platform.h +8 -0
 	 */
 	CC_ATTR_HOST_SEV_SNP,
+
+	/**
+	 * @CC_ATTR_GUEST_SNP_SECURE_TSC: SNP Secure TSC is active.
+	 *
+	 * The platform/OS is running as a guest/virtual machine and actively
+	 * using AMD SEV-SNP Secure TSC feature.
+	 */
+	CC_ATTR_GUEST_SNP_SECURE_TSC,

Maybe move this up above the host related attribute so that it is grouped with the other guest attributes.

Quoted 8 lines
include/linux/cc_platform.h +0 -0
 };
 
 #ifdef CONFIG_ARCH_HAS_CC_PLATFORM
arch/x86/coco/core.c +0 -0
index 0f81f70aca82..5b9a358a3254 100644

arch/x86/coco/core.c static bool noinstr amd_cc_platform_has(enum cc_attr attr) · line 105

Quoted 6 lines
arch/x86/coco/core.c +3 -0
@@ -100,6 +100,9 @@ static bool noinstr amd_cc_platform_has(enum cc_attr attr)
 	case CC_ATTR_HOST_SEV_SNP:
 		return cc_flags.host_sev_snp;
 
+	case CC_ATTR_GUEST_SNP_SECURE_TSC:
+		return sev_status & MSR_AMD64_SNP_SECURE_TSC;
+

Ditto here. Move this up above the host check.

Also, should this be:

	return (sev_status & MSR_AMD64_SEV_SNP_ENABLED) &&
	       (sev_status & MSR_AMD64_SNP_SECURE_TSC);

?

Quoted 122 lines
arch/x86/coco/core.c +0 -0
 	default:
 		return false;
 	}
arch/x86/coco/sev/core.c +97 -0
index a61898c7f114..39683101b526 100644
@@ -96,6 +96,14 @@ static u64 sev_hv_features __ro_after_init;
 /* Secrets page physical address from the CC blob */
 static u64 secrets_pa __ro_after_init;
 
+/*
+ * For Secure TSC guests, the BP fetches TSC_INFO using SNP guest messaging and
+ * initializes snp_tsc_scale and snp_tsc_offset. These values are replicated
+ * across the APs VMSA fields (TSC_SCALE and TSC_OFFSET).
+ */
+static u64 snp_tsc_scale __ro_after_init;
+static u64 snp_tsc_offset __ro_after_init;
+
 /* #VC handler runtime per-CPU data */
 struct sev_es_runtime_data {
 	struct ghcb ghcb_page;
@@ -1277,6 +1285,12 @@ static int wakeup_cpu_via_vmgexit(u32 apic_id, unsigned long start_ip)
 	vmsa->vmpl		= snp_vmpl;
 	vmsa->sev_features	= sev_status >> 2;
 
+	/* Populate AP's TSC scale/offset to get accurate TSC values. */
+	if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC)) {
+		vmsa->tsc_scale = snp_tsc_scale;
+		vmsa->tsc_offset = snp_tsc_offset;
+	}
+
 	/* Switch the page over to a VMSA page now that it is initialized */
 	ret = snp_set_vmsa(vmsa, caa, apic_id, true);
 	if (ret) {
@@ -3127,3 +3141,105 @@ int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req
 }
 EXPORT_SYMBOL_GPL(snp_send_guest_request);
 
+static int __init snp_get_tsc_info(void)
+{
+	struct snp_guest_request_ioctl *rio;
+	struct snp_tsc_info_resp *tsc_resp;
+	struct snp_tsc_info_req *tsc_req;
+	struct snp_msg_desc *mdesc;
+	struct snp_guest_req *req;
+	unsigned char *buf;
+	int rc = -ENOMEM;
+
+	tsc_req = kzalloc(sizeof(*tsc_req), GFP_KERNEL);
+	if (!tsc_req)
+		return rc;
+
+	tsc_resp = kzalloc(sizeof(*tsc_resp), GFP_KERNEL);
+	if (!tsc_resp)
+		goto e_free_tsc_req;
+
+	req = kzalloc(sizeof(*req), GFP_KERNEL);
+	if (!req)
+		goto e_free_tsc_resp;
+
+	rio = kzalloc(sizeof(*rio), GFP_KERNEL);
+	if (!rio)
+		goto e_free_req;
+
+	/*
+	 * The intermediate response buffer is used while decrypting the
+	 * response payload. Make sure that it has enough space to cover
+	 * the authtag.
+	 */
+	buf = kzalloc(SNP_TSC_INFO_RESP_SZ + AUTHTAG_LEN, GFP_KERNEL);
+	if (!buf)
+		goto e_free_rio;
+
+	mdesc = snp_msg_alloc();
+	if (IS_ERR_OR_NULL(mdesc))
+		goto e_free_buf;
+
+	rc = snp_msg_init(mdesc, snp_vmpl);
+	if (rc)
+		goto e_free_mdesc;
+
+	req->msg_version = MSG_HDR_VER;
+	req->msg_type = SNP_MSG_TSC_INFO_REQ;
+	req->vmpck_id = snp_vmpl;
+	req->req_buf = tsc_req;
+	req->req_sz = sizeof(*tsc_req);
+	req->resp_buf = buf;
+	req->resp_sz = sizeof(*tsc_resp) + AUTHTAG_LEN;
+	req->exit_code = SVM_VMGEXIT_GUEST_REQUEST;
+
+	rc = snp_send_guest_request(mdesc, req, rio);
+	if (rc)
+		goto e_request;
+
+	memcpy(tsc_resp, buf, sizeof(*tsc_resp));
+	pr_debug("%s: response status 0x%x scale 0x%llx offset 0x%llx factor 0x%x\n",
+		 __func__, tsc_resp->status, tsc_resp->tsc_scale, tsc_resp->tsc_offset,
+		 tsc_resp->tsc_factor);
+
+	if (tsc_resp->status == 0) {
+		snp_tsc_scale = tsc_resp->tsc_scale;
+		snp_tsc_offset = tsc_resp->tsc_offset;
+	} else {
+		pr_err("Failed to get TSC info, response status 0x%x\n", tsc_resp->status);
+		rc = -EIO;
+	}
+
+e_request:
+	/* The response buffer contains sensitive data, explicitly clear it. */
+	memzero_explicit(buf, sizeof(buf));
+	memzero_explicit(tsc_resp, sizeof(*tsc_resp));
+e_free_mdesc:
+	snp_msg_free(mdesc);
+e_free_buf:
+	kfree(buf);
+e_free_rio:
+	kfree(rio);
+e_free_req:
+	kfree(req);
+ e_free_tsc_resp:
+	kfree(tsc_resp);

arch/x86/coco/sev/core.c int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req · line 3236

Quoted 10 lines
arch/x86/coco/sev/core.c +10 -0
+e_free_tsc_req:
+	kfree(tsc_req);
+
+	return rc;
+}
+
+void __init snp_secure_tsc_prepare(void)
+{
+	if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP) ||
+	    !cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))

If you make the change above, you only need to check for SNP_SECURE_TSC.

Thanks,
Tom

Quoted 22 lines
arch/x86/coco/sev/core.c +9 -0
+		return;
+
+	if (snp_get_tsc_info()) {
+		pr_alert("Unable to retrieve Secure TSC info from ASP\n");
+		sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_SECURE_TSC);
+	}
+
+	pr_debug("SecureTSC enabled");
+}
arch/x86/mm/mem_encrypt.c +2 -0
index 0a120d85d7bb..95bae74fdab2 100644
@@ -94,6 +94,8 @@ void __init mem_encrypt_init(void)
 	/* Call into SWIOTLB to update the SWIOTLB DMA buffers */
 	swiotlb_update_mem_attributes();
 
+	snp_secure_tsc_prepare();
+
 	print_mem_encrypt_feature_info();
 }
 
#25 lore raw

Re: [PATCH v15 03/13] x86/sev: Add Secure TSC support for SNP guests

Quoted 11 lines · Tom Lendacky
Quoted 10 lines · Nikunj A Dadhania

Add support for Secure TSC in SNP-enabled guests. Secure TSC allows guests to securely use RDTSC/RDTSCP instructions, ensuring that the parameters used cannot be altered by the hypervisor once the guest is launched.

Secure TSC-enabled guests need to query TSC information from the AMD Security Processor. This communication channel is encrypted between the AMD Security Processor and the guest, with the hypervisor acting merely as a conduit to deliver the guest messages to the AMD Security Processor. Each message is protected with AEAD (AES-256 GCM).

Quoted 5 lines
Quoted 3 lines

Just some minor nits if you have to respin...

Yes, I will be spinning a new version.

Quoted 112 lines
Quoted 112 lines
---
 arch/x86/include/asm/sev-common.h |   1 +
 arch/x86/include/asm/sev.h        |  22 ++++++
 arch/x86/include/asm/svm.h        |   6 +-
 include/linux/cc_platform.h       |   8 +++
 arch/x86/coco/core.c              |   3 +
 arch/x86/coco/sev/core.c          | 116 ++++++++++++++++++++++++++++++
 arch/x86/mm/mem_encrypt.c         |   2 +
 7 files changed, 156 insertions(+), 2 deletions(-)
arch/x86/include/asm/sev-common.h +1 -0
index 50f5666938c0..6ef92432a5ce 100644
@@ -206,6 +206,7 @@ struct snp_psc_desc {
 #define GHCB_TERM_NO_SVSM		7	/* SVSM is not advertised in the secrets page */
 #define GHCB_TERM_SVSM_VMPL0		8	/* SVSM is present but has set VMPL to 0 */
 #define GHCB_TERM_SVSM_CAA		9	/* SVSM is present but CAA is not page aligned */
+#define GHCB_TERM_SECURE_TSC		10	/* Secure TSC initialization failed */
 
 #define GHCB_RESP_CODE(v)		((v) & GHCB_MSR_INFO_MASK)
 
arch/x86/include/asm/sev.h +22 -0
index 53f3048f484e..9fd02efef08e 100644
@@ -146,6 +146,9 @@ enum msg_type {
 	SNP_MSG_VMRK_REQ,
 	SNP_MSG_VMRK_RSP,
 
+	SNP_MSG_TSC_INFO_REQ = 17,
+	SNP_MSG_TSC_INFO_RSP,
+
 	SNP_MSG_TYPE_MAX
 };
 
@@ -174,6 +177,22 @@ struct snp_guest_msg {
 	u8 payload[PAGE_SIZE - sizeof(struct snp_guest_msg_hdr)];
 } __packed;
 
+#define SNP_TSC_INFO_REQ_SZ	128
+#define SNP_TSC_INFO_RESP_SZ	128
+
+struct snp_tsc_info_req {
+	u8 rsvd[SNP_TSC_INFO_REQ_SZ];
+} __packed;
+
+struct snp_tsc_info_resp {
+	u32 status;
+	u32 rsvd1;
+	u64 tsc_scale;
+	u64 tsc_offset;
+	u32 tsc_factor;
+	u8 rsvd2[100];
+} __packed;
+
 struct snp_guest_req {
 	void *req_buf;
 	size_t req_sz;
@@ -473,6 +492,8 @@ void snp_msg_free(struct snp_msg_desc *mdesc);
 int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
 			   struct snp_guest_request_ioctl *rio);
 
+void __init snp_secure_tsc_prepare(void);
+
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
 #define snp_vmpl 0
@@ -514,6 +535,7 @@ static inline struct snp_msg_desc *snp_msg_alloc(void) { return NULL; }
 static inline void snp_msg_free(struct snp_msg_desc *mdesc) { }
 static inline int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
 					 struct snp_guest_request_ioctl *rio) { return -ENODEV; }
+static inline void __init snp_secure_tsc_prepare(void) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/include/asm/svm.h +4 -2
index 2b59b9951c90..92e18798f197 100644
@@ -417,7 +417,9 @@ struct sev_es_save_area {
 	u8 reserved_0x298[80];
 	u32 pkru;
 	u32 tsc_aux;
-	u8 reserved_0x2f0[24];
+	u64 tsc_scale;
+	u64 tsc_offset;
+	u8 reserved_0x300[8];
 	u64 rcx;
 	u64 rdx;
 	u64 rbx;
@@ -564,7 +566,7 @@ static inline void __unused_size_checks(void)
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x1c0);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x248);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x298);
-	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x2f0);
+	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x300);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x320);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x380);
 	BUILD_BUG_RESERVED_OFFSET(sev_es_save_area, 0x3f0);
include/linux/cc_platform.h +3 -0
index caa4b4430634..cb7103dc124f 100644
@@ -88,6 +88,14 @@ enum cc_attr {
 	 * enabled to run SEV-SNP guests.
 	 */
 	CC_ATTR_HOST_SEV_SNP,
+
+	/**
+	 * @CC_ATTR_GUEST_SNP_SECURE_TSC: SNP Secure TSC is active.
Quoted 8 lines

include/linux/cc_platform.h enum cc_attr { · line 98

Quoted 5 lines
include/linux/cc_platform.h +5 -0
+	 *
+	 * The platform/OS is running as a guest/virtual machine and actively
+	 * using AMD SEV-SNP Secure TSC feature.
+	 */
+	CC_ATTR_GUEST_SNP_SECURE_TSC,

Maybe move this up above the host related attribute so that it is grouped with the other guest attributes.

Sure

Quoted 12 lines
Quoted 12 lines
include/linux/cc_platform.h +0 -0
 };
 
 #ifdef CONFIG_ARCH_HAS_CC_PLATFORM
arch/x86/coco/core.c +0 -0
index 0f81f70aca82..5b9a358a3254 100644
@@ -100,6 +100,9 @@ static bool noinstr amd_cc_platform_has(enum cc_attr attr)
 	case CC_ATTR_HOST_SEV_SNP:
 		return cc_flags.host_sev_snp;
 
Quoted 5 lines

arch/x86/coco/core.c static bool noinstr amd_cc_platform_has(enum cc_attr attr) · line 105

Quoted 3 lines
arch/x86/coco/core.c +3 -0
+	case CC_ATTR_GUEST_SNP_SECURE_TSC:
+		return sev_status & MSR_AMD64_SNP_SECURE_TSC;
+

Ditto here. Move this up above the host check.

Ack.

Quoted 7 lines

Also, should this be:

	return (sev_status & MSR_AMD64_SEV_SNP_ENABLED) &&
	       (sev_status & MSR_AMD64_SNP_SECURE_TSC);

?

Yes, we can do this and the below mentioned change.

Quoted 126 lines
Quoted 126 lines
arch/x86/coco/core.c +0 -0
 	default:
 		return false;
 	}
arch/x86/coco/sev/core.c +101 -0
index a61898c7f114..39683101b526 100644
@@ -96,6 +96,14 @@ static u64 sev_hv_features __ro_after_init;
 /* Secrets page physical address from the CC blob */
 static u64 secrets_pa __ro_after_init;
 
+/*
+ * For Secure TSC guests, the BP fetches TSC_INFO using SNP guest messaging and
+ * initializes snp_tsc_scale and snp_tsc_offset. These values are replicated
+ * across the APs VMSA fields (TSC_SCALE and TSC_OFFSET).
+ */
+static u64 snp_tsc_scale __ro_after_init;
+static u64 snp_tsc_offset __ro_after_init;
+
 /* #VC handler runtime per-CPU data */
 struct sev_es_runtime_data {
 	struct ghcb ghcb_page;
@@ -1277,6 +1285,12 @@ static int wakeup_cpu_via_vmgexit(u32 apic_id, unsigned long start_ip)
 	vmsa->vmpl		= snp_vmpl;
 	vmsa->sev_features	= sev_status >> 2;
 
+	/* Populate AP's TSC scale/offset to get accurate TSC values. */
+	if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC)) {
+		vmsa->tsc_scale = snp_tsc_scale;
+		vmsa->tsc_offset = snp_tsc_offset;
+	}
+
 	/* Switch the page over to a VMSA page now that it is initialized */
 	ret = snp_set_vmsa(vmsa, caa, apic_id, true);
 	if (ret) {
@@ -3127,3 +3141,105 @@ int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req
 }
 EXPORT_SYMBOL_GPL(snp_send_guest_request);
 
+static int __init snp_get_tsc_info(void)
+{
+	struct snp_guest_request_ioctl *rio;
+	struct snp_tsc_info_resp *tsc_resp;
+	struct snp_tsc_info_req *tsc_req;
+	struct snp_msg_desc *mdesc;
+	struct snp_guest_req *req;
+	unsigned char *buf;
+	int rc = -ENOMEM;
+
+	tsc_req = kzalloc(sizeof(*tsc_req), GFP_KERNEL);
+	if (!tsc_req)
+		return rc;
+
+	tsc_resp = kzalloc(sizeof(*tsc_resp), GFP_KERNEL);
+	if (!tsc_resp)
+		goto e_free_tsc_req;
+
+	req = kzalloc(sizeof(*req), GFP_KERNEL);
+	if (!req)
+		goto e_free_tsc_resp;
+
+	rio = kzalloc(sizeof(*rio), GFP_KERNEL);
+	if (!rio)
+		goto e_free_req;
+
+	/*
+	 * The intermediate response buffer is used while decrypting the
+	 * response payload. Make sure that it has enough space to cover
+	 * the authtag.
+	 */
+	buf = kzalloc(SNP_TSC_INFO_RESP_SZ + AUTHTAG_LEN, GFP_KERNEL);
+	if (!buf)
+		goto e_free_rio;
+
+	mdesc = snp_msg_alloc();
+	if (IS_ERR_OR_NULL(mdesc))
+		goto e_free_buf;
+
+	rc = snp_msg_init(mdesc, snp_vmpl);
+	if (rc)
+		goto e_free_mdesc;
+
+	req->msg_version = MSG_HDR_VER;
+	req->msg_type = SNP_MSG_TSC_INFO_REQ;
+	req->vmpck_id = snp_vmpl;
+	req->req_buf = tsc_req;
+	req->req_sz = sizeof(*tsc_req);
+	req->resp_buf = buf;
+	req->resp_sz = sizeof(*tsc_resp) + AUTHTAG_LEN;
+	req->exit_code = SVM_VMGEXIT_GUEST_REQUEST;
+
+	rc = snp_send_guest_request(mdesc, req, rio);
+	if (rc)
+		goto e_request;
+
+	memcpy(tsc_resp, buf, sizeof(*tsc_resp));
+	pr_debug("%s: response status 0x%x scale 0x%llx offset 0x%llx factor 0x%x\n",
+		 __func__, tsc_resp->status, tsc_resp->tsc_scale, tsc_resp->tsc_offset,
+		 tsc_resp->tsc_factor);
+
+	if (tsc_resp->status == 0) {
+		snp_tsc_scale = tsc_resp->tsc_scale;
+		snp_tsc_offset = tsc_resp->tsc_offset;
+	} else {
+		pr_err("Failed to get TSC info, response status 0x%x\n", tsc_resp->status);
+		rc = -EIO;
+	}
+
+e_request:
+	/* The response buffer contains sensitive data, explicitly clear it. */
+	memzero_explicit(buf, sizeof(buf));
+	memzero_explicit(tsc_resp, sizeof(*tsc_resp));
+e_free_mdesc:
+	snp_msg_free(mdesc);
+e_free_buf:
+	kfree(buf);
+e_free_rio:
+	kfree(rio);
+e_free_req:
+	kfree(req);
+ e_free_tsc_resp:
+	kfree(tsc_resp);
+e_free_tsc_req:
+	kfree(tsc_req);
+
+	return rc;
Quoted 8 lines

arch/x86/coco/sev/core.c int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req · line 3236

Quoted 6 lines
arch/x86/coco/sev/core.c +6 -0
+}
+
+void __init snp_secure_tsc_prepare(void)
+{
+	if (!cc_platform_has(CC_ATTR_GUEST_SEV_SNP) ||
+	    !cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))

If you make the change above, you only need to check for SNP_SECURE_TSC.

Ack

Regards
Nikunj

#26 lore raw

Re: [PATCH v15 03/13] x86/sev: Add Secure TSC support for SNP guests

In reply to Nikunj A Dadhania
Quoted 120 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +2 -0
index a61898c7f114..39683101b526 100644
@@ -96,6 +96,14 @@ static u64 sev_hv_features __ro_after_init;
 /* Secrets page physical address from the CC blob */
 static u64 secrets_pa __ro_after_init;
 
+/*
+ * For Secure TSC guests, the BP fetches TSC_INFO using SNP guest

messaging and
+ * initializes snp_tsc_scale and snp_tsc_offset. These values are replicated
+ * across the APs VMSA fields (TSC_SCALE and TSC_OFFSET).
+ */

arch/x86/coco/sev/core.c +4 -0
+static u64 snp_tsc_scale __ro_after_init;
+static u64 snp_tsc_offset __ro_after_init;
+
 /* #VC handler runtime per-CPU data */
 struct sev_es_runtime_data {
 	struct ghcb ghcb_page;
@@ -1277,6 +1285,12 @@ static int wakeup_cpu_via_vmgexit(u32 apic_id,
 unsigned long start_ip)
 	vmsa->vmpl		= snp_vmpl;
 	vmsa->sev_features	= sev_status >> 2;
 
+	/* Populate AP's TSC scale/offset to get accurate TSC

values. */

arch/x86/coco/sev/core.c +5 -0
+	if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC)) {
+		vmsa->tsc_scale = snp_tsc_scale;
+		vmsa->tsc_offset = snp_tsc_offset;
+	}
+
 	/* Switch the page over to a VMSA page now that it is

initialized */

arch/x86/coco/sev/core.c +0 -0
 	ret = snp_set_vmsa(vmsa, caa, apic_id, true);
 	if (ret) {
@@ -3127,3 +3141,105 @@ int snp_send_guest_request(struct

snp_msg_desc *mdesc, struct snp_guest_req *req
}
EXPORT_SYMBOL_GPL(snp_send_guest_request);

arch/x86/coco/sev/core.c +28 -0
+static int __init snp_get_tsc_info(void)
+{
+	struct snp_guest_request_ioctl *rio;
+	struct snp_tsc_info_resp *tsc_resp;
+	struct snp_tsc_info_req *tsc_req;
+	struct snp_msg_desc *mdesc;
+	struct snp_guest_req *req;
+	unsigned char *buf;
+	int rc = -ENOMEM;
+
+	tsc_req = kzalloc(sizeof(*tsc_req), GFP_KERNEL);
+	if (!tsc_req)
+		return rc;
+
+	tsc_resp = kzalloc(sizeof(*tsc_resp), GFP_KERNEL);
+	if (!tsc_resp)
+		goto e_free_tsc_req;
+
+	req = kzalloc(sizeof(*req), GFP_KERNEL);
+	if (!req)
+		goto e_free_tsc_resp;
+
+	rio = kzalloc(sizeof(*rio), GFP_KERNEL);
+	if (!rio)
+		goto e_free_req;
+
+	/*
+	 * The intermediate response buffer is used while decrypting

the

arch/x86/coco/sev/core.c +1 -0
+	 * response payload. Make sure that it has enough space to

cover

arch/x86/coco/sev/core.c +3 -0
+	 * the authtag.
+	 */
+	buf = kzalloc(SNP_TSC_INFO_RESP_SZ + AUTHTAG_LEN,

GFP_KERNEL);

arch/x86/coco/sev/core.c +26 -0
+	if (!buf)
+		goto e_free_rio;
+
+	mdesc = snp_msg_alloc();
+	if (IS_ERR_OR_NULL(mdesc))
+		goto e_free_buf;
+
+	rc = snp_msg_init(mdesc, snp_vmpl);
+	if (rc)
+		goto e_free_mdesc;
+
+	req->msg_version = MSG_HDR_VER;
+	req->msg_type = SNP_MSG_TSC_INFO_REQ;
+	req->vmpck_id = snp_vmpl;
+	req->req_buf = tsc_req;
+	req->req_sz = sizeof(*tsc_req);
+	req->resp_buf = buf;
+	req->resp_sz = sizeof(*tsc_resp) + AUTHTAG_LEN;
+	req->exit_code = SVM_VMGEXIT_GUEST_REQUEST;
+
+	rc = snp_send_guest_request(mdesc, req, rio);
+	if (rc)
+		goto e_request;
+
+	memcpy(tsc_resp, buf, sizeof(*tsc_resp));
+	pr_debug("%s: response status 0x%x scale 0x%llx offset

0x%llx factor 0x%x\n",

arch/x86/coco/sev/core.c +1 -0
+		 __func__, tsc_resp->status, tsc_resp->tsc_scale,

tsc_resp->tsc_offset,

arch/x86/coco/sev/core.c +7 -0
+		 tsc_resp->tsc_factor);
+
+	if (tsc_resp->status == 0) {
+		snp_tsc_scale = tsc_resp->tsc_scale;
+		snp_tsc_offset = tsc_resp->tsc_offset;
+	} else {
+		pr_err("Failed to get TSC info, response status

0x%x\n", tsc_resp->status);

arch/x86/coco/sev/core.c +5 -0
+		rc = -EIO;
+	}
+
+e_request:
+	/* The response buffer contains sensitive data, explicitly

clear it. */

arch/x86/coco/sev/core.c int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req · line 3216

Quoted 2 lines
arch/x86/coco/sev/core.c +2 -0
+	memzero_explicit(buf, sizeof(buf));
+	memzero_explicit(tsc_resp, sizeof(*tsc_resp));

buf is an unsigned char *, so by using sizeof(buf) you are not zeroing the entire buffer.
Also, I see no point in having a separate tsc_resp buffer just to copy the response from buf to tsc_resp, if you just use a single buffer with size (SNP_TSC_INFO_RESP_SZ + AUTHTAG_LEN) and parse the response from that buffer you will avoid the double buffer allocation, the memory copying, and the double zeroing.

#27 lore raw

Re: [PATCH v15 03/13] x86/sev: Add Secure TSC support for SNP guests

Quoted 16 lines · Francesco Lavra

arch/x86/coco/sev/core.c int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req · line 3216

Quoted 6 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +3 -0
+
+e_request:
+	/* The response buffer contains sensitive data, explicitly

clear it. */

arch/x86/coco/sev/core.c +2 -0
+	memzero_explicit(buf, sizeof(buf));
+	memzero_explicit(tsc_resp, sizeof(*tsc_resp));

buf is an unsigned char *, so by using sizeof(buf) you are not zeroing the entire buffer.
Also, I see no point in having a separate tsc_resp buffer just to copy the response from buf to tsc_resp, if you just use a single buffer with size (SNP_TSC_INFO_RESP_SZ + AUTHTAG_LEN) and parse the response from that buffer you will avoid the double buffer allocation, the memory copying, and the double zeroing.

Makes sense, will update.

Regards
Nikunj

#28 lore raw

PATCH v15 4/13 x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

In reply to Nikunj A Dadhania

Secure TSC enabled guests should not write to MSR_IA32_TSC(10H) register as the subsequent TSC value reads are undefined. For AMD platform, MSR_IA32_TSC is intercepted by the hypervisor, MSR_IA32_TSC read/write accesses should not exit to the hypervisor for such guests.

Accesses to MSR_IA32_TSC needs special handling in the #VC handler for the guests with Secure TSC enabled. Writes to MSR_IA32_TSC should be ignored, and reads of MSR_IA32_TSC should return the result of the RDTSC instruction.

---
 arch/x86/coco/sev/core.c | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)
arch/x86/coco/sev/core.c +28 -0
index 39683101b526..af28fb962309 100644
@@ -1433,6 +1433,31 @@ static enum es_result __vc_handle_msr_caa(struct pt_regs *regs, bool write)
 	return ES_OK;
 }
 
+/*
+ * TSC related accesses should not exit to the hypervisor when a guest is
+ * executing with SecureTSC enabled, so special handling is required for
+ * accesses of MSR_IA32_TSC:
+ *
+ * Writes: Writing to MSR_IA32_TSC can cause subsequent reads
+ *         of the TSC to return undefined values, so ignore all
+ *         writes.
+ * Reads:  Reads of MSR_IA32_TSC should return the current TSC
+ *         value, use the value returned by RDTSC.
+ */
+static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
+{
+	u64 tsc;
+
+	if (write)
+		return ES_OK;
+
+	tsc = rdtsc_ordered();
+	regs->ax = lower_32_bits(tsc);
+	regs->dx = upper_32_bits(tsc);
+
+	return ES_OK;
+}
+
 static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 {
 	struct pt_regs *regs = ctxt->regs;
@@ -1445,6 +1470,9 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	if (regs->cx == MSR_SVSM_CAA)
 		return __vc_handle_msr_caa(regs, write);
 
+	if (regs->cx == MSR_IA32_TSC && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
+		return __vc_handle_msr_tsc(regs, write);
+
 	ghcb_set_rcx(ghcb, regs->cx);
 	if (write) {
 		ghcb_set_rax(ghcb, regs->ax);
Signature · 1 line
2.34.1
maintainer
#29 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

Quoted 2 lines · Nikunj A Dadhania

Secure TSC enabled guests should not write to MSR_IA32_TSC(10H) register as the subsequent TSC value reads are undefined.

What does that mean exactly?

I'd prefer if we issued a WARN_ONCE() there on the write to catch any offenders.

*NO ONE* should be writing the TSC MSR but that's a different story.

IOW, something like this ontop of yours?

---

arch/x86/coco/sev/core.c +14 -5
index c22cb2ea4b99..050170eb28e6 100644
@@ -1443,9 +1443,15 @@ static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
 {
 	u64 tsc;
 
-	if (write)
-		return ES_OK;
+	if (!(sev_status & MSR_AMD64_SNP_SECURE_TSC))
+		goto read_tsc;
+
+	if (write) {
+		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
+		return ES_UNSUPPORTED;
+	}
 
+read_tsc:
 	tsc = rdtsc_ordered();
 	regs->ax = lower_32_bits(tsc);
 	regs->dx = upper_32_bits(tsc);
@@ -1462,11 +1468,14 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	/* Is it a WRMSR? */
 	write = ctxt->insn.opcode.bytes[1] == 0x30;
 
-	if (regs->cx == MSR_SVSM_CAA)
+	switch(regs->cx) {
+	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
-
-	if (regs->cx == MSR_IA32_TSC && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
+	case MSR_IA32_TSC:
 		return __vc_handle_msr_tsc(regs, write);
+	default:
+		break;
+	}
 
 	ghcb_set_rcx(ghcb, regs->cx);
 	if (write) {
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#30 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Nikunj A Dadhania

Secure TSC enabled guests should not write to MSR_IA32_TSC(10H) register as the subsequent TSC value reads are undefined.

What does that mean exactly?

That is the warning from the APM: 15.36.18 Secure TSC

"Guests that run with Secure TSC enabled are not expected to perform writes to the TSC MSR (10h). If such a write occurs, subsequent TSC values read are undefined."

What I make out of it is: if a write is performed to the TSC MSR, subsequent reads of TSC is not reliable/trusted.

That was the reason to ignore such writes in the #VC handler.

Quoted 3 lines

I'd prefer if we issued a WARN_ONCE() there on the write to catch any offenders.

Do you also want to terminate the offending guest?

ES_UNSUPPORTED return will do that.

Quoted 12 lines

*NO ONE* should be writing the TSC MSR but that's a different story.

IOW, something like this ontop of yours?

---

arch/x86/coco/sev/core.c +0 -0
index c22cb2ea4b99..050170eb28e6 100644

arch/x86/coco/sev/core.c static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write) · line 1447

Quoted 7 lines
arch/x86/coco/sev/core.c +2 -2
@@ -1443,9 +1443,15 @@ static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
 {
 	u64 tsc;
 
-	if (write)
-		return ES_OK;
+	if (!(sev_status & MSR_AMD64_SNP_SECURE_TSC))
+		goto read_tsc;

This is changing the behavior for SEV-ES and SNP guests(non SECURE_TSC), TSC MSR reads are converted to RDTSC. This is a good optimization. But just wanted to bring up the subtle impact.

arch/x86/coco/sev/core.c static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write) · line 1451

Quoted 4 lines
arch/x86/coco/sev/core.c +4 -0
+
+	if (write) {
+		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
+		return ES_UNSUPPORTED;

Sure, we can add a WARN_ONCE().

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1471

Quoted 12 lines
arch/x86/coco/sev/core.c +3 -1
+	}
 
+read_tsc:
 	tsc = rdtsc_ordered();
 	regs->ax = lower_32_bits(tsc);
 	regs->dx = upper_32_bits(tsc);
@@ -1462,11 +1468,14 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	/* Is it a WRMSR? */
 	write = ctxt->insn.opcode.bytes[1] == 0x30;
 
-	if (regs->cx == MSR_SVSM_CAA)
+	switch(regs->cx) {

Yes, I was thinking about a switch, as there will be more such instances when we enable newer features.

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1481

Quoted 13 lines
arch/x86/coco/sev/core.c +5 -2
+	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
-
-	if (regs->cx == MSR_IA32_TSC && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
+	case MSR_IA32_TSC:
 		return __vc_handle_msr_tsc(regs, write);
+	default:
+		break;
+	}
 
 	ghcb_set_rcx(ghcb, regs->cx);
 	if (write) {

Regards,
Nikunj

maintainer
#31 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

Quoted 8 lines · Nikunj A. Dadhania

That is the warning from the APM: 15.36.18 Secure TSC

"Guests that run with Secure TSC enabled are not expected to perform writes to the TSC MSR (10h). If such a write occurs, subsequent TSC values read are undefined."

What I make out of it is: if a write is performed to the TSC MSR, subsequent reads of TSC is not reliable/trusted.

Basically, what happens on baremetal too.

Quoted 3 lines

Do you also want to terminate the offending guest?

ES_UNSUPPORTED return will do that.

I guess that would be too harsh. I guess a warn and a ES_OK should be fine for now.

Quoted 3 lines

This is changing the behavior for SEV-ES and SNP guests(non SECURE_TSC), TSC MSR reads are converted to RDTSC. This is a good optimization. But just wanted to bring up the subtle impact.

That RDTSC happens still in the guest, right? But in its #VC handler. Versus it being a HV GHCB protocol call. I guess this conversion should be a separate patch in case there's some issues like the HV intercepting RDTSC... i.e., VMEXIT_RDTSC.

We should probably handle that case too and then fallback to the GHCB call. Or is there a catch 22 I'm missing here...

Quoted 2 lines

Yes, I was thinking about a switch, as there will be more such instances when we enable newer features.

Exactly.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#32 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Nikunj A. Dadhania

Do you also want to terminate the offending guest?

ES_UNSUPPORTED return will do that.

I guess that would be too harsh. I guess a warn and a ES_OK should be fine for now.

Yes, that will be better.

Quoted 6 lines
Quoted 3 lines

This is changing the behavior for SEV-ES and SNP guests(non SECURE_TSC), TSC MSR reads are converted to RDTSC. This is a good optimization. But just wanted to bring up the subtle impact.

That RDTSC happens still in the guest, right? But in its #VC handler. Versus it being a HV GHCB protocol call.

Yes, and the change is working fine, I have verified it.

Quoted 3 lines

I guess this conversion should be a separate
patch in case there's some issues like the HV intercepting RDTSC... i.e., VMEXIT_RDTSC.

I tried instrumenting code to intercept RDTSC and RDTSCP, KVM does not handle EXIT_RDTSC and the guest (including non-secure guests) crashes pretty early with the following in the host kernel log:

kvm_amd: kvm [2153024]: vcpu0, guest rIP: 0xbbea5fc2 svm: unexpected exit reason 0x6e

Quoted 2 lines

We should probably handle that case too and then fallback to the GHCB call. Or is there a catch 22 I'm missing here...

Regards,
Nikunj

#33 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

In reply to Nikunj A. Dadhania
Quoted 41 lines · Nikunj A. Dadhania
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Nikunj A Dadhania

Secure TSC enabled guests should not write to MSR_IA32_TSC(10H) register as the subsequent TSC value reads are undefined.

What does that mean exactly?

That is the warning from the APM: 15.36.18 Secure TSC

"Guests that run with Secure TSC enabled are not expected to perform writes to the TSC MSR (10h). If such a write occurs, subsequent TSC values read are undefined."

What I make out of it is: if a write is performed to the TSC MSR, subsequent reads of TSC is not reliable/trusted.

That was the reason to ignore such writes in the #VC handler.

Quoted 3 lines

I'd prefer if we issued a WARN_ONCE() there on the write to catch any offenders.

Do you also want to terminate the offending guest?

ES_UNSUPPORTED return will do that.

Quoted 15 lines

*NO ONE* should be writing the TSC MSR but that's a different story.

IOW, something like this ontop of yours?

---

arch/x86/coco/sev/core.c +0 -0
index c22cb2ea4b99..050170eb28e6 100644
@@ -1443,9 +1443,15 @@ static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
 {
 	u64 tsc;
 
Quoted 8 lines

arch/x86/coco/sev/core.c static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write) · line 1447

Quoted 4 lines
arch/x86/coco/sev/core.c +2 -2
-	if (write)
-		return ES_OK;
+	if (!(sev_status & MSR_AMD64_SNP_SECURE_TSC))
+		goto read_tsc;

This is changing the behavior for SEV-ES and SNP guests(non SECURE_TSC), TSC MSR reads are converted to RDTSC. This is a good optimization. But just wanted to bring up the subtle impact.

Right, I think it should still flow through the GHCB MSR request for non-Secure TSC guests.

Quoted 7 lines

arch/x86/coco/sev/core.c static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write) · line 1451

Quoted 5 lines
arch/x86/coco/sev/core.c +4 -0
+
+	if (write) {
+		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
+		return ES_UNSUPPORTED;

Sure, we can add a WARN_ONCE().

You'll want to test this... IIRC, I'm not sure if a WARN_ONCE() will be properly printed when issued within the #VC handler (since it will generate a nested #VC).

Thanks,
Tom

Quoted 33 lines

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1471

Quoted 13 lines
arch/x86/coco/sev/core.c +3 -1
+	}
 
+read_tsc:
 	tsc = rdtsc_ordered();
 	regs->ax = lower_32_bits(tsc);
 	regs->dx = upper_32_bits(tsc);
@@ -1462,11 +1468,14 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	/* Is it a WRMSR? */
 	write = ctxt->insn.opcode.bytes[1] == 0x30;
 
-	if (regs->cx == MSR_SVSM_CAA)
+	switch(regs->cx) {

Yes, I was thinking about a switch, as there will be more such instances when we enable newer features.

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1481

Quoted 13 lines
arch/x86/coco/sev/core.c +5 -2
+	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
-
-	if (regs->cx == MSR_IA32_TSC && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
+	case MSR_IA32_TSC:
 		return __vc_handle_msr_tsc(regs, write);
+	default:
+		break;
+	}
 
 	ghcb_set_rcx(ghcb, regs->cx);
 	if (write) {

Regards,
Nikunj

#34 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

Quoted 10 lines · Tom Lendacky
Quoted 8 lines · Nikunj A. Dadhania

arch/x86/coco/sev/core.c static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write) · line 1451

Quoted 5 lines · Borislav Petkov

On Tue, Dec 03, 2024 at 02:30:36PM +0530, Nikunj A Dadhania wrote:

arch/x86/coco/sev/core.c +4 -0
+
+	if (write) {
+		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
+		return ES_UNSUPPORTED;

Sure, we can add a WARN_ONCE().

Quoted 3 lines

You'll want to test this... IIRC, I'm not sure if a WARN_ONCE() will be properly printed when issued within the #VC handler (since it will generate a nested #VC).

Right, a write to TSC MSR generates the following splat:

[   17.450076] ------------[ cut here ]------------
[   17.450077] TSC MSR writes are verboten!
[   17.450079] WARNING: CPU: 0 PID: 617 at arch/x86/coco/sev/core.c:1456 vc_handle_exitcode.part.0+0xe54/0x1110
[   17.450090] CPU: 0 UID: 0 PID: 617 Comm: wrmsr Tainted: G S                 6.13.0-rc1-00093-g5e3143d631a9-dirty #121

Regards,
Nikunj

maintainer
#35 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

In reply to Tom Lendacky
Quoted 6 lines · Tom Lendacky
Quoted 3 lines

This is changing the behavior for SEV-ES and SNP guests(non SECURE_TSC), TSC MSR reads are converted to RDTSC. This is a good optimization. But just wanted to bring up the subtle impact.

Right, I think it should still flow through the GHCB MSR request for non-Secure TSC guests.

Why?

I'm trying to think of a reason but I'm getting confused by what needs to happen where and when... :-\

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#36 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

Quoted 5 lines · Borislav Petkov
Quoted 4 lines · Tom Lendacky
Quoted 3 lines

This is changing the behavior for SEV-ES and SNP guests(non SECURE_TSC), TSC MSR reads are converted to RDTSC. This is a good optimization. But just wanted to bring up the subtle impact.

Quoted 7 lines
Quoted 2 lines

Right, I think it should still flow through the GHCB MSR request for non-Secure TSC guests.

Why?

I'm trying to think of a reason but I'm getting confused by what needs to happen where and when... :-\

It could be any reason... maybe the hypervisor wants to know when this MSR used in order to tell the guest owner to update their code. Writing to or reading from that MSR is not that common, so I would think we want to keep the same behavior that has been in effect.

But if we do want to make this change, maybe do it separate from the Secure TSC series since it alters the behavior of SEV-ES guests and SEV-SNP guests without Secure TSC.

Thanks,
Tom

Quoted 1 line
maintainer
#37 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

Quoted 4 lines · Tom Lendacky

It could be any reason... maybe the hypervisor wants to know when this MSR used in order to tell the guest owner to update their code. Writing to or reading from that MSR is not that common, so I would think we want to keep the same behavior that has been in effect.

Ah, I thought you're gonna say something along the lines of, yeah, we must use the HV GHCB protocol because of <raisins> and there's no other way this could work.

Quoted 3 lines

But if we do want to make this change, maybe do it separate from the Secure TSC series since it alters the behavior of SEV-ES guests and SEV-SNP guests without Secure TSC.

Already suggested so - this should be a separate patch.

It would be interesting to see if it brings any improvement by avoiding the HV call... especially since RDTSC is used a *lot* and prominently at that in sched_clock, for example.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#38 lore raw

Re: [PATCH v15 04/13] x86/sev: Change TSC MSR behavior for Secure TSC enabled guests

Quoted 14 lines · Borislav Petkov
Quoted 4 lines · Tom Lendacky

It could be any reason... maybe the hypervisor wants to know when this MSR used in order to tell the guest owner to update their code. Writing to or reading from that MSR is not that common, so I would think we want to keep the same behavior that has been in effect.

Ah, I thought you're gonna say something along the lines of, yeah, we must use the HV GHCB protocol because of <raisins> and there's no other way this could work.

Quoted 3 lines

But if we do want to make this change, maybe do it separate from the Secure TSC series since it alters the behavior of SEV-ES guests and SEV-SNP guests without Secure TSC.

Quoted 5 lines

Already suggested so - this should be a separate patch.

It would be interesting to see if it brings any improvement by avoiding the HV call... especially since RDTSC is used a *lot* and prominently at that in sched_clock, for example.

I doubt you would notice anything since it doesn't look like Linux ever reads from or writes to MSR_IA32_TSC, preferring to just use RDTSC or RDTSCP which aren't usually intercepted at all and so never goes out to the HV.

Thanks,
Tom

Quoted 1 line
#39 lore raw

PATCH v15 5/13 x86/sev: Prevent RDTSC/RDTSCP interception for Secure TSC enabled guests

In reply to Nikunj A Dadhania

The hypervisor should not be intercepting RDTSC/RDTSCP when Secure TSC is enabled. A #VC exception will be generated if the RDTSC/RDTSCP instructions are being intercepted. If this should occur and Secure TSC is enabled, guest execution should be terminated as the guest cannot rely on the TSC value provided by the hypervisor.

---
 arch/x86/coco/sev/shared.c | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
arch/x86/coco/sev/shared.c +14 -0
index 71de53194089..879ab48b705c 100644
@@ -1140,6 +1140,20 @@ static enum es_result vc_handle_rdtsc(struct ghcb *ghcb,
 	bool rdtscp = (exit_code == SVM_EXIT_RDTSCP);
 	enum es_result ret;
 
+	/*
+	 * The hypervisor should not be intercepting RDTSC/RDTSCP when Secure
+	 * TSC is enabled. A #VC exception will be generated if the RDTSC/RDTSCP
+	 * instructions are being intercepted. If this should occur and Secure
+	 * TSC is enabled, guest execution should be terminated as the guest
+	 * cannot rely on the TSC value provided by the hypervisor.
+	 *
+	 * This file is included from kernel/sev.c and boot/compressed/sev.c,
+	 * use sev_status here as cc_platform_has() is not available when
+	 * compiling boot/compressed/sev.c.
+	 */
+	if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+		return ES_VMM_ERROR;
+
 	ret = sev_es_ghcb_hv_call(ghcb, ctxt, exit_code, 0, 0);
 	if (ret != ES_OK)
 		return ret;
Signature · 1 line
2.34.1
maintainer
#40 lore raw

Re: [PATCH v15 05/13] x86/sev: Prevent RDTSC/RDTSCP interception for Secure TSC enabled guests

Quoted 9 lines · Nikunj A Dadhania
arch/x86/coco/sev/shared.c +1 -0
index 71de53194089..879ab48b705c 100644
@@ -1140,6 +1140,20 @@ static enum es_result vc_handle_rdtsc(struct ghcb *ghcb,
 	bool rdtscp = (exit_code == SVM_EXIT_RDTSCP);
 	enum es_result ret;
 
+	/*

arch/x86/coco/sev/shared.c static enum es_result vc_handle_rdtsc(struct ghcb *ghcb, · line 1153

Quoted 10 lines
arch/x86/coco/sev/shared.c +10 -0
+	 * The hypervisor should not be intercepting RDTSC/RDTSCP when Secure
+	 * TSC is enabled. A #VC exception will be generated if the RDTSC/RDTSCP
+	 * instructions are being intercepted. If this should occur and Secure
+	 * TSC is enabled, guest execution should be terminated as the guest
+	 * cannot rely on the TSC value provided by the hypervisor.
+	 *
+	 * This file is included from kernel/sev.c and boot/compressed/sev.c,
+	 * use sev_status here as cc_platform_has() is not available when
+	 * compiling boot/compressed/sev.c.
+	 */
arch/x86/coco/sev/shared.c +0 -4
index 71c7024eb597..2e4122f8aa6b 100644
@@ -1147,10 +1147,6 @@ static enum es_result vc_handle_rdtsc(struct ghcb *ghcb,
 	 * instructions are being intercepted. If this should occur and Secure
 	 * TSC is enabled, guest execution should be terminated as the guest
 	 * cannot rely on the TSC value provided by the hypervisor.
-	 *
-	 * This file is included from kernel/sev.c and boot/compressed/sev.c,
-	 * use sev_status here as cc_platform_has() is not available when
-	 * compiling boot/compressed/sev.c.
 	 */
 	if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
 		return ES_VMM_ERROR;
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#41 lore raw

PATCH v15 6/13 x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

In reply to Nikunj A Dadhania

The hypervisor should not be intercepting GUEST_TSC_FREQ MSR(0xcOO10134) when Secure TSC is enabled. A #VC exception will be generated if the GUEST_TSC_FREQ MSR is being intercepted. If this should occur and SecureTSC is enabled, terminate guest execution.

---
 arch/x86/include/asm/msr-index.h | 1 +
 arch/x86/coco/sev/core.c         | 8 ++++++++
 2 files changed, 9 insertions(+)
arch/x86/include/asm/msr-index.h +1 -0
index 3ae84c3b8e6d..233be13cc21f 100644
@@ -608,6 +608,7 @@
 #define MSR_AMD_PERF_CTL		0xc0010062
 #define MSR_AMD_PERF_STATUS		0xc0010063
 #define MSR_AMD_PSTATE_DEF_BASE		0xc0010064
+#define MSR_AMD64_GUEST_TSC_FREQ	0xc0010134
 #define MSR_AMD64_OSVW_ID_LENGTH	0xc0010140
 #define MSR_AMD64_OSVW_STATUS		0xc0010141
 #define MSR_AMD_PPIN_CTL		0xc00102f0
arch/x86/coco/sev/core.c +8 -0
index af28fb962309..59c5e716fdd1 100644
@@ -1473,6 +1473,14 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	if (regs->cx == MSR_IA32_TSC && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
 		return __vc_handle_msr_tsc(regs, write);
 
+	/*
+	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
+	 * enabled. Terminate the SNP guest when the interception is enabled.
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
+		return ES_VMM_ERROR;
+
+
 	ghcb_set_rcx(ghcb, regs->cx);
 	if (write) {
 		ghcb_set_rax(ghcb, regs->ax);
Signature · 1 line
2.34.1
maintainer
#42 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 6 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +0 -0
index af28fb962309..59c5e716fdd1 100644
@@ -1473,6 +1473,14 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	if (regs->cx == MSR_IA32_TSC && (sev_status & MSR_AMD64_SNP_SECURE_TSC))

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1483

Quoted 10 lines
arch/x86/coco/sev/core.c +8 -0
 		return __vc_handle_msr_tsc(regs, write);
 
+	/*
+	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
+	 * enabled. Terminate the SNP guest when the interception is enabled.
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
+		return ES_VMM_ERROR;
+
+

If you merge this logic into the switch-case, the patch becomes even easier and the code cleaner:

arch/x86/coco/sev/core.c +8 -0
index 050170eb28e6..35d9a3bb4b06 100644
@@ -1446,6 +1446,13 @@ static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
 	if (!(sev_status & MSR_AMD64_SNP_SECURE_TSC))
 		goto read_tsc;
 
+	/*
+	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
+	 * enabled. Terminate the SNP guest when the interception is enabled.
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ)
+		return ES_VMM_ERROR;
+
 	if (write) {
 		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
 		return ES_UNSUPPORTED;
@@ -1472,6 +1479,7 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
 	case MSR_IA32_TSC:
+	case MSR_AMD64_GUEST_TSC_FREQ:
 		return __vc_handle_msr_tsc(regs, write);
 	default:
 		break;
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#43 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 9 lines · Borislav Petkov
Quoted 8 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +0 -0
index af28fb962309..59c5e716fdd1 100644
@@ -1473,6 +1473,14 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	if (regs->cx == MSR_IA32_TSC && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
 		return __vc_handle_msr_tsc(regs, write);
 
Quoted 11 lines

arch/x86/coco/sev/core.c static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)

Quoted 8 lines
arch/x86/coco/sev/core.c +8 -0
+	/*
+	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
+	 * enabled. Terminate the SNP guest when the interception is enabled.
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
+		return ES_VMM_ERROR;
+
+

If you merge this logic into the switch-case, the patch becomes even easier and the code cleaner:

This is incorrect, for a non-Secure TSC guest, a read of intercepted MSR_AMD64_GUEST_TSC_FREQ will return value of rdtsc_ordered(). This is an invalid MSR when SecureTSC is not enabled.

Quoted 21 lines
arch/x86/coco/sev/core.c +7 -0
index 050170eb28e6..35d9a3bb4b06 100644
@@ -1446,6 +1446,13 @@ static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
 	if (!(sev_status & MSR_AMD64_SNP_SECURE_TSC))
 		goto read_tsc;
 
+	/*
+	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
+	 * enabled. Terminate the SNP guest when the interception is enabled.
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ)
+		return ES_VMM_ERROR;
+
 	if (write) {
 		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
 		return ES_UNSUPPORTED;

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1485

Quoted 7 lines
arch/x86/coco/sev/core.c +1 -0
@@ -1472,6 +1479,7 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
 	case MSR_IA32_TSC:
+	case MSR_AMD64_GUEST_TSC_FREQ:
 		return __vc_handle_msr_tsc(regs, write);
 	default:
 		break;

Regards
Nikunj

maintainer
#44 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 3 lines · Nikunj A Dadhania

This is incorrect, for a non-Secure TSC guest, a read of intercepted MSR_AMD64_GUEST_TSC_FREQ will return value of rdtsc_ordered(). This is an invalid MSR when SecureTSC is not enabled.

So how would you change this diff to fix this?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#45 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 6 lines · Borislav Petkov
Quoted 3 lines · Nikunj A Dadhania

This is incorrect, for a non-Secure TSC guest, a read of intercepted MSR_AMD64_GUEST_TSC_FREQ will return value of rdtsc_ordered(). This is an invalid MSR when SecureTSC is not enabled.

So how would you change this diff to fix this?

How about the below change, this also keeps the behavior intact for non-Secure TSC guests (SEV-ES/SNP):

arch/x86/coco/sev/core.c +9 -12
index 25a4c47f58c4..fa57adf5a2c6 100644
@@ -1448,15 +1448,18 @@ static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
 {
 	u64 tsc;
 
-	if (!(sev_status & MSR_AMD64_SNP_SECURE_TSC))
-		goto read_tsc;
+	/*
+	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
+	 * enabled. Terminate the SNP guest when the interception is enabled.
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ)
+		return ES_VMM_ERROR;
 
 	if (write) {
 		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
 		return ES_OK;
 	}
 
-read_tsc:
 	tsc = rdtsc_ordered();
 	regs->ax = lower_32_bits(tsc);
 	regs->dx = upper_32_bits(tsc);
@@ -1477,19 +1480,13 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
 	case MSR_IA32_TSC:
-		return __vc_handle_msr_tsc(regs, write);
+	case MSR_AMD64_GUEST_TSC_FREQ:
+		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+			return __vc_handle_msr_tsc(regs, write);
 	default:
 		break;
 	}
 
-	/*
-	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
-	 * enabled. Terminate the SNP guest when the interception is enabled.
-	 */
-	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
-		return ES_VMM_ERROR;
-
-
 	ghcb_set_rcx(ghcb, regs->cx);
 	if (write) {
 		ghcb_set_rax(ghcb, regs->ax);

---

Regards
Nikunj

maintainer
#46 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1485

Quoted 8 lines · Nikunj A. Dadhania
arch/x86/coco/sev/core.c +3 -1
@@ -1477,19 +1480,13 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
 	case MSR_IA32_TSC:
-		return __vc_handle_msr_tsc(regs, write);
+	case MSR_AMD64_GUEST_TSC_FREQ:
+		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+			return __vc_handle_msr_tsc(regs, write);

Now push that conditional inside the function too.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#47 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 9 lines · Borislav Petkov

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1485

Quoted 8 lines · Nikunj A. Dadhania
arch/x86/coco/sev/core.c +3 -1
@@ -1477,19 +1480,13 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
 	case MSR_IA32_TSC:
-		return __vc_handle_msr_tsc(regs, write);
+	case MSR_AMD64_GUEST_TSC_FREQ:
+		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+			return __vc_handle_msr_tsc(regs, write);

With the above change, non-Secure TSC SNP guest will return following error:

$ sudo rdmsr 0xc0010134
rdmsr: CPU 0 cannot read MSR 0xc0010134
$
Quoted 1 line

Now push that conditional inside the function too.

With the condition inside the function, even tough the MSR is not valid in this configuration, I am getting value 0. Is this behavior acceptable ?

$ sudo rdmsr 0xc0010134
0
$

Regards,
Nikunj

maintainer
#48 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 3 lines · Nikunj A. Dadhania

With the condition inside the function, even tough the MSR is not valid in this configuration, I am getting value 0. Is this behavior acceptable ?

The whole untested diff, should DTRT this time:

arch/x86/coco/sev/core.c +24 -13
index 10980898c054..96a9ee93f9cb 100644
@@ -1441,10 +1441,25 @@ static enum es_result __vc_handle_msr_caa(struct pt_regs *regs, bool write)
  */
 static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
 {
+	bool sec_tsc = sev_status & MSR_AMD64_SNP_SECURE_TSC;
 	u64 tsc;
 
-	if (write)
-		return ES_OK;
+	/*
+	 * The GUEST_TSC_FREQ MSR should not be intercepted when secure
+	 * TSC is enabled so terminate the guest. For non-secure TSC
+	 * guests, that MSR is #GP(0).
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ) {
+		if (sec_tsc)
+			return ES_VMM_ERROR;
+		else
+			return ES_UNSUPPORTED;
+	}
+
+	if (write) {
+		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
+		return ES_UNSUPPORTED;
+	}
 
 	tsc = rdtsc_ordered();
 	regs->ax = lower_32_bits(tsc);
@@ -1462,19 +1477,15 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	/* Is it a WRMSR? */
 	write = ctxt->insn.opcode.bytes[1] == 0x30;
 
-	if (regs->cx == MSR_SVSM_CAA)
+	switch(regs->cx) {
+	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
-
-	if (regs->cx == MSR_IA32_TSC && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
+	case MSR_IA32_TSC:
+	case MSR_AMD64_GUEST_TSC_FREQ:
 		return __vc_handle_msr_tsc(regs, write);
-
-	/*
-	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
-	 * enabled. Terminate the SNP guest when the interception is enabled.
-	 */
-	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
-		return ES_VMM_ERROR;
-
+	default:
+		break;
+	}
 
 	ghcb_set_rcx(ghcb, regs->cx);
 	if (write) {
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#49 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 6 lines · Borislav Petkov
Quoted 3 lines · Nikunj A. Dadhania

With the condition inside the function, even tough the MSR is not valid in this configuration, I am getting value 0. Is this behavior acceptable ?

The whole untested diff, should DTRT this time:

I have tested the diff and ES_UNSUPPORTED causes unexpected termination of SNP guest(without SecureTSC).

$ sudo wrmsr 0x10 0
KVM: unknown exit reason 24
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a00f11
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff0 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
...
$ sudo wrmsr 0xc0010134 0
KVM: unknown exit reason 24
EAX=00000000 EBX=00000000 ECX=00000000 EDX=00a00f11
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
...

IMO, the below change appropriately handles all the conditions well and does not affect SNP guests without SecureTSC.

arch/x86/coco/sev/core.c +43 -1
index 84e4e64decf7..a8977c68695b 100644
@@ -1428,6 +1428,40 @@ static enum es_result __vc_handle_msr_caa(struct pt_regs *regs, bool write)
 	return ES_OK;
 }
 
+/*
+ * TSC related accesses should not exit to the hypervisor when a guest is
+ * executing with SecureTSC enabled, so special handling is required for
+ * accesses of MSR_IA32_TSC and MSR_AMD64_GUEST_TSC_FREQ:
+ *
+ * Writes: Writing to MSR_IA32_TSC can cause subsequent reads
+ *         of the TSC to return undefined values, so ignore all
+ *         writes.
+ * Reads:  Reads of MSR_IA32_TSC should return the current TSC
+ *         value, use the value returned by RDTSC.
+ */
+static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
+{
+	u64 tsc;
+
+	/*
+	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
+	 * enabled. Terminate the SNP guest when the interception is enabled.
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ)
+		return ES_VMM_ERROR;
+
+	if (write) {
+		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
+		return ES_OK;
+	}
+
+	tsc = rdtsc_ordered();
+	regs->ax = lower_32_bits(tsc);
+	regs->dx = upper_32_bits(tsc);
+
+	return ES_OK;
+}
+
 static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 {
 	struct pt_regs *regs = ctxt->regs;
@@ -1437,8 +1471,16 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	/* Is it a WRMSR? */
 	write = ctxt->insn.opcode.bytes[1] == 0x30;
 
-	if (regs->cx == MSR_SVSM_CAA)
+	switch (regs->cx) {
+	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
+	case MSR_IA32_TSC:
+	case MSR_AMD64_GUEST_TSC_FREQ:
+		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+			return __vc_handle_msr_tsc(regs, write);
+	default:
+		break;
+	}
 
 	ghcb_set_rcx(ghcb, regs->cx);
 	if (write) {
maintainer
#50 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1480

Quoted 12 lines · Nikunj A. Dadhania
arch/x86/coco/sev/core.c +6 -1
@@ -1437,8 +1471,16 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	/* Is it a WRMSR? */
 	write = ctxt->insn.opcode.bytes[1] == 0x30;
 
-	if (regs->cx == MSR_SVSM_CAA)
+	switch (regs->cx) {
+	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
+	case MSR_IA32_TSC:
+	case MSR_AMD64_GUEST_TSC_FREQ:
+		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+			return __vc_handle_msr_tsc(regs, write);

Again: move all the logic inside __vc_handle_msr_tsc().

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#51 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 5 lines · Borislav Petkov
Quoted 4 lines · Nikunj A. Dadhania
arch/x86/coco/sev/core.c +0 -0
@@ -1437,8 +1471,16 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	/* Is it a WRMSR? */
 	write = ctxt->insn.opcode.bytes[1] == 0x30;
 
Quoted 11 lines

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1480

Quoted 8 lines
arch/x86/coco/sev/core.c +6 -1
-	if (regs->cx == MSR_SVSM_CAA)
+	switch (regs->cx) {
+	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
+	case MSR_IA32_TSC:
+	case MSR_AMD64_GUEST_TSC_FREQ:
+		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+			return __vc_handle_msr_tsc(regs, write);

Again: move all the logic inside __vc_handle_msr_tsc().

Boris, I have mentioned in my previous reply why this is incorrect for non-SecureTSC guests with examples and had asked for your confirmation if it is an acceptable behaviour[1] and Tom has also objected to the change of behavior for non-SecureTSC guest[2].

I think we are dragging this a little too far and the implementation[3] that I gave is good without any side effects.

Regards
Nikunj

1: https://lore.kernel.org/all/[email protected]
2: https://lore.kernel.org/all/[email protected]
3: https://lore.kernel.org/all/[email protected]

maintainer
#52 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 1 line · Nikunj A. Dadhania

I think we are dragging this a little too far

What does that mean?

Is there some time limit I don't know about, on how long patches should be reviewed on the mailing list?

Quoted 1 line

and the implementation[3] that I gave is good without any side effects.

Well, apparently not good enough - otherwise I won't say anything, would I? And I wouldn't review your patches on my holiday, would I?

Geez.

Now lemme try again, this time in greater detail with the hope it is more clear.

If you handle TSC MSRs and then you have a function __vc_handle_msr_tsc() then *all* handling should happen there! There should not be if-conditionals in the switch-case which makes following the code flow harder. That's why I'm asking you to push the conditional inside. So that everything is concentrated in a single function!

But there's this thing with handling TSC MSRs and non-STSC guests and that needs special, later handling and decision.

So *that* needs to be made obvious.

As in: I will handle the TSC MSRs for STSC guests and the other flow for non-STSC guests should remain. For now.

And make that goddamn explicit.

One possible way to do that is this:

arch/x86/coco/sev/core.c +4 -2
index 6235286a0eda..61100532c259 100644
@@ -1439,7 +1439,7 @@ static enum es_result __vc_handle_msr_caa(struct pt_regs *regs, bool write)
  * Reads:  Reads of MSR_IA32_TSC should return the current TSC
  *         value, use the value returned by RDTSC.
  */
-static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
+static enum es_result __vc_handle_secure_tsc_msrs(struct pt_regs *regs, bool write)
 {
 	u64 tsc;
 
@@ -1477,7 +1477,9 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_IA32_TSC:
 	case MSR_AMD64_GUEST_TSC_FREQ:
 		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
-			return __vc_handle_msr_tsc(regs, write);
+			return __vc_handle_secure_tsc_msrs(regs, write);
+		else
+			break;
 	default:
 		break;
 	}

---

You can still push the if-conditional inside the function but that'll need more surgery as you need a different retval to tell vc_handle_msr() to fall-through to the GHCB HV call instead of returning, yadda, yadda so this version above is shorter.

And it can be revisited later, when we decide what we wanna do with TSC MSRs on !STSC guests.

IOW, the code is still clear and there's enough breadcrumbs left to know what needs to happen there in the future.

Versus: lemme drop my enablement patches and disappear and the maintainers can mop up after me. Who cares! :-(

I hope this makes it a lot more clear now.

And again, if this takes too long for you, just lemme know: I have absolutely no problem if someone else who's faster reviews your code - I have more than enough TODOs on my plate so not dealing with this would be more than welcome for me.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#53 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 8 lines · Borislav Petkov

On Thu, Jan 02, 2025 at 10:33:26AM +0530, Nikunj A. Dadhania wrote:

As in: I will handle the TSC MSRs for STSC guests and the other flow for non-STSC guests should remain. For now.

And make that goddamn explicit.

One possible way to do that is this:

I agree, if renaming helps to make it explicit, this is perfect. Thanks.

Quoted 16 lines
arch/x86/coco/sev/core.c +1 -1
index 6235286a0eda..61100532c259 100644
@@ -1439,7 +1439,7 @@ static enum es_result __vc_handle_msr_caa(struct pt_regs *regs, bool write)
  * Reads:  Reads of MSR_IA32_TSC should return the current TSC
  *         value, use the value returned by RDTSC.
  */
-static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
+static enum es_result __vc_handle_secure_tsc_msrs(struct pt_regs *regs, bool write)
 {
 	u64 tsc;
 
@@ -1477,7 +1477,9 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_IA32_TSC:

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)

Quoted 10 lines
arch/x86/coco/sev/core.c +3 -2
 	case MSR_AMD64_GUEST_TSC_FREQ:
 		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
-			return __vc_handle_msr_tsc(regs, write);
+			return __vc_handle_secure_tsc_msrs(regs, write);
+		else
+			break;
 	default:
 		break;
 	}
---

Regards,
Nikunj

#54 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 26 lines · Nikunj A. Dadhania
Quoted 8 lines · Borislav Petkov

On Thu, Jan 02, 2025 at 10:33:26AM +0530, Nikunj A. Dadhania wrote:

As in: I will handle the TSC MSRs for STSC guests and the other flow for non-STSC guests should remain. For now.

And make that goddamn explicit.

One possible way to do that is this:

I agree, if renaming helps to make it explicit, this is perfect. Thanks.

Quoted 14 lines
arch/x86/coco/sev/core.c +1 -1
index 6235286a0eda..61100532c259 100644
@@ -1439,7 +1439,7 @@ static enum es_result __vc_handle_msr_caa(struct pt_regs *regs, bool write)
  * Reads:  Reads of MSR_IA32_TSC should return the current TSC
  *         value, use the value returned by RDTSC.
  */
-static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
+static enum es_result __vc_handle_secure_tsc_msrs(struct pt_regs *regs, bool write)
 {
 	u64 tsc;
 
Quoted 8 lines

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1482

Quoted 8 lines
arch/x86/coco/sev/core.c +3 -1
@@ -1477,7 +1477,9 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_IA32_TSC:
 	case MSR_AMD64_GUEST_TSC_FREQ:
 		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
-			return __vc_handle_msr_tsc(regs, write);
+			return __vc_handle_secure_tsc_msrs(regs, write);
+		else
+			break;

There's a return as part of the if, so no reason for the else. Just put the break in the normal place and it reads much clearer.

Thanks,
Tom

Quoted 7 lines

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)

Quoted 4 lines
arch/x86/coco/sev/core.c +0 -1
 	default:
 		break;
 	}
---

Regards,
Nikunj

maintainer
#55 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

Quoted 11 lines · Tom Lendacky
Quoted 8 lines

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1482

Quoted 8 lines
arch/x86/coco/sev/core.c +3 -1
@@ -1477,7 +1477,9 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_IA32_TSC:
 	case MSR_AMD64_GUEST_TSC_FREQ:
 		if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
-			return __vc_handle_msr_tsc(regs, write);
+			return __vc_handle_secure_tsc_msrs(regs, write);
+		else
+			break;

There's a return as part of the if, so no reason for the else. Just put the break in the normal place and it reads much clearer.

I guess that's in the eye of the beholder. I prefer a balanced

if
foo
else
bar

as it is as obvious and clear as it gets. Especially if it is interwoven in a switch-case like here.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#56 lore raw

Re: [PATCH v15 06/13] x86/sev: Prevent GUEST_TSC_FREQ MSR interception for Secure TSC enabled guests

In reply to Nikunj A Dadhania
Quoted 22 lines · Nikunj A Dadhania
Quoted 9 lines · Borislav Petkov
Quoted 8 lines · Nikunj A Dadhania
arch/x86/coco/sev/core.c +0 -0
index af28fb962309..59c5e716fdd1 100644
@@ -1473,6 +1473,14 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	if (regs->cx == MSR_IA32_TSC && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
 		return __vc_handle_msr_tsc(regs, write);
 
Quoted 11 lines

arch/x86/coco/sev/core.c static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)

Quoted 8 lines
arch/x86/coco/sev/core.c +8 -0
+	/*
+	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
+	 * enabled. Terminate the SNP guest when the interception is enabled.
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ && (sev_status & MSR_AMD64_SNP_SECURE_TSC))
+		return ES_VMM_ERROR;
+
+

If you merge this logic into the switch-case, the patch becomes even easier and the code cleaner:

Quoted 3 lines

This is incorrect, for a non-Secure TSC guest, a read of intercepted MSR_AMD64_GUEST_TSC_FREQ will return value of rdtsc_ordered(). This is an invalid MSR when SecureTSC is not enabled.

For the non-Secure TSC guest, I still think that we should continue to use the GHCB MSR NAE event instead of switching to using rdtsc_ordered().

Thanks,
Tom

Quoted 32 lines
Quoted 22 lines
arch/x86/coco/sev/core.c +7 -0
index 050170eb28e6..35d9a3bb4b06 100644
@@ -1446,6 +1446,13 @@ static enum es_result __vc_handle_msr_tsc(struct pt_regs *regs, bool write)
 	if (!(sev_status & MSR_AMD64_SNP_SECURE_TSC))
 		goto read_tsc;
 
+	/*
+	 * GUEST_TSC_FREQ should not be intercepted when Secure TSC is
+	 * enabled. Terminate the SNP guest when the interception is enabled.
+	 */
+	if (regs->cx == MSR_AMD64_GUEST_TSC_FREQ)
+		return ES_VMM_ERROR;
+
 	if (write) {
 		WARN_ONCE(1, "TSC MSR writes are verboten!\n");
 		return ES_UNSUPPORTED;

arch/x86/coco/sev/core.c static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt) · line 1485

Quoted 7 lines
arch/x86/coco/sev/core.c +1 -0
@@ -1472,6 +1479,7 @@ static enum es_result vc_handle_msr(struct ghcb *ghcb, struct es_em_ctxt *ctxt)
 	case MSR_SVSM_CAA:
 		return __vc_handle_msr_caa(regs, write);
 	case MSR_IA32_TSC:
+	case MSR_AMD64_GUEST_TSC_FREQ:
 		return __vc_handle_msr_tsc(regs, write);
 	default:
 		break;

Regards
Nikunj

#57 lore raw

PATCH v15 7/13 x86/sev: Mark Secure TSC as reliable clocksource

In reply to Nikunj A Dadhania

In SNP guest environment with Secure TSC enabled, unlike other clock sources (such as HPET, ACPI timer, APIC, etc.), the RDTSC instruction is handled without causing a VM exit, resulting in minimal overhead and jitters. Hence, mark Secure TSC as the only reliable clock source, bypassing unstable calibration.

---
 arch/x86/mm/mem_encrypt_amd.c | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/mm/mem_encrypt_amd.c +4 -0
index 774f9677458f..fa0bc52ef707 100644
@@ -541,6 +541,10 @@ void __init sme_early_init(void)
 	 * kernel mapped.
 	 */
 	snp_update_svsm_ca();
+
+	/* Mark the TSC as reliable when Secure TSC is enabled */
+	if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+		setup_force_cpu_cap(X86_FEATURE_TSC_RELIABLE);
 }
 
 void __init mem_encrypt_free_decrypted_mem(void)
Signature · 1 line
2.34.1
maintainer
#58 lore raw

Re: [PATCH v15 07/13] x86/sev: Mark Secure TSC as reliable clocksource

arch/x86/mm/mem_encrypt_amd.c void __init sme_early_init(void) · line 547

Quoted 12 lines · Nikunj A Dadhania
arch/x86/mm/mem_encrypt_amd.c +4 -0
index 774f9677458f..fa0bc52ef707 100644
@@ -541,6 +541,10 @@ void __init sme_early_init(void)
 	 * kernel mapped.
 	 */
 	snp_update_svsm_ca();
+
+	/* Mark the TSC as reliable when Secure TSC is enabled */
+	if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+		setup_force_cpu_cap(X86_FEATURE_TSC_RELIABLE);

What happens if someone writes MSR 0x10 on some CPU and thus makes the TSCs on the host unsynchronized and that CPU runs a SecureTSC guest?

That guest would use RDTSC and get wrong values and turn the guest into a mess, right?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#59 lore raw

Re: [PATCH v15 07/13] x86/sev: Mark Secure TSC as reliable clocksource

Quoted 17 lines · Borislav Petkov

arch/x86/mm/mem_encrypt_amd.c void __init sme_early_init(void) · line 547

Quoted 12 lines · Nikunj A Dadhania
arch/x86/mm/mem_encrypt_amd.c +4 -0
index 774f9677458f..fa0bc52ef707 100644
@@ -541,6 +541,10 @@ void __init sme_early_init(void)
 	 * kernel mapped.
 	 */
 	snp_update_svsm_ca();
+
+	/* Mark the TSC as reliable when Secure TSC is enabled */
+	if (sev_status & MSR_AMD64_SNP_SECURE_TSC)
+		setup_force_cpu_cap(X86_FEATURE_TSC_RELIABLE);

What happens if someone writes MSR 0x10 on some CPU and thus makes the TSCs on the host unsynchronized and that CPU runs a SecureTSC guest?

Quoted 2 lines

That guest would use RDTSC and get wrong values and turn the guest into a mess, right?

No, SecureTSC guest keeps on ticking forward even when the HV has written to MSR 0x10 on the CPU where the SecureTSC guest is running.

I performed following experiment to confirm the behavior:

1) Started the SecureTSC guest pinned to CPU10

host $ taskset -c 10 ./bootg_sectsc.sh

2) Read TSC MSR on guest

guest $ sudo rdmsr 0x10
8005a5b2d634c

3) Set TSC MSR to 0 on CPU10 on host.

host $ sudo wrmsr -p 10 0x10 0
host $ sudo rdmsr -p 10 0x10
4846ad0

4) Read TSC MSR on guest again

guest $ sudo rdmsr 0x10
8005d18a7144f

Regards,
Nikunj

#60 lore raw

PATCH v15 8/13 x86/cpu/amd: Do not print FW_BUG for Secure TSC

In reply to Nikunj A Dadhania

When Secure TSC is enabled and TscInvariant (bit 8) in CPUID_8000_0007_edx is set, the kernel complains with the below firmware bug:

[Firmware Bug]: TSC doesn't count with P0 frequency!

Secure TSC does not need to run at P0 frequency; the TSC frequency is set by the VMM as part of the SNP_LAUNCH_START command. Skip this check when Secure TSC is enabled

---
 arch/x86/kernel/cpu/amd.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/amd.c +2 -1
index d8408aafeed9..af9dd545f0ec 100644
@@ -370,7 +370,8 @@ static void bsp_determine_snp(struct cpuinfo_x86 *c)
 
 static void bsp_init_amd(struct cpuinfo_x86 *c)
 {
-	if (cpu_has(c, X86_FEATURE_CONSTANT_TSC)) {
+	if (cpu_has(c, X86_FEATURE_CONSTANT_TSC) &&
+	    !cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC)) {
 
 		if (c->x86 > 0x10 ||
 		    (c->x86 == 0x10 && c->x86_model >= 0x2)) {
Signature · 1 line
2.34.1
maintainer
#61 lore raw

Re: [PATCH v15 08/13] x86/cpu/amd: Do not print FW_BUG for Secure TSC

Quoted 4 lines · Nikunj A Dadhania

When Secure TSC is enabled and TscInvariant (bit 8) in CPUID_8000_0007_edx is set, the kernel complains with the below firmware bug:

[Firmware Bug]: TSC doesn't count with P0 frequency!

This happens in a normal guest too:

[    0.000000] [Firmware Bug]: TSC doesn't count with P0 frequency!
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
...
[    0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2023.11-8 02/21/2024

...

If anything, this should check X86_FEATURE_HYPERVISOR but this is just a silly warning so you don't really need to do anything about it.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#62 lore raw

Re: [PATCH v15 08/13] x86/cpu/amd: Do not print FW_BUG for Secure TSC

Quoted 13 lines · Borislav Petkov
Quoted 4 lines · Nikunj A Dadhania

When Secure TSC is enabled and TscInvariant (bit 8) in CPUID_8000_0007_edx is set, the kernel complains with the below firmware bug:

[Firmware Bug]: TSC doesn't count with P0 frequency!

This happens in a normal guest too:

[    0.000000] [Firmware Bug]: TSC doesn't count with P0 frequency!
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009ffff] usable
...
[    0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2023.11-8 02/21/2024
Quoted 4 lines

...

If anything, this should check X86_FEATURE_HYPERVISOR but this is just a silly warning so you don't really need to do anything about it.

In that case, let me drop this patch.

Regards,
Nikunj

#63 lore raw

PATCH v15 9/13 tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

In reply to Nikunj A Dadhania

Calibrating the TSC frequency using the kvmclock is not correct for SecureTSC enabled guests. Use the platform provided TSC frequency via the GUEST_TSC_FREQ MSR (C001_0134h).

---
 arch/x86/include/asm/sev.h |  2 ++
 arch/x86/coco/sev/core.c   | 16 ++++++++++++++++
 arch/x86/kernel/tsc.c      |  5 +++++
 3 files changed, 23 insertions(+)
arch/x86/include/asm/sev.h +2 -0
index 9fd02efef08e..c4dca06b3b01 100644
@@ -493,6 +493,7 @@ int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req
 			   struct snp_guest_request_ioctl *rio);
 
 void __init snp_secure_tsc_prepare(void);
+void __init snp_secure_tsc_init(void);
 
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
@@ -536,6 +537,7 @@ static inline void snp_msg_free(struct snp_msg_desc *mdesc) { }
 static inline int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
 					 struct snp_guest_request_ioctl *rio) { return -ENODEV; }
 static inline void __init snp_secure_tsc_prepare(void) { }
+static inline void __init snp_secure_tsc_init(void) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/coco/sev/core.c +16 -0
index 59c5e716fdd1..1bc668883058 100644
@@ -3279,3 +3279,19 @@ void __init snp_secure_tsc_prepare(void)
 
 	pr_debug("SecureTSC enabled");
 }
+
+static unsigned long securetsc_get_tsc_khz(void)
+{
+	unsigned long long tsc_freq_mhz;
+
+	setup_force_cpu_cap(X86_FEATURE_TSC_KNOWN_FREQ);
+	rdmsrl(MSR_AMD64_GUEST_TSC_FREQ, tsc_freq_mhz);
+
+	return (unsigned long)(tsc_freq_mhz * 1000);
+}
+
+void __init snp_secure_tsc_init(void)
+{
+	x86_platform.calibrate_cpu = securetsc_get_tsc_khz;
+	x86_platform.calibrate_tsc = securetsc_get_tsc_khz;
+}
arch/x86/kernel/tsc.c +5 -0
index 67aeaba4ba9c..c0eef924b84e 100644
@@ -30,6 +30,7 @@
 #include <asm/i8259.h>
 #include <asm/topology.h>
 #include <asm/uv/uv.h>
+#include <asm/sev.h>
 
 unsigned int __read_mostly cpu_khz;	/* TSC clocks / usec, not used here */
 EXPORT_SYMBOL(cpu_khz);
@@ -1515,6 +1516,10 @@ void __init tsc_early_init(void)
 	/* Don't change UV TSC multi-chassis synchronization */
 	if (is_early_uv_system())
 		return;
+
+	if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))
+		snp_secure_tsc_init();
+
 	if (!determine_cpu_tsc_frequencies(true))
 		return;
 	tsc_enable_sched_clock();
Signature · 1 line
2.34.1
#64 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 36 lines · Nikunj A Dadhania

Calibrating the TSC frequency using the kvmclock is not correct for SecureTSC enabled guests. Use the platform provided TSC frequency via the GUEST_TSC_FREQ MSR (C001_0134h).

---
 arch/x86/include/asm/sev.h |  2 ++
 arch/x86/coco/sev/core.c   | 16 ++++++++++++++++
 arch/x86/kernel/tsc.c      |  5 +++++
 3 files changed, 23 insertions(+)
arch/x86/include/asm/sev.h +2 -0
index 9fd02efef08e..c4dca06b3b01 100644
@@ -493,6 +493,7 @@ int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req
 			   struct snp_guest_request_ioctl *rio);
 
 void __init snp_secure_tsc_prepare(void);
+void __init snp_secure_tsc_init(void);
 
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
@@ -536,6 +537,7 @@ static inline void snp_msg_free(struct snp_msg_desc *mdesc) { }
 static inline int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
 					 struct snp_guest_request_ioctl *rio) { return -ENODEV; }
 static inline void __init snp_secure_tsc_prepare(void) { }
+static inline void __init snp_secure_tsc_init(void) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/coco/sev/core.c +0 -0
index 59c5e716fdd1..1bc668883058 100644

arch/x86/coco/sev/core.c void __init snp_secure_tsc_prepare(void) · line 3288

Quoted 10 lines
arch/x86/coco/sev/core.c +7 -0
@@ -3279,3 +3279,19 @@ void __init snp_secure_tsc_prepare(void)
 
 	pr_debug("SecureTSC enabled");
 }
+
+static unsigned long securetsc_get_tsc_khz(void)
+{
+	unsigned long long tsc_freq_mhz;
+
+	setup_force_cpu_cap(X86_FEATURE_TSC_KNOWN_FREQ);
+	rdmsrl(MSR_AMD64_GUEST_TSC_FREQ, tsc_freq_mhz);

This should never change, right? Can this be put in snp_secure_tsc_init() and just return a saved value that is already in khz form? No reason to perform the MSR access and multiplication every time.

Thanks,
Tom

Quoted 32 lines
arch/x86/coco/sev/core.c +9 -0
+
+	return (unsigned long)(tsc_freq_mhz * 1000);
+}
+
+void __init snp_secure_tsc_init(void)
+{
+	x86_platform.calibrate_cpu = securetsc_get_tsc_khz;
+	x86_platform.calibrate_tsc = securetsc_get_tsc_khz;
+}
arch/x86/kernel/tsc.c +5 -0
index 67aeaba4ba9c..c0eef924b84e 100644
@@ -30,6 +30,7 @@
 #include <asm/i8259.h>
 #include <asm/topology.h>
 #include <asm/uv/uv.h>
+#include <asm/sev.h>
 
 unsigned int __read_mostly cpu_khz;	/* TSC clocks / usec, not used here */
 EXPORT_SYMBOL(cpu_khz);
@@ -1515,6 +1516,10 @@ void __init tsc_early_init(void)
 	/* Don't change UV TSC multi-chassis synchronization */
 	if (is_early_uv_system())
 		return;
+
+	if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))
+		snp_secure_tsc_init();
+
 	if (!determine_cpu_tsc_frequencies(true))
 		return;
 	tsc_enable_sched_clock();
#65 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 38 lines · Tom Lendacky
Quoted 37 lines · Nikunj A Dadhania

Calibrating the TSC frequency using the kvmclock is not correct for SecureTSC enabled guests. Use the platform provided TSC frequency via the GUEST_TSC_FREQ MSR (C001_0134h).

---
 arch/x86/include/asm/sev.h |  2 ++
 arch/x86/coco/sev/core.c   | 16 ++++++++++++++++
 arch/x86/kernel/tsc.c      |  5 +++++
 3 files changed, 23 insertions(+)
arch/x86/include/asm/sev.h +2 -0
index 9fd02efef08e..c4dca06b3b01 100644
@@ -493,6 +493,7 @@ int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req
 			   struct snp_guest_request_ioctl *rio);
 
 void __init snp_secure_tsc_prepare(void);
+void __init snp_secure_tsc_init(void);
 
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
@@ -536,6 +537,7 @@ static inline void snp_msg_free(struct snp_msg_desc *mdesc) { }
 static inline int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req,
 					 struct snp_guest_request_ioctl *rio) { return -ENODEV; }
 static inline void __init snp_secure_tsc_prepare(void) { }
+static inline void __init snp_secure_tsc_init(void) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/coco/sev/core.c +0 -0
index 59c5e716fdd1..1bc668883058 100644
@@ -3279,3 +3279,19 @@ void __init snp_secure_tsc_prepare(void)
 
Quoted 13 lines

arch/x86/coco/sev/core.c void __init snp_secure_tsc_prepare(void) · line 3288

Quoted 9 lines
arch/x86/coco/sev/core.c +7 -0
 	pr_debug("SecureTSC enabled");
 }
+
+static unsigned long securetsc_get_tsc_khz(void)
+{
+	unsigned long long tsc_freq_mhz;
+
+	setup_force_cpu_cap(X86_FEATURE_TSC_KNOWN_FREQ);
+	rdmsrl(MSR_AMD64_GUEST_TSC_FREQ, tsc_freq_mhz);

This should never change, right? Can this be put in snp_secure_tsc_init() and just return a saved value that is already in khz form? No reason to perform the MSR access and multiplication every time.

This happens a couple of times during the boot, so I think this does not have much overhead. Something like below ?

arch/x86/coco/sev/core.c +7 -4
index c7870294a957..69b65c4c850c 100644
@@ -103,6 +103,7 @@ static u64 secrets_pa __ro_after_init;
  */
 static u64 snp_tsc_scale __ro_after_init;
 static u64 snp_tsc_offset __ro_after_init;
+static u64 snp_tsc_freq_khz __ro_after_init;
 
 /* #VC handler runtime per-CPU data */
 struct sev_es_runtime_data {
@@ -3282,16 +3283,18 @@ void __init snp_secure_tsc_prepare(void)
 
 static unsigned long securetsc_get_tsc_khz(void)
 {
-	unsigned long long tsc_freq_mhz;
-
 	setup_force_cpu_cap(X86_FEATURE_TSC_KNOWN_FREQ);
-	rdmsrl(MSR_AMD64_GUEST_TSC_FREQ, tsc_freq_mhz);
 
-	return (unsigned long)(tsc_freq_mhz * 1000);
+	return snp_tsc_freq_khz;
 }
 
 void __init snp_secure_tsc_init(void)
 {
+	unsigned long long tsc_freq_mhz;
+
+	rdmsrl(MSR_AMD64_GUEST_TSC_FREQ, tsc_freq_mhz);
+	snp_tsc_freq_khz = (unsigned long)(tsc_freq_mhz * 1000);
+
 	x86_platform.calibrate_cpu = securetsc_get_tsc_khz;
 	x86_platform.calibrate_tsc = securetsc_get_tsc_khz;
 }

---

Regards
Nikunj

#66 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 15 lines · Nikunj A Dadhania
Quoted 12 lines · Tom Lendacky
Quoted 11 lines · Nikunj A Dadhania

Calibrating the TSC frequency using the kvmclock is not correct for SecureTSC enabled guests. Use the platform provided TSC frequency via the GUEST_TSC_FREQ MSR (C001_0134h).

---
 arch/x86/include/asm/sev.h |  2 ++
 arch/x86/coco/sev/core.c   | 16 ++++++++++++++++
 arch/x86/kernel/tsc.c      |  5 +++++
 3 files changed, 23 insertions(+)

arch/x86/coco/sev/core.c void __init snp_secure_tsc_prepare(void) · line 3286

Quoted 6 lines
arch/x86/coco/sev/core.c +0 -2
@@ -3282,16 +3283,18 @@ void __init snp_secure_tsc_prepare(void)
 
 static unsigned long securetsc_get_tsc_khz(void)
 {
-	unsigned long long tsc_freq_mhz;
-
 	setup_force_cpu_cap(X86_FEATURE_TSC_KNOWN_FREQ);

I was thinking even this can be moved.

Thanks,
Tom

Quoted 22 lines
arch/x86/coco/sev/core.c +6 -3
-	rdmsrl(MSR_AMD64_GUEST_TSC_FREQ, tsc_freq_mhz);
 
-	return (unsigned long)(tsc_freq_mhz * 1000);
+	return snp_tsc_freq_khz;
 }
 
 void __init snp_secure_tsc_init(void)
 {
+	unsigned long long tsc_freq_mhz;
+
+	rdmsrl(MSR_AMD64_GUEST_TSC_FREQ, tsc_freq_mhz);
+	snp_tsc_freq_khz = (unsigned long)(tsc_freq_mhz * 1000);
+
 	x86_platform.calibrate_cpu = securetsc_get_tsc_khz;
 	x86_platform.calibrate_tsc = securetsc_get_tsc_khz;
 }
 
 
---

Regards
Nikunj

#67 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 17 lines · Tom Lendacky
Quoted 16 lines · Nikunj A Dadhania
Quoted 12 lines · Tom Lendacky
Quoted 11 lines · Nikunj A Dadhania

Calibrating the TSC frequency using the kvmclock is not correct for SecureTSC enabled guests. Use the platform provided TSC frequency via the GUEST_TSC_FREQ MSR (C001_0134h).

---
 arch/x86/include/asm/sev.h |  2 ++
 arch/x86/coco/sev/core.c   | 16 ++++++++++++++++
 arch/x86/kernel/tsc.c      |  5 +++++
 3 files changed, 23 insertions(+)
(quoted diff) +0 -0
@@ -3282,16 +3283,18 @@ void __init snp_secure_tsc_prepare(void)
 
Quoted 7 lines

arch/x86/coco/sev/core.c void __init snp_secure_tsc_prepare(void) · line 3286

Quoted 5 lines
arch/x86/coco/sev/core.c +0 -2
 static unsigned long securetsc_get_tsc_khz(void)
 {
-	unsigned long long tsc_freq_mhz;
-
 	setup_force_cpu_cap(X86_FEATURE_TSC_KNOWN_FREQ);

I was thinking even this can be moved.

Yes, I am testing that out. If there is no dependency, I will move it.

Regards,
Nikunj

maintainer
#68 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

In reply to Nikunj A Dadhania
Quoted 1 line · Nikunj A Dadhania

Subject: Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for...

The tip tree preferred format for patch subject prefixes is 'subsys/component:', e.g. 'x86/apic:', 'x86/mm/fault:', 'sched/fair:', 'genirq/core:'. Please do not use file names or complete file paths as prefix. 'git log path/to/file' should give you a reasonable hint in most cases.

Audit your whole set pls.

Review

Quoted 4 lines
(quoted diff) +4 -0
+void __init snp_secure_tsc_init(void)
+{
+	x86_platform.calibrate_cpu = securetsc_get_tsc_khz;
+	x86_platform.calibrate_tsc = securetsc_get_tsc_khz;

The fact that you assign the same function to two different function ptrs already hints at some sort of improper functionality split.

Quoted 13 lines
arch/x86/kernel/tsc.c +1 -0
index 67aeaba4ba9c..c0eef924b84e 100644
@@ -30,6 +30,7 @@
 #include <asm/i8259.h>
 #include <asm/topology.h>
 #include <asm/uv/uv.h>
+#include <asm/sev.h>
 
 unsigned int __read_mostly cpu_khz;	/* TSC clocks / usec, not used here */
 EXPORT_SYMBOL(cpu_khz);

arch/x86/kernel/tsc.c void __init tsc_early_init(void) · line 1521

Quoted 6 lines
arch/x86/kernel/tsc.c +3 -0
@@ -1515,6 +1516,10 @@ void __init tsc_early_init(void)
 	/* Don't change UV TSC multi-chassis synchronization */
 	if (is_early_uv_system())
 		return;
+
+	if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))
+		snp_secure_tsc_init();
arch/x86/coco/sev/core.c +3 -0
index aac066d798ef..24e7c6cf3e29 100644
@@ -3287,6 +3287,9 @@ static unsigned long securetsc_get_tsc_khz(void)
 
 void __init snp_secure_tsc_init(void)
 {
+	if (!cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))
+		return;
+
 	x86_platform.calibrate_cpu = securetsc_get_tsc_khz;
 	x86_platform.calibrate_tsc = securetsc_get_tsc_khz;
 }
arch/x86/kernel/tsc.c +1 -2
index c0eef924b84e..0864b314c26a 100644
@@ -1517,8 +1517,7 @@ void __init tsc_early_init(void)
 	if (is_early_uv_system())
 		return;
 
-	if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))
-		snp_secure_tsc_init();
+	snp_secure_tsc_init();
 
 	if (!determine_cpu_tsc_frequencies(true))
 		return;
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#69 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 11 lines · Borislav Petkov
Quoted 1 line · Nikunj A Dadhania

Subject: Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for...

The tip tree preferred format for patch subject prefixes is 'subsys/component:', e.g. 'x86/apic:', 'x86/mm/fault:', 'sched/fair:', 'genirq/core:'. Please do not use file names or complete file paths as prefix. 'git log path/to/file' should give you a reasonable hint in most cases.

Audit your whole set pls.

Sure, will update

Quoted 7 lines

Review

Quoted 4 lines
(quoted diff) +4 -0
+void __init snp_secure_tsc_init(void)
+{
+	x86_platform.calibrate_cpu = securetsc_get_tsc_khz;
+	x86_platform.calibrate_tsc = securetsc_get_tsc_khz;

The fact that you assign the same function to two different function ptrs already hints at some sort of improper functionality split.

As kvm-clock would have set the callbacks, I need to point them to securetsc_get_tsc_khz().

arch/x86/kernel/kvmclock.c:     x86_platform.calibrate_tsc = kvm_get_tsc_khz;
arch/x86/kernel/kvmclock.c:     x86_platform.calibrate_cpu = kvm_get_tsc_khz;

For virtualized environments, I see that all of them are assigning the same functions to different function ptrs.

arch/x86/kernel/cpu/mshyperv.c:         x86_platform.calibrate_tsc = hv_get_tsc_khz;
arch/x86/kernel/cpu/mshyperv.c:         x86_platform.calibrate_cpu = hv_get_tsc_khz;
arch/x86/kernel/cpu/vmware.c:           x86_platform.calibrate_tsc = vmware_get_tsc_khz;
arch/x86/kernel/cpu/vmware.c:           x86_platform.calibrate_cpu = vmware_get_tsc_khz;
arch/x86/kernel/jailhouse.c:    x86_platform.calibrate_cpu              = jailhouse_get_tsc;
arch/x86/kernel/jailhouse.c:    x86_platform.calibrate_tsc              = jailhouse_get_tsc;
Quoted 41 lines
Quoted 20 lines
arch/x86/kernel/tsc.c +4 -0
index 67aeaba4ba9c..c0eef924b84e 100644
@@ -30,6 +30,7 @@
 #include <asm/i8259.h>
 #include <asm/topology.h>
 #include <asm/uv/uv.h>
+#include <asm/sev.h>
 
 unsigned int __read_mostly cpu_khz;	/* TSC clocks / usec, not used here */
 EXPORT_SYMBOL(cpu_khz);
@@ -1515,6 +1516,10 @@ void __init tsc_early_init(void)
 	/* Don't change UV TSC multi-chassis synchronization */
 	if (is_early_uv_system())
 		return;
+
+	if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))
+		snp_secure_tsc_init();
arch/x86/coco/sev/core.c +3 -0
index aac066d798ef..24e7c6cf3e29 100644
@@ -3287,6 +3287,9 @@ static unsigned long securetsc_get_tsc_khz(void)
 
 void __init snp_secure_tsc_init(void)
 {
+	if (!cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))
+		return;
+
 	x86_platform.calibrate_cpu = securetsc_get_tsc_khz;
 	x86_platform.calibrate_tsc = securetsc_get_tsc_khz;
 }
arch/x86/kernel/tsc.c +0 -0
index c0eef924b84e..0864b314c26a 100644

arch/x86/kernel/tsc.c void __init tsc_early_init(void) · line 1523

Quoted 9 lines
arch/x86/kernel/tsc.c +1 -2
@@ -1517,8 +1517,7 @@ void __init tsc_early_init(void)
 	if (is_early_uv_system())
 		return;
 
-	if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))
-		snp_secure_tsc_init();
+	snp_secure_tsc_init();
 
 	if (!determine_cpu_tsc_frequencies(true))
 		return;

Sure will update.

Regards,
Nikunj

maintainer
#70 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 6 lines · Nikunj A. Dadhania

As kvm-clock would have set the callbacks, I need to point them to securetsc_get_tsc_khz().

arch/x86/kernel/kvmclock.c:     x86_platform.calibrate_tsc = kvm_get_tsc_khz;
arch/x86/kernel/kvmclock.c:     x86_platform.calibrate_cpu = kvm_get_tsc_khz;

For virtualized environments, I see that all of them are assigning the same functions to different function ptrs.

So just because the others do it, you should do it too without even figuring out *why*?

How about you do some git archeology and get to the actual reason why there are two different function pointers and then analyze whether a secure TSC guest can use the same function for both?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#71 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 10 lines · Borislav Petkov
Quoted 6 lines · Nikunj A. Dadhania

As kvm-clock would have set the callbacks, I need to point them to securetsc_get_tsc_khz().

arch/x86/kernel/kvmclock.c:     x86_platform.calibrate_tsc = kvm_get_tsc_khz;
arch/x86/kernel/kvmclock.c:     x86_platform.calibrate_cpu = kvm_get_tsc_khz;

For virtualized environments, I see that all of them are assigning the same functions to different function ptrs.

So just because the others do it, you should do it too without even figuring out *why*?

Again: As kvm-clock has over-ridden both the callbacks, SecureTSC needs to override them with its own.

Regards,
Nikunj

maintainer
#72 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 2 lines · Nikunj A. Dadhania

Again: As kvm-clock has over-ridden both the callbacks, SecureTSC needs to override them with its own.

Again?

Where do you state this fact?

Because I don't see it in the commit message:

"Calibrating the TSC frequency using the kvmclock is not correct for SecureTSC enabled guests. Use the platform provided TSC frequency via the GUEST_TSC_FREQ MSR (C001_0134h)."

Yes, you had this in your reply but that's not good enough.

So again: you need to explain exactly *why* you're doing stuff in patches because I don't have a crystal ball and I don't have special capabilities of reading people's minds. If I had those, I wouldn't be doing this.

And if you had read my reply properly you would've realized that this is not really what I'm asking. I'm asking why you have to assign the *same* function to both function pointers.

And if you had done some git archeology, you would've found this:

aa297292d708 ("x86/tsc: Enumerate SKL cpu_khz and tsc_khz via CPUID")

and then you would've been able to state that you can assign the same function to both function ptrs because the difference between CPU base and TSC frequency does not apply in this case.

But that's too much to ask, right? :-(

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#73 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Nikunj A. Dadhania

Again: As kvm-clock has over-ridden both the callbacks, SecureTSC needs to override them with its own.

Again?

Where do you state this fact?

Quoted 7 lines

Because I don't see it in the commit message:

"Calibrating the TSC frequency using the kvmclock is not correct for SecureTSC enabled guests. Use the platform provided TSC frequency via the GUEST_TSC_FREQ MSR (C001_0134h)."

Yes, you had this in your reply but that's not good enough.

Sure, how about the below:

For SecureTSC enabled guests, TSC frequency should be obtained from the platform provided GUEST_TSC_FREQ MSR (C001_0134h). As paravirt clock(kvm-clock) has over-ridden x86_platform.calibrate_cpu() and x86_platform.calibrate_tsc() callbacks, SecureTSC needs to override them with its own callbacks.

Regards
Nikunj

maintainer
#74 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 6 lines · Nikunj A. Dadhania

Sure, how about the below:

For SecureTSC enabled guests, TSC frequency should be obtained from the platform provided GUEST_TSC_FREQ MSR (C001_0134h). As paravirt clock(kvm-clock) has over-ridden x86_platform.calibrate_cpu() and x86_platform.calibrate_tsc() callbacks, SecureTSC needs to override them with its own callbacks.

Not really.

It's like you're in a contest of "how to write a commit message which is the shortest and has as less information as possible."

Go back, read the subthread and summarize, please, what has been discussed here and for patch 12.

I'm missing the big picture about the relationship between kvmclock and TSC in STSC guests.

After asking so many questions, it sounds to me like this patch and patch 12 should be merged into one and there it should be explained what the strategy is when a STSC guest loads and how kvmclock is supposed to be handled, what is allowed, what is warned about, when the guest terminates what is tainted, yadda yadda.

This all belongs in a single patch logically.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#75 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 8 lines · Borislav Petkov
Quoted 6 lines · Nikunj A. Dadhania

Sure, how about the below:

For SecureTSC enabled guests, TSC frequency should be obtained from the platform provided GUEST_TSC_FREQ MSR (C001_0134h). As paravirt clock(kvm-clock) has over-ridden x86_platform.calibrate_cpu() and x86_platform.calibrate_tsc() callbacks, SecureTSC needs to override them with its own callbacks.

Quoted 4 lines

Not really.

It's like you're in a contest of "how to write a commit message which is the shortest and has as less information as possible."

That is not the goal :-)

Patch 9

---------------------------------------------------------------------------
x86/tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Although the kernel switches over to stable TSC clocksource instead of kvm-clock, TSC frequency calibration still keeps on using the kvm-clock based frequency calibration. This is due to kvmclock_init() updating the x86_platform's CPU and TSC callbacks unconditionally.

For SecureTSC enabled guests, use the GUEST_TSC_FREQ MSR to discover the TSC frequency instead of relying on kvm-clock based frequency calibration. Override both CPU and TSC frequency calibration callbacks with securetsc_get_tsc_khz(). As difference between CPU base and TSC frequency does not apply in this case same callback is being used.
---------------------------------------------------------------------------

Quoted 3 lines

Go back, read the subthread and summarize, please, what has been discussed here and for patch 12.

Here is the updated commit log for patch 12:

---------------------------------------------------------------------------
x86/kvmclock: Warn when kvmclock is selected for SecureTSC enabled guests

Warn users when kvmclock is selected as the clock source for SecureTSC enabled guests. Users can change the clock source to kvm-clock using the sysfs interface while running on a Secure TSC enabled guest. Switching to the hypervisor-controlled kvmclock defeats the purpose of using SecureTSC.

Taint the kernel and issue a warning to the user when the clock source switches to kvmclock, ensuring they are aware of the change and its implications.

---------------------------------------------------------------------------

Quoted 2 lines

I'm missing the big picture about the relationship between kvmclock and TSC in STSC guests.

kvmclock_init() always runs before tsc_early_init(). kvm-clock registers the following during the initialization:

1) TSC calibration callbacks (addressed by patch 9)
2) sched clock (addressed by patch 11)
3) kvm-clock as the clocksource (addressed by patch 10)
4) wall clock callbacks (we don't have a solution for this yet)

I had a summary about this here [1], below snippet with slight modifications after review:

---
To summarise this thread with respect to TSC vs KVM clock, there three key questions:

1) When should kvmclock init be done?
2) How should the TSC frequency be discovered?
3) What should be the sched clock source and how should it be selected in a generic way?

○ Legacy CPU/VMs: VMs running on platforms without non-stop/constant TSC
+ kvm-clock should be registered before tsc-early/tsc
+ Need to calibrate TSC frequency
+ Use kvmclock wallclock
+ Use kvmclock for sched clock

○ Modern CPU/VMs: VMs running on platforms supporting constant, non-stop and reliable TSC
+ kvm-clock should be registered before tsc-early/tsc
+ TSC Frequency:
For SecureTSC: GUEST_TSC_FREQ MSR (C001_0134h) provides the TSC frequency, other AMD guests need to calibrate the TSC frequency.
+ Use kvmclock wallclock
+ Use TSC for sched clock

----

Quoted 7 lines

After asking so many questions, it sounds to me like this patch and patch 12 should be merged into one and there it should be explained what the strategy is when a STSC guest loads and how kvmclock is supposed to be handled, what is allowed, what is warned about, when the guest terminates what is tainted, yadda yadda.

Quoted 1 line

This all belongs in a single patch logically.

Regards
Nikunj

1: https://lore.kernel.org/lkml/[email protected]/

maintainer
#76 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 8 lines · Nikunj A. Dadhania

---------------------------------------------------------------------------
x86/tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Although the kernel switches over to stable TSC clocksource instead of kvm-clock, TSC frequency calibration still keeps on using the kvm-clock based frequency calibration. This is due to kvmclock_init() updating the x86_platform's CPU and TSC callbacks unconditionally.

Quoted 6 lines

For SecureTSC enabled guests, use the GUEST_TSC_FREQ MSR to discover the TSC frequency instead of relying on kvm-clock based frequency calibration. Override both CPU and TSC frequency calibration callbacks with securetsc_get_tsc_khz(). As difference between CPU base and TSC frequency does not apply in this case same callback is being used.
---------------------------------------------------------------------------

Ok.

Quoted 8 lines

---------------------------------------------------------------------------
x86/kvmclock: Warn when kvmclock is selected for SecureTSC enabled guests

Warn users when kvmclock is selected as the clock source for SecureTSC enabled guests. Users can change the clock source to kvm-clock using the sysfs interface while running on a Secure TSC enabled guest. Switching to the hypervisor-controlled kvmclock defeats the purpose of using SecureTSC.

Quoted 4 lines

Taint the kernel and issue a warning to the user when the clock source switches to kvmclock, ensuring they are aware of the change and its implications.

---------------------------------------------------------------------------

Ok.

Quoted 1 line

○ Modern CPU/VMs: VMs running on platforms supporting constant, non-stop and reliable TSC

Modern?

What guarantees do you have on "modern" setups that the HV has no control over the TSC MSRs? None.

The only guarantee you have is when the TSC MSRs are not intercepted - IOW, you're a STSC guest.

So none of that modern stuff means anything - your only case is a STSC guest where you can somewhat reliably know in the guest that the host is not lying to you.

So the only configuration is a STSC guest - everything else should use kvm-clock.

AFAIU.

Quoted 6 lines
Quoted 6 lines

After asking so many questions, it sounds to me like this patch and patch 12 should be merged into one and there it should be explained what the strategy is when a STSC guest loads and how kvmclock is supposed to be handled, what is allowed, what is warned about, when the guest terminates what is tainted, yadda yadda.

Quoted 1 line

This all belongs in a single patch logically.

Now, why aren't you merging patch 9 and 12 into one and calling it:

"Switch Secure TSC guests away from kvm-clock"

or so, where you switch a STSC guest to use the TSC MSRs and warn/taint/terminate the guest if the user chooses otherwise?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#77 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 5 lines · Borislav Petkov
Quoted 1 line · Nikunj A. Dadhania

○ Modern CPU/VMs: VMs running on platforms supporting constant, non-stop and reliable TSC

Modern?

Meaning platforms/CPU that support SNP/TDX, they have constant, non-stop and invariant TSC.

Quoted 2 lines

What guarantees do you have on "modern" setups that the HV has no control over the TSC MSRs? None.

None.

But, non-secure guests uses the regular TSC when the guest is booted with TscInvariant bit set, although it doesn't switch the sched clock and tsc calibration. The guest initially picks up kvm-clock instead of tsc-early as it was registered earlier and both the clocks have the same clock rating(299). But at a later point in time clocksource switches to regular TSC(clock rating 300) from kvm-clock

[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000001] kvm-clock: using sched offset of 1799357702246960 cycles
[    0.001493] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.006289] tsc: Detected 1996.249 MHz processor
[    0.305123] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x398cadd9d93, max_idle_ns: 881590552906 ns
[    1.045759] clocksource: Switched to clocksource kvm-clock
[    1.141326] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x398cadd9d93, max_idle_ns: 881590552906 ns
[    1.144634] clocksource: Switched to clocksource tsc
Quoted 6 lines

The only guarantee you have is when the TSC MSRs are not intercepted - IOW, you're a STSC guest.

So none of that modern stuff means anything - your only case is a STSC guest where you can somewhat reliably know in the guest that the host is not lying to you.

That was my understanding and implementation in v11, where Sean suggested that VMs running on CPUs supporting stable and always running TSC, should switch over to TSC properly[1] and [2], in a generic way.

Quoted 3 lines

So the only configuration is a STSC guest - everything else should use kvm-clock.

That is not right, if non-secure guest is booted with TscInvariant bit set, guest will start using TSC as the clocksource, unfortunately sched clock keeps on using kvm-clock :(

Quoted 8 lines

AFAIU.

Quoted 5 lines
Quoted 5 lines

After asking so many questions, it sounds to me like this patch and patch 12 should be merged into one and there it should be explained what the strategy is when a STSC guest loads and how kvmclock is supposed to be handled, what is allowed, what is warned about, when the guest terminates what is tainted, yadda yadda.

Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

This all belongs in a single patch logically.

Now, why aren't you merging patch 9 and 12 into one and calling it:

"Switch Secure TSC guests away from kvm-clock"

or so, where you switch a STSC guest to use the TSC MSRs and warn/taint/terminate the guest if the user chooses otherwise?

I am going to merge both the patches, but wanted your input on the commit wordings.

Thanks,
Nikunj

1. https://lore.kernel.org/kvm/[email protected]/
2. https://lore.kernel.org/kvm/[email protected]/

maintainer
#78 lore raw

Re: [PATCH v15 09/13] tsc: Use the GUEST_TSC_FREQ MSR for discovering TSC frequency

Quoted 3 lines · Nikunj A. Dadhania

That was my understanding and implementation in v11, where Sean suggested that VMs running on CPUs supporting stable and always running TSC, should switch over to TSC properly[1] and [2], in a generic way.

Sure, you can do that. But that doesn't get rid of the fact that until the HV is safely blocked from touching the TSC MSRs, you cannot trust it fully. IOW, only Secure TSC can give you that, I don't know what provisions TDX has about that.

And then, even if you can get the HV out of the picture, the underlying hw is not guaranteed either. That's why I keep returning to TSC watchdog disable logic in check_system_tsc_reliable() - only then you can somewhat trust the TSC hardware.

Quoted 3 lines

That is not right, if non-secure guest is booted with TscInvariant bit set, guest will start using TSC as the clocksource, unfortunately sched clock keeps on using kvm-clock :(

Again: you can switch to the TSC as much as you want to but until the hypervisor is out of the picture, you can't really trust it.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#79 lore raw

PATCH v15 10/13 tsc: Upgrade TSC clocksource rating

In reply to Nikunj A Dadhania

In virtualized environments running on modern CPUs, the underlying platforms guarantees to have a stable, always running TSC, i.e. that the TSC is a superior timesource as compared to other clock sources (such as kvmclock, HPET, ACPI timer, APIC, etc.).

Upgrade the rating of the early and regular clock source to prefer TSC over other clock sources when TSC is invariant, non-stop and stable.

Cc Alexey Makhalov <[email protected]>
Cc Juergen Gross <[email protected]>
Cc Boris Ostrovsky <[email protected]>
---
 arch/x86/kernel/tsc.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
arch/x86/kernel/tsc.c +17 -0
index c0eef924b84e..900edcde0c9e 100644
@@ -1265,6 +1265,21 @@ static void __init check_system_tsc_reliable(void)
 		tsc_disable_clocksource_watchdog();
 }
 
+static void __init upgrade_clock_rating(struct clocksource *tsc_early,
+					struct clocksource *tsc)
+{
+	/*
+	 * Upgrade the clock rating for TSC early and regular clocksource when
+	 * the underlying platform provides non-stop, invaraint and stable TSC.
+	 */
+	if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) &&
+	    boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&
+	    !tsc_unstable) {
+		tsc_early->rating = 449;
+		tsc->rating = 450;
+	}
+}
+
 /*
  * Make an educated guess if the TSC is trustworthy and synchronized
  * over all CPUs.
@@ -1566,6 +1581,8 @@ void __init tsc_init(void)
 	if (tsc_clocksource_reliable || no_tsc_watchdog)
 		tsc_disable_clocksource_watchdog();
 
+	upgrade_clock_rating(&clocksource_tsc_early, &clocksource_tsc);
+
 	clocksource_register_khz(&clocksource_tsc_early, tsc_khz);
 	detect_art();
 }
Signature · 1 line
2.34.1
maintainer
#80 lore raw

Re: [PATCH v15 10/13] tsc: Upgrade TSC clocksource rating

Quoted 4 lines · Nikunj A Dadhania

In virtualized environments running on modern CPUs, the underlying platforms guarantees to have a stable, always running TSC, i.e. that the TSC is a superior timesource as compared to other clock sources (such as kvmclock, HPET, ACPI timer, APIC, etc.).

That paragraph sounds like marketing fluff and can't be more far away from the truth. We still can't have a stable clocksource in the currently ending 2024 without someone f*cking with it.

Quoted 2 lines

Upgrade the rating of the early and regular clock source to prefer TSC over other clock sources when TSC is invariant, non-stop and stable.

I don't think so...

Have you read all that gunk in check_system_tsc_reliable() and the commits touching that logic which disables the TSC clocksource watchdog caused by all the hw crap that is being produced?

Quoted 14 lines
Cc Alexey Makhalov <[email protected]>
Cc Juergen Gross <[email protected]>
Cc Boris Ostrovsky <[email protected]>
---
 arch/x86/kernel/tsc.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
arch/x86/kernel/tsc.c +0 -0
index c0eef924b84e..900edcde0c9e 100644

arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1273

Quoted 9 lines
arch/x86/kernel/tsc.c +6 -0
@@ -1265,6 +1265,21 @@ static void __init check_system_tsc_reliable(void)
 		tsc_disable_clocksource_watchdog();
 }
 
+static void __init upgrade_clock_rating(struct clocksource *tsc_early,
+					struct clocksource *tsc)
+{
+	/*
+	 * Upgrade the clock rating for TSC early and regular clocksource when
+	 * the underlying platform provides non-stop, invaraint and stable TSC.

Unknown word [invaraint] in comment.
Suggestions: ['invariant', 'inerrant', 'invent', 'intranet', 'infant', 'unvaried', 'informant', 'ingrained', 'entrant', 'inherent', 'unafraid', 'infuriate', 'inferring', 'univalent', 'infringe', 'infringed', 'infuriating']

Spellchecker pls.

arch/x86/kernel/tsc.c static void __init check_system_tsc_reliable(void) · line 1276

Quoted 3 lines
arch/x86/kernel/tsc.c +3 -0
+	 */
+	if (boot_cpu_has(X86_FEATURE_CONSTANT_TSC) &&
+	    boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&

check_for_deprecated_apis: WARNING: arch/x86/kernel/tsc.c:1275: Do not use boot_cpu_has() - use cpu_feature_enabled() instead.
check_for_deprecated_apis: WARNING: arch/x86/kernel/tsc.c:1276: Do not use boot_cpu_has() - use cpu_feature_enabled() instead.

Quoted 21 lines
arch/x86/kernel/tsc.c +8 -0
+	    !tsc_unstable) {
+		tsc_early->rating = 449;
+		tsc->rating = 450;
+	}
+}
+
 /*
  * Make an educated guess if the TSC is trustworthy and synchronized
  * over all CPUs.
@@ -1566,6 +1581,8 @@ void __init tsc_init(void)
 	if (tsc_clocksource_reliable || no_tsc_watchdog)
 		tsc_disable_clocksource_watchdog();
 
+	upgrade_clock_rating(&clocksource_tsc_early, &clocksource_tsc);
+
 	clocksource_register_khz(&clocksource_tsc_early, tsc_khz);
 	detect_art();
 }
Signature · 1 line
2.34.1
Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#81 lore raw

Re: [PATCH v15 10/13] tsc: Upgrade TSC clocksource rating

Quoted 7 lines · Borislav Petkov
Quoted 2 lines · Nikunj A Dadhania

Upgrade the rating of the early and regular clock source to prefer TSC over other clock sources when TSC is invariant, non-stop and stable.

I don't think so...

This is what was suggested by tglx:

"So if you know you want TSC to be selected, then upgrade the rating of
both the early and the regular TSC clocksource and be done with it."

https://lore.kernel.org/lkml/87setgzvn5.ffs@tglx/

Regards,
Nikunj

maintainer
#82 lore raw

Re: [PATCH v15 10/13] tsc: Upgrade TSC clocksource rating

Quoted 4 lines · Nikunj A. Dadhania

This is what was suggested by tglx:

"So if you know you want TSC to be selected, then upgrade the rating of
both the early and the regular TSC clocksource and be done with it."

I highly doubt that he saw what you have now:

Your commit message is talking about virtualized environments but your diff is doing a global, unconditional change which affects *everything*.

If we are going to do this, we need to put this at least through the meatgrinder and run it on *everything* that's out there and see what falls out before we make it the default.

But if tglx thinks this is fine, I won't object - I've uttered my scepticism already.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#83 lore raw

Re: [PATCH v15 10/13] tsc: Upgrade TSC clocksource rating

Quoted 10 lines · Borislav Petkov
Quoted 4 lines · Nikunj A. Dadhania

This is what was suggested by tglx:

"So if you know you want TSC to be selected, then upgrade the rating of
both the early and the regular TSC clocksource and be done with it."

I highly doubt that he saw what you have now:

Your commit message is talking about virtualized environments but your diff is doing a global, unconditional change which affects *everything*.

Right, let me limit this only to virtualized environments as part of CONFIG_PARAVIRT.

Subject: [PATCH] x86/tsc: Upgrade TSC clocksource rating for guests

Hypervisor platform setup (x86_hyper_init::init_platform) routines register their own PV clock sources (KVM, HyperV, and Xen) at different clock ratings resulting in selection of PV clock source even though a stable TSC clock source is available. Upgrade the clock rating of the TSC early and regular clock source to prefer TSC over PV clock sources when TSC is invariant, non-stop and stable

Cc Alexey Makhalov <[email protected]>
Cc Juergen Gross <[email protected]>
Cc Boris Ostrovsky <[email protected]>
---
 arch/x86/kernel/tsc.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
arch/x86/kernel/tsc.c +23 -0
index 34dec0b72ea8..5c6831a42889 100644
@@ -274,10 +274,31 @@ bool using_native_sched_clock(void)
 {
 	return static_call_query(pv_sched_clock) == native_sched_clock;
 }
+
+/*
+ * Upgrade the clock rating for TSC early and regular clocksource when the
+ * underlying platform provides non-stop, invariant, and stable TSC. TSC
+ * early/regular clocksource will be preferred over other para-virtualized clock
+ * sources.
+ */
+static void __init upgrade_clock_rating(struct clocksource *tsc_early,
+					struct clocksource *tsc)
+{
+	if (cpu_feature_enabled(X86_FEATURE_HYPERVISOR) &&
+	    cpu_feature_enabled(X86_FEATURE_CONSTANT_TSC) &&
+	    cpu_feature_enabled(X86_FEATURE_NONSTOP_TSC) &&
+	    !tsc_unstable) {
+		tsc_early->rating = 449;
+		tsc->rating = 450;
+	}
+}
 #else
 u64 sched_clock_noinstr(void) __attribute__((alias("native_sched_clock")));
 
 bool using_native_sched_clock(void) { return true; }
+
+static void __init upgrade_clock_rating(struct clocksource *tsc_early,
+					struct clocksource *tsc) { }
 #endif
 
 notrace u64 sched_clock(void)
@@ -1564,6 +1585,8 @@ void __init tsc_init(void)
 	if (tsc_clocksource_reliable || no_tsc_watchdog)
 		tsc_disable_clocksource_watchdog();
 
+	upgrade_clock_rating(&clocksource_tsc_early, &clocksource_tsc);
+
 	clocksource_register_khz(&clocksource_tsc_early, tsc_khz);
 	detect_art();
 }
Signature · 1 line
2.34.1
maintainer
#84 lore raw

Re: [PATCH v15 10/13] tsc: Upgrade TSC clocksource rating

Quoted 2 lines · Nikunj A. Dadhania

Right, let me limit this only to virtualized environments as part of CONFIG_PARAVIRT.

That's not what you do below - you check whether you're running as a guest.

And that's not sufficient as I just said. I think the only somewhat reliable thing you can do is when you're running as a STSC guest.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#85 lore raw

Re: [PATCH v15 10/13] tsc: Upgrade TSC clocksource rating

Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Nikunj A. Dadhania

Right, let me limit this only to virtualized environments as part of CONFIG_PARAVIRT.

That's not what you do below - you check whether you're running as a guest.

And that's not sufficient as I just said. I think the only somewhat reliable thing you can do is when you're running as a STSC guest.

This is for all guest running with TSC that is constant, non-stop and stable[1] (and detailed response here [2]), STSC is one of them.

Thanks
Nikunj

1. https://lore.kernel.org/kvm/[email protected]/
2. https://lore.kernel.org/all/[email protected]/

#86 lore raw

PATCH v15 11/13 tsc: Switch to native sched clock

In reply to Nikunj A Dadhania

Although the kernel switches over to stable TSC clocksource instead of PV clocksource, the scheduler still keeps on using PV clocks as the sched clock source. This is because KVM, Xen and VMWare, switch the paravirt sched clock handler in their init routines. HyperV is the only PV clock source that checks if the platform provides an invariant TSC and does not switch to PV sched clock.

When switching back to stable TSC, restore the scheduler clock to native_sched_clock().

As the clock selection happens in the stop machine context, schedule delayed work to update the static_call()

Cc Alexey Makhalov <[email protected]>
Cc Juergen Gross <[email protected]>
Cc Boris Ostrovsky <[email protected]>
---
 arch/x86/kernel/tsc.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
arch/x86/kernel/tsc.c +19 -0
index 900edcde0c9e..a337c7e0b26e 100644
@@ -273,10 +273,25 @@ bool using_native_sched_clock(void)
 {
 	return static_call_query(pv_sched_clock) == native_sched_clock;
 }
+
+static void enable_native_sc_work(struct work_struct *work)
+{
+	pr_info("using native sched clock\n");
+	paravirt_set_sched_clock(native_sched_clock);
+}
+static DECLARE_DELAYED_WORK(enable_native_sc, enable_native_sc_work);
+
+static void enable_native_sched_clock(void)
+{
+	if (!using_native_sched_clock())
+		schedule_delayed_work(&enable_native_sc, 0);
+}
 #else
 u64 sched_clock_noinstr(void) __attribute__((alias("native_sched_clock")));
 
 bool using_native_sched_clock(void) { return true; }
+
+static void enable_native_sched_clock(void) { }
 #endif
 
 notrace u64 sched_clock(void)
@@ -1158,6 +1173,10 @@ static void tsc_cs_tick_stable(struct clocksource *cs)
 static int tsc_cs_enable(struct clocksource *cs)
 {
 	vclocks_set_used(VDSO_CLOCKMODE_TSC);
+
+	/* Restore native_sched_clock() when switching to TSC */
+	enable_native_sched_clock();
+
 	return 0;
 }
 
Signature · 1 line
2.34.1
#87 lore raw

PATCH v15 12/13 x86/kvmclock: Abort SecureTSC enabled guest when kvmclock is selected

In reply to Nikunj A Dadhania

SecureTSC enabled guests should use TSC as the only clock source, terminate the guest with appropriate code when clock source switches to hypervisor controlled kvmclock.

---
 arch/x86/include/asm/sev-common.h | 1 +
 arch/x86/include/asm/sev.h        | 2 ++
 arch/x86/coco/sev/shared.c        | 3 +--
 arch/x86/kernel/kvmclock.c        | 9 +++++++++
 4 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/include/asm/sev-common.h +1 -0
index 6ef92432a5ce..ad0743800b0e 100644
@@ -207,6 +207,7 @@ struct snp_psc_desc {
 #define GHCB_TERM_SVSM_VMPL0		8	/* SVSM is present but has set VMPL to 0 */
 #define GHCB_TERM_SVSM_CAA		9	/* SVSM is present but CAA is not page aligned */
 #define GHCB_TERM_SECURE_TSC		10	/* Secure TSC initialization failed */
+#define GHCB_TERM_SECURE_TSC_KVMCLOCK	11	/* KVM clock selected instead of Secure TSC */
 
 #define GHCB_RESP_CODE(v)		((v) & GHCB_MSR_INFO_MASK)
 
arch/x86/include/asm/sev.h +2 -0
index c4dca06b3b01..12b167fd6475 100644
@@ -494,6 +494,7 @@ int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req
 
 void __init snp_secure_tsc_prepare(void);
 void __init snp_secure_tsc_init(void);
+void __noreturn sev_es_terminate(unsigned int set, unsigned int reason);
 
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
@@ -538,6 +539,7 @@ static inline int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_
 					 struct snp_guest_request_ioctl *rio) { return -ENODEV; }
 static inline void __init snp_secure_tsc_prepare(void) { }
 static inline void __init snp_secure_tsc_init(void) { }
+static inline void sev_es_terminate(unsigned int set, unsigned int reason) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/coco/sev/shared.c +1 -2
index 879ab48b705c..840149556241 100644
@@ -117,8 +117,7 @@ static bool __init sev_es_check_cpu_features(void)
 	return true;
 }
 
-static void __head __noreturn
-sev_es_terminate(unsigned int set, unsigned int reason)
+void __head __noreturn sev_es_terminate(unsigned int set, unsigned int reason)
 {
 	u64 val = GHCB_MSR_TERM_REQ;
 
arch/x86/kernel/kvmclock.c +9 -0
index 5b2c15214a6b..39dda04b5ba0 100644
@@ -21,6 +21,7 @@
 #include <asm/hypervisor.h>
 #include <asm/x86_init.h>
 #include <asm/kvmclock.h>
+#include <asm/sev.h>
 
 static int kvmclock __initdata = 1;
 static int kvmclock_vsyscall __initdata = 1;
@@ -150,6 +151,14 @@ bool kvm_check_and_clear_guest_paused(void)
 
 static int kvm_cs_enable(struct clocksource *cs)
 {
+	/*
+	 * For a guest with SecureTSC enabled, the TSC should be the only clock
+	 * source. Abort the guest when kvmclock is selected as the clock
+	 * source.
+	 */
+	if (WARN_ON(cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC)))
+		sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_SECURE_TSC_KVMCLOCK);
+
 	vclocks_set_used(VDSO_CLOCKMODE_PVCLOCK);
 	return 0;
 }
Signature · 1 line
2.34.1
Reviewed-by on v15 12/13
#88 lore raw

Re: [PATCH v15 12/13] x86/kvmclock: Abort SecureTSC enabled guest when kvmclock is selected

Quoted 5 lines · Nikunj A Dadhania

SecureTSC enabled guests should use TSC as the only clock source, terminate the guest with appropriate code when clock source switches to hypervisor controlled kvmclock.

Quoted 80 lines
---
 arch/x86/include/asm/sev-common.h | 1 +
 arch/x86/include/asm/sev.h        | 2 ++
 arch/x86/coco/sev/shared.c        | 3 +--
 arch/x86/kernel/kvmclock.c        | 9 +++++++++
 4 files changed, 13 insertions(+), 2 deletions(-)
arch/x86/include/asm/sev-common.h +1 -0
index 6ef92432a5ce..ad0743800b0e 100644
@@ -207,6 +207,7 @@ struct snp_psc_desc {
 #define GHCB_TERM_SVSM_VMPL0		8	/* SVSM is present but has set VMPL to 0 */
 #define GHCB_TERM_SVSM_CAA		9	/* SVSM is present but CAA is not page aligned */
 #define GHCB_TERM_SECURE_TSC		10	/* Secure TSC initialization failed */
+#define GHCB_TERM_SECURE_TSC_KVMCLOCK	11	/* KVM clock selected instead of Secure TSC */
 
 #define GHCB_RESP_CODE(v)		((v) & GHCB_MSR_INFO_MASK)
 
arch/x86/include/asm/sev.h +2 -0
index c4dca06b3b01..12b167fd6475 100644
@@ -494,6 +494,7 @@ int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_guest_req *req
 
 void __init snp_secure_tsc_prepare(void);
 void __init snp_secure_tsc_init(void);
+void __noreturn sev_es_terminate(unsigned int set, unsigned int reason);
 
 #else	/* !CONFIG_AMD_MEM_ENCRYPT */
 
@@ -538,6 +539,7 @@ static inline int snp_send_guest_request(struct snp_msg_desc *mdesc, struct snp_
 					 struct snp_guest_request_ioctl *rio) { return -ENODEV; }
 static inline void __init snp_secure_tsc_prepare(void) { }
 static inline void __init snp_secure_tsc_init(void) { }
+static inline void sev_es_terminate(unsigned int set, unsigned int reason) { }
 
 #endif	/* CONFIG_AMD_MEM_ENCRYPT */
 
arch/x86/coco/sev/shared.c +1 -2
index 879ab48b705c..840149556241 100644
@@ -117,8 +117,7 @@ static bool __init sev_es_check_cpu_features(void)
 	return true;
 }
 
-static void __head __noreturn
-sev_es_terminate(unsigned int set, unsigned int reason)
+void __head __noreturn sev_es_terminate(unsigned int set, unsigned int reason)
 {
 	u64 val = GHCB_MSR_TERM_REQ;
 
arch/x86/kernel/kvmclock.c +9 -0
index 5b2c15214a6b..39dda04b5ba0 100644
@@ -21,6 +21,7 @@
 #include <asm/hypervisor.h>
 #include <asm/x86_init.h>
 #include <asm/kvmclock.h>
+#include <asm/sev.h>
 
 static int kvmclock __initdata = 1;
 static int kvmclock_vsyscall __initdata = 1;
@@ -150,6 +151,14 @@ bool kvm_check_and_clear_guest_paused(void)
 
 static int kvm_cs_enable(struct clocksource *cs)
 {
+	/*
+	 * For a guest with SecureTSC enabled, the TSC should be the only clock
+	 * source. Abort the guest when kvmclock is selected as the clock
+	 * source.
+	 */
+	if (WARN_ON(cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC)))
+		sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_SECURE_TSC_KVMCLOCK);
+
 	vclocks_set_used(VDSO_CLOCKMODE_PVCLOCK);
 	return 0;
 }
maintainer
#89 lore raw

Re: [PATCH v15 12/13] x86/kvmclock: Abort SecureTSC enabled guest when kvmclock is selected

In reply to Nikunj A Dadhania
Quoted 3 lines · Nikunj A Dadhania

SecureTSC enabled guests should use TSC as the only clock source, terminate the guest with appropriate code when clock source switches to hypervisor controlled kvmclock.

This looks silly. Why not prevent kvm_register_clock() from succeeding simply on a secure-TSC guest?

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#90 lore raw

Re: [PATCH v15 12/13] x86/kvmclock: Abort SecureTSC enabled guest when kvmclock is selected

Quoted 6 lines · Borislav Petkov
Quoted 3 lines · Nikunj A Dadhania

SecureTSC enabled guests should use TSC as the only clock source, terminate the guest with appropriate code when clock source switches to hypervisor controlled kvmclock.

This looks silly.

I can drop this patch, and if the admin wants to change the clock source to kvm-clock from Secure TSC, that will be permitted.

Quoted 2 lines

Why not prevent kvm_register_clock() from succeeding simply on a secure-TSC guest?

Or we can have something like below that will prevent kvm-clock registration:

arch/x86/kernel/kvmclock.c +6 -8
index 39dda04b5ba0..e68683a65018 100644
@@ -151,14 +151,6 @@ bool kvm_check_and_clear_guest_paused(void)
 
 static int kvm_cs_enable(struct clocksource *cs)
 {
-	/*
-	 * For a guest with SecureTSC enabled, the TSC should be the only clock
-	 * source. Abort the guest when kvmclock is selected as the clock
-	 * source.
-	 */
-	if (WARN_ON(cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC)))
-		sev_es_terminate(SEV_TERM_SET_LINUX, GHCB_TERM_SECURE_TSC_KVMCLOCK);
-
 	vclocks_set_used(VDSO_CLOCKMODE_PVCLOCK);
 	return 0;
 }
@@ -352,6 +344,12 @@ void __init kvmclock_init(void)
 	    boot_cpu_has(X86_FEATURE_NONSTOP_TSC) &&
 	    !check_tsc_unstable())
 		kvm_clock.rating = 299;
+        /*
+         * For a guest with SecureTSC enabled, the TSC clock source should be
+         * used. Prevent the kvm-clock source registration.
+         */
+        if (cc_platform_has(CC_ATTR_GUEST_SNP_SECURE_TSC))
+		return;
 
 	clocksource_register_hz(&kvm_clock, NSEC_PER_SEC);
 	pv_info.name = "KVM";
maintainer
#91 lore raw

Re: [PATCH v15 12/13] x86/kvmclock: Abort SecureTSC enabled guest when kvmclock is selected

Quoted 2 lines · Nikunj A. Dadhania

I can drop this patch, and if the admin wants to change the clock source to kvm-clock from Secure TSC, that will be permitted.

Why would the admin want that and why would we even support that?

Your commit message starts with:

"SecureTSC enabled guests should use TSC as the only clock source, terminate the guest with appropriate code when clock source switches to hypervisor controlled kvmclock."

So what are we even doing here?

You do notice that you're missing answering the "why" question on all those, which is kinda the most important one. You're adding support for something but the supported use cases are not really clear...

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#92 lore raw

Re: [PATCH v15 12/13] x86/kvmclock: Abort SecureTSC enabled guest when kvmclock is selected

Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Nikunj A. Dadhania

I can drop this patch, and if the admin wants to change the clock source to kvm-clock from Secure TSC, that will be permitted.

Why would the admin want that and why would we even support that?

I am not saying that admin will do that, I am saying that it is a possibility.

Changing clocksource is supported via sysfs interface:

echo "kvm-clock" > /sys/devices/system/clocksource/clocksource0/current_clocksource

This is what we are trying to prevent in this patch.

Regards,
Nikunj

maintainer
#93 lore raw

Re: [PATCH v15 12/13] x86/kvmclock: Abort SecureTSC enabled guest when kvmclock is selected

Quoted 7 lines · Nikunj A. Dadhania
Quoted 4 lines · Borislav Petkov
Quoted 2 lines · Nikunj A. Dadhania

I can drop this patch, and if the admin wants to change the clock source to kvm-clock from Secure TSC, that will be permitted.

Quoted 7 lines
Quoted 1 line

Why would the admin want that and why would we even support that?

I am not saying that admin will do that, I am saying that it is a possibility.

Changing clocksource is supported via sysfs interface:

echo "kvm-clock" > /sys/devices/system/clocksource/clocksource0/current_clocksource

You can do that in the guest, right?

Not on the host.

If so, are you basically saying that users will be able to kill their guests simply by switching the clocksource?

Because this would be dumb of us.

And then the real thing to do should be something along the lines of

"You're running a Secure TSC guest, changing the clocksource is not allowed!"

or even better we warn when the user changes it but allow the change and taint the kernel.

...

Yeah, stuff like that comes out only when I scratch the surface and ask about it. And then people complain about it taking too long.

Well, tough luck.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#94 lore raw

Re: [PATCH v15 12/13] x86/kvmclock: Abort SecureTSC enabled guest when kvmclock is selected

Quoted 10 lines · Borislav Petkov
Quoted 9 lines · Nikunj A. Dadhania
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Nikunj A. Dadhania

I can drop this patch, and if the admin wants to change the clock source to kvm-clock from Secure TSC, that will be permitted.

Why would the admin want that and why would we even support that?

Quoted 7 lines
Quoted 5 lines

I am not saying that admin will do that, I am saying that it is a possibility.

Changing clocksource is supported via sysfs interface:

echo "kvm-clock" > /sys/devices/system/clocksource/clocksource0/current_clocksource

You can do that in the guest, right?

Yes.

Quoted 2 lines

Not on the host.

Right, kvm-clock is not available on host.

Quoted 11 lines

If so, are you basically saying that users will be able to kill their guests simply by switching the clocksource?

Because this would be dumb of us.

And then the real thing to do should be something along the lines of

"You're running a Secure TSC guest, changing the clocksource is not allowed!"

or even better we warn when the user changes it but allow the change and taint the kernel.

Sure, that sounds better. I will keep the warning and taint the kernel.

Regards,
Nikunj

#95 lore raw

PATCH v15 13/13 x86/sev: Allow Secure TSC feature for SNP guests

In reply to Nikunj A Dadhania

Now that all the required plumbing is done for enabling SNP Secure TSC feature, add Secure TSC to SNP features present list.

---
 arch/x86/boot/compressed/sev.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/boot/compressed/sev.c +2 -1
index cd44e120fe53..bb55934c1cee 100644
@@ -401,7 +401,8 @@ void do_boot_stage2_vc(struct pt_regs *regs, unsigned long exit_code)
  * by the guest kernel. As and when a new feature is implemented in the
  * guest kernel, a corresponding bit should be added to the mask.
  */
-#define SNP_FEATURES_PRESENT	MSR_AMD64_SNP_DEBUG_SWAP
+#define SNP_FEATURES_PRESENT	(MSR_AMD64_SNP_DEBUG_SWAP |	\
+				 MSR_AMD64_SNP_SECURE_TSC)
 
 u64 snp_get_unsupported_features(u64 status)
 {
Signature · 1 line
2.34.1