Skip to contentSkip to conversation

Support microcode updates affecting SGX

RFC PATCH Superseded Cathy Zhang 4 participants 28 messages Apr 1, 2022 – Apr 6, 2022

Subsystems (2)

v3 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

v3 cover Support microcode updates affecting SGX

v2:
https://lore.kernel.org/lkml/[email protected]/

Changes since v2:
- Changes are made in patch "x86/sgx: Introduce mechanism to prevent new initializations of EPC pages" by moving SGX2 related changes out. It allows this series to be applied on tip/x86/sgx branch with only picking up some auxiliary changes from SGX2 series, rather than depend on the whole set. (Jarkko Sakkinen, Reinette Chatre)

Changes since v1:
- Remove the sysfs file svnupdate. (Thomas Gleixner, Dave Hansen)
- Let late microcode load path call ENCLS[EUPDATESVN] procedure directly. (Borislav Petkov)
- Update cover letter by removing saying that "...triggered by administrators via sysfs...".
- Drop the patch for documentation change.

cover letter:

== General Microcode Background ==

Historically, microcode updates are applied by the BIOS or early in boot. In recent years, several trends have made these old approaches less palatable.

First, the cadence of microcode updates has increased to deliver security mitigations. Second, the value of those updates has increased, meaning that any delay in applying them is unacceptable. Third, users have become accustomed to approaches like hot patching their kernels and have a growing aversion to reboots in general.

Users want microcode updates to behave more like a hot patching a kernel and less like a BIOS update.

== SGX Attestation Background ==

SGX enclaves have an attestation mechanism. An enclave might, for instance, need to attest to its state before it is given a special decryption key. Since SGX must trust the CPU microcode, attestation incorporates the microcode versions of all processors on the system and is affected by microcode updates. This allows the entity to which the enclave is attesting to make deployment decisions based on the microcode version. For example, an enclave might be denied a decryption key if it runs on a system that has old microcode without a specific mitigation.

Unfortunately, this attestation metric (called CPUSVN) is only a snapshot. When the kernel first uses SGX (successfully executes any ENCLS instruction), SGX inspects all CPUs in the system and incorporates a record of their microcode versions into CPUSVN. Today, that value is locked and is not updated until a reboot.

== Problems ==

This means that, although the microcode may be update, enclaves can never attest to this fact. Enclaves are stuck attesting to the old version until a reboot.

Old enclaves created before the microcode update are presumed to be compromised must not be allowed to attest with the new microcode version.

== Solution ==

EUPDATESVN is a new SGX instruction which allows enclave attestation to include information about updated microcode without a reboot.

Whenever a microcode update affects SGX, the SGX attestation architecture assumes that all running enclaves and cryptographic assets (like internal SGX encryption keys) have been compromised. To mitigate the impact of this presumed compromise, EUPDATESVN success requires that all SGX memory to be marked as "unused" and its contents destroyed. This requirement ensures that no compromised enclave can survive the EUPDATESVN procedure and provides an opportunity to generate new cryptographic assets.

This series implements the infrastructure needed to track and tear down bare-metal enclaves and then run EUPDATESVN, it will be called by the late microcode load path after the microcode update.

This is a very slow operation. It is, of course, exceedingly disruptive to enclaves but should be infrequent as microcode updates are released on the order of every few months. Also, this is not the first piece of the SGX architecture which will destroy all enclave contents.

A follow-on series will add Virtual EPC (KVM guest) support.

Here is the spec for your reference:
https://cdrdv2.intel.com/v1/dl/getContent/648682?explicitVersion=true

This is series is based on tip/x86/sgx with the following additionally applied:

"x86/sgx: Export sgx_encl_ewb_cpumask()"
https://lore.kernel.org/lkml/[email protected]/T/#m44e2b931e82a87a8b2c80058130182eb747fbcf0
"x86/sgx: Rename sgx_encl_ewb_cpumask() as sgx_encl_cpumask()"
https://lore.kernel.org/lkml/[email protected]/T/#mf6268a66b5c48ca9a18a772b6eaea097c315dc1d
"x86/sgx: Make sgx_ipi_cb() available internally"
https://lore.kernel.org/lkml/[email protected]/T/#ma3330f8ee8136aa084d0a2b5f110331e37f44c52
"x86/sgx: Keep record of SGX page type"
https://lore.kernel.org/lkml/[email protected]/T/#m4ae80fdf67ad330119bfc2abaea845baa24ed14c

Cathy Zhang (10):
x86/sgx: Introduce mechanism to prevent new initializations of EPC
pages
x86/sgx: Provide VA page non-NULL owner
x86/sgx: Save enclave pointer for VA page
x86/sgx: Keep record for SGX VA and Guest page type x86/sgx: Save the size of each EPC section
x86/sgx: Forced EPC page zapping for EUPDATESVN x86/sgx: Define error codes for ENCLS[EUPDATESVN] x86/sgx: Implement ENCLS[EUPDATESVN]
x86/cpu: Call ENCLS[EUPDATESVN] procedure in microcode update x86/sgx: Call ENCLS[EUPDATESVN] during SGX initialization

 arch/x86/include/asm/microcode.h |   5 +
 arch/x86/include/asm/sgx.h       |  48 +++-
 arch/x86/kernel/cpu/sgx/encl.h   |   4 +-
 arch/x86/kernel/cpu/sgx/encls.h  |  14 +
 arch/x86/kernel/cpu/sgx/sgx.h    |  23 +-
 arch/x86/kernel/cpu/common.c     |   9 +
 arch/x86/kernel/cpu/sgx/encl.c   |  40 ++-
 arch/x86/kernel/cpu/sgx/ioctl.c  |  28 +-
 arch/x86/kernel/cpu/sgx/main.c   | 459 ++++++++++++++++++++++++++++++-
 arch/x86/kernel/cpu/sgx/virt.c   |  22 ++
 10 files changed, 628 insertions(+), 24 deletions(-)
Signature · 1 line
2.17.1
#1 lore raw

RFC PATCH v3 1/10 x86/sgx: Introduce mechanism to prevent new initializations of EPC pages

== Background ==

EUPDATESVN is a new SGX instruction which allows enclave attestation to include information about updated microcode without a reboot.

The SGX hardware maintains metadata for each enclave page to help enforce its security guarantees. This includes things like a record of the enclave to which the page belongs and the type of the page:
SGX metadata like "VA" or "SECS" pages, or regular enclave pages like those that store user data.

Before an EUPDATESVN operation can be successful, all SGX memory (aka. EPC) must be marked as "unused" in the SGX hardware metadata (aka, EPCM). The SGX microcode now maintains a reference count of pages which are unused to aid in determining when all pages reach the "unused" state.

Both bare-metal and KVM guest EPC must be made unused. To increase the chance of a successful EUPDATESVN, the kernel prevents existing enclaves from creating new, valid pages and prevents new enclave creation (creating an enclave involves initializing a "SECS" page).

The entire EUPDATESVN process is very slow since it potentially affects gigabytes of enclave memory. It can potentially take seconds or minutes to complete. Userspace may encounter -EBUSY errors during the update and is expected to retry.

== Patch contents ==

Introduce mechanism to prevent new initializations of EPC pages.

Use a flag to indicate when SGX EPC pages are "locked", which means it's not allowed to allocate new EPC page for use. Check it in all paths that can initialize an EPC page. Use SRCU to ensure that the flag is visible across the system before proceeding with an update.

Add checks to all sites that call SGX instructions that can transition pages from unused to initialized to ensure that the SRCU lock is held.

---
Changes since v2:
- Move out the SGX2 related change to remove the dependency. (Jarkko Sakkinen, Reinette Chatre).

---
 arch/x86/kernel/cpu/sgx/encls.h |  8 +++++++
 arch/x86/kernel/cpu/sgx/sgx.h   |  3 +++
 arch/x86/kernel/cpu/sgx/encl.c  | 19 +++++++++++++++++
 arch/x86/kernel/cpu/sgx/ioctl.c | 25 +++++++++++++++++++++-
 arch/x86/kernel/cpu/sgx/main.c  | 37 +++++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/sgx/virt.c  | 20 ++++++++++++++++++
 6 files changed, 111 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/encls.h +8 -0
index fa04a73daf9c..60321c5f5718 100644
@@ -138,6 +138,8 @@ static inline bool encls_failed(int ret)
 
 static inline int __ecreate(struct sgx_pageinfo *pginfo, void *secs)
 {
+	lockdep_assert_held(&sgx_lock_epc_srcu);
+
 	return __encls_2(ECREATE, pginfo, secs);
 }
 
@@ -148,6 +150,8 @@ static inline int __eextend(void *secs, void *addr)
 
 static inline int __eadd(struct sgx_pageinfo *pginfo, void *addr)
 {
+	lockdep_assert_held(&sgx_lock_epc_srcu);
+
 	return __encls_2(EADD, pginfo, addr);
 }
 
@@ -179,6 +183,8 @@ static inline int __etrack(void *addr)
 static inline int __eldu(struct sgx_pageinfo *pginfo, void *addr,
 			 void *va)
 {
+	lockdep_assert_held(&sgx_lock_epc_srcu);
+
 	return __encls_ret_3(ELDU, pginfo, addr, va);
 }
 
@@ -191,6 +197,8 @@ static inline int __epa(void *addr)
 {
 	unsigned long rbx = SGX_PAGE_TYPE_VA;
 
+	lockdep_assert_held(&sgx_lock_epc_srcu);
+
 	return __encls_2(EPA, rbx, addr);
 }
 
arch/x86/kernel/cpu/sgx/sgx.h +3 -0
index b30cee4de903..d7a1490d90bb 100644
@@ -103,4 +103,7 @@ static inline int __init sgx_vepc_init(void)
 
 void sgx_update_lepubkeyhash(u64 *lepubkeyhash);
 
+extern struct srcu_struct sgx_lock_epc_srcu;
+bool sgx_epc_is_locked(void);
+
 #endif /* _X86_SGX_H */
arch/x86/kernel/cpu/sgx/encl.c +19 -0
index 02ff9ac83985..68c8d65a8dee 100644
@@ -142,6 +142,7 @@ static vm_fault_t sgx_vma_fault(struct vm_fault *vmf)
 	unsigned long phys_addr;
 	struct sgx_encl *encl;
 	vm_fault_t ret;
+	int srcu_idx;
 
 	encl = vma->vm_private_data;
 
@@ -153,11 +154,18 @@ static vm_fault_t sgx_vma_fault(struct vm_fault *vmf)
 	if (unlikely(!encl))
 		return VM_FAULT_SIGBUS;
 
+	srcu_idx = srcu_read_lock(&sgx_lock_epc_srcu);
+	if (sgx_epc_is_locked()) {
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+		return VM_FAULT_SIGBUS;
+	}
+
 	mutex_lock(&encl->lock);
 
 	entry = sgx_encl_load_page(encl, addr, vma->vm_flags);
 	if (IS_ERR(entry)) {
 		mutex_unlock(&encl->lock);
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
 
 		if (PTR_ERR(entry) == -EBUSY)
 			return VM_FAULT_NOPAGE;
@@ -170,12 +178,14 @@ static vm_fault_t sgx_vma_fault(struct vm_fault *vmf)
 	ret = vmf_insert_pfn(vma, addr, PFN_DOWN(phys_addr));
 	if (ret != VM_FAULT_NOPAGE) {
 		mutex_unlock(&encl->lock);
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
 
 		return VM_FAULT_SIGBUS;
 	}
 
 	sgx_encl_test_and_clear_young(vma->vm_mm, entry);
 	mutex_unlock(&encl->lock);
+	srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
 
 	return VM_FAULT_NOPAGE;
 }
@@ -323,6 +333,7 @@ static int sgx_vma_access(struct vm_area_struct *vma, unsigned long addr,
 	struct sgx_encl_page *entry = NULL;
 	char data[sizeof(unsigned long)];
 	unsigned long align;
+	int srcu_idx;
 	int offset;
 	int cnt;
 	int ret = 0;
@@ -339,8 +350,15 @@ static int sgx_vma_access(struct vm_area_struct *vma, unsigned long addr,
 		return -EFAULT;
 
 	for (i = 0; i < len; i += cnt) {
+		srcu_idx = srcu_read_lock(&sgx_lock_epc_srcu);
+		if (sgx_epc_is_locked()) {
+			ret = -EBUSY;
+			goto out;
+		}
+
 		entry = sgx_encl_reserve_page(encl, (addr + i) & PAGE_MASK,
 					      vma->vm_flags);
+
 		if (IS_ERR(entry)) {
 			ret = PTR_ERR(entry);
 			break;
@@ -366,6 +384,7 @@ static int sgx_vma_access(struct vm_area_struct *vma, unsigned long addr,
 
 out:
 		mutex_unlock(&encl->lock);
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
 
 		if (ret)
 			break;
arch/x86/kernel/cpu/sgx/ioctl.c +24 -1
index b3c2e8d58142..00668e50848d 100644
@@ -147,6 +147,7 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)
 static long sgx_ioc_enclave_create(struct sgx_encl *encl, void __user *arg)
 {
 	struct sgx_enclave_create create_arg;
+	int srcu_idx;
 	void *secs;
 	int ret;
 
@@ -162,9 +163,20 @@ static long sgx_ioc_enclave_create(struct sgx_encl *encl, void __user *arg)
 
 	if (copy_from_user(secs, (void __user *)create_arg.src, PAGE_SIZE))
 		ret = -EFAULT;
-	else
+	else {
+		srcu_idx = srcu_read_lock(&sgx_lock_epc_srcu);
+		if (sgx_epc_is_locked()) {
+			srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+			ret = -EBUSY;
+			goto out;
+		}
+
 		ret = sgx_encl_create(encl, secs);
 
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+	}
+
+out:
 	kfree(secs);
 	return ret;
 }
@@ -418,6 +430,7 @@ static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg)
 	struct sgx_enclave_add_pages add_arg;
 	struct sgx_secinfo secinfo;
 	unsigned long c;
+	int srcu_idx;
 	int ret;
 
 	if (!test_bit(SGX_ENCL_CREATED, &encl->flags) ||
@@ -455,8 +468,18 @@ static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg)
 		if (need_resched())
 			cond_resched();
 
+		srcu_idx = srcu_read_lock(&sgx_lock_epc_srcu);
+		if (sgx_epc_is_locked()) {
+			ret = -EBUSY;
+			srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+			break;
+		}
+
 		ret = sgx_encl_add_page(encl, add_arg.src + c, add_arg.offset + c,
 					&secinfo, add_arg.flags);
+
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+
 		if (ret)
 			break;
 	}
arch/x86/kernel/cpu/sgx/main.c +37 -0
index b3aba1e1274c..10360f06c0df 100644
@@ -23,6 +23,17 @@ static int sgx_nr_epc_sections;
 static struct task_struct *ksgxd_tsk;
 static DECLARE_WAIT_QUEUE_HEAD(ksgxd_waitq);
 static DEFINE_XARRAY(sgx_epc_address_space);
+/*
+ * The flag sgx_epc_locked prevents any new SGX flows that
+ * may attempt to allocate a new EPC page.
+ */
+static bool __rcu sgx_epc_locked;
+/*
+ * By synchronizing around sgx_epc_locked SRCU ensures that any executing
+ * SGX flows have completed before proceeding with an SVN update. New SGX flows
+ * will be prevented from starting during an SVN update.
+ */
+DEFINE_SRCU(sgx_lock_epc_srcu);
 
 /*
  * These variables are part of the state of the reclaimer, and must be accessed
@@ -407,6 +418,8 @@ static bool sgx_should_reclaim(unsigned long watermark)
 
 static int ksgxd(void *p)
 {
+	int srcu_idx;
+
 	set_freezable();
 
 	/*
@@ -427,9 +440,15 @@ static int ksgxd(void *p)
 				     kthread_should_stop() ||
 				     sgx_should_reclaim(SGX_NR_HIGH_PAGES));
 
+		srcu_idx = srcu_read_lock(&sgx_lock_epc_srcu);
+		if (sgx_epc_is_locked())
+			goto maybe_resched;
+
 		if (sgx_should_reclaim(SGX_NR_HIGH_PAGES))
 			sgx_reclaim_pages();
 
+maybe_resched:
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
 		cond_resched();
 	}
 
@@ -972,3 +991,21 @@ static int __init sgx_init(void)
 }
 
 device_initcall(sgx_init);
+
+static void sgx_lock_epc(void)
+{
+	sgx_epc_locked = true;
+	synchronize_srcu(&sgx_lock_epc_srcu);
+}
+
+static void sgx_unlock_epc(void)
+{
+	sgx_epc_locked = false;
+	synchronize_srcu(&sgx_lock_epc_srcu);
+}
+
+bool sgx_epc_is_locked(void)
+{
+	lockdep_assert_held(&sgx_lock_epc_srcu);
+	return sgx_epc_locked;
+}
arch/x86/kernel/cpu/sgx/virt.c +20 -0
index 6a77a14eee38..e953816d7c8b 100644
@@ -75,10 +75,21 @@ static vm_fault_t sgx_vepc_fault(struct vm_fault *vmf)
 {
 	struct vm_area_struct *vma = vmf->vma;
 	struct sgx_vepc *vepc = vma->vm_private_data;
+	int srcu_idx;
 	int ret;
 
 	mutex_lock(&vepc->lock);
+	srcu_idx = srcu_read_lock(&sgx_lock_epc_srcu);
+
+	if (sgx_epc_is_locked()) {
+		ret = -EBUSY;
+		goto out_unlock;
+	}
+
 	ret = __sgx_vepc_fault(vepc, vma, vmf->address);
+
+out_unlock:
+	srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
 	mutex_unlock(&vepc->lock);
 
 	if (!ret)
@@ -331,6 +342,7 @@ int __init sgx_vepc_init(void)
 int sgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void __user *secs,
 		     int *trapnr)
 {
+	int srcu_idx;
 	int ret;
 
 	/*
@@ -347,6 +359,12 @@ int sgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void __user *secs,
 	if (WARN_ON_ONCE(!access_ok(secs, PAGE_SIZE)))
 		return -EINVAL;
 
+	srcu_idx = srcu_read_lock(&sgx_lock_epc_srcu);
+	if (sgx_epc_is_locked()) {
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+		return -EBUSY;
+	}
+
 	__uaccess_begin();
 	ret = __ecreate(pageinfo, (void *)secs);
 	__uaccess_end();
@@ -356,6 +374,8 @@ int sgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void __user *secs,
 		return -EFAULT;
 	}
 
+	srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+
 	/* ECREATE doesn't return an error code, it faults or succeeds. */
 	WARN_ON_ONCE(ret);
 	return 0;
Signature · 1 line
2.17.1
maintainer
#2 lore raw

Re: [RFC PATCH v3 01/10] x86/sgx: Introduce mechanism to prevent new initializations of EPC pages

Quoted 21 lines · Cathy Zhang
arch/x86/kernel/cpu/sgx/ioctl.c +5 -1
index b3c2e8d58142..00668e50848d 100644
@@ -147,6 +147,7 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)
 static long sgx_ioc_enclave_create(struct sgx_encl *encl, void __user *arg)
 {
 	struct sgx_enclave_create create_arg;
+	int srcu_idx;
 	void *secs;
 	int ret;
 
@@ -162,9 +163,20 @@ static long sgx_ioc_enclave_create(struct sgx_encl *encl, void __user *arg)
 
 	if (copy_from_user(secs, (void __user *)create_arg.src, PAGE_SIZE))
 		ret = -EFAULT;
-	else
+	else {
+		srcu_idx = srcu_read_lock(&sgx_lock_epc_srcu);
+		if (sgx_epc_is_locked()) {
+			srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);

arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_create(struct sgx_encl *encl, void __user *arg) · line 179

Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c +8 -0
+			ret = -EBUSY;
+			goto out;
+		}
+
 		ret = sgx_encl_create(encl, secs);
 
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+	}
+
+out:

Nit: I'd rename this as 'err:'.

arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_create(struct sgx_encl *encl, void __user *arg) · line 182

Quoted 3 lines
arch/x86/kernel/cpu/sgx/ioctl.c +0 -0
 	kfree(secs);
 	return ret;
 }

Please, take advantage of the label you created anyway:

  	if (copy_from_user(secs, (void __user *)create_arg.src, PAGE_SIZE {
  		ret = -EFAULT;
                goto out;
        }

Then fail cases have the exact same rollback procedure, and the code is easier to read because it is less nested.

BR, Jarkko

#3 lore raw

RE: [RFC PATCH v3 01/10] x86/sgx: Introduce mechanism to prevent new initializations of EPC pages

Hi Jarkko,

Quoted 40 lines

-----Original Message-----
From: Jarkko Sakkinen <[email protected]>
Sent: Sunday, April 3, 2022 4:14 PM
To: Zhang, Cathy <[email protected]>
Cc: [email protected]; [email protected]; Chatre, Reinette <[email protected]>; Hansen, Dave <[email protected]>; Raj, Ashok <[email protected]>
Subject: Re: [RFC PATCH v3 01/10] x86/sgx: Introduce mechanism to prevent new initializations of EPC pages

Quoted 29 lines · Cathy Zhang
(quoted diff) +0 -0

b/arch/x86/kernel/cpu/sgx/ioctl.c index b3c2e8d58142..00668e50848d 100644

arch/x86/kernel/cpu/sgx/ioctl.c +0 -0
@@ -147,6 +147,7 @@ static int sgx_encl_create(struct sgx_encl *encl,

struct sgx_secs *secs) static long sgx_ioc_enclave_create(struct sgx_encl *encl, void __user *arg) {

arch/x86/kernel/cpu/sgx/ioctl.c +9 -1
 	struct sgx_enclave_create create_arg;
+	int srcu_idx;
 	void *secs;
 	int ret;
@@ -162,9 +163,20 @@ static long sgx_ioc_enclave_create(struct
 sgx_encl *encl, void __user *arg)
 
 	if (copy_from_user(secs, (void __user *)create_arg.src, PAGE_SIZE))
 		ret = -EFAULT;
-	else
+	else {
+		srcu_idx = srcu_read_lock(&sgx_lock_epc_srcu);
+		if (sgx_epc_is_locked()) {
+			srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+			ret = -EBUSY;
+			goto out;
+		}
+
 		ret = sgx_encl_create(encl, secs);
 
Quoted 6 lines

arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_create(struct sgx_encl *encl, void __user *arg) · line 179

Quoted 4 lines
arch/x86/kernel/cpu/sgx/ioctl.c +4 -0
+		srcu_read_unlock(&sgx_lock_epc_srcu, srcu_idx);
+	}
+
+out:

Nit: I'd rename this as 'err:'.

Renamed as 'err'.

Quoted 7 lines
Quoted 4 lines
arch/x86/kernel/cpu/sgx/ioctl.c +0 -0
 	kfree(secs);
 	return ret;

}

Please, take advantage of the label you created anyway:

Quoted 7 lines
  	if (copy_from_user(secs, (void __user *)create_arg.src, PAGE_SIZE {
  		ret = -EFAULT;
                goto out;
        }

Then fail cases have the exact same rollback procedure, and the code is easier to read because it is less nested.

Update is done. The error branches in this function will all "goto err".

Quoted 3 lines

BR, Jarkko

#4 lore raw

RFC PATCH v3 2/10 x86/sgx: Provide VA page non-NULL owner

In reply to Cathy Zhang

Provide non-NULL owner for VA page, which tells that the EPC page is allocated for use.

---
 arch/x86/kernel/cpu/sgx/encl.h  | 3 +--
 arch/x86/kernel/cpu/sgx/sgx.h   | 2 +-
 arch/x86/kernel/cpu/sgx/encl.c  | 5 +++--
 arch/x86/kernel/cpu/sgx/ioctl.c | 2 +-
 4 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/sgx/encl.h +1 -2
index 7cdc351bc273..0c8571fc30cf 100644
@@ -112,8 +112,7 @@ int sgx_encl_get_backing(struct sgx_encl *encl, unsigned long page_index,
 void sgx_encl_put_backing(struct sgx_backing *backing, bool do_write);
 int sgx_encl_test_and_clear_young(struct mm_struct *mm,
 				  struct sgx_encl_page *page);
-
-struct sgx_epc_page *sgx_alloc_va_page(void);
+struct sgx_epc_page *sgx_alloc_va_page(struct sgx_va_page *va_page);
 unsigned int sgx_alloc_va_slot(struct sgx_va_page *va_page);
 void sgx_free_va_slot(struct sgx_va_page *va_page, unsigned int offset);
 bool sgx_va_page_full(struct sgx_va_page *va_page);
arch/x86/kernel/cpu/sgx/sgx.h +1 -1
index d7a1490d90bb..f8ed9deac18b 100644
@@ -33,7 +33,7 @@ struct sgx_epc_page {
 	unsigned int section;
 	u16 flags;
 	u16 poison;
-	struct sgx_encl_page *owner;
+	void *owner;
 	struct list_head list;
 };
 
arch/x86/kernel/cpu/sgx/encl.c +3 -2
index 68c8d65a8dee..c0725111cc25 100644
@@ -753,6 +753,7 @@ int sgx_encl_test_and_clear_young(struct mm_struct *mm,
 
 /**
  * sgx_alloc_va_page() - Allocate a Version Array (VA) page
+ * @va_page:	struct sgx_va_page connected to this VA page
  *
  * Allocate a free EPC page and convert it to a Version Array (VA) page.
  *
@@ -760,12 +761,12 @@ int sgx_encl_test_and_clear_young(struct mm_struct *mm,
  *   a VA page,
  *   -errno otherwise
  */
-struct sgx_epc_page *sgx_alloc_va_page(void)
+struct sgx_epc_page *sgx_alloc_va_page(struct sgx_va_page *va_page)
 {
 	struct sgx_epc_page *epc_page;
 	int ret;
 
-	epc_page = sgx_alloc_epc_page(NULL, true);
+	epc_page = sgx_alloc_epc_page(va_page, true);
 	if (IS_ERR(epc_page))
 		return ERR_CAST(epc_page);
 
arch/x86/kernel/cpu/sgx/ioctl.c +1 -1
index 00668e50848d..f0ce96bd462a 100644
@@ -30,7 +30,7 @@ static struct sgx_va_page *sgx_encl_grow(struct sgx_encl *encl)
 		if (!va_page)
 			return ERR_PTR(-ENOMEM);
 
-		va_page->epc_page = sgx_alloc_va_page();
+		va_page->epc_page = sgx_alloc_va_page(va_page);
 		if (IS_ERR(va_page->epc_page)) {
 			err = ERR_CAST(va_page->epc_page);
 			kfree(va_page);
Signature · 1 line
2.17.1
maintainer
#5 lore raw

Re: [RFC PATCH v3 02/10] x86/sgx: Provide VA page non-NULL owner

Quoted 4 lines · Cathy Zhang

Provide non-NULL owner for VA page, which tells that the EPC page is allocated for use.

So here the code change does not look all that bad but the commit message completely explanation of it.

"Provided non-NULL owner" is not something that I can map to any reasonable action that , i.e. I don't know what the heck it even means :-) It's just jargon, isn't it?

Staart with

"Make epc->owner type-agnostic by changing it's type to void *. ..."

Then you should answer exactly two questions in the commit message:

* Who is the owner? For regular pages it's the logical enclave page.
* For exactly what use it is allocated for?

BR, Jarkko

#6 lore raw

RFC PATCH v3 3/10 x86/sgx: Save enclave pointer for VA page

In reply to Cathy Zhang

Tearing down all enclaves is required by SGX SVN update, which involves running the ENCLS[EREMOVE] instruction on every EPC page. This (tearing down all enclaves) should be coordinated with any enclaves that may be in the process of existing and thus already be running ENCLS[EREMOVE] as part of enclave release.

In support of this coordination, it is required to know which enclave owns each in-use EPC page. It is already possible to locate the owning enclave of SECS and regular pages but not for VA pages.

Save the enclave pointer for each VA page to support locating its owning enclave.

Note: to track 2T EPC memory, this scheme of tracking will use additional 8M memory.

---
 arch/x86/kernel/cpu/sgx/encl.h  | 1 +
 arch/x86/kernel/cpu/sgx/ioctl.c | 1 +
 2 files changed, 2 insertions(+)
arch/x86/kernel/cpu/sgx/encl.h +1 -0
index 0c8571fc30cf..59fbd4ed5c64 100644
@@ -76,6 +76,7 @@ struct sgx_va_page {
 	struct sgx_epc_page *epc_page;
 	DECLARE_BITMAP(slots, SGX_VA_SLOT_COUNT);
 	struct list_head list;
+	struct sgx_encl *encl;
 };
 
 struct sgx_backing {
arch/x86/kernel/cpu/sgx/ioctl.c +1 -0
index f0ce96bd462a..da82abbb81b4 100644
@@ -30,6 +30,7 @@ static struct sgx_va_page *sgx_encl_grow(struct sgx_encl *encl)
 		if (!va_page)
 			return ERR_PTR(-ENOMEM);
 
+		va_page->encl = encl;
 		va_page->epc_page = sgx_alloc_va_page(va_page);
 		if (IS_ERR(va_page->epc_page)) {
 			err = ERR_CAST(va_page->epc_page);
Signature · 1 line
2.17.1
maintainer
#7 lore raw

Re: [RFC PATCH v3 03/10] x86/sgx: Save enclave pointer for VA page

Quoted 49 lines · Cathy Zhang

Tearing down all enclaves is required by SGX SVN update, which involves running the ENCLS[EREMOVE] instruction on every EPC page. This (tearing down all enclaves) should be coordinated with any enclaves that may be in the process of existing and thus already be running ENCLS[EREMOVE] as part of enclave release.

In support of this coordination, it is required to know which enclave owns each in-use EPC page. It is already possible to locate the owning enclave of SECS and regular pages but not for VA pages.

Save the enclave pointer for each VA page to support locating its owning enclave.

Note: to track 2T EPC memory, this scheme of tracking will use additional 8M memory.

---
 arch/x86/kernel/cpu/sgx/encl.h  | 1 +
 arch/x86/kernel/cpu/sgx/ioctl.c | 1 +
 2 files changed, 2 insertions(+)
arch/x86/kernel/cpu/sgx/encl.h +1 -0
index 0c8571fc30cf..59fbd4ed5c64 100644
@@ -76,6 +76,7 @@ struct sgx_va_page {
 	struct sgx_epc_page *epc_page;
 	DECLARE_BITMAP(slots, SGX_VA_SLOT_COUNT);
 	struct list_head list;
+	struct sgx_encl *encl;
 };
 
 struct sgx_backing {
arch/x86/kernel/cpu/sgx/ioctl.c +1 -0
index f0ce96bd462a..da82abbb81b4 100644
@@ -30,6 +30,7 @@ static struct sgx_va_page *sgx_encl_grow(struct sgx_encl *encl)
 		if (!va_page)
 			return ERR_PTR(-ENOMEM);
 
+		va_page->encl = encl;
 		va_page->epc_page = sgx_alloc_va_page(va_page);
 		if (IS_ERR(va_page->epc_page)) {
 			err = ERR_CAST(va_page->epc_page);
Signature · 1 line
2.17.1

Squash this with the previous patch. Then the code will provide answer to one of the questions that I mentioned in 2/10 review.

BR, Jarkko

#8 lore raw

RE: [RFC PATCH v3 03/10] x86/sgx: Save enclave pointer for VA page

Quoted 68 lines

-----Original Message-----
From: Jarkko Sakkinen <[email protected]>
Sent: Sunday, April 3, 2022 6:08 PM
To: Zhang, Cathy <[email protected]>
Cc: [email protected]; [email protected]; Chatre, Reinette <[email protected]>; Hansen, Dave <[email protected]>; Raj, Ashok <[email protected]>
Subject: Re: [RFC PATCH v3 03/10] x86/sgx: Save enclave pointer for VA page

arch/x86/kernel/cpu/sgx/ioctl.c static struct sgx_va_page *sgx_encl_grow(struct

Quoted 41 lines · Cathy Zhang

Tearing down all enclaves is required by SGX SVN update, which involves running the ENCLS[EREMOVE] instruction on every EPC page. This (tearing down all enclaves) should be coordinated with any enclaves that may be in the process of existing and thus already be running ENCLS[EREMOVE] as part of enclave release.

In support of this coordination, it is required to know which enclave owns each in-use EPC page. It is already possible to locate the owning enclave of SECS and regular pages but not for VA pages.

Save the enclave pointer for each VA page to support locating its owning enclave.

Note: to track 2T EPC memory, this scheme of tracking will use additional 8M memory.

---
 arch/x86/kernel/cpu/sgx/encl.h  | 1 +
arch/x86/kernel/cpu/sgx/ioctl.c | 1 +
 2 files changed, 2 insertions(+)
(quoted diff) +0 -0

b/arch/x86/kernel/cpu/sgx/encl.h index 0c8571fc30cf..59fbd4ed5c64 100644

arch/x86/kernel/cpu/sgx/encl.h +1 -0
@@ -76,6 +76,7 @@ struct sgx_va_page {
 	struct sgx_epc_page *epc_page;
 	DECLARE_BITMAP(slots, SGX_VA_SLOT_COUNT);
 	struct list_head list;
+	struct sgx_encl *encl;
 };
 
 struct sgx_backing {
(quoted diff) +0 -0

b/arch/x86/kernel/cpu/sgx/ioctl.c index f0ce96bd462a..da82abbb81b4 100644

arch/x86/kernel/cpu/sgx/ioctl.c +0 -0
@@ -30,6 +30,7 @@ static struct sgx_va_page *sgx_encl_grow(struct

sgx_encl *encl)

Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c +1 -1
 		if (!va_page)
 			return ERR_PTR(-ENOMEM);
 
+		va_page->encl = encl;
 		va_page->epc_page = sgx_alloc_va_page(va_page);
 		if (IS_ERR(va_page->epc_page)) {
 			err = ERR_CAST(va_page->epc_page);
--

2.17.1

Squash this with the previous patch. Then the code will provide answer to one of the questions that I mentioned in 2/10 review.

Quoted 1 line

BR, Jarkko

Squashed. How about re-write the commit log as follows:

x86/sgx: Save enclave pointer for VA page

Tearing down all enclaves is required by SGX SVN update, which involves running the ENCLS[EREMOVE] instruction on every EPC page. This (tearing down all enclaves) should be coordinated with any enclaves that may be in the process of existing and thus already be running ENCLS[EREMOVE] as part of enclave release.

In support of this coordination, it is required to know which enclave owns each in-use EPC page. It is already possible to locate the owning enclave of SECS and regular pages but not for VA pages.

Make the following changes for VA page's location:
1) Make epc->owner type-agnostic by change its type to 'void *'. So, besides "struct sgx_encl_page", it can have other types, like "struct sgx_va_page".
2) Save the enclave pointer for each VA page to support locating its owning enclave.

Note: to track 2T EPC memory, this scheme of tracking will use additional 8M memory.

maintainer
#9 lore raw

Re: [RFC PATCH v3 03/10] x86/sgx: Save enclave pointer for VA page

Quoted 91 lines · Cathy
Quoted 65 lines

-----Original Message-----
From: Jarkko Sakkinen <[email protected]>
Sent: Sunday, April 3, 2022 6:08 PM
To: Zhang, Cathy <[email protected]>
Cc: [email protected]; [email protected]; Chatre, Reinette <[email protected]>; Hansen, Dave <[email protected]>; Raj, Ashok <[email protected]>
Subject: Re: [RFC PATCH v3 03/10] x86/sgx: Save enclave pointer for VA page

arch/x86/kernel/cpu/sgx/ioctl.c static struct sgx_va_page *sgx_encl_grow(struct

Quoted 41 lines · Cathy Zhang

Tearing down all enclaves is required by SGX SVN update, which involves running the ENCLS[EREMOVE] instruction on every EPC page. This (tearing down all enclaves) should be coordinated with any enclaves that may be in the process of existing and thus already be running ENCLS[EREMOVE] as part of enclave release.

In support of this coordination, it is required to know which enclave owns each in-use EPC page. It is already possible to locate the owning enclave of SECS and regular pages but not for VA pages.

Save the enclave pointer for each VA page to support locating its owning enclave.

Note: to track 2T EPC memory, this scheme of tracking will use additional 8M memory.

---
 arch/x86/kernel/cpu/sgx/encl.h  | 1 +
arch/x86/kernel/cpu/sgx/ioctl.c | 1 +

 2 files changed, 2 insertions(+)
(quoted diff) +0 -0

b/arch/x86/kernel/cpu/sgx/encl.h index 0c8571fc30cf..59fbd4ed5c64 100644

arch/x86/kernel/cpu/sgx/encl.h +1 -0
@@ -76,6 +76,7 @@ struct sgx_va_page {
         struct sgx_epc_page *epc_page;
         DECLARE_BITMAP(slots, SGX_VA_SLOT_COUNT);
         struct list_head list;
+       struct sgx_encl *encl;
  };
 
  struct sgx_backing {
(quoted diff) +0 -0

b/arch/x86/kernel/cpu/sgx/ioctl.c index f0ce96bd462a..da82abbb81b4 100644

arch/x86/kernel/cpu/sgx/ioctl.c +0 -0
@@ -30,6 +30,7 @@ static struct sgx_va_page *sgx_encl_grow(struct

sgx_encl *encl)

Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c +1 -1
                 if (!va_page)
                         return ERR_PTR(-ENOMEM);
 
+               va_page->encl = encl;
                 va_page->epc_page = sgx_alloc_va_page(va_page);
                 if (IS_ERR(va_page->epc_page)) {
                         err = ERR_CAST(va_page->epc_page);
--

2.17.1

Quoted 4 lines

Squash this with the previous patch. Then the code will provide answer to one of the questions that I mentioned in 2/10 review.

BR, Jarkko

Squashed. How about re-write the commit log as follows:

x86/sgx: Save enclave pointer for VA page

Tearing down all enclaves is required by SGX SVN update, which involves running the ENCLS[EREMOVE] instruction on every EPC page. This (tearing down all enclaves) should be coordinated with any enclaves that may be in the process of existing and thus already be running ENCLS[EREMOVE] as part of enclave release.

In support of this coordination, it is required to know which enclave owns each in-use EPC page. It is already possible to locate the owning enclave of SECS and regular pages but not for VA pages.

Make the following changes for VA page's location:
1) Make epc->owner type-agnostic by change its type to 'void *'. So, besides "struct sgx_encl_page", it can have other types, like "struct sgx_va_page".
2) Save the enclave pointer for each VA page to support locating its owning enclave.

Quoted 2 lines

Note: to track 2T EPC memory, this scheme of tracking will use additional 8M memory.

Sounds good to me.

BR, Jarkko

#10 lore raw

RFC PATCH v3 4/10 x86/sgx: Keep record for SGX VA and Guest page type

In reply to Cathy Zhang

Regular enclave EPC pages have sgx_encl_page as their owner, but SGX VA page and KVM guest EPC page are maintained by different owner structures.

SGX CPUSVN update requires to know the EPC page owner's status and then decide how to handle the page.

Keep a record of page type for SGX VA and KVM guest page while the other EPC pages already have their type tracked, so that CPUSVN update can get EPC page's owner by type and handle it then.

---
 arch/x86/kernel/cpu/sgx/sgx.h  | 4 ++++
 arch/x86/kernel/cpu/sgx/encl.c | 2 ++
 arch/x86/kernel/cpu/sgx/virt.c | 2 ++
 3 files changed, 8 insertions(+)
arch/x86/kernel/cpu/sgx/sgx.h +4 -0
index f8ed9deac18b..0b036f19cca1 100644
@@ -28,6 +28,10 @@
 
 /* Pages on free list */
 #define SGX_EPC_PAGE_IS_FREE		BIT(1)
+/* VA page */
+#define SGX_EPC_PAGE_VA			BIT(2)
+/* Pages allocated for KVM guest */
+#define SGX_EPC_PAGE_GUEST		BIT(3)
 
 struct sgx_epc_page {
 	unsigned int section;
arch/x86/kernel/cpu/sgx/encl.c +2 -0
index c0725111cc25..a01a72637e2e 100644
@@ -777,6 +777,8 @@ struct sgx_epc_page *sgx_alloc_va_page(struct sgx_va_page *va_page)
 		return ERR_PTR(-EFAULT);
 	}
 
+	epc_page->flags |= SGX_EPC_PAGE_VA;
+
 	return epc_page;
 }
 
arch/x86/kernel/cpu/sgx/virt.c +2 -0
index e953816d7c8b..acdf72769a39 100644
@@ -50,6 +50,8 @@ static int __sgx_vepc_fault(struct sgx_vepc *vepc,
 	if (IS_ERR(epc_page))
 		return PTR_ERR(epc_page);
 
+	epc_page->flags |= SGX_EPC_PAGE_GUEST;
+
 	ret = xa_err(xa_store(&vepc->page_array, index, epc_page, GFP_KERNEL));
 	if (ret)
 		goto err_free;
Signature · 1 line
2.17.1
maintainer
#11 lore raw

Re: [RFC PATCH v3 04/10] x86/sgx: Keep record for SGX VA and Guest page type

Quoted 23 lines · Cathy Zhang

Regular enclave EPC pages have sgx_encl_page as their owner, but SGX VA page and KVM guest EPC page are maintained by different owner structures.

SGX CPUSVN update requires to know the EPC page owner's status and then decide how to handle the page.

Keep a record of page type for SGX VA and KVM guest page while the other EPC pages already have their type tracked, so that CPUSVN update can get EPC page's owner by type and handle it then.

---
 arch/x86/kernel/cpu/sgx/sgx.h  | 4 ++++
 arch/x86/kernel/cpu/sgx/encl.c | 2 ++
 arch/x86/kernel/cpu/sgx/virt.c | 2 ++
 3 files changed, 8 insertions(+)
arch/x86/kernel/cpu/sgx/sgx.h +0 -0
index f8ed9deac18b..0b036f19cca1 100644

arch/x86/kernel/cpu/sgx/sgx.h line 34

Quoted 7 lines
arch/x86/kernel/cpu/sgx/sgx.h +4 -0
@@ -28,6 +28,10 @@
 
 /* Pages on free list */
 #define SGX_EPC_PAGE_IS_FREE		BIT(1)
+/* VA page */
+#define SGX_EPC_PAGE_VA			BIT(2)
+/* Pages allocated for KVM guest */
+#define SGX_EPC_PAGE_GUEST		BIT(3)

Please rename this as either SGX_EPC_PAGE_KVM_GUEST or SGX_EPC_PAGE_KVM. Then it is hard to not remember what it is.

BR, Jarkko

#12 lore raw

RE: [RFC PATCH v3 04/10] x86/sgx: Keep record for SGX VA and Guest page type

Quoted 38 lines

-----Original Message-----
From: Jarkko Sakkinen <[email protected]>
Sent: Sunday, April 3, 2022 6:11 PM
To: Zhang, Cathy <[email protected]>
Cc: [email protected]; [email protected]; Chatre, Reinette <[email protected]>; Hansen, Dave <[email protected]>; Raj, Ashok <[email protected]>
Subject: Re: [RFC PATCH v3 04/10] x86/sgx: Keep record for SGX VA and Guest page type

Quoted 25 lines · Cathy Zhang

Regular enclave EPC pages have sgx_encl_page as their owner, but SGX VA page and KVM guest EPC page are maintained by different owner structures.

SGX CPUSVN update requires to know the EPC page owner's status and then decide how to handle the page.

Keep a record of page type for SGX VA and KVM guest page while the other EPC pages already have their type tracked, so that CPUSVN update can get EPC page's owner by type and handle it then.

---
 arch/x86/kernel/cpu/sgx/sgx.h  | 4 ++++
arch/x86/kernel/cpu/sgx/encl.c | 2 ++  arch/x86/kernel/cpu/sgx/virt.c
| 2 ++
 3 files changed, 8 insertions(+)
(quoted diff) +0 -0

b/arch/x86/kernel/cpu/sgx/sgx.h index f8ed9deac18b..0b036f19cca1 100644

arch/x86/kernel/cpu/sgx/sgx.h +0 -0
@@ -28,6 +28,10 @@
 
Quoted 10 lines

arch/x86/kernel/cpu/sgx/sgx.h line 34

Quoted 6 lines
arch/x86/kernel/cpu/sgx/sgx.h +4 -0
 /* Pages on free list */
 #define SGX_EPC_PAGE_IS_FREE		BIT(1)
+/* VA page */
+#define SGX_EPC_PAGE_VA			BIT(2)
+/* Pages allocated for KVM guest */
+#define SGX_EPC_PAGE_GUEST		BIT(3)

Please rename this as either SGX_EPC_PAGE_KVM_GUEST or SGX_EPC_PAGE_KVM.
Then it is hard to not remember what it is.

Renamed as SGX_EPC_PAGE_KVM_GUEST.

Quoted 2 lines

BR, Jarkko

#13 lore raw

RFC PATCH v3 5/10 x86/sgx: Save the size of each EPC section

In reply to Cathy Zhang

For SGX CPUSVN update process will check all EPC pages in each section to ensure they will be marked as unused, it requires to know the size of each EPC section to end the loop.

---
 arch/x86/kernel/cpu/sgx/sgx.h  | 1 +
 arch/x86/kernel/cpu/sgx/main.c | 1 +
 2 files changed, 2 insertions(+)
arch/x86/kernel/cpu/sgx/sgx.h +1 -0
index 0b036f19cca1..45ba6fcabfda 100644
@@ -63,6 +63,7 @@ struct sgx_epc_section {
 	void *virt_addr;
 	struct sgx_epc_page *pages;
 	struct sgx_numa_node *node;
+	u64 size;
 };
 
 extern struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS];
arch/x86/kernel/cpu/sgx/main.c +1 -0
index 10360f06c0df..031c1402cd7e 100644
@@ -665,6 +665,7 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size,
 	}
 
 	section->phys_addr = phys_addr;
+	section->size = size;
 	xa_store_range(&sgx_epc_address_space, section->phys_addr,
 		       phys_addr + size - 1, section, GFP_KERNEL);
 
Signature · 1 line
2.17.1
maintainer
#14 lore raw

Re: [RFC PATCH v3 05/10] x86/sgx: Save the size of each EPC section

Quoted 3 lines · Cathy Zhang

For SGX CPUSVN update process will check all EPC pages in each section to ensure they will be marked as unused, it requires to know the size of each EPC section to end the loop.

Why is the size required to end the loop? It's missing.

Quoted 33 lines
---
 arch/x86/kernel/cpu/sgx/sgx.h  | 1 +
 arch/x86/kernel/cpu/sgx/main.c | 1 +
 2 files changed, 2 insertions(+)
arch/x86/kernel/cpu/sgx/sgx.h +1 -0
index 0b036f19cca1..45ba6fcabfda 100644
@@ -63,6 +63,7 @@ struct sgx_epc_section {
 	void *virt_addr;
 	struct sgx_epc_page *pages;
 	struct sgx_numa_node *node;
+	u64 size;
 };
 
 extern struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS];
arch/x86/kernel/cpu/sgx/main.c +1 -0
index 10360f06c0df..031c1402cd7e 100644
@@ -665,6 +665,7 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size,
 	}
 
 	section->phys_addr = phys_addr;
+	section->size = size;
 	xa_store_range(&sgx_epc_address_space, section->phys_addr,
 		       phys_addr + size - 1, section, GFP_KERNEL);
 
Signature · 1 line
2.17.1

BR, Jarkko

#15 lore raw

RE: [RFC PATCH v3 05/10] x86/sgx: Save the size of each EPC section

Quoted 11 lines

-----Original Message-----
From: Jarkko Sakkinen <[email protected]>
Sent: Sunday, April 3, 2022 6:15 PM
To: Zhang, Cathy <[email protected]>
Cc: [email protected]; [email protected]; Chatre, Reinette <[email protected]>; Hansen, Dave <[email protected]>; Raj, Ashok <[email protected]>
Subject: Re: [RFC PATCH v3 05/10] x86/sgx: Save the size of each EPC section

Quoted 6 lines
Quoted 3 lines · Cathy Zhang

For SGX CPUSVN update process will check all EPC pages in each section to ensure they will be marked as unused, it requires to know the size of each EPC section to end the loop.

Why is the size required to end the loop? It's missing.

How about re-write as follows:

SGX CPUSVN update process should check all EPC pages to ensure they will be marked as unused. For EPC pages are stored in EPC sections, it's required to save the size of each section, as the indicator for the end of each section's traversing.

Quoted 39 lines

arch/x86/kernel/cpu/sgx/main.c static bool __init sgx_setup_epc_section(u64

Quoted 26 lines
---
 arch/x86/kernel/cpu/sgx/sgx.h  | 1 +
 arch/x86/kernel/cpu/sgx/main.c | 1 +
 2 files changed, 2 insertions(+)
(quoted diff) +0 -0

b/arch/x86/kernel/cpu/sgx/sgx.h index 0b036f19cca1..45ba6fcabfda 100644

arch/x86/kernel/cpu/sgx/sgx.h +1 -0
@@ -63,6 +63,7 @@ struct sgx_epc_section {
 	void *virt_addr;
 	struct sgx_epc_page *pages;
 	struct sgx_numa_node *node;
+	u64 size;
 };
 
 extern struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS];
(quoted diff) +0 -0

b/arch/x86/kernel/cpu/sgx/main.c index 10360f06c0df..031c1402cd7e 100644

arch/x86/kernel/cpu/sgx/main.c +0 -0
@@ -665,6 +665,7 @@ static bool __init sgx_setup_epc_section(u64

phys_addr, u64 size,

Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +1 -1
 	}
 
 	section->phys_addr = phys_addr;
+	section->size = size;
 	xa_store_range(&sgx_epc_address_space, section->phys_addr,
 		       phys_addr + size - 1, section, GFP_KERNEL);
 
--

2.17.1

BR, Jarkko

maintainer
#16 lore raw

Re: [RFC PATCH v3 05/10] x86/sgx: Save the size of each EPC section

Quoted 16 lines · Cathy
Quoted 16 lines

-----Original Message-----
From: Jarkko Sakkinen <[email protected]>
Sent: Sunday, April 3, 2022 6:15 PM
To: Zhang, Cathy <[email protected]>
Cc: [email protected]; [email protected]; Chatre, Reinette <[email protected]>; Hansen, Dave <[email protected]>; Raj, Ashok <[email protected]>
Subject: Re: [RFC PATCH v3 05/10] x86/sgx: Save the size of each EPC section

Quoted 3 lines · Cathy Zhang

For SGX CPUSVN update process will check all EPC pages in each section to ensure they will be marked as unused, it requires to know the size of each EPC section to end the loop.

Quoted 8 lines
Quoted 1 line

Why is the size required to end the loop? It's missing.

How about re-write as follows:

SGX CPUSVN update process should check all EPC pages to ensure they will be marked as unused. For EPC pages are stored in EPC sections, it's required to save the size of each section, as the indicator for the end of each section's traversing.

Sounds good to me.

BR, Jarkko

#17 lore raw

RFC PATCH v3 6/10 x86/sgx: Forced EPC page zapping for EUPDATESVN

In reply to Cathy Zhang

Before an EUPDATESVN instruction can be successful, all enclave pages (EPC) must be marked as unused in the SGX hardware metadata (EPCM).

A page becomes unused when an issued EREMOVE instruction succeeds. To prepare for EUPDATESVN, loop over all SGX pages and attempt to EREMOVE them. This is fatal to running enclaves and destroys all enclave state and memory contents. This destruction is by design and mitigates any compromise of enclaves or the SGX hardware itself which occurred before the microcode update.

An EREMOVE operation on a page may fail for a few reasons. Each has its own mitigations.

First, EREMOVE will fail if an enclave that uses the page is executing. Send an IPI to all CPUs that might be running the enclave to force it out of the enclave long enough to EREMOVE the page. Other CPUs might enter the enclave in the meantime, so this is not a rock-solid guarantee.

Second, EREMOVE can fail on special SGX metadata pages, such as SECS and VA. EREMOVE will work on them only after the normal SGX pages that depend on them have been EREMOVE'd. Defer handling those pages and repeat EREMOVE after the dependency has been addressed.

---
 arch/x86/kernel/cpu/sgx/sgx.h  |  13 ++
 arch/x86/kernel/cpu/sgx/encl.c |  14 +-
 arch/x86/kernel/cpu/sgx/main.c | 347 +++++++++++++++++++++++++++++++++
 3 files changed, 373 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/sgx.h +13 -0
index 45ba6fcabfda..33d66473986f 100644
@@ -32,6 +32,17 @@
 #define SGX_EPC_PAGE_VA			BIT(2)
 /* Pages allocated for KVM guest */
 #define SGX_EPC_PAGE_GUEST		BIT(3)
+/*
+ * Pages, failed to be zapped (EREMOVED)
+ * by SGX CPUSVN update process.
+ */
+#define SGX_EPC_PAGE_ZAP_TRACKED	BIT(4)
+/*
+ * Pages, the associated enclave is being
+ * released while SGX CPUSVN update is
+ * running.
+ */
+#define SGX_EPC_PAGE_IN_RELEASE		BIT(5)
 
 struct sgx_epc_page {
 	unsigned int section;
@@ -110,5 +121,7 @@ void sgx_update_lepubkeyhash(u64 *lepubkeyhash);
 
 extern struct srcu_struct sgx_lock_epc_srcu;
 bool sgx_epc_is_locked(void);
+void sgx_zap_wakeup(void);
+void sgx_zap_abort(void);
 
 #endif /* _X86_SGX_H */
arch/x86/kernel/cpu/sgx/encl.c +13 -1
index a01a72637e2e..be177a5e3292 100644
@@ -457,6 +457,12 @@ void sgx_encl_release(struct kref *ref)
 	WARN_ON_ONCE(encl->secs_child_cnt);
 	WARN_ON_ONCE(encl->secs.epc_page);
 
+	/*
+	 * EPC pages were freed and EREMOVE was executed. Wake
+	 * up any zappers which were waiting for this.
+	 */
+	sgx_zap_wakeup();
+
 	kfree(encl);
 }
 
@@ -840,8 +846,14 @@ void sgx_encl_free_epc_page(struct sgx_epc_page *page)
 	WARN_ON_ONCE(page->flags & SGX_EPC_PAGE_RECLAIMER_TRACKED);
 
 	ret = __eremove(sgx_get_epc_virt_addr(page));
-	if (WARN_ONCE(ret, EREMOVE_ERROR_MESSAGE, ret, ret))
+	if (WARN_ONCE(ret, EREMOVE_ERROR_MESSAGE, ret, ret)) {
+		/*
+		 * The EREMOVE failed. If a CPUSVN is in progress,
+		 * it is now expected to fail. Notify it.
+		 */
+		sgx_zap_abort();
 		return;
+	}
 
 	sgx_free_epc_page(page);
 }
arch/x86/kernel/cpu/sgx/main.c +347 -0

362 lines not on this page · raw message

Signature · 1 line
2.17.1
maintainer
#18 lore raw

Re: [RFC PATCH v3 06/10] x86/sgx: Forced EPC page zapping for EUPDATESVN

Quoted 14 lines · Cathy Zhang

Before an EUPDATESVN instruction can be successful, all enclave pages (EPC) must be marked as unused in the SGX hardware metadata (EPCM).

A page becomes unused when an issued EREMOVE instruction succeeds. To prepare for EUPDATESVN, loop over all SGX pages and attempt to EREMOVE them. This is fatal to running enclaves and destroys all enclave state and memory contents. This destruction is by design and mitigates any compromise of enclaves or the SGX hardware itself which occurred before the microcode update.

An EREMOVE operation on a page may fail for a few reasons. Each has its own mitigations.

Quoted 8 lines

First, EREMOVE will fail if an enclave that uses the page is executing. Send an IPI to all CPUs that might be running the enclave to force it out of the enclave long enough to EREMOVE the page. Other CPUs might enter the enclave in the meantime, so this is not a rock-solid guarantee.

Second, EREMOVE can fail on special SGX metadata pages, such as SECS and VA. EREMOVE will work on them only after the normal SGX

Ignoring concurrency rules that apply to any type of EPC page, VA page can be removed at any point of time, i.e. the first sentence in this paragraph is not factually true.

BR, Jarkko

#19 lore raw

RE: [RFC PATCH v3 06/10] x86/sgx: Forced EPC page zapping for EUPDATESVN

Quoted 32 lines

-----Original Message-----
From: Jarkko Sakkinen <[email protected]>
Sent: Sunday, April 3, 2022 6:21 PM
To: Zhang, Cathy <[email protected]>
Cc: [email protected]; [email protected]; Chatre, Reinette <[email protected]>; Hansen, Dave <[email protected]>; Raj, Ashok <[email protected]>
Subject: Re: [RFC PATCH v3 06/10] x86/sgx: Forced EPC page zapping for EUPDATESVN

Quoted 19 lines · Cathy Zhang

Before an EUPDATESVN instruction can be successful, all enclave pages (EPC) must be marked as unused in the SGX hardware metadata (EPCM).

A page becomes unused when an issued EREMOVE instruction succeeds. To prepare for EUPDATESVN, loop over all SGX pages and attempt to EREMOVE them. This is fatal to running enclaves and destroys all enclave state and memory contents. This destruction is by design and mitigates any compromise of enclaves or the SGX hardware itself which occurred before the microcode update.

An EREMOVE operation on a page may fail for a few reasons. Each has its own mitigations.

First, EREMOVE will fail if an enclave that uses the page is executing. Send an IPI to all CPUs that might be running the enclave to force it out of the enclave long enough to EREMOVE the page. Other CPUs might enter the enclave in the meantime, so this is not a rock-solid guarantee.

Quoted 6 lines
Quoted 2 lines

Second, EREMOVE can fail on special SGX metadata pages, such as SECS and VA. EREMOVE will work on them only after the normal SGX

Ignoring concurrency rules that apply to any type of EPC page, VA page can be removed at any point of time, i.e. the first sentence in this paragraph is not factually true.

Yes, the SGX metadata pages failed to be EREMOVEd and be tracked for a later retry is SECS pages.

Removed "VA" from the sentence Jarkko mentioned, Thanks for pointing out!

Quoted 2 lines

BR, Jarkko

#20 lore raw

RFC PATCH v3 7/10 x86/sgx: Define error codes for ENCLS[EUPDATESVN]

In reply to Cathy Zhang

Add error codes for ENCLS[EUPDATESVN], then SGX CPUSVN update process can know the execution state of EUPDATESVN and notify userspace.

---
 arch/x86/include/asm/sgx.h | 10 ++++++++++
 1 file changed, 10 insertions(+)
arch/x86/include/asm/sgx.h +10 -0
index 03ec7e32659c..4a8ca7281229 100644
@@ -65,17 +65,27 @@ enum sgx_encls_function {
 
 /**
  * enum sgx_return_code - The return code type for ENCLS, ENCLU and ENCLV
+ * %SGX_EPC_PAGE_CONFLICT	Page is being written by other ENCLS function.
  * %SGX_NOT_TRACKED:		Previous ETRACK's shootdown sequence has not
  *				been completed yet.
  * %SGX_CHILD_PRESENT		SECS has child pages present in the EPC.
  * %SGX_INVALID_EINITTOKEN:	EINITTOKEN is invalid and enclave signer's
  *				public key does not match IA32_SGXLEPUBKEYHASH.
+ * %SGX_INSUFFICIENT_ENTROPY:	Insufficient entropy in RNG.
+ * %SGX_EPC_NOT_READY:		EPC is not ready for SVN update.
+ * %SGX_NO_UPDATE:		EUPDATESVN was successful, but CPUSVN was not
+ *				updated because current SVN was not newer than
+ *				CPUSVN.
  * %SGX_UNMASKED_EVENT:		An unmasked event, e.g. INTR, was received
  */
 enum sgx_return_code {
+	SGX_EPC_PAGE_CONFLICT		= 7,
 	SGX_NOT_TRACKED			= 11,
 	SGX_CHILD_PRESENT		= 13,
 	SGX_INVALID_EINITTOKEN		= 16,
+	SGX_INSUFFICIENT_ENTROPY	= 29,
+	SGX_EPC_NOT_READY		= 30,
+	SGX_NO_UPDATE			= 31,
 	SGX_UNMASKED_EVENT		= 128,
 };
 
Signature · 1 line
2.17.1
#21 lore raw

RFC PATCH v3 8/10 x86/sgx: Implement ENCLS[EUPDATESVN]

In reply to Cathy Zhang

The SGX attestation architecture assumes a compromise of all running enclaves and cryptographic assets (like internal SGX encryption keys) whenever a microcode update affects SGX. To mitigate the impact of this presumed compromise, a new supervisor SGX instruction:
ENCLS[EUPDATESVN], is introduced to update SGX microcode version and generate new cryptographic assets in runtime after SGX microcode update.

EUPDATESVN requires that SGX memory to be marked as "unused" before it will succeed. This ensures that no compromised enclave can survive the process and provides an opportunity to generate new cryptographic assets.

---
Changes since v1:
- Print message for each return code to notify userspace the ENCLS[EUPDATESVN] execution status.

---
 arch/x86/include/asm/sgx.h      | 33 +++++++++++----------
 arch/x86/kernel/cpu/sgx/encls.h |  6 ++++
 arch/x86/kernel/cpu/sgx/main.c  | 52 +++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+), 15 deletions(-)
arch/x86/include/asm/sgx.h +18 -15
index 4a8ca7281229..74bcb6841a4b 100644
@@ -26,23 +26,26 @@
 #define SGX_CPUID_EPC_SECTION	0x1
 /* The bitmask for the EPC section type. */
 #define SGX_CPUID_EPC_MASK	GENMASK(3, 0)
+/* EUPDATESVN presence indication */
+#define SGX_CPUID_EUPDATESVN	BIT(10)
 
 enum sgx_encls_function {
-	ECREATE	= 0x00,
-	EADD	= 0x01,
-	EINIT	= 0x02,
-	EREMOVE	= 0x03,
-	EDGBRD	= 0x04,
-	EDGBWR	= 0x05,
-	EEXTEND	= 0x06,
-	ELDU	= 0x08,
-	EBLOCK	= 0x09,
-	EPA	= 0x0A,
-	EWB	= 0x0B,
-	ETRACK	= 0x0C,
-	EAUG	= 0x0D,
-	EMODPR	= 0x0E,
-	EMODT	= 0x0F,
+	ECREATE		= 0x00,
+	EADD		= 0x01,
+	EINIT		= 0x02,
+	EREMOVE		= 0x03,
+	EDGBRD		= 0x04,
+	EDGBWR		= 0x05,
+	EEXTEND		= 0x06,
+	ELDU		= 0x08,
+	EBLOCK		= 0x09,
+	EPA		= 0x0A,
+	EWB		= 0x0B,
+	ETRACK		= 0x0C,
+	EAUG		= 0x0D,
+	EMODPR		= 0x0E,
+	EMODT		= 0x0F,
+	EUPDATESVN	= 0x18,
 };
 
 /**
arch/x86/kernel/cpu/sgx/encls.h +6 -0
index 60321c5f5718..8455f385e817 100644
@@ -208,4 +208,10 @@ static inline int __ewb(struct sgx_pageinfo *pginfo, void *addr,
 	return __encls_ret_3(EWB, pginfo, addr, va);
 }
 
+/* Update CPUSVN at runtime. */
+static inline int __eupdatesvn(void)
+{
+	return __encls_ret_1(EUPDATESVN, "");
+}
+
 #endif /* _X86_ENCLS_H */
arch/x86/kernel/cpu/sgx/main.c +52 -0

55 lines not on this page · raw message

Signature · 1 line
2.17.1
#22 lore raw

RFC PATCH v3 9/10 x86/cpu: Call ENCLS[EUPDATESVN] procedure in microcode update

In reply to Cathy Zhang

EUPDATESVN is the SGX instruction which allows enclave attestation to include information about updated microcode without a reboot.

Microcode updates which affect SGX require two phases:

1. Do the main microcode update
2. Make the new CPUSVN available for enclave attestation via EUPDATESVN.

Before a EUPDATESVN can succeed, all enclave pages (EPC) must be marked as unused in the SGX metadata (EPCM). This operation destroys all preexisting SGX enclave data and metadata. This is by design and mitigates the impact of vulnerabilities that may have compromised enclaves or the SGX hardware itself prior to the update.

---
Changes since v1:
- Remove the sysfs file svnupdate. (Thomas Gleixner, Dave Hansen)
- Let late microcode load path call ENCLS[EUPDATESVN] procedure directly. (Borislav Petkov)
- Redefine update_cpusvn_intel() to return void instead of int.

---
 arch/x86/include/asm/microcode.h |  5 +++++
 arch/x86/include/asm/sgx.h       |  5 +++++
 arch/x86/kernel/cpu/common.c     |  9 +++++++++
 arch/x86/kernel/cpu/sgx/main.c   | 12 ++++++++++++
 4 files changed, 31 insertions(+)
arch/x86/include/asm/microcode.h +5 -0
index d6bfdfb0f0af..1ba66b9fe198 100644
@@ -3,6 +3,7 @@
 #define _ASM_X86_MICROCODE_H
 
 #include <asm/cpu.h>
+#include <asm/sgx.h>
 #include <linux/earlycpio.h>
 #include <linux/initrd.h>
 
@@ -137,4 +138,8 @@ static inline void load_ucode_ap(void)				{ }
 static inline void reload_early_microcode(void)			{ }
 #endif
 
+#ifndef update_cpusvn_intel
+static inline void update_cpusvn_intel(void) {}
+#endif
+
 #endif /* _ASM_X86_MICROCODE_H */
arch/x86/include/asm/sgx.h +5 -0
index 74bcb6841a4b..5867f5a78d13 100644
@@ -409,4 +409,9 @@ int sgx_virt_einit(void __user *sigstruct, void __user *token,
 int sgx_set_attribute(unsigned long *allowed_attributes,
 		      unsigned int attribute_fd);
 
+#ifdef CONFIG_X86_SGX
+void update_cpusvn_intel(void);
+#define update_cpusvn_intel update_cpusvn_intel
+#endif
+
 #endif /* _ASM_X86_SGX_H */
arch/x86/kernel/cpu/common.c +9 -0
index 7b8382c11788..f53fd877ba0d 100644
@@ -59,6 +59,7 @@
 #include <asm/cpu_device_id.h>
 #include <asm/uv/uv.h>
 #include <asm/sigframe.h>
+#include <asm/sgx.h>
 
 #include "cpu.h"
 
@@ -2086,6 +2087,14 @@ void microcode_check(void)
 
 	perf_check_microcode();
 
+	/*
+	 * SGX related microcode update requires EUPDATESVN to update CPUSVN, which
+	 * will destroy all enclaves to ensure EPC is not in use. If SGX is configured
+	 * and EUPDATESVN is supported, call the EUPDATESVN procecure.
+	 */
+	if (IS_ENABLED(CONFIG_X86_SGX) && (cpuid_eax(SGX_CPUID) & SGX_CPUID_EUPDATESVN))
+		update_cpusvn_intel();
+
 	/* Reload CPUID max function as it might've changed. */
 	info.cpuid_level = cpuid_eax(0);
 
arch/x86/kernel/cpu/sgx/main.c +12 -0
index 784780314762..20800c543110 100644
@@ -1409,3 +1409,15 @@ static int sgx_updatesvn(void)
 
 	return ret;
 }
+
+void update_cpusvn_intel(void)
+{
+	sgx_lock_epc();
+	if (sgx_zap_pages())
+		goto out;
+
+	sgx_updatesvn();
+
+out:
+	sgx_unlock_epc();
+}
Signature · 1 line
2.17.1
maintainer
#23 lore raw

Re: [RFC PATCH v3 09/10] x86/cpu: Call ENCLS[EUPDATESVN] procedure in microcode update

arch/x86/kernel/cpu/common.c void microcode_check(void) · line 2095

Quoted 10 lines · Cathy Zhang
arch/x86/kernel/cpu/common.c +6 -0
@@ -2086,6 +2087,14 @@ void microcode_check(void)
 
 	perf_check_microcode();
 
+	/*
+	 * SGX related microcode update requires EUPDATESVN to update CPUSVN, which
+	 * will destroy all enclaves to ensure EPC is not in use. If SGX is configured
+	 * and EUPDATESVN is supported, call the EUPDATESVN procecure.
+	 */
+	if (IS_ENABLED(CONFIG_X86_SGX) && (cpuid_eax(SGX_CPUID) & SGX_CPUID_EUPDATESVN))

Stick all that above...

arch/x86/kernel/cpu/common.c void microcode_check(void) · line 2096

Quoted 1 line
arch/x86/kernel/cpu/common.c +1 -0
+		update_cpusvn_intel();

...inside this function so that you have only the call here. Inside the function you can do all the ifdeffery, checking and commenting and so on. Just like perf_check_microcode() does.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#24 lore raw

RE: [RFC PATCH v3 09/10] x86/cpu: Call ENCLS[EUPDATESVN] procedure in microcode update

Hi Boris,

Thanks for helping review!

Quoted 18 lines

-----Original Message-----
From: Borislav Petkov <[email protected]>
Sent: Friday, April 1, 2022 10:30 PM
To: Zhang, Cathy <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; Chatre, Reinette <[email protected]>; Hansen, Dave <[email protected]>; Raj, Ashok <[email protected]>
Subject: Re: [RFC PATCH v3 09/10] x86/cpu: Call ENCLS[EUPDATESVN] procedure in microcode update

arch/x86/kernel/cpu/common.c void microcode_check(void) · line 2091

Quoted 6 lines · Cathy Zhang
arch/x86/kernel/cpu/common.c +2 -0
@@ -2086,6 +2087,14 @@ void microcode_check(void)
 
 	perf_check_microcode();
 
+	/*
+	 * SGX related microcode update requires EUPDATESVN to update

CPUSVN, which

Quoted 8 lines

Review void microcode_check(void) · line 2092

Quoted 1 line
(quoted diff) +1 -0
+	 * will destroy all enclaves to ensure EPC is not in use. If SGX is

configured

arch/x86/kernel/cpu/common.c void microcode_check(void)

Quoted 4 lines
arch/x86/kernel/cpu/common.c +4 -0
+	 * and EUPDATESVN is supported, call the EUPDATESVN procecure.
+	 */
+	if (IS_ENABLED(CONFIG_X86_SGX) && (cpuid_eax(SGX_CPUID) &
+SGX_CPUID_EUPDATESVN))

Stick all that above...

The comment is re-written as follows:

/*
* SGX attestation incorporates the microcode versions of all processors
* on the system and is affected by microcode updates. So, update SGX
* attestation metric (called CPUSVN) to ensure enclaves attest to the
* new version after microcode update.
*/

Quoted 7 lines

arch/x86/kernel/cpu/common.c void microcode_check(void) · line 2096

Quoted 2 lines
arch/x86/kernel/cpu/common.c +1 -0
+		update_cpusvn_intel();

...inside this function so that you have only the call here. Inside the function you can do all the ifdeffery, checking and commenting and so on. Just like perf_check_microcode() does.

Thanks for showing me the example! I've referred.

Quoted 7 lines

Thx.

--
Regards/Gruss,
Boris.

https://people.kernel.org/tglx/notes-about-netiquette

#25 lore raw

Re: [RFC PATCH v3 09/10] x86/cpu: Call ENCLS[EUPDATESVN] procedure in microcode update

In reply to Cathy Zhang
Quoted 32 lines · Cathy Zhang
arch/x86/include/asm/microcode.h +4 -0
+#ifndef update_cpusvn_intel
+static inline void update_cpusvn_intel(void) {}
+#endif
+
 #endif /* _ASM_X86_MICROCODE_H */
arch/x86/include/asm/sgx.h +5 -0
index 74bcb6841a4b..5867f5a78d13 100644
@@ -409,4 +409,9 @@ int sgx_virt_einit(void __user *sigstruct, void __user *token,
 int sgx_set_attribute(unsigned long *allowed_attributes,
 		      unsigned int attribute_fd);
 
+#ifdef CONFIG_X86_SGX
+void update_cpusvn_intel(void);
+#define update_cpusvn_intel update_cpusvn_intel
+#endif
+
 #endif /* _ASM_X86_SGX_H */
arch/x86/kernel/cpu/common.c +1 -0
index 7b8382c11788..f53fd877ba0d 100644
@@ -59,6 +59,7 @@
 #include <asm/cpu_device_id.h>
 #include <asm/uv/uv.h>
 #include <asm/sigframe.h>
+#include <asm/sgx.h>
 
 #include "cpu.h"
 

arch/x86/kernel/cpu/common.c void microcode_check(void) · line 2096

Quoted 10 lines
arch/x86/kernel/cpu/common.c +7 -0
@@ -2086,6 +2087,14 @@ void microcode_check(void)
 
 	perf_check_microcode();
 
+	/*
+	 * SGX related microcode update requires EUPDATESVN to update CPUSVN, which
+	 * will destroy all enclaves to ensure EPC is not in use. If SGX is configured
+	 * and EUPDATESVN is supported, call the EUPDATESVN procecure.
+	 */
+	if (IS_ENABLED(CONFIG_X86_SGX) && (cpuid_eax(SGX_CPUID) & SGX_CPUID_EUPDATESVN))
+		update_cpusvn_intel();

In addition to what Borislav said, these #ifdefs are not how we do things.

The update_cpusvn_intel() shouldn't be declared in two different headers. Just imagine what happens if some code happens to include the microcode header *then* the sgx.h header.

Please define both the 'static inline' stub *and* the declaration in sgx.h. Then you won't even need the #ifndef trickery. Please also add an "sgx_" to its name prefix while you're at it.

That comment is also not great. A reader will likely have *ZERO* idea what EUPDATESVN is or what a CPUSVN is. Try to focus on what the code *means* rather than just repeat the if() conditions in the comment.

#26 lore raw

RE: [RFC PATCH v3 09/10] x86/cpu: Call ENCLS[EUPDATESVN] procedure in microcode update

Hi Dave,

Thanks for your comments and suggestions!

Quoted 58 lines

-----Original Message-----
From: Hansen, Dave <[email protected]>
Sent: Friday, April 1, 2022 11:42 PM
To: Zhang, Cathy <[email protected]>; [email protected]; [email protected]
Cc: [email protected]; Chatre, Reinette <[email protected]>; Raj, Ashok <[email protected]>
Subject: Re: [RFC PATCH v3 09/10] x86/cpu: Call ENCLS[EUPDATESVN] procedure in microcode update

Quoted 18 lines · Cathy Zhang
arch/x86/include/asm/microcode.h +3 -0
+#ifndef update_cpusvn_intel
+static inline void update_cpusvn_intel(void) {} #endif
+
 #endif /* _ASM_X86_MICROCODE_H */
arch/x86/include/asm/sgx.h +4 -0
index 74bcb6841a4b..5867f5a78d13 100644
@@ -409,4 +409,9 @@ int sgx_virt_einit(void __user *sigstruct, void
 __user *token,  int sgx_set_attribute(unsigned long *allowed_attributes,
 		      unsigned int attribute_fd);
 
+#ifdef CONFIG_X86_SGX
+void update_cpusvn_intel(void);
+#define update_cpusvn_intel update_cpusvn_intel #endif
+
 #endif /* _ASM_X86_SGX_H */
(quoted diff) +0 -0
Quoted 1 line

b/arch/x86/kernel/cpu/common.c index 7b8382c11788..f53fd877ba0d

100644

Quoted 11 lines
arch/x86/kernel/cpu/common.c +1 -0
@@ -59,6 +59,7 @@
 #include <asm/cpu_device_id.h>
 #include <asm/uv/uv.h>
 #include <asm/sigframe.h>
+#include <asm/sgx.h>
 
 #include "cpu.h"
 

arch/x86/kernel/cpu/common.c void microcode_check(void) · line 2091

Quoted 5 lines
arch/x86/kernel/cpu/common.c +2 -0
@@ -2086,6 +2087,14 @@ void microcode_check(void)
 
 	perf_check_microcode();
 
+	/*
+	 * SGX related microcode update requires EUPDATESVN to update

CPUSVN, which

arch/x86/kernel/cpu/common.c void microcode_check(void) · line 2092

Quoted 1 line
arch/x86/kernel/cpu/common.c +1 -0
+	 * will destroy all enclaves to ensure EPC is not in use. If SGX is

configured

arch/x86/kernel/cpu/common.c void microcode_check(void)

Quoted 3 lines
arch/x86/kernel/cpu/common.c +3 -0
+	 * and EUPDATESVN is supported, call the EUPDATESVN procecure.
+	 */
+	if (IS_ENABLED(CONFIG_X86_SGX) && (cpuid_eax(SGX_CPUID) &

SGX_CPUID_EUPDATESVN))

arch/x86/kernel/cpu/common.c void microcode_check(void) · line 2096

Quoted 1 line
arch/x86/kernel/cpu/common.c +1 -0
+		update_cpusvn_intel();

In addition to what Borislav said, these #ifdefs are not how we do things.

Quoted 7 lines

The update_cpusvn_intel() shouldn't be declared in two different headers. Just imagine what happens if some code happens to include the microcode header *then* the sgx.h header.

Please define both the 'static inline' stub *and* the declaration in sgx.h. Then you won't even need the #ifndef trickery. Please also add an "sgx_" to its name prefix while you're at it.

Moved to sgx.h and added "sgx_".

Quoted 4 lines

That comment is also not great. A reader will likely have *ZERO* idea what EUPDATESVN is or what a CPUSVN is. Try to focus on what the code *means* rather than just repeat the if() conditions in the comment.

Yes, I see. The update comment is shown in my reply to Boris. Please take a look.

#27 lore raw

RFC PATCH v3 10/10 x86/sgx: Call ENCLS[EUPDATESVN] during SGX initialization

In reply to Cathy Zhang

A snapshot of the processor microcode SVN is taken each boot cycle at the time when Intel SGX is first used. This results in microcode updates being loadable at any time, fixing microcode issues. However, if system boot up through kexec() from error recovery, no hardware reset happens, any SGX leaf execution during boot up is not assumed as the first use in such case, and no snapshot of SVN is taken. So, it's necessary to call ENCLS[EUPDATESVN] to update SVN automatically, rather than waiting for the admin to do it when he/she is even not aware of that.

Call ENCLS[EUPDATESVN] after sanitizing pages will increase the chance of success, for it requires that EPC is empty.

---
Changes since v1:
- Update accordingly for update_cpusvn_intel() return *void*.

---
 arch/x86/kernel/cpu/sgx/main.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +9 -1
index 20800c543110..48eb10f6c6fc 100644
@@ -426,6 +426,7 @@ static bool sgx_should_reclaim(unsigned long watermark)
 	       !list_empty(&sgx_active_page_list);
 }
 
+void update_cpusvn_intel(void);
 static int ksgxd(void *p)
 {
 	int srcu_idx;
@@ -440,7 +441,14 @@ static int ksgxd(void *p)
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
 	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	if (!WARN_ON(!list_empty(&sgx_dirty_page_list))) {
+		/*
+		 * Do SVN update for kexec(). It should complete without error, for
+		 * all EPC pages are unused at this point.
+		 */
+		if (cpuid_eax(SGX_CPUID) & SGX_CPUID_EUPDATESVN)
+			update_cpusvn_intel();
+	}
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())
Signature · 1 line
2.17.1