Skip to contentSkip to conversation

x86/sgx: Fix feature control rebase

PATCH In review Sean Christopherson 2 participants 8 messages Feb 1, 2020 – Feb 4, 2020

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover x86/sgx: Fix feature control rebase

Two patches to fix the Feature Control MSR bit definitions, which didn't get correctly rebased.

Patch 03 reworks SGX handling of Feature Control to use the new feat_ctl.c code, which was the entire point of that series. Patch 03 is a drop-in replacement for 4249f9b240b7 ("x86/cpu/intel: Detect SGX supprt").

*** DISCLAIMER ***
These patches are untested against your master, as your master doesn't boot on my system. The fully tested version is available at
https://github.com/sean-jc/linux.git, branch sgx/for_jarkko_v25_rebased. I'll dig into the boot issue next week.

Sean Christopherson (3):
x86/msr: Fixup "Add Intel SGX hardware bits"
x86/msr: Fixup "Intel SGX Launch Control hardware bits" x86/cpu: Configure SGX support when initializing feature control MSR

 arch/x86/include/asm/msr-index.h |  9 ++------
 arch/x86/kernel/cpu/feat_ctl.c   | 29 +++++++++++++++++++++++-
 arch/x86/kernel/cpu/intel.c      | 39 --------------------------------
 3 files changed, 30 insertions(+), 47 deletions(-)
Signature · 1 line
2.24.1
#1 lore raw

PATCH 1/3 x86/msr: Fixup "Add Intel SGX hardware bits"

Fix the Feature Control MSR bit definition for SGX that got borked during the rebase to the latest upstream.

Fixes 8813e054c085 ("x86/cpufeatures: x86/msr: Add Intel SGX hardware bits")
---
 arch/x86/include/asm/msr-index.h | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)
arch/x86/include/asm/msr-index.h +1 -7
index 75f41095f9b9..3ddc6336aaa2 100644
@@ -564,6 +564,7 @@
 #define FEAT_CTL_LOCKED				BIT(0)
 #define FEAT_CTL_VMX_ENABLED_INSIDE_SMX		BIT(1)
 #define FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX	BIT(2)
+#define FEAT_CTL_SGX_ENABLED			BIT(18)
 #define FEAT_CTL_LMCE_ENABLED			BIT(20)
 
 #define MSR_IA32_TSC_ADJUST             0x0000003b
@@ -573,13 +574,6 @@
 
 #define MSR_IA32_XSS			0x00000da0
 
-#define FEATURE_CONTROL_LOCKED				(1<<0)
-#define FEATURE_CONTROL_VMXON_ENABLED_INSIDE_SMX	(1<<1)
-#define FEATURE_CONTROL_VMXON_ENABLED_OUTSIDE_SMX	(1<<2)
-#define FEATURE_CONTROL_SGX_LE_WR			(1<<17)
-#define FEATURE_CONTROL_SGX_ENABLE			(1<<18)
-#define FEATURE_CONTROL_LMCE				(1<<20)
-
 #define MSR_IA32_APICBASE		0x0000001b
 #define MSR_IA32_APICBASE_BSP		(1<<8)
 #define MSR_IA32_APICBASE_ENABLE	(1<<11)
Signature · 1 line
2.24.1
#2 lore raw

PATCH 2/3 x86/msr: Fixup "Intel SGX Launch Control hardware bits"

In reply to Sean Christopherson

Fix the Feature Control MSR bit definition for SGX LC that got borked during the rebase to the latest upstream.

Note, the name diverges from what was previously used in the SGX series. The decision made by/with Boris was to follow the SDM names.

Fixes 24670c2036be ("x86/cpufeatures: x86/msr: Intel SGX Launch Control hardware bits")
---
 arch/x86/include/asm/msr-index.h | 1 +
 1 file changed, 1 insertion(+)
arch/x86/include/asm/msr-index.h +1 -0
index 3ddc6336aaa2..a0776c262820 100644
@@ -564,6 +564,7 @@
 #define FEAT_CTL_LOCKED				BIT(0)
 #define FEAT_CTL_VMX_ENABLED_INSIDE_SMX		BIT(1)
 #define FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX	BIT(2)
+#define FEAT_CTL_SGX_LC_ENABLED			BIT(17)
 #define FEAT_CTL_SGX_ENABLED			BIT(18)
 #define FEAT_CTL_LMCE_ENABLED			BIT(20)
 
Signature · 1 line
2.24.1
#3 lore raw

PATCH 3/3 x86/cpu: Configure SGX support when initializing feature control MSR

In reply to Sean Christopherson

Configure SGX as part of feature control MSR initialization and update the associated X86_FEATURE flags accordingly. Because the kernel will require the LE hash MSRs to be writable when running native enclaves, disable X86_FEATURE_SGX (and all derivatives) if SGX Launch Control is not (or cannot) be fully enabled via feature control MSR.

Note, unlike VMX, clear the X86_FEATURE_SGX* flags for all CPUs if any CPU lacks SGX support as the kernel expects SGX to be available on all CPUs. X86_FEATURE_VMX is intentionally cleared only for the current CPU so that KVM can provide additional information if KVM fails to load, e.g. print which CPU doesn't support VMX. KVM/VMX requires additional per-CPU enabling, e.g. to set CR4.VMXE and do VMXON, and so already has the necessary infrastructure to do per-CPU checks. SGX on the other hand doesn't require additional enabling, so clearing the feature flags on all CPUs means the SGX subsystem doesn't need to manually do support checks on a per-CPU basis.

---

This reverts commit 4249f9b240b7, i.e. is a drop-in replacement.

Fixes 4249f9b240b7 ("x86/cpu/intel: Detect SGX supprt")
 arch/x86/kernel/cpu/feat_ctl.c | 29 ++++++++++++++++++++++++-
 arch/x86/kernel/cpu/intel.c    | 39 ----------------------------------
 2 files changed, 28 insertions(+), 40 deletions(-)
arch/x86/kernel/cpu/feat_ctl.c +28 -1
index 0268185bef94..b16b71a6da74 100644
@@ -92,6 +92,14 @@ static void init_vmx_capabilities(struct cpuinfo_x86 *c)
 }
 #endif /* CONFIG_X86_VMX_FEATURE_NAMES */
 
+static void clear_sgx_caps(void)
+{
+	setup_clear_cpu_cap(X86_FEATURE_SGX);
+	setup_clear_cpu_cap(X86_FEATURE_SGX_LC);
+	setup_clear_cpu_cap(X86_FEATURE_SGX1);
+	setup_clear_cpu_cap(X86_FEATURE_SGX2);
+}
+
 void init_ia32_feat_ctl(struct cpuinfo_x86 *c)
 {
 	bool tboot = tboot_enabled();
@@ -99,6 +107,7 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)
 
 	if (rdmsrl_safe(MSR_IA32_FEAT_CTL, &msr)) {
 		clear_cpu_cap(c, X86_FEATURE_VMX);
+		clear_sgx_caps();
 		return;
 	}
 
@@ -123,13 +132,21 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)
 			msr |= FEAT_CTL_VMX_ENABLED_INSIDE_SMX;
 	}
 
+	/*
+	 * Enable SGX if and only if the kernel supports SGX and Launch Control
+	 * is supported, i.e. disable SGX if the LE hash MSRs can't be written.
+	 */
+	if (cpu_has(c, X86_FEATURE_SGX) && cpu_has(c, X86_FEATURE_SGX_LC) &&
+	    IS_ENABLED(CONFIG_INTEL_SGX))
+		msr |= FEAT_CTL_SGX_ENABLED | FEAT_CTL_SGX_LC_ENABLED;
+
 	wrmsrl(MSR_IA32_FEAT_CTL, msr);
 
 update_caps:
 	set_cpu_cap(c, X86_FEATURE_MSR_IA32_FEAT_CTL);
 
 	if (!cpu_has(c, X86_FEATURE_VMX))
-		return;
+		goto update_sgx;
 
 	if ( (tboot && !(msr & FEAT_CTL_VMX_ENABLED_INSIDE_SMX)) ||
 	    (!tboot && !(msr & FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX))) {
@@ -142,4 +159,14 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)
 		init_vmx_capabilities(c);
 #endif
 	}
+
+update_sgx:
+	if (!cpu_has(c, X86_FEATURE_SGX) || !cpu_has(c, X86_FEATURE_SGX_LC)) {
+		clear_sgx_caps();
+	} else if (!(msr & FEAT_CTL_SGX_ENABLED) ||
+		   !(msr & FEAT_CTL_SGX_LC_ENABLED)) {
+		if (IS_ENABLED(CONFIG_INTEL_SGX))
+			pr_err_once("SGX disabled by BIOS\n");
+		clear_sgx_caps();
+	}
 }
arch/x86/kernel/cpu/intel.c +0 -39
index cab0940784a7..be82cd5841c3 100644
@@ -542,40 +542,6 @@ static void detect_tme(struct cpuinfo_x86 *c)
 	c->x86_phys_bits -= keyid_bits;
 }
 
-static void __maybe_unused detect_sgx(struct cpuinfo_x86 *c)
-{
-	unsigned long long fc;
-
-	rdmsrl(MSR_IA32_FEAT_CTL, fc);
-	if (!(fc & FEATURE_CONTROL_LOCKED)) {
-		pr_err_once("sgx: The feature control MSR is not locked\n");
-		goto err_unsupported;
-	}
-
-	if (!(fc & FEATURE_CONTROL_SGX_ENABLE)) {
-		pr_err_once("sgx: SGX is not enabled in IA32_FEATURE_CONTROL MSR\n");
-		goto err_unsupported;
-	}
-
-	if (!cpu_has(c, X86_FEATURE_SGX1)) {
-		pr_err_once("sgx: SGX1 instruction set is not supported\n");
-		goto err_unsupported;
-	}
-
-	if (!(fc & FEATURE_CONTROL_SGX_LE_WR)) {
-		pr_info_once("sgx: The launch control MSRs are not writable\n");
-		goto err_unsupported;
-	}
-
-	return;
-
-err_unsupported:
-	setup_clear_cpu_cap(X86_FEATURE_SGX);
-	setup_clear_cpu_cap(X86_FEATURE_SGX1);
-	setup_clear_cpu_cap(X86_FEATURE_SGX2);
-	setup_clear_cpu_cap(X86_FEATURE_SGX_LC);
-}
-
 static void init_cpuid_fault(struct cpuinfo_x86 *c)
 {
 	u64 msr;
@@ -712,11 +678,6 @@ static void init_intel(struct cpuinfo_x86 *c)
 	if (cpu_has(c, X86_FEATURE_TME))
 		detect_tme(c);
 
-#ifdef CONFIG_INTEL_SGX
-	if (cpu_has(c, X86_FEATURE_SGX))
-		detect_sgx(c);
-#endif
-
 	init_intel_misc_features(c);
 
 	if (tsx_ctrl_state == TSX_CTRL_ENABLE)
Signature · 1 line
2.24.1
#4 lore raw
In reply to Sean Christopherson
Quoted 7 lines · Sean Christopherson

Two patches to fix the Feature Control MSR bit definitions, which didn't get correctly rebased.

Patch 03 reworks SGX handling of Feature Control to use the new feat_ctl.c code, which was the entire point of that series. Patch 03 is a drop-in replacement for 4249f9b240b7 ("x86/cpu/intel: Detect SGX supprt").

Quoted 5 lines

*** DISCLAIMER ***
These patches are untested against your master, as your master doesn't boot on my system. The fully tested version is available at
https://github.com/sean-jc/linux.git, branch sgx/for_jarkko_v25_rebased. I'll dig into the boot issue next week.

Possibly it was related that I was temporarily on top of x86/tip? It is now again on top of Linus' tree. I created v25-rc2 tag that has these updates:

tag v25-rc2

Tagger: Jarkko Sakkinen <[email protected]>
Date:   Sat Feb 1 21:27:24 2020 +0200

x86/sgx: v25-rc2

* Fix a double-free issue when SGX_IOC_ENCLAVE_ADD_PAGES
fails on executing ENCLS[EADD]. The rollback path executed radix_tree_delete() on the same address twice when this happened.
* Return -EINTR instead of -ERESTARTSYS in SGX_IOC_ENCLAVE_ADD_PAGES when a signal is pending.
* As requested by Borislav, move the CPUID 0x12 features to their own word in cpufeatures.
* Sean fixed a bug from sgx_reclaimer_write() where sgx_encl_put_backing() was called with an uninitialized pointer when sgx_encl_get_backing() fails.
* Migrated /dev/sgx/* to misc. This is future-proof as struct miscdevice has 'groups' for setting up sysfs attributes for the device.
* Use device_initcall instead of subsys_initcall so that misc_class is initialized before SGX is initialized.
* Return -EACCES in SGX_IOC_ENCLAVE_INIT when caller tries to select enclave attributes that we the kernel does not allow it to set instead of -EINVAL.
* Unless SGX public key MSRs are writable always deny the feature from Linux. Previously this was only denied from driver. How VMs should be supported is not really part of initial patch set, which makes this an obvious choice.
* Rewrote the documentation. Lessened the fine-grained micro architecture details as they can be looked up from Intel SDM in order to make the core ideas more approachable.
-----BEGIN PGP SIGNATURE-----

iJYEABYIAD4WIQRE6pSOnaBC00OEHEIaerohdGur0gUCXjXRMiAcamFya2tvLnNh a2tpbmVuQGxpbnV4LmludGVsLmNvbQAKCRAaerohdGur0ihBAQC3VE2u6zyPYFLN hCBFEF3LKqpNk26DjkO9M5tRZfUhSgEA/sF6AKHJRDqYUePW6N6Rtc3GOZY9DmbD R+qOg0XcbAs=
=OtqV
-----END PGP SIGNATURE-----

If you fix any regression, use solely this tag as the baseline for fixes. I worry about possible merge conflicts with the master.

/Jarkko

#5 lore raw
Quoted 57 lines · Jarkko Sakkinen
Quoted 7 lines · Sean Christopherson

Two patches to fix the Feature Control MSR bit definitions, which didn't get correctly rebased.

Patch 03 reworks SGX handling of Feature Control to use the new feat_ctl.c code, which was the entire point of that series. Patch 03 is a drop-in replacement for 4249f9b240b7 ("x86/cpu/intel: Detect SGX supprt").

Quoted 5 lines

*** DISCLAIMER ***
These patches are untested against your master, as your master doesn't boot on my system. The fully tested version is available at
https://github.com/sean-jc/linux.git, branch sgx/for_jarkko_v25_rebased. I'll dig into the boot issue next week.

Possibly it was related that I was temporarily on top of x86/tip? It is now again on top of Linus' tree. I created v25-rc2 tag that has these updates:

tag v25-rc2

Tagger: Jarkko Sakkinen <[email protected]>
Date:   Sat Feb 1 21:27:24 2020 +0200

x86/sgx: v25-rc2

* Fix a double-free issue when SGX_IOC_ENCLAVE_ADD_PAGES
fails on executing ENCLS[EADD]. The rollback path executed radix_tree_delete() on the same address twice when this happened.
* Return -EINTR instead of -ERESTARTSYS in SGX_IOC_ENCLAVE_ADD_PAGES when a signal is pending.
* As requested by Borislav, move the CPUID 0x12 features to their own word in cpufeatures.
* Sean fixed a bug from sgx_reclaimer_write() where sgx_encl_put_backing() was called with an uninitialized pointer when sgx_encl_get_backing() fails.
* Migrated /dev/sgx/* to misc. This is future-proof as struct miscdevice has 'groups' for setting up sysfs attributes for the device.
* Use device_initcall instead of subsys_initcall so that misc_class is initialized before SGX is initialized.
* Return -EACCES in SGX_IOC_ENCLAVE_INIT when caller tries to select enclave attributes that we the kernel does not allow it to set instead of -EINVAL.
* Unless SGX public key MSRs are writable always deny the feature from Linux. Previously this was only denied from driver. How VMs should be supported is not really part of initial patch set, which makes this an obvious choice.
* Rewrote the documentation. Lessened the fine-grained micro architecture details as they can be looked up from Intel SDM in order to make the core ideas more approachable.
-----BEGIN PGP SIGNATURE-----

iJYEABYIAD4WIQRE6pSOnaBC00OEHEIaerohdGur0gUCXjXRMiAcamFya2tvLnNh a2tpbmVuQGxpbnV4LmludGVsLmNvbQAKCRAaerohdGur0ihBAQC3VE2u6zyPYFLN hCBFEF3LKqpNk26DjkO9M5tRZfUhSgEA/sF6AKHJRDqYUePW6N6Rtc3GOZY9DmbD R+qOg0XcbAs=
=OtqV
-----END PGP SIGNATURE-----

Quoted 4 lines

If you fix any regression, use solely this tag as the baseline for fixes. I worry about possible merge conflicts with the master.

/Jarkko

As far as I'm concerned master has now everything for v25, so I removed the tag. Just test against master.

/Jarkko

#6 lore raw
Quoted 8 lines · Jarkko Sakkinen
Quoted 4 lines · Jarkko Sakkinen

If you fix any regression, use solely this tag as the baseline for fixes. I worry about possible merge conflicts with the master.

/Jarkko

As far as I'm concerned master has now everything for v25, so I removed the tag. Just test against master.

Sent a fix for the boot issue.

Can you also rebase to Linus' latest tree? Or any tree that containts commit 8df5bb4a03b0 ("char: hpet: Fix out-of-bounds read bug"). Spent the morning bisecting another boot crash due to the hpet bug...

#7 lore raw
Quoted 7 lines · Sean Christopherson
Quoted 6 lines · Jarkko Sakkinen
Quoted 4 lines · Jarkko Sakkinen

If you fix any regression, use solely this tag as the baseline for fixes. I worry about possible merge conflicts with the master.

/Jarkko

Quoted 8 lines
Quoted 2 lines

As far as I'm concerned master has now everything for v25, so I removed the tag. Just test against master.

Sent a fix for the boot issue.

Can you also rebase to Linus' latest tree? Or any tree that containts commit 8df5bb4a03b0 ("char: hpet: Fix out-of-bounds read bug"). Spent the morning bisecting another boot crash due to the hpet bug...

OK, I'll send the patch set today.

/Jarkko