Skip to contentSkip to conversation

x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

PATCH In review Dave Hansen 6 participants 25 messages Jul 20, 2022 – Aug 18, 2022

Subsystems (3)
lore
Share & tools
lore

Dave Hansen

author maintainer
#0 lore raw

PATCH v2 1/1 x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

Changes from v1:
* Make sure SGX_ATTR_ASYNC_EXIT_NOTIFY is in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.

--

Short Version:

Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

Long Version:

== SGX Attribute Background ==

The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.

As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.

The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.

For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.

== AEX Notify Background ==

"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:

Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.

Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

== The Problem ==

These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.

== The Solution ==

Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes. Instead of adding to the open-coded lists of individual attributes, add named lists of privileged (disallowed by default) and unprivileged (allowed by default) attributes. Add the AEX notify attribute as an unprivileged attribute, which will keep the kernel from rejecting enclaves with it set.

I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware (and new microcde) to test it.

Testing on bare-metal and in VMs accompanied by Tested-by's would be much appreciated. (This means you, Intel folks who actually have systems with the microcode that can do this.)

1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true

Cc Jarkko Sakkinen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Kai Huang <[email protected]>
Cc Haitao Huang <[email protected]>
---
 arch/x86/include/asm/sgx.h      | 33 ++++++++++++++++++++++++++-------
 arch/x86/kernel/cpu/sgx/ioctl.c |  2 +-
 arch/x86/kvm/cpuid.c            |  4 +---
 3 files changed, 28 insertions(+), 11 deletions(-)
arch/x86/include/asm/sgx.h +26 -7
index 3f9334ef67cd..3004dfe76498 100644
@@ -110,17 +110,36 @@ enum sgx_miscselect {
  * %SGX_ATTR_EINITTOKENKEY:	Allow to use token signing key that is used to
  *				sign cryptographic tokens that can be passed to
  *				EINIT as an authorization to run an enclave.
+ * %SGX_ATTR_ASYNC_EXIT_NOTIFY:	Allow enclaves to be notified after an
+ *				asynchronous exit has occurred.
  */
 enum sgx_attribute {
-	SGX_ATTR_INIT		= BIT(0),
-	SGX_ATTR_DEBUG		= BIT(1),
-	SGX_ATTR_MODE64BIT	= BIT(2),
-	SGX_ATTR_PROVISIONKEY	= BIT(4),
-	SGX_ATTR_EINITTOKENKEY	= BIT(5),
-	SGX_ATTR_KSS		= BIT(7),
+	SGX_ATTR_INIT		   = BIT(0),
+	SGX_ATTR_DEBUG		   = BIT(1),
+	SGX_ATTR_MODE64BIT	   = BIT(2),
+				  /* BIT(3) is reserved */
+	SGX_ATTR_PROVISIONKEY	   = BIT(4),
+	SGX_ATTR_EINITTOKENKEY	   = BIT(5),
+				  /* BIT(6) is for CET */
+	SGX_ATTR_KSS		   = BIT(7),
+				  /* BIT(8) is reserved */
+				  /* BIT(9) is reserved */
+	SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10),
 };
 
-#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | BIT_ULL(6) | GENMASK_ULL(63, 8))
+#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | \
+				 BIT_ULL(6) | \
+				 BIT_ULL(8) | \
+				 BIT_ULL(9) | \
+				 GENMASK_ULL(63, 11))
+
+#define SGX_ATTR_UNPRIV_MASK	(SGX_ATTR_DEBUG	    | \
+				 SGX_ATTR_MODE64BIT | \
+				 SGX_ATTR_KSS	    | \
+				 SGX_ATTR_ASYNC_EXIT_NOTIFY)
+
+#define SGX_ATTR_PRIV_MASK	(SGX_ATTR_PROVISIONKEY	| \
+				 SGX_ATTR_EINITTOKENKEY)
 
 /**
  * struct sgx_secs - SGX Enclave Control Structure (SECS)
arch/x86/kernel/cpu/sgx/ioctl.c +1 -1
index 83df20e3e633..37d523895244 100644
@@ -110,7 +110,7 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)
 	encl->base = secs->base;
 	encl->size = secs->size;
 	encl->attributes = secs->attributes;
-	encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT | SGX_ATTR_KSS;
+	encl->attributes_mask = SGX_ATTR_UNPRIV_MASK;
 
 	/* Set only after completion, as encl->lock has not been taken. */
 	set_bit(SGX_ENCL_CREATED, &encl->flags);
arch/x86/kvm/cpuid.c +1 -3
index 0c1ba6aa0765..96a73b5b4369 100644
@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		 * userspace.  ATTRIBUTES.XFRM is not adjusted as userspace is
 		 * expected to derive it from supported XCR0.
 		 */
-		entry->eax &= SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |
-			      SGX_ATTR_PROVISIONKEY | SGX_ATTR_EINITTOKENKEY |
-			      SGX_ATTR_KSS;
+		entry->eax &= SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK;
 		entry->ebx &= 0;
 		break;
 	/* Intel PT */
Signature · 1 line
2.34.1
#1 lore raw
Quoted 4 lines · Dave Hansen

Changes from v1:
* Make sure SGX_ATTR_ASYNC_EXIT_NOTIFY is in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.

Heh, I was wondering if KVM needed to be updated.

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 1025

Quoted 12 lines
arch/x86/kvm/cpuid.c +1 -3
index 0c1ba6aa0765..96a73b5b4369 100644
@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		 * userspace.  ATTRIBUTES.XFRM is not adjusted as userspace is
 		 * expected to derive it from supported XCR0.
 		 */
-		entry->eax &= SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |
-			      SGX_ATTR_PROVISIONKEY | SGX_ATTR_EINITTOKENKEY |
-			      SGX_ATTR_KSS;
+		entry->eax &= SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK;

It may seem like a maintenance burdern, and it is to some extent, but I think it's better for KVM to have to explicitly "enable" each flag. There is no guarantee that a new feature will not require additional KVM enabling, i.e. we want the pain of having to manually update KVM so that we get "feature X isn't virtualized" complaints and not "I upgraded my kernel and my enclaves broke" bug reports.

I don't think it's likely that attribute-based features will require additional enabling since there aren't any virtualization controls for the ENCLU side of things (ENCLU is effectively disabled by blocking ENCLS[ECREATE]), but updating KVM isn't particularly difficult so I'd rather be paranoid.

#2 lore raw
Quoted 20 lines · Sean Christopherson

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 1025

Quoted 12 lines · Dave Hansen
arch/x86/kvm/cpuid.c +1 -3
index 0c1ba6aa0765..96a73b5b4369 100644
@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		 * userspace.  ATTRIBUTES.XFRM is not adjusted as userspace is
 		 * expected to derive it from supported XCR0.
 		 */
-		entry->eax &= SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |
-			      SGX_ATTR_PROVISIONKEY | SGX_ATTR_EINITTOKENKEY |
-			      SGX_ATTR_KSS;
+		entry->eax &= SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK;

It may seem like a maintenance burdern, and it is to some extent, but I think it's better for KVM to have to explicitly "enable" each flag. There is no guarantee that a new feature will not require additional KVM enabling, i.e. we want the pain of having to manually update KVM so that we get "feature X isn't virtualized" complaints and not "I upgraded my kernel and my enclaves broke" bug reports.

Quoted 4 lines

I don't think it's likely that attribute-based features will require additional enabling since there aren't any virtualization controls for the ENCLU side of things (ENCLU is effectively disabled by blocking ENCLS[ECREATE]), but updating KVM isn't particularly difficult so I'd rather be paranoid.

How about something where KVM gets to keep a discrete mask, but where it's at least defined next to the attributes, something like:

/*
* These attributes will be advertised to KVM guests as being
* available. This includes privileged attributes. Only add
* to this list when host-side KVM does not require additional
* enabling for the attribute.
*/

#define SGX_ATTR_KVM_MASK       (SGX_ATTR_DEBUG         | \
                                 SGX_ATTR_MODE64BIT     | \
                                 SGX_ATTR_PROVISIONKEY  | \
                                 SGX_ATTR_EINITTOKENKEY | \
                                 SGX_ATTR_KSS           | \

SGX_ATTR_ASYNC_EXIT_NOTIFY)

That at least has a *chance* of someone seeing it who goes to add a new attribute.

#3 lore raw
Quoted 40 lines · Dave Hansen
Quoted 20 lines · Sean Christopherson

arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 1025

Quoted 12 lines · Dave Hansen
arch/x86/kvm/cpuid.c +1 -3
index 0c1ba6aa0765..96a73b5b4369 100644
@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		 * userspace.  ATTRIBUTES.XFRM is not adjusted as userspace is
 		 * expected to derive it from supported XCR0.
 		 */
-		entry->eax &= SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |
-			      SGX_ATTR_PROVISIONKEY | SGX_ATTR_EINITTOKENKEY |
-			      SGX_ATTR_KSS;
+		entry->eax &= SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK;

It may seem like a maintenance burdern, and it is to some extent, but I think it's better for KVM to have to explicitly "enable" each flag. There is no guarantee that a new feature will not require additional KVM enabling, i.e. we want the pain of having to manually update KVM so that we get "feature X isn't virtualized" complaints and not "I upgraded my kernel and my enclaves broke" bug reports.

Quoted 4 lines

I don't think it's likely that attribute-based features will require additional enabling since there aren't any virtualization controls for the ENCLU side of things (ENCLU is effectively disabled by blocking ENCLS[ECREATE]), but updating KVM isn't particularly difficult so I'd rather be paranoid.

How about something where KVM gets to keep a discrete mask, but where it's at least defined next to the attributes, something like:

/*
* These attributes will be advertised to KVM guests as being
* available. This includes privileged attributes. Only add
* to this list when host-side KVM does not require additional
* enabling for the attribute.
*/

#define SGX_ATTR_KVM_MASK       (SGX_ATTR_DEBUG         | \
                                 SGX_ATTR_MODE64BIT     | \
                                 SGX_ATTR_PROVISIONKEY  | \
                                 SGX_ATTR_EINITTOKENKEY | \
                                 SGX_ATTR_KSS           | \
Quoted 4 lines

SGX_ATTR_ASYNC_EXIT_NOTIFY)

That at least has a *chance* of someone seeing it who goes to add a new attribute.

Hmm, what if we enforce it in code with a compile-time assert? That will make it even harder to screw things up, and it also avoids a scenario where someone extends SGX_ATTR_KVM_MASK without getting approval from KVM folks. And conversely, KVM won't need to touch SGX files if there's ever a need to tweak KVM behavior.

/*
* Index 1: SECS.ATTRIBUTES. ATTRIBUTES are restricted a la
* feature flags. Advertise all supported flags, including
* privileged attributes that require explicit opt-in from
* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is
* expected to derive it from supported XCR0.
*/

#define KVM_SGX_ATTR_ALLOWED_MASK (SGX_ATTR_DEBUG |		\
				   SGX_ATTR_MODE64BIT |		\
				   SGX_ATTR_PROVISIONKEY |	\
				   SGX_ATTR_EINITTOKENKEY |	\
				   SGX_ATTR_KSS |		\
				   SGX_ATTR_ASYNC_EXIT_NOTIFY)

#define KVM_SGX_ATTR_DENIED_MASK (0)

		/*
		 * Assert that KVM explicitly allows or denies exposing all
		 * features, i.e. detect attempts to add kernel support without
		 * also updating KVM.
		 */
		BUILD_BUG_ON((KVM_SGX_ATTR_ALLOWED_MASK | KVM_SGX_ATTR_DENIED_MASK) !=
			     (SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK));

		entry->eax &= KVM_SGX_ATTR_ALLOWED_MASK;
		entry->ebx &= 0;
		break;
#4 lore raw
Quoted 7 lines · Sean Christopherson
		/*
		 * Assert that KVM explicitly allows or denies exposing all
		 * features, i.e. detect attempts to add kernel support without
		 * also updating KVM.
		 */
		BUILD_BUG_ON((KVM_SGX_ATTR_ALLOWED_MASK | KVM_SGX_ATTR_DENIED_MASK) !=
			     (SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK));

Looks good to me. I'll do that for the next version. Thanks!

#5 lore raw
In reply to Dave Hansen
Quoted 14 lines · Dave Hansen
arch/x86/kvm/cpuid.c +1 -4
@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
  		 * userspace.  ATTRIBUTES.XFRM is not adjusted as userspace is
  		 * expected to derive it from supported XCR0.
  		 */
-		entry->eax &= SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |
-			      SGX_ATTR_PROVISIONKEY | SGX_ATTR_EINITTOKENKEY |
-			      SGX_ATTR_KSS;
+		entry->eax &= SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK;
  		entry->ebx &= 0;
  		break;
  	/* Intel PT */
--
Quoted 1 line

2.34.1

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?

(quoted diff) +0 -0

b/arch/x86/include/asm/cpufeatures.h
index 6466a58b9cff..d2ebb38b31e7 100644

arch/x86/include/asm/cpufeatures.h +0 -0
@@ -296,6 +296,7 @@
 #define X86_FEATURE_PER_THREAD_MBA     (11*32+ 7) /* "" Per-thread Memory

Bandwidth Allocation */
#define X86_FEATURE_SGX1 (11*32+ 8) /* "" Basic SGX */
#define X86_FEATURE_SGX2 (11*32+ 9) /* "" SGX Enclave Dynamic
Memory Management (EDMM) */
+#define X86_FEATURE_SGX_EDECCSSA (11*32+10) /* "" SGX EDECCSSA user leaf
function */

/* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */
#define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */

arch/x86/kvm/cpuid.c +1 -1
index d47222ab8e6e..121456653417 100644
@@ -612,7 +612,7 @@ void kvm_set_cpu_caps(void)
        );
 
        kvm_cpu_cap_init_scattered(CPUID_12_EAX,
-               SF(SGX1) | SF(SGX2)
+               SF(SGX1) | SF(SGX2) | SF(SGX_EDECCSSA)
        );
 
        kvm_cpu_cap_mask(CPUID_8000_0001_ECX,
arch/x86/kvm/reverse_cpuid.h +1 -0
index a19d473d0184..4e5b8444f161 100644
@@ -23,6 +23,7 @@ enum kvm_only_cpuid_leafs {
 /* Intel-defined SGX sub-features, CPUID level 0x12 (EAX). */
 #define KVM_X86_FEATURE_SGX1           KVM_X86_FEATURE(CPUID_12_EAX, 0)
 #define KVM_X86_FEATURE_SGX2           KVM_X86_FEATURE(CPUID_12_EAX, 1)
+#define KVM_X86_FEATURE_SGX_EDECCSSA   KVM_X86_FEATURE(CPUID_12_EAX, 11)
 
 struct cpuid_reg {
        u32 function;
@@ -78,6 +79,8 @@ static __always_inline u32 __feature_translate(int

x86_feature)

                return KVM_X86_FEATURE_SGX1;
        else if (x86_feature == X86_FEATURE_SGX2)
                return KVM_X86_FEATURE_SGX2;

+ else if (x86_feature == X86_FEATURE_SGX_EDECCSSA)
+ return KVM_X86_FEATURE_SGX_EDECCSSA;

        return x86_feature;

}

#6 lore raw
Quoted 4 lines · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?

Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:

Quoted 2 lines

EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...

If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.

Oh, and the one-liner patch that I was promised for enabling this is getting a _wee_ bit longer than one line.

#7 lore raw
Quoted 11 lines · Dave Hansen
Quoted 4 lines · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?

Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:

Quoted 2 lines

EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...

Yeah, it's enumerated separately.

Quoted 5 lines

If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.

Oh, and the one-liner patch that I was promised for enabling this is getting a _wee_ bit longer than one line.

Heh, fool me once...

#8 lore raw
In reply to Dave Hansen
Quoted 6 lines · Dave Hansen
Quoted 4 lines · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?

Quoted 9 lines

Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:

Quoted 2 lines

EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...

If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.

They are enumerated separately, but looks in practice the notify handler will use it to switch back to the correct/targeted CSSA to continue to run normally after handling the exit notify. This is my understanding of the "facilitate" mean in the spec.

Btw, in real hardware I think the two should come together, meaning no real hardware will only support one.

Haitao, could you give us more information?

Signature · 2 lines
Thanks,
-Kai
#9 lore raw
Quoted 25 lines · Kai Huang
Quoted 9 lines · Dave Hansen
Quoted 1 line · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be

used

Quoted 2 lines

together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should

also

Quoted 1 line

advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below

(untested)?

Quoted 9 lines

Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:

Quoted 2 lines

EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...

If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.

They are enumerated separately, but looks in practice the notify handler will use it to switch back to the correct/targeted CSSA to continue to run normally after handling the exit notify. This is my understanding of the "facilitate" mean in the spec.

Quoted 5 lines

Btw, in real hardware I think the two should come together, meaning no real hardware will only support one.

Haitao, could you give us more information?

You are right. They are enumerated separately and HW should come with both or neither. My understanding it is also possible for enclaves choose not to receive AEX notify but still use EDECCSSA.

Thanks
Haitao

#10 lore raw
Quoted 34 lines · Haitao Huang
Quoted 32 lines · Kai Huang
Quoted 9 lines · Dave Hansen
Quoted 1 line · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be

used

Quoted 2 lines

together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should

also

Quoted 1 line

advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below

(untested)?

Quoted 9 lines

Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:

Quoted 2 lines

EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...

If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.

They are enumerated separately, but looks in practice the notify handler will
use it to switch back to the correct/targeted CSSA to continue to run normally
after handling the exit notify. This is my understanding of the "facilitate"
mean in the spec.

Btw, in real hardware I think the two should come together, meaning no real
hardware will only support one.

Quoted 8 lines
Quoted 2 lines

Haitao, could you give us more information?

You are right. They are enumerated separately and HW should come with both or neither.
My understanding it is also possible for enclaves choose not to receive AEX notify
but still use EDECCSSA.

What is the use case of using EDECCSSA w/o using AEX notify?

If I understand correctly EDECCSSA effectively switches to another thread (using the previous SSA, which is the context of another TCS thread if I understand correctly). Won't this cause problem?

It probably makes sense to use only AEX notify w/o using EDECCSSA though, but this should be the case that the enclave detects serious attack and don't want to continue to run normally. In another words, it is enclave's choice, but hardware should always support both AEX notify and EDECCSSA.

Signature · 2 lines
Thanks,
-Kai
#11 lore raw
Quoted 48 lines · Kai Huang
Quoted 38 lines · Haitao Huang
Quoted 3 lines · Kai Huang
Quoted 2 lines · Dave Hansen
Quoted 1 line · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should

be

Quoted 8 lines
Quoted 8 lines

used

Quoted 2 lines

together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should

also

Quoted 1 line

advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below

(untested)?

Sounds like a great follow-on patch! It doesn't seem truly

functionally

Quoted 6 lines
Quoted 6 lines

required from the spec:

Quoted 2 lines

EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...

If that's wrong or imprecise, I'd love to hear more about it and

also

Quoted 4 lines
Quoted 2 lines

about how the spec will be updated.

They are enumerated separately, but looks in practice the notify

handler

Quoted 11 lines

will
use it to switch back to the correct/targeted CSSA to continue to run normally
after handling the exit notify. This is my understanding of the "facilitate"
mean in the spec.

Btw, in real hardware I think the two should come together, meaning no real
hardware will only support one.

Quoted 8 lines
Quoted 2 lines

Haitao, could you give us more information?

You are right. They are enumerated separately and HW should come with both or neither.
My understanding it is also possible for enclaves choose not to receive AEX notify
but still use EDECCSSA.

Quoted 3 lines

What is the use case of using EDECCSSA w/o using AEX notify? If I understand correctly EDECCSSA effectively switches to another thread (using the previous SSA, which is the context of another TCS thread if I understand
correctly). Won't this cause problem?

No. Decrementing CSSA is equivalent to popping stack frames, not switching threads.
In some cases such as so-called "first stage" exception handling, one could pop CSSA back to the previous after resetting CPU context and stack frame appropriate to the "second stage" or "real" exception handling routine, then jump to the handler directly. This could improve exception handling performance by saving an EEXIT/ERESUME trip.

Quoted 5 lines

It probably makes sense to use only AEX notify w/o using EDECCSSA though, but this should be the case that the enclave detects serious attack and don't want to continue to run normally. In another words, it is enclave's choice, but hardware should always support both AEX notify and EDECCSSA.

#12 lore raw
Quoted 59 lines · Haitao Huang
Quoted 50 lines · Kai Huang
Quoted 39 lines · Haitao Huang
Quoted 3 lines · Kai Huang
Quoted 2 lines · Dave Hansen
Quoted 1 line · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should

be

Quoted 8 lines
Quoted 8 lines

used

Quoted 2 lines

together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should

also

Quoted 1 line

advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below

(untested)?

Sounds like a great follow-on patch! It doesn't seem truly

functionally

Quoted 6 lines
Quoted 6 lines

required from the spec:

Quoted 2 lines

EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...

If that's wrong or imprecise, I'd love to hear more about it and

also

Quoted 4 lines
Quoted 2 lines

about how the spec will be updated.

They are enumerated separately, but looks in practice the notify

handler

Quoted 11 lines

will
use it to switch back to the correct/targeted CSSA to continue to run normally
after handling the exit notify. This is my understanding of the "facilitate"
mean in the spec.

Btw, in real hardware I think the two should come together, meaning no real
hardware will only support one.

Quoted 9 lines
Quoted 2 lines

Haitao, could you give us more information?

You are right. They are enumerated separately and HW should come with both
or neither.
My understanding it is also possible for enclaves choose not to receive AEX notify
but still use EDECCSSA.

Quoted 6 lines

What is the use case of using EDECCSSA w/o using AEX notify? If I understand correctly EDECCSSA effectively switches to another thread (using
the previous SSA, which is the context of another TCS thread if I understand
correctly). Won't this cause problem?

Quoted 9 lines

No. Decrementing CSSA is equivalent to popping stack frames, not switching threads.
In some cases such as so-called "first stage" exception handling, one could pop CSSA back to the previous after resetting CPU context and stack frame appropriate to the "second stage" or "real" exception handling routine, then jump to the handler directly. This could improve exception handling performance by saving an EEXIT/ERESUME trip.

Looking at the AEX-notify spec again, EDECCSSA does below:

(* At this point, the instruction is guaranteed to complete *) CR_TCS_PA.CSSA := CR_TCS_PA.CSSA - 1;
CR_GPR_PA := Physical_Address(DS:TMP_GPR);

It doens't reset the RIP to CR_GPA_PA.RIP so looks yes you are right. It only "popping the stack frame" but doesn't switch thread.

But the pseudo code of EDECCSSA only updates the CR_TCS_PA and CR_GPR_PA registers (forget about XSAVE not), but doesn't manually updating the actual CPU registers such as GPRs. Are the actual CPU registers updated automatically when CR_xx are updated?

Signature · 2 lines
Thanks,
-Kai
#13 lore raw
Quoted 84 lines · Kai Huang
Quoted 67 lines · Haitao Huang
Quoted 7 lines · Kai Huang
Quoted 6 lines · Haitao Huang
Quoted 3 lines · Kai Huang
Quoted 2 lines · Dave Hansen
Quoted 1 line · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA]

should

Quoted 4 lines
Quoted 4 lines

be

Quoted 3 lines
Quoted 3 lines

used

Quoted 2 lines

together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we

should

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

also

Quoted 1 line

advertise the ENCLU[EDECCSSA] support in guest's CPUID, like

below

Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines

(untested)?

Sounds like a great follow-on patch! It doesn't seem truly

functionally

Quoted 4 lines
Quoted 4 lines

required from the spec:

Quoted 2 lines

EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification

handling...

Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines

If that's wrong or imprecise, I'd love to hear more about it and

also

Quoted 4 lines
Quoted 2 lines

about how the spec will be updated.

They are enumerated separately, but looks in practice the notify

handler

Quoted 2 lines

will
use it to switch back to the correct/targeted CSSA to continue to

run

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

normally
after handling the exit notify. This is my understanding of the "facilitate"
mean in the spec.

Btw, in real hardware I think the two should come together,

meaning no

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

real
hardware will only support one.

Haitao, could you give us more information?

You are right. They are enumerated separately and HW should come

with

Quoted 3 lines
Quoted 3 lines

both
or neither.
My understanding it is also possible for enclaves choose not to

receive

Quoted 10 lines
Quoted 3 lines

AEX notify
but still use EDECCSSA.

What is the use case of using EDECCSSA w/o using AEX notify? If I understand correctly EDECCSSA effectively switches to another thread (using
the previous SSA, which is the context of another TCS thread if I understand
correctly). Won't this cause problem?

Quoted 9 lines

No. Decrementing CSSA is equivalent to popping stack frames, not switching threads.
In some cases such as so-called "first stage" exception handling, one could pop CSSA back to the previous after resetting CPU context and stack frame appropriate to the "second stage" or "real" exception handling routine, then jump to the handler directly. This could improve exception handling performance by saving an EEXIT/ERESUME trip.

Looking at the AEX-notify spec again, EDECCSSA does below:

(* At this point, the instruction is guaranteed to complete *) CR_TCS_PA.CSSA := CR_TCS_PA.CSSA - 1;
CR_GPR_PA := Physical_Address(DS:TMP_GPR);

Quoted 8 lines

It doens't reset the RIP to CR_GPA_PA.RIP so looks yes you are right. It only "popping the stack frame" but doesn't switch thread.

But the pseudo code of EDECCSSA only updates the CR_TCS_PA and CR_GPR_PA registers (forget about XSAVE not), but doesn't manually updating the actual CPU registers such as GPRs. Are the actual CPU registers updated automatically when CR_xx are updated?

No, the enclave code is supposed to do that. Here is are a few more details on the flow I mentioned.

On any AEX event, CPU saves states including GPR/XSave into SSA[0]. When AEX-notify is turned off, for enclaves to handle exceptions occurred inside enclave, user space must do EENTER with the same TCS on which the exception occurred. EENTER would give a clean slate of GPR and SSA[1] becomes active for next AEX. It's enclave's responsibility to save GPR/XSave states in SSA[0] to some place (e.g., stack), then EDECCSSA, then jump to the "second stage" handler. (Note now SSA[0] is reactivated and ready if another AEX occurs). The second stage handler then fixes the situation that caused the original AEX, restore CPU context from the saved SSA[0] states, jump back to original place where exception happened.

#14 lore raw
In reply to Kai Huang
Quoted 4 lines · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?

Kai, would you care to send a new version of this with a proper SoB and changelog integrating what you've learned since posting it? It can be merged along with AEX notify itself.

#15 lore raw
Quoted 9 lines · Dave Hansen
Quoted 4 lines · Kai Huang

Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?

Kai, would you care to send a new version of this with a proper SoB and changelog integrating what you've learned since posting it? It can be merged along with AEX notify itself.

Sure will do today.

Signature · 2 lines
Thanks,
-Kai
maintainer Acked-by on v2 1/1
#16 lore raw
In reply to Dave Hansen
Quoted 184 lines · Dave Hansen

Changes from v1:
* Make sure SGX_ATTR_ASYNC_EXIT_NOTIFY is in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.

--

Short Version:

Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

Long Version:

== SGX Attribute Background ==

The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.

As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.

The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.

For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.

== AEX Notify Background ==

"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:

Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.

Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

== The Problem ==

These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.

== The Solution ==

Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes. Instead of adding to the open-coded lists of individual attributes, add named lists of privileged (disallowed by default) and unprivileged (allowed by default) attributes. Add the AEX notify attribute as an unprivileged attribute, which will keep the kernel from rejecting enclaves with it set.

I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware (and new microcde) to test it.

Testing on bare-metal and in VMs accompanied by Tested-by's would be much appreciated. (This means you, Intel folks who actually have systems with the microcode that can do this.)

1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true

Cc Jarkko Sakkinen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Kai Huang <[email protected]>
Cc Haitao Huang <[email protected]>
---
 arch/x86/include/asm/sgx.h      | 33 ++++++++++++++++++++++++++-------
 arch/x86/kernel/cpu/sgx/ioctl.c |  2 +-
 arch/x86/kvm/cpuid.c            |  4 +---
 3 files changed, 28 insertions(+), 11 deletions(-)
arch/x86/include/asm/sgx.h +26 -7
index 3f9334ef67cd..3004dfe76498 100644
@@ -110,17 +110,36 @@ enum sgx_miscselect {
  * %SGX_ATTR_EINITTOKENKEY:	Allow to use token signing key that is used to
  *				sign cryptographic tokens that can be passed to
  *				EINIT as an authorization to run an enclave.
+ * %SGX_ATTR_ASYNC_EXIT_NOTIFY:	Allow enclaves to be notified after an
+ *				asynchronous exit has occurred.
  */
 enum sgx_attribute {
-	SGX_ATTR_INIT		= BIT(0),
-	SGX_ATTR_DEBUG		= BIT(1),
-	SGX_ATTR_MODE64BIT	= BIT(2),
-	SGX_ATTR_PROVISIONKEY	= BIT(4),
-	SGX_ATTR_EINITTOKENKEY	= BIT(5),
-	SGX_ATTR_KSS		= BIT(7),
+	SGX_ATTR_INIT		   = BIT(0),
+	SGX_ATTR_DEBUG		   = BIT(1),
+	SGX_ATTR_MODE64BIT	   = BIT(2),
+				  /* BIT(3) is reserved */
+	SGX_ATTR_PROVISIONKEY	   = BIT(4),
+	SGX_ATTR_EINITTOKENKEY	   = BIT(5),
+				  /* BIT(6) is for CET */
+	SGX_ATTR_KSS		   = BIT(7),
+				  /* BIT(8) is reserved */
+				  /* BIT(9) is reserved */
+	SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10),
 };
 
-#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | BIT_ULL(6) | GENMASK_ULL(63, 8))
+#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | \
+				 BIT_ULL(6) | \
+				 BIT_ULL(8) | \
+				 BIT_ULL(9) | \
+				 GENMASK_ULL(63, 11))
+
+#define SGX_ATTR_UNPRIV_MASK	(SGX_ATTR_DEBUG	    | \
+				 SGX_ATTR_MODE64BIT | \
+				 SGX_ATTR_KSS	    | \
+				 SGX_ATTR_ASYNC_EXIT_NOTIFY)
+
+#define SGX_ATTR_PRIV_MASK	(SGX_ATTR_PROVISIONKEY	| \
+				 SGX_ATTR_EINITTOKENKEY)
 
 /**
  * struct sgx_secs - SGX Enclave Control Structure (SECS)
arch/x86/kernel/cpu/sgx/ioctl.c +1 -1
index 83df20e3e633..37d523895244 100644
@@ -110,7 +110,7 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)
 	encl->base = secs->base;
 	encl->size = secs->size;
 	encl->attributes = secs->attributes;
-	encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT | SGX_ATTR_KSS;
+	encl->attributes_mask = SGX_ATTR_UNPRIV_MASK;
 
 	/* Set only after completion, as encl->lock has not been taken. */
 	set_bit(SGX_ENCL_CREATED, &encl->flags);
arch/x86/kvm/cpuid.c +1 -3
index 0c1ba6aa0765..96a73b5b4369 100644
@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)
 		 * userspace.  ATTRIBUTES.XFRM is not adjusted as userspace is
 		 * expected to derive it from supported XCR0.
 		 */
-		entry->eax &= SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |
-			      SGX_ATTR_PROVISIONKEY | SGX_ATTR_EINITTOKENKEY |
-			      SGX_ATTR_KSS;
+		entry->eax &= SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK;
 		entry->ebx &= 0;
 		break;
 	/* Intel PT */
Signature · 1 line
2.34.1

BR, Jarkkko

Tested-by on v2 1/1
#17 lore raw
In reply to Dave Hansen
Quoted 182 lines · Dave Hansen

Changes from v1:
* Make sure SGX_ATTR_ASYNC_EXIT_NOTIFY is in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.

--

Short Version:

Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

Long Version:

== SGX Attribute Background ==

The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.

As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.

The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.

For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.

== AEX Notify Background ==

"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:

Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.

Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

== The Problem ==

These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.

== The Solution ==

Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes. Instead of adding to the open-coded lists of individual attributes, add named lists of privileged (disallowed by default) and unprivileged (allowed by default) attributes. Add the AEX notify attribute as an unprivileged attribute, which will keep the kernel from rejecting enclaves with it set.

I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware (and new microcde) to test it.

Testing on bare-metal and in VMs accompanied by Tested-by's would be much appreciated. (This means you, Intel folks who actually have systems with the microcode that can do this.)

1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true

Cc Jarkko Sakkinen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Kai Huang <[email protected]>
Cc Haitao Huang <[email protected]>
---
 arch/x86/include/asm/sgx.h      | 33 ++++++++++++++++++++++++++-------
 arch/x86/kernel/cpu/sgx/ioctl.c |  2 +-
 arch/x86/kvm/cpuid.c            |  4 +---
 3 files changed, 28 insertions(+), 11 deletions(-)
arch/x86/include/asm/sgx.h +0 -0
index 3f9334ef67cd..3004dfe76498 100644
@@ -110,17 +110,36 @@ enum sgx_miscselect {
  * %SGX_ATTR_EINITTOKENKEY:	Allow to use token signing key that is used  

to
* sign cryptographic tokens that can be passed to
* EINIT as an authorization to run an enclave.
+ * %SGX_ATTR_ASYNC_EXIT_NOTIFY: Allow enclaves to be notified after an
+ * asynchronous exit has occurred.
*/
enum sgx_attribute {

arch/x86/include/asm/sgx.h +11 -7
-	SGX_ATTR_INIT		= BIT(0),
-	SGX_ATTR_DEBUG		= BIT(1),
-	SGX_ATTR_MODE64BIT	= BIT(2),
-	SGX_ATTR_PROVISIONKEY	= BIT(4),
-	SGX_ATTR_EINITTOKENKEY	= BIT(5),
-	SGX_ATTR_KSS		= BIT(7),
+	SGX_ATTR_INIT		   = BIT(0),
+	SGX_ATTR_DEBUG		   = BIT(1),
+	SGX_ATTR_MODE64BIT	   = BIT(2),
+				  /* BIT(3) is reserved */
+	SGX_ATTR_PROVISIONKEY	   = BIT(4),
+	SGX_ATTR_EINITTOKENKEY	   = BIT(5),
+				  /* BIT(6) is for CET */
+	SGX_ATTR_KSS		   = BIT(7),
+				  /* BIT(8) is reserved */
+				  /* BIT(9) is reserved */
+	SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10),
 };
-#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | BIT_ULL(6) |  

GENMASK_ULL(63, 8))

arch/x86/include/asm/sgx.h +13 -0
+#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | \
+				 BIT_ULL(6) | \
+				 BIT_ULL(8) | \
+				 BIT_ULL(9) | \
+				 GENMASK_ULL(63, 11))
+
+#define SGX_ATTR_UNPRIV_MASK	(SGX_ATTR_DEBUG	    | \
+				 SGX_ATTR_MODE64BIT | \
+				 SGX_ATTR_KSS	    | \
+				 SGX_ATTR_ASYNC_EXIT_NOTIFY)
+
+#define SGX_ATTR_PRIV_MASK	(SGX_ATTR_PROVISIONKEY	| \
+				 SGX_ATTR_EINITTOKENKEY)

/**
* struct sgx_secs - SGX Enclave Control Structure (SECS)

arch/x86/kernel/cpu/sgx/ioctl.c +1 -1
index 83df20e3e633..37d523895244 100644
@@ -110,7 +110,7 @@ static int sgx_encl_create(struct sgx_encl *encl,  
 struct sgx_secs *secs)
 	encl->base = secs->base;
 	encl->size = secs->size;
 	encl->attributes = secs->attributes;
-	encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |  
 SGX_ATTR_KSS;
+	encl->attributes_mask = SGX_ATTR_UNPRIV_MASK;
 	/* Set only after completion, as encl->lock has not been taken. */
arch/x86/kernel/cpu/sgx/ioctl.c +0 -0
 	set_bit(SGX_ENCL_CREATED, &encl->flags);
arch/x86/kvm/cpuid.c +1 -3
index 0c1ba6aa0765..96a73b5b4369 100644
@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct  
 kvm_cpuid_array *array, u32 function)
 		 * userspace.  ATTRIBUTES.XFRM is not adjusted as userspace is
 		 * expected to derive it from supported XCR0.
 		 */
-		entry->eax &= SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT |
-			      SGX_ATTR_PROVISIONKEY | SGX_ATTR_EINITTOKENKEY |
-			      SGX_ATTR_KSS;
+		entry->eax &= SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK;
 		entry->ebx &= 0;
 		break;
Quoted 1 line

/* Intel PT */

Thanks
Haitao

#19 lore raw
Quoted 8 lines · Kai Huang
Quoted 5 lines

Thanks
Haitao

Hi Haitao,

Quoted 8 lines

Could you also help to test in a VM?

You will also need below patch in order to use EDECCSSA in the guest:

https://lore.kernel.org/lkml/[email protected]/

When you create the VM, please use -cpu host.

Hi Haitao,

Do you have any update?

If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?

Signature · 2 lines
Thanks,
-Kai
maintainer Acked-by
#20 lore raw
Quoted 18 lines · Kai
Quoted 8 lines · Kai Huang
Quoted 5 lines

Thanks
Haitao

Hi Haitao,

Quoted 8 lines

Could you also help to test in a VM?

You will also need below patch in order to use EDECCSSA in the guest:

https://lore.kernel.org/lkml/[email protected]/

When you create the VM, please use -cpu host.

Quoted 6 lines

Hi Haitao,

Do you have any update?

If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?

I can give ack for this, given that it is so obvious:

Would give reviewed-by if there was ucode update available, and I could test this.

BR, Jarkko

#21 lore raw
Quoted 27 lines · Kai
Quoted 19 lines · Kai Huang
Quoted 8 lines
Quoted 5 lines

Thanks
Haitao

Hi Haitao,

Quoted 9 lines

Could you also help to test in a VM?

You will also need below patch in order to use EDECCSSA in the guest:

https://lore.kernel.org/lkml/20220727115442.464380-1-kai.huang@intel
.com/

When you create the VM, please use -cpu host.

Quoted 6 lines

Hi Haitao,

Do you have any update?

If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?

Quoted 7 lines

I can give ack for this, given that it is so obvious:

Would give reviewed-by if there was ucode update available, and I could test this.

I don't know whether there's ucode update available. Maybe Haitao has more information.

I talked to Haitao and I'll try to verify this in a VM.

Thanks,
-Kai

#22 lore raw
Quoted 35 lines · Kai
Quoted 27 lines · Kai
Quoted 19 lines · Kai Huang
Quoted 8 lines
Quoted 5 lines

Thanks
Haitao

Hi Haitao,

Quoted 9 lines

Could you also help to test in a VM?

You will also need below patch in order to use EDECCSSA in the guest:

https://lore.kernel.org/lkml/20220727115442.464380-1-kai.huang@intel
.com/

When you create the VM, please use -cpu host.

Quoted 6 lines

Hi Haitao,

Do you have any update?

If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?

Quoted 7 lines

I can give ack for this, given that it is so obvious:

Would give reviewed-by if there was ucode update available, and I could test this.

Quoted 6 lines

I don't know whether there's ucode update available. Maybe Haitao has more information.

I talked to Haitao and I'll try to verify this in a VM.

Thanks,
-Kai

There's no release for Icelake existing yet with AEX Notify.

BR, Jarkko

Tested-by
#23 lore raw
In reply to Huang, Kai
Quoted 31 lines · Kai
Quoted 31 lines · Kai
Quoted 19 lines · Kai Huang
Quoted 8 lines
Quoted 5 lines

Thanks
Haitao

Hi Haitao,

Quoted 9 lines

Could you also help to test in a VM?

You will also need below patch in order to use EDECCSSA in the guest:

https://lore.kernel.org/lkml/20220727115442.464380-1-kai.huang@intel
.com/

When you create the VM, please use -cpu host.

Quoted 6 lines

Hi Haitao,

Do you have any update?

If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?

I can give ack for this, given that it is so obvious:

Quoted 9 lines
Quoted 3 lines

Would give reviewed-by if there was ucode update available, and I could test this.

I don't know whether there's ucode update available. Maybe Haitao has more information.

I talked to Haitao and I'll try to verify this in a VM.

Together with the patch to expose EDECCSSA to guest:

https://lore.kernel.org/linux-sgx/[email protected]/T/#u

I have tested both AEX-notify and EDECCSSA work in KVM guest. So:

Signature · 2 lines
Thanks,
-Kai
maintainer
#24 lore raw
In reply to Dave Hansen
Quoted 86 lines · Dave Hansen

Changes from v1:
* Make sure SGX_ATTR_ASYNC_EXIT_NOTIFY is in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.

--

Short Version:

Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

Long Version:

== SGX Attribute Background ==

The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.

As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.

The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.

For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.

== AEX Notify Background ==

"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:

Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.

Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

== The Problem ==

These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.

== The Solution ==

Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes. Instead of adding to the open-coded lists of individual attributes, add named lists of privileged (disallowed by default) and unprivileged (allowed by default) attributes. Add the AEX notify attribute as an unprivileged attribute, which will keep the kernel from rejecting enclaves with it set.

I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware (and new microcde) to test it.

Testing on bare-metal and in VMs accompanied by Tested-by's would be much appreciated. (This means you, Intel folks who actually have systems with the microcode that can do this.)

1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true

Quoted 13 lines
Cc Jarkko Sakkinen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Sean Christopherson <[email protected]>
Cc Kai Huang <[email protected]>
Cc Haitao Huang <[email protected]>

I think it would make sense to have a co-maintainer with better access to unreleased ucode patches, if anyone is willing to consider. Perhaps someone from Intel given the constraints.

This would help with features such as AEX Notify. I can work out issues with existing hardware and also make sure that the whole stack is usable (as I'm also consumer for SGX).

BR, Jarkko