Skip to contentSkip to conversation

x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

PATCH Superseded Dave Hansen 4 participants 9 messages Jul 5, 2022 – Jul 28, 2022

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Dave Hansen

author maintainer
#0 lore raw

PATCH 1/1 x86/sgx: Allow enclaves to use Asynchrounous Exit Notification

Short Version:

Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

Long Version:

== SGX Attribute Background ==

The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.

As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.

The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.

For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.

== AEX Notify Background ==

"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:

Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.

Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

== The Problem ==

These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.

== The Solution ==

Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes which allows it to be used in enclaves.

I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware to test it handy. Tested-by's would be much appreciated.

1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true

Cc Jarkko Sakkinen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
---
 arch/x86/include/asm/sgx.h | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)
arch/x86/include/asm/sgx.h +18 -7
index 3f9334ef67cd..f7328d8efd83 100644
@@ -110,17 +110,28 @@ enum sgx_miscselect {
  * %SGX_ATTR_EINITTOKENKEY:	Allow to use token signing key that is used to
  *				sign cryptographic tokens that can be passed to
  *				EINIT as an authorization to run an enclave.
+ * %SGX_ATTR_ASYNC_EXIT_NOTIFY:	Allow enclaves to be notified after an
+ *				asynchronous exit has occurred.
  */
 enum sgx_attribute {
-	SGX_ATTR_INIT		= BIT(0),
-	SGX_ATTR_DEBUG		= BIT(1),
-	SGX_ATTR_MODE64BIT	= BIT(2),
-	SGX_ATTR_PROVISIONKEY	= BIT(4),
-	SGX_ATTR_EINITTOKENKEY	= BIT(5),
-	SGX_ATTR_KSS		= BIT(7),
+	SGX_ATTR_INIT		   = BIT(0),
+	SGX_ATTR_DEBUG		   = BIT(1),
+	SGX_ATTR_MODE64BIT	   = BIT(2),
+				  /* BIT(3) is reserved */
+	SGX_ATTR_PROVISIONKEY	   = BIT(4),
+	SGX_ATTR_EINITTOKENKEY	   = BIT(5),
+				  /* BIT(6) is for CET */
+	SGX_ATTR_KSS		   = BIT(7),
+				  /* BIT(8) is reserved */
+				  /* BIT(9) is reserved */
+	SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10),
 };
 
-#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | BIT_ULL(6) | GENMASK_ULL(63, 8))
+#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | \
+				 BIT_ULL(6) | \
+				 BIT_ULL(8) | \
+				 BIT_ULL(9) | \
+				 GENMASK_ULL(63, 11))
 
 /**
  * struct sgx_secs - SGX Enclave Control Structure (SECS)
Signature · 1 line
2.34.1
maintainer
#1 lore raw
Quoted 60 lines · Dave Hansen

Short Version:

Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

Long Version:

== SGX Attribute Background ==

The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.

As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.

The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.

For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.

== AEX Notify Background ==

"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:

Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.

Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

== The Problem ==

These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.

== The Solution ==

Quoted 7 lines

Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes which allows it to be used in enclaves.

I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware to test it handy. Tested-by's would be much appreciated.

Is this available on recent ucode updates e.g. for Icelake or Geminilake?

Quoted 61 lines

1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true

Cc Jarkko Sakkinen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
---
 arch/x86/include/asm/sgx.h | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)
arch/x86/include/asm/sgx.h +18 -7
index 3f9334ef67cd..f7328d8efd83 100644
@@ -110,17 +110,28 @@ enum sgx_miscselect {
  * %SGX_ATTR_EINITTOKENKEY:	Allow to use token signing key that is used to
  *				sign cryptographic tokens that can be passed to
  *				EINIT as an authorization to run an enclave.
+ * %SGX_ATTR_ASYNC_EXIT_NOTIFY:	Allow enclaves to be notified after an
+ *				asynchronous exit has occurred.
  */
 enum sgx_attribute {
-	SGX_ATTR_INIT		= BIT(0),
-	SGX_ATTR_DEBUG		= BIT(1),
-	SGX_ATTR_MODE64BIT	= BIT(2),
-	SGX_ATTR_PROVISIONKEY	= BIT(4),
-	SGX_ATTR_EINITTOKENKEY	= BIT(5),
-	SGX_ATTR_KSS		= BIT(7),
+	SGX_ATTR_INIT		   = BIT(0),
+	SGX_ATTR_DEBUG		   = BIT(1),
+	SGX_ATTR_MODE64BIT	   = BIT(2),
+				  /* BIT(3) is reserved */
+	SGX_ATTR_PROVISIONKEY	   = BIT(4),
+	SGX_ATTR_EINITTOKENKEY	   = BIT(5),
+				  /* BIT(6) is for CET */
+	SGX_ATTR_KSS		   = BIT(7),
+				  /* BIT(8) is reserved */
+				  /* BIT(9) is reserved */
+	SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10),
 };
 
-#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | BIT_ULL(6) | GENMASK_ULL(63, 8))
+#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | \
+				 BIT_ULL(6) | \
+				 BIT_ULL(8) | \
+				 BIT_ULL(9) | \
+				 GENMASK_ULL(63, 11))
 
 /**
  * struct sgx_secs - SGX Enclave Control Structure (SECS)
Signature · 1 line
2.34.1

BR, Jarkko

maintainer
#2 lore raw
Quoted 65 lines · Jarkko Sakkinen
Quoted 64 lines · Dave Hansen

Short Version:

Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

Long Version:

== SGX Attribute Background ==

The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.

As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.

The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.

For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.

== AEX Notify Background ==

"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:

Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.

Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

== The Problem ==

These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.

== The Solution ==

Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes which allows it to be used in enclaves.

Quoted 6 lines
Quoted 3 lines

I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware to test it handy. Tested-by's would be much appreciated.

Is this available on recent ucode updates e.g. for Icelake or Geminilake?

I mean it would not take me long to upgrade our exception handling flow to this. Then I can run our full test suite on it. But this will of course require ucode update for Icelake.

AEX Notify will actually just simplify everything. We kind of simulate "AEX Notify" already with EENTER to execute in-enclave exception handler before doing actual ERESUME.

BR, Jarkko

#3 lore raw
In reply to Jarkko Sakkinen
Quoted 2 lines · Jarkko Sakkinen

Is this available on recent ucode updates e.g. for Icelake or Geminilake?

I do not believe it is currently publicly available.

It *really* needs some @intel.com Tested-by's.

#4 lore raw
In reply to Dave Hansen

Hi Dave

I think you need add this change in arch/x86/kernel/cpu/sgx/ioctl.c, inside sgx_encl_create function:

-	encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT | SGX_ATTR_KSS;
+	encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT | SGX_ATTR_KSS | SGX_ATTR_ASYNC_EXIT_NOTIFY;

Otherwise, EINIT ioctl fails with this new attribute set in enclave secs.

Thanks
Haitao

Quoted 125 lines · Dave Hansen

Short Version:

Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

Long Version:

== SGX Attribute Background ==

The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.

As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.

The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.

For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.

== AEX Notify Background ==

"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:

Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.

Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.

AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.

== The Problem ==

These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.

== The Solution ==

Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes which allows it to be used in enclaves.

I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware to test it handy. Tested-by's would be much appreciated.

1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true

Cc Jarkko Sakkinen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
---
 arch/x86/include/asm/sgx.h | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)
arch/x86/include/asm/sgx.h +0 -0
index 3f9334ef67cd..f7328d8efd83 100644
@@ -110,17 +110,28 @@ enum sgx_miscselect {
  * %SGX_ATTR_EINITTOKENKEY:	Allow to use token signing key that is used  

to
* sign cryptographic tokens that can be passed to
* EINIT as an authorization to run an enclave.
+ * %SGX_ATTR_ASYNC_EXIT_NOTIFY: Allow enclaves to be notified after an
+ * asynchronous exit has occurred.
*/
enum sgx_attribute {

arch/x86/include/asm/sgx.h +11 -7
-	SGX_ATTR_INIT		= BIT(0),
-	SGX_ATTR_DEBUG		= BIT(1),
-	SGX_ATTR_MODE64BIT	= BIT(2),
-	SGX_ATTR_PROVISIONKEY	= BIT(4),
-	SGX_ATTR_EINITTOKENKEY	= BIT(5),
-	SGX_ATTR_KSS		= BIT(7),
+	SGX_ATTR_INIT		   = BIT(0),
+	SGX_ATTR_DEBUG		   = BIT(1),
+	SGX_ATTR_MODE64BIT	   = BIT(2),
+				  /* BIT(3) is reserved */
+	SGX_ATTR_PROVISIONKEY	   = BIT(4),
+	SGX_ATTR_EINITTOKENKEY	   = BIT(5),
+				  /* BIT(6) is for CET */
+	SGX_ATTR_KSS		   = BIT(7),
+				  /* BIT(8) is reserved */
+				  /* BIT(9) is reserved */
+	SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10),
 };
-#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | BIT_ULL(6) |  

GENMASK_ULL(63, 8))

arch/x86/include/asm/sgx.h +5 -0
+#define SGX_ATTR_RESERVED_MASK	(BIT_ULL(3) | \
+				 BIT_ULL(6) | \
+				 BIT_ULL(8) | \
+				 BIT_ULL(9) | \
+				 GENMASK_ULL(63, 11))

/**
* struct sgx_secs - SGX Enclave Control Structure (SECS)

#5 lore raw
Quoted 10 lines · Haitao Huang

I think you need add this change in arch/x86/kernel/cpu/sgx/ioctl.c, inside sgx_encl_create function:

-    encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT | SGX_ATTR_KSS;
+    encl->attributes_mask = SGX_ATTR_DEBUG | SGX_ATTR_MODE64BIT | SGX_ATTR_KSS | SGX_ATTR_ASYNC_EXIT_NOTIFY;

Otherwise, EINIT ioctl fails with this new attribute set in enclave secs.

Makes sense. I'll add it there.

I also just realized we need to update arch/x86/kvm/cpuid.c as well. It keeps a complete list of guest-available SGX attributes.

#6 lore raw
In reply to Haitao Huang

Oh, and, btw... I don't have any hardware that can actually test this patch. It can't be applied until I have an actual tested-by from someone with the hardware.

#7 lore raw
Quoted 3 lines · Dave Hansen

Oh, and, btw... I don't have any hardware that can actually test this patch. It can't be applied until I have an actual tested-by from someone with the hardware.

I can ask someone in our team to test and report back results with Tested-by when you post updated patch. Thanks
Haitao

maintainer
#8 lore raw
In reply to Dave Hansen
Quoted 3 lines · Dave Hansen

Oh, and, btw... I don't have any hardware that can actually test this patch. It can't be applied until I have an actual tested-by from someone with the hardware.

... I bet that NUC7's will never get a patch

BR, Jarkko