x86/sgx: Allow enclaves to use Asynchrounous Exit Notification
Dave Hansen
maintainerPATCH v2 1/1 x86/sgx: Allow enclaves to use Asynchrounous Exit Notification
Changes from v1:
* Make sure SGX_ATTR_ASYNC_EXIT_NOTIFY is in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.
--
Short Version:
Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].
AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.
Long Version:
== SGX Attribute Background ==
The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.
As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.
The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.
For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.
== AEX Notify Background ==
"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:
Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.
Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.
AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.
== The Problem ==
These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.
== The Solution ==
Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes. Instead of adding to the open-coded lists of individual attributes, add named lists of privileged (disallowed by default) and unprivileged (allowed by default) attributes. Add the AEX notify attribute as an unprivileged attribute, which will keep the kernel from rejecting enclaves with it set.
I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware (and new microcde) to test it.
Testing on bare-metal and in VMs accompanied by Tested-by's would be much appreciated. (This means you, Intel folks who actually have systems with the microcode that can do this.)
1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true
--- arch/x86/include/asm/sgx.h | 33 ++++++++++++++++++++++++++------- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- arch/x86/kvm/cpuid.c | 4 +--- 3 files changed, 28 insertions(+), 11 deletions(-)
arch/x86/include/asm/sgx.h +26 -7
@@ -110,17 +110,36 @@ enum sgx_miscselect {* %SGX_ATTR_EINITTOKENKEY: Allow to use token signing key that is used to * sign cryptographic tokens that can be passed to * EINIT as an authorization to run an enclave. +* %SGX_ATTR_ASYNC_EXIT_NOTIFY: Allow enclaves to be notified after an +* asynchronous exit has occurred. */ enum sgx_attribute { -SGX_ATTR_INIT =BIT (0 ), -SGX_ATTR_DEBUG =BIT (1 ), -SGX_ATTR_MODE64BIT =BIT (2 ), -SGX_ATTR_PROVISIONKEY =BIT (4 ), -SGX_ATTR_EINITTOKENKEY =BIT (5 ), -SGX_ATTR_KSS =BIT (7 ), +SGX_ATTR_INIT =BIT (0 ), +SGX_ATTR_DEBUG =BIT (1 ), +SGX_ATTR_MODE64BIT =BIT (2 ), +/* BIT(3) is reserved */ +SGX_ATTR_PROVISIONKEY =BIT (4 ), +SGX_ATTR_EINITTOKENKEY =BIT (5 ), +/* BIT(6) is for CET */ +SGX_ATTR_KSS =BIT (7 ), +/* BIT(8) is reserved */ +/* BIT(9) is reserved */ +SGX_ATTR_ASYNC_EXIT_NOTIFY =BIT (10 ), }; -#define SGX_ATTR_RESERVED_MASK (BIT_ULL(3) | BIT_ULL(6) | GENMASK_ULL(63, 8)) +#define SGX_ATTR_RESERVED_MASK (BIT_ULL(3) | \ + BIT_ULL(6) | \ + BIT_ULL(8) | \ + BIT_ULL(9) | \ + GENMASK_ULL(63, 11)) + +#define SGX_ATTR_UNPRIV_MASK (SGX_ATTR_DEBUG | \ + SGX_ATTR_MODE64BIT | \ + SGX_ATTR_KSS | \ + SGX_ATTR_ASYNC_EXIT_NOTIFY) + +#define SGX_ATTR_PRIV_MASK (SGX_ATTR_PROVISIONKEY | \ + SGX_ATTR_EINITTOKENKEY)/** * struct sgx_secs - SGX Enclave Control Structure (SECS)
arch/x86/kernel/cpu/sgx/ioctl.c +1 -1
@@ -110,7 +110,7 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) encl->base = secs->base; encl->size = secs->size; encl->attributes = secs->attributes; - encl->attributes_mask =SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT |SGX_ATTR_KSS ; + encl->attributes_mask =SGX_ATTR_UNPRIV_MASK ;/* Set only after completion, as encl->lock has not been taken. */ set_bit (SGX_ENCL_CREATED , &encl->flags);
arch/x86/kvm/cpuid.c +1 -3
@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is * expected to derive it from supported XCR0. */ - entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | -SGX_ATTR_PROVISIONKEY |SGX_ATTR_EINITTOKENKEY | -SGX_ATTR_KSS ; + entry->eax &=SGX_ATTR_PRIV_MASK |SGX_ATTR_UNPRIV_MASK ; entry->ebx &=0 ;break ;/* Intel PT */
Signature · 1 line
2.34.1
Sean Christopherson
maintainerQuoted 4 lines · Dave Hansen
Changes from v1:
* Make sureSGX_ATTR_ASYNC_EXIT_NOTIFYis in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.
Heh, I was wondering if KVM needed to be updated.
arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 1025
Quoted 12 lines
arch/x86/kvm/cpuid.c +1
-3@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is * expected to derive it from supported XCR0. */ - entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | -SGX_ATTR_PROVISIONKEY |SGX_ATTR_EINITTOKENKEY | -SGX_ATTR_KSS ; + entry->eax &=SGX_ATTR_PRIV_MASK |SGX_ATTR_UNPRIV_MASK ;
It may seem like a maintenance burdern, and it is to some extent, but I think it's better for KVM to have to explicitly "enable" each flag. There is no guarantee that a new feature will not require additional KVM enabling, i.e. we want the pain of having to manually update KVM so that we get "feature X isn't virtualized" complaints and not "I upgraded my kernel and my enclaves broke" bug reports.
I don't think it's likely that attribute-based features will require additional enabling since there aren't any virtualization controls for the ENCLU side of things (ENCLU is effectively disabled by blocking ENCLS[ECREATE]), but updating KVM isn't particularly difficult so I'd rather be paranoid.
Quoted 20 lines · Sean Christopherson
arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 1025
Quoted 12 lines · Dave Hansen
arch/x86/kvm/cpuid.c +1
-3@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is * expected to derive it from supported XCR0. */ - entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | -SGX_ATTR_PROVISIONKEY |SGX_ATTR_EINITTOKENKEY | -SGX_ATTR_KSS ; + entry->eax &=SGX_ATTR_PRIV_MASK |SGX_ATTR_UNPRIV_MASK ;It may seem like a maintenance burdern, and it is to some extent, but I think it's better for KVM to have to explicitly "enable" each flag. There is no guarantee that a new feature will not require additional KVM enabling, i.e. we want the pain of having to manually update KVM so that we get "feature X isn't virtualized" complaints and not "I upgraded my kernel and my enclaves broke" bug reports.
Quoted 4 lines
I don't think it's likely that attribute-based features will require additional enabling since there aren't any virtualization controls for the ENCLU side of things (ENCLU is effectively disabled by blocking ENCLS[ECREATE]), but updating KVM isn't particularly difficult so I'd rather be paranoid.
How about something where KVM gets to keep a discrete mask, but where it's at least defined next to the attributes, something like:
/*
* These attributes will be advertised to KVM guests as being
* available. This includes privileged attributes. Only add
* to this list when host-side KVM does not require additional
* enabling for the attribute.
*/
#define SGX_ATTR_KVM_MASK (SGX_ATTR_DEBUG | \
SGX_ATTR_MODE64BIT | \
SGX_ATTR_PROVISIONKEY | \
SGX_ATTR_EINITTOKENKEY | \
SGX_ATTR_KSS | \ SGX_ATTR_ASYNC_EXIT_NOTIFY)
That at least has a *chance* of someone seeing it who goes to add a new attribute.
Sean Christopherson
maintainerQuoted 40 lines · Dave Hansen
Quoted 20 lines · Sean Christopherson
arch/x86/kvm/cpuid.c static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function) · line 1025
Quoted 12 lines · Dave Hansen
arch/x86/kvm/cpuid.c +1
-3@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is * expected to derive it from supported XCR0. */ - entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | -SGX_ATTR_PROVISIONKEY |SGX_ATTR_EINITTOKENKEY | -SGX_ATTR_KSS ; + entry->eax &=SGX_ATTR_PRIV_MASK |SGX_ATTR_UNPRIV_MASK ;It may seem like a maintenance burdern, and it is to some extent, but I think it's better for KVM to have to explicitly "enable" each flag. There is no guarantee that a new feature will not require additional KVM enabling, i.e. we want the pain of having to manually update KVM so that we get "feature X isn't virtualized" complaints and not "I upgraded my kernel and my enclaves broke" bug reports.
Quoted 4 lines
I don't think it's likely that attribute-based features will require additional enabling since there aren't any virtualization controls for the ENCLU side of things (ENCLU is effectively disabled by blocking ENCLS[ECREATE]), but updating KVM isn't particularly difficult so I'd rather be paranoid.
How about something where KVM gets to keep a discrete mask, but where it's at least defined next to the attributes, something like:
/*
* These attributes will be advertised to KVM guests as being
* available. This includes privileged attributes. Only add
* to this list when host-side KVM does not require additional
* enabling for the attribute.
*/#define SGX_ATTR_KVM_MASK (SGX_ATTR_DEBUG | \ SGX_ATTR_MODE64BIT | \ SGX_ATTR_PROVISIONKEY | \ SGX_ATTR_EINITTOKENKEY | \ SGX_ATTR_KSS | \
Quoted 4 lines
SGX_ATTR_ASYNC_EXIT_NOTIFY)That at least has a *chance* of someone seeing it who goes to add a new attribute.
Hmm, what if we enforce it in code with a compile-time assert? That will make it even harder to screw things up, and it also avoids a scenario where someone extends SGX_ATTR_KVM_MASK without getting approval from KVM folks. And conversely, KVM won't need to touch SGX files if there's ever a need to tweak KVM behavior.
/*
* Index 1: SECS.ATTRIBUTES. ATTRIBUTES are restricted a la
* feature flags. Advertise all supported flags, including
* privileged attributes that require explicit opt-in from
* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is
* expected to derive it from supported XCR0.
*/
#define KVM_SGX_ATTR_ALLOWED_MASK (SGX_ATTR_DEBUG | \ SGX_ATTR_MODE64BIT | \ SGX_ATTR_PROVISIONKEY | \ SGX_ATTR_EINITTOKENKEY | \ SGX_ATTR_KSS | \ SGX_ATTR_ASYNC_EXIT_NOTIFY)
#define KVM_SGX_ATTR_DENIED_MASK (0)
/* * Assert that KVM explicitly allows or denies exposing all * features, i.e. detect attempts to add kernel support without * also updating KVM. */ BUILD_BUG_ON((KVM_SGX_ATTR_ALLOWED_MASK | KVM_SGX_ATTR_DENIED_MASK) != (SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK)); entry->eax &= KVM_SGX_ATTR_ALLOWED_MASK; entry->ebx &= 0; break;
Quoted 7 lines · Sean Christopherson
/* * Assert that KVM explicitly allows or denies exposing all * features, i.e. detect attempts to add kernel support without * also updating KVM. */ BUILD_BUG_ON((KVM_SGX_ATTR_ALLOWED_MASK | KVM_SGX_ATTR_DENIED_MASK) != (SGX_ATTR_PRIV_MASK | SGX_ATTR_UNPRIV_MASK));
Looks good to me. I'll do that for the next version. Thanks!
Quoted 14 lines · Dave Hansen
arch/x86/kvm/cpuid.c +1
-4@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is * expected to derive it from supported XCR0. */ - entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | - SGX_ATTR_PROVISIONKEY | SGX_ATTR_EINITTOKENKEY | - SGX_ATTR_KSS; + entry->eax &=SGX_ATTR_PRIV_MASK |SGX_ATTR_UNPRIV_MASK ; entry->ebx &= 0; break; /* Intel PT */ --
Quoted 1 line
2.34.1
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new SGX_ATTR_ASYNC_EXIT_NOTIFY bit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?
(quoted diff) +0 -0
b/arch/x86/include/asm/cpufeatures.h
index 6466a58b9cff..d2ebb38b31e7 100644
arch/x86/include/asm/cpufeatures.h +0 -0
@@ -296,6 +296,7 @@#define X86_FEATURE_PER_THREAD_MBA (11*32+ 7)/* "" Per-thread Memory
Bandwidth Allocation */
#define X86_FEATURE_SGX1 (11*32+ 8) /* "" Basic SGX */
#define X86_FEATURE_SGX2 (11*32+ 9) /* "" SGX Enclave Dynamic
Memory Management (EDMM) */
+#define X86_FEATURE_SGX_EDECCSSA (11*32+10) /* "" SGX EDECCSSA user leaf
function */
/* Intel-defined CPU features, CPUID level 0x00000007:1 (EAX), word 12 */
#define X86_FEATURE_AVX_VNNI (12*32+ 4) /* AVX VNNI instructions */
arch/x86/kvm/cpuid.c +1 -1
@@ -612,7 +612,7 @@ void kvm_set_cpu_caps(void) );kvm_cpu_cap_init_scattered (CPUID_12_EAX , -SF (SGX1 ) |SF (SGX2 ) +SF (SGX1 ) |SF (SGX2 ) |SF (SGX_EDECCSSA ) ); kvm_cpu_cap_mask(CPUID_8000_0001_ECX,
arch/x86/kvm/reverse_cpuid.h +1 -0
@@ -23,6 +23,7 @@ enum kvm_only_cpuid_leafs {/* Intel-defined SGX sub-features, CPUID level 0x12 (EAX). */ #define KVM_X86_FEATURE_SGX1 KVM_X86_FEATURE(CPUID_12_EAX, 0)#define KVM_X86_FEATURE_SGX2 KVM_X86_FEATURE(CPUID_12_EAX, 1) +#define KVM_X86_FEATURE_SGX_EDECCSSA KVM_X86_FEATURE(CPUID_12_EAX, 11)struct cpuid_reg {u32 function;
@@ -78,6 +79,8 @@ static __always_inline u32 __feature_translate(int
x86_feature)
return KVM_X86_FEATURE_SGX1;
else if (x86_feature == X86_FEATURE_SGX2)
return KVM_X86_FEATURE_SGX2;+ else if (x86_feature == X86_FEATURE_SGX_EDECCSSA)
+ return KVM_X86_FEATURE_SGX_EDECCSSA;
return x86_feature;
}
Quoted 4 lines · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?
Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:
Quoted 2 lines
EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...
If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.
Oh, and the one-liner patch that I was promised for enabling this is getting a _wee_ bit longer than one line.
Sean Christopherson
maintainerQuoted 11 lines · Dave Hansen
Quoted 4 lines · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:
Quoted 2 lines
EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...
Yeah, it's enumerated separately.
Quoted 5 lines
If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.
Oh, and the one-liner patch that I was promised for enabling this is getting a _wee_ bit longer than one line.
Heh, fool me once...
Quoted 6 lines · Dave Hansen
Quoted 4 lines · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?
Quoted 9 lines
Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:
Quoted 2 lines
EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...
If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.
They are enumerated separately, but looks in practice the notify handler will use it to switch back to the correct/targeted CSSA to continue to run normally after handling the exit notify. This is my understanding of the "facilitate" mean in the spec.
Btw, in real hardware I think the two should come together, meaning no real hardware will only support one.
Haitao, could you give us more information?
Signature · 2 lines
Thanks, -Kai
Quoted 25 lines · Kai Huang
Quoted 9 lines · Dave Hansen
Quoted 1 line · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be
used
Quoted 2 lines
together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think we shouldalso
Quoted 1 line
advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below
(untested)?
Quoted 9 lines
Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:
Quoted 2 lines
EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...
If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.
They are enumerated separately, but looks in practice the notify handler will use it to switch back to the correct/targeted CSSA to continue to run normally after handling the exit notify. This is my understanding of the "facilitate" mean in the spec.
Quoted 5 lines
Btw, in real hardware I think the two should come together, meaning no real hardware will only support one.
Haitao, could you give us more information?
You are right. They are enumerated separately and HW should come with both or neither. My understanding it is also possible for enclaves choose not to receive AEX notify but still use EDECCSSA.
Thanks
Haitao
Quoted 34 lines · Haitao Huang
Quoted 32 lines · Kai Huang
Quoted 9 lines · Dave Hansen
Quoted 1 line · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be
used
Quoted 2 lines
together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think we shouldalso
Quoted 1 line
advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below
(untested)?
Quoted 9 lines
Sounds like a great follow-on patch! It doesn't seem truly functionally required from the spec:
Quoted 2 lines
EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...
If that's wrong or imprecise, I'd love to hear more about it and also about how the spec will be updated.
They are enumerated separately, but looks in practice the notify handler will
use it to switch back to the correct/targeted CSSA to continue to run normally
after handling the exit notify. This is my understanding of the "facilitate"
mean in the spec.Btw, in real hardware I think the two should come together, meaning no real
hardware will only support one.
Quoted 8 lines
Quoted 2 lines
Haitao, could you give us more information?
You are right. They are enumerated separately and HW should come with both or neither.
My understanding it is also possible for enclaves choose not to receive AEX notify
but still use EDECCSSA.
What is the use case of using EDECCSSA w/o using AEX notify?
If I understand correctly EDECCSSA effectively switches to another thread (using the previous SSA, which is the context of another TCS thread if I understand correctly). Won't this cause problem?
It probably makes sense to use only AEX notify w/o using EDECCSSA though, but this should be the case that the enclave detects serious attack and don't want to continue to run normally. In another words, it is enclave's choice, but hardware should always support both AEX notify and EDECCSSA.
Signature · 2 lines
Thanks, -Kai
Quoted 48 lines · Kai Huang
Quoted 38 lines · Haitao Huang
Quoted 3 lines · Kai Huang
Quoted 2 lines · Dave Hansen
Quoted 1 line · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should
be
Quoted 8 lines
Quoted 8 lines
used
Quoted 2 lines
together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think we shouldalso
Quoted 1 line
advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below
(untested)?
Sounds like a great follow-on patch! It doesn't seem truly
functionally
Quoted 6 lines
Quoted 6 lines
required from the spec:
Quoted 2 lines
EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...
If that's wrong or imprecise, I'd love to hear more about it and
also
Quoted 4 lines
Quoted 2 lines
about how the spec will be updated.
They are enumerated separately, but looks in practice the notify
handler
Quoted 11 lines
will
use it to switch back to the correct/targeted CSSA to continue to run normally
after handling the exit notify. This is my understanding of the "facilitate"
mean in the spec.Btw, in real hardware I think the two should come together, meaning no real
hardware will only support one.Quoted 8 lines
Quoted 2 lines
Haitao, could you give us more information?
You are right. They are enumerated separately and HW should come with both or neither.
My understanding it is also possible for enclaves choose not to receive AEX notify
but still use EDECCSSA.
Quoted 3 lines
What is the use case of using EDECCSSA w/o using AEX notify? If I understand correctly EDECCSSA effectively switches to another thread (using the previous SSA, which is the context of another TCS thread if I understand
correctly). Won't this cause problem?
No. Decrementing CSSA is equivalent to popping stack frames, not switching threads.
In some cases such as so-called "first stage" exception handling, one could pop CSSA back to the previous after resetting CPU context and stack frame appropriate to the "second stage" or "real" exception handling routine, then jump to the handler directly. This could improve exception handling performance by saving an EEXIT/ERESUME trip.
Quoted 5 lines
It probably makes sense to use only AEX notify w/o using EDECCSSA though, but this should be the case that the enclave detects serious attack and don't want to continue to run normally. In another words, it is enclave's choice, but hardware should always support both AEX notify and EDECCSSA.
Quoted 59 lines · Haitao Huang
Quoted 50 lines · Kai Huang
Quoted 39 lines · Haitao Huang
Quoted 3 lines · Kai Huang
Quoted 2 lines · Dave Hansen
Quoted 1 line · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should
be
Quoted 8 lines
Quoted 8 lines
used
Quoted 2 lines
together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think we shouldalso
Quoted 1 line
advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below
(untested)?
Sounds like a great follow-on patch! It doesn't seem truly
functionally
Quoted 6 lines
Quoted 6 lines
required from the spec:
Quoted 2 lines
EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification handling...
If that's wrong or imprecise, I'd love to hear more about it and
also
Quoted 4 lines
Quoted 2 lines
about how the spec will be updated.
They are enumerated separately, but looks in practice the notify
handler
Quoted 11 lines
will
use it to switch back to the correct/targeted CSSA to continue to run normally
after handling the exit notify. This is my understanding of the "facilitate"
mean in the spec.Btw, in real hardware I think the two should come together, meaning no real
hardware will only support one.Quoted 9 lines
Quoted 2 lines
Haitao, could you give us more information?
You are right. They are enumerated separately and HW should come with both
or neither.
My understanding it is also possible for enclaves choose not to receive AEX notify
but still use EDECCSSA.Quoted 6 lines
What is the use case of using EDECCSSA w/o using AEX notify? If I understand correctly EDECCSSA effectively switches to another thread (using
the previous SSA, which is the context of another TCS thread if I understand
correctly). Won't this cause problem?
Quoted 9 lines
No. Decrementing CSSA is equivalent to popping stack frames, not switching threads.
In some cases such as so-called "first stage" exception handling, one could pop CSSA back to the previous after resetting CPU context and stack frame appropriate to the "second stage" or "real" exception handling routine, then jump to the handler directly. This could improve exception handling performance by saving an EEXIT/ERESUME trip.
Looking at the AEX-notify spec again, EDECCSSA does below:
(* At this point, the instruction is guaranteed to complete *) CR_TCS_PA.CSSA := CR_TCS_PA.CSSA - 1;CR_GPR_PA := Physical_Address(DS:TMP_GPR);
It doens't reset the RIP to CR_GPA_PA.RIP so looks yes you are right. It only "popping the stack frame" but doesn't switch thread.
But the pseudo code of EDECCSSA only updates the CR_TCS_PA and CR_GPR_PA registers (forget about XSAVE not), but doesn't manually updating the actual CPU registers such as GPRs. Are the actual CPU registers updated automatically when CR_xx are updated?
Signature · 2 lines
Thanks, -Kai
Quoted 84 lines · Kai Huang
Quoted 67 lines · Haitao Huang
Quoted 7 lines · Kai Huang
Quoted 6 lines · Haitao Huang
Quoted 3 lines · Kai Huang
Quoted 2 lines · Dave Hansen
Quoted 1 line · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA]
should
Quoted 4 lines
Quoted 4 lines
be
Quoted 3 lines
Quoted 3 lines
used
Quoted 2 lines
together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think weshould
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
also
Quoted 1 line
advertise the ENCLU[EDECCSSA] support in guest's CPUID, like
below
Quoted 8 lines
Quoted 8 lines
Quoted 3 lines
Quoted 3 lines
(untested)?
Sounds like a great follow-on patch! It doesn't seem truly
functionally
Quoted 4 lines
Quoted 4 lines
required from the spec:
Quoted 2 lines
EDECCSSA is a new Intel SGX user leaf function (ENCLU[EDECCSSA]) that can facilitate AEX notification
handling...
Quoted 10 lines
Quoted 10 lines
Quoted 2 lines
Quoted 2 lines
If that's wrong or imprecise, I'd love to hear more about it and
also
Quoted 4 lines
Quoted 2 lines
about how the spec will be updated.
They are enumerated separately, but looks in practice the notify
handler
Quoted 2 lines
will
use it to switch back to the correct/targeted CSSA to continue torun
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
normally
after handling the exit notify. This is my understanding of the "facilitate"
mean in the spec.Btw, in real hardware I think the two should come together,
meaning no
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
real
hardware will only support one.Haitao, could you give us more information?
You are right. They are enumerated separately and HW should come
with
Quoted 3 lines
Quoted 3 lines
both
or neither.
My understanding it is also possible for enclaves choose not toreceive
Quoted 10 lines
Quoted 3 lines
AEX notify
but still use EDECCSSA.What is the use case of using EDECCSSA w/o using AEX notify? If I understand correctly EDECCSSA effectively switches to another thread (using
the previous SSA, which is the context of another TCS thread if I understand
correctly). Won't this cause problem?Quoted 9 lines
No. Decrementing CSSA is equivalent to popping stack frames, not switching threads.
In some cases such as so-called "first stage" exception handling, one could pop CSSA back to the previous after resetting CPU context and stack frame appropriate to the "second stage" or "real" exception handling routine, then jump to the handler directly. This could improve exception handling performance by saving an EEXIT/ERESUME trip.Looking at the AEX-notify spec again, EDECCSSA does below:
(* At this point, the instruction is guaranteed to complete *) CR_TCS_PA.CSSA := CR_TCS_PA.CSSA - 1;
CR_GPR_PA:=Physical_Address(DS:TMP_GPR);
Quoted 8 lines
It doens't reset the RIP to CR_GPA_PA.RIP so looks yes you are right. It only "popping the stack frame" but doesn't switch thread.
But the pseudo code of EDECCSSA only updates the
CR_TCS_PAandCR_GPR_PAregisters (forget about XSAVE not), but doesn't manually updating the actual CPU registers such as GPRs. Are the actual CPU registers updated automatically whenCR_xxare updated?
No, the enclave code is supposed to do that. Here is are a few more details on the flow I mentioned.
On any AEX event, CPU saves states including GPR/XSave into SSA[0]. When AEX-notify is turned off, for enclaves to handle exceptions occurred inside enclave, user space must do EENTER with the same TCS on which the exception occurred. EENTER would give a clean slate of GPR and SSA[1] becomes active for next AEX. It's enclave's responsibility to save GPR/XSave states in SSA[0] to some place (e.g., stack), then EDECCSSA, then jump to the "second stage" handler. (Note now SSA[0] is reactivated and ready if another AEX occurs). The second stage handler then fixes the situation that caused the original AEX, restore CPU context from the saved SSA[0] states, jump back to original place where exception happened.
Quoted 4 lines · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?
Kai, would you care to send a new version of this with a proper SoB and changelog integrating what you've learned since posting it? It can be merged along with AEX notify itself.
Quoted 9 lines · Dave Hansen
Quoted 4 lines · Kai Huang
Did a quick look at the spec. It appears ENCLU[EDECCSSA] should be used together with AEX-notify. So besides advertising the new
SGX_ATTR_ASYNC_EXIT_NOTIFYbit to the KVM guest, I think we should also advertise the ENCLU[EDECCSSA] support in guest's CPUID, like below (untested)?Kai, would you care to send a new version of this with a proper SoB and changelog integrating what you've learned since posting it? It can be merged along with AEX notify itself.
Sure will do today.
Signature · 2 lines
Thanks, -Kai
Quoted 184 lines · Dave Hansen
Changes from v1:
* Make sureSGX_ATTR_ASYNC_EXIT_NOTIFYis in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.--
Short Version:
Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].
AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.
Long Version:
== SGX Attribute Background ==
The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.
As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.
The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.
For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.
== AEX Notify Background ==
"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:
Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.
Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.
AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.
== The Problem ==
These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.
== The Solution ==
Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes. Instead of adding to the open-coded lists of individual attributes, add named lists of privileged (disallowed by default) and unprivileged (allowed by default) attributes. Add the AEX notify attribute as an unprivileged attribute, which will keep the kernel from rejecting enclaves with it set.
I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware (and new microcde) to test it.
Testing on bare-metal and in VMs accompanied by Tested-by's would be much appreciated. (This means you, Intel folks who actually have systems with the microcode that can do this.)
1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=trueSigned-off-by Dave Hansen <[email protected]>Cc Jarkko Sakkinen <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Sean Christopherson <[email protected]>Cc Kai Huang <[email protected]>Cc Haitao Huang <[email protected]>--- arch/x86/include/asm/sgx.h | 33 ++++++++++++++++++++++++++------- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- arch/x86/kvm/cpuid.c | 4 +--- 3 files changed, 28 insertions(+), 11 deletions(-)arch/x86/include/asm/sgx.h +26
-7@@ -110,17 +110,36 @@ enum sgx_miscselect {* %SGX_ATTR_EINITTOKENKEY: Allow to use token signing key that is used to * sign cryptographic tokens that can be passed to * EINIT as an authorization to run an enclave. +* %SGX_ATTR_ASYNC_EXIT_NOTIFY: Allow enclaves to be notified after an +* asynchronous exit has occurred. */ enum sgx_attribute { -SGX_ATTR_INIT =BIT (0 ), -SGX_ATTR_DEBUG =BIT (1 ), -SGX_ATTR_MODE64BIT =BIT (2 ), -SGX_ATTR_PROVISIONKEY =BIT (4 ), -SGX_ATTR_EINITTOKENKEY =BIT (5 ), -SGX_ATTR_KSS =BIT (7 ), +SGX_ATTR_INIT =BIT (0 ), +SGX_ATTR_DEBUG =BIT (1 ), +SGX_ATTR_MODE64BIT =BIT (2 ), +/* BIT(3) is reserved */ +SGX_ATTR_PROVISIONKEY =BIT (4 ), +SGX_ATTR_EINITTOKENKEY =BIT (5 ), +/* BIT(6) is for CET */ +SGX_ATTR_KSS =BIT (7 ), +/* BIT(8) is reserved */ +/* BIT(9) is reserved */ +SGX_ATTR_ASYNC_EXIT_NOTIFY =BIT (10 ), }; -#define SGX_ATTR_RESERVED_MASK (BIT_ULL(3) | BIT_ULL(6) | GENMASK_ULL(63, 8)) +#define SGX_ATTR_RESERVED_MASK (BIT_ULL(3) | \ + BIT_ULL(6) | \ + BIT_ULL(8) | \ + BIT_ULL(9) | \ + GENMASK_ULL(63, 11)) + +#define SGX_ATTR_UNPRIV_MASK (SGX_ATTR_DEBUG | \ + SGX_ATTR_MODE64BIT | \ + SGX_ATTR_KSS | \ + SGX_ATTR_ASYNC_EXIT_NOTIFY) + +#define SGX_ATTR_PRIV_MASK (SGX_ATTR_PROVISIONKEY | \ + SGX_ATTR_EINITTOKENKEY)/** * struct sgx_secs - SGX Enclave Control Structure (SECS) arch/x86/kernel/cpu/sgx/ioctl.c +1
-1@@ -110,7 +110,7 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) encl->base = secs->base; encl->size = secs->size; encl->attributes = secs->attributes; - encl->attributes_mask =SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT |SGX_ATTR_KSS ; + encl->attributes_mask =SGX_ATTR_UNPRIV_MASK ;/* Set only after completion, as encl->lock has not been taken. */ set_bit (SGX_ENCL_CREATED , &encl->flags);arch/x86/kvm/cpuid.c +1
-3@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(struct kvm_cpuid_array *array, u32 function)* userspace. ATTRIBUTES.XFRM is not adjusted as userspace is * expected to derive it from supported XCR0. */ - entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | -SGX_ATTR_PROVISIONKEY |SGX_ATTR_EINITTOKENKEY | -SGX_ATTR_KSS ; + entry->eax &=SGX_ATTR_PRIV_MASK |SGX_ATTR_UNPRIV_MASK ; entry->ebx &=0 ;break ;/* Intel PT */ Signature · 1 line
2.34.1
BR, Jarkkko
Quoted 182 lines · Dave Hansen
Changes from v1:
* Make sureSGX_ATTR_ASYNC_EXIT_NOTIFYis in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.--
Short Version:
Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].
AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.
Long Version:
== SGX Attribute Background ==
The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.
As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.
The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.
For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.
== AEX Notify Background ==
"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:
Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.
Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.
AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.
== The Problem ==
These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.
== The Solution ==
Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes. Instead of adding to the open-coded lists of individual attributes, add named lists of privileged (disallowed by default) and unprivileged (allowed by default) attributes. Add the AEX notify attribute as an unprivileged attribute, which will keep the kernel from rejecting enclaves with it set.
I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware (and new microcde) to test it.
Testing on bare-metal and in VMs accompanied by Tested-by's would be much appreciated. (This means you, Intel folks who actually have systems with the microcode that can do this.)
1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=trueSigned-off-by Dave Hansen <[email protected]>Cc Jarkko Sakkinen <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Sean Christopherson <[email protected]>Cc Kai Huang <[email protected]>Cc Haitao Huang <[email protected]>--- arch/x86/include/asm/sgx.h | 33 ++++++++++++++++++++++++++------- arch/x86/kernel/cpu/sgx/ioctl.c | 2 +- arch/x86/kvm/cpuid.c | 4 +--- 3 files changed, 28 insertions(+), 11 deletions(-)arch/x86/include/asm/sgx.h +0
-0@@ -110,17 +110,36 @@ enum sgx_miscselect {* %SGX_ATTR_EINITTOKENKEY: Allow to use token signing key that is used to
* sign cryptographic tokens that can be passed to
* EINIT as an authorization to run an enclave.
+ * %SGX_ATTR_ASYNC_EXIT_NOTIFY: Allow enclaves to be notified after an
+ * asynchronous exit has occurred.
*/
enumsgx_attribute{arch/x86/include/asm/sgx.h +11
-7-SGX_ATTR_INIT =BIT (0 ), -SGX_ATTR_DEBUG =BIT (1 ), -SGX_ATTR_MODE64BIT =BIT (2 ), -SGX_ATTR_PROVISIONKEY =BIT (4 ), -SGX_ATTR_EINITTOKENKEY =BIT (5 ), - SGX_ATTR_KSS = BIT(7), +SGX_ATTR_INIT =BIT (0 ), +SGX_ATTR_DEBUG =BIT (1 ), +SGX_ATTR_MODE64BIT =BIT (2 ), +/* BIT(3) is reserved */ +SGX_ATTR_PROVISIONKEY =BIT (4 ), +SGX_ATTR_EINITTOKENKEY =BIT (5 ), +/* BIT(6) is for CET */ +SGX_ATTR_KSS =BIT (7 ), +/* BIT(8) is reserved */ +/* BIT(9) is reserved */ + SGX_ATTR_ASYNC_EXIT_NOTIFY = BIT(10), }; -#define SGX_ATTR_RESERVED_MASK (BIT_ULL(3) | BIT_ULL(6) |
GENMASK_ULL(63, 8))arch/x86/include/asm/sgx.h +13
-0+#define SGX_ATTR_RESERVED_MASK (BIT_ULL(3) | \ + BIT_ULL(6) | \ + BIT_ULL(8) | \ + BIT_ULL(9) | \ + GENMASK_ULL(63, 11)) + +#define SGX_ATTR_UNPRIV_MASK (SGX_ATTR_DEBUG | \ + SGX_ATTR_MODE64BIT | \ + SGX_ATTR_KSS | \ + SGX_ATTR_ASYNC_EXIT_NOTIFY) + +#define SGX_ATTR_PRIV_MASK (SGX_ATTR_PROVISIONKEY | \ + SGX_ATTR_EINITTOKENKEY)/**
* structsgx_secs- SGX Enclave Control Structure (SECS)arch/x86/kernel/cpu/sgx/ioctl.c +1
-1@@ -110,7 +110,7 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) encl->base = secs->base; encl->size = secs->size; encl->attributes = secs->attributes; - encl->attributes_mask =SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT |SGX_ATTR_KSS ; + encl->attributes_mask =SGX_ATTR_UNPRIV_MASK ;/* Set only after completion, as encl->lock has not been taken. */ arch/x86/kernel/cpu/sgx/ioctl.c +0
-0set_bit (SGX_ENCL_CREATED , &encl->flags);arch/x86/kvm/cpuid.c +1
-3@@ -1022,9 +1022,7 @@ static inline int __do_cpuid_func(structkvm_cpuid_array *array, u32 function) * userspace. ATTRIBUTES.XFRM is not adjusted as userspace is * expected to derive it from supported XCR0. */ - entry->eax &=SGX_ATTR_DEBUG |SGX_ATTR_MODE64BIT | -SGX_ATTR_PROVISIONKEY |SGX_ATTR_EINITTOKENKEY | -SGX_ATTR_KSS ; + entry->eax &=SGX_ATTR_PRIV_MASK |SGX_ATTR_UNPRIV_MASK ; entry->ebx &=0 ;break ;
Quoted 1 line
/* Intel PT */
Thanks
Haitao
Quoted 5 lines
Tested-by Haitao Huang <[email protected]>Thanks
Haitao
Hi Haitao,
Could you also help to test in a VM?
You will also need below patch in order to use EDECCSSA in the guest:
https://lore.kernel.org/lkml/[email protected]/
When you create the VM, please use -cpu host.
Signature · 2 lines
Thanks, -Kai
Quoted 8 lines · Kai Huang
Quoted 5 lines
Tested-by Haitao Huang <[email protected]>Thanks
HaitaoHi Haitao,
Quoted 8 lines
Could you also help to test in a VM?
You will also need below patch in order to use EDECCSSA in the guest:
https://lore.kernel.org/lkml/[email protected]/
When you create the VM, please use -cpu host.
Hi Haitao,
Do you have any update?
If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?
Signature · 2 lines
Thanks, -Kai
Quoted 18 lines · Kai
Quoted 8 lines · Kai Huang
Quoted 5 lines
Tested-by Haitao Huang <[email protected]>Thanks
HaitaoHi Haitao,
Quoted 8 lines
Could you also help to test in a VM?
You will also need below patch in order to use EDECCSSA in the guest:
https://lore.kernel.org/lkml/[email protected]/
When you create the VM, please use -cpu host.
Quoted 6 lines
Hi Haitao,
Do you have any update?
If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?
I can give ack for this, given that it is so obvious:
Would give reviewed-by if there was ucode update available, and I could test this.
BR, Jarkko
Quoted 27 lines · Kai
Quoted 19 lines · Kai Huang
Quoted 8 lines
Quoted 5 lines
Tested-by Haitao Huang <[email protected]>Thanks
HaitaoHi Haitao,
Quoted 9 lines
Could you also help to test in a VM?
You will also need below patch in order to use EDECCSSA in the guest:
https://lore.kernel.org/lkml/20220727115442.464380-1-kai.huang@intel
.com/When you create the VM, please use -cpu host.
Quoted 6 lines
Hi Haitao,
Do you have any update?
If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?
Quoted 7 lines
I can give ack for this, given that it is so obvious:
Acked-by Jarkko Sakkinen <[email protected]>Would give reviewed-by if there was ucode update available, and I could test this.
I don't know whether there's ucode update available. Maybe Haitao has more information.
I talked to Haitao and I'll try to verify this in a VM.
Thanks,
-Kai
[email protected]
maintainerQuoted 35 lines · Kai
Quoted 27 lines · Kai
Quoted 19 lines · Kai Huang
Quoted 8 lines
Quoted 5 lines
Tested-by Haitao Huang <[email protected]>Thanks
HaitaoHi Haitao,
Quoted 9 lines
Could you also help to test in a VM?
You will also need below patch in order to use EDECCSSA in the guest:
https://lore.kernel.org/lkml/20220727115442.464380-1-kai.huang@intel
.com/When you create the VM, please use -cpu host.
Quoted 6 lines
Hi Haitao,
Do you have any update?
If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?
Quoted 7 lines
I can give ack for this, given that it is so obvious:
Acked-by Jarkko Sakkinen <[email protected]>Would give reviewed-by if there was ucode update available, and I could test this.
Quoted 6 lines
I don't know whether there's ucode update available. Maybe Haitao has more information.
I talked to Haitao and I'll try to verify this in a VM.
Thanks,
-Kai
There's no release for Icelake existing yet with AEX Notify.
BR, Jarkko
Huang, Kai
Tested-byQuoted 31 lines · Kai
Quoted 31 lines · Kai
Quoted 19 lines · Kai Huang
Quoted 8 lines
Quoted 5 lines
Tested-by Haitao Huang <[email protected]>Thanks
HaitaoHi Haitao,
Quoted 9 lines
Could you also help to test in a VM?
You will also need below patch in order to use EDECCSSA in the guest:
https://lore.kernel.org/lkml/20220727115442.464380-1-kai.huang@intel
.com/When you create the VM, please use -cpu host.
Quoted 6 lines
Hi Haitao,
Do you have any update?
If it's not easy for you to verify in VM, could you let me know how to set up the testing machine so I can have a try?
I can give ack for this, given that it is so obvious:
Acked-by Jarkko Sakkinen <[email protected]>
Quoted 9 lines
Quoted 3 lines
Would give reviewed-by if there was ucode update available, and I could test this.
I don't know whether there's ucode update available. Maybe Haitao has more information.
I talked to Haitao and I'll try to verify this in a VM.
Together with the patch to expose EDECCSSA to guest:
https://lore.kernel.org/linux-sgx/[email protected]/T/#u
I have tested both AEX-notify and EDECCSSA work in KVM guest. So:
Signature · 2 lines
Thanks, -Kai
Jarkko Sakkinen
maintainerQuoted 86 lines · Dave Hansen
Changes from v1:
* Make sureSGX_ATTR_ASYNC_EXIT_NOTIFYis in the masks that are used at bare-metal enclave initialization and that enumerates available attributes to KVM guests.--
Short Version:
Allow enclaves to use the new Asynchronous EXit (AEX) notification mechanism. This mechanism lets enclaves run a handler after an AEX event. These handlers can run mitigations for things like SGX-Step[1].
AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.
Long Version:
== SGX Attribute Background ==
The SGX architecture includes a list of SGX "attributes". These attributes ensure consistency and transparency around specific enclave features.
As a simple example, the "DEBUG" attribute allows an enclave to be debugged, but also destroys virtually all of SGX security. Using attributes, enclaves can know that they are being debugged. Attributes also affect enclave attestation so an enclave can, for instance, be denied access to secrets while it is being debugged.
The kernel keeps a list of known attributes and will only initialize enclaves that use a known set of attributes. This kernel policy eliminates the chance that a new SGX attribute could cause undesired effects.
For example, imagine a new attribute was added called "PROVISIONKEY2" that provided similar functionality to "PROVISIIONKEY". A kernel policy that allowed indiscriminate use of unknown attributes and thus PROVISIONKEY2 would undermine the existing kernel policy which limits use of PROVISIONKEY enclaves.
== AEX Notify Background ==
"Intel Architecture Instruction Set Extensions and Future Features - Version 45" is out[2]. There is a new chapter:
Asynchronous Enclave Exit Notify and the EDECCSSA User Leaf Function.
Enclaves exit can be either synchronous and consensual (EEXIT for instance) or asynchronous (on an interrupt or fault). The asynchronous ones can evidently be exploited to single step enclaves[1], on top of which other naughty things can be built.
AEX Notify will be made available both on upcoming processors and on some older processors through microcode updates.
== The Problem ==
These attacks are currently entirely opaque to the enclave since the hardware does the save/restore under the covers. The Asynchronous Enclave Exit Notify (AEX Notify) mechanism provides enclaves an ability to detect and mitigate potential exposure to these kinds of attacks.
== The Solution ==
Define the new attribute value for AEX Notification. Ensure the attribute is cleared from the list reserved attributes. Instead of adding to the open-coded lists of individual attributes, add named lists of privileged (disallowed by default) and unprivileged (allowed by default) attributes. Add the AEX notify attribute as an unprivileged attribute, which will keep the kernel from rejecting enclaves with it set.
I just built this and ran it to make sure there were no obvious regressions since I do not have the hardware (and new microcde) to test it.
Testing on bare-metal and in VMs accompanied by Tested-by's would be much appreciated. (This means you, Intel folks who actually have systems with the microcode that can do this.)
1. https://github.com/jovanbulck/sgx-step
2. https://cdrdv2.intel.com/v1/dl/getContent/671368?explicitVersion=true
Quoted 13 lines
Signed-off-by Dave Hansen <[email protected]>Cc Jarkko Sakkinen <[email protected]>Cc Andy Lutomirski <[email protected]>Cc Thomas Gleixner <[email protected]>Cc Ingo Molnar <[email protected]>Cc Borislav Petkov <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Sean Christopherson <[email protected]>Cc Kai Huang <[email protected]>Cc Haitao Huang <[email protected]>
I think it would make sense to have a co-maintainer with better access to unreleased ucode patches, if anyone is willing to consider. Perhaps someone from Intel given the constraints.
This would help with features such as AEX Notify. I can work out issues with existing hardware and also make sure that the whole stack is usable (as I'm also consumer for SGX).
BR, Jarkko