x86: Add vDSO exception fixup for SGX

arch/x86/entry/vdso/vsgx_enter_enclave.c new file +5 -0

+       /*
+        * Invoke the caller's exit handler if one was provided.  The return
+        * value tells us whether to re-enter the enclave (EENTER or ERESUME)
+        * or to return (EEXIT).
+        */

Quoted 1 line · Sean Christopherson

+

Quoted 5 lines

arch/x86/entry/vdso/vsgx_enter_enclave.c new file +5 -0

+       /*
+        * Invoke the caller's exit handler if one was provided.  The return
+        * value tells us whether to re-enter the enclave (EENTER or ERESUME)
+        * or to return (EEXIT).
+        */

arch/x86/entry/vdso/vsgx_enter_enclave.c line 116

Quoted 10 lines

arch/x86/entry/vdso/vsgx_enter_enclave.c new file +10 -0

+       if (exit_handler) {
+               leaf = exit_handler(exit_info, tcs, priv);
+               if (leaf == SGX_EENTER || leaf == SGX_ERESUME)
+                       goto enter_enclave;
+               if (leaf == SGX_EEXIT)
+                       return 0;
+               return -EINVAL;
+       } else if (leaf != SGX_EEXIT) {
+               return -EFAULT;
+       }

This still seems overcomplicated to me. How about letting the requested leaf (EENTER or ERESUME) be a parameter to the function and then just returning here? As it stands, you're requiring any ERESUME that gets issued (other than the implicit ones) to be issued in the same call stack, which is very awkward if you're doing something like forwarding the fault to a different task over a socket and then waiting in epoll_wait() or similar before resuming the enclave.

+Cc: linux-sgx, Haitao, Greg and Jethro

My apologies for neglecting to cc the SGX folks, original thread is here:

https://lkml.kernel.org/r/[email protected]

Quoted 20 lines · Andy Lutomirski

Quoted 1 line · Sean Christopherson

+

Quoted 5 lines

arch/x86/entry/vdso/vsgx_enter_enclave.c new file +5 -0

+       /*
+        * Invoke the caller's exit handler if one was provided.  The return
+        * value tells us whether to re-enter the enclave (EENTER or ERESUME)
+        * or to return (EEXIT).
+        */

arch/x86/entry/vdso/vsgx_enter_enclave.c line 116

Quoted 10 lines

arch/x86/entry/vdso/vsgx_enter_enclave.c new file +10 -0

+       if (exit_handler) {
+               leaf = exit_handler(exit_info, tcs, priv);
+               if (leaf == SGX_EENTER || leaf == SGX_ERESUME)
+                       goto enter_enclave;
+               if (leaf == SGX_EEXIT)
+                       return 0;
+               return -EINVAL;
+       } else if (leaf != SGX_EEXIT) {
+               return -EFAULT;
+       }

Quoted 7 lines

This still seems overcomplicated to me. How about letting the requested leaf (EENTER or ERESUME) be a parameter to the function and then just returning here? As it stands, you're requiring any ERESUME that gets issued (other than the implicit ones) to be issued in the same call stack, which is very awkward if you're doing something like forwarding the fault to a different task over a socket and then waiting in epoll_wait() or similar before resuming the enclave.

Ah, yeah, wasn't thinking about usage models where the enclave could get passed off to a different thread.

What about supporting both, i.e. keep the exit handler but make it 100% optional? And simplify the exit_handler to effectively return a boolean, i.e. "exit or continue".

Something like this:

notrace long __vdso_sgx_enter_enclave(u32 op, void *tcs, void *priv,
struct sgx_enclave_exit_info *exit_info, sgx_enclave_exit_handler *exit_handler)

{
        u64 rdi, rsi, rdx;
        u32 leaf;
        long ret;

        if (!tcs || !exit_info)
                return -EINVAL;

enter_enclave:

        if (op != SGX_EENTER && op != SGX_ERESUME)
                return -EINVAL;

        <same core code>

        /*
         * Invoke the caller's exit handler if one was provided.  The return
         * value tells us whether to re-enter the enclave (EENTER or ERESUME)
         * or to return (EEXIT).
         */
        if (exit_handler) {
                if (exit_handler(exit_info, tcs, priv)) {
                        op = exit_info->leaf;
                        goto enter_enclave;
                }
        }

        if (exit_info->leaf == SGX_EEXIT)
                return -EFAULT;

        return 0;
}

I like that the exit handler allows userspace to trap/panic with the full call stack in place, and in a dedicated path, i.e. outside of the basic enter/exit code. An exit handler probably doesn't fundamentally change what userspace can do with respect to debugging/reporting, but I think it would actually simplify some userspace implementations, e.g. I'd use it in my tests like so:

long fault_handler(struct sgx_enclave_exit_info *exit_info, void *tcs, void *priv)

{
        if (exit_info->leaf == SGX_EEXIT)
                return 0;

        <report exception and die/hang>
}

Quoted 12 lines · Sean Christopherson

I like that the exit handler allows userspace to trap/panic with the full call stack in place, and in a dedicated path, i.e. outside of the basic enter/exit code. An exit handler probably doesn't fundamentally change what userspace can do with respect to debugging/reporting, but I think it would actually simplify some userspace implementations, e.g. I'd use it in my tests like so:

long fault_handler(struct sgx_enclave_exit_info *exit_info, void *tcs, void *priv)

{
        if (exit_info->leaf == SGX_EEXIT)
                return 0;

Quoted 3 lines

<report exception and die/hang>
}

Hmm. That't not totally silly, although you could accomplish almost the same thing by wrapping the vDSO helper in another function.

Quoted 16 lines · Andy Lutomirski

Quoted 12 lines · Sean Christopherson

I like that the exit handler allows userspace to trap/panic with the full call stack in place, and in a dedicated path, i.e. outside of the basic enter/exit code. An exit handler probably doesn't fundamentally change what userspace can do with respect to debugging/reporting, but I think it would actually simplify some userspace implementations, e.g. I'd use it in my tests like so:

long fault_handler(struct sgx_enclave_exit_info *exit_info, void *tcs, void *priv)

{
       if (exit_info->leaf == SGX_EEXIT)
               return 0;

Quoted 6 lines

Quoted 3 lines

<report exception and die/hang>
}

Hmm. That't not totally silly, although you could accomplish almost the same thing by wrapping the vDSO helper in another function.

True, but I think there's value in having the option to intercept an exception at the exact(ish) point of failure, without having to return up the stack.

The enclave has full access to the process' memory space, including the untrsuted stack. It's not too far fetched to envision a scenario where the enclave corrupts the stack even if the enclave isn't intentionally using the stack, e.g. the host passes in variable that a points at the stack instead of whatever memory is supposed to be shared between the enclave and host. It'd be nice to have the ability to triage something like that without having to e.g. configure breakpoints on the stack.

Quoted 29 lines · Sean Christopherson

Quoted 16 lines · Andy Lutomirski

Quoted 12 lines · Sean Christopherson

I like that the exit handler allows userspace to trap/panic with the full call stack in place, and in a dedicated path, i.e. outside of the basic enter/exit code. An exit handler probably doesn't fundamentally change what userspace can do with respect to debugging/reporting, but I think it would actually simplify some userspace implementations, e.g. I'd use it in my tests like so:

long fault_handler(struct sgx_enclave_exit_info *exit_info, void *tcs, void *priv)

{
       if (exit_info->leaf == SGX_EEXIT)
               return 0;

Quoted 6 lines

Quoted 3 lines

<report exception and die/hang>
}

Hmm. That't not totally silly, although you could accomplish almost the same thing by wrapping the vDSO helper in another function.

True, but I think there's value in having the option to intercept an exception at the exact(ish) point of failure, without having to return up the stack.

Quoted 7 lines

The enclave has full access to the process' memory space, including the untrsuted stack. It's not too far fetched to envision a scenario where the enclave corrupts the stack even if the enclave isn't intentionally using the stack, e.g. the host passes in variable that a points at the stack instead of whatever memory is supposed to be shared between the enclave and host. It'd be nice to have the ability to triage something like that without having to e.g. configure breakpoints on the stack.

Ah, I see. You’re saying that, if the non-enclave stare is corrupted such that RIP is okay and RSP still points somewhere reasonable but the return address is garbage, then we can at least get to the fault handler and print something?

This only works if the fault handler pointer itself is okay, though, which somewhat limits the usefulness, given that its pointer is quite likely to be on the stack very close to the return address.

I really wish the ENCLU instruction had seen fit to preserve some registers.

Quoted 39 lines · Andy Lutomirski

Quoted 29 lines

Quoted 18 lines · Sean Christopherson

On Fri, Dec 07, 2018 at 09:56:09AM -0800, Andy Lutomirski wrote:

Quoted 12 lines · Sean Christopherson

I like that the exit handler allows userspace to trap/panic with the full call stack in place, and in a dedicated path, i.e. outside of the basic enter/exit code. An exit handler probably doesn't fundamentally change what userspace can do with respect to debugging/reporting, but I think it would actually simplify some userspace implementations, e.g. I'd use it in my tests like so:

long fault_handler(struct sgx_enclave_exit_info *exit_info, void *tcs, void *priv)

{
      if (exit_info->leaf == SGX_EEXIT)
              return 0;

Quoted 6 lines

Quoted 3 lines

<report exception and die/hang>
}

Hmm. That't not totally silly, although you could accomplish almost the same thing by wrapping the vDSO helper in another function.

True, but I think there's value in having the option to intercept an exception at the exact(ish) point of failure, without having to return up the stack.

Quoted 7 lines

The enclave has full access to the process' memory space, including the untrsuted stack. It's not too far fetched to envision a scenario where the enclave corrupts the stack even if the enclave isn't intentionally using the stack, e.g. the host passes in variable that a points at the stack instead of whatever memory is supposed to be shared between the enclave and host. It'd be nice to have the ability to triage something like that without having to e.g. configure breakpoints on the stack.

Quoted 4 lines

Ah, I see. You’re saying that, if the non-enclave stare is corrupted such that RIP is okay and RSP still points somewhere reasonable but the return address is garbage, then we can at least get to the fault handler and print something?

Yep. Even for something more subtle like GPR corruption it could dump the entire call stack before attempting to return back up.

Quoted 3 lines

This only works if the fault handler pointer itself is okay, though, which somewhat limits the usefulness, given that its pointer is quite likely to be on the stack very close to the return address.

Yeah, it's not a silver bullet by any means, but it does seem useful for at least some scenarios. Even exploding when invoking the handler instead of at a random point might prove useful, e.g. "calling my exit handler exploded, maybe my enclave corrupted the stack!".

Quoted 2 lines

I really wish the ENCLU instruction had seen fit to preserve some registers.

Speaking of preserving registers, the asm blob needs to mark RBX as clobbered since it's modified for EEXIT.

Quoted 5 lines · Sean Christopherson

Speaking of preserving registers, the asm blob needs to mark RBX as clobbered since it's modified for EEXIT.

Have fun with that. The x86_32 compiler seems to really like having its PIC register preserved, and you may get some lovely compiler errors.

Quoted 18 lines · Sean Christopherson

Quoted 7 lines · Andy Lutomirski

Ah, I see. You’re saying that, if the non-enclave stare is corrupted such that RIP is okay and RSP still points somewhere reasonable but the return address is garbage, then we can at least get to the fault handler and print something?

Yep. Even for something more subtle like GPR corruption it could dump the entire call stack before attempting to return back up.

Quoted 3 lines

This only works if the fault handler pointer itself is okay, though, which somewhat limits the usefulness, given that its pointer is quite likely to be on the stack very close to the return address.

Quoted 4 lines

Yeah, it's not a silver bullet by any means, but it does seem useful for at least some scenarios. Even exploding when invoking the handler instead of at a random point might prove useful, e.g. "calling my exit handler exploded, maybe my enclave corrupted the stack!".

Here’s another idea: calculate some little hash or other checksum of RSP, RBP, and perhaps a couple words on the stack, and do:

call __vdso_enclave_corrupted_state

If you get a mismatch after return. That function could be:

call __vdso_enclave_corrupted_state:
ud2

And now the debug trace makes it very clear what happened.

This may or may not be worth the effort.

But ISTM the enclave is almost as likely to corrupt the host state and the. EEXIT as it is to corrupt the host state and then fault.

Quoted 18 lines · Andy Lutomirski

Quoted 18 lines · Sean Christopherson

Quoted 7 lines · Andy Lutomirski

Ah, I see. You’re saying that, if the non-enclave stare is corrupted such that RIP is okay and RSP still points somewhere reasonable but the return address is garbage, then we can at least get to the fault handler and print something?

Yep. Even for something more subtle like GPR corruption it could dump the entire call stack before attempting to return back up.

Quoted 3 lines

This only works if the fault handler pointer itself is okay, though, which somewhat limits the usefulness, given that its pointer is quite likely to be on the stack very close to the return address.

Quoted 7 lines

Quoted 4 lines

Yeah, it's not a silver bullet by any means, but it does seem useful for at least some scenarios. Even exploding when invoking the handler instead of at a random point might prove useful, e.g. "calling my exit handler exploded, maybe my enclave corrupted the stack!".

Here’s another idea: calculate some little hash or other checksum of RSP, RBP, and perhaps a couple words on the stack, and do:

Corrupting RSP and RBP as opposed to the stack memory seems much less likely since the enclave would have to poke into the save state area. And as much as I dislike the practice of intentionally manipulating SSA.RSP, preventing the user from doing something because we're "helping" doesn't seem right.

Quoted 10 lines

call __vdso_enclave_corrupted_state

If you get a mismatch after return. That function could be:

call __vdso_enclave_corrupted_state:
ud2

And now the debug trace makes it very clear what happened.

This may or may not be worth the effort.

Running a checksum on the stack for every exit doesn't seem like it'd be worth the effort, especially since this type of bug should be quite rare, at least in production environments.

If we want to pursue the checksum idea I think the easiest approach would be to combine it with an exit_handler and do a simple check on the handler. It'd be minimal overhead in the fast path and would flag cases where invoking exit_handle() would explode, while deferring all other checks to the user.

Quoted 10 lines · Sean Christopherson

Running a checksum on the stack for every exit doesn't seem like it'd be worth the effort, especially since this type of bug should be quite rare, at least in production environments.

If we want to pursue the checksum idea I think the easiest approach would be to combine it with an exit_handler and do a simple check on the handler. It'd be minimal overhead in the fast path and would flag cases where invoking exit_handle() would explode, while deferring all other checks to the user.

How about this variant?

#define MAGIC 0xaaaabbbbccccddddul
#define RETADDR_HASH ((unsigned long)__builtin_return_address(0) ^ MAGIC)

void foo(void)

{
    volatile unsigned long hash = RETADDR_HASH;

    /* placeholder for your actual code */
    asm volatile ("nop");

    if (hash != RETADDR_HASH)
        asm volatile ("ud2");
}

But I have a real argument for dropping exit_handler: in this new age of Spectre, the indirect call is a retpoline, and it's therefore quite slow.

So I'm not saying NAK, but I do think it's unnecessary.

Quoted 29 lines · Andy Lutomirski

Quoted 10 lines · Sean Christopherson

Running a checksum on the stack for every exit doesn't seem like it'd be worth the effort, especially since this type of bug should be quite rare, at least in production environments.

If we want to pursue the checksum idea I think the easiest approach would be to combine it with an exit_handler and do a simple check on the handler. It'd be minimal overhead in the fast path and would flag cases where invoking exit_handle() would explode, while deferring all other checks to the user.

How about this variant?

#define MAGIC 0xaaaabbbbccccddddul
#define RETADDR_HASH ((unsigned long)__builtin_return_address(0) ^ MAGIC)

void foo(void)

{
    volatile unsigned long hash = RETADDR_HASH;

    /* placeholder for your actual code */
    asm volatile ("nop");

    if (hash != RETADDR_HASH)
        asm volatile ("ud2");
}

Quoted 3 lines

But I have a real argument for dropping exit_handler: in this new age of Spectre, the indirect call is a retpoline, and it's therefore quite slow.

Technically slower, but would the extra CALL+RET pair even be noticeable in the grand scheme of SGX?

Quoted 35 lines · Sean Christopherson

Quoted 32 lines · Andy Lutomirski

Quoted 10 lines · Sean Christopherson

Running a checksum on the stack for every exit doesn't seem like it'd be worth the effort, especially since this type of bug should be quite rare, at least in production environments.

If we want to pursue the checksum idea I think the easiest approach would be to combine it with an exit_handler and do a simple check on the handler. It'd be minimal overhead in the fast path and would flag cases where invoking exit_handle() would explode, while deferring all other checks to the user.

How about this variant?

#define MAGIC 0xaaaabbbbccccddddul
#define RETADDR_HASH ((unsigned long)__builtin_return_address(0) ^ MAGIC)

void foo(void)

{
    volatile unsigned long hash = RETADDR_HASH;

    /* placeholder for your actual code */
    asm volatile ("nop");

    if (hash != RETADDR_HASH)
        asm volatile ("ud2");
}

But I have a real argument for dropping exit_handler: in this new age of Spectre, the indirect call is a retpoline, and it's therefore quite slow.

Quoted 2 lines

Technically slower, but would the extra CALL+RET pair even be noticeable in the grand scheme of SGX?

But it's CALL, CALL, MOV to overwrite return address, intentionally midpredicted RET, and RET because Spectre. That whole sequence seems to be several tens of cycles, so it's a lot worse than just CALL+RET. Whether it's noticeable overall is a fair question, though.

Quoted 40 lines · Andy Lutomirski

Quoted 37 lines · Sean Christopherson

Quoted 29 lines · Andy Lutomirski

Quoted 10 lines · Sean Christopherson

Running a checksum on the stack for every exit doesn't seem like it'd be worth the effort, especially since this type of bug should be quite rare, at least in production environments.

If we want to pursue the checksum idea I think the easiest approach would be to combine it with an exit_handler and do a simple check on the handler. It'd be minimal overhead in the fast path and would flag cases where invoking exit_handle() would explode, while deferring all other checks to the user.

How about this variant?

#define MAGIC 0xaaaabbbbccccddddul
#define RETADDR_HASH ((unsigned long)__builtin_return_address(0) ^ MAGIC)

void foo(void)

{
    volatile unsigned long hash = RETADDR_HASH;

    /* placeholder for your actual code */
    asm volatile ("nop");

    if (hash != RETADDR_HASH)
        asm volatile ("ud2");
}

Quoted 3 lines

But I have a real argument for dropping exit_handler: in this new age of Spectre, the indirect call is a retpoline, and it's therefore quite slow.

Technically slower, but would the extra CALL+RET pair even be noticeable in the grand scheme of SGX?

Quoted 4 lines

But it's CALL, CALL, MOV to overwrite return address, intentionally midpredicted RET, and RET because Spectre. That whole sequence seems to be several tens of cycles, so it's a lot worse than just CALL+RET. Whether it's noticeable overall is a fair question, though.

I was thinking of the case where the handler re-entered the enclave vs. leaving and re-calling the vDSO, which would be RET+CALL and some other stuff.

Intel Software Guard Extensions (SGX) SGX introduces a new CPL3-only enclave mode that runs as a sort of black box shared object that is hosted by an untrusted normal CPL3 process.

Enclave transitions have semantics that are a lovely blend of SYCALL, SYSRET and VM-Exit. In a non-faulting scenario, entering and exiting an enclave can only be done through SGX-specific instructions, EENTER and EEXIT respectively. EENTER+EEXIT is analogous to SYSCALL+SYSRET, e.g. EENTER/SYSCALL load RCX with the next RIP and EEXIT/SYSRET load RIP from R{B,C}X.

But in a faulting/interrupting scenario, enclave transitions act more like VM-Exit and VMRESUME. Maintaining the black box nature of the enclave means that hardware must automatically switch CPU context when an Asynchronous Exiting Event (AEE) occurs, an AEE being any interrupt or exception (exceptions are AEEs because asynchronous in this context is relative to the enclave and not CPU execution, e.g. the enclave doesn't get an opportunity to save/fuzz CPU state).

Like VM-Exits, all AEEs jump to a common location, referred to as the Asynchronous Exiting Point (AEP). The AEP is specified at enclave entry via register passed to EENTER/ERESUME, similar to how the hypervisor specifies the VM-Exit point (via VMCS.HOST_RIP at VMLAUNCH/VMRESUME). Resuming the enclave/VM after the exiting event is handled is done via ERESUME/VMRESUME respectively. In SGX, AEEs that are handled by the kernel, e.g. INTR, NMI and most page faults, IRET will journey back to the AEP which then ERESUMEs th enclave.

Enclaves also behave a bit like VMs in the sense that they can generate exceptions as part of their normal operation that for all intents and purposes need to handled in the enclave/VM. However, unlike VMX, SGX doesn't allow the host to modify its guest's, a.k.a. enclave's, state, as doing so would circumvent the enclave's security. So to handle an exception, the enclave must first be re-entered through the normal EENTER flow (SYSCALL/SYSRET behavior), and then resumed via ERESUME (VMRESUME behavior) after the source of the exception is resolved.

All of the above is just the tip of the iceberg when it comes to running an enclave. But, SGX was designed in such a way that the host process can utilize a library to build, launch and run an enclave. This is roughly analogous to how e.g. libc implementations are used by most applications so that the application can focus on its business logic.

The big gotcha is that because enclaves can generate *and* handle exceptions, any SGX library must be prepared to handle nearly any exception at any time (well, any time a thread is executing in an enclave). In Linux, this means the SGX library must register a signal handler in order to intercept relevant exceptions and forward them to the enclave (or in some cases, take action on behalf of the enclave). Unfortunately, Linux's signal mechanism doesn't mesh well with libraries, e.g. signal handlers are process wide, are difficult to chain, etc... This becomes particularly nasty when using multiple levels of libraries that register signal handlers, e.g. running an enclave via cgo inside of the Go runtime.

In comes vDSO to save the day. Now that vDSO can fixup exceptions, add a function, __vdso_sgx_enter_enclave(), to wrap enclave transitions and intercept any exceptions that occur when running the enclave.

__vdso_sgx_enter_enclave() accepts four parameters:

- A pointer to a Thread Control Structure (TCS). A TCS is a page within the enclave that defines/tracks the context of an enclave thread.

- An opaque pointer passed to the enclave via RDI, e.g. to marshal data into the enclave.

- A pointer to a struct sgx_enclave_exit_info that is used to relay exit/fault information back to the caller. The primary variable in the exit info is the ENCLU leaf at the time of exit, which can be queried to determine whether the enclave exited cleanly (EEXIT), or took an exception (EENTER or ERESUME).

The exact leaf is captured, instead of e.g. a fault flag, so that the caller can identity whether a fault occurred in the enclave or
on EENTER. A fault on EENTER generally means the enclave has died
and needs to be restarted.

On a clean EEXIT, registers RDI, RSI and RDX are captured as-is, e.g. to pass data out of the enclave. On a fault that is reported to the caller, the exit info is stuffed (by way of the vDSO fixup handler) with the trapnr, error_code and address. Not all enclave exits are reported to the caller, e.g. interrupts and faults that are handled by the kernel do not trigger fixup and IRET back to ENCLU[ERESUME], i.e. unconditionally resume the enclave.

- An optional exit handler that, when defined, is invoked prior to returning. The callback provides the enclave's untrusted runtime an opportunity to resolve a fault or service a remote procedure call without losing its stack context.

In addition to allowing the runtime to do silly shenanigans with its stack, e.g. pushing data onto the stack from within the enclave, the handler approach preserves the full call stack for debugging purposes. For example, to accept a new EPC page into the enclave, an enclave with a single TCS must re-EENTER the enclave using the same TCS to EACCEPT the new page prior to executing ERESUME to restart at the fault context. The handler approach allows reentry without undoing the original call stack, i.e. preserves the view of the original interrupted call.

Note that this effectively requires userspace to implement an exit handler if they want to support correctable enclave faults, as there is no other way to request ERESUME.

On Thu, Dec 06, 2018 at 02:19:22PM -0800, Sean Christopherson wrote:

Good morning, I hope the week is ending well for everyone.

The audience for the issues that Sean is addressing are the groups that have developed and are delivering an Untrusted RunTime System (URTS) as a component of SGX Platform SoftWare (PSW). At the current time I believe that is Intel and us, although there may be stealth initiatives as well.

Sean is obviously coordinating with and supporting the Intel SDK/PSW team. SGX has now been in the wild for 2-3 years so there is work other then the reference implementation in the field. The purpose of this mail is to make sure that everyone understands the issues and ramifications of changes that may end up in 'Flag Day' events, if nothing else so we can get the best possible implementation put forward.

Baidu and Fortanix are working on Trusted RunTime Systems (TRTS) based on RUST, I believe, so this will affect them to the extent that they are implementing their own low level enclave runtime support or they may be simply building on top of the low level Intel TRTS. Perhaps Jethro would comment on these issues if he could.

Quoted 37 lines

Intel Software Guard Extensions (SGX) SGX introduces a new CPL3-only enclave mode that runs as a sort of black box shared object that is hosted by an untrusted normal CPL3 process.

Enclave transitions have semantics that are a lovely blend of SYCALL, SYSRET and VM-Exit. In a non-faulting scenario, entering and exiting an enclave can only be done through SGX-specific instructions, EENTER and EEXIT respectively. EENTER+EEXIT is analogous to SYSCALL+SYSRET, e.g. EENTER/SYSCALL load RCX with the next RIP and EEXIT/SYSRET load RIP from R{B,C}X.

But in a faulting/interrupting scenario, enclave transitions act more like VM-Exit and VMRESUME. Maintaining the black box nature of the enclave means that hardware must automatically switch CPU context when an Asynchronous Exiting Event (AEE) occurs, an AEE being any interrupt or exception (exceptions are AEEs because asynchronous in this context is relative to the enclave and not CPU execution, e.g. the enclave doesn't get an opportunity to save/fuzz CPU state).

Like VM-Exits, all AEEs jump to a common location, referred to as the Asynchronous Exiting Point (AEP). The AEP is specified at enclave entry via register passed to EENTER/ERESUME, similar to how the hypervisor specifies the VM-Exit point (via VMCS.HOST_RIP at VMLAUNCH/VMRESUME). Resuming the enclave/VM after the exiting event is handled is done via ERESUME/VMRESUME respectively. In SGX, AEEs that are handled by the kernel, e.g. INTR, NMI and most page faults, IRET will journey back to the AEP which then ERESUMEs th enclave.

Enclaves also behave a bit like VMs in the sense that they can generate exceptions as part of their normal operation that for all intents and purposes need to handled in the enclave/VM. However, unlike VMX, SGX doesn't allow the host to modify its guest's, a.k.a. enclave's, state, as doing so would circumvent the enclave's security. So to handle an exception, the enclave must first be re-entered through the normal EENTER flow (SYSCALL/SYSRET behavior), and then resumed via ERESUME (VMRESUME behavior) after the source of the exception is resolved.

Quoted 6 lines

All of the above is just the tip of the iceberg when it comes to running an enclave. But, SGX was designed in such a way that the host process can utilize a library to build, launch and run an enclave. This is roughly analogous to how e.g. libc implementations are used by most applications so that the application can focus on its business logic.

Just to make sure we are on the same page.

When you refer to 'build' an enclave I assume you mean to construct the enclave image from a compiled shared object file. Or are you suggesting an environment where the library loads dynamically generated object code into an enclave using Enclave Dynamic Memory Management (EDMM)? Building, i.e. compiling and linking an enclave, doesn't seem to be the province of library support.

Perhaps a more accurate phrase would be; 'to load, initialize and execute an enclave image'.

To step back further and frame the issue most precisely. What the VDSO work is proposing to support is shifting from a model where applications 'own' enclaves to a model where dynamically linked shared libraries 'own' enclaves, correct?

With the VDSO model you are proposing an environment where library developers can implement SGX/enclave based protections of code and data which the actual application linking against the library would be totally unaware of, correct?

To summarize succinctly, would it be correct to assert that there are three possible advantages to the VDSO approach:

1.) Shared libraries can own enclaves without the knowledge of applications.

2.) EDMM responses can be implemented more efficiently.

3.) Reduction in enclave attack surface.

With respect to point three, perhaps the most important attack on SGX security guarantees to date has been documented in Lee et.al.'s dark-ROP attack. A significant aspect of that attack was AEE based probing of enclave execution. Do you have reflections with respect to how the proposed archictecture would lessen or facilitate such attacks?

The economics of software development seem to be motivating the use of libOS approaches to porting applications to enclaves which has significant implications with respect to AEE based probing attacks.

Quoted 4 lines

Note that this effectively requires userspace to implement an exit handler if they want to support correctable enclave faults, as there is no other way to request ERESUME.

Once again to be clear for those of us that have investments in the existing ABI.

I'm assuming that in the proposed model the URTS would interrogate the VDSO to determine the availability of entry and exception handling support and then setup the appropriate infrastructure and exit handler?

As a result, do you anticipate the need for a 'flag day' with respect to URTS/PSW/SDK support for all of this?

In addition, would you anticipate anything in the design that would be problematic for environments where the application would choose to implement an enclave in addition to linking against a library that implements enclaves?

Baidu and Fortanix are working on Trusted RunTime Systems (TRTS) based on RUST, I believe, so this will affect them to the extent that they are implementing their own low level enclave runtime support or they may be simply building on top of the low level Intel TRTS. Perhaps Jethro would comment on these issues if he could.

I'm assuming that in the proposed model the URTS would interrogate the VDSO to determine the availability of entry and exception handling support and then setup the appropriate infrastructure and exit handler? VDSO's are typically the domain of the system library. Given the nature of SGX I couldn't even conceive of Glibc offering support and, if it was acceptable to provide support, the potential timeframe that would be involved in seeing deployment in the field.

As a result, do you anticipate the need for a 'flag day' with respect to URTS/PSW/SDK support for all of this?

arch/x86/entry/vdso/vsgx_enter_enclave.c new file +0 -0

new file mode 100644
index 000000000000..896c2eb079bb

arch/x86/entry/vdso/vsgx_enter_enclave.c new file +18 -0

+enter_enclave:
+	asm volatile(
+		/*
+		 * When an event occurs in an enclave, hardware first exits the
+		 * enclave to the AEP, switching CPU context along the way, and
+		 * *then* delivers the event as usual.  As part of the context
+		 * switching, registers are loaded with synthetic state (except
+		 * BP and SP, which are saved/restored).  The defined synthetic
+		 * state loads registers so that simply executing ENCLU will do
+		 * ERESUME, e.g. RAX=4, RBX=TCS and RCX=AEP after an AEE.  So,
+		 * we only need to load RAX, RBX and RCX for the initial entry.
+		 * The AEP can point at that same ENCLU, fixup will jump us out
+		 * if an exception was unhandled.
+		 */
+		"	lea	1f(%%rip), %%rcx\n"
+		"1:	enclu\n"
+		"2:\n"
+

This is not sufficient to support the Fortanix SGX ABI calling convention, which was designed to be mostly compatible with the SysV 64-bit calling convention. The following registers need to be passed in to an enclave from userspace: RDI, RSI, RDX, R8, R9, R10. The following registers need to be passed out from an enclave to userspace: RDI, RSI, RDX, R8, R9.

Quoted 8 lines · Jethro Beekman

This is not sufficient to support the Fortanix SGX ABI calling convention, which was designed to be mostly compatible with the SysV 64-bit calling convention. The following registers need to be passed in to an enclave from userspace: RDI, RSI, RDX, R8, R9, R10. The following registers need to be passed out from an enclave to userspace: RDI, RSI, RDX, R8, R9.

Are you asking nicely to change the new Linux ABI to be consistent with your existing ABI? Or, are you saying that the new ABI *must* be compatible with this previous out-of-tree implementation?

Quoted 6 lines · Jethro Beekman

This is not sufficient to support the Fortanix SGX ABI calling convention, which was designed to be mostly compatible with the SysV 64-bit calling convention. The following registers need to be passed in to an enclave from userspace: RDI, RSI, RDX, R8, R9, R10. The following registers need to be passed out from an enclave to userspace: RDI, RSI, RDX, R8, R9.

Are you asking nicely to change the new Linux ABI to be consistent with your existing ABI? Or, are you saying that the new ABI *must* be compatible with this previous out-of-tree implementation?

Quoted 11 lines · Dave Hansen

Quoted 6 lines · Jethro Beekman

This is not sufficient to support the Fortanix SGX ABI calling convention, which was designed to be mostly compatible with the SysV 64-bit calling convention. The following registers need to be passed in to an enclave from userspace: RDI, RSI, RDX, R8, R9, R10. The following registers need to be passed out from an enclave to userspace: RDI, RSI, RDX, R8, R9.

Are you asking nicely to change the new Linux ABI to be consistent with your existing ABI? Or, are you saying that the new ABI *must* be compatible with this previous out-of-tree implementation?

What's being discussed here is one of the alternatives for SGX fault handling, meant to improve the current status quo of having to use a signal handler.

I'm merely providing a data point that the currently proposed solution is not sufficient to support current use of the (ring 3) ENCLU instruction. You might find this useful in determining whether proposed kernel features will actually be used by users, and in further developing this solution or other solutions to the fault handling issue.

If going with the vDSO solution, I think something with semantics closer to the actual instruction would be preferred, like the following:

notrace __attribute__((naked)) long __vdso_sgx_enclu_with_aep()

{
	asm volatile(
		"	lea	2f(%%rip), %%rcx\n"
		"1:	enclu\n"
		"2:     ret\n"
		".pushsection .fixup, \"ax\" \n"
		"3:	jmp 2b\n"
		".popsection\n"
		_ASM_VDSO_EXTABLE_HANDLE(1b, 3b)
		:::
	);
}