Skip to contentSkip to conversation

Intel SGX1 support

PATCH Superseded Jarkko Sakkinen 15 participants 314 messages Apr 17, 2019 – Jul 17, 2019 1 message not in the archive Page 2 of 3

Subsystems (9)

v20 of 40 Superseded Latest is v40 →

lore
Share & tools
lore
#171 lore raw

RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson
Quoted 49 lines

From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 10:55 AM

Quoted 2 lines · Sean Christopherson

Hmm, I've been thinking more about pulling permissions from the source page. Conceptually I'm not sure we need to meet the same requirements

as

Quoted 1 line

non-enclave DSOs while the enclave is being built, i.e. do we really

need

Quoted 3 lines

to force userspace to fully map the enclave in normal memory?

Consider the Graphene scenario where it's building an enclave on the

fly.

Quoted 2 lines

Pulling permissions from the source VMAs means Graphene has to map the code pages of the enclave with X. This means Graphene will need

EXEDMOD

Quoted 1 line

(or EXECMEM if Graphene isn't careful). In a non-SGX scenario this

makes

Quoted 3 lines

perfect sense since there is no way to verify the end result of RW->RX.

But for SGX, assuming enclaves are whitelisted by their sigstruct

(checked

Quoted 1 line

at EINIT) and because page permissions affect sigstruct.MRENCLAVE, it

*is*

Quoted 1 line

possible to verify the resulting RX contents. E.g. for the purposes

of

Quoted 9 lines

LSMs, can't we use the .sigstruct file as a proxy for the enclave and require FILE__EXECUTE on the .sigstruct inode to map/run the enclave?

Stephen, is my logic sound?

If so...

- Require FILE__READ+FILE__EXECUTE on .sigstruct to mmap() the

enclave.

Quoted 2 lines

- Prevent userspace from mapping the enclave with permissions beyond

the

Quoted 1 line

original permissions of the enclave. This can be done by

populating

Quoted 1 line

VM_MAY{READ,WRITE,EXEC} from the SECINFO (same basic concept as

Andy's

Quoted 1 line

proposals). E.g. pre-EINIT, mmap() and mprotect() can only

succeed

Quoted 3 lines

with PROT_NONE.

- Require FILE__{READ,WRITE,EXECUTE} on /dev/sgx/enclave for

Quoted 8 lines

simplicity,

Quoted 1 line

or provide an alternate SGX_IOC_MPROTECT if we want to sidestep

the

Quoted 1 line

FILE__WRITE requirement.

One more thought. EADD (and the equivalent SGX2 flow) could do security_mmap_file() with a NULL file on the SECINFO permissions, which would trigger PROCESS_EXECMEM if an enclave attempts to map a page RWX.

If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.

Quoted 5 lines
Quoted 2 lines

No changes are required to LSMs, SGX1 has a single LSM touchpoint in

its

Quoted 2 lines

mmap(), and I *think* the only required userspace change is to mmap() PROT_NONE when allocating the enclave's virtual address range.

I'm not sure I understand the motivation behind this proposal to decouple initial EPC permissions from source pages.

I don't think it a big deal to fully mmap() enclave files, which have to be parsed by user mode anyway to determine various things including but not limited to the size of heap(s), size and number of TCSs/stacks/TLS areas, and the overall enclave size. So with PHDRs parsed, it's trivial to mmap() each segment with permissions from its PHDR.

Quoted 4 lines
Quoted 3 lines

As for Graphene, it doesn't need extra permissions to run its enclaves, it just needs a way to install .sigstruct, which is a generic

permissions

Quoted 15 lines
Quoted 7 lines

problem and not SGX specific.

For SGX2 maybe:

- No additional requirements to map an EAUG'd page as RW page. Not aligned with standard MAP_SHARED behavior, but we really don't

want

Quoted 7 lines

to require FILE__WRITE, and thus allow writes to .sigstruct.

- Require FILE__EXECMOD on the .sigstruct to map previously writable page as executable (which indirectly includes all EAUG'd pages). Wiring this up will be a little funky, but we again we don't want to require FILE__WRITE on .sigstruct.

I'm lost. Why is EAUG tied to permissions on .sigstruct?

-Cedric

#172 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 39 lines · Cedric
Quoted 39 lines

From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 10:55 AM

Quoted 33 lines · Sean Christopherson

Hmm, I've been thinking more about pulling permissions from the source page. Conceptually I'm not sure we need to meet the same requirements as non-enclave DSOs while the enclave is being built, i.e. do we really need to force userspace to fully map the enclave in normal memory?

Consider the Graphene scenario where it's building an enclave on the fly. Pulling permissions from the source VMAs means Graphene has to map the code pages of the enclave with X. This means Graphene will need EXEDMOD (or EXECMEM if Graphene isn't careful). In a non-SGX scenario this makes perfect sense since there is no way to verify the end result of RW->RX.

But for SGX, assuming enclaves are whitelisted by their sigstruct (checked at EINIT) and because page permissions affect sigstruct.MRENCLAVE, it *is* possible to verify the resulting RX contents. E.g. for the purposes of LSMs, can't we use the .sigstruct file as a proxy for the enclave and require FILE__EXECUTE on the .sigstruct inode to map/run the enclave?

Stephen, is my logic sound?

If so...

- Require FILE__READ+FILE__EXECUTE on .sigstruct to mmap() the enclave.

- Prevent userspace from mapping the enclave with permissions beyond the original permissions of the enclave. This can be done by populating VM_MAY{READ,WRITE,EXEC} from the SECINFO (same basic concept as Andy's proposals). E.g. pre-EINIT, mmap() and mprotect() can only succeed with PROT_NONE.

- Require FILE__{READ,WRITE,EXECUTE} on /dev/sgx/enclave for simplicity, or provide an alternate SGX_IOC_MPROTECT if we want to sidestep the FILE__WRITE requirement.

Quoted 8 lines
Quoted 3 lines

One more thought. EADD (and the equivalent SGX2 flow) could do security_mmap_file() with a NULL file on the SECINFO permissions, which would trigger PROCESS_EXECMEM if an enclave attempts to map a page RWX.

If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.

My point is that enclaves have different properties than shared objects.

Normal LSM behavior with regard to executing files is to label files with e.g. FILE__EXECUTE. Because an enclave must be built to the exact specifications of .sigstruct, requring FILE__EXECUTE on the .sigstruct is effectively the same as requiring FILE__EXECUTE on the enclave itself.

Addressing your scenario of loading an executable page in EPC, doing so would require one of the following:

- Ability to install a .sigstruct with FILE__EXECUTE

- PROCESS__EXECMEM

- FILE__EXECMOD and SGX2 support

Quoted 5 lines

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.

There is no FILE__EXECMEM check, only PROCESS__EXECMEM and FILE__EXECMOD. I assume you're referring to the latter?

I don't see a fundamental difference between having RWX in an enclave and RWX in normal memory, either way the process can execute arbitrary code, i.e. PROCESS__EXECMEM is appropriate. Yes, an enclave will #UD on certain instructions, but that's easily sidestepped by having a trampoline in the host (marked RX) and piping arbitrary code into the enclave. Or using EEXIT to do a bit of ROP.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

No changes are required to LSMs, SGX1 has a single LSM touchpoint in

its

Quoted 2 lines

mmap(), and I *think* the only required userspace change is to mmap() PROT_NONE when allocating the enclave's virtual address range.

I'm not sure I understand the motivation behind this proposal to decouple initial EPC permissions from source pages.

Pulling permissions from source pages means userspace needs to fully map the in normal memory, including marking pages executable. That exposes the loader to having executable pages in its address space that it has no intention of executing (outside of the enclave). And for Graphene, it means having to actively avoid PROCESS__EXECMEM, e.g. by using a dummy backing file to build the enclave instead of anon memory.

Quoted 17 lines

I don't think it a big deal to fully mmap() enclave files, which have to be parsed by user mode anyway to determine various things including but not limited to the size of heap(s), size and number of TCSs/stacks/TLS areas, and the overall enclave size. So with PHDRs parsed, it's trivial to mmap() each segment with permissions from its PHDR.

Quoted 11 lines
Quoted 11 lines

As for Graphene, it doesn't need extra permissions to run its enclaves, it just needs a way to install .sigstruct, which is a generic permissions problem and not SGX specific.

For SGX2 maybe:

- No additional requirements to map an EAUG'd page as RW page. Not aligned with standard MAP_SHARED behavior, but we really don't want to require FILE__WRITE, and thus allow writes to .sigstruct.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines

- Require FILE__EXECMOD on the .sigstruct to map previously writable page as executable (which indirectly includes all EAUG'd pages). Wiring this up will be a little funky, but we again we don't want to require FILE__WRITE on .sigstruct.

I'm lost. Why is EAUG tied to permissions on .sigstruct?

Because for the purposes of LSM checks, .sigstruct is the enclave's backing file, and mapping a previously writable enclave page as exectuable is roughly equivalent to mapping a CoW'd page as exectuable.

maintainer
#173 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 6 lines · Sean Christopherson
Quoted 3 lines · Cedric
Quoted 3 lines

From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 10:55 AM

Quoted 6 lines

I don't see a fundamental difference between having RWX in an enclave and RWX in normal memory, either way the process can execute arbitrary code, i.e. PROCESS__EXECMEM is appropriate. Yes, an enclave will #UD on certain instructions, but that's easily sidestepped by having a trampoline in the host (marked RX) and piping arbitrary code into the enclave. Or using EEXIT to do a bit of ROP.

There's a difference, albeit a somewhat weak one, if sigstructs are whitelisted. FILE__EXECMOD on
either /dev/sgx/enclave or on the sigstruct is not an entirely crazy way to express this.

#174 lore raw

RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson
Quoted 11 lines

From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 12:14 PM

My point is that enclaves have different properties than shared objects.

Normal LSM behavior with regard to executing files is to label files with e.g. FILE__EXECUTE. Because an enclave must be built to the exact specifications of .sigstruct, requring FILE__EXECUTE on the .sigstruct is effectively the same as requiring FILE__EXECUTE on the enclave itself.

Quoted 8 lines

Addressing your scenario of loading an executable page in EPC, doing so would require one of the following:

- Ability to install a .sigstruct with FILE__EXECUTE

- PROCESS__EXECMEM

- FILE__EXECMOD and SGX2 support

Now I got your point. It sounds a great idea to me!

But instead of using .sigstruct file, I'd still recommend using file mapping (i.e. SIGSTRUCT needs to reside in executable memory). But then there'll be a hole - a process having FILE__EXECMOD on any file could use that file as a SIGSTRUCT. Probably we'll need a new type in SELinux to label enclave/sigstruct files.

Quoted 10 lines
Quoted 3 lines

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have

been checked.

Quoted 3 lines

For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.

There is no FILE__EXECMEM check, only PROCESS__EXECMEM and FILE__EXECMOD. I assume you're referring to the latter?

Yes.

Quoted 7 lines

I don't see a fundamental difference between having RWX in an enclave and RWX in normal memory, either way the process can execute arbitrary code, i.e. PROCESS__EXECMEM is appropriate. Yes, an enclave will #UD on certain instructions, but that's easily sidestepped by having a trampoline in the host (marked RX) and piping arbitrary code into the enclave. Or using EEXIT to do a bit of ROP.

I'm with you.

With your proposal only FILE__EXECMOD is needed on /dev/sgx/enclave to launch Graphene enclaves or the like.

Quoted 11 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

No changes are required to LSMs, SGX1 has a single LSM touchpoint in

its

Quoted 2 lines

mmap(), and I *think* the only required userspace change is to mmap() PROT_NONE when allocating the enclave's virtual address

range.

Quoted 3 lines

I'm not sure I understand the motivation behind this proposal to decouple initial EPC permissions from source pages.

Quoted 6 lines

Pulling permissions from source pages means userspace needs to fully map the in normal memory, including marking pages executable. That exposes the loader to having executable pages in its address space that it has no intention of executing (outside of the enclave). And for Graphene, it means having to actively avoid PROCESS__EXECMEM, e.g. by using a dummy backing file to build the enclave instead of anon memory.

Agreed.

Quoted 27 lines
Quoted 6 lines

I don't think it a big deal to fully mmap() enclave files, which have to be parsed by user mode anyway to determine various things including but not limited to the size of heap(s), size and number of TCSs/stacks/TLS areas, and the overall enclave size. So with PHDRs parsed, it's trivial to mmap() each segment with permissions from its

PHDR.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

As for Graphene, it doesn't need extra permissions to run its enclaves, it just needs a way to install .sigstruct, which is a generic permissions problem and not SGX specific.

For SGX2 maybe:

- No additional requirements to map an EAUG'd page as RW page.

Not

Quoted 1 line
Quoted 1 line
Quoted 1 line

aligned with standard MAP_SHARED behavior, but we really don't

want

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

to require FILE__WRITE, and thus allow writes to .sigstruct.

- Require FILE__EXECMOD on the .sigstruct to map previously

writable

Quoted 1 line
Quoted 1 line
Quoted 1 line

page as executable (which indirectly includes all EAUG'd

pages).

Quoted 1 line
Quoted 1 line
Quoted 1 line

Wiring this up will be a little funky, but we again we don't

want

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

to require FILE__WRITE on .sigstruct.

I'm lost. Why is EAUG tied to permissions on .sigstruct?

Because for the purposes of LSM checks, .sigstruct is the enclave's backing file, and mapping a previously writable enclave page as exectuable is roughly equivalent to mapping a CoW'd page as exectuable.

I think I've got your idea. You are trying to use permissions on .sigstruct to determine whether EAUG will be available to that specific enclave. Am I right?

I'd tie EAUG to the permissions of /dev/sgx/enclave instead. But why? There are couple of reasons. For one, a SIGSTRUCT identifies the behavior of the enclave, hence the SGX features needed by that enclave. So if an enclave requires EAUG, the .sigstruct has to allow EAUG or the enclave wouldn't work. That means the system admin wouldn't have a choice but to match up what's needed by the enclave. For two, whether to allow, say loading code dynamically into an enclave, depends on whether the host process can tolerate the inherent risk. And that decision is seldom made on individual enclaves but to the host process as a whole. And /dev/sgx/enclave serves that purpose.

-Cedric

#175 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 16 lines · Cedric
Quoted 16 lines

From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 12:14 PM

My point is that enclaves have different properties than shared objects.

Normal LSM behavior with regard to executing files is to label files with e.g. FILE__EXECUTE. Because an enclave must be built to the exact specifications of .sigstruct, requring FILE__EXECUTE on the .sigstruct is effectively the same as requiring FILE__EXECUTE on the enclave itself.

Addressing your scenario of loading an executable page in EPC, doing so would require one of the following:

- Ability to install a .sigstruct with FILE__EXECUTE

Quoted 8 lines
Quoted 3 lines

- PROCESS__EXECMEM

- FILE__EXECMOD and SGX2 support

Now I got your point. It sounds a great idea to me!

But instead of using .sigstruct file, I'd still recommend using file mapping (i.e. SIGSTRUCT needs to reside in executable memory). But then there'll be a

Why? Even in the Graphene case the final .sigstruct can be known ahead of time. Userspace can always use memfd() if it's generating SIGSTRUCT on the fly.

Quoted 11 lines

hole - a process having FILE__EXECMOD on any file could use that file as a SIGSTRUCT. Probably we'll need a new type in SELinux to label enclave/sigstruct files.

Quoted 6 lines

I don't see a fundamental difference between having RWX in an enclave and RWX in normal memory, either way the process can execute arbitrary code, i.e. PROCESS__EXECMEM is appropriate. Yes, an enclave will #UD on certain instructions, but that's easily sidestepped by having a trampoline in the host (marked RX) and piping arbitrary code into the enclave. Or using EEXIT to do a bit of ROP.

Quoted 4 lines

I'm with you.

With your proposal only FILE__EXECMOD is needed on /dev/sgx/enclave to launch Graphene enclaves or the like.

It wouldn't even need FILE__EXECMOD, assuming Graphene does all of its libc rewriting before building the enclave, i.e. doesn't EADD RWX pages.

Quoted 51 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

No changes are required to LSMs, SGX1 has a single LSM touchpoint in

its

Quoted 2 lines

mmap(), and I *think* the only required userspace change is to mmap() PROT_NONE when allocating the enclave's virtual address

range.

Quoted 3 lines

I'm not sure I understand the motivation behind this proposal to decouple initial EPC permissions from source pages.

Quoted 6 lines

Pulling permissions from source pages means userspace needs to fully map the in normal memory, including marking pages executable. That exposes the loader to having executable pages in its address space that it has no intention of executing (outside of the enclave). And for Graphene, it means having to actively avoid PROCESS__EXECMEM, e.g. by using a dummy backing file to build the enclave instead of anon memory.

Agreed.

Quoted 32 lines
Quoted 6 lines

I don't think it a big deal to fully mmap() enclave files, which have to be parsed by user mode anyway to determine various things including but not limited to the size of heap(s), size and number of TCSs/stacks/TLS areas, and the overall enclave size. So with PHDRs parsed, it's trivial to mmap() each segment with permissions from its

PHDR.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines

As for Graphene, it doesn't need extra permissions to run its enclaves, it just needs a way to install .sigstruct, which is a generic permissions problem and not SGX specific.

For SGX2 maybe:

- No additional requirements to map an EAUG'd page as RW page.

Not

Quoted 1 line
Quoted 1 line
Quoted 1 line

aligned with standard MAP_SHARED behavior, but we really don't

want

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

to require FILE__WRITE, and thus allow writes to .sigstruct.

- Require FILE__EXECMOD on the .sigstruct to map previously

writable

Quoted 1 line
Quoted 1 line
Quoted 1 line

page as executable (which indirectly includes all EAUG'd

pages).

Quoted 1 line
Quoted 1 line
Quoted 1 line

Wiring this up will be a little funky, but we again we don't

want

Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

to require FILE__WRITE on .sigstruct.

I'm lost. Why is EAUG tied to permissions on .sigstruct?

Quoted 7 lines
Quoted 3 lines

Because for the purposes of LSM checks, .sigstruct is the enclave's backing file, and mapping a previously writable enclave page as exectuable is roughly equivalent to mapping a CoW'd page as exectuable.

I think I've got your idea. You are trying to use permissions on .sigstruct to determine whether EAUG will be available to that specific enclave. Am I right?

Yep.

Quoted 9 lines

I'd tie EAUG to the permissions of /dev/sgx/enclave instead. But why? There are couple of reasons. For one, a SIGSTRUCT identifies the behavior of the enclave, hence the SGX features needed by that enclave. So if an enclave requires EAUG, the .sigstruct has to allow EAUG or the enclave wouldn't work. That means the system admin wouldn't have a choice but to match up what's needed by the enclave. For two, whether to allow, say loading code dynamically into an enclave, depends on whether the host process can tolerate the inherent risk. And that decision is seldom made on individual enclaves but to the host process as a whole. And /dev/sgx/enclave serves that purpose.

I think I'd be ok either way? What I really care about is having line of sight to a sane way to support for SGX2, and both seem sane. I.e. we can hash this detail out when SGX2 goes in.

maintainer
#176 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Xing, Cedric
Quoted 55 lines · Cedric
Quoted 55 lines

From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 10:55 AM

Quoted 2 lines · Sean Christopherson

Hmm, I've been thinking more about pulling permissions from the source page. Conceptually I'm not sure we need to meet the same requirements

as

Quoted 1 line

non-enclave DSOs while the enclave is being built, i.e. do we really

need

Quoted 3 lines

to force userspace to fully map the enclave in normal memory?

Consider the Graphene scenario where it's building an enclave on the

fly.

Quoted 2 lines

Pulling permissions from the source VMAs means Graphene has to map the code pages of the enclave with X. This means Graphene will need

EXEDMOD

Quoted 1 line

(or EXECMEM if Graphene isn't careful). In a non-SGX scenario this

makes

Quoted 3 lines

perfect sense since there is no way to verify the end result of RW->RX.

But for SGX, assuming enclaves are whitelisted by their sigstruct

(checked

Quoted 1 line

at EINIT) and because page permissions affect sigstruct.MRENCLAVE, it

*is*

Quoted 1 line

possible to verify the resulting RX contents. E.g. for the purposes

of

Quoted 9 lines

LSMs, can't we use the .sigstruct file as a proxy for the enclave and require FILE__EXECUTE on the .sigstruct inode to map/run the enclave?

Stephen, is my logic sound?

If so...

- Require FILE__READ+FILE__EXECUTE on .sigstruct to mmap() the

enclave.

Quoted 2 lines

- Prevent userspace from mapping the enclave with permissions beyond

the

Quoted 1 line

original permissions of the enclave. This can be done by

populating

Quoted 1 line

VM_MAY{READ,WRITE,EXEC} from the SECINFO (same basic concept as

Andy's

Quoted 1 line

proposals). E.g. pre-EINIT, mmap() and mprotect() can only

succeed

Quoted 3 lines

with PROT_NONE.

- Require FILE__{READ,WRITE,EXECUTE} on /dev/sgx/enclave for

simplicity,

Quoted 1 line

or provide an alternate SGX_IOC_MPROTECT if we want to sidestep

the

Quoted 1 line

FILE__WRITE requirement.

Quoted 7 lines
Quoted 3 lines

One more thought. EADD (and the equivalent SGX2 flow) could do security_mmap_file() with a NULL file on the SECINFO permissions, which would trigger PROCESS_EXECMEM if an enclave attempts to map a page RWX.

If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.

So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.

I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:

int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);

Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)

(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

--Andy

#177 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 13 lines · Andy Lutomirski
Quoted 11 lines · Cedric

If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.

Quoted 4 lines

So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.

Was this in response to Cedric's comment, or to my comment?

Quoted 19 lines

I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:

int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);

Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)

(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)

Quoted 7 lines

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?

What's the value prop beyond whitelisting sigstructs? Realistically, I doubt LSMs/users will want to take the performance hit of scanning the source bytes every time an enclave is loaded.

We could add seomthing like security_enclave_mprotect() in lieu of abusing security_file_mprotect(), but passing the full source bytes seems a bit much.

#178 lore raw

RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 19 lines

From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 1:04 PM

Quoted 1 line · Andy Lutomirski

On Fri, May 24, 2019 at 11:34 AM Xing, Cedric <[email protected]>

Quoted 5 lines · wrote:
Quoted 5 lines

If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable,

as an executable page in EPC.

Quoted 6 lines
Quoted 6 lines

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole

Quoted 8 lines

process hosting it.

Quoted 5 lines

So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.

Was this in response to Cedric's comment, or to my comment?

Creating RWX source page requires PROCESS_EXECMEM. But as I responded to Sean earlier, I think his proposal of "aggregating" all "initial" permission checks into a single SIGSTRUCT check is probably a better approach.

Quoted 10 lines
Quoted 10 lines

I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:

int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);

This is exactly what I was thinking. But with Sean's proposal this is probably no longer necessary.

Quoted 18 lines
Quoted 10 lines

Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)

(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)

Quoted 7 lines

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

Quoted 3 lines

This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?

In the context of SELinux, new types could be defined to be associated with SIGSTRUCT (or more precisely, files containing SIGSTRUCTs). Then the LSM hook (I'd propose security_sgx_initialize_enclave) could enforce whatever...

Quoted 8 lines

What's the value prop beyond whitelisting sigstructs? Realistically, I doubt LSMs/users will want to take the performance hit of scanning the source bytes every time an enclave is loaded.

We could add seomthing like security_enclave_mprotect() in lieu of abusing security_file_mprotect(), but passing the full source bytes seems a bit much.

I'd just use /dev/sgx/enclave to govern "runtime" permissions any EPC page can mmap()/mprotect() to. Then we won't need any code changes in LSM.

-Cedric

maintainer
#179 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson
Quoted 15 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 11 lines · Cedric

If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.

Quoted 6 lines
Quoted 4 lines

So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.

Was this in response to Cedric's comment, or to my comment?

Yours. I think that requiring source pages to be actually mapped W is not such a great idea.

Quoted 28 lines
Quoted 20 lines

I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:

int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);

Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)

(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)

Quoted 7 lines

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

Quoted 3 lines

This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?

In addition.

But I named the function badly and gave it a bad signature, which confused you. Let's try again:

int security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);

Maybe some really fancy future LSM would also want loff_t source_offset, but it's probably not terribly useful. This same callback would be used for EAUG.

Following up on your discussion with Cedric about sigstruct, the other callback would be something like:

int security_enclave_init(struct file *sigstruct_file);

The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:

int security_enclave_load_zeros(unsigned int maxperm);

An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.

But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.

Am I making more sense now?

#180 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 22 lines · Andy Lutomirski
Quoted 20 lines · Sean Christopherson
Quoted 17 lines · Andy Lutomirski
Quoted 11 lines · Cedric

If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.

So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.

Quoted 4 lines
Quoted 1 line

Was this in response to Cedric's comment, or to my comment?

Yours. I think that requiring source pages to be actually mapped W is not such a great idea.

I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:

  if ((SECINFO.PERMS & RWX) == RWX) {
      ret = security_mmap_file(NULL, RWX, ???);
      if (ret)
          return ret;
  }

But that's a moot point if we add security_enclave_load() or whatever.

Quoted 33 lines
Quoted 29 lines
Quoted 21 lines

I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:

int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);

Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)

(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)

Quoted 7 lines

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

Quoted 3 lines

This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?

Quoted 7 lines

In addition.

But I named the function badly and gave it a bad signature, which confused you. Let's try again:

int security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);

I prefer security_enclave_load(), "from_memory" seems redundant at best.

Quoted 7 lines

Maybe some really fancy future LSM would also want loff_t source_offset, but it's probably not terribly useful. This same callback would be used for EAUG.

Following up on your discussion with Cedric about sigstruct, the other callback would be something like:

Quoted 7 lines

int security_enclave_init(struct file *sigstruct_file);

The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:

int security_enclave_load_zeros(unsigned int maxperm);

What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.

Quoted 9 lines

An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.

But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.

Am I making more sense now?

Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:

int security_enclave_load(struct file *file, struct vm_area_struct *vma,

                             unsigned long prot);

And for ultimate flexibility we could pass both .sigstruct and the file pointer for /dev/sgx/enclave, but that seems a bit ridiculous.

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

#181 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 80 lines · Sean Christopherson
Quoted 24 lines · Andy Lutomirski
Quoted 20 lines · Sean Christopherson
Quoted 19 lines · Andy Lutomirski
Quoted 7 lines · Cedric

If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.

Quoted 5 lines

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.

So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.

Quoted 4 lines
Quoted 1 line

Was this in response to Cedric's comment, or to my comment?

Yours. I think that requiring source pages to be actually mapped W is not such a great idea.

I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:

 if ((SECINFO.PERMS & RWX) == RWX) {
     ret = security_mmap_file(NULL, RWX, ???);
     if (ret)
         return ret;

}

But that's a moot point if we add security_enclave_load() or whatever.

Quoted 35 lines
Quoted 29 lines
Quoted 21 lines

I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:

int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);

Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)

(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)

Quoted 7 lines

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

Quoted 3 lines

This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?

In addition.

Quoted 7 lines
Quoted 5 lines

But I named the function badly and gave it a bad signature, which confused you. Let's try again:

int security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);

I prefer security_enclave_load(), "from_memory" seems redundant at best.

Fine with me.

Quoted 14 lines
Quoted 14 lines

Maybe some really fancy future LSM would also want loff_t source_offset, but it's probably not terribly useful. This same callback would be used for EAUG.

Following up on your discussion with Cedric about sigstruct, the other callback would be something like:

int security_enclave_init(struct file *sigstruct_file);

The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:

Quoted 6 lines
Quoted 1 line

int security_enclave_load_zeros(unsigned int maxperm);

What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.

This is how EAUG could ask if RWX is okay. If an enclave is internally doing dynamic loading, the it will need a heap page with maxperm = RWX. (If it’s well designed, it will make it RW and then RX, either by changing SECINFO or by asking the host to mprotect() it, but it still needs the overall RWX mask.).

Also, do real SGX1 enclave formats have BSS? If so, then either we need an ioctl or load zeros or user code is going to load from /dev/zero or just from the heap, but the LSM is going to play better with an ioctl, I suspect :)

Quoted 11 lines
Quoted 10 lines

An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.

But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.

Am I making more sense now?

Quoted 8 lines

Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:

int security_enclave_load(struct file *file, struct vm_area_struct *vma,

                            unsigned long prot);

And for ultimate flexibility we could pass both .sigstruct and the file pointer for /dev/sgx/enclave, but that seems a bit ridiculous.

I agree.

Quoted 6 lines

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

The benefit of putting it on .sigstruct is that it can be per-enclave.

As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty look.

This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.

(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)

#182 lore raw

RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 86 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Friday, May 24, 2019 4:42 PM

Quoted 6 lines · Sean Christopherson
Quoted 5 lines · Andy Lutomirski
Quoted 1 line · Sean Christopherson
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Andy Lutomirski
Quoted 6 lines · Cedric

If "initial permissions" for enclaves are less restrictive than
shared objects, then it'd become a backdoor for circumventing LSM
when enclave whitelisting is *not* in place. For example, an
adversary may load a page, which would otherwise never be executable, as an executable

page in EPC.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

In the case a RWX page is needed, the calling process has to have
a RWX page serving as the source for EADD so PROCESS__EXECMEM will
have been checked. For SGX2, changing an EPC page to RWX is
subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a
security benefit because it only affects the enclave but not the whole process hosting

it.

Quoted 62 lines
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines

So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.

Was this in response to Cedric's comment, or to my comment?

Yours. I think that requiring source pages to be actually mapped W is not such a great idea.

I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:

 if ((SECINFO.PERMS & RWX) == RWX) {
     ret = security_mmap_file(NULL, RWX, ???);
     if (ret)
         return ret;

}

But that's a moot point if we add security_enclave_load() or whatever.

Quoted 38 lines
Quoted 29 lines
Quoted 21 lines

I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:

int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);

Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)

(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)

Quoted 7 lines

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

Quoted 3 lines

This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?

In addition.

But I named the function badly and gave it a bad signature, which confused you. Let's try again:

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

int security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);

I prefer security_enclave_load(), "from_memory" seems redundant at best.

Fine with me.

If we think of EADD as a way of mmap()'ing an enclave file into memory, would this security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Maybe some really fancy future LSM would also want loff_t source_offset, but it's probably not terribly useful. This same callback would be used for EAUG.

EAUG always zeroes the EPC page before making it available to an enclave. So I don't think there's anything needed to done here.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Following up on your discussion with Cedric about sigstruct, the other callback would be something like:

int security_enclave_init(struct file *sigstruct_file);

I'd still insist in using a pointer rather than a file, for reasons that we've discussed before. For those who can't recall, the major reason is that most implementation would embed SIGSTRUCT into the same file as the enclave (or at least I don't want to prevent anyone from doing so), which could also be part of another file, such as a shared object or even the main executable itself. It could be difficult to obtain a fd in those cases. memfd won't work because it can't retain the same attributes of the original file containing the SIGSTRUCT.

After all, what matters is the attributes associated with the backing file, which could be easily retrieve from vm_file of the covering VMA. So for the sake of flexibility, let's stay with what we've agreed before - a pointer to SIGSTRUCT.

Quoted 12 lines
Quoted 11 lines
Quoted 6 lines

The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:

int security_enclave_load_zeros(unsigned int maxperm);

What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.

Quoted 4 lines

This is how EAUG could ask if RWX is okay. If an enclave is internally doing dynamic loading, the it will need a heap page with maxperm = RWX. (If it’s well designed, it will make it RW and then RX, either by changing SECINFO or by asking the host to mprotect() it, but it still needs the overall RWX mask.).

Any new page EAUG'ed will start in RW (as dictated by SGX ISA). EACCEPTCOPY will then change it to RX. RWX is never needed for all practical purposes. This in fact could be gated by mprotect() and the attributes associated with /dev/sgx/enclave. In the case of SELinux, FILE__EXECMOD is the right attribute and mprotect() will take care of all the rest. I don't see why the driver need a role here.

Quoted 4 lines

Also, do real SGX1 enclave formats have BSS? If so, then either we need an ioctl or load zeros or user code is going to load from /dev/zero or just from the heap, but the LSM is going to play better with an ioctl, I suspect :)

Yes, it does. But an enclave would either measure BSS, in which case the initial bytes have to be zero or MRENCLAVE will change; or zero BSS explicitly in its initialization code.

But from LSM's perspective it makes no difference than EADD'ing a page with non-zero content. And security_enclave_load(NULL, RW) would take care of it in exactly in the same way.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.

With SGX2/EDMM, RWX is *never* needed for all practical purposes.

In theory, in terms of security, no page shall be made executable while it is still being prepared. So W and X shall always be mutually exclusive, regardless it's in EPC or regular memory.

RWX is only needed in SGX1, as a workaround for certain usages, because EPCM permissions can never change at runtime.

Quoted 11 lines
Quoted 11 lines
Quoted 7 lines

But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.

Am I making more sense now?

Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:

Quoted 7 lines
Quoted 5 lines

int security_enclave_load(struct file *file, struct vm_area_struct *vma,

                            unsigned long prot);

And for ultimate flexibility we could pass both .sigstruct and the file pointer for /dev/sgx/enclave, but that seems a bit ridiculous.

I agree.

Loosely speaking, an enclave (including initial contents of all of its pages and their permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.

Theoretically speaking, if LSM can make a decision at EINIT by means of security_enclave_load(), then security_enclave_load() is never needed.

In practice, I support keeping both because security_enclave_load() can only approve an enumerable set while security_enclave_load() can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced by security_enclave_load().

Quoted 15 lines
Quoted 7 lines

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

The benefit of putting it on .sigstruct is that it can be per-enclave.

As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty look.

Quoted 8 lines

This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.

(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)

I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX, i.e. FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.

maintainer
#183 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 87 lines · Cedric
Quoted 87 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Friday, May 24, 2019 4:42 PM

Quoted 6 lines · Sean Christopherson
Quoted 5 lines · Andy Lutomirski
Quoted 1 line · Sean Christopherson
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Andy Lutomirski
Quoted 6 lines · Cedric

If "initial permissions" for enclaves are less restrictive than
shared objects, then it'd become a backdoor for circumventing LSM
when enclave whitelisting is *not* in place. For example, an
adversary may load a page, which would otherwise never be executable, as an executable

page in EPC.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

In the case a RWX page is needed, the calling process has to have
a RWX page serving as the source for EADD so PROCESS__EXECMEM will
have been checked. For SGX2, changing an EPC page to RWX is
subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a
security benefit because it only affects the enclave but not the whole process hosting

it.

Quoted 62 lines
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines

So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.

Was this in response to Cedric's comment, or to my comment?

Yours. I think that requiring source pages to be actually mapped W is not such a great idea.

I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:

 if ((SECINFO.PERMS & RWX) == RWX) {
     ret = security_mmap_file(NULL, RWX, ???);
     if (ret)
         return ret;

}

But that's a moot point if we add security_enclave_load() or whatever.

Quoted 38 lines
Quoted 29 lines
Quoted 21 lines

I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:

int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);

Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)

(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)

Quoted 7 lines

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

Quoted 3 lines

This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?

In addition.

But I named the function badly and gave it a bad signature, which confused you. Let's try again:

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

int security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);

I prefer security_enclave_load(), "from_memory" seems redundant at best.

Fine with me.

If we think of EADD as a way of mmap()'ing an enclave file into memory, would this security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?

Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Maybe some really fancy future LSM would also want loff_t source_offset, but it's probably not terribly useful. This same callback would be used for EAUG.

EAUG always zeroes the EPC page before making it available to an enclave. So I don't think there's anything needed to done here.

Duh. So security_enclave_load_zeros() for EAUG. See below.

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Following up on your discussion with Cedric about sigstruct, the other callback would be something like:

int security_enclave_init(struct file *sigstruct_file);

I'd still insist in using a pointer rather than a file, for reasons that we've discussed before. For those who can't recall, the major reason is that most implementation would embed SIGSTRUCT into the same file as the enclave (or at least I don't want to prevent anyone from doing so), which could also be part of another file, such as a shared object or even the main executable itself. It could be difficult to obtain a fd in those cases. memfd won't work because it can't retain the same attributes of the original file containing the SIGSTRUCT.

After all, what matters is the attributes associated with the backing file, which could be easily retrieve from vm_file of the covering VMA. So for the sake of flexibility, let's stay with what we've agreed before - a pointer to SIGSTRUCT.

I'm okay with this, except for one nastiness: there's a big difference between a file that is just a sigstruct and a file that contains essentially arbitrary data plus a sigstruct at an arbitrary offset. We could do something tricky like saying that SIGSTRUCT can be in a file that's just a SIGSTRUCT or it can be in a special SIGSTRUCT ELF note in a file that isn't just a SIGSTRUCT, but that could be annoyingly restrictive.

If it's going to be in an arbitrary file, then I think the signature needs to be more like:

int security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);

So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines

The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:

int security_enclave_load_zeros(unsigned int maxperm);

What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.

Quoted 6 lines
Quoted 4 lines

This is how EAUG could ask if RWX is okay. If an enclave is internally doing dynamic loading, the it will need a heap page with maxperm = RWX. (If it’s well designed, it will make it RW and then RX, either by changing SECINFO or by asking the host to mprotect() it, but it still needs the overall RWX mask.).

Any new page EAUG'ed will start in RW (as dictated by SGX ISA). EACCEPTCOPY will then change it to RX. RWX is never needed for all practical purposes. This in fact could be gated by mprotect() and the attributes associated with /dev/sgx/enclave. In the case of SELinux, FILE__EXECMOD is the right attribute and mprotect() will take care of all the rest. I don't see why the driver need a role here.

I find the SDM's discussion of EAUG, EACCEPT, and EACCEPTCOPY to be extremely confusing. My copy of the SDM has EACCEPT's SECINFO argument as "Read access permitted by Non Enclave". Is that an error?
And is EACCEPTCOPY just EACCEPT + memcpy or is there some other
fundamental difference? 38.5.7 doesn't even mention EACCEPTCOPY.

Anyway, all my confusion aside, I was talking about the page table, not the EPCM. I think the enclave should need permission to write its own content into a page that will ever become X, and the enclave's untrusted host library would do this by adding the page with MAXPERM=RWX and then mapping/mprotecting it as PROT_WRITE and then (later or simultaneously) PROT_EXEC.

Since SGX2 doesn't seem to have a way to add an initialized page to EPC after an enclave starts, I guess that it's impossible to have the enclave do something like dlopen() without MAXPERM=RWX. So be it. Maybe someone will find this annoying someday and SGX3 will add EAUG-but-don't-zero and EACCEPT-with-existing-contents.

Quoted 9 lines
Quoted 5 lines

Also, do real SGX1 enclave formats have BSS? If so, then either we need an ioctl or load zeros or user code is going to load from /dev/zero or just from the heap, but the LSM is going to play better with an ioctl, I suspect :)

Yes, it does. But an enclave would either measure BSS, in which case the initial bytes have to be zero or MRENCLAVE will change; or zero BSS explicitly in its initialization code.

But from LSM's perspective it makes no difference than EADD'ing a page with non-zero content. And security_enclave_load(NULL, RW) would take care of it in exactly in the same way.

Sure, I suppose the same hook with NULL parameters would be equivalent.

Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.

With SGX2/EDMM, RWX is *never* needed for all practical purposes.

In theory, in terms of security, no page shall be made executable while it is still being prepared. So W and X shall always be mutually exclusive, regardless it's in EPC or regular memory.

RWX is only needed in SGX1, as a workaround for certain usages, because EPCM permissions can never change at runtime.

As above, I think I disagree. MAXPERM is intended as an upper bound on the permissions that a page can ever have, at least until it's EREMOVEd and re-added. Since there's no EAUG-but-don't-zero, EAUG with MAXPERM.W=0 is basically useless because the page can never contain anything other than zeros, so a dynamically allocated page that is ever executed has to have MAXPERM=RWX or MAXPERM=WX. And that will need special permissions, which I think is consistent with your recent emails on how this could all map to SELinux permissions.

Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 8 lines

But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.

Am I making more sense now?

Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:

int security_enclave_load(struct file *file, struct vm_area_struct *vma,

                            unsigned long prot);

And for ultimate flexibility we could pass both .sigstruct and the file pointer for /dev/sgx/enclave, but that seems a bit ridiculous.

Quoted 7 lines
Quoted 1 line

I agree.

Loosely speaking, an enclave (including initial contents of all of its pages and their permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.

Theoretically speaking, if LSM can make a decision at EINIT by means of security_enclave_load(), then security_enclave_load() is never needed.

In practice, I support keeping both because security_enclave_load() can only approve an enumerable set while security_enclave_load() can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced by security_enclave_load().

I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also, security_enclave_load() provides no protection against loading a mishmash of two different enclave files. I see security_enclave_init() as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" and security_enclave_load() as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."

Quoted 19 lines
Quoted 19 lines
Quoted 8 lines

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

The benefit of putting it on .sigstruct is that it can be per-enclave.

As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty look.

This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.

Quoted 7 lines
Quoted 5 lines

(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)

I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX, i.e. FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.

Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual mprotect() / mmap() permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply that PROT_WRITE | PROT_EXEC is allowed.

#184 lore raw

RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 102 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PM

Quoted 5 lines · Cedric
Quoted 5 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Friday, May 24, 2019 4:42 PM

Quoted 1 line

On May 24, 2019, at 3:41 PM, Sean Christopherson <[email protected]>

Quoted 92 lines · wrote:
Quoted 91 lines
Quoted 5 lines
Quoted 5 lines · Andy Lutomirski
Quoted 1 line · Sean Christopherson
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Andy Lutomirski
Quoted 7 lines · Cedric

If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable

page in EPC.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting

it.

Quoted 60 lines
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines

So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.

Was this in response to Cedric's comment, or to my comment?

Yours. I think that requiring source pages to be actually mapped W is not such a great idea.

I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:

 if ((SECINFO.PERMS & RWX) == RWX) {
     ret = security_mmap_file(NULL, RWX, ???);
     if (ret)
         return ret;

}

But that's a moot point if we add security_enclave_load() or whatever.

Quoted 36 lines
Quoted 30 lines
Quoted 22 lines

I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:

int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);

Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)

(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)

Quoted 7 lines

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

Quoted 3 lines

This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?

In addition.

Quoted 7 lines
Quoted 5 lines

But I named the function badly and gave it a bad signature, which confused you. Let's try again:

int security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);

I prefer security_enclave_load(), "from_memory" seems redundant at best.

Fine with me.

Quoted 7 lines
Quoted 1 line

If we think of EADD as a way of mmap()'ing an enclave file into memory, would this

security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?

Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.

What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.

But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to security_mmap_file() by default?

Quoted 32 lines
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Maybe some really fancy future LSM would also want loff_t source_offset, but it's probably not terribly useful. This same callback would be used for EAUG.

EAUG always zeroes the EPC page before making it available to an enclave. So I don't

think there's anything needed to done here.

Duh. So security_enclave_load_zeros() for EAUG. See below.

Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

Following up on your discussion with Cedric about sigstruct, the other callback would be something like:

int security_enclave_init(struct file *sigstruct_file);

I'd still insist in using a pointer rather than a file, for reasons that we've discussed

before. For those who can't recall, the major reason is that most implementation would embed SIGSTRUCT into the same file as the enclave (or at least I don't want to prevent anyone from doing so), which could also be part of another file, such as a shared object or even the main executable itself. It could be difficult to obtain a fd in those cases. memfd won't work because it can't retain the same attributes of the original file containing the SIGSTRUCT.

Quoted 2 lines

After all, what matters is the attributes associated with the backing file, which could

be easily retrieve from vm_file of the covering VMA. So for the sake of flexibility, let's stay with what we've agreed before - a pointer to SIGSTRUCT.

Quoted 6 lines

I'm okay with this, except for one nastiness: there's a big difference between a file that is just a sigstruct and a file that contains essentially arbitrary data plus a sigstruct at an arbitrary offset.
We could do something tricky like saying that SIGSTRUCT can be in a file that's just a SIGSTRUCT or it can be in a special SIGSTRUCT ELF note in a file that isn't just a SIGSTRUCT, but that could be annoyingly restrictive.

Agreed. Approving a file implies approving all SIGSTRUCTs within that file. But I guess it wouldn't cause practical problems.

Quoted 8 lines

If it's going to be in an arbitrary file, then I think the signature needs to be more like:

int security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);

So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.

Wouldn't the pair of { sigstruct_vma, sigstruct_offset } be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer - vma->vm_start)?

Quoted 22 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 9 lines

The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also
want:

int security_enclave_load_zeros(unsigned int maxperm);

What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.

Quoted 4 lines
Quoted 4 lines

This is how EAUG could ask if RWX is okay. If an enclave is
internally doing dynamic loading, the it will need a heap page with maxperm = RWX. (If it’s well designed, it will make it RW and then RX, either by changing SECINFO or by asking the host to mprotect() it, but it still

needs the overall RWX mask.).

Quoted 2 lines

Any new page EAUG'ed will start in RW (as dictated by SGX ISA). EACCEPTCOPY will then

Quoted 8 lines

change it to RX. RWX is never needed for all practical purposes. This in fact could be gated by mprotect() and the attributes associated with /dev/sgx/enclave. In the case of SELinux, FILE__EXECMOD is the right attribute and mprotect() will take care of all the rest. I don't see why the driver need a role here.

I find the SDM's discussion of EAUG, EACCEPT, and EACCEPTCOPY to be extremely confusing. My copy of the SDM has EACCEPT's SECINFO argument as "Read access permitted by Non Enclave". Is that an error?

I'm confused by those descriptions too. Guess I cannot comment if that's an error or not.

Anyway, per our internal documents, for EAUG, SECINFO has to be set to PT_REG|RW. For EACCEPT, SGX ISA compares supplied SECINFO with EPCM attributes and returns an error if they don't match. EACCEPTCOPY only works on pending pages (i.e. SECINFO.P must be set), and sets EPCM access permissions to whatever supplied in SECINFO.

Quoted 2 lines

And is EACCEPTCOPY just EACCEPT + memcpy or is there some other fundamental difference?
38.5.7 doesn't even mention EACCEPTCOPY.

2 differences: 1) EACCEPT only *compares* but EACCEPTCOPY *sets* EPCM permissions; and 2) EACCEPTCOPY does EACCEPT+memcpy atomically.

Quoted 6 lines

Anyway, all my confusion aside, I was talking about the page table, not the EPCM. I think the enclave should need permission to write its own content into a page that will ever become X, and the enclave's untrusted host library would do this by adding the page with MAXPERM=RWX and then mapping/mprotecting it as PROT_WRITE and then (later or simultaneously) PROT_EXEC.

I was talking about the same thing. A code page in EPC will start in RW (both EPCM and PTE) and end in RX (both EPCM and PTE). EACCEPTCOPY takes care of EPCM, while mprotect() could take care of PTE as long as /dev/sgx/enclave has FILE__EXECMOD.

I understand your intention to enclave pages to segments with different MAXPERMs. My concern is though the host process may not always have a priori knowledge on which ranges to be used as code vs. data. After all, only the weakest link matters in security so I think what a host process cares would be whether the enclave loads code dynamically, or expands its data segments only, or neither. And for that reason, I think it more "user friendly" to keep just one MAXPERM - i.e. the most permissive one. Then we could associate that with /dev/sgx/enclave so as to relieve the driver from keeping track of too many things.

Quoted 4 lines

Since SGX2 doesn't seem to have a way to add an initialized page to EPC after an enclave starts, I guess that it's impossible to have the enclave do something like dlopen() without MAXPERM=RWX. So be it.

That's true. The reason behind it is SGX doesn’t trust anything from outside. So non-predetermined contents must be measured (e.g. EADD+EEXTEND), but it's hard to measure (or attest to) dynamically added contents so we decided to allow predetermined contents (i.e. all zeros in the case of EAUG) only.

Quoted 2 lines

Maybe someone will find this annoying someday and SGX3 will add EAUG-but-don't-zero and EACCEPT-with-existing-contents.

From security perspective, accepting a page that is measured/hashed to XYZ is equivalent to overwriting that page with content hashed to XYZ. EACCEPTCOPY actually does the latter. The annoying part is due to the mismatch between SGX ISA and the s/w model adopted by LSM, but that has nothing to do with security.

Quoted 33 lines
Quoted 9 lines
Quoted 7 lines

Also, do real SGX1 enclave formats have BSS? If so, then either we need an ioctl or load zeros or user code is going to load from /dev/zero or just from the heap, but the LSM is going to play better with an ioctl, I suspect :)

Yes, it does. But an enclave would either measure BSS, in which case the initial bytes

have to be zero or MRENCLAVE will change; or zero BSS explicitly in its initialization code.

Quoted 2 lines

But from LSM's perspective it makes no difference than EADD'ing a page with non-zero

content. And security_enclave_load(NULL, RW) would take care of it in exactly in the same way.

Sure, I suppose the same hook with NULL parameters would be equivalent.

Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.

With SGX2/EDMM, RWX is *never* needed for all practical purposes.

In theory, in terms of security, no page shall be made executable while it is still

being prepared. So W and X shall always be mutually exclusive, regardless it's in EPC or regular memory.

Quoted 2 lines

RWX is only needed in SGX1, as a workaround for certain usages, because EPCM permissions

can never change at runtime.

Quoted 7 lines

As above, I think I disagree. MAXPERM is intended as an upper bound on the permissions that a page can ever have, at least until it's EREMOVEd and re-added. Since there's no EAUG-but-don't-zero, EAUG with MAXPERM.W=0 is basically useless because the page can never contain anything other than zeros, so a dynamically allocated page that is ever executed has to have MAXPERM=RWX or MAXPERM=WX. And that will need special permissions, which I think is consistent with your recent emails on how this could all map to SELinux permissions.

I'm totally with you. What I was trying to say was that only W or X would be needed at any given time. That said, MAXPERM=RWX but PROCESS__EXECMEM will not be needed, while FILE__EXECMOD will be needed only on /dev/sgx/enclave. So the inherent risk is contained.

Quoted 35 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 9 lines

But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.

Am I making more sense now?

Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:

int security_enclave_load(struct file *file, struct vm_area_struct *vma,

                            unsigned long prot);
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

And for ultimate flexibility we could pass both .sigstruct and the file pointer for /dev/sgx/enclave, but that seems a bit ridiculous.

I agree.

Loosely speaking, an enclave (including initial contents of all of its pages and their

permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.

Quoted 2 lines

Theoretically speaking, if LSM can make a decision at EINIT by means of

security_enclave_load(), then security_enclave_load() is never needed.

Quoted 2 lines

In practice, I support keeping both because security_enclave_load() can only approve an

enumerable set while security_enclave_load() can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced by security_enclave_load().

Quoted 7 lines

I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also, security_enclave_load() provides no protection against loading a mishmash of two different enclave files. I see
security_enclave_init() as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT"
and security_enclave_load() as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."

Sorry for the confusion. I was saying the same thing except that the decision of security_enclave_load() doesn't have to depend on SIGSTRUCT. Given your prototype of security_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".

Quoted 29 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

Quoted 6 lines
Quoted 6 lines

The benefit of putting it on .sigstruct is that it can be per-enclave.

As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty

look.

Quoted 12 lines
Quoted 11 lines

This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.

(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)

Quoted 7 lines
Quoted 1 line

I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX, i.e.

FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.

Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual mprotect() /
mmap() permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply that PROT_WRITE | PROT_EXEC is allowed.

If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?

To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of /dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on /dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of -EPERM returned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of IOC_EINIT.

#185 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 15 lines · Cedric
Quoted 7 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PM

Quoted 3 lines · Cedric

If we think of EADD as a way of mmap()'ing an enclave file into memory, would this

Quoted 7 lines

security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?

Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.

Quoted 8 lines

What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.

But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to security_mmap_file() by default?

Mapping the enclave will still go through security_mmap_file(), the extra security_enclave_load() hook allows the mmap() to use PROT_NONE.

Quoted 11 lines
Quoted 7 lines

If it's going to be in an arbitrary file, then I think the signature needs to be more like:

int security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);

So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.

Wouldn't the pair of { sigstruct_vma, sigstruct_offset } be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer - vma->vm_start)?

VMA has vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?

  int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);
Quoted 22 lines
Quoted 14 lines
Quoted 1 line

Loosely speaking, an enclave (including initial contents of all of its pages and their

permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.

Quoted 2 lines

Theoretically speaking, if LSM can make a decision at EINIT by means of

security_enclave_load(), then security_enclave_load() is never needed.

Quoted 2 lines

In practice, I support keeping both because security_enclave_load() can only approve an

enumerable set while security_enclave_load() can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced by security_enclave_load().

Quoted 7 lines

I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also, security_enclave_load() provides no protection against loading a mishmash of two different enclave files. I see security_enclave_init() as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" and security_enclave_load() as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."

Quoted 6 lines

Sorry for the confusion. I was saying the same thing except that the decision of security_enclave_load() doesn't have to depend on SIGSTRUCT. Given your prototype of security_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".

Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling /dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using /dev/sgx/enclave means users can't do per-enclave permissions, period.

What is your objection to working on the sigstruct?

Quoted 25 lines
Quoted 25 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

Quoted 6 lines
Quoted 6 lines

The benefit of putting it on .sigstruct is that it can be per-enclave.

As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty

look.

Quoted 12 lines
Quoted 11 lines

This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.

(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX, i.e.

FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.

Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual mprotect() / mmap() permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply that PROT_WRITE | PROT_EXEC is allowed.

Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that sgx_enclave_load() with a NULL vma implies a transition from RW. For example, SELinux would interpret sgx_enclave_load(NULL, RX) as requiring FILE__EXECMOD.

As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e. sgx_enclave_load(NULL, RX) already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs from MAY_RW to MAY_RX.

For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit ioctl() to add the to-be-EAUG'd range. That just leaves us wanting an ioctl() to set the post-EACCEPT{COPY} permissions.

E.g.:

    ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */

    mprotect(addr, size, RW);
    ...

    EACCEPTCOPY -> EAUG /* page fault handler */

    ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */

    mprotect(addr, size, RX);

    ... 

And making ACTIVATE_REGION a single-shot per page eliminates the need for the MAXPERMS concept (see below).

Quoted 15 lines

If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?

To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of /dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on /dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of -EPERM returned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of IOC_EINIT.

Aha!

Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:

- Initial permissions for *EADD* pages are explicitly handled via sgx_enclave_load() with the exact SECINFO permissions.

- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.

- Runtime permissions break down as follows:

      R   - N/A, subset of RW (EAUG)
      W   - N/A, subset of RW (EAUG) and x86 paging can't do W
      X   - N/A, subset of RX (x86 paging can't do XO)
      RW  - Handled by EAUG LSM hook (uses RW unconditionally)
      WX  - N/A, subset of RWX (x86 paging can't do WX)
      RX  - Handled by ACTIVATE_REGION

RWX - Handled by ACTIVATE_REGION

In other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):

- <prefix>_ACTIVATE_REGION(vma, perms) -> sgx_enclave_load(NULL, perms)

- <prefix>_ADD_REGION(vma) -> sgx_enclave_load(vma, SECINFO.perms)

- <prefix>_ADD_REGION(NULL) -> sgx_enclave_load(NULL, RW)

then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of sgx_enclave_load(..., RW) would need to be different than normal shared memory, e.g. FILE__WRITE should *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without* FILE__EXECMOD, it'd have the necessary information to do so.

The userspace changes are fairly minimal:

- For SGX1, use PROT_NONE for the initial mmap() and refactor ADD_PAGE to ADD_REGION.

- For SGX2, do an explicit ADD_REGION on the ranges to be EAUG'd, and an ACTIVATE_REGION to make a region RX or R (no extra ioctl() required to keep RW permissions).

Because ACTIVATE_REGION can only be done once per page, to do *abitrary* mprotect() transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.

- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.

- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.

maintainer
#186 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 117 lines · Sean Christopherson
Quoted 15 lines · Cedric
Quoted 7 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PM

Quoted 3 lines · Cedric

If we think of EADD as a way of mmap()'ing an enclave file into memory, would this

Quoted 7 lines

security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?

Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.

Quoted 8 lines

What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.

But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to security_mmap_file() by default?

Mapping the enclave will still go through security_mmap_file(), the extra security_enclave_load() hook allows the mmap() to use PROT_NONE.

Quoted 11 lines
Quoted 7 lines

If it's going to be in an arbitrary file, then I think the signature needs to be more like:

int security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);

So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.

Wouldn't the pair of { sigstruct_vma, sigstruct_offset } be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer - vma->vm_start)?

VMA has vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?

  int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);
Quoted 22 lines
Quoted 14 lines
Quoted 1 line

Loosely speaking, an enclave (including initial contents of all of its pages and their

permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.

Quoted 2 lines

Theoretically speaking, if LSM can make a decision at EINIT by means of

security_enclave_load(), then security_enclave_load() is never needed.

Quoted 2 lines

In practice, I support keeping both because security_enclave_load() can only approve an

enumerable set while security_enclave_load() can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced by security_enclave_load().

Quoted 7 lines

I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also, security_enclave_load() provides no protection against loading a mishmash of two different enclave files. I see security_enclave_init() as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" and security_enclave_load() as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."

Quoted 6 lines

Sorry for the confusion. I was saying the same thing except that the decision of security_enclave_load() doesn't have to depend on SIGSTRUCT. Given your prototype of security_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".

Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling /dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using /dev/sgx/enclave means users can't do per-enclave permissions, period.

What is your objection to working on the sigstruct?

Quoted 25 lines
Quoted 25 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

Quoted 6 lines
Quoted 6 lines

The benefit of putting it on .sigstruct is that it can be per-enclave.

As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty

look.

Quoted 12 lines
Quoted 11 lines

This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.

(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX, i.e.

FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.

Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual mprotect() / mmap() permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply that PROT_WRITE | PROT_EXEC is allowed.

Quoted 4 lines

Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that sgx_enclave_load() with a NULL vma implies a transition from RW. For example, SELinux would interpret sgx_enclave_load(NULL, RX) as requiring FILE__EXECMOD.

You lost me here. What operation triggers this callback? And wouldn't sgx_enclave_load(NULL, RX) sometimes be a transition from RO or just some fresh executable zero bytes?

Quoted 12 lines

As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e. sgx_enclave_load(NULL, RX) already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs from MAY_RW to MAY_RX.

For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit ioctl() to add the to-be-EAUG'd range. That just leaves us wanting an ioctl() to set the post-EACCEPT{COPY} permissions.

E.g.:

Quoted 10 lines
    ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */

    mprotect(addr, size, RW);
    ...

    EACCEPTCOPY -> EAUG /* page fault handler */

    ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */

    mprotect(addr, size, RX);

In the maxperm model, this mprotect() will fail unless MAXPERM contains RX, which could only happen if MAXPERM=RWX. So, regardless of how it's actually mapped to SELinux policy, MAXPERM=RWX is functionally like EXECMOD and actual RWX PTEs are functionally like EXECMEM.

Quoted 30 lines
    ...

And making ACTIVATE_REGION a single-shot per page eliminates the need for the MAXPERMS concept (see below).

Quoted 15 lines

If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?

To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of /dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on /dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of -EPERM returned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of IOC_EINIT.

Aha!

Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:

- Initial permissions for *EADD* pages are explicitly handled via sgx_enclave_load() with the exact SECINFO permissions.

Quoted 7 lines

- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.

- Runtime permissions break down as follows:

      R   - N/A, subset of RW (EAUG)
      W   - N/A, subset of RW (EAUG) and x86 paging can't do W
      X   - N/A, subset of RX (x86 paging can't do XO)

Sure it can! You just have a hypervisor that maps a PA bit to EPT no-read. Then you can use that PA bit to suppress read. Also, Linux already abuses PKRU to simulate XO, although that won't work for enclaves.

Quoted 26 lines
      RW  - Handled by EAUG LSM hook (uses RW unconditionally)
      WX  - N/A, subset of RWX (x86 paging can't do WX)
      RX  - Handled by ACTIVATE_REGION

RWX - Handled by ACTIVATE_REGION

In other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):

- <prefix>_ACTIVATE_REGION(vma, perms) -> sgx_enclave_load(NULL, perms)

- <prefix>_ADD_REGION(vma) -> sgx_enclave_load(vma, SECINFO.perms)

- <prefix>_ADD_REGION(NULL) -> sgx_enclave_load(NULL, RW)

then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of sgx_enclave_load(..., RW) would need to be different than normal shared memory, e.g. FILE__WRITE should *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without* FILE__EXECMOD, it'd have the necessary information to do so.

The userspace changes are fairly minimal:

- For SGX1, use PROT_NONE for the initial mmap() and refactor ADD_PAGE to ADD_REGION.

Quoted 9 lines

- For SGX2, do an explicit ADD_REGION on the ranges to be EAUG'd, and an ACTIVATE_REGION to make a region RX or R (no extra ioctl() required to keep RW permissions).

Because ACTIVATE_REGION can only be done once per page, to do *abitrary* mprotect() transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.

I may be misunderstanding or just be biased to my own proposal, but this seems potentially more complicated and less flexible than the MAXPERM model. One of the main things that made me come up with MAXPERM is that I wanted to avoid any complicated PTE/VMA modification or runtime changes. So, with MAXPERM, we still need to track the MAXPERM bits per page, but we don't ever need to *change* them or to worry about what is or is not mapped anywhere at any given time. With ACTIVATE_REGION, don't we need to make sure that we don't have a second VMA pointing at the same pages? Or am I just confused?

Quoted 2 lines

- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.

I may still be missing something, but, for SGX1, it's possible at least in principle for the enclave to request, via ocall or similar, that the untrusted runtime do mprotect(). It's not even such a bad idea. Honestly, enclaves *shouldn't* have anything actually writable and executable at once because the enclaves don't want to be easily exploited.

Quoted 5 lines

- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.

#187 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 11 lines · Andy Lutomirski
Quoted 5 lines · Sean Christopherson

Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that sgx_enclave_load() with a NULL vma implies a transition from RW. For example, SELinux would interpret sgx_enclave_load(NULL, RX) as requiring FILE__EXECMOD.

You lost me here. What operation triggers this callback? And wouldn't sgx_enclave_load(NULL, RX) sometimes be a transition from RO or just some fresh executable zero bytes?

An explicit ioctl() after EACCEPTCOPY to update the allowed permissions. For all intents and purposes, the EAUG'd page must start RW. Maybe a better way to phrase it is that at some point the page must be writable to have any value whatsover. EACCEPTCOPY explicitly requires the page to be at least RW. EACCEPT technically doesn't require RW, but a RO or RX zero page is useless. Userspace could still EACCEPT with RO or RX, but SGX would assume a minimum of RW for the purposes of the LSM check.

Quoted 20 lines
Quoted 20 lines

As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e. sgx_enclave_load(NULL, RX) already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs from MAY_RW to MAY_RX.

For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit ioctl() to add the to-be-EAUG'd range. That just leaves us wanting an ioctl() to set the post-EACCEPT{COPY} permissions.

E.g.:

    ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */

    mprotect(addr, size, RW);
    ...

    EACCEPTCOPY -> EAUG /* page fault handler */

    ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */
Quoted 7 lines
Quoted 1 line
    mprotect(addr, size, RX);

In the maxperm model, this mprotect() will fail unless MAXPERM contains RX, which could only happen if MAXPERM=RWX. So, regardless of how it's actually mapped to SELinux policy, MAXPERM=RWX is functionally like EXECMOD and actual RWX PTEs are functionally like EXECMEM.

Yep, same idea, except in the proposed flow ACTIVATE_REGION.

Quoted 37 lines
Quoted 29 lines
    ...

And making ACTIVATE_REGION a single-shot per page eliminates the need for the MAXPERMS concept (see below).

Quoted 15 lines

If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?

To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of /dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on /dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of -EPERM returned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of IOC_EINIT.

Aha!

Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:

- Initial permissions for *EADD* pages are explicitly handled via sgx_enclave_load() with the exact SECINFO permissions.

Quoted 7 lines

- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.

- Runtime permissions break down as follows:

      R   - N/A, subset of RW (EAUG)
      W   - N/A, subset of RW (EAUG) and x86 paging can't do W
      X   - N/A, subset of RX (x86 paging can't do XO)
Quoted 4 lines

Sure it can! You just have a hypervisor that maps a PA bit to EPT no-read. Then you can use that PA bit to suppress read. Also, Linux already abuses PKRU to simulate XO, although that won't work for enclaves.

Heh, I intentionally said "x86 paging" to rule out EPT :-) I'm pretty sure it's a moot point though, I have a hard time believing an LSM will allow RW->X and not RW->RX.

Quoted 36 lines
Quoted 26 lines
      RW  - Handled by EAUG LSM hook (uses RW unconditionally)
      WX  - N/A, subset of RWX (x86 paging can't do WX)
      RX  - Handled by ACTIVATE_REGION

RWX - Handled by ACTIVATE_REGION

In other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):

- <prefix>_ACTIVATE_REGION(vma, perms) -> sgx_enclave_load(NULL, perms)

- <prefix>_ADD_REGION(vma) -> sgx_enclave_load(vma, SECINFO.perms)

- <prefix>_ADD_REGION(NULL) -> sgx_enclave_load(NULL, RW)

then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of sgx_enclave_load(..., RW) would need to be different than normal shared memory, e.g. FILE__WRITE should *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without* FILE__EXECMOD, it'd have the necessary information to do so.

The userspace changes are fairly minimal:

- For SGX1, use PROT_NONE for the initial mmap() and refactor ADD_PAGE to ADD_REGION.

Quoted 9 lines

- For SGX2, do an explicit ADD_REGION on the ranges to be EAUG'd, and an ACTIVATE_REGION to make a region RX or R (no extra ioctl() required to keep RW permissions).

Because ACTIVATE_REGION can only be done once per page, to do *abitrary* mprotect() transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.

Quoted 9 lines

I may be misunderstanding or just be biased to my own proposal, but this seems potentially more complicated and less flexible than the MAXPERM model. One of the main things that made me come up with MAXPERM is that I wanted to avoid any complicated PTE/VMA modification or runtime changes. So, with MAXPERM, we still need to track the MAXPERM bits per page, but we don't ever need to *change* them or to worry about what is or is not mapped anywhere at any given time. With ACTIVATE_REGION, don't we need to make sure that we don't have a second VMA pointing at the same pages? Or am I just confused?

In theory, it's still your MAXPERM model, but with the unnecessary states removed and the others enforced/handled by the natural SGX transitions instead of explictly in ioctls. Underneath the hood the SGX driver would still need to track the MAXPERM.

With SGX1, SECINFO == MAXPERM. With SGX2, ACTIVATE_REGION == MAXPERM, with the implication that the previous state is always RW.

Quoted 9 lines
Quoted 2 lines

- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.

I may still be missing something, but, for SGX1, it's possible at least in principle for the enclave to request, via ocall or similar, that the untrusted runtime do mprotect(). It's not even such a bad idea. Honestly, enclaves *shouldn't* have anything actually writable and executable at once because the enclaves don't want to be easily exploited.

Yes, but the *EPCM* permissions are immutable. So if an enclave wants to do RW->RX it has to intialize its pages to RWX. And because the untrusted runtime is, ahem, untrusted, the enclave cannot rely on userspace to never map its pages RWX. In other words, from a enclave security perspective, an SGX1 enclave+runtime that uses RW->RX is no different than an enclave that uses RWX. Using your earlier terminology, an SGX1 enclave *should* get a dirty looks if maps a page RWX in the EPCM, even if it only intends RW->RX behavior.

Quoted 4 lines
Quoted 4 lines

- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.

#188 lore raw

RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 16 lines

From: Christopherson, Sean J
Sent: Tuesday, May 28, 2019 2:41 PM

Quoted 11 lines · Andy Lutomirski
Quoted 5 lines · Sean Christopherson

Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that sgx_enclave_load() with a NULL vma implies a transition from RW. For example, SELinux would interpret sgx_enclave_load(NULL, RX) as requiring FILE__EXECMOD.

You lost me here. What operation triggers this callback? And wouldn't sgx_enclave_load(NULL, RX) sometimes be a transition from RO or just some fresh executable zero bytes?

Quoted 6 lines

An explicit ioctl() after EACCEPTCOPY to update the allowed permissions. For all intents and purposes, the EAUG'd page must start RW. Maybe a better way to phrase it is that at some point the page must be writable to have any value whatsover. EACCEPTCOPY explicitly requires the page to be at least RW. EACCEPT technically doesn't require RW, but a RO or RX zero page is useless. Userspace could still EACCEPT with RO or RX, but SGX would assume a minimum of RW for the purposes of the LSM check.

Why is an explicit ioctl() necessary after EACCEPTCOPY? Or why is mprotect() not sufficient?

I tend to agree on Andy's MAXPERM model where MAXPERM never changes once established.

Quoted 71 lines
Quoted 21 lines
Quoted 21 lines

As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e. sgx_enclave_load(NULL, RX) already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs from MAY_RW to MAY_RX.

For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit ioctl() to add the to-be-EAUG'd range. That just leaves us wanting an ioctl() to set the post-EACCEPT{COPY} permissions.

E.g.:

    ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */

    mprotect(addr, size, RW);
    ...

    EACCEPTCOPY -> EAUG /* page fault handler */

    ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */
Quoted 7 lines
Quoted 1 line
    mprotect(addr, size, RX);

In the maxperm model, this mprotect() will fail unless MAXPERM contains RX, which could only happen if MAXPERM=RWX. So, regardless of how it's actually mapped to SELinux policy, MAXPERM=RWX is functionally like EXECMOD and actual RWX PTEs are functionally like EXECMEM.

Yep, same idea, except in the proposed flow ACTIVATE_REGION.

Quoted 8 lines
Quoted 8 lines
    ...

And making ACTIVATE_REGION a single-shot per page eliminates the need for the MAXPERMS concept (see below).

Quoted 3 lines

If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?

Quoted 10 lines
Quoted 10 lines
Quoted 10 lines

To be a bit more clear, system admin sets MAXPERM upper bound in
the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of
/dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on /dev/sgx/enclave, then everything
will just work. Otherwise, it fails in the form of -EPERM returned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care
of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still
required for initial code pages even though system admins could disable dynamically

loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial"

Quoted 19 lines
Quoted 18 lines
Quoted 2 lines

mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of IOC_EINIT.

Aha!

Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:

- Initial permissions for *EADD* pages are explicitly handled via sgx_enclave_load() with the exact SECINFO permissions.

- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.

- Runtime permissions break down as follows:

      R   - N/A, subset of RW (EAUG)
      W   - N/A, subset of RW (EAUG) and x86 paging can't do W
      X   - N/A, subset of RX (x86 paging can't do XO)
Quoted 18 lines
Quoted 4 lines

Sure it can! You just have a hypervisor that maps a PA bit to EPT no-read. Then you can use that PA bit to suppress read. Also, Linux already abuses PKRU to simulate XO, although that won't work for enclaves.

Heh, I intentionally said "x86 paging" to rule out EPT :-) I'm pretty sure it's a moot point though, I have a hard time believing an LSM will allow RW->X and not RW->RX.

Quoted 10 lines
Quoted 10 lines
      RW  - Handled by EAUG LSM hook (uses RW unconditionally)
      WX  - N/A, subset of RWX (x86 paging can't do WX)
      RX  - Handled by ACTIVATE_REGION

RWX - Handled by ACTIVATE_REGION

In other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):

- <prefix>_ACTIVATE_REGION(vma, perms) -> sgx_enclave_load(NULL,
perms)

I'm not sure on what security_enclave_load()'s decision would be based.

Quoted 38 lines
Quoted 28 lines
Quoted 21 lines

- <prefix>_ADD_REGION(vma) -> sgx_enclave_load(vma, SECINFO.perms)

- <prefix>_ADD_REGION(NULL) -> sgx_enclave_load(NULL, RW)

then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of sgx_enclave_load(..., RW) would need to be different than normal shared memory, e.g. FILE__WRITE should *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without* FILE__EXECMOD, it'd have the necessary information to do so.

The userspace changes are fairly minimal:

- For SGX1, use PROT_NONE for the initial mmap() and refactor ADD_PAGE to ADD_REGION.

- For SGX2, do an explicit ADD_REGION on the ranges to be EAUG'd, and an ACTIVATE_REGION to make a region RX or R (no extra ioctl() required to keep RW permissions).

Quoted 6 lines

Because ACTIVATE_REGION can only be done once per page, to do *abitrary*
mprotect() transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.

Quoted 9 lines

I may be misunderstanding or just be biased to my own proposal, but this seems potentially more complicated and less flexible than the MAXPERM model. One of the main things that made me come up with MAXPERM is that I wanted to avoid any complicated PTE/VMA modification or runtime changes. So, with MAXPERM, we still need to track the MAXPERM bits per page, but we don't ever need to *change* them or to worry about what is or is not mapped anywhere at any given time. With ACTIVATE_REGION, don't we need to make sure that we don't have a second VMA pointing at the same pages? Or am I just confused?

Quoted 3 lines

In theory, it's still your MAXPERM model, but with the unnecessary states removed and the others enforced/handled by the natural SGX transitions instead of explictly in ioctls. Underneath the hood the SGX driver would still need to track the MAXPERM.

What are the "unnecessary states" removed?

I'm not sure understand the proposal fully. The whole thing looks to me like the driver is undertaking things that should/would otherwise be done by mmap()/mprotect() syscalls. It also imposes unnecessary restrictions on user mode code, such as mmap(PROT_NONE), ACTIVATE_REGION can be called only once, etc. What'd happen if ACTIVATE_REGION is called with a range spanning multiple/partial VMAs? What'd happen if an enclave was unmapped than mapped again? I'd say the proposal is unintuitive at least.

In theory, if the driver can keep track of MAXPERM for all pages within an enclave, then it could fail mmap() if the requested prot conflicts with any page's MAXPERM within that range. Otherwise, MAXPERM could be copied into VM_MAY* flags then mprotect() will just follow through. Wouldn't that be a much simpler and more intuitive approach?

Quoted 25 lines

With SGX1, SECINFO == MAXPERM. With SGX2, ACTIVATE_REGION == MAXPERM, with the implication that the previous state is always RW.

Quoted 9 lines
Quoted 2 lines

- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.

I may still be missing something, but, for SGX1, it's possible at least in principle for the enclave to request, via ocall or similar, that the untrusted runtime do mprotect(). It's not even such a bad idea. Honestly, enclaves *shouldn't* have anything actually writable and executable at once because the enclaves don't want to be easily exploited.

Yes, but the *EPCM* permissions are immutable. So if an enclave wants to do RW->RX it has to intialize its pages to RWX. And because the untrusted runtime is, ahem, untrusted, the enclave cannot rely on userspace to never map its pages RWX. In other words, from a enclave security perspective, an SGX1 enclave+runtime that uses RW->RX is no different than an enclave that uses RWX. Using your earlier terminology, an SGX1 enclave *should* get a dirty looks if maps a page RWX in the EPCM, even if it only intends RW->RX behavior.

Quoted 4 lines
Quoted 4 lines

- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.

#189 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 16 lines · Cedric
Quoted 16 lines

From: Christopherson, Sean J
Sent: Tuesday, May 28, 2019 2:41 PM

Quoted 11 lines · Andy Lutomirski
Quoted 5 lines · Sean Christopherson

Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that sgx_enclave_load() with a NULL vma implies a transition from RW. For example, SELinux would interpret sgx_enclave_load(NULL, RX) as requiring FILE__EXECMOD.

You lost me here. What operation triggers this callback? And wouldn't sgx_enclave_load(NULL, RX) sometimes be a transition from RO or just some fresh executable zero bytes?

Quoted 8 lines
Quoted 6 lines

An explicit ioctl() after EACCEPTCOPY to update the allowed permissions. For all intents and purposes, the EAUG'd page must start RW. Maybe a better way to phrase it is that at some point the page must be writable to have any value whatsover. EACCEPTCOPY explicitly requires the page to be at least RW. EACCEPT technically doesn't require RW, but a RO or RX zero page is useless. Userspace could still EACCEPT with RO or RX, but SGX would assume a minimum of RW for the purposes of the LSM check.

Why is an explicit ioctl() necessary after EACCEPTCOPY? Or why is mprotect() not sufficient?

Ignore this, I was trying to avoid having to add a vm_ops mprotect(), which Andy pointed out was silly.

Quoted 5 lines
Quoted 3 lines

In theory, it's still your MAXPERM model, but with the unnecessary states removed and the others enforced/handled by the natural SGX transitions instead of explictly in ioctls. Underneath the hood the SGX driver would still need to track the MAXPERM.

What are the "unnecessary states" removed?

Andy proposed taking full RWX in MAXPERMs, but really we only need "can writes ever happen to this page", as that allows the SGX driver to avoid having to track if a page has been mapped PROT_WRITE by any VMA in any process.

Quoted 8 lines

I'm not sure understand the proposal fully. The whole thing looks to me like the driver is undertaking things that should/would otherwise be done by mmap()/mprotect() syscalls. It also imposes unnecessary restrictions on user mode code, such as mmap(PROT_NONE), ACTIVATE_REGION can be called only once, etc. What'd happen if ACTIVATE_REGION is called with a range spanning multiple/partial VMAs? What'd happen if an enclave was unmapped than mapped again? I'd say the proposal is unintuitive at least.

Quoted 5 lines

In theory, if the driver can keep track of MAXPERM for all pages within an enclave, then it could fail mmap() if the requested prot conflicts with any page's MAXPERM within that range. Otherwise, MAXPERM could be copied into VM_MAY* flags then mprotect() will just follow through. Wouldn't that be a much simpler and more intuitive approach?

Ignore all this, again I was trying to avoid hooking mprotect().

#190 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson
Quoted 191 lines · Sean Christopherson
Quoted 15 lines · Cedric
Quoted 7 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PM

Quoted 3 lines · Cedric

If we think of EADD as a way of mmap()'ing an enclave file into memory, would this

Quoted 7 lines

security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?

Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.

Quoted 8 lines

What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.

But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to security_mmap_file() by default?

Mapping the enclave will still go through security_mmap_file(), the extra security_enclave_load() hook allows the mmap() to use PROT_NONE.

Quoted 11 lines
Quoted 7 lines

If it's going to be in an arbitrary file, then I think the signature needs to be more like:

int security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);

So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.

Wouldn't the pair of { sigstruct_vma, sigstruct_offset } be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer - vma->vm_start)?

VMA has vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?

   int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);
Quoted 22 lines
Quoted 14 lines
Quoted 1 line

Loosely speaking, an enclave (including initial contents of all of its pages and their

permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.

Quoted 2 lines

Theoretically speaking, if LSM can make a decision at EINIT by means of

security_enclave_load(), then security_enclave_load() is never needed.

Quoted 2 lines

In practice, I support keeping both because security_enclave_load() can only approve an

enumerable set while security_enclave_load() can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced by security_enclave_load().

Quoted 7 lines

I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also, security_enclave_load() provides no protection against loading a mishmash of two different enclave files. I see security_enclave_init() as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" and security_enclave_load() as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."

Quoted 6 lines

Sorry for the confusion. I was saying the same thing except that the decision of security_enclave_load() doesn't have to depend on SIGSTRUCT. Given your prototype of security_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".

Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling /dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using /dev/sgx/enclave means users can't do per-enclave permissions, period.

What is your objection to working on the sigstruct?

Quoted 25 lines
Quoted 25 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

Quoted 6 lines
Quoted 6 lines

The benefit of putting it on .sigstruct is that it can be per-enclave.

As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty

look.

Quoted 12 lines
Quoted 11 lines

This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.

(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)

Quoted 7 lines
Quoted 7 lines
Quoted 1 line

I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX, i.e.

FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.

Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual mprotect() / mmap() permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply that PROT_WRITE | PROT_EXEC is allowed.

Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that sgx_enclave_load() with a NULL vma implies a transition from RW. For example, SELinux would interpret sgx_enclave_load(NULL, RX) as requiring FILE__EXECMOD.

As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e. sgx_enclave_load(NULL, RX) already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs from MAY_RW to MAY_RX.

For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit ioctl() to add the to-be-EAUG'd range. That just leaves us wanting an ioctl() to set the post-EACCEPT{COPY} permissions.

E.g.:

     ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */

     mprotect(addr, size, RW);
     ...

     EACCEPTCOPY -> EAUG /* page fault handler */

     ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */

     mprotect(addr, size, RX);

     ...

And making ACTIVATE_REGION a single-shot per page eliminates the need for the MAXPERMS concept (see below).

Quoted 15 lines

If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?

To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of /dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on /dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of -EPERM returned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of IOC_EINIT.

Aha!

Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:

- Initial permissions for *EADD* pages are explicitly handled via sgx_enclave_load() with the exact SECINFO permissions.

- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.

- Runtime permissions break down as follows:

       R   - N/A, subset of RW (EAUG)
       W   - N/A, subset of RW (EAUG) and x86 paging can't do W
       X   - N/A, subset of RX (x86 paging can't do XO)
       RW  - Handled by EAUG LSM hook (uses RW unconditionally)
       WX  - N/A, subset of RWX (x86 paging can't do WX)
       RX  - Handled by ACTIVATE_REGION

RWX - Handled by ACTIVATE_REGION

In other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):

- <prefix>_ACTIVATE_REGION(vma, perms) -> sgx_enclave_load(NULL, perms)

- <prefix>_ADD_REGION(vma) -> sgx_enclave_load(vma, SECINFO.perms)

Quoted 8 lines

- <prefix>_ADD_REGION(NULL) -> sgx_enclave_load(NULL, RW)

then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of sgx_enclave_load(..., RW) would need to be different than normal shared memory, e.g. FILE__WRITE should *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without* FILE__EXECMOD, it'd have the necessary information to do so.

Assuming that sgx_enclave_load() is a LSM hook (probably named security_enclave_load() instead), then:

a) Does the sigstruct file get passed to this hook in every case, even when vma is NULL? I think the answer is yes, just want to confirm.

b) Should we use a different hook for ACTIVATE_REGION than for ADD_REGION or is the distinction between them irrelevant/unnecessary from an access control point of view? At present LSM/SELinux won't be able to distinguish ACTIVATE_REGION(vma, RW) from ADD_REGION(NULL) above since they will both invoke the same hook with the same arguments IIUC. Does it matter? It's ok if the answer is no, just want to confirm.

c) Is there still also a separate security_enclave_init() hook that will be called, and if so, how does it differ and when is it called relative to security_enclave_load()?

d) What checks were you envisioning each of these calls making?

With the separate security_enclave_*() hooks, we could define and use new ENCLAVE__* permissions, e.g. ENCLAVE__LOAD, ENCLAVE__INIT, ENCLAVE__EXECUTE, ENCLAVE__EXECMEM, ENCLAVE__EXECMOD, if we want to distinguish these operations from regular file mmap/mprotect operations.

Quoted 23 lines

The userspace changes are fairly minimal:

- For SGX1, use PROT_NONE for the initial mmap() and refactor ADD_PAGE to ADD_REGION.

- For SGX2, do an explicit ADD_REGION on the ranges to be EAUG'd, and an ACTIVATE_REGION to make a region RX or R (no extra ioctl() required to keep RW permissions).

Because ACTIVATE_REGION can only be done once per page, to do *abitrary* mprotect() transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.

- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.

- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.

#191 lore raw

RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 215 lines

From: [email protected] [mailto:[email protected]] On Behalf Of Stephen Smalley
Sent: Wednesday, May 29, 2019 7:08 AM

Quoted 21 lines · Sean Christopherson
Quoted 20 lines · Cedric
Quoted 7 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PM

Quoted 3 lines · Cedric

If we think of EADD as a way of mmap()'ing an enclave file into memory, would this

Quoted 7 lines

security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?

Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.

What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.

Quoted 8 lines
Quoted 3 lines

But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to security_mmap_file() by default?

Mapping the enclave will still go through security_mmap_file(), the extra security_enclave_load() hook allows the mmap() to use PROT_NONE.

Quoted 1 line
Quoted 1 line

If it's going to be in an arbitrary file, then I think the signature needs to be more

like:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

int security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t

sigstruct_offset,

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

const sgx_sigstruct *sigstruct);

So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.

Quoted 10 lines
Quoted 3 lines

Wouldn't the pair of { sigstruct_vma, sigstruct_offset } be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer - vma->vm_start)?

VMA has vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?

   int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);
Quoted 1 line
Quoted 1 line
Quoted 1 line

Loosely speaking, an enclave (including initial contents of all of its pages and

their

Quoted 1 line
Quoted 1 line
Quoted 1 line

permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision

resistant

Quoted 1 line
Quoted 1 line
Quoted 1 line

property of SHA-2). So only one is needed for a decision, and either one would lead to

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

same decision. So I don't see anything making any sense here.

Quoted 2 lines

Theoretically speaking, if LSM can make a decision at EINIT by means of

security_enclave_load(), then security_enclave_load() is never needed.

Quoted 2 lines

In practice, I support keeping both because security_enclave_load() can only approve

an

Quoted 1 line
Quoted 1 line
Quoted 1 line

enumerable set while security_enclave_load() can approve a non-enumerable set of

enclaves.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Moreover, in order to determine the validity of a MRENCLAVE (as in development of a

policy

Quoted 1 line
Quoted 1 line
Quoted 1 line

or in creation of a white/black list), system admins will need the audit log produced

by

Quoted 17 lines
Quoted 10 lines
Quoted 9 lines

security_enclave_load().

I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also, security_enclave_load() provides no protection against loading a mishmash of two different enclave files. I see security_enclave_init() as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" and security_enclave_load() as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."

Quoted 6 lines

Sorry for the confusion. I was saying the same thing except that the decision of security_enclave_load() doesn't have to depend on SIGSTRUCT. Given your prototype of security_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".

Quoted 35 lines

Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling /dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using /dev/sgx/enclave means users can't do per-enclave permissions, period.

What is your objection to working on the sigstruct?

Quoted 26 lines
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

Quoted 6 lines
Quoted 6 lines

The benefit of putting it on .sigstruct is that it can be per-enclave.

As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty

look.

Quoted 13 lines
Quoted 11 lines

This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.

(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)

I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX,

i.e.

Quoted 82 lines
Quoted 6 lines
Quoted 6 lines

FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.

Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual mprotect() / mmap() permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply that PROT_WRITE | PROT_EXEC is allowed.

Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that sgx_enclave_load() with a NULL vma implies a transition from RW. For example, SELinux would interpret sgx_enclave_load(NULL, RX) as requiring FILE__EXECMOD.

As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e. sgx_enclave_load(NULL, RX) already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs from MAY_RW to MAY_RX.

For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit ioctl() to add the to-be-EAUG'd range. That just leaves us wanting an ioctl() to set the post-EACCEPT{COPY} permissions.

E.g.:

     ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */

     mprotect(addr, size, RW);
     ...

     EACCEPTCOPY -> EAUG /* page fault handler */

     ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */

     mprotect(addr, size, RX);

     ...

And making ACTIVATE_REGION a single-shot per page eliminates the need for the MAXPERMS concept (see below).

Quoted 15 lines

If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?

To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of /dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on /dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of -EPERM returned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of IOC_EINIT.

Aha!

Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:

- Initial permissions for *EADD* pages are explicitly handled via sgx_enclave_load() with the exact SECINFO permissions.

- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.

- Runtime permissions break down as follows:

       R   - N/A, subset of RW (EAUG)
       W   - N/A, subset of RW (EAUG) and x86 paging can't do W
       X   - N/A, subset of RX (x86 paging can't do XO)
       RW  - Handled by EAUG LSM hook (uses RW unconditionally)
       WX  - N/A, subset of RWX (x86 paging can't do WX)
       RX  - Handled by ACTIVATE_REGION

RWX - Handled by ACTIVATE_REGION

In other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):

- <prefix>_ACTIVATE_REGION(vma, perms) -> sgx_enclave_load(NULL, perms)

- <prefix>_ADD_REGION(vma) -> sgx_enclave_load(vma, SECINFO.perms)

Quoted 8 lines

- <prefix>_ADD_REGION(NULL) -> sgx_enclave_load(NULL, RW)

then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of sgx_enclave_load(..., RW) would need to be different than normal shared memory, e.g. FILE__WRITE should *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without* FILE__EXECMOD, it'd have the necessary information to do so.

Quoted 5 lines

Assuming that sgx_enclave_load() is a LSM hook (probably named security_enclave_load() instead), then:

a) Does the sigstruct file get passed to this hook in every case, even when vma is NULL? I think the answer is yes, just want to confirm.

I'm confused.

In the case of EADD (non-NULL vma), are we passing both vma and sigstruct file? If so, which file dictates allowed permissions, vma->vm_file or sigstruct, or both???

In the case of EAUG (NULL vma), all other parameters are constant for any given enclave. Then why do we call this same hook for every region added, assuming the hook will return the same value everytime anyway?

And it looks like ACTIVATE_REGION is needed only because the proposed security_enclave_load() would base its decision on the sigstruct file. An alternative is to base that decision on /dev/sgx/enclave. Of course the former has finer granularity but is that really necessary? From security perspective, only the weakest link matters. FILE__EXECMOD on a regular shared object could allow exploits of all bugs throughout the host process because code within that shared object is modifiable by not only itself but also any code within that same process. In contrast, FILE__EXECMOD on /dev/sgx/enclave only allows enclaves to modify themselves. They cannot modify each other, neither can "untrusted" code outside of enclaves modify any of them. So it doesn't look like a weaker link to me. Moreover, requiring FILE__EXECMOD on sigstruct means it could be used as a target buffer for code injection attacks. IMHO that *lowers* the security of the whole process.

Quoted 11 lines

b) Should we use a different hook for ACTIVATE_REGION than for ADD_REGION or is the distinction between them irrelevant/unnecessary from an access control point of view? At present LSM/SELinux won't be able to distinguish ACTIVATE_REGION(vma, RW) from ADD_REGION(NULL) above since they will both invoke the same hook with the same arguments IIUC. Does it matter? It's ok if the answer is no, just want to confirm.

c) Is there still also a separate security_enclave_init() hook that will be called, and if so, how does it differ and when is it called relative to security_enclave_load()?

I think security_enclave_init() will always be useful, as it offers a way for LSM to implement whitelisting/blacklisting. Of course an LSM module like SELinux can look into the backing inode too. I think the hook should have a signature like:

int security_enclave_init(struct sgx_sigstruct __user *sigstruct);

An LSM that cares about the backing file could look into vm_file of the VMA covering the buffer, while an LSM that cares the sigstruct itself (e.g. signing key) could just look into the buffer.

Quoted 7 lines

d) What checks were you envisioning each of these calls making?

With the separate security_enclave_*() hooks, we could define and use new ENCLAVE__* permissions, e.g. ENCLAVE__LOAD, ENCLAVE__INIT, ENCLAVE__EXECUTE, ENCLAVE__EXECMEM, ENCLAVE__EXECMOD, if we want to distinguish these operations from regular file mmap/mprotect operations.

I'm not sure if these ENCLAVE__* flags are an overkill, unless we want to enforce an enclave file cannot be loaded as a regular shared object or vice versa.

Quoted 24 lines
Quoted 24 lines

The userspace changes are fairly minimal:

- For SGX1, use PROT_NONE for the initial mmap() and refactor ADD_PAGE to ADD_REGION.

- For SGX2, do an explicit ADD_REGION on the ranges to be EAUG'd, and an ACTIVATE_REGION to make a region RX or R (no extra ioctl() required to keep RW permissions).

Because ACTIVATE_REGION can only be done once per page, to do *abitrary* mprotect() transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.

- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.

- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.

#192 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 215 lines · Cedric
Quoted 215 lines

From: [email protected] [mailto:[email protected]] On Behalf Of Stephen Smalley
Sent: Wednesday, May 29, 2019 7:08 AM

Quoted 21 lines · Sean Christopherson
Quoted 20 lines · Cedric
Quoted 7 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PM

Quoted 3 lines · Cedric

If we think of EADD as a way of mmap()'ing an enclave file into memory, would this

Quoted 7 lines

security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?

Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.

What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.

Quoted 8 lines
Quoted 3 lines

But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to security_mmap_file() by default?

Mapping the enclave will still go through security_mmap_file(), the extra security_enclave_load() hook allows the mmap() to use PROT_NONE.

Quoted 1 line
Quoted 1 line

If it's going to be in an arbitrary file, then I think the signature needs to be more

like:

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

int security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t

sigstruct_offset,

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

const sgx_sigstruct *sigstruct);

So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.

Quoted 10 lines
Quoted 3 lines

Wouldn't the pair of { sigstruct_vma, sigstruct_offset } be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer - vma->vm_start)?

VMA has vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?

    int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);
Quoted 1 line
Quoted 1 line
Quoted 1 line

Loosely speaking, an enclave (including initial contents of all of its pages and

their

Quoted 1 line
Quoted 1 line
Quoted 1 line

permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision

resistant

Quoted 1 line
Quoted 1 line
Quoted 1 line

property of SHA-2). So only one is needed for a decision, and either one would lead to

the

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

same decision. So I don't see anything making any sense here.

Quoted 2 lines

Theoretically speaking, if LSM can make a decision at EINIT by means of

security_enclave_load(), then security_enclave_load() is never needed.

Quoted 2 lines

In practice, I support keeping both because security_enclave_load() can only approve

an

Quoted 1 line
Quoted 1 line
Quoted 1 line

enumerable set while security_enclave_load() can approve a non-enumerable set of

enclaves.

Quoted 1 line
Quoted 1 line
Quoted 1 line

Moreover, in order to determine the validity of a MRENCLAVE (as in development of a

policy

Quoted 1 line
Quoted 1 line
Quoted 1 line

or in creation of a white/black list), system admins will need the audit log produced

by

Quoted 17 lines
Quoted 10 lines
Quoted 9 lines

security_enclave_load().

I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also, security_enclave_load() provides no protection against loading a mishmash of two different enclave files. I see security_enclave_init() as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" and security_enclave_load() as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."

Quoted 6 lines

Sorry for the confusion. I was saying the same thing except that the decision of security_enclave_load() doesn't have to depend on SIGSTRUCT. Given your prototype of security_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".

Quoted 35 lines

Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling /dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using /dev/sgx/enclave means users can't do per-enclave permissions, period.

What is your objection to working on the sigstruct?

Quoted 26 lines
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.

Quoted 6 lines
Quoted 6 lines

The benefit of putting it on .sigstruct is that it can be per-enclave.

As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty

look.

Quoted 13 lines
Quoted 11 lines

This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.

(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)

I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX,

i.e.

Quoted 90 lines
Quoted 6 lines
Quoted 6 lines

FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.

Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual mprotect() / mmap() permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply that PROT_WRITE | PROT_EXEC is allowed.

Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that sgx_enclave_load() with a NULL vma implies a transition from RW. For example, SELinux would interpret sgx_enclave_load(NULL, RX) as requiring FILE__EXECMOD.

As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e. sgx_enclave_load(NULL, RX) already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs from MAY_RW to MAY_RX.

For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit ioctl() to add the to-be-EAUG'd range. That just leaves us wanting an ioctl() to set the post-EACCEPT{COPY} permissions.

E.g.:

      ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */

      mprotect(addr, size, RW);
      ...

      EACCEPTCOPY -> EAUG /* page fault handler */

      ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */

      mprotect(addr, size, RX);

      ...

And making ACTIVATE_REGION a single-shot per page eliminates the need for the MAXPERMS concept (see below).

Quoted 15 lines

If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?

To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of /dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on /dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of -EPERM returned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of IOC_EINIT.

Aha!

Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:

- Initial permissions for *EADD* pages are explicitly handled via sgx_enclave_load() with the exact SECINFO permissions.

- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.

- Runtime permissions break down as follows:
R - N/A, subset of RW (EAUG)
W - N/A, subset of RW (EAUG) and x86 paging can't do W
X - N/A, subset of RX (x86 paging can't do XO)
RW - Handled by EAUG LSM hook (uses RW unconditionally) WX - N/A, subset of RWX (x86 paging can't do WX) RX - Handled by ACTIVATE_REGION
RWX - Handled by ACTIVATE_REGION

In other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):

- <prefix>_ACTIVATE_REGION(vma, perms) -> sgx_enclave_load(NULL, perms)

- <prefix>_ADD_REGION(vma) -> sgx_enclave_load(vma, SECINFO.perms)

- <prefix>_ADD_REGION(NULL) -> sgx_enclave_load(NULL, RW)

then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of sgx_enclave_load(..., RW) would need to be different than normal shared memory, e.g. FILE__WRITE should *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without* FILE__EXECMOD, it'd have the necessary information to do so.

Quoted 7 lines
Quoted 5 lines

Assuming that sgx_enclave_load() is a LSM hook (probably named security_enclave_load() instead), then:

a) Does the sigstruct file get passed to this hook in every case, even when vma is NULL? I think the answer is yes, just want to confirm.

I'm confused.

I'm finding it difficult to follow as well, so my questions are just an attempt to understand the latest model.

Quoted 1 line

In the case of EADD (non-NULL vma), are we passing both vma and sigstruct file? If so, which file dictates allowed permissions, vma->vm_file or sigstruct, or both???

My impression was that they were going to pass both, but the sigstruct file is the target of permission checks. The vma if non-NULL would be used to determine whether PROT_EXEC is being added or was already present and whether EXECMOD needs to be checked (i.e. copy-on-write has occurred and PROT_EXEC is being added).

Quoted 1 line

In the case of EAUG (NULL vma), all other parameters are constant for any given enclave. Then why do we call this same hook for every region added, assuming the hook will return the same value everytime anyway?

Yes, I was wondering about that as well.

Quoted 1 line

And it looks like ACTIVATE_REGION is needed only because the proposed security_enclave_load() would base its decision on the sigstruct file. An alternative is to base that decision on /dev/sgx/enclave. Of course the former has finer granularity but is that really necessary? From security perspective, only the weakest link matters. FILE__EXECMOD on a regular shared object could allow exploits of all bugs throughout the host process because code within that shared object is modifiable by not only itself but also any code within that same process. In contrast, FILE__EXECMOD on /dev/sgx/enclave only allows enclaves to modify themselves. They cannot modify each other, neither can "untrusted" code outside of enclaves modify any of them. So it doesn't look like a weaker link to me. Moreover, requiring FILE__EXECMOD on sigstruct means it could be used as a target buffer for code injection attacks. IMHO that *lowers* the security of the whole process.

This is partly why I suggested separate ENCLAVE__EXECMOD and other checks below, so we can distinguish between FILE__EXECMOD versus ENCLAVE__EXECMOD on the sigstruct file. If using /dev/sgx/enclave as the target, then we don't need a separate permission per se but we lose the per-sigstruct granularity.

Quoted 13 lines
Quoted 9 lines

b) Should we use a different hook for ACTIVATE_REGION than for ADD_REGION or is the distinction between them irrelevant/unnecessary from an access control point of view? At present LSM/SELinux won't be able to distinguish ACTIVATE_REGION(vma, RW) from ADD_REGION(NULL) above since they will both invoke the same hook with the same arguments IIUC. Does it matter? It's ok if the answer is no, just want to confirm.

Quoted 3 lines

c) Is there still also a separate security_enclave_init() hook that will be called, and if so, how does it differ and when is it called relative to security_enclave_load()?

Quoted 5 lines

I think security_enclave_init() will always be useful, as it offers a way for LSM to implement whitelisting/blacklisting. Of course an LSM module like SELinux can look into the backing inode too. I think the hook should have a signature like:

int security_enclave_init(struct sgx_sigstruct __user *sigstruct);

An LSM that cares about the backing file could look into vm_file of the VMA covering the buffer, while an LSM that cares the sigstruct itself (e.g. signing key) could just look into the buffer.

I'm not a fan of passing __user pointers to LSM hooks. And it certainly shouldn't be looking at the buffer since it could change between the time of check and time of use.

Quoted 10 lines
Quoted 8 lines

d) What checks were you envisioning each of these calls making?

With the separate security_enclave_*() hooks, we could define and use new ENCLAVE__* permissions, e.g. ENCLAVE__LOAD, ENCLAVE__INIT, ENCLAVE__EXECUTE, ENCLAVE__EXECMEM, ENCLAVE__EXECMOD, if we want to distinguish these operations from regular file mmap/mprotect operations.

I'm not sure if these ENCLAVE__* flags are an overkill, unless we want to enforce an enclave file cannot be loaded as a regular shared object or vice versa.

ENCLAVE__LOAD and/or ENCLAVE__INIT would be to support whitelisting of what enclaves can be loaded/initialized by the process. That's separate from the W^X discussion. Those permissions would be between the process and either the sigstruct file or the enclave file (the consensus seemed to be the sigstruct file as the stronger/more complete binding of the enclave). We probably only need one of those two permission checks not both.

ENCLAVE__EXECUTE, ENCLAVE__EXECMEM, ENCLAVE__EXECMOD would allow distinctions between host process mmap/mprotect PROT_EXEC operations (which would continue to apply FILE__EXECUTE, PROCESS__EXECMEM, and FILE__EXECMOD checks if appropriate) and the driver's setting of initial and runtime permissions (which would apply ENCLAVE__EXECUTE, ENCLAVE__EXECMEM, and ENCLAVE__EXECMOD checks if appropriate). That's particularly helpful if we are using the sigstruct or enclave file as the target of all checks instead of /dev/sgx/enclave, so that we don't have to allow FILE__EXECUTE or FILE__EXECMOD to the sigstruct file by the host process.

Quoted 26 lines
Quoted 25 lines
Quoted 25 lines

The userspace changes are fairly minimal:

- For SGX1, use PROT_NONE for the initial mmap() and refactor ADD_PAGE to ADD_REGION.

- For SGX2, do an explicit ADD_REGION on the ranges to be EAUG'd, and an ACTIVATE_REGION to make a region RX or R (no extra ioctl() required to keep RW permissions).

Because ACTIVATE_REGION can only be done once per page, to do *abitrary* mprotect() transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.

- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.

- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.

maintainer
#193 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Hi all-

After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.

Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.

First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:

- You can create a WX or RWX mapping if and only if you have EXECMEM.

- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.

- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.

And I have two design proposals. One is static and one is dynamic. To implement either one, we will probably need a new .may_mprotect vm operation, and that operation can call an LSM hook. Or we can give LSMs a way to detect that a given vm_area_struct is an enclave. As I see it, this is an implementation detail that is certainly solveable.

Static proposal:

EADD takes an execute_intent flag. It calls a new hook:

  int security_enclave_load(struct vm_area_struct *source, bool execute_intent);

This hook will fail if execute_intent==true and the caller has neither EXECUTE, EXECMOD, nor EXECMEM.

EAUG sets execute_intent = false.

EINIT takes a sigstruct pointer. SGX can (when initially upstreamed or later on once there's demand) call a new hook:

security_enclave_init(struct sigstruct *sigstruct, struct
vm_area_struct *source);

mmap() and mprotect() will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.

Dynamic proposal:

EADD does not take any special flags. It does something like this internally:

bool execute_intent = true;
int security_enclave_load(struct vm_area_struct *source, bool
*execute_intent);

The implementation of security_enclave_load() may set *execute_intent to false. The driver records execute_intent after the LSM is done.

mmap() and mprotect() will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.

A benefit of the static proposal is that audit failures due to a lack of EXECUTE permission are easy to implement and to understand in the lods. With the dynamic model, we can only really audit the lack of EXECMOD or EXECMEM. A benefit of the dynamic model is that we hide what is arguably a decently large wart from the API.

#194 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 10 lines · Andy Lutomirski

Hi all-

After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.

Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.

Quoted 8 lines

First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:

- You can create a WX or RWX mapping if and only if you have EXECMEM.

- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.

EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or /dev/sgx/enclave?

Quoted 3 lines

- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.

What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or /dev/sgx/enclave?

Quoted 11 lines

And I have two design proposals. One is static and one is dynamic. To implement either one, we will probably need a new .may_mprotect vm operation, and that operation can call an LSM hook. Or we can give LSMs a way to detect that a given vm_area_struct is an enclave. As I see it, this is an implementation detail that is certainly solveable.

Static proposal:

Quoted 6 lines

EADD takes an execute_intent flag. It calls a new hook:

   int security_enclave_load(struct vm_area_struct *source, bool execute_intent);

This hook will fail if execute_intent==true and the caller has neither EXECUTE, EXECMOD, nor EXECMEM.

EADD execute_intent flag is originally provided by whom (userspace or driver) on what basis? Which file is referenced by source->vm_file? Why trigger all three checks up front versus only checking if needed? Won't this trigger a lot of unnecessary EXECMOD and EXECMEM denials that will need to be dontaudit'd? What if there is a mismatch between execute_intent and the initial permissions?

Quoted 8 lines

EAUG sets execute_intent = false.

EINIT takes a sigstruct pointer. SGX can (when initially upstreamed or later on once there's demand) call a new hook:

security_enclave_init(struct sigstruct *sigstruct, struct
vm_area_struct *source);

Is struct sigstruct the same as struct sgx_sigstruct in the current patches (i.e. just the sigstruct data, no file)? What file is referenced by source->vm_file (the sigstruct or the enclave or /dev/sgx/enclave)? Is this hook only for enforcing a whitelist on what enclaves can be loaded? What is the target of the check?

Quoted 4 lines

mmap() and mprotect() will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.

Does this occur for both setting initial permissions and runtime permissions or just runtime? Both userspace- and driver-initiated mmap/mprotect operations or just userspace-initiated ones? Does the driver use interfaces that call the mmap/mprotect hooks or lower level functions?

Quoted 5 lines

Dynamic proposal:

Quoted 8 lines

EADD does not take any special flags. It does something like this internally:

bool execute_intent = true;
int security_enclave_load(struct vm_area_struct *source, bool
*execute_intent);

The implementation of security_enclave_load() may set *execute_intent to false. The driver records execute_intent after the LSM is done.

On what basis does LSM decide whether to set *execute_intent? If the process lacks all three permissions? What if there is a mismatch with the initial permissions?

Quoted 14 lines

mmap() and mprotect() will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.

A benefit of the static proposal is that audit failures due to a lack of EXECUTE permission are easy to implement and to understand in the lods. With the dynamic model, we can only really audit the lack of EXECMOD or EXECMEM. A benefit of the dynamic model is that we hide what is arguably a decently large wart from the API.

maintainer
#195 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 16 lines · Stephen Smalley
Quoted 14 lines · Andy Lutomirski

Hi all-

After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.

Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.

First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:

Quoted 7 lines
Quoted 4 lines

- You can create a WX or RWX mapping if and only if you have EXECMEM.

- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.

EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or /dev/sgx/enclave?

I leave that decision to you :) The user should need permission to do an execmod thing on an enclave, however that wants to be encoded.

Quoted 7 lines
Quoted 4 lines

- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.

What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or /dev/sgx/enclave?

Enclave file -- that is, the file backing the vma from which the data is loaded.

Quoted 19 lines
Quoted 12 lines

And I have two design proposals. One is static and one is dynamic. To implement either one, we will probably need a new .may_mprotect vm operation, and that operation can call an LSM hook. Or we can give LSMs a way to detect that a given vm_area_struct is an enclave. As I see it, this is an implementation detail that is certainly solveable.

Static proposal:

Quoted 6 lines

EADD takes an execute_intent flag. It calls a new hook:

   int security_enclave_load(struct vm_area_struct *source, bool execute_intent);

This hook will fail if execute_intent==true and the caller has neither EXECUTE, EXECMOD, nor EXECMEM.

Quoted 6 lines

EADD execute_intent flag is originally provided by whom (userspace or driver) on what basis? Which file is referenced by source->vm_file? Why trigger all three checks up front versus only checking if needed? Won't this trigger a lot of unnecessary EXECMOD and EXECMEM denials that will need to be dontaudit'd? What if there is a mismatch between execute_intent and the initial permissions?

It's provided by userspace based on whether it thinks the data in question is enclave code. source->vm_file is the file from which the code is being loaded. I'm assuming that the user code will only set excute_intent ==true if it actually wants to execute the code, so, if there's a denial, it will be fatal. The normal case will be that the request will be granted on the basis of EXECUTE.

Quoted 7 lines
Quoted 7 lines

EAUG sets execute_intent = false.

EINIT takes a sigstruct pointer. SGX can (when initially upstreamed or later on once there's demand) call a new hook:

Quoted 8 lines
Quoted 2 lines

security_enclave_init(struct sigstruct *sigstruct, struct
vm_area_struct *source);

Is struct sigstruct the same as struct sgx_sigstruct in the current patches (i.e. just the sigstruct data, no file)? What file is referenced by source->vm_file (the sigstruct or the enclave or /dev/sgx/enclave)? Is this hook only for enforcing a whitelist on what enclaves can be loaded? What is the target of the check?

sigstruct is just the data. source->vm_file is the file from which the sigstruct came, which could be a .sigstruct file or could be the main executable or a DSO that contains an embedded enclave. The sigstruct data is there so that an LSM (not necessarily SELinux) could check MRENCLAVE or MRSIGNER, and the source is there so that the file's label can be checked.

Quoted 11 lines
Quoted 5 lines

mmap() and mprotect() will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.

Does this occur for both setting initial permissions and runtime permissions or just runtime? Both userspace- and driver-initiated mmap/mprotect operations or just userspace-initiated ones? Does the driver use interfaces that call the mmap/mprotect hooks or lower level functions?

These would occur for any mmap(), mprotect(), or ioctl() that changes VMA permissions. Actually arranging for the hooks to be called is an implementation detail that might require a new .mprotect vm_operation. As an alternative, security_enclave_init() or similar could supply may_execmod and may_execmem flags to the driver, and the driver could do these checks on its own when mmap() and mprotect() happen without a new LSM callback.

Quoted 12 lines
Quoted 12 lines

Dynamic proposal:

EADD does not take any special flags. It does something like this internally:

bool execute_intent = true;
int security_enclave_load(struct vm_area_struct *source, bool
*execute_intent);

Quoted 7 lines
Quoted 2 lines

The implementation of security_enclave_load() may set *execute_intent to false. The driver records execute_intent after the LSM is done.

On what basis does LSM decide whether to set *execute_intent? If the process lacks all three permissions? What if there is a mismatch with the initial permissions?

I think it would set *execute_intent=false if the process lacks EXECUTE on source->vm_file. I'm not sure any more complexity is required. If the enclave has EXECMOD, then it will still work on the basis of the mmap/mprotect rules.

Quoted 15 lines
Quoted 14 lines

mmap() and mprotect() will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.

A benefit of the static proposal is that audit failures due to a lack of EXECUTE permission are easy to implement and to understand in the lods. With the dynamic model, we can only really audit the lack of EXECMOD or EXECMEM. A benefit of the dynamic model is that we hide what is arguably a decently large wart from the API.

#196 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 19 lines · Andy Lutomirski
Quoted 18 lines · Stephen Smalley
Quoted 16 lines · Andy Lutomirski

Hi all-

After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.

Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.

First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:

- You can create a WX or RWX mapping if and only if you have EXECMEM.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.

EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or /dev/sgx/enclave?

I leave that decision to you :) The user should need permission to do an execmod thing on an enclave, however that wants to be encoded.

But that decision dictates how the SGX API handles sigstruct. If LSMs want to associate EXECMOD with sigstruct, then SGX needs to take sigstruct early and hold a reference to the file for the lifetime of the enclave. And if we're going to do that, the whole approach of inheriting permissions from source VMAs becomes unnecessary complexity.

Quoted 9 lines
Quoted 7 lines
Quoted 4 lines

- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.

What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or /dev/sgx/enclave?

Enclave file -- that is, the file backing the vma from which the data is loaded.

It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).

I have two gripes with that approach:

- Requires enclave builder to mark enclave pages executable in the non-enclave VMAs, which may unnecessarily require EXECMOD on the source file, or even worse, EXECMEM, and potentially increases the attack surface since the file must be executable.

- Is completely unnecessary if the enclave holds a reference to the sigstruct file, as LSMs can easily apply labels to the sigstruct, e.g. EXECUTE on the sigstruct instead of EXECUTE on the source file.

After the bajillion mails we've generated, AIUI we've come up with two concepts that are viable: inheriting permissions from the source VMA vs. using sigstruct as a proxy for the enclave. Andy's proposals rely on the inheritance concept. The proposal below is based on the sigstruct proxy concept.

For those not familiar with SGX details, sigstruct can be used as a proxy because hardware enforces that the measurement stored in the sigstruct exactly matches the measurement generated by the enclave build process, e.g. adding a page as RX instead of R will change the measurement.

Core Concepts:
- FILE_{READ,WRITE,EXEC} on /dev/sgx/enclave effectively gates access to EPC. All real world enclaves will need all three permissions.
- sigstruct is the proxy for enclave from an LSM perspective, e.g. SELinux can define ENCLAVE__EXECUTE and ENCLAVE__EXECMOD and apply them to the sigstruct file.
- Take sigstruct at ECREATE so that ADD_REGION immediately followed by mprotect() works as expected (because SGX.mprotect() needs sigstruct to pass to security_enclave_mprotect(), see below).
- SGX driver takes a reference to the backing sigstruct file if it exists so that the file can be provided to LSMs during mprotect().
- Optional: SGX driver *requires* sigstruct to be backed by file, purely to enforce userspace infrastructure is in place for LSM support.

W^X handling:
- mmap() to /dev/sgx/enclave only allowed with PROT_NONE, i.e. force userspace through mprotect() to simplify the kernel implementation.
- Add vm_ops mprotect() ops hook (I'll refer to SGX's implementation as SGX.mprotect())
- Take explicit ALLOW_WRITE at ADD_REGION, a.k.a. EADD
- ADD_REGION also used to describe EAUG region (tentatively for SGX2).
- Track "can be written at some point in time (past or future)" as ALLOW_WRITE (to avoid confusiong with MAY_WRITE). A priori knowledge of writability avoids having to track/coordinate PROT_WRITE across VMAs and MMs.
- SGX.mprotect() returns -EPERM if PROT_WRITE && !ALLOW_WRITE.
- Add security_enclave_mprotect() LSM hook, called by SGX.mprotect(), e.g. int security_enclave_mprotect(struct file *sigstruct, unsigned long prot, bool allow_write)
- Intention is that EXECMOD is required if PROT_EXEC and ALLOW_WRITE.

Enclave {white,black}listing:
- Optional/Future: add security_enclave_create(), invoked during SGX ECREATE ioctl(), e.g.
int security_enclave_create(struct vm_area_struct *sigstruct)

- If this LSM hook is implemented, having sigstruct at ECREATE allows LSMs to determine whether or not the enclave is allowed to execute before allocating EPC for the enclave, e.g. unwanted enclaves can't DoS wanted enclaves.

LSM implementation possibilities:

- Define ENCLAVE__EXECUTE and ENCLAVE__EXECMOD, require them on the process. Does not require sigstruct to be backed by file, but cannot achieve per-enclave granularity.

- Define ENCLAVE__EXECUTE and ENCLAVE__EXECMOD, require them on the sigstruct, i.e. force sigstruct to reside in filesystem. Allows per-enclave granularity.

- Reuse FILE__EXECUTE and FILE__EXECMOD on sigstruct. Likely has implications that may or may not be concerning, e.g. the sigstruct file itself is weirdly executable.

- Adding ENCLAVE__EXECUTE and ENCLAVE__EXECMOD means the sigstruct, which may be emdedded in the same file as the enclave, does *not* require FILE__EXECUTE or FILE__EXECMOD, e.g. can be read-only.

- LSMs can (will?) require ENCLAVE__EXECUTE and ENCLAVE__EXECMOD to effectively map an enclave, even if the process acquired the enclave via SCM_RIGHTS (enclaves are tracked by fds). This is good or bad depending on your perspective.

Userspace changes:

- EADD ioctl adds flags param to take ALLOW_WRITE

- ECREATE ioctl takes sigstruct instead of EINIT

- Initial mmap() must be PROT_NONE.

- sigstruct likely needs to reside in a file (this may not affect some userspace implementations).

maintainer
#197 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 41 lines · Sean Christopherson
Quoted 19 lines · Andy Lutomirski
Quoted 18 lines · Stephen Smalley
Quoted 16 lines · Andy Lutomirski

Hi all-

After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.

Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.

First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:

- You can create a WX or RWX mapping if and only if you have EXECMEM.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.

EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or /dev/sgx/enclave?

I leave that decision to you :) The user should need permission to do an execmod thing on an enclave, however that wants to be encoded.

But that decision dictates how the SGX API handles sigstruct. If LSMs want to associate EXECMOD with sigstruct, then SGX needs to take sigstruct early and hold a reference to the file for the lifetime of the enclave. And if we're going to do that, the whole approach of inheriting permissions from source VMAs becomes unnecessary complexity.

Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or /dev/sgx/enclave?

Enclave file -- that is, the file backing the vma from which the data is loaded.

It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).

I actually meant for it to *not* work like this. I don't want the source VMA to have to be VM_EXEC. I think the LSM should just check permissions on ->vm_file.

#198 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 6 lines · Andy Lutomirski
Quoted 4 lines · Sean Christopherson
Quoted 1 line · Andy Lutomirski

Enclave file -- that is, the file backing the vma from which the data is loaded.

Quoted 7 lines
Quoted 3 lines

It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).

I actually meant for it to *not* work like this. I don't want the source VMA to have to be VM_EXEC. I think the LSM should just check permissions on ->vm_file.

But if ->vm_file is NULL, i.e. the enclave is not backed by a file, then PROCESS__EXECMEM is required (or more likely, ENCLAVE__EXECMEM).

In practice, it's the same net effect of using sigstruct as a proxy, i.e. *something* has to get to the file system to avoid EXECMEM. But putting the entire enclave to the filesystem seems like a heaver lift than dumping the sigstruct.

And if sigstruct needs to be in the file system for security_enclave_create/init()...

maintainer
#199 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Andy Lutomirski
Quoted 7 lines · Sean Christopherson
Quoted 1 line · Andy Lutomirski

Enclave file -- that is, the file backing the vma from which the data is loaded.

It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).

Quoted 7 lines
Quoted 3 lines

I actually meant for it to *not* work like this. I don't want the source VMA to have to be VM_EXEC. I think the LSM should just check permissions on ->vm_file.

But if ->vm_file is NULL, i.e. the enclave is not backed by a file, then PROCESS__EXECMEM is required (or more likely, ENCLAVE__EXECMEM).

If ->vm_file is NULL, then I think some privilege is needed. I suppose the policy could have a new lesser permission EXECUNTRUSTED which is like EXECMOD but you can't modify it. I'm not convinced this is particular important.

#200 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 18 lines · Andy Lutomirski
Quoted 16 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 7 lines · Sean Christopherson
Quoted 1 line · Andy Lutomirski

Enclave file -- that is, the file backing the vma from which the data is loaded.

It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).

I actually meant for it to *not* work like this. I don't want the source VMA to have to be VM_EXEC. I think the LSM should just check permissions on ->vm_file.

Quoted 8 lines
Quoted 3 lines

But if ->vm_file is NULL, i.e. the enclave is not backed by a file, then PROCESS__EXECMEM is required (or more likely, ENCLAVE__EXECMEM).

If ->vm_file is NULL, then I think some privilege is needed. I suppose the policy could have a new lesser permission EXECUNTRUSTED which is like EXECMOD but you can't modify it. I'm not convinced this is particular important.

Assuming MRENCLAVE generated by Graphene or any other hosting scheme are stable[1], then avoiding EXEC<whatever> means the user can effectively whitelist what enclaves are runnable by Graphene, even if the kernel doesn't implement security_enclave_create/init().

I agree that it probably isn't all that important, it's more of a "why not" argument, i.e. what is gained by not using sigstruct as a proxy?

[1] What in the world is being attested if MRENCLAVE isn't stable?

#201 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

On Thu, May 30, 2019 at 02:36:01PM -0700, Sean Christopherson wrote:

Good morning, I hope everyone had a pleasant weekend.

Quoted 10 lines

Assuming MRENCLAVE generated by Graphene or any other hosting scheme are stable[1], then avoiding EXEC<whatever> means the user can effectively whitelist what enclaves are runnable by Graphene, even if the kernel doesn't implement security_enclave_create/init().

I agree that it probably isn't all that important, it's more of a "why not" argument, i.e. what is gained by not using sigstruct as a proxy?

[1] What in the world is being attested if MRENCLAVE isn't stable?

The cryptographic identity of the entity that signed the enclave and generated the SIGSTRUCT.

At the risk of being the monotone in the choir, any relevant SGX security controls require verifying the identity of whoever signed the identity characteristics (SIGSTRUCT) of the image that initiates the execution of an SGX TEE. Other then verifying the initial execution image, the MRENCLAVE value isn't all that relevant.

This issue is further evidenced by the fact that sealing data to an enclave uses the MRSIGNER variant of ENCLU[EGETKEY] key derivation.

The current work on LSM controls seems to focus on the identity of the entity that is requesting the image to be loaded rather then who actually signed, and presumably authored, the code. As I have previously noted, with SGX2/EDMM, a platform owner may not even have any visibility into the code that an SGX TEE may ultimately load and execute.

Any security relevant LSM control in this space has to focus on providing the platform owner the ability to take action based on the contents of the SIGSTRUCT of the initiating image. In addition to the identity of who is requesting the image to be loaded.

Have a good week.

Dr. Greg

As always,

Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.

PH: 701-281-1686
FAX: 701-281-3949 EMAIL: [email protected]
------------------------------------------------------------------------------
"Experience is something you don't get until just after you need it."
-- Olivier

#202 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson
Quoted 9 lines · Sean Christopherson

Assuming MRENCLAVE generated by Graphene or any other hosting scheme are stable[1], then avoiding EXEC<whatever> means the user can effectively whitelist what enclaves are runnable by Graphene, even if the kernel doesn't implement security_enclave_create/init().

I agree that it probably isn't all that important, it's more of a "why not" argument, i.e. what is gained by not using sigstruct as a proxy?

[1] What in the world is being attested if MRENCLAVE isn't stable?

If I've understood correctly, Graphene uses a single loader enclave that loads the executable in.

/Jarkko

#203 lore raw

RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Andy Lutomirski

Hi Andy,

I'm just curious how Sean convinced you that "MAXPERM doesn't work". More specifically, I'm objecting to the statement of "the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed".

I'm still new to LSM so my understanding may not be correct. But I think LSM policy defines a boundary that "loosely" restricts what a process can do. By "loosely", I mean it is usually more permissive than a process needs. For example, FILE__EXECMOD basically says there are self-modifying code in a file, but it isn't specific on which pages contain self-modifying code, hence *all* pages are allowed mprotect(RW->RX) even though only a (small) subset actually need it. LSM policies are static too, as FILE__EXECMOD is given by a system admin to be associated with the disk file, instead of being requested/programmed by any processes loading/mapping that file.

So I think the same rationale applies to enclaves. Your original idea of MAXPERM is the policy set forth by system admin and shall *never* change at runtime. If an enclave is dynamically linked and needs to bring in code pages at runtime, the admin needs to enable it by setting, say ENCLAVE__EXECMOD, in the sigstruct file. Then all EAUG'ed pages will receive RWX as MAXPERM. The process would then mprotect() selective pages to be RX but which exact set of pages doesn't concern LSM usually.

Quoted 50 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Thursday, May 30, 2019 12:21 PM

Quoted 45 lines · Sean Christopherson
Quoted 20 lines · Andy Lutomirski
Quoted 19 lines · Stephen Smalley
Quoted 17 lines · Andy Lutomirski

Hi all-

After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.

Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.

First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's
discretion:

- You can create a WX or RWX mapping if and only if you have EXECMEM.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

- To create an X mapping of an enclave page that has ever been
W, you need EXECMOD.

EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or /dev/sgx/enclave?

I leave that decision to you :) The user should need permission to do an execmod thing on an enclave, however that wants to be encoded.

But that decision dictates how the SGX API handles sigstruct. If LSMs want to associate EXECMOD with sigstruct, then SGX needs to take sigstruct early and hold a reference to the file for the lifetime of the enclave. And if we're going to do that, the whole approach of inheriting permissions from source VMAs becomes unnecessary complexity.

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

- To create an X mapping of an enclave page that came from
EADD, you need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.

What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or /dev/sgx/enclave?

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Enclave file -- that is, the file backing the vma from which the data is loaded.

It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).

I actually meant for it to *not* work like this. I don't want the source VMA to have to be VM_EXEC. I think the LSM should just check permissions on ->vm_file.

-Cedric

#204 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 7 lines · Cedric

So I think the same rationale applies to enclaves. Your original idea of MAXPERM is the policy set forth by system admin and shall *never* change at runtime. If an enclave is dynamically linked and needs to bring in code pages at runtime, the admin needs to enable it by setting, say ENCLAVE__EXECMOD, in the sigstruct file. Then all EAUG'ed pages will receive RWX as MAXPERM. The process would then mprotect() selective pages to be RX but which exact set of pages doesn't concern LSM usually.

Because passing RWX means the enclave "requires" EXECMOD even if it never actually does a RW->RX transition. It's not broken per se, but at the very least it's decidedly odd.

Dynamically detecting the EXECMOD case is not difficult and has the advantage of simplifying userspace loaders, e.g. all EAUG pages are tagged ALLOW_WRITE and the kernel takes care of the rest.

I *think* auditing/learning is also messed up with a MAXPERMS approach, as mprotect() would fail (due to MAXPERMS clearing MAY_{READ,WRITE,EXEC}) before it calls security_file_mprotect(). Hooking mprotect() is the obvious workaround, but then it's looking a lot like the new proposals.

In other words, the new proposals are rooted in the MAXPERMS concept, e.g. MAXPERM is effectively "I want EXECMOD", which gets distilled down to ALLOW_WRITE (or ALLOW_EXEC in Andy's proposal).

#205 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson
Quoted 4 lines · Sean Christopherson

- Requires enclave builder to mark enclave pages executable in the non-enclave VMAs, which may unnecessarily require EXECMOD on the source file, or even worse, EXECMEM, and potentially increases the attack surface since the file must be executable.

Enclave builder marks *non-enclave pages*? Not following.

Quoted 11 lines

W^X handling:
- mmap() to /dev/sgx/enclave only allowed with PROT_NONE, i.e. force userspace through mprotect() to simplify the kernel implementation.
- Add vm_ops mprotect() ops hook (I'll refer to SGX's implementation as SGX.mprotect())
- Take explicit ALLOW_WRITE at ADD_REGION, a.k.a. EADD
- ADD_REGION also used to describe EAUG region (tentatively for SGX2).
- Track "can be written at some point in time (past or future)" as ALLOW_WRITE (to avoid confusiong with MAY_WRITE). A priori knowledge of writability avoids having to track/coordinate PROT_WRITE across VMAs and MMs.

Still not sure why you want to use vm_ops instead of file_operations.

The approach I've been proposing earlier in this email thread before these new proposals can be summarized from hook perspective as:

- Allow mmap() only before ECREATE and require it to be size of the ELRANGE (ECREATE ioctl would check this). This would be with PROT_NONE.
- Disallow mprotect() before EINIT. Requires a new callback to file_operations like mmap() has.
- After EINIT check for each mprotect() that it matches the permissions of underlying enclave pages. Disallow mmap() after EINIT.

/Jarkko

#206 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Andy Lutomirski
Quoted 5 lines · Andy Lutomirski
Quoted 2 lines

What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or /dev/sgx/enclave?

Enclave file -- that is, the file backing the vma from which the data is loaded.

Wonder why KVM gets away without having this given that enclaves are lot alike VMs.

Quoted 6 lines

It's provided by userspace based on whether it thinks the data in question is enclave code. source->vm_file is the file from which the code is being loaded. I'm assuming that the user code will only set excute_intent ==true if it actually wants to execute the code, so, if there's a denial, it will be fatal. The normal case will be that the request will be granted on the basis of EXECUTE.

AFAIK user spaces tells that already with the SECINFO flags. I don't get why we need a duplicate parameter.

/Jarkko

maintainer
#207 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 11 lines · Jarkko Sakkinen
Quoted 5 lines · Andy Lutomirski
Quoted 2 lines

What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or /dev/sgx/enclave?

Enclave file -- that is, the file backing the vma from which the data is loaded.

Wonder why KVM gets away without having this given that enclaves are lot alike VMs.

I would argue it's because access to /dev/kvm means you can execute whatever you code you want in a VM. I don't see how this is avoidable. On the other hand, it would be nice for SGX to not imply this same sort of "execute anything" right, especially since, unlike KVM, SGX is not a sandbox.

#208 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Jarkko Sakkinen
Quoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Andy Lutomirski
Quoted 2 lines

What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or /dev/sgx/enclave?

Enclave file -- that is, the file backing the vma from which the data is loaded.

Wonder why KVM gets away without having this given that enclaves are lot alike VMs.

From a memory management perspective, VMs are not at all like enclaves. An enclave is an extension of its host, i.e. runs in the same address. This isn't strictly necessary, e.g. an enclave could run in a sandbox process, but even then the enclave will be running with the kernel's standard page tables.

A VM is a essentially an opaque blob of data that gets loaded into memory. KVM builds a completely different set of page tables for the VM, the VM has it's own file system (or perhaps doesn't have a file system at all), etc... Ignoring Spectre and L1TF, the VM is contained to its own world.

There are a lot of ways for a userspace VMM to expose things beyond raw memory, but doing so requires the appropriate permissions.

And practically speaking, all traditional VMs will effectively need RWX memory, i.e. Qemu (or any other userspace VMM) would be required to have EXECMEM permissions, which would be a net negative for security.

Quoted 9 lines
Quoted 6 lines

It's provided by userspace based on whether it thinks the data in question is enclave code. source->vm_file is the file from which the code is being loaded. I'm assuming that the user code will only set excute_intent ==true if it actually wants to execute the code, so, if there's a denial, it will be fatal. The normal case will be that the request will be granted on the basis of EXECUTE.

AFAIK user spaces tells that already with the SECINFO flags. I don't get why we need a duplicate parameter.

Please read through the RFC, I think it address a lot of your questions. Hopefully that will help us avoid some thrash.

#209 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 2 lines · Sean Christopherson

Please read through the RFC, I think it address a lot of your questions. Hopefully that will help us avoid some thrash.

I promise to read it through with detail albeit I just said that as a patch set it is broken :-) Internals still need documentation tho...

/Jarkko

#210 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Stephen Smalley
Quoted 4 lines · Stephen Smalley

Does this occur for both setting initial permissions and runtime permissions or just runtime? Both userspace- and driver-initiated mmap/mprotect operations or just userspace-initiated ones? Does the driver use interfaces that call the mmap/mprotect hooks or lower level functions?

The driver never initiates mmap() or mprotect().

/Jarkko

#211 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Andy Lutomirski
Quoted 3 lines · Andy Lutomirski

- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.

Source file? EADD ioctl takes memory buffer in right now.

Quoted 5 lines

And I have two design proposals. One is static and one is dynamic. To implement either one, we will probably need a new .may_mprotect vm operation, and that operation can call an LSM hook. Or we can give LSMs a way to detect that a given vm_area_struct is an enclave. As I see it, this is an implementation detail that is certainly solveable.

Why VM operation and not file operation?

Quoted 7 lines

EADD takes an execute_intent flag. It calls a new hook:

  int security_enclave_load(struct vm_area_struct *source, bool execute_intent);

This hook will fail if execute_intent==true and the caller has neither EXECUTE, EXECMOD, nor EXECMEM.

Quoted 7 lines

EAUG sets execute_intent = false.

EINIT takes a sigstruct pointer. SGX can (when initially upstreamed or later on once there's demand) call a new hook:

security_enclave_init(struct sigstruct *sigstruct, struct
vm_area_struct *source);

What is the source VMA in these callbacks? Why is @execute_intent needed anyway as a ioctl arugment and not deduced from SECINFO?

/Jarkko

#212 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson

On Fri, May 24, 2019 at 01:03:33PM -0700, Sean Christopherson wrote:

Good morning, I hope the weekend is going well for everyone. Skunky holiday weather out here in West-Central Minnesota.

Quoted 13 lines · Andy Lutomirski
Quoted 7 lines

If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.

This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?

Quoted 7 lines

What's the value prop beyond whitelisting sigstructs? Realistically, I doubt LSMs/users will want to take the performance hit of scanning the source bytes every time an enclave is loaded.

We could add seomthing like security_enclave_mprotect() in lieu of abusing security_file_mprotect(), but passing the full source bytes seems a bit much.

It would seem that we hold the moniker of responsibility for this conversation, since without our provocation regarding cryptographic verification of enclave source, there would be a driver headed upstream whose only constraint against W^X sourced executable code, running with full confidentiality and integrity protections, would be a character device with o666 permissions. Given that, a couple of reflections to facilitate further conversation, if nothing else for the benefit of Jonathan Corbet and his bystanders... :-)

As the conversations to date have indicated, imposing LSM controls on enclave executable code is a bit problematic, in no small part since it is the theological equivalent of driving a square peg into a round hole. SGX, as a technology, was designed around the concept of cryptographic verification of code provenance and origin.

The decision to take that off the table, for reasons of political idealogy only, means that mainstream Linux will not be a platform that can achieve the full hardware security capabilities and protections of SGX, nor will mainstream Linux be able to enjoy full protections from the technology itself.

We will be dealing with that, from a driver and runtime perspective, but that is a conversation for another day.

The issue of SGX2 and Enclave Dynamic Memory Management (EDMM) has come up and to date there doesn't appear to have been a serious conversation regarding whether or not all of the LSM machinations in the world will make any difference when this technology goes mainline. The agenda driving mainlining of the driver is to support Graphene for cloud based solutions and without EDMM, dynamic code loading support is decidedly more problematic.

Dynamic enclave code loading isn't problematic from a security perspective when the code is being loaded from the platform itself, since presumably, the encompassing conversation will result in LSM controls being applied to the necessary code paths. However, with the ability to exploit SGX2 instructions, an enclave with adverserial intent could simply setup a mutually attested security context and pull whatever executable code it wants from the INTERNET at large, using an encrypted and integrity protected communications channel.

That has at least been our interpretation and experience with the ENCLU[EMODPE] and ENCLU[EACCEPTCOPY] instructions and the out-of-tree driver. Given the use of an encrypted channel, and the fact that these instructions are ring 3 enclave mode only, it would seem that all of the LSM controls in the world won't have visibility or control over code that is being loaded and executed using such a mechanism.

We could have arguably missed something that the new driver will do to address this issue. To date the only discussion seems to have been about controls over ENCLS[EAUG], which are arguably a bit blunt for this purpose.

In the land of SGX, if one is intellectually honest from an engineering perspective, the only solid security contract one has to work with is the notion of cryptographic identity. Hence our concern and patches that implemented an absolutely minimal footprint ring-0 control infrastructure over the contents of an enclave's SIGSTRUCT. Which is where we have arguably circled back to after 3-4 months and one kernel release cycle.

Wrapping an LSM hook around our policy mechanism would seem to achieve, from a security perspective, about the same level of security effect that more major and invasive modifications would achieve, given Cedric's proposal to inherit page permissions from the source, which is what our runtime already does.

As always, apologies for excessive verbosity beyond LKML sensibilities.

Best wishes for a pleasant remainder of the spring weekend to everyone.

Dr. Greg

As always,

Dr. G.W. Wettstein, Ph.D.   Enjellic Systems Development, LLC.
4206 N. 19th Ave.           Specializing in information infra-structure
Fargo, ND  58102            development.

PH: 701-281-1686
FAX: 701-281-3949 EMAIL: [email protected]
------------------------------------------------------------------------------
"Heaven goes by favor. If it went by merit, you would stay out and your
dog would go in."
-- Mark Twain

maintainer
#213 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Xing, Cedric
Quoted 66 lines · Cedric

Hi Andy,

Quoted 63 lines

From: Andy Lutomirski [mailto:[email protected]]
Sent: Thursday, May 23, 2019 6:18 PM

Quoted 17 lines · Sean Christopherson
Quoted 15 lines · Andy Lutomirski
Quoted 13 lines · Sean Christopherson
Quoted 10 lines · Jarkko Sakkinen
Quoted 7 lines · Sean Christopherson

But actually, there's no need to disallow mmap() after ECREATE since the LSM checks also apply to mmap(), e.g. FILE__EXECUTE would be needed to
mmap() any enclave pages PROT_EXEC. I guess my past self thought mmap() bypassed LSM checks? The real problem is that mmap()'ng an existing enclave would require FILE__WRITE and FILE__EXECUTE, which puts us back at square one.

I'm lost with the constraints we want to set.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

As is today, SELinux policies would require enclave loaders to
have FILE__WRITE and FILE__EXECUTE permissions on
/dev/sgx/enclave. Presumably other LSMs have similar
requirements. Requiring all processes to have
FILE__{WRITE,EXECUTE} permissions means the permissions don't add much value, e.g. they can't be used to distinguish between an enclave that is being loaded from an unmodified file and an enclave that is being

generated on the fly, e.g. Graphene.

Quoted 18 lines
Quoted 12 lines
Quoted 5 lines

Looking back at Andy's mail, he was talking about requiring FILE__EXECUTE to run an enclave, so perhaps it's only FILE__WRITE that we're trying to special case.

I thought about this some more, and I have a new proposal that helps address the ELRANGE alignment issue and the permission issue at the cost of some extra verbosity. Maybe you all can poke holes in it :) The basic idea is to make everything more explicit from a user's perspective. Here's how it works:

Quoted 5 lines

Opening /dev/sgx/enclave gives an enclave_fd that, by design, doesn't give EXECUTE or WRITE. mmap() on the enclave_fd only works if you pass PROT_NONE and gives the correct alignment. The resulting VMA cannot be mprotected or mremapped. It can't be mmapped at all until

Quoted 7 lines

I assume you're thinking of clearing all VM_MAY* flags in sgx_mmap()?

Quoted 1 line

after ECREATE because the alignment isn't known before that.

I don't follow. The alignment is known because userspace knows the size of its enclave. The initial unknown is the address, but that becomes known once the initial mmap() completes.

[...]

I think I made the mistake of getting too carried away with implementation details rather than just getting to the point. And I misremembered the ECREATE flow -- oops. Let me try again. First, here are some problems with some earlier proposals (mine, yours Cedric's):

Quoted 6 lines
Quoted 2 lines

- Having the EADD operation always work but have different effects depending on the
source memory permissions is, at the very least, confusing.

Inheriting permissions from source pages IMHO is the easiest way to validate the EPC permissions without any changes to LSM. And the argument about its security is also easy to make.

I understand that it may take some effort to document it properly but otherwise don't see any practical issues with it.

My objection is to the fact that it's implicit. I have no problem with some operation succeeding if the source address is X and failing if it's !X, but I don't think it's fantastic to have it succeed in either case but do different things.

For what it's worth, while this is a bit of a theoretical issue for X, but I think it's a real problem with W. To avoid accidentally mapping an enclave page X and then later mapping the same page W (potentially in a different VMA), I think it will be a lot simpler if the driver can track which pages are allowed to ever be W. We definitely *don't* want an interface in which the eventual writability of a page is inferred from the W permission on the source address, since we do *not* want to force anyone to map their enclave file PROT_WRITE or even to open it O_RDWR.

With the explicit MAXPERM passed in, this issue goes away. You can specify W if you want W.

Quoted 25 lines
Quoted 25 lines

- If we want to encourage user programs to be well-behaved, we want to make it easy to
map the RX parts of an enclave RX, the RW parts RW, the RO parts R, etc. But this interacts poorly with the sgx_mmap() alignment magic, as you've pointed out.

- We don't want to couple LSMs with SGX too tightly.

So here's how a nice interface might work:

int enclave_fd = open("/dev/sgx/enclave", O_RDWR);

/* enclave_fd points to a totally blank enclave. Before ECREATE, we need to decide on an address. */

void *addr = mmap(NULL, size, PROT_NONE, MAP_SHARED, enclave_fd, 0);

/* we have an address! */

ioctl(enclave_fd, ECREATE, ...);

/* now add some data to the enclave. We want the RWX addition to fail
immediately unless we have the relevant LSM pemission. Similarly, we
want the RX addition to fail immediately unless the source VMA is appropriate. */

Quoted 6 lines
Quoted 4 lines

ioctl(enclave_fd, EADD, rx_source_1, MAXPERM=RX, ...); [the ... includes SECINFO, which the kernel doesn't really care about] ioctl(enclave_fd, EADD, ro_source_1, MAXPERM=RX ...); ioctl(enclave_fd, EADD, rw_source_1, MAXPERM=RW ...); ioctl(enclave_fd, EADD, rwx_source_1, MAXPERM=RWX ...);

If MAXPERM is taken from ioctl parameters, the real question here is how to validate MAXPERM. Guess we shouldn't allow arbitrary MAXPERM to be specified by user code, and the only logical source I can think of is from the source pages (or from the enclave source file, but memory mapping is preferred because it offers more flexibility).

That's exactly what I intended here. If you specify MAXPERM=RX, then the kernel can validate that the source address is executable.

Quoted 12 lines
Quoted 12 lines

ioctl(enclave_fd, EINIT, ...); /* presumably pass sigstruct_fd here, too. */

/* at this point, all is well except that the enclave is mapped PROT_NONE. There are a couple ways I can imagine to fix this. */

We could use mmap:

mmap(baseaddr+offset, len, PROT_READ, MAP_SHARED | MAP_FIXED, enclave_fd, 0); /* only succeeds if MAXPERM & R == R */

Quoted 6 lines
Quoted 2 lines

But this has some annoying implications with regard to sgx_get_unmapped_area(). We could use an ioctl:

There's an easy fix. Just let sgx_get_unmapped_area() do the natural alignment only if MAP_FIXED is *not* set, otherwise, honor both address and len.

But mmap() is subject to LSM check (probably against /dev/sgx/enclave?). How to do mmap(RX) if FILE__EXECUTE is *not* granted for /dev/sgx/enclave, even if MAXPERM=RX?

I think we just let /dev/sgx/enclave be FILE__EXECUTE. We don't *have* to make it so that FILE__WRITE and FILE__EXECUTE on /dev/sgx/enclave means you can create RWX enclave mappings.

Quoted 4 lines
Quoted 4 lines

ioctl(enclave_fd, SGX_IOC_MPROTECT, offset, len, PROT_READ);

Quoted 8 lines
Quoted 6 lines

which has the potentially nice property that we can completely bypass the LSM hooks, because the LSM has *already* vetted everything when the EADD calls were allowed. Or we could maybe even just use
mprotect() itself:

mprotect(baseaddr + offset, len, PROT_READ);

How to bypass LSM hooks in this mprotect()?

Hmm. I guess we either use FILE__WRITE and FILE__EXECUTE or we use ioctl().

Quoted 18 lines
Quoted 8 lines

Or, for the really evil option, we could use a bit of magic in .fault and do nothing here. Instead we'd make the initial mapping PROT_READ|PROT_WRITE|PROT_EXEC and have .fault actually instantiate the PTEs with the intersection of the VMA permissions and MAXPERM. I don't think I like this alternative, since it feels more magical than needed and it will be harder to debug. I like the fact that /proc/self/maps shows the actual permissions in all the other variants.

Agreed.

Quoted 7 lines

All of the rest of the crud in my earlier email was just implementation details. The point I was trying to make was that I think it's possible to implement this without making too much of a mess internally. I think I favor the mprotect() approach since it makes the behavior fairly obvious.

Quoted 9 lines
Quoted 6 lines

I don't think any of this needs to change for SGX2. We'd have an
ioctl() that does EAUG and specifies MAXPERM. Trying to mprotect() a page that hasn't been added yet with any permission other than PROT_NONE would fail. I suppose we might end up needing a way to let the EAUG operation *change* MAXPERM, and this operation would have to do some more LSM checks and walk all the existing mappings to make sure they're consistent with the new MAXPERM.

EAUG ioctl could be a solution, but isn't optimal at least. What we've done is #PF based. Specifically, an SGX2 enclave will have its heap mapped as RW, but without any pages populated before EINIT. Then when the enclave needs a new page in its heap, it issues EACCEPT, which will cause a #PF and the driver will respond by EAUG a new EPC page. And then the enclave will be resumed and the faulted EACCEPT will be retried (and succeed).

If the driver works like that, then whatever call sets up this lazily allocated heap could do the MAXPERM part.

That being said, is the performance advantage from putting this logic in the kernel instead of in the untrusted part of the SDK really worthwhile?

Quoted 11 lines
Quoted 6 lines

As an aside, I wonder if Linus et all would be okay with a new MAP_FULLY_ALIGNED mmap() flag that allocated memory aligned to the requested size. Then we could get rid of yet another bit of magic.

--Andy

I've also got a chance to think more about it lately.

When we talk about EPC page permissions with SGX2 in mind, I think we should distinguish between initial permissions and runtime permissions. Initial permissions refer to the page permissions set at EADD. They are technically set by "untrusted" code so should go by policies similar to those applicable to regular shared objects. Runtime permissions refer to the permissions granted by EMODPE, EAUG and EACCEPTCOPY. They are resulted from inherent behavior of the enclave, which in theory is determined by the enclave's measurements (MRENCLAVE and/or MRSIGNER).

Quoted 8 lines

And we have 2 distinct files to work with - the enclave file and /dev/sgx/enclave. And I consider the enclave file a logical source for initial permissions while /dev/sgx/enclave is a means to control runtime permissions. Then we can have a simpler approach like the pseudo code below.

/**
* Summary:
* - The enclave file resembles a shared object that contains RO/RX/RW segments
* - FILE__* are assigned to /dev/sgx/enclave, to determine acceptable permissions to mmap()/mprotect(), valid combinations are
* + FILE__READ - Allow SGX1 enclaves only
* + FILE__READ|FILE__WRITE - Allow SGX2 enclaves to expand data segments (e.g. heaps, stacks, etc.)

I think this is a non-starter :( FILE__WRITE also means that you can write to the file, and the admin / policy author will almost never want to allow that.

#214 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 11 lines · Andy Lutomirski
Quoted 6 lines · Cedric

/**
* Summary:
* - The enclave file resembles a shared object that contains RO/RX/RW segments
* - FILE__* are assigned to /dev/sgx/enclave, to determine acceptable permissions to mmap()/mprotect(), valid combinations are
* + FILE__READ - Allow SGX1 enclaves only
* + FILE__READ|FILE__WRITE - Allow SGX2 enclaves to expand data segments (e.g. heaps, stacks, etc.)

I think this is a non-starter :( FILE__WRITE also means that you can write to the file, and the admin / policy author will almost never want to allow that.

Why would FILE__WRITE on /dev/sgx/enclave be a problem? An actual write to /dev/sgx/enclave would yield -EINVAL, no?

#215 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Andy Lutomirski
Quoted 7 lines · Cedric

/**
* Summary:
* - The enclave file resembles a shared object that contains RO/RX/RW segments
* - FILE__* are assigned to /dev/sgx/enclave, to determine acceptable permissions to mmap()/mprotect(), valid combinations are
* + FILE__READ - Allow SGX1 enclaves only
* + FILE__READ|FILE__WRITE - Allow SGX2 enclaves to expand data segments (e.g. heaps, stacks, etc.)

Quoted 6 lines
Quoted 3 lines

I think this is a non-starter :( FILE__WRITE also means that you can write to the file, and the admin / policy author will almost never want to allow that.

Why would FILE__WRITE on /dev/sgx/enclave be a problem? An actual write to /dev/sgx/enclave would yield -EINVAL, no?

Bah, read it wrong — FILE__WRITE on the enclave file on disk is no good.

#216 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Andy Lutomirski
Quoted 60 lines · Andy Lutomirski
Quoted 20 lines · Sean Christopherson
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Sean Christopherson

But actually, there's no need to disallow mmap() after ECREATE since the LSM checks also apply to mmap(), e.g. FILE__EXECUTE would be needed to mmap() any enclave pages PROT_EXEC. I guess my past self thought mmap() bypassed LSM checks? The real problem is that mmap()'ng an existing enclave would require FILE__WRITE and FILE__EXECUTE, which puts us back at square one.

I'm lost with the constraints we want to set.

As is today, SELinux policies would require enclave loaders to have FILE__WRITE and FILE__EXECUTE permissions on /dev/sgx/enclave. Presumably other LSMs have similar requirements. Requiring all processes to have FILE__{WRITE,EXECUTE} permissions means the permissions don't add much value, e.g. they can't be used to distinguish between an enclave that is being loaded from an unmodified file and an enclave that is being generated on the fly, e.g. Graphene.

Quoted 4 lines

Looking back at Andy's mail, he was talking about requiring FILE__EXECUTE to run an enclave, so perhaps it's only FILE__WRITE that we're trying to special case.

I thought about this some more, and I have a new proposal that helps address the ELRANGE alignment issue and the permission issue at the cost of some extra verbosity. Maybe you all can poke holes in it :) The basic idea is to make everything more explicit from a user's perspective. Here's how it works:

Opening /dev/sgx/enclave gives an enclave_fd that, by design, doesn't give EXECUTE or WRITE. mmap() on the enclave_fd only works if you pass PROT_NONE and gives the correct alignment. The resulting VMA cannot be mprotected or mremapped. It can't be mmapped at all until after ECREATE because the alignment isn't known before that.

Associated with the enclave are a bunch (up to 7) "enclave segment inodes". These are anon_inodes that are created automagically. An enclave segment is a group of pages, not necessary contiguous, with an upper bound on the memory permissions. Each enclave page belongs to a segment. When you do EADD, you tell the driver what segment you're adding to. [0] This means that EADD gets an extra argument that is a permission mask for the page -- in addition to the initial SECINFO, you also pass to EADD something to the effect of "I promise never to map this with permissions greater than RX".

Then we just need some way to mmap a region from an enclave segment. This could be done by having a way to get an fd for an enclave segment or it could be done by having a new ioctl SGX_IOC_MAP_SEGMENT. User code would use this operation to replace, MAP_FIXED-style, ranges from the big PROT_NONE mapping with the relevant pages from the enclave segment. The resulting vma would only have VM_MAYWRITE if the segment is W, only have VM_MAYEXEC if the segment is X, and only have VM_MAYREAD if the segment is R. Depending on implementation details, the VMAs might need to restrict mremap() to avoid mapping pages that aren't part of the segment in question.

Quoted 8 lines

It's plausible that this whole thing works without the magic segment inodes under the hood, but figuring that out would need a careful look at how all the core mm bits and LSM bits work together.

To get all the LSM stuff to work, SELinux will need some way to automatically assign an appropriate label to the segment inodes. I assume that such a mechanism already exists and gets used for things like sockets, but I haven't actually confirmed this.

I don't follow that. socket inodes are not anon inodes, and anon inodes have no per-instance data by definition, and typically you're only dealing with a single anon inode for all files, and hence they were long ago marked S_PRIVATE and exempted from all LSM checking except for EXECMEM on mmap/mprotect PROT_EXEC. We have no way to perform useful security checking on them currently. socket inodes we can label from their creating process but even that's not going to support multiple labels for different sockets created by the same process unless the process explicitly used setsockcreatecon(3) aka /proc/self/attr/sockcreate

Quoted 5 lines

[0] There needs to be some vaguely intelligent semantics if you EADD the *same* address more than once. A simple solution would be to disallow it if the segments don't match.

#217 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Andy Lutomirski
Quoted 33 lines · Andy Lutomirski
Quoted 20 lines · Sean Christopherson
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Sean Christopherson

But actually, there's no need to disallow mmap() after ECREATE since the LSM checks also apply to mmap(), e.g. FILE__EXECUTE would be needed to mmap() any enclave pages PROT_EXEC. I guess my past self thought mmap() bypassed LSM checks? The real problem is that mmap()'ng an existing enclave would require FILE__WRITE and FILE__EXECUTE, which puts us back at square one.

I'm lost with the constraints we want to set.

As is today, SELinux policies would require enclave loaders to have FILE__WRITE and FILE__EXECUTE permissions on /dev/sgx/enclave. Presumably other LSMs have similar requirements. Requiring all processes to have FILE__{WRITE,EXECUTE} permissions means the permissions don't add much value, e.g. they can't be used to distinguish between an enclave that is being loaded from an unmodified file and an enclave that is being generated on the fly, e.g. Graphene.

Quoted 4 lines

Looking back at Andy's mail, he was talking about requiring FILE__EXECUTE to run an enclave, so perhaps it's only FILE__WRITE that we're trying to special case.

I thought about this some more, and I have a new proposal that helps address the ELRANGE alignment issue and the permission issue at the cost of some extra verbosity. Maybe you all can poke holes in it :) The basic idea is to make everything more explicit from a user's perspective. Here's how it works:

Quoted 5 lines

Opening /dev/sgx/enclave gives an enclave_fd that, by design, doesn't give EXECUTE or WRITE. mmap() on the enclave_fd only works if you pass PROT_NONE and gives the correct alignment. The resulting VMA cannot be mprotected or mremapped. It can't be mmapped at all until after ECREATE because the alignment isn't known before that.

How to deny mprotect()? struct file_operations does not have callback for that (AFAIK).

Quoted 25 lines

Associated with the enclave are a bunch (up to 7) "enclave segment inodes". These are anon_inodes that are created automagically. An enclave segment is a group of pages, not necessary contiguous, with an upper bound on the memory permissions. Each enclave page belongs to a segment. When you do EADD, you tell the driver what segment you're adding to. [0] This means that EADD gets an extra argument that is a permission mask for the page -- in addition to the initial SECINFO, you also pass to EADD something to the effect of "I promise never to map this with permissions greater than RX".

Then we just need some way to mmap a region from an enclave segment. This could be done by having a way to get an fd for an enclave segment or it could be done by having a new ioctl SGX_IOC_MAP_SEGMENT. User code would use this operation to replace, MAP_FIXED-style, ranges from the big PROT_NONE mapping with the relevant pages from the enclave segment. The resulting vma would only have VM_MAYWRITE if the segment is W, only have VM_MAYEXEC if the segment is X, and only have VM_MAYREAD if the segment is R. Depending on implementation details, the VMAs might need to restrict mremap() to avoid mapping pages that aren't part of the segment in question.

It's plausible that this whole thing works without the magic segment inodes under the hood, but figuring that out would need a careful look at how all the core mm bits and LSM bits work together.

Quoted 8 lines

To get all the LSM stuff to work, SELinux will need some way to automatically assign an appropriate label to the segment inodes. I assume that such a mechanism already exists and gets used for things like sockets, but I haven't actually confirmed this.

[0] There needs to be some vaguely intelligent semantics if you EADD the *same* address more than once. A simple solution would be to disallow it if the segments don't match.

What if instead simply:

- Require to do PROT_NONE mmap() for the ELRANGE before ECREATE.
- Disallow mprotect() up until EINIT.
- Given that we have a callback for mprotect() check that permissions match EADD'd permissions.

/Jarkko

#218 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson
Quoted 19 lines · Sean Christopherson
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Sean Christopherson

But actually, there's no need to disallow mmap() after ECREATE since the LSM checks also apply to mmap(), e.g. FILE__EXECUTE would be needed to mmap() any enclave pages PROT_EXEC. I guess my past self thought mmap() bypassed LSM checks? The real problem is that mmap()'ng an existing enclave would require FILE__WRITE and FILE__EXECUTE, which puts us back at square one.

I'm lost with the constraints we want to set.

As is today, SELinux policies would require enclave loaders to have FILE__WRITE and FILE__EXECUTE permissions on /dev/sgx/enclave. Presumably other LSMs have similar requirements. Requiring all processes to have FILE__{WRITE,EXECUTE} permissions means the permissions don't add much value, e.g. they can't be used to distinguish between an enclave that is being loaded from an unmodified file and an enclave that is being generated on the fly, e.g. Graphene.

Quoted 3 lines

Looking back at Andy's mail, he was talking about requiring FILE__EXECUTE to run an enclave, so perhaps it's only FILE__WRITE that we're trying to special case.

Argh, as I was working through Andy's latest proposal I realized that I was subconciously making FILE__READ imply FILE__EXECUTE.

The idea behind inheriting permissions from the source VMA is to exempt "standard" enclaves from needing FILE__WRITE. But if we don't add an exemption for FILE__EXECUTE as well, then all enclaves need FILE__EXECUTE, which means FILE__EXECUTE can't be used to identify the case where userspace is mapping an inherited PROT_WRITE page as PROT_EXEC. And if the SGX magic exempts FILE__EXECUTE, then FILE__READ implies FILE__EXECUTE.

Yuck.

#219 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson
Quoted 5 lines · Sean Christopherson

1. Do nothing. Userspace would essentially be required to mmap() the enclave after EINIT, which is ugly but not breaking since userspace could mmap() the enclave with a placeholder VMA prior to building the enclave, and then a series of mmap() to establish its "real" mapping.

What it'd break to return error if mmap() is done before EINIT?

Quoted 2 lines

2. Propagate the permissions from EADD to the VMAs of the current mm if the entire EADD range is mapped and the mapping is PROT_NONE.

Right now you can do multiple mmap's. If the mmap's must be done after EINIT, the driver could check that permissions match the permissions in that range.

This leaves open how to deal with mprotect() but if the process does not have FILE__WRITE I guess you cannot do much.

Quoted 3 lines

3. Propagate the permissions from EADD to the VMAs of all mm structs that have mapped some piece of the enclave, following the matching rules from #2.

For me it looks that allowing mmap's only after EINIT would result the least confusing implemntation.

/Jarkko

#220 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 19 lines · Jarkko Sakkinen
Quoted 5 lines · Sean Christopherson

1. Do nothing. Userspace would essentially be required to mmap() the enclave after EINIT, which is ugly but not breaking since userspace could mmap() the enclave with a placeholder VMA prior to building the enclave, and then a series of mmap() to establish its "real" mapping.

What it'd break to return error if mmap() is done before EINIT?

Quoted 2 lines

2. Propagate the permissions from EADD to the VMAs of the current mm if the entire EADD range is mapped and the mapping is PROT_NONE.

Right now you can do multiple mmap's. If the mmap's must be done after EINIT, the driver could check that permissions match the permissions in that range.

This leaves open how to deal with mprotect() but if the process does not have FILE__WRITE I guess you cannot do much.

Quoted 6 lines
Quoted 3 lines

3. Propagate the permissions from EADD to the VMAs of all mm structs that have mapped some piece of the enclave, following the matching rules from #2.

For me it looks that allowing mmap's only after EINIT would result the least confusing implemntation.

Obvious problem is of course the requirement of fixed mapping, which is of course nasty.

/Jarkko

#221 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Andy Lutomirski
Quoted 3 lines · Andy Lutomirski

As far as I know from this whole discussion, we still haven't come up with any credible way to avoid tracking, per enclave page, whether that page came from unmodified PROT_EXEC memory.

So is this in the context that the enclave is read from another VMA and not through a file descriptor? Is that locked in?

/Jarkko

#222 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

Quoted 7 lines · Jarkko Sakkinen
Quoted 3 lines · Andy Lutomirski

As far as I know from this whole discussion, we still haven't come up with any credible way to avoid tracking, per enclave page, whether that page came from unmodified PROT_EXEC memory.

So is this in the context that the enclave is read from another VMA and not through a file descriptor? Is that locked in?

No need to answer. Got in page from Sean's response.

/Jarkko

#223 lore raw

Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)

In reply to Sean Christopherson
Quoted 4 lines · Sean Christopherson

The SGX ioctl() would need to take mmap_sem for write, but we can mitigate that issue by changing the ioctl() to take a range of memory instead of a single page. That'd also provide "EADD batching" that folks have requested.

This should be easy enough to add as the EADD operations are already batched internally to a worker thread.

/Jarkko

#224 lore raw
In reply to Sean Christopherson
Quoted 22 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 11 lines · Jarkko Sakkinen
Quoted 8 lines · Jarkko Sakkinen

I did study through SDK's file format and realized that it does not does make sense after all to embed one.

To implement it properly you would probably need a new syscall (lets say sgx_load_enclave) and also that enclaves are not just executables binaries. It is hard to find a generic format for them as applications range from simply protecting part of an application to running a containter inside enclave.

Quoted 7 lines
Quoted 2 lines

I'm still puzzling what kind of changes you were discussing considering SGX_IOC_ENCLAVE_ADD_PAGE.

I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?)

Quoted 7 lines

Nit: SECINFO is just the struct passed to EADD, I think what you're really asking is "can the EPCM permissions be changed after the fact".

And the answer is, yes.

On SGX2 hardware, the enclave can extend the EPCM permissions at runtime via ENCLU[EMODPE], e.g. to make a page writable.

Small correction: it is EMODPR.

Anyway, it is good to mention that these would require EACCEPT from the enclave side. In order to take advantage of this is in a malicous enclave, one would require SELinux/IMA/whatnot policy to have permitted it in the first place.

Thus, it cannot be said that it breaks the security policy if this would happen because policy has allowed to use the particular enclave.

Quoted 8 lines

Hardware also doesn't prevent doing EADD to the same virtual address multiple times, e.g. an enclave could EADD a RX page, and then EADD a RW page at the same virtual address with different data. The second EADD will affect MRENCLAVE, but so long as it's accounted for by the enclave's signer, it's "legal". SGX_IOC_ENCLAVE_ADD_PAGE *does* prevent adding the "same" page to an enclave multiple times, so effectively this scenario is blocked by the current implementation, but it's more of a side effect (of a sane implementation) as opposed to deliberately preventing shenanigans.

If the security policy can define who can create legit SIGSTRUCT files, this should not be a problem. Neither should be how EEXTEND is used.

This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?

If we can sort this question out, then SIGSTRUCT-centered way to control enclave would actually be robust.

/Jarkko

#225 lore raw
Quoted 4 lines · Jarkko Sakkinen

This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?

Also wondering if a path would be better than plain fd for defining a reasonable policy i.e. have sigstruct_path as part of the ioctl parameters and not sigstruct_fd.

/Jarkko

#226 lore raw
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Jarkko Sakkinen

This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?

The policy will presumably check the label on the file that the fd points to.

Quoted 4 lines

Also wondering if a path would be better than plain fd for defining a reasonable policy i.e. have sigstruct_path as part of the ioctl parameters and not sigstruct_fd.

It would save two syscalls at the cost of a decent amount of complexity.

#227 lore raw
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Jarkko Sakkinen

This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?

The policy will presumably check the label on the file that the fd points to.

Right (checked SELinux documentation).

Got one idea from this. Right now creation and initialization does not require any VMAs to be created (since v20). Requiring to map a VMA for copying the data would bring in my opinion a glitch to this model that we have done effort to build up.

What if we similarly change EADD ioctl in a way that it'd take an fd and an offset? This way we can enforce policy to the source where the enclave data is loaded from. On the other hand, loading SIGSTRUCT from fd enforces a legit structure for the enclave.

This would still allow to construct enclaves in VMA independent way.

Quoted 6 lines
Quoted 4 lines

Also wondering if a path would be better than plain fd for defining a reasonable policy i.e. have sigstruct_path as part of the ioctl parameters and not sigstruct_fd.

It would save two syscalls at the cost of a decent amount of complexity.

And also using fd gives robustness because it allows SIGSTRUCT pulled from a file containing it (among other things).

/Jarkko

#228 lore raw
Quoted 20 lines · Jarkko Sakkinen
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Jarkko Sakkinen

This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?

The policy will presumably check the label on the file that the fd points to.

Right (checked SELinux documentation).

Got one idea from this. Right now creation and initialization does not require any VMAs to be created (since v20). Requiring to map a VMA for copying the data would bring in my opinion a glitch to this model that we have done effort to build up.

Quoted 6 lines

What if we similarly change EADD ioctl in a way that it'd take an fd and an offset? This way we can enforce policy to the source where the enclave data is loaded from. On the other hand, loading SIGSTRUCT from fd enforces a legit structure for the enclave.

This would still allow to construct enclaves in VMA independent way.

The API would turn into this:

/**
* struct sgx_enclave_add_page - parameter structure for the
* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
* @fd: file containing the page data
* @offset: offset in the file containing the page data
* @secinfo: address for the SECINFO data
* @mrmask: bitmask for the measured 256 byte chunks
*/

struct sgx_enclave_add_page {
	__u64	fd;
	__u64	offset;
	__u64	secinfo;
	__u16	mrmask;
} __attribute__((__packed__));

/**
* struct sgx_enclave_init - parameter structure for the
* %SGX_IOC_ENCLAVE_INIT ioctl
* @fd: file containing the sigstruct
* @offset: offset in the file containing the sigstruct
*/

struct sgx_enclave_init {
	__u64	fd;
	__u64	offset;
};

/Jarkko

#229 lore raw
Quoted 52 lines · Jarkko Sakkinen
Quoted 20 lines · Jarkko Sakkinen
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Jarkko Sakkinen

This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?

The policy will presumably check the label on the file that the fd points to.

Right (checked SELinux documentation).

Got one idea from this. Right now creation and initialization does not require any VMAs to be created (since v20). Requiring to map a VMA for copying the data would bring in my opinion a glitch to this model that we have done effort to build up.

Quoted 6 lines

What if we similarly change EADD ioctl in a way that it'd take an fd and an offset? This way we can enforce policy to the source where the enclave data is loaded from. On the other hand, loading SIGSTRUCT from fd enforces a legit structure for the enclave.

This would still allow to construct enclaves in VMA independent way.

The API would turn into this:

/**
* struct sgx_enclave_add_page - parameter structure for the
* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
* @fd: file containing the page data
* @offset: offset in the file containing the page data
* @secinfo: address for the SECINFO data
* @mrmask: bitmask for the measured 256 byte chunks
*/

struct sgx_enclave_add_page {
	__u64	fd;
	__u64	offset;
	__u64	secinfo;
	__u16	mrmask;
} __attribute__((__packed__));

/**
* struct sgx_enclave_init - parameter structure for the
* %SGX_IOC_ENCLAVE_INIT ioctl
* @fd: file containing the sigstruct
* @offset: offset in the file containing the sigstruct
*/

Quoted 4 lines
struct sgx_enclave_init {
	__u64	fd;
	__u64	offset;
};

The change to EADD/EINIT ioctl's would be simply fget/kernel_read/fput sequence replacing copy_from_user().

/Jarkko

#230 lore raw
In reply to Jarkko Sakkinen
Quoted 10 lines · Jarkko Sakkinen
Quoted 9 lines · Sean Christopherson
Quoted 4 lines · Andy Lutomirski

I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?)

Nit: SECINFO is just the struct passed to EADD, I think what you're really asking is "can the EPCM permissions be changed after the fact".

Quoted 6 lines
Quoted 4 lines

And the answer is, yes.

On SGX2 hardware, the enclave can extend the EPCM permissions at runtime via ENCLU[EMODPE], e.g. to make a page writable.

Small correction: it is EMODPR.

No, I'm referring to EMODPE, note the ENCLU classification.

Quoted 4 lines

Anyway, it is good to mention that these would require EACCEPT from the enclave side. In order to take advantage of this is in a malicous enclave, one would require SELinux/IMA/whatnot policy to have permitted it in the first place.

EMODPE doesn't require EACCEPT or any equivalent from the kernel. As you alluded to, the page tables would still need to allow PROT_EXEC. I was simply trying to answer Andy's question regarding SECINFO.

Quoted 2 lines

Thus, it cannot be said that it breaks the security policy if this would happen because policy has allowed to use the particular enclave.

#231 lore raw
Quoted 20 lines · Sean Christopherson
Quoted 10 lines · Jarkko Sakkinen
Quoted 9 lines · Sean Christopherson
Quoted 4 lines · Andy Lutomirski

I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?)

Nit: SECINFO is just the struct passed to EADD, I think what you're really asking is "can the EPCM permissions be changed after the fact".

Quoted 6 lines
Quoted 4 lines

And the answer is, yes.

On SGX2 hardware, the enclave can extend the EPCM permissions at runtime via ENCLU[EMODPE], e.g. to make a page writable.

Small correction: it is EMODPR.

No, I'm referring to EMODPE, note the ENCLU classification.

Quoted 8 lines
Quoted 4 lines

Anyway, it is good to mention that these would require EACCEPT from the enclave side. In order to take advantage of this is in a malicous enclave, one would require SELinux/IMA/whatnot policy to have permitted it in the first place.

EMODPE doesn't require EACCEPT or any equivalent from the kernel. As you alluded to, the page tables would still need to allow PROT_EXEC. I was simply trying to answer Andy's question regarding SECINFO.

Ah, have to admit that I had totally forgot EMODPE :-) Have not had to deal with that opcode that much.

/Jarkko

#232 lore raw
In reply to Andy Lutomirski
Quoted 16 lines · Andy Lutomirski
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines · Jarkko Sakkinen

I did study through SDK's file format and realized that it does not does make sense after all to embed one.

Quoted 8 lines
Quoted 5 lines

To implement it properly you would probably need a new syscall (lets say sgx_load_enclave) and also that enclaves are not just executables binaries. It is hard to find a generic format for them as applications range from simply protecting part of an application to running a containter inside enclave.

I'm still puzzling what kind of changes you were discussing considering SGX_IOC_ENCLAVE_ADD_PAGE.

Quoted 7 lines

I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?) Sadly, we don't really have the a nice in-kernel API for that right now. You could do down_read(mmap_sem) and find_vma(). Arguably there is no value to checking that PKRU allows execute to the data.

OK, so you would actually go on to check whether the VMA where the data is copied contains executable data?

What if SECINFO.X is not set and you EADD to region that is executable in the enclave VMA? E.g. have RWX VMA for the enclave just to give a simple example. Then you could carry executable code in the data sections of the binary.

/Jarkko

#233 lore raw
Quoted 25 lines · Jarkko Sakkinen
Quoted 16 lines · Andy Lutomirski
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines · Jarkko Sakkinen

I did study through SDK's file format and realized that it does not does make sense after all to embed one.

Quoted 8 lines
Quoted 5 lines

To implement it properly you would probably need a new syscall (lets say sgx_load_enclave) and also that enclaves are not just executables binaries. It is hard to find a generic format for them as applications range from simply protecting part of an application to running a containter inside enclave.

I'm still puzzling what kind of changes you were discussing considering SGX_IOC_ENCLAVE_ADD_PAGE.

Quoted 7 lines

I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?) Sadly, we don't really have the a nice in-kernel API for that right now. You could do down_read(mmap_sem) and find_vma(). Arguably there is no value to checking that PKRU allows execute to the data.

Quoted 7 lines

OK, so you would actually go on to check whether the VMA where the data is copied contains executable data?

What if SECINFO.X is not set and you EADD to region that is executable in the enclave VMA? E.g. have RWX VMA for the enclave just to give a simple example. Then you could carry executable code in the data sections of the binary.

This would require to be done after the enclave is initialized of course with EMODPR, which requires the enclave to accept the permission change with EACCEPT, which limits somewhat. This means that the static enclave would be fully covered.

/Jarkko

#234 lore raw
In reply to Andy Lutomirski
Quoted 24 lines · Andy Lutomirski
Quoted 14 lines · Jethro Beekman
Quoted 3 lines · Cedric

Hi Jethro,

Quoted 1 line

ELF files are explicitly designed such that you can map them (with

mmap)

Quoted 2 lines
Quoted 2 lines

in 4096-byte chunks. However, sometimes there's overlap and you will sometimes see that a particular offset is mapped twice because the

first

Quoted 1 line
Quoted 1 line

half of the page in the file belongs to an RX range and the second

half

Quoted 3 lines
Quoted 2 lines

to an R-only range. Also, ELF files don't (normally) describe stack, heap, etc. which you do need for enclaves.

Quoted 8 lines
Quoted 1 line

You have probably misread my email. By mmap(), I meant the enclave

file would be mapped via *multiple* mmap() calls, in the same way as what dlopen() would do in loading regular shared object. The intention here is to make the enclave file subject to the same checks as regular shared objects.

No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's on-disk representation.

Quoted 6 lines

For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.

Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.

By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant. We are loading enclave files as RO and copying those into EPC. An IMA policy can enforce RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.

And sgx enclaves are always signed and integrity protected and verified at the time of EINIT. So if EINIT passes, we know the content loaded (including permission flags) is matching the sigstruct. But sigstruct/signature is part of the file, should be accounted for in IMA measurement of the whole file, so it is also verified by IMA during file open, right?

The only potential gap/difference comparing to regular ELF executable or DSOs:for enclaves, we need mmap portions of enclave linear range with RW to do EADD IOC, then mprotect those pages to RX after EINIT. But this is operated on enclave fd provided by driver. So we can have an SELinux policy say: only this type of processes is allowed to open enclave fd, and allowed to do mmap/mprotect with read, write, execute on it. Wouldn't that be enough?

Thanks
Haitao

maintainer
#235 lore raw
Quoted 39 lines · Haitao Huang
Quoted 28 lines · Andy Lutomirski
Quoted 20 lines · Jethro Beekman
Quoted 3 lines · Cedric

Hi Jethro,

Quoted 1 line

ELF files are explicitly designed such that you can map them (with

mmap)

Quoted 2 lines
Quoted 2 lines

in 4096-byte chunks. However, sometimes there's overlap and you will sometimes see that a particular offset is mapped twice because the

first

Quoted 1 line
Quoted 1 line

half of the page in the file belongs to an RX range and the second

half

Quoted 4 lines
Quoted 2 lines

to an R-only range. Also, ELF files don't (normally) describe stack, heap, etc. which you do need for enclaves.

You have probably misread my email. By mmap(), I meant the enclave

file would be mapped via *multiple* mmap() calls, in the same way as what dlopen() would do in loading regular shared object. The intention here is to make the enclave file subject to the same checks as regular shared objects.

Quoted 5 lines

No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's on-disk representation.

Quoted 6 lines

For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.

Quoted 7 lines

Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.

By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant.

The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:

Quoted 2 lines

We are loading
enclave files as RO and copying those into EPC.

Which is, semantically, a lot like loading a normal file as RO and then mprotecting() it to RX, which is disallowed under quite a few LSM policies.

Quoted 3 lines

An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.

If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.

Quoted 7 lines

And sgx enclaves are always signed and integrity protected and verified at the time of EINIT. So if EINIT passes, we know the content loaded (including permission flags) is matching the sigstruct. But sigstruct/signature is part of the file, should be accounted for in IMA measurement of the whole file, so it is also verified by IMA during file open, right?

This does work, but only if the kernel parses that file so that the kernel can trust that the enclave data actually came from the file as intended. And moving the parsing to the kernel seems like a mess that no one really wants to do.

#236 lore raw
Quoted 64 lines · Andy Lutomirski
Quoted 44 lines · Haitao Huang
Quoted 9 lines · Andy Lutomirski
Quoted 7 lines · Jethro Beekman
Quoted 3 lines · Cedric

Hi Jethro,

Quoted 1 line

ELF files are explicitly designed such that you can map them (with

mmap)

Quoted 1 line
Quoted 1 line

in 4096-byte chunks. However, sometimes there's overlap and you

will

Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line

sometimes see that a particular offset is mapped twice because the

first

Quoted 1 line
Quoted 1 line

half of the page in the file belongs to an RX range and the second

half

Quoted 1 line
Quoted 1 line

to an R-only range. Also, ELF files don't (normally) describe

stack,

Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

heap, etc. which you do need for enclaves.

You have probably misread my email. By mmap(), I meant the enclave

file would be mapped via *multiple* mmap() calls, in the same way as what dlopen() would do in loading regular shared object. The

intention

Quoted 1 line
Quoted 1 line

here is to make the enclave file subject to the same checks as

regular

Quoted 5 lines
Quoted 5 lines

shared objects.

No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's

on-disk

Quoted 9 lines
Quoted 2 lines

representation.

For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.

Quoted 7 lines

Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.

By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant.

The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:

Quoted 2 lines

We are loading
enclave files as RO and copying those into EPC.

Which is, semantically, a lot like loading a normal file as RO and then mprotecting() it to RX, which is disallowed under quite a few LSM policies.

Quoted 7 lines
Quoted 3 lines

An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.

If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.

But in this case, file itself is mapped as RO treated like data and it is not for execution. SGX enclave pages have EPCM enforced permissions. So from SELinux point of view I would think it can treat it as RO and that's fine.

Quoted 8 lines
Quoted 7 lines

And sgx enclaves are always signed and integrity protected and verified at the time of EINIT. So if EINIT passes, we know the content loaded (including permission flags) is matching the sigstruct. But sigstruct/signature is part of the file, should be accounted for in IMA measurement of the whole file, so it is also verified by IMA during file open, right?

Quoted 4 lines

This does work, but only if the kernel parses that file so that the kernel can trust that the enclave data actually came from the file as intended. And moving the parsing to the kernel seems like a mess that no one really wants to do.

If kernel only needs to know the source bytes are from a file that passed IMA integrity and SELinux RO enforcement, then it can just check if the source pointer belongs to a VMA with valid fd and no parsing or checking permissions needed. Understood if you want to make enclave file code segment stick to the RX semantics mentioned above, then this doesn't qualify.

Thanks

Haitao

maintainer
#237 lore raw
Quoted 67 lines · Haitao Huang
Quoted 66 lines · Andy Lutomirski
Quoted 39 lines · Haitao Huang
Quoted 28 lines · Andy Lutomirski
Quoted 20 lines · Jethro Beekman
Quoted 3 lines · Cedric

Hi Jethro,

Quoted 1 line

ELF files are explicitly designed such that you can map them (with

mmap)

Quoted 2 lines
Quoted 2 lines

in 4096-byte chunks. However, sometimes there's overlap and you will sometimes see that a particular offset is mapped twice because the

first

Quoted 1 line
Quoted 1 line

half of the page in the file belongs to an RX range and the second

half

Quoted 4 lines
Quoted 2 lines

to an R-only range. Also, ELF files don't (normally) describe stack, heap, etc. which you do need for enclaves.

You have probably misread my email. By mmap(), I meant the enclave

file would be mapped via *multiple* mmap() calls, in the same way as what dlopen() would do in loading regular shared object. The intention here is to make the enclave file subject to the same checks as regular shared objects.

Quoted 5 lines

No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's on-disk representation.

Quoted 6 lines

For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.

Quoted 7 lines

Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.

By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant.

The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:

Quoted 2 lines

We are loading
enclave files as RO and copying those into EPC.

Which is, semantically, a lot like loading a normal file as RO and then mprotecting() it to RX, which is disallowed under quite a few LSM policies.

Quoted 3 lines

An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.

Quoted 5 lines
Quoted 3 lines

If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.

But in this case, file itself is mapped as RO treated like data and it is not for execution. SGX enclave pages have EPCM enforced permissions. So from SELinux point of view I would think it can treat it as RO and that's fine.

As an example, SELinux has an “execute” permission (via security_mmap_file — see file_map_prot_check()) that controls whether you can execute code from that file. If you lack this permission on a file, you may still be able to map it PROT_READ, but you may not map it PROT_EXEC. Similarly, if you want to malloc() some memory, write *code* to it, and execute it, you need a specific permission.

So, unless we somehow think it’s okay for SGX to break the existing model, we need to respect these restrictions in the SGX driver. In other words, we either need to respect execmem, etc or require PROT_EXEC or the equivalent. I like the latter a lot more.

#238 lore raw
Quoted 92 lines · Andy Lutomirski
Quoted 79 lines · Haitao Huang
Quoted 78 lines · Andy Lutomirski
Quoted 51 lines · Haitao Huang
Quoted 1 line · Andy Lutomirski

On Fri, May 10, 2019 at 12:04 PM Jethro Beekman

<[email protected]>

Quoted 6 lines
Quoted 5 lines · wrote:
Quoted 3 lines · Cedric

Hi Jethro,

Quoted 1 line

ELF files are explicitly designed such that you can map them

(with

Quoted 2 lines
Quoted 2 lines

mmap)

Quoted 1 line
Quoted 1 line

in 4096-byte chunks. However, sometimes there's overlap and you

will

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

sometimes see that a particular offset is mapped twice because

the

Quoted 2 lines
Quoted 2 lines

first

Quoted 1 line
Quoted 1 line

half of the page in the file belongs to an RX range and the

second

Quoted 2 lines
Quoted 2 lines

half

Quoted 1 line
Quoted 1 line

to an R-only range. Also, ELF files don't (normally) describe

stack,

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

heap, etc. which you do need for enclaves.

You have probably misread my email. By mmap(), I meant the

enclave

Quoted 1 line
Quoted 1 line

file would be mapped via *multiple* mmap() calls, in the same way

as

Quoted 1 line
Quoted 1 line

what dlopen() would do in loading regular shared object. The

intention

Quoted 1 line
Quoted 1 line

here is to make the enclave file subject to the same checks as

regular

Quoted 5 lines
Quoted 5 lines

shared objects.

No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's

on-disk

Quoted 4 lines
Quoted 2 lines

representation.

For a tiny bit of background, Linux (AFAIK*) makes no effort to

ensure

Quoted 5 lines

the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.

Quoted 7 lines

Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.

By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant.

The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:

Quoted 2 lines

We are loading
enclave files as RO and copying those into EPC.

Which is, semantically, a lot like loading a normal file as RO and then mprotecting() it to RX, which is disallowed under quite a few LSM policies.

Quoted 3 lines

An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.

Quoted 5 lines
Quoted 3 lines

If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.

But in this case, file itself is mapped as RO treated like data and it is not for execution. SGX enclave pages have EPCM enforced permissions. So from SELinux point of view I would think it can treat it as RO and that's fine.

As an example, SELinux has an “execute” permission (via security_mmap_file — see file_map_prot_check()) that controls whether you can execute code from that file. If you lack this permission on a file, you may still be able to map it PROT_READ, but you may not map it PROT_EXEC. Similarly, if you want to malloc() some memory, write *code* to it, and execute it, you need a specific permission.

Quoted 4 lines

So, unless we somehow think it’s okay for SGX to break the existing model, we need to respect these restrictions in the SGX driver. In other words, we either need to respect execmem, etc or require PROT_EXEC or the equivalent. I like the latter a lot more.

What puzzles me is that this restriction does not add real value to security.
When enclave files are mapped with PROT_READ, without SE execute permission. No breakage to LSM model in normal process address space as no one can execute code directly from the file in normal memory. When enclave is built and loaded into EPC by EADDs, if the SIGSTRUCT is trusted (either signer or MRENCLAVE), EINIT will guarantee security (both integrity and permissions). LSM may not like the fact the a piece of code got loaded into EPC page without specifically giving SE execute permission. However, LSM can be used to control what SIGSTRUCTs can be trusted as you suggested and indirectly enforce what code got executed inside EPC.

So to me, only SIGSTRUCT verification would add value.

#239 lore raw
In reply to Andy Lutomirski

Hi Everyone,

I think we are talking about 2 different kinds of criteria for determining the sanity of an enclave.

The first kind determines an enclave's sanity by generally accepted good practices. For example, no executable pages shall ever be writable.

The second kind determines an enclave's sanity by "who stands behind it", such as whether the file containing SIGSTRUCT has the proper SELinux label/type, or whether the signing key is trusted.

I'd say those 2 kinds of criteria should be orthogonal because they don't get into each other's way and it could also be beneficial to enable both at the same time. For example, a user may want to allow launching enclaves signed by himself/herself only, however, as a human being he/she may make mistakes so would also like to ensure no RWX pages even for those enclaves explicitly authorized.

I think those 2 kinds of criteria could be abstracted as 2 new LSM hooks - security_sgx_add_pages() and security_sgx_initialize_enclave().

security_sgx_add_pages() is invoked by SGX driver to determine if a range of source pages are allowed to be EADD'ed with the requested EPCM attributes, and by default it returns 0 (allowed) iff the requested EPCM attributes are a subset of the permissions of the VMA covering that range of source pages. An (SGX-aware) LSM module/policy could employ different criteria, such as making sure the source pages are backed by an enclave file (using SELinux label, for example).

security_sgx_initialize_enclave() is invoked by SGX driver to determine if a given SIGSTRUCT is valid (hence allowed to be EINIT'ed), and it always returns 0 (allowed) by default. An LSM implementation may enforce custom policies, such as whether the signing public key is trusted by the current user, or whether it was backed (mmap()'ed) by an authorized file (e.g. of an expected type in the case of SELinux, or located in a particular directory in the case of AppArmor).

With regard to SGX2/EDMM (Enclave Dynamic Memory Management) support, RW->RX transitions are inevitable to support certain usages such as dynamic loading/linking, meaning those usages may be blocked by existing policies. But from security perspective, I think they should be allowed by default (i.e. for non-SGX-aware LSM modules/policies) because such permission changes are inherent behaviors of the enclave itself, which is considered "sane" after passing all the checks performed in security_sgx_add_pages()/security_sgx_initialize_enclave(). Of course, an SGX-aware LSM module/policy shall be allowed to override. How about adding a new security_sgx_mprotect() LSM hook?

Quoted 95 lines

From: Andy Lutomirski [mailto:[email protected]]

Quoted 1 line

On May 14, 2019, at 8:30 AM, Haitao Huang

Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines

On Tue, 14 May 2019 10:17:29 -0500, Andy Lutomirski <[email protected]>

Quoted 49 lines · wrote:
Quoted 49 lines
Quoted 46 lines · Haitao Huang
Quoted 40 lines · Andy Lutomirski
Quoted 24 lines · Jethro Beekman
Quoted 4 lines · Cedric

Hi Jethro,

Quoted 2 lines

ELF files are explicitly designed such that you can map them (with

mmap)

Quoted 3 lines
Quoted 3 lines

in 4096-byte chunks. However, sometimes there's overlap and you will sometimes see that a particular offset is mapped twice because the

first

Quoted 2 lines
Quoted 2 lines

half of the page in the file belongs to an RX range and the second

half

Quoted 5 lines
Quoted 2 lines

to an R-only range. Also, ELF files don't (normally) describe stack, heap, etc. which you do need for enclaves.

You have probably misread my email. By mmap(), I meant the enclave

file would be mapped via *multiple* mmap() calls, in the same way as what dlopen() would do in loading regular shared object. The intention here is to make the enclave file subject to the same checks as regular shared objects.

Quoted 5 lines

No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's on-disk representation.

For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so
configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.

By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux

compliant.

Quoted 15 lines
Quoted 15 lines

The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:

Quoted 2 lines

We are loading
enclave files as RO and copying those into EPC.

Which is, semantically, a lot like loading a normal file as RO and then mprotecting() it to RX, which is disallowed under quite a few LSM policies.

Quoted 3 lines

An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.

Quoted 5 lines
Quoted 3 lines

If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.

But in this case, file itself is mapped as RO treated like data and it

is not for execution. SGX enclave pages have EPCM enforced permissions. So from SELinux point of view I would think it can treat it as RO and that's fine.

As an example, SELinux has an “execute” permission (via security_mmap_file — see file_map_prot_check()) that controls whether you can execute code from that file. If you lack this permission on a file, you may still be able to map it PROT_READ, but you may not map it PROT_EXEC. Similarly, if you want to malloc() some memory, write *code* to it, and execute it, you need a specific permission.

Quoted 4 lines

So, unless we somehow think it’s okay for SGX to break the existing model, we need to respect these restrictions in the SGX driver. In other words, we either need to respect execmem, etc or require PROT_EXEC or the equivalent. I like the latter a lot more.

-Cedric

#240 lore raw
Quoted 6 lines · Cedric

Hi Everyone,

I think we are talking about 2 different kinds of criteria for determining the sanity of an enclave.

The first kind determines an enclave's sanity by generally accepted good practices. For example, no executable pages shall ever be writable.

We'll have to trust user space doing mmap with right permissions as SELinux does not enforce which segment to be RW and which to be RX. The file needs to have SELinux EXECUTE and WRITE both, if we need map some segments with RW and others with RX.

We could say EINIT would ensure user is doing the right thing because it would fail if user map permission wrongly. Then the extra mmaps are redundant of doing SIGSTRUCT verification.

Additionally, per Sean's comments, after EADD in current implementation, we will still need PROCESS_EXECMEM for mprotect on enclave fd to change some EPC pages PTE to RX before enclave can execute. So I don't think mmap the source enclave file would gain anything in addition to what your proposed security_sgx_initialize_enclave() does.

Since security_sgx_initialize_enclave() is a lot like launch control policy enforcement we discussed a lot and resolved, I tend to agree with Andy's assessment we can just do nothing for the initial merge and add hooks needed if someone wants them. And the initial merge would require the enclave hosting processes ask for PROCESS_EXECMEM permission to do mmap/mprotect with enclave fd.

#241 lore raw
In reply to Dave Hansen

Hi Dave,

Quoted 7 lines · Jethro Beekman
Quoted 2 lines

It does assume a specific format, namely, that the memory layout (including page types/permissions) of the enclave can be represented

in

Quoted 2 lines

a "flat file" on disk, or at least that the enclave memory contents consist of 4096-byte chunks in that file.

Quoted 8 lines

I _think_ Cedric's point is that, to the kernel, /lib/x86_64-linux-gnu/libc.so.6 is a "flat file" because the kernel doesn't have any part in parsing the executable format of a shared library.

I actually don't know how it works, though. Do we just just trust that the userspace parsing of the .so format is correct? Do we just assume that any part of a file passing IMA checks can be PROT_EXEC?

The key idea here is for enclave files to "inherit" the checks applicable to regular shared objects. And how we are going to do it is for user process to map every loadable segment of the enclave file into its address space using *multiple* mmap() calls, just in the same way as dlopen() would do for loading regular shared objects. Then those open()/mmap() syscalls will trigger all applicable checks (by means of security_file_open(), security_mmap_file() and ima_file_mmap() hooks) transparently. That said, IMA/LSM implementations/policies will dictate the success/failure of those open()/mmap() syscalls. And by depending EPCM attributes on permissions of source VMAs, no EXEC page could be ever created unless the source page (which has to be EXEC, btw) has passed IMA/LSM checks (as if it were loaded from a regular shared object file).

-Cedric

#242 lore raw
In reply to Jethro Beekman
Quoted 11 lines · Jethro Beekman
Quoted 10 lines · Andy Lutomirski

With plain mmap() + mprotect(), the LSM will prevent you from making memory that *was* writable executable. This is by design and SELinux supports it. I don’t remember the name of the associated SELinux permission off the top of my head.

If we start enforcing equivalent rules on SGX, then the current API will simply not allow enclaves to be loaded — no matter how you slice it, loading an enclave with the current API is indistinguishable from making arbitrary data executable.

Quoted 4 lines

Yes this is exactly what I intended here: a very simple change that stops SGX from confusing LSM. Just by enforcing that everything that looks like a memory write (EADD, EAUG, EDBGWR, etc.) actually requires write permissions, reality and LSM should be on the same page.

And how so? You create writeable AND executable memory. That's a nono and you can argue in circles, that's not going to change with any of your proposed changes. Andy clearly made a proposal which solves it in a proper way.

Thanks,

tglx

#243 lore raw
In reply to Jethro Beekman
Quoted 18 lines · Jethro Beekman
Quoted 17 lines · Thomas Gleixner
Quoted 9 lines · Jethro Beekman
Quoted 5 lines · Andy Lutomirski

There are many,
many Linux systems that enforce a policy that *all* executable text needs to come from a verified source. On these systems, you can't mmap some writable memory, write to it, and then change it to executable.

How is this implemented on those systems? AFAIK there's no kernel config option that changes the semantics of mmap as you describe.

That has nothing to do with mmap() semantics. You mmap() writeable memory and then you change the permissions via mprotect(). mprotect() calls into LSM and depending on policy and security model this will reject the request.

Quoted 6 lines
Quoted 2 lines

Andy was pointing out that the SGX ioctl bypasses the LSM mechanics which is obviously a bad thing.

We could modify the driver such that when you call ioctl EADD, the page table permissions need to be the PAGEINFO.SECINFO.FLAGS | PROT_WRITE, otherwise you get EPERM or so. After EADD, if you want, you can restrict

Actually, I don't think you even need to include PAGEINFO.SECINFO.FLAGS, you just need to ensure PROT_WRITE. Regular page table checks take care of PAGEINFO.SECINFO.FLAGS.

--
Jethro Beekman | Fortanix

#244 lore raw
In reply to Dr. Greg
Quoted 3 lines · Dr. Greg

3.) The approach we propose addresses these issues while imposing no functional limitations on how Linux platform owners can use enclave technology.

Let's say v20 gets merged. Is there any reason your approach can not be rebased on top of that code, reviewed and merged later?

#245 lore raw

cover An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

In reply to Jarkko Sakkinen

The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.

This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new __vdso_sgx_enter_enclave() will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null), __vdso_sgx_enter_enclave() will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).

The SGX selftest is augmented to test out the new callback interface, and to serve as a simple example to showcase how to use the callback interface in practice.

Cedric Xing (3):
selftests/x86: Fixed Makefile for SGX selftest
x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing
on untrusted stack
selftests/x86: Augment SGX selftest to test new
__vdso_sgx_enter_enclave() and its callback interface

 arch/x86/entry/vdso/vsgx_enter_enclave.S   | 156 ++++++++++++---------
 arch/x86/include/uapi/asm/sgx.h            |  14 +-
 tools/testing/selftests/x86/Makefile       |  12 +-
 tools/testing/selftests/x86/sgx/Makefile   |  45 +++---
 tools/testing/selftests/x86/sgx/main.c     | 123 +++++++++++++---
 tools/testing/selftests/x86/sgx/sgx_call.S |  40 +++++-
 6 files changed, 264 insertions(+), 126 deletions(-)
Signature · 1 line
2.17.1
#246 lore raw

RFC PATCH 1/3 selftests/x86: Fixed Makefile for SGX selftest

The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.

---
 tools/testing/selftests/x86/Makefile     | 12 +++----
 tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++---------------
 2 files changed, 22 insertions(+), 35 deletions(-)
tools/testing/selftests/x86/Makefile +6 -6
index 4fc9a42f56ea..1294c5f5b6ca 100644
@@ -70,11 +70,11 @@ all_32: $(BINARIES_32)
 
 all_64: $(BINARIES_64)
 
-all_64: $(SUBDIRS_64)
-	@for DIR in $(SUBDIRS_64); do			\
-		BUILD_TARGET=$(OUTPUT)/$$DIR;		\
-		mkdir $$BUILD_TARGET  -p;		\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+all_64: | $(SUBDIRS_64)
+	@for DIR in $|; do					\
+		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
+		mkdir $$BUILD_TARGET  -p;			\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
 	done
 
 EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)
@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01)
 all: warn_32bit_failure
 
 warn_32bit_failure:
-	@echo "Warning: you seem to have a broken 32-bit build" 2>&1; 	\
+	@echo "Warning: you seem to have a broken 32-bit build" 2>&1;	\
 	echo "environment.  This will reduce test coverage of 64-bit" 2>&1; \
 	echo "kernels.  If you are using a Debian-like distribution," 2>&1; \
 	echo "try:"; 2>&1; \
tools/testing/selftests/x86/sgx/Makefile +16 -29

63 lines not on this page · raw message

Signature · 1 line
2.17.1
#247 lore raw

RFC PATCH 2/3 x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

In reply to Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++++---------
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 100 insertions(+), 70 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +93 -63
index fe0bf6671d6d..210f4366374a 100644
@@ -14,88 +14,118 @@
 .code64
 .section .text, "ax"
 
-#ifdef SGX_KERNEL_DOC
 /**
  * __vdso_sgx_enter_enclave() - Enter an SGX enclave
  *
  * @leaf:	**IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:	**IN \%rbx** - TCS, must be non-NULL
- * @ex_info:	**IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer
+ * @tcs:	**IN 0x08(\%rsp)** - TCS, must be non-NULL
+ * @ex_info:	**IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo'
+ *				     pointer
+ * @callback:	**IN 0x18(\%rsp)** - Optional callback function to be called on
+ *				     enclave exit or exception
  *
  * Return:
  *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is
- *  not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults
+ *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not
+ *  allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value
+ *  returned from ``callback`` (if one is supplied).
  *
  * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant with the
- * x86-64 ABI, i.e. cannot be called from standard C code.   As noted above,
- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``.  All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc...  Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * x86-64 ABI, i.e. cannot be called from standard C code. As noted above,
+ * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and
+ * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other registers
+ * will be passed through to the enclave as is. All registers except ``%rbp``
+ * must be treated as volatile from the caller's perspective, including but not
+ * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave being
+ * run **must** preserve the untrusted ``%rbp``.
+ *
+ * ``callback`` has the following signature:
+ * int callback(long rdi, long rsi, long rdx,
+ *		struct sgx_enclave_exinfo *ex_info, long r8, long r9,
+ *		void *tcs, long ursp);
+ * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``, ``%rbx``
+ * and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``, ``%rdx``,
+ * ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave exited/excepted,
+ * can be accessed directly as input parameters, while other GPRs can be
+ * accessed in assembly if needed.
+ * A positive value returned from ``callback`` will be treated as an ENCLU leaf
+ * (e.g. EENTER/ERESUME) to reenter the enclave, while 0 or a negative return
+ * value will be passed back to the caller of __vdso_sgx_enter_enclave().
+ * It is also **safe** to ``longjmp()`` out of ``callback``.
  */
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-			 struct sgx_enclave_exception *ex_info)
-{
-	if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-		return -EINVAL;
-
-	if (!tcs)
-		return -EINVAL;
-
-	try {
-		ENCLU[leaf];
-	} catch (exception) {
-		if (e)
-			*e = exception;
-		return -EFAULT;
-	}
-
-	return 0;
-}
-#endif
 ENTRY(__vdso_sgx_enter_enclave)
-	/* EENTER <= leaf <= ERESUME */
+	/* Prolog */
+	.cfi_startproc
+	push	%rbp
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbp, 0
+	mov	%rsp, %rbp
+	.cfi_def_cfa_register	%rbp
+
+1:	/* EENTER <= leaf <= ERESUME */
 	cmp	$0x2, %eax
-	jb	bad_input
-
+	jb	6f
 	cmp	$0x3, %eax
-	ja	bad_input
+	ja	6f
 
-	/* TCS must be non-NULL */
-	test	%rbx, %rbx
-	je	bad_input
+	/* Load TCS and AEP */
+	mov	0x10(%rbp), %rbx
+	lea	2f(%rip), %rcx
 
-	/* Save @exception_info */
-	push	%rcx
-
-	/* Load AEP for ENCLU */
-	lea	1f(%rip),  %rcx
-1:	enclu
-
-	add	$0x8, %rsp
-	xor	%eax, %eax
-	ret
-
-bad_input:
-	mov     $(-EINVAL), %rax
-	ret
-
-.pushsection .fixup, "ax"
-	/* Re-load @exception_info and fill it (if it's non-NULL) */
-2:	pop	%rcx
-	test    %rcx, %rcx
-	je      3f
+	/* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:	enclu
 
+	/* EEXIT path */
+	xor	%ebx, %ebx
+3:	mov	0x18(%rbp), %rcx
+	jrcxz	4f
 	mov	%eax, EX_LEAF(%rcx)
-	mov	%di,  EX_TRAPNR(%rcx)
-	mov	%si,  EX_ERROR_CODE(%rcx)
+	jnc	4f
+	mov	%di, EX_TRAPNR(%rcx)
+	mov	%si, EX_ERROR_CODE(%rcx)
 	mov	%rdx, EX_ADDRESS(%rcx)
-3:	mov	$(-EFAULT), %rax
+
+4:	/* Call *callback if supplied */
+	mov	0x20(%rbp), %rax
+	test	%rax, %rax
+	cmovz	%rbx, %rax
+	jz	7f
+	/* Align stack and clear RFLAGS.DF per x86_64 ABI */
+	mov	%rsp, %rbx
+	and	$-0x10, %rsp
+	cld
+	/* Parameters for *callback */
+	push	%rbx
+	push	0x10(%rbp)
+	/* Call via retpoline */
+	call	40f
+	/* Cleanup stack */
+	mov	%rbx, %rsp
+	jmp	1b
+40:	/* retpoline */
+	call	42f
+41:	pause
+	lfence
+	jmp	41b
+42:	mov	%rax, (%rsp)
+	ret
+
+5:	/* Exception path */
+	mov	$-EFAULT, %ebx
+	stc
+	jmp	3b
+
+6:	/* Unsupported ENCLU leaf */
+	cmp	$0, %eax
+	jle	7f
+	mov	$-EINVAL, %eax
+
+7:	/* Epilog */
+	leave
+	.cfi_def_cfa		%rsp, 8
 	ret
-.popsection
+	.cfi_endproc
 
-_ASM_VDSO_EXTABLE_HANDLE(1b, 2b)
+_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)
 
 ENDPROC(__vdso_sgx_enter_enclave)
arch/x86/include/uapi/asm/sgx.h +7 -7
index 9ed690a38c70..50d2b5143e5e 100644
@@ -24,7 +24,7 @@
 
 /**
  * struct sgx_enclave_create - parameter structure for the
- *                             %SGX_IOC_ENCLAVE_CREATE ioctl
+ *			       %SGX_IOC_ENCLAVE_CREATE ioctl
  * @src:	address for the SECS page data
  */
 struct sgx_enclave_create  {
@@ -33,7 +33,7 @@ struct sgx_enclave_create  {
 
 /**
  * struct sgx_enclave_add_page - parameter structure for the
- *                               %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
+ *				 %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
  * @addr:	address within the ELRANGE
  * @src:	address for the page data
  * @secinfo:	address for the SECINFO data
@@ -49,7 +49,7 @@ struct sgx_enclave_add_page {
 
 /**
  * struct sgx_enclave_init - parameter structure for the
- *                           %SGX_IOC_ENCLAVE_INIT ioctl
+ *			     %SGX_IOC_ENCLAVE_INIT ioctl
  * @sigstruct:	address for the SIGSTRUCT data
  */
 struct sgx_enclave_init {
@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute {
 };
 
 /**
- * struct sgx_enclave_exception - structure to report exceptions encountered in
- *				  __vdso_sgx_enter_enclave()
+ * struct sgx_enclave_exinfo - structure to report exceptions encountered in
+ *			       __vdso_sgx_enter_enclave()
  *
- * @leaf:	ENCLU leaf from \%eax at time of exception
+ * @leaf:	ENCLU leaf from \%eax at time of exception/exit
  * @trapnr:	exception trap number, a.k.a. fault vector
  * @error_code:	exception error code
  * @address:	exception address, e.g. CR2 on a #PF
  * @reserved:	reserved for future use
  */
-struct sgx_enclave_exception {
+struct sgx_enclave_exinfo {
 	__u32 leaf;
 	__u16 trapnr;
 	__u16 error_code;
Signature · 1 line
2.17.1
#248 lore raw

Re: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 4 lines · Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).

Quoted 10 lines

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.

Please resend this series with the new code as a separate function, not a modification to the existing function. Andy made it pretty clear that he prefers a separate vDSO function, and it'll be a lot easier to provide feedback on the code if we're working from a clean slate. If you want to push to get everything lumped into a single function then by all means do so on LKML, but for the RFC let's keep the focus on the code itself and simplify life for reviewers.

Quoted 5 lines · Andy Lutomirski

If the answer to both questions is yes, then it seems like it could be reasonable to support it in the vDSO. But I still think it should probably be a different vDSO entry point so that the normal case doesn't become more complicated.

Quoted 15 lines

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S. * 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits. * __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation). * __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation). * The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

Again, wrap at 75 chars or earlier. And incorporate checkpatch into your git send-email flow if you haven't done so already, e.g. checkpatch should have warned about lines longer than 75 chars.

Quoted 2 lines

---

maintainer
#249 lore raw

Re: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

In reply to Cedric Xing
Quoted 85 lines · Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left
off by the enclave as a parameter to the callback.

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++++---------
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 100 insertions(+), 70 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +17 -11
index fe0bf6671d6d..210f4366374a 100644
@@ -14,88 +14,118 @@
 .code64
 .section .text, "ax"
 
-#ifdef SGX_KERNEL_DOC
 /**
  * __vdso_sgx_enter_enclave() - Enter an SGX enclave
  *
  * @leaf:      **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:       **IN \%rbx** - TCS, must be non-NULL
- * @ex_info:   **IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer
+ * @tcs:       **IN 0x08(\%rsp)** - TCS, must be non-NULL
+ * @ex_info:   **IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo'
+ *                                  pointer
+ * @callback:  **IN 0x18(\%rsp)** - Optional callback function to be called on
+ *                                  enclave exit or exception
  *
  * Return:
  *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is
- *  not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults
+ *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not
+ *  allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value
+ *  returned from ``callback`` (if one is supplied).
  *
  * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant with the
- * x86-64 ABI, i.e. cannot be called from standard C code.   As noted above,
- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``.  All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc...  Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * x86-64 ABI, i.e. cannot be called from standard C code. As noted above,
+ * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and
+ * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other registers
+ * will be passed through to the enclave as is. All registers except ``%rbp``
+ * must be treated as volatile from the caller's perspective, including but not
+ * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave being
+ * run **must** preserve the untrusted ``%rbp``.
+ *
+ * ``callback`` has the following signature:

arch/x86/entry/vdso/vsgx_enter_enclave.S line 52

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +10 -0
+ * int callback(long rdi, long rsi, long rdx,
+ *             struct sgx_enclave_exinfo *ex_info, long r8, long r9,
+ *             void *tcs, long ursp);
+ * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``, ``%rbx``
+ * and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``, ``%rdx``,
+ * ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave exited/excepted,
+ * can be accessed directly as input parameters, while other GPRs can be
+ * accessed in assembly if needed.
+ * A positive value returned from ``callback`` will be treated as an ENCLU leaf
+ * (e.g. EENTER/ERESUME) to reenter the enclave, while 0 or a negative return

"to reenter the enclave without popping the extra data pushed by the enclave off the stack" or similar. We really don't want a situation where someone puts all there "keep on going" logic in the callback and the stack usage grows without bound.

arch/x86/entry/vdso/vsgx_enter_enclave.S line 54

Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2 -0
+ * value will be passed back to the caller of __vdso_sgx_enter_enclave().
+ * It is also **safe** to ``longjmp()`` out of ``callback``.

I'm not sure that "safe" needs emphasis.

Quoted 76 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +20 -48
  */
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-                        struct sgx_enclave_exception *ex_info)
-{
-       if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-               return -EINVAL;
-
-       if (!tcs)
-               return -EINVAL;
-
-       try {
-               ENCLU[leaf];
-       } catch (exception) {
-               if (e)
-                       *e = exception;
-               return -EFAULT;
-       }
-
-       return 0;
-}
-#endif
 ENTRY(__vdso_sgx_enter_enclave)
-       /* EENTER <= leaf <= ERESUME */
+       /* Prolog */
+       .cfi_startproc
+       push    %rbp
+       .cfi_adjust_cfa_offset  8
+       .cfi_rel_offset         %rbp, 0
+       mov     %rsp, %rbp
+       .cfi_def_cfa_register   %rbp
+
+1:     /* EENTER <= leaf <= ERESUME */
        cmp     $0x2, %eax
-       jb      bad_input
-
+       jb      6f
        cmp     $0x3, %eax
-       ja      bad_input
+       ja      6f
 
-       /* TCS must be non-NULL */
-       test    %rbx, %rbx
-       je      bad_input
+       /* Load TCS and AEP */
+       mov     0x10(%rbp), %rbx
+       lea     2f(%rip), %rcx
 
-       /* Save @exception_info */
-       push    %rcx
-
-       /* Load AEP for ENCLU */
-       lea     1f(%rip),  %rcx
-1:     enclu
-
-       add     $0x8, %rsp
-       xor     %eax, %eax
-       ret
-
-bad_input:
-       mov     $(-EINVAL), %rax
-       ret
-
-.pushsection .fixup, "ax"
-       /* Re-load @exception_info and fill it (if it's non-NULL) */
-2:     pop     %rcx
-       test    %rcx, %rcx
-       je      3f
+       /* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:     enclu
 
+       /* EEXIT path */
+       xor     %ebx, %ebx
+3:     mov     0x18(%rbp), %rcx
+       jrcxz   4f
        mov     %eax, EX_LEAF(%rcx)
-       mov     %di,  EX_TRAPNR(%rcx)

arch/x86/entry/vdso/vsgx_enter_enclave.S line 90

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +7 -2
-       mov     %si,  EX_ERROR_CODE(%rcx)
+       jnc     4f
+       mov     %di, EX_TRAPNR(%rcx)
+       mov     %si, EX_ERROR_CODE(%rcx)
        mov     %rdx, EX_ADDRESS(%rcx)
-3:     mov     $(-EFAULT), %rax
+
+4:     /* Call *callback if supplied */
+       mov     0x20(%rbp), %rax
+       test    %rax, %rax

Maybe have a comment like "At this point, the effective return value is in RBX. If there is no callback, then return it."

arch/x86/entry/vdso/vsgx_enter_enclave.S line 94

Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4 -0
+       cmovz   %rbx, %rax
+       jz      7f
+       /* Align stack and clear RFLAGS.DF per x86_64 ABI */
+       mov     %rsp, %rbx

Whoa, this is too subtle here. Can you update the comment to clarify that the uRSP value set by the enclave needs to be saved so that the enclave can be resumed if needed?

arch/x86/entry/vdso/vsgx_enter_enclave.S line 103

Quoted 9 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +9 -0
+       and     $-0x10, %rsp
+       cld
+       /* Parameters for *callback */
+       push    %rbx
+       push    0x10(%rbp)
+       /* Call via retpoline */
+       call    40f
+       /* Cleanup stack */
+       mov     %rbx, %rsp

To me, "Cleanup stack" makes me think that you're restoring the original RSP, but you're actually just undoing in the stack alignment. How about "Undo stack alignment"?

But I'm not seeing the code that causes a return value RAX <= 0 to just return.

Quoted 18 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +18 -0
+       jmp     1b
+40:    /* retpoline */
+       call    42f
+41:    pause
+       lfence
+       jmp     41b
+42:    mov     %rax, (%rsp)
+       ret
+
+5:     /* Exception path */
+       mov     $-EFAULT, %ebx
+       stc
+       jmp     3b
+
+6:     /* Unsupported ENCLU leaf */
+       cmp     $0, %eax
+       jle     7f
+       mov     $-EINVAL, %eax

arch/x86/entry/vdso/vsgx_enter_enclave.S line 129

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6 -2
+
+7:     /* Epilog */
+       leave
+       .cfi_def_cfa            %rsp, 8
        ret
-.popsection
+       .cfi_endproc
 
-_ASM_VDSO_EXTABLE_HANDLE(1b, 2b)
+_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)

--Andy

#250 lore raw

RE: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Hi Andy,

I sent out my RFC patch v2 last night, that has your suggestions incorporated, plus a new unwind test to single step through the vDSO API to test out the CFI directives. Hopefully it is able to address all of your concerns. It's worth noting that, given the current patch fixes up #DB and #BP at ENCLU, the unwind test cannot run to completion. I assume Sean will revise the fixup code soon.

Thanks!

-Cedric

Quoted 310 lines

-----Original Message-----
From: Andy Lutomirski [mailto:[email protected]]
Sent: Monday, April 22, 2019 6:26 PM
To: Xing, Cedric <[email protected]>
Cc: LKML <[email protected]>; X86 ML <[email protected]>; linux- [email protected]; Andrew Morton <[email protected]>; Hansen, Dave <[email protected]>; Christopherson, Sean J <[email protected]>; [email protected]; [email protected]; Ayoun, Serge <[email protected]>; Katz-zamir, Shay <[email protected]>; Huang, Haitao <[email protected]>; Andy Shevchenko <[email protected]>; Thomas Gleixner <[email protected]>; Svahn, Kai <[email protected]>; Borislav Petkov <[email protected]>; Josh Triplett <[email protected]>; Andrew Lutomirski <[email protected]>; Huang, Kai <[email protected]>; David Rientjes <[email protected]>; Jarkko Sakkinen <[email protected]>
Subject: Re: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 18 lines · Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave
eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.

Quoted 3 lines

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct

sgx_enclave_exinfo'

Quoted 1 line

because it is filled upon both AEX (i.e. exceptions) and normal

enclave

Quoted 2 lines

exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead

of %rsp in

Quoted 2 lines

the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback

function to

Quoted 2 lines

be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon

enclave exits

Quoted 2 lines

(same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at

enclave

Quoted 1 line

exit to address data "pushed" by the enclave. A positive value

returned by

Quoted 1 line

the callback will be treated as an ENCLU leaf for re-entering the

enclave,

Quoted 7 lines

while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++++--------

-

Quoted 9 lines
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 100 insertions(+), 70 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -0
index fe0bf6671d6d..210f4366374a 100644

arch/x86/entry/vdso/vsgx_enter_enclave.S line 23

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -3
@@ -14,88 +14,118 @@
 .code64
 .section .text, "ax"
 
-#ifdef SGX_KERNEL_DOC
 /**
  * __vdso_sgx_enter_enclave() - Enter an SGX enclave
  *
  * @leaf:      **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:       **IN \%rbx** - TCS, must be non-NULL
- * @ex_info:   **IN \%rcx** - Optional 'struct sgx_enclave_exception'

pointer

arch/x86/entry/vdso/vsgx_enter_enclave.S line 22

Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2 -0
+ * @tcs:       **IN 0x08(\%rsp)** - TCS, must be non-NULL
+ * @ex_info:   **IN 0x10(\%rsp)** - Optional 'struct

sgx_enclave_exinfo'

arch/x86/entry/vdso/vsgx_enter_enclave.S line 24

Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2 -0
+ *                                  pointer
+ * @callback:  **IN 0x18(\%rsp)** - Optional callback function to be

called on

Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -1
+ *                                  enclave exit or exception
  *
  * Return:
  *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU

leaf is
- * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave

Quoted 8 lines

faults
+ * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU
+ leaf is not
+ * allowed, %-EFAULT if ENCLU or the enclave faults, or a
+ non-positive value
+ * returned from ``callback`` (if one is supplied).
*
* **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant

with the

Quoted 1 line

- * x86-64 ABI, i.e. cannot be called from standard C code. As noted

above,

Quoted 32 lines

- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``. All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * x86-64 ABI, i.e. cannot be called from standard C code. As noted
+ above,
+ * input parameters must be passed via ``%eax``, ``8(%rsp)``,
+ ``0x10(%rsp)`` and
+ * ``0x18(%rsp)``, with the return value passed via ``%eax``. All
+ other registers
+ * will be passed through to the enclave as is. All registers except
+ ``%rbp``
+ * must be treated as volatile from the caller's perspective,
+ including but not
+ * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the
+ enclave being
+ * run **must** preserve the untrusted ``%rbp``.
+ *
+ * ``callback`` has the following signature:
+ * int callback(long rdi, long rsi, long rdx,
+ * struct sgx_enclave_exinfo *ex_info, long r8, long r9,
+ * void *tcs, long ursp);
+ * ``callback`` **shall** follow x86_64 ABI. All GPRs **except**
+ ``%rax``, ``%rbx``
+ * and ``rcx`` are passed through to ``callback``. ``%rdi``,
+ ``%rsi``, ``%rdx``,
+ * ``%r8``, ``%r9``, along with the value of ``%rsp`` when the

Quoted 8 lines

+ enclave exited/excepted,
+ * can be accessed directly as input parameters, while other GPRs can
+ be
+ * accessed in assembly if needed.
+ * A positive value returned from ``callback`` will be treated as an
+ ENCLU leaf
+ * (e.g. EENTER/ERESUME) to reenter the enclave, while 0 or a
+ negative return

"to reenter the enclave without popping the extra data pushed by the enclave off the stack" or similar. We really don't want a situation where someone puts all there "keep on going" logic in the callback and the stack usage grows without bound.

Quoted 1 line

+ * value will be passed back to the caller of

__vdso_sgx_enter_enclave().

Quoted 1 line

+ * It is also **safe** to ``longjmp()`` out of ``callback``.

I'm not sure that "safe" needs emphasis.

Quoted 76 lines

*/

arch/x86/entry/vdso/vsgx_enter_enclave.S +20 -48
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-                        struct sgx_enclave_exception *ex_info)
-{
-       if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-               return -EINVAL;
-
-       if (!tcs)
-               return -EINVAL;
-
-       try {
-               ENCLU[leaf];
-       } catch (exception) {
-               if (e)
-                       *e = exception;
-               return -EFAULT;
-       }
-
-       return 0;
-}
-#endif
 ENTRY(__vdso_sgx_enter_enclave)
-       /* EENTER <= leaf <= ERESUME */
+       /* Prolog */
+       .cfi_startproc
+       push    %rbp
+       .cfi_adjust_cfa_offset  8
+       .cfi_rel_offset         %rbp, 0
+       mov     %rsp, %rbp
+       .cfi_def_cfa_register   %rbp
+
+1:     /* EENTER <= leaf <= ERESUME */
        cmp     $0x2, %eax
-       jb      bad_input
-
+       jb      6f
        cmp     $0x3, %eax
-       ja      bad_input
+       ja      6f
 
-       /* TCS must be non-NULL */
-       test    %rbx, %rbx
-       je      bad_input
+       /* Load TCS and AEP */
+       mov     0x10(%rbp), %rbx
+       lea     2f(%rip), %rcx
 
-       /* Save @exception_info */
-       push    %rcx
-
-       /* Load AEP for ENCLU */
-       lea     1f(%rip),  %rcx
-1:     enclu
-
-       add     $0x8, %rsp
-       xor     %eax, %eax
-       ret
-
-bad_input:
-       mov     $(-EINVAL), %rax
-       ret
-
-.pushsection .fixup, "ax"
-       /* Re-load @exception_info and fill it (if it's non-NULL) */
-2:     pop     %rcx
-       test    %rcx, %rcx
-       je      3f
+       /* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:     enclu
 
+       /* EEXIT path */
+       xor     %ebx, %ebx
+3:     mov     0x18(%rbp), %rcx
+       jrcxz   4f
        mov     %eax, EX_LEAF(%rcx)
-       mov     %di,  EX_TRAPNR(%rcx)

arch/x86/entry/vdso/vsgx_enter_enclave.S line 90

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +7 -2
-       mov     %si,  EX_ERROR_CODE(%rcx)
+       jnc     4f
+       mov     %di, EX_TRAPNR(%rcx)
+       mov     %si, EX_ERROR_CODE(%rcx)
        mov     %rdx, EX_ADDRESS(%rcx)
-3:     mov     $(-EFAULT), %rax
+
+4:     /* Call *callback if supplied */
+       mov     0x20(%rbp), %rax
+       test    %rax, %rax

Maybe have a comment like "At this point, the effective return value is in RBX. If there is no callback, then return it."

arch/x86/entry/vdso/vsgx_enter_enclave.S line 94

Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4 -0
+       cmovz   %rbx, %rax
+       jz      7f
+       /* Align stack and clear RFLAGS.DF per x86_64 ABI */
+       mov     %rsp, %rbx

Whoa, this is too subtle here. Can you update the comment to clarify that the uRSP value set by the enclave needs to be saved so that the enclave can be resumed if needed?

arch/x86/entry/vdso/vsgx_enter_enclave.S line 103

Quoted 9 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +9 -0
+       and     $-0x10, %rsp
+       cld
+       /* Parameters for *callback */
+       push    %rbx
+       push    0x10(%rbp)
+       /* Call via retpoline */
+       call    40f
+       /* Cleanup stack */
+       mov     %rbx, %rsp

To me, "Cleanup stack" makes me think that you're restoring the original RSP, but you're actually just undoing in the stack alignment. How about "Undo stack alignment"?

But I'm not seeing the code that causes a return value RAX <= 0 to just return.

Quoted 18 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +18 -0
+       jmp     1b
+40:    /* retpoline */
+       call    42f
+41:    pause
+       lfence
+       jmp     41b
+42:    mov     %rax, (%rsp)
+       ret
+
+5:     /* Exception path */
+       mov     $-EFAULT, %ebx
+       stc
+       jmp     3b
+
+6:     /* Unsupported ENCLU leaf */
+       cmp     $0, %eax
+       jle     7f
+       mov     $-EINVAL, %eax

arch/x86/entry/vdso/vsgx_enter_enclave.S line 129

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6 -2
+
+7:     /* Epilog */
+       leave
+       .cfi_def_cfa            %rsp, 8
        ret
-.popsection
+       .cfi_endproc
 
-_ASM_VDSO_EXTABLE_HANDLE(1b, 2b)
+_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)

--Andy

#251 lore raw

Re: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

In reply to Cedric Xing
Quoted 44 lines · Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left
off by the enclave as a parameter to the callback.

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++++---------
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 100 insertions(+), 70 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -0
index fe0bf6671d6d..210f4366374a 100644

arch/x86/entry/vdso/vsgx_enter_enclave.S line 17

Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -1
@@ -14,88 +14,118 @@
 .code64
 .section .text, "ax"
 
-#ifdef SGX_KERNEL_DOC

This #ifdef and the pseudo-C code below has a functional purpose. From the original commit:

Note, the C-like pseudocode describing the assembly routine is wrapped in a non-existent macro instead of in a comment to trick kernel-doc into auto-parsing the documentation and function prototype. This is a double win as the pseudocode is intended to aid kernel developers, not userland enclave developers.

We don't need full pseudocode, but a C-like prototype is necessary to get the kernel-doc comment parsed correctly.

I'll try to look at the actual code later today.

Quoted 69 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +29 -30
 /**
  * __vdso_sgx_enter_enclave() - Enter an SGX enclave
  *
  * @leaf:	**IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:	**IN \%rbx** - TCS, must be non-NULL
- * @ex_info:	**IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer
+ * @tcs:	**IN 0x08(\%rsp)** - TCS, must be non-NULL
+ * @ex_info:	**IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo'
+ *				     pointer
+ * @callback:	**IN 0x18(\%rsp)** - Optional callback function to be called on
+ *				     enclave exit or exception
  *
  * Return:
  *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is
- *  not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults
+ *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not
+ *  allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value
+ *  returned from ``callback`` (if one is supplied).
  *
  * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant with the
- * x86-64 ABI, i.e. cannot be called from standard C code.   As noted above,
- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``.  All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc...  Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * x86-64 ABI, i.e. cannot be called from standard C code. As noted above,
+ * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and
+ * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other registers
+ * will be passed through to the enclave as is. All registers except ``%rbp``
+ * must be treated as volatile from the caller's perspective, including but not
+ * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave being
+ * run **must** preserve the untrusted ``%rbp``.
+ *
+ * ``callback`` has the following signature:
+ * int callback(long rdi, long rsi, long rdx,
+ *		struct sgx_enclave_exinfo *ex_info, long r8, long r9,
+ *		void *tcs, long ursp);
+ * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``, ``%rbx``
+ * and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``, ``%rdx``,
+ * ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave exited/excepted,
+ * can be accessed directly as input parameters, while other GPRs can be
+ * accessed in assembly if needed.
+ * A positive value returned from ``callback`` will be treated as an ENCLU leaf
+ * (e.g. EENTER/ERESUME) to reenter the enclave, while 0 or a negative return
+ * value will be passed back to the caller of __vdso_sgx_enter_enclave().
+ * It is also **safe** to ``longjmp()`` out of ``callback``.
  */
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-			 struct sgx_enclave_exception *ex_info)
-{
-	if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-		return -EINVAL;
-
-	if (!tcs)
-		return -EINVAL;
-
-	try {
-		ENCLU[leaf];
-	} catch (exception) {
-		if (e)
-			*e = exception;
-		return -EFAULT;
-	}
-
-	return 0;
-}
-#endif
#252 lore raw

Re: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 57 lines · Sean Christopherson
Quoted 49 lines · Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left
off by the enclave as a parameter to the callback.

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++++---------
arch/x86/include/uapi/asm/sgx.h | 14 +-

2 files changed, 100 insertions(+), 70 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -0
index fe0bf6671d6d..210f4366374a 100644
@@ -14,88 +14,118 @@

.code64
.section .text, "ax"

Quoted 1 line

-#ifdef SGX_KERNEL_DOC

This #ifdef and the pseudo-C code below has a functional purpose. From the original commit:

Quoted 8 lines

Note, the C-like pseudocode describing the assembly routine is wrapped in a non-existent macro instead of in a comment to trick kernel-doc into auto-parsing the documentation and function prototype. This is a double win as the pseudocode is intended to aid kernel developers, not userland enclave developers.

We don't need full pseudocode, but a C-like prototype is necessary to get the kernel-doc comment parsed correctly.

That should be explained in a comment :)

—Andy

#253 lore raw

RFC PATCH 3/3 selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

In reply to Cedric Xing

Given the changes to __vdso_sgx_enter_enclave(), the selftest is augmented to test the newly added callback interface. This addtional test marks the whole enclave range as PROT_READ, and calls mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes.

---
 tools/testing/selftests/x86/sgx/main.c     | 123 ++++++++++++++++++---
 tools/testing/selftests/x86/sgx/sgx_call.S |  40 ++++++-
 2 files changed, 142 insertions(+), 21 deletions(-)
tools/testing/selftests/x86/sgx/main.c +105 -18
index e2265f841fb0..234cfbad14a5 100644
@@ -9,6 +9,7 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/mman.h>
 #include <sys/stat.h>
@@ -18,6 +19,10 @@
 #include "../../../../../arch/x86/kernel/cpu/sgx/arch.h"
 #include "../../../../../arch/x86/include/uapi/asm/sgx.h"
 
+#define _Q(x)	__Q(x)
+#define __Q(x)	#x
+#define ERRLN	"Line " _Q(__LINE__)
+
 static const uint64_t MAGIC = 0x1122334455667788ULL;
 
 struct vdso_symtab {
@@ -138,7 +143,7 @@ static bool encl_create(int dev_fd, unsigned long bin_size,
 	base = mmap(NULL, secs->size, PROT_READ | PROT_WRITE | PROT_EXEC,
 		    MAP_SHARED, dev_fd, 0);
 	if (base == MAP_FAILED) {
-		perror("mmap");
+		perror(ERRLN);
 		return false;
 	}
 
@@ -224,24 +229,113 @@ static bool encl_load(struct sgx_secs *secs, unsigned long bin_size)
 	return false;
 }
 
-void sgx_call(void *rdi, void *rsi, void *tcs,
-	      struct sgx_enclave_exception *exception,
-	      void *eenter);
+int sgx_call(void *rdi, void *rsi, long rdx, void *rcx, void *r8, void *r9,
+	     void *tcs, struct sgx_enclave_exinfo *ei, void *cb, void *eenter);
+
+static void show_enclave_exinfo(const struct sgx_enclave_exinfo *exinfop,
+				const char *header)
+{
+	printf("%s: leaf:%d", header, exinfop->leaf);
+	if (exinfop->leaf != 4)
+		printf(" trap#:%d ec:%d addr:0x%llx\n", exinfop->trapnr,
+			exinfop->error_code, exinfop->address);
+	else printf("\n");
+}
+
+static void test1(void *eenter, struct sgx_secs *secs)
+{
+	uint64_t result = 0;
+	struct sgx_enclave_exinfo exinfo;
+
+	printf("[1] Entering the enclave without callback.\n");
+
+	printf("Input: 0x%lx\n Expect: Same as input\n", MAGIC);
+	sgx_call((void *)&MAGIC, &result, 0, NULL, NULL, NULL,
+		 (void *)secs->base, &exinfo, NULL, eenter);
+	if (result != MAGIC) {
+		fprintf(stderr, "0x%lx != 0x%lx\n", result, MAGIC);
+		exit(1);
+	}
+	printf(" Output: 0x%lx\n", result);
+
+	printf("Input: Null TCS\n Expect: #PF at EENTER\n");
+	sgx_call((void *)&MAGIC, &result, 0, NULL, NULL, NULL,
+		 NULL, &exinfo, NULL, eenter);
+	show_enclave_exinfo(&exinfo, " Exit");
+	if (exinfo.leaf != 2 /*EENTER*/ || exinfo.trapnr != 14 /*#PF*/)
+		exit(1);
+}
+
+static int enclave_ex_callback(long rdi, long rsi, long rdx,
+	struct sgx_enclave_exinfo *ei, long r8, long r9, void *tcs, long ursp)
+{
+	show_enclave_exinfo(ei, "  callback");
+
+	switch (ei->leaf)
+	{
+	case 4:
+		return 0;
+	case 3:
+	case 2:
+		if (ei->trapnr != 14 /*#PF*/ || (ei->error_code & 1) == 0) {
+			fprintf(stderr, ERRLN ": Unexpected exception\n");
+			exit(1);
+		}
+
+		if (mprotect((void*)(ei->address & -0x1000), 0x1000,
+			     ((ei->error_code & 2) ? PROT_WRITE : 0) |
+			     ((ei->error_code & 0x10) ? PROT_EXEC : 0) |
+			     PROT_READ)) {
+			perror(ERRLN);
+			exit(1);
+		}
+
+		return ei->leaf == 2 ? -EAGAIN : ei->leaf;
+	}
+	return -EINVAL;
+}
+
+static void test2(void *eenter, struct sgx_secs *secs)
+{
+	uint64_t result = 0;
+	struct sgx_enclave_exinfo exinfo;
+
+	printf("[2] Entering the enclave with callback.\n");
+
+	printf("Input: 0x%lx\n Expect: Same as input\n", MAGIC);
+	sgx_call((void *)&MAGIC, &result, 0, NULL, NULL, NULL,
+		 (void *)secs->base, &exinfo, enclave_ex_callback, eenter);
+	if (result != MAGIC) {
+		fprintf(stderr, "0x%lx != 0x%lx\n", result, MAGIC);
+		exit(1);
+	}
+	printf(" Output: 0x%lx\n", result);
+
+	printf("Input: Read-only enclave (0x%lx-0x%lx)\n"
+	       " Expect: #PFs to be fixed by callback\n",
+	       secs->base, secs->base + (encl_bin_end - encl_bin) - 1);
+	if (mprotect((void*)secs->base, encl_bin_end - encl_bin, PROT_READ)) {
+		perror(ERRLN);
+		exit(1);
+	}
+	while (sgx_call((void *)&MAGIC, &result, 0, NULL, NULL, NULL,
+			(void*)secs->base, &exinfo, enclave_ex_callback,
+			eenter) == -EAGAIN);
+	show_enclave_exinfo(&exinfo, " Exit");
+	if (exinfo.leaf != 4 /*EEXIT*/)
+		exit(1);
+}
 
 int main(int argc, char *argv[], char *envp[])
 {
 	unsigned long bin_size = encl_bin_end - encl_bin;
 	unsigned long ss_size = encl_ss_end - encl_ss;
-	struct sgx_enclave_exception exception;
 	Elf64_Sym *eenter_sym;
 	struct vdso_symtab symtab;
 	struct sgx_secs secs;
-	uint64_t result = 0;
 	void *eenter;
 	void *addr;
 
-	memset(&exception, 0, sizeof(exception));
-
 	addr = vdso_get_base_addr(envp);
 	if (!addr)
 		exit(1);
@@ -266,14 +360,7 @@ int main(int argc, char *argv[], char *envp[])
 	if (!encl_load(&secs, bin_size))
 		exit(1);
 
-	printf("Input: 0x%lx\n", MAGIC);
-	sgx_call((void *)&MAGIC, &result, (void *)secs.base, &exception,
-		 eenter);
-	if (result != MAGIC) {
-		fprintf(stderr, "0x%lx != 0x%lx\n", result, MAGIC);
-		exit(1);
-	}
-
-	printf("Output: 0x%lx\n", result);
-	exit(0);
+	test1(eenter, &secs);
+	test2(eenter, &secs);
+	return 0;
 }
tools/testing/selftests/x86/sgx/sgx_call.S +37 -3
index 14bd0a044199..da8f687a60d2 100644
@@ -7,9 +7,43 @@
 
 	.global sgx_call
 sgx_call:
+	.cfi_startproc
+	push	%r15
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r15, 0
+	push	%r14
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r14, 0
+	push	%r13
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r13, 0
+	push	%r12
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r12, 0
 	push	%rbx
-	mov	$0x02, %rax
-	mov	%rdx, %rbx
-	call	*%r8
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbx, 0
+	push	$0
+	.cfi_adjust_cfa_offset	8
+	push	0x48(%rsp)
+	.cfi_adjust_cfa_offset	8
+	push	0x48(%rsp)
+	.cfi_adjust_cfa_offset	8
+	push	0x48(%rsp)
+	.cfi_adjust_cfa_offset	8
+	mov	$2, %eax
+	call	*0x68(%rsp)
+	add	$0x20, %rsp
+	.cfi_adjust_cfa_offset	-0x20
 	pop	%rbx
+	.cfi_adjust_cfa_offset	-8
+	pop	%r12
+	.cfi_adjust_cfa_offset	-8
+	pop	%r13
+	.cfi_adjust_cfa_offset	-8
+	pop	%r14
+	.cfi_adjust_cfa_offset	-8
+	pop	%r15
+	.cfi_adjust_cfa_offset	-8
 	ret
+	.cfi_endproc
Signature · 1 line
2.17.1
maintainer
#254 lore raw

Re: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

Quoted 5 lines · Cedric Xing

Given the changes to __vdso_sgx_enter_enclave(), the selftest is augmented to test the newly added callback interface. This addtional test marks the whole enclave range as PROT_READ, and calls mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes.

Nifty.

What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then do_debug() will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?

--Andy

#255 lore raw

Re: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

Quoted 9 lines · Andy Lutomirski
Quoted 5 lines · Cedric Xing

Given the changes to __vdso_sgx_enter_enclave(), the selftest is augmented to test the newly added callback interface. This addtional test marks the whole enclave range as PROT_READ, and calls mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes.

Nifty.

Quoted 7 lines

What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then do_debug() will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?

My money would be on the code being broken as opposed to you remembering wrong. I'll take a look at it tomorrow.

#256 lore raw

Re: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

In reply to Andy Lutomirski
Quoted 7 lines · Andy Lutomirski

What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then do_debug() will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?

The code is indeed broken, and I don't see a sane way to make it not broken other than to never do vDSO fixup on #DB or #BP. But that's probably the right thing to do anyways since an attached debugger is likely the intended recipient the 99.9999999% of the time.

The crux of the matter is that it's impossible to identify whether or not a #DB/#BP originated from within an enclave, e.g. an INT3 in an enclave will look identical to an INT3 at the AEP. Even if hardware provided a magic flag, #DB still has scenarios where the intended recipient is ambiguous, e.g. data breakpoint encountered in the enclave but on an address outside of the enclave, breakpoint encountered in the enclave and a code breakpoint on the AEP, etc...

maintainer
#257 lore raw

Re: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

Quoted 15 lines · Sean Christopherson
Quoted 7 lines · Andy Lutomirski

What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then do_debug() will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?

The code is indeed broken, and I don't see a sane way to make it not broken other than to never do vDSO fixup on #DB or #BP. But that's probably the right thing to do anyways since an attached debugger is likely the intended recipient the 99.9999999% of the time.

Quoted 7 lines

The crux of the matter is that it's impossible to identify whether or not a #DB/#BP originated from within an enclave, e.g. an INT3 in an enclave will look identical to an INT3 at the AEP. Even if hardware provided a magic flag, #DB still has scenarios where the intended recipient is ambiguous, e.g. data breakpoint encountered in the enclave but on an address outside of the enclave, breakpoint encountered in the enclave and a code breakpoint on the AEP, etc...

Ugh. It sounds like ignoring the fixup for #DB is the right call. But what happens if the enclave contains an INT3 or ICEBP instruction?
Are they magically promoted to #GP, perhaps?

As a maybe possible alternative, if we made it so that the AEX address was not the same as the ENCLU, could we usefully distinguish these exceptions based on RIP? I suppose it's also worth considering whether page faults from *inside* the enclave should result in SIGSEGV or result in a fixup. We certainly want page faults from the ENCLU instruction itself to get fixed up, but maybe we want most exceptions inside the enclave to work a bit differently. Of course, if we do this, we need to make sure that the semantics of returning from the signal handler are reasonable.

#258 lore raw

Re: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

Quoted 25 lines · Andy Lutomirski
Quoted 15 lines · Sean Christopherson
Quoted 7 lines · Andy Lutomirski

What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then do_debug() will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?

The code is indeed broken, and I don't see a sane way to make it not broken other than to never do vDSO fixup on #DB or #BP. But that's probably the right thing to do anyways since an attached debugger is likely the intended recipient the 99.9999999% of the time.

Quoted 7 lines

The crux of the matter is that it's impossible to identify whether or not a #DB/#BP originated from within an enclave, e.g. an INT3 in an enclave will look identical to an INT3 at the AEP. Even if hardware provided a magic flag, #DB still has scenarios where the intended recipient is ambiguous, e.g. data breakpoint encountered in the enclave but on an address outside of the enclave, breakpoint encountered in the enclave and a code breakpoint on the AEP, etc...

Quoted 3 lines

Ugh. It sounds like ignoring the fixup for #DB is the right call. But what happens if the enclave contains an INT3 or ICEBP instruction?
Are they magically promoted to #GP, perhaps?

#UD for opt-out, a.k.a. non-debug, enclaves. Delivered "normally" for opt-in debug enclaves, except they're fault-like instead of trap-like.

Quoted 3 lines

As a maybe possible alternative, if we made it so that the AEX address was not the same as the ENCLU, could we usefully distinguish these exceptions based on RIP?

Not really, because a user could set a code breakpoint on the AEX or insert an INT3, e.g. to break on exit from the enclave.

Theoretically the kernel could cross-reference addresses to determine whether or not a DRx match occurred on an enclave address, but a) that'd be pretty ugly to implement and b) there would still be ambiguity, e.g. if there's a code breakpoint on the AEX and a #DB occurs in the enclave, then DR6 will record both the in-enclave DRx match and the AEX (non-enclave) DRx match.

Quoted 7 lines

I suppose it's also worth considering
whether page faults from *inside* the enclave should result in SIGSEGV or result in a fixup. We certainly want page faults from the ENCLU instruction itself to get fixed up, but maybe we want most exceptions inside the enclave to work a bit differently. Of course, if we do this, we need to make sure that the semantics of returning from the signal handler are reasonable.

Hmm, I'm pretty sure any fault that is 100% in the domain of the enclave should result in fixup.

Here are a few use cases off the top of my head that would require the enclave's runtime to intercept the signal, either to reenter the enclave or to feed the fault into the enclave's handler:

- Handle EPC invalidation, e.g. due to VM migration, while a thread is in the enclave since the resulting #PF can occur inside the enclave.

- During enclave development, configure the runtime to call into the enclave on any exception so that the enclave can dump register state.

- Implement copy-on-write or lazy allocation using SGX2 instructions, which would require feeding the #PF back into the enclave. Purely theoretical AFAIK, but lazy allocation in particular could be interesting, e.g. don't allocate .bss pages at startup time.

- An enclave and its runtime might feed #UDs back into the enclave, e.g. to run an unmodified binary in an enclave by wrapping it in a shim of sorts.

#259 lore raw

Re: [RFC PATCH v1 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

In reply to Cedric Xing
Quoted 20 lines · Cedric Xing

The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.

This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new __vdso_sgx_enter_enclave() will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null), __vdso_sgx_enter_enclave() will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).

Quoted 3 lines

The SGX selftest is augmented to test out the new callback interface, and to serve as a simple example to showcase how to use the callback interface in practice.

Please wrap your emails, or use an editor that will do it for you. 75 chars, plus or minus a few, is generally preferred.

Quoted 18 lines

Cedric Xing (3):
selftests/x86: Fixed Makefile for SGX selftest
x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing
on untrusted stack
selftests/x86: Augment SGX selftest to test new
__vdso_sgx_enter_enclave() and its callback interface

 arch/x86/entry/vdso/vsgx_enter_enclave.S   | 156 ++++++++++++---------
 arch/x86/include/uapi/asm/sgx.h            |  14 +-
 tools/testing/selftests/x86/Makefile       |  12 +-
 tools/testing/selftests/x86/sgx/Makefile   |  45 +++---
 tools/testing/selftests/x86/sgx/main.c     | 123 +++++++++++++---
 tools/testing/selftests/x86/sgx/sgx_call.S |  40 +++++-
 6 files changed, 264 insertions(+), 126 deletions(-)
Signature · 1 line
2.17.1
#260 lore raw

v2 cover An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

In reply to Cedric Xing

The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.

This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new __vdso_sgx_enter_enclave() will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null), __vdso_sgx_enter_enclave() will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).

The SGX selftest is augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in __vdso_sgx_enter_enclave() by single-stepping through it and unwinding call stack at every instruction.

v2:
- Revised comments in __vdso_sgx_enter_enclave(). See patch 2/3.
- Added stack unwind test. See patch 3/3.

v1: https://lkml.org/lkml/2019/4/22/871

Note: This patchset is based upon SGX1 patch v20
(https://lkml.org/lkml/2019/4/17/344) by Jarkko Sakkinen

Cedric Xing (3):
selftests/x86: Fixed Makefile for SGX selftest
x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing
on untrusted stack
selftests/x86: Augment SGX selftest to test new
__vdso_sgx_enter_enclave() and its callback interface

 arch/x86/entry/vdso/vsgx_enter_enclave.S   | 175 +++++++----
 arch/x86/include/uapi/asm/sgx.h            |  14 +-
 tools/testing/selftests/x86/Makefile       |  12 +-
 tools/testing/selftests/x86/sgx/Makefile   |  49 ++--
 tools/testing/selftests/x86/sgx/main.c     | 323 ++++++++++++++++++---
 tools/testing/selftests/x86/sgx/sgx_call.S |  40 ++-
 6 files changed, 471 insertions(+), 142 deletions(-)
Signature · 1 line
2.17.1
#261 lore raw

RFC PATCH v2 1/3 selftests/x86: Fixed Makefile for SGX selftest

The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.

---
 tools/testing/selftests/x86/Makefile     | 12 +++----
 tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++---------------
 2 files changed, 22 insertions(+), 35 deletions(-)
tools/testing/selftests/x86/Makefile +6 -6
index 4fc9a42f56ea..1294c5f5b6ca 100644
@@ -70,11 +70,11 @@ all_32: $(BINARIES_32)
 
 all_64: $(BINARIES_64)
 
-all_64: $(SUBDIRS_64)
-	@for DIR in $(SUBDIRS_64); do			\
-		BUILD_TARGET=$(OUTPUT)/$$DIR;		\
-		mkdir $$BUILD_TARGET  -p;		\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+all_64: | $(SUBDIRS_64)
+	@for DIR in $|; do					\
+		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
+		mkdir $$BUILD_TARGET  -p;			\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
 	done
 
 EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)
@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01)
 all: warn_32bit_failure
 
 warn_32bit_failure:
-	@echo "Warning: you seem to have a broken 32-bit build" 2>&1; 	\
+	@echo "Warning: you seem to have a broken 32-bit build" 2>&1;	\
 	echo "environment.  This will reduce test coverage of 64-bit" 2>&1; \
 	echo "kernels.  If you are using a Debian-like distribution," 2>&1; \
 	echo "try:"; 2>&1; \
tools/testing/selftests/x86/sgx/Makefile +16 -29

63 lines not on this page · raw message

Signature · 1 line
2.17.1
#262 lore raw

Re: [RFC PATCH v2 1/3] selftests/x86: Fixed Makefile for SGX selftest

Quoted 18 lines · Cedric Xing

The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.

---
 tools/testing/selftests/x86/Makefile     | 12 +++----
 tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++---------------
 2 files changed, 22 insertions(+), 35 deletions(-)
tools/testing/selftests/x86/Makefile +0 -0
index 4fc9a42f56ea..1294c5f5b6ca 100644
@@ -70,11 +70,11 @@ all_32: $(BINARIES_32)
 
 all_64: $(BINARIES_64)
 

tools/testing/selftests/x86/Makefile all_32: $(BINARIES_32) · line 77

Quoted 10 lines
tools/testing/selftests/x86/Makefile +5 -5
-all_64: $(SUBDIRS_64)
-	@for DIR in $(SUBDIRS_64); do			\
-		BUILD_TARGET=$(OUTPUT)/$$DIR;		\
-		mkdir $$BUILD_TARGET  -p;		\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+all_64: | $(SUBDIRS_64)
+	@for DIR in $|; do					\
+		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
+		mkdir $$BUILD_TARGET  -p;			\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\

This is not fix for anything. It is change in semantics. This diff should be isolated to its own commit as you are changing something outside of SGX scope.

tools/testing/selftests/x86/Makefile ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) · line 93

Quoted 9 lines
tools/testing/selftests/x86/Makefile +1 -1
 	done
 
 EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)
@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01)
 all: warn_32bit_failure
 
 warn_32bit_failure:
-	@echo "Warning: you seem to have a broken 32-bit build" 2>&1; 	\
+	@echo "Warning: you seem to have a broken 32-bit build" 2>&1;	\

Please clean this up.

Quoted 74 lines
tools/testing/selftests/x86/Makefile +0 -0
 	echo "environment.  This will reduce test coverage of 64-bit" 2>&1; \
 	echo "kernels.  If you are using a Debian-like distribution," 2>&1; \
 	echo "try:"; 2>&1; \
tools/testing/selftests/x86/sgx/Makefile +16 -29
index 1fd6f2708e81..3af15d7c8644 100644
@@ -2,47 +2,34 @@ top_srcdir = ../../../../..
 
 include ../../lib.mk
 
-HOST_CFLAGS := -Wall -Werror -g $(INCLUDES) -fPIC
-ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIC \
+ifeq ($(shell $(CC) -dumpmachine | cut --delimiter=- -f1),x86_64)
+all: all_64
+endif
+
+HOST_CFLAGS := -Wall -Werror -g $(INCLUDES)
+ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIE \
 	       -fno-stack-protector -mrdrnd $(INCLUDES)
 
 TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx
 all_64: $(TEST_CUSTOM_PROGS)
 
-$(TEST_CUSTOM_PROGS): $(OUTPUT)/main.o $(OUTPUT)/sgx_call.o \
-		      $(OUTPUT)/encl_piggy.o
+$(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o
 	$(CC) $(HOST_CFLAGS) -o $@ $^
 
-$(OUTPUT)/main.o: main.c
-	$(CC) $(HOST_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/sgx_call.o: sgx_call.S
-	$(CC) $(HOST_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/encl_piggy.o: $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
-	$(CC) $(HOST_CFLAGS) -c encl_piggy.S -o $@
+$(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
+	$(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@
 
-$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf $(OUTPUT)/sgxsign
+$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf
 	objcopy --remove-section=.got.plt -O binary $< $@
 
-$(OUTPUT)/encl.elf: $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o
-	$(CC) $(ENCL_CFLAGS) -T encl.lds -o $@ $^
+$(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S
+	$(CC) $(ENCL_CFLAGS) -T $^ -o $@
 
-$(OUTPUT)/encl.o: encl.c
-	$(CC) $(ENCL_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/encl_bootstrap.o: encl_bootstrap.S
-	$(CC) $(ENCL_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/encl.ss: $(OUTPUT)/encl.bin  $(OUTPUT)/sgxsign
-	$(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
+$(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin
+	$^ $@
 
 $(OUTPUT)/sgxsign: sgxsign.c
 	$(CC) -o $@ $< -lcrypto
 
-EXTRA_CLEAN := $(OUTPUT)/sgx-selftest $(OUTPUT)/sgx-selftest.o \
-	       $(OUTPUT)/sgx_call.o $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss \
-	       $(OUTPUT)/encl.elf $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o \
-	       $(OUTPUT)/sgxsign
-
-.PHONY: clean
+EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(addprefix $(OUTPUT)/,	\
+		encl.elf encl.bin encl.ss encl_piggy.o sgxsign)
Signature · 1 line
2.17.1

What are all these changes to the makefile? I don't see mention of them in the commit message.

/Jarkko

#263 lore raw

Re: [RFC PATCH v2 1/3] selftests/x86: Fixed Makefile for SGX selftest

Quoted 19 lines · Jarkko Sakkinen
Quoted 18 lines · Cedric Xing

The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.

---
  tools/testing/selftests/x86/Makefile     | 12 +++----
  tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++---------------
  2 files changed, 22 insertions(+), 35 deletions(-)
tools/testing/selftests/x86/Makefile +0 -0
index 4fc9a42f56ea..1294c5f5b6ca 100644
@@ -70,11 +70,11 @@ all_32: $(BINARIES_32)
 
  all_64: $(BINARIES_64)
 
Quoted 14 lines

tools/testing/selftests/x86/Makefile all_32: $(BINARIES_32) · line 77

Quoted 10 lines
tools/testing/selftests/x86/Makefile +5 -5
-all_64: $(SUBDIRS_64)
-	@for DIR in $(SUBDIRS_64); do			\
-		BUILD_TARGET=$(OUTPUT)/$$DIR;		\
-		mkdir $$BUILD_TARGET  -p;		\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+all_64: | $(SUBDIRS_64)
+	@for DIR in $|; do					\
+		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
+		mkdir $$BUILD_TARGET  -p;			\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\

This is not fix for anything. It is change in semantics. This diff should be isolated to its own commit as you are changing something outside of SGX scope.

I have removed all changes to x86/Makefile from v4.

The current x86/Makefile only builds but cannot run sgx selftest. I don't see it as an urgent issue though.

Quoted 75 lines

tools/testing/selftests/x86/Makefile ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) · line 93

Quoted 9 lines
tools/testing/selftests/x86/Makefile +1 -1
  	done
 
  EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)
@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01)
  all: warn_32bit_failure
 
  warn_32bit_failure:
-	@echo "Warning: you seem to have a broken 32-bit build" 2>&1; 	\
+	@echo "Warning: you seem to have a broken 32-bit build" 2>&1;	\

Please clean this up.

Quoted 63 lines
tools/testing/selftests/x86/Makefile +0 -0
  	echo "environment.  This will reduce test coverage of 64-bit" 2>&1; \
  	echo "kernels.  If you are using a Debian-like distribution," 2>&1; \
  	echo "try:"; 2>&1; \
tools/testing/selftests/x86/sgx/Makefile +14 -23
index 1fd6f2708e81..3af15d7c8644 100644
@@ -2,47 +2,34 @@ top_srcdir = ../../../../..
 
  include ../../lib.mk
 
-HOST_CFLAGS := -Wall -Werror -g $(INCLUDES) -fPIC
-ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIC \
+ifeq ($(shell $(CC) -dumpmachine | cut --delimiter=- -f1),x86_64)
+all: all_64
+endif
+
+HOST_CFLAGS := -Wall -Werror -g $(INCLUDES)
+ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIE \
  	       -fno-stack-protector -mrdrnd $(INCLUDES)
 
  TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx
  all_64: $(TEST_CUSTOM_PROGS)
 
-$(TEST_CUSTOM_PROGS): $(OUTPUT)/main.o $(OUTPUT)/sgx_call.o \
-		      $(OUTPUT)/encl_piggy.o
+$(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o
  	$(CC) $(HOST_CFLAGS) -o $@ $^
 
-$(OUTPUT)/main.o: main.c
-	$(CC) $(HOST_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/sgx_call.o: sgx_call.S
-	$(CC) $(HOST_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/encl_piggy.o: $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
-	$(CC) $(HOST_CFLAGS) -c encl_piggy.S -o $@
+$(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
+	$(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@
 
-$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf $(OUTPUT)/sgxsign
+$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf
  	objcopy --remove-section=.got.plt -O binary $< $@
 
-$(OUTPUT)/encl.elf: $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o
-	$(CC) $(ENCL_CFLAGS) -T encl.lds -o $@ $^
+$(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S
+	$(CC) $(ENCL_CFLAGS) -T $^ -o $@
 
-$(OUTPUT)/encl.o: encl.c
-	$(CC) $(ENCL_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/encl_bootstrap.o: encl_bootstrap.S
-	$(CC) $(ENCL_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/encl.ss: $(OUTPUT)/encl.bin  $(OUTPUT)/sgxsign
-	$(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
+$(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin
+	$^ $@
 
  $(OUTPUT)/sgxsign: sgxsign.c
  	$(CC) -o $@ $< -lcrypto
 
Quoted 14 lines
Quoted 11 lines
tools/testing/selftests/x86/sgx/Makefile +2 -6
-EXTRA_CLEAN := $(OUTPUT)/sgx-selftest $(OUTPUT)/sgx-selftest.o \
-	       $(OUTPUT)/sgx_call.o $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss \
-	       $(OUTPUT)/encl.elf $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o \
-	       $(OUTPUT)/sgxsign
-
-.PHONY: clean
+EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(addprefix $(OUTPUT)/,	\
+		encl.elf encl.bin encl.ss encl_piggy.o sgxsign)
Signature · 1 line
2.17.1

What are all these changes to the makefile? I don't see mention of them in the commit message.

Added description to the commit message in v4.

Quoted 2 lines

/Jarkko

#264 lore raw

RFC PATCH v2 2/3 x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

In reply to Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 175 ++++++++++++++++-------
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 128 insertions(+), 61 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +121 -54
index fe0bf6671d6d..debecb0d785c 100644
@@ -19,83 +19,150 @@
  * __vdso_sgx_enter_enclave() - Enter an SGX enclave
  *
  * @leaf:	**IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:	**IN \%rbx** - TCS, must be non-NULL
- * @ex_info:	**IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer
+ * @tcs:	**IN 0x08(\%rsp)** - TCS, must be non-NULL
+ * @ex_info:	**IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo'
+ *				     pointer
+ * @callback:	**IN 0x18(\%rsp)** - Optional callback function to be called on
+ *				     enclave exit or exception
  *
  * Return:
  *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is
- *  not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults
+ *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not
+ *  allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value
+ *  returned from ``callback`` (if one is supplied).
  *
  * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant with the
- * x86-64 ABI, i.e. cannot be called from standard C code.   As noted above,
- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``.  All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc...  Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * x86-64 ABI, i.e. cannot be called from standard C code. As noted above,
+ * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and
+ * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other
+ * registers will be passed through to the enclave as is. All registers except
+ * ``%rbp`` must be treated as volatile from the caller's perspective, including
+ * but not limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the
+ * enclave being run **must** preserve the untrusted ``%rbp``.
+ *
+ * ``callback`` has the following signature:
+ * int callback(long rdi, long rsi, long rdx,
+ *		struct sgx_enclave_exinfo *exinfo, long r8, long r9,
+ *		void *tcs, long ursp);
+ * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``,
+ * ``%rbx`` and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``,
+ * ``%rdx``, ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave
+ * exited/excepted, can be accessed directly as input parameters, while other
+ * GPRs can be accessed in assembly if needed.  A positive value returned from
+ * ``callback`` will be treated as an ENCLU leaf (e.g. EENTER/ERESUME) to
+ * reenter the enclave (without popping the extra data pushed by the enclave off
+ * the stack), while 0 (zero) or a negative return value will be passed back to
+ * the caller of __vdso_sgx_enter_enclave(). It is also safe to leave
+ * ``callback`` via ``longjmp()`` or by throwing a C++ exception.
  */
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-			 struct sgx_enclave_exception *ex_info)
+typedef int (*sgx_callback)(long rdi, long rsi, long rdx,
+			    struct sgx_enclave_exinfo *exinfo, long r8,
+			    long r9, void *tcs, long ursp);
+int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+			     struct sgx_enclave_exinfo *exinfo,
+			     sgx_callback callback)
 {
-	if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-		return -EINVAL;
-
-	if (!tcs)
-		return -EINVAL;
-
-	try {
-		ENCLU[leaf];
-	} catch (exception) {
-		if (e)
-			*e = exception;
-		return -EFAULT;
+	while (leaf == EENTER || leaf == ERESUME) {
+		int rc;
+		try {
+			ENCLU[leaf];
+			rc = 0;
+			if (exinfo)
+				exinfo->leaf = EEXIT;
+		} catch (exception) {
+			rc = -EFAULT;
+			if (exinfo)
+				*exinfo = exception;
+		}
+
+		leaf = callback ? (*callback)(
+			rdi, rsi, rdx, exinfo, r8, r9, tcs, ursp) : rc;
 	}
 
-	return 0;
+	return leaf > 0 ? -EINVAL : leaf;
 }
 #endif
 ENTRY(__vdso_sgx_enter_enclave)
-	/* EENTER <= leaf <= ERESUME */
+	/* Prolog */
+	.cfi_startproc
+	push	%rbp
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbp, 0
+	mov	%rsp, %rbp
+	.cfi_def_cfa_register	%rbp
+
+1:	/* EENTER <= leaf <= ERESUME */
 	cmp	$0x2, %eax
-	jb	bad_input
-
+	jb	6f
 	cmp	$0x3, %eax
-	ja	bad_input
+	ja	6f
 
-	/* TCS must be non-NULL */
-	test	%rbx, %rbx
-	je	bad_input
+	/* Load TCS and AEP */
+	mov	0x10(%rbp), %rbx
+	lea	2f(%rip), %rcx
 
-	/* Save @exception_info */
-	push	%rcx
+	/* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:	enclu
 
-	/* Load AEP for ENCLU */
-	lea	1f(%rip),  %rcx
-1:	enclu
+	/* EEXIT path */
+	xor	%ebx, %ebx
+3:	mov	0x18(%rbp), %rcx
+	jrcxz	4f
+	mov	%eax, EX_LEAF(%rcx)
+	jnc	4f
+	mov	%di, EX_TRAPNR(%rcx)
+	mov	%si, EX_ERROR_CODE(%rcx)
+	mov	%rdx, EX_ADDRESS(%rcx)
 
-	add	$0x8, %rsp
-	xor	%eax, %eax
+4:	/* Call *callback if supplied */
+	mov	0x20(%rbp), %rax
+	test	%rax, %rax
+	/* At this point, %ebx holds the effective return value, which shall be
+	 * returned if no callback is specified */
+	cmovz	%rbx, %rax
+	jz	7f
+	/* Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be
+	 * restored after *callback returns. */
+	mov	%rsp, %rbx
+	and	$-0x10, %rsp
+	/* Clear RFLAGS.DF per x86_64 ABI */
+	cld
+	/* Parameters for *callback */
+	push	%rbx
+	push	0x10(%rbp)
+	/* Call *%rax via retpoline */
+	call	40f
+	/* Restore %rsp to its original value left off by the enclave from last
+	 * exit */
+	mov	%rbx, %rsp
+	/* Positive return value from *callback will be interpreted as an ENCLU
+	 * leaf, while a non-positive value will be interpreted as the return
+	 * value to be passed back to the caller. */
+	jmp	1b
+40:	/* retpoline */
+	call	42f
+41:	pause
+	lfence
+	jmp	41b
+42:	mov	%rax, (%rsp)
 	ret
 
-bad_input:
-	mov     $(-EINVAL), %rax
-	ret
+5:	/* Exception path */
+	mov	$-EFAULT, %ebx
+	stc
+	jmp	3b
 
-.pushsection .fixup, "ax"
-	/* Re-load @exception_info and fill it (if it's non-NULL) */
-2:	pop	%rcx
-	test    %rcx, %rcx
-	je      3f
+6:	/* Unsupported ENCLU leaf */
+	cmp	$0, %eax
+	jle	7f
+	mov	$-EINVAL, %eax
 
-	mov	%eax, EX_LEAF(%rcx)
-	mov	%di,  EX_TRAPNR(%rcx)
-	mov	%si,  EX_ERROR_CODE(%rcx)
-	mov	%rdx, EX_ADDRESS(%rcx)
-3:	mov	$(-EFAULT), %rax
+7:	/* Epilog */
+	leave
+	.cfi_def_cfa		%rsp, 8
 	ret
-.popsection
+	.cfi_endproc
 
-_ASM_VDSO_EXTABLE_HANDLE(1b, 2b)
+_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)
 
 ENDPROC(__vdso_sgx_enter_enclave)
arch/x86/include/uapi/asm/sgx.h +7 -7
index 9ed690a38c70..50d2b5143e5e 100644
@@ -24,7 +24,7 @@
 
 /**
  * struct sgx_enclave_create - parameter structure for the
- *                             %SGX_IOC_ENCLAVE_CREATE ioctl
+ *			       %SGX_IOC_ENCLAVE_CREATE ioctl
  * @src:	address for the SECS page data
  */
 struct sgx_enclave_create  {
@@ -33,7 +33,7 @@ struct sgx_enclave_create  {
 
 /**
  * struct sgx_enclave_add_page - parameter structure for the
- *                               %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
+ *				 %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
  * @addr:	address within the ELRANGE
  * @src:	address for the page data
  * @secinfo:	address for the SECINFO data
@@ -49,7 +49,7 @@ struct sgx_enclave_add_page {
 
 /**
  * struct sgx_enclave_init - parameter structure for the
- *                           %SGX_IOC_ENCLAVE_INIT ioctl
+ *			     %SGX_IOC_ENCLAVE_INIT ioctl
  * @sigstruct:	address for the SIGSTRUCT data
  */
 struct sgx_enclave_init {
@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute {
 };
 
 /**
- * struct sgx_enclave_exception - structure to report exceptions encountered in
- *				  __vdso_sgx_enter_enclave()
+ * struct sgx_enclave_exinfo - structure to report exceptions encountered in
+ *			       __vdso_sgx_enter_enclave()
  *
- * @leaf:	ENCLU leaf from \%eax at time of exception
+ * @leaf:	ENCLU leaf from \%eax at time of exception/exit
  * @trapnr:	exception trap number, a.k.a. fault vector
  * @error_code:	exception error code
  * @address:	exception address, e.g. CR2 on a #PF
  * @reserved:	reserved for future use
  */
-struct sgx_enclave_exception {
+struct sgx_enclave_exinfo {
 	__u32 leaf;
 	__u16 trapnr;
 	__u16 error_code;
Signature · 1 line
2.17.1
#265 lore raw

Re: [RFC PATCH v2 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 4 lines · Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).

Quoted 10 lines

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.

AIUI, you and Andy had an off-list discussion regarding rewriting __vdso_sgx_enter_enclave() vs. providing a second vDSO function. Can you please summarize the discussion so that it's clear why you're pursuing a single function?

In the end I probably agree that's it's desirable to have a single ABI and vDSO function since the cost is basically that the non-callback case needs to pass params via the stack instead of registers, but I do miss the simplicity of separate implementations.

Quoted 50 lines

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 175 ++++++++++++++++-------
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 128 insertions(+), 61 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +8 -5
index fe0bf6671d6d..debecb0d785c 100644
@@ -19,83 +19,150 @@
  * __vdso_sgx_enter_enclave() - Enter an SGX enclave
  *
  * @leaf:	**IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:	**IN \%rbx** - TCS, must be non-NULL
- * @ex_info:	**IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer
+ * @tcs:	**IN 0x08(\%rsp)** - TCS, must be non-NULL
+ * @ex_info:	**IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo'
+ *				     pointer
+ * @callback:	**IN 0x18(\%rsp)** - Optional callback function to be called on
+ *				     enclave exit or exception
  *
  * Return:
  *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is
- *  not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults
+ *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not
+ *  allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value
+ *  returned from ``callback`` (if one is supplied).
  *
  * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant with the
- * x86-64 ABI, i.e. cannot be called from standard C code.   As noted above,

arch/x86/entry/vdso/vsgx_enter_enclave.S line 39

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5 -5
- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``.  All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc...  Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * x86-64 ABI, i.e. cannot be called from standard C code. As noted above,
+ * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and
+ * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other
+ * registers will be passed through to the enclave as is. All registers except
+ * ``%rbp`` must be treated as volatile from the caller's perspective, including

Hmm, this is my fault since I wrote the original comment, but stating "All registers except %rbp must be treated as volatile" is confusing, e.g. at first I thought it meant that the assembly blob was mucking with the registers and that they couldn't be used to pass information out of the enclave.

Maybe something like:

* Register ABI:

 *  - As noted above, input parameters are passed via %eax and the stack.
 *  - The return value is passed via %eax.
 *  - %rbx and %rcx must be treated as volatile as they are modified as part of

* enclaves transitions and are used as scratch regs.
* - %rbp **must** be preserved by the enclave in all cases, as it is used to
* reference paramaters when handling enclave exits.
* - %rdx, %rdi, %rsi and %r8-%r15 may be freely modified by the enclave
* and will not be passed back to the caller untouched.
* - %rsp is saved/restored across __vdso_sgx_enter_enclave(), but is passed
* as-is to the callback if one is provided, i.e. the enclave may use u_rsp
* to pass information to its associated callback.
*
* Callback ABI:
* <more information about callback ABI>
*

Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +8 -0
+ * but not limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the
+ * enclave being run **must** preserve the untrusted ``%rbp``.
+ *
+ * ``callback`` has the following signature:
+ * int callback(long rdi, long rsi, long rdx,
+ *		struct sgx_enclave_exinfo *exinfo, long r8, long r9,
+ *		void *tcs, long ursp);
+ * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``,

arch/x86/entry/vdso/vsgx_enter_enclave.S line 57

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +9 -0
+ * ``%rbx`` and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``,
+ * ``%rdx``, ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave
+ * exited/excepted, can be accessed directly as input parameters, while other
+ * GPRs can be accessed in assembly if needed.  A positive value returned from
+ * ``callback`` will be treated as an ENCLU leaf (e.g. EENTER/ERESUME) to
+ * reenter the enclave (without popping the extra data pushed by the enclave off
+ * the stack), while 0 (zero) or a negative return value will be passed back to
+ * the caller of __vdso_sgx_enter_enclave(). It is also safe to leave
+ * ``callback`` via ``longjmp()`` or by throwing a C++ exception.
  */

While this may format nicely in .rst (I haven't checked), it's damn near unreadable in its raw form. Escaping '%' in kernel-doc is unresolved at this point, but this definitely is an argument for allowing the %CONST interpretation to be disabled entirely.

longjmp() is probably the only thing that needs to be explicitly marked, I think it would be better to simply say "the callback" instead of using ``callback``, i.e. use regular English instead of referencing the param.

Quoted 12 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6 -5
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-			 struct sgx_enclave_exception *ex_info)
+typedef int (*sgx_callback)(long rdi, long rsi, long rdx,
+			    struct sgx_enclave_exinfo *exinfo, long r8,
+			    long r9, void *tcs, long ursp);
+int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+			     struct sgx_enclave_exinfo *exinfo,
+			     sgx_callback callback)
 {
-	if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-		return -EINVAL;
-

arch/x86/entry/vdso/vsgx_enter_enclave.S line 65

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -9
-	if (!tcs)
-		return -EINVAL;
-
-	try {
-		ENCLU[leaf];
-	} catch (exception) {
-		if (e)
-			*e = exception;
-		return -EFAULT;
+	while (leaf == EENTER || leaf == ERESUME) {

This gives the impression that looping is the common case. I'd prefer using 'goto' to show that the common case is a simple fall through whereas the callback case can effectively loop on ENCLU.

Quoted 47 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +29 -9
+		int rc;
+		try {
+			ENCLU[leaf];
+			rc = 0;
+			if (exinfo)
+				exinfo->leaf = EEXIT;
+		} catch (exception) {
+			rc = -EFAULT;
+			if (exinfo)
+				*exinfo = exception;
+		}
+
+		leaf = callback ? (*callback)(
+			rdi, rsi, rdx, exinfo, r8, r9, tcs, ursp) : rc;
 	}
 
-	return 0;
+	return leaf > 0 ? -EINVAL : leaf;
 }
 #endif
 ENTRY(__vdso_sgx_enter_enclave)
-	/* EENTER <= leaf <= ERESUME */
+	/* Prolog */
+	.cfi_startproc
+	push	%rbp
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbp, 0
+	mov	%rsp, %rbp
+	.cfi_def_cfa_register	%rbp
+
+1:	/* EENTER <= leaf <= ERESUME */
 	cmp	$0x2, %eax
-	jb	bad_input
-
+	jb	6f
 	cmp	$0x3, %eax
-	ja	bad_input
+	ja	6f
 
-	/* TCS must be non-NULL */
-	test	%rbx, %rbx
-	je	bad_input
+	/* Load TCS and AEP */
+	mov	0x10(%rbp), %rbx
+	lea	2f(%rip), %rcx
 
-	/* Save @exception_info */

arch/x86/entry/vdso/vsgx_enter_enclave.S line 109

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5 -4
-	push	%rcx
+	/* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:	enclu
 
-	/* Load AEP for ENCLU */
-	lea	1f(%rip),  %rcx
-1:	enclu
+	/* EEXIT path */
+	xor	%ebx, %ebx
+3:	mov	0x18(%rbp), %rcx

@exinfo is optional, i.e. %ecx needs to be tested before its used.

arch/x86/entry/vdso/vsgx_enter_enclave.S line 110

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+	jrcxz	4f

I dislike the number of flags and values that are stashed away only to be consumed later, it makes the code hard to follow. AFAICT, a lot of the shenanigans are done to reuse code because exinfo was overloaded, but that's actually pointless, i.e. it's unnecessarily complex.

Overlading the struct is pointless becase if you make it mandatory when using a callback then it can be nullified (when passed to the callback) to indicate EEXIT. If it's still optional, then the callback needs an extra param to explicitly indicate EEXIT vs. -EFAULT, otherwise the callback has no indication whatsover of why it was invoked.

My preference is for the latter since it's more explicit and obvious.

Tangetially related, I'm not opposed to renaming it 'struct sgx_enclave_exception_info' for clarity.

arch/x86/entry/vdso/vsgx_enter_enclave.S line 115

Quoted 5 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5 -0
+	mov	%eax, EX_LEAF(%rcx)
+	jnc	4f
+	mov	%di, EX_TRAPNR(%rcx)
+	mov	%si, EX_ERROR_CODE(%rcx)
+	mov	%rdx, EX_ADDRESS(%rcx)

My strong preference would be to organize the code to separate the various flows, i.e. happy common case, invalid input, exception handler and callback invocation. And make the common case a straight shot so that it's more obvious what happens in the happy non-callback case.

arch/x86/entry/vdso/vsgx_enter_enclave.S line 121

Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5 -2
-	add	$0x8, %rsp
-	xor	%eax, %eax
+4:	/* Call *callback if supplied */
+	mov	0x20(%rbp), %rax
+	test	%rax, %rax
+	/* At this point, %ebx holds the effective return value, which shall be
+	 * returned if no callback is specified */

Use kernel-doc format for multi-line comments. Missing punctionation at the end. Blank lines between logical blocks would also help readability.

E.g.:

4: /* Call *callback if supplied */
mov 0x20(%rbp), %rax
test %rax, %rax

/*
* At this point, %ebx holds the effective return value, which shall be
* returned if no callback is specified.
*/
cmovz %rbx, %rax
jz 7f

arch/x86/entry/vdso/vsgx_enter_enclave.S line 134

Quoted 13 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +13 -0
+	cmovz	%rbx, %rax
+	jz	7f
+	/* Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be
+	 * restored after *callback returns. */
+	mov	%rsp, %rbx
+	and	$-0x10, %rsp
+	/* Clear RFLAGS.DF per x86_64 ABI */
+	cld
+	/* Parameters for *callback */
+	push	%rbx
+	push	0x10(%rbp)
+	/* Call *%rax via retpoline */
+	call	40f

Another case of stashing a value it consuming it later. This one is especially brutal since %rax was loaded with CMOVcc, which means the reader needs to backtrack even further to understand what %rax contains at this point.

arch/x86/entry/vdso/vsgx_enter_enclave.S line 140

Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6 -0
+	/* Restore %rsp to its original value left off by the enclave from last
+	 * exit */
+	mov	%rbx, %rsp
+	/* Positive return value from *callback will be interpreted as an ENCLU
+	 * leaf, while a non-positive value will be interpreted as the return
+	 * value to be passed back to the caller. */

Please use imperative mood in the comments, i.e. simply state what the code is doing. E.g. "will be interpreted" makes it sound like something else is processing the callback's return value.

Quoted 33 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +16 -13
+	jmp	1b
+40:	/* retpoline */
+	call	42f
+41:	pause
+	lfence
+	jmp	41b
+42:	mov	%rax, (%rsp)
 	ret
 
-bad_input:
-	mov     $(-EINVAL), %rax
-	ret
+5:	/* Exception path */
+	mov	$-EFAULT, %ebx
+	stc
+	jmp	3b
 
-.pushsection .fixup, "ax"
-	/* Re-load @exception_info and fill it (if it's non-NULL) */
-2:	pop	%rcx
-	test    %rcx, %rcx
-	je      3f
+6:	/* Unsupported ENCLU leaf */
+	cmp	$0, %eax
+	jle	7f
+	mov	$-EINVAL, %eax
 
-	mov	%eax, EX_LEAF(%rcx)
-	mov	%di,  EX_TRAPNR(%rcx)
-	mov	%si,  EX_ERROR_CODE(%rcx)
-	mov	%rdx, EX_ADDRESS(%rcx)
-3:	mov	$(-EFAULT), %rax
+7:	/* Epilog */

arch/x86/entry/vdso/vsgx_enter_enclave.S line 168

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4 -2
+	leave
+	.cfi_def_cfa		%rsp, 8
 	ret
-.popsection
+	.cfi_endproc
 
-_ASM_VDSO_EXTABLE_HANDLE(1b, 2b)
+_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)
 
 ENDPROC(__vdso_sgx_enter_enclave)

Putting everything together, sans comments, I'd prefer something like the following. Pasted in raw code as that'll hopefully be easier to review and discuss than a diff.

Note, swapping 'long rc' and '... *e' would allow the callback flow to save one memory access, but IMO the exception info belongs at the end since it's optional.

#ifdef SGX_KERNEL_DOC
typedef int (*sgx_callback)(long rdi, long rsi, long rdx, long ret, long r8,

			    long r9, void *tcs, long ursp,
			    struct sgx_enclave_exception_info *e);
int __vdso_sgx_enter_enclave(int leaf, void *tcs,
				struct sgx_enclave_exception_info *e,
				sgx_callback callback)
{
	int ret;

enter_enclave:

	if (leaf != EENTER && leaf != ERESUME) {
		ret = -EINVAL;
		goto out;
	}

	try {
		ENCLU[leaf];
		ret = 0;
	} catch (exception) {
		ret = -EFAULT;
		if (e)
			*e = exception;
	}
	if (callback)
		goto do_callback;
out:
	return ret;

do_callback:

	leaf = (*callback)(rdi, rsi, rdx, ret, r8, r9, e, tcs, ursp);
	if (leaf <= 0)
		goto out;
	goto enter_enclave;
}
#endif
ENTRY(__vdso_sgx_enter_enclave)
	/* Prolog */
	.cfi_startproc
	push	%rbp
	.cfi_adjust_cfa_offset	8
	.cfi_rel_offset		%rbp, 0
	mov	%rsp, %rbp
	.cfi_def_cfa_register	%rbp

1: /* EENTER <= leaf <= ERESUME */

	cmp	$0x2, %eax
	jb	5f
	cmp	$0x3, %eax
	ja	5f

	/* Load TCS and AEP */
	mov	0x10(%rbp), %rbx
	lea	2f(%rip), %rcx

	/* Single ENCLU serving as both EENTER and AEP (ERESUME) */

2: enclu

	/* EEXIT branches here unless the enclave is doing something fancy. */
	xor	%eax, %eax

3: cmp $0, 0x20(%rbp)

	jne	8f

4: leave

	.cfi_def_cfa		%rsp, 8
	ret

5: mov $(-EINVAL), %rax

	jmp	4b

6: mov 0x18(%rbp), %rcx

	test    %rcx, %rcx
	je	7f

	/* Fill optional exception info. */
	mov	%eax, EX_LEAF(%rcx)
	mov	%di,  EX_TRAPNR(%rcx)
	mov	%si,  EX_ERROR_CODE(%rcx)
	mov	%rdx, EX_ADDRESS(%rcx)
7:	mov	$(-EFAULT), %rax
	jmp	3b

8: /*

	 * Align stack per x86_64 ABI.  Save the original %rsp in %rbx to be
	 * restored after the callback returns.
	 */
	mov	%rsp, %rbx
	and	$-0x10, %rsp

	/* Push @e, u_rsp and @tcs as parameters to the callback. */
	push	0x18(%rbp)
	push	%rbx
	push	0x10(%rbp)

	/* Pass the "return value" to the callback via %rcx. */
	mov	%rax, %rcx

	/* Clear RFLAGS.DF per x86_64 ABI */
	cld

	/* Load the callback pointer to %rax and invoke it via retpoline. */
	mov	0x20(%rbp), %rax
	call	40f

	/* Restore %rsp to its post-exit value. */
	mov	%rbx, %rsp

	/*
	 * If the return from callback is zero or negative, return immediately,
	 * otherwise attempt to re-execute ENCLU with the return interpreted as
	 * the requested ENCLU leaf.
	 */
	cmp	$0, %eax
	jle	4b
	jmp	1b

40: /* retpoline */

	call	42f
41:	pause
	lfence
	jmp	41b

42: mov %rax, (%rsp)
ret
.cfi_endproc

_ASM_VDSO_EXTABLE_HANDLE(2b, 6b)

ENDPROC(__vdso_sgx_enter_enclave)

#266 lore raw

RE: [RFC PATCH v2 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Hi Sean,

Quoted 6 lines

AIUI, you and Andy had an off-list discussion regarding rewriting __vdso_sgx_enter_enclave() vs. providing a second vDSO function. Can you
please summarize the discussion so that it's clear why you're pursuing a single function?

Quoted 7 lines

In the end I probably agree that's it's desirable to have a single ABI and
vDSO function since the cost is basically that the non-callback case needs
to pass params via the stack instead of registers, but I do miss the simplicity of separate implementations.

The major reason is we don't want to maintain 2 different ABIs (preserving %rsp vs. %rbp). Given the new one covers all known use cases, we prefer not to keep the other one.

Quoted 112 lines
Quoted 1 line

Here's the summary of API/ABI changes in this patch. More details

could be

Quoted 2 lines

found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct

sgx_enclave_exinfo'

Quoted 1 line

because it is filled upon both AEX (i.e. exceptions) and normal

enclave

Quoted 2 lines

exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead

of %rsp in

Quoted 2 lines

the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback

function to

Quoted 2 lines

be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon

enclave exits

Quoted 2 lines

(same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at

enclave

Quoted 1 line

exit to address data "pushed" by the enclave. A positive value

returned by

Quoted 1 line

the callback will be treated as an ENCLU leaf for re-entering the

enclave,

Quoted 7 lines

while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 175 ++++++++++++++++------

-

Review

Quoted 4 lines
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 128 insertions(+), 61 deletions(-)
(quoted diff) +0 -0

b/arch/x86/entry/vdso/vsgx_enter_enclave.S

arch/x86/entry/vdso/vsgx_enter_enclave.S line 23

Quoted 9 lines

index fe0bf6671d6d..debecb0d785c 100644

arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -2
@@ -19,83 +19,150 @@
  * __vdso_sgx_enter_enclave() - Enter an SGX enclave
  *
  * @leaf:	**IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:	**IN \%rbx** - TCS, must be non-NULL
- * @ex_info:	**IN \%rcx** - Optional 'struct sgx_enclave_exception'

pointer

arch/x86/entry/vdso/vsgx_enter_enclave.S line 23

Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2 -0
+ * @tcs:	**IN 0x08(\%rsp)** - TCS, must be non-NULL
+ * @ex_info:	**IN 0x10(\%rsp)** - Optional 'struct

sgx_enclave_exinfo'

arch/x86/entry/vdso/vsgx_enter_enclave.S line 25

Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2 -0
+ *				     pointer
+ * @callback:	**IN 0x18(\%rsp)** - Optional callback function to

be called on

arch/x86/entry/vdso/vsgx_enter_enclave.S

Quoted 5 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -1
+ *				     enclave exit or exception
  *
  * Return:
  *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU

leaf is

arch/x86/entry/vdso/vsgx_enter_enclave.S line 28

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -1
- *  not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave

faults

arch/x86/entry/vdso/vsgx_enter_enclave.S line 30

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU

leaf is not

arch/x86/entry/vdso/vsgx_enter_enclave.S line 31

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ *  allowed, %-EFAULT if ENCLU or the enclave faults, or a non-

positive value

arch/x86/entry/vdso/vsgx_enter_enclave.S

Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ *  returned from ``callback`` (if one is supplied).
  *
  * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant

with the

arch/x86/entry/vdso/vsgx_enter_enclave.S line 31

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -1
- * x86-64 ABI, i.e. cannot be called from standard C code.   As noted

above,

arch/x86/entry/vdso/vsgx_enter_enclave.S line 32

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -1
- * input parameters must be passed via ``%eax``, ``%rbx`` and

``%rcx``, with

arch/x86/entry/vdso/vsgx_enter_enclave.S line 33

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -1
- * the return value passed via ``%eax``.  All registers except

``%rsp`` must

arch/x86/entry/vdso/vsgx_enter_enclave.S line 34

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -1
- * be treated as volatile from the caller's perspective, including

but not

arch/x86/entry/vdso/vsgx_enter_enclave.S line 35

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -1
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc...  Conversely, the

enclave

arch/x86/entry/vdso/vsgx_enter_enclave.S line 35

Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -1
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * x86-64 ABI, i.e. cannot be called from standard C code. As noted

above,

arch/x86/entry/vdso/vsgx_enter_enclave.S line 36

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * input parameters must be passed via ``%eax``, ``8(%rsp)``,

``0x10(%rsp)`` and

arch/x86/entry/vdso/vsgx_enter_enclave.S line 37

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * ``0x18(%rsp)``, with the return value passed via ``%eax``. All

other

arch/x86/entry/vdso/vsgx_enter_enclave.S line 38

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * registers will be passed through to the enclave as is. All

registers except

arch/x86/entry/vdso/vsgx_enter_enclave.S line 39

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * ``%rbp`` must be treated as volatile from the caller's perspective,

including

Hmm, this is my fault since I wrote the original comment, but stating "All registers except %rbp must be treated as volatile" is confusing, e.g.
at first I thought it meant that the assembly blob was mucking with the registers and that they couldn't be used to pass information out of the enclave.

Maybe something like:

* Register ABI:

 *  - As noted above, input parameters are passed via %eax and the stack.
 *  - The return value is passed via %eax.
 *  - %rbx and %rcx must be treated as volatile as they are modified as

part of
* enclaves transitions and are used as scratch regs.
* - %rbp **must** be preserved by the enclave in all cases, as it is
used to
* reference paramaters when handling enclave exits.
* - %rdx, %rdi, %rsi and %r8-%r15 may be freely modified by the
enclave
* and will not be passed back to the caller untouched.
* - %rsp is saved/restored across __vdso_sgx_enter_enclave(), but is

Quoted 8 lines

passed
* as-is to the callback if one is provided, i.e. the enclave may use
u_rsp
* to pass information to its associated callback.
*
* Callback ABI:
* <more information about callback ABI>
*

Really appreciated! I'll update the comments in the next revision.

Quoted 25 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S line 40

Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * but not limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely,

the

arch/x86/entry/vdso/vsgx_enter_enclave.S

Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +7 -0
+ * enclave being run **must** preserve the untrusted ``%rbp``.
+ *
+ * ``callback`` has the following signature:
+ * int callback(long rdi, long rsi, long rdx,
+ *		struct sgx_enclave_exinfo *exinfo, long r8, long r9,
+ *		void *tcs, long ursp);
+ * ``callback`` **shall** follow x86_64 ABI. All GPRs **except**

``%rax``,

arch/x86/entry/vdso/vsgx_enter_enclave.S line 48

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * ``%rbx`` and ``rcx`` are passed through to ``callback``. ``%rdi``,

``%rsi``,

arch/x86/entry/vdso/vsgx_enter_enclave.S line 49

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * ``%rdx``, ``%r8``, ``%r9``, along with the value of ``%rsp`` when

the enclave

arch/x86/entry/vdso/vsgx_enter_enclave.S line 50

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * exited/excepted, can be accessed directly as input parameters,

while other

arch/x86/entry/vdso/vsgx_enter_enclave.S line 51

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * GPRs can be accessed in assembly if needed.  A positive value

returned from

arch/x86/entry/vdso/vsgx_enter_enclave.S line 52

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * ``callback`` will be treated as an ENCLU leaf (e.g. EENTER/ERESUME)

to

arch/x86/entry/vdso/vsgx_enter_enclave.S line 53

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * reenter the enclave (without popping the extra data pushed by the

enclave off

arch/x86/entry/vdso/vsgx_enter_enclave.S line 54

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+ * the stack), while 0 (zero) or a negative return value will be

passed back to

Quoted 9 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S line 57

Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2 -0
+ * the caller of __vdso_sgx_enter_enclave(). It is also safe to leave
+ * ``callback`` via ``longjmp()`` or by throwing a C++ exception.
  */

While this may format nicely in .rst (I haven't checked), it's damn near unreadable in its raw form. Escaping '%' in kernel-doc is unresolved at this point, but this definitely is an argument for allowing the %CONST interpretation to be disabled entirely.

I know very little about kernel-doc. Not quite sure which markup means what. Or if you don't mind, could you just write up the whole thing (you have written half of it in your email already) so I can include it into my next revision?

Quoted 4 lines

longjmp() is probably the only thing that needs to be explicitly marked, I think it would be better to simply say "the callback" instead of using ``callback``, i.e. use regular English instead of referencing the param.

Are you suggesting not to mention C++ exception here?

Quoted 19 lines
Quoted 19 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6 -12
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-			 struct sgx_enclave_exception *ex_info)
+typedef int (*sgx_callback)(long rdi, long rsi, long rdx,
+			    struct sgx_enclave_exinfo *exinfo, long r8,
+			    long r9, void *tcs, long ursp);
+int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+			     struct sgx_enclave_exinfo *exinfo,
+			     sgx_callback callback)
 {
-	if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-		return -EINVAL;
-
-	if (!tcs)
-		return -EINVAL;
-
-	try {
-		ENCLU[leaf];
-	} catch (exception) {
-		if (e)
Quoted 8 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S line 65

Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -2
-			*e = exception;
-		return -EFAULT;
+	while (leaf == EENTER || leaf == ERESUME) {

This gives the impression that looping is the common case. I'd prefer using 'goto' to show that the common case is a simple fall through whereas
the callback case can effectively loop on ENCLU.

Looping IMO is indeed the common case. Just think of the case of an enclave making a sequence of o-calls.

Quoted 61 lines
Quoted 48 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +29 -9
+		int rc;
+		try {
+			ENCLU[leaf];
+			rc = 0;
+			if (exinfo)
+				exinfo->leaf = EEXIT;
+		} catch (exception) {
+			rc = -EFAULT;
+			if (exinfo)
+				*exinfo = exception;
+		}
+
+		leaf = callback ? (*callback)(
+			rdi, rsi, rdx, exinfo, r8, r9, tcs, ursp) : rc;
 	}
 
-	return 0;
+	return leaf > 0 ? -EINVAL : leaf;
 }
 #endif
 ENTRY(__vdso_sgx_enter_enclave)
-	/* EENTER <= leaf <= ERESUME */
+	/* Prolog */
+	.cfi_startproc
+	push	%rbp
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbp, 0
+	mov	%rsp, %rbp
+	.cfi_def_cfa_register	%rbp
+
+1:	/* EENTER <= leaf <= ERESUME */
 	cmp	$0x2, %eax
-	jb	bad_input
-
+	jb	6f
 	cmp	$0x3, %eax
-	ja	bad_input
+	ja	6f
 
-	/* TCS must be non-NULL */
-	test	%rbx, %rbx
-	je	bad_input
+	/* Load TCS and AEP */
+	mov	0x10(%rbp), %rbx
+	lea	2f(%rip), %rcx
 
-	/* Save @exception_info */

arch/x86/entry/vdso/vsgx_enter_enclave.S line 109

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5 -4
-	push	%rcx
+	/* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:	enclu
 
-	/* Load AEP for ENCLU */
-	lea	1f(%rip),  %rcx
-1:	enclu
+	/* EEXIT path */
+	xor	%ebx, %ebx
+3:	mov	0x18(%rbp), %rcx

@exinfo is optional, i.e. %ecx needs to be tested before its used.

Quoted 1 line

arch/x86/entry/vdso/vsgx_enter_enclave.S line 110

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+	jrcxz	4f

The above instruction (i.e. jrcxz) does test %rcx.

Quoted 6 lines

I dislike the number of flags and values that are stashed away only to be consumed later, it makes the code hard to follow. AFAICT, a lot of the shenanigans are done to reuse code because exinfo was overloaded, but that's actually pointless, i.e. it's unnecessarily complex.

Quoted 7 lines

Overlading the struct is pointless becase if you make it mandatory when using a callback then it can be nullified (when passed to the callback) to indicate EEXIT. If it's still optional, then the callback needs an extra param to explicitly indicate EEXIT vs. -EFAULT, otherwise the callback has no indication whatsover of why it was invoked.

My preference is for the latter since it's more explicit and obvious.

I wouldn't nullify @exinfo at EEXIT because that could be used as a "context" pointer to the callback.

Making it optionally because the callback can still use other registers (e.g. %rdi) determine the reason without it. For example, an enclave may set %rdi to non-zero to signify o-call/e-ret while zero (as set by AEX) would indicate an exception. My intention is not to impose unnecessary restrictions on applications.

Quoted 3 lines

Tangetially related, I'm not opposed to renaming it 'struct sgx_enclave_exception_info' for clarity.

I'd still prefer sgx_enclave_exinfo to imply its use in both _ex_it and _ex_ception cases, for the reason stated above.

Quoted 12 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S line 115

Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5 -0
+	mov	%eax, EX_LEAF(%rcx)
+	jnc	4f
+	mov	%di, EX_TRAPNR(%rcx)
+	mov	%si, EX_ERROR_CODE(%rcx)
+	mov	%rdx, EX_ADDRESS(%rcx)

My strong preference would be to organize the code to separate the various
flows, i.e. happy common case, invalid input, exception handler and callback invocation. And make the common case a straight shot so that it's more obvious what happens in the happy non-callback case.

I'd prefer more concise code. After all, this is assembly without optimization done by the compiler. If it were for C, I'd agree with you because either case would end up in roughly the same code (in terms of code size or efficiency) due to the optimizer.

Quoted 19 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S

Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4 -2
-	add	$0x8, %rsp
-	xor	%eax, %eax
+4:	/* Call *callback if supplied */
+	mov	0x20(%rbp), %rax
+	test	%rax, %rax
+	/* At this point, %ebx holds the effective return value, which

shall be

arch/x86/entry/vdso/vsgx_enter_enclave.S line 121

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+	 * returned if no callback is specified */

Use kernel-doc format for multi-line comments. Missing punctionation at the end. Blank lines between logical blocks would also help readability.

E.g.:

4: /* Call *callback if supplied */
mov 0x20(%rbp), %rax
test %rax, %rax

Quoted 7 lines

/*
* At this point, %ebx holds the effective return value, which
shall be
* returned if no callback is specified.
*/
cmovz %rbx, %rax
jz 7f

I know very little about kernel-doc. Would you please point me to documents describing it? What you suggested here will of course be incorporated into my next revision.

Quoted 4 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S

Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +3 -0
+	cmovz	%rbx, %rax
+	jz	7f
+	/* Align stack per x86_64 ABI. The original %rsp is saved in %rbx

to be

Quoted 15 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S line 134

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +10 -0
+	 * restored after *callback returns. */
+	mov	%rsp, %rbx
+	and	$-0x10, %rsp
+	/* Clear RFLAGS.DF per x86_64 ABI */
+	cld
+	/* Parameters for *callback */
+	push	%rbx
+	push	0x10(%rbp)
+	/* Call *%rax via retpoline */
+	call	40f

Another case of stashing a value it consuming it later. This one is especially brutal since %rax was loaded with CMOVcc, which means the reader needs to backtrack even further to understand what %rax contains at this point.

%rax was loaded only 2 instructions ago, while what %rbx is containing is described in the comment immediately above it. I think I have done my best to make it readable. After all, this is assembly so can't expect something like giving proper names to variables in C because that just can't be done in assembly.

Quoted 7 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S line 134

Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+	/* Restore %rsp to its original value left off by the enclave from

last

arch/x86/entry/vdso/vsgx_enter_enclave.S

Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +3 -0
+	 * exit */
+	mov	%rbx, %rsp
+	/* Positive return value from *callback will be interpreted as an

ENCLU

Quoted 7 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S line 138

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+	 * leaf, while a non-positive value will be interpreted as the

return

arch/x86/entry/vdso/vsgx_enter_enclave.S line 140

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+	 * value to be passed back to the caller. */

Please use imperative mood in the comments, i.e. simply state what the code is doing. E.g. "will be interpreted" makes it sound like something else is processing the callback's return value.

Will do in the next revision.

Quoted 49 lines
Quoted 33 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +16 -13
+	jmp	1b
+40:	/* retpoline */
+	call	42f
+41:	pause
+	lfence
+	jmp	41b
+42:	mov	%rax, (%rsp)
 	ret
 
-bad_input:
-	mov     $(-EINVAL), %rax
-	ret
+5:	/* Exception path */
+	mov	$-EFAULT, %ebx
+	stc
+	jmp	3b
 
-.pushsection .fixup, "ax"
-	/* Re-load @exception_info and fill it (if it's non-NULL) */
-2:	pop	%rcx
-	test    %rcx, %rcx
-	je      3f
+6:	/* Unsupported ENCLU leaf */
+	cmp	$0, %eax
+	jle	7f
+	mov	$-EINVAL, %eax
 
-	mov	%eax, EX_LEAF(%rcx)
-	mov	%di,  EX_TRAPNR(%rcx)
-	mov	%si,  EX_ERROR_CODE(%rcx)
-	mov	%rdx, EX_ADDRESS(%rcx)
-3:	mov	$(-EFAULT), %rax
+7:	/* Epilog */

arch/x86/entry/vdso/vsgx_enter_enclave.S line 168

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4 -2
+	leave
+	.cfi_def_cfa		%rsp, 8
 	ret
-.popsection
+	.cfi_endproc
 
-_ASM_VDSO_EXTABLE_HANDLE(1b, 2b)
+_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)
 
 ENDPROC(__vdso_sgx_enter_enclave)

Putting everything together, sans comments, I'd prefer something like the
following. Pasted in raw code as that'll hopefully be easier to review and discuss than a diff.

Quoted 3 lines

Note, swapping 'long rc' and '... *e' would allow the callback flow to save one memory access, but IMO the exception info belongs at the end since it's optional.

You have probably misunderstood my code. 'ret' is _not_ passed while @exinfo is passed in %rcx. exinfo->leaf implies the reason of the callback.

Or are you suggesting to add 'ret' as one more parameter to the callback? I don't think it necessary because exinfo contains everything the callback would ever need.

Quoted 128 lines

#ifdef SGX_KERNEL_DOC
typedef int (*sgx_callback)(long rdi, long rsi, long rdx, long ret, long r8,

			    long r9, void *tcs, long ursp,
			    struct sgx_enclave_exception_info *e);
int __vdso_sgx_enter_enclave(int leaf, void *tcs,
				struct sgx_enclave_exception_info *e,
				sgx_callback callback)
{
	int ret;

enter_enclave:

	if (leaf != EENTER && leaf != ERESUME) {
		ret = -EINVAL;
		goto out;
	}

	try {
		ENCLU[leaf];
		ret = 0;
	} catch (exception) {
		ret = -EFAULT;
		if (e)
			*e = exception;
	}
	if (callback)
		goto do_callback;
out:
	return ret;

do_callback:

	leaf = (*callback)(rdi, rsi, rdx, ret, r8, r9, e, tcs, ursp);
	if (leaf <= 0)
		goto out;
	goto enter_enclave;
}
#endif
ENTRY(__vdso_sgx_enter_enclave)
	/* Prolog */
	.cfi_startproc
	push	%rbp
	.cfi_adjust_cfa_offset	8
	.cfi_rel_offset		%rbp, 0
	mov	%rsp, %rbp
	.cfi_def_cfa_register	%rbp

1: /* EENTER <= leaf <= ERESUME */

	cmp	$0x2, %eax
	jb	5f
	cmp	$0x3, %eax
	ja	5f

	/* Load TCS and AEP */
	mov	0x10(%rbp), %rbx
	lea	2f(%rip), %rcx

	/* Single ENCLU serving as both EENTER and AEP (ERESUME) */

2: enclu

	/* EEXIT branches here unless the enclave is doing something fancy.
*/
	xor	%eax, %eax

3: cmp $0, 0x20(%rbp)

	jne	8f

4: leave

	.cfi_def_cfa		%rsp, 8
	ret

5: mov $(-EINVAL), %rax

	jmp	4b

6: mov 0x18(%rbp), %rcx

	test    %rcx, %rcx
	je	7f

	/* Fill optional exception info. */
	mov	%eax, EX_LEAF(%rcx)
	mov	%di,  EX_TRAPNR(%rcx)
	mov	%si,  EX_ERROR_CODE(%rcx)
	mov	%rdx, EX_ADDRESS(%rcx)
7:	mov	$(-EFAULT), %rax
	jmp	3b

8: /*
* Align stack per x86_64 ABI. Save the original %rsp in %rbx to
be

	 * restored after the callback returns.
	 */
	mov	%rsp, %rbx
	and	$-0x10, %rsp

	/* Push @e, u_rsp and @tcs as parameters to the callback. */
	push	0x18(%rbp)
	push	%rbx
	push	0x10(%rbp)

	/* Pass the "return value" to the callback via %rcx. */
	mov	%rax, %rcx

	/* Clear RFLAGS.DF per x86_64 ABI */
	cld

	/* Load the callback pointer to %rax and invoke it via retpoline.
*/
	mov	0x20(%rbp), %rax
	call	40f

	/* Restore %rsp to its post-exit value. */
	mov	%rbx, %rsp

	/*
	 * If the return from callback is zero or negative, return
immediately,
	 * otherwise attempt to re-execute ENCLU with the return
interpreted as
	 * the requested ENCLU leaf.
	 */
	cmp	$0, %eax
	jle	4b
	jmp	1b

40: /* retpoline */

	call	42f
41:	pause
	lfence
	jmp	41b
Quoted 7 lines

42: mov %rax, (%rsp)
ret
.cfi_endproc

_ASM_VDSO_EXTABLE_HANDLE(2b, 6b)

ENDPROC(__vdso_sgx_enter_enclave)

Your code will work. The only missing thing is a ".cfi_def_cfa %rbp, 16" after 'ret' instruction in block 4. I decided to keep "leave; ret" at the end to avoid confusion around the frame pointer. As you may also notice, GCC generated code usually does the same thing.

-Cedric

#267 lore raw

Re: [RFC PATCH v2 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 10 lines · Cedric
Quoted 5 lines

While this may format nicely in .rst (I haven't checked), it's damn near unreadable in its raw form. Escaping '%' in kernel-doc is unresolved at this point, but this definitely is an argument for allowing the %CONST interpretation to be disabled entirely.

I know very little about kernel-doc. Not quite sure which markup means what. Or if you don't mind, could you just write up the whole thing (you have written half of it in your email already) so I can include it into my next revision?

Hmm, at this point I'd say just ignore the comment entirely. It's something we need to sort out in the "official" series anyways.

Quoted 7 lines
Quoted 5 lines

longjmp() is probably the only thing that needs to be explicitly marked, I think it would be better to simply say "the callback" instead of using ``callback``, i.e. use regular English instead of referencing the param.

Are you suggesting not to mention C++ exception here?

No, I was saying that rather than do the backtick markup on callback, just say "the callback". The copious amount of markup makes it difficult to read the raw text.

Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6 -12
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-			 struct sgx_enclave_exception *ex_info)
+typedef int (*sgx_callback)(long rdi, long rsi, long rdx,
+			    struct sgx_enclave_exinfo *exinfo, long r8,
+			    long r9, void *tcs, long ursp);
+int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+			     struct sgx_enclave_exinfo *exinfo,
+			     sgx_callback callback)
 {
-	if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-		return -EINVAL;
-
-	if (!tcs)
-		return -EINVAL;
-
-	try {
-		ENCLU[leaf];
-	} catch (exception) {
-		if (e)
Quoted 8 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S line 65

Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -2
-			*e = exception;
-		return -EFAULT;
+	while (leaf == EENTER || leaf == ERESUME) {

This gives the impression that looping is the common case. I'd prefer using 'goto' to show that the common case is a simple fall through whereas
the callback case can effectively loop on ENCLU.

Quoted 2 lines

Looping IMO is indeed the common case. Just think of the case of an enclave making a sequence of o-calls.

Except that looping is only done when using a callback, and that most definitely is not the common case.

Quoted 60 lines
Quoted 60 lines
Quoted 47 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +29 -9
+		int rc;
+		try {
+			ENCLU[leaf];
+			rc = 0;
+			if (exinfo)
+				exinfo->leaf = EEXIT;
+		} catch (exception) {
+			rc = -EFAULT;
+			if (exinfo)
+				*exinfo = exception;
+		}
+
+		leaf = callback ? (*callback)(
+			rdi, rsi, rdx, exinfo, r8, r9, tcs, ursp) : rc;
 	}
 
-	return 0;
+	return leaf > 0 ? -EINVAL : leaf;
 }
 #endif
 ENTRY(__vdso_sgx_enter_enclave)
-	/* EENTER <= leaf <= ERESUME */
+	/* Prolog */
+	.cfi_startproc
+	push	%rbp
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbp, 0
+	mov	%rsp, %rbp
+	.cfi_def_cfa_register	%rbp
+
+1:	/* EENTER <= leaf <= ERESUME */
 	cmp	$0x2, %eax
-	jb	bad_input
-
+	jb	6f
 	cmp	$0x3, %eax
-	ja	bad_input
+	ja	6f
 
-	/* TCS must be non-NULL */
-	test	%rbx, %rbx
-	je	bad_input
+	/* Load TCS and AEP */
+	mov	0x10(%rbp), %rbx
+	lea	2f(%rip), %rcx
 
-	/* Save @exception_info */

arch/x86/entry/vdso/vsgx_enter_enclave.S line 109

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5 -4
-	push	%rcx
+	/* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:	enclu
 
-	/* Load AEP for ENCLU */
-	lea	1f(%rip),  %rcx
-1:	enclu
+	/* EEXIT path */
+	xor	%ebx, %ebx
+3:	mov	0x18(%rbp), %rcx

@exinfo is optional, i.e. %ecx needs to be tested before its used.

Quoted 3 lines
Quoted 1 line

arch/x86/entry/vdso/vsgx_enter_enclave.S line 110

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+	jrcxz	4f

The above instruction (i.e. jrcxz) does test %rcx.

I love x86 ISA.

Quoted 21 lines
Quoted 5 lines

I dislike the number of flags and values that are stashed away only to be consumed later, it makes the code hard to follow. AFAICT, a lot of the shenanigans are done to reuse code because exinfo was overloaded, but that's actually pointless, i.e. it's unnecessarily complex.

Quoted 7 lines

Overlading the struct is pointless becase if you make it mandatory when using a callback then it can be nullified (when passed to the callback) to indicate EEXIT. If it's still optional, then the callback needs an extra param to explicitly indicate EEXIT vs. -EFAULT, otherwise the callback has no indication whatsover of why it was invoked.

My preference is for the latter since it's more explicit and obvious.

I wouldn't nullify @exinfo at EEXIT because that could be used as a "context" pointer to the callback.

Making it optionally because the callback can still use other registers (e.g. %rdi) determine the reason without it. For example, an enclave may set %rdi to non-zero to signify o-call/e-ret while zero (as set by AEX) would indicate an exception. My intention is not to impose unnecessary restrictions on applications.

Quoted 5 lines
Quoted 2 lines

Tangetially related, I'm not opposed to renaming it 'struct sgx_enclave_exception_info' for clarity.

I'd still prefer sgx_enclave_exinfo to imply its use in both _ex_it and _ex_ception cases, for the reason stated above.

Because an enclave *could* choose a different route? The cost is essentially one memory access, which is likely offset by the fact that the callback can directly check %rcx for the exit reason. You are not the sole consumer of this code.

Quoted 12 lines
Quoted 11 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S line 115

Quoted 5 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5 -0
+	mov	%eax, EX_LEAF(%rcx)
+	jnc	4f
+	mov	%di, EX_TRAPNR(%rcx)
+	mov	%si, EX_ERROR_CODE(%rcx)
+	mov	%rdx, EX_ADDRESS(%rcx)

My strong preference would be to organize the code to separate the various
flows, i.e. happy common case, invalid input, exception handler and callback invocation. And make the common case a straight shot so that it's more obvious what happens in the happy non-callback case.

Quoted 4 lines

I'd prefer more concise code. After all, this is assembly without optimization done by the compiler. If it were for C, I'd agree with you because either case would end up in roughly the same code (in terms of code size or efficiency) due to the optimizer.

a) You've argued multiple times this isn't performance critical code.

b) A straight shot is optimal for the common case where userspace is not using a callback.

c) The performance difference is something like one or two memory accesses. That's peanuts compared to how readable the code is by people without prior knowledge of exactly what this code is doing.

Quoted 26 lines
Quoted 18 lines

arch/x86/entry/vdso/vsgx_enter_enclave.S

Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4 -2
-	add	$0x8, %rsp
-	xor	%eax, %eax
+4:	/* Call *callback if supplied */
+	mov	0x20(%rbp), %rax
+	test	%rax, %rax
+	/* At this point, %ebx holds the effective return value, which

shall be

arch/x86/entry/vdso/vsgx_enter_enclave.S line 121

Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1 -0
+	 * returned if no callback is specified */

Use kernel-doc format for multi-line comments. Missing punctionation at the end. Blank lines between logical blocks would also help readability.

E.g.:

4: /* Call *callback if supplied */
mov 0x20(%rbp), %rax
test %rax, %rax

Quoted 7 lines

/*
* At this point, %ebx holds the effective return value, which
shall be
* returned if no callback is specified.
*/
cmovz %rbx, %rax
jz 7f

Quoted 3 lines

I know very little about kernel-doc. Would you please point me to documents describing it? What you suggested here will of course be incorporated into my next revision.

Documentation/doc-guide/kernel-doc.rst

[...]

Quoted 10 lines
Quoted 3 lines

Note, swapping 'long rc' and '... *e' would allow the callback flow to save one memory access, but IMO the exception info belongs at the end since it's optional.

You have probably misunderstood my code. 'ret' is _not_ passed while @exinfo is passed in %rcx. exinfo->leaf implies the reason of the callback.

Or are you suggesting to add 'ret' as one more parameter to the callback? I don't think it necessary because exinfo contains everything the callback would ever need.

The latter, because exinfo is optional, as stated above.

#268 lore raw

Re: [RFC PATCH v2 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 7 lines · Sean Christopherson
Quoted 6 lines · Cedric
Quoted 5 lines

While this may format nicely in .rst (I haven't checked), it's damn near unreadable in its raw form. Escaping '%' in kernel-doc is unresolved at this point, but this definitely is an argument for allowing the %CONST interpretation to be disabled entirely.

Quoted 7 lines
Quoted 4 lines

I know very little about kernel-doc. Not quite sure which markup means what. Or if you don't mind, could you just write up the whole thing (you have written half of it in your email already) so I can include it into my next revision?

Hmm, at this point I'd say just ignore the comment entirely. It's something we need to sort out in the "official" series anyways.

To get started: https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt

/Jarkko

#269 lore raw

RFC PATCH v2 3/3 selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

In reply to Cedric Xing

This patch augments SGX selftest with two new tests.

The first test exercises the newly added callback interface, by marking the whole enclave range as PROT_READ, then calling mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes. This test also serves as an example to demonstrate the callback interface.

The second test single-steps through __vdso_sgx_enter_enclave() to make sure the call stack can be unwound at every instruction within that vDSO API. Its purpose is to validate the hand-crafted CFI directives in the assembly.

---
 tools/testing/selftests/x86/sgx/Makefile   |   6 +-
 tools/testing/selftests/x86/sgx/main.c     | 323 ++++++++++++++++++---
 tools/testing/selftests/x86/sgx/sgx_call.S |  40 ++-
 3 files changed, 322 insertions(+), 47 deletions(-)
tools/testing/selftests/x86/sgx/Makefile +3 -3
index 3af15d7c8644..31f937e220c4 100644
@@ -14,16 +14,16 @@ TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx
 all_64: $(TEST_CUSTOM_PROGS)
 
 $(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o
-	$(CC) $(HOST_CFLAGS) -o $@ $^
+	$(CC) $(HOST_CFLAGS) -o $@ $^ -lunwind -ldl -Wl,--defsym,__image_base=0 -pie
 
 $(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
 	$(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@
 
 $(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf
-	objcopy --remove-section=.got.plt -O binary $< $@
+	objcopy -O binary $< $@
 
 $(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S
-	$(CC) $(ENCL_CFLAGS) -T $^ -o $@
+	$(CC) $(ENCL_CFLAGS) -T $^ -o $@ -Wl,--build-id=none
 
 $(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin
 	$^ $@
tools/testing/selftests/x86/sgx/main.c +282 -41

388 lines not on this page · raw message

tools/testing/selftests/x86/sgx/sgx_call.S +37 -3
index 14bd0a044199..ca2c0c947758 100644
@@ -7,9 +7,43 @@
 
 	.global sgx_call
 sgx_call:
+	.cfi_startproc
+	push	%r15
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r15, 0
+	push	%r14
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r14, 0
+	push	%r13
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r13, 0
+	push	%r12
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r12, 0
 	push	%rbx
-	mov	$0x02, %rax
-	mov	%rdx, %rbx
-	call	*%r8
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbx, 0
+	push	$0
+	.cfi_adjust_cfa_offset	8
+	push	0x48(%rsp)
+	.cfi_adjust_cfa_offset	8
+	push	0x48(%rsp)
+	.cfi_adjust_cfa_offset	8
+	push	0x48(%rsp)
+	.cfi_adjust_cfa_offset	8
+	mov	$2, %eax
+	call	*eenter(%rip)
+	add	$0x20, %rsp
+	.cfi_adjust_cfa_offset	-0x20
 	pop	%rbx
+	.cfi_adjust_cfa_offset	-8
+	pop	%r12
+	.cfi_adjust_cfa_offset	-8
+	pop	%r13
+	.cfi_adjust_cfa_offset	-8
+	pop	%r14
+	.cfi_adjust_cfa_offset	-8
+	pop	%r15
+	.cfi_adjust_cfa_offset	-8
 	ret
+	.cfi_endproc
Signature · 1 line
2.17.1
#270 lore raw

Re: [RFC PATCH v2 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

Quoted 101 lines · Cedric Xing

This patch augments SGX selftest with two new tests.

The first test exercises the newly added callback interface, by marking the whole enclave range as PROT_READ, then calling mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes. This test also serves as an example to demonstrate the callback interface.

The second test single-steps through __vdso_sgx_enter_enclave() to make sure the call stack can be unwound at every instruction within that vDSO API. Its purpose is to validate the hand-crafted CFI directives in the assembly.

---
 tools/testing/selftests/x86/sgx/Makefile   |   6 +-
 tools/testing/selftests/x86/sgx/main.c     | 323 ++++++++++++++++++---
 tools/testing/selftests/x86/sgx/sgx_call.S |  40 ++-
 3 files changed, 322 insertions(+), 47 deletions(-)
tools/testing/selftests/x86/sgx/Makefile +3 -3
index 3af15d7c8644..31f937e220c4 100644
@@ -14,16 +14,16 @@ TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx
 all_64: $(TEST_CUSTOM_PROGS)
 
 $(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o
-	$(CC) $(HOST_CFLAGS) -o $@ $^
+	$(CC) $(HOST_CFLAGS) -o $@ $^ -lunwind -ldl -Wl,--defsym,__image_base=0 -pie
 
 $(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
 	$(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@
 
 $(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf
-	objcopy --remove-section=.got.plt -O binary $< $@
+	objcopy -O binary $< $@
 
 $(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S
-	$(CC) $(ENCL_CFLAGS) -T $^ -o $@
+	$(CC) $(ENCL_CFLAGS) -T $^ -o $@ -Wl,--build-id=none
 
 $(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin
 	$^ $@
tools/testing/selftests/x86/sgx/main.c +19 -9
index e2265f841fb0..d3e53c71306d 100644
@@ -1,6 +1,7 @@
 // SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
 // Copyright(c) 2016-18 Intel Corporation.
 
+#define _GNU_SOURCE
 #include <elf.h>
 #include <fcntl.h>
 #include <stdbool.h>
@@ -9,16 +10,31 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <errno.h>
 #include <sys/ioctl.h>
 #include <sys/mman.h>
 #include <sys/stat.h>
-#include <sys/time.h>
+#include <sys/auxv.h>
+#include <signal.h>
+#include <sys/ucontext.h>
+
+#define UNW_LOCAL_ONLY
+#include <libunwind.h>
+
 #include "encl_piggy.h"
 #include "defines.h"
 #include "../../../../../arch/x86/kernel/cpu/sgx/arch.h"
 #include "../../../../../arch/x86/include/uapi/asm/sgx.h"
 
-static const uint64_t MAGIC = 0x1122334455667788ULL;
+#define _Q(x)	__Q(x)
+#define __Q(x)	#x
+#define ERRLN	"Line " _Q(__LINE__)
+
+#define X86_EFLAGS_TF	(1ul << 8)
+
+extern char __image_base[];
+size_t eenter;
+static size_t vdso_base;
 
 struct vdso_symtab {
 	Elf64_Sym *elf_symtab;
@@ -26,20 +42,11 @@ struct vdso_symtab {
 	Elf64_Word *elf_hashtab;
 };
 
-static void *vdso_get_base_addr(char *envp[])
+static void vdso_init(void)
 {
-	Elf64_auxv_t *auxv;
-	int i;
-
-	for (i = 0; envp[i]; i++);
-	auxv = (Elf64_auxv_t *)&envp[i + 1];
-

tools/testing/selftests/x86/sgx/main.c struct vdso_symtab { · line 50

Quoted 10 lines
tools/testing/selftests/x86/sgx/main.c +3 -6
-	for (i = 0; auxv[i].a_type != AT_NULL; i++) {
-		if (auxv[i].a_type == AT_SYSINFO_EHDR)
-			return (void *)auxv[i].a_un.a_val;
-	}
-
-	return NULL;
+	vdso_base = getauxval(AT_SYSINFO_EHDR);
+	if (!vdso_base)
+		exit(1);
 }

The clean up makes sense but should be a separate patch i.e. one logical change per patch. Right now the patch does other mods than the ones explcitly stated in the commit message.

I'd suggest open coding vdso_init() to the call site in that patch.

Please try to always minimize for diff's.

Quoted 81 lines
tools/testing/selftests/x86/sgx/main.c +55 -5
 static Elf64_Dyn *vdso_get_dyntab(void *addr)
@@ -66,8 +73,9 @@ static void *vdso_get_dyn(void *addr, Elf64_Dyn *dyntab, Elf64_Sxword tag)
 	return NULL;
 }
 
-static bool vdso_get_symtab(void *addr, struct vdso_symtab *symtab)
+static bool vdso_get_symtab(struct vdso_symtab *symtab)
 {
+	void *addr = (void *)vdso_base;
 	Elf64_Dyn *dyntab = vdso_get_dyntab(addr);
 
 	symtab->elf_symtab = vdso_get_dyn(addr, dyntab, DT_SYMTAB);
@@ -138,7 +146,7 @@ static bool encl_create(int dev_fd, unsigned long bin_size,
 	base = mmap(NULL, secs->size, PROT_READ | PROT_WRITE | PROT_EXEC,
 		    MAP_SHARED, dev_fd, 0);
 	if (base == MAP_FAILED) {
-		perror("mmap");
+		perror(ERRLN);
 		return false;
 	}
 
@@ -224,35 +232,271 @@ static bool encl_load(struct sgx_secs *secs, unsigned long bin_size)
 	return false;
 }
 
-void sgx_call(void *rdi, void *rsi, void *tcs,
-	      struct sgx_enclave_exception *exception,
-	      void *eenter);
+int sgx_call(void *rdi, void *rsi, long rdx, void *rcx, void *r8, void *r9,
+	     void *tcs, struct sgx_enclave_exinfo *ei, void *cb);
+
+static void show_enclave_exinfo(const struct sgx_enclave_exinfo *exinfop,
+				const char *header)
+{
+	static const char * const enclu_leaves[] = {
+		"EREPORT",
+		"EGETKEY",
+		"EENTER",
+		"ERESUME",
+		"EEXIT"
+	};
+	static const char * const exception_names[] = {
+		"#DE",
+		"#DB",
+		"NMI",
+		"#BP",
+		"#OF",
+		"#BR",
+		"#UD",
+		"#NM",
+		"#DF",
+		"CSO",
+		"#TS",
+		"#NP",
+		"#SS",
+		"#GP",
+		"#PF",
+		"Unknown",
+		"#MF",
+		"#AC",
+		"#MC",
+		"#XM",
+		"#VE",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown"
+	};
+
+	printf("%s: leaf:%s(%d)", header,
+		enclu_leaves[exinfop->leaf], exinfop->leaf);
+	if (exinfop->leaf != 4)
+		printf(" trap:%s(%d) ec:%d addr:0x%llx\n",

tools/testing/selftests/x86/sgx/main.c static bool encl_load(struct sgx_secs *secs, unsigned long bin_size) · line 295

Quoted 10 lines
tools/testing/selftests/x86/sgx/main.c +8 -1
+			exception_names[exinfop->trapnr], exinfop->trapnr,
+			exinfop->error_code, exinfop->address);
+	else
+		printf("\n");
+}
+
+static const uint64_t MAGIC = 0x1122334455667788ULL;
 
-int main(int argc, char *argv[], char *envp[])
+static void test1(struct sgx_secs *secs)

test1, test2 and test3 are not too descriptive names. Every patch should make the code base cleaner, not messier.

/Jarkko

#271 lore raw

Re: [RFC PATCH v2 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

Quoted 211 lines · Jarkko Sakkinen
Quoted 101 lines · Cedric Xing

This patch augments SGX selftest with two new tests.

The first test exercises the newly added callback interface, by marking the whole enclave range as PROT_READ, then calling mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes. This test also serves as an example to demonstrate the callback interface.

The second test single-steps through __vdso_sgx_enter_enclave() to make sure the call stack can be unwound at every instruction within that vDSO API. Its purpose is to validate the hand-crafted CFI directives in the assembly.

---
  tools/testing/selftests/x86/sgx/Makefile   |   6 +-
  tools/testing/selftests/x86/sgx/main.c     | 323 ++++++++++++++++++---
  tools/testing/selftests/x86/sgx/sgx_call.S |  40 ++-
  3 files changed, 322 insertions(+), 47 deletions(-)
tools/testing/selftests/x86/sgx/Makefile +3 -3
index 3af15d7c8644..31f937e220c4 100644
@@ -14,16 +14,16 @@ TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx
  all_64: $(TEST_CUSTOM_PROGS)
 
  $(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o
-	$(CC) $(HOST_CFLAGS) -o $@ $^
+	$(CC) $(HOST_CFLAGS) -o $@ $^ -lunwind -ldl -Wl,--defsym,__image_base=0 -pie
 
  $(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
  	$(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@
 
  $(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf
-	objcopy --remove-section=.got.plt -O binary $< $@
+	objcopy -O binary $< $@
 
  $(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S
-	$(CC) $(ENCL_CFLAGS) -T $^ -o $@
+	$(CC) $(ENCL_CFLAGS) -T $^ -o $@ -Wl,--build-id=none
 
  $(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin
  	$^ $@
tools/testing/selftests/x86/sgx/main.c +19 -9
index e2265f841fb0..d3e53c71306d 100644
@@ -1,6 +1,7 @@
  // SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause)
  // Copyright(c) 2016-18 Intel Corporation.
 
+#define _GNU_SOURCE
  #include <elf.h>
  #include <fcntl.h>
  #include <stdbool.h>
@@ -9,16 +10,31 @@
  #include <stdlib.h>
  #include <string.h>
  #include <unistd.h>
+#include <errno.h>
  #include <sys/ioctl.h>
  #include <sys/mman.h>
  #include <sys/stat.h>
-#include <sys/time.h>
+#include <sys/auxv.h>
+#include <signal.h>
+#include <sys/ucontext.h>
+
+#define UNW_LOCAL_ONLY
+#include <libunwind.h>
+
  #include "encl_piggy.h"
  #include "defines.h"
  #include "../../../../../arch/x86/kernel/cpu/sgx/arch.h"
  #include "../../../../../arch/x86/include/uapi/asm/sgx.h"
 
-static const uint64_t MAGIC = 0x1122334455667788ULL;
+#define _Q(x)	__Q(x)
+#define __Q(x)	#x
+#define ERRLN	"Line " _Q(__LINE__)
+
+#define X86_EFLAGS_TF	(1ul << 8)
+
+extern char __image_base[];
+size_t eenter;
+static size_t vdso_base;
 
  struct vdso_symtab {
  	Elf64_Sym *elf_symtab;
@@ -26,20 +42,11 @@ struct vdso_symtab {
  	Elf64_Word *elf_hashtab;
  };
 
-static void *vdso_get_base_addr(char *envp[])
+static void vdso_init(void)
  {
-	Elf64_auxv_t *auxv;
-	int i;
-
-	for (i = 0; envp[i]; i++);
-	auxv = (Elf64_auxv_t *)&envp[i + 1];
-

tools/testing/selftests/x86/sgx/main.c struct vdso_symtab { · line 50

Quoted 10 lines
tools/testing/selftests/x86/sgx/main.c +3 -6
-	for (i = 0; auxv[i].a_type != AT_NULL; i++) {
-		if (auxv[i].a_type == AT_SYSINFO_EHDR)
-			return (void *)auxv[i].a_un.a_val;
-	}
-
-	return NULL;
+	vdso_base = getauxval(AT_SYSINFO_EHDR);
+	if (!vdso_base)
+		exit(1);
  }

The clean up makes sense but should be a separate patch i.e. one logical change per patch. Right now the patch does other mods than the ones explcitly stated in the commit message.

I'd suggest open coding vdso_init() to the call site in that patch.

Please try to always minimize for diff's.

Quoted 89 lines
tools/testing/selftests/x86/sgx/main.c +62 -5
  static Elf64_Dyn *vdso_get_dyntab(void *addr)
@@ -66,8 +73,9 @@ static void *vdso_get_dyn(void *addr, Elf64_Dyn *dyntab, Elf64_Sxword tag)
  	return NULL;
  }
 
-static bool vdso_get_symtab(void *addr, struct vdso_symtab *symtab)
+static bool vdso_get_symtab(struct vdso_symtab *symtab)
  {
+	void *addr = (void *)vdso_base;
  	Elf64_Dyn *dyntab = vdso_get_dyntab(addr);
 
  	symtab->elf_symtab = vdso_get_dyn(addr, dyntab, DT_SYMTAB);
@@ -138,7 +146,7 @@ static bool encl_create(int dev_fd, unsigned long bin_size,
  	base = mmap(NULL, secs->size, PROT_READ | PROT_WRITE | PROT_EXEC,
  		    MAP_SHARED, dev_fd, 0);
  	if (base == MAP_FAILED) {
-		perror("mmap");
+		perror(ERRLN);
  		return false;
  	}
 
@@ -224,35 +232,271 @@ static bool encl_load(struct sgx_secs *secs, unsigned long bin_size)
  	return false;
  }
 
-void sgx_call(void *rdi, void *rsi, void *tcs,
-	      struct sgx_enclave_exception *exception,
-	      void *eenter);
+int sgx_call(void *rdi, void *rsi, long rdx, void *rcx, void *r8, void *r9,
+	     void *tcs, struct sgx_enclave_exinfo *ei, void *cb);
+
+static void show_enclave_exinfo(const struct sgx_enclave_exinfo *exinfop,
+				const char *header)
+{
+	static const char * const enclu_leaves[] = {
+		"EREPORT",
+		"EGETKEY",
+		"EENTER",
+		"ERESUME",
+		"EEXIT"
+	};
+	static const char * const exception_names[] = {
+		"#DE",
+		"#DB",
+		"NMI",
+		"#BP",
+		"#OF",
+		"#BR",
+		"#UD",
+		"#NM",
+		"#DF",
+		"CSO",
+		"#TS",
+		"#NP",
+		"#SS",
+		"#GP",
+		"#PF",
+		"Unknown",
+		"#MF",
+		"#AC",
+		"#MC",
+		"#XM",
+		"#VE",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown",
+		"Unknown"
+	};
+
+	printf("%s: leaf:%s(%d)", header,
+		enclu_leaves[exinfop->leaf], exinfop->leaf);
+	if (exinfop->leaf != 4)
+		printf(" trap:%s(%d) ec:%d addr:0x%llx\n",
+			exception_names[exinfop->trapnr], exinfop->trapnr,
+			exinfop->error_code, exinfop->address);
+	else
+		printf("\n");
+}
+
+static const uint64_t MAGIC = 0x1122334455667788ULL;
 
Quoted 5 lines

tools/testing/selftests/x86/sgx/main.c static bool encl_load(struct sgx_secs *secs, unsigned long bin_size) · line 295

Quoted 2 lines
tools/testing/selftests/x86/sgx/main.c +1 -1
-int main(int argc, char *argv[], char *envp[])
+static void test1(struct sgx_secs *secs)

test1, test2 and test3 are not too descriptive names. Every patch should make the code base cleaner, not messier.

I don't think it that important, as there are many test## occurrences in existing selftests. Anyway, I've added comments to those test functions to brief what is done. Hope you'll find them helpful.

Quoted 2 lines

/Jarkko

#272 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

In reply to Cedric Xing
Quoted 19 lines · Cedric Xing

The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.

This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new __vdso_sgx_enter_enclave() will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null), __vdso_sgx_enter_enclave() will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).

Quoted 4 lines

The SGX selftest is augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in __vdso_sgx_enter_enclave() by single-stepping through it and unwinding call stack at every instruction.

Why does the SDK anyway use real enclaves when step debugging? I don't think kernel needs to scale to that. For me it looks more like a bad architectural choice in the SDK implementation than anything else.

You should design SDK in a way that it doesn't run the code inside real enclave at first. It is just the sanest development model to use. The current SDK has an unacceptable requirement that the workstation used to develop code destined to run inside enclave must possess an appropriate hardware. I don't think that is acceptable no matter what kind of API kernel provides to invoke enclaves.

I think "real" debug enclaves are only good for production testing when you have more or less ready to release/deploy something but still want to be able to get a memory dump of the enclave on occassion.

With these conclusions I think the current vDSO API is sufficient for Linux.

/Jarkko

#273 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

Quoted 20 lines · Jarkko Sakkinen
Quoted 19 lines · Cedric Xing

The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.

This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new __vdso_sgx_enter_enclave() will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null), __vdso_sgx_enter_enclave() will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).

Quoted 8 lines
Quoted 4 lines

The SGX selftest is augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in __vdso_sgx_enter_enclave() by single-stepping through it and unwinding call stack at every instruction.

Why does the SDK anyway use real enclaves when step debugging? I don't think kernel needs to scale to that. For me it looks more like a bad architectural choice in the SDK implementation than anything else.

Intel's SGX SDK *does* support simulation mode for debugging enclaves outside of SGX mode, or even on non-SGX platforms. But nothing can replace the real SGX environment, hence SGX ISA supports debugging real enclaves.

Quoted 6 lines

You should design SDK in a way that it doesn't run the code inside real enclave at first. It is just the sanest development model to use. The current SDK has an unacceptable requirement that the workstation used to develop code destined to run inside enclave must possess an appropriate hardware. I don't think that is acceptable no matter what kind of API kernel provides to invoke enclaves.

SDK supports simulation on non-SGX platforms.

Quoted 3 lines

I think "real" debug enclaves are only good for production testing when you have more or less ready to release/deploy something but still want to be able to get a memory dump of the enclave on occassion.

That's true.

Quoted 2 lines

With these conclusions I think the current vDSO API is sufficient for Linux.

The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.

The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.

Quoted 2 lines

/Jarkko

#274 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

Quoted 5 lines · Cedric
Quoted 2 lines

With these conclusions I think the current vDSO API is sufficient for Linux.

The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.

And how that is useful?

Quoted 5 lines

The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.

Of course I won't merge code of which usefulness I don't understand.

/Jarkko

#275 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

Quoted 8 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines

With these conclusions I think the current vDSO API is sufficient for Linux.

The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.

And how that is useful?

There is a lengthy discussion on its usefulness so I don't want to repeat. In short, it allows using untrusted stack as a convenient method to exchange data with the enclave. It is currently being used by Intel's SGX SDK for e/o-calls parameters.

Quoted 7 lines
Quoted 5 lines

The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.

Of course I won't merge code of which usefulness I don't understand.

Sure.

Any other questions I can help with?

Quoted 2 lines

/Jarkko

#276 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

Quoted 21 lines · Cedric
Quoted 8 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines

With these conclusions I think the current vDSO API is sufficient for Linux.

The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.

And how that is useful?

There is a lengthy discussion on its usefulness so I don't want to repeat. In short, it allows using untrusted stack as a convenient method to exchange data with the enclave. It is currently being used by Intel's SGX SDK for e/o-calls parameters.

Quoted 6 lines
Quoted 5 lines

The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.

Quoted 5 lines
Quoted 1 line

Of course I won't merge code of which usefulness I don't understand.

Sure.

Any other questions I can help with?

I dissected my concerns in other email. We can merge this feature after v21 if it makes sense.

/Jarkko

#277 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

Quoted 22 lines · Jarkko Sakkinen
Quoted 21 lines · Cedric
Quoted 8 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines

With these conclusions I think the current vDSO API is sufficient for Linux.

The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.

And how that is useful?

There is a lengthy discussion on its usefulness so I don't want to repeat. In short, it allows using untrusted stack as a convenient method to exchange data with the enclave. It is currently being used by Intel's SGX SDK for e/o-calls parameters.

Quoted 6 lines
Quoted 5 lines

The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

Of course I won't merge code of which usefulness I don't understand.

Sure.

Any other questions I can help with?

I dissected my concerns in other email. We can merge this feature after v21 if it makes sense.

Sent out v3 of vDSO changes last night. I hope your concerns have been properly addressed.

The new vDSO API is a community consensus. I can help on whatever technical problems you may have but I don't see a reason you should reject it.

Quoted 2 lines

/Jarkko

#278 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

In reply to Jarkko Sakkinen
Quoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines

With these conclusions I think the current vDSO API is sufficient for Linux.

The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.

And how that is useful?

Quoted 7 lines
Quoted 5 lines

The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.

Of course I won't merge code of which usefulness I don't understand.

I re-read the cover letter [1] because it usually is the place to "pitch" a feature.

It fails to address two things:

1. How and in what circumstances is an untrusted stack is a better vessel for handling exceptions than the register based approach that we already have?
2. How is it simpler approach? There is a strong claim of simplicity and convenience without anything backing it.
3. Why we need both register and stack based approach co-exist? I'd go with one approach for a new API without any legacy whatsoever.

This really needs a better pitch before we can consider doing anything to it.

Also, in [2] there is talk about the next revision. Maybe the way go forward is to address the three issues I found in the cover letter and fix whatever needed to be fixed in the actual patches?

[1] https://lkml.org/lkml/2019/4/24/84
[2] https://lkml.org/lkml/2019/4/25/1170

/Jarkko

#279 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

Quoted 18 lines · Jarkko Sakkinen
Quoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines

With these conclusions I think the current vDSO API is sufficient for Linux.

The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.

And how that is useful?

Quoted 7 lines
Quoted 5 lines

The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.

Of course I won't merge code of which usefulness I don't understand.

Quoted 8 lines

I re-read the cover letter [1] because it usually is the place to "pitch" a feature.

It fails to address two things:

1. How and in what circumstances is an untrusted stack is a better vessel for handling exceptions than the register based approach that we already have?

We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.

Quoted 2 lines

2. How is it simpler approach? There is a strong claim of simplicity and convenience without anything backing it.

The major benefits in terms of simplicity realize in user mode applications. It's always a trade-off. This vDSO API takes 10-20 more lines than the original one but would save hundreds or even thousands in user applications.

Again, I don't want to repeat everything as you can look back at the lengthy discussion to dig out the details.

Quoted 2 lines

3. Why we need both register and stack based approach co-exist? I'd go with one approach for a new API without any legacy whatsoever.

Neither is a legacy to the other. Supporting both approaches is by design. Again, the goal is to enable more vessels because there's *no* single best vessel.

Quoted 9 lines

This really needs a better pitch before we can consider doing anything to it.

Also, in [2] there is talk about the next revision. Maybe the way go forward is to address the three issues I found in the cover letter and fix whatever needed to be fixed in the actual patches?

[1] https://lkml.org/lkml/2019/4/24/84
[2] https://lkml.org/lkml/2019/4/25/1170

Let me update the commit message for the vDSO API and send you a patch.

Quoted 2 lines

/Jarkko

#280 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

Quoted 24 lines · Cedric
Quoted 23 lines · Jarkko Sakkinen
Quoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines

With these conclusions I think the current vDSO API is sufficient for Linux.

The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.

And how that is useful?

Quoted 7 lines
Quoted 5 lines

The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.

Of course I won't merge code of which usefulness I don't understand.

I re-read the cover letter [1] because it usually is the place to "pitch" a feature.

It fails to address two things:

Quoted 7 lines
Quoted 3 lines

1. How and in what circumstances is an untrusted stack is a better vessel for handling exceptions than the register based approach that we already have?

We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.

I think reasonable metric is actually the coverage of the Intel SDK based enclaves. How widely are they in the wild? If the user base is large, it should be reasonable to support this just based on that.

/Jarkko

#281 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

Quoted 8 lines · Jarkko Sakkinen
Quoted 3 lines · Cedric

We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.

I think reasonable metric is actually the coverage of the Intel SDK based enclaves. How widely are they in the wild? If the user base is large, it should be reasonable to support this just based on that.

Large enough that Andy agreed to take the vDSO code with the optional callback, despite his personal opinion being that mucking with uR{B,S}P from within the enclave is poor form.

#282 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

Quoted 6 lines · Sean Christopherson
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines · Cedric

We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.

Quoted 7 lines
Quoted 3 lines

I think reasonable metric is actually the coverage of the Intel SDK based enclaves. How widely are they in the wild? If the user base is large, it should be reasonable to support this just based on that.

Large enough that Andy agreed to take the vDSO code with the optional callback, despite his personal opinion being that mucking with uR{B,S}P from within the enclave is poor form.

OK, the cover letter empahasized things that did not make sense to me, which made me to do my initial conclusions. I don't recall even reading the word "coverage" from it.

Anyways, I'm sure we can land this after v21 has been published now that the rationale is clear.

/Jarkko

#283 lore raw

Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack

In reply to Jarkko Sakkinen
Quoted 29 lines · Jarkko Sakkinen
Quoted 28 lines · Cedric
Quoted 26 lines · Jarkko Sakkinen
Quoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines

With these conclusions I think the current vDSO API is sufficient for Linux.

The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.

And how that is useful?

Quoted 7 lines
Quoted 5 lines

The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.

Of course I won't merge code of which usefulness I don't understand.

I re-read the cover letter [1] because it usually is the place to "pitch" a feature.

It fails to address two things:

1. How and in what circumstances is an untrusted stack is a better vessel for handling exceptions than the register based approach that we already have?

Quoted 7 lines
Quoted 3 lines

We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.

I think reasonable metric is actually the coverage of the Intel SDK based enclaves. How widely are they in the wild? If the user base is large, it should be reasonable to support this just based on that.

I don't know how many existing enclaves out there, but definitely larger than 0 (zero), while user base for the old API is definitely 0. What are you worrying, really?

Quoted 2 lines

/Jarkko

#284 lore raw

v3 cover x86/sgx: Amend vDSO API to allow enclave/host parameter passing on untrusted stack

In reply to Cedric Xing

This patchset is based upon, and can be applied cleanly on SGX1 patch v20
(https://lkml.org/lkml/2019/4/17/344) by Jarkko Sakkinen.

The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. After all, passing data via untrusted stack is very easy to implement (by enclaves), with essentially no overhead, therefore is very suitable for exchanging data in small amounts, so could be desirable by future SGX applications as well.

This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. And in order to service o-calls and to preserve the untrusted stack upon exceptions, the new vDSO API takes one more optional parameter - "callback", which if supplied, will be invoked on all enclave exits (including normal and asynchronous exits). Ample details regarding the new ABI have been documented as comments inside the source code located in arch/x86/entry/vsgx_enter_enclave.S

Please note that there was a lengthy discussion on what is the "best" approach for passing parameters for trusted/untrusted calls. Unfortunately there's no single "best" approach that fits all use cases, hence this new ABI has been designed intentionally to accommodate varieties. Therefore, to those not interested in using the untrusted stack, whatever worked with the old ABI proposed by Sean will continue to work with this new ABI.

The SGX selftest has been augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in __vdso_sgx_enter_enclave() by single-stepping through it and unwinding call stack at every instruction. Please note that the selftest CANNOT run to completion yet, as it depends on the vDSO fixup code to signal the process upon #DB/#BP inside enclaves (rather than the current behavior of branching to the handler in vDSO).

Changelog:
· This is version 3 of this patch series with the following changes.
- Per Andy Lutomirski and Sean Christopherson, revised comments and their format in arch/x86/entry/vsgx_enter_enclave.S
- Per Jarkko Sakkinen, revised the cover letter to articulate motivation and objective of this patchset.
· v2 - https://patchwork.kernel.org/cover/10914161/
· v1 - https://patchwork.kernel.org/cover/10911615/

Cedric Xing (3):
selftests/x86: Fixed Makefile for SGX selftest
x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing
on untrusted stack
selftests/x86: Augment SGX selftest to test new
__vdso_sgx_enter_enclave() and its callback interface

 arch/x86/entry/vdso/vsgx_enter_enclave.S   | 214 ++++++++++----
 arch/x86/include/uapi/asm/sgx.h            |  14 +-
 tools/testing/selftests/x86/Makefile       |  12 +-
 tools/testing/selftests/x86/sgx/Makefile   |  49 ++--
 tools/testing/selftests/x86/sgx/main.c     | 323 ++++++++++++++++++---
 tools/testing/selftests/x86/sgx/sgx_call.S |  40 ++-
 6 files changed, 500 insertions(+), 152 deletions(-)
Signature · 1 line
2.17.1
#285 lore raw

RFC PATCH v3 1/3 selftests/x86: Fixed Makefile for SGX selftest

The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.

---
 tools/testing/selftests/x86/Makefile     | 12 +++----
 tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++---------------
 2 files changed, 22 insertions(+), 35 deletions(-)
tools/testing/selftests/x86/Makefile +6 -6
index 4fc9a42f56ea..1294c5f5b6ca 100644
@@ -70,11 +70,11 @@ all_32: $(BINARIES_32)
 
 all_64: $(BINARIES_64)
 
-all_64: $(SUBDIRS_64)
-	@for DIR in $(SUBDIRS_64); do			\
-		BUILD_TARGET=$(OUTPUT)/$$DIR;		\
-		mkdir $$BUILD_TARGET  -p;		\
-		make OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
+all_64: | $(SUBDIRS_64)
+	@for DIR in $|; do					\
+		BUILD_TARGET=$(OUTPUT)/$$DIR;			\
+		mkdir $$BUILD_TARGET  -p;			\
+		$(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@;	\
 	done
 
 EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)
@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01)
 all: warn_32bit_failure
 
 warn_32bit_failure:
-	@echo "Warning: you seem to have a broken 32-bit build" 2>&1; 	\
+	@echo "Warning: you seem to have a broken 32-bit build" 2>&1;	\
 	echo "environment.  This will reduce test coverage of 64-bit" 2>&1; \
 	echo "kernels.  If you are using a Debian-like distribution," 2>&1; \
 	echo "try:"; 2>&1; \
tools/testing/selftests/x86/sgx/Makefile +16 -29

63 lines not on this page · raw message

Signature · 1 line
2.17.1
#286 lore raw

RFC PATCH v3 2/3 x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

In reply to Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls) on the untrusted stack.

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 214 ++++++++++++++++-------
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 157 insertions(+), 71 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +150 -64
index fe0bf6671d6d..62f28c01b3c8 100644
@@ -14,88 +14,174 @@
 .code64
 .section .text, "ax"
 
-#ifdef SGX_KERNEL_DOC
 /**
  * __vdso_sgx_enter_enclave() - Enter an SGX enclave
  *
- * @leaf:	**IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:	**IN \%rbx** - TCS, must be non-NULL
- * @ex_info:	**IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer
+ * Parameters:
+ *	@leaf, passed in %eax, must be either EENTER(2) or ERESUME(3)
+ *	@tcs, passed on stack at 8(%rsp), is the linear address of TCS
+ *	@exinfo, passed on stack at 0x10(%rsp), is optional, and if non-NULL,
+ *	shall point to an sgx_enclave_exinfo structure to receive information
+ *	about the enclave exit
+ *	@callback, passed on stack at 0x18(%rsp), is optiona, and if non-NULL,
+ *	points to a callback function that will be invoked after the enclave
+ *	exits
  *
- * Return:
- *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is
- *  not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults
+ * Returns:
+ *	$0 (zero) on a clean exit from the enclave
+ *	$-EINVAL will be returned if leaf isn't either EENTER or ERESUME
+ *	Other negative values could also be returned as the return value from
+ *	the callback function
  *
- * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant with the
- * x86-64 ABI, i.e. cannot be called from standard C code.   As noted above,
- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``.  All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc...  Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * IMPORTANT! This API is **not** compliant with x86-64 ABI but adopts a
+ * proprietary calling convention, described below:
+ *   · As noted above, input parameters are passed via %eax and the stack.
+ *   · The return value is passed via %eax.
+ *   · %rbx and %rcx must be treated as volatile as they are modified as part
+ *     of enclaves transitions and are used as scratch regs.
+ *   · %rdx, %rdi, %rsi and %r8-%r15 are passed as is and may be freely
+ *     modified by the enclave. Values left in those registers will not be
+ *     altered either, so will be visiable to the callback or the caller (if no
+ *     callback is specified).
+ *   · %rsp is saved/restored across __vdso_sgx_enter_enclave().
+ *
+ * A callback function, if supplied, shall have the following signature:
+ *
+ *	int callback(long rdi, long rsi, long rdx,
+ *		     struct sgx_enclave_exinfo *exinfo, long r8, long r9,
+ *		     void *tcs, long ursp);
+ *
+ * Callback functions shall comply to x86_64 ABI.
+ *   · All registers left off by the enclave are passed as is except %rax, %rbx
+ *     and %rcx. %rdi, %rsi, %r8 and %9 could be accessed as function
+ *     parameters, while other registers could be access in assembly code if
+ *     needed.
+ *   · Positive return values from the callback will be interpreted as ENCLU
+ *     leafs to re-enter the enclave. Currently only EENTER(2) and ERESUME(3)
+ *     are supported, while all other positive return values will result in
+ *     $-EINVAL returned to the caller of __vdso_sgx_enter_enclave().
+ *   · $0 (zero) or negative return values will be passed back to the caller of
+ *     __vdso_sgx_enter_enclave() as is.
+ *
+ * Pseudo-code:
+ *
+ * typedef int (*sgx_callback)(long rdi, long rsi, long rdx,
+ *			       struct sgx_enclave_exinfo *exinfo, long r8,
+ *			       long r9, void *tcs, long ursp);
+ *
+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ *				struct sgx_enclave_exinfo *exinfo,
+ *				sgx_callback callback)
+ * {
+ *	while (leaf == EENTER || leaf == ERESUME) {
+ *		int rc;
+ *		try {
+ *			ENCLU[leaf];
+ *			rc = 0;
+ *			if (exinfo)
+ *				exinfo->leaf = EEXIT;
+ *		} catch (exception) {
+ *			rc = -EFAULT;
+ *			if (exinfo)
+ *				*exinfo = exception;
+ *		}
+ *
+ *		leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ *						   r8, r9, tcs, ursp);
+ *	}
+ *
+ *	return leaf > 0 ? -EINVAL : leaf;
+ * }
  */
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-			 struct sgx_enclave_exception *ex_info)
-{
-	if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-		return -EINVAL;
-
-	if (!tcs)
-		return -EINVAL;
-
-	try {
-		ENCLU[leaf];
-	} catch (exception) {
-		if (e)
-			*e = exception;
-		return -EFAULT;
-	}
-
-	return 0;
-}
-#endif
 ENTRY(__vdso_sgx_enter_enclave)
-	/* EENTER <= leaf <= ERESUME */
+	/* Prolog */
+	.cfi_startproc
+	push	%rbp
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbp, 0
+	mov	%rsp, %rbp
+	.cfi_def_cfa_register	%rbp
+
+1:	/* EENTER <= leaf <= ERESUME */
 	cmp	$0x2, %eax
-	jb	bad_input
-
+	jb	6f
 	cmp	$0x3, %eax
-	ja	bad_input
+	ja	6f
 
-	/* TCS must be non-NULL */
-	test	%rbx, %rbx
-	je	bad_input
+	/* Load TCS and AEP */
+	mov	0x10(%rbp), %rbx
+	lea	2f(%rip), %rcx
 
-	/* Save @exception_info */
-	push	%rcx
+	/* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:	enclu
 
-	/* Load AEP for ENCLU */
-	lea	1f(%rip),  %rcx
-1:	enclu
+	/* EEXIT path */
+	xor	%ebx, %ebx
+3:	mov	0x18(%rbp), %rcx
+	jrcxz	4f
+	mov	%eax, EX_LEAF(%rcx)
+	jnc	4f
+	mov	%di, EX_TRAPNR(%rcx)
+	mov	%si, EX_ERROR_CODE(%rcx)
+	mov	%rdx, EX_ADDRESS(%rcx)
 
-	add	$0x8, %rsp
-	xor	%eax, %eax
+4:	/* Call *callback if supplied */
+	mov	0x20(%rbp), %rax
+	test	%rax, %rax
+	/*
+	 * At this point, %ebx holds the effective return value, which shall be
+	 * returned if no callback is specified
+	 */
+	cmovz	%rbx, %rax
+	jz	7f
+	/*
+	 * Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be
+	 * restored after *callback returns.
+	 */
+	mov	%rsp, %rbx
+	and	$-0x10, %rsp
+	/* Clear RFLAGS.DF per x86_64 ABI */
+	cld
+	/* Parameters for *callback */
+	push	%rbx
+	push	0x10(%rbp)
+	/* Call *%rax via retpoline */
+	call	40f
+	/*
+	 * Restore %rsp to its original value left off by the enclave from last
+	 * exit
+	 */
+	mov	%rbx, %rsp
+	/*
+	 * Positive return value from *callback will be interpreted as an ENCLU
+	 * leaf, while a non-positive value will be interpreted as the return
+	 * value to be passed back to the caller.
+	 */
+	jmp	1b
+40:	/* retpoline */
+	call	42f
+41:	pause
+	lfence
+	jmp	41b
+42:	mov	%rax, (%rsp)
 	ret
 
-bad_input:
-	mov     $(-EINVAL), %rax
-	ret
+5:	/* Exception path */
+	mov	$-EFAULT, %ebx
+	stc
+	jmp	3b
 
-.pushsection .fixup, "ax"
-	/* Re-load @exception_info and fill it (if it's non-NULL) */
-2:	pop	%rcx
-	test    %rcx, %rcx
-	je      3f
+6:	/* Unsupported ENCLU leaf */
+	cmp	$0, %eax
+	jle	7f
+	mov	$-EINVAL, %eax
 
-	mov	%eax, EX_LEAF(%rcx)
-	mov	%di,  EX_TRAPNR(%rcx)
-	mov	%si,  EX_ERROR_CODE(%rcx)
-	mov	%rdx, EX_ADDRESS(%rcx)
-3:	mov	$(-EFAULT), %rax
+7:	/* Epilog */
+	leave
+	.cfi_def_cfa		%rsp, 8
 	ret
-.popsection
+	.cfi_endproc
 
-_ASM_VDSO_EXTABLE_HANDLE(1b, 2b)
+_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)
 
 ENDPROC(__vdso_sgx_enter_enclave)
arch/x86/include/uapi/asm/sgx.h +7 -7
index 9ed690a38c70..50d2b5143e5e 100644
@@ -24,7 +24,7 @@
 
 /**
  * struct sgx_enclave_create - parameter structure for the
- *                             %SGX_IOC_ENCLAVE_CREATE ioctl
+ *			       %SGX_IOC_ENCLAVE_CREATE ioctl
  * @src:	address for the SECS page data
  */
 struct sgx_enclave_create  {
@@ -33,7 +33,7 @@ struct sgx_enclave_create  {
 
 /**
  * struct sgx_enclave_add_page - parameter structure for the
- *                               %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
+ *				 %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
  * @addr:	address within the ELRANGE
  * @src:	address for the page data
  * @secinfo:	address for the SECINFO data
@@ -49,7 +49,7 @@ struct sgx_enclave_add_page {
 
 /**
  * struct sgx_enclave_init - parameter structure for the
- *                           %SGX_IOC_ENCLAVE_INIT ioctl
+ *			     %SGX_IOC_ENCLAVE_INIT ioctl
  * @sigstruct:	address for the SIGSTRUCT data
  */
 struct sgx_enclave_init {
@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute {
 };
 
 /**
- * struct sgx_enclave_exception - structure to report exceptions encountered in
- *				  __vdso_sgx_enter_enclave()
+ * struct sgx_enclave_exinfo - structure to report exceptions encountered in
+ *			       __vdso_sgx_enter_enclave()
  *
- * @leaf:	ENCLU leaf from \%eax at time of exception
+ * @leaf:	ENCLU leaf from \%eax at time of exception/exit
  * @trapnr:	exception trap number, a.k.a. fault vector
  * @error_code:	exception error code
  * @address:	exception address, e.g. CR2 on a #PF
  * @reserved:	reserved for future use
  */
-struct sgx_enclave_exception {
+struct sgx_enclave_exinfo {
 	__u32 leaf;
 	__u16 trapnr;
 	__u16 error_code;
Signature · 1 line
2.17.1
#287 lore raw

Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 44 lines · Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls) on the untrusted stack.

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 214 ++++++++++++++++-------
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 157 insertions(+), 71 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -0
index fe0bf6671d6d..62f28c01b3c8 100644

arch/x86/entry/vdso/vsgx_enter_enclave.S line 19

Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -1
@@ -14,88 +14,174 @@
 .code64
 .section .text, "ax"
 
-#ifdef SGX_KERNEL_DOC
 /**
  * __vdso_sgx_enter_enclave() - Enter an SGX enclave
  *

This was already there but here should not be empty line.

arch/x86/entry/vdso/vsgx_enter_enclave.S line 23

Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +0 -3
- * @leaf:	**IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:	**IN \%rbx** - TCS, must be non-NULL
- * @ex_info:	**IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer

Does not align with kdoc standards.

* @leaf: ENCLU leaf, must be EENTER or ERESUME
* @tcs: TCS, must be non-NULL
* @ex_info: Optional 'struct sgx_enclave_exception' pointer

Quoted 77 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +64 -11
+ * Parameters:
+ *	@leaf, passed in %eax, must be either EENTER(2) or ERESUME(3)
+ *	@tcs, passed on stack at 8(%rsp), is the linear address of TCS
+ *	@exinfo, passed on stack at 0x10(%rsp), is optional, and if non-NULL,
+ *	shall point to an sgx_enclave_exinfo structure to receive information
+ *	about the enclave exit
+ *	@callback, passed on stack at 0x18(%rsp), is optiona, and if non-NULL,
+ *	points to a callback function that will be invoked after the enclave
+ *	exits
  *
- * Return:
- *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is
- *  not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults
+ * Returns:
+ *	$0 (zero) on a clean exit from the enclave
+ *	$-EINVAL will be returned if leaf isn't either EENTER or ERESUME
+ *	Other negative values could also be returned as the return value from
+ *	the callback function
  *
- * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant with the
- * x86-64 ABI, i.e. cannot be called from standard C code.   As noted above,
- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``.  All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc...  Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * IMPORTANT! This API is **not** compliant with x86-64 ABI but adopts a
+ * proprietary calling convention, described below:
+ *   · As noted above, input parameters are passed via %eax and the stack.
+ *   · The return value is passed via %eax.
+ *   · %rbx and %rcx must be treated as volatile as they are modified as part
+ *     of enclaves transitions and are used as scratch regs.
+ *   · %rdx, %rdi, %rsi and %r8-%r15 are passed as is and may be freely
+ *     modified by the enclave. Values left in those registers will not be
+ *     altered either, so will be visiable to the callback or the caller (if no
+ *     callback is specified).
+ *   · %rsp is saved/restored across __vdso_sgx_enter_enclave().
+ *
+ * A callback function, if supplied, shall have the following signature:
+ *
+ *	int callback(long rdi, long rsi, long rdx,
+ *		     struct sgx_enclave_exinfo *exinfo, long r8, long r9,
+ *		     void *tcs, long ursp);
+ *
+ * Callback functions shall comply to x86_64 ABI.
+ *   · All registers left off by the enclave are passed as is except %rax, %rbx
+ *     and %rcx. %rdi, %rsi, %r8 and %9 could be accessed as function
+ *     parameters, while other registers could be access in assembly code if
+ *     needed.
+ *   · Positive return values from the callback will be interpreted as ENCLU
+ *     leafs to re-enter the enclave. Currently only EENTER(2) and ERESUME(3)
+ *     are supported, while all other positive return values will result in
+ *     $-EINVAL returned to the caller of __vdso_sgx_enter_enclave().
+ *   · $0 (zero) or negative return values will be passed back to the caller of
+ *     __vdso_sgx_enter_enclave() as is.
+ *
+ * Pseudo-code:
+ *
+ * typedef int (*sgx_callback)(long rdi, long rsi, long rdx,
+ *			       struct sgx_enclave_exinfo *exinfo, long r8,
+ *			       long r9, void *tcs, long ursp);
+ *
+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ *				struct sgx_enclave_exinfo *exinfo,
+ *				sgx_callback callback)
+ * {
+ *	while (leaf == EENTER || leaf == ERESUME) {
+ *		int rc;
+ *		try {
+ *			ENCLU[leaf];
+ *			rc = 0;
+ *			if (exinfo)
+ *				exinfo->leaf = EEXIT;
+ *		} catch (exception) {
+ *			rc = -EFAULT;
+ *			if (exinfo)

arch/x86/entry/vdso/vsgx_enter_enclave.S line 95

Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +9 -0
+ *				*exinfo = exception;
+ *		}
+ *
+ *		leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ *						   r8, r9, tcs, ursp);
+ *	}
+ *
+ *	return leaf > 0 ? -EINVAL : leaf;
+ * }
  */

Please remove all of this documentation or write more punctual documentation and follow the standard:

https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt

Not going to maintain the above. Rather do not add documentation at all than the above.

Quoted 146 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +77 -49
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-			 struct sgx_enclave_exception *ex_info)
-{
-	if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-		return -EINVAL;
-
-	if (!tcs)
-		return -EINVAL;
-
-	try {
-		ENCLU[leaf];
-	} catch (exception) {
-		if (e)
-			*e = exception;
-		return -EFAULT;
-	}
-
-	return 0;
-}
-#endif
 ENTRY(__vdso_sgx_enter_enclave)
-	/* EENTER <= leaf <= ERESUME */
+	/* Prolog */
+	.cfi_startproc
+	push	%rbp
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbp, 0
+	mov	%rsp, %rbp
+	.cfi_def_cfa_register	%rbp
+
+1:	/* EENTER <= leaf <= ERESUME */
 	cmp	$0x2, %eax
-	jb	bad_input
-
+	jb	6f
 	cmp	$0x3, %eax
-	ja	bad_input
+	ja	6f
 
-	/* TCS must be non-NULL */
-	test	%rbx, %rbx
-	je	bad_input
+	/* Load TCS and AEP */
+	mov	0x10(%rbp), %rbx
+	lea	2f(%rip), %rcx
 
-	/* Save @exception_info */
-	push	%rcx
+	/* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:	enclu
 
-	/* Load AEP for ENCLU */
-	lea	1f(%rip),  %rcx
-1:	enclu
+	/* EEXIT path */
+	xor	%ebx, %ebx
+3:	mov	0x18(%rbp), %rcx
+	jrcxz	4f
+	mov	%eax, EX_LEAF(%rcx)
+	jnc	4f
+	mov	%di, EX_TRAPNR(%rcx)
+	mov	%si, EX_ERROR_CODE(%rcx)
+	mov	%rdx, EX_ADDRESS(%rcx)
 
-	add	$0x8, %rsp
-	xor	%eax, %eax
+4:	/* Call *callback if supplied */
+	mov	0x20(%rbp), %rax
+	test	%rax, %rax
+	/*
+	 * At this point, %ebx holds the effective return value, which shall be
+	 * returned if no callback is specified
+	 */
+	cmovz	%rbx, %rax
+	jz	7f
+	/*
+	 * Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be
+	 * restored after *callback returns.
+	 */
+	mov	%rsp, %rbx
+	and	$-0x10, %rsp
+	/* Clear RFLAGS.DF per x86_64 ABI */
+	cld
+	/* Parameters for *callback */
+	push	%rbx
+	push	0x10(%rbp)
+	/* Call *%rax via retpoline */
+	call	40f
+	/*
+	 * Restore %rsp to its original value left off by the enclave from last
+	 * exit
+	 */
+	mov	%rbx, %rsp
+	/*
+	 * Positive return value from *callback will be interpreted as an ENCLU
+	 * leaf, while a non-positive value will be interpreted as the return
+	 * value to be passed back to the caller.
+	 */
+	jmp	1b
+40:	/* retpoline */
+	call	42f
+41:	pause
+	lfence
+	jmp	41b
+42:	mov	%rax, (%rsp)
 	ret
 
-bad_input:
-	mov     $(-EINVAL), %rax
-	ret
+5:	/* Exception path */
+	mov	$-EFAULT, %ebx
+	stc
+	jmp	3b
 
-.pushsection .fixup, "ax"
-	/* Re-load @exception_info and fill it (if it's non-NULL) */
-2:	pop	%rcx
-	test    %rcx, %rcx
-	je      3f
+6:	/* Unsupported ENCLU leaf */
+	cmp	$0, %eax
+	jle	7f
+	mov	$-EINVAL, %eax
 
-	mov	%eax, EX_LEAF(%rcx)
-	mov	%di,  EX_TRAPNR(%rcx)
-	mov	%si,  EX_ERROR_CODE(%rcx)
-	mov	%rdx, EX_ADDRESS(%rcx)
-3:	mov	$(-EFAULT), %rax
+7:	/* Epilog */
+	leave
+	.cfi_def_cfa		%rsp, 8
 	ret
-.popsection
+	.cfi_endproc
 
-_ASM_VDSO_EXTABLE_HANDLE(1b, 2b)
+_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)
 
 ENDPROC(__vdso_sgx_enter_enclave)
arch/x86/include/uapi/asm/sgx.h +0 -0
index 9ed690a38c70..50d2b5143e5e 100644

arch/x86/include/uapi/asm/sgx.h line 27

Quoted 5 lines
arch/x86/include/uapi/asm/sgx.h +1 -1
@@ -24,7 +24,7 @@
 
 /**
  * struct sgx_enclave_create - parameter structure for the
- *                             %SGX_IOC_ENCLAVE_CREATE ioctl
+ *			       %SGX_IOC_ENCLAVE_CREATE ioctl

You have bunch of these clutter diff's in your patch. Please get rid of them.

Quoted 22 lines
arch/x86/include/uapi/asm/sgx.h +2 -2
  * @src:	address for the SECS page data
  */
 struct sgx_enclave_create  {
@@ -33,7 +33,7 @@ struct sgx_enclave_create  {
 
 /**
  * struct sgx_enclave_add_page - parameter structure for the
- *                               %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
+ *				 %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
  * @addr:	address within the ELRANGE
  * @src:	address for the page data
  * @secinfo:	address for the SECINFO data
@@ -49,7 +49,7 @@ struct sgx_enclave_add_page {
 
 /**
  * struct sgx_enclave_init - parameter structure for the
- *                           %SGX_IOC_ENCLAVE_INIT ioctl
+ *			     %SGX_IOC_ENCLAVE_INIT ioctl
  * @sigstruct:	address for the SIGSTRUCT data
  */
 struct sgx_enclave_init {

arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_set_attribute { · line 70

Quoted 7 lines
arch/x86/include/uapi/asm/sgx.h +2 -2
@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute {
 };
 
 /**
- * struct sgx_enclave_exception - structure to report exceptions encountered in
- *				  __vdso_sgx_enter_enclave()
+ * struct sgx_enclave_exinfo - structure to report exceptions encountered in
+ *			       __vdso_sgx_enter_enclave()

If you want to rename a struct it should be its own commit. Anyway, I'd say that this unnecessary.

Quoted 16 lines
arch/x86/include/uapi/asm/sgx.h +2 -2
  *
- * @leaf:	ENCLU leaf from \%eax at time of exception
+ * @leaf:	ENCLU leaf from \%eax at time of exception/exit
  * @trapnr:	exception trap number, a.k.a. fault vector
  * @error_code:	exception error code
  * @address:	exception address, e.g. CR2 on a #PF
  * @reserved:	reserved for future use
  */
-struct sgx_enclave_exception {
+struct sgx_enclave_exinfo {
 	__u32 leaf;
 	__u16 trapnr;
 	__u16 error_code;
Signature · 1 line
2.17.1

Summary: I can live with the general idea but the patch itself is somewhat half-finished still.

/Jarkko

#288 lore raw

Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

In reply to Cedric Xing
Quoted 1 line · Cedric Xing

-#ifdef SGX_KERNEL_DOC

Why is this removed?

Quoted 16 lines

+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ * struct sgx_enclave_exinfo *exinfo,
+ * sgx_callback callback)
+ * {
+ * while (leaf == EENTER || leaf == ERESUME) {
+ * int rc;
+ * try {
+ * ENCLU[leaf];
+ * rc = 0;
+ * if (exinfo)
+ * exinfo->leaf = EEXIT;
+ * } catch (exception) {
+ * rc = -EFAULT;
+ * if (exinfo)
+ * *exinfo = exception;
+ * }

Quoted 8 lines

+ *
+ * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ * r8, r9, tcs, ursp);
+ * }
+ *
+ * return leaf > 0 ? -EINVAL : leaf;
+ * }
*/

What is this? C++ and anyway there is already a source code. No need to duplicate with pseudo-code. Only adds maintenance burde. Please get rid of this.

/Jarkko

#289 lore raw

Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 30 lines · Jarkko Sakkinen
Quoted 1 line · Cedric Xing

-#ifdef SGX_KERNEL_DOC

Why is this removed?

Quoted 16 lines

+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ * struct sgx_enclave_exinfo *exinfo,
+ * sgx_callback callback)
+ * {
+ * while (leaf == EENTER || leaf == ERESUME) {
+ * int rc;
+ * try {
+ * ENCLU[leaf];
+ * rc = 0;
+ * if (exinfo)
+ * exinfo->leaf = EEXIT;
+ * } catch (exception) {
+ * rc = -EFAULT;
+ * if (exinfo)
+ * *exinfo = exception;
+ * }

Quoted 8 lines

+ *
+ * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ * r8, r9, tcs, ursp);
+ * }
+ *
+ * return leaf > 0 ? -EINVAL : leaf;
+ * }
*/

Quoted 3 lines

What is this? C++ and anyway there is already a source code. No need to duplicate with pseudo-code. Only adds maintenance burde. Please get rid of this.

Adding C pseudo-code was my idea, e.g. it already exists in v20. Declaring a psuedo-C function coerces kernel-doc into generating documentation for the asm routine. IIRC, fully defining the function is not required for docs, but IMO it's significantly easier to gain an understanding of a blob of asm if there is higher level pseudocode.

#290 lore raw

Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 35 lines · Sean Christopherson
Quoted 30 lines · Jarkko Sakkinen
Quoted 1 line · Cedric Xing

-#ifdef SGX_KERNEL_DOC

Why is this removed?

Quoted 16 lines

+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ * struct sgx_enclave_exinfo *exinfo,
+ * sgx_callback callback)
+ * {
+ * while (leaf == EENTER || leaf == ERESUME) {
+ * int rc;
+ * try {
+ * ENCLU[leaf];
+ * rc = 0;
+ * if (exinfo)
+ * exinfo->leaf = EEXIT;
+ * } catch (exception) {
+ * rc = -EFAULT;
+ * if (exinfo)
+ * *exinfo = exception;
+ * }

Quoted 8 lines

+ *
+ * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ * r8, r9, tcs, ursp);
+ * }
+ *
+ * return leaf > 0 ? -EINVAL : leaf;
+ * }
*/

Quoted 3 lines

What is this? C++ and anyway there is already a source code. No need to duplicate with pseudo-code. Only adds maintenance burde. Please get rid of this.

Quoted 5 lines

Adding C pseudo-code was my idea, e.g. it already exists in v20. Declaring a psuedo-C function coerces kernel-doc into generating documentation for the asm routine. IIRC, fully defining the function is not required for docs, but IMO it's significantly easier to gain an understanding of a blob of asm if there is higher level pseudocode.

The way to do this right would be to write a documentation block before kdoc's for the functions. It is described in the last section of

https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt0

I.e. organize the file as

1. Documentation block describing the theory of operation.
2. Functions and their associated documentations.

/Jarkko

#291 lore raw

Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 45 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 30 lines · Jarkko Sakkinen
Quoted 1 line · Cedric Xing

-#ifdef SGX_KERNEL_DOC

Why is this removed?

Quoted 16 lines

+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ * struct sgx_enclave_exinfo *exinfo,
+ * sgx_callback callback)
+ * {
+ * while (leaf == EENTER || leaf == ERESUME) {
+ * int rc;
+ * try {
+ * ENCLU[leaf];
+ * rc = 0;
+ * if (exinfo)
+ * exinfo->leaf = EEXIT;
+ * } catch (exception) {
+ * rc = -EFAULT;
+ * if (exinfo)
+ * *exinfo = exception;
+ * }

Quoted 8 lines

+ *
+ * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ * r8, r9, tcs, ursp);
+ * }
+ *
+ * return leaf > 0 ? -EINVAL : leaf;
+ * }
*/

Quoted 3 lines

What is this? C++ and anyway there is already a source code. No need to duplicate with pseudo-code. Only adds maintenance burde. Please get rid of this.

Quoted 5 lines

Adding C pseudo-code was my idea, e.g. it already exists in v20. Declaring a psuedo-C function coerces kernel-doc into generating documentation for the asm routine. IIRC, fully defining the function is not required for docs, but IMO it's significantly easier to gain an understanding of a blob of asm if there is higher level pseudocode.

The way to do this right would be to write a documentation block before kdoc's for the functions. It is described in the last section of

Quoted 6 lines

https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt0

I.e. organize the file as

1. Documentation block describing the theory of operation.
2. Functions and their associated documentations.

The kernel doc parser straight up doesn't work on asm functions, hence the shenanigans to coerce it into thinking it's parsing a normal C function.

#292 lore raw

Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 5 lines · Sean Christopherson
Quoted 5 lines

The way to do this right would be to write a documentation block before kdoc's for the functions. It is described in the last section of

https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt0

Quoted 7 lines
Quoted 4 lines

I.e. organize the file as

1. Documentation block describing the theory of operation.
2. Functions and their associated documentations.

The kernel doc parser straight up doesn't work on asm functions, hence the shenanigans to coerce it into thinking it's parsing a normal C function.

Aah. I think I was too hazy with my comment. Looked it from patchwork and the documentation is mostly just fine.

For this patch this cruft nees to be removed:

- * %SGX_IOC_ENCLAVE_INIT ioctl
+ * %SGX_IOC_ENCLAVE_INIT ioctl

These make squashing the patch properly nasty.

Also there is one unwanted rename. Did not find anything else obviously wrong.

/Jarkko

#293 lore raw

Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

Quoted 19 lines · Jarkko Sakkinen
Quoted 5 lines · Sean Christopherson
Quoted 5 lines

The way to do this right would be to write a documentation block before kdoc's for the functions. It is described in the last section of

https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt0

Quoted 7 lines
Quoted 4 lines

I.e. organize the file as

1. Documentation block describing the theory of operation.
2. Functions and their associated documentations.

The kernel doc parser straight up doesn't work on asm functions, hence the shenanigans to coerce it into thinking it's parsing a normal C function.

Aah. I think I was too hazy with my comment. Looked it from patchwork and the documentation is mostly just fine.

For this patch this cruft nees to be removed:

Quoted 7 lines

- * %SGX_IOC_ENCLAVE_INIT ioctl
+ * %SGX_IOC_ENCLAVE_INIT ioctl

These make squashing the patch properly nasty.

Also there is one unwanted rename. Did not find anything else obviously wrong.

Reformatted the comments. Tested with kernel-doc. Now v4 could be converted and displayed nicely as man pages.

Quoted 2 lines

/Jarkko

#294 lore raw

RFC PATCH v3 3/3 selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface

In reply to Cedric Xing

This patch augments SGX selftest with two new tests.

The first test exercises the newly added callback interface, by marking the whole enclave range as PROT_READ, then calling mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes. This test also serves as an example to demonstrate the callback interface.

The second test single-steps through __vdso_sgx_enter_enclave() to make sure the call stack can be unwound at every instruction within that vDSO API. Its purpose is to validate the hand-crafted CFI directives in the assembly.

---
 tools/testing/selftests/x86/sgx/Makefile   |   6 +-
 tools/testing/selftests/x86/sgx/main.c     | 323 ++++++++++++++++++---
 tools/testing/selftests/x86/sgx/sgx_call.S |  40 ++-
 3 files changed, 322 insertions(+), 47 deletions(-)
tools/testing/selftests/x86/sgx/Makefile +3 -3
index 3af15d7c8644..31f937e220c4 100644
@@ -14,16 +14,16 @@ TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx
 all_64: $(TEST_CUSTOM_PROGS)
 
 $(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o
-	$(CC) $(HOST_CFLAGS) -o $@ $^
+	$(CC) $(HOST_CFLAGS) -o $@ $^ -lunwind -ldl -Wl,--defsym,__image_base=0 -pie
 
 $(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
 	$(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@
 
 $(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf
-	objcopy --remove-section=.got.plt -O binary $< $@
+	objcopy -O binary $< $@
 
 $(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S
-	$(CC) $(ENCL_CFLAGS) -T $^ -o $@
+	$(CC) $(ENCL_CFLAGS) -T $^ -o $@ -Wl,--build-id=none
 
 $(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin
 	$^ $@
tools/testing/selftests/x86/sgx/main.c +282 -41

388 lines not on this page · raw message

tools/testing/selftests/x86/sgx/sgx_call.S +37 -3
index 14bd0a044199..ca2c0c947758 100644
@@ -7,9 +7,43 @@
 
 	.global sgx_call
 sgx_call:
+	.cfi_startproc
+	push	%r15
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r15, 0
+	push	%r14
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r14, 0
+	push	%r13
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r13, 0
+	push	%r12
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%r12, 0
 	push	%rbx
-	mov	$0x02, %rax
-	mov	%rdx, %rbx
-	call	*%r8
+	.cfi_adjust_cfa_offset	8
+	.cfi_rel_offset		%rbx, 0
+	push	$0
+	.cfi_adjust_cfa_offset	8
+	push	0x48(%rsp)
+	.cfi_adjust_cfa_offset	8
+	push	0x48(%rsp)
+	.cfi_adjust_cfa_offset	8
+	push	0x48(%rsp)
+	.cfi_adjust_cfa_offset	8
+	mov	$2, %eax
+	call	*eenter(%rip)
+	add	$0x20, %rsp
+	.cfi_adjust_cfa_offset	-0x20
 	pop	%rbx
+	.cfi_adjust_cfa_offset	-8
+	pop	%r12
+	.cfi_adjust_cfa_offset	-8
+	pop	%r13
+	.cfi_adjust_cfa_offset	-8
+	pop	%r14
+	.cfi_adjust_cfa_offset	-8
+	pop	%r15
+	.cfi_adjust_cfa_offset	-8
 	ret
+	.cfi_endproc
Signature · 1 line
2.17.1
#295 lore raw

Re: [RFC PATCH v3 0/3] x86/sgx: Amend vDSO API to allow enclave/host parameter passing on untrusted stack

In reply to Cedric Xing
Quoted 38 lines · Cedric Xing

This patchset is based upon, and can be applied cleanly on SGX1 patch v20
(https://lkml.org/lkml/2019/4/17/344) by Jarkko Sakkinen.

The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. After all, passing data via untrusted stack is very easy to implement (by enclaves), with essentially no overhead, therefore is very suitable for exchanging data in small amounts, so could be desirable by future SGX applications as well.

This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. And in order to service o-calls and to preserve the untrusted stack upon exceptions, the new vDSO API takes one more optional parameter - "callback", which if supplied, will be invoked on all enclave exits (including normal and asynchronous exits). Ample details regarding the new ABI have been documented as comments inside the source code located in arch/x86/entry/vsgx_enter_enclave.S

Please note that there was a lengthy discussion on what is the "best" approach for passing parameters for trusted/untrusted calls. Unfortunately there's no single "best" approach that fits all use cases, hence this new ABI has been designed intentionally to accommodate varieties. Therefore, to those not interested in using the untrusted stack, whatever worked with the old ABI proposed by Sean will continue to work with this new ABI.

The SGX selftest has been augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in __vdso_sgx_enter_enclave() by single-stepping through it and unwinding call stack at every instruction. Please note that the selftest CANNOT run to completion yet, as it depends on the vDSO fixup code to signal the process upon #DB/#BP inside enclaves (rather than the current behavior of branching to the handler in vDSO).

Quoted 8 lines

Changelog:
· This is version 3 of this patch series with the following changes.
- Per Andy Lutomirski and Sean Christopherson, revised comments and their format in arch/x86/entry/vsgx_enter_enclave.S
- Per Jarkko Sakkinen, revised the cover letter to articulate motivation and objective of this patchset.
· v2 - https://patchwork.kernel.org/cover/10914161/
· v1 - https://patchwork.kernel.org/cover/10911615/

1. I agree with the high level idea.
2. The patches do changes out of scope.

Generally, when doing kernel patches, even for a running patch set, please do not do anything extra.

It is also stated in the kernel process;

https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html#separate-your-changes

Once these are fully cleaned up we can merge them.

/Jarkko

#296 lore raw

v4 cover x86/sgx: Amend vDSO API to allow enclave/host parameter passing on untrusted stack

In reply to Cedric Xing

This patchset is based upon, and can be applied cleanly on SGX1 patch v20
(https://lkml.org/lkml/2019/4/17/344) by Jarkko Sakkinen.

The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. After all, passing data via untrusted stack is very easy to implement (by enclaves), with essentially no overhead, therefore is very suitable for exchanging data in small amounts, so could be desirable by future SGX applications as well.

This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. And in order to service o-calls and to preserve the untrusted stack upon exceptions, the new vDSO API takes one more optional parameter - "callback", which if supplied, will be invoked on all enclave exits (including normal and asynchronous exits). Ample details regarding the new ABI have been documented as comments inside the source code located in arch/x86/entry/vsgx_enter_enclave.S

Please note that there was a lengthy discussion on what is the "best" approach for passing parameters for trusted/untrusted calls. Unfortunately there's no single "best" approach that fits all use cases, hence this new ABI has been designed intentionally to accommodate varieties. Therefore, to those not interested in using the untrusted stack, whatever worked with the old ABI proposed by Sean will continue to work with this new ABI.

The SGX selftest has been augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in __vdso_sgx_enter_enclave() by single-stepping through it and unwinding call stack at every instruction.

Changelog:
· This is version 4 of this patch series with the following changes.
- Removed unrelated cosmetic changes.
- Rewrote and reformatted comments in arch/x86/entry/vdso/vsgx_enter_enclave.S to follow kernel-doc conventions. New comments now can be converted to nice looking man pages.
- Fixed minor issues in the unwinding selftest and now it can run to completion successfully with Sean's fix in vDSO fixup code
(https://patchwork.kernel.org/patch/11040801/). Comments have also been added to describe the tests done.
· v3 - https://patchwork.kernel.org/cover/11039263/
· v2 - https://patchwork.kernel.org/cover/10914161/
· v1 - https://patchwork.kernel.org/cover/10911615/

Cedric Xing (3):
selftests/x86/sgx: Fix Makefile for SGX selftest
x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing
on untrusted stack
selftests/x86/sgx: Augment SGX selftest to test vDSO API

 arch/x86/entry/vdso/vsgx_enter_enclave.S   | 310 ++++++++++++++-----
 arch/x86/include/uapi/asm/sgx.h            |  14 +-
 tools/testing/selftests/x86/sgx/Makefile   |  49 ++-
 tools/testing/selftests/x86/sgx/main.c     | 344 ++++++++++++++++++---
 tools/testing/selftests/x86/sgx/sgx_call.S |  40 ++-
 5 files changed, 600 insertions(+), 157 deletions(-)
Signature · 1 line
2.17.1
#297 lore raw

RFC PATCH v4 1/3 selftests/x86/sgx: Fix Makefile for SGX selftest

The original x86/sgx/Makefile didn't work when "x86/sgx" was specified as the test target, nor did it work with "run_tests" as the make target. Yet another problem was that it breaks 32-bit only build. This patch fixes those problems, along with adjustments to compiler/linker options and simplifications to the build rules.

---
 tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++---------------
 1 file changed, 16 insertions(+), 29 deletions(-)
tools/testing/selftests/x86/sgx/Makefile +16 -29
index 1fd6f2708e81..3af15d7c8644 100644
@@ -2,47 +2,34 @@ top_srcdir = ../../../../..
 
 include ../../lib.mk
 
-HOST_CFLAGS := -Wall -Werror -g $(INCLUDES) -fPIC
-ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIC \
+ifeq ($(shell $(CC) -dumpmachine | cut --delimiter=- -f1),x86_64)
+all: all_64
+endif
+
+HOST_CFLAGS := -Wall -Werror -g $(INCLUDES)
+ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIE \
 	       -fno-stack-protector -mrdrnd $(INCLUDES)
 
 TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx
 all_64: $(TEST_CUSTOM_PROGS)
 
-$(TEST_CUSTOM_PROGS): $(OUTPUT)/main.o $(OUTPUT)/sgx_call.o \
-		      $(OUTPUT)/encl_piggy.o
+$(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o
 	$(CC) $(HOST_CFLAGS) -o $@ $^
 
-$(OUTPUT)/main.o: main.c
-	$(CC) $(HOST_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/sgx_call.o: sgx_call.S
-	$(CC) $(HOST_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/encl_piggy.o: $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
-	$(CC) $(HOST_CFLAGS) -c encl_piggy.S -o $@
+$(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
+	$(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@
 
-$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf $(OUTPUT)/sgxsign
+$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf
 	objcopy --remove-section=.got.plt -O binary $< $@
 
-$(OUTPUT)/encl.elf: $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o
-	$(CC) $(ENCL_CFLAGS) -T encl.lds -o $@ $^
+$(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S
+	$(CC) $(ENCL_CFLAGS) -T $^ -o $@
 
-$(OUTPUT)/encl.o: encl.c
-	$(CC) $(ENCL_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/encl_bootstrap.o: encl_bootstrap.S
-	$(CC) $(ENCL_CFLAGS) -c $< -o $@
-
-$(OUTPUT)/encl.ss: $(OUTPUT)/encl.bin  $(OUTPUT)/sgxsign
-	$(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss
+$(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin
+	$^ $@
 
 $(OUTPUT)/sgxsign: sgxsign.c
 	$(CC) -o $@ $< -lcrypto
 
-EXTRA_CLEAN := $(OUTPUT)/sgx-selftest $(OUTPUT)/sgx-selftest.o \
-	       $(OUTPUT)/sgx_call.o $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss \
-	       $(OUTPUT)/encl.elf $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o \
-	       $(OUTPUT)/sgxsign
-
-.PHONY: clean
+EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(addprefix $(OUTPUT)/,	\
+		encl.elf encl.bin encl.ss encl_piggy.o sgxsign)
Signature · 1 line
2.17.1
#298 lore raw

Re: [RFC PATCH v4 1/3] selftests/x86/sgx: Fix Makefile for SGX selftest

Quoted 7 lines · Cedric Xing

The original x86/sgx/Makefile didn't work when "x86/sgx" was specified as the test target, nor did it work with "run_tests" as the make target. Yet another problem was that it breaks 32-bit only build. This patch fixes those problems, along with adjustments to compiler/linker options and simplifications to the build rules.

You must split this in quite a few separate patches:

1. One for each fix.
2. At least one patch for each change in compiler and linker options with a commit message clearly expalaining why the change was made.
3. One for each simplification.

We don't support 32-bit build:

config INTEL_SGX

	bool "Intel SGX core functionality"
	depends on X86_64 && CPU_SUP_INTEL

/Jarkko

#299 lore raw

Re: [RFC PATCH v4 1/3] selftests/x86/sgx: Fix Makefile for SGX selftest

Quoted 16 lines · Jarkko Sakkinen
Quoted 7 lines · Cedric Xing

The original x86/sgx/Makefile didn't work when "x86/sgx" was specified as the test target, nor did it work with "run_tests" as the make target. Yet another problem was that it breaks 32-bit only build. This patch fixes those problems, along with adjustments to compiler/linker options and simplifications to the build rules.

You must split this in quite a few separate patches:

1. One for each fix.
2. At least one patch for each change in compiler and linker options with a commit message clearly expalaining why the change was made.
3. One for each simplification.

Quoted 5 lines

We don't support 32-bit build:

config INTEL_SGX

	bool "Intel SGX core functionality"
	depends on X86_64 && CPU_SUP_INTEL

This is not to say that changes suck. This just in "unreviewable" state as far as the kernel processes go...

/Jarkko

#300 lore raw

Re: [RFC PATCH v4 1/3] selftests/x86/sgx: Fix Makefile for SGX selftest

Quoted 17 lines · Jarkko Sakkinen
Quoted 16 lines · Jarkko Sakkinen
Quoted 7 lines · Cedric Xing

The original x86/sgx/Makefile didn't work when "x86/sgx" was specified as the test target, nor did it work with "run_tests" as the make target. Yet another problem was that it breaks 32-bit only build. This patch fixes those problems, along with adjustments to compiler/linker options and simplifications to the build rules.

You must split this in quite a few separate patches:

1. One for each fix.
2. At least one patch for each change in compiler and linker options with a commit message clearly expalaining why the change was made.
3. One for each simplification.

Quoted 8 lines
Quoted 5 lines

We don't support 32-bit build:

config INTEL_SGX

	bool "Intel SGX core functionality"
	depends on X86_64 && CPU_SUP_INTEL

This is not to say that changes suck. This just in "unreviewable" state as far as the kernel processes go...

Please note that your patchset hasn't been upstreamed yet. Your Makefile is problematic to begin with. Technically it's your job to make it work before sending out any patches. You didn't explain what's done for each line of Makefile in your commit message either.

Not saying documentation is unimportant, but the purposes for those changes are obvious and easy to understand for anyone having reasonable knowledge on how Makefile works.

I'm totally fine not fixing the Makefile. You can just leave them out.

Quoted 2 lines

/Jarkko

#301 lore raw

Re: [RFC PATCH v4 1/3] selftests/x86/sgx: Fix Makefile for SGX selftest

Quoted 4 lines · Cedric

Please note that your patchset hasn't been upstreamed yet. Your Makefile is problematic to begin with. Technically it's your job to make it work before sending out any patches. You didn't explain what's done for each line of Makefile in your commit message either.

Yes, it is different case to do the initial version of the whole thing that suggest fixes to it. The latter needs to have more granularity. Bug fixes in any type of software development should be isolated to separate change sets. It is just a sane QA practice.

Quoted 5 lines

Not saying documentation is unimportant, but the purposes for those changes are obvious and easy to understand for anyone having reasonable knowledge on how Makefile works.

I'm totally fine not fixing the Makefile. You can just leave them out.

/Jarkko

#302 lore raw

RFC PATCH v4 2/3 x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack

In reply to Cedric Xing

The previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls) on the untrusted stack.

This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.

Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.

---
 arch/x86/entry/vdso/vsgx_enter_enclave.S | 310 +++++++++++++++++------
 arch/x86/include/uapi/asm/sgx.h          |  14 +-
 2 files changed, 242 insertions(+), 82 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +235 -75
index fe0bf6671d6d..a96542ba6945 100644
@@ -6,96 +6,256 @@
 
 #include "extable.h"
 
-#define EX_LEAF		0*8
-#define EX_TRAPNR	0*8+4
-#define EX_ERROR_CODE	0*8+6
-#define EX_ADDRESS	1*8
+#define EX_LEAF     0*8
+#define EX_TRAPNR   0*8+4
+#define EX_ERROR_CODE   0*8+6
+#define EX_ADDRESS  1*8
 
 .code64
 .section .text, "ax"
 
 #ifdef SGX_KERNEL_DOC
 /**
- * __vdso_sgx_enter_enclave() - Enter an SGX enclave
+ * typedef sgx_ex_callback - Callback function for __vdso_sgx_enter_enclave()
  *
- * @leaf:	**IN \%eax** - ENCLU leaf, must be EENTER or ERESUME
- * @tcs:	**IN \%rbx** - TCS, must be non-NULL
- * @ex_info:	**IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer
+ * @rdi:    value of %%rdi register at enclave exit
+ * @rsi:    value of %%rsi register at enclave exit
+ * @rdx:    value of %%rdx register at enclave exit
+ * @exinfo: pointer to a sgx_enclave_exinfo structure, which was passed to
+ *      __vdso_sgx_enter_enclave() as input
+ * @r8:     value of %%r8 register at enclave exit
+ * @r9:     value of %%r9 register at enclave exit
+ * @tcs:    TCS used by __vdso_sgx_enter_enclave() to enter the enclave,
+ *      could be used to re-enter the
+ *      enclave
+ * @ursp:   value of %%rsp register at enclave exit
+ *
+ * This is the callback function to be invoked upon enclave exits, including
+ * normal exits (as result of EEXIT), and asynchronous exits (AEX) due to
+ * exceptions occurred at EENTER or within the enclave.
+ *
+ * This callback is expected to follow x86_64 ABI.
  *
  * Return:
- *  **OUT \%eax** -
- *  %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is
- *  not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults
- *
- * **Important!**  __vdso_sgx_enter_enclave() is **NOT** compliant with the
- * x86-64 ABI, i.e. cannot be called from standard C code.   As noted above,
- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``.  All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc...  Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ *
+ * EENTER(2) - causes __vdso_sgx_enter_enclave() to issue ENCLU[EENTER] on the
+ * same TCS. All GPRs left off by this callback function will be passed through
+ * back to the enclave, except %%rax, %%rbx and %%rcx, which are clobbered by
+ * ENCLU[EENTER] instruction.
+ *
+ * ERESUME(3) - causes __vdso_sgx_enter_enclave() to issue ENCLU[ERESUME] on
+ * the same TCS.
+ *
+ * 0 (zero) or negative returned values will be returned back to
+ * __vdso_sgx_enter_enclave()'s caller as is.
+ *
+ * All other values will cause -EINVAL to be returned to
+ * __vdso_sgx_enter_enclave()'s caller.
+ *
+ * Note: All general purpose registers (GPRs) left off by the enclave are
+ * passed through to this function, except %%rax, %%rbx and %%rcx, which are
+ * used internally by __vdso_sgx_enter_enclave(). Some of those registers are
+ * accessible as function parameters (i.e. @rdi, @rsi, @rdx, @r8, @r9 and
+ * @ursp), while others can be accessed only from assembly code.
  */
-__vdso_sgx_enter_enclave(u32 leaf, void *tcs,
-			 struct sgx_enclave_exception *ex_info)
-{
-	if (leaf != SGX_EENTER && leaf != SGX_ERESUME)
-		return -EINVAL;
+typedef int sgx_ex_callback(long rdi, long rsi, long rdx,
+                struct sgx_enclave_exinfo *exinfo,
+                long r8, long r9, void *tcs, long ursp);
 
-	if (!tcs)
-		return -EINVAL;
+/**
+ * __vdso_sgx_enter_enclave() - Enter an SGX enclave and capture exceptions
+ *
+ * @leaf:
+ *  passed in %%eax, must be either EENTER(2) or ERESUME(3)
+ * @tcs:
+ *  passed on stack at 8(%%rsp), is the linear address of TCS
+ * @exinfo:
+ *  passed on stack at 0x10(%%rsp), optional, and if non-NULL, shall point
+ *  to an sgx_enclave_exinfo structure to receive information about the
+ *  enclave exit
+ * @callback:
+ *  passed on stack at 0x18(%%rsp), optional, and if non-NULL, points to a
+ *  callback function to be invoked at enclave exits
+ *
+ * __vdso_sgx_enter_enclave() issues either ENCLU[EENTER] or ENCLU[ERESUME] on
+ * @tcs depending on @leaf.
+ *
+ * IMPORTANT! This API is not compliant with x86-64 ABI but adopts a
+ * proprietary calling convention. Please see NOTES section below for details.
+ *
+ * On an enclave exit, @exinfo->leaf will be set to the ENCLU leaf at exit, if
+ * @exinfo is not NULL. That is, @exinfo->leaf may be one of the following:
+ *
+ *   * EEXIT:   Normal exit due to ENCLU[EEXIT] within the enclave. All other
+ *      members will remain intact.
+ *
+ *   * ERESUME: Asynchronous exit due to exceptions within the enclave.
+ *      @exinfo->trapnr, @exinfo->error_code and @exinfo->address are
+ *      set to the trap number, error code and fault address,
+ *      respectively.
+ *
+ *   * EENTER:  Exception occurred when trying to enter the enclave.
+ *      @exinfo->trapnr, @exinfo->error_code and @exinfo->address are
+ *      set to the trap number, error code and fault address,
+ *      accordingly.
+ *
+ * If @callback is NULL, 0 (zero) is returned if the enclave has been entered
+ * and exited normally, or -EFAULT if any exception has occurred, or -EINVAL if
+ * @leaf on input is neither EENTER or ERESUME.
+ *
+ * If @callback is not NULL, it is invoked at enclave exit, and then actions
+ * will be taken depending on its return value - i.e. positive value will be
+ * treated as ENCLU leaf to re-enter the enclave, while 0 (zero) or negative
+ * values will be returned back to the caller as is. Unrecognized leaf values
+ * will cause -EINVAL to be returned.
+ *
+ * Return:
+ *
+ * 0 (zero) is returned on a successful entry and normal exit from the enclave.
+ *
+ * -EINVAL is returned if @leaf is neither EENTER nor ERESUME, or if @callback
+ * is not NULL and returns a positive value that is neither EENTER nor ERESUME
+ * after the enclave exits.
+ *
+ * -EFAULT is returned if an exception has occurred at EENTER or during
+ * execution of the enclave and @callback is NULL, or if @callback is not NULL
+ * and it returns -EFAULT after the enclave exits.
+ *
+ * Other values may be returned as the return value from @callback if it is not
+ * NULL.
+ *
+ * Note: __vdso_sgx_enter_enclave() adopts a proprietary calling convention,
+ * described below:
+ *
+ *    * As noted above, input parameters are passed via %%eax and the stack.
+ *
+ *    * %%rbx and %%rcx must be treated as volatile as they are modified as part
+ *  of enclaves transitions and are used as scratch regs.
+ *
+ *    * %%rdx, %%rdi, %%rsi and %%r8-%%r15 are passed as is and may be freely
+ *  modified by the enclave. Values left in those registers will not be
+ *  altered either, so will be visiable to the callback or the caller (if no
+ *  callback is specified).
+ *
+ *    * %%rsp could be decremented by the enclave to allocate temporary space on
+ *      the untrusted stack. Temporary space allocated this way is retained in
+ *      the context of @callback, and will be freed (i.e. %%rsp will be
+ *      restored) before __vdso_sgx_enter_enclave() returns.
+ */
+int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+                 struct sgx_enclave_exinfo *exinfo,
+                 sgx_ex_callback *callback);
+{
+     while (leaf == EENTER || leaf == ERESUME) {
+    int rc;
+    try {
+        ENCLU[leaf];
+        rc = 0;
+        if (exinfo)
+            exinfo->leaf = EEXIT;
+    } catch (exception) {
+        rc = -EFAULT;
+        if (exinfo)
+            *exinfo = exception;
+    }
 
-	try {
-		ENCLU[leaf];
-	} catch (exception) {
-		if (e)
-			*e = exception;
-		return -EFAULT;
-	}
+    leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+                       r8, r9, tcs, ursp);
+     }
 
-	return 0;
+     return leaf > 0 ? -EINVAL : leaf;
 }
 #endif
+
 ENTRY(__vdso_sgx_enter_enclave)
-	/* EENTER <= leaf <= ERESUME */
-	cmp	$0x2, %eax
-	jb	bad_input
-
-	cmp	$0x3, %eax
-	ja	bad_input
-
-	/* TCS must be non-NULL */
-	test	%rbx, %rbx
-	je	bad_input
-
-	/* Save @exception_info */
-	push	%rcx
-
-	/* Load AEP for ENCLU */
-	lea	1f(%rip),  %rcx
-1:	enclu
-
-	add	$0x8, %rsp
-	xor	%eax, %eax
-	ret
-
-bad_input:
-	mov     $(-EINVAL), %rax
-	ret
-
-.pushsection .fixup, "ax"
-	/* Re-load @exception_info and fill it (if it's non-NULL) */
-2:	pop	%rcx
-	test    %rcx, %rcx
-	je      3f
-
-	mov	%eax, EX_LEAF(%rcx)
-	mov	%di,  EX_TRAPNR(%rcx)
-	mov	%si,  EX_ERROR_CODE(%rcx)
-	mov	%rdx, EX_ADDRESS(%rcx)
-3:	mov	$(-EFAULT), %rax
-	ret
-.popsection
-
-_ASM_VDSO_EXTABLE_HANDLE(1b, 2b)
+    /* Prolog */
+    .cfi_startproc
+    push    %rbp
+    .cfi_adjust_cfa_offset  8
+    .cfi_rel_offset     %rbp, 0
+    mov %rsp, %rbp
+    .cfi_def_cfa_register   %rbp
+
+1:  /* EENTER <= leaf <= ERESUME */
+    cmp $0x2, %eax
+    jb  6f
+    cmp $0x3, %eax
+    ja  6f
+
+    /* Load TCS and AEP */
+    mov 0x10(%rbp), %rbx
+    lea 2f(%rip), %rcx
+
+    /* Single ENCLU serving as both EENTER and AEP (ERESUME) */
+2:  enclu
+
+    /* EEXIT path */
+    xor %ebx, %ebx
+3:  mov 0x18(%rbp), %rcx
+    jrcxz   4f
+    mov %eax, EX_LEAF(%rcx)
+    jnc 4f
+    mov %di, EX_TRAPNR(%rcx)
+    mov %si, EX_ERROR_CODE(%rcx)
+    mov %rdx, EX_ADDRESS(%rcx)
+
+4:  /* Call *callback if supplied */
+    mov 0x20(%rbp), %rax
+    test    %rax, %rax
+    /*
+     * At this point, %ebx holds the effective return value, which shall be
+     * returned if no callback is specified
+     */
+    cmovz   %rbx, %rax
+    jz  7f
+    /*
+     * Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be
+     * restored after *callback returns.
+     */
+    mov %rsp, %rbx
+    and $-0x10, %rsp
+    /* Clear RFLAGS.DF per x86_64 ABI */
+    cld
+    /* Parameters for *callback */
+    push    %rbx
+    push    0x10(%rbp)
+    /* Call *%rax via retpoline */
+    call    40f
+    /*
+     * Restore %rsp to its original value left off by the enclave from last
+     * exit
+     */
+    mov %rbx, %rsp
+    /*
+     * Positive return value from *callback will be interpreted as an ENCLU
+     * leaf, while a non-positive value will be interpreted as the return
+     * value to be passed back to the caller.
+     */
+    jmp 1b
+40: /* retpoline */
+    call    42f
+41: pause
+    lfence
+    jmp 41b
+42: mov %rax, (%rsp)
+    ret
+
+5:  /* Exception path */
+    mov $-EFAULT, %ebx
+    stc
+    jmp 3b
+
+6:  /* Unsupported ENCLU leaf */
+    cmp $0, %eax
+    jle 7f
+    mov $-EINVAL, %eax
+
+7:  /* Epilog */
+    leave
+    .cfi_def_cfa        %rsp, 8
+    ret
+    .cfi_endproc
+
+_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)
 
 ENDPROC(__vdso_sgx_enter_enclave)
arch/x86/include/uapi/asm/sgx.h +7 -7
index 9ed690a38c70..50d2b5143e5e 100644
@@ -24,7 +24,7 @@
 
 /**
  * struct sgx_enclave_create - parameter structure for the
- *                             %SGX_IOC_ENCLAVE_CREATE ioctl
+ *			       %SGX_IOC_ENCLAVE_CREATE ioctl
  * @src:	address for the SECS page data
  */
 struct sgx_enclave_create  {
@@ -33,7 +33,7 @@ struct sgx_enclave_create  {
 
 /**
  * struct sgx_enclave_add_page - parameter structure for the
- *                               %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
+ *				 %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
  * @addr:	address within the ELRANGE
  * @src:	address for the page data
  * @secinfo:	address for the SECINFO data
@@ -49,7 +49,7 @@ struct sgx_enclave_add_page {
 
 /**
  * struct sgx_enclave_init - parameter structure for the
- *                           %SGX_IOC_ENCLAVE_INIT ioctl
+ *			     %SGX_IOC_ENCLAVE_INIT ioctl
  * @sigstruct:	address for the SIGSTRUCT data
  */
 struct sgx_enclave_init {
@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute {
 };
 
 /**
- * struct sgx_enclave_exception - structure to report exceptions encountered in
- *				  __vdso_sgx_enter_enclave()
+ * struct sgx_enclave_exinfo - structure to report exceptions encountered in
+ *			       __vdso_sgx_enter_enclave()
  *
- * @leaf:	ENCLU leaf from \%eax at time of exception
+ * @leaf:	ENCLU leaf from \%eax at time of exception/exit
  * @trapnr:	exception trap number, a.k.a. fault vector
  * @error_code:	exception error code
  * @address:	exception address, e.g. CR2 on a #PF
  * @reserved:	reserved for future use
  */
-struct sgx_enclave_exception {
+struct sgx_enclave_exinfo {
 	__u32 leaf;
 	__u16 trapnr;
 	__u16 error_code;
Signature · 1 line
2.17.1