Intel SGX1 support
Subsystems (9)
RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonQuoted 49 lines
From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 10:55 AMQuoted 2 lines · Sean Christopherson
Hmm, I've been thinking more about pulling permissions from the source page. Conceptually I'm not sure we need to meet the same requirements
as
Quoted 1 line
non-enclave DSOs while the enclave is being built, i.e. do we really
need
Quoted 3 lines
to force userspace to fully map the enclave in normal memory?
Consider the Graphene scenario where it's building an enclave on the
fly.
Quoted 2 lines
Pulling permissions from the source VMAs means Graphene has to map the code pages of the enclave with X. This means Graphene will need
EXEDMOD
Quoted 1 line
(or EXECMEM if Graphene isn't careful). In a non-SGX scenario this
makes
Quoted 3 lines
perfect sense since there is no way to verify the end result of
RW->RX.But for SGX, assuming enclaves are whitelisted by their sigstruct
(checked
Quoted 1 line
at EINIT) and because page permissions affect sigstruct.MRENCLAVE, it
*is*
Quoted 1 line
possible to verify the resulting RX contents. E.g. for the purposes
of
Quoted 9 lines
LSMs, can't we use the .sigstruct file as a proxy for the enclave and require
FILE__EXECUTEon the .sigstruct inode to map/run the enclave?Stephen, is my logic sound?
If so...
- Require FILE__READ+FILE__EXECUTE on .sigstruct to
mmap()theenclave.
Quoted 2 lines
- Prevent userspace from mapping the enclave with permissions beyond
the
Quoted 1 line
original permissions of the enclave. This can be done by
populating
Quoted 1 line
VM_MAY{READ,WRITE,EXEC} from the SECINFO (same basic concept as
Andy's
Quoted 1 line
proposals). E.g. pre-EINIT,
mmap()andmprotect()can onlysucceed
Quoted 3 lines
with
PROT_NONE.- Require FILE__{READ,WRITE,EXECUTE} on
/dev/sgx/enclavefor
Quoted 8 lines
simplicity,
Quoted 1 line
or provide an alternate
SGX_IOC_MPROTECTif we want to sidestepthe
Quoted 1 line
FILE__WRITErequirement.One more thought. EADD (and the equivalent SGX2 flow) could do
security_mmap_file()with a NULL file on the SECINFO permissions, which would triggerPROCESS_EXECMEMif an enclave attempts to map a page RWX.
If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so PROCESS__EXECMEM will have been checked. For SGX2, changing an EPC page to RWX is subject to FILE__EXECMEM on /dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.
Quoted 5 lines
Quoted 2 lines
No changes are required to LSMs, SGX1 has a single LSM touchpoint in
its
Quoted 2 lines
mmap(), and I *think* the only required userspace change is tommap()PROT_NONEwhen allocating the enclave's virtual address range.
I'm not sure I understand the motivation behind this proposal to decouple initial EPC permissions from source pages.
I don't think it a big deal to fully mmap() enclave files, which have to be parsed by user mode anyway to determine various things including but not limited to the size of heap(s), size and number of TCSs/stacks/TLS areas, and the overall enclave size. So with PHDRs parsed, it's trivial to mmap() each segment with permissions from its PHDR.
Quoted 4 lines
Quoted 3 lines
As for Graphene, it doesn't need extra permissions to run its enclaves, it just needs a way to install .sigstruct, which is a generic
permissions
Quoted 15 lines
Quoted 7 lines
problem and not SGX specific.
For SGX2 maybe:
- No additional requirements to map an EAUG'd page as RW page. Not aligned with standard
MAP_SHAREDbehavior, but we really don'twant
Quoted 7 lines
to require
FILE__WRITE, and thus allow writes to .sigstruct.- Require
FILE__EXECMODon the .sigstruct to map previously writable page as executable (which indirectly includes all EAUG'd pages). Wiring this up will be a little funky, but we again we don't want to requireFILE__WRITEon .sigstruct.
I'm lost. Why is EAUG tied to permissions on .sigstruct?
-Cedric
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 39 lines · Cedric
Quoted 39 lines
From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 10:55 AMQuoted 33 lines · Sean Christopherson
Hmm, I've been thinking more about pulling permissions from the source page. Conceptually I'm not sure we need to meet the same requirements as non-enclave DSOs while the enclave is being built, i.e. do we really need to force userspace to fully map the enclave in normal memory?
Consider the Graphene scenario where it's building an enclave on the fly. Pulling permissions from the source VMAs means Graphene has to map the code pages of the enclave with X. This means Graphene will need EXEDMOD (or EXECMEM if Graphene isn't careful). In a non-SGX scenario this makes perfect sense since there is no way to verify the end result of
RW->RX.But for SGX, assuming enclaves are whitelisted by their sigstruct (checked at EINIT) and because page permissions affect sigstruct.MRENCLAVE, it *is* possible to verify the resulting RX contents. E.g. for the purposes of LSMs, can't we use the .sigstruct file as a proxy for the enclave and require
FILE__EXECUTEon the .sigstruct inode to map/run the enclave?Stephen, is my logic sound?
If so...
- Require FILE__READ+FILE__EXECUTE on .sigstruct to
mmap()the enclave.- Prevent userspace from mapping the enclave with permissions beyond the original permissions of the enclave. This can be done by populating VM_MAY{READ,WRITE,EXEC} from the SECINFO (same basic concept as Andy's proposals). E.g. pre-EINIT,
mmap()andmprotect()can only succeed withPROT_NONE.- Require FILE__{READ,WRITE,EXECUTE} on
/dev/sgx/enclavefor simplicity, or provide an alternateSGX_IOC_MPROTECTif we want to sidestep theFILE__WRITErequirement.
Quoted 8 lines
Quoted 3 lines
One more thought. EADD (and the equivalent SGX2 flow) could do
security_mmap_file()with a NULL file on the SECINFO permissions, which would triggerPROCESS_EXECMEMif an enclave attempts to map a page RWX.If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.
My point is that enclaves have different properties than shared objects.
Normal LSM behavior with regard to executing files is to label files with e.g. FILE__EXECUTE. Because an enclave must be built to the exact specifications of .sigstruct, requring FILE__EXECUTE on the .sigstruct is effectively the same as requiring FILE__EXECUTE on the enclave itself.
Addressing your scenario of loading an executable page in EPC, doing so would require one of the following:
- Ability to install a .sigstruct with FILE__EXECUTE
- PROCESS__EXECMEM
- FILE__EXECMOD and SGX2 support
Quoted 5 lines
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so
PROCESS__EXECMEMwill have been checked. For SGX2, changing an EPC page to RWX is subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.
There is no FILE__EXECMEM check, only PROCESS__EXECMEM and FILE__EXECMOD. I assume you're referring to the latter?
I don't see a fundamental difference between having RWX in an enclave and RWX in normal memory, either way the process can execute arbitrary code, i.e. PROCESS__EXECMEM is appropriate. Yes, an enclave will #UD on certain instructions, but that's easily sidestepped by having a trampoline in the host (marked RX) and piping arbitrary code into the enclave. Or using EEXIT to do a bit of ROP.
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
No changes are required to LSMs, SGX1 has a single LSM touchpoint in
its
Quoted 2 lines
mmap(), and I *think* the only required userspace change is tommap()PROT_NONEwhen allocating the enclave's virtual address range.I'm not sure I understand the motivation behind this proposal to decouple initial EPC permissions from source pages.
Pulling permissions from source pages means userspace needs to fully map the in normal memory, including marking pages executable. That exposes the loader to having executable pages in its address space that it has no intention of executing (outside of the enclave). And for Graphene, it means having to actively avoid PROCESS__EXECMEM, e.g. by using a dummy backing file to build the enclave instead of anon memory.
Quoted 17 lines
I don't think it a big deal to fully
mmap()enclave files, which have to be parsed by user mode anyway to determine various things including but not limited to the size of heap(s), size and number of TCSs/stacks/TLS areas, and the overall enclave size. So with PHDRs parsed, it's trivial tommap()each segment with permissions from its PHDR.Quoted 11 lines
Quoted 11 lines
As for Graphene, it doesn't need extra permissions to run its enclaves, it just needs a way to install .sigstruct, which is a generic permissions problem and not SGX specific.
For SGX2 maybe:
- No additional requirements to map an EAUG'd page as RW page. Not aligned with standard
MAP_SHAREDbehavior, but we really don't want to requireFILE__WRITE, and thus allow writes to .sigstruct.
Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
- Require
FILE__EXECMODon the .sigstruct to map previously writable page as executable (which indirectly includes all EAUG'd pages). Wiring this up will be a little funky, but we again we don't want to requireFILE__WRITEon .sigstruct.I'm lost. Why is EAUG tied to permissions on .sigstruct?
Because for the purposes of LSM checks, .sigstruct is the enclave's backing file, and mapping a previously writable enclave page as exectuable is roughly equivalent to mapping a CoW'd page as exectuable.
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 6 lines · Sean Christopherson
Quoted 3 lines · Cedric
Quoted 3 lines
From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 10:55 AM
Quoted 6 lines
I don't see a fundamental difference between having RWX in an enclave and RWX in normal memory, either way the process can execute arbitrary code, i.e.
PROCESS__EXECMEMis appropriate. Yes, an enclave will #UD on certain instructions, but that's easily sidestepped by having a trampoline in the host (marked RX) and piping arbitrary code into the enclave. Or using EEXIT to do a bit of ROP.
There's a difference, albeit a somewhat weak one, if sigstructs are whitelisted. FILE__EXECMOD on
either /dev/sgx/enclave or on the sigstruct is not an entirely crazy way to express this.
RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonQuoted 11 lines
From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 12:14 PMMy point is that enclaves have different properties than shared objects.
Normal LSM behavior with regard to executing files is to label files with e.g.
FILE__EXECUTE. Because an enclave must be built to the exact specifications of .sigstruct, requringFILE__EXECUTEon the .sigstruct is effectively the same as requiringFILE__EXECUTEon the enclave itself.
Quoted 8 lines
Addressing your scenario of loading an executable page in EPC, doing so would require one of the following:
- Ability to install a .sigstruct with
FILE__EXECUTE-
PROCESS__EXECMEM-
FILE__EXECMODand SGX2 support
Now I got your point. It sounds a great idea to me!
But instead of using .sigstruct file, I'd still recommend using file mapping (i.e. SIGSTRUCT needs to reside in executable memory). But then there'll be a hole - a process having FILE__EXECMOD on any file could use that file as a SIGSTRUCT. Probably we'll need a new type in SELinux to label enclave/sigstruct files.
Quoted 10 lines
Quoted 3 lines
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so
PROCESS__EXECMEMwill havebeen checked.
Quoted 3 lines
For SGX2, changing an EPC page to RWX is subject to
FILE__EXECMEMon/dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.There is no
FILE__EXECMEMcheck, onlyPROCESS__EXECMEMandFILE__EXECMOD. I assume you're referring to the latter?
Yes.
Quoted 7 lines
I don't see a fundamental difference between having RWX in an enclave and RWX in normal memory, either way the process can execute arbitrary code, i.e.
PROCESS__EXECMEMis appropriate. Yes, an enclave will #UD on certain instructions, but that's easily sidestepped by having a trampoline in the host (marked RX) and piping arbitrary code into the enclave. Or using EEXIT to do a bit of ROP.
I'm with you.
With your proposal only FILE__EXECMOD is needed on /dev/sgx/enclave to launch Graphene enclaves or the like.
Quoted 11 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
No changes are required to LSMs, SGX1 has a single LSM touchpoint in
its
Quoted 2 lines
mmap(), and I *think* the only required userspace change is tommap()PROT_NONEwhen allocating the enclave's virtual addressrange.
Quoted 3 lines
I'm not sure I understand the motivation behind this proposal to decouple initial EPC permissions from source pages.
Quoted 6 lines
Pulling permissions from source pages means userspace needs to fully map the in normal memory, including marking pages executable. That exposes the loader to having executable pages in its address space that it has no intention of executing (outside of the enclave). And for Graphene, it means having to actively avoid
PROCESS__EXECMEM, e.g. by using a dummy backing file to build the enclave instead of anon memory.
Agreed.
Quoted 27 lines
Quoted 6 lines
I don't think it a big deal to fully
mmap()enclave files, which have to be parsed by user mode anyway to determine various things including but not limited to the size of heap(s), size and number of TCSs/stacks/TLS areas, and the overall enclave size. So with PHDRs parsed, it's trivial tommap()each segment with permissions from itsPHDR.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
As for Graphene, it doesn't need extra permissions to run its enclaves, it just needs a way to install .sigstruct, which is a generic permissions problem and not SGX specific.
For SGX2 maybe:
- No additional requirements to map an EAUG'd page as RW page.
Not
Quoted 1 line
Quoted 1 line
Quoted 1 line
aligned with standard
MAP_SHAREDbehavior, but we really don'twant
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to require
FILE__WRITE, and thus allow writes to .sigstruct.- Require
FILE__EXECMODon the .sigstruct to map previouslywritable
Quoted 1 line
Quoted 1 line
Quoted 1 line
page as executable (which indirectly includes all EAUG'd
pages).
Quoted 1 line
Quoted 1 line
Quoted 1 line
Wiring this up will be a little funky, but we again we don't
want
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
to require
FILE__WRITEon .sigstruct.I'm lost. Why is EAUG tied to permissions on .sigstruct?
Because for the purposes of LSM checks, .sigstruct is the enclave's backing file, and mapping a previously writable enclave page as exectuable is roughly equivalent to mapping a CoW'd page as exectuable.
I think I've got your idea. You are trying to use permissions on .sigstruct to determine whether EAUG will be available to that specific enclave. Am I right?
I'd tie EAUG to the permissions of /dev/sgx/enclave instead. But why? There are couple of reasons. For one, a SIGSTRUCT identifies the behavior of the enclave, hence the SGX features needed by that enclave. So if an enclave requires EAUG, the .sigstruct has to allow EAUG or the enclave wouldn't work. That means the system admin wouldn't have a choice but to match up what's needed by the enclave. For two, whether to allow, say loading code dynamically into an enclave, depends on whether the host process can tolerate the inherent risk. And that decision is seldom made on individual enclaves but to the host process as a whole. And /dev/sgx/enclave serves that purpose.
-Cedric
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 16 lines · Cedric
Quoted 16 lines
From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 12:14 PMMy point is that enclaves have different properties than shared objects.
Normal LSM behavior with regard to executing files is to label files with e.g.
FILE__EXECUTE. Because an enclave must be built to the exact specifications of .sigstruct, requringFILE__EXECUTEon the .sigstruct is effectively the same as requiringFILE__EXECUTEon the enclave itself.Addressing your scenario of loading an executable page in EPC, doing so would require one of the following:
- Ability to install a .sigstruct with
FILE__EXECUTE
Quoted 8 lines
Quoted 3 lines
-
PROCESS__EXECMEM-
FILE__EXECMODand SGX2 supportNow I got your point. It sounds a great idea to me!
But instead of using .sigstruct file, I'd still recommend using file mapping (i.e. SIGSTRUCT needs to reside in executable memory). But then there'll be a
Why? Even in the Graphene case the final .sigstruct can be known ahead of time. Userspace can always use memfd() if it's generating SIGSTRUCT on the fly.
Quoted 11 lines
hole - a process having
FILE__EXECMODon any file could use that file as a SIGSTRUCT. Probably we'll need a new type in SELinux to label enclave/sigstruct files.Quoted 6 lines
I don't see a fundamental difference between having RWX in an enclave and RWX in normal memory, either way the process can execute arbitrary code, i.e.
PROCESS__EXECMEMis appropriate. Yes, an enclave will #UD on certain instructions, but that's easily sidestepped by having a trampoline in the host (marked RX) and piping arbitrary code into the enclave. Or using EEXIT to do a bit of ROP.
Quoted 4 lines
I'm with you.
With your proposal only
FILE__EXECMODis needed on/dev/sgx/enclaveto launch Graphene enclaves or the like.
It wouldn't even need FILE__EXECMOD, assuming Graphene does all of its libc rewriting before building the enclave, i.e. doesn't EADD RWX pages.
Quoted 51 lines
Quoted 10 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
No changes are required to LSMs, SGX1 has a single LSM touchpoint in
its
Quoted 2 lines
mmap(), and I *think* the only required userspace change is tommap()PROT_NONEwhen allocating the enclave's virtual addressrange.
Quoted 3 lines
I'm not sure I understand the motivation behind this proposal to decouple initial EPC permissions from source pages.
Quoted 6 lines
Pulling permissions from source pages means userspace needs to fully map the in normal memory, including marking pages executable. That exposes the loader to having executable pages in its address space that it has no intention of executing (outside of the enclave). And for Graphene, it means having to actively avoid
PROCESS__EXECMEM, e.g. by using a dummy backing file to build the enclave instead of anon memory.Agreed.
Quoted 32 lines
Quoted 6 lines
I don't think it a big deal to fully
mmap()enclave files, which have to be parsed by user mode anyway to determine various things including but not limited to the size of heap(s), size and number of TCSs/stacks/TLS areas, and the overall enclave size. So with PHDRs parsed, it's trivial tommap()each segment with permissions from itsPHDR.
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
As for Graphene, it doesn't need extra permissions to run its enclaves, it just needs a way to install .sigstruct, which is a generic permissions problem and not SGX specific.
For SGX2 maybe:
- No additional requirements to map an EAUG'd page as RW page.
Not
Quoted 1 line
Quoted 1 line
Quoted 1 line
aligned with standard
MAP_SHAREDbehavior, but we really don'twant
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
to require
FILE__WRITE, and thus allow writes to .sigstruct.- Require
FILE__EXECMODon the .sigstruct to map previouslywritable
Quoted 1 line
Quoted 1 line
Quoted 1 line
page as executable (which indirectly includes all EAUG'd
pages).
Quoted 1 line
Quoted 1 line
Quoted 1 line
Wiring this up will be a little funky, but we again we don't
want
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
to require
FILE__WRITEon .sigstruct.I'm lost. Why is EAUG tied to permissions on .sigstruct?
Quoted 7 lines
Quoted 3 lines
Because for the purposes of LSM checks, .sigstruct is the enclave's backing file, and mapping a previously writable enclave page as exectuable is roughly equivalent to mapping a CoW'd page as exectuable.
I think I've got your idea. You are trying to use permissions on .sigstruct to determine whether EAUG will be available to that specific enclave. Am I right?
Yep.
Quoted 9 lines
I'd tie EAUG to the permissions of
/dev/sgx/enclaveinstead. But why? There are couple of reasons. For one, a SIGSTRUCT identifies the behavior of the enclave, hence the SGX features needed by that enclave. So if an enclave requires EAUG, the .sigstruct has to allow EAUG or the enclave wouldn't work. That means the system admin wouldn't have a choice but to match up what's needed by the enclave. For two, whether to allow, say loading code dynamically into an enclave, depends on whether the host process can tolerate the inherent risk. And that decision is seldom made on individual enclaves but to the host process as a whole. And/dev/sgx/enclaveserves that purpose.
I think I'd be ok either way? What I really care about is having line of sight to a sane way to support for SGX2, and both seem sane. I.e. we can hash this detail out when SGX2 goes in.
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Xing, CedricQuoted 55 lines · Cedric
Quoted 55 lines
From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 10:55 AMQuoted 2 lines · Sean Christopherson
Hmm, I've been thinking more about pulling permissions from the source page. Conceptually I'm not sure we need to meet the same requirements
as
Quoted 1 line
non-enclave DSOs while the enclave is being built, i.e. do we really
need
Quoted 3 lines
to force userspace to fully map the enclave in normal memory?
Consider the Graphene scenario where it's building an enclave on the
fly.
Quoted 2 lines
Pulling permissions from the source VMAs means Graphene has to map the code pages of the enclave with X. This means Graphene will need
EXEDMOD
Quoted 1 line
(or EXECMEM if Graphene isn't careful). In a non-SGX scenario this
makes
Quoted 3 lines
perfect sense since there is no way to verify the end result of
RW->RX.But for SGX, assuming enclaves are whitelisted by their sigstruct
(checked
Quoted 1 line
at EINIT) and because page permissions affect sigstruct.MRENCLAVE, it
*is*
Quoted 1 line
possible to verify the resulting RX contents. E.g. for the purposes
of
Quoted 9 lines
LSMs, can't we use the .sigstruct file as a proxy for the enclave and require
FILE__EXECUTEon the .sigstruct inode to map/run the enclave?Stephen, is my logic sound?
If so...
- Require FILE__READ+FILE__EXECUTE on .sigstruct to
mmap()theenclave.
Quoted 2 lines
- Prevent userspace from mapping the enclave with permissions beyond
the
Quoted 1 line
original permissions of the enclave. This can be done by
populating
Quoted 1 line
VM_MAY{READ,WRITE,EXEC} from the SECINFO (same basic concept as
Andy's
Quoted 1 line
proposals). E.g. pre-EINIT,
mmap()andmprotect()can onlysucceed
Quoted 3 lines
with
PROT_NONE.- Require FILE__{READ,WRITE,EXECUTE} on
/dev/sgx/enclaveforsimplicity,
Quoted 1 line
or provide an alternate
SGX_IOC_MPROTECTif we want to sidestepthe
Quoted 1 line
FILE__WRITErequirement.
Quoted 7 lines
Quoted 3 lines
One more thought. EADD (and the equivalent SGX2 flow) could do
security_mmap_file()with a NULL file on the SECINFO permissions, which would triggerPROCESS_EXECMEMif an enclave attempts to map a page RWX.If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so
PROCESS__EXECMEMwill have been checked. For SGX2, changing an EPC page to RWX is subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.
So the permission would be like FILE__EXECMOD on the source enclave page, because it would be mapped MAP_ANONYMOUS, PROT_WRITE? MAP_SHARED, PROT_WRITE isn't going to work because that means you can modify the file.
I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:
int security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);
Then you don't have to muck with mapping anything PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)
(I'm proposing passing in the source_vma because this hook would be called with mmap_sem held for read to avoid a TOCTOU race.)
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around mprotect() calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.
--Andy
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 13 lines · Andy Lutomirski
Quoted 11 lines · Cedric
If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so
PROCESS__EXECMEMwill have been checked. For SGX2, changing an EPC page to RWX is subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.
Quoted 4 lines
So the permission would be like
FILE__EXECMODon the source enclave page, because it would be mappedMAP_ANONYMOUS,PROT_WRITE?MAP_SHARED,PROT_WRITEisn't going to work because that means you can modify the file.
Was this in response to Cedric's comment, or to my comment?
Quoted 19 lines
I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:
int
security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);Then you don't have to muck with mapping anything
PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)(I'm proposing passing in the
source_vmabecause this hook would be called withmmap_semheld for read to avoid a TOCTOU race.)
Quoted 7 lines
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around
mprotect()calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.
This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?
What's the value prop beyond whitelisting sigstructs? Realistically, I doubt LSMs/users will want to take the performance hit of scanning the source bytes every time an enclave is loaded.
We could add seomthing like security_enclave_mprotect() in lieu of abusing security_file_mprotect(), but passing the full source bytes seems a bit much.
RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 19 lines
From: [email protected] [mailto:linux-sgx- [email protected]] On Behalf Of Sean Christopherson
Sent: Friday, May 24, 2019 1:04 PMQuoted 1 line · Andy Lutomirski
On Fri, May 24, 2019 at 11:34 AM Xing, Cedric <[email protected]>
Quoted 5 lines · wrote:
Quoted 5 lines
If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable,
as an executable page in EPC.
Quoted 6 lines
Quoted 6 lines
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so
PROCESS__EXECMEMwill have been checked. For SGX2, changing an EPC page to RWX is subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole
Quoted 8 lines
process hosting it.
Quoted 5 lines
So the permission would be like
FILE__EXECMODon the source enclave page, because it would be mappedMAP_ANONYMOUS,PROT_WRITE?MAP_SHARED,PROT_WRITEisn't going to work because that means you can modify the file.Was this in response to Cedric's comment, or to my comment?
Creating RWX source page requires PROCESS_EXECMEM. But as I responded to Sean earlier, I think his proposal of "aggregating" all "initial" permission checks into a single SIGSTRUCT check is probably a better approach.
Quoted 10 lines
Quoted 10 lines
I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:
int
security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);
This is exactly what I was thinking. But with Sean's proposal this is probably no longer necessary.
Quoted 18 lines
Quoted 10 lines
Then you don't have to muck with mapping anything
PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)(I'm proposing passing in the
source_vmabecause this hook would be called withmmap_semheld for read to avoid a TOCTOU race.)Quoted 7 lines
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around
mprotect()calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.
Quoted 3 lines
This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?
In the context of SELinux, new types could be defined to be associated with SIGSTRUCT (or more precisely, files containing SIGSTRUCTs). Then the LSM hook (I'd propose security_sgx_initialize_enclave) could enforce whatever...
Quoted 8 lines
What's the value prop beyond whitelisting sigstructs? Realistically, I doubt LSMs/users will want to take the performance hit of scanning the source bytes every time an enclave is loaded.
We could add seomthing like
security_enclave_mprotect()in lieu of abusingsecurity_file_mprotect(), but passing the full source bytes seems a bit much.
I'd just use /dev/sgx/enclave to govern "runtime" permissions any EPC page can mmap()/mprotect() to. Then we won't need any code changes in LSM.
-Cedric
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonQuoted 15 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 11 lines · Cedric
If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so
PROCESS__EXECMEMwill have been checked. For SGX2, changing an EPC page to RWX is subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.
Quoted 6 lines
Quoted 4 lines
So the permission would be like
FILE__EXECMODon the source enclave page, because it would be mappedMAP_ANONYMOUS,PROT_WRITE?MAP_SHARED,PROT_WRITEisn't going to work because that means you can modify the file.Was this in response to Cedric's comment, or to my comment?
Yours. I think that requiring source pages to be actually mapped W is not such a great idea.
Quoted 28 lines
Quoted 20 lines
I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:
int
security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);Then you don't have to muck with mapping anything
PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)(I'm proposing passing in the
source_vmabecause this hook would be called withmmap_semheld for read to avoid a TOCTOU race.)Quoted 7 lines
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around
mprotect()calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.
Quoted 3 lines
This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?
In addition.
But I named the function badly and gave it a bad signature, which confused you. Let's try again:
int security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);
Maybe some really fancy future LSM would also want loff_t source_offset, but it's probably not terribly useful. This same callback would be used for EAUG.
Following up on your discussion with Cedric about sigstruct, the other callback would be something like:
int security_enclave_init(struct file *sigstruct_file);
The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:
int security_enclave_load_zeros(unsigned int maxperm);
An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.
But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.
Am I making more sense now?
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 22 lines · Andy Lutomirski
Quoted 20 lines · Sean Christopherson
Quoted 17 lines · Andy Lutomirski
Quoted 11 lines · Cedric
If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so
PROCESS__EXECMEMwill have been checked. For SGX2, changing an EPC page to RWX is subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.So the permission would be like
FILE__EXECMODon the source enclave page, because it would be mappedMAP_ANONYMOUS,PROT_WRITE?MAP_SHARED,PROT_WRITEisn't going to work because that means you can modify the file.
Quoted 4 lines
Quoted 1 line
Was this in response to Cedric's comment, or to my comment?
Yours. I think that requiring source pages to be actually mapped W is not such a great idea.
I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:
if ((SECINFO.PERMS & RWX) == RWX) {
ret = security_mmap_file(NULL, RWX, ???);
if (ret)
return ret;
}But that's a moot point if we add security_enclave_load() or whatever.
Quoted 33 lines
Quoted 29 lines
Quoted 21 lines
I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:
int
security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);Then you don't have to muck with mapping anything
PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)(I'm proposing passing in the
source_vmabecause this hook would be called withmmap_semheld for read to avoid a TOCTOU race.)Quoted 7 lines
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around
mprotect()calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.Quoted 3 lines
This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?
Quoted 7 lines
In addition.
But I named the function badly and gave it a bad signature, which confused you. Let's try again:
int
security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);
I prefer security_enclave_load(), "from_memory" seems redundant at best.
Quoted 7 lines
Maybe some really fancy future LSM would also want
loff_tsource_offset, but it's probably not terribly useful. This same callback would be used for EAUG.Following up on your discussion with Cedric about sigstruct, the other callback would be something like:
Quoted 7 lines
int
security_enclave_init(struct file *sigstruct_file);The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:
int
security_enclave_load_zeros(unsigned int maxperm);
What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.
Quoted 9 lines
An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.
But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.
Am I making more sense now?
Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:
int security_enclave_load(struct file *file, struct vm_area_struct *vma,
unsigned long prot);
And for ultimate flexibility we could pass both .sigstruct and the file pointer for /dev/sgx/enclave, but that seems a bit ridiculous.
Passing both would allow tying EXECMOD to /dev/sgx/enclave as Cedric wanted (without having to play games and pass /dev/sgx/enclave to security_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 80 lines · Sean Christopherson
Quoted 24 lines · Andy Lutomirski
Quoted 20 lines · Sean Christopherson
Quoted 19 lines · Andy Lutomirski
Quoted 7 lines · Cedric
If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable page in EPC.
Quoted 5 lines
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so
PROCESS__EXECMEMwill have been checked. For SGX2, changing an EPC page to RWX is subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hosting it.So the permission would be like
FILE__EXECMODon the source enclave page, because it would be mappedMAP_ANONYMOUS,PROT_WRITE?MAP_SHARED,PROT_WRITEisn't going to work because that means you can modify the file.Quoted 4 lines
Quoted 1 line
Was this in response to Cedric's comment, or to my comment?
Yours. I think that requiring source pages to be actually mapped W is not such a great idea.
I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:
if ((SECINFO.PERMS & RWX) == RWX) { ret = security_mmap_file(NULL, RWX, ???); if (ret) return ret;}
But that's a moot point if we add
security_enclave_load()or whatever.Quoted 35 lines
Quoted 29 lines
Quoted 21 lines
I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:
int
security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);Then you don't have to muck with mapping anything
PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)(I'm proposing passing in the
source_vmabecause this hook would be called withmmap_semheld for read to avoid a TOCTOU race.)Quoted 7 lines
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around
mprotect()calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.Quoted 3 lines
This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?
In addition.
Quoted 7 lines
Quoted 5 lines
But I named the function badly and gave it a bad signature, which confused you. Let's try again:
int
security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);I prefer
security_enclave_load(), "from_memory" seems redundant at best.
Fine with me.
Quoted 14 lines
Quoted 14 lines
Maybe some really fancy future LSM would also want
loff_tsource_offset, but it's probably not terribly useful. This same callback would be used for EAUG.Following up on your discussion with Cedric about sigstruct, the other callback would be something like:
int
security_enclave_init(struct file *sigstruct_file);The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:
Quoted 6 lines
Quoted 1 line
int
security_enclave_load_zeros(unsigned int maxperm);What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.
This is how EAUG could ask if RWX is okay. If an enclave is internally doing dynamic loading, the it will need a heap page with maxperm = RWX. (If it’s well designed, it will make it RW and then RX, either by changing SECINFO or by asking the host to mprotect() it, but it still needs the overall RWX mask.).
Also, do real SGX1 enclave formats have BSS? If so, then either we need an ioctl or load zeros or user code is going to load from /dev/zero or just from the heap, but the LSM is going to play better with an ioctl, I suspect :)
Quoted 11 lines
Quoted 10 lines
An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.
But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.
Am I making more sense now?
Quoted 8 lines
Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:
int security_enclave_load(struct file *file, struct
vm_area_struct*vma,unsigned long prot);And for ultimate flexibility we could pass both .sigstruct and the file pointer for
/dev/sgx/enclave, but that seems a bit ridiculous.
I agree.
Quoted 6 lines
Passing both would allow tying EXECMOD to
/dev/sgx/enclaveas Cedric wanted (without having to play games and pass/dev/sgx/enclavetosecurity_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.
The benefit of putting it on .sigstruct is that it can be per-enclave.
As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty look.
This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.
(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)
RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 86 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Friday, May 24, 2019 4:42 PMQuoted 6 lines · Sean Christopherson
Quoted 5 lines · Andy Lutomirski
Quoted 1 line · Sean Christopherson
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Andy Lutomirski
Quoted 6 lines · Cedric
If "initial permissions" for enclaves are less restrictive than
shared objects, then it'd become a backdoor for circumventing LSM
when enclave whitelisting is *not* in place. For example, an
adversary may load a page, which would otherwise never be executable, as an executablepage in EPC.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In the case a RWX page is needed, the calling process has to have
a RWX page serving as the source for EADD soPROCESS__EXECMEMwill
have been checked. For SGX2, changing an EPC page to RWX is
subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a
security benefit because it only affects the enclave but not the whole process hostingit.
Quoted 62 lines
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
So the permission would be like
FILE__EXECMODon the source enclave page, because it would be mappedMAP_ANONYMOUS,PROT_WRITE?MAP_SHARED,PROT_WRITEisn't going to work because that means you can modify the file.Was this in response to Cedric's comment, or to my comment?
Yours. I think that requiring source pages to be actually mapped W is not such a great idea.
I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:
if ((SECINFO.PERMS & RWX) == RWX) { ret = security_mmap_file(NULL, RWX, ???); if (ret) return ret;}
But that's a moot point if we add
security_enclave_load()or whatever.Quoted 38 lines
Quoted 29 lines
Quoted 21 lines
I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:
int
security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);Then you don't have to muck with mapping anything
PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)(I'm proposing passing in the
source_vmabecause this hook would be called withmmap_semheld for read to avoid a TOCTOU race.)Quoted 7 lines
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around
mprotect()calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.Quoted 3 lines
This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?
In addition.
But I named the function badly and gave it a bad signature, which confused you. Let's try again:
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
int
security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);I prefer
security_enclave_load(), "from_memory" seems redundant at best.Fine with me.
If we think of EADD as a way of mmap()'ing an enclave file into memory, would this security_enclave_load() be the same as security_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Maybe some really fancy future LSM would also want
loff_tsource_offset, but it's probably not terribly useful. This same callback would be used for EAUG.
EAUG always zeroes the EPC page before making it available to an enclave. So I don't think there's anything needed to done here.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Following up on your discussion with Cedric about sigstruct, the other callback would be something like:
int
security_enclave_init(struct file *sigstruct_file);
I'd still insist in using a pointer rather than a file, for reasons that we've discussed before. For those who can't recall, the major reason is that most implementation would embed SIGSTRUCT into the same file as the enclave (or at least I don't want to prevent anyone from doing so), which could also be part of another file, such as a shared object or even the main executable itself. It could be difficult to obtain a fd in those cases. memfd won't work because it can't retain the same attributes of the original file containing the SIGSTRUCT.
After all, what matters is the attributes associated with the backing file, which could be easily retrieve from vm_file of the covering VMA. So for the sake of flexibility, let's stay with what we've agreed before - a pointer to SIGSTRUCT.
Quoted 12 lines
Quoted 11 lines
Quoted 6 lines
The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:
int
security_enclave_load_zeros(unsigned int maxperm);What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.
Quoted 4 lines
This is how EAUG could ask if RWX is okay. If an enclave is internally doing dynamic loading, the it will need a heap page with maxperm = RWX. (If it’s well designed, it will make it RW and then RX, either by changing SECINFO or by asking the host to
mprotect()it, but it still needs the overall RWX mask.).
Any new page EAUG'ed will start in RW (as dictated by SGX ISA). EACCEPTCOPY will then change it to RX. RWX is never needed for all practical purposes. This in fact could be gated by mprotect() and the attributes associated with /dev/sgx/enclave. In the case of SELinux, FILE__EXECMOD is the right attribute and mprotect() will take care of all the rest. I don't see why the driver need a role here.
Quoted 4 lines
Also, do real SGX1 enclave formats have BSS? If so, then either we need an ioctl or load zeros or user code is going to load from
/dev/zeroor just from the heap, but the LSM is going to play better with an ioctl, I suspect :)
Yes, it does. But an enclave would either measure BSS, in which case the initial bytes have to be zero or MRENCLAVE will change; or zero BSS explicitly in its initialization code.
But from LSM's perspective it makes no difference than EADD'ing a page with non-zero content. And security_enclave_load(NULL, RW) would take care of it in exactly in the same way.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.
With SGX2/EDMM, RWX is *never* needed for all practical purposes.
In theory, in terms of security, no page shall be made executable while it is still being prepared. So W and X shall always be mutually exclusive, regardless it's in EPC or regular memory.
RWX is only needed in SGX1, as a workaround for certain usages, because EPCM permissions can never change at runtime.
Quoted 11 lines
Quoted 11 lines
Quoted 7 lines
But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.
Am I making more sense now?
Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:
Quoted 7 lines
Quoted 5 lines
int security_enclave_load(struct file *file, struct
vm_area_struct*vma,unsigned long prot);And for ultimate flexibility we could pass both .sigstruct and the file pointer for
/dev/sgx/enclave, but that seems a bit ridiculous.I agree.
Loosely speaking, an enclave (including initial contents of all of its pages and their permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.
Theoretically speaking, if LSM can make a decision at EINIT by means of security_enclave_load(), then security_enclave_load() is never needed.
In practice, I support keeping both because security_enclave_load() can only approve an enumerable set while security_enclave_load() can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced by security_enclave_load().
Quoted 15 lines
Quoted 7 lines
Passing both would allow tying EXECMOD to
/dev/sgx/enclaveas Cedric wanted (without having to play games and pass/dev/sgx/enclavetosecurity_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.The benefit of putting it on .sigstruct is that it can be per-enclave.
As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty look.
Quoted 8 lines
This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.
(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)
I'm lost here. Dynamically linked enclaves, if running on SGX2, would need RW->RX, i.e. FILE__EXECMOD on /dev/sgx/enclave. But they never need RWX, i.e. PROCESS__EXECMEM.
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 87 lines · Cedric
Quoted 87 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Friday, May 24, 2019 4:42 PMQuoted 6 lines · Sean Christopherson
Quoted 5 lines · Andy Lutomirski
Quoted 1 line · Sean Christopherson
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines · Andy Lutomirski
Quoted 6 lines · Cedric
If "initial permissions" for enclaves are less restrictive than
shared objects, then it'd become a backdoor for circumventing LSM
when enclave whitelisting is *not* in place. For example, an
adversary may load a page, which would otherwise never be executable, as an executablepage in EPC.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
In the case a RWX page is needed, the calling process has to have
a RWX page serving as the source for EADD soPROCESS__EXECMEMwill
have been checked. For SGX2, changing an EPC page to RWX is
subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a
security benefit because it only affects the enclave but not the whole process hostingit.
Quoted 62 lines
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
So the permission would be like
FILE__EXECMODon the source enclave page, because it would be mappedMAP_ANONYMOUS,PROT_WRITE?MAP_SHARED,PROT_WRITEisn't going to work because that means you can modify the file.Was this in response to Cedric's comment, or to my comment?
Yours. I think that requiring source pages to be actually mapped W is not such a great idea.
I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:
if ((SECINFO.PERMS & RWX) == RWX) { ret = security_mmap_file(NULL, RWX, ???); if (ret) return ret;}
But that's a moot point if we add
security_enclave_load()or whatever.Quoted 38 lines
Quoted 29 lines
Quoted 21 lines
I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:
int
security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);Then you don't have to muck with mapping anything
PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)(I'm proposing passing in the
source_vmabecause this hook would be called withmmap_semheld for read to avoid a TOCTOU race.)Quoted 7 lines
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around
mprotect()calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.Quoted 3 lines
This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?
In addition.
But I named the function badly and gave it a bad signature, which confused you. Let's try again:
Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
int
security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);I prefer
security_enclave_load(), "from_memory" seems redundant at best.Fine with me.
If we think of EADD as a way of mmap()'ing an enclave file into memory, would this
security_enclave_load()be the same assecurity_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?
Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Maybe some really fancy future LSM would also want
loff_tsource_offset, but it's probably not terribly useful. This same callback would be used for EAUG.EAUG always zeroes the EPC page before making it available to an enclave. So I don't think there's anything needed to done here.
Duh. So security_enclave_load_zeros() for EAUG. See below.
Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Following up on your discussion with Cedric about sigstruct, the other callback would be something like:
int
security_enclave_init(struct file *sigstruct_file);I'd still insist in using a pointer rather than a file, for reasons that we've discussed before. For those who can't recall, the major reason is that most implementation would embed SIGSTRUCT into the same file as the enclave (or at least I don't want to prevent anyone from doing so), which could also be part of another file, such as a shared object or even the main executable itself. It could be difficult to obtain a fd in those cases. memfd won't work because it can't retain the same attributes of the original file containing the SIGSTRUCT.
After all, what matters is the attributes associated with the backing file, which could be easily retrieve from
vm_fileof the covering VMA. So for the sake of flexibility, let's stay with what we've agreed before - a pointer to SIGSTRUCT.
I'm okay with this, except for one nastiness: there's a big difference between a file that is just a sigstruct and a file that contains essentially arbitrary data plus a sigstruct at an arbitrary offset. We could do something tricky like saying that SIGSTRUCT can be in a file that's just a SIGSTRUCT or it can be in a special SIGSTRUCT ELF note in a file that isn't just a SIGSTRUCT, but that could be annoyingly restrictive.
If it's going to be in an arbitrary file, then I think the signature needs to be more like:
int security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);
So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.
Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 7 lines
The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also want:
int
security_enclave_load_zeros(unsigned int maxperm);What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.
Quoted 6 lines
Quoted 4 lines
This is how EAUG could ask if RWX is okay. If an enclave is internally doing dynamic loading, the it will need a heap page with maxperm = RWX. (If it’s well designed, it will make it RW and then RX, either by changing SECINFO or by asking the host to
mprotect()it, but it still needs the overall RWX mask.).Any new page EAUG'ed will start in RW (as dictated by SGX ISA). EACCEPTCOPY will then change it to RX. RWX is never needed for all practical purposes. This in fact could be gated by
mprotect()and the attributes associated with/dev/sgx/enclave. In the case of SELinux,FILE__EXECMODis the right attribute andmprotect()will take care of all the rest. I don't see why the driver need a role here.
I find the SDM's discussion of EAUG, EACCEPT, and EACCEPTCOPY to be extremely confusing. My copy of the SDM has EACCEPT's SECINFO argument as "Read access permitted by Non Enclave". Is that an error?
And is EACCEPTCOPY just EACCEPT + memcpy or is there some other
fundamental difference? 38.5.7 doesn't even mention EACCEPTCOPY.
Anyway, all my confusion aside, I was talking about the page table, not the EPCM. I think the enclave should need permission to write its own content into a page that will ever become X, and the enclave's untrusted host library would do this by adding the page with MAXPERM=RWX and then mapping/mprotecting it as PROT_WRITE and then (later or simultaneously) PROT_EXEC.
Since SGX2 doesn't seem to have a way to add an initialized page to EPC after an enclave starts, I guess that it's impossible to have the enclave do something like dlopen() without MAXPERM=RWX. So be it. Maybe someone will find this annoying someday and SGX3 will add EAUG-but-don't-zero and EACCEPT-with-existing-contents.
Quoted 9 lines
Quoted 5 lines
Also, do real SGX1 enclave formats have BSS? If so, then either we need an ioctl or load zeros or user code is going to load from
/dev/zeroor just from the heap, but the LSM is going to play better with an ioctl, I suspect :)Yes, it does. But an enclave would either measure BSS, in which case the initial bytes have to be zero or MRENCLAVE will change; or zero BSS explicitly in its initialization code.
But from LSM's perspective it makes no difference than EADD'ing a page with non-zero content. And
security_enclave_load(NULL, RW)would take care of it in exactly in the same way.
Sure, I suppose the same hook with NULL parameters would be equivalent.
Quoted 11 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.
With SGX2/EDMM, RWX is *never* needed for all practical purposes.
In theory, in terms of security, no page shall be made executable while it is still being prepared. So W and X shall always be mutually exclusive, regardless it's in EPC or regular memory.
RWX is only needed in SGX1, as a workaround for certain usages, because EPCM permissions can never change at runtime.
As above, I think I disagree. MAXPERM is intended as an upper bound on the permissions that a page can ever have, at least until it's EREMOVEd and re-added. Since there's no EAUG-but-don't-zero, EAUG with MAXPERM.W=0 is basically useless because the page can never contain anything other than zeros, so a dynamically allocated page that is ever executed has to have MAXPERM=RWX or MAXPERM=WX. And that will need special permissions, which I think is consistent with your recent emails on how this could all map to SELinux permissions.
Quoted 18 lines
Quoted 18 lines
Quoted 17 lines
Quoted 8 lines
But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.
Am I making more sense now?
Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:
int security_enclave_load(struct file *file, struct
vm_area_struct*vma,unsigned long prot);And for ultimate flexibility we could pass both .sigstruct and the file pointer for
/dev/sgx/enclave, but that seems a bit ridiculous.
Quoted 7 lines
Quoted 1 line
I agree.
Loosely speaking, an enclave (including initial contents of all of its pages and their permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.
Theoretically speaking, if LSM can make a decision at EINIT by means of
security_enclave_load(), thensecurity_enclave_load()is never needed.In practice, I support keeping both because
security_enclave_load()can only approve an enumerable set whilesecurity_enclave_load()can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced bysecurity_enclave_load().
I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also, security_enclave_load() provides no protection against loading a mishmash of two different enclave files. I see security_enclave_init() as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" and security_enclave_load() as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."
Quoted 19 lines
Quoted 19 lines
Quoted 8 lines
Passing both would allow tying EXECMOD to
/dev/sgx/enclaveas Cedric wanted (without having to play games and pass/dev/sgx/enclavetosecurity_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.The benefit of putting it on .sigstruct is that it can be per-enclave.
As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirty look.
This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.
Quoted 7 lines
Quoted 5 lines
(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)
I'm lost here. Dynamically linked enclaves, if running on SGX2, would need
RW->RX, i.e.FILE__EXECMODon/dev/sgx/enclave. But they never need RWX, i.e.PROCESS__EXECMEM.
Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual mprotect() / mmap() permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply that PROT_WRITE | PROT_EXEC is allowed.
RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 102 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PMQuoted 5 lines · Cedric
Quoted 5 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Friday, May 24, 2019 4:42 PMQuoted 1 line
On May 24, 2019, at 3:41 PM, Sean Christopherson <[email protected]>
Quoted 92 lines · wrote:
Quoted 91 lines
Quoted 5 lines
Quoted 5 lines · Andy Lutomirski
Quoted 1 line · Sean Christopherson
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines · Andy Lutomirski
Quoted 7 lines · Cedric
If "initial permissions" for enclaves are less restrictive than shared objects, then it'd become a backdoor for circumventing LSM when enclave whitelisting is *not* in place. For example, an adversary may load a page, which would otherwise never be executable, as an executable
page in EPC.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
In the case a RWX page is needed, the calling process has to have a RWX page serving as the source for EADD so
PROCESS__EXECMEMwill have been checked. For SGX2, changing an EPC page to RWX is subject toFILE__EXECMEMon/dev/sgx/enclave, which I see as a security benefit because it only affects the enclave but not the whole process hostingit.
Quoted 60 lines
Quoted 10 lines
Quoted 7 lines
Quoted 5 lines
So the permission would be like
FILE__EXECMODon the source enclave page, because it would be mappedMAP_ANONYMOUS,PROT_WRITE?MAP_SHARED,PROT_WRITEisn't going to work because that means you can modify the file.Was this in response to Cedric's comment, or to my comment?
Yours. I think that requiring source pages to be actually mapped W is not such a great idea.
I wasn't requiring source pages to be mapped W. At least I didn't intend to require W. What I was trying to say is that SGX could trigger an EXECMEM check if userspace attempted to EADD or EAUG an enclave page with RWX permissions, e.g.:
if ((SECINFO.PERMS & RWX) == RWX) { ret = security_mmap_file(NULL, RWX, ???); if (ret) return ret;}
But that's a moot point if we add
security_enclave_load()or whatever.Quoted 36 lines
Quoted 30 lines
Quoted 22 lines
I'm starting to think that looking at the source VMA permission bits or source PTE permission bits is putting a bit too much policy into the driver as opposed to the LSM. How about delegating the whole thing to an LSM hook? The EADD operation would invoke a new hook, something like:
int
security_enclave_load_bytes(void *source_addr, struct vm_area_struct *source_vma, loff_t source_offset, unsigned int maxperm);Then you don't have to muck with mapping anything
PROT_EXEC. Instead you load from a mapping of a file and the LSM applies whatever policy it feels appropriate. If the first pass gets something wrong, the application or library authors can take it up with the SELinux folks without breaking the whole ABI :)(I'm proposing passing in the
source_vmabecause this hook would be called withmmap_semheld for read to avoid a TOCTOU race.)Quoted 7 lines
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around
mprotect()calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.Quoted 3 lines
This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?
In addition.
Quoted 7 lines
Quoted 5 lines
But I named the function badly and gave it a bad signature, which confused you. Let's try again:
int
security_enclave_load_from_memory(const struct vm_area_struct *source, unsigned int maxperm);I prefer
security_enclave_load(), "from_memory" seems redundant at best.Fine with me.
Quoted 7 lines
Quoted 1 line
If we think of EADD as a way of mmap()'ing an enclave file into memory, would this
security_enclave_load()be the same assecurity_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.
What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.
But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to security_mmap_file() by default?
Quoted 32 lines
Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Maybe some really fancy future LSM would also want
loff_tsource_offset, but it's probably not terribly useful. This same callback would be used for EAUG.EAUG always zeroes the EPC page before making it available to an enclave. So I don't
think there's anything needed to done here.
Duh. So
security_enclave_load_zeros()for EAUG. See below.Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Following up on your discussion with Cedric about sigstruct, the other callback would be something like:
int
security_enclave_init(struct file *sigstruct_file);I'd still insist in using a pointer rather than a file, for reasons that we've discussed
before. For those who can't recall, the major reason is that most implementation would embed SIGSTRUCT into the same file as the enclave (or at least I don't want to prevent anyone from doing so), which could also be part of another file, such as a shared object or even the main executable itself. It could be difficult to obtain a fd in those cases. memfd won't work because it can't retain the same attributes of the original file containing the SIGSTRUCT.
Quoted 2 lines
After all, what matters is the attributes associated with the backing file, which could
be easily retrieve from
vm_fileof the covering VMA. So for the sake of flexibility, let's stay with what we've agreed before - a pointer to SIGSTRUCT.
Quoted 6 lines
I'm okay with this, except for one nastiness: there's a big difference between a file that is just a sigstruct and a file that contains essentially arbitrary data plus a sigstruct at an arbitrary offset.
We could do something tricky like saying that SIGSTRUCT can be in a file that's just a SIGSTRUCT or it can be in a special SIGSTRUCT ELF note in a file that isn't just a SIGSTRUCT, but that could be annoyingly restrictive.
Agreed. Approving a file implies approving all SIGSTRUCTs within that file. But I guess it wouldn't cause practical problems.
Quoted 8 lines
If it's going to be in an arbitrary file, then I think the signature needs to be more like:
int
security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.
Wouldn't the pair of { sigstruct_vma, sigstruct_offset } be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer - vma->vm_start)?
Quoted 22 lines
Quoted 15 lines
Quoted 15 lines
Quoted 14 lines
Quoted 9 lines
The main issue I see is that we also want to control the enclave's ability to have RWX pages or to change a W page to X. We might also
want:int
security_enclave_load_zeros(unsigned int maxperm);What's the use case for this? @maxperm will always be at least RW in this case, otherwise the page is useless to the enclave, and if the enclave can write the page, the fact that it started as zeros is irrelevant.
Quoted 4 lines
Quoted 4 lines
This is how EAUG could ask if RWX is okay. If an enclave is
internally doing dynamic loading, the it will need a heap page with maxperm = RWX. (If it’s well designed, it will make it RW and then RX, either by changing SECINFO or by asking the host tomprotect()it, but it stillneeds the overall RWX mask.).
Quoted 2 lines
Any new page EAUG'ed will start in RW (as dictated by SGX ISA). EACCEPTCOPY will then
Quoted 8 lines
change it to RX. RWX is never needed for all practical purposes. This in fact could be gated by
mprotect()and the attributes associated with/dev/sgx/enclave. In the case of SELinux,FILE__EXECMODis the right attribute andmprotect()will take care of all the rest. I don't see why the driver need a role here.I find the SDM's discussion of EAUG, EACCEPT, and EACCEPTCOPY to be extremely confusing. My copy of the SDM has EACCEPT's SECINFO argument as "Read access permitted by Non Enclave". Is that an error?
I'm confused by those descriptions too. Guess I cannot comment if that's an error or not.
Anyway, per our internal documents, for EAUG, SECINFO has to be set to PT_REG|RW. For EACCEPT, SGX ISA compares supplied SECINFO with EPCM attributes and returns an error if they don't match. EACCEPTCOPY only works on pending pages (i.e. SECINFO.P must be set), and sets EPCM access permissions to whatever supplied in SECINFO.
Quoted 2 lines
And is EACCEPTCOPY just EACCEPT + memcpy or is there some other fundamental difference?
38.5.7 doesn't even mention EACCEPTCOPY.
2 differences: 1) EACCEPT only *compares* but EACCEPTCOPY *sets* EPCM permissions; and 2) EACCEPTCOPY does EACCEPT+memcpy atomically.
Quoted 6 lines
Anyway, all my confusion aside, I was talking about the page table, not the EPCM. I think the enclave should need permission to write its own content into a page that will ever become X, and the enclave's untrusted host library would do this by adding the page with MAXPERM=RWX and then mapping/mprotecting it as
PROT_WRITEand then (later or simultaneously)PROT_EXEC.
I was talking about the same thing. A code page in EPC will start in RW (both EPCM and PTE) and end in RX (both EPCM and PTE). EACCEPTCOPY takes care of EPCM, while mprotect() could take care of PTE as long as /dev/sgx/enclave has FILE__EXECMOD.
I understand your intention to enclave pages to segments with different MAXPERMs. My concern is though the host process may not always have a priori knowledge on which ranges to be used as code vs. data. After all, only the weakest link matters in security so I think what a host process cares would be whether the enclave loads code dynamically, or expands its data segments only, or neither. And for that reason, I think it more "user friendly" to keep just one MAXPERM - i.e. the most permissive one. Then we could associate that with /dev/sgx/enclave so as to relieve the driver from keeping track of too many things.
Quoted 4 lines
Since SGX2 doesn't seem to have a way to add an initialized page to EPC after an enclave starts, I guess that it's impossible to have the enclave do something like
dlopen()without MAXPERM=RWX. So be it.
That's true. The reason behind it is SGX doesn’t trust anything from outside. So non-predetermined contents must be measured (e.g. EADD+EEXTEND), but it's hard to measure (or attest to) dynamically added contents so we decided to allow predetermined contents (i.e. all zeros in the case of EAUG) only.
Quoted 2 lines
Maybe someone will find this annoying someday and SGX3 will add EAUG-but-don't-zero and EACCEPT-with-existing-contents.
From security perspective, accepting a page that is measured/hashed to XYZ is equivalent to overwriting that page with content hashed to XYZ. EACCEPTCOPY actually does the latter. The annoying part is due to the mismatch between SGX ISA and the s/w model adopted by LSM, but that has nothing to do with security.
Quoted 33 lines
Quoted 9 lines
Quoted 7 lines
Also, do real SGX1 enclave formats have BSS? If so, then either we need an ioctl or load zeros or user code is going to load from
/dev/zeroor just from the heap, but the LSM is going to play better with an ioctl, I suspect :)Yes, it does. But an enclave would either measure BSS, in which case the initial bytes
have to be zero or MRENCLAVE will change; or zero BSS explicitly in its initialization code.
Quoted 2 lines
But from LSM's perspective it makes no difference than EADD'ing a page with non-zero
content. And
security_enclave_load(NULL, RW)would take care of it in exactly in the same way.Sure, I suppose the same hook with NULL parameters would be equivalent.
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
An enclave that's going to modify its own code will need memory with maxperm = RWX or WX.
With SGX2/EDMM, RWX is *never* needed for all practical purposes.
In theory, in terms of security, no page shall be made executable while it is still
being prepared. So W and X shall always be mutually exclusive, regardless it's in EPC or regular memory.
Quoted 2 lines
RWX is only needed in SGX1, as a workaround for certain usages, because EPCM permissions
can never change at runtime.
Quoted 7 lines
As above, I think I disagree. MAXPERM is intended as an upper bound on the permissions that a page can ever have, at least until it's EREMOVEd and re-added. Since there's no EAUG-but-don't-zero, EAUG with MAXPERM.W=0 is basically useless because the page can never contain anything other than zeros, so a dynamically allocated page that is ever executed has to have MAXPERM=RWX or MAXPERM=WX. And that will need special permissions, which I think is consistent with your recent emails on how this could all map to SELinux permissions.
I'm totally with you. What I was trying to say was that only W or X would be needed at any given time. That said, MAXPERM=RWX but PROCESS__EXECMEM will not be needed, while FILE__EXECMOD will be needed only on /dev/sgx/enclave. So the inherent risk is contained.
Quoted 35 lines
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Quoted 9 lines
But this is a bit awkward if the LSM's decision depends on the sigstruct. We could get fancy and require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct.
Am I making more sense now?
Yep. Requiring .sigstruct at ECREATE would be trivial. If we wanted flexibility we could do:
int security_enclave_load(struct file *file, struct
vm_area_struct*vma,unsigned long prot);Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
And for ultimate flexibility we could pass both .sigstruct and the file pointer for
/dev/sgx/enclave, but that seems a bit ridiculous.I agree.
Loosely speaking, an enclave (including initial contents of all of its pages and their
permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.
Quoted 2 lines
Theoretically speaking, if LSM can make a decision at EINIT by means of
security_enclave_load(), thensecurity_enclave_load()is never needed.Quoted 2 lines
In practice, I support keeping both because
security_enclave_load()can only approve anenumerable set while
security_enclave_load()can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced bysecurity_enclave_load().
Quoted 7 lines
I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also,
security_enclave_load()provides no protection against loading a mishmash of two different enclave files. I seesecurity_enclave_init()as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT"
andsecurity_enclave_load()as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."
Sorry for the confusion. I was saying the same thing except that the decision of security_enclave_load() doesn't have to depend on SIGSTRUCT. Given your prototype of security_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".
Quoted 29 lines
Quoted 10 lines
Quoted 10 lines
Quoted 9 lines
Passing both would allow tying EXECMOD to
/dev/sgx/enclaveas Cedric wanted (without having to play games and pass/dev/sgx/enclavetosecurity_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.Quoted 6 lines
Quoted 6 lines
The benefit of putting it on .sigstruct is that it can be per-enclave.
As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirtylook.
Quoted 12 lines
Quoted 11 lines
This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.
(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)
Quoted 7 lines
Quoted 1 line
I'm lost here. Dynamically linked enclaves, if running on SGX2, would need
RW->RX, i.e.
FILE__EXECMODon/dev/sgx/enclave. But they never need RWX, i.e.PROCESS__EXECMEM.Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual
mprotect()/mmap()permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply thatPROT_WRITE|PROT_EXECis allowed.
If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?
To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of /dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on /dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of -EPERM returned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of IOC_EINIT.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 15 lines · Cedric
Quoted 7 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PMQuoted 3 lines · Cedric
If we think of EADD as a way of mmap()'ing an enclave file into memory, would this
Quoted 7 lines
security_enclave_load()be the same assecurity_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.
Quoted 8 lines
What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.
But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to
security_mmap_file()by default?
Mapping the enclave will still go through security_mmap_file(), the extra security_enclave_load() hook allows the mmap() to use PROT_NONE.
Quoted 11 lines
Quoted 7 lines
If it's going to be in an arbitrary file, then I think the signature needs to be more like:
int
security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.
Wouldn't the pair of {
sigstruct_vma,sigstruct_offset} be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer -vma->vm_start)?
VMA has vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?
int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);
Quoted 22 lines
Quoted 14 lines
Quoted 1 line
Loosely speaking, an enclave (including initial contents of all of its pages and their
permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.
Quoted 2 lines
Theoretically speaking, if LSM can make a decision at EINIT by means of
security_enclave_load(), thensecurity_enclave_load()is never needed.Quoted 2 lines
In practice, I support keeping both because
security_enclave_load()can only approve anenumerable set while
security_enclave_load()can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced bysecurity_enclave_load().Quoted 7 lines
I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also,
security_enclave_load()provides no protection against loading a mishmash of two different enclave files. I seesecurity_enclave_init()as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" andsecurity_enclave_load()as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."
Quoted 6 lines
Sorry for the confusion. I was saying the same thing except that the decision of
security_enclave_load()doesn't have to depend on SIGSTRUCT. Given your prototype ofsecurity_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".
Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling /dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using /dev/sgx/enclave means users can't do per-enclave permissions, period.
What is your objection to working on the sigstruct?
Quoted 25 lines
Quoted 25 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Passing both would allow tying EXECMOD to
/dev/sgx/enclaveas Cedric wanted (without having to play games and pass/dev/sgx/enclavetosecurity_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.Quoted 6 lines
Quoted 6 lines
The benefit of putting it on .sigstruct is that it can be per-enclave.
As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirtylook.
Quoted 12 lines
Quoted 11 lines
This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.
(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
I'm lost here. Dynamically linked enclaves, if running on SGX2, would need
RW->RX, i.e.
FILE__EXECMODon/dev/sgx/enclave. But they never need RWX, i.e.PROCESS__EXECMEM.Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual
mprotect()/mmap()permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply thatPROT_WRITE|PROT_EXECis allowed.
Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that sgx_enclave_load() with a NULL vma implies a transition from RW. For example, SELinux would interpret sgx_enclave_load(NULL, RX) as requiring FILE__EXECMOD.
As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e. sgx_enclave_load(NULL, RX) already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs from MAY_RW to MAY_RX.
For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit ioctl() to add the to-be-EAUG'd range. That just leaves us wanting an ioctl() to set the post-EACCEPT{COPY} permissions.
E.g.:
ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */
mprotect(addr, size, RW);
...
EACCEPTCOPY -> EAUG /* page fault handler */
ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */
mprotect(addr, size, RX);
... And making ACTIVATE_REGION a single-shot per page eliminates the need for the MAXPERMS concept (see below).
Quoted 15 lines
If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?
To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of
/dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on/dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of-EPERMreturned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part ofIOC_EINIT.
Aha!
Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:
- Initial permissions for *EADD* pages are explicitly handled via sgx_enclave_load() with the exact SECINFO permissions.
- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.
- Runtime permissions break down as follows:
R - N/A, subset of RW (EAUG)
W - N/A, subset of RW (EAUG) and x86 paging can't do W
X - N/A, subset of RX (x86 paging can't do XO)
RW - Handled by EAUG LSM hook (uses RW unconditionally)
WX - N/A, subset of RWX (x86 paging can't do WX)
RX - Handled by ACTIVATE_REGION RWX - Handled by ACTIVATE_REGION
In other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):
- <prefix>_ACTIVATE_REGION(vma, perms) -> sgx_enclave_load(NULL, perms)
- <prefix>_ADD_REGION(vma) -> sgx_enclave_load(vma, SECINFO.perms)
- <prefix>_ADD_REGION(NULL) -> sgx_enclave_load(NULL, RW)
then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of sgx_enclave_load(..., RW) would need to be different than normal shared memory, e.g. FILE__WRITE should *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without* FILE__EXECMOD, it'd have the necessary information to do so.
The userspace changes are fairly minimal:
- For SGX1, use PROT_NONE for the initial mmap() and refactor ADD_PAGE to ADD_REGION.
- For SGX2, do an explicit ADD_REGION on the ranges to be EAUG'd, and an ACTIVATE_REGION to make a region RX or R (no extra ioctl() required to keep RW permissions).
Because ACTIVATE_REGION can only be done once per page, to do *abitrary* mprotect() transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.
- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.
- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 117 lines · Sean Christopherson
Quoted 15 lines · Cedric
Quoted 7 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PMQuoted 3 lines · Cedric
If we think of EADD as a way of mmap()'ing an enclave file into memory, would this
Quoted 7 lines
security_enclave_load()be the same assecurity_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.
Quoted 8 lines
What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.
But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to
security_mmap_file()by default?Mapping the enclave will still go through
security_mmap_file(), the extrasecurity_enclave_load()hook allows themmap()to usePROT_NONE.Quoted 11 lines
Quoted 7 lines
If it's going to be in an arbitrary file, then I think the signature needs to be more like:
int
security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.
Wouldn't the pair of {
sigstruct_vma,sigstruct_offset} be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer -vma->vm_start)?VMA has
vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);Quoted 22 lines
Quoted 14 lines
Quoted 1 line
Loosely speaking, an enclave (including initial contents of all of its pages and their
permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.
Quoted 2 lines
Theoretically speaking, if LSM can make a decision at EINIT by means of
security_enclave_load(), thensecurity_enclave_load()is never needed.Quoted 2 lines
In practice, I support keeping both because
security_enclave_load()can only approve anenumerable set while
security_enclave_load()can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced bysecurity_enclave_load().Quoted 7 lines
I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also,
security_enclave_load()provides no protection against loading a mishmash of two different enclave files. I seesecurity_enclave_init()as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" andsecurity_enclave_load()as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."Quoted 6 lines
Sorry for the confusion. I was saying the same thing except that the decision of
security_enclave_load()doesn't have to depend on SIGSTRUCT. Given your prototype ofsecurity_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling
/dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using/dev/sgx/enclavemeans users can't do per-enclave permissions, period.What is your objection to working on the sigstruct?
Quoted 25 lines
Quoted 25 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Passing both would allow tying EXECMOD to
/dev/sgx/enclaveas Cedric wanted (without having to play games and pass/dev/sgx/enclavetosecurity_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.Quoted 6 lines
Quoted 6 lines
The benefit of putting it on .sigstruct is that it can be per-enclave.
As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirtylook.
Quoted 12 lines
Quoted 11 lines
This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.
(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
I'm lost here. Dynamically linked enclaves, if running on SGX2, would need
RW->RX, i.e.
FILE__EXECMODon/dev/sgx/enclave. But they never need RWX, i.e.PROCESS__EXECMEM.Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual
mprotect()/mmap()permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply thatPROT_WRITE|PROT_EXECis allowed.
Quoted 4 lines
Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that
sgx_enclave_load()with a NULL vma implies a transition from RW. For example, SELinux would interpretsgx_enclave_load(NULL, RX)as requiringFILE__EXECMOD.
You lost me here. What operation triggers this callback? And wouldn't sgx_enclave_load(NULL, RX) sometimes be a transition from RO or just some fresh executable zero bytes?
Quoted 12 lines
As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e.
sgx_enclave_load(NULL, RX)already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs fromMAY_RWtoMAY_RX.For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit
ioctl()to add the to-be-EAUG'd range. That just leaves us wanting anioctl()to set the post-EACCEPT{COPY} permissions.E.g.:
Quoted 10 lines
ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */
mprotect(addr, size, RW);
...
EACCEPTCOPY -> EAUG /* page fault handler */
ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */
mprotect(addr, size, RX);In the maxperm model, this mprotect() will fail unless MAXPERM contains RX, which could only happen if MAXPERM=RWX. So, regardless of how it's actually mapped to SELinux policy, MAXPERM=RWX is functionally like EXECMOD and actual RWX PTEs are functionally like EXECMEM.
Quoted 30 lines
...And making
ACTIVATE_REGIONa single-shot per page eliminates the need for the MAXPERMS concept (see below).Quoted 15 lines
If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?
To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of
/dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on/dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of-EPERMreturned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part ofIOC_EINIT.Aha!
Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:
- Initial permissions for *EADD* pages are explicitly handled via
sgx_enclave_load()with the exact SECINFO permissions.
Quoted 7 lines
- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.
- Runtime permissions break down as follows:
R - N/A, subset of RW (EAUG) W - N/A, subset of RW (EAUG) and x86 paging can't do W X - N/A, subset of RX (x86 paging can't do XO)
Sure it can! You just have a hypervisor that maps a PA bit to EPT no-read. Then you can use that PA bit to suppress read. Also, Linux already abuses PKRU to simulate XO, although that won't work for enclaves.
Quoted 26 lines
RW - Handled by EAUG LSM hook (uses RW unconditionally) WX - N/A, subset of RWX (x86 paging can't do WX) RX - Handled by ACTIVATE_REGIONRWX - Handled by
ACTIVATE_REGIONIn other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):
- <prefix>_ACTIVATE_REGION(vma, perms) ->
sgx_enclave_load(NULL, perms)- <prefix>_ADD_REGION(vma) ->
sgx_enclave_load(vma, SECINFO.perms)- <prefix>_ADD_REGION(NULL) ->
sgx_enclave_load(NULL, RW)then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of
sgx_enclave_load(..., RW)would need to be different than normal shared memory, e.g.FILE__WRITEshould *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without*FILE__EXECMOD, it'd have the necessary information to do so.The userspace changes are fairly minimal:
- For SGX1, use
PROT_NONEfor the initialmmap()and refactorADD_PAGEtoADD_REGION.
Quoted 9 lines
- For SGX2, do an explicit
ADD_REGIONon the ranges to be EAUG'd, and anACTIVATE_REGIONto make a region RX or R (no extraioctl()required to keep RW permissions).Because
ACTIVATE_REGIONcan only be done once per page, to do *abitrary*mprotect()transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.
I may be misunderstanding or just be biased to my own proposal, but this seems potentially more complicated and less flexible than the MAXPERM model. One of the main things that made me come up with MAXPERM is that I wanted to avoid any complicated PTE/VMA modification or runtime changes. So, with MAXPERM, we still need to track the MAXPERM bits per page, but we don't ever need to *change* them or to worry about what is or is not mapped anywhere at any given time. With ACTIVATE_REGION, don't we need to make sure that we don't have a second VMA pointing at the same pages? Or am I just confused?
Quoted 2 lines
- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.
I may still be missing something, but, for SGX1, it's possible at least in principle for the enclave to request, via ocall or similar, that the untrusted runtime do mprotect(). It's not even such a bad idea. Honestly, enclaves *shouldn't* have anything actually writable and executable at once because the enclaves don't want to be easily exploited.
Quoted 5 lines
- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 11 lines · Andy Lutomirski
Quoted 5 lines · Sean Christopherson
Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that
sgx_enclave_load()with a NULL vma implies a transition from RW. For example, SELinux would interpretsgx_enclave_load(NULL, RX)as requiringFILE__EXECMOD.You lost me here. What operation triggers this callback? And wouldn't
sgx_enclave_load(NULL, RX)sometimes be a transition from RO or just some fresh executable zero bytes?
An explicit ioctl() after EACCEPTCOPY to update the allowed permissions. For all intents and purposes, the EAUG'd page must start RW. Maybe a better way to phrase it is that at some point the page must be writable to have any value whatsover. EACCEPTCOPY explicitly requires the page to be at least RW. EACCEPT technically doesn't require RW, but a RO or RX zero page is useless. Userspace could still EACCEPT with RO or RX, but SGX would assume a minimum of RW for the purposes of the LSM check.
Quoted 20 lines
Quoted 20 lines
As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e.
sgx_enclave_load(NULL, RX)already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs fromMAY_RWtoMAY_RX.For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit
ioctl()to add the to-be-EAUG'd range. That just leaves us wanting anioctl()to set the post-EACCEPT{COPY} permissions.E.g.:
ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */ mprotect(addr, size, RW); ... EACCEPTCOPY -> EAUG /* page fault handler */ ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */
Quoted 7 lines
Quoted 1 line
mprotect(addr, size, RX);In the maxperm model, this
mprotect()will fail unless MAXPERM contains RX, which could only happen if MAXPERM=RWX. So, regardless of how it's actually mapped to SELinux policy, MAXPERM=RWX is functionally like EXECMOD and actual RWX PTEs are functionally like EXECMEM.
Yep, same idea, except in the proposed flow ACTIVATE_REGION.
Quoted 37 lines
Quoted 29 lines
...And making
ACTIVATE_REGIONa single-shot per page eliminates the need for the MAXPERMS concept (see below).Quoted 15 lines
If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?
To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of
/dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on/dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of-EPERMreturned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part ofIOC_EINIT.Aha!
Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:
- Initial permissions for *EADD* pages are explicitly handled via
sgx_enclave_load()with the exact SECINFO permissions.Quoted 7 lines
- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.
- Runtime permissions break down as follows:
R - N/A, subset of RW (EAUG) W - N/A, subset of RW (EAUG) and x86 paging can't do W X - N/A, subset of RX (x86 paging can't do XO)
Quoted 4 lines
Sure it can! You just have a hypervisor that maps a PA bit to EPT no-read. Then you can use that PA bit to suppress read. Also, Linux already abuses PKRU to simulate XO, although that won't work for enclaves.
Heh, I intentionally said "x86 paging" to rule out EPT :-) I'm pretty sure it's a moot point though, I have a hard time believing an LSM will allow RW->X and not RW->RX.
Quoted 36 lines
Quoted 26 lines
RW - Handled by EAUG LSM hook (uses RW unconditionally) WX - N/A, subset of RWX (x86 paging can't do WX) RX - Handled by ACTIVATE_REGIONRWX - Handled by
ACTIVATE_REGIONIn other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):
- <prefix>_ACTIVATE_REGION(vma, perms) ->
sgx_enclave_load(NULL, perms)- <prefix>_ADD_REGION(vma) ->
sgx_enclave_load(vma, SECINFO.perms)- <prefix>_ADD_REGION(NULL) ->
sgx_enclave_load(NULL, RW)then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of
sgx_enclave_load(..., RW)would need to be different than normal shared memory, e.g.FILE__WRITEshould *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without*FILE__EXECMOD, it'd have the necessary information to do so.The userspace changes are fairly minimal:
- For SGX1, use
PROT_NONEfor the initialmmap()and refactorADD_PAGEtoADD_REGION.Quoted 9 lines
- For SGX2, do an explicit
ADD_REGIONon the ranges to be EAUG'd, and anACTIVATE_REGIONto make a region RX or R (no extraioctl()required to keep RW permissions).Because
ACTIVATE_REGIONcan only be done once per page, to do *abitrary*mprotect()transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.
Quoted 9 lines
I may be misunderstanding or just be biased to my own proposal, but this seems potentially more complicated and less flexible than the MAXPERM model. One of the main things that made me come up with MAXPERM is that I wanted to avoid any complicated PTE/VMA modification or runtime changes. So, with MAXPERM, we still need to track the MAXPERM bits per page, but we don't ever need to *change* them or to worry about what is or is not mapped anywhere at any given time. With
ACTIVATE_REGION, don't we need to make sure that we don't have a second VMA pointing at the same pages? Or am I just confused?
In theory, it's still your MAXPERM model, but with the unnecessary states removed and the others enforced/handled by the natural SGX transitions instead of explictly in ioctls. Underneath the hood the SGX driver would still need to track the MAXPERM.
With SGX1, SECINFO == MAXPERM. With SGX2, ACTIVATE_REGION == MAXPERM, with the implication that the previous state is always RW.
Quoted 9 lines
Quoted 2 lines
- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.
I may still be missing something, but, for SGX1, it's possible at least in principle for the enclave to request, via ocall or similar, that the untrusted runtime do
mprotect(). It's not even such a bad idea. Honestly, enclaves *shouldn't* have anything actually writable and executable at once because the enclaves don't want to be easily exploited.
Yes, but the *EPCM* permissions are immutable. So if an enclave wants to do RW->RX it has to intialize its pages to RWX. And because the untrusted runtime is, ahem, untrusted, the enclave cannot rely on userspace to never map its pages RWX. In other words, from a enclave security perspective, an SGX1 enclave+runtime that uses RW->RX is no different than an enclave that uses RWX. Using your earlier terminology, an SGX1 enclave *should* get a dirty looks if maps a page RWX in the EPCM, even if it only intends RW->RX behavior.
Quoted 4 lines
Quoted 4 lines
- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.
RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 16 lines
From: Christopherson, Sean J
Sent: Tuesday, May 28, 2019 2:41 PMQuoted 11 lines · Andy Lutomirski
Quoted 5 lines · Sean Christopherson
Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that
sgx_enclave_load()with a NULL vma implies a transition from RW. For example, SELinux would interpretsgx_enclave_load(NULL, RX)as requiringFILE__EXECMOD.You lost me here. What operation triggers this callback? And wouldn't
sgx_enclave_load(NULL, RX)sometimes be a transition from RO or just some fresh executable zero bytes?
Quoted 6 lines
An explicit
ioctl()after EACCEPTCOPY to update the allowed permissions. For all intents and purposes, the EAUG'd page must start RW. Maybe a better way to phrase it is that at some point the page must be writable to have any value whatsover. EACCEPTCOPY explicitly requires the page to be at least RW. EACCEPT technically doesn't require RW, but a RO or RX zero page is useless. Userspace could still EACCEPT with RO or RX, but SGX would assume a minimum of RW for the purposes of the LSM check.
Why is an explicit ioctl() necessary after EACCEPTCOPY? Or why is mprotect() not sufficient?
I tend to agree on Andy's MAXPERM model where MAXPERM never changes once established.
Quoted 71 lines
Quoted 21 lines
Quoted 21 lines
As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e.
sgx_enclave_load(NULL, RX)already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs fromMAY_RWtoMAY_RX.For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit
ioctl()to add the to-be-EAUG'd range. That just leaves us wanting anioctl()to set the post-EACCEPT{COPY} permissions.E.g.:
ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */ mprotect(addr, size, RW); ... EACCEPTCOPY -> EAUG /* page fault handler */ ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */Quoted 7 lines
Quoted 1 line
mprotect(addr, size, RX);In the maxperm model, this
mprotect()will fail unless MAXPERM contains RX, which could only happen if MAXPERM=RWX. So, regardless of how it's actually mapped to SELinux policy, MAXPERM=RWX is functionally like EXECMOD and actual RWX PTEs are functionally like EXECMEM.Yep, same idea, except in the proposed flow
ACTIVATE_REGION.Quoted 8 lines
Quoted 8 lines
...And making
ACTIVATE_REGIONa single-shot per page eliminates the need for the MAXPERMS concept (see below).Quoted 3 lines
If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?
Quoted 10 lines
Quoted 10 lines
Quoted 10 lines
To be a bit more clear, system admin sets MAXPERM upper bound in
the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of/dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on/dev/sgx/enclave, then everything
will just work. Otherwise, it fails in the form of-EPERMreturned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care
of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still
required for initial code pages even though system admins could disable dynamicallyloaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial"
Quoted 19 lines
Quoted 18 lines
Quoted 2 lines
mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part of
IOC_EINIT.Aha!
Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:
- Initial permissions for *EADD* pages are explicitly handled via
sgx_enclave_load()with the exact SECINFO permissions.- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.
- Runtime permissions break down as follows:
R - N/A, subset of RW (EAUG) W - N/A, subset of RW (EAUG) and x86 paging can't do W X - N/A, subset of RX (x86 paging can't do XO)
Quoted 18 lines
Quoted 4 lines
Sure it can! You just have a hypervisor that maps a PA bit to EPT no-read. Then you can use that PA bit to suppress read. Also, Linux already abuses PKRU to simulate XO, although that won't work for enclaves.
Heh, I intentionally said "x86 paging" to rule out EPT :-) I'm pretty sure it's a moot point though, I have a hard time believing an LSM will allow
RW->Xand notRW->RX.Quoted 10 lines
Quoted 10 lines
RW - Handled by EAUG LSM hook (uses RW unconditionally) WX - N/A, subset of RWX (x86 paging can't do WX) RX - Handled by ACTIVATE_REGIONRWX - Handled by
ACTIVATE_REGIONIn other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):
- <prefix>_ACTIVATE_REGION(vma, perms) -> sgx_enclave_load(NULL,
perms)
I'm not sure on what security_enclave_load()'s decision would be based.
Quoted 38 lines
Quoted 28 lines
Quoted 21 lines
- <prefix>_ADD_REGION(vma) ->
sgx_enclave_load(vma, SECINFO.perms)- <prefix>_ADD_REGION(NULL) ->
sgx_enclave_load(NULL, RW)then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of
sgx_enclave_load(..., RW)would need to be different than normal shared memory, e.g.FILE__WRITEshould *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without*FILE__EXECMOD, it'd have the necessary information to do so.The userspace changes are fairly minimal:
- For SGX1, use
PROT_NONEfor the initialmmap()and refactorADD_PAGEtoADD_REGION.- For SGX2, do an explicit
ADD_REGIONon the ranges to be EAUG'd, and anACTIVATE_REGIONto make a region RX or R (no extraioctl()required to keep RW permissions).Quoted 6 lines
Because
ACTIVATE_REGIONcan only be done once per page, to do *abitrary*mprotect()transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.Quoted 9 lines
I may be misunderstanding or just be biased to my own proposal, but this seems potentially more complicated and less flexible than the MAXPERM model. One of the main things that made me come up with MAXPERM is that I wanted to avoid any complicated PTE/VMA modification or runtime changes. So, with MAXPERM, we still need to track the MAXPERM bits per page, but we don't ever need to *change* them or to worry about what is or is not mapped anywhere at any given time. With
ACTIVATE_REGION, don't we need to make sure that we don't have a second VMA pointing at the same pages? Or am I just confused?
Quoted 3 lines
In theory, it's still your MAXPERM model, but with the unnecessary states removed and the others enforced/handled by the natural SGX transitions instead of explictly in ioctls. Underneath the hood the SGX driver would still need to track the MAXPERM.
What are the "unnecessary states" removed?
I'm not sure understand the proposal fully. The whole thing looks to me like the driver is undertaking things that should/would otherwise be done by mmap()/mprotect() syscalls. It also imposes unnecessary restrictions on user mode code, such as mmap(PROT_NONE), ACTIVATE_REGION can be called only once, etc. What'd happen if ACTIVATE_REGION is called with a range spanning multiple/partial VMAs? What'd happen if an enclave was unmapped than mapped again? I'd say the proposal is unintuitive at least.
In theory, if the driver can keep track of MAXPERM for all pages within an enclave, then it could fail mmap() if the requested prot conflicts with any page's MAXPERM within that range. Otherwise, MAXPERM could be copied into VM_MAY* flags then mprotect() will just follow through. Wouldn't that be a much simpler and more intuitive approach?
Quoted 25 lines
With SGX1, SECINFO == MAXPERM. With SGX2,
ACTIVATE_REGION== MAXPERM, with the implication that the previous state is always RW.Quoted 9 lines
Quoted 2 lines
- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.
I may still be missing something, but, for SGX1, it's possible at least in principle for the enclave to request, via ocall or similar, that the untrusted runtime do
mprotect(). It's not even such a bad idea. Honestly, enclaves *shouldn't* have anything actually writable and executable at once because the enclaves don't want to be easily exploited.Yes, but the *EPCM* permissions are immutable. So if an enclave wants to do
RW->RXit has to intialize its pages to RWX. And because the untrusted runtime is, ahem, untrusted, the enclave cannot rely on userspace to never map its pages RWX. In other words, from a enclave security perspective, an SGX1 enclave+runtime that usesRW->RXis no different than an enclave that uses RWX. Using your earlier terminology, an SGX1 enclave *should* get a dirty looks if maps a page RWX in the EPCM, even if it only intendsRW->RXbehavior.Quoted 4 lines
Quoted 4 lines
- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 16 lines · Cedric
Quoted 16 lines
From: Christopherson, Sean J
Sent: Tuesday, May 28, 2019 2:41 PMQuoted 11 lines · Andy Lutomirski
Quoted 5 lines · Sean Christopherson
Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that
sgx_enclave_load()with a NULL vma implies a transition from RW. For example, SELinux would interpretsgx_enclave_load(NULL, RX)as requiringFILE__EXECMOD.You lost me here. What operation triggers this callback? And wouldn't
sgx_enclave_load(NULL, RX)sometimes be a transition from RO or just some fresh executable zero bytes?
Quoted 8 lines
Quoted 6 lines
An explicit
ioctl()after EACCEPTCOPY to update the allowed permissions. For all intents and purposes, the EAUG'd page must start RW. Maybe a better way to phrase it is that at some point the page must be writable to have any value whatsover. EACCEPTCOPY explicitly requires the page to be at least RW. EACCEPT technically doesn't require RW, but a RO or RX zero page is useless. Userspace could still EACCEPT with RO or RX, but SGX would assume a minimum of RW for the purposes of the LSM check.Why is an explicit
ioctl()necessary after EACCEPTCOPY? Or why ismprotect()not sufficient?
Ignore this, I was trying to avoid having to add a vm_ops mprotect(), which Andy pointed out was silly.
Quoted 5 lines
Quoted 3 lines
In theory, it's still your MAXPERM model, but with the unnecessary states removed and the others enforced/handled by the natural SGX transitions instead of explictly in ioctls. Underneath the hood the SGX driver would still need to track the MAXPERM.
What are the "unnecessary states" removed?
Andy proposed taking full RWX in MAXPERMs, but really we only need "can writes ever happen to this page", as that allows the SGX driver to avoid having to track if a page has been mapped PROT_WRITE by any VMA in any process.
Quoted 8 lines
I'm not sure understand the proposal fully. The whole thing looks to me like the driver is undertaking things that should/would otherwise be done by mmap()/mprotect() syscalls. It also imposes unnecessary restrictions on user mode code, such as
mmap(PROT_NONE),ACTIVATE_REGIONcan be called only once, etc. What'd happen ifACTIVATE_REGIONis called with a range spanning multiple/partial VMAs? What'd happen if an enclave was unmapped than mapped again? I'd say the proposal is unintuitive at least.
Quoted 5 lines
In theory, if the driver can keep track of MAXPERM for all pages within an enclave, then it could fail
mmap()if the requested prot conflicts with any page's MAXPERM within that range. Otherwise, MAXPERM could be copied into VM_MAY* flags thenmprotect()will just follow through. Wouldn't that be a much simpler and more intuitive approach?
Ignore all this, again I was trying to avoid hooking mprotect().
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonQuoted 191 lines · Sean Christopherson
Quoted 15 lines · Cedric
Quoted 7 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PMQuoted 3 lines · Cedric
If we think of EADD as a way of mmap()'ing an enclave file into memory, would this
Quoted 7 lines
security_enclave_load()be the same assecurity_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.
Quoted 8 lines
What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.
But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to
security_mmap_file()by default?Mapping the enclave will still go through
security_mmap_file(), the extrasecurity_enclave_load()hook allows themmap()to usePROT_NONE.Quoted 11 lines
Quoted 7 lines
If it's going to be in an arbitrary file, then I think the signature needs to be more like:
int
security_enclave_init(struct vm_area_struct *sigstruct_vma, loff_t sigstruct_offset, const sgx_sigstruct *sigstruct);So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.
Wouldn't the pair of {
sigstruct_vma,sigstruct_offset} be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer -vma->vm_start)?VMA has
vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);Quoted 22 lines
Quoted 14 lines
Quoted 1 line
Loosely speaking, an enclave (including initial contents of all of its pages and their
permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision resistant property of SHA-2). So only one is needed for a decision, and either one would lead to the same decision. So I don't see anything making any sense here.
Quoted 2 lines
Theoretically speaking, if LSM can make a decision at EINIT by means of
security_enclave_load(), thensecurity_enclave_load()is never needed.Quoted 2 lines
In practice, I support keeping both because
security_enclave_load()can only approve anenumerable set while
security_enclave_load()can approve a non-enumerable set of enclaves. Moreover, in order to determine the validity of a MRENCLAVE (as in development of a policy or in creation of a white/black list), system admins will need the audit log produced bysecurity_enclave_load().Quoted 7 lines
I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also,
security_enclave_load()provides no protection against loading a mishmash of two different enclave files. I seesecurity_enclave_init()as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" andsecurity_enclave_load()as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."Quoted 6 lines
Sorry for the confusion. I was saying the same thing except that the decision of
security_enclave_load()doesn't have to depend on SIGSTRUCT. Given your prototype ofsecurity_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling
/dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using/dev/sgx/enclavemeans users can't do per-enclave permissions, period.What is your objection to working on the sigstruct?
Quoted 25 lines
Quoted 25 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Passing both would allow tying EXECMOD to
/dev/sgx/enclaveas Cedric wanted (without having to play games and pass/dev/sgx/enclavetosecurity_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.Quoted 6 lines
Quoted 6 lines
The benefit of putting it on .sigstruct is that it can be per-enclave.
As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirtylook.
Quoted 12 lines
Quoted 11 lines
This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.
(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)
Quoted 7 lines
Quoted 7 lines
Quoted 1 line
I'm lost here. Dynamically linked enclaves, if running on SGX2, would need
RW->RX, i.e.
FILE__EXECMODon/dev/sgx/enclave. But they never need RWX, i.e.PROCESS__EXECMEM.Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual
mprotect()/mmap()permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply thatPROT_WRITE|PROT_EXECis allowed.Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that
sgx_enclave_load()with a NULL vma implies a transition from RW. For example, SELinux would interpretsgx_enclave_load(NULL, RX)as requiringFILE__EXECMOD.As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e.
sgx_enclave_load(NULL, RX)already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs fromMAY_RWtoMAY_RX.For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit
ioctl()to add the to-be-EAUG'd range. That just leaves us wanting anioctl()to set the post-EACCEPT{COPY} permissions.E.g.:
ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */ mprotect(addr, size, RW); ... EACCEPTCOPY -> EAUG /* page fault handler */ ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */ mprotect(addr, size, RX); ...And making
ACTIVATE_REGIONa single-shot per page eliminates the need for the MAXPERMS concept (see below).Quoted 15 lines
If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?
To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of
/dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on/dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of-EPERMreturned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part ofIOC_EINIT.Aha!
Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:
- Initial permissions for *EADD* pages are explicitly handled via
sgx_enclave_load()with the exact SECINFO permissions.- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.
- Runtime permissions break down as follows:
R - N/A, subset of RW (EAUG) W - N/A, subset of RW (EAUG) and x86 paging can't do W X - N/A, subset of RX (x86 paging can't do XO) RW - Handled by EAUG LSM hook (uses RW unconditionally) WX - N/A, subset of RWX (x86 paging can't do WX) RX - Handled by ACTIVATE_REGIONRWX - Handled by
ACTIVATE_REGIONIn other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):
- <prefix>_ACTIVATE_REGION(vma, perms) ->
sgx_enclave_load(NULL, perms)- <prefix>_ADD_REGION(vma) ->
sgx_enclave_load(vma, SECINFO.perms)
Quoted 8 lines
- <prefix>_ADD_REGION(NULL) ->
sgx_enclave_load(NULL, RW)then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of
sgx_enclave_load(..., RW)would need to be different than normal shared memory, e.g.FILE__WRITEshould *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without*FILE__EXECMOD, it'd have the necessary information to do so.
Assuming that sgx_enclave_load() is a LSM hook (probably named security_enclave_load() instead), then:
a) Does the sigstruct file get passed to this hook in every case, even when vma is NULL? I think the answer is yes, just want to confirm.
b) Should we use a different hook for ACTIVATE_REGION than for ADD_REGION or is the distinction between them irrelevant/unnecessary from an access control point of view? At present LSM/SELinux won't be able to distinguish ACTIVATE_REGION(vma, RW) from ADD_REGION(NULL) above since they will both invoke the same hook with the same arguments IIUC. Does it matter? It's ok if the answer is no, just want to confirm.
c) Is there still also a separate security_enclave_init() hook that will be called, and if so, how does it differ and when is it called relative to security_enclave_load()?
d) What checks were you envisioning each of these calls making?
With the separate security_enclave_*() hooks, we could define and use new ENCLAVE__* permissions, e.g. ENCLAVE__LOAD, ENCLAVE__INIT, ENCLAVE__EXECUTE, ENCLAVE__EXECMEM, ENCLAVE__EXECMOD, if we want to distinguish these operations from regular file mmap/mprotect operations.
Quoted 23 lines
The userspace changes are fairly minimal:
- For SGX1, use
PROT_NONEfor the initialmmap()and refactorADD_PAGEtoADD_REGION.- For SGX2, do an explicit
ADD_REGIONon the ranges to be EAUG'd, and anACTIVATE_REGIONto make a region RX or R (no extraioctl()required to keep RW permissions).Because
ACTIVATE_REGIONcan only be done once per page, to do *abitrary*mprotect()transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.
- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.
RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 215 lines
From: [email protected] [mailto:[email protected]] On Behalf Of Stephen Smalley
Sent: Wednesday, May 29, 2019 7:08 AMQuoted 21 lines · Sean Christopherson
Quoted 20 lines · Cedric
Quoted 7 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PMQuoted 3 lines · Cedric
If we think of EADD as a way of mmap()'ing an enclave file into memory, would this
Quoted 7 lines
security_enclave_load()be the same assecurity_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.
What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.
Quoted 8 lines
Quoted 3 lines
But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to
security_mmap_file()by default?Mapping the enclave will still go through
security_mmap_file(), the extrasecurity_enclave_load()hook allows themmap()to usePROT_NONE.Quoted 1 line
Quoted 1 line
If it's going to be in an arbitrary file, then I think the signature needs to be more
like:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
int security_enclave_init(struct
vm_area_struct*sigstruct_vma,loff_t
sigstruct_offset,Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
const
sgx_sigstruct*sigstruct);So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.
Quoted 10 lines
Quoted 3 lines
Wouldn't the pair of {
sigstruct_vma,sigstruct_offset} be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer -vma->vm_start)?VMA has
vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);Quoted 1 line
Quoted 1 line
Quoted 1 line
Loosely speaking, an enclave (including initial contents of all of its pages and
their
Quoted 1 line
Quoted 1 line
Quoted 1 line
permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision
resistant
Quoted 1 line
Quoted 1 line
Quoted 1 line
property of SHA-2). So only one is needed for a decision, and either one would lead to
the
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
same decision. So I don't see anything making any sense here.
Quoted 2 lines
Theoretically speaking, if LSM can make a decision at EINIT by means of
security_enclave_load(), thensecurity_enclave_load()is never needed.Quoted 2 lines
In practice, I support keeping both because
security_enclave_load()can only approvean
Quoted 1 line
Quoted 1 line
Quoted 1 line
enumerable set while
security_enclave_load()can approve a non-enumerable set ofenclaves.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Moreover, in order to determine the validity of a MRENCLAVE (as in development of a
policy
Quoted 1 line
Quoted 1 line
Quoted 1 line
or in creation of a white/black list), system admins will need the audit log produced
by
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
security_enclave_load().I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also,
security_enclave_load()provides no protection against loading a mishmash of two different enclave files. I seesecurity_enclave_init()as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" andsecurity_enclave_load()as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."Quoted 6 lines
Sorry for the confusion. I was saying the same thing except that the decision of
security_enclave_load()doesn't have to depend on SIGSTRUCT. Given your prototype ofsecurity_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".Quoted 35 lines
Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling
/dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using/dev/sgx/enclavemeans users can't do per-enclave permissions, period.What is your objection to working on the sigstruct?
Quoted 26 lines
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Passing both would allow tying EXECMOD to
/dev/sgx/enclaveas Cedric wanted (without having to play games and pass/dev/sgx/enclavetosecurity_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.Quoted 6 lines
Quoted 6 lines
The benefit of putting it on .sigstruct is that it can be per-enclave.
As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirtylook.
Quoted 13 lines
Quoted 11 lines
This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.
(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)
I'm lost here. Dynamically linked enclaves, if running on SGX2, would need
RW->RX,i.e.
Quoted 82 lines
Quoted 6 lines
Quoted 6 lines
FILE__EXECMODon/dev/sgx/enclave. But they never need RWX, i.e.PROCESS__EXECMEM.Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual
mprotect()/mmap()permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply thatPROT_WRITE|PROT_EXECis allowed.Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that
sgx_enclave_load()with a NULL vma implies a transition from RW. For example, SELinux would interpretsgx_enclave_load(NULL, RX)as requiringFILE__EXECMOD.As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e.
sgx_enclave_load(NULL, RX)already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs fromMAY_RWtoMAY_RX.For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit
ioctl()to add the to-be-EAUG'd range. That just leaves us wanting anioctl()to set the post-EACCEPT{COPY} permissions.E.g.:
ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */ mprotect(addr, size, RW); ... EACCEPTCOPY -> EAUG /* page fault handler */ ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */ mprotect(addr, size, RX); ...And making
ACTIVATE_REGIONa single-shot per page eliminates the need for the MAXPERMS concept (see below).Quoted 15 lines
If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?
To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of
/dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on/dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of-EPERMreturned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part ofIOC_EINIT.Aha!
Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:
- Initial permissions for *EADD* pages are explicitly handled via
sgx_enclave_load()with the exact SECINFO permissions.- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.
- Runtime permissions break down as follows:
R - N/A, subset of RW (EAUG) W - N/A, subset of RW (EAUG) and x86 paging can't do W X - N/A, subset of RX (x86 paging can't do XO) RW - Handled by EAUG LSM hook (uses RW unconditionally) WX - N/A, subset of RWX (x86 paging can't do WX) RX - Handled by ACTIVATE_REGIONRWX - Handled by
ACTIVATE_REGIONIn other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):
- <prefix>_ACTIVATE_REGION(vma, perms) ->
sgx_enclave_load(NULL, perms)- <prefix>_ADD_REGION(vma) ->
sgx_enclave_load(vma, SECINFO.perms)Quoted 8 lines
- <prefix>_ADD_REGION(NULL) ->
sgx_enclave_load(NULL, RW)then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of
sgx_enclave_load(..., RW)would need to be different than normal shared memory, e.g.FILE__WRITEshould *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without*FILE__EXECMOD, it'd have the necessary information to do so.
Quoted 5 lines
Assuming that
sgx_enclave_load()is a LSM hook (probably namedsecurity_enclave_load()instead), then:a) Does the sigstruct file get passed to this hook in every case, even when vma is NULL? I think the answer is yes, just want to confirm.
I'm confused.
In the case of EADD (non-NULL vma), are we passing both vma and sigstruct file? If so, which file dictates allowed permissions, vma->vm_file or sigstruct, or both???
In the case of EAUG (NULL vma), all other parameters are constant for any given enclave. Then why do we call this same hook for every region added, assuming the hook will return the same value everytime anyway?
And it looks like ACTIVATE_REGION is needed only because the proposed security_enclave_load() would base its decision on the sigstruct file. An alternative is to base that decision on /dev/sgx/enclave. Of course the former has finer granularity but is that really necessary? From security perspective, only the weakest link matters. FILE__EXECMOD on a regular shared object could allow exploits of all bugs throughout the host process because code within that shared object is modifiable by not only itself but also any code within that same process. In contrast, FILE__EXECMOD on /dev/sgx/enclave only allows enclaves to modify themselves. They cannot modify each other, neither can "untrusted" code outside of enclaves modify any of them. So it doesn't look like a weaker link to me. Moreover, requiring FILE__EXECMOD on sigstruct means it could be used as a target buffer for code injection attacks. IMHO that *lowers* the security of the whole process.
Quoted 11 lines
b) Should we use a different hook for
ACTIVATE_REGIONthan forADD_REGIONor is the distinction between them irrelevant/unnecessary from an access control point of view? At present LSM/SELinux won't be able to distinguishACTIVATE_REGION(vma, RW)fromADD_REGION(NULL)above since they will both invoke the same hook with the same arguments IIUC. Does it matter? It's ok if the answer is no, just want to confirm.c) Is there still also a separate
security_enclave_init()hook that will be called, and if so, how does it differ and when is it called relative tosecurity_enclave_load()?
I think security_enclave_init() will always be useful, as it offers a way for LSM to implement whitelisting/blacklisting. Of course an LSM module like SELinux can look into the backing inode too. I think the hook should have a signature like:
int security_enclave_init(struct sgx_sigstruct __user *sigstruct);
An LSM that cares about the backing file could look into vm_file of the VMA covering the buffer, while an LSM that cares the sigstruct itself (e.g. signing key) could just look into the buffer.
Quoted 7 lines
d) What checks were you envisioning each of these calls making?
With the separate security_enclave_*() hooks, we could define and use new ENCLAVE__* permissions, e.g.
ENCLAVE__LOAD,ENCLAVE__INIT,ENCLAVE__EXECUTE,ENCLAVE__EXECMEM,ENCLAVE__EXECMOD, if we want to distinguish these operations from regular file mmap/mprotect operations.
I'm not sure if these ENCLAVE__* flags are an overkill, unless we want to enforce an enclave file cannot be loaded as a regular shared object or vice versa.
Quoted 24 lines
Quoted 24 lines
The userspace changes are fairly minimal:
- For SGX1, use
PROT_NONEfor the initialmmap()and refactorADD_PAGEtoADD_REGION.- For SGX2, do an explicit
ADD_REGIONon the ranges to be EAUG'd, and anACTIVATE_REGIONto make a region RX or R (no extraioctl()required to keep RW permissions).Because
ACTIVATE_REGIONcan only be done once per page, to do *abitrary*mprotect()transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.
- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 215 lines · Cedric
Quoted 215 lines
From: [email protected] [mailto:[email protected]] On Behalf Of Stephen Smalley
Sent: Wednesday, May 29, 2019 7:08 AMQuoted 21 lines · Sean Christopherson
Quoted 20 lines · Cedric
Quoted 7 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Saturday, May 25, 2019 5:58 PMQuoted 3 lines · Cedric
If we think of EADD as a way of mmap()'ing an enclave file into memory, would this
Quoted 7 lines
security_enclave_load()be the same assecurity_mmap_file(source_vma->vm_file, maxperm, MAP_PRIVATE), except that the target is now EPC instead of regular pages?Hmm, that's clever. Although it seems plausible that an LSM would want to allow RX or RWX of a given file page but only in the context of an approved enclave, so I think it should still be its own hook.
What do you mean by "in the context of an approved enclave"? EPC pages are *inaccessible* to any software until after EINIT. So it would never be a security concern to EADD a page with wrong permissions as long as the enclave would be denied eventually by LSM at EINIT.
Quoted 8 lines
Quoted 3 lines
But I acknowledge the difference between loading a page into regular memory vs. into EPC. So it's beneficial to have a separate hook, which if not hooked, would pass through to
security_mmap_file()by default?Mapping the enclave will still go through
security_mmap_file(), the extrasecurity_enclave_load()hook allows themmap()to usePROT_NONE.Quoted 1 line
Quoted 1 line
If it's going to be in an arbitrary file, then I think the signature needs to be more
like:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
int security_enclave_init(struct
vm_area_struct*sigstruct_vma,loff_t
sigstruct_offset,Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
const
sgx_sigstruct*sigstruct);So that the LSM still has the opportunity to base its decision on the contents of the SIGSTRUCT. Actually, we need that change regardless.
Quoted 10 lines
Quoted 3 lines
Wouldn't the pair of {
sigstruct_vma,sigstruct_offset} be the same as just a pointer, because the VMA could be looked up using the pointer and the offset would then be (pointer -vma->vm_start)?VMA has
vm_file, e.g. the .sigstruct file labeled by LSMs. That being said, why does the LSM need the VMA? E.g. why not this?int security_enclave_init(struct file *file, struct sgx_sigstruct *sigstruct);Quoted 1 line
Quoted 1 line
Quoted 1 line
Loosely speaking, an enclave (including initial contents of all of its pages and
their
Quoted 1 line
Quoted 1 line
Quoted 1 line
permissions) and its MRENCLAVE are a 1-to-1 correspondence (given the collision
resistant
Quoted 1 line
Quoted 1 line
Quoted 1 line
property of SHA-2). So only one is needed for a decision, and either one would lead to
the
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
same decision. So I don't see anything making any sense here.
Quoted 2 lines
Theoretically speaking, if LSM can make a decision at EINIT by means of
security_enclave_load(), thensecurity_enclave_load()is never needed.Quoted 2 lines
In practice, I support keeping both because
security_enclave_load()can only approvean
Quoted 1 line
Quoted 1 line
Quoted 1 line
enumerable set while
security_enclave_load()can approve a non-enumerable set ofenclaves.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Moreover, in order to determine the validity of a MRENCLAVE (as in development of a
policy
Quoted 1 line
Quoted 1 line
Quoted 1 line
or in creation of a white/black list), system admins will need the audit log produced
by
Quoted 17 lines
Quoted 10 lines
Quoted 9 lines
security_enclave_load().I'm confused. Things like MRSIGNER aren't known until the SIGSTRUCT shows up. Also,
security_enclave_load()provides no protection against loading a mishmash of two different enclave files. I seesecurity_enclave_init()as "verify this SIGSTRUCT against your policy on who may sign enclaves and/or grant EXECMOD depending on SIGSTRUCT" andsecurity_enclave_load()as "implement your EXECMOD / EXECUTE / WRITE / whatever policy and possibly check enclave files for some label."Quoted 6 lines
Sorry for the confusion. I was saying the same thing except that the decision of
security_enclave_load()doesn't have to depend on SIGSTRUCT. Given your prototype ofsecurity_enclave_load(), I think we are on the same page. I made the above comment to object to the idea of "require that the sigstruct be supplied before any EADD operations so that the maxperm decisions can depend on the sigstruct".Quoted 35 lines
Except that having the sigstruct allows using the sigstruct as the proxy for the enclave. I think the last big disconnect is that Andy and I want to tie everything to an enclave-specific file, i.e. sigstruct, while you are proposing labeling
/dev/sgx/enclave. If someone wants to cram several sigstructs into a single file, so be it, but using/dev/sgx/enclavemeans users can't do per-enclave permissions, period.What is your objection to working on the sigstruct?
Quoted 26 lines
Quoted 26 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Passing both would allow tying EXECMOD to
/dev/sgx/enclaveas Cedric wanted (without having to play games and pass/dev/sgx/enclavetosecurity_enclave_load()), but I don't think there's anything fundamentally broken with using .sigstruct for EXECMOD. It requires more verbose labeling, but that's not a bad thing.Quoted 6 lines
Quoted 6 lines
The benefit of putting it on .sigstruct is that it can be per-enclave.
As I understand it from Fedora packaging, the way this works on distros is generally that a package will include some files and
their associated labels, and, if the package needs EXECMOD, then the files are labeled with EXECMOD and the author of the relevant code might get a dirtylook.
Quoted 13 lines
Quoted 11 lines
This could translate to the author of an exclave that needs RWX regions getting a dirty look without leaking this permission into other enclaves.
(In my opinion, the dirty looks are actually the best security benefit of the entire concept of LSMs making RWX difficult. A sufficiently creative attacker can almost always bypass W^X restrictions once they’ve pwned you, but W^X makes it harder to pwn you in the first place, and SELinux makes it really obvious when packaging a program that doesn’t respect W^X. The upshot is that a lot of programs got fixed.)
I'm lost here. Dynamically linked enclaves, if running on SGX2, would need
RW->RX,i.e.
Quoted 90 lines
Quoted 6 lines
Quoted 6 lines
FILE__EXECMODon/dev/sgx/enclave. But they never need RWX, i.e.PROCESS__EXECMEM.Hmm. If we want to make this distinction, we need something a big richer than my proposed callbacks. A check of the actual
mprotect()/mmap()permissions would also be needed. Specifically, allowing MAXPERM=RWX wouldn't imply thatPROT_WRITE|PROT_EXECis allowed.Actually, I think we do have everything we need from an LSM perspective. LSMs just need to understand that
sgx_enclave_load()with a NULL vma implies a transition from RW. For example, SELinux would interpretsgx_enclave_load(NULL, RX)as requiringFILE__EXECMOD.As Cedric mentioned earlier, the host process doesn't necessarily know which pages will end up RW vs RX, i.e.
sgx_enclave_load(NULL, RX)already has to be invoked at runtime, and when that happens, the kernel can take the opportunity to change the VMAs fromMAY_RWtoMAY_RX.For simplicity in the kernel and clarity in userspace, it makes sense to require an explicit
ioctl()to add the to-be-EAUG'd range. That just leaves us wanting anioctl()to set the post-EACCEPT{COPY} permissions.E.g.:
ioctl(<prefix>_ADD_REGION, { NULL }) /* NULL == EAUG, MAY_RW */ mprotect(addr, size, RW); ... EACCEPTCOPY -> EAUG /* page fault handler */ ioctl(<prefix>_ACTIVATE_REGION, { addr, size, RX}) /* MAY_RX */ mprotect(addr, size, RX); ...And making
ACTIVATE_REGIONa single-shot per page eliminates the need for the MAXPERMS concept (see below).Quoted 15 lines
If we keep only one MAXPERM, wouldn't this be the current behavior of mmap()/mprotect()?
To be a bit more clear, system admin sets MAXPERM upper bound in the form of FILE__{READ|WRITE|EXECUTE|EXECMOD} of
/dev/sgx/enclave. Then for a process/enclave, if what it requires falls below what's allowed on/dev/sgx/enclave, then everything will just work. Otherwise, it fails in the form of-EPERMreturned from mmap()/mprotect(). Please note that MAXPERM here applies to "runtime" permissions, while "initial" permissions are taken care of by security_enclave_{load|init}. "initial" permissions could be more permissive than "runtime" permissions, e.g., RX is still required for initial code pages even though system admins could disable dynamically loaded code pages by *not* giving FILE__{EXECUTE|EXECMOD}. Therefore, the "initial" mapping would still have to be done by the driver (to bypass LSM), either via a new ioctl or as part ofIOC_EINIT.Aha!
Starting with Cedric's assertion that initial permissions can be taken directly from SECINFO:
- Initial permissions for *EADD* pages are explicitly handled via
sgx_enclave_load()with the exact SECINFO permissions.- Initial permissions for *EAUG* are unconditionally RW. EACCEPTCOPY requires the target EPC page to be RW, and EACCEPT with RO is useless.
- Runtime permissions break down as follows:
R - N/A, subset of RW (EAUG)
W - N/A, subset of RW (EAUG) and x86 paging can't do W
X - N/A, subset of RX (x86 paging can't do XO)
RW - Handled by EAUG LSM hook (uses RW unconditionally) WX - N/A, subset of RWX (x86 paging can't do WX) RX - Handled byACTIVATE_REGION
RWX - Handled byACTIVATE_REGIONIn other words, if we define the SGX -> LSM calls as follows (minus the file pointer and other params for brevity):
- <prefix>_ACTIVATE_REGION(vma, perms) ->
sgx_enclave_load(NULL, perms)- <prefix>_ADD_REGION(vma) ->
sgx_enclave_load(vma, SECINFO.perms)- <prefix>_ADD_REGION(NULL) ->
sgx_enclave_load(NULL, RW)then SGX and LSMs have all the information and hooks needed. The catch is that the LSM semantics of
sgx_enclave_load(..., RW)would need to be different than normal shared memory, e.g.FILE__WRITEshould *not* be required, but that's ok since it's an SGX specific hook. And if for some reason an LSM wanted to gate access to EAUG *without*FILE__EXECMOD, it'd have the necessary information to do so.
Quoted 7 lines
Quoted 5 lines
Assuming that
sgx_enclave_load()is a LSM hook (probably namedsecurity_enclave_load()instead), then:a) Does the sigstruct file get passed to this hook in every case, even when vma is NULL? I think the answer is yes, just want to confirm.
I'm confused.
I'm finding it difficult to follow as well, so my questions are just an attempt to understand the latest model.
Quoted 1 line
In the case of EADD (non-NULL vma), are we passing both vma and sigstruct file? If so, which file dictates allowed permissions,
vma->vm_fileor sigstruct, or both???
My impression was that they were going to pass both, but the sigstruct file is the target of permission checks. The vma if non-NULL would be used to determine whether PROT_EXEC is being added or was already present and whether EXECMOD needs to be checked (i.e. copy-on-write has occurred and PROT_EXEC is being added).
Quoted 1 line
In the case of EAUG (NULL vma), all other parameters are constant for any given enclave. Then why do we call this same hook for every region added, assuming the hook will return the same value everytime anyway?
Yes, I was wondering about that as well.
Quoted 1 line
And it looks like
ACTIVATE_REGIONis needed only because the proposedsecurity_enclave_load()would base its decision on the sigstruct file. An alternative is to base that decision on/dev/sgx/enclave. Of course the former has finer granularity but is that really necessary? From security perspective, only the weakest link matters.FILE__EXECMODon a regular shared object could allow exploits of all bugs throughout the host process because code within that shared object is modifiable by not only itself but also any code within that same process. In contrast,FILE__EXECMODon/dev/sgx/enclaveonly allows enclaves to modify themselves. They cannot modify each other, neither can "untrusted" code outside of enclaves modify any of them. So it doesn't look like a weaker link to me. Moreover, requiringFILE__EXECMODon sigstruct means it could be used as a target buffer for code injection attacks. IMHO that *lowers* the security of the whole process.
This is partly why I suggested separate ENCLAVE__EXECMOD and other checks below, so we can distinguish between FILE__EXECMOD versus ENCLAVE__EXECMOD on the sigstruct file. If using /dev/sgx/enclave as the target, then we don't need a separate permission per se but we lose the per-sigstruct granularity.
Quoted 13 lines
Quoted 9 lines
b) Should we use a different hook for
ACTIVATE_REGIONthan forADD_REGIONor is the distinction between them irrelevant/unnecessary from an access control point of view? At present LSM/SELinux won't be able to distinguishACTIVATE_REGION(vma, RW)fromADD_REGION(NULL)above since they will both invoke the same hook with the same arguments IIUC. Does it matter? It's ok if the answer is no, just want to confirm.Quoted 3 lines
c) Is there still also a separate
security_enclave_init()hook that will be called, and if so, how does it differ and when is it called relative tosecurity_enclave_load()?
Quoted 5 lines
I think
security_enclave_init()will always be useful, as it offers a way for LSM to implement whitelisting/blacklisting. Of course an LSM module like SELinux can look into the backing inode too. I think the hook should have a signature like:int
security_enclave_init(struct sgx_sigstruct __user *sigstruct);An LSM that cares about the backing file could look into
vm_fileof the VMA covering the buffer, while an LSM that cares the sigstruct itself (e.g. signing key) could just look into the buffer.
I'm not a fan of passing __user pointers to LSM hooks. And it certainly shouldn't be looking at the buffer since it could change between the time of check and time of use.
Quoted 10 lines
Quoted 8 lines
d) What checks were you envisioning each of these calls making?
With the separate security_enclave_*() hooks, we could define and use new ENCLAVE__* permissions, e.g.
ENCLAVE__LOAD,ENCLAVE__INIT,ENCLAVE__EXECUTE,ENCLAVE__EXECMEM,ENCLAVE__EXECMOD, if we want to distinguish these operations from regular file mmap/mprotect operations.I'm not sure if these ENCLAVE__* flags are an overkill, unless we want to enforce an enclave file cannot be loaded as a regular shared object or vice versa.
ENCLAVE__LOAD and/or ENCLAVE__INIT would be to support whitelisting of what enclaves can be loaded/initialized by the process. That's separate from the W^X discussion. Those permissions would be between the process and either the sigstruct file or the enclave file (the consensus seemed to be the sigstruct file as the stronger/more complete binding of the enclave). We probably only need one of those two permission checks not both.
ENCLAVE__EXECUTE, ENCLAVE__EXECMEM, ENCLAVE__EXECMOD would allow distinctions between host process mmap/mprotect PROT_EXEC operations (which would continue to apply FILE__EXECUTE, PROCESS__EXECMEM, and FILE__EXECMOD checks if appropriate) and the driver's setting of initial and runtime permissions (which would apply ENCLAVE__EXECUTE, ENCLAVE__EXECMEM, and ENCLAVE__EXECMOD checks if appropriate). That's particularly helpful if we are using the sigstruct or enclave file as the target of all checks instead of /dev/sgx/enclave, so that we don't have to allow FILE__EXECUTE or FILE__EXECMOD to the sigstruct file by the host process.
Quoted 26 lines
Quoted 25 lines
Quoted 25 lines
The userspace changes are fairly minimal:
- For SGX1, use
PROT_NONEfor the initialmmap()and refactorADD_PAGEtoADD_REGION.- For SGX2, do an explicit
ADD_REGIONon the ranges to be EAUG'd, and anACTIVATE_REGIONto make a region RX or R (no extraioctl()required to keep RW permissions).Because
ACTIVATE_REGIONcan only be done once per page, to do *abitrary*mprotect()transitions, userspace would need to set the added/activated permissions to be a superset of the transitions, e.g. RW -> RX would require RWX, but that's a non-issue.- For SGX1 it's a nop since it's impossible to change the EPCM permissions, i.e. the page would need to be RWX regardless.
- For SGX2, userspace can suck it up and request RWX to do completely arbitrary transitions (working as intended), or the kernel can support trimming (removing) pages from an enclave, which would allow userspace to do "arbitrary" transitions by first removing the page.
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Hi all-
After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.
Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.
First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:
- You can create a WX or RWX mapping if and only if you have EXECMEM.
- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.
- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.
And I have two design proposals. One is static and one is dynamic. To implement either one, we will probably need a new .may_mprotect vm operation, and that operation can call an LSM hook. Or we can give LSMs a way to detect that a given vm_area_struct is an enclave. As I see it, this is an implementation detail that is certainly solveable.
Static proposal:
EADD takes an execute_intent flag. It calls a new hook:
int security_enclave_load(struct vm_area_struct *source, bool execute_intent);
This hook will fail if execute_intent==true and the caller has neither EXECUTE, EXECMOD, nor EXECMEM.
EAUG sets execute_intent = false.
EINIT takes a sigstruct pointer. SGX can (when initially upstreamed or later on once there's demand) call a new hook:
security_enclave_init(struct sigstruct *sigstruct, structvm_area_struct *source);
mmap() and mprotect() will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.
Dynamic proposal:
EADD does not take any special flags. It does something like this internally:
bool execute_intent = true;
int security_enclave_load(struct vm_area_struct *source, bool*execute_intent);
The implementation of security_enclave_load() may set *execute_intent to false. The driver records execute_intent after the LSM is done.
mmap() and mprotect() will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.
A benefit of the static proposal is that audit failures due to a lack of EXECUTE permission are easy to implement and to understand in the lods. With the dynamic model, we can only really audit the lack of EXECMOD or EXECMEM. A benefit of the dynamic model is that we hide what is arguably a decently large wart from the API.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 10 lines · Andy Lutomirski
Hi all-
After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.
Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.
Quoted 8 lines
First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:
- You can create a WX or RWX mapping if and only if you have EXECMEM.
- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.
EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or /dev/sgx/enclave?
Quoted 3 lines
- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.
What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or /dev/sgx/enclave?
Quoted 11 lines
And I have two design proposals. One is static and one is dynamic. To implement either one, we will probably need a new .may_mprotect vm operation, and that operation can call an LSM hook. Or we can give LSMs a way to detect that a given
vm_area_structis an enclave. As I see it, this is an implementation detail that is certainly solveable.Static proposal:
Quoted 6 lines
EADD takes an
execute_intentflag. It calls a new hook:int security_enclave_load(struct vm_area_struct *source, bool execute_intent);This hook will fail if execute_intent==true and the caller has neither EXECUTE, EXECMOD, nor EXECMEM.
EADD execute_intent flag is originally provided by whom (userspace or driver) on what basis? Which file is referenced by source->vm_file? Why trigger all three checks up front versus only checking if needed? Won't this trigger a lot of unnecessary EXECMOD and EXECMEM denials that will need to be dontaudit'd? What if there is a mismatch between execute_intent and the initial permissions?
Quoted 8 lines
EAUG sets
execute_intent= false.EINIT takes a sigstruct pointer. SGX can (when initially upstreamed or later on once there's demand) call a new hook:
security_enclave_init(struct sigstruct *sigstruct, struct
vm_area_struct*source);
Is struct sigstruct the same as struct sgx_sigstruct in the current patches (i.e. just the sigstruct data, no file)? What file is referenced by source->vm_file (the sigstruct or the enclave or /dev/sgx/enclave)? Is this hook only for enforcing a whitelist on what enclaves can be loaded? What is the target of the check?
Quoted 4 lines
mmap()andmprotect()will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.
Does this occur for both setting initial permissions and runtime permissions or just runtime? Both userspace- and driver-initiated mmap/mprotect operations or just userspace-initiated ones? Does the driver use interfaces that call the mmap/mprotect hooks or lower level functions?
Quoted 5 lines
Dynamic proposal:
Quoted 8 lines
EADD does not take any special flags. It does something like this internally:
bool
execute_intent= true;
int security_enclave_load(structvm_area_struct*source, bool*execute_intent);The implementation of
security_enclave_load()may set*execute_intentto false. The driver recordsexecute_intentafter the LSM is done.
On what basis does LSM decide whether to set *execute_intent? If the process lacks all three permissions? What if there is a mismatch with the initial permissions?
Quoted 14 lines
mmap()andmprotect()will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.A benefit of the static proposal is that audit failures due to a lack of EXECUTE permission are easy to implement and to understand in the lods. With the dynamic model, we can only really audit the lack of EXECMOD or EXECMEM. A benefit of the dynamic model is that we hide what is arguably a decently large wart from the API.
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 16 lines · Stephen Smalley
Quoted 14 lines · Andy Lutomirski
Hi all-
After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.
Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.
First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:
Quoted 7 lines
Quoted 4 lines
- You can create a WX or RWX mapping if and only if you have EXECMEM.
- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or
/dev/sgx/enclave?
I leave that decision to you :) The user should need permission to do an execmod thing on an enclave, however that wants to be encoded.
Quoted 7 lines
Quoted 4 lines
- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or
/dev/sgx/enclave?
Enclave file -- that is, the file backing the vma from which the data is loaded.
Quoted 19 lines
Quoted 12 lines
And I have two design proposals. One is static and one is dynamic. To implement either one, we will probably need a new .may_mprotect vm operation, and that operation can call an LSM hook. Or we can give LSMs a way to detect that a given
vm_area_structis an enclave. As I see it, this is an implementation detail that is certainly solveable.Static proposal:
Quoted 6 lines
EADD takes an
execute_intentflag. It calls a new hook:int security_enclave_load(struct vm_area_struct *source, bool execute_intent);This hook will fail if execute_intent==true and the caller has neither EXECUTE, EXECMOD, nor EXECMEM.
Quoted 6 lines
EADD
execute_intentflag is originally provided by whom (userspace or driver) on what basis? Which file is referenced bysource->vm_file? Why trigger all three checks up front versus only checking if needed? Won't this trigger a lot of unnecessary EXECMOD and EXECMEM denials that will need to be dontaudit'd? What if there is a mismatch betweenexecute_intentand the initial permissions?
It's provided by userspace based on whether it thinks the data in question is enclave code. source->vm_file is the file from which the code is being loaded. I'm assuming that the user code will only set excute_intent ==true if it actually wants to execute the code, so, if there's a denial, it will be fatal. The normal case will be that the request will be granted on the basis of EXECUTE.
Quoted 7 lines
Quoted 7 lines
EAUG sets
execute_intent= false.EINIT takes a sigstruct pointer. SGX can (when initially upstreamed or later on once there's demand) call a new hook:
Quoted 8 lines
Quoted 2 lines
security_enclave_init(struct sigstruct *sigstruct, struct
vm_area_struct*source);Is struct sigstruct the same as struct
sgx_sigstructin the current patches (i.e. just the sigstruct data, no file)? What file is referenced bysource->vm_file(the sigstruct or the enclave or/dev/sgx/enclave)? Is this hook only for enforcing a whitelist on what enclaves can be loaded? What is the target of the check?
sigstruct is just the data. source->vm_file is the file from which the sigstruct came, which could be a .sigstruct file or could be the main executable or a DSO that contains an embedded enclave. The sigstruct data is there so that an LSM (not necessarily SELinux) could check MRENCLAVE or MRSIGNER, and the source is there so that the file's label can be checked.
Quoted 11 lines
Quoted 5 lines
mmap()andmprotect()will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.Does this occur for both setting initial permissions and runtime permissions or just runtime? Both userspace- and driver-initiated mmap/mprotect operations or just userspace-initiated ones? Does the driver use interfaces that call the mmap/mprotect hooks or lower level functions?
These would occur for any mmap(), mprotect(), or ioctl() that changes VMA permissions. Actually arranging for the hooks to be called is an implementation detail that might require a new .mprotect vm_operation. As an alternative, security_enclave_init() or similar could supply may_execmod and may_execmem flags to the driver, and the driver could do these checks on its own when mmap() and mprotect() happen without a new LSM callback.
Quoted 12 lines
Quoted 12 lines
Dynamic proposal:
EADD does not take any special flags. It does something like this internally:
bool
execute_intent= true;
int security_enclave_load(structvm_area_struct*source, bool*execute_intent);
Quoted 7 lines
Quoted 2 lines
The implementation of
security_enclave_load()may set*execute_intentto false. The driver recordsexecute_intentafter the LSM is done.On what basis does LSM decide whether to set
*execute_intent? If the process lacks all three permissions? What if there is a mismatch with the initial permissions?
I think it would set *execute_intent=false if the process lacks EXECUTE on source->vm_file. I'm not sure any more complexity is required. If the enclave has EXECMOD, then it will still work on the basis of the mmap/mprotect rules.
Quoted 15 lines
Quoted 14 lines
mmap()andmprotect()will require EXECMEM to create WX or RWX mappings. They will require EXECMOD to create RX or X mappings of an execute_intent==false page. They require no permissions in the other cases.A benefit of the static proposal is that audit failures due to a lack of EXECUTE permission are easy to implement and to understand in the lods. With the dynamic model, we can only really audit the lack of EXECMOD or EXECMEM. A benefit of the dynamic model is that we hide what is arguably a decently large wart from the API.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 19 lines · Andy Lutomirski
Quoted 18 lines · Stephen Smalley
Quoted 16 lines · Andy Lutomirski
Hi all-
After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.
Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.
First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:
- You can create a WX or RWX mapping if and only if you have EXECMEM.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or
/dev/sgx/enclave?I leave that decision to you :) The user should need permission to do an execmod thing on an enclave, however that wants to be encoded.
But that decision dictates how the SGX API handles sigstruct. If LSMs want to associate EXECMOD with sigstruct, then SGX needs to take sigstruct early and hold a reference to the file for the lifetime of the enclave. And if we're going to do that, the whole approach of inheriting permissions from source VMAs becomes unnecessary complexity.
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or
/dev/sgx/enclave?Enclave file -- that is, the file backing the vma from which the data is loaded.
It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).
I have two gripes with that approach:
- Requires enclave builder to mark enclave pages executable in the non-enclave VMAs, which may unnecessarily require EXECMOD on the source file, or even worse, EXECMEM, and potentially increases the attack surface since the file must be executable.
- Is completely unnecessary if the enclave holds a reference to the sigstruct file, as LSMs can easily apply labels to the sigstruct, e.g. EXECUTE on the sigstruct instead of EXECUTE on the source file.
After the bajillion mails we've generated, AIUI we've come up with two concepts that are viable: inheriting permissions from the source VMA vs. using sigstruct as a proxy for the enclave. Andy's proposals rely on the inheritance concept. The proposal below is based on the sigstruct proxy concept.
For those not familiar with SGX details, sigstruct can be used as a proxy because hardware enforces that the measurement stored in the sigstruct exactly matches the measurement generated by the enclave build process, e.g. adding a page as RX instead of R will change the measurement.
Core Concepts:
- FILE_{READ,WRITE,EXEC} on /dev/sgx/enclave effectively gates access to EPC. All real world enclaves will need all three permissions.
- sigstruct is the proxy for enclave from an LSM perspective, e.g. SELinux can define ENCLAVE__EXECUTE and ENCLAVE__EXECMOD and apply them to the sigstruct file.
- Take sigstruct at ECREATE so that ADD_REGION immediately followed by mprotect() works as expected (because SGX.mprotect() needs sigstruct to pass to security_enclave_mprotect(), see below).
- SGX driver takes a reference to the backing sigstruct file if it exists so that the file can be provided to LSMs during mprotect().
- Optional: SGX driver *requires* sigstruct to be backed by file, purely to enforce userspace infrastructure is in place for LSM support.
W^X handling:
- mmap() to /dev/sgx/enclave only allowed with PROT_NONE, i.e. force userspace through mprotect() to simplify the kernel implementation.
- Add vm_ops mprotect() ops hook (I'll refer to SGX's implementation as SGX.mprotect())
- Take explicit ALLOW_WRITE at ADD_REGION, a.k.a. EADD
- ADD_REGION also used to describe EAUG region (tentatively for SGX2).
- Track "can be written at some point in time (past or future)" as ALLOW_WRITE (to avoid confusiong with MAY_WRITE). A priori knowledge of writability avoids having to track/coordinate PROT_WRITE across VMAs and MMs.
- SGX.mprotect() returns -EPERM if PROT_WRITE && !ALLOW_WRITE.
- Add security_enclave_mprotect() LSM hook, called by SGX.mprotect(), e.g. int security_enclave_mprotect(struct file *sigstruct, unsigned long prot, bool allow_write)
- Intention is that EXECMOD is required if PROT_EXEC and ALLOW_WRITE.
Enclave {white,black}listing:
- Optional/Future: add security_enclave_create(), invoked during SGX ECREATE ioctl(), e.g.
int security_enclave_create(struct vm_area_struct *sigstruct)
- If this LSM hook is implemented, having sigstruct at ECREATE allows LSMs to determine whether or not the enclave is allowed to execute before allocating EPC for the enclave, e.g. unwanted enclaves can't DoS wanted enclaves.
LSM implementation possibilities:
- Define ENCLAVE__EXECUTE and ENCLAVE__EXECMOD, require them on the process. Does not require sigstruct to be backed by file, but cannot achieve per-enclave granularity.
- Define ENCLAVE__EXECUTE and ENCLAVE__EXECMOD, require them on the sigstruct, i.e. force sigstruct to reside in filesystem. Allows per-enclave granularity.
- Reuse FILE__EXECUTE and FILE__EXECMOD on sigstruct. Likely has implications that may or may not be concerning, e.g. the sigstruct file itself is weirdly executable.
- Adding ENCLAVE__EXECUTE and ENCLAVE__EXECMOD means the sigstruct, which may be emdedded in the same file as the enclave, does *not* require FILE__EXECUTE or FILE__EXECMOD, e.g. can be read-only.
- LSMs can (will?) require ENCLAVE__EXECUTE and ENCLAVE__EXECMOD to effectively map an enclave, even if the process acquired the enclave via SCM_RIGHTS (enclaves are tracked by fds). This is good or bad depending on your perspective.
Userspace changes:
- EADD ioctl adds flags param to take ALLOW_WRITE
- ECREATE ioctl takes sigstruct instead of EINIT
- Initial mmap() must be PROT_NONE.
- sigstruct likely needs to reside in a file (this may not affect some userspace implementations).
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 41 lines · Sean Christopherson
Quoted 19 lines · Andy Lutomirski
Quoted 18 lines · Stephen Smalley
Quoted 16 lines · Andy Lutomirski
Hi all-
After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.
Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.
First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's discretion:
- You can create a WX or RWX mapping if and only if you have EXECMEM.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
- To create an X mapping of an enclave page that has ever been W, you
need EXECMOD.EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or
/dev/sgx/enclave?I leave that decision to you :) The user should need permission to do an execmod thing on an enclave, however that wants to be encoded.
But that decision dictates how the SGX API handles sigstruct. If LSMs want to associate EXECMOD with sigstruct, then SGX needs to take sigstruct early and hold a reference to the file for the lifetime of the enclave. And if we're going to do that, the whole approach of inheriting permissions from source VMAs becomes unnecessary complexity.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or
/dev/sgx/enclave?Enclave file -- that is, the file backing the vma from which the data is loaded.
It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).
I actually meant for it to *not* work like this. I don't want the source VMA to have to be VM_EXEC. I think the LSM should just check permissions on ->vm_file.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 6 lines · Andy Lutomirski
Quoted 4 lines · Sean Christopherson
Quoted 1 line · Andy Lutomirski
Enclave file -- that is, the file backing the vma from which the data is loaded.
Quoted 7 lines
Quoted 3 lines
It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).
I actually meant for it to *not* work like this. I don't want the source VMA to have to be
VM_EXEC. I think the LSM should just check permissions on ->vm_file.
But if ->vm_file is NULL, i.e. the enclave is not backed by a file, then PROCESS__EXECMEM is required (or more likely, ENCLAVE__EXECMEM).
In practice, it's the same net effect of using sigstruct as a proxy, i.e. *something* has to get to the file system to avoid EXECMEM. But putting the entire enclave to the filesystem seems like a heaver lift than dumping the sigstruct.
And if sigstruct needs to be in the file system for security_enclave_create/init()...
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Andy Lutomirski
Quoted 7 lines · Sean Christopherson
Quoted 1 line · Andy Lutomirski
Enclave file -- that is, the file backing the vma from which the data is loaded.
It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).
Quoted 7 lines
Quoted 3 lines
I actually meant for it to *not* work like this. I don't want the source VMA to have to be
VM_EXEC. I think the LSM should just check permissions on ->vm_file.But if ->vm_file is NULL, i.e. the enclave is not backed by a file, then
PROCESS__EXECMEMis required (or more likely,ENCLAVE__EXECMEM).
If ->vm_file is NULL, then I think some privilege is needed. I suppose the policy could have a new lesser permission EXECUNTRUSTED which is like EXECMOD but you can't modify it. I'm not convinced this is particular important.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 18 lines · Andy Lutomirski
Quoted 16 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 7 lines · Sean Christopherson
Quoted 1 line · Andy Lutomirski
Enclave file -- that is, the file backing the vma from which the data is loaded.
It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).
I actually meant for it to *not* work like this. I don't want the source VMA to have to be
VM_EXEC. I think the LSM should just check permissions on ->vm_file.
Quoted 8 lines
Quoted 3 lines
But if ->vm_file is NULL, i.e. the enclave is not backed by a file, then
PROCESS__EXECMEMis required (or more likely,ENCLAVE__EXECMEM).If ->vm_file is NULL, then I think some privilege is needed. I suppose the policy could have a new lesser permission EXECUNTRUSTED which is like EXECMOD but you can't modify it. I'm not convinced this is particular important.
Assuming MRENCLAVE generated by Graphene or any other hosting scheme are stable[1], then avoiding EXEC<whatever> means the user can effectively whitelist what enclaves are runnable by Graphene, even if the kernel doesn't implement security_enclave_create/init().
I agree that it probably isn't all that important, it's more of a "why not" argument, i.e. what is gained by not using sigstruct as a proxy?
[1] What in the world is being attested if MRENCLAVE isn't stable?
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
On Thu, May 30, 2019 at 02:36:01PM -0700, Sean Christopherson wrote:
Good morning, I hope everyone had a pleasant weekend.
Quoted 10 lines
Assuming MRENCLAVE generated by Graphene or any other hosting scheme are stable[1], then avoiding EXEC<whatever> means the user can effectively whitelist what enclaves are runnable by Graphene, even if the kernel doesn't implement security_enclave_create/init().
I agree that it probably isn't all that important, it's more of a "why not" argument, i.e. what is gained by not using sigstruct as a proxy?
[1] What in the world is being attested if MRENCLAVE isn't stable?
The cryptographic identity of the entity that signed the enclave and generated the SIGSTRUCT.
At the risk of being the monotone in the choir, any relevant SGX security controls require verifying the identity of whoever signed the identity characteristics (SIGSTRUCT) of the image that initiates the execution of an SGX TEE. Other then verifying the initial execution image, the MRENCLAVE value isn't all that relevant.
This issue is further evidenced by the fact that sealing data to an enclave uses the MRSIGNER variant of ENCLU[EGETKEY] key derivation.
The current work on LSM controls seems to focus on the identity of the entity that is requesting the image to be loaded rather then who actually signed, and presumably authored, the code. As I have previously noted, with SGX2/EDMM, a platform owner may not even have any visibility into the code that an SGX TEE may ultimately load and execute.
Any security relevant LSM control in this space has to focus on providing the platform owner the ability to take action based on the contents of the SIGSTRUCT of the initiating image. In addition to the identity of who is requesting the image to be loaded.
Have a good week.
Dr. Greg
As always,
Dr. G.W. Wettstein, Ph.D. Enjellic Systems Development, LLC. 4206 N. 19th Ave. Specializing in information infra-structure Fargo, ND 58102 development.
PH: 701-281-1686
FAX: 701-281-3949 EMAIL: [email protected]
------------------------------------------------------------------------------
"Experience is something you don't get until just after you need it."
-- Olivier
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonQuoted 9 lines · Sean Christopherson
Assuming MRENCLAVE generated by Graphene or any other hosting scheme are stable[1], then avoiding EXEC<whatever> means the user can effectively whitelist what enclaves are runnable by Graphene, even if the kernel doesn't implement security_enclave_create/init().
I agree that it probably isn't all that important, it's more of a "why not" argument, i.e. what is gained by not using sigstruct as a proxy?
[1] What in the world is being attested if MRENCLAVE isn't stable?
If I've understood correctly, Graphene uses a single loader enclave that loads the executable in.
/Jarkko
RE: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Andy LutomirskiHi Andy,
I'm just curious how Sean convinced you that "MAXPERM doesn't work". More specifically, I'm objecting to the statement of "the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed".
I'm still new to LSM so my understanding may not be correct. But I think LSM policy defines a boundary that "loosely" restricts what a process can do. By "loosely", I mean it is usually more permissive than a process needs. For example, FILE__EXECMOD basically says there are self-modifying code in a file, but it isn't specific on which pages contain self-modifying code, hence *all* pages are allowed mprotect(RW->RX) even though only a (small) subset actually need it. LSM policies are static too, as FILE__EXECMOD is given by a system admin to be associated with the disk file, instead of being requested/programmed by any processes loading/mapping that file.
So I think the same rationale applies to enclaves. Your original idea of MAXPERM is the policy set forth by system admin and shall *never* change at runtime. If an enclave is dynamically linked and needs to bring in code pages at runtime, the admin needs to enable it by setting, say ENCLAVE__EXECMOD, in the sigstruct file. Then all EAUG'ed pages will receive RWX as MAXPERM. The process would then mprotect() selective pages to be RX but which exact set of pages doesn't concern LSM usually.
Quoted 50 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Thursday, May 30, 2019 12:21 PMQuoted 45 lines · Sean Christopherson
Quoted 20 lines · Andy Lutomirski
Quoted 19 lines · Stephen Smalley
Quoted 17 lines · Andy Lutomirski
Hi all-
After an offline discussion with Sean yesterday, here are some updates to the user API parts of my proposal.
Unfortunately, Sean convinced me that MAXPERM doesn't work the way I described it because, for SGX2, the enclave loader won't know at load time whether a given EAUG-ed page will ever be executed. So here's an update.
First, here are the requrements as I see them, where EXECUTE, EXECMOD, and EXECMEM could be substituted with other rules at the LSM's
discretion:- You can create a WX or RWX mapping if and only if you have EXECMEM.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
- To create an X mapping of an enclave page that has ever been
W, you need EXECMOD.EXECMOD to what file? The enclave file from which the page's content originated, the sigstruct file, or
/dev/sgx/enclave?I leave that decision to you :) The user should need permission to do an execmod thing on an enclave, however that wants to be encoded.
But that decision dictates how the SGX API handles sigstruct. If LSMs want to associate EXECMOD with sigstruct, then SGX needs to take sigstruct early and hold a reference to the file for the lifetime of the enclave. And if we're going to do that, the whole approach of inheriting permissions from source VMAs becomes unnecessary complexity.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
- To create an X mapping of an enclave page that came from
EADD, you need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or
/dev/sgx/enclave?
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Enclave file -- that is, the file backing the vma from which the data is loaded.
It wasn't explicitly called out in Andy's proposal(s), but the idea is that the SGX driver would effectively inherit permissions from the source VMA (EADD needs a source for the initial value of the encave page).
I actually meant for it to *not* work like this. I don't want the source VMA to have to be
VM_EXEC. I think the LSM should just check permissions on ->vm_file.
-Cedric
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 7 lines · Cedric
So I think the same rationale applies to enclaves. Your original idea of MAXPERM is the policy set forth by system admin and shall *never* change at runtime. If an enclave is dynamically linked and needs to bring in code pages at runtime, the admin needs to enable it by setting, say
ENCLAVE__EXECMOD, in the sigstruct file. Then all EAUG'ed pages will receive RWX as MAXPERM. The process would thenmprotect()selective pages to be RX but which exact set of pages doesn't concern LSM usually.
Because passing RWX means the enclave "requires" EXECMOD even if it never actually does a RW->RX transition. It's not broken per se, but at the very least it's decidedly odd.
Dynamically detecting the EXECMOD case is not difficult and has the advantage of simplifying userspace loaders, e.g. all EAUG pages are tagged ALLOW_WRITE and the kernel takes care of the rest.
I *think* auditing/learning is also messed up with a MAXPERMS approach, as mprotect() would fail (due to MAXPERMS clearing MAY_{READ,WRITE,EXEC}) before it calls security_file_mprotect(). Hooking mprotect() is the obvious workaround, but then it's looking a lot like the new proposals.
In other words, the new proposals are rooted in the MAXPERMS concept, e.g. MAXPERM is effectively "I want EXECMOD", which gets distilled down to ALLOW_WRITE (or ALLOW_EXEC in Andy's proposal).
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonQuoted 4 lines · Sean Christopherson
- Requires enclave builder to mark enclave pages executable in the non-enclave VMAs, which may unnecessarily require EXECMOD on the source file, or even worse, EXECMEM, and potentially increases the attack surface since the file must be executable.
Enclave builder marks *non-enclave pages*? Not following.
Quoted 11 lines
W^X handling:
-mmap()to/dev/sgx/enclaveonly allowed withPROT_NONE, i.e. force userspace throughmprotect()to simplify the kernel implementation.
- Addvm_opsmprotect()ops hook (I'll refer to SGX's implementation asSGX.mprotect())
- Take explicitALLOW_WRITEatADD_REGION, a.k.a. EADD
-ADD_REGIONalso used to describe EAUG region (tentatively for SGX2).
- Track "can be written at some point in time (past or future)" asALLOW_WRITE(to avoid confusiong withMAY_WRITE). A priori knowledge of writability avoids having to track/coordinatePROT_WRITEacross VMAs and MMs.
Still not sure why you want to use vm_ops instead of file_operations.
The approach I've been proposing earlier in this email thread before these new proposals can be summarized from hook perspective as:
- Allow mmap() only before ECREATE and require it to be size of the ELRANGE (ECREATE ioctl would check this). This would be with PROT_NONE.
- Disallow mprotect() before EINIT. Requires a new callback to file_operations like mmap() has.
- After EINIT check for each mprotect() that it matches the permissions of underlying enclave pages. Disallow mmap() after EINIT.
/Jarkko
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Andy LutomirskiQuoted 5 lines · Andy Lutomirski
Quoted 2 lines
What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or
/dev/sgx/enclave?Enclave file -- that is, the file backing the vma from which the data is loaded.
Wonder why KVM gets away without having this given that enclaves are lot alike VMs.
Quoted 6 lines
It's provided by userspace based on whether it thinks the data in question is enclave code.
source->vm_fileis the file from which the code is being loaded. I'm assuming that the user code will only setexcute_intent==true if it actually wants to execute the code, so, if there's a denial, it will be fatal. The normal case will be that the request will be granted on the basis of EXECUTE.
AFAIK user spaces tells that already with the SECINFO flags. I don't get why we need a duplicate parameter.
/Jarkko
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 11 lines · Jarkko Sakkinen
Quoted 5 lines · Andy Lutomirski
Quoted 2 lines
What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or
/dev/sgx/enclave?Enclave file -- that is, the file backing the vma from which the data is loaded.
Wonder why KVM gets away without having this given that enclaves are lot alike VMs.
I would argue it's because access to /dev/kvm means you can execute whatever you code you want in a VM. I don't see how this is avoidable. On the other hand, it would be nice for SGX to not imply this same sort of "execute anything" right, especially since, unlike KVM, SGX is not a sandbox.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Jarkko SakkinenQuoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Andy Lutomirski
Quoted 2 lines
What is the "source file" i.e. the target of the check? Enclave file, sigstruct file, or
/dev/sgx/enclave?Enclave file -- that is, the file backing the vma from which the data is loaded.
Wonder why KVM gets away without having this given that enclaves are lot alike VMs.
From a memory management perspective, VMs are not at all like enclaves. An enclave is an extension of its host, i.e. runs in the same address. This isn't strictly necessary, e.g. an enclave could run in a sandbox process, but even then the enclave will be running with the kernel's standard page tables.
A VM is a essentially an opaque blob of data that gets loaded into memory. KVM builds a completely different set of page tables for the VM, the VM has it's own file system (or perhaps doesn't have a file system at all), etc... Ignoring Spectre and L1TF, the VM is contained to its own world.
There are a lot of ways for a userspace VMM to expose things beyond raw memory, but doing so requires the appropriate permissions.
And practically speaking, all traditional VMs will effectively need RWX memory, i.e. Qemu (or any other userspace VMM) would be required to have EXECMEM permissions, which would be a net negative for security.
Quoted 9 lines
Quoted 6 lines
It's provided by userspace based on whether it thinks the data in question is enclave code.
source->vm_fileis the file from which the code is being loaded. I'm assuming that the user code will only setexcute_intent==true if it actually wants to execute the code, so, if there's a denial, it will be fatal. The normal case will be that the request will be granted on the basis of EXECUTE.AFAIK user spaces tells that already with the SECINFO flags. I don't get why we need a duplicate parameter.
Please read through the RFC, I think it address a lot of your questions. Hopefully that will help us avoid some thrash.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 2 lines · Sean Christopherson
Please read through the RFC, I think it address a lot of your questions. Hopefully that will help us avoid some thrash.
I promise to read it through with detail albeit I just said that as a patch set it is broken :-) Internals still need documentation tho...
/Jarkko
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Stephen SmalleyQuoted 4 lines · Stephen Smalley
Does this occur for both setting initial permissions and runtime permissions or just runtime? Both userspace- and driver-initiated mmap/mprotect operations or just userspace-initiated ones? Does the driver use interfaces that call the mmap/mprotect hooks or lower level functions?
The driver never initiates mmap() or mprotect().
/Jarkko
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Andy LutomirskiQuoted 3 lines · Andy Lutomirski
- To create an X mapping of an enclave page that came from EADD, you
need EXECUTE on the source file. Optionally, we could also permit this if you have EXECMOD.
Source file? EADD ioctl takes memory buffer in right now.
Quoted 5 lines
And I have two design proposals. One is static and one is dynamic. To implement either one, we will probably need a new .may_mprotect vm operation, and that operation can call an LSM hook. Or we can give LSMs a way to detect that a given
vm_area_structis an enclave. As I see it, this is an implementation detail that is certainly solveable.
Why VM operation and not file operation?
Quoted 7 lines
EADD takes an
execute_intentflag. It calls a new hook:int security_enclave_load(struct vm_area_struct *source, bool execute_intent);This hook will fail if execute_intent==true and the caller has neither EXECUTE, EXECMOD, nor EXECMEM.
Quoted 7 lines
EAUG sets
execute_intent= false.EINIT takes a sigstruct pointer. SGX can (when initially upstreamed or later on once there's demand) call a new hook:
security_enclave_init(struct sigstruct *sigstruct, struct
vm_area_struct*source);
What is the source VMA in these callbacks? Why is @execute_intent needed anyway as a ioctl arugment and not deduced from SECINFO?
/Jarkko
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonOn Fri, May 24, 2019 at 01:03:33PM -0700, Sean Christopherson wrote:
Good morning, I hope the weekend is going well for everyone. Skunky holiday weather out here in West-Central Minnesota.
Quoted 13 lines · Andy Lutomirski
Quoted 7 lines
If we go this route, the only substantial change to the existing driver that's needed for an initial upstream merge is the maxperm mechanism and whatever hopefully minimal API changes are needed to allow users to conveniently set up the mappings. And we don't need to worry about how to hack around
mprotect()calling into the LSM, because the LSM will actually be aware of SGX and can just do the right thing.This doesn't address restricting which processes can run which enclaves, it only allows restricting the build flow. Or are you suggesting this be done in addition to whitelisting sigstructs?
Quoted 7 lines
What's the value prop beyond whitelisting sigstructs? Realistically, I doubt LSMs/users will want to take the performance hit of scanning the source bytes every time an enclave is loaded.
We could add seomthing like
security_enclave_mprotect()in lieu of abusingsecurity_file_mprotect(), but passing the full source bytes seems a bit much.
It would seem that we hold the moniker of responsibility for this conversation, since without our provocation regarding cryptographic verification of enclave source, there would be a driver headed upstream whose only constraint against W^X sourced executable code, running with full confidentiality and integrity protections, would be a character device with o666 permissions. Given that, a couple of reflections to facilitate further conversation, if nothing else for the benefit of Jonathan Corbet and his bystanders... :-)
As the conversations to date have indicated, imposing LSM controls on enclave executable code is a bit problematic, in no small part since it is the theological equivalent of driving a square peg into a round hole. SGX, as a technology, was designed around the concept of cryptographic verification of code provenance and origin.
The decision to take that off the table, for reasons of political idealogy only, means that mainstream Linux will not be a platform that can achieve the full hardware security capabilities and protections of SGX, nor will mainstream Linux be able to enjoy full protections from the technology itself.
We will be dealing with that, from a driver and runtime perspective, but that is a conversation for another day.
The issue of SGX2 and Enclave Dynamic Memory Management (EDMM) has come up and to date there doesn't appear to have been a serious conversation regarding whether or not all of the LSM machinations in the world will make any difference when this technology goes mainline. The agenda driving mainlining of the driver is to support Graphene for cloud based solutions and without EDMM, dynamic code loading support is decidedly more problematic.
Dynamic enclave code loading isn't problematic from a security perspective when the code is being loaded from the platform itself, since presumably, the encompassing conversation will result in LSM controls being applied to the necessary code paths. However, with the ability to exploit SGX2 instructions, an enclave with adverserial intent could simply setup a mutually attested security context and pull whatever executable code it wants from the INTERNET at large, using an encrypted and integrity protected communications channel.
That has at least been our interpretation and experience with the ENCLU[EMODPE] and ENCLU[EACCEPTCOPY] instructions and the out-of-tree driver. Given the use of an encrypted channel, and the fact that these instructions are ring 3 enclave mode only, it would seem that all of the LSM controls in the world won't have visibility or control over code that is being loaded and executed using such a mechanism.
We could have arguably missed something that the new driver will do to address this issue. To date the only discussion seems to have been about controls over ENCLS[EAUG], which are arguably a bit blunt for this purpose.
In the land of SGX, if one is intellectually honest from an engineering perspective, the only solid security contract one has to work with is the notion of cryptographic identity. Hence our concern and patches that implemented an absolutely minimal footprint ring-0 control infrastructure over the contents of an enclave's SIGSTRUCT. Which is where we have arguably circled back to after 3-4 months and one kernel release cycle.
Wrapping an LSM hook around our policy mechanism would seem to achieve, from a security perspective, about the same level of security effect that more major and invasive modifications would achieve, given Cedric's proposal to inherit page permissions from the source, which is what our runtime already does.
As always, apologies for excessive verbosity beyond LKML sensibilities.
Best wishes for a pleasant remainder of the spring weekend to everyone.
Dr. Greg
As always,
Dr. G.W. Wettstein, Ph.D. Enjellic Systems Development, LLC. 4206 N. 19th Ave. Specializing in information infra-structure Fargo, ND 58102 development.
PH: 701-281-1686
FAX: 701-281-3949 EMAIL: [email protected]
------------------------------------------------------------------------------
"Heaven goes by favor. If it went by merit, you would stay out and your
dog would go in."
-- Mark Twain
Andy Lutomirski
maintainerRe: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Xing, CedricQuoted 66 lines · Cedric
Hi Andy,
Quoted 63 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Thursday, May 23, 2019 6:18 PMQuoted 17 lines · Sean Christopherson
Quoted 15 lines · Andy Lutomirski
Quoted 13 lines · Sean Christopherson
Quoted 10 lines · Jarkko Sakkinen
Quoted 7 lines · Sean Christopherson
But actually, there's no need to disallow
mmap()after ECREATE since the LSM checks also apply tommap(), e.g.FILE__EXECUTEwould be needed tommap()any enclave pagesPROT_EXEC. I guess my past self thoughtmmap()bypassed LSM checks? The real problem is that mmap()'ng an existing enclave would requireFILE__WRITEandFILE__EXECUTE, which puts us back at square one.I'm lost with the constraints we want to set.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
As is today, SELinux policies would require enclave loaders to
haveFILE__WRITEandFILE__EXECUTEpermissions on/dev/sgx/enclave. Presumably other LSMs have similar
requirements. Requiring all processes to have
FILE__{WRITE,EXECUTE} permissions means the permissions don't add much value, e.g. they can't be used to distinguish between an enclave that is being loaded from an unmodified file and an enclave that is beinggenerated on the fly, e.g. Graphene.
Quoted 18 lines
Quoted 12 lines
Quoted 5 lines
Looking back at Andy's mail, he was talking about requiring
FILE__EXECUTEto run an enclave, so perhaps it's onlyFILE__WRITEthat we're trying to special case.I thought about this some more, and I have a new proposal that helps address the ELRANGE alignment issue and the permission issue at the cost of some extra verbosity. Maybe you all can poke holes in it :) The basic idea is to make everything more explicit from a user's perspective. Here's how it works:
Quoted 5 lines
Opening
/dev/sgx/enclavegives anenclave_fdthat, by design, doesn't give EXECUTE or WRITE.mmap()on theenclave_fdonly works if you passPROT_NONEand gives the correct alignment. The resulting VMA cannot be mprotected or mremapped. It can't be mmapped at all untilQuoted 7 lines
I assume you're thinking of clearing all VM_MAY* flags in
sgx_mmap()?Quoted 1 line
after ECREATE because the alignment isn't known before that.
I don't follow. The alignment is known because userspace knows the size of its enclave. The initial unknown is the address, but that becomes known once the initial
mmap()completes.[...]
I think I made the mistake of getting too carried away with implementation details rather than just getting to the point. And I misremembered the ECREATE flow -- oops. Let me try again. First, here are some problems with some earlier proposals (mine, yours Cedric's):
Quoted 6 lines
Quoted 2 lines
- Having the EADD operation always work but have different effects depending on the
source memory permissions is, at the very least, confusing.Inheriting permissions from source pages IMHO is the easiest way to validate the EPC permissions without any changes to LSM. And the argument about its security is also easy to make.
I understand that it may take some effort to document it properly but otherwise don't see any practical issues with it.
My objection is to the fact that it's implicit. I have no problem with some operation succeeding if the source address is X and failing if it's !X, but I don't think it's fantastic to have it succeed in either case but do different things.
For what it's worth, while this is a bit of a theoretical issue for X, but I think it's a real problem with W. To avoid accidentally mapping an enclave page X and then later mapping the same page W (potentially in a different VMA), I think it will be a lot simpler if the driver can track which pages are allowed to ever be W. We definitely *don't* want an interface in which the eventual writability of a page is inferred from the W permission on the source address, since we do *not* want to force anyone to map their enclave file PROT_WRITE or even to open it O_RDWR.
With the explicit MAXPERM passed in, this issue goes away. You can specify W if you want W.
Quoted 25 lines
Quoted 25 lines
- If we want to encourage user programs to be well-behaved, we want to make it easy to
map the RX parts of an enclave RX, the RW parts RW, the RO parts R, etc. But this interacts poorly with thesgx_mmap()alignment magic, as you've pointed out.- We don't want to couple LSMs with SGX too tightly.
So here's how a nice interface might work:
int
enclave_fd=open("/dev/sgx/enclave", O_RDWR);/*
enclave_fdpoints to a totally blank enclave. Before ECREATE, we need to decide on an address. */void *addr =
mmap(NULL, size, PROT_NONE, MAP_SHARED, enclave_fd, 0);/* we have an address! */
ioctl(enclave_fd, ECREATE, ...);/* now add some data to the enclave. We want the RWX addition to fail
immediately unless we have the relevant LSM pemission. Similarly, we
want the RX addition to fail immediately unless the source VMA is appropriate. */
Quoted 6 lines
Quoted 4 lines
ioctl(enclave_fd, EADD, rx_source_1, MAXPERM=RX, ...); [the ... includes SECINFO, which the kernel doesn't really care about]ioctl(enclave_fd, EADD, ro_source_1, MAXPERM=RX ...);ioctl(enclave_fd, EADD, rw_source_1, MAXPERM=RW ...);ioctl(enclave_fd, EADD, rwx_source_1, MAXPERM=RWX ...);If MAXPERM is taken from ioctl parameters, the real question here is how to validate MAXPERM. Guess we shouldn't allow arbitrary MAXPERM to be specified by user code, and the only logical source I can think of is from the source pages (or from the enclave source file, but memory mapping is preferred because it offers more flexibility).
That's exactly what I intended here. If you specify MAXPERM=RX, then the kernel can validate that the source address is executable.
Quoted 12 lines
Quoted 12 lines
ioctl(enclave_fd, EINIT, ...); /* presumably passsigstruct_fdhere, too. *//* at this point, all is well except that the enclave is mapped
PROT_NONE. There are a couple ways I can imagine to fix this. */We could use mmap:
mmap(baseaddr+offset, len, PROT_READ, MAP_SHARED | MAP_FIXED, enclave_fd, 0); /* only succeeds if MAXPERM & R == R */
Quoted 6 lines
Quoted 2 lines
But this has some annoying implications with regard to
sgx_get_unmapped_area(). We could use an ioctl:There's an easy fix. Just let
sgx_get_unmapped_area()do the natural alignment only ifMAP_FIXEDis *not* set, otherwise, honor both address and len.But
mmap()is subject to LSM check (probably against/dev/sgx/enclave?). How to do mmap(RX) ifFILE__EXECUTEis *not* granted for/dev/sgx/enclave, even if MAXPERM=RX?
I think we just let /dev/sgx/enclave be FILE__EXECUTE. We don't *have* to make it so that FILE__WRITE and FILE__EXECUTE on /dev/sgx/enclave means you can create RWX enclave mappings.
Quoted 4 lines
Quoted 4 lines
ioctl(enclave_fd, SGX_IOC_MPROTECT, offset, len, PROT_READ);
Quoted 8 lines
Quoted 6 lines
which has the potentially nice property that we can completely bypass the LSM hooks, because the LSM has *already* vetted everything when the EADD calls were allowed. Or we could maybe even just use
mprotect()itself:
mprotect(baseaddr + offset, len, PROT_READ);How to bypass LSM hooks in this
mprotect()?
Hmm. I guess we either use FILE__WRITE and FILE__EXECUTE or we use ioctl().
Quoted 18 lines
Quoted 8 lines
Or, for the really evil option, we could use a bit of magic in .fault and do nothing here. Instead we'd make the initial mapping PROT_READ|PROT_WRITE|PROT_EXEC and have .fault actually instantiate the PTEs with the intersection of the VMA permissions and MAXPERM. I don't think I like this alternative, since it feels more magical than needed and it will be harder to debug. I like the fact that
/proc/self/mapsshows the actual permissions in all the other variants.Agreed.
Quoted 7 lines
All of the rest of the crud in my earlier email was just implementation details. The point I was trying to make was that I think it's possible to implement this without making too much of a mess internally. I think I favor the
mprotect()approach since it makes the behavior fairly obvious.
Quoted 9 lines
Quoted 6 lines
I don't think any of this needs to change for SGX2. We'd have an
ioctl()that does EAUG and specifies MAXPERM. Trying tomprotect()a page that hasn't been added yet with any permission other thanPROT_NONEwould fail. I suppose we might end up needing a way to let the EAUG operation *change* MAXPERM, and this operation would have to do some more LSM checks and walk all the existing mappings to make sure they're consistent with the new MAXPERM.EAUG ioctl could be a solution, but isn't optimal at least. What we've done is #PF based. Specifically, an SGX2 enclave will have its heap mapped as RW, but without any pages populated before EINIT. Then when the enclave needs a new page in its heap, it issues EACCEPT, which will cause a #PF and the driver will respond by EAUG a new EPC page. And then the enclave will be resumed and the faulted EACCEPT will be retried (and succeed).
If the driver works like that, then whatever call sets up this lazily allocated heap could do the MAXPERM part.
That being said, is the performance advantage from putting this logic in the kernel instead of in the untrusted part of the SDK really worthwhile?
Quoted 11 lines
Quoted 6 lines
As an aside, I wonder if Linus et all would be okay with a new
MAP_FULLY_ALIGNEDmmap()flag that allocated memory aligned to the requested size. Then we could get rid of yet another bit of magic.--Andy
I've also got a chance to think more about it lately.
When we talk about EPC page permissions with SGX2 in mind, I think we should distinguish between initial permissions and runtime permissions. Initial permissions refer to the page permissions set at EADD. They are technically set by "untrusted" code so should go by policies similar to those applicable to regular shared objects. Runtime permissions refer to the permissions granted by EMODPE, EAUG and EACCEPTCOPY. They are resulted from inherent behavior of the enclave, which in theory is determined by the enclave's measurements (MRENCLAVE and/or MRSIGNER).
Quoted 8 lines
And we have 2 distinct files to work with - the enclave file and
/dev/sgx/enclave. And I consider the enclave file a logical source for initial permissions while/dev/sgx/enclaveis a means to control runtime permissions. Then we can have a simpler approach like the pseudo code below./**
* Summary:
* - The enclave file resembles a shared object that contains RO/RX/RW segments
* - FILE__* are assigned to/dev/sgx/enclave, to determine acceptable permissions to mmap()/mprotect(), valid combinations are
* +FILE__READ- Allow SGX1 enclaves only
* + FILE__READ|FILE__WRITE - Allow SGX2 enclaves to expand data segments (e.g. heaps, stacks, etc.)
I think this is a non-starter :( FILE__WRITE also means that you can write to the file, and the admin / policy author will almost never want to allow that.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 11 lines · Andy Lutomirski
Quoted 6 lines · Cedric
/**
* Summary:
* - The enclave file resembles a shared object that contains RO/RX/RW segments
* - FILE__* are assigned to/dev/sgx/enclave, to determine acceptable permissions to mmap()/mprotect(), valid combinations are
* +FILE__READ- Allow SGX1 enclaves only
* + FILE__READ|FILE__WRITE - Allow SGX2 enclaves to expand data segments (e.g. heaps, stacks, etc.)I think this is a non-starter :(
FILE__WRITEalso means that you can write to the file, and the admin / policy author will almost never want to allow that.
Why would FILE__WRITE on /dev/sgx/enclave be a problem? An actual write to /dev/sgx/enclave would yield -EINVAL, no?
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 12 lines · Sean Christopherson
Quoted 10 lines · Andy Lutomirski
Quoted 7 lines · Cedric
/**
* Summary:
* - The enclave file resembles a shared object that contains RO/RX/RW segments
* - FILE__* are assigned to/dev/sgx/enclave, to determine acceptable permissions to mmap()/mprotect(), valid combinations are
* +FILE__READ- Allow SGX1 enclaves only
* + FILE__READ|FILE__WRITE - Allow SGX2 enclaves to expand data segments (e.g. heaps, stacks, etc.)
Quoted 6 lines
Quoted 3 lines
I think this is a non-starter :(
FILE__WRITEalso means that you can write to the file, and the admin / policy author will almost never want to allow that.Why would
FILE__WRITEon/dev/sgx/enclavebe a problem? An actual write to/dev/sgx/enclavewould yield-EINVAL, no?
Bah, read it wrong — FILE__WRITE on the enclave file on disk is no good.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Andy LutomirskiQuoted 60 lines · Andy Lutomirski
Quoted 20 lines · Sean Christopherson
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Sean Christopherson
But actually, there's no need to disallow
mmap()after ECREATE since the LSM checks also apply tommap(), e.g.FILE__EXECUTEwould be needed tommap()any enclave pagesPROT_EXEC. I guess my past self thoughtmmap()bypassed LSM checks? The real problem is that mmap()'ng an existing enclave would requireFILE__WRITEandFILE__EXECUTE, which puts us back at square one.I'm lost with the constraints we want to set.
As is today, SELinux policies would require enclave loaders to have
FILE__WRITEandFILE__EXECUTEpermissions on/dev/sgx/enclave. Presumably other LSMs have similar requirements. Requiring all processes to have FILE__{WRITE,EXECUTE} permissions means the permissions don't add much value, e.g. they can't be used to distinguish between an enclave that is being loaded from an unmodified file and an enclave that is being generated on the fly, e.g. Graphene.Quoted 4 lines
Looking back at Andy's mail, he was talking about requiring
FILE__EXECUTEto run an enclave, so perhaps it's onlyFILE__WRITEthat we're trying to special case.I thought about this some more, and I have a new proposal that helps address the ELRANGE alignment issue and the permission issue at the cost of some extra verbosity. Maybe you all can poke holes in it :) The basic idea is to make everything more explicit from a user's perspective. Here's how it works:
Opening
/dev/sgx/enclavegives anenclave_fdthat, by design, doesn't give EXECUTE or WRITE.mmap()on theenclave_fdonly works if you passPROT_NONEand gives the correct alignment. The resulting VMA cannot be mprotected or mremapped. It can't be mmapped at all until after ECREATE because the alignment isn't known before that.Associated with the enclave are a bunch (up to 7) "enclave segment inodes". These are
anon_inodesthat are created automagically. An enclave segment is a group of pages, not necessary contiguous, with an upper bound on the memory permissions. Each enclave page belongs to a segment. When you do EADD, you tell the driver what segment you're adding to. [0] This means that EADD gets an extra argument that is a permission mask for the page -- in addition to the initial SECINFO, you also pass to EADD something to the effect of "I promise never to map this with permissions greater than RX".Then we just need some way to mmap a region from an enclave segment. This could be done by having a way to get an fd for an enclave segment or it could be done by having a new ioctl
SGX_IOC_MAP_SEGMENT. User code would use this operation to replace, MAP_FIXED-style, ranges from the bigPROT_NONEmapping with the relevant pages from the enclave segment. The resulting vma would only haveVM_MAYWRITEif the segment is W, only haveVM_MAYEXECif the segment is X, and only haveVM_MAYREADif the segment is R. Depending on implementation details, the VMAs might need to restrictmremap()to avoid mapping pages that aren't part of the segment in question.
Quoted 8 lines
It's plausible that this whole thing works without the magic segment inodes under the hood, but figuring that out would need a careful look at how all the core mm bits and LSM bits work together.
To get all the LSM stuff to work, SELinux will need some way to automatically assign an appropriate label to the segment inodes. I assume that such a mechanism already exists and gets used for things like sockets, but I haven't actually confirmed this.
I don't follow that. socket inodes are not anon inodes, and anon inodes have no per-instance data by definition, and typically you're only dealing with a single anon inode for all files, and hence they were long ago marked S_PRIVATE and exempted from all LSM checking except for EXECMEM on mmap/mprotect PROT_EXEC. We have no way to perform useful security checking on them currently. socket inodes we can label from their creating process but even that's not going to support multiple labels for different sockets created by the same process unless the process explicitly used setsockcreatecon(3) aka /proc/self/attr/sockcreate
Quoted 5 lines
[0] There needs to be some vaguely intelligent semantics if you EADD the *same* address more than once. A simple solution would be to disallow it if the segments don't match.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Andy LutomirskiQuoted 33 lines · Andy Lutomirski
Quoted 20 lines · Sean Christopherson
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Sean Christopherson
But actually, there's no need to disallow
mmap()after ECREATE since the LSM checks also apply tommap(), e.g.FILE__EXECUTEwould be needed tommap()any enclave pagesPROT_EXEC. I guess my past self thoughtmmap()bypassed LSM checks? The real problem is that mmap()'ng an existing enclave would requireFILE__WRITEandFILE__EXECUTE, which puts us back at square one.I'm lost with the constraints we want to set.
As is today, SELinux policies would require enclave loaders to have
FILE__WRITEandFILE__EXECUTEpermissions on/dev/sgx/enclave. Presumably other LSMs have similar requirements. Requiring all processes to have FILE__{WRITE,EXECUTE} permissions means the permissions don't add much value, e.g. they can't be used to distinguish between an enclave that is being loaded from an unmodified file and an enclave that is being generated on the fly, e.g. Graphene.Quoted 4 lines
Looking back at Andy's mail, he was talking about requiring
FILE__EXECUTEto run an enclave, so perhaps it's onlyFILE__WRITEthat we're trying to special case.I thought about this some more, and I have a new proposal that helps address the ELRANGE alignment issue and the permission issue at the cost of some extra verbosity. Maybe you all can poke holes in it :) The basic idea is to make everything more explicit from a user's perspective. Here's how it works:
Quoted 5 lines
Opening
/dev/sgx/enclavegives anenclave_fdthat, by design, doesn't give EXECUTE or WRITE.mmap()on theenclave_fdonly works if you passPROT_NONEand gives the correct alignment. The resulting VMA cannot be mprotected or mremapped. It can't be mmapped at all until after ECREATE because the alignment isn't known before that.
How to deny mprotect()? struct file_operations does not have callback for that (AFAIK).
Quoted 25 lines
Associated with the enclave are a bunch (up to 7) "enclave segment inodes". These are
anon_inodesthat are created automagically. An enclave segment is a group of pages, not necessary contiguous, with an upper bound on the memory permissions. Each enclave page belongs to a segment. When you do EADD, you tell the driver what segment you're adding to. [0] This means that EADD gets an extra argument that is a permission mask for the page -- in addition to the initial SECINFO, you also pass to EADD something to the effect of "I promise never to map this with permissions greater than RX".Then we just need some way to mmap a region from an enclave segment. This could be done by having a way to get an fd for an enclave segment or it could be done by having a new ioctl
SGX_IOC_MAP_SEGMENT. User code would use this operation to replace, MAP_FIXED-style, ranges from the bigPROT_NONEmapping with the relevant pages from the enclave segment. The resulting vma would only haveVM_MAYWRITEif the segment is W, only haveVM_MAYEXECif the segment is X, and only haveVM_MAYREADif the segment is R. Depending on implementation details, the VMAs might need to restrictmremap()to avoid mapping pages that aren't part of the segment in question.It's plausible that this whole thing works without the magic segment inodes under the hood, but figuring that out would need a careful look at how all the core mm bits and LSM bits work together.
Quoted 8 lines
To get all the LSM stuff to work, SELinux will need some way to automatically assign an appropriate label to the segment inodes. I assume that such a mechanism already exists and gets used for things like sockets, but I haven't actually confirmed this.
[0] There needs to be some vaguely intelligent semantics if you EADD the *same* address more than once. A simple solution would be to disallow it if the segments don't match.
What if instead simply:
- Require to do PROT_NONE mmap() for the ELRANGE before ECREATE.
- Disallow mprotect() up until EINIT.
- Given that we have a callback for mprotect() check that permissions match EADD'd permissions.
/Jarkko
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonQuoted 19 lines · Sean Christopherson
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Sean Christopherson
But actually, there's no need to disallow
mmap()after ECREATE since the LSM checks also apply tommap(), e.g.FILE__EXECUTEwould be needed tommap()any enclave pagesPROT_EXEC. I guess my past self thoughtmmap()bypassed LSM checks? The real problem is that mmap()'ng an existing enclave would requireFILE__WRITEandFILE__EXECUTE, which puts us back at square one.I'm lost with the constraints we want to set.
As is today, SELinux policies would require enclave loaders to have
FILE__WRITEandFILE__EXECUTEpermissions on/dev/sgx/enclave. Presumably other LSMs have similar requirements. Requiring all processes to have FILE__{WRITE,EXECUTE} permissions means the permissions don't add much value, e.g. they can't be used to distinguish between an enclave that is being loaded from an unmodified file and an enclave that is being generated on the fly, e.g. Graphene.
Quoted 3 lines
Looking back at Andy's mail, he was talking about requiring
FILE__EXECUTEto run an enclave, so perhaps it's onlyFILE__WRITEthat we're trying to special case.
Argh, as I was working through Andy's latest proposal I realized that I was subconciously making FILE__READ imply FILE__EXECUTE.
The idea behind inheriting permissions from the source VMA is to exempt "standard" enclaves from needing FILE__WRITE. But if we don't add an exemption for FILE__EXECUTE as well, then all enclaves need FILE__EXECUTE, which means FILE__EXECUTE can't be used to identify the case where userspace is mapping an inherited PROT_WRITE page as PROT_EXEC. And if the SGX magic exempts FILE__EXECUTE, then FILE__READ implies FILE__EXECUTE.
Yuck.
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonQuoted 5 lines · Sean Christopherson
1. Do nothing. Userspace would essentially be required to
mmap()the enclave after EINIT, which is ugly but not breaking since userspace couldmmap()the enclave with a placeholder VMA prior to building the enclave, and then a series ofmmap()to establish its "real" mapping.
What it'd break to return error if mmap() is done before EINIT?
Quoted 2 lines
2. Propagate the permissions from EADD to the VMAs of the current mm if the entire EADD range is mapped and the mapping is
PROT_NONE.
Right now you can do multiple mmap's. If the mmap's must be done after EINIT, the driver could check that permissions match the permissions in that range.
This leaves open how to deal with mprotect() but if the process does not have FILE__WRITE I guess you cannot do much.
Quoted 3 lines
3. Propagate the permissions from EADD to the VMAs of all mm structs that have mapped some piece of the enclave, following the matching rules from #2.
For me it looks that allowing mmap's only after EINIT would result the least confusing implemntation.
/Jarkko
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 19 lines · Jarkko Sakkinen
Quoted 5 lines · Sean Christopherson
1. Do nothing. Userspace would essentially be required to
mmap()the enclave after EINIT, which is ugly but not breaking since userspace couldmmap()the enclave with a placeholder VMA prior to building the enclave, and then a series ofmmap()to establish its "real" mapping.What it'd break to return error if
mmap()is done before EINIT?Quoted 2 lines
2. Propagate the permissions from EADD to the VMAs of the current mm if the entire EADD range is mapped and the mapping is
PROT_NONE.Right now you can do multiple mmap's. If the mmap's must be done after EINIT, the driver could check that permissions match the permissions in that range.
This leaves open how to deal with
mprotect()but if the process does not haveFILE__WRITEI guess you cannot do much.
Quoted 6 lines
Quoted 3 lines
3. Propagate the permissions from EADD to the VMAs of all mm structs that have mapped some piece of the enclave, following the matching rules from #2.
For me it looks that allowing mmap's only after EINIT would result the least confusing implemntation.
Obvious problem is of course the requirement of fixed mapping, which is of course nasty.
/Jarkko
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Andy LutomirskiQuoted 3 lines · Andy Lutomirski
As far as I know from this whole discussion, we still haven't come up with any credible way to avoid tracking, per enclave page, whether that page came from unmodified
PROT_EXECmemory.
So is this in the context that the enclave is read from another VMA and not through a file descriptor? Is that locked in?
/Jarkko
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
Quoted 7 lines · Jarkko Sakkinen
Quoted 3 lines · Andy Lutomirski
As far as I know from this whole discussion, we still haven't come up with any credible way to avoid tracking, per enclave page, whether that page came from unmodified
PROT_EXECmemory.So is this in the context that the enclave is read from another VMA and not through a file descriptor? Is that locked in?
No need to answer. Got in page from Sean's response.
/Jarkko
Re: SGX vs LSM (Re: [PATCH v20 00/28] Intel SGX1 support)
In reply to Sean ChristophersonQuoted 4 lines · Sean Christopherson
The SGX
ioctl()would need to takemmap_semfor write, but we can mitigate that issue by changing theioctl()to take a range of memory instead of a single page. That'd also provide "EADD batching" that folks have requested.
This should be easy enough to add as the EADD operations are already batched internally to a worker thread.
/Jarkko
Quoted 22 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 11 lines · Jarkko Sakkinen
Quoted 8 lines · Jarkko Sakkinen
I did study through SDK's file format and realized that it does not does make sense after all to embed one.
To implement it properly you would probably need a new syscall (lets say
sgx_load_enclave) and also that enclaves are not just executables binaries. It is hard to find a generic format for them as applications range from simply protecting part of an application to running a containter inside enclave.Quoted 7 lines
Quoted 2 lines
I'm still puzzling what kind of changes you were discussing considering
SGX_IOC_ENCLAVE_ADD_PAGE.I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?)
Quoted 7 lines
Nit: SECINFO is just the struct passed to EADD, I think what you're really asking is "can the EPCM permissions be changed after the fact".
And the answer is, yes.
On SGX2 hardware, the enclave can extend the EPCM permissions at runtime via ENCLU[EMODPE], e.g. to make a page writable.
Small correction: it is EMODPR.
Anyway, it is good to mention that these would require EACCEPT from the enclave side. In order to take advantage of this is in a malicous enclave, one would require SELinux/IMA/whatnot policy to have permitted it in the first place.
Thus, it cannot be said that it breaks the security policy if this would happen because policy has allowed to use the particular enclave.
Quoted 8 lines
Hardware also doesn't prevent doing EADD to the same virtual address multiple times, e.g. an enclave could EADD a RX page, and then EADD a RW page at the same virtual address with different data. The second EADD will affect MRENCLAVE, but so long as it's accounted for by the enclave's signer, it's "legal".
SGX_IOC_ENCLAVE_ADD_PAGE*does* prevent adding the "same" page to an enclave multiple times, so effectively this scenario is blocked by the current implementation, but it's more of a side effect (of a sane implementation) as opposed to deliberately preventing shenanigans.
If the security policy can define who can create legit SIGSTRUCT files, this should not be a problem. Neither should be how EEXTEND is used.
This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?
If we can sort this question out, then SIGSTRUCT-centered way to control enclave would actually be robust.
/Jarkko
Quoted 4 lines · Jarkko Sakkinen
This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?
Also wondering if a path would be better than plain fd for defining a reasonable policy i.e. have sigstruct_path as part of the ioctl parameters and not sigstruct_fd.
/Jarkko
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Jarkko Sakkinen
This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?
The policy will presumably check the label on the file that the fd points to.
Quoted 4 lines
Also wondering if a path would be better than plain fd for defining a reasonable policy i.e. have
sigstruct_pathas part of the ioctl parameters and notsigstruct_fd.
It would save two syscalls at the cost of a decent amount of complexity.
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Jarkko Sakkinen
This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?
The policy will presumably check the label on the file that the fd points to.
Right (checked SELinux documentation).
Got one idea from this. Right now creation and initialization does not require any VMAs to be created (since v20). Requiring to map a VMA for copying the data would bring in my opinion a glitch to this model that we have done effort to build up.
What if we similarly change EADD ioctl in a way that it'd take an fd and an offset? This way we can enforce policy to the source where the enclave data is loaded from. On the other hand, loading SIGSTRUCT from fd enforces a legit structure for the enclave.
This would still allow to construct enclaves in VMA independent way.
Quoted 6 lines
Quoted 4 lines
Also wondering if a path would be better than plain fd for defining a reasonable policy i.e. have
sigstruct_pathas part of the ioctl parameters and notsigstruct_fd.It would save two syscalls at the cost of a decent amount of complexity.
And also using fd gives robustness because it allows SIGSTRUCT pulled from a file containing it (among other things).
/Jarkko
Quoted 20 lines · Jarkko Sakkinen
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Jarkko Sakkinen
This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?
The policy will presumably check the label on the file that the fd points to.
Right (checked SELinux documentation).
Got one idea from this. Right now creation and initialization does not require any VMAs to be created (since v20). Requiring to map a VMA for copying the data would bring in my opinion a glitch to this model that we have done effort to build up.
Quoted 6 lines
What if we similarly change EADD ioctl in a way that it'd take an fd and an offset? This way we can enforce policy to the source where the enclave data is loaded from. On the other hand, loading SIGSTRUCT from fd enforces a legit structure for the enclave.
This would still allow to construct enclaves in VMA independent way.
The API would turn into this:
/**
* struct sgx_enclave_add_page - parameter structure for the
* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
* @fd: file containing the page data
* @offset: offset in the file containing the page data
* @secinfo: address for the SECINFO data
* @mrmask: bitmask for the measured 256 byte chunks
*/
struct sgx_enclave_add_page {
__u64 fd;
__u64 offset;
__u64 secinfo;
__u16 mrmask;
} __attribute__((__packed__));/**
* struct sgx_enclave_init - parameter structure for the
* %SGX_IOC_ENCLAVE_INIT ioctl
* @fd: file containing the sigstruct
* @offset: offset in the file containing the sigstruct
*/
struct sgx_enclave_init {
__u64 fd;
__u64 offset;
};/Jarkko
Quoted 52 lines · Jarkko Sakkinen
Quoted 20 lines · Jarkko Sakkinen
Quoted 11 lines · Andy Lutomirski
Quoted 9 lines · Jarkko Sakkinen
Quoted 6 lines · Jarkko Sakkinen
This brings me to an open question in Andy's model: lets say that we change the source for SIGSTRUCT from memory address to fd. How can the policy prevent the use not creating a file containing a SIGSTRUCT and passing fd of that to the EINIT ioctl?
The policy will presumably check the label on the file that the fd points to.
Right (checked SELinux documentation).
Got one idea from this. Right now creation and initialization does not require any VMAs to be created (since v20). Requiring to map a VMA for copying the data would bring in my opinion a glitch to this model that we have done effort to build up.
Quoted 6 lines
What if we similarly change EADD ioctl in a way that it'd take an fd and an offset? This way we can enforce policy to the source where the enclave data is loaded from. On the other hand, loading SIGSTRUCT from fd enforces a legit structure for the enclave.
This would still allow to construct enclaves in VMA independent way.
The API would turn into this:
/**
* structsgx_enclave_add_page- parameter structure for the
* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
* @fd: file containing the page data
* @offset: offset in the file containing the page data
* @secinfo: address for the SECINFO data
* @mrmask: bitmask for the measured 256 byte chunks
*/struct sgx_enclave_add_page { __u64 fd; __u64 offset; __u64 secinfo; __u16 mrmask; } __attribute__((__packed__));/**
* structsgx_enclave_init- parameter structure for the
* %SGX_IOC_ENCLAVE_INIT ioctl
* @fd: file containing the sigstruct
* @offset: offset in the file containing the sigstruct
*/
Quoted 4 lines
struct sgx_enclave_init {
__u64 fd;
__u64 offset;
};The change to EADD/EINIT ioctl's would be simply fget/kernel_read/fput sequence replacing copy_from_user().
/Jarkko
Quoted 10 lines · Jarkko Sakkinen
Quoted 9 lines · Sean Christopherson
Quoted 4 lines · Andy Lutomirski
I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?)
Nit: SECINFO is just the struct passed to EADD, I think what you're really asking is "can the EPCM permissions be changed after the fact".
Quoted 6 lines
Quoted 4 lines
And the answer is, yes.
On SGX2 hardware, the enclave can extend the EPCM permissions at runtime via ENCLU[EMODPE], e.g. to make a page writable.
Small correction: it is EMODPR.
No, I'm referring to EMODPE, note the ENCLU classification.
Quoted 4 lines
Anyway, it is good to mention that these would require EACCEPT from the enclave side. In order to take advantage of this is in a malicous enclave, one would require SELinux/IMA/whatnot policy to have permitted it in the first place.
EMODPE doesn't require EACCEPT or any equivalent from the kernel. As you alluded to, the page tables would still need to allow PROT_EXEC. I was simply trying to answer Andy's question regarding SECINFO.
Quoted 2 lines
Thus, it cannot be said that it breaks the security policy if this would happen because policy has allowed to use the particular enclave.
Quoted 20 lines · Sean Christopherson
Quoted 10 lines · Jarkko Sakkinen
Quoted 9 lines · Sean Christopherson
Quoted 4 lines · Andy Lutomirski
I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?)
Nit: SECINFO is just the struct passed to EADD, I think what you're really asking is "can the EPCM permissions be changed after the fact".
Quoted 6 lines
Quoted 4 lines
And the answer is, yes.
On SGX2 hardware, the enclave can extend the EPCM permissions at runtime via ENCLU[EMODPE], e.g. to make a page writable.
Small correction: it is EMODPR.
No, I'm referring to EMODPE, note the ENCLU classification.
Quoted 8 lines
Quoted 4 lines
Anyway, it is good to mention that these would require EACCEPT from the enclave side. In order to take advantage of this is in a malicous enclave, one would require SELinux/IMA/whatnot policy to have permitted it in the first place.
EMODPE doesn't require EACCEPT or any equivalent from the kernel. As you alluded to, the page tables would still need to allow
PROT_EXEC. I was simply trying to answer Andy's question regarding SECINFO.
Ah, have to admit that I had totally forgot EMODPE :-) Have not had to deal with that opcode that much.
/Jarkko
Quoted 16 lines · Andy Lutomirski
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines · Jarkko Sakkinen
I did study through SDK's file format and realized that it does not does make sense after all to embed one.
Quoted 8 lines
Quoted 5 lines
To implement it properly you would probably need a new syscall (lets say
sgx_load_enclave) and also that enclaves are not just executables binaries. It is hard to find a generic format for them as applications range from simply protecting part of an application to running a containter inside enclave.I'm still puzzling what kind of changes you were discussing considering
SGX_IOC_ENCLAVE_ADD_PAGE.
Quoted 7 lines
I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?) Sadly, we don't really have the a nice in-kernel API for that right now. You could do
down_read(mmap_sem)andfind_vma(). Arguably there is no value to checking that PKRU allows execute to the data.
OK, so you would actually go on to check whether the VMA where the data is copied contains executable data?
What if SECINFO.X is not set and you EADD to region that is executable in the enclave VMA? E.g. have RWX VMA for the enclave just to give a simple example. Then you could carry executable code in the data sections of the binary.
/Jarkko
Quoted 25 lines · Jarkko Sakkinen
Quoted 16 lines · Andy Lutomirski
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines · Jarkko Sakkinen
I did study through SDK's file format and realized that it does not does make sense after all to embed one.
Quoted 8 lines
Quoted 5 lines
To implement it properly you would probably need a new syscall (lets say
sgx_load_enclave) and also that enclaves are not just executables binaries. It is hard to find a generic format for them as applications range from simply protecting part of an application to running a containter inside enclave.I'm still puzzling what kind of changes you were discussing considering
SGX_IOC_ENCLAVE_ADD_PAGE.Quoted 7 lines
I think it's as simple as requiring that, if SECINFO.X is set, then the src pointer points to the appropriate number of bytes of executable memory. (Unless there's some way for an enclave to change SECINFO after the fact -- is there?) Sadly, we don't really have the a nice in-kernel API for that right now. You could do
down_read(mmap_sem)andfind_vma(). Arguably there is no value to checking that PKRU allows execute to the data.
Quoted 7 lines
OK, so you would actually go on to check whether the VMA where the data is copied contains executable data?
What if SECINFO.X is not set and you EADD to region that is executable in the enclave VMA? E.g. have RWX VMA for the enclave just to give a simple example. Then you could carry executable code in the data sections of the binary.
This would require to be done after the enclave is initialized of course with EMODPR, which requires the enclave to accept the permission change with EACCEPT, which limits somewhat. This means that the static enclave would be fully covered.
/Jarkko
Quoted 24 lines · Andy Lutomirski
Quoted 14 lines · Jethro Beekman
Quoted 3 lines · Cedric
Hi Jethro,
Quoted 1 line
ELF files are explicitly designed such that you can map them (with
mmap)
Quoted 2 lines
Quoted 2 lines
in 4096-byte chunks. However, sometimes there's overlap and you will sometimes see that a particular offset is mapped twice because the
first
Quoted 1 line
Quoted 1 line
half of the page in the file belongs to an RX range and the second
half
Quoted 3 lines
Quoted 2 lines
to an R-only range. Also, ELF files don't (normally) describe stack, heap, etc. which you do need for enclaves.
Quoted 8 lines
Quoted 1 line
You have probably misread my email. By
mmap(), I meant the enclavefile would be mapped via *multiple*
mmap()calls, in the same way as whatdlopen()would do in loading regular shared object. The intention here is to make the enclave file subject to the same checks as regular shared objects.No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's on-disk representation.
Quoted 6 lines
For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.
Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.
By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant. We are loading enclave files as RO and copying those into EPC. An IMA policy can enforce RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.
And sgx enclaves are always signed and integrity protected and verified at the time of EINIT. So if EINIT passes, we know the content loaded (including permission flags) is matching the sigstruct. But sigstruct/signature is part of the file, should be accounted for in IMA measurement of the whole file, so it is also verified by IMA during file open, right?
The only potential gap/difference comparing to regular ELF executable or DSOs:for enclaves, we need mmap portions of enclave linear range with RW to do EADD IOC, then mprotect those pages to RX after EINIT. But this is operated on enclave fd provided by driver. So we can have an SELinux policy say: only this type of processes is allowed to open enclave fd, and allowed to do mmap/mprotect with read, write, execute on it. Wouldn't that be enough?
Thanks
Haitao
Andy Lutomirski
maintainerQuoted 39 lines · Haitao Huang
Quoted 28 lines · Andy Lutomirski
Quoted 20 lines · Jethro Beekman
Quoted 3 lines · Cedric
Hi Jethro,
Quoted 1 line
ELF files are explicitly designed such that you can map them (with
mmap)
Quoted 2 lines
Quoted 2 lines
in 4096-byte chunks. However, sometimes there's overlap and you will sometimes see that a particular offset is mapped twice because the
first
Quoted 1 line
Quoted 1 line
half of the page in the file belongs to an RX range and the second
half
Quoted 4 lines
Quoted 2 lines
to an R-only range. Also, ELF files don't (normally) describe stack, heap, etc. which you do need for enclaves.
You have probably misread my email. By
mmap(), I meant the enclavefile would be mapped via *multiple*
mmap()calls, in the same way as whatdlopen()would do in loading regular shared object. The intention here is to make the enclave file subject to the same checks as regular shared objects.Quoted 5 lines
No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's on-disk representation.Quoted 6 lines
For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.
Quoted 7 lines
Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.
By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant.
The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:
Quoted 2 lines
We are loading
enclave files as RO and copying those into EPC.
Which is, semantically, a lot like loading a normal file as RO and then mprotecting() it to RX, which is disallowed under quite a few LSM policies.
Quoted 3 lines
An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.
If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.
Quoted 7 lines
And sgx enclaves are always signed and integrity protected and verified at the time of EINIT. So if EINIT passes, we know the content loaded (including permission flags) is matching the sigstruct. But sigstruct/signature is part of the file, should be accounted for in IMA measurement of the whole file, so it is also verified by IMA during file open, right?
This does work, but only if the kernel parses that file so that the kernel can trust that the enclave data actually came from the file as intended. And moving the parsing to the kernel seems like a mess that no one really wants to do.
Quoted 64 lines · Andy Lutomirski
Quoted 44 lines · Haitao Huang
Quoted 9 lines · Andy Lutomirski
Quoted 7 lines · Jethro Beekman
Quoted 3 lines · Cedric
Hi Jethro,
Quoted 1 line
ELF files are explicitly designed such that you can map them (with
mmap)
Quoted 1 line
Quoted 1 line
in 4096-byte chunks. However, sometimes there's overlap and you
will
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
sometimes see that a particular offset is mapped twice because the
first
Quoted 1 line
Quoted 1 line
half of the page in the file belongs to an RX range and the second
half
Quoted 1 line
Quoted 1 line
to an R-only range. Also, ELF files don't (normally) describe
stack,
Quoted 5 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
heap, etc. which you do need for enclaves.
You have probably misread my email. By
mmap(), I meant the enclavefile would be mapped via *multiple*
mmap()calls, in the same way as whatdlopen()would do in loading regular shared object. Theintention
Quoted 1 line
Quoted 1 line
here is to make the enclave file subject to the same checks as
regular
Quoted 5 lines
Quoted 5 lines
shared objects.
No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave'son-disk
Quoted 9 lines
Quoted 2 lines
representation.
For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.
Quoted 7 lines
Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.
By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant.
The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:
Quoted 2 lines
We are loading
enclave files as RO and copying those into EPC.Which is, semantically, a lot like loading a normal file as RO and then
mprotecting()it to RX, which is disallowed under quite a few LSM policies.
Quoted 7 lines
Quoted 3 lines
An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.
But in this case, file itself is mapped as RO treated like data and it is not for execution. SGX enclave pages have EPCM enforced permissions. So from SELinux point of view I would think it can treat it as RO and that's fine.
Quoted 8 lines
Quoted 7 lines
And sgx enclaves are always signed and integrity protected and verified at the time of EINIT. So if EINIT passes, we know the content loaded (including permission flags) is matching the sigstruct. But sigstruct/signature is part of the file, should be accounted for in IMA measurement of the whole file, so it is also verified by IMA during file open, right?
Quoted 4 lines
This does work, but only if the kernel parses that file so that the kernel can trust that the enclave data actually came from the file as intended. And moving the parsing to the kernel seems like a mess that no one really wants to do.
If kernel only needs to know the source bytes are from a file that passed IMA integrity and SELinux RO enforcement, then it can just check if the source pointer belongs to a VMA with valid fd and no parsing or checking permissions needed. Understood if you want to make enclave file code segment stick to the RX semantics mentioned above, then this doesn't qualify.
Thanks
Haitao
Andy Lutomirski
maintainerQuoted 67 lines · Haitao Huang
Quoted 66 lines · Andy Lutomirski
Quoted 39 lines · Haitao Huang
Quoted 28 lines · Andy Lutomirski
Quoted 20 lines · Jethro Beekman
Quoted 3 lines · Cedric
Hi Jethro,
Quoted 1 line
ELF files are explicitly designed such that you can map them (with
mmap)
Quoted 2 lines
Quoted 2 lines
in 4096-byte chunks. However, sometimes there's overlap and you will sometimes see that a particular offset is mapped twice because the
first
Quoted 1 line
Quoted 1 line
half of the page in the file belongs to an RX range and the second
half
Quoted 4 lines
Quoted 2 lines
to an R-only range. Also, ELF files don't (normally) describe stack, heap, etc. which you do need for enclaves.
You have probably misread my email. By
mmap(), I meant the enclavefile would be mapped via *multiple*
mmap()calls, in the same way as whatdlopen()would do in loading regular shared object. The intention here is to make the enclave file subject to the same checks as regular shared objects.Quoted 5 lines
No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's on-disk representation.Quoted 6 lines
For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.
Quoted 7 lines
Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.
By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant.
The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:
Quoted 2 lines
We are loading
enclave files as RO and copying those into EPC.Which is, semantically, a lot like loading a normal file as RO and then
mprotecting()it to RX, which is disallowed under quite a few LSM policies.Quoted 3 lines
An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.
Quoted 5 lines
Quoted 3 lines
If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.
But in this case, file itself is mapped as RO treated like data and it is not for execution. SGX enclave pages have EPCM enforced permissions. So from SELinux point of view I would think it can treat it as RO and that's fine.
As an example, SELinux has an “execute” permission (via security_mmap_file — see file_map_prot_check()) that controls whether you can execute code from that file. If you lack this permission on a file, you may still be able to map it PROT_READ, but you may not map it PROT_EXEC. Similarly, if you want to malloc() some memory, write *code* to it, and execute it, you need a specific permission.
So, unless we somehow think it’s okay for SGX to break the existing model, we need to respect these restrictions in the SGX driver. In other words, we either need to respect execmem, etc or require PROT_EXEC or the equivalent. I like the latter a lot more.
Quoted 92 lines · Andy Lutomirski
Quoted 79 lines · Haitao Huang
Quoted 78 lines · Andy Lutomirski
Quoted 51 lines · Haitao Huang
Quoted 1 line · Andy Lutomirski
On Fri, May 10, 2019 at 12:04 PM Jethro Beekman
Quoted 6 lines
Quoted 5 lines · wrote:
Quoted 3 lines · Cedric
Hi Jethro,
Quoted 1 line
ELF files are explicitly designed such that you can map them
(with
Quoted 2 lines
Quoted 2 lines
mmap)
Quoted 1 line
Quoted 1 line
in 4096-byte chunks. However, sometimes there's overlap and you
will
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
sometimes see that a particular offset is mapped twice because
the
Quoted 2 lines
Quoted 2 lines
first
Quoted 1 line
Quoted 1 line
half of the page in the file belongs to an RX range and the
second
Quoted 2 lines
Quoted 2 lines
half
Quoted 1 line
Quoted 1 line
to an R-only range. Also, ELF files don't (normally) describe
stack,
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
heap, etc. which you do need for enclaves.
You have probably misread my email. By
mmap(), I meant theenclave
Quoted 1 line
Quoted 1 line
file would be mapped via *multiple*
mmap()calls, in the same wayas
Quoted 1 line
Quoted 1 line
what
dlopen()would do in loading regular shared object. Theintention
Quoted 1 line
Quoted 1 line
here is to make the enclave file subject to the same checks as
regular
Quoted 5 lines
Quoted 5 lines
shared objects.
No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave'son-disk
Quoted 4 lines
Quoted 2 lines
representation.
For a tiny bit of background, Linux (AFAIK*) makes no effort to
ensure
Quoted 5 lines
the complete integrity of DSOs. What Linux *does* do (if so configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.
Quoted 7 lines
Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.
By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux compliant.
The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:
Quoted 2 lines
We are loading
enclave files as RO and copying those into EPC.Which is, semantically, a lot like loading a normal file as RO and then
mprotecting()it to RX, which is disallowed under quite a few LSM policies.Quoted 3 lines
An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.Quoted 5 lines
Quoted 3 lines
If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.
But in this case, file itself is mapped as RO treated like data and it is not for execution. SGX enclave pages have EPCM enforced permissions. So from SELinux point of view I would think it can treat it as RO and that's fine.
As an example, SELinux has an “execute” permission (via
security_mmap_file— seefile_map_prot_check()) that controls whether you can execute code from that file. If you lack this permission on a file, you may still be able to map itPROT_READ, but you may not map itPROT_EXEC. Similarly, if you want tomalloc()some memory, write *code* to it, and execute it, you need a specific permission.
Quoted 4 lines
So, unless we somehow think it’s okay for SGX to break the existing model, we need to respect these restrictions in the SGX driver. In other words, we either need to respect execmem, etc or require
PROT_EXECor the equivalent. I like the latter a lot more.
What puzzles me is that this restriction does not add real value to security.
When enclave files are mapped with PROT_READ, without SE execute permission. No breakage to LSM model in normal process address space as no one can execute code directly from the file in normal memory. When enclave is built and loaded into EPC by EADDs, if the SIGSTRUCT is trusted (either signer or MRENCLAVE), EINIT will guarantee security (both integrity and permissions). LSM may not like the fact the a piece of code got loaded into EPC page without specifically giving SE execute permission. However, LSM can be used to control what SIGSTRUCTs can be trusted as you suggested and indirectly enforce what code got executed inside EPC.
So to me, only SIGSTRUCT verification would add value.
Hi Everyone,
I think we are talking about 2 different kinds of criteria for determining the sanity of an enclave.
The first kind determines an enclave's sanity by generally accepted good practices. For example, no executable pages shall ever be writable.
The second kind determines an enclave's sanity by "who stands behind it", such as whether the file containing SIGSTRUCT has the proper SELinux label/type, or whether the signing key is trusted.
I'd say those 2 kinds of criteria should be orthogonal because they don't get into each other's way and it could also be beneficial to enable both at the same time. For example, a user may want to allow launching enclaves signed by himself/herself only, however, as a human being he/she may make mistakes so would also like to ensure no RWX pages even for those enclaves explicitly authorized.
I think those 2 kinds of criteria could be abstracted as 2 new LSM hooks - security_sgx_add_pages() and security_sgx_initialize_enclave().
security_sgx_add_pages() is invoked by SGX driver to determine if a range of source pages are allowed to be EADD'ed with the requested EPCM attributes, and by default it returns 0 (allowed) iff the requested EPCM attributes are a subset of the permissions of the VMA covering that range of source pages. An (SGX-aware) LSM module/policy could employ different criteria, such as making sure the source pages are backed by an enclave file (using SELinux label, for example).
security_sgx_initialize_enclave() is invoked by SGX driver to determine if a given SIGSTRUCT is valid (hence allowed to be EINIT'ed), and it always returns 0 (allowed) by default. An LSM implementation may enforce custom policies, such as whether the signing public key is trusted by the current user, or whether it was backed (mmap()'ed) by an authorized file (e.g. of an expected type in the case of SELinux, or located in a particular directory in the case of AppArmor).
With regard to SGX2/EDMM (Enclave Dynamic Memory Management) support, RW->RX transitions are inevitable to support certain usages such as dynamic loading/linking, meaning those usages may be blocked by existing policies. But from security perspective, I think they should be allowed by default (i.e. for non-SGX-aware LSM modules/policies) because such permission changes are inherent behaviors of the enclave itself, which is considered "sane" after passing all the checks performed in security_sgx_add_pages()/security_sgx_initialize_enclave(). Of course, an SGX-aware LSM module/policy shall be allowed to override. How about adding a new security_sgx_mprotect() LSM hook?
Quoted 95 lines
From: Andy Lutomirski [mailto:[email protected]]
Quoted 1 line
On May 14, 2019, at 8:30 AM, Haitao Huang
Quoted 2 lines · <[email protected]> wrote:
Quoted 2 lines
On Tue, 14 May 2019 10:17:29 -0500, Andy Lutomirski <[email protected]>
Quoted 49 lines · wrote:
Quoted 49 lines
Quoted 46 lines · Haitao Huang
Quoted 40 lines · Andy Lutomirski
Quoted 24 lines · Jethro Beekman
Quoted 4 lines · Cedric
Hi Jethro,
Quoted 2 lines
ELF files are explicitly designed such that you can map them (with
mmap)
Quoted 3 lines
Quoted 3 lines
in 4096-byte chunks. However, sometimes there's overlap and you will sometimes see that a particular offset is mapped twice because the
first
Quoted 2 lines
Quoted 2 lines
half of the page in the file belongs to an RX range and the second
half
Quoted 5 lines
Quoted 2 lines
to an R-only range. Also, ELF files don't (normally) describe stack, heap, etc. which you do need for enclaves.
You have probably misread my email. By
mmap(), I meant the enclavefile would be mapped via *multiple*
mmap()calls, in the same way as whatdlopen()would do in loading regular shared object. The intention here is to make the enclave file subject to the same checks as regular shared objects.Quoted 5 lines
No, I didn't misread your email. My original point still stands:
requiring that an enclave's memory is created from one or more mmap calls of a file puts significant restrictions on the enclave's on-disk representation.For a tiny bit of background, Linux (AFAIK*) makes no effort to ensure the complete integrity of DSOs. What Linux *does* do (if so
configured) is to make sure that only approved data is mapped executable. So, if you want to have some bytes be executable, those bytes have to come from a file that passes the relevant LSM and IMA checks.Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Given this, I just want to step back a little to understand the exact issue that SGX is causing here for LSM/IMA. Sorry if I missed points discussed earlier.
By the time of EADD, enclave file is opened and should have passed IMA and SELinux policy enforcement gates if any. We really don't need extra mmaps on the enclave files to be IMA and SELinux
compliant.
Quoted 15 lines
Quoted 15 lines
The problem, as i see it, is that they passed the *wrong* checks, because, as you noticed:
Quoted 2 lines
We are loading
enclave files as RO and copying those into EPC.Which is, semantically, a lot like loading a normal file as RO and then
mprotecting()it to RX, which is disallowed under quite a few LSM policies.Quoted 3 lines
An IMA policy can enforce
RO files (or any file). And SELinux policy can say which processes can open the file for what permissions. No extra needed here.Quoted 5 lines
Quoted 3 lines
If SELinux says a process may open a file as RO, that does *not* mean that it can be opened as RX.
But in this case, file itself is mapped as RO treated like data and it
is not for execution. SGX enclave pages have EPCM enforced permissions. So from SELinux point of view I would think it can treat it as RO and that's fine.
As an example, SELinux has an “execute” permission (via
security_mmap_file— seefile_map_prot_check()) that controls whether you can execute code from that file. If you lack this permission on a file, you may still be able to map itPROT_READ, but you may not map itPROT_EXEC. Similarly, if you want tomalloc()some memory, write *code* to it, and execute it, you need a specific permission.
Quoted 4 lines
So, unless we somehow think it’s okay for SGX to break the existing model, we need to respect these restrictions in the SGX driver. In other words, we either need to respect execmem, etc or require
PROT_EXECor the equivalent. I like the latter a lot more.
-Cedric
Quoted 6 lines · Cedric
Hi Everyone,
I think we are talking about 2 different kinds of criteria for determining the sanity of an enclave.
The first kind determines an enclave's sanity by generally accepted good practices. For example, no executable pages shall ever be writable.
We'll have to trust user space doing mmap with right permissions as SELinux does not enforce which segment to be RW and which to be RX. The file needs to have SELinux EXECUTE and WRITE both, if we need map some segments with RW and others with RX.
We could say EINIT would ensure user is doing the right thing because it would fail if user map permission wrongly. Then the extra mmaps are redundant of doing SIGSTRUCT verification.
Additionally, per Sean's comments, after EADD in current implementation, we will still need PROCESS_EXECMEM for mprotect on enclave fd to change some EPC pages PTE to RX before enclave can execute. So I don't think mmap the source enclave file would gain anything in addition to what your proposed security_sgx_initialize_enclave() does.
Since security_sgx_initialize_enclave() is a lot like launch control policy enforcement we discussed a lot and resolved, I tend to agree with Andy's assessment we can just do nothing for the initial merge and add hooks needed if someone wants them. And the initial merge would require the enclave hosting processes ask for PROCESS_EXECMEM permission to do mmap/mprotect with enclave fd.
Hi Dave,
Quoted 7 lines · Jethro Beekman
Quoted 2 lines
It does assume a specific format, namely, that the memory layout (including page types/permissions) of the enclave can be represented
in
Quoted 2 lines
a "flat file" on disk, or at least that the enclave memory contents consist of 4096-byte chunks in that file.
Quoted 8 lines
I _think_ Cedric's point is that, to the kernel,
/lib/x86_64-linux-gnu/libc.so.6is a "flat file" because the kernel doesn't have any part in parsing the executable format of a shared library.I actually don't know how it works, though. Do we just just trust that the userspace parsing of the .so format is correct? Do we just assume that any part of a file passing IMA checks can be
PROT_EXEC?
The key idea here is for enclave files to "inherit" the checks applicable to regular shared objects. And how we are going to do it is for user process to map every loadable segment of the enclave file into its address space using *multiple* mmap() calls, just in the same way as dlopen() would do for loading regular shared objects. Then those open()/mmap() syscalls will trigger all applicable checks (by means of security_file_open(), security_mmap_file() and ima_file_mmap() hooks) transparently. That said, IMA/LSM implementations/policies will dictate the success/failure of those open()/mmap() syscalls. And by depending EPCM attributes on permissions of source VMAs, no EXEC page could be ever created unless the source page (which has to be EXEC, btw) has passed IMA/LSM checks (as if it were loaded from a regular shared object file).
-Cedric
Quoted 11 lines · Jethro Beekman
Quoted 10 lines · Andy Lutomirski
With plain
mmap()+mprotect(), the LSM will prevent you from making memory that *was* writable executable. This is by design and SELinux supports it. I don’t remember the name of the associated SELinux permission off the top of my head.If we start enforcing equivalent rules on SGX, then the current API will simply not allow enclaves to be loaded — no matter how you slice it, loading an enclave with the current API is indistinguishable from making arbitrary data executable.
Quoted 4 lines
Yes this is exactly what I intended here: a very simple change that stops SGX from confusing LSM. Just by enforcing that everything that looks like a memory write (EADD, EAUG, EDBGWR, etc.) actually requires write permissions, reality and LSM should be on the same page.
And how so? You create writeable AND executable memory. That's a nono and you can argue in circles, that's not going to change with any of your proposed changes. Andy clearly made a proposal which solves it in a proper way.
Thanks,
tglx
Quoted 18 lines · Jethro Beekman
Quoted 17 lines · Thomas Gleixner
Quoted 9 lines · Jethro Beekman
Quoted 5 lines · Andy Lutomirski
There are many,
many Linux systems that enforce a policy that *all* executable text needs to come from a verified source. On these systems, you can't mmap some writable memory, write to it, and then change it to executable.How is this implemented on those systems? AFAIK there's no kernel config option that changes the semantics of mmap as you describe.
That has nothing to do with
mmap()semantics. Yoummap()writeable memory and then you change the permissions viamprotect().mprotect()calls into LSM and depending on policy and security model this will reject the request.
Quoted 6 lines
Quoted 2 lines
Andy was pointing out that the SGX ioctl bypasses the LSM mechanics which is obviously a bad thing.
We could modify the driver such that when you call ioctl EADD, the page table permissions need to be the PAGEINFO.SECINFO.FLAGS |
PROT_WRITE, otherwise you get EPERM or so. After EADD, if you want, you can restrict
Actually, I don't think you even need to include PAGEINFO.SECINFO.FLAGS, you just need to ensure PROT_WRITE. Regular page table checks take care of PAGEINFO.SECINFO.FLAGS.
--
Jethro Beekman | Fortanix
Quoted 3 lines · Dr. Greg
3.) The approach we propose addresses these issues while imposing no functional limitations on how Linux platform owners can use enclave technology.
Let's say v20 gets merged. Is there any reason your approach can not be rebased on top of that code, reviewed and merged later?
cover An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
In reply to Jarkko SakkinenThe current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.
This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new __vdso_sgx_enter_enclave() will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null), __vdso_sgx_enter_enclave() will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).
The SGX selftest is augmented to test out the new callback interface, and to serve as a simple example to showcase how to use the callback interface in practice.
Cedric Xing (3):
selftests/x86: Fixed Makefile for SGX selftest
x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing
on untrusted stack
selftests/x86: Augment SGX selftest to test new
__vdso_sgx_enter_enclave() and its callback interface
arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++--------- arch/x86/include/uapi/asm/sgx.h | 14 +- tools/testing/selftests/x86/Makefile | 12 +- tools/testing/selftests/x86/sgx/Makefile | 45 +++--- tools/testing/selftests/x86/sgx/main.c | 123 +++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 +++++- 6 files changed, 264 insertions(+), 126 deletions(-)
Signature · 1 line
2.17.1
RFC PATCH 1/3 selftests/x86: Fixed Makefile for SGX selftest
The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.
--- tools/testing/selftests/x86/Makefile | 12 +++---- tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++--------------- 2 files changed, 22 insertions(+), 35 deletions(-)
tools/testing/selftests/x86/Makefile +6 -6
@@ -70,11 +70,11 @@ all_32: $(BINARIES_32) all_64: $(BINARIES_64) -all_64: $(SUBDIRS_64) - @for DIR in $(SUBDIRS_64); do \ - BUILD_TARGET=$(OUTPUT)/$$DIR; \ - mkdir $$BUILD_TARGET -p; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ +all_64: | $(SUBDIRS_64) + @for DIR in $|; do \ + BUILD_TARGET=$(OUTPUT)/$$DIR; \ + mkdir $$BUILD_TARGET -p; \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ done EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)
@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) all: warn_32bit_failure warn_32bit_failure: - @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \ + @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \ echo "environment. This will reduce test coverage of 64-bit" 2>&1; \ echo "kernels. If you are using a Debian-like distribution," 2>&1; \ echo "try:"; 2>&1; \
tools/testing/selftests/x86/sgx/Makefile +16 -29
63 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH 2/3 x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
In reply to Cedric XingThe previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).
This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.
Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.
--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++++--------- arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 100 insertions(+), 70 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +93 -63
@@ -14,88 +14,118 @@ .code64 .section .text, "ax" -#ifdef SGX_KERNEL_DOC /** * __vdso_sgx_enter_enclave() - Enter an SGX enclave * * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer + * @tcs: **IN 0x08(\%rsp)** - TCS, must be non-NULL + * @ex_info: **IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo' + * pointer + * @callback: **IN 0x18(\%rsp)** - Optional callback function to be called on + * enclave exit or exception * * Return: * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is - * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults + * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not + * allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value + * returned from ``callback`` (if one is supplied). * * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the - * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, - * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with - * the return value passed via ``%eax``. All registers except ``%rsp`` must - * be treated as volatile from the caller's perspective, including but not - * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave - * being run **must** preserve the untrusted ``%rsp`` and stack. + * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, + * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and + * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other registers + * will be passed through to the enclave as is. All registers except ``%rbp`` + * must be treated as volatile from the caller's perspective, including but not + * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave being + * run **must** preserve the untrusted ``%rbp``. + * + * ``callback`` has the following signature: + * int callback(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *ex_info, long r8, long r9, + * void *tcs, long ursp); + * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``, ``%rbx`` + * and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``, ``%rdx``, + * ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave exited/excepted, + * can be accessed directly as input parameters, while other GPRs can be + * accessed in assembly if needed. + * A positive value returned from ``callback`` will be treated as an ENCLU leaf + * (e.g. EENTER/ERESUME) to reenter the enclave, while 0 or a negative return + * value will be passed back to the caller of __vdso_sgx_enter_enclave(). + * It is also **safe** to ``longjmp()`` out of ``callback``. */ -__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) -{ - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; - - if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e) - *e = exception; - return -EFAULT; - } - - return 0; -} -#endif ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax - jb bad_input - + jb 6f cmp $0x3, %eax - ja bad_input + ja 6f - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx - /* Save @exception_info */ - push %rcx - - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu - - add $0x8, %rsp - xor %eax, %eax - ret - -bad_input: - mov $(-EINVAL), %rax - ret - -.pushsection .fixup, "ax" - /* Re-load @exception_info and fill it (if it's non-NULL) */ -2: pop %rcx - test %rcx, %rcx - je 3f + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx + jrcxz 4f mov %eax, EX_LEAF(%rcx) - mov %di, EX_TRAPNR(%rcx) - mov %si, EX_ERROR_CODE(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) mov %rdx, EX_ADDRESS(%rcx) -3: mov $(-EFAULT), %rax + +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %rax + cmovz %rbx, %rax + jz 7f + /* Align stack and clear RFLAGS.DF per x86_64 ABI */ + mov %rsp, %rbx + and $-0x10, %rsp + cld + /* Parameters for *callback */ + push %rbx + push 0x10(%rbp) + /* Call via retpoline */ + call 40f + /* Cleanup stack */ + mov %rbx, %rsp + jmp 1b +40: /* retpoline */ + call 42f +41: pause + lfence + jmp 41b +42: mov %rax, (%rsp) + ret + +5: /* Exception path */ + mov $-EFAULT, %ebx + stc + jmp 3b + +6: /* Unsupported ENCLU leaf */ + cmp $0, %eax + jle 7f + mov $-EINVAL, %eax + +7: /* Epilog */ + leave + .cfi_def_cfa %rsp, 8 ret -.popsection + .cfi_endproc -_ASM_VDSO_EXTABLE_HANDLE(1b, 2b) +_ASM_VDSO_EXTABLE_HANDLE(2b, 5b) ENDPROC(__vdso_sgx_enter_enclave)
arch/x86/include/uapi/asm/sgx.h +7 -7
@@ -24,7 +24,7 @@/** * struct sgx_enclave_create - parameter structure for the -* %SGX_IOC_ENCLAVE_CREATE ioctl +* %SGX_IOC_ENCLAVE_CREATE ioctl * @src: address for the SECS page data */ struct sgx_enclave_create {
@@ -33,7 +33,7 @@ struct sgx_enclave_create {/** * struct sgx_enclave_add_page - parameter structure for the -* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl +* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl * @addr: address within the ELRANGE * @src: address for the page data * @secinfo: address for the SECINFO data
@@ -49,7 +49,7 @@ struct sgx_enclave_add_page {/** * struct sgx_enclave_init - parameter structure for the -* %SGX_IOC_ENCLAVE_INIT ioctl +* %SGX_IOC_ENCLAVE_INIT ioctl * @sigstruct: address for the SIGSTRUCT data */ struct sgx_enclave_init {
@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute { };/** -* struct sgx_enclave_exception - structure to report exceptions encountered in -* __vdso_sgx_enter_enclave() +* struct sgx_enclave_exinfo - structure to report exceptions encountered in +* __vdso_sgx_enter_enclave() * -* @leaf: ENCLU leaf from \%eax at time of exception +* @leaf: ENCLU leaf from \%eax at time of exception/exit * @trapnr: exception trap number, a.k.a. fault vector * @error_code: exception error code * @address: exception address, e.g. CR2 on a #PF * @reserved: reserved for future use */ -struct sgx_enclave_exception { +struct sgx_enclave_exinfo {__u32 leaf;__u16 trapnr;__u16 error_code;
Signature · 1 line
2.17.1
Re: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 4 lines · Cedric Xing
The previous
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).
Quoted 10 lines
This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then
__vdso_sgx_enter_enclave()can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after__vdso_sgx_enter_enclave()returns. Therefore,__vdso_sgx_enter_enclave()has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.
Please resend this series with the new code as a separate function, not a modification to the existing function. Andy made it pretty clear that he prefers a separate vDSO function, and it'll be a lot easier to provide feedback on the code if we're working from a clean slate. If you want to push to get everything lumped into a single function then by all means do so on LKML, but for the RFC let's keep the focus on the code itself and simplify life for reviewers.
Quoted 5 lines · Andy Lutomirski
If the answer to both questions is yes, then it seems like it could be reasonable to support it in the vDSO. But I still think it should probably be a different vDSO entry point so that the normal case doesn't become more complicated.
Quoted 15 lines
Here's the summary of API/ABI changes in this patch. More details could be found in
arch/x86/entry/vdso/vsgx_enter_enclave.S. * 'structsgx_enclave_exception' is renamed to 'structsgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits. *__vdso_sgx_enter_enclave()anchors its frame using %rbp (instead of %rsp in the previous implementation). *__vdso_sgx_enter_enclave()takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause__vdso_sgx_enter_enclave()to return upon enclave exits (same behavior as previous implementation). * The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of__vdso_sgx_enter_enclave()to its caller. It's also safe to leave callback bylongjmp()or by throwing a C++ exception.
Again, wrap at 75 chars or earlier. And incorporate checkpatch into your git send-email flow if you haven't done so already, e.g. checkpatch should have warned about lines longer than 75 chars.
Quoted 2 lines
Signed-off-by Cedric Xing <[email protected]>---
Andy Lutomirski
maintainerRe: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
In reply to Cedric XingQuoted 85 lines · Cedric Xing
The previous
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then
__vdso_sgx_enter_enclave()can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after__vdso_sgx_enter_enclave()returns. Therefore,__vdso_sgx_enter_enclave()has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left
off by the enclave as a parameter to the callback.Here's the summary of API/ABI changes in this patch. More details could be found in
arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'structsgx_enclave_exception' is renamed to 'structsgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
*__vdso_sgx_enter_enclave()anchors its frame using %rbp (instead of %rsp in the previous implementation).
*__vdso_sgx_enter_enclave()takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause__vdso_sgx_enter_enclave()to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of__vdso_sgx_enter_enclave()to its caller. It's also safe to leave callback bylongjmp()or by throwing a C++ exception.Signed-off-by Cedric Xing <[email protected]>--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++++--------- arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 100 insertions(+), 70 deletions(-)arch/x86/entry/vdso/vsgx_enter_enclave.S +17
-11@@ -14,88 +14,118 @@ .code64 .section .text, "ax" -#ifdef SGX_KERNEL_DOC /** * __vdso_sgx_enter_enclave() - Enter an SGX enclave * * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer + * @tcs: **IN 0x08(\%rsp)** - TCS, must be non-NULL + * @ex_info: **IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo' + * pointer + * @callback: **IN 0x18(\%rsp)** - Optional callback function to be called on + * enclave exit or exception * * Return: * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is - * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults + * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not + * allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value + * returned from ``callback`` (if one is supplied). * * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the - * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, - * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with - * the return value passed via ``%eax``. All registers except ``%rsp`` must - * be treated as volatile from the caller's perspective, including but not - * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave - * being run **must** preserve the untrusted ``%rsp`` and stack. + * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, + * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and + * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other registers + * will be passed through to the enclave as is. All registers except ``%rbp`` + * must be treated as volatile from the caller's perspective, including but not + * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave being + * run **must** preserve the untrusted ``%rbp``. + * + * ``callback`` has the following signature:
arch/x86/entry/vdso/vsgx_enter_enclave.S line 52
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +10
-0+ * int callback(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *ex_info, long r8, long r9, + * void *tcs, long ursp); + * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``, ``%rbx`` + * and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``, ``%rdx``, + * ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave exited/excepted, + * can be accessed directly as input parameters, while other GPRs can be + * accessed in assembly if needed. + * A positive value returned from ``callback`` will be treated as an ENCLU leaf + * (e.g. EENTER/ERESUME) to reenter the enclave, while 0 or a negative return
"to reenter the enclave without popping the extra data pushed by the enclave off the stack" or similar. We really don't want a situation where someone puts all there "keep on going" logic in the callback and the stack usage grows without bound.
arch/x86/entry/vdso/vsgx_enter_enclave.S line 54
Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2
-0+ * value will be passed back to the caller of __vdso_sgx_enter_enclave(). + * It is also **safe** to ``longjmp()`` out of ``callback``.
I'm not sure that "safe" needs emphasis.
Quoted 76 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +20
-48*/ -__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) -{ - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; - - if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e) - *e = exception; - return -EFAULT; - } - - return 0; -} -#endif ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax - jb bad_input - + jb 6f cmp $0x3, %eax - ja bad_input + ja 6f - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx - /* Save @exception_info */ - push %rcx - - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu - - add $0x8, %rsp - xor %eax, %eax - ret - -bad_input: - mov $(-EINVAL), %rax - ret - -.pushsection .fixup, "ax" - /* Re-load @exception_info and fill it (if it's non-NULL) */ -2: pop %rcx - test %rcx, %rcx - je 3f + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx + jrcxz 4f mov %eax, EX_LEAF(%rcx) - mov %di, EX_TRAPNR(%rcx)
arch/x86/entry/vdso/vsgx_enter_enclave.S line 90
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +7
-2- mov %si, EX_ERROR_CODE(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) mov %rdx, EX_ADDRESS(%rcx) -3: mov $(-EFAULT), %rax + +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %rax
Maybe have a comment like "At this point, the effective return value is in RBX. If there is no callback, then return it."
arch/x86/entry/vdso/vsgx_enter_enclave.S line 94
Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4
-0+ cmovz %rbx, %rax + jz 7f + /* Align stack and clear RFLAGS.DF per x86_64 ABI */ + mov %rsp, %rbx
Whoa, this is too subtle here. Can you update the comment to clarify that the uRSP value set by the enclave needs to be saved so that the enclave can be resumed if needed?
arch/x86/entry/vdso/vsgx_enter_enclave.S line 103
Quoted 9 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +9
-0+ and $-0x10, %rsp + cld + /* Parameters for *callback */ + push %rbx + push 0x10(%rbp) + /* Call via retpoline */ + call 40f + /* Cleanup stack */ + mov %rbx, %rsp
To me, "Cleanup stack" makes me think that you're restoring the original RSP, but you're actually just undoing in the stack alignment. How about "Undo stack alignment"?
But I'm not seeing the code that causes a return value RAX <= 0 to just return.
Quoted 18 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +18
-0+ jmp 1b +40: /* retpoline */ + call 42f +41: pause + lfence + jmp 41b +42: mov %rax, (%rsp) + ret + +5: /* Exception path */ + mov $-EFAULT, %ebx + stc + jmp 3b + +6: /* Unsupported ENCLU leaf */ + cmp $0, %eax + jle 7f + mov $-EINVAL, %eax
arch/x86/entry/vdso/vsgx_enter_enclave.S line 129
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6
-2+ +7: /* Epilog */ + leave + .cfi_def_cfa %rsp, 8 ret -.popsection + .cfi_endproc -_ASM_VDSO_EXTABLE_HANDLE(1b, 2b) +_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)
--Andy
RE: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Hi Andy,
I sent out my RFC patch v2 last night, that has your suggestions incorporated, plus a new unwind test to single step through the vDSO API to test out the CFI directives. Hopefully it is able to address all of your concerns. It's worth noting that, given the current patch fixes up #DB and #BP at ENCLU, the unwind test cannot run to completion. I assume Sean will revise the fixup code soon.
Thanks!
-Cedric
Quoted 310 lines
-----Original Message-----
From: Andy Lutomirski [mailto:[email protected]]
Sent: Monday, April 22, 2019 6:26 PM
To: Xing, Cedric <[email protected]>
Cc: LKML <[email protected]>; X86 ML <[email protected]>; linux- [email protected]; Andrew Morton <[email protected]>; Hansen, Dave <[email protected]>; Christopherson, Sean J <[email protected]>; [email protected]; [email protected]; Ayoun, Serge <[email protected]>; Katz-zamir, Shay <[email protected]>; Huang, Haitao <[email protected]>; Andy Shevchenko <[email protected]>; Thomas Gleixner <[email protected]>; Svahn, Kai <[email protected]>; Borislav Petkov <[email protected]>; Josh Triplett <[email protected]>; Andrew Lutomirski <[email protected]>; Huang, Kai <[email protected]>; David Rientjes <[email protected]>; Jarkko Sakkinen <[email protected]>
Subject: Re: [RFC PATCH v1 2/3] x86/vdso: Modify__vdso_sgx_enter_enclave()to allow parameter passing on untrusted stackQuoted 18 lines · Cedric Xing
The previous
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then
__vdso_sgx_enter_enclave()can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after__vdso_sgx_enter_enclave()returns. Therefore,__vdso_sgx_enter_enclave()has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave
eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.Quoted 3 lines
Here's the summary of API/ABI changes in this patch. More details could be found in
arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'structsgx_enclave_exception' is renamed to 'struct
sgx_enclave_exinfo'Quoted 1 line
because it is filled upon both AEX (i.e. exceptions) and normal
enclave
Quoted 2 lines
exits.
*__vdso_sgx_enter_enclave()anchors its frame using %rbp (insteadof %rsp in
Quoted 2 lines
the previous implementation).
*__vdso_sgx_enter_enclave()takes one more parameter - a callbackfunction to
Quoted 2 lines
be invoked upon enclave exits. This callback is optional, and if not supplied, will cause
__vdso_sgx_enter_enclave()to return uponenclave exits
Quoted 2 lines
(same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp atenclave
Quoted 1 line
exit to address data "pushed" by the enclave. A positive value
returned by
Quoted 1 line
the callback will be treated as an ENCLU leaf for re-entering the
enclave,
Quoted 7 lines
while a zero or negative value will be passed through as the return value of
__vdso_sgx_enter_enclave()to its caller. It's also safe to leave callback bylongjmp()or by throwing a C++ exception.Signed-off-by Cedric Xing <[email protected]>--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++++---------
Quoted 9 lines
arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 100 insertions(+), 70 deletions(-)arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-0arch/x86/entry/vdso/vsgx_enter_enclave.S line 23
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-3@@ -14,88 +14,118 @@ .code64 .section .text, "ax" -#ifdef SGX_KERNEL_DOC /** * __vdso_sgx_enter_enclave() - Enter an SGX enclave * * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception'pointer
arch/x86/entry/vdso/vsgx_enter_enclave.S line 22
Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2
-0+ * @tcs: **IN 0x08(\%rsp)** - TCS, must be non-NULL + * @ex_info: **IN 0x10(\%rsp)** - Optional 'struct
sgx_enclave_exinfo'arch/x86/entry/vdso/vsgx_enter_enclave.S line 24
Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2
-0+ * pointer + * @callback: **IN 0x18(\%rsp)** - Optional callback function to becalled on
Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-1+ * enclave exit or exception * * Return: * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLUleaf is
- * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclaveQuoted 8 lines
faults
+ * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU
+ leaf is not
+ * allowed, %-EFAULT if ENCLU or the enclave faults, or a
+ non-positive value
+ * returned from ``callback`` (if one is supplied).
*
* **Important!**__vdso_sgx_enter_enclave()is **NOT** compliantwith the
Quoted 1 line
- * x86-64 ABI, i.e. cannot be called from standard C code. As noted
above,
Quoted 32 lines
- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with
- * the return value passed via ``%eax``. All registers except ``%rsp`` must
- * be treated as volatile from the caller's perspective, including but not
- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave
- * being run **must** preserve the untrusted ``%rsp`` and stack.
+ * x86-64 ABI, i.e. cannot be called from standard C code. As noted
+ above,
+ * input parameters must be passed via ``%eax``, ``8(%rsp)``,
+ ``0x10(%rsp)`` and
+ * ``0x18(%rsp)``, with the return value passed via ``%eax``. All
+ other registers
+ * will be passed through to the enclave as is. All registers except
+ ``%rbp``
+ * must be treated as volatile from the caller's perspective,
+ including but not
+ * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the
+ enclave being
+ * run **must** preserve the untrusted ``%rbp``.
+ *
+ * ``callback`` has the following signature:
+ * int callback(long rdi, long rsi, long rdx,
+ * structsgx_enclave_exinfo*ex_info, long r8, long r9,
+ * void *tcs, long ursp);
+ * ``callback`` **shall** followx86_64ABI. All GPRs **except**
+ ``%rax``, ``%rbx``
+ * and ``rcx`` are passed through to ``callback``. ``%rdi``,
+ ``%rsi``, ``%rdx``,
+ * ``%r8``, ``%r9``, along with the value of ``%rsp`` when theQuoted 8 lines
+ enclave exited/excepted,
+ * can be accessed directly as input parameters, while other GPRs can
+ be
+ * accessed in assembly if needed.
+ * A positive value returned from ``callback`` will be treated as an
+ ENCLU leaf
+ * (e.g. EENTER/ERESUME) to reenter the enclave, while 0 or a
+ negative return"to reenter the enclave without popping the extra data pushed by the enclave off the stack" or similar. We really don't want a situation where someone puts all there "keep on going" logic in the callback and the stack usage grows without bound.
Quoted 1 line
+ * value will be passed back to the caller of
__vdso_sgx_enter_enclave().Quoted 1 line
+ * It is also **safe** to ``longjmp()`` out of ``callback``.
I'm not sure that "safe" needs emphasis.
Quoted 76 lines
*/
arch/x86/entry/vdso/vsgx_enter_enclave.S +20
-48-__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) -{ - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; - - if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e) - *e = exception; - return -EFAULT; - } - - return 0; -} -#endif ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax - jb bad_input - + jb 6f cmp $0x3, %eax - ja bad_input + ja 6f - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx - /* Save @exception_info */ - push %rcx - - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu - - add $0x8, %rsp - xor %eax, %eax - ret - -bad_input: - mov $(-EINVAL), %rax - ret - -.pushsection .fixup, "ax" - /* Re-load @exception_info and fill it (if it's non-NULL) */ -2: pop %rcx - test %rcx, %rcx - je 3f + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx + jrcxz 4f mov %eax, EX_LEAF(%rcx) - mov %di, EX_TRAPNR(%rcx)arch/x86/entry/vdso/vsgx_enter_enclave.S line 90
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +7
-2- mov %si, EX_ERROR_CODE(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) mov %rdx, EX_ADDRESS(%rcx) -3: mov $(-EFAULT), %rax + +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %raxMaybe have a comment like "At this point, the effective return value is in RBX. If there is no callback, then return it."
arch/x86/entry/vdso/vsgx_enter_enclave.S line 94
Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4
-0+ cmovz %rbx, %rax + jz 7f + /* Align stack and clear RFLAGS.DF per x86_64 ABI */ + mov %rsp, %rbxWhoa, this is too subtle here. Can you update the comment to clarify that the uRSP value set by the enclave needs to be saved so that the enclave can be resumed if needed?
arch/x86/entry/vdso/vsgx_enter_enclave.S line 103
Quoted 9 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +9
-0+ and $-0x10, %rsp + cld + /* Parameters for *callback */ + push %rbx + push 0x10(%rbp) + /* Call via retpoline */ + call 40f + /* Cleanup stack */ + mov %rbx, %rspTo me, "Cleanup stack" makes me think that you're restoring the original RSP, but you're actually just undoing in the stack alignment. How about "Undo stack alignment"?
But I'm not seeing the code that causes a return value RAX <= 0 to just return.
Quoted 18 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +18
-0+ jmp 1b +40: /* retpoline */ + call 42f +41: pause + lfence + jmp 41b +42: mov %rax, (%rsp) + ret + +5: /* Exception path */ + mov $-EFAULT, %ebx + stc + jmp 3b + +6: /* Unsupported ENCLU leaf */ + cmp $0, %eax + jle 7f + mov $-EINVAL, %eaxarch/x86/entry/vdso/vsgx_enter_enclave.S line 129
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6
-2+ +7: /* Epilog */ + leave + .cfi_def_cfa %rsp, 8 ret -.popsection + .cfi_endproc -_ASM_VDSO_EXTABLE_HANDLE(1b, 2b) +_ASM_VDSO_EXTABLE_HANDLE(2b, 5b)--Andy
Re: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
In reply to Cedric XingQuoted 44 lines · Cedric Xing
The previous
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then
__vdso_sgx_enter_enclave()can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after__vdso_sgx_enter_enclave()returns. Therefore,__vdso_sgx_enter_enclave()has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left
off by the enclave as a parameter to the callback.Here's the summary of API/ABI changes in this patch. More details could be found in
arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'structsgx_enclave_exception' is renamed to 'structsgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
*__vdso_sgx_enter_enclave()anchors its frame using %rbp (instead of %rsp in the previous implementation).
*__vdso_sgx_enter_enclave()takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause__vdso_sgx_enter_enclave()to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of__vdso_sgx_enter_enclave()to its caller. It's also safe to leave callback bylongjmp()or by throwing a C++ exception.Signed-off-by Cedric Xing <[email protected]>--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++++--------- arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 100 insertions(+), 70 deletions(-)arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-0
arch/x86/entry/vdso/vsgx_enter_enclave.S line 17
Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-1@@ -14,88 +14,118 @@ .code64 .section .text, "ax" -#ifdef SGX_KERNEL_DOC
This #ifdef and the pseudo-C code below has a functional purpose. From the original commit:
Note, the C-like pseudocode describing the assembly routine is wrapped in a non-existent macro instead of in a comment to trick kernel-doc into auto-parsing the documentation and function prototype. This is a double win as the pseudocode is intended to aid kernel developers, not userland enclave developers.
We don't need full pseudocode, but a C-like prototype is necessary to get the kernel-doc comment parsed correctly.
I'll try to look at the actual code later today.
Quoted 69 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +29
-30/** * __vdso_sgx_enter_enclave() - Enter an SGX enclave * * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer + * @tcs: **IN 0x08(\%rsp)** - TCS, must be non-NULL + * @ex_info: **IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo' + * pointer + * @callback: **IN 0x18(\%rsp)** - Optional callback function to be called on + * enclave exit or exception * * Return: * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is - * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults + * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not + * allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value + * returned from ``callback`` (if one is supplied). * * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the - * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, - * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with - * the return value passed via ``%eax``. All registers except ``%rsp`` must - * be treated as volatile from the caller's perspective, including but not - * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave - * being run **must** preserve the untrusted ``%rsp`` and stack. + * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, + * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and + * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other registers + * will be passed through to the enclave as is. All registers except ``%rbp`` + * must be treated as volatile from the caller's perspective, including but not + * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave being + * run **must** preserve the untrusted ``%rbp``. + * + * ``callback`` has the following signature: + * int callback(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *ex_info, long r8, long r9, + * void *tcs, long ursp); + * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``, ``%rbx`` + * and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``, ``%rdx``, + * ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave exited/excepted, + * can be accessed directly as input parameters, while other GPRs can be + * accessed in assembly if needed. + * A positive value returned from ``callback`` will be treated as an ENCLU leaf + * (e.g. EENTER/ERESUME) to reenter the enclave, while 0 or a negative return + * value will be passed back to the caller of __vdso_sgx_enter_enclave(). + * It is also **safe** to ``longjmp()`` out of ``callback``. */ -__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) -{ - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; - - if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e) - *e = exception; - return -EFAULT; - } - - return 0; -} -#endif
Re: [RFC PATCH v1 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 57 lines · Sean Christopherson
Quoted 49 lines · Cedric Xing
The previous
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then
__vdso_sgx_enter_enclave()can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after__vdso_sgx_enter_enclave()returns. Therefore,__vdso_sgx_enter_enclave()has been changed to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left
off by the enclave as a parameter to the callback.Here's the summary of API/ABI changes in this patch. More details could be found in
arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'structsgx_enclave_exception' is renamed to 'structsgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
*__vdso_sgx_enter_enclave()anchors its frame using %rbp (instead of %rsp in the previous implementation).
*__vdso_sgx_enter_enclave()takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause__vdso_sgx_enter_enclave()to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of__vdso_sgx_enter_enclave()to its caller. It's also safe to leave callback bylongjmp()or by throwing a C++ exception.Signed-off-by Cedric Xing <[email protected]>---
arch/x86/entry/vdso/vsgx_enter_enclave.S| 156 ++++++++++++++---------arch/x86/include/uapi/asm/sgx.h| 14 +-2 files changed, 100 insertions(+), 70 deletions(-)arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-0@@ -14,88 +14,118 @@.code64
.section .text, "ax"Quoted 1 line
-#ifdef
SGX_KERNEL_DOCThis #ifdef and the pseudo-C code below has a functional purpose. From the original commit:
Quoted 8 lines
Note, the C-like pseudocode describing the assembly routine is wrapped in a non-existent macro instead of in a comment to trick kernel-doc into auto-parsing the documentation and function prototype. This is a double win as the pseudocode is intended to aid kernel developers, not userland enclave developers.
We don't need full pseudocode, but a C-like prototype is necessary to get the kernel-doc comment parsed correctly.
That should be explained in a comment :)
—Andy
RFC PATCH 3/3 selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
In reply to Cedric XingGiven the changes to __vdso_sgx_enter_enclave(), the selftest is augmented to test the newly added callback interface. This addtional test marks the whole enclave range as PROT_READ, and calls mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes.
--- tools/testing/selftests/x86/sgx/main.c | 123 ++++++++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 ++++++- 2 files changed, 142 insertions(+), 21 deletions(-)
tools/testing/selftests/x86/sgx/main.c +105 -18
@@ -9,6 +9,7 @@#include <stdlib.h> #include <string.h> #include <unistd.h> +#include <errno.h> #include <sys/ioctl.h> #include <sys/mman.h> #include <sys/stat.h>
@@ -18,6 +19,10 @@#include "../../../../../arch/x86/kernel/cpu/sgx/arch.h" #include "../../../../../arch/x86/include/uapi/asm/sgx.h" +#define _Q (x) __Q(x) +#define __Q (x) #x +#define ERRLN "Line " _Q(__LINE__) +static const uint64_t MAGIC =0x1122334455667788ULL ;struct vdso_symtab {
@@ -138,7 +143,7 @@ static bool encl_create(int dev_fd, unsigned long bin_size, base =mmap (NULL , secs->size,PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_SHARED , dev_fd,0 );if (base ==MAP_FAILED ) { -perror ("mmap" ); +perror (ERRLN );return false; }
@@ -224,24 +229,113 @@ static bool encl_load(struct sgx_secs *secs, unsigned long bin_size)return false; } -void sgx_call (void *rdi,void *rsi,void *tcs, -struct sgx_enclave_exception *exception, -void *eenter); +int sgx_call (void *rdi,void *rsi,long rdx,void *rcx,void *r8,void *r9, +void *tcs,struct sgx_enclave_exinfo *ei,void *cb,void *eenter); + +static void show_enclave_exinfo (const struct sgx_enclave_exinfo *exinfop, +const char *header) +{ +printf ("%s: leaf:%d" , header, exinfop->leaf); +if (exinfop->leaf !=4 ) +printf (" trap#:%d ec:%d addr:0x%llx\n" , exinfop->trapnr, + exinfop->error_code, exinfop->address); +else printf ("\n" ); +} + +static void test1 (void *eenter,struct sgx_secs *secs) +{ +uint64_t result =0 ; +struct sgx_enclave_exinfo exinfo; + +printf ("[1] Entering the enclave without callback.\n" ); + +printf ("Input: 0x%lx\n Expect: Same as input\n" ,MAGIC ); +sgx_call ((void *)&MAGIC , &result,0 ,NULL ,NULL ,NULL , + (void *)secs->base, &exinfo,NULL , eenter); +if (result !=MAGIC ) { +fprintf (stderr,"0x%lx != 0x%lx\n" , result,MAGIC ); +exit (1 ); + } +printf (" Output: 0x%lx\n" , result); + +printf ("Input: Null TCS\n Expect: #PF at EENTER\n" ); +sgx_call ((void *)&MAGIC , &result,0 ,NULL ,NULL ,NULL , +NULL , &exinfo,NULL , eenter); +show_enclave_exinfo (&exinfo," Exit" ); +if (exinfo.leaf !=2 /*EENTER*/ || exinfo.trapnr !=14 /*#PF*/ ) +exit (1 ); +} + +static int enclave_ex_callback (long rdi,long rsi,long rdx, +struct sgx_enclave_exinfo *ei,long r8,long r9,void *tcs,long ursp) +{ +show_enclave_exinfo (ei," callback" ); + +switch (ei->leaf) + { +case 4 : +return 0 ; +case 3 : +case 2 : +if (ei->trapnr !=14 /*#PF*/ || (ei->error_code &1 ) ==0 ) { +fprintf (stderr,ERRLN ": Unexpected exception\n" ); +exit (1 ); + } + +if (mprotect ((void *)(ei->address &-0x1000 ),0x1000 , + ((ei->error_code &2 ) ?PROT_WRITE :0 ) | + ((ei->error_code &0x10 ) ?PROT_EXEC :0 ) | +PROT_READ )) { +perror (ERRLN ); +exit (1 ); + } + +return ei->leaf ==2 ? -EAGAIN : ei->leaf; + } +return -EINVAL ; +} + +static void test2 (void *eenter,struct sgx_secs *secs) +{ +uint64_t result =0 ; +struct sgx_enclave_exinfo exinfo; + +printf ("[2] Entering the enclave with callback.\n" ); + +printf ("Input: 0x%lx\n Expect: Same as input\n" ,MAGIC ); +sgx_call ((void *)&MAGIC , &result,0 ,NULL ,NULL ,NULL , + (void *)secs->base, &exinfo, enclave_ex_callback, eenter); +if (result !=MAGIC ) { +fprintf (stderr,"0x%lx != 0x%lx\n" , result,MAGIC ); +exit (1 ); + } +printf (" Output: 0x%lx\n" , result); + +printf ("Input: Read-only enclave (0x%lx-0x%lx)\n" +" Expect: #PFs to be fixed by callback\n" , + secs->base, secs->base + (encl_bin_end - encl_bin) -1 ); +if (mprotect ((void *)secs->base, encl_bin_end - encl_bin,PROT_READ )) { +perror (ERRLN ); +exit (1 ); + } +while (sgx_call ((void *)&MAGIC , &result,0 ,NULL ,NULL ,NULL , + (void *)secs->base, &exinfo, enclave_ex_callback, + eenter) == -EAGAIN ); +show_enclave_exinfo (&exinfo," Exit" ); +if (exinfo.leaf !=4 /*EEXIT*/ ) +exit (1 ); +}int main (int argc,char *argv[],char *envp[]) {unsigned long bin_size = encl_bin_end - encl_bin;unsigned long ss_size = encl_ss_end - encl_ss; -struct sgx_enclave_exception exception;Elf64_Sym *eenter_sym;struct vdso_symtab symtab;struct sgx_secs secs; -uint64_t result =0 ;void *eenter;void *addr; -memset (&exception,0 ,sizeof (exception)); - addr =vdso_get_base_addr (envp);if (!addr)exit (1 );
@@ -266,14 +360,7 @@ int main(int argc, char *argv[], char *envp[])if (!encl_load (&secs, bin_size))exit (1 ); -printf ("Input: 0x%lx\n" ,MAGIC ); -sgx_call ((void *)&MAGIC , &result, (void *)secs.base, &exception, - eenter); -if (result !=MAGIC ) { -fprintf (stderr,"0x%lx != 0x%lx\n" , result,MAGIC ); -exit (1 ); - } - -printf ("Output: 0x%lx\n" , result); -exit (0 ); +test1 (eenter, &secs); +test2 (eenter, &secs); +return 0 ; }
tools/testing/selftests/x86/sgx/sgx_call.S +37 -3
@@ -7,9 +7,43 @@ .global sgx_call sgx_call: + .cfi_startproc + push %r15 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r15, 0 + push %r14 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r14, 0 + push %r13 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r13, 0 + push %r12 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r12, 0 push %rbx - mov $0x02, %rax - mov %rdx, %rbx - call *%r8 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbx, 0 + push $0 + .cfi_adjust_cfa_offset 8 + push 0x48(%rsp) + .cfi_adjust_cfa_offset 8 + push 0x48(%rsp) + .cfi_adjust_cfa_offset 8 + push 0x48(%rsp) + .cfi_adjust_cfa_offset 8 + mov $2, %eax + call *0x68(%rsp) + add $0x20, %rsp + .cfi_adjust_cfa_offset -0x20 pop %rbx + .cfi_adjust_cfa_offset -8 + pop %r12 + .cfi_adjust_cfa_offset -8 + pop %r13 + .cfi_adjust_cfa_offset -8 + pop %r14 + .cfi_adjust_cfa_offset -8 + pop %r15 + .cfi_adjust_cfa_offset -8 ret + .cfi_endproc
Signature · 1 line
2.17.1
Andy Lutomirski
maintainerRe: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
Quoted 5 lines · Cedric Xing
Given the changes to
__vdso_sgx_enter_enclave(), the selftest is augmented to test the newly added callback interface. This addtional test marks the whole enclave range asPROT_READ, and callsmprotect()upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes.
Nifty.
What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then do_debug() will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?
--Andy
Re: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
Quoted 9 lines · Andy Lutomirski
Quoted 5 lines · Cedric Xing
Given the changes to
__vdso_sgx_enter_enclave(), the selftest is augmented to test the newly added callback interface. This addtional test marks the whole enclave range asPROT_READ, and callsmprotect()upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes.Nifty.
Quoted 7 lines
What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then
do_debug()will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?
My money would be on the code being broken as opposed to you remembering wrong. I'll take a look at it tomorrow.
Re: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
In reply to Andy LutomirskiQuoted 7 lines · Andy Lutomirski
What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then
do_debug()will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?
The code is indeed broken, and I don't see a sane way to make it not broken other than to never do vDSO fixup on #DB or #BP. But that's probably the right thing to do anyways since an attached debugger is likely the intended recipient the 99.9999999% of the time.
The crux of the matter is that it's impossible to identify whether or not a #DB/#BP originated from within an enclave, e.g. an INT3 in an enclave will look identical to an INT3 at the AEP. Even if hardware provided a magic flag, #DB still has scenarios where the intended recipient is ambiguous, e.g. data breakpoint encountered in the enclave but on an address outside of the enclave, breakpoint encountered in the enclave and a code breakpoint on the AEP, etc...
Andy Lutomirski
maintainerRe: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
Quoted 15 lines · Sean Christopherson
Quoted 7 lines · Andy Lutomirski
What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then
do_debug()will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?The code is indeed broken, and I don't see a sane way to make it not broken other than to never do vDSO fixup on #DB or #BP. But that's probably the right thing to do anyways since an attached debugger is likely the intended recipient the 99.9999999% of the time.
Quoted 7 lines
The crux of the matter is that it's impossible to identify whether or not a #DB/#BP originated from within an enclave, e.g. an INT3 in an enclave will look identical to an INT3 at the AEP. Even if hardware provided a magic flag, #DB still has scenarios where the intended recipient is ambiguous, e.g. data breakpoint encountered in the enclave but on an address outside of the enclave, breakpoint encountered in the enclave and a code breakpoint on the AEP, etc...
Ugh. It sounds like ignoring the fixup for #DB is the right call. But what happens if the enclave contains an INT3 or ICEBP instruction?
Are they magically promoted to #GP, perhaps?
As a maybe possible alternative, if we made it so that the AEX address was not the same as the ENCLU, could we usefully distinguish these exceptions based on RIP? I suppose it's also worth considering whether page faults from *inside* the enclave should result in SIGSEGV or result in a fixup. We certainly want page faults from the ENCLU instruction itself to get fixed up, but maybe we want most exceptions inside the enclave to work a bit differently. Of course, if we do this, we need to make sure that the semantics of returning from the signal handler are reasonable.
Re: [RFC PATCH v1 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
Quoted 25 lines · Andy Lutomirski
Quoted 15 lines · Sean Christopherson
Quoted 7 lines · Andy Lutomirski
What's not tested here is running this code with EFLAGS.TF set and making sure that it unwinds correctly. Also, Jarkko, unless I missed something, the vDSO extable code likely has a bug. If you run the instruction right before ENCLU with EFLAGS.TF set, then
do_debug()will eat the SIGTRAP and skip to the exception handler. Similarly, if you put an instruction breakpoint on ENCLU, it'll get skipped. Or is the code actually correct and am I just remembering wrong?The code is indeed broken, and I don't see a sane way to make it not broken other than to never do vDSO fixup on #DB or #BP. But that's probably the right thing to do anyways since an attached debugger is likely the intended recipient the 99.9999999% of the time.
Quoted 7 lines
The crux of the matter is that it's impossible to identify whether or not a #DB/#BP originated from within an enclave, e.g. an INT3 in an enclave will look identical to an INT3 at the AEP. Even if hardware provided a magic flag, #DB still has scenarios where the intended recipient is ambiguous, e.g. data breakpoint encountered in the enclave but on an address outside of the enclave, breakpoint encountered in the enclave and a code breakpoint on the AEP, etc...
Quoted 3 lines
Ugh. It sounds like ignoring the fixup for #DB is the right call. But what happens if the enclave contains an INT3 or ICEBP instruction?
Are they magically promoted to #GP, perhaps?
#UD for opt-out, a.k.a. non-debug, enclaves. Delivered "normally" for opt-in debug enclaves, except they're fault-like instead of trap-like.
Quoted 3 lines
As a maybe possible alternative, if we made it so that the AEX address was not the same as the ENCLU, could we usefully distinguish these exceptions based on RIP?
Not really, because a user could set a code breakpoint on the AEX or insert an INT3, e.g. to break on exit from the enclave.
Theoretically the kernel could cross-reference addresses to determine whether or not a DRx match occurred on an enclave address, but a) that'd be pretty ugly to implement and b) there would still be ambiguity, e.g. if there's a code breakpoint on the AEX and a #DB occurs in the enclave, then DR6 will record both the in-enclave DRx match and the AEX (non-enclave) DRx match.
Quoted 7 lines
I suppose it's also worth considering
whether page faults from *inside* the enclave should result in SIGSEGV or result in a fixup. We certainly want page faults from the ENCLU instruction itself to get fixed up, but maybe we want most exceptions inside the enclave to work a bit differently. Of course, if we do this, we need to make sure that the semantics of returning from the signal handler are reasonable.
Hmm, I'm pretty sure any fault that is 100% in the domain of the enclave should result in fixup.
Here are a few use cases off the top of my head that would require the enclave's runtime to intercept the signal, either to reenter the enclave or to feed the fault into the enclave's handler:
- Handle EPC invalidation, e.g. due to VM migration, while a thread is in the enclave since the resulting #PF can occur inside the enclave.
- During enclave development, configure the runtime to call into the enclave on any exception so that the enclave can dump register state.
- Implement copy-on-write or lazy allocation using SGX2 instructions, which would require feeding the #PF back into the enclave. Purely theoretical AFAIK, but lazy allocation in particular could be interesting, e.g. don't allocate .bss pages at startup time.
- An enclave and its runtime might feed #UDs back into the enclave, e.g. to run an unmodified binary in an enclave by wrapping it in a shim of sorts.
Re: [RFC PATCH v1 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
In reply to Cedric XingQuoted 20 lines · Cedric Xing
The current proposed
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.This patchset introduces a new ABI for
__vdso_sgx_enter_enclave()to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new__vdso_sgx_enter_enclave()will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null),__vdso_sgx_enter_enclave()will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).
Quoted 3 lines
The SGX selftest is augmented to test out the new callback interface, and to serve as a simple example to showcase how to use the callback interface in practice.
Please wrap your emails, or use an editor that will do it for you. 75 chars, plus or minus a few, is generally preferred.
Quoted 18 lines
Cedric Xing (3):
selftests/x86: FixedMakefilefor SGX selftest
x86/vdso: Modify__vdso_sgx_enter_enclave()to allow parameter passing
on untrusted stack
selftests/x86: Augment SGX selftest to test new
__vdso_sgx_enter_enclave()and its callback interfacearch/x86/entry/vdso/vsgx_enter_enclave.S | 156 ++++++++++++--------- arch/x86/include/uapi/asm/sgx.h | 14 +- tools/testing/selftests/x86/Makefile | 12 +- tools/testing/selftests/x86/sgx/Makefile | 45 +++--- tools/testing/selftests/x86/sgx/main.c | 123 +++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 +++++- 6 files changed, 264 insertions(+), 126 deletions(-)Signature · 1 line
2.17.1
v2 cover An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
In reply to Cedric XingThe current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.
This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new __vdso_sgx_enter_enclave() will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null), __vdso_sgx_enter_enclave() will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).
The SGX selftest is augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in __vdso_sgx_enter_enclave() by single-stepping through it and unwinding call stack at every instruction.
v2:
- Revised comments in __vdso_sgx_enter_enclave(). See patch 2/3.
- Added stack unwind test. See patch 3/3.
v1: https://lkml.org/lkml/2019/4/22/871
Note: This patchset is based upon SGX1 patch v20
(https://lkml.org/lkml/2019/4/17/344) by Jarkko Sakkinen
Cedric Xing (3):
selftests/x86: Fixed Makefile for SGX selftest
x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing
on untrusted stack
selftests/x86: Augment SGX selftest to test new
__vdso_sgx_enter_enclave() and its callback interface
arch/x86/entry/vdso/vsgx_enter_enclave.S | 175 +++++++---- arch/x86/include/uapi/asm/sgx.h | 14 +- tools/testing/selftests/x86/Makefile | 12 +- tools/testing/selftests/x86/sgx/Makefile | 49 ++-- tools/testing/selftests/x86/sgx/main.c | 323 ++++++++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 ++- 6 files changed, 471 insertions(+), 142 deletions(-)
Signature · 1 line
2.17.1
RFC PATCH v2 1/3 selftests/x86: Fixed Makefile for SGX selftest
The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.
--- tools/testing/selftests/x86/Makefile | 12 +++---- tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++--------------- 2 files changed, 22 insertions(+), 35 deletions(-)
tools/testing/selftests/x86/Makefile +6 -6
@@ -70,11 +70,11 @@ all_32: $(BINARIES_32) all_64: $(BINARIES_64) -all_64: $(SUBDIRS_64) - @for DIR in $(SUBDIRS_64); do \ - BUILD_TARGET=$(OUTPUT)/$$DIR; \ - mkdir $$BUILD_TARGET -p; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ +all_64: | $(SUBDIRS_64) + @for DIR in $|; do \ + BUILD_TARGET=$(OUTPUT)/$$DIR; \ + mkdir $$BUILD_TARGET -p; \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ done EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)
@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) all: warn_32bit_failure warn_32bit_failure: - @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \ + @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \ echo "environment. This will reduce test coverage of 64-bit" 2>&1; \ echo "kernels. If you are using a Debian-like distribution," 2>&1; \ echo "try:"; 2>&1; \
tools/testing/selftests/x86/sgx/Makefile +16 -29
63 lines not on this page · raw message
Signature · 1 line
2.17.1
Re: [RFC PATCH v2 1/3] selftests/x86: Fixed Makefile for SGX selftest
Quoted 18 lines · Cedric Xing
The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.
Signed-off-by Cedric Xing <[email protected]>--- tools/testing/selftests/x86/Makefile | 12 +++---- tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++--------------- 2 files changed, 22 insertions(+), 35 deletions(-)tools/testing/selftests/x86/Makefile +0
-0@@ -70,11 +70,11 @@ all_32: $(BINARIES_32) all_64: $(BINARIES_64)
tools/testing/selftests/x86/Makefile all_32: $(BINARIES_32) · line 77
Quoted 10 lines
tools/testing/selftests/x86/Makefile +5
-5-all_64: $(SUBDIRS_64) - @for DIR in $(SUBDIRS_64); do \ - BUILD_TARGET=$(OUTPUT)/$$DIR; \ - mkdir $$BUILD_TARGET -p; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ +all_64: | $(SUBDIRS_64) + @for DIR in $|; do \ + BUILD_TARGET=$(OUTPUT)/$$DIR; \ + mkdir $$BUILD_TARGET -p; \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \
This is not fix for anything. It is change in semantics. This diff should be isolated to its own commit as you are changing something outside of SGX scope.
tools/testing/selftests/x86/Makefile ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) · line 93
Quoted 9 lines
tools/testing/selftests/x86/Makefile +1
-1done EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) all: warn_32bit_failure warn_32bit_failure: - @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \ + @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \
Please clean this up.
Quoted 74 lines
tools/testing/selftests/x86/Makefile +0
-0echo "environment. This will reduce test coverage of 64-bit" 2>&1; \ echo "kernels. If you are using a Debian-like distribution," 2>&1; \ echo "try:"; 2>&1; \tools/testing/selftests/x86/sgx/Makefile +16
-29@@ -2,47 +2,34 @@ top_srcdir = ../../../../.. include ../../lib.mk -HOST_CFLAGS := -Wall -Werror -g $(INCLUDES) -fPIC -ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIC \ +ifeq ($(shell $(CC) -dumpmachine | cut --delimiter=- -f1),x86_64) +all: all_64 +endif + +HOST_CFLAGS := -Wall -Werror -g $(INCLUDES) +ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIE \ -fno-stack-protector -mrdrnd $(INCLUDES) TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx all_64: $(TEST_CUSTOM_PROGS) -$(TEST_CUSTOM_PROGS): $(OUTPUT)/main.o $(OUTPUT)/sgx_call.o \ - $(OUTPUT)/encl_piggy.o +$(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o $(CC) $(HOST_CFLAGS) -o $@ $^ -$(OUTPUT)/main.o: main.c - $(CC) $(HOST_CFLAGS) -c $< -o $@ - -$(OUTPUT)/sgx_call.o: sgx_call.S - $(CC) $(HOST_CFLAGS) -c $< -o $@ - -$(OUTPUT)/encl_piggy.o: $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss - $(CC) $(HOST_CFLAGS) -c encl_piggy.S -o $@ +$(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss + $(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@ -$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf $(OUTPUT)/sgxsign +$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf objcopy --remove-section=.got.plt -O binary $< $@ -$(OUTPUT)/encl.elf: $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o - $(CC) $(ENCL_CFLAGS) -T encl.lds -o $@ $^ +$(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S + $(CC) $(ENCL_CFLAGS) -T $^ -o $@ -$(OUTPUT)/encl.o: encl.c - $(CC) $(ENCL_CFLAGS) -c $< -o $@ - -$(OUTPUT)/encl_bootstrap.o: encl_bootstrap.S - $(CC) $(ENCL_CFLAGS) -c $< -o $@ - -$(OUTPUT)/encl.ss: $(OUTPUT)/encl.bin $(OUTPUT)/sgxsign - $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss +$(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin + $^ $@ $(OUTPUT)/sgxsign: sgxsign.c $(CC) -o $@ $< -lcrypto -EXTRA_CLEAN := $(OUTPUT)/sgx-selftest $(OUTPUT)/sgx-selftest.o \ - $(OUTPUT)/sgx_call.o $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss \ - $(OUTPUT)/encl.elf $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o \ - $(OUTPUT)/sgxsign - -.PHONY: clean +EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(addprefix $(OUTPUT)/, \ + encl.elf encl.bin encl.ss encl_piggy.o sgxsign)Signature · 1 line
2.17.1
What are all these changes to the makefile? I don't see mention of them in the commit message.
/Jarkko
Re: [RFC PATCH v2 1/3] selftests/x86: Fixed Makefile for SGX selftest
Quoted 19 lines · Jarkko Sakkinen
Quoted 18 lines · Cedric Xing
The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.
Signed-off-by Cedric Xing <[email protected]>--- tools/testing/selftests/x86/Makefile | 12 +++---- tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++--------------- 2 files changed, 22 insertions(+), 35 deletions(-)tools/testing/selftests/x86/Makefile +0
-0@@ -70,11 +70,11 @@ all_32: $(BINARIES_32) all_64: $(BINARIES_64)
Quoted 14 lines
tools/testing/selftests/x86/Makefile all_32: $(BINARIES_32) · line 77
Quoted 10 lines
tools/testing/selftests/x86/Makefile +5
-5-all_64: $(SUBDIRS_64) - @for DIR in $(SUBDIRS_64); do \ - BUILD_TARGET=$(OUTPUT)/$$DIR; \ - mkdir $$BUILD_TARGET -p; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ +all_64: | $(SUBDIRS_64) + @for DIR in $|; do \ + BUILD_TARGET=$(OUTPUT)/$$DIR; \ + mkdir $$BUILD_TARGET -p; \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \This is not fix for anything. It is change in semantics. This diff should be isolated to its own commit as you are changing something outside of SGX scope.
I have removed all changes to x86/Makefile from v4.
The current x86/Makefile only builds but cannot run sgx selftest. I don't see it as an urgent issue though.
Quoted 75 lines
tools/testing/selftests/x86/Makefile ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) · line 93
Quoted 9 lines
tools/testing/selftests/x86/Makefile +1
-1done EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) all: warn_32bit_failure warn_32bit_failure: - @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \ + @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \Please clean this up.
Quoted 63 lines
tools/testing/selftests/x86/Makefile +0
-0echo "environment. This will reduce test coverage of 64-bit" 2>&1; \ echo "kernels. If you are using a Debian-like distribution," 2>&1; \ echo "try:"; 2>&1; \tools/testing/selftests/x86/sgx/Makefile +14
-23@@ -2,47 +2,34 @@ top_srcdir = ../../../../.. include ../../lib.mk -HOST_CFLAGS := -Wall -Werror -g $(INCLUDES) -fPIC -ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIC \ +ifeq ($(shell $(CC) -dumpmachine | cut --delimiter=- -f1),x86_64) +all: all_64 +endif + +HOST_CFLAGS := -Wall -Werror -g $(INCLUDES) +ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIE \ -fno-stack-protector -mrdrnd $(INCLUDES) TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx all_64: $(TEST_CUSTOM_PROGS) -$(TEST_CUSTOM_PROGS): $(OUTPUT)/main.o $(OUTPUT)/sgx_call.o \ - $(OUTPUT)/encl_piggy.o +$(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o $(CC) $(HOST_CFLAGS) -o $@ $^ -$(OUTPUT)/main.o: main.c - $(CC) $(HOST_CFLAGS) -c $< -o $@ - -$(OUTPUT)/sgx_call.o: sgx_call.S - $(CC) $(HOST_CFLAGS) -c $< -o $@ - -$(OUTPUT)/encl_piggy.o: $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss - $(CC) $(HOST_CFLAGS) -c encl_piggy.S -o $@ +$(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss + $(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@ -$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf $(OUTPUT)/sgxsign +$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf objcopy --remove-section=.got.plt -O binary $< $@ -$(OUTPUT)/encl.elf: $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o - $(CC) $(ENCL_CFLAGS) -T encl.lds -o $@ $^ +$(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S + $(CC) $(ENCL_CFLAGS) -T $^ -o $@ -$(OUTPUT)/encl.o: encl.c - $(CC) $(ENCL_CFLAGS) -c $< -o $@ - -$(OUTPUT)/encl_bootstrap.o: encl_bootstrap.S - $(CC) $(ENCL_CFLAGS) -c $< -o $@ - -$(OUTPUT)/encl.ss: $(OUTPUT)/encl.bin $(OUTPUT)/sgxsign - $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss +$(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin + $^ $@ $(OUTPUT)/sgxsign: sgxsign.c $(CC) -o $@ $< -lcrypto
Quoted 14 lines
Quoted 11 lines
tools/testing/selftests/x86/sgx/Makefile +2
-6-EXTRA_CLEAN := $(OUTPUT)/sgx-selftest $(OUTPUT)/sgx-selftest.o \ - $(OUTPUT)/sgx_call.o $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss \ - $(OUTPUT)/encl.elf $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o \ - $(OUTPUT)/sgxsign - -.PHONY: clean +EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(addprefix $(OUTPUT)/, \ + encl.elf encl.bin encl.ss encl_piggy.o sgxsign)Signature · 1 line
2.17.1What are all these changes to the makefile? I don't see mention of them in the commit message.
Added description to the commit message in v4.
Quoted 2 lines
/Jarkko
RFC PATCH v2 2/3 x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
In reply to Cedric XingThe previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).
This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.
Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.
--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 175 ++++++++++++++++------- arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 128 insertions(+), 61 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +121 -54
@@ -19,83 +19,150 @@ * __vdso_sgx_enter_enclave() - Enter an SGX enclave * * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer + * @tcs: **IN 0x08(\%rsp)** - TCS, must be non-NULL + * @ex_info: **IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo' + * pointer + * @callback: **IN 0x18(\%rsp)** - Optional callback function to be called on + * enclave exit or exception * * Return: * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is - * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults + * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not + * allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value + * returned from ``callback`` (if one is supplied). * * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the - * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, - * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with - * the return value passed via ``%eax``. All registers except ``%rsp`` must - * be treated as volatile from the caller's perspective, including but not - * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave - * being run **must** preserve the untrusted ``%rsp`` and stack. + * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, + * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and + * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other + * registers will be passed through to the enclave as is. All registers except + * ``%rbp`` must be treated as volatile from the caller's perspective, including + * but not limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the + * enclave being run **must** preserve the untrusted ``%rbp``. + * + * ``callback`` has the following signature: + * int callback(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *exinfo, long r8, long r9, + * void *tcs, long ursp); + * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``, + * ``%rbx`` and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``, + * ``%rdx``, ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave + * exited/excepted, can be accessed directly as input parameters, while other + * GPRs can be accessed in assembly if needed. A positive value returned from + * ``callback`` will be treated as an ENCLU leaf (e.g. EENTER/ERESUME) to + * reenter the enclave (without popping the extra data pushed by the enclave off + * the stack), while 0 (zero) or a negative return value will be passed back to + * the caller of __vdso_sgx_enter_enclave(). It is also safe to leave + * ``callback`` via ``longjmp()`` or by throwing a C++ exception. */ -__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) +typedef int (*sgx_callback)(long rdi, long rsi, long rdx, + struct sgx_enclave_exinfo *exinfo, long r8, + long r9, void *tcs, long ursp); +int __vdso_sgx_enter_enclave(int leaf, void *tcs, + struct sgx_enclave_exinfo *exinfo, + sgx_callback callback) { - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; - - if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e) - *e = exception; - return -EFAULT; + while (leaf == EENTER || leaf == ERESUME) { + int rc; + try { + ENCLU[leaf]; + rc = 0; + if (exinfo) + exinfo->leaf = EEXIT; + } catch (exception) { + rc = -EFAULT; + if (exinfo) + *exinfo = exception; + } + + leaf = callback ? (*callback)( + rdi, rsi, rdx, exinfo, r8, r9, tcs, ursp) : rc; } - return 0; + return leaf > 0 ? -EINVAL : leaf; } #endif ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax - jb bad_input - + jb 6f cmp $0x3, %eax - ja bad_input + ja 6f - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx - /* Save @exception_info */ - push %rcx + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx + jrcxz 4f + mov %eax, EX_LEAF(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) + mov %rdx, EX_ADDRESS(%rcx) - add $0x8, %rsp - xor %eax, %eax +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %rax + /* At this point, %ebx holds the effective return value, which shall be + * returned if no callback is specified */ + cmovz %rbx, %rax + jz 7f + /* Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be + * restored after *callback returns. */ + mov %rsp, %rbx + and $-0x10, %rsp + /* Clear RFLAGS.DF per x86_64 ABI */ + cld + /* Parameters for *callback */ + push %rbx + push 0x10(%rbp) + /* Call *%rax via retpoline */ + call 40f + /* Restore %rsp to its original value left off by the enclave from last + * exit */ + mov %rbx, %rsp + /* Positive return value from *callback will be interpreted as an ENCLU + * leaf, while a non-positive value will be interpreted as the return + * value to be passed back to the caller. */ + jmp 1b +40: /* retpoline */ + call 42f +41: pause + lfence + jmp 41b +42: mov %rax, (%rsp) ret -bad_input: - mov $(-EINVAL), %rax - ret +5: /* Exception path */ + mov $-EFAULT, %ebx + stc + jmp 3b -.pushsection .fixup, "ax" - /* Re-load @exception_info and fill it (if it's non-NULL) */ -2: pop %rcx - test %rcx, %rcx - je 3f +6: /* Unsupported ENCLU leaf */ + cmp $0, %eax + jle 7f + mov $-EINVAL, %eax - mov %eax, EX_LEAF(%rcx) - mov %di, EX_TRAPNR(%rcx) - mov %si, EX_ERROR_CODE(%rcx) - mov %rdx, EX_ADDRESS(%rcx) -3: mov $(-EFAULT), %rax +7: /* Epilog */ + leave + .cfi_def_cfa %rsp, 8 ret -.popsection + .cfi_endproc -_ASM_VDSO_EXTABLE_HANDLE(1b, 2b) +_ASM_VDSO_EXTABLE_HANDLE(2b, 5b) ENDPROC(__vdso_sgx_enter_enclave)
arch/x86/include/uapi/asm/sgx.h +7 -7
@@ -24,7 +24,7 @@/** * struct sgx_enclave_create - parameter structure for the -* %SGX_IOC_ENCLAVE_CREATE ioctl +* %SGX_IOC_ENCLAVE_CREATE ioctl * @src: address for the SECS page data */ struct sgx_enclave_create {
@@ -33,7 +33,7 @@ struct sgx_enclave_create {/** * struct sgx_enclave_add_page - parameter structure for the -* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl +* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl * @addr: address within the ELRANGE * @src: address for the page data * @secinfo: address for the SECINFO data
@@ -49,7 +49,7 @@ struct sgx_enclave_add_page {/** * struct sgx_enclave_init - parameter structure for the -* %SGX_IOC_ENCLAVE_INIT ioctl +* %SGX_IOC_ENCLAVE_INIT ioctl * @sigstruct: address for the SIGSTRUCT data */ struct sgx_enclave_init {
@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute { };/** -* struct sgx_enclave_exception - structure to report exceptions encountered in -* __vdso_sgx_enter_enclave() +* struct sgx_enclave_exinfo - structure to report exceptions encountered in +* __vdso_sgx_enter_enclave() * -* @leaf: ENCLU leaf from \%eax at time of exception +* @leaf: ENCLU leaf from \%eax at time of exception/exit * @trapnr: exception trap number, a.k.a. fault vector * @error_code: exception error code * @address: exception address, e.g. CR2 on a #PF * @reserved: reserved for future use */ -struct sgx_enclave_exception { +struct sgx_enclave_exinfo {__u32 leaf;__u16 trapnr;__u16 error_code;
Signature · 1 line
2.17.1
Re: [RFC PATCH v2 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 4 lines · Cedric Xing
The previous
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls).
Quoted 10 lines
This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then
__vdso_sgx_enter_enclave()can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after__vdso_sgx_enter_enclave()returns. Therefore,__vdso_sgx_enter_enclave()has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.
AIUI, you and Andy had an off-list discussion regarding rewriting __vdso_sgx_enter_enclave() vs. providing a second vDSO function. Can you please summarize the discussion so that it's clear why you're pursuing a single function?
In the end I probably agree that's it's desirable to have a single ABI and vDSO function since the cost is basically that the non-callback case needs to pass params via the stack instead of registers, but I do miss the simplicity of separate implementations.
Quoted 50 lines
Here's the summary of API/ABI changes in this patch. More details could be found in
arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'structsgx_enclave_exception' is renamed to 'structsgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
*__vdso_sgx_enter_enclave()anchors its frame using %rbp (instead of %rsp in the previous implementation).
*__vdso_sgx_enter_enclave()takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause__vdso_sgx_enter_enclave()to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of__vdso_sgx_enter_enclave()to its caller. It's also safe to leave callback bylongjmp()or by throwing a C++ exception.Signed-off-by Cedric Xing <[email protected]>--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 175 ++++++++++++++++------- arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 128 insertions(+), 61 deletions(-)arch/x86/entry/vdso/vsgx_enter_enclave.S +8
-5@@ -19,83 +19,150 @@ * __vdso_sgx_enter_enclave() - Enter an SGX enclave * * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer + * @tcs: **IN 0x08(\%rsp)** - TCS, must be non-NULL + * @ex_info: **IN 0x10(\%rsp)** - Optional 'struct sgx_enclave_exinfo' + * pointer + * @callback: **IN 0x18(\%rsp)** - Optional callback function to be called on + * enclave exit or exception * * Return: * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is - * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults + * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is not + * allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value + * returned from ``callback`` (if one is supplied). * * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the - * x86-64 ABI, i.e. cannot be called from standard C code. As noted above,
arch/x86/entry/vdso/vsgx_enter_enclave.S line 39
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5
-5- * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with - * the return value passed via ``%eax``. All registers except ``%rsp`` must - * be treated as volatile from the caller's perspective, including but not - * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave - * being run **must** preserve the untrusted ``%rsp`` and stack. + * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, + * input parameters must be passed via ``%eax``, ``8(%rsp)``, ``0x10(%rsp)`` and + * ``0x18(%rsp)``, with the return value passed via ``%eax``. All other + * registers will be passed through to the enclave as is. All registers except + * ``%rbp`` must be treated as volatile from the caller's perspective, including
Hmm, this is my fault since I wrote the original comment, but stating "All registers except %rbp must be treated as volatile" is confusing, e.g. at first I thought it meant that the assembly blob was mucking with the registers and that they couldn't be used to pass information out of the enclave.
Maybe something like:
* Register ABI:
* - As noted above, input parameters are passed via %eax and the stack. * - The return value is passed via %eax. * - %rbx and %rcx must be treated as volatile as they are modified as part of
* enclaves transitions and are used as scratch regs.
* - %rbp **must** be preserved by the enclave in all cases, as it is used to
* reference paramaters when handling enclave exits.
* - %rdx, %rdi, %rsi and %r8-%r15 may be freely modified by the enclave
* and will not be passed back to the caller untouched.
* - %rsp is saved/restored across __vdso_sgx_enter_enclave(), but is passed
* as-is to the callback if one is provided, i.e. the enclave may use u_rsp
* to pass information to its associated callback.
*
* Callback ABI:
* <more information about callback ABI>
*
Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +8
-0+ * but not limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the + * enclave being run **must** preserve the untrusted ``%rbp``. + * + * ``callback`` has the following signature: + * int callback(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *exinfo, long r8, long r9, + * void *tcs, long ursp); + * ``callback`` **shall** follow x86_64 ABI. All GPRs **except** ``%rax``,
arch/x86/entry/vdso/vsgx_enter_enclave.S line 57
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +9
-0+ * ``%rbx`` and ``rcx`` are passed through to ``callback``. ``%rdi``, ``%rsi``, + * ``%rdx``, ``%r8``, ``%r9``, along with the value of ``%rsp`` when the enclave + * exited/excepted, can be accessed directly as input parameters, while other + * GPRs can be accessed in assembly if needed. A positive value returned from + * ``callback`` will be treated as an ENCLU leaf (e.g. EENTER/ERESUME) to + * reenter the enclave (without popping the extra data pushed by the enclave off + * the stack), while 0 (zero) or a negative return value will be passed back to + * the caller of __vdso_sgx_enter_enclave(). It is also safe to leave + * ``callback`` via ``longjmp()`` or by throwing a C++ exception. */
While this may format nicely in .rst (I haven't checked), it's damn near unreadable in its raw form. Escaping '%' in kernel-doc is unresolved at this point, but this definitely is an argument for allowing the %CONST interpretation to be disabled entirely.
longjmp() is probably the only thing that needs to be explicitly marked, I think it would be better to simply say "the callback" instead of using ``callback``, i.e. use regular English instead of referencing the param.
Quoted 12 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6
-5-__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) +typedef int (*sgx_callback)(long rdi, long rsi, long rdx, + struct sgx_enclave_exinfo *exinfo, long r8, + long r9, void *tcs, long ursp); +int __vdso_sgx_enter_enclave(int leaf, void *tcs, + struct sgx_enclave_exinfo *exinfo, + sgx_callback callback) { - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; -
arch/x86/entry/vdso/vsgx_enter_enclave.S line 65
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-9- if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e) - *e = exception; - return -EFAULT; + while (leaf == EENTER || leaf == ERESUME) {
This gives the impression that looping is the common case. I'd prefer using 'goto' to show that the common case is a simple fall through whereas the callback case can effectively loop on ENCLU.
Quoted 47 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +29
-9+ int rc; + try { + ENCLU[leaf]; + rc = 0; + if (exinfo) + exinfo->leaf = EEXIT; + } catch (exception) { + rc = -EFAULT; + if (exinfo) + *exinfo = exception; + } + + leaf = callback ? (*callback)( + rdi, rsi, rdx, exinfo, r8, r9, tcs, ursp) : rc; } - return 0; + return leaf > 0 ? -EINVAL : leaf; } #endif ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax - jb bad_input - + jb 6f cmp $0x3, %eax - ja bad_input + ja 6f - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx - /* Save @exception_info */
arch/x86/entry/vdso/vsgx_enter_enclave.S line 109
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5
-4- push %rcx + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx
@exinfo is optional, i.e. %ecx needs to be tested before its used.
arch/x86/entry/vdso/vsgx_enter_enclave.S line 110
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ jrcxz 4f
I dislike the number of flags and values that are stashed away only to be consumed later, it makes the code hard to follow. AFAICT, a lot of the shenanigans are done to reuse code because exinfo was overloaded, but that's actually pointless, i.e. it's unnecessarily complex.
Overlading the struct is pointless becase if you make it mandatory when using a callback then it can be nullified (when passed to the callback) to indicate EEXIT. If it's still optional, then the callback needs an extra param to explicitly indicate EEXIT vs. -EFAULT, otherwise the callback has no indication whatsover of why it was invoked.
My preference is for the latter since it's more explicit and obvious.
Tangetially related, I'm not opposed to renaming it 'struct sgx_enclave_exception_info' for clarity.
arch/x86/entry/vdso/vsgx_enter_enclave.S line 115
Quoted 5 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5
-0+ mov %eax, EX_LEAF(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) + mov %rdx, EX_ADDRESS(%rcx)
My strong preference would be to organize the code to separate the various flows, i.e. happy common case, invalid input, exception handler and callback invocation. And make the common case a straight shot so that it's more obvious what happens in the happy non-callback case.
arch/x86/entry/vdso/vsgx_enter_enclave.S line 121
Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5
-2- add $0x8, %rsp - xor %eax, %eax +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %rax + /* At this point, %ebx holds the effective return value, which shall be + * returned if no callback is specified */
Use kernel-doc format for multi-line comments. Missing punctionation at the end. Blank lines between logical blocks would also help readability.
E.g.:
4: /* Call *callback if supplied */
mov 0x20(%rbp), %rax
test %rax, %rax
/*
* At this point, %ebx holds the effective return value, which shall be
* returned if no callback is specified.
*/
cmovz %rbx, %rax
jz 7f
arch/x86/entry/vdso/vsgx_enter_enclave.S line 134
Quoted 13 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +13
-0+ cmovz %rbx, %rax + jz 7f + /* Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be + * restored after *callback returns. */ + mov %rsp, %rbx + and $-0x10, %rsp + /* Clear RFLAGS.DF per x86_64 ABI */ + cld + /* Parameters for *callback */ + push %rbx + push 0x10(%rbp) + /* Call *%rax via retpoline */ + call 40f
Another case of stashing a value it consuming it later. This one is especially brutal since %rax was loaded with CMOVcc, which means the reader needs to backtrack even further to understand what %rax contains at this point.
arch/x86/entry/vdso/vsgx_enter_enclave.S line 140
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6
-0+ /* Restore %rsp to its original value left off by the enclave from last + * exit */ + mov %rbx, %rsp + /* Positive return value from *callback will be interpreted as an ENCLU + * leaf, while a non-positive value will be interpreted as the return + * value to be passed back to the caller. */
Please use imperative mood in the comments, i.e. simply state what the code is doing. E.g. "will be interpreted" makes it sound like something else is processing the callback's return value.
Quoted 33 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +16
-13+ jmp 1b +40: /* retpoline */ + call 42f +41: pause + lfence + jmp 41b +42: mov %rax, (%rsp) ret -bad_input: - mov $(-EINVAL), %rax - ret +5: /* Exception path */ + mov $-EFAULT, %ebx + stc + jmp 3b -.pushsection .fixup, "ax" - /* Re-load @exception_info and fill it (if it's non-NULL) */ -2: pop %rcx - test %rcx, %rcx - je 3f +6: /* Unsupported ENCLU leaf */ + cmp $0, %eax + jle 7f + mov $-EINVAL, %eax - mov %eax, EX_LEAF(%rcx) - mov %di, EX_TRAPNR(%rcx) - mov %si, EX_ERROR_CODE(%rcx) - mov %rdx, EX_ADDRESS(%rcx) -3: mov $(-EFAULT), %rax +7: /* Epilog */
arch/x86/entry/vdso/vsgx_enter_enclave.S line 168
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4
-2+ leave + .cfi_def_cfa %rsp, 8 ret -.popsection + .cfi_endproc -_ASM_VDSO_EXTABLE_HANDLE(1b, 2b) +_ASM_VDSO_EXTABLE_HANDLE(2b, 5b) ENDPROC(__vdso_sgx_enter_enclave)
Putting everything together, sans comments, I'd prefer something like the following. Pasted in raw code as that'll hopefully be easier to review and discuss than a diff.
Note, swapping 'long rc' and '... *e' would allow the callback flow to save one memory access, but IMO the exception info belongs at the end since it's optional.
#ifdef SGX_KERNEL_DOC
typedef int (*sgx_callback)(long rdi, long rsi, long rdx, long ret, long r8,
long r9, void *tcs, long ursp,
struct sgx_enclave_exception_info *e);
int __vdso_sgx_enter_enclave(int leaf, void *tcs,
struct sgx_enclave_exception_info *e,
sgx_callback callback)
{
int ret;enter_enclave:
if (leaf != EENTER && leaf != ERESUME) {
ret = -EINVAL;
goto out;
}
try {
ENCLU[leaf];
ret = 0;
} catch (exception) {
ret = -EFAULT;
if (e)
*e = exception;
}
if (callback)
goto do_callback;
out:
return ret;do_callback:
leaf = (*callback)(rdi, rsi, rdx, ret, r8, r9, e, tcs, ursp); if (leaf <= 0) goto out; goto enter_enclave; } #endif ENTRY(__vdso_sgx_enter_enclave) /* Prolog */ .cfi_startproc push %rbp .cfi_adjust_cfa_offset 8 .cfi_rel_offset %rbp, 0 mov %rsp, %rbp .cfi_def_cfa_register %rbp
1: /* EENTER <= leaf <= ERESUME */
cmp $0x2, %eax jb 5f cmp $0x3, %eax ja 5f /* Load TCS and AEP */ mov 0x10(%rbp), %rbx lea 2f(%rip), %rcx /* Single ENCLU serving as both EENTER and AEP (ERESUME) */
2: enclu
/* EEXIT branches here unless the enclave is doing something fancy. */ xor %eax, %eax
3: cmp $0, 0x20(%rbp)
jne 8f
4: leave
.cfi_def_cfa %rsp, 8 ret
5: mov $(-EINVAL), %rax
jmp 4b
6: mov 0x18(%rbp), %rcx
test %rcx, %rcx je 7f /* Fill optional exception info. */ mov %eax, EX_LEAF(%rcx) mov %di, EX_TRAPNR(%rcx) mov %si, EX_ERROR_CODE(%rcx) mov %rdx, EX_ADDRESS(%rcx) 7: mov $(-EFAULT), %rax jmp 3b
8: /*
* Align stack per x86_64 ABI. Save the original %rsp in %rbx to be * restored after the callback returns. */ mov %rsp, %rbx and $-0x10, %rsp /* Push @e, u_rsp and @tcs as parameters to the callback. */ push 0x18(%rbp) push %rbx push 0x10(%rbp) /* Pass the "return value" to the callback via %rcx. */ mov %rax, %rcx /* Clear RFLAGS.DF per x86_64 ABI */ cld /* Load the callback pointer to %rax and invoke it via retpoline. */ mov 0x20(%rbp), %rax call 40f /* Restore %rsp to its post-exit value. */ mov %rbx, %rsp /* * If the return from callback is zero or negative, return immediately, * otherwise attempt to re-execute ENCLU with the return interpreted as * the requested ENCLU leaf. */ cmp $0, %eax jle 4b jmp 1b
40: /* retpoline */
call 42f 41: pause lfence jmp 41b
42: mov %rax, (%rsp)
ret
.cfi_endproc
_ASM_VDSO_EXTABLE_HANDLE(2b, 6b)
ENDPROC(__vdso_sgx_enter_enclave)
RE: [RFC PATCH v2 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Hi Sean,
Quoted 6 lines
AIUI, you and Andy had an off-list discussion regarding rewriting
__vdso_sgx_enter_enclave()vs. providing a second vDSO function. Can you
please summarize the discussion so that it's clear why you're pursuing a single function?
Quoted 7 lines
In the end I probably agree that's it's desirable to have a single ABI and
vDSO function since the cost is basically that the non-callback case needs
to pass params via the stack instead of registers, but I do miss the simplicity of separate implementations.
The major reason is we don't want to maintain 2 different ABIs (preserving %rsp vs. %rbp). Given the new one covers all known use cases, we prefer not to keep the other one.
Quoted 112 lines
Quoted 1 line
Here's the summary of API/ABI changes in this patch. More details
could be
Quoted 2 lines
found in
arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'structsgx_enclave_exception' is renamed to 'struct
sgx_enclave_exinfo'Quoted 1 line
because it is filled upon both AEX (i.e. exceptions) and normal
enclave
Quoted 2 lines
exits.
*__vdso_sgx_enter_enclave()anchors its frame using %rbp (insteadof %rsp in
Quoted 2 lines
the previous implementation).
*__vdso_sgx_enter_enclave()takes one more parameter - a callbackfunction to
Quoted 2 lines
be invoked upon enclave exits. This callback is optional, and if not supplied, will cause
__vdso_sgx_enter_enclave()to return uponenclave exits
Quoted 2 lines
(same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp atenclave
Quoted 1 line
exit to address data "pushed" by the enclave. A positive value
returned by
Quoted 1 line
the callback will be treated as an ENCLU leaf for re-entering the
enclave,
Quoted 7 lines
while a zero or negative value will be passed through as the return value of
__vdso_sgx_enter_enclave()to its caller. It's also safe to leave callback bylongjmp()or by throwing a C++ exception.Signed-off-by Cedric Xing <[email protected]>--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 175 ++++++++++++++++-------
Review
Quoted 4 lines
arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 128 insertions(+), 61 deletions(-)(quoted diff) +0
-0
b/arch/x86/entry/vdso/vsgx_enter_enclave.Sarch/x86/entry/vdso/vsgx_enter_enclave.S line 23
Quoted 9 lines
index
fe0bf6671d6d..debecb0d785c100644arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-2@@ -19,83 +19,150 @@ * __vdso_sgx_enter_enclave() - Enter an SGX enclave * * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception'pointer
arch/x86/entry/vdso/vsgx_enter_enclave.S line 23
Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2
-0+ * @tcs: **IN 0x08(\%rsp)** - TCS, must be non-NULL + * @ex_info: **IN 0x10(\%rsp)** - Optional 'struct
sgx_enclave_exinfo'arch/x86/entry/vdso/vsgx_enter_enclave.S line 25
Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2
-0+ * pointer + * @callback: **IN 0x18(\%rsp)** - Optional callback function tobe called on
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 5 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-1+ * enclave exit or exception * * Return: * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLUleaf is
arch/x86/entry/vdso/vsgx_enter_enclave.S line 28
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-1- * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclavefaults
arch/x86/entry/vdso/vsgx_enter_enclave.S line 30
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLUleaf is not
arch/x86/entry/vdso/vsgx_enter_enclave.S line 31
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * allowed, %-EFAULT if ENCLU or the enclave faults, or a non-positive value
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * returned from ``callback`` (if one is supplied). * * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliantwith the
arch/x86/entry/vdso/vsgx_enter_enclave.S line 31
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-1- * x86-64 ABI, i.e. cannot be called from standard C code. As notedabove,
arch/x86/entry/vdso/vsgx_enter_enclave.S line 32
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-1- * input parameters must be passed via ``%eax``, ``%rbx`` and``%rcx``, with
arch/x86/entry/vdso/vsgx_enter_enclave.S line 33
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-1- * the return value passed via ``%eax``. All registers except``%rsp`` must
arch/x86/entry/vdso/vsgx_enter_enclave.S line 34
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-1- * be treated as volatile from the caller's perspective, includingbut not
arch/x86/entry/vdso/vsgx_enter_enclave.S line 35
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-1- * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, theenclave
arch/x86/entry/vdso/vsgx_enter_enclave.S line 35
Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-1- * being run **must** preserve the untrusted ``%rsp`` and stack. + * x86-64 ABI, i.e. cannot be called from standard C code. As notedabove,
arch/x86/entry/vdso/vsgx_enter_enclave.S line 36
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * input parameters must be passed via ``%eax``, ``8(%rsp)``,``0x10(%rsp)`` and
arch/x86/entry/vdso/vsgx_enter_enclave.S line 37
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * ``0x18(%rsp)``, with the return value passed via ``%eax``. Allother
arch/x86/entry/vdso/vsgx_enter_enclave.S line 38
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * registers will be passed through to the enclave as is. Allregisters except
arch/x86/entry/vdso/vsgx_enter_enclave.S line 39
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * ``%rbp`` must be treated as volatile from the caller's perspective,including
Hmm, this is my fault since I wrote the original comment, but stating "All registers except %rbp must be treated as volatile" is confusing, e.g.
at first I thought it meant that the assembly blob was mucking with the registers and that they couldn't be used to pass information out of the enclave.Maybe something like:
* Register ABI:
* - As noted above, input parameters are passed via %eax and the stack. * - The return value is passed via %eax. * - %rbx and %rcx must be treated as volatile as they are modified aspart of
* enclaves transitions and are used as scratch regs.
* - %rbp **must** be preserved by the enclave in all cases, as it is
used to
* reference paramaters when handling enclave exits.
* - %rdx, %rdi, %rsi and %r8-%r15 may be freely modified by the
enclave
* and will not be passed back to the caller untouched.
* - %rsp is saved/restored across__vdso_sgx_enter_enclave(), but is
Quoted 8 lines
passed
* as-is to the callback if one is provided, i.e. the enclave may useu_rsp
* to pass information to its associated callback.
*
* Callback ABI:
* <more information about callback ABI>
*
Really appreciated! I'll update the comments in the next revision.
Quoted 25 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 40
Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * but not limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely,the
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +7
-0+ * enclave being run **must** preserve the untrusted ``%rbp``. + * + * ``callback`` has the following signature: + * int callback(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *exinfo, long r8, long r9, + * void *tcs, long ursp); + * ``callback`` **shall** follow x86_64 ABI. All GPRs **except**``%rax``,
arch/x86/entry/vdso/vsgx_enter_enclave.S line 48
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * ``%rbx`` and ``rcx`` are passed through to ``callback``. ``%rdi``,``%rsi``,
arch/x86/entry/vdso/vsgx_enter_enclave.S line 49
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * ``%rdx``, ``%r8``, ``%r9``, along with the value of ``%rsp`` whenthe enclave
arch/x86/entry/vdso/vsgx_enter_enclave.S line 50
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * exited/excepted, can be accessed directly as input parameters,while other
arch/x86/entry/vdso/vsgx_enter_enclave.S line 51
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * GPRs can be accessed in assembly if needed. A positive valuereturned from
arch/x86/entry/vdso/vsgx_enter_enclave.S line 52
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * ``callback`` will be treated as an ENCLU leaf (e.g. EENTER/ERESUME)to
arch/x86/entry/vdso/vsgx_enter_enclave.S line 53
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * reenter the enclave (without popping the extra data pushed by theenclave off
arch/x86/entry/vdso/vsgx_enter_enclave.S line 54
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * the stack), while 0 (zero) or a negative return value will bepassed back to
Quoted 9 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 57
Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +2
-0+ * the caller of __vdso_sgx_enter_enclave(). It is also safe to leave + * ``callback`` via ``longjmp()`` or by throwing a C++ exception. */While this may format nicely in .rst (I haven't checked), it's damn near unreadable in its raw form. Escaping '%' in kernel-doc is unresolved at this point, but this definitely is an argument for allowing the %CONST interpretation to be disabled entirely.
I know very little about kernel-doc. Not quite sure which markup means what. Or if you don't mind, could you just write up the whole thing (you have written half of it in your email already) so I can include it into my next revision?
Quoted 4 lines
longjmp()is probably the only thing that needs to be explicitly marked, I think it would be better to simply say "the callback" instead of using ``callback``, i.e. use regular English instead of referencing the param.
Are you suggesting not to mention C++ exception here?
Quoted 19 lines
Quoted 19 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6
-12-__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) +typedef int (*sgx_callback)(long rdi, long rsi, long rdx, + struct sgx_enclave_exinfo *exinfo, long r8, + long r9, void *tcs, long ursp); +int __vdso_sgx_enter_enclave(int leaf, void *tcs, + struct sgx_enclave_exinfo *exinfo, + sgx_callback callback) { - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; - - if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e)
Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 65
Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-2- *e = exception; - return -EFAULT; + while (leaf == EENTER || leaf == ERESUME) {This gives the impression that looping is the common case. I'd prefer using 'goto' to show that the common case is a simple fall through whereas
the callback case can effectively loop on ENCLU.
Looping IMO is indeed the common case. Just think of the case of an enclave making a sequence of o-calls.
Quoted 61 lines
Quoted 48 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +29
-9+ int rc; + try { + ENCLU[leaf]; + rc = 0; + if (exinfo) + exinfo->leaf = EEXIT; + } catch (exception) { + rc = -EFAULT; + if (exinfo) + *exinfo = exception; + } + + leaf = callback ? (*callback)( + rdi, rsi, rdx, exinfo, r8, r9, tcs, ursp) : rc; } - return 0; + return leaf > 0 ? -EINVAL : leaf; } #endif ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax - jb bad_input - + jb 6f cmp $0x3, %eax - ja bad_input + ja 6f - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx - /* Save @exception_info */arch/x86/entry/vdso/vsgx_enter_enclave.S line 109
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5
-4- push %rcx + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx@exinfo is optional, i.e. %ecx needs to be tested before its used.
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S line 110
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ jrcxz 4f
The above instruction (i.e. jrcxz) does test %rcx.
Quoted 6 lines
I dislike the number of flags and values that are stashed away only to be consumed later, it makes the code hard to follow. AFAICT, a lot of the shenanigans are done to reuse code because exinfo was overloaded, but that's actually pointless, i.e. it's unnecessarily complex.
Quoted 7 lines
Overlading the struct is pointless becase if you make it mandatory when using a callback then it can be nullified (when passed to the callback) to indicate EEXIT. If it's still optional, then the callback needs an extra param to explicitly indicate EEXIT vs.
-EFAULT, otherwise the callback has no indication whatsover of why it was invoked.My preference is for the latter since it's more explicit and obvious.
I wouldn't nullify @exinfo at EEXIT because that could be used as a "context" pointer to the callback.
Making it optionally because the callback can still use other registers (e.g. %rdi) determine the reason without it. For example, an enclave may set %rdi to non-zero to signify o-call/e-ret while zero (as set by AEX) would indicate an exception. My intention is not to impose unnecessary restrictions on applications.
Quoted 3 lines
Tangetially related, I'm not opposed to renaming it 'struct
sgx_enclave_exception_info' for clarity.
I'd still prefer sgx_enclave_exinfo to imply its use in both _ex_it and _ex_ception cases, for the reason stated above.
Quoted 12 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 115
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5
-0+ mov %eax, EX_LEAF(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) + mov %rdx, EX_ADDRESS(%rcx)My strong preference would be to organize the code to separate the various
flows, i.e. happy common case, invalid input, exception handler and callback invocation. And make the common case a straight shot so that it's more obvious what happens in the happy non-callback case.
I'd prefer more concise code. After all, this is assembly without optimization done by the compiler. If it were for C, I'd agree with you because either case would end up in roughly the same code (in terms of code size or efficiency) due to the optimizer.
Quoted 19 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4
-2- add $0x8, %rsp - xor %eax, %eax +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %rax + /* At this point, %ebx holds the effective return value, whichshall be
arch/x86/entry/vdso/vsgx_enter_enclave.S line 121
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * returned if no callback is specified */Use kernel-doc format for multi-line comments. Missing punctionation at the end. Blank lines between logical blocks would also help readability.
E.g.:
4: /* Call *callback if supplied */
mov 0x20(%rbp), %rax
test %rax, %rax
Quoted 7 lines
/*
* At this point, %ebx holds the effective return value, which
shall be
* returned if no callback is specified.
*/
cmovz %rbx, %rax
jz 7f
I know very little about kernel-doc. Would you please point me to documents describing it? What you suggested here will of course be incorporated into my next revision.
Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +3
-0+ cmovz %rbx, %rax + jz 7f + /* Align stack per x86_64 ABI. The original %rsp is saved in %rbxto be
Quoted 15 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 134
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +10
-0+ * restored after *callback returns. */ + mov %rsp, %rbx + and $-0x10, %rsp + /* Clear RFLAGS.DF per x86_64 ABI */ + cld + /* Parameters for *callback */ + push %rbx + push 0x10(%rbp) + /* Call *%rax via retpoline */ + call 40fAnother case of stashing a value it consuming it later. This one is especially brutal since %rax was loaded with CMOVcc, which means the reader needs to backtrack even further to understand what %rax contains at this point.
%rax was loaded only 2 instructions ago, while what %rbx is containing is described in the comment immediately above it. I think I have done my best to make it readable. After all, this is assembly so can't expect something like giving proper names to variables in C because that just can't be done in assembly.
Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 134
Quoted 2 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ /* Restore %rsp to its original value left off by the enclave fromlast
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +3
-0+ * exit */ + mov %rbx, %rsp + /* Positive return value from *callback will be interpreted as anENCLU
Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 138
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * leaf, while a non-positive value will be interpreted as thereturn
arch/x86/entry/vdso/vsgx_enter_enclave.S line 140
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * value to be passed back to the caller. */Please use imperative mood in the comments, i.e. simply state what the code is doing. E.g. "will be interpreted" makes it sound like something else is processing the callback's return value.
Will do in the next revision.
Quoted 49 lines
Quoted 33 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +16
-13+ jmp 1b +40: /* retpoline */ + call 42f +41: pause + lfence + jmp 41b +42: mov %rax, (%rsp) ret -bad_input: - mov $(-EINVAL), %rax - ret +5: /* Exception path */ + mov $-EFAULT, %ebx + stc + jmp 3b -.pushsection .fixup, "ax" - /* Re-load @exception_info and fill it (if it's non-NULL) */ -2: pop %rcx - test %rcx, %rcx - je 3f +6: /* Unsupported ENCLU leaf */ + cmp $0, %eax + jle 7f + mov $-EINVAL, %eax - mov %eax, EX_LEAF(%rcx) - mov %di, EX_TRAPNR(%rcx) - mov %si, EX_ERROR_CODE(%rcx) - mov %rdx, EX_ADDRESS(%rcx) -3: mov $(-EFAULT), %rax +7: /* Epilog */arch/x86/entry/vdso/vsgx_enter_enclave.S line 168
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4
-2+ leave + .cfi_def_cfa %rsp, 8 ret -.popsection + .cfi_endproc -_ASM_VDSO_EXTABLE_HANDLE(1b, 2b) +_ASM_VDSO_EXTABLE_HANDLE(2b, 5b) ENDPROC(__vdso_sgx_enter_enclave)Putting everything together, sans comments, I'd prefer something like the
following. Pasted in raw code as that'll hopefully be easier to review and discuss than a diff.
Quoted 3 lines
Note, swapping 'long rc' and '... *e' would allow the callback flow to save one memory access, but IMO the exception info belongs at the end since it's optional.
You have probably misunderstood my code. 'ret' is _not_ passed while @exinfo is passed in %rcx. exinfo->leaf implies the reason of the callback.
Or are you suggesting to add 'ret' as one more parameter to the callback? I don't think it necessary because exinfo contains everything the callback would ever need.
Quoted 128 lines
#ifdef
SGX_KERNEL_DOC
typedef int (*sgx_callback)(long rdi, long rsi, long rdx, long ret, long r8,long r9, void *tcs, long ursp, struct sgx_enclave_exception_info *e); int __vdso_sgx_enter_enclave(int leaf, void *tcs, struct sgx_enclave_exception_info *e, sgx_callback callback) { int ret;
enter_enclave:if (leaf != EENTER && leaf != ERESUME) { ret = -EINVAL; goto out; } try { ENCLU[leaf]; ret = 0; } catch (exception) { ret = -EFAULT; if (e) *e = exception; } if (callback) goto do_callback; out: return ret;
do_callback:leaf = (*callback)(rdi, rsi, rdx, ret, r8, r9, e, tcs, ursp); if (leaf <= 0) goto out; goto enter_enclave; } #endif ENTRY(__vdso_sgx_enter_enclave) /* Prolog */ .cfi_startproc push %rbp .cfi_adjust_cfa_offset 8 .cfi_rel_offset %rbp, 0 mov %rsp, %rbp .cfi_def_cfa_register %rbp1: /* EENTER <= leaf <= ERESUME */
cmp $0x2, %eax jb 5f cmp $0x3, %eax ja 5f /* Load TCS and AEP */ mov 0x10(%rbp), %rbx lea 2f(%rip), %rcx /* Single ENCLU serving as both EENTER and AEP (ERESUME) */2: enclu
/* EEXIT branches here unless the enclave is doing something fancy. */ xor %eax, %eax3: cmp $0, 0x20(%rbp)
jne 8f4: leave
.cfi_def_cfa %rsp, 8 ret5: mov $(-EINVAL), %rax
jmp 4b6: mov 0x18(%rbp), %rcx
test %rcx, %rcx je 7f /* Fill optional exception info. */ mov %eax, EX_LEAF(%rcx) mov %di, EX_TRAPNR(%rcx) mov %si, EX_ERROR_CODE(%rcx) mov %rdx, EX_ADDRESS(%rcx) 7: mov $(-EFAULT), %rax jmp 3b8: /*
* Align stack perx86_64ABI. Save the original %rsp in %rbx to
be* restored after the callback returns. */ mov %rsp, %rbx and $-0x10, %rsp /* Push @e, u_rsp and @tcs as parameters to the callback. */ push 0x18(%rbp) push %rbx push 0x10(%rbp) /* Pass the "return value" to the callback via %rcx. */ mov %rax, %rcx /* Clear RFLAGS.DF per x86_64 ABI */ cld /* Load the callback pointer to %rax and invoke it via retpoline. */ mov 0x20(%rbp), %rax call 40f /* Restore %rsp to its post-exit value. */ mov %rbx, %rsp /* * If the return from callback is zero or negative, return immediately, * otherwise attempt to re-execute ENCLU with the return interpreted as * the requested ENCLU leaf. */ cmp $0, %eax jle 4b jmp 1b40: /* retpoline */
call 42f 41: pause lfence jmp 41b
Quoted 7 lines
42: mov %rax, (%rsp)
ret
.cfi_endproc
_ASM_VDSO_EXTABLE_HANDLE(2b, 6b)
ENDPROC(__vdso_sgx_enter_enclave)
Your code will work. The only missing thing is a ".cfi_def_cfa %rbp, 16" after 'ret' instruction in block 4. I decided to keep "leave; ret" at the end to avoid confusion around the frame pointer. As you may also notice, GCC generated code usually does the same thing.
-Cedric
Re: [RFC PATCH v2 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 10 lines · Cedric
Quoted 5 lines
While this may format nicely in .rst (I haven't checked), it's damn near unreadable in its raw form. Escaping '%' in kernel-doc is unresolved at this point, but this definitely is an argument for allowing the %CONST interpretation to be disabled entirely.
I know very little about kernel-doc. Not quite sure which markup means what. Or if you don't mind, could you just write up the whole thing (you have written half of it in your email already) so I can include it into my next revision?
Hmm, at this point I'd say just ignore the comment entirely. It's something we need to sort out in the "official" series anyways.
Quoted 7 lines
Quoted 5 lines
longjmp()is probably the only thing that needs to be explicitly marked, I think it would be better to simply say "the callback" instead of using ``callback``, i.e. use regular English instead of referencing the param.Are you suggesting not to mention C++ exception here?
No, I was saying that rather than do the backtick markup on callback, just say "the callback". The copious amount of markup makes it difficult to read the raw text.
Quoted 29 lines
Quoted 20 lines
Quoted 20 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +6
-12-__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) +typedef int (*sgx_callback)(long rdi, long rsi, long rdx, + struct sgx_enclave_exinfo *exinfo, long r8, + long r9, void *tcs, long ursp); +int __vdso_sgx_enter_enclave(int leaf, void *tcs, + struct sgx_enclave_exinfo *exinfo, + sgx_callback callback) { - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; - - if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e)Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 65
Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-2- *e = exception; - return -EFAULT; + while (leaf == EENTER || leaf == ERESUME) {This gives the impression that looping is the common case. I'd prefer using 'goto' to show that the common case is a simple fall through whereas
the callback case can effectively loop on ENCLU.
Quoted 2 lines
Looping IMO is indeed the common case. Just think of the case of an enclave making a sequence of o-calls.
Except that looping is only done when using a callback, and that most definitely is not the common case.
Quoted 60 lines
Quoted 60 lines
Quoted 47 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +29
-9+ int rc; + try { + ENCLU[leaf]; + rc = 0; + if (exinfo) + exinfo->leaf = EEXIT; + } catch (exception) { + rc = -EFAULT; + if (exinfo) + *exinfo = exception; + } + + leaf = callback ? (*callback)( + rdi, rsi, rdx, exinfo, r8, r9, tcs, ursp) : rc; } - return 0; + return leaf > 0 ? -EINVAL : leaf; } #endif ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax - jb bad_input - + jb 6f cmp $0x3, %eax - ja bad_input + ja 6f - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx - /* Save @exception_info */arch/x86/entry/vdso/vsgx_enter_enclave.S line 109
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5
-4- push %rcx + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx@exinfo is optional, i.e. %ecx needs to be tested before its used.
Quoted 3 lines
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S line 110
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ jrcxz 4fThe above instruction (i.e. jrcxz) does test %rcx.
I love x86 ISA.
Quoted 21 lines
Quoted 5 lines
I dislike the number of flags and values that are stashed away only to be consumed later, it makes the code hard to follow. AFAICT, a lot of the shenanigans are done to reuse code because exinfo was overloaded, but that's actually pointless, i.e. it's unnecessarily complex.
Quoted 7 lines
Overlading the struct is pointless becase if you make it mandatory when using a callback then it can be nullified (when passed to the callback) to indicate EEXIT. If it's still optional, then the callback needs an extra param to explicitly indicate EEXIT vs.
-EFAULT, otherwise the callback has no indication whatsover of why it was invoked.My preference is for the latter since it's more explicit and obvious.
I wouldn't nullify @exinfo at EEXIT because that could be used as a "context" pointer to the callback.
Making it optionally because the callback can still use other registers (e.g. %rdi) determine the reason without it. For example, an enclave may set %rdi to non-zero to signify o-call/e-ret while zero (as set by AEX) would indicate an exception. My intention is not to impose unnecessary restrictions on applications.
Quoted 5 lines
Quoted 2 lines
Tangetially related, I'm not opposed to renaming it 'struct
sgx_enclave_exception_info' for clarity.I'd still prefer
sgx_enclave_exinfoto imply its use in both_ex_itand_ex_ceptioncases, for the reason stated above.
Because an enclave *could* choose a different route? The cost is essentially one memory access, which is likely offset by the fact that the callback can directly check %rcx for the exit reason. You are not the sole consumer of this code.
Quoted 12 lines
Quoted 11 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 115
Quoted 5 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +5
-0+ mov %eax, EX_LEAF(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) + mov %rdx, EX_ADDRESS(%rcx)My strong preference would be to organize the code to separate the various
flows, i.e. happy common case, invalid input, exception handler and callback invocation. And make the common case a straight shot so that it's more obvious what happens in the happy non-callback case.
Quoted 4 lines
I'd prefer more concise code. After all, this is assembly without optimization done by the compiler. If it were for C, I'd agree with you because either case would end up in roughly the same code (in terms of code size or efficiency) due to the optimizer.
a) You've argued multiple times this isn't performance critical code.
b) A straight shot is optimal for the common case where userspace is not using a callback.
c) The performance difference is something like one or two memory accesses. That's peanuts compared to how readable the code is by people without prior knowledge of exactly what this code is doing.
Quoted 26 lines
Quoted 18 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +4
-2- add $0x8, %rsp - xor %eax, %eax +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %rax + /* At this point, %ebx holds the effective return value, whichshall be
arch/x86/entry/vdso/vsgx_enter_enclave.S line 121
Quoted 1 line
arch/x86/entry/vdso/vsgx_enter_enclave.S +1
-0+ * returned if no callback is specified */Use kernel-doc format for multi-line comments. Missing punctionation at the end. Blank lines between logical blocks would also help readability.
E.g.:
4: /* Call *callback if supplied */
mov 0x20(%rbp), %rax
test %rax, %raxQuoted 7 lines
/*
* At this point, %ebx holds the effective return value, which
shall be
* returned if no callback is specified.
*/
cmovz %rbx, %rax
jz 7f
Quoted 3 lines
I know very little about kernel-doc. Would you please point me to documents describing it? What you suggested here will of course be incorporated into my next revision.
Documentation/doc-guide/kernel-doc.rst
[...]
Quoted 10 lines
Quoted 3 lines
Note, swapping 'long rc' and '... *e' would allow the callback flow to save one memory access, but IMO the exception info belongs at the end since it's optional.
You have probably misunderstood my code. 'ret' is _not_ passed while @exinfo is passed in %rcx.
exinfo->leafimplies the reason of the callback.Or are you suggesting to add 'ret' as one more parameter to the callback? I don't think it necessary because exinfo contains everything the callback would ever need.
The latter, because exinfo is optional, as stated above.
Re: [RFC PATCH v2 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 7 lines · Sean Christopherson
Quoted 6 lines · Cedric
Quoted 5 lines
While this may format nicely in .rst (I haven't checked), it's damn near unreadable in its raw form. Escaping '%' in kernel-doc is unresolved at this point, but this definitely is an argument for allowing the %CONST interpretation to be disabled entirely.
Quoted 7 lines
Quoted 4 lines
I know very little about kernel-doc. Not quite sure which markup means what. Or if you don't mind, could you just write up the whole thing (you have written half of it in your email already) so I can include it into my next revision?
Hmm, at this point I'd say just ignore the comment entirely. It's something we need to sort out in the "official" series anyways.
To get started: https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt
/Jarkko
RFC PATCH v2 3/3 selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
In reply to Cedric XingThis patch augments SGX selftest with two new tests.
The first test exercises the newly added callback interface, by marking the whole enclave range as PROT_READ, then calling mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes. This test also serves as an example to demonstrate the callback interface.
The second test single-steps through __vdso_sgx_enter_enclave() to make sure the call stack can be unwound at every instruction within that vDSO API. Its purpose is to validate the hand-crafted CFI directives in the assembly.
--- tools/testing/selftests/x86/sgx/Makefile | 6 +- tools/testing/selftests/x86/sgx/main.c | 323 ++++++++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 ++- 3 files changed, 322 insertions(+), 47 deletions(-)
tools/testing/selftests/x86/sgx/Makefile +3 -3
@@ -14,16 +14,16 @@ TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx all_64: $(TEST_CUSTOM_PROGS) $(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o - $(CC) $(HOST_CFLAGS) -o $@ $^ + $(CC) $(HOST_CFLAGS) -o $@ $^ -lunwind -ldl -Wl,--defsym,__image_base=0 -pie $(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss $(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@ $(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf - objcopy --remove-section=.got.plt -O binary $< $@ + objcopy -O binary $< $@ $(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S - $(CC) $(ENCL_CFLAGS) -T $^ -o $@ + $(CC) $(ENCL_CFLAGS) -T $^ -o $@ -Wl,--build-id=none $(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $^ $@
tools/testing/selftests/x86/sgx/main.c +282 -41
388 lines not on this page · raw message
tools/testing/selftests/x86/sgx/sgx_call.S +37 -3
@@ -7,9 +7,43 @@ .global sgx_call sgx_call: + .cfi_startproc + push %r15 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r15, 0 + push %r14 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r14, 0 + push %r13 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r13, 0 + push %r12 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r12, 0 push %rbx - mov $0x02, %rax - mov %rdx, %rbx - call *%r8 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbx, 0 + push $0 + .cfi_adjust_cfa_offset 8 + push 0x48(%rsp) + .cfi_adjust_cfa_offset 8 + push 0x48(%rsp) + .cfi_adjust_cfa_offset 8 + push 0x48(%rsp) + .cfi_adjust_cfa_offset 8 + mov $2, %eax + call *eenter(%rip) + add $0x20, %rsp + .cfi_adjust_cfa_offset -0x20 pop %rbx + .cfi_adjust_cfa_offset -8 + pop %r12 + .cfi_adjust_cfa_offset -8 + pop %r13 + .cfi_adjust_cfa_offset -8 + pop %r14 + .cfi_adjust_cfa_offset -8 + pop %r15 + .cfi_adjust_cfa_offset -8 ret + .cfi_endproc
Signature · 1 line
2.17.1
Re: [RFC PATCH v2 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
Quoted 101 lines · Cedric Xing
This patch augments SGX selftest with two new tests.
The first test exercises the newly added callback interface, by marking the whole enclave range as
PROT_READ, then callingmprotect()upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes. This test also serves as an example to demonstrate the callback interface.The second test single-steps through
__vdso_sgx_enter_enclave()to make sure the call stack can be unwound at every instruction within that vDSO API. Its purpose is to validate the hand-crafted CFI directives in the assembly.Signed-off-by Cedric Xing <[email protected]>--- tools/testing/selftests/x86/sgx/Makefile | 6 +- tools/testing/selftests/x86/sgx/main.c | 323 ++++++++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 ++- 3 files changed, 322 insertions(+), 47 deletions(-)tools/testing/selftests/x86/sgx/Makefile +3
-3@@ -14,16 +14,16 @@ TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx all_64: $(TEST_CUSTOM_PROGS) $(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o - $(CC) $(HOST_CFLAGS) -o $@ $^ + $(CC) $(HOST_CFLAGS) -o $@ $^ -lunwind -ldl -Wl,--defsym,__image_base=0 -pie $(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss $(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@ $(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf - objcopy --remove-section=.got.plt -O binary $< $@ + objcopy -O binary $< $@ $(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S - $(CC) $(ENCL_CFLAGS) -T $^ -o $@ + $(CC) $(ENCL_CFLAGS) -T $^ -o $@ -Wl,--build-id=none $(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $^ $@tools/testing/selftests/x86/sgx/main.c +19
-9@@ -1,6 +1,7 @@// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) // Copyright(c) 2016-18 Intel Corporation. +#define _GNU_SOURCE#include <elf.h> #include <fcntl.h> #include <stdbool.h> @@ -9,16 +10,31 @@#include <stdlib.h> #include <string.h> #include <unistd.h> +#include <errno.h> #include <sys/ioctl.h> #include <sys/mman.h> #include <sys/stat.h> -#include <sys/time.h> +#include <sys/auxv.h> +#include <signal.h> +#include <sys/ucontext.h> + +#define UNW_LOCAL_ONLY +#include <libunwind.h> +#include "encl_piggy.h" #include "defines.h" #include "../../../../../arch/x86/kernel/cpu/sgx/arch.h" #include "../../../../../arch/x86/include/uapi/asm/sgx.h" -static const uint64_t MAGIC =0x1122334455667788ULL ; +#define _Q (x) __Q(x) +#define __Q (x) #x +#define ERRLN "Line " _Q(__LINE__) + +#define X86_EFLAGS_TF (1ul << 8) + +extern char __image_base[]; +size_t eenter; +static size_t vdso_base;struct vdso_symtab {Elf64_Sym *elf_symtab;@@ -26,20 +42,11 @@ struct vdso_symtab {Elf64_Word *elf_hashtab; }; -static void *vdso_get_base_addr (char *envp[]) +static void vdso_init (void ) { -Elf64_auxv_t *auxv; -int i; - -for (i =0 ; envp[i]; i++); - auxv = (Elf64_auxv_t *)&envp[i +1 ]; -
tools/testing/selftests/x86/sgx/main.c struct vdso_symtab { · line 50
Quoted 10 lines
tools/testing/selftests/x86/sgx/main.c +3
-6-for (i =0 ; auxv[i].a_type !=AT_NULL ; i++) { -if (auxv[i].a_type ==AT_SYSINFO_EHDR ) -return (void *)auxv[i].a_un.a_val; - } - -return NULL ; + vdso_base =getauxval (AT_SYSINFO_EHDR ); +if (!vdso_base) +exit (1 ); }
The clean up makes sense but should be a separate patch i.e. one logical change per patch. Right now the patch does other mods than the ones explcitly stated in the commit message.
I'd suggest open coding vdso_init() to the call site in that patch.
Please try to always minimize for diff's.
Quoted 81 lines
tools/testing/selftests/x86/sgx/main.c +55
-5static Elf64_Dyn *vdso_get_dyntab (void *addr)@@ -66,8 +73,9 @@ static void *vdso_get_dyn(void *addr, Elf64_Dyn *dyntab, Elf64_Sxword tag)return NULL ; } -static bool vdso_get_symtab (void *addr,struct vdso_symtab *symtab) +static bool vdso_get_symtab (struct vdso_symtab *symtab) { +void *addr = (void *)vdso_base;Elf64_Dyn *dyntab =vdso_get_dyntab (addr); symtab->elf_symtab =vdso_get_dyn (addr, dyntab,DT_SYMTAB );@@ -138,7 +146,7 @@ static bool encl_create(int dev_fd, unsigned long bin_size, base =mmap (NULL , secs->size,PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_SHARED , dev_fd,0 );if (base ==MAP_FAILED ) { -perror ("mmap" ); +perror (ERRLN );return false; }@@ -224,35 +232,271 @@ static bool encl_load(struct sgx_secs *secs, unsigned long bin_size)return false; } -void sgx_call (void *rdi,void *rsi,void *tcs, -struct sgx_enclave_exception *exception, -void *eenter); +int sgx_call (void *rdi,void *rsi,long rdx,void *rcx,void *r8,void *r9, +void *tcs,struct sgx_enclave_exinfo *ei,void *cb); + +static void show_enclave_exinfo (const struct sgx_enclave_exinfo *exinfop, +const char *header) +{ +static const char *const enclu_leaves[] = { +"EREPORT" , +"EGETKEY" , +"EENTER" , +"ERESUME" , +"EEXIT" + }; +static const char *const exception_names[] = { +"#DE" , +"#DB" , +"NMI" , +"#BP" , +"#OF" , +"#BR" , +"#UD" , +"#NM" , +"#DF" , +"CSO" , +"#TS" , +"#NP" , +"#SS" , +"#GP" , +"#PF" , +"Unknown" , +"#MF" , +"#AC" , +"#MC" , +"#XM" , +"#VE" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" + }; + +printf ("%s: leaf:%s(%d)" , header, + enclu_leaves[exinfop->leaf], exinfop->leaf); +if (exinfop->leaf !=4 ) + printf(" trap:%s(%d) ec:%d addr:0x%llx\n",
tools/testing/selftests/x86/sgx/main.c static bool encl_load(struct sgx_secs *secs, unsigned long bin_size) · line 295
Quoted 10 lines
tools/testing/selftests/x86/sgx/main.c +8
-1+ exception_names[exinfop->trapnr], exinfop->trapnr, + exinfop->error_code, exinfop->address); + else + printf("\n"); +} + +static const uint64_t MAGIC =0x1122334455667788ULL ; -int main (int argc,char *argv[],char *envp[]) +static void test1 (struct sgx_secs *secs)
test1, test2 and test3 are not too descriptive names. Every patch should make the code base cleaner, not messier.
/Jarkko
Re: [RFC PATCH v2 3/3] selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
Quoted 211 lines · Jarkko Sakkinen
Quoted 101 lines · Cedric Xing
This patch augments SGX selftest with two new tests.
The first test exercises the newly added callback interface, by marking the whole enclave range as
PROT_READ, then callingmprotect()upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes. This test also serves as an example to demonstrate the callback interface.The second test single-steps through
__vdso_sgx_enter_enclave()to make sure the call stack can be unwound at every instruction within that vDSO API. Its purpose is to validate the hand-crafted CFI directives in the assembly.Signed-off-by Cedric Xing <[email protected]>--- tools/testing/selftests/x86/sgx/Makefile | 6 +- tools/testing/selftests/x86/sgx/main.c | 323 ++++++++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 ++- 3 files changed, 322 insertions(+), 47 deletions(-)tools/testing/selftests/x86/sgx/Makefile +3
-3@@ -14,16 +14,16 @@ TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx all_64: $(TEST_CUSTOM_PROGS) $(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o - $(CC) $(HOST_CFLAGS) -o $@ $^ + $(CC) $(HOST_CFLAGS) -o $@ $^ -lunwind -ldl -Wl,--defsym,__image_base=0 -pie $(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss $(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@ $(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf - objcopy --remove-section=.got.plt -O binary $< $@ + objcopy -O binary $< $@ $(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S - $(CC) $(ENCL_CFLAGS) -T $^ -o $@ + $(CC) $(ENCL_CFLAGS) -T $^ -o $@ -Wl,--build-id=none $(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $^ $@tools/testing/selftests/x86/sgx/main.c +19
-9@@ -1,6 +1,7 @@// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) // Copyright(c) 2016-18 Intel Corporation. +#define _GNU_SOURCE#include <elf.h> #include <fcntl.h> #include <stdbool.h> @@ -9,16 +10,31 @@#include <stdlib.h> #include <string.h> #include <unistd.h> +#include <errno.h> #include <sys/ioctl.h> #include <sys/mman.h> #include <sys/stat.h> -#include <sys/time.h> +#include <sys/auxv.h> +#include <signal.h> +#include <sys/ucontext.h> + +#define UNW_LOCAL_ONLY +#include <libunwind.h> +#include "encl_piggy.h" #include "defines.h" #include "../../../../../arch/x86/kernel/cpu/sgx/arch.h" #include "../../../../../arch/x86/include/uapi/asm/sgx.h" -static const uint64_t MAGIC =0x1122334455667788ULL ; +#define _Q (x) __Q(x) +#define __Q (x) #x +#define ERRLN "Line " _Q(__LINE__) + +#define X86_EFLAGS_TF (1ul << 8) + +extern char __image_base[]; +size_t eenter; +static size_t vdso_base;struct vdso_symtab {Elf64_Sym *elf_symtab;@@ -26,20 +42,11 @@ struct vdso_symtab {Elf64_Word *elf_hashtab; }; -static void *vdso_get_base_addr (char *envp[]) +static void vdso_init (void ) { -Elf64_auxv_t *auxv; -int i; - -for (i =0 ; envp[i]; i++); - auxv = (Elf64_auxv_t *)&envp[i +1 ]; -tools/testing/selftests/x86/sgx/main.c struct vdso_symtab { · line 50
Quoted 10 lines
tools/testing/selftests/x86/sgx/main.c +3
-6-for (i =0 ; auxv[i].a_type !=AT_NULL ; i++) { -if (auxv[i].a_type ==AT_SYSINFO_EHDR ) -return (void *)auxv[i].a_un.a_val; - } - -return NULL ; + vdso_base =getauxval (AT_SYSINFO_EHDR ); +if (!vdso_base) +exit (1 ); }The clean up makes sense but should be a separate patch i.e. one logical change per patch. Right now the patch does other mods than the ones explcitly stated in the commit message.
I'd suggest open coding
vdso_init()to the call site in that patch.Please try to always minimize for diff's.
Quoted 89 lines
tools/testing/selftests/x86/sgx/main.c +62
-5static Elf64_Dyn *vdso_get_dyntab (void *addr)@@ -66,8 +73,9 @@ static void *vdso_get_dyn(void *addr, Elf64_Dyn *dyntab, Elf64_Sxword tag)return NULL ; } -static bool vdso_get_symtab (void *addr,struct vdso_symtab *symtab) +static bool vdso_get_symtab (struct vdso_symtab *symtab) { +void *addr = (void *)vdso_base;Elf64_Dyn *dyntab =vdso_get_dyntab (addr); symtab->elf_symtab =vdso_get_dyn (addr, dyntab,DT_SYMTAB );@@ -138,7 +146,7 @@ static bool encl_create(int dev_fd, unsigned long bin_size, base =mmap (NULL , secs->size,PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_SHARED , dev_fd,0 );if (base ==MAP_FAILED ) { -perror ("mmap" ); +perror (ERRLN );return false; }@@ -224,35 +232,271 @@ static bool encl_load(struct sgx_secs *secs, unsigned long bin_size)return false; } -void sgx_call (void *rdi,void *rsi,void *tcs, -struct sgx_enclave_exception *exception, -void *eenter); +int sgx_call (void *rdi,void *rsi,long rdx,void *rcx,void *r8,void *r9, +void *tcs,struct sgx_enclave_exinfo *ei,void *cb); + +static void show_enclave_exinfo (const struct sgx_enclave_exinfo *exinfop, +const char *header) +{ +static const char *const enclu_leaves[] = { +"EREPORT" , +"EGETKEY" , +"EENTER" , +"ERESUME" , +"EEXIT" + }; +static const char *const exception_names[] = { +"#DE" , +"#DB" , +"NMI" , +"#BP" , +"#OF" , +"#BR" , +"#UD" , +"#NM" , +"#DF" , +"CSO" , +"#TS" , +"#NP" , +"#SS" , +"#GP" , +"#PF" , +"Unknown" , +"#MF" , +"#AC" , +"#MC" , +"#XM" , +"#VE" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" , +"Unknown" + }; + +printf ("%s: leaf:%s(%d)" , header, + enclu_leaves[exinfop->leaf], exinfop->leaf); +if (exinfop->leaf !=4 ) +printf (" trap:%s(%d) ec:%d addr:0x%llx\n" , + exception_names[exinfop->trapnr], exinfop->trapnr, + exinfop->error_code, exinfop->address); +else +printf ("\n" ); +} + +static const uint64_t MAGIC =0x1122334455667788ULL ;
Quoted 5 lines
tools/testing/selftests/x86/sgx/main.c static bool encl_load(struct sgx_secs *secs, unsigned long bin_size) · line 295
Quoted 2 lines
tools/testing/selftests/x86/sgx/main.c +1
-1-int main (int argc,char *argv[],char *envp[]) +static void test1 (struct sgx_secs *secs)test1, test2 and test3 are not too descriptive names. Every patch should make the code base cleaner, not messier.
I don't think it that important, as there are many test## occurrences in existing selftests. Anyway, I've added comments to those test functions to brief what is done. Hope you'll find them helpful.
Quoted 2 lines
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
In reply to Cedric XingQuoted 19 lines · Cedric Xing
The current proposed
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.This patchset introduces a new ABI for
__vdso_sgx_enter_enclave()to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new__vdso_sgx_enter_enclave()will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null),__vdso_sgx_enter_enclave()will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).
Quoted 4 lines
The SGX selftest is augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in
__vdso_sgx_enter_enclave()by single-stepping through it and unwinding call stack at every instruction.
Why does the SDK anyway use real enclaves when step debugging? I don't think kernel needs to scale to that. For me it looks more like a bad architectural choice in the SDK implementation than anything else.
You should design SDK in a way that it doesn't run the code inside real enclave at first. It is just the sanest development model to use. The current SDK has an unacceptable requirement that the workstation used to develop code destined to run inside enclave must possess an appropriate hardware. I don't think that is acceptable no matter what kind of API kernel provides to invoke enclaves.
I think "real" debug enclaves are only good for production testing when you have more or less ready to release/deploy something but still want to be able to get a memory dump of the enclave on occassion.
With these conclusions I think the current vDSO API is sufficient for Linux.
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
Quoted 20 lines · Jarkko Sakkinen
Quoted 19 lines · Cedric Xing
The current proposed
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. And given its simplicity and convenience, it could be desired by future SGX applications as well.This patchset introduces a new ABI for
__vdso_sgx_enter_enclave()to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. Additionally, this new__vdso_sgx_enter_enclave()will take one more parameter - a callback function, to be invoked upon all enclave exits (both AEX and normal exits). The callback function will be given the value of %rsp left off by the enclave, so that data "pushed" by the enclave (if any) could be addressed/accessed. Please note that the callback function is optional, and if not supplied (i.e. null),__vdso_sgx_enter_enclave()will just return (i.e. behave the same as the current implementation) after the enclave exits (or AEX due to exceptions).
Quoted 8 lines
Quoted 4 lines
The SGX selftest is augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in
__vdso_sgx_enter_enclave()by single-stepping through it and unwinding call stack at every instruction.Why does the SDK anyway use real enclaves when step debugging? I don't think kernel needs to scale to that. For me it looks more like a bad architectural choice in the SDK implementation than anything else.
Intel's SGX SDK *does* support simulation mode for debugging enclaves outside of SGX mode, or even on non-SGX platforms. But nothing can replace the real SGX environment, hence SGX ISA supports debugging real enclaves.
Quoted 6 lines
You should design SDK in a way that it doesn't run the code inside real enclave at first. It is just the sanest development model to use. The current SDK has an unacceptable requirement that the workstation used to develop code destined to run inside enclave must possess an appropriate hardware. I don't think that is acceptable no matter what kind of API kernel provides to invoke enclaves.
SDK supports simulation on non-SGX platforms.
Quoted 3 lines
I think "real" debug enclaves are only good for production testing when you have more or less ready to release/deploy something but still want to be able to get a memory dump of the enclave on occassion.
That's true.
Quoted 2 lines
With these conclusions I think the current vDSO API is sufficient for Linux.
The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.
The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.
Quoted 2 lines
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
Quoted 5 lines · Cedric
Quoted 2 lines
With these conclusions I think the current vDSO API is sufficient for Linux.
The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.
And how that is useful?
Quoted 5 lines
The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.
Of course I won't merge code of which usefulness I don't understand.
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
Quoted 8 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines
With these conclusions I think the current vDSO API is sufficient for Linux.
The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.
And how that is useful?
There is a lengthy discussion on its usefulness so I don't want to repeat. In short, it allows using untrusted stack as a convenient method to exchange data with the enclave. It is currently being used by Intel's SGX SDK for e/o-calls parameters.
Quoted 7 lines
Quoted 5 lines
The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.
Of course I won't merge code of which usefulness I don't understand.
Sure.
Any other questions I can help with?
Quoted 2 lines
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
Quoted 21 lines · Cedric
Quoted 8 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines
With these conclusions I think the current vDSO API is sufficient for Linux.
The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.
And how that is useful?
There is a lengthy discussion on its usefulness so I don't want to repeat. In short, it allows using untrusted stack as a convenient method to exchange data with the enclave. It is currently being used by Intel's SGX SDK for e/o-calls parameters.
Quoted 6 lines
Quoted 5 lines
The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.
Quoted 5 lines
Quoted 1 line
Of course I won't merge code of which usefulness I don't understand.
Sure.
Any other questions I can help with?
I dissected my concerns in other email. We can merge this feature after v21 if it makes sense.
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
Quoted 22 lines · Jarkko Sakkinen
Quoted 21 lines · Cedric
Quoted 8 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines
With these conclusions I think the current vDSO API is sufficient for Linux.
The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.
And how that is useful?
There is a lengthy discussion on its usefulness so I don't want to repeat. In short, it allows using untrusted stack as a convenient method to exchange data with the enclave. It is currently being used by Intel's SGX SDK for e/o-calls parameters.
Quoted 6 lines
Quoted 5 lines
The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Of course I won't merge code of which usefulness I don't understand.
Sure.
Any other questions I can help with?
I dissected my concerns in other email. We can merge this feature after v21 if it makes sense.
Sent out v3 of vDSO changes last night. I hope your concerns have been properly addressed.
The new vDSO API is a community consensus. I can help on whatever technical problems you may have but I don't see a reason you should reject it.
Quoted 2 lines
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
In reply to Jarkko SakkinenQuoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines
With these conclusions I think the current vDSO API is sufficient for Linux.
The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.
And how that is useful?
Quoted 7 lines
Quoted 5 lines
The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.
Of course I won't merge code of which usefulness I don't understand.
I re-read the cover letter [1] because it usually is the place to "pitch" a feature.
It fails to address two things:
1. How and in what circumstances is an untrusted stack is a better vessel for handling exceptions than the register based approach that we already have?
2. How is it simpler approach? There is a strong claim of simplicity and convenience without anything backing it.
3. Why we need both register and stack based approach co-exist? I'd go with one approach for a new API without any legacy whatsoever.
This really needs a better pitch before we can consider doing anything to it.
Also, in [2] there is talk about the next revision. Maybe the way go forward is to address the three issues I found in the cover letter and fix whatever needed to be fixed in the actual patches?
[1] https://lkml.org/lkml/2019/4/24/84
[2] https://lkml.org/lkml/2019/4/25/1170
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
Quoted 18 lines · Jarkko Sakkinen
Quoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines
With these conclusions I think the current vDSO API is sufficient for Linux.
The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.
And how that is useful?
Quoted 7 lines
Quoted 5 lines
The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.
Of course I won't merge code of which usefulness I don't understand.
Quoted 8 lines
I re-read the cover letter [1] because it usually is the place to "pitch" a feature.
It fails to address two things:
1. How and in what circumstances is an untrusted stack is a better vessel for handling exceptions than the register based approach that we already have?
We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.
Quoted 2 lines
2. How is it simpler approach? There is a strong claim of simplicity and convenience without anything backing it.
The major benefits in terms of simplicity realize in user mode applications. It's always a trade-off. This vDSO API takes 10-20 more lines than the original one but would save hundreds or even thousands in user applications.
Again, I don't want to repeat everything as you can look back at the lengthy discussion to dig out the details.
Quoted 2 lines
3. Why we need both register and stack based approach co-exist? I'd go with one approach for a new API without any legacy whatsoever.
Neither is a legacy to the other. Supporting both approaches is by design. Again, the goal is to enable more vessels because there's *no* single best vessel.
Quoted 9 lines
This really needs a better pitch before we can consider doing anything to it.
Also, in [2] there is talk about the next revision. Maybe the way go forward is to address the three issues I found in the cover letter and fix whatever needed to be fixed in the actual patches?
[1] https://lkml.org/lkml/2019/4/24/84
[2] https://lkml.org/lkml/2019/4/25/1170
Let me update the commit message for the vDSO API and send you a patch.
Quoted 2 lines
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
Quoted 24 lines · Cedric
Quoted 23 lines · Jarkko Sakkinen
Quoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines
With these conclusions I think the current vDSO API is sufficient for Linux.
The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.
And how that is useful?
Quoted 7 lines
Quoted 5 lines
The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.
Of course I won't merge code of which usefulness I don't understand.
I re-read the cover letter [1] because it usually is the place to "pitch" a feature.
It fails to address two things:
Quoted 7 lines
Quoted 3 lines
1. How and in what circumstances is an untrusted stack is a better vessel for handling exceptions than the register based approach that we already have?
We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.
I think reasonable metric is actually the coverage of the Intel SDK based enclaves. How widely are they in the wild? If the user base is large, it should be reasonable to support this just based on that.
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
Quoted 8 lines · Jarkko Sakkinen
Quoted 3 lines · Cedric
We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.
I think reasonable metric is actually the coverage of the Intel SDK based enclaves. How widely are they in the wild? If the user base is large, it should be reasonable to support this just based on that.
Large enough that Andy agreed to take the vDSO code with the optional callback, despite his personal opinion being that mucking with uR{B,S}P from within the enclave is poor form.
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
Quoted 6 lines · Sean Christopherson
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines · Cedric
We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.
Quoted 7 lines
Quoted 3 lines
I think reasonable metric is actually the coverage of the Intel SDK based enclaves. How widely are they in the wild? If the user base is large, it should be reasonable to support this just based on that.
Large enough that Andy agreed to take the vDSO code with the optional callback, despite his personal opinion being that mucking with uR{B,S}P from within the enclave is poor form.
OK, the cover letter empahasized things that did not make sense to me, which made me to do my initial conclusions. I don't recall even reading the word "coverage" from it.
Anyways, I'm sure we can land this after v21 has been published now that the rationale is clear.
/Jarkko
Re: [RFC PATCH v2 0/3] An alternative __vdso_sgx_enter_enclave() to allow enclave/host parameter passing using untrusted stack
In reply to Jarkko SakkinenQuoted 29 lines · Jarkko Sakkinen
Quoted 28 lines · Cedric
Quoted 26 lines · Jarkko Sakkinen
Quoted 9 lines · Jarkko Sakkinen
Quoted 5 lines · Cedric
Quoted 2 lines
With these conclusions I think the current vDSO API is sufficient for Linux.
The new vDSO API is to support data exchange on stack. It has nothing to do with debugging. BTW, the community has closed on this.
And how that is useful?
Quoted 7 lines
Quoted 5 lines
The CFI directives are for stack unwinding. They don't affect what the code does so you can just treat them as NOPs if you don't understand what they do. However, they are useful to not only debuggers but also exception handling code. libunwind also has a setjmp()/longjmp() implementation based on CFI directives.
Of course I won't merge code of which usefulness I don't understand.
I re-read the cover letter [1] because it usually is the place to "pitch" a feature.
It fails to address two things:
1. How and in what circumstances is an untrusted stack is a better vessel for handling exceptions than the register based approach that we already have?
Quoted 7 lines
Quoted 3 lines
We are not judging which vessel is better (or the best) among all possible vessels. We are trying to enable more vessels. Every vessel has its pros and cons so there's *no* single best vessel.
I think reasonable metric is actually the coverage of the Intel SDK based enclaves. How widely are they in the wild? If the user base is large, it should be reasonable to support this just based on that.
I don't know how many existing enclaves out there, but definitely larger than 0 (zero), while user base for the old API is definitely 0. What are you worrying, really?
Quoted 2 lines
/Jarkko
v3 cover x86/sgx: Amend vDSO API to allow enclave/host parameter passing on untrusted stack
In reply to Cedric XingThis patchset is based upon, and can be applied cleanly on SGX1 patch v20
(https://lkml.org/lkml/2019/4/17/344) by Jarkko Sakkinen.
The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. After all, passing data via untrusted stack is very easy to implement (by enclaves), with essentially no overhead, therefore is very suitable for exchanging data in small amounts, so could be desirable by future SGX applications as well.
This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. And in order to service o-calls and to preserve the untrusted stack upon exceptions, the new vDSO API takes one more optional parameter - "callback", which if supplied, will be invoked on all enclave exits (including normal and asynchronous exits). Ample details regarding the new ABI have been documented as comments inside the source code located in arch/x86/entry/vsgx_enter_enclave.S
Please note that there was a lengthy discussion on what is the "best" approach for passing parameters for trusted/untrusted calls. Unfortunately there's no single "best" approach that fits all use cases, hence this new ABI has been designed intentionally to accommodate varieties. Therefore, to those not interested in using the untrusted stack, whatever worked with the old ABI proposed by Sean will continue to work with this new ABI.
The SGX selftest has been augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in __vdso_sgx_enter_enclave() by single-stepping through it and unwinding call stack at every instruction. Please note that the selftest CANNOT run to completion yet, as it depends on the vDSO fixup code to signal the process upon #DB/#BP inside enclaves (rather than the current behavior of branching to the handler in vDSO).
Changelog:
· This is version 3 of this patch series with the following changes.
- Per Andy Lutomirski and Sean Christopherson, revised comments and their format in arch/x86/entry/vsgx_enter_enclave.S
- Per Jarkko Sakkinen, revised the cover letter to articulate motivation and objective of this patchset.
· v2 - https://patchwork.kernel.org/cover/10914161/
· v1 - https://patchwork.kernel.org/cover/10911615/
Cedric Xing (3):
selftests/x86: Fixed Makefile for SGX selftest
x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing
on untrusted stack
selftests/x86: Augment SGX selftest to test new
__vdso_sgx_enter_enclave() and its callback interface
arch/x86/entry/vdso/vsgx_enter_enclave.S | 214 ++++++++++---- arch/x86/include/uapi/asm/sgx.h | 14 +- tools/testing/selftests/x86/Makefile | 12 +- tools/testing/selftests/x86/sgx/Makefile | 49 ++-- tools/testing/selftests/x86/sgx/main.c | 323 ++++++++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 ++- 6 files changed, 500 insertions(+), 152 deletions(-)
Signature · 1 line
2.17.1
RFC PATCH v3 1/3 selftests/x86: Fixed Makefile for SGX selftest
The original x86/sgx/Makefile doesn't work when 'x86/sgx' is specified as the test target. This patch fixes that problem, along with minor changes to the dependencies between 'x86' and 'x86/sgx' in selftests/x86/Makefile.
--- tools/testing/selftests/x86/Makefile | 12 +++---- tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++--------------- 2 files changed, 22 insertions(+), 35 deletions(-)
tools/testing/selftests/x86/Makefile +6 -6
@@ -70,11 +70,11 @@ all_32: $(BINARIES_32) all_64: $(BINARIES_64) -all_64: $(SUBDIRS_64) - @for DIR in $(SUBDIRS_64); do \ - BUILD_TARGET=$(OUTPUT)/$$DIR; \ - mkdir $$BUILD_TARGET -p; \ - make OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ +all_64: | $(SUBDIRS_64) + @for DIR in $|; do \ + BUILD_TARGET=$(OUTPUT)/$$DIR; \ + mkdir $$BUILD_TARGET -p; \ + $(MAKE) OUTPUT=$$BUILD_TARGET -C $$DIR $@; \ done EXTRA_CLEAN := $(BINARIES_32) $(BINARIES_64)
@@ -90,7 +90,7 @@ ifeq ($(CAN_BUILD_I386)$(CAN_BUILD_X86_64),01) all: warn_32bit_failure warn_32bit_failure: - @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \ + @echo "Warning: you seem to have a broken 32-bit build" 2>&1; \ echo "environment. This will reduce test coverage of 64-bit" 2>&1; \ echo "kernels. If you are using a Debian-like distribution," 2>&1; \ echo "try:"; 2>&1; \
tools/testing/selftests/x86/sgx/Makefile +16 -29
63 lines not on this page · raw message
Signature · 1 line
2.17.1
RFC PATCH v3 2/3 x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
In reply to Cedric XingThe previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls) on the untrusted stack.
This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.
Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.
--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 214 ++++++++++++++++------- arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 157 insertions(+), 71 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +150 -64
@@ -14,88 +14,174 @@ .code64 .section .text, "ax" -#ifdef SGX_KERNEL_DOC /** * __vdso_sgx_enter_enclave() - Enter an SGX enclave * - * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer + * Parameters: + * @leaf, passed in %eax, must be either EENTER(2) or ERESUME(3) + * @tcs, passed on stack at 8(%rsp), is the linear address of TCS + * @exinfo, passed on stack at 0x10(%rsp), is optional, and if non-NULL, + * shall point to an sgx_enclave_exinfo structure to receive information + * about the enclave exit + * @callback, passed on stack at 0x18(%rsp), is optiona, and if non-NULL, + * points to a callback function that will be invoked after the enclave + * exits * - * Return: - * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is - * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults + * Returns: + * $0 (zero) on a clean exit from the enclave + * $-EINVAL will be returned if leaf isn't either EENTER or ERESUME + * Other negative values could also be returned as the return value from + * the callback function * - * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the - * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, - * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with - * the return value passed via ``%eax``. All registers except ``%rsp`` must - * be treated as volatile from the caller's perspective, including but not - * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave - * being run **must** preserve the untrusted ``%rsp`` and stack. + * IMPORTANT! This API is **not** compliant with x86-64 ABI but adopts a + * proprietary calling convention, described below: + * · As noted above, input parameters are passed via %eax and the stack. + * · The return value is passed via %eax. + * · %rbx and %rcx must be treated as volatile as they are modified as part + * of enclaves transitions and are used as scratch regs. + * · %rdx, %rdi, %rsi and %r8-%r15 are passed as is and may be freely + * modified by the enclave. Values left in those registers will not be + * altered either, so will be visiable to the callback or the caller (if no + * callback is specified). + * · %rsp is saved/restored across __vdso_sgx_enter_enclave(). + * + * A callback function, if supplied, shall have the following signature: + * + * int callback(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *exinfo, long r8, long r9, + * void *tcs, long ursp); + * + * Callback functions shall comply to x86_64 ABI. + * · All registers left off by the enclave are passed as is except %rax, %rbx + * and %rcx. %rdi, %rsi, %r8 and %9 could be accessed as function + * parameters, while other registers could be access in assembly code if + * needed. + * · Positive return values from the callback will be interpreted as ENCLU + * leafs to re-enter the enclave. Currently only EENTER(2) and ERESUME(3) + * are supported, while all other positive return values will result in + * $-EINVAL returned to the caller of __vdso_sgx_enter_enclave(). + * · $0 (zero) or negative return values will be passed back to the caller of + * __vdso_sgx_enter_enclave() as is. + * + * Pseudo-code: + * + * typedef int (*sgx_callback)(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *exinfo, long r8, + * long r9, void *tcs, long ursp); + * + * int __vdso_sgx_enter_enclave(int leaf, void *tcs, + * struct sgx_enclave_exinfo *exinfo, + * sgx_callback callback) + * { + * while (leaf == EENTER || leaf == ERESUME) { + * int rc; + * try { + * ENCLU[leaf]; + * rc = 0; + * if (exinfo) + * exinfo->leaf = EEXIT; + * } catch (exception) { + * rc = -EFAULT; + * if (exinfo) + * *exinfo = exception; + * } + * + * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo, + * r8, r9, tcs, ursp); + * } + * + * return leaf > 0 ? -EINVAL : leaf; + * } */ -__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) -{ - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; - - if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e) - *e = exception; - return -EFAULT; - } - - return 0; -} -#endif ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax - jb bad_input - + jb 6f cmp $0x3, %eax - ja bad_input + ja 6f - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx - /* Save @exception_info */ - push %rcx + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx + jrcxz 4f + mov %eax, EX_LEAF(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) + mov %rdx, EX_ADDRESS(%rcx) - add $0x8, %rsp - xor %eax, %eax +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %rax + /* + * At this point, %ebx holds the effective return value, which shall be + * returned if no callback is specified + */ + cmovz %rbx, %rax + jz 7f + /* + * Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be + * restored after *callback returns. + */ + mov %rsp, %rbx + and $-0x10, %rsp + /* Clear RFLAGS.DF per x86_64 ABI */ + cld + /* Parameters for *callback */ + push %rbx + push 0x10(%rbp) + /* Call *%rax via retpoline */ + call 40f + /* + * Restore %rsp to its original value left off by the enclave from last + * exit + */ + mov %rbx, %rsp + /* + * Positive return value from *callback will be interpreted as an ENCLU + * leaf, while a non-positive value will be interpreted as the return + * value to be passed back to the caller. + */ + jmp 1b +40: /* retpoline */ + call 42f +41: pause + lfence + jmp 41b +42: mov %rax, (%rsp) ret -bad_input: - mov $(-EINVAL), %rax - ret +5: /* Exception path */ + mov $-EFAULT, %ebx + stc + jmp 3b -.pushsection .fixup, "ax" - /* Re-load @exception_info and fill it (if it's non-NULL) */ -2: pop %rcx - test %rcx, %rcx - je 3f +6: /* Unsupported ENCLU leaf */ + cmp $0, %eax + jle 7f + mov $-EINVAL, %eax - mov %eax, EX_LEAF(%rcx) - mov %di, EX_TRAPNR(%rcx) - mov %si, EX_ERROR_CODE(%rcx) - mov %rdx, EX_ADDRESS(%rcx) -3: mov $(-EFAULT), %rax +7: /* Epilog */ + leave + .cfi_def_cfa %rsp, 8 ret -.popsection + .cfi_endproc -_ASM_VDSO_EXTABLE_HANDLE(1b, 2b) +_ASM_VDSO_EXTABLE_HANDLE(2b, 5b) ENDPROC(__vdso_sgx_enter_enclave)
arch/x86/include/uapi/asm/sgx.h +7 -7
@@ -24,7 +24,7 @@/** * struct sgx_enclave_create - parameter structure for the -* %SGX_IOC_ENCLAVE_CREATE ioctl +* %SGX_IOC_ENCLAVE_CREATE ioctl * @src: address for the SECS page data */ struct sgx_enclave_create {
@@ -33,7 +33,7 @@ struct sgx_enclave_create {/** * struct sgx_enclave_add_page - parameter structure for the -* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl +* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl * @addr: address within the ELRANGE * @src: address for the page data * @secinfo: address for the SECINFO data
@@ -49,7 +49,7 @@ struct sgx_enclave_add_page {/** * struct sgx_enclave_init - parameter structure for the -* %SGX_IOC_ENCLAVE_INIT ioctl +* %SGX_IOC_ENCLAVE_INIT ioctl * @sigstruct: address for the SIGSTRUCT data */ struct sgx_enclave_init {
@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute { };/** -* struct sgx_enclave_exception - structure to report exceptions encountered in -* __vdso_sgx_enter_enclave() +* struct sgx_enclave_exinfo - structure to report exceptions encountered in +* __vdso_sgx_enter_enclave() * -* @leaf: ENCLU leaf from \%eax at time of exception +* @leaf: ENCLU leaf from \%eax at time of exception/exit * @trapnr: exception trap number, a.k.a. fault vector * @error_code: exception error code * @address: exception address, e.g. CR2 on a #PF * @reserved: reserved for future use */ -struct sgx_enclave_exception { +struct sgx_enclave_exinfo {__u32 leaf;__u16 trapnr;__u16 error_code;
Signature · 1 line
2.17.1
Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 44 lines · Cedric Xing
The previous
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls) on the untrusted stack.This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then
__vdso_sgx_enter_enclave()can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after__vdso_sgx_enter_enclave()returns. Therefore,__vdso_sgx_enter_enclave()has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.Here's the summary of API/ABI changes in this patch. More details could be found in
arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'structsgx_enclave_exception' is renamed to 'structsgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
*__vdso_sgx_enter_enclave()anchors its frame using %rbp (instead of %rsp in the previous implementation).
*__vdso_sgx_enter_enclave()takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause__vdso_sgx_enter_enclave()to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of__vdso_sgx_enter_enclave()to its caller. It's also safe to leave callback bylongjmp()or by throwing a C++ exception.Signed-off-by Cedric Xing <[email protected]>--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 214 ++++++++++++++++------- arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 157 insertions(+), 71 deletions(-)arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-0
arch/x86/entry/vdso/vsgx_enter_enclave.S line 19
Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-1@@ -14,88 +14,174 @@ .code64 .section .text, "ax" -#ifdef SGX_KERNEL_DOC /** * __vdso_sgx_enter_enclave() - Enter an SGX enclave *
This was already there but here should not be empty line.
arch/x86/entry/vdso/vsgx_enter_enclave.S line 23
Quoted 3 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +0
-3- * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer
Does not align with kdoc standards.
* @leaf: ENCLU leaf, must be EENTER or ERESUME
* @tcs: TCS, must be non-NULL
* @ex_info: Optional 'struct sgx_enclave_exception' pointer
Quoted 77 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +64
-11+ * Parameters: + * @leaf, passed in %eax, must be either EENTER(2) or ERESUME(3) + * @tcs, passed on stack at 8(%rsp), is the linear address of TCS + * @exinfo, passed on stack at 0x10(%rsp), is optional, and if non-NULL, + * shall point to an sgx_enclave_exinfo structure to receive information + * about the enclave exit + * @callback, passed on stack at 0x18(%rsp), is optiona, and if non-NULL, + * points to a callback function that will be invoked after the enclave + * exits * - * Return: - * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is - * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults + * Returns: + * $0 (zero) on a clean exit from the enclave + * $-EINVAL will be returned if leaf isn't either EENTER or ERESUME + * Other negative values could also be returned as the return value from + * the callback function * - * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the - * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, - * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with - * the return value passed via ``%eax``. All registers except ``%rsp`` must - * be treated as volatile from the caller's perspective, including but not - * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave - * being run **must** preserve the untrusted ``%rsp`` and stack. + * IMPORTANT! This API is **not** compliant with x86-64 ABI but adopts a + * proprietary calling convention, described below: + * · As noted above, input parameters are passed via %eax and the stack. + * · The return value is passed via %eax. + * · %rbx and %rcx must be treated as volatile as they are modified as part + * of enclaves transitions and are used as scratch regs. + * · %rdx, %rdi, %rsi and %r8-%r15 are passed as is and may be freely + * modified by the enclave. Values left in those registers will not be + * altered either, so will be visiable to the callback or the caller (if no + * callback is specified). + * · %rsp is saved/restored across __vdso_sgx_enter_enclave(). + * + * A callback function, if supplied, shall have the following signature: + * + * int callback(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *exinfo, long r8, long r9, + * void *tcs, long ursp); + * + * Callback functions shall comply to x86_64 ABI. + * · All registers left off by the enclave are passed as is except %rax, %rbx + * and %rcx. %rdi, %rsi, %r8 and %9 could be accessed as function + * parameters, while other registers could be access in assembly code if + * needed. + * · Positive return values from the callback will be interpreted as ENCLU + * leafs to re-enter the enclave. Currently only EENTER(2) and ERESUME(3) + * are supported, while all other positive return values will result in + * $-EINVAL returned to the caller of __vdso_sgx_enter_enclave(). + * · $0 (zero) or negative return values will be passed back to the caller of + * __vdso_sgx_enter_enclave() as is. + * + * Pseudo-code: + * + * typedef int (*sgx_callback)(long rdi, long rsi, long rdx, + * struct sgx_enclave_exinfo *exinfo, long r8, + * long r9, void *tcs, long ursp); + * + * int __vdso_sgx_enter_enclave(int leaf, void *tcs, + * struct sgx_enclave_exinfo *exinfo, + * sgx_callback callback) + * { + * while (leaf == EENTER || leaf == ERESUME) { + * int rc; + * try { + * ENCLU[leaf]; + * rc = 0; + * if (exinfo) + * exinfo->leaf = EEXIT; + * } catch (exception) { + * rc = -EFAULT; + * if (exinfo)
arch/x86/entry/vdso/vsgx_enter_enclave.S line 95
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +9
-0+ * *exinfo = exception; + * } + * + * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo, + * r8, r9, tcs, ursp); + * } + * + * return leaf > 0 ? -EINVAL : leaf; + * } */
Please remove all of this documentation or write more punctual documentation and follow the standard:
https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt
Not going to maintain the above. Rather do not add documentation at all than the above.
Quoted 146 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S +77
-49-__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) -{ - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; - - if (!tcs) - return -EINVAL; - - try { - ENCLU[leaf]; - } catch (exception) { - if (e) - *e = exception; - return -EFAULT; - } - - return 0; -} -#endif ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ cmp $0x2, %eax - jb bad_input - + jb 6f cmp $0x3, %eax - ja bad_input + ja 6f - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx - /* Save @exception_info */ - push %rcx + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx + jrcxz 4f + mov %eax, EX_LEAF(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) + mov %rdx, EX_ADDRESS(%rcx) - add $0x8, %rsp - xor %eax, %eax +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %rax + /* + * At this point, %ebx holds the effective return value, which shall be + * returned if no callback is specified + */ + cmovz %rbx, %rax + jz 7f + /* + * Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be + * restored after *callback returns. + */ + mov %rsp, %rbx + and $-0x10, %rsp + /* Clear RFLAGS.DF per x86_64 ABI */ + cld + /* Parameters for *callback */ + push %rbx + push 0x10(%rbp) + /* Call *%rax via retpoline */ + call 40f + /* + * Restore %rsp to its original value left off by the enclave from last + * exit + */ + mov %rbx, %rsp + /* + * Positive return value from *callback will be interpreted as an ENCLU + * leaf, while a non-positive value will be interpreted as the return + * value to be passed back to the caller. + */ + jmp 1b +40: /* retpoline */ + call 42f +41: pause + lfence + jmp 41b +42: mov %rax, (%rsp) ret -bad_input: - mov $(-EINVAL), %rax - ret +5: /* Exception path */ + mov $-EFAULT, %ebx + stc + jmp 3b -.pushsection .fixup, "ax" - /* Re-load @exception_info and fill it (if it's non-NULL) */ -2: pop %rcx - test %rcx, %rcx - je 3f +6: /* Unsupported ENCLU leaf */ + cmp $0, %eax + jle 7f + mov $-EINVAL, %eax - mov %eax, EX_LEAF(%rcx) - mov %di, EX_TRAPNR(%rcx) - mov %si, EX_ERROR_CODE(%rcx) - mov %rdx, EX_ADDRESS(%rcx) -3: mov $(-EFAULT), %rax +7: /* Epilog */ + leave + .cfi_def_cfa %rsp, 8 ret -.popsection + .cfi_endproc -_ASM_VDSO_EXTABLE_HANDLE(1b, 2b) +_ASM_VDSO_EXTABLE_HANDLE(2b, 5b) ENDPROC(__vdso_sgx_enter_enclave)arch/x86/include/uapi/asm/sgx.h +0
-0
arch/x86/include/uapi/asm/sgx.h line 27
Quoted 5 lines
arch/x86/include/uapi/asm/sgx.h +1
-1@@ -24,7 +24,7 @@/** * struct sgx_enclave_create - parameter structure for the -* %SGX_IOC_ENCLAVE_CREATE ioctl +* %SGX_IOC_ENCLAVE_CREATE ioctl
You have bunch of these clutter diff's in your patch. Please get rid of them.
Quoted 22 lines
arch/x86/include/uapi/asm/sgx.h +2
-2* @src: address for the SECS page data */ struct sgx_enclave_create {@@ -33,7 +33,7 @@ struct sgx_enclave_create {/** * struct sgx_enclave_add_page - parameter structure for the -* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl +* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl * @addr: address within the ELRANGE * @src: address for the page data * @secinfo: address for the SECINFO data @@ -49,7 +49,7 @@ struct sgx_enclave_add_page {/** * struct sgx_enclave_init - parameter structure for the -* %SGX_IOC_ENCLAVE_INIT ioctl +* %SGX_IOC_ENCLAVE_INIT ioctl * @sigstruct: address for the SIGSTRUCT data */ struct sgx_enclave_init {
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_set_attribute { · line 70
Quoted 7 lines
arch/x86/include/uapi/asm/sgx.h +2
-2@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute { };/** -* struct sgx_enclave_exception - structure to report exceptions encountered in -* __vdso_sgx_enter_enclave() +* struct sgx_enclave_exinfo - structure to report exceptions encountered in +* __vdso_sgx_enter_enclave()
If you want to rename a struct it should be its own commit. Anyway, I'd say that this unnecessary.
Quoted 16 lines
arch/x86/include/uapi/asm/sgx.h +2
-2* -* @leaf: ENCLU leaf from \%eax at time of exception +* @leaf: ENCLU leaf from \%eax at time of exception/exit * @trapnr: exception trap number, a.k.a. fault vector * @error_code: exception error code * @address: exception address, e.g. CR2 on a #PF * @reserved: reserved for future use */ -struct sgx_enclave_exception { +struct sgx_enclave_exinfo {__u32 leaf;__u16 trapnr;__u16 error_code;Signature · 1 line
2.17.1
Summary: I can live with the general idea but the patch itself is somewhat half-finished still.
/Jarkko
Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
In reply to Cedric XingQuoted 1 line · Cedric Xing
-#ifdef
SGX_KERNEL_DOC
Why is this removed?
Quoted 16 lines
+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ * structsgx_enclave_exinfo*exinfo,
+ *sgx_callbackcallback)
+ * {
+ * while (leaf == EENTER || leaf == ERESUME) {
+ * int rc;
+ * try {
+ * ENCLU[leaf];
+ * rc = 0;
+ * if (exinfo)
+ *exinfo->leaf= EEXIT;
+ * } catch (exception) {
+ * rc =-EFAULT;
+ * if (exinfo)
+ * *exinfo = exception;
+ * }
Quoted 8 lines
+ *
+ * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ * r8, r9, tcs, ursp);
+ * }
+ *
+ * return leaf > 0 ?-EINVAL: leaf;
+ * }
*/
What is this? C++ and anyway there is already a source code. No need to duplicate with pseudo-code. Only adds maintenance burde. Please get rid of this.
/Jarkko
Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 30 lines · Jarkko Sakkinen
Quoted 1 line · Cedric Xing
-#ifdef
SGX_KERNEL_DOCWhy is this removed?
Quoted 16 lines
+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ * structsgx_enclave_exinfo*exinfo,
+ *sgx_callbackcallback)
+ * {
+ * while (leaf == EENTER || leaf == ERESUME) {
+ * int rc;
+ * try {
+ * ENCLU[leaf];
+ * rc = 0;
+ * if (exinfo)
+ *exinfo->leaf= EEXIT;
+ * } catch (exception) {
+ * rc =-EFAULT;
+ * if (exinfo)
+ * *exinfo = exception;
+ * }Quoted 8 lines
+ *
+ * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ * r8, r9, tcs, ursp);
+ * }
+ *
+ * return leaf > 0 ?-EINVAL: leaf;
+ * }
*/
Quoted 3 lines
What is this? C++ and anyway there is already a source code. No need to duplicate with pseudo-code. Only adds maintenance burde. Please get rid of this.
Adding C pseudo-code was my idea, e.g. it already exists in v20. Declaring a psuedo-C function coerces kernel-doc into generating documentation for the asm routine. IIRC, fully defining the function is not required for docs, but IMO it's significantly easier to gain an understanding of a blob of asm if there is higher level pseudocode.
Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 35 lines · Sean Christopherson
Quoted 30 lines · Jarkko Sakkinen
Quoted 1 line · Cedric Xing
-#ifdef
SGX_KERNEL_DOCWhy is this removed?
Quoted 16 lines
+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ * structsgx_enclave_exinfo*exinfo,
+ *sgx_callbackcallback)
+ * {
+ * while (leaf == EENTER || leaf == ERESUME) {
+ * int rc;
+ * try {
+ * ENCLU[leaf];
+ * rc = 0;
+ * if (exinfo)
+ *exinfo->leaf= EEXIT;
+ * } catch (exception) {
+ * rc =-EFAULT;
+ * if (exinfo)
+ * *exinfo = exception;
+ * }Quoted 8 lines
+ *
+ * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ * r8, r9, tcs, ursp);
+ * }
+ *
+ * return leaf > 0 ?-EINVAL: leaf;
+ * }
*/Quoted 3 lines
What is this? C++ and anyway there is already a source code. No need to duplicate with pseudo-code. Only adds maintenance burde. Please get rid of this.
Quoted 5 lines
Adding C pseudo-code was my idea, e.g. it already exists in v20. Declaring a psuedo-C function coerces kernel-doc into generating documentation for the asm routine. IIRC, fully defining the function is not required for docs, but IMO it's significantly easier to gain an understanding of a blob of asm if there is higher level pseudocode.
The way to do this right would be to write a documentation block before kdoc's for the functions. It is described in the last section of
https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt0
I.e. organize the file as
1. Documentation block describing the theory of operation.
2. Functions and their associated documentations.
/Jarkko
Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 45 lines · Jarkko Sakkinen
Quoted 35 lines · Sean Christopherson
Quoted 30 lines · Jarkko Sakkinen
Quoted 1 line · Cedric Xing
-#ifdef
SGX_KERNEL_DOCWhy is this removed?
Quoted 16 lines
+ * int __vdso_sgx_enter_enclave(int leaf, void *tcs,
+ * structsgx_enclave_exinfo*exinfo,
+ *sgx_callbackcallback)
+ * {
+ * while (leaf == EENTER || leaf == ERESUME) {
+ * int rc;
+ * try {
+ * ENCLU[leaf];
+ * rc = 0;
+ * if (exinfo)
+ *exinfo->leaf= EEXIT;
+ * } catch (exception) {
+ * rc =-EFAULT;
+ * if (exinfo)
+ * *exinfo = exception;
+ * }Quoted 8 lines
+ *
+ * leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo,
+ * r8, r9, tcs, ursp);
+ * }
+ *
+ * return leaf > 0 ?-EINVAL: leaf;
+ * }
*/Quoted 3 lines
What is this? C++ and anyway there is already a source code. No need to duplicate with pseudo-code. Only adds maintenance burde. Please get rid of this.
Quoted 5 lines
Adding C pseudo-code was my idea, e.g. it already exists in v20. Declaring a psuedo-C function coerces kernel-doc into generating documentation for the asm routine. IIRC, fully defining the function is not required for docs, but IMO it's significantly easier to gain an understanding of a blob of asm if there is higher level pseudocode.
The way to do this right would be to write a documentation block before kdoc's for the functions. It is described in the last section of
Quoted 6 lines
https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt0
I.e. organize the file as
1. Documentation block describing the theory of operation.
2. Functions and their associated documentations.
The kernel doc parser straight up doesn't work on asm functions, hence the shenanigans to coerce it into thinking it's parsing a normal C function.
Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 5 lines · Sean Christopherson
Quoted 5 lines
The way to do this right would be to write a documentation block before kdoc's for the functions. It is described in the last section of
https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt0
Quoted 7 lines
Quoted 4 lines
I.e. organize the file as
1. Documentation block describing the theory of operation.
2. Functions and their associated documentations.The kernel doc parser straight up doesn't work on asm functions, hence the shenanigans to coerce it into thinking it's parsing a normal C function.
Aah. I think I was too hazy with my comment. Looked it from patchwork and the documentation is mostly just fine.
For this patch this cruft nees to be removed:
- * %SGX_IOC_ENCLAVE_INIT ioctl
+ * %SGX_IOC_ENCLAVE_INIT ioctl
These make squashing the patch properly nasty.
Also there is one unwanted rename. Did not find anything else obviously wrong.
/Jarkko
Re: [RFC PATCH v3 2/3] x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
Quoted 19 lines · Jarkko Sakkinen
Quoted 5 lines · Sean Christopherson
Quoted 5 lines
The way to do this right would be to write a documentation block before kdoc's for the functions. It is described in the last section of
https://www.kernel.org/doc/Documentation/kernel-doc-nano-HOWTO.txt0
Quoted 7 lines
Quoted 4 lines
I.e. organize the file as
1. Documentation block describing the theory of operation.
2. Functions and their associated documentations.The kernel doc parser straight up doesn't work on asm functions, hence the shenanigans to coerce it into thinking it's parsing a normal C function.
Aah. I think I was too hazy with my comment. Looked it from patchwork and the documentation is mostly just fine.
For this patch this cruft nees to be removed:
Quoted 7 lines
- * %SGX_IOC_ENCLAVE_INIT ioctl
+ * %SGX_IOC_ENCLAVE_INIT ioctlThese make squashing the patch properly nasty.
Also there is one unwanted rename. Did not find anything else obviously wrong.
Reformatted the comments. Tested with kernel-doc. Now v4 could be converted and displayed nicely as man pages.
Quoted 2 lines
/Jarkko
RFC PATCH v3 3/3 selftests/x86: Augment SGX selftest to test new __vdso_sgx_enter_enclave() and its callback interface
In reply to Cedric XingThis patch augments SGX selftest with two new tests.
The first test exercises the newly added callback interface, by marking the whole enclave range as PROT_READ, then calling mprotect() upon #PFs to add necessary PTE permissions per PFEC (#PF Error Code) until the enclave finishes. This test also serves as an example to demonstrate the callback interface.
The second test single-steps through __vdso_sgx_enter_enclave() to make sure the call stack can be unwound at every instruction within that vDSO API. Its purpose is to validate the hand-crafted CFI directives in the assembly.
--- tools/testing/selftests/x86/sgx/Makefile | 6 +- tools/testing/selftests/x86/sgx/main.c | 323 ++++++++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 ++- 3 files changed, 322 insertions(+), 47 deletions(-)
tools/testing/selftests/x86/sgx/Makefile +3 -3
@@ -14,16 +14,16 @@ TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx all_64: $(TEST_CUSTOM_PROGS) $(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o - $(CC) $(HOST_CFLAGS) -o $@ $^ + $(CC) $(HOST_CFLAGS) -o $@ $^ -lunwind -ldl -Wl,--defsym,__image_base=0 -pie $(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss $(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@ $(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf - objcopy --remove-section=.got.plt -O binary $< $@ + objcopy -O binary $< $@ $(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S - $(CC) $(ENCL_CFLAGS) -T $^ -o $@ + $(CC) $(ENCL_CFLAGS) -T $^ -o $@ -Wl,--build-id=none $(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $^ $@
tools/testing/selftests/x86/sgx/main.c +282 -41
388 lines not on this page · raw message
tools/testing/selftests/x86/sgx/sgx_call.S +37 -3
@@ -7,9 +7,43 @@ .global sgx_call sgx_call: + .cfi_startproc + push %r15 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r15, 0 + push %r14 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r14, 0 + push %r13 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r13, 0 + push %r12 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %r12, 0 push %rbx - mov $0x02, %rax - mov %rdx, %rbx - call *%r8 + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbx, 0 + push $0 + .cfi_adjust_cfa_offset 8 + push 0x48(%rsp) + .cfi_adjust_cfa_offset 8 + push 0x48(%rsp) + .cfi_adjust_cfa_offset 8 + push 0x48(%rsp) + .cfi_adjust_cfa_offset 8 + mov $2, %eax + call *eenter(%rip) + add $0x20, %rsp + .cfi_adjust_cfa_offset -0x20 pop %rbx + .cfi_adjust_cfa_offset -8 + pop %r12 + .cfi_adjust_cfa_offset -8 + pop %r13 + .cfi_adjust_cfa_offset -8 + pop %r14 + .cfi_adjust_cfa_offset -8 + pop %r15 + .cfi_adjust_cfa_offset -8 ret + .cfi_endproc
Signature · 1 line
2.17.1
Re: [RFC PATCH v3 0/3] x86/sgx: Amend vDSO API to allow enclave/host parameter passing on untrusted stack
In reply to Cedric XingQuoted 38 lines · Cedric Xing
This patchset is based upon, and can be applied cleanly on SGX1 patch v20
(https://lkml.org/lkml/2019/4/17/344) by Jarkko Sakkinen.The current proposed
__vdso_sgx_enter_enclave()requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. After all, passing data via untrusted stack is very easy to implement (by enclaves), with essentially no overhead, therefore is very suitable for exchanging data in small amounts, so could be desirable by future SGX applications as well.This patchset introduces a new ABI for
__vdso_sgx_enter_enclave()to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. And in order to service o-calls and to preserve the untrusted stack upon exceptions, the new vDSO API takes one more optional parameter - "callback", which if supplied, will be invoked on all enclave exits (including normal and asynchronous exits). Ample details regarding the new ABI have been documented as comments inside the source code located inarch/x86/entry/vsgx_enter_enclave.SPlease note that there was a lengthy discussion on what is the "best" approach for passing parameters for trusted/untrusted calls. Unfortunately there's no single "best" approach that fits all use cases, hence this new ABI has been designed intentionally to accommodate varieties. Therefore, to those not interested in using the untrusted stack, whatever worked with the old ABI proposed by Sean will continue to work with this new ABI.
The SGX selftest has been augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in
__vdso_sgx_enter_enclave()by single-stepping through it and unwinding call stack at every instruction. Please note that the selftest CANNOT run to completion yet, as it depends on the vDSO fixup code to signal the process upon #DB/#BP inside enclaves (rather than the current behavior of branching to the handler in vDSO).
Quoted 8 lines
Changelog:
· This is version 3 of this patch series with the following changes.
- Per Andy Lutomirski and Sean Christopherson, revised comments and their format inarch/x86/entry/vsgx_enter_enclave.S
- Per Jarkko Sakkinen, revised the cover letter to articulate motivation and objective of this patchset.
· v2 - https://patchwork.kernel.org/cover/10914161/
· v1 - https://patchwork.kernel.org/cover/10911615/
1. I agree with the high level idea.
2. The patches do changes out of scope.
Generally, when doing kernel patches, even for a running patch set, please do not do anything extra.
It is also stated in the kernel process;
https://www.kernel.org/doc/html/v4.17/process/submitting-patches.html#separate-your-changes
Once these are fully cleaned up we can merge them.
/Jarkko
v4 cover x86/sgx: Amend vDSO API to allow enclave/host parameter passing on untrusted stack
In reply to Cedric XingThis patchset is based upon, and can be applied cleanly on SGX1 patch v20
(https://lkml.org/lkml/2019/4/17/344) by Jarkko Sakkinen.
The current proposed __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating space on the untrusted stack. However, there are existing enclaves (e.g. those built with current Intel SGX SDK libraries) relying on the untrusted stack for passing parameters to untrusted functions (aka. o-calls), which requires allocating space on the untrusted stack by enclaves. After all, passing data via untrusted stack is very easy to implement (by enclaves), with essentially no overhead, therefore is very suitable for exchanging data in small amounts, so could be desirable by future SGX applications as well.
This patchset introduces a new ABI for __vdso_sgx_enter_enclave() to anchor its stack frame on %rbp (instead of %rsp), so as to allow enclaves to "push" onto the untrusted stack by decrementing the untrusted %rsp. And in order to service o-calls and to preserve the untrusted stack upon exceptions, the new vDSO API takes one more optional parameter - "callback", which if supplied, will be invoked on all enclave exits (including normal and asynchronous exits). Ample details regarding the new ABI have been documented as comments inside the source code located in arch/x86/entry/vsgx_enter_enclave.S
Please note that there was a lengthy discussion on what is the "best" approach for passing parameters for trusted/untrusted calls. Unfortunately there's no single "best" approach that fits all use cases, hence this new ABI has been designed intentionally to accommodate varieties. Therefore, to those not interested in using the untrusted stack, whatever worked with the old ABI proposed by Sean will continue to work with this new ABI.
The SGX selftest has been augmented by two new tests. One exercises the new callback interface, and serves as a simple example to showcase how to use it; while the other validates the hand-crafted CFI directives in __vdso_sgx_enter_enclave() by single-stepping through it and unwinding call stack at every instruction.
Changelog:
· This is version 4 of this patch series with the following changes.
- Removed unrelated cosmetic changes.
- Rewrote and reformatted comments in arch/x86/entry/vdso/vsgx_enter_enclave.S to follow kernel-doc conventions. New comments now can be converted to nice looking man pages.
- Fixed minor issues in the unwinding selftest and now it can run to completion successfully with Sean's fix in vDSO fixup code
(https://patchwork.kernel.org/patch/11040801/). Comments have also been added to describe the tests done.
· v3 - https://patchwork.kernel.org/cover/11039263/
· v2 - https://patchwork.kernel.org/cover/10914161/
· v1 - https://patchwork.kernel.org/cover/10911615/
Cedric Xing (3):
selftests/x86/sgx: Fix Makefile for SGX selftest
x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing
on untrusted stack
selftests/x86/sgx: Augment SGX selftest to test vDSO API
arch/x86/entry/vdso/vsgx_enter_enclave.S | 310 ++++++++++++++----- arch/x86/include/uapi/asm/sgx.h | 14 +- tools/testing/selftests/x86/sgx/Makefile | 49 ++- tools/testing/selftests/x86/sgx/main.c | 344 ++++++++++++++++++--- tools/testing/selftests/x86/sgx/sgx_call.S | 40 ++- 5 files changed, 600 insertions(+), 157 deletions(-)
Signature · 1 line
2.17.1
RFC PATCH v4 1/3 selftests/x86/sgx: Fix Makefile for SGX selftest
The original x86/sgx/Makefile didn't work when "x86/sgx" was specified as the test target, nor did it work with "run_tests" as the make target. Yet another problem was that it breaks 32-bit only build. This patch fixes those problems, along with adjustments to compiler/linker options and simplifications to the build rules.
--- tools/testing/selftests/x86/sgx/Makefile | 45 +++++++++--------------- 1 file changed, 16 insertions(+), 29 deletions(-)
tools/testing/selftests/x86/sgx/Makefile +16 -29
@@ -2,47 +2,34 @@ top_srcdir = ../../../../.. include ../../lib.mk -HOST_CFLAGS := -Wall -Werror -g $(INCLUDES) -fPIC -ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIC \ +ifeq ($(shell $(CC) -dumpmachine | cut --delimiter=- -f1),x86_64) +all: all_64 +endif + +HOST_CFLAGS := -Wall -Werror -g $(INCLUDES) +ENCL_CFLAGS := -Wall -Werror -static -nostdlib -nostartfiles -fPIE \ -fno-stack-protector -mrdrnd $(INCLUDES) TEST_CUSTOM_PROGS := $(OUTPUT)/test_sgx all_64: $(TEST_CUSTOM_PROGS) -$(TEST_CUSTOM_PROGS): $(OUTPUT)/main.o $(OUTPUT)/sgx_call.o \ - $(OUTPUT)/encl_piggy.o +$(TEST_CUSTOM_PROGS): main.c sgx_call.S $(OUTPUT)/encl_piggy.o $(CC) $(HOST_CFLAGS) -o $@ $^ -$(OUTPUT)/main.o: main.c - $(CC) $(HOST_CFLAGS) -c $< -o $@ - -$(OUTPUT)/sgx_call.o: sgx_call.S - $(CC) $(HOST_CFLAGS) -c $< -o $@ - -$(OUTPUT)/encl_piggy.o: $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss - $(CC) $(HOST_CFLAGS) -c encl_piggy.S -o $@ +$(OUTPUT)/encl_piggy.o: encl_piggy.S $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss + $(CC) $(HOST_CFLAGS) -I$(OUTPUT) -c $< -o $@ -$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf $(OUTPUT)/sgxsign +$(OUTPUT)/encl.bin: $(OUTPUT)/encl.elf objcopy --remove-section=.got.plt -O binary $< $@ -$(OUTPUT)/encl.elf: $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o - $(CC) $(ENCL_CFLAGS) -T encl.lds -o $@ $^ +$(OUTPUT)/encl.elf: encl.lds encl.c encl_bootstrap.S + $(CC) $(ENCL_CFLAGS) -T $^ -o $@ -$(OUTPUT)/encl.o: encl.c - $(CC) $(ENCL_CFLAGS) -c $< -o $@ - -$(OUTPUT)/encl_bootstrap.o: encl_bootstrap.S - $(CC) $(ENCL_CFLAGS) -c $< -o $@ - -$(OUTPUT)/encl.ss: $(OUTPUT)/encl.bin $(OUTPUT)/sgxsign - $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss +$(OUTPUT)/encl.ss: $(OUTPUT)/sgxsign signing_key.pem $(OUTPUT)/encl.bin + $^ $@ $(OUTPUT)/sgxsign: sgxsign.c $(CC) -o $@ $< -lcrypto -EXTRA_CLEAN := $(OUTPUT)/sgx-selftest $(OUTPUT)/sgx-selftest.o \ - $(OUTPUT)/sgx_call.o $(OUTPUT)/encl.bin $(OUTPUT)/encl.ss \ - $(OUTPUT)/encl.elf $(OUTPUT)/encl.o $(OUTPUT)/encl_bootstrap.o \ - $(OUTPUT)/sgxsign - -.PHONY: clean +EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(addprefix $(OUTPUT)/, \ + encl.elf encl.bin encl.ss encl_piggy.o sgxsign)
Signature · 1 line
2.17.1
Re: [RFC PATCH v4 1/3] selftests/x86/sgx: Fix Makefile for SGX selftest
Quoted 7 lines · Cedric Xing
The original x86/sgx/Makefile didn't work when "x86/sgx" was specified as the test target, nor did it work with "
run_tests" as the make target. Yet another problem was that it breaks 32-bit only build. This patch fixes those problems, along with adjustments to compiler/linker options and simplifications to the build rules.Signed-off-by Cedric Xing <[email protected]>
You must split this in quite a few separate patches:
1. One for each fix.
2. At least one patch for each change in compiler and linker options with a commit message clearly expalaining why the change was made.
3. One for each simplification.
We don't support 32-bit build:
config INTEL_SGX
bool "Intel SGX core functionality" depends on X86_64 && CPU_SUP_INTEL
/Jarkko
Re: [RFC PATCH v4 1/3] selftests/x86/sgx: Fix Makefile for SGX selftest
Quoted 16 lines · Jarkko Sakkinen
Quoted 7 lines · Cedric Xing
The original x86/sgx/Makefile didn't work when "x86/sgx" was specified as the test target, nor did it work with "
run_tests" as the make target. Yet another problem was that it breaks 32-bit only build. This patch fixes those problems, along with adjustments to compiler/linker options and simplifications to the build rules.Signed-off-by Cedric Xing <[email protected]>You must split this in quite a few separate patches:
1. One for each fix.
2. At least one patch for each change in compiler and linker options with a commit message clearly expalaining why the change was made.
3. One for each simplification.
Quoted 5 lines
We don't support 32-bit build:
config
INTEL_SGXbool "Intel SGX core functionality" depends on X86_64 && CPU_SUP_INTEL
This is not to say that changes suck. This just in "unreviewable" state as far as the kernel processes go...
/Jarkko
Re: [RFC PATCH v4 1/3] selftests/x86/sgx: Fix Makefile for SGX selftest
Quoted 17 lines · Jarkko Sakkinen
Quoted 16 lines · Jarkko Sakkinen
Quoted 7 lines · Cedric Xing
The original x86/sgx/Makefile didn't work when "x86/sgx" was specified as the test target, nor did it work with "
run_tests" as the make target. Yet another problem was that it breaks 32-bit only build. This patch fixes those problems, along with adjustments to compiler/linker options and simplifications to the build rules.Signed-off-by Cedric Xing <[email protected]>You must split this in quite a few separate patches:
1. One for each fix.
2. At least one patch for each change in compiler and linker options with a commit message clearly expalaining why the change was made.
3. One for each simplification.
Quoted 8 lines
Quoted 5 lines
We don't support 32-bit build:
config
INTEL_SGXbool "Intel SGX core functionality" depends on X86_64 && CPU_SUP_INTELThis is not to say that changes suck. This just in "unreviewable" state as far as the kernel processes go...
Please note that your patchset hasn't been upstreamed yet. Your Makefile is problematic to begin with. Technically it's your job to make it work before sending out any patches. You didn't explain what's done for each line of Makefile in your commit message either.
Not saying documentation is unimportant, but the purposes for those changes are obvious and easy to understand for anyone having reasonable knowledge on how Makefile works.
I'm totally fine not fixing the Makefile. You can just leave them out.
Quoted 2 lines
/Jarkko
Re: [RFC PATCH v4 1/3] selftests/x86/sgx: Fix Makefile for SGX selftest
Quoted 4 lines · Cedric
Please note that your patchset hasn't been upstreamed yet. Your
Makefileis problematic to begin with. Technically it's your job to make it work before sending out any patches. You didn't explain what's done for each line ofMakefilein your commit message either.
Yes, it is different case to do the initial version of the whole thing that suggest fixes to it. The latter needs to have more granularity. Bug fixes in any type of software development should be isolated to separate change sets. It is just a sane QA practice.
Quoted 5 lines
Not saying documentation is unimportant, but the purposes for those changes are obvious and easy to understand for anyone having reasonable knowledge on how
Makefileworks.I'm totally fine not fixing the
Makefile. You can just leave them out.
/Jarkko
RFC PATCH v4 2/3 x86/vdso: Modify __vdso_sgx_enter_enclave() to allow parameter passing on untrusted stack
In reply to Cedric XingThe previous __vdso_sgx_enter_enclave() requires enclaves to preserve %rsp, which prohibits enclaves from allocating and passing parameters for untrusted function calls (aka. o-calls) on the untrusted stack.
This patch addresses the problem above by introducing a new ABI that preserves %rbp instead of %rsp. Then __vdso_sgx_enter_enclave() can anchor its frame using %rbp so that enclaves are allowed to allocate space on the untrusted stack by decrementing %rsp. Please note that the stack space allocated in such way will be part of __vdso_sgx_enter_enclave()'s frame so will be freed after __vdso_sgx_enter_enclave() returns. Therefore, __vdso_sgx_enter_enclave() has been revised to take a callback function as an optional parameter, which if supplied, will be invoked upon enclave exits (both AEX (Asynchronous Enclave eXit) and normal exits), with the value of %rsp left off by the enclave as a parameter to the callback.
Here's the summary of API/ABI changes in this patch. More details could be found in arch/x86/entry/vdso/vsgx_enter_enclave.S.
* 'struct sgx_enclave_exception' is renamed to 'struct sgx_enclave_exinfo' because it is filled upon both AEX (i.e. exceptions) and normal enclave exits.
* __vdso_sgx_enter_enclave() anchors its frame using %rbp (instead of %rsp in the previous implementation).
* __vdso_sgx_enter_enclave() takes one more parameter - a callback function to be invoked upon enclave exits. This callback is optional, and if not supplied, will cause __vdso_sgx_enter_enclave() to return upon enclave exits (same behavior as previous implementation).
* The callback function is given as a parameter the value of %rsp at enclave exit to address data "pushed" by the enclave. A positive value returned by the callback will be treated as an ENCLU leaf for re-entering the enclave, while a zero or negative value will be passed through as the return value of __vdso_sgx_enter_enclave() to its caller. It's also safe to leave callback by longjmp() or by throwing a C++ exception.
--- arch/x86/entry/vdso/vsgx_enter_enclave.S | 310 +++++++++++++++++------ arch/x86/include/uapi/asm/sgx.h | 14 +- 2 files changed, 242 insertions(+), 82 deletions(-)
arch/x86/entry/vdso/vsgx_enter_enclave.S +235 -75
@@ -6,96 +6,256 @@ #include "extable.h" -#define EX_LEAF 0*8 -#define EX_TRAPNR 0*8+4 -#define EX_ERROR_CODE 0*8+6 -#define EX_ADDRESS 1*8 +#define EX_LEAF 0*8 +#define EX_TRAPNR 0*8+4 +#define EX_ERROR_CODE 0*8+6 +#define EX_ADDRESS 1*8 .code64 .section .text, "ax" #ifdef SGX_KERNEL_DOC /** - * __vdso_sgx_enter_enclave() - Enter an SGX enclave + * typedef sgx_ex_callback - Callback function for __vdso_sgx_enter_enclave() * - * @leaf: **IN \%eax** - ENCLU leaf, must be EENTER or ERESUME - * @tcs: **IN \%rbx** - TCS, must be non-NULL - * @ex_info: **IN \%rcx** - Optional 'struct sgx_enclave_exception' pointer + * @rdi: value of %%rdi register at enclave exit + * @rsi: value of %%rsi register at enclave exit + * @rdx: value of %%rdx register at enclave exit + * @exinfo: pointer to a sgx_enclave_exinfo structure, which was passed to + * __vdso_sgx_enter_enclave() as input + * @r8: value of %%r8 register at enclave exit + * @r9: value of %%r9 register at enclave exit + * @tcs: TCS used by __vdso_sgx_enter_enclave() to enter the enclave, + * could be used to re-enter the + * enclave + * @ursp: value of %%rsp register at enclave exit + * + * This is the callback function to be invoked upon enclave exits, including + * normal exits (as result of EEXIT), and asynchronous exits (AEX) due to + * exceptions occurred at EENTER or within the enclave. + * + * This callback is expected to follow x86_64 ABI. * * Return: - * **OUT \%eax** - - * %0 on a clean entry/exit to/from the enclave, %-EINVAL if ENCLU leaf is - * not allowed or if TCS is NULL, %-EFAULT if ENCLU or the enclave faults - * - * **Important!** __vdso_sgx_enter_enclave() is **NOT** compliant with the - * x86-64 ABI, i.e. cannot be called from standard C code. As noted above, - * input parameters must be passed via ``%eax``, ``%rbx`` and ``%rcx``, with - * the return value passed via ``%eax``. All registers except ``%rsp`` must - * be treated as volatile from the caller's perspective, including but not - * limited to GPRs, EFLAGS.DF, MXCSR, FCW, etc... Conversely, the enclave - * being run **must** preserve the untrusted ``%rsp`` and stack. + * + * EENTER(2) - causes __vdso_sgx_enter_enclave() to issue ENCLU[EENTER] on the + * same TCS. All GPRs left off by this callback function will be passed through + * back to the enclave, except %%rax, %%rbx and %%rcx, which are clobbered by + * ENCLU[EENTER] instruction. + * + * ERESUME(3) - causes __vdso_sgx_enter_enclave() to issue ENCLU[ERESUME] on + * the same TCS. + * + * 0 (zero) or negative returned values will be returned back to + * __vdso_sgx_enter_enclave()'s caller as is. + * + * All other values will cause -EINVAL to be returned to + * __vdso_sgx_enter_enclave()'s caller. + * + * Note: All general purpose registers (GPRs) left off by the enclave are + * passed through to this function, except %%rax, %%rbx and %%rcx, which are + * used internally by __vdso_sgx_enter_enclave(). Some of those registers are + * accessible as function parameters (i.e. @rdi, @rsi, @rdx, @r8, @r9 and + * @ursp), while others can be accessed only from assembly code. */ -__vdso_sgx_enter_enclave(u32 leaf, void *tcs, - struct sgx_enclave_exception *ex_info) -{ - if (leaf != SGX_EENTER && leaf != SGX_ERESUME) - return -EINVAL; +typedef int sgx_ex_callback(long rdi, long rsi, long rdx, + struct sgx_enclave_exinfo *exinfo, + long r8, long r9, void *tcs, long ursp); - if (!tcs) - return -EINVAL; +/** + * __vdso_sgx_enter_enclave() - Enter an SGX enclave and capture exceptions + * + * @leaf: + * passed in %%eax, must be either EENTER(2) or ERESUME(3) + * @tcs: + * passed on stack at 8(%%rsp), is the linear address of TCS + * @exinfo: + * passed on stack at 0x10(%%rsp), optional, and if non-NULL, shall point + * to an sgx_enclave_exinfo structure to receive information about the + * enclave exit + * @callback: + * passed on stack at 0x18(%%rsp), optional, and if non-NULL, points to a + * callback function to be invoked at enclave exits + * + * __vdso_sgx_enter_enclave() issues either ENCLU[EENTER] or ENCLU[ERESUME] on + * @tcs depending on @leaf. + * + * IMPORTANT! This API is not compliant with x86-64 ABI but adopts a + * proprietary calling convention. Please see NOTES section below for details. + * + * On an enclave exit, @exinfo->leaf will be set to the ENCLU leaf at exit, if + * @exinfo is not NULL. That is, @exinfo->leaf may be one of the following: + * + * * EEXIT: Normal exit due to ENCLU[EEXIT] within the enclave. All other + * members will remain intact. + * + * * ERESUME: Asynchronous exit due to exceptions within the enclave. + * @exinfo->trapnr, @exinfo->error_code and @exinfo->address are + * set to the trap number, error code and fault address, + * respectively. + * + * * EENTER: Exception occurred when trying to enter the enclave. + * @exinfo->trapnr, @exinfo->error_code and @exinfo->address are + * set to the trap number, error code and fault address, + * accordingly. + * + * If @callback is NULL, 0 (zero) is returned if the enclave has been entered + * and exited normally, or -EFAULT if any exception has occurred, or -EINVAL if + * @leaf on input is neither EENTER or ERESUME. + * + * If @callback is not NULL, it is invoked at enclave exit, and then actions + * will be taken depending on its return value - i.e. positive value will be + * treated as ENCLU leaf to re-enter the enclave, while 0 (zero) or negative + * values will be returned back to the caller as is. Unrecognized leaf values + * will cause -EINVAL to be returned. + * + * Return: + * + * 0 (zero) is returned on a successful entry and normal exit from the enclave. + * + * -EINVAL is returned if @leaf is neither EENTER nor ERESUME, or if @callback + * is not NULL and returns a positive value that is neither EENTER nor ERESUME + * after the enclave exits. + * + * -EFAULT is returned if an exception has occurred at EENTER or during + * execution of the enclave and @callback is NULL, or if @callback is not NULL + * and it returns -EFAULT after the enclave exits. + * + * Other values may be returned as the return value from @callback if it is not + * NULL. + * + * Note: __vdso_sgx_enter_enclave() adopts a proprietary calling convention, + * described below: + * + * * As noted above, input parameters are passed via %%eax and the stack. + * + * * %%rbx and %%rcx must be treated as volatile as they are modified as part + * of enclaves transitions and are used as scratch regs. + * + * * %%rdx, %%rdi, %%rsi and %%r8-%%r15 are passed as is and may be freely + * modified by the enclave. Values left in those registers will not be + * altered either, so will be visiable to the callback or the caller (if no + * callback is specified). + * + * * %%rsp could be decremented by the enclave to allocate temporary space on + * the untrusted stack. Temporary space allocated this way is retained in + * the context of @callback, and will be freed (i.e. %%rsp will be + * restored) before __vdso_sgx_enter_enclave() returns. + */ +int __vdso_sgx_enter_enclave(int leaf, void *tcs, + struct sgx_enclave_exinfo *exinfo, + sgx_ex_callback *callback); +{ + while (leaf == EENTER || leaf == ERESUME) { + int rc; + try { + ENCLU[leaf]; + rc = 0; + if (exinfo) + exinfo->leaf = EEXIT; + } catch (exception) { + rc = -EFAULT; + if (exinfo) + *exinfo = exception; + } - try { - ENCLU[leaf]; - } catch (exception) { - if (e) - *e = exception; - return -EFAULT; - } + leaf = !callback ? rc: (*callback)(rdi, rsi, rdx, exinfo, + r8, r9, tcs, ursp); + } - return 0; + return leaf > 0 ? -EINVAL : leaf; } #endif + ENTRY(__vdso_sgx_enter_enclave) - /* EENTER <= leaf <= ERESUME */ - cmp $0x2, %eax - jb bad_input - - cmp $0x3, %eax - ja bad_input - - /* TCS must be non-NULL */ - test %rbx, %rbx - je bad_input - - /* Save @exception_info */ - push %rcx - - /* Load AEP for ENCLU */ - lea 1f(%rip), %rcx -1: enclu - - add $0x8, %rsp - xor %eax, %eax - ret - -bad_input: - mov $(-EINVAL), %rax - ret - -.pushsection .fixup, "ax" - /* Re-load @exception_info and fill it (if it's non-NULL) */ -2: pop %rcx - test %rcx, %rcx - je 3f - - mov %eax, EX_LEAF(%rcx) - mov %di, EX_TRAPNR(%rcx) - mov %si, EX_ERROR_CODE(%rcx) - mov %rdx, EX_ADDRESS(%rcx) -3: mov $(-EFAULT), %rax - ret -.popsection - -_ASM_VDSO_EXTABLE_HANDLE(1b, 2b) + /* Prolog */ + .cfi_startproc + push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + +1: /* EENTER <= leaf <= ERESUME */ + cmp $0x2, %eax + jb 6f + cmp $0x3, %eax + ja 6f + + /* Load TCS and AEP */ + mov 0x10(%rbp), %rbx + lea 2f(%rip), %rcx + + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +2: enclu + + /* EEXIT path */ + xor %ebx, %ebx +3: mov 0x18(%rbp), %rcx + jrcxz 4f + mov %eax, EX_LEAF(%rcx) + jnc 4f + mov %di, EX_TRAPNR(%rcx) + mov %si, EX_ERROR_CODE(%rcx) + mov %rdx, EX_ADDRESS(%rcx) + +4: /* Call *callback if supplied */ + mov 0x20(%rbp), %rax + test %rax, %rax + /* + * At this point, %ebx holds the effective return value, which shall be + * returned if no callback is specified + */ + cmovz %rbx, %rax + jz 7f + /* + * Align stack per x86_64 ABI. The original %rsp is saved in %rbx to be + * restored after *callback returns. + */ + mov %rsp, %rbx + and $-0x10, %rsp + /* Clear RFLAGS.DF per x86_64 ABI */ + cld + /* Parameters for *callback */ + push %rbx + push 0x10(%rbp) + /* Call *%rax via retpoline */ + call 40f + /* + * Restore %rsp to its original value left off by the enclave from last + * exit + */ + mov %rbx, %rsp + /* + * Positive return value from *callback will be interpreted as an ENCLU + * leaf, while a non-positive value will be interpreted as the return + * value to be passed back to the caller. + */ + jmp 1b +40: /* retpoline */ + call 42f +41: pause + lfence + jmp 41b +42: mov %rax, (%rsp) + ret + +5: /* Exception path */ + mov $-EFAULT, %ebx + stc + jmp 3b + +6: /* Unsupported ENCLU leaf */ + cmp $0, %eax + jle 7f + mov $-EINVAL, %eax + +7: /* Epilog */ + leave + .cfi_def_cfa %rsp, 8 + ret + .cfi_endproc + +_ASM_VDSO_EXTABLE_HANDLE(2b, 5b) ENDPROC(__vdso_sgx_enter_enclave)
arch/x86/include/uapi/asm/sgx.h +7 -7
@@ -24,7 +24,7 @@/** * struct sgx_enclave_create - parameter structure for the -* %SGX_IOC_ENCLAVE_CREATE ioctl +* %SGX_IOC_ENCLAVE_CREATE ioctl * @src: address for the SECS page data */ struct sgx_enclave_create {
@@ -33,7 +33,7 @@ struct sgx_enclave_create {/** * struct sgx_enclave_add_page - parameter structure for the -* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl +* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl * @addr: address within the ELRANGE * @src: address for the page data * @secinfo: address for the SECINFO data
@@ -49,7 +49,7 @@ struct sgx_enclave_add_page {/** * struct sgx_enclave_init - parameter structure for the -* %SGX_IOC_ENCLAVE_INIT ioctl +* %SGX_IOC_ENCLAVE_INIT ioctl * @sigstruct: address for the SIGSTRUCT data */ struct sgx_enclave_init {
@@ -66,16 +66,16 @@ struct sgx_enclave_set_attribute { };/** -* struct sgx_enclave_exception - structure to report exceptions encountered in -* __vdso_sgx_enter_enclave() +* struct sgx_enclave_exinfo - structure to report exceptions encountered in +* __vdso_sgx_enter_enclave() * -* @leaf: ENCLU leaf from \%eax at time of exception +* @leaf: ENCLU leaf from \%eax at time of exception/exit * @trapnr: exception trap number, a.k.a. fault vector * @error_code: exception error code * @address: exception address, e.g. CR2 on a #PF * @reserved: reserved for future use */ -struct sgx_enclave_exception { +struct sgx_enclave_exinfo {__u32 leaf;__u16 trapnr;__u16 error_code;
Signature · 1 line
2.17.1