x86/sgx and selftests/sgx: Support SGX2
Subsystems (4)
PATCH v2 17/32 selftests/sgx: Add test for EPCM permission changes
In reply to Reinette ChatreEPCM permission changes could be made from within (to relax permissions) or out (to restrict permissions) the enclave. Kernel support is needed when permissions are restricted to be able to call the privileged ENCLS[EMODPR] instruction and ensure PTEs allowing the restricted permissions are flushed. EPCM permissions can be relaxed via ENCLU[EMODPE] from within the enclave but the enclave still depends on the kernel to install PTEs with the new permissions.
Add a test that exercises a few of the enclave page permission flows:
1) Test starts with a RW (from enclave and kernel perspective) enclave page that is mapped via a RW VMA.
2) Use the SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS ioctl() to restrict the enclave (EPCM) page permissions to read-only (kernel removes PTE in the process).
3) Run ENCLU[EACCEPT] from within the enclave to accept the new page permissions.
4) Attempt to write to the enclave page from within the enclave - this should fail with a page fault on the PTE since the page table entry accurately reflects the (read-only) EPCM permissions.
5) Restore EPCM permissions to RW by running ENCLU[EMODPE] from within the enclave.
6) Attempt to write to the enclave page from within the enclave - this should fail again with a page fault because even though the EPCM permissions are RW the PTE does not yet reflect that.
7) Use the SGX_IOC_ENCLAVE_RELAX_PERMISSIONS ioctl() to inform the kernel of new page permissions and PTEs will accurately reflect RW EPCM permissions.
8) Writing to enclave page from within enclave succeeds.
---
Changes since V1:
- Adapt test to the kernel interface changes: the ioctl() name change and providing entire secinfo as parameter.
- Remove the ENCLU[EACCEPT] call after permissions are relaxed since the new flow no longer results in the EPCM PR bit being set.
- Rewrite error path to reduce line lengths.
tools/testing/selftests/sgx/defines.h | 15 ++ tools/testing/selftests/sgx/main.c | 253 ++++++++++++++++++++++++ tools/testing/selftests/sgx/test_encl.c | 38 ++++ 3 files changed, 306 insertions(+)
tools/testing/selftests/sgx/defines.h +15 -0
@@ -24,6 +24,8 @@ enum encl_op_type {ENCL_OP_PUT_TO_ADDRESS ,ENCL_OP_GET_FROM_ADDRESS ,ENCL_OP_NOP , +ENCL_OP_EACCEPT , +ENCL_OP_EMODPE , ENCL_OP_MAX, };
@@ -53,4 +55,17 @@ struct encl_op_get_from_addr { uint64_t addr; }; +struct encl_op_eaccept { +struct encl_op_header header; +uint64_t epc_addr; +uint64_t flags; +uint64_t ret; +}; + +struct encl_op_emodpe { +struct encl_op_header header; +uint64_t epc_addr; +uint64_t flags; +}; +#endif /* DEFINES_H */
tools/testing/selftests/sgx/main.c +253 -0
263 lines not on this page · raw message
tools/testing/selftests/sgx/test_encl.c +38 -0
@@ -11,6 +11,42 @@*/ static uint8_t encl_buffer[8192 ] = {1 }; +enum sgx_enclu_function { +EACCEPT =0x5 , +EMODPE =0x6 , +}; + +static void do_encl_emodpe (void *_op) +{ +struct sgx_secinfo secinfo__aligned(sizeof(struct sgx_secinfo)) = {0 }; +struct encl_op_emodpe *op = _op; + + secinfo.flags = op->flags; + + asmvolatile (".byte 0x0f, 0x01, 0xd7" + : + :"a" (EMODPE ), +"b" (&secinfo), +"c" (op->epc_addr)); +} + +static void do_encl_eaccept (void *_op) +{ +struct sgx_secinfo secinfo__aligned(sizeof(struct sgx_secinfo)) = {0 }; +struct encl_op_eaccept *op = _op; +int rax; + + secinfo.flags = op->flags; + + asmvolatile (".byte 0x0f, 0x01, 0xd7" + :"=a" (rax) + :"a" (EACCEPT ), +"b" (&secinfo), +"c" (op->epc_addr)); + + op->ret = rax; +} +static void *memcpy (void *dest,const void *src,size_t n) {size_t i;
@@ -62,6 +98,8 @@ void encl_body(void *rdi, void *rsi) do_encl_op_put_to_addr, do_encl_op_get_from_addr, do_encl_op_nop, + do_encl_eaccept, + do_encl_emodpe, };struct encl_op_header *op = (struct encl_op_header *)rdi;
Signature · 1 line
2.25.1
PATCH v2 18/32 selftests/sgx: Add test for TCS page permission changes
In reply to Reinette ChatreKernel should not allow permission changes on TCS pages. Add test to confirm this behavior.
---
Changes since V1:
- Adapt test to the kernel interface changes: the ioctl() name change and providing entire secinfo as parameter.
- Rewrite error path to reduce line lengths.
tools/testing/selftests/sgx/main.c | 74 ++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+)
tools/testing/selftests/sgx/main.c +74 -0
@@ -121,6 +121,24 @@ static Elf64_Sym *vdso_symtab_get(struct vdso_symtab *symtab, const char *name)return NULL ; } +/* +* Return the offset in the enclave where the TCS segment can be found. +* The first RW segment loaded is the TCS. +*/ +static off_t encl_get_tcs_offset (struct encl *encl) +{ +int i; + +for (i =0 ; i < encl->nr_segments; i++) { +struct encl_segment *seg = &encl->segment_tbl[i]; + +if (i ==0 && seg->prot == (PROT_READ |PROT_WRITE )) +return seg->offset; + } + +return -1 ; +} +/* * Return the offset in the enclave where the data segment can be found. * The first RW segment loaded is the TCS, skip that to get info on the
@@ -567,6 +585,62 @@ TEST_F(enclave, pte_permissions)EXPECT_EQ (self->run.exception_addr,0 ); } +/* +* Modifying permissions of TCS page should not be possible. +*/ +TEST_F(enclave, tcs_permissions) +{ +struct sgx_enclave_restrict_perm ioc; +struct sgx_secinfo secinfo; +int ret, errno_save; + +ASSERT_TRUE (setup_test_encl (ENCL_HEAP_SIZE_DEFAULT , &self->encl, _metadata)); + +memset (&self->run,0 ,sizeof (self->run)); + self->run.tcs = self->encl.encl_base; + +memset (&ioc,0 ,sizeof (ioc)); +memset (&secinfo,0 ,sizeof (secinfo)); + +/* +* Ensure kernel supports needed ioctl() and system supports needed +* commands. +*/ + + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS , &ioc); + errno_save = ret ==-1 ? errno :0 ; + +/* +* Invalid parameters were provided during sanity check, +* expect command to fail. +*/ +ASSERT_EQ (ret,-1 ); + +/* ret == -1 */ +if (errno_save ==ENOTTY ) +SKIP (return, +"Kernel does not support SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS ioctl()" ); +else if (errno_save ==ENODEV ) +SKIP (return,"System does not support SGX2" ); + +/* +* Attempt to make TCS page read-only. This is not allowed and +* should be prevented by the kernel. +*/ + secinfo.flags =PROT_READ ; + ioc.offset =encl_get_tcs_offset (&self->encl); + ioc.length =PAGE_SIZE ; + ioc.secinfo = (unsigned long )&secinfo; + + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS , &ioc); + errno_save = ret ==-1 ? errno :0 ; + +EXPECT_EQ (ret,-1 ); +EXPECT_EQ (errno_save,EINVAL ); +EXPECT_EQ (ioc.result,0 ); +EXPECT_EQ (ioc.count,0 ); +} +/* * Enclave page permission test. *
Signature · 1 line
2.25.1
PATCH v2 19/32 x86/sgx: Support adding of pages to an initialized enclave
In reply to Reinette ChatreWith SGX1 an enclave needs to be created with its maximum memory demands allocated. Pages cannot be added to an enclave after it is initialized. SGX2 introduces a new function, ENCLS[EAUG], that can be used to add pages to an initialized enclave. With SGX2 the enclave still needs to set aside address space for its maximum memory demands during enclave creation, but all pages need not be added before enclave initialization. Pages can be added during enclave runtime.
Add support for dynamically adding pages to an initialized enclave, architecturally limited to RW permission. Add pages via the page fault handler at the time an enclave address without a backing enclave page is accessed, potentially directly reclaiming pages if no free pages are available.
The enclave is still required to run ENCLU[EACCEPT] on the page before it can be used. A useful flow is for the enclave to run ENCLU[EACCEPT] on an uninitialized address. This will trigger the page fault handler that will add the enclave page and return execution to the enclave to repeat the ENCLU[EACCEPT] instruction, this time successful.
If the enclave accesses an uninitialized address in another way, for example by expanding the enclave stack to a page that has not yet been added, then the page fault handler would add the page on the first write but upon returning to the enclave the instruction that triggered the page fault would be repeated and since ENCLU[EACCEPT] was not run yet it would trigger a second page fault, this time with the SGX flag set in the page fault error code. This can only be recovered by entering the enclave again and directly running the ENCLU[EACCEPT] instruction on the now initialized address.
Accessing an uninitialized address from outside the enclave also triggers this flow but the page will remain inaccessible (access will result in #PF) until accepted from within the enclave via ENCLU[EACCEPT].
The page is added with the architecturally constrained RW permissions as runtime as well as maximum allowed permissions. It is understood that there are some use cases, for example code relocation, that requires RWX maximum permissions. Supporting these use cases require guidance from user space policy before such maximum permissions can be allowed. Integration with user policy is deferred.
---
Changes since V1:
- Fix subject line "to initialized" -> "to an initialized" (Jarkko).
- Move text about hardware's PENDING state to the patch that introduces the ENCLS[EAUG] wrapper (Jarkko).
- Ensure kernel-doc uses brackets when referring to function.
arch/x86/kernel/cpu/sgx/encl.c | 133 ++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/encl.h | 2 + arch/x86/kernel/cpu/sgx/ioctl.c | 4 +- 3 files changed, 137 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/sgx/encl.c +133 -0
@@ -124,6 +124,128 @@ struct sgx_encl_page *sgx_encl_load_page(struct sgx_encl *encl,return entry; } +/** +* sgx_encl_eaug_page() - Dynamically add page to initialized enclave +* @vma: VMA obtained from fault info from where page is accessed +* @encl: enclave accessing the page +* @addr: address that triggered the page fault +* +* When an initialized enclave accesses a page with no backing EPC page +* on a SGX2 system then the EPC can be added dynamically via the SGX2 +* ENCLS[EAUG] instruction. +* +* Returns: Appropriate vm_fault_t: VM_FAULT_NOPAGE when PTE was installed +* successfully, VM_FAULT_SIGBUS or VM_FAULT_OOM as error otherwise. +*/ +static vm_fault_t sgx_encl_eaug_page (struct vm_area_struct *vma, +struct sgx_encl *encl,unsigned long addr) +{ +struct sgx_pageinfo pginfo = {0 }; +struct sgx_encl_page *encl_page; +struct sgx_epc_page *epc_page; +struct sgx_va_page *va_page; +unsigned long phys_addr; +unsigned long prot; +vm_fault_t vmret; +int ret; + +if (!test_bit (SGX_ENCL_INITIALIZED , &encl->flags)) +return VM_FAULT_SIGBUS ; + + encl_page =kzalloc (sizeof (*encl_page),GFP_KERNEL ); +if (!encl_page) +return VM_FAULT_OOM ; + + encl_page->desc = addr; + encl_page->encl = encl; + +/* +* Adding a regular page that is architecturally allowed to only +* be created with RW permissions. +* TBD: Interface with user space policy to support max permissions +* of RWX. +*/ + prot =PROT_READ |PROT_WRITE ; + encl_page->vm_run_prot_bits =calc_vm_prot_bits (prot,0 ); + encl_page->vm_max_prot_bits = encl_page->vm_run_prot_bits; + + epc_page =sgx_alloc_epc_page (encl_page, true); +if (IS_ERR (epc_page)) { +kfree (encl_page); +return VM_FAULT_SIGBUS ; + } + + va_page =sgx_encl_grow (encl); +if (IS_ERR (va_page)) { + ret =PTR_ERR (va_page); + goto err_out_free; + } + +mutex_lock (&encl->lock); + +/* +* Copy comment from sgx_encl_add_page() to maintain guidance in +* this similar flow: +* Adding to encl->va_pages must be done under encl->lock. Ditto for +* deleting (via sgx_encl_shrink()) in the error path. +*/ +if (va_page) +list_add (&va_page->list, &encl->va_pages); + + ret =xa_insert (&encl->page_array,PFN_DOWN (encl_page->desc), + encl_page,GFP_KERNEL ); +/* +* If ret == -EBUSY then page was created in another flow while +* running without encl->lock +*/ +if (ret) + goto err_out_unlock; + + pginfo.secs = (unsigned long )sgx_get_epc_virt_addr (encl->secs.epc_page); + pginfo.addr = encl_page->desc &PAGE_MASK ; + pginfo.metadata =0 ; + + ret =__eaug (&pginfo,sgx_get_epc_virt_addr (epc_page)); +if (ret) + goto err_out; + + encl_page->encl = encl; + encl_page->epc_page = epc_page; + encl_page->type =SGX_PAGE_TYPE_REG ; + encl->secs_child_cnt++; + +sgx_mark_page_reclaimable (encl_page->epc_page); + + phys_addr =sgx_get_epc_phys_addr (epc_page); +/* +* Do not undo everything when creating PTE entry fails - next #PF +* would find page ready for a PTE. +* PAGE_SHARED because protection is forced to be RW above and COW +* is not supported. +*/ + vmret =vmf_insert_pfn_prot (vma, addr,PFN_DOWN (phys_addr), +PAGE_SHARED ); +if (vmret !=VM_FAULT_NOPAGE ) { +mutex_unlock (&encl->lock); +return VM_FAULT_SIGBUS ; + } +mutex_unlock (&encl->lock); +return VM_FAULT_NOPAGE ; + +err_out: +xa_erase (&encl->page_array,PFN_DOWN (encl_page->desc)); + +err_out_unlock: +sgx_encl_shrink (encl, va_page); +mutex_unlock (&encl->lock); + +err_out_free: +sgx_encl_free_epc_page (epc_page); +kfree (encl_page); + +return VM_FAULT_SIGBUS ; +} +static vm_fault_t sgx_vma_fault (struct vm_fault *vmf) {unsigned long addr = (unsigned long )vmf->address;
@@ -145,6 +267,17 @@ static vm_fault_t sgx_vma_fault(struct vm_fault *vmf)if (unlikely (!encl))return VM_FAULT_SIGBUS ; +/* +* The page_array keeps track of all enclave pages, whether they +* are swapped out or not. If there is no entry for this page and +* the system supports SGX2 then it is possible to dynamically add +* a new enclave page. This is only possible for an initialized +* enclave that will be checked for right away. +*/ +if (cpu_feature_enabled (X86_FEATURE_SGX2 ) && + (!xa_load (&encl->page_array,PFN_DOWN (addr)))) +return sgx_encl_eaug_page (vma, encl, addr); +mutex_lock (&encl->lock); entry =sgx_encl_load_page (encl, addr);
arch/x86/kernel/cpu/sgx/encl.h +2 -0
@@ -123,4 +123,6 @@ void sgx_encl_free_epc_page(struct sgx_epc_page *page);struct sgx_encl_page *sgx_encl_load_page (struct sgx_encl *encl,unsigned long addr); +struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl); +void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page);#endif /* _X86_ENCL_H */
arch/x86/kernel/cpu/sgx/ioctl.c +2 -2
@@ -17,7 +17,7 @@#include "encl.h" #include "encls.h" -static struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl) +struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl) {struct sgx_va_page *va_page =NULL ;void *err;
@@ -43,7 +43,7 @@ static struct sgx_va_page *sgx_encl_grow(struct sgx_encl *encl)return va_page; } -static void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page) +void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page) { encl->page_cnt--;
Signature · 1 line
2.25.1
Jarkko Sakkinen
maintainerRe: [PATCH V2 19/32] x86/sgx: Support adding of pages to an initialized enclave
Quoted 242 lines · Reinette Chatre
With SGX1 an enclave needs to be created with its maximum memory demands allocated. Pages cannot be added to an enclave after it is initialized. SGX2 introduces a new function, ENCLS[EAUG], that can be used to add pages to an initialized enclave. With SGX2 the enclave still needs to set aside address space for its maximum memory demands during enclave creation, but all pages need not be added before enclave initialization. Pages can be added during enclave runtime.
Add support for dynamically adding pages to an initialized enclave, architecturally limited to RW permission. Add pages via the page fault handler at the time an enclave address without a backing enclave page is accessed, potentially directly reclaiming pages if no free pages are available.
The enclave is still required to run ENCLU[EACCEPT] on the page before it can be used. A useful flow is for the enclave to run ENCLU[EACCEPT] on an uninitialized address. This will trigger the page fault handler that will add the enclave page and return execution to the enclave to repeat the ENCLU[EACCEPT] instruction, this time successful.
If the enclave accesses an uninitialized address in another way, for example by expanding the enclave stack to a page that has not yet been added, then the page fault handler would add the page on the first write but upon returning to the enclave the instruction that triggered the page fault would be repeated and since ENCLU[EACCEPT] was not run yet it would trigger a second page fault, this time with the SGX flag set in the page fault error code. This can only be recovered by entering the enclave again and directly running the ENCLU[EACCEPT] instruction on the now initialized address.
Accessing an uninitialized address from outside the enclave also triggers this flow but the page will remain inaccessible (access will result in #PF) until accepted from within the enclave via ENCLU[EACCEPT].
The page is added with the architecturally constrained RW permissions as runtime as well as maximum allowed permissions. It is understood that there are some use cases, for example code relocation, that requires RWX maximum permissions. Supporting these use cases require guidance from user space policy before such maximum permissions can be allowed. Integration with user policy is deferred.
Signed-off-by Reinette Chatre <[email protected]>---
Changes since V1:
- Fix subject line "to initialized" -> "to an initialized" (Jarkko).
- Move text about hardware's PENDING state to the patch that introduces the ENCLS[EAUG] wrapper (Jarkko).
- Ensure kernel-doc uses brackets when referring to function.arch/x86/kernel/cpu/sgx/encl.c | 133 ++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/encl.h | 2 + arch/x86/kernel/cpu/sgx/ioctl.c | 4 +- 3 files changed, 137 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/sgx/encl.c +133
-0@@ -124,6 +124,128 @@ struct sgx_encl_page *sgx_encl_load_page(struct sgx_encl *encl,return entry; } +/** +* sgx_encl_eaug_page() - Dynamically add page to initialized enclave +* @vma: VMA obtained from fault info from where page is accessed +* @encl: enclave accessing the page +* @addr: address that triggered the page fault +* +* When an initialized enclave accesses a page with no backing EPC page +* on a SGX2 system then the EPC can be added dynamically via the SGX2 +* ENCLS[EAUG] instruction. +* +* Returns: Appropriate vm_fault_t: VM_FAULT_NOPAGE when PTE was installed +* successfully, VM_FAULT_SIGBUS or VM_FAULT_OOM as error otherwise. +*/ +static vm_fault_t sgx_encl_eaug_page (struct vm_area_struct *vma, +struct sgx_encl *encl,unsigned long addr) +{ +struct sgx_pageinfo pginfo = {0 }; +struct sgx_encl_page *encl_page; +struct sgx_epc_page *epc_page; +struct sgx_va_page *va_page; +unsigned long phys_addr; +unsigned long prot; +vm_fault_t vmret; +int ret; + +if (!test_bit (SGX_ENCL_INITIALIZED , &encl->flags)) +return VM_FAULT_SIGBUS ; + + encl_page =kzalloc (sizeof (*encl_page),GFP_KERNEL ); +if (!encl_page) +return VM_FAULT_OOM ; + + encl_page->desc = addr; + encl_page->encl = encl; + +/* +* Adding a regular page that is architecturally allowed to only +* be created with RW permissions. +* TBD: Interface with user space policy to support max permissions +* of RWX. +*/ + prot =PROT_READ |PROT_WRITE ; + encl_page->vm_run_prot_bits =calc_vm_prot_bits (prot,0 ); + encl_page->vm_max_prot_bits = encl_page->vm_run_prot_bits; + + epc_page =sgx_alloc_epc_page (encl_page, true); +if (IS_ERR (epc_page)) { +kfree (encl_page); +return VM_FAULT_SIGBUS ; + } + + va_page =sgx_encl_grow (encl); +if (IS_ERR (va_page)) { + ret =PTR_ERR (va_page); + goto err_out_free; + } + +mutex_lock (&encl->lock); + +/* +* Copy comment from sgx_encl_add_page() to maintain guidance in +* this similar flow: +* Adding to encl->va_pages must be done under encl->lock. Ditto for +* deleting (via sgx_encl_shrink()) in the error path. +*/ +if (va_page) +list_add (&va_page->list, &encl->va_pages); + + ret =xa_insert (&encl->page_array,PFN_DOWN (encl_page->desc), + encl_page,GFP_KERNEL ); +/* +* If ret == -EBUSY then page was created in another flow while +* running without encl->lock +*/ +if (ret) + goto err_out_unlock; + + pginfo.secs = (unsigned long )sgx_get_epc_virt_addr (encl->secs.epc_page); + pginfo.addr = encl_page->desc &PAGE_MASK ; + pginfo.metadata =0 ; + + ret =__eaug (&pginfo,sgx_get_epc_virt_addr (epc_page)); +if (ret) + goto err_out; + + encl_page->encl = encl; + encl_page->epc_page = epc_page; + encl_page->type =SGX_PAGE_TYPE_REG ; + encl->secs_child_cnt++; + +sgx_mark_page_reclaimable (encl_page->epc_page); + + phys_addr =sgx_get_epc_phys_addr (epc_page); +/* +* Do not undo everything when creating PTE entry fails - next #PF +* would find page ready for a PTE. +* PAGE_SHARED because protection is forced to be RW above and COW +* is not supported. +*/ + vmret =vmf_insert_pfn_prot (vma, addr,PFN_DOWN (phys_addr), +PAGE_SHARED ); +if (vmret !=VM_FAULT_NOPAGE ) { +mutex_unlock (&encl->lock); +return VM_FAULT_SIGBUS ; + } +mutex_unlock (&encl->lock); +return VM_FAULT_NOPAGE ; + +err_out: +xa_erase (&encl->page_array,PFN_DOWN (encl_page->desc)); + +err_out_unlock: +sgx_encl_shrink (encl, va_page); +mutex_unlock (&encl->lock); + +err_out_free: +sgx_encl_free_epc_page (epc_page); +kfree (encl_page); + +return VM_FAULT_SIGBUS ; +} +static vm_fault_t sgx_vma_fault (struct vm_fault *vmf) {unsigned long addr = (unsigned long )vmf->address;@@ -145,6 +267,17 @@ static vm_fault_t sgx_vma_fault(struct vm_fault *vmf)if (unlikely (!encl))return VM_FAULT_SIGBUS ; +/* +* The page_array keeps track of all enclave pages, whether they +* are swapped out or not. If there is no entry for this page and +* the system supports SGX2 then it is possible to dynamically add +* a new enclave page. This is only possible for an initialized +* enclave that will be checked for right away. +*/ +if (cpu_feature_enabled (X86_FEATURE_SGX2 ) && + (!xa_load (&encl->page_array,PFN_DOWN (addr)))) +return sgx_encl_eaug_page (vma, encl, addr); +mutex_lock (&encl->lock); entry =sgx_encl_load_page (encl, addr);arch/x86/kernel/cpu/sgx/encl.h +2
-0@@ -123,4 +123,6 @@ void sgx_encl_free_epc_page(struct sgx_epc_page *page);struct sgx_encl_page *sgx_encl_load_page (struct sgx_encl *encl,unsigned long addr); +struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl); +void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page);#endif /* _X86_ENCL_H */ arch/x86/kernel/cpu/sgx/ioctl.c +2
-2@@ -17,7 +17,7 @@#include "encl.h" #include "encls.h" -static struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl) +struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl) {struct sgx_va_page *va_page =NULL ;void *err;@@ -43,7 +43,7 @@ static struct sgx_va_page *sgx_encl_grow(struct sgx_encl *encl)return va_page; } -static void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page) +void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page) { encl->page_cnt--;Signature · 1 line
2.25.1
Quickly looking through also this sequence is possible:
1. Enclave's run-time flow ignores the whole EACCEPT but instead a memory dereference will initialize the sequence.
2. This causes #PF handler to do EAUG and after the enclave is re-entered the vDSO exists because the page is not EACCEPT'd.
2. Enclave host enter in-enclave exception handler, which does EACCEPT.
Can you confirm this? I'm planning to test this patch by implementing EAUG support in Rust for Enarx. At this point I'm not yet sure whether I choose EACCEPT initiated or memory deference initiated code path but I think it is good if the kernel implementation is good enough to support both.
Other than that, this looks super solid!
BR, Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH V2 19/32] x86/sgx: Support adding of pages to an initialized enclave
Quoted 252 lines · Jarkko Sakkinen
Quoted 242 lines · Reinette Chatre
With SGX1 an enclave needs to be created with its maximum memory demands allocated. Pages cannot be added to an enclave after it is initialized. SGX2 introduces a new function, ENCLS[EAUG], that can be used to add pages to an initialized enclave. With SGX2 the enclave still needs to set aside address space for its maximum memory demands during enclave creation, but all pages need not be added before enclave initialization. Pages can be added during enclave runtime.
Add support for dynamically adding pages to an initialized enclave, architecturally limited to RW permission. Add pages via the page fault handler at the time an enclave address without a backing enclave page is accessed, potentially directly reclaiming pages if no free pages are available.
The enclave is still required to run ENCLU[EACCEPT] on the page before it can be used. A useful flow is for the enclave to run ENCLU[EACCEPT] on an uninitialized address. This will trigger the page fault handler that will add the enclave page and return execution to the enclave to repeat the ENCLU[EACCEPT] instruction, this time successful.
If the enclave accesses an uninitialized address in another way, for example by expanding the enclave stack to a page that has not yet been added, then the page fault handler would add the page on the first write but upon returning to the enclave the instruction that triggered the page fault would be repeated and since ENCLU[EACCEPT] was not run yet it would trigger a second page fault, this time with the SGX flag set in the page fault error code. This can only be recovered by entering the enclave again and directly running the ENCLU[EACCEPT] instruction on the now initialized address.
Accessing an uninitialized address from outside the enclave also triggers this flow but the page will remain inaccessible (access will result in #PF) until accepted from within the enclave via ENCLU[EACCEPT].
The page is added with the architecturally constrained RW permissions as runtime as well as maximum allowed permissions. It is understood that there are some use cases, for example code relocation, that requires RWX maximum permissions. Supporting these use cases require guidance from user space policy before such maximum permissions can be allowed. Integration with user policy is deferred.
Signed-off-by Reinette Chatre <[email protected]>---
Changes since V1:
- Fix subject line "to initialized" -> "to an initialized" (Jarkko).
- Move text about hardware's PENDING state to the patch that introduces the ENCLS[EAUG] wrapper (Jarkko).
- Ensure kernel-doc uses brackets when referring to function.arch/x86/kernel/cpu/sgx/encl.c | 133 ++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/encl.h | 2 + arch/x86/kernel/cpu/sgx/ioctl.c | 4 +- 3 files changed, 137 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/sgx/encl.c +133
-0@@ -124,6 +124,128 @@ struct sgx_encl_page *sgx_encl_load_page(struct sgx_encl *encl,return entry; } +/** +* sgx_encl_eaug_page() - Dynamically add page to initialized enclave +* @vma: VMA obtained from fault info from where page is accessed +* @encl: enclave accessing the page +* @addr: address that triggered the page fault +* +* When an initialized enclave accesses a page with no backing EPC page +* on a SGX2 system then the EPC can be added dynamically via the SGX2 +* ENCLS[EAUG] instruction. +* +* Returns: Appropriate vm_fault_t: VM_FAULT_NOPAGE when PTE was installed +* successfully, VM_FAULT_SIGBUS or VM_FAULT_OOM as error otherwise. +*/ +static vm_fault_t sgx_encl_eaug_page (struct vm_area_struct *vma, +struct sgx_encl *encl,unsigned long addr) +{ +struct sgx_pageinfo pginfo = {0 }; +struct sgx_encl_page *encl_page; +struct sgx_epc_page *epc_page; +struct sgx_va_page *va_page; +unsigned long phys_addr; +unsigned long prot; +vm_fault_t vmret; +int ret; + +if (!test_bit (SGX_ENCL_INITIALIZED , &encl->flags)) +return VM_FAULT_SIGBUS ; + + encl_page =kzalloc (sizeof (*encl_page),GFP_KERNEL ); +if (!encl_page) +return VM_FAULT_OOM ; + + encl_page->desc = addr; + encl_page->encl = encl; + +/* +* Adding a regular page that is architecturally allowed to only +* be created with RW permissions. +* TBD: Interface with user space policy to support max permissions +* of RWX. +*/ + prot =PROT_READ |PROT_WRITE ; + encl_page->vm_run_prot_bits =calc_vm_prot_bits (prot,0 ); + encl_page->vm_max_prot_bits = encl_page->vm_run_prot_bits; + + epc_page =sgx_alloc_epc_page (encl_page, true); +if (IS_ERR (epc_page)) { +kfree (encl_page); +return VM_FAULT_SIGBUS ; + } + + va_page =sgx_encl_grow (encl); +if (IS_ERR (va_page)) { + ret =PTR_ERR (va_page); + goto err_out_free; + } + +mutex_lock (&encl->lock); + +/* +* Copy comment from sgx_encl_add_page() to maintain guidance in +* this similar flow: +* Adding to encl->va_pages must be done under encl->lock. Ditto for +* deleting (via sgx_encl_shrink()) in the error path. +*/ +if (va_page) +list_add (&va_page->list, &encl->va_pages); + + ret =xa_insert (&encl->page_array,PFN_DOWN (encl_page->desc), + encl_page,GFP_KERNEL ); +/* +* If ret == -EBUSY then page was created in another flow while +* running without encl->lock +*/ +if (ret) + goto err_out_unlock; + + pginfo.secs = (unsigned long )sgx_get_epc_virt_addr (encl->secs.epc_page); + pginfo.addr = encl_page->desc &PAGE_MASK ; + pginfo.metadata =0 ; + + ret =__eaug (&pginfo,sgx_get_epc_virt_addr (epc_page)); +if (ret) + goto err_out; + + encl_page->encl = encl; + encl_page->epc_page = epc_page; + encl_page->type =SGX_PAGE_TYPE_REG ; + encl->secs_child_cnt++; + +sgx_mark_page_reclaimable (encl_page->epc_page); + + phys_addr =sgx_get_epc_phys_addr (epc_page); +/* +* Do not undo everything when creating PTE entry fails - next #PF +* would find page ready for a PTE. +* PAGE_SHARED because protection is forced to be RW above and COW +* is not supported. +*/ + vmret =vmf_insert_pfn_prot (vma, addr,PFN_DOWN (phys_addr), +PAGE_SHARED ); +if (vmret !=VM_FAULT_NOPAGE ) { +mutex_unlock (&encl->lock); +return VM_FAULT_SIGBUS ; + } +mutex_unlock (&encl->lock); +return VM_FAULT_NOPAGE ; + +err_out: +xa_erase (&encl->page_array,PFN_DOWN (encl_page->desc)); + +err_out_unlock: +sgx_encl_shrink (encl, va_page); +mutex_unlock (&encl->lock); + +err_out_free: +sgx_encl_free_epc_page (epc_page); +kfree (encl_page); + +return VM_FAULT_SIGBUS ; +} +static vm_fault_t sgx_vma_fault (struct vm_fault *vmf) {unsigned long addr = (unsigned long )vmf->address;@@ -145,6 +267,17 @@ static vm_fault_t sgx_vma_fault(struct vm_fault *vmf)if (unlikely (!encl))return VM_FAULT_SIGBUS ; +/* +* The page_array keeps track of all enclave pages, whether they +* are swapped out or not. If there is no entry for this page and +* the system supports SGX2 then it is possible to dynamically add +* a new enclave page. This is only possible for an initialized +* enclave that will be checked for right away. +*/ +if (cpu_feature_enabled (X86_FEATURE_SGX2 ) && + (!xa_load (&encl->page_array,PFN_DOWN (addr)))) +return sgx_encl_eaug_page (vma, encl, addr); +mutex_lock (&encl->lock); entry =sgx_encl_load_page (encl, addr);arch/x86/kernel/cpu/sgx/encl.h +2
-0@@ -123,4 +123,6 @@ void sgx_encl_free_epc_page(struct sgx_epc_page *page);struct sgx_encl_page *sgx_encl_load_page (struct sgx_encl *encl,unsigned long addr); +struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl); +void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page);#endif /* _X86_ENCL_H */ arch/x86/kernel/cpu/sgx/ioctl.c +2
-2@@ -17,7 +17,7 @@#include "encl.h" #include "encls.h" -static struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl) +struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl) {struct sgx_va_page *va_page =NULL ;void *err;@@ -43,7 +43,7 @@ static struct sgx_va_page *sgx_encl_grow(struct sgx_encl *encl)return va_page; } -static void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page) +void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page) { encl->page_cnt--;Signature · 1 line
2.25.1Quickly looking through also this sequence is possible:
1. Enclave's run-time flow ignores the whole EACCEPT but instead a memory dereference will initialize the sequence.
2. This causes #PF handler to do EAUG and after the enclave is re-entered the vDSO exists because the page is not EACCEPT'd.
2. Enclave host enter in-enclave exception handler, which does EACCEPT.
Quoted 6 lines
Can you confirm this? I'm planning to test this patch by implementing EAUG support in Rust for Enarx. At this point I'm not yet sure whether I choose EACCEPT initiated or memory deference initiated code path but I think it is good if the kernel implementation is good enough to support both.
Other than that, this looks super solid!
I got my answer:
I could almost give reviewed-by but I need to write the user space implementation first to check that this works for Enarx.
BR, Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH V2 19/32] x86/sgx: Support adding of pages to an initialized enclave
Quoted 258 lines · Jarkko Sakkinen
Quoted 257 lines · Jarkko Sakkinen
Quoted 242 lines · Reinette Chatre
With SGX1 an enclave needs to be created with its maximum memory demands allocated. Pages cannot be added to an enclave after it is initialized. SGX2 introduces a new function, ENCLS[EAUG], that can be used to add pages to an initialized enclave. With SGX2 the enclave still needs to set aside address space for its maximum memory demands during enclave creation, but all pages need not be added before enclave initialization. Pages can be added during enclave runtime.
Add support for dynamically adding pages to an initialized enclave, architecturally limited to RW permission. Add pages via the page fault handler at the time an enclave address without a backing enclave page is accessed, potentially directly reclaiming pages if no free pages are available.
The enclave is still required to run ENCLU[EACCEPT] on the page before it can be used. A useful flow is for the enclave to run ENCLU[EACCEPT] on an uninitialized address. This will trigger the page fault handler that will add the enclave page and return execution to the enclave to repeat the ENCLU[EACCEPT] instruction, this time successful.
If the enclave accesses an uninitialized address in another way, for example by expanding the enclave stack to a page that has not yet been added, then the page fault handler would add the page on the first write but upon returning to the enclave the instruction that triggered the page fault would be repeated and since ENCLU[EACCEPT] was not run yet it would trigger a second page fault, this time with the SGX flag set in the page fault error code. This can only be recovered by entering the enclave again and directly running the ENCLU[EACCEPT] instruction on the now initialized address.
Accessing an uninitialized address from outside the enclave also triggers this flow but the page will remain inaccessible (access will result in #PF) until accepted from within the enclave via ENCLU[EACCEPT].
The page is added with the architecturally constrained RW permissions as runtime as well as maximum allowed permissions. It is understood that there are some use cases, for example code relocation, that requires RWX maximum permissions. Supporting these use cases require guidance from user space policy before such maximum permissions can be allowed. Integration with user policy is deferred.
Signed-off-by Reinette Chatre <[email protected]>---
Changes since V1:
- Fix subject line "to initialized" -> "to an initialized" (Jarkko).
- Move text about hardware's PENDING state to the patch that introduces the ENCLS[EAUG] wrapper (Jarkko).
- Ensure kernel-doc uses brackets when referring to function.arch/x86/kernel/cpu/sgx/encl.c | 133 ++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/encl.h | 2 + arch/x86/kernel/cpu/sgx/ioctl.c | 4 +- 3 files changed, 137 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/sgx/encl.c +133
-0@@ -124,6 +124,128 @@ struct sgx_encl_page *sgx_encl_load_page(struct sgx_encl *encl,return entry; } +/** +* sgx_encl_eaug_page() - Dynamically add page to initialized enclave +* @vma: VMA obtained from fault info from where page is accessed +* @encl: enclave accessing the page +* @addr: address that triggered the page fault +* +* When an initialized enclave accesses a page with no backing EPC page +* on a SGX2 system then the EPC can be added dynamically via the SGX2 +* ENCLS[EAUG] instruction. +* +* Returns: Appropriate vm_fault_t: VM_FAULT_NOPAGE when PTE was installed +* successfully, VM_FAULT_SIGBUS or VM_FAULT_OOM as error otherwise. +*/ +static vm_fault_t sgx_encl_eaug_page (struct vm_area_struct *vma, +struct sgx_encl *encl,unsigned long addr) +{ +struct sgx_pageinfo pginfo = {0 }; +struct sgx_encl_page *encl_page; +struct sgx_epc_page *epc_page; +struct sgx_va_page *va_page; +unsigned long phys_addr; +unsigned long prot; +vm_fault_t vmret; +int ret; + +if (!test_bit (SGX_ENCL_INITIALIZED , &encl->flags)) +return VM_FAULT_SIGBUS ; + + encl_page =kzalloc (sizeof (*encl_page),GFP_KERNEL ); +if (!encl_page) +return VM_FAULT_OOM ; + + encl_page->desc = addr; + encl_page->encl = encl; + +/* +* Adding a regular page that is architecturally allowed to only +* be created with RW permissions. +* TBD: Interface with user space policy to support max permissions +* of RWX. +*/ + prot =PROT_READ |PROT_WRITE ; + encl_page->vm_run_prot_bits =calc_vm_prot_bits (prot,0 ); + encl_page->vm_max_prot_bits = encl_page->vm_run_prot_bits; + + epc_page =sgx_alloc_epc_page (encl_page, true); +if (IS_ERR (epc_page)) { +kfree (encl_page); +return VM_FAULT_SIGBUS ; + } + + va_page =sgx_encl_grow (encl); +if (IS_ERR (va_page)) { + ret =PTR_ERR (va_page); + goto err_out_free; + } + +mutex_lock (&encl->lock); + +/* +* Copy comment from sgx_encl_add_page() to maintain guidance in +* this similar flow: +* Adding to encl->va_pages must be done under encl->lock. Ditto for +* deleting (via sgx_encl_shrink()) in the error path. +*/ +if (va_page) +list_add (&va_page->list, &encl->va_pages); + + ret =xa_insert (&encl->page_array,PFN_DOWN (encl_page->desc), + encl_page,GFP_KERNEL ); +/* +* If ret == -EBUSY then page was created in another flow while +* running without encl->lock +*/ +if (ret) + goto err_out_unlock; + + pginfo.secs = (unsigned long )sgx_get_epc_virt_addr (encl->secs.epc_page); + pginfo.addr = encl_page->desc &PAGE_MASK ; + pginfo.metadata =0 ; + + ret =__eaug (&pginfo,sgx_get_epc_virt_addr (epc_page)); +if (ret) + goto err_out; + + encl_page->encl = encl; + encl_page->epc_page = epc_page; + encl_page->type =SGX_PAGE_TYPE_REG ; + encl->secs_child_cnt++; + +sgx_mark_page_reclaimable (encl_page->epc_page); + + phys_addr =sgx_get_epc_phys_addr (epc_page); +/* +* Do not undo everything when creating PTE entry fails - next #PF +* would find page ready for a PTE. +* PAGE_SHARED because protection is forced to be RW above and COW +* is not supported. +*/ + vmret =vmf_insert_pfn_prot (vma, addr,PFN_DOWN (phys_addr), +PAGE_SHARED ); +if (vmret !=VM_FAULT_NOPAGE ) { +mutex_unlock (&encl->lock); +return VM_FAULT_SIGBUS ; + } +mutex_unlock (&encl->lock); +return VM_FAULT_NOPAGE ; + +err_out: +xa_erase (&encl->page_array,PFN_DOWN (encl_page->desc)); + +err_out_unlock: +sgx_encl_shrink (encl, va_page); +mutex_unlock (&encl->lock); + +err_out_free: +sgx_encl_free_epc_page (epc_page); +kfree (encl_page); + +return VM_FAULT_SIGBUS ; +} +static vm_fault_t sgx_vma_fault (struct vm_fault *vmf) {unsigned long addr = (unsigned long )vmf->address;@@ -145,6 +267,17 @@ static vm_fault_t sgx_vma_fault(struct vm_fault *vmf)if (unlikely (!encl))return VM_FAULT_SIGBUS ; +/* +* The page_array keeps track of all enclave pages, whether they +* are swapped out or not. If there is no entry for this page and +* the system supports SGX2 then it is possible to dynamically add +* a new enclave page. This is only possible for an initialized +* enclave that will be checked for right away. +*/ +if (cpu_feature_enabled (X86_FEATURE_SGX2 ) && + (!xa_load (&encl->page_array,PFN_DOWN (addr)))) +return sgx_encl_eaug_page (vma, encl, addr); +mutex_lock (&encl->lock); entry =sgx_encl_load_page (encl, addr);arch/x86/kernel/cpu/sgx/encl.h +2
-0@@ -123,4 +123,6 @@ void sgx_encl_free_epc_page(struct sgx_epc_page *page);struct sgx_encl_page *sgx_encl_load_page (struct sgx_encl *encl,unsigned long addr); +struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl); +void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page);#endif /* _X86_ENCL_H */ arch/x86/kernel/cpu/sgx/ioctl.c +2
-2@@ -17,7 +17,7 @@#include "encl.h" #include "encls.h" -static struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl) +struct sgx_va_page *sgx_encl_grow (struct sgx_encl *encl) {struct sgx_va_page *va_page =NULL ;void *err;@@ -43,7 +43,7 @@ static struct sgx_va_page *sgx_encl_grow(struct sgx_encl *encl)return va_page; } -static void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page) +void sgx_encl_shrink (struct sgx_encl *encl,struct sgx_va_page *va_page) { encl->page_cnt--;Signature · 1 line
2.25.1Quickly looking through also this sequence is possible:
1. Enclave's run-time flow ignores the whole EACCEPT but instead a memory dereference will initialize the sequence.
2. This causes #PF handler to do EAUG and after the enclave is re-entered the vDSO exists because the page is not EACCEPT'd.
2. Enclave host enter in-enclave exception handler, which does EACCEPT.Can you confirm this? I'm planning to test this patch by implementing EAUG support in Rust for Enarx. At this point I'm not yet sure whether I choose EACCEPT initiated or memory deference initiated code path but I think it is good if the kernel implementation is good enough to support both.
Quoted 8 lines
Quoted 1 line
Other than that, this looks super solid!
I got my answer:
I could almost give reviewed-by but I need to write the user space implementation first to check that this works for Enarx.
Do you know if it is possible to do EAUG, EMODPR and the do a single EACCEPT for both? Just looking at pseudo-code, it looked doable but I need to check this.
I.e. EAUG has this
EPCM(DS:RCX).BLOCKED := 0;
EPCM(DS:RCX).PENDING := 1;
EPCM(DS:RCX).MODIFIED := 0;
EPCM(DS:RCX).PR := 0;
(* associate the EPCPAGE with the SECS by storing the SECS identifier of DS:TMP_SECS *) Update EPCM(DS:RCX) SECS identifier to reference DS:TMP_SECS identifier; (* Set EPCM valid fields *)
EPCM(DS:RCX).VALID := 1;
And EMODPR only checks .VALID.
Doing two EACCEPT rounds is a bit rough as you have the page available in a kind of "stalled' state.
/Jarkko
Re: [PATCH V2 19/32] x86/sgx: Support adding of pages to an initialized enclave
Hi Jarkko,
On 2/20/2022 10:40 AM, Jarkko Sakkinen wrote:
...
Quoted 15 lines
Do you know if it is possible to do EAUG, EMODPR and the do a single EACCEPT for both? Just looking at pseudo-code, it looked doable but I need to check this.
I.e. EAUG has this
EPCM(DS:RCX).BLOCKED := 0;
EPCM(DS:RCX).PENDING := 1;
EPCM(DS:RCX).MODIFIED := 0;
EPCM(DS:RCX).PR := 0;
(* associate the EPCPAGE with the SECS by storing the SECS identifier of DS:TMP_SECS *) Update EPCM(DS:RCX) SECS identifier to reference DS:TMP_SECS identifier; (* Set EPCM valid fields *)
EPCM(DS:RCX).VALID := 1;
Quoted 1 line
And EMODPR only checks .VALID.
After that check there is also:
IF (EPCM(DS:RCX).PENDING is not 0 or (EPCM(DS:RCX).MODIFIED is not 0) )
THEN
RFLAGS.ZF := 1;
RAX := SGX_PAGE_NOT_MODIFIABLE;
GOTO DONE;FI;
Attempting the SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS ioctl() on a recently added page (EAUG) that has not yet been EACCEPTed is thus expected to fail with errno of EFAULT (indicating ENCLS[EMODPR] failure) and the returned structure's result field set to 20 (SGX_PAGE_NOT_MODIFIABLE).
I confirmed this behavior by modifying the "augment" kselftest test by adding a SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS call between the new memory access and the EACCEPT.
Reinette
Jarkko Sakkinen
maintainerRe: [PATCH V2 19/32] x86/sgx: Support adding of pages to an initialized enclave
Quoted 30 lines · Reinette Chatre
Hi Jarkko,
On 2/20/2022 10:40 AM, Jarkko Sakkinen wrote:
...Quoted 15 lines
Do you know if it is possible to do EAUG, EMODPR and the do a single EACCEPT for both? Just looking at pseudo-code, it looked doable but I need to check this.
I.e. EAUG has this
EPCM(DS:RCX).BLOCKED := 0;
EPCM(DS:RCX).PENDING := 1;
EPCM(DS:RCX).MODIFIED := 0;
EPCM(DS:RCX).PR := 0;
(* associate the EPCPAGE with the SECS by storing the SECS identifier of DS:TMP_SECS *) Update EPCM(DS:RCX) SECS identifier to reference DS:TMP_SECS identifier; (* Set EPCM valid fields *)
EPCM(DS:RCX).VALID := 1;Quoted 1 line
And EMODPR only checks .VALID.
After that check there is also:
IF (EPCM(DS:RCX).PENDING is not 0 or (EPCM(DS:RCX).MODIFIED is not 0) )THEN RFLAGS.ZF := 1; RAX := SGX_PAGE_NOT_MODIFIABLE; GOTO DONE;FI;
Quoted 8 lines
Attempting the
SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONSioctl()on a recently added page (EAUG) that has not yet been EACCEPTed is thus expected to fail with errno of EFAULT (indicating ENCLS[EMODPR] failure) and the returned structure's result field set to 20 (SGX_PAGE_NOT_MODIFIABLE).I confirmed this behavior by modifying the "augment" kselftest test by adding a
SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONScall between the new memory access and the EACCEPT.
Thank you, also Mark confirmed this.
BR, Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH V2 19/32] x86/sgx: Support adding of pages to an initialized enclave
In reply to Reinette ChatreQuoted 6 lines · Reinette Chatre
arch/x86/kernel/cpu/sgx/encl.c +6
-0+ encl_page =kzalloc (sizeof (*encl_page),GFP_KERNEL ); +if (!encl_page) +return VM_FAULT_OOM ; + + encl_page->desc = addr; + encl_page->encl = encl;
arch/x86/kernel/cpu/sgx/encl.c struct sgx_encl_page *sgx_encl_load_page(struct sgx_encl *encl, · line 170
Quoted 10 lines
arch/x86/kernel/cpu/sgx/encl.c +10
-0+ +/* +* Adding a regular page that is architecturally allowed to only +* be created with RW permissions. +* TBD: Interface with user space policy to support max permissions +* of RWX. +*/ + prot =PROT_READ |PROT_WRITE ; + encl_page->vm_run_prot_bits =calc_vm_prot_bits (prot,0 ); + encl_page->vm_max_prot_bits = encl_page->vm_run_prot_bits;
You should use sgx_encl_page_alloc() here and not reinvent wheel.
I wrote a patch that exports it:
https://lore.kernel.org/linux-sgx/[email protected]/T/#u
BR, Jarkko
PATCH v2 20/32 x86/sgx: Tighten accessible memory range after enclave initialization
In reply to Reinette ChatreBefore an enclave is initialized the enclave's memory range is unknown. The enclave's memory range is learned at the time it is created via the SGX_IOC_ENCLAVE_CREATE ioctl() where the provided memory range is obtained from an earlier mmap() of /dev/sgx_enclave. After an enclave is initialized its memory can be mapped into user space (mmap()) from where it can be entered at its defined entry points.
With the enclave's memory range known after it is initialized there is no reason why it should be possible to map memory outside this range.
Lock down access to the initialized enclave's memory range by denying any attempt to map memory outside its memory range.
Locking down the memory range also makes adding pages to an initialized enclave more efficient. Pages are added to an initialized enclave by accessing memory that belongs to the enclave's memory range but not yet backed by an enclave page. If it is possible for user space to map memory that does not form part of the enclave then an access to this memory would eventually fail. Failures range from a prompt general protection fault if the access was an ENCLU[EACCEPT] from within the enclave, or a page fault via the vDSO if it was another access from within the enclave, or a SIGBUS (also resulting from a page fault) if the access was from outside the enclave.
Disallowing invalid memory to be mapped in the first place avoids preventable failures.
---
Changes since V1:
- Add comment (Jarkko).
arch/x86/kernel/cpu/sgx/encl.c | 5 +++++ 1 file changed, 5 insertions(+)
arch/x86/kernel/cpu/sgx/encl.c +5 -0
@@ -403,6 +403,11 @@ int sgx_encl_may_map(struct sgx_encl *encl, unsigned long start,XA_STATE (xas, &encl->page_array,PFN_DOWN (start)); +/* Disallow mapping outside enclave's address range. */ +if (test_bit (SGX_ENCL_INITIALIZED , &encl->flags) && + (start < encl->base || end > encl->base + encl->size)) +return -EACCES ; +/* * Disallow READ_IMPLIES_EXEC tasks as their VMA permissions might * conflict with the enclave page permissions.
Signature · 1 line
2.25.1
PATCH v2 21/32 selftests/sgx: Test two different SGX2 EAUG flows
In reply to Reinette ChatreEnclave pages can be added to an initialized enclave when an address belonging to the enclave but without a backing page is accessed from within the enclave.
Accessing memory without a backing enclave page from within an enclave can be in different ways:
1) Pre-emptively run ENCLU[EACCEPT]. Since the addition of a page always needs to be accepted by the enclave via ENCLU[EACCEPT] this flow is efficient since the first execution of ENCLU[EACCEPT] triggers the addition of the page and when execution returns to the same instruction the second execution would be successful as an acceptance of the page.
2) A direct read or write. The flow where a direct read or write triggers the page addition execution cannot resume from the instruction (read/write) that triggered the fault but instead the enclave needs to be entered at a different entry point to run needed ENCLU[EACCEPT] before execution can return to the original entry point and the read/write instruction that faulted.
Add tests for both flows.
---
Changes since v1:
- Replace __cpuid() definition and usage with __cpuid_count().
- Fix accuracy of comments.
tools/testing/selftests/sgx/main.c | 243 +++++++++++++++++++++++++++++ 1 file changed, 243 insertions(+)
tools/testing/selftests/sgx/main.c +243 -0
@@ -86,6 +86,15 @@ static bool vdso_get_symtab(void *addr, struct vdso_symtab *symtab)return true; } +static inline int sgx2_supported (void ) +{ +unsigned int eax, ebx, ecx, edx; + +__cpuid_count (SGX_CPUID ,0x0 , eax, ebx, ecx, edx); + +return eax &0x2 ; +} +static unsigned long elf_sym_hash (const char *name) {unsigned long h =0 , high;
@@ -882,4 +891,238 @@ TEST_F(enclave, epcm_permissions)EXPECT_EQ (self->run.exception_addr,0 ); } +/* +* Test the addition of pages to an initialized enclave via writing to +* a page belonging to the enclave's address space but was not added +* during enclave creation. +*/ +TEST_F(enclave, augment) +{ +struct encl_op_get_from_addr get_addr_op; +struct encl_op_put_to_addr put_addr_op; +struct encl_op_eaccept eaccept_op; +size_t total_size =0 ; +void *addr; +int i; + +if (!sgx2_supported ()) +SKIP (return,"SGX2 not supported" ); + +ASSERT_TRUE (setup_test_encl (ENCL_HEAP_SIZE_DEFAULT , &self->encl, _metadata)); + +memset (&self->run,0 ,sizeof (self->run)); + self->run.tcs = self->encl.encl_base; + +for (i =0 ; i < self->encl.nr_segments; i++) { +struct encl_segment *seg = &self->encl.segment_tbl[i]; + + total_size += seg->size; + } + +/* +* Actual enclave size is expected to be larger than the loaded +* test enclave since enclave size must be a power of 2 in bytes +* and test_encl does not consume it all. +*/ +EXPECT_LT (total_size +PAGE_SIZE , self->encl.encl_size); + +/* +* Create memory mapping for the page that will be added. New +* memory mapping is for one page right after all existing +* mappings. +*/ + addr =mmap ((void *)self->encl.encl_base + total_size,PAGE_SIZE , +PROT_READ |PROT_WRITE |PROT_EXEC , +MAP_SHARED |MAP_FIXED , self->encl.fd,0 ); +EXPECT_NE (addr,MAP_FAILED ); + + self->run.exception_vector =0 ; + self->run.exception_error_code =0 ; + self->run.exception_addr =0 ; + +/* +* Attempt to write to the new page from within enclave. +* Expected to fail since page is not (yet) part of the enclave. +* The first #PF will trigger the addition of the page to the +* enclave, but since the new page needs an EACCEPT from within the +* enclave before it can be used it would not be possible +* to successfully return to the failing instruction. This is the +* cause of the second #PF captured here having the SGX bit set, +* it is from hardware preventing the page from being used. +*/ + put_addr_op.value =MAGIC ; + put_addr_op.addr = (unsigned long )addr; + put_addr_op.header.type =ENCL_OP_PUT_TO_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&put_addr_op, &self->run, true),0 ); + +EXPECT_EQ (self->run.function,ERESUME ); +EXPECT_EQ (self->run.exception_vector,14 ); +EXPECT_EQ (self->run.exception_addr, (unsigned long )addr); + +if (self->run.exception_error_code ==0x6 ) { +munmap (addr,PAGE_SIZE ); +SKIP (return,"Kernel does not support adding pages to initialized enclave" ); + } + +EXPECT_EQ (self->run.exception_error_code,0x8007 ); + + self->run.exception_vector =0 ; + self->run.exception_error_code =0 ; + self->run.exception_addr =0 ; + +/* Handle AEX by running EACCEPT from new entry point. */ + self->run.tcs = self->encl.encl_base +PAGE_SIZE ; + + eaccept_op.epc_addr = self->encl.encl_base + total_size; + eaccept_op.flags =SGX_SECINFO_R |SGX_SECINFO_W |SGX_SECINFO_REG |SGX_SECINFO_PENDING ; + eaccept_op.ret =0 ; + eaccept_op.header.type =ENCL_OP_EACCEPT ; + +EXPECT_EQ (ENCL_CALL (&eaccept_op, &self->run, true),0 ); + +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); +EXPECT_EQ (eaccept_op.ret,0 ); + +/* Can now return to main TCS to resume execution. */ + self->run.tcs = self->encl.encl_base; + +EXPECT_EQ (vdso_sgx_enter_enclave ((unsigned long )&put_addr_op,0 ,0 , +ERESUME ,0 ,0 , + &self->run), +0 ); + +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +/* +* Read memory from newly added page that was just written to, +* confirming that data previously written (MAGIC) is present. +*/ + get_addr_op.value =0 ; + get_addr_op.addr = (unsigned long )addr; + get_addr_op.header.type =ENCL_OP_GET_FROM_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&get_addr_op, &self->run, true),0 ); + +EXPECT_EQ (get_addr_op.value,MAGIC ); +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +munmap (addr,PAGE_SIZE ); +} + +/* +* Test for the addition of pages to an initialized enclave via a +* pre-emptive run of EACCEPT on page to be added. +*/ +TEST_F(enclave, augment_via_eaccept) +{ +struct encl_op_get_from_addr get_addr_op; +struct encl_op_put_to_addr put_addr_op; +struct encl_op_eaccept eaccept_op; +size_t total_size =0 ; +void *addr; +int i; + +if (!sgx2_supported ()) +SKIP (return,"SGX2 not supported" ); + +ASSERT_TRUE (setup_test_encl (ENCL_HEAP_SIZE_DEFAULT , &self->encl, _metadata)); + +memset (&self->run,0 ,sizeof (self->run)); + self->run.tcs = self->encl.encl_base; + +for (i =0 ; i < self->encl.nr_segments; i++) { +struct encl_segment *seg = &self->encl.segment_tbl[i]; + + total_size += seg->size; + } + +/* +* Actual enclave size is expected to be larger than the loaded +* test enclave since enclave size must be a power of 2 in bytes while +* test_encl does not consume it all. +*/ +EXPECT_LT (total_size +PAGE_SIZE , self->encl.encl_size); + +/* +* mmap() a page at end of existing enclave to be used for dynamic +* EPC page. +*/ + + addr =mmap ((void *)self->encl.encl_base + total_size,PAGE_SIZE , +PROT_READ |PROT_WRITE |PROT_EXEC ,MAP_SHARED |MAP_FIXED , + self->encl.fd,0 ); +EXPECT_NE (addr,MAP_FAILED ); + + self->run.exception_vector =0 ; + self->run.exception_error_code =0 ; + self->run.exception_addr =0 ; + +/* +* Run EACCEPT on new page to trigger the #PF->EAUG->EACCEPT(again +* without a #PF). All should be transparent to userspace. +*/ + eaccept_op.epc_addr = self->encl.encl_base + total_size; + eaccept_op.flags =SGX_SECINFO_R |SGX_SECINFO_W |SGX_SECINFO_REG |SGX_SECINFO_PENDING ; + eaccept_op.ret =0 ; + eaccept_op.header.type =ENCL_OP_EACCEPT ; + +EXPECT_EQ (ENCL_CALL (&eaccept_op, &self->run, true),0 ); + +if (self->run.exception_vector ==14 && + self->run.exception_error_code ==4 && + self->run.exception_addr == self->encl.encl_base + total_size) { +munmap (addr,PAGE_SIZE ); +SKIP (return,"Kernel does not support adding pages to initialized enclave" ); + } + +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); +EXPECT_EQ (eaccept_op.ret,0 ); + +/* +* New page should be accessible from within enclave - attempt to +* write to it. +*/ + put_addr_op.value =MAGIC ; + put_addr_op.addr = (unsigned long )addr; + put_addr_op.header.type =ENCL_OP_PUT_TO_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&put_addr_op, &self->run, true),0 ); + +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +/* +* Read memory from newly added page that was just written to, +* confirming that data previously written (MAGIC) is present. +*/ + get_addr_op.value =0 ; + get_addr_op.addr = (unsigned long )addr; + get_addr_op.header.type =ENCL_OP_GET_FROM_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&get_addr_op, &self->run, true),0 ); + +EXPECT_EQ (get_addr_op.value,MAGIC ); +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +munmap (addr,PAGE_SIZE ); +} +TEST_HARNESS_MAIN
Signature · 1 line
2.25.1
Jarkko Sakkinen
maintainerRe: [PATCH V2 21/32] selftests/sgx: Test two different SGX2 EAUG flows
tools/testing/selftests/sgx/main.c TEST_F(enclave, epcm_permissions) · line 936
Quoted 3 lines · Reinette Chatre
tools/testing/selftests/sgx/main.c +3
-0+ addr =mmap ((void *)self->encl.encl_base + total_size,PAGE_SIZE , +PROT_READ |PROT_WRITE |PROT_EXEC , +MAP_SHARED |MAP_FIXED , self->encl.fd,0 );
Maybe inline comment to just state that this is expected to work because the range does not contain enclave pages, just as a reminder (had xref to sgx_encl_may_map() to check this assumption). Otherwise, fine.
BR, Jarkko
PATCH v2 22/32 x86/sgx: Support modifying SGX page type
In reply to Reinette ChatreEvery enclave contains one or more Thread Control Structures (TCS). The TCS contains meta-data used by the hardware to save and restore thread specific information when entering/exiting the enclave. With SGX1 an enclave needs to be created with enough TCSs to support the largest number of threads expecting to use the enclave and enough enclave pages to meet all its anticipated memory demands. In SGX1 all pages remain in the enclave until the enclave is unloaded.
SGX2 introduces a new function, ENCLS[EMODT], that is used to change the type of an enclave page from a regular (SGX_PAGE_TYPE_REG) enclave page to a TCS (SGX_PAGE_TYPE_TCS) page or change the type from a regular (SGX_PAGE_TYPE_REG) or TCS (SGX_PAGE_TYPE_TCS)
page to a trimmed (SGX_PAGE_TYPE_TRIM) page (setting it up for later removal).
With the existing support of dynamically adding regular enclave pages to an initialized enclave and changing the page type to TCS it is possible to dynamically increase the number of threads supported by an enclave.
Changing the enclave page type to SGX_PAGE_TYPE_TRIM is the first step of dynamically removing pages from an initialized enclave. The complete page removal flow is:
1) Change the type of the pages to be removed to SGX_PAGE_TYPE_TRIM using the SGX_IOC_ENCLAVE_MODIFY_TYPE ioctl() introduced here.
2) Approve the page removal by running ENCLU[EACCEPT] from within the enclave.
3) Initiate actual page removal using the ioctl() introduced in the following patch.
Add ioctl() SGX_IOC_ENCLAVE_MODIFY_TYPE to support changing SGX enclave page types within an initialized enclave. With SGX_IOC_ENCLAVE_MODIFY_TYPE the user specifies a page range and the enclave page type to be applied to all pages in the provided range. The ioctl() itself can return an error code based on failures encountered by the kernel. It is also possible for SGX specific failures to be encountered. Add a result output parameter to communicate the SGX return code. It is possible for the enclave page type change request to fail on any page within the provided range. Support partial success by returning the number of pages that were successfully changed.
After the page type is changed the page continues to be accessible from the kernel perspective with page table entries and internal state. The page may be moved to swap. Any access until ENCLU[EACCEPT] will encounter a page fault with SGX flag set in error code.
---
Changes since V1:
- Remove the "Earlier changes ..." paragraph (Jarkko).
- Change "new ioctl" text to "Add SGX_IOC_ENCLAVE_MOD_TYPE" (Jarkko).
- Discussion about EPCM interaction and the EPCM MODIFIED bit is moved to new patch that introduces the ENCLS[EMODT] wrapper while keeping the higher level discussion on page accessibility in
this commit log (Jarkko).
- Rename SGX_IOC_PAGE_MODT ioctl() to SGX_IOC_ENCLAVE_MODIFY_TYPE (Jarkko).
- Rename struct sgx_page_modt to struct sgx_enclave_modt in support of ioctl() rename.
- Rename sgx_page_modt() to sgx_enclave_modt() and sgx_ioc_page_modt() to sgx_ioc_enclave_modt() in support of ioctl() rename.
- Provide secinfo as parameter to ioctl() instead of just page type (Jarkko).
- Update comments to refer to new ioctl() names.
- Use new SGX2 checking helper().
- Use ETRACK flow utility.
- Move kernel-doc to function that provides documentation for Documentation/x86/sgx.rst.
- Remove redundant comment.
- Use offset/length validation utility.
- Make explicit which members of struct sgx_enclave_modt are for output (Dave).
arch/x86/include/uapi/asm/sgx.h | 20 +++ arch/x86/kernel/cpu/sgx/ioctl.c | 212 ++++++++++++++++++++++++++++++++ 2 files changed, 232 insertions(+)
arch/x86/include/uapi/asm/sgx.h +20 -0
@@ -33,6 +33,8 @@ enum sgx_page_flags { _IOWR(SGX_MAGIC, 0x05, struct sgx_enclave_relax_perm)#define SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS \ _IOWR(SGX_MAGIC, 0x06, struct sgx_enclave_restrict_perm) +#define SGX_IOC_ENCLAVE_MODIFY_TYPE \ + _IOWR(SGX_MAGIC, 0x07, struct sgx_enclave_modt)/** * struct sgx_enclave_create - parameter structure for the
@@ -116,6 +118,24 @@ struct sgx_enclave_restrict_perm {__u64 count; }; +/** +* struct sgx_enclave_modt - parameters for %SGX_IOC_ENCLAVE_MODIFY_TYPE +* @offset: starting page offset (page aligned relative to enclave base +* address defined in SECS) +* @length: length of memory (multiple of the page size) +* @secinfo: address for the SECINFO data containing the new type +* for pages in range described by @offset and @length +* @result: (output) SGX result code of ENCLS[EMODT] function +* @count: (output) bytes successfully changed (multiple of page size) +*/ +struct sgx_enclave_modt { +__u64 offset; +__u64 length; +__u64 secinfo; +__u64 result; +__u64 count; +}; +struct sgx_enclave_run ;/**
arch/x86/kernel/cpu/sgx/ioctl.c +212 -0
224 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 23/32 x86/sgx: Support complete page removal
In reply to Reinette ChatreThe SGX2 page removal flow was introduced in previous patch and is as follows:
1) Change the type of the pages to be removed to SGX_PAGE_TYPE_TRIM using the ioctl() SGX_IOC_ENCLAVE_MODIFY_TYPE introduced in previous patch.
2) Approve the page removal by running ENCLU[EACCEPT] from within the enclave.
3) Initiate actual page removal using the ioctl() SGX_IOC_ENCLAVE_REMOVE_PAGES introduced here.
Support the final step of the SGX2 page removal flow with ioctl() SGX_IOC_ENCLAVE_REMOVE_PAGES. With this ioctl() the user specifies a page range that should be removed. All pages in the provided range should have the SGX_PAGE_TYPE_TRIM page type and the request will fail with EPERM (Operation not permitted) if a page that does not have the correct type is encountered. Page removal can fail on any page within the provided range. Support partial success by returning the number of pages that were successfully removed.
Since actual page removal will succeed even if ENCLU[EACCEPT] was not run from within the enclave the ENCLU[EMODPR] instruction with RWX permissions is used as a no-op mechanism to ensure ENCLU[EACCEPT] was successfully run from within the enclave before the enclave page is removed.
If the user omits running SGX_IOC_ENCLAVE_REMOVE_PAGES the pages will still be removed when the enclave is unloaded.
---
Changes since V1:
- Update comments to refer to new ioctl() names SGX_IOC_PAGE_MODT -> SGX_IOC_ENCLAVE_MODIFY_TYPE.
- Fix kernel-doc to have () as part of function name.
- Change name of ioctl():
SGX_IOC_PAGE_REMOVE -> SGX_IOC_ENCLAVE_REMOVE_PAGES (Jarkko).
- With the above name change the page removal ioctl() has its name aligned with existing SGX_IOC_ENCLAVE_ADD_PAGES ioctl(). Also align naming of struct and functions:
struct sgx_page_remove -> struct sgx_enclave_remove_pages sgx_page_remove() -> sgx_encl_remove_pages() sgx_ioc_page_remove() -> sgx_ioc_enclave_remove_pages()
- Use new SGX2 checking helper.
- When loading enclave page, make error code consistent with other instances to help user distinguish between permanent and temporary failures.
- Move kernel-doc to function that provides documentation for Documentation/x86/sgx.rst.
- Remove redundant comment.
- Use offset/length validation utility.
- Make explicit which member of struct sgx_enclave_remove_pages is for output (Dave).
arch/x86/include/uapi/asm/sgx.h | 21 +++++ arch/x86/kernel/cpu/sgx/ioctl.c | 145 ++++++++++++++++++++++++++++++++ 2 files changed, 166 insertions(+)
arch/x86/include/uapi/asm/sgx.h +21 -0
@@ -35,6 +35,8 @@ enum sgx_page_flags { _IOWR(SGX_MAGIC, 0x06, struct sgx_enclave_restrict_perm)#define SGX_IOC_ENCLAVE_MODIFY_TYPE \ _IOWR(SGX_MAGIC, 0x07, struct sgx_enclave_modt) +#define SGX_IOC_ENCLAVE_REMOVE_PAGES \ + _IOWR(SGX_MAGIC, 0x08, struct sgx_enclave_remove_pages)/** * struct sgx_enclave_create - parameter structure for the
@@ -136,6 +138,25 @@ struct sgx_enclave_modt {__u64 count; }; +/** +* struct sgx_enclave_remove_pages - %SGX_IOC_ENCLAVE_REMOVE_PAGES parameters +* @offset: starting page offset (page aligned relative to enclave base +* address defined in SECS) +* @length: length of memory (multiple of the page size) +* @count: (output) bytes successfully changed (multiple of page size) +* +* Regular (PT_REG) or TCS (PT_TCS) can be removed from an initialized +* enclave if the system supports SGX2. First, the %SGX_IOC_ENCLAVE_MODIFY_TYPE +* ioctl() should be used to change the page type to PT_TRIM. After that +* succeeds ENCLU[EACCEPT] should be run from within the enclave and then +* %SGX_IOC_ENCLAVE_REMOVE_PAGES can be used to complete the page removal. +*/ +struct sgx_enclave_remove_pages { +__u64 offset; +__u64 length; +__u64 count; +}; +struct sgx_enclave_run ;/**
arch/x86/kernel/cpu/sgx/ioctl.c +145 -0
157 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 24/32 Documentation/x86: Introduce enclave runtime management section
In reply to Reinette ChatreEnclave runtime management is introduced following the pattern of the section describing enclave building. Provide a brief summary of enclave runtime management, pointing to the functions implementing the ioctl()s that will contain details within their kernel-doc.
---
Changes since V1:
- New patch.
Documentation/x86/sgx.rst | 16 ++++++++++++++++ 1 file changed, 16 insertions(+)
Documentation/x86/sgx.rst +16 -0
@@ -138,6 +138,22 @@ pages and establish enclave page permissions. sgx_ioc_enclave_init sgx_ioc_enclave_provision +Enclave runtime management +-------------------------- + +Systems supporting SGX2 additionally support changes to initialized +enclaves: modifying enclave page permissions and type, and dynamically +adding and removing of enclave pages. When an enclave accesses an address +within its address range that does not have a backing page then a new +regular page will be dynamically added to the enclave. The enclave is +still required to run EACCEPT on the new page before it can be used. + +.. kernel-doc:: arch/x86/kernel/cpu/sgx/ioctl.c + :functions: sgx_ioc_enclave_relax_perm + sgx_ioc_enclave_restrict_perm + sgx_ioc_enclave_modt + sgx_ioc_enclave_remove_pages + Enclave vDSO ------------
Signature · 1 line
2.25.1
PATCH v2 25/32 selftests/sgx: Introduce dynamic entry point
In reply to Reinette ChatreThe test enclave (test_encl.elf) is built with two initialized Thread Control Structures (TCS) included in the binary. Both TCS are initialized with the same entry point, encl_entry, that correctly computes the absolute address of the stack based on the stack of each TCS that is also built into the binary.
A new TCS can be added dynamically to the enclave and requires to be initialized with an entry point used to enter the enclave. Since the existing entry point, encl_entry, assumes that the TCS and its stack exists at particular offsets within the binary it is not able to handle a dynamically added TCS and its stack.
Introduce a new entry point, encl_dyn_entry, that initializes the absolute address of that thread's stack to the address immediately preceding the TCS itself. It is now possible to dynamically add a contiguous memory region to the enclave with the new stack preceding the new TCS. With the new TCS initialized with encl_dyn_entry as entry point the absolute address of the stack is computed correctly on entry.
---
No changes since V1.
tools/testing/selftests/sgx/test_encl_bootstrap.S | 6 ++++++ 1 file changed, 6 insertions(+)
tools/testing/selftests/sgx/test_encl_bootstrap.S +6 -0
@@ -45,6 +45,12 @@ encl_entry: # TCS #2. By adding the value of encl_stack to it, we get # the absolute address for the stack. lea (encl_stack)(%rbx), %rax + jmp encl_entry_core +encl_dyn_entry: + # Entry point for dynamically created TCS page expected to follow + # its stack directly. + lea -1(%rbx), %rax +encl_entry_core: xchg %rsp, %rax push %rax
Signature · 1 line
2.25.1
PATCH v2 26/32 selftests/sgx: Introduce TCS initialization enclave operation
In reply to Reinette ChatreThe Thread Control Structure (TCS) contains meta-data used by the hardware to save and restore thread specific information when entering/exiting the enclave. A TCS can be added to an initialized enclave by first adding a new regular enclave page, initializing the content of the new page from within the enclave, and then changing that page's type to a TCS.
Support the initialization of a TCS from within the enclave. The variable information needed that should be provided from outside the enclave is the address of the TCS, address of the State Save Area (SSA), and the entry point that the thread should use to enter the enclave. With this information provided all needed fields of a TCS can be initialized.
---
No changes since V1.
tools/testing/selftests/sgx/defines.h | 8 +++++++ tools/testing/selftests/sgx/test_encl.c | 30 +++++++++++++++++++++++++ 2 files changed, 38 insertions(+)
tools/testing/selftests/sgx/defines.h +8 -0
@@ -26,6 +26,7 @@ enum encl_op_type {ENCL_OP_NOP ,ENCL_OP_EACCEPT ,ENCL_OP_EMODPE , +ENCL_OP_INIT_TCS_PAGE , ENCL_OP_MAX, };
@@ -68,4 +69,11 @@ struct encl_op_emodpe { uint64_t flags; }; +struct encl_op_init_tcs_page { +struct encl_op_header header; +uint64_t tcs_page; +uint64_t ssa; +uint64_t entry; +}; +#endif /* DEFINES_H */
tools/testing/selftests/sgx/test_encl.c +30 -0
@@ -57,6 +57,35 @@ static void *memcpy(void *dest, const void *src, size_t n)return dest; } +static void *memset (void *dest,int c,size_t n) +{ +size_t i; + +for (i =0 ; i < n; i++) + ((char *)dest)[i] = c; + +return dest; +} + +static void do_encl_init_tcs_page (void *_op) +{ +struct encl_op_init_tcs_page *op = _op; +void *tcs = (void *)op->tcs_page; +uint32_t val_32; + +memset (tcs,0 ,16 );/* STATE and FLAGS */ +memcpy (tcs +16 , &op->ssa,8 );/* OSSA */ +memset (tcs +24 ,0 ,4 );/* CSSA */ + val_32 =1 ; +memcpy (tcs +28 , &val_32,4 );/* NSSA */ +memcpy (tcs +32 , &op->entry,8 );/* OENTRY */ +memset (tcs +40 ,0 ,24 );/* AEP, OFSBASE, OGSBASE */ + val_32 =0xFFFFFFFF ; +memcpy (tcs +64 , &val_32,4 );/* FSLIMIT */ +memcpy (tcs +68 , &val_32,4 );/* GSLIMIT */ +memset (tcs +72 ,0 ,4024 );/* Reserved */ +} +static void do_encl_op_put_to_buf (void *op) {struct encl_op_put_to_buf *op2 = op;
@@ -100,6 +129,7 @@ void encl_body(void *rdi, void *rsi) do_encl_op_nop, do_encl_eaccept, do_encl_emodpe, + do_encl_init_tcs_page, };struct encl_op_header *op = (struct encl_op_header *)rdi;
Signature · 1 line
2.25.1
PATCH v2 27/32 selftests/sgx: Test complete changing of page type flow
In reply to Reinette ChatreSupport for changing an enclave page's type enables an initialized enclave to be expanded with support for more threads by changing the type of a regular enclave page to that of a Thread Control Structure (TCS). Additionally, being able to change a TCS or regular enclave page's type to be trimmed (SGX_PAGE_TYPE_TRIM) initiates the removal of the page from the enclave.
Test changing page type to TCS as well as page removal flows in two phases: In the first phase support for a new thread is dynamically added to an initialized enclave and in the second phase the pages associated with the new thread are removed from the enclave. As an additional sanity check after the second phase the page used as a TCS page during the first phase is added back as a regular page and ensured that it can be written to (which is not possible if it was a TCS page).
---
Changes since V1:
- Update to support ioctl() name change (SGX_IOC_PAGE_MODT -> SGX_IOC_ENCLAVE_MODIFY_TYPE) and provide secinfo as parameter instead of just page type (Jarkko).
- Update test to reflect page removal ioctl() and struct name change:
SGX_IOC_PAGE_REMOVE->SGX_IOC_ENCLAVE_REMOVE_PAGES,
struct sgx_page_remove -> struct sgx_enclave_remove_pages (Jarkko).
- Use ioctl() instead of ioctl (Dave).
tools/testing/selftests/sgx/load.c | 41 ++++ tools/testing/selftests/sgx/main.c | 347 +++++++++++++++++++++++++++++ tools/testing/selftests/sgx/main.h | 1 + 3 files changed, 389 insertions(+)
tools/testing/selftests/sgx/load.c +41 -0
@@ -130,6 +130,47 @@ static bool encl_ioc_add_pages(struct encl *encl, struct encl_segment *seg)return true; } +/* +* Parse the enclave code's symbol table to locate and return address of +* the provided symbol +*/ +uint64_t encl_get_entry (struct encl *encl,const char *symbol) +{ +Elf64_Shdr *sections; +Elf64_Sym *symtab; +Elf64_Ehdr *ehdr; +char *sym_names; +int num_sym; +int i; + + ehdr = encl->bin; + sections = encl->bin + ehdr->e_shoff; + +for (i =0 ; i < ehdr->e_shnum; i++) { +if (sections[i].sh_type ==SHT_SYMTAB ) { + symtab = (Elf64_Sym *)((char *)encl->bin + sections[i].sh_offset); + num_sym = sections[i].sh_size / sections[i].sh_entsize; +break ; + } + } + +for (i =0 ; i < ehdr->e_shnum; i++) { +if (sections[i].sh_type ==SHT_STRTAB ) { + sym_names = (char *)encl->bin + sections[i].sh_offset; +break ; + } + } + +for (i =0 ; i < num_sym; i++) { +Elf64_Sym *sym = &symtab[i]; + +if (!strcmp (symbol, sym_names + sym->st_name)) +return (uint64_t )sym->st_value; + } + +return 0 ; +} +bool encl_load (const char *path,struct encl *encl,unsigned long heap_size) {const char device_path[] ="/dev/sgx_enclave" ;
tools/testing/selftests/sgx/main.c +347 -0
351 lines not on this page · raw message
tools/testing/selftests/sgx/main.h +1 -0
@@ -38,6 +38,7 @@ void encl_delete(struct encl *ctx);bool encl_load (const char *path,struct encl *encl,unsigned long heap_size);bool encl_measure (struct encl *encl);bool encl_build (struct encl *encl); +uint64_t encl_get_entry (struct encl *encl,const char *symbol);int sgx_enter_enclave (void *rdi,void *rsi,long rdx,u32 function,void *r8,void *r9,struct sgx_enclave_run *run);
Signature · 1 line
2.25.1
PATCH v2 28/32 selftests/sgx: Test faulty enclave behavior
In reply to Reinette ChatreRemoving a page from an initialized enclave involves three steps:
first the user requests changing the page type to SGX_PAGE_TYPE_TRIM via an ioctl(), on success the ENCLU[EACCEPT] instruction needs to be run from within the enclave to accept the page removal, finally the user requests page removal to be completed via an ioctl(). Only after acceptance (ENCLU[EACCEPT]) from within the enclave can the kernel remove the page from a running enclave.
Test the behavior when the user's request to change the page type succeeds, but the ENCLU[EACCEPT] instruction is not run before the ioctl() requesting page removal is run. This should not be permitted.
---
Changes since V1:
- Update to support ioctl() name change (SGX_IOC_PAGE_MODT -> SGX_IOC_ENCLAVE_MODIFY_TYPE) and provide secinfo as parameter instead of just page type (Jarkko).
- Update test to reflect page removal ioctl() and struct name change:
SGX_IOC_PAGE_REMOVE->SGX_IOC_ENCLAVE_REMOVE_PAGES,
struct sgx_page_remove -> struct sgx_enclave_remove_pages (Jarkko).
- Use ioctl() instead of ioctl in text (Dave).
tools/testing/selftests/sgx/main.c | 116 +++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+)
tools/testing/selftests/sgx/main.c +116 -0
@@ -1472,4 +1472,120 @@ TEST_F(enclave, tcs_create)munmap (addr,3 *PAGE_SIZE ); } +/* +* Ensure sane behavior if user requests page removal, does not run +* EACCEPT from within enclave but still attempts to finalize page removal +* with the SGX_IOC_ENCLAVE_REMOVE_PAGES ioctl(). The latter should fail +* because the removal was not EACCEPTed from within the enclave. +*/ +TEST_F(enclave, remove_added_page_no_eaccept) +{ +struct sgx_enclave_remove_pages remove_ioc; +struct encl_op_get_from_addr get_addr_op; +struct encl_op_put_to_addr put_addr_op; +struct sgx_enclave_modt modt_ioc; +struct sgx_secinfo secinfo; +unsigned long data_start; +int ret, errno_save; + +ASSERT_TRUE (setup_test_encl (ENCL_HEAP_SIZE_DEFAULT , &self->encl, _metadata)); + +memset (&self->run,0 ,sizeof (self->run)); + self->run.tcs = self->encl.encl_base; + +/* +* Hardware (SGX2) and kernel support is needed for this test. Start +* with check that test has a chance of succeeding. +*/ +memset (&modt_ioc,0 ,sizeof (modt_ioc)); + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &modt_ioc); + +if (ret ==-1 ) { +if (errno ==ENOTTY ) +SKIP (return,"Kernel does not support SGX_IOC_ENCLAVE_MODIFY_TYPE ioctl()" ); +else if (errno ==ENODEV ) +SKIP (return,"System does not support SGX2" ); + } + +/* +* Invalid parameters were provided during sanity check, +* expect command to fail. +*/ +EXPECT_EQ (ret,-1 ); + +/* +* Page that will be removed is the second data page in the .data +* segment. This forms part of the local encl_buffer within the +* enclave. +*/ + data_start = self->encl.encl_base + +encl_get_data_offset (&self->encl) +PAGE_SIZE ; + +/* +* Sanity check that page at @data_start is writable before +* removing it. +* +* Start by writing MAGIC to test page. +*/ + put_addr_op.value =MAGIC ; + put_addr_op.addr = data_start; + put_addr_op.header.type =ENCL_OP_PUT_TO_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&put_addr_op, &self->run, true),0 ); + +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +/* +* Read memory that was just written to, confirming that data +* previously written (MAGIC) is present. +*/ + get_addr_op.value =0 ; + get_addr_op.addr = data_start; + get_addr_op.header.type =ENCL_OP_GET_FROM_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&get_addr_op, &self->run, true),0 ); + +EXPECT_EQ (get_addr_op.value,MAGIC ); +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +/* Start page removal by requesting change of page type to PT_TRIM */ +memset (&modt_ioc,0 ,sizeof (modt_ioc)); +memset (&secinfo,0 ,sizeof (secinfo)); + + secinfo.flags =SGX_PAGE_TYPE_TRIM <<8 ; + modt_ioc.offset =encl_get_data_offset (&self->encl) +PAGE_SIZE ; + modt_ioc.length =PAGE_SIZE ; + modt_ioc.secinfo = (unsigned long )&secinfo; + + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &modt_ioc); + errno_save = ret ==-1 ? errno :0 ; + +EXPECT_EQ (ret,0 ); +EXPECT_EQ (errno_save,0 ); +EXPECT_EQ (modt_ioc.result,0 ); +EXPECT_EQ (modt_ioc.count,4096 ); + +/* Skip EACCEPT */ + +/* Send final ioctl() to complete page removal */ +memset (&remove_ioc,0 ,sizeof (remove_ioc)); + + remove_ioc.offset =encl_get_data_offset (&self->encl) +PAGE_SIZE ; + remove_ioc.length =PAGE_SIZE ; + + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_REMOVE_PAGES , &remove_ioc); + errno_save = ret ==-1 ? errno :0 ; + +/* Operation not permitted since EACCEPT was omitted. */ +EXPECT_EQ (ret,-1 ); +EXPECT_EQ (errno_save,EPERM ); +EXPECT_EQ (remove_ioc.count,0 ); +} +TEST_HARNESS_MAIN
Signature · 1 line
2.25.1
PATCH v2 29/32 selftests/sgx: Test invalid access to removed enclave page
In reply to Reinette ChatreRemoving a page from an initialized enclave involves three steps:
(1) the user requests changing the page type to SGX_PAGE_TYPE_TRIM via the SGX_IOC_ENCLAVE_MODIFY_TYPE ioctl(), (2) on success the ENCLU[EACCEPT] instruction is run from within the enclave to accept the page removal, (3) the user initiates the actual removal of the page via the SGX_IOC_ENCLAVE_REMOVE_PAGES ioctl().
Test two possible invalid accesses during the page removal flow:
* Test the behavior when a request to remove the page by changing its type to SGX_PAGE_TYPE_TRIM completes successfully but instead of executing ENCLU[EACCEPT] from within the enclave the enclave attempts to read from the page. Even though the page is accessible from the page table entries its type is SGX_PAGE_TYPE_TRIM and thus not accessible according to SGX. The expected behavior is a page fault with the SGX flag set in the error code.
* Test the behavior when the page type is changed successfully and ENCLU[EACCEPT] was run from within the enclave. The final ioctl(), SGX_IOC_ENCLAVE_REMOVE_PAGES, is omitted and replaced with an attempt to access the page. Even though the page is accessible from the page table entries its type is SGX_PAGE_TYPE_TRIM and thus not accessible according to SGX. The expected behavior is a page fault with the SGX flag set in the error code.
---
Changes since V1:
- Update to support ioctl() name change (SGX_IOC_PAGE_MODT -> SGX_IOC_ENCLAVE_MODIFY_TYPE) and provide secinfo as parameter instead of just page type (Jarkko).
- Use ioctl() instead of ioctl (Dave).
tools/testing/selftests/sgx/main.c | 247 +++++++++++++++++++++++++++++ 1 file changed, 247 insertions(+)
tools/testing/selftests/sgx/main.c +247 -0
@@ -1588,4 +1588,251 @@ TEST_F(enclave, remove_added_page_no_eaccept)EXPECT_EQ (remove_ioc.count,0 ); } +/* +* Request enclave page removal but instead of correctly following with +* EACCEPT a read attempt to page is made from within the enclave. +*/ +TEST_F(enclave, remove_added_page_invalid_access) +{ +struct encl_op_get_from_addr get_addr_op; +struct encl_op_put_to_addr put_addr_op; +struct sgx_enclave_modt ioc; +struct sgx_secinfo secinfo; +unsigned long data_start; +int ret, errno_save; + +ASSERT_TRUE (setup_test_encl (ENCL_HEAP_SIZE_DEFAULT , &self->encl, _metadata)); + +memset (&self->run,0 ,sizeof (self->run)); + self->run.tcs = self->encl.encl_base; + +/* +* Hardware (SGX2) and kernel support is needed for this test. Start +* with check that test has a chance of succeeding. +*/ +memset (&ioc,0 ,sizeof (ioc)); + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &ioc); + +if (ret ==-1 ) { +if (errno ==ENOTTY ) +SKIP (return,"Kernel does not support SGX_IOC_ENCLAVE_MODIFY_TYPE ioctl()" ); +else if (errno ==ENODEV ) +SKIP (return,"System does not support SGX2" ); + } + +/* +* Invalid parameters were provided during sanity check, +* expect command to fail. +*/ +EXPECT_EQ (ret,-1 ); + +/* +* Page that will be removed is the second data page in the .data +* segment. This forms part of the local encl_buffer within the +* enclave. +*/ + data_start = self->encl.encl_base + +encl_get_data_offset (&self->encl) +PAGE_SIZE ; + +/* +* Sanity check that page at @data_start is writable before +* removing it. +* +* Start by writing MAGIC to test page. +*/ + put_addr_op.value =MAGIC ; + put_addr_op.addr = data_start; + put_addr_op.header.type =ENCL_OP_PUT_TO_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&put_addr_op, &self->run, true),0 ); + +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +/* +* Read memory that was just written to, confirming that data +* previously written (MAGIC) is present. +*/ + get_addr_op.value =0 ; + get_addr_op.addr = data_start; + get_addr_op.header.type =ENCL_OP_GET_FROM_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&get_addr_op, &self->run, true),0 ); + +EXPECT_EQ (get_addr_op.value,MAGIC ); +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +/* Start page removal by requesting change of page type to PT_TRIM. */ +memset (&ioc,0 ,sizeof (ioc)); +memset (&secinfo,0 ,sizeof (secinfo)); + + secinfo.flags =SGX_PAGE_TYPE_TRIM <<8 ; + ioc.offset =encl_get_data_offset (&self->encl) +PAGE_SIZE ; + ioc.length =PAGE_SIZE ; + ioc.secinfo = (unsigned long )&secinfo; + + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &ioc); + errno_save = ret ==-1 ? errno :0 ; + +EXPECT_EQ (ret,0 ); +EXPECT_EQ (errno_save,0 ); +EXPECT_EQ (ioc.result,0 ); +EXPECT_EQ (ioc.count,4096 ); + +/* +* Read from page that was just removed. +*/ + get_addr_op.value =0 ; + +EXPECT_EQ (ENCL_CALL (&get_addr_op, &self->run, true),0 ); + +/* +* From kernel perspective the page is present but according to SGX the +* page should not be accessible so a #PF with SGX bit set is +* expected. +*/ + +EXPECT_EQ (self->run.function,ERESUME ); +EXPECT_EQ (self->run.exception_vector,14 ); +EXPECT_EQ (self->run.exception_error_code,0x8005 ); +EXPECT_EQ (self->run.exception_addr, data_start); +} + +/* +* Request enclave page removal and correctly follow with +* EACCEPT but do not follow with removal ioctl() but instead a read attempt +* to removed page is made from within the enclave. +*/ +TEST_F(enclave, remove_added_page_invalid_access_after_eaccept) +{ +struct encl_op_get_from_addr get_addr_op; +struct encl_op_put_to_addr put_addr_op; +struct encl_op_eaccept eaccept_op; +struct sgx_enclave_modt ioc; +struct sgx_secinfo secinfo; +unsigned long data_start; +int ret, errno_save; + +ASSERT_TRUE (setup_test_encl (ENCL_HEAP_SIZE_DEFAULT , &self->encl, _metadata)); + +memset (&self->run,0 ,sizeof (self->run)); + self->run.tcs = self->encl.encl_base; + +/* +* Hardware (SGX2) and kernel support is needed for this test. Start +* with check that test has a chance of succeeding. +*/ +memset (&ioc,0 ,sizeof (ioc)); + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &ioc); + +if (ret ==-1 ) { +if (errno ==ENOTTY ) +SKIP (return,"Kernel does not support SGX_IOC_ENCLAVE_MODIFY_TYPE ioctl()" ); +else if (errno ==ENODEV ) +SKIP (return,"System does not support SGX2" ); + } + +/* +* Invalid parameters were provided during sanity check, +* expect command to fail. +*/ +EXPECT_EQ (ret,-1 ); + +/* +* Page that will be removed is the second data page in the .data +* segment. This forms part of the local encl_buffer within the +* enclave. +*/ + data_start = self->encl.encl_base + +encl_get_data_offset (&self->encl) +PAGE_SIZE ; + +/* +* Sanity check that page at @data_start is writable before +* removing it. +* +* Start by writing MAGIC to test page. +*/ + put_addr_op.value =MAGIC ; + put_addr_op.addr = data_start; + put_addr_op.header.type =ENCL_OP_PUT_TO_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&put_addr_op, &self->run, true),0 ); + +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +/* +* Read memory that was just written to, confirming that data +* previously written (MAGIC) is present. +*/ + get_addr_op.value =0 ; + get_addr_op.addr = data_start; + get_addr_op.header.type =ENCL_OP_GET_FROM_ADDRESS ; + +EXPECT_EQ (ENCL_CALL (&get_addr_op, &self->run, true),0 ); + +EXPECT_EQ (get_addr_op.value,MAGIC ); +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); + +/* Start page removal by requesting change of page type to PT_TRIM. */ +memset (&ioc,0 ,sizeof (ioc)); +memset (&secinfo,0 ,sizeof (secinfo)); + + secinfo.flags =SGX_PAGE_TYPE_TRIM <<8 ; + ioc.offset =encl_get_data_offset (&self->encl) +PAGE_SIZE ; + ioc.length =PAGE_SIZE ; + ioc.secinfo = (unsigned long )&secinfo; + + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &ioc); + errno_save = ret ==-1 ? errno :0 ; + +EXPECT_EQ (ret,0 ); +EXPECT_EQ (errno_save,0 ); +EXPECT_EQ (ioc.result,0 ); +EXPECT_EQ (ioc.count,4096 ); + + eaccept_op.epc_addr = (unsigned long )data_start; + eaccept_op.ret =0 ; + eaccept_op.flags =SGX_SECINFO_TRIM |SGX_SECINFO_MODIFIED ; + eaccept_op.header.type =ENCL_OP_EACCEPT ; + +EXPECT_EQ (ENCL_CALL (&eaccept_op, &self->run, true),0 ); + +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); +EXPECT_EQ (eaccept_op.ret,0 ); + +/* Skip ioctl() to remove page. */ + +/* +* Read from page that was just removed. +*/ + get_addr_op.value =0 ; + +EXPECT_EQ (ENCL_CALL (&get_addr_op, &self->run, true),0 ); + +/* +* From kernel perspective the page is present but according to SGX the +* page should not be accessible so a #PF with SGX bit set is +* expected. +*/ + +EXPECT_EQ (self->run.function,ERESUME ); +EXPECT_EQ (self->run.exception_vector,14 ); +EXPECT_EQ (self->run.exception_error_code,0x8005 ); +EXPECT_EQ (self->run.exception_addr, data_start); +} +TEST_HARNESS_MAIN
Signature · 1 line
2.25.1
PATCH v2 30/32 selftests/sgx: Test reclaiming of untouched page
In reply to Reinette ChatreRemoving a page from an initialized enclave involves three steps:
(1) the user requests changing the page type to PT_TRIM via the SGX_IOC_ENCLAVE_MODIFY_TYPE ioctl()
(2) on success the ENCLU[EACCEPT] instruction is run from within the enclave to accept the page removal
(3) the user initiates the actual removal of the page via the SGX_IOC_ENCLAVE_REMOVE_PAGES ioctl().
Remove a page that has never been accessed. This means that when the first ioctl() requesting page removal arrives, there will be no page table entry, yet a valid page table entry needs to exist for the ENCLU[EACCEPT] function to succeed. In this test it is verified that a page table entry can still be installed for a page that is in the process of being removed.
---
Changes since V1:
- Update to support ioctl() name change (SGX_IOC_PAGE_MODT -> SGX_IOC_ENCLAVE_MODIFY_TYPE) and provide secinfo as parameter instead of just page type (Jarkko).
- Update test to reflect page removal ioctl() and struct name change:
SGX_IOC_PAGE_REMOVE->SGX_IOC_ENCLAVE_REMOVE_PAGES,
struct sgx_page_remove -> struct sgx_enclave_remove_pages (Jarkko).
- Ensure test is skipped when SGX2 not supported by kernel.
tools/testing/selftests/sgx/main.c | 82 ++++++++++++++++++++++++++++++ 1 file changed, 82 insertions(+)
tools/testing/selftests/sgx/main.c +82 -0
@@ -1835,4 +1835,86 @@ TEST_F(enclave, remove_added_page_invalid_access_after_eaccept)EXPECT_EQ (self->run.exception_addr, data_start); } +TEST_F(enclave, remove_untouched_page) +{ +struct sgx_enclave_remove_pages remove_ioc; +struct encl_op_eaccept eaccept_op; +struct sgx_enclave_modt modt_ioc; +struct sgx_secinfo secinfo; +unsigned long data_start; +int ret, errno_save; + +ASSERT_TRUE (setup_test_encl (ENCL_HEAP_SIZE_DEFAULT , &self->encl, _metadata)); + +/* +* Hardware (SGX2) and kernel support is needed for this test. Start +* with check that test has a chance of succeeding. +*/ +memset (&modt_ioc,0 ,sizeof (modt_ioc)); + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &modt_ioc); + +if (ret ==-1 ) { +if (errno ==ENOTTY ) +SKIP (return,"Kernel does not support SGX_IOC_ENCLAVE_MODIFY_TYPE ioctl()" ); +else if (errno ==ENODEV ) +SKIP (return,"System does not support SGX2" ); + } + +/* +* Invalid parameters were provided during sanity check, +* expect command to fail. +*/ +EXPECT_EQ (ret,-1 ); + +/* SGX2 is supported by kernel and hardware, test can proceed. */ +memset (&self->run,0 ,sizeof (self->run)); + self->run.tcs = self->encl.encl_base; + + data_start = self->encl.encl_base + +encl_get_data_offset (&self->encl) +PAGE_SIZE ; + +memset (&modt_ioc,0 ,sizeof (modt_ioc)); +memset (&secinfo,0 ,sizeof (secinfo)); + + secinfo.flags =SGX_PAGE_TYPE_TRIM <<8 ; + modt_ioc.offset =encl_get_data_offset (&self->encl) +PAGE_SIZE ; + modt_ioc.length =PAGE_SIZE ; + modt_ioc.secinfo = (unsigned long )&secinfo; + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &modt_ioc); + errno_save = ret ==-1 ? errno :0 ; + +EXPECT_EQ (ret,0 ); +EXPECT_EQ (errno_save,0 ); +EXPECT_EQ (modt_ioc.result,0 ); +EXPECT_EQ (modt_ioc.count,4096 ); + +/* +* Enter enclave via TCS #1 and approve page removal by sending +* EACCEPT for removed page. +*/ + + eaccept_op.epc_addr = data_start; + eaccept_op.flags =SGX_SECINFO_TRIM |SGX_SECINFO_MODIFIED ; + eaccept_op.ret =0 ; + eaccept_op.header.type =ENCL_OP_EACCEPT ; + +EXPECT_EQ (ENCL_CALL (&eaccept_op, &self->run, true),0 ); +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); +EXPECT_EQ (eaccept_op.ret,0 ); + +memset (&remove_ioc,0 ,sizeof (remove_ioc)); + + remove_ioc.offset =encl_get_data_offset (&self->encl) +PAGE_SIZE ; + remove_ioc.length =PAGE_SIZE ; + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_REMOVE_PAGES , &remove_ioc); + errno_save = ret ==-1 ? errno :0 ; + +EXPECT_EQ (ret,0 ); +EXPECT_EQ (errno_save,0 ); +EXPECT_EQ (remove_ioc.count,4096 ); +} +TEST_HARNESS_MAIN
Signature · 1 line
2.25.1
PATCH v2 31/32 x86/sgx: Free up EPC pages directly to support large page ranges
In reply to Reinette ChatreThe page reclaimer ensures availability of EPC pages across all enclaves. In support of this it runs independently from the individual enclaves in order to take locks from the different enclaves as it writes pages to swap.
When needing to load a page from swap an EPC page needs to be available for its contents to be loaded into. Loading an existing enclave page from swap does not reclaim EPC pages directly if none are available, instead the reclaimer is woken when the available EPC pages are found to be below a watermark.
When iterating over a large number of pages in an oversubscribed environment there is a race between the reclaimer woken up and EPC pages reclaimed fast enough for the page operations to proceed.
Ensure there are EPC pages available before attempting to load a page that may potentially be pulled from swap into an available EPC page.
---
Changes since v1:
- Reword commit message.
arch/x86/kernel/cpu/sgx/ioctl.c | 6 ++++++ arch/x86/kernel/cpu/sgx/main.c | 6 ++++++ arch/x86/kernel/cpu/sgx/sgx.h | 1 + 3 files changed, 13 insertions(+)
arch/x86/kernel/cpu/sgx/ioctl.c +6 -0
@@ -962,6 +962,8 @@ static long sgx_enclave_restrict_perm(struct sgx_encl *encl,for (c =0 ; c < modp->length; c +=PAGE_SIZE ) { addr = encl->base + modp->offset + c; +sgx_direct_reclaim (); +mutex_lock (&encl->lock); entry =sgx_encl_load_page (encl, addr);
@@ -1156,6 +1158,8 @@ static long sgx_enclave_modt(struct sgx_encl *encl,for (c =0 ; c < modt->length; c +=PAGE_SIZE ) { addr = encl->base + modt->offset + c; +sgx_direct_reclaim (); +mutex_lock (&encl->lock); entry =sgx_encl_load_page (encl, addr);
@@ -1354,6 +1358,8 @@ static long sgx_encl_remove_pages(struct sgx_encl *encl,for (c =0 ; c < params->length; c +=PAGE_SIZE ) { addr = encl->base + params->offset + c; +sgx_direct_reclaim (); +mutex_lock (&encl->lock); entry =sgx_encl_load_page (encl, addr);
arch/x86/kernel/cpu/sgx/main.c +6 -0
@@ -370,6 +370,12 @@ static bool sgx_should_reclaim(unsigned long watermark) !list_empty (&sgx_active_page_list); } +void sgx_direct_reclaim (void ) +{ +if (sgx_should_reclaim (SGX_NR_LOW_PAGES )) +sgx_reclaim_pages (); +} +static int ksgxd (void *p) {set_freezable ();
arch/x86/kernel/cpu/sgx/sgx.h +1 -0
@@ -86,6 +86,7 @@ static inline void *sgx_get_epc_virt_addr(struct sgx_epc_page *page)struct sgx_epc_page *__sgx_alloc_epc_page (void );void sgx_free_epc_page (struct sgx_epc_page *page); +void sgx_direct_reclaim (void );void sgx_mark_page_reclaimable (struct sgx_epc_page *page);int sgx_unmark_page_reclaimable (struct sgx_epc_page *page);struct sgx_epc_page *sgx_alloc_epc_page (void *owner,bool reclaim);
Signature · 1 line
2.25.1
PATCH v2 32/32 selftests/sgx: Page removal stress test
In reply to Reinette ChatreCreate enclave with additional heap that consumes all physical SGX memory and then remove it.
Depending on the available SGX memory this test could take a significant time to run (several minutes) as it (1) creates the enclave, (2) changes the type of every page to be trimmed,
(3) enters the enclave once per page to run EACCEPT, before
(4) the pages are finally removed.
---
Changes since V1:
- Exit test completely on first failure of EACCEPT of a removed page. Since this is an oversubscribed test the number of pages on which this is attempted can be significant and in case of failure the per-page error logging would overwhelm the system.
- Update test to call renamed ioctl() (SGX_IOC_PAGE_MODT -> SGX_IOC_ENCLAVE_MODIFY_TYPE) and provide secinfo as parameter (Jarkko).
- Fixup definitions to be reverse xmas tree.
- Update test to reflect page removal ioctl() and struct name change:
SGX_IOC_PAGE_REMOVE->SGX_IOC_ENCLAVE_REMOVE_PAGES,
struct sgx_page_remove -> struct sgx_enclave_remove_pages (Jarkko).
- Ensure test is skipped when SGX2 not supported by kernel.
- Cleanup comments.
tools/testing/selftests/sgx/main.c | 122 +++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+)
tools/testing/selftests/sgx/main.c +122 -0
@@ -378,7 +378,129 @@ TEST_F(enclave, unclobbered_vdso_oversubscribed)EXPECT_EQ (get_op.value,MAGIC );EXPECT_EEXIT (&self->run);EXPECT_EQ (self->run.user_data,0 ); +} + +TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) +{ +struct sgx_enclave_remove_pages remove_ioc; +struct encl_op_get_from_buf get_op; +struct encl_op_eaccept eaccept_op; +struct encl_op_put_to_buf put_op; +struct sgx_enclave_modt modt_ioc; +struct sgx_secinfo secinfo; +struct encl_segment *heap; +unsigned long total_mem; +int ret, errno_save; +unsigned long addr; +unsigned long i; + +/* +* Create enclave with additional heap that is as big as all +* available physical SGX memory. +*/ + total_mem =get_total_epc_mem (); +ASSERT_NE (total_mem,0 ); +TH_LOG ("Creating an enclave with %lu bytes heap may take a while ..." , + total_mem); +ASSERT_TRUE (setup_test_encl (total_mem, &self->encl, _metadata)); + +/* +* Hardware (SGX2) and kernel support is needed for this test. Start +* with check that test has a chance of succeeding. +*/ +memset (&modt_ioc,0 ,sizeof (modt_ioc)); + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &modt_ioc); + +if (ret ==-1 ) { +if (errno ==ENOTTY ) +SKIP (return,"Kernel does not support SGX_IOC_ENCLAVE_MODIFY_TYPE ioctl()" ); +else if (errno ==ENODEV ) +SKIP (return,"System does not support SGX2" ); + } + +/* +* Invalid parameters were provided during sanity check, +* expect command to fail. +*/ +EXPECT_EQ (ret,-1 ); + +/* SGX2 is supported by kernel and hardware, test can proceed. */ +memset (&self->run,0 ,sizeof (self->run)); + self->run.tcs = self->encl.encl_base; + + heap = &self->encl.segment_tbl[self->encl.nr_segments -1 ]; + + put_op.header.type =ENCL_OP_PUT_TO_BUFFER ; + put_op.value =MAGIC ; + +EXPECT_EQ (ENCL_CALL (&put_op, &self->run, false),0 ); + +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.user_data,0 ); + + get_op.header.type =ENCL_OP_GET_FROM_BUFFER ; + get_op.value =0 ; + +EXPECT_EQ (ENCL_CALL (&get_op, &self->run, false),0 ); + +EXPECT_EQ (get_op.value,MAGIC ); +EXPECT_EEXIT (&self->run); +EXPECT_EQ (self->run.user_data,0 ); + +/* Trim entire heap. */ +memset (&modt_ioc,0 ,sizeof (modt_ioc)); +memset (&secinfo,0 ,sizeof (secinfo)); + + secinfo.flags =SGX_PAGE_TYPE_TRIM <<8 ; + modt_ioc.offset = heap->offset; + modt_ioc.length = heap->size; + modt_ioc.secinfo = (unsigned long )&secinfo; + +TH_LOG ("Changing type of %zd bytes to trimmed may take a while ..." , + heap->size); + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_MODIFY_TYPE , &modt_ioc); + errno_save = ret ==-1 ? errno :0 ; + +EXPECT_EQ (ret,0 ); +EXPECT_EQ (errno_save,0 ); +EXPECT_EQ (modt_ioc.result,0 ); +EXPECT_EQ (modt_ioc.count, heap->size); + +/* EACCEPT all removed pages. */ + addr = self->encl.encl_base + heap->offset; + + eaccept_op.flags =SGX_SECINFO_TRIM |SGX_SECINFO_MODIFIED ; + eaccept_op.header.type =ENCL_OP_EACCEPT ; + +TH_LOG ("Entering enclave to run EACCEPT for each page of %zd bytes may take a while ..." , + heap->size); +for (i =0 ; i < heap->size; i +=4096 ) { + eaccept_op.epc_addr = addr + i; + eaccept_op.ret =0 ; +EXPECT_EQ (ENCL_CALL (&eaccept_op, &self->run, true),0 ); + +EXPECT_EQ (self->run.exception_vector,0 ); +EXPECT_EQ (self->run.exception_error_code,0 ); +EXPECT_EQ (self->run.exception_addr,0 ); +ASSERT_EQ (eaccept_op.ret,0 ); +ASSERT_EQ (self->run.function,EEXIT ); + } + +/* Complete page removal. */ +memset (&remove_ioc,0 ,sizeof (remove_ioc)); + + remove_ioc.offset = heap->offset; + remove_ioc.length = heap->size; + +TH_LOG ("Removing %zd bytes from enclave may take a while ..." , + heap->size); + ret =ioctl (self->encl.fd,SGX_IOC_ENCLAVE_REMOVE_PAGES , &remove_ioc); + errno_save = ret ==-1 ? errno :0 ; + +EXPECT_EQ (ret,0 ); +EXPECT_EQ (errno_save,0 ); +EXPECT_EQ (remove_ioc.count, heap->size); }TEST_F (enclave, clobbered_vdso)
Signature · 1 line
2.25.1
1. This interface looks very odd to me. mmap() is the kernel interface for changing user space memory maps. Why are we introducing a new interface for this? You can just simply add a new mmap flag (i.e. MAP_SGX_TCS*) and then figure out which SGX instructions to execute based on the desired state of the memory maps. If you do this, none of the following ioctls are needed:
* SGX_IOC_ENCLAVE_RELAX_PERMISSIONS
* SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS
* SGX_IOC_ENCLAVE_REMOVE_PAGES
* SGX_IOC_ENCLAVE_MODIFY_TYPE
It also means that languages don't have to grow support for all these ioctls. Instead, they can just reuse the existing mmap() bindings with the new flag. Also, multiple operations can be combined into a single mmap() call, amortizing the changes over a single context switch.
2. Automatically adding pages with hard-coded permissions in a fault handler seems like a really bad idea. How do you distinguish between accesses which should result in an updated mapping and accesses that should result in a fault? IMHO, all unmapped page accesses should result in a page fault. mmap() should be called first to identify the correct permissions for these pages. Then the page handler should be updated to use the permissions from the mapping when backfilling physical pages. If I understand correctly, this should also obviate the need for the weird userspace callback to allow for execute permissions.
3. Implementing as I've suggested also means that we can lock down an enclave, for example - after code has been JITed, by closing the file descriptor. Once the file descriptor used to create the enclave is closed, no further mmap() can be performed on the enclave. Attempting to do EACCEPT on an unmapped page will generate a page fault.
* - I'm aware that a new flag might be frowned upon. I see a few other options:
1. reuse an existing flag which doesn't make sense in this context
2. communicate the page type in the offset argument
3. keep SGX_IOC_ENCLAVE_MODIFY_TYPE
Quoted 231 lines · Reinette Chatre
V1: https://lore.kernel.org/linux-sgx/[email protected]/
Changes since V1 that directly impact user space:
- SGX2 permission changes changed from a singleioctl()namedSGX_IOC_PAGE_MODPto two new ioctl()s:
SGX_IOC_ENCLAVE_RELAX_PERMISSIONSandSGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS, supported by two different parameter structures (SGX_IOC_ENCLAVE_RELAX_PERMISSIONSdoes not support a result output parameter) (Jarkko).User space flow impact: After user space runs ENCLU[EMODPE] it needs to call
SGX_IOC_ENCLAVE_RELAX_PERMISSIONSto have PTEs updated. Previously runningSGX_IOC_PAGE_MODPin this scenario resulted in EPCM.PR being set but callingSGX_IOC_ENCLAVE_RELAX_PERMISSIONSwill not result in EPCM.PR being set anymore and thus no need for an additional ENCLU[EACCEPT].-
SGX_IOC_ENCLAVE_RELAX_PERMISSIONSandSGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS
obtain new permissions from secinfo as parameter instead of the permissions directly (Jarkko).-
ioctl()supporting SGX2 page type change is renamed fromSGX_IOC_PAGE_MODTtoSGX_IOC_ENCLAVE_MODIFY_TYPE(Jarkko).-
SGX_IOC_ENCLAVE_MODIFY_TYPEobtains new page type from secinfo as parameter instead of the page type directly (Jarkko).-
ioctl()supporting SGX2 page removal is renamed fromSGX_IOC_PAGE_REMOVEtoSGX_IOC_ENCLAVE_REMOVE_PAGES(Jarkko).- All
ioctl()parameter structures have been renamed as a result of theioctl()renaming:
SGX_IOC_ENCLAVE_RELAX_PERMISSIONS=> structsgx_enclave_relax_permSGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS=> structsgx_enclave_restrict_permSGX_IOC_ENCLAVE_MODIFY_TYPE=> structsgx_enclave_modtSGX_IOC_ENCLAVE_REMOVE_PAGES=> structsgx_enclave_remove_pagesChanges since V1 that do not directly impact user space:
- Number of patches in series increased from 25 to 32 primarily because of splitting the original submission:
- Wrappers for the new SGX2 functions are introduced in three separate patches replacing the original "x86/sgx: Add wrappers for SGX2 functions"
(Jarkko).
- Moving and renamingsgx_encl_ewb_cpumask()is done with two patches replacing the original "x86/sgx: Use more generic name for enclave cpumask function" (Jarkko).
- Support for SGX2 EPCM permission changes is split into twoioctls(), one for relaxing and one for restricting permissions, each introduced by a new patch replacing the original "x86/sgx: Support enclave page permission changes" (Jarkko).
- Extracted code used by existingioctls()for usage by new ioctl()s into a new utility in new patch "x86/sgx: Create utility to validate user provided offset and length" (Dave did not specifically ask for this but it addresses his review feedback).
- Two new Documentation patches to support the SGX2 work ("Documentation/x86: Introduce enclave runtime management") and a dedicated section on the enclave permission management ("Documentation/x86: Document SGX permission details") (Andy).
- Most patches were reworked to improve the language by:
* aiming to refer to exact item instead of English rephrasing (Jarkko).
* useioctl()instead of ioctl throughout (Dave).
* Use "relaxed" instead of "exceed" when referring to permissions(Dave).- Improved documentation with several additions to
Documentation/x86/sgx.rst.
- Many smaller changes, please refer to individual patches.Hi Everybody,
The current Linux kernel support for SGX includes support for SGX1 that requires that an enclave be created with properties that accommodate all usages over its (the enclave's) lifetime. This includes properties such as permissions of enclave pages, the number of enclave pages, and the number of threads supported by the enclave.
Consequences of this requirement to have the enclave be created to accommodate all usages include:
* pages needing to support relocated code are required to have RWX permissions for their entire lifetime,
* an enclave needs to be created with the maximum stack and heap projected to be needed during the enclave's entire lifetime which can be longer than the processes running within it,
* an enclave needs to be created with support for the maximum number of threads projected to run in the enclave.Since SGX1 a few more functions were introduced, collectively called SGX2, that support modifications to an initialized enclave. Hardware supporting these functions are already available as listed on
https://github.com/ayeks/SGX-hardwareThis series adds support for SGX2, also referred to as Enclave Dynamic Memory Management (EDMM). This includes:
* Support modifying permissions of regular enclave pages belonging to an initialized enclave. New permissions are not allowed to exceed the originally vetted permissions. For example, RX isn't allowed unless the page was originally added with RX or RWX.
Modifying permissions is accomplished with two new ioctl()s:
SGX_IOC_ENCLAVE_RELAX_PERMISSIONSandSGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS.* Support dynamic addition of regular enclave pages to an initialized enclave. Pages are added with RW permissions as their "originally vetted permissions" (see previous bullet) and thus not allowed to be made executable at this time. Enabling dynamically added pages to obtain executable permissions require integration with user space policy that is deferred until the core SGX2 enabling is complete. Pages are dynamically added to an initialized enclave from the SGX page fault handler.
* Support expanding an initialized enclave to accommodate more threads. More threads can be accommodated by an enclave with the addition of Thread Control Structure (TCS) pages that is done by changing the type of regular enclave pages to TCS pages using a new
ioctl()SGX_IOC_ENCLAVE_MODIFY_TYPE.* Support removing regular and TCS pages from an initialized enclave. Removing pages is accomplished in two stages as supported by two new ioctl()s
SGX_IOC_ENCLAVE_MODIFY_TYPE(sameioctl()as mentioned in previous bullet) andSGX_IOC_ENCLAVE_REMOVE_PAGES.* Tests covering all the new flows, some edge cases, and one comprehensive stress scenario.
No additional work is needed to support SGX2 in a virtualized environment. The tests included in this series can also be run from a guest and was tested with the recent QEMU release based on 6.2.0 that supports SGX.
Patches 1 to 14 prepares the existing code for SGX2 support by introducing the SGX2 functions, making sure pages remain accessible after their enclave permissions are changed, and tracking enclave page types as well as runtime permissions as needed by SGX2.
Patches 15 through 32 are a mix of x86/sgx and selftests/sgx patches that follow the format where first an SGX2 feature is enabled and then followed by tests of the new feature and/or tests of scenarios that combine SGX2 features enabled up to that point.
In two cases (patches 20 and 31) code in support of SGX2 is separated out with detailed motivation to support the review.
This series is based on v5.17-rc2 with the following fixes additionally applied:
"selftests/sgx: Remove extra newlines in test output"
https://lore.kernel.org/linux-sgx/16317683a1822bbd44ab3ca48b60a9a217ac24de.1643754040.git.reinette.chatre@intel.com/
"selftests/sgx: Ensure enclave data available during debug print"
https://lore.kernel.org/linux-sgx/eaaeeb9122916d831942fc8a3043c687137314c1.1643754040.git.reinette.chatre@intel.com/
"selftests/sgx: Do not attempt enclave build without valid enclave"
https://lore.kernel.org/linux-sgx/4e4ea6d70c286c209964bec1e8d29ac8e692748b.1643754040.git.reinette.chatre@intel.com/
"selftests/sgx: Fix NULL-pointer-dereference upon early test failure"
https://lore.kernel.org/linux-sgx/89824888783fd8e770bfc64530c7549650a41851.1643754040.git.reinette.chatre@intel.com/
"x86/sgx: Add poison handling to reclaimer"
https://lore.kernel.org/linux-sgx/dcc95eb2aaefb042527ac50d0a50738c7c160dac.1643830353.git.reinette.chatre@intel.com/
"x86/sgx: Silence softlockup detection when releasing large enclaves"
https://lore.kernel.org/linux-sgx/b5e9f218064aa76e3026f778e1ad0a1d823e3db8.1643133224.git.reinette.chatre@intel.com/Your feedback will be greatly appreciated.
Regards,
Reinette
Reinette Chatre (32):
x86/sgx: Add short descriptions to ENCLS wrappers
x86/sgx: Add wrapper for SGX2 EMODPR function
x86/sgx: Add wrapper for SGX2 EMODT function
x86/sgx: Add wrapper for SGX2 EAUG functionDocumentation/x86: Document SGX permission details x86/sgx: Support VMA permissions more relaxed than enclave permissions x86/sgx: Addpfn_mkwrite()handler for present PTEs x86/sgx: x86/sgx: Addsgx_encl_page->vm_run_prot_bitsfor dynamic
permission changes
x86/sgx: Exportsgx_encl_ewb_cpumask()
x86/sgx: Renamesgx_encl_ewb_cpumask()assgx_encl_cpumask()x86/sgx: Move PTE zap code to newsgx_zap_enclave_ptes()x86/sgx: Makesgx_ipi_cb()available internally
x86/sgx: Create utility to validate user provided offset and length x86/sgx: Keep record of SGX page type
x86/sgx: Support relaxing of enclave page permissions x86/sgx: Support restricting of enclave page permissions selftests/sgx: Add test for EPCM permission changes selftests/sgx: Add test for TCS page permission changes x86/sgx: Support adding of pages to an initialized enclave x86/sgx: Tighten accessible memory range after enclave initialization selftests/sgx: Test two different SGX2 EAUG flows
x86/sgx: Support modifying SGX page type
x86/sgx: Support complete page removal
Documentation/x86: Introduce enclave runtime management section selftests/sgx: Introduce dynamic entry point selftests/sgx: Introduce TCS initialization enclave operation selftests/sgx: Test complete changing of page type flow selftests/sgx: Test faulty enclave behavior
selftests/sgx: Test invalid access to removed enclave page selftests/sgx: Test reclaiming of untouched page
x86/sgx: Free up EPC pages directly to support large page ranges selftests/sgx: Page removal stress testDocumentation/x86/sgx.rst | 64 +- arch/x86/include/asm/sgx.h | 8 + arch/x86/include/uapi/asm/sgx.h | 81 + arch/x86/kernel/cpu/sgx/encl.c | 334 +++- arch/x86/kernel/cpu/sgx/encl.h | 12 +- arch/x86/kernel/cpu/sgx/encls.h | 33 + arch/x86/kernel/cpu/sgx/ioctl.c | 831 ++++++++- arch/x86/kernel/cpu/sgx/main.c | 70 +- arch/x86/kernel/cpu/sgx/sgx.h | 3 + tools/testing/selftests/sgx/defines.h | 23 + tools/testing/selftests/sgx/load.c | 41 + tools/testing/selftests/sgx/main.c | 1484 +++++++++++++++++ tools/testing/selftests/sgx/main.h | 1 + tools/testing/selftests/sgx/test_encl.c | 68 + .../selftests/sgx/test_encl_bootstrap.S | 6 + 15 files changed, 2963 insertions(+), 96 deletions(-)base-commit:
26291c54e111ff6ba87a164d85d4a4e134b7315c
prerequisite-patch-id:3c3908f1c3536cc04ba020fb3e81f51395b44223
prerequisite-patch-id:e860923423c3387cf6fdcceb2fa41dc5e9454ef4
prerequisite-patch-id:986260c8bc4255eb61e2c4afa88d2b723e376423
prerequisite-patch-id:ba014a99fced2b57d5d9e2dfb9d80ddf4333c13e
prerequisite-patch-id:65cbb72889b6353a5639b984615d12019136b270
prerequisite-patch-id:e3296a2f0345a77c8a7ca91f76697ae2e1dca21f--
2.25.1
Hi Nathaniel,
Quoted 3 lines · Nathaniel McCallum
1. This interface looks very odd to me.
mmap()is the kernel interface for changing user space memory maps. Why are we introducing a new interface for this?
mmap() is the kernel interface used to create new mappings in the virtual address space of the calling process. This is different from the permissions and properties of the underlying file/memory being mapped.
A new interface is introduced because changes need to be made to the permissions and properties of the underlying enclave. A new virtual address space is not needed nor should existing VMAs be impacted.
This is similar to how mmap() is not used to change file permissions.
VMA permissions are separate from enclave page permissions as found in the EPCM (Enclave Page Cache Map). The current implementation (SGX1) already distinguishes between the VMA and EPCM permissions - for example, it is already possible to create a read-only VMA from enclave pages that have RW EPCM permissions. mmap() of a portion of EPC memory with a particular permission does not imply that the underlying EPCM permissions (should)have that permission.
Quoted 10 lines
You can just simply add a new mmap flag (i.e.
MAP_SGX_TCS*) and then figure out which SGX instructions to execute based on the desired state of the memory maps. If you do this, none of the following ioctls are needed:*
SGX_IOC_ENCLAVE_RELAX_PERMISSIONS
*SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS
*SGX_IOC_ENCLAVE_REMOVE_PAGES
*SGX_IOC_ENCLAVE_MODIFY_TYPE
Quoted 7 lines
It also means that languages don't have to grow support for all these ioctls. Instead, they can just reuse the existing
mmap()bindings with the new flag. Also, multiple operations can be combined into a singlemmap()call, amortizing the changes over a single context switch.2. Automatically adding pages with hard-coded permissions in a fault handler seems like a really bad idea.
Could you please elaborate why this is a bad idea?
Quoted 3 lines
How do you distinguish between
accesses which should result in an updated mapping and accesses that should result in a fault?
Accesses that should result in an updated mapping have two requirements:
(a) address accessed belongs to the enclave based on the address range specified during enclave create
(b) there is no backing enclave page for the address
Quoted 6 lines
IMHO, all unmapped page accesses should
result in a page fault.mmap()should be called first to identify the correct permissions for these pages.
Then the page handler should be
updated to use the permissions from the mapping when backfilling physical pages. If I understand correctly, this should also obviate
Regular enclave pages can _only_ be dynamically added with RW permission.
SGX2's support for adding regular pages to an enclave via the EAUG instruction is architecturally set at RW. The OS cannot change those permissions via the EAUG instruction nor can the OS do so with a different/additional instruction because:
* the OS is not able to relax permissions since that can only be done from within the enclave with ENCLU[EMODPE], thus it is not possible for the OS to dynamically add pages via EAUG as RW and then relax permissions to RWX.
* the OS is not able to EAUG a page and immediately attempt an EMODPR either as Jarkko also recently inquired about:
https://lore.kernel.org/linux-sgx/[email protected]/
Quoted 2 lines
the need for the weird userspace callback to allow for execute permissions.
User policy integration would always be required to allow execute permissions on a writable page. This is not expected to be a userspace callback but instead integration with existing user policy subsystem(s).
Quoted 6 lines
3. Implementing as I've suggested also means that we can lock down an enclave, for example - after code has been JITed, by closing the file descriptor. Once the file descriptor used to create the enclave is closed, no further
mmap()can be performed on the enclave. Attempting to do EACCEPT on an unmapped page will generate a page fault.
This is not clear to me. If the file descriptor is closed and no further mmap() is allowed then how would a process be able to enter the enclave to execute code within it?
This series does indeed lock down the address range to ensure that it is not possible to map memory that does not belong to the enclave after the enclave is created. Please see:
https://lore.kernel.org/linux-sgx/1b833dbce6c937f71523f4aaf4b2181b9673519f.1644274683.git.reinette.chatre@intel.com/
Quoted 6 lines
* - I'm aware that a new flag might be frowned upon. I see a few other options:
1. reuse an existing flag which doesn't make sense in this context
2. communicate the page type in the offset argument
3. keepSGX_IOC_ENCLAVE_MODIFY_TYPE
Reinette
Quoted 18 lines · Reinette Chatre
Hi Nathaniel,
Quoted 3 lines · Nathaniel McCallum
1. This interface looks very odd to me.
mmap()is the kernel interface for changing user space memory maps. Why are we introducing a new interface for this?
mmap()is the kernel interface used to create new mappings in the virtual address space of the calling process. This is different from the permissions and properties of the underlying file/memory being mapped.A new interface is introduced because changes need to be made to the permissions and properties of the underlying enclave. A new virtual address space is not needed nor should existing VMAs be impacted.
This is similar to how
mmap()is not used to change file permissions.
Quoted 7 lines
VMA permissions are separate from enclave page permissions as found in the EPCM (Enclave Page Cache Map). The current implementation (SGX1) already distinguishes between the VMA and EPCM permissions - for example, it is already possible to create a read-only VMA from enclave pages that have RW EPCM permissions.
mmap()of a portion of EPC memory with a particular permission does not imply that the underlying EPCM permissions (should)have that permission.
Yes. BUT... unlike the file permissions, this leaks an implementation detail.
The user process is governed by VMA permissions. And during enclave creation, it had to mmap() all the enclave regions to their final VMA permissions. So during enclave creation you have to use mmap() but after enclave creation you use custom APIs? That's inconsistent at best.
Forcing userspace to worry about the (mostly undocumented!) interactions between EPC, PTE and VMA permissions makes these APIs hard to use and difficult to reason about.
When I call SGX_IOC_ENCLAVE_RELAX_PERMISSIONS, do I also have to call mmap() to update the VMA permissions to match? It isn't clear. Nor is it really clear why I'm calling completely separate APIs.
Quoted 15 lines
Quoted 15 lines
You can just simply add a new mmap flag (i.e.
MAP_SGX_TCS*) and then figure out which SGX instructions to execute based on the desired state of the memory maps. If you do this, none of the following ioctls are needed:*
SGX_IOC_ENCLAVE_RELAX_PERMISSIONS
*SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS
*SGX_IOC_ENCLAVE_REMOVE_PAGES
*SGX_IOC_ENCLAVE_MODIFY_TYPEIt also means that languages don't have to grow support for all these ioctls. Instead, they can just reuse the existing
mmap()bindings with the new flag. Also, multiple operations can be combined into a singlemmap()call, amortizing the changes over a single context switch.
Quoted 4 lines
Quoted 2 lines
2. Automatically adding pages with hard-coded permissions in a fault handler seems like a really bad idea.
Could you please elaborate why this is a bad idea?
Because implementations that miss this subtlety suddenly have pages with magic permissions. Magic is bad. Explicit is good.
Quoted 8 lines
Quoted 3 lines
How do you distinguish between
accesses which should result in an updated mapping and accesses that should result in a fault?Accesses that should result in an updated mapping have two requirements:
(a) address accessed belongs to the enclave based on the address range specified during enclave create
(b) there is no backing enclave page for the address
What happens if the enclave is buggy? Or has been compromised. In both of those cases, there should be a userspace visible fault and pages should not be added.
Quoted 9 lines
Quoted 6 lines
IMHO, all unmapped page accesses should
result in a page fault.mmap()should be called first to identify the correct permissions for these pages.
Then the page handler should be
updated to use the permissions from the mapping when backfilling physical pages. If I understand correctly, this should also obviateRegular enclave pages can _only_ be dynamically added with RW permission.
Quoted 10 lines
SGX2's support for adding regular pages to an enclave via the EAUG instruction is architecturally set at RW. The OS cannot change those permissions via the EAUG instruction nor can the OS do so with a different/additional instruction because:
* the OS is not able to relax permissions since that can only be done from within the enclave with ENCLU[EMODPE], thus it is not possible for the OS to dynamically add pages via EAUG as RW and then relax permissions to RWX.
* the OS is not able to EAUG a page and immediately attempt an EMODPR either as Jarkko also recently inquired about:
https://lore.kernel.org/linux-sgx/[email protected]/
This design looks... unfinished. EAUG takes a PAGEINFO in RBX, but PAGEINFO.SECINFO must be zeroed and EAUG instead sets magic hard-coded permissions. Why doesn't EAUG just respect the permissions in PAGEINFO.SECINFO? We aren't told.
Further, if the enclave can do EMODPE, why does SGX_IOC_ENCLAVE_RELAX_PERMISSIONS even exist? None of the documentation explains what this ioctl even does. Does it update PTE permissions? VMA permissions? Nobody knows without reading the source code.
Userspace should not be bothered with the subtle details of the interaction between EPC, PTE and VMA permissions. But this API does everything it can do to expose all these details to userspace. And it doesn't bother to document them (probably because it is hard). It would be much better to avoid exposing these details to userspace.
IMHO, there should be a simple flow like this (if EAUG respects PAGEINFO.SECINFO):
1. Non-enclave calls mmap()/munmap().
2. Enclave issues EACCEPT, if necessary.
3. Enclave issues EMODPE, if necessary.
Notice that in the second step above, during the mmap() call, the kernel ensures that EPC, PTE and VMA are in sync and fails if they cannot be made to be compatible. Also note that in the above flow EAUG instructions can be efficiently batched.
Given the current poor state of the EAUG instruction, we might need to do this flow instead:
1. Enclave issues EACCEPT, if necessary. (Add RW pages...)
2. Non-enclave calls mmap()/munmap().
3. Enclave issues EACCEPT, if necessary.
4. Enclave issues EMODPE, if necessary.
However, doing EAUG only via the page access handler means that there is no way to batch EAUG instructions and this forces a context switch for every page you want to add. This has to be terrible for performance. Note specifically that the SDM calls out batching, which is currently impossible under this patch set. 35.5.7 - "Page allocation operations may be batched to improve efficiency."
As it stands today, if I want to add 256MiB of pages to an enclave, I'll have to do 2^16 context switches. That doesn't seem scalable.
Quoted 6 lines
Quoted 2 lines
the need for the weird userspace callback to allow for execute permissions.
User policy integration would always be required to allow execute permissions on a writable page. This is not expected to be a userspace callback but instead integration with existing user policy subsystem(s).
Why? This isn't documented.
Quoted 9 lines
Quoted 5 lines
3. Implementing as I've suggested also means that we can lock down an enclave, for example - after code has been JITed, by closing the file descriptor. Once the file descriptor used to create the enclave is closed, no further
mmap()can be performed on the enclave. Attempting to do EACCEPT on an unmapped page will generate a page fault.This is not clear to me. If the file descriptor is closed and no further
mmap()is allowed then how would a process be able to enter the enclave to execute code within it?
EENTER (or the vdso function) with the address of a TCS page, like normal. In Enarx, we don't retain the enclave fd after the final mmap() following EINIT. Everything works just fine.
Quoted 4 lines
This series does indeed lock down the address range to ensure that it is not possible to map memory that does not belong to the enclave after the enclave is created. Please see:
https://lore.kernel.org/linux-sgx/1b833dbce6c937f71523f4aaf4b2181b9673519f.1644274683.git.reinette.chatre@intel.com/
That's not what I'm talking about. I'm talking about a workflow like this:
1. Enclave initialization: ECREATE ... EINIT
2. EENTER
3. Enclave JITs some code (changes page permissions)
4. EEXIT
5. Close enclave fd.
6. EENTER
7. If an enclave attempts page modifications, a fault occurs.
Think of this similar to seccomp(). The enclave wants to do some dynamic page table manipulation. But then it wants to lock down page table modification so that, if compromised, attackers have no ability to obtain RWX permissions.
Hi Nathaniel,
Quoted 28 lines · Nathaniel McCallum
Quoted 18 lines · Reinette Chatre
Hi Nathaniel,
Quoted 3 lines · Nathaniel McCallum
1. This interface looks very odd to me.
mmap()is the kernel interface for changing user space memory maps. Why are we introducing a new interface for this?
mmap()is the kernel interface used to create new mappings in the virtual address space of the calling process. This is different from the permissions and properties of the underlying file/memory being mapped.A new interface is introduced because changes need to be made to the permissions and properties of the underlying enclave. A new virtual address space is not needed nor should existing VMAs be impacted.
This is similar to how
mmap()is not used to change file permissions.Quoted 7 lines
VMA permissions are separate from enclave page permissions as found in the EPCM (Enclave Page Cache Map). The current implementation (SGX1) already distinguishes between the VMA and EPCM permissions - for example, it is already possible to create a read-only VMA from enclave pages that have RW EPCM permissions.
mmap()of a portion of EPC memory with a particular permission does not imply that the underlying EPCM permissions (should)have that permission.
Quoted 1 line
Yes. BUT... unlike the file permissions, this leaks an implementation detail.
Not really - just like a RW file can be mapped read-only or RW, RW enclave memory can be mapped read-only or RW.
Quoted 6 lines
The user process is governed by VMA permissions. And during enclave creation, it had to
mmap()all the enclave regions to their final VMA permissions. So during enclave creation you have to usemmap()but after enclave creation you use custom APIs? That's inconsistent at best.
No. ioctl()s are consistently used to manage enclave memory.
The existing ioctls() SGX_IOC_ENCLAVE_CREATE, SGX_IOC_ENCLAVE_ADD_PAGES, and SGX_IOC_ENCLAVE_INIT are used to set up to initialize the enclave memory.
The new ioctls() are used to manage enclave memory after enclave initialization.
The enclave memory is thus managed with a consistent interface.
mmap() is required before SGX_IOC_ENCLAVE_CREATE to obtain a base address for the enclave that is required by the ioctl(). The rest of the ioctl()s, existing and new, are consistent in interface by not requiring a memory mapping but instead work from an offset from the base address.
Quoted 3 lines
Forcing userspace to worry about the (mostly undocumented!) interactions between EPC, PTE and VMA permissions makes these APIs hard to use and difficult to reason about.
This is not new. The current SGX1 user space is already prevented from creating a mapping of enclave memory that is more relaxed than the enclave memory. For example, if the enclave memory has RW EPCM permissions then it is not possible to mmap() that memory as RWX.
Quoted 3 lines
When I call
SGX_IOC_ENCLAVE_RELAX_PERMISSIONS, do I also have to callmmap()to update the VMA permissions to match? It isn't clear. Nor is
mprotect() may be the better call to use.
Quoted 17 lines
it really clear why I'm calling completely separate APIs.
Quoted 15 lines
Quoted 15 lines
You can just simply add a new mmap flag (i.e.
MAP_SGX_TCS*) and then figure out which SGX instructions to execute based on the desired state of the memory maps. If you do this, none of the following ioctls are needed:*
SGX_IOC_ENCLAVE_RELAX_PERMISSIONS
*SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS
*SGX_IOC_ENCLAVE_REMOVE_PAGES
*SGX_IOC_ENCLAVE_MODIFY_TYPEIt also means that languages don't have to grow support for all these ioctls. Instead, they can just reuse the existing
mmap()bindings with the new flag. Also, multiple operations can be combined into a singlemmap()call, amortizing the changes over a single context switch.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
2. Automatically adding pages with hard-coded permissions in a fault handler seems like a really bad idea.
Could you please elaborate why this is a bad idea?
Because implementations that miss this subtlety suddenly have pages with magic permissions. Magic is bad. Explicit is good.
There is no magic. Any new pages have to be accepted by the enclave. The enclave will not be able to access these pages unless explicitly accepted, ENCLU[EACCEPT], from within the enclave.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
How do you distinguish between
accesses which should result in an updated mapping and accesses that should result in a fault?
Quoted 8 lines
Quoted 4 lines
Accesses that should result in an updated mapping have two requirements:
(a) address accessed belongs to the enclave based on the address range specified during enclave create
(b) there is no backing enclave page for the addressWhat happens if the enclave is buggy? Or has been compromised. In both of those cases, there should be a userspace visible fault and pages should not be added.
If user space accesses a memory address with a regular read/write that results in a new page added then there is indeed a user space visible fault. You can see this flow in action in the "augment" test case in
https://lore.kernel.org/linux-sgx/32c1116934a588bd3e6c174684e3e36a05c0a4d4.1644274683.git.reinette.chatre@intel.com/
If user space indeed wants the page after encountering such a fault then it needs to enter the enclave again, from a different entry point, to run ENCLU[EACCEPT], before it can return to the original entry point to continue execution from the instruction that triggered the original read/write.
The only flow where a page is added without a user space visible fault is when user space explicitly runs the ENCLU[EACCEPT] to do so.
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
IMHO, all unmapped page accesses should
result in a page fault.mmap()should be called first to identify the correct permissions for these pages.
Then the page handler should be
updated to use the permissions from the mapping when backfilling physical pages. If I understand correctly, this should also obviateRegular enclave pages can _only_ be dynamically added with RW permission.
Quoted 10 lines
SGX2's support for adding regular pages to an enclave via the EAUG instruction is architecturally set at RW. The OS cannot change those permissions via the EAUG instruction nor can the OS do so with a different/additional instruction because:
* the OS is not able to relax permissions since that can only be done from within the enclave with ENCLU[EMODPE], thus it is not possible for the OS to dynamically add pages via EAUG as RW and then relax permissions to RWX.
* the OS is not able to EAUG a page and immediately attempt an EMODPR either as Jarkko also recently inquired about:
https://lore.kernel.org/linux-sgx/[email protected]/
Quoted 4 lines
This design looks... unfinished. EAUG takes a PAGEINFO in RBX, but PAGEINFO.SECINFO must be zeroed and EAUG instead sets magic hard-coded permissions. Why doesn't EAUG just respect the permissions in PAGEINFO.SECINFO? We aren't told.
This design is finished and respects the hardware specification. You can find the details in the SDM's documentation of the EAUG function.
If the SECINFO field has a value then the hardware requires it to indicate that it is a new shadow stack page being added, not a regular page. Support for shadow stack pages is not in scope for this work. Attempting to dynamically add a regular page with explicit permissions will result in a #GP(0).
The only way to add a regular enclave page is to make the SECINFO field empty and doing so forces the page type to be a regular page and the permissions to be RW.
Quoted 6 lines
Further, if the enclave can do EMODPE, why does
SGX_IOC_ENCLAVE_RELAX_PERMISSIONSeven exist? None of the documentation explains what this ioctl even does. Does it update PTE permissions? VMA permissions? Nobody knows without reading the source code.
Build the documentation (after applying this series) and it should contain all the information you are searching for. As is the current custom in the SGX documentation the built documentation pulls its content from the kernel doc of the functions that implement the core of the user space interactions.
Quoted 9 lines
Userspace should not be bothered with the subtle details of the interaction between EPC, PTE and VMA permissions. But this API does everything it can do to expose all these details to userspace. And it doesn't bother to document them (probably because it is hard). It would be much better to avoid exposing these details to userspace.
IMHO, there should be a simple flow like this (if EAUG respects PAGEINFO.SECINFO):
EAUG does not respect PAGEINFO.SECINFO for regular pages.
Quoted 18 lines
1. Non-enclave calls mmap()/munmap().
2. Enclave issues EACCEPT, if necessary.
3. Enclave issues EMODPE, if necessary.Notice that in the second step above, during the
mmap()call, the kernel ensures that EPC, PTE and VMA are in sync and fails if they cannot be made to be compatible. Also note that in the above flow EAUG instructions can be efficiently batched.Given the current poor state of the EAUG instruction, we might need to do this flow instead:
1. Enclave issues EACCEPT, if necessary. (Add RW pages...)
2. Non-enclave calls mmap()/munmap().
3. Enclave issues EACCEPT, if necessary.
4. Enclave issues EMODPE, if necessary.
Quoted 6 lines
However, doing EAUG only via the page access handler means that there is no way to batch EAUG instructions and this forces a context switch for every page you want to add. This has to be terrible for performance. Note specifically that the SDM calls out batching, which is currently impossible under this patch set. 35.5.7 - "Page allocation operations may be batched to improve efficiency."
These page functions are all per-page so it is not possible to add multiple pages with a single instruction. It is indeed possible to pre-fault pages.
Quoted 2 lines
As it stands today, if I want to add 256MiB of pages to an enclave, I'll have to do 2^16 context switches. That doesn't seem scalable.
No. Running ENCLU[EACCEPT] on each of the pages within that range should not need any explicit context switch out of the enclave. See the "augment_via_eaccept" test case in:
https://lore.kernel.org/linux-sgx/32c1116934a588bd3e6c174684e3e36a05c0a4d4.1644274683.git.reinette.chatre@intel.com/
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
the need for the weird userspace callback to allow for execute permissions.
User policy integration would always be required to allow execute permissions on a writable page. This is not expected to be a userspace callback but instead integration with existing user policy subsystem(s).
Why? This isn't documented.
This is similar to the existing policies involved in managing the permissions of mapped memory. When user space calls mprotect() to change permissions of a mapped region then the kernel will not blindly allow the permissions but instead ensure that it is allowed based on user policy by calling the LSM (Linux Security Module) hooks.
You can learn more about LSM and various security modules at:Documentation/security/lsm.rst Documentation/admin-guide/LSM/*
You can compare what is needed here to what is currently done when user space attempts to make some memory executable (see:
mm/mprotect.c:do_mprotect_key()->security_file_mprotect()). User policy needs to help the kernel determine if this is allowed. For example, when SELinux is the security module of choice then the process or file (depending on what type of memory is being changed) needs to have a special permission (PROCESS__EXECHEAP, PROCESS__EXECSTACK, or FILE__EXECMOD) assigned by user space to allow this.
Integration with user space policy is required for RWX of dynamically added pages to be supported. In this series dynamically added pages will not be allowed to be made executable, a follow-up series will add support for user policy integration to support RWX permissions of dynamically added pages.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
3. Implementing as I've suggested also means that we can lock down an enclave, for example - after code has been JITed, by closing the file descriptor. Once the file descriptor used to create the enclave is closed, no further
mmap()can be performed on the enclave. Attempting to do EACCEPT on an unmapped page will generate a page fault.
Quoted 7 lines
Quoted 3 lines
This is not clear to me. If the file descriptor is closed and no further
mmap()is allowed then how would a process be able to enter the enclave to execute code within it?EENTER (or the vdso function) with the address of a TCS page, like normal. In Enarx, we don't retain the enclave fd after the final
mmap()following EINIT. Everything works just fine.
The OS fault handler is responsible for managing the PTEs that is required for the enclave to be able to access the memory within the enclave. The OS fault handler is attached to a VMA that is created with mmap().
Quoted 8 lines
Quoted 5 lines
This series does indeed lock down the address range to ensure that it is not possible to map memory that does not belong to the enclave after the enclave is created. Please see:
https://lore.kernel.org/linux-sgx/1b833dbce6c937f71523f4aaf4b2181b9673519f.1644274683.git.reinette.chatre@intel.com/That's not what I'm talking about. I'm talking about a workflow like this:
Quoted 7 lines
1. Enclave initialization: ECREATE ... EINIT
2. EENTER
3. Enclave JITs some code (changes page permissions)
4. EEXIT
5. Close enclave fd.
6. EENTER
7. If an enclave attempts page modifications, a fault occurs.
The original fd that was created to obtain the enclave base address may be closed at (5) but the executable and data portions of the enclave still needs to be mapped afterwards to be able to have OS support for managing the PTEs that the enclave depends on to access those pages.
Quoted 5 lines
Think of this similar to
seccomp(). The enclave wants to do some dynamic page table manipulation. But then it wants to lock down page table modification so that, if compromised, attackers have no ability to obtain RWX permissions.
Reinette
Reinette,
Perhaps it would be better for us to have a shared understanding on how the patches as posted are supposed to work in the most common cases? I'm thinking here of projects such as Enarx, Gramine and Occulum, which all have a similar process. Namely they execute an executable (called exec in the below chart) which has things like syscalls handled by a shim. These two components (shim and exec) are supported by a non-enclave userspace runtime. Given this common architectural pattern, this is how I understand adding pages via an exec call to mmap() to work.
This only covers adding RW pages. I haven't even tackled permission changes yet. Is that understanding correct? If not, please provide an alternative sequence diagram to explain how you expect this to be used.
Quoted 307 lines · Reinette Chatre
Hi Nathaniel,
Quoted 28 lines · Nathaniel McCallum
Quoted 18 lines · Reinette Chatre
Hi Nathaniel,
Quoted 3 lines · Nathaniel McCallum
1. This interface looks very odd to me.
mmap()is the kernel interface for changing user space memory maps. Why are we introducing a new interface for this?
mmap()is the kernel interface used to create new mappings in the virtual address space of the calling process. This is different from the permissions and properties of the underlying file/memory being mapped.A new interface is introduced because changes need to be made to the permissions and properties of the underlying enclave. A new virtual address space is not needed nor should existing VMAs be impacted.
This is similar to how
mmap()is not used to change file permissions.Quoted 7 lines
VMA permissions are separate from enclave page permissions as found in the EPCM (Enclave Page Cache Map). The current implementation (SGX1) already distinguishes between the VMA and EPCM permissions - for example, it is already possible to create a read-only VMA from enclave pages that have RW EPCM permissions.
mmap()of a portion of EPC memory with a particular permission does not imply that the underlying EPCM permissions (should)have that permission.Quoted 1 line
Yes. BUT... unlike the file permissions, this leaks an implementation detail.
Not really - just like a RW file can be mapped read-only or RW, RW enclave memory can be mapped read-only or RW.
Quoted 6 lines
The user process is governed by VMA permissions. And during enclave creation, it had to
mmap()all the enclave regions to their final VMA permissions. So during enclave creation you have to usemmap()but after enclave creation you use custom APIs? That's inconsistent at best.No. ioctl()s are consistently used to manage enclave memory.
The existing
ioctls()SGX_IOC_ENCLAVE_CREATE,SGX_IOC_ENCLAVE_ADD_PAGES, andSGX_IOC_ENCLAVE_INITare used to set up to initialize the enclave memory.The new
ioctls()are used to manage enclave memory after enclave initialization.The enclave memory is thus managed with a consistent interface.
mmap()is required beforeSGX_IOC_ENCLAVE_CREATEto obtain a base address for the enclave that is required by theioctl(). The rest of the ioctl()s, existing and new, are consistent in interface by not requiring a memory mapping but instead work from an offset from the base address.Quoted 3 lines
Forcing userspace to worry about the (mostly undocumented!) interactions between EPC, PTE and VMA permissions makes these APIs hard to use and difficult to reason about.
This is not new. The current SGX1 user space is already prevented from creating a mapping of enclave memory that is more relaxed than the enclave memory. For example, if the enclave memory has RW EPCM permissions then it is not possible to
mmap()that memory as RWX.Quoted 3 lines
When I call
SGX_IOC_ENCLAVE_RELAX_PERMISSIONS, do I also have to callmmap()to update the VMA permissions to match? It isn't clear. Nor is
mprotect()may be the better call to use.Quoted 17 lines
it really clear why I'm calling completely separate APIs.
Quoted 15 lines
Quoted 15 lines
You can just simply add a new mmap flag (i.e.
MAP_SGX_TCS*) and then figure out which SGX instructions to execute based on the desired state of the memory maps. If you do this, none of the following ioctls are needed:*
SGX_IOC_ENCLAVE_RELAX_PERMISSIONS
*SGX_IOC_ENCLAVE_RESTRICT_PERMISSIONS
*SGX_IOC_ENCLAVE_REMOVE_PAGES
*SGX_IOC_ENCLAVE_MODIFY_TYPEIt also means that languages don't have to grow support for all these ioctls. Instead, they can just reuse the existing
mmap()bindings with the new flag. Also, multiple operations can be combined into a singlemmap()call, amortizing the changes over a single context switch.Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
2. Automatically adding pages with hard-coded permissions in a fault handler seems like a really bad idea.
Could you please elaborate why this is a bad idea?
Because implementations that miss this subtlety suddenly have pages with magic permissions. Magic is bad. Explicit is good.
There is no magic. Any new pages have to be accepted by the enclave. The enclave will not be able to access these pages unless explicitly accepted, ENCLU[EACCEPT], from within the enclave.
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
How do you distinguish between
accesses which should result in an updated mapping and accesses that should result in a fault?Quoted 8 lines
Quoted 4 lines
Accesses that should result in an updated mapping have two requirements:
(a) address accessed belongs to the enclave based on the address range specified during enclave create
(b) there is no backing enclave page for the addressWhat happens if the enclave is buggy? Or has been compromised. In both of those cases, there should be a userspace visible fault and pages should not be added.
If user space accesses a memory address with a regular read/write that results in a new page added then there is indeed a user space visible fault. You can see this flow in action in the "augment" test case in
https://lore.kernel.org/linux-sgx/32c1116934a588bd3e6c174684e3e36a05c0a4d4.1644274683.git.reinette.chatre@intel.com/If user space indeed wants the page after encountering such a fault then it needs to enter the enclave again, from a different entry point, to run ENCLU[EACCEPT], before it can return to the original entry point to continue execution from the instruction that triggered the original read/write.
The only flow where a page is added without a user space visible fault is when user space explicitly runs the ENCLU[EACCEPT] to do so.
Quoted 21 lines
Quoted 10 lines
Quoted 7 lines
IMHO, all unmapped page accesses should
result in a page fault.mmap()should be called first to identify the correct permissions for these pages.
Then the page handler should be
updated to use the permissions from the mapping when backfilling physical pages. If I understand correctly, this should also obviateRegular enclave pages can _only_ be dynamically added with RW permission.
Quoted 10 lines
SGX2's support for adding regular pages to an enclave via the EAUG instruction is architecturally set at RW. The OS cannot change those permissions via the EAUG instruction nor can the OS do so with a different/additional instruction because:
* the OS is not able to relax permissions since that can only be done from within the enclave with ENCLU[EMODPE], thus it is not possible for the OS to dynamically add pages via EAUG as RW and then relax permissions to RWX.
* the OS is not able to EAUG a page and immediately attempt an EMODPR either as Jarkko also recently inquired about:
https://lore.kernel.org/linux-sgx/[email protected]/Quoted 4 lines
This design looks... unfinished. EAUG takes a PAGEINFO in RBX, but PAGEINFO.SECINFO must be zeroed and EAUG instead sets magic hard-coded permissions. Why doesn't EAUG just respect the permissions in PAGEINFO.SECINFO? We aren't told.
This design is finished and respects the hardware specification. You can find the details in the SDM's documentation of the EAUG function.
If the SECINFO field has a value then the hardware requires it to indicate that it is a new shadow stack page being added, not a regular page. Support for shadow stack pages is not in scope for this work. Attempting to dynamically add a regular page with explicit permissions will result in a #GP(0).
The only way to add a regular enclave page is to make the SECINFO field empty and doing so forces the page type to be a regular page and the permissions to be RW.
Quoted 6 lines
Further, if the enclave can do EMODPE, why does
SGX_IOC_ENCLAVE_RELAX_PERMISSIONSeven exist? None of the documentation explains what this ioctl even does. Does it update PTE permissions? VMA permissions? Nobody knows without reading the source code.Build the documentation (after applying this series) and it should contain all the information you are searching for. As is the current custom in the SGX documentation the built documentation pulls its content from the kernel doc of the functions that implement the core of the user space interactions.
Quoted 9 lines
Userspace should not be bothered with the subtle details of the interaction between EPC, PTE and VMA permissions. But this API does everything it can do to expose all these details to userspace. And it doesn't bother to document them (probably because it is hard). It would be much better to avoid exposing these details to userspace.
IMHO, there should be a simple flow like this (if EAUG respects PAGEINFO.SECINFO):
EAUG does not respect PAGEINFO.SECINFO for regular pages.
Quoted 18 lines
1. Non-enclave calls mmap()/munmap().
2. Enclave issues EACCEPT, if necessary.
3. Enclave issues EMODPE, if necessary.Notice that in the second step above, during the
mmap()call, the kernel ensures that EPC, PTE and VMA are in sync and fails if they cannot be made to be compatible. Also note that in the above flow EAUG instructions can be efficiently batched.Given the current poor state of the EAUG instruction, we might need to do this flow instead:
1. Enclave issues EACCEPT, if necessary. (Add RW pages...)
2. Non-enclave calls mmap()/munmap().
3. Enclave issues EACCEPT, if necessary.
4. Enclave issues EMODPE, if necessary.Quoted 6 lines
However, doing EAUG only via the page access handler means that there is no way to batch EAUG instructions and this forces a context switch for every page you want to add. This has to be terrible for performance. Note specifically that the SDM calls out batching, which is currently impossible under this patch set. 35.5.7 - "Page allocation operations may be batched to improve efficiency."
These page functions are all per-page so it is not possible to add multiple pages with a single instruction. It is indeed possible to pre-fault pages.
Quoted 2 lines
As it stands today, if I want to add 256MiB of pages to an enclave, I'll have to do 2^16 context switches. That doesn't seem scalable.
No. Running ENCLU[EACCEPT] on each of the pages within that range should not need any explicit context switch out of the enclave. See the "
augment_via_eaccept" test case in:
https://lore.kernel.org/linux-sgx/32c1116934a588bd3e6c174684e3e36a05c0a4d4.1644274683.git.reinette.chatre@intel.com/Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
the need for the weird userspace callback to allow for execute permissions.
User policy integration would always be required to allow execute permissions on a writable page. This is not expected to be a userspace callback but instead integration with existing user policy subsystem(s).
Why? This isn't documented.
This is similar to the existing policies involved in managing the permissions of mapped memory. When user space calls
mprotect()to change permissions of a mapped region then the kernel will not blindly allow the permissions but instead ensure that it is allowed based on user policy by calling the LSM (Linux Security Module) hooks.You can learn more about LSM and various security modules at:
Documentation/security/lsm.rstDocumentation/admin-guide/LSM/*You can compare what is needed here to what is currently done when user space attempts to make some memory executable (see:
mm/mprotect.c:do_mprotect_key()->security_file_mprotect()). User policy needs to help the kernel determine if this is allowed. For example, when SELinux is the security module of choice then the process or file (depending on what type of memory is being changed) needs to have a special permission (PROCESS__EXECHEAP,PROCESS__EXECSTACK, orFILE__EXECMOD) assigned by user space to allow this.Integration with user space policy is required for RWX of dynamically added pages to be supported. In this series dynamically added pages will not be allowed to be made executable, a follow-up series will add support for user policy integration to support RWX permissions of dynamically added pages.
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
3. Implementing as I've suggested also means that we can lock down an enclave, for example - after code has been JITed, by closing the file descriptor. Once the file descriptor used to create the enclave is closed, no further
mmap()can be performed on the enclave. Attempting to do EACCEPT on an unmapped page will generate a page fault.Quoted 7 lines
Quoted 3 lines
This is not clear to me. If the file descriptor is closed and no further
mmap()is allowed then how would a process be able to enter the enclave to execute code within it?EENTER (or the vdso function) with the address of a TCS page, like normal. In Enarx, we don't retain the enclave fd after the final
mmap()following EINIT. Everything works just fine.The OS fault handler is responsible for managing the PTEs that is required for the enclave to be able to access the memory within the enclave. The OS fault handler is attached to a VMA that is created with
mmap().Quoted 8 lines
Quoted 5 lines
This series does indeed lock down the address range to ensure that it is not possible to map memory that does not belong to the enclave after the enclave is created. Please see:
https://lore.kernel.org/linux-sgx/1b833dbce6c937f71523f4aaf4b2181b9673519f.1644274683.git.reinette.chatre@intel.com/That's not what I'm talking about. I'm talking about a workflow like this:
Quoted 7 lines
1. Enclave initialization: ECREATE ... EINIT
2. EENTER
3. Enclave JITs some code (changes page permissions)
4. EEXIT
5. Close enclave fd.
6. EENTER
7. If an enclave attempts page modifications, a fault occurs.The original fd that was created to obtain the enclave base address may be closed at (5) but the executable and data portions of the enclave still needs to be mapped afterwards to be able to have OS support for managing the PTEs that the enclave depends on to access those pages.
Quoted 5 lines
Think of this similar to
seccomp(). The enclave wants to do some dynamic page table manipulation. But then it wants to lock down page table modification so that, if compromised, attackers have no ability to obtain RWX permissions.Reinette
Hi Nathaniel,
Quoted 10 lines · Nathaniel McCallum
Perhaps it would be better for us to have a shared understanding on how the patches as posted are supposed to work in the most common cases? I'm thinking here of projects such as Enarx, Gramine and Occulum, which all have a similar process. Namely they execute an executable (called exec in the below chart) which has things like syscalls handled by a shim. These two components (shim and exec) are supported by a non-enclave userspace runtime. Given this common architectural pattern, this is how I understand adding pages via an exec call to
mmap()to work.
Quoted 6 lines
This only covers adding RW pages. I haven't even tackled permission changes yet. Is that understanding correct? If not, please provide an alternative sequence diagram to explain how you expect this to be used.
Please find my attempt linked below:
The changes include:
* Move mmap() to occur before attempting EACCEPT on the addresses. This is required for EACCEPT (as well as any subsequent access from within the enclave) to be able to access the pages.
* Remove AEX[1] to the runtime within the loop. After EAUG returns execution will return to the instruction pointer that triggered the #PF, EACCEPT, this will cause the EACCEPT to be run again, this time succeeding.
This is based on the implementation within this series. When supporting the new ioctl() requested by Jarkko there will be an additional ioctl() required before the loop.
Reinette
Quoted 33 lines · Reinette Chatre
Hi Nathaniel,
Quoted 10 lines · Nathaniel McCallum
Perhaps it would be better for us to have a shared understanding on how the patches as posted are supposed to work in the most common cases? I'm thinking here of projects such as Enarx, Gramine and Occulum, which all have a similar process. Namely they execute an executable (called exec in the below chart) which has things like syscalls handled by a shim. These two components (shim and exec) are supported by a non-enclave userspace runtime. Given this common architectural pattern, this is how I understand adding pages via an exec call to
mmap()to work.Quoted 6 lines
This only covers adding RW pages. I haven't even tackled permission changes yet. Is that understanding correct? If not, please provide an alternative sequence diagram to explain how you expect this to be used.
Please find my attempt linked below:
The changes include:
* Movemmap()to occur before attempting EACCEPT on the addresses. This is required for EACCEPT (as well as any subsequent access from within the enclave) to be able to access the pages.
* Remove AEX[1] to the runtime within the loop. After EAUG returns execution will return to the instruction pointer that triggered the #PF, EACCEPT, this will cause the EACCEPT to be run again, this time succeeding.
Quoted 3 lines
This is based on the implementation within this series. When supporting the new
ioctl()requested by Jarkko there will be an additionalioctl()required before the loop.
My comments below correspond to the arrow numbers in the diagram.
2. When the runtime receives the AEX, it doesn't have enough knowledge to know whether or not to ask the kernel for an mmap(). So it has to reenter the shim.
3. The shim has to handle the syscall instruction routing it to the enclave's memory management subsystem.
4. The shim has to do bookkeeping and decide if additional pages are even needed. If pages are already allocated, for example, it can skip directly to step 13. However, if modifications are needed, it will go to steps 5-12.
5-12. This is the part that represents new code from the kernel's perspective for SGX2. It is also in a performance critical path and should be evaluated with greater scrutiny. The number of context switches is O(2N + 4) for each new allocated block, where N is the number of pages: a context switch occurs at step 5, 6, 7, 8, 9/10 and
12. However, this can be reduced to O(4) for each new allocated block with a simple modification:
The interesting thing about this pattern is that this can be done for all page modification types except EMODT. For example, here's the same process for changing a mapping from RW to RX:
My point in this thread has always been that it is an anti-feature to presume that there is a need to treat EPC and VLA permissions separately. This is a performance sink and it optimizes for a use case which doesn't exist. Nobody actually wants there to be a mismatch between EPC and VLA permissions.
So, besides EMODT, the only userspace interface we need is mmap()/mprotect()/munmap(). The kernel should either succeed the mmap()/mprotect()/munmap() syscall if the EPC permissions can be made compatible or should fail otherwise.
Another interesting property arises from this flow. Since the EPC and VLA permissions are always synchronized from the perspective of userspace, in cases where the memory state between the kernel and the exec layer is roughly synchronous, bookkeeping in the shim can be implemented without any persistent memory between syscall handling events. So, for example, the shim can implement brk() and mmap()/munmap()/mprotect() with just two pointers: one to the break position and one to the lowest mmap().
It is true that this basically commits enclave authors to doing all EACCEPT calls immediately after modifications. But I suspect everyone will do this anyway since there is no efficient (read: performant) way for shims to handle page faults. So trying to do this lazily will just result in a huge decrease in performance.
Hi Nathaniel,
Quoted 63 lines · Nathaniel McCallum
Quoted 33 lines · Reinette Chatre
Hi Nathaniel,
Quoted 10 lines · Nathaniel McCallum
Perhaps it would be better for us to have a shared understanding on how the patches as posted are supposed to work in the most common cases? I'm thinking here of projects such as Enarx, Gramine and Occulum, which all have a similar process. Namely they execute an executable (called exec in the below chart) which has things like syscalls handled by a shim. These two components (shim and exec) are supported by a non-enclave userspace runtime. Given this common architectural pattern, this is how I understand adding pages via an exec call to
mmap()to work.Quoted 6 lines
This only covers adding RW pages. I haven't even tackled permission changes yet. Is that understanding correct? If not, please provide an alternative sequence diagram to explain how you expect this to be used.
Please find my attempt linked below:
The changes include:
* Movemmap()to occur before attempting EACCEPT on the addresses. This is required for EACCEPT (as well as any subsequent access from within the enclave) to be able to access the pages.
* Remove AEX[1] to the runtime within the loop. After EAUG returns execution will return to the instruction pointer that triggered the #PF, EACCEPT, this will cause the EACCEPT to be run again, this time succeeding.Quoted 3 lines
This is based on the implementation within this series. When supporting the new
ioctl()requested by Jarkko there will be an additionalioctl()required before the loop.My comments below correspond to the arrow numbers in the diagram.
2. When the runtime receives the AEX, it doesn't have enough knowledge to know whether or not to ask the kernel for an
mmap(). So it has to reenter the shim.3. The shim has to handle the syscall instruction routing it to the enclave's memory management subsystem.
4. The shim has to do bookkeeping and decide if additional pages are even needed. If pages are already allocated, for example, it can skip directly to step 13. However, if modifications are needed, it will go to steps 5-12.
5-12. This is the part that represents new code from the kernel's perspective for SGX2. It is also in a performance critical path and should be evaluated with greater scrutiny. The number of context switches is O(2N + 4) for each new allocated block, where N is the number of pages: a context switch occurs at step 5, 6, 7, 8, 9/10 and
12. However, this can be reduced to O(4) for each new allocated block with a simple modification:
Your optimized proposal is possible in the current implementation as follows:
You can think of that EACCEPT instruction similar to a current (SGX1) enclave memory read or write when the enclave page is not currently in the EPC, for example, if the enclave memory being accessed is swapped out and need to be decrypted and loaded back. Instead of ENCLS[ELDU] incorporated to load the enclave page back into EPC, ENCLS[EAUG] is incorporated to create a new EPC page.
You can find an example of such a flow involving EACCEPT in the "augment_via_eaccept" test found in "[PATCH V2 21/32] selftests/sgx: Test two different SGX2 EAUG flows"
Quoted 11 lines
The interesting thing about this pattern is that this can be done for all page modification types except EMODT. For example, here's the same process for changing a mapping from RW to RX:
My point in this thread has always been that it is an anti-feature to presume that there is a need to treat EPC and VLA permissions separately. This is a performance sink and it optimizes for a use case which doesn't exist. Nobody actually wants there to be a mismatch between EPC and VLA permissions.
I assume you mean VMA permissions. It is hard for me to trust the statement that nobody wants there to be a mismatch since VMA permissions being separate from EPC permissions is an intentional (as documented) and integral part of the current SGX ABI. Current SGX implementation explicitly checks for and supports VMA mappings with permissions different from EPC permissions.
This SGX2 implementation follows and respects the current ABI and changing ABI cannot be taken lightly.
Reinette
Jarkko Sakkinen
maintainerQuoted 69 lines · Nathaniel McCallum
Quoted 33 lines · Reinette Chatre
Hi Nathaniel,
Quoted 10 lines · Nathaniel McCallum
Perhaps it would be better for us to have a shared understanding on how the patches as posted are supposed to work in the most common cases? I'm thinking here of projects such as Enarx, Gramine and Occulum, which all have a similar process. Namely they execute an executable (called exec in the below chart) which has things like syscalls handled by a shim. These two components (shim and exec) are supported by a non-enclave userspace runtime. Given this common architectural pattern, this is how I understand adding pages via an exec call to
mmap()to work.Quoted 6 lines
This only covers adding RW pages. I haven't even tackled permission changes yet. Is that understanding correct? If not, please provide an alternative sequence diagram to explain how you expect this to be used.
Please find my attempt linked below:
The changes include:
* Movemmap()to occur before attempting EACCEPT on the addresses. This is required for EACCEPT (as well as any subsequent access from within the enclave) to be able to access the pages.
* Remove AEX[1] to the runtime within the loop. After EAUG returns execution will return to the instruction pointer that triggered the #PF, EACCEPT, this will cause the EACCEPT to be run again, this time succeeding.Quoted 3 lines
This is based on the implementation within this series. When supporting the new
ioctl()requested by Jarkko there will be an additionalioctl()required before the loop.My comments below correspond to the arrow numbers in the diagram.
2. When the runtime receives the AEX, it doesn't have enough knowledge to know whether or not to ask the kernel for an
mmap(). So it has to reenter the shim.3. The shim has to handle the syscall instruction routing it to the enclave's memory management subsystem.
4. The shim has to do bookkeeping and decide if additional pages are even needed. If pages are already allocated, for example, it can skip directly to step 13. However, if modifications are needed, it will go to steps 5-12.
5-12. This is the part that represents new code from the kernel's perspective for SGX2. It is also in a performance critical path and should be evaluated with greater scrutiny. The number of context switches is O(2N + 4) for each new allocated block, where N is the number of pages: a context switch occurs at step 5, 6, 7, 8, 9/10 and
12. However, this can be reduced to O(4) for each new allocated block with a simple modification:The interesting thing about this pattern is that this can be done for all page modification types except EMODT. For example, here's the same process for changing a mapping from RW to RX:
Quoted 7 lines
My point in this thread has always been that it is an anti-feature to presume that there is a need to treat EPC and VLA permissions separately. This is a performance sink and it optimizes for a use case which doesn't exist. Nobody actually wants there to be a mismatch between EPC and VLA permissions.
I would not touch pre-initialization, EADD'd pages. The reason is backwards compatibility.
For post-initialization, options are still open.
Quoted 4 lines
So, besides EMODT, the only userspace interface we need is mmap()/mprotect()/munmap(). The kernel should either succeed the mmap()/mprotect()/munmap() syscall if the EPC permissions can be made compatible or should fail otherwise.
For mmap() it is the enclave who sets the permissions, not kernel, i.e. you get a half-broken mmap() implementation. Kernel does EAUG, enclave does EACCEPTCOPY.
I think what you're asking is too simple to be true, and even if we could do it, it might limit possibilities to optimize user space, e.g. because there is two ENCLU leaf functions (EMODPE, EACCEPTCOPY) and one ENCLS leaf function (EMODPR), which can modify permissions.
A kernel syscall is essentially something that can be fully serviced by the kernel. This is not such situation. The work is split.
BR, Jarkko