Intel SGX foundations
Subsystems (11)
v38 cover Intel SGX foundations
Intel(R) SGX is a set of CPU instructions that can be used by applications to set aside private regions of code and data. The code outside the enclave is disallowed to access the memory inside the enclave by the CPU access control.
There is a new hardware unit in the processor called Memory Encryption Engine (MEE) starting from the Skylake microacrhitecture. BIOS can define one or many MEE regions that can hold enclave data by configuring them with PRMRR registers.
The MEE automatically encrypts the data leaving the processor package to the MEE regions. The data is encrypted using a random key whose life-time is exactly one power cycle.
The current implementation requires that the firmware sets IA32_SGXLEPUBKEYHASH* MSRs as writable so that ultimately the kernel can decide what enclaves it wants run. The implementation does not create any bottlenecks to support read-only MSRs later on.
You can tell if your CPU supports SGX by looking into /proc/cpuinfo:
cat /proc/cpuinfo | grep sgx
v38:
* Fast iteration because I had email server issues with v37. Using temporary SMTP for this (mail.kapsi.fi).
* Return -EINTR when no data is processed, just like read().
https://patchwork.kernel.org/patch/11773941/
* Remove cruft from SGX_IOC_ENCLAVE_ADD_PAGES and fix the return value.
https://lore.kernel.org/linux-sgx/[email protected]/T/#mc643ef2ab477f5f7aa5d463d883d1308eb44d6f1
v37:
* Remove MODULE_*().
https://lore.kernel.org/linux-sgx/[email protected]/
* Fix mmap() type check.
https://lore.kernel.org/linux-sgx/20200827152051.GB22351@sjchrist-ice/
* Fix ioctl-number.txt.
https://lore.kernel.org/linux-sgx/[email protected]/
* Fix SPDX identifier in arch/x86/include/uapi/asm/sgx.h
https://lore.kernel.org/linux-sgx/[email protected]/
* Consistently document "@encl: an enclave pointer".
https://lore.kernel.org/linux-sgx/[email protected]/
* Renamed SGX_IOC_ENCLAVE_SET_ATTRIBUTE as SGX_IOC_ENCLAVE_PROVISION and
cleaned up associated code. Also fixed issues of code introduced too
early that were caused by the split. Now it should be more streamlined.
https://lore.kernel.org/linux-sgx/[email protected]/
* Fixed signed integer shift overflow:
https://github.com/intel/linux-sgx-driver/pull/16/commits d27ca6071b2b28e2e789d265eda796dd9fc65a64
* Fixed off-by-one error in a size calculation:
https://github.com/intel/SGXDataCenterAttestationPrimitives/commit/e44cc238becf584cc079aef40b557c6af9a03f38
* Rework vDSO API with a context structure for IO data.
https://lore.kernel.org/linux-sgx/[email protected]/
* Refined commentary about retry-loop with ENCLS[EINIT] based on Sean's response.
https://lore.kernel.org/linux-sgx/[email protected]/
* Return positive number from SGX_IOC_ENCLAVE_ADD_PAGES indicating the
number of pages processed and set a fixed 1MB limit for length. In addition,
on interrupt, return 0 and number of processed pages instead of -EINTR.
https://lore.kernel.org/linux-sgx/[email protected]/
v36:
* Make a self-contained mprotect() handler.
* Move from radix_tree to xarray, which is more modern and robust data structure for managing large sparse arrays. Rename encl->page_tree as encl->page_array.
v35:
* Add missing SGX_ENCL_DEAD check to sgx_ioctl().
v34:
* Remove SGX_ENCL_DEAD checks from ioctl.c, as the file is open while executing an ioctl.
* Split driver patch into base patch and one patch for each ioctl.
* Move encl->flags check to the beginning of each ioctl. Return consistently -EINVAL if they don't match the expected values. Before, sometimes -EFAULT was returned instead of -EINVAL.
* Rename vm_ops->may_mprotect as vm_ops->mprotect() and move the call right before call to mprotect_fixup().
v33:
* Rebased to v5.8-rc1. Given the core changes (mmap_lock and #DB handling), it made sense to update the series.
* Refined documentation about how remote attestation is done in SGX.
v32:
* v31 contained not fully cleaned up main.c after merger of main.c and reclaim.c. Fixed in this version.
* Rebased to v5.7. Forgot to do this for v31.
v31:
* Unset SGX_ENCL_IOCTL in the error path of checking encl->flags in order to prevent leaving it set and thus block any further ioctl calls.
* Added missing cleanup_srcu_struct() call to sgx_encl_release().
* Take encl->lock in sgx_encl_add_page() in order to prevent races with the page reclaimer.
* Fix a use-after-free bug from the page reclaimer. Call kref_put() for encl->refcount only after putting enclave page back to the active page list because it could be the last ref to the enclave.
* Filter any CPU disallowed values from sigstruct->vendor SGX_IOC_ENCLAVE_INIT.
* Use bits 0-7 of page descriptor for the EPC section index. This should be enough for long term needs.
* Refined naming for functions that allocate and free EPC pages to be more sound and consistent.
* Merge main.c and reclaim.c into one.
v30:
Bunch of tags added. No actual code changes.
v29:
* The selftest has been moved to selftests/sgx. Because SGX is an execution environment of its own, it really isn't a great fit with more "standard" x86 tests.
The RSA key is now generated on fly and the whole signing process has been made as part of the enclave loader instead of signing the enclave during the compilation time.
Finally, the enclave loader loads now the test enclave directly from its ELF file, which means that ELF file does not need to be coverted as raw binary during the build process.
* Version the mm_list instead of using on synchronize_mm() when adding new entries. We hold the write lock for the mm_struct, and dup_mm() can thus deadlock with the page reclaimer, which could hold the lock for the old mm_struct.
* Disallow mmap(PROT_NONE) from /dev/sgx. Any mapping (e.g. anonymous) can be used to reserve the address range. Now /dev/sgx supports only opaque mappings to the (initialized) enclave data.
* Make the vDSO callable directly from C by preserving RBX and taking leaf from RCX.
v28:
* Documented to Documentation/x86/sgx.rst how the kernel manages the enclave ownership.
* Removed non-LC flow from sgx_einit().
* Removed struct sgx_einittoken since only the size of the corresponding microarchitectural structure is used in the series ATM.
v27:
* Disallow RIE processes to use enclaves as there could a permission conflict between VMA and enclave permissions.
* In the documentation, replace "grep /proc/cpuinfo" with "grep sgx /proc/cpuinfo".
v26:
* Fixed the commit author in "x86/sgx: Linux Enclave Driver", which was changed in v25 by mistake.
* Addressed a bunch of grammar mistakes in sgx.rst (thanks Randy once again for such a detailed feedback).
* Added back the MAINTAINERS update commit, which was mistakenly removed in v25.
* EREMOVE's for SECS cannot be done while sanitizing an EPC section. The CPU does not allow to remove a SECS page before all of its children have been removed, and a child page can be in some other section than the one currently being processed. Thus, removed special SECS processing from sgx_sanitize_page() and instead put sections through it twice. In the 2nd round the lists should only contain SECS pages.
v25:
* Fix a double-free issue when SGX_IOC_ENCLAVE_ADD_PAGES
fails on executing ENCLS[EADD]. The rollback path executed radix_tree_delete() on the same address twice when this happened.
* Return -EINTR instead of -ERESTARTSYS in SGX_IOC_ENCLAVE_ADD_PAGES when a signal is pending.
* As requested by Borislav, move the CPUID 0x12 features to their own word in cpufeatures.
* Sean fixed a bug from sgx_reclaimer_write() where sgx_encl_put_backing() was called with an uninitialized pointer when sgx_encl_get_backing() fails.
* Migrated /dev/sgx/* to misc. This is future-proof as struct miscdevice has 'groups' for setting up sysfs attributes for the device.
* Use device_initcall instead of subsys_initcall so that misc_class is initialized before SGX is initialized.
* Return -EACCES in SGX_IOC_ENCLAVE_INIT when caller tries to select enclave attributes that we the kernel does not allow it to set instead of -EINVAL.
* Unless SGX public key MSRs are writable always deny the feature from Linux. Previously this was only denied from driver. How VMs should be supported is not really part of initial patch set, which makes this an obvious choice.
* Cleaned up and refined documentation to be more approachable.
v24:
* Reclaim unmeasured and TCS pages (regression in v23).
* Replace usages of GFP_HIGHUSER with GFP_KERNEL.
* Return -EIO on when EADD or EEXTEND fails in %SGX_IOC_ENCLAVE_ADD_PAGES and use the same rollback (destroy enclave). This can happen when host suspends itself unknowingly to a VM running enclaves. From -EIO the user space can deduce what happened.
* Have a separate @count in struct sgx_enclave_add_pages to output number of bytes processed instead of overwriting the input parameters for clarity and more importantly that the API provides means for partial processing (@count could be less than @length in success case).
v23:
* Replace SGX_ENCLAVE_ADD_PAGE with SGX_ENCLAVE_ADD_PAGES. Replace @mrmask with %SGX_PAGE_MEASURE flag.
* Return -EIO instead of -ECANCELED when ptrace() fails to read a TCS page.
* In the reclaimer, pin page before ENCLS[EBLOCK] because pinning can fail (because of OOM) even in legit behaviour and after EBLOCK the reclaiming flow can be only reverted by killing the whole enclave.
* Fixed SGX_ATTR_RESERVED_MASK. Bit 7 was marked as reserved while in fact it should have been bit 6 (Table 37-3 in the SDM).
* Return -EPERM from SGX_IOC_ENCLAVE_INIT when ENCLS[EINIT] returns an SGX error code.
v22:
* Refined bunch commit messages and added associated SDM references as many of them were too exhausting and some outdated.
* Alignment checks have been removed from mmap() because it does not define the ELRANGE. VMAs only act as windows to the enclave. The semantics compare somewhat how mmap() works with regular files.
* We now require user space addresses given to SGX_IOC_ENCLAVE_ADD_PAGE to be page aligned so that we can pass the page directly to EADD and do not have to do an extra copy. This was made effectively possible by removing the worker thread for adding pages.
* The selftest build files have been refined throughout of various glitches and work properly in a cross compilation environment such as BuildRoot. In addition, libcalls fail the build with an assertion in the linker script, if they end up to the enclave binary.
* CONFIG_INTEL_SGX_DRIVER has been removed because you cannot use SGX core for anything without having the driver. This could change when KVM support is added.
* We require zero permissions in SECINFO for TCS pages because the CPU overwrites SECINFO flags with zero permissions and measures the page only after that. Allowing to pass TCS with non-zero permissions would cause mismatching measurement between the one provided in SIGSTRUCT and the one computed by the CPU.
* Obviously lots of small fixes and clean ups (does make sense to document them all).
v21:
* Check on mmap() that the VMA does cover an area that does not have enclave pages. Only mapping with PROT_NONE can do that to reserve initial address space for an enclave.
* Check om mmap() and mprotect() that the VMA permissions do not surpass the enclave permissions.
* Remove two refcounts from vma_close(): mm_list and encl->refcount. Enclave refcount is only need for swapper/enclave sync and we can remove mm_list refcount by destroying mm_struct when the process is closed. By not having vm_close() the Linux MM can merge VMAs.
* Do not naturally align MAP_FIXED address.
* Numerous small fixes and clean ups.
* Use SRCU for synchronizing the list of mm_struct's.
* Move to stack based call convention in the vDSO.
v20:
* Fine-tune Kconfig messages and spacing and remove MMU_NOTIFIER dependency as MMU notifiers are no longer used in the driver.
* Use mm_users instead of mm_count as refcount for mm_struct as mm_count only protects from deleting mm_struct, not removing its contents.
* Sanitize EPC when the reclaimer thread starts by doing EREMOVE for all of them. They could be in initialized state when the kernel starts because it might be spawned by kexec().
* Documentation overhaul.
* Use a device /dev/sgx/provision for delivering the provision token instead of securityfs.
* Create a reference to the enclave when already when opening /dev/sgx/enclave. The file is then associated with this enclave only. mmap() can be done at free at any point and always get a reference to the enclave. To summarize the file now represents the enclave.
v19:
* Took 3-4 months but in some sense this was more like a rewrite of most of the corners of the source code. If I've forgotten to deal with some feedback, please don't shout me. Make a remark and I will fix it for the next version. Hopefully there won't be this big turnovers anymore.
* Validate SECS attributes properly against CPUID given attributes and against allowed attributes. SECS attributes are the ones that are enforced whereas SIGSTRUCT attributes tell what is required to run the enclave.
* Add KSS (Key Sharing Support) to the enclave attributes.
* Deny MAP_PRIVATE as an enclave is always a shared memory entity.
* Revert back to shmem backing storage so that it can be easily shared by multiple processes.
* Split the recognization of an ENCLS leaf failure by using three functions to detect it: encsl_faulted(), encls_returned_code() and sgx_failed(). encls_failed() is only caused by a spurious expections that should never happen. Thus, it is not defined as an inline function in order to easily insert a kprobe to it.
* Move low-level enclave management routines, page fault handler and page reclaiming routines from driver to the core. These cannot be separated from each other as they are heavily interdependent. The rationale is that the core does not call any code from the driver.
* Allow the driver to be compiled as a module now that it no code is using its routines and it only uses exported symbols. Now the driver is essentially just a thin ioctl layer.
* Reworked the driver to maintain a list of mm_struct's. The VMA callbacks add new entries to this list as the process is forked. Each entry has its own refcount because they have a different life-cycle as the enclave does. In effect @tgid and @mm have been removed from struct sgx_encl and we allow forking by removing VM_DONTCOPY from vm flags.
* Generate a cpu mask in the reclaimer from the cpu mask's of all mm_struct's. This will kick out the hardware threads out of the enclave from multiple processes. It is not a local variable because it would eat too much of the stack space but instead a field in struct sgx_encl.
* Allow forking i.e. remove VM_DONTCOPY. I did not change the API because the old API scaled to the workload that Andy described. The codebase is now mostly API independent i.e. changing the API is a small task. For me the proper trigger to chanage it is a as concrete as possible workload that cannot be fulfilled. I hope you understand my thinking here. I don't want to change anything w/o proper basis but I'm ready to change anything if there is a proper basis. I do not have any kind of attachment to any particular type of API.
* Add Sean's vDSO ENCLS(EENTER) patches and update selftest to use the new vDSO.
v18:
* Update the ioctl-number.txt.
* Move the driver under arch/x86.
* Add SGX features (SGX, SGX1, SGX2) to the disabled-features.h.
* Rename the selftest as test_sgx (previously sgx-selftest).
* In order to enable process accounting, swap EPC pages and PCMD's to a VMA instead of shmem.
* Allow only to initialize and run enclaves with a subset of {DEBUG, MODE64BIT} set.
* Add SGX_IOC_ENCLAVE_SET_ATTRIBUTE to allow an enclave to have privileged attributes e.g. PROVISIONKEY.
v17:
* Add a simple selftest.
* Fix a null pointer dereference to section->pages when its allocation fails.
* Add Sean's description of the exception handling to the documentation.
v16:
* Fixed SOB's in the commits that were a bit corrupted in v15.
* Implemented exceptio handling properly to detect_sgx().
* Use GENMASK() to define SGX_CPUID_SUB_LEAF_TYPE_MASK.
* Updated the documentation to use rst definition lists.
* Added the missing Documentation/x86/index.rst, which has a link to intel_sgx.rst. Now the SGX and uapi documentation is properly generated with 'make htmldocs'.
* While enumerating EPC sections, if an undefined section is found, fail the driver initialization instead of continuing the initialization.
* Issue a warning if there are more than %SGX_MAX_EPC_SECTIONS.
* Remove copyright notice from arch/x86/include/asm/sgx.h.
* Migrated from ioremap_cache() to memremap().
v15:
* Split into more digestable size patches.
* Lots of small fixes and clean ups.
* Signal a "plain" SIGSEGV on an EPCM violation.
v14:
* Change the comment about X86_FEATURE_SGX_LC from “SGX launch configuration” to “SGX launch control”.
* Move the SGX-related CPU feature flags as part of the Linux defined virtual leaf 8.
* Add SGX_ prefix to the constants defining the ENCLS leaf functions.
* Use GENMASK*() and BIT*() in sgx_arch.h instead of raw hex numbers.
* Refine the long description for CONFIG_INTEL_SGX_CORE.
* Do not use pr_*_ratelimited() in the driver. The use of the rate limited versions is legacy cruft from the prototyping phase.
* Detect sleep with SGX_INVALID_EINIT_TOKEN instead of counting power cycles.
* Manually prefix with “sgx:” in the core SGX code instead of redefining pr_fmt.
* Report if IA32_SGXLEPUBKEYHASHx MSRs are not writable in the driver instead of core because it is a driver requirement.
* Change prompt to bool in the entry for CONFIG_INTEL_SGX_CORE because the default is ‘n’.
* Rename struct sgx_epc_bank as struct sgx_epc_section in order to match the SDM.
* Allocate struct sgx_epc_page instances one at a time.
* Use “__iomem void *” pointers for the mapped EPC memory consistently.
* Retry once on SGX_INVALID_TOKEN in sgx_einit() instead of counting power cycles.
* Call enclave swapping operations directly from the driver instead of calling them .indirectly through struct sgx_epc_page_ops because indirect calls are not required yet as the patch set does not contain the KVM support.
* Added special signal SEGV_SGXERR to notify about SGX EPCM violation errors.
v13:
* Always use SGX_CPUID constant instead of a hardcoded value.
* Simplified and documented the macros and functions for ENCLS leaves.
* Enable sgx_free_page() to free active enclave pages on demand in order to allow sgx_invalidate() to delete enclave pages. It no longer performs EREMOVE if a page is in the process of being reclaimed.
* Use PM notifier per enclave so that we don't have to traverse the global list of active EPC pages to find enclaves.
* Removed unused SGX_LE_ROLLBACK constant from uapi/asm/sgx.h
* Always use ioremap() to map EPC banks as we only support 64-bit kernel.
* Invalidate IA32_SGXLEPUBKEYHASH cache used by sgx_einit() when going to sleep.
v12:
* Split to more narrow scoped commits in order to ease the review process and use co-developed-by tag for co-authors of commits instead of listing them in the source files.
* Removed cruft EXPORT_SYMBOL() declarations and converted to static variables.
* Removed in-kernel LE i.e. this version of the SGX software stack only supports unlocked IA32_SGXLEPUBKEYHASHx MSRs.
* Refined documentation on launching enclaves, swapping and enclave construction.
* Refined sgx_arch.h to include alignment information for every struct that requires it and removed structs that are not needed without an LE.
* Got rid of SGX_CPUID.
* SGX detection now prints log messages about firmware configuration issues.
v11:
* Polished ENCLS wrappers with refined exception handling.
* ksgxswapd was not stopped (regression in v5) in sgx_page_cache_teardown(), which causes a leaked kthread after driver deinitialization.
* Shutdown sgx_le_proxy when going to suspend because its EPC pages will be invalidated when resuming, which will cause it not function properly anymore.
* Set EINITTOKEN.VALID to zero for a token that is passed when SGXLEPUBKEYHASH matches MRSIGNER as alloc_page() does not give a zero page.
* Fixed the check in sgx_edbgrd() for a TCS page. Allowed to read offsets around the flags field, which causes a #GP. Only flags read is readable.
* On read access memcpy() call inside sgx_vma_access() had src and dest parameters in wrong order.
* The build issue with CONFIG_KASAN is now fixed. Added undefined symbols to LE even if “KASAN_SANITIZE := false” was set in the makefile.
* Fixed a regression in the #PF handler. If a page has SGX_ENCL_PAGE_RESERVED flag the #PF handler should unconditionally fail. It did not, which caused weird races when trying to change other parts of swapping code.
* EPC management has been refactored to a flat LRU cache and moved to arch/x86. The swapper thread reads a cluster of EPC pages and swaps all of them. It can now swap from multiple enclaves in the same round.
* For the sake of consistency with SGX_IOC_ENCLAVE_ADD_PAGE, return -EINVAL when an enclave is already initialized or dead instead of zero.
v10:
* Cleaned up anon inode based IPC between the ring-0 and ring-3 parts of the driver.
* Unset the reserved flag from an enclave page if EDBGRD/WR fails (regression in v6).
* Close the anon inode when LE is stopped (regression in v9).
* Update the documentation with a more detailed description of SGX.
v9:
* Replaced kernel-LE IPC based on pipes with an anonymous inode. The driver does not require anymore new exports.
v8:
* Check that public key MSRs match the LE public key hash in the driver initialization when the MSRs are read-only.
* Fix the race in VA slot allocation by checking the fullness immediately after succeesful allocation.
* Fix the race in hash mrsigner calculation between the launch enclave and user enclaves by having a separate lock for hash calculation.
v7:
* Fixed offset calculation in sgx_edbgr/wr(). Address was masked with PAGE_MASK when it should have been masked with ~PAGE_MASK.
* Fixed a memory leak in sgx_ioc_enclave_create().
* Simplified swapping code by using a pointer array for a cluster instead of a linked list.
* Squeezed struct sgx_encl_page to 32 bytes.
* Fixed deferencing of an RSA key on OpenSSL 1.1.0.
* Modified TC's CMAC to use kernel AES-NI. Restructured the code
a bit in order to better align with kernel conventions.
v6:
* Fixed semaphore underrun when accessing /dev/sgx from the launch enclave.
* In sgx_encl_create() s/IS_ERR(secs)/IS_ERR(encl)/.
* Removed virtualization chapter from the documentation.
* Changed the default filename for the signing key as signing_key.pem.
* Reworked EPC management in a way that instead of a linked list of struct sgx_epc_page instances there is an array of integers that encodes address and bank of an EPC page (the same data as 'pa' field earlier). The locking has been moved to the EPC bank level instead of a global lock.
* Relaxed locking requirements for EPC management. EPC pages can be released back to the EPC bank concurrently.
* Cleaned up ptrace() code.
* Refined commit messages for new architectural constants.
* Sorted includes in every source file.
* Sorted local variable declarations according to the line length in every function.
* Style fixes based on Darren's comments to sgx_le.c.
v5:
* Described IPC between the Launch Enclave and kernel in the commit messages.
* Fixed all relevant checkpatch.pl issues that I have forgot fix in earlier versions except those that exist in the imported TinyCrypt code.
* Fixed spelling mistakes in the documentation.
* Forgot to check the return value of sgx_drv_subsys_init().
* Encapsulated properly page cache init and teardown.
* Collect epc pages to a temp list in sgx_add_epc_bank
* Removed SGX_ENCLAVE_INIT_ARCH constant.
v4:
* Tied life-cycle of the sgx_le_proxy process to /dev/sgx.
* Removed __exit annotation from sgx_drv_subsys_exit().
* Fixed a leak of a backing page in sgx_process_add_page_req() in the case when vm_insert_pfn() fails.
* Removed unused symbol exports for sgx_page_cache.c.
* Updated sgx_alloc_page() to require encl parameter and documented the behavior (Sean Christopherson).
* Refactored a more lean API for sgx_encl_find() and documented the behavior.
* Moved #PF handler to sgx_fault.c.
* Replaced subsys_system_register() with plain bus_register().
* Retry EINIT 2nd time only if MSRs are not locked.
v3:
* Check that FEATURE_CONTROL_LOCKED and FEATURE_CONTROL_SGX_ENABLE are set.
* Return -ERESTARTSYS in __sgx_encl_add_page() when sgx_alloc_page() fails.
* Use unused bits in epc_page->pa to store the bank number.
* Removed #ifdef for WQ_NONREENTRANT.
* If mmu_notifier_register() fails with -EINTR, return -ERESTARTSYS.
* Added --remove-section=.got.plt to objcopy flags in order to prevent a dummy .got.plt, which will cause an inconsistent size for the LE.
* Documented sgx_encl_* functions.
* Added remark about AES implementation used inside the LE.
* Removed redundant sgx_sys_exit() from le/main.c.
* Fixed struct sgx_secinfo alignment from 128 to 64 bytes.
* Validate miscselect in sgx_encl_create().
* Fixed SSA frame size calculation to take the misc region into account.
* Implemented consistent exception handling to __encls() and __encls_ret().
* Implemented a proper device model in order to allow sysfs attributes and in-kernel API.
* Cleaned up various "find enclave" implementations to the unified sgx_encl_find().
* Validate that vm_pgoff is zero.
* Discard backing pages with shmem_truncate_range() after EADD.
* Added missing EEXTEND operations to LE signing and launch.
* Fixed SSA size for GPRS region from 168 to 184 bytes.
* Fixed the checks for TCS flags. Now DBGOPTIN is allowed.
* Check that TCS addresses are in ELRANGE and not just page aligned.
* Require kernel to be compiled with X64_64 and CPU_SUP_INTEL.
* Fixed an incorrect value for SGX_ATTR_DEBUG from 0x01 to 0x02.
v2:
* get_rand_uint32() changed the value of the pointer instead of value where it is pointing at.
* Launch enclave incorrectly used sigstruct attributes-field instead of enclave attributes-field.
* Removed unused struct sgx_add_page_req from sgx_ioctl.c
* Removed unused sgx_has_sgx2.
* Updated arch/x86/include/asm/sgx.h so that it provides stub implementations when sgx in not enabled.
* Removed cruft rdmsr-calls from sgx_set_pubkeyhash_msrs().
* return -ENOMEM in sgx_alloc_page() when VA pages consume too much space
* removed unused global sgx_nr_pids
* moved sgx_encl_release to sgx_encl.c
* return -ERESTARTSYS instead of -EINTR in sgx_encl_init()
Jarkko Sakkinen (14):
x86/sgx: Add SGX microarchitectural data structures x86/sgx: Add wrappers for ENCLS leaf functions x86/cpu/intel: Add nosgx kernel parameter
x86/sgx: Add __sgx_alloc_epc_page() and sgx_free_epc_page() x86/sgx: Add SGX enclave driver
x86/sgx: Add SGX_IOC_ENCLAVE_CREATE
x86/sgx: Add SGX_IOC_ENCLAVE_ADD_PAGES
x86/sgx: Add SGX_IOC_ENCLAVE_INIT
x86/sgx: Enable provisioning for remote attestation x86/sgx: Add a page reclaimer
x86/sgx: ptrace() support for the SGX driver selftests/x86: Add a selftest for SGX
docs: x86/sgx: Document SGX micro architecture and kernel internals x86/sgx: Update MAINTAINERS
Sean Christopherson (10):
x86/cpufeatures: x86/msr: Add Intel SGX hardware bits x86/cpufeatures: x86/msr: Add Intel SGX Launch Control hardware bits x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX x86/cpu/intel: Detect SGX support
x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections
mm: Add vm_ops->mprotect()
x86/vdso: Add support for exception fixup in vDSO functions x86/fault: Add helper function to sanitize error code x86/traps: Attempt to fixup exceptions in vDSO before signaling x86/vdso: Implement a vDSO for Intel SGX enclave call
.../admin-guide/kernel-parameters.txt | 2 + .../userspace-api/ioctl/ioctl-number.rst | 1 + Documentation/x86/index.rst | 1 + Documentation/x86/sgx.rst | 200 +++++ MAINTAINERS | 11 + arch/x86/Kconfig | 17 + arch/x86/entry/vdso/Makefile | 8 +- arch/x86/entry/vdso/extable.c | 46 + arch/x86/entry/vdso/extable.h | 29 + arch/x86/entry/vdso/vdso-layout.lds.S | 9 +- arch/x86/entry/vdso/vdso.lds.S | 1 + arch/x86/entry/vdso/vdso2c.h | 50 +- arch/x86/entry/vdso/vsgx_enter_enclave.S | 157 ++++ arch/x86/include/asm/cpufeature.h | 5 +- arch/x86/include/asm/cpufeatures.h | 8 +- arch/x86/include/asm/disabled-features.h | 18 +- arch/x86/include/asm/enclu.h | 8 + arch/x86/include/asm/msr-index.h | 8 + arch/x86/include/asm/required-features.h | 2 +- arch/x86/include/asm/traps.h | 14 +- arch/x86/include/asm/vdso.h | 5 + arch/x86/include/uapi/asm/sgx.h | 203 +++++ arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/common.c | 4 + arch/x86/kernel/cpu/feat_ctl.c | 41 +- arch/x86/kernel/cpu/sgx/Makefile | 5 + arch/x86/kernel/cpu/sgx/arch.h | 341 ++++++++ arch/x86/kernel/cpu/sgx/driver.c | 204 +++++ arch/x86/kernel/cpu/sgx/driver.h | 32 + arch/x86/kernel/cpu/sgx/encl.c | 754 ++++++++++++++++ arch/x86/kernel/cpu/sgx/encl.h | 128 +++ arch/x86/kernel/cpu/sgx/encls.h | 238 +++++ arch/x86/kernel/cpu/sgx/ioctl.c | 823 ++++++++++++++++++ arch/x86/kernel/cpu/sgx/main.c | 753 ++++++++++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 65 ++ arch/x86/kernel/traps.c | 19 +- arch/x86/mm/fault.c | 45 +- include/linux/mm.h | 3 + mm/mprotect.c | 5 +- tools/testing/selftests/Makefile | 1 + tools/testing/selftests/sgx/.gitignore | 2 + tools/testing/selftests/sgx/Makefile | 53 ++ tools/testing/selftests/sgx/call.S | 44 + tools/testing/selftests/sgx/defines.h | 21 + tools/testing/selftests/sgx/load.c | 277 ++++++ tools/testing/selftests/sgx/main.c | 232 +++++ tools/testing/selftests/sgx/main.h | 38 + tools/testing/selftests/sgx/sigstruct.c | 395 +++++++++ tools/testing/selftests/sgx/test_encl.c | 20 + tools/testing/selftests/sgx/test_encl.lds | 40 + .../selftests/sgx/test_encl_bootstrap.S | 89 ++ 51 files changed, 5446 insertions(+), 30 deletions(-) create mode 100644 Documentation/x86/sgx.rst create mode 100644 arch/x86/entry/vdso/extable.c create mode 100644 arch/x86/entry/vdso/extable.h create mode 100644 arch/x86/entry/vdso/vsgx_enter_enclave.S create mode 100644 arch/x86/include/asm/enclu.h create mode 100644 arch/x86/include/uapi/asm/sgx.h create mode 100644 arch/x86/kernel/cpu/sgx/Makefile create mode 100644 arch/x86/kernel/cpu/sgx/arch.h create mode 100644 arch/x86/kernel/cpu/sgx/driver.c create mode 100644 arch/x86/kernel/cpu/sgx/driver.h create mode 100644 arch/x86/kernel/cpu/sgx/encl.c create mode 100644 arch/x86/kernel/cpu/sgx/encl.h create mode 100644 arch/x86/kernel/cpu/sgx/encls.h create mode 100644 arch/x86/kernel/cpu/sgx/ioctl.c create mode 100644 arch/x86/kernel/cpu/sgx/main.c create mode 100644 arch/x86/kernel/cpu/sgx/sgx.h create mode 100644 tools/testing/selftests/sgx/.gitignore create mode 100644 tools/testing/selftests/sgx/Makefile create mode 100644 tools/testing/selftests/sgx/call.S create mode 100644 tools/testing/selftests/sgx/defines.h create mode 100644 tools/testing/selftests/sgx/load.c create mode 100644 tools/testing/selftests/sgx/main.c create mode 100644 tools/testing/selftests/sgx/main.h create mode 100644 tools/testing/selftests/sgx/sigstruct.c create mode 100644 tools/testing/selftests/sgx/test_encl.c create mode 100644 tools/testing/selftests/sgx/test_encl.lds create mode 100644 tools/testing/selftests/sgx/test_encl_bootstrap.S
Signature · 1 line
2.25.1
PATCH v38 1/24 x86/cpufeatures: x86/msr: Add Intel SGX hardware bits
From: Sean Christopherson <[email protected]>
Add X86_FEATURE_SGX from CPUID.(EAX=7, ECX=1), which informs whether the CPU has SGX.
Add X86_FEATURE_SGX1 and X86_FEATURE_SGX2 from CPUID.(EAX=12H, ECX=0), which describe the level of SGX support available [1].
Add IA32_FEATURE_CONTROL.SGX_ENABLE. BIOS can use this bit to opt-in SGX before locking the feature control MSR [2].
[1] Intel SDM: 36.7.2 Intel® SGX Resource Enumeration Leaves
[2] Intel SDM: 36.7.1 Intel® SGX Opt-In Configuration
--- arch/x86/include/asm/cpufeature.h | 5 +++-- arch/x86/include/asm/cpufeatures.h | 7 ++++++- arch/x86/include/asm/disabled-features.h | 18 +++++++++++++++--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/include/asm/required-features.h | 2 +- arch/x86/kernel/cpu/common.c | 4 ++++ 6 files changed, 30 insertions(+), 7 deletions(-)
arch/x86/include/asm/cpufeature.h +3 -2
@@ -30,6 +30,7 @@ enum cpuid_leafsCPUID_7_ECX ,CPUID_8000_0007_EBX ,CPUID_7_EDX , + CPUID_12_EAX, };#ifdef CONFIG_X86_FEATURE_NAMES
@@ -89,7 +90,7 @@ extern const char * const x86_bug_flags[NBUGINTS*32]; CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 17, feature_bit) || \ CHECK_BIT_IN_MASK_WORD(REQUIRED_MASK, 18, feature_bit) || \ REQUIRED_MASK_CHECK || \ - BUILD_BUG_ON_ZERO(NCAPINTS != 19)) + BUILD_BUG_ON_ZERO(NCAPINTS != 20))#define DISABLED_MASK_BIT_SET (feature_bit) \ ( CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 0, feature_bit) || \
@@ -112,7 +113,7 @@ extern const char * const x86_bug_flags[NBUGINTS*32]; CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 17, feature_bit) || \ CHECK_BIT_IN_MASK_WORD(DISABLED_MASK, 18, feature_bit) || \ DISABLED_MASK_CHECK || \ - BUILD_BUG_ON_ZERO(NCAPINTS != 19)) + BUILD_BUG_ON_ZERO(NCAPINTS != 20))#define cpu_has (c, bit) \ (__builtin_constant_p(bit) && REQUIRED_MASK_BIT_SET(bit) ? 1 : \
arch/x86/include/asm/cpufeatures.h +6 -1
@@ -13,7 +13,7 @@/* * Defines x86 CPU feature bits */ -#define NCAPINTS 19/* N 32-bit words worth of info */ +#define NCAPINTS 20/* N 32-bit words worth of info */ #define NBUGINTS 1/* N 32-bit bug flags */ /*
@@ -240,6 +240,7 @@/* Intel-defined CPU features, CPUID level 0x00000007:0 (EBX), word 9 */ #define X86_FEATURE_FSGSBASE ( 9*32+ 0)/* RDFSBASE, WRFSBASE, RDGSBASE, WRGSBASE instructions*/ #define X86_FEATURE_TSC_ADJUST ( 9*32+ 1)/* TSC adjustment MSR 0x3B */ +#define X86_FEATURE_SGX ( 9*32+ 2)/* Software Guard Extensions */ #define X86_FEATURE_BMI1 ( 9*32+ 3)/* 1st group bit manipulation extensions */ #define X86_FEATURE_HLE ( 9*32+ 4)/* Hardware Lock Elision */ #define X86_FEATURE_AVX2 ( 9*32+ 5)/* AVX2 instructions */
@@ -377,6 +378,10 @@#define X86_FEATURE_CORE_CAPABILITIES (18*32+30)/* "" IA32_CORE_CAPABILITIES MSR */ #define X86_FEATURE_SPEC_CTRL_SSBD (18*32+31)/* "" Speculative Store Bypass Disable */ +/* Intel-defined SGX features, CPUID level 0x00000012:0 (EAX), word 19 */ +#define X86_FEATURE_SGX1 (19*32+ 0)/* SGX1 leaf functions */ +#define X86_FEATURE_SGX2 (19*32+ 1)/* SGX2 leaf functions */ +/* * BUG word(s) */
arch/x86/include/asm/disabled-features.h +15 -3
@@ -28,13 +28,18 @@# define DISABLE_CYRIX_ARR (1<<(X86_FEATURE_CYRIX_ARR & 31))# define DISABLE_CENTAUR_MCR (1<<(X86_FEATURE_CENTAUR_MCR & 31))# define DISABLE_PCID 0 +# define DISABLE_SGX1 0 +# define DISABLE_SGX2 0#else # define DISABLE_VME 0# define DISABLE_K6_MTRR 0# define DISABLE_CYRIX_ARR 0# define DISABLE_CENTAUR_MCR 0# define DISABLE_PCID (1<<(X86_FEATURE_PCID & 31)) -#endif /* CONFIG_X86_64 */ +# define DISABLE_SGX1 (1<<(X86_FEATURE_SGX1 & 31)) +# define DISABLE_SGX2 (1<<(X86_FEATURE_SGX2 & 31)) +#endif /* CONFIG_X86_64 */ +#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS # define DISABLE_PKU 0
@@ -56,6 +61,12 @@# define DISABLE_PTI (1 << (X86_FEATURE_PTI & 31))#endif +#ifdef CONFIG_INTEL_SGX +# define DISABLE_SGX 0 +#else +# define DISABLE_SGX (1 << (X86_FEATURE_SGX & 31)) +#endif +/* * Make sure to add features to the correct mask */
@@ -68,7 +79,7 @@#define DISABLED_MASK6 0#define DISABLED_MASK7 (DISABLE_PTI)#define DISABLED_MASK8 0 -#define DISABLED_MASK9 (DISABLE_SMAP) +#define DISABLED_MASK9 (DISABLE_SMAP|DISABLE_SGX)#define DISABLED_MASK10 0#define DISABLED_MASK11 0#define DISABLED_MASK12 0
@@ -78,6 +89,7 @@#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE|DISABLE_LA57|DISABLE_UMIP)#define DISABLED_MASK17 0#define DISABLED_MASK18 0 -#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) +#define DISABLED_MASK19 (DISABLE_SGX1|DISABLE_SGX2) +#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)#endif /* _ASM_X86_DISABLED_FEATURES_H */
arch/x86/include/asm/msr-index.h +1 -0
@@ -602,6 +602,7 @@#define FEAT_CTL_LOCKED BIT(0)#define FEAT_CTL_VMX_ENABLED_INSIDE_SMX BIT(1)#define FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX BIT(2) +#define FEAT_CTL_SGX_ENABLED BIT(18)#define FEAT_CTL_LMCE_ENABLED BIT(20)#define MSR_IA32_TSC_ADJUST 0x0000003b
arch/x86/include/asm/required-features.h +1 -1
@@ -101,6 +101,6 @@#define REQUIRED_MASK16 0#define REQUIRED_MASK17 0#define REQUIRED_MASK18 0 -#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19) +#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 20)#endif /* _ASM_X86_REQUIRED_FEATURES_H */
arch/x86/kernel/cpu/common.c +4 -0
@@ -931,6 +931,10 @@ void get_cpu_cap(struct cpuinfo_x86 *c) c->x86_capability[CPUID_D_1_EAX ] = eax; } +/* Additional Intel-defined SGX flags: level 0x00000012 */ +if (c->cpuid_level >=0x00000012 ) + c->x86_capability[CPUID_12_EAX ] =cpuid_eax (0x00000012 ); +/* AMD-defined flags: level 0x80000001 */ eax =cpuid_eax (0x80000000 ); c->extended_cpuid_level = eax;
Signature · 1 line
2.25.1
PATCH v38 2/24 x86/cpufeatures: x86/msr: Add Intel SGX Launch Control hardware bits
In reply to Jarkko SakkinenFrom: Sean Christopherson <[email protected]>
Add X86_FEATURE_SGX_LC, which informs whether or not the CPU supports SGX Launch Control.
Add MSR_IA32_SGXLEPUBKEYHASH{0, 1, 2, 3}, which when combined contain a SHA256 hash of a 3072-bit RSA public key. SGX backed software packages, so called enclaves, are always signed. All enclaves signed with the public key are unconditionally allowed to initialize. [1]
Add FEAT_CTL_SGX_LC_ENABLED, which informs whether the aformentioned MSRs are writable or not. If the bit is off, the public key MSRs are read-only for the OS.
If the MSRs are read-only, the platform must provide a launch enclave (LE). LE can create cryptographic tokens for other enclaves that they can pass together with their signature to the ENCLS(EINIT) opcode, which is used to initialize enclaves.
Linux is unlikely to support the locked configuration because it takes away the control of the launch decisions from the kernel.
[1] Intel SDM: 38.1.4 Intel SGX Launch Control Configuration
--- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/msr-index.h | 7 +++++++ 2 files changed, 8 insertions(+)
arch/x86/include/asm/cpufeatures.h +1 -0
@@ -354,6 +354,7 @@#define X86_FEATURE_CLDEMOTE (16*32+25)/* CLDEMOTE instruction */ #define X86_FEATURE_MOVDIRI (16*32+27)/* MOVDIRI instruction */ #define X86_FEATURE_MOVDIR64B (16*32+28)/* MOVDIR64B instruction */ +#define X86_FEATURE_SGX_LC (16*32+30)/* Software Guard Extensions Launch Control */ /* AMD-defined CPU features, CPUID level 0x80000007 (EBX), word 17 */ #define X86_FEATURE_OVERFLOW_RECOV (17*32+ 0)/* MCA overflow recovery support */
arch/x86/include/asm/msr-index.h +7 -0
@@ -602,6 +602,7 @@#define FEAT_CTL_LOCKED BIT(0)#define FEAT_CTL_VMX_ENABLED_INSIDE_SMX BIT(1)#define FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX BIT(2) +#define FEAT_CTL_SGX_LC_ENABLED BIT(17)#define FEAT_CTL_SGX_ENABLED BIT(18)#define FEAT_CTL_LMCE_ENABLED BIT(20)
@@ -622,6 +623,12 @@#define MSR_IA32_UCODE_WRITE 0x00000079#define MSR_IA32_UCODE_REV 0x0000008b +/* Intel SGX Launch Enclave Public Key Hash MSRs */ +#define MSR_IA32_SGXLEPUBKEYHASH0 0x0000008C +#define MSR_IA32_SGXLEPUBKEYHASH1 0x0000008D +#define MSR_IA32_SGXLEPUBKEYHASH2 0x0000008E +#define MSR_IA32_SGXLEPUBKEYHASH3 0x0000008F +#define MSR_IA32_SMM_MONITOR_CTL 0x0000009b#define MSR_IA32_SMBASE 0x0000009e
Signature · 1 line
2.25.1
PATCH v38 3/24 x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX
In reply to Jarkko SakkinenFrom: Sean Christopherson <[email protected]>
Include SGX bit to the PF error codes and throw SIGSEGV with PF_SGX when a #PF with SGX set happens.
CPU throws a #PF with the SGX set in the event of Enclave Page Cache Map (EPCM) conflict. The EPCM is a CPU-internal table, which describes the properties for a enclave page. Enclaves are measured and signed software entities, which SGX hosts. [1]
Although the primary purpose of the EPCM conflict checks is to prevent malicious accesses to an enclave, an illegit access can happen also for legit reasons.
All SGX reserved memory, including EPCM is encrypted with a transient key that does not survive from the power transition. Throwing a SIGSEGV allows user space software to react when this happens (e.g. recreate the enclave, which was invalidated).
[1] Intel SDM: 36.5.1 Enclave Page Cache Map (EPCM)
--- arch/x86/include/asm/traps.h | 14 ++++++++------ arch/x86/mm/fault.c | 13 +++++++++++++ 2 files changed, 21 insertions(+), 6 deletions(-)
arch/x86/include/asm/traps.h +8 -6
@@ -44,12 +44,13 @@ void __noreturn handle_stack_overflow(const char *message,/* * Page fault error code bits: * -* bit 0 == 0: no page found 1: protection fault -* bit 1 == 0: read access 1: write access -* bit 2 == 0: kernel-mode access 1: user-mode access -* bit 3 == 1: use of reserved bit detected -* bit 4 == 1: fault was an instruction fetch -* bit 5 == 1: protection keys block access +* bit 0 == 0: no page found 1: protection fault +* bit 1 == 0: read access 1: write access +* bit 2 == 0: kernel-mode access 1: user-mode access +* bit 3 == 1: use of reserved bit detected +* bit 4 == 1: fault was an instruction fetch +* bit 5 == 1: protection keys block access +* bit 15 == 1: inside SGX enclave */ enum x86_pf_error_code {X86_PF_PROT =1 <<0 ,
@@ -58,5 +59,6 @@ enum x86_pf_error_code {X86_PF_RSVD =1 <<3 ,X86_PF_INSTR =1 <<4 ,X86_PF_PK =1 <<5 , + X86_PF_SGX = 1 << 15, };#endif /* _ASM_X86_TRAPS_H */
arch/x86/mm/fault.c +13 -0
@@ -1054,6 +1054,19 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)if (error_code &X86_PF_PK )return 1 ; +/* +* Access is blocked by the Enclave Page Cache Map (EPCM), i.e. the +* access is allowed by the PTE but not the EPCM. This usually happens +* when the EPCM is yanked out from under us, e.g. by hardware after a +* suspend/resume cycle. In any case, software, i.e. the kernel, can't +* fix the source of the fault as the EPCM can't be directly modified by +* software. Handle the fault as an access error in order to signal +* userspace so that userspace can rebuild their enclave(s), even though +* userspace may not have actually violated access permissions. +*/ +if (unlikely (error_code &X86_PF_SGX )) +return 1 ; +/* * Make sure to check the VMA so that we do not perform * faults just to hit a X86_PF_PK as soon as we fill in a
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 03/24] x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX
Quoted 21 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
Include SGX bit to the PF error codes and throw SIGSEGV with
PF_SGXwhen a #PF with SGX set happens.CPU throws a #PF with the SGX set in the event of Enclave Page Cache Map (EPCM) conflict. The EPCM is a CPU-internal table, which describes the properties for a enclave page. Enclaves are measured and signed software entities, which SGX hosts. [1]
Although the primary purpose of the EPCM conflict checks is to prevent malicious accesses to an enclave, an illegit access can happen also for legit reasons.
All SGX reserved memory, including EPCM is encrypted with a transient key that does not survive from the power transition. Throwing a SIGSEGV allows user space software to react when this happens (e.g. recreate the enclave, which was invalidated).
[1] Intel SDM: 36.5.1 Enclave Page Cache Map (EPCM)
Quoted 7 lines
Acked-by Jethro Beekman <[email protected]>Reviewed-by Darren Kenny <[email protected]>Reviewed-by Borislav Petkov <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>--- arch/x86/include/asm/traps.h | 14 ++++++++------
The SEV-ES patchset moved that to arch/x86/include/asm/trap_pf.h but fixing that up is real easy. Simply do this search-replace
s!arch/x86/include/asm/traps.h!arch/x86/include/asm/trap_pf.h!
on this patch.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 03/24] x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX
Quoted 30 lines · Borislav Petkov
Quoted 21 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
Include SGX bit to the PF error codes and throw SIGSEGV with
PF_SGXwhen a #PF with SGX set happens.CPU throws a #PF with the SGX set in the event of Enclave Page Cache Map (EPCM) conflict. The EPCM is a CPU-internal table, which describes the properties for a enclave page. Enclaves are measured and signed software entities, which SGX hosts. [1]
Although the primary purpose of the EPCM conflict checks is to prevent malicious accesses to an enclave, an illegit access can happen also for legit reasons.
All SGX reserved memory, including EPCM is encrypted with a transient key that does not survive from the power transition. Throwing a SIGSEGV allows user space software to react when this happens (e.g. recreate the enclave, which was invalidated).
[1] Intel SDM: 36.5.1 Enclave Page Cache Map (EPCM)
Quoted 7 lines
Acked-by Jethro Beekman <[email protected]>Reviewed-by Darren Kenny <[email protected]>Reviewed-by Borislav Petkov <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>--- arch/x86/include/asm/traps.h | 14 ++++++++------
Quoted 6 lines
The SEV-ES patchset moved that to
arch/x86/include/asm/trap_pf.hbut fixing that up is real easy. Simply do this search-replaces!arch/x86/include/asm/traps.h!arch/x86/include/asm/trap_pf.h!
on this patch.
Hey, I just did
git fetch mainline && git rebase mainline/master
Zero conflicts and there is no trap_pf.h yet. So this is not yet in Linus' tree?
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 03/24] x86/mm: x86/sgx: Signal SIGSEGV with PF_SGX
Quoted 2 lines · Jarkko Sakkinen
Zero conflicts and there is no
trap_pf.hyet. So this is not yet in Linus' tree?
tip tree: https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/log/?h=x86/seves
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v38 4/24 x86/sgx: Add SGX microarchitectural data structures
In reply to Jarkko SakkinenDefine the SGX microarchitectural data structures used by various SGX opcodes. This is not an exhaustive representation of all SGX data structures but only those needed by the kernel.
The data structures are described in:
Intel SDM: 37.6 INTEL® SGX DATA STRUCTURES OVERVIEW
--- arch/x86/kernel/cpu/sgx/arch.h | 341 +++++++++++++++++++++++++++++++++ 1 file changed, 341 insertions(+) create mode 100644 arch/x86/kernel/cpu/sgx/arch.h
arch/x86/kernel/cpu/sgx/arch.h new file +341 -0
@@ -0,0 +1,341 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */ +/** +* Copyright(c) 2016-18 Intel Corporation. +* +* Contains data structures defined by the SGX architecture. Data structures +* defined by the Linux software stack should not be placed here. +*/ +#ifndef _ASM_X86_SGX_ARCH_H +#define _ASM_X86_SGX_ARCH_H + +#include <linux/bits.h> +#include <linux/types.h> + +#define SGX_CPUID 0x12 +#define SGX_CPUID_FIRST_VARIABLE_SUB_LEAF 2 + +/** +* enum sgx_return_code - The return code type for ENCLS, ENCLU and ENCLV +* %SGX_NOT_TRACKED: Previous ETRACK's shootdown sequence has not +* been completed yet. +* %SGX_INVALID_EINITTOKEN: EINITTOKEN is invalid and enclave signer's +* public key does not match IA32_SGXLEPUBKEYHASH. +* %SGX_UNMASKED_EVENT: An unmasked event, e.g. INTR, was received +*/ +enum sgx_return_code { +SGX_NOT_TRACKED =11 , +SGX_INVALID_EINITTOKEN =16 , +SGX_UNMASKED_EVENT =128 , +}; + +/** +* enum sgx_sub_leaf_types - SGX CPUID variable sub-leaf types +* %SGX_CPUID_SUB_LEAF_INVALID: Indicates this sub-leaf is invalid. +* %SGX_CPUID_SUB_LEAF_EPC_SECTION: Sub-leaf enumerates an EPC section. +*/ +enum sgx_sub_leaf_types { +SGX_CPUID_SUB_LEAF_INVALID =0x0 , +SGX_CPUID_SUB_LEAF_EPC_SECTION =0x1 , +}; + +#define SGX_CPUID_SUB_LEAF_TYPE_MASK GENMASK(3, 0) + +#define SGX_MODULUS_SIZE 384 + +/** +* enum sgx_miscselect - additional information to an SSA frame +* %SGX_MISC_EXINFO: Report #PF or #GP to the SSA frame. +* +* Save State Area (SSA) is a stack inside the enclave used to store processor +* state when an exception or interrupt occurs. This enum defines additional +* information stored to an SSA frame. +*/ +enum sgx_miscselect { +SGX_MISC_EXINFO =BIT (0 ), +}; + +#define SGX_MISC_RESERVED_MASK GENMASK_ULL(63, 1) + +#define SGX_SSA_GPRS_SIZE 184 +#define SGX_SSA_MISC_EXINFO_SIZE 16 + +/** +* enum sgx_attributes - the attributes field in &struct sgx_secs +* %SGX_ATTR_INIT: Enclave can be entered (is initialized). +* %SGX_ATTR_DEBUG: Allow ENCLS(EDBGRD) and ENCLS(EDBGWR). +* %SGX_ATTR_MODE64BIT: Tell that this a 64-bit enclave. +* %SGX_ATTR_PROVISIONKEY: Allow to use provisioning keys for remote +* attestation. +* %SGX_ATTR_KSS: Allow to use key separation and sharing (KSS). +* %SGX_ATTR_EINITTOKENKEY: Allow to use token signing key that is used to +* sign cryptographic tokens that can be passed to +* EINIT as an authorization to run an enclave. +*/ +enum sgx_attribute { +SGX_ATTR_INIT =BIT (0 ), +SGX_ATTR_DEBUG =BIT (1 ), +SGX_ATTR_MODE64BIT =BIT (2 ), +SGX_ATTR_PROVISIONKEY =BIT (4 ), +SGX_ATTR_EINITTOKENKEY =BIT (5 ), +SGX_ATTR_KSS =BIT (7 ), +}; + +#define SGX_ATTR_RESERVED_MASK (BIT_ULL(3) | BIT_ULL(6) | GENMASK_ULL(63, 8)) + +/** +* struct sgx_secs - SGX Enclave Control Structure (SECS) +* @size: size of the address space +* @base: base address of the address space +* @ssa_frame_size: size of an SSA frame +* @miscselect: additional information stored to an SSA frame +* @attributes: attributes for enclave +* @xfrm: XSave-Feature Request Mask (subset of XCR0) +* @mrenclave: SHA256-hash of the enclave contents +* @mrsigner: SHA256-hash of the public key used to sign the SIGSTRUCT +* @config_id: a user-defined value that is used in key derivation +* @isv_prod_id: a user-defined value that is used in key derivation +* @isv_svn: a user-defined value that is used in key derivation +* @config_svn: a user-defined value that is used in key derivation +* +* SGX Enclave Control Structure (SECS) is a special enclave page that is not +* visible in the address space. In fact, this structure defines the address +* range and other global attributes for the enclave and it is the first EPC +* page created for any enclave. It is moved from a temporary buffer to an EPC +* by the means of ENCLS(ECREATE) leaf. +*/ +struct sgx_secs { +u64 size; +u64 base; +u32 ssa_frame_size; +u32 miscselect; +u8 reserved1[24 ]; +u64 attributes; +u64 xfrm; +u32 mrenclave[8 ]; +u8 reserved2[32 ]; +u32 mrsigner[8 ]; +u8 reserved3[32 ]; +u32 config_id[16 ]; +u16 isv_prod_id; +u16 isv_svn; +u16 config_svn; +u8 reserved4[3834 ]; +} __packed; + +/** +* enum sgx_tcs_flags - execution flags for TCS +* %SGX_TCS_DBGOPTIN: If enabled allows single-stepping and breakpoints +* inside an enclave. It is cleared by EADD but can +* be set later with EDBGWR. +*/ +enum sgx_tcs_flags { +SGX_TCS_DBGOPTIN =0x01 , +}; + +#define SGX_TCS_RESERVED_MASK GENMASK_ULL(63, 1) +#define SGX_TCS_RESERVED_SIZE 4024 + +/** +* struct sgx_tcs - Thread Control Structure (TCS) +* @state: used to mark an entered TCS +* @flags: execution flags (cleared by EADD) +* @ssa_offset: SSA stack offset relative to the enclave base +* @ssa_index: the current SSA frame index (cleard by EADD) +* @nr_ssa_frames: the number of frame in the SSA stack +* @entry_offset: entry point offset relative to the enclave base +* @exit_addr: address outside the enclave to exit on an exception or +* interrupt +* @fs_offset: offset relative to the enclave base to become FS +* segment inside the enclave +* @gs_offset: offset relative to the enclave base to become GS +* segment inside the enclave +* @fs_limit: size to become a new FS-limit (only 32-bit enclaves) +* @gs_limit: size to become a new GS-limit (only 32-bit enclaves) +* +* Thread Control Structure (TCS) is an enclave page visible in its address +* space that defines an entry point inside the enclave. A thread enters inside +* an enclave by supplying address of TCS to ENCLU(EENTER). A TCS can be entered +* by only one thread at a time. +*/ +struct sgx_tcs { +u64 state; +u64 flags; +u64 ssa_offset; +u32 ssa_index; +u32 nr_ssa_frames; +u64 entry_offset; +u64 exit_addr; +u64 fs_offset; +u64 gs_offset; +u32 fs_limit; +u32 gs_limit; +u8 reserved[SGX_TCS_RESERVED_SIZE ]; +} __packed; + +/** +* struct sgx_pageinfo - an enclave page descriptor +* @addr: address of the enclave page +* @contents: pointer to the page contents +* @metadata: pointer either to a SECINFO or PCMD instance +* @secs: address of the SECS page +*/ +struct sgx_pageinfo { +u64 addr; +u64 contents; +u64 metadata; +u64 secs; +} __packed__aligned(32) ; + + +/** +* enum sgx_page_type - bits in the SECINFO flags defining the page type +* %SGX_PAGE_TYPE_SECS: a SECS page +* %SGX_PAGE_TYPE_TCS: a TCS page +* %SGX_PAGE_TYPE_REG: a regular page +* %SGX_PAGE_TYPE_VA: a VA page +* %SGX_PAGE_TYPE_TRIM: a page in trimmed state +*/ +enum sgx_page_type { +SGX_PAGE_TYPE_SECS , +SGX_PAGE_TYPE_TCS , +SGX_PAGE_TYPE_REG , +SGX_PAGE_TYPE_VA , +SGX_PAGE_TYPE_TRIM , +}; + +#define SGX_NR_PAGE_TYPES 5 +#define SGX_PAGE_TYPE_MASK GENMASK(7, 0) + +/** +* enum sgx_secinfo_flags - the flags field in &struct sgx_secinfo +* %SGX_SECINFO_R: allow read +* %SGX_SECINFO_W: allow write +* %SGX_SECINFO_X: allow execution +* %SGX_SECINFO_SECS: a SECS page +* %SGX_SECINFO_TCS: a TCS page +* %SGX_SECINFO_REG: a regular page +* %SGX_SECINFO_VA: a VA page +* %SGX_SECINFO_TRIM: a page in trimmed state +*/ +enum sgx_secinfo_flags { +SGX_SECINFO_R =BIT (0 ), +SGX_SECINFO_W =BIT (1 ), +SGX_SECINFO_X =BIT (2 ), +SGX_SECINFO_SECS = (SGX_PAGE_TYPE_SECS <<8 ), +SGX_SECINFO_TCS = (SGX_PAGE_TYPE_TCS <<8 ), +SGX_SECINFO_REG = (SGX_PAGE_TYPE_REG <<8 ), +SGX_SECINFO_VA = (SGX_PAGE_TYPE_VA <<8 ), +SGX_SECINFO_TRIM = (SGX_PAGE_TYPE_TRIM <<8 ), +}; + +#define SGX_SECINFO_PERMISSION_MASK GENMASK_ULL(2, 0) +#define SGX_SECINFO_PAGE_TYPE_MASK (SGX_PAGE_TYPE_MASK << 8) +#define SGX_SECINFO_RESERVED_MASK ~(SGX_SECINFO_PERMISSION_MASK | \ + SGX_SECINFO_PAGE_TYPE_MASK) + +/** +* struct sgx_secinfo - describes attributes of an EPC page +* @flags: permissions and type +* +* Used together with ENCLS leaves that add or modify an EPC page to an +* enclave to define page permissions and type. +*/ +struct sgx_secinfo { +u64 flags; +u8 reserved[56 ]; +} __packed__aligned(64) ; + +#define SGX_PCMD_RESERVED_SIZE 40 + +/** +* struct sgx_pcmd - Paging Crypto Metadata (PCMD) +* @enclave_id: enclave identifier +* @mac: MAC over PCMD, page contents and isvsvn +* +* PCMD is stored for every swapped page to the regular memory. When ELDU loads +* the page back it recalculates the MAC by using a isvsvn number stored in a +* VA page. Together these two structures bring integrity and rollback +* protection. +*/ +struct sgx_pcmd { +struct sgx_secinfo secinfo; +u64 enclave_id; +u8 reserved[SGX_PCMD_RESERVED_SIZE ]; +u8 mac[16 ]; +} __packed__aligned(128) ; + +#define SGX_SIGSTRUCT_RESERVED1_SIZE 84 +#define SGX_SIGSTRUCT_RESERVED2_SIZE 20 +#define SGX_SIGSTRUCT_RESERVED3_SIZE 32 +#define SGX_SIGSTRUCT_RESERVED4_SIZE 12 + +/** +* struct sgx_sigstruct_header - defines author of the enclave +* @header1: constant byte string +* @vendor: must be either 0x0000 or 0x8086 +* @date: YYYYMMDD in BCD +* @header2: costant byte string +* @swdefined: software defined value +*/ +struct sgx_sigstruct_header { +u64 header1[2 ]; +u32 vendor; +u32 date; +u64 header2[2 ]; +u32 swdefined; +u8 reserved1[84 ]; +} __packed; + +/** +* struct sgx_sigstruct_body - defines contents of the enclave +* @miscselect: additional information stored to an SSA frame +* @misc_mask: required miscselect in SECS +* @attributes: attributes for enclave +* @xfrm: XSave-Feature Request Mask (subset of XCR0) +* @attributes_mask: required attributes in SECS +* @xfrm_mask: required XFRM in SECS +* @mrenclave: SHA256-hash of the enclave contents +* @isvprodid: a user-defined value that is used in key derivation +* @isvsvn: a user-defined value that is used in key derivation +*/ +struct sgx_sigstruct_body { +u32 miscselect; +u32 misc_mask; +u8 reserved2[20 ]; +u64 attributes; +u64 xfrm; +u64 attributes_mask; +u64 xfrm_mask; +u8 mrenclave[32 ]; +u8 reserved3[32 ]; +u16 isvprodid; +u16 isvsvn; +} __packed; + +/** +* struct sgx_sigstruct - an enclave signature +* @header: defines author of the enclave +* @modulus: the modulus of the public key +* @exponent: the exponent of the public key +* @signature: the signature calculated over the fields except modulus, +* @body: defines contents of the enclave +* @q1: a value used in RSA signature verification +* @q2: a value used in RSA signature verification +* +* Header and body are the parts that are actual signed. The remaining fields +* define the signature of the enclave. +*/ +struct sgx_sigstruct { +struct sgx_sigstruct_header header; +u8 modulus[SGX_MODULUS_SIZE ]; +u32 exponent; +u8 signature[SGX_MODULUS_SIZE ]; +struct sgx_sigstruct_body body; +u8 reserved4[12 ]; +u8 q1[SGX_MODULUS_SIZE ]; +u8 q2[SGX_MODULUS_SIZE ]; +} __packed; + +#define SGX_LAUNCH_TOKEN_SIZE 304 + +#endif /* _ASM_X86_SGX_ARCH_H */
Signature · 1 line
2.25.1
PATCH v38 5/24 x86/sgx: Add wrappers for ENCLS leaf functions
In reply to Jarkko SakkinenENCLS is a ring 0 instruction, which contains a set of leaf functions for managing an enclave. Enclaves are measured and signed software entities, which are protected by asserting the outside memory accesses and memory encryption.
Add a two-layer macro system along with an encoding scheme to allow wrappers to return trap numbers along ENCLS-specific error codes. The bottom layer of the macro system splits between the leafs that return an error code and those that do not. The second layer generates the correct input/output annotations based on the number of operands for each leaf function.
ENCLS leaf functions are documented in
Intel SDM: 36.6 ENCLAVE INSTRUCTIONS AND INTEL®
--- arch/x86/kernel/cpu/sgx/encls.h | 238 ++++++++++++++++++++++++++++++++ 1 file changed, 238 insertions(+) create mode 100644 arch/x86/kernel/cpu/sgx/encls.h
arch/x86/kernel/cpu/sgx/encls.h new file +238 -0
@@ -0,0 +1,238 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */ +#ifndef _X86_ENCLS_H +#define _X86_ENCLS_H + +#include <linux/bitops.h> +#include <linux/err.h> +#include <linux/io.h> +#include <linux/rwsem.h> +#include <linux/types.h> +#include <asm/asm.h> +#include <asm/traps.h> +#include "sgx.h" + +enum sgx_encls_leaf { +ECREATE =0x00 , +EADD =0x01 , +EINIT =0x02 , +EREMOVE =0x03 , +EDGBRD =0x04 , +EDGBWR =0x05 , +EEXTEND =0x06 , +ELDU =0x08 , +EBLOCK =0x09 , +EPA =0x0A , +EWB =0x0B , +ETRACK =0x0C , +}; + +/** +* ENCLS_FAULT_FLAG - flag signifying an ENCLS return code is a trapnr +* +* ENCLS has its own (positive value) error codes and also generates +* ENCLS specific #GP and #PF faults. And the ENCLS values get munged +* with system error codes as everything percolates back up the stack. +* Unfortunately (for us), we need to precisely identify each unique +* error code, e.g. the action taken if EWB fails varies based on the +* type of fault and on the exact SGX error code, i.e. we can't simply +* convert all faults to -EFAULT. +* +* To make all three error types coexist, we set bit 30 to identify an +* ENCLS fault. Bit 31 (technically bits N:31) is used to differentiate +* between positive (faults and SGX error codes) and negative (system +* error codes) values. +*/ +#define ENCLS_FAULT_FLAG 0x40000000 + +/* Retrieve the encoded trapnr from the specified return code. */ +#define ENCLS_TRAPNR (r) ((r) & ~ENCLS_FAULT_FLAG) + +/* Issue a WARN() about an ENCLS leaf. */ +#define ENCLS_WARN (r, name) { \ + do { \ + int _r = (r); \ + WARN_ONCE(_r, "%s returned %d (0x%x)\n", (name), _r, _r); \ + } while (0); \ +} + +/** +* encls_failed() - Check if an ENCLS leaf function failed +* @ret: the return value of an ENCLS leaf function call +* +* Check if an ENCLS leaf function failed. This happens when the leaf function +* causes a fault that is not caused by an EPCM conflict or when the leaf +* function returns a non-zero value. +*/ +static inline bool encls_failed (int ret) +{ +int epcm_trapnr; + +if (boot_cpu_has (X86_FEATURE_SGX2 )) + epcm_trapnr =X86_TRAP_PF ; +else + epcm_trapnr =X86_TRAP_GP ; + +if (ret &ENCLS_FAULT_FLAG ) +return ENCLS_TRAPNR (ret) != epcm_trapnr; + +return !!ret; +} + +/** +* __encls_ret_N - encode an ENCLS leaf that returns an error code in EAX +* @rax: leaf number +* @inputs: asm inputs for the leaf +* +* Emit assembly for an ENCLS leaf that returns an error code, e.g. EREMOVE. +* And because SGX isn't complex enough as it is, leafs that return an error +* code also modify flags. +* +* Return: +* 0 on success, +* SGX error code on failure +*/ +#define __encls_ret_N(rax, inputs...) \ + ({ \ + int ret; \ + asm volatile( \ + "1: .byte 0x0f, 0x01, 0xcf;\n\t" \ + "2:\n" \ + ".section .fixup,\"ax\"\n" \ + "3: orl $"__stringify(ENCLS_FAULT_FLAG)",%%eax\n" \ + " jmp 2b\n" \ + ".previous\n" \ + _ASM_EXTABLE_FAULT(1b, 3b) \ + : "=a"(ret) \ + : "a"(rax), inputs \ + : "memory", "cc"); \ + ret; \ + }) + +#define __encls_ret_1 (rax, rcx) \ + ({ \ + __encls_ret_N(rax, "c"(rcx)); \ + }) + +#define __encls_ret_2 (rax, rbx, rcx) \ + ({ \ + __encls_ret_N(rax, "b"(rbx), "c"(rcx)); \ + }) + +#define __encls_ret_3 (rax, rbx, rcx, rdx) \ + ({ \ + __encls_ret_N(rax, "b"(rbx), "c"(rcx), "d"(rdx)); \ + }) + +/** +* __encls_N - encode an ENCLS leaf that doesn't return an error code +* @rax: leaf number +* @rbx_out: optional output variable +* @inputs: asm inputs for the leaf +* +* Emit assembly for an ENCLS leaf that does not return an error code, +* e.g. ECREATE. Leaves without error codes either succeed or fault. +* @rbx_out is an optional parameter for use by EDGBRD, which returns +* the requested value in RBX. +* +* Return: +* 0 on success, +* trapnr with ENCLS_FAULT_FLAG set on fault +*/ +#define __encls_N(rax, rbx_out, inputs...) \ + ({ \ + int ret; \ + asm volatile( \ + "1: .byte 0x0f, 0x01, 0xcf;\n\t" \ + " xor %%eax,%%eax;\n" \ + "2:\n" \ + ".section .fixup,\"ax\"\n" \ + "3: orl $"__stringify(ENCLS_FAULT_FLAG)",%%eax\n" \ + " jmp 2b\n" \ + ".previous\n" \ + _ASM_EXTABLE_FAULT(1b, 3b) \ + : "=a"(ret), "=b"(rbx_out) \ + : "a"(rax), inputs \ + : "memory"); \ + ret; \ + }) + +#define __encls_2 (rax, rbx, rcx) \ + ({ \ + unsigned long ign_rbx_out; \ + __encls_N(rax, ign_rbx_out, "b"(rbx), "c"(rcx)); \ + }) + +#define __encls_1_1 (rax, data, rcx) \ + ({ \ + unsigned long rbx_out; \ + int ret = __encls_N(rax, rbx_out, "c"(rcx)); \ + if (!ret) \ + data = rbx_out; \ + ret; \ + }) + +static inline int __ecreate (struct sgx_pageinfo *pginfo,void *secs) +{ +return __encls_2 (ECREATE , pginfo, secs); +} + +static inline int __eextend (void *secs,void *addr) +{ +return __encls_2 (EEXTEND , secs, addr); +} + +static inline int __eadd (struct sgx_pageinfo *pginfo,void *addr) +{ +return __encls_2 (EADD , pginfo, addr); +} + +static inline int __einit (void *sigstruct,void *token,void *secs) +{ +return __encls_ret_3 (EINIT , sigstruct, secs, token); +} + +static inline int __eremove (void *addr) +{ +return __encls_ret_1 (EREMOVE , addr); +} + +static inline int __edbgwr (void *addr,unsigned long *data) +{ +return __encls_2 (EDGBWR , *data, addr); +} + +static inline int __edbgrd (void *addr,unsigned long *data) +{ +return __encls_1_1 (EDGBRD , *data, addr); +} + +static inline int __etrack (void *addr) +{ +return __encls_ret_1 (ETRACK , addr); +} + +static inline int __eldu (struct sgx_pageinfo *pginfo,void *addr, +void *va) +{ +return __encls_ret_3 (ELDU , pginfo, addr, va); +} + +static inline int __eblock (void *addr) +{ +return __encls_ret_1 (EBLOCK , addr); +} + +static inline int __epa (void *addr) +{ +unsigned long rbx =SGX_PAGE_TYPE_VA ; + +return __encls_2 (EPA , rbx, addr); +} + +static inline int __ewb (struct sgx_pageinfo *pginfo,void *addr, +void *va) +{ +return __encls_ret_3 (EWB , pginfo, addr, va); +} + +#endif /* _X86_ENCLS_H */
Signature · 1 line
2.25.1
PATCH v38 6/24 x86/cpu/intel: Detect SGX support
In reply to Jarkko SakkinenFrom: Sean Christopherson <[email protected]>
Configure SGX as part of feature control MSR initialization and update the associated X86_FEATURE flags accordingly. Because the kernel will require the LE hash MSRs to be writable when running native enclaves, disable X86_FEATURE_SGX (and all derivatives) if SGX Launch Control is not (or cannot) be fully enabled via feature control MSR.
The check is done for every CPU, not just BSP, in order to verify that MSR_IA32_FEATURE_CONTROL is correctly configured on all CPUs. The other parts of the kernel, like the enclave driver, expect the same configuration from all CPUs.
Note, unlike VMX, clear the X86_FEATURE_SGX* flags for all CPUs if any CPU lacks SGX support as the kernel expects SGX to be available on all CPUs. X86_FEATURE_VMX is intentionally cleared only for the current CPU so that KVM can provide additional information if KVM fails to load, e.g. print which CPU doesn't support VMX. KVM/VMX requires additional per-CPU enabling, e.g. to set CR4.VMXE and do VMXON, and so already has the necessary infrastructure to do per-CPU checks. SGX on the other hand doesn't require additional enabling, so clearing the feature flags on all CPUs means the SGX subsystem doesn't need to manually do support checks on a per-CPU basis.
--- arch/x86/kernel/cpu/feat_ctl.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/feat_ctl.c +31 -1
@@ -93,16 +93,35 @@ static void init_vmx_capabilities(struct cpuinfo_x86 *c) }#endif /* CONFIG_X86_VMX_FEATURE_NAMES */ +static void clear_sgx_caps (void ) +{ +setup_clear_cpu_cap (X86_FEATURE_SGX ); +setup_clear_cpu_cap (X86_FEATURE_SGX_LC ); +setup_clear_cpu_cap (X86_FEATURE_SGX1 ); +setup_clear_cpu_cap (X86_FEATURE_SGX2 ); +} +void init_ia32_feat_ctl (struct cpuinfo_x86 *c) {bool tboot =tboot_enabled (); +bool enable_sgx;u64 msr;if (rdmsrl_safe (MSR_IA32_FEAT_CTL , &msr)) {clear_cpu_cap (c,X86_FEATURE_VMX ); +clear_sgx_caps ();return ; } +/* +* Enable SGX if and only if the kernel supports SGX and Launch Control +* is supported, i.e. disable SGX if the LE hash MSRs can't be written. +*/ + enable_sgx =cpu_has (c,X86_FEATURE_SGX ) && +cpu_has (c,X86_FEATURE_SGX1 ) && +cpu_has (c,X86_FEATURE_SGX_LC ) && +IS_ENABLED (CONFIG_INTEL_SGX ); +if (msr &FEAT_CTL_LOCKED ) goto update_caps;
@@ -124,13 +143,16 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c) msr |=FEAT_CTL_VMX_ENABLED_INSIDE_SMX ; } +if (enable_sgx) + msr |=FEAT_CTL_SGX_ENABLED |FEAT_CTL_SGX_LC_ENABLED ; +wrmsrl (MSR_IA32_FEAT_CTL , msr); update_caps:set_cpu_cap (c,X86_FEATURE_MSR_IA32_FEAT_CTL );if (!cpu_has (c,X86_FEATURE_VMX )) -return ; + goto update_sgx;if ( (tboot && !(msr &FEAT_CTL_VMX_ENABLED_INSIDE_SMX )) || (!tboot && !(msr &FEAT_CTL_VMX_ENABLED_OUTSIDE_SMX ))) {
@@ -143,4 +165,12 @@ void init_ia32_feat_ctl(struct cpuinfo_x86 *c)init_vmx_capabilities (c);#endif } + +update_sgx: +if (!(msr &FEAT_CTL_SGX_ENABLED ) || + !(msr &FEAT_CTL_SGX_LC_ENABLED ) || !enable_sgx) { +if (enable_sgx) +pr_err_once ("SGX disabled by BIOS\n" ); +clear_sgx_caps (); + } }
Signature · 1 line
2.25.1
PATCH v38 7/24 x86/cpu/intel: Add nosgx kernel parameter
In reply to Jarkko SakkinenAdd kernel parameter to disable Intel SGX kernel support.
--- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/x86/kernel/cpu/feat_ctl.c | 9 +++++++++ 2 files changed, 11 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +2 -0
@@ -3339,6 +3339,8 @@ nosep [BUGS=X86-32] Disables x86 SYSENTER/SYSEXIT support. + nosgx [X86-64,SGX] Disables Intel SGX kernel support. + nosmp [SMP] Tells an SMP kernel to act as a UP kernel, and disable the IO APIC. legacy for "maxcpus=0".
arch/x86/kernel/cpu/feat_ctl.c +9 -0
@@ -101,6 +101,15 @@ static void clear_sgx_caps(void)setup_clear_cpu_cap (X86_FEATURE_SGX2 ); } +static int __init nosgx (char *str) +{ +clear_sgx_caps (); + +return 0 ; +} + +early_param ("nosgx" , nosgx); +void init_ia32_feat_ctl (struct cpuinfo_x86 *c) {bool tboot =tboot_enabled ();
Signature · 1 line
2.25.1
PATCH v38 8/24 x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections
In reply to Jarkko SakkinenFrom: Sean Christopherson <[email protected]>
Enumerate Enclave Page Cache (EPC) sections via CPUID and add the data structures necessary to track EPC pages so that they can be easily borrowed for different uses.
Embed section index to the first eight bits of the EPC page descriptor. Existing client hardware supports only a single section, while upcoming server hardware will support at most eight sections. Thus, eight bits should be enough for long term needs.
--- arch/x86/Kconfig | 17 +++ arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/sgx/Makefile | 2 + arch/x86/kernel/cpu/sgx/main.c | 216 +++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 52 ++++++++ 5 files changed, 288 insertions(+) create mode 100644 arch/x86/kernel/cpu/sgx/Makefile create mode 100644 arch/x86/kernel/cpu/sgx/main.c create mode 100644 arch/x86/kernel/cpu/sgx/sgx.h
arch/x86/Kconfig +17 -0
@@ -1927,6 +1927,23 @@ config X86_INTEL_TSX_MODE_AUTO side channel attacks- equals the tsx=auto command line parameter. endchoice +config INTEL_SGX + bool "Intel SGX" + depends on X86_64 && CPU_SUP_INTEL + depends on CRYPTO=y + depends on CRYPTO_SHA256=y + select SRCU + select MMU_NOTIFIER + help + Intel(R) Software Guard eXtensions (SGX) is a set of CPU instructions + that can be used by applications to set aside private regions of code + and data, referred to as enclaves. An enclave's private memory can + only be accessed by code running within the enclave. Accesses from + outside the enclave, including other enclaves, are disallowed by + hardware. + + If unsure, say N. + config EFI bool "EFI runtime service support" depends on ACPI
arch/x86/kernel/cpu/Makefile +1 -0
@@ -48,6 +48,7 @@ obj-$(CONFIG_X86_MCE) += mce/ obj-$(CONFIG_MTRR) += mtrr/ obj-$(CONFIG_MICROCODE) += microcode/ obj-$(CONFIG_X86_CPU_RESCTRL) += resctrl/ +obj-$(CONFIG_INTEL_SGX) += sgx/ obj-$(CONFIG_X86_LOCAL_APIC) += perfctr-watchdog.o
arch/x86/kernel/cpu/sgx/Makefile new file +2 -0
@@ -0,0 +1,2 @@ +obj-y += \ + main.o
arch/x86/kernel/cpu/sgx/main.c new file +216 -0
216 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/sgx.h new file +52 -0
52 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v38 9/24 x86/sgx: Add __sgx_alloc_epc_page() and sgx_free_epc_page()
In reply to Jarkko SakkinenAdd __sgx_alloc_epc_page(), which iterates through EPC sections and borrows a page structure that is not used by anyone else. When a page is no longer needed it must be released with sgx_free_epc_page(). This function implicitly calls ENCLS[EREMOVE], which will return the page to the uninitialized state (i.e. not required from caller part).
--- arch/x86/kernel/cpu/sgx/main.c | 62 ++++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 3 ++ 2 files changed, 65 insertions(+)
arch/x86/kernel/cpu/sgx/main.c +62 -0
@@ -83,6 +83,68 @@ static bool __init sgx_page_reclaimer_init(void)return true; } +static struct sgx_epc_page *__sgx_alloc_epc_page_from_section (struct sgx_epc_section *section) +{ +struct sgx_epc_page *page; + +if (list_empty (§ion->page_list)) +return NULL ; + + page = list_first_entry(§ion->page_list, struct sgx_epc_page, list); +list_del_init (&page->list); + +return page; +} + +/** +* __sgx_alloc_epc_page() - Allocate an EPC page +* +* Iterate through EPC sections and borrow a free EPC page to the caller. When a +* page is no longer needed it must be released with sgx_free_epc_page(). +* +* Return: +* an EPC page, +* -errno on error +*/ +struct sgx_epc_page *__sgx_alloc_epc_page (void ) +{ +struct sgx_epc_section *section; +struct sgx_epc_page *page; +int i; + +for (i =0 ; i < sgx_nr_epc_sections; i++) { + section = &sgx_epc_sections[i]; +spin_lock (§ion->lock); + page =__sgx_alloc_epc_page_from_section (section); +spin_unlock (§ion->lock); + +if (page) +return page; + } + +return ERR_PTR (-ENOMEM ); +} + +/** +* sgx_free_epc_page() - Free an EPC page +* @page: an EPC page +* +* Call EREMOVE for an EPC page and insert it back to the list of free pages. +*/ +void sgx_free_epc_page (struct sgx_epc_page *page) +{ +struct sgx_epc_section *section =sgx_get_epc_section (page); +int ret; + + ret =__eremove (sgx_get_epc_addr (page)); +if (WARN_ONCE (ret,"EREMOVE returned %d (0x%x)" , ret, ret)) +return ; + +spin_lock (§ion->lock); +list_add_tail (&page->list, §ion->page_list); +spin_unlock (§ion->lock); +} +static void __init sgx_free_epc_section (struct sgx_epc_section *section) {struct sgx_epc_page *page;
arch/x86/kernel/cpu/sgx/sgx.h +3 -0
@@ -49,4 +49,7 @@ static inline void *sgx_get_epc_addr(struct sgx_epc_page *page) return section->va + (page->desc & PAGE_MASK) - section->pa; } +struct sgx_epc_page *__sgx_alloc_epc_page (void ); +void sgx_free_epc_page (struct sgx_epc_page *page); +#endif /* _X86_SGX_H */
Signature · 1 line
2.25.1
PATCH v38 10/24 mm: Add vm_ops->mprotect()
In reply to Jarkko SakkinenFrom: Sean Christopherson <[email protected]>
Add vm_ops()->mprotect() for additional constraints for a VMA.
Intel Software Guard eXtensions (SGX) will use this callback to add two constraints:
1. Verify that the address range does not have holes: each page address must be filled with an enclave page.
2. Verify that VMA permissions won't surpass the permissions of any enclave page within the address range. Enclave cryptographically sealed permissions for each page address that set the upper limit for possible VMA permissions. Not respecting this can cause #GP's to be emitted.
--- include/linux/mm.h | 3 +++ mm/mprotect.c | 5 ++++- 2 files changed, 7 insertions(+), 1 deletion(-)
include/linux/mm.h +3 -0
@@ -547,6 +547,9 @@ struct vm_operations_struct {void (*close)(struct vm_area_struct * area);int (*split)(struct vm_area_struct * area,unsigned long addr);int (*mremap)(struct vm_area_struct * area); +int (*mprotect)(struct vm_area_struct *vma, +struct vm_area_struct **pprev,unsigned long start, +unsigned long end,unsigned long newflags);vm_fault_t (*fault)(struct vm_fault *vmf);vm_fault_t (*huge_fault)(struct vm_fault *vmf,enum page_entry_size pe_size);
mm/mprotect.c +4 -1
@@ -610,7 +610,10 @@ static int do_mprotect_pkey(unsigned long start, size_t len, tmp = vma->vm_end;if (tmp > end) tmp = end; - error =mprotect_fixup (vma, &prev, nstart, tmp, newflags); +if (vma->vm_ops && vma->vm_ops->mprotect) + error = vma->vm_ops->mprotect (vma, &prev, nstart, tmp, newflags); +else + error =mprotect_fixup (vma, &prev, nstart, tmp, newflags);if (error) goto out; nstart = tmp;
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 22 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
Add vm_ops()->mprotect() for additional constraints for a VMA.
Intel Software Guard eXtensions (SGX) will use this callback to add two constraints:
1. Verify that the address range does not have holes: each page address must be filled with an enclave page.
2. Verify that VMA permissions won't surpass the permissions of any enclave page within the address range. Enclave cryptographically sealed permissions for each page address that set the upper limit for possible VMA permissions. Not respecting this can cause #GP's to be emitted.Cc Andrew Morton <[email protected]>Cc Matthew Wilcox <[email protected]>Acked-by Jethro Beekman <[email protected]>Reviewed-by Darren Kenny <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Co-developed-by Jarkko Sakkinen <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>
Quoted 4 lines
--- include/linux/mm.h | 3 +++ mm/mprotect.c | 5 ++++- 2 files changed, 7 insertions(+), 1 deletion(-)
Needs an ACK from an mm person.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteAndy Lutomirski
maintainerRe: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Jarkko SakkinenQuoted 8 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
Add vm_ops()->mprotect() for additional constraints for a VMA.
Intel Software Guard eXtensions (SGX) will use this callback to add two constraints:
Quoted 6 lines
1. Verify that the address range does not have holes: each page address must be filled with an enclave page.
2. Verify that VMA permissions won't surpass the permissions of any enclave page within the address range. Enclave cryptographically sealed permissions for each page address that set the upper limit for possible VMA permissions. Not respecting this can cause #GP's to be emitted.
It's been awhile since I looked at this. Can you remind us: is this just preventing userspace from shooting itself in the foot or is this something more important?
--Andy
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()'
Quoted 17 lines · Andy Lutomirski
Quoted 8 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
Add vm_ops()->mprotect() for additional constraints for a VMA.
Intel Software Guard eXtensions (SGX) will use this callback to add two constraints:
Quoted 6 lines
1. Verify that the address range does not have holes: each page address must be filled with an enclave page.
2. Verify that VMA permissions won't surpass the permissions of any enclave page within the address range. Enclave cryptographically sealed permissions for each page address that set the upper limit for possible VMA permissions. Not respecting this can cause #GP's to be emitted.
Quoted 5 lines
It's been awhile since I looked at this. Can you remind us: is this just preventing userspace from shooting itself in the foot or is this something more important?
--Andy
Haitao found this:
https://patchwork.kernel.org/patch/10978327/
The way I understand it, for an LSM hook it makes sense that the mprotect() can deduce a single permission for an enclave address range. With those constraints it is possible.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Andy LutomirskiQuoted 10 lines · Andy Lutomirski
Quoted 8 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
Add vm_ops()->mprotect() for additional constraints for a VMA.
Intel Software Guard eXtensions (SGX) will use this callback to add two constraints:
Quoted 6 lines
Quoted 6 lines
1. Verify that the address range does not have holes: each page address must be filled with an enclave page.
2. Verify that VMA permissions won't surpass the permissions of any enclave page within the address range. Enclave cryptographically sealed permissions for each page address that set the upper limit for possible VMA permissions. Not respecting this can cause #GP's to be emitted.
Side note, #GP is wrong. EPCM violations are #PFs. Skylake CPUs #GP, but that's technically an errata. But this isn't the real motivation, e.g. userspace can already trigger #GP/#PF by reading/writing a bad address, SGX simply adds another flavor.
Quoted 3 lines
It's been awhile since I looked at this. Can you remind us: is this just preventing userspace from shooting itself in the foot or is this something more important?
Something more important, it's used to prevent userspace from circumventing a noexec filesystem by loading code into an enclave, and to give the kernel the option of adding enclave specific LSM policies in the future.
The source file (if one exists) for the enclave is long gone when the enclave is actually mmap()'d and mprotect()'d. To enforce noexec, the requested permissions for a given page are snapshotted when the page is added to the enclave, i.e. when the enclave is built. Enclave pages that will be executable must originate from an a MAYEXEC VMA, e.g. the source page can't come from a noexec file system.
The ->mprotect() hook allows SGX to reject mprotect() if userspace is declaring permissions beyond what are allowed, e.g. trying to map an enclave page with EXEC permissions when the page was added to the enclave without EXEC.
Future LSM policies have a similar need due to vm_file always pointing at /dev/sgx/enclave, e.g. policies couldn't be attached to a specific enclave. ->mprotect() again allows enforcing permissions at map time that were checked at enclave build time, e.g. via an LSM hook.
Deferring ->mprotect() until LSM support is added (if it ever is) would be problematic due to SGX2. With SGX2, userspace can extend permissions of an enclave page (for the CPU's EPC Map entry, not the kernel's page tables) without bouncing through the kernel. Without ->mprotect () enforcement. userspace could do EADD(RW) -> mprotect(RWX) -> EMODPE(X) to gain W+X. We want to disallow such a flow now, i.e. force userspace to do EADD(RW,X), so that the hypothetical LSM hook would have all information at EADD(), i.e. would be aware of the EXEC permission, without creating divergent behavior based on whether or not an LSM is active.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 50 lines · Sean Christopherson
Quoted 10 lines · Andy Lutomirski
Quoted 10 lines
On Tue, Sep 15, 2020 at 4:28 AM Jarkko Sakkinen
<[email protected]> wrote:From: Sean Christopherson <[email protected]>
Add vm_ops()->mprotect() for additional constraints for a VMA.
Intel Software Guard eXtensions (SGX) will use this callback to add two constraints:
Quoted 6 lines
Quoted 6 lines
1. Verify that the address range does not have holes: each page address must be filled with an enclave page.
2. Verify that VMA permissions won't surpass the permissions of any enclave page within the address range. Enclave cryptographically sealed permissions for each page address that set the upper limit for possible VMA permissions. Not respecting this can cause #GP's to be emitted.Side note, #GP is wrong. EPCM violations are #PFs. Skylake CPUs #GP, but that's technically an errata. But this isn't the real motivation, e.g. userspace can already trigger #GP/#PF by reading/writing a bad address, SGX simply adds another flavor.
Quoted 3 lines
It's been awhile since I looked at this. Can you remind us: is this just preventing userspace from shooting itself in the foot or is this something more important?
Something more important, it's used to prevent userspace from circumventing a noexec filesystem by loading code into an enclave, and to give the kernel the option of adding enclave specific LSM policies in the future.
The source file (if one exists) for the enclave is long gone when the enclave is actually mmap()'d and mprotect()'d. To enforce noexec, the requested permissions for a given page are snapshotted when the page is added to the enclave, i.e. when the enclave is built. Enclave pages that will be executable must originate from an a MAYEXEC VMA, e.g. the source page can't come from a noexec file system.
The ->mprotect() hook allows SGX to reject
mprotect()if userspace is declaring permissions beyond what are allowed, e.g. trying to map an enclave page with EXEC permissions when the page was added to the enclave without EXEC.Future LSM policies have a similar need due to
vm_filealways pointing at/dev/sgx/enclave, e.g. policies couldn't be attached to a specific enclave. ->mprotect() again allows enforcing permissions at map time that were checked at enclave build time, e.g. via an LSM hook.
Quoted 9 lines
Deferring ->mprotect() until LSM support is added (if it ever is) would be problematic due to SGX2. With SGX2, userspace can extend permissions of an enclave page (for the CPU's EPC Map entry, not the kernel's page tables) without bouncing through the kernel. Without ->mprotect () enforcement. userspace could do EADD(RW) -> mprotect(RWX) -> EMODPE(X) to gain W+X. We want to disallow such a flow now, i.e. force userspace to do
EADD(RW,X), so that the hypothetical LSM hook would have all information atEADD(), i.e. would be aware of the EXEC permission, without creating divergent behavior based on whether or not an LSM is active.
That’s what I thought. Can we get this in the changelog?
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 60 lines · Andy Lutomirski
Quoted 50 lines · Sean Christopherson
Quoted 10 lines · Andy Lutomirski
Quoted 10 lines
On Tue, Sep 15, 2020 at 4:28 AM Jarkko Sakkinen
<[email protected]> wrote:From: Sean Christopherson <[email protected]>
Add vm_ops()->mprotect() for additional constraints for a VMA.
Intel Software Guard eXtensions (SGX) will use this callback to add two constraints:
Quoted 6 lines
Quoted 6 lines
1. Verify that the address range does not have holes: each page address must be filled with an enclave page.
2. Verify that VMA permissions won't surpass the permissions of any enclave page within the address range. Enclave cryptographically sealed permissions for each page address that set the upper limit for possible VMA permissions. Not respecting this can cause #GP's to be emitted.Side note, #GP is wrong. EPCM violations are #PFs. Skylake CPUs #GP, but that's technically an errata. But this isn't the real motivation, e.g. userspace can already trigger #GP/#PF by reading/writing a bad address, SGX simply adds another flavor.
Quoted 3 lines
It's been awhile since I looked at this. Can you remind us: is this just preventing userspace from shooting itself in the foot or is this something more important?
Something more important, it's used to prevent userspace from circumventing a noexec filesystem by loading code into an enclave, and to give the kernel the option of adding enclave specific LSM policies in the future.
The source file (if one exists) for the enclave is long gone when the enclave is actually mmap()'d and mprotect()'d. To enforce noexec, the requested permissions for a given page are snapshotted when the page is added to the enclave, i.e. when the enclave is built. Enclave pages that will be executable must originate from an a MAYEXEC VMA, e.g. the source page can't come from a noexec file system.
The ->mprotect() hook allows SGX to reject
mprotect()if userspace is declaring permissions beyond what are allowed, e.g. trying to map an enclave page with EXEC permissions when the page was added to the enclave without EXEC.Future LSM policies have a similar need due to
vm_filealways pointing at/dev/sgx/enclave, e.g. policies couldn't be attached to a specific enclave. ->mprotect() again allows enforcing permissions at map time that were checked at enclave build time, e.g. via an LSM hook.Quoted 9 lines
Deferring ->mprotect() until LSM support is added (if it ever is) would be problematic due to SGX2. With SGX2, userspace can extend permissions of an enclave page (for the CPU's EPC Map entry, not the kernel's page tables) without bouncing through the kernel. Without ->mprotect () enforcement. userspace could do EADD(RW) -> mprotect(RWX) -> EMODPE(X) to gain W+X. We want to disallow such a flow now, i.e. force userspace to do
EADD(RW,X), so that the hypothetical LSM hook would have all information atEADD(), i.e. would be aware of the EXEC permission, without creating divergent behavior based on whether or not an LSM is active.
Quoted 1 line
That’s what I thought. Can we get this in the changelog?
I rewrote the commit message.
"
mm: Add 'mprotect' callback to vm_ops
Intel Sofware Guard eXtensions (SGX) allows creation of executable blobs called enclaves, of which page permissions are defined when the enclave is first loaded. Once an enclave is loaded and initialized, it can be mapped to the process address space.
Enclave permissions can be dynamically modified by using ENCLS[EMODPE] instruction. We want to limit its use to not allow higher permissions than the ones defined when the enclave was first created.
Add 'mprotect' hook to vm_ops, so that we can implement a callback for SGX that will check that {mmap, mprotect}() permissions do not surpass any of the page permissions in the address range defined.
This is required in order to be able to make any access control decisions when enclave pages are loaded.
"
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 2 lines · Jarkko Sakkinen
Intel Sofware Guard eXtensions (SGX) allows creation of executable blobs called enclaves, of which page permissions are defined when the enclave
"of which" => "for which"
Quoted 2 lines
is first loaded. Once an enclave is loaded and initialized, it can be mapped to the process address space.
Could you compare and contrast this a *bit* with existing executables? What's special about SGX? ELF executables have page permissions inside the binary too. Why don't we use this mechanism for them?
Quoted 1 line
Enclave permissions can be dynamically modified by using ENCLS[EMODPE]
I'm not sure this sentence matters. I'm not sure why I care what the instruction is named that does this. But, it _sounds_ here like an enclave can adjust its own permissions directly with ENCLS[EMODPE].
Quoted 2 lines
instruction. We want to limit its use to not allow higher permissions than the ones defined when the enclave was first created.
Rather than higher and lower, please use stronger and weaker.
Also, please get rid of the "we".
Quoted 3 lines
Add 'mprotect' hook to
vm_ops, so that we can implement a callback for SGX that will check that {mmap, mprotect}() permissions do not surpass any of the page permissions in the address range defined.
"check" => "ensure"
Quoted 2 lines
This is required in order to be able to make any access control decisions when enclave pages are loaded.
Now I'm confused. I actually don't think I have a strong understanding of how an enclave actually gets loaded, how mmap() and mprotect() are normally used and what this hook is intended to thwart.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 5 lines · Dave Hansen
Quoted 1 line
Enclave permissions can be dynamically modified by using ENCLS[EMODPE]
I'm not sure this sentence matters. I'm not sure why I care what the instruction is named that does this. But, it _sounds_ here like an enclave can adjust its own permissions directly with ENCLS[EMODPE].
If there was no EMODPE, I would drop this patch from the patch set. It is the only reason I keep it. There are no other hard reasons to have this.
Quoted 3 lines
Now I'm confused. I actually don't think I have a strong understanding of how an enclave actually gets loaded, how
mmap()andmprotect()are normally used and what this hook is intended to thwart.
Enclave gets loaded with the ioctl API so ATM we rely only on DAC permissions.
In future you might want to have LSM hooks to improve granularity in two points:
1. When pages are added using SGX_IOC_ENCLAVE ADD_PAGES.
2. When enclave is initialized using SGX_IOC_ENCLAVE_INIT
In both you might want to make a policy decision based on origin and page permissions.
If we do not limit mmap(), enclave could later on upgrade its permissions with EMODPE and do mmap().
No matter what kind of LSM hooks or whatever possible improvements are done later on to access control, they won't work unless we have this because the permissions could be different than the original.
With this change you can still do EMODPE inside an enclave, but you don't gain anything with it because your max permissions are capped during the build time.
I'm not sure what exactly from this is missing from the commit message but if you get this explanation maybe you can help me out with that.
Thank you for the feedback.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Dave HansenQuoted 4 lines · Dave Hansen
Quoted 2 lines · Jarkko Sakkinen
Intel Sofware Guard eXtensions (SGX) allows creation of executable blobs called enclaves, of which page permissions are defined when the enclave
Quoted 8 lines
"of which" => "for which"
Quoted 2 lines
is first loaded. Once an enclave is loaded and initialized, it can be mapped to the process address space.
Could you compare and contrast this a *bit* with existing executables? What's special about SGX? ELF executables have page permissions inside the binary too. Why don't we use this mechanism for them?
There is no standard file format for enclaves. They are dynamically built. And the way enclaves are used as part of an app and throwing container inside enclave differ. A single format would no work too well for all possible use cases.
I cannot formally prove this of course but it is highly unlikely that we could ever define such a format. Thus, the security focus is allow to verify from origin.
And the existing ecosystem around SGX is already too large to suddenly move to a single format. User base, I guess, is also an argument. This is not yet mainline code so technically we have zero ABI debt but I still think this is a valid point because SGX is already widely used.
I'm not really sure what would be the best way to nail this information to the commit message but I'll try to figure out something before I send the next version of the patch set.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Dave HansenQuoted 3 lines · Dave Hansen
Now I'm confused. I actually don't think I have a strong understanding of how an enclave actually gets loaded, how
mmap()andmprotect()are normally used and what this hook is intended to thwart.
You saw my other comments. I scraped this together based on your feedback and my responses:
"
mm: Add 'mprotect' callback to vm_ops
Intel Sofware Guard eXtensions (SGX) allows creation of blobs called enclaves, for which page permissions are defined when the enclave is first loaded. Once an enclave is loaded and initialized, it can be mapped to the process address space.
There is no standard file format for enclaves. They are dynamically built and the ways how enclaves are deployed differ greatly. For an app you might want to have a simple static binary, but on the other hand for a container you might want to dynamically create the whole thing at run-time. Also, the existing ecosystem for SGX is already large, which would make the task very hard.
Finally, even if there was a standard format, one would still want a dynamic way to add pages to the enclave. One big reason for this is that enclaves have load time defined pages that represent entry points to the enclave. Each entry point can service one hardware thread at a time and you might want to run-time parametrize this depending on your environment.
The consequence is that enclaves are best created with an ioctl API and the access control can be based only to the origin of the source file for the enclave data, i.e. on VMA file pointer and page permissions. For example, this could be done with LSM hooks that are triggered in the appropriate ioctl's and they could make the access control decision based on this information.
Unfortunately, there is ENCLS[EMODPE] that a running enclave can use to upgrade its permissions. If we do not limit mmap() and mprotect(), enclave could upgrade its permissions by using EMODPE followed by an appropriate mprotect() call. This would be completely hidden from the kernel.
Add 'mprotect' hook to vm_ops, so that a callback can be implemeted for SGX that will ensure that {mmap, mprotect}() permissions do not surpass any of the original page permissions. This feature allows to maintain and refine sane access control for enclaves.
"
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 12 lines · Jarkko Sakkinen
The consequence is that enclaves are best created with an ioctl API and the access control can be based only to the origin of the source file for the enclave data, i.e. on VMA file pointer and page permissions. For example, this could be done with LSM hooks that are triggered in the appropriate ioctl's and they could make the access control decision based on this information.
Unfortunately, there is ENCLS[EMODPE] that a running enclave can use to upgrade its permissions. If we do not limit
mmap()andmprotect(), enclave could upgrade its permissions by using EMODPE followed by an appropriatemprotect()call. This would be completely hidden from the kernel.
Quoted 4 lines
Add 'mprotect' hook to
vm_ops, so that a callback can be implemeted for SGX that will ensure that {mmap, mprotect}() permissions do not surpass any of the original page permissions. This feature allows to maintain and refine sane access control for enclaves.
Maybe I'm just being dense, but I still don't have a clear idea what function this hook serves.
I understand that SGX has an orthogonal set of page permissions to the normal x86 page tables. It needs these so that the OS can't play nasty tricks on the enclave, like removing read-only protections that provide hardening.
But, I still don't get the connection to mprotect() and the x86 paging permissions. If the enclave's permissions are orthogonal, then why bother with this hook? Why does the OS view of the enclave's memory matter?
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 21 lines · Dave Hansen
Quoted 12 lines · Jarkko Sakkinen
The consequence is that enclaves are best created with an ioctl API and the access control can be based only to the origin of the source file for the enclave data, i.e. on VMA file pointer and page permissions. For example, this could be done with LSM hooks that are triggered in the appropriate ioctl's and they could make the access control decision based on this information.
Unfortunately, there is ENCLS[EMODPE] that a running enclave can use to upgrade its permissions. If we do not limit
mmap()andmprotect(), enclave could upgrade its permissions by using EMODPE followed by an appropriatemprotect()call. This would be completely hidden from the kernel.Quoted 4 lines
Add 'mprotect' hook to
vm_ops, so that a callback can be implemeted for SGX that will ensure that {mmap, mprotect}() permissions do not surpass any of the original page permissions. This feature allows to maintain and refine sane access control for enclaves.Maybe I'm just being dense, but I still don't have a clear idea what function this hook serves.
Quoted 8 lines
I understand that SGX has an orthogonal set of page permissions to the normal x86 page tables. It needs these so that the OS can't play nasty tricks on the enclave, like removing read-only protections that provide hardening.
But, I still don't get the connection to
mprotect()and the x86 paging permissions. If the enclave's permissions are orthogonal, then why bother with this hook? Why does the OS view of the enclave's memory matter?
For the purpose of this discussion, ignore the enclave permissions. The only reason they're mentioned is to explain the background (well, try to) of how we ended up at an ->mprotect() hook. There was a great deal of discussion in the past about whether or not we could use enclave permissions to enforce OS permissions. The TL:DR version is that because of ENCLU[EMODPE], the answer is "no".
What we're preventing via ->mprotect() is using SGX to bypass existing restrictions on code execution, e.g. noexec and LSM policies.
Because code must first be loaded into an enclave before it can be executed, all enclaves are kind of a variant on (in SELinux terminology) either EXECMOD or EXECMEM. I.e. it's simply not possible to execute an enclave by mapping the source file as executable. This effectively allows userspace to bypass a noexec FS by loading code into an enclave without EXEC perms on the source file, only on /dev/sgx/enclave, and denying EXEC on /dev/sgx/enclave would prevent running _any_ enclave.
The ->mprotect() hook is used by SGX to require userspace to declare what permissions are allowed on any given enclave page, e.g. SGX's mmap()/mprotect() requires all underlying enclave pages to be declared as executable if the mmap()/mprotect() is specifying VM_EXEC. By requiring userspace to declare their intent up front, SGX can then enforce noexec by requiring pages that are declared as executable to have VM_MAYEXEC set in the source VMA when loading code into the enclave.
As Jarkko pointed out, an alternative to adding ->mprotect() would be to simply require VM_MAYEXEC on _all_ source VMAs when loading code into the enclave. That would work, albeit with the potentially undesirable side effect of preventing loading any part of an enclave from e.g. a noexec, readonly FS.
But, unconditionally requiring VM_MAYEXEC doesn't address the Linux Security Module hooks for mmap() and mprotect(), which could also be bypassed by abusing SGX. E.g. a process could gain arbitrary code execution by loading code from anonymous memory into an enclave, as the LSM checks hooks at mmap()/mprotect() will see always vm_file=/dev/sgx/enclave. An LSM could deny a process access to /dev/sgx/enclave, but again that is very coarse granularity.
By requiring userspace to declare permissions up front (when loading code/data into an enclave), SGX can make explicit upcalls to LSMs hooks at load time so that an LSM can enforce a meaningful policy, e.g. require all enclave code to originate from an executable file system. This series doesn't actually implement the LSM integration, but it does ensure that _if_ we want to add LSM support in the future, we can do so without breaking userspace.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Dave HansenQuoted 21 lines · Dave Hansen
Quoted 12 lines · Jarkko Sakkinen
The consequence is that enclaves are best created with an ioctl API and the access control can be based only to the origin of the source file for the enclave data, i.e. on VMA file pointer and page permissions. For example, this could be done with LSM hooks that are triggered in the appropriate ioctl's and they could make the access control decision based on this information.
Unfortunately, there is ENCLS[EMODPE] that a running enclave can use to upgrade its permissions. If we do not limit
mmap()andmprotect(), enclave could upgrade its permissions by using EMODPE followed by an appropriatemprotect()call. This would be completely hidden from the kernel.Quoted 4 lines
Add 'mprotect' hook to
vm_ops, so that a callback can be implemeted for SGX that will ensure that {mmap, mprotect}() permissions do not surpass any of the original page permissions. This feature allows to maintain and refine sane access control for enclaves.Maybe I'm just being dense, but I still don't have a clear idea what function this hook serves.
Quoted 8 lines
I understand that SGX has an orthogonal set of page permissions to the normal x86 page tables. It needs these so that the OS can't play nasty tricks on the enclave, like removing read-only protections that provide hardening.
But, I still don't get the connection to
mprotect()and the x86 paging permissions. If the enclave's permissions are orthogonal, then why bother with this hook? Why does the OS view of the enclave's memory matter?
Consider SGX_IOC_ENCLAVE_ADD_PAGES. It essentially takes from MAC perpective two items:
1. Source represented as a VMA, and in the end as vm_file, i.e. the origin. This good place to anchor a security policy.
2. Permissions.
Since enclaves are dynamically constructed and do not go through execve() path but are dynamically loaded this is a good place hook make an LSM hook. Lets imagine that we have an LSM hook in place.
Now when enclave the enclave is loaded the LSM makes decisions based on its policy, permissions and the origin of the enclave file. After this decisions has been made it is sane to assume that the enclave could not be run with higher permissions, right?
Now what an adversary could do after the enclave has been loaded and initialized is to call EMODPE inside the enclave to upgrade permissions to some pages. After that mprotect() can be applied to use the new permissions.
The mprotect() callback keeps any higher permissions than originally accepted by the MAC policy unusable.
Also without any LSM hooks this allows to have a more privileged launcher process to build enclaves (e.g. have someone to own /dev/sgx/enclave) and send fd's to other processes to run them. And with that callback in place the launcher process can cap the permissions at build time.
That's where the indirect connection originates from. Should I write this user story to the commit message as an addition. Would that help?
For v39 I did make a change that noexec partitions are categorically discluded as an origin for enclaves. Before that it was that NX pages from such partitions are allowed but I thought that it is better simplify things since they are anyway complex. I'm 100% we can make that kind of support for noexec partitions later on, if anyone ever wants that anyway.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Sean ChristophersonQuoted 31 lines · Sean Christopherson
Quoted 10 lines · Andy Lutomirski
Quoted 8 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
Add vm_ops()->mprotect() for additional constraints for a VMA.
Intel Software Guard eXtensions (SGX) will use this callback to add two constraints:
Quoted 6 lines
Quoted 6 lines
1. Verify that the address range does not have holes: each page address must be filled with an enclave page.
2. Verify that VMA permissions won't surpass the permissions of any enclave page within the address range. Enclave cryptographically sealed permissions for each page address that set the upper limit for possible VMA permissions. Not respecting this can cause #GP's to be emitted.Side note, #GP is wrong. EPCM violations are #PFs. Skylake CPUs #GP, but that's technically an errata. But this isn't the real motivation, e.g. userspace can already trigger #GP/#PF by reading/writing a bad address, SGX simply adds another flavor.
Quoted 3 lines
It's been awhile since I looked at this. Can you remind us: is this just preventing userspace from shooting itself in the foot or is this something more important?
Something more important, it's used to prevent userspace from circumventing a noexec filesystem by loading code into an enclave, and to give the kernel the option of adding enclave specific LSM policies in the future.
Quoted 6 lines
The source file (if one exists) for the enclave is long gone when the enclave is actually mmap()'d and mprotect()'d. To enforce noexec, the requested permissions for a given page are snapshotted when the page is added to the enclave, i.e. when the enclave is built. Enclave pages that will be executable must originate from an a MAYEXEC VMA, e.g. the source page can't come from a noexec file system.
noexec check is done in __sgx_encl_add_page(), not in this callback. sgx_vma_mprotect() calls sgx_encl_may_map(), which iterates the addresses, checks that permissions are not surpassed and there are no holes.
Quoted 3 lines
The ->mprotect() hook allows SGX to reject
mprotect()if userspace is declaring permissions beyond what are allowed, e.g. trying to map an enclave page with EXEC permissions when the page was added to the enclave without EXEC.
For my ear this is tautological because if user space would use differing permissions, it would exactly soot itself in the foot.
What really should be documented is to answer why we consider an enclave mappable, if and only if the conditions are that I described about sgx_encl_may_map() functioning.
This is obviously part of the answer [*], i.e. with pharsing LSM hook requires an enclave to have a state, which is compatible with the mmap permissions and is fully populated.
The 2nd part of the answer is the answer to the question: why we want to feed LSM hooks enclaves exactly in this state.
What would you answer to that?
[*] https://lore.kernel.org/linux-sgx/[email protected]/T/#u
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 1 line · Jarkko Sakkinen
What really should be documented is to answer why we consider an enclave
~~
the (editing mistake)
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Jarkko SakkinenQuoted 2 lines · Jarkko Sakkinen
The 2nd part of the answer is the answer to the question: why we want to feed LSM hooks enclaves exactly in this state.
The question can be further refined as why: why this is the best possible set of substates to filter in?
"no holes" part is obvious as the consequence of not surpassing permissions of any of the pages in range, as you could otherwise break the state with ioctl(SGX_ENCLAVE_ADD_PAGES) with permssions that are below the mmap permissions.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Jarkko SakkinenQuoted 39 lines · Jarkko Sakkinen
Quoted 31 lines · Sean Christopherson
Quoted 10 lines · Andy Lutomirski
Quoted 8 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
Add vm_ops()->mprotect() for additional constraints for a VMA.
Intel Software Guard eXtensions (SGX) will use this callback to add two constraints:
Quoted 6 lines
Quoted 6 lines
1. Verify that the address range does not have holes: each page address must be filled with an enclave page.
2. Verify that VMA permissions won't surpass the permissions of any enclave page within the address range. Enclave cryptographically sealed permissions for each page address that set the upper limit for possible VMA permissions. Not respecting this can cause #GP's to be emitted.Side note, #GP is wrong. EPCM violations are #PFs. Skylake CPUs #GP, but that's technically an errata. But this isn't the real motivation, e.g. userspace can already trigger #GP/#PF by reading/writing a bad address, SGX simply adds another flavor.
Quoted 3 lines
It's been awhile since I looked at this. Can you remind us: is this just preventing userspace from shooting itself in the foot or is this something more important?
Something more important, it's used to prevent userspace from circumventing a noexec filesystem by loading code into an enclave, and to give the kernel the option of adding enclave specific LSM policies in the future.
Quoted 6 lines
The source file (if one exists) for the enclave is long gone when the enclave is actually mmap()'d and mprotect()'d. To enforce noexec, the requested permissions for a given page are snapshotted when the page is added to the enclave, i.e. when the enclave is built. Enclave pages that will be executable must originate from an a MAYEXEC VMA, e.g. the source page can't come from a noexec file system.
Quoted 4 lines
noexec check is done in
__sgx_encl_add_page(), not in this callback.sgx_vma_mprotect()callssgx_encl_may_map(), which iterates the addresses, checks that permissions are not surpassed and there are no holes.
Yes, that's what I said.
Quoted 6 lines
Quoted 3 lines
The ->mprotect() hook allows SGX to reject
mprotect()if userspace is declaring permissions beyond what are allowed, e.g. trying to map an enclave page with EXEC permissions when the page was added to the enclave without EXEC.For my ear this is tautological because if user space would use differing permissions, it would exactly soot itself in the foot.
Not on SGX2 hardware. The kernel can't disable ENCLU[MODPE].
Quoted 4 lines
What really should be documented is to answer why we consider an enclave mappable, if and only if the conditions are that I described about
sgx_encl_may_map()functioning.
Quoted 8 lines
This is obviously part of the answer [*], i.e. with pharsing LSM hook requires an enclave to have a state, which is compatible with the mmap permissions and is fully populated.
The 2nd part of the answer is the answer to the question: why we want to feed LSM hooks enclaves exactly in this state.
What would you answer to that?
I would copy-paste the part of the response that was snipped...
Quoted 3 lines
[*] https://lore.kernel.org/linux-sgx/[email protected]/T/#u
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 12 lines · Sean Christopherson
Quoted 11 lines · Jarkko Sakkinen
Quoted 9 lines · Sean Christopherson
a noexec filesystem by loading code into an enclave, and to give the kernel the option of adding enclave specific LSM policies in the future.
The source file (if one exists) for the enclave is long gone when the enclave is actually mmap()'d and mprotect()'d. To enforce noexec, the requested permissions for a given page are snapshotted when the page is added to the enclave, i.e. when the enclave is built. Enclave pages that will be executable must originate from an a MAYEXEC VMA, e.g. the source page can't come from a noexec file system.
Quoted 6 lines
Quoted 4 lines
noexec check is done in
__sgx_encl_add_page(), not in this callback.sgx_vma_mprotect()callssgx_encl_may_map(), which iterates the addresses, checks that permissions are not surpassed and there are no holes.Yes, that's what I said.
sgx_encl_add_page() will remove such page. The callback does not interact with this process as such pages never get to the enclave.
Quoted 1 line
I would copy-paste the part of the response that was snipped...
I do agree with the main conclusions but it contains also things that I do not see relating that much, like noexec partitions. It goes too far in detail what will LSM's end up doing. I absolutely do not want to forecast too far how LSM hooks would work.
Since we do not have ioctl's for EMODPE and such, I see EMODPE as the only reason for doing this right now. Otherwise, we are in trouble with any possible LSM callbacks. For any sort of access control decision, things decided must stick.
I would add something like this to the commit message largely based on your text:
"SGX stores the permissions for each page when they are first added, and will implement this callback to check that mmap() or mprotect() does not surpass these permissions in the requested address range.
This is done to prevent using EMODPE upgrading permissions of a page after mmap() or mprotect() has been done, which would prevent any sort of LSM callbacks to be implemented later on because the access control decision could deprecate."
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 18 lines · Jarkko Sakkinen
Quoted 17 lines · Sean Christopherson
Quoted 15 lines · Jarkko Sakkinen
Quoted 9 lines · Sean Christopherson
a noexec filesystem by loading code into an enclave, and to give the kernel the option of adding enclave specific LSM policies in the future.
The source file (if one exists) for the enclave is long gone when the enclave is actually mmap()'d and mprotect()'d. To enforce noexec, the requested permissions for a given page are snapshotted when the page is added to the enclave, i.e. when the enclave is built. Enclave pages that will be executable must originate from an a MAYEXEC VMA, e.g. the source page can't come from a noexec file system.
noexec check is done in
__sgx_encl_add_page(), not in this callback.sgx_vma_mprotect()callssgx_encl_may_map(), which iterates the addresses, checks that permissions are not surpassed and there are no holes.
Quoted 4 lines
Quoted 1 line
Yes, that's what I said.
sgx_encl_add_page()will remove such page. The callback does not interact with this process as such pages never get to the enclave.
I think we're in violent agreement, mostly.
Userspace can add the page without EXEC permissions in the EPCM, and thus avoid the noexec/VM_MAYEXEC check. The enclave can then do EMODPE to gain EXEC permissions in the EPMC. Without the ->mprotect() hook, we wouldn't be able to detect/prevent such shenanigans.
Quoted 4 lines
Quoted 1 line
I would copy-paste the part of the response that was snipped...
I do agree with the main conclusions but it contains also things that I do not see relating that much, like noexec partitions.
As above, this does directly related to noexec/VM_MAYEXEC.
Quoted 2 lines
It goes too far in detail what will LSM's end up doing. I absolutely do not want to forecast too far how LSM hooks would work.
That's fine, I was responding to Andy's question, not intending to write a changelog.
Quoted 4 lines
Since we do not have ioctl's for EMODPE and such, I see EMODPE as the only reason for doing this right now. Otherwise, we are in trouble with any possible LSM callbacks. For any sort of access control decision, things decided must stick.
Yes, again, violent agreement :-).
Quoted 11 lines
I would add something like this to the commit message largely based on your text:
"SGX stores the permissions for each page when they are first added, and will implement this callback to check that
mmap()ormprotect()does not surpass these permissions in the requested address range.This is done to prevent using EMODPE upgrading permissions of a page after
mmap()ormprotect()has been done, which would prevent any sort of LSM callbacks to be implemented later on because the access control decision could deprecate."
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 24 lines · Sean Christopherson
Quoted 18 lines · Jarkko Sakkinen
Quoted 17 lines · Sean Christopherson
Quoted 15 lines · Jarkko Sakkinen
Quoted 9 lines · Sean Christopherson
a noexec filesystem by loading code into an enclave, and to give the kernel the option of adding enclave specific LSM policies in the future.
The source file (if one exists) for the enclave is long gone when the enclave is actually mmap()'d and mprotect()'d. To enforce noexec, the requested permissions for a given page are snapshotted when the page is added to the enclave, i.e. when the enclave is built. Enclave pages that will be executable must originate from an a MAYEXEC VMA, e.g. the source page can't come from a noexec file system.
noexec check is done in
__sgx_encl_add_page(), not in this callback.sgx_vma_mprotect()callssgx_encl_may_map(), which iterates the addresses, checks that permissions are not surpassed and there are no holes.Quoted 4 lines
Quoted 1 line
Yes, that's what I said.
sgx_encl_add_page()will remove such page. The callback does not interact with this process as such pages never get to the enclave.
Quoted 6 lines
I think we're in violent agreement, mostly.
Userspace can add the page without EXEC permissions in the EPCM, and thus avoid the noexec/VM_MAYEXEC check. The enclave can then do EMODPE to gain EXEC permissions in the EPMC. Without the ->mprotect() hook, we wouldn't be able to detect/prevent such shenanigans.
Right, the VM_MAYEXEC in the code is nested under VM_EXEC check.
I'm only wondering why not block noexec completely with any permissions, i.e. why not just have unconditional VM_MAYEXEC check?
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 32 lines · Jarkko Sakkinen
Quoted 24 lines · Sean Christopherson
Quoted 18 lines · Jarkko Sakkinen
Quoted 17 lines · Sean Christopherson
Quoted 15 lines · Jarkko Sakkinen
Quoted 9 lines · Sean Christopherson
a noexec filesystem by loading code into an enclave, and to give the kernel the option of adding enclave specific LSM policies in the future.
The source file (if one exists) for the enclave is long gone when the enclave is actually mmap()'d and mprotect()'d. To enforce noexec, the requested permissions for a given page are snapshotted when the page is added to the enclave, i.e. when the enclave is built. Enclave pages that will be executable must originate from an a MAYEXEC VMA, e.g. the source page can't come from a noexec file system.
noexec check is done in
__sgx_encl_add_page(), not in this callback.sgx_vma_mprotect()callssgx_encl_may_map(), which iterates the addresses, checks that permissions are not surpassed and there are no holes.Quoted 4 lines
Quoted 1 line
Yes, that's what I said.
sgx_encl_add_page()will remove such page. The callback does not interact with this process as such pages never get to the enclave.Quoted 6 lines
I think we're in violent agreement, mostly.
Userspace can add the page without EXEC permissions in the EPCM, and thus avoid the noexec/VM_MAYEXEC check. The enclave can then do EMODPE to gain EXEC permissions in the EPMC. Without the ->mprotect() hook, we wouldn't be able to detect/prevent such shenanigans.
Quoted 4 lines
Right, the
VM_MAYEXECin the code is nested underVM_EXECcheck.I'm only wondering why not block noexec completely with any permissions, i.e. why not just have unconditional
VM_MAYEXECcheck?
I.e. why not this:
static int __sgx_encl_add_page(struct sgx_encl *encl,
struct sgx_encl_page *encl_page, struct sgx_epc_page *epc_page, struct sgx_secinfo *secinfo, unsigned long src)
{
struct sgx_pageinfo pginfo;
struct vm_area_struct *vma;
struct page *src_page;
int ret;
vma = find_vma(current->mm, src);
if (!vma)
return -EFAULT;
if (!(vma->vm_flags & VM_MAYEXEC))
return -EACCES;I'm not seeing the reason for "partial support" for noexec partitions.
If there is a good reason, fine, let's just then document it.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 31 lines · Jarkko Sakkinen
Quoted 10 lines · Jarkko Sakkinen
Quoted 4 lines · Sean Christopherson
Userspace can add the page without EXEC permissions in the EPCM, and thus avoid the noexec/VM_MAYEXEC check. The enclave can then do EMODPE to gain EXEC permissions in the EPMC. Without the ->mprotect() hook, we wouldn't be able to detect/prevent such shenanigans.
Right, the
VM_MAYEXECin the code is nested underVM_EXECcheck.I'm only wondering why not block noexec completely with any permissions, i.e. why not just have unconditional
VM_MAYEXECcheck?I.e. why not this:
static int __sgx_encl_add_page(struct
sgx_encl*encl,
structsgx_encl_page*encl_page, structsgx_epc_page*epc_page, structsgx_secinfo*secinfo, unsigned long src){ struct sgx_pageinfo pginfo; struct vm_area_struct *vma; struct page *src_page; int ret; vma = find_vma(current->mm, src); if (!vma) return -EFAULT; if (!(vma->vm_flags & VM_MAYEXEC)) return -EACCES;
Quoted 3 lines
I'm not seeing the reason for "partial support" for noexec partitions.
If there is a good reason, fine, let's just then document it.
There are scenarios I can contrive, e.g. loading an enclave from a noexec filesystem without having to copy the entire enclave to anon memory, or loading a data payload from a noexec FS.
They're definitely contrived scenarios, but given that we also want the ->mprotect() hook/behavior for potential LSM interaction, supporting said contrived scenarios costs is "free".
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 36 lines · Sean Christopherson
Quoted 31 lines · Jarkko Sakkinen
Quoted 10 lines · Jarkko Sakkinen
Quoted 4 lines · Sean Christopherson
Userspace can add the page without EXEC permissions in the EPCM, and thus avoid the noexec/VM_MAYEXEC check. The enclave can then do EMODPE to gain EXEC permissions in the EPMC. Without the ->mprotect() hook, we wouldn't be able to detect/prevent such shenanigans.
Right, the
VM_MAYEXECin the code is nested underVM_EXECcheck.I'm only wondering why not block noexec completely with any permissions, i.e. why not just have unconditional
VM_MAYEXECcheck?I.e. why not this:
static int __sgx_encl_add_page(struct
sgx_encl*encl,
structsgx_encl_page*encl_page, structsgx_epc_page*epc_page, structsgx_secinfo*secinfo, unsigned long src){ struct sgx_pageinfo pginfo; struct vm_area_struct *vma; struct page *src_page; int ret; vma = find_vma(current->mm, src); if (!vma) return -EFAULT; if (!(vma->vm_flags & VM_MAYEXEC)) return -EACCES;Quoted 3 lines
I'm not seeing the reason for "partial support" for noexec partitions.
If there is a good reason, fine, let's just then document it.
Quoted 7 lines
There are scenarios I can contrive, e.g. loading an enclave from a noexec filesystem without having to copy the entire enclave to anon memory, or loading a data payload from a noexec FS.
They're definitely contrived scenarios, but given that we also want the ->mprotect() hook/behavior for potential LSM interaction, supporting said contrived scenarios costs is "free".
For me this has caused months of confusion and misunderstanding of this feature. I only recently realized that "oh, right, we invented this".
They are contrived scenarios enough that they should be considered when the workloads hit.
Either we fully support noexec or not at all. Any "partial" thing is a two edged sword: it can bring some robustness with the price of complexity and possible unknown uknown scenarios where they might become API issue.
I rather think later on how to extend API in some way to enable such contrivid scenarios rather than worrying about how this could be abused.
The whole SGX is complex beast already so lets not add any extra when there is no a hard requirement to do so.
I'll categorically deny noexec in the next patch set version.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 64 lines · Jarkko Sakkinen
Quoted 40 lines · Sean Christopherson
Quoted 3 lines · Jarkko Sakkinen
Quoted 2 lines · Jarkko Sakkinen
Quoted 1 line · Sean Christopherson
Userspace can add the page without EXEC permissions in the EPCM,
and thus
Quoted 1 line
Quoted 1 line
Quoted 1 line
avoid the noexec/VM_MAYEXEC check. The enclave can then do
EMODPE to gain
Quoted 1 line
Quoted 1 line
Quoted 1 line
EXEC permissions in the EPMC. Without the ->mprotect() hook, we
wouldn't
Quoted 5 lines
Quoted 5 lines
Quoted 1 line
be able to detect/prevent such shenanigans.
Right, the
VM_MAYEXECin the code is nested underVM_EXECcheck.I'm only wondering why not block noexec completely with any
permissions,
Quoted 21 lines
Quoted 1 line
i.e. why not just have unconditional
VM_MAYEXECcheck?I.e. why not this:
static int __sgx_encl_add_page(struct
sgx_encl*encl,
structsgx_encl_page*encl_page, structsgx_epc_page*epc_page, structsgx_secinfo*secinfo, unsigned long src){ struct sgx_pageinfo pginfo; struct vm_area_struct *vma; struct page *src_page; int ret; vma = find_vma(current->mm, src); if (!vma) return -EFAULT; if (!(vma->vm_flags & VM_MAYEXEC)) return -EACCES;Quoted 3 lines
I'm not seeing the reason for "partial support" for noexec partitions.
If there is a good reason, fine, let's just then document it.
Quoted 8 lines
There are scenarios I can contrive, e.g. loading an enclave from a noexec filesystem without having to copy the entire enclave to anon memory, or loading a data payload from a noexec FS.
They're definitely contrived scenarios, but given that we also want the ->mprotect() hook/behavior for potential LSM interaction, supporting said
contrived scenarios costs is "free".For me this has caused months of confusion and misunderstanding of this feature. I only recently realized that "oh, right, we invented this".
They are contrived scenarios enough that they should be considered when the workloads hit.
Either we fully support noexec or not at all. Any "partial" thing is a two edged sword: it can bring some robustness with the price of complexity and possible unknown uknown scenarios where they might become API issue.
I rather think later on how to extend API in some way to enable such contrivid scenarios rather than worrying about how this could be abused.
Quoted 6 lines
The whole SGX is complex beast already so lets not add any extra when there is no a hard requirement to do so.
I'll categorically deny noexec in the next patch set version.
/Jarkko
There are use cases supported currently in which enclave binary is received via IPC/RPC and held in buffers before EADD. Denying noexec altogether would break those, right?
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 9 lines · Haitao Huang
Quoted 3 lines · Jarkko Sakkinen
I'll categorically deny noexec in the next patch set version.
/Jarkko
There are use cases supported currently in which enclave binary is received via IPC/RPC and held in buffers before EADD. Denying noexec altogether would break those, right?
No. noexec only applies to file-backed VMAs, what you're describing is loading an enclave from an anon VMA, which will still have VM_MAYEXEC.
I believe you're thinking of SELinux's EXECMEM, which is required to execute from anonymous memory, and which we talked about (more than once) applying to SGX enclaves.
That being said, I still dislike the idea of requiring VM_MAYEXEC, it's a hack that doesn't really buy us much, if anything.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 11 lines · Sean Christopherson
Quoted 8 lines · Haitao Huang
Quoted 3 lines · Jarkko Sakkinen
I'll categorically deny noexec in the next patch set version.
/Jarkko
There are use cases supported currently in which enclave binary is received via IPC/RPC and held in buffers before EADD. Denying noexec altogether would break those, right?
No. noexec only applies to file-backed VMAs, what you're describing is loading an enclave from an anon VMA, which will still have
VM_MAYEXEC.
Maybe I'm just stupid, but I still don't get the scenario that's being thwarted or why it is valuable. The SDM is worthless on what EMODPE does or what its restrictions are.
In pseudo-C, it's something logically like this for the "nice" case:
ptr = mmap("/some/executable", PROT_EXEC);
ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size);
mmap(sgx_fd);
EENTER;And we're trying to thwart:
ptr = mmap("/mnt/noexec/file", PROT_READ);
ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size);
mmap(sgx_fd);
EENTER;because that loads data into the enclave which is executable but which was not executable normally. But, we're allowing this from anonymous memory, so this would seem to work:
ptr = mmap("/mnt/noexec/file", PROT_READ);
buffer = malloc(PAGE_SIZE);
memcpy(buffer, ptr, PAGE_SIZE);
// need mprotect(buf, PROT_EXEC)???
ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, buffer, size);
mmap(sgx_fd);
EENTER;and give the same result. What am I missing?
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 43 lines · Dave Hansen
Quoted 11 lines · Sean Christopherson
Quoted 8 lines · Haitao Huang
Quoted 3 lines · Jarkko Sakkinen
I'll categorically deny noexec in the next patch set version.
/Jarkko
There are use cases supported currently in which enclave binary is received via IPC/RPC and held in buffers before EADD. Denying noexec altogether would break those, right?
No. noexec only applies to file-backed VMAs, what you're describing is loading an enclave from an anon VMA, which will still have
VM_MAYEXEC.Maybe I'm just stupid, but I still don't get the scenario that's being thwarted or why it is valuable. The SDM is worthless on what EMODPE does or what its restrictions are.
In pseudo-C, it's something logically like this for the "nice" case:
ptr = mmap("/some/executable", PROT_EXEC); ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size); mmap(sgx_fd); EENTER;And we're trying to thwart:
ptr = mmap("/mnt/noexec/file", PROT_READ); ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size); mmap(sgx_fd); EENTER;because that loads data into the enclave which is executable but which was not executable normally. But, we're allowing this from anonymous memory, so this would seem to work:
ptr = mmap("/mnt/noexec/file", PROT_READ); buffer = malloc(PAGE_SIZE); memcpy(buffer, ptr, PAGE_SIZE); // need mprotect(buf, PROT_EXEC)??? ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, buffer, size); mmap(sgx_fd); EENTER;
Quoted 1 line
and give the same result. What am I missing?
The last example, where the enclave is copied to a buffer, is out of scope for noexec. But, it is in scope for LSMs, e.g. for this last example, we could add an LSM upcall so that SELinux could require PROCESS_EXECMEM (or an SGX specific equivalent).
The ->mprotect() hook gives us direct line of sight to such LSM upcalls, e.g. I have fully working code that implements the LSM integration. LSM support isn't in this series because the only thing everyone could agree on in terms of an LSM implementation is that it wasn't needed for initial upstreaming :-)
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 26 lines · Sean Christopherson
Quoted 26 lines
In pseudo-C, it's something logically like this for the "nice" case:
ptr = mmap("/some/executable", PROT_EXEC); ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size); mmap(sgx_fd); EENTER;And we're trying to thwart:
ptr = mmap("/mnt/noexec/file", PROT_READ); ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size); mmap(sgx_fd); EENTER;because that loads data into the enclave which is executable but which was not executable normally. But, we're allowing this from anonymous memory, so this would seem to work:
ptr = mmap("/mnt/noexec/file", PROT_READ); buffer = malloc(PAGE_SIZE); memcpy(buffer, ptr, PAGE_SIZE); // need mprotect(buf, PROT_EXEC)??? ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, buffer, size); mmap(sgx_fd); EENTER;
Quoted 5 lines
Quoted 1 line
and give the same result. What am I missing?
The last example, where the enclave is copied to a buffer, is out of scope for noexec. But, it is in scope for LSMs, e.g. for this last example, we could add an LSM upcall so that SELinux could require
PROCESS_EXECMEM(or an SGX specific equivalent).
Why don't we just declare enclave memory as "out of scope for noexec" in the same way that anonymous memory is, and just discard this patch? That doesn't seem too much of a stretch.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 28 lines · Dave Hansen
Quoted 27 lines · Sean Christopherson
Quoted 27 lines
In pseudo-C, it's something logically like this for the "nice" case:
ptr = mmap("/some/executable", PROT_EXEC); ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size); mmap(sgx_fd); EENTER;And we're trying to thwart:
ptr = mmap("/mnt/noexec/file", PROT_READ); ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size); mmap(sgx_fd); EENTER;because that loads data into the enclave which is executable but which was not executable normally. But, we're allowing this from anonymous memory, so this would seem to work:
ptr = mmap("/mnt/noexec/file", PROT_READ); buffer = malloc(PAGE_SIZE); memcpy(buffer, ptr, PAGE_SIZE); // need mprotect(buf, PROT_EXEC)??? ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, buffer, size); mmap(sgx_fd); EENTER;and give the same result. What am I missing?
Quoted 8 lines
Quoted 4 lines
The last example, where the enclave is copied to a buffer, is out of scope for noexec. But, it is in scope for LSMs, e.g. for this last example, we could add an LSM upcall so that SELinux could require
PROCESS_EXECMEM(or an SGX specific equivalent).Why don't we just declare enclave memory as "out of scope for noexec" in the same way that anonymous memory is, and just discard this patch? That doesn't seem too much of a stretch.
Because we lose line of sight to LSM support. Without enforcing "declare perms at load time" in the initial series, we would create an ABI where userspace could load an enclave page with only READ permissions and then map the enclave with whatever permissions it wants, without any convenient way for SGX to call into the LSM.
Retroactively enforcing permissions at load time would break the ABI, or at least yield different behavior based on the mere existence of LSMs, e.g. if LSMs are supported, suddenly the ADD_PAGES w/ READ -> mmap(RWX) flow breaks, even if there is no LSM policy denying that behavior.
Enforcing LSM policies using the existing mmap()/mprotect() hooks doesn't work well because the only information available is a fd pointing at /dev/sgx/enclave, which is largely useless because /dev/sgx/enclave must be map SHARED w/ RWX to run an enclave. We explored things like grabbing a reference to the source file for later verification, but that means pinning files for the entire lifetime of an enclave.
Enforcing noexec was an easy/obvious addition since we need 99% of the code for potential LSM support anyways.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
On 9/24/20 1:25 PM, Sean Christopherson wrote:
...
Quoted 9 lines
Quoted 3 lines
Why don't we just declare enclave memory as "out of scope for noexec" in the same way that anonymous memory is, and just discard this patch? That doesn't seem too much of a stretch.
Because we lose line of sight to LSM support. Without enforcing "declare perms at load time" in the initial series, we would create an ABI where userspace could load an enclave page with only READ permissions and then map the enclave with whatever permissions it wants, without any convenient way for SGX to call into the LSM.
This argument holds no water for me. LSMs are all about taking what would otherwise be perfectly acceptable behavior and breaking them in the name of security. They fundamentally break applications that used to work just fine and also did totally sane things.
Quoted 4 lines
Retroactively enforcing permissions at load time would break the ABI, or at least yield different behavior based on the mere existence of LSMs, e.g. if LSMs are supported, suddenly the
ADD_PAGESw/ READ -> mmap(RWX) flow breaks, even if there is no LSM policy denying that behavior.
I'm a security dummy. All I know is that when I see something like this:
if (security_vm_enough_memory_mm(mm, grow)) ...
I know to ignore it because I like my systems to boot and I'm not using those hooks. :)
How could the mere presence of an LSM change the behavior of one of these hooks? Don't they have to actually hook into the specific place and actively go trying to change the behavior at that site?
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 12 lines · Dave Hansen
On 9/24/20 1:25 PM, Sean Christopherson wrote:
...Quoted 9 lines
Quoted 3 lines
Why don't we just declare enclave memory as "out of scope for noexec" in the same way that anonymous memory is, and just discard this patch? That doesn't seem too much of a stretch.
Because we lose line of sight to LSM support. Without enforcing "declare perms at load time" in the initial series, we would create an ABI where userspace could load an enclave page with only READ permissions and then map the enclave with whatever permissions it wants, without any convenient way for SGX to call into the LSM.
Quoted 4 lines
This argument holds no water for me. LSMs are all about taking what would otherwise be perfectly acceptable behavior and breaking them in the name of security. They fundamentally break applications that used to work just fine and also did totally sane things.
Well, I see that having this gives a better assets for a separate loader. It can load and initialize the enclave and a consumer can then use it but cannot override its permissions.
I don't even remember exactly how SELinux permissions played together. I don't think it is required as an argument for this patch.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Dave HansenQuoted 12 lines · Dave Hansen
On 9/24/20 1:25 PM, Sean Christopherson wrote:
...Quoted 9 lines
Quoted 3 lines
Why don't we just declare enclave memory as "out of scope for noexec" in the same way that anonymous memory is, and just discard this patch? That doesn't seem too much of a stretch.
Because we lose line of sight to LSM support. Without enforcing "declare perms at load time" in the initial series, we would create an ABI where userspace could load an enclave page with only READ permissions and then map the enclave with whatever permissions it wants, without any convenient way for SGX to call into the LSM.
Quoted 4 lines
This argument holds no water for me. LSMs are all about taking what would otherwise be perfectly acceptable behavior and breaking them in the name of security. They fundamentally break applications that used to work just fine and also did totally sane things.
It's not about LSMs breaking things, they can obviously do that without any help. The concern is that, unless we lay the groundwork now, adding support for LSMs in the future will break existing applications or create an unholy mess of an ABI.
If we want to support LSM policy for enclave page permissions, checking LSM policies at load time and hooking mprotect() to enforce the policy at run time is by far the cleanest solution of the many ideas we discussed.
The problem is that enforcing permissions via mprotect() needs to be done unconditionally, otherwise we end up with weird behavior where the existence of an LSM will change what is/isn't allowed, even if the LSM(s) has no SGX policy whatsover.
And on the flip side, enforcing permissions unconditionally will break backwards compatibility.
I'm a-ok if we want to kill off the ->mprotect() hook, so long as we acknowledge that in doing so we are likely throwing in the towel on supporting LSM policies for enclave page permissions.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 4 lines · Sean Christopherson
The problem is that enforcing permissions via
mprotect()needs to be done unconditionally, otherwise we end up with weird behavior where the existence of an LSM will change what is/isn't allowed, even if the LSM(s) has no SGX policy whatsover.
Could we make this a bit less abstract, please?
Could someone point to code or another examples that demonstrates how the mere existence of an LSM will change what is/isn't allowed?
I can't seem to wrap my head around it as-is.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 6 lines · Dave Hansen
Quoted 4 lines · Sean Christopherson
The problem is that enforcing permissions via
mprotect()needs to be done unconditionally, otherwise we end up with weird behavior where the existence of an LSM will change what is/isn't allowed, even if the LSM(s) has no SGX policy whatsover.
Quoted 6 lines
Could we make this a bit less abstract, please?
Could someone point to code or another examples that demonstrates how the mere existence of an LSM will change what is/isn't allowed?
I can't seem to wrap my head around it as-is.
Without pre-declared permissions, loading and running an enclave would be:
ptr = malloc(size);
memcpy(ptr, evil_shenanigans, size);
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, ptr, size);
...
ioctl(sgx_fd, ENCLAVE_INIT);
enclave = mmap(sgx_fd, ... PROT_READ);
mprotect(enclave, ..., PROT_READ | PROT_EXEC);
EENTER;With the existing security_file_mprotect(), an LSM will see:
vma->vm_file ~= /dev/sgx/enclave
prot = PROT_READ | PROT_EXEC
From a policy perspective, the LSM can't do much because every enclave is backed by /dev/sgx/enclave, and all enclaves need READ and EXEC perms, i.e. a policy can only deny all enclaves or allow enclaves.
The solution we came up with is to require userspace to declare the intended permissions at build time so that an LSM hook can be invoked when the source VMA is availble, e.g. in this example, the LSM would see that the process is loading executable code into an enclave from an anonymous VMA:
ptr = malloc(size);
memcpy(ptr, evil_shenanigans, size);
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, SGX_PROT_READ | SGX_PROT_EXEC, ptr, size);
{
ret = security_enclave_load(ptr, prot);
if (ret)
return ret;
enclave_page->declared_prot = prot;
}
...
ioctl(sgx_fd, ENCLAVE_INIT);and then enforce the declared perms via ->mprotect()
enclave = mmap(sgx_fd, ... PROT_READ);
mprotect(enclave, ..., PROT_READ | PROT_EXEC);
|
-> sgx_mprotect(...)
{
if (~enclave_page->declared_prot & prot)
return -EACCES;
}
EENTER;So the above would be allowed, but this would fail (irrespective of LSM behavior):
ptr = malloc(size);
memcpy(ptr, evil_shenanigans, size);
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, SGX_PROT_READ, ptr, size);
...
ioctl(sgx_fd, ENCLAVE_INIT);
enclave = mmap(sgx_fd, ... PROT_READ);
mprotect(enclave, ..., PROT_READ | PROT_EXEC);My concern is that if we merge this
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, SGX_PROT_READ | SGX_PROT_EXEC, ptr, size);
without ->mprotect(), we can't actually enforce the declared protections. And if we drop the field altogether:
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, ptr, size);
then we can't implement security_enclave_load().
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Thanks for the walkthrough. The thing that clicked for me seeing those examples was how the earlier ioctl(ADD_PAGE) is "bound" to later enforcement actions at enclave PTE creation time.
Quoted 10 lines · Sean Christopherson
My concern is that if we merge this
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, SGX_PROT_READ | SGX_PROT_EXEC, ptr, size);without ->mprotect(), we can't actually enforce the declared protections. And if we drop the field altogether:
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, ptr, size);then we can't implement
security_enclave_load().
To me, it's perfectly OK to have parts of the ABI which are unused. It sure makes them harder to test if there are no actual users in the code, but if it solves a real problem with the ABI, I'm fine with it.
Let's see if I can put all the pieces together.
Background:
1. SGX enclave pages are populated with data by copying data to them from normal memory via: ioctl(sgx_fd, ENCLAVE_ADD_PAGE, src_ptr...);
2. We want to be able to restrict those normal memory data sources. For instance, before copying data to an executable enclave page, we might ensure that the source is executable.
3. Enclave page permissions are dynamic just like normal permissions and can be adjusted at runtime with mprotect() (along with a corresponding special instruction inside the enclave)
4. The original data source may have have long since vanished at the time when enclave page permission are established (mmap() or mprotect())
Solution:
The solution is to force enclaves creators to declare their intent up front to ioctl(ENCLAVE_ADD_PAGE). This intent can me immediately compared to the source data mapping (and rejected if necessary). It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or the enclave itself are consistent with the earlier declared permissions.
Essentially, this means that whenever the kernel is asked to change an enclave PTE, it needs to ensure the change is consistent with that stashed intent. There is an existing vm_ops->mmap() hook which allows SGX to do that for mmap(). However, there is no ->mprotect() hook. Add a vm_ops->mprotect() hook so that mprotect() operations which are inconsistent with any page's stashed intent can be rejected by the driver.
Implications:
However, there is currently no implementation of the intent checks at the time of ioctl(ENCLAVE_ADD_PAGE). That means that the intent argument (SGX_PROT_*) is currently unused.
--
Is that all correct? Did I miss anything?
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 46 lines · Dave Hansen
Thanks for the walkthrough. The thing that clicked for me seeing those examples was how the earlier
ioctl(ADD_PAGE)is "bound" to later enforcement actions at enclave PTE creation time.Quoted 10 lines · Sean Christopherson
My concern is that if we merge this
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, SGX_PROT_READ | SGX_PROT_EXEC, ptr, size);without ->mprotect(), we can't actually enforce the declared protections. And if we drop the field altogether:
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, ptr, size);then we can't implement
security_enclave_load().To me, it's perfectly OK to have parts of the ABI which are unused. It sure makes them harder to test if there are no actual users in the code, but if it solves a real problem with the ABI, I'm fine with it.
Let's see if I can put all the pieces together.
Background:
1. SGX enclave pages are populated with data by copying data to them from normal memory via:
ioctl(sgx_fd, ENCLAVE_ADD_PAGE, src_ptr...);
2. We want to be able to restrict those normal memory data sources. For instance, before copying data to an executable enclave page, we might ensure that the source is executable.
3. Enclave page permissions are dynamic just like normal permissions and can be adjusted at runtime withmprotect()(along with a corresponding special instruction inside the enclave)
4. The original data source may have have long since vanished at the time when enclave page permission are established (mmap()ormprotect())Solution:
The solution is to force enclaves creators to declare their intent up front to
ioctl(ENCLAVE_ADD_PAGE). This intent can me immediately compared to the source data mapping (and rejected if necessary). It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or the enclave itself are consistent with the earlier declared permissions.
Quoted 6 lines
Essentially, this means that whenever the kernel is asked to change an enclave PTE, it needs to ensure the change is consistent with that stashed intent. There is an existing
vm_ops->mmap()hook which allows SGX to do that formmap(). However, there is no ->mprotect() hook. Add avm_ops->mprotect()hook so thatmprotect()operations which are inconsistent with any page's stashed intent can be rejected by the driver.
Yes to all of the above.
Quoted 4 lines
Implications:
However, there is currently no implementation of the intent checks at the time of
ioctl(ENCLAVE_ADD_PAGE).
Correct.
Quoted 1 line
That means that the intent argument (SGX_PROT_*) is currently unused.
No, the intent argument is used (eventually) by SGX's ->mprotect() implementation, i.e. sgx_mprotect() enforces that the actual protections are a subset of the declared/intended protections.
If ->mprotect() is not merged, then it yes, it will be unused. And therein lies the problem as the kernel can't start using/enforcing the intent without breaking userspace. E.g. an enclave loaded with SGX_PROT_READ but mprotect()'d with PROT_READ | PROT_EXEC would break if sgx_mprotect() came along.
One way to avoid introducing ->mprotect() would be to require all enclaves to declare all pages with READ|WRITE|EXEC. Then we could drop sgx_mprotect() since the mprotect() permissions are guaranteed to be a subset of the declared permissions. That would have the added bonus of eliminating the per-page checks in sgx_mmap()/sgx_mprotect(), though I've no idea if that is a meaningful optmization or it's lost in the noise.
The big downside of requiring READ|WRITE|EXEC is that it will make life hell for a LSM policy owner if they ever want to apply EXECMEM or EXECMOD style restritions on enclaves, i.e. if SELinux folks want to add security_enclave_load().
I find that I'm more or less ok with that approach, in no small part because introducing security_enclave_load() might be a pretty big "if", e.g. security folks may decide that they'd rather allow/deny enclaves based on the measurement or signer of the enclave and eschew per-page checks entirely.
Quoted 3 lines
--
Is that all correct? Did I miss anything?
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 6 lines · Sean Christopherson
Quoted 1 line
That means that the intent argument (SGX_PROT_*) is currently unused.
No, the intent argument is used (eventually) by SGX's ->mprotect() implementation, i.e.
sgx_mprotect()enforces that the actual protections are a subset of the declared/intended protections.If ->mprotect() is not merged, then it yes, it will be unused.
OK, I think I've got it.
I think I'm OK with adding ->mprotect(). As long as folks buy into the argument that intent needs to be checked at mmap() time, they obviously need to be checked at mprotect() too.
Jarkko, if you want to try and rewrite the changelog, capturing the discussion here and reply, I think I can ack the resulting patch. I don't know if that will satisfy the request from Boris from an ack from a "mm person", but we can at least start there. :)
Please be judicious in what you include in the changelog. There's been a lot of detritus in them. Let's keep it as short, sweet, simple and on topic as we can.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 13 lines · Dave Hansen
Quoted 6 lines · Sean Christopherson
Quoted 1 line
That means that the intent argument (SGX_PROT_*) is currently unused.
No, the intent argument is used (eventually) by SGX's ->mprotect() implementation, i.e.
sgx_mprotect()enforces that the actual protections are a subset of the declared/intended protections.If ->mprotect() is not merged, then it yes, it will be unused.
OK, I think I've got it.
Quoted 8 lines
I think I'm OK with adding ->mprotect(). As long as folks buy into the argument that intent needs to be checked at
mmap()time, they obviously need to be checked atmprotect()too.Jarkko, if you want to try and rewrite the changelog, capturing the discussion here and reply, I think I can ack the resulting patch. I don't know if that will satisfy the request from Boris from an ack from a "mm person", but we can at least start there. :)
I think I agree. ->mprotect seems reasonable to me.
FWIW, I don’t think I should ack this particular thing — it was, to a decent extent, my suggestion in the first place, so I’m biased. I think it turned into something reasonable, and the ->mprotect mechanism seems easily supportable and plausibly useful for other purposes down the road.
Quoted 4 lines
Please be judicious in what you include in the changelog. There's been a lot of detritus in them. Let's keep it as short, sweet, simple and on topic as we can.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Dave HansenQuoted 10 lines · Dave Hansen
Quoted 6 lines · Sean Christopherson
Quoted 1 line
That means that the intent argument (SGX_PROT_*) is currently unused.
No, the intent argument is used (eventually) by SGX's ->mprotect() implementation, i.e.
sgx_mprotect()enforces that the actual protections are a subset of the declared/intended protections.If ->mprotect() is not merged, then it yes, it will be unused.
OK, I think I've got it.
Quoted 8 lines
I think I'm OK with adding ->mprotect(). As long as folks buy into the argument that intent needs to be checked at
mmap()time, they obviously need to be checked atmprotect()too.Jarkko, if you want to try and rewrite the changelog, capturing the discussion here and reply, I think I can ack the resulting patch. I don't know if that will satisfy the request from Boris from an ack from a "mm person", but we can at least start there. :)
I think what it needs, based on what I've read, is the step by step description of the EMODPE scenarion without this callback and with it.
I think other important thing to underline is that an LSM or any other security measure can only do a sane decision when the enclave is loaded. At that point we know the source (vm_file).
I.e. when you are doing mmap() or mprotect() you don't have that information. The permissions kind of describe the contact made at that point of time.
Quoted 3 lines
Please be judicious in what you include in the changelog. There's been a lot of detritus in them. Let's keep it as short, sweet, simple and on topic as we can.
Of course.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 15 lines · Jarkko Sakkinen
Quoted 14 lines · Dave Hansen
Quoted 6 lines · Sean Christopherson
Quoted 1 line
That means that the intent argument (SGX_PROT_*) is currently unused.
No, the intent argument is used (eventually) by SGX's ->mprotect() implementation, i.e.
sgx_mprotect()enforces that the actual protections are a subset of the declared/intended protections.If ->mprotect() is not merged, then it yes, it will be unused.
OK, I think I've got it.
I think I'm OK with adding ->mprotect(). As long as folks buy into the argument that intent needs to be checked at
mmap()time, they obviously need to be checked atmprotect()too.
Quoted 7 lines
Quoted 4 lines
Jarkko, if you want to try and rewrite the changelog, capturing the discussion here and reply, I think I can ack the resulting patch. I don't know if that will satisfy the request from Boris from an ack from a "mm person", but we can at least start there. :)
I think what it needs, based on what I've read, is the step by step description of the EMODPE scenarion without this callback and with it.
EMODPE is virtually irrelevant for this whole thing. The x86 PTE permissions still specify the most restrictive permissions, which is what matters the most.
We care about the _worst_ the enclave can do, not what it imposes on itself on top of that.
Quoted 3 lines
I think other important thing to underline is that an LSM or any other security measure can only do a sane decision when the enclave is loaded. At that point we know the source (
vm_file).
Right, you know the source, but it can be anonymous or a file.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 24 lines · Dave Hansen
Quoted 15 lines · Jarkko Sakkinen
Quoted 14 lines · Dave Hansen
Quoted 6 lines · Sean Christopherson
Quoted 1 line
That means that the intent argument (SGX_PROT_*) is currently unused.
No, the intent argument is used (eventually) by SGX's ->mprotect() implementation, i.e.
sgx_mprotect()enforces that the actual protections are a subset of the declared/intended protections.If ->mprotect() is not merged, then it yes, it will be unused.
OK, I think I've got it.
I think I'm OK with adding ->mprotect(). As long as folks buy into the argument that intent needs to be checked at
mmap()time, they obviously need to be checked atmprotect()too.Quoted 7 lines
Quoted 4 lines
Jarkko, if you want to try and rewrite the changelog, capturing the discussion here and reply, I think I can ack the resulting patch. I don't know if that will satisfy the request from Boris from an ack from a "mm person", but we can at least start there. :)
I think what it needs, based on what I've read, is the step by step description of the EMODPE scenarion without this callback and with it.
Quoted 6 lines
EMODPE is virtually irrelevant for this whole thing. The x86 PTE permissions still specify the most restrictive permissions, which is what matters the most.
We care about the _worst_ the enclave can do, not what it imposes on itself on top of that.
AFAIK it is not, or what we are protecting against with this anyway then?
Let say an LSM makes decision for the permissions based on origin. If we do not have this you can:
1. EMODPE
2. mprotect
I.e. whatever LSM decides, won't matter.
The other case, noexec, is now unconditionally denied.
Quoted 5 lines
Quoted 3 lines
I think other important thing to underline is that an LSM or any other security measure can only do a sane decision when the enclave is loaded. At that point we know the source (
vm_file).Right, you know the source, but it can be anonymous or a file.
They are both origin, the point being that you know what you're dealing with when you build the enclave, not when you map it.
This is my current rewrite of the commit message in my master branch:
"
mm: Add 'mprotect' callback to vm_ops
Intel Sofware Guard eXtensions (SGX) allows creation of blobs called enclaves, for which page permissions are defined when the enclave is first loaded. Once an enclave is loaded and initialized, it can be mapped to the process address space.
There is no standard file format for enclaves. They are dynamically built and the ways how enclaves are deployed differ greatly. For an app you might want to have a simple static binary, but on the other hand for a container you might want to dynamically create the whole thing at run-time. Also, the existing ecosystem for SGX is already large, which would make the task very hard.
Finally, even if there was a standard format, one would still want a dynamic way to add pages to the enclave. One big reason for this is that enclaves have load time defined pages that represent entry points to the enclave. Each entry point can service one hardware thread at a time and you might want to run-time parametrize this depending on your environment.
The consequence is that enclaves are best created with an ioctl API and the access control can be based only to the origin of the source file for the enclave data, i.e. on VMA file pointer and page permissions. For example, this could be done with LSM hooks that are triggered in the appropriate ioctl's and they could make the access control decision based on this information.
Unfortunately, there is ENCLS[EMODPE] that a running enclave can use to upgrade its permissions. If we do not limit mmap() and mprotect(), enclave could upgrade its permissions by using EMODPE followed by an appropriate mprotect() call. This would be completely hidden from the kernel.
Add 'mprotect' hook to vm_ops, so that a callback can be implemeted for SGX that will ensure that {mmap, mprotect}() permissions do not surpass any of the original page permissions. This feature allows to maintain and refine sane access control for enclaves.
"
I'm mostly happy with this but am open for change suggestions.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 8 lines · Jarkko Sakkinen
Quoted 6 lines · Dave Hansen
EMODPE is virtually irrelevant for this whole thing. The x86 PTE permissions still specify the most restrictive permissions, which is what matters the most.
We care about the _worst_ the enclave can do, not what it imposes on itself on top of that.
Quoted 8 lines
AFAIK it is not, or what we are protecting against with this anyway then?
Let say an LSM makes decision for the permissions based on origin. If we do not have this you can:
1. EMODPE
2. mprotect
The thing that matters is that the enclave needs relaxed permissions from the kernel. What it *ALSO* needs to do to *ITSELF* to get those permissions is entirely irrelevant to the kernel.
Quoted 25 lines
I.e. whatever LSM decides, won't matter.
The other case, noexec, is now unconditionally denied.
Quoted 5 lines
Quoted 3 lines
I think other important thing to underline is that an LSM or any other security measure can only do a sane decision when the enclave is loaded. At that point we know the source (
vm_file).Right, you know the source, but it can be anonymous or a file.
They are both origin, the point being that you know what you're dealing with when you build the enclave, not when you map it.
This is my current rewrite of the commit message in my master branch:
"
mm: Add 'mprotect' callback tovm_opsIntel Sofware Guard eXtensions (SGX) allows creation of blobs called enclaves, for which page permissions are defined when the enclave is first loaded. Once an enclave is loaded and initialized, it can be mapped to the process address space.
Quoted 6 lines
There is no standard file format for enclaves. They are dynamically built and the ways how enclaves are deployed differ greatly. For an app you might want to have a simple static binary, but on the other hand for a container you might want to dynamically create the whole thing at run-time. Also, the existing ecosystem for SGX is already large, which would make the task very hard.
I'm sorry I ever mentioned the file format. Please remove any mention of it. It's irrelevant. This entire paragraph is irrelevant.
Quoted 5 lines
Finally, even if there was a standard format, one would still want a dynamic way to add pages to the enclave. One big reason for this is that enclaves have load time defined pages that represent entry points to the enclave. Each entry point can service one hardware thread at a time and you might want to run-time parametrize this depending on your environment.
I also don't know what this paragraph has to do with the mprotect() hook. Please remove it.
Quoted 3 lines
The consequence is that enclaves are best created with an ioctl API and the access control can be based only to the origin of the source file for the enclave data, i.e. on VMA file pointer and page permissions. For example,
It's not strictly page permissions, though. It's actually VMA permissions. The thing you copy from might be the zero page, and even though it has Write=0 page permissions, apps are completely OK to write to the address. This is the WRITE vs. MAY_WRITE semantic in the VMA flags.
It's also not just about *files*. Anonymous memory might or might not be a valid source for enclave data based on LSM hooks.
Quoted 3 lines
this could be done with LSM hooks that are triggered in the appropriate ioctl's and they could make the access control decision based on this information.
This "appropriate ioctl's" is not good changelog material. Please use those bytes to convey actual information.
... this could be done with LSM hooks which restrict the source of enclave page data
I don't care that it's an ioctl(), really. What matters is what the ioctl() does: copy data into enclave pages.
Quoted 4 lines
Unfortunately, there is ENCLS[EMODPE] that a running enclave can use to upgrade its permissions. If we do not limit
mmap()andmprotect(), enclave could upgrade its permissions by using EMODPE followed by an appropriatemprotect()call. This would be completely hidden from the kernel.
There's too much irrelevant info.
I'll say it again: all that matters is that enclaves can legitimately, safely, and securely have a need for the kernel to change page permissions. That's *IT*. EMODPE just happens to be part of the mechanism that makes these permission changes safe for enclaves. It's a side show.
Quoted 4 lines
Add 'mprotect' hook to
vm_ops, so that a callback can be implemeted for SGX that will ensure that {mmap, mprotect}() permissions do not surpass any of the original page permissions. This feature allows to maintain and refine sane access control for enclaves.
Instead of "original", I'd stick to the "source" page nomenclature. There are also "original" permissions with mprotect().
Also, it's literally OK for the enclave page permissions to surpass the original (source) page permissions. That sentence is incorrect, or at least misleadingly imprecise.
Quoted 1 line
I'm mostly happy with this but am open for change suggestions.
I wrote a pretty nice description of this. It was about 90% correct, shorter, and conveyed more information. I'd suggest starting with that.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 15 lines · Dave Hansen
Quoted 14 lines · Jarkko Sakkinen
Quoted 6 lines · Dave Hansen
EMODPE is virtually irrelevant for this whole thing. The x86 PTE permissions still specify the most restrictive permissions, which is what matters the most.
We care about the _worst_ the enclave can do, not what it imposes on itself on top of that.
AFAIK it is not, or what we are protecting against with this anyway then?
Let say an LSM makes decision for the permissions based on origin. If we do not have this you can:
Quoted 6 lines
Quoted 2 lines
1. EMODPE
2. mprotectThe thing that matters is that the enclave needs relaxed permissions from the kernel. What it *ALSO* needs to do to *ITSELF* to get those permissions is entirely irrelevant to the kernel.
Lets try to find the root of misunderstanding, shall we?
I'm assuming that your statement is encapsulated to:
https://lore.kernel.org/linux-sgx/[email protected]/
I do agree with this.
And here is a direct quote:
"It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or the enclave itself are consistent with the earlier declared permissions."
Without the mprotect callback in place, by using EMODPE and mprotect, one could surpass the permisssions that we declared earlier.
With the callback in place this is not possible. EMODPE can be freely used but mprotect() always caps the permissions. It enables us to *not care* about EMODPE.
My problem is that I fully agree what you say in your description but disagree on that EMODPE should not be mentioned.
Quoted 9 lines
Quoted 6 lines
There is no standard file format for enclaves. They are dynamically built and the ways how enclaves are deployed differ greatly. For an app you might want to have a simple static binary, but on the other hand for a container you might want to dynamically create the whole thing at run-time. Also, the existing ecosystem for SGX is already large, which would make the task very hard.
I'm sorry I ever mentioned the file format. Please remove any mention of it. It's irrelevant. This entire paragraph is irrelevant.
Not sure if it is or not. It is merely to state that execve() path is possible. Perhaps should be toned down in some way. Should be like the a small remark at most.
Quoted 8 lines
Quoted 5 lines
Finally, even if there was a standard format, one would still want a dynamic way to add pages to the enclave. One big reason for this is that enclaves have load time defined pages that represent entry points to the enclave. Each entry point can service one hardware thread at a time and you might want to run-time parametrize this depending on your environment.
I also don't know what this paragraph has to do with the
mprotect()hook. Please remove it.
Agreed.
Quoted 11 lines
Quoted 3 lines
The consequence is that enclaves are best created with an ioctl API and the access control can be based only to the origin of the source file for the enclave data, i.e. on VMA file pointer and page permissions. For example,
It's not strictly page permissions, though. It's actually VMA permissions. The thing you copy from might be the zero page, and even though it has Write=0 page permissions, apps are completely OK to write to the address. This is the WRITE vs.
MAY_WRITEsemantic in the VMA flags.It's also not just about *files*. Anonymous memory might or might not be a valid source for enclave data based on LSM hooks.
Yes, this should be refined, agreed. Source can be either anonymous page or a file, I do of course understand that.
Quoted 4 lines
Quoted 3 lines
this could be done with LSM hooks that are triggered in the appropriate ioctl's and they could make the access control decision based on this information.
Quoted 8 lines
This "appropriate ioctl's" is not good changelog material. Please use those bytes to convey actual information.
... this could be done with LSM hooks which restrict the source of enclave page data
I don't care that it's an
ioctl(), really. What matters is what theioctl()does: copy data into enclave pages.
Agreed.
Quoted 5 lines
Quoted 4 lines
Unfortunately, there is ENCLS[EMODPE] that a running enclave can use to upgrade its permissions. If we do not limit
mmap()andmprotect(), enclave could upgrade its permissions by using EMODPE followed by an appropriatemprotect()call. This would be completely hidden from the kernel.
Quoted 7 lines
There's too much irrelevant info.
I'll say it again: all that matters is that enclaves can legitimately, safely, and securely have a need for the kernel to change page permissions. That's *IT*. EMODPE just happens to be part of the mechanism that makes these permission changes safe for enclaves. It's a side show.
Disagree on this. I wrote my statement about. Maybe it should not be driving argument but should be definitely part of the description.
Quoted 11 lines
Quoted 4 lines
Add 'mprotect' hook to
vm_ops, so that a callback can be implemeted for SGX that will ensure that {mmap, mprotect}() permissions do not surpass any of the original page permissions. This feature allows to maintain and refine sane access control for enclaves.Instead of "original", I'd stick to the "source" page nomenclature. There are also "original" permissions with
mprotect().Also, it's literally OK for the enclave page permissions to surpass the original (source) page permissions. That sentence is incorrect, or at least misleadingly imprecise.
Yes it is. It's fine to use EMODPE to upgrade the perms, and by having this hook, doing anything nasty with it is impossible.
Quoted 4 lines
Quoted 1 line
I'm mostly happy with this but am open for change suggestions.
I wrote a pretty nice description of this. It was about 90% correct, shorter, and conveyed more information. I'd suggest starting with that.
I should have added a disclaimer that my description is not up to date. Just do not want to make anything final before this EMODPE discussion has some conclusion.
I'm fine using your description as basis for the commit message if as long as these few details are settled.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 2 lines · Jarkko Sakkinen
My problem is that I fully agree what you say in your description but disagree on that EMODPE should not be mentioned.
I'll just be very clear: I'm not willing to ack any patch with a changelog that has more than a passing mention of EMODPE.
Do what you think is best, but if sticking to your guns may deplete the pool of folks willing to ack your patch.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 9 lines · Dave Hansen
Quoted 2 lines · Jarkko Sakkinen
My problem is that I fully agree what you say in your description but disagree on that EMODPE should not be mentioned.
I'll just be very clear: I'm not willing to ack any patch with a changelog that has more than a passing mention of EMODPE.
Do what you think is best, but if sticking to your guns may deplete the pool of folks willing to ack your patch.
I do see it mentioned in other responses too in this thread, and not just mine.
And here is even a request to get it to the changelog:
https://lore.kernel.org/linux-sgx/[email protected]/
I'm absolutely fine not to mention EMODPE but after re-reading the thread, it is not like there is one voice on it. I don't really care all that much whether it is mentioned or not but there should be some reasonable logic behind the decision.
PS. I just noticed that my previous response did not reach lore so I bounced it again.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 17 lines · Jarkko Sakkinen
Quoted 9 lines · Dave Hansen
Quoted 3 lines · Jarkko Sakkinen
My problem is that I fully agree what you say in your description but disagree on that EMODPE should not be mentioned.
I'll just be very clear: I'm not willing to ack any patch with a changelog that has more than a passing mention of EMODPE.
Do what you think is best, but if sticking to your guns may deplete the pool of folks willing to ack your patch.
I do see it mentioned in other responses too in this thread, and not just mine.
Quoted 8 lines
And here is even a request to get it to the changelog:
https://lore.kernel.org/linux-sgx/[email protected]/
I'm absolutely fine not to mention EMODPE but after re-reading the thread, it is not like there is one voice on it. I don't really care all that much whether it is mentioned or not but there should be some reasonable logic behind the decision.
I don’t personally care that much about EMODPE but, you could probably get the point across with something like:
SGX’s EPCM permission bits do not obviate the need to enforce these rules in the PTEs because enclaves can freely modify the EPCM permissions using EMODPE.
IOW, EMODPE is not really special here; rather, EMODPE’s existence demonstrates that EADD / EEXTEND are not special.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 9 lines · Andy Lutomirski
I don’t personally care that much about EMODPE but, you could probably get the point across with something like:
SGX’s EPCM permission bits do not obviate the need to enforce these rules in the PTEs because enclaves can freely modify the EPCM permissions using EMODPE.
IOW, EMODPE is not really special here; rather, EMODPE’s existence demonstrates that EADD / EEXTEND are not special.
So I did "disagree and commit" with this one. I'm not actually diagreeing on anything what Dave wrote, on the contrary it is an understandable high level description. I just thought that it would not hurt to remark that the ISA contains such peculiarities as EMODPE.
I did only very rudimentary clean up for the text (e.g. fix the ioctl name, add shortt summary and not much else).
Does not make sense to waste more time to this. I'll move on to implement the missing boot time patching for the vDSO so that we get the next version out.
"
mm: Add 'mprotect' hook to struct vm_operations_struct
Background
==========
1. SGX enclave pages are populated with data by copying data to them from normal memory via ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES).
2. We want to be able to restrict those normal memory data sources. For instance, before copying data to an executable enclave page, we might ensure that the source is executable.
3. Enclave page permissions are dynamic just like normal permissions and can be adjusted at runtime with mprotect() (along with a corresponding special instruction inside the enclave).
4. The original data source may have have long since vanished at the time when enclave page permission are established (mmap() or mprotect()).
Solution
========
The solution is to force enclaves creators to declare their intent up front to ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES). This intent can me immediately compared to the source data mapping (and rejected if necessary). It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or the enclave itself are consistent with the earlier declared permissions.
Essentially, this means that whenever the kernel is asked to change an enclave PTE, it needs to ensure the change is consistent with that stashed intent. There is an existing vm_ops->mmap() hook which allows SGX to do that for mmap(). However, there is no ->mprotect() hook. Add a vm_ops->mprotect() hook so that mprotect() operations which are inconsistent with any page's stashed intent can be rejected by the driver.
Implications
============
However, there is currently no implementation of the intent checks at the time of ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES). That means that the intent argument (SGX_PROT_*) is currently unused.
"
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 26 lines · Jarkko Sakkinen
Quoted 9 lines · Andy Lutomirski
I don’t personally care that much about EMODPE but, you could probably get the point across with something like:
SGX’s EPCM permission bits do not obviate the need to enforce these rules in the PTEs because enclaves can freely modify the EPCM permissions using EMODPE.
IOW, EMODPE is not really special here; rather, EMODPE’s existence demonstrates that EADD / EEXTEND are not special.
So I did "disagree and commit" with this one. I'm not actually diagreeing on anything what Dave wrote, on the contrary it is an understandable high level description. I just thought that it would not hurt to remark that the ISA contains such peculiarities as EMODPE.
I did only very rudimentary clean up for the text (e.g. fix the ioctl name, add shortt summary and not much else).
Does not make sense to waste more time to this. I'll move on to implement the missing boot time patching for the vDSO so that we get the next version out.
"
mm: Add 'mprotect' hook to structvm_operations_struct
Quoted 8 lines
Background
==========1. SGX enclave pages are populated with data by copying data to them from normal memory via
ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES).
2. We want to be able to restrict those normal memory data sources. For instance, before copying data to an executable enclave page, we might ensure that the source is executable.
I know I wrote that. I suck, and I wrote it in a changelog-unacceptable way. Folks dislike the use of "we" in these things. Here's a better version:
2. It is desirable to be able to restrict those normal memory data sources. For instance, the kernel can ensure that the source is executable, before copying data to an executable enclave page.
Quoted 10 lines
3. Enclave page permissions are dynamic just like normal permissions and can be adjusted at runtime with
mprotect()(along with a corresponding special instruction inside the enclave).
4. The original data source may have have long since vanished at the time when enclave page permission are established (mmap()ormprotect()).Solution
========
Quoted 6 lines
The solution is to force enclaves creators to declare their intent up front to
ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES). This intent can me immediately compared to the source data mapping (and rejected if necessary). It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or the enclave itself are consistent with the earlier declared permissions.
Let's also say "... or *requested* by the enclave itself ...", since the enclave itself can't directly make syscalls.
Quoted 7 lines
Essentially, this means that whenever the kernel is asked to change an enclave PTE, it needs to ensure the change is consistent with that stashed intent. There is an existing
vm_ops->mmap()hook which allows SGX to do that formmap(). However, there is no ->mprotect() hook. Add avm_ops->mprotect()hook so thatmprotect()operations which are inconsistent with any page's stashed intent can be rejected by the driver.
Quoted 6 lines
Implications
============However, there is currently no implementation of the intent checks at the time of
ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES). That means that the intent argument (SGX_PROT_*) is currently unused.
This was incorrect to say. Sean corrected me on this point. Could you look through the thread and incorporate that?
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 61 lines · Dave Hansen
Quoted 26 lines · Jarkko Sakkinen
Quoted 9 lines · Andy Lutomirski
I don’t personally care that much about EMODPE but, you could probably get the point across with something like:
SGX’s EPCM permission bits do not obviate the need to enforce these rules in the PTEs because enclaves can freely modify the EPCM permissions using EMODPE.
IOW, EMODPE is not really special here; rather, EMODPE’s existence demonstrates that EADD / EEXTEND are not special.
So I did "disagree and commit" with this one. I'm not actually diagreeing on anything what Dave wrote, on the contrary it is an understandable high level description. I just thought that it would not hurt to remark that the ISA contains such peculiarities as EMODPE.
I did only very rudimentary clean up for the text (e.g. fix the ioctl name, add shortt summary and not much else).
Does not make sense to waste more time to this. I'll move on to implement the missing boot time patching for the vDSO so that we get the next version out.
"
mm: Add 'mprotect' hook to structvm_operations_structQuoted 8 lines
Background
==========1. SGX enclave pages are populated with data by copying data to them from normal memory via
ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES).
2. We want to be able to restrict those normal memory data sources. For instance, before copying data to an executable enclave page, we might ensure that the source is executable.I know I wrote that. I suck, and I wrote it in a changelog-unacceptable way. Folks dislike the use of "we" in these things. Here's a better version:
2. It is desirable to be able to restrict those normal memory data sources. For instance, the kernel can ensure that the source is executable, before copying data to an executable enclave page.
Quoted 10 lines
3. Enclave page permissions are dynamic just like normal permissions and can be adjusted at runtime with
mprotect()(along with a corresponding special instruction inside the enclave).
4. The original data source may have have long since vanished at the time when enclave page permission are established (mmap()ormprotect()).Solution
========Quoted 6 lines
The solution is to force enclaves creators to declare their intent up front to
ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES). This intent can me immediately compared to the source data mapping (and rejected if necessary). It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or the enclave itself are consistent with the earlier declared permissions.
Quoted 2 lines
Let's also say "... or *requested* by the enclave itself ...", since the enclave itself can't directly make syscalls.
Yes, it is definitely more understandable way to say it. Do you mind if I rephrase it as:
"It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or requested the enclave by itself (e.g. by issuing ECLU[EMODPE]) are consistent with the earlier declared permissions."
I'd just mention EMODPE as an example, but I'm also perfectly fine leaving that out :-) Not a big deal for me.
Also, should there be commas, i.e. ", or requested the enclave by itself,"? I suck with English comma rules.
Quoted 10 lines
Quoted 10 lines
Essentially, this means that whenever the kernel is asked to change an enclave PTE, it needs to ensure the change is consistent with that stashed intent. There is an existing
vm_ops->mmap()hook which allows SGX to do that formmap(). However, there is no ->mprotect() hook. Add avm_ops->mprotect()hook so thatmprotect()operations which are inconsistent with any page's stashed intent can be rejected by the driver.Implications
============
Quoted 6 lines
Quoted 3 lines
However, there is currently no implementation of the intent checks at the time of
ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES). That means that the intent argument (SGX_PROT_*) is currently unused.This was incorrect to say. Sean corrected me on this point. Could you look through the thread and incorporate that?
OK, so we are probably talking about [1] here, right?
There are at least two checks done with and without the callback:
A. TCS pages are required to be passed with zero EPCM permissions,
the reason being that CPU will silently overwrite its permissions zero. This check resides in sgx_validate_secinfo() [2].
B. noexec partitions are unconditionally disallowed. This check
resides in __sgx_encl_add_page() [2].
TCS pages are funky in the way that they are also required to be mapped as RW. OK, then you might argue that why not just require RW as ioctl supplied permissions. That would cause a mismatch with CPU calculated MRENCLAVE and one calculated to SIGSTRUCT. I've written in past quite length explanation of this to the kdoc of sgx_ioc_enclave_add_pages() [2].
This was something that I did not find from Sean's response.
To encompensate all this information in a paragraph, I'd write along the lines of
"ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES, ...) checks for every page that Thread Control Structure (TCS) pages are always added with zero permissions and no pages are sourced from noexec partitions. TCS pages are pages that work as entry points to the enclave. This is the basic acceptance criteria for any enclave page before it gets mapped.
After finishing this, the ioctl will project the enclave permissions to the corresponding VMA permissions and stores the result for later lookup. For regular pages this is an identity mapping but as an exception TCS pages are unconditionally mapped as RW VMA permssion even though their enclave permissions are zero. This required by the ISA.
This information will be used by sgx_mmap() and sgx_vma_protect() to enforce that higher permissions than the projected permissions will not be used by checking this for each every page in the address range.
By doing this, we give assets for LSM's to make decisions during the build time based on projected VMA permissions and the source VMA (either a file or anonymous mapping) that hold when the enclave is finally mapped to the visible memory."
Is this sufficient?
[1] https://lore.kernel.org/linux-sgx/[email protected]/T/#m7e84af96d63dd8ec528422cfc942f42e3bdf4356
[2] https://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-sgx.git/tree/arch/x86/kernel/cpu/sgx/ioctl.c
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 68 lines · Jarkko Sakkinen
Quoted 61 lines · Dave Hansen
Quoted 26 lines · Jarkko Sakkinen
Quoted 9 lines · Andy Lutomirski
I don’t personally care that much about EMODPE but, you could probably get the point across with something like:
SGX’s EPCM permission bits do not obviate the need to enforce these rules in the PTEs because enclaves can freely modify the EPCM permissions using EMODPE.
IOW, EMODPE is not really special here; rather, EMODPE’s existence demonstrates that EADD / EEXTEND are not special.
So I did "disagree and commit" with this one. I'm not actually diagreeing on anything what Dave wrote, on the contrary it is an understandable high level description. I just thought that it would not hurt to remark that the ISA contains such peculiarities as EMODPE.
I did only very rudimentary clean up for the text (e.g. fix the ioctl name, add shortt summary and not much else).
Does not make sense to waste more time to this. I'll move on to implement the missing boot time patching for the vDSO so that we get the next version out.
"
mm: Add 'mprotect' hook to structvm_operations_structQuoted 8 lines
Background
==========1. SGX enclave pages are populated with data by copying data to them from normal memory via
ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES).
2. We want to be able to restrict those normal memory data sources. For instance, before copying data to an executable enclave page, we might ensure that the source is executable.I know I wrote that. I suck, and I wrote it in a changelog-unacceptable way. Folks dislike the use of "we" in these things. Here's a better version:
2. It is desirable to be able to restrict those normal memory data sources. For instance, the kernel can ensure that the source is executable, before copying data to an executable enclave page.
Quoted 10 lines
3. Enclave page permissions are dynamic just like normal permissions and can be adjusted at runtime with
mprotect()(along with a corresponding special instruction inside the enclave).
4. The original data source may have have long since vanished at the time when enclave page permission are established (mmap()ormprotect()).Solution
========Quoted 6 lines
The solution is to force enclaves creators to declare their intent up front to
ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES). This intent can me immediately compared to the source data mapping (and rejected if necessary). It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or the enclave itself are consistent with the earlier declared permissions.Quoted 2 lines
Let's also say "... or *requested* by the enclave itself ...", since the enclave itself can't directly make syscalls.
Yes, it is definitely more understandable way to say it. Do you mind if I rephrase it as:
Quoted 7 lines
"It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or requested the enclave by itself (e.g. by issuing ECLU[EMODPE]) are consistent with the earlier declared permissions."
I'd just mention EMODPE as an example, but I'm also perfectly fine leaving that out :-) Not a big deal for me.
If I say it's a big deal for me, will you remove the bloody thing?
Mentioning EMODPE is a distraction for this patch. It's a big distraction because it makes it sound like it is some kind of *peer* of mmap()/mprotect(). It's not. It's subservient to x86 paging protections and thus *IRRELEVANT* for this unless you care about the intricacies of writing enclaves.
It's a big deal to me.
Also, I've tried to give this feedback previously, but the paging permissions are also essentially irrelevant.
Quoted 2 lines
Also, should there be commas, i.e. ", or requested the enclave by itself,"? I suck with English comma rules.
I'd just say this:
It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations are consistent with the earlier declared permissions.
Yours was starting to look pretty run on.
Quoted 5 lines
"
ioctl(fd, SGX_IOC_ENCLAVE_ADD_PAGES, ...)checks for every page that Thread Control Structure (TCS) pages are always added with zero permissions and no pages are sourced from noexec partitions. TCS pages are pages that work as entry points to the enclave. This is the basic acceptance criteria for any enclave page before it gets mapped.
This is going off into the SGX weeds again.
We don't need to justify the ABI for an ioctl() introduced in a different patch in *THIS* patch. Just remove this, please.
Quoted 3 lines
After finishing this, the ioctl will project the enclave permissions to the corresponding VMA permissions and stores the result for later lookup.
That sounds vaguely relevant, although I'm not sure what permission projection is. You use that terminology over and over, so you probably need to define it.
Quoted 3 lines
For regular pages this is an identity mapping but as an
exception TCS pages are unconditionally mapped as RW VMA permssion even though their enclave permissions are zero. This required by the ISA.
I don't think this is relevant.
Quoted 4 lines
This information will be used by
sgx_mmap()andsgx_vma_protect()to enforce that higher permissions than the projected permissions will not be used by checking this for each every page in the address range.
I've given this feedback before. Please don't use "higher" and "lower" permissions. "Stronger/weaker" is my preferred terminology.
You also don't have to *NAME* the functions. If I want to know where a structure field is, grep is a better way to find that.
Does this lose any meaning if we just say:
This information will be to ensure that enclave PTEs will not be created with permissions weaker than the source data
?
Quoted 4 lines
By doing this, we give assets for LSM's to make decisions during the build time based on projected VMA permissions and the source VMA (either a file or anonymous mapping) that hold when the enclave is finally mapped to the visible memory."
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Dave HansenQuoted 15 lines · Dave Hansen
Quoted 14 lines · Jarkko Sakkinen
Quoted 6 lines · Dave Hansen
EMODPE is virtually irrelevant for this whole thing. The x86 PTE permissions still specify the most restrictive permissions, which is what matters the most.
We care about the _worst_ the enclave can do, not what it imposes on itself on top of that.
AFAIK it is not, or what we are protecting against with this anyway then?
Let say an LSM makes decision for the permissions based on origin. If we do not have this you can:
Quoted 6 lines
Quoted 2 lines
1. EMODPE
2. mprotectThe thing that matters is that the enclave needs relaxed permissions from the kernel. What it *ALSO* needs to do to *ITSELF* to get those permissions is entirely irrelevant to the kernel.
Lets try to find the root of misunderstanding, shall we?
I'm assuming that your statement is encapsulated to:
https://lore.kernel.org/linux-sgx/[email protected]/
I do agree with this.
And here is a direct quote:
"It is also stashed off and then later compared with enclave PTEs to ensure that any future mmap()/mprotect() operations performed by the enclave creator or the enclave itself are consistent with the earlier declared permissions."
Without the mprotect callback in place, by using EMODPE and mprotect, one could surpass the permisssions that we declared earlier.
With the callback in place this is not possible. EMODPE can be freely used but mprotect() always caps the permissions. It enables us to *not care* about EMODPE.
My problem is that I fully agree what you say in your description but disagree on that EMODPE should not be mentioned.
Quoted 9 lines
Quoted 6 lines
There is no standard file format for enclaves. They are dynamically built and the ways how enclaves are deployed differ greatly. For an app you might want to have a simple static binary, but on the other hand for a container you might want to dynamically create the whole thing at run-time. Also, the existing ecosystem for SGX is already large, which would make the task very hard.
I'm sorry I ever mentioned the file format. Please remove any mention of it. It's irrelevant. This entire paragraph is irrelevant.
Not sure if it is or not. It is merely to state that execve() path is possible. Perhaps should be toned down in some way. Should be like the a small remark at most.
Quoted 8 lines
Quoted 5 lines
Finally, even if there was a standard format, one would still want a dynamic way to add pages to the enclave. One big reason for this is that enclaves have load time defined pages that represent entry points to the enclave. Each entry point can service one hardware thread at a time and you might want to run-time parametrize this depending on your environment.
I also don't know what this paragraph has to do with the
mprotect()hook. Please remove it.
Agreed.
Quoted 11 lines
Quoted 3 lines
The consequence is that enclaves are best created with an ioctl API and the access control can be based only to the origin of the source file for the enclave data, i.e. on VMA file pointer and page permissions. For example,
It's not strictly page permissions, though. It's actually VMA permissions. The thing you copy from might be the zero page, and even though it has Write=0 page permissions, apps are completely OK to write to the address. This is the WRITE vs.
MAY_WRITEsemantic in the VMA flags.It's also not just about *files*. Anonymous memory might or might not be a valid source for enclave data based on LSM hooks.
Yes, this should be refined, agreed. Source can be either anonymous page or a file, I do of course understand that.
Quoted 4 lines
Quoted 3 lines
this could be done with LSM hooks that are triggered in the appropriate ioctl's and they could make the access control decision based on this information.
Quoted 8 lines
This "appropriate ioctl's" is not good changelog material. Please use those bytes to convey actual information.
... this could be done with LSM hooks which restrict the source of enclave page data
I don't care that it's an
ioctl(), really. What matters is what theioctl()does: copy data into enclave pages.
Agreed.
Quoted 5 lines
Quoted 4 lines
Unfortunately, there is ENCLS[EMODPE] that a running enclave can use to upgrade its permissions. If we do not limit
mmap()andmprotect(), enclave could upgrade its permissions by using EMODPE followed by an appropriatemprotect()call. This would be completely hidden from the kernel.
Quoted 7 lines
There's too much irrelevant info.
I'll say it again: all that matters is that enclaves can legitimately, safely, and securely have a need for the kernel to change page permissions. That's *IT*. EMODPE just happens to be part of the mechanism that makes these permission changes safe for enclaves. It's a side show.
Disagree on this. I wrote my statement about. Maybe it should not be driving argument but should be definitely part of the description.
Quoted 11 lines
Quoted 4 lines
Add 'mprotect' hook to
vm_ops, so that a callback can be implemeted for SGX that will ensure that {mmap, mprotect}() permissions do not surpass any of the original page permissions. This feature allows to maintain and refine sane access control for enclaves.Instead of "original", I'd stick to the "source" page nomenclature. There are also "original" permissions with
mprotect().Also, it's literally OK for the enclave page permissions to surpass the original (source) page permissions. That sentence is incorrect, or at least misleadingly imprecise.
Yes it is. It's fine to use EMODPE to upgrade the perms, and by having this hook, doing anything nasty with it is impossible.
Quoted 4 lines
Quoted 1 line
I'm mostly happy with this but am open for change suggestions.
I wrote a pretty nice description of this. It was about 90% correct, shorter, and conveyed more information. I'd suggest starting with that.
I should have added a disclaimer that my description is not up to date. Just do not want to make anything final before this EMODPE discussion has some conclusion.
I'm fine using your description as basis for the commit message if as long as these few details are settled.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Dave HansenOn Mon, Sep 28, 2020 at 07:04:38AM -0700, Dave Hansen wrote:
Good morning, I hope the weekend is going well for everyone.
Quoted 28 lines · Jarkko Sakkinen
Quoted 15 lines · Dave Hansen
Quoted 14 lines · Sean Christopherson
Quoted 6 lines
Quoted 1 line
That means that the intent argument (SGX_PROT_*) is currently unused.
No, the intent argument is used (eventually) by SGX's ->mprotect() implementation, i.e.
sgx_mprotect()enforces that the actual protections are a subset of the declared/intended protections.If ->mprotect() is not merged, then it yes, it will be unused.
OK, I think I've got it.
I think I'm OK with adding ->mprotect(). As long as folks buy into the argument that intent needs to be checked at
mmap()time, they obviously need to be checked atmprotect()too.Quoted 7 lines
Quoted 4 lines
Jarkko, if you want to try and rewrite the changelog, capturing the discussion here and reply, I think I can ack the resulting patch. I don't know if that will satisfy the request from Boris from an ack from a "mm person", but we can at least start there. :)
I think what it needs, based on what I've read, is the step by step description of the EMODPE scenarion without this callback and with it.
EMODPE is virtually irrelevant for this whole thing. The x86 PTE permissions still specify the most restrictive permissions, which is what matters the most.
Quoted 8 lines
We care about the _worst_ the enclave can do, not what it imposes on itself on top of that.
Quoted 3 lines
I think other important thing to underline is that an LSM or any other security measure can only do a sane decision when the enclave is loaded. At that point we know the source (
vm_file).Right, you know the source, but it can be anonymous or a file.
Or it could be loaded over the network in encrypted form by the enclave itself.
Sean, admirably, wants to peer into the future and set the driver up from an architectural perspective, to future proof it for the imposition of security controls. So it would seem helpful to peer a bit.
If I can paraphrase/simplify the discussion to date; the best the kernel can do with respect to SGX is to impose controls, via mprotect, that limit the maximum permissions of an enclave page to whatever was specified when the enclave was loaded/initialized.
So here is the question that would seem to need answering:
Is this even a relevant control if we cede the notion of dynamically loadable enclave code, which is the objective of SGX2 hardware, which will in all likelihood be the only relevant hardware implementation in the future?
The answer to this could very well be yes if the objective is to provide a method for a platform owner to explicitly block dynamically loadable enclave code. Since there seems to be a desire for immense clarity in the changelogs surrounding all of this, framing the discussion in something practical like this may be of assistance.
One of the desires of the SGX user community is to not allow visibility into enclave code, this is one of the notions/objectives of confidential computing. The Protected Code Loader that was added by Intel to their PSW is an acknowledgement of this fact. EDMM and dynamically loadable code makes doing this much more efficient so that would seem to be the face of the future.
My apologies for 'delaying' the driver even more. I was accused of that about a year ago but it appears I didn't do too much damage... :-)
Best wishes for a productive week.
Dr. Greg
As always,
Dr. Greg Wettstein, Ph.D, Worker Autonomously self-defensive
Enjellic Systems Development, LLC IOT platforms and edge devices.
4206 N. 19th Ave.
Fargo, ND 58102
PH: 701-281-1686 EMAIL: [email protected]
------------------------------------------------------------------------------
"If you get to thinkin' you're a person of some influence, try
orderin' somebody else's dog around."
-- Cowboy Wisdom
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 4 lines · Dr. Greg
Is this even a relevant control if we cede the notion of dynamically loadable enclave code, which is the objective of SGX2 hardware, which will in all likelihood be the only relevant hardware implementation in the future?
Yes, it's still relevant. Giving the thumbs up to dynamically loadable code is not a purely binary decision, e.g. a user/admin can allow RW->RX transitions but still disallow full RWX permissions.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
On Mon, Oct 19, 2020 at 02:31:05PM -0700, Sean Christopherson wrote:
Good morning, I hope the day is starting well for everyone.
Quoted 9 lines · Dr. Greg
Quoted 4 lines
Is this even a relevant control if we cede the notion of dynamically loadable enclave code, which is the objective of SGX2 hardware, which will in all likelihood be the only relevant hardware implementation in the future?
Yes, it's still relevant. Giving the thumbs up to dynamically loadable code is not a purely binary decision, e.g. a user/admin can allow
RW->RXtransitions but still disallow full RWX permissions.
With respect to the security issue at hand, the only relevant issue would seem to be if a page had write permissions at one time in its trajectory to having execute permisions, isn't this correct?
The next paragraph of my reply wasn't included in your reply, but I did state that the mprotect hook would be relevant if its purpose was to disallow this permission trajectory and in the process disable enclave dynamic code loading and execution.
So to assist everyone in understanding this issue and the security implications involved, is the ultimate purpose of the mprotect hook to disable dynamic code loading?
Have a good day.
Dr. Greg
As always,
Dr. Greg Wettstein, Ph.D, Worker Autonomously self-defensive
Enjellic Systems Development, LLC IOT platforms and edge devices.
4206 N. 19th Ave.
Fargo, ND 58102
PH: 701-281-1686 EMAIL: [email protected]
------------------------------------------------------------------------------
"Those who will not study history are doomed to debug it."
-- Barry Shein
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 10 lines · Dr. Greg
On Mon, Oct 19, 2020 at 02:31:05PM -0700, Sean Christopherson wrote:
Good morning, I hope the day is starting well for everyone.
Quoted 5 lines · Dr. Greg
Quoted 4 lines
Is this even a relevant control if we cede the notion of dynamically loadable enclave code, which is the objective of SGX2 hardware, which will in all likelihood be the only relevant hardware implementation in the future?
Quoted 7 lines
Quoted 3 lines
Yes, it's still relevant. Giving the thumbs up to dynamically loadable code is not a purely binary decision, e.g. a user/admin can allow
RW->RXtransitions but still disallow full RWX permissions.With respect to the security issue at hand, the only relevant issue would seem to be if a page had write permissions at one time in its trajectory to having execute permisions, isn't this correct?
No. RW->RX has different properties than RWX. E.g. an enclave that dynamically loads code is not the same thing as an enclave that allows simultaneously writing and executing a page.
Quoted 8 lines
The next paragraph of my reply wasn't included in your reply, but I did state that the mprotect hook would be relevant if its purpose was to disallow this permission trajectory and in the process disable enclave dynamic code loading and execution.
So to assist everyone in understanding this issue and the security implications involved, is the ultimate purpose of the mprotect hook to disable dynamic code loading?
No, it's to provide line of sight to enforcing granular LSM checks on enclave pages. Jumping back to the RWX thing, as a defense in depth measure, a policy owner could set an SELinux policy to never allow RWX, even for enclaves that dynamically load code.
Whether or not having per-page granluarity on enclave permission checks is valuable/interesting is debatable, e.g. it's why LSM integration is notably absent from the this series. But, adding the ->mprotect() hook is relatively cheap and can always be removed if it's deemed unnecessary in the long run. The reverse is not true; omitting ->mprotect() commits the kernel to an ABI that is either ugly and potentially painful (require all enclaves to declare full RWX permissions), or flat out prevents adding granular LSM support in the future (do nothing).
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
On Tue, Oct 20, 2020 at 09:40:00AM -0700, Sean Christopherson wrote:
Good morning, I hope the week has gone well for everyone.
Quoted 9 lines · Dr. Greg
Quoted 4 lines
With respect to the security issue at hand, the only relevant issue would seem to be if a page had write permissions at one time in its trajectory to having execute permisions, isn't this correct?
No.
RW->RXhas different properties than RWX. E.g. an enclave that dynamically loads code is not the same thing as an enclave that allows simultaneously writing and executing a page.
Yes, it is certainly correct that a platform administrator may want to restrict RWX, given that it makes an enclave susceptible to potential arbitrary code execution if there is a programming error in the enclave.
However, I think it is important for everyone interested in these issues, to reflect back on what started all of this and that was Andy's concern that the initial incantations of the driver allowed execution of arbitrary memory without the ability of the LSM to get a 'look' at the code/memory.
My point in all of this is that a permissions trajectory for an enclave that allows for write permissions on a path that terminates in X permissions opens the door for arbitrary memory execution that the platform security architect has no insight into or that the LSM will have any control over.
There is no guarantee that dynamically loaded code has to come into the enclave via anything that the operating system has visibility into. If the enclave can toggle RW->RX it is free to dynamically load code, in encrypted form over the network and then execute it.
In fact, I would posit that this model will be a primary use for dynamic code loading. The SGX user community views 'confidential computing' as much about protecting visibility into algorithms and code as it is about data that is being operated on. In certain unnamed venues where I have consulted it is the primary concern.
So in the broadest sense, we have spent a year worrying about if and how the LSM will have visibility into enclave based code and in the end the only really relevant security mechanism available is limiting page permission transitions that prevent dynamic code loading. Modulo of course the issue with RWX, where a platform owner may elect to try and prevent an enclave writer from shooting themselves in the foot.
The issue at hand is that the primary security threat of the technology is the same as what the user community wants to use it for. Joanna Rutkowska called that out a half decade ago when she first reviewed the technology.
Quoted 14 lines
Quoted 8 lines
The next paragraph of my reply wasn't included in your reply, but I did state that the mprotect hook would be relevant if its purpose was to disallow this permission trajectory and in the process disable enclave dynamic code loading and execution.
So to assist everyone in understanding this issue and the security implications involved, is the ultimate purpose of the mprotect hook to disable dynamic code loading?
No, it's to provide line of sight to enforcing granular LSM checks on enclave pages. Jumping back to the RWX thing, as a defense in depth measure, a policy owner could set an SELinux policy to never allow RWX, even for enclaves that dynamically load code.
Quoted 9 lines
Whether or not having per-page granluarity on enclave permission checks is valuable/interesting is debatable, e.g. it's why LSM integration is notably absent from the this series. But, adding the ->mprotect() hook is relatively cheap and can always be removed if it's deemed unnecessary in the long run. The reverse is not true; omitting ->mprotect() commits the kernel to an ABI that is either ugly and potentially painful (require all enclaves to declare full RWX permissions), or flat out prevents adding granular LSM support in the future (do nothing).
I believe your analysis with respect to the ability to remove ->mprotect is flawed. The long standing consensus has been that functionality never gets broken. Once ->mprotect is a component of the ABI it would have to be left intact since it could potentially break things that elected to use it. On the other hand there is a long standing history of adding functionality.
I can't bring myself to believe that LSM's are going to be written that will be making enclave security decisions on a page by page basis. Given what I have written above, I think all of this comes down to giving platform administrators one of three decisions, in order of most to least secure:
1.) Block dynamic code loading and execution.
2.) Block access to RWX pages.
3.) The wild west - no restrictions on enclave page protection manipulation.
From a security perspective I would argue for the wisdom of making option 1 unconditional via a kernel command-line parameter.
It may be that ->mprotect is the right mechanism to implement this. If that is the case, frame the discussion and documentation so that it reflects the actual security threat and the consideration and means for dealing with it.
Hopefully all of this is useful to the stakeholders in this technology.
Have a good weekend.
Dr. Greg
As always,
Dr. Greg Wettstein, Ph.D, Worker Autonomously self-defensive
Enjellic Systems Development, LLC IOT platforms and edge devices.
4206 19th Ave. N.
Fargo, ND 58102
PH: 701-281-1686 EMAIL: [email protected]
------------------------------------------------------------------------------
"Politics is the business of getting power and privilege without possessing
merit."
-- P.J. O'Rourke
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 14 lines · Dr. Greg
I can't bring myself to believe that LSM's are going to be written that will be making enclave security decisions on a page by page basis. Given what I have written above, I think all of this comes down to giving platform administrators one of three decisions, in order of most to least secure:
1.) Block dynamic code loading and execution.
I don’t understand what you’re trying to say. Unless we’re going to split enclaves into multiple VMAs with different permissions, how do you expect to block dynamic code loading unless you have separate RW and RX pages? That would be “page-by-page”, right?
Quoted 29 lines
2.) Block access to RWX pages.
3.) The wild west - no restrictions on enclave page protection manipulation.
From a security perspective I would argue for the wisdom of making option 1 unconditional via a kernel command-line parameter.
It may be that ->mprotect is the right mechanism to implement this. If that is the case, frame the discussion and documentation so that it reflects the actual security threat and the consideration and means for dealing with it.
Hopefully all of this is useful to the stakeholders in this technology.
Have a good weekend.
Dr. Greg
As always,
Dr. Greg Wettstein, Ph.D, Worker Autonomously self-defensive
Enjellic Systems Development, LLC IOT platforms and edge devices.
4206 19th Ave. N.
Fargo, ND 58102
PH: 701-281-1686 EMAIL: [email protected]
------------------------------------------------------------------------------
"Politics is the business of getting power and privilege without possessing merit."-- P.J. O'Rourke
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
On Sat, Oct 24, 2020 at 08:33:21AM -0700, Andy Lutomirski wrote:
Good morning, I hope the week is starting well for everyone.
Quoted 7 lines
Quoted 7 lines · Dr. Greg
I can't bring myself to believe that LSM's are going to be written that will be making enclave security decisions on a page by page basis. Given what I have written above, I think all of this comes down to giving platform administrators one of three decisions, in order of most to least secure:
Quoted 6 lines
Quoted 1 line
1.) Block dynamic code loading and execution.
I don't understand what you're trying to say. Unless we're going to split enclaves into multiple VMAs with different permissions, how do you expect to block dynamic code loading unless you have separate RW and RX pages? That would be ???page-by-page???, right?
I believe that there are multiple knobs that can be manipulated to achieve the effects that are desired. I am advocating that we have a clear and reasoned understanding, with appropriate documentation, as to what we are doing and why we are doing it.
To advance this understanding, I would advocate that anyone interested in these issues review the papers that describe the design and use of the SGX2 architecture instructions. Fore convenience we have them available on our FTP server via the following links:
ftp://ftp.enjellic.com/pub/sgx/docs/hasp4.pdf
ftp://ftp.enjellic.com/pub/sgx/docs/hasp5.pdf
They are both from HASP 2016 and are readily available elsewhere as well.
The first paper discusses the SGX2 architecture instructions and why they were implemented. The second paper discusses how Enclave Dynamic Memory Management (EDMM) can be implemented from the OS perspective using these instructions.
Full disclosure, I've had the opportunity to have direct exchanges with a number of the authors on those papers.
The design of all this was heavily influenced by Marcus Peinado's group at Microsoft. They did the first implementation of an SGX library OS (Haven) and in the process addressed the issue of dynamic memory and code loading. Their work also resulted in the documentation of a number of errata in the SGX silicon implementation.
Like the company or not, I don't think the case can be made that they don't understand operating system design and theory. It would thus seem prudent to understand the prior art that went into the design of all this.
To summarize at a high level, EDMM requires a rather intricate choreography between enclave code and the device driver, the latter of which needs to execute ring-0 privileged instructions in order to expand the virtual address space of an enclave.
The easiest way to generically block dynamic code loading is to not allow the ENCLS[EAUG] instruction to be executed, the purpose of which is to augment a defined but sparse ELRANGE with additional physical pages from the EPC. It doesn't require ->mprotect or anything else, just a physical decision by the OS to not allow execution of that instruction.
All of which is consistent with my recomendation for a global disable knob on the kernel command-line for sites that do not want to tolerate completely anonymous code execution.
Since this driver does not yet support EDMM, the most immediate situation that we are dealing with are the potential security implications of SGX2 ENCLU instructions being executed inside of an enclave. The most principal issue is the ENCLU[EMODPE] instruction that allows a running enclave to extend the current permissions of a page.
I've been assuming that Sean's desire for ->mprotect is to block the ability of an initialized enclave, on a non-EDMM enabled driver, to collaborate with its untrusted component to self-modify its page permissions and thus allow execution of code that the operating system has no visibility into. That would make far more sense then the notion of someone trying to create an LSM that makes page by page security decisions.
The open question in all of this is that the EDMM paper, as well as the SDM, indicate the effects of an ENCLU[EMODPE] are immediate inside of a running enclave. I'm assuming that this does NOT mean that once a context of execution is running in enclave mode it would be capable of evading standard page protections but the 'immediate' is somewhat disquieting and probably deserves clarification, despite Dave Hansen's adament concerns about discussing the instruction... :-)
At the risk of being widely unpopular, it may be worth calling the question if we shouldn't bite the bullet and implement full SGX2 support in the driver. Given the trajectory that SGX is on it is what everyone wants and will be using. We are currently in somewhat of a bad place since we have to be cognizant of the implications of SGX2 hardware, which is going to be the only relevant platform for this driver, without actually addressing all of the issues it brings to the table.
Hopefully all of the above helps advance an understanding of the issues at hand.
Have a good day.
Dr. Greg
As always,
Dr. Greg Wettstein, Ph.D, Worker Autonomously self-defensive
Enjellic Systems Development, LLC IOT platforms and edge devices.
4206 N. 19th Ave.
Fargo, ND 58102
PH: 701-281-1686 EMAIL: [email protected]
------------------------------------------------------------------------------
"Five year projections, are you kidding me. We don't know what we are
supposed to be doing at the 4 o'clock meeting this afternoon."
-- Terry Wieland
Resurrection
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 11 lines · Dr. Greg
On Sat, Oct 24, 2020 at 08:33:21AM -0700, Andy Lutomirski wrote:
The easiest way to generically block dynamic code loading is to not allow the ENCLS[EAUG] instruction to be executed, the purpose of which is to augment a defined but sparse ELRANGE with additional physical pages from the EPC. It doesn't require ->mprotect or anything else, just a physical decision by the OS to not allow execution of that instruction.
I’m pretty sure that one can dynamically load code without EAUG. It
would require preallocation, but I can’t see why EAUG changes anything from a security policy perspective.
Quoted 12 lines
All of which is consistent with my recomendation for a global disable knob on the kernel command-line for sites that do not want to tolerate completely anonymous code execution.
Since this driver does not yet support EDMM, the most immediate situation that we are dealing with are the potential security implications of SGX2 ENCLU instructions being executed inside of an enclave. The most principal issue is the ENCLU[EMODPE] instruction that allows a running enclave to extend the current permissions of a page.
Quoted 7 lines
I've been assuming that Sean's desire for ->mprotect is to block the ability of an initialized enclave, on a non-EDMM enabled driver, to collaborate with its untrusted component to self-modify its page permissions and thus allow execution of code that the operating system has no visibility into. That would make far more sense then the notion of someone trying to create an LSM that makes page by page security decisions.
If you remove every mention of EDMM from that description, and you realize that the ability for LSMs to implement this sort of policy is basically the same as for the core SGX code to do so, then I agree.
The addition of EDMM will not change anything here per se, except that we’re a lot more likely to encounter enclaves doing interesting things with EMODPE once EDMM is enabled.
Quoted 8 lines
The open question in all of this is that the EDMM paper, as well as the SDM, indicate the effects of an ENCLU[EMODPE] are immediate inside of a running enclave. I'm assuming that this does NOT mean that once a context of execution is running in enclave mode it would be capable of evading standard page protections but the 'immediate' is somewhat disquieting and probably deserves clarification, despite Dave Hansen's adament concerns about discussing the instruction... :-)
If EMODPE writes an entry into the TLB that violates PTE permissions, then we have a real problem. I would be very surprised if this were to be the case.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
Quoted 11 lines · Andy Lutomirski
Quoted 8 lines · Dr. Greg
The open question in all of this is that the EDMM paper, as well as the SDM, indicate the effects of an ENCLU[EMODPE] are immediate inside of a running enclave. I'm assuming that this does NOT mean that once a context of execution is running in enclave mode it would be capable of evading standard page protections but the 'immediate' is somewhat disquieting and probably deserves clarification, despite Dave Hansen's adament concerns about discussing the instruction... :-)
If EMODPE writes an entry into the TLB that violates PTE permissions, then we have a real problem. I would be very surprised if this were to be the case.
EMODPE only affects the EPCM, it doesn't magically change the PTEs or insert into the TLB. The "immediate" wording in the whitepaper is differentiating it from EMODPR and EMODT, where the modifications only take effect after they have been verified by the enclave via EACCEPT.
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Dave HansenQuoted 28 lines · Dave Hansen
Quoted 27 lines · Sean Christopherson
Quoted 27 lines
In pseudo-C, it's something logically like this for the "nice" case:
ptr = mmap("/some/executable", PROT_EXEC); ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size); mmap(sgx_fd); EENTER;And we're trying to thwart:
ptr = mmap("/mnt/noexec/file", PROT_READ); ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, ptr, size); mmap(sgx_fd); EENTER;because that loads data into the enclave which is executable but which was not executable normally. But, we're allowing this from anonymous memory, so this would seem to work:
ptr = mmap("/mnt/noexec/file", PROT_READ); buffer = malloc(PAGE_SIZE); memcpy(buffer, ptr, PAGE_SIZE); // need mprotect(buf, PROT_EXEC)??? ioctl(sgx_fd, ADD_ENCLAVE_PAGE, SGX_PROT_EXEC, buffer, size); mmap(sgx_fd); EENTER;and give the same result. What am I missing?
Quoted 8 lines
Quoted 4 lines
The last example, where the enclave is copied to a buffer, is out of scope for noexec. But, it is in scope for LSMs, e.g. for this last example, we could add an LSM upcall so that SELinux could require
PROCESS_EXECMEM(or an SGX specific equivalent).Why don't we just declare enclave memory as "out of scope for noexec" in the same way that anonymous memory is, and just discard this patch? That doesn't seem too much of a stretch.
I did that already for v39. It unconditionally discards noexec partitions.
I see EMODPE as the key driver for this patch, not noexec partitions.
I.e. post you've done SGX_IOC_ENCLAVE_INIT you are capped when it comes to permissions.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Sean ChristophersonQuoted 14 lines · Sean Christopherson
Quoted 9 lines · Haitao Huang
Quoted 3 lines · Jarkko Sakkinen
I'll categorically deny noexec in the next patch set version.
/Jarkko
There are use cases supported currently in which enclave binary is received via IPC/RPC and held in buffers before EADD. Denying noexec altogether would break those, right?
No. noexec only applies to file-backed VMAs, what you're describing is loading an enclave from an anon VMA, which will still have
VM_MAYEXEC.
Quoted 6 lines
I believe you're thinking of SELinux's EXECMEM, which is required to execute from anonymous memory, and which we talked about (more than once) applying to SGX enclaves.
That being said, I still dislike the idea of requiring
VM_MAYEXEC, it's a hack that doesn't really buy us much, if anything.
I think it makes sense as long as it is not half-way there solution.
Either require it for the full binary or not at all. I'm fine with either.
/Jarkko
Re: [PATCH v38 10/24] mm: Add vm_ops->mprotect()
In reply to Haitao HuangQuoted 17 lines · Haitao Huang
Quoted 17 lines
For me this has caused months of confusion and misunderstanding of this feature. I only recently realized that "oh, right, we invented this".
They are contrived scenarios enough that they should be considered when the workloads hit.
Either we fully support noexec or not at all. Any "partial" thing is a two edged sword: it can bring some robustness with the price of complexity and possible unknown uknown scenarios where they might become API issue.
I rather think later on how to extend API in some way to enable such contrivid scenarios rather than worrying about how this could be abused.
The whole SGX is complex beast already so lets not add any extra when there is no a hard requirement to do so.
Quoted 7 lines
Quoted 3 lines
I'll categorically deny noexec in the next patch set version.
/Jarkko
There are use cases supported currently in which enclave binary is received via IPC/RPC and held in buffers before EADD. Denying noexec altogether would break those, right?
I do not see why data cannot be provided at run-time. AFAIK, it is not different from executables how this works when it comes to noexec.
/Jarkko
PATCH v38 11/24 x86/sgx: Add SGX enclave driver
In reply to Jarkko SakkinenIntel Software Guard eXtensions (SGX) is a set of CPU instructions that can be used by applications to set aside private regions of code and data. The code outside the SGX hosted software entity is prevented from accessing the memory inside the enclave by the CPU. We call these entities as enclaves.
Add a driver that provides an ioctl API to construct and run enclaves. Enclaves are constructed from pages residing in reserved physical memory areas. The contents of these pages can only be accessed when they are mapped as part of an enclave, by a hardware thread running inside the enclave.
The starting state of an enclave consists of a fixed measured set of pages that are copied to the EPC during the construction process by using the opcode ENCLS leaf functions and Software Enclave Control Structure (SECS) that defines the enclave properties.
Enclaves are constructed by using ENCLS leaf functions ECREATE, EADD and EINIT. ECREATE initializes SECS, EADD copies pages from system memory to the EPC and EINIT checks a given signed measurement and moves the enclave into a state ready for execution.
An initialized enclave can only be accessed through special Thread Control Structure (TCS) pages by using ENCLU (ring-3 only) leaf EENTER. This leaf function converts a thread into enclave mode and continues the execution in the offset defined by the TCS provided to EENTER. An enclave is exited through syscall, exception, interrupts or by explicitly calling another ENCLU leaf EEXIT.
The mmap() permissions are capped by the contained enclave page permissions. The mapped areas must also be populated, i.e. each page address must contain a page. This logic is implemented in sgx_encl_may_map().
--- arch/x86/kernel/cpu/sgx/Makefile | 2 + arch/x86/kernel/cpu/sgx/driver.c | 173 ++++++++++++++++ arch/x86/kernel/cpu/sgx/driver.h | 29 +++ arch/x86/kernel/cpu/sgx/encl.c | 331 +++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/encl.h | 85 ++++++++ arch/x86/kernel/cpu/sgx/main.c | 11 + 6 files changed, 631 insertions(+) create mode 100644 arch/x86/kernel/cpu/sgx/driver.c create mode 100644 arch/x86/kernel/cpu/sgx/driver.h create mode 100644 arch/x86/kernel/cpu/sgx/encl.c create mode 100644 arch/x86/kernel/cpu/sgx/encl.h
arch/x86/kernel/cpu/sgx/Makefile +2 -0
@@ -1,2 +1,4 @@ obj-y += \ + driver.o \ + encl.o \ main.o
arch/x86/kernel/cpu/sgx/driver.c new file +173 -0
173 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/driver.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) */ +#ifndef __ARCH_SGX_DRIVER_H__ +#define __ARCH_SGX_DRIVER_H__ + +#include <crypto/hash.h> +#include <linux/kref.h> +#include <linux/mmu_notifier.h> +#include <linux/radix-tree.h> +#include <linux/rwsem.h> +#include <linux/sched.h> +#include <linux/workqueue.h> +#include "sgx.h" + +#define SGX_EINIT_SPIN_COUNT 20 +#define SGX_EINIT_SLEEP_COUNT 50 +#define SGX_EINIT_SLEEP_TIME 20 + +extern u64 sgx_encl_size_max_32; +extern u64 sgx_encl_size_max_64; +extern u32 sgx_misc_reserved_mask; +extern u64 sgx_attributes_reserved_mask; +extern u64 sgx_xfrm_reserved_mask; +extern u32 sgx_xsave_size_tbl[64 ]; + +long sgx_ioctl (struct file *filep,unsigned int cmd,unsigned long arg); + +int sgx_drv_init (void ); + +#endif /* __ARCH_X86_SGX_DRIVER_H__ */
arch/x86/kernel/cpu/sgx/encl.c new file +331 -0
331 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/encl.h new file +85 -0
85 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/main.c +11 -0
@@ -9,6 +9,8 @@#include <linux/sched/mm.h> #include <linux/sched/signal.h> #include <linux/slab.h> +#include "driver.h" +#include "encl.h" #include "encls.h" struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS ];
@@ -260,6 +262,8 @@ static bool __init sgx_page_cache_init(void)static void __init sgx_init (void ) { +int ret; +if (!boot_cpu_has (X86_FEATURE_SGX ))return ;
@@ -269,8 +273,15 @@ static void __init sgx_init(void)if (!sgx_page_reclaimer_init ()) goto err_page_cache; + ret =sgx_drv_init (); +if (ret) + goto err_kthread; +return ; +err_kthread: +kthread_stop (ksgxswapd_tsk); + err_page_cache:sgx_page_cache_teardown (); }
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 11/24] x86/sgx: Add SGX enclave driver
Quoted 4 lines · Jarkko Sakkinen
Intel Software Guard eXtensions (SGX) is a set of CPU instructions that can be used by applications to set aside private regions of code and data. The code outside the SGX hosted software entity is prevented from accessing the memory inside the enclave by the CPU. We call these entities as enclaves.
It was correct before:
-memory inside the enclave by the CPU. We call these entities enclaves. +memory inside the enclave by the CPU. We call these entities as enclaves.
"as " is not needed.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 11/24] x86/sgx: Add SGX enclave driver
Quoted 11 lines · Borislav Petkov
Quoted 4 lines · Jarkko Sakkinen
Intel Software Guard eXtensions (SGX) is a set of CPU instructions that can be used by applications to set aside private regions of code and data. The code outside the SGX hosted software entity is prevented from accessing the memory inside the enclave by the CPU. We call these entities as enclaves.
It was correct before:
(quoted diff) +1
-1-memory inside the enclave by the CPU. We call these entities enclaves. +memory inside the enclave by the CPU. We call these entities as enclaves.
Quoted 1 line
"as " is not needed.
I reworded it. Thank you.
Quoted 6 lines
Signature · 5 lines
-- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Re: [PATCH v38 11/24] x86/sgx: Add SGX enclave driver
In reply to Jarkko SakkinenQuoted 18 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/driver.c new file +18
-0+int __init sgx_drv_init (void ) +{ +unsigned int eax, ebx, ecx, edx; +u64 attr_mask, xfrm_mask; +int ret; +int i; + +if (!boot_cpu_has (X86_FEATURE_SGX_LC )) { +pr_info ("The public key MSRs are not writable.\n" ); +return -ENODEV ; + } + +cpuid_count (SGX_CPUID ,0 , &eax, &ebx, &ecx, &edx); + sgx_misc_reserved_mask = ~ebx |SGX_MISC_RESERVED_MASK ; + sgx_encl_size_max_64 =1ULL << ((edx >>8 ) &0xFF ); + sgx_encl_size_max_32 =1ULL << (edx &0xFF ); + +cpuid_count (SGX_CPUID ,1 , &eax, &ebx, &ecx, &edx);
arch/x86/kernel/cpu/sgx/driver.c line 159
Quoted 10 lines
arch/x86/kernel/cpu/sgx/driver.c new file +10
-0+ + attr_mask = (((u64 )ebx) <<32 ) + (u64 )eax; + sgx_attributes_reserved_mask = ~attr_mask |SGX_ATTR_RESERVED_MASK ; + +if (boot_cpu_has (X86_FEATURE_OSXSAVE )) { + xfrm_mask = (((u64 )edx) <<32 ) + (u64 )ecx; + +for (i =2 ; i <64 ; i++) { +cpuid_count (0x0D , i, &eax, &ebx, &ecx, &edx); +if ((1UL << i) & xfrm_mask)
Any reason not to use BIT()? The max size computations are arguably not bit operation, but XFRM is a set of bits.
Quoted 14 lines
arch/x86/kernel/cpu/sgx/driver.c new file +14
-0+ sgx_xsave_size_tbl[i] = eax + ebx; + } + + sgx_xfrm_reserved_mask = ~xfrm_mask; + } + + ret =misc_register (&sgx_dev_enclave); +if (ret) { +pr_err ("Creating /dev/sgx/enclave failed with %d.\n" , ret); +return ret; + } + +return 0 ; +}
Re: [PATCH v38 11/24] x86/sgx: Add SGX enclave driver
Quoted 24 lines · Sean Christopherson
Quoted 23 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/driver.c new file +23
-0+int __init sgx_drv_init (void ) +{ +unsigned int eax, ebx, ecx, edx; +u64 attr_mask, xfrm_mask; +int ret; +int i; + +if (!boot_cpu_has (X86_FEATURE_SGX_LC )) { +pr_info ("The public key MSRs are not writable.\n" ); +return -ENODEV ; + } + +cpuid_count (SGX_CPUID ,0 , &eax, &ebx, &ecx, &edx); + sgx_misc_reserved_mask = ~ebx |SGX_MISC_RESERVED_MASK ; + sgx_encl_size_max_64 =1ULL << ((edx >>8 ) &0xFF ); + sgx_encl_size_max_32 =1ULL << (edx &0xFF ); + +cpuid_count (SGX_CPUID ,1 , &eax, &ebx, &ecx, &edx); + + attr_mask = (((u64 )ebx) <<32 ) + (u64 )eax; + sgx_attributes_reserved_mask = ~attr_mask |SGX_ATTR_RESERVED_MASK ; + +if (boot_cpu_has (X86_FEATURE_OSXSAVE )) {
Quoted 8 lines
arch/x86/kernel/cpu/sgx/driver.c line 159
Quoted 5 lines
arch/x86/kernel/cpu/sgx/driver.c new file +5
-0+ xfrm_mask = (((u64 )edx) <<32 ) + (u64 )ecx; + +for (i =2 ; i <64 ; i++) { +cpuid_count (0x0D , i, &eax, &ebx, &ecx, &edx); +if ((1UL << i) & xfrm_mask)Any reason not to use
BIT()? The max size computations are arguably not bit operation, but XFRM is a set of bits.
AFAIK, yes.
If you have bandwidth drop a patch for this and provision thing. I'm busy with the documentation. If not, that's fine, I'll do it once I'm able to.
Thanks.
/Jarkko
PATCH v38 12/24 x86/sgx: Add SGX_IOC_ENCLAVE_CREATE
In reply to Jarkko SakkinenAdd an ioctl that performs ENCLS[ECREATE], which creates SGX Enclave Control Structure for the enclave. SECS contains attributes about the enclave that are used by the hardware and cannot be directly accessed by software, as SECS resides in the EPC.
One essential field in SECS is a field that stores the SHA256 of the measured enclave pages. This field, MRENCLAVE, is initialized by the ECREATE instruction and updated by every EADD and EEXTEND operation. Finally, EINIT locks down the value.
--- .../userspace-api/ioctl/ioctl-number.rst | 1 + arch/x86/include/uapi/asm/sgx.h | 25 ++ arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/driver.c | 12 + arch/x86/kernel/cpu/sgx/driver.h | 1 + arch/x86/kernel/cpu/sgx/ioctl.c | 220 ++++++++++++++++++ 6 files changed, 260 insertions(+) create mode 100644 arch/x86/include/uapi/asm/sgx.h create mode 100644 arch/x86/kernel/cpu/sgx/ioctl.c
Documentation/userspace-api/ioctl/ioctl-number.rst +1 -0
@@ -323,6 +323,7 @@ Code Seq# Include File Comments <mailto:[email protected]> 0xA3 90-9F linux/dtlk.h 0xA4 00-1F uapi/linux/tee.h Generic TEE subsystem +0xA4 00-1F uapi/asm/sgx.h <mailto:[email protected]> 0xAA 00-3F linux/uapi/linux/userfaultfd.h 0xAB 00-1F linux/nbd.h 0xAC 00-1F linux/raw.h
arch/x86/include/uapi/asm/sgx.h new file +25 -0
@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */ +/* +* Copyright(c) 2016-19 Intel Corporation. +*/ +#ifndef _UAPI_ASM_X86_SGX_H +#define _UAPI_ASM_X86_SGX_H + +#include <linux/types.h> +#include <linux/ioctl.h> + +#define SGX_MAGIC 0xA4 + +#define SGX_IOC_ENCLAVE_CREATE \ + _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create) + +/** +* struct sgx_enclave_create - parameter structure for the +* %SGX_IOC_ENCLAVE_CREATE ioctl +* @src: address for the SECS page data +*/ +struct sgx_enclave_create { +__u64 src; +}; + +#endif /* _UAPI_ASM_X86_SGX_H */
arch/x86/kernel/cpu/sgx/Makefile +1 -0
@@ -1,4 +1,5 @@ obj-y += \ driver.o \ encl.o \ + ioctl.o \ main.o
arch/x86/kernel/cpu/sgx/driver.c +12 -0
@@ -114,10 +114,22 @@ static unsigned long sgx_get_unmapped_area(struct file *file,return current->mm->get_unmapped_area (file, addr, len, pgoff, flags); } +#ifdef CONFIG_COMPAT +static long sgx_compat_ioctl (struct file *filep,unsigned int cmd, +unsigned long arg) +{ +return sgx_ioctl (filep, cmd, arg); +} +#endif +static const struct file_operations sgx_encl_fops = { .owner =THIS_MODULE , .open = sgx_open, .release = sgx_release, + .unlocked_ioctl = sgx_ioctl, +#ifdef CONFIG_COMPAT + .compat_ioctl = sgx_compat_ioctl, +#endif .mmap = sgx_mmap, .get_unmapped_area = sgx_get_unmapped_area, };
arch/x86/kernel/cpu/sgx/driver.h +1 -0
@@ -9,6 +9,7 @@#include <linux/rwsem.h> #include <linux/sched.h> #include <linux/workqueue.h> +#include <uapi/asm/sgx.h> #include "sgx.h" #define SGX_EINIT_SPIN_COUNT 20
arch/x86/kernel/cpu/sgx/ioctl.c new file +220 -0
220 lines not on this page · raw message
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 12/24] x86/sgx: Add SGX_IOC_ENCLAVE_CREATE
Quoted 13 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/ioctl.c new file +13
-0+static int sgx_validate_secs (const struct sgx_secs *secs) +{ +u64 max_size = (secs->attributes &SGX_ATTR_MODE64BIT ) ? + sgx_encl_size_max_64 : sgx_encl_size_max_32; + +if (secs->size < (2 *PAGE_SIZE ) || !is_power_of_2 (secs->size)) +return -EINVAL ; + +if (secs->base & (secs->size -1 )) +return -EINVAL ; + + if (secs->miscselect & sgx_misc_reserved_mask || + secs->attributes & sgx_attributes_reserved_mask ||
arch/x86/kernel/cpu/sgx/ioctl.c line 63
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +10
-0+ secs->xfrm & sgx_xfrm_reserved_mask) +return -EINVAL ; + +if (secs->size > max_size) +return -EINVAL ; + +if (!(secs->xfrm &XFEATURE_MASK_FP ) || + !(secs->xfrm &XFEATURE_MASK_SSE ) || + (((secs->xfrm >>XFEATURE_BNDREGS ) &1 ) != + ((secs->xfrm >>XFEATURE_BNDCSR ) &1 )))
Let that last line stick out so that you have each statement on a single line.
arch/x86/kernel/cpu/sgx/ioctl.c line 70
Quoted 7 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +7
-0+return -EINVAL ; + +if (!secs->ssa_frame_size) +return -EINVAL ; + +if (sgx_calc_ssa_frame_size (secs->miscselect, secs->xfrm) > + secs->ssa_frame_size)
Let that stick out.
Quoted 4 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +4
-0+return -EINVAL ; + + if (memchr_inv(secs->reserved1, 0, sizeof(secs->reserved1)) || + memchr_inv(secs->reserved2, 0, sizeof(secs->reserved2)) ||
arch/x86/kernel/cpu/sgx/ioctl.c line 84
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +10
-0+memchr_inv (secs->reserved3,0 ,sizeof (secs->reserved3)) || + memchr_inv(secs->reserved4, 0, sizeof(secs->reserved4))) + return -EINVAL; + +return 0 ; +} + +static int sgx_encl_create (struct sgx_encl *encl,struct sgx_secs *secs) +{ +unsigned long encl_size = secs->size +PAGE_SIZE ;
You're still using secs->size before validation. I know, we will return early if sgx_validate_secs() fails but pls move that addition after the validation call.
...
Quoted 3 lines
+/**
+ *sgx_ioc_enclave_create- handler for %SGX_IOC_ENCLAVE_CREATE
+ * @filep: open file to/dev/sgx
Dammit, how many times do I have to type this comment here?!
"That's
@encl: enclave pointer
or so."
There's no filep - there is an encl!
Quoted 4 lines
+ * @arg: userspace pointer to a struct
sgx_enclave_createinstance
+ *
+ * Allocate kernel data structures for a new enclave and execute ECREATE after
+ * verifying the correctness of the provided SECS.
... which is done in sgx_validate_secs()."
Yes, spell it out, otherwise one has to wonder where that validation is happening in the function *below* because the comment is over it - not over sgx_validate_secs().
And yes, you need to spell stuff like that out because this SGX crap is complex and it better be properly documented!
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 12/24] x86/sgx: Add SGX_IOC_ENCLAVE_CREATE
Quoted 47 lines · Borislav Petkov
Quoted 13 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/ioctl.c new file +13
-0+static int sgx_validate_secs (const struct sgx_secs *secs) +{ +u64 max_size = (secs->attributes &SGX_ATTR_MODE64BIT ) ? + sgx_encl_size_max_64 : sgx_encl_size_max_32; + +if (secs->size < (2 *PAGE_SIZE ) || !is_power_of_2 (secs->size)) +return -EINVAL ; + +if (secs->base & (secs->size -1 )) +return -EINVAL ; + + if (secs->miscselect & sgx_misc_reserved_mask || + secs->attributes & sgx_attributes_reserved_mask ||arch/x86/kernel/cpu/sgx/ioctl.c line 63
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +10
-0+ secs->xfrm & sgx_xfrm_reserved_mask) +return -EINVAL ; + +if (secs->size > max_size) +return -EINVAL ; + +if (!(secs->xfrm &XFEATURE_MASK_FP ) || + !(secs->xfrm &XFEATURE_MASK_SSE ) || + (((secs->xfrm >>XFEATURE_BNDREGS ) &1 ) != + ((secs->xfrm >>XFEATURE_BNDCSR ) &1 )))Let that last line stick out so that you have each statement on a single line.
arch/x86/kernel/cpu/sgx/ioctl.c line 70
Quoted 7 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +7
-0+return -EINVAL ; + +if (!secs->ssa_frame_size) +return -EINVAL ; + +if (sgx_calc_ssa_frame_size (secs->miscselect, secs->xfrm) > + secs->ssa_frame_size)Let that stick out.
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +10
-0+return -EINVAL ; + +if (memchr_inv (secs->reserved1,0 ,sizeof (secs->reserved1)) || +memchr_inv (secs->reserved2,0 ,sizeof (secs->reserved2)) || +memchr_inv (secs->reserved3,0 ,sizeof (secs->reserved3)) || +memchr_inv (secs->reserved4,0 ,sizeof (secs->reserved4))) +return -EINVAL ; + +return 0 ; +}
Quoted 8 lines
arch/x86/kernel/cpu/sgx/ioctl.c line 84
Quoted 4 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +4
-0+ +static int sgx_encl_create (struct sgx_encl *encl,struct sgx_secs *secs) +{ +unsigned long encl_size = secs->size +PAGE_SIZE ;You're still using
secs->sizebefore validation. I know, we will return early ifsgx_validate_secs()fails but pls move that addition after the validation call.
Is this appropriate:
/* The extra page in swap space goes to SECS. */
encl_size = secs->size + PAGE_SIZE;
backing = shmem_file_setup("SGX backing", encl_size + (encl_size >> 5),
VM_NORESERVE);
if (IS_ERR(backing)) {
ret = PTR_ERR(backing);
goto err_out_shrink;
}Quoted 8 lines
...
Quoted 3 lines
+/**
+ *sgx_ioc_enclave_create- handler for %SGX_IOC_ENCLAVE_CREATE
+ * @filep: open file to/dev/sgxDammit, how many times do I have to type this comment here?!
Quoted 7 lines
"That's
@encl: enclave pointer
or so."
There's no filep - there is an encl!
I'm not actually sure what has happened. As you can easily grep, the rename is done in five other sites. I also see a similar problem in EINIT, which I will fix.
git grep "enclave pointer" arch/x86/kernel/cpu/sgx/ | wc -l 5
Quoted 5 lines
Quoted 4 lines
+ * @arg: userspace pointer to a struct
sgx_enclave_createinstance
+ *
+ * Allocate kernel data structures for a new enclave and execute ECREATE after
+ * verifying the correctness of the provided SECS.
Quoted 8 lines
... which is done in
sgx_validate_secs()."Yes, spell it out, otherwise one has to wonder where that validation is happening in the function *below* because the comment is over it - not over
sgx_validate_secs().And yes, you need to spell stuff like that out because this SGX crap is complex and it better be properly documented!
I agree with this but I also think it would make sense to rephrase "verifying the correctness of the provided SECS" with something more informative.
I would rephrase as:
"... after checking that the provided data for SECS meets the expectations of ENCLS[ECREATE] for an unitialized enclave and size of the address space does not surpass the platform expectations. This validation is executed by sgx_validate_secs()."
Is this sufficient for you, or do you have further suggestions?
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 12/24] x86/sgx: Add SGX_IOC_ENCLAVE_CREATE
Quoted 12 lines · Jarkko Sakkinen
Is this appropriate:
/* The extra page in swap space goes to SECS. */ encl_size = secs->size + PAGE_SIZE; backing = shmem_file_setup("SGX backing", encl_size + (encl_size >> 5), VM_NORESERVE); if (IS_ERR(backing)) { ret = PTR_ERR(backing); goto err_out_shrink; }
Yap, thanks.
Quoted 10 lines
I agree with this but I also think it would make sense to rephrase "verifying the correctness of the provided SECS" with something more informative.
I would rephrase as:
"... after checking that the provided data for SECS meets the expectations of ENCLS[ECREATE] for an unitialized enclave and size of the address space does not surpass the platform expectations. This validation is executed by
sgx_validate_secs()."
s/executed/done/
Quoted 1 line
Is this sufficient for you, or do you have further suggestions?
Yes, no further suggestions.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 12/24] x86/sgx: Add SGX_IOC_ENCLAVE_CREATE
Quoted 2 lines · Borislav Petkov
Quoted 2 lines
"... after checking that the provided data for SECS meets the expectations of ENCLS[ECREATE] for an unitialized enclave and size of the address
There is a typo (should be uninitialized).
Quoted 5 lines
Quoted 1 line
Is this sufficient for you, or do you have further suggestions?
Yes, no further suggestions.
Thx.
Great, thank you.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Re: [PATCH v38 12/24] x86/sgx: Add SGX_IOC_ENCLAVE_CREATE
In reply to Jarkko SakkinenQuoted 182 lines · Jarkko Sakkinen
Add an ioctl that performs ENCLS[ECREATE], which creates SGX Enclave Control Structure for the enclave. SECS contains attributes about the enclave that are used by the hardware and cannot be directly accessed by software, as SECS resides in the EPC.
One essential field in SECS is a field that stores the SHA256 of the measured enclave pages. This field, MRENCLAVE, is initialized by the ECREATE instruction and updated by every EADD and EEXTEND operation. Finally, EINIT locks down the value.
Acked-by Jethro Beekman <[email protected]>Tested-by Jethro Beekman <[email protected]>Tested-by Haitao Huang <[email protected]>Tested-by Chunyang Hui <[email protected]>Tested-by Jordan Hand <[email protected]>Tested-by Nathaniel McCallum <[email protected]>Tested-by Seth Moore <[email protected]>Tested-by Darren Kenny <[email protected]>Reviewed-by Darren Kenny <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Co-developed-by Suresh Siddha <[email protected]>Signed-off-by Suresh Siddha <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>--- .../userspace-api/ioctl/ioctl-number.rst | 1 + arch/x86/include/uapi/asm/sgx.h | 25 ++ arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/driver.c | 12 + arch/x86/kernel/cpu/sgx/driver.h | 1 + arch/x86/kernel/cpu/sgx/ioctl.c | 220 ++++++++++++++++++ 6 files changed, 260 insertions(+) create mode 100644 arch/x86/include/uapi/asm/sgx.h create mode 100644 arch/x86/kernel/cpu/sgx/ioctl.cDocumentation/userspace-api/ioctl/ioctl-number.rst +0
-0@@ -323,6 +323,7 @@ Code Seq# IncludeFile Comments
<mailto:[email protected]>0xA3 90-9F linux/dtlk.h 0xA4 00-1F uapi/linux/tee.h Generic TEE subsystem +0xA4 00-1F uapi/asm/sgx.h <mailto:[email protected]> 0xAA 00-3F linux/uapi/linux/userfaultfd.h 0xAB 00-1F linux/nbd.h 0xAC 00-1F linux/raw.harch/x86/include/uapi/asm/sgx.h new file +1
-0@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */
arch/x86/include/uapi/asm/sgx.h new file +24
-0+/* +* Copyright(c) 2016-19 Intel Corporation. +*/ +#ifndef _UAPI_ASM_X86_SGX_H +#define _UAPI_ASM_X86_SGX_H + +#include <linux/types.h> +#include <linux/ioctl.h> + +#define SGX_MAGIC 0xA4 + +#define SGX_IOC_ENCLAVE_CREATE \ + _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create) + +/** +* struct sgx_enclave_create - parameter structure for the +* %SGX_IOC_ENCLAVE_CREATE ioctl +* @src: address for the SECS page data +*/ +struct sgx_enclave_create { +__u64 src; +}; + +#endif /* _UAPI_ASM_X86_SGX_H */ arch/x86/kernel/cpu/sgx/Makefile +1
-0@@ -1,4 +1,5 @@ obj-y += \ driver.o \ encl.o \ + ioctl.o \ main.oarch/x86/kernel/cpu/sgx/driver.c +12
-0@@ -114,10 +114,22 @@ static unsigned long sgx_get_unmapped_area(structfile *file, return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); } +#ifdef CONFIG_COMPAT +static long sgx_compat_ioctl (struct file *filep,unsigned int cmd, +unsigned long arg) +{ +return sgx_ioctl (filep, cmd, arg); +} +#endif +static const struct file_operations sgx_encl_fops = { .owner =THIS_MODULE , .open = sgx_open, .release = sgx_release, + .unlocked_ioctl = sgx_ioctl, +#ifdef CONFIG_COMPAT + .compat_ioctl = sgx_compat_ioctl, +#endif .mmap = sgx_mmap, .get_unmapped_area = sgx_get_unmapped_area, };arch/x86/kernel/cpu/sgx/driver.h +1
-0@@ -9,6 +9,7 @@#include <linux/rwsem.h> #include <linux/sched.h> #include <linux/workqueue.h> +#include <uapi/asm/sgx.h> #include "sgx.h" #define SGX_EINIT_SPIN_COUNT 20arch/x86/kernel/cpu/sgx/ioctl.c new file +46
-0@@ -0,0 +1,220 @@ +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) +// Copyright(c) 2016-19 Intel Corporation. + +#include <asm/mman.h> +#include <linux/mman.h> +#include <linux/delay.h> +#include <linux/file.h> +#include <linux/hashtable.h> +#include <linux/highmem.h> +#include <linux/ratelimit.h> +#include <linux/sched/signal.h> +#include <linux/shmem_fs.h> +#include <linux/slab.h> +#include <linux/suspend.h> +#include "driver.h" +#include "encl.h" +#include "encls.h" + +static u32 sgx_calc_ssa_frame_size (u32 miscselect,u64 xfrm) +{ +u32 size_max =PAGE_SIZE ; +u32 size; +int i; + +for (i =2 ; i <64 ; i++) { +if (!((1 << i) & xfrm)) +continue ; + + size =SGX_SSA_GPRS_SIZE + sgx_xsave_size_tbl[i]; + +if (miscselect &SGX_MISC_EXINFO ) + size +=SGX_SSA_MISC_EXINFO_SIZE ; + +if (size > size_max) + size_max = size; + } + +return PFN_UP (size_max); +} + +static int sgx_validate_secs (const struct sgx_secs *secs) +{ +u64 max_size = (secs->attributes &SGX_ATTR_MODE64BIT ) ? + sgx_encl_size_max_64 : sgx_encl_size_max_32; + +if (secs->size < (2 *PAGE_SIZE ) || !is_power_of_2 (secs->size))
arch/x86/kernel/cpu/sgx/ioctl.c line 56
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +10
-0+return -EINVAL ; + +if (secs->base & (secs->size -1 )) +return -EINVAL ; + +if (secs->miscselect & sgx_misc_reserved_mask || + secs->attributes & sgx_attributes_reserved_mask || + secs->xfrm & sgx_xfrm_reserved_mask) +return -EINVAL ; +
Attributes should not be enforced against what's available on platform but checked and enforced against the mask for enforcement in sigstruct. For example an enclave could opt to sign with AVX bit in xfrm, but still be loadable on a platform without it if the sigstruct->body.attributes_mask does not turn that bit on.
Suggest to remove the above check,and do following during EINIt in sgx_encl_init:
- /* Check that the required attributes have been authorized. */
+ /* Check that the requested attributes have been authorized. */
if (encl->secs_attributes & ~encl->allowed_attributes)
return -EACCES;+ /* Check that mandatory features are supported. */
+ if (sigstruct->body.attributes & sigstruct->body.attributes_mask &
+ sgx_attributes_reserved_mask)
+ return -EINVAL;
+ if (sigstruct->body.miscselect & sigstruct->body.misc_mask &
+ sgx_misc_reserved_mask)
+ return -EINVAL;
+ if (sigstruct->body.xfrm & sigstruct->body.xfrm_mask &
+ sgx_xfrm_reserved_mask)
+ return -EINVAL;
+
ret = sgx_get_key_hash(sigstruct->modulus, mrsigner);
Haitao
Re: [PATCH v38 12/24] x86/sgx: Add SGX_IOC_ENCLAVE_CREATE
Quoted 211 lines · Haitao Huang
Quoted 188 lines · Jarkko Sakkinen
Add an ioctl that performs ENCLS[ECREATE], which creates SGX Enclave Control Structure for the enclave. SECS contains attributes about the enclave that are used by the hardware and cannot be directly accessed by software, as SECS resides in the EPC.
One essential field in SECS is a field that stores the SHA256 of the measured enclave pages. This field, MRENCLAVE, is initialized by the ECREATE instruction and updated by every EADD and EEXTEND operation. Finally, EINIT locks down the value.
Acked-by Jethro Beekman <[email protected]>Tested-by Jethro Beekman <[email protected]>Tested-by Haitao Huang <[email protected]>Tested-by Chunyang Hui <[email protected]>Tested-by Jordan Hand <[email protected]>Tested-by Nathaniel McCallum <[email protected]>Tested-by Seth Moore <[email protected]>Tested-by Darren Kenny <[email protected]>Reviewed-by Darren Kenny <[email protected]>Co-developed-by Sean Christopherson <[email protected]>Signed-off-by Sean Christopherson <[email protected]>Co-developed-by Suresh Siddha <[email protected]>Signed-off-by Suresh Siddha <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>--- .../userspace-api/ioctl/ioctl-number.rst | 1 + arch/x86/include/uapi/asm/sgx.h | 25 ++ arch/x86/kernel/cpu/sgx/Makefile | 1 + arch/x86/kernel/cpu/sgx/driver.c | 12 + arch/x86/kernel/cpu/sgx/driver.h | 1 + arch/x86/kernel/cpu/sgx/ioctl.c | 220 ++++++++++++++++++ 6 files changed, 260 insertions(+) create mode 100644 arch/x86/include/uapi/asm/sgx.h create mode 100644 arch/x86/kernel/cpu/sgx/ioctl.cDocumentation/userspace-api/ioctl/ioctl-number.rst +0
-0@@ -323,6 +323,7 @@ Code Seq# Include FileComments
<mailto:[email protected]>0xA3 90-9F linux/dtlk.h 0xA4 00-1F uapi/linux/tee.h Generic TEE subsystem +0xA4 00-1F uapi/asm/sgx.h <mailto:[email protected]> 0xAA 00-3F linux/uapi/linux/userfaultfd.h 0xAB 00-1F linux/nbd.h 0xAC 00-1F linux/raw.harch/x86/include/uapi/asm/sgx.h new file +1
-0@@ -0,0 +1,25 @@ +/* SPDX-License-Identifier: ((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) */
arch/x86/include/uapi/asm/sgx.h new file +24
-0+/* +* Copyright(c) 2016-19 Intel Corporation. +*/ +#ifndef _UAPI_ASM_X86_SGX_H +#define _UAPI_ASM_X86_SGX_H + +#include <linux/types.h> +#include <linux/ioctl.h> + +#define SGX_MAGIC 0xA4 + +#define SGX_IOC_ENCLAVE_CREATE \ + _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create) + +/** +* struct sgx_enclave_create - parameter structure for the +* %SGX_IOC_ENCLAVE_CREATE ioctl +* @src: address for the SECS page data +*/ +struct sgx_enclave_create { +__u64 src; +}; + +#endif /* _UAPI_ASM_X86_SGX_H */ arch/x86/kernel/cpu/sgx/Makefile +1
-0@@ -1,4 +1,5 @@ obj-y += \ driver.o \ encl.o \ + ioctl.o \ main.oarch/x86/kernel/cpu/sgx/driver.c +12
-0@@ -114,10 +114,22 @@ static unsigned long sgx_get_unmapped_area(structfile *file, return current->mm->get_unmapped_area(file, addr, len, pgoff, flags); } +#ifdef CONFIG_COMPAT +static long sgx_compat_ioctl (struct file *filep,unsigned int cmd, +unsigned long arg) +{ +return sgx_ioctl (filep, cmd, arg); +} +#endif +static const struct file_operations sgx_encl_fops = { .owner =THIS_MODULE , .open = sgx_open, .release = sgx_release, + .unlocked_ioctl = sgx_ioctl, +#ifdef CONFIG_COMPAT + .compat_ioctl = sgx_compat_ioctl, +#endif .mmap = sgx_mmap, .get_unmapped_area = sgx_get_unmapped_area, };arch/x86/kernel/cpu/sgx/driver.h +1
-0@@ -9,6 +9,7 @@#include <linux/rwsem.h> #include <linux/sched.h> #include <linux/workqueue.h> +#include <uapi/asm/sgx.h> #include "sgx.h" #define SGX_EINIT_SPIN_COUNT 20arch/x86/kernel/cpu/sgx/ioctl.c new file +46
-0@@ -0,0 +1,220 @@ +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) +// Copyright(c) 2016-19 Intel Corporation. + +#include <asm/mman.h> +#include <linux/mman.h> +#include <linux/delay.h> +#include <linux/file.h> +#include <linux/hashtable.h> +#include <linux/highmem.h> +#include <linux/ratelimit.h> +#include <linux/sched/signal.h> +#include <linux/shmem_fs.h> +#include <linux/slab.h> +#include <linux/suspend.h> +#include "driver.h" +#include "encl.h" +#include "encls.h" + +static u32 sgx_calc_ssa_frame_size (u32 miscselect,u64 xfrm) +{ +u32 size_max =PAGE_SIZE ; +u32 size; +int i; + +for (i =2 ; i <64 ; i++) { +if (!((1 << i) & xfrm)) +continue ; + + size =SGX_SSA_GPRS_SIZE + sgx_xsave_size_tbl[i]; + +if (miscselect &SGX_MISC_EXINFO ) + size +=SGX_SSA_MISC_EXINFO_SIZE ; + +if (size > size_max) + size_max = size; + } + +return PFN_UP (size_max); +} + +static int sgx_validate_secs (const struct sgx_secs *secs) +{ +u64 max_size = (secs->attributes &SGX_ATTR_MODE64BIT ) ? + sgx_encl_size_max_64 : sgx_encl_size_max_32; + +if (secs->size < (2 *PAGE_SIZE ) || !is_power_of_2 (secs->size))arch/x86/kernel/cpu/sgx/ioctl.c line 56
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c new file +10
-0+return -EINVAL ; + +if (secs->base & (secs->size -1 )) +return -EINVAL ; + +if (secs->miscselect & sgx_misc_reserved_mask || + secs->attributes & sgx_attributes_reserved_mask || + secs->xfrm & sgx_xfrm_reserved_mask) +return -EINVAL ; +Attributes should not be enforced against what's available on platform but checked and enforced against the mask for enforcement in sigstruct. For example an enclave could opt to sign with AVX bit in xfrm, but still be loadable on a platform without it if the
sigstruct->body.attributes_maskdoes not turn that bit on.Suggest to remove the above check,and do following during EINIt in
sgx_encl_init:
Review
Quoted 2 lines
(quoted diff) +1
-1- /* Check that the required attributes have been authorized. */ + /* Check that the requested attributes have been authorized. */
This is as bad and useless comment as the existing was. In the paragraph you wrote above you state the things that are useful.
So: I'll take the code change and put the response above here instead because it is says the right things :-) (maybe with very minor edit)
Quoted 16 lines
if (encl->secs_attributes & ~encl->allowed_attributes) return -EACCES;(quoted diff) +11
-0+ /* Check that mandatory features are supported. */ + if (sigstruct->body.attributes & sigstruct->body.attributes_mask & + sgx_attributes_reserved_mask) + return -EINVAL; + if (sigstruct->body.miscselect & sigstruct->body.misc_mask & + sgx_misc_reserved_mask) + return -EINVAL; + if (sigstruct->body.xfrm & sigstruct->body.xfrm_mask & + sgx_xfrm_reserved_mask) + return -EINVAL; + ret = sgx_get_key_hash(sigstruct->modulus, mrsigner);
Quoted 1 line
Haitao
Thanks a lot!
/Jarkko
PATCH v38 13/24 x86/sgx: Add SGX_IOC_ENCLAVE_ADD_PAGES
In reply to Jarkko SakkinenAdd an ioctl, which performs ENCLS[EADD] that adds new visible page to an enclave, and optionally ENCLS[EEXTEND] operations that hash the page to the enclave measurement. By visible we mean a page that can be mapped to the address range of an enclave.
--- arch/x86/include/uapi/asm/sgx.h | 28 +++ arch/x86/kernel/cpu/sgx/ioctl.c | 294 ++++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 1 + 3 files changed, 323 insertions(+)
arch/x86/include/uapi/asm/sgx.h +28 -0
@@ -8,10 +8,21 @@#include <linux/types.h> #include <linux/ioctl.h> +/** +* enum sgx_epage_flags - page control flags +* %SGX_PAGE_MEASURE: Measure the page contents with a sequence of +* ENCLS[EEXTEND] operations. +*/ +enum sgx_page_flags { +SGX_PAGE_MEASURE =0x01 , +}; +#define SGX_MAGIC 0xA4#define SGX_IOC_ENCLAVE_CREATE \ _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create) +#define SGX_IOC_ENCLAVE_ADD_PAGES \ + _IOWR(SGX_MAGIC, 0x01, struct sgx_enclave_add_pages)/** * struct sgx_enclave_create - parameter structure for the
@@ -22,4 +33,21 @@ struct sgx_enclave_create { __u64 src; }; +/** +* struct sgx_enclave_add_pages - parameter structure for the +* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl +* @src: start address for the page data +* @offset: starting page offset +* @length: length of the data (multiple of the page size) +* @secinfo: address for the SECINFO data +* @flags: page control flags +*/ +struct sgx_enclave_add_pages { +__u64 src; +__u64 offset; +__u64 length; +__u64 secinfo; +__u64 flags; +}; +#endif /* _UAPI_ASM_X86_SGX_H */
arch/x86/kernel/cpu/sgx/ioctl.c +294 -0
306 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/sgx.h +1 -0
@@ -34,6 +34,7 @@ struct sgx_epc_section {#define SGX_EPC_SECTION_MASK GENMASK(7, 0)#define SGX_MAX_EPC_SECTIONS (SGX_EPC_SECTION_MASK + 1) +#define SGX_MAX_ADD_PAGES_LENGTH 0x100000extern struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS ];
Signature · 1 line
2.25.1
PATCH v38 14/24 x86/sgx: Add SGX_IOC_ENCLAVE_INIT
In reply to Jarkko SakkinenAdd an ioctl that performs ENCLS[EINIT], which locks down the measurement and initializes the enclave for entrance. After this, new pages can no longer be added.
--- arch/x86/include/uapi/asm/sgx.h | 11 ++ arch/x86/kernel/cpu/sgx/encl.h | 2 + arch/x86/kernel/cpu/sgx/ioctl.c | 195 ++++++++++++++++++++++++++++++++ 3 files changed, 208 insertions(+)
arch/x86/include/uapi/asm/sgx.h +11 -0
@@ -23,6 +23,8 @@ enum sgx_page_flags { _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create)#define SGX_IOC_ENCLAVE_ADD_PAGES \ _IOWR(SGX_MAGIC, 0x01, struct sgx_enclave_add_pages) +#define SGX_IOC_ENCLAVE_INIT \ + _IOW(SGX_MAGIC, 0x02, struct sgx_enclave_init)/** * struct sgx_enclave_create - parameter structure for the
@@ -50,4 +52,13 @@ struct sgx_enclave_add_pages { __u64 flags; }; +/** +* struct sgx_enclave_init - parameter structure for the +* %SGX_IOC_ENCLAVE_INIT ioctl +* @sigstruct: address for the SIGSTRUCT data +*/ +struct sgx_enclave_init { +__u64 sigstruct; +}; +#endif /* _UAPI_ASM_X86_SGX_H */
arch/x86/kernel/cpu/sgx/encl.h +2 -0
@@ -70,6 +70,8 @@ struct sgx_encl {struct xarray page_array;struct sgx_encl_page secs;cpumask_t cpumask; +unsigned long attributes; +unsigned long attributes_mask; };extern const struct vm_operations_struct sgx_vm_ops;
arch/x86/kernel/cpu/sgx/ioctl.c +195 -0
219 lines not on this page · raw message
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 536
Quoted 7 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/ioctl.c +7
-0+static int sgx_einit (struct sgx_sigstruct *sigstruct,void *token, +struct sgx_epc_page *secs,u64 *lepubkeyhash) +{ +int ret; + +preempt_disable (); +sgx_update_lepubkeyhash_msrs (lepubkeyhash, false);
So this will update the cached copies *and* the MSRs itself if what's cached is stale...
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 538
Quoted 2 lines
arch/x86/kernel/cpu/sgx/ioctl.c +2
-0+ ret =__einit (sigstruct, token,sgx_get_epc_addr (secs)); +if (ret ==SGX_INVALID_EINITTOKEN ) {
... so why would it return this error here?
Definition of this error says:
* %SGX_INVALID_EINITTOKEN: EINITTOKEN is invalid and enclave signer's
* public key does not match IA32_SGXLEPUBKEYHASH.
when you just updated them?!
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 539
Quoted 1 line
arch/x86/kernel/cpu/sgx/ioctl.c +1
-0+sgx_update_lepubkeyhash_msrs (lepubkeyhash, true);
So why force a second time?
Quoted 7 lines
arch/x86/kernel/cpu/sgx/ioctl.c +7
-0+ ret =__einit (sigstruct, token,sgx_get_epc_addr (secs)); + } +preempt_enable (); +return ret; +} + +static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 556
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c +10
-0+ void *token) +{ +u64 mrsigner[4 ]; +int ret; +int i; +int j; + +/* Deny initializing enclaves with attributes (namely provisioning) +* that have not been explicitly allowed. +*/
Comments style is with the first line empty:
/*
* A sentence ending with a full-stop.
* Another sentence. ...
* More sentences. ...
*/
Quoted 48 lines
arch/x86/kernel/cpu/sgx/ioctl.c +48
-0+if (encl->attributes & ~encl->attributes_mask) +return -EACCES ; + + ret =sgx_get_key_hash (sigstruct->modulus, mrsigner); +if (ret) +return ret; + +mutex_lock (&encl->lock); + +/* +* ENCLS[EINIT] is interruptible because it has such a high latency, +* e.g. 50k+ cycles on success. If an IRQ/NMI/SMI becomes pending, +* EINIT may fail with SGX_UNMASKED_EVENT so that the event can be +* serviced. +*/ +for (i =0 ; i <SGX_EINIT_SLEEP_COUNT ; i++) { +for (j =0 ; j <SGX_EINIT_SPIN_COUNT ; j++) { + ret =sgx_einit (sigstruct, token, encl->secs.epc_page, + mrsigner); +if (ret ==SGX_UNMASKED_EVENT ) +continue ; +else +break ; + } + +if (ret !=SGX_UNMASKED_EVENT ) +break ; + +msleep_interruptible (SGX_EINIT_SLEEP_TIME ); + +if (signal_pending (current)) { + ret = -ERESTARTSYS ; + goto err_out; + } + } + +if (ret &ENCLS_FAULT_FLAG ) { +if (encls_failed (ret)) +ENCLS_WARN (ret,"EINIT" ); + +sgx_encl_destroy (encl); + ret = -EFAULT ; + }else if (ret) { +pr_debug ("EINIT returned %d\n" , ret); + ret = -EPERM ; + }else { +atomic_or (SGX_ENCL_INITIALIZED , &encl->flags); + }
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 614
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c +10
-0+ +err_out: +mutex_unlock (&encl->lock); +return ret; +} + +/** +* sgx_ioc_enclave_init - handler for %SGX_IOC_ENCLAVE_INIT +* +* @filep: open file to /dev/sgx
Aaand again:
"@encl: pointer to an enclave instance (via ioctl() file pointer)"
this is also from a previous review.
Quoted 13 lines
+ * @arg: userspace pointer to a struct
sgx_enclave_initinstance
+ *
+ * Flush any outstanding enqueued EADD operations and perform EINIT. The
+ * Launch Enclave Public Key Hash MSRs are rewritten as necessary to match
+ * the enclave's MRSIGNER, which is caculated from the provided sigstruct.
+ *
+ * Return:+ * 0 on success, + * SGX error code on EINIT failure, + * -errno otherwise+ */
arch/x86/kernel/cpu/sgx/ioctl.c +2
-0+static long sgx_ioc_enclave_init (struct sgx_encl *encl,void __user *arg) +{
...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
Quoted 19 lines · Borislav Petkov
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 536
Quoted 7 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/ioctl.c +7
-0+static int sgx_einit (struct sgx_sigstruct *sigstruct,void *token, +struct sgx_epc_page *secs,u64 *lepubkeyhash) +{ +int ret; + +preempt_disable (); +sgx_update_lepubkeyhash_msrs (lepubkeyhash, false);So this will update the cached copies *and* the MSRs itself if what's cached is stale...
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 538
Quoted 2 lines
arch/x86/kernel/cpu/sgx/ioctl.c +2
-0+ ret =__einit (sigstruct, token,sgx_get_epc_addr (secs)); +if (ret ==SGX_INVALID_EINITTOKEN ) {... so why would it return this error here?
Definition of this error says:
Quoted 8 lines
* %SGX_INVALID_EINITTOKEN: EINITTOKEN is invalid and enclave signer's
* public key does not matchIA32_SGXLEPUBKEYHASH.when you just updated them?!
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 539
Quoted 1 line
arch/x86/kernel/cpu/sgx/ioctl.c +1
-0+sgx_update_lepubkeyhash_msrs (lepubkeyhash, true);So why force a second time?
The LE pubkey hash MSRs are special snowflakes. They get reset to Intel's default key on any loss of EPC, e.g. if the system does a suspend/resume cycle. The approach we took (obviously) is to assume the kernel's cache can be stale at any given time. The alternative would be to try and track loss of EPC conditions and emulate the reset, but that's a bit dicey on bare metal as any missed case would hose SGX, and in a VM it's theoretically impossible to handle as a particularly unhelpful VMM could emulate loss of EPC at will.
Yes, this need a big fat comment.
Quoted 5 lines
Quoted 5 lines
arch/x86/kernel/cpu/sgx/ioctl.c +5
-0+ ret =__einit (sigstruct, token,sgx_get_epc_addr (secs)); + } +preempt_enable (); +return ret; +}
Borislav Petkov
maintainerRe: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
Quoted 8 lines · Sean Christopherson
The LE pubkey hash MSRs are special snowflakes. They get reset to Intel's default key on any loss of EPC, e.g. if the system does a suspend/resume cycle. The approach we took (obviously) is to assume the kernel's cache can be stale at any given time. The alternative would be to try and track loss of EPC conditions and emulate the reset, but that's a bit dicey on bare metal as any missed case would hose SGX, and in a VM it's theoretically impossible to handle as a particularly unhelpful VMM could emulate loss of EPC at will.
Lemme try to understand this: the system could suspend/resume right here:
sgx_update_lepubkeyhash_msrs(lepubkeyhash, false);
<--- suspend/resume
ret = __einit(sigstruct, token, sgx_get_epc_addr(secs));
and thus the MSRs would have the default key so you'd need the second __einit() call?
But what happens if the system suspends before the second __einit() call?
Why don't you simply drop that @enforce param and let the caller handle any retries?
Or is the scenario something different?
Or you could perhaps disable suspend/resume around it, maybe something like lock_system_sleep() or so, from a quick grep...
Quoted 1 line
Yes, this need a big fat comment.
Oh yeah.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteA message that is not in the archive (sent off-list or not yet received): [email protected]
Borislav Petkov
maintainerRe: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
Quoted 5 lines · Sean Christopherson
That was effectively my original suggestion as well, check for a stale cache and retry indefinitely. I capitulated because it did feel like I was being overly paranoid. I'm obviously ok going the retry indefinitely route :-).
Right, so if EINIT is so expensive, why does it matter how many cyccles WRMSR has? I.e., you don't really need to cache - you simply write the 4 MSRs and you're done. Simple.
As to "indefinitely" - caller can increment a counter which counts how many times it returned SGX_INVALID_EINITTOKEN. I guess when it reaches some too high number which should not be reached during normal usage patterns, you can give up and issue a message to say that counter reached max retries or so but other than that, you should be ok. That thing is running interruptible in a loop anyway...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
Quoted 11 lines · Borislav Petkov
Quoted 5 lines · Sean Christopherson
That was effectively my original suggestion as well, check for a stale cache and retry indefinitely. I capitulated because it did feel like I was being overly paranoid. I'm obviously ok going the retry indefinitely route :-).
Right, so if EINIT is so expensive, why does it matter how many cyccles WRMSR has? I.e., you don't really need to cache - you simply write the 4 MSRs and you're done. Simple.
Quoted 6 lines
As to "indefinitely" - caller can increment a counter which counts how many times it returned
SGX_INVALID_EINITTOKEN. I guess when it reaches some too high number which should not be reached during normal usage patterns, you can give up and issue a message to say that counter reached max retries or so but other than that, you should be ok. That thing is running interruptible in a loop anyway...
The way I see it after reading the thread is:
1. Start with simpler always-update-MSRs in this patch set.
2. If this ever causes a bottleneck, then we will fix it.
Quoted 7 lines
Thx.
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Re: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
In reply to Borislav PetkovQuoted 11 lines · Borislav Petkov
Quoted 5 lines · Sean Christopherson
That was effectively my original suggestion as well, check for a stale cache and retry indefinitely. I capitulated because it did feel like I was being overly paranoid. I'm obviously ok going the retry indefinitely route :-).
Right, so if EINIT is so expensive, why does it matter how many cyccles WRMSR has? I.e., you don't really need to cache - you simply write the 4 MSRs and you're done. Simple.
Quoted 6 lines
As to "indefinitely" - caller can increment a counter which counts how many times it returned
SGX_INVALID_EINITTOKEN. I guess when it reaches some too high number which should not be reached during normal usage patterns, you can give up and issue a message to say that counter reached max retries or so but other than that, you should be ok. That thing is running interruptible in a loop anyway...
I don't see why you'd need to retry indefinitely. Yes the MSRs may not match the cached value for “reasons”, but if after you've written them once it still doesn't work, clearly either 1) an “unhelpful” VMM is actively messing with the MSRs which I'd say is at best a VMM bug or 2) there was an EPC reset and your enclave is now invalid anyway, so no need to EINIT.
--
Jethro Beekman | Fortanix
Borislav Petkov
maintainerRe: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
Quoted 6 lines · Jethro Beekman
I don't see why you'd need to retry indefinitely. Yes the MSRs may not match the cached value for “reasons”, but if after you've written them once it still doesn't work, clearly either 1) an “unhelpful” VMM is actively messing with the MSRs which I'd say is at best a VMM bug or 2) there was an EPC reset and your enclave is now invalid anyway, so no need to EINIT.
/me likes that even more.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
Quoted 9 lines · Borislav Petkov
Quoted 6 lines · Jethro Beekman
I don't see why you'd need to retry indefinitely. Yes the MSRs may not match the cached value for “reasons”, but if after you've written them once it still doesn't work, clearly either 1) an “unhelpful” VMM is actively messing with the MSRs which I'd say is at best a VMM bug or 2) there was an EPC reset and your enclave is now invalid anyway, so no need to EINIT.
/me likes that even more.
OK, so I did not follow this particular discussion in high detail, so as a sanity check I'll preview my changes.
I'd refine sgx_update_lepubkeyhash_msrs() to:
static void sgx_update_lepubkeyhash_msrs(u64 *lepubkeyhash)
{
u64 *cache;
int i;
preempt_disable();
cache = per_cpu(sgx_lepubkeyhash_cache, smp_processor_id());
for (i = 0; i < 4; i++) {
if (lepubkeyhash[i] != cache[i]) {
wrmsrl(MSR_IA32_SGXLEPUBKEYHASH0 + i, lepubkeyhash[i]);
cache[i] = lepubkeyhash[i];
}
}
preempt_enable();
}I'd drop sgx_einit() completely.
Finally, in sgx_encl_init() I would call sgx_update_lepubkeyhash_msrs() directly:
/* ... */
sgx_update_lepubkeyhash_msrs(mrsigner); ret = __einit(sigstruct, token, sgx_get_epc_addr(secs));
/* ... */
These staments would replace the call to sgx_einit().
Do I have the correct understanding?
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
Quoted 1 line · Jarkko Sakkinen
OK, so I did not follow this particular discussion in high detail,
What do you mean you did not follow it? It is not a huge subthread in your mbox.
Quoted 1 line
so as a sanity check I'll preview my changes.
Sorry, but you'll have to read threads properly like everyone else.
In any case, my preference would be make it the simplest possible: no cache and try EINIT only once.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
Quoted 5 lines · Borislav Petkov
Quoted 1 line · Jarkko Sakkinen
OK, so I did not follow this particular discussion in high detail,
What do you mean you did not follow it? It is not a huge subthread in your mbox.
I focused on other subthreads but I've read it now through. It has been waiting in the queue. Can't do everything simultaneously. I did skim each message as they came for this one though.
Quoted 6 lines
Quoted 1 line
so as a sanity check I'll preview my changes.
Sorry, but you'll have to read threads properly like everyone else.
In any case, my preference would be make it the simplest possible: no cache and try EINIT only once.
The main reason skimming was that I was thinking that perhaps Sean might send a patch for this one and I would then read the thread again in better detail and check that the patch matches the conclusions, as he has been more active on this one.
Anyway, now that I've read it through in detail, I did just change the init simply as:
secs_addr = sgx_get_epc_addr(encl->secs.epc_page);
preempt_disable();
ret = __einit(sigstruct, token, secs_addr);preempt_enable();
I use a local for address to make the code more readable instead of calling sgx_get_epc_addr() inside __einit().
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Re: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
In reply to Jethro BeekmanQuoted 11 lines · Jethro Beekman
Quoted 10 lines · Borislav Petkov
Quoted 5 lines · Sean Christopherson
That was effectively my original suggestion as well, check for a stale cache and retry indefinitely. I capitulated because it did feel like I was being overly paranoid. I'm obviously ok going the retry indefinitely route :-).
Right, so if EINIT is so expensive, why does it matter how many cyccles WRMSR has? I.e., you don't really need to cache - you simply write the 4 MSRs and you're done. Simple.
Hmm, true. The 1200+ cycles to write the hash MSRs (they're 3x slower than other MSRs) seems scary, but compared to the 60k cycles it really doesn't matter.
Quoted 7 lines
Quoted 6 lines
As to "indefinitely" - caller can increment a counter which counts how many times it returned
SGX_INVALID_EINITTOKEN. I guess when it reaches some too high number which should not be reached during normal usage patterns, you can give up and issue a message to say that counter reached max retries or so but other than that, you should be ok. That thing is running interruptible in a loop anyway...
Quoted 5 lines
I don't see why you'd need to retry indefinitely. Yes the MSRs may not match the cached value for “reasons”, but if after you've written them once it still doesn't work, clearly either 1) an “unhelpful” VMM is actively messing with the MSRs which I'd say is at best a VMM bug or 2) there was an EPC reset and your enclave is now invalid anyway, so no need to EINIT.
Ah, also true, I overlooked that an MSR reset would also kill the enclave.
So yeah, this can be simplified to:
if (SGX_LC) {
for (i = 0; i < 4; i++)
wrmsrl(...);
}
return __einit(...);Re: [PATCH v38 14/24] x86/sgx: Add SGX_IOC_ENCLAVE_INIT
In reply to Sean ChristophersonQuoted 38 lines · Sean Christopherson
Quoted 19 lines · Borislav Petkov
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 536
Quoted 7 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/ioctl.c +7
-0+static int sgx_einit (struct sgx_sigstruct *sigstruct,void *token, +struct sgx_epc_page *secs,u64 *lepubkeyhash) +{ +int ret; + +preempt_disable (); +sgx_update_lepubkeyhash_msrs (lepubkeyhash, false);So this will update the cached copies *and* the MSRs itself if what's cached is stale...
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 538
Quoted 2 lines
arch/x86/kernel/cpu/sgx/ioctl.c +2
-0+ ret =__einit (sigstruct, token,sgx_get_epc_addr (secs)); +if (ret ==SGX_INVALID_EINITTOKEN ) {... so why would it return this error here?
Definition of this error says:
Quoted 8 lines
* %SGX_INVALID_EINITTOKEN: EINITTOKEN is invalid and enclave signer's
* public key does not matchIA32_SGXLEPUBKEYHASH.when you just updated them?!
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_add_pages(struct sgx_encl *encl, void __user *arg) · line 539
Quoted 1 line
arch/x86/kernel/cpu/sgx/ioctl.c +1
-0+sgx_update_lepubkeyhash_msrs (lepubkeyhash, true);So why force a second time?
The LE pubkey hash MSRs are special snowflakes. They get reset to Intel's default key on any loss of EPC, e.g. if the system does a suspend/resume cycle. The approach we took (obviously) is to assume the kernel's cache can be stale at any given time. The alternative would be to try and track loss of EPC conditions and emulate the reset, but that's a bit dicey on bare metal as any missed case would hose SGX, and in a VM it's theoretically impossible to handle as a particularly unhelpful VMM could emulate loss of EPC at will.
Quoted 1 line
Yes, this need a big fat comment.
Thanks, please provide one :-)
/Jarkko
PATCH v38 15/24 x86/sgx: Enable provisioning for remote attestation
In reply to Jarkko SakkinenProvisioning Certification Enclave (PCE), the root of trust for other enclaves, generates a signing key from a fused key called Provisioning Certification Key. PCE can then use this key to certify an attestation key of a Quoting Enclave (QE), e.g. we get the chain of trust down to the hardware if the Intel signed PCE is used.
To use the needed keys, ATTRIBUTE.PROVISIONKEY is required but should be only allowed for those who actually need it so that only the trusted parties can certify QE's.
Obviously the attestation service should know the public key of the used PCE and that way detect illegit attestation, but whitelisting the legit users still adds an additional layer of defence.
Add new device file called /dev/sgx/provision. The sole purpose of this file is to provide file descriptors that act as privilege tokens to allow to build enclaves with ATTRIBUTE.PROVISIONKEY set. A new ioctl called SGX_IOC_ENCLAVE_PROVISION is used to assign this token to an enclave.
--- arch/x86/include/uapi/asm/sgx.h | 11 ++++++++ arch/x86/kernel/cpu/sgx/driver.c | 18 ++++++++++++ arch/x86/kernel/cpu/sgx/driver.h | 2 ++ arch/x86/kernel/cpu/sgx/ioctl.c | 47 ++++++++++++++++++++++++++++++++ 4 files changed, 78 insertions(+)
arch/x86/include/uapi/asm/sgx.h +11 -0
@@ -25,6 +25,8 @@ enum sgx_page_flags { _IOWR(SGX_MAGIC, 0x01, struct sgx_enclave_add_pages)#define SGX_IOC_ENCLAVE_INIT \ _IOW(SGX_MAGIC, 0x02, struct sgx_enclave_init) +#define SGX_IOC_ENCLAVE_PROVISION \ + _IOW(SGX_MAGIC, 0x03, struct sgx_enclave_provision)/** * struct sgx_enclave_create - parameter structure for the
@@ -61,4 +63,13 @@ struct sgx_enclave_init { __u64 sigstruct; }; +/** +* struct sgx_enclave_provision - parameter structure for the +* %SGX_IOC_ENCLAVE_PROVISION ioctl +* @attribute_fd: file handle of the attribute file in the securityfs +*/ +struct sgx_enclave_provision { +__u64 attribute_fd; +}; +#endif /* _UAPI_ASM_X86_SGX_H */
arch/x86/kernel/cpu/sgx/driver.c +18 -0
@@ -134,6 +134,10 @@ static const struct file_operations sgx_encl_fops = { .get_unmapped_area = sgx_get_unmapped_area, }; +const struct file_operations sgx_provision_fops = { + .owner =THIS_MODULE , +}; +static struct miscdevice sgx_dev_enclave = { .minor =MISC_DYNAMIC_MINOR , .name ="enclave" ,
@@ -141,6 +145,13 @@ static struct miscdevice sgx_dev_enclave = { .fops = &sgx_encl_fops, }; +static struct miscdevice sgx_dev_provision = { + .minor =MISC_DYNAMIC_MINOR , + .name ="provision" , + .nodename ="sgx/provision" , + .fops = &sgx_provision_fops, +}; +int __init sgx_drv_init (void ) {unsigned int eax, ebx, ecx, edx;
@@ -181,5 +192,12 @@ int __init sgx_drv_init(void)return ret; } + ret =misc_register (&sgx_dev_provision); +if (ret) { +pr_err ("Creating /dev/sgx/provision failed with %d.\n" , ret); +misc_deregister (&sgx_dev_enclave); +return ret; + } +return 0 ; }
arch/x86/kernel/cpu/sgx/driver.h +2 -0
@@ -23,6 +23,8 @@ extern u64 sgx_attributes_reserved_mask;extern u64 sgx_xfrm_reserved_mask;extern u32 sgx_xsave_size_tbl[64 ]; +extern const struct file_operations sgx_provision_fops; +long sgx_ioctl (struct file *filep,unsigned int cmd,unsigned long arg);int sgx_drv_init (void );
arch/x86/kernel/cpu/sgx/ioctl.c +47 -0
59 lines not on this page · raw message
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 15/24] x86/sgx: Enable provisioning for remote attestation
arch/x86/kernel/cpu/sgx/driver.c int __init sgx_drv_init(void) · line 198
Quoted 8 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/driver.c +4
-0@@ -181,5 +192,12 @@ int __init sgx_drv_init(void)return ret; } + ret =misc_register (&sgx_dev_provision); +if (ret) { +pr_err ("Creating /dev/sgx/provision failed with %d.\n" , ret); +misc_deregister (&sgx_dev_enclave);
The comment over misc_deregister() says:
* Unregister a miscellaneous device that was previously
* successfully registered with misc_register().
but this is not a successful registration here, in the if (ret) case...
Quoted 23 lines
arch/x86/kernel/cpu/sgx/driver.c +3
-0+return ret; + } +return 0 ; }arch/x86/kernel/cpu/sgx/driver.h +2
-0@@ -23,6 +23,8 @@ extern u64 sgx_attributes_reserved_mask;extern u64 sgx_xfrm_reserved_mask;extern u32 sgx_xsave_size_tbl[64 ]; +extern const struct file_operations sgx_provision_fops; +long sgx_ioctl (struct file *filep,unsigned int cmd,unsigned long arg);int sgx_drv_init (void );arch/x86/kernel/cpu/sgx/ioctl.c +0
-0
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_init(struct sgx_encl *encl, void __user *arg) · line 678
Quoted 6 lines
arch/x86/kernel/cpu/sgx/ioctl.c +3
-0@@ -673,6 +673,50 @@ static long sgx_ioc_enclave_init(struct sgx_encl *encl, void __user *arg)return ret; } +/** +* sgx_ioc_enclave_set_attribute - handler for %SGX_IOC_ENCLAVE_PROVISION +* @filep: open file to /dev/sgx
^^^^^^
Can you guess what my comment to that would be...?
Quoted 7 lines
arch/x86/kernel/cpu/sgx/ioctl.c +7
-0+* @arg: userspace pointer to a struct sgx_enclave_provision instance +* +* Mark the enclave as being allowed to access a restricted attribute bit. +* The requested attribute is specified via the attribute_fd field in the +* provided struct sgx_enclave_provision. The attribute_fd must be a +* handle to an SGX attribute file, e.g. "/dev/sgx/provision". +*
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_init(struct sgx_encl *encl, void __user *arg) · line 695
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c +10
-0+* Failure to explicitly request access to a restricted attribute will cause +* sgx_ioc_enclave_init() to fail. Currently, the only restricted attribute +* is access to the PROVISION_KEY. +* +* Note, access to the EINITTOKEN_KEY is disallowed entirely. +* +* Return: 0 on success, -errno otherwise +*/ +static long sgx_ioc_enclave_provision (struct sgx_encl *encl, +void __user *arg)
No need for the line break: both function args can fit on the same line.
...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 15/24] x86/sgx: Enable provisioning for remote attestation
Quoted 12 lines · Borislav Petkov
arch/x86/kernel/cpu/sgx/driver.c int __init sgx_drv_init(void) · line 198
Quoted 8 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/driver.c +4
-0@@ -181,5 +192,12 @@ int __init sgx_drv_init(void)return ret; } + ret =misc_register (&sgx_dev_provision); +if (ret) { +pr_err ("Creating /dev/sgx/provision failed with %d.\n" , ret); +misc_deregister (&sgx_dev_enclave);The comment over
misc_deregister()says:
Quoted 4 lines
* Unregister a miscellaneous device that was previously
* successfully registered withmisc_register().but this is not a successful registration here, in the if (ret) case...
'sgx_dev_enclave' is successfully register whenever that happens. Am I missing something here?
Quoted 5 lines
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_init(struct sgx_encl *encl, void __user *arg) · line 678
Quoted 2 lines
arch/x86/kernel/cpu/sgx/ioctl.c +2
-0+* sgx_ioc_enclave_set_attribute - handler for %SGX_IOC_ENCLAVE_PROVISION +* @filep: open file to /dev/sgx ^^^^^^Can you guess what my comment to that would be...?
There is also another incosistency that I fixed: the first line should have 'sgx_ioc_enclave_provision'.
Quoted 4 lines
arch/x86/kernel/cpu/sgx/ioctl.c static long sgx_ioc_enclave_init(struct sgx_encl *encl, void __user *arg) · line 695
Quoted 2 lines
arch/x86/kernel/cpu/sgx/ioctl.c +2
-0+static long sgx_ioc_enclave_provision (struct sgx_encl *encl, +void __user *arg)No need for the line break: both function args can fit on the same line.
Fixed this one too, thanks.
Quoted 8 lines
...
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 15/24] x86/sgx: Enable provisioning for remote attestation
Quoted 2 lines · Jarkko Sakkinen
'
sgx_dev_enclave' is successfully register whenever that happens. Am I missing something here?
No, I was missing the fact that you have *two* misc devices and you're deregistering the first one in the error path of the second.
Forget what I said. ;-\
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 15/24] x86/sgx: Enable provisioning for remote attestation
Quoted 8 lines · Borislav Petkov
Quoted 2 lines · Jarkko Sakkinen
'
sgx_dev_enclave' is successfully register whenever that happens. Am I missing something here?No, I was missing the fact that you have *two* misc devices and you're deregistering the first one in the error path of the second.
Forget what I said. ;-\
Ok, great.
BTW, probably would make sense to rewrite the short summary as:
"x86/sgx: Add SGX_IOC_ENCLAVE_PROVISION"
for the sake of consistency with the three precursoring patches.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 15/24] x86/sgx: Enable provisioning for remote attestation
Quoted 5 lines · Jarkko Sakkinen
BTW, probably would make sense to rewrite the short summary as:
"x86/sgx: Add
SGX_IOC_ENCLAVE_PROVISION"for the sake of consistency with the three precursoring patches.
Ok.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v38 16/24 x86/sgx: Add a page reclaimer
In reply to Jarkko SakkinenThere is a limited amount of EPC available. Therefore, some of it must be copied to the regular memory, and only subset kept in the SGX reserved memory. While kernel cannot directly access enclave memory, SGX provides a set of ENCLS leaf functions to perform reclaiming.
This commits implements a page reclaimer by using these leaf functions. It picks the victim pages in LRU fashion from all the enclaves running in the system. The thread ksgxswapd reclaims pages on the event when the number of free EPC pages goes below SGX_NR_LOW_PAGES up until it reaches SGX_NR_HIGH_PAGES.
sgx_alloc_epc_page() can optionally directly reclaim pages with @reclaim set true. A caller must also supply owner for each page so that the reclaimer can access the associated enclaves. This is needed for locking, as most of the ENCLS leafs cannot be executed concurrently for an enclave. The owner is also needed for accessing SECS, which is required to be resident when its child pages are being reclaimed.
--- arch/x86/kernel/cpu/sgx/driver.c | 1 + arch/x86/kernel/cpu/sgx/encl.c | 340 +++++++++++++++++++++- arch/x86/kernel/cpu/sgx/encl.h | 41 +++ arch/x86/kernel/cpu/sgx/ioctl.c | 77 ++++- arch/x86/kernel/cpu/sgx/main.c | 464 +++++++++++++++++++++++++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 9 + 6 files changed, 925 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/sgx/driver.c +1 -0
@@ -29,6 +29,7 @@ static int sgx_open(struct inode *inode, struct file *file)atomic_set (&encl->flags,0 );kref_init (&encl->refcount);xa_init (&encl->page_array); +INIT_LIST_HEAD (&encl->va_pages);mutex_init (&encl->lock);INIT_LIST_HEAD (&encl->mm_list);spin_lock_init (&encl->mm_lock);
arch/x86/kernel/cpu/sgx/encl.c +338 -2
390 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/encl.h +41 -0
85 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/ioctl.c +72 -5
144 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/main.c +464 -0
512 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/sgx.h +9 -0
@@ -15,6 +15,7 @@struct sgx_epc_page {unsigned long desc; +struct sgx_encl_page *owner;struct list_head list; };
@@ -27,6 +28,7 @@ struct sgx_epc_page {struct sgx_epc_section {unsigned long pa;void *va; +unsigned long free_cnt;struct list_head page_list;struct list_head unsanitized_page_list;spinlock_t lock;
@@ -35,6 +37,10 @@ struct sgx_epc_section {#define SGX_EPC_SECTION_MASK GENMASK(7, 0)#define SGX_MAX_EPC_SECTIONS (SGX_EPC_SECTION_MASK + 1)#define SGX_MAX_ADD_PAGES_LENGTH 0x100000 +#define SGX_EPC_PAGE_RECLAIMABLE BIT(8) +#define SGX_NR_TO_SCAN 16 +#define SGX_NR_LOW_PAGES 32 +#define SGX_NR_HIGH_PAGES 64extern struct sgx_epc_section sgx_epc_sections[SGX_MAX_EPC_SECTIONS ];
@@ -50,7 +56,10 @@ static inline void *sgx_get_epc_addr(struct sgx_epc_page *page) return section->va + (page->desc & PAGE_MASK) - section->pa; } +void sgx_mark_page_reclaimable (struct sgx_epc_page *page); +int sgx_unmark_page_reclaimable (struct sgx_epc_page *page);struct sgx_epc_page *__sgx_alloc_epc_page (void ); +struct sgx_epc_page *sgx_alloc_epc_page (void *owner,bool reclaim);void sgx_free_epc_page (struct sgx_epc_page *page);#endif /* _X86_SGX_H */
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
Quoted 6 lines · Jarkko Sakkinen
There is a limited amount of EPC available. Therefore, some of it must be copied to the regular memory, and only subset kept in the SGX reserved memory. While kernel cannot directly access enclave memory, SGX provides a set of ENCLS leaf functions to perform reclaiming.
This commits implements a page reclaimer by using these leaf functions. It
s/This commits implements/Implement/
Quoted 4 lines
picks the victim pages in LRU fashion from all the enclaves running in the system. The thread ksgxswapd reclaims pages on the event when the number of free EPC pages goes below
SGX_NR_LOW_PAGESup until it reachesSGX_NR_HIGH_PAGES.
...
Had to grep the SDM for that "eldu" magic. It could use a comment:
/*
* ELDU: Load an EPC page as unblocked. For more info, see "OS
* Management of EPC Pages" in the SDM.
*/
or so, to denote what's going on here.
Quoted 25 lines
arch/x86/kernel/cpu/sgx/encl.c +25
-0+static struct sgx_epc_page *sgx_encl_eldu (struct sgx_encl_page *encl_page, +struct sgx_epc_page *secs_page) +{ +unsigned long va_offset =SGX_ENCL_PAGE_VA_OFFSET (encl_page); +struct sgx_encl *encl = encl_page->encl; +struct sgx_epc_page *epc_page; +int ret; + + epc_page =sgx_alloc_epc_page (encl_page, false); +if (IS_ERR (epc_page)) +return epc_page; + + ret =__sgx_encl_eldu (encl_page, epc_page, secs_page); +if (ret) { +sgx_free_epc_page (epc_page); +return ERR_PTR (ret); + } + +sgx_free_va_slot (encl_page->va_page, va_offset); +list_move (&encl_page->va_page->list, &encl->va_pages); + encl_page->desc &= ~SGX_ENCL_PAGE_VA_OFFSET_MASK ; + encl_page->epc_page = epc_page; + +return epc_page; +}
...
arch/x86/kernel/cpu/sgx/encl.h line 23
Quoted 10 lines
arch/x86/kernel/cpu/sgx/encl.h +2
-0@@ -19,6 +19,10 @@/** * enum sgx_encl_page_desc - defines bits for an enclave page's descriptor +* %SGX_ENCL_PAGE_RECLAIMED: The page is in the process of being +* reclaimed.
SGX_ENCL_PAGE_RECLAIMED means, tho, that the page has been reclaimed already.
I guess that reads better:
+ if (entry->desc & SGX_ENCL_PAGE_BEING_RECLAIMED)
+ return ERR_PTR(-EBUSY);
Quoted 5 lines
arch/x86/kernel/cpu/sgx/encl.h +2
-0+* %SGX_ENCL_PAGE_VA_OFFSET_MASK: Holds the offset in the Version Array +* (VA) page for a swapped page. * %SGX_ENCL_PAGE_ADDR_MASK: Holds the virtual address of the page. * * The page address for SECS is zero and is used by the subsystem to recognize
...
Quoted 4 lines
arch/x86/kernel/cpu/sgx/encl.h +0
-0@@ -86,24 +123,34 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) {unsigned long encl_size = secs->size +PAGE_SIZE ;struct sgx_epc_page *secs_epc;
arch/x86/kernel/cpu/sgx/encl.h static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) · line 135
Quoted 10 lines
arch/x86/kernel/cpu/sgx/encl.h +5
-0+struct sgx_va_page *va_page;struct sgx_pageinfo pginfo;struct sgx_secinfo secinfo;struct file *backing;long ret; + va_page =sgx_encl_grow (encl); +if (IS_ERR (va_page)) +return PTR_ERR (va_page); +else if (va_page)
Not "else" simply?
AFAICT, sgx_encl_grow() would either return an ERR_PTR or the actual page...
Also, should the growing happen *after* the SECS validation?
Quoted 26 lines
arch/x86/kernel/cpu/sgx/encl.h +9
-4+list_add (&va_page->list, &encl->va_pages); +if (sgx_validate_secs (secs)) {pr_debug ("invalid SECS\n" ); -return -EINVAL ; + ret = -EINVAL ; + goto err_out_shrink; } backing =shmem_file_setup ("SGX backing" , encl_size + (encl_size >>5 ),VM_NORESERVE ); -if (IS_ERR (backing)) -return PTR_ERR (backing); +if (IS_ERR (backing)) { + ret =PTR_ERR (backing); + goto err_out_shrink; + } encl->backing = backing; - secs_epc =__sgx_alloc_epc_page (); + secs_epc =sgx_alloc_epc_page (&encl->secs, true);if (IS_ERR (secs_epc)) { ret =PTR_ERR (secs_epc); goto err_out_backing;
arch/x86/kernel/cpu/sgx/encl.h static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) · line 205
Quoted 8 lines
arch/x86/kernel/cpu/sgx/encl.h +3
-0@@ -151,6 +198,9 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)fput (encl->backing); encl->backing =NULL ; +err_out_shrink: +sgx_encl_shrink (encl, va_page); +return ret; }
...
Let's add some comment blurb about "Write back/invalidate an EPC page" to at least start to explain what that "ewb" is.
Quoted 51 lines
arch/x86/kernel/cpu/sgx/encl.h +51
-0+static void sgx_encl_ewb (struct sgx_epc_page *epc_page, +struct sgx_backing *backing) +{ +struct sgx_encl_page *encl_page = epc_page->owner; +struct sgx_encl *encl = encl_page->encl; +struct sgx_va_page *va_page; +unsigned int va_offset; +void *va_slot; +int ret; + + encl_page->desc &= ~SGX_ENCL_PAGE_RECLAIMED ; + + va_page = list_first_entry(&encl->va_pages, struct sgx_va_page, + list); + va_offset =sgx_alloc_va_slot (va_page); + va_slot =sgx_get_epc_addr (va_page->epc_page) + va_offset; +if (sgx_va_page_full (va_page)) +list_move_tail (&va_page->list, &encl->va_pages); + + ret =__sgx_encl_ewb (epc_page, va_slot, backing); +if (ret ==SGX_NOT_TRACKED ) { + ret =__etrack (sgx_get_epc_addr (encl->secs.epc_page)); +if (ret) { +if (encls_failed (ret)) +ENCLS_WARN (ret,"ETRACK" ); + } + + ret =__sgx_encl_ewb (epc_page, va_slot, backing); +if (ret ==SGX_NOT_TRACKED ) { +/* +* Slow path, send IPIs to kick cpus out of the +* enclave. Note, it's imperative that the cpu +* mask is generated *after* ETRACK, else we'll +* miss cpus that entered the enclave between +* generating the mask and incrementing epoch. +*/ +on_each_cpu_mask (sgx_encl_ewb_cpumask (encl), + sgx_ipi_cb,NULL ,1 ); + ret =__sgx_encl_ewb (epc_page, va_slot, backing); + } + } + +if (ret) { +if (encls_failed (ret)) +ENCLS_WARN (ret,"EWB" ); + +sgx_free_va_slot (va_page, va_offset); + }else { + encl_page->desc |= va_offset; + encl_page->va_page = va_page; + }
...
Quoted 4 lines
arch/x86/kernel/cpu/sgx/encl.h +4
-0+static void sgx_reclaim_pages (void ) +{ +struct sgx_epc_page *chunk[SGX_NR_TO_SCAN ]; +struct sgx_backing backing[SGX_NR_TO_SCAN ];
arch/x86/kernel/cpu/sgx/encl.h
Quoted 10 lines
arch/x86/kernel/cpu/sgx/encl.h +10
-0+struct sgx_epc_section *section; +struct sgx_encl_page *encl_page; +struct sgx_epc_page *epc_page; +int cnt =0 ; +int ret; +int i; + +spin_lock (&sgx_active_page_list_lock); +for (i =0 ; i <SGX_NR_TO_SCAN ; i++) { +if (list_empty (&sgx_active_page_list))
Isn't it enough to do this once, i.e., not in the loop? You're holding sgx_active_page_list_lock...
Quoted 73 lines
arch/x86/kernel/cpu/sgx/encl.h +71
-0+break ; + + epc_page =list_first_entry (&sgx_active_page_list, + struct sgx_epc_page, list); +list_del_init (&epc_page->list); + encl_page = epc_page->owner; + +if (kref_get_unless_zero (&encl_page->encl->refcount) !=0 ) + chunk[cnt++] = epc_page; +else +/* The owner is freeing the page. No need to add the +* page back to the list of reclaimable pages. +*/ + epc_page->desc &= ~SGX_EPC_PAGE_RECLAIMABLE ; + } +spin_unlock (&sgx_active_page_list_lock); + +for (i =0 ; i < cnt; i++) { + epc_page = chunk[i]; + encl_page = epc_page->owner; + +if (!sgx_reclaimer_age (epc_page)) + goto skip; + + ret =sgx_encl_get_backing (encl_page->encl, +SGX_ENCL_PAGE_INDEX (encl_page), + &backing[i]); +if (ret) + goto skip; + +mutex_lock (&encl_page->encl->lock); + encl_page->desc |=SGX_ENCL_PAGE_RECLAIMED ; +mutex_unlock (&encl_page->encl->lock); +continue ; + +skip: +spin_lock (&sgx_active_page_list_lock); +list_add_tail (&epc_page->list, &sgx_active_page_list); +spin_unlock (&sgx_active_page_list_lock); + +kref_put (&encl_page->encl->refcount, sgx_encl_release); + + chunk[i] =NULL ; + } + +for (i =0 ; i < cnt; i++) { + epc_page = chunk[i]; +if (epc_page) +sgx_reclaimer_block (epc_page); + } + +for (i =0 ; i < cnt; i++) { + epc_page = chunk[i]; +if (!epc_page) +continue ; + + encl_page = epc_page->owner; +sgx_reclaimer_write (epc_page, &backing[i]); +sgx_encl_put_backing (&backing[i], true); + +kref_put (&encl_page->encl->refcount, sgx_encl_release); + epc_page->desc &= ~SGX_EPC_PAGE_RECLAIMABLE ; + + section =sgx_get_epc_section (epc_page); +spin_lock (§ion->lock); +list_add_tail (&epc_page->list, §ion->page_list); + section->free_cnt++; +spin_unlock (§ion->lock); + } +} +static void sgx_sanitize_section (struct sgx_epc_section *section)
...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
Quoted 7 lines · Borislav Petkov
Quoted 5 lines
+ * %SGX_ENCL_PAGE_VA_OFFSET_MASK: Holds the offset in the Version Array
+ * (VA) page for a swapped page.
* %SGX_ENCL_PAGE_ADDR_MASK: Holds the virtual address of the page.
*
* The page address for SECS is zero and is used by the subsystem to recognize...
So what's wrong over here?
Quoted 17 lines
Quoted 4 lines
arch/x86/kernel/cpu/sgx/ioctl.c +0
-0@@ -86,24 +123,34 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) {unsigned long encl_size = secs->size +PAGE_SIZE ;struct sgx_epc_page *secs_epc;arch/x86/kernel/cpu/sgx/ioctl.c static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) · line 135
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c +5
-0+struct sgx_va_page *va_page;struct sgx_pageinfo pginfo;struct sgx_secinfo secinfo;struct file *backing;long ret; + va_page =sgx_encl_grow (encl); +if (IS_ERR (va_page)) +return PTR_ERR (va_page); +else if (va_page)Not "else" simply?
Quoted 4 lines
AFAICT,
sgx_encl_grow()would either return anERR_PTRor the actual page...Also, should the growing happen *after* the SECS validation?
Yes, it would make sense. I'll move it.
Quoted 2 lines
Let's add some comment blurb about "Write back/invalidate an EPC page" to at least start to explain what that "ewb" is.
OK, I'll try to write something understandable.
Quoted 6 lines
arch/x86/kernel/cpu/sgx/main.c line 319
Quoted 3 lines
arch/x86/kernel/cpu/sgx/main.c +3
-0+spin_lock (&sgx_active_page_list_lock); +for (i =0 ; i <SGX_NR_TO_SCAN ; i++) { +if (list_empty (&sgx_active_page_list))Isn't it enough to do this once, i.e., not in the loop? You're holding
sgx_active_page_list_lock...
I think that would make sense. Distantly analogous to the EINIT discussion. Too complex code for yet to be known problem workloads I'd say.
Quoted 3 lines
Quoted 1 line
arch/x86/kernel/cpu/sgx/main.c +0
-0static void sgx_sanitize_section (struct sgx_epc_section *section)...
Is this about documenting this function?
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Thanks, this was valuable feedback.
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
Quoted 10 lines · Jarkko Sakkinen
Quoted 7 lines · Borislav Petkov
Quoted 5 lines
+ * %SGX_ENCL_PAGE_VA_OFFSET_MASK: Holds the offset in the Version Array
+ * (VA) page for a swapped page.
* %SGX_ENCL_PAGE_ADDR_MASK: Holds the virtual address of the page.
*
* The page address for SECS is zero and is used by the subsystem to recognize...
So what's wrong over here?
Nothing - that's just an ellipsis: https://en.wikipedia.org/wiki/Ellipsis
I do that because otherwise you'd have to scroll through pages of quoted mail for no good reason and that ain't friendly at all. And I notice people not trimming their replies and not quoting only the text they reply to. They should not wonder if their emails remain unread.
Thx for the rest.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
Quoted 12 lines · Borislav Petkov
Quoted 10 lines · Jarkko Sakkinen
Quoted 7 lines · Borislav Petkov
Quoted 5 lines
+ * %SGX_ENCL_PAGE_VA_OFFSET_MASK: Holds the offset in the Version Array
+ * (VA) page for a swapped page.
* %SGX_ENCL_PAGE_ADDR_MASK: Holds the virtual address of the page.
*
* The page address for SECS is zero and is used by the subsystem to recognize...
So what's wrong over here?
Quoted 8 lines
Nothing - that's just an ellipsis: https://en.wikipedia.org/wiki/Ellipsis
I do that because otherwise you'd have to scroll through pages of quoted mail for no good reason and that ain't friendly at all. And I notice people not trimming their replies and not quoting only the text they reply to. They should not wonder if their emails remain unread.
Thx for the rest.
OK, I'm just almost paranoic in not addressing all feedback, thanks for the explanation :-)
Like in this version, I'll try to xref all changes so it will be at least easier spot if something is missing.
Quoted 6 lines
Signature · 5 lines
-- Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Re: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
In reply to Jarkko SakkinenQuoted 7 lines · Jarkko Sakkinen
Quoted 6 lines · Borislav Petkov
arch/x86/kernel/cpu/sgx/main.c line 319
Quoted 3 lines
arch/x86/kernel/cpu/sgx/main.c +3
-0+spin_lock (&sgx_active_page_list_lock); +for (i =0 ; i <SGX_NR_TO_SCAN ; i++) { +if (list_empty (&sgx_active_page_list))Isn't it enough to do this once, i.e., not in the loop? You're holding
sgx_active_page_list_lock...
Argh, I missed this until I looked at Jarkko's updated tree.
The reason for checking list_empty() on every iteration is that the loop is greedy, i.e. it tries to grab and reclaim up to 16 (SGX_NR_TO_SCAN) EPC pages at a time.
Quoted 3 lines
I think that would make sense. Distantly analogous to the EINIT discussion. Too complex code for yet to be known problem workloads I'd say.
Nooooo. Please no.
This will most definitely impact workloads running a single large enclave, or a process running multiple enclaves, as we'll lose the batching of ETRACK and IPIs. ETRACK isn't a big deal, but the IPIs are painful and could thrash the system. E.g. in the current code, reclaiming 10 pages from an enclave with actively running threads will generate one round of IPIs to CPUs associated with the enclave (based on the mm struct). Going to per-page reclaim will likely result in 10 rounds of IPIs as threads will reenter the enclave immediately after each IPI wave.
Having to grab the section spinlock for every page is also (very) problematic. Reclaiming one page at a time makes it nigh impossible for the reclaimer to keep up as every allocation attempt acquires the spinlock. I.e. the reclaimer has to contend with every CPU that is faulting in an EPC page or is adding a page to the enclave. In my experiments with the EPC cgroup, even batching 16 pages may be insufficient to make "forward progress". That's not an apples to apples comparison, but it's a rough equivalent of what will happen with the reclaim thread versus EPC page faults. We can (and should) play with scaling the number of reclaim threads, but at this stage, that's an exercise best left to post-upstreaming.
I can't do A/B testing to provide numbers, because the changes in Jarkko's tree break basic reclaim. Which is a nice segue into my last point: this is by far the most subtle code in the SGX code base; I really, really don't want to be making last minute changes in this area unless they are absolutely necessary or obviously benign. Even if/when we get the basic reclaim functional again, and assuming it doesn't have performance issues, I wouldn't be comfortable merging the code without hammering it with the EPC cgroup tests for multiple days (on top of the 1+ days to rebased the EPC cgroup). Past testing with the cgroup found multiple bugs with races between CPUs that required hitting a window that was open for less than 10 instructions.
Re: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
Quoted 15 lines · Sean Christopherson
Quoted 7 lines · Jarkko Sakkinen
Quoted 6 lines · Borislav Petkov
arch/x86/kernel/cpu/sgx/main.c line 319
Quoted 3 lines
arch/x86/kernel/cpu/sgx/main.c +3
-0+spin_lock (&sgx_active_page_list_lock); +for (i =0 ; i <SGX_NR_TO_SCAN ; i++) { +if (list_empty (&sgx_active_page_list))Isn't it enough to do this once, i.e., not in the loop? You're holding
sgx_active_page_list_lock...Argh, I missed this until I looked at Jarkko's updated tree.
The reason for checking
list_empty()on every iteration is that the loop is greedy, i.e. it tries to grab and reclaim up to 16 (SGX_NR_TO_SCAN) EPC pages at a time.
Quoted 5 lines
Quoted 3 lines
I think that would make sense. Distantly analogous to the EINIT discussion. Too complex code for yet to be known problem workloads I'd say.
Nooooo. Please no.
I added this comment in the beginning of the sgx_reclaim_pages() based on your response:
/*
* Take a fixed number of pages from the head of the active page pool and
* reclaim them to the enclave's private shmem files. Skip the pages, which have
* been accessed since the last scan. Move those pages to the tail of active
* page pool so that the pages get scanned in LRU like fashion.
*
* Batch process a chunk of pages (at the moment 16) in order to degrade amount
* of IPI's and ETRACK's potentially required. sgx_encl_ewb() does degrade a bit
* among the HW threads with three stage EWB pipeline (EWB, ETRACK + EWB and IPI
* + EWB) but not sufficiently. Reclaiming one page at a time would also be
* problematic as it would increase the lock contention too much, which would
* halt forward progress.
*/
And reverted reclaimer patch as it was. Do you have anything in mind that I should add or modify in it?
/Jarkko
Re: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
Quoted 39 lines · Jarkko Sakkinen
Quoted 15 lines · Sean Christopherson
Quoted 7 lines · Jarkko Sakkinen
Quoted 6 lines · Borislav Petkov
arch/x86/kernel/cpu/sgx/main.c line 319
Quoted 3 lines
arch/x86/kernel/cpu/sgx/main.c +3
-0+spin_lock (&sgx_active_page_list_lock); +for (i =0 ; i <SGX_NR_TO_SCAN ; i++) { +if (list_empty (&sgx_active_page_list))Isn't it enough to do this once, i.e., not in the loop? You're holding
sgx_active_page_list_lock...Argh, I missed this until I looked at Jarkko's updated tree.
The reason for checking
list_empty()on every iteration is that the loop is greedy, i.e. it tries to grab and reclaim up to 16 (SGX_NR_TO_SCAN) EPC pages at a time.Quoted 5 lines
Quoted 3 lines
I think that would make sense. Distantly analogous to the EINIT discussion. Too complex code for yet to be known problem workloads I'd say.
Nooooo. Please no.
I added this comment in the beginning of the
sgx_reclaim_pages()based on your response:/*
* Take a fixed number of pages from the head of the active page pool and
* reclaim them to the enclave's private shmem files. Skip the pages, which have
* been accessed since the last scan. Move those pages to the tail of active
* page pool so that the pages get scanned in LRU like fashion.
*
* Batch process a chunk of pages (at the moment 16) in order to degrade amount
* of IPI's and ETRACK's potentially required.sgx_encl_ewb()does degrade a bit
* among the HW threads with three stage EWB pipeline (EWB, ETRACK + EWB and IPI
* + EWB) but not sufficiently. Reclaiming one page at a time would also be
* problematic as it would increase the lock contention too much, which would
* halt forward progress.
*/
Quoted 2 lines
And reverted reclaimer patch as it was. Do you have anything in mind that I should add or modify in it?
Nope, can't think of anything.
Re: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
In reply to Borislav PetkovQuoted 26 lines · Borislav Petkov
Quoted 5 lines · Jarkko Sakkinen
+ * %SGX_ENCL_PAGE_VA_OFFSET_MASK: Holds the offset in the Version Array
+ * (VA) page for a swapped page.
* %SGX_ENCL_PAGE_ADDR_MASK: Holds the virtual address of the page.
*
* The page address for SECS is zero and is used by the subsystem to recognize...
Quoted 4 lines
arch/x86/kernel/cpu/sgx/ioctl.c +0
-0@@ -86,24 +123,34 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) {unsigned long encl_size = secs->size +PAGE_SIZE ;struct sgx_epc_page *secs_epc;arch/x86/kernel/cpu/sgx/ioctl.c static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) · line 135
Quoted 10 lines
arch/x86/kernel/cpu/sgx/ioctl.c +5
-0+struct sgx_va_page *va_page;struct sgx_pageinfo pginfo;struct sgx_secinfo secinfo;struct file *backing;long ret; + va_page =sgx_encl_grow (encl); +if (IS_ERR (va_page)) +return PTR_ERR (va_page); +else if (va_page)Not "else" simply?
Quoted 3 lines
AFAICT,
sgx_encl_grow()would either return anERR_PTRor the actual page...
The "else if" is correct. Version Array (VA) pages have 512 slots that hold metadata for evicted EPC pages, i.e. swapping a page out of the EPC requires a VA slot. For simplicity (LOL), the approach we are taking for initial support is to reserve a VA slot when adding a page to the enclave[*]. In most cases, reserving a slot does not require allocating a new VA page, e.g. to
reserve slots 1-511 of the "current" VA page. The if-elif is handling the
case where the current VA page is fully reserved and a new one needs to be allocated. The if handles the error, the elif handles success, i.e.
if (IS_ERR(va_page)) <- needed a new VA page, allocation failed
return PTR_ERR(va_page);
else if (va_page) <- needed a new VA page, allocation succeeded
list_add(&va_page->list, &encl->va_pages);
else
<- reused the current VA page
When reusing a VA page, we obviously don't want to readd the page to the list of va_pages, and the error handling path also shouldn't free the VA page.
Quoted 10 lines
Also, should the growing happen *after* the SECS validation?
Quoted 8 lines
arch/x86/kernel/cpu/sgx/ioctl.c +4
-1+list_add (&va_page->list, &encl->va_pages); +if (sgx_validate_secs (secs)) {pr_debug ("invalid SECS\n" ); -return -EINVAL ; + ret = -EINVAL ; + goto err_out_shrink; }
Borislav Petkov
maintainerRe: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
Quoted 3 lines · Sean Christopherson
The "else if" is correct. Version Array (VA) pages have 512 slots that hold metadata for evicted EPC pages, i.e. swapping a page out of the EPC requires a VA slot. For simplicity (LOL),
I'll say.
Quoted 7 lines
the approach we are taking for initial support is to reserve a VA slot when adding a page to the enclave[*]. In most cases, reserving a slot does not require allocating a new VA page, e.g. to reserve slots 1-511 of the "current" VA page. The if-elif is handling the case where the current VA page is fully reserved and a new one needs to be allocated. The if handles the error, the elif handles success, i.e.
Quoted 6 lines
if (
IS_ERR(va_page)) <- needed a new VA page, allocation failed
returnPTR_ERR(va_page);
else if (va_page) <- needed a new VA page, allocation succeeded
list_add(&va_page->list, &encl->va_pages);
else
<- reused the current VA page
Aha, in that case va_page will be NULL but you're not using it in that function later and that current VA page is on encl->va_pages which others can get from there. Ok, gotcha.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 16/24] x86/sgx: Add a page reclaimer
In reply to Sean ChristophersonQuoted 6 lines
if (
IS_ERR(va_page)) <- needed a new VA page, allocation failed
returnPTR_ERR(va_page);
else if (va_page) <- needed a new VA page, allocation succeeded
list_add(&va_page->list, &encl->va_pages);
else
<- reused the current VA page
/* else the tail page of the VA page list had free slots. */
I added this comment as "the missing branch".
/Jarkko
PATCH v38 17/24 x86/sgx: ptrace() support for the SGX driver
In reply to Jarkko SakkinenAdd VMA callbacks for ptrace() that can be used with debug enclaves. With debug enclaves data can be read and write the memory word at a time by using ENCLS(EDBGRD) and ENCLS(EDBGWR) leaf instructions.
--- arch/x86/kernel/cpu/sgx/encl.c | 87 ++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+)
arch/x86/kernel/cpu/sgx/encl.c +87 -0
@@ -333,10 +333,97 @@ static int sgx_vma_mprotect(struct vm_area_struct *vma,return mprotect_fixup (vma, pprev, start, end, newflags); } +static int sgx_edbgrd (struct sgx_encl *encl,struct sgx_encl_page *page, +unsigned long addr,void *data) +{ +unsigned long offset = addr & ~PAGE_MASK ; +int ret; + + + ret =__edbgrd (sgx_get_epc_addr (page->epc_page) + offset, data); +if (ret) +return -EIO ; + +return 0 ; +} + +static int sgx_edbgwr (struct sgx_encl *encl,struct sgx_encl_page *page, +unsigned long addr,void *data) +{ +unsigned long offset = addr & ~PAGE_MASK ; +int ret; + + ret =__edbgwr (sgx_get_epc_addr (page->epc_page) + offset, data); +if (ret) +return -EIO ; + +return 0 ; +} + +static int sgx_vma_access (struct vm_area_struct *vma,unsigned long addr, +void *buf,int len,int write) +{ +struct sgx_encl *encl = vma->vm_private_data; +struct sgx_encl_page *entry =NULL ; +char data[sizeof (unsigned long )]; +unsigned long align; +unsigned int flags; +int offset; +int cnt; +int ret =0 ; +int i; + +/* If process was forked, VMA is still there but vm_private_data is set +* to NULL. +*/ +if (!encl) +return -EFAULT ; + + flags =atomic_read (&encl->flags); + +if (!(flags &SGX_ENCL_DEBUG ) || !(flags &SGX_ENCL_INITIALIZED ) || + (flags &SGX_ENCL_DEAD )) +return -EFAULT ; + +for (i =0 ; i < len; i += cnt) { + entry =sgx_encl_reserve_page (encl, (addr + i) &PAGE_MASK ); +if (IS_ERR (entry)) { + ret =PTR_ERR (entry); +break ; + } + + align =ALIGN_DOWN (addr + i,sizeof (unsigned long )); + offset = (addr + i) & (sizeof (unsigned long ) -1 ); + cnt =sizeof (unsigned long ) - offset; + cnt =min (cnt, len - i); + + ret =sgx_edbgrd (encl, entry, align, data); +if (ret) + goto out; + +if (write) { +memcpy (data + offset, buf + i, cnt); + ret =sgx_edbgwr (encl, entry, align, data); +if (ret) + goto out; + }else +memcpy (buf + i, data + offset, cnt); + +out: +mutex_unlock (&encl->lock); + +if (ret) +break ; + } + +return ret <0 ? ret : i; +} +const struct vm_operations_struct sgx_vm_ops = { .open = sgx_vma_open, .fault = sgx_vma_fault, .mprotect = sgx_vma_mprotect, + .access = sgx_vma_access, };/**
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver
Quoted 2 lines
Subject: Re: [PATCH v38 17/24] x86/sgx:
ptrace()support for the SGX driver
... x86/sgx: Add ptrace() support...
subject needs a verb.
Quoted 2 lines · Jarkko Sakkinen
Add VMA callbacks for
ptrace()that can be used with debug enclaves. With debug enclaves data can be read and write the memory word at a time
I think you wanna say here
"... data can be read and/or written a memory word at a time by using..."
Quoted 32 lines
by using ENCLS(EDBGRD) and ENCLS(EDBGWR) leaf instructions.
Acked-by Jethro Beekman <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>--- arch/x86/kernel/cpu/sgx/encl.c | 87 ++++++++++++++++++++++++++++++++++ 1 file changed, 87 insertions(+)arch/x86/kernel/cpu/sgx/encl.c +16
-0@@ -333,10 +333,97 @@ static int sgx_vma_mprotect(struct vm_area_struct *vma,return mprotect_fixup (vma, pprev, start, end, newflags); } +static int sgx_edbgrd (struct sgx_encl *encl,struct sgx_encl_page *page, +unsigned long addr,void *data) +{ +unsigned long offset = addr & ~PAGE_MASK ; +int ret; + + + ret =__edbgrd (sgx_get_epc_addr (page->epc_page) + offset, data); +if (ret) +return -EIO ; + +return 0 ; +} + +static int sgx_edbgwr (struct sgx_encl *encl,struct sgx_encl_page *page, +unsigned long addr,void *data)
arch/x86/kernel/cpu/sgx/encl.c static int sgx_vma_mprotect(struct vm_area_struct *vma, · line 361
Quoted 10 lines
arch/x86/kernel/cpu/sgx/encl.c +10
-0+{ +unsigned long offset = addr & ~PAGE_MASK ; +int ret; + + ret =__edbgwr (sgx_get_epc_addr (page->epc_page) + offset, data); +if (ret) +return -EIO ; + +return 0 ; +}
I know those are supposed to correspond to the ENCLS leafs but the function names are totally unreadable. I guess you could name them
sgx_encl_dbg_readsgx_encl_dbg_write
and leave the lowlevel helpers like the insn names.
Quoted 6 lines
arch/x86/kernel/cpu/sgx/encl.c +6
-0+static int sgx_vma_access (struct vm_area_struct *vma,unsigned long addr, +void *buf,int len,int write) +{ +struct sgx_encl *encl = vma->vm_private_data; +struct sgx_encl_page *entry =NULL ; +char data[sizeof (unsigned long )];
arch/x86/kernel/cpu/sgx/encl.c static int sgx_vma_mprotect(struct vm_area_struct *vma, · line 378
Quoted 10 lines
arch/x86/kernel/cpu/sgx/encl.c +10
-0+unsigned long align; +unsigned int flags; +int offset; +int cnt; +int ret =0 ; +int i; + +/* If process was forked, VMA is still there but vm_private_data is set +* to NULL. +*/
Kernel comments style is:
/*
* A sentence ending with a full-stop.
* Another sentence. ...
* More sentences. ...
*/
Quoted 21 lines
arch/x86/kernel/cpu/sgx/encl.c +21
-0+if (!encl) +return -EFAULT ; + + flags =atomic_read (&encl->flags); + +if (!(flags &SGX_ENCL_DEBUG ) || !(flags &SGX_ENCL_INITIALIZED ) || + (flags &SGX_ENCL_DEAD )) +return -EFAULT ; + +for (i =0 ; i < len; i += cnt) { + entry =sgx_encl_reserve_page (encl, (addr + i) &PAGE_MASK ); +if (IS_ERR (entry)) { + ret =PTR_ERR (entry); +break ; + } + + align =ALIGN_DOWN (addr + i,sizeof (unsigned long )); + offset = (addr + i) & (sizeof (unsigned long ) -1 ); + cnt =sizeof (unsigned long ) - offset; + cnt =min (cnt, len - i); +
arch/x86/kernel/cpu/sgx/encl.c static int sgx_vma_mprotect(struct vm_area_struct *vma, · line 409
Quoted 10 lines
arch/x86/kernel/cpu/sgx/encl.c +10
-0+ ret =sgx_edbgrd (encl, entry, align, data); +if (ret) + goto out; + +if (write) { +memcpy (data + offset, buf + i, cnt); + ret =sgx_edbgwr (encl, entry, align, data); +if (ret) + goto out; + }else
} else {arch/x86/kernel/cpu/sgx/encl.c static int sgx_vma_mprotect(struct vm_area_struct *vma, · line 410
Quoted 1 line
arch/x86/kernel/cpu/sgx/encl.c +1
-0+memcpy (buf + i, data + offset, cnt);
}
Put the else branch in {} too.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver
Quoted 6 lines · Borislav Petkov
Quoted 2 lines
Subject: Re: [PATCH v38 17/24] x86/sgx:
ptrace()support for the SGX driver... x86/sgx: Add ptrace() support...subject needs a verb.
Quoted 7 lines
Quoted 2 lines · Jarkko Sakkinen
Add VMA callbacks for
ptrace()that can be used with debug enclaves. With debug enclaves data can be read and write the memory word at a timeI think you wanna say here
"... data can be read and/or written a memory word at a time by using..."
I also fixed the other issues but I'll paste here the rewrite that I did for the commit message:
"
x86/sgx: Add ptrace() support for the SGX driver
Intel Sofware Guard eXtensions (SGX) allows creation of executable blobs called enclaves, which cannot be accessed by default when not executing inside the enclave. Enclaves can be entered by only using predefined memory addresses, which are defined the enclave is loaded.
However, enclaves can defined as debug enclaves during the load time. In debug enclaves data can be read and/or written a memory word at a time by using by using ENCLS[EDBGRD] and ENCLS[EDBGWR] leaf instructions.
Add 'access' implementation to vm_ops with the help of these functions. This allows to use ptrace() with debug enclaves.
"
I also think that mm CC would make sense also for this patch.
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver
Quoted 4 lines · Jarkko Sakkinen
Intel Sofware Guard eXtensions (SGX) allows creation of executable blobs called enclaves, which cannot be accessed by default when not executing inside the enclave. Enclaves can be entered by only using predefined memory addresses, which are defined the enclave is loaded.
^ "when" or "before". I think it is before.
Quoted 1 line
However, enclaves can defined as debug enclaves during the load time. In
"However, enclaves can be defined as debug enclaves at load time."
Quoted 2 lines
debug enclaves data can be read and/or written a memory word at a time by using by using ENCLS[EDBGRD] and ENCLS[EDBGWR] leaf instructions.
only one "by using" is enough.
Quoted 1 line
Add 'access' implementation to
vm_opswith the help of these functions.
"Add an ->access virtual MM function for accessing the enclave's memory... "
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver
Quoted 3 lines · Borislav Petkov
Quoted 1 line
Add 'access' implementation to
vm_opswith the help of these functions."Add an ->access virtual MM function for accessing the enclave's memory... "
Thank you. I wrote the last paragraph like this:
"Add an '->access' virtual function for accessing the enclave's memory to vm_ops by using these functions. This allows to use ptrace() with debug enclaves."
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver
Quoted 9 lines · Jarkko Sakkinen
Quoted 3 lines · Borislav Petkov
Quoted 1 line
Add 'access' implementation to
vm_opswith the help of these functions."Add an ->access virtual MM function for accessing the enclave's memory... "
Thank you. I wrote the last paragraph like this:
"Add an '->access' virtual function for accessing the enclave's memory to
vm_opsby using these functions. This allows to useptrace()with
"to vm_ops" must come after "function".
But lemme ask what is "vm_ops"?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver'
Quoted 6 lines · Borislav Petkov
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines · Borislav Petkov
Quoted 1 line
Add 'access' implementation to
vm_opswith the help of these functions."Add an ->access virtual MM function for accessing the enclave's memory... "
Quoted 8 lines
Quoted 4 lines
Thank you. I wrote the last paragraph like this:
"Add an '->access' virtual function for accessing the enclave's memory to
vm_opsby using these functions. This allows to useptrace()with"to
vm_ops" must come after "function".But lemme ask what is "
vm_ops"?
I assume this is a rethorical question and I notice what I suggested looks as bad as my earlier commit message :-)
So, I gave it some thought that and decided to "open code" the paragraph as
"Add sgx_vma_access() function that implements 'access' virtual function of struct vm_operations_struct. Use formentioned leaf instructions to achieve read and write primitives for the enclave memory."
I think this starts to have the right balance and is understandable.
Still open for futher suggestion of course.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Re: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver'
Quoted 22 lines · Jarkko Sakkinen
Quoted 6 lines · Borislav Petkov
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines · Borislav Petkov
Quoted 1 line
Add 'access' implementation to
vm_opswith the help of these functions."Add an ->access virtual MM function for accessing the enclave's memory... "
Quoted 8 lines
Quoted 4 lines
Thank you. I wrote the last paragraph like this:
"Add an '->access' virtual function for accessing the enclave's memory to
vm_opsby using these functions. This allows to useptrace()with"to
vm_ops" must come after "function".But lemme ask what is "
vm_ops"?I assume this is a rethorical question and I notice what I suggested looks as bad as my earlier commit message :-)
So, I gave it some thought that and decided to "open code" the paragraph as
Quoted 7 lines
"Add
sgx_vma_access()function that implements 'access' virtual function of structvm_operations_struct. Use formentioned leaf instructions to achieve read and write primitives for the enclave memory."I think this starts to have the right balance and is understandable.
Still open for futher suggestion of course.
I'm not sure if I said it already but I also added cc to linux-mm (same CC's in the patch as with mprotect callback commit). This should also have mm ack I think.
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver'
Quoted 3 lines · Jarkko Sakkinen
I'm not sure if I said it already but I also added cc to linux-mm (same CC's in the patch as with mprotect callback commit). This should also have mm ack I think.
Why? This is adding ptrace functionality to enclaves which is purely arch/x86/.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver'
Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Jarkko Sakkinen
I'm not sure if I said it already but I also added cc to linux-mm (same CC's in the patch as with mprotect callback commit). This should also have mm ack I think.
Why? This is adding ptrace functionality to enclaves which is purely
arch/x86/.
You are right. No changes to vm_operations_struct itself.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver'
In reply to Jarkko SakkinenQuoted 1 line · Jarkko Sakkinen
I assume this is a rethorical question
Of course - our text should not be write-only.
Quoted 8 lines
and I notice what I suggested
looks as bad as my earlier commit message :-)So, I gave it some thought that and decided to "open code" the paragraph as
"Add
sgx_vma_access()function that implements 'access' virtual function of structvm_operations_struct. Use formentioned leaf instructions to
"aforementioned"
Quoted 1 line
achieve read and write primitives for the enclave memory."
I'd say "to provide read and write access to enclave memory."
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 17/24] x86/sgx: ptrace() support for the SGX driver'
Quoted 8 lines · Borislav Petkov
Quoted 1 line · Jarkko Sakkinen
I assume this is a rethorical question
Of course - our text should not be write-only.
Quoted 3 lines
and I notice what I suggested
looks as bad as my earlier commit message :-)
Quoted 7 lines
Quoted 5 lines
So, I gave it some thought that and decided to "open code" the paragraph as
"Add
sgx_vma_access()function that implements 'access' virtual function of structvm_operations_struct. Use formentioned leaf instructions to"aforementioned"
Ugh, that the was worst spelling I've done for a while.
Quoted 3 lines
Quoted 1 line
achieve read and write primitives for the enclave memory."
I'd say "to provide read and write access to enclave memory."
Sound better yes, thanks.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Updated the commit message accordingly. I also rebased now to x86 tip (and yes, merge conflicts were trivial to sort out).
/Jarkko
PATCH v38 18/24 x86/vdso: Add support for exception fixup in vDSO functions
In reply to Jarkko SakkinenFrom: Sean Christopherson <[email protected]>
The basic concept and implementation is very similar to the kernel's exception fixup mechanism. The key differences are that the kernel handler is hardcoded and the fixup entry addresses are relative to the overall table as opposed to individual entries.
Hardcoding the kernel handler avoids the need to figure out how to get userspace code to point at a kernel function. Given that the expected usage is to propagate information to userspace, dumping all fault information into registers is likely the desired behavior for the vast majority of yet-to-be-created functions. Use registers DI, SI and DX to communicate fault information, which follows Linux's ABI for register consumption and hopefully avoids conflict with hardware features that might leverage the fixup capabilities, e.g. register usage for SGX instructions was at least partially designed with calling conventions in mind.
Making fixup addresses relative to the overall table allows the table to be stripped from the final vDSO image (it's a kernel construct) without complicating the offset logic, e.g. entry-relative addressing would also need to account for the table's location relative to the image.
Regarding stripping the table, modify vdso2c to extract the table from the raw, a.k.a. unstripped, data and dump it as a standalone byte array in the resulting .c file. The original base of the table, its length and a pointer to the byte array are captured in struct vdso_image. Alternatively, the table could be dumped directly into the struct, but because the number of entries can vary per image, that would require either hardcoding a max sized table into the struct definition or defining the table as a flexible length array. The flexible length array approach has zero benefits, e.g. the base/size are still needed, and prevents reusing the extraction code, while hardcoding the max size adds ongoing maintenance just to avoid exporting the explicit size.
The immediate use case is for Intel Software Guard Extensions (SGX). SGX introduces a new CPL3-only "enclave" mode that runs as a sort of black box shared object that is hosted by an untrusted "normal" CPl3 process.
Entering an enclave can only be done through SGX-specific instructions, EENTER and ERESUME, and is a non-trivial process. Because of the complexity of transitioning to/from an enclave, the vast majority of enclaves are expected to utilize a library to handle the actual transitions. This is roughly analogous to how e.g. libc implementations are used by most applications.
Another crucial characteristic of SGX enclaves is that they can generate exceptions as part of their normal (at least as "normal" as SGX can be) operation that need to be handled *in* the enclave and/or are unique to SGX.
And because they are essentially fancy shared objects, a process can host any number of enclaves, each of which can execute multiple threads simultaneously.
Putting everything together, userspace enclaves will utilize a library that must be prepared to handle any and (almost) all exceptions any time at least one thread may be executing in an enclave. Leveraging signals to handle the enclave exceptions is unpleasant, to put it mildly, e.g. the SGX library must constantly (un)register its signal handler based on whether or not at least one thread is executing in an enclave, and filter and forward exceptions that aren't related to its enclaves. This becomes particularly nasty when using multiple levels of libraries that register signal handlers, e.g. running an enclave via cgo inside of the Go runtime.
Enabling exception fixup in vDSO allows the kernel to provide a vDSO function that wraps the low-level transitions to/from the enclave, i.e. the EENTER and ERESUME instructions. The vDSO function can intercept exceptions that would otherwise generate a signal and return the fault information directly to its caller, thus avoiding the need to juggle signal handlers.
Note that unlike the kernel's _ASM_EXTABLE_HANDLE implementation, the 'C' version of _ASM_VDSO_EXTABLE_HANDLE doesn't use a pre-compiled assembly macro. Duplicating four lines of code is simpler than adding the necessary infrastructure to generate pre-compiled assembly and the intended benefit of massaging GCC's inlining algorithm is unlikely to realized in the vDSO any time soon, if ever.
--- arch/x86/entry/vdso/Makefile | 6 ++-- arch/x86/entry/vdso/extable.c | 46 ++++++++++++++++++++++++ arch/x86/entry/vdso/extable.h | 29 ++++++++++++++++ arch/x86/entry/vdso/vdso-layout.lds.S | 9 ++++- arch/x86/entry/vdso/vdso2c.h | 50 ++++++++++++++++++++++++++- arch/x86/include/asm/vdso.h | 5 +++ 6 files changed, 140 insertions(+), 5 deletions(-) create mode 100644 arch/x86/entry/vdso/extable.c create mode 100644 arch/x86/entry/vdso/extable.h
arch/x86/entry/vdso/Makefile +3 -3
@@ -31,7 +31,7 @@ vobjs32-y := vdso32/note.o vdso32/system_call.o vdso32/sigreturn.o vobjs32-y += vdso32/vclock_gettime.o # files to link into kernel -obj-y += vma.o +obj-y += vma.o extable.o KASAN_SANITIZE_vma.o := y UBSAN_SANITIZE_vma.o := y KCSAN_SANITIZE_vma.o := y
@@ -130,8 +130,8 @@ $(obj)/%-x32.o: $(obj)/%.o FORCE targets += vdsox32.lds $(vobjx32s-y) -$(obj)/%.so: OBJCOPYFLAGS := -S -$(obj)/%.so: $(obj)/%.so.dbg FORCE +$(obj)/%.so: OBJCOPYFLAGS := -S --remove-section __ex_table +$(obj)/%.so: $(obj)/%.so.dbg $(call if_changed,objcopy) $(obj)/vdsox32.so.dbg: $(obj)/vdsox32.lds $(vobjx32s) FORCE
arch/x86/entry/vdso/extable.c new file +46 -0
46 lines not on this page · raw message
arch/x86/entry/vdso/extable.h new file +29 -0
@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __VDSO_EXTABLE_H +#define __VDSO_EXTABLE_H + +/* +* Inject exception fixup for vDSO code. Unlike normal exception fixup, +* vDSO uses a dedicated handler the addresses are relative to the overall +* exception table, not each individual entry. +*/ +#ifdef __ASSEMBLY__ +#define _ASM_VDSO_EXTABLE_HANDLE (from, to) \ + ASM_VDSO_EXTABLE_HANDLE from to + +.macro ASM_VDSO_EXTABLE_HANDLE from:req to:req + .pushsection __ex_table, "a" + .long (\from) - __ex_table + .long (\to) - __ex_table + .popsection +.endm +#else +#define _ASM_VDSO_EXTABLE_HANDLE (from, to) \ + ".pushsection __ex_table, \"a\"\n" \ + ".long (" #from ") - __ex_table\n" \ + ".long (" #to ") - __ex_table\n" \ + ".popsection\n" +#endif + +#endif /* __VDSO_EXTABLE_H */ +
arch/x86/entry/vdso/vdso-layout.lds.S +8 -1
@@ -75,11 +75,18 @@ SECTIONS * stuff that isn't used at runtime in between. */ - .text : { *(.text*) } :text =0x90909090, + .text : { + *(.text*) + *(.fixup) + } :text =0x90909090, + + .altinstructions : { *(.altinstructions) } :text .altinstr_replacement : { *(.altinstr_replacement) } :text + __ex_table : { *(__ex_table) } :text + /DISCARD/ : { *(.discard) *(.discard.*)
arch/x86/entry/vdso/vdso2c.h +49 -1
80 lines not on this page · raw message
arch/x86/include/asm/vdso.h +5 -0
@@ -15,6 +15,8 @@ struct vdso_image {unsigned long size;/* Always a multiple of PAGE_SIZE */ unsigned long alt, alt_len; +unsigned long extable_base, extable_len; +const void *extable;long sym_vvar_start;/* Negative offset to the vvar area */
@@ -45,6 +47,9 @@ extern void __init init_vdso_image(const struct vdso_image *image);extern int map_vdso_once (const struct vdso_image *image,unsigned long addr); +extern bool fixup_vdso_exception (struct pt_regs *regs,int trapnr, +unsigned long error_code, +unsigned long fault_addr);#endif /* __ASSEMBLER__ */ #endif /* _ASM_X86_VDSO_H */
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 18/24] x86/vdso: Add support for exception fixup in vDSO functions
Quoted 6 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
The basic concept and implementation is very similar to the kernel's exception fixup mechanism. The key differences are that the kernel handler is hardcoded and the fixup entry addresses are relative to the overall table as opposed to individual entries.
...
This gist of this commit message should be also in Documentation/x86/sgx.rst
And I already said the same thing during v33 review:
"That is a very good explanation and I would prefer if it would be in a sgx-specific README or so instead of it getting lost in git..."
...
Quoted 25 lines
arch/x86/entry/vdso/extable.h new file +19
-0@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __VDSO_EXTABLE_H +#define __VDSO_EXTABLE_H + +/* +* Inject exception fixup for vDSO code. Unlike normal exception fixup, +* vDSO uses a dedicated handler the addresses are relative to the overall +* exception table, not each individual entry. +*/ +#ifdef __ASSEMBLY__ +#define _ASM_VDSO_EXTABLE_HANDLE (from, to) \ + ASM_VDSO_EXTABLE_HANDLE from to + +.macro ASM_VDSO_EXTABLE_HANDLE from:req to:req + .pushsection __ex_table, "a" + .long (\from) - __ex_table + .long (\to) - __ex_table + .popsection +.endm
arch/x86/entry/vdso/extable.h line 29
Quoted 10 lines
arch/x86/entry/vdso/extable.h new file +10
-0+#else +#define _ASM_VDSO_EXTABLE_HANDLE (from, to) \ + ".pushsection __ex_table, \"a\"\n" \ + ".long (" #from ") - __ex_table\n" \ + ".long (" #to ") - __ex_table\n" \ + ".popsection\n" +#endif + +#endif /* __VDSO_EXTABLE_H */ +
Also from last time:
.git/rebase-apply/patch:122: new blank line at EOF. +
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 18/24] x86/vdso: Add support for exception fixup in vDSO functions
Quoted 13 lines · Borislav Petkov
Quoted 6 lines · Jarkko Sakkinen
From: Sean Christopherson <[email protected]>
The basic concept and implementation is very similar to the kernel's exception fixup mechanism. The key differences are that the kernel handler is hardcoded and the fixup entry addresses are relative to the overall table as opposed to individual entries.
...
This gist of this commit message should be also in
Documentation/x86/sgx.rst
Quoted 6 lines
And I already said the same thing during v33 review:
"That is a very good explanation and I would prefer if it would be in a sgx-specific README or so instead of it getting lost in git..."
...
This is not technically SGX specific patch. Is SGX documentation the correct place for this?
Quoted 38 lines
Quoted 25 lines
arch/x86/entry/vdso/extable.h new file +19
-0@@ -0,0 +1,29 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __VDSO_EXTABLE_H +#define __VDSO_EXTABLE_H + +/* +* Inject exception fixup for vDSO code. Unlike normal exception fixup, +* vDSO uses a dedicated handler the addresses are relative to the overall +* exception table, not each individual entry. +*/ +#ifdef __ASSEMBLY__ +#define _ASM_VDSO_EXTABLE_HANDLE (from, to) \ + ASM_VDSO_EXTABLE_HANDLE from to + +.macro ASM_VDSO_EXTABLE_HANDLE from:req to:req + .pushsection __ex_table, "a" + .long (\from) - __ex_table + .long (\to) - __ex_table + .popsection +.endmarch/x86/entry/vdso/extable.h line 29
Quoted 10 lines
arch/x86/entry/vdso/extable.h new file +10
-0+#else +#define _ASM_VDSO_EXTABLE_HANDLE (from, to) \ + ".pushsection __ex_table, \"a\"\n" \ + ".long (" #from ") - __ex_table\n" \ + ".long (" #to ") - __ex_table\n" \ + ".popsection\n" +#endif + +#endif /* __VDSO_EXTABLE_H */ +Also from last time:
Quoted 2 lines
.git/rebase-apply/patch:122: new blank line at EOF. +
From checkpatch I only get:
ERROR: Macros with complex values should be enclosed in parentheses #193: FILE: arch/x86/entry/vdso/extable.h:11:
+#define _ASM_VDSO_EXTABLE_HANDLE(from, to) \
+ ASM_VDSO_EXTABLE_HANDLE from to
ERROR: need consistent spacing around '*' (ctx:WxV)
#244: FILE: arch/x86/entry/vdso/vdso2c.h:8:
+static void BITSFUNC(copy)(FILE *outfile, const unsigned char *data, size_t len)
^
ERROR: need consistent spacing around '*' (ctx:WxV) #263: FILE: arch/x86/entry/vdso/vdso2c.h:27:
+ FILE *outfile, ELF(Shdr) *sec, const char *name)
I did fix it now. Thanks.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 18/24] x86/vdso: Add support for exception fixup in vDSO functions
Quoted 2 lines · Jarkko Sakkinen
This is not technically SGX specific patch. Is SGX documentation the correct place for this?
So what is it then? It is SGX implementation-specific. Why would you not put it in the documentation?!
Quoted 1 line
From checkpatch I only get:
Please concentrate and start reading more carefully:
".git/rebase-apply/patch:122: new blank line at EOF."
Would that error come from checkpatch?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 18/24] x86/vdso: Add support for exception fixup in vDSO functions
Quoted 7 lines · Borislav Petkov
Quoted 2 lines · Jarkko Sakkinen
This is not technically SGX specific patch. Is SGX documentation the correct place for this?
So what is it then? It is SGX implementation-specific. Why would you not put it in the documentation?!
Quoted 7 lines
Quoted 1 line
From checkpatch I only get:
Please concentrate and start reading more carefully:
".git/rebase-apply/patch:122: new blank line at EOF."
Would that error come from checkpatch?
Nope. And I did fully read what you wrote. I just mentioned that more in the tone that I should (and will) do also git am test from now on, at least for mainline tree (when applicable) and x86 tip.
Right now the static tests that I do are checkpatch and sparse. Any other suggestions are welcome.
I would also also coccicheck but have had some version issues with it in Ubuntu, which I use as my host OS. Cannot recall what was the exact issue, has been a while since I last tried it.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
PATCH v38 19/24 x86/fault: Add helper function to sanitize error code
In reply to Jarkko SakkinenFrom: Sean Christopherson <[email protected]>
Add helper function to sanitize error code to prepare for vDSO exception fixup, which will expose the error code to userspace and runs before set_signal_archinfo(), i.e. suppresses the signal when fixup is successful.
--- arch/x86/mm/fault.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-)
arch/x86/mm/fault.c +17 -7
@@ -555,6 +555,18 @@ pgtable_bad(struct pt_regs *regs, unsigned long error_code,oops_end (flags, regs, sig); } +static void sanitize_error_code (unsigned long address, +unsigned long *error_code) +{ +/* +* To avoid leaking information about the kernel page +* table layout, pretend that user-mode accesses to +* kernel addresses are always protection faults. +*/ +if (address >=TASK_SIZE_MAX ) + *error_code |=X86_PF_PROT ; +} +static void set_signal_archinfo (unsigned long address,unsigned long error_code) {
@@ -611,6 +623,8 @@ no_context(struct pt_regs *regs, unsigned long error_code,* faulting through the emulate_vsyscall() logic. */ if (current->thread.sig_on_uaccess_err && signal) { +sanitize_error_code (address, &error_code); +set_signal_archinfo (address, error_code);/* XXX: hwpoison faults will set the wrong code. */
@@ -759,13 +773,7 @@ __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,if (is_errata100 (regs, address))return ; -/* -* To avoid leaking information about the kernel page table -* layout, pretend that user-mode accesses to kernel addresses -* are always protection faults. -*/ -if (address >=TASK_SIZE_MAX ) - error_code |=X86_PF_PROT ; +sanitize_error_code (address, &error_code);if (likely (show_unhandled_signals))show_signal_msg (regs, error_code, address, tsk);
@@ -884,6 +892,8 @@ do_sigbus(struct pt_regs *regs, unsigned long error_code, unsigned long address,if (is_prefetch (regs, error_code, address))return ; +sanitize_error_code (address, &error_code); +set_signal_archinfo (address, error_code);#ifdef CONFIG_MEMORY_FAILURE
Signature · 1 line
2.25.1
PATCH v38 20/24 x86/traps: Attempt to fixup exceptions in vDSO before signaling
In reply to Jarkko SakkinenFrom: Sean Christopherson <[email protected]>
vDSO functions can now leverage an exception fixup mechanism similar to kernel exception fixup. For vDSO exception fixup, the initial user is Intel's Software Guard Extensions (SGX), which will wrap the low-level transitions to/from the enclave, i.e. EENTER and ERESUME instructions, in a vDSO function and leverage fixup to intercept exceptions that would otherwise generate a signal. This allows the vDSO wrapper to return the fault information directly to its caller, obviating the need for SGX applications and libraries to juggle signal handlers.
Attempt to fixup vDSO exceptions immediately prior to populating and sending signal information. Except for the delivery mechanism, an exception in a vDSO function should be treated like any other exception in userspace, e.g. any fault that is successfully handled by the kernel should not be directly visible to userspace.
Although it's debatable whether or not all exceptions are of interest to enclaves, defer to the vDSO fixup to decide whether to do fixup or generate a signal. Future users of vDSO fixup, if there ever are any, will undoubtedly have different requirements than SGX enclaves, e.g. the fixup vs. signal logic can be made function specific if/when necessary.
--- arch/x86/kernel/traps.c | 19 ++++++++++++++++--- arch/x86/mm/fault.c | 8 ++++++++ 2 files changed, 24 insertions(+), 3 deletions(-)
arch/x86/kernel/traps.c +16 -3
@@ -59,6 +59,7 @@#include <asm/umip.h> #include <asm/insn.h> #include <asm/insn-eval.h> +#include <asm/vdso.h> #ifdef CONFIG_X86_64 #include <asm/x86_init.h>
@@ -116,6 +117,9 @@ do_trap_no_signal(struct task_struct *tsk, int trapnr, const char *str, tsk->thread.error_code = error_code; tsk->thread.trap_nr = trapnr;die (str, regs, error_code); + }else { +if (fixup_vdso_exception (regs, trapnr, error_code,0 )) +return 0 ; }/*
@@ -549,6 +553,9 @@ DEFINE_IDTENTRY_ERRORCODE(exc_general_protection) tsk->thread.error_code = error_code; tsk->thread.trap_nr =X86_TRAP_GP ; +if (fixup_vdso_exception (regs,X86_TRAP_GP , error_code,0 )) +return ; +show_signal (tsk,SIGSEGV ,"" , desc, regs, error_code);force_sig (SIGSEGV ); goto exit;
@@ -827,9 +834,12 @@ static void handle_debug(struct pt_regs *regs, unsigned long dr6, bool user)#endif if (notify_die (DIE_DEBUG ,"debug" , regs, (long )&dr6,0 , -SIGTRAP ) ==NOTIFY_STOP ) { -return ; - } +SIGTRAP ) ==NOTIFY_STOP ) + goto out; + +if (user_mode (regs) && +fixup_vdso_exception (regs,X86_TRAP_DB ,0 ,0 )) + goto out;/* It's safe to allow irq's after DR6 has been saved */ cond_local_irq_enable (regs);
@@ -981,6 +991,9 @@ static void math_error(struct pt_regs *regs, int trapnr)if (!si_code) goto exit; +if (fixup_vdso_exception (regs, trapnr,0 ,0 )) +return ; +force_sig_fault (SIGFPE , si_code, (void __user *)uprobe_get_trap_addr (regs)); exit:
arch/x86/mm/fault.c +8 -0
@@ -30,6 +30,7 @@#include <asm/cpu_entry_area.h> /* exception stack */ #include <asm/pgtable_areas.h> /* VMALLOC_START, ... */ #include <asm/kvm_para.h> /* kvm_handle_async_pf */ +#include <asm/vdso.h> /* fixup_vdso_exception() */ #define CREATE_TRACE_POINTS #include <asm/trace/exceptions.h>
@@ -775,6 +776,10 @@ __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,sanitize_error_code (address, &error_code); +if (fixup_vdso_exception (regs,X86_TRAP_PF , error_code, + address)) +return ; +if (likely (show_unhandled_signals))show_signal_msg (regs, error_code, address, tsk);
@@ -894,6 +899,9 @@ do_sigbus(struct pt_regs *regs, unsigned long error_code, unsigned long address,sanitize_error_code (address, &error_code); +if (fixup_vdso_exception (regs,X86_TRAP_PF , error_code, address)) +return ; +set_signal_archinfo (address, error_code);#ifdef CONFIG_MEMORY_FAILURE
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 20/24] x86/traps: Attempt to fixup exceptions in vDSO before signaling
Quoted 13 lines · Jarkko Sakkinen
arch/x86/mm/fault.c +1
-0@@ -30,6 +30,7 @@#include <asm/cpu_entry_area.h> /* exception stack */ #include <asm/pgtable_areas.h> /* VMALLOC_START, ... */ #include <asm/kvm_para.h> /* kvm_handle_async_pf */ +#include <asm/vdso.h> /* fixup_vdso_exception() */ #define CREATE_TRACE_POINTS #include <asm/trace/exceptions.h>
arch/x86/mm/fault.c __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code, · line 780
Quoted 5 lines
arch/x86/mm/fault.c +2
-0@@ -775,6 +776,10 @@ __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,sanitize_error_code (address, &error_code); +if (fixup_vdso_exception (regs,X86_TRAP_PF , error_code, + address))
No need to break that line.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 20/24] x86/traps: Attempt to fixup exceptions in vDSO before signaling
Quoted 15 lines · Borislav Petkov
Quoted 14 lines · Jarkko Sakkinen
arch/x86/mm/fault.c +1
-0@@ -30,6 +30,7 @@#include <asm/cpu_entry_area.h> /* exception stack */ #include <asm/pgtable_areas.h> /* VMALLOC_START, ... */ #include <asm/kvm_para.h> /* kvm_handle_async_pf */ +#include <asm/vdso.h> /* fixup_vdso_exception() */ #define CREATE_TRACE_POINTS #include <asm/trace/exceptions.h> @@ -775,6 +776,10 @@ __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code,
Quoted 6 lines
arch/x86/mm/fault.c __bad_area_nosemaphore(struct pt_regs *regs, unsigned long error_code, · line 780
Quoted 4 lines
arch/x86/mm/fault.c +2
-0sanitize_error_code (address, &error_code); +if (fixup_vdso_exception (regs,X86_TRAP_PF , error_code, + address))No need to break that line.
Thanks, fixed.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
PATCH v38 21/24 x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to Jarkko SakkinenFrom: Sean Christopherson <[email protected]>
An SGX runtime must be aware of the exceptions, which happen inside an enclave. Introduce a vDSO call that wraps EENTER/ERESUME cycle and returns the CPU exception back to the caller exactly when it happens.
Kernel fixups the exception information to RDI, RSI and RDX. The SGX call vDSO handler fills this information to the user provided buffer or alternatively trigger user provided callback at the time of the exception.
The calling convention is custom and does not follow System V x86-64 ABI.
--- arch/x86/entry/vdso/Makefile | 2 + arch/x86/entry/vdso/vdso.lds.S | 1 + arch/x86/entry/vdso/vsgx_enter_enclave.S | 157 +++++++++++++++++++++++ arch/x86/include/asm/enclu.h | 8 ++ arch/x86/include/uapi/asm/sgx.h | 128 ++++++++++++++++++ 5 files changed, 296 insertions(+) create mode 100644 arch/x86/entry/vdso/vsgx_enter_enclave.S create mode 100644 arch/x86/include/asm/enclu.h
arch/x86/entry/vdso/Makefile +2 -0
@@ -29,6 +29,7 @@ VDSO32-$(CONFIG_IA32_EMULATION) := y vobjs-y := vdso-note.o vclock_gettime.o vgetcpu.o vobjs32-y := vdso32/note.o vdso32/system_call.o vdso32/sigreturn.o vobjs32-y += vdso32/vclock_gettime.o +vobjs-$(VDSO64-y) += vsgx_enter_enclave.o # files to link into kernel obj-y += vma.o extable.o
@@ -100,6 +101,7 @@ $(vobjs): KBUILD_CFLAGS := $(filter-out $(GCC_PLUGINS_CFLAGS) $(RETPOLINE_CFLAGS CFLAGS_REMOVE_vclock_gettime.o = -pg CFLAGS_REMOVE_vdso32/vclock_gettime.o = -pg CFLAGS_REMOVE_vgetcpu.o = -pg +CFLAGS_REMOVE_vsgx_enter_enclave.o = -pg # # X32 processes use x32 vDSO to access 64bit kernel data.
arch/x86/entry/vdso/vdso.lds.S +1 -0
@@ -27,6 +27,7 @@ VERSION { __vdso_time; clock_getres; __vdso_clock_getres; + __vdso_sgx_enter_enclave; local: *; }; }
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +157 -0
157 lines not on this page · raw message
arch/x86/include/asm/enclu.h new file +8 -0
@@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ENCLU_H +#define _ASM_X86_ENCLU_H + +#define EENTER 0x02 +#define ERESUME 0x03 + +#endif /* _ASM_X86_ENCLU_H */
arch/x86/include/uapi/asm/sgx.h +128 -0
132 lines not on this page · raw message
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 5 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +0
-0
Why not simply
arch/x86/entry/vdso/sgx.S
?
Quoted 9 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +8
-0@@ -0,0 +1,157 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include <linux/linkage.h> +#include <asm/export.h> +#include <asm/errno.h> +#include <asm/enclu.h> + +#include "extable.h"
arch/x86/entry/vdso/vsgx_enter_enclave.S line 18
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +10
-0+ +/* Offset of 'struct sgx_enclave_run' relative to %rbp. */ +#define SGX_ENCLAVE_RUN_PTR 2*8 + +/* Offsets into 'struct sgx_enclave_run'. */ +#define SGX_ENCLAVE_RUN_TSC 0*8 +#define SGX_ENCLAVE_RUN_FLAGS 1*8 +#define SGX_ENCLAVE_RUN_EXIT_REASON 1*8 + 4 +#define SGX_ENCLAVE_RUN_USER_HANDLER 2*8 +/* #define SGX_ENCLAVE_RUN_USER_DATA 3*8 */
What's with that guy? Left here as documentation showing what's at 3*8 offset?
arch/x86/entry/vdso/vsgx_enter_enclave.S line 22
Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+#define SGX_ENCLAVE_RUN_EXCEPTION 4*8 + +#define SGX_SYNCHRONOUS_EXIT 0 +#define SGX_EXCEPTION_EXIT 1
Those are in ...uapi/asm/sgx.h too. Unify?
arch/x86/entry/vdso/vsgx_enter_enclave.S line 35
Quoted 13 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +13
-0+ +/* Offsets into sgx_enter_enclave.exception. */ +#define SGX_EX_LEAF 0*8 +#define SGX_EX_TRAPNR 0*8+4 +#define SGX_EX_ERROR_CODE 0*8+6 +#define SGX_EX_ADDRESS 1*8 + +.code64 +.section .text, "ax" + +SYM_FUNC_START(__vdso_sgx_enter_enclave) + /* Prolog */ + .cfi_startproc
Are the CFI annotations for userspace?
Quoted 13 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +13
-0+ push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + push %rbx + .cfi_rel_offset %rbx, -8 + + mov %ecx, %eax +.Lenter_enclave: + /* EENTER <= leaf <= ERESUME */ + cmp $EENTER, %eax + jb .Linvalid_input
arch/x86/entry/vdso/vsgx_enter_enclave.S line 58
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +10
-0+ cmp $ERESUME, %eax + ja .Linvalid_input + + mov SGX_ENCLAVE_RUN_PTR(%rbp), %rcx + + /* No flags are currently defined/supported. */ + cmpl $0, SGX_ENCLAVE_RUN_FLAGS(%rcx) + jne .Linvalid_input + + /* Load TCS and AEP */
TSC
I can see why you would write "TCS" though - there's a thread control structure thing too in that patch.
Quoted 42 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +42
-0+ mov SGX_ENCLAVE_RUN_TSC(%rcx), %rbx + lea .Lasync_exit_pointer(%rip), %rcx + + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +.Lasync_exit_pointer: +.Lenclu_eenter_eresume: + enclu + + /* EEXIT jumps here unless the enclave is doing something fancy. */ + mov SGX_ENCLAVE_RUN_PTR(%rbp), %rbx + + /* Set exit_reason. */ + movl $SGX_SYNCHRONOUS_EXIT, SGX_ENCLAVE_RUN_EXIT_REASON(%rbx) + + /* Invoke userspace's exit handler if one was provided. */ +.Lhandle_exit: + cmpq $0, SGX_ENCLAVE_RUN_USER_HANDLER(%rbx) + jne .Linvoke_userspace_handler + + /* Success, in the sense that ENCLU was attempted. */ + xor %eax, %eax + +.Lout: + pop %rbx + leave + .cfi_def_cfa %rsp, 8 + ret + + /* The out-of-line code runs with the pre-leave stack frame. */ + .cfi_def_cfa %rbp, 16 + +.Linvalid_input: + mov $(-EINVAL), %eax + jmp .Lout + +.Lhandle_exception: + mov SGX_ENCLAVE_RUN_PTR(%rbp), %rbx + + /* Set the exit_reason and exception info. */ + movl $SGX_EXCEPTION_EXIT, SGX_ENCLAVE_RUN_EXIT_REASON(%rbx) + + mov %eax, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_LEAF)(%rbx)
arch/x86/entry/vdso/vsgx_enter_enclave.S line 110
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +10
-0+ mov %di, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_TRAPNR)(%rbx) + mov %si, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_ERROR_CODE)(%rbx) + mov %rdx, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_ADDRESS)(%rbx) + jmp .Lhandle_exit + +.Linvoke_userspace_handler: + /* Pass the untrusted RSP (at exit) to the callback via %rcx. */ + mov %rsp, %rcx + + /* Save @e, %rbx is about to be clobbered. */
I've been wondering what that "@e" thing is and I believe I've found it at the
end: "... @e, the optional sgx_enclave_exception struct"
Yes?
Please rewrite what that "e" is here - I think you wanna say "struct sgx_enclave_run.exception" instead to clarify.
...
arch/x86/include/uapi/asm/sgx.h
Quoted 18 lines
arch/x86/include/asm/enclu.h new file +8
-0@@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ENCLU_H +#define _ASM_X86_ENCLU_H + +#define EENTER 0x02 +#define ERESUME 0x03 + +#endif /* _ASM_X86_ENCLU_H */ arch/x86/include/uapi/asm/sgx.h +0
-0
Are all those defines being added to the uapi header, also actually needed by userspace?
Quoted 9 lines
arch/x86/include/uapi/asm/sgx.h +5
-0@@ -72,4 +72,132 @@ struct sgx_enclave_provision {__u64 attribute_fd; }; +#define SGX_SYNCHRONOUS_EXIT 0 +#define SGX_EXCEPTION_EXIT 1 + +struct sgx_enclave_run ; +
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_provision { · line 89
Quoted 10 lines
arch/x86/include/uapi/asm/sgx.h +10
-0+/** +* typedef sgx_enclave_exit_handler_t - Exit handler function accepted by +* __vdso_sgx_enter_enclave() +* +* @rdi: RDI at the time of enclave exit +* @rsi: RSI at the time of enclave exit +* @rdx: RDX at the time of enclave exit +* @ursp: RSP at the time of enclave exit (untrusted stack) +* @r8: R8 at the time of enclave exit +* @r9: R9 at the time of enclave exit
I'm sure you can avoid that "at the time of enclave exit" repetition.
...
Quoted 12 lines
arch/x86/include/uapi/asm/sgx.h +12
-0+/** +* typedef vdso_sgx_enter_enclave_t - Prototype for __vdso_sgx_enter_enclave(), +* a vDSO function to enter an SGX enclave. +* +* @rdi: Pass-through value for RDI +* @rsi: Pass-through value for RSI +* @rdx: Pass-through value for RDX +* @leaf: ENCLU leaf, must be EENTER or ERESUME +* @r8: Pass-through value for R8 +* @r9: Pass-through value for R9 +* @r: struct sgx_enclave_run, must be non-NULL +*
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_provision { · line 166
Quoted 10 lines
arch/x86/include/uapi/asm/sgx.h +10
-0+* NOTE: __vdso_sgx_enter_enclave() does not ensure full compliance with the +* x86-64 ABI, e.g. doesn't handle XSAVE state. Except for non-volatile +* general purpose registers, EFLAGS.DF, and RSP alignment, preserving/setting +* state in accordance with the x86-64 ABI is the responsibility of the enclave +* and its runtime, i.e. __vdso_sgx_enter_enclave() cannot be called from C +* code without careful consideration by both the enclave and its runtime. +* +* All general purpose registers except RAX, RBX and RCX are passed as-is to +* the enclave. RAX, RBX and RCX are consumed by EENTER and ERESUME and are +* loaded with @leaf, asynchronous exit pointer, and @tcs respectively.
That tcs is probably struct sgx_enclave_run.tcs?
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_provision { · line 169
Quoted 2 lines
arch/x86/include/uapi/asm/sgx.h +2
-0+* RBP and the stack are used to anchor __vdso_sgx_enter_enclave() to the +* pre-enclave state, e.g. to retrieve @e and @handler after an enclave exit.
Expand @e and do you mean "@user_handler" with "@handler"?
Quoted 37 lines
arch/x86/include/uapi/asm/sgx.h +33
-0+* All other registers are available for use by the enclave and its runtime, +* e.g. an enclave can push additional data onto the stack (and modify RSP) to +* pass information to the optional exit handler (see below). +* +* Most exceptions reported on ENCLU, including those that occur within the +* enclave, are fixed up and reported synchronously instead of being delivered +* via a standard signal. Debug Exceptions (#DB) and Breakpoints (#BP) are +* never fixed up and are always delivered via standard signals. On synchrously +* reported exceptions, -EFAULT is returned and details about the exception are +* recorded in @e, the optional sgx_enclave_exception struct. +* +* If an exit handler is provided, the handler will be invoked on synchronous +* exits from the enclave and for all synchronously reported exceptions. In +* latter case, @e is filled prior to invoking the handler. +* +* The exit handler's return value is interpreted as follows: +* >0: continue, restart __vdso_sgx_enter_enclave() with @ret as @leaf +* 0: success, return @ret to the caller +* <0: error, return @ret to the caller +* +* The exit handler may transfer control, e.g. via longjmp() or C++ exception, +* without returning to __vdso_sgx_enter_enclave(). +* +* Return: +* 0 on success (ENCLU reached), +* -EINVAL if ENCLU leaf is not allowed, +* -errno for all other negative values returned by the userspace exit handler +*/ +typedef int (*vdso_sgx_enter_enclave_t )(unsigned long rdi,unsigned long rsi, +unsigned long rdx,unsigned int leaf, +unsigned long r8,unsigned long r9, +struct sgx_enclave_run *r); +#endif /* _UAPI_ASM_X86_SGX_H */ Signature · 1 line
2.25.1
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 9 lines · Borislav Petkov
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 5 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +0
-0Why not simply
Quoted 3 lines
arch/x86/entry/vdso/sgx.S?
I renamed it as vsgx.S (for the sake of convention).
Quoted 14 lines
Quoted 14 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +13
-0@@ -0,0 +1,157 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include <linux/linkage.h> +#include <asm/export.h> +#include <asm/errno.h> +#include <asm/enclu.h> + +#include "extable.h" + +/* Offset of 'struct sgx_enclave_run' relative to %rbp. */ +#define SGX_ENCLAVE_RUN_PTR 2*8 + +/* Offsets into 'struct sgx_enclave_run'. */
Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 18
Quoted 5 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +5
-0+#define SGX_ENCLAVE_RUN_TSC 0*8 +#define SGX_ENCLAVE_RUN_FLAGS 1*8 +#define SGX_ENCLAVE_RUN_EXIT_REASON 1*8 + 4 +#define SGX_ENCLAVE_RUN_USER_HANDLER 2*8 +/* #define SGX_ENCLAVE_RUN_USER_DATA 3*8 */What's with that guy? Left here as documentation showing what's at 3*8 offset?
Yes.
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 22
Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+#define SGX_ENCLAVE_RUN_EXCEPTION 4*8 + +#define SGX_SYNCHRONOUS_EXIT 0 +#define SGX_EXCEPTION_EXIT 1Those are in
...uapi/asm/sgx.htoo. Unify?
I have not authored this patch but what I would propose is to use just raw value in the place of these constants. It is practially just a boolean value.
I can also add sgx_vdso.h to uapi directory. I just don't see the point.
Holding on for feedback with this one.
Quoted 7 lines
Quoted 7 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +7
-0+ +/* Offsets into sgx_enter_enclave.exception. */ +#define SGX_EX_LEAF 0*8 +#define SGX_EX_TRAPNR 0*8+4 +#define SGX_EX_ERROR_CODE 0*8+6 +#define SGX_EX_ADDRESS 1*8 +
Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 35
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+.code64 +.section .text, "ax" + +SYM_FUNC_START(__vdso_sgx_enter_enclave) + /* Prolog */ + .cfi_startprocAre the CFI annotations for userspace?
Yes, for stack unwinding. However, I would leave them out. Holding on for further feedback with this tho.
Quoted 26 lines
Quoted 13 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +13
-0+ push %rbp + .cfi_adjust_cfa_offset 8 + .cfi_rel_offset %rbp, 0 + mov %rsp, %rbp + .cfi_def_cfa_register %rbp + push %rbx + .cfi_rel_offset %rbx, -8 + + mov %ecx, %eax +.Lenter_enclave: + /* EENTER <= leaf <= ERESUME */ + cmp $EENTER, %eax + jb .Linvalid_inputarch/x86/entry/vdso/vsgx_enter_enclave.S line 58
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +10
-0+ cmp $ERESUME, %eax + ja .Linvalid_input + + mov SGX_ENCLAVE_RUN_PTR(%rbp), %rcx + + /* No flags are currently defined/supported. */ + cmpl $0, SGX_ENCLAVE_RUN_FLAGS(%rcx) + jne .Linvalid_input + + /* Load TCS and AEP */TSC
Quoted 2 lines
I can see why you would write "TCS" though - there's a thread control structure thing too in that patch.
Renamed.
Quoted 56 lines
Quoted 42 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +42
-0+ mov SGX_ENCLAVE_RUN_TSC(%rcx), %rbx + lea .Lasync_exit_pointer(%rip), %rcx + + /* Single ENCLU serving as both EENTER and AEP (ERESUME) */ +.Lasync_exit_pointer: +.Lenclu_eenter_eresume: + enclu + + /* EEXIT jumps here unless the enclave is doing something fancy. */ + mov SGX_ENCLAVE_RUN_PTR(%rbp), %rbx + + /* Set exit_reason. */ + movl $SGX_SYNCHRONOUS_EXIT, SGX_ENCLAVE_RUN_EXIT_REASON(%rbx) + + /* Invoke userspace's exit handler if one was provided. */ +.Lhandle_exit: + cmpq $0, SGX_ENCLAVE_RUN_USER_HANDLER(%rbx) + jne .Linvoke_userspace_handler + + /* Success, in the sense that ENCLU was attempted. */ + xor %eax, %eax + +.Lout: + pop %rbx + leave + .cfi_def_cfa %rsp, 8 + ret + + /* The out-of-line code runs with the pre-leave stack frame. */ + .cfi_def_cfa %rbp, 16 + +.Linvalid_input: + mov $(-EINVAL), %eax + jmp .Lout + +.Lhandle_exception: + mov SGX_ENCLAVE_RUN_PTR(%rbp), %rbx + + /* Set the exit_reason and exception info. */ + movl $SGX_EXCEPTION_EXIT, SGX_ENCLAVE_RUN_EXIT_REASON(%rbx) + + mov %eax, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_LEAF)(%rbx)arch/x86/entry/vdso/vsgx_enter_enclave.S line 110
Quoted 10 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +10
-0+ mov %di, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_TRAPNR)(%rbx) + mov %si, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_ERROR_CODE)(%rbx) + mov %rdx, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_ADDRESS)(%rbx) + jmp .Lhandle_exit + +.Linvoke_userspace_handler: + /* Pass the untrusted RSP (at exit) to the callback via %rcx. */ + mov %rsp, %rcx + + /* Save @e, %rbx is about to be clobbered. */I've been wondering what that "@e" thing is and I believe I've found it at the
end: "... @e, the optionalsgx_enclave_exceptionstruct"
Quoted 4 lines
Yes?
Please rewrite what that "e" is here - I think you wanna say "struct sgx_enclave_run.exception" instead to clarify.
I agree. Open coded them as "struct sgx_enclave_exception".
Quoted 16 lines
...
Quoted 13 lines
arch/x86/include/asm/enclu.h new file +7
-0@@ -0,0 +1,8 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_ENCLU_H +#define _ASM_X86_ENCLU_H + +#define EENTER 0x02 +#define ERESUME 0x03 +
Quoted 8 lines
arch/x86/include/uapi/asm/sgx.h
Quoted 5 lines
arch/x86/include/asm/enclu.h new file +1
-0+#endif /* _ASM_X86_ENCLU_H */ arch/x86/include/uapi/asm/sgx.h +0
-0Are all those defines being added to the uapi header, also actually needed by userspace?
I'd remove the two constants, as it is clearly just a boolean value.
Quoted 13 lines
Quoted 13 lines
arch/x86/include/uapi/asm/sgx.h +9
-0@@ -72,4 +72,132 @@ struct sgx_enclave_provision {__u64 attribute_fd; }; +#define SGX_SYNCHRONOUS_EXIT 0 +#define SGX_EXCEPTION_EXIT 1 + +struct sgx_enclave_run ; + +/** +* typedef sgx_enclave_exit_handler_t - Exit handler function accepted by +* __vdso_sgx_enter_enclave() +*
Quoted 8 lines
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_provision { · line 89
Quoted 6 lines
arch/x86/include/uapi/asm/sgx.h +6
-0+* @rdi: RDI at the time of enclave exit +* @rsi: RSI at the time of enclave exit +* @rdx: RDX at the time of enclave exit +* @ursp: RSP at the time of enclave exit (untrusted stack) +* @r8: R8 at the time of enclave exit +* @r9: R9 at the time of enclave exit I'm sure you can avoid that "at the time of enclave exit" repetition.
I edited this as
/**
* typedef sgx_enclave_exit_handler_t - Exit handler function accepted by
* __vdso_sgx_enter_enclave()
* @rdi: RDI snapshot
* @rsi: RSI snapshot
* @rdx: RDX snapshot
* @rsp: RSP snapshot (untrusted stack)
* @r8: R8 snapshot
* @r9: R9 snapshot
* @run: Pointer to the caller provided struct sgx_enclave_run
*
* Return:
* 0 or negative to exit vDSO
* positive to re-enter enclave (must be EENTER or ERESUME leaf) */
typedef int (*sgx_enclave_exit_handler_t)(long rdi, long rsi, long rdx,
long rsp, long r8, long r9, struct sgx_enclave_run *run);
Quoted 23 lines
Quoted 12 lines
arch/x86/include/uapi/asm/sgx.h +12
-0+/** +* typedef vdso_sgx_enter_enclave_t - Prototype for __vdso_sgx_enter_enclave(), +* a vDSO function to enter an SGX enclave. +* +* @rdi: Pass-through value for RDI +* @rsi: Pass-through value for RSI +* @rdx: Pass-through value for RDX +* @leaf: ENCLU leaf, must be EENTER or ERESUME +* @r8: Pass-through value for R8 +* @r9: Pass-through value for R9 +* @r: struct sgx_enclave_run, must be non-NULL +* arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_provision { · line 166
Quoted 10 lines
arch/x86/include/uapi/asm/sgx.h +10
-0+* NOTE: __vdso_sgx_enter_enclave() does not ensure full compliance with the +* x86-64 ABI, e.g. doesn't handle XSAVE state. Except for non-volatile +* general purpose registers, EFLAGS.DF, and RSP alignment, preserving/setting +* state in accordance with the x86-64 ABI is the responsibility of the enclave +* and its runtime, i.e. __vdso_sgx_enter_enclave() cannot be called from C +* code without careful consideration by both the enclave and its runtime. +* +* All general purpose registers except RAX, RBX and RCX are passed as-is to +* the enclave. RAX, RBX and RCX are consumed by EENTER and ERESUME and are +* loaded with @leaf, asynchronous exit pointer, and @tcs respectively.
Quoted 1 line
That tcs is probably struct sgx_enclave_run.tcs?
Yes.
Quoted 4 lines
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_provision { · line 169
Quoted 2 lines
arch/x86/include/uapi/asm/sgx.h +2
-0+* RBP and the stack are used to anchor __vdso_sgx_enter_enclave() to the +* pre-enclave state, e.g. to retrieve @e and @handler after an enclave exit. Expand @e and do you mean "@user_handler" with "@handler"?
I replaced this and similar fields as @run.user_handler and so forth.
Quoted 30 lines
Quoted 30 lines
arch/x86/include/uapi/asm/sgx.h +29
-0+* All other registers are available for use by the enclave and its runtime, +* e.g. an enclave can push additional data onto the stack (and modify RSP) to +* pass information to the optional exit handler (see below). +* +* Most exceptions reported on ENCLU, including those that occur within the +* enclave, are fixed up and reported synchronously instead of being delivered +* via a standard signal. Debug Exceptions (#DB) and Breakpoints (#BP) are +* never fixed up and are always delivered via standard signals. On synchrously +* reported exceptions, -EFAULT is returned and details about the exception are +* recorded in @e, the optional sgx_enclave_exception struct. +* +* If an exit handler is provided, the handler will be invoked on synchronous +* exits from the enclave and for all synchronously reported exceptions. In +* latter case, @e is filled prior to invoking the handler. +* +* The exit handler's return value is interpreted as follows: +* >0: continue, restart __vdso_sgx_enter_enclave() with @ret as @leaf +* 0: success, return @ret to the caller +* <0: error, return @ret to the caller +* +* The exit handler may transfer control, e.g. via longjmp() or C++ exception, +* without returning to __vdso_sgx_enter_enclave(). +* +* Return: +* 0 on success (ENCLU reached), +* -EINVAL if ENCLU leaf is not allowed, +* -errno for all other negative values returned by the userspace exit handler +*/ +typedef int (*vdso_sgx_enter_enclave_t)(unsigned long rdi, unsigned long rsi,
Quoted 14 lines
Quoted 8 lines
arch/x86/include/uapi/asm/sgx.h +4
-0+ unsigned long rdx, unsigned int leaf, + unsigned long r8, unsigned long r9, + struct sgx_enclave_run *r); +#endif /* _UAPI_ASM_X86_SGX_H */ Signature · 1 line
2.25.1Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
Also, I renamed 'r' as 'run' in some places.
End result:
I'm wondering this sentence:
"The calling convention is custom and does not follow System V x86-64 ABI."
AFAIK, now the vDSO is fully C-callable but waiting for feedback before removing it.
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 1 line · Jarkko Sakkinen
I renamed it as
vsgx.S(for the sake of convention).
Right.
Quoted 5 lines
I have not authored this patch but what I would propose is to use just raw value in the place of these constants. It is practially just a boolean value.
I can also add
sgx_vdso.hto uapi directory. I just don't see the point.
Just be very cautious what you add to the uapi/ directory because it becomes API and there's no changing it. That's why I point you guys to it, to think hard what you expose there and that it becomes contract with luserspace.
Quoted 4 lines
Quoted 2 lines
I can see why you would write "TCS" though - there's a thread control structure thing too in that patch.
Renamed.
See Sean's reply.
Quoted 9 lines
/**
* typedefsgx_enclave_exit_handler_t- Exit handler function accepted by
*__vdso_sgx_enter_enclave()
* @rdi: RDI snapshot
* @rsi: RSI snapshot
* @rdx: RDX snapshot
* @rsp: RSP snapshot (untrusted stack)
* @r8: R8 snapshot
* @r9: R9 snapshot
I'd say here:
"The registers' content is the snapshot made at enclave exit."
Quoted 9 lines
Also, I renamed 'r' as 'run' in some places.
End result:
I'm wondering this sentence:
"The calling convention is custom and does not follow System V x86-64 ABI."
Yeah, I was wondering what that meant too.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 6 lines · Borislav Petkov
Quoted 4 lines
Quoted 2 lines
I can see why you would write "TCS" though - there's a thread control structure thing too in that patch.
Renamed.
See Sean's reply.
I did not get Sean's reply, and neither can find it from lore:
https://lore.kernel.org/linux-sgx/[email protected]/T/#t
Quoted 9 lines
Quoted 8 lines
* typedef
sgx_enclave_exit_handler_t- Exit handler function accepted by
*__vdso_sgx_enter_enclave()
* @rdi: RDI snapshot
* @rsi: RSI snapshot
* @rdx: RDX snapshot
* @rsp: RSP snapshot (untrusted stack)
* @r8: R8 snapshot
* @r9: R9 snapshot
Quoted 3 lines
I'd say here:
"The registers' content is the snapshot made at enclave exit."
I'd make that a description and take away individual parameter descriptions. Is that fine?
Quoted 11 lines
Quoted 9 lines
Also, I renamed 'r' as 'run' in some places.
End result:
I'm wondering this sentence:
"The calling convention is custom and does not follow System V x86-64 ABI."
Yeah, I was wondering what that meant too.
I'll refine that one based on my own and Jethro's feedback.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 3 lines · Jarkko Sakkinen
I did not get Sean's reply, and neither can find it from lore:
Yah, your mail server upgrade broke a lot of stuff. And lore even says it is not there:
2020-09-25 11:43 ` Jethro Beekman
[not found] ` <[email protected]> <---
2020-09-25 1:04 ` Jarkko Sakkinen
Lemme bounce it to you.
Quoted 2 lines
I'd make that a description and take away individual parameter descriptions. Is that fine?
Sure.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 5 lines · Borislav Petkov
Quoted 3 lines · Jarkko Sakkinen
I did not get Sean's reply, and neither can find it from lore:
Quoted 8 lines
Yah, your mail server upgrade broke a lot of stuff. And lore even says it is not there:
2020-09-25 11:43 ` Jethro Beekman
[not found] ` <[email protected]> <---2020-09-25 1:04 ` Jarkko Sakkinen
Lemme bounce it to you.
Thank you. I think I have it correctly in my tree. And I actually noticed that I had the original email stored in wrong archive folder on my machine (sorry about that), so did I receive it after all, but it does not exist in lore.
Quoted 4 lines
Quoted 2 lines
I'd make that a description and take away individual parameter descriptions. Is that fine?
Sure.
/**
* typedef sgx_enclave_exit_handler_t - Exit handler function accepted by
* __vdso_sgx_enter_enclave()
* @run: Pointer to the caller provided struct sgx_enclave_run
*
* The register parameters contain the snapshot of their values at enclave
* exit
*
* Return:
* 0 or negative to exit vDSO
* positive to re-enter enclave (must be EENTER or ERESUME leaf) */
typedef int (*sgx_enclave_exit_handler_t)(long rdi, long rsi, long rdx,
long rsp, long r8, long r9, struct sgx_enclave_run *run);
I think this looks reasonable now.
Another minor clean up I made is:
struct sgx_enclave_run {
__u64 tcs;
__u32 flags;
__u32 exit_reason;
__u64 user_handler;
__u64 user_data;I.e. got rid of the "user_handler union. Makes the struc less confusing looking and is consistent with the other structs.
I've been thinking about this tail:
union {
struct sgx_enclave_exception exception;
/* Pad the entire struct to 256 bytes. */
__u8 pad[256 - 32];
};
};I'd just replace this with
__u64 exception; };
And do something like (just writing it to the email to show the idea, have not even compiled this):
- mov %eax, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_LEAF)(%rbx) - mov %di, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_TRAPNR)(%rbx) - mov %si, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_ERROR_CODE)(%rbx) - mov %rdx, (SGX_ENCLAVE_RUN_EXCEPTION + SGX_EX_ADDRESS)(%rbx) + mov SGX_ENCLAVE_RUN_EXCEPTION(%rbx), %rbx + + mov %eax, (SGX_EX_LEAF)(%rbx) + mov %di, (SGX_EX_TRAPNR)(%rbx) + mov %si, (SGX_EX_ERROR_CODE)(%rbx) + mov %rdx, (SGX_EX_ADDRESS)(%rbx)
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to Jarkko SakkinenQuoted 10 lines · Jarkko Sakkinen
End result:
I'm wondering this sentence:
"The calling convention is custom and does not follow System V x86-64 ABI."
AFAIK, now the vDSO is fully C-callable but waiting for feedback before removing it.
It's C-callable *iff your enclave follows the System V x86-64 ABI*. In addition, all registers not clobbered by the SGX ISA are passed to the enclave, not just those specified as parameter passing registers in the ABI. This is intentional to make the vDSO function usable in applications that use the current flexibility of ENCLU.
--
Jethro Beekman | Fortanix
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 9 lines · Jethro Beekman
Quoted 8 lines · Jarkko Sakkinen
End result:
I'm wondering this sentence:
"The calling convention is custom and does not follow System V x86-64 ABI."
Quoted 8 lines
Quoted 2 lines
AFAIK, now the vDSO is fully C-callable but waiting for feedback before removing it.
It's C-callable *iff your enclave follows the System V x86-64 ABI*. In addition, all registers not clobbered by the SGX ISA are passed to the enclave, not just those specified as parameter passing registers in the ABI. This is intentional to make the vDSO function usable in applications that use the current flexibility of ENCLU.
Hold on, I really want to fix this bit of documentation before sending any new version, so I'll explain how I understand it.
I think it is just SystemV ABI call with six parameters in the usual GPRs (rdi, rsi, rdx, rcx, r8, r9).
I'm looking at vdso_sgx_enter_enclave.
What I'm not getting?
Quoted 2 lines
--
Jethro Beekman | Fortanix
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 22 lines · Jarkko Sakkinen
Quoted 9 lines · Jethro Beekman
Quoted 8 lines · Jarkko Sakkinen
End result:
I'm wondering this sentence:
"The calling convention is custom and does not follow System V x86-64 ABI."
Quoted 8 lines
Quoted 2 lines
AFAIK, now the vDSO is fully C-callable but waiting for feedback before removing it.
It's C-callable *iff your enclave follows the System V x86-64 ABI*. In addition, all registers not clobbered by the SGX ISA are passed to the enclave, not just those specified as parameter passing registers in the ABI. This is intentional to make the vDSO function usable in applications that use the current flexibility of ENCLU.
Hold on, I really want to fix this bit of documentation before sending any new version, so I'll explain how I understand it.
Quoted 6 lines
I think it is just SystemV ABI call with six parameters in the usual GPRs (rdi, rsi, rdx, rcx, r8, r9).
I'm looking at
vdso_sgx_enter_enclave.What I'm not getting?
This can't be observed from looking at the code in vdso_sgx_enter_enclave, because what makes this "custom" is the absence of code or code in the enclave.
If you call vdso_sgx_enter_enclave() from C but your enclave doesn't respect the ABI (for example, it clobbers callee-saved registers), your code will break. But if you call vdso_sgx_enter_enclave from assembly, you can use enclaves with any ABI as long as your assembly and the enclave agree on that ABI.
Alternatively, when using assembly, I can pass stuff to the enclave in registers besides rdi, rsi, rdx, rcx, r8, r9, just as if I were manually calling ENCLU from assembly.
The vDSO assembly has been carefully written to support both scenarios by only using rax, rbx, rcx, rbp, rsp and passing rdi, rsi, rdx, r8, r9 (and everything else).
Quoted 10 lines
+ * NOTE:
__vdso_sgx_enter_enclave()does not ensure full compliance with the
+ * x86-64 ABI, e.g. doesn't handle XSAVE state. Except for non-volatile
+ * general purpose registers, EFLAGS.DF, and RSP alignment, preserving/setting
+ * state in accordance with the x86-64 ABI is the responsibility of the enclave
+ * and its runtime, i.e.__vdso_sgx_enter_enclave()cannot be called from C
+ * code without careful consideration by both the enclave and its runtime.
+ *
+ * All general purpose registers except RAX, RBX and RCX are passed as-is to
+ * the enclave. RAX, RBX and RCX are consumed by EENTER and ERESUME and are
+ * loaded with @leaf, asynchronous exit pointer, and @tcs respectively.
Perhaps this should be updated to read "All general purpose registers except RAX, RBX, RCX, RBP and RSP ..."
--
Jethro Beekman | Fortanix
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 10 lines · Sean Christopherson
Quoted 9 lines · Borislav Petkov
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 5 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +0
-0Why not simply
Quoted 5 lines
Quoted 3 lines
arch/x86/entry/vdso/sgx.S?
I really like typing?
I fixed bunch of things. For that sentence I need feedback + for any other possible changes please send a patch if required.
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to a message not in the archiveQuoted 7 lines · Sean Christopherson
Quoted 5 lines
Why not simply
arch/x86/entry/vdso/sgx.S?
I really like typing?
I'll say.
Quoted 2 lines
Yes, to call out that there's a field there, but a field that the vDSO should never touch.
You wanna enforce that in the vdso code?
Because if it is there, luserspace will touch it.
Quoted 6 lines
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 22
Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+#define SGX_ENCLAVE_RUN_EXCEPTION 4*8 + +#define SGX_SYNCHRONOUS_EXIT 0 +#define SGX_EXCEPTION_EXIT 1Those are in
...uapi/asm/sgx.htoo. Unify?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
What about this here?
Quoted 3 lines
Quoted 1 line
Are the CFI annotations for userspace?
Yes, though that's a 90% confident "yes".
Looks like we wanna support it:
f24f91088427 ("x86/vdso: Define BUILD_VDSO while building and emit .eh_frame in asm")
Quoted 2 lines
Argh, it's actually supposed to be TCS,
SGX_ENCLAVE_RUN_TSCis the one that's wrong.
Whoopsie :-)
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 9 lines · Borislav Petkov
Quoted 6 lines
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 22
Quoted 4 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+#define SGX_ENCLAVE_RUN_EXCEPTION 4*8 + +#define SGX_SYNCHRONOUS_EXIT 0 +#define SGX_EXCEPTION_EXIT 1Those are in
...uapi/asm/sgx.htoo. Unify?^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
What about this here?
Repeating myself, but since there is only 0 and 1, I would just use 0 and 1.
Otherwise, I think I pretty much got these comments sorted out.
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to a message not in the archiveQuoted 5 lines · Sean Christopherson
Quoted 2 lines
I can see why you would write "TCS" though - there's a thread control structure thing too in that patch.
Argh, it's actually supposed to be TCS,
SGX_ENCLAVE_RUN_TSCis the one that's wrong.
So I presume that I fixed this one already:
https://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-sgx.git/tree/arch/x86/entry/vdso/vsgx.S
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to Jarkko SakkinenQuoted 6 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +0
-0
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 9 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +8
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline: + call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + ret
I hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Whichever of these two large series lands first is going to inflict fixing this problem on the other.
As the vdso text is global (to a first approximation), it must not be a retpoline if any other process is liable to want to use CET-SS.
If the retpoline really does need to stay, then the vdso probably needs to gain suitable __x86_indirect_thunk_%reg thunks which are patched at boot based on the system properties.
~Andrew
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 11 lines · Andrew Cooper
Quoted 10 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +3
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline: + call 2f
Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 5 lines
+1: pause
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+ lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
I know CET only conceptual level (separate stack holding return addresses as an measure against return oriented programming (ROP)).
Quoted 5 lines
Whichever of these two large series lands first is going to inflict fixing this problem on the other.
As the vdso text is global (to a first approximation), it must not be a retpoline if any other process is liable to want to use CET-SS.
Why is that?
Quoted 5 lines
If the retpoline really does need to stay, then the vdso probably needs to gain suitable __x86_indirect_thunk_%reg thunks which are patched at boot based on the system properties.
~Andrew
aka without CET it is patched? With CET, not?
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 19 lines · Jarkko Sakkinen
Quoted 10 lines · Andrew Cooper
Quoted 9 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +2
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline:Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+ call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 2 lines
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
CET Shadow Stacks detect attacks which modify the return address on the stack.
Retpoline *is* a ROP gadget. It really does modify the return address on the stack, even if its purpose is defensive (vs Spectre v2) rather than malicious.
Quoted 6 lines
Quoted 5 lines
Whichever of these two large series lands first is going to inflict fixing this problem on the other.
As the vdso text is global (to a first approximation), it must not be a retpoline if any other process is liable to want to use CET-SS.
Why is that?
Because when CET-SS is enabled, the ret will suffer a #CP exception (return address on the stack not matching the one recorded in the shadow stack), which I presume/hope is wired into SIGSEGV.
~Andrew
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 34 lines · Andrew Cooper
Quoted 19 lines · Jarkko Sakkinen
Quoted 10 lines · Andrew Cooper
Quoted 9 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +2
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline:Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+ call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 2 lines
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
CET Shadow Stacks detect attacks which modify the return address on the stack.
Retpoline *is* a ROP gadget. It really does modify the return address on the stack, even if its purpose is defensive (vs Spectre v2) rather than malicious.
Quoted 3 lines
Quoted 3 lines
Whichever of these two large series lands first is going to inflict fixing this problem on the other.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
As the vdso text is global (to a first approximation), it must not be a retpoline if any other process is liable to want to use CET-SS.
Why is that?
Because when CET-SS is enabled, the ret will suffer a #CP exception (return address on the stack not matching the one recorded in the shadow stack), which I presume/hope is wired into SIGSEGV.
Here is the CET compatible retpoline:
endbr64
/* Check if shadow stack is in use. NB: R11 is the only usable
scratch register for function calls. */
xorl %r11d, %r11d
rdsspq %r11
testq %r11, %r11
jnz 3f
call 2f
1:
pause
lfence
jmp 1b
2:
mov %rax, (%rsp)
ret
3:
/* Shadow stack is in use. Make the indirect call. */ call *%rax
ret
Signature · 1 line
H.J.
Andy Lutomirski
maintainerRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 58 lines · H.J. Lu
Quoted 34 lines · Andrew Cooper
Quoted 19 lines · Jarkko Sakkinen
Quoted 10 lines · Andrew Cooper
Quoted 9 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +2
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline:Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+ call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 2 lines
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
CET Shadow Stacks detect attacks which modify the return address on the stack.
Retpoline *is* a ROP gadget. It really does modify the return address on the stack, even if its purpose is defensive (vs Spectre v2) rather than malicious.
Quoted 3 lines
Quoted 3 lines
Whichever of these two large series lands first is going to inflict fixing this problem on the other.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
As the vdso text is global (to a first approximation), it must not be a retpoline if any other process is liable to want to use CET-SS.
Why is that?
Because when CET-SS is enabled, the ret will suffer a #CP exception (return address on the stack not matching the one recorded in the shadow stack), which I presume/hope is wired into SIGSEGV.
Here is the CET compatible retpoline:
endbr64
/* Check if shadow stack is in use. NB: R11 is the only usable
scratch register for function calls. */
xorl %r11d, %r11d
rdsspq %r11
testq %r11, %r11
jnz 3f
call 2f
1:
pause
lfence
Quoted 8 lines
jmp 1b
2:
mov %rax, (%rsp)
ret
3:
/* Shadow stack is in use. Make the indirect call. */ call *%rax
ret
What do we expect user programs to do on CET systems? It would be nice if we could instead ALTERNATIVE this out if X86_FEATURE_SHSTK.
--Andy
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 19 lines · Andy Lutomirski
Quoted 11 lines
endbr64
/* Check if shadow stack is in use. NB: R11 is the only usable
scratch register for function calls. */
xorl %r11d, %r11d
rdsspq %r11
testq %r11, %r11
jnz 3f
call 2f
1:
pause
lfenceQuoted 8 lines
jmp 1b
2:
mov %rax, (%rsp)
ret
3:
/* Shadow stack is in use. Make the indirect call. */ call *%rax
ret
Quoted 2 lines
What do we expect user programs to do on CET systems? It would be nice if we could instead ALTERNATIVE this out if
X86_FEATURE_SHSTK.
Shouldn't we just be able to use X86_FEATURE_RETPOLINE?
We probably need a mechanism to force X86_FEATURE_SHSTK and X86_FEATURE_RETPOLINE to be mutually exclusive if we don't have one already.
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 20 lines · Dave Hansen
Quoted 19 lines · Andy Lutomirski
Quoted 19 lines
endbr64
/* Check if shadow stack is in use. NB: R11 is the only usable
scratch register for function calls. */
xorl %r11d, %r11d
rdsspq %r11
testq %r11, %r11
jnz 3f
call 2f
1:
pause
lfence
jmp 1b
2:
mov %rax, (%rsp)
ret
3:
/* Shadow stack is in use. Make the indirect call. */ call *%rax
ret
Quoted 7 lines
Quoted 2 lines
What do we expect user programs to do on CET systems? It would be nice if we could instead ALTERNATIVE this out if
X86_FEATURE_SHSTK.Shouldn't we just be able to use
X86_FEATURE_RETPOLINE?We probably need a mechanism to force
X86_FEATURE_SHSTKandX86_FEATURE_RETPOLINEto be mutually exclusive if we don't have one already.
First of all: lets go with boot time patching instead of dynamic detection. It's both easier to implement and by all other merits makes a lot more sense. It was just a thing that I've not used before.
That sorted out, does it matter which direction I look it at? I could use either feature flag as basis (and I do not have a personal preference here).
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to Andy LutomirskiQuoted 68 lines · Andy Lutomirski
Quoted 58 lines · H.J. Lu
Quoted 34 lines · Andrew Cooper
Quoted 19 lines · Jarkko Sakkinen
Quoted 10 lines · Andrew Cooper
Quoted 9 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +2
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline:Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+ call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 2 lines
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
CET Shadow Stacks detect attacks which modify the return address on the stack.
Retpoline *is* a ROP gadget. It really does modify the return address on the stack, even if its purpose is defensive (vs Spectre v2) rather than malicious.
Quoted 3 lines
Quoted 3 lines
Whichever of these two large series lands first is going to inflict fixing this problem on the other.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
As the vdso text is global (to a first approximation), it must not be a retpoline if any other process is liable to want to use CET-SS.
Why is that?
Because when CET-SS is enabled, the ret will suffer a #CP exception (return address on the stack not matching the one recorded in the shadow stack), which I presume/hope is wired into SIGSEGV.
Here is the CET compatible retpoline:
endbr64
/* Check if shadow stack is in use. NB: R11 is the only usable
scratch register for function calls. */
xorl %r11d, %r11d
rdsspq %r11
testq %r11, %r11
jnz 3f
call 2f
1:
pause
lfenceQuoted 8 lines
jmp 1b
2:
mov %rax, (%rsp)
ret
3:
/* Shadow stack is in use. Make the indirect call. */ call *%rax
ret
Quoted 4 lines
What do we expect user programs to do on CET systems? It would be nice if we could instead ALTERNATIVE this out if
X86_FEATURE_SHSTK.--Andy
I'm open to do either solution. My thinking was to initially do things vsgx.S local (i.e. consider ALTERNATIVE post upstreaming) and use the above solution but I'm also fine doing ALTERNATIVE. Dave kindly briefed on details how that thing works and it should be perfectly usable for our use case.
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 73 lines · Jarkko Sakkinen
Quoted 71 lines · Andy Lutomirski
Quoted 58 lines · H.J. Lu
Quoted 34 lines · Andrew Cooper
Quoted 19 lines · Jarkko Sakkinen
Quoted 10 lines · Andrew Cooper
Quoted 9 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +2
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline:Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+ call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 2 lines
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
CET Shadow Stacks detect attacks which modify the return address on the stack.
Retpoline *is* a ROP gadget. It really does modify the return address on the stack, even if its purpose is defensive (vs Spectre v2) rather than malicious.
Quoted 3 lines
Quoted 3 lines
Whichever of these two large series lands first is going to inflict fixing this problem on the other.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
As the vdso text is global (to a first approximation), it must not be a retpoline if any other process is liable to want to use CET-SS.
Why is that?
Because when CET-SS is enabled, the ret will suffer a #CP exception (return address on the stack not matching the one recorded in the shadow stack), which I presume/hope is wired into SIGSEGV.
Here is the CET compatible retpoline:
endbr64
/* Check if shadow stack is in use. NB: R11 is the only usable
scratch register for function calls. */
xorl %r11d, %r11d
rdsspq %r11
testq %r11, %r11
jnz 3f
call 2f
1:
pause
lfenceQuoted 8 lines
jmp 1b
2:
mov %rax, (%rsp)
ret
3:
/* Shadow stack is in use. Make the indirect call. */ call *%rax
retWhat do we expect user programs to do on CET systems? It would be nice if we could instead ALTERNATIVE this out if
X86_FEATURE_SHSTK.
Quoted 8 lines
Quoted 1 line
--Andy
I'm open to do either solution. My thinking was to initially do things
vsgx.Slocal (i.e. consider ALTERNATIVE post upstreaming) and use the above solution but I'm also fine doing ALTERNATIVE. Dave kindly briefed on details how that thing works and it should be perfectly usable for our use case.
Since SHSTK and IBT are enabled per process, not the whole machine, are you going to patch vDSO on a per-process basis?
Signature · 1 line
H.J.
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 8 lines · H.J. Lu
Quoted 6 lines
I'm open to do either solution. My thinking was to initially do things
vsgx.Slocal (i.e. consider ALTERNATIVE post upstreaming) and use the above solution but I'm also fine doing ALTERNATIVE. Dave kindly briefed on details how that thing works and it should be perfectly usable for our use case.Since SHSTK and IBT are enabled per process, not the whole machine, are you going to patch vDSO on a per-process basis?
No.
Retpolines mitigate Spectre v2 attacks. If you're not vulnerable to Spectre v2, you don't need retpolines.
All processors which support CET *also* have hardware mitigations against Spectre v2 and don't need retpolines. Here's all of the possibilities:
CET=y, BUG_SPECTRE_V2=y: does not exist CET=n, BUG_SPECTRE_V2=y: vulnerable, use retpoline CET=y, BUG_SPECTRE_V2=n: no retpoline, not vulnerable CET=n, BUG_SPECTRE_V2=n: no retpoline, not vulnerable
Andy Lutomirski
maintainerRe: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 16 lines · Dave Hansen
Quoted 8 lines · H.J. Lu
Quoted 6 lines
I'm open to do either solution. My thinking was to initially do things
vsgx.Slocal (i.e. consider ALTERNATIVE post upstreaming) and use the above solution but I'm also fine doing ALTERNATIVE. Dave kindly briefed on details how that thing works and it should be perfectly usable for our use case.Since SHSTK and IBT are enabled per process, not the whole machine, are you going to patch vDSO on a per-process basis?
No.
Retpolines mitigate Spectre v2 attacks. If you're not vulnerable to Spectre v2, you don't need retpolines.
Quoted 8 lines
All processors which support CET *also* have hardware mitigations against Spectre v2 and don't need retpolines. Here's all of the possibilities:
CET=y, BUG_SPECTRE_V2=y: does not exist CET=n, BUG_SPECTRE_V2=y: vulnerable, use retpoline CET=y, BUG_SPECTRE_V2=n: no retpoline, not vulnerable CET=n, BUG_SPECTRE_V2=n: no retpoline, not vulnerable
Just to confirm: does this mean that the CPU mitigates against user code mistraining the branch predictors for CPL0? Because this is the vDSO, and the situation we're actually concerned about is user code mistraining its own branch predictors. This could happen cross-process or within the same process.
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 19 lines · Andy Lutomirski
Quoted 18 lines · Dave Hansen
Quoted 8 lines · H.J. Lu
Quoted 6 lines
I'm open to do either solution. My thinking was to initially do things
vsgx.Slocal (i.e. consider ALTERNATIVE post upstreaming) and use the above solution but I'm also fine doing ALTERNATIVE. Dave kindly briefed on details how that thing works and it should be perfectly usable for our use case.Since SHSTK and IBT are enabled per process, not the whole machine, are you going to patch vDSO on a per-process basis?
No.
Retpolines mitigate Spectre v2 attacks. If you're not vulnerable to Spectre v2, you don't need retpolines.
All processors which support CET *also* have hardware mitigations against Spectre v2 and don't need retpolines. Here's all of the possibilities:
Quoted 6 lines
Quoted 4 lines
CET=y, BUG_SPECTRE_V2=y: does not exist CET=n, BUG_SPECTRE_V2=y: vulnerable, use retpoline CET=y, BUG_SPECTRE_V2=n: no retpoline, not vulnerable CET=n, BUG_SPECTRE_V2=n: no retpoline, not vulnerable
Just to confirm: does this mean that the CPU mitigates against user code mistraining the branch predictors for CPL0?
If (and only if) you have eIBRS enabled.
eIBRS should be available on all CET-capable hardware, and Linux ought to use it by default.
Quoted 4 lines
Because this is the
vDSO, and the situation we're actually concerned about is user code mistraining its own branch predictors. This could happen cross-process or within the same process.
There is nothing (in Intel parts) which prevents mode same-mode training of indirect branches, either in user or kernel space.
However, an IBPB on context switch should prevent cross-process trailing attacks.
~Andrew
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 10 lines · Andrew Cooper
Quoted 6 lines
Quoted 4 lines
CET=y, BUG_SPECTRE_V2=y: does not exist CET=n, BUG_SPECTRE_V2=y: vulnerable, use retpoline CET=y, BUG_SPECTRE_V2=n: no retpoline, not vulnerable CET=n, BUG_SPECTRE_V2=n: no retpoline, not vulnerable
Just to confirm: does this mean that the CPU mitigates against user code mistraining the branch predictors for CPL0?
If (and only if) you have eIBRS enabled.
eIBRS should be available on all CET-capable hardware, and Linux ought to use it by default.
You're totally right, of course. I was (wrongly) thinking about this VDSO retpoline as kernel code.
There's another wrinkle here. Let's say we're vulnerable to a Spectre-v2-style attack and we want to mitigate it on CET hardware that has enhanced IBRS. I'm not sure how reliable of a mitigation retpolines are on enhanced IBRS hardware. Intel has recommended _against_ using them in some cases:
"On processors that support enhanced IBRS, it should be used for mitigation instead of retpoline."
I actually authored that bit of the whitepaper, and I recall that this was not simply a recommendation based on performance advantages of using enhanced IBRS. I can dig through some old email if we decide that we want to explore using a retpoline on enhanced IBRS hardware.
But, let's take a step back. The changelog for this patch needs to at least have:
1. What is the attack being mitigated by the retpoline?
2. Do we actually want to mitigate it?
3. What options are there to mitigate it?
4. Which option does this patch use and why?
Right now, there's not even a comment about this.
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 20 lines · Dave Hansen
Quoted 10 lines · Andrew Cooper
Quoted 6 lines
Quoted 4 lines
CET=y, BUG_SPECTRE_V2=y: does not exist CET=n, BUG_SPECTRE_V2=y: vulnerable, use retpoline CET=y, BUG_SPECTRE_V2=n: no retpoline, not vulnerable CET=n, BUG_SPECTRE_V2=n: no retpoline, not vulnerable
Just to confirm: does this mean that the CPU mitigates against user code mistraining the branch predictors for CPL0?
If (and only if) you have eIBRS enabled.
eIBRS should be available on all CET-capable hardware, and Linux ought to use it by default.
You're totally right, of course. I was (wrongly) thinking about this VDSO retpoline as kernel code.
There's another wrinkle here. Let's say we're vulnerable to a Spectre-v2-style attack and we want to mitigate it on CET hardware that has enhanced IBRS. I'm not sure how reliable of a mitigation retpolines are on enhanced IBRS hardware. Intel has recommended _against_ using them in some cases:
Quoted 7 lines
"On processors that support enhanced IBRS, it should be used for mitigation instead of retpoline."
I actually authored that bit of the whitepaper, and I recall that this was not simply a recommendation based on performance advantages of using enhanced IBRS. I can dig through some old email if we decide that we want to explore using a retpoline on enhanced IBRS hardware.
If only life were simple.
In light of https://arxiv.org/abs/2008.02307 which managed to demonstrate that the original KAISER was actually a speculative attack and nothing to do with the prefetch instruction, a discussion about same-mode training happened.
The updated recommendation given was to continue using retpoline as well as eIBRS to prevent same-mode training of the syscall indirect branch. Josh (CC'd) has been doing a lot of work to find and fix other speculative leaks in this area.
For Skylake uarch and later, even if an RSB underflow leads to a BTB lookup, it still requires an interrupt/NMI to hit one of two instruction boundaries to empty the RSB, and an attacker with that level of control probably has more interesting things to be trying to do.
Without retpoline (or something even more expensive such as IRET-ing around), an attacker can still create speculative type confusion between different system calls, when eIBRS is active.
Once you mix CET-SS in, this breaks, unless you're prepared to update the retpoline gadget to include a WRSS to modify the shadow stack alongside the regular stack. Add this to the large pile of fun for whomever has the privileg^W chore of implementing supervisor CET support.
Quoted 10 lines
But, let's take a step back. The changelog for this patch needs to at least have:
1. What is the attack being mitigated by the retpoline?
2. Do we actually want to mitigate it?
3. What options are there to mitigate it?
4. Which option does this patch use and why?Right now, there's not even a comment about this.
I agree. The reason for using a retpoline here in the first place is unclear.
~Andrew
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to H.J. LuQuoted 57 lines · H.J. Lu
Quoted 34 lines · Andrew Cooper
Quoted 19 lines · Jarkko Sakkinen
Quoted 10 lines · Andrew Cooper
Quoted 9 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +2
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline:Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+ call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 2 lines
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
CET Shadow Stacks detect attacks which modify the return address on the stack.
Retpoline *is* a ROP gadget. It really does modify the return address on the stack, even if its purpose is defensive (vs Spectre v2) rather than malicious.
Quoted 3 lines
Quoted 3 lines
Whichever of these two large series lands first is going to inflict fixing this problem on the other.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
As the vdso text is global (to a first approximation), it must not be a retpoline if any other process is liable to want to use CET-SS.
Why is that?
Because when CET-SS is enabled, the ret will suffer a #CP exception (return address on the stack not matching the one recorded in the shadow stack), which I presume/hope is wired into SIGSEGV.
Here is the CET compatible retpoline:
endbr64
/* Check if shadow stack is in use. NB: R11 is the only usable
scratch register for function calls. */
xorl %r11d, %r11d
rdsspq %r11
testq %r11, %r11
jnz 3f
call 2f
1:
pause
lfence
Quoted 8 lines
jmp 1b
2:
mov %rax, (%rsp)
ret
3:
/* Shadow stack is in use. Make the indirect call. */ call *%rax
ret
Right, so I have actually two alternatives: this and boot time patching:
https://lkml.org/lkml/2020/9/25/1122
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to Andrew CooperQuoted 23 lines · Andrew Cooper
Quoted 19 lines · Jarkko Sakkinen
Quoted 10 lines · Andrew Cooper
Quoted 9 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +2
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline:Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+ call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 2 lines
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
Quoted 6 lines
CET Shadow Stacks detect attacks which modify the return address on the stack.
Retpoline *is* a ROP gadget. It really does modify the return address on the stack, even if its purpose is defensive (vs Spectre v2) rather than malicious.
Aah. I get that, yes.
Kernel is full of retpoline but I presume that ring-0 does not use CET.
The situation with callback is follows: for a run-time the user_handler by all practical means is always the same. There is ever only one user handler that gets executed. I.e. the indirect callback will always lead to the same thing. I wonder how much assets an adversary would get if we just remove retpoline bits (not much thinking done yet on that).
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 26 lines · Jarkko Sakkinen
Quoted 25 lines · Andrew Cooper
Quoted 19 lines · Jarkko Sakkinen
Quoted 10 lines · Andrew Cooper
Quoted 9 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +2
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline:Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+ call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 2 lines
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
CET Shadow Stacks detect attacks which modify the return address on the stack.
Quoted 6 lines
Quoted 3 lines
Retpoline *is* a ROP gadget. It really does modify the return address on the stack, even if its purpose is defensive (vs Spectre v2) rather than malicious.
Aah. I get that, yes.
Kernel is full of retpoline but I presume that ring-0 does not use CET.
No-one has implemented support CET-SS support for Linux itself yet, but other kernels do have it working.
~Andrew
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 30 lines · Andrew Cooper
Quoted 29 lines · Jarkko Sakkinen
Quoted 28 lines · Andrew Cooper
Quoted 19 lines · Jarkko Sakkinen
Quoted 10 lines · Andrew Cooper
Quoted 9 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +2
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline:Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 6 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +6
-0+ call 2f +1: pause + lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 2 lines
CET goes beyond my expertise. Can you describe, at least rudimentary, how this code is not compatible?
CET Shadow Stacks detect attacks which modify the return address on the stack.
Retpoline *is* a ROP gadget. It really does modify the return address on the stack, even if its purpose is defensive (vs Spectre v2) rather than malicious.
Quoted 8 lines
Quoted 3 lines
Aah. I get that, yes.
Kernel is full of retpoline but I presume that ring-0 does not use CET.
No-one has implemented support CET-SS support for Linux itself yet, but other kernels do have it working.
~Andrew
Haitao, can you point out the user handler callback in the Intel SGX runtime?
There is only one single global callback in a practical deployment provided by the runtime. AFAIK, it just copies values, does not do any rountrips and is generally very trivial peace of code but it is better to check it before final say.
I've now experimented with ALTERNATIVE() and it can be definitely made work. I'm just thinking that would it be best not use retpoline at all?
My guess is that the callback would not have much applicability in Spectre'ish attacks but do not have enough expertise on that to even semiformally conclude it.
My intention is to find reasonable conclusions to drop it instead of adding more complexity to the vDSO.
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to Andrew CooperQuoted 11 lines · Andrew Cooper
Quoted 10 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +3
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline: + call 2f
Quoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 5 lines
+1: pause
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+ lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Hi Jarkko,
These 1: and 2: targets are reached only from these few lines? If they are direct call/jmp targets, I think it is OK in terms of CET. If they are reached from an instruction like "jmp *%rax", then we need to put in an "endbr64".
Yu-cheng
Quoted 13 lines
Whichever of these two large series lands first is going to inflict fixing this problem on the other.
As the vdso text is global (to a first approximation), it must not be a retpoline if any other process is liable to want to use CET-SS.
If the retpoline really does need to stay, then the vdso probably needs to gain suitable __x86_indirect_thunk_%reg thunks which are patched at boot based on the system properties.
~Andrew
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 22 lines · Yu-cheng
Quoted 11 lines · Andrew Cooper
Quoted 10 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +3
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline: + call 2fQuoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 5 lines
+1: pause
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+ lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 7 lines
Hi Jarkko,
These 1: and 2: targets are reached only from these few lines? If they are direct call/jmp targets, I think it is OK in terms of CET. If they are reached from an instruction like "jmp *%rax", then we need to put in an "endbr64".
This also isn't compatible with shadow stack.
Signature · 1 line
H.J.
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
Quoted 25 lines · H.J. Lu
Quoted 24 lines · Yu-cheng
Quoted 11 lines · Andrew Cooper
Quoted 10 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +3
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline: + call 2fQuoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 5 lines
+1: pause
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+ lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Hi Jarkko,
Quoted 8 lines
Quoted 5 lines
These 1: and 2: targets are reached only from these few lines? If they are direct call/jmp targets, I think it is OK in terms of CET. If they are reached from an instruction like "jmp *%rax", then we need to put in an "endbr64".
This also isn't compatible with shadow stack.
Then, when shadow stack is enabled, disable this?
Yu-cheng
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to H.J. LuQuoted 25 lines · H.J. Lu
Quoted 24 lines · Yu-cheng
Quoted 11 lines · Andrew Cooper
Quoted 10 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +3
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline: + call 2fQuoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 5 lines
+1: pause
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+ lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Hi Jarkko,
Quoted 10 lines
Quoted 5 lines
These 1: and 2: targets are reached only from these few lines? If they are direct call/jmp targets, I think it is OK in terms of CET. If they are reached from an instruction like "jmp *%rax", then we need to put in an "endbr64".
This also isn't compatible with shadow stack.
Signature · 1 line
H.J.
I have the now full picture of the problem thanks to Andrew's response [1]. And Dave Hansen just explained me in detail the context and background with [2]. So I'd guess this will get sorted out soon.
If you don't mind I'll CC you to this commit when I send the next version?
[1] https://lkml.org/lkml/2020/9/28/1153
[2] https://lkml.org/lkml/2020/9/25/1122
/Jarkko
Re: [PATCH v38 21/24] x86/vdso: Implement a vDSO for Intel SGX enclave call
In reply to Yu, Yu-chengQuoted 21 lines · Yu-cheng
Quoted 11 lines · Andrew Cooper
Quoted 10 lines · Jarkko Sakkinen
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +3
-0@@ -0,0 +1,157 @@ +SYM_FUNC_START(__vdso_sgx_enter_enclave) +.Lretpoline: + call 2fQuoted 8 lines
arch/x86/entry/vdso/vsgx_enter_enclave.S line 152
Quoted 5 lines
+1: pause
arch/x86/entry/vdso/vsgx_enter_enclave.S new file +4
-0+ lfence + jmp 1b +2: mov %rax, (%rsp) + retI hate to throw further spanners in the work, but this is not compatible with CET, and the user shadow stack work in progress.
Quoted 8 lines
Hi Jarkko,
These 1: and 2: targets are reached only from these few lines? If they are direct call/jmp targets, I think it is OK in terms of CET. If they are reached from an instruction like "jmp *%rax", then we need to put in an "endbr64".
Yu-cheng
The invocation is over here:
/* Load the callback pointer to %rax and invoke it via retpoline. */ mov SGX_ENCLAVE_RUN_USER_HANDLER(%rax), %rax call .Lretpoline
/Jarkko
PATCH v38 22/24 selftests/x86: Add a selftest for SGX
In reply to Jarkko SakkinenAdd a selftest for SGX. It is a trivial test where a simple enclave copies one 64-bit word of memory between two memory locations.
--- tools/testing/selftests/Makefile | 1 + tools/testing/selftests/sgx/.gitignore | 2 + tools/testing/selftests/sgx/Makefile | 53 +++ tools/testing/selftests/sgx/call.S | 44 ++ tools/testing/selftests/sgx/defines.h | 21 + tools/testing/selftests/sgx/load.c | 277 ++++++++++++ tools/testing/selftests/sgx/main.c | 232 ++++++++++ tools/testing/selftests/sgx/main.h | 38 ++ tools/testing/selftests/sgx/sigstruct.c | 395 ++++++++++++++++++ tools/testing/selftests/sgx/test_encl.c | 20 + tools/testing/selftests/sgx/test_encl.lds | 40 ++ .../selftests/sgx/test_encl_bootstrap.S | 89 ++++ 12 files changed, 1212 insertions(+) create mode 100644 tools/testing/selftests/sgx/.gitignore create mode 100644 tools/testing/selftests/sgx/Makefile create mode 100644 tools/testing/selftests/sgx/call.S create mode 100644 tools/testing/selftests/sgx/defines.h create mode 100644 tools/testing/selftests/sgx/load.c create mode 100644 tools/testing/selftests/sgx/main.c create mode 100644 tools/testing/selftests/sgx/main.h create mode 100644 tools/testing/selftests/sgx/sigstruct.c create mode 100644 tools/testing/selftests/sgx/test_encl.c create mode 100644 tools/testing/selftests/sgx/test_encl.lds create mode 100644 tools/testing/selftests/sgx/test_encl_bootstrap.S
tools/testing/selftests/Makefile +1 -0
@@ -68,6 +68,7 @@ TARGETS += user TARGETS += vm TARGETS += x86 TARGETS += zram +TARGETS += sgx #Please keep the TARGETS list alphabetically sorted # Run "make quicktest=1 run_tests" or # "make quicktest=1 kselftest" from top level Makefile
tools/testing/selftests/sgx/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +test_sgx +test_encl.elf
tools/testing/selftests/sgx/Makefile new file +53 -0
53 lines not on this page · raw message
tools/testing/selftests/sgx/call.S new file +44 -0
44 lines not on this page · raw message
tools/testing/selftests/sgx/defines.h new file +21 -0
@@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright(c) 2016-19 Intel Corporation. +*/ + +#ifndef DEFINES_H +#define DEFINES_H + +#include <stdint.h> + +#define PAGE_SIZE 4096 +#define PAGE_MASK (~(PAGE_SIZE - 1)) + +#define __aligned (x) __attribute__((__aligned__(x))) +#define __packed __attribute__((packed)) + +#include "../../../../arch/x86/kernel/cpu/sgx/arch.h" +#include "../../../../arch/x86/include/asm/enclu.h" +#include "../../../../arch/x86/include/uapi/asm/sgx.h" + +#endif /* DEFINES_H */
tools/testing/selftests/sgx/load.c new file +277 -0
277 lines not on this page · raw message
tools/testing/selftests/sgx/main.c new file +232 -0
232 lines not on this page · raw message
tools/testing/selftests/sgx/main.h new file +38 -0
@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* Copyright(c) 2016-19 Intel Corporation. +*/ + +#ifndef MAIN_H +#define MAIN_H + +struct encl_segment { +off_t offset; +size_t size; +unsigned int prot; +unsigned int flags; +}; + +struct encl { +int fd; +void *bin; +off_t bin_size; +void *src; +size_t src_size; +size_t encl_size; +off_t encl_base; +unsigned int nr_segments; +struct encl_segment *segment_tbl; +struct sgx_secs secs; +struct sgx_sigstruct sigstruct; +}; + +void encl_delete (struct encl *ctx); +bool encl_load (const char *path,struct encl *encl); +bool encl_measure (struct encl *encl); +bool encl_build (struct encl *encl); + +int sgx_call_vdso (void *rdi,void *rsi,long rdx,u32 leaf,void *r8,void *r9, +struct sgx_enclave_run *run); + +#endif /* MAIN_H */
tools/testing/selftests/sgx/sigstruct.c new file +395 -0
395 lines not on this page · raw message
tools/testing/selftests/sgx/test_encl.c new file +20 -0
@@ -0,0 +1,20 @@ +// SPDX-License-Identifier: (GPL-2.0 OR BSD-3-Clause) +// Copyright(c) 2016-18 Intel Corporation. + +#include <stddef.h> +#include "defines.h" + +static void *memcpy (void *dest,const void *src,size_t n) +{ +size_t i; + +for (i =0 ; i < n; i++) + ((char *)dest)[i] = ((char *)src)[i]; + +return dest; +} + +void encl_body (void *rdi,void *rsi) +{ +memcpy (rsi, rdi,8 ); +}
tools/testing/selftests/sgx/test_encl.lds new file +40 -0
@@ -0,0 +1,40 @@ +OUTPUT_FORMAT(elf64-x86-64) + +PHDRS +{ + tcs PT_LOAD; + text PT_LOAD; + data PT_LOAD; +} + +SECTIONS +{ + . = 0; + .tcs : { + *(.tcs*) + } : tcs + + . = ALIGN(4096); + .text : { + *(.text*) + *(.rodata*) + } : text + + . = ALIGN(4096); + .data : { + *(.data*) + } : data + + /DISCARD/ : { + *(.comment*) + *(.note*) + *(.debug*) + *(.eh_frame*) + } +} + +ASSERT(!DEFINED(.altinstructions), "ALTERNATIVES are not supported in enclaves") +ASSERT(!DEFINED(.altinstr_replacement), "ALTERNATIVES are not supported in enclaves") +ASSERT(!DEFINED(.discard.retpoline_safe), "RETPOLINE ALTERNATIVES are not supported in enclaves") +ASSERT(!DEFINED(.discard.nospec), "RETPOLINE ALTERNATIVES are not supported in enclaves") +ASSERT(!DEFINED(.got.plt), "Libcalls are not supported in enclaves")
tools/testing/selftests/sgx/test_encl_bootstrap.S new file +89 -0
89 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v38 23/24 docs: x86/sgx: Document SGX micro architecture and kernel internals
In reply to Jarkko SakkinenDocument the Intel SGX kernel architecture. The fine-grained micro architecture details can be looked up from Intel SDM Volume 3D.
--- Documentation/x86/index.rst | 1 + Documentation/x86/sgx.rst | 200 ++++++++++++++++++++++++++++++++++++ 2 files changed, 201 insertions(+) create mode 100644 Documentation/x86/sgx.rst
Documentation/x86/index.rst +1 -0
@@ -30,3 +30,4 @@ x86-specific Documentation usb-legacy-support i386/index x86_64/index + sgx
Documentation/x86/sgx.rst new file +200 -0
200 lines not on this page · raw message
Signature · 1 line
2.25.1
Borislav Petkov
maintainerRe: [PATCH v38 23/24] docs: x86/sgx: Document SGX micro architecture and kernel internals
This patch must come first in the set so that other reviewers can read what all this jazz is about first, before exposing them to the gory details later.
Documentation/x86/sgx.rst line 5
Quoted 11 lines · Jarkko Sakkinen
Documentation/x86/sgx.rst new file +5
-0@@ -0,0 +1,200 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============ +Architecture +============
Only the title of the document should be sandwiched between two lines of
============
as that is what appears in the documentation TOC. As it is right now,it says "Architecture" but it should say "Software Guard eXtensions (SGX)"
Architecture should follow only with a bottom line as a section name:
Architecture
============
Documentation/x86/sgx.rst line 9
Quoted 4 lines
Documentation/x86/sgx.rst new file +4
-0+ +*Software Guard eXtensions (SGX)* is a set of instructions that enable ring-3 +applications to set aside private regions of code and data. These regions are +called enclaves. An enclave can be entered to a fixed set of entry points. Only
s / to / at /
Documentation/x86/sgx.rst line 20
Quoted 11 lines
Documentation/x86/sgx.rst new file +11
-0+a CPU running inside the enclave can access its code and data. + +The support can be determined by + + ``grep sgx /proc/cpuinfo`` + +Enclave Page Cache +================== + +SGX utilizes an *Enclave Page Cache (EPC)* to store pages that are associated +with an enclave. It is contained in a BIOS reserved region of physical memory.
BIOS-reserved
Documentation/x86/sgx.rst line 26
Quoted 6 lines
Documentation/x86/sgx.rst new file +6
-0+Unlike pages used for regular memory, pages can only be accessed outside the +enclave for different purposes with the instructions **ENCLS**, **ENCLV** and +**ENCLU**. + +Direct memory accesses to an enclave can be only done by a CPU executing inside +the enclave. An enclave can be entered with **ENCLU[EENTER]** to a fixed set of
s / to / at /
Quoted 19 lines
Documentation/x86/sgx.rst new file +19
-0+entry points. However, a CPU executing inside the enclave can do outside memory +accesses. + +Page Types +---------- + +**SGX Enclave Control Structure (SECS)** + Enclave's address range, attributes and other global data are defined + by this structure. + +**Regular (REG)** + Regular EPC pages contain the code and data of an enclave. + +**Thread Control Structure (TCS)** + Thread Control Structure pages define the entry points to an enclave and + track the execution state of an enclave thread. + +**Version Array (VA)** + Version Array pages contain 512 slots, each of which can contain a version
Documentation/x86/sgx.rst line 55
Quoted 10 lines
Documentation/x86/sgx.rst new file +10
-0+ number for a page evicted from the EPC. + +Enclave Page Cache Map +---------------------- + +The processor tracks EPC pages via the *Enclave Page Cache Map (EPCM)*. EPCM +contains an entry for each EPC page, which describes the owning enclave, access +rights and page type among the other things. + +The permissions from EPCM is consulted if and only if walking the kernel page
s/ is / are /
Documentation/x86/sgx.rst line 59
Quoted 4 lines
Documentation/x86/sgx.rst new file +4
-0+tables succeeds. The total permissions are thus a conjunction between page table +and EPCM permissions. + +For all intents and purposes the SGX architecture allows the processor to
^ ,
Quoted 20 lines
Documentation/x86/sgx.rst new file +20
-0+invalidate all EPCM entries at will, i.e. requires that software be prepared to +handle an EPCM fault at any time. The contents of EPC are encrypted with an +ephemeral key, which is lost on power transitions. + +EPC management +============== + +EPC pages do not have ``struct page`` instances. They are IO memory from kernel +perspective. The consequence is that they are always mapped as shared memory. +Kernel defines ``/dev/sgx/enclave`` that can be mapped as ``MAP_SHARED`` to +define the address range for an enclave. + +EPC Over-subscription +===================== + +When the amount of free EPC pages goes below a low watermark the swapping thread +starts reclaiming pages. The pages that do not have the **A** bit set are +selected as victim pages. + +Launch Control
Documentation/x86/sgx.rst line 89
Quoted 10 lines
Documentation/x86/sgx.rst new file +10
-0+============== + +SGX provides a launch control mechanism. After all enclave pages have been +copied, kernel executes **ENCLS[EINIT]**, which initializes the enclave. Only +after this the CPU can execute inside the enclave. + +This leaf function takes an RSA-3072 signature of the enclave measurement and an +optional cryptographic token. Linux does not take advantage of launch tokens. +The instruction checks that the signature is signed with the key defined in +**IA32_SGXLEPUBKEYHASH?** MSRs and the measurement is correct. If so, the
That '?' wants to be '[0123]' perhaps?
Documentation/x86/sgx.rst line 92
Quoted 3 lines
Documentation/x86/sgx.rst new file +3
-0+enclave is allowed to be executed. + +MSRs can be configured by the BIOS to be either readable or writable. Linux
"Those MSRs..."
Documentation/x86/sgx.rst
Quoted 5 lines
Documentation/x86/sgx.rst new file +5
-0+supports only writable configuration in order to give full control to the kernel +on launch control policy. Readable configuration requires the use of previously +mentioned launch tokens. + +The current kernel implementation supports only writable MSRs.
This sentence is repeated a couple of lines above.
Quoted 4 lines
The launch is
Documentation/x86/sgx.rst new file +3
-0+performed by setting the MSRs to the hash of the enclave signer's public key. +The alternative would be to have *a launch enclave* that would be signed with +the key set into MSRs, which would then generate launch tokens for other
Documentation/x86/sgx.rst line 110
Quoted 10 lines
Documentation/x86/sgx.rst new file +10
-0+enclaves. This would only make sense with read-only MSRs, and thus the option +has been discarded. + +Attestation +=========== + +Local Attestation +----------------- + +In local attestation an enclave creates a **REPORT** data structure with
^ ,
Quoted 6 lines
Documentation/x86/sgx.rst new file +6
-0+**ENCLS[EREPORT]**, which describes the origin of an enclave. In particular, it +contains a AES-CMAC of the enclave contents signed with a report key unique to +each processor. All enclaves have access to this key. + +This mechanism can also be used in addition as a communication channel as the +**REPORT** data structure includes a 64-byte field for variable information.
...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteRe: [PATCH v38 23/24] docs: x86/sgx: Document SGX micro architecture and kernel internals
Quoted 6 lines · Borislav Petkov
Documentation/x86/sgx.rst line 89
Quoted 4 lines
Documentation/x86/sgx.rst new file +4
-0+This leaf function takes an RSA-3072 signature of the enclave measurement and an +optional cryptographic token. Linux does not take advantage of launch tokens. +The instruction checks that the signature is signed with the key defined in +**IA32_SGXLEPUBKEYHASH?** MSRs and the measurement is correct. If so, theThat '?' wants to be '[0123]' perhaps?
What do you think of this:
"The leaf instruction checks that the measurement is correct and signature is signed with the key hashed to the four +**IA32_SGXLEPUBKEYHASH{0, 1, 2, 3}** MSRs representing the SHA256 of a public key."
I though that little more detail would make sense here (i.e. what do the MSRs represent).
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v38 23/24] docs: x86/sgx: Document SGX micro architecture and kernel internals
Quoted 8 lines · Jarkko Sakkinen
Quoted 6 lines · Borislav Petkov
Documentation/x86/sgx.rst line 89
Quoted 4 lines
Documentation/x86/sgx.rst new file +4
-0+This leaf function takes an RSA-3072 signature of the enclave measurement and an +optional cryptographic token. Linux does not take advantage of launch tokens. +The instruction checks that the signature is signed with the key defined in +**IA32_SGXLEPUBKEYHASH?** MSRs and the measurement is correct. If so, theThat '?' wants to be '[0123]' perhaps?
Quoted 6 lines
What do you think of this:
"The leaf instruction checks that the measurement is correct and signature is signed with the key hashed to the four +**IA32_SGXLEPUBKEYHASH{0, 1, 2, 3}** MSRs representing the SHA256 of a public key."
ACK.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePATCH v38 24/24 x86/sgx: Update MAINTAINERS
In reply to Jarkko SakkinenAdd the maintainer information for the SGX subsystem.
--- MAINTAINERS | 11 +++++++++++ 1 file changed, 11 insertions(+)
MAINTAINERS +11 -0
@@ -9051,6 +9051,17 @@ F: Documentation/x86/intel_txt.rst F: arch/x86/kernel/tboot.c F: include/linux/tboot.h +INTEL SGX +M: Jarkko Sakkinen <[email protected]> +M: Sean Christopherson <[email protected]> +L: [email protected] +S: Maintained +Q: https://patchwork.kernel.org/project/intel-sgx/list/ +T: git https://github.com/jsakkine-intel/linux-sgx.git +F: arch/x86/include/uapi/asm/sgx.h +F: arch/x86/kernel/cpu/sgx/* +K: \bSGX_ + INTERCONNECT API M: Georgi Djakov <[email protected]> L: [email protected]
Signature · 1 line
2.25.1
Re: [PATCH v38 24/24] x86/sgx: Update MAINTAINERS
Quoted 14 lines · Jarkko Sakkinen
Add the maintainer information for the SGX subsystem.
Cc Thomas Gleixner <[email protected]>Cc Borislav Petkov <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>--- MAINTAINERS | 11 +++++++++++ 1 file changed, 11 insertions(+)MAINTAINERS +0
-0
MAINTAINERS F: Documentation/x86/intel_txt.rst · line 9060
Quoted 10 lines
MAINTAINERS +7
-0@@ -9051,6 +9051,17 @@ F: Documentation/x86/intel_txt.rst F: arch/x86/kernel/tboot.c F: include/linux/tboot.h +INTEL SGX +M: Jarkko Sakkinen <[email protected]> +M: Sean Christopherson <[email protected]> +L: [email protected] +S: Maintained +Q: https://patchwork.kernel.org/project/intel-sgx/list/ +T: git https://github.com/jsakkine-intel/linux-sgx.git
This is stale, you moved your tree to kernel.org.
Quoted 10 lines
MAINTAINERS +4
-0+F: arch/x86/include/uapi/asm/sgx.h +F: arch/x86/kernel/cpu/sgx/* +K: \bSGX_ + INTERCONNECT API M: Georgi Djakov <[email protected]> L: [email protected]Signature · 1 line
2.25.1
Re: [PATCH v38 24/24] x86/sgx: Update MAINTAINERS
Quoted 18 lines · Sean Christopherson
Quoted 17 lines · Jarkko Sakkinen
Add the maintainer information for the SGX subsystem.
Cc Thomas Gleixner <[email protected]>Cc Borislav Petkov <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>--- MAINTAINERS | 11 +++++++++++ 1 file changed, 11 insertions(+)MAINTAINERS +0
-0@@ -9051,6 +9051,17 @@ F: Documentation/x86/intel_txt.rst F: arch/x86/kernel/tboot.c F: include/linux/tboot.h
Quoted 9 lines
MAINTAINERS F: Documentation/x86/intel_txt.rst · line 9060
Quoted 7 lines
MAINTAINERS +7
-0+INTEL SGX +M: Jarkko Sakkinen <[email protected]> +M: Sean Christopherson <[email protected]> +L: [email protected] +S: Maintained +Q: https://patchwork.kernel.org/project/intel-sgx/list/ +T: git https://github.com/jsakkine-intel/linux-sgx.gitThis is stale, you moved your tree to kernel.org.
That's correct. It is already updated in my GIT tree:
https://git.kernel.org/pub/scm/linux/kernel/git/jarkko/linux-sgx.git/tree/MAINTAINERS
/Jarkko