x86/sgx: Use mmu_notifier.release() instead of per-vma refcounting
RFC PATCH 1/1 x86/sgx: Use mmu_notifier.release() instead of per-vma refcounting
Using per-vma refcounting to track mm_structs associated with an enclave requires hooking .vm_close(), which in turn prevents the mm from merging vmas (precisely to allow refcounting).
Avoid refcounting encl_mm altogether by registering an mmu_notifier at .mmap(), removing the dying encl_mm at mmu_notifier.release() and protecting mm_list during reclaim via a per-enclave SRCU.
Removing refcounting/vm_close() allows merging of enclave vmas, at the cost of delaying removal of encl_mm structs from mm_list, i.e. an mm is disassociated from an enclave when the mm exits or the enclave dies, as opposed to when the last vma (in a given mm) is closed.
The impact of delying encl_mm removal is its memory footprint and whatever overhead is incurred during EPC reclaim (to walk an mm's vmas). Practically speaking, a stale encl_mm will exist for a meaningful amount of time if and only if the enclave is mapped in a long-lived process and then passed off to another long-lived process. It is expected that the vast majority of use cases will not encounter this condition, e.g. even using a daemon to build enclaves should not result in a stale encl_mm as the builder should never need to mmap() the enclave.
Even if there are scenarios that lead to defunct encl_mms, the cost is likely far outweighed by the benefits of reducing the number of vmas across all enclaves.
Note, using SRCU to protect mm_list is not strictly necessary, i.e. the existing walker with encl_mm refcounting could be massaged to work with mmu_notifier.release(), but the resulting code is subtle and fragile (I never actually got it working). The primary issue is that an encl_mm can't be moved off the list until its refcount goes to zero, otherwise the custom walker goes off into the weeds. The refcount requirement then prevents using mm_list to identify if an mmu_notifier.release() has fired, i.e. another mechanism is needed to guard against races between exit_mmap() and sgx_release().
---
I really wanted to send this as a two patch series to first introduce mmu_notifier and then move to SRCU, but making the existing mm_list walker play nice with mmu_notifier was a never ending nightmare.
The end result isn't as clean as I would like, but I do think it's an improvement over the per-vma refcounting. Maybe Jarkko's reclaim-based reaping will cure all our ills :-)
arch/x86/Kconfig | 2 + arch/x86/kernel/cpu/sgx/driver/ioctl.c | 14 -- arch/x86/kernel/cpu/sgx/driver/main.c | 38 ++++ arch/x86/kernel/cpu/sgx/encl.c | 234 +++++++++++-------------- arch/x86/kernel/cpu/sgx/encl.h | 19 +- arch/x86/kernel/cpu/sgx/reclaim.c | 71 +++----- 6 files changed, 182 insertions(+), 196 deletions(-)
arch/x86/Kconfig +2 -0
@@ -1918,6 +1918,8 @@ config X86_INTEL_MEMORY_PROTECTION_KEYS config INTEL_SGX bool "Intel SGX core functionality" depends on X86_64 && CPU_SUP_INTEL + select MMU_NOTIFIER + select SRCU ---help--- Intel(R) SGX is a set of CPU instructions that can be used by applications to set aside private regions of code and data, referred
arch/x86/kernel/cpu/sgx/driver/ioctl.c +0 -14
@@ -276,7 +276,6 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) {unsigned long encl_size = secs->size +PAGE_SIZE ;struct sgx_epc_page *secs_epc; -struct sgx_encl_mm *encl_mm;unsigned long ssaframesize;struct sgx_pageinfo pginfo;struct sgx_secinfo secinfo;
@@ -311,12 +310,6 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)INIT_WORK (&encl->work, sgx_add_page_worker); - encl_mm =sgx_encl_mm_add (encl, current->mm); -if (IS_ERR (encl_mm)) { - ret =PTR_ERR (encl_mm); - goto err_out; - } - secs_epc =sgx_alloc_page (&encl->secs, true);if (IS_ERR (secs_epc)) { ret =PTR_ERR (secs_epc);
@@ -369,13 +362,6 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs) encl->backing =NULL ; } -if (!list_empty (&encl->mm_list)) { - encl_mm = list_first_entry(&encl->mm_list, struct sgx_encl_mm, - list); -list_del (&encl_mm->list); -kfree (encl_mm); - } -mutex_unlock (&encl->lock);return ret; }
arch/x86/kernel/cpu/sgx/driver/main.c +38 -0
@@ -25,6 +25,7 @@ u32 sgx_xsave_size_tbl[64];static int sgx_open (struct inode *inode,struct file *file) {struct sgx_encl *encl; +int ret; encl =kzalloc (sizeof (*encl),GFP_KERNEL );if (!encl)
@@ -38,6 +39,12 @@ static int sgx_open(struct inode *inode, struct file *file)INIT_LIST_HEAD (&encl->mm_list);spin_lock_init (&encl->mm_lock); + ret =init_srcu_struct (&encl->srcu); +if (ret) { +kfree (encl); +return ret; + } + file->private_data = encl;return 0 ;
@@ -46,6 +53,32 @@ static int sgx_open(struct inode *inode, struct file *file)static int sgx_release (struct inode *inode,struct file *file) {struct sgx_encl *encl = file->private_data; +struct sgx_encl_mm *encl_mm; + +/* +* Objects can't be *moved* off an RCU protected list (deletion is ok), +* nor can the object be freed until after synchronize_srcu(). +*/ +restart: +spin_lock (&encl->mm_lock); +if (list_empty (&encl->mm_list)) { + encl_mm =NULL ; + }else { + encl_mm = list_first_entry(&encl->mm_list, struct sgx_encl_mm, + list); +list_del_rcu (&encl_mm->list); + } +spin_unlock (&encl->mm_lock); + +if (encl_mm) { +synchronize_srcu (&encl->srcu); + +mmu_notifier_unregister (&encl_mm->mmu_notifier, encl_mm->mm); + +sgx_encl_mm_release (encl_mm); + + goto restart; + }kref_put (&encl->refcount, sgx_encl_release);
@@ -63,6 +96,11 @@ static long sgx_compat_ioctl(struct file *filep, unsigned int cmd,static int sgx_mmap (struct file *file,struct vm_area_struct *vma) {struct sgx_encl *encl = file->private_data; +int ret; + + ret =sgx_encl_mm_add (encl, vma->vm_mm); +if (ret) +return ret; vma->vm_ops = &sgx_vm_ops; vma->vm_flags |=VM_PFNMAP |VM_DONTEXPAND |VM_DONTDUMP |VM_IO ;
arch/x86/kernel/cpu/sgx/encl.c +107 -127
271 lines not on this page · raw message
arch/x86/kernel/cpu/sgx/encl.h +13 -6
@@ -9,9 +9,11 @@#include <linux/kref.h> #include <linux/list.h> #include <linux/mm_types.h> +#include <linux/mmu_notifier.h> #include <linux/mutex.h> #include <linux/notifier.h> #include <linux/radix-tree.h> +#include <linux/srcu.h> #include <linux/workqueue.h> /**
@@ -57,8 +59,10 @@ enum sgx_encl_flags {struct sgx_encl_mm {struct sgx_encl *encl;struct mm_struct *mm; -struct kref refcount;struct list_head list; +struct mmu_notifier mmu_notifier; +struct work_struct release_work; +struct rcu_head rcu; };struct sgx_encl {
@@ -72,6 +76,7 @@ struct sgx_encl {spinlock_t mm_lock;struct file *backing;struct kref refcount; +struct srcu_struct srcu;unsigned long base;unsigned long size;unsigned long ssaframesize;
@@ -118,11 +123,13 @@ void sgx_encl_destroy(struct sgx_encl *encl);void sgx_encl_release (struct kref *ref);pgoff_t sgx_encl_get_index (struct sgx_encl *encl,struct sgx_encl_page *page);struct page *sgx_encl_get_backing_page (struct sgx_encl *encl,pgoff_t index); -struct sgx_encl_mm *sgx_encl_next_mm (struct sgx_encl *encl, -struct sgx_encl_mm *encl_mm,int *iter); -struct sgx_encl_mm *sgx_encl_mm_add (struct sgx_encl *encl, -struct mm_struct *mm); -void sgx_encl_mm_release (struct kref *ref); +int sgx_encl_mm_add (struct sgx_encl *encl,struct mm_struct *mm); +static inline void sgx_encl_mm_release (struct sgx_encl_mm *encl_mm) +{ +kref_put (&encl_mm->encl->refcount, sgx_encl_release); + +kfree (encl_mm); +}int sgx_encl_test_and_clear_young (struct mm_struct *mm,struct sgx_encl_page *page); struct sgx_encl_page *sgx_encl_reserve_page(struct sgx_encl *encl,
arch/x86/kernel/cpu/sgx/reclaim.c +22 -49
124 lines not on this page · raw message
Signature · 1 line
2.21.0
Quoted 36 lines · Sean Christopherson
Using per-vma refcounting to track
mm_structsassociated with an enclave requires hooking .vm_close(), which in turn prevents the mm from merging vmas (precisely to allow refcounting).Avoid refcounting
encl_mmaltogether by registering anmmu_notifierat .mmap(), removing the dyingencl_mmatmmu_notifier.release()and protectingmm_listduring reclaim via a per-enclave SRCU.Removing refcounting/vm_close() allows merging of enclave vmas, at the cost of delaying removal of
encl_mmstructs frommm_list, i.e. an mm is disassociated from an enclave when the mm exits or the enclave dies, as opposed to when the last vma (in a given mm) is closed.The impact of delying
encl_mmremoval is its memory footprint and whatever overhead is incurred during EPC reclaim (to walk an mm's vmas). Practically speaking, a staleencl_mmwill exist for a meaningful amount of time if and only if the enclave is mapped in a long-lived process and then passed off to another long-lived process. It is expected that the vast majority of use cases will not encounter this condition, e.g. even using a daemon to build enclaves should not result in a staleencl_mmas the builder should never need tommap()the enclave.Even if there are scenarios that lead to defunct
encl_mms, the cost is likely far outweighed by the benefits of reducing the number of vmas across all enclaves.Note, using SRCU to protect
mm_listis not strictly necessary, i.e. the existing walker withencl_mmrefcounting could be massaged to work withmmu_notifier.release(), but the resulting code is subtle and fragile (I never actually got it working). The primary issue is that anencl_mmcan't be moved off the list until its refcount goes to zero, otherwise the custom walker goes off into the weeds. The refcount requirement then prevents usingmm_listto identify if anmmu_notifier.release()has fired, i.e. another mechanism is needed to guard against races betweenexit_mmap()andsgx_release().
Quoted 3 lines
Cc Dave Hansen <[email protected]>Cc Andy Lutomirski <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
OK, so what this needs is to split two patches. That can be seen from your last paragraph (you are implying it with your note). Then this can be sanely reviewed.
And no, this does not fight by any means with the changes that I'm doing.
/Jarkko
Quoted 41 lines · Jarkko Sakkinen
Quoted 36 lines · Sean Christopherson
Using per-vma refcounting to track
mm_structsassociated with an enclave requires hooking .vm_close(), which in turn prevents the mm from merging vmas (precisely to allow refcounting).Avoid refcounting
encl_mmaltogether by registering anmmu_notifierat .mmap(), removing the dyingencl_mmatmmu_notifier.release()and protectingmm_listduring reclaim via a per-enclave SRCU.Removing refcounting/vm_close() allows merging of enclave vmas, at the cost of delaying removal of
encl_mmstructs frommm_list, i.e. an mm is disassociated from an enclave when the mm exits or the enclave dies, as opposed to when the last vma (in a given mm) is closed.The impact of delying
encl_mmremoval is its memory footprint and whatever overhead is incurred during EPC reclaim (to walk an mm's vmas). Practically speaking, a staleencl_mmwill exist for a meaningful amount of time if and only if the enclave is mapped in a long-lived process and then passed off to another long-lived process. It is expected that the vast majority of use cases will not encounter this condition, e.g. even using a daemon to build enclaves should not result in a staleencl_mmas the builder should never need tommap()the enclave.Even if there are scenarios that lead to defunct
encl_mms, the cost is likely far outweighed by the benefits of reducing the number of vmas across all enclaves.Note, using SRCU to protect
mm_listis not strictly necessary, i.e. the existing walker withencl_mmrefcounting could be massaged to work withmmu_notifier.release(), but the resulting code is subtle and fragile (I never actually got it working). The primary issue is that anencl_mmcan't be moved off the list until its refcount goes to zero, otherwise the custom walker goes off into the weeds. The refcount requirement then prevents usingmm_listto identify if anmmu_notifier.release()has fired, i.e. another mechanism is needed to guard against races betweenexit_mmap()andsgx_release().Quoted 3 lines
Cc Dave Hansen <[email protected]>Cc Andy Lutomirski <[email protected]>Signed-off-by Sean Christopherson <[email protected]>
Quoted 6 lines
OK, so what this needs is to split two patches. That can be seen from your last paragraph (you are implying it with your note). Then this can be sanely reviewed.
And no, this does not fight by any means with the changes that I'm doing.
OK, so please ignore what I just said. You're right what I asked could not be sanely done :-)
/Jarkko