x86/sgx: Clean up and enhance add pages ioctl
Subsystems (3)
cover x86/sgx: Clean up and enhance add pages ioctl
This series is intended to be included in v21 of Jarkko's SGX series and applies on Jarkko's current master:
dfc89a83b5bc ("docs: x86/sgx: Document the enclave API")
The primary goal of the series is to tweak the ioctl for adding pages to an enclave so that it is somewhat extensible, e.g. add a flags field that can be reused for access control integration and SGX2/EAUG, and add a size field so that multiple pages can be added in a single call (batching EADD has been mentioned at various times in the past).
The secondary goal is to improve the performance of building enclaves. Handling multiple pages in a single call helps somewhat, but the real win (for some enclaves) is using the kernel's zero page as the source for EADD when possible.
Sean Christopherson (7):
x86/sgx: Remove dead code to handle non-existent IOR ioctl x86/sgx: Remove unnecessary @cmd parameter from ioctl helpers x86/sgx: Let ioctl helpers do copy to/from user
x86/sgx: Allow userspace to add multiple pages in single ioctl() x86/sgx: Add flag to zero added region instead of copying from source x86/sgx: Use the actual zero page as the source when adding zero pages x86/sgx: Add a reserved field to sgx_enclave_add_region to drop
'packed'
Documentation/x86/sgx/3.API.rst | 2 +- arch/x86/include/uapi/asm/sgx.h | 30 +-- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 252 +++++++++++++++---------- 3 files changed, 171 insertions(+), 113 deletions(-)
Signature · 1 line
2.21.0
PATCH 1/7 x86/sgx: Remove dead code to handle non-existent IOR ioctl
There are currently no SGX ioctls that require writing data back to userspace.
--- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-)
arch/x86/kernel/cpu/sgx/driver/ioctl.c +1 -7
@@ -839,11 +839,5 @@ long sgx_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)if (copy_from_user (data, (void __user *)arg,_IOC_SIZE (cmd)))return -EFAULT ; - ret =handler (filep, cmd, (unsigned long )((void *)data)); -if (!ret && (cmd &IOC_OUT )) { -if (copy_to_user ((void __user *)arg, data,_IOC_SIZE (cmd))) -return -EFAULT ; - } - -return ret; +return handler (filep, cmd, (unsigned long )((void *)data)); }
Signature · 1 line
2.21.0
PATCH 2/7 x86/sgx: Remove unnecessary @cmd parameter from ioctl helpers
In reply to Sean ChristophersonEach ioctl command is directly and uniquely associated with a handler, i.e. the command is implied by the function itself.
--- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-)
arch/x86/kernel/cpu/sgx/driver/ioctl.c +6 -15
@@ -383,7 +383,6 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)/** * sgx_ioc_enclave_create - handler for %SGX_IOC_ENCLAVE_CREATE * @filep: open file to /dev/sgx -* @cmd: the command value * @arg: pointer to an &sgx_enclave_create instance * * Allocate kernel data structures for a new enclave and execute ECREATE after
@@ -397,8 +396,7 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)* 0 on success, * -errno otherwise */ -static long sgx_ioc_enclave_create (struct file *filep,unsigned int cmd, -unsigned long arg) +static long sgx_ioc_enclave_create (struct file *filep,unsigned long arg) {struct sgx_enclave_create *createp = (struct sgx_enclave_create *)arg;struct sgx_encl *encl = filep->private_data;
@@ -579,7 +577,6 @@ static int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr,* sgx_ioc_enclave_add_page - handler for %SGX_IOC_ENCLAVE_ADD_PAGE * * @filep: open file to /dev/sgx -* @cmd: the command value * @arg: pointer to an &sgx_enclave_add_page instance * * Add a page to an uninitialized enclave (EADD), and optionally extend the
@@ -593,8 +590,7 @@ static int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr,* 0 on success, * -errno otherwise */ -static long sgx_ioc_enclave_add_page (struct file *filep,unsigned int cmd, -unsigned long arg) +static long sgx_ioc_enclave_add_page (struct file *filep,unsigned long arg) {struct sgx_enclave_add_page *addp = (void *)arg;struct sgx_encl *encl = filep->private_data;
@@ -721,7 +717,6 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,* sgx_ioc_enclave_init - handler for %SGX_IOC_ENCLAVE_INIT * * @filep: open file to /dev/sgx -* @cmd: the command value * @arg: pointer to an &sgx_enclave_init instance * * Flush any outstanding enqueued EADD operations and perform EINIT. The
@@ -733,8 +728,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,* SGX error code on EINIT failure, * -errno otherwise */ -static long sgx_ioc_enclave_init (struct file *filep,unsigned int cmd, -unsigned long arg) +static long sgx_ioc_enclave_init (struct file *filep,unsigned long arg) {struct sgx_enclave_init *initp = (struct sgx_enclave_init *)arg;struct sgx_encl *encl = filep->private_data;
@@ -770,7 +764,6 @@ static long sgx_ioc_enclave_init(struct file *filep, unsigned int cmd,/** * sgx_ioc_enclave_set_attribute - handler for %SGX_IOC_ENCLAVE_SET_ATTRIBUTE * @filep: open file to /dev/sgx -* @cmd: the command value * @arg: pointer to a struct sgx_enclave_set_attribute instance * * Mark the enclave as being allowed to access a restricted attribute bit.
@@ -786,8 +779,7 @@ static long sgx_ioc_enclave_init(struct file *filep, unsigned int cmd,* * Return: 0 on success, -errno otherwise */ -static long sgx_ioc_enclave_set_attribute (struct file *filep,unsigned int cmd, -unsigned long arg) +static long sgx_ioc_enclave_set_attribute (struct file *filep,unsigned long arg) {struct sgx_enclave_set_attribute *params = (void *)arg;struct sgx_encl *encl = filep->private_data;
@@ -810,8 +802,7 @@ static long sgx_ioc_enclave_set_attribute(struct file *filep, unsigned int cmd,return ret; } -typedef long (*sgx_ioc_t )(struct file *filep,unsigned int cmd, -unsigned long arg); +typedef long (*sgx_ioc_t )(struct file *filep,unsigned long arg);long sgx_ioctl (struct file *filep,unsigned int cmd,unsigned long arg) {
@@ -839,5 +830,5 @@ long sgx_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)if (copy_from_user (data, (void __user *)arg,_IOC_SIZE (cmd)))return -EFAULT ; -return handler (filep, cmd, (unsigned long )((void *)data)); +return handler (filep, (unsigned long )((void *)data)); }
Signature · 1 line
2.21.0
PATCH 3/7 x86/sgx: Let ioctl helpers do copy to/from user
In reply to Sean ChristophersonA future patch will extend SGX_IOC_ENCLAVE_ADD_PAGE to allow userspace to add multiple pages in a single ioctl. Because the number of pages could be quite high, it's possible the ioctl could be interrupted, in which case the driver will need to "return" the number of pages that have been queued, e.g. so that userspace can restart the ioctl in the event that the signal wasn't fatal.
In short, SGX_IOC_ENCLAVE_ADD_PAGE will become a conditional _IOWR and will only do copy_to_user() on a non-zero return value. Rather than implement funky logic, pass the raw userspace argument to the helpers so that they may do copy_{to,from}_user() at will. This kills pre-copying the data, but the value added by pre-copy is tenuous, e.g. it consolidates two lines of code but requires use of a fixed size buffer as well as an indirect handler and thus retpoline. And the pre-copy may lead to reader confusion due to @arg being a user pointer in some flows and a kernel pointer in others. Lastly, in the event that a pure _IOR ioctl is added, there won't be a need to update the existing code (to skip copy_from_user()).
--- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 73 ++++++++++++-------------- 1 file changed, 34 insertions(+), 39 deletions(-)
arch/x86/kernel/cpu/sgx/driver/ioctl.c +34 -39
@@ -383,7 +383,7 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)/** * sgx_ioc_enclave_create - handler for %SGX_IOC_ENCLAVE_CREATE * @filep: open file to /dev/sgx -* @arg: pointer to an &sgx_enclave_create instance +* @arg: userspace pointer to a struct sgx_enclave_create instance * * Allocate kernel data structures for a new enclave and execute ECREATE after * verifying the correctness of the provided SECS.
@@ -396,25 +396,27 @@ static int sgx_encl_create(struct sgx_encl *encl, struct sgx_secs *secs)* 0 on success, * -errno otherwise */ -static long sgx_ioc_enclave_create (struct file *filep,unsigned long arg) +static long sgx_ioc_enclave_create (struct file *filep,void __user *arg) { -struct sgx_enclave_create *createp = (struct sgx_enclave_create *)arg;struct sgx_encl *encl = filep->private_data; +struct sgx_enclave_create ecreate;struct page *secs_page;struct sgx_secs *secs;int ret; +if (copy_from_user (&ecreate, arg,sizeof (ecreate))) +return -EFAULT ; + secs_page =alloc_page (GFP_HIGHUSER );if (!secs_page)return -ENOMEM ; secs =kmap (secs_page); -if (copy_from_user (secs, (void __user *)createp->src,sizeof (*secs))) { +if (copy_from_user (secs, (void __user *)ecreate.src,sizeof (*secs))) { ret = -EFAULT ; goto out; } - ret =sgx_encl_create (encl, secs); out:
@@ -577,7 +579,7 @@ static int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr,* sgx_ioc_enclave_add_page - handler for %SGX_IOC_ENCLAVE_ADD_PAGE * * @filep: open file to /dev/sgx -* @arg: pointer to an &sgx_enclave_add_page instance +* @arg: userspace pointer to a struct sgx_enclave_add_page instance * * Add a page to an uninitialized enclave (EADD), and optionally extend the * enclave's measurement with the contents of the page (EEXTEND). EADD and
@@ -590,16 +592,19 @@ static int sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr,* 0 on success, * -errno otherwise */ -static long sgx_ioc_enclave_add_page (struct file *filep,unsigned long arg) +static long sgx_ioc_enclave_add_page (struct file *filep,void __user *arg) { -struct sgx_enclave_add_page *addp = (void *)arg;struct sgx_encl *encl = filep->private_data; +struct sgx_enclave_add_page addp;struct sgx_secinfo secinfo;struct page *data_page;void *data;int ret; -if (copy_from_user (&secinfo, (void __user *)addp->secinfo, +if (copy_from_user (&addp, arg,sizeof (addp))) +return -EFAULT ; + +if (copy_from_user (&secinfo, (void __user *)addp.secinfo,sizeof (secinfo)))return -EFAULT ;
@@ -609,12 +614,12 @@ static long sgx_ioc_enclave_add_page(struct file *filep, unsigned long arg) data =kmap (data_page); -if (copy_from_user ((void *)data, (void __user *)addp->src,PAGE_SIZE )) { +if (copy_from_user ((void *)data, (void __user *)addp.src,PAGE_SIZE )) { ret = -EFAULT ; goto out; } - ret =sgx_encl_add_page (encl, addp->addr, data, &secinfo, addp->mrmask); + ret =sgx_encl_add_page (encl, addp.addr, data, &secinfo, addp.mrmask);if (ret) goto out;
@@ -717,7 +722,7 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,* sgx_ioc_enclave_init - handler for %SGX_IOC_ENCLAVE_INIT * * @filep: open file to /dev/sgx -* @arg: pointer to an &sgx_enclave_init instance +* @arg: userspace pointer to a struct sgx_enclave_init instance * * Flush any outstanding enqueued EADD operations and perform EINIT. The * Launch Enclave Public Key Hash MSRs are rewritten as necessary to match
@@ -728,15 +733,18 @@ static int sgx_encl_init(struct sgx_encl *encl, struct sgx_sigstruct *sigstruct,* SGX error code on EINIT failure, * -errno otherwise */ -static long sgx_ioc_enclave_init (struct file *filep,unsigned long arg) +static long sgx_ioc_enclave_init (struct file *filep,void __user *arg) { -struct sgx_enclave_init *initp = (struct sgx_enclave_init *)arg;struct sgx_encl *encl = filep->private_data;struct sgx_einittoken *einittoken;struct sgx_sigstruct *sigstruct; +struct sgx_enclave_init einit;struct page *initp_page;int ret; +if (copy_from_user (&einit, arg,sizeof (einit))) +return -EFAULT ; + initp_page =alloc_page (GFP_HIGHUSER );if (!initp_page)return -ENOMEM ;
@@ -746,13 +754,12 @@ static long sgx_ioc_enclave_init(struct file *filep, unsigned long arg) ((unsigned long )sigstruct +PAGE_SIZE /2 );memset (einittoken,0 ,sizeof (*einittoken)); -if (copy_from_user (sigstruct, (void __user *)initp->sigstruct, +if (copy_from_user (sigstruct, (void __user *)einit.sigstruct,sizeof (*sigstruct))) { ret = -EFAULT ; goto out; } - ret =sgx_encl_init (encl, sigstruct, einittoken); out:
@@ -764,7 +771,7 @@ static long sgx_ioc_enclave_init(struct file *filep, unsigned long arg)/** * sgx_ioc_enclave_set_attribute - handler for %SGX_IOC_ENCLAVE_SET_ATTRIBUTE * @filep: open file to /dev/sgx -* @arg: pointer to a struct sgx_enclave_set_attribute instance +* @arg: userspace pointer to a struct sgx_enclave_set_attribute instance * * Mark the enclave as being allowed to access a restricted attribute bit. * The requested attribute is specified via the attribute_fd field in the
@@ -779,14 +786,17 @@ static long sgx_ioc_enclave_init(struct file *filep, unsigned long arg)* * Return: 0 on success, -errno otherwise */ -static long sgx_ioc_enclave_set_attribute (struct file *filep,unsigned long arg) +static long sgx_ioc_enclave_set_attribute (struct file *filep,void __user *arg) { -struct sgx_enclave_set_attribute *params = (void *)arg;struct sgx_encl *encl = filep->private_data; +struct sgx_enclave_set_attribute params;struct file *attribute_file;int ret; - attribute_file =fget (params->attribute_fd); +if (copy_from_user (¶ms, arg,sizeof (params))) +return -EFAULT ; + + attribute_file =fget (params.attribute_fd);if (!attribute_file->f_op)return -EINVAL ;
@@ -802,33 +812,18 @@ static long sgx_ioc_enclave_set_attribute(struct file *filep, unsigned long arg)return ret; } -typedef long (*sgx_ioc_t )(struct file *filep,unsigned long arg); -long sgx_ioctl (struct file *filep,unsigned int cmd,unsigned long arg) { -char data[256 ]; -sgx_ioc_t handler =NULL ; -long ret; -switch (cmd) {case SGX_IOC_ENCLAVE_CREATE : - handler = sgx_ioc_enclave_create; -break ; +return sgx_ioc_enclave_create (filep, (void __user *)arg);case SGX_IOC_ENCLAVE_ADD_PAGE : - handler = sgx_ioc_enclave_add_page; -break ; +return sgx_ioc_enclave_add_page (filep, (void __user *)arg);case SGX_IOC_ENCLAVE_INIT : - handler = sgx_ioc_enclave_init; -break ; +return sgx_ioc_enclave_init (filep, (void __user *)arg);case SGX_IOC_ENCLAVE_SET_ATTRIBUTE : - handler = sgx_ioc_enclave_set_attribute; -break ; +return sgx_ioc_enclave_set_attribute (filep, (void __user *)arg);default :return -ENOIOCTLCMD ; } - -if (copy_from_user (data, (void __user *)arg,_IOC_SIZE (cmd))) -return -EFAULT ; - -return handler (filep, (unsigned long )((void *)data)); }
Signature · 1 line
2.21.0
PATCH 4/7 x86/sgx: Allow userspace to add multiple pages in single ioctl()
In reply to Sean Christopherson...to improve performance when building enclaves by reducing the number of user<->system transitions. Rather than provide arbitrary batching, e.g. with per-page SECINFO and mrmask, take advantage of the fact that any sane enclave will have large swaths of pages with identical properties, e.g. code vs. data sections.
For simplicity and stability in the initial implementation, loop over the existing add page flow instead of taking a more agressive approach, which would require tracking transitions between VMAs and holding mmap_sem for an extended duration.
On an error, update the userspace struct to reflect progress made, e.g. so that the ioctl can be re-invoked to finish adding pages after a non- fatal error.
--- Documentation/x86/sgx/3.API.rst | 2 +- arch/x86/include/uapi/asm/sgx.h | 21 ++-- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 128 +++++++++++++++++-------- 3 files changed, 98 insertions(+), 53 deletions(-)
Documentation/x86/sgx/3.API.rst +1 -1
@@ -22,6 +22,6 @@ controls the `PROVISON_KEY` attribute. .. kernel-doc:: arch/x86/kernel/cpu/sgx/driver/ioctl.c :functions: sgx_ioc_enclave_create - sgx_ioc_enclave_add_page + sgx_ioc_enclave_add_region sgx_ioc_enclave_init sgx_ioc_enclave_set_attribute
arch/x86/include/uapi/asm/sgx.h +11 -10
@@ -12,8 +12,8 @@#define SGX_IOC_ENCLAVE_CREATE \ _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create) -#define SGX_IOC_ENCLAVE_ADD_PAGE \ - _IOW(SGX_MAGIC, 0x01, struct sgx_enclave_add_page) +#define SGX_IOC_ENCLAVE_ADD_REGION \ + _IOWR(SGX_MAGIC, 0x01, struct sgx_enclave_add_region)#define SGX_IOC_ENCLAVE_INIT \ _IOW(SGX_MAGIC, 0x02, struct sgx_enclave_init)#define SGX_IOC_ENCLAVE_SET_ATTRIBUTE \
@@ -32,21 +32,22 @@ struct sgx_enclave_create { };/** -* struct sgx_enclave_add_page - parameter structure for the -* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl -* @addr: address within the ELRANGE -* @src: address for the page data -* @secinfo: address for the SECINFO data -* @mrmask: bitmask for the measured 256 byte chunks +* struct sgx_enclave_add_region - parameter structure for the +* %SGX_IOC_ENCLAVE_ADD_REGION ioctl +* @addr: start address within the ELRANGE +* @src: start address for the pages' data +* @size: size of region, in bytes +* @secinfo: address of the SECINFO data (common to the entire region) +* @mrmask: bitmask of 256 byte chunks to measure (applied per 4k page) */ -struct sgx_enclave_add_page { +struct sgx_enclave_add_region {__u64 addr;__u64 src; +__u64 size;__u64 secinfo;__u16 mrmask; } __attribute__((__packed__ )); -/** * struct sgx_enclave_init - parameter structure for the * %SGX_IOC_ENCLAVE_INIT ioctl
arch/x86/kernel/cpu/sgx/driver/ioctl.c +86 -42
177 lines not on this page · raw message
Signature · 1 line
2.21.0
Re: [PATCH 4/7] x86/sgx: Allow userspace to add multiple pages in single ioctl()
Quoted 11 lines · Sean Christopherson
...to improve performance when building enclaves by reducing the number of user<->system transitions. Rather than provide arbitrary batching, e.g. with per-page SECINFO and mrmask, take advantage of the fact that any sane enclave will have large swaths of pages with identical properties, e.g. code vs. data sections.
For simplicity and stability in the initial implementation, loop over the existing add page flow instead of taking a more agressive approach, which would require tracking transitions between VMAs and holding
mmap_semfor an extended duration.
Quoted 5 lines
On an error, update the userspace struct to reflect progress made, e.g. so that the ioctl can be re-invoked to finish adding pages after a non- fatal error.
Signed-off-by Sean Christopherson <[email protected]>
Probably not going to look at this before other things are settled.
/Jarkko
Re: [PATCH 4/7] x86/sgx: Allow userspace to add multiple pages in single ioctl()
In reply to Sean ChristophersonQuoted 64 lines · Sean Christopherson
...to improve performance when building enclaves by reducing the number of user<->system transitions. Rather than provide arbitrary batching, e.g. with per-page SECINFO and mrmask, take advantage of the fact that any sane enclave will have large swaths of pages with identical properties, e.g. code vs. data sections.
For simplicity and stability in the initial implementation, loop over the existing add page flow instead of taking a more agressive approach, which would require tracking transitions between VMAs and holding
mmap_semfor an extended duration.On an error, update the userspace struct to reflect progress made, e.g. so that the ioctl can be re-invoked to finish adding pages after a non- fatal error.
Signed-off-by Sean Christopherson <[email protected]>--- Documentation/x86/sgx/3.API.rst | 2 +- arch/x86/include/uapi/asm/sgx.h | 21 ++-- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 128 +++++++++++++++++-------- 3 files changed, 98 insertions(+), 53 deletions(-)Documentation/x86/sgx/3.API.rst +1
-1@@ -22,6 +22,6 @@ controls the `PROVISON_KEY` attribute. .. kernel-doc:: arch/x86/kernel/cpu/sgx/driver/ioctl.c :functions: sgx_ioc_enclave_create - sgx_ioc_enclave_add_page + sgx_ioc_enclave_add_region sgx_ioc_enclave_init sgx_ioc_enclave_set_attributearch/x86/include/uapi/asm/sgx.h +7
-8@@ -12,8 +12,8 @@#define SGX_IOC_ENCLAVE_CREATE \ _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create) -#define SGX_IOC_ENCLAVE_ADD_PAGE \ - _IOW(SGX_MAGIC, 0x01, struct sgx_enclave_add_page) +#define SGX_IOC_ENCLAVE_ADD_REGION \ + _IOWR(SGX_MAGIC, 0x01, struct sgx_enclave_add_region)#define SGX_IOC_ENCLAVE_INIT \ _IOW(SGX_MAGIC, 0x02, struct sgx_enclave_init)#define SGX_IOC_ENCLAVE_SET_ATTRIBUTE \@@ -32,21 +32,22 @@ struct sgx_enclave_create { };/** -* struct sgx_enclave_add_page - parameter structure for the -* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl -* @addr: address within the ELRANGE -* @src: address for the page data -* @secinfo: address for the SECINFO data -* @mrmask: bitmask for the measured 256 byte chunks +* struct sgx_enclave_add_region - parameter structure for the +* %SGX_IOC_ENCLAVE_ADD_REGION ioctl +* @addr: start address within the ELRANGE +* @src: start address for the pages' data +* @size: size of region, in bytes
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_create { · line 48
Quoted 10 lines
arch/x86/include/uapi/asm/sgx.h +4
-1+* @secinfo: address of the SECINFO data (common to the entire region) +* @mrmask: bitmask of 256 byte chunks to measure (applied per 4k page) */ -struct sgx_enclave_add_page { +struct sgx_enclave_add_region {__u64 addr;__u64 src; +__u64 size;__u64 secinfo;__u16 mrmask;
Considering:
1. I might want to load multiple pages that are not consecutive in memory.
2. Repeating mrmask (other than 0 or ~0) doesn't really make sense for ranges.
I'd be in favor of an approach that passes an array of sgx_enclave_add_page instead.
Somewhat unrelated: have you considered optionally "gifting" enclave source pages to the kernel (as in vmsplice)? That would avoid the copy_from_user.
--
Jethro Beekman | Fortanix
Re: [PATCH 4/7] x86/sgx: Allow userspace to add multiple pages in single ioctl()
Quoted 75 lines · Jethro Beekman
Quoted 61 lines · Sean Christopherson
...to improve performance when building enclaves by reducing the number of user<->system transitions. Rather than provide arbitrary batching, e.g. with per-page SECINFO and mrmask, take advantage of the fact that any sane enclave will have large swaths of pages with identical properties, e.g. code vs. data sections.
For simplicity and stability in the initial implementation, loop over the existing add page flow instead of taking a more agressive approach, which would require tracking transitions between VMAs and holding
mmap_semfor an extended duration.On an error, update the userspace struct to reflect progress made, e.g. so that the ioctl can be re-invoked to finish adding pages after a non- fatal error.
Signed-off-by Sean Christopherson <[email protected]>--- Documentation/x86/sgx/3.API.rst | 2 +- arch/x86/include/uapi/asm/sgx.h | 21 ++-- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 128 +++++++++++++++++-------- 3 files changed, 98 insertions(+), 53 deletions(-)Documentation/x86/sgx/3.API.rst +1
-1@@ -22,6 +22,6 @@ controls the `PROVISON_KEY` attribute. .. kernel-doc:: arch/x86/kernel/cpu/sgx/driver/ioctl.c :functions: sgx_ioc_enclave_create - sgx_ioc_enclave_add_page + sgx_ioc_enclave_add_region sgx_ioc_enclave_init sgx_ioc_enclave_set_attributearch/x86/include/uapi/asm/sgx.h +7
-8@@ -12,8 +12,8 @@#define SGX_IOC_ENCLAVE_CREATE \ _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create) -#define SGX_IOC_ENCLAVE_ADD_PAGE \ - _IOW(SGX_MAGIC, 0x01, struct sgx_enclave_add_page) +#define SGX_IOC_ENCLAVE_ADD_REGION \ + _IOWR(SGX_MAGIC, 0x01, struct sgx_enclave_add_region)#define SGX_IOC_ENCLAVE_INIT \ _IOW(SGX_MAGIC, 0x02, struct sgx_enclave_init)#define SGX_IOC_ENCLAVE_SET_ATTRIBUTE \@@ -32,21 +32,22 @@ struct sgx_enclave_create { };/** -* struct sgx_enclave_add_page - parameter structure for the -* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl -* @addr: address within the ELRANGE -* @src: address for the page data -* @secinfo: address for the SECINFO data -* @mrmask: bitmask for the measured 256 byte chunks +* struct sgx_enclave_add_region - parameter structure for the +* %SGX_IOC_ENCLAVE_ADD_REGION ioctl +* @addr: start address within the ELRANGE +* @src: start address for the pages' data +* @size: size of region, in bytes arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_create { · line 47
Quoted 10 lines
arch/x86/include/uapi/asm/sgx.h +4
-1+* @secinfo: address of the SECINFO data (common to the entire region) +* @mrmask: bitmask of 256 byte chunks to measure (applied per 4k page) */ -struct sgx_enclave_add_page { +struct sgx_enclave_add_region {__u64 addr;__u64 src; +__u64 size;__u64 secinfo;__u16 mrmask;Considering:
Quoted 6 lines
1. I might want to load multiple pages that are not consecutive in memory.
2. Repeating mrmask (other than 0 or ~0) doesn't really make sense for ranges.I'd be in favor of an approach that passes an array of
sgx_enclave_add_pageinstead.
I'm not opposed to taking an array. The region approach seemed simpler at first glance, but that may not be the case, especially if we get rid of the workqueue. I'll play around with it.
Quoted 2 lines
Somewhat unrelated: have you considered optionally "gifting" enclave source pages to the kernel (as in vmsplice)? That would avoid the
copy_from_user.
If we ditch the workqueue then we probably don't even need to gift the page, e.g. I think we allocate the EPC page prior to taking mmap_sem, and then simply do gup+kmap around EADD. We'd just need to be careful about not allocating EPC pages for ioctls that are guaranteed to fail.
Re: [PATCH 4/7] x86/sgx: Allow userspace to add multiple pages in single ioctl()
Quoted 88 lines · Sean Christopherson
Quoted 75 lines · Jethro Beekman
Quoted 61 lines · Sean Christopherson
...to improve performance when building enclaves by reducing the number of user<->system transitions. Rather than provide arbitrary batching, e.g. with per-page SECINFO and mrmask, take advantage of the fact that any sane enclave will have large swaths of pages with identical properties, e.g. code vs. data sections.
For simplicity and stability in the initial implementation, loop over the existing add page flow instead of taking a more agressive approach, which would require tracking transitions between VMAs and holding
mmap_semfor an extended duration.On an error, update the userspace struct to reflect progress made, e.g. so that the ioctl can be re-invoked to finish adding pages after a non- fatal error.
Signed-off-by Sean Christopherson <[email protected]>--- Documentation/x86/sgx/3.API.rst | 2 +- arch/x86/include/uapi/asm/sgx.h | 21 ++-- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 128 +++++++++++++++++-------- 3 files changed, 98 insertions(+), 53 deletions(-)Documentation/x86/sgx/3.API.rst +1
-1@@ -22,6 +22,6 @@ controls the `PROVISON_KEY` attribute. .. kernel-doc:: arch/x86/kernel/cpu/sgx/driver/ioctl.c :functions: sgx_ioc_enclave_create - sgx_ioc_enclave_add_page + sgx_ioc_enclave_add_region sgx_ioc_enclave_init sgx_ioc_enclave_set_attributearch/x86/include/uapi/asm/sgx.h +7
-8@@ -12,8 +12,8 @@#define SGX_IOC_ENCLAVE_CREATE \ _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create) -#define SGX_IOC_ENCLAVE_ADD_PAGE \ - _IOW(SGX_MAGIC, 0x01, struct sgx_enclave_add_page) +#define SGX_IOC_ENCLAVE_ADD_REGION \ + _IOWR(SGX_MAGIC, 0x01, struct sgx_enclave_add_region)#define SGX_IOC_ENCLAVE_INIT \ _IOW(SGX_MAGIC, 0x02, struct sgx_enclave_init)#define SGX_IOC_ENCLAVE_SET_ATTRIBUTE \@@ -32,21 +32,22 @@ struct sgx_enclave_create { };/** -* struct sgx_enclave_add_page - parameter structure for the -* %SGX_IOC_ENCLAVE_ADD_PAGE ioctl -* @addr: address within the ELRANGE -* @src: address for the page data -* @secinfo: address for the SECINFO data -* @mrmask: bitmask for the measured 256 byte chunks +* struct sgx_enclave_add_region - parameter structure for the +* %SGX_IOC_ENCLAVE_ADD_REGION ioctl +* @addr: start address within the ELRANGE +* @src: start address for the pages' data +* @size: size of region, in bytes arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_create { · line 47
Quoted 10 lines
arch/x86/include/uapi/asm/sgx.h +4
-1+* @secinfo: address of the SECINFO data (common to the entire region) +* @mrmask: bitmask of 256 byte chunks to measure (applied per 4k page) */ -struct sgx_enclave_add_page { +struct sgx_enclave_add_region {__u64 addr;__u64 src; +__u64 size;__u64 secinfo;__u16 mrmask;Considering:
Quoted 6 lines
1. I might want to load multiple pages that are not consecutive in memory.
2. Repeating mrmask (other than 0 or ~0) doesn't really make sense for ranges.I'd be in favor of an approach that passes an array of
sgx_enclave_add_pageinstead.I'm not opposed to taking an array. The region approach seemed simpler at first glance, but that may not be the case, especially if we get rid of the workqueue. I'll play around with it.
Quoted 9 lines
Quoted 2 lines
Somewhat unrelated: have you considered optionally "gifting" enclave source pages to the kernel (as in vmsplice)? That would avoid the
copy_from_user.If we ditch the workqueue then we probably don't even need to gift the page, e.g. I think we allocate the EPC page prior to taking
mmap_sem, and then simply do gup+kmap around EADD. We'd just need to be careful about not allocating EPC pages for ioctls that are guaranteed to fail.
Why gup + kmap? Can't you just do STAC; EADD; CLAC? (Using the appropriate C helpers, of course.)
--Andy
Re: [PATCH 4/7] x86/sgx: Allow userspace to add multiple pages in single ioctl()
Quoted 11 lines · Andy Lutomirski
Quoted 6 lines · Sean Christopherson
If we ditch the workqueue then we probably don't even need to gift the page, e.g. I think we allocate the EPC page prior to taking
mmap_sem, and then simply do gup+kmap around EADD. We'd just need to be careful about not allocating EPC pages for ioctls that are guaranteed to fail.Why gup + kmap? Can't you just do STAC; EADD; CLAC? (Using the appropriate C helpers, of course.)
That should work as well, we already have exception fixup on ENCLS. I obviously haven't given much thought to what all we can do once the workqueue goes bye bye :-)
RE: [PATCH 4/7] x86/sgx: Allow userspace to add multiple pages in single ioctl()
In reply to Sean ChristophersonQuoted 89 lines
From: Christopherson, Sean J
Sent: Thursday, June 13, 2019 9:52 AMQuoted 7 lines · Jethro Beekman
Quoted 6 lines · Sean Christopherson
...to improve performance when building enclaves by reducing the number of user<->system transitions. Rather than provide arbitrary batching, e.g. with per-page SECINFO and mrmask, take advantage of the fact that any sane enclave will have large swaths of pages with identical properties, e.g. code vs. data sections.
Quoted 6 lines
Quoted 6 lines
For simplicity and stability in the initial implementation, loop over the existing add page flow instead of taking a more agressive approach, which would require tracking transitions between VMAs and holding
mmap_semfor an extended duration.On an error, update the userspace struct to reflect progress made,
e.g.
Quoted 44 lines
Quoted 44 lines
so that the ioctl can be re-invoked to finish adding pages after a non- fatal error.
Signed-off-by Sean Christopherson <[email protected]>--- Documentation/x86/sgx/3.API.rst | 2 +- arch/x86/include/uapi/asm/sgx.h | 21 ++-- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 128 +++++++++++++++++-------- 3 files changed, 98 insertions(+), 53 deletions(-)(quoted diff) +0
-0
b/Documentation/x86/sgx/3.API.rstindexb113aeb05f54..44550aa41073100644Documentation/x86/sgx/3.API.rst +1
-1@@ -22,6 +22,6 @@ controls the `PROVISON_KEY` attribute. .. kernel-doc:: arch/x86/kernel/cpu/sgx/driver/ioctl.c :functions: sgx_ioc_enclave_create - sgx_ioc_enclave_add_page + sgx_ioc_enclave_add_region sgx_ioc_enclave_init sgx_ioc_enclave_set_attribute diff --git
a/arch/x86/include/uapi/asm/sgx.hb/arch/x86/include/uapi/asm/sgx.hindex9ed690a38c70..30d114f6b3bd100644arch/x86/include/uapi/asm/sgx.h +2
-1@@ -12,8 +12,8 @@#define SGX_IOC_ENCLAVE_CREATE \ _IOW(SGX_MAGIC, 0x00, struct sgx_enclave_create) -#define SGX_IOC_ENCLAVE_ADD_PAGE \ - _IOW(SGX_MAGIC, 0x01, struct sgx_enclave_add_page) +#define SGX_IOC_ENCLAVE_ADD_REGION \ + _IOWR(SGX_MAGIC, 0x01, struct sgx_enclave_add_region)#define SGX_IOC_ENCLAVE_INIT \ _IOW(SGX_MAGIC, 0x02, struct sgx_enclave_init)#define SGX_IOC_ENCLAVE_SET_ATTRIBUTE \ @@ -32,21 +32,22 @@ struct
sgx_enclave_create{
};
/**
- * structsgx_enclave_add_page- parameter structure for the
- * %SGX_IOC_ENCLAVE_ADD_PAGE ioctl
- * @addr: address within the ELRANGE
- * @src: address for the page dataQuoted 8 lines
Quoted 8 lines
- * @secinfo: address for the SECINFO data
- * @mrmask: bitmask for the measured 256 byte chunks
+ * structsgx_enclave_add_region- parameter structure for the
+ * %SGX_IOC_ENCLAVE_ADD_REGION ioctl
+ * @addr: start address within the ELRANGE
+ * @src: start address for the pages' data
+ * @size: size of region, in bytes
+ * @secinfo: address of the SECINFO data (common to the entireregion)
Quoted 1 line
Quoted 1 line
+ * @mrmask: bitmask of 256 byte chunks to measure (applied per 4k
page)
Quoted 11 lines
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_create { · line 48
Quoted 8 lines
*/
arch/x86/include/uapi/asm/sgx.h +2
-1-struct sgx_enclave_add_page { +struct sgx_enclave_add_region {__u64 addr;__u64 src; +__u64 size;__u64 secinfo;__u16 mrmask;Considering:
Quoted 1 line
1. I might want to load multiple pages that are not consecutive in
memory.
Quoted 3 lines
2. Repeating mrmask (other than 0 or ~0) doesn't really make sense for ranges.
Quoted 6 lines
Quoted 2 lines
I'd be in favor of an approach that passes an array of
sgx_enclave_add_pageinstead.I'm not opposed to taking an array. The region approach seemed simpler at first glance, but that may not be the case, especially if we get rid of the workqueue. I'll play around with it.
In practice, ELF segments are usually mapped continuously in memory. Other components of an enclave such as heaps and stack, are usually trunks of zeroes. I'd just stay simple. After all, there's no requirement that everything has to be done in a single ioctl.
PATCH 5/7 x86/sgx: Add flag to zero added region instead of copying from source
In reply to Sean ChristophersonFor some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Use bit 8 for the SGX_ZERO_REGION flag to avoid an anticipated conflict with passing PROT_{READ,WRITE,EXEC} in bits 2:0, and to leave room in case there is a need for additional protection related bits.
--- arch/x86/include/uapi/asm/sgx.h | 5 +++++ arch/x86/kernel/cpu/sgx/driver/ioctl.c | 15 ++++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-)
arch/x86/include/uapi/asm/sgx.h +5 -0
@@ -31,6 +31,9 @@ struct sgx_enclave_create {__u64 src; }; +/* Zero an added region instead of copying data from a source page. */ +#define SGX_ZERO_REGION 0x100 +/** * struct sgx_enclave_add_region - parameter structure for the * %SGX_IOC_ENCLAVE_ADD_REGION ioctl
@@ -38,6 +41,7 @@ struct sgx_enclave_create {* @src: start address for the pages' data * @size: size of region, in bytes * @secinfo: address of the SECINFO data (common to the entire region) +* @flags: miscellaneous flags * @mrmask: bitmask of 256 byte chunks to measure (applied per 4k page) */ struct sgx_enclave_add_region {
@@ -45,6 +49,7 @@ struct sgx_enclave_add_region {__u64 src;__u64 size;__u64 secinfo; +__u32 flags;__u16 mrmask; } __attribute__((__packed__));
arch/x86/kernel/cpu/sgx/driver/ioctl.c +12 -3
@@ -459,6 +459,9 @@ static int sgx_validate_tcs(struct sgx_encl *encl, struct sgx_tcs *tcs) {int i; +if (!tcs) +return -EINVAL ; +if (tcs->flags &SGX_TCS_RESERVED_MASK )return -EINVAL ;
@@ -510,7 +513,10 @@ static int sgx_encl_queue_page(struct sgx_encl *encl, } backing_ptr =kmap (backing); -memcpy (backing_ptr, data,PAGE_SIZE ); +if (data) +memcpy (backing_ptr, data,PAGE_SIZE ); +else +memset (backing_ptr,0 ,PAGE_SIZE );kunmap (backing);if (page_type ==SGX_SECINFO_TCS ) encl_page->desc |=SGX_ENCL_PAGE_TCS ;
@@ -576,12 +582,15 @@ static int __sgx_encl_add_page(struct sgx_encl *encl, unsigned long addr,static int sgx_encl_add_page (struct sgx_encl *encl,unsigned long addr,unsigned long src,struct sgx_secinfo *secinfo, -unsigned int mrmask) +unsigned int mrmask,unsigned long flags) {struct page *data_page;void *data;int ret; +if (flags &SGX_ZERO_REGION ) +return __sgx_encl_add_page (encl, addr,NULL , secinfo, mrmask); + data_page =alloc_page (GFP_HIGHUSER );if (!data_page)return -ENOMEM ;
@@ -658,7 +667,7 @@ static long sgx_ioc_enclave_add_region(struct file *filep, void __user *arg)cond_resched (); ret =sgx_encl_add_page (encl, region.addr + i, region.src + i, - &secinfo, region.mrmask); + &secinfo, region.mrmask, region.flags);if (ret)break ; }
Signature · 1 line
2.21.0
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
Quoted 11 lines · Andy Lutomirski
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Quoted 2 lines
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Performance. A while back I wrote a patch set to remove the worker queue and discovered that it tanks enclave build time when the enclave is being hosted by a Golang application. Here's a snippet from a mail discussing the code.
The bad news is that I don't think we can remove the add page worker as applications with userspace schedulers, e.g. Go's M:N scheduler, can see a 10x or more throughput improvement when using the worker queue. I did a bit of digging for the Golang case to make sure I wasn't doing something horribly stupid/naive and found that it's a generic issue in Golang with blocking (or just long-running) system calls. Because Golang multiplexes Goroutines on top of OS threads, blocking syscalls introduce latency and context switching overhead, e.g. Go's scheduler will spin up a new OS thread to service other Goroutines after it realizes the syscall has blocked, and will later destroy one of the OS threads so that it doesn't build up too many unused.
IIRC, the scenario is spinning up several goroutines, each building an enclave. I played around with adding a flag to do a synchronous EADD but didn't see a meaningful change in performance for the simple case. Supporting both the worker queue and direct paths was complex enough that I decided it wasn't worth the trouble for initial upstreaming.
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
Quoted 37 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Quoted 2 lines
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Performance. A while back I wrote a patch set to remove the worker queue and discovered that it tanks enclave build time when the enclave is being hosted by a Golang application. Here's a snippet from a mail discussing the code.
The bad news is that I don't think we can remove the add page worker as applications with userspace schedulers, e.g. Go's M:N scheduler, can see a 10x or more throughput improvement when using the worker queue. I did a bit of digging for the Golang case to make sure I wasn't doing something horribly stupid/naive and found that it's a generic issue in Golang with blocking (or just long-running) system calls. Because Golang multiplexes Goroutines on top of OS threads, blocking syscalls introduce latency and context switching overhead, e.g. Go's scheduler will spin up a new OS thread to service other Goroutines after it realizes the syscall has blocked, and will later destroy one of the OS threads so that it doesn't build up too many unused.
Quoted 5 lines
IIRC, the scenario is spinning up several goroutines, each building an enclave. I played around with adding a flag to do a synchronous EADD but didn't see a meaningful change in performance for the simple case. Supporting both the worker queue and direct paths was complex enough that I decided it wasn't worth the trouble for initial upstreaming.
Sigh.
It seems silly to add a workaround for a language that has trouble calling somewhat-but-not-too-slow syscalls or ioctls.
How about fixing this in Go directly? Either convince the golang people to add a way to allocate a real thread for a particular region of code or have the Go SGX folks write a bit of C code to do a whole bunch of ioctls and have Go call *that*. Then the mess stays in Go where it belongs.
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
Quoted 10 lines · Andy Lutomirski
Sigh.
It seems silly to add a workaround for a language that has trouble calling somewhat-but-not-too-slow syscalls or ioctls.
How about fixing this in Go directly? Either convince the golang people to add a way to allocate a real thread for a particular region of code or have the Go SGX folks write a bit of C code to do a whole bunch of ioctls and have Go call *that*. Then the mess stays in Go where it belongs.
A worker thread would be only appropriate if the existing SGX code was already in the mainline because then it would break the user space.
Doing it either way right now does not break anything so there is no case for having it.
/Jarkko
RE: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
In reply to Andy LutomirskiQuoted 49 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Friday, June 07, 2019 12:32 PMQuoted 1 line
On Jun 6, 2019, at 10:32 AM, Sean Christopherson
Quoted 16 lines · <[email protected]> wrote:
Quoted 13 lines · Andy Lutomirski
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Quoted 2 lines
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Quoted 6 lines
Performance. A while back I wrote a patch set to remove the worker queue and discovered that it tanks enclave build time when the enclave is being hosted by a Golang application. Here's a snippet from a mail discussing the code.
The bad news is that I don't think we can remove the add page
worker
Quoted 9 lines
as applications with userspace schedulers, e.g. Go's M:N scheduler, can see a 10x or more throughput improvement when using the worker queue. I did a bit of digging for the Golang case to make sure I wasn't doing something horribly stupid/naive and found that it's a generic issue in Golang with blocking (or just long-running) system calls. Because Golang multiplexes Goroutines on top of OS threads, blocking syscalls introduce latency and context switching overhead, e.g. Go's scheduler will spin up a new OS thread to service other Goroutines after it realizes the syscall has blocked, and will
later
Quoted 8 lines
destroy one of the OS threads so that it doesn't build up too many unused.
IIRC, the scenario is spinning up several goroutines, each building an enclave. I played around with adding a flag to do a synchronous EADD but didn't see a meaningful change in performance for the simple case. Supporting both the worker queue and direct paths was complex enough that I decided it wasn't worth the trouble for initial upstreaming.
Sigh.
Quoted 8 lines
It seems silly to add a workaround for a language that has trouble calling somewhat-but-not-too-slow syscalls or ioctls.
How about fixing this in Go directly? Either convince the golang people to add a way to allocate a real thread for a particular region of code or have the Go SGX folks write a bit of C code to do a whole bunch of ioctls and have Go call *that*. Then the mess stays in Go where it belongs.
The add page worker is less about performance but more about concurrency restrictions in SGX ISA. EADD/EEXTEND are slow instructions and both take lock on the SECS. So if there's another EADD/EEXTEND in progress then it will #GP.
In practice, no sane applications will EADD in multiple threads because that would make MRENCLAVE unpredictable. But adversary may use that just to trigger #GP in kernel. Therefore, the SGX module would have to lock around EADD/EEXTEND anyway. And then we figured it would be better to have the add page worker so that no lock would be necessary, plus (small) improvement in performance.
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
Quoted 63 lines · Cedric
Quoted 49 lines
From: Andy Lutomirski [mailto:[email protected]]
Sent: Friday, June 07, 2019 12:32 PMQuoted 1 line
On Jun 6, 2019, at 10:32 AM, Sean Christopherson
Quoted 16 lines · <[email protected]> wrote:
Quoted 13 lines · Andy Lutomirski
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Quoted 2 lines
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Quoted 6 lines
Performance. A while back I wrote a patch set to remove the worker queue and discovered that it tanks enclave build time when the enclave is being hosted by a Golang application. Here's a snippet from a mail discussing the code.
The bad news is that I don't think we can remove the add page
worker
Quoted 9 lines
as applications with userspace schedulers, e.g. Go's M:N scheduler, can see a 10x or more throughput improvement when using the worker queue. I did a bit of digging for the Golang case to make sure I wasn't doing something horribly stupid/naive and found that it's a generic issue in Golang with blocking (or just long-running) system calls. Because Golang multiplexes Goroutines on top of OS threads, blocking syscalls introduce latency and context switching overhead, e.g. Go's scheduler will spin up a new OS thread to service other Goroutines after it realizes the syscall has blocked, and will
later
Quoted 8 lines
destroy one of the OS threads so that it doesn't build up too many unused.
IIRC, the scenario is spinning up several goroutines, each building an enclave. I played around with adding a flag to do a synchronous EADD but didn't see a meaningful change in performance for the simple case. Supporting both the worker queue and direct paths was complex enough that I decided it wasn't worth the trouble for initial upstreaming.
Sigh.
Quoted 8 lines
It seems silly to add a workaround for a language that has trouble calling somewhat-but-not-too-slow syscalls or ioctls.
How about fixing this in Go directly? Either convince the golang people to add a way to allocate a real thread for a particular region of code or have the Go SGX folks write a bit of C code to do a whole bunch of ioctls and have Go call *that*. Then the mess stays in Go where it belongs.
The add page worker is less about performance but more about concurrency restrictions in SGX ISA. EADD/EEXTEND are slow instructions and both take lock on the SECS. So if there's another EADD/EEXTEND in progress then it will #GP.
Quoted 6 lines
In practice, no sane applications will EADD in multiple threads because that would make MRENCLAVE unpredictable. But adversary may use that just to trigger #GP in kernel. Therefore, the SGX module would have to lock around EADD/EEXTEND anyway. And then we figured it would be better to have the add page worker so that no lock would be necessary, plus (small) improvement in performance.
That may have been true years ago, but it doesn't reflect the current reality. The ADD_PAGE ioctl *and* the worker function both take the enclave's lock. The worker function actually takes it twice, once to pull the request off the queue, and once to do EADD/EEXTEND. The lock is temporarily released by the worker function to avoid deadlock in case EPC page allocation triggers reclaim.
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
In reply to Andy LutomirskiQuoted 44 lines · Andy Lutomirski
Quoted 36 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Quoted 2 lines
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Performance. A while back I wrote a patch set to remove the worker queue and discovered that it tanks enclave build time when the enclave is being hosted by a Golang application. Here's a snippet from a mail discussing the code.
The bad news is that I don't think we can remove the add page worker as applications with userspace schedulers, e.g. Go's M:N scheduler, can see a 10x or more throughput improvement when using the worker queue. I did a bit of digging for the Golang case to make sure I wasn't doing something horribly stupid/naive and found that it's a generic issue in Golang with blocking (or just long-running) system calls. Because Golang multiplexes Goroutines on top of OS threads, blocking syscalls introduce latency and context switching overhead, e.g. Go's scheduler will spin up a new OS thread to service other Goroutines after it realizes the syscall has blocked, and will later destroy one of the OS threads so that it doesn't build up too many unused.
Quoted 5 lines
IIRC, the scenario is spinning up several goroutines, each building an enclave. I played around with adding a flag to do a synchronous EADD but didn't see a meaningful change in performance for the simple case. Supporting both the worker queue and direct paths was complex enough that I decided it wasn't worth the trouble for initial upstreaming.
Sigh.
Quoted 7 lines
It seems silly to add a workaround for a language that has trouble calling somewhat-but-not-too-slow syscalls or ioctls.
How about fixing this in Go directly? Either convince the golang people to add a way to allocate a real thread for a particular region of code or have the Go SGX folks write a bit of C code to do a whole bunch of ioctls and have Go call *that*. Then the mess stays in Go where it belongs.
Actually, I'm pretty sure changing the ioctl() from ADD_PAGE to ADD_REGION would eliminate the worst of the golang slowdown without requiring userspace to get super fancy. I'm in favor of eliminating the work queue, especially if the UAPI is changed to allow adding multiple pages in a single syscall.
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
Quoted 53 lines · Sean Christopherson
Quoted 44 lines · Andy Lutomirski
Quoted 36 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Quoted 2 lines
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Performance. A while back I wrote a patch set to remove the worker queue and discovered that it tanks enclave build time when the enclave is being hosted by a Golang application. Here's a snippet from a mail discussing the code.
The bad news is that I don't think we can remove the add page worker as applications with userspace schedulers, e.g. Go's M:N scheduler, can see a 10x or more throughput improvement when using the worker queue. I did a bit of digging for the Golang case to make sure I wasn't doing something horribly stupid/naive and found that it's a generic issue in Golang with blocking (or just long-running) system calls. Because Golang multiplexes Goroutines on top of OS threads, blocking syscalls introduce latency and context switching overhead, e.g. Go's scheduler will spin up a new OS thread to service other Goroutines after it realizes the syscall has blocked, and will later destroy one of the OS threads so that it doesn't build up too many unused.
Quoted 5 lines
IIRC, the scenario is spinning up several goroutines, each building an enclave. I played around with adding a flag to do a synchronous EADD but didn't see a meaningful change in performance for the simple case. Supporting both the worker queue and direct paths was complex enough that I decided it wasn't worth the trouble for initial upstreaming.
Sigh.
Quoted 7 lines
It seems silly to add a workaround for a language that has trouble calling somewhat-but-not-too-slow syscalls or ioctls.
How about fixing this in Go directly? Either convince the golang people to add a way to allocate a real thread for a particular region of code or have the Go SGX folks write a bit of C code to do a whole bunch of ioctls and have Go call *that*. Then the mess stays in Go where it belongs.
Quoted 6 lines
Actually, I'm pretty sure changing the
ioctl()fromADD_PAGEtoADD_REGIONwould eliminate the worst of the golang slowdown without requiring userspace to get super fancy. I'm in favor of eliminating the work queue, especially if the UAPI is changed to allow adding multiple pages in a single syscall.
I don't know if this is going to matter a whole lot, but have you considered the performance impact of needing to the EPC paging while doing the EADD ioctl and how this interacts with having a workqueue?
--
Jethro Beekman | Fortanix
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
Quoted 61 lines · Jethro Beekman
Quoted 53 lines · Sean Christopherson
Quoted 44 lines · Andy Lutomirski
Quoted 36 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Quoted 2 lines
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Performance. A while back I wrote a patch set to remove the worker queue and discovered that it tanks enclave build time when the enclave is being hosted by a Golang application. Here's a snippet from a mail discussing the code.
The bad news is that I don't think we can remove the add page worker as applications with userspace schedulers, e.g. Go's M:N scheduler, can see a 10x or more throughput improvement when using the worker queue. I did a bit of digging for the Golang case to make sure I wasn't doing something horribly stupid/naive and found that it's a generic issue in Golang with blocking (or just long-running) system calls. Because Golang multiplexes Goroutines on top of OS threads, blocking syscalls introduce latency and context switching overhead, e.g. Go's scheduler will spin up a new OS thread to service other Goroutines after it realizes the syscall has blocked, and will later destroy one of the OS threads so that it doesn't build up too many unused.
Quoted 5 lines
IIRC, the scenario is spinning up several goroutines, each building an enclave. I played around with adding a flag to do a synchronous EADD but didn't see a meaningful change in performance for the simple case. Supporting both the worker queue and direct paths was complex enough that I decided it wasn't worth the trouble for initial upstreaming.
Sigh.
Quoted 7 lines
It seems silly to add a workaround for a language that has trouble calling somewhat-but-not-too-slow syscalls or ioctls.
How about fixing this in Go directly? Either convince the golang people to add a way to allocate a real thread for a particular region of code or have the Go SGX folks write a bit of C code to do a whole bunch of ioctls and have Go call *that*. Then the mess stays in Go where it belongs.
Quoted 6 lines
Actually, I'm pretty sure changing the
ioctl()fromADD_PAGEtoADD_REGIONwould eliminate the worst of the golang slowdown without requiring userspace to get super fancy. I'm in favor of eliminating the work queue, especially if the UAPI is changed to allow adding multiple pages in a single syscall.
Quoted 3 lines
I don't know if this is going to matter a whole lot, but have you considered the performance impact of needing to the EPC paging while doing the EADD ioctl and how this interacts with having a workqueue?
Yep, other than the goroutine case, eliminating the workqueue doesn't substantially affect performance in either direction, regardless of the pressure on the EPC.
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
Quoted 63 lines · Sean Christopherson
Quoted 61 lines · Jethro Beekman
Quoted 59 lines · Sean Christopherson
Quoted 44 lines · Andy Lutomirski
Quoted 36 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Quoted 2 lines
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Performance. A while back I wrote a patch set to remove the worker queue and discovered that it tanks enclave build time when the enclave is being hosted by a Golang application. Here's a snippet from a mail discussing the code.
The bad news is that I don't think we can remove the add page worker as applications with userspace schedulers, e.g. Go's M:N scheduler, can see a 10x or more throughput improvement when using the worker queue. I did a bit of digging for the Golang case to make sure I wasn't doing something horribly stupid/naive and found that it's a generic issue in Golang with blocking (or just long-running) system calls. Because Golang multiplexes Goroutines on top of OS threads, blocking syscalls introduce latency and context switching overhead, e.g. Go's scheduler will spin up a new OS thread to service other Goroutines after it realizes the syscall has blocked, and will later destroy one of the OS threads so that it doesn't build up too many unused.
Quoted 5 lines
IIRC, the scenario is spinning up several goroutines, each building an enclave. I played around with adding a flag to do a synchronous EADD but didn't see a meaningful change in performance for the simple case. Supporting both the worker queue and direct paths was complex enough that I decided it wasn't worth the trouble for initial upstreaming.
Sigh.
Quoted 7 lines
It seems silly to add a workaround for a language that has trouble calling somewhat-but-not-too-slow syscalls or ioctls.
How about fixing this in Go directly? Either convince the golang people to add a way to allocate a real thread for a particular region of code or have the Go SGX folks write a bit of C code to do a whole bunch of ioctls and have Go call *that*. Then the mess stays in Go where it belongs.
Actually, I'm pretty sure changing the
ioctl()fromADD_PAGEtoADD_REGIONwould eliminate the worst of the golang slowdown without requiring userspace to get super fancy. I'm in favor of eliminating the work queue, especially if the UAPI is changed to allow adding multiple pages in a single syscall.
Quoted 7 lines
Quoted 3 lines
I don't know if this is going to matter a whole lot, but have you considered the performance impact of needing to the EPC paging while doing the EADD ioctl and how this interacts with having a workqueue?
Yep, other than the goroutine case, eliminating the workqueue doesn't substantially affect performance in either direction, regardless of the pressure on the EPC.
It should get rid of some extra copies and allocations, no?
Re: [PATCH 5/7] x86/sgx: Add flag to zero added region instead of copying from source
Quoted 69 lines · Andy Lutomirski
Quoted 67 lines · Sean Christopherson
Quoted 64 lines · Jethro Beekman
Quoted 53 lines · Sean Christopherson
Quoted 44 lines · Andy Lutomirski
Quoted 36 lines · Sean Christopherson
Quoted 13 lines · Andy Lutomirski
Quoted 8 lines · Sean Christopherson
For some enclaves, e.g. an enclave with a small code footprint and a large working set, the vast majority of pages added to the enclave are zero pages. Introduce a flag to denote such zero pages. The major benefit of the flag will be realized in a future patch to use Linux's actual zero page as the source, as opposed to explicitly zeroing the enclave's backing memory.
Quoted 2 lines
I feel like I probably asked this at some point, but why is there a workqueue here at all?
Performance. A while back I wrote a patch set to remove the worker queue and discovered that it tanks enclave build time when the enclave is being hosted by a Golang application. Here's a snippet from a mail discussing the code.
The bad news is that I don't think we can remove the add page worker as applications with userspace schedulers, e.g. Go's M:N scheduler, can see a 10x or more throughput improvement when using the worker queue. I did a bit of digging for the Golang case to make sure I wasn't doing something horribly stupid/naive and found that it's a generic issue in Golang with blocking (or just long-running) system calls. Because Golang multiplexes Goroutines on top of OS threads, blocking syscalls introduce latency and context switching overhead, e.g. Go's scheduler will spin up a new OS thread to service other Goroutines after it realizes the syscall has blocked, and will later destroy one of the OS threads so that it doesn't build up too many unused.
Quoted 5 lines
IIRC, the scenario is spinning up several goroutines, each building an enclave. I played around with adding a flag to do a synchronous EADD but didn't see a meaningful change in performance for the simple case. Supporting both the worker queue and direct paths was complex enough that I decided it wasn't worth the trouble for initial upstreaming.
Sigh.
Quoted 7 lines
It seems silly to add a workaround for a language that has trouble calling somewhat-but-not-too-slow syscalls or ioctls.
How about fixing this in Go directly? Either convince the golang people to add a way to allocate a real thread for a particular region of code or have the Go SGX folks write a bit of C code to do a whole bunch of ioctls and have Go call *that*. Then the mess stays in Go where it belongs.
Quoted 6 lines
Actually, I'm pretty sure changing the
ioctl()fromADD_PAGEtoADD_REGIONwould eliminate the worst of the golang slowdown without requiring userspace to get super fancy. I'm in favor of eliminating the work queue, especially if the UAPI is changed to allow adding multiple pages in a single syscall.I don't know if this is going to matter a whole lot, but have you considered the performance impact of needing to the EPC paging while doing the EADD ioctl and how this interacts with having a workqueue?
Quoted 5 lines
Quoted 3 lines
Yep, other than the goroutine case, eliminating the workqueue doesn't substantially affect performance in either direction, regardless of the pressure on the EPC.
It should get rid of some extra copies and allocations, no?
My experiments were fairly dumb, e.g. removed the workqueue without doing extra enhancements like avoiding copy_from_user. IIRC, the type of allocation changed, but the number of copies/allocations stayed the same.
PATCH 6/7 x86/sgx: Use the actual zero page as the source when adding zero pages
In reply to Sean ChristophersonUsing the zero page avoids dirtying the backing page, inserting TLB entries, the cost of memset, etc... For some enclaves, e.g. an enclave with a small code footprint and a large working set, this results in a 20%+ reduction in enclave build time.
--- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 50 ++++++++++++++++---------- 1 file changed, 32 insertions(+), 18 deletions(-)
arch/x86/kernel/cpu/sgx/driver/ioctl.c +32 -18
@@ -19,6 +19,7 @@ struct sgx_add_page_req {struct sgx_secinfo secinfo;unsigned long mrmask;struct list_head list; +bool zero_page; };static int sgx_encl_grow (struct sgx_encl *encl)
@@ -76,6 +77,7 @@ static bool sgx_process_add_page_req(struct sgx_add_page_req *req,struct sgx_pageinfo pginfo;struct page *backing;unsigned long addr; +void *contents;int ret;int i;
@@ -84,9 +86,15 @@ static bool sgx_process_add_page_req(struct sgx_add_page_req *req, addr =SGX_ENCL_PAGE_ADDR (encl_page); - backing =sgx_encl_get_backing_page (encl, page_index); -if (IS_ERR (backing)) -return false; +if (!req->zero_page) { + backing =sgx_encl_get_backing_page (encl, page_index); +if (IS_ERR (backing)) +return false; + contents =kmap_atomic (backing); + }else { + backing =NULL ; + contents =__va (page_to_pfn (ZERO_PAGE (0 )) <<PAGE_SHIFT ); + }/* * The SECINFO field must be 64-byte aligned, copy it to a local
@@ -99,11 +107,13 @@ static bool sgx_process_add_page_req(struct sgx_add_page_req *req, pginfo.secs = (unsigned long )sgx_epc_addr (encl->secs.epc_page); pginfo.addr = addr; pginfo.metadata = (unsigned long )&secinfo; - pginfo.contents = (unsigned long )kmap_atomic (backing); + pginfo.contents = (unsigned long )contents; ret =__eadd (&pginfo,sgx_epc_addr (epc_page)); -kunmap_atomic ((void *)(unsigned long )pginfo.contents); -put_page (backing); +if (backing) { +kunmap_atomic (contents); +put_page (backing); + }if (ret) {if (encls_failed (ret))
@@ -506,18 +516,20 @@ static int sgx_encl_queue_page(struct sgx_encl *encl,if (!req)return -ENOMEM ; - backing =sgx_encl_get_backing_page (encl, page_index); -if (IS_ERR (backing)) { -kfree (req); -return PTR_ERR (backing); - } +if (data) { + backing =sgx_encl_get_backing_page (encl, page_index); +if (IS_ERR (backing)) { +kfree (req); +return PTR_ERR (backing); + } - backing_ptr =kmap (backing); -if (data) + backing_ptr =kmap (backing);memcpy (backing_ptr, data,PAGE_SIZE ); -else -memset (backing_ptr,0 ,PAGE_SIZE ); -kunmap (backing); +kunmap (backing); + }else { + backing =NULL ; + req->zero_page = true; + }if (page_type ==SGX_SECINFO_TCS ) encl_page->desc |=SGX_ENCL_PAGE_TCS ;memcpy (&req->secinfo, secinfo,sizeof (*secinfo));
@@ -529,8 +541,10 @@ static int sgx_encl_queue_page(struct sgx_encl *encl,list_add_tail (&req->list, &encl->add_page_reqs);if (empty)queue_work (sgx_encl_wq, &encl->work); -set_page_dirty (backing); -put_page (backing); +if (backing) { +set_page_dirty (backing); +put_page (backing); + }return 0 ; }
Signature · 1 line
2.21.0
PATCH 7/7 x86/sgx: Add a reserved field to sgx_enclave_add_region to drop 'packed'
In reply to Sean ChristophersonA reserved field could prove useful in the future, and packed structs can result in inefficiencies due to its impact on alignment.
--- arch/x86/include/uapi/asm/sgx.h | 4 +++- arch/x86/kernel/cpu/sgx/driver/ioctl.c | 3 +++ 2 files changed, 6 insertions(+), 1 deletion(-)
arch/x86/include/uapi/asm/sgx.h +3 -1
@@ -43,6 +43,7 @@ struct sgx_enclave_create {* @secinfo: address of the SECINFO data (common to the entire region) * @flags: miscellaneous flags * @mrmask: bitmask of 256 byte chunks to measure (applied per 4k page) +* @reserved: reserved for future use, must be zero */ struct sgx_enclave_add_region {__u64 addr;
@@ -51,7 +52,8 @@ struct sgx_enclave_add_region {__u64 secinfo;__u32 flags;__u16 mrmask; -} __attribute__((__packed__)); +__u16 reserved; +};/** * struct sgx_enclave_init - parameter structure for the
arch/x86/kernel/cpu/sgx/driver/ioctl.c +3 -0
@@ -663,6 +663,9 @@ static long sgx_ioc_enclave_add_region(struct file *filep, void __user *arg)if (!IS_ALIGNED (region.addr,4096 ) || !IS_ALIGNED (region.size,4096 ))return -EINVAL ; +if (region.reserved) +return -EINVAL ; +if (!region.size)return 0 ;
Signature · 1 line
2.21.0
Re: [PATCH 7/7] x86/sgx: Add a reserved field to sgx_enclave_add_region to drop 'packed'
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_add_region { · line 56
Quoted 9 lines · Sean Christopherson
struct
sgx_enclave_add_region{arch/x86/include/uapi/asm/sgx.h +2
-1__u64 addr;@@ -51,7 +52,8 @@ struct sgx_enclave_add_region {__u64 secinfo;__u32 flags;__u16 mrmask; -} __attribute__((__packed__)); +__u16 reserved; +};
Without __packed__ do we have strong enough guarantees from the compiler for a stable ABI?
Re: [PATCH 7/7] x86/sgx: Add a reserved field to sgx_enclave_add_region to drop 'packed'
Quoted 14 lines · Dave Hansen
arch/x86/include/uapi/asm/sgx.h struct sgx_enclave_add_region { · line 56
Quoted 9 lines · Sean Christopherson
struct sgx_enclave_add_region { __u64 addr;arch/x86/include/uapi/asm/sgx.h +2
-1@@ -51,7 +52,8 @@ struct sgx_enclave_add_region {__u64 secinfo;__u32 flags;__u16 mrmask; -} __attribute__((__packed__)); +__u16 reserved; +};Without
__packed__do we have strong enough guarantees from the compiler for a stable ABI?
We rely on this kind of thing for uapi on a regular basis, so I sure hope so.
Re: [PATCH 7/7] x86/sgx: Add a reserved field to sgx_enclave_add_region to drop 'packed'
In reply to Sean ChristophersonQuoted 4 lines · Sean Christopherson
A reserved field could prove useful in the future, and packed structs can result in inefficiencies due to its impact on alignment.
Signed-off-by Sean Christopherson <[email protected]>
Rather should extend mrmask just to 32 bits.
/Jarkko
Re: [PATCH 7/7] x86/sgx: Add a reserved field to sgx_enclave_add_region to drop 'packed'
Quoted 7 lines · Jarkko Sakkinen
Quoted 4 lines · Sean Christopherson
A reserved field could prove useful in the future, and packed structs can result in inefficiencies due to its impact on alignment.
Signed-off-by Sean Christopherson <[email protected]>Rather should extend mrmask just to 32 bits.
I considered that option, but extending mrmask would probably confusing to userspace, e.g. we'd have to document that bits 31:15 are ignored.
Re: [PATCH 7/7] x86/sgx: Add a reserved field to sgx_enclave_add_region to drop 'packed'
Quoted 11 lines · Sean Christopherson
Quoted 7 lines · Jarkko Sakkinen
Quoted 4 lines · Sean Christopherson
A reserved field could prove useful in the future, and packed structs can result in inefficiencies due to its impact on alignment.
Signed-off-by Sean Christopherson <[email protected]>Rather should extend mrmask just to 32 bits.
I considered that option, but extending mrmask would probably confusing to userspace, e.g. we'd have to document that bits 31:15 are ignored.
I meant u64.
Not a big deal. With the reserved field we would have to document the same in different terms.
/Jarkko
Re: [PATCH 7/7] x86/sgx: Add a reserved field to sgx_enclave_add_region to drop 'packed'
In reply to Sean ChristophersonQuoted 5 lines · Sean Christopherson
Quoted 4 lines · Jarkko Sakkinen
Quoted 3 lines · Sean Christopherson
A reserved field could prove useful in the future, and packed structs can result in inefficiencies due to its impact on alignment.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Signed-off-by Sean Christopherson <[email protected]>Rather should extend mrmask just to 32 bits.
I considered that option, but extending mrmask would probably confusing to userspace, e.g. we'd have to document that bits 31:15 are ignored.
Yeah this doesn't make a whole lot of sense to me.
--
Jethro Beekman | Fortanix
Quoted 11 lines · Sean Christopherson
This series is intended to be included in v21 of Jarkko's SGX series and applies on Jarkko's current master:
dfc89a83b5bc("docs: x86/sgx: Document the enclave API")The primary goal of the series is to tweak the ioctl for adding pages to an enclave so that it is somewhat extensible, e.g. add a flags field that can be reused for access control integration and SGX2/EAUG, and add a size field so that multiple pages can be added in a single call (batching EADD has been mentioned at various times in the past).
Quoted 4 lines
The secondary goal is to improve the performance of building enclaves. Handling multiple pages in a single call helps somewhat, but the real win (for some enclaves) is using the kernel's zero page as the source for EADD when possible.
For v21 I will add 1, 2 and 6 from these. Thanks.
I'll change the size or mrmask field to 32 bits.
/Jarkko
Quoted 12 lines · Jarkko Sakkinen
Quoted 11 lines · Sean Christopherson
This series is intended to be included in v21 of Jarkko's SGX series and applies on Jarkko's current master:
dfc89a83b5bc("docs: x86/sgx: Document the enclave API")The primary goal of the series is to tweak the ioctl for adding pages to an enclave so that it is somewhat extensible, e.g. add a flags field that can be reused for access control integration and SGX2/EAUG, and add a size field so that multiple pages can be added in a single call (batching EADD has been mentioned at various times in the past).
Quoted 6 lines
Quoted 4 lines
The secondary goal is to improve the performance of building enclaves. Handling multiple pages in a single call helps somewhat, but the real win (for some enclaves) is using the kernel's zero page as the source for EADD when possible.
For v21 I will add 1, 2 and 6 from these. Thanks.
I meant to say 1, 2 and 3 (typo).
/Jarkko