x86/sgx: Rename fallback labels in sgx_init()
Subsystems (3)
Jarkko Sakkinen
maintainerPATCH v9 1/2 x86/sgx: Rename fallback labels in sgx_init()
It's hard to add new content to this function because it is time consuming to match fallback and its cause. Rename labels in a way that the name of error label refers to the site where failure happened. This way it is easier to keep on track what is going on.
---
v5:
* A new patch.
--- arch/x86/kernel/cpu/sgx/main.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/sgx/main.c +6 -6
@@ -803,12 +803,12 @@ static int __init sgx_init(void)if (!sgx_page_reclaimer_init ()) { ret = -ENOMEM ; - goto err_page_cache; + goto err_reclaimer; } ret =misc_register (&sgx_dev_provision);if (ret) - goto err_kthread; + goto err_provision;/* * Always try to initialize the native *and* KVM drivers.
@@ -821,17 +821,17 @@ static int __init sgx_init(void) ret =sgx_drv_init ();if (sgx_vepc_init () && ret) - goto err_provision; + goto err_driver;return 0 ; -err_provision: +err_driver:misc_deregister (&sgx_dev_provision); -err_kthread: +err_provision:kthread_stop (ksgxd_tsk); -err_page_cache: +err_reclaimer:for (i =0 ; i < sgx_nr_epc_sections; i++) {vfree (sgx_epc_sections[i].pages);memunmap (sgx_epc_sections[i].virt_addr);
Signature · 1 line
2.32.0
Jarkko Sakkinen
maintainerPATCH v9 2/2 x86/sgx: Add an attribute for the amount of SGX memory in a NUMA node
The amount of SGX memory on the system is determined by the BIOS and it varies wildly between systems. It can be from dozens of MB's on desktops or VM's, up to many GB's on servers. Just like for regular memory, it is sometimes useful to know the amount of usable SGX memory in the system.
Add an attribute for the amount of SGX memory in bytes to each NUMA node. The path is /sys/devices/system/node/node[0-9]*/sgx/size. Calculate these values by summing up EPC section sizes for each node during the driver initalization.
---
v9:
* Fix racy initialization of sysfs attributes:
https://lore.kernel.org/linux-sgx/[email protected]/
v8:
* Fix a bug in sgx_numa_init(): node->dev should be only set after sysfe_create_group(). Otherwise, sysfs_remove_group() will issue a warning in sgx_numa_exit(), when sgx_create_group() is unsuccessful, because the group does not exist.
v7:
* Shorten memory_size to size. The prefix makes the name only longer but does not clarify things more than "size" would.
* Use device_attribute instead of kobj_attribute.
* Use named attribute group instead of creating raw kobject just for the "sgx" subdirectory.
v6:
* Initialize node->size to zero in sgx_setup_epc_section(), when the node is first accessed.
v5
* A new patch based on the discussion on
https://lore.kernel.org/linux-sgx/[email protected]/T/#t
--- Documentation/ABI/stable/sysfs-devices-node | 7 +++++ arch/x86/include/asm/sgx.h | 2 ++ arch/x86/kernel/cpu/sgx/main.c | 31 +++++++++++++++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 2 ++ drivers/base/node.c | 16 ++++++++++- 5 files changed, 57 insertions(+), 1 deletion(-)
Documentation/ABI/stable/sysfs-devices-node +7 -0
@@ -176,3 +176,10 @@ Contact: Keith Busch <[email protected]> Description: The cache write policy: 0 for write-back, 1 for write-through, other or unknown. + +What: /sys/devices/system/node/nodeX/sgx/size +Date: October 2021 +Contact: Jarkko Sakkinen <[email protected]> +Description: + Total available physical SGX memory, also known as Enclave Page + Cache (EPC), in bytes.
arch/x86/include/asm/sgx.h +2 -0
@@ -365,6 +365,8 @@ struct sgx_sigstruct {* comment! */ +extern const struct attribute_group sgx_node_group; +#ifdef CONFIG_X86_SGX_KVM int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs,int *trapnr);
arch/x86/kernel/cpu/sgx/main.c +31 -0
@@ -714,9 +714,12 @@ static bool __init sgx_page_cache_init(void)spin_lock_init (&sgx_numa_nodes[nid].lock);INIT_LIST_HEAD (&sgx_numa_nodes[nid].free_page_list);node_set (nid, sgx_numa_mask); + sgx_numa_nodes[nid].size =0 ; } sgx_epc_sections[i].node = &sgx_numa_nodes[nid]; + sgx_numa_nodes[nid].dev = &node_devices[nid]->dev; + sgx_numa_nodes[nid].size += size; sgx_nr_epc_sections++; }
@@ -790,6 +793,34 @@ int sgx_set_attribute(unsigned long *allowed_attributes, }EXPORT_SYMBOL_GPL (sgx_set_attribute); +#ifdef CONFIG_NUMA +static ssize_t size_show (struct device *dev,struct device_attribute *attr,char *buf) +{ +unsigned long size =0 ; +int nid; + +for (nid =0 ; nid <num_possible_nodes (); nid++) { +if (dev == sgx_numa_nodes[nid].dev) { + size = sgx_numa_nodes[nid].size; +break ; + } + } + +return sysfs_emit (buf,"%lu\n" , size); +} +DEVICE_ATTR_RO (size); + +static struct attribute *sgx_node_attrs[] = { + &dev_attr_size.attr, +NULL , +}; + +const struct attribute_group sgx_node_group = { + .name ="sgx" , + .attrs = sgx_node_attrs, +}; +#endif /* CONFIG_NUMA */ +static int __init sgx_init (void ) {int ret;
arch/x86/kernel/cpu/sgx/sgx.h +2 -0
@@ -39,6 +39,8 @@ struct sgx_epc_page {*/ struct sgx_numa_node {struct list_head free_page_list; +struct device *dev; +unsigned long size;spinlock_t lock; };
drivers/base/node.c +15 -1
@@ -20,6 +20,9 @@#include <linux/pm_runtime.h> #include <linux/swap.h> #include <linux/slab.h> +#ifdef CONFIG_X86_SGX +#include <asm/sgx.h> +#endif static struct bus_type node_subsys = { .name ="node" ,
@@ -565,7 +568,18 @@ static struct attribute *node_dev_attrs[] = { &dev_attr_vmstat.attr,NULL }; -ATTRIBUTE_GROUPS (node_dev); + +static const struct attribute_group node_dev_group = { + .attrs = node_dev_attrs, +}; + +static const struct attribute_group *node_dev_groups[] = { + &node_dev_group, +#ifdef CONFIG_X86_SGX + &sgx_node_group, +#endif /* CONFIG_X86_SGX */ +NULL , +};#ifdef CONFIG_HUGETLBFS /*
Signature · 1 line
2.32.0
Greg Kroah-Hartman
maintainerRe: [PATCH v9 2/2] x86/sgx: Add an attribute for the amount of SGX memory in a NUMA node
Quoted 100 lines · Jarkko Sakkinen
The amount of SGX memory on the system is determined by the BIOS and it varies wildly between systems. It can be from dozens of MB's on desktops or VM's, up to many GB's on servers. Just like for regular memory, it is sometimes useful to know the amount of usable SGX memory in the system.
Add an attribute for the amount of SGX memory in bytes to each NUMA node. The path is /sys/devices/system/node/node[0-9]*/sgx/size. Calculate these values by summing up EPC section sizes for each node during the driver initalization.
Cc Greg Kroah-Hartman <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>---
v9:
* Fix racy initialization of sysfs attributes:
https://lore.kernel.org/linux-sgx/[email protected]/v8:
* Fix a bug insgx_numa_init():node->devshould be only set aftersysfe_create_group(). Otherwise,sysfs_remove_group()will issue a warning insgx_numa_exit(), whensgx_create_group()is unsuccessful, because the group does not exist.v7:
* Shortenmemory_sizeto size. The prefix makes the name only longer but does not clarify things more than "size" would.
* Usedevice_attributeinstead ofkobj_attribute.
* Use named attribute group instead of creating raw kobject just for the "sgx" subdirectory.v6:
* Initializenode->sizeto zero insgx_setup_epc_section(), when the node is first accessed.v5
* A new patch based on the discussion on
https://lore.kernel.org/linux-sgx/[email protected]/T/#t--- Documentation/ABI/stable/sysfs-devices-node | 7 +++++ arch/x86/include/asm/sgx.h | 2 ++ arch/x86/kernel/cpu/sgx/main.c | 31 +++++++++++++++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 2 ++ drivers/base/node.c | 16 ++++++++++- 5 files changed, 57 insertions(+), 1 deletion(-)Documentation/ABI/stable/sysfs-devices-node +7
-0@@ -176,3 +176,10 @@ Contact: Keith Busch <[email protected]> Description: The cache write policy: 0 for write-back, 1 for write-through, other or unknown. + +What: /sys/devices/system/node/nodeX/sgx/size +Date: October 2021 +Contact: Jarkko Sakkinen <[email protected]> +Description: + Total available physical SGX memory, also known as Enclave Page + Cache (EPC), in bytes.arch/x86/include/asm/sgx.h +2
-0@@ -365,6 +365,8 @@ struct sgx_sigstruct {* comment! */ +extern const struct attribute_group sgx_node_group; +#ifdef CONFIG_X86_SGX_KVM int sgx_virt_ecreate (struct sgx_pageinfo *pageinfo,void __user *secs,int *trapnr);arch/x86/kernel/cpu/sgx/main.c +9
-0@@ -714,9 +714,12 @@ static bool __init sgx_page_cache_init(void)spin_lock_init (&sgx_numa_nodes[nid].lock);INIT_LIST_HEAD (&sgx_numa_nodes[nid].free_page_list);node_set (nid, sgx_numa_mask); + sgx_numa_nodes[nid].size =0 ; } sgx_epc_sections[i].node = &sgx_numa_nodes[nid]; + sgx_numa_nodes[nid].dev = &node_devices[nid]->dev; + sgx_numa_nodes[nid].size += size; sgx_nr_epc_sections++; }@@ -790,6 +793,34 @@ int sgx_set_attribute(unsigned long *allowed_attributes, }EXPORT_SYMBOL_GPL (sgx_set_attribute); +#ifdef CONFIG_NUMA +static ssize_t size_show (struct device *dev,struct device_attribute *attr,char *buf) +{ +unsigned long size =0 ; +int nid; +
arch/x86/kernel/cpu/sgx/main.c int sgx_set_attribute(unsigned long *allowed_attributes, · line 811
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +10
-0+for (nid =0 ; nid <num_possible_nodes (); nid++) { +if (dev == sgx_numa_nodes[nid].dev) { + size = sgx_numa_nodes[nid].size; +break ; + } + } + +return sysfs_emit (buf,"%lu\n" , size); +} +DEVICE_ATTR_RO (size);
static?
Jarkko Sakkinen
maintainerRe: [PATCH v9 2/2] x86/sgx: Add an attribute for the amount of SGX memory in a NUMA node
Quoted 105 lines · Greg Kroah-Hartman
Quoted 104 lines · Jarkko Sakkinen
The amount of SGX memory on the system is determined by the BIOS and it varies wildly between systems. It can be from dozens of MB's on desktops or VM's, up to many GB's on servers. Just like for regular memory, it is sometimes useful to know the amount of usable SGX memory in the system.
Add an attribute for the amount of SGX memory in bytes to each NUMA node. The path is /sys/devices/system/node/node[0-9]*/sgx/size. Calculate these values by summing up EPC section sizes for each node during the driver initalization.
Cc Greg Kroah-Hartman <[email protected]>Signed-off-by Jarkko Sakkinen <[email protected]>---
v9:
* Fix racy initialization of sysfs attributes:
https://lore.kernel.org/linux-sgx/[email protected]/v8:
* Fix a bug insgx_numa_init():node->devshould be only set after sysfe_create_group(). Otherwise,sysfs_remove_group()will issue a warning insgx_numa_exit(), whensgx_create_group()is unsuccessful, because the group does not exist.v7:
* Shortenmemory_sizeto size. The prefix makes the name only longer but does not clarify things more than "size" would.
* Usedevice_attributeinstead ofkobj_attribute.
* Use named attribute group instead of creating raw kobject just for the "sgx" subdirectory.v6:
* Initializenode->sizeto zero insgx_setup_epc_section(), when the node is first accessed.v5
* A new patch based on the discussion on
https://lore.kernel.org/linux-sgx/[email protected]/T/#t
---
Documentation/ABI/stable/sysfs-devices-node | 7 +++++ arch/x86/include/asm/sgx.h | 2 ++ arch/x86/kernel/cpu/sgx/main.c | 31 +++++++++++++++++++++ arch/x86/kernel/cpu/sgx/sgx.h | 2 ++ drivers/base/node.c | 16 ++++++++++-5 files changed, 57 insertions(+), 1 deletion(-)Documentation/ABI/stable/sysfs-devices-node +0
-0@@ -176,3 +176,10 @@ Contact: Keith Busch <[email protected]>Description:
The cache write policy: 0 for write-back, 1 for write-through, other or unknown.Documentation/ABI/stable/sysfs-devices-node +7
-0+ +What: /sys/devices/system/node/nodeX/sgx/size +Date: October 2021 +Contact: Jarkko Sakkinen <[email protected]> +Description: + Total available physical SGX memory, also known as Enclave Page + Cache (EPC), in bytes.arch/x86/include/asm/sgx.h +0
-0@@ -365,6 +365,8 @@ struct sgx_sigstruct {* comment! */ arch/x86/include/asm/sgx.h +2
-0+extern const struct attribute_group sgx_node_group; +#ifdef
CONFIG_X86_SGX_KVM
intsgx_virt_ecreate(struct sgx_pageinfo *pageinfo, void __user *secs, int *trapnr);arch/x86/kernel/cpu/sgx/main.c +1
-0@@ -714,9 +714,12 @@ static bool __init sgx_page_cache_init(void) spin_lock_init(&sgx_numa_nodes[nid].lock); INIT_LIST_HEAD(&sgx_numa_nodes[nid].free_page_list); node_set(nid, sgx_numa_mask); + sgx_numa_nodes[nid].size = 0; }
sgx_epc_sections[i].node = &sgx_numa_nodes[nid]; + sgx_numa_nodes[nid].dev = &node_devices[nid]->dev; + sgx_numa_nodes[nid].size += size;
sgx_nr_epc_sections++;
}arch/x86/kernel/cpu/sgx/main.c +0
-0@@ -790,6 +793,34 @@ int sgx_set_attribute(unsigned long *allowed_attributes, } EXPORT_SYMBOL_GPL(sgx_set_attribute);arch/x86/kernel/cpu/sgx/main.c +10
-0+#ifdef CONFIG_NUMA +static ssize_t size_show (struct device *dev,struct device_attribute *attr,char *buf) +{ + unsigned long size = 0; + int nid; + + for (nid = 0; nid < num_possible_nodes(); nid++) { + if (dev == sgx_numa_nodes[nid].dev) { + size = sgx_numa_nodes[nid].size; + break;
Quoted 8 lines
arch/x86/kernel/cpu/sgx/main.c int sgx_set_attribute(unsigned long *allowed_attributes, · line 811
Quoted 6 lines
+ }
+ }arch/x86/kernel/cpu/sgx/main.c +4
-0+ + return sysfs_emit(buf, "%lu\n", size); +} +DEVICE_ATTR_RO (size);static?
The named ("sgx") attribute group is exported:
extern const struct attribute_group sgx_node_group;
and added as an additional group of the node device:
static const struct attribute_group node_dev_group = {
.attrs = node_dev_attrs,
};
static const struct attribute_group *node_dev_groups[] = {
&node_dev_group,
#ifdef CONFIG_X86_SGX
&sgx_node_group,
#endif /* CONFIG_X86_SGX */
NULL,
};
An alternative would to define a helper, e.g.
ssize_t sgx_get_node_size(struct device *dev);
And use this in drivers/base/node.c to implement the attribute. I'm fine either way...
/Jarkko
Greg Kroah-Hartman
maintainerRe: [PATCH v9 2/2] x86/sgx: Add an attribute for the amount of SGX memory in a NUMA node
Quoted 18 lines · Jarkko Sakkinen
Quoted 9 lines
Quoted 9 lines
arch/x86/kernel/cpu/sgx/main.c +9
-0+static ssize_t size_show (struct device *dev,struct device_attribute *attr,char *buf) +{ + unsigned long size = 0; + int nid; + + for (nid = 0; nid < num_possible_nodes(); nid++) { + if (dev == sgx_numa_nodes[nid].dev) { + size = sgx_numa_nodes[nid].size; + break;Quoted 8 lines
arch/x86/kernel/cpu/sgx/main.c int sgx_set_attribute(unsigned long *allowed_attributes, · line 811
Quoted 6 lines
+ }
+ }arch/x86/kernel/cpu/sgx/main.c +4
-0+ + return sysfs_emit(buf, "%lu\n", size); +} +DEVICE_ATTR_RO (size);static?
Quoted 3 lines
The named ("sgx") attribute group is exported:
extern const struct
attribute_groupsgx_node_group;
That's fine, I am objecting to the fact that you now have added a global symbol called device_attr_size. Please make that static as that is a VERY generic name.
thanks,
greg k-h
Jarkko Sakkinen
maintainerRe: [PATCH v9 2/2] x86/sgx: Add an attribute for the amount of SGX memory in a NUMA node
Quoted 19 lines · Greg Kroah-Hartman
Quoted 18 lines · Jarkko Sakkinen
Quoted 17 lines
Quoted 5 lines
arch/x86/kernel/cpu/sgx/main.c +5
-0+static ssize_t size_show (struct device *dev,struct device_attribute *attr,char *buf) +{ + unsigned long size = 0; + int nid; +arch/x86/kernel/cpu/sgx/main.c int sgx_set_attribute(unsigned long *allowed_attributes, · line 811
Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +10
-0+ for (nid = 0; nid < num_possible_nodes(); nid++) { + if (dev == sgx_numa_nodes[nid].dev) { + size = sgx_numa_nodes[nid].size; + break; + } + } + + return sysfs_emit(buf, "%lu\n", size); +} +DEVICE_ATTR_RO (size);static?
Quoted 7 lines
Quoted 3 lines
The named ("sgx") attribute group is exported:
extern const struct
attribute_groupsgx_node_group;That's fine, I am objecting to the fact that you now have added a global symbol called device_attr_size. Please make that static as that is a VERY generic name.
Right, got it! Thank you.
/Jarkko
Re: [PATCH v9 2/2] x86/sgx: Add an attribute for the amount of SGX memory in a NUMA node
In reply to Jarkko SakkinenDocumentation/ABI/stable/sysfs-devices-node Contact: Keith Busch <[email protected]> · line 182
Quoted 3 lines · Jarkko Sakkinen
Documentation/ABI/stable/sysfs-devices-node +3
-0+What: /sys/devices/system/node/nodeX/sgx/size +Date: October 2021 +Contact: Jarkko Sakkinen <[email protected]>
I don't think we should do something *entirely* SGX-specific here. The only question to me is whether any non-SGX users want something like this and who they are.
Here are some ideas I like more than an "sgx/" directory:
/sys/devices/system/node/nodeX/arch/sgx_size /sys/devices/system/node/nodeX/x86/sgx_size /sys/devices/system/node/nodeX/coco/sgx_size
There's somebody else *today* who is trying to do something in the same general area: per-node platform-specific memory encryption capabilities:
Quoted 1 line
Also, could we please think through how this will look if we add more attributes? I can imagine wanting both:
* total SGX memory available
* total SGX memory present
But those would be quite hard to differentiate if we have just an "sgx_size".
Wouldn't it be much nicer to name them things like:
sgx_present_bytes
sgx_available_bytes
?
In other words, can we please try to think just a bit into the future on this one? What other SGX things will we want to export like this? Outside of SGX, who else wants stuff _like_ this?
Jarkko Sakkinen
maintainerRe: [PATCH v9 2/2] x86/sgx: Add an attribute for the amount of SGX memory in a NUMA node
Quoted 9 lines · Dave Hansen
Documentation/ABI/stable/sysfs-devices-node Contact: Keith Busch <[email protected]> · line 182
Quoted 3 lines · Jarkko Sakkinen
Documentation/ABI/stable/sysfs-devices-node +3
-0+What: /sys/devices/system/node/nodeX/sgx/size +Date: October 2021 +Contact: Jarkko Sakkinen <[email protected]>I don't think we should do something *entirely* SGX-specific here. The only question to me is whether any non-SGX users want something like this and who they are.
Quoted 5 lines
Here are some ideas I like more than an "sgx/" directory:
/sys/devices/system/node/nodeX/arch/sgx_size
/sys/devices/system/node/nodeX/x86/sgx_size/sys/devices/system/node/nodeX/coco/sgx_size
Sure, I can rename the attribute group as "x86".
Quoted 4 lines
There's somebody else *today* who is trying to do something in the same general area: per-node platform-specific memory encryption capabilities:
Quoted 1 line
Martin's is adding a new attribute as part of pre-existing attribute group for node device, where as my patch is adding a completely new named attribute group.
Quoted 11 lines
Also, could we please think through how this will look if we add more attributes? I can imagine wanting both:
* total SGX memory available
* total SGX memory presentBut those would be quite hard to differentiate if we have just an "
sgx_size".Wouldn't it be much nicer to name them things like:
Quoted 8 lines
sgx_present_bytes
sgx_available_bytes?
In other words, can we please try to think just a bit into the future on this one? What other SGX things will we want to export like this? Outside of SGX, who else wants stuff _like_ this?
I don't mind renaming the attribute but maybe it should sgx_total_bytes, just because in some other sysfs attributes that keyword is used to refer all of the bytes?
/Jarkko
Jarkko Sakkinen
maintainerRe: [PATCH v9 2/2] x86/sgx: Add an attribute for the amount of SGX memory in a NUMA node
Quoted 10 lines · Jarkko Sakkinen
Quoted 9 lines · Dave Hansen
Documentation/ABI/stable/sysfs-devices-node Contact: Keith Busch <[email protected]> · line 182
Quoted 3 lines · Jarkko Sakkinen
Documentation/ABI/stable/sysfs-devices-node +3
-0+What: /sys/devices/system/node/nodeX/sgx/size +Date: October 2021 +Contact: Jarkko Sakkinen <[email protected]>I don't think we should do something *entirely* SGX-specific here. The only question to me is whether any non-SGX users want something like this and who they are.
Quoted 7 lines
Quoted 5 lines
Here are some ideas I like more than an "sgx/" directory:
/sys/devices/system/node/nodeX/arch/sgx_size
/sys/devices/system/node/nodeX/x86/sgx_size/sys/devices/system/node/nodeX/coco/sgx_sizeSure, I can rename the attribute group as "x86".
Dave,
So, if I rename sgx_node_group as arch_node_dev_group, we also need something like CONFIG_ARCH_HAS_NODE_DEV_GROUP defined in arch/Kconfig, so that the group definition in drivers/base/node.c becomes:
static const struct attribute_group *node_dev_groups[] = {
&node_dev_group,
#ifdef CONFIG_ARCH_HAS_NODE_DEV_GROUP
&arch_node_dev_group,
#endif
NULL,
}
Does this make sense to you, and reflect your feedback in a reasonable manner?
/Jarkko