x86/sgx: Do not update sgx_nr_free_pages in sgx_setup_epc_section()
Jarkko Sakkinen
maintainerPATCH v2 1/2 x86/sgx: Do not update sgx_nr_free_pages in sgx_setup_epc_section()
Now that the sanitization process will make pages available by calling sgx_free_epc_page(), sgx_setup_epc_section() should not touch to sgx_nr_free_pages. This will result sgx_nr_free_pages to contain 2x the number of actual free pages. Simply, remove the statement.
--- arch/x86/kernel/cpu/sgx/main.c | 1 - 1 file changed, 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +0 -1
@@ -657,7 +657,6 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size,list_add_tail (§ion->pages[i].list, &sgx_dirty_page_list); } - sgx_nr_free_pages += nr_pages;return true; }
Signature · 1 line
2.31.1
Jarkko Sakkinen
maintainerPATCH v2 2/2 x86/sgx: Add sgx_nr_{all, free}_pages to the debugfs
Add two debugs attributes:
* /sys/kernel/debug/x86/sgx_nr_all_pages
* /sys/kernel/debug/x86/sgx_nr_free_pages
These provide useful statistics for testing purposes.
E.g. on a NUC7CJYH2, when no enclaves are running, and EPC set to 32 MB:
$ sudo cat /sys/kernel/debug/x86/sgx_nr_all_pages 5632
$ sudo cat /sys/kernel/debug/x86/sgx_nr_free_pages 5632
---
v2:
* Use debugfs_create_ulong():
https://lore.kernel.org/linux-sgx/[email protected]/
arch/x86/kernel/cpu/sgx/main.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +12 -1
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright(c) 2016-20 Intel Corporation. */ +#include <linux/debugfs.h> #include <linux/freezer.h> #include <linux/highmem.h> #include <linux/kthread.h>
@@ -25,7 +26,10 @@ static DECLARE_WAIT_QUEUE_HEAD(ksgxd_waitq);static LIST_HEAD (sgx_active_page_list);static DEFINE_SPINLOCK (sgx_reclaimer_lock); -/* The free page list lock protected variables prepend the lock. */ +/* The number of EPC pages in total in all nodes. */ +static unsigned long sgx_nr_all_pages; + +/* The number of free EPC pages in all nodes. */ static unsigned long sgx_nr_free_pages;/* Nodes with one or more EPC sections. */
@@ -657,6 +661,8 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size,list_add_tail (§ion->pages[i].list, &sgx_dirty_page_list); } + sgx_nr_all_pages += nr_pages; +return true; }
@@ -750,6 +756,11 @@ static int __init sgx_init(void)if (ret) goto err_kthread; +#ifdef CONFIG_DEBUG_FS +debugfs_create_ulong ("sgx_nr_all_pages" ,0400 , arch_debugfs_dir,NULL , &sgx_nr_all_pages); +debugfs_create_ulong ("sgx_nr_free_pages" ,0400 , arch_debugfs_dir,NULL , &sgx_nr_free_pages); +#endif /* CONFIG_DEBUG_FS */ +return 0 ; err_kthread:
Signature · 1 line
2.31.1
Borislav Petkov
maintainerRe: [PATCH v2 2/2] x86/sgx: Add sgx_nr_{all, free}_pages to the debugfs
Quoted 4 lines · Jarkko Sakkinen
Add two debugs attributes:
*
/sys/kernel/debug/x86/sgx_nr_all_pages
*/sys/kernel/debug/x86/sgx_nr_free_pages
Definitely under /sys/kernel/debug/x86/sgx/...
^^^^^^
Quoted 1 line
These provide useful statistics for testing purposes.
Testing what exactly?
Also, if those are EPC pages, why isn't "epc" in the name?
Quoted 7 lines
E.g. on a NUC7CJYH2, when no enclaves are running, and EPC set to 32 MB:
$ sudo cat /sys/kernel/debug/x86/sgx_nr_all_pages 5632$ sudo cat /sys/kernel/debug/x86/sgx_nr_free_pages 5632
I have no clue what that is useful for. You want to account how many of the EPC pages on all nodes are in use? What for?
Are those globally useful for people? If so, they need to go to sysfs along with documentation what they do.
If not, you can keep this patch in your tree for your own testing.
Quoted 23 lines
Signed-off-by Jarkko Sakkinen <[email protected]>---
v2:
* Usedebugfs_create_ulong():
https://lore.kernel.org/linux-sgx/[email protected]/arch/x86/kernel/cpu/sgx/main.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +1
-0@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright(c) 2016-20 Intel Corporation. */ +#include <linux/debugfs.h> #include <linux/freezer.h> #include <linux/highmem.h> #include <linux/kthread.h>
arch/x86/kernel/cpu/sgx/main.c static DECLARE_WAIT_QUEUE_HEAD(ksgxd_waitq); · line 33
Quoted 9 lines
arch/x86/kernel/cpu/sgx/main.c +4
-1@@ -25,7 +26,10 @@ static DECLARE_WAIT_QUEUE_HEAD(ksgxd_waitq);static LIST_HEAD (sgx_active_page_list);static DEFINE_SPINLOCK (sgx_reclaimer_lock); -/* The free page list lock protected variables prepend the lock. */ +/* The number of EPC pages in total in all nodes. */ +static unsigned long sgx_nr_all_pages; + +/* The number of free EPC pages in all nodes. */ static unsigned long sgx_nr_free_pages;
Ok, you're fixing the comment here. Good.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJarkko Sakkinen
maintainerRe: [PATCH v2 2/2] x86/sgx: Add sgx_nr_{all, free}_pages to the debugfs
Quoted 6 lines · Borislav Petkov
Quoted 4 lines · Jarkko Sakkinen
Add two debugs attributes:
*
/sys/kernel/debug/x86/sgx_nr_all_pages
*/sys/kernel/debug/x86/sgx_nr_free_pages
Quoted 8 lines
Definitely under
/sys/kernel/debug/x86/sgx/...^^^^^^Quoted 1 line
These provide useful statistics for testing purposes.
Testing what exactly?
Also, if those are EPC pages, why isn't "epc" in the name?
When debugging the SGX code it is useful to quickly check the amount of EPC pages, and also total amount of EPC available.
I left out "epc" because they are already prefixed with "sgx_".
Quoted 8 lines
Quoted 7 lines
E.g. on a NUC7CJYH2, when no enclaves are running, and EPC set to 32 MB:
$ sudo cat /sys/kernel/debug/x86/sgx_nr_all_pages 5632$ sudo cat /sys/kernel/debug/x86/sgx_nr_free_pages 5632
Quoted 7 lines
I have no clue what that is useful for. You want to account how many of the EPC pages on all nodes are in use? What for?
Are those globally useful for people? If so, they need to go to sysfs along with documentation what they do.
If not, you can keep this patch in your tree for your own testing.
E.g. when stress testing this might be useful information to scale the workload for example, or even sample the number of EPC pages. When otherwise testing this might be useful to catch any leaks. I created 1/2 based on what I saw with these variables.
debugfs was my first shot, but for sure these could be sysfs.
Quoted 26 lines
Quoted 26 lines
Signed-off-by Jarkko Sakkinen <[email protected]>---
v2:
* Usedebugfs_create_ulong():
https://lore.kernel.org/linux-sgx/[email protected]/arch/x86/kernel/cpu/sgx/main.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +1
-0@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0 /* Copyright(c) 2016-20 Intel Corporation. */ +#include <linux/debugfs.h> #include <linux/freezer.h> #include <linux/highmem.h> #include <linux/kthread.h> @@ -25,7 +26,10 @@ static DECLARE_WAIT_QUEUE_HEAD(ksgxd_waitq);static LIST_HEAD (sgx_active_page_list);static DEFINE_SPINLOCK (sgx_reclaimer_lock);
Quoted 8 lines
arch/x86/kernel/cpu/sgx/main.c static DECLARE_WAIT_QUEUE_HEAD(ksgxd_waitq); · line 33
Quoted 6 lines
arch/x86/kernel/cpu/sgx/main.c +4
-1-/* The free page list lock protected variables prepend the lock. */ +/* The number of EPC pages in total in all nodes. */ +static unsigned long sgx_nr_all_pages; + +/* The number of free EPC pages in all nodes. */ static unsigned long sgx_nr_free_pages;Ok, you're fixing the comment here. Good.
Should that be part of the first patch?
Quoted 8 lines
Thx.
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v2 2/2] x86/sgx: Add sgx_nr_{all, free}_pages to the debugfs
Quoted 1 line · Jarkko Sakkinen
I left out "epc" because they are already prefixed with "
sgx_".
Are there any other "page" types which are going to be figurating in some pseudofs or is "sgx" == "epc" in this case?
Quoted 1 line
debugfs was my first shot, but for sure these could be sysfs.
Ok, let's keep it in debugfs for now, it can always be made an ABI later and moved to sysfs. But pls document what those are and what they do and that when in debugfs, there are no guarantees that these interfaces will be there in the future.
Quoted 1 line
Should that be part of the first patch?
Yes pls.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJarkko Sakkinen
maintainerRe: [PATCH v2 2/2] x86/sgx: Add sgx_nr_{all, free}_pages to the debugfs
Quoted 6 lines · Borislav Petkov
Quoted 1 line · Jarkko Sakkinen
I left out "epc" because they are already prefixed with "
sgx_".Are there any other "page" types which are going to be figurating in some pseudofs or is "sgx" == "epc" in this case?
Quoted 6 lines
Quoted 1 line
debugfs was my first shot, but for sure these could be sysfs.
Ok, let's keep it in debugfs for now, it can always be made an ABI later and moved to sysfs. But pls document what those are and what they do and that when in debugfs, there are no guarantees that these interfaces will be there in the future.
I think these attributes are quite useful information to have available so I'd go actually doing sysfs attributes and create Documentation/ABI/stable/sysfs-driver-sgx to document them.
Given that they would go then to the sysfs directory of the driver, then probably the legit names for the attributes ought to be:
- nr_all_epc_pages
- nr_free_epc_pages
What do you think?
PS. One useful case that I forgot to mention is that I use these to give idea what I gave EPC size in the BIOS. Now my EPC is set to 32 MB, and these report 20 MB of EPC pages. It's because other metadata (e.g. EPCM containing page attributes) is also stored in this area.
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v2 2/2] x86/sgx: Add sgx_nr_{all, free}_pages to the debugfs
Quoted 3 lines · Jarkko Sakkinen
I think these attributes are quite useful information to have available so I'd go actually doing sysfs attributes and create
Documentation/ABI/stable/sysfs-driver-sgxto document them.
testing/
This directory documents interfaces that are felt to be stable, as the main development of this interface has been completed.
This sounds better for a start. From Documentation/ABI/README.
Quoted 7 lines
Given that they would go then to the sysfs directory of the driver, then probably the legit names for the attributes ought to be:
-
nr_all_epc_pages
-nr_free_epc_pagesWhat do you think?
Sounds ok to me.
Quoted 4 lines
PS. One useful case that I forgot to mention is that I use these to give idea what I gave EPC size in the BIOS. Now my EPC is set to 32 MB, and these report 20 MB of EPC pages. It's because other metadata (e.g. EPCM containing page attributes) is also stored in this area.
Just remember to put yourself in the user's shoes and think whether they make sense to her/him.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJarkko Sakkinen
maintainerRe: [PATCH v2 2/2] x86/sgx: Add sgx_nr_{all, free}_pages to the debugfs
In reply to Jarkko SakkinenQuoted 18 lines · Jarkko Sakkinen
Quoted 6 lines · Borislav Petkov
Quoted 1 line · Jarkko Sakkinen
I left out "epc" because they are already prefixed with "
sgx_".Are there any other "page" types which are going to be figurating in some pseudofs or is "sgx" == "epc" in this case?
Quoted 6 lines
Quoted 1 line
debugfs was my first shot, but for sure these could be sysfs.
Ok, let's keep it in debugfs for now, it can always be made an ABI later and moved to sysfs. But pls document what those are and what they do and that when in debugfs, there are no guarantees that these interfaces will be there in the future.
I think these attributes are quite useful information to have available so I'd go actually doing sysfs attributes and create
Documentation/ABI/stable/sysfs-driver-sgxto document them.
Quoted 7 lines
Given that they would go then to the sysfs directory of the driver, then probably the legit names for the attributes ought to be:
-
nr_all_epc_pages
-nr_free_epc_pagesWhat do you think?
Actually I think read-only sysctl attributes would be a better idea.
The rationale for this is that we have two misc devices sgx_enclave and sgx_provision, and these are global attributes even applicable to KVM.
It does not matter functionality-wise, but API-wise it'd look stupid to directly associate to sgx_enclave.
I.e. I'd add something along the lines of
static struct ctl_path x86_sysctl_path[] = {
{ .procname = "kernel", },
{ .procname = "x86", },
{ }
};static struct ctl_table x86_sysctl_table[] = {
{
.procname = "sgx_nr_all_pages",
.mode = 0444,
/* rest ... */
},
{
.procname = "sgx_nr_free_pages",
.mode = 0444,
/* rest ... */
},
{ }
};And write Documentation/x86/proc.rst.
/Jarkko
Borislav Petkov
maintainerRe: [PATCH v2 2/2] x86/sgx: Add sgx_nr_{all, free}_pages to the debugfs
Quoted 1 line · Jarkko Sakkinen
Actually I think read-only sysctl attributes would be a better idea.
I still think debugfs is the right *start* for this: you play with them, see what makes sense and what not, tweak them, etc, and then you cast them in stone.
Not cast them in stone and see if anyone is even interested. So pls keep them in debugfs for now - you can always do whatever, later, when it turns out that those are useful.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJarkko Sakkinen
maintainerRe: [PATCH v2 2/2] x86/sgx: Add sgx_nr_{all, free}_pages to the debugfs
Quoted 6 lines · Borislav Petkov
Quoted 1 line · Jarkko Sakkinen
Actually I think read-only sysctl attributes would be a better idea.
I still think debugfs is the right *start* for this: you play with them, see what makes sense and what not, tweak them, etc, and then you cast them in stone.
Yup. Also better to see what makes sense in the context of virtualization. Too many (unknown) unknowns to make the right call.
Quoted 3 lines
Not cast them in stone and see if anyone is even interested. So pls keep them in debugfs for now - you can always do whatever, later, when it turns out that those are useful.
I'll do that.
Quoted 7 lines
Thx.
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerQuoted 16 lines · Jarkko Sakkinen
Now that the sanitization process will make pages available by calling
sgx_free_epc_page(),sgx_setup_epc_section()should not touch tosgx_nr_free_pages. This will resultsgx_nr_free_pagesto contain 2x the number of actual free pages. Simply, remove the statement.Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list")Signed-off-by Jarkko Sakkinen <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 1 - 1 file changed, 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0
arch/x86/kernel/cpu/sgx/main.c static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size, · line 662
Quoted 7 lines
arch/x86/kernel/cpu/sgx/main.c +0
-1@@ -657,7 +657,6 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size,list_add_tail (§ion->pages[i].list, &sgx_dirty_page_list); } - sgx_nr_free_pages += nr_pages;return true; }
First of all, I don't know how I didn't catch this:
/* The free page list lock protected variables prepend the lock. */ static unsigned long sgx_nr_free_pages;
I need some sort of translator to understand what this comment means. I can guess what is trying to tell me.
Which leads to my question: what is sgx_nr_free_pages supposed to denote?
Because I understand the callpath
sgx_page_cache_init
...
for (i = 0; i < ARRAY_SIZE(sgx_epc_sections); i++) {
...
sgx_setup_epc_section
...
sgx_nr_free_pages += nr_pages;as adding the number of pages of each new EPC section to the total number of the free pages. Unless that variable accounts something else.
So what does this variable actually mean?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJarkko Sakkinen
maintainerQuoted 45 lines · Borislav Petkov
Quoted 16 lines · Jarkko Sakkinen
Now that the sanitization process will make pages available by calling
sgx_free_epc_page(),sgx_setup_epc_section()should not touch tosgx_nr_free_pages. This will resultsgx_nr_free_pagesto contain 2x the number of actual free pages. Simply, remove the statement.Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list")Signed-off-by Jarkko Sakkinen <[email protected]>--- arch/x86/kernel/cpu/sgx/main.c | 1 - 1 file changed, 1 deletion(-)arch/x86/kernel/cpu/sgx/main.c +0
-0arch/x86/kernel/cpu/sgx/main.c static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size, · line 662
Quoted 7 lines
arch/x86/kernel/cpu/sgx/main.c +0
-1@@ -657,7 +657,6 @@ static bool __init sgx_setup_epc_section(u64 phys_addr, u64 size,list_add_tail (§ion->pages[i].list, &sgx_dirty_page_list); } - sgx_nr_free_pages += nr_pages;return true; }First of all, I don't know how I didn't catch this:
/* The free page list lock protected variables prepend the lock. */ static unsigned long
sgx_nr_free_pages;I need some sort of translator to understand what this comment means. I can guess what is trying to tell me.
Which leads to my question: what is
sgx_nr_free_pagessupposed to denote?Because I understand the callpath
sgx_page_cache_init
...for (i = 0; i < ARRAY_SIZE(sgx_epc_sections); i++) { ... sgx_setup_epc_section ... sgx_nr_free_pages += nr_pages;
Quoted 4 lines
as adding the number of pages of each new EPC section to the total number of the free pages. Unless that variable accounts something else.
So what does this variable actually mean?
It's used for only to trigger watermark for reclaiming. I.e. causes ksgxd to trigger. And it gives the number of total free EPC pages in all NUMA nodes.
/Jarkko
Borislav Petkov
maintainerQuoted 12 lines · Jarkko Sakkinen
Quoted 12 lines
Which leads to my question: what is
sgx_nr_free_pagessupposed to denote?Because I understand the callpath
sgx_page_cache_init
...for (i = 0; i < ARRAY_SIZE(sgx_epc_sections); i++) { ... sgx_setup_epc_section ... sgx_nr_free_pages += nr_pages;
Quoted 8 lines
Quoted 4 lines
as adding the number of pages of each new EPC section to the total number of the free pages. Unless that variable accounts something else.
So what does this variable actually mean?
It's used for only to trigger watermark for reclaiming. I.e. causes ksgxd to trigger. And it gives the number of total free EPC pages in all NUMA nodes.
So the callpath I laid out above is adding the number of pages of each section to the total free EPC pages number.
Why is that wrong and why is your patch needed?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJarkko Sakkinen
maintainerQuoted 18 lines · Borislav Petkov
Quoted 17 lines · Jarkko Sakkinen
Quoted 16 lines
Which leads to my question: what is
sgx_nr_free_pagessupposed to denote?Because I understand the callpath
sgx_page_cache_init
...for (i = 0; i < ARRAY_SIZE(sgx_epc_sections); i++) { ... sgx_setup_epc_section ... sgx_nr_free_pages += nr_pages;as adding the number of pages of each new EPC section to the total number of the free pages. Unless that variable accounts something else.
So what does this variable actually mean?
Quoted 8 lines
Quoted 3 lines
It's used for only to trigger watermark for reclaiming. I.e. causes ksgxd to trigger. And it gives the number of total free EPC pages in all NUMA nodes.
So the callpath I laid out above is adding the number of pages of each section to the total free EPC pages number.
Why is that wrong and why is your patch needed?
As part of "x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list" pages are processed from a global list by ksgxd.
This in turn introduces change to sanitization:
- if (!ret)
- list_move(&page->list, §ion->page_list);
- else
+ if (!ret) {
+ /*
+ * page is now sanitized. Make it available via the SGX
+ * page allocator:
+ */
+ list_del(&page->list);
+ sgx_free_epc_page(page);
+ } else {
+ /* The page is not yet clean - move to the dirty list. */
list_move_tail(&page->list, &dirty);
-
- spin_unlock(§ion->lock);
+ }
This is done for the reason that it is best to keep the logic to assign available-for-use EPC pages to correct NUMA lists in a single location.
The regression is that the sgx_nr_free_pages is also incremented by sgx_free_epc_pages(), and thus it ends up having double the number of pages available.
/Jarkko
Borislav Petkov
maintainerQuoted 3 lines · Jarkko Sakkinen
The regression is that the
sgx_nr_free_pagesis also incremented bysgx_free_epc_pages(), and thus it ends up having double the number of pages available.
So when you add a new EPC section with sgx_setup_epc_section(), those new pages in "nr_pages" are initially not going to be accounted anywhere? Or is that sgx_nr_all_pages? And you do that in your second patch...
But those new pages coming in *are* free pages so they should be in the free pages count too, IMHO.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteJarkko Sakkinen
maintainerQuoted 5 lines · Borislav Petkov
Quoted 3 lines · Jarkko Sakkinen
The regression is that the
sgx_nr_free_pagesis also incremented bysgx_free_epc_pages(), and thus it ends up having double the number of pages available.
Quoted 7 lines
So when you add a new EPC section with
sgx_setup_epc_section(), those new pages in "nr_pages" are initially not going to be accounted anywhere? Or is thatsgx_nr_all_pages? And you do that in your second patch...But those new pages coming in *are* free pages so they should be in the free pages count too, IMHO.
They are not in the "free_page_list" before sanitization process has put them to there. So in that way the count is also better in sync with this fix.
It was even before NUMA patches kind of out-of-sync, i.e. free pages count was incremented before putting them to the free list, but it didn't matter that much because sanitization is fairly fast and it only prevented ksgxd to trigger small time after the booth.
I think I'll send the fix for this now as a separate patch with a better descriptino so that I can use more time to the attributes and implement that properly at once.
Quoted 5 lines
Signature · 4 lines
Regards/Gruss, Boris. https://people.kernel.org/tglx/notes-about-netiquette
/Jarkko
Borislav Petkov
maintainerQuoted 3 lines · Jarkko Sakkinen
They are not in the "
free_page_list" before sanitization process has put them to there. So in that way the count is also better in sync with this fix.
This is the bit of information I was looking for. This needs to be in the commit message.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette