Skip to contentSkip to conversation

x86/sgx: Do not consider unsanitized pages an error

PATCH Superseded Jarkko Sakkinen 3 participants 9 messages Aug 25, 2022

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Jarkko Sakkinen

author maintainer
#0 lore raw

PATCH v3 1/1 x86/sgx: Do not consider unsanitized pages an error

If sgx_dirty_page_list ends up being non-empty, currently this triggers WARN_ON(), which produces a lot of noise, and can potentially crash the kernel, depending on the kernel command line.

However, if the SGX subsystem initialization is retracted, the sanitization process could end up in the middle, and sgx_dirty_page_list be left non-empty for legit reasons.

Replace this faulty behavior with more verbose version __sgx_sanitize_pages(), which can optionally print EREMOVE error code and the number of unsanitized pages.

Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list")
Cc Haitao Huang <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Reinette Chatre <[email protected]>

---
v3:
- Remove WARN_ON().
- Tuned comments and the commit message a bit.

v2:
- Replaced WARN_ON() with optional pr_info() inside __sgx_sanitize_pages().
- Rewrote the commit message.
- Added the fixes tag.

---
 arch/x86/kernel/cpu/sgx/main.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/sgx/main.c +13 -7
index 515e2a5f25bb..d204520a5e26 100644
@@ -50,16 +50,17 @@ static LIST_HEAD(sgx_dirty_page_list);
  * from the input list, and made available for the page allocator. SECS pages
  * prepending their children in the input list are left intact.
  */
-static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
+static void __sgx_sanitize_pages(struct list_head *dirty_page_list, bool verbose)
 {
 	struct sgx_epc_page *page;
+	int dirty_count = 0;
 	LIST_HEAD(dirty);
 	int ret;
 
 	/* dirty_page_list is thread-local, no need for a lock: */
 	while (!list_empty(dirty_page_list)) {
 		if (kthread_should_stop())
-			return;
+			break;
 
 		page = list_first_entry(dirty_page_list, struct sgx_epc_page, list);
 
@@ -90,14 +91,22 @@ static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
 			list_del(&page->list);
 			sgx_free_epc_page(page);
 		} else {
+			if (verbose)
+				pr_err_ratelimited(EREMOVE_ERROR_MESSAGE, ret, ret);
+
 			/* The page is not yet clean - move to the dirty list. */
 			list_move_tail(&page->list, &dirty);
+			dirty_count++;
 		}
 
 		cond_resched();
 	}
 
 	list_splice(&dirty, dirty_page_list);
+
+	/* Can happen, when the initialization is retracted: */
+	if (verbose && dirty_count > 0)
+		pr_info("%d unsanitized pages\n", dirty_count);
 }
 
 static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)
@@ -394,11 +403,8 @@ static int ksgxd(void *p)
 	 * Sanitize pages in order to recover from kexec(). The 2nd pass is
 	 * required for SECS pages, whose child pages blocked EREMOVE.
 	 */
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
-
-	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	__sgx_sanitize_pages(&sgx_dirty_page_list, false);
+	__sgx_sanitize_pages(&sgx_dirty_page_list, true);
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())
Signature · 1 line
2.37.1
#1 lore raw
Quoted 3 lines · Jarkko Sakkinen

However, if the SGX subsystem initialization is retracted, the sanitization process could end up in the middle, and sgx_dirty_page_list be left non-empty for legit reasons.

What does "retraction" mean in this context?

Reviewed-by on v3 1/1
#5 lore raw
In reply to Jarkko Sakkinen

Hi Jarkko,

Quoted 51 lines · Jarkko Sakkinen

If sgx_dirty_page_list ends up being non-empty, currently this triggers WARN_ON(), which produces a lot of noise, and can potentially crash the kernel, depending on the kernel command line.

However, if the SGX subsystem initialization is retracted, the sanitization process could end up in the middle, and sgx_dirty_page_list be left non-empty for legit reasons.

Replace this faulty behavior with more verbose version __sgx_sanitize_pages(), which can optionally print EREMOVE error code and the number of unsanitized pages.

Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list")
Cc Haitao Huang <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Reinette Chatre <[email protected]>

---
v3:
- Remove WARN_ON().
- Tuned comments and the commit message a bit.

v2:
- Replaced WARN_ON() with optional pr_info() inside __sgx_sanitize_pages().
- Rewrote the commit message.
- Added the fixes tag.

---
 arch/x86/kernel/cpu/sgx/main.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/sgx/main.c +0 -0
index 515e2a5f25bb..d204520a5e26 100644
@@ -50,16 +50,17 @@ static LIST_HEAD(sgx_dirty_page_list);
  * from the input list, and made available for the page allocator. SECS  

pages
* prepending their children in the input list are left intact. */

arch/x86/kernel/cpu/sgx/main.c +1 -1
-static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
+static void __sgx_sanitize_pages(struct list_head *dirty_page_list,  

bool verbose)
{

arch/x86/kernel/cpu/sgx/main.c +1 -0
 	struct sgx_epc_page *page;
+	int dirty_count = 0;
 	LIST_HEAD(dirty);
 	int ret;
Quoted 1 line
	/* dirty_page_list is thread-local, no need for a lock: */

Just a nitpick,
Although it is not added in this patch, the above comment is not accurate. The list is accessed one thread only: filled first in main thread, then only ever accessed here.

IIUC, could you remove or update that comment?

Other than that, FWIW:

Thanks
Haitao

author maintainer
#2 lore raw
In reply to Dave Hansen
Quoted 6 lines · Dave Hansen
Quoted 3 lines · Jarkko Sakkinen

However, if the SGX subsystem initialization is retracted, the sanitization process could end up in the middle, and sgx_dirty_page_list be left non-empty for legit reasons.

What does "retraction" mean in this context?

Rest of the initialization failing or features not detected (-ENODEV).

BR, Jarkko

#3 lore raw
Quoted 7 lines · Jarkko Sakkinen
Quoted 5 lines · Dave Hansen
Quoted 3 lines · Jarkko Sakkinen

However, if the SGX subsystem initialization is retracted, the sanitization process could end up in the middle, and sgx_dirty_page_list be left non-empty for legit reasons.

What does "retraction" mean in this context?

Rest of the initialization failing or features not detected (-ENODEV).

Can you please work on communicating better descriptions of the problems? This really isn't good enough.

I think you're talking about sgx_init(). It launches ksgxd from sgx_page_reclaimer_init() which sets about sanitizing the 'dirty_page_list'. After launching ksgxd, if later actions in sgx_init() (misc_register(), sgx_drv_init(), sgx_vepc_init()) fail, ksgxd will be stopped prematurely.

This will leave pages in 'sgx_dirty_page_list' after __sgx_sanitize_pages() has completed, which results in a WARN_ON().

The WARN_ON() is really only valid when __sgx_sanitize_pages() runs to completion *and* fails to empty 'sgx_dirty_page_list'.

Is that it?

If so, could you please give the changelog another go?

author maintainer
#6 lore raw
In reply to Haitao Huang
Quoted 66 lines · Haitao Huang

Hi Jarkko,

Quoted 54 lines · Jarkko Sakkinen

If sgx_dirty_page_list ends up being non-empty, currently this triggers WARN_ON(), which produces a lot of noise, and can potentially crash the kernel, depending on the kernel command line.

However, if the SGX subsystem initialization is retracted, the sanitization
process could end up in the middle, and sgx_dirty_page_list be left non-empty for legit reasons.

Replace this faulty behavior with more verbose version __sgx_sanitize_pages(), which can optionally print EREMOVE error code and the number of unsanitized pages.

Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with

sgx_dirty_page_list")

Cc Haitao Huang <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Reinette Chatre <[email protected]>

---
v3:
- Remove WARN_ON().
- Tuned comments and the commit message a bit.

v2:
- Replaced WARN_ON() with optional pr_info() inside __sgx_sanitize_pages().
- Rewrote the commit message.
- Added the fixes tag.

---
 arch/x86/kernel/cpu/sgx/main.c | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/sgx/main.c +0 -0
index 515e2a5f25bb..d204520a5e26 100644
@@ -50,16 +50,17 @@ static LIST_HEAD(sgx_dirty_page_list);
  * from the input list, and made available for the page allocator. SECS

pages
* prepending their children in the input list are left intact. */

arch/x86/kernel/cpu/sgx/main.c +1 -1
-static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
+static void __sgx_sanitize_pages(struct list_head *dirty_page_list,

bool verbose)
{

arch/x86/kernel/cpu/sgx/main.c +1 -0
 	struct sgx_epc_page *page;
+	int dirty_count = 0;
 	LIST_HEAD(dirty);
 	int ret;
Quoted 1 line
	/* dirty_page_list is thread-local, no need for a lock: */

Just a nitpick,
Although it is not added in this patch, the above comment is not accurate. The list is accessed one thread only: filled first in main thread, then only ever accessed here.

Quoted 1 line

IIUC, could you remove or update that comment?

Well, if we cut hairs here, it's actually expectation for the caller, right? :-)

arch/x86/kernel/cpu/sgx/main.c +2 -1
index d204520a5e26..c6f416307812 100644
@@ -49,6 +49,8 @@ static LIST_HEAD(sgx_dirty_page_list);
  * Reset post-kexec EPC pages to the uninitialized state. The pages are removed
  * from the input list, and made available for the page allocator. SECS pages
  * prepending their children in the input list are left intact.
+ *
+ * @dirty_page_list must be thread-local, i.e. no need for a lock.
  */
 static void __sgx_sanitize_pages(struct list_head *dirty_page_list, bool verbose)
 {
@@ -57,7 +59,6 @@ static void __sgx_sanitize_pages(struct list_head *dirty_page_list, bool verbose
        LIST_HEAD(dirty);
        int ret;
 
-       /* dirty_page_list is thread-local, no need for a lock: */
        while (!list_empty(dirty_page_list)) {
                if (kthread_should_stop())
                        break;
Quoted 5 lines

Other than that, FWIW:

Thanks
Haitao

Thank you.

BR, Jarkko

#7 lore raw
In reply to Jarkko Sakkinen
Quoted 7 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/main.c +3 -0
+	/* Can happen, when the initialization is retracted: */
+	if (verbose && dirty_count > 0)
+		pr_info("%d unsanitized pages\n", dirty_count);
 }
 
 static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)

arch/x86/kernel/cpu/sgx/main.c static int ksgxd(void *p) · line 407

Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +2 -5
@@ -394,11 +403,8 @@ static int ksgxd(void *p)
 	 * Sanitize pages in order to recover from kexec(). The 2nd pass is
 	 * required for SECS pages, whose child pages blocked EREMOVE.
 	 */
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
-
-	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	__sgx_sanitize_pages(&sgx_dirty_page_list, false);
+	__sgx_sanitize_pages(&sgx_dirty_page_list, true);

This is backwards, IMNHO.

Make __sgx_sanitize_pages() return the number of pages that it leaves dirty.

	__sgx_sanitize_pages(&sgx_dirty_page_list)
	left_dirty = __sgx_sanitize_pages(&sgx_dirty_page_list);
	if (left_dirty)
		pr_warn(...);

That rids us of the mystery true/false and puts the pr_warn() in a place that makes logical sense. Then, let's either *not* do the

	pr_err_ratelimited(EREMOVE_ERROR_MESSAGE, ret, ret);

at all, or make it an unconditional pr_warn_ratelimited(). They're not going to be common and multiple messages are virtually worthless anyway.

I actually think a common tracepoint, or out-of-line ENCLS/ENCLU functions that can be easily ftraced are a much better idea than a one-off pr_whatever().

author maintainer
#4 lore raw
In reply to Dave Hansen
Quoted 11 lines · Dave Hansen
Quoted 7 lines · Jarkko Sakkinen
Quoted 5 lines · Dave Hansen
Quoted 3 lines · Jarkko Sakkinen

However, if the SGX subsystem initialization is retracted, the sanitization process could end up in the middle, and sgx_dirty_page_list be left non-empty for legit reasons.

What does "retraction" mean in this context?

Rest of the initialization failing or features not detected (-ENODEV).

Can you please work on communicating better descriptions of the problems? This really isn't good enough.

Sure, I can put more detail into this patch.

If you speak in general about commit messages, picking the correct granularity is somewhat easy to fail because different people have different expectations on that. If denoted, I'm happy to write more detailed description, if the original is not granular enough.

Quoted 5 lines

I think you're talking about sgx_init(). It launches ksgxd from sgx_page_reclaimer_init() which sets about sanitizing the 'dirty_page_list'. After launching ksgxd, if later actions in sgx_init() (misc_register(), sgx_drv_init(), sgx_vepc_init()) fail, ksgxd will be stopped prematurely.

It's a bit more complicated, as either sgx_drv_init() or sgx_vepc_init() can fail without premature end for ksgxd.

So the exact conditions for premature stop are:

"In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped."

Quoted 5 lines

This will leave pages in 'sgx_dirty_page_list' after __sgx_sanitize_pages() has completed, which results in a WARN_ON().

The WARN_ON() is really only valid when __sgx_sanitize_pages() runs to completion *and* fails to empty 'sgx_dirty_page_list'.

This is correct.

Quoted 1 line

Is that it?

Just thinking if pr_warn() should be used if running to the completion and failing to empty the list. A bit more information to the klog on conditions, and not much extra complexity. What do you think?

Quoted 1 line

If so, could you please give the changelog another go?

BR, Jarkko

author maintainer
#8 lore raw
In reply to Dave Hansen
Quoted 21 lines · Dave Hansen
Quoted 7 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/main.c +3 -0
+	/* Can happen, when the initialization is retracted: */
+	if (verbose && dirty_count > 0)
+		pr_info("%d unsanitized pages\n", dirty_count);
 }
 
 static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)

arch/x86/kernel/cpu/sgx/main.c static int ksgxd(void *p) · line 407

Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +2 -5
@@ -394,11 +403,8 @@ static int ksgxd(void *p)
 	 * Sanitize pages in order to recover from kexec(). The 2nd pass is
 	 * required for SECS pages, whose child pages blocked EREMOVE.
 	 */
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
-
-	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	__sgx_sanitize_pages(&sgx_dirty_page_list, false);
+	__sgx_sanitize_pages(&sgx_dirty_page_list, true);

This is backwards, IMNHO.

Quoted 7 lines

Make __sgx_sanitize_pages() return the number of pages that it leaves dirty.

	__sgx_sanitize_pages(&sgx_dirty_page_list)
	left_dirty = __sgx_sanitize_pages(&sgx_dirty_page_list);
	if (left_dirty)
		pr_warn(...);

I like this and my patch has already the counter in place so why not.

Quoted 11 lines

That rids us of the mystery true/false and puts the pr_warn() in a place that makes logical sense. Then, let's either *not* do the

	pr_err_ratelimited(EREMOVE_ERROR_MESSAGE, ret, ret);

at all, or make it an unconditional pr_warn_ratelimited(). They're not going to be common and multiple messages are virtually worthless anyway.

I actually think a common tracepoint, or out-of-line ENCLS/ENCLU functions that can be easily ftraced are a much better idea than a one-off pr_whatever().

I like the tracepoint idea more than out-of-line ENCLS/ENCLU because out-of-line is more "intrusive" change to the code semantics than a tracepoint.

BR, Jarkko