Skip to contentSkip to conversation

x86/sgx: Test and fixes

PATCH In review Jarkko Sakkinen 5 participants 56 messages Aug 30, 2022 – Sep 2, 2022

Subsystems (3)
lore
Share & tools
lore

Jarkko Sakkinen

author maintainer
#0 lore raw

cover x86/sgx: Test and fixes

A collection of tests and fixes for SGX.

Haitao Huang (2):
x86/sgx: Handle VA page allocation failure for EAUG on PF. selftests/sgx: retry the ioctls returned with EAGAIN

Jarkko Sakkinen (2):
x86/sgx: Do not consider unsanitized pages an error selftests/sgx: Add a bpftrace script for tracking allocation errors

Kristen Carlson Accardi (1):
selftests/sgx: Ignore OpenSSL 3.0 deprecated functions warning

Vijay Dhanraj (1):
selftests/sgx: Add SGX selftest augment_via_eaccept_long

 arch/x86/kernel/cpu/sgx/encl.c             |   5 +-
 arch/x86/kernel/cpu/sgx/main.c             |  19 ++-
 tools/testing/selftests/sgx/alloc-error.bt |   7 +
 tools/testing/selftests/sgx/load.c         |   5 +-
 tools/testing/selftests/sgx/main.c         | 180 +++++++++++++++++----
 tools/testing/selftests/sgx/main.h         |   3 +-
 tools/testing/selftests/sgx/sigstruct.c    |   8 +-
 7 files changed, 189 insertions(+), 38 deletions(-)
 create mode 100644 tools/testing/selftests/sgx/alloc-error.bt
Signature · 1 line
2.37.2
author maintainer
#1 lore raw

PATCH 1/6 x86/sgx: Do not consider unsanitized pages an error

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

This triggers WARN_ON() because sgx_dirty_page_list ends up being non-empty, and dumps the call stack:

[    0.000000] Linux version 6.0.0-rc2 (root@4beb429beb4a) (gcc (Debian
11.3.0-3) 11.3.0, GNU ld (GNU Binutils for Debian) 2.38) #382 SMP
PREEMPT_DYNAMIC Fri Aug 26 12:52:15 UTC 2022
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.0.0-rc2
root=UUID=56f398e0-1e25-4fda-aa9f-611dece4b333 ro quiet
module_blacklist=psmouse initcall_debug log_buf_len=4M cryptomgr.notests
[…]
[    0.268089] calling  sgx_init+0x0/0x409 @ 1
[    0.268103] sgx: EPC section 0x40200000-0x45f7ffff
[    0.268591] ------------[ cut here ]------------
[    0.268592] WARNING: CPU: 6 PID: 83 at
arch/x86/kernel/cpu/sgx/main.c:401 ksgxd+0x1b7/0x1d0
[    0.268598] Modules linked in:
[    0.268600] CPU: 6 PID: 83 Comm: ksgxd Not tainted 6.0.0-rc2 #382
[    0.268603] Hardware name: Dell Inc. XPS 13 9370/0RMYH9, BIOS 1.21.0
07/06/2022
[    0.268604] RIP: 0010:ksgxd+0x1b7/0x1d0
[    0.268607] Code: ff e9 f2 fe ff ff 48 89 df e8 75 07 0e 00 84 c0 0f
84 c3 fe ff ff 31 ff e8 e6 07 0e 00 84 c0 0f 85 94 fe ff ff e9 af fe ff
ff <0f> 0b e9 7f fe ff ff e8 dd 9c 95 00 66 66 2e 0f 1f 84 00 00 00 00
[    0.268608] RSP: 0000:ffffb6c7404f3ed8 EFLAGS: 00010287
[    0.268610] RAX: ffffb6c740431a10 RBX: ffff8dcd8117b400 RCX:
0000000000000000
[    0.268612] RDX: 0000000080000000 RSI: ffffb6c7404319d0 RDI:
00000000ffffffff
[    0.268613] RBP: ffff8dcd820a4d80 R08: ffff8dcd820a4180 R09:
ffff8dcd820a4180
[    0.268614] R10: 0000000000000000 R11: 0000000000000006 R12:
ffffb6c74006bce0
[    0.268615] R13: ffff8dcd80e63880 R14: ffffffffa8a60f10 R15:
0000000000000000
[    0.268616] FS:  0000000000000000(0000) GS:ffff8dcf25580000(0000)
knlGS:0000000000000000
[    0.268617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.268619] CR2: 0000000000000000 CR3: 0000000213410001 CR4:
00000000003706e0
[    0.268620] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[    0.268621] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[    0.268622] Call Trace:
[    0.268624]  <TASK>
[    0.268627]  ? _raw_spin_lock_irqsave+0x24/0x60
[    0.268632]  ? _raw_spin_unlock_irqrestore+0x23/0x40
[    0.268634]  ? __kthread_parkme+0x36/0x90
[    0.268637]  kthread+0xe5/0x110
[    0.268639]  ? kthread_complete_and_exit+0x20/0x20
[    0.268642]  ret_from_fork+0x1f/0x30
[    0.268647]  </TASK>
[    0.268648] ---[ end trace 0000000000000000 ]---
[    0.268694] initcall sgx_init+0x0/0x409 returned -19 after 603 usecs

Ultimately this can crash the kernel, if the following is set:

/proc/sys/kernel/panic_on_warn

Print a simple warning instead, and improve the output by printing the number of unsanitized pages, in order to provide debug informnation for future needs.

Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list")

---
Cc: Haitao Huang <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc: Reinette Chatre <[email protected]>

v5:
- Add the klog dump and sysctl option to the commit message.

v4:
- Explain expectations for dirty_page_list in the function header, instead of an inline comment.
- Improve commit message to explain the conditions better.
- Return the number of pages left dirty to ksgxd() and print warning after the 2nd call, if there are any.

v3:
- Remove WARN_ON().
- Tuned comments and the commit message a bit.

v2:
- Replaced WARN_ON() with optional pr_info() inside __sgx_sanitize_pages().
- Rewrote the commit message.
- Added the fixes tag.

---
 arch/x86/kernel/cpu/sgx/main.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)
arch/x86/kernel/cpu/sgx/main.c +13 -6
index 515e2a5f25bb..903100fcfce3 100644
@@ -49,17 +49,20 @@ static LIST_HEAD(sgx_dirty_page_list);
  * Reset post-kexec EPC pages to the uninitialized state. The pages are removed
  * from the input list, and made available for the page allocator. SECS pages
  * prepending their children in the input list are left intact.
+ *
+ * Contents of the @dirty_page_list must be thread-local, i.e.
+ * not shared by multiple threads.
  */
-static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
+static int __sgx_sanitize_pages(struct list_head *dirty_page_list)
 {
 	struct sgx_epc_page *page;
+	int left_dirty = 0;
 	LIST_HEAD(dirty);
 	int ret;
 
-	/* dirty_page_list is thread-local, no need for a lock: */
 	while (!list_empty(dirty_page_list)) {
 		if (kthread_should_stop())
-			return;
+			break;
 
 		page = list_first_entry(dirty_page_list, struct sgx_epc_page, list);
 
@@ -92,12 +95,14 @@ static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
 		} else {
 			/* The page is not yet clean - move to the dirty list. */
 			list_move_tail(&page->list, &dirty);
+			left_dirty++;
 		}
 
 		cond_resched();
 	}
 
 	list_splice(&dirty, dirty_page_list);
+	return left_dirty;
 }
 
 static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)
@@ -388,6 +393,8 @@ void sgx_reclaim_direct(void)
 
 static int ksgxd(void *p)
 {
+	int left_dirty;
+
 	set_freezable();
 
 	/*
@@ -395,10 +402,10 @@ static int ksgxd(void *p)
 	 * required for SECS pages, whose child pages blocked EREMOVE.
 	 */
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
-	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	left_dirty = __sgx_sanitize_pages(&sgx_dirty_page_list);
+	if (left_dirty)
+		pr_warn("%d unsanitized pages\n", left_dirty);
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())
Signature · 1 line
2.37.2
#2 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

To help the reader understand the subject of this patch it may help to explain that prematurely stopping ksgxd may leave some unsanitized pages, but that is not a problem since SGX cannot be used on the platform anyway.

Quoted 3 lines

This triggers WARN_ON() because sgx_dirty_page_list ends up being non-empty, and dumps the call stack:

Traces like below can be frowned upon. I recommend that you follow the guidance in "Backtraces in commit mesages"(sic) in Documentation/process/submitting-patches.rst.

Quoted 52 lines
[    0.000000] Linux version 6.0.0-rc2 (root@4beb429beb4a) (gcc (Debian
11.3.0-3) 11.3.0, GNU ld (GNU Binutils for Debian) 2.38) #382 SMP
PREEMPT_DYNAMIC Fri Aug 26 12:52:15 UTC 2022
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-6.0.0-rc2
root=UUID=56f398e0-1e25-4fda-aa9f-611dece4b333 ro quiet
module_blacklist=psmouse initcall_debug log_buf_len=4M cryptomgr.notests
[…]
[    0.268089] calling  sgx_init+0x0/0x409 @ 1
[    0.268103] sgx: EPC section 0x40200000-0x45f7ffff
[    0.268591] ------------[ cut here ]------------
[    0.268592] WARNING: CPU: 6 PID: 83 at
arch/x86/kernel/cpu/sgx/main.c:401 ksgxd+0x1b7/0x1d0
[    0.268598] Modules linked in:
[    0.268600] CPU: 6 PID: 83 Comm: ksgxd Not tainted 6.0.0-rc2 #382
[    0.268603] Hardware name: Dell Inc. XPS 13 9370/0RMYH9, BIOS 1.21.0
07/06/2022
[    0.268604] RIP: 0010:ksgxd+0x1b7/0x1d0
[    0.268607] Code: ff e9 f2 fe ff ff 48 89 df e8 75 07 0e 00 84 c0 0f
84 c3 fe ff ff 31 ff e8 e6 07 0e 00 84 c0 0f 85 94 fe ff ff e9 af fe ff
ff <0f> 0b e9 7f fe ff ff e8 dd 9c 95 00 66 66 2e 0f 1f 84 00 00 00 00
[    0.268608] RSP: 0000:ffffb6c7404f3ed8 EFLAGS: 00010287
[    0.268610] RAX: ffffb6c740431a10 RBX: ffff8dcd8117b400 RCX:
0000000000000000
[    0.268612] RDX: 0000000080000000 RSI: ffffb6c7404319d0 RDI:
00000000ffffffff
[    0.268613] RBP: ffff8dcd820a4d80 R08: ffff8dcd820a4180 R09:
ffff8dcd820a4180
[    0.268614] R10: 0000000000000000 R11: 0000000000000006 R12:
ffffb6c74006bce0
[    0.268615] R13: ffff8dcd80e63880 R14: ffffffffa8a60f10 R15:
0000000000000000
[    0.268616] FS:  0000000000000000(0000) GS:ffff8dcf25580000(0000)
knlGS:0000000000000000
[    0.268617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    0.268619] CR2: 0000000000000000 CR3: 0000000213410001 CR4:
00000000003706e0
[    0.268620] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[    0.268621] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[    0.268622] Call Trace:
[    0.268624]  <TASK>
[    0.268627]  ? _raw_spin_lock_irqsave+0x24/0x60
[    0.268632]  ? _raw_spin_unlock_irqrestore+0x23/0x40
[    0.268634]  ? __kthread_parkme+0x36/0x90
[    0.268637]  kthread+0xe5/0x110
[    0.268639]  ? kthread_complete_and_exit+0x20/0x20
[    0.268642]  ret_from_fork+0x1f/0x30
[    0.268647]  </TASK>
[    0.268648] ---[ end trace 0000000000000000 ]---
[    0.268694] initcall sgx_init+0x0/0x409 returned -19 after 603 usecs
Quoted 7 lines

Ultimately this can crash the kernel, if the following is set:

/proc/sys/kernel/panic_on_warn

Print a simple warning instead, and improve the output by printing the number of unsanitized pages, in order to provide debug informnation for future needs.

informnation -> information

...

Quoted 5 lines
Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list")

Should this go to stable?

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 54

Quoted 12 lines
arch/x86/kernel/cpu/sgx/main.c +3 -0
index 515e2a5f25bb..903100fcfce3 100644
@@ -49,17 +49,20 @@ static LIST_HEAD(sgx_dirty_page_list);
  * Reset post-kexec EPC pages to the uninitialized state. The pages are removed
  * from the input list, and made available for the page allocator. SECS pages
  * prepending their children in the input list are left intact.
+ *
+ * Contents of the @dirty_page_list must be thread-local, i.e.
+ * not shared by multiple threads.

Did you intend to mention something about the needed locking here? It looks like some information is lost during the move to the function description.

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 59

Quoted 6 lines
arch/x86/kernel/cpu/sgx/main.c +2 -1
  */
-static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
+static int __sgx_sanitize_pages(struct list_head *dirty_page_list)
 {
 	struct sgx_epc_page *page;
+	int left_dirty = 0;

I do not know how many pages this code should be ready for but at least this could handle more by being an unsigned int considering that it is always positive ... maybe even unsigned long?

Quoted 39 lines
arch/x86/kernel/cpu/sgx/main.c +5 -2
 	LIST_HEAD(dirty);
 	int ret;
 
-	/* dirty_page_list is thread-local, no need for a lock: */
 	while (!list_empty(dirty_page_list)) {
 		if (kthread_should_stop())
-			return;
+			break;
 
 		page = list_first_entry(dirty_page_list, struct sgx_epc_page, list);
 
@@ -92,12 +95,14 @@ static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
 		} else {
 			/* The page is not yet clean - move to the dirty list. */
 			list_move_tail(&page->list, &dirty);
+			left_dirty++;
 		}
 
 		cond_resched();
 	}
 
 	list_splice(&dirty, dirty_page_list);
+	return left_dirty;
 }
 
 static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)
@@ -388,6 +393,8 @@ void sgx_reclaim_direct(void)
 
 static int ksgxd(void *p)
 {
+	int left_dirty;
+
 	set_freezable();
 
 	/*
@@ -395,10 +402,10 @@ static int ksgxd(void *p)
 	 * required for SECS pages, whose child pages blocked EREMOVE.
 	 */
 	__sgx_sanitize_pages(&sgx_dirty_page_list);

arch/x86/kernel/cpu/sgx/main.c static int ksgxd(void *p) · line 411

Quoted 10 lines
arch/x86/kernel/cpu/sgx/main.c +3 -3
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
-	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	left_dirty = __sgx_sanitize_pages(&sgx_dirty_page_list);
+	if (left_dirty)
+		pr_warn("%d unsanitized pages\n", left_dirty);
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())

Reinette

#3 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

Btw currently EPC pages assigned to KVM guest cannot be reclaimed, so theoretically ksgxd can be moved to sgx_drv_init(), but who knows someday we will decide to make KVM guest EPC pages to be able to be reclaimed. :)

Signature · 2 lines
Thanks,
-Kai
author maintainer
#4 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 17 lines · Kai
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Quoted 5 lines

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

I would focus fixing the existing flow rather than reinventing the flow.

It can be made to work, and therefore it is IMHO correct action to take.

Quoted 3 lines

Btw currently EPC pages assigned to KVM guest cannot be reclaimed, so theoretically ksgxd can be moved to sgx_drv_init(), but who knows someday we will decide to make KVM guest EPC pages to be able to be reclaimed. :)

I'm open to changes but it is in my opinion out of context for this.

Quoted 8 lines
Signature · 3 lines
-- 
Thanks,
-Kai

BR, Jarkko

#5 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 24 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 17 lines · Kai
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Quoted 5 lines

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

Quoted 3 lines

I would focus fixing the existing flow rather than reinventing the flow.

It can be made to work, and therefore it is IMHO correct action to take.

From another perspective, the *existing flow* is the reason which causes this bug. A real fix is to fix the flow itself.

Quoted 8 lines
Quoted 4 lines

Btw currently EPC pages assigned to KVM guest cannot be reclaimed, so theoretically ksgxd can be moved to sgx_drv_init(), but who knows someday we will decide to make KVM guest EPC pages to be able to be reclaimed. :)

I'm open to changes but it is in my opinion out of context for this.

Yeah. I was expressing the reason I suggested to move sgx_page_reclaimer_init() to the end of sgx_init(), but not to sgx_drv_init().

But moving to sgx_drv_init() also makes sense to me given KVM guest EPC pages are not reclaimable now. For now there's no reason to run ksgxd() if only virtual EPC driver is enabled. We can move sgx_page_reclaimer_init() out of sgx_drv_init() when we add KVM guest EPC page reclaiming support (if it happens in the future).

Signature · 2 lines
Thanks,
-Kai
author maintainer
#6 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 25 lines · Kai
Quoted 24 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 22 lines · Kai
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

Quoted 6 lines
Quoted 3 lines

I would focus fixing the existing flow rather than reinventing the flow.

It can be made to work, and therefore it is IMHO correct action to take.

From another perspective, the *existing flow* is the reason which causes this bug. A real fix is to fix the flow itself.

Any existing flow in part of the kernel can have a bug. That does not mean that switching flow would be proper way to fix a bug.

BR, Jarkko

#7 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 30 lines · On Wed, 2022-08-31 at 05:44 +0300, [email protected] wrote:
Quoted 29 lines · Kai
Quoted 27 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 17 lines · Kai
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Quoted 5 lines

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

I would focus fixing the existing flow rather than reinventing the flow.

It can be made to work, and therefore it is IMHO correct action to take.

Quoted 8 lines
Quoted 2 lines

From another perspective, the *existing flow* is the reason which causes this bug. A real fix is to fix the flow itself.

Any existing flow in part of the kernel can have a bug. That does not mean that switching flow would be proper way to fix a bug.

BR, Jarkko

Yes but I think this is only true when the flow is reasonable. If the flow itself isn't reasonable, we should fix the flow (given it's easy to fix AFAICT).

Anyway, let us also hear from others.

Signature · 2 lines
Thanks,
-Kai
author maintainer
#8 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 38 lines · Kai
Quoted 37 lines · On Wed, 2022-08-31 at 05:44 +0300, [email protected] wrote:
Quoted 25 lines · Kai
Quoted 24 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 22 lines · Kai
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

Quoted 6 lines
Quoted 3 lines

I would focus fixing the existing flow rather than reinventing the flow.

It can be made to work, and therefore it is IMHO correct action to take.

From another perspective, the *existing flow* is the reason which causes this bug. A real fix is to fix the flow itself.

Any existing flow in part of the kernel can have a bug. That does not mean that switching flow would be proper way to fix a bug.

Quoted 6 lines
Quoted 1 line

BR, Jarkko

Yes but I think this is only true when the flow is reasonable. If the flow itself isn't reasonable, we should fix the flow (given it's easy to fix AFAICT).

Anyway, let us also hear from others.

The flow can be made to work without issues, which in the context of a bug fix is exactly what a bug fix should do. Not more or less.

You don't gain any measurable value for the user with this switch idea.

BR, Jarkko

author maintainer
#9 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 44 lines · On Wed, Aug 31, 2022 at 05:57:22AM +0300, [email protected] wrote:
Quoted 43 lines · Kai
Quoted 30 lines · On Wed, 2022-08-31 at 05:44 +0300, [email protected] wrote:
Quoted 29 lines · Kai
Quoted 27 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 17 lines · Kai
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Quoted 5 lines

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

I would focus fixing the existing flow rather than reinventing the flow.

It can be made to work, and therefore it is IMHO correct action to take.

Quoted 8 lines
Quoted 2 lines

From another perspective, the *existing flow* is the reason which causes this bug. A real fix is to fix the flow itself.

Any existing flow in part of the kernel can have a bug. That does not mean that switching flow would be proper way to fix a bug.

BR, Jarkko

Yes but I think this is only true when the flow is reasonable. If the flow itself isn't reasonable, we should fix the flow (given it's easy to fix AFAICT).

Quoted 8 lines
Quoted 1 line

Anyway, let us also hear from others.

The flow can be made to work without issues, which in the context of a bug fix is exactly what a bug fix should do. Not more or less.

You don't gain any measurable value for the user with this switch idea.

And besides this not proper way to review patch anyway because you did not review the code. I'll focus on fix what is broken e.g. so that it is easy to backport to stable and distro kernels, and call it a day. It certainly does not have to make code "perfect", as long as known bugs are sorted out.

You are welcome to review the next version of the patch, once I've resolved the issues that were pointed out by Reinette, if you still see some issue but this type of speculative discussion is frankly just wasting everyones time.

(need to check my mutt config, do not know why it is not always putting real name to from field)

BR, Jarkko

#10 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 51 lines · Jarkko Sakkinen
Quoted 50 lines · On Wed, Aug 31, 2022 at 05:57:22AM +0300, [email protected] wrote:
Quoted 38 lines · Kai
Quoted 37 lines · On Wed, 2022-08-31 at 05:44 +0300, [email protected] wrote:
Quoted 25 lines · Kai
Quoted 24 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 22 lines · Kai
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

Quoted 6 lines
Quoted 3 lines

I would focus fixing the existing flow rather than reinventing the flow.

It can be made to work, and therefore it is IMHO correct action to take.

From another perspective, the *existing flow* is the reason which causes this bug. A real fix is to fix the flow itself.

Any existing flow in part of the kernel can have a bug. That does not mean that switching flow would be proper way to fix a bug.

Quoted 6 lines
Quoted 1 line

BR, Jarkko

Yes but I think this is only true when the flow is reasonable. If the flow itself isn't reasonable, we should fix the flow (given it's easy to fix AFAICT).

Anyway, let us also hear from others.

The flow can be made to work without issues, which in the context of a bug fix is exactly what a bug fix should do. Not more or less.

Quoted 6 lines
Quoted 2 lines

You don't gain any measurable value for the user with this switch idea.

And besides this not proper way to review patch anyway because you did not review the code.

I did review the code, but I couldn't agree on the fix. That's why I expressed my view here.

Quoted 4 lines

I'll focus on fix what is broken e.g. so that it
is easy to backport to stable and distro kernels, and call it a day. It certainly does not have to make code "perfect", as long as known bugs are sorted out.

Why cannot the fix which fixes the flow go to stable?

Quoted 5 lines

You are welcome to review the next version of the patch, once I've resolved the issues that were pointed out by Reinette, if you still see some issue but this type of speculative discussion is frankly just wasting everyones time.

Hmm.. Why pointing out a better fix (my perspective of course) is wasting everyone's time?

Signature · 2 lines
Thanks,
-Kai
author maintainer
#11 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 71 lines · Kai
Quoted 51 lines · Jarkko Sakkinen
Quoted 50 lines · On Wed, Aug 31, 2022 at 05:57:22AM +0300, [email protected] wrote:
Quoted 38 lines · Kai
Quoted 37 lines · On Wed, 2022-08-31 at 05:44 +0300, [email protected] wrote:
Quoted 25 lines · Kai
Quoted 24 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 22 lines · Kai
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

Quoted 6 lines
Quoted 3 lines

I would focus fixing the existing flow rather than reinventing the flow.

It can be made to work, and therefore it is IMHO correct action to take.

From another perspective, the *existing flow* is the reason which causes this bug. A real fix is to fix the flow itself.

Any existing flow in part of the kernel can have a bug. That does not mean that switching flow would be proper way to fix a bug.

Quoted 6 lines
Quoted 1 line

BR, Jarkko

Yes but I think this is only true when the flow is reasonable. If the flow itself isn't reasonable, we should fix the flow (given it's easy to fix AFAICT).

Anyway, let us also hear from others.

The flow can be made to work without issues, which in the context of a bug fix is exactly what a bug fix should do. Not more or less.

Quoted 6 lines
Quoted 2 lines

You don't gain any measurable value for the user with this switch idea.

And besides this not proper way to review patch anyway because you did not review the code.

I did review the code, but I couldn't agree on the fix. That's why I expressed my view here.

Quoted 4 lines

I'll focus on fix what is broken e.g. so that it
is easy to backport to stable and distro kernels, and call it a day. It certainly does not have to make code "perfect", as long as known bugs are sorted out.

Why cannot the fix which fixes the flow go to stable?

Quoted 1 line
Quoted 7 lines
Quoted 4 lines

You are welcome to review the next version of the patch, once I've resolved the issues that were pointed out by Reinette, if you still see some issue but this type of speculative discussion is frankly just wasting everyones time.

Hmm.. Why pointing out a better fix (my perspective of course) is wasting everyone's time?

There was not a single inline comment.

BR, Jarkko

#12 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

In reply to [email protected]
Quoted 41 lines · On Wed, 2022-08-31 at 05:57 +0300, [email protected] wrote:
Quoted 40 lines · Kai
Quoted 38 lines · On Wed, 2022-08-31 at 05:44 +0300, [email protected] wrote:
Quoted 25 lines · Kai
Quoted 24 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 22 lines · Kai
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure to sgx_drv_init() won't immediately result in ksgxd to stop prematurally. As long as KVM SGX can be initialized successfully, sgx_init() still returns 0.

Btw I was thinking whether we should move sgx_page_reclaimer_init() to the end of sgx_init(), after we make sure at least one of the driver and the KVM SGX is initialized successfully. Then the code change in this patch won't be necessary if I understand correctly. AFAICT there's no good reason to start the ksgxd at early stage before we are sure either the driver or KVM SGX will work.

Quoted 6 lines
Quoted 3 lines

I would focus fixing the existing flow rather than reinventing the flow.

It can be made to work, and therefore it is IMHO correct action to take.

From another perspective, the *existing flow* is the reason which causes this bug. A real fix is to fix the flow itself.

Any existing flow in part of the kernel can have a bug. That does not mean that switching flow would be proper way to fix a bug.

BR, Jarkko

Quoted 8 lines
Quoted 4 lines

Yes but I think this is only true when the flow is reasonable. If the flow itself isn't reasonable, we should fix the flow (given it's easy to fix AFAICT).

Anyway, let us also hear from others.

The flow can be made to work without issues, which in the context of a bug fix is exactly what a bug fix should do. Not more or less.

No. To me the flow itself is buggy. There's no reason to start ksgxd() before at least SGX driver is initialized to work.

Patching the buggy flow is more like a workaround, but isn't a real fix.

Quoted 3 lines

You don't gain any measurable value for the user with this switch idea.

There is actual gain by moving sgx_page_reclaimer_init() to sgx_drv_init(), or only calling sgx_page_reclaimer_init() when sgx_drv_init() returns success:

If somehow sgx_drv_init() fails to initialize, ksgxd() won't run.

Currently, if SGX driver fails to initialize but virtual EPC initializes successfully, ksgxd() still runs. However it achieves nothing but only wastes CPU cycles.

Signature · 2 lines
Thanks,
-Kai
#13 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Hi Kai

Quoted 61 lines · Kai
Quoted 60 lines · On Wed, 2022-08-31 at 05:57 +0300, [email protected] wrote:
Quoted 9 lines · Kai
Quoted 8 lines · On Wed, 2022-08-31 at 05:44 +0300, [email protected] wrote:
Quoted 7 lines · Kai
Quoted 6 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 5 lines · Kai
Quoted 4 lines · Reinette Chatre

Hi Jarkko,

Quoted 1 line · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision

device fails, and

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

neither sgx_drv_init() nor sgx_vepc_init() succeeds, then

ksgxd will be

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

prematurely stopped.

I do not think misc_register() is required to fail for the

scenario to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is

started

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

(via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance, failure

to sgx_drv_init()

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

won't immediately result in ksgxd to stop prematurally. As

long as KVM SGX can

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

be initialized successfully, sgx_init() still returns 0.

Btw I was thinking whether we should move

sgx_page_reclaimer_init() to the end

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

of sgx_init(), after we make sure at least one of the driver

and the KVM SGX is

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

initialized successfully. Then the code change in this patch

won't be necessary

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

if I understand correctly. AFAICT there's no good reason to

start the ksgxd at

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

early stage before we are sure either the driver or KVM SGX

will work.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I would focus fixing the existing flow rather than reinventing

the flow.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

It can be made to work, and therefore it is IMHO correct action

to take.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

From another perspective, the *existing flow* is the reason which

causes this

Quoted 9 lines
Quoted 7 lines
Quoted 1 line

bug. A real fix is to fix the flow itself.

Any existing flow in part of the kernel can have a bug. That does not mean that switching flow would be proper way to fix a bug.

BR, Jarkko

Yes but I think this is only true when the flow is reasonable. If

the flow

Quoted 1 line

itself isn't reasonable, we should fix the flow (given it's easy to

fix AFAICT).

Quoted 1 line
Quoted 9 lines
Quoted 5 lines
Quoted 1 line

Anyway, let us also hear from others.

The flow can be made to work without issues, which in the context of a bug fix is exactly what a bug fix should do. Not more or less.

No. To me the flow itself is buggy. There's no reason to start ksgxd() before at least SGX driver is initialized to work.

Will it cause racing if we expose dev nodes to user space before ksgxd is started and sensitization done?

Quoted 6 lines

Patching the buggy flow is more like a workaround, but isn't a real fix.

Quoted 3 lines

You don't gain any measurable value for the user with this switch idea.

Quoted 10 lines

There is actual gain by moving sgx_page_reclaimer_init() to sgx_drv_init(), or only calling sgx_page_reclaimer_init() when sgx_drv_init() returns success:

If somehow sgx_drv_init() fails to initialize, ksgxd() won't run.

Currently, if SGX driver fails to initialize but virtual EPC initializes successfully, ksgxd() still runs. However it achieves nothing but only wastes CPU cycles.

You still need ksgxd for sanitizing (at least) and swapping (potentially) even if only virtual EPC initializes.

Thanks
Haitao

author maintainer
#14 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 70 lines · Haitao Huang

Hi Kai

Quoted 67 lines · Kai
Quoted 65 lines · On Wed, 2022-08-31 at 05:57 +0300, [email protected] wrote:
Quoted 9 lines · Kai
Quoted 8 lines · On Wed, 2022-08-31 at 05:44 +0300, [email protected] wrote:
Quoted 7 lines · Kai
Quoted 6 lines · On Wed, 2022-08-31 at 05:15 +0300, [email protected] wrote:
Quoted 5 lines · Kai
Quoted 4 lines · Reinette Chatre

Hi Jarkko,

Quoted 1 line · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision

device fails, and

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

neither sgx_drv_init() nor sgx_vepc_init() succeeds,

then ksgxd will be

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

prematurely stopped.

I do not think misc_register() is required to fail for

the scenario to

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd

is started

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line

(via sgx_page_reclaimer_init()) ...".

IMHO "a failure" might be too vague. For instance,

failure to sgx_drv_init()

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

won't immediately result in ksgxd to stop prematurally.

As long as KVM SGX can

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

be initialized successfully, sgx_init() still returns 0.

Btw I was thinking whether we should move

sgx_page_reclaimer_init() to the end

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

of sgx_init(), after we make sure at least one of the

driver and the KVM SGX is

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

initialized successfully. Then the code change in this

patch won't be necessary

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

if I understand correctly. AFAICT there's no good reason

to start the ksgxd at

Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

early stage before we are sure either the driver or KVM

SGX will work.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

I would focus fixing the existing flow rather than

reinventing the flow.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

It can be made to work, and therefore it is IMHO correct

action to take.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

From another perspective, the *existing flow* is the reason

which causes this

Quoted 9 lines
Quoted 7 lines
Quoted 1 line

bug. A real fix is to fix the flow itself.

Any existing flow in part of the kernel can have a bug. That does not mean that switching flow would be proper way to fix a bug.

BR, Jarkko

Yes but I think this is only true when the flow is reasonable. If

the flow

Quoted 1 line

itself isn't reasonable, we should fix the flow (given it's easy

to fix AFAICT).

Quoted 2 lines

Anyway, let us also hear from others.

The flow can be made to work without issues, which in the context of a bug fix is exactly what a bug fix should do. Not more or less.

Quoted 7 lines
Quoted 4 lines

No. To me the flow itself is buggy. There's no reason to start ksgxd() before
at least SGX driver is initialized to work.

Will it cause racing if we expose dev nodes to user space before ksgxd is started and sensitization done?

I'll to explain this.

So the point is to fix the issue at hand, and fix it locally.

Changing initialization order is simply out of context. It's not really an argument for or against changing it

We are fixing sanitization here, and only that with zero side-effects to any other semantics.

It's dictated by the development process [*] but more importantly it's also just plain common sense.

[*] https://www.kernel.org/doc/html/v5.19/process/submitting-patches.html#separate-your-changes

BR, Jarkko

#15 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Jarkko, Kai and Haitao,

Can you three please start trimming your replies? You don't need to and should not quote the entirety of your messages every time you reply.

Quoted 6 lines · On 8/31/22 11:28, [email protected] wrote:
Quoted 2 lines

Will it cause racing if we expose dev nodes to user space before ksgxd is started and sensitization done?

I'll to explain this.

So the point is to fix the issue at hand, and fix it locally.

Quoted 8 lines

Changing initialization order is simply out of context. It's not really an argument for or against changing it

We are fixing sanitization here, and only that with zero side-effects to any other semantics.

It's dictated by the development process [*] but more importantly it's also just plain common sense.

Kai, I think your suggestion is reasonable. You make a good point about not needing ksgxd for vepc.

*But*, I think it's a bit too much for a bugfix that's headed to -stable. I'm concerned that it will have unintended side effects, *especially* when there's a working, tested alternative.

author maintainer
#16 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 21 lines · Dave Hansen

Jarkko, Kai and Haitao,

Can you three please start trimming your replies? You don't need to and should not quote the entirety of your messages every time you reply.

Quoted 6 lines · On 8/31/22 11:28, [email protected] wrote:
Quoted 2 lines

Will it cause racing if we expose dev nodes to user space before ksgxd is started and sensitization done?

I'll to explain this.

So the point is to fix the issue at hand, and fix it locally.

Quoted 8 lines

Changing initialization order is simply out of context. It's not really an argument for or against changing it

We are fixing sanitization here, and only that with zero side-effects to any other semantics.

It's dictated by the development process [*] but more importantly it's also just plain common sense.

Quoted 6 lines

Kai, I think your suggestion is reasonable. You make a good point about not needing ksgxd for vepc.

*But*, I think it's a bit too much for a bugfix that's headed to -stable. I'm concerned that it will have unintended side effects, *especially* when there's a working, tested alternative.

Yeah, I also actually *do* agree that the suggestions could be reasonable.

BR, Jarkko

author maintainer
#17 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

In reply to Dave Hansen
Quoted 4 lines · Dave Hansen

Jarkko, Kai and Haitao,

Can you three please start trimming your replies? You don't need to and should not quote the entirety of your messages every time you reply.

Sure, sorry about that.

BR, Jarkko

#18 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

In reply to Dave Hansen
Quoted 21 lines · Dave Hansen

Jarkko, Kai and Haitao,

Can you three please start trimming your replies? You don't need to and should not quote the entirety of your messages every time you reply.

Quoted 6 lines · On 8/31/22 11:28, [email protected] wrote:
Quoted 2 lines

Will it cause racing if we expose dev nodes to user space before ksgxd is started and sensitization done?

I'll to explain this.

So the point is to fix the issue at hand, and fix it locally.

Quoted 8 lines

Changing initialization order is simply out of context. It's not really an argument for or against changing it

We are fixing sanitization here, and only that with zero side-effects to any other semantics.

It's dictated by the development process [*] but more importantly it's also just plain common sense.

Quoted 6 lines

Kai, I think your suggestion is reasonable. You make a good point about not needing ksgxd for vepc.

*But*, I think it's a bit too much for a bugfix that's headed to -stable. I'm concerned that it will have unintended side effects, *especially* when there's a working, tested alternative.

Agreed. Thanks Dave/Jarkko.

Signature · 2 lines
Thanks,
-Kai
author maintainer
#19 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 22 lines · Kai
Quoted 21 lines · Dave Hansen

Jarkko, Kai and Haitao,

Can you three please start trimming your replies? You don't need to and should not quote the entirety of your messages every time you reply.

Quoted 14 lines · On 8/31/22 11:28, [email protected] wrote:
Quoted 2 lines

Will it cause racing if we expose dev nodes to user space before ksgxd is started and sensitization done?

I'll to explain this.

So the point is to fix the issue at hand, and fix it locally.

Changing initialization order is simply out of context. It's not really an argument for or against changing it

We are fixing sanitization here, and only that with zero side-effects to any other semantics.

It's dictated by the development process [*] but more importantly it's also just plain common sense.

Quoted 8 lines
Quoted 6 lines

Kai, I think your suggestion is reasonable. You make a good point about not needing ksgxd for vepc.

*But*, I think it's a bit too much for a bugfix that's headed to -stable. I'm concerned that it will have unintended side effects, *especially* when there's a working, tested alternative.

Agreed. Thanks Dave/Jarkko.

Please do a patch. It's a very reasonable suggestion when considered out of context of this bug.

If you go really rigid with this, the compilation process should not compile in sanitization process in the case when only vepc is enabled. It's useless functionality in that case.

BR, Jarkko

#20 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 35 lines · On Fri, 2022-09-02 at 01:27 +0300, [email protected] wrote:
Quoted 22 lines · Kai
Quoted 21 lines · Dave Hansen

Jarkko, Kai and Haitao,

Can you three please start trimming your replies? You don't need to and should not quote the entirety of your messages every time you reply.

Quoted 14 lines · On 8/31/22 11:28, [email protected] wrote:
Quoted 2 lines

Will it cause racing if we expose dev nodes to user space before ksgxd is started and sensitization done?

I'll to explain this.

So the point is to fix the issue at hand, and fix it locally.

Changing initialization order is simply out of context. It's not really an argument for or against changing it

We are fixing sanitization here, and only that with zero side-effects to any other semantics.

It's dictated by the development process [*] but more importantly it's also just plain common sense.

Quoted 8 lines
Quoted 6 lines

Kai, I think your suggestion is reasonable. You make a good point about not needing ksgxd for vepc.

*But*, I think it's a bit too much for a bugfix that's headed to -stable. I'm concerned that it will have unintended side effects, *especially* when there's a working, tested alternative.

Agreed. Thanks Dave/Jarkko.

Please do a patch. It's a very reasonable suggestion when considered out of context of this bug.

Quoted 6 lines

If you go really rigid with this, the compilation process should not compile in sanitization process in the case when only vepc is enabled. It's useless functionality in that case.

BR, Jarkko

Yeah I am planning to work out one to see how it goes.

Signature · 2 lines
Thanks,
-Kai
author maintainer
#21 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 36 lines · Kai
Quoted 35 lines · On Fri, 2022-09-02 at 01:27 +0300, [email protected] wrote:
Quoted 22 lines · Kai
Quoted 21 lines · Dave Hansen

Jarkko, Kai and Haitao,

Can you three please start trimming your replies? You don't need to and should not quote the entirety of your messages every time you reply.

Quoted 14 lines · On 8/31/22 11:28, [email protected] wrote:
Quoted 2 lines

Will it cause racing if we expose dev nodes to user space before ksgxd is started and sensitization done?

I'll to explain this.

So the point is to fix the issue at hand, and fix it locally.

Changing initialization order is simply out of context. It's not really an argument for or against changing it

We are fixing sanitization here, and only that with zero side-effects to any other semantics.

It's dictated by the development process [*] but more importantly it's also just plain common sense.

Quoted 8 lines
Quoted 6 lines

Kai, I think your suggestion is reasonable. You make a good point about not needing ksgxd for vepc.

*But*, I think it's a bit too much for a bugfix that's headed to -stable. I'm concerned that it will have unintended side effects, *especially* when there's a working, tested alternative.

Agreed. Thanks Dave/Jarkko.

Please do a patch. It's a very reasonable suggestion when considered out of context of this bug.

Quoted 8 lines
Quoted 6 lines

If you go really rigid with this, the compilation process should not compile in sanitization process in the case when only vepc is enabled. It's useless functionality in that case.

BR, Jarkko

Yeah I am planning to work out one to see how it goes.

Looking forward to try it out :-)

BR, Jarkko

#22 lore raw

RE: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 4 lines
Quoted 2 lines

Yeah I am planning to work out one to see how it goes.

Looking forward to try it out :-)

Just let you know I'll work on it perhaps late next week or the week after, since I have other higher priority tasks to do. :)

Thanks,
-Kai

author maintainer
#23 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

In reply to Reinette Chatre
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

This would be the fixed version of the sentence:

"
In sgx_init(), if misc_register() fails or misc_register() succeeds but neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped. This may leave some unsanitized pages, which does not matter, because SGX will be disabled for the whole power cycle. "

I want to keep the end states listed and not make it more abstract.

The second sentence addresses the remark below.

Quoted 5 lines

To help the reader understand the subject of this patch it may help to explain that prematurely stopping ksgxd may leave some unsanitized pages, but that is not a problem since SGX cannot be used on the platform anyway.

Quoted 10 lines
Quoted 3 lines

This triggers WARN_ON() because sgx_dirty_page_list ends up being non-empty, and dumps the call stack:

Traces like below can be frowned upon. I recommend that you follow the guidance in "Backtraces in commit mesages"(sic) in Documentation/process/submitting-patches.rst.

Quoted 2 lines
[    0.268592] WARNING: CPU: 6 PID: 83 at

arch/x86/kernel/cpu/sgx/main.c:401 ksgxd+0x1b7/0x1d0

Is this good enough? I had not actually spotted this section before but nice that it exists. Apparently has been added in 5.12.

Quoted 10 lines
Quoted 8 lines
Quoted 1 line

Ultimately this can crash the kernel, if the following is set:

/proc/sys/kernel/panic_on_warn

Print a simple warning instead, and improve the output by printing the number of unsanitized pages, in order to provide debug informnation for future needs.

informnation -> information

+1

Quoted 11 lines

...

Quoted 5 lines
Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list")

Should this go to stable?

I guess it should. The hard reason for this that it can panic the kernel.

Quoted 16 lines

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 54

Quoted 13 lines
arch/x86/kernel/cpu/sgx/main.c +3 -0
index 515e2a5f25bb..903100fcfce3 100644
@@ -49,17 +49,20 @@ static LIST_HEAD(sgx_dirty_page_list);
  * Reset post-kexec EPC pages to the uninitialized state. The pages are removed
  * from the input list, and made available for the page allocator. SECS pages
  * prepending their children in the input list are left intact.
+ *
+ * Contents of the @dirty_page_list must be thread-local, i.e.
+ * not shared by multiple threads.

Did you intend to mention something about the needed locking here? It looks like some information is lost during the move to the function description.

Nothing about the locking that concerns the parameter, as the sentence defines clear constraints for the caller.

Quoted 11 lines

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 59

Quoted 7 lines
arch/x86/kernel/cpu/sgx/main.c +2 -1
  */
-static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
+static int __sgx_sanitize_pages(struct list_head *dirty_page_list)
 {
 	struct sgx_epc_page *page;
+	int left_dirty = 0;

I do not know how many pages this code should be ready for but at least this could handle more by being an unsigned int considering that it is always positive ... maybe even unsigned long?

I would go for 'long'. More information below.

Quoted 48 lines
Quoted 48 lines
arch/x86/kernel/cpu/sgx/main.c +8 -5
 	LIST_HEAD(dirty);
 	int ret;
 
-	/* dirty_page_list is thread-local, no need for a lock: */
 	while (!list_empty(dirty_page_list)) {
 		if (kthread_should_stop())
-			return;
+			break;
 
 		page = list_first_entry(dirty_page_list, struct sgx_epc_page, list);
 
@@ -92,12 +95,14 @@ static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
 		} else {
 			/* The page is not yet clean - move to the dirty list. */
 			list_move_tail(&page->list, &dirty);
+			left_dirty++;
 		}
 
 		cond_resched();
 	}
 
 	list_splice(&dirty, dirty_page_list);
+	return left_dirty;
 }
 
 static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)
@@ -388,6 +393,8 @@ void sgx_reclaim_direct(void)
 
 static int ksgxd(void *p)
 {
+	int left_dirty;
+
 	set_freezable();
 
 	/*
@@ -395,10 +402,10 @@ static int ksgxd(void *p)
 	 * required for SECS pages, whose child pages blocked EREMOVE.
 	 */
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
-	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	left_dirty = __sgx_sanitize_pages(&sgx_dirty_page_list);
+	if (left_dirty)
+		pr_warn("%d unsanitized pages\n", left_dirty);
 
Quoted 5 lines

arch/x86/kernel/cpu/sgx/main.c static int ksgxd(void *p) · line 411

Quoted 2 lines
arch/x86/kernel/cpu/sgx/main.c +0 -0
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())

Reinette

We need to return -ECANCELED on premature stop, and number of pages otherwise.

In premature stop, nothing should be printed, as the number is by practical means a random number. Otherwise, it is an indicator of a bug in the driver, and therefore a non-zero number should be printed pr_err(), if that happens after the second call.

Thanks for feedback.

BR, Jarkko

author maintainer
#24 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 161 lines · Jarkko Sakkinen
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

This would be the fixed version of the sentence:

"
In sgx_init(), if misc_register() fails or misc_register() succeeds but neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped. This may leave some unsanitized pages, which does not matter, because SGX will be disabled for the whole power cycle. "

I want to keep the end states listed and not make it more abstract.

The second sentence addresses the remark below.

Quoted 5 lines

To help the reader understand the subject of this patch it may help to explain that prematurely stopping ksgxd may leave some unsanitized pages, but that is not a problem since SGX cannot be used on the platform anyway.

Quoted 10 lines
Quoted 3 lines

This triggers WARN_ON() because sgx_dirty_page_list ends up being non-empty, and dumps the call stack:

Traces like below can be frowned upon. I recommend that you follow the guidance in "Backtraces in commit mesages"(sic) in Documentation/process/submitting-patches.rst.

Quoted 2 lines
[    0.268592] WARNING: CPU: 6 PID: 83 at

arch/x86/kernel/cpu/sgx/main.c:401 ksgxd+0x1b7/0x1d0

Is this good enough? I had not actually spotted this section before but nice that it exists. Apparently has been added in 5.12.

Quoted 10 lines
Quoted 8 lines
Quoted 1 line

Ultimately this can crash the kernel, if the following is set:

/proc/sys/kernel/panic_on_warn

Print a simple warning instead, and improve the output by printing the number of unsanitized pages, in order to provide debug informnation for future needs.

informnation -> information

+1

Quoted 11 lines

...

Quoted 5 lines
Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list")

Should this go to stable?

I guess it should. The hard reason for this that it can panic the kernel.

Quoted 16 lines

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 54

Quoted 13 lines
arch/x86/kernel/cpu/sgx/main.c +3 -0
index 515e2a5f25bb..903100fcfce3 100644
@@ -49,17 +49,20 @@ static LIST_HEAD(sgx_dirty_page_list);
  * Reset post-kexec EPC pages to the uninitialized state. The pages are removed
  * from the input list, and made available for the page allocator. SECS pages
  * prepending their children in the input list are left intact.
+ *
+ * Contents of the @dirty_page_list must be thread-local, i.e.
+ * not shared by multiple threads.

Did you intend to mention something about the needed locking here? It looks like some information is lost during the move to the function description.

Nothing about the locking that concerns the parameter, as the sentence defines clear constraints for the caller.

Quoted 11 lines

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 59

Quoted 7 lines
arch/x86/kernel/cpu/sgx/main.c +2 -1
  */
-static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
+static int __sgx_sanitize_pages(struct list_head *dirty_page_list)
 {
 	struct sgx_epc_page *page;
+	int left_dirty = 0;

I do not know how many pages this code should be ready for but at least this could handle more by being an unsigned int considering that it is always positive ... maybe even unsigned long?

I would go for 'long'. More information below.

Quoted 48 lines
Quoted 48 lines
arch/x86/kernel/cpu/sgx/main.c +8 -5
 	LIST_HEAD(dirty);
 	int ret;
 
-	/* dirty_page_list is thread-local, no need for a lock: */
 	while (!list_empty(dirty_page_list)) {
 		if (kthread_should_stop())
-			return;
+			break;
 
 		page = list_first_entry(dirty_page_list, struct sgx_epc_page, list);
 
@@ -92,12 +95,14 @@ static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
 		} else {
 			/* The page is not yet clean - move to the dirty list. */
 			list_move_tail(&page->list, &dirty);
+			left_dirty++;
 		}
 
 		cond_resched();
 	}
 
 	list_splice(&dirty, dirty_page_list);
+	return left_dirty;
 }
 
 static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)
@@ -388,6 +393,8 @@ void sgx_reclaim_direct(void)
 
 static int ksgxd(void *p)
 {
+	int left_dirty;
+
 	set_freezable();
 
 	/*
@@ -395,10 +402,10 @@ static int ksgxd(void *p)
 	 * required for SECS pages, whose child pages blocked EREMOVE.
 	 */
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
-	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	left_dirty = __sgx_sanitize_pages(&sgx_dirty_page_list);
+	if (left_dirty)
+		pr_warn("%d unsanitized pages\n", left_dirty);
 
Quoted 5 lines

arch/x86/kernel/cpu/sgx/main.c static int ksgxd(void *p) · line 411

Quoted 2 lines
arch/x86/kernel/cpu/sgx/main.c +0 -0
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())

Reinette

Quoted 8 lines

We need to return -ECANCELED on premature stop, and number of pages otherwise.

In premature stop, nothing should be printed, as the number is by practical means a random number. Otherwise, it is an indicator of a bug in the driver, and therefore a non-zero number should be printed pr_err(), if that happens after the second call.

I.e. even though we print less we get more *information* what is going inside the kernel. Warning is not correct for either path IMHO.

BR, Jarkko

author maintainer
#25 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

Quoted 171 lines · Jarkko Sakkinen
Quoted 161 lines · Jarkko Sakkinen
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

This would be the fixed version of the sentence:

"
In sgx_init(), if misc_register() fails or misc_register() succeeds but neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped. This may leave some unsanitized pages, which does not matter, because SGX will be disabled for the whole power cycle. "

I want to keep the end states listed and not make it more abstract.

The second sentence addresses the remark below.

Quoted 5 lines

To help the reader understand the subject of this patch it may help to explain that prematurely stopping ksgxd may leave some unsanitized pages, but that is not a problem since SGX cannot be used on the platform anyway.

Quoted 10 lines
Quoted 3 lines

This triggers WARN_ON() because sgx_dirty_page_list ends up being non-empty, and dumps the call stack:

Traces like below can be frowned upon. I recommend that you follow the guidance in "Backtraces in commit mesages"(sic) in Documentation/process/submitting-patches.rst.

Quoted 2 lines
[    0.268592] WARNING: CPU: 6 PID: 83 at

arch/x86/kernel/cpu/sgx/main.c:401 ksgxd+0x1b7/0x1d0

Is this good enough? I had not actually spotted this section before but nice that it exists. Apparently has been added in 5.12.

Quoted 10 lines
Quoted 8 lines
Quoted 1 line

Ultimately this can crash the kernel, if the following is set:

/proc/sys/kernel/panic_on_warn

Print a simple warning instead, and improve the output by printing the number of unsanitized pages, in order to provide debug informnation for future needs.

informnation -> information

+1

Quoted 11 lines

...

Quoted 5 lines
Fixes 51ab30eb2ad4 ("x86/sgx: Replace section->init_laundry_list with sgx_dirty_page_list")

Should this go to stable?

I guess it should. The hard reason for this that it can panic the kernel.

Quoted 16 lines

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 54

Quoted 13 lines
arch/x86/kernel/cpu/sgx/main.c +3 -0
index 515e2a5f25bb..903100fcfce3 100644
@@ -49,17 +49,20 @@ static LIST_HEAD(sgx_dirty_page_list);
  * Reset post-kexec EPC pages to the uninitialized state. The pages are removed
  * from the input list, and made available for the page allocator. SECS pages
  * prepending their children in the input list are left intact.
+ *
+ * Contents of the @dirty_page_list must be thread-local, i.e.
+ * not shared by multiple threads.

Did you intend to mention something about the needed locking here? It looks like some information is lost during the move to the function description.

Nothing about the locking that concerns the parameter, as the sentence defines clear constraints for the caller.

Quoted 11 lines

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 59

Quoted 7 lines
arch/x86/kernel/cpu/sgx/main.c +2 -1
  */
-static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
+static int __sgx_sanitize_pages(struct list_head *dirty_page_list)
 {
 	struct sgx_epc_page *page;
+	int left_dirty = 0;

I do not know how many pages this code should be ready for but at least this could handle more by being an unsigned int considering that it is always positive ... maybe even unsigned long?

I would go for 'long'. More information below.

Quoted 48 lines
Quoted 48 lines
arch/x86/kernel/cpu/sgx/main.c +8 -5
 	LIST_HEAD(dirty);
 	int ret;
 
-	/* dirty_page_list is thread-local, no need for a lock: */
 	while (!list_empty(dirty_page_list)) {
 		if (kthread_should_stop())
-			return;
+			break;
 
 		page = list_first_entry(dirty_page_list, struct sgx_epc_page, list);
 
@@ -92,12 +95,14 @@ static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
 		} else {
 			/* The page is not yet clean - move to the dirty list. */
 			list_move_tail(&page->list, &dirty);
+			left_dirty++;
 		}
 
 		cond_resched();
 	}
 
 	list_splice(&dirty, dirty_page_list);
+	return left_dirty;
 }
 
 static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)
@@ -388,6 +393,8 @@ void sgx_reclaim_direct(void)
 
 static int ksgxd(void *p)
 {
+	int left_dirty;
+
 	set_freezable();
 
 	/*
@@ -395,10 +402,10 @@ static int ksgxd(void *p)
 	 * required for SECS pages, whose child pages blocked EREMOVE.
 	 */
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
-	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	left_dirty = __sgx_sanitize_pages(&sgx_dirty_page_list);
+	if (left_dirty)
+		pr_warn("%d unsanitized pages\n", left_dirty);
 
Quoted 5 lines

arch/x86/kernel/cpu/sgx/main.c static int ksgxd(void *p) · line 411

Quoted 2 lines
arch/x86/kernel/cpu/sgx/main.c +0 -0
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())

Reinette

Quoted 8 lines

We need to return -ECANCELED on premature stop, and number of pages otherwise.

In premature stop, nothing should be printed, as the number is by practical means a random number. Otherwise, it is an indicator of a bug in the driver, and therefore a non-zero number should be printed pr_err(), if that happens after the second call.

Quoted 3 lines

I.e. even though we print less we get more *information* what is going inside the kernel. Warning is not correct for either path IMHO.

Oh, sorry, I forgot one thing. The devices should be actually deinitialized in the error case. We do not want to leave a broken driver running.

BR, Jarkko

#26 lore raw

Re: [PATCH 1/6] x86/sgx: Do not consider unsanitized pages an error

In reply to Jarkko Sakkinen

Hi Jarkko,

Quoted 22 lines · Jarkko Sakkinen
Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 3 lines · Jarkko Sakkinen

In sgx_init(), if misc_register() for the provision device fails, and neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped.

I do not think misc_register() is required to fail for the scenario to be triggered (rather use "or" than "and"?). Perhaps just "In sgx_init(), if a failure is encountered after ksgxd is started (via sgx_page_reclaimer_init()) ...".

This would be the fixed version of the sentence:

"
In sgx_init(), if misc_register() fails or misc_register() succeeds but neither sgx_drv_init() nor sgx_vepc_init() succeeds, then ksgxd will be prematurely stopped. This may leave some unsanitized pages, which does not matter, because SGX will be disabled for the whole power cycle. "

Quoted 3 lines

I want to keep the end states listed and not make it more abstract.

The second sentence addresses the remark below.

Thank you for capturing this. It sounds good.

Quoted 14 lines
Quoted 14 lines

To help the reader understand the subject of this patch it may help to explain that prematurely stopping ksgxd may leave some unsanitized pages, but that is not a problem since SGX cannot be used on the platform anyway.

Quoted 3 lines

This triggers WARN_ON() because sgx_dirty_page_list ends up being non-empty, and dumps the call stack:

Traces like below can be frowned upon. I recommend that you follow the guidance in "Backtraces in commit mesages"(sic) in Documentation/process/submitting-patches.rst.

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines
[    0.268592] WARNING: CPU: 6 PID: 83 at

arch/x86/kernel/cpu/sgx/main.c:401 ksgxd+0x1b7/0x1d0

Is this good enough? I had not actually spotted this section before but nice that it exists. Apparently has been added in 5.12.

The timestamp should be removed, it is among the things listed as "distracting information". In this case the backtrace and registers within the trace are not adding value but I think it is important to mention the kernel version somewhere for folks to be able to interpret the line number provided.

Yet I see you kept the whole trace in V2 ?

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
arch/x86/kernel/cpu/sgx/main.c +0 -0
index 515e2a5f25bb..903100fcfce3 100644
@@ -49,17 +49,20 @@ static LIST_HEAD(sgx_dirty_page_list);
  * Reset post-kexec EPC pages to the uninitialized state. The pages are removed
Quoted 8 lines

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 54

Quoted 5 lines
arch/x86/kernel/cpu/sgx/main.c +3 -0
  * from the input list, and made available for the page allocator. SECS pages
  * prepending their children in the input list are left intact.
+ *
+ * Contents of the @dirty_page_list must be thread-local, i.e.
+ * not shared by multiple threads.

Did you intend to mention something about the needed locking here? It looks like some information is lost during the move to the function description.

Quoted 2 lines

Nothing about the locking that concerns the parameter, as the sentence defines clear constraints for the caller.

ok

Quoted 69 lines
Quoted 12 lines

arch/x86/kernel/cpu/sgx/main.c static LIST_HEAD(sgx_dirty_page_list); · line 59

Quoted 8 lines
arch/x86/kernel/cpu/sgx/main.c +2 -1
  */
-static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
+static int __sgx_sanitize_pages(struct list_head *dirty_page_list)
 {
 	struct sgx_epc_page *page;
+	int left_dirty = 0;

I do not know how many pages this code should be ready for but at least this could handle more by being an unsigned int considering that it is always positive ... maybe even unsigned long?

I would go for 'long'. More information below.

Quoted 48 lines
Quoted 48 lines
arch/x86/kernel/cpu/sgx/main.c +8 -5
 	LIST_HEAD(dirty);
 	int ret;
 
-	/* dirty_page_list is thread-local, no need for a lock: */
 	while (!list_empty(dirty_page_list)) {
 		if (kthread_should_stop())
-			return;
+			break;
 
 		page = list_first_entry(dirty_page_list, struct sgx_epc_page, list);
 
@@ -92,12 +95,14 @@ static void __sgx_sanitize_pages(struct list_head *dirty_page_list)
 		} else {
 			/* The page is not yet clean - move to the dirty list. */
 			list_move_tail(&page->list, &dirty);
+			left_dirty++;
 		}
 
 		cond_resched();
 	}
 
 	list_splice(&dirty, dirty_page_list);
+	return left_dirty;
 }
 
 static bool sgx_reclaimer_age(struct sgx_epc_page *epc_page)
@@ -388,6 +393,8 @@ void sgx_reclaim_direct(void)
 
 static int ksgxd(void *p)
 {
+	int left_dirty;
+
 	set_freezable();
 
 	/*
@@ -395,10 +402,10 @@ static int ksgxd(void *p)
 	 * required for SECS pages, whose child pages blocked EREMOVE.
 	 */
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
-	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
-	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	left_dirty = __sgx_sanitize_pages(&sgx_dirty_page_list);
+	if (left_dirty)
+		pr_warn("%d unsanitized pages\n", left_dirty);
 
Quoted 5 lines

arch/x86/kernel/cpu/sgx/main.c static int ksgxd(void *p) · line 411

Quoted 2 lines
arch/x86/kernel/cpu/sgx/main.c +0 -0
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())

Reinette

Quoted 9 lines

We need to return -ECANCELED on premature stop, and number of pages otherwise.

In premature stop, nothing should be printed, as the number is by practical means a random number. Otherwise, it is an indicator of a bug in the driver, and therefore a non-zero number should be printed pr_err(), if that happens after the second call.

Good point.

Quoted 3 lines

Oh, sorry, I forgot one thing. The devices should be actually deinitialized in the error case. We do not want to leave a broken driver running.

Is this not already done on the error path of sgx_init()?

Reinette

author maintainer
#27 lore raw

PATCH 2/6 x86/sgx: Handle VA page allocation failure for EAUG on PF.

In reply to Jarkko Sakkinen

From: Haitao Huang <[email protected]>

VM_FAULT_NOPAGE is expected behaviour for -EBUSY failure path, when augmenting a page, as this means that the reclaimer thread has been triggered, and the intention is just to round-trip in ring-3, and retry with a new page fault.

Fixes 5a90d2c3f5ef ("x86/sgx: Support adding of pages to an initialized enclave")

---
v2:
* Removed reviewed-by, no other changes.

---
 arch/x86/kernel/cpu/sgx/encl.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/encl.c +4 -1
index 24c1bb8eb196..de92c1c8b79d 100644
@@ -344,8 +344,11 @@ static vm_fault_t sgx_encl_eaug_page(struct vm_area_struct *vma,
 	}
 
 	va_page = sgx_encl_grow(encl, false);
-	if (IS_ERR(va_page))
+	if (IS_ERR(va_page)) {
+		if (PTR_ERR(va_page) == -EBUSY)
+			vmret =  VM_FAULT_NOPAGE;
 		goto err_out_epc;
+	}
 
 	if (va_page)
 		list_add(&va_page->list, &encl->va_pages);
Signature · 1 line
2.37.2
Reviewed-by on v1 2/6
#28 lore raw

Re: [PATCH 2/6] x86/sgx: Handle VA page allocation failure for EAUG on PF.

Hi Jarkko and Haitao,

Quoted 6 lines · Jarkko Sakkinen

From: Haitao Huang <[email protected]>

VM_FAULT_NOPAGE is expected behaviour for -EBUSY failure path, when augmenting a page, as this means that the reclaimer thread has been triggered, and the intention is just to round-trip in ring-3, and retry with a new page fault.

Thank you very much for finding and fixing this issue.

Quoted 17 lines
Fixes 5a90d2c3f5ef ("x86/sgx: Support adding of pages to an initialized enclave")

---
v2:
* Removed reviewed-by, no other changes.

---
 arch/x86/kernel/cpu/sgx/encl.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/encl.c +0 -0
index 24c1bb8eb196..de92c1c8b79d 100644

arch/x86/kernel/cpu/sgx/encl.c static vm_fault_t sgx_encl_eaug_page(struct vm_area_struct *vma, · line 349

Quoted 7 lines
arch/x86/kernel/cpu/sgx/encl.c +3 -1
@@ -344,8 +344,11 @@ static vm_fault_t sgx_encl_eaug_page(struct vm_area_struct *vma,
 	}
 
 	va_page = sgx_encl_grow(encl, false);
-	if (IS_ERR(va_page))
+	if (IS_ERR(va_page)) {
+		if (PTR_ERR(va_page) == -EBUSY)
+			vmret =  VM_FAULT_NOPAGE;

Some unnecessary white space remained after the copy of this snippet. No need to propagate that mistake.

arch/x86/kernel/cpu/sgx/encl.c static vm_fault_t sgx_encl_eaug_page(struct vm_area_struct *vma, · line 354

Quoted 5 lines
arch/x86/kernel/cpu/sgx/encl.c +1 -0
 		goto err_out_epc;
+	}
 
 	if (va_page)
 		list_add(&va_page->list, &encl->va_pages);

Apart from the white space nitpick this looks good.

Thank you very much.

Reinette

author maintainer
#29 lore raw

PATCH 3/6 selftests/sgx: Ignore OpenSSL 3.0 deprecated functions warning

In reply to Jarkko Sakkinen

From: Kristen Carlson Accardi <[email protected]>

OpenSSL 3.0 deprecates some of the functions used in the SGX selftests, causing build errors on new distros. For now ignore the warnings until support for the functions is no longer available and mark FIXME so that it can be clear this should be removed at some point.

---
 tools/testing/selftests/sgx/sigstruct.c | 6 ++++++
 1 file changed, 6 insertions(+)
tools/testing/selftests/sgx/sigstruct.c +6 -0
index 50c5ab1aa6fa..a07896a46364 100644
@@ -17,6 +17,12 @@
 #include "defines.h"
 #include "main.h"
 
+/*
+ * FIXME: OpenSSL 3.0 has deprecated some functions. For now just ignore
+ * the warnings.
+ */
+#pragma GCC diagnostic ignored "-Wdeprecated-declarations"
+
 struct q1q2_ctx {
 	BN_CTX *bn_ctx;
 	BIGNUM *m;
Signature · 1 line
2.37.2
#30 lore raw

Re: [PATCH 3/6] selftests/sgx: Ignore OpenSSL 3.0 deprecated functions warning

Hi Jarkko,

Quoted 11 lines · Jarkko Sakkinen

From: Kristen Carlson Accardi <[email protected]>

OpenSSL 3.0 deprecates some of the functions used in the SGX selftests, causing build errors on new distros. For now ignore the warnings until support for the functions is no longer available and mark FIXME so that it can be clear this should be removed at some point.

---

This one can be dropped from this series. The fix already made it into v6.0-rc3 via selftest.

Reinette

author maintainer
#31 lore raw

Re: [PATCH 3/6] selftests/sgx: Ignore OpenSSL 3.0 deprecated functions warning

Quoted 11 lines · Reinette Chatre

Hi Jarkko,

Quoted 8 lines · Jarkko Sakkinen

From: Kristen Carlson Accardi <[email protected]>

OpenSSL 3.0 deprecates some of the functions used in the SGX selftests, causing build errors on new distros. For now ignore the warnings until support for the functions is no longer available and mark FIXME so that it can be clear this should be removed at some point.

Quoted 6 lines
Quoted 3 lines

---

This one can be dropped from this series. The fix already made it into v6.0-rc3 via selftest.

ok

BR, Jarkko

author maintainer
#32 lore raw

PATCH 4/6 selftests/sgx: Add SGX selftest augment_via_eaccept_long

In reply to Jarkko Sakkinen

From: Vijay Dhanraj <[email protected]>

Add a new test case which is same as augment_via_eaccept but adds a larger number of EPC pages to stress test EAUG via EACCEPT.

---
v2:
- Addressed Reinette's feedback:
https://lore.kernel.org/linux-sgx/[email protected]/

---
 tools/testing/selftests/sgx/load.c      |   5 +-
 tools/testing/selftests/sgx/main.c      | 141 +++++++++++++++++++++---
 tools/testing/selftests/sgx/main.h      |   3 +-
 tools/testing/selftests/sgx/sigstruct.c |   2 +-
 4 files changed, 129 insertions(+), 22 deletions(-)
tools/testing/selftests/sgx/load.c +3 -2
index 94bdeac1cf04..7de1b15c90b1 100644
@@ -171,7 +171,8 @@ uint64_t encl_get_entry(struct encl *encl, const char *symbol)
 	return 0;
 }
 
-bool encl_load(const char *path, struct encl *encl, unsigned long heap_size)
+bool encl_load(const char *path, struct encl *encl, unsigned long heap_size,
+			   unsigned long edmm_size)
 {
 	const char device_path[] = "/dev/sgx_enclave";
 	struct encl_segment *seg;
@@ -300,7 +301,7 @@ bool encl_load(const char *path, struct encl *encl, unsigned long heap_size)
 
 	encl->src_size = encl->segment_tbl[j].offset + encl->segment_tbl[j].size;
 
-	for (encl->encl_size = 4096; encl->encl_size < encl->src_size; )
+	for (encl->encl_size = 4096; encl->encl_size < encl->src_size + edmm_size;)
 		encl->encl_size <<= 1;
 
 	return true;
tools/testing/selftests/sgx/main.c +123 -18

263 lines not on this page · raw message

tools/testing/selftests/sgx/main.h +2 -1
index fc585be97e2f..fe5d39ac0e1e 100644
@@ -35,7 +35,8 @@ extern unsigned char sign_key[];
 extern unsigned char sign_key_end[];
 
 void encl_delete(struct encl *ctx);
-bool encl_load(const char *path, struct encl *encl, unsigned long heap_size);
+bool encl_load(const char *path, struct encl *encl, unsigned long heap_size,
+			   unsigned long edmm_size);
 bool encl_measure(struct encl *encl);
 bool encl_build(struct encl *encl);
 uint64_t encl_get_entry(struct encl *encl, const char *symbol);
tools/testing/selftests/sgx/sigstruct.c +1 -1
index a07896a46364..decd767434d5 100644
@@ -349,7 +349,7 @@ bool encl_measure(struct encl *encl)
 	if (!ctx)
 		goto err;
 
-	if (!mrenclave_ecreate(ctx, encl->src_size))
+	if (!mrenclave_ecreate(ctx, encl->encl_size))
 		goto err;
 
 	for (i = 0; i < encl->nr_segments; i++) {
Signature · 1 line
2.37.2
#33 lore raw

Re: [PATCH 4/6] selftests/sgx: Add SGX selftest augment_via_eaccept_long

Hi Vijay and Jarkko,

Quoted 12 lines · Jarkko Sakkinen

From: Vijay Dhanraj <[email protected]>

Add a new test case which is same as augment_via_eaccept but adds a larger number of EPC pages to stress test EAUG via EACCEPT.

---
v2:
- Addressed Reinette's feedback:
https://lore.kernel.org/linux-sgx/[email protected]/

Quoted 6 lines
---
 tools/testing/selftests/sgx/load.c      |   5 +-
 tools/testing/selftests/sgx/main.c      | 141 +++++++++++++++++++++---
 tools/testing/selftests/sgx/main.h      |   3 +-
 tools/testing/selftests/sgx/sigstruct.c |   2 +-
 4 files changed, 129 insertions(+), 22 deletions(-)

There seems to be at least three patches merged into one here:
1) Update SGX selftests to create enclaves with provided size dedicated to EDMM (this change causes a lot of noise and distracts from the test addition).
2) The mrenclave_ecreate() fix (which is still incomplete).
3) The actual test addition.

tools/testing/selftests/sgx/load.c uint64_t encl_get_entry(struct encl *encl, const char *symbol) · line 176

Quoted 12 lines
tools/testing/selftests/sgx/load.c +2 -1
index 94bdeac1cf04..7de1b15c90b1 100644
@@ -171,7 +171,8 @@ uint64_t encl_get_entry(struct encl *encl, const char *symbol)
 	return 0;
 }
 
-bool encl_load(const char *path, struct encl *encl, unsigned long heap_size)
+bool encl_load(const char *path, struct encl *encl, unsigned long heap_size,
+			   unsigned long edmm_size)
 {

checkpatch.pl informs about alignment issues above and also a few other places.

Quoted 16 lines
tools/testing/selftests/sgx/load.c +1 -1
 	const char device_path[] = "/dev/sgx_enclave";
 	struct encl_segment *seg;
@@ -300,7 +301,7 @@ bool encl_load(const char *path, struct encl *encl, unsigned long heap_size)
 
 	encl->src_size = encl->segment_tbl[j].offset + encl->segment_tbl[j].size;
 
-	for (encl->encl_size = 4096; encl->encl_size < encl->src_size; )
+	for (encl->encl_size = 4096; encl->encl_size < encl->src_size + edmm_size;)
 		encl->encl_size <<= 1;
 
 	return true;
tools/testing/selftests/sgx/main.c +0 -0
index 9820b3809c69..867e98502120 100644

tools/testing/selftests/sgx/main.c line 29

Quoted 9 lines
tools/testing/selftests/sgx/main.c +6 -0
@@ -21,8 +21,15 @@
 #include "../kselftest_harness.h"
 #include "main.h"
 
+/*
+ * The size was chosen based on a bug report:
+ * https://lore.kernel.org/linux-sgx/DM8PR11MB55912A7F47A84EC9913A6352F6999@DM8PR11MB5591.namprd11.prod.outlook.com/
+ */
+static const unsigned long EDMM_SIZE_LONG = 8L * 1024L * 1024L * 1024L; // 8 GB
+static const unsigned long TIMEOUT_LONG = 900; /* seconds */

It is interesting that in this small snippet there are three different comment styles (multi-line comment preceding code, tail comment using /**/, and tail comment using //) :)

tools/testing/selftests/sgx/main.c line 32

Quoted 3 lines
tools/testing/selftests/sgx/main.c +1 -0
 static const uint64_t MAGIC = 0x1122334455667788ULL;
 static const uint64_t MAGIC2 = 0x8877665544332211ULL;
+

Addition of empty line here.

Quoted 32 lines
tools/testing/selftests/sgx/main.c +4 -3
 vdso_sgx_enter_enclave_t vdso_sgx_enter_enclave;
 
 /*
@@ -173,7 +180,8 @@ FIXTURE(enclave) {
 };
 
 static bool setup_test_encl(unsigned long heap_size, struct encl *encl,
-			    struct __test_metadata *_metadata)
+			    struct __test_metadata *_metadata,
+			    unsigned long edmm_size)
 {
 	Elf64_Sym *sgx_enter_enclave_sym = NULL;
 	struct vdso_symtab symtab;
@@ -183,7 +191,7 @@ static bool setup_test_encl(unsigned long heap_size, struct encl *encl,
 	unsigned int i;
 	void *addr;
 
-	if (!encl_load("test_encl.elf", encl, heap_size)) {
+	if (!encl_load("test_encl.elf", encl, heap_size, edmm_size)) {
 		encl_delete(encl);
 		TH_LOG("Failed to load the test enclave.");
 		return false;
@@ -284,7 +292,7 @@ TEST_F(enclave, unclobbered_vdso)
 	struct encl_op_get_from_buf get_op;
 	struct encl_op_put_to_buf put_op;
 
-	ASSERT_TRUE(setup_test_encl(ENCL_HEAP_SIZE_DEFAULT, &self->encl, _metadata));
+	ASSERT_TRUE(setup_test_encl(ENCL_HEAP_SIZE_DEFAULT, &self->encl, _metadata, 0));
 
 	memset(&self->run, 0, sizeof(self->run));
 	self->run.tcs = self->encl.encl_base;

tools/testing/selftests/sgx/main.c TEST_F(enclave, unclobbered_vdso_oversubscribed) · line 369

Quoted 6 lines
tools/testing/selftests/sgx/main.c +1 -1
@@ -357,7 +365,7 @@ TEST_F(enclave, unclobbered_vdso_oversubscribed)
 
 	total_mem = get_total_epc_mem();
 	ASSERT_NE(total_mem, 0);
-	ASSERT_TRUE(setup_test_encl(total_mem, &self->encl, _metadata));
+	ASSERT_TRUE(setup_test_encl(total_mem, &self->encl, _metadata, 0));
 

The support for EDMM size and all the call site changes could be moved to a separate patch to make the changes easier to parse.

...

Quoted 39 lines
tools/testing/selftests/sgx/main.c +35 -0
@@ -1210,6 +1218,103 @@ TEST_F(enclave, augment_via_eaccept)
 	munmap(addr, PAGE_SIZE);
 }
 
+/*
+ * Test for the addition of large number of pages to an initialized enclave via
+ * a pre-emptive run of EACCEPT on every page to be added.
+ */
+TEST_F_TIMEOUT(enclave, augment_via_eaccept_long, TIMEOUT_LONG)
+{
+	struct encl_op_get_from_addr get_addr_op;
+	struct encl_op_put_to_addr put_addr_op;
+	struct encl_op_eaccept eaccept_op;
+	size_t total_size = 0;
+	unsigned long i;
+	void *addr;
+
+	if (!sgx2_supported())
+		SKIP(return, "SGX2 not supported");
+
+	ASSERT_TRUE(setup_test_encl(ENCL_HEAP_SIZE_DEFAULT, &self->encl, _metadata,
+				    EDMM_SIZE_LONG));
+
+	memset(&self->run, 0, sizeof(self->run));
+	self->run.tcs = self->encl.encl_base;
+
+	for (i = 0; i < self->encl.nr_segments; i++) {
+		struct encl_segment *seg = &self->encl.segment_tbl[i];
+
+		total_size += seg->size;
+	}
+
+	/*
+	 * mmap() every page at end of existing enclave to be used for
+	 * EDMM.
+	 */
+	addr = mmap((void *)self->encl.encl_base + total_size, EDMM_SIZE_LONG,
+			PROT_READ | PROT_WRITE | PROT_EXEC, MAP_SHARED | MAP_FIXED,
+			self->encl.fd, 0);

tools/testing/selftests/sgx/main.c TEST_F(enclave, augment_via_eaccept) · line 1265

Quoted 10 lines
tools/testing/selftests/sgx/main.c +10 -0
+	EXPECT_NE(addr, MAP_FAILED);
+
+	self->run.exception_vector = 0;
+	self->run.exception_error_code = 0;
+	self->run.exception_addr = 0;
+
+	/*
+	 * Run EACCEPT on new page to trigger the #PF->EAUG->EACCEPT(again
+	 * without a #PF). All should be transparent to userspace.
+	 */

s/on new page/on every page/ ?

Quoted 12 lines
tools/testing/selftests/sgx/main.c +12 -0
+	eaccept_op.flags = SGX_SECINFO_R | SGX_SECINFO_W | SGX_SECINFO_REG | SGX_SECINFO_PENDING;
+	eaccept_op.ret = 0;
+	eaccept_op.header.type = ENCL_OP_EACCEPT;
+
+	for (i = 0; i < EDMM_SIZE_LONG; i += 4096) {
+		eaccept_op.epc_addr = (uint64_t)(addr + i);
+
+		EXPECT_EQ(ENCL_CALL(&eaccept_op, &self->run, true), 0);
+		if (self->run.exception_vector == 14 &&
+			self->run.exception_error_code == 4 &&
+			self->run.exception_addr == self->encl.encl_base) {
+			munmap(addr, EDMM_SIZE_LONG);

tools/testing/selftests/sgx/main.c TEST_F(enclave, augment_via_eaccept) · line 1287

Quoted 10 lines
tools/testing/selftests/sgx/main.c +10 -0
+			SKIP(return, "Kernel does not support adding pages to initialized enclave");
+		}
+
+		EXPECT_EQ(self->run.exception_vector, 0);
+		EXPECT_EQ(self->run.exception_error_code, 0);
+		EXPECT_EQ(self->run.exception_addr, 0);
+		ASSERT_EQ(eaccept_op.ret, 0);
+		ASSERT_EQ(self->run.function, EEXIT);
+	}
+

I see that you removed all comments here after the discussion about what comment would be appropriate. This may be ok but it does seem awkward that there are two parts to this section and the second part still has a comment while the first does not. How about the comment I provided in
https://lore.kernel.org/linux-sgx/[email protected]/ :

/*
* Pool of pages were successfully added to enclave. Perform sanity
* check on first page of the pool only to ensure data can be written
* to and read from a dynamically added enclave page.
*/

Quoted 33 lines
tools/testing/selftests/sgx/main.c +30 -0
+	put_addr_op.value = MAGIC;
+	put_addr_op.addr = (unsigned long)addr;
+	put_addr_op.header.type = ENCL_OP_PUT_TO_ADDRESS;
+
+	EXPECT_EQ(ENCL_CALL(&put_addr_op, &self->run, true), 0);
+
+	EXPECT_EEXIT(&self->run);
+	EXPECT_EQ(self->run.exception_vector, 0);
+	EXPECT_EQ(self->run.exception_error_code, 0);
+	EXPECT_EQ(self->run.exception_addr, 0);
+
+	/*
+	 * Read memory from newly added page that was just written to,
+	 * confirming that data previously written (MAGIC) is present.
+	 */
+	get_addr_op.value = 0;
+	get_addr_op.addr = (unsigned long)addr;
+	get_addr_op.header.type = ENCL_OP_GET_FROM_ADDRESS;
+
+	EXPECT_EQ(ENCL_CALL(&get_addr_op, &self->run, true), 0);
+
+	EXPECT_EQ(get_addr_op.value, MAGIC);
+	EXPECT_EEXIT(&self->run);
+	EXPECT_EQ(self->run.exception_vector, 0);
+	EXPECT_EQ(self->run.exception_error_code, 0);
+	EXPECT_EQ(self->run.exception_addr, 0);
+
+	munmap(addr, EDMM_SIZE_LONG);
+}
+
 /*
  * SGX2 page type modification test in two phases:
  * Phase 1:

...

tools/testing/selftests/sgx/sigstruct.c bool encl_measure(struct encl *encl) · line 355

Quoted 13 lines
tools/testing/selftests/sgx/sigstruct.c +1 -1
index a07896a46364..decd767434d5 100644
@@ -349,7 +349,7 @@ bool encl_measure(struct encl *encl)
 	if (!ctx)
 		goto err;
 
-	if (!mrenclave_ecreate(ctx, encl->src_size))
+	if (!mrenclave_ecreate(ctx, encl->encl_size))
 		goto err;
 
 	for (i = 0; i < encl->nr_segments; i++) {

As I mentioned in feedback to previous version, even though encl_size is now provided, this misses that mrenclave_ecreate() continues to recompute it within.

Reinette

author maintainer
#34 lore raw

Re: [PATCH 4/6] selftests/sgx: Add SGX selftest augment_via_eaccept_long

Quoted 22 lines · Reinette Chatre

Hi Vijay and Jarkko,

Quoted 12 lines · Jarkko Sakkinen

From: Vijay Dhanraj <[email protected]>

Add a new test case which is same as augment_via_eaccept but adds a larger number of EPC pages to stress test EAUG via EACCEPT.

---
v2:
- Addressed Reinette's feedback:
https://lore.kernel.org/linux-sgx/[email protected]/

Quoted 6 lines
---
 tools/testing/selftests/sgx/load.c      |   5 +-
 tools/testing/selftests/sgx/main.c      | 141 +++++++++++++++++++++---
 tools/testing/selftests/sgx/main.h      |   3 +-
 tools/testing/selftests/sgx/sigstruct.c |   2 +-
 4 files changed, 129 insertions(+), 22 deletions(-)
Quoted 6 lines

There seems to be at least three patches merged into one here:
1) Update SGX selftests to create enclaves with provided size dedicated to EDMM (this change causes a lot of noise and distracts from the test addition).
2) The mrenclave_ecreate() fix (which is still incomplete).
3) The actual test addition.

I would agree on this on a kernel patch but not for kselftest patch. It does not really give useful value here. This adds a test and that is a good enough granularity in my opinion, unless some major architecture work is required as precursory. It is not the case here.

Quoted 8 lines
Quoted 8 lines
tools/testing/selftests/sgx/load.c +0 -0
index 94bdeac1cf04..7de1b15c90b1 100644
@@ -171,7 +171,8 @@ uint64_t encl_get_entry(struct encl *encl, const char *symbol)
 	return 0;
 }
 
Quoted 6 lines

tools/testing/selftests/sgx/load.c uint64_t encl_get_entry(struct encl *encl, const char *symbol) · line 176

Quoted 4 lines
tools/testing/selftests/sgx/load.c +2 -1
-bool encl_load(const char *path, struct encl *encl, unsigned long heap_size)
+bool encl_load(const char *path, struct encl *encl, unsigned long heap_size,
+			   unsigned long edmm_size)
 {

checkpatch.pl informs about alignment issues above and also a few other places.

Weird. I did run checkpatch through these. Will revisit.

Quoted 20 lines
Quoted 20 lines
tools/testing/selftests/sgx/load.c +1 -1
 	const char device_path[] = "/dev/sgx_enclave";
 	struct encl_segment *seg;
@@ -300,7 +301,7 @@ bool encl_load(const char *path, struct encl *encl, unsigned long heap_size)
 
 	encl->src_size = encl->segment_tbl[j].offset + encl->segment_tbl[j].size;
 
-	for (encl->encl_size = 4096; encl->encl_size < encl->src_size; )
+	for (encl->encl_size = 4096; encl->encl_size < encl->src_size + edmm_size;)
 		encl->encl_size <<= 1;
 
 	return true;
tools/testing/selftests/sgx/main.c +0 -0
index 9820b3809c69..867e98502120 100644
@@ -21,8 +21,15 @@
 #include "../kselftest_harness.h"
 #include "main.h"
 
Quoted 10 lines

tools/testing/selftests/sgx/main.c line 29

Quoted 6 lines
tools/testing/selftests/sgx/main.c +6 -0
+/*
+ * The size was chosen based on a bug report:
+ * https://lore.kernel.org/linux-sgx/DM8PR11MB55912A7F47A84EC9913A6352F6999@DM8PR11MB5591.namprd11.prod.outlook.com/
+ */
+static const unsigned long EDMM_SIZE_LONG = 8L * 1024L * 1024L * 1024L; // 8 GB
+static const unsigned long TIMEOUT_LONG = 900; /* seconds */

It is interesting that in this small snippet there are three different comment styles (multi-line comment preceding code, tail comment using /**/, and tail comment using //) :)

Oops. Will fix.

Quoted 6 lines

tools/testing/selftests/sgx/main.c line 32

Quoted 4 lines
tools/testing/selftests/sgx/main.c +1 -0
 static const uint64_t MAGIC = 0x1122334455667788ULL;
 static const uint64_t MAGIC2 = 0x8877665544332211ULL;
+

Addition of empty line here.

Will remove.

Quoted 89 lines
Quoted 33 lines
tools/testing/selftests/sgx/main.c +4 -3
 vdso_sgx_enter_enclave_t vdso_sgx_enter_enclave;
 
 /*
@@ -173,7 +180,8 @@ FIXTURE(enclave) {
 };
 
 static bool setup_test_encl(unsigned long heap_size, struct encl *encl,
-			    struct __test_metadata *_metadata)
+			    struct __test_metadata *_metadata,
+			    unsigned long edmm_size)
 {
 	Elf64_Sym *sgx_enter_enclave_sym = NULL;
 	struct vdso_symtab symtab;
@@ -183,7 +191,7 @@ static bool setup_test_encl(unsigned long heap_size, struct encl *encl,
 	unsigned int i;
 	void *addr;
 
-	if (!encl_load("test_encl.elf", encl, heap_size)) {
+	if (!encl_load("test_encl.elf", encl, heap_size, edmm_size)) {
 		encl_delete(encl);
 		TH_LOG("Failed to load the test enclave.");
 		return false;
@@ -284,7 +292,7 @@ TEST_F(enclave, unclobbered_vdso)
 	struct encl_op_get_from_buf get_op;
 	struct encl_op_put_to_buf put_op;
 
-	ASSERT_TRUE(setup_test_encl(ENCL_HEAP_SIZE_DEFAULT, &self->encl, _metadata));
+	ASSERT_TRUE(setup_test_encl(ENCL_HEAP_SIZE_DEFAULT, &self->encl, _metadata, 0));
 
 	memset(&self->run, 0, sizeof(self->run));
 	self->run.tcs = self->encl.encl_base;

tools/testing/selftests/sgx/main.c TEST_F(enclave, unclobbered_vdso_oversubscribed) · line 369

Quoted 6 lines
tools/testing/selftests/sgx/main.c +1 -1
@@ -357,7 +365,7 @@ TEST_F(enclave, unclobbered_vdso_oversubscribed)
 
 	total_mem = get_total_epc_mem();
 	ASSERT_NE(total_mem, 0);
-	ASSERT_TRUE(setup_test_encl(total_mem, &self->encl, _metadata));
+	ASSERT_TRUE(setup_test_encl(total_mem, &self->encl, _metadata, 0));
 

The support for EDMM size and all the call site changes could be moved to a separate patch to make the changes easier to parse.

...

Quoted 43 lines
tools/testing/selftests/sgx/main.c +39 -0
@@ -1210,6 +1218,103 @@ TEST_F(enclave, augment_via_eaccept)
 	munmap(addr, PAGE_SIZE);
 }
 
+/*
+ * Test for the addition of large number of pages to an initialized enclave via
+ * a pre-emptive run of EACCEPT on every page to be added.
+ */
+TEST_F_TIMEOUT(enclave, augment_via_eaccept_long, TIMEOUT_LONG)
+{
+	struct encl_op_get_from_addr get_addr_op;
+	struct encl_op_put_to_addr put_addr_op;
+	struct encl_op_eaccept eaccept_op;
+	size_t total_size = 0;
+	unsigned long i;
+	void *addr;
+
+	if (!sgx2_supported())
+		SKIP(return, "SGX2 not supported");
+
+	ASSERT_TRUE(setup_test_encl(ENCL_HEAP_SIZE_DEFAULT, &self->encl, _metadata,
+				    EDMM_SIZE_LONG));
+
+	memset(&self->run, 0, sizeof(self->run));
+	self->run.tcs = self->encl.encl_base;
+
+	for (i = 0; i < self->encl.nr_segments; i++) {
+		struct encl_segment *seg = &self->encl.segment_tbl[i];
+
+		total_size += seg->size;
+	}
+
+	/*
+	 * mmap() every page at end of existing enclave to be used for
+	 * EDMM.
+	 */
+	addr = mmap((void *)self->encl.encl_base + total_size, EDMM_SIZE_LONG,
+			PROT_READ | PROT_WRITE | PROT_EXEC, MAP_SHARED | MAP_FIXED,
+			self->encl.fd, 0);
+	EXPECT_NE(addr, MAP_FAILED);
+
+	self->run.exception_vector = 0;
+	self->run.exception_error_code = 0;
Quoted 8 lines

tools/testing/selftests/sgx/main.c TEST_F(enclave, augment_via_eaccept) · line 1265

Quoted 6 lines
tools/testing/selftests/sgx/main.c +6 -0
+	self->run.exception_addr = 0;
+
+	/*
+	 * Run EACCEPT on new page to trigger the #PF->EAUG->EACCEPT(again
+	 * without a #PF). All should be transparent to userspace.
+	 */

s/on new page/on every page/ ?

+1

Quoted 30 lines
Quoted 13 lines
tools/testing/selftests/sgx/main.c +12 -0
+	eaccept_op.flags = SGX_SECINFO_R | SGX_SECINFO_W | SGX_SECINFO_REG | SGX_SECINFO_PENDING;
+	eaccept_op.ret = 0;
+	eaccept_op.header.type = ENCL_OP_EACCEPT;
+
+	for (i = 0; i < EDMM_SIZE_LONG; i += 4096) {
+		eaccept_op.epc_addr = (uint64_t)(addr + i);
+
+		EXPECT_EQ(ENCL_CALL(&eaccept_op, &self->run, true), 0);
+		if (self->run.exception_vector == 14 &&
+			self->run.exception_error_code == 4 &&
+			self->run.exception_addr == self->encl.encl_base) {
+			munmap(addr, EDMM_SIZE_LONG);

tools/testing/selftests/sgx/main.c TEST_F(enclave, augment_via_eaccept) · line 1287

Quoted 10 lines
tools/testing/selftests/sgx/main.c +10 -0
+			SKIP(return, "Kernel does not support adding pages to initialized enclave");
+		}
+
+		EXPECT_EQ(self->run.exception_vector, 0);
+		EXPECT_EQ(self->run.exception_error_code, 0);
+		EXPECT_EQ(self->run.exception_addr, 0);
+		ASSERT_EQ(eaccept_op.ret, 0);
+		ASSERT_EQ(self->run.function, EEXIT);
+	}
+

I see that you removed all comments here after the discussion about what comment would be appropriate. This may be ok but it does seem awkward that there are two parts to this section and the second part still has a comment while the first does not. How about the comment I provided in
https://lore.kernel.org/linux-sgx/[email protected]/ :

Quoted 5 lines

/*
* Pool of pages were successfully added to enclave. Perform sanity
* check on first page of the pool only to ensure data can be written
* to and read from a dynamically added enclave page.
*/

+1

Quoted 49 lines
Quoted 34 lines
tools/testing/selftests/sgx/main.c +30 -0
+	put_addr_op.value = MAGIC;
+	put_addr_op.addr = (unsigned long)addr;
+	put_addr_op.header.type = ENCL_OP_PUT_TO_ADDRESS;
+
+	EXPECT_EQ(ENCL_CALL(&put_addr_op, &self->run, true), 0);
+
+	EXPECT_EEXIT(&self->run);
+	EXPECT_EQ(self->run.exception_vector, 0);
+	EXPECT_EQ(self->run.exception_error_code, 0);
+	EXPECT_EQ(self->run.exception_addr, 0);
+
+	/*
+	 * Read memory from newly added page that was just written to,
+	 * confirming that data previously written (MAGIC) is present.
+	 */
+	get_addr_op.value = 0;
+	get_addr_op.addr = (unsigned long)addr;
+	get_addr_op.header.type = ENCL_OP_GET_FROM_ADDRESS;
+
+	EXPECT_EQ(ENCL_CALL(&get_addr_op, &self->run, true), 0);
+
+	EXPECT_EQ(get_addr_op.value, MAGIC);
+	EXPECT_EEXIT(&self->run);
+	EXPECT_EQ(self->run.exception_vector, 0);
+	EXPECT_EQ(self->run.exception_error_code, 0);
+	EXPECT_EQ(self->run.exception_addr, 0);
+
+	munmap(addr, EDMM_SIZE_LONG);
+}
+
 /*
  * SGX2 page type modification test in two phases:
  * Phase 1:

...

Quoted 12 lines
tools/testing/selftests/sgx/sigstruct.c +1 -1
index a07896a46364..decd767434d5 100644
@@ -349,7 +349,7 @@ bool encl_measure(struct encl *encl)
 	if (!ctx)
 		goto err;
 
-	if (!mrenclave_ecreate(ctx, encl->src_size))
+	if (!mrenclave_ecreate(ctx, encl->encl_size))
 		goto err;
 
Quoted 8 lines

tools/testing/selftests/sgx/sigstruct.c bool encl_measure(struct encl *encl) · line 355

Quoted 1 line
tools/testing/selftests/sgx/sigstruct.c +0 -0
 	for (i = 0; i < encl->nr_segments; i++) {

As I mentioned in feedback to previous version, even though encl_size is now provided, this misses that mrenclave_ecreate() continues to recompute it within.

Reinette

I simply forgot this one, sorry. Will fix.

BR, Jarkko

#35 lore raw

Re: [PATCH 4/6] selftests/sgx: Add SGX selftest augment_via_eaccept_long

Hi Jarkko,

Quoted 28 lines · Jarkko Sakkinen
Quoted 20 lines · Reinette Chatre
Quoted 12 lines · Jarkko Sakkinen

From: Vijay Dhanraj <[email protected]>

Add a new test case which is same as augment_via_eaccept but adds a larger number of EPC pages to stress test EAUG via EACCEPT.

---
v2:
- Addressed Reinette's feedback:
https://lore.kernel.org/linux-sgx/[email protected]/

Quoted 6 lines
---
 tools/testing/selftests/sgx/load.c      |   5 +-
 tools/testing/selftests/sgx/main.c      | 141 +++++++++++++++++++++---
 tools/testing/selftests/sgx/main.h      |   3 +-
 tools/testing/selftests/sgx/sigstruct.c |   2 +-
 4 files changed, 129 insertions(+), 22 deletions(-)
Quoted 6 lines

There seems to be at least three patches merged into one here:
1) Update SGX selftests to create enclaves with provided size dedicated to EDMM (this change causes a lot of noise and distracts from the test addition).
2) The mrenclave_ecreate() fix (which is still incomplete).
3) The actual test addition.

Quoted 4 lines

I would agree on this on a kernel patch but not for kselftest patch. It does not really give useful value here. This adds a test and that is a good enough granularity in my opinion, unless some major architecture work is required as precursory. It is not the case here.

I must say that for many good reasons this goes against one of the fundamental rules of kernel patches: separate logical changes into separate patches. This is your domain though so of course the work within it follows your guidance and I will not pursue it further.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
tools/testing/selftests/sgx/load.c +0 -0
index 94bdeac1cf04..7de1b15c90b1 100644
@@ -171,7 +171,8 @@ uint64_t encl_get_entry(struct encl *encl, const char *symbol)
 	return 0;
 }
 
Quoted 8 lines
Quoted 6 lines

tools/testing/selftests/sgx/load.c uint64_t encl_get_entry(struct encl *encl, const char *symbol) · line 176

Quoted 4 lines
tools/testing/selftests/sgx/load.c +2 -1
-bool encl_load(const char *path, struct encl *encl, unsigned long heap_size)
+bool encl_load(const char *path, struct encl *encl, unsigned long heap_size,
+			   unsigned long edmm_size)
 {

checkpatch.pl informs about alignment issues above and also a few other places.

Weird. I did run checkpatch through these. Will revisit.

I usually run checkpatch.pl with "--strict".

Reinette

author maintainer
#36 lore raw

Re: [PATCH 4/6] selftests/sgx: Add SGX selftest augment_via_eaccept_long

Quoted 36 lines · Reinette Chatre

Hi Jarkko,

Quoted 28 lines · Jarkko Sakkinen
Quoted 20 lines · Reinette Chatre
Quoted 12 lines · Jarkko Sakkinen

From: Vijay Dhanraj <[email protected]>

Add a new test case which is same as augment_via_eaccept but adds a larger number of EPC pages to stress test EAUG via EACCEPT.

---
v2:
- Addressed Reinette's feedback:
https://lore.kernel.org/linux-sgx/[email protected]/

Quoted 6 lines
---
 tools/testing/selftests/sgx/load.c      |   5 +-
 tools/testing/selftests/sgx/main.c      | 141 +++++++++++++++++++++---
 tools/testing/selftests/sgx/main.h      |   3 +-
 tools/testing/selftests/sgx/sigstruct.c |   2 +-
 4 files changed, 129 insertions(+), 22 deletions(-)
Quoted 6 lines

There seems to be at least three patches merged into one here:
1) Update SGX selftests to create enclaves with provided size dedicated to EDMM (this change causes a lot of noise and distracts from the test addition).
2) The mrenclave_ecreate() fix (which is still incomplete).
3) The actual test addition.

Quoted 4 lines

I would agree on this on a kernel patch but not for kselftest patch. It does not really give useful value here. This adds a test and that is a good enough granularity in my opinion, unless some major architecture work is required as precursory. It is not the case here.

Quoted 4 lines

I must say that for many good reasons this goes against one of the fundamental rules of kernel patches: separate logical changes into separate patches. This is your domain though so of course the work within it follows your guidance and I will not pursue it further.

I don't consider kselftest patch exactly same as kernel patch but I can split this. What would be good enough?

Quoted 1 line

I usually run checkpatch.pl with "--strict".

I honestly did not know that this option was available :-) First time I hear of it.

Thanks.

Quoted 2 lines

Reinette

BR, Jarkko

#37 lore raw

Re: [PATCH 4/6] selftests/sgx: Add SGX selftest augment_via_eaccept_long

Hi Jarkko,

Quoted 17 lines · Jarkko Sakkinen
Quoted 16 lines · Reinette Chatre
Quoted 14 lines · Jarkko Sakkinen
Quoted 8 lines · Reinette Chatre

On 8/29/2022 8:12 PM, Jarkko Sakkinen wrote:

There seems to be at least three patches merged into one here:
1) Update SGX selftests to create enclaves with provided size dedicated to EDMM (this change causes a lot of noise and distracts from the test addition).
2) The mrenclave_ecreate() fix (which is still incomplete).
3) The actual test addition.

I would agree on this on a kernel patch but not for kselftest patch. It does not really give useful value here. This adds a test and that is a good enough granularity in my opinion, unless some major architecture work is required as precursory. It is not the case here.

Quoted 6 lines
Quoted 4 lines

I must say that for many good reasons this goes against one of the fundamental rules of kernel patches: separate logical changes into separate patches. This is your domain though so of course the work within it follows your guidance and I will not pursue it further.

I don't consider kselftest patch exactly same as kernel patch

You are not alone.

Quoted 1 line

but I can split this. What would be good enough?

I identified three candidate patches in my original response that is quoted above, but as I mentioned I understand the sentiment and this is your domain so I will not insist on it.

Reinette

author maintainer
#38 lore raw

Re: [PATCH 4/6] selftests/sgx: Add SGX selftest augment_via_eaccept_long

Quoted 27 lines · Reinette Chatre

Hi Jarkko,

Quoted 17 lines · Jarkko Sakkinen
Quoted 16 lines · Reinette Chatre
Quoted 14 lines · Jarkko Sakkinen
Quoted 8 lines · Reinette Chatre

On 8/29/2022 8:12 PM, Jarkko Sakkinen wrote:

There seems to be at least three patches merged into one here:
1) Update SGX selftests to create enclaves with provided size dedicated to EDMM (this change causes a lot of noise and distracts from the test addition).
2) The mrenclave_ecreate() fix (which is still incomplete).
3) The actual test addition.

I would agree on this on a kernel patch but not for kselftest patch. It does not really give useful value here. This adds a test and that is a good enough granularity in my opinion, unless some major architecture work is required as precursory. It is not the case here.

Quoted 6 lines
Quoted 4 lines

I must say that for many good reasons this goes against one of the fundamental rules of kernel patches: separate logical changes into separate patches. This is your domain though so of course the work within it follows your guidance and I will not pursue it further.

I don't consider kselftest patch exactly same as kernel patch

Quoted 7 lines

You are not alone.

Quoted 1 line

but I can split this. What would be good enough?

I identified three candidate patches in my original response that is quoted above, but as I mentioned I understand the sentiment and this is your domain so I will not insist on it.

OK, fair enough, I'll rework on this. It's my domain but at least my own aim is always only satisfy on consensus :-)

BR, Jarkko

author maintainer
#39 lore raw

Re: [PATCH 4/6] selftests/sgx: Add SGX selftest augment_via_eaccept_long

Quoted 32 lines · Jarkko Sakkinen
Quoted 31 lines · Reinette Chatre

Hi Jarkko,

Quoted 17 lines · Jarkko Sakkinen
Quoted 16 lines · Reinette Chatre
Quoted 14 lines · Jarkko Sakkinen
Quoted 8 lines · Reinette Chatre

On 8/29/2022 8:12 PM, Jarkko Sakkinen wrote:

There seems to be at least three patches merged into one here:
1) Update SGX selftests to create enclaves with provided size dedicated to EDMM (this change causes a lot of noise and distracts from the test addition).
2) The mrenclave_ecreate() fix (which is still incomplete).
3) The actual test addition.

I would agree on this on a kernel patch but not for kselftest patch. It does not really give useful value here. This adds a test and that is a good enough granularity in my opinion, unless some major architecture work is required as precursory. It is not the case here.

Quoted 6 lines
Quoted 4 lines

I must say that for many good reasons this goes against one of the fundamental rules of kernel patches: separate logical changes into separate patches. This is your domain though so of course the work within it follows your guidance and I will not pursue it further.

I don't consider kselftest patch exactly same as kernel patch

You are not alone.

Quoted 1 line

but I can split this. What would be good enough?

Quoted 7 lines
Quoted 3 lines

I identified three candidate patches in my original response that is quoted above, but as I mentioned I understand the sentiment and this is your domain so I will not insist on it.

OK, fair enough, I'll rework on this. It's my domain but at least my own aim is always only satisfy on consensus :-)

I'll also split the patch set because this is not as urgent as getting the fixes in. There will be separate fixes and kselftest improvements patch sets.

BR, Jarkko

author maintainer
#40 lore raw

PATCH 5/6 selftests/sgx: retry the ioctls returned with EAGAIN

In reply to Jarkko Sakkinen

From: Haitao Huang <[email protected]>

For EMODT and EREMOVE ioctls with a large range, kernel may not finish in one shot and return EAGAIN error code and count of bytes of EPC pages on that operations are finished successfully.

Change the unclobbered_vdso_oversubscribed_remove test to rerun the ioctls in a loop, updating offset and length using the byte count returned in each iteration.

---
 tools/testing/selftests/sgx/main.c | 39 +++++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 9 deletions(-)
tools/testing/selftests/sgx/main.c +30 -9
index 867e98502120..3268d8b01b0b 100644
@@ -399,7 +399,7 @@ TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900)
 	unsigned long total_mem;
 	int ret, errno_save;
 	unsigned long addr;
-	unsigned long i;
+	unsigned long i, count;
 
 	/*
 	 * Create enclave with additional heap that is as big as all
@@ -461,16 +461,27 @@ TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900)
 	modt_ioc.offset = heap->offset;
 	modt_ioc.length = heap->size;
 	modt_ioc.page_type = SGX_PAGE_TYPE_TRIM;
-
+	count = 0;
 	TH_LOG("Changing type of %zd bytes to trimmed may take a while ...",
 	       heap->size);
-	ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_MODIFY_TYPES, &modt_ioc);
-	errno_save = ret == -1 ? errno : 0;
+	do {
+		ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_MODIFY_TYPES, &modt_ioc);
+		errno_save = ret == -1 ? errno : 0;
+		if (errno_save == EAGAIN) {
+			count += modt_ioc.count;
+			modt_ioc.offset += modt_ioc.count;
+			modt_ioc.length -= modt_ioc.count;
+			modt_ioc.result = 0;
+			modt_ioc.count = 0;
+		} else
+			break;
+	} while (modt_ioc.length != 0);
 
 	EXPECT_EQ(ret, 0);
 	EXPECT_EQ(errno_save, 0);
 	EXPECT_EQ(modt_ioc.result, 0);
-	EXPECT_EQ(modt_ioc.count, heap->size);
+	count += modt_ioc.count;
+	EXPECT_EQ(count, heap->size);
 
 	/* EACCEPT all removed pages. */
 	addr = self->encl.encl_base + heap->offset;
@@ -498,15 +509,25 @@ TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900)
 
 	remove_ioc.offset = heap->offset;
 	remove_ioc.length = heap->size;
-
+	count = 0;
 	TH_LOG("Removing %zd bytes from enclave may take a while ...",
 	       heap->size);
-	ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_REMOVE_PAGES, &remove_ioc);
-	errno_save = ret == -1 ? errno : 0;
+	do {
+		ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_REMOVE_PAGES, &remove_ioc);
+		errno_save = ret == -1 ? errno : 0;
+		if (errno_save == EAGAIN) {
+			count += remove_ioc.count;
+			remove_ioc.offset += remove_ioc.count;
+			remove_ioc.length -= remove_ioc.count;
+			remove_ioc.count = 0;
+		} else
+			break;
+	} while (remove_ioc.length != 0);
 
 	EXPECT_EQ(ret, 0);
 	EXPECT_EQ(errno_save, 0);
-	EXPECT_EQ(remove_ioc.count, heap->size);
+	count += remove_ioc.count;
+	EXPECT_EQ(count, heap->size);
 }
 
 TEST_F(enclave, clobbered_vdso)
Signature · 1 line
2.37.2
#41 lore raw

Re: [PATCH 5/6] selftests/sgx: retry the ioctls returned with EAGAIN

Hi Haitao and Jarkko,

selftests/sgx: Retry the ioctl()s returned with EAGAIN

Quoted 3 lines · Jarkko Sakkinen

From: Haitao Huang <[email protected]>

For EMODT and EREMOVE ioctls with a large range, kernel

ioctl()s?

Quoted 6 lines

may not finish in one shot and return EAGAIN error code and count of bytes of EPC pages on that operations are finished successfully.

Change the unclobbered_vdso_oversubscribed_remove test to rerun the ioctls in a loop, updating offset and length

ioctl()s?

Quoted 2 lines

using the byte count returned in each iteration.

This is a valuable addition, thank you very much.

Quoted 12 lines
---
 tools/testing/selftests/sgx/main.c | 39 +++++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 9 deletions(-)
tools/testing/selftests/sgx/main.c +0 -0
index 867e98502120..3268d8b01b0b 100644

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 402

Quoted 5 lines
tools/testing/selftests/sgx/main.c +1 -1
@@ -399,7 +399,7 @@ TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900)
 	unsigned long total_mem;
 	int ret, errno_save;
 	unsigned long addr;
-	unsigned long i;
+	unsigned long i, count;

Reverse fir tree?

Quoted 11 lines
tools/testing/selftests/sgx/main.c +1 -1
 	/*
 	 * Create enclave with additional heap that is as big as all
@@ -461,16 +461,27 @@ TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900)
 	modt_ioc.offset = heap->offset;
 	modt_ioc.length = heap->size;
 	modt_ioc.page_type = SGX_PAGE_TYPE_TRIM;
-
+	count = 0;
 	TH_LOG("Changing type of %zd bytes to trimmed may take a while ...",
 	       heap->size);

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 474

Quoted 10 lines
tools/testing/selftests/sgx/main.c +8 -2
-	ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_MODIFY_TYPES, &modt_ioc);
-	errno_save = ret == -1 ? errno : 0;
+	do {
+		ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_MODIFY_TYPES, &modt_ioc);
+		errno_save = ret == -1 ? errno : 0;
+		if (errno_save == EAGAIN) {
+			count += modt_ioc.count;
+			modt_ioc.offset += modt_ioc.count;
+			modt_ioc.length -= modt_ioc.count;
+			modt_ioc.result = 0;

As part of the test I think it would be helpful to know if there was a result code in here. What do you think of failing the test if it is not zero?

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 477

Quoted 3 lines
tools/testing/selftests/sgx/main.c +3 -0
+			modt_ioc.count = 0;
+		} else
+			break;

Watch out for unbalanced braces (also later in patch). This causes checkpatch.pl noise.

Quoted 31 lines
tools/testing/selftests/sgx/main.c +14 -4
+	} while (modt_ioc.length != 0);
 
 	EXPECT_EQ(ret, 0);
 	EXPECT_EQ(errno_save, 0);
 	EXPECT_EQ(modt_ioc.result, 0);
-	EXPECT_EQ(modt_ioc.count, heap->size);
+	count += modt_ioc.count;
+	EXPECT_EQ(count, heap->size);
 
 	/* EACCEPT all removed pages. */
 	addr = self->encl.encl_base + heap->offset;
@@ -498,15 +509,25 @@ TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900)
 
 	remove_ioc.offset = heap->offset;
 	remove_ioc.length = heap->size;
-
+	count = 0;
 	TH_LOG("Removing %zd bytes from enclave may take a while ...",
 	       heap->size);
-	ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_REMOVE_PAGES, &remove_ioc);
-	errno_save = ret == -1 ? errno : 0;
+	do {
+		ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_REMOVE_PAGES, &remove_ioc);
+		errno_save = ret == -1 ? errno : 0;
+		if (errno_save == EAGAIN) {
+			count += remove_ioc.count;
+			remove_ioc.offset += remove_ioc.count;
+			remove_ioc.length -= remove_ioc.count;
+			remove_ioc.count = 0;
+		} else
+			break;

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 533

Quoted 10 lines
tools/testing/selftests/sgx/main.c +3 -1
+	} while (remove_ioc.length != 0);
 
 	EXPECT_EQ(ret, 0);
 	EXPECT_EQ(errno_save, 0);
-	EXPECT_EQ(remove_ioc.count, heap->size);
+	count += remove_ioc.count;
+	EXPECT_EQ(count, heap->size);
 }
 
 TEST_F(enclave, clobbered_vdso)

Reinette

author maintainer
#42 lore raw

Re: [PATCH 5/6] selftests/sgx: retry the ioctls returned with EAGAIN

Quoted 6 lines · Reinette Chatre

Hi Haitao and Jarkko,

selftests/sgx: Retry the ioctl()s returned with EAGAIN

Quoted 6 lines
Quoted 3 lines · Jarkko Sakkinen

From: Haitao Huang <[email protected]>

For EMODT and EREMOVE ioctls with a large range, kernel

ioctl()s?

Ioctl is common enough to be considered as noun and is widely phrased like that in commit messages. I don't see any added clarity.

Quoted 22 lines
Quoted 7 lines

may not finish in one shot and return EAGAIN error code and count of bytes of EPC pages on that operations are finished successfully.

Change the unclobbered_vdso_oversubscribed_remove test to rerun the ioctls in a loop, updating offset and length

ioctl()s?

Quoted 2 lines

using the byte count returned in each iteration.

This is a valuable addition, thank you very much.

Quoted 7 lines
---
 tools/testing/selftests/sgx/main.c | 39 +++++++++++++++++++++++-------
 1 file changed, 30 insertions(+), 9 deletions(-)
Quoted 12 lines

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 402

Quoted 10 lines
tools/testing/selftests/sgx/main.c +1 -1
index 867e98502120..3268d8b01b0b 100644
@@ -399,7 +399,7 @@ TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900)
 	unsigned long total_mem;
 	int ret, errno_save;
 	unsigned long addr;
-	unsigned long i;
+	unsigned long i, count;

Reverse fir tree?

+1

Quoted 17 lines
Quoted 17 lines
tools/testing/selftests/sgx/main.c +4 -3
 	/*
 	 * Create enclave with additional heap that is as big as all
@@ -461,16 +461,27 @@ TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900)
 	modt_ioc.offset = heap->offset;
 	modt_ioc.length = heap->size;
 	modt_ioc.page_type = SGX_PAGE_TYPE_TRIM;
-
+	count = 0;
 	TH_LOG("Changing type of %zd bytes to trimmed may take a while ...",
 	       heap->size);
-	ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_MODIFY_TYPES, &modt_ioc);
-	errno_save = ret == -1 ? errno : 0;
+	do {
+		ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_MODIFY_TYPES, &modt_ioc);
+		errno_save = ret == -1 ? errno : 0;
Quoted 8 lines

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 474

Quoted 5 lines
tools/testing/selftests/sgx/main.c +5 -0
+		if (errno_save == EAGAIN) {
+			count += modt_ioc.count;
+			modt_ioc.offset += modt_ioc.count;
+			modt_ioc.length -= modt_ioc.count;
+			modt_ioc.result = 0;

As part of the test I think it would be helpful to know if there was a result code in here. What do you think of failing the test if it is not zero?

I would not mind doing that.

Quoted 7 lines

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 477

Quoted 4 lines
tools/testing/selftests/sgx/main.c +3 -0
+			modt_ioc.count = 0;
+		} else
+			break;

Watch out for unbalanced braces (also later in patch). This causes checkpatch.pl noise.

Again. I did run checkpatch to all of these. Will revisit.

Quoted 41 lines
Quoted 41 lines
tools/testing/selftests/sgx/main.c +17 -5
+	} while (modt_ioc.length != 0);
 
 	EXPECT_EQ(ret, 0);
 	EXPECT_EQ(errno_save, 0);
 	EXPECT_EQ(modt_ioc.result, 0);
-	EXPECT_EQ(modt_ioc.count, heap->size);
+	count += modt_ioc.count;
+	EXPECT_EQ(count, heap->size);
 
 	/* EACCEPT all removed pages. */
 	addr = self->encl.encl_base + heap->offset;
@@ -498,15 +509,25 @@ TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900)
 
 	remove_ioc.offset = heap->offset;
 	remove_ioc.length = heap->size;
-
+	count = 0;
 	TH_LOG("Removing %zd bytes from enclave may take a while ...",
 	       heap->size);
-	ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_REMOVE_PAGES, &remove_ioc);
-	errno_save = ret == -1 ? errno : 0;
+	do {
+		ret = ioctl(self->encl.fd, SGX_IOC_ENCLAVE_REMOVE_PAGES, &remove_ioc);
+		errno_save = ret == -1 ? errno : 0;
+		if (errno_save == EAGAIN) {
+			count += remove_ioc.count;
+			remove_ioc.offset += remove_ioc.count;
+			remove_ioc.length -= remove_ioc.count;
+			remove_ioc.count = 0;
+		} else
+			break;
+	} while (remove_ioc.length != 0);
 
 	EXPECT_EQ(ret, 0);
 	EXPECT_EQ(errno_save, 0);
-	EXPECT_EQ(remove_ioc.count, heap->size);
+	count += remove_ioc.count;
+	EXPECT_EQ(count, heap->size);
 }
 
Quoted 3 lines

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 533

Quoted 1 line
tools/testing/selftests/sgx/main.c +0 -0
 TEST_F(enclave, clobbered_vdso)

Reinette

BR, Jarkko

#43 lore raw

Re: [PATCH 5/6] selftests/sgx: retry the ioctls returned with EAGAIN

Hi Jarkko,

Quoted 10 lines · Jarkko Sakkinen
Quoted 9 lines · Reinette Chatre

Hi Haitao and Jarkko,

selftests/sgx: Retry the ioctl()s returned with EAGAIN

Quoted 2 lines · Jarkko Sakkinen

From: Haitao Huang <[email protected]>

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

For EMODT and EREMOVE ioctls with a large range, kernel

ioctl()s?

Ioctl is common enough to be considered as noun and is widely phrased like that in commit messages. I don't see any added clarity.

ok. I was asked to make this change in the SGX2 patches and thought that I should propagate this advice :)

Quoted 8 lines
Quoted 6 lines

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 477

Quoted 3 lines
tools/testing/selftests/sgx/main.c +3 -0
+			modt_ioc.count = 0;
+		} else
+			break;

Watch out for unbalanced braces (also later in patch). This causes checkpatch.pl noise.

Again. I did run checkpatch to all of these. Will revisit.

It looks like I see it because I use "checkpatch.pl --strict".

Reinette

author maintainer
#44 lore raw

Re: [PATCH 5/6] selftests/sgx: retry the ioctls returned with EAGAIN

Quoted 15 lines · Reinette Chatre

Hi Jarkko,

Quoted 12 lines · Jarkko Sakkinen
Quoted 11 lines · Reinette Chatre

Hi Haitao and Jarkko,

selftests/sgx: Retry the ioctl()s returned with EAGAIN

Quoted 3 lines · Jarkko Sakkinen

From: Haitao Huang <[email protected]>

For EMODT and EREMOVE ioctls with a large range, kernel

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

ioctl()s?

Ioctl is common enough to be considered as noun and is widely phrased like that in commit messages. I don't see any added clarity.

ok. I was asked to make this change in the SGX2 patches and thought that I should propagate this advice :)

I can use the other form too, np.

Quoted 11 lines
Quoted 9 lines
Quoted 7 lines

tools/testing/selftests/sgx/main.c TEST_F_TIMEOUT(enclave, unclobbered_vdso_oversubscribed_remove, 900) · line 477

Quoted 4 lines
tools/testing/selftests/sgx/main.c +3 -0
+			modt_ioc.count = 0;
+		} else
+			break;

Watch out for unbalanced braces (also later in patch). This causes checkpatch.pl noise.

Again. I did run checkpatch to all of these. Will revisit.

It looks like I see it because I use "checkpatch.pl --strict".

Thanks BTW for pointing this out :-)

Quoted 1 line

Reinette

BR, Jarkko

#45 lore raw

Re: [PATCH 5/6] selftests/sgx: retry the ioctls returned with EAGAIN

In reply to Jarkko Sakkinen
Quoted 8 lines · Jarkko Sakkinen
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines

From: Haitao Huang <[email protected]>

For EMODT and EREMOVE ioctls with a large range, kernel

ioctl()s?

Ioctl is common enough to be considered as noun and is widely phrased like that in commit messages. I don't see any added clarity.

I definitely prefer ioctl().

author maintainer
#46 lore raw

Re: [PATCH 5/6] selftests/sgx: retry the ioctls returned with EAGAIN

Quoted 11 lines · Dave Hansen
Quoted 8 lines · Jarkko Sakkinen
Quoted 5 lines · Jarkko Sakkinen
Quoted 3 lines

From: Haitao Huang <[email protected]>

For EMODT and EREMOVE ioctls with a large range, kernel

ioctl()s?

Ioctl is common enough to be considered as noun and is widely phrased like that in commit messages. I don't see any added clarity.

I definitely prefer ioctl().

I can use it.

BR, Jarkko

author maintainer
#47 lore raw

PATCH 6/6 selftests/sgx: Add a bpftrace script for tracking allocation errors

In reply to Jarkko Sakkinen
---
 tools/testing/selftests/sgx/alloc-error.bt | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 tools/testing/selftests/sgx/alloc-error.bt
tools/testing/selftests/sgx/alloc-error.bt new file +7 -0
new file mode 100644
index 000000000000..9268d50dea29
@@ -0,0 +1,7 @@
+kr:sgx_alloc_epc_page /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_alloc_epc_page: retval=%d\n", (int64)retval);
+}
+
+kr:sgx_encl_page_alloc /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_encl_page_alloc: retval=%d\n", (int64)retval);
+}
Signature · 1 line
2.37.2
#48 lore raw

Re: [PATCH 6/6] selftests/sgx: Add a bpftrace script for tracking allocation errors

Hi Jarkko,

Quoted 12 lines · Jarkko Sakkinen
---
 tools/testing/selftests/sgx/alloc-error.bt | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 tools/testing/selftests/sgx/alloc-error.bt
tools/testing/selftests/sgx/alloc-error.bt new file +0 -0
new file mode 100644
index 000000000000..9268d50dea29

tools/testing/selftests/sgx/alloc-error.bt line 7

Quoted 7 lines
tools/testing/selftests/sgx/alloc-error.bt new file +7 -0
@@ -0,0 +1,7 @@
+kr:sgx_alloc_epc_page /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_alloc_epc_page: retval=%d\n", (int64)retval);
+}
+
+kr:sgx_encl_page_alloc /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_encl_page_alloc: retval=%d\n", (int64)retval);
+}

Could there be a snippet of comments in this new file to guide users on how to use this script?

Reinette

author maintainer
#49 lore raw

Re: [PATCH 6/6] selftests/sgx: Add a bpftrace script for tracking allocation errors

Quoted 18 lines · Reinette Chatre

Hi Jarkko,

Quoted 15 lines · Jarkko Sakkinen
---
 tools/testing/selftests/sgx/alloc-error.bt | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 tools/testing/selftests/sgx/alloc-error.bt
tools/testing/selftests/sgx/alloc-error.bt new file +3 -0
new file mode 100644
index 000000000000..9268d50dea29
@@ -0,0 +1,7 @@
+kr:sgx_alloc_epc_page /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_alloc_epc_page: retval=%d\n", (int64)retval);
+}
Quoted 8 lines

tools/testing/selftests/sgx/alloc-error.bt line 7

Quoted 4 lines
tools/testing/selftests/sgx/alloc-error.bt new file +4 -0
+
+kr:sgx_encl_page_alloc /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_encl_page_alloc: retval=%d\n", (int64)retval);
+}

Could there be a snippet of comments in this new file to guide users on how to use this script?

Do not mean to be rude but I'm not sure what there is to guide but I'm open for ideas.

BR, Jarkko

#50 lore raw

Re: [PATCH 6/6] selftests/sgx: Add a bpftrace script for tracking allocation errors

Hi Jarkko,

Quoted 28 lines · Jarkko Sakkinen
Quoted 18 lines · Reinette Chatre

Hi Jarkko,

Quoted 15 lines · Jarkko Sakkinen
---
 tools/testing/selftests/sgx/alloc-error.bt | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 tools/testing/selftests/sgx/alloc-error.bt
tools/testing/selftests/sgx/alloc-error.bt new file +3 -0
new file mode 100644
index 000000000000..9268d50dea29
@@ -0,0 +1,7 @@
+kr:sgx_alloc_epc_page /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_alloc_epc_page: retval=%d\n", (int64)retval);
+}
Quoted 8 lines

tools/testing/selftests/sgx/alloc-error.bt line 7

Quoted 4 lines
tools/testing/selftests/sgx/alloc-error.bt new file +4 -0
+
+kr:sgx_encl_page_alloc /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_encl_page_alloc: retval=%d\n", (int64)retval);
+}

Could there be a snippet of comments in this new file to guide users on how to use this script?

Quoted 2 lines

Do not mean to be rude but I'm not sure what there is to guide but I'm open for ideas.

How about something like below in comments as part of the script:

"bpftrace script using kretprobe to trace returns of some key functions
in support of tracking allocation errors."

This is essentially in the subject line of the patch but that information will be lost when somebody looks at the script and tries to figure out what it is.

Reinette

author maintainer
#51 lore raw

Re: [PATCH 6/6] selftests/sgx: Add a bpftrace script for tracking allocation errors

Quoted 31 lines · Reinette Chatre

Hi Jarkko,

Quoted 28 lines · Jarkko Sakkinen
Quoted 26 lines · Reinette Chatre

Hi Jarkko,

Quoted 12 lines · Jarkko Sakkinen
---
 tools/testing/selftests/sgx/alloc-error.bt | 7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 tools/testing/selftests/sgx/alloc-error.bt
tools/testing/selftests/sgx/alloc-error.bt new file +0 -0
new file mode 100644
index 000000000000..9268d50dea29

tools/testing/selftests/sgx/alloc-error.bt line 7

Quoted 7 lines
tools/testing/selftests/sgx/alloc-error.bt new file +7 -0
@@ -0,0 +1,7 @@
+kr:sgx_alloc_epc_page /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_alloc_epc_page: retval=%d\n", (int64)retval);
+}
+
+kr:sgx_encl_page_alloc /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_encl_page_alloc: retval=%d\n", (int64)retval);
+}

Could there be a snippet of comments in this new file to guide users on how to use this script?

Quoted 7 lines
Quoted 2 lines

Do not mean to be rude but I'm not sure what there is to guide but I'm open for ideas.

How about something like below in comments as part of the script:

"bpftrace script using kretprobe to trace returns of some key functions
in support of tracking allocation errors."

I think comments that I put (before seeing) also
make it clear enough (not to say that what you
had not been a valid alternative).

BR, Jarkko

#52 lore raw

Re: [PATCH 6/6] selftests/sgx: Add a bpftrace script for tracking allocation errors

In reply to Jarkko Sakkinen

tools/testing/selftests/sgx/alloc-error.bt line 7

Quoted 13 lines · Jarkko Sakkinen
tools/testing/selftests/sgx/alloc-error.bt new file +7 -0
new file mode 100644
index 000000000000..9268d50dea29
@@ -0,0 +1,7 @@
+kr:sgx_alloc_epc_page /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_alloc_epc_page: retval=%d\n", (int64)retval);
+}
+
+kr:sgx_encl_page_alloc /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_encl_page_alloc: retval=%d\n", (int64)retval);
+}

I guess this doesn't _hurt_, but it's also not exactly the easiest way to get this done. You need a whole bpf toolchain. You could also just do:

perf probe 'sgx_encl_page_alloc%return $retval'

Even *that* can be replicated in a few scant lines of shell code echoing into /sys/kernel/debug/tracing.

author maintainer
#53 lore raw

Re: [PATCH 6/6] selftests/sgx: Add a bpftrace script for tracking allocation errors

Quoted 18 lines · Dave Hansen

tools/testing/selftests/sgx/alloc-error.bt line 7

Quoted 13 lines · Jarkko Sakkinen
tools/testing/selftests/sgx/alloc-error.bt new file +7 -0
new file mode 100644
index 000000000000..9268d50dea29
@@ -0,0 +1,7 @@
+kr:sgx_alloc_epc_page /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_alloc_epc_page: retval=%d\n", (int64)retval);
+}
+
+kr:sgx_encl_page_alloc /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_encl_page_alloc: retval=%d\n", (int64)retval);
+}

I guess this doesn't _hurt_, but it's also not exactly the easiest way to get this done. You need a whole bpf toolchain. You could also just do:

Quoted 4 lines

perf probe 'sgx_encl_page_alloc%return $retval'

Even *that* can be replicated in a few scant lines of shell code echoing into /sys/kernel/debug/tracing.

Thanks, I have not used perf that much. What if I replace this with a shell script using perf? How do you use that for two kretprobes?

BR, Jarkko

#54 lore raw

Re: [PATCH 6/6] selftests/sgx: Add a bpftrace script for tracking allocation errors

Quoted 10 lines · Jarkko Sakkinen
Quoted 10 lines

tools/testing/selftests/sgx/alloc-error.bt line 7

Quoted 7 lines
tools/testing/selftests/sgx/alloc-error.bt new file +7 -0
+kr:sgx_alloc_epc_page /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_alloc_epc_page: retval=%d\n", (int64)retval);
+}
+
+kr:sgx_encl_page_alloc /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_encl_page_alloc: retval=%d\n", (int64)retval);
+}

I guess this doesn't _hurt_, but it's also not exactly the easiest way to get this done. You need a whole bpf toolchain. You could also just do:

Quoted 7 lines
Quoted 4 lines

perf probe 'sgx_encl_page_alloc%return $retval'

Even *that* can be replicated in a few scant lines of shell code echoing into /sys/kernel/debug/tracing.

Thanks, I have not used perf that much. What if I replace this with a shell script using perf? How do you use that for two kretprobes?

The manpage is pretty good.

But, I'd proably be doing something along these lines:

	perf probe 'sgx_encl_page_alloc%return ret=$retval'
	perf record -e probe:sgx_encl_page_alloc -aR	\
		--filter='ret >= 0xwhatever' sleep 1
	perf script

There are probably shorter ways to do it, but I'm pretty sure that works.

author maintainer
#55 lore raw

Re: [PATCH 6/6] selftests/sgx: Add a bpftrace script for tracking allocation errors

Quoted 21 lines · Dave Hansen
Quoted 10 lines · Jarkko Sakkinen
Quoted 10 lines

tools/testing/selftests/sgx/alloc-error.bt line 7

Quoted 7 lines
tools/testing/selftests/sgx/alloc-error.bt new file +7 -0
+kr:sgx_alloc_epc_page /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_alloc_epc_page: retval=%d\n", (int64)retval);
+}
+
+kr:sgx_encl_page_alloc /(uint64)retval >= (uint64)(-4095)/ {
+		 printf("sgx_encl_page_alloc: retval=%d\n", (int64)retval);
+}

I guess this doesn't _hurt_, but it's also not exactly the easiest way to get this done. You need a whole bpf toolchain. You could also just do:

Quoted 7 lines
Quoted 4 lines

perf probe 'sgx_encl_page_alloc%return $retval'

Even *that* can be replicated in a few scant lines of shell code echoing into /sys/kernel/debug/tracing.

Thanks, I have not used perf that much. What if I replace this with a shell script using perf? How do you use that for two kretprobes?

The manpage is pretty good.

Quoted 8 lines

But, I'd proably be doing something along these lines:

	perf probe 'sgx_encl_page_alloc%return ret=$retval'
	perf record -e probe:sgx_encl_page_alloc -aR	\
		--filter='ret >= 0xwhatever' sleep 1
	perf script

There are probably shorter ways to do it, but I'm pretty sure that works.

Thanks, will give it a shot.

BR, Jarkko