Skip to contentSkip to conversation

x86/sgx: Suppress WARN on inability to sanitize EPC if ksgxd is stopped

PATCH In review Sean Christopherson 5 participants 7 messages Jun 16, 2021 – Jun 25, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/sgx: Suppress WARN on inability to sanitize EPC if ksgxd is stopped

Don't WARN on having unsanitized EPC pages if ksgxd is stopped early, e.g. if sgx_init() realizes there will be no downstream consumers of EPC. If ksgxd is stopped early, EPC pages may be left on the dirty list, but that's ok because ksgxd is only stopped if SGX initialization failed or if the kernel is going down. In either case, the EPC won't be used.

This bug was exposed by the addition of KVM support, but has existed and was hittable since the original sanitization code was added. Prior to adding KVM support, if Launch Control was not fully enabled, e.g. when running on older hardware, sgx_init() bailed immediately before spawning ksgxd because X86_FEATURE_SGX was cleared if X86_FEATURE_SGX_LC was unsupported.

With KVM support, sgx_drv_init() handles the X86_FEATURE_SGX_LC check manually, so now there's any easy-to-hit case where sgx_init() will spawn ksgxd and _then_ fail to initialize, which results in sgx_init() stopping ksgxd before it finishes sanitizing the EPC.

Prior to KVM support, the bug was much harder to hit because it basically required char device registration to fail.

Fixes e7e0545299d8 ("x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections")

---

Lightly tested due to lack of hardware. I hacked the flow to verify that stopping early will leave work pending, and that rechecking should_stop() suppress the resulting WARN.

 arch/x86/kernel/cpu/sgx/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +1 -1
index ad904747419e..fbad2b9625a5 100644
@@ -425,7 +425,7 @@ static int ksgxd(void *p)
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
 	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	WARN_ON(!list_empty(&sgx_dirty_page_list) && !kthread_should_stop());
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())
Signature · 1 line
2.32.0.272.g935e593368-goog
#1 lore raw
Quoted 49 lines · Sean Christopherson

Don't WARN on having unsanitized EPC pages if ksgxd is stopped early, e.g. if sgx_init() realizes there will be no downstream consumers of EPC. If ksgxd is stopped early, EPC pages may be left on the dirty list, but that's ok because ksgxd is only stopped if SGX initialization failed or if the kernel is going down. In either case, the EPC won't be used.

This bug was exposed by the addition of KVM support, but has existed and was hittable since the original sanitization code was added. Prior to adding KVM support, if Launch Control was not fully enabled, e.g. when running on older hardware, sgx_init() bailed immediately before spawning ksgxd because X86_FEATURE_SGX was cleared if X86_FEATURE_SGX_LC was unsupported.

With KVM support, sgx_drv_init() handles the X86_FEATURE_SGX_LC check manually, so now there's any easy-to-hit case where sgx_init() will spawn ksgxd and _then_ fail to initialize, which results in sgx_init() stopping ksgxd before it finishes sanitizing the EPC.

Prior to KVM support, the bug was much harder to hit because it basically required char device registration to fail.

Fixes e7e0545299d8 ("x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections")

---

Lightly tested due to lack of hardware. I hacked the flow to verify that stopping early will leave work pending, and that rechecking should_stop() suppress the resulting WARN.

 arch/x86/kernel/cpu/sgx/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +1 -1
index ad904747419e..fbad2b9625a5 100644
@@ -425,7 +425,7 @@ static int ksgxd(void *p)
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
 	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	WARN_ON(!list_empty(&sgx_dirty_page_list) && !kthread_should_stop());
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())
Signature · 1 line
2.32.0.272.g935e593368-goog

I applied this patch on 5.13-rc6, and it no longer causes to trigger WARN_ON() on my NUC:

```

[    0.669411] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.669412] software IO TLB: mapped [mem 0x0000000017cb9000-0x000000001bcb9000] (64MB)
[    0.672788] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.672805] sgx: EPC section 0x30200000-0x35f7ffff
[    0.674239] Initialise system trusted keyrings
[    0.674254] Key type blacklist registered

```

Regards,
Du Cheng

maintainer
#2 lore raw
Quoted 65 lines · Du Cheng
Quoted 49 lines · Sean Christopherson

Don't WARN on having unsanitized EPC pages if ksgxd is stopped early, e.g. if sgx_init() realizes there will be no downstream consumers of EPC. If ksgxd is stopped early, EPC pages may be left on the dirty list, but that's ok because ksgxd is only stopped if SGX initialization failed or if the kernel is going down. In either case, the EPC won't be used.

This bug was exposed by the addition of KVM support, but has existed and was hittable since the original sanitization code was added. Prior to adding KVM support, if Launch Control was not fully enabled, e.g. when running on older hardware, sgx_init() bailed immediately before spawning ksgxd because X86_FEATURE_SGX was cleared if X86_FEATURE_SGX_LC was unsupported.

With KVM support, sgx_drv_init() handles the X86_FEATURE_SGX_LC check manually, so now there's any easy-to-hit case where sgx_init() will spawn ksgxd and _then_ fail to initialize, which results in sgx_init() stopping ksgxd before it finishes sanitizing the EPC.

Prior to KVM support, the bug was much harder to hit because it basically required char device registration to fail.

Fixes e7e0545299d8 ("x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections")

---

Lightly tested due to lack of hardware. I hacked the flow to verify that stopping early will leave work pending, and that rechecking should_stop() suppress the resulting WARN.

 arch/x86/kernel/cpu/sgx/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +1 -1
index ad904747419e..fbad2b9625a5 100644
@@ -425,7 +425,7 @@ static int ksgxd(void *p)
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
 	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	WARN_ON(!list_empty(&sgx_dirty_page_list) && !kthread_should_stop());
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())
Signature · 1 line
2.32.0.272.g935e593368-goog

I applied this patch on 5.13-rc6, and it no longer causes to trigger WARN_ON() on my NUC:

```

[    0.669411] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.669412] software IO TLB: mapped [mem 0x0000000017cb9000-0x000000001bcb9000] (64MB)
[    0.672788] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.672805] sgx: EPC section 0x30200000-0x35f7ffff
[    0.674239] Initialise system trusted keyrings
[    0.674254] Key type blacklist registered

```
Quoted 2 lines

Regards,
Du Cheng

Can you thus give a tested-by for this?

/Jarkko

Tested-by
#3 lore raw
Quoted 64 lines · Jarkko Sakkinen
Quoted 63 lines · Du Cheng
Quoted 49 lines · Sean Christopherson

Don't WARN on having unsanitized EPC pages if ksgxd is stopped early, e.g. if sgx_init() realizes there will be no downstream consumers of EPC. If ksgxd is stopped early, EPC pages may be left on the dirty list, but that's ok because ksgxd is only stopped if SGX initialization failed or if the kernel is going down. In either case, the EPC won't be used.

This bug was exposed by the addition of KVM support, but has existed and was hittable since the original sanitization code was added. Prior to adding KVM support, if Launch Control was not fully enabled, e.g. when running on older hardware, sgx_init() bailed immediately before spawning ksgxd because X86_FEATURE_SGX was cleared if X86_FEATURE_SGX_LC was unsupported.

With KVM support, sgx_drv_init() handles the X86_FEATURE_SGX_LC check manually, so now there's any easy-to-hit case where sgx_init() will spawn ksgxd and _then_ fail to initialize, which results in sgx_init() stopping ksgxd before it finishes sanitizing the EPC.

Prior to KVM support, the bug was much harder to hit because it basically required char device registration to fail.

Fixes e7e0545299d8 ("x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections")

---

Lightly tested due to lack of hardware. I hacked the flow to verify that stopping early will leave work pending, and that rechecking should_stop() suppress the resulting WARN.

 arch/x86/kernel/cpu/sgx/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +1 -1
index ad904747419e..fbad2b9625a5 100644
@@ -425,7 +425,7 @@ static int ksgxd(void *p)
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
 	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	WARN_ON(!list_empty(&sgx_dirty_page_list) && !kthread_should_stop());
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())
Signature · 1 line
2.32.0.272.g935e593368-goog

I applied this patch on 5.13-rc6, and it no longer causes to trigger WARN_ON() on my NUC:

```

[    0.669411] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    0.669412] software IO TLB: mapped [mem 0x0000000017cb9000-0x000000001bcb9000] (64MB)
[    0.672788] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.672805] sgx: EPC section 0x30200000-0x35f7ffff
[    0.674239] Initialise system trusted keyrings
[    0.674254] Key type blacklist registered
Quoted 8 lines
Quoted 4 lines

```

Regards,
Du Cheng

Can you thus give a tested-by for this?

/Jarkko

Certainly.

maintainer
#4 lore raw
In reply to Sean Christopherson
Quoted 39 lines · Sean Christopherson

Don't WARN on having unsanitized EPC pages if ksgxd is stopped early, e.g. if sgx_init() realizes there will be no downstream consumers of EPC. If ksgxd is stopped early, EPC pages may be left on the dirty list, but that's ok because ksgxd is only stopped if SGX initialization failed or if the kernel is going down. In either case, the EPC won't be used.

This bug was exposed by the addition of KVM support, but has existed and was hittable since the original sanitization code was added. Prior to adding KVM support, if Launch Control was not fully enabled, e.g. when running on older hardware, sgx_init() bailed immediately before spawning ksgxd because X86_FEATURE_SGX was cleared if X86_FEATURE_SGX_LC was unsupported.

With KVM support, sgx_drv_init() handles the X86_FEATURE_SGX_LC check manually, so now there's any easy-to-hit case where sgx_init() will spawn ksgxd and _then_ fail to initialize, which results in sgx_init() stopping ksgxd before it finishes sanitizing the EPC.

Prior to KVM support, the bug was much harder to hit because it basically required char device registration to fail.

Fixes e7e0545299d8 ("x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections")

---

Lightly tested due to lack of hardware. I hacked the flow to verify that stopping early will leave work pending, and that rechecking should_stop() suppress the resulting WARN.

 arch/x86/kernel/cpu/sgx/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +0 -0
index ad904747419e..fbad2b9625a5 100644

arch/x86/kernel/cpu/sgx/main.c static int ksgxd(void *p) · line 431

Quoted 8 lines
arch/x86/kernel/cpu/sgx/main.c +1 -1
@@ -425,7 +425,7 @@ static int ksgxd(void *p)
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
 	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	WARN_ON(!list_empty(&sgx_dirty_page_list) && !kthread_should_stop());
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())

Hmm, this looks weird. Why aren't we starting ksgxd only after *everything* has initialized successfully? I.e., after both kvm and native drivers' init functions have succeeded?

Then you won't have to do this kthread_should_stop() thing after the fact.

Btw, you have the same thing in the while loop's termination condition two lines down which, if I have to look at it later, would make me scratch head as to what TH is going on here.

Thx.

Signature · 4 lines
Regards/Gruss,
    Boris.

https://people.kernel.org/tglx/notes-about-netiquette
#5 lore raw
Quoted 3 lines · Borislav Petkov

Hmm, this looks weird. Why aren't we starting ksgxd only after *everything* has initialized successfully? I.e., after both kvm and native drivers' init functions have succeeded?

ksgxd has two roles. I think that's why it looks weird.

The obvious role is its use as the kswapd equivalent for SGX.

But, it's also used to speed up SGX initialization. It "sanitizes" the EPC asynchronously because it can take quite a while. That's why it gets launched off early. If it gets interrupted, that's when this warning can trigger.

I think you're suggesting that we just defer starting ksgxd until we *know* it won't be interrupted, basically moving sgx_page_reclaimer_init() down below sgx_drv_init() and sgx_vepc_init().

While I can see why it's best to get it going as early as possible, I don't see much going on in those init functions that would justify needing to fork off ksgx earlier. Am I missing anything?

maintainer Reviewed-by on v1 1/1
#6 lore raw
In reply to Sean Christopherson
Quoted 18 lines · Sean Christopherson

Don't WARN on having unsanitized EPC pages if ksgxd is stopped early, e.g. if sgx_init() realizes there will be no downstream consumers of EPC. If ksgxd is stopped early, EPC pages may be left on the dirty list, but that's ok because ksgxd is only stopped if SGX initialization failed or if the kernel is going down. In either case, the EPC won't be used.

This bug was exposed by the addition of KVM support, but has existed and was hittable since the original sanitization code was added. Prior to adding KVM support, if Launch Control was not fully enabled, e.g. when running on older hardware, sgx_init() bailed immediately before spawning ksgxd because X86_FEATURE_SGX was cleared if X86_FEATURE_SGX_LC was unsupported.

With KVM support, sgx_drv_init() handles the X86_FEATURE_SGX_LC check manually, so now there's any easy-to-hit case where sgx_init() will spawn ksgxd and _then_ fail to initialize, which results in sgx_init() stopping ksgxd before it finishes sanitizing the EPC.

Quoted 7 lines

Prior to KVM support, the bug was much harder to hit because it basically required char device registration to fail.

Fixes e7e0545299d8 ("x86/sgx: Initialize metadata for Enclave Page Cache (EPC) sections")

---

Thank you.

Quoted 24 lines

Lightly tested due to lack of hardware. I hacked the flow to verify that stopping early will leave work pending, and that rechecking should_stop() suppress the resulting WARN.

 arch/x86/kernel/cpu/sgx/main.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/kernel/cpu/sgx/main.c +1 -1
index ad904747419e..fbad2b9625a5 100644
@@ -425,7 +425,7 @@ static int ksgxd(void *p)
 	__sgx_sanitize_pages(&sgx_dirty_page_list);
 
 	/* sanity check: */
-	WARN_ON(!list_empty(&sgx_dirty_page_list));
+	WARN_ON(!list_empty(&sgx_dirty_page_list) && !kthread_should_stop());
 
 	while (!kthread_should_stop()) {
 		if (try_to_freeze())
Signature · 1 line
2.32.0.272.g935e593368-goog

/Jarkko