Skip to contentSkip to conversation

x86/sgx: WARN once if EREMOVE fails when killing an enclave

PATCH In review Sean Christopherson 2 participants 6 messages Oct 8, 2019 – Oct 10, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/sgx: WARN once if EREMOVE fails when killing an enclave

WARN if EREMOVE fails when destroying an enclave. sgx_encl_release() uses the non-WARN __sgx_free_page() when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail as sgx_encl_destroy() is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.

---
 arch/x86/kernel/cpu/sgx/encl.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/sgx/encl.c +9 -2
index 54ca827e68a9..a6786e7ae40e 100644
@@ -463,16 +463,23 @@ void sgx_encl_destroy(struct sgx_encl *encl)
 	struct sgx_encl_page *entry;
 	struct radix_tree_iter iter;
 	void **slot;
+	int r;
 
 	atomic_or(SGX_ENCL_DEAD, &encl->flags);
 
 	radix_tree_for_each_slot(slot, &encl->page_tree, &iter, 0) {
 		entry = *slot;
 		if (entry->epc_page) {
-			if (!__sgx_free_page(entry->epc_page)) {
+			/*
+			 * Freeing the page can fail if it's in the process of
+			 * being reclaimed (-EBUSY), but EREMOVE itself should
+			 * not fail at this point.
+			 */
+			r = __sgx_free_page(entry->epc_page);
+			WARN_ONCE(r > 0, "sgx: EREMOVE returned %d (0x%x)", r, r);
+			if (!r) {
 				encl->secs_child_cnt--;
 				entry->epc_page = NULL;
-
 			}
 
 			radix_tree_delete(&entry->encl->page_tree,
Signature · 1 line
2.22.0
#1 lore raw
Quoted 44 lines · Sean Christopherson

WARN if EREMOVE fails when destroying an enclave. sgx_encl_release() uses the non-WARN __sgx_free_page() when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail as sgx_encl_destroy() is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.

---
 arch/x86/kernel/cpu/sgx/encl.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/sgx/encl.c +9 -2
index 54ca827e68a9..a6786e7ae40e 100644
@@ -463,16 +463,23 @@ void sgx_encl_destroy(struct sgx_encl *encl)
 	struct sgx_encl_page *entry;
 	struct radix_tree_iter iter;
 	void **slot;
+	int r;
 
 	atomic_or(SGX_ENCL_DEAD, &encl->flags);
 
 	radix_tree_for_each_slot(slot, &encl->page_tree, &iter, 0) {
 		entry = *slot;
 		if (entry->epc_page) {
-			if (!__sgx_free_page(entry->epc_page)) {
+			/*
+			 * Freeing the page can fail if it's in the process of
+			 * being reclaimed (-EBUSY), but EREMOVE itself should
+			 * not fail at this point.
+			 */
+			r = __sgx_free_page(entry->epc_page);
+			WARN_ONCE(r > 0, "sgx: EREMOVE returned %d (0x%x)", r, r);
+			if (!r) {
 				encl->secs_child_cnt--;
 				entry->epc_page = NULL;
-
 			}
 
 			radix_tree_delete(&entry->encl->page_tree,
Signature · 1 line
2.22.0

Intended for v23, forgot to tag the subject...

#2 lore raw
In reply to Sean Christopherson
Quoted 8 lines · Sean Christopherson

WARN if EREMOVE fails when destroying an enclave. sgx_encl_release() uses the non-WARN __sgx_free_page() when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail as sgx_encl_destroy() is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.

For me this concludes that I will manually convert all the call sites to use __sgx_free_page() and add appropriate warnings. I agree with Borislav's conclusions here.

/Jarkko

#3 lore raw
Quoted 8 lines · Jarkko Sakkinen
Quoted 7 lines · Sean Christopherson

WARN if EREMOVE fails when destroying an enclave. sgx_encl_release() uses the non-WARN __sgx_free_page() when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail as sgx_encl_destroy() is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.

Quoted 5 lines
Quoted 1 line

For me this concludes that I will manually convert all the call sites to use __sgx_free_page() and add appropriate warnings. I agree with Borislav's conclusions here.

Argh, now we have a bunch of call sites that can silently leak EPC pages, and I'm seeing timeouts during testing that strongly suggest pages are being leaked...

#4 lore raw
Quoted 11 lines · Sean Christopherson
Quoted 10 lines · Jarkko Sakkinen
Quoted 8 lines · Sean Christopherson

WARN if EREMOVE fails when destroying an enclave. sgx_encl_release() uses the non-WARN __sgx_free_page() when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail as sgx_encl_destroy() is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.

Quoted 7 lines
Quoted 3 lines

For me this concludes that I will manually convert all the call sites to use __sgx_free_page() and add appropriate warnings. I agree with Borislav's conclusions here.

Argh, now we have a bunch of call sites that can silently leak EPC pages, and I'm seeing timeouts during testing that strongly suggest pages are being leaked...

Confirmed that we're leaking pages, but it's not related to the -EBUSY case in sgx_free_page(). Debug in progress...

As to the sgx_free_page() thing, I think we can invert the old WARN logic and make everyone happy. I'll send a patch.

#5 lore raw
Quoted 20 lines · Sean Christopherson
Quoted 11 lines · Sean Christopherson
Quoted 10 lines · Jarkko Sakkinen
Quoted 8 lines · Sean Christopherson

WARN if EREMOVE fails when destroying an enclave. sgx_encl_release() uses the non-WARN __sgx_free_page() when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail as sgx_encl_destroy() is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.

Quoted 7 lines
Quoted 3 lines

For me this concludes that I will manually convert all the call sites to use __sgx_free_page() and add appropriate warnings. I agree with Borislav's conclusions here.

Argh, now we have a bunch of call sites that can silently leak EPC pages, and I'm seeing timeouts during testing that strongly suggest pages are being leaked...

Quoted 5 lines

Confirmed that we're leaking pages, but it's not related to the -EBUSY case in sgx_free_page(). Debug in progress...

As to the sgx_free_page() thing, I think we can invert the old WARN logic and make everyone happy. I'll send a patch.

Figured out what's up. I'm testing in a VM with multiple EPC sections. Because of a change in v23[*], sgx_nr_free_pages is getting corrupted due to non-atomic concurrent writes. When it drops below 0 and wraps to a high value the swap thread stops reclaiming and things grind to a halt.

[*] https://patchwork.kernel.org/patch/11146733/#22887361