x86/sgx: WARN once if EREMOVE fails when killing an enclave
PATCH 1/1 x86/sgx: WARN once if EREMOVE fails when killing an enclave
WARN if EREMOVE fails when destroying an enclave. sgx_encl_release() uses the non-WARN __sgx_free_page() when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail as sgx_encl_destroy() is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.
--- arch/x86/kernel/cpu/sgx/encl.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/sgx/encl.c +9 -2
@@ -463,16 +463,23 @@ void sgx_encl_destroy(struct sgx_encl *encl)struct sgx_encl_page *entry;struct radix_tree_iter iter;void **slot; +int r;atomic_or (SGX_ENCL_DEAD , &encl->flags);radix_tree_for_each_slot (slot, &encl->page_tree, &iter,0 ) { entry = *slot;if (entry->epc_page) { -if (!__sgx_free_page (entry->epc_page)) { +/* +* Freeing the page can fail if it's in the process of +* being reclaimed (-EBUSY), but EREMOVE itself should +* not fail at this point. +*/ + r =__sgx_free_page (entry->epc_page); +WARN_ONCE (r >0 ,"sgx: EREMOVE returned %d (0x%x)" , r, r); +if (!r) { encl->secs_child_cnt--; entry->epc_page =NULL ; - } radix_tree_delete(&entry->encl->page_tree,
Signature · 1 line
2.22.0
Quoted 44 lines · Sean Christopherson
WARN if EREMOVE fails when destroying an enclave.
sgx_encl_release()uses the non-WARN__sgx_free_page()when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail assgx_encl_destroy()is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.Signed-off-by Sean Christopherson <[email protected]>--- arch/x86/kernel/cpu/sgx/encl.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)arch/x86/kernel/cpu/sgx/encl.c +9
-2@@ -463,16 +463,23 @@ void sgx_encl_destroy(struct sgx_encl *encl)struct sgx_encl_page *entry;struct radix_tree_iter iter;void **slot; +int r;atomic_or (SGX_ENCL_DEAD , &encl->flags);radix_tree_for_each_slot (slot, &encl->page_tree, &iter,0 ) { entry = *slot;if (entry->epc_page) { -if (!__sgx_free_page (entry->epc_page)) { +/* +* Freeing the page can fail if it's in the process of +* being reclaimed (-EBUSY), but EREMOVE itself should +* not fail at this point. +*/ + r =__sgx_free_page (entry->epc_page); +WARN_ONCE (r >0 ,"sgx: EREMOVE returned %d (0x%x)" , r, r); +if (!r) { encl->secs_child_cnt--; entry->epc_page =NULL ; - } radix_tree_delete(&entry->encl->page_tree,Signature · 1 line
2.22.0
Intended for v23, forgot to tag the subject...
Quoted 8 lines · Sean Christopherson
WARN if EREMOVE fails when destroying an enclave.
sgx_encl_release()uses the non-WARN__sgx_free_page()when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail assgx_encl_destroy()is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.Signed-off-by Sean Christopherson <[email protected]>
For me this concludes that I will manually convert all the call sites to use __sgx_free_page() and add appropriate warnings. I agree with Borislav's conclusions here.
/Jarkko
Quoted 8 lines · Jarkko Sakkinen
Quoted 7 lines · Sean Christopherson
WARN if EREMOVE fails when destroying an enclave.
sgx_encl_release()uses the non-WARN__sgx_free_page()when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail assgx_encl_destroy()is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.
Quoted 5 lines
Quoted 1 line
Signed-off-by Sean Christopherson <[email protected]>For me this concludes that I will manually convert all the call sites to use
__sgx_free_page()and add appropriate warnings. I agree with Borislav's conclusions here.
Argh, now we have a bunch of call sites that can silently leak EPC pages, and I'm seeing timeouts during testing that strongly suggest pages are being leaked...
Quoted 11 lines · Sean Christopherson
Quoted 10 lines · Jarkko Sakkinen
Quoted 8 lines · Sean Christopherson
WARN if EREMOVE fails when destroying an enclave.
sgx_encl_release()uses the non-WARN__sgx_free_page()when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail assgx_encl_destroy()is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.Signed-off-by Sean Christopherson <[email protected]>
Quoted 7 lines
Quoted 3 lines
For me this concludes that I will manually convert all the call sites to use
__sgx_free_page()and add appropriate warnings. I agree with Borislav's conclusions here.Argh, now we have a bunch of call sites that can silently leak EPC pages, and I'm seeing timeouts during testing that strongly suggest pages are being leaked...
Confirmed that we're leaking pages, but it's not related to the -EBUSY case in sgx_free_page(). Debug in progress...
As to the sgx_free_page() thing, I think we can invert the old WARN logic and make everyone happy. I'll send a patch.
Quoted 20 lines · Sean Christopherson
Quoted 11 lines · Sean Christopherson
Quoted 10 lines · Jarkko Sakkinen
Quoted 8 lines · Sean Christopherson
WARN if EREMOVE fails when destroying an enclave.
sgx_encl_release()uses the non-WARN__sgx_free_page()when freeing pages as some pages may be in the process of being reclaimed, i.e. are owned by the reclaimer. But EREMOVE should never fail assgx_encl_destroy()is only called when the enclave cannot have active threads, e.g. prior to EINIT and when the enclave is being released.Signed-off-by Sean Christopherson <[email protected]>Quoted 7 lines
Quoted 3 lines
For me this concludes that I will manually convert all the call sites to use
__sgx_free_page()and add appropriate warnings. I agree with Borislav's conclusions here.Argh, now we have a bunch of call sites that can silently leak EPC pages, and I'm seeing timeouts during testing that strongly suggest pages are being leaked...
Quoted 5 lines
Confirmed that we're leaking pages, but it's not related to the
-EBUSYcase insgx_free_page(). Debug in progress...As to the
sgx_free_page()thing, I think we can invert the old WARN logic and make everyone happy. I'll send a patch.
Figured out what's up. I'm testing in a VM with multiple EPC sections. Because of a change in v23[*], sgx_nr_free_pages is getting corrupted due to non-atomic concurrent writes. When it drops below 0 and wraps to a high value the swap thread stops reclaiming and things grind to a halt.