x86/sgx: Synchronize encl->srcu in sgx_encl_release().
Jarkko Sakkinen
maintainerPATCH 1/1 x86/sgx: Synchronize encl->srcu in sgx_encl_release().
Each sgx_mmun_notifier_release() starts a grace period, which means that one extra synchronize_rcu() in sgx_encl_release(). Add it there.
sgx_release() has the loop that drains the list but with bad luck the entry is already gone from the list before that loop processes it.
--- arch/x86/kernel/cpu/sgx/encl.c | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/kernel/cpu/sgx/encl.c +7 -0
@@ -438,6 +438,13 @@ void sgx_encl_release(struct kref *ref)if (encl->backing)fput (encl->backing); +/* +* Each sgx_mmun_notifier_release() starts a grace period. Thus one +* "extra" synchronize_rcu() is required here. This can go undetected by +* sgx_release() when it drains the mm list. +*/ +synchronize_srcu (&encl->srcu); +cleanup_srcu_struct (&encl->srcu);WARN_ON_ONCE (!list_empty (&encl->mm_list));
Signature · 1 line
2.27.0
Quoted 1 line · Jarkko Sakkinen
Each
sgx_mmun_notifier_release()starts a grace period, which means that
Should be sgx_mmu_notifier_release(), here and in the comment.
Quoted 4 lines
one extra
synchronize_rcu()insgx_encl_release(). Add it there.
sgx_release()has the loop that drains the list but with bad luck the entry is already gone from the list before that loop processes it.
Why not include the actual analysis that "proves" the bug? The splat that Haitao reported would also be useful info.
Quoted 4 lines
Fixes 1728ab54b4be ("x86/sgx: Add a page reclaimer")Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Reported-by Sean Christopherson <[email protected]>
Haitao reported the bug, and for all intents and purposes provided the fix. I just did the analysis to verify that there was a legitimate bug and that the synchronization in sgx_encl_release() was indeed necessary.
Quoted 26 lines
Signed-off-by Jarkko Sakkinen <[email protected]>--- arch/x86/kernel/cpu/sgx/encl.c | 7 +++++++ 1 file changed, 7 insertions(+)arch/x86/kernel/cpu/sgx/encl.c +7
-0@@ -438,6 +438,13 @@ void sgx_encl_release(struct kref *ref)if (encl->backing)fput (encl->backing); +/* +* Each sgx_mmun_notifier_release() starts a grace period. Thus one +* "extra" synchronize_rcu() is required here. This can go undetected by +* sgx_release() when it drains the mm list. +*/ +synchronize_srcu (&encl->srcu); +cleanup_srcu_struct (&encl->srcu);WARN_ON_ONCE (!list_empty (&encl->mm_list));Signature · 1 line
2.27.0
Jarkko Sakkinen
maintainerQuoted 4 lines · Sean Christopherson
Quoted 1 line · Jarkko Sakkinen
Each
sgx_mmun_notifier_release()starts a grace period, which means thatShould be
sgx_mmu_notifier_release(), here and in the comment.
Thanks.
Quoted 7 lines
Quoted 4 lines
one extra
synchronize_rcu()insgx_encl_release(). Add it there.
sgx_release()has the loop that drains the list but with bad luck the entry is already gone from the list before that loop processes it.Why not include the actual analysis that "proves" the bug? The splat that Haitao reported would also be useful info.
True. I can include a snippet of dmesg to the commit message.
Quoted 8 lines
Quoted 4 lines
Fixes 1728ab54b4be ("x86/sgx: Add a page reclaimer")Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Reported-by Sean Christopherson <[email protected]>Haitao reported the bug, and for all intents and purposes provided the fix. I just did the analysis to verify that there was a legitimate bug and that the synchronization in
sgx_encl_release()was indeed necessary.
Good and valid point. The way I see it, the tags should be:
Haitao pointed out the bug but from your analysis I could resolve that this is the fix to implement, and was able to write the long description for the commit.
Does this make sense to you?
/Jarkko
Jarkko Sakkinen
maintainerQuoted 29 lines · Jarkko Sakkinen
Quoted 4 lines · Sean Christopherson
Quoted 1 line · Jarkko Sakkinen
Each
sgx_mmun_notifier_release()starts a grace period, which means thatShould be
sgx_mmu_notifier_release(), here and in the comment.Thanks.
Quoted 7 lines
Quoted 4 lines
one extra
synchronize_rcu()insgx_encl_release(). Add it there.
sgx_release()has the loop that drains the list but with bad luck the entry is already gone from the list before that loop processes it.Why not include the actual analysis that "proves" the bug? The splat that Haitao reported would also be useful info.
True. I can include a snippet of dmesg to the commit message.
Quoted 8 lines
Quoted 4 lines
Fixes 1728ab54b4be ("x86/sgx: Add a page reclaimer")Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Reported-by Sean Christopherson <[email protected]>Haitao reported the bug, and for all intents and purposes provided the fix. I just did the analysis to verify that there was a legitimate bug and that the synchronization in
sgx_encl_release()was indeed necessary.Good and valid point. The way I see it, the tags should be:
Quoted 8 lines
Reported-by Haitao Huang <[email protected]>Suggested-by Sean Christopherson <[email protected]>Haitao pointed out the bug but from your analysis I could resolve that this is the fix to implement, and was able to write the long description for the commit.
Does this make sense to you?
I'm sending v2 next week (this week on vacation).
/Jarkko
Quoted 42 lines · Jarkko Sakkinen
Quoted 41 lines · Jarkko Sakkinen
Quoted 2 lines · Sean Christopherson
Quoted 1 line · Jarkko Sakkinen
Each
sgx_mmun_notifier_release()starts a grace period, which meansthat
Quoted 2 lines
Should be
sgx_mmu_notifier_release(), here and in the comment.Thanks.
Quoted 3 lines
Quoted 3 lines
one extra
synchronize_rcu()insgx_encl_release(). Add it there.
sgx_release()has the loop that drains the list but with bad luckthe
Quoted 3 lines
Quoted 1 line
entry is already gone from the list before that loop processes it.
Why not include the actual analysis that "proves" the bug? The splat
that
Quoted 1 line
Haitao reported would also be useful info.
True. I can include a snippet of dmesg to the commit message.
Quoted 6 lines
Quoted 4 lines
Fixes 1728ab54b4be ("x86/sgx: Add a page reclaimer")Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Reported-by Sean Christopherson <[email protected]>Haitao reported the bug, and for all intents and purposes provided
the fix. I
Quoted 1 line
just did the analysis to verify that there was a legitimate bug and
that the
Quoted 1 line
synchronization in
sgx_encl_release()was indeed necessary.Good and valid point. The way I see it, the tags should be:
Reported-by Haitao Huang <[email protected]>Suggested-by Sean Christopherson <[email protected]>Haitao pointed out the bug but from your analysis I could resolve that this is the fix to implement, and was able to write the long description for the commit.
Quoted 5 lines
Quoted 1 line
Does this make sense to you?
I'm sending v2 next week (this week on vacation).
/Jarkko
I don't mind either how tags are assigned. But our testing reveals significant latency introduced in scenarios of heavy loading/unloading enclaves. synchronize_srcu_expedited fixed the issue. Please analyze and confirm if that's more appropriate than synchronize_srcu here.
Jarkko Sakkinen
maintainerQuoted 47 lines · Haitao Huang
Quoted 44 lines · Jarkko Sakkinen
Quoted 42 lines · Jarkko Sakkinen
Quoted 2 lines · Sean Christopherson
Quoted 1 line · Jarkko Sakkinen
Each
sgx_mmun_notifier_release()starts a grace period, whichmeans that
Quoted 2 lines
Should be
sgx_mmu_notifier_release(), here and in the comment.Thanks.
Quoted 3 lines
Quoted 3 lines
one extra
synchronize_rcu()insgx_encl_release(). Add it there.
sgx_release()has the loop that drains the list but with badluck the
Quoted 3 lines
Quoted 1 line
entry is already gone from the list before that loop processes it.
Why not include the actual analysis that "proves" the bug? The
splat that
Quoted 1 line
Haitao reported would also be useful info.
True. I can include a snippet of dmesg to the commit message.
Quoted 6 lines
Quoted 4 lines
Fixes 1728ab54b4be ("x86/sgx: Add a page reclaimer")Cc Borislav Petkov <[email protected]>Cc Dave Hansen <[email protected]>Reported-by Sean Christopherson <[email protected]>Haitao reported the bug, and for all intents and purposes provided
the fix. I
Quoted 1 line
just did the analysis to verify that there was a legitimate bug
and that the
Quoted 1 line
synchronization in
sgx_encl_release()was indeed necessary.Good and valid point. The way I see it, the tags should be:
Reported-by Haitao Huang <[email protected]>Suggested-by Sean Christopherson <[email protected]>Haitao pointed out the bug but from your analysis I could resolve that this is the fix to implement, and was able to write the long description for the commit.
Does this make sense to you?
Quoted 8 lines
Quoted 3 lines
I'm sending v2 next week (this week on vacation).
/Jarkko
I don't mind either how tags are assigned. But our testing reveals significant latency introduced in scenarios of heavy loading/unloading enclaves.
synchronize_srcu_expeditedfixed the issue. Please analyze and confirm if that's more appropriate thansynchronize_srcuhere.
I don't see any obvious reason why *_expedited could not be used here, as most of the time sync's are taken care of sgx_release() loop, and the final sync is with sgx_mmu_notifier_release(). More aggressive spinning should not do any harm here.
About the tags. I just try to get them right, and it is sometimes not straight-forward. So I guess, with all things considered, I'll put suggested-by from you. Once I get a refined patch out, try it out with your workloads and provide me tested-by, if it is working for you.
/Jarkko
Quoted 7 lines · Jarkko Sakkinen
Quoted 3 lines · Sean Christopherson
Haitao reported the bug, and for all intents and purposes provided the fix. I just did the analysis to verify that there was a legitimate bug and that the synchronization in
sgx_encl_release()was indeed necessary.Good and valid point. The way I see it, the tags should be:
Quoted 8 lines
Reported-by Haitao Huang <[email protected]>Suggested-by Sean Christopherson <[email protected]>Haitao pointed out the bug but from your analysis I could resolve that this is the fix to implement, and was able to write the long description for the commit.
Does this make sense to you?
Yep, works for me.
Jarkko Sakkinen
maintainerQuoted 11 lines · Sean Christopherson
Quoted 10 lines · Jarkko Sakkinen
Quoted 3 lines · Sean Christopherson
Haitao reported the bug, and for all intents and purposes provided the fix. I just did the analysis to verify that there was a legitimate bug and that the synchronization in
sgx_encl_release()was indeed necessary.Good and valid point. The way I see it, the tags should be:
Reported-by Haitao Huang <[email protected]>Suggested-by Sean Christopherson <[email protected]>
Quoted 7 lines
Quoted 5 lines
Haitao pointed out the bug but from your analysis I could resolve that this is the fix to implement, and was able to write the long description for the commit.
Does this make sense to you?
Yep, works for me.
I'll just add two suggested-by's. Process guide does not forbid that and it best describes matters.
/Jarkko