x86/sgx: Free backing memory after faulting the enclave page
Subsystems (3)
Jarkko Sakkinen
maintainerPATCH 1/1 x86/sgx: Free backing memory after faulting the enclave page
The backing memory was not freed, after loading it back to the SGX reserved memory. This problem was not prevalent with the systems that were available at the time when SGX was first upstreamed, as the swapped memory could grow only up to 128 MB. However, Icelake Server can have gigabytes of SGX memory, and thus this has become a real bottleneck.
Free the swap space for other use by calling shmem_truncate_range(), when a page is faulted back.
--- arch/x86/kernel/cpu/sgx/encl.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/kernel/cpu/sgx/encl.c +4 -0
@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, {unsigned long va_offset = encl_page->desc &SGX_ENCL_PAGE_VA_OFFSET_MASK ;struct sgx_encl *encl = encl_page->encl; +struct inode *inode =file_inode (encl->backing);struct sgx_pageinfo pginfo;struct sgx_backing b;pgoff_t page_index;
@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,sgx_encl_put_backing (&b, false); +/* Free the backing memory. */ +shmem_truncate_range (inode,PFN_PHYS (page_index),PFN_PHYS (page_index) +PAGE_SIZE -1 ); +return ret; }
Signature · 1 line
2.32.0
Quoted 8 lines · Jarkko Sakkinen
The backing memory was not freed, after loading it back to the SGX reserved memory. This problem was not prevalent with the systems that were available at the time when SGX was first upstreamed, as the swapped memory could grow only up to 128 MB. However, Icelake Server can have gigabytes of SGX memory, and thus this has become a real bottleneck.
Free the swap space for other use by calling
shmem_truncate_range(), when a page is faulted back.
This needs a _bit_ more context. Could you include a few sentences about what backing memory is?
It's also not a big deal but it is nice to include something like:
This was found by inspection.
and:
Also, what is the end-user-visible effect of this bug? What would a user see if they were impacted by this? How did you determine that this fixed the issue? This memory will be recovered when the enclave is torn down, right?
Do we also need to deal with truncating the PCMD? (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)
Jarkko Sakkinen
maintainerQuoted 13 lines · Dave Hansen
Quoted 8 lines · Jarkko Sakkinen
The backing memory was not freed, after loading it back to the SGX reserved memory. This problem was not prevalent with the systems that were available at the time when SGX was first upstreamed, as the swapped memory could grow only up to 128 MB. However, Icelake Server can have gigabytes of SGX memory, and thus this has become a real bottleneck.
Free the swap space for other use by calling
shmem_truncate_range(), when a page is faulted back.This needs a _bit_ more context. Could you include a few sentences about what backing memory is?
Quoted 7 lines
It's also not a big deal but it is nice to include something like:
This was found by inspection.and:
Reported-by Dave Hansen <[email protected]>
Oops, I'm sorry, I was planning to add this but forgot to do it.
Quoted 4 lines
Also, what is the end-user-visible effect of this bug? What would a user see if they were impacted by this? How did you determine that this fixed the issue? This memory will be recovered when the enclave is torn down, right?
You're absolutely right.
I just realized how to make the whole thing concrete and reproduce OOM with the 128 MB EPC that I have in my i5-9600KF desktop. I'll just reconfigure my test VM to have sufficiently small amount of RAM, and come up with an appropriate workload.
Quoted 4 lines
Do we also need to deal with truncating the PCMD? (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)
Yes.
This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.
I guess it makes sense to put this into patch of its own.
/Jarkko
Jarkko Sakkinen
maintainerQuoted 4 lines · Jarkko Sakkinen
This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.
We have bookkeeping in place for this: encl->page_array.
/Jarkko
Quoted 10 lines · Jarkko Sakkinen
Quoted 4 lines
Do we also need to deal with truncating the PCMD? (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)
Yes.
This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.
I was thinking we could just read the page. If it's all 0's, truncate it.
Jarkko Sakkinen
maintainerQuoted 5 lines · Dave Hansen
Quoted 4 lines · Jarkko Sakkinen
Quoted 4 lines
Do we also need to deal with truncating the PCMD? (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)
Quoted 8 lines
Quoted 6 lines
Yes.
This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.
I was thinking we could just read the page. If it's all 0's, truncate it.
Hmm... did ELDU zero PCMD as a side-effect?
It should be fairly effecient just to check the pages by using encl->page_tree.
/Jarkko
Quoted 6 lines · Jarkko Sakkinen
Quoted 5 lines · Dave Hansen
Quoted 4 lines · Jarkko Sakkinen
Quoted 4 lines
Do we also need to deal with truncating the PCMD? (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)
Quoted 8 lines
Quoted 7 lines
Quoted 6 lines
Yes.
This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.
I was thinking we could just read the page. If it's all 0's, truncate it.
Hmm... did ELDU zero PCMD as a side-effect?
I don't think so, but there's nothing stopping us from doing it ourselves.
Quoted 2 lines
It should be fairly effecient just to check the pages by using
encl->page_tree.
That sounds more complicated and slower than what I suggested. You could even just check the refcount on the page. I _think_ page cache pages have a refcount of 2. So, look for the refcount that means "no more PCMD in this page", and just free it if so.
Jarkko Sakkinen
maintainerQuoted 9 lines · Dave Hansen
Quoted 8 lines · Jarkko Sakkinen
Quoted 7 lines · Dave Hansen
Quoted 6 lines · Jarkko Sakkinen
Quoted 4 lines
Do we also need to deal with truncating the PCMD? (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)
Yes.
Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines
This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.
I was thinking we could just read the page. If it's all 0's, truncate it.
Hmm... did ELDU zero PCMD as a side-effect?
I don't think so, but there's nothing stopping us from doing it ourselves.
Ok.
Quoted 7 lines
Quoted 2 lines
It should be fairly effecient just to check the pages by using
encl->page_tree.That sounds more complicated and slower than what I suggested. You could even just check the refcount on the page. I _think_ page cache pages have a refcount of 2. So, look for the refcount that means "no more PCMD in this page", and just free it if so.
Umh, so... there is total 32 PCMD's per one page.
/Jarkko
Quoted 7 lines · Jarkko Sakkinen
Quoted 6 lines
Quoted 2 lines
It should be fairly effecient just to check the pages by using
encl->page_tree.That sounds more complicated and slower than what I suggested. You could even just check the refcount on the page. I _think_ page cache pages have a refcount of 2. So, look for the refcount that means "no more PCMD in this page", and just free it if so.
Umh, so... there is total 32 PCMD's per one page.
When you place PCMD in a page, you do a get_page(). The refcount goes up by one. So, a PCMD page with one PCMD will (I think) have a refcount of 3. If you totally fill it up with 31 *more* PCMD entries, it will have a refcount of 34. You do *not* do a put_page() on the PCMD page at the end of the allocation phase.
When the backing storage is freed, you drop the refcount. So, going from 32 PCMD entries to 31 entries in a page, you go from 34->33.
When that refcount drops to 2, you know there is no more data in the page that's useful. At that point you can truncate it out of the backing storage.
There's no reason to scan the page, or a boatload of other metadata. Just keep a refcount. Just use the *existing* 'struct page' refcount.
Jarkko Sakkinen
maintainerQuoted 18 lines · Dave Hansen
Quoted 7 lines · Jarkko Sakkinen
Quoted 6 lines
Quoted 2 lines
It should be fairly effecient just to check the pages by using
encl->page_tree.That sounds more complicated and slower than what I suggested. You could even just check the refcount on the page. I _think_ page cache pages have a refcount of 2. So, look for the refcount that means "no more PCMD in this page", and just free it if so.
Umh, so... there is total 32 PCMD's per one page.
When you place PCMD in a page, you do a get_page(). The refcount goes up by one. So, a PCMD page with one PCMD will (I think) have a refcount of 3. If you totally fill it up with 31 *more* PCMD entries, it will have a refcount of 34. You do *not* do a
put_page()on the PCMD page at the end of the allocation phase.When the backing storage is freed, you drop the refcount. So, going from 32 PCMD entries to 31 entries in a page, you go from 34->33.
Quoted 6 lines
When that refcount drops to 2, you know there is no more data in the page that's useful. At that point you can truncate it out of the backing storage.
There's no reason to scan the page, or a boatload of other metadata. Just keep a refcount. Just use the *existing* 'struct page' refcount.
Right! Thank you, I'll use this approach, and check that the refcount actually behaves that way you described.
/Jarkko
Quoted 9 lines · Jarkko Sakkinen
Quoted 9 lines
When you place PCMD in a page, you do a get_page(). The refcount goes up by one. So, a PCMD page with one PCMD will (I think) have a refcount of 3. If you totally fill it up with 31 *more* PCMD entries, it will have a refcount of 34. You do *not* do a
put_page()on the PCMD page at the end of the allocation phase.When the backing storage is freed, you drop the refcount. So, going from 32 PCMD entries to 31 entries in a page, you go from 34->33.
Quoted 8 lines
Quoted 6 lines
When that refcount drops to 2, you know there is no more data in the page that's useful. At that point you can truncate it out of the backing storage.
There's no reason to scan the page, or a boatload of other metadata. Just keep a refcount. Just use the *existing* 'struct page' refcount.
Right! Thank you, I'll use this approach, and check that the refcount actually behaves that way you described.
Thinking about this a bit more... We don't want to use the normal get/put_page() refcount for this. If we do, it will pin the page while there is any data in it, preventing it from being reclaimed (swapped).
That isn't to say that we can't keep *a* refcount, just that we can't use the page refcount for this.
I still like the idea of just scanning the whole page for zeros.
Jarkko Sakkinen
maintainerQuoted 19 lines · Dave Hansen
Quoted 9 lines · Jarkko Sakkinen
Quoted 9 lines
When you place PCMD in a page, you do a get_page(). The refcount goes up by one. So, a PCMD page with one PCMD will (I think) have a refcount of 3. If you totally fill it up with 31 *more* PCMD entries, it will have a refcount of 34. You do *not* do a
put_page()on the PCMD page at the end of the allocation phase.When the backing storage is freed, you drop the refcount. So, going from 32 PCMD entries to 31 entries in a page, you go from 34->33.
Quoted 8 lines
Quoted 6 lines
When that refcount drops to 2, you know there is no more data in the page that's useful. At that point you can truncate it out of the backing storage.
There's no reason to scan the page, or a boatload of other metadata. Just keep a refcount. Just use the *existing* 'struct page' refcount.
Right! Thank you, I'll use this approach, and check that the refcount actually behaves that way you described.
Quoted 8 lines
Thinking about this a bit more... We don't want to use the normal get/put_page() refcount for this. If we do, it will pin the page while there is any data in it, preventing it from being reclaimed (swapped).
That isn't to say that we can't keep *a* refcount, just that we can't use the page refcount for this.
I still like the idea of just scanning the whole page for zeros.
I can try that route first. I like the property in zeroing that it has predicatable O(1) cost.
/Jarkko
Quoted 11 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/encl.c +1
-0@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, {unsigned long va_offset = encl_page->desc &SGX_ENCL_PAGE_VA_OFFSET_MASK ;struct sgx_encl *encl = encl_page->encl; +struct inode *inode =file_inode (encl->backing);struct sgx_pageinfo pginfo;struct sgx_backing b;pgoff_t page_index;
arch/x86/kernel/cpu/sgx/encl.c static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, · line 68
Quoted 8 lines
arch/x86/kernel/cpu/sgx/encl.c +3
-0@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,sgx_encl_put_backing (&b, false); +/* Free the backing memory. */ +shmem_truncate_range (inode,PFN_PHYS (page_index),PFN_PHYS (page_index) +PAGE_SIZE -1 ); +return ret; }
This also misses tearing down the backing storage if it is in place at sgx_encl_release().
Does a entry->epc_page==NULL page in there guarantee that it has backing storage?
Jarkko Sakkinen
maintainerQuoted 15 lines · Dave Hansen
Quoted 14 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/encl.c +1
-0@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, {unsigned long va_offset = encl_page->desc &SGX_ENCL_PAGE_VA_OFFSET_MASK ;struct sgx_encl *encl = encl_page->encl; +struct inode *inode =file_inode (encl->backing);struct sgx_pageinfo pginfo;struct sgx_backing b;pgoff_t page_index;@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,sgx_encl_put_backing (&b, false);
Quoted 8 lines
arch/x86/kernel/cpu/sgx/encl.c static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, · line 68
Quoted 5 lines
arch/x86/kernel/cpu/sgx/encl.c +3
-0+/* Free the backing memory. */ +shmem_truncate_range (inode,PFN_PHYS (page_index),PFN_PHYS (page_index) +PAGE_SIZE -1 ); +return ret; }This also misses tearing down the backing storage if it is in place at
sgx_encl_release().
Hmm... sgx_encl_release() does fput(). Isn't that enough to tear it down, or does it require explicit truncate, i.e. something like
shmem_truncate_range(file_inode(encl->backing), encl->base, encl->size - 1);
Quoted 2 lines
Does a entry->epc_page==NULL page in there guarantee that it has backing storage?
Yes, it is an invariant. That what I was thinking to use for PCMD: iterate 32 pages and check if they have a faulted page.
/Jarkko
Quoted 25 lines · Jarkko Sakkinen
Quoted 15 lines · Dave Hansen
Quoted 14 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/encl.c +1
-0@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, {unsigned long va_offset = encl_page->desc &SGX_ENCL_PAGE_VA_OFFSET_MASK ;struct sgx_encl *encl = encl_page->encl; +struct inode *inode =file_inode (encl->backing);struct sgx_pageinfo pginfo;struct sgx_backing b;pgoff_t page_index;@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,sgx_encl_put_backing (&b, false);Quoted 8 lines
arch/x86/kernel/cpu/sgx/encl.c static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, · line 68
Quoted 5 lines
arch/x86/kernel/cpu/sgx/encl.c +3
-0+/* Free the backing memory. */ +shmem_truncate_range (inode,PFN_PHYS (page_index),PFN_PHYS (page_index) +PAGE_SIZE -1 ); +return ret; }This also misses tearing down the backing storage if it is in place at
sgx_encl_release().
Quoted 4 lines
Hmm...
sgx_encl_release()doesfput(). Isn't that enough to tear it down, or does it require explicit truncate, i.e. something likeshmem_truncate_range(file_inode(encl->backing), encl->base, encl->size - 1);
That's true, the page cache should all be torn down along with the fput(). *But*, it would be a very nice property if the backing storage was empty by this point. It essentially ensures that no enclave-runtime cases missed truncating the backing storage away.
Quoted 5 lines
Quoted 2 lines
Does a entry->epc_page==NULL page in there guarantee that it has backing storage?
Yes, it is an invariant. That what I was thinking to use for PCMD: iterate 32 pages and check if they have a faulted page.
I think the rule should be that entry->epc_page==NULL enclave pages have backing storage. All entry->epc_page!=NULL do *not* have backing storage.
Jarkko Sakkinen
maintainerQuoted 29 lines · Dave Hansen
Quoted 28 lines · Jarkko Sakkinen
Quoted 15 lines · Dave Hansen
Quoted 14 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/encl.c +1
-0@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, {unsigned long va_offset = encl_page->desc &SGX_ENCL_PAGE_VA_OFFSET_MASK ;struct sgx_encl *encl = encl_page->encl; +struct inode *inode =file_inode (encl->backing);struct sgx_pageinfo pginfo;struct sgx_backing b;pgoff_t page_index;@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,sgx_encl_put_backing (&b, false);Quoted 8 lines
arch/x86/kernel/cpu/sgx/encl.c static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, · line 68
Quoted 5 lines
arch/x86/kernel/cpu/sgx/encl.c +3
-0+/* Free the backing memory. */ +shmem_truncate_range (inode,PFN_PHYS (page_index),PFN_PHYS (page_index) +PAGE_SIZE -1 ); +return ret; }This also misses tearing down the backing storage if it is in place at
sgx_encl_release().Hmm...
sgx_encl_release()doesfput(). Isn't that enough to tear it down, or does it require explicit truncate, i.e. something like
Quoted 6 lines
Quoted 1 line
shmem_truncate_range(file_inode(encl->backing), encl->base, encl->size - 1);That's true, the page cache should all be torn down along with the
fput(). *But*, it would be a very nice property if the backing storage was empty by this point. It essentially ensures that no enclave-runtime cases missed truncating the backing storage away.
What if an enclave is released a point when all of its pages are swapped out? Or even simpler case would an enclave that is larger than all of EPC.
What can be made sure is that for all pages, which are in EPC, the backing page is truncated.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Does a entry->epc_page==NULL page in there guarantee that it has backing storage?
Yes, it is an invariant. That what I was thinking to use for PCMD: iterate 32 pages and check if they have a faulted page.
I think the rule should be that entry->epc_page==NULL enclave pages have backing storage. All entry->epc_page!=NULL do *not* have backing storage.
Yes, that is the goal of this patch.
/Jarkko
Quoted 9 lines · Jarkko Sakkinen
Quoted 4 lines · Dave Hansen
That's true, the page cache should all be torn down along with the
fput(). *But*, it would be a very nice property if the backing storage was empty by this point. It essentially ensures that no enclave-runtime cases missed truncating the backing storage away.What if an enclave is released a point when all of its pages are swapped out? Or even simpler case would an enclave that is larger than all of EPC.
In this loop:
void sgx_encl_release(struct kref *ref)
{
...
xa_for_each(&encl->page_array, index, entry) {
if (entry->epc_page == NULL)
// truncate backing storageQuoted 2 lines
What can be made sure is that for all pages, which are in EPC, the backing page is truncated.
Right, and that should be utterly trivial to do.