Skip to contentSkip to conversation

x86/sgx: Free backing memory after faulting the enclave page

PATCH Superseded Jarkko Sakkinen 2 participants 17 messages Nov 3, 2021 – Nov 8, 2021

Subsystems (3)

v1 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

Jarkko Sakkinen

author maintainer
#0 lore raw

PATCH 1/1 x86/sgx: Free backing memory after faulting the enclave page

The backing memory was not freed, after loading it back to the SGX reserved memory. This problem was not prevalent with the systems that were available at the time when SGX was first upstreamed, as the swapped memory could grow only up to 128 MB. However, Icelake Server can have gigabytes of SGX memory, and thus this has become a real bottleneck.

Free the swap space for other use by calling shmem_truncate_range(), when a page is faulted back.

Fixes 1728ab54b4be ("x86/sgx: Add a page reclaimer")
---
 arch/x86/kernel/cpu/sgx/encl.c | 4 ++++
 1 file changed, 4 insertions(+)
arch/x86/kernel/cpu/sgx/encl.c +4 -0
index 001808e3901c..f2d3f2e5028f 100644
@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 {
 	unsigned long va_offset = encl_page->desc & SGX_ENCL_PAGE_VA_OFFSET_MASK;
 	struct sgx_encl *encl = encl_page->encl;
+	struct inode *inode = file_inode(encl->backing);
 	struct sgx_pageinfo pginfo;
 	struct sgx_backing b;
 	pgoff_t page_index;
@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 
 	sgx_encl_put_backing(&b, false);
 
+	/* Free the backing memory. */
+	shmem_truncate_range(inode, PFN_PHYS(page_index), PFN_PHYS(page_index) + PAGE_SIZE - 1);
+
 	return ret;
 }
 
Signature · 1 line
2.32.0
#1 lore raw
Quoted 8 lines · Jarkko Sakkinen

The backing memory was not freed, after loading it back to the SGX reserved memory. This problem was not prevalent with the systems that were available at the time when SGX was first upstreamed, as the swapped memory could grow only up to 128 MB. However, Icelake Server can have gigabytes of SGX memory, and thus this has become a real bottleneck.

Free the swap space for other use by calling shmem_truncate_range(), when a page is faulted back.

This needs a _bit_ more context. Could you include a few sentences about what backing memory is?

It's also not a big deal but it is nice to include something like:

This was found by inspection.

and:

Also, what is the end-user-visible effect of this bug? What would a user see if they were impacted by this? How did you determine that this fixed the issue? This memory will be recovered when the enclave is torn down, right?

Do we also need to deal with truncating the PCMD? (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)

author maintainer
#2 lore raw
Quoted 13 lines · Dave Hansen
Quoted 8 lines · Jarkko Sakkinen

The backing memory was not freed, after loading it back to the SGX reserved memory. This problem was not prevalent with the systems that were available at the time when SGX was first upstreamed, as the swapped memory could grow only up to 128 MB.  However, Icelake Server can have gigabytes of SGX memory, and thus this has become a real bottleneck.

Free the swap space for other use by calling shmem_truncate_range(), when a page is faulted back.

This needs a _bit_ more context.  Could you include a few sentences about what backing memory is?

Quoted 7 lines

It's also not a big deal but it is nice to include something like:

        This was found by inspection.

and:

Oops, I'm sorry, I was planning to add this but forgot to do it.

Quoted 4 lines

Also, what is the end-user-visible effect of this bug?  What would a user see if they were impacted by this?  How did you determine that this fixed the issue?  This memory will be recovered when the enclave is torn down, right?

You're absolutely right.

I just realized how to make the whole thing concrete and reproduce OOM with the 128 MB EPC that I have in my i5-9600KF desktop. I'll just reconfigure my test VM to have sufficiently small amount of RAM, and come up with an appropriate workload.

Quoted 4 lines

Do we also need to deal with truncating the PCMD?  (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)

Yes.

This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.

I guess it makes sense to put this into patch of its own.

/Jarkko

author maintainer
#3 lore raw
Quoted 4 lines · Jarkko Sakkinen

This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.

We have bookkeeping in place for this: encl->page_array.

/Jarkko

#4 lore raw
In reply to Jarkko Sakkinen
Quoted 10 lines · Jarkko Sakkinen
Quoted 4 lines

Do we also need to deal with truncating the PCMD?  (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)

Yes.

This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.

I was thinking we could just read the page. If it's all 0's, truncate it.

author maintainer
#5 lore raw
Quoted 5 lines · Dave Hansen
Quoted 4 lines · Jarkko Sakkinen
Quoted 4 lines

Do we also need to deal with truncating the PCMD?  (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)

Quoted 8 lines
Quoted 6 lines

Yes.

This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.

I was thinking we could just read the page.  If it's all 0's, truncate it.

Hmm... did ELDU zero PCMD as a side-effect?

It should be fairly effecient just to check the pages by using encl->page_tree.

/Jarkko

#6 lore raw
Quoted 6 lines · Jarkko Sakkinen
Quoted 5 lines · Dave Hansen
Quoted 4 lines · Jarkko Sakkinen
Quoted 4 lines

Do we also need to deal with truncating the PCMD?  (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)

Quoted 8 lines
Quoted 7 lines
Quoted 6 lines

Yes.

This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.

I was thinking we could just read the page.  If it's all 0's, truncate it.

Hmm... did ELDU zero PCMD as a side-effect?

I don't think so, but there's nothing stopping us from doing it ourselves.

Quoted 2 lines

It should be fairly effecient just to check the pages by using encl->page_tree.

That sounds more complicated and slower than what I suggested. You could even just check the refcount on the page. I _think_ page cache pages have a refcount of 2. So, look for the refcount that means "no more PCMD in this page", and just free it if so.

author maintainer
#7 lore raw
Quoted 9 lines · Dave Hansen
Quoted 8 lines · Jarkko Sakkinen
Quoted 7 lines · Dave Hansen
Quoted 6 lines · Jarkko Sakkinen
Quoted 4 lines

Do we also need to deal with truncating the PCMD?  (For those watching along at home, there are two things SGX swaps to RAM: the actual page data and also some metadata that ensures page integrity and helps prevent things like rolling back to old versions of swapped pages)

Yes.

Quoted 8 lines
Quoted 6 lines
Quoted 5 lines
Quoted 4 lines

This can be achieved by iterating through all of the enclave pages, which share the same shmem page for storing their PCMD's, as the one being faulted back. If none of those pages is swapped, the PCMD page can safely truncated.

I was thinking we could just read the page.  If it's all 0's, truncate it.

Hmm... did ELDU zero PCMD as a side-effect?

I don't think so, but there's nothing stopping us from doing it ourselves.

Ok.

Quoted 7 lines
Quoted 2 lines

It should be fairly effecient just to check the pages by using encl->page_tree.

That sounds more complicated and slower than what I suggested. You could even just check the refcount on the page. I _think_ page cache pages have a refcount of 2. So, look for the refcount that means "no more PCMD in this page", and just free it if so.

Umh, so... there is total 32 PCMD's per one page.

/Jarkko

#8 lore raw
Quoted 7 lines · Jarkko Sakkinen
Quoted 6 lines
Quoted 2 lines

It should be fairly effecient just to check the pages by using encl->page_tree.

That sounds more complicated and slower than what I suggested. You could even just check the refcount on the page. I _think_ page cache pages have a refcount of 2. So, look for the refcount that means "no more PCMD in this page", and just free it if so.

Umh, so... there is total 32 PCMD's per one page.

When you place PCMD in a page, you do a get_page(). The refcount goes up by one. So, a PCMD page with one PCMD will (I think) have a refcount of 3. If you totally fill it up with 31 *more* PCMD entries, it will have a refcount of 34. You do *not* do a put_page() on the PCMD page at the end of the allocation phase.

When the backing storage is freed, you drop the refcount. So, going from 32 PCMD entries to 31 entries in a page, you go from 34->33.

When that refcount drops to 2, you know there is no more data in the page that's useful. At that point you can truncate it out of the backing storage.

There's no reason to scan the page, or a boatload of other metadata. Just keep a refcount. Just use the *existing* 'struct page' refcount.

author maintainer
#9 lore raw
Quoted 18 lines · Dave Hansen
Quoted 7 lines · Jarkko Sakkinen
Quoted 6 lines
Quoted 2 lines

It should be fairly effecient just to check the pages by using encl->page_tree.

That sounds more complicated and slower than what I suggested.  You could even just check the refcount on the page.  I _think_ page cache pages have a refcount of 2.  So, look for the refcount that means "no more PCMD in this page", and just free it if so.

Umh, so... there is total 32 PCMD's per one page.

When you place PCMD in a page, you do a get_page().  The refcount goes up by one.  So, a PCMD page with one PCMD will (I think) have a refcount of 3.  If you totally fill it up with 31 *more* PCMD entries, it will have a refcount of 34.  You do *not* do a put_page() on the PCMD page at the end of the allocation phase.

When the backing storage is freed, you drop the refcount.  So, going from 32 PCMD entries to 31 entries in a page, you go from 34->33.

Quoted 6 lines

When that refcount drops to 2, you know there is no more data in the page that's useful.  At that point you can truncate it out of the backing storage.

There's no reason to scan the page, or a boatload of other metadata. Just keep a refcount.  Just use the *existing* 'struct page' refcount.

Right! Thank you, I'll use this approach, and check that the refcount actually behaves that way you described.

/Jarkko

#10 lore raw
Quoted 9 lines · Jarkko Sakkinen
Quoted 9 lines

When you place PCMD in a page, you do a get_page().  The refcount goes up by one.  So, a PCMD page with one PCMD will (I think) have a refcount of 3.  If you totally fill it up with 31 *more* PCMD entries, it will have a refcount of 34.  You do *not* do a put_page() on the PCMD page at the end of the allocation phase.

When the backing storage is freed, you drop the refcount.  So, going from 32 PCMD entries to 31 entries in a page, you go from 34->33.

Quoted 8 lines
Quoted 6 lines

When that refcount drops to 2, you know there is no more data in the page that's useful.  At that point you can truncate it out of the backing storage.

There's no reason to scan the page, or a boatload of other metadata. Just keep a refcount.  Just use the *existing* 'struct page' refcount.

Right! Thank you, I'll use this approach, and check that the refcount actually behaves that way you described.

Thinking about this a bit more... We don't want to use the normal get/put_page() refcount for this. If we do, it will pin the page while there is any data in it, preventing it from being reclaimed (swapped).

That isn't to say that we can't keep *a* refcount, just that we can't use the page refcount for this.

I still like the idea of just scanning the whole page for zeros.

author maintainer
#11 lore raw
Quoted 19 lines · Dave Hansen
Quoted 9 lines · Jarkko Sakkinen
Quoted 9 lines

When you place PCMD in a page, you do a get_page().  The refcount goes up by one.  So, a PCMD page with one PCMD will (I think) have a refcount of 3.  If you totally fill it up with 31 *more* PCMD entries, it will have a refcount of 34.  You do *not* do a put_page() on the PCMD page at the end of the allocation phase.

When the backing storage is freed, you drop the refcount.  So, going from 32 PCMD entries to 31 entries in a page, you go from 34->33.

Quoted 8 lines
Quoted 6 lines

When that refcount drops to 2, you know there is no more data in the page that's useful.  At that point you can truncate it out of the backing storage.

There's no reason to scan the page, or a boatload of other metadata. Just keep a refcount.  Just use the *existing* 'struct page' refcount.

Right! Thank you, I'll use this approach, and check that the refcount actually behaves that way you described.

Quoted 8 lines

Thinking about this a bit more... We don't want to use the normal get/put_page() refcount for this. If we do, it will pin the page while there is any data in it, preventing it from being reclaimed (swapped).

That isn't to say that we can't keep *a* refcount, just that we can't use the page refcount for this.

I still like the idea of just scanning the whole page for zeros.

I can try that route first. I like the property in zeroing that it has predicatable O(1) cost.

/Jarkko

#12 lore raw
In reply to Jarkko Sakkinen
Quoted 11 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/encl.c +1 -0
@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 {
 	unsigned long va_offset = encl_page->desc & SGX_ENCL_PAGE_VA_OFFSET_MASK;
 	struct sgx_encl *encl = encl_page->encl;
+	struct inode *inode = file_inode(encl->backing);
 	struct sgx_pageinfo pginfo;
 	struct sgx_backing b;
 	pgoff_t page_index;

arch/x86/kernel/cpu/sgx/encl.c static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, · line 68

Quoted 8 lines
arch/x86/kernel/cpu/sgx/encl.c +3 -0
@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 
 	sgx_encl_put_backing(&b, false);
 
+	/* Free the backing memory. */
+	shmem_truncate_range(inode, PFN_PHYS(page_index), PFN_PHYS(page_index) + PAGE_SIZE - 1);
+
 	return ret;
 }

This also misses tearing down the backing storage if it is in place at sgx_encl_release().

Does a entry->epc_page==NULL page in there guarantee that it has backing storage?

author maintainer
#13 lore raw
Quoted 15 lines · Dave Hansen
Quoted 14 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/encl.c +1 -0
@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 {
 	unsigned long va_offset = encl_page->desc & SGX_ENCL_PAGE_VA_OFFSET_MASK;
 	struct sgx_encl *encl = encl_page->encl;
+	struct inode *inode = file_inode(encl->backing);
 	struct sgx_pageinfo pginfo;
 	struct sgx_backing b;
 	pgoff_t page_index;
@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 
 	sgx_encl_put_backing(&b, false);
 
Quoted 8 lines

arch/x86/kernel/cpu/sgx/encl.c static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, · line 68

Quoted 5 lines
arch/x86/kernel/cpu/sgx/encl.c +3 -0
+	/* Free the backing memory. */
+	shmem_truncate_range(inode, PFN_PHYS(page_index), PFN_PHYS(page_index) + PAGE_SIZE - 1);
+
 	return ret;
 }

This also misses tearing down the backing storage if it is in place at sgx_encl_release().

Hmm... sgx_encl_release() does fput(). Isn't that enough to tear it down, or does it require explicit truncate, i.e. something like

        shmem_truncate_range(file_inode(encl->backing), encl->base, encl->size - 1);
Quoted 2 lines

Does a entry->epc_page==NULL page in there guarantee that it has backing storage?

Yes, it is an invariant. That what I was thinking to use for PCMD: iterate 32 pages and check if they have a faulted page.

/Jarkko

#14 lore raw
Quoted 25 lines · Jarkko Sakkinen
Quoted 15 lines · Dave Hansen
Quoted 14 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/encl.c +1 -0
@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 {
 	unsigned long va_offset = encl_page->desc & SGX_ENCL_PAGE_VA_OFFSET_MASK;
 	struct sgx_encl *encl = encl_page->encl;
+	struct inode *inode = file_inode(encl->backing);
 	struct sgx_pageinfo pginfo;
 	struct sgx_backing b;
 	pgoff_t page_index;
@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 
 	sgx_encl_put_backing(&b, false);
 
Quoted 8 lines

arch/x86/kernel/cpu/sgx/encl.c static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, · line 68

Quoted 5 lines
arch/x86/kernel/cpu/sgx/encl.c +3 -0
+	/* Free the backing memory. */
+	shmem_truncate_range(inode, PFN_PHYS(page_index), PFN_PHYS(page_index) + PAGE_SIZE - 1);
+
 	return ret;
 }

This also misses tearing down the backing storage if it is in place at sgx_encl_release().

Quoted 4 lines

Hmm... sgx_encl_release() does fput(). Isn't that enough to tear it down, or does it require explicit truncate, i.e. something like

        shmem_truncate_range(file_inode(encl->backing), encl->base, encl->size - 1);

That's true, the page cache should all be torn down along with the fput(). *But*, it would be a very nice property if the backing storage was empty by this point. It essentially ensures that no enclave-runtime cases missed truncating the backing storage away.

Quoted 5 lines
Quoted 2 lines

Does a entry->epc_page==NULL page in there guarantee that it has backing storage?

Yes, it is an invariant. That what I was thinking to use for PCMD: iterate 32 pages and check if they have a faulted page.

I think the rule should be that entry->epc_page==NULL enclave pages have backing storage. All entry->epc_page!=NULL do *not* have backing storage.

author maintainer
#15 lore raw
Quoted 29 lines · Dave Hansen
Quoted 28 lines · Jarkko Sakkinen
Quoted 15 lines · Dave Hansen
Quoted 14 lines · Jarkko Sakkinen
arch/x86/kernel/cpu/sgx/encl.c +1 -0
@@ -22,6 +22,7 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 {
 	unsigned long va_offset = encl_page->desc & SGX_ENCL_PAGE_VA_OFFSET_MASK;
 	struct sgx_encl *encl = encl_page->encl;
+	struct inode *inode = file_inode(encl->backing);
 	struct sgx_pageinfo pginfo;
 	struct sgx_backing b;
 	pgoff_t page_index;
@@ -60,6 +61,9 @@ static int __sgx_encl_eldu(struct sgx_encl_page *encl_page,
 
 	sgx_encl_put_backing(&b, false);
 
Quoted 8 lines

arch/x86/kernel/cpu/sgx/encl.c static int __sgx_encl_eldu(struct sgx_encl_page *encl_page, · line 68

Quoted 5 lines
arch/x86/kernel/cpu/sgx/encl.c +3 -0
+	/* Free the backing memory. */
+	shmem_truncate_range(inode, PFN_PHYS(page_index), PFN_PHYS(page_index) + PAGE_SIZE - 1);
+
 	return ret;
 }

This also misses tearing down the backing storage if it is in place at sgx_encl_release().

Hmm... sgx_encl_release() does fput(). Isn't that enough to tear it down, or does it require explicit truncate, i.e. something like

Quoted 6 lines
Quoted 1 line
        shmem_truncate_range(file_inode(encl->backing), encl->base, encl->size - 1);

That's true, the page cache should all be torn down along with the fput(). *But*, it would be a very nice property if the backing storage was empty by this point. It essentially ensures that no enclave-runtime cases missed truncating the backing storage away.

What if an enclave is released a point when all of its pages are swapped out? Or even simpler case would an enclave that is larger than all of EPC.

What can be made sure is that for all pages, which are in EPC, the backing page is truncated.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Does a entry->epc_page==NULL page in there guarantee that it has backing storage?

Yes, it is an invariant. That what I was thinking to use for PCMD: iterate 32 pages and check if they have a faulted page.

I think the rule should be that entry->epc_page==NULL enclave pages have backing storage. All entry->epc_page!=NULL do *not* have backing storage.

Yes, that is the goal of this patch.

/Jarkko

#16 lore raw
Quoted 9 lines · Jarkko Sakkinen
Quoted 4 lines · Dave Hansen

That's true, the page cache should all be torn down along with the fput(). *But*, it would be a very nice property if the backing storage was empty by this point. It essentially ensures that no enclave-runtime cases missed truncating the backing storage away.

What if an enclave is released a point when all of its pages are swapped out? Or even simpler case would an enclave that is larger than all of EPC.

In this loop:

void sgx_encl_release(struct kref *ref)

{
	...
        xa_for_each(&encl->page_array, index, entry) {
		if (entry->epc_page == NULL)
			// truncate backing storage
Quoted 2 lines

What can be made sure is that for all pages, which are in EPC, the backing page is truncated.

Right, and that should be utterly trivial to do.