Skip to contentSkip to conversation

x86/sgx: Fix deadloop in __sgx_alloc_epc_page()

PATCH In review Aaron Lu 4 participants 13 messages Aug 29, 2024 – Sep 4, 2024

Subsystems (2)
lore
Share & tools
lore

Aaron Lu

author
#0 lore raw

PATCH 1/1 x86/sgx: Fix deadloop in __sgx_alloc_epc_page()

When current node doesn't have a EPC section configured by firmware and all other EPC sections memory are used up, CPU can stuck inside the while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because nid_of_current is not set in sgx_numa_mask.

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can be good for performance but that's not a requirement functionality wise.

Fixes 901ddbb9ecf5 ("x86/sgx: Add a basic NUMA allocation scheme to sgx_alloc_epc_page()")

---
This issue is found by Zhimin when doing internal testing and no external bug report has been sent out so there is no Closes: tag.

 arch/x86/kernel/cpu/sgx/main.c | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)
arch/x86/kernel/cpu/sgx/main.c +14 -13
index 1a000acd933a..694fcf7a5e3a 100644
@@ -475,24 +475,25 @@ struct sgx_epc_page *__sgx_alloc_epc_page(void)
 {
 	struct sgx_epc_page *page;
 	int nid_of_current = numa_node_id();
-	int nid = nid_of_current;
+	int nid_start, nid;
 
-	if (node_isset(nid_of_current, sgx_numa_mask)) {
-		page = __sgx_alloc_epc_page_from_node(nid_of_current);
-		if (page)
-			return page;
-	}
-
-	/* Fall back to the non-local NUMA nodes: */
-	while (true) {
-		nid = next_node_in(nid, sgx_numa_mask);
-		if (nid == nid_of_current)
-			break;
+	/*
+	 * Try local node first. If it doesn't have an EPC section,
+	 * fall back to the non-local NUMA nodes.
+	 */
+	if (node_isset(nid_of_current, sgx_numa_mask))
+		nid_start = nid_of_current;
+	else
+		nid_start = next_node_in(nid_of_current, sgx_numa_mask);
 
+	nid = nid_start;
+	do {
 		page = __sgx_alloc_epc_page_from_node(nid);
 		if (page)
 			return page;
-	}
+
+		nid = next_node_in(nid, sgx_numa_mask);
+	} while (nid != nid_start);
 
 	return ERR_PTR(-ENOMEM);
 }

base-commit: a85536e1bce722cb184abbac98068217874bdd6e

Signature · 1 line
2.45.2
Reviewed-by on v1 1/1
#1 lore raw
Quoted 8 lines · Aaron Lu

When current node doesn't have a EPC section configured by firmware and all other EPC sections memory are used up, CPU can stuck inside the while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because nid_of_current is not set in sgx_numa_mask.

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can

seup -> set up.

Quoted 6 lines

be good for performance but that's not a requirement functionality wise.

Fixes 901ddbb9ecf5 ("x86/sgx: Add a basic NUMA allocation scheme to sgx_alloc_epc_page()")
#2 lore raw
In reply to Aaron Lu

Actually run spell check this time ...

Quoted 1 line · Aaron Lu

When current node doesn't have a EPC section configured by firmware and

"current node" -> "the current node"

"a EPC section" -> "an EPC section"

Quoted 1 line

all other EPC sections memory are used up, CPU can stuck inside the

"EPC sections memory" -> "EPC sections"

"can stuck" -> "can get stuck"

Quoted 8 lines

while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because nid_of_current is not set in sgx_numa_mask.

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can be good for performance but that's not a requirement functionality wise.

[...]

author
#3 lore raw
Quoted 19 lines · Kai

Actually run spell check this time ...

Quoted 1 line · Aaron Lu

When current node doesn't have a EPC section configured by firmware and

"current node" -> "the current node"

"a EPC section" -> "an EPC section"

Quoted 1 line

all other EPC sections memory are used up, CPU can stuck inside the

"EPC sections memory" -> "EPC sections"

"can stuck" -> "can get stuck"

Quoted 4 lines

while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because nid_of_current is not set in sgx_numa_mask.

Quoted 6 lines
Quoted 4 lines

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can be good for performance but that's not a requirement functionality wise.

[...]

Thank you Kai for your detailed review, will reword the changelog according to your suggestions when sending the next version.

Acked-by on v1 1/1
#4 lore raw
In reply to Aaron Lu

Generally, I think it's a bad idea to refer to function names in subjects. This, for instance would be much more informative:

x86/sgx: Fix deadlock in SGX NUMA node search

Quoted 9 lines · Aaron Lu

When current node doesn't have a EPC section configured by firmware and all other EPC sections memory are used up, CPU can stuck inside the while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because nid_of_current is not set in sgx_numa_mask.

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can be good for performance but that's not a requirement functionality wise.

The changelog is a little rough, but I think Kai gave some good suggestions. The other thing you can do is dump the text in chatgpt (or whatever) and have it fix your grammar. It actually does a pretty decent job.

Also, you didn't say _how_ you fixed this. That needs to be in here. Something along the lines of:

Rework the loop to start and end on *a* node that has SGX memory. This avoids the deadlock looking for the current SGX- lacking node to show up in the loop when it never will.

The code looks fine, so feel free to add:

Also, I do think we should probably add some kind of sanity warning to the SGX code in another patch. If a node on an SGX system has CPUs and memory, it's very likely it will also have some EPC. It can be something soft like a pr_info(), but I think it would be nice to have.

author
#5 lore raw
Quoted 4 lines · Dave Hansen

Generally, I think it's a bad idea to refer to function names in subjects. This, for instance would be much more informative:

x86/sgx: Fix deadlock in SGX NUMA node search

Indeed, will use this as subject, thanks.

Quoted 7 lines · Aaron Lu
Quoted 6 lines

When current node doesn't have a EPC section configured by firmware and all other EPC sections memory are used up, CPU can stuck inside the while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because nid_of_current is not set in sgx_numa_mask.

Quoted 8 lines
Quoted 3 lines

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can be good for performance but that's not a requirement functionality wise.

The changelog is a little rough, but I think Kai gave some good suggestions. The other thing you can do is dump the text in chatgpt (or whatever) and have it fix your grammar. It actually does a pretty decent job.

Thanks for the suggestion.

Quoted 7 lines

Also, you didn't say _how_ you fixed this. That needs to be in here. Something along the lines of:

Rework the loop to start and end on *a* node that has SGX memory. This avoids the deadlock looking for the current SGX- lacking node to show up in the loop when it never will.

Will add this to the changelog, thanks for the write-up.

Quoted 4 lines

The code looks fine, so feel free to add:

Thanks.

Quoted 5 lines

Also, I do think we should probably add some kind of sanity warning to the SGX code in another patch. If a node on an SGX system has CPUs and memory, it's very likely it will also have some EPC. It can be something soft like a pr_info(), but I think it would be nice to have.

I think there are systems with valid reason to not setup an EPC section per node, e.g. a 8 sockets system with SNC=2, there would be a total of 16 nodes and it's not possible to have one EPC section per node because the upper limit of EPC sections is 8. I'm not sure a warning is appropriate here, what do you think?

#6 lore raw
Quoted 9 lines · Aaron Lu
Quoted 4 lines

Also, I do think we should probably add some kind of sanity warning to the SGX code in another patch. If a node on an SGX system has CPUs and memory, it's very likely it will also have some EPC. It can be something soft like a pr_info(), but I think it would be nice to have.

I think there are systems with valid reason to not setup an EPC section per node, e.g. a 8 sockets system with SNC=2, there would be a total of 16 nodes and it's not possible to have one EPC section per node because the upper limit of EPC sections is 8. I'm not sure a warning is appropriate here, what do you think?

While possible, those systems are pretty rare. I don't think a softly-worded pr_info() will scare anyone too much.

author
#7 lore raw
Quoted 5 lines · Dave Hansen
Quoted 4 lines · Aaron Lu
Quoted 4 lines

Also, I do think we should probably add some kind of sanity warning to the SGX code in another patch. If a node on an SGX system has CPUs and memory, it's very likely it will also have some EPC. It can be something soft like a pr_info(), but I think it would be nice to have.

Quoted 8 lines
Quoted 5 lines

I think there are systems with valid reason to not setup an EPC section per node, e.g. a 8 sockets system with SNC=2, there would be a total of 16 nodes and it's not possible to have one EPC section per node because the upper limit of EPC sections is 8. I'm not sure a warning is appropriate here, what do you think?

While possible, those systems are pretty rare. I don't think a softly-worded pr_info() will scare anyone too much.

Understood.

Maybe something like below?

From e49a78f27956b3d62c5ef962320e63dc3eeb897c Mon Sep 17 00:00:00 2001
From: Aaron Lu <[email protected]>
Date: Mon, 2 Sep 2024 11:46:07 +0800
Subject: [PATCH] x86/sgx: Log information when a node lacks an EPC section

For optimized performance, firmware typically distributes EPC sections evenly across different NUMA nodes. However, there are scenarios where a node may have both CPUs and memory but no EPC section configured. For example, in an 8-socket system with a Sub-Numa-Cluster=2 setup, there are a total of 16 nodes. Given that the maximum number of supported EPC sections is 8, it is simply not feasible to assign one EPC section to each node. This configuration is not incorrect - SGX will still operate correctly; it is just not optimized from a NUMA standpoint.

For this reason, log a message when a node with both CPUs and memory lacks an EPC section. This will provide users with a hint as to why they might be experiencing less-than-ideal performance when running SGX enclaves.

---
 arch/x86/kernel/cpu/sgx/main.c | 7 +++++++
 1 file changed, 7 insertions(+)
arch/x86/kernel/cpu/sgx/main.c +7 -0
index 694fcf7a5e3a..3a79105455f1 100644
@@ -848,6 +848,13 @@ static bool __init sgx_page_cache_init(void)
 		return false;
 	}
 
+	for_each_online_node(nid) {
+		if (!node_isset(nid, sgx_numa_mask) &&
+		    node_state(nid, N_MEMORY) && node_state(nid, N_CPU))
+			pr_info("node%d has both CPUs and memory but doesn't have an EPC section\n",
+				nid);
+	}
+
 	return true;
 }
 
Signature · 1 line
2.45.2
maintainer
#8 lore raw
In reply to Aaron Lu
Quoted 4 lines · Aaron Lu

When current node doesn't have a EPC section configured by firmware and all other EPC sections memory are used up, CPU can stuck inside the while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because

                                                     ~~~~

Oh *that* while loop ;-) Please be more specific.

Quoted 5 lines

nid_of_current is not set in sgx_numa_mask.

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can be good for performance but that's not a requirement functionality wise.

This lacks any description of what is done to __sgx_alloc_epc_page().

Quoted 5 lines
Fixes 901ddbb9ecf5 ("x86/sgx: Add a basic NUMA allocation scheme to sgx_alloc_epc_page()")

BR, Jarkko

author
#9 lore raw
Quoted 8 lines · Jarkko Sakkinen
Quoted 4 lines · Aaron Lu

When current node doesn't have a EPC section configured by firmware and all other EPC sections memory are used up, CPU can stuck inside the while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because

                                                     ~~~~

Oh *that* while loop ;-) Please be more specific.

What about:
Note how nid_of_current will never be equal to nid in the while loop that searches an available EPC page from remote nodes because nid_of_current is not set in sgx_numa_mask.

Quoted 7 lines
Quoted 5 lines

nid_of_current is not set in sgx_numa_mask.

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can be good for performance but that's not a requirement functionality wise.

This lacks any description of what is done to __sgx_alloc_epc_page().

Will add what Dave suggested on how the problem is fixed to the changelog.

Quoted 5 lines
Quoted 5 lines
Fixes 901ddbb9ecf5 ("x86/sgx: Add a basic NUMA allocation scheme to sgx_alloc_epc_page()")

Thanks,
Aaron

maintainer
#10 lore raw
Quoted 6 lines · Aaron Lu
Quoted 5 lines · Jarkko Sakkinen
Quoted 4 lines · Aaron Lu

When current node doesn't have a EPC section configured by firmware and all other EPC sections memory are used up, CPU can stuck inside the while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because

Quoted 8 lines
Quoted 3 lines
                                                     ~~~~

Oh *that* while loop ;-) Please be more specific.

What about:
Note how nid_of_current will never be equal to nid in the while loop that searches an available EPC page from remote nodes because nid_of_current is not set in sgx_numa_mask.

That would work I think!

Quoted 10 lines
Quoted 8 lines
Quoted 6 lines

nid_of_current is not set in sgx_numa_mask.

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can be good for performance but that's not a requirement functionality wise.

This lacks any description of what is done to __sgx_alloc_epc_page().

Will add what Dave suggested on how the problem is fixed to the changelog.

Great. I think the code change is correct reflecting these additions. I'll look the next version as a whole but with high probability I can ack that as long as the commit message has these updates.

Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Fixes 901ddbb9ecf5 ("x86/sgx: Add a basic NUMA allocation scheme to sgx_alloc_epc_page()")

Thanks,
Aaron

BR, Jarkko

author
#11 lore raw
Quoted 9 lines · Jarkko Sakkinen
Quoted 8 lines · Aaron Lu
Quoted 7 lines · Jarkko Sakkinen
Quoted 4 lines · Aaron Lu

When current node doesn't have a EPC section configured by firmware and all other EPC sections memory are used up, CPU can stuck inside the while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because

                                                     ~~~~
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Oh *that* while loop ;-) Please be more specific.

What about:
Note how nid_of_current will never be equal to nid in the while loop that searches an available EPC page from remote nodes because nid_of_current is not set in sgx_numa_mask.

That would work I think!

While rewriting the changelog, I find it more natural to explain this "while loop" when I first mentioned it, i.e.

When the current node doesn't have an EPC section configured by firmware and all other EPC sections are used up, CPU can get stuck inside the while loop that looks for an available EPC page from remote nodes indefinitely, leading to a soft lockup. Note how nid_of_current will never be equal to nid in that while loop because nid_of_current is not set in sgx_numa_mask.

I hope this looks fine to you.

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines

nid_of_current is not set in sgx_numa_mask.

Also worth mentioning is that it's perfectly fine for firmware to not seup an EPC section on a node. Setting an EPC section on each node can be good for performance but that's not a requirement functionality wise.

Quoted 7 lines
Quoted 3 lines
Quoted 1 line

This lacks any description of what is done to __sgx_alloc_epc_page().

Will add what Dave suggested on how the problem is fixed to the changelog.

Great. I think the code change is correct reflecting these additions. I'll look the next version as a whole but with high probability I can ack that as long as the commit message has these updates.

Thanks.

Quoted 11 lines
Quoted 9 lines
Quoted 6 lines
Quoted 6 lines
Fixes 901ddbb9ecf5 ("x86/sgx: Add a basic NUMA allocation scheme to sgx_alloc_epc_page()")

Thanks,
Aaron

BR, Jarkko

maintainer
#12 lore raw
Quoted 22 lines · Aaron Lu
Quoted 9 lines · Jarkko Sakkinen
Quoted 8 lines · Aaron Lu
Quoted 7 lines · Jarkko Sakkinen
Quoted 4 lines · Aaron Lu

When current node doesn't have a EPC section configured by firmware and all other EPC sections memory are used up, CPU can stuck inside the while loop in __sgx_alloc_epc_page() forever and soft lockup will happen. Note how nid_of_current will never equal to nid in that while loop because

                                                     ~~~~
Quoted 8 lines
Quoted 6 lines
Quoted 1 line

Oh *that* while loop ;-) Please be more specific.

What about:
Note how nid_of_current will never be equal to nid in the while loop that searches an available EPC page from remote nodes because nid_of_current is not set in sgx_numa_mask.

That would work I think!

While rewriting the changelog, I find it more natural to explain this "while loop" when I first mentioned it, i.e.

Quoted 8 lines

When the current node doesn't have an EPC section configured by firmware and all other EPC sections are used up, CPU can get stuck inside the while loop that looks for an available EPC page from remote nodes indefinitely, leading to a soft lockup. Note how nid_of_current will never be equal to nid in that while loop because nid_of_current is not set in sgx_numa_mask.

I hope this looks fine to you.

Yes, it is. I just want the storyline to the commit message as a reminder why we did this, that's all. It helps a lot later on when having to look into commit history.

BR, Jarkko