Skip to contentSkip to conversation

sched/numa, mm: do not promote folios to nodes not set N_MEMORY

PATCH In review Byungchul Park 5 participants 8 messages Feb 16, 2024 – Feb 19, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/1 sched/numa, mm: do not promote folios to nodes not set N_MEMORY

Changes from v2:
1. Rewrite the comment in code and the commit message becasue it turns out that this patch is not the real fix for the oops descriped. The real fix goes in another patch below:

https://lore.kernel.org/lkml/[email protected]/

Changes from v1:
1. Trim the verbose oops in the commit message. (feedbacked by Phil Auld)
2. Rewrite a comment in code. (feedbacked by Phil Auld)

--->8---
From 150af2f78e19217a1d03e47e3ee5279684590fb4 Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Fri, 16 Feb 2024 20:18:10 +0900
Subject: [PATCH v3] sched/numa, mm: do not promote folios to nodes not set N_MEMORY

A numa node might not have its local memory but CPUs. Promoting a folio to the node's local memory is nonsense. So avoid nodes not set N_MEMORY from getting promoted.

---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/fair.c +7 -0
index d7a3c63a2171..7ed9ef3c0134 100644
@@ -1828,6 +1828,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;
 
+	/*
+	 * A node of dst_nid might not have its local memory. Promoting
+	 * a folio to the node is meaningless.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
 	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.
Signature · 1 line
2.17.1
#1 lore raw
Quoted 4 lines · Byungchul Park

From 150af2f78e19217a1d03e47e3ee5279684590fb4 Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Fri, 16 Feb 2024 20:18:10 +0900
Subject: [PATCH v3] sched/numa, mm: do not promote folios to nodes not set N_MEMORY

"do not try to promote folios to memoryless nodes"

because AFAICS we are just trying.
Even if should_numa_migrate_memory() returns true, I assume that we will fail somewhere down the chain e.g: migrate_pages() when we see that this node does not any memory, right?

Quoted 3 lines

A numa node might not have its local memory but CPUs. Promoting a folio to the node's local memory is nonsense. So avoid nodes not set N_MEMORY from getting promoted.

If you talk about memoryless nodes everybody gets it better IMHO. "Memoryless nodes do not have any memory to migrate to, so stop trying it."

Quoted 10 lines
---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/fair.c +0 -0
index d7a3c63a2171..7ed9ef3c0134 100644

kernel/sched/fair.c bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, · line 1836

Quoted 9 lines
kernel/sched/fair.c +6 -0
@@ -1828,6 +1828,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;
 
+	/*
+	 * A node of dst_nid might not have its local memory. Promoting
+	 * a folio to the node is meaningless.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;

"Cannot migrate to memoryless nodes"

seems shorter and more clear.

So, what happens when we return true here? will we fail at migrate_pages() I guess? That is quite down the road so I guess this check can save us some time.

Signature · 2 lines
Oscar Salvador
SUSE Labs
Reviewed-by
#2 lore raw
Quoted 7 lines · Oscar Salvador
Quoted 4 lines · Byungchul Park

From 150af2f78e19217a1d03e47e3ee5279684590fb4 Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Fri, 16 Feb 2024 20:18:10 +0900
Subject: [PATCH v3] sched/numa, mm: do not promote folios to nodes not set N_MEMORY

"do not try to promote folios to memoryless nodes"

Thinking some more, promote might be misleading, just something like "do not try to migrate memory to memoryless nodes".

As this is not a bug fix but an optimization, as we will fail anyways in migrate_misplaced_folio() when migrate_balanced_pgdat() notices that we do not have any memory on that code.

With the other comments addressed:

Quoted 47 lines

because AFAICS we are just trying.
Even if should_numa_migrate_memory() returns true, I assume that we will fail somewhere down the chain e.g: migrate_pages() when we see that this node does not any memory, right?

Quoted 3 lines

A numa node might not have its local memory but CPUs. Promoting a folio to the node's local memory is nonsense. So avoid nodes not set N_MEMORY from getting promoted.

If you talk about memoryless nodes everybody gets it better IMHO. "Memoryless nodes do not have any memory to migrate to, so stop trying it."

Quoted 10 lines
---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/fair.c +0 -0
index d7a3c63a2171..7ed9ef3c0134 100644

kernel/sched/fair.c bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, · line 1836

Quoted 9 lines
kernel/sched/fair.c +6 -0
@@ -1828,6 +1828,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;
 
+	/*
+	 * A node of dst_nid might not have its local memory. Promoting
+	 * a folio to the node is meaningless.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;

"Cannot migrate to memoryless nodes"

seems shorter and more clear.

So, what happens when we return true here? will we fail at migrate_pages() I guess? That is quite down the road so I guess this check can save us some time.

Signature · 2 lines
Oscar Salvador
SUSE Labs
Signature · 2 lines
Oscar Salvador
SUSE Labs
#3 lore raw
Quoted 11 lines · Oscar Salvador
Quoted 7 lines · Oscar Salvador
Quoted 4 lines · Byungchul Park

From 150af2f78e19217a1d03e47e3ee5279684590fb4 Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Fri, 16 Feb 2024 20:18:10 +0900
Subject: [PATCH v3] sched/numa, mm: do not promote folios to nodes not set N_MEMORY

"do not try to promote folios to memoryless nodes"

Thinking some more, promote might be misleading, just something like "do not try to migrate memory to memoryless nodes".

Thank you. I will.

Quoted 7 lines

As this is not a bug fix but an optimization, as we will fail anyways in migrate_misplaced_folio() when migrate_balanced_pgdat() notices that we do not have any memory on that code.

With the other comments addressed:

Thank you for reviewing.

Byungchul

Quoted 50 lines
Quoted 46 lines

because AFAICS we are just trying.
Even if should_numa_migrate_memory() returns true, I assume that we will fail somewhere down the chain e.g: migrate_pages() when we see that this node does not any memory, right?

Quoted 3 lines

A numa node might not have its local memory but CPUs. Promoting a folio to the node's local memory is nonsense. So avoid nodes not set N_MEMORY from getting promoted.

If you talk about memoryless nodes everybody gets it better IMHO. "Memoryless nodes do not have any memory to migrate to, so stop trying it."

Quoted 10 lines
---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/fair.c +0 -0
index d7a3c63a2171..7ed9ef3c0134 100644

kernel/sched/fair.c bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, · line 1836

Quoted 9 lines
kernel/sched/fair.c +6 -0
@@ -1828,6 +1828,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;
 
+	/*
+	 * A node of dst_nid might not have its local memory. Promoting
+	 * a folio to the node is meaningless.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;

"Cannot migrate to memoryless nodes"

seems shorter and more clear.

So, what happens when we return true here? will we fail at migrate_pages() I guess? That is quite down the road so I guess this check can save us some time.

Signature · 2 lines
Oscar Salvador
SUSE Labs
Signature · 2 lines
Oscar Salvador
SUSE Labs
Reviewed-by
#4 lore raw
In reply to Oscar Salvador
Quoted 11 lines · Oscar Salvador
Quoted 7 lines · Oscar Salvador
Quoted 4 lines · Byungchul Park

From 150af2f78e19217a1d03e47e3ee5279684590fb4 Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Fri, 16 Feb 2024 20:18:10 +0900
Subject: [PATCH v3] sched/numa, mm: do not promote folios to nodes not set N_MEMORY

"do not try to promote folios to memoryless nodes"

Thinking some more, promote might be misleading, just something like "do not try to migrate memory to memoryless nodes".

Yes. Does this also want an unlikely()? Not that it would be measurable.

Quoted 3 lines

As this is not a bug fix but an optimization, as we will fail anyways in migrate_misplaced_folio() when migrate_balanced_pgdat() notices that we do not have any memory on that code.

This should be in the changelog and the subject is misleading as well.

Quoted 57 lines

With the other comments addressed:

Quoted 47 lines

because AFAICS we are just trying.
Even if should_numa_migrate_memory() returns true, I assume that we will fail somewhere down the chain e.g: migrate_pages() when we see that this node does not any memory, right?

Quoted 3 lines

A numa node might not have its local memory but CPUs. Promoting a folio to the node's local memory is nonsense. So avoid nodes not set N_MEMORY from getting promoted.

If you talk about memoryless nodes everybody gets it better IMHO. "Memoryless nodes do not have any memory to migrate to, so stop trying it."

Quoted 10 lines
---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/fair.c +0 -0
index d7a3c63a2171..7ed9ef3c0134 100644

kernel/sched/fair.c bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, · line 1836

Quoted 9 lines
kernel/sched/fair.c +6 -0
@@ -1828,6 +1828,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;
 
+	/*
+	 * A node of dst_nid might not have its local memory. Promoting
+	 * a folio to the node is meaningless.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;

"Cannot migrate to memoryless nodes"

seems shorter and more clear.

So, what happens when we return true here? will we fail at migrate_pages() I guess? That is quite down the road so I guess this check can save us some time.

--
Oscar Salvador
SUSE Labs

Signature · 2 lines
Oscar Salvador
SUSE Labs
#5 lore raw
In reply to Oscar Salvador
Quoted 7 lines · Oscar Salvador
Quoted 4 lines · Byungchul Park

From 150af2f78e19217a1d03e47e3ee5279684590fb4 Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Fri, 16 Feb 2024 20:18:10 +0900
Subject: [PATCH v3] sched/numa, mm: do not promote folios to nodes not set N_MEMORY

"do not try to promote folios to memoryless nodes"

Thank you. I will.

Quoted 4 lines

because AFAICS we are just trying.
Even if should_numa_migrate_memory() returns true, I assume that we will fail somewhere down the chain e.g: migrate_pages() when we see that this node does not any memory, right?

Yes.

Quoted 6 lines
Quoted 3 lines

A numa node might not have its local memory but CPUs. Promoting a folio to the node's local memory is nonsense. So avoid nodes not set N_MEMORY from getting promoted.

If you talk about memoryless nodes everybody gets it better IMHO. "Memoryless nodes do not have any memory to migrate to, so stop trying it."

Much better.

Quoted 22 lines
Quoted 10 lines
---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/fair.c +0 -0
index d7a3c63a2171..7ed9ef3c0134 100644

kernel/sched/fair.c bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, · line 1836

Quoted 9 lines
kernel/sched/fair.c +6 -0
@@ -1828,6 +1828,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;
 
+	/*
+	 * A node of dst_nid might not have its local memory. Promoting
+	 * a folio to the node is meaningless.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;

"Cannot migrate to memoryless nodes"

Quoted 1 line

seems shorter and more clear.

Agree.

Byungchul

Quoted 8 lines

So, what happens when we return true here? will we fail at migrate_pages() I guess? That is quite down the road so I guess this check can save us some time.

Signature · 2 lines
Oscar Salvador
SUSE Labs
Reviewed-by
#6 lore raw
Quoted 49 lines · On Mon, Feb 19, 2024 at 11:08:54AM +0900 Byungchul Park wrote:
Quoted 7 lines · Oscar Salvador
Quoted 4 lines · Byungchul Park

From 150af2f78e19217a1d03e47e3ee5279684590fb4 Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Fri, 16 Feb 2024 20:18:10 +0900
Subject: [PATCH v3] sched/numa, mm: do not promote folios to nodes not set N_MEMORY

"do not try to promote folios to memoryless nodes"

Thank you. I will.

Quoted 4 lines

because AFAICS we are just trying.
Even if should_numa_migrate_memory() returns true, I assume that we will fail somewhere down the chain e.g: migrate_pages() when we see that this node does not any memory, right?

Yes.

Quoted 6 lines
Quoted 3 lines

A numa node might not have its local memory but CPUs. Promoting a folio to the node's local memory is nonsense. So avoid nodes not set N_MEMORY from getting promoted.

If you talk about memoryless nodes everybody gets it better IMHO. "Memoryless nodes do not have any memory to migrate to, so stop trying it."

Much better.

Quoted 22 lines
Quoted 10 lines
---
 kernel/sched/fair.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/sched/fair.c +0 -0
index d7a3c63a2171..7ed9ef3c0134 100644

kernel/sched/fair.c bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, · line 1836

Quoted 9 lines
kernel/sched/fair.c +6 -0
@@ -1828,6 +1828,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;
 
+	/*
+	 * A node of dst_nid might not have its local memory. Promoting
+	 * a folio to the node is meaningless.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;

"Cannot migrate to memoryless nodes"

Quoted 4 lines
Quoted 1 line

seems shorter and more clear.

Agree.

We clearly have dst_cpu when we call this so I still think a check could be done farther up. But this one still looks reasonable to me.

Thanks,
Phil

Quoted 11 lines

Byungchul

Quoted 8 lines

So, what happens when we return true here? will we fail at migrate_pages() I guess? That is quite down the road so I guess this check can save us some time.

Signature · 2 lines
Oscar Salvador
SUSE Labs
Acked-by on v3 1/1
#7 lore raw
In reply to Byungchul Park
Quoted 18 lines · Byungchul Park

Changes from v2:
1. Rewrite the comment in code and the commit message becasue it turns out that this patch is not the real fix for the oops descriped. The real fix goes in another patch below:

https://lore.kernel.org/lkml/[email protected]/

Changes from v1:
1. Trim the verbose oops in the commit message. (feedbacked by Phil Auld)
2. Rewrite a comment in code. (feedbacked by Phil Auld)

--->8---
From 150af2f78e19217a1d03e47e3ee5279684590fb4 Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Fri, 16 Feb 2024 20:18:10 +0900
Subject: [PATCH v3] sched/numa, mm: do not promote folios to nodes not set N_MEMORY

Quoted 3 lines

A numa node might not have its local memory but CPUs. Promoting a folio to the node's local memory is nonsense. So avoid nodes not set N_MEMORY from getting promoted.

So there is no bug/panic that can be triggered and this is not a "fix" but an optimization?

Quoted 14 lines
---
  kernel/sched/fair.c | 7 +++++++
  1 file changed, 7 insertions(+)
kernel/sched/fair.c +0 -0
index d7a3c63a2171..7ed9ef3c0134 100644
@@ -1828,6 +1828,13 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
  	int dst_nid = cpu_to_node(dst_cpu);
  	int last_cpupid, this_cpupid;
 

kernel/sched/fair.c bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, · line 1840

Quoted 10 lines
kernel/sched/fair.c +7 -0
+	/*
+	 * A node of dst_nid might not have its local memory. Promoting
+	 * a folio to the node is meaningless.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
  	/*
  	 * The pages in slow memory node should be migrated according
  	 * to hot/cold instead of private/shared.
Signature · 3 lines
Cheers,

David / dhildenb