Skip to contentSkip to conversation

sched/numa, mm: do not try to migrate memory to memoryless nodes

PATCH Applied Byungchul Park 4 participants 11 messages Feb 19, 2024 – Feb 20, 2024

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 sched/numa, mm: do not try to migrate memory to memoryless nodes

Changes from v3:
1. Rewrite the comment in code and the commit message to make it more clear. (feedbacked by Oscar Salvador)
2. Add "Reviewed-by: Oscar Salvador <[email protected]>"

Changes from v2:
1. Rewrite the comment in code and the commit message becasue it turns out that this patch is not the real fix for the oops descriped. The real fix goes in another patch below:

https://lore.kernel.org/lkml/[email protected]/

Changes from v1:
1. Trim the verbose oops in the commit message. (feedbacked by Phil Auld)
2. Rewrite a comment in code. (feedbacked by Phil Auld)

--->8---
From 98f5d472c08e3ed5b9b6543290d392a2e50fcf3c Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Mon, 19 Feb 2024 13:10:47 +0900
Subject: [PATCH v4] sched/numa, mm: do not try to migrate memory to memoryless nodes

Memoryless nodes do not have any memory to migrate to, so stop trying it.

---
 kernel/sched/fair.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/fair.c +6 -0
index d7a3c63a2171..3e3b44ae72d1 100644
@@ -1828,6 +1828,12 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;
 
+	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
 	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.
Signature · 1 line
2.17.1
Reviewed-by on v4 1/1
#1 lore raw

Hi, Byungchul,

Quoted 39 lines · Byungchul Park

Changes from v3:
1. Rewrite the comment in code and the commit message to make it more clear. (feedbacked by Oscar Salvador)
2. Add "Reviewed-by: Oscar Salvador <[email protected]>"

Changes from v2:
1. Rewrite the comment in code and the commit message becasue it turns out that this patch is not the real fix for the oops descriped. The real fix goes in another patch below:

https://lore.kernel.org/lkml/[email protected]/

Changes from v1:
1. Trim the verbose oops in the commit message. (feedbacked by Phil Auld)
2. Rewrite a comment in code. (feedbacked by Phil Auld)

--->8---

Quoted 1 line

From 98f5d472c08e3ed5b9b6543290d392a2e50fcf3c Mon Sep 17 00:00:00 2001

From: Byungchul Park <[email protected]>
Date: Mon, 19 Feb 2024 13:10:47 +0900
Subject: [PATCH v4] sched/numa, mm: do not try to migrate memory to memoryless nodes

Memoryless nodes do not have any memory to migrate to, so stop trying it.

---
 kernel/sched/fair.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/fair.c +0 -0
index d7a3c63a2171..3e3b44ae72d1 100644
@@ -1828,6 +1828,12 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;

kernel/sched/fair.c bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio, · line 1839

Quoted 10 lines
kernel/sched/fair.c +6 -0
 
+	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
 	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.

Good catch!

IIUC, you will use patch as fix to the issue in

https://lore.kernel.org/lkml/[email protected]/

If so, we need the Fixes: tag to make it land in -stable properly.

Feel free to add

maintainer Reviewed-by Oscar Salvador Reviewed-by Huang, Ying Reviewed-by Phil Auld
#2 lore raw
Quoted 12 lines · Ying

Review

Quoted 9 lines
(quoted diff) +6 -0
+	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
 	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.

Good catch!

Quoted 5 lines

IIUC, you will use patch as fix to the issue in

https://lore.kernel.org/lkml/[email protected]/

If so, we need the Fixes: tag to make it land in -stable properly.

Yes, this changelog is missing rather a lot of important information.

I pulled together the below, please check.

From: Byungchul Park <[email protected]>
Subject: sched/numa, mm: do not try to migrate memory to memoryless nodes
Date: Mon, 19 Feb 2024 13:10:47 +0900

With numa balancing on, when a numa system is running where a numa node doesn't have its local memory so it has no managed zones, the following oops has been observed. It's because wakeup_kswapd() is called with a wrong zone index, -1. Fixed it by checking the index before calling wakeup_kswapd().

Quoted 36 lines
BUG: unable to handle page fault for address: 00000000000033f3
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 2 PID: 895 Comm: masim Not tainted 6.6.0-dirty #255
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
   rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
RIP: 0010:wakeup_kswapd (./linux/mm/vmscan.c:7812)
Code: (omitted)
RSP: 0000:ffffc90004257d58 EFLAGS: 00010286
RAX: ffffffffffffffff RBX: ffff88883fff0480 RCX: 0000000000000003
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88883fff0480
RBP: ffffffffffffffff R08: ff0003ffffffffff R09: ffffffffffffffff
R10: ffff888106c95540 R11: 0000000055555554 R12: 0000000000000003
R13: 0000000000000000 R14: 0000000000000000 R15: ffff88883fff0940
FS:  00007fc4b8124740(0000) GS:ffff888827c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000033f3 CR3: 000000026cc08004 CR4: 0000000000770ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
 <TASK>

? __die
? page_fault_oops
? __pte_offset_map_lock
? exc_page_fault
? asm_exc_page_fault
? wakeup_kswapd
migrate_misplaced_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault

Quoted 9 lines
RIP: 0033:0x55b897ba0808
Code: (omitted)
RSP: 002b:00007ffeefa821a0 EFLAGS: 00010287
RAX: 000055b89983acd0 RBX: 00007ffeefa823f8 RCX: 000055b89983acd0
RDX: 00007fc2f8122010 RSI: 0000000000020000 RDI: 000055b89983acd0
RBP: 00007ffeefa821a0 R08: 0000000000000037 R09: 0000000000000075
R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
R13: 00007ffeefa82410 R14: 000055b897ba5dd8 R15: 00007fc4b8340000
 </TASK>

Fix this by avoiding any attempt to migrate memory to memoryless nodes.

Fixes c574bbe917036 ("NUMA balancing: optimize page placement for memory tiering system")
Cc Benjamin Segall <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc Vincent Guittot <[email protected]>

---

 kernel/sched/fair.c |    6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/fair.c renamed from kernel/sched/fair.c~sched-numa-mm-do-not-try-to-migrate-memory-to-memoryless-nodes +6 -0
@@ -1831,6 +1831,12 @@ bool should_numa_migrate_memory(struct t
 	int last_cpupid, this_cpupid;
 
 	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
+	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.
 	 */

_

#3 lore raw
Quoted 14 lines · Andrew Morton
Quoted 12 lines · Ying

kernel/sched/fair.c bool should_numa_migrate_memory(struct t · line 1841

Quoted 9 lines
kernel/sched/fair.c renamed from kernel/sched/fair.c~sched-numa-mm-do-not-try-to-migrate-memory-to-memoryless-nodes +6 -0
+	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
 	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.

Good catch!

Quoted 7 lines
Quoted 5 lines

IIUC, you will use patch as fix to the issue in

https://lore.kernel.org/lkml/[email protected]/

If so, we need the Fixes: tag to make it land in -stable properly.

Yes, this changelog is missing rather a lot of important information.

This is not the root cause fix any more but just optimization. That's why I didn't add Fixes: tag and cc [email protected] in here.

Instead, I added Fixes: tag and cc'ed [email protected] in the real fix patch. check the following link please:

https://lore.kernel.org/lkml/[email protected]/

Byungchul

Quoted 101 lines

I pulled together the below, please check.

From: Byungchul Park <[email protected]>
Subject: sched/numa, mm: do not try to migrate memory to memoryless nodes
Date: Mon, 19 Feb 2024 13:10:47 +0900

With numa balancing on, when a numa system is running where a numa node doesn't have its local memory so it has no managed zones, the following oops has been observed. It's because wakeup_kswapd() is called with a wrong zone index, -1. Fixed it by checking the index before calling wakeup_kswapd().

Quoted 36 lines
BUG: unable to handle page fault for address: 00000000000033f3
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 2 PID: 895 Comm: masim Not tainted 6.6.0-dirty #255
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
   rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
RIP: 0010:wakeup_kswapd (./linux/mm/vmscan.c:7812)
Code: (omitted)
RSP: 0000:ffffc90004257d58 EFLAGS: 00010286
RAX: ffffffffffffffff RBX: ffff88883fff0480 RCX: 0000000000000003
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88883fff0480
RBP: ffffffffffffffff R08: ff0003ffffffffff R09: ffffffffffffffff
R10: ffff888106c95540 R11: 0000000055555554 R12: 0000000000000003
R13: 0000000000000000 R14: 0000000000000000 R15: ffff88883fff0940
FS:  00007fc4b8124740(0000) GS:ffff888827c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000033f3 CR3: 000000026cc08004 CR4: 0000000000770ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
 <TASK>

? __die
? page_fault_oops
? __pte_offset_map_lock
? exc_page_fault
? asm_exc_page_fault
? wakeup_kswapd
migrate_misplaced_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault

Quoted 9 lines
RIP: 0033:0x55b897ba0808
Code: (omitted)
RSP: 002b:00007ffeefa821a0 EFLAGS: 00010287
RAX: 000055b89983acd0 RBX: 00007ffeefa823f8 RCX: 000055b89983acd0
RDX: 00007fc2f8122010 RSI: 0000000000020000 RDI: 000055b89983acd0
RBP: 00007ffeefa821a0 R08: 0000000000000037 R09: 0000000000000075
R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
R13: 00007ffeefa82410 R14: 000055b897ba5dd8 R15: 00007fc4b8340000
 </TASK>

Fix this by avoiding any attempt to migrate memory to memoryless nodes.

Fixes c574bbe917036 ("NUMA balancing: optimize page placement for memory tiering system")
Cc Benjamin Segall <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc Vincent Guittot <[email protected]>

---

 kernel/sched/fair.c |    6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/fair.c renamed from kernel/sched/fair.c~sched-numa-mm-do-not-try-to-migrate-memory-to-memoryless-nodes +6 -0
@@ -1831,6 +1831,12 @@ bool should_numa_migrate_memory(struct t
 	int last_cpupid, this_cpupid;
 
 	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
+	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.
 	 */

_

maintainer
#4 lore raw
Quoted 9 lines · On Tue, 20 Feb 2024 10:53:43 +0900 Byungchul Park <[email protected]> wrote:
Quoted 7 lines
Quoted 5 lines

IIUC, you will use patch as fix to the issue in

https://lore.kernel.org/lkml/[email protected]/

If so, we need the Fixes: tag to make it land in -stable properly.

Yes, this changelog is missing rather a lot of important information.

This is not the root cause fix any more but just optimization.

It would have been helpful to have told us this in the changelog :(

Quoted 7 lines

That's
why I didn't add Fixes: tag and cc [email protected] in here.

Instead, I added Fixes: tag and cc'ed [email protected] in the real fix patch. check the following link please:

https://lore.kernel.org/lkml/[email protected]/

But doesn't this patch "sched/numa, mm: do not try to migrate memory to memoryless nodes" also fix the bug? Do we truly need both?

#5 lore raw
Quoted 6 lines · Andrew Morton
Quoted 4 lines · On Tue, 20 Feb 2024 10:53:43 +0900 Byungchul Park <[email protected]> wrote:
Quoted 4 lines
Quoted 4 lines

IIUC, you will use patch as fix to the issue in

https://lore.kernel.org/lkml/[email protected]/

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
Quoted 1 line

If so, we need the Fixes: tag to make it land in -stable properly.

Yes, this changelog is missing rather a lot of important information.

This is not the root cause fix any more but just optimization.

It would have been helpful to have told us this in the changelog :(

Sorry for that and making you guys confused.

Quoted 10 lines
Quoted 7 lines

That's
why I didn't add Fixes: tag and cc [email protected] in here.

Instead, I added Fixes: tag and cc'ed [email protected] in the real fix patch. check the following link please:

https://lore.kernel.org/lkml/[email protected]/

But doesn't this patch "sched/numa, mm: do not try to migrate memory to memoryless nodes" also fix the bug? Do we truly need both?

Yes. The oops is gone with "sched/numa, mm: do not try to migrate memory to memoryless nodes". However, as you know, wrongly manipulating array index is very dangerous - what hackers are looking for. Even with security in mind, both are necessary. Plus, no one gurantees the problematic code is gone through with a numa node that has no managed zones in the future.

Byungchul

author Reviewed-by Davidlohr Bueso Acked-by David Hildenbrand
#6 lore raw
In reply to Andrew Morton
Quoted 16 lines · Andrew Morton
Quoted 14 lines · Ying

kernel/sched/fair.c bool should_numa_migrate_memory(struct t · line 1841

Quoted 9 lines
kernel/sched/fair.c renamed from kernel/sched/fair.c~sched-numa-mm-do-not-try-to-migrate-memory-to-memoryless-nodes +6 -0
+	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
 	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.

Good catch!

IIUC, you will use patch as fix to the issue in

Quoted 7 lines
Quoted 3 lines

https://lore.kernel.org/lkml/[email protected]/

If so, we need the Fixes: tag to make it land in -stable properly.

Yes, this changelog is missing rather a lot of important information.

I pulled together the below, please check.

Plus, two guys gave the tags for the optimization patch as well:

Byungchul

Quoted 98 lines

From: Byungchul Park <[email protected]>
Subject: sched/numa, mm: do not try to migrate memory to memoryless nodes
Date: Mon, 19 Feb 2024 13:10:47 +0900

With numa balancing on, when a numa system is running where a numa node doesn't have its local memory so it has no managed zones, the following oops has been observed. It's because wakeup_kswapd() is called with a wrong zone index, -1. Fixed it by checking the index before calling wakeup_kswapd().

Quoted 36 lines
BUG: unable to handle page fault for address: 00000000000033f3
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 2 PID: 895 Comm: masim Not tainted 6.6.0-dirty #255
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
   rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
RIP: 0010:wakeup_kswapd (./linux/mm/vmscan.c:7812)
Code: (omitted)
RSP: 0000:ffffc90004257d58 EFLAGS: 00010286
RAX: ffffffffffffffff RBX: ffff88883fff0480 RCX: 0000000000000003
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88883fff0480
RBP: ffffffffffffffff R08: ff0003ffffffffff R09: ffffffffffffffff
R10: ffff888106c95540 R11: 0000000055555554 R12: 0000000000000003
R13: 0000000000000000 R14: 0000000000000000 R15: ffff88883fff0940
FS:  00007fc4b8124740(0000) GS:ffff888827c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000033f3 CR3: 000000026cc08004 CR4: 0000000000770ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
 <TASK>

? __die
? page_fault_oops
? __pte_offset_map_lock
? exc_page_fault
? asm_exc_page_fault
? wakeup_kswapd
migrate_misplaced_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault

Quoted 9 lines
RIP: 0033:0x55b897ba0808
Code: (omitted)
RSP: 002b:00007ffeefa821a0 EFLAGS: 00010287
RAX: 000055b89983acd0 RBX: 00007ffeefa823f8 RCX: 000055b89983acd0
RDX: 00007fc2f8122010 RSI: 0000000000020000 RDI: 000055b89983acd0
RBP: 00007ffeefa821a0 R08: 0000000000000037 R09: 0000000000000075
R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
R13: 00007ffeefa82410 R14: 000055b897ba5dd8 R15: 00007fc4b8340000
 </TASK>

Fix this by avoiding any attempt to migrate memory to memoryless nodes.

Fixes c574bbe917036 ("NUMA balancing: optimize page placement for memory tiering system")
Cc Benjamin Segall <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc Vincent Guittot <[email protected]>

---

 kernel/sched/fair.c |    6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/fair.c renamed from kernel/sched/fair.c~sched-numa-mm-do-not-try-to-migrate-memory-to-memoryless-nodes +6 -0
@@ -1831,6 +1831,12 @@ bool should_numa_migrate_memory(struct t
 	int last_cpupid, this_cpupid;
 
 	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
+	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.
 	 */

_

#7 lore raw
In reply to Andrew Morton
Quoted 16 lines · Andrew Morton
Quoted 14 lines · Ying

kernel/sched/fair.c bool should_numa_migrate_memory(struct t · line 1841

Quoted 9 lines
kernel/sched/fair.c renamed from kernel/sched/fair.c~sched-numa-mm-do-not-try-to-migrate-memory-to-memoryless-nodes +6 -0
+	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
 	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.

Good catch!

IIUC, you will use patch as fix to the issue in

Quoted 7 lines
Quoted 3 lines

https://lore.kernel.org/lkml/[email protected]/

If so, we need the Fixes: tag to make it land in -stable properly.

Yes, this changelog is missing rather a lot of important information.

I pulled together the below, please check.

To make it more clear, I need to explain it more. I posted the following two patches while resolving the oops issue. However, two are going on for different purposes.

1) https://lkml.kernel.org/r/[email protected]

I started this patch as the fix for the oops. However, I found the root cause comes from using -1 as an array index. So let the root cause fix go with another thread, 2). Nevertheless, 1) is still necessary as a *reasonable optimization* but not the real fix any more.

2) https://lkml.kernel.org/r/[email protected]

I found the root cause of the oops comes from using -1 as an array index. So moved all the oops message, Fixes: tag, and cc stable to here. Long story short, 2) is the *real fix* for the oops.

Byungchul

Quoted 98 lines

From: Byungchul Park <[email protected]>
Subject: sched/numa, mm: do not try to migrate memory to memoryless nodes
Date: Mon, 19 Feb 2024 13:10:47 +0900

With numa balancing on, when a numa system is running where a numa node doesn't have its local memory so it has no managed zones, the following oops has been observed. It's because wakeup_kswapd() is called with a wrong zone index, -1. Fixed it by checking the index before calling wakeup_kswapd().

Quoted 36 lines
BUG: unable to handle page fault for address: 00000000000033f3
#PF: supervisor read access in kernel mode
#PF: error_code(0x0000) - not-present page
PGD 0 P4D 0
Oops: 0000 [#1] PREEMPT SMP NOPTI
CPU: 2 PID: 895 Comm: masim Not tainted 6.6.0-dirty #255
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS
   rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
RIP: 0010:wakeup_kswapd (./linux/mm/vmscan.c:7812)
Code: (omitted)
RSP: 0000:ffffc90004257d58 EFLAGS: 00010286
RAX: ffffffffffffffff RBX: ffff88883fff0480 RCX: 0000000000000003
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff88883fff0480
RBP: ffffffffffffffff R08: ff0003ffffffffff R09: ffffffffffffffff
R10: ffff888106c95540 R11: 0000000055555554 R12: 0000000000000003
R13: 0000000000000000 R14: 0000000000000000 R15: ffff88883fff0940
FS:  00007fc4b8124740(0000) GS:ffff888827c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000033f3 CR3: 000000026cc08004 CR4: 0000000000770ee0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
 <TASK>

? __die
? page_fault_oops
? __pte_offset_map_lock
? exc_page_fault
? asm_exc_page_fault
? wakeup_kswapd
migrate_misplaced_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault

Quoted 9 lines
RIP: 0033:0x55b897ba0808
Code: (omitted)
RSP: 002b:00007ffeefa821a0 EFLAGS: 00010287
RAX: 000055b89983acd0 RBX: 00007ffeefa823f8 RCX: 000055b89983acd0
RDX: 00007fc2f8122010 RSI: 0000000000020000 RDI: 000055b89983acd0
RBP: 00007ffeefa821a0 R08: 0000000000000037 R09: 0000000000000075
R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000000
R13: 00007ffeefa82410 R14: 000055b897ba5dd8 R15: 00007fc4b8340000
 </TASK>

Fix this by avoiding any attempt to migrate memory to memoryless nodes.

Fixes c574bbe917036 ("NUMA balancing: optimize page placement for memory tiering system")
Cc Benjamin Segall <[email protected]>
Cc Daniel Bristot de Oliveira <[email protected]>
Cc Dietmar Eggemann <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Juri Lelli <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Steven Rostedt <[email protected]>
Cc Valentin Schneider <[email protected]>
Cc Vincent Guittot <[email protected]>

---

 kernel/sched/fair.c |    6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/fair.c renamed from kernel/sched/fair.c~sched-numa-mm-do-not-try-to-migrate-memory-to-memoryless-nodes +6 -0
@@ -1831,6 +1831,12 @@ bool should_numa_migrate_memory(struct t
 	int last_cpupid, this_cpupid;
 
 	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
+	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.
 	 */

_

maintainer
#8 lore raw
Quoted 8 lines · On Tue, 20 Feb 2024 11:33:04 +0900 Byungchul Park <[email protected]> wrote:
Quoted 3 lines

Yes, this changelog is missing rather a lot of important information.

I pulled together the below, please check.

To make it more clear, I need to explain it more. I posted the following two patches while resolving the oops issue. However, two are going on for different purposes.

Quoted 7 lines

1) https://lkml.kernel.org/r/[email protected]

I started this patch as the fix for the oops. However, I found the root cause comes from using -1 as an array index. So let the root cause fix go with another thread, 2). Nevertheless, 1) is still necessary as a *reasonable optimization* but not the real fix any more.

Well I altered this patch's changelog to tell readers that it is an optimization. But one does wonder why it isn't simply a bugfix. Attempting to migrate to a memoryless node is clearly as error. Presumably the called code handles it somehow, but in what fashion and at what cost?

Quoted 6 lines

2) https://lkml.kernel.org/r/[email protected]

I found the root cause of the oops comes from using -1 as an array index. So moved all the oops message, Fixes: tag, and cc stable to here. Long story short, 2) is the *real fix* for the oops.

#9 lore raw
Quoted 18 lines · Andrew Morton
Quoted 8 lines · On Tue, 20 Feb 2024 11:33:04 +0900 Byungchul Park <[email protected]> wrote:
Quoted 3 lines

Yes, this changelog is missing rather a lot of important information.

I pulled together the below, please check.

To make it more clear, I need to explain it more. I posted the following two patches while resolving the oops issue. However, two are going on for different purposes.

Quoted 7 lines

1) https://lkml.kernel.org/r/[email protected]

I started this patch as the fix for the oops. However, I found the root cause comes from using -1 as an array index. So let the root cause fix go with another thread, 2). Nevertheless, 1) is still necessary as a *reasonable optimization* but not the real fix any more.

Quoted 3 lines

Well I altered this patch's changelog to tell readers that it is an optimization. But one does wonder why it isn't simply a bugfix. Attempting to migrate to a memoryless node is clearly as error.

I agree with what Oscar Salvador said:

"As this is not a bug fix but an optimization, as we will fail anyways in migrate_misplaced_folio() when migrate_balanced_pgdat() notices that we do not have any memory on that node."

https://lore.kernel.org/lkml/[email protected]/

So assuming all the related code works correctly, the migration will safely fail even without this optimization patch.

Byungchul

Quoted 9 lines

Presumably the called code handles it somehow, but in what fashion and at what cost?

Quoted 6 lines

2) https://lkml.kernel.org/r/[email protected]

I found the root cause of the oops comes from using -1 as an array index. So moved all the oops message, Fixes: tag, and cc stable to here. Long story short, 2) is the *real fix* for the oops.

Reviewed-by on v4 1/1
#10 lore raw
In reply to Byungchul Park
Quoted 53 lines · On Mon, Feb 19, 2024 at 01:19:20PM +0900 Byungchul Park wrote:

Changes from v3:
1. Rewrite the comment in code and the commit message to make it more clear. (feedbacked by Oscar Salvador)
2. Add "Reviewed-by: Oscar Salvador <[email protected]>"

Changes from v2:
1. Rewrite the comment in code and the commit message becasue it turns out that this patch is not the real fix for the oops descriped. The real fix goes in another patch below:

https://lore.kernel.org/lkml/[email protected]/

Changes from v1:
1. Trim the verbose oops in the commit message. (feedbacked by Phil Auld)
2. Rewrite a comment in code. (feedbacked by Phil Auld)

--->8---
From 98f5d472c08e3ed5b9b6543290d392a2e50fcf3c Mon Sep 17 00:00:00 2001
From: Byungchul Park <[email protected]>
Date: Mon, 19 Feb 2024 13:10:47 +0900
Subject: [PATCH v4] sched/numa, mm: do not try to migrate memory to memoryless nodes

Memoryless nodes do not have any memory to migrate to, so stop trying it.

---
 kernel/sched/fair.c | 6 ++++++
 1 file changed, 6 insertions(+)
kernel/sched/fair.c +6 -0
index d7a3c63a2171..3e3b44ae72d1 100644
@@ -1828,6 +1828,12 @@ bool should_numa_migrate_memory(struct task_struct *p, struct folio *folio,
 	int dst_nid = cpu_to_node(dst_cpu);
 	int last_cpupid, this_cpupid;
 
+	/*
+	 * Cannot migrate to memoryless nodes.
+	 */
+	if (!node_state(dst_nid, N_MEMORY))
+		return false;
+
 	/*
 	 * The pages in slow memory node should be migrated according
 	 * to hot/cold instead of private/shared.
Signature · 1 line
2.17.1

Sorry, hadn't gotten far enough to see this version when I replied to v3...

Cheers,
Phil