sched/numa: skip VMA scanning on memory pinned
v2 cover sched/numa: skip VMA scanning on memory pinned
v1->v2:
1. add perf improvment numbers in commit log. Yet to find perf diff on will-it-scale, so not included here. Plan to run more workloads.
2. add tracepoint.
3. To peterz's comment, this will make it impossible to attract tasks to those memory just like other VMA skippings. This is the current implementation, I think we can improve that in the future, but at the moment it's probabaly better to keep it consistent.
Libo Chen (2):
sched/numa: skip VMA scanning on memory pinned to one NUMA node via
cpuset.mems
sched/numa: Add tracepoint that tracks the skipping of numa balancing
due to cpuset memory pinning
include/trace/events/sched.h | 31 +++++++++++++++++++++++++++++++ kernel/sched/fair.c | 9 +++++++++ 2 files changed, 40 insertions(+)
Signature · 1 line
2.43.5
PATCH v2 1/2 sched/numa: skip VMA scanning on memory pinned to one NUMA node via cpuset.mems
When the memory of the current task is pinned to one NUMA node by cgroup, there is no point in continuing the rest of VMA scanning and hinting page faults as they will just be overhead. With this change, there will be no more unnecessary PTE updates or page faults in this scenario.
We have seen up to a 6x improvement on a typical java workload running on VMs with memory and CPU pinned to one NUMA node via cpuset in a two-socket AARCH64 system. With the same pinning, on a 18-cores-per-socket Intel platform, we have seen 20% improvment in a microbench that creates a 30-vCPU selftest KVM guest with 4GB memory, where each vCPU reads 4KB pages in a fixed number of loops.
--- kernel/sched/fair.c | 7 +++++++ 1 file changed, 7 insertions(+)
kernel/sched/fair.c +7 -0
@@ -3329,6 +3329,13 @@ static void task_numa_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; +/* +* Memory is pinned to only one NUMA node via cpuset.mems, naturally +* no page can be migrated. +*/ +if (nodes_weight (cpuset_current_mems_allowed) ==1 ) +return ; +if (!mm->numa_next_scan) { mm->numa_next_scan = now +msecs_to_jiffies (sysctl_numa_balancing_scan_delay);
Signature · 1 line
2.43.5
Re: [PATCH v2 1/2] sched/numa: skip VMA scanning on memory pinned to one NUMA node via cpuset.mems
Quoted 22 lines · Libo Chen
When the memory of the current task is pinned to one NUMA node by cgroup, there is no point in continuing the rest of VMA scanning and hinting page faults as they will just be overhead. With this change, there will be no more unnecessary PTE updates or page faults in this scenario.
We have seen up to a 6x improvement on a typical java workload running on VMs with memory and CPU pinned to one NUMA node via cpuset in a two-socket AARCH64 system. With the same pinning, on a 18-cores-per-socket Intel platform, we have seen 20% improvment in a microbench that creates a 30-vCPU selftest KVM guest with 4GB memory, where each vCPU reads 4KB pages in a fixed number of loops.
Signed-off-by Libo Chen <[email protected]>--- kernel/sched/fair.c | 7 +++++++ 1 file changed, 7 insertions(+)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static void task_numa_work(struct callback_head *work) · line 3338
Quoted 10 lines
kernel/sched/fair.c +7
-0@@ -3329,6 +3329,13 @@ static void task_numa_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; +/* +* Memory is pinned to only one NUMA node via cpuset.mems, naturally +* no page can be migrated. +*/ +if (nodes_weight (cpuset_current_mems_allowed) ==1 ) +return ; +
Check cpusets_enabled() first?
Signature · 2 lines
Mel Gorman SUSE Labs
Re: [PATCH v2 1/2] sched/numa: skip VMA scanning on memory pinned to one NUMA node via cpuset.mems
Quoted 26 lines · Mel Gorman
Quoted 25 lines · Libo Chen
When the memory of the current task is pinned to one NUMA node by cgroup, there is no point in continuing the rest of VMA scanning and hinting page faults as they will just be overhead. With this change, there will be no more unnecessary PTE updates or page faults in this scenario.
We have seen up to a 6x improvement on a typical java workload running on VMs with memory and CPU pinned to one NUMA node via cpuset in a two-socket AARCH64 system. With the same pinning, on a 18-cores-per-socket Intel platform, we have seen 20% improvment in a microbench that creates a 30-vCPU selftest KVM guest with 4GB memory, where each vCPU reads 4KB pages in a fixed number of loops.
Signed-off-by Libo Chen <[email protected]>--- kernel/sched/fair.c | 7 +++++++ 1 file changed, 7 insertions(+)kernel/sched/fair.c +0
-0@@ -3329,6 +3329,13 @@ static void task_numa_work(struct callback_head *work)if (p->flags &PF_EXITING )return ;
Quoted 10 lines
kernel/sched/fair.c static void task_numa_work(struct callback_head *work) · line 3338
Quoted 7 lines
kernel/sched/fair.c +7
-0+/* +* Memory is pinned to only one NUMA node via cpuset.mems, naturally +* no page can be migrated. +*/ +if (nodes_weight (cpuset_current_mems_allowed) ==1 ) +return ; +Check
cpusets_enabled()first?
Hi Mel,
Yeah, can add that but isn't a bit redundant since nodes_weight(cpuset_current_mems_allowed) will just return #nodes which doesn't equal to 1 when !cpusets_enabled() and there are >= 2 numa nodes?
Libo
PATCH v2 2/2 sched/numa: Add tracepoint that tracks the skipping of numa balancing due to cpuset memory pinning
In reply to Libo ChenUnlike sched_skip_vma_numa tracepoint which tracks skipped VMAs, this tracks the task subjected to cpuset.mems pinning and prints out its allowed memory node mask.
--- include/trace/events/sched.h | 31 +++++++++++++++++++++++++++++++ kernel/sched/fair.c | 4 +++- 2 files changed, 34 insertions(+), 1 deletion(-)
include/trace/events/sched.h +31 -0
@@ -745,6 +745,37 @@ TRACE_EVENT(sched_skip_vma_numa, __entry->vm_end, __print_symbolic(__entry->reason, NUMAB_SKIP_REASON)) ); + +TRACE_EVENT (sched_skip_cpuset_numa, + + TP_PROTO(struct task_struct *tsk, nodemask_t *mem_allowed_ptr), + +TP_ARGS (tsk, mem_allowed_ptr), + +TP_STRUCT__entry ( + __array( char, comm, TASK_COMM_LEN ) + __field( pid_t, pid ) + __field( pid_t, tgid ) + __field( pid_t, ngid ) + __array( unsigned long, mem_allowed, BITS_TO_LONGS(MAX_NUMNODES)) + ), + +TP_fast_assign ( + memcpy(__entry->comm, tsk->comm, TASK_COMM_LEN); + __entry->pid = task_pid_nr(tsk); + __entry->tgid = task_tgid_nr(tsk); + __entry->ngid = task_numa_group_id(tsk); + memcpy(__entry->mem_allowed, mem_allowed_ptr->bits, + sizeof(__entry->mem_allowed)); + ), + +TP_printk ("comm=%s pid=%d tgid=%d ngid=%d mem_node_allowed_mask=%lx" , + __entry->comm, + __entry->pid, + __entry->tgid, + __entry->ngid, + __entry->mem_allowed[0 ]) +);#endif /* CONFIG_NUMA_BALANCING */ /*
kernel/sched/fair.c +3 -1
@@ -3333,8 +3333,10 @@ static void task_numa_work(struct callback_head *work)* Memory is pinned to only one NUMA node via cpuset.mems, naturally * no page can be migrated. */ -if (nodes_weight (cpuset_current_mems_allowed) ==1 ) +if (nodes_weight (cpuset_current_mems_allowed) ==1 ) { +trace_sched_skip_cpuset_numa (current, &cpuset_current_mems_allowed);return ; + }if (!mm->numa_next_scan) { mm->numa_next_scan = now +
Signature · 1 line
2.43.5
Re: [PATCH v2 2/2] sched/numa: Add tracepoint that tracks the skipping of numa balancing due to cpuset memory pinning
forgot to add Steven Rostedt.
Quoted 31 lines · Libo Chen
Unlike
sched_skip_vma_numatracepoint which tracks skipped VMAs, this tracks the task subjected to cpuset.mems pinning and prints out its allowed memory node mask.--- include/trace/events/sched.h | 31 +++++++++++++++++++++++++++++++ kernel/sched/fair.c | 4 +++- 2 files changed, 34 insertions(+), 1 deletion(-)include/trace/events/sched.h +15
-0@@ -745,6 +745,37 @@ TRACE_EVENT(sched_skip_vma_numa, __entry->vm_end, __print_symbolic(__entry->reason, NUMAB_SKIP_REASON)) ); + +TRACE_EVENT(sched_skip_cpuset_numa, + + TP_PROTO(struct task_struct *tsk, nodemask_t *mem_allowed_ptr), + + TP_ARGS(tsk, mem_allowed_ptr), + + TP_STRUCT__entry( + __array( char, comm, TASK_COMM_LEN ) + __field( pid_t, pid ) + __field( pid_t, tgid ) + __field( pid_t, ngid ) + __array( unsigned long, mem_allowed, BITS_TO_LONGS(MAX_NUMNODES)) + ), +
include/trace/events/sched.h TRACE_EVENT(sched_skip_vma_numa, · line 772
Quoted 10 lines
include/trace/events/sched.h +10
-0+TP_fast_assign ( + memcpy(__entry->comm, tsk->comm, TASK_COMM_LEN); + __entry->pid =task_pid_nr (tsk); + __entry->tgid =task_tgid_nr (tsk); + __entry->ngid =task_numa_group_id (tsk); +memcpy (__entry->mem_allowed, mem_allowed_ptr->bits, +sizeof (__entry->mem_allowed)); + ), + + TP_printk("comm=%s pid=%d tgid=%d ngid=%d mem_node_allowed_mask=%lx",
I cannot find a way to print out nodemask_t nicely here with %*pbl. So I fall back to just raw hex value. Will be grateful if someone knows a better way to print nodemask nicely in a tracepoint
Quoted 25 lines
include/trace/events/sched.h +6
-0+ __entry->comm, + __entry->pid, + __entry->tgid, + __entry->ngid, + __entry->mem_allowed[0]) +);#endif /* CONFIG_NUMA_BALANCING */ /* kernel/sched/fair.c +3
-1@@ -3333,8 +3333,10 @@ static void task_numa_work(struct callback_head *work)* Memory is pinned to only one NUMA node via cpuset.mems, naturally * no page can be migrated. */ -if (nodes_weight (cpuset_current_mems_allowed) ==1 ) +if (nodes_weight (cpuset_current_mems_allowed) ==1 ) { +trace_sched_skip_cpuset_numa (current, &cpuset_current_mems_allowed);return ; + }if (!mm->numa_next_scan) { mm->numa_next_scan = now +
Steven Rostedt
maintainerRe: [PATCH v2 2/2] sched/numa: Add tracepoint that tracks the skipping of numa balancing due to cpuset memory pinning
Quoted 6 lines · On Wed, 26 Mar 2025 17:40:02 -0700 Libo Chen <[email protected]> wrote:
include/trace/events/sched.h TRACE_EVENT(sched_skip_vma_numa, · line 772
Quoted 1 line
include/trace/events/sched.h +1
-0+ TP_printk("comm=%s pid=%d tgid=%d ngid=%d mem_node_allowed_mask=%lx",I cannot find a way to print out
nodemask_tnicely here with %*pbl. So I fall back to just raw hex value. Will be grateful if someone knows a better way to print nodemask nicely in a tracepoint
Ug.
I guess I need to add support for that. Both in the boot verifier as well as in libtraceevent.
-- Steve
Re: [PATCH v2 2/2] sched/numa: Add tracepoint that tracks the skipping of numa balancing due to cpuset memory pinning
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 26 Mar 2025 17:40:02 -0700 Libo Chen <[email protected]> wrote:
include/trace/events/sched.h TRACE_EVENT(sched_skip_vma_numa, · line 772
Quoted 1 line
include/trace/events/sched.h +1
-0+ TP_printk("comm=%s pid=%d tgid=%d ngid=%d mem_node_allowed_mask=%lx",I cannot find a way to print out
nodemask_tnicely here with %*pbl. So I fall back to just raw hex value. Will be grateful if someone knows a better way to print nodemask nicely in a tracepoint
Quoted 5 lines
Ug.
I guess I need to add support for that. Both in the boot verifier as well as in libtraceevent.
That will be fantastic, I think this is quite useful.
Thanks,
Libo
Quoted 1 line
-- Steve
Steven Rostedt
maintainerRe: [PATCH v2 2/2] sched/numa: Add tracepoint that tracks the skipping of numa balancing due to cpuset memory pinning
Quoted 11 lines · On Wed, 26 Mar 2025 21:32:32 -0700 Libo Chen <[email protected]> wrote:
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 26 Mar 2025 17:40:02 -0700 Libo Chen <[email protected]> wrote:
include/trace/events/sched.h TRACE_EVENT(sched_skip_vma_numa, · line 772
Quoted 1 line
include/trace/events/sched.h +1
-0+ TP_printk("comm=%s pid=%d tgid=%d ngid=%d mem_node_allowed_mask=%lx",I cannot find a way to print out
nodemask_tnicely here with %*pbl. So I fall back to just raw hex value. Will be grateful if someone knows a better way to print nodemask nicely in a tracepoint
Quoted 7 lines
Quoted 5 lines
Ug.
I guess I need to add support for that. Both in the boot verifier as well as in libtraceevent.
That will be fantastic, I think this is quite useful.
I'm curious to what was broken. Just the user space side? Or was the kernel side broken too?
I notice that i2c_write has "[%*phD]" which means it worked, and also skipped out on the verification tests. This should be fixed.
But trace-cmd and perf will not be able to print that out either.
Now %*pbl works in the kernel (not checked), but probably doesn't work for trace-cmd.
-- Steve
Re: [PATCH v2 2/2] sched/numa: Add tracepoint that tracks the skipping of numa balancing due to cpuset memory pinning
Quoted 14 lines · Steven Rostedt
Quoted 11 lines · On Wed, 26 Mar 2025 21:32:32 -0700 Libo Chen <[email protected]> wrote:
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 26 Mar 2025 17:40:02 -0700 Libo Chen <[email protected]> wrote:
include/trace/events/sched.h TRACE_EVENT(sched_skip_vma_numa, · line 772
Quoted 1 line
include/trace/events/sched.h +1
-0+ TP_printk("comm=%s pid=%d tgid=%d ngid=%d mem_node_allowed_mask=%lx",I cannot find a way to print out
nodemask_tnicely here with %*pbl. So I fall back to just raw hex value. Will be grateful if someone knows a better way to print nodemask nicely in a tracepoint
Quoted 8 lines
Quoted 7 lines
Quoted 5 lines
Ug.
I guess I need to add support for that. Both in the boot verifier as well as in libtraceevent.
That will be fantastic, I think this is quite useful.
Oops just saw this right now, not sure how I missed it.
Quoted 3 lines
I'm curious to what was broken. Just the user space side? Or was the kernel side broken too?
That's a good point, I originally only tried %*pbl with trace-cmd, but I tested your patch without trace-cmd. So yeah %*pbl may already be working in the kernel, I will report back later today.
-- Libo
Quoted 9 lines
I notice that
i2c_writehas "[%*phD]" which means it worked, and also skipped out on the verification tests. This should be fixed.But trace-cmd and perf will not be able to print that out either.
Now %*pbl works in the kernel (not checked), but probably doesn't work for trace-cmd.
-- Steve
Re: [PATCH v2 2/2] sched/numa: Add tracepoint that tracks the skipping of numa balancing due to cpuset memory pinning
Quoted 32 lines · Libo Chen
Quoted 14 lines · Steven Rostedt
Quoted 11 lines · On Wed, 26 Mar 2025 21:32:32 -0700 Libo Chen <[email protected]> wrote:
Quoted 10 lines · Steven Rostedt
Quoted 6 lines · On Wed, 26 Mar 2025 17:40:02 -0700 Libo Chen <[email protected]> wrote:
include/trace/events/sched.h TRACE_EVENT(sched_skip_vma_numa, · line 772
Quoted 1 line
include/trace/events/sched.h +1
-0+ TP_printk("comm=%s pid=%d tgid=%d ngid=%d mem_node_allowed_mask=%lx",I cannot find a way to print out
nodemask_tnicely here with %*pbl. So I fall back to just raw hex value. Will be grateful if someone knows a better way to print nodemask nicely in a tracepointQuoted 8 lines
Quoted 7 lines
Quoted 5 lines
Ug.
I guess I need to add support for that. Both in the boot verifier as well as in libtraceevent.
That will be fantastic, I think this is quite useful.
Oops just saw this right now, not sure how I missed it.
Quoted 3 lines
I'm curious to what was broken. Just the user space side? Or was the kernel side broken too?
Quoted 6 lines
That's a good point, I originally only tried %*pbl with trace-cmd, but I tested your patch without trace-cmd. So yeah %*pbl may already be working in the kernel, I will report back later today.
-- Libo
Okay yes, it does prints out to trace log. Not seeing the output is due to the userspace tool not the lack of check.
-- Libo
Quoted 10 lines
Quoted 9 lines
I notice that
i2c_writehas "[%*phD]" which means it worked, and also skipped out on the verification tests. This should be fixed.But trace-cmd and perf will not be able to print that out either.
Now %*pbl works in the kernel (not checked), but probably doesn't work for trace-cmd.
-- Steve