numa/core updates
Subsystems (3)
cover numa/core updates
I've been testing wider workloads and here's two more small and obvious patches rounding up numa/core behavior around the edges.
The NUMA code should now be pretty unintrusive to all but the long-running, memory-intense workloads where it's expected to make a (positive) difference.
Short-run workloads like kbuild or hackbench don't trigger the NUMA code now. The limits can be reconsidered later on, iteratively - the goal now is to not regress.
Thanks,
Ingo
-------------->
Ingo Molnar (2):
sched: Exclude pinned tasks from the NUMA-balancing logic
sched: Add RSS filter to NUMA-balancing
include/linux/sched.h | 1 + kernel/sched/core.c | 6 ++++++ kernel/sched/debug.c | 1 + kernel/sched/fair.c | 53 +++++++++++++++++++++++++++++++++++++++++++++---- kernel/sched/features.h | 1 + kernel/sysctl.c | 7 +++++++ 6 files changed, 65 insertions(+), 4 deletions(-)
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/2 sched: Exclude pinned tasks from the NUMA-balancing logic
Don't try to NUMA-balance hard-bound tasks in vein. This also makes it easier to compare hard-bound workloads against NUMA-balanced workloads, because the NUMA code will be completely inactive for those hard-bound tasks.
( Keep a debugging feature flag around: for development it
makes sense to observe what NUMA balancing tries to do with hard-affine tasks. )
[ Note: the duplicated test condition will be consolidated
in the next patch. ]
--- kernel/sched/core.c | 6 ++++++ kernel/sched/debug.c | 1 + kernel/sched/fair.c | 7 +++++++ kernel/sched/features.h | 1 + 4 files changed, 15 insertions(+)
kernel/sched/core.c +6 -0
@@ -4664,6 +4664,12 @@ void do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask)cpumask_copy (&p->cpus_allowed, new_mask); p->nr_cpus_allowed =cpumask_weight (new_mask); + +#ifdef CONFIG_NUMA_BALANCING +/* Don't disturb hard-bound tasks: */ +if (sched_feat (NUMA_EXCLUDE_AFFINE ) && (p->nr_cpus_allowed !=num_online_cpus ())) + p->numa_shared =-1 ; +#endif }/*
kernel/sched/debug.c +1 -0
@@ -448,6 +448,7 @@ void proc_sched_show_task(struct task_struct *p, struct seq_file *m) nr_switches = p->nvcsw + p->nivcsw; +P (nr_cpus_allowed);#ifdef CONFIG_SCHEDSTATS PN (se.statistics.wait_start);PN (se.statistics.sleep_start);
kernel/sched/fair.c +7 -0
@@ -2495,6 +2495,13 @@ static void task_tick_numa(struct rq *rq, struct task_struct *curr)if (!curr->mm || (curr->flags &PF_EXITING ) || !curr->numa_faults)return ; +/* Don't disturb hard-bound tasks: */ +if (sched_feat (NUMA_EXCLUDE_AFFINE ) && (curr->nr_cpus_allowed !=num_online_cpus ())) { +if (curr->numa_shared >=0 ) + curr->numa_shared =-1 ; +return ; + } +task_tick_numa_scan (rq, curr);task_tick_numa_placement (rq, curr); }
kernel/sched/features.h +1 -0
@@ -77,6 +77,7 @@ SCHED_FEAT(WAKE_ON_IDEAL_CPU, false) SCHED_FEAT(NUMA, true) SCHED_FEAT(NUMA_BALANCE_ALL, false) SCHED_FEAT(NUMA_BALANCE_INTERNODE, false) +SCHED_FEAT(NUMA_EXCLUDE_AFFINE, true) SCHED_FEAT(NUMA_LB, false) SCHED_FEAT(NUMA_GROUP_LB_COMPRESS, true)SCHED_FEAT (NUMA_GROUP_LB_SPREAD , true)
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 2/2 sched: Add RSS filter to NUMA-balancing
In reply to Ingo MolnarNUMA-balancing, combined with NUMA-affine memory migration, is a relatively long-term process (compared to the typical time scale of scheduling) that takes time to establish and converge - on the time scale of of several seconds or more.
Small, short-lived and don't have much of a NUMA placement cost to begin with, so don't NUMA-balance them. A task needs to execute long enough and needs to establish a large enough user-space memory image to benefit from more intelligent NUMA balancing.
We already have a CPU time limit before tasks are affected by NUMA balancing - this change adds the memory equivalent:
by introducing an RSS limit of 128 MBs.
In practice this excludes most short-lived tasks - the limit is in fact probably a bit on the conservative side - but with intrusive kernel features conservative is good.
The /proc/sys/kernel/sched_numa_rss_threshold_mb value can be tuned runtime - setting it to 0 turns off this filter.
To implement the RSS filter first factor out a clean task_numa_candidate() function and comment on the various reasons of why we wouldn't want to begin to NUMA-balance a particular task (yet). Then add the RSS check.
Note, we are using the p->hiwater_rss value instead of the current RSS size. We do this to avoid tasks flipping in and out of the limit, if their RSS fluctuates around the limit. The RSS high-water value increases monotonically in the life-time of a task, so there's a single, precise transition to NUMA-balancing as the limit is crossed.
--- include/linux/sched.h | 1 + kernel/sched/fair.c | 50 ++++++++++++++++++++++++++++++++++++++++++++------ kernel/sysctl.c | 7 +++++++ 3 files changed, 52 insertions(+), 6 deletions(-)
include/linux/sched.h +1 -0
@@ -2059,6 +2059,7 @@ extern unsigned int sysctl_sched_numa_scan_period_min;extern unsigned int sysctl_sched_numa_scan_period_max;extern unsigned int sysctl_sched_numa_scan_size_min;extern unsigned int sysctl_sched_numa_scan_size_max; +extern unsigned int sysctl_sched_numa_rss_threshold;extern unsigned int sysctl_sched_numa_settle_count;#ifdef CONFIG_SCHED_DEBUG
kernel/sched/fair.c +44 -6
67 lines not on this page · raw message
kernel/sysctl.c +7 -0
@@ -388,6 +388,13 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec, }, { + .procname = "sched_numa_rss_threshold_mb", + .data = &sysctl_sched_numa_rss_threshold, + .maxlen = sizeof(unsigned int), + .mode = 0644, + .proc_handler = proc_dointvec, + }, + { .procname = "sched_numa_settle_count", .data = &sysctl_sched_numa_settle_count, .maxlen = sizeof(unsigned int),
Signature · 7 lines
1.7.11.7 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH v2 2/2 sched: Add RSS filter to NUMA-balancing
Updated -v2 patch: RSS high-water calculation has a performance trick, so mm->hiwater_rss must be used together with get_mm_rss.
Thanks,
Ingo
--------------------------->