Scan for an idle sibling in a single pass
v2 cover Scan for an idle sibling in a single pass
Changlog since v1
o Move extern declaration to header for coding style
o Remove unnecessary parameter from __select_idle_cpu
This series of 5 patches reposts three patches from Peter entitled "select_idle_sibling() wreckage". It only scans the runqueues in a single pass when searching for an idle sibling.
Two patches from Peter were dropped. The first patch altered how scan depth was calculated. Scan depth deletion is a random number generator with two major limitations. The avg_idle time is based on the time between a CPU going idle and being woken up clamped approximately by 2*sysctl_sched_migration_cost. This is difficult to compare in a sensible fashion to avg_scan_cost. The second issue is that only the avg_scan_cost of scan failures is recorded and it does not decay. This requires deeper surgery that would justify a patch on its own although Peter notes that
https://lkml.kernel.org/r/[email protected] is potentially useful for an alternative avg_idle metric.
The second patch dropped converted the idle core scan throttling mechanism to SIS_PROP. While this would unify the throttling of core and CPU scanning, it was not free of regressions and has_idle_cores is a fairly effective throttling mechanism with the caveat that it can have a lot of false positives for workloads like hackbench.
Peter's series tried to solve three problems at once, this subset addresses one problem. As with anything select_idle_sibling, it's a mix of wins and losses but won more than it lost across a range of workloads and machines.
kernel/sched/core.c | 18 +++-- kernel/sched/fair.c | 157 ++++++++++++++++++++-------------------- kernel/sched/features.h | 1 - kernel/sched/sched.h | 2 + 4 files changed, 93 insertions(+), 85 deletions(-)
Signature · 1 line
2.26.2
PATCH v2 1/5 sched/fair: Remove SIS_AVG_CPU
SIS_AVG_CPU was introduced as a means of avoiding a search when the average search cost indicated that the search would likely fail. It was a blunt instrument and disabled by commit 4c77b18cf8b7 ("sched/fair: Make select_idle_cpu() more aggressive") and later replaced with a proportional search depth by commit 1ad3aaf3fcd2 ("sched/core: Implement new approach to scale select_idle_cpu()").
While there are corner cases where SIS_AVG_CPU is better, it has now been disabled for almost three years. As the intent of SIS_PROP is to reduce the time complexity of select_idle_cpu(), lets drop SIS_AVG_CPU and focus on SIS_PROP as a throttling mechanism.
--- kernel/sched/fair.c | 20 +++++++++----------- kernel/sched/features.h | 1 - 2 files changed, 9 insertions(+), 12 deletions(-)
kernel/sched/fair.c +9 -11
@@ -6145,7 +6145,6 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t {struct cpumask *cpus =this_cpu_cpumask_var_ptr (select_idle_mask);struct sched_domain *this_sd; -u64 avg_cost, avg_idle;u64 time;int this =smp_processor_id ();int cpu, nr =INT_MAX ;
@@ -6154,18 +6153,17 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tif (!this_sd)return -1 ; -/* -* Due to large variance we need a large fuzz factor; hackbench in -* particularly is sensitive here. -*/ - avg_idle =this_rq ()->avg_idle /512 ; - avg_cost = this_sd->avg_scan_cost +1 ; +if (sched_feat (SIS_PROP )) { +u64 avg_cost, avg_idle, span_avg; -if (sched_feat (SIS_AVG_CPU ) && avg_idle < avg_cost) -return -1 ; +/* +* Due to large variance we need a large fuzz factor; +* hackbench in particularly is sensitive here. +*/ + avg_idle =this_rq ()->avg_idle /512 ; + avg_cost = this_sd->avg_scan_cost +1 ; -if (sched_feat (SIS_PROP )) { -u64 span_avg = sd->span_weight * avg_idle; + span_avg = sd->span_weight * avg_idle;if (span_avg >4 *avg_cost) nr =div_u64 (span_avg, avg_cost);else
kernel/sched/features.h +0 -1
@@ -54,7 +54,6 @@ SCHED_FEAT(TTWU_QUEUE, true)/* * When doing wakeups, attempt to limit superfluous scans of the LLC domain. */ -SCHED_FEAT(SIS_AVG_CPU, false)SCHED_FEAT (SIS_PROP , true)/*
Signature · 1 line
2.26.2
PATCH v2 2/5 sched/fair: Move avg_scan_cost calculations under SIS_PROP
In reply to Mel GormanAs noted by Vincent Guittot, avg_scan_costs are calculated for SIS_PROP even if SIS_PROP is disabled. Move the time calculations under a SIS_PROP check and while we are at it, exclude the cost of initialising the CPU mask from the average scan cost.
--- kernel/sched/fair.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-)
kernel/sched/fair.c +8 -6
@@ -6153,6 +6153,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tif (!this_sd)return -1 ; +cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); +if (sched_feat (SIS_PROP )) {u64 avg_cost, avg_idle, span_avg;
@@ -6168,11 +6170,9 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t nr =div_u64 (span_avg, avg_cost); else nr =4 ; - } - - time =cpu_clock (this); -cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); + time =cpu_clock (this); + }for_each_cpu_wrap (cpu, cpus, target) {if (!--nr)
@@ -6181,8 +6181,10 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tbreak ; } - time =cpu_clock (this) - time; -update_avg (&this_sd->avg_scan_cost, time); +if (sched_feat (SIS_PROP )) { + time =cpu_clock (this) - time; +update_avg (&this_sd->avg_scan_cost, time); + }return cpu; }
Signature · 1 line
2.26.2
PATCH v2 3/5 sched/fair: Make select_idle_cpu() proportional to cores
In reply to Mel GormanFrom: Peter Zijlstra (Intel) <[email protected]>
Instead of calculating how many (logical) CPUs to scan, compute how many cores to scan.
This changes behaviour for anything !SMT2.
--- kernel/sched/core.c | 18 +++++++++++++----- kernel/sched/fair.c | 12 ++++++++++-- kernel/sched/sched.h | 2 ++ 3 files changed, 25 insertions(+), 7 deletions(-)
kernel/sched/core.c +13 -5
@@ -7444,11 +7444,19 @@ int sched_cpu_activate(unsigned int cpu)balance_push_set (cpu, false);#ifdef CONFIG_SCHED_SMT -/* -* When going up, increment the number of cores with SMT present. -*/ -if (cpumask_weight (cpu_smt_mask (cpu)) ==2 ) -static_branch_inc_cpuslocked (&sched_smt_present); +do { +int weight =cpumask_weight (cpu_smt_mask (cpu)); + +if (weight > sched_smt_weight) + sched_smt_weight = weight; + +/* +* When going up, increment the number of cores with SMT present. +*/ +if (weight ==2 ) +static_branch_inc_cpuslocked (&sched_smt_present); + + }while (0 );#endif set_cpu_active (cpu, true);
kernel/sched/fair.c +10 -2
@@ -6010,6 +6010,8 @@ static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *pDEFINE_STATIC_KEY_FALSE (sched_smt_present);EXPORT_SYMBOL_GPL (sched_smt_present); +int sched_smt_weight__read_mostly =1 ; +static inline void set_idle_cores (int cpu,int val) {struct sched_domain_shared *sds;
@@ -6124,6 +6126,8 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t#else /* CONFIG_SCHED_SMT */ +#define sched_smt_weight 1 +static inline int select_idle_core (struct task_struct *p,struct sched_domain *sd,int target) {return -1 ;
@@ -6136,6 +6140,8 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd#endif /* CONFIG_SCHED_SMT */ +#define sis_min_cores 2 +/* * Scan the LLC domain for idle CPUs; this is dynamically regulated by * comparing the average scan cost (tracked in sd->avg_scan_cost) against the
@@ -6166,10 +6172,12 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t avg_cost = this_sd->avg_scan_cost +1 ; span_avg = sd->span_weight * avg_idle; -if (span_avg >4 *avg_cost) +if (span_avg > sis_min_cores*avg_cost) nr =div_u64 (span_avg, avg_cost);else - nr =4 ; + nr = sis_min_cores; + + nr *= sched_smt_weight; time =cpu_clock (this); }
kernel/sched/sched.h +2 -0
@@ -1107,6 +1107,8 @@ static inline void update_idle_core(struct rq *rq) __update_idle_core(rq); } +extern int sched_smt_weight; +#else static inline void update_idle_core (struct rq *rq) { }#endif
Signature · 1 line
2.26.2
Re: [PATCH 3/5] sched/fair: Make select_idle_cpu() proportional to cores
Quoted 77 lines · Mel Gorman
From: Peter Zijlstra (Intel) <[email protected]>
Instead of calculating how many (logical) CPUs to scan, compute how many cores to scan.
This changes behaviour for anything !SMT2.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Mel Gorman <[email protected]>--- kernel/sched/core.c | 18 +++++++++++++----- kernel/sched/fair.c | 12 ++++++++++-- kernel/sched/sched.h | 2 ++ 3 files changed, 25 insertions(+), 7 deletions(-)kernel/sched/core.c +13
-5@@ -7444,11 +7444,19 @@ int sched_cpu_activate(unsigned int cpu)balance_push_set (cpu, false);#ifdef CONFIG_SCHED_SMT -/* -* When going up, increment the number of cores with SMT present. -*/ -if (cpumask_weight (cpu_smt_mask (cpu)) ==2 ) -static_branch_inc_cpuslocked (&sched_smt_present); +do { +int weight =cpumask_weight (cpu_smt_mask (cpu)); + +if (weight > sched_smt_weight) + sched_smt_weight = weight; + +/* +* When going up, increment the number of cores with SMT present. +*/ +if (weight ==2 ) +static_branch_inc_cpuslocked (&sched_smt_present); + + }while (0 );#endif set_cpu_active (cpu, true);kernel/sched/fair.c +6
-0@@ -6010,6 +6010,8 @@ static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *pDEFINE_STATIC_KEY_FALSE (sched_smt_present);EXPORT_SYMBOL_GPL (sched_smt_present); +int sched_smt_weight__read_mostly =1 ; +static inline void set_idle_cores (int cpu,int val) {struct sched_domain_shared *sds;@@ -6124,6 +6126,8 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t#else /* CONFIG_SCHED_SMT */ +#define sched_smt_weight 1 +static inline int select_idle_core (struct task_struct *p,struct sched_domain *sd,int target) {return -1 ;@@ -6136,6 +6140,8 @@ static inline int select_idle_smt(struct task_struct *p, struct sched_domain *sd#endif /* CONFIG_SCHED_SMT */ +#define sis_min_cores 2 +/* * Scan the LLC domain for idle CPUs; this is dynamically regulated by * comparing the average scan cost (tracked in sd->avg_scan_cost) against the @@ -6166,10 +6172,12 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t avg_cost = this_sd->avg_scan_cost +1 ;
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6180
Quoted 10 lines
kernel/sched/fair.c +4
-2span_avg = sd->span_weight * avg_idle; -if (span_avg >4 *avg_cost) +if (span_avg > sis_min_cores*avg_cost) nr =div_u64 (span_avg, avg_cost);else - nr =4 ; + nr = sis_min_cores; + + nr *= sched_smt_weight;
Is it better to put this into an inline wrapper to hide sched_smt_weight if !CONFIG_SCHED_SMT?
Thanks,
-Aubrey
Re: [PATCH 3/5] sched/fair: Make select_idle_cpu() proportional to cores
Quoted 16 lines · Aubrey
Quoted 16 lines
<SNIP>
kernel/sched/fair.c +2
-0@@ -6124,6 +6126,8 @@ static int select_idle_smt(struct task_struct *p, struct sched_domain *sd, int t#else /* CONFIG_SCHED_SMT */ +#define sched_smt_weight 1 +static inline int select_idle_core (struct task_struct *p,struct sched_domain *sd,int target) {return -1 ;@@ -6166,10 +6172,12 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t avg_cost = this_sd->avg_scan_cost +1 ;
Quoted 12 lines
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6180
Quoted 9 lines
kernel/sched/fair.c +4
-2span_avg = sd->span_weight * avg_idle; -if (span_avg >4 *avg_cost) +if (span_avg > sis_min_cores*avg_cost) nr =div_u64 (span_avg, avg_cost);else - nr =4 ; + nr = sis_min_cores; + + nr *= sched_smt_weight;Is it better to put this into an inline wrapper to hide
sched_smt_weightif !CONFIG_SCHED_SMT?
There already is a #define sched_smt_weight for !CONFIG_SCHED_SMT and I do not think an inline wrapper would make it more readable or maintainable.
Signature · 2 lines
Mel Gorman SUSE Labs
PATCH v2 4/5 sched/fair: Remove select_idle_smt()
In reply to Mel GormanFrom: Peter Zijlstra (Intel) <[email protected]>
In order to make the next patch more readable, and to quantify the actual effectiveness of this pass, start by removing it.
--- kernel/sched/fair.c | 30 ------------------------------ 1 file changed, 30 deletions(-)
kernel/sched/fair.c +0 -30
@@ -6103,27 +6103,6 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int return -1; } -/* -* Scan the local SMT mask for idle CPUs. -*/ -static int select_idle_smt (struct task_struct *p,struct sched_domain *sd,int target) -{ -int cpu; - -if (!static_branch_likely (&sched_smt_present)) -return -1 ; - -for_each_cpu (cpu,cpu_smt_mask (target)) { -if (!cpumask_test_cpu (cpu, p->cpus_ptr) || - !cpumask_test_cpu (cpu,sched_domain_span (sd))) -continue ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -return cpu; - } - -return -1 ; -} -#else /* CONFIG_SCHED_SMT */ #define sched_smt_weight 1
@@ -6133,11 +6112,6 @@ static inline int select_idle_core(struct task_struct *p, struct sched_domain *s return -1; } -static inline int select_idle_smt (struct task_struct *p,struct sched_domain *sd,int target) -{ -return -1 ; -} -#endif /* CONFIG_SCHED_SMT */ #define sis_min_cores 2
@@ -6331,10 +6305,6 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)if ((unsigned )i < nr_cpumask_bits)return i; - i =select_idle_smt (p, sd, target); -if ((unsigned )i < nr_cpumask_bits) -return i; -return target; }
Signature · 1 line
2.26.2
PATCH v2 5/5 sched/fair: Merge select_idle_core/cpu()
In reply to Mel GormanFrom: Peter Zijlstra (Intel) <[email protected]>
Both select_idle_core() and select_idle_cpu() do a loop over the same cpumask. Observe that by clearing the already visited CPUs, we can fold the iteration and iterate a core at a time.
All we need to do is remember any non-idle CPU we encountered while scanning for an idle core. This way we'll only iterate every CPU once.
--- kernel/sched/fair.c | 97 +++++++++++++++++++++++++++------------------ 1 file changed, 59 insertions(+), 38 deletions(-)
kernel/sched/fair.c +59 -38
@@ -6006,6 +6006,14 @@ static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *preturn new_cpu; } +static inline int __select_idle_cpu (struct task_struct *p,int core,struct cpumask *cpus) +{ +if (available_idle_cpu (core) ||sched_idle_cpu (core)) +return core; + +return -1 ; +} +#ifdef CONFIG_SCHED_SMT DEFINE_STATIC_KEY_FALSE (sched_smt_present);EXPORT_SYMBOL_GPL (sched_smt_present);
@@ -6066,40 +6074,34 @@ void __update_idle_core(struct rq *rq)* there are no idle cores left in the system; tracked through * sd_llc->shared->has_idle_cores and enabled through update_idle_core() above. */ -static int select_idle_core (struct task_struct *p,struct sched_domain *sd,int target) +static int select_idle_core (struct task_struct *p,int core,struct cpumask *cpus,int *idle_cpu) { -struct cpumask *cpus =this_cpu_cpumask_var_ptr (select_idle_mask); -int core, cpu; +bool idle = true; +int cpu;if (!static_branch_likely (&sched_smt_present)) -return -1 ; - -if (!test_idle_cores (target, false)) -return -1 ; - -cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); +return __select_idle_cpu (p, core, cpus); -for_each_cpu_wrap (core, cpus, target) { -bool idle = true; - -for_each_cpu (cpu,cpu_smt_mask (core)) { -if (!available_idle_cpu (cpu)) { - idle = false; -break ; +for_each_cpu (cpu,cpu_smt_mask (core)) { +if (!available_idle_cpu (cpu)) { + idle = false; +if (*idle_cpu ==-1 ) { +if (sched_idle_cpu (cpu) &&cpumask_test_cpu (cpu, p->cpus_ptr)) { + *idle_cpu = cpu; +break ; + } +continue ; } +break ; } - -if (idle) -return core; - -cpumask_andnot (cpus, cpus,cpu_smt_mask (core)); +if (*idle_cpu ==-1 &&cpumask_test_cpu (cpu, p->cpus_ptr)) + *idle_cpu = cpu; } -/* -* Failed to find an idle core; stop looking for one. -*/ -set_idle_cores (target,0 ); +if (idle) +return core; +cpumask_andnot (cpus, cpus,cpu_smt_mask (core));return -1 ; }
@@ -6107,9 +6109,18 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int#define sched_smt_weight 1 -static inline int select_idle_core (struct task_struct *p,struct sched_domain *sd,int target) +static inline void set_idle_cores (int cpu,int val) { -return -1 ; +} + +static inline bool test_idle_cores (int cpu,bool def) +{ +return def; +} + +static inline int select_idle_core (struct task_struct *p,int core,struct cpumask *cpus,int *idle_cpu) +{ +return __select_idle_cpu (p, core, cpus); }#endif /* CONFIG_SCHED_SMT */
@@ -6124,10 +6135,11 @@ static inline int select_idle_core(struct task_struct *p, struct sched_domain *sstatic int select_idle_cpu (struct task_struct *p,struct sched_domain *sd,int target) {struct cpumask *cpus =this_cpu_cpumask_var_ptr (select_idle_mask); +int i, cpu, idle_cpu =-1 , nr =INT_MAX ; +bool smt =test_idle_cores (target, false); +int this =smp_processor_id ();struct sched_domain *this_sd;u64 time; -int this =smp_processor_id (); -int cpu, nr =INT_MAX ; this_sd =rcu_dereference (*this_cpu_ptr (&sd_llc));if (!this_sd)
@@ -6135,7 +6147,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tcpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); -if (sched_feat (SIS_PROP )) { +if (sched_feat (SIS_PROP ) && !smt) {u64 avg_cost, avg_idle, span_avg;/*
@@ -6159,16 +6171,29 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ; -if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; +if (smt) { + i =select_idle_core (p, cpu, cpus, &idle_cpu); +if ((unsigned int )i < nr_cpumask_bits) +return i; + + }else { + i =__select_idle_cpu (p, cpu, cpus); +if ((unsigned int )i < nr_cpumask_bits) { + idle_cpu = i; +break ; + } + } } -if (sched_feat (SIS_PROP )) { +if (smt) +set_idle_cores (this, false); + +if (sched_feat (SIS_PROP ) && !smt) { time =cpu_clock (this) - time;update_avg (&this_sd->avg_scan_cost, time); } -return cpu; +return idle_cpu; }/*
@@ -6297,10 +6322,6 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)if (!sd)return target; - i =select_idle_core (p, sd, target); -if ((unsigned )i < nr_cpumask_bits) -return i; - i =select_idle_cpu (p, sd, target);if ((unsigned )i < nr_cpumask_bits)return i;
Signature · 1 line
2.26.2
Re: [PATCH 5/5] sched/fair: Merge select_idle_core/cpu()
Quoted 20 lines · Mel Gorman
From: Peter Zijlstra (Intel) <[email protected]>
Both
select_idle_core()andselect_idle_cpu()do a loop over the same cpumask. Observe that by clearing the already visited CPUs, we can fold the iteration and iterate a core at a time.All we need to do is remember any non-idle CPU we encountered while scanning for an idle core. This way we'll only iterate every CPU once.
Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Mel Gorman <[email protected]>--- kernel/sched/fair.c | 97 +++++++++++++++++++++++++++------------------ 1 file changed, 59 insertions(+), 38 deletions(-)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *p · line 6009
Quoted 4 lines
kernel/sched/fair.c +1
-0@@ -6006,6 +6006,14 @@ static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *preturn new_cpu; } +static inline int __select_idle_cpu (struct task_struct *p,int core,struct cpumask *cpus)
Sorry if I missed anything, why p and cpus are needed here?
Quoted 106 lines
kernel/sched/fair.c +40
-29+{ +if (available_idle_cpu (core) ||sched_idle_cpu (core)) +return core; + +return -1 ; +} +#ifdef CONFIG_SCHED_SMT DEFINE_STATIC_KEY_FALSE (sched_smt_present);EXPORT_SYMBOL_GPL (sched_smt_present);@@ -6066,40 +6074,34 @@ void __update_idle_core(struct rq *rq)* there are no idle cores left in the system; tracked through * sd_llc->shared->has_idle_cores and enabled through update_idle_core() above. */ -static int select_idle_core (struct task_struct *p,struct sched_domain *sd,int target) +static int select_idle_core (struct task_struct *p,int core,struct cpumask *cpus,int *idle_cpu) { -struct cpumask *cpus =this_cpu_cpumask_var_ptr (select_idle_mask); -int core, cpu; +bool idle = true; +int cpu;if (!static_branch_likely (&sched_smt_present)) -return -1 ; - -if (!test_idle_cores (target, false)) -return -1 ; - -cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); +return __select_idle_cpu (p, core, cpus); -for_each_cpu_wrap (core, cpus, target) { -bool idle = true; - -for_each_cpu (cpu,cpu_smt_mask (core)) { -if (!available_idle_cpu (cpu)) { - idle = false; -break ; +for_each_cpu (cpu,cpu_smt_mask (core)) { +if (!available_idle_cpu (cpu)) { + idle = false; +if (*idle_cpu ==-1 ) { +if (sched_idle_cpu (cpu) &&cpumask_test_cpu (cpu, p->cpus_ptr)) { + *idle_cpu = cpu; +break ; + } +continue ; } +break ; } - -if (idle) -return core; - -cpumask_andnot (cpus, cpus,cpu_smt_mask (core)); +if (*idle_cpu ==-1 &&cpumask_test_cpu (cpu, p->cpus_ptr)) + *idle_cpu = cpu; } -/* -* Failed to find an idle core; stop looking for one. -*/ -set_idle_cores (target,0 ); +if (idle) +return core; +cpumask_andnot (cpus, cpus,cpu_smt_mask (core));return -1 ; }@@ -6107,9 +6109,18 @@ static int select_idle_core(struct task_struct *p, struct sched_domain *sd, int#define sched_smt_weight 1 -static inline int select_idle_core (struct task_struct *p,struct sched_domain *sd,int target) +static inline void set_idle_cores (int cpu,int val) { -return -1 ; +} + +static inline bool test_idle_cores (int cpu,bool def) +{ +return def; +} + +static inline int select_idle_core (struct task_struct *p,int core,struct cpumask *cpus,int *idle_cpu) +{ +return __select_idle_cpu (p, core, cpus); }#endif /* CONFIG_SCHED_SMT */ @@ -6124,10 +6135,11 @@ static inline int select_idle_core(struct task_struct *p, struct sched_domain *sstatic int select_idle_cpu (struct task_struct *p,struct sched_domain *sd,int target) {struct cpumask *cpus =this_cpu_cpumask_var_ptr (select_idle_mask); +int i, cpu, idle_cpu =-1 , nr =INT_MAX ; +bool smt =test_idle_cores (target, false); +int this =smp_processor_id ();struct sched_domain *this_sd;u64 time; -int this =smp_processor_id (); -int cpu, nr =INT_MAX ; this_sd =rcu_dereference (*this_cpu_ptr (&sd_llc));if (!this_sd)
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6150
Quoted 5 lines
kernel/sched/fair.c +1
-1@@ -6135,7 +6147,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tcpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); -if (sched_feat (SIS_PROP )) { +if (sched_feat (SIS_PROP ) && !smt) {
Is it possible the system does have a idle core, but I still don't want to scan the entire llc domain?
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6173
Quoted 7 lines
kernel/sched/fair.c +0
-0u64 avg_cost, avg_idle, span_avg;/* @@ -6159,16 +6171,29 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ;
It looks like nr only makes sense when smt = false now, can it be moved into else branch below?
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6177
Quoted 6 lines
kernel/sched/fair.c +4
-2-if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; +if (smt) { + i =select_idle_core (p, cpu, cpus, &idle_cpu); +if ((unsigned int )i < nr_cpumask_bits) +return i;
What if the last idle core is selected here, should we set_idle_cores false before return?
Quoted 36 lines
kernel/sched/fair.c +13
-6+ + }else { + i =__select_idle_cpu (p, cpu, cpus); +if ((unsigned int )i < nr_cpumask_bits) { + idle_cpu = i; +break ; + } + } } -if (sched_feat (SIS_PROP )) { +if (smt) +set_idle_cores (this, false); + +if (sched_feat (SIS_PROP ) && !smt) { time =cpu_clock (this) - time;update_avg (&this_sd->avg_scan_cost, time); } -return cpu; +return idle_cpu; }/* @@ -6297,10 +6322,6 @@ static int select_idle_sibling(struct task_struct *p, int prev, int target)if (!sd)return target; - i =select_idle_core (p, sd, target); -if ((unsigned )i < nr_cpumask_bits) -return i; - i =select_idle_cpu (p, sd, target);if ((unsigned )i < nr_cpumask_bits)return i;
Re: [PATCH 5/5] sched/fair: Merge select_idle_core/cpu()
Quoted 8 lines · Aubrey
Quoted 8 lines
kernel/sched/fair.c +0
-0@@ -6006,6 +6006,14 @@ static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *preturn new_cpu; }
Quoted 4 lines
kernel/sched/fair.c static inline int find_idlest_cpu(struct sched_domain *sd, struct task_struct *p · line 6009
Quoted 1 line
kernel/sched/fair.c +1
-0+static inline int __select_idle_cpu (struct task_struct *p,int core,struct cpumask *cpus)Sorry if I missed anything, why p and cpus are needed here?
They are not needed. The original code was matching the calling pattern for select_idle_core() which needs p and cpus to check if sibling CPUs are allowed.
Quoted 9 lines
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6150
Quoted 6 lines
kernel/sched/fair.c +1
-1@@ -6135,7 +6147,7 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tcpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); -if (sched_feat (SIS_PROP )) { +if (sched_feat (SIS_PROP ) && !smt) {Is it possible the system does have a idle core, but I still don't want to scan the entire llc domain?
This version is matching historical behaviour. To limit the scan for cores, select_idle_core() would need to obey SIS_PROP and that patch was dropped as it introduced regressions. It would only be considered once SIS_PROP had better metrics for limiting the depth of the search.
Quoted 10 lines
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6173
Quoted 7 lines
kernel/sched/fair.c +0
-0u64 avg_cost, avg_idle, span_avg;/* @@ -6159,16 +6171,29 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int tfor_each_cpu_wrap (cpu, cpus, target) {if (!--nr)return -1 ;It looks like nr only makes sense when smt = false now, can it be moved into else branch below?
It can. I expect the saving to be marginal and it will need to move back when/if select_idle_core() obeys SIS_PROP.
Quoted 9 lines
kernel/sched/fair.c static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, int t · line 6177
Quoted 6 lines
kernel/sched/fair.c +4
-2-if (available_idle_cpu (cpu) ||sched_idle_cpu (cpu)) -break ; +if (smt) { + i =select_idle_core (p, cpu, cpus, &idle_cpu); +if ((unsigned int )i < nr_cpumask_bits) +return i;What if the last idle core is selected here, should we
set_idle_coresfalse before return?
We'd have to check what bits were still set in the cpus mask and determine if they represent an idle core. I severely doubt it would be worth the cost given that the availability of idle cores can change at any instant.
Signature · 2 lines
Mel Gorman SUSE Labs