sched: Introduce Cache aware scheduling
Subsystems (4)
cover sched: Introduce Cache aware scheduling
This is a respin of the cache-aware scheduling proposed by Peter[1]. In this patch set, some known issues in [1] were addressed, and the performance regression was investigated and mitigated.
Cache-aware scheduling aims to aggregate tasks with potential shared resources into the same cache domain. This approach enhances cache locality, thereby optimizing system performance by reducing cache misses and improving data access efficiency.
In the current implementation, threads within the same process are considered as entities that potentially share resources. Cache-aware scheduling monitors the CPU occupancy of each cache domain for every process. Based on this monitoring, it endeavors to migrate threads within a given process to its cache-hot domains, with the goal of maximizing cache locality.
Patch 1 constitutes the fundamental cache-aware scheduling. It is the same patch as [1]. Patch 2 comprises a series of fixes for Patch 1, including compiling warnings and functional fixes.
Patch 3 fixes performance degradation that arise from excessive task migrations within the preferred LLC domain.
Patch 4 further alleviates performance regressions when the preferred LLC becomes saturated. Patch 5 introduces ftrace events, which is used to track task migrations triggered by wakeup and load balancer. This addition facilitate performance regression analysis.
The patch set is applied on top of v6.14 sched/core,
commit 4ba7518327c6 ("sched/debug: Print the local group's asym_prefer_cpu")
schbench was tested on EMR and Zen3 Milan. An improvement in tail latency was observed when the LLC was underloaded; however, some regressions were still evident when the LLC was saturated. Additionally, the load balance should be adjusted to further address these regressions.
[1] https://lore.kernel.org/all/[email protected]/
Chen Yu (4):
sched: Several fixes for cache aware scheduling
sched: Avoid task migration within its preferred LLC
sched: Inhibit cache aware scheduling if the preferred LLC is over
aggregated
sched: Add ftrace to track task migration and load balance within and
across LLC
Peter Zijlstra (1):
sched: Cache aware load-balancing
include/linux/mm_types.h | 44 ++++ include/linux/sched.h | 4 + include/linux/sched/topology.h | 4 + include/trace/events/sched.h | 51 ++++ init/Kconfig | 4 + kernel/fork.c | 5 + kernel/sched/core.c | 13 +- kernel/sched/fair.c | 461 +++++++++++++++++++++++++++++++-- kernel/sched/features.h | 1 + kernel/sched/sched.h | 8 + 10 files changed, 569 insertions(+), 26 deletions(-)
Signature · 1 line
2.25.1
RFC PATCH 1/5 sched: Cache aware load-balancing
From: Peter Zijlstra <[email protected]>
Hi all,
One of the many things on the eternal todo list has been finishing the below hackery.
It is an attempt at modelling cache affinity -- and while the patch really only targets LLC, it could very well be extended to also apply to clusters (L2). Specifically any case of multiple cache domains inside a node.
Anyway, I wrote this about a year ago, and I mentioned this at the recent OSPM conf where Gautham and Prateek expressed interest in playing with this code.
So here goes, very rough and largely unproven code ahead :-)
It applies to current tip/master, but I know it will fail the __percpu validation that sits in -next, although that shouldn't be terribly hard to fix up.
As is, it only computes a CPU inside the LLC that has the highest recent runtime, this CPU is then used in the wake-up path to steer towards this LLC and in task_hot() to limit migrations away from it.
More elaborate things could be done, notably there is an XXX in there somewhere about finding the best LLC inside a NODE (interaction with NUMA_BALANCING).
--- include/linux/mm_types.h | 44 ++++++ include/linux/sched.h | 4 + init/Kconfig | 4 + kernel/fork.c | 5 + kernel/sched/core.c | 13 +- kernel/sched/fair.c | 330 +++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 8 + 7 files changed, 388 insertions(+), 20 deletions(-)
include/linux/mm_types.h +44 -0
@@ -893,6 +893,12 @@ struct mm_cid { };#endif +struct mm_sched { +u64 runtime; +unsigned long epoch; +unsigned long occ; +}; +struct kioctx_table ;struct iommu_mm_data ;struct mm_struct {
@@ -983,6 +989,17 @@ struct mm_struct {*/ raw_spinlock_t cpus_allowed_lock;#endif +#ifdef CONFIG_SCHED_CACHE +/* +* Track per-cpu-per-process occupancy as a proxy for cache residency. +* See account_mm_sched() and ... +*/ +struct mm_sched __percpu *pcpu_sched; +raw_spinlock_t mm_sched_lock; +unsigned long mm_sched_epoch; +int mm_sched_cpu; +#endif +#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */ #endif
@@ -1393,6 +1410,33 @@ static inline unsigned int mm_cid_size(void)static inline void mm_set_cpus_allowed (struct mm_struct *mm,const struct cpumask *cpumask) { }#endif /* CONFIG_SCHED_MM_CID */ +#ifdef CONFIG_SCHED_CACHE +extern void mm_init_sched (struct mm_struct *mm,struct mm_sched *pcpu_sched); + +static inline int mm_alloc_sched_noprof (struct mm_struct *mm) +{ + struct mm_sched *pcpu_sched = alloc_percpu_noprof(struct mm_sched); +if (!pcpu_sched) +return -ENOMEM ; + +mm_init_sched (mm, pcpu_sched); +return 0 ; +} + +#define mm_alloc_sched (...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) + +static inline void mm_destroy_sched (struct mm_struct *mm) +{ +free_percpu (mm->pcpu_sched); + mm->pcpu_sched =NULL ; +} +#else /* !CONFIG_SCHED_CACHE */ + +static inline int mm_alloc_sched (struct mm_struct *mm) {return 0 ; } +static inline void mm_destroy_sched (struct mm_struct *mm) { } + +#endif /* CONFIG_SCHED_CACHE */ +struct mmu_gather ;extern void tlb_gather_mmu (struct mmu_gather *tlb,struct mm_struct *mm);extern void tlb_gather_mmu_fullmm (struct mmu_gather *tlb,struct mm_struct *mm);
include/linux/sched.h +4 -0
@@ -1399,6 +1399,10 @@ struct task_struct {unsigned long numa_pages_migrated;#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +struct callback_head cache_work; +#endif +#ifdef CONFIG_RSEQ struct rseq __user *rseq;u32 rseq_len;
init/Kconfig +4 -0
@@ -950,6 +950,10 @@ config NUMA_BALANCING This system will be inactive on UMA systems. +config SCHED_CACHE + bool "Cache aware scheduler" + default y + config NUMA_BALANCING_DEFAULT_ENABLED bool "Automatically enable NUMA aware memory/task placement" default y
kernel/fork.c +5 -0
@@ -1331,6 +1331,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,if (mm_alloc_cid (mm, p)) goto fail_cid; +if (mm_alloc_sched (mm)) + goto fail_sched; +if (percpu_counter_init_many (mm->rss_stat,0 ,GFP_KERNEL_ACCOUNT ,NR_MM_COUNTERS )) goto fail_pcpu;
@@ -1340,6 +1343,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,return mm; fail_pcpu: +mm_destroy_sched (mm); +fail_sched:mm_destroy_cid (mm); fail_cid:destroy_context (mm);
kernel/sched/core.c +10 -3
@@ -4524,6 +4524,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->migration_pending =NULL ;#endif init_sched_mm_cid (p); +init_sched_mm (p); }DEFINE_STATIC_KEY_FALSE (sched_numa_balancing);
@@ -8528,6 +8529,7 @@ static struct kmem_cache *task_group_cache __ro_after_init;void __init sched_init (void ) { +unsigned long now = jiffies;unsigned long ptr =0 ;int i;
@@ -8602,7 +8604,7 @@ void __init sched_init(void)raw_spin_lock_init (&rq->__lock); rq->nr_running =0 ; rq->calc_load_active =0 ; - rq->calc_load_update = jiffies +LOAD_FREQ ; + rq->calc_load_update = now +LOAD_FREQ ;init_cfs_rq (&rq->cfs);init_rt_rq (&rq->rt);init_dl_rq (&rq->dl);
@@ -8646,7 +8648,7 @@ void __init sched_init(void) rq->cpu_capacity =SCHED_CAPACITY_SCALE ; rq->balance_callback = &balance_push_callback; rq->active_balance =0 ; - rq->next_balance = jiffies; + rq->next_balance = now; rq->push_cpu =0 ; rq->cpu = i; rq->online =0 ;
@@ -8658,7 +8660,7 @@ void __init sched_init(void)rq_attach_root (rq, &def_root_domain);#ifdef CONFIG_NO_HZ_COMMON - rq->last_blocked_load_update_tick = jiffies; + rq->last_blocked_load_update_tick = now;atomic_set (&rq->nohz_flags,0 );INIT_CSD (&rq->nohz_csd, nohz_csd_func, rq);
@@ -8683,6 +8685,11 @@ void __init sched_init(void) rq->core_cookie =0UL ;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spin_lock_init (&rq->cpu_epoch_lock); + rq->cpu_epoch_next = now; +#endif +zalloc_cpumask_var_node (&rq->scratch_mask,GFP_KERNEL ,cpu_to_node (i)); }
kernel/sched/fair.c +313 -17
450 lines not on this page · raw message
kernel/sched/sched.h +8 -0
@@ -1173,6 +1173,12 @@ struct rq {u64 clock_pelt_idle_copy;u64 clock_idle_copy;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spinlock_t cpu_epoch_lock; +u64 cpu_runtime; +unsigned long cpu_epoch; +unsigned long cpu_epoch_next; +#endif atomic_t nr_iowait;
@@ -3887,6 +3893,8 @@ static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) { }static inline void init_sched_mm_cid (struct task_struct *t) { }#endif /* !CONFIG_SCHED_MM_CID */ +extern void init_sched_mm (struct task_struct *p); +extern u64 avg_vruntime (struct cfs_rq *cfs_rq);extern int entity_eligible (struct cfs_rq *cfs_rq,struct sched_entity *se);#ifdef CONFIG_SMP
Signature · 1 line
2.25.1
kernel test robot
build botRe: [RFC PATCH 1/5] sched: Cache aware load-balancing
kernel test robot noticed the following build warnings:
Hi Chen,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/sched/core]
[also build test WARNING on akpm-mm/mm-everything peterz-queue/sched/core linus/master v6.15-rc3 next-20250417] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Chen-Yu/sched-Cache-aware-load-balancing/20250421-113804 base: tip/sched/core patch link: https://lore.kernel.org/r/391c48836585786ed32d66df9534366459684383.1745199017.git.yu.c.chen%40intel.com patch subject: [RFC PATCH 1/5] sched: Cache aware load-balancing config: alpha-randconfig-r121-20250421 (https://download.01.org/0day-ci/archive/20250422/[email protected]/config) compiler: alpha-linux-gcc (GCC) 8.5.0 reproduce: (https://download.01.org/0day-ci/archive/20250422/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
kernel/sched/fair.c: In function '__update_mm_sched':
>> kernel/sched/fair.c:1217:34: warning: division by zero [-Wdiv-by-zero]
n = (delta + EPOCH_PERIOD - 1) / EPOCH_PERIOD;
^
In file included from include/asm-generic/percpu.h:7,
from arch/alpha/include/asm/percpu.h:17,
from include/linux/irqflags.h:19,
from include/linux/spinlock.h:59,
from include/linux/sched.h:2217,
from include/linux/ratelimit.h:6,
from include/linux/dev_printk.h:16,
from include/linux/device.h:15,
from include/linux/energy_model.h:5,
from kernel/sched/fair.c:23:
kernel/sched/fair.c: In function 'task_cache_work':
kernel/sched/fair.c:1321:38: error: 'sd_llc' undeclared (first use in this function); did you mean 'd_alloc'?
struct sched_domain *sd = per_cpu(sd_llc, cpu);
^~~~~~
include/linux/percpu-defs.h:220:47: note: in definition of macro '__verify_pcpu_ptr'
const void __percpu *__vpp_verify = (typeof((ptr) + 0))NULL; \
^~~
include/linux/percpu-defs.h:272:29: note: in expansion of macro 'per_cpu_ptr'
#define per_cpu(var, cpu) (*per_cpu_ptr(&(var), cpu))
^~~~~~~~~~~
kernel/sched/fair.c:1321:30: note: in expansion of macro 'per_cpu'
struct sched_domain *sd = per_cpu(sd_llc, cpu);
^~~~~~~
kernel/sched/fair.c:1321:38: note: each undeclared identifier is reported only once for each function it appears in
struct sched_domain *sd = per_cpu(sd_llc, cpu);
^~~~~~
include/linux/percpu-defs.h:220:47: note: in definition of macro '__verify_pcpu_ptr'
const void __percpu *__vpp_verify = (typeof((ptr) + 0))NULL; \
^~~
include/linux/percpu-defs.h:272:29: note: in expansion of macro 'per_cpu_ptr'
#define per_cpu(var, cpu) (*per_cpu_ptr(&(var), cpu))
^~~~~~~~~~~
kernel/sched/fair.c:1321:30: note: in expansion of macro 'per_cpu'
struct sched_domain *sd = per_cpu(sd_llc, cpu);
^~~~~~~
In file included from include/linux/bitmap.h:11,
from include/linux/cpumask.h:12,
from include/linux/energy_model.h:4,
from kernel/sched/fair.c:23:
kernel/sched/fair.c:1325:20: error: implicit declaration of function 'sched_domain_span'; did you mean 'perf_domain_span'? [-Werror=implicit-function-declaration]
for_each_cpu(i, sched_domain_span(sd)) {
^~~~~~~~~~~~~~~~~
include/linux/find.h:586:41: note: in definition of macro 'for_each_set_bit'
for ((bit) = 0; (bit) = find_next_bit((addr), (size), (bit)), (bit) < (size); (bit)++)
^~~~
include/linux/cpumask.h:333:24: note: in expansion of macro 'cpumask_bits'
for_each_set_bit(cpu, cpumask_bits(mask), small_cpumask_bits)
^~~~~~~~~~~~
kernel/sched/fair.c:1325:4: note: in expansion of macro 'for_each_cpu'
for_each_cpu(i, sched_domain_span(sd)) {
^~~~~~~~~~~~
include/linux/cpumask_types.h:18:37: error: invalid type argument of '->' (have 'int')
#define cpumask_bits(maskp) ((maskp)->bits)
^~
include/linux/find.h:586:41: note: in definition of macro 'for_each_set_bit'
for ((bit) = 0; (bit) = find_next_bit((addr), (size), (bit)), (bit) < (size); (bit)++)
^~~~
include/linux/cpumask.h:333:24: note: in expansion of macro 'cpumask_bits'
for_each_set_bit(cpu, cpumask_bits(mask), small_cpumask_bits)
^~~~~~~~~~~~
kernel/sched/fair.c:1325:4: note: in expansion of macro 'for_each_cpu'
for_each_cpu(i, sched_domain_span(sd)) {
^~~~~~~~~~~~
include/linux/find.h:586:62: warning: left-hand operand of comma expression has no effect [-Wunused-value]
for ((bit) = 0; (bit) = find_next_bit((addr), (size), (bit)), (bit) < (size); (bit)++)
^
include/linux/cpumask.h:333:2: note: in expansion of macro 'for_each_set_bit'
for_each_set_bit(cpu, cpumask_bits(mask), small_cpumask_bits)
^~~~~~~~~~~~~~~~
kernel/sched/fair.c:1325:4: note: in expansion of macro 'for_each_cpu'
for_each_cpu(i, sched_domain_span(sd)) {
^~~~~~~~~~~~
In file included from include/asm-generic/percpu.h:7,
from arch/alpha/include/asm/percpu.h:17,
from include/linux/irqflags.h:19,
from include/linux/spinlock.h:59,
from include/linux/sched.h:2217,
from include/linux/ratelimit.h:6,
from include/linux/dev_printk.h:16,
from include/linux/device.h:15,
from include/linux/energy_model.h:5,
from kernel/sched/fair.c:23:
kernel/sched/fair.c:1335:19: error: 'sd_llc_id' undeclared (first use in this function)
per_cpu(sd_llc_id, i), occ, m_occ, m_cpu, nr);
^~~~~~~~~
include/linux/percpu-defs.h:220:47: note: in definition of macro '__verify_pcpu_ptr'
const void __percpu *__vpp_verify = (typeof((ptr) + 0))NULL; \
^~~
include/linux/percpu-defs.h:272:29: note: in expansion of macro 'per_cpu_ptr'
#define per_cpu(var, cpu) (*per_cpu_ptr(&(var), cpu))
^~~~~~~~~~~
kernel/sched/fair.c:1335:11: note: in expansion of macro 'per_cpu'
per_cpu(sd_llc_id, i), occ, m_occ, m_cpu, nr);
^~~~~~~ In file included from include/linux/bitmap.h:11,
from include/linux/cpumask.h:12, from include/linux/energy_model.h:4, from kernel/sched/fair.c:23:
vim +1217 kernel/sched/fair.c
1208
1209 static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)
1210 {
1211 lockdep_assert_held(&rq->cpu_epoch_lock);
1212
1213 unsigned long n, now = jiffies;
1214 long delta = now - rq->cpu_epoch_next;
1215
1216 if (delta > 0) {
> 1217 n = (delta + EPOCH_PERIOD - 1) / EPOCH_PERIOD;
1218 rq->cpu_epoch += n;
1219 rq->cpu_epoch_next += n * EPOCH_PERIOD;
1220 __shr_u64(&rq->cpu_runtime, n);
1221 }
1222
1223 n = rq->cpu_epoch - pcpu_sched->epoch;
1224 if (n) {
1225 pcpu_sched->epoch += n;
1226 __shr_u64(&pcpu_sched->runtime, n);
1227 }
1228 }
1229
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RFC PATCH 2/5 sched: Several fixes for cache aware scheduling
In reply to Chen Yu1. Fix the compile errors on per-CPU allocation.
2. Enqueue tasks to the target CPU instead of the current CPU; otherwise, the per-CPU occupancy will be messed up.
3. Fix the NULL LLC sched domain issue(Libo Chen).
4. Avoid duplicated epoch check in task_tick_cache()
5. Introduce sched feature SCHED_CACHE to control cache aware scheduling
TBD suggestion in previous version:
move cache_work from per task to per mm_struct, consider the actual cpu capacity in fraction_mm_sched() (Abel Wu)
--- include/linux/mm_types.h | 4 ++-- kernel/sched/fair.c | 15 +++++++++------ kernel/sched/features.h | 1 + 3 files changed, 12 insertions(+), 8 deletions(-)
include/linux/mm_types.h +2 -2
@@ -1411,11 +1411,11 @@ static inline void mm_set_cpus_allowed(struct mm_struct *mm, const struct cpumas#endif /* CONFIG_SCHED_MM_CID */ #ifdef CONFIG_SCHED_CACHE -extern void mm_init_sched (struct mm_struct *mm,struct mm_sched *pcpu_sched); +extern void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *pcpu_sched);static inline int mm_alloc_sched_noprof (struct mm_struct *mm) { - struct mm_sched *pcpu_sched = alloc_percpu_noprof(struct mm_sched); + struct mm_sched __percpu *pcpu_sched = alloc_percpu_noprof(struct mm_sched);if (!pcpu_sched)return -ENOMEM ;
kernel/sched/fair.c +9 -6
@@ -1175,7 +1175,7 @@ static s64 update_curr_se(struct rq *rq, struct sched_entity *curr)#define EPOCH_PERIOD (HZ/100)/* 10 ms */ #define EPOCH_OLD 5/* 50 ms */ -void mm_init_sched (struct mm_struct *mm,struct mm_sched *_pcpu_sched) +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) {unsigned long epoch;int i;
@@ -1254,7 +1254,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (!mm || !mm->pcpu_sched)return ; - pcpu_sched =this_cpu_ptr (p->mm->pcpu_sched); + pcpu_sched =per_cpu_ptr (p->mm->pcpu_sched,cpu_of (rq)); scoped_guard (raw_spinlock, &rq->cpu_epoch_lock) {__update_mm_sched (rq, pcpu_sched);
@@ -1286,9 +1286,6 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p)guard (raw_spinlock)(&mm->mm_sched_lock); -if (mm->mm_sched_epoch == rq->cpu_epoch) -return ; -if (work->next == work) {task_work_add (p, work,TWA_RESUME );WRITE_ONCE (mm->mm_sched_epoch, rq->cpu_epoch);
@@ -1322,6 +1319,9 @@ static void task_cache_work(struct callback_head *work)unsigned long occ, m_occ =0 , a_occ =0 ;int m_cpu =-1 , nr =0 , i; +if (!sd) +continue ; +for_each_cpu (i,sched_domain_span (sd)) { occ =fraction_mm_sched (cpu_rq (i),per_cpu_ptr (mm->pcpu_sched, i));
@@ -8796,6 +8796,9 @@ static int select_cache_cpu(struct task_struct *p, int prev_cpu)struct mm_struct *mm = p->mm;int cpu; +if (!sched_feat (SCHED_CACHE )) +return prev_cpu; +if (!mm || p->nr_cpus_allowed ==1 )return prev_cpu;
@@ -9550,7 +9553,7 @@ static int task_hot(struct task_struct *p, struct lb_env *env)return 0 ;#ifdef CONFIG_SCHED_CACHE -if (p->mm && p->mm->pcpu_sched) { +if (sched_feat (SCHED_CACHE ) && p->mm && p->mm->pcpu_sched) {/* * XXX things like Skylake have non-inclusive L3 and might not * like this L3 centric view. What to do about L2 stickyness ?
kernel/sched/features.h +1 -0
@@ -87,6 +87,7 @@ SCHED_FEAT(TTWU_QUEUE, true)*/ SCHED_FEAT(SIS_UTIL, true) +SCHED_FEAT (SCHED_CACHE , true)/* * Issue a WARN when we do multiple update_rq_clock() calls * in a single rq->lock section. Default disabled because the
Signature · 1 line
2.25.1
RFC PATCH 3/5 sched: Avoid task migration within its preferred LLC
In reply to Chen YuIt was found that when running schbench, there is a significant amount of in-LLC task migration, even if the wakee is woken up on its preferred LLC. This leads to core-to-core latency and impairs performance.
Inhibit task migration if the wakee is already in its preferred LLC. Meanwhile, prevent the load balancer from treating the task as cache-hot if this task is being migrated out of its preferred LLC, rather than comparing the occupancy between CPUs.
With this enhancement applied, the in-LLC task migration has been reduced a lot(use PATCH 5/5 to verify).
It was found that when schbench is running, there is a significant amount of in-LLC task migration, even if the wakee is woken up on its preferred LLC. This leads to core-to-core latency and impairs performance.
Inhibit task migration if the wakee is already in its preferred LLC. Meanwhile, prevent the load balancer from treating the task as cache-hot if this task is being migrated out of its preferred LLC, instead of comparing occupancy between CPUs directly.
With this enhancement applied, the in-LLC task migration has been reduced significantly, (use PATCH 5/5 to verify).
--- kernel/sched/fair.c | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-)
kernel/sched/fair.c +13 -8
@@ -8806,6 +8806,12 @@ static int select_cache_cpu(struct task_struct *p, int prev_cpu)if (cpu <0 )return prev_cpu; +/* +* No need to migrate the task if previous and preferred CPU +* are in the same LLC. +*/ +if (cpus_share_cache (prev_cpu, cpu)) +return prev_cpu;if (static_branch_likely (&sched_numa_balancing) &&__migrate_degrades_locality (p, prev_cpu, cpu, false) >0 ) {
@@ -9553,14 +9559,13 @@ static int task_hot(struct task_struct *p, struct lb_env *env)return 0 ;#ifdef CONFIG_SCHED_CACHE -if (sched_feat (SCHED_CACHE ) && p->mm && p->mm->pcpu_sched) { -/* -* XXX things like Skylake have non-inclusive L3 and might not -* like this L3 centric view. What to do about L2 stickyness ? -*/ -return per_cpu_ptr (p->mm->pcpu_sched, env->src_cpu)->occ > -per_cpu_ptr (p->mm->pcpu_sched, env->dst_cpu)->occ; - } +/* +* Don't migrate task out of its preferred LLC. +*/ +if (sched_feat (SCHED_CACHE ) && p->mm && p->mm->mm_sched_cpu >=0 && +cpus_share_cache (env->src_cpu, p->mm->mm_sched_cpu) && + !cpus_share_cache (env->src_cpu, env->dst_cpu)) +return 1 ;#endif delta =rq_clock_task (env->src_rq) - p->se.exec_start;
Signature · 1 line
2.25.1
RFC PATCH 4/5 sched: Inhibit cache aware scheduling if the preferred LLC is over aggregated
In reply to Chen YuIt is found that when the process's preferred LLC gets saturated by too many threads, task contention is very frequent and causes performance regression.
Save the per LLC statistics calculated by periodic load balance. The statistics include the average utilization and the average number of runnable tasks. The task wakeup path for cache aware scheduling manipulates these statistics to inhibit cache aware scheduling to avoid performance regression. When either the average utilization of the preferred LLC has reached 25%, or the average number of runnable tasks has exceeded 1/3 of the LLC weight, the cache aware wakeup is disabled. Only when the process has more threads than the LLC weight will this restriction be enabled.
Running schbench via mmtests on a Xeon platform, which has 2 sockets, each socket has 60 Cores/120 CPUs. The DRAM interleave is enabled across NUMA nodes via BIOS, so there are 2 "LLCs" in 1 NUMA node.
compare-mmtests.pl --directory work/log --benchmark schbench --names baseline,sched_cache
baselin sched_cach
baseline sched_cache
Lat 50.0th-qrtle-1 6.00 ( 0.00%) 6.00 ( 0.00%) Lat 90.0th-qrtle-1 10.00 ( 0.00%) 9.00 ( 10.00%) Lat 99.0th-qrtle-1 29.00 ( 0.00%) 13.00 ( 55.17%) Lat 99.9th-qrtle-1 35.00 ( 0.00%) 21.00 ( 40.00%) Lat 20.0th-qrtle-1 266.00 ( 0.00%) 266.00 ( 0.00%) Lat 50.0th-qrtle-2 8.00 ( 0.00%) 6.00 ( 25.00%) Lat 90.0th-qrtle-2 10.00 ( 0.00%) 10.00 ( 0.00%) Lat 99.0th-qrtle-2 19.00 ( 0.00%) 18.00 ( 5.26%) Lat 99.9th-qrtle-2 27.00 ( 0.00%) 29.00 ( -7.41%) Lat 20.0th-qrtle-2 533.00 ( 0.00%) 507.00 ( 4.88%) Lat 50.0th-qrtle-4 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-4 8.00 ( 0.00%) 5.00 ( 37.50%) Lat 99.0th-qrtle-4 14.00 ( 0.00%) 9.00 ( 35.71%) Lat 99.9th-qrtle-4 22.00 ( 0.00%) 14.00 ( 36.36%) Lat 20.0th-qrtle-4 1070.00 ( 0.00%) 995.00 ( 7.01%) Lat 50.0th-qrtle-8 5.00 ( 0.00%) 5.00 ( 0.00%) Lat 90.0th-qrtle-8 7.00 ( 0.00%) 5.00 ( 28.57%) Lat 99.0th-qrtle-8 12.00 ( 0.00%) 11.00 ( 8.33%) Lat 99.9th-qrtle-8 19.00 ( 0.00%) 16.00 ( 15.79%) Lat 20.0th-qrtle-8 2140.00 ( 0.00%) 2140.00 ( 0.00%) Lat 50.0th-qrtle-16 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-16 7.00 ( 0.00%) 5.00 ( 28.57%) Lat 99.0th-qrtle-16 12.00 ( 0.00%) 10.00 ( 16.67%) Lat 99.9th-qrtle-16 17.00 ( 0.00%) 14.00 ( 17.65%) Lat 20.0th-qrtle-16 4296.00 ( 0.00%) 4200.00 ( 2.23%) Lat 50.0th-qrtle-32 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-32 8.00 ( 0.00%) 6.00 ( 25.00%) Lat 99.0th-qrtle-32 12.00 ( 0.00%) 10.00 ( 16.67%) Lat 99.9th-qrtle-32 17.00 ( 0.00%) 14.00 ( 17.65%) Lat 20.0th-qrtle-32 8496.00 ( 0.00%) 8528.00 ( -0.38%) Lat 50.0th-qrtle-64 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-64 8.00 ( 0.00%) 8.00 ( 0.00%) Lat 99.0th-qrtle-64 12.00 ( 0.00%) 12.00 ( 0.00%) Lat 99.9th-qrtle-64 17.00 ( 0.00%) 17.00 ( 0.00%) Lat 20.0th-qrtle-64 17120.00 ( 0.00%) 17120.00 ( 0.00%) Lat 50.0th-qrtle-128 7.00 ( 0.00%) 7.00 ( 0.00%) Lat 90.0th-qrtle-128 9.00 ( 0.00%) 9.00 ( 0.00%) Lat 99.0th-qrtle-128 13.00 ( 0.00%) 14.00 ( -7.69%) Lat 99.9th-qrtle-128 20.00 ( 0.00%) 20.00 ( 0.00%) Lat 20.0th-qrtle-128 31776.00 ( 0.00%) 30496.00 ( 4.03%) Lat 50.0th-qrtle-239 9.00 ( 0.00%) 9.00 ( 0.00%) Lat 90.0th-qrtle-239 14.00 ( 0.00%) 18.00 ( -28.57%) Lat 99.0th-qrtle-239 43.00 ( 0.00%) 56.00 ( -30.23%) Lat 99.9th-qrtle-239 106.00 ( 0.00%) 483.00 (-355.66%) Lat 20.0th-qrtle-239 30176.00 ( 0.00%) 29984.00 ( 0.64%)
We can see overall latency improvement and some throughput degradation when the system gets saturated.
Also, we run schbench (old version) on an EPYC 7543 system, which has 4 NUMA nodes, and each node has 4 LLCs. Monitor the 99.0th latency:
case load baseline(std%) compare%( std%) normal 4-mthreads-1-workers 1.00 ( 6.47) +9.02 ( 4.68) normal 4-mthreads-2-workers 1.00 ( 3.25) +28.03 ( 8.76) normal 4-mthreads-4-workers 1.00 ( 6.67) -4.32 ( 2.58) normal 4-mthreads-8-workers 1.00 ( 2.38) +1.27 ( 2.41) normal 4-mthreads-16-workers 1.00 ( 5.61) -8.48 ( 4.39) normal 4-mthreads-31-workers 1.00 ( 9.31) -0.22 ( 9.77)
When the LLC is underloaded, the latency improvement is observed. When the LLC gets saturated, we observe some degradation.
The aggregation of tasks will move tasks towards the preferred LLC pretty quickly during wake ups. However load balance will tend to move tasks away from the aggregated LLC. The two migrations are in the opposite directions and tend to bounce tasks between LLCs. Such task migrations should be impeded in load balancing as long as the home LLC. We're working on fixing up the load balancing path to address such issues.
--- include/linux/sched/topology.h | 4 ++ kernel/sched/fair.c | 101 ++++++++++++++++++++++++++++++++- 2 files changed, 104 insertions(+), 1 deletion(-)
include/linux/sched/topology.h +4 -0
@@ -78,6 +78,10 @@ struct sched_domain_shared {atomic_t nr_busy_cpus;int has_idle_cores;int nr_idle_scan; +#ifdef CONFIG_SCHED_CACHE +unsigned long util_avg; +u64 nr_avg; +#endif };struct sched_domain {
kernel/sched/fair.c +100 -1
131 lines not on this page · raw message
Signature · 1 line
2.25.1
kernel test robot
build botRe: [RFC PATCH 4/5] sched: Inhibit cache aware scheduling if the preferred LLC is over aggregated
kernel test robot noticed the following build warnings:
Hi Chen,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on tip/sched/core]
[also build test WARNING on akpm-mm/mm-everything peterz-queue/sched/core linus/master v6.15-rc3 next-20250417] [If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Chen-Yu/sched-Cache-aware-load-balancing/20250421-113804 base: tip/sched/core patch link: https://lore.kernel.org/r/2c45f6db1efef84c6c1ed514a8d24a9bc4a2ca4b.1745199017.git.yu.c.chen%40intel.com patch subject: [RFC PATCH 4/5] sched: Inhibit cache aware scheduling if the preferred LLC is over aggregated config: arc-randconfig-r122-20250421 (https://download.01.org/0day-ci/archive/20250422/[email protected]/config) compiler: arc-linux-gcc (GCC) 14.2.0 reproduce: (https://download.01.org/0day-ci/archive/20250422/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
sparse warnings: (new ones prefixed by >>)
kernel/sched/fair.c:1318:51: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct sched_domain *sd @@ got struct sched_domain [noderef] __rcu * @@
kernel/sched/fair.c:1318:51: sparse: expected struct sched_domain *sd
kernel/sched/fair.c:1318:51: sparse: got struct sched_domain [noderef] __rcu *
kernel/sched/fair.c:1432:39: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/fair.c:1432:39: sparse: expected struct task_struct *donor
kernel/sched/fair.c:1432:39: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/fair.c:1495:34: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct sched_entity const *se @@ got struct sched_entity [noderef] __rcu * @@
kernel/sched/fair.c:1495:34: sparse: expected struct sched_entity const *se
kernel/sched/fair.c:1495:34: sparse: got struct sched_entity [noderef] __rcu *
kernel/sched/fair.c:13238:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@
kernel/sched/fair.c:13238:9: sparse: expected struct sched_domain *[assigned] sd
kernel/sched/fair.c:13238:9: sparse: got struct sched_domain [noderef] __rcu *parent
kernel/sched/fair.c:7238:58: sparse: sparse: incorrect type in argument 2 (different address spaces) @@ expected struct task_struct *p @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/fair.c:7238:58: sparse: expected struct task_struct *p
kernel/sched/fair.c:7238:58: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/fair.c:8632:20: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@
kernel/sched/fair.c:8632:20: sparse: expected struct sched_domain *[assigned] sd
kernel/sched/fair.c:8632:20: sparse: got struct sched_domain [noderef] __rcu *parent
kernel/sched/fair.c:8937:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] tmp @@ got struct sched_domain [noderef] __rcu *parent @@
kernel/sched/fair.c:8937:9: sparse: expected struct sched_domain *[assigned] tmp
kernel/sched/fair.c:8937:9: sparse: got struct sched_domain [noderef] __rcu *parent
kernel/sched/fair.c:9081:39: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct task_struct *donor @@ got struct task_struct [noderef] __rcu *donor @@
kernel/sched/fair.c:9081:39: sparse: expected struct task_struct *donor
kernel/sched/fair.c:9081:39: sparse: got struct task_struct [noderef] __rcu *donor
kernel/sched/fair.c:9112:37: sparse: sparse: incorrect type in argument 1 (different address spaces) @@ expected struct task_struct *tsk @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/fair.c:9112:37: sparse: expected struct task_struct *tsk
kernel/sched/fair.c:9112:37: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/fair.c:9330:38: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct task_struct *curr @@ got struct task_struct [noderef] __rcu *curr @@
kernel/sched/fair.c:9330:38: sparse: expected struct task_struct *curr
kernel/sched/fair.c:9330:38: sparse: got struct task_struct [noderef] __rcu *curr
kernel/sched/fair.c:10374:40: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct sched_domain *child @@ got struct sched_domain [noderef] __rcu *child @@
kernel/sched/fair.c:10374:40: sparse: expected struct sched_domain *child
kernel/sched/fair.c:10374:40: sparse: got struct sched_domain [noderef] __rcu *child
>> kernel/sched/fair.c:10705:42: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct sched_domain *sd @@ got struct sched_domain [noderef] __rcu *child @@
kernel/sched/fair.c:10705:42: sparse: expected struct sched_domain *sd
kernel/sched/fair.c:10705:42: sparse: got struct sched_domain [noderef] __rcu *child
kernel/sched/fair.c:11045:22: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/fair.c:11045:22: sparse: struct task_struct [noderef] __rcu *
kernel/sched/fair.c:11045:22: sparse: struct task_struct *
kernel/sched/fair.c:12490:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@
kernel/sched/fair.c:12490:9: sparse: expected struct sched_domain *[assigned] sd
kernel/sched/fair.c:12490:9: sparse: got struct sched_domain [noderef] __rcu *parent
kernel/sched/fair.c:12140:44: sparse: sparse: incorrect type in initializer (different address spaces) @@ expected struct sched_domain *sd_parent @@ got struct sched_domain [noderef] __rcu *parent @@
kernel/sched/fair.c:12140:44: sparse: expected struct sched_domain *sd_parent
kernel/sched/fair.c:12140:44: sparse: got struct sched_domain [noderef] __rcu *parent
kernel/sched/fair.c:12599:9: sparse: sparse: incorrect type in assignment (different address spaces) @@ expected struct sched_domain *[assigned] sd @@ got struct sched_domain [noderef] __rcu *parent @@
kernel/sched/fair.c:12599:9: sparse: expected struct sched_domain *[assigned] sd
kernel/sched/fair.c:12599:9: sparse: got struct sched_domain [noderef] __rcu *parent
kernel/sched/fair.c:1234:26: sparse: sparse: context imbalance in 'fraction_mm_sched' - wrong count at exit kernel/sched/fair.c:1276:13: sparse: sparse: context imbalance in 'task_tick_cache' - different lock contexts for basic block kernel/sched/fair.c:6949:35: sparse: sparse: marked inline, but without a definition
kernel/sched/fair.c: note: in included file:
kernel/sched/sched.h:2496:9: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2496:9: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2496:9: sparse: struct task_struct *
kernel/sched/fair.c: note: in included file (through arch/arc/include/asm/irqflags.h, include/linux/irqflags.h, include/linux/spinlock.h, ...):
arch/arc/include/asm/irqflags-arcv2.h:80:31: sparse: sparse: undefined identifier '__builtin_arc_lr' arch/arc/include/asm/irqflags-arcv2.h:83:17: sparse: sparse: undefined identifier '__builtin_arc_sr' kernel/sched/fair.c: note: in included file:
kernel/sched/sched.h:2284:26: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2284:26: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2284:26: sparse: struct task_struct *
kernel/sched/sched.h:2284:26: sparse: sparse: incompatible types in comparison expression (different address spaces):
kernel/sched/sched.h:2284:26: sparse: struct task_struct [noderef] __rcu *
kernel/sched/sched.h:2284:26: sparse: struct task_struct *
vim +10705 kernel/sched/fair.c
10694
10695 #ifdef CONFIG_SCHED_CACHE
10696 /*
10697 * Save this sched group's statistic for later use:
10698 * The task wakeup and load balance can make better
10699 * decision based on these statistics.
10700 */
10701 static void update_sg_if_llc(struct lb_env *env, struct sg_lb_stats *sgs,
10702 struct sched_group *group)
10703 {
10704 /* Find the sched domain that spans this group. */
10705 struct sched_domain *sd = env->sd->child;
10706 struct sched_domain_shared *sd_share;
10707 u64 last_nr;
10708
10709 if (!sched_feat(SCHED_CACHE) || env->idle == CPU_NEWLY_IDLE)
10710 return;
10711
10712 /* only care the sched domain that spans 1 LLC */
10713 if (!sd || !(sd->flags & SD_SHARE_LLC) ||
10714 !sd->parent || (sd->parent->flags & SD_SHARE_LLC))
10715 return;
10716
10717 sd_share = rcu_dereference(per_cpu(sd_llc_shared,
10718 cpumask_first(sched_group_span(group))));
10719 if (!sd_share)
10720 return;
10721
10722 last_nr = READ_ONCE(sd_share->nr_avg);
10723 update_avg(&last_nr, sgs->sum_nr_running);
10724
10725 if (likely(READ_ONCE(sd_share->util_avg) != sgs->group_util))
10726 WRITE_ONCE(sd_share->util_avg, sgs->group_util);
10727
10728 WRITE_ONCE(sd_share->nr_avg, last_nr);
10729 }
10730 #else
10731 static inline void update_sg_if_llc(struct lb_env *env, struct sg_lb_stats *sgs,
10732 struct sched_group *group)
10733 {
10734 }
10735 #endif
10736
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [RFC PATCH 4/5] sched: Inhibit cache aware scheduling if the preferred LLC is over aggregated
In reply to Chen YuHi Chen Yu,
Quoted 82 lines · Chen Yu
It is found that when the process's preferred LLC gets saturated by too many threads, task contention is very frequent and causes performance regression.
Save the per LLC statistics calculated by periodic load balance. The statistics include the average utilization and the average number of runnable tasks. The task wakeup path for cache aware scheduling manipulates these statistics to inhibit cache aware scheduling to avoid performance regression. When either the average utilization of the preferred LLC has reached 25%, or the average number of runnable tasks has exceeded 1/3 of the LLC weight, the cache aware wakeup is disabled. Only when the process has more threads than the LLC weight will this restriction be enabled.
Running schbench via mmtests on a Xeon platform, which has 2 sockets, each socket has 60 Cores/120 CPUs. The DRAM interleave is enabled across NUMA nodes via BIOS, so there are 2 "LLCs" in 1 NUMA node.
compare-mmtests.pl --directory work/log --benchmark schbench --names baseline,sched_cache
baselinsched_cach
baselinesched_cacheLat 50.0th-qrtle-1 6.00 ( 0.00%) 6.00 ( 0.00%) Lat 90.0th-qrtle-1 10.00 ( 0.00%) 9.00 ( 10.00%) Lat 99.0th-qrtle-1 29.00 ( 0.00%) 13.00 ( 55.17%) Lat 99.9th-qrtle-1 35.00 ( 0.00%) 21.00 ( 40.00%) Lat 20.0th-qrtle-1 266.00 ( 0.00%) 266.00 ( 0.00%) Lat 50.0th-qrtle-2 8.00 ( 0.00%) 6.00 ( 25.00%) Lat 90.0th-qrtle-2 10.00 ( 0.00%) 10.00 ( 0.00%) Lat 99.0th-qrtle-2 19.00 ( 0.00%) 18.00 ( 5.26%) Lat 99.9th-qrtle-2 27.00 ( 0.00%) 29.00 ( -7.41%) Lat 20.0th-qrtle-2 533.00 ( 0.00%) 507.00 ( 4.88%) Lat 50.0th-qrtle-4 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-4 8.00 ( 0.00%) 5.00 ( 37.50%) Lat 99.0th-qrtle-4 14.00 ( 0.00%) 9.00 ( 35.71%) Lat 99.9th-qrtle-4 22.00 ( 0.00%) 14.00 ( 36.36%) Lat 20.0th-qrtle-4 1070.00 ( 0.00%) 995.00 ( 7.01%) Lat 50.0th-qrtle-8 5.00 ( 0.00%) 5.00 ( 0.00%) Lat 90.0th-qrtle-8 7.00 ( 0.00%) 5.00 ( 28.57%) Lat 99.0th-qrtle-8 12.00 ( 0.00%) 11.00 ( 8.33%) Lat 99.9th-qrtle-8 19.00 ( 0.00%) 16.00 ( 15.79%) Lat 20.0th-qrtle-8 2140.00 ( 0.00%) 2140.00 ( 0.00%) Lat 50.0th-qrtle-16 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-16 7.00 ( 0.00%) 5.00 ( 28.57%) Lat 99.0th-qrtle-16 12.00 ( 0.00%) 10.00 ( 16.67%) Lat 99.9th-qrtle-16 17.00 ( 0.00%) 14.00 ( 17.65%) Lat 20.0th-qrtle-16 4296.00 ( 0.00%) 4200.00 ( 2.23%) Lat 50.0th-qrtle-32 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-32 8.00 ( 0.00%) 6.00 ( 25.00%) Lat 99.0th-qrtle-32 12.00 ( 0.00%) 10.00 ( 16.67%) Lat 99.9th-qrtle-32 17.00 ( 0.00%) 14.00 ( 17.65%) Lat 20.0th-qrtle-32 8496.00 ( 0.00%) 8528.00 ( -0.38%) Lat 50.0th-qrtle-64 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-64 8.00 ( 0.00%) 8.00 ( 0.00%) Lat 99.0th-qrtle-64 12.00 ( 0.00%) 12.00 ( 0.00%) Lat 99.9th-qrtle-64 17.00 ( 0.00%) 17.00 ( 0.00%) Lat 20.0th-qrtle-64 17120.00 ( 0.00%) 17120.00 ( 0.00%) Lat 50.0th-qrtle-128 7.00 ( 0.00%) 7.00 ( 0.00%) Lat 90.0th-qrtle-128 9.00 ( 0.00%) 9.00 ( 0.00%) Lat 99.0th-qrtle-128 13.00 ( 0.00%) 14.00 ( -7.69%) Lat 99.9th-qrtle-128 20.00 ( 0.00%) 20.00 ( 0.00%) Lat 20.0th-qrtle-128 31776.00 ( 0.00%) 30496.00 ( 4.03%) Lat 50.0th-qrtle-239 9.00 ( 0.00%) 9.00 ( 0.00%) Lat 90.0th-qrtle-239 14.00 ( 0.00%) 18.00 ( -28.57%) Lat 99.0th-qrtle-239 43.00 ( 0.00%) 56.00 ( -30.23%) Lat 99.9th-qrtle-239 106.00 ( 0.00%) 483.00 (-355.66%) Lat 20.0th-qrtle-239 30176.00 ( 0.00%) 29984.00 ( 0.64%)We can see overall latency improvement and some throughput degradation when the system gets saturated.
Also, we run schbench (old version) on an EPYC 7543 system, which has 4 NUMA nodes, and each node has 4 LLCs. Monitor the 99.0th latency:
case load baseline(std%) compare%( std%) normal 4-mthreads-1-workers 1.00 ( 6.47) +9.02 ( 4.68) normal 4-mthreads-2-workers 1.00 ( 3.25) +28.03 ( 8.76) normal 4-mthreads-4-workers 1.00 ( 6.67) -4.32 ( 2.58) normal 4-mthreads-8-workers 1.00 ( 2.38) +1.27 ( 2.41) normal 4-mthreads-16-workers 1.00 ( 5.61) -8.48 ( 4.39) normal 4-mthreads-31-workers 1.00 ( 9.31) -0.22 ( 9.77)When the LLC is underloaded, the latency improvement is observed. When the LLC gets saturated, we observe some degradation.
[..snip..]
Quoted 10 lines
kernel/sched/fair.c +10
-0+static bool valid_target_cpu (int cpu,struct task_struct *p) +{ +int nr_running, llc_weight; +unsigned long util, llc_cap; + +if (!get_llc_stats (cpu, &nr_running, &llc_weight, + &util)) +return false; + + llc_cap = llc_weight *SCHED_CAPACITY_SCALE ;
kernel/sched/fair.c static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu) · line 8829
Quoted 10 lines
kernel/sched/fair.c +10
-0+ +/* +* If this process has many threads, be careful to avoid +* task stacking on the preferred LLC, by checking the system's +* utilization and runnable tasks. Otherwise, if this +* process does not have many threads, honor the cache +* aware wakeup. +*/ +if (get_nr_threads (p) < llc_weight) +return true;
IIUC, there might be scenarios were llc might be already overloaded with threads of other process. In that case, we will be returning true for p in above condition and don't check the below conditions. Shouldn't we check the below two conditions either way?
Tested this patch with real life workload Daytrader, didn't see any regression. It spawns lot of threads and is CPU intensive. So, I think it's not impacted due to the below conditions.
Also, in schbench numbers provided by you, there is a degradation in saturated case. Is it due to the overhead in computing the preferred llc which is not being used due to below conditions?
Thanks,
Madadi Vineeth Reddy
Quoted 16 lines
kernel/sched/fair.c +16
-0+ +/* +* Check if it exceeded 25% of average utiliazation, +* or if it exceeded 33% of CPUs. This is a magic number +* that did not cause heavy cache contention on Xeon or +* Zen. +*/ +if (util *4 >= llc_cap) +return false; + +if (nr_running *3 >= llc_weight) +return false; + +return true; +} +
[..snip..]
Re: [RFC PATCH 4/5] sched: Inhibit cache aware scheduling if the preferred LLC is over aggregated
Hi Madadi,
Quoted 105 lines · Madadi Vineeth Reddy
Hi Chen Yu,
Quoted 82 lines · Chen Yu
It is found that when the process's preferred LLC gets saturated by too many threads, task contention is very frequent and causes performance regression.
Save the per LLC statistics calculated by periodic load balance. The statistics include the average utilization and the average number of runnable tasks. The task wakeup path for cache aware scheduling manipulates these statistics to inhibit cache aware scheduling to avoid performance regression. When either the average utilization of the preferred LLC has reached 25%, or the average number of runnable tasks has exceeded 1/3 of the LLC weight, the cache aware wakeup is disabled. Only when the process has more threads than the LLC weight will this restriction be enabled.
Running schbench via mmtests on a Xeon platform, which has 2 sockets, each socket has 60 Cores/120 CPUs. The DRAM interleave is enabled across NUMA nodes via BIOS, so there are 2 "LLCs" in 1 NUMA node.
compare-mmtests.pl --directory work/log --benchmark schbench --names baseline,sched_cache
baselinsched_cach
baselinesched_cacheLat 50.0th-qrtle-1 6.00 ( 0.00%) 6.00 ( 0.00%) Lat 90.0th-qrtle-1 10.00 ( 0.00%) 9.00 ( 10.00%) Lat 99.0th-qrtle-1 29.00 ( 0.00%) 13.00 ( 55.17%) Lat 99.9th-qrtle-1 35.00 ( 0.00%) 21.00 ( 40.00%) Lat 20.0th-qrtle-1 266.00 ( 0.00%) 266.00 ( 0.00%) Lat 50.0th-qrtle-2 8.00 ( 0.00%) 6.00 ( 25.00%) Lat 90.0th-qrtle-2 10.00 ( 0.00%) 10.00 ( 0.00%) Lat 99.0th-qrtle-2 19.00 ( 0.00%) 18.00 ( 5.26%) Lat 99.9th-qrtle-2 27.00 ( 0.00%) 29.00 ( -7.41%) Lat 20.0th-qrtle-2 533.00 ( 0.00%) 507.00 ( 4.88%) Lat 50.0th-qrtle-4 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-4 8.00 ( 0.00%) 5.00 ( 37.50%) Lat 99.0th-qrtle-4 14.00 ( 0.00%) 9.00 ( 35.71%) Lat 99.9th-qrtle-4 22.00 ( 0.00%) 14.00 ( 36.36%) Lat 20.0th-qrtle-4 1070.00 ( 0.00%) 995.00 ( 7.01%) Lat 50.0th-qrtle-8 5.00 ( 0.00%) 5.00 ( 0.00%) Lat 90.0th-qrtle-8 7.00 ( 0.00%) 5.00 ( 28.57%) Lat 99.0th-qrtle-8 12.00 ( 0.00%) 11.00 ( 8.33%) Lat 99.9th-qrtle-8 19.00 ( 0.00%) 16.00 ( 15.79%) Lat 20.0th-qrtle-8 2140.00 ( 0.00%) 2140.00 ( 0.00%) Lat 50.0th-qrtle-16 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-16 7.00 ( 0.00%) 5.00 ( 28.57%) Lat 99.0th-qrtle-16 12.00 ( 0.00%) 10.00 ( 16.67%) Lat 99.9th-qrtle-16 17.00 ( 0.00%) 14.00 ( 17.65%) Lat 20.0th-qrtle-16 4296.00 ( 0.00%) 4200.00 ( 2.23%) Lat 50.0th-qrtle-32 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-32 8.00 ( 0.00%) 6.00 ( 25.00%) Lat 99.0th-qrtle-32 12.00 ( 0.00%) 10.00 ( 16.67%) Lat 99.9th-qrtle-32 17.00 ( 0.00%) 14.00 ( 17.65%) Lat 20.0th-qrtle-32 8496.00 ( 0.00%) 8528.00 ( -0.38%) Lat 50.0th-qrtle-64 6.00 ( 0.00%) 5.00 ( 16.67%) Lat 90.0th-qrtle-64 8.00 ( 0.00%) 8.00 ( 0.00%) Lat 99.0th-qrtle-64 12.00 ( 0.00%) 12.00 ( 0.00%) Lat 99.9th-qrtle-64 17.00 ( 0.00%) 17.00 ( 0.00%) Lat 20.0th-qrtle-64 17120.00 ( 0.00%) 17120.00 ( 0.00%) Lat 50.0th-qrtle-128 7.00 ( 0.00%) 7.00 ( 0.00%) Lat 90.0th-qrtle-128 9.00 ( 0.00%) 9.00 ( 0.00%) Lat 99.0th-qrtle-128 13.00 ( 0.00%) 14.00 ( -7.69%) Lat 99.9th-qrtle-128 20.00 ( 0.00%) 20.00 ( 0.00%) Lat 20.0th-qrtle-128 31776.00 ( 0.00%) 30496.00 ( 4.03%) Lat 50.0th-qrtle-239 9.00 ( 0.00%) 9.00 ( 0.00%) Lat 90.0th-qrtle-239 14.00 ( 0.00%) 18.00 ( -28.57%) Lat 99.0th-qrtle-239 43.00 ( 0.00%) 56.00 ( -30.23%) Lat 99.9th-qrtle-239 106.00 ( 0.00%) 483.00 (-355.66%) Lat 20.0th-qrtle-239 30176.00 ( 0.00%) 29984.00 ( 0.64%)We can see overall latency improvement and some throughput degradation when the system gets saturated.
Also, we run schbench (old version) on an EPYC 7543 system, which has 4 NUMA nodes, and each node has 4 LLCs. Monitor the 99.0th latency:
case load baseline(std%) compare%( std%) normal 4-mthreads-1-workers 1.00 ( 6.47) +9.02 ( 4.68) normal 4-mthreads-2-workers 1.00 ( 3.25) +28.03 ( 8.76) normal 4-mthreads-4-workers 1.00 ( 6.67) -4.32 ( 2.58) normal 4-mthreads-8-workers 1.00 ( 2.38) +1.27 ( 2.41) normal 4-mthreads-16-workers 1.00 ( 5.61) -8.48 ( 4.39) normal 4-mthreads-31-workers 1.00 ( 9.31) -0.22 ( 9.77)When the LLC is underloaded, the latency improvement is observed. When the LLC gets saturated, we observe some degradation.
[..snip..]
Quoted 17 lines
kernel/sched/fair.c +17
-0+static bool valid_target_cpu (int cpu,struct task_struct *p) +{ +int nr_running, llc_weight; +unsigned long util, llc_cap; + +if (!get_llc_stats (cpu, &nr_running, &llc_weight, + &util)) +return false; + + llc_cap = llc_weight *SCHED_CAPACITY_SCALE ; + +/* +* If this process has many threads, be careful to avoid +* task stacking on the preferred LLC, by checking the system's +* utilization and runnable tasks. Otherwise, if this +* process does not have many threads, honor the cache +* aware wakeup.
Quoted 8 lines
kernel/sched/fair.c static int find_energy_efficient_cpu(struct task_struct *p, int prev_cpu) · line 8829
Quoted 3 lines
kernel/sched/fair.c +3
-0+*/ +if (get_nr_threads (p) < llc_weight) +return true;IIUC, there might be scenarios were llc might be already overloaded with threads of other process. In that case, we will be returning true for p in above condition and don't check the below conditions. Shouldn't we check the below two conditions either way?
The reason why get_nr_threads() was used is that we don't know if the following threshold is suitable for different workloads. We chose 25% and 33% because we found that it worked well for workload A, but was too low for workload B. Workload B requires the cache-aware scheduling to be enabled in any case, and the number of threads in B is smaller than the llc_weight. Therefore, we use the above check to meet the requirements of B. What you said is correct. We can remove the above checks on nr_thread and make the combination of utilization and nr_running a mandatory check, and then conduct further tuning.>
Quoted 1 line
Tested this patch with real life workload Daytrader, didn't see any regression.
Good to know the regression is gone.
Quoted 6 lines
It spawns lot of threads and is CPU intensive. So, I think it's not impacted due to the below conditions.
Also, in schbench numbers provided by you, there is a degradation in saturated case. Is it due to the overhead in computing the preferred llc which is not being used due to below conditions?
Yes, the overhead of preferred LLC calculation could be one part, and we also suspect that the degradation might be tied to the task migrations. We still observed more task migrations than the baseline, even when the system was saturated (in theory, after 25% is exceeded, we should fallback to the generic task wakeup path). We haven't dug into that yet, and we can conduct an investigation in the following days.
thanks,
Chenyu>
Quoted 22 lines
Thanks,
Madadi Vineeth ReddyQuoted 16 lines
kernel/sched/fair.c +16
-0+ +/* +* Check if it exceeded 25% of average utiliazation, +* or if it exceeded 33% of CPUs. This is a magic number +* that did not cause heavy cache contention on Xeon or +* Zen. +*/ +if (util *4 >= llc_cap) +return false; + +if (nr_running *3 >= llc_weight) +return false; + +return true; +} +[..snip..]
Re: [RFC PATCH 4/5] sched: Inhibit cache aware scheduling if the preferred LLC is over aggregated
Quoted 8 lines · Yu C
Quoted 7 lines
It spawns lot of threads and is CPU intensive. So, I think it's not impacted due to the below conditions.
Also, in schbench numbers provided by you, there is a degradation in saturated case. Is it due to the overhead in computing the preferred llc which is not being used due to below conditions?
Quoted 6 lines
Yes, the overhead of preferred LLC calculation could be one part, and we also suspect that the degradation might be tied to the task migrations. We still observed more task migrations than the baseline, even when the system was saturated (in theory, after 25% is exceeded, we should fallback to the generic task wakeup path). We haven't dug into that yet, and we can conduct an investigation in the following days.
In the saturation case it is mostly the tail latency that has regression. The preferred LLC has a tendency to have higher load than the other LLCs. Load balancer will try to move tasks out and wake balance will try to move it back to the preferred LLC. This increases the task migrations and affect tail latency.
Tim
Re: [RFC PATCH 4/5] sched: Inhibit cache aware scheduling if the preferred LLC is over aggregated
Hi Tim,
Quoted 16 lines · Tim Chen
Quoted 8 lines · Yu C
Quoted 7 lines
It spawns lot of threads and is CPU intensive. So, I think it's not impacted due to the below conditions.
Also, in schbench numbers provided by you, there is a degradation in saturated case. Is it due to the overhead in computing the preferred llc which is not being used due to below conditions?
Quoted 6 lines
Yes, the overhead of preferred LLC calculation could be one part, and we also suspect that the degradation might be tied to the task migrations. We still observed more task migrations than the baseline, even when the system was saturated (in theory, after 25% is exceeded, we should fallback to the generic task wakeup path). We haven't dug into that yet, and we can conduct an investigation in the following days.
Quoted 5 lines
In the saturation case it is mostly the tail latency that has regression. The preferred LLC has a tendency to have higher load than the other LLCs. Load balancer will try to move tasks out and wake balance will try to move it back to the preferred LLC. This increases the task migrations and affect tail latency.
Why would the task be moved back to the preferred LLC in wakeup path for the saturated case? The checks shouldn't allow it right?
Thanks,
Madadi Vineeth Reddy
Quoted 2 lines
Tim
Re: [RFC PATCH 4/5] sched: Inhibit cache aware scheduling if the preferred LLC is over aggregated
Quoted 19 lines · Madadi Vineeth Reddy
Hi Tim,
Quoted 16 lines · Tim Chen
Quoted 14 lines · Yu C
Quoted 7 lines
It spawns lot of threads and is CPU intensive. So, I think it's not impacted due to the below conditions.
Also, in schbench numbers provided by you, there is a degradation in saturated case. Is it due to the overhead in computing the preferred llc which is not being used due to below conditions?
Yes, the overhead of preferred LLC calculation could be one part, and we also suspect that the degradation might be tied to the task migrations. We still observed more task migrations than the baseline, even when the system was saturated (in theory, after 25% is exceeded, we should fallback to the generic task wakeup path). We haven't dug into that yet, and we can conduct an investigation in the following days.
Quoted 8 lines
Quoted 5 lines
In the saturation case it is mostly the tail latency that has regression. The preferred LLC has a tendency to have higher load than the other LLCs. Load balancer will try to move tasks out and wake balance will try to move it back to the preferred LLC. This increases the task migrations and affect tail latency.
Why would the task be moved back to the preferred LLC in wakeup path for the saturated case? The checks shouldn't allow it right?
The task wake ups happens very frequently in schbench and it takes a while for utilization to catch up. The utilization of the LLC is updated at the load balance time of LLC.
So once utilization falls below the utilization threshold, there is a window where the woken tasks will rush into the preferred LLC until the utilization is updated at the next load balance time.
Tim
Quoted 7 lines
Thanks,
Madadi Vineeth ReddyQuoted 2 lines
Tim
Re: [RFC PATCH 4/5] sched: Inhibit cache aware scheduling if the preferred LLC is over aggregated
In reply to Chen, Yu CQuoted 117 lines · Yu C
Hi Madadi,
Quoted 105 lines · Madadi Vineeth Reddy
Hi Chen Yu,
Quoted 82 lines · Chen Yu
It is found that when the process's preferred LLC gets saturated by too many threads, task contention is very frequent and causes performance regression.
Save the per LLC statistics calculated by periodic load balance. The statistics include the average utilization and the average number of runnable tasks. The task wakeup path for cache aware scheduling manipulates these statistics to inhibit cache aware scheduling to avoid performance regression. When either the average utilization of the preferred LLC has reached 25%, or the average number of runnable tasks has exceeded 1/3 of the LLC weight, the cache aware wakeup is disabled. Only when the process has more threads than the LLC weight will this restriction be enabled.
Running schbench via mmtests on a Xeon platform, which has 2 sockets, each socket has 60 Cores/120 CPUs. The DRAM interleave is enabled across NUMA nodes via BIOS, so there are 2 "LLCs" in 1 NUMA node.
compare-mmtests.pl --directory work/log --benchmark schbench --names baseline,sched_cache baselin
sched_cachbaselinesched_cache
Lat 50.0th-qrtle-1 6.00 ( 0.00%) 6.00 ( 0.00%)
Lat 90.0th-qrtle-1 10.00 ( 0.00%) 9.00 ( 10.00%)
Lat 99.0th-qrtle-1 29.00 ( 0.00%) 13.00 ( 55.17%)
Lat 99.9th-qrtle-1 35.00 ( 0.00%) 21.00 ( 40.00%)
Lat 20.0th-qrtle-1 266.00 ( 0.00%) 266.00 ( 0.00%)
Lat 50.0th-qrtle-2 8.00 ( 0.00%) 6.00 ( 25.00%)
Lat 90.0th-qrtle-2 10.00 ( 0.00%) 10.00 ( 0.00%)
Lat 99.0th-qrtle-2 19.00 ( 0.00%) 18.00 ( 5.26%)
Lat 99.9th-qrtle-2 27.00 ( 0.00%) 29.00 ( -7.41%)
Lat 20.0th-qrtle-2 533.00 ( 0.00%) 507.00 ( 4.88%)
Lat 50.0th-qrtle-4 6.00 ( 0.00%) 5.00 ( 16.67%)
Lat 90.0th-qrtle-4 8.00 ( 0.00%) 5.00 ( 37.50%)
Lat 99.0th-qrtle-4 14.00 ( 0.00%) 9.00 ( 35.71%)
Lat 99.9th-qrtle-4 22.00 ( 0.00%) 14.00 ( 36.36%)
Lat 20.0th-qrtle-4 1070.00 ( 0.00%) 995.00 ( 7.01%)
Lat 50.0th-qrtle-8 5.00 ( 0.00%) 5.00 ( 0.00%)
Lat 90.0th-qrtle-8 7.00 ( 0.00%) 5.00 ( 28.57%)
Lat 99.0th-qrtle-8 12.00 ( 0.00%) 11.00 ( 8.33%)
Lat 99.9th-qrtle-8 19.00 ( 0.00%) 16.00 ( 15.79%)
Lat 20.0th-qrtle-8 2140.00 ( 0.00%) 2140.00 ( 0.00%)
Lat 50.0th-qrtle-16 6.00 ( 0.00%) 5.00 ( 16.67%)
Lat 90.0th-qrtle-16 7.00 ( 0.00%) 5.00 ( 28.57%)
Lat 99.0th-qrtle-16 12.00 ( 0.00%) 10.00 ( 16.67%)
Lat 99.9th-qrtle-16 17.00 ( 0.00%) 14.00 ( 17.65%)
Lat 20.0th-qrtle-16 4296.00 ( 0.00%) 4200.00 ( 2.23%)
Lat 50.0th-qrtle-32 6.00 ( 0.00%) 5.00 ( 16.67%)
Lat 90.0th-qrtle-32 8.00 ( 0.00%) 6.00 ( 25.00%)
Lat 99.0th-qrtle-32 12.00 ( 0.00%) 10.00 ( 16.67%)
Lat 99.9th-qrtle-32 17.00 ( 0.00%) 14.00 ( 17.65%)
Lat 20.0th-qrtle-32 8496.00 ( 0.00%) 8528.00 ( -0.38%)
Lat 50.0th-qrtle-64 6.00 ( 0.00%) 5.00 ( 16.67%)
Lat 90.0th-qrtle-64 8.00 ( 0.00%) 8.00 ( 0.00%)
Lat 99.0th-qrtle-64 12.00 ( 0.00%) 12.00 ( 0.00%)
Lat 99.9th-qrtle-64 17.00 ( 0.00%) 17.00 ( 0.00%)
Lat 20.0th-qrtle-64 17120.00 ( 0.00%) 17120.00 ( 0.00%)
Lat 50.0th-qrtle-128 7.00 ( 0.00%) 7.00 ( 0.00%)
Lat 90.0th-qrtle-128 9.00 ( 0.00%) 9.00 ( 0.00%)
Lat 99.0th-qrtle-128 13.00 ( 0.00%) 14.00 ( -7.69%)
Lat 99.9th-qrtle-128 20.00 ( 0.00%) 20.00 ( 0.00%)
Lat 20.0th-qrtle-128 31776.00 ( 0.00%) 30496.00 ( 4.03%)
Lat 50.0th-qrtle-239 9.00 ( 0.00%) 9.00 ( 0.00%)
Lat 90.0th-qrtle-239 14.00 ( 0.00%) 18.00 ( -28.57%)
Lat 99.0th-qrtle-239 43.00 ( 0.00%) 56.00 ( -30.23%)
Lat 99.9th-qrtle-239 106.00 ( 0.00%) 483.00 (-355.66%)
Lat 20.0th-qrtle-239 30176.00 ( 0.00%) 29984.00 ( 0.64%)We can see overall latency improvement and some throughput degradation when the system gets saturated.
Also, we run schbench (old version) on an EPYC 7543 system, which has 4 NUMA nodes, and each node has 4 LLCs. Monitor the 99.0th latency:
case load baseline(std%) compare%( std%) normal 4-mthreads-1-workers 1.00 ( 6.47) +9.02 ( 4.68) normal 4-mthreads-2-workers 1.00 ( 3.25) +28.03 ( 8.76) normal 4-mthreads-4-workers 1.00 ( 6.67) -4.32 ( 2.58) normal 4-mthreads-8-workers 1.00 ( 2.38) +1.27 ( 2.41) normal 4-mthreads-16-workers 1.00 ( 5.61) -8.48 ( 4.39) normal 4-mthreads-31-workers 1.00 ( 9.31) -0.22 ( 9.77)
When the LLC is underloaded, the latency improvement is observed. When the LLC gets saturated, we observe some degradation.
[..snip..]
Quoted 17 lines
kernel/sched/fair.c +17
-0+static bool valid_target_cpu (int cpu,struct task_struct *p) +{ + int nr_running, llc_weight; + unsigned long util, llc_cap; + + if (!get_llc_stats(cpu, &nr_running, &llc_weight, + &util)) + return false; + + llc_cap = llc_weight * SCHED_CAPACITY_SCALE; + + /* +* If this process has many threads, be careful to avoid +* task stacking on the preferred LLC, by checking the system's +* utilization and runnable tasks. Otherwise, if this +* process does not have many threads, honor the cache +* aware wakeup. Quoted 8 lines
Quoted 3 lines
+ */
+ if (get_nr_threads(p)<llc_weight)
+ return true;IIUC, there might be scenarios were llc might be already overloaded with threads of other process. In that case, we will be returning true for p in above condition and don't check the below conditions. Shouldn't we check the below two conditions either way?
Quoted 1 line
The reason why
get_nr_threads()was used is that we don't know if the following threshold is suitable for different workloads. We chose 25% and 33% because we found that it worked well for workload A, but was too low for workload B. Workload B requires the cache-aware scheduling to be enabled in any case, and the number of threads in B is smaller than thellc_weight. Therefore, we use the above check to meet the requirements of B. What you said is correct. We can remove the above checks onnr_threadand make the combination of utilization andnr_runninga mandatory check, and then conduct further tuning.>
Thanks Chen. It's always tricky to make all workloads happy. As long as we're not regressing too much on the others, it should be fine I guess given the overall impact is positive.
JFYI, In Power10, LLC is at a small core level containing 4 threads. So, nr_running on LLC can't be more than 1 for cache aware scheduling to work.
Quoted 4 lines
Quoted 1 line
Tested this patch with real life workload Daytrader, didn't see any regression.
Good to know the regression is gone.
Quoted 8 lines
Quoted 6 lines
It spawns lot of threads and is CPU intensive. So, I think it's not impacted due to the below conditions.
Also, in schbench numbers provided by you, there is a degradation in saturated case. Is it due to the overhead in computing the preferred llc which is not being used due to below conditions?
Yes, the overhead of preferred LLC calculation could be one part, and we also suspect that the degradation might be tied to the task migrations. We still observed more task migrations than the baseline, even when the system was saturated (in theory, after 25% is exceeded, we should fallback to the generic task wakeup path). We haven't dug into that yet, and we can conduct an investigation in the following days.
Interesting. I will also try to look into these extra migrations.
Thanks,
Madadi Vineeth Reddy
Quoted 25 lines
thanks,
Chenyu>Quoted 22 lines
Thanks,
Madadi Vineeth ReddyQuoted 16 lines
kernel/sched/fair.c +16
-0+ + /* + * Check if it exceeded 25% of average utiliazation, + * or if it exceeded 33% of CPUs. This is a magic number + * that did not cause heavy cache contention on Xeon or + * Zen. + */ + if (util * 4 >= llc_cap) + return false; + + if (nr_running * 3 >= llc_weight) + return false; + + return true; +} +[..snip..]
RFC PATCH 5/5 sched: Add ftrace to track task migration and load balance within and across LLC
In reply to Chen Yu[Not for upstream]
Introduce these ftrace events for debugging purposes. The task migration activity is an important indicator to infer the performance regression.
Use the following bpftrace script to capture the task migrations:
tracepoint:sched:sched_attach_task
{
$src_cpu = args->src_cpu;
$dst_cpu = args->dst_cpu;
$src_llc = args->src_llc;
$dst_llc = args->dst_llc;
$idle = args->idle;
if ($src_llc == $dst_llc) {
@lb_mig_1llc[$idle] = count();
} else {
@lb_mig_2llc[$idle] = count(); }
}
tracepoint:sched:sched_select_task_rq
{
$new_cpu = args->new_cpu;
$old_cpu = args->old_cpu;
$new_llc = args->new_llc;
$old_llc = args->old_llc;
if ($new_cpu != $old_cpu) {
if ($new_llc == $old_llc) {
@wake_mig_1llc[$new_llc] = count();
} else {
@wake_mig_2llc = count();
} }
}
interval:s:10
{
time("\n%H:%M:%S scheduler statistics: \n");
print(@lb_mig_1llc);
clear(@lb_mig_1llc);
print(@lb_mig_2llc);
clear(@lb_mig_2llc);
print(@wake_mig_1llc);
clear(@wake_mig_1llc);
print(@wake_mig_2llc);
clear(@wake_mig_2llc);
}--- include/trace/events/sched.h | 51 ++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 24 ++++++++++++----- 2 files changed, 69 insertions(+), 6 deletions(-)
include/trace/events/sched.h +51 -0
@@ -10,6 +10,57 @@#include <linux/tracepoint.h> #include <linux/binfmts.h> +TRACE_EVENT (sched_attach_task, + + TP_PROTO(int src_cpu, int dst_cpu, int src_llc, int dst_llc, int idle), + +TP_ARGS (src_cpu, dst_cpu, src_llc, dst_llc, idle), + +TP_STRUCT__entry ( + __field( int, src_cpu ) + __field( int, dst_cpu ) + __field( int, src_llc ) + __field( int, dst_llc ) +__field ( int, idle ) + ), + +TP_fast_assign ( + __entry->src_cpu = src_cpu; + __entry->dst_cpu = dst_cpu; + __entry->src_llc = src_llc; + __entry->dst_llc = dst_llc; + __entry->idle = idle; + ), + +TP_printk ("src_cpu=%d dst_cpu=%d src_llc=%d dst_llc=%d idle=%d" , + __entry->src_cpu, __entry->dst_cpu, __entry->src_llc, + __entry->dst_llc, __entry->idle) +); + +TRACE_EVENT (sched_select_task_rq, + + TP_PROTO(int new_cpu, int old_cpu, int new_llc, int old_llc), + +TP_ARGS (new_cpu, old_cpu, new_llc, old_llc), + +TP_STRUCT__entry ( + __field( int, new_cpu ) + __field( int, old_cpu ) + __field( int, new_llc ) +__field ( int, old_llc ) + ), + +TP_fast_assign ( + __entry->new_cpu = new_cpu; + __entry->old_cpu = old_cpu; + __entry->new_llc = new_llc; + __entry->old_llc = old_llc; + ), + +TP_printk ("new_cpu=%d old_cpu=%d new_llc=%d old_llc=%d" , + __entry->new_cpu, __entry->old_cpu, __entry->new_llc, __entry->old_llc) +); +/* * Tracepoint for calling kthread_stop, performed to end a kthread: */
kernel/sched/fair.c +18 -6
@@ -8902,7 +8902,7 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags)int sync = (wake_flags &WF_SYNC ) && !(current->flags &PF_EXITING );struct sched_domain *tmp, *sd =NULL ;int cpu =smp_processor_id (); -int new_cpu = prev_cpu; +int new_cpu = prev_cpu, orig_prev_cpu = prev_cpu;int want_affine =0 ;/* SD_flags and WF_flags share the first nibble */ int sd_flag = wake_flags &0xF ;
@@ -8965,6 +8965,10 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int wake_flags) new_cpu =select_idle_sibling (p, prev_cpu, new_cpu); } +trace_sched_select_task_rq (new_cpu, orig_prev_cpu, +per_cpu (sd_llc_id, new_cpu), +per_cpu (sd_llc_id, orig_prev_cpu)); +return new_cpu; }
@@ -10026,11 +10030,17 @@ static int detach_tasks(struct lb_env *env)/* * attach_task() -- attach the task detached by detach_task() to its new rq. */ -static void attach_task (struct rq *rq,struct task_struct *p) +static void attach_task (struct rq *rq,struct task_struct *p,struct lb_env *env) {lockdep_assert_rq_held (rq);WARN_ON_ONCE (task_rq (p) != rq); + +if (env) +trace_sched_attach_task (env->src_cpu, env->dst_cpu, +per_cpu (sd_llc_id, env->src_cpu), +per_cpu (sd_llc_id, env->dst_cpu), + env->idle);activate_task (rq, p,ENQUEUE_NOCLOCK );wakeup_preempt (rq, p,0 ); }
@@ -10039,13 +10049,13 @@ static void attach_task(struct rq *rq, struct task_struct *p)* attach_one_task() -- attaches the task returned from detach_one_task() to * its new rq. */ -static void attach_one_task (struct rq *rq,struct task_struct *p) +static void attach_one_task (struct rq *rq,struct task_struct *p,struct lb_env *env) {struct rq_flags rf;rq_lock (rq, &rf);update_rq_clock (rq); -attach_task (rq, p); +attach_task (rq, p, env);rq_unlock (rq, &rf); }
@@ -10066,7 +10076,7 @@ static void attach_tasks(struct lb_env *env) p = list_first_entry(tasks, struct task_struct, se.group_node);list_del_init (&p->se.group_node); -attach_task (env->dst_rq, p); +attach_task (env->dst_rq, p, env); }rq_unlock (env->dst_rq, &rf);
@@ -12457,6 +12467,7 @@ static int active_load_balance_cpu_stop(void *data)struct sched_domain *sd;struct task_struct *p =NULL ;struct rq_flags rf; +struct lb_env env_tmp;rq_lock_irq (busiest_rq, &rf);/*
@@ -12512,6 +12523,7 @@ static int active_load_balance_cpu_stop(void *data) }else {schedstat_inc (sd->alb_failed); } +memcpy (&env_tmp, &env,sizeof (env)); }rcu_read_unlock (); out_unlock:
@@ -12519,7 +12531,7 @@ static int active_load_balance_cpu_stop(void *data)rq_unlock (busiest_rq, &rf);if (p) -attach_one_task (target_rq, p); +attach_one_task (target_rq, p, sd ? &env_tmp :NULL );local_irq_enable ();
Signature · 1 line
2.25.1
K Prateek Nayak
reviewerHello Chenyu,
Quoted 23 lines · Chen Yu
This is a respin of the cache-aware scheduling proposed by Peter[1]. In this patch set, some known issues in [1] were addressed, and the performance regression was investigated and mitigated.
Cache-aware scheduling aims to aggregate tasks with potential shared resources into the same cache domain. This approach enhances cache locality, thereby optimizing system performance by reducing cache misses and improving data access efficiency.
In the current implementation, threads within the same process are considered as entities that potentially share resources. Cache-aware scheduling monitors the CPU occupancy of each cache domain for every process. Based on this monitoring, it endeavors to migrate threads within a given process to its cache-hot domains, with the goal of maximizing cache locality.
Patch 1 constitutes the fundamental cache-aware scheduling. It is the same patch as [1]. Patch 2 comprises a series of fixes for Patch 1, including compiling warnings and functional fixes.
Patch 3 fixes performance degradation that arise from excessive task migrations within the preferred LLC domain.
Patch 4 further alleviates performance regressions when the preferred LLC becomes saturated. Patch 5 introduces ftrace events, which is used to track task migrations triggered by wakeup and load balancer. This addition facilitate performance regression analysis.
Quoted 3 lines
The patch set is applied on top of v6.14 sched/core,
commit4ba7518327c6("sched/debug: Print the local group'sasym_prefer_cpu")
Thank you for working on this! I have been a bit preoccupied but I promise to look into the regressions I've reported below sometime this week and report back soon on what seems to make them unhappy.
tl;dr
o Most regressions aren't as severe as v1 thanks to all the work from you and Abel.
o I too see schbench regress in fully loaded cases but the old schbench tail latencies improve when #threads < #CPUs in LLC
o There is a consistent regression in tbench - what I presume is happening there is all threads of "tbench_srv" share an mm and and all the tbench clients share an mm but for best performance, the wakeups between client and server must be local (same core / same LLC) but either the cost of additional search build up or the clients get co-located as one set of entities and the servers get colocated as another set of entities leading to mostly remote wakeups.
Not too sure if netperf has similar architecture as tbench but that too sees a regression.
o Longer running benchmarks see a regression. Still not sure if this is because of additional search or something else.
I'll leave the full results below:
o Machine details
- 3rd Generation EPYC System
- 2 sockets each with 64C/128T
- NPS1 (Each socket is a NUMA node)
- C2 Disabled (POLL and C1(MWAIT) remained enabled)
o Benchmark results
==================================================================
Test : hackbench
Units : Normalized time in seconds
Interpretation: Lower is better
Statistic : AMean
==================================================================
Case: tip[pct imp](CV) cache_aware_lb[pct imp](CV)
1-groups 1.00 [ -0.00]( 9.02) 1.03 [ -3.38](11.44) 2-groups 1.00 [ -0.00]( 6.86) 0.98 [ 2.20]( 6.61) 4-groups 1.00 [ -0.00]( 2.73) 1.00 [ 0.42]( 4.00) 8-groups 1.00 [ -0.00]( 1.21) 1.04 [ -4.00]( 5.59) 16-groups 1.00 [ -0.00]( 0.97) 1.01 [ -0.52]( 2.12)
==================================================================
Test : tbench
Units : Normalized throughput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: tip[pct imp](CV) cache_aware_lb[pct imp](CV)
1 1.00 [ 0.00]( 0.67) 0.96 [ -3.95]( 0.55)
2 1.00 [ 0.00]( 0.85) 0.98 [ -1.69]( 0.65)
4 1.00 [ 0.00]( 0.52) 0.96 [ -3.68]( 0.09)
8 1.00 [ 0.00]( 0.92) 0.96 [ -4.06]( 0.43)
16 1.00 [ 0.00]( 1.01) 0.95 [ -5.19]( 1.65)
32 1.00 [ 0.00]( 1.35) 0.95 [ -4.79]( 0.29)
64 1.00 [ 0.00]( 1.22) 0.94 [ -6.49]( 1.46)
128 1.00 [ 0.00]( 2.39) 0.92 [ -7.61]( 1.41)
256 1.00 [ 0.00]( 1.83) 0.92 [ -8.24]( 0.35)
512 1.00 [ 0.00]( 0.17) 0.93 [ -7.08]( 0.22)
1024 1.00 [ 0.00]( 0.31) 0.91 [ -8.57]( 0.29) ==================================================================
Test : stream-10
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: tip[pct imp](CV) cache_aware_lb[pct imp](CV)
Copy 1.00 [ 0.00]( 8.24) 1.03 [ 2.66]( 6.15)
Scale 1.00 [ 0.00]( 5.62) 0.99 [ -1.43]( 6.32)
Add 1.00 [ 0.00]( 6.18) 0.97 [ -3.12]( 5.70)
Triad 1.00 [ 0.00]( 5.29) 1.01 [ 1.31]( 3.82) ==================================================================
Test : stream-100
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: tip[pct imp](CV) cache_aware_lb[pct imp](CV)
Copy 1.00 [ 0.00]( 2.92) 0.99 [ -1.47]( 5.02)
Scale 1.00 [ 0.00]( 4.80) 0.98 [ -2.08]( 5.53)
Add 1.00 [ 0.00]( 4.35) 0.98 [ -1.85]( 4.26)
Triad 1.00 [ 0.00]( 2.30) 0.99 [ -0.84]( 1.83) ==================================================================
Test : netperf
Units : Normalized Througput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: tip[pct imp](CV) cache_aware_lb[pct imp](CV)
1-clients 1.00 [ 0.00]( 0.17) 0.97 [ -2.55]( 0.50) 2-clients 1.00 [ 0.00]( 0.77) 0.97 [ -2.52]( 0.20) 4-clients 1.00 [ 0.00]( 0.93) 0.97 [ -3.30]( 0.54) 8-clients 1.00 [ 0.00]( 0.87) 0.96 [ -3.98]( 1.19) 16-clients 1.00 [ 0.00]( 1.15) 0.96 [ -4.16]( 1.06) 32-clients 1.00 [ 0.00]( 1.00) 0.95 [ -5.47]( 0.96) 64-clients 1.00 [ 0.00]( 1.37) 0.94 [ -5.75]( 1.64) 128-clients 1.00 [ 0.00]( 0.99) 0.92 [ -8.50]( 1.49) 256-clients 1.00 [ 0.00]( 3.23) 0.90 [-10.22]( 2.86) 512-clients 1.00 [ 0.00](58.43) 0.90 [-10.28](47.59)
==================================================================
Test : schbench
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) cache_aware_lb[pct imp](CV)
1 1.00 [ -0.00]( 5.59) 0.55 [ 45.00](11.17)
2 1.00 [ -0.00](14.29) 0.52 [ 47.62]( 7.53)
4 1.00 [ -0.00]( 1.24) 0.57 [ 42.55]( 5.73)
8 1.00 [ -0.00](11.16) 1.06 [ -6.12]( 2.92)
16 1.00 [ -0.00]( 6.81) 1.12 [-12.28](11.09)
32 1.00 [ -0.00]( 6.99) 1.05 [ -5.26](12.48)
64 1.00 [ -0.00]( 6.00) 0.96 [ 4.21](18.31)
128 1.00 [ -0.00]( 3.26) 1.63 [-62.84](36.71)
256 1.00 [ -0.00](19.29) 0.97 [ 3.25]( 4.94)
512 1.00 [ -0.00]( 1.48) 1.05 [ -4.71]( 5.11) ==================================================================
Test : new-schbench-requests-per-second
Units : Normalized Requests per second
Interpretation: Higher is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) cache_aware_lb[pct imp](CV)
1 1.00 [ 0.00]( 0.00) 0.95 [ -4.99]( 0.48)
2 1.00 [ 0.00]( 0.26) 0.96 [ -3.82]( 0.55)
4 1.00 [ 0.00]( 0.15) 0.95 [ -4.96]( 0.27)
8 1.00 [ 0.00]( 0.15) 0.99 [ -0.58]( 0.00)
16 1.00 [ 0.00]( 0.00) 1.00 [ -0.29]( 0.15)
32 1.00 [ 0.00]( 4.88) 1.04 [ 4.27]( 2.42)
64 1.00 [ 0.00]( 5.57) 0.87 [-13.10](11.51)
128 1.00 [ 0.00]( 0.34) 0.97 [ -3.13]( 0.58)
256 1.00 [ 0.00]( 1.95) 1.02 [ 1.83]( 0.15)
512 1.00 [ 0.00]( 0.44) 1.00 [ 0.48]( 0.12) ==================================================================
Test : new-schbench-wakeup-latency
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) cache_aware_lb[pct imp](CV)
1 1.00 [ -0.00]( 4.19) 1.00 [ -0.00](14.91)
2 1.00 [ -0.00]( 3.78) 0.93 [ 7.14]( 0.00)
4 1.00 [ -0.00]( 8.91) 0.80 [ 20.00]( 4.43)
8 1.00 [ -0.00]( 7.45) 1.00 [ -0.00]( 7.45)
16 1.00 [ -0.00]( 4.08) 1.00 [ -0.00](10.79)
32 1.00 [ -0.00](16.90) 0.93 [ 6.67](10.00)
64 1.00 [ -0.00]( 9.11) 1.12 [-12.50]( 0.00)
128 1.00 [ -0.00]( 7.05) 2.43 [-142.86](24.47)
256 1.00 [ -0.00]( 4.32) 1.02 [ -2.34]( 1.20)
512 1.00 [ -0.00]( 0.35) 1.01 [ -0.77]( 0.40) ==================================================================
Test : new-schbench-request-latency
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) cache_aware_lb[pct imp](CV)
1 1.00 [ -0.00]( 0.78) 1.16 [-15.70]( 2.14)
2 1.00 [ -0.00]( 0.81) 1.13 [-13.11]( 0.62)
4 1.00 [ -0.00]( 0.24) 1.26 [-26.11](16.43)
8 1.00 [ -0.00]( 1.30) 1.03 [ -3.46]( 0.81)
16 1.00 [ -0.00]( 1.11) 1.02 [ -2.12]( 1.85)
32 1.00 [ -0.00]( 5.94) 0.96 [ 4.05]( 4.48)
64 1.00 [ -0.00]( 6.27) 1.06 [ -6.01]( 6.67)
128 1.00 [ -0.00]( 0.21) 1.12 [-12.31]( 2.61)
256 1.00 [ -0.00](13.73) 1.06 [ -6.30]( 3.37)
512 1.00 [ -0.00]( 0.95) 1.05 [ -4.85]( 0.61) ==================================================================
Test : Various longer running benchmarks
Units : %diff in throughput reported
Interpretation: Higher is better
Statistic : Median
==================================================================
Benchmarks: %diff ycsb-cassandra -1.21% ycsb-mongodb -0.69%
deathstarbench-1x -7.40% deathstarbench-2x -3.80% deathstarbench-3x -3.99% deathstarbench-6x -3.02%
hammerdb+mysql 16VU -2.59%
hammerdb+mysql 64VU -1.05%
Also, could you fold the below diff into your Patch2:
kernel/sched/fair.c +2 -2
@@ -7694,8 +7694,6 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, boolint i, cpu, idle_cpu =-1 , nr =INT_MAX ;struct sched_domain_shared *sd_share; -cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); -if (sched_feat (SIS_UTIL )) { sd_share =rcu_dereference (per_cpu (sd_llc_shared, target));if (sd_share) {
@@ -7707,6 +7705,8 @@ static int select_idle_cpu(struct task_struct *p, struct sched_domain *sd, bool } } +cpumask_and (cpus,sched_domain_span (sd), p->cpus_ptr); +if (static_branch_unlikely (&sched_cluster_active)) {struct sched_group *sg = sd->groups;
---
If the SIS_UTIL cut off hits, that result of the cpumask_and() is of no use. To save some additional cycles, especially in cases where we target the LLC frequently and the search bails out because the LLC is busy, this overhead can be easily avoided. Since select_idle_cpu() can now be called twice per wakeup, this overhead can be visible in benchmarks like hackbench.
Signature · 2 lines
Thanks and Regards, Prateek
Quoted 19 lines
schbench was tested on EMR and Zen3 Milan. An improvement in tail latency was observed when the LLC was underloaded; however, some regressions were still evident when the LLC was saturated. Additionally, the load balance should be adjusted to further address these regressions.
[1] https://lore.kernel.org/all/[email protected]/
Chen Yu (4):
sched: Several fixes for cache aware scheduling
sched: Avoid task migration within its preferred LLC
sched: Inhibit cache aware scheduling if the preferred LLC is over
aggregated
sched: Add ftrace to track task migration and load balance within and
across LLCPeter Zijlstra (1):
sched: Cache aware load-balancing
Hi Prateek,
Quoted 26 lines · K Prateek Nayak
Hello Chenyu,
Quoted 23 lines · Chen Yu
This is a respin of the cache-aware scheduling proposed by Peter[1]. In this patch set, some known issues in [1] were addressed, and the performance regression was investigated and mitigated.
Cache-aware scheduling aims to aggregate tasks with potential shared resources into the same cache domain. This approach enhances cache locality, thereby optimizing system performance by reducing cache misses and improving data access efficiency.
In the current implementation, threads within the same process are considered as entities that potentially share resources. Cache-aware scheduling monitors the CPU occupancy of each cache domain for every process. Based on this monitoring, it endeavors to migrate threads within a given process to its cache-hot domains, with the goal of maximizing cache locality.
Patch 1 constitutes the fundamental cache-aware scheduling. It is the same patch as [1]. Patch 2 comprises a series of fixes for Patch 1, including compiling warnings and functional fixes.
Patch 3 fixes performance degradation that arise from excessive task migrations within the preferred LLC domain.
Patch 4 further alleviates performance regressions when the preferred LLC becomes saturated. Patch 5 introduces ftrace events, which is used to track task migrations triggered by wakeup and load balancer. This addition facilitate performance regression analysis.
Quoted 8 lines
Quoted 3 lines
The patch set is applied on top of v6.14 sched/core,
commit4ba7518327c6("sched/debug: Print the local group'sasym_prefer_cpu")Thank you for working on this! I have been a bit preoccupied but I promise to look into the regressions I've reported below sometime this week and report back soon on what seems to make them unhappy.
Thanks for your time on this testings.
Quoted 8 lines
tl;dr
o Most regressions aren't as severe as v1 thanks to all the work from you and Abel.
o I too see schbench regress in fully loaded cases but the old schbench tail latencies improve when #threads < #CPUs in LLC
Quoted 8 lines
o There is a consistent regression in tbench - what I presume is happening there is all threads of "
tbench_srv" share an mm and and all the tbench clients share an mm but for best performance, the wakeups between client and server must be local (same core / same LLC) but either the cost of additional search build up or the clients get co-located as one set of entities and the servers get colocated as another set of entities leading to mostly remote wakeups.
This is a good point. If A and B are both multi-threaded processes, and A interacts with B frequently, we should not only consider aggregating the threads within A and B, but also placing A and B together. I'm not sure if WF_SYNC is carried along and takes effect during the tbench socket wakeup process. I'll also try tbench/netperf testings.
Quoted 136 lines
Not too sure if netperf has similar architecture as tbench but that too sees a regression.
o Longer running benchmarks see a regression. Still not sure if this is because of additional search or something else.
I'll leave the full results below:
o Machine details
- 3rd Generation EPYC System
- 2 sockets each with 64C/128T
- NPS1 (Each socket is a NUMA node)- C2 Disabled (POLL and C1(MWAIT) remained enabled)
o Benchmark results
==================================================================
Test : hackbench
Units : Normalized time in seconds
Interpretation: Lower is better
Statistic : AMean
==================================================================
Case: tip[pct imp](CV) cache_aware_lb[pct imp](CV) 1-groups 1.00 [ -0.00]( 9.02) 1.03 [ -3.38](11.44) 2-groups 1.00 [ -0.00]( 6.86) 0.98 [ 2.20]( 6.61) 4-groups 1.00 [ -0.00]( 2.73) 1.00 [ 0.42]( 4.00) 8-groups 1.00 [ -0.00]( 1.21) 1.04 [ -4.00]( 5.59) 16-groups 1.00 [ -0.00]( 0.97) 1.01 [ -0.52]( 2.12)==================================================================
Test : tbench
Units : Normalized throughput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: tip[pct imp](CV) cache_aware_lb[pct imp](CV) 1 1.00 [ 0.00]( 0.67) 0.96 [ -3.95]( 0.55) 2 1.00 [ 0.00]( 0.85) 0.98 [ -1.69]( 0.65) 4 1.00 [ 0.00]( 0.52) 0.96 [ -3.68]( 0.09) 8 1.00 [ 0.00]( 0.92) 0.96 [ -4.06]( 0.43) 16 1.00 [ 0.00]( 1.01) 0.95 [ -5.19]( 1.65) 32 1.00 [ 0.00]( 1.35) 0.95 [ -4.79]( 0.29) 64 1.00 [ 0.00]( 1.22) 0.94 [ -6.49]( 1.46) 128 1.00 [ 0.00]( 2.39) 0.92 [ -7.61]( 1.41) 256 1.00 [ 0.00]( 1.83) 0.92 [ -8.24]( 0.35) 512 1.00 [ 0.00]( 0.17) 0.93 [ -7.08]( 0.22) 1024 1.00 [ 0.00]( 0.31) 0.91 [ -8.57]( 0.29)==================================================================
Test : stream-10
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: tip[pct imp](CV) cache_aware_lb[pct imp](CV) Copy 1.00 [ 0.00]( 8.24) 1.03 [ 2.66]( 6.15) Scale 1.00 [ 0.00]( 5.62) 0.99 [ -1.43]( 6.32) Add 1.00 [ 0.00]( 6.18) 0.97 [ -3.12]( 5.70) Triad 1.00 [ 0.00]( 5.29) 1.01 [ 1.31]( 3.82)==================================================================
Test : stream-100
Units : Normalized Bandwidth, MB/s
Interpretation: Higher is better
Statistic : HMean
==================================================================
Test: tip[pct imp](CV) cache_aware_lb[pct imp](CV) Copy 1.00 [ 0.00]( 2.92) 0.99 [ -1.47]( 5.02) Scale 1.00 [ 0.00]( 4.80) 0.98 [ -2.08]( 5.53) Add 1.00 [ 0.00]( 4.35) 0.98 [ -1.85]( 4.26) Triad 1.00 [ 0.00]( 2.30) 0.99 [ -0.84]( 1.83)==================================================================
Test : netperf
Units : Normalized Througput
Interpretation: Higher is better
Statistic : AMean
==================================================================
Clients: tip[pct imp](CV) cache_aware_lb[pct imp](CV) 1-clients 1.00 [ 0.00]( 0.17) 0.97 [ -2.55]( 0.50) 2-clients 1.00 [ 0.00]( 0.77) 0.97 [ -2.52]( 0.20) 4-clients 1.00 [ 0.00]( 0.93) 0.97 [ -3.30]( 0.54) 8-clients 1.00 [ 0.00]( 0.87) 0.96 [ -3.98]( 1.19) 16-clients 1.00 [ 0.00]( 1.15) 0.96 [ -4.16]( 1.06) 32-clients 1.00 [ 0.00]( 1.00) 0.95 [ -5.47]( 0.96) 64-clients 1.00 [ 0.00]( 1.37) 0.94 [ -5.75]( 1.64) 128-clients 1.00 [ 0.00]( 0.99) 0.92 [ -8.50]( 1.49) 256-clients 1.00 [ 0.00]( 3.23) 0.90 [-10.22]( 2.86) 512-clients 1.00 [ 0.00](58.43) 0.90 [-10.28](47.59)==================================================================
Test : schbench
Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) cache_aware_lb[pct imp](CV) 1 1.00 [ -0.00]( 5.59) 0.55 [ 45.00](11.17) 2 1.00 [ -0.00](14.29) 0.52 [ 47.62]( 7.53) 4 1.00 [ -0.00]( 1.24) 0.57 [ 42.55]( 5.73) 8 1.00 [ -0.00](11.16) 1.06 [ -6.12]( 2.92) 16 1.00 [ -0.00]( 6.81) 1.12 [-12.28](11.09) 32 1.00 [ -0.00]( 6.99) 1.05 [ -5.26](12.48) 64 1.00 [ -0.00]( 6.00) 0.96 [ 4.21](18.31) 128 1.00 [ -0.00]( 3.26) 1.63 [-62.84](36.71) 256 1.00 [ -0.00](19.29) 0.97 [ 3.25]( 4.94) 512 1.00 [ -0.00]( 1.48) 1.05 [ -4.71]( 5.11)==================================================================
Test : new-schbench-requests-per-second Units : Normalized Requests per second
Interpretation: Higher is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) cache_aware_lb[pct imp](CV) 1 1.00 [ 0.00]( 0.00) 0.95 [ -4.99]( 0.48) 2 1.00 [ 0.00]( 0.26) 0.96 [ -3.82]( 0.55) 4 1.00 [ 0.00]( 0.15) 0.95 [ -4.96]( 0.27) 8 1.00 [ 0.00]( 0.15) 0.99 [ -0.58]( 0.00) 16 1.00 [ 0.00]( 0.00) 1.00 [ -0.29]( 0.15) 32 1.00 [ 0.00]( 4.88) 1.04 [ 4.27]( 2.42) 64 1.00 [ 0.00]( 5.57) 0.87 [-13.10](11.51) 128 1.00 [ 0.00]( 0.34) 0.97 [ -3.13]( 0.58) 256 1.00 [ 0.00]( 1.95) 1.02 [ 1.83]( 0.15) 512 1.00 [ 0.00]( 0.44) 1.00 [ 0.48]( 0.12)
Quoted 15 lines
==================================================================
Test : new-schbench-wakeup-latency Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) cache_aware_lb[pct imp](CV) 1 1.00 [ -0.00]( 4.19) 1.00 [ -0.00](14.91) 2 1.00 [ -0.00]( 3.78) 0.93 [ 7.14]( 0.00) 4 1.00 [ -0.00]( 8.91) 0.80 [ 20.00]( 4.43) 8 1.00 [ -0.00]( 7.45) 1.00 [ -0.00]( 7.45) 16 1.00 [ -0.00]( 4.08) 1.00 [ -0.00](10.79) 32 1.00 [ -0.00](16.90) 0.93 [ 6.67](10.00) 64 1.00 [ -0.00]( 9.11) 1.12 [-12.50]( 0.00) 128 1.00 [ -0.00]( 7.05) 2.43 [-142.86](24.47)
OK, this was what I saw too. I'm looking into this.
Quoted 33 lines
256 1.00 [ -0.00]( 4.32) 1.02 [ -2.34]( 1.20) 512 1.00 [ -0.00]( 0.35) 1.01 [ -0.77]( 0.40)
==================================================================
Test : new-schbench-request-latency Units : Normalized 99th percentile latency in us
Interpretation: Lower is better
Statistic : Median
==================================================================
#workers: tip[pct imp](CV) cache_aware_lb[pct imp](CV) 1 1.00 [ -0.00]( 0.78) 1.16 [-15.70]( 2.14) 2 1.00 [ -0.00]( 0.81) 1.13 [-13.11]( 0.62) 4 1.00 [ -0.00]( 0.24) 1.26 [-26.11](16.43) 8 1.00 [ -0.00]( 1.30) 1.03 [ -3.46]( 0.81) 16 1.00 [ -0.00]( 1.11) 1.02 [ -2.12]( 1.85) 32 1.00 [ -0.00]( 5.94) 0.96 [ 4.05]( 4.48) 64 1.00 [ -0.00]( 6.27) 1.06 [ -6.01]( 6.67) 128 1.00 [ -0.00]( 0.21) 1.12 [-12.31]( 2.61) 256 1.00 [ -0.00](13.73) 1.06 [ -6.30]( 3.37) 512 1.00 [ -0.00]( 0.95) 1.05 [ -4.85]( 0.61)==================================================================
Test : Various longer running benchmarks Units : %diff in throughput reported
Interpretation: Higher is better Statistic : Median
==================================================================
Benchmarks: %diff ycsb-cassandra -1.21% ycsb-mongodb -0.69%
Quoted 8 lines
deathstarbench-1x -7.40%
deathstarbench-2x -3.80%
deathstarbench-3x -3.99%
deathstarbench-6x -3.02%hammerdb+mysql 16VU -2.59%
hammerdb+mysql 64VU -1.05%
For long-duration task, the penalty of remote cache access is severe. This might indicate a similar issue as tbench/netperf as you mentioned, different processes are aggregated to different LLCs, but these processes interact with each other and WF_SYNC did not take effect.
Quoted 3 lines
Also, could you fold the below diff into your Patch2:
Sure, let me apply it and do the test.
thanks,
Chenyu
Quoted 33 lines
kernel/sched/fair.c +0
-0@@ -7694,8 +7694,6 @@ static int select_idle_cpu(struct task_struct *p,struct
sched_domain*sd, bool
int i, cpu,idle_cpu= -1, nr =INT_MAX;
structsched_domain_shared*sd_share;-
cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr);kernel/sched/fair.c +0
-1- if (sched_feat(SIS_UTIL)) { sd_share = rcu_dereference(per_cpu(sd_llc_shared, target)); if (sd_share) {@@ -7707,6 +7705,8 @@ static int select_idle_cpu(struct task_struct *p,struct
sched_domain*sd, bool
}
}+
cpumask_and(cpus, sched_domain_span(sd), p->cpus_ptr); +
if (static_branch_unlikely(&sched_cluster_active)) { structsched_group*sg =sd->groups;---
If the
SIS_UTILcut off hits, that result of thecpumask_and()is of no use. To save some additional cycles, especially in cases where we target the LLC frequently and the search bails out because the LLC is busy, this overhead can be easily avoided. Sinceselect_idle_cpu()can now be called twice per wakeup, this overhead can be visible in benchmarks like hackbench.