Cache Aware Scheduling
Subsystems (4)
cover Cache Aware Scheduling
There had been 4 RFC postings of this patch set. We've incorporated the feedbacks and comments and now would like to post this patch set for consideration of inclusion to mainline. The patches are based on the original patch proposed by Peter[1].
The goal of the patch series is to aggregate tasks sharing data to the same LLC cache domain, thereby reducing cache bouncing and cache misses, and improve data access efficiency. In the current implementation, threads within the same process are considered as entities that potentially share resources.
The changes from v4 RFC patches are minor. Most are commit log and
and code clean ups per feedbacks. Several bugs were fixed:
1. A memory leak of not freeing cache aware scheduling structure when struct mm is freed.
2. A false sharing regression involving nr_running_avg.
3. Bug for initializing cache aware scheduling structures for system with no L3.
Peter suggested enhancing the patch set to allow task aggregation into secondary LLCs when the preferred LLC becomes overloaded. We have not implemented that in this version. In our previous testing, maintaining stable LLC preferences proved important to avoid excessive task migrations, which can undermine cache locality benefits. Additionally, migrating tasks between primary and secondary LLCs often caused cache bouncing, making the locality gains from using a secondary LLC marginal. We would have to take a closer look to see if such scheme can can be done without the such problems.
The following tunables control under /sys/kernel/debug/sched/ control the behavior of cache aware scheduling:
1. llc_aggr_tolerance Controls how aggressive we aggregate tasks to their preferred LLC, based on a process's RSS size and number of running threads. Processes that have smaller memory footprint and fewer number of tasks will benefit better from aggregation. Varies between 0 to 100 0: Cache aware scheduling is disabled 1: Process with RSS greater than LLC size,
or running threads more than number of cpu cores/LLC skip aggregation
100: Aggressive; a process's threads are aggregated regardless of
RSS or running threads.
For example, with a 32MB L3 cache 8 cores in L3:
llc_aggr_tolerance=1 -> process with RSS > 32MB, or nr_running_avg >
8 are skipped. llc_aggr_tolerance=99 -> process with RSS > 784GB
or nr_running_avg > 785 are skipped. 784GB = (1 + (99 - 1) * 256)
* 32MB.
785 = (1 + (99 - 1) * 8).Currently this knob is a global control. Considering that different workloads have different requirements for task consolidation, it would be ideal to introduce per process control for this knob via prctl in the future.
2. llc_overload_pct, llc_imb_pct
We'll always try to move a task to its preferred LLC if an LLC's average core utilization is below llc_overload_pct (default to 50%). Otherwise, the utilization of preferred LLC has to be not more than llc_imb_pct (default to 20%) to move a task to it. This is to prevent overloading on the preferred LLC.
3. llc_epoch_period
Controls how often the scheduler collect LLC occupancy of a process (default to 10 msec)
4. llc_epoch_affinity_timeout
Detect that if a process has not run for llc_epoch_affinity_timeout (default to 50 msec), it loses its cache preference.
Test results:
The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.
The second test platform is a AMD Milan. There are 2 Nodes and 64 CPUs per node. Each node has 8 CCXs and each CCX has 8 CPUs.
The third test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.
[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when there is 1 group with different number of fd pairs(threads) within this process. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows ~10% throughput improvement. Other micro-workloads did not show much difference.
Milan:
No obvious difference is observed so far.
Genoa:
ChaCha20-xiangshan shows 44% throughput improvement.
[Sapphire Rapids details]
[hackbench]
Hackbench show overall improvement when there is only 1 group, with different number of fd(pairs). This is the expected behavior because this test scenario would benefit from cache aware load balance most. Other number of groups shows not much difference(using default fd = 20).
groups baseline sched_cache
Min 1 37.5960 ( 0.00%) 26.4340 ( 29.69%) Min 3 38.7050 ( 0.00%) 38.6920 ( 0.03%) Min 5 39.4550 ( 0.00%) 38.6280 ( 2.10%) Min 7 51.4270 ( 0.00%) 50.6790 ( 1.45%) Min 12 62.8540 ( 0.00%) 63.6590 ( -1.28%) Min 16 74.0160 ( 0.00%) 74.7480 ( -0.99%) Amean 1 38.4768 ( 0.00%) 26.7146 * 30.57%* Amean 3 39.0750 ( 0.00%) 39.5586 ( -1.24%) Amean 5 41.5178 ( 0.00%) 41.2766 ( 0.58%) Amean 7 52.1164 ( 0.00%) 51.5152 ( 1.15%) Amean 12 63.9052 ( 0.00%) 64.0420 ( -0.21%) Amean 16 74.5812 ( 0.00%) 75.4318 ( -1.14%) BAmean-99 1 38.2027 ( 0.00%) 26.5500 ( 30.50%) BAmean-99 3 38.8725 ( 0.00%) 39.2225 ( -0.90%) BAmean-99 5 41.1898 ( 0.00%) 41.0037 ( 0.45%) BAmean-99 7 51.8645 ( 0.00%) 51.4453 ( 0.81%) BAmean-99 12 63.6317 ( 0.00%) 63.9307 ( -0.47%) BAmean-99 16 74.4528 ( 0.00%) 75.2113 ( -1.02%)
[schbench]
Wakeup Latencies 99.0th improvement is observed.
threads baseline sched_cache change 1 13.80(1.10) 14.80(2.86) -7.25% 2 12.00(1.00) 8.00(2.12) +33.33% 4 9.00(0.00) 5.60(0.89) +37.78% 8 9.00(0.00) 6.40(1.14) +28.89% 16 9.20(0.45) 6.20(0.84) +32.61% 32 9.60(0.55) 7.00(0.71) +27.08% 64 10.80(0.45) 8.40(0.55) +22.22% 128 12.60(0.55) 11.40(0.55) +9.52% 239 14.00(0.00) 14.20(0.45) -1.43%
[stream]
No much difference is observed.
baseline sc
GB/sec copy-2 35.00 ( 0.00%) 34.79 ( -0.60%) GB/sec scale-2 24.04 ( 0.00%) 23.90 ( -0.58%) GB/sec add-2 28.98 ( 0.00%) 28.92 ( -0.22%) GB/sec triad-2 28.32 ( 0.00%) 28.31 ( -0.04%)
[netperf]
No much difference is observed(consider the stdev).
nr_pairs netperf netperf
Hmean 60 1023.44 ( 0.00%) 1021.87 ( -0.15%) BHmean-99 60 1023.78 ( 0.00%) 1022.22 ( -0.15%) Hmean 120 792.09 ( 0.00%) 793.75 ( 0.21%) BHmean-99 120 792.36 ( 0.00%) 794.04 ( 0.21%) Hmean 180 513.42 ( 0.00%) 513.53 ( 0.02%) BHmean-99 180 513.81 ( 0.00%) 513.80 ( -0.00%) Hmean 240 387.09 ( 0.00%) 387.33 ( 0.06%) BHmean-99 240 387.18 ( 0.00%) 387.45 ( 0.07%) Hmean 300 316.04 ( 0.00%) 315.68 ( -0.12%) BHmean-99 300 316.12 ( 0.00%) 315.77 ( -0.11%) Hmean 360 496.38 ( 0.00%) 455.49 ( -8.24%) BHmean-99 360 499.88 ( 0.00%) 458.17 ( -8.34%) Hmean 420 497.32 ( 0.00%) 501.84 ( 0.91%) BHmean-99 420 499.90 ( 0.00%) 504.56 ( 0.93%) Hmean 480 417.62 ( 0.00%) 432.25 ( 3.50%) BHmean-99 480 419.96 ( 0.00%) 434.43 ( 3.45%)
In above case of 360 pairs, although there is a performance drop of 8.24%, the corresponding:
HCoeffVar 360 23.78 ( 0.00%) 29.52 ( -24.15%)
shows that the regression is within the run-to-run variance.
[Milan details]
default settings:
[hackbench]
Min 1 50.8170 ( 0.00%) 51.1890 ( -0.73%) Min 3 59.3610 ( 0.00%) 58.6080 ( 1.27%) Min 5 94.9760 ( 0.00%) 96.0210 ( -1.10%) Min 7 123.3270 ( 0.00%) 124.1680 ( -0.68%) Min 12 179.2000 ( 0.00%) 181.8390 ( -1.47%) Min 16 238.8680 ( 0.00%) 242.6390 ( -1.58%) Amean 1 51.6614 ( 0.00%) 51.3630 ( 0.58%) Amean 3 60.1886 ( 0.00%) 59.4542 ( 1.22%) Amean 5 95.7602 ( 0.00%) 96.8338 ( -1.12%) Amean 7 124.0332 ( 0.00%) 124.4406 ( -0.33%) Amean 12 181.0324 ( 0.00%) 182.9220 ( -1.04%) Amean 16 239.5556 ( 0.00%) 243.3556 * -1.59%* BAmean-99 1 51.5335 ( 0.00%) 51.3338 ( 0.39%) BAmean-99 3 59.7848 ( 0.00%) 59.0958 ( 1.15%) BAmean-99 5 95.6698 ( 0.00%) 96.5450 ( -0.91%) BAmean-99 7 123.8478 ( 0.00%) 124.3760 ( -0.43%) BAmean-99 12 180.8035 ( 0.00%) 182.5135 ( -0.95%) BAmean-99 16 239.1933 ( 0.00%) 243.0570 ( -1.62%)
[schbench]
threads baseline sched_cache change 1 12.00(2.00) 11.00(0.71) +8.33% 2 12.40(0.89) 13.80(0.84) -11.29% 4 14.20(0.45) 14.80(0.45) -4.23% 8 16.00(0.00) 15.80(0.45) +1.25% 16 16.00(0.00) 16.00(0.71) 0.00% 32 19.40(0.55) 18.60(0.55) +4.12% 63 22.20(0.45) 23.20(0.45) -4.50%
[stream]
No obvious difference is found.
export STREAM_SIZE=$((128000000))
baseline sched_cache GB/sec copy-16 726.48 ( 0.00%) 715.60 ( -1.50%) GB/sec scale-16 577.71 ( 0.00%) 577.03 ( -0.12%) GB/sec add-16 678.85 ( 0.00%) 672.87 ( -0.88%) GB/sec triad-16 735.52 ( 0.00%) 729.05 ( -0.88%)
[netperf]
No much difference is observed.
nr_pairs baseline sched_cache
Hmean 32 755.98 ( 0.00%) 755.17 ( -0.11%) BHmean-99 32 756.42 ( 0.00%) 755.40 ( -0.13%) Hmean 64 677.38 ( 0.00%) 669.75 ( -1.13%) BHmean-99 64 677.50 ( 0.00%) 669.86 ( -1.13%) Hmean 96 498.52 ( 0.00%) 496.73 ( -0.36%) BHmean-99 96 498.69 ( 0.00%) 496.93 ( -0.35%) Hmean 128 604.38 ( 0.00%) 604.22 ( -0.03%) BHmean-99 128 604.87 ( 0.00%) 604.87 ( 0.00%) Hmean 160 471.67 ( 0.00%) 468.29 ( -0.72%) BHmean-99 160 474.34 ( 0.00%) 471.05 ( -0.69%) Hmean 192 381.18 ( 0.00%) 384.88 ( 0.97%) BHmean-99 192 383.30 ( 0.00%) 386.82 ( 0.92%) Hmean 224 327.79 ( 0.00%) 326.05 ( -0.53%) BHmean-99 224 329.85 ( 0.00%) 327.87 ( -0.60%) Hmean 256 284.61 ( 0.00%) 300.52 ( 5.59%) BHmean-99 256 286.41 ( 0.00%) 302.06 ( 5.47%)
[Genoa details]
[ChaCha20-xiangshan]
ChaCha20-xiangshan is a simple benchmark using a static build of an 8-thread Verilator of XiangShan(RISC-V). The README file can be found here[2]. The score depends on how aggressive the user set the /sys/kernel/debug/sched/llc_aggr_tolerance. Using the default values, there is no much difference observed. While setting the /sys/kernel/debug/sched/llc_aggr_tolerance to 100, 44% improvment is observed.
baseline:
Host time spent: 50,868ms
sched_cache:
Host time spent: 28,349ms
The time has been reduced by 44%.
Thanks to everyone who participated and provided valuable suggestions for the previous versions. Your comments and tests on the latest version are also greatly appreciated in advance.
Tim
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://github.com/yu-chen-surf/chacha20-xiangshan/blob/master/README.eng.md
RFC v4:
[3] https://lore.kernel.org/all/[email protected]/
RFC v3
[4] https://lore.kernel.org/all/[email protected]/
RFC v2:
[5] https://lore.kernel.org/lkml/[email protected]/
Chen Yu (7):
sched/fair: Record per-LLC utilization to guide cache-aware scheduling
decisions
sched/fair: Introduce helper functions to enforce LLC migration policy sched/fair: Introduce a static key to enable cache aware only for
multi LLCs
sched/fair: Exclude processes with many threads from cache-aware
scheduling
sched/fair: Disable cache aware scheduling for processes with high
thread counts
sched/fair: Avoid cache-aware scheduling for memory-heavy processes sched/fair: Add user control to adjust the tolerance of cache-aware
scheduling
Peter Zijlstra (Intel) (1):
sched/fair: Add infrastructure for cache-aware load balancing
Tim Chen (11):
sched/fair: Add LLC index mapping for CPUs
sched/fair: Assign preferred LLC ID to processes sched/fair: Track LLC-preferred tasks per runqueue sched/fair: Introduce per runqueue task LLC preference counter sched/fair: Count tasks prefering each LLC in a sched group sched/fair: Prioritize tasks preferring destination LLC during
balancing
sched/fair: Identify busiest sched_group for LLC-aware load balancing sched/fair: Add migrate_llc_task migration type for cache-aware
balancing
sched/fair: Handle moving single tasks to/from their preferred LLC sched/fair: Consider LLC preference when selecting tasks for load
balancing
sched/fair: Respect LLC preference in task migration and detach
include/linux/cacheinfo.h | 21 +- include/linux/mm_types.h | 45 ++ include/linux/sched.h | 5 + include/linux/sched/topology.h | 4 + include/linux/threads.h | 10 + init/Kconfig | 20 + init/init_task.c | 3 + kernel/fork.c | 6 + kernel/sched/core.c | 18 + kernel/sched/debug.c | 56 ++ kernel/sched/fair.c | 1022 +++++++++++++++++++++++++++++++- kernel/sched/features.h | 1 + kernel/sched/sched.h | 27 + kernel/sched/topology.c | 61 +- 14 files changed, 1283 insertions(+), 16 deletions(-)
Signature · 1 line
2.32.0
PATCH 1/19 sched/fair: Add infrastructure for cache-aware load balancing
From: "Peter Zijlstra (Intel)" <[email protected]>
Cache-aware load balancing aims to aggregate tasks with potential shared resources into the same cache domain. This approach enhances cache locality, thereby optimizing system performance by reducing cache misses and improving data access efficiency.
In the current implementation, threads within the same process are considered as entities that potentially share resources. Cache-aware load balancing monitors the CPU occupancy of each cache domain for every process. Based on this monitoring, it endeavors to migrate threads within a given process to its cache-hot domains, with the goal of maximizing cache locality.
It is an attempt at modelling cache affinity. While the patch series only targets LLC, it could very well be extended to clusters (L2), or other kind of domains grouping inside a node.
As it stands, the mechanism only computes a CPU within the LLC that has the highest recent runtime; this CPU is then used in the load balance path in subsequent patches to steer toward this LLC.
More elaborate measures could be added later in NUMA_BALANCING: for example, migrating task A to its preferred LLC when it has spare CPU capacity, or swapping task A with another running task B in task A’s preferred LLC.
--- include/linux/mm_types.h | 44 ++++++ include/linux/sched.h | 4 + init/Kconfig | 11 ++ kernel/fork.c | 6 + kernel/sched/core.c | 6 + kernel/sched/fair.c | 288 +++++++++++++++++++++++++++++++++++++++ kernel/sched/features.h | 1 + kernel/sched/sched.h | 8 ++ 8 files changed, 368 insertions(+)
include/linux/mm_types.h +44 -0
@@ -927,6 +927,11 @@ struct mm_cid { };#endif +struct mm_sched { +u64 runtime; +unsigned long epoch; +}; +struct kioctx_table ;struct iommu_mm_data ;struct mm_struct {
@@ -1017,6 +1022,17 @@ struct mm_struct {*/ raw_spinlock_t cpus_allowed_lock;#endif +#ifdef CONFIG_SCHED_CACHE +/* +* Track per-cpu-per-process occupancy as a proxy for cache residency. +* See account_mm_sched() and ... +*/ +struct mm_sched __percpu *pcpu_sched; +raw_spinlock_t mm_sched_lock; +unsigned long mm_sched_epoch; +int mm_sched_cpu; +#endif +#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */ #endif
@@ -1436,6 +1452,34 @@ static inline unsigned int mm_cid_size(void)static inline void mm_set_cpus_allowed (struct mm_struct *mm,const struct cpumask *cpumask) { }#endif /* CONFIG_SCHED_MM_CID */ +#ifdef CONFIG_SCHED_CACHE +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *pcpu_sched); + +static inline int mm_alloc_sched_noprof (struct mm_struct *mm) +{ + struct mm_sched __percpu *pcpu_sched = alloc_percpu_noprof(struct mm_sched); + +if (!pcpu_sched) +return -ENOMEM ; + +mm_init_sched (mm, pcpu_sched); +return 0 ; +} + +#define mm_alloc_sched (...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) + +static inline void mm_destroy_sched (struct mm_struct *mm) +{ +free_percpu (mm->pcpu_sched); + mm->pcpu_sched =NULL ; +} +#else /* !CONFIG_SCHED_CACHE */ + +static inline int mm_alloc_sched (struct mm_struct *mm) {return 0 ; } +static inline void mm_destroy_sched (struct mm_struct *mm) { } + +#endif /* CONFIG_SCHED_CACHE */ +struct mmu_gather ;extern void tlb_gather_mmu (struct mmu_gather *tlb,struct mm_struct *mm);extern void tlb_gather_mmu_fullmm (struct mmu_gather *tlb,struct mm_struct *mm);
include/linux/sched.h +4 -0
@@ -1400,6 +1400,10 @@ struct task_struct {unsigned long numa_pages_migrated;#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +struct callback_head cache_work; +#endif +#ifdef CONFIG_RSEQ struct rseq __user *rseq;u32 rseq_len;
init/Kconfig +11 -0
@@ -970,6 +970,17 @@ config NUMA_BALANCING This system will be inactive on UMA systems. +config SCHED_CACHE + bool "Cache aware load balance" + default y + depends on SMP + help + When enabled, the scheduler will attempt to aggregate tasks from + the same process onto a single Last Level Cache (LLC) domain when + possible. This improves cache locality by keeping tasks that share + resources within the same cache domain, reducing cache misses and + lowering data access latency. + config NUMA_BALANCING_DEFAULT_ENABLED bool "Automatically enable NUMA aware memory/task placement" default y
kernel/fork.c +6 -0
@@ -680,6 +680,7 @@ void __mmdrop(struct mm_struct *mm)cleanup_lazy_tlbs (mm);WARN_ON_ONCE (mm == current->active_mm); +mm_destroy_sched (mm);mm_free_pgd (mm);mm_free_id (mm);destroy_context (mm);
@@ -1079,6 +1080,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,if (mm_alloc_cid (mm, p)) goto fail_cid; +if (mm_alloc_sched (mm)) + goto fail_sched; +if (percpu_counter_init_many (mm->rss_stat,0 ,GFP_KERNEL_ACCOUNT ,NR_MM_COUNTERS )) goto fail_pcpu;
@@ -1088,6 +1092,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,return mm; fail_pcpu: +mm_destroy_sched (mm); +fail_sched:mm_destroy_cid (mm); fail_cid:destroy_context (mm);
kernel/sched/core.c +6 -0
@@ -4520,6 +4520,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p) p->wake_entry.u_flags =CSD_TYPE_TTWU ; p->migration_pending =NULL ;init_sched_mm_cid (p); +init_sched_mm (p); }DEFINE_STATIC_KEY_FALSE (sched_numa_balancing);
@@ -8821,6 +8822,11 @@ void __init sched_init(void) rq->core_cookie =0UL ;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spin_lock_init (&rq->cpu_epoch_lock); + rq->cpu_epoch_next = jiffies; +#endif +zalloc_cpumask_var_node (&rq->scratch_mask,GFP_KERNEL ,cpu_to_node (i)); }
kernel/sched/fair.c +288 -0
312 lines not on this page · raw message
kernel/sched/features.h +1 -0
@@ -87,6 +87,7 @@ SCHED_FEAT(TTWU_QUEUE, true)*/ SCHED_FEAT(SIS_UTIL, true) +SCHED_FEAT (SCHED_CACHE , true)/* * Issue a WARN when we do multiple update_rq_clock() calls * in a single rq->lock section. Default disabled because the
kernel/sched/sched.h +8 -0
@@ -1166,6 +1166,12 @@ struct rq {u64 clock_pelt_idle_copy;u64 clock_idle_copy;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spinlock_t cpu_epoch_lock____cacheline_aligned ; +u64 cpu_runtime; +unsigned long cpu_epoch; +unsigned long cpu_epoch_next; +#endif atomic_t nr_iowait;
@@ -3790,6 +3796,8 @@ static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) { }static inline void init_sched_mm_cid (struct task_struct *t) { }#endif /* !CONFIG_SCHED_MM_CID */ +extern void init_sched_mm (struct task_struct *p); +extern u64 avg_vruntime (struct cfs_rq *cfs_rq);extern int entity_eligible (struct cfs_rq *cfs_rq,struct sched_entity *se); static inline
Signature · 1 line
2.32.0
Re: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
Quoted 7 lines · Tim Chen
From: "Peter Zijlstra (Intel)" <[email protected]>
Cache-aware load balancing aims to aggregate tasks with potential shared resources into the same cache domain. This approach enhances cache locality, thereby optimizing system performance by reducing cache misses and improving data access efficiency.
[snip]
Quoted 43 lines
kernel/sched/fair.c +43
-0+static void get_scan_cpumasks (cpumask_var_t cpus,int cache_cpu, +int pref_nid,int curr_cpu) +{ +#ifdef CONFIG_NUMA_BALANCING +/* First honor the task's preferred node. */ +if (pref_nid !=NUMA_NO_NODE ) +cpumask_or (cpus, cpus,cpumask_of_node (pref_nid)); +#endif + +/* Next honor the task's cache CPU if it is not included. */ +if (cache_cpu !=-1 && !cpumask_test_cpu (cache_cpu, cpus)) +cpumask_or (cpus, cpus, +cpumask_of_node (cpu_to_node (cache_cpu))); + +/* +* Lastly make sure that the task's current running node is +* considered. +*/ +if (!cpumask_test_cpu (curr_cpu, cpus)) +cpumask_or (cpus, cpus,cpumask_of_node (cpu_to_node (curr_cpu))); +} + +static void __no_profile task_cache_work(struct callback_head *work) +{ +struct task_struct *p = current; +struct mm_struct *mm = p->mm; +unsigned long m_a_occ =0 ; +unsigned long curr_m_a_occ =0 ; +int cpu, m_a_cpu =-1 , cache_cpu, + pref_nid =NUMA_NO_NODE , curr_cpu; +cpumask_var_t cpus; + +WARN_ON_ONCE (work != &p->cache_work); + + work->next = work; + +if (p->flags &PF_EXITING ) +return ; + +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL )) +return ; + + curr_cpu =task_cpu (p);
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1399
Quoted 10 lines
kernel/sched/fair.c +10
-0+ cache_cpu = mm->mm_sched_cpu; +#ifdef CONFIG_NUMA_BALANCING +if (static_branch_likely (&sched_numa_balancing)) + pref_nid = p->numa_preferred_nid; +#endif + +scoped_guard (cpus_read_lock) { +get_scan_cpumasks (cpus, cache_cpu, + pref_nid, curr_cpu); +
IIUC, get_scan_cpumasks ORs together the preferred NUMA node, cache CPU's node, and current CPU's node. This could result in scanning multiple nodes, not preferring the NUMA preferred node.
Quoted 34 lines
kernel/sched/fair.c +34
-0+for_each_cpu (cpu, cpus) { +/* XXX sched_cluster_active */ +struct sched_domain *sd =per_cpu (sd_llc, cpu); +unsigned long occ, m_occ =0 , a_occ =0 ; +int m_cpu =-1 , i; + +if (!sd) +continue ; + +for_each_cpu (i,sched_domain_span (sd)) { + occ =fraction_mm_sched (cpu_rq (i), +per_cpu_ptr (mm->pcpu_sched, i)); + a_occ += occ; +if (occ > m_occ) { + m_occ = occ; + m_cpu = i; + } + } + +/* +* Compare the accumulated occupancy of each LLC. The +* reason for using accumulated occupancy rather than average +* per CPU occupancy is that it works better in asymmetric LLC +* scenarios. +* For example, if there are 2 threads in a 4CPU LLC and 3 +* threads in an 8CPU LLC, it might be better to choose the one +* with 3 threads. However, this would not be the case if the +* occupancy is divided by the number of CPUs in an LLC (i.e., +* if average per CPU occupancy is used). +* Besides, NUMA balancing fault statistics behave similarly: +* the total number of faults per node is compared rather than +* the average number of faults per CPU. This strategy is also +* followed here. +*/
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1443
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (a_occ > m_a_occ) { + m_a_occ = a_occ; + m_a_cpu = m_cpu; + } + +if (llc_id (cpu) ==llc_id (mm->mm_sched_cpu)) + curr_m_a_occ = a_occ; + +cpumask_andnot (cpus, cpus,sched_domain_span (sd)); + }
This means NUMA preference has no effect on the selection, except in the unlikely case of exactly equal occupancy across LLCs on different nodes (where iteration order determines the winner).
How does it handle when cache locality and memory locality conflict? Shouldn't numa preferred node get preference? Also scanning multiple nodes add overhead, so can restricting it to numa preferred node be better and scan others only when there is no numa preferred node?
Let me know if I am missing anything.
Thanks,
Madadi Vineeth Reddy
Quoted 88 lines
kernel/sched/fair.c +40
-0+ } + +if (m_a_occ > (2 * curr_m_a_occ)) { +/* +* Avoid switching mm_sched_cpu too fast. +* The reason to choose 2X is because: +* 1. It is better to keep the preferred LLC stable, +* rather than changing it frequently and cause migrations +* 2. 2X means the new preferred LLC has at least 1 more +* busy CPU than the old one(200% vs 100%, eg) +* 3. 2X is chosen based on test results, as it delivers +* the optimal performance gain so far. +*/ + mm->mm_sched_cpu = m_a_cpu; + } + +free_cpumask_var (cpus); +} + +void init_sched_mm (struct task_struct *p) +{ +struct callback_head *work = &p->cache_work; + +init_task_work (work, task_cache_work); + work->next = work; +} + +#else + +static inline void account_mm_sched (struct rq *rq,struct task_struct *p, +s64 delta_exec) { } + +void init_sched_mm (struct task_struct *p) { } + +static void task_tick_cache (struct rq *rq,struct task_struct *p) { } + +#endif +/* * Used by other classes to account runtime. */ @@ -13031,6 +13317,8 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)if (static_branch_unlikely (&sched_numa_balancing))task_tick_numa (rq, curr); +task_tick_cache (rq, curr); +update_misfit_status (curr, rq);check_update_overutilized_status (task_rq (curr));kernel/sched/features.h +1
-0@@ -87,6 +87,7 @@ SCHED_FEAT(TTWU_QUEUE, true)*/ SCHED_FEAT(SIS_UTIL, true) +SCHED_FEAT (SCHED_CACHE , true)/* * Issue a WARN when we do multiple update_rq_clock() calls * in a single rq->lock section. Default disabled because the kernel/sched/sched.h +8
-0@@ -1166,6 +1166,12 @@ struct rq {u64 clock_pelt_idle_copy;u64 clock_idle_copy;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spinlock_t cpu_epoch_lock____cacheline_aligned ; +u64 cpu_runtime; +unsigned long cpu_epoch; +unsigned long cpu_epoch_next; +#endif atomic_t nr_iowait;@@ -3790,6 +3796,8 @@ static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) { }static inline void init_sched_mm_cid (struct task_struct *t) { }#endif /* !CONFIG_SCHED_MM_CID */ +extern void init_sched_mm (struct task_struct *p); +extern u64 avg_vruntime (struct cfs_rq *cfs_rq);extern int entity_eligible (struct cfs_rq *cfs_rq,struct sched_entity *se); static inline
Re: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
Quoted 8 lines · Madadi Vineeth Reddy
Quoted 7 lines · Tim Chen
From: "Peter Zijlstra (Intel)" <[email protected]>
Cache-aware load balancing aims to aggregate tasks with potential shared resources into the same cache domain. This approach enhances cache locality, thereby optimizing system performance by reducing cache misses and improving data access efficiency.
[snip]
Quoted 27 lines
Quoted 27 lines
kernel/sched/fair.c +27
-0+static void __no_profile task_cache_work(struct callback_head *work) +{ +struct task_struct *p = current; +struct mm_struct *mm = p->mm; +unsigned long m_a_occ =0 ; +unsigned long curr_m_a_occ =0 ; +int cpu, m_a_cpu =-1 , cache_cpu, + pref_nid =NUMA_NO_NODE , curr_cpu; +cpumask_var_t cpus; + +WARN_ON_ONCE (work != &p->cache_work); + + work->next = work; + +if (p->flags &PF_EXITING ) +return ; + +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL )) +return ; + + curr_cpu =task_cpu (p); + cache_cpu = mm->mm_sched_cpu; +#ifdef CONFIG_NUMA_BALANCING +if (static_branch_likely (&sched_numa_balancing)) + pref_nid = p->numa_preferred_nid; +#endif +
Quoted 9 lines
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1399
Quoted 4 lines
kernel/sched/fair.c +4
-0+scoped_guard (cpus_read_lock) { +get_scan_cpumasks (cpus, cache_cpu, + pref_nid, curr_cpu); +IIUC,
get_scan_cpumasksORs together the preferred NUMA node, cache CPU's node, and current CPU's node. This could result in scanning multiple nodes, not preferring the NUMA preferred node.
Yes, it is possible, please see comments below.
Quoted 49 lines
Quoted 34 lines
kernel/sched/fair.c +34
-0+for_each_cpu (cpu, cpus) { +/* XXX sched_cluster_active */ +struct sched_domain *sd =per_cpu (sd_llc, cpu); +unsigned long occ, m_occ =0 , a_occ =0 ; +int m_cpu =-1 , i; + +if (!sd) +continue ; + +for_each_cpu (i,sched_domain_span (sd)) { + occ =fraction_mm_sched (cpu_rq (i), +per_cpu_ptr (mm->pcpu_sched, i)); + a_occ += occ; +if (occ > m_occ) { + m_occ = occ; + m_cpu = i; + } + } + +/* +* Compare the accumulated occupancy of each LLC. The +* reason for using accumulated occupancy rather than average +* per CPU occupancy is that it works better in asymmetric LLC +* scenarios. +* For example, if there are 2 threads in a 4CPU LLC and 3 +* threads in an 8CPU LLC, it might be better to choose the one +* with 3 threads. However, this would not be the case if the +* occupancy is divided by the number of CPUs in an LLC (i.e., +* if average per CPU occupancy is used). +* Besides, NUMA balancing fault statistics behave similarly: +* the total number of faults per node is compared rather than +* the average number of faults per CPU. This strategy is also +* followed here. +*/ kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1443
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (a_occ > m_a_occ) { + m_a_occ = a_occ; + m_a_cpu = m_cpu; + } + +if (llc_id (cpu) ==llc_id (mm->mm_sched_cpu)) + curr_m_a_occ = a_occ; + +cpumask_andnot (cpus, cpus,sched_domain_span (sd)); + }This means NUMA preference has no effect on the selection, except in the unlikely case of exactly equal occupancy across LLCs on different nodes (where iteration order determines the winner).
Quoted 5 lines
How does it handle when cache locality and memory locality conflict? Shouldn't numa preferred node get preference? Also scanning multiple nodes add overhead, so can restricting it to numa preferred node be better and scan others only when there is no numa preferred node?
Basically, yes, you're right. Ideally, we should prioritize the NUMA preferred node as the top priority. There's one case I find hard to
handle: the NUMA preferred node is per task rather than per process. It's possible that different threads of the same process have different preferred nodes; as a result, the process-wide preferred LLC could bounce between different nodes, which might cause costly task migrations across nodes. As a workaround, we tried to keep the scan CPU mask covering the process's current preferred LLC to ensure the old preferred LLC is included in the candidates. After all, we have a 2X threshold for switching the preferred LLC.
thanks,
Chenyu
Re: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
Quoted 106 lines · Yu C
Quoted 8 lines · Madadi Vineeth Reddy
Quoted 7 lines · Tim Chen
From: "Peter Zijlstra (Intel)" <[email protected]>
Cache-aware load balancing aims to aggregate tasks with potential shared resources into the same cache domain. This approach enhances cache locality, thereby optimizing system performance by reducing cache misses and improving data access efficiency.
[snip]
Quoted 27 lines
Quoted 27 lines
kernel/sched/fair.c +27
-0+static void __no_profile task_cache_work(struct callback_head *work) +{ +struct task_struct *p = current; +struct mm_struct *mm = p->mm; +unsigned long m_a_occ =0 ; +unsigned long curr_m_a_occ =0 ; +int cpu, m_a_cpu =-1 , cache_cpu, + pref_nid =NUMA_NO_NODE , curr_cpu; +cpumask_var_t cpus; + +WARN_ON_ONCE (work != &p->cache_work); + + work->next = work; + +if (p->flags &PF_EXITING ) +return ; + +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL )) +return ; + + curr_cpu =task_cpu (p); + cache_cpu = mm->mm_sched_cpu; +#ifdef CONFIG_NUMA_BALANCING +if (static_branch_likely (&sched_numa_balancing)) + pref_nid = p->numa_preferred_nid; +#endif +Quoted 9 lines
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1399
Quoted 4 lines
kernel/sched/fair.c +4
-0+scoped_guard (cpus_read_lock) { +get_scan_cpumasks (cpus, cache_cpu, + pref_nid, curr_cpu); +IIUC,
get_scan_cpumasksORs together the preferred NUMA node, cache CPU's node, and current CPU's node. This could result in scanning multiple nodes, not preferring the NUMA preferred node.Yes, it is possible, please see comments below.
Quoted 49 lines
Quoted 34 lines
kernel/sched/fair.c +34
-0+for_each_cpu (cpu, cpus) { +/* XXX sched_cluster_active */ +struct sched_domain *sd =per_cpu (sd_llc, cpu); +unsigned long occ, m_occ =0 , a_occ =0 ; +int m_cpu =-1 , i; + +if (!sd) +continue ; + +for_each_cpu (i,sched_domain_span (sd)) { + occ =fraction_mm_sched (cpu_rq (i), +per_cpu_ptr (mm->pcpu_sched, i)); + a_occ += occ; +if (occ > m_occ) { + m_occ = occ; + m_cpu = i; + } + } + +/* +* Compare the accumulated occupancy of each LLC. The +* reason for using accumulated occupancy rather than average +* per CPU occupancy is that it works better in asymmetric LLC +* scenarios. +* For example, if there are 2 threads in a 4CPU LLC and 3 +* threads in an 8CPU LLC, it might be better to choose the one +* with 3 threads. However, this would not be the case if the +* occupancy is divided by the number of CPUs in an LLC (i.e., +* if average per CPU occupancy is used). +* Besides, NUMA balancing fault statistics behave similarly: +* the total number of faults per node is compared rather than +* the average number of faults per CPU. This strategy is also +* followed here. +*/ kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1443
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (a_occ > m_a_occ) { + m_a_occ = a_occ; + m_a_cpu = m_cpu; + } + +if (llc_id (cpu) ==llc_id (mm->mm_sched_cpu)) + curr_m_a_occ = a_occ; + +cpumask_andnot (cpus, cpus,sched_domain_span (sd)); + }This means NUMA preference has no effect on the selection, except in the unlikely case of exactly equal occupancy across LLCs on different nodes (where iteration order determines the winner).
Quoted 5 lines
How does it handle when cache locality and memory locality conflict? Shouldn't numa preferred node get preference? Also scanning multiple nodes add overhead, so can restricting it to numa preferred node be better and scan others only when there is no numa preferred node?
Quoted 10 lines
Basically, yes, you're right. Ideally, we should prioritize the NUMA preferred node as the top priority. There's one case I find hard to
handle: the NUMA preferred node is per task rather than per process. It's possible that different threads of the same process have different preferred nodes; as a result, the process-wide preferred LLC could bounce between different nodes, which might cause costly task migrations across nodes. As a workaround, we tried to keep the scan CPU mask covering the process's current preferred LLC to ensure the old preferred LLC is included in the candidates. After all, we have a 2X threshold for switching the preferred LLC.
If tasks in a process had different preferred nodes, they would belong to different numa_groups, and majority of their data would be from different NUMA nodes.
To resolve such conflict, we'll need to change the aggregation of tasks by process, to aggregation of tasks by numa_group when NUMA balancing is enabled. This probably makes more sense as tasks in a numa_group have more shared data and would benefit from co-locating in the same cache.
Thanks.
Tim
Re: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
Quoted 4 lines · Tim Chen
Quoted 3 lines · Yu C
Quoted 2 lines · Madadi Vineeth Reddy
Quoted 1 line · Tim Chen
From: "Peter Zijlstra (Intel)" <[email protected]>
[snip]
Quoted 22 lines
Quoted 7 lines
Quoted 6 lines
How does it handle when cache locality and memory locality conflict? Shouldn't numa preferred node get preference? Also scanning multiple nodes add overhead, so can restricting it to numa preferred node be better and scan others only when there is no numa preferred node?
Quoted 10 lines
Basically, yes, you're right. Ideally, we should prioritize the NUMA preferred node as the top priority. There's one case I find hard to
handle: the NUMA preferred node is per task rather than per process. It's possible that different threads of the same process have different preferred nodes; as a result, the process-wide preferred LLC could bounce between different nodes, which might cause costly task migrations across nodes. As a workaround, we tried to keep the scan CPU mask covering the process's current preferred LLC to ensure the old preferred LLC is included in the candidates. After all, we have a 2X threshold for switching the preferred LLC.If tasks in a process had different preferred nodes, they would belong to different
numa_groups, and majority of their data would be from different NUMA nodes.
Quoted 6 lines
To resolve such conflict, we'll need to change the aggregation of tasks by process, to aggregation of tasks by
numa_groupwhen NUMA balancing is enabled. This probably makes more sense as tasks in anuma_grouphave more shared data and would benefit from co-locating in the same cache.
Yes, this could be an enhancement if the NUMA balance is enabled.
thanks,
Chenyu
Peter Zijlstra
maintainerRe: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
In reply to Madadi Vineeth ReddyQuoted 49 lines · Madadi Vineeth Reddy
Quoted 49 lines
kernel/sched/fair.c +49
-0+static void get_scan_cpumasks (cpumask_var_t cpus,int cache_cpu, +int pref_nid,int curr_cpu) +{ +#ifdef CONFIG_NUMA_BALANCING +/* First honor the task's preferred node. */ +if (pref_nid !=NUMA_NO_NODE ) +cpumask_or (cpus, cpus,cpumask_of_node (pref_nid)); +#endif + +/* Next honor the task's cache CPU if it is not included. */ +if (cache_cpu !=-1 && !cpumask_test_cpu (cache_cpu, cpus)) +cpumask_or (cpus, cpus, +cpumask_of_node (cpu_to_node (cache_cpu))); + +/* +* Lastly make sure that the task's current running node is +* considered. +*/ +if (!cpumask_test_cpu (curr_cpu, cpus)) +cpumask_or (cpus, cpus,cpumask_of_node (cpu_to_node (curr_cpu))); +} + +static void __no_profile task_cache_work(struct callback_head *work) +{ +struct task_struct *p = current; +struct mm_struct *mm = p->mm; +unsigned long m_a_occ =0 ; +unsigned long curr_m_a_occ =0 ; +int cpu, m_a_cpu =-1 , cache_cpu, + pref_nid =NUMA_NO_NODE , curr_cpu; +cpumask_var_t cpus; + +WARN_ON_ONCE (work != &p->cache_work); + + work->next = work; + +if (p->flags &PF_EXITING ) +return ; + +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL )) +return ; + + curr_cpu =task_cpu (p); + cache_cpu = mm->mm_sched_cpu; +#ifdef CONFIG_NUMA_BALANCING +if (static_branch_likely (&sched_numa_balancing)) + pref_nid = p->numa_preferred_nid; +#endif +
Quoted 8 lines
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1399
Quoted 4 lines
kernel/sched/fair.c +4
-0+scoped_guard (cpus_read_lock) { +get_scan_cpumasks (cpus, cache_cpu, + pref_nid, curr_cpu); +IIUC,
get_scan_cpumasksORs together the preferred NUMA node, cache CPU's node, and current CPU's node. This could result in scanning multiple nodes, not preferring the NUMA preferred node.
So this used to be online_mask, and is now magically changed to this more limited mask.
Could you split this change out and have it have a justification?
Re: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
Quoted 59 lines · Peter Zijlstra
Quoted 49 lines · Madadi Vineeth Reddy
Quoted 49 lines
kernel/sched/fair.c +49
-0+static void get_scan_cpumasks (cpumask_var_t cpus,int cache_cpu, +int pref_nid,int curr_cpu) +{ +#ifdef CONFIG_NUMA_BALANCING +/* First honor the task's preferred node. */ +if (pref_nid !=NUMA_NO_NODE ) +cpumask_or (cpus, cpus,cpumask_of_node (pref_nid)); +#endif + +/* Next honor the task's cache CPU if it is not included. */ +if (cache_cpu !=-1 && !cpumask_test_cpu (cache_cpu, cpus)) +cpumask_or (cpus, cpus, +cpumask_of_node (cpu_to_node (cache_cpu))); + +/* +* Lastly make sure that the task's current running node is +* considered. +*/ +if (!cpumask_test_cpu (curr_cpu, cpus)) +cpumask_or (cpus, cpus,cpumask_of_node (cpu_to_node (curr_cpu))); +} + +static void __no_profile task_cache_work(struct callback_head *work) +{ +struct task_struct *p = current; +struct mm_struct *mm = p->mm; +unsigned long m_a_occ =0 ; +unsigned long curr_m_a_occ =0 ; +int cpu, m_a_cpu =-1 , cache_cpu, + pref_nid =NUMA_NO_NODE , curr_cpu; +cpumask_var_t cpus; + +WARN_ON_ONCE (work != &p->cache_work); + + work->next = work; + +if (p->flags &PF_EXITING ) +return ; + +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL )) +return ; + + curr_cpu =task_cpu (p); + cache_cpu = mm->mm_sched_cpu; +#ifdef CONFIG_NUMA_BALANCING +if (static_branch_likely (&sched_numa_balancing)) + pref_nid = p->numa_preferred_nid; +#endif +Quoted 8 lines
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1399
Quoted 4 lines
kernel/sched/fair.c +4
-0+scoped_guard (cpus_read_lock) { +get_scan_cpumasks (cpus, cache_cpu, + pref_nid, curr_cpu); +IIUC,
get_scan_cpumasksORs together the preferred NUMA node, cache CPU's node, and current CPU's node. This could result in scanning multiple nodes, not preferring the NUMA preferred node.
Quoted 4 lines
So this used to be
online_mask, and is now magically changed to this more limited mask.Could you split this change out and have it have a justification?
OK, we will do this and provide an explanation.
thanks,
Chenyu
kernel test robot
build botRe: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
In reply to Tim Chenkernel test robot noticed a 5.1% improvement of will-it-scale.per_thread_ops on:
Hello,
kernel test robot noticed a 5.1% improvement of will-it-scale.per_thread_ops on:
commit: ddf7df94672b42db9a86b3225cf9ebcfdfefc506 ("[PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing")
url: https://github.com/intel-lab-lkp/linux/commits/Tim-Chen/sched-fair-Add-infrastructure-for-cache-aware-load-balancing/20251012-022248
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 45b7f780739a3145aeef24d2dfa02517a6c82ed6
patch link: https://lore.kernel.org/all/865b852e3fdef6561c9e0a5be9a94aec8a68cdea.1760206683.git.tim.c.chen@linux.intel.com/
patch subject: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
testcase: will-it-scale
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 64 threads 2 sockets Intel(R) Xeon(R) Gold 6346 CPU @ 3.10GHz (Ice Lake) with 256G memory parameters:
nr_task: 100% mode: thread test: mmap1 cpufreq_governor: performance
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20251023/[email protected]
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/thread/100%/debian-13-x86_64-20250902.cgz/lkp-icl-2sp7/mmap1/will-it-scale
commit:
45b7f78073 ("sched: Fix some typos in include/linux/preempt.h") ddf7df9467 ("sched/fair: Add infrastructure for cache-aware load balancing")
45b7f780739a3145 ddf7df94672b42db9a86b3225cf
---------------- ---------------------------
%stddev %change %stddev
\ | \
12.30 ± 4% +1.2 13.54 ± 2% turbostat.C6%
16510 ± 2% +11.0% 18333 ± 5% perf-c2c.HITM.local
14007 ± 2% +11.9% 15670 ± 5% perf-c2c.HITM.remote
30518 ± 2% +11.4% 34003 ± 5% perf-c2c.HITM.total
4216 ±117% -82.9% 720.83 ± 87% sched_debug.cfs_rq:/.load_avg.max
230030 ± 6% +21.9% 280349 ± 3% sched_debug.cpu.nr_switches.min
32696287 +100.0% 65398751 sched_debug.sysctl_sched.sysctl_sched_features
71075 +5.1% 74710 will-it-scale.64.threads
1109 +5.1% 1166 will-it-scale.per_thread_ops
71075 +5.1% 74710 will-it-scale.workload
20012272 +1.8% 20374679 perf-stat.i.branch-misses
20662368 +4.4% 21568525 perf-stat.i.cache-references
181255 ± 6% +10.5% 200376 ± 2% perf-stat.i.context-switches
177.39 +5.8% 187.65 perf-stat.i.cpu-migrations
22226 ± 3% -5.9% 20924 ± 2% perf-stat.i.cycles-between-cache-misses
2.83 ± 6% +10.5% 3.13 ± 2% perf-stat.i.metric.K/sec
0.26 +0.0 0.27 perf-stat.overall.branch-miss-rate%
1.609e+08 -5.6% 1.518e+08 perf-stat.overall.path-length
19825311 +1.9% 20209803 perf-stat.ps.branch-misses
20712472 +4.4% 21614327 perf-stat.ps.cache-references
180352 ± 6% +10.5% 199276 ± 2% perf-stat.ps.context-switches
177.05 +5.8% 187.32 perf-stat.ps.cpu-migrations
47.47 -0.1 47.36 perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap
48.49 -0.1 48.40 perf-profile.calltrace.cycles-pp.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe
48.39 -0.1 48.30 perf-profile.calltrace.cycles-pp.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
47.36 -0.1 47.27 perf-profile.calltrace.cycles-pp.osq_lock.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.do_syscall_64
48.44 -0.1 48.35 perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.__vm_munmap.__x64_sys_munmap.do_syscall_64
48.34 -0.1 48.25 perf-profile.calltrace.cycles-pp.rwsem_down_write_slowpath.down_write_killable.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe
49.70 -0.1 49.62 perf-profile.calltrace.cycles-pp.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
49.70 -0.1 49.62 perf-profile.calltrace.cycles-pp.__vm_munmap.__x64_sys_munmap.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
49.71 -0.1 49.64 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__munmap
49.73 -0.1 49.66 perf-profile.calltrace.cycles-pp.__munmap
49.71 -0.1 49.64 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__munmap
49.11 -0.1 49.06 perf-profile.calltrace.cycles-pp.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
0.52 +0.0 0.54 perf-profile.calltrace.cycles-pp.vms_gather_munmap_vmas.do_vmi_align_munmap.do_vmi_munmap.__vm_munmap.__x64_sys_munmap
0.54 ± 2% +0.0 0.56 perf-profile.calltrace.cycles-pp.do_mmap.vm_mmap_pgoff.do_syscall_64.entry_SYSCALL_64_after_hwframe.__mmap
0.67 ± 3% +0.1 0.74 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
0.68 ± 3% +0.1 0.74 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
0.71 ± 3% +0.1 0.78 ± 2% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
0.99 ± 3% +0.1 1.10 ± 2% perf-profile.calltrace.cycles-pp.common_startup_64
0.97 ± 3% +0.1 1.08 ± 2% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
0.97 ± 3% +0.1 1.08 ± 2% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
0.97 ± 3% +0.1 1.08 ± 2% perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
94.85 -0.2 94.65 perf-profile.children.cycles-pp.osq_lock
96.81 -0.2 96.64 perf-profile.children.cycles-pp.rwsem_down_write_slowpath
96.87 -0.2 96.70 perf-profile.children.cycles-pp.down_write_killable
98.91 -0.1 98.80 perf-profile.children.cycles-pp.do_syscall_64
98.91 -0.1 98.80 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
49.70 -0.1 49.62 perf-profile.children.cycles-pp.__x64_sys_munmap
49.70 -0.1 49.62 perf-profile.children.cycles-pp.__vm_munmap
49.73 -0.1 49.66 perf-profile.children.cycles-pp.__munmap
0.05 +0.0 0.06 perf-profile.children.cycles-pp.wake_q_add
0.23 +0.0 0.24 perf-profile.children.cycles-pp.kmem_cache_free
0.15 ± 2% +0.0 0.16 perf-profile.children.cycles-pp.anon_vma_clone
0.07 +0.0 0.08 ± 4% perf-profile.children.cycles-pp.ttwu_queue_wakelist
0.07 ± 5% +0.0 0.08 ± 5% perf-profile.children.cycles-pp.pick_next_task_fair
0.08 ± 4% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.__pick_next_task
0.07 +0.0 0.08 ± 5% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
0.22 +0.0 0.24 ± 3% perf-profile.children.cycles-pp.vma_expand
0.09 ± 4% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.dequeue_task_fair
0.10 ± 3% +0.0 0.12 ± 3% perf-profile.children.cycles-pp.schedule_idle
0.08 ± 4% +0.0 0.10 ± 3% perf-profile.children.cycles-pp.dequeue_entity
0.09 ± 4% +0.0 0.11 ± 4% perf-profile.children.cycles-pp.try_to_block_task
0.45 +0.0 0.47 perf-profile.children.cycles-pp.__split_vma
0.08 ± 5% +0.0 0.10 ± 4% perf-profile.children.cycles-pp.dequeue_entities
0.16 ± 2% +0.0 0.18 ± 4% perf-profile.children.cycles-pp.commit_merge
0.16 +0.0 0.18 ± 5% perf-profile.children.cycles-pp.vma_complete
0.12 ± 7% +0.0 0.14 ± 3% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.25 +0.0 0.27 ± 3% perf-profile.children.cycles-pp.vma_merge_new_range
0.54 +0.0 0.56 perf-profile.children.cycles-pp.do_mmap
0.08 ± 5% +0.0 0.11 ± 4% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
0.05 ± 7% +0.0 0.08 ± 4% perf-profile.children.cycles-pp.update_curr
0.04 ± 44% +0.0 0.08 ± 6% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.41 +0.0 0.44 ± 2% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.39 +0.0 0.43 ± 2% perf-profile.children.cycles-pp.tick_nohz_handler
0.18 ± 4% +0.0 0.21 ± 3% perf-profile.children.cycles-pp.schedule
0.17 ± 3% +0.0 0.21 ± 2% perf-profile.children.cycles-pp.schedule_preempt_disabled
0.18 ± 2% +0.0 0.22 ± 2% perf-profile.children.cycles-pp.try_to_wake_up
0.18 ± 4% +0.0 0.22 ± 3% perf-profile.children.cycles-pp.wake_up_q
0.35 +0.0 0.39 ± 2% perf-profile.children.cycles-pp.update_process_times
0.24 ± 3% +0.0 0.28 ± 3% perf-profile.children.cycles-pp.sched_tick
0.25 ± 3% +0.0 0.29 ± 3% perf-profile.children.cycles-pp.rwsem_wake
0.07 +0.0 0.12 ± 3% perf-profile.children.cycles-pp._raw_spin_lock
0.28 ± 2% +0.0 0.33 ± 2% perf-profile.children.cycles-pp.__schedule
0.18 ± 3% +0.1 0.23 ± 5% perf-profile.children.cycles-pp.task_tick_fair
0.00 +0.1 0.06 perf-profile.children.cycles-pp.update_se
0.69 ± 3% +0.1 0.76 ± 2% perf-profile.children.cycles-pp.cpuidle_enter
0.69 ± 3% +0.1 0.76 ± 2% perf-profile.children.cycles-pp.cpuidle_enter_state
0.72 ± 3% +0.1 0.79 ± 2% perf-profile.children.cycles-pp.cpuidle_idle_call
0.36 ± 3% +0.1 0.44 ± 2% perf-profile.children.cycles-pp.intel_idle_irq
0.99 ± 3% +0.1 1.10 ± 2% perf-profile.children.cycles-pp.common_startup_64
0.99 ± 3% +0.1 1.10 ± 2% perf-profile.children.cycles-pp.cpu_startup_entry
0.99 ± 3% +0.1 1.10 ± 2% perf-profile.children.cycles-pp.do_idle
0.97 ± 3% +0.1 1.08 ± 2% perf-profile.children.cycles-pp.start_secondary
94.33 -0.2 94.10 perf-profile.self.cycles-pp.osq_lock
0.32 -0.0 0.29 ± 2% perf-profile.self.cycles-pp.rwsem_down_write_slowpath
0.07 ± 5% -0.0 0.05 perf-profile.self.cycles-pp.vms_complete_munmap_vmas
0.05 +0.0 0.06 perf-profile.self.cycles-pp.wake_q_add
0.06 +0.0 0.07 ± 5% perf-profile.self.cycles-pp._raw_spin_lock
0.06 ± 6% +0.0 0.08 ± 4% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
0.04 ± 44% +0.0 0.08 ± 6% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.34 ± 4% +0.1 0.42 perf-profile.self.cycles-pp.intel_idle_irqDisclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
K Prateek Nayak
reviewerRe: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
In reply to Tim ChenHello Tim,
On 10/11/2025 11:54 PM, Tim Chen wrote:
[..snip..]
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1179
Quoted 8 lines
static s64
update_se(struct rq *rq, struct sched_entity *se){kernel/sched/fair.c +1
-0u64 now =rq_clock_task (rq);@@ -1174,6 +1176,7 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)trace_sched_stat_runtime (running, delta_exec);account_group_exec_runtime (running, delta_exec); +account_mm_sched (rq, donor, delta_exec);
Shouldn't we attribute this to "rq->curr"/"running" since that is the task which is actually running on the CPU (with "rq->curr->mm" being the one that is being used on CPU) as opposed to the "donor" which is just providing the vruntime context?
Quoted 3 lines
kernel/sched/fair.c +0
-0/* cgroup time is always accounted against the donor */ cgroup_account_cputime (donor, delta_exec);
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 01/19] sched/fair: Add infrastructure for cache-aware load balancing
Hi Prateek,
Quoted 11 lines · K Prateek Nayak
Hello Tim,
On 10/11/2025 11:54 PM, Tim Chen wrote:
[..snip..]
Quoted 5 lines
static s64 update_se(struct rq *rq, struct sched_entity *se) { u64 now = rq_clock_task(rq);(quoted diff) +0
-0@@ -1174,6 +1176,7 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)
Quoted 9 lines
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1179
Quoted 3 lines
kernel/sched/fair.c +1
-0trace_sched_stat_runtime (running, delta_exec);account_group_exec_runtime (running, delta_exec); +account_mm_sched (rq, donor, delta_exec);Shouldn't we attribute this to "rq->curr"/"running" since that is the task which is actually running on the CPU (with "
rq->curr->mm" being the one that is being used on CPU) as opposed to the "donor" which is just providing the vruntime context?
This is a good point. I'm not quite familiar with proxy execution, but after studying commit aa4f74dfd42b ("sched: Fix runtime accounting w/ split exec & sched contexts"), it seems that if it is related to the raw running time, the duration should be accumulated to rq->curr, the
actual proxy task. If it is related to vruntime - which might be
directly tied to the task selection/duration strategy-we should accumulate the delta to rq->donor, whose context is being borrowed. Is this a convention, or did we encounter any issues before aa4f74dfd42b?
I think it makes sense to change to rq->curr. As in "[PATCH 16/19] sched/fair: Exclude processes with many threads from cache-aware scheduling," we use rq->curr to determine how many active threads the process has. We should also change rq->donor to rq->curr in this [PATCH 1/19] to keep them consistent.
thanks,
Chenyu
Quoted 4 lines
Quoted 3 lines
kernel/sched/fair.c +0
-0/* cgroup time is always accounted against the donor */ cgroup_account_cputime (donor, delta_exec);
PATCH 2/19 sched/fair: Record per-LLC utilization to guide cache-aware scheduling decisions
In reply to Tim ChenFrom: Chen Yu <[email protected]>
When a system becomes busy and a process’s preferred LLC is saturated with too many threads, tasks within that LLC migrate frequently. These in LLC migrations introduce latency and degrade performance. To avoid this, task aggregation should be suppressed when the preferred LLC is overloaded, which requires a metric to indicate LLC utilization.
Record per LLC utilization/cpu capacity during periodic load balancing. These statistics will be used in later patches to decide whether tasks should be aggregated into their preferred LLC.
--- include/linux/sched/topology.h | 4 ++ kernel/sched/fair.c | 73 ++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+)
include/linux/sched/topology.h +4 -0
@@ -77,6 +77,10 @@ struct sched_domain_shared {atomic_t nr_busy_cpus;int has_idle_cores;int nr_idle_scan; +#ifdef CONFIG_SCHED_CACHE +unsigned long util_avg; +unsigned long capacity____cacheline_aligned_in_smp ; +#endif };struct sched_domain {
kernel/sched/fair.c +73 -0
91 lines not on this page · raw message
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 02/19] sched/fair: Record per-LLC utilization to guide cache-aware scheduling decisions
Quoted 17 lines · Tim Chen
kernel/sched/fair.c +17
-0+static void record_sg_llc_stats (struct lb_env *env, +struct sg_lb_stats *sgs, +struct sched_group *group) +{ +/* +* Find the child domain on env->dst_cpu. This domain +* is either the domain that spans this group(if the +* group is a local group), or the sibling domain of +* this group. +*/ +struct sched_domain *sd = env->sd->child; +struct sched_domain_shared *sd_share; + +if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE ) +return ; + +/* only care about sched domains spanning a LLC */
kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10587
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (sd !=rcu_dereference (per_cpu (sd_llc, env->dst_cpu))) +return ; + +/* +* At this point we know this group spans a LLC domain. +* Record the statistic of this group in its corresponding +* shared LLC domain. +*/ + sd_share =rcu_dereference (per_cpu (sd_llc_shared, +cpumask_first (sched_group_span (group))));
Isn't this sd->shared ? Or did I loose the plot somewhere?
Quoted 9 lines
kernel/sched/fair.c +9
-0+if (!sd_share) +return ; + +if (READ_ONCE (sd_share->util_avg) != sgs->group_util) +WRITE_ONCE (sd_share->util_avg, sgs->group_util); + +if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity)) +WRITE_ONCE (sd_share->capacity, sgs->group_capacity); +}
Re: [PATCH 02/19] sched/fair: Record per-LLC utilization to guide cache-aware scheduling decisions
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Tim Chen
kernel/sched/fair.c +21
-0+static void record_sg_llc_stats (struct lb_env *env, +struct sg_lb_stats *sgs, +struct sched_group *group) +{ +/* +* Find the child domain on env->dst_cpu. This domain +* is either the domain that spans this group(if the +* group is a local group), or the sibling domain of +* this group. +*/ +struct sched_domain *sd = env->sd->child; +struct sched_domain_shared *sd_share; + +if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE ) +return ; + +/* only care about sched domains spanning a LLC */ +if (sd !=rcu_dereference (per_cpu (sd_llc, env->dst_cpu))) +return ; + +/*
Quoted 9 lines
kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10587
Quoted 6 lines
kernel/sched/fair.c +6
-0+* At this point we know this group spans a LLC domain. +* Record the statistic of this group in its corresponding +* shared LLC domain. +*/ + sd_share =rcu_dereference (per_cpu (sd_llc_shared, +cpumask_first (sched_group_span (group))));Isn't this
sd->shared? Or did I loose the plot somewhere?
The sd here refers to the domain that covers the local_group, which is derived from env->dst_cpu. Meanwhile, sd_share corresponds to the domain covering the 'group' that may be a sibling of the local_group. Our goal is to update the statistics of this latter 'group'. It is assumed that the local_group and its sibling 'group' have the same CPU weight, meaning they each cover one LLC. We check sd simply because it is the only domain we can obtain via env->sd->child (apologies, but I haven’t found a way to get the corresponding domain that spans the 'group' itself).
thanks,
Chenyu
K Prateek Nayak
reviewerRe: [PATCH 02/19] sched/fair: Record per-LLC utilization to guide cache-aware scheduling decisions
In reply to Tim ChenHello Tim,
Quoted 7 lines · Tim Chen
kernel/sched/fair.c +7
-0+#ifdef CONFIG_SCHED_CACHE +/* +* Record the statistics for this scheduler group for later +* use. These values guide load balancing on aggregating tasks +* to a LLC. +*/ +static void record_sg_llc_stats(struct lb_env *env,
kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10571
Quoted 10 lines
kernel/sched/fair.c +10
-0+ struct sg_lb_stats *sgs, + struct sched_group *group) +{ +/* +* Find the child domain on env->dst_cpu. This domain +* is either the domain that spans this group(if the +* group is a local group), or the sibling domain of +* this group. +*/ +struct sched_domain *sd = env->sd->child;
Was this intentionally done to limit the update to sg_llc_stats to the load balancing period of "sd_llc->parent"?
Can't this be done with update_idle_cpu_scan()? I believe it is more frequent, "sds->total_capacity" from caller gives you the equivalent of "group_capacity", and "group_util" is already calculated as "sum_util".
Checking "sd_llc->parent" there should be sufficient to check if there are multiple LLC domains or not. Thoughts?
Quoted 25 lines
kernel/sched/fair.c +25
-0+struct sched_domain_shared *sd_share; + +if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE ) +return ; + +/* only care about sched domains spanning a LLC */ +if (sd !=rcu_dereference (per_cpu (sd_llc, env->dst_cpu))) +return ; + +/* +* At this point we know this group spans a LLC domain. +* Record the statistic of this group in its corresponding +* shared LLC domain. +*/ + sd_share =rcu_dereference (per_cpu (sd_llc_shared, +cpumask_first (sched_group_span (group)))); +if (!sd_share) +return ; + +if (READ_ONCE (sd_share->util_avg) != sgs->group_util) +WRITE_ONCE (sd_share->util_avg, sgs->group_util); + +if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity)) +WRITE_ONCE (sd_share->capacity, sgs->group_capacity); +}
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 02/19] sched/fair: Record per-LLC utilization to guide cache-aware scheduling decisions
Hi Prateek,
Quoted 24 lines · K Prateek Nayak
Hello Tim,
Quoted 7 lines · Tim Chen
kernel/sched/fair.c +7
-0+#ifdef CONFIG_SCHED_CACHE +/* +* Record the statistics for this scheduler group for later +* use. These values guide load balancing on aggregating tasks +* to a LLC. +*/ +static void record_sg_llc_stats(struct lb_env *env,kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10571
Quoted 10 lines
kernel/sched/fair.c +10
-0+ struct sg_lb_stats *sgs, + struct sched_group *group) +{ +/* +* Find the child domain on env->dst_cpu. This domain +* is either the domain that spans this group(if the +* group is a local group), or the sibling domain of +* this group. +*/ +struct sched_domain *sd = env->sd->child;Was this intentionally done to limit the update to
sg_llc_statsto the load balancing period of "sd_llc->parent"?
Quoted 7 lines
Can't this be done with
update_idle_cpu_scan()? I believe it is more frequent, "sds->total_capacity" from caller gives you the equivalent of "group_capacity", and "group_util" is already calculated as "sum_util".Checking "
sd_llc->parent" there should be sufficient to check if there are multiple LLC domains or not. Thoughts?
The original idea was to calculate the statistics for the CPUs within one LLC, and set the tag for that sched group as well as its sg_lb_stats (but not at the sched domain scope). With this flag set in that sched group, we can perform some comparisons in update_sd_pick_busiest() to determine if that sched group has any tasks that need to be moved to other LLC sched groups. If we do this in update_idle_cpu_scan(), might it be a bit late for update_sd_pick_busiest()?
thanks,
Chenyu
Quoted 26 lines
Quoted 25 lines
kernel/sched/fair.c +25
-0+struct sched_domain_shared *sd_share; + +if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE ) +return ; + +/* only care about sched domains spanning a LLC */ +if (sd !=rcu_dereference (per_cpu (sd_llc, env->dst_cpu))) +return ; + +/* +* At this point we know this group spans a LLC domain. +* Record the statistic of this group in its corresponding +* shared LLC domain. +*/ + sd_share =rcu_dereference (per_cpu (sd_llc_shared, +cpumask_first (sched_group_span (group)))); +if (!sd_share) +return ; + +if (READ_ONCE (sd_share->util_avg) != sgs->group_util) +WRITE_ONCE (sd_share->util_avg, sgs->group_util); + +if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity)) +WRITE_ONCE (sd_share->capacity, sgs->group_capacity); +}
K Prateek Nayak
reviewerRe: [PATCH 02/19] sched/fair: Record per-LLC utilization to guide cache-aware scheduling decisions
Hello Chenyu,
Quoted 35 lines · Yu C
Hi Prateek,
Quoted 24 lines · K Prateek Nayak
Hello Tim,
Quoted 7 lines · Tim Chen
kernel/sched/fair.c +7
-0+#ifdef CONFIG_SCHED_CACHE +/* +* Record the statistics for this scheduler group for later +* use. These values guide load balancing on aggregating tasks +* to a LLC. +*/ +static void record_sg_llc_stats(struct lb_env *env,kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10571
Quoted 10 lines
kernel/sched/fair.c +10
-0+ struct sg_lb_stats *sgs, + struct sched_group *group) +{ + /* + * Find the child domain on env->dst_cpu. This domain + * is either the domain that spans this group(if the + * group is a local group), or the sibling domain of + * this group. + */ + struct sched_domain *sd = env->sd->child;Was this intentionally done to limit the update to
sg_llc_statsto the load balancing period of "sd_llc->parent"?Quoted 7 lines
Can't this be done with
update_idle_cpu_scan()? I believe it is more frequent, "sds->total_capacity" from caller gives you the equivalent of "group_capacity", and "group_util" is already calculated as "sum_util".Checking "
sd_llc->parent" there should be sufficient to check if there are multiple LLC domains or not. Thoughts?
Quoted 7 lines
The original idea was to calculate the statistics for the CPUs within one LLC, and set the tag for that sched group as well as its
sg_lb_stats(but not at the sched domain scope). With this flag set in that sched group, we can perform some comparisons inupdate_sd_pick_busiest()to determine if that sched group has any tasks that need to be moved to other LLC sched groups. If we do this inupdate_idle_cpu_scan(), might it be a bit late forupdate_sd_pick_busiest()?
Once I got to Patch 10, the location of record_sg_llc_stats() became more clear w.r.t. the following call to llc_balance(). Thank you for clarifying and sorry for the noise.
Signature · 2 lines
Thanks and Regards, Prateek
PATCH 3/19 sched/fair: Introduce helper functions to enforce LLC migration policy
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Cache-aware scheduling aggregates threads onto their preferred LLC, mainly through load balancing. When the preferred LLC becomes saturated, more threads are still placed there, increasing latency. A mechanism is needed to limit aggregation so that the preferred LLC does not become overloaded.
Introduce helper functions can_migrate_llc() and can_migrate_llc_task() to enforce the LLC migration policy:
1. Aggregate a task to its preferred LLC if both source and destination LLCs are not too busy (<50% utilization, tunable), or if doing so will not leave the preferred LLC much more imbalanced than the non-preferred one (>20% utilization difference, tunable, similar to imbalance_pct of the LLC domain).
2. Allow moving a task from overloaded preferred LLC to a non preferred LLC if this will not cause the non preferred LLC to become too imbalanced to cause a later migration back.
3. If both LLCs are too busy, let the generic load balance to spread the tasks.
This hysteresis prevents tasks from being migrated into and out of the preferred LLC frequently (back and forth): the threshold for migrating a task out of its preferred LLC is higher than that for migrating it into the LLC.
Since aggregation tends to make the preferred LLC busier than others, the imbalance tolerance is controlled by llc_imb_pct. If set to 0, tasks may still aggregate to the preferred LLC as long as it is not more utilized than the source LLC, preserving the preference.
--- kernel/sched/debug.c | 4 ++ kernel/sched/fair.c | 145 +++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 5 ++ 3 files changed, 154 insertions(+)
kernel/sched/debug.c +4 -0
@@ -523,6 +523,10 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +debugfs_create_u32 ("llc_overload_pct" ,0644 , debugfs_sched, &llc_overload_pct); +debugfs_create_u32 ("llc_imb_pct" ,0644 , debugfs_sched, &llc_imb_pct); +#endif debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();
kernel/sched/fair.c +145 -0
163 lines not on this page · raw message
kernel/sched/sched.h +5 -0
@@ -2797,6 +2797,11 @@ extern unsigned int sysctl_numa_balancing_scan_period_max;extern unsigned int sysctl_numa_balancing_scan_size;extern unsigned int sysctl_numa_balancing_hot_threshold; +#ifdef CONFIG_SCHED_CACHE +extern unsigned int llc_overload_pct; +extern unsigned int llc_imb_pct; +#endif +#ifdef CONFIG_SCHED_HRTICK /*
Signature · 1 line
2.32.0
PATCH 4/19 sched/fair: Introduce a static key to enable cache aware only for multi LLCs
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Enable cache-aware load balancing only if at least 1 NUMA node has more than one LLC.
--- kernel/sched/fair.c | 15 ++++++++++++--- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 14 ++++++++++++-- 3 files changed, 25 insertions(+), 5 deletions(-)
kernel/sched/fair.c +12 -3
@@ -1208,6 +1208,14 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_allowed); + +static inline bool sched_cache_enabled (void ) +{ +return sched_feat (SCHED_CACHE ) && +static_branch_likely (&sched_cache_allowed); +} +static int llc_id (int cpu) {if (cpu <0 )
@@ -1294,7 +1302,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_sched *pcpu_sched;unsigned long epoch; -if (!sched_feat (SCHED_CACHE )) +if (!sched_cache_enabled ())return ;if (p->sched_class != &fair_sched_class)
@@ -1330,7 +1338,7 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p)struct callback_head *work = &p->cache_work;struct mm_struct *mm = p->mm; -if (!sched_feat (SCHED_CACHE )) +if (!sched_cache_enabled ())return ;if (!mm || !mm->pcpu_sched)
@@ -10716,7 +10724,8 @@ static void record_sg_llc_stats(struct lb_env *env,struct sched_domain *sd = env->sd->child;struct sched_domain_shared *sd_share; -if (!sched_feat (SCHED_CACHE ) || env->idle ==CPU_NEWLY_IDLE ) +if (!sched_cache_enabled () || + env->idle ==CPU_NEWLY_IDLE )return ;/* only care about sched domains spanning a LLC */
kernel/sched/sched.h +1 -0
@@ -2800,6 +2800,7 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern struct static_key_false sched_cache_allowed;#endif #ifdef CONFIG_SCHED_HRTICK
kernel/sched/topology.c +12 -2
@@ -2444,6 +2444,7 @@ static int build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attr) { enum s_alloc alloc_state = sa_none; +bool has_multi_llcs = false;struct sched_domain *sd;struct s_data d;struct rq *rq =NULL ;
@@ -2530,10 +2531,12 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att* between LLCs and memory channels. */ nr_llcs = sd->span_weight / child->span_weight; -if (nr_llcs ==1 ) +if (nr_llcs ==1 ) { imb = sd->span_weight >>3 ; -else + }else { imb = nr_llcs; + has_multi_llcs = true; + } imb =max (1U , imb); sd->imb_numa_nr = imb;
@@ -2581,6 +2584,13 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (has_cluster)static_branch_inc_cpuslocked (&sched_cluster_active); +#ifdef CONFIG_SCHED_CACHE +if (has_multi_llcs) { +static_branch_enable_cpuslocked (&sched_cache_allowed); +pr_info ("Cache aware load balance enabled.\n" ); + } +#endif +if (rq && sched_debug_verbose)pr_info ("root domain span: %*pbl\n" ,cpumask_pr_args (cpu_map));
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
Quoted 20 lines · Tim Chen
From: Chen Yu <[email protected]>
Enable cache-aware load balancing only if at least 1 NUMA node has more than one LLC.
Suggested-by Libo Chen <[email protected]>Suggested-by Adam Li <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 15 ++++++++++++--- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 14 ++++++++++++-- 3 files changed, 25 insertions(+), 5 deletions(-)kernel/sched/fair.c +0
-0
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1217
Quoted 10 lines
kernel/sched/fair.c +7
-0@@ -1208,6 +1208,14 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_allowed); + +static inline bool sched_cache_enabled (void ) +{ +return sched_feat (SCHED_CACHE ) && +static_branch_likely (&sched_cache_allowed); +}
Urgh; do we really need _2_ static keys stacked for this? I'm thinking one should be well enough.
Re: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
Quoted 24 lines · Peter Zijlstra
Quoted 23 lines · Tim Chen
From: Chen Yu <[email protected]>
Enable cache-aware load balancing only if at least 1 NUMA node has more than one LLC.
Suggested-by Libo Chen <[email protected]>Suggested-by Adam Li <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 15 ++++++++++++--- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 14 ++++++++++++-- 3 files changed, 25 insertions(+), 5 deletions(-)kernel/sched/fair.c +0
-0@@ -1208,6 +1208,14 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ;
Quoted 10 lines
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1217
Quoted 7 lines
kernel/sched/fair.c +7
-0+DEFINE_STATIC_KEY_FALSE (sched_cache_allowed); + +static inline bool sched_cache_enabled (void ) +{ +return sched_feat (SCHED_CACHE ) && +static_branch_likely (&sched_cache_allowed); +}Urgh; do we really need _2_ static keys stacked for this? I'm thinking one should be well enough.
SCHED_CACHE allows user space to turn on/off the feature at runtime, while sched_cache_allow is a hardware capability. This capability is
disabled if there are no multiple LLCs within one node. I’m not sure
if using one key could support the above two scenarios.
thanks,
Chenyu
Re: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
Quoted 36 lines · Yu C
Quoted 26 lines · Peter Zijlstra
Quoted 25 lines · Tim Chen
From: Chen Yu <[email protected]>
Enable cache-aware load balancing only if at least 1 NUMA node has more than one LLC.
Suggested-by Libo Chen <[email protected]>Suggested-by Adam Li <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
kernel/sched/fair.c | 15 ++++++++++++---
kernel/sched/sched.h | 1 +
kernel/sched/topology.c| 14 ++++++++++++--3 files changed, 25 insertions(+), 5 deletions(-)kernel/sched/fair.c +2
-0@@ -1208,6 +1208,14 @@ static s64 update_se(struct rq *rq, struct sched_entity *se) __read_mostly unsigned int llc_overload_pct = 50; __read_mostly unsigned int llc_imb_pct = 20; +DEFINE_STATIC_KEY_FALSE (sched_cache_allowed); +Quoted 8 lines
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1217
Quoted 5 lines
kernel/sched/fair.c +5
-0+static inline bool sched_cache_enabled (void ) +{ + return sched_feat(SCHED_CACHE) && + static_branch_likely(&sched_cache_allowed); +}Urgh; do we really need _2_ static keys stacked for this? I'm thinking one should be well enough.
Quoted 2 lines
SCHED_CACHEallows user space to turn on/off the feature at runtime, whilesched_cache_allowis a hardware capability. This capability is
isn't it possible use only static_branch_likely(&sched_cache_allowed) at runtime?
Enable that key only if FEAT is set. Disable when unset. That way you could use only one static branch at runtime.
Also, I am not sure if the FEATURE should be true by default. I know it maybe unused but IMO it should be true by default only when its proven there are no regression. One should be aware of their topology to enable it.
Quoted 3 lines
disabled if there are no multiple LLCs within one node. I’m not sure
if using one key could support the above two scenarios.
It is possible to have multiple NUMA nodes. One node may have multiple LLC while other
one may have only 1 LLC. what happens in that case?
I am yet to go through the series(hopefully this week). Maybe its handled already.
Re: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
Quoted 39 lines · Shrikanth Hegde
Quoted 36 lines · Yu C
Quoted 34 lines · Peter Zijlstra
Quoted 20 lines · Tim Chen
From: Chen Yu <[email protected]>
Enable cache-aware load balancing only if at least 1 NUMA node has more than one LLC.
Suggested-by Libo Chen <[email protected]>Suggested-by Adam Li <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
kernel/sched/fair.c | 15 ++++++++++++---
kernel/sched/sched.h | 1 +
kernel/sched/topology.c| 14 ++++++++++++--3 files changed, 25 insertions(+), 5 deletions(-)kernel/sched/fair.c +0
-0kernel/sched/fair.c static s64 update_se(struct rq *rq, struct · line 1217
Quoted 10 lines
kernel/sched/fair.c +7
-0@@ -1208,6 +1208,14 @@ static s64 update_se(struct rq *rq, struct sched_entity *se) __read_mostly unsigned int llc_overload_pct = 50; __read_mostly unsigned int llc_imb_pct = 20; +DEFINE_STATIC_KEY_FALSE (sched_cache_allowed); + +static inline bool sched_cache_enabled (void ) +{ + return sched_feat(SCHED_CACHE) && + static_branch_likely(&sched_cache_allowed); +}Urgh; do we really need _2_ static keys stacked for this? I'm thinking one should be well enough.
Quoted 8 lines
Quoted 2 lines
SCHED_CACHEallows user space to turn on/off the feature at runtime, whilesched_cache_allowis a hardware capability. This capability isisn't it possible use only
static_branch_likely(&sched_cache_allowed)at runtime?Enable that key only if FEAT is set. Disable when unset. That way you could use only one static branch at runtime.
Oh, do you mean only using sched_cache_allowed in sched_cache_enabled()? I misunderstood that Peter suggested introducing only one key. I did not quite catch up, do you mean, we should monitor the switch of FEAT and modify sched_cache_allowed when needed, and the OS only queries the sched_cache_allowed at runtime? I'll take a deeper look tomorrow.
Oh, do you mean only using sched_cache_allowed in sched_cache_enabled()? I misunderstood that Peter suggested introducing only one key. But I didn't quite catch up - do you mean we should monitor the switch of FEAT, modify sched_cache_allowed when needed, and that the OS only queries sched_cache_allowed at runtime?
Quoted 2 lines
Also, I am not sure if the FEATURE should be true by default. I know it maybe unused but IMO it should be true by default only when its proven there are no regression.
Yes we tried very hard to not bring regressions during the past.
Quoted 6 lines
One should be aware of their topology to enable it.
Quoted 3 lines
disabled if there are no multiple LLCs within one node. I’m not sure if using one key could support the above two scenarios.
It is possible to have multiple NUMA nodes. One node may have multiple LLC while otherone may have only 1 LLC. what happens in that case?
In this case, it will be enabled, and the cache-aware load balancing will occur on that node with multiple LLCs. (Only a domain with SD_SHARE_LLC set, and whose parent domain does not have SD_SHARE_LLC set, will initiate the cache-aware load balancing.)
thanks,
Chenyu
Quoted 1 line
I am yet to go through the series(hopefully this week). Maybe its handled already.
Peter Zijlstra
maintainerRe: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
Quoted 6 lines · Yu C
Oh, do you mean only using
sched_cache_allowedinsched_cache_enabled()? I misunderstood that Peter suggested introducing only one key. But I didn't quite catch up - do you mean we should monitor the switch of FEAT, modifysched_cache_allowedwhen needed, and that the OS only queriessched_cache_allowed
at runtime?
Just don't use sched_feat(), add a debugfs file like numa_balancing and then combine the userspace and topology information when setting the one static_branch.
Re: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
Quoted 9 lines · Peter Zijlstra
Quoted 6 lines · Yu C
Oh, do you mean only using
sched_cache_allowedinsched_cache_enabled()? I misunderstood that Peter suggested introducing only one key. But I didn't quite catch up - do you mean we should monitor the switch of FEAT, modifysched_cache_allowedwhen needed, and that the OS only queriessched_cache_allowed
at runtime?
Quoted 3 lines
Just don't use
sched_feat(), add a debugfs file likenuma_balancingand then combine the userspace and topology information when setting the onestatic_branch.
Got it, will do like this.
thanks,
Chenyu
Peter Zijlstra
maintainerRe: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
In reply to Chen, Yu CQuoted 35 lines · Yu C
Quoted 25 lines · Peter Zijlstra
Quoted 24 lines · Tim Chen
From: Chen Yu <[email protected]>
Enable cache-aware load balancing only if at least 1 NUMA node has more than one LLC.
Suggested-by Libo Chen <[email protected]>Suggested-by Adam Li <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 15 ++++++++++++--- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 14 ++++++++++++-- 3 files changed, 25 insertions(+), 5 deletions(-)kernel/sched/fair.c +2
-0@@ -1208,6 +1208,14 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_allowed); +Quoted 8 lines
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1217
Quoted 5 lines
kernel/sched/fair.c +5
-0+static inline bool sched_cache_enabled (void ) +{ +return sched_feat (SCHED_CACHE ) && +static_branch_likely (&sched_cache_allowed); +}Urgh; do we really need _2_ static keys stacked for this? I'm thinking one should be well enough.
Quoted 4 lines
SCHED_CACHEallows user space to turn on/off the feature at runtime, whilesched_cache_allowis a hardware capability. This capability is
disabled if there are no multiple LLCs within one node. I’m not sure
if using one key could support the above two scenarios.
Of course it can! There is one decision 'is cache aware crap enabled', you only need one branch for that. You just need to make the code that sets the branch state a little more complicated.
Things like sysctl/kernel/sched_schedstats sysctl/kernel/sched_energy_aware debugfs/sched/numa_balancing all work like this.
Take the sched_energy_aware one, that is very similar; it will only enable if the topology supports energy aware stuff. But then still lets userspace disable it.
Same for this. The static branch condition should be:
'topology-has-multi-llc && userspace-wants-it'.
K Prateek Nayak
reviewerRe: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
In reply to Tim ChenHello Tim,
kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2538
Quoted 11 lines · Tim Chen
kernel/sched/topology.c +3
-2@@ -2530,10 +2531,12 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att* between LLCs and memory channels. */ nr_llcs = sd->span_weight / child->span_weight; -if (nr_llcs ==1 ) +if (nr_llcs ==1 ) { imb = sd->span_weight >>3 ; -else + }else { imb = nr_llcs; + has_multi_llcs = true;
One caution: this will not hold if all the CPUs aren't online during boot. One case I can think of is when the kernel is booted with "maxcpus" cmdline and CPUs are hotplugged later.
Unfortunately, I don't think we even have the raw topology data from the arch/ side under such scenario to accurately make a call if the system contains single or multiple LLC :(
I'm not sure if it is feasible but assuming the task_work() cannot run if &sched_cache_allowed is false, can the fist instance of the task work for sched_cache do the necessary setup?
Quoted 18 lines
kernel/sched/topology.c +8
-0+ } imb =max (1U , imb); sd->imb_numa_nr = imb;@@ -2581,6 +2584,13 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (has_cluster)static_branch_inc_cpuslocked (&sched_cluster_active); +#ifdef CONFIG_SCHED_CACHE +if (has_multi_llcs) { +static_branch_enable_cpuslocked (&sched_cache_allowed); +pr_info ("Cache aware load balance enabled.\n" ); + } +#endif +if (rq && sched_debug_verbose)pr_info ("root domain span: %*pbl\n" ,cpumask_pr_args (cpu_map));
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
Hi Prateek,
Quoted 19 lines · K Prateek Nayak
Hello Tim,
kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2538
Quoted 11 lines · Tim Chen
kernel/sched/topology.c +3
-2@@ -2530,10 +2531,12 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att* between LLCs and memory channels. */ nr_llcs = sd->span_weight / child->span_weight; -if (nr_llcs ==1 ) +if (nr_llcs ==1 ) { imb = sd->span_weight >>3 ; -else + }else { imb = nr_llcs; + has_multi_llcs = true;One caution: this will not hold if all the CPUs aren't online during boot. One case I can think of is when the kernel is booted with "maxcpus" cmdline and CPUs are hotplugged later.
Quoted 8 lines
Unfortunately, I don't think we even have the raw topology data from the
arch/side under such scenario to accurately make a call if the system contains single or multiple LLC :(I'm not sure if it is feasible but assuming the
task_work()cannot run if&sched_cache_allowedis false, can the fist instance of the task work forsched_cachedo the necessary setup?
build_sched_domains() might get invoked to rebuild the corresponding sched domains during CPU hotplug via cpuset subsystem. So if the CPU gets online after bootup, we still have the chance to detect multiple LLCs I suppose?
I did a check on my VM:
root@ubuntu:/sys/devices/system/cpu# lscpu CPU(s): 32 On-line CPU(s) list: 0-7 root@ubuntu:/sys/devices/system/cpu# echo 1 > cpu31/online Tracing ... Hit Ctrl-C to end. ^C
@build_sched_domains[
build_sched_domains+5
partition_sched_domains+613
cpuset_update_active_cpus+838
sched_cpu_activate+272
cpuhp_invoke_callback+340
cpuhp_thread_fun+139
smpboot_thread_fn+238
kthread+249
ret_from_fork+193
ret_from_fork_asm+26]: 1
thanks,
Chenyu
Re: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
Quoted 26 lines · Yu C
Hi Prateek,
Quoted 23 lines · K Prateek Nayak
Hello Tim,
kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2538
Quoted 11 lines · Tim Chen
kernel/sched/topology.c +3
-2@@ -2530,10 +2531,12 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att* between LLCs and memory channels. */ nr_llcs = sd->span_weight / child->span_weight; -if (nr_llcs ==1 ) +if (nr_llcs ==1 ) { imb = sd->span_weight >>3 ; -else + }else { imb = nr_llcs; + has_multi_llcs = true;One caution: this will not hold if all the CPUs aren't online during boot. One case I can think of is when the kernel is booted with "maxcpus" cmdline and CPUs are hotplugged later.
Unfortunately, I don't think we even have the raw topology data from the
arch/side under such scenario to accurately make a call if the system contains single or multiple LLC :(
Quoted 8 lines
Quoted 4 lines
I'm not sure if it is feasible but assuming the
task_work()cannot run if&sched_cache_allowedis false, can the fist instance of the task work forsched_cachedo the necessary setup?
build_sched_domains()might get invoked to rebuild the corresponding sched domains during CPU hotplug via cpuset subsystem. So if the CPU gets online after bootup, we still have the chance to detect multiple LLCs I suppose?
The case Pratek brought up of adding CPUs and enabling SCHED_CACHE should be covered.
The trickier case is if we disable SCHED_CACHE when CPUs are offlined and multi_cpus becomes false. We'll need to clear out rq->nr_pref_llcs data and tasks' preferred LLC would need to be cleared. Or else the accounting could be skewed we bring CPU online later and again re-enable SCHED_CACHE. So far we haven't done that when we disable SCHED_CACHE from an enabled state.
Tim
Quoted 24 lines
I did a check on my VM:
root@ubuntu:/sys/devices/system/cpu# lscpu CPU(s): 32 On-line CPU(s) list: 0-7 root@ubuntu:/sys/devices/system/cpu# echo 1 > cpu31/online Tracing ... Hit Ctrl-C to end. ^C@build_sched_domains[
build_sched_domains+5 partition_sched_domains+613 cpuset_update_active_cpus+838 sched_cpu_activate+272 cpuhp_invoke_callback+340 cpuhp_thread_fun+139 smpboot_thread_fn+238 kthread+249 ret_from_fork+193 ret_from_fork_asm+26]: 1
thanks,
Chenyu
Re: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
Quoted 36 lines · Tim Chen
Quoted 26 lines · Yu C
Hi Prateek,
Quoted 23 lines · K Prateek Nayak
Hello Tim,
kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2538
Quoted 11 lines · Tim Chen
kernel/sched/topology.c +3
-2@@ -2530,10 +2531,12 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att* between LLCs and memory channels. */ nr_llcs = sd->span_weight / child->span_weight; -if (nr_llcs ==1 ) +if (nr_llcs ==1 ) { imb = sd->span_weight >>3 ; -else + }else { imb = nr_llcs; + has_multi_llcs = true;One caution: this will not hold if all the CPUs aren't online during boot. One case I can think of is when the kernel is booted with "maxcpus" cmdline and CPUs are hotplugged later.
Unfortunately, I don't think we even have the raw topology data from the
arch/side under such scenario to accurately make a call if the system contains single or multiple LLC :(Quoted 8 lines
Quoted 4 lines
I'm not sure if it is feasible but assuming the
task_work()cannot run if&sched_cache_allowedis false, can the fist instance of the task work forsched_cachedo the necessary setup?
build_sched_domains()might get invoked to rebuild the corresponding sched domains during CPU hotplug via cpuset subsystem. So if the CPU gets online after bootup, we still have the chance to detect multiple LLCs I suppose?
Quoted 7 lines
The case Pratek brought up of adding CPUs and enabling
SCHED_CACHEshould be covered.The trickier case is if we disable
SCHED_CACHEwhen CPUs are offlined andmulti_cpusbecomes false. We'll need to clear outrq->nr_pref_llcsdata and tasks' preferred LLC would need to be cleared. Or else the accounting could be skewed we bring CPU online later and again re-enableSCHED_CACHE.
To safely keep the data consistent, do we need to add hook when SCHED_CACHE is disabled at runtime, to clear all the stale data? In that way, another race condition between SCHED_CACHE enabling vs EQ/DQ might also be covered.
thanks,
Chenyu
Quoted 2 lines
So far we haven't done that when we disable
SCHED_CACHEfrom an enabled state.
K Prateek Nayak
reviewerRe: [PATCH 04/19] sched/fair: Introduce a static key to enable cache aware only for multi LLCs
In reply to Chen, Yu CHello Chenyu,
Quoted 3 lines · Yu C
build_sched_domains()might get invoked to rebuild the corresponding sched domains during CPU hotplug via cpuset subsystem. So if the CPU gets online after bootup, we still have the chance to detect multiple LLCs I suppose?
Ah yes! Thank you for confirming.
Signature · 2 lines
Thanks and Regards, Prateek
PATCH 5/19 sched/fair: Add LLC index mapping for CPUs
In reply to Tim ChenIntroduce an index mapping between CPUs and their LLCs. This provides a continuous per LLC index needed for cache-aware load balancing in later patches.
The existing per_cpu llc_id usually points to the first CPU of the LLC domain, which is sparse and unsuitable as an array index. Using llc_id directly would waste memory.
With the new mapping, CPUs in the same LLC share a continuous index:
per_cpu(llc_idx, CPU=0...15) = 0
per_cpu(llc_idx, CPU=16...31) = 1
per_cpu(llc_idx, CPU=32...47) = 2
...
The maximum number of LLCs is limited by CONFIG_NR_LLCS. If the number of LLCs available exceeds CONFIG_NR_LLCS, the cache aware load balance is disabled. To further save memory, this array could be converted to dynamic allocation in the future, or the LLC index could be made NUMA node-wide.
As mentioned by Adam, if there is no domain with SD_SHARE_LLC, the function update_llc_idx() should not be invoked to update the index; otherwise, it will generate an invalid index.
--- include/linux/threads.h | 10 +++++++++ init/Kconfig | 9 ++++++++ kernel/sched/fair.c | 11 ++++++++++ kernel/sched/sched.h | 2 ++ kernel/sched/topology.c | 47 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 79 insertions(+)
include/linux/threads.h +10 -0
@@ -20,6 +20,16 @@/* Places which use this should consider cpumask_var_t. */ #define NR_CPUS CONFIG_NR_CPUS +#ifndef CONFIG_NR_LLCS +#define CONFIG_NR_LLCS 1 +#endif + +#if CONFIG_NR_LLCS >NR_CPUS +#define NR_LLCS NR_CPUS +#else +#define NR_LLCS CONFIG_NR_LLCS +#endif +#define MIN_THREADS_LEFT_FOR_ROOT 4/*
init/Kconfig +9 -0
@@ -981,6 +981,15 @@ config SCHED_CACHE resources within the same cache domain, reducing cache misses and lowering data access latency. +config NR_LLCS + int "Maximum number of Last Level Caches" + range 2 1024 + depends on SMP && SCHED_CACHE + default 64 + help + This allows you to specify the maximum number of last level caches + this kernel will support for cache aware scheduling. + config NUMA_BALANCING_DEFAULT_ENABLED bool "Automatically enable NUMA aware memory/task placement" default y
kernel/sched/fair.c +11 -0
@@ -1224,6 +1224,17 @@ static int llc_id(int cpu)return per_cpu (sd_llc_id, cpu); } +/* +* continuous LLC index, starting from 0. +*/ +static inline int llc_idx (int cpu) +{ +if (cpu <0 ) +return -1 ; + +return per_cpu (sd_llc_idx, cpu); +} +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) {unsigned long epoch;
kernel/sched/sched.h +2 -0
@@ -2039,6 +2039,7 @@ static inline struct sched_domain *lowest_flag_domain(int cpu, int flag) DECLARE_PER_CPU(struct sched_domain __rcu *, sd_llc);DECLARE_PER_CPU (int, sd_llc_size);DECLARE_PER_CPU (int, sd_llc_id); +DECLARE_PER_CPU (int, sd_llc_idx);DECLARE_PER_CPU (int, sd_share_id); DECLARE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DECLARE_PER_CPU(struct sched_domain __rcu *, sd_numa);
@@ -2047,6 +2048,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);extern struct static_key_false sched_asym_cpucapacity;extern struct static_key_false sched_cluster_active; +extern int max_llcs; static __always_inline bool sched_asym_cpucap_active(void) {
kernel/sched/topology.c +47 -0
71 lines not on this page · raw message
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 05/19] sched/fair: Add LLC index mapping for CPUs
Quoted 1 line · Tim Chen
Introduce an index mapping between CPUs and their LLCs. This provides
"Introduce a *dense* mapping ...", since we already have a mapping, but as you explain below, that is sparse and not well suited for indexing.
Quoted 10 lines
a continuous per LLC index needed for cache-aware load balancing in later patches.
The maximum number of LLCs is limited by
CONFIG_NR_LLCS. If the number of LLCs available exceedsCONFIG_NR_LLCS, the cache aware load balance is disabled. To further save memory, this array could be converted to dynamic allocation in the future, or the LLC index could be made NUMA node-wide.
init/Kconfig config SCHED_CACHE · line 991
Quoted 8 lines
init/Kconfig +8
-0+config NR_LLCS + int "Maximum number of Last Level Caches" + range 2 1024 + depends on SMP && SCHED_CACHE + default 64 + help + This allows you to specify the maximum number of last level caches + this kernel will support for cache aware scheduling.
Not really a fan of this max thing. I suppose I'll see the use in the next few patches, but ideally we'd start with the dynamic solution.
Peter Zijlstra
maintainerRe: [PATCH 05/19] sched/fair: Add LLC index mapping for CPUs
In reply to Tim Chenkernel/sched/topology.c static void destroy_sched_domains(struct sched_domain *sd) · line 665
Quoted 12 lines · Tim Chen
kernel/sched/topology.c +1
-0@@ -659,6 +659,7 @@ static void destroy_sched_domains(struct sched_domain *sd) DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc);DEFINE_PER_CPU (int, sd_llc_size);DEFINE_PER_CPU (int, sd_llc_id); +DEFINE_PER_CPU (int, sd_llc_idx);DEFINE_PER_CPU (int, sd_share_id); DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa);
There is literally *ONE* user of sd_llc_id, cpus_share_cache(), surely that can equally use sd_llc_idx?
That is to say, do we really need two numbers for this?
Re: [PATCH 05/19] sched/fair: Add LLC index mapping for CPUs
Quoted 17 lines · Peter Zijlstra
kernel/sched/topology.c static void destroy_sched_domains(struct sched_domain *sd) · line 665
Quoted 12 lines · Tim Chen
kernel/sched/topology.c +1
-0@@ -659,6 +659,7 @@ static void destroy_sched_domains(struct sched_domain *sd) DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc);DEFINE_PER_CPU (int, sd_llc_size);DEFINE_PER_CPU (int, sd_llc_id); +DEFINE_PER_CPU (int, sd_llc_idx);DEFINE_PER_CPU (int, sd_share_id); DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa);There is literally *ONE* user of
sd_llc_id,cpus_share_cache(), surely that can equally usesd_llc_idx?
Quoted 1 line
That is to say, do we really need two numbers for this?
Okay, will look into removing sd_llc_id
Tim
PATCH 6/19 sched/fair: Assign preferred LLC ID to processes
In reply to Tim ChenWith cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar to numa_preferred_nid in NUMA balancing.
--- include/linux/sched.h | 1 + init/init_task.c | 3 +++ kernel/sched/fair.c | 7 +++++++ 3 files changed, 11 insertions(+)
include/linux/sched.h +1 -0
@@ -1402,6 +1402,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; +int preferred_llc;#endif #ifdef CONFIG_RSEQ
init/init_task.c +3 -0
@@ -188,6 +188,9 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .numa_group = NULL, .numa_faults = NULL,#endif +#ifdef CONFIG_SCHED_CACHE + .preferred_llc = -1, +#endif #if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS ) .kasan_depth = 1,#endif
kernel/sched/fair.c +7 -0
@@ -1312,6 +1312,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched;unsigned long epoch; +int mm_sched_llc =-1 ;if (!sched_cache_enabled ())return ;
@@ -1342,6 +1343,12 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; } + +if (mm->mm_sched_cpu !=-1 ) + mm_sched_llc =per_cpu (sd_llc_id, mm->mm_sched_cpu); + +if (p->preferred_llc != mm_sched_llc) + p->preferred_llc = mm_sched_llc; }static void task_tick_cache (struct rq *rq,struct task_struct *p)
Signature · 1 line
2.32.0
Re: [PATCH 06/19] sched/fair: Assign preferred LLC ID to processes
(Copied the question from Vern as the email seems to not reach LKML)
Quoted 9 lines · Tim Chen
Quoted 8 lines · vernhao
Tim Chen<[email protected]> wrote:
With cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar tonuma_preferred_nidin NUMA balancing.Signed-off-by Tim Chen <[email protected]>
[snip]
Quoted 28 lines
Quoted 10 lines
+
+ if (mm->mm_sched_cpu!= -1)
+mm_sched_llc=per_cpu(sd_llc_id, mm->mm_sched_cpu);In high-concurrency multi-threaded scenarios, not all threads handle same events, so their hot data in the LLC is not completely shared. Therefore, if every thread's preferred LLC is migrated to the LLC pointed to by
mm->mm_sched_cpu, this would lead to the incorrect assumption that all threads prefer the same LLC, thereby intensifying competition between LLCs.Yes, that's the reason why we stop aggregating to the preferred LLC once the the utilization of the
LLC becomes too high relative to the other LLCs.If you know your threads characteristics before hand on which of them share data together, you probably can use cgroup/cpuset from user space to separate out the threads.
There's not enough info from occupancy data for OS to group the threads by data sharing. Perhaps an alternative if NUMA balancing is on is to group tasks by their task numa group instead of by mm.
That would incur the page scanning overhead etc and make cache aware scheduling be dependent on NUMA balancing.
Quoted 1 line
Quoted 7 lines
Quoted 3 lines
So I'm wondering, why not move ‘mm->mm_sched_cpu’ to ‘task_struct’, so that each thread can individually track its preferred LLC? What are the losses in doing so?
You would need a way to group related tasks together and put them on the same LLC. Either group them by mm or some other means.
While Vern's use case is common in production environments, switching to per-task_struct prefer_llc might not aggregate the threads to dedicated LLCs. It is possible that each thread will stick to its old LLC because the thread was forked there and the occupancy is high on that old LLC. As a result, threads are randomly "pinned" to different LLCs.
The question becomes: how can we figure out the threads that share data? Can the kernel detect this, or get the hint from user space?
Yes, the numa_group in NUMA load balancing indicates that several tasks manipulate the same page, which could be an indicator. Besides, if task A frequently wakes up task B, does it mean A and B have the potential to share data? Furthermore, if task A wakes up B via a pipe, it might also indicate that A has something to share with B. I just wonder if we can introduce a structure to gather this information together.
thanks,
Chenyu
Peter Zijlstra
maintainerRe: [PATCH 06/19] sched/fair: Assign preferred LLC ID to processes
Quoted 2 lines · Yu C
The question becomes: how can we figure out the threads that share data? Can the kernel detect this, or get the hint from user space?
This needs the PMU, then you can steer using cache-miss ratios. But then people will hate us for using counters.
Quoted 7 lines
Yes, the
numa_groupin NUMA load balancing indicates that several tasks manipulate the same page, which could be an indicator. Besides, if task A frequently wakes up task B, does it mean A and B have the potential to share data? Furthermore, if task A wakes up B via a pipe, it might also indicate that A has something to share with B. I just wonder if we can introduce a structure to gather this information together.
The wakeup or pipe relation might be small relative to the working set. Consider a sharded in memory database, where the query comes in through the pipe/socket/wakeup. This query is small, but then it needs to go trawl through its memory to find the answer.
Something we *could* look at -- later -- is an interface to create thread groups, such that userspace that is clever enough can communicate this. But then there is the ago old question, will there be sufficient users to justify the maintenance of said interface.
Re: [PATCH 06/19] sched/fair: Assign preferred LLC ID to processes
Quoted 21 lines · Peter Zijlstra
Quoted 2 lines · Yu C
The question becomes: how can we figure out the threads that share data? Can the kernel detect this, or get the hint from user space?
This needs the PMU, then you can steer using cache-miss ratios. But then people will hate us for using counters.
Quoted 7 lines
Yes, the
numa_groupin NUMA load balancing indicates that several tasks manipulate the same page, which could be an indicator. Besides, if task A frequently wakes up task B, does it mean A and B have the potential to share data? Furthermore, if task A wakes up B via a pipe, it might also indicate that A has something to share with B. I just wonder if we can introduce a structure to gather this information together.The wakeup or pipe relation might be small relative to the working set. Consider a sharded in memory database, where the query comes in through the pipe/socket/wakeup. This query is small, but then it needs to go trawl through its memory to find the answer.
Quoted 4 lines
Something we *could* look at -- later -- is an interface to create thread groups, such that userspace that is clever enough can communicate this. But then there is the ago old question, will there be sufficient users to justify the maintenance of said interface.
OK, that could be an enhancement in the future if we encounter more use cases.
thanks,
Chenyu
Re: [PATCH 06/19] sched/fair: Assign preferred LLC ID to processes
In reply to Peter ZijlstraQuoted 21 lines · Peter Zijlstra
Quoted 2 lines · Yu C
The question becomes: how can we figure out the threads that share data? Can the kernel detect this, or get the hint from user space?
This needs the PMU, then you can steer using cache-miss ratios. But then people will hate us for using counters.
Quoted 7 lines
Yes, the
numa_groupin NUMA load balancing indicates that several tasks manipulate the same page, which could be an indicator. Besides, if task A frequently wakes up task B, does it mean A and B have the potential to share data? Furthermore, if task A wakes up B via a pipe, it might also indicate that A has something to share with B. I just wonder if we can introduce a structure to gather this information together.The wakeup or pipe relation might be small relative to the working set. Consider a sharded in memory database, where the query comes in through the pipe/socket/wakeup. This query is small, but then it needs to go trawl through its memory to find the answer.
Quoted 4 lines
Something we *could* look at -- later -- is an interface to create thread groups, such that userspace that is clever enough can communicate this. But then there is the ago old question, will there be sufficient users to justify the maintenance of said interface.
I did not intend to digress too far, but since this issue has been brought up, a wild guess came to me - could the "interface to create thread groups" here refer to something like the filesystem for memory cgroup
v2 thread mode? I just heard that some cloud users might split the threads of a single process into different thread groups, where threads within each group share data with one another (for example, when performing K-V hashing operations). Using cgroup for this purpose might be a bit overkill, though, considering that cgroup itself is designed for resource partitioning rather than identifying tasks sharing data. Meanwhile, the hierarchy of cgroup could also cause some overhead. If there were a single-layer thread partitioning mechanism - similar to the resctrl filesystem - wouldn’t that allow us to avoid modifying too much user business code while minimizing coupling with existing kernel components?
thanks,
Chenyu
Re: [PATCH 06/19] sched/fair: Assign preferred LLC ID to processes
Quoted 27 lines · Yu C
Quoted 21 lines · Peter Zijlstra
Quoted 2 lines · Yu C
The question becomes: how can we figure out the threads that share data? Can the kernel detect this, or get the hint from user space?
This needs the PMU, then you can steer using cache-miss ratios. But then people will hate us for using counters.
Quoted 7 lines
Yes, the
numa_groupin NUMA load balancing indicates that several tasks manipulate the same page, which could be an indicator. Besides, if task A frequently wakes up task B, does it mean A and B have the potential to share data? Furthermore, if task A wakes up B via a pipe, it might also indicate that A has something to share with B. I just wonder if we can introduce a structure to gather this information together.The wakeup or pipe relation might be small relative to the working set. Consider a sharded in memory database, where the query comes in through the pipe/socket/wakeup. This query is small, but then it needs to go trawl through its memory to find the answer.
Quoted 4 lines
Something we *could* look at -- later -- is an interface to create thread groups, such that userspace that is clever enough can communicate this. But then there is the ago old question, will there be sufficient users to justify the maintenance of said interface.
Quoted 7 lines
I did not intend to digress too far, but since this issue has been brought up, a wild guess came to me - could the "interface to create thread groups" here refer to something like the filesystem for memory cgroup v2 thread mode? I just heard that some cloud users might split the threads of a single process into different thread groups, where threads within each group share data with one another (for example, when performing K-V hashing operations).
Yes, in our internal business, we encountered similar issues. The actual scenario is on AMD virtual machines,
where businesses would spawn multiple concurrent threads, for example, around 900 threads, with over 600 threads
handling hash or key-value computations, more than 100 threads dealing with network transmission, and some others handling
background logging or monitoring. These threads do not share same hot L3 cache data. so concentrating these threads would only
exacerbate contention.
Can we differentiate these types of threads? It's obvious that the current configuration approach cannot meet the requirements
and will only cause more L3 cache race. Can we use cgroup or other methods, or configure through system calls to make
distinctions (the application may not be willing to modify the code) ?
Quoted 12 lines
Using cgroup for this purpose might be a bit overkill, though, considering that cgroup itself is designed for resource partitioning rather than identifying tasks sharing data. Meanwhile, the hierarchy of cgroup could also cause some overhead. If there were a single-layer thread partitioning mechanism - similar to the resctrl filesystem - wouldn’t that allow us to avoid modifying too much user business code while minimizing coupling with existing kernel components?
thanks,
Chenyu
PATCH 7/19 sched/fair: Track LLC-preferred tasks per runqueue
In reply to Tim ChenFor each runqueue, track the number of tasks with an LLC preference and how many of them are running on their preferred LLC. This mirrors nr_numa_running and nr_preferred_running for NUMA balancing, and will be used by cache-aware load balancing in later patches.
--- kernel/sched/core.c | 12 +++++++++++ kernel/sched/fair.c | 47 +++++++++++++++++++++++++++++++++++++++++++- kernel/sched/sched.h | 7 +++++++ 3 files changed, 65 insertions(+), 1 deletion(-)
kernel/sched/core.c +12 -0
@@ -529,6 +529,18 @@ void __trace_set_current_state(int state_value) }EXPORT_SYMBOL (__trace_set_current_state); +#ifdef CONFIG_SMP +int task_llc (const struct task_struct *p) +{ +return per_cpu (sd_llc_id,task_cpu (p)); +} +#else +int task_llc (const struct task_struct *p) +{ +return 0 ; +} +#endif +/* * Serialization rules: *
kernel/sched/fair.c +46 -1
93 lines not on this page · raw message
kernel/sched/sched.h +7 -0
@@ -1098,6 +1098,10 @@ struct rq {unsigned int nr_preferred_running;unsigned int numa_migrate_on;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc_running; +unsigned int nr_llc_running; +#endif #ifdef CONFIG_NO_HZ_COMMON unsigned long last_blocked_load_update_tick;unsigned int has_blocked_load;
@@ -1952,6 +1956,9 @@ init_numa_balancing(unsigned long clone_flags, struct task_struct *p)#endif /* !CONFIG_NUMA_BALANCING */ +void reset_llc_stats (struct rq *rq); +int task_llc (const struct task_struct *p); + static inline void queue_balance_callback(struct rq *rq, struct balance_callback *head,
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
kernel/sched/fair.c account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4042
Quoted 6 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks);
Here and...
kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4053
Quoted 7 lines
kernel/sched/fair.c +1
-0} cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));
... here, could you please check the compiler is doing CSE of task_of()?
kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4060
Quoted 7 lines
kernel/sched/fair.c +4
-0list_del_init (&se->group_node); } cfs_rq->nr_queued--; + +/* safeguard to clear the cache aware data */ +if (!parent_entity (se) && !cfs_rq->nr_queued) +reset_llc_stats (rq_of (cfs_rq));
I'm confused -- why?
Re: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
Quoted 10 lines · Peter Zijlstra
kernel/sched/fair.c account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4042
Quoted 6 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks);Here and...
Quoted 9 lines
kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4053
Quoted 7 lines
kernel/sched/fair.c +1
-0} cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));... here, could you please check the compiler is doing CSE of
task_of()?
Will consolidate those task_of(se).
Quoted 10 lines
kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4060
Quoted 8 lines
kernel/sched/fair.c +4
-0list_del_init (&se->group_node); } cfs_rq->nr_queued--; + +/* safeguard to clear the cache aware data */ +if (!parent_entity (se) && !cfs_rq->nr_queued) +reset_llc_stats (rq_of (cfs_rq));I'm confused -- why?
Was put here in early code development to make sure things would not go haywire. Will remove them. Probably some warning of having tasks preferring some LLC when there're no task in queue is more appropriate.
Tim
Peter Zijlstra
maintainerRe: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
Quoted 21 lines · Tim Chen
Quoted 10 lines · Peter Zijlstra
kernel/sched/fair.c account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4042
Quoted 6 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks);Here and...
Quoted 9 lines
kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4053
Quoted 7 lines
kernel/sched/fair.c +1
-0} cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));... here, could you please check the compiler is doing CSE of
task_of()?
Quoted 1 line
Will consolidate those
task_of(se).
And rq_of(). But really, check code-gen, it *should* DTRT and CSE the lot. If it doesn't, then do it manually.
Re: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Tim Chen
Quoted 19 lines · Peter Zijlstra
kernel/sched/fair.c account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4042
Quoted 6 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks);Here and...
kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4053
Quoted 7 lines
kernel/sched/fair.c +1
-0} cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));... here, could you please check the compiler is doing CSE of
task_of()?
Quoted 4 lines
Quoted 1 line
Will consolidate those
task_of(se).And
rq_of(). But really, check code-gen, it *should* DTRT and CSE the lot. If it doesn't, then do it manually.
Looking at the assembly dump, it does look like compiler is doing the right thing and not computing those twice.
Tim
K Prateek Nayak
reviewerRe: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
In reply to Tim ChenHello Tim,
Quoted 10 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks); } cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);
kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4060
Quoted 10 lines
kernel/sched/fair.c +5
-0if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));list_del_init (&se->group_node); } cfs_rq->nr_queued--; + +/* safeguard to clear the cache aware data */ +if (!parent_entity (se) && !cfs_rq->nr_queued) +reset_llc_stats (rq_of (cfs_rq));
Instead of relying on reset_llc_stats() hack, I think a better approach would be to have a "p->se.llc_sched_active" flag similar to how uclamp has "uc_se->active" and we set this in account_llc_enqueue() which will still check for sched_cache_enabled() but account_llc_dequeue() would only check for "p->se.llc_sched_active" to decrement the stats and then unset the flag.
That way, we cannot have an imbalanced accounting. Thoughts?
Quoted 2 lines
(quoted diff) +0
-0}
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
Quoted 31 lines · K Prateek Nayak
Hello Tim,
Quoted 10 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks); } cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4060
Quoted 10 lines
kernel/sched/fair.c +5
-0if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));list_del_init (&se->group_node); } cfs_rq->nr_queued--; + +/* safeguard to clear the cache aware data */ +if (!parent_entity (se) && !cfs_rq->nr_queued) +reset_llc_stats (rq_of (cfs_rq));Instead of relying on
reset_llc_stats()hack, I think a better approach would be to have a "p->se.llc_sched_active" flag similar to how uclamp has "uc_se->active" and we set this inaccount_llc_enqueue()which will still check forsched_cache_enabled()butaccount_llc_dequeue()would only check for "p->se.llc_sched_active" to decrement the stats and then unset the flag.
Quoted 2 lines
That way, we cannot have an imbalanced accounting. Thoughts?
I suppose what you mean is to avoid the race condition between enabling sched_cache and EQ/DE_LLC, similar to uclamp:
enqueue(taskA)
// sched_cache gets enabled
enqueue(taskB)
dequeue(taskA)
// Must not decrement rq->llc_pref for taskA
dequeue(taskB)We'll think more about this.
thanks,
Chenyu
Re: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
Quoted 35 lines · Yu C
Quoted 31 lines · K Prateek Nayak
Hello Tim,
Quoted 10 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks); } cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4060
Quoted 10 lines
kernel/sched/fair.c +5
-0if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));list_del_init (&se->group_node); } cfs_rq->nr_queued--; + +/* safeguard to clear the cache aware data */ +if (!parent_entity (se) && !cfs_rq->nr_queued) +reset_llc_stats (rq_of (cfs_rq));Instead of relying on
reset_llc_stats()hack, I think a better approach would be to have a "p->se.llc_sched_active" flag similar to how uclamp has "uc_se->active" and we set this inaccount_llc_enqueue()which will still check forsched_cache_enabled()butaccount_llc_dequeue()would only check for "p->se.llc_sched_active" to decrement the stats and then unset the flag.Quoted 2 lines
That way, we cannot have an imbalanced accounting. Thoughts?
Quoted 8 lines
I suppose what you mean is to avoid the race condition between enabling
sched_cacheand EQ/DE_LLC, similar to uclamp:enqueue(taskA) // sched_cache gets enabled enqueue(taskB) dequeue(taskA) // Must not decrement rq->llc_pref for taskA
For this case, task A is already on rq when sched cache get enabled. But task A's preferred_llc is still -1.
If we dequeue it while its preferred_llc is still -1, it won't affect rq->llc_pref.
If we change its preferred_llc to llc_i before we dequeue it, then rq->llc_pref[llc_i] will be incremented first.
Then when we dequeue task A, we will decrement it. We are still accounting rq->llc_pref[llc_i] correctly with current code.
The trickier case is if we need to dynamically resize rq->llc_pref[]. We need to make sure that we lock the rq to prevent enqueue/dequeue, switch it to a larger size rq->llc_pref[], copy the old data over, then switch over to the larger sized rq->llc_pref[] and unlock rq to keep the accounting straight.
Tim
Quoted 10 lines
dequeue(taskB)
We'll think more about this.
thanks,
Chenyu
K Prateek Nayak
reviewerRe: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
Hello Tim,
Quoted 51 lines · Tim Chen
Quoted 35 lines · Yu C
Quoted 31 lines · K Prateek Nayak
Hello Tim,
Quoted 10 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks); } cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4060
Quoted 10 lines
kernel/sched/fair.c +5
-0if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));list_del_init (&se->group_node); } cfs_rq->nr_queued--; + +/* safeguard to clear the cache aware data */ +if (!parent_entity (se) && !cfs_rq->nr_queued) +reset_llc_stats (rq_of (cfs_rq));Instead of relying on
reset_llc_stats()hack, I think a better approach would be to have a "p->se.llc_sched_active" flag similar to how uclamp has "uc_se->active" and we set this inaccount_llc_enqueue()which will still check forsched_cache_enabled()butaccount_llc_dequeue()would only check for "p->se.llc_sched_active" to decrement the stats and then unset the flag.Quoted 2 lines
That way, we cannot have an imbalanced accounting. Thoughts?
Quoted 8 lines
I suppose what you mean is to avoid the race condition between enabling
sched_cacheand EQ/DE_LLC, similar to uclamp:enqueue(taskA) // sched_cache gets enabled enqueue(taskB) dequeue(taskA) // Must not decrement rq->llc_pref for taskAFor this case, task A is already on rq when sched cache get enabled. But task A's
preferred_llcis still -1.If we dequeue it while its
preferred_llcis still -1, it won't affectrq->llc_pref.
Quoted 6 lines
If we change its
preferred_llctollc_ibefore we dequeue it, then rq->llc_pref[llc_i] will be incremented first.Then when we dequeue task A, we will decrement it. We are still accounting rq->llc_pref[llc_i] correctly with current code.
So what I really disliked was having reset_llc_stats() to reset the stat but looking at it again, that too is guarded by sched_cache_enabled() counter so I think the counters can still go out of balance if:
/* Cache aware scheduling enabled */
enqueue(TaskA) /* nr_llc_running = 1 */
enqueue(TaskB) /* nr_llc_running = 2 */
enqueue(TaskC) /* nr_llc_running = 3 */
dequeue(TaskA) /* nr_llc_running = 2 */
/* Cache aware scheduling disabled */ dequeue(TaskB) /* nr_llc_running = 2 */
dequeue(TaskC) /* nr_llc_running = 2 */
/* nr_running == 0; nr_llc_running = 2 */
/* Cache aware scheduling enabled again */
enqueue(TaskD) /* nr_llc_running = 3 */
enqueue(TaskE) /* nr_llc_running = 4 */
...
At some later point if nr_running reaches 0 again, then "nr_llc_running" is finally reset to 0 but until then it can show inaccurate value if users repeatedly toggle the feature depending on the workload running.
Quoted 7 lines
The trickier case is if we need to dynamically resize rq->llc_pref[]. We need to make sure that we lock the rq to prevent enqueue/dequeue, switch it to a larger size rq->llc_pref[], copy the old data over, then switch over to the larger sized rq->llc_pref[] and unlock rq to keep the accounting straight.
When that happens, we'll have to modify sched domains since something has changed in the system / cpuset and rq_attach_root() could be a good place to do it at when rq is offlined and onlined once again with the rq_lock held.
Only issue is if the partition splits the LLC in which case, we'll have two LLC index - one for for first half and another for the second half and we'll have to re-account the task to this new index.
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
Quoted 66 lines · K Prateek Nayak
Hello Tim,
Quoted 51 lines · Tim Chen
Quoted 35 lines · Yu C
Quoted 31 lines · K Prateek Nayak
Hello Tim,
Quoted 10 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks); } cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4060
Quoted 10 lines
kernel/sched/fair.c +5
-0if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));list_del_init (&se->group_node); } cfs_rq->nr_queued--; + +/* safeguard to clear the cache aware data */ +if (!parent_entity (se) && !cfs_rq->nr_queued) +reset_llc_stats (rq_of (cfs_rq));Instead of relying on
reset_llc_stats()hack, I think a better approach would be to have a "p->se.llc_sched_active" flag similar to how uclamp has "uc_se->active" and we set this inaccount_llc_enqueue()which will still check forsched_cache_enabled()butaccount_llc_dequeue()would only check for "p->se.llc_sched_active" to decrement the stats and then unset the flag.Quoted 2 lines
That way, we cannot have an imbalanced accounting. Thoughts?
Quoted 8 lines
I suppose what you mean is to avoid the race condition between enabling
sched_cacheand EQ/DE_LLC, similar to uclamp:enqueue(taskA) // sched_cache gets enabled enqueue(taskB) dequeue(taskA) // Must not decrement rq->llc_pref for taskAFor this case, task A is already on rq when sched cache get enabled. But task A's
preferred_llcis still -1.If we dequeue it while its
preferred_llcis still -1, it won't affectrq->llc_pref.Quoted 6 lines
If we change its
preferred_llctollc_ibefore we dequeue it, then rq->llc_pref[llc_i] will be incremented first.Then when we dequeue task A, we will decrement it. We are still accounting rq->llc_pref[llc_i] correctly with current code.
So what I really disliked was having
reset_llc_stats()to reset the stat but looking at it again, that too is guarded bysched_cache_enabled()counter so I think the counters can still go out of balance if:
Quoted 9 lines
/* Cache aware scheduling enabled */
enqueue(TaskA) /* nr_llc_running = 1 */
enqueue(TaskB) /* nr_llc_running = 2 */
enqueue(TaskC) /* nr_llc_running = 3 */
dequeue(TaskA) /* nr_llc_running = 2 */
/* Cache aware scheduling disabled */
dequeue(TaskB) /* nr_llc_running = 2 */If we introduce the mechanism you suggested previously:
"enable p->llc_sched_active in account_llc_enqueue(), which will still check sched_cache_enabled(), but account_llc_dequeue() only checks p->llc_sched_active to decrement the stats. Then the above scenario might be covered: dequeue(TaskB) will decrease nr_llc_running even if cache aware is disabled. Another idea is to reset all CPU statistics when cache aware scheduling is disabled at runtime, this might also avoid several race conditions, for example cpu hotplug vs cache aware scheduling.
thanks,
Chenyu
K Prateek Nayak
reviewerRe: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
In reply to Chen, Yu CHello Chenyu,
Quoted 13 lines · Yu C
Quoted 9 lines
Instead of relying on
reset_llc_stats()hack, I think a better approach would be to have a "p->se.llc_sched_active" flag similar to how uclamp has "uc_se->active" and we set this inaccount_llc_enqueue()which will still check forsched_cache_enabled()butaccount_llc_dequeue()would only check for "p->se.llc_sched_active" to decrement the stats and then unset the flag.That way, we cannot have an imbalanced accounting. Thoughts?
I suppose what you mean is to avoid the race condition between enabling
sched_cacheand EQ/DE_LLC, similar to uclamp:
Quoted 6 lines
enqueue(taskA)
//sched_cachegets enabled enqueue(taskB)
dequeue(taskA)
// Must not decrementrq->llc_preffor taskA dequeue(taskB)
Yup! We can have
enqueue(p)
account_llc_enqueue(p)
if (sched_cache_enabled())
p->se.llc_sched_active = true;
rq->nr_llc_running += (p->preferred_llc != -1);
rq->nr_pref_llc_running += (p->preferred_llc == task_llc(p));...
dequeue(p)
account_llc_dequeue(p)
if (p->se.llc_sched_active)
rq->nr_llc_running -= (p->preferred_llc != -1);
rq->nr_pref_llc_running -= (p->preferred_llc == task_llc(p));We can also have a single bit for "llc_sched_active" in the task_struct next to the "sched_task_hot" bit instead of using the hole in sched_entity after "sched_delayed"
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 07/19] sched/fair: Track LLC-preferred tasks per runqueue
In reply to K Prateek NayakQuoted 31 lines · K Prateek Nayak
Hello Tim,
Quoted 10 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -3999,6 +4038,7 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)struct rq *rq =rq_of (cfs_rq);account_numa_enqueue (rq,task_of (se)); +account_llc_enqueue (rq,task_of (se));list_add (&se->group_node, &rq->cfs_tasks); } cfs_rq->nr_queued++;@@ -4010,9 +4050,14 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)update_load_sub (&cfs_rq->load, se->load.weight);kernel/sched/fair.c account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) · line 4060
Quoted 10 lines
kernel/sched/fair.c +5
-0if (entity_is_task (se)) {account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +account_llc_dequeue (rq_of (cfs_rq),task_of (se));list_del_init (&se->group_node); } cfs_rq->nr_queued--; + +/* safeguard to clear the cache aware data */ +if (!parent_entity (se) && !cfs_rq->nr_queued) +reset_llc_stats (rq_of (cfs_rq));Instead of relying on
reset_llc_stats()hack, I think a better approach would be to have a "p->se.llc_sched_active" flag similar to how uclamp has "uc_se->active" and we set this inaccount_llc_enqueue()which will still check forsched_cache_enabled()butaccount_llc_dequeue()would only check for "p->se.llc_sched_active" to decrement the stats and then unset the flag.
Quoted 1 line
That way, we cannot have an imbalanced accounting. Thoughts?
With our current accounting method, we should not have imbalanced accounting even if you turn on sched_cache after the scheduler has started running (see my reply to Chen Yu's follow up). That reset_llc_stats() hack
should not be needed as Peter pointed out.
We will change that check to warning under debug option.
Tim
Quoted 3 lines
Quoted 3 lines
(quoted diff) +0
-0}
PATCH 8/19 sched/fair: Introduce per runqueue task LLC preference counter
In reply to Tim ChenEach runqueue is assigned a static array where each element tracks the number of tasks preferring a given LLC, indexed from 0 to NR_LLCS.
For example, rq->nr_pref_llc[3] = 2 signifies that there are 2 tasks on this runqueue which prefer to run within LLC3 (indexed from 0 to NR_LLCS
The load balancer can use this information to identify busy runqueues and migrate tasks to their preferred LLC domains.
--- kernel/sched/fair.c | 35 +++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 2 files changed, 36 insertions(+)
kernel/sched/fair.c +35 -0
@@ -1235,22 +1235,51 @@ static inline int llc_idx(int cpu)return per_cpu (sd_llc_idx, cpu); } +static inline int pref_llc_idx (struct task_struct *p) +{ +return llc_idx (p->preferred_llc); +} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) { +int pref_llc; +if (!sched_cache_enabled ())return ; rq->nr_llc_running += (p->preferred_llc !=-1 ); rq->nr_pref_llc_running += (p->preferred_llc ==task_llc (p)); + +if (p->preferred_llc <0 ) +return ; + + pref_llc =pref_llc_idx (p); +if (pref_llc <0 ) +return ; + + ++rq->nr_pref_llc[pref_llc]; }static void account_llc_dequeue (struct rq *rq,struct task_struct *p) { +int pref_llc; +if (!sched_cache_enabled ())return ; rq->nr_llc_running -= (p->preferred_llc !=-1 ); rq->nr_pref_llc_running -= (p->preferred_llc ==task_llc (p)); + +if (p->preferred_llc <0 ) +return ; + + pref_llc =pref_llc_idx (p); +if (pref_llc <0 ) +return ; + +/* avoid negative counter */ +if (rq->nr_pref_llc[pref_llc] >0 ) + --rq->nr_pref_llc[pref_llc]; }void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)
@@ -1524,10 +1553,16 @@ void init_sched_mm(struct task_struct *p)void reset_llc_stats (struct rq *rq) { +int i =0 ; +if (!sched_cache_enabled ())return ; rq->nr_llc_running =0 ; + +for (i =0 ; i < max_llcs; ++i) + rq->nr_pref_llc[i] =0 ; + rq->nr_pref_llc_running =0 ; }
kernel/sched/sched.h +1 -0
@@ -1101,6 +1101,7 @@ struct rq {#ifdef CONFIG_SCHED_CACHE unsigned int nr_pref_llc_running;unsigned int nr_llc_running; +unsigned int nr_pref_llc[NR_LLCS ];#endif #ifdef CONFIG_NO_HZ_COMMON unsigned long last_blocked_load_update_tick;
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 08/19] sched/fair: Introduce per runqueue task LLC preference counter
Quoted 59 lines · Tim Chen
Each runqueue is assigned a static array where each element tracks the number of tasks preferring a given LLC, indexed from 0 to
NR_LLCS.For example, rq->nr_pref_llc[3] = 2 signifies that there are 2 tasks on this runqueue which prefer to run within LLC3 (indexed from 0 to
NR_LLCSThe load balancer can use this information to identify busy runqueues and migrate tasks to their preferred LLC domains.
Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 35 +++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 2 files changed, 36 insertions(+)kernel/sched/fair.c +19
-0@@ -1235,22 +1235,51 @@ static inline int llc_idx(int cpu)return per_cpu (sd_llc_idx, cpu); } +static inline int pref_llc_idx (struct task_struct *p) +{ +return llc_idx (p->preferred_llc); +} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) { +int pref_llc; +if (!sched_cache_enabled ())return ; rq->nr_llc_running += (p->preferred_llc !=-1 ); rq->nr_pref_llc_running += (p->preferred_llc ==task_llc (p)); + +if (p->preferred_llc <0 ) +return ; + + pref_llc =pref_llc_idx (p); +if (pref_llc <0 ) +return ; + + ++rq->nr_pref_llc[pref_llc]; }static void account_llc_dequeue (struct rq *rq,struct task_struct *p) { +int pref_llc; +if (!sched_cache_enabled ())return ; rq->nr_llc_running -= (p->preferred_llc !=-1 ); rq->nr_pref_llc_running -= (p->preferred_llc ==task_llc (p)); +
kernel/sched/fair.c static inline int llc_idx(int cpu) · line 1282
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (p->preferred_llc <0 ) +return ; + + pref_llc =pref_llc_idx (p); +if (pref_llc <0 ) +return ; + +/* avoid negative counter */ +if (rq->nr_pref_llc[pref_llc] >0 ) + --rq->nr_pref_llc[pref_llc];
How!? Also, please use post increment/decrement operators.
Quoted 9 lines
kernel/sched/fair.c +2
-0}void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)@@ -1524,10 +1553,16 @@ void init_sched_mm(struct task_struct *p)void reset_llc_stats (struct rq *rq) { +int i =0 ; +
kernel/sched/fair.c void init_sched_mm(struct task_struct *p) · line 1567
Quoted 10 lines
kernel/sched/fair.c +4
-0if (!sched_cache_enabled ())return ; rq->nr_llc_running =0 ; + +for (i =0 ; i < max_llcs; ++i) + rq->nr_pref_llc[i] =0 ; + rq->nr_pref_llc_running =0 ; }
Still don't understand why this thing exists..
kernel/sched/sched.h struct rq { · line 1104
Quoted 10 lines
kernel/sched/sched.h +1
-0@@ -1101,6 +1101,7 @@ struct rq {#ifdef CONFIG_SCHED_CACHE unsigned int nr_pref_llc_running;unsigned int nr_llc_running; +unsigned int nr_pref_llc[NR_LLCS ];
Gah, yeah, lets not do this. Just (re)alloc the thing on topology changes or something.
Re: [PATCH 08/19] sched/fair: Introduce per runqueue task LLC preference counter
Quoted 57 lines · Peter Zijlstra
Quoted 56 lines · Tim Chen
Each runqueue is assigned a static array where each element tracks the number of tasks preferring a given LLC, indexed from 0 to
NR_LLCS.For example, rq->nr_pref_llc[3] = 2 signifies that there are 2 tasks on this runqueue which prefer to run within LLC3 (indexed from 0 to
NR_LLCSThe load balancer can use this information to identify busy runqueues and migrate tasks to their preferred LLC domains.
Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 35 +++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 1 + 2 files changed, 36 insertions(+)kernel/sched/fair.c +18
-0@@ -1235,22 +1235,51 @@ static inline int llc_idx(int cpu)return per_cpu (sd_llc_idx, cpu); } +static inline int pref_llc_idx (struct task_struct *p) +{ +return llc_idx (p->preferred_llc); +} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) { +int pref_llc; +if (!sched_cache_enabled ())return ; rq->nr_llc_running += (p->preferred_llc !=-1 ); rq->nr_pref_llc_running += (p->preferred_llc ==task_llc (p)); + +if (p->preferred_llc <0 ) +return ; + + pref_llc =pref_llc_idx (p); +if (pref_llc <0 ) +return ; + + ++rq->nr_pref_llc[pref_llc]; }static void account_llc_dequeue (struct rq *rq,struct task_struct *p) { +int pref_llc; +if (!sched_cache_enabled ())return ;
Quoted 15 lines
kernel/sched/fair.c static inline int llc_idx(int cpu) · line 1282
Quoted 13 lines
kernel/sched/fair.c +11
-0rq->nr_llc_running -= (p->preferred_llc !=-1 ); rq->nr_pref_llc_running -= (p->preferred_llc ==task_llc (p)); + +if (p->preferred_llc <0 ) +return ; + + pref_llc =pref_llc_idx (p); +if (pref_llc <0 ) +return ; + +/* avoid negative counter */ +if (rq->nr_pref_llc[pref_llc] >0 ) + --rq->nr_pref_llc[pref_llc];How!? Also, please use post increment/decrement operators.
Will change the rq->nr_pref_llc[pref_llc] <= 0 to a warning instead, and update the decrement to post operator.
Quoted 13 lines
Quoted 13 lines
kernel/sched/fair.c +2
-0}void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)@@ -1524,10 +1553,16 @@ void init_sched_mm(struct task_struct *p)void reset_llc_stats (struct rq *rq) { +int i =0 ; +if (!sched_cache_enabled ())return ;
Quoted 9 lines
kernel/sched/fair.c void init_sched_mm(struct task_struct *p) · line 1567
Quoted 7 lines
kernel/sched/fair.c +4
-0rq->nr_llc_running =0 ; + +for (i =0 ; i < max_llcs; ++i) + rq->nr_pref_llc[i] =0 ; + rq->nr_pref_llc_running =0 ; }Still don't understand why this thing exists..
Will remove this or change this to a debug
warning for the case when rq has no fair task.
Quoted 14 lines
kernel/sched/sched.h struct rq { · line 1104
Quoted 11 lines
kernel/sched/sched.h +1
-0@@ -1101,6 +1101,7 @@ struct rq {#ifdef CONFIG_SCHED_CACHE unsigned int nr_pref_llc_running;unsigned int nr_llc_running; +unsigned int nr_pref_llc[NR_LLCS ];Gah, yeah, lets not do this. Just (re)alloc the thing on topology changes or something.
Will have to think about how to keep the tasks' preference consistent with nr_pref_llc with the new array. Perhaps make it size of NR_CPUS so we will allocate
once and don't have to resize and reallocate it, and fill it back up with the right data.
Tim
Peter Zijlstra
maintainerRe: [PATCH 08/19] sched/fair: Introduce per runqueue task LLC preference counter
Quoted 8 lines · Tim Chen
Quoted 5 lines
kernel/sched/fair.c static inline int llc_idx(int cpu) · line 1282
Quoted 3 lines
kernel/sched/fair.c +3
-0+/* avoid negative counter */ +if (rq->nr_pref_llc[pref_llc] >0 ) + --rq->nr_pref_llc[pref_llc];How!? Also, please use post increment/decrement operators.
Will change the rq->nr_pref_llc[pref_llc] <= 0 to a warning instead, and update the decrement to post operator.
That WARN will still add code. Note how none of the nr_*_running decrements have checks on. You fundamentally cannot remove a task from the runqueue that hasn't first been enqueued.
If you get mis-matches something is *very* busted.
Re: [PATCH 08/19] sched/fair: Introduce per runqueue task LLC preference counter
In reply to Tim ChenQuoted 6 lines · Tim Chen
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Tim Chen
Each runqueue is assigned a static array where each element tracks the number of tasks preferring a given LLC, indexed from 0 to
NR_LLCS.
[snip]
Quoted 15 lines
Quoted 14 lines
kernel/sched/sched.h struct rq { · line 1104
Quoted 11 lines
kernel/sched/sched.h +1
-0@@ -1101,6 +1101,7 @@ struct rq {#ifdef CONFIG_SCHED_CACHE unsigned int nr_pref_llc_running;unsigned int nr_llc_running; +unsigned int nr_pref_llc[NR_LLCS ];Gah, yeah, lets not do this. Just (re)alloc the thing on topology changes or something.
Quoted 7 lines
Will have to think about how to keep the tasks' preference consistent with
nr_pref_llcwith the new array. Perhaps make it size ofNR_CPUSso we will allocate
once and don't have to resize and reallocate it, and fill it back up with the right data.Tim
IIUC, what Peter meant is to dynamically allocate the array size based on the actual number of LLCs computed in build_sched_domains() or somesuch rather than statically allocating NR_LLCS which is 64 by default.
Making it NR_CPUS would be even larger and waste more memory on systems with few LLCs.
Thanks,
Madadi Vineeth Reddy
Re: [PATCH 08/19] sched/fair: Introduce per runqueue task LLC preference counter
Quoted 31 lines · Madadi Vineeth Reddy
Quoted 6 lines · Tim Chen
Quoted 5 lines · Peter Zijlstra
Quoted 4 lines · Tim Chen
Each runqueue is assigned a static array where each element tracks the number of tasks preferring a given LLC, indexed from 0 to
NR_LLCS.[snip]
Quoted 21 lines
Quoted 14 lines
kernel/sched/sched.h struct rq { · line 1104
Quoted 11 lines
kernel/sched/sched.h +1
-0@@ -1101,6 +1101,7 @@ struct rq {#ifdef CONFIG_SCHED_CACHE unsigned int nr_pref_llc_running;unsigned int nr_llc_running; +unsigned int nr_pref_llc[NR_LLCS ];Gah, yeah, lets not do this. Just (re)alloc the thing on topology changes or something.
Will have to think about how to keep the tasks' preference consistent with
nr_pref_llcwith the new array. Perhaps make it size ofNR_CPUSso we will allocate
once and don't have to resize and reallocate it, and fill it back up with the right data.
Quoted 6 lines
Quoted 1 line
Tim
IIUC, what Peter meant is to dynamically allocate the array size based on the actual number of LLCs computed in
build_sched_domains()or somesuch rather than statically allocatingNR_LLCSwhich is 64 by default.
OK, this might involve dynamic adjustment/data synchronization of the nr_pref_llc buffer during CPU hotplug, and we'll make some modifications to it.
thanks,
Chenyu
Quoted 5 lines
Making it
NR_CPUSwould be even larger and waste more memory on systems with few LLCs.Thanks,
Madadi Vineeth Reddy
PATCH 9/19 sched/fair: Count tasks prefering each LLC in a sched group
In reply to Tim ChenDuring LLC load balancing, tabulate the number of tasks on each runqueue that prefer a given destination LLC in a sched group.
For example, consider a system with 4 LLC sched groups (LLC0 to LLC3) balancing towards LLC3. LLC0 has 3 tasks preferring LLC3, LLC1 has 2, and LLC2 has 1. LLC0, having the most tasks preferring LLC3, is selected as the busiest source to pick tasks from.
Within a source LLC, the total number of tasks preferring a destination LLC is computed by summing counts across all CPUs in that runqueue. For instance, if LLC0 has CPU0 with 2 tasks and CPU1 with 1 task preferring LLC3, the total for LLC0 is 3.
These statistics allow the load balancer to choose tasks from source sched groups that best match their preferred LLCs.
--- kernel/sched/fair.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
kernel/sched/fair.c +11 -0
@@ -10399,6 +10399,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc[NR_LLCS ]; +#endif };/*
@@ -10891,6 +10894,14 @@ static inline void update_sg_lb_stats(struct lb_env *env,if (cpu_overutilized (i)) *sg_overutilized =1 ; +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +int j; + +for (j =0 ; j < max_llcs; ++j) + sgs->nr_pref_llc[j] += rq->nr_pref_llc[j]; + } +#endif /* * No need to call idle_cpu() if nr_running is not 0 */
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 09/19] sched/fair: Count tasks prefering each LLC in a sched group
Quoted 26 lines · Tim Chen
During LLC load balancing, tabulate the number of tasks on each runqueue that prefer a given destination LLC in a sched group.
For example, consider a system with 4 LLC sched groups (LLC0 to LLC3) balancing towards LLC3. LLC0 has 3 tasks preferring LLC3, LLC1 has 2, and LLC2 has 1. LLC0, having the most tasks preferring LLC3, is selected as the busiest source to pick tasks from.
Within a source LLC, the total number of tasks preferring a destination LLC is computed by summing counts across all CPUs in that runqueue. For instance, if LLC0 has CPU0 with 2 tasks and CPU1 with 1 task preferring LLC3, the total for LLC0 is 3.
These statistics allow the load balancer to choose tasks from source sched groups that best match their preferred LLCs.
Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 11 +++++++++++ 1 file changed, 11 insertions(+)kernel/sched/fair.c +0
-0
kernel/sched/fair.c struct sg_lb_stats { · line 10405
Quoted 7 lines
kernel/sched/fair.c +3
-0@@ -10399,6 +10399,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc[NR_LLCS ]; +#endif };
Hahahaha, no! We have this on-stack, this cannot be.
Re: [PATCH 09/19] sched/fair: Count tasks prefering each LLC in a sched group
Quoted 22 lines · Peter Zijlstra
Quoted 21 lines · Tim Chen
During LLC load balancing, tabulate the number of tasks on each runqueue that prefer a given destination LLC in a sched group.
For example, consider a system with 4 LLC sched groups (LLC0 to LLC3) balancing towards LLC3. LLC0 has 3 tasks preferring LLC3, LLC1 has 2, and LLC2 has 1. LLC0, having the most tasks preferring LLC3, is selected as the busiest source to pick tasks from.
Within a source LLC, the total number of tasks preferring a destination LLC is computed by summing counts across all CPUs in that runqueue. For instance, if LLC0 has CPU0 with 2 tasks and CPU1 with 1 task preferring LLC3, the total for LLC0 is 3.
These statistics allow the load balancer to choose tasks from source sched groups that best match their preferred LLCs.
Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 11 +++++++++++ 1 file changed, 11 insertions(+)
Quoted 14 lines
kernel/sched/fair.c struct sg_lb_stats { · line 10405
Quoted 12 lines
kernel/sched/fair.c +3
-0@@ -10399,6 +10399,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc[NR_LLCS ]; +#endif };Hahahaha, no! We have this on-stack, this cannot be.
Okay, will allocate it off stack.
Tim
Peter Zijlstra
maintainerRe: [PATCH 09/19] sched/fair: Count tasks prefering each LLC in a sched group
In reply to Tim Chenkernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 10902
Quoted 6 lines · Tim Chen
kernel/sched/fair.c +6
-0+#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +int j; + +for (j =0 ; j < max_llcs; ++j) + sgs->nr_pref_llc[j] += rq->nr_pref_llc[j];
We live in the year 2025 and have embraced c99, please write as:
for (int j = 0; j < max_llcs; j++)
Quoted 2 lines
kernel/sched/fair.c +2
-0+ } +#endif
Re: [PATCH 09/19] sched/fair: Count tasks prefering each LLC in a sched group
Quoted 11 lines · Peter Zijlstra
kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 10902
Quoted 6 lines · Tim Chen
kernel/sched/fair.c +6
-0+#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +int j; + +for (j =0 ; j < max_llcs; ++j) + sgs->nr_pref_llc[j] += rq->nr_pref_llc[j];We live in the year 2025 and have embraced c99, please write as:
Quoted 1 line
for (int j = 0; j < max_llcs; j++)
Will do.
Quoted 3 lines
Quoted 3 lines
kernel/sched/fair.c +2
-0+ } +#endif
K Prateek Nayak
reviewerRe: [PATCH 09/19] sched/fair: Count tasks prefering each LLC in a sched group
In reply to Tim ChenHello Tim,
kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 10904
Quoted 8 lines · Tim Chen
kernel/sched/fair.c +8
-0+#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +int j; + +for (j =0 ; j < max_llcs; ++j) + sgs->nr_pref_llc[j] += rq->nr_pref_llc[j]; + } +#endif
If I'm not mistaken, we only compare "sds->nr_pref_llc[llc_idx(env->dst_cpu)]"
and the destination LLC is always fixes. Do we need to aggregate the data for all the LLCs? Is a single "nr_pref_llc_dest" enough?
Quoted 3 lines
kernel/sched/fair.c +0
-0/* * No need to call idle_cpu() if nr_running is not 0 */
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 09/19] sched/fair: Count tasks prefering each LLC in a sched group
Quoted 16 lines · K Prateek Nayak
Hello Tim,
kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 10904
Quoted 8 lines · Tim Chen
kernel/sched/fair.c +8
-0+#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +int j; + +for (j =0 ; j < max_llcs; ++j) + sgs->nr_pref_llc[j] += rq->nr_pref_llc[j]; + } +#endif If I'm not mistaken, we only compare "sds->nr_pref_llc[llc_idx(env->dst_cpu)]"
and the destination LLC is always fixes. Do we need to aggregate the data for all the LLCs? Is a single "nr_pref_llc_dest" enough?
Yes. Only the nr_pref_llc entry corresponding to the destination LLC is going to be used later to find either the LLC or run queue to be chosen for balancing. We can skip accounting for the other LLCs and save some memory here.
Tim
Quoted 4 lines
Quoted 4 lines
kernel/sched/fair.c +0
-0/* * No need to call idle_cpu() if nr_running is not 0 */
PATCH 10/19 sched/fair: Prioritize tasks preferring destination LLC during balancing
In reply to Tim ChenDuring LLC load balancing, first check for tasks that prefer the destination LLC and balance them to it before others.
Mark source sched groups containing tasks preferring non local LLCs with the group_llc_balance flag. This ensures the load balancer later pulls or pushes these tasks toward their preferred LLCs.
--- kernel/sched/fair.c | 43 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-)
kernel/sched/fair.c +41 -2
@@ -9822,8 +9822,7 @@ static __maybe_unused enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu else return mig_unrestricted; -return can_migrate_llc (src_cpu, dst_cpu, - task_util(p), to_pref); + return can_migrate_llc(src_cpu, dst_cpu, task_util(p), to_pref); }#else
@@ -10394,6 +10393,7 @@ struct sg_lb_stats {enum group_type group_type;unsigned int group_asym_packing;/* Tasks should be moved to preferred CPU */ unsigned int group_smt_balance;/* Task on busy SMT be moved */ +unsigned int group_llc_balance;/* Tasks should be moved to preferred LLC */ unsigned long group_misfit_task_load;/* A CPU has a task too big for its capacity */ #ifdef CONFIG_NUMA_BALANCING unsigned int nr_numa_running;
@@ -10849,11 +10849,45 @@ static void record_sg_llc_stats(struct lb_env *env,if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity)) WRITE_ONCE(sd_share->capacity, sgs->group_capacity); } + +/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false; + +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC )) +return false; + + llc =llc_idx (env->dst_cpu); +if (sgs->nr_pref_llc[llc] >0 && +can_migrate_llc (env->src_cpu, env->dst_cpu,0 , true) == mig_llc) +return true; + +return false; +}#else static inline void record_sg_llc_stats (struct lb_env *env,struct sg_lb_stats *sgs,struct sched_group *group) { } + +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +return false; +}#endif /**
@@ -10954,6 +10988,11 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_type =group_classify (env->sd->imbalance_pct, group, sgs);record_sg_llc_stats (env, sgs, group); + +/* Check for tasks in this group can be moved to their preferred LLC */ +if (!local_group &&llc_balance (env, sgs, group)) + sgs->group_llc_balance =1 ; +/* Computing avg_load makes sense only when group is overloaded */ if (sgs->group_type == group_overloaded) sgs->avg_load = (sgs->group_load * SCHED_CAPACITY_SCALE) /
Signature · 1 line
2.32.0
kernel test robot
build botRe: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
kernel test robot noticed "UBSAN:array-index-out-of-bounds_in_kernel/sched/fair.c" on:
Hello,
kernel test robot noticed "UBSAN:array-index-out-of-bounds_in_kernel/sched/fair.c" on:
commit: a9872e774986636a909d5cc6f6bde8b44add9f33 ("[PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing")
url: https://github.com/intel-lab-lkp/linux/commits/Tim-Chen/sched-fair-Add-infrastructure-for-cache-aware-load-balancing/20251012-022248
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 45b7f780739a3145aeef24d2dfa02517a6c82ed6
patch link: https://lore.kernel.org/all/ca1946de63ad9f0ae99e079a74d70c55879cc0b6.1760206683.git.tim.c.chen@linux.intel.com/
patch subject: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
in testcase: boot
config: i386-randconfig-003-20251012
compiler: clang-20
test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 4G
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
[ 0.527179][ T2] ------------[ cut here ]------------ [ 0.527563][ T2] UBSAN: array-index-out-of-bounds in kernel/sched/fair.c:10965:6 [ 0.527563][ T2] index -1 is out of range for type 'unsigned int[8]' [ 0.527563][ T2] CPU: 0 UID: 0 PID: 2 Comm: kthreadd Not tainted 6.17.0-rc4-00027-ga9872e774986 #1 PREEMPTLAZY [ 0.527563][ T2] Call Trace: [ 0.527563][ T2] ? __dump_stack (lib/dump_stack.c:95) [ 0.527563][ T2] ? dump_stack_lvl (lib/dump_stack.c:123) [ 0.527563][ T2] ? ubsan_epilogue.llvm.14679606394994327670 (lib/dump_stack.c:129 lib/ubsan.c:233) [ 0.527563][ T2] ? __ubsan_handle_out_of_bounds (lib/ubsan.c:?) [ 0.527563][ T2] ? sched_balance_rq (kernel/sched/fair.c:?) [ 0.527563][ T2] ? sched_balance_newidle (kernel/sched/fair.c:13514) [ 0.527563][ T2] ? pick_next_task_fair (kernel/sched/fair.c:9328) [ 0.527563][ T2] ? dequeue_task (kernel/sched/core.c:2136) [ 0.527563][ T2] ? __pick_next_task (kernel/sched/core.c:5983) [ 0.527563][ T2] ? __schedule (kernel/sched/core.c:?) [ 0.527563][ T2] ? schedule (kernel/sched/core.c:7025 kernel/sched/core.c:7039) [ 0.527563][ T2] ? kthreadd (kernel/kthread.c:835) [ 0.527563][ T2] ? kthread_stop_put (kernel/kthread.c:818) [ 0.527563][ T2] ? ret_from_fork (arch/x86/kernel/process.c:154) [ 0.527563][ T2] ? kthread_stop_put (kernel/kthread.c:818) [ 0.527563][ T2] ? debug_smp_processor_id (lib/smp_processor_id.c:58) [ 0.527563][ T2] ? kthread_stop_put (kernel/kthread.c:818) [ 0.527563][ T2] ? ret_from_fork_asm (arch/x86/entry/entry_32.S:737) [ 0.527563][ T2] ? entry_INT80_32 (arch/x86/entry/entry_32.S:945) [ 0.527563][ T2] ---[ end trace ]---
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20251015/[email protected]
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Peter Zijlstra
maintainerRe: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
In reply to Tim ChenQuoted 11 lines · Tim Chen
During LLC load balancing, first check for tasks that prefer the destination LLC and balance them to it before others.
Mark source sched groups containing tasks preferring non local LLCs with the
group_llc_balanceflag. This ensures the load balancer later pulls or pushes these tasks toward their preferred LLCs.Co-developed-by Chen Yu <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
For me this patch is cut too fine; it only sets group_llc_balance but then we don't see how it is used.
Quoted 42 lines
kernel/sched/fair.c | 43 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-)kernel/sched/fair.c +15
-2@@ -9822,8 +9822,7 @@ static __maybe_unused enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu else return mig_unrestricted; -return can_migrate_llc (src_cpu, dst_cpu, - task_util(p), to_pref); + return can_migrate_llc(src_cpu, dst_cpu, task_util(p), to_pref); }#else @@ -10394,6 +10393,7 @@ struct sg_lb_stats {enum group_type group_type;unsigned int group_asym_packing;/* Tasks should be moved to preferred CPU */ unsigned int group_smt_balance;/* Task on busy SMT be moved */ +unsigned int group_llc_balance;/* Tasks should be moved to preferred LLC */ unsigned long group_misfit_task_load;/* A CPU has a task too big for its capacity */ #ifdef CONFIG_NUMA_BALANCING unsigned int nr_numa_running;@@ -10849,11 +10849,45 @@ static void record_sg_llc_stats(struct lb_env *env,if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity))WRITE_ONCE (sd_share->capacity, sgs->group_capacity); } + +/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false;
kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10874
Quoted 10 lines
kernel/sched/fair.c +10
-0+ +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC )) +return false; + + llc =llc_idx (env->dst_cpu); + if (sgs->nr_pref_llc[llc] > 0 &&
Nit: s/> 0// would be the same, right?
Quoted 20 lines
kernel/sched/fair.c +11
-0+ can_migrate_llc(env->src_cpu, env->dst_cpu, 0, true) == mig_llc) +return true; + + return false; +}#else static inline void record_sg_llc_stats (struct lb_env *env,struct sg_lb_stats *sgs,struct sched_group *group) { } + +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +return false; +}#endif /**
kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 10994
Quoted 7 lines
kernel/sched/fair.c +4
-0@@ -10954,6 +10988,11 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_type =group_classify (env->sd->imbalance_pct, group, sgs);record_sg_llc_stats (env, sgs, group); + +/* Check for tasks in this group can be moved to their preferred LLC */ +if (!local_group &&llc_balance (env, sgs, group)) + sgs->group_llc_balance =1 ;
We now have 3 (or so) branches that start with:
if (!local_group &&
perhaps collate that some?
Quoted 7 lines
kernel/sched/fair.c +1
-0+/* Computing avg_load makes sense only when group is overloaded */ if (sgs->group_type == group_overloaded) sgs->avg_load = (sgs->group_load * SCHED_CAPACITY_SCALE) /Signature · 1 line
2.32.0
Re: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Tim Chen
During LLC load balancing, first check for tasks that prefer the destination LLC and balance them to it before others.
Mark source sched groups containing tasks preferring non local LLCs with the
group_llc_balanceflag. This ensures the load balancer later pulls or pushes these tasks toward their preferred LLCs.
Quoted 7 lines
Quoted 4 lines
Co-developed-by Chen Yu <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
For me this patch is cut too fine; it only sets
group_llc_balancebut then we don't see how it is used.
Okay, will combine this patch with the following one.
Quoted 47 lines
Quoted 47 lines
kernel/sched/fair.c | 43 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-)kernel/sched/fair.c +19
-2@@ -9822,8 +9822,7 @@ static __maybe_unused enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu else return mig_unrestricted; -return can_migrate_llc (src_cpu, dst_cpu, - task_util(p), to_pref); + return can_migrate_llc(src_cpu, dst_cpu, task_util(p), to_pref); }#else @@ -10394,6 +10393,7 @@ struct sg_lb_stats {enum group_type group_type;unsigned int group_asym_packing;/* Tasks should be moved to preferred CPU */ unsigned int group_smt_balance;/* Task on busy SMT be moved */ +unsigned int group_llc_balance;/* Tasks should be moved to preferred LLC */ unsigned long group_misfit_task_load;/* A CPU has a task too big for its capacity */ #ifdef CONFIG_NUMA_BALANCING unsigned int nr_numa_running;@@ -10849,11 +10849,45 @@ static void record_sg_llc_stats(struct lb_env *env,if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity))WRITE_ONCE (sd_share->capacity, sgs->group_capacity); } + +/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false; + +if (env->sd->flags &SD_SHARE_LLC ) +return false; +
Quoted 8 lines
kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10874
Quoted 6 lines
kernel/sched/fair.c +6
-0+/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC )) +return false; + + llc =llc_idx (env->dst_cpu); + if (sgs->nr_pref_llc[llc] > 0 &&Nit: s/> 0// would be the same, right?
Sure.
Quoted 31 lines
Quoted 21 lines
kernel/sched/fair.c +11
-0+ can_migrate_llc(env->src_cpu, env->dst_cpu, 0, true) == mig_llc) +return true; + + return false; +}#else static inline void record_sg_llc_stats (struct lb_env *env,struct sg_lb_stats *sgs,struct sched_group *group) { } + +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +return false; +}#endif /** kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 10994
Quoted 7 lines
kernel/sched/fair.c +4
-0@@ -10954,6 +10988,11 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_type =group_classify (env->sd->imbalance_pct, group, sgs);record_sg_llc_stats (env, sgs, group); + +/* Check for tasks in this group can be moved to their preferred LLC */ +if (!local_group &&llc_balance (env, sgs, group)) + sgs->group_llc_balance =1 ;We now have 3 (or so) branches that start with:
Quoted 3 lines
if (!local_group &&perhaps collate that some?
Sure.
Quoted 8 lines
Quoted 8 lines
kernel/sched/fair.c +1
-0+/* Computing avg_load makes sense only when group is overloaded */ if (sgs->group_type == group_overloaded) sgs->avg_load = (sgs->group_load * SCHED_CAPACITY_SCALE) /Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
In reply to Tim ChenQuoted 8 lines · Tim Chen
kernel/sched/fair.c +8
-0+static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false;
kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10874
Quoted 10 lines
kernel/sched/fair.c +10
-0+ +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC )) +return false; + + llc =llc_idx (env->dst_cpu); + if (sgs->nr_pref_llc[llc] > 0 &&
Robot says llc can be -1 here, and it don't like doing out of bound array access.
Quoted 5 lines
kernel/sched/fair.c +5
-0+ can_migrate_llc(env->src_cpu, env->dst_cpu, 0, true) == mig_llc) +return true; + +return false; +}
Re: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
Quoted 15 lines · Peter Zijlstra
Quoted 13 lines · Tim Chen
kernel/sched/fair.c +13
-0+static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false; + +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */
Quoted 9 lines
kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10874
Quoted 5 lines
kernel/sched/fair.c +5
-0+if (child && !(child->flags &SD_SHARE_LLC )) +return false; + + llc =llc_idx (env->dst_cpu); + if (sgs->nr_pref_llc[llc] > 0 &&Robot says llc can be -1 here, and it don't like doing out of bound array access.
Humm, there seems to be a race condition during bootup with build_sched_domains(), where the per-cpu(sd_llc_idx) is reset to -1 at the beginning. We might need to add a sanity check here.
thanks,
Chenyu
Quoted 5 lines
Quoted 5 lines
kernel/sched/fair.c +5
-0+ can_migrate_llc(env->src_cpu, env->dst_cpu, 0, true) == mig_llc) +return true; + +return false; +}
Re: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
In reply to Tim ChenHi Tim,
Quoted 18 lines · Tim Chen
kernel/sched/fair.c +14
-0@@ -10849,11 +10849,45 @@ static void record_sg_llc_stats(struct lb_env *env,if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity))WRITE_ONCE (sd_share->capacity, sgs->group_capacity); } + +/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false; +
kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10875
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC )) +return false; + + llc =llc_idx (env->dst_cpu); +if (sgs->nr_pref_llc[llc] >0 && +can_migrate_llc (env->src_cpu, env->dst_cpu,0 , true) == mig_llc)
llc_balance() is called from update_sg_lb_stats() and at that time, env->src_cpu is not determined yet so should not be used here?
Quoted 4 lines
kernel/sched/fair.c +4
-0+return true; + +return false; +}
Re: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
Hi Aaron,
Quoted 26 lines · Aaron Lu
Hi Tim,
Quoted 23 lines · Tim Chen
kernel/sched/fair.c +19
-0@@ -10849,11 +10849,45 @@ static void record_sg_llc_stats(struct lb_env *env,if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity))WRITE_ONCE (sd_share->capacity, sgs->group_capacity); } + +/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false; + +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC ))
Quoted 9 lines
kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10875
Quoted 5 lines
kernel/sched/fair.c +5
-0+return false; + + llc =llc_idx (env->dst_cpu); +if (sgs->nr_pref_llc[llc] >0 && +can_migrate_llc (env->src_cpu, env->dst_cpu,0 , true) == mig_llc)
llc_balance()is called fromupdate_sg_lb_stats()and at that time,env->src_cpuis not determined yet so should not be used here?
You are right, I think we should check the candidate group's first CPU rather than the env->src_cpu. Will fix it in the next version. Thanks a lot!
chenyu
Quoted 4 lines
Quoted 4 lines
kernel/sched/fair.c +4
-0+return true; + +return false; +}
Re: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
Quoted 39 lines · Yu C
Hi Aaron,
Quoted 26 lines · Aaron Lu
Hi Tim,
Quoted 23 lines · Tim Chen
kernel/sched/fair.c +19
-0@@ -10849,11 +10849,45 @@ static void record_sg_llc_stats(struct lb_env *env,if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity))WRITE_ONCE (sd_share->capacity, sgs->group_capacity); } + +/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false; + +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC ))Quoted 9 lines
kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10875
Quoted 5 lines
kernel/sched/fair.c +5
-0+return false; + + llc =llc_idx (env->dst_cpu); +if (sgs->nr_pref_llc[llc] >0 && +can_migrate_llc (env->src_cpu, env->dst_cpu,0 , true) == mig_llc)
llc_balance()is called fromupdate_sg_lb_stats()and at that time,env->src_cpuis not determined yet so should not be used here?
Quoted 2 lines
You are right, I think we should check the candidate group's first CPU rather than the
env->src_cpu. Will fix it in the next version.
Looks like can_migrate_llc() doesn't care an exact cpu but any cpu in the same LLC should do, so either the candidate group's first cpu or any other cpus in that group should make no difference.
It might be more intuitive to prototype can_migrate_llc() with sd_shared as param than using cpu, just a thought.
Re: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
Quoted 40 lines · Aaron Lu
Quoted 39 lines · Yu C
Hi Aaron,
Quoted 35 lines · Aaron Lu
Hi Tim,
Quoted 18 lines · Tim Chen
kernel/sched/fair.c +14
-0@@ -10849,11 +10849,45 @@ static void record_sg_llc_stats(struct lb_env *env,if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity))WRITE_ONCE (sd_share->capacity, sgs->group_capacity); } + +/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false; +kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10875
Quoted 10 lines
kernel/sched/fair.c +10
-0+if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC )) +return false; + + llc =llc_idx (env->dst_cpu); +if (sgs->nr_pref_llc[llc] >0 && +can_migrate_llc (env->src_cpu, env->dst_cpu,0 , true) == mig_llc)
llc_balance()is called fromupdate_sg_lb_stats()and at that time,env->src_cpuis not determined yet so should not be used here?
Quoted 7 lines
Quoted 2 lines
You are right, I think we should check the candidate group's first CPU rather than the
env->src_cpu. Will fix it in the next version.Looks like
can_migrate_llc()doesn't care an exact cpu but any cpu in the same LLC should do, so either the candidate group's first cpu or any other cpus in that group should make no difference.
Yes, actually the cache aware is based on LLC rather than CPU. It was a historic reason that the original proposal was based on CPU in the wakeup path.
Quoted 2 lines
It might be more intuitive to prototype
can_migrate_llc()withsd_sharedas param than using cpu, just a thought.
I understand your concern, will think about this.
thanks,
Chenyu
K Prateek Nayak
reviewerRe: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
In reply to Tim ChenHello Tim,
Quoted 8 lines · Tim Chen
kernel/sched/fair.c +8
-0+/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child;
kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10870
Quoted 10 lines
kernel/sched/fair.c +10
-0+int llc; + +if (!sched_cache_enabled ()) +return false; + +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC ))
nit. You can just check group->flags here.
Quoted 24 lines
kernel/sched/fair.c +15
-0+return false; + + llc =llc_idx (env->dst_cpu); +if (sgs->nr_pref_llc[llc] >0 && +can_migrate_llc (env->src_cpu, env->dst_cpu,0 , true) == mig_llc) +return true; + + return false; +}#else static inline void record_sg_llc_stats (struct lb_env *env,struct sg_lb_stats *sgs,struct sched_group *group) { } + +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +return false; +}#endif /**
kernel/sched/fair.c static inline void update_sg_lb_stats(struct lb_env *env, · line 10990
Quoted 3 lines
kernel/sched/fair.c +0
-0@@ -10954,6 +10988,11 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_type =group_classify (env->sd->imbalance_pct, group, sgs);record_sg_llc_stats (env, sgs, group);
Okay, I see the intention of recording this based on group stats. Sorry for the noise on Patch 2.
Quoted 8 lines
kernel/sched/fair.c +5
-0+ +/* Check for tasks in this group can be moved to their preferred LLC */ +if (!local_group &&llc_balance (env, sgs, group)) + sgs->group_llc_balance =1 ; +/* Computing avg_load makes sense only when group is overloaded */ if (sgs->group_type == group_overloaded) sgs->avg_load = (sgs->group_load * SCHED_CAPACITY_SCALE) /
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 10/19] sched/fair: Prioritize tasks preferring destination LLC during balancing
Quoted 15 lines · K Prateek Nayak
Hello Tim,
Quoted 12 lines · Tim Chen
kernel/sched/fair.c +12
-0+/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain *child = env->sd->child; +int llc; + +if (!sched_cache_enabled ()) +return false;
Quoted 9 lines
kernel/sched/fair.c static void record_sg_llc_stats(struct lb_env *env, · line 10870
Quoted 6 lines
kernel/sched/fair.c +6
-0+ +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +/* only care about task migration among LLCs */ +if (child && !(child->flags &SD_SHARE_LLC ))nit. You can just check
group->flagshere.
Got it, we will simplify the code.
thanks,
Chenyu
PATCH 11/19 sched/fair: Identify busiest sched_group for LLC-aware load balancing
In reply to Tim ChenThe load balancer selects the busiest sched_group and migrates tasks to less busy groups to distribute load across CPUs.
With cache-aware scheduling enabled, the busiest sched_group is the one with most tasks preferring the destination LLC. If the group has the llc_balance flag set, cache aware load balancing is triggered.
Introduce the helper function update_llc_busiest() to identify the sched_group with the most tasks preferring the destination LLC.
--- kernel/sched/fair.c | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-)
kernel/sched/fair.c +38 -1
@@ -10877,6 +10877,23 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs, return false; } + +static bool update_llc_busiest (struct lb_env *env, +struct sg_lb_stats *busiest, +struct sg_lb_stats *sgs) +{ +int idx; + +/* Only the candidate with llc_balance needs to be taken care of */ +if (!sgs->group_llc_balance) +return false; + +/* +* There are more tasks that want to run on dst_cpu's LLC. +*/ + idx =llc_idx (env->dst_cpu); +return sgs->nr_pref_llc[idx] > busiest->nr_pref_llc[idx]; +}#else static inline void record_sg_llc_stats (struct lb_env *env,struct sg_lb_stats *sgs,struct sched_group *group)
@@ -10888,6 +10905,13 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs, {return false; } + +static bool update_llc_busiest (struct lb_env *env, +struct sg_lb_stats *busiest, +struct sg_lb_stats *sgs) +{ +return false; +}#endif /**
@@ -11035,6 +11059,17 @@ static bool update_sd_pick_busiest(struct lb_env *env, sds->local_stat.group_type != group_has_spare))return false; +/* deal with prefer LLC load balance, if failed, fall into normal load balance */ +if (update_llc_busiest (env, busiest, sgs)) +return true; + +/* +* If the busiest group has tasks with LLC preference, +* skip normal load balance. +*/ +if (busiest->group_llc_balance) +return false; +if (sgs->group_type > busiest->group_type)return true;
@@ -11942,9 +11977,11 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)/* * Try to move all excess tasks to a sibling domain of the busiest * group's child domain. +* Also do so if we can move some tasks that prefer the local LLC. */ if (sds.prefer_sibling && local->group_type == group_has_spare && -sibling_imbalance (env, &sds, busiest, local) >1 ) + (busiest->group_llc_balance || +sibling_imbalance (env, &sds, busiest, local) >1 )) goto force_balance;if (busiest->group_type != group_overloaded) {
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 11/19] sched/fair: Identify busiest sched_group for LLC-aware load balancing
Quoted 7 lines · Tim Chen
kernel/sched/fair.c +3
-0@@ -11035,6 +11059,17 @@ static bool update_sd_pick_busiest(struct lb_env *env, sds->local_stat.group_type != group_has_spare))return false; +/* deal with prefer LLC load balance, if failed, fall into normal load balance */ +if (update_llc_busiest (env, busiest, sgs)) +return true;
kernel/sched/fair.c static bool update_sd_pick_busiest(struct lb_env *env, · line 11074
Quoted 10 lines
kernel/sched/fair.c +8
-0+ +/* +* If the busiest group has tasks with LLC preference, +* skip normal load balance. +*/ +if (busiest->group_llc_balance) +return false; +if (sgs->group_type > busiest->group_type)return true;
This feels weird.. should we really override things like group_imbalance or group_misfit_task ?
Re: [PATCH 11/19] sched/fair: Identify busiest sched_group for LLC-aware load balancing
Quoted 6 lines · Peter Zijlstra
Quoted 4 lines · Tim Chen
kernel/sched/fair.c +0
-0@@ -11035,6 +11059,17 @@ static bool update_sd_pick_busiest(struct lb_env *env, sds->local_stat.group_type != group_has_spare))return false;
Quoted 16 lines
kernel/sched/fair.c static bool update_sd_pick_busiest(struct lb_env *env, · line 11074
Quoted 13 lines
kernel/sched/fair.c +11
-0+/* deal with prefer LLC load balance, if failed, fall into normal load balance */ +if (update_llc_busiest (env, busiest, sgs)) +return true; + +/* +* If the busiest group has tasks with LLC preference, +* skip normal load balance. +*/ +if (busiest->group_llc_balance) +return false; +if (sgs->group_type > busiest->group_type)return true;This feels weird.. should we really override things like
group_imbalanceorgroup_misfit_task?
Probably make sense to move the group_llc_balance priority to be lower than group_imblanace and group_misfit_task. Will update accordingly.
Tim
PATCH 12/19 sched/fair: Add migrate_llc_task migration type for cache-aware balancing
In reply to Tim ChenIntroduce a new migration type, migrate_llc_task, to support cache-aware load balancing.
After identifying the busiest sched_group (having the most tasks preferring the destination LLC), mark migrations with this type. During load balancing, each runqueue in the busiest sched_group is examined, and the runqueue with the highest number of tasks preferring the destination CPU is selected as the busiest runqueue.
--- kernel/sched/fair.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-)
kernel/sched/fair.c +31 -1
@@ -9504,7 +9504,8 @@ enum migration_type { migrate_load =0 , migrate_util, migrate_task, - migrate_misfit + migrate_misfit, + migrate_llc_task };#define LBF_ALL_PINNED 0x01
@@ -10082,6 +10083,10 @@ static int detach_tasks(struct lb_env *env) env->imbalance -= util;break ; +case migrate_llc_task: + env->imbalance--; +break ; +case migrate_task: env->imbalance--;break ;
@@ -11733,6 +11738,15 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *sreturn ; } +#ifdef CONFIG_SCHED_CACHE +if (busiest->group_llc_balance) { +/* Move a task that prefer local LLC */ + env->migration_type = migrate_llc_task; + env->imbalance =1 ; +return ; + } +#endif +if (busiest->group_type == group_imbalanced) {/* * In the group_imb case we cannot rely on group-wide averages
@@ -12041,6 +12055,10 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,struct rq *busiest =NULL , *rq;unsigned long busiest_util =0 , busiest_load =0 , busiest_capacity =1 ;unsigned int busiest_nr =0 ; +#ifdef CONFIG_SCHED_CACHE +unsigned int busiest_pref_llc =0 ; +int dst_llc; +#endif int i;for_each_cpu_and (i,sched_group_span (group), env->cpus) {
@@ -12149,6 +12167,16 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env, }break ; +case migrate_llc_task: +#ifdef CONFIG_SCHED_CACHE + dst_llc =llc_idx (env->dst_cpu); +if (!cpus_share_cache (env->dst_cpu, rq->cpu) && + busiest_pref_llc < rq->nr_pref_llc[dst_llc]) { + busiest_pref_llc = rq->nr_pref_llc[dst_llc]; + busiest = rq; + } +#endif +break ;case migrate_task:if (busiest_nr < nr_running) { busiest_nr = nr_running;
@@ -12331,6 +12359,8 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sdcase migrate_misfit:__schedstat_add (sd->lb_imbalance_misfit[idle], env->imbalance);break ; +case migrate_llc_task: +break ; } }
Signature · 1 line
2.32.0
K Prateek Nayak
reviewerRe: [PATCH 12/19] sched/fair: Add migrate_llc_task migration type for cache-aware balancing
Hello Tim,
kernel/sched/fair.c static struct rq *sched_balance_find_src_rq(struct lb_env *env, · line 12173
Quoted 8 lines · Tim Chen
kernel/sched/fair.c +4
-0@@ -12149,6 +12167,16 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env, }break ; +case migrate_llc_task: +#ifdef CONFIG_SCHED_CACHE + dst_llc =llc_idx (env->dst_cpu); + if (!cpus_share_cache(env->dst_cpu, rq->cpu) &&
Busiest group is always a non-local group right? Can cpus_share_cache() ever happen given we are looking at groups of first !SD_LLC_SHARE domain for "migrate_llc_task"?
Quoted 9 lines
kernel/sched/fair.c +6
-0+ busiest_pref_llc < rq->nr_pref_llc[dst_llc]) { + busiest_pref_llc = rq->nr_pref_llc[dst_llc]; + busiest = rq; + } +#endif +break ;case migrate_task:if (busiest_nr < nr_running) { busiest_nr = nr_running;
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 12/19] sched/fair: Add migrate_llc_task migration type for cache-aware balancing
Quoted 7 lines · K Prateek Nayak
Hello Tim,
Quoted 4 lines · Tim Chen
(quoted diff) +0
-0@@ -12149,6 +12167,16 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env, } break;
Quoted 8 lines
kernel/sched/fair.c static struct rq *sched_balance_find_src_rq(struct lb_env *env, · line 12173
Quoted 4 lines
kernel/sched/fair.c +4
-0+case migrate_llc_task: +#ifdef CONFIG_SCHED_CACHE + dst_llc =llc_idx (env->dst_cpu); + if (!cpus_share_cache(env->dst_cpu, rq->cpu) &&Busiest group is always a non-local group right? Can
cpus_share_cache()ever happen given we are looking at groups of first !SD_LLC_SHARE domain for "migrate_llc_task"?
That's a good point. We should have already checked that busiest and local are not in the same LLC when we mark migrate_llc_task. We shouldn't need to do cpus_share_cache() check here.
Tim
Quoted 10 lines
Quoted 10 lines
kernel/sched/fair.c +6
-0+ busiest_pref_llc < rq->nr_pref_llc[dst_llc]) { + busiest_pref_llc = rq->nr_pref_llc[dst_llc]; + busiest = rq; + } +#endif +break ;case migrate_task:if (busiest_nr < nr_running) { busiest_nr = nr_running;
PATCH 13/19 sched/fair: Handle moving single tasks to/from their preferred LLC
In reply to Tim ChenIf the busiest runqueue has only one task, active balancing may be invoked to move it. However, before migration, check whether the task is running on its preferred LLC.
Do not move a lone task to another LLC if it would move the task away from its preferred LLC or cause excessive imbalance between LLCs.
--- kernel/sched/fair.c | 62 ++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 59 insertions(+), 3 deletions(-)
kernel/sched/fair.c +59 -3
@@ -9826,12 +9826,53 @@ static __maybe_unused enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpureturn can_migrate_llc (src_cpu, dst_cpu,task_util (p), to_pref); } +static inline bool +break_llc_locality (struct lb_env *env) +{ +if (!sched_cache_enabled ()) +return false; + +if (cpus_share_cache (env->src_cpu, env->dst_cpu)) +return false; +/* +* All tasks prefer to stay on their current CPU. +* Do not pull a task from its preferred CPU if: +* 1. It is the only task running there; OR +* 2. Migrating it away from its preferred LLC would violate +* the cache-aware scheduling policy. +*/ +if (env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable) { +unsigned long util =0 ; +struct task_struct *cur; + +if (env->src_rq->nr_running <=1 ) +return true; + +rcu_read_lock (); + cur =rcu_dereference (env->src_rq->curr); +if (cur) + util =task_util (cur); +rcu_read_unlock (); + +if (can_migrate_llc (env->src_cpu, env->dst_cpu, + util, false) == mig_forbid) +return true; + } + +return false; +}#else static inline bool get_llc_stats (int cpu,unsigned long *util,unsigned long *cap) {return false; } + +static inline bool +break_llc_locality (struct lb_env *env) +{ +return false; +}#endif /* * can_migrate_task - may task p from runqueue rq be migrated to this_cpu?
@@ -12247,6 +12288,9 @@ static int need_active_balance(struct lb_env *env) {struct sched_domain *sd = env->sd; +if (break_llc_locality (env)) +return 0 ; +if (asym_active_balance (env))return 1 ;
@@ -12266,7 +12310,8 @@ static int need_active_balance(struct lb_env *env)return 1 ; } -if (env->migration_type == migrate_misfit) +if (env->migration_type == migrate_misfit || + env->migration_type == migrate_llc_task)return 1 ;return 0 ;
@@ -12711,9 +12756,20 @@ static int active_load_balance_cpu_stop(void *data) goto out_unlock;/* Is there any task to move? */ -if (busiest_rq->nr_running <=1 ) - goto out_unlock; +if (busiest_rq->nr_running <=1 ) { +#ifdef CONFIG_SCHED_CACHE +int llc =llc_idx (target_cpu); +if (!sched_cache_enabled ()) + goto out_unlock; + +if (llc <0 ) + goto out_unlock; +/* don't migrate if no task prefers target */ + if (busiest_rq->nr_pref_llc[llc] < 1) +#endif + goto out_unlock; + }/* * This condition is "impossible", if it occurs * we need to fix it. Originally reported by
Signature · 1 line
2.32.0
PATCH 14/19 sched/fair: Consider LLC preference when selecting tasks for load balancing
In reply to Tim ChenCurrently, task selection from the busiest runqueue ignores LLC preferences. Reorder tasks in the busiest queue to prioritize selection as follows:
1. Tasks preferring the destination CPU's LLC
2. Tasks with no LLC preference
3. Tasks preferring an LLC different from their current one
4. Tasks preferring the LLC they are currently on
This improves the likelihood that tasks are migrated to their preferred LLC.
--- kernel/sched/fair.c | 66 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-)
kernel/sched/fair.c +65 -1
@@ -10036,6 +10036,68 @@ static struct task_struct *detach_one_task(struct lb_env *env)return NULL ; } +#ifdef CONFIG_SCHED_CACHE +/* +* Prepare lists to detach tasks in the following order: +* 1. tasks that prefer dst cpu's LLC +* 2. tasks that have no preference in LLC +* 3. tasks that prefer LLC other than the ones they are on +* 4. tasks that prefer the LLC that they are currently on. +*/ +static struct list_head +*order_tasks_by_llc (struct lb_env *env,struct list_head *tasks) +{ +struct task_struct *p; +LIST_HEAD (pref_old_llc); +LIST_HEAD (pref_new_llc); +LIST_HEAD (no_pref_llc); +LIST_HEAD (pref_other_llc); + +if (!sched_cache_enabled ()) +return tasks; + +if (cpus_share_cache (env->dst_cpu, env->src_cpu)) +return tasks; + +while (!list_empty (tasks)) { + p = list_last_entry(tasks, struct task_struct, se.group_node); + +if (p->preferred_llc ==llc_id (env->dst_cpu)) { +list_move (&p->se.group_node, &pref_new_llc); +continue ; + } + +if (p->preferred_llc ==llc_id (env->src_cpu)) { +list_move (&p->se.group_node, &pref_old_llc); +continue ; + } + +if (p->preferred_llc ==-1 ) { +list_move (&p->se.group_node, &no_pref_llc); +continue ; + } + +list_move (&p->se.group_node, &pref_other_llc); + } + +/* +* We detach tasks from list tail in detach tasks. Put tasks +* to be chosen first at end of list. +*/ +list_splice (&pref_new_llc, tasks); +list_splice (&no_pref_llc, tasks); +list_splice (&pref_other_llc, tasks); +list_splice (&pref_old_llc, tasks); +return tasks; +} +#else +static inline struct list_head +*order_tasks_by_llc (struct lb_env *env,struct list_head *tasks) +{ +return tasks; +} +#endif +/* * detach_tasks() -- tries to detach up to imbalance load/util/tasks from * busiest_rq, as part of a balancing operation within domain "sd".
@@ -10044,7 +10106,7 @@ static struct task_struct *detach_one_task(struct lb_env *env)*/ static int detach_tasks (struct lb_env *env) { -struct list_head *tasks = &env->src_rq->cfs_tasks; +struct list_head *tasks;unsigned long util, load;struct task_struct *p;int detached =0 ;
@@ -10063,6 +10125,8 @@ static int detach_tasks(struct lb_env *env)if (env->imbalance <=0 )return 0 ; + tasks =order_tasks_by_llc (env, &env->src_rq->cfs_tasks); +while (!list_empty (tasks)) {/* * We don't want to steal all, otherwise we may be treated likewise,
Signature · 1 line
2.32.0
PATCH 15/19 sched/fair: Respect LLC preference in task migration and detach
In reply to Tim ChenDuring the final step of load balancing, can_migrate_task() now considers a task's LLC preference before moving it out of its preferred LLC.
Additionally, add checks in detach_tasks() to prevent selecting tasks that prefer their current LLC.
--- kernel/sched/fair.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-)
kernel/sched/fair.c +22 -2
@@ -9801,8 +9801,8 @@ static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu,* Check if task p can migrate from src_cpu to dst_cpu * in terms of cache aware load balance. */ -static __maybe_unused enum llc_mig can_migrate_llc_task (int src_cpu,int dst_cpu, -struct task_struct *p) +static enum llc_mig can_migrate_llc_task (int src_cpu,int dst_cpu, +struct task_struct *p) {struct mm_struct *mm;bool to_pref;
@@ -9969,6 +9969,12 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)if (env->flags &LBF_ACTIVE_LB )return 1 ; +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled () && +can_migrate_llc_task (env->src_cpu, env->dst_cpu, p) == mig_forbid) +return 0 ; +#endif + degrades =migrate_degrades_locality (p, env);if (!degrades) hot =task_hot (p, env);
@@ -10227,6 +10233,20 @@ static int detach_tasks(struct lb_env *env)if (env->imbalance <=0 )break ; +#ifdef CONFIG_SCHED_CACHE +/* +* Don't detach more tasks if the remaining tasks want +* to stay. We know the remaining tasks all prefer the +* current LLC, because after order_tasks_by_llc(), the +* tasks that prefer the current LLC are at the tail of +* the list. The inhibition of detachment is to avoid too +* many tasks being migrated out of the preferred LLC. +*/ +if (sched_cache_enabled () && detached && p->preferred_llc !=-1 && +llc_id (env->src_cpu) == p->preferred_llc) +break ; +#endif +continue ; next:if (p->sched_task_hot)
Signature · 1 line
2.32.0
K Prateek Nayak
reviewerRe: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
Hello Tim,
kernel/sched/fair.c int can_migrate_task(struct task_struct *p, struct lb_env *env) · line 9980
Quoted 13 lines · Tim Chen
kernel/sched/fair.c +6
-0@@ -9969,6 +9969,12 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)if (env->flags &LBF_ACTIVE_LB )return 1 ; +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled () && +can_migrate_llc_task (env->src_cpu, env->dst_cpu, p) == mig_forbid) +return 0 ; +#endif + degrades =migrate_degrades_locality (p, env);if (!degrades) hot =task_hot (p, env);
Should we care for task_hot() w.r.t. migration cost if a task is being moved to a preferred LLC?
Also, should we leave out tasks under core scheduling from the llc aware lb? Even discount them when calculating "mm->nr_running_avg"?
Quoted 6 lines
kernel/sched/fair.c +2
-0@@ -10227,6 +10233,20 @@ static int detach_tasks(struct lb_env *env)if (env->imbalance <=0 )break ; +#ifdef CONFIG_SCHED_CACHE +/*
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10247
Quoted 10 lines
kernel/sched/fair.c +10
-0+* Don't detach more tasks if the remaining tasks want +* to stay. We know the remaining tasks all prefer the +* current LLC, because after order_tasks_by_llc(), the +* tasks that prefer the current LLC are at the tail of +* the list. The inhibition of detachment is to avoid too +* many tasks being migrated out of the preferred LLC. +*/ +if (sched_cache_enabled () && detached && p->preferred_llc !=-1 && +llc_id (env->src_cpu) == p->preferred_llc) +break ;
In all cases? Should we check can_migrate_llc() wrt to util migrated and then make a call if we should move the preferred LLC tasks or not?
Perhaps disallow it the first time if "nr_balance_failed" is 0 but subsequent failed attempts should perhaps explore breaking the preferred llc restriction if there is an imbalance and we are under "mig_unrestricted" conditions.
Quoted 5 lines
kernel/sched/fair.c +2
-0+#endif +continue ; next:if (p->sched_task_hot)
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
Hi Prateek,
Quoted 7 lines · K Prateek Nayak
Hello Tim,
Quoted 4 lines · Tim Chen
kernel/sched/fair.c +0
-0@@ -9969,6 +9969,12 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)if (env->flags &LBF_ACTIVE_LB )return 1 ;
Quoted 13 lines
kernel/sched/fair.c int can_migrate_task(struct task_struct *p, struct lb_env *env) · line 9980
Quoted 9 lines
kernel/sched/fair.c +6
-0+#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled () && +can_migrate_llc_task (env->src_cpu, env->dst_cpu, p) == mig_forbid) +return 0 ; +#endif + degrades =migrate_degrades_locality (p, env);if (!degrades) hot =task_hot (p, env);Should we care for
task_hot()w.r.t. migration cost if a task is being moved to a preferred LLC?
This is a good question. The decision not to migrate a task when its LLC preference is violated takes priority over the check in task_hot().
The main reason is that we want cache aware aggregation to be more
aggressive than generic migration; otherwise, cache-aware migration
might not take effect according to our previous test. This seems to
be a trade-off. Another consideration might be: should we consider
the occupancy of a single thread or that of the entire process?
For example, suppose t0, t1, and t2 belong to the same process. t0
and t1 are running on the process's preferred LLC0, while t2 is
running on the non-preferred LLC1. Even though t2 has high occupancy
on LLC1 (making it cache-hot on LLC1), we might still want to move t2
to LLC0 if t0, t1, and t2 read from and write to each other - since we don't want to generate cross-LLC access.
Quoted 3 lines
Also, should we leave out tasks under core scheduling from the llc aware lb? Even discount them when calculating "
mm->nr_running_avg"?
Yes, it seems that the cookie match check case was missed, which is embedded in task_hot(). I suppose you are referring to the p->core_cookie check; I'll look into this direction.
Quoted 4 lines
Quoted 4 lines
kernel/sched/fair.c +0
-0@@ -10227,6 +10233,20 @@ static int detach_tasks(struct lb_env *env)if (env->imbalance <=0 )break ;
Quoted 16 lines
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10247
Quoted 12 lines
kernel/sched/fair.c +12
-0+#ifdef CONFIG_SCHED_CACHE +/* +* Don't detach more tasks if the remaining tasks want +* to stay. We know the remaining tasks all prefer the +* current LLC, because after order_tasks_by_llc(), the +* tasks that prefer the current LLC are at the tail of +* the list. The inhibition of detachment is to avoid too +* many tasks being migrated out of the preferred LLC. +*/ +if (sched_cache_enabled () && detached && p->preferred_llc !=-1 && +llc_id (env->src_cpu) == p->preferred_llc) +break ;In all cases? Should we check
can_migrate_llc()wrt to util migrated and then make a call if we should move the preferred LLC tasks or not?
Prior to this "stop of detaching tasks", we performed a can_migrate_task(p) to determine if the detached p is dequeued from its preferred LLC, and in can_migrate_task(), we use can_migrate_llc_task() -> can_migrate_llc() to carry out the check. That is to say, only when certain tasks have been detached, will we stop further detaching.
Quoted 5 lines
Perhaps disallow it the first time if "
nr_balance_failed" is 0 but subsequent failed attempts should perhaps explore breaking the preferred llc restriction if there is an imbalance and we are under "mig_unrestricted" conditions.
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above can_migrate_llc() check, I suppose we have raised the threshold for stopping "task detachment"?
thanks,
Chenyu
Quoted 6 lines
Quoted 5 lines
kernel/sched/fair.c +2
-0+#endif +continue ; next:if (p->sched_task_hot)
Re: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
Quoted 67 lines · Yu C
Hi Prateek,
Quoted 7 lines · K Prateek Nayak
Hello Tim,
Quoted 4 lines · Tim Chen
kernel/sched/fair.c +0
-0@@ -9969,6 +9969,12 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)if (env->flags &LBF_ACTIVE_LB )return 1 ;Quoted 13 lines
kernel/sched/fair.c int can_migrate_task(struct task_struct *p, struct lb_env *env) · line 9980
Quoted 9 lines
kernel/sched/fair.c +6
-0+#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled () && +can_migrate_llc_task (env->src_cpu, env->dst_cpu, p) == mig_forbid) +return 0 ; +#endif + degrades =migrate_degrades_locality (p, env);if (!degrades) hot =task_hot (p, env);Should we care for
task_hot()w.r.t. migration cost if a task is being moved to a preferred LLC?This is a good question. The decision not to migrate a task when its LLC preference is violated takes priority over the check in
task_hot().The main reason is that we want cache aware aggregation to be more aggressive than generic migration; otherwise, cache-aware migration
might not take effect according to our previous test. This seems to
be a trade-off. Another consideration might be: should we consider the occupancy of a single thread or that of the entire process? For example, suppose t0, t1, and t2 belong to the same process. t0 and t1 are running on the process's preferred LLC0, while t2 is running on the non-preferred LLC1. Even though t2 has high occupancy on LLC1 (making it cache-hot on LLC1), we might still want to move t2 to LLC0 if t0, t1, and t2 read from and write to each other - since we don't want to generate cross-LLC access.Quoted 3 lines
Also, should we leave out tasks under core scheduling from the llc aware lb? Even discount them when calculating "
mm->nr_running_avg"?Yes, it seems that the cookie match check case was missed, which is embedded in
task_hot(). I suppose you are referring to thep->core_cookiecheck; I'll look into this direction.Quoted 4 lines
Quoted 4 lines
kernel/sched/fair.c +0
-0@@ -10227,6 +10233,20 @@ static int detach_tasks(struct lb_env *env)if (env->imbalance <=0 )break ;Quoted 16 lines
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10247
Quoted 12 lines
kernel/sched/fair.c +12
-0+#ifdef CONFIG_SCHED_CACHE +/* +* Don't detach more tasks if the remaining tasks want +* to stay. We know the remaining tasks all prefer the +* current LLC, because after order_tasks_by_llc(), the +* tasks that prefer the current LLC are at the tail of +* the list. The inhibition of detachment is to avoid too +* many tasks being migrated out of the preferred LLC. +*/ +if (sched_cache_enabled () && detached && p->preferred_llc !=-1 && +llc_id (env->src_cpu) == p->preferred_llc) +break ;In all cases? Should we check
can_migrate_llc()wrt to util migrated and then make a call if we should move the preferred LLC tasks or not?
Quoted 12 lines
Prior to this "stop of detaching tasks", we performed a
can_migrate_task(p)to determine if the detached p is dequeued from its preferred LLC, and incan_migrate_task(), we usecan_migrate_llc_task()->can_migrate_llc()to carry out the check. That is to say, only when certain tasks have been detached, will we stop further detaching.Quoted 5 lines
Perhaps disallow it the first time if "
nr_balance_failed" is 0 but subsequent failed attempts should perhaps explore breaking the preferred llc restriction if there is an imbalance and we are under "mig_unrestricted" conditions.
Pratek,
We have to actually allow for imbalance between LLCs with task aggregation.
Say we have 2 LLCs and only one process running. Suppose all tasks in the process can fit in one LLC and not overload it. Then we should not pull tasks from the preferred LLC, and allow the imbalance. If we balance the tasks the second time around, that will defeat the purpose.
That's why we have the knob llc_overload_pct (50%), which will start spreading tasks to non-preferred LLC once load in preferred LLC excees 50%. And llc_imb_pct(20%), which allows for a 20% higher load between preferred LLC and non-preferred LLC if the preferred LLC is operating above 50%.
So if we ignore the LLC policy totally the second time around, we may be breaking LLC aggregation and have tasks be moved to their non-preferred LLC.
Will take a closer look to see if nr_balance_failed > 0 because we cannot move tasks to their preferred LLC repeatedly, and if we should do anything different to balance tasks better without violating LLC preference.
Tim
Quoted 14 lines
I suppose you are suggesting that the threshold for stopping task detachment
should be higher. With the abovecan_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?thanks,
ChenyuQuoted 6 lines
Quoted 5 lines
kernel/sched/fair.c +2
-0+#endif +continue ; next:if (p->sched_task_hot)
K Prateek Nayak
reviewerRe: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
Hello Tim,
Quoted 40 lines · Tim Chen
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
kernel/sched/fair.c +7
-0+#ifdef CONFIG_SCHED_CACHE +/* +* Don't detach more tasks if the remaining tasks want +* to stay. We know the remaining tasks all prefer the +* current LLC, because after order_tasks_by_llc(), the +* tasks that prefer the current LLC are at the tail of +* the list. The inhibition of detachment is to avoid too Quoted 9 lines
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10247
Quoted 5 lines
kernel/sched/fair.c +5
-0+* many tasks being migrated out of the preferred LLC. +*/ +if (sched_cache_enabled () && detached && p->preferred_llc !=-1 && +llc_id (env->src_cpu) == p->preferred_llc) +break ;In all cases? Should we check
can_migrate_llc()wrt to util migrated and then make a call if we should move the preferred LLC tasks or not?Quoted 12 lines
Prior to this "stop of detaching tasks", we performed a
can_migrate_task(p)to determine if the detached p is dequeued from its preferred LLC, and incan_migrate_task(), we usecan_migrate_llc_task()->can_migrate_llc()to carry out the check. That is to say, only when certain tasks have been detached, will we stop further detaching.Quoted 5 lines
Perhaps disallow it the first time if "
nr_balance_failed" is 0 but subsequent failed attempts should perhaps explore breaking the preferred llc restriction if there is an imbalance and we are under "mig_unrestricted" conditions.Pratek,
We have to actually allow for imbalance between LLCs with task aggregation.
Say we have 2 LLCs and only one process running. Suppose all tasks in the process can fit in one LLC and not overload it. Then we should not pull tasks from the preferred LLC, and allow the imbalance. If we balance the tasks the second time around, that will defeat the purpose.
Quoted 7 lines
That's why we have the knob
llc_overload_pct(50%), which will start spreading tasks to non-preferred LLC once load in preferred LLC excees 50%. Andllc_imb_pct(20%), which allows for a 20% higher load between preferred LLC and non-preferred LLC if the preferred LLC is operating above 50%.So if we ignore the LLC policy totally the second time around, we may be breaking LLC aggregation and have tasks be moved to their non-preferred LLC.
Ack! I have replied to Chenyu's response with an example of "migrate_load" case that, as per my understanding, would be restricted by this condition. If I'm missing something, please do let me know. Otherwise, the intention looks good to me.
Quoted 5 lines
Will take a closer look to see if
nr_balance_failed> 0 because we cannot move tasks to their preferred LLC repeatedly, and if we should do anything different to balance tasks better without violating LLC preference.
Thank you!
Signature · 2 lines
Thanks and Regards, Prateek
K Prateek Nayak
reviewerRe: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
In reply to Chen, Yu CHello Chenyu,
Quoted 26 lines · Yu C
Hi Prateek,
Quoted 19 lines · K Prateek Nayak
Hello Tim,
Quoted 9 lines · Tim Chen
kernel/sched/fair.c +0
-0@@ -9969,6 +9969,12 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) if (env->flags & LBF_ACTIVE_LB) return 1;+#ifdef
CONFIG_SCHED_CACHE
+ if (sched_cache_enabled()&&
+can_migrate_llc_task(env->src_cpu, env->dst_cpu, p)==mig_forbid) + return 0;(quoted diff) +2
-0+#endif +Quoted 3 lines
degrades =
migrate_degrades_locality(p, env); if (!degrades)
hot =task_hot(p, env);Should we care for
task_hot()w.r.t. migration cost if a task is being moved to a preferred LLC?This is a good question. The decision not to migrate a task when its LLC preference is violated takes priority over the check in
task_hot().
Quoted 10 lines
The main reason is that we want cache aware aggregation to be more
aggressive than generic migration; otherwise, cache-aware migration
might not take effect according to our previous test. This seems to
be a trade-off. Another consideration might be: should we consider
the occupancy of a single thread or that of the entire process?
For example, suppose t0, t1, and t2 belong to the same process. t0
and t1 are running on the process's preferred LLC0, while t2 is
running on the non-preferred LLC1. Even though t2 has high occupancy
on LLC1 (making it cache-hot on LLC1), we might still want to move t2
to LLC0 if t0, t1, and t2 read from and write to each other - since we don't want to generate cross-LLC access.
Makes sense. That would need some heuristics based on the avg_running to know which LLC can be be a potential target with fewest migrations. But then again, in a dynamic system things change so quickly - what you have now seems to be a good start to further optimize on top of.
Quoted 7 lines
Quoted 4 lines
Also, should we leave out tasks under core scheduling from the llc aware lb? Even discount them when calculating "
mm->nr_running_avg"?Yes, it seems that the cookie match check case was missed, which is embedded in
task_hot(). I suppose you are referring to thep->core_cookiecheck; I'll look into this direction.
Yup! I think if user has opted into core scheduling, they should ideally not bother about cache aware scheduling.
Quoted 33 lines
Quoted 11 lines
Quoted 11 lines
kernel/sched/fair.c +0
-0@@ -10227,6 +10233,20 @@ static int detach_tasks(struct lb_env *env) if (env->imbalance <= 0) break;+#ifdef
CONFIG_SCHED_CACHE
+ /*
+ * Don't detach more tasks if the remaining tasks want + * to stay. We know the remaining tasks all prefer the + * current LLC, because afterorder_tasks_by_llc(), the + * tasks that prefer the current LLC are at the tail of + * the list. The inhibition of detachment is to avoid tooQuoted 9 lines
Quoted 5 lines
+ * many tasks being migrated out of the preferred LLC. + */
+ if (sched_cache_enabled()&& detached &&p->preferred_llc!= -1 && +llc_id(env->src_cpu)==p->preferred_llc)
+ break;In all cases? Should we check
can_migrate_llc()wrt to util migrated and then make a call if we should move the preferred LLC tasks or not?Prior to this "stop of detaching tasks", we performed a
can_migrate_task(p)to determine if the detached p is dequeued from its preferred LLC, and incan_migrate_task(), we usecan_migrate_llc_task()->can_migrate_llc()to carry out the check. That is to say, only when certain tasks have been detached, will we stop further detaching.Quoted 5 lines
Perhaps disallow it the first time if "
nr_balance_failed" is 0 but subsequent failed attempts should perhaps explore breaking the preferred llc restriction if there is an imbalance and we are under "mig_unrestricted" conditions.
Quoted 3 lines
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above
can_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?
Say the LLC is under heavy load and we only have overloaded groups. can_migrate_llc() would return "mig_unrestricted" since fits_llc_capacity() would return false.
Since we are under "migrate_load", sched_balance_find_src_rq() has returned the CPU with the highest load which could very well be the CPU with with a large number of preferred LLC tasks.
sched_cache_enabled() is still true and when detach_tasks() reaches one of these preferred llc tasks (which comes at the very end of the tasks list), we break out even if env->imbalance > 0 leaving potential imbalance for the "migrate_load" case.
Instead, we can account for the util moved out of the src_llc and after accounting for it, check if can_migrate_llc() would return "mig_forbid" for the src llc.
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
On 10/29/2025 11:54 AM, K Prateek Nayak wrote:
[snip]
Quoted 44 lines
Quoted 32 lines
Quoted 10 lines
Quoted 10 lines
kernel/sched/fair.c +6
-0@@ -10227,6 +10233,20 @@ static int detach_tasks(struct lb_env *env) if (env->imbalance <= 0) break; +#ifdef CONFIG_SCHED_CACHE + /* +* Don't detach more tasks if the remaining tasks want +* to stay. We know the remaining tasks all prefer the +* current LLC, because after order_tasks_by_llc(), the +* tasks that prefer the current LLC are at the tail of +* the list. The inhibition of detachment is to avoid too Quoted 9 lines
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10247
Quoted 5 lines
kernel/sched/fair.c +5
-0+* many tasks being migrated out of the preferred LLC. +*/ + if (sched_cache_enabled() && detached && p->preferred_llc != -1 && + llc_id(env->src_cpu) == p->preferred_llc) + break;In all cases? Should we check
can_migrate_llc()wrt to util migrated and then make a call if we should move the preferred LLC tasks or not?Prior to this "stop of detaching tasks", we performed a
can_migrate_task(p)to determine if the detached p is dequeued from its preferred LLC, and incan_migrate_task(), we usecan_migrate_llc_task()->can_migrate_llc()to carry out the check. That is to say, only when certain tasks have been detached, will we stop further detaching.Quoted 5 lines
Perhaps disallow it the first time if "
nr_balance_failed" is 0 but subsequent failed attempts should perhaps explore breaking the preferred llc restriction if there is an imbalance and we are under "mig_unrestricted" conditions.Quoted 3 lines
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above
can_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?Say the LLC is under heavy load and we only have overloaded groups.
can_migrate_llc()would return "mig_unrestricted" sincefits_llc_capacity()would return false.Since we are under "
migrate_load",sched_balance_find_src_rq()has returned the CPU with the highest load which could very well be the CPU with with a large number of preferred LLC tasks.
Quoted 9 lines
sched_cache_enabled()is still true and whendetach_tasks()reaches one of these preferred llc tasks (which comes at the very end of the tasks list), we break out even ifenv->imbalance> 0 leaving potential imbalance for the "migrate_load" case.Instead, we can account for the util moved out of the
src_llcand after accounting for it, check ifcan_migrate_llc()would return "mig_forbid" for the src llc.
I see your point, the original decision matrix intends to spread the tasks when both LLCs are overloaded. (src is the preferred LLC, dst is non-preferred LLC)
src \ dst 30% 40% 50% 60% 30% N N N N 40% N N N N 50% N N G G 60% Y N G G
src : src_util dst : dst_util Y : Yes, migrate N : No, do not migrate G : let the Generic load balance to even the load.
I suppose the reason why the code breaks the rule here is because as Tim mentioned in another thread, to inhibit the task bouncing between LLCs.
thanks,
Chenyu
Re: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
In reply to K Prateek NayakQuoted 49 lines · K Prateek Nayak
Hello Chenyu,
Quoted 26 lines · Yu C
Hi Prateek,
Quoted 19 lines · K Prateek Nayak
Hello Tim,
Quoted 9 lines · Tim Chen
kernel/sched/fair.c +0
-0@@ -9969,6 +9969,12 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env) if (env->flags & LBF_ACTIVE_LB) return 1;+#ifdef
CONFIG_SCHED_CACHE
+ if (sched_cache_enabled()&&
+can_migrate_llc_task(env->src_cpu, env->dst_cpu, p)==mig_forbid) + return 0;(quoted diff) +2
-0+#endif +Quoted 3 lines
degrades =
migrate_degrades_locality(p, env); if (!degrades)
hot =task_hot(p, env);Should we care for
task_hot()w.r.t. migration cost if a task is being moved to a preferred LLC?This is a good question. The decision not to migrate a task when its LLC preference is violated takes priority over the check in
task_hot().Quoted 10 lines
The main reason is that we want cache aware aggregation to be more
aggressive than generic migration; otherwise, cache-aware migration
might not take effect according to our previous test. This seems to
be a trade-off. Another consideration might be: should we consider
the occupancy of a single thread or that of the entire process?
For example, suppose t0, t1, and t2 belong to the same process. t0
and t1 are running on the process's preferred LLC0, while t2 is
running on the non-preferred LLC1. Even though t2 has high occupancy
on LLC1 (making it cache-hot on LLC1), we might still want to move t2
to LLC0 if t0, t1, and t2 read from and write to each other - since we don't want to generate cross-LLC access.Makes sense. That would need some heuristics based on the
avg_runningto know which LLC can be be a potential target with fewest migrations. But then again, in a dynamic system things change so quickly - what you have now seems to be a good start to further optimize on top of.Quoted 4 lines
Quoted 4 lines
Also, should we leave out tasks under core scheduling from the llc aware lb? Even discount them when calculating "
mm->nr_running_avg"?
Quoted 26 lines
Quoted 3 lines
Yes, it seems that the cookie match check case was missed, which is embedded in
task_hot(). I suppose you are referring to thep->core_cookiecheck; I'll look into this direction.Yup! I think if user has opted into core scheduling, they should ideally not bother about cache aware scheduling.
Quoted 19 lines
Quoted 19 lines
Quoted 8 lines
kernel/sched/fair.c +0
-0@@ -10227,6 +10233,20 @@ static int detach_tasks(struct lb_env *env) if (env->imbalance <= 0) break;+#ifdef
CONFIG_SCHED_CACHE
+ /*
+ * Don't detach more tasks if the remaining tasks want + * to stay. We know the remaining tasks all prefer theQuoted 8 lines
+ * current LLC, because after
order_tasks_by_llc(), the + * tasks that prefer the current LLC are at the tail of + * the list. The inhibition of detachment is to avoid too + * many tasks being migrated out of the preferred LLC. + */
+ if (sched_cache_enabled()&& detached &&p->preferred_llc!= -1 && +llc_id(env->src_cpu)==p->preferred_llc)
+ break;In all cases?
Not in all cases, but only when we know that the remaining tasks prefer to stay in current LLC and not be moved to an LLC it doesn't like.
I think we need to add the check that llc_id(env->dst_cpu) != p->preferred_llc in the above condition
Quoted 24 lines
Quoted 16 lines
Quoted 3 lines
Should we check
can_migrate_llc()wrt to util migrated and then make a call if we should move the preferred LLC tasks or not?Prior to this "stop of detaching tasks", we performed a
can_migrate_task(p)to determine if the detached p is dequeued from its preferred LLC, and incan_migrate_task(), we usecan_migrate_llc_task()->can_migrate_llc()to carry out the check. That is to say, only when certain tasks have been detached, will we stop further detaching.Quoted 5 lines
Perhaps disallow it the first time if "
nr_balance_failed" is 0 but subsequent failed attempts should perhaps explore breaking the preferred llc restriction if there is an imbalance and we are under "mig_unrestricted" conditions.Quoted 3 lines
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above
can_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?Say the LLC is under heavy load and we only have overloaded groups.
can_migrate_llc()would return "mig_unrestricted" sincefits_llc_capacity()would return false.
Quoted 8 lines
Since we are under "
migrate_load",sched_balance_find_src_rq()has returned the CPU with the highest load which could very well be the CPU with with a large number of preferred LLC tasks.
sched_cache_enabled()is still true and whendetach_tasks()reaches one of these preferred llc tasks (which comes at the very end of the tasks list),
we break out even ifenv->imbalance> 0 leaving
Yes, but at least one task has been removed to even the load (making forward progress) and the remaining tasks all wish to stay in the current LLC and will
preferred not to be moved. My thought was to not even all the load out
in one shot and pull more tasks out of their preferred LLC.
If the imbalance still remain, we'll come to that in the next load balance.
Pulling tasks more slowly when we come to tasks that preferred to stay (if possible) would also help to prevent tasks bouncing between LLC.
Tim
Quoted 5 lines
potential imbalance for the "
migrate_load" case.Instead, we can account for the util moved out of the
src_llcand after accounting for it, check ifcan_migrate_llc()would return "mig_forbid" for the src llc.
K Prateek Nayak
reviewerRe: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
Hello Tim,
Quoted 17 lines · Tim Chen
Quoted 8 lines
Quoted 3 lines
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above
can_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?Say the LLC is under heavy load and we only have overloaded groups.
can_migrate_llc()would return "mig_unrestricted" sincefits_llc_capacity()would return false.Quoted 8 lines
Since we are under "
migrate_load",sched_balance_find_src_rq()has returned the CPU with the highest load which could very well be the CPU with with a large number of preferred LLC tasks.
sched_cache_enabled()is still true and whendetach_tasks()reaches one of these preferred llc tasks (which comes at the very end of the tasks list),
we break out even ifenv->imbalance> 0 leaving
Quoted 5 lines
Yes, but at least one task has been removed to even the load (making forward progress) and the remaining tasks all wish to stay in the current LLC and will
preferred not to be moved. My thought was to not even all the load out
in one shot and pull more tasks out of their preferred LLC.
If the imbalance still remain, we'll come to that in the next load balance.
In that case, can we spoof a LBF_ALL_PINNED for the case where we start hitting preferred task. That way, the main lb loop will goto redo and try to find another busy CPU to pull tasks from.
Quoted 6 lines
Pulling tasks more slowly when we come to tasks that preferred to stay (if possible) would also help to prevent tasks bouncing between LLC.
Tim
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
Quoted 20 lines · K Prateek Nayak
Hello Tim,
Quoted 17 lines · Tim Chen
Quoted 16 lines
Quoted 3 lines
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above
can_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?Say the LLC is under heavy load and we only have overloaded groups.
can_migrate_llc()would return "mig_unrestricted" sincefits_llc_capacity()would return false.Since we are under "
migrate_load",sched_balance_find_src_rq()has returned the CPU with the highest load which could very well be the CPU with with a large number of preferred LLC tasks.
sched_cache_enabled()is still true and whendetach_tasks()reaches one of these preferred llc tasks (which comes at the very end of the tasks list),
we break out even ifenv->imbalance> 0 leaving
Quoted 7 lines
Quoted 5 lines
Yes, but at least one task has been removed to even the load (making forward progress) and the remaining tasks all wish to stay in the current LLC and will
preferred not to be moved. My thought was to not even all the load out
in one shot and pull more tasks out of their preferred LLC.
If the imbalance still remain, we'll come to that in the next load balance.In that case, can we spoof a
LBF_ALL_PINNEDfor the case where we start
In the code chunk (with fix I mentioned in last reply):
+#ifdef CONFIG_SCHED_CACHE
+ /*
+ * Don't detach more tasks if the remaining tasks want
+ * to stay. We know the remaining tasks all prefer the
+ * current LLC, because after order_tasks_by_llc(), the
+ * tasks that prefer the current LLC are at the tail of
+ * the list. The inhibition of detachment is to avoid too
+ * many tasks being migrated out of the preferred LLC.
+ */
+ if (sched_cache_enabled() && detached && p->preferred_llc != -1 &&
+ llc_id(env->src_cpu) == p->preferred_llc &&
llc_id(env->dst_cpu) != p->preferred_llc)
+ break;
We have already pulled at least one task when we stop detaching because we know that all the remaining tasks want to stay in it current LLC. "detached" is non zero when we break. So LBF_ALL_PINNED would be cleared. We will only exit the detach_tasks loop when there are truly no tasks that can be moved and it is truly a LBF_ALL_PINNED case.
We should not be causing problem with the LBF_ALL_PINNED.
Tim
Quoted 9 lines
hitting preferred task. That way, the main lb loop will goto redo and try to find another busy CPU to pull tasks from.
Quoted 6 lines
Pulling tasks more slowly when we come to tasks that preferred to stay (if possible) would also help to prevent tasks bouncing between LLC.
Tim
K Prateek Nayak
reviewerRe: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
Hello Tim,
Quoted 45 lines · Tim Chen
Quoted 20 lines · K Prateek Nayak
Hello Tim,
Quoted 17 lines · Tim Chen
Quoted 16 lines
Quoted 3 lines
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above
can_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?Say the LLC is under heavy load and we only have overloaded groups.
can_migrate_llc()would return "mig_unrestricted" sincefits_llc_capacity()would return false.Since we are under "
migrate_load",sched_balance_find_src_rq()has returned the CPU with the highest load which could very well be the CPU with with a large number of preferred LLC tasks.
sched_cache_enabled()is still true and whendetach_tasks()reaches one of these preferred llc tasks (which comes at the very end of the tasks list),
we break out even ifenv->imbalance> 0 leavingQuoted 7 lines
Quoted 5 lines
Yes, but at least one task has been removed to even the load (making forward progress) and the remaining tasks all wish to stay in the current LLC and will
preferred not to be moved. My thought was to not even all the load out
in one shot and pull more tasks out of their preferred LLC.
If the imbalance still remain, we'll come to that in the next load balance.In that case, can we spoof a
LBF_ALL_PINNEDfor the case where we startIn the code chunk (with fix I mentioned in last reply):
(quoted diff) +11
-0+#ifdef CONFIG_SCHED_CACHE + /* + * Don't detach more tasks if the remaining tasks want + * to stay. We know the remaining tasks all prefer the + * current LLC, because after order_tasks_by_llc(), the + * tasks that prefer the current LLC are at the tail of + * the list. The inhibition of detachment is to avoid too + * many tasks being migrated out of the preferred LLC. + */ + if (sched_cache_enabled() && detached && p->preferred_llc != -1 && + llc_id(env->src_cpu) == p->preferred_llc &&llc_id(env->dst_cpu) != p->preferred_llc)(quoted diff) +1
-0+ break;
Quoted 5 lines
We have already pulled at least one task when we stop detaching because we know that all the remaining tasks want to stay in it current LLC. "detached" is non zero when we break. So
LBF_ALL_PINNEDwould be cleared. We will only exit thedetach_tasksloop when there are truly no tasks that can be moved and it is truly aLBF_ALL_PINNEDcase.
So what I was suggesting is something like:
@@ -10251,6 +10252,7 @@ static int detach_tasks(struct lb_env *env)
unsigned long util, load;
struct task_struct *p;
int detached = 0;
+ bool preserve_preferred;
lockdep_assert_rq_held(env->src_rq);
@@ -10268,6 +10270,10 @@ static int detach_tasks(struct lb_env *env)
tasks = order_tasks_by_llc(env, &env->src_rq->cfs_tasks);
+ preserve_preferred = sched_cache_enabled() &&
+ !(env->sd->flags & SD_SHARE_LLC) &&
+ !sd->nr_balance_failed;
+
while (!list_empty(tasks)) {
/*
* We don't want to steal all, otherwise we may be treated likewise,
@@ -10370,16 +10376,15 @@ static int detach_tasks(struct lb_env *env)
#ifdef CONFIG_SCHED_CACHE
/*
- * Don't detach more tasks if the remaining tasks want
- * to stay. We know the remaining tasks all prefer the
- * current LLC, because after order_tasks_by_llc(), the
- * tasks that prefer the current LLC are at the tail of
- * the list. The inhibition of detachment is to avoid too
- * many tasks being migrated out of the preferred LLC.
+ * We've hit tasks that prefer src LLC while balancing between LLCs.
+ * If previous balances have been successful, pretend the rest of the
+ * tasks on this CPU are pinned and let the main load balancing loop
+ * find another target CPU to pull from if imbalance exists.
*/
- if (sched_cache_enabled() && detached && p->preferred_llc != -1 &&
- llc_id(env->src_cpu) == p->preferred_llc)
+ if (preserve_preferred && detached && llc_id(env->src_cpu) == p->preferred_llc) {
+ env->flags |= LBF_ALL_PINNED;
break;
+ }
#endif
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
Hi Prateek,
Quoted 65 lines · K Prateek Nayak
Hello Tim,
Quoted 45 lines · Tim Chen
Quoted 20 lines · K Prateek Nayak
Hello Tim,
Quoted 17 lines · Tim Chen
Quoted 16 lines
Quoted 3 lines
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above
can_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?Say the LLC is under heavy load and we only have overloaded groups.
can_migrate_llc()would return "mig_unrestricted" sincefits_llc_capacity()would return false.Since we are under "
migrate_load",sched_balance_find_src_rq()has returned the CPU with the highest load which could very well be the CPU with with a large number of preferred LLC tasks.
sched_cache_enabled()is still true and whendetach_tasks()reaches one of these preferred llc tasks (which comes at the very end of the tasks list),
we break out even ifenv->imbalance> 0 leavingQuoted 7 lines
Quoted 5 lines
Yes, but at least one task has been removed to even the load (making forward progress) and the remaining tasks all wish to stay in the current LLC and will
preferred not to be moved. My thought was to not even all the load out
in one shot and pull more tasks out of their preferred LLC.
If the imbalance still remain, we'll come to that in the next load balance.In that case, can we spoof a
LBF_ALL_PINNEDfor the case where we startIn the code chunk (with fix I mentioned in last reply):
(quoted diff) +11
-0+#ifdef CONFIG_SCHED_CACHE + /* + * Don't detach more tasks if the remaining tasks want + * to stay. We know the remaining tasks all prefer the + * current LLC, because after order_tasks_by_llc(), the + * tasks that prefer the current LLC are at the tail of + * the list. The inhibition of detachment is to avoid too + * many tasks being migrated out of the preferred LLC. + */ + if (sched_cache_enabled() && detached && p->preferred_llc != -1 && + llc_id(env->src_cpu) == p->preferred_llc &&llc_id(env->dst_cpu) != p->preferred_llc)(quoted diff) +1
-0+ break;Quoted 5 lines
We have already pulled at least one task when we stop detaching because we know that all the remaining tasks want to stay in it current LLC. "detached" is non zero when we break. So
LBF_ALL_PINNEDwould be cleared. We will only exit thedetach_tasksloop when there are truly no tasks that can be moved and it is truly aLBF_ALL_PINNEDcase.So what I was suggesting is something like:
(quoted diff) +1
-0@@ -10251,6 +10252,7 @@ static int detach_tasks(struct lb_env *env) unsigned long util, load; struct task_struct *p; int detached = 0; + bool preserve_preferred; lockdep_assert_rq_held(env->src_rq);
Review static int detach_tasks(struct lb_env *env) · line 10274
Quoted 5 lines
(quoted diff) +2
-0@@ -10268,6 +10270,10 @@ static int detach_tasks(struct lb_env *env) tasks = order_tasks_by_llc(env, &env->src_rq->cfs_tasks); + preserve_preferred = sched_cache_enabled() && + !(env->sd->flags & SD_SHARE_LLC) &&
Maybe also check (env->sd->child->flag & SD_SHARE_LLC) because we only care about the domain that is the parent of a LLC domain.
Quoted 14 lines
(quoted diff) +2
-5+ !sd->nr_balance_failed; + while (!list_empty(tasks)) { /* * We don't want to steal all, otherwise we may be treated likewise,@@ -10370,16 +10376,15 @@ static int detach_tasks(struct lb_env *env) #ifdef CONFIG_SCHED_CACHE /* - * Don't detach more tasks if the remaining tasks want - * to stay. We know the remaining tasks all prefer the - * current LLC, because after order_tasks_by_llc(), the - * tasks that prefer the current LLC are at the tail of - * the list. The inhibition of detachment is to avoid too
Review static int detach_tasks(struct lb_env *env) · line 10385
Quoted 10 lines
(quoted diff) +6
-3- * many tasks being migrated out of the preferred LLC. + * We've hit tasks that prefer src LLC while balancing between LLCs. + * If previous balances have been successful, pretend the rest of the + * tasks on this CPU are pinned and let the main load balancing loop + * find another target CPU to pull from if imbalance exists. */ - if (sched_cache_enabled() && detached && p->preferred_llc != -1 && - llc_id(env->src_cpu) == p->preferred_llc) + if (preserve_preferred && detached && llc_id(env->src_cpu) == p->preferred_llc) { + env->flags |= LBF_ALL_PINNED;
Let me try to understand this strategy: if all previous migrations on this sched_domain have succeeded, it means that even if we stop migrating tasks out of this busiest CPU from now on, it won’t matter because the imbalance has already been mitigated. If we stop the migration, we should look for other busy CPUs to pull some tasks from. One concern is that setting LBF_ALL_PINNED and only clearing env->dst_cpu will trigger a full re-scan of the entire sched_domain, which might be costly-especially on large LLCs. We can try this to see if it has any impact on the benchmark.
thanks,
Chenyu
Quoted 5 lines
(quoted diff) +1
-0break; + } #endif
Re: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
Quoted 103 lines · Yu C
Hi Prateek,
Quoted 65 lines · K Prateek Nayak
Hello Tim,
Quoted 45 lines · Tim Chen
Quoted 20 lines · K Prateek Nayak
Hello Tim,
Quoted 17 lines · Tim Chen
Quoted 16 lines
Quoted 3 lines
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above
can_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?Say the LLC is under heavy load and we only have overloaded groups.
can_migrate_llc()would return "mig_unrestricted" sincefits_llc_capacity()would return false.Since we are under "
migrate_load",sched_balance_find_src_rq()has returned the CPU with the highest load which could very well be the CPU with with a large number of preferred LLC tasks.
sched_cache_enabled()is still true and whendetach_tasks()reaches one of these preferred llc tasks (which comes at the very end of the tasks list),
we break out even ifenv->imbalance> 0 leavingQuoted 7 lines
Quoted 5 lines
Yes, but at least one task has been removed to even the load (making forward progress) and the remaining tasks all wish to stay in the current LLC and will
preferred not to be moved. My thought was to not even all the load out
in one shot and pull more tasks out of their preferred LLC.
If the imbalance still remain, we'll come to that in the next load balance.In that case, can we spoof a
LBF_ALL_PINNEDfor the case where we startIn the code chunk (with fix I mentioned in last reply):
(quoted diff) +11
-0+#ifdef CONFIG_SCHED_CACHE + /* + * Don't detach more tasks if the remaining tasks want + * to stay. We know the remaining tasks all prefer the + * current LLC, because after order_tasks_by_llc(), the + * tasks that prefer the current LLC are at the tail of + * the list. The inhibition of detachment is to avoid too + * many tasks being migrated out of the preferred LLC. + */ + if (sched_cache_enabled() && detached && p->preferred_llc != -1 && + llc_id(env->src_cpu) == p->preferred_llc &&llc_id(env->dst_cpu) != p->preferred_llc)(quoted diff) +1
-0+ break;Quoted 5 lines
We have already pulled at least one task when we stop detaching because we know that all the remaining tasks want to stay in it current LLC. "detached" is non zero when we break. So
LBF_ALL_PINNEDwould be cleared. We will only exit thedetach_tasksloop when there are truly no tasks that can be moved and it is truly aLBF_ALL_PINNEDcase.So what I was suggesting is something like:
(quoted diff) +1
-0@@ -10251,6 +10252,7 @@ static int detach_tasks(struct lb_env *env) unsigned long util, load; struct task_struct *p; int detached = 0; + bool preserve_preferred; lockdep_assert_rq_held(env->src_rq);Review static int detach_tasks(struct lb_env *env) · line 10274
Quoted 5 lines
(quoted diff) +2
-0@@ -10268,6 +10270,10 @@ static int detach_tasks(struct lb_env *env) tasks = order_tasks_by_llc(env, &env->src_rq->cfs_tasks); + preserve_preferred = sched_cache_enabled() && + !(env->sd->flags & SD_SHARE_LLC) &&Maybe also check (
env->sd->child->flag&SD_SHARE_LLC) because we only care about the domain that is the parent of a LLC domain.Quoted 14 lines
(quoted diff) +2
-5+ !sd->nr_balance_failed; + while (!list_empty(tasks)) { /* * We don't want to steal all, otherwise we may be treated likewise,@@ -10370,16 +10376,15 @@ static int detach_tasks(struct lb_env *env) #ifdef CONFIG_SCHED_CACHE /* - * Don't detach more tasks if the remaining tasks want - * to stay. We know the remaining tasks all prefer the - * current LLC, because after order_tasks_by_llc(), the - * tasks that prefer the current LLC are at the tail of - * the list. The inhibition of detachment is to avoid tooReview static int detach_tasks(struct lb_env *env) · line 10385
Quoted 10 lines
(quoted diff) +6
-3- * many tasks being migrated out of the preferred LLC. + * We've hit tasks that prefer src LLC while balancing between LLCs. + * If previous balances have been successful, pretend the rest of the + * tasks on this CPU are pinned and let the main load balancing loop + * find another target CPU to pull from if imbalance exists. */ - if (sched_cache_enabled() && detached && p->preferred_llc != -1 && - llc_id(env->src_cpu) == p->preferred_llc) + if (preserve_preferred && detached && llc_id(env->src_cpu) == p->preferred_llc) { + env->flags |= LBF_ALL_PINNED;Let me try to understand this strategy: if all previous migrations
Quoted 8 lines
on this
sched_domainhave succeeded, it means that even if we stop migrating tasks out of this busiest CPU from now on, it won’t matter because the imbalance has already been mitigated. If we stop the migration, we should look for other busy CPUs to pull some tasks from. One concern is that settingLBF_ALL_PINNEDand only clearingenv->dst_cpuwill trigger a full re-scan of the entiresched_domain, which might be costly-especially on large LLCs. We can try this to see if it has any impact on the benchmark.
I think it does cause update_sd_lb_stats() to be called again with the previous rq taken out. So we are spending more CPU cycles to find an alternative task to balance to try to preserve LLC preference.
Tim
Quoted 9 lines
thanks,
ChenyuQuoted 5 lines
(quoted diff) +1
-0break; + } #endif
Re: [PATCH 15/19] sched/fair: Respect LLC preference in task migration and detach
In reply to K Prateek NayakQuoted 88 lines · K Prateek Nayak
Hello Tim,
Quoted 45 lines · Tim Chen
Quoted 20 lines · K Prateek Nayak
Hello Tim,
Quoted 17 lines · Tim Chen
Quoted 16 lines
Quoted 3 lines
I suppose you are suggesting that the threshold for stopping task detachment should be higher. With the above
can_migrate_llc()check, I suppose we have raised the threshold for stopping "task detachment"?Say the LLC is under heavy load and we only have overloaded groups.
can_migrate_llc()would return "mig_unrestricted" sincefits_llc_capacity()would return false.Since we are under "
migrate_load",sched_balance_find_src_rq()has returned the CPU with the highest load which could very well be the CPU with with a large number of preferred LLC tasks.
sched_cache_enabled()is still true and whendetach_tasks()reaches one of these preferred llc tasks (which comes at the very end of the tasks list),
we break out even ifenv->imbalance> 0 leavingQuoted 7 lines
Quoted 5 lines
Yes, but at least one task has been removed to even the load (making forward progress) and the remaining tasks all wish to stay in the current LLC and will
preferred not to be moved. My thought was to not even all the load out
in one shot and pull more tasks out of their preferred LLC.
If the imbalance still remain, we'll come to that in the next load balance.In that case, can we spoof a
LBF_ALL_PINNEDfor the case where we startIn the code chunk (with fix I mentioned in last reply):
(quoted diff) +11
-0+#ifdef CONFIG_SCHED_CACHE + /* + * Don't detach more tasks if the remaining tasks want + * to stay. We know the remaining tasks all prefer the + * current LLC, because after order_tasks_by_llc(), the + * tasks that prefer the current LLC are at the tail of + * the list. The inhibition of detachment is to avoid too + * many tasks being migrated out of the preferred LLC. + */ + if (sched_cache_enabled() && detached && p->preferred_llc != -1 && + llc_id(env->src_cpu) == p->preferred_llc &&llc_id(env->dst_cpu) != p->preferred_llc)(quoted diff) +1
-0+ break;Quoted 5 lines
We have already pulled at least one task when we stop detaching because we know that all the remaining tasks want to stay in it current LLC. "detached" is non zero when we break. So
LBF_ALL_PINNEDwould be cleared. We will only exit thedetach_tasksloop when there are truly no tasks that can be moved and it is truly aLBF_ALL_PINNEDcase.So what I was suggesting is something like:
(quoted diff) +8
-6@@ -10251,6 +10252,7 @@ static int detach_tasks(struct lb_env *env) unsigned long util, load; struct task_struct *p; int detached = 0; + bool preserve_preferred; lockdep_assert_rq_held(env->src_rq);@@ -10268,6 +10270,10 @@ static int detach_tasks(struct lb_env *env) tasks = order_tasks_by_llc(env, &env->src_rq->cfs_tasks); + preserve_preferred = sched_cache_enabled() && + !(env->sd->flags & SD_SHARE_LLC) && + !sd->nr_balance_failed; + while (!list_empty(tasks)) { /* * We don't want to steal all, otherwise we may be treated likewise,@@ -10370,16 +10376,15 @@ static int detach_tasks(struct lb_env *env) #ifdef CONFIG_SCHED_CACHE /* - * Don't detach more tasks if the remaining tasks want - * to stay. We know the remaining tasks all prefer the - * current LLC, because after order_tasks_by_llc(), the - * tasks that prefer the current LLC are at the tail of - * the list. The inhibition of detachment is to avoid too - * many tasks being migrated out of the preferred LLC. + * We've hit tasks that prefer src LLC while balancing between LLCs. + * If previous balances have been successful, pretend the rest of the + * tasks on this CPU are pinned and let the main load balancing loop
Review static int detach_tasks(struct lb_env *env) · line 10389
Quoted 10 lines
(quoted diff) +4
-2+ * find another target CPU to pull from if imbalance exists. */ - if (sched_cache_enabled() && detached && p->preferred_llc != -1 && - llc_id(env->src_cpu) == p->preferred_llc) + if (preserve_preferred && detached && llc_id(env->src_cpu) == p->preferred_llc) { + env->flags |= LBF_ALL_PINNED; break; + } #endif
You have a good point. If all the remaining tasks on the rq prefer src_llc, we should find an alternative rq to pull tasks from by specifying LBF_ALL_PINNED.
This policy makes sense for the migrate_task and migrate_llc_task case. I have to think about the migrate_util case, where the source group is overloaded and dst group has spare capacity, and tasks in the source group are in their preferred LLC.
Tim
PATCH 16/19 sched/fair: Exclude processes with many threads from cache-aware scheduling
In reply to Tim ChenFrom: Chen Yu <[email protected]>
A performance regression was observed by Prateek when running hackbench with many threads per process (high fd count). To avoid this, processes with a large number of active threads are excluded from cache-aware scheduling.
With sched_cache enabled, record the number of active threads in each process during the periodic task_cache_work(). While iterating over CPUs, if the currently running task belongs to the same process as the task that launched task_cache_work(), increment the active thread count.
If the count exceeds the number of CPUs in the process's preferred LLC, sched_cache will avoid aggregating too many threads into a single LLC domain.
--- include/linux/mm_types.h | 1 + kernel/sched/fair.c | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 2 deletions(-)
include/linux/mm_types.h +1 -0
@@ -1031,6 +1031,7 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; +u64 nr_running_avg____cacheline_aligned_in_smp ;#endif #ifdef CONFIG_MMU
kernel/sched/fair.c +12 -2
@@ -1451,12 +1451,13 @@ static void get_scan_cpumasks(cpumask_var_t cpus, int cache_cpu, static void __no_profile task_cache_work(struct callback_head *work) { -struct task_struct *p = current; +struct task_struct *p = current, *cur;struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ;unsigned long curr_m_a_occ =0 ;int cpu, m_a_cpu =-1 , cache_cpu, - pref_nid =NUMA_NO_NODE , curr_cpu; + pref_nid =NUMA_NO_NODE , curr_cpu, + nr_running =0 ;cpumask_var_t cpus;WARN_ON_ONCE (work != &p->cache_work);
@@ -1497,6 +1498,14 @@ static void __no_profile task_cache_work(struct callback_head *work) m_occ = occ; m_cpu = i; } + +rcu_read_lock (); + cur =rcu_dereference (cpu_rq (i)->curr); +if (cur && !(cur->flags & (PF_EXITING |PF_KTHREAD )) && + cur->mm == mm) + nr_running++; +rcu_read_unlock (); + }/*
@@ -1540,6 +1549,7 @@ static void __no_profile task_cache_work(struct callback_head *work) mm->mm_sched_cpu = m_a_cpu; } +update_avg (&mm->nr_running_avg, nr_running);free_cpumask_var (cpus); }
Signature · 1 line
2.32.0
kernel test robot
build botRe: [PATCH 16/19] sched/fair: Exclude processes with many threads from cache-aware scheduling
kernel test robot noticed a 2.1% regression of will-it-scale.per_thread_ops on:
Hello,
kernel test robot noticed a 2.1% regression of will-it-scale.per_thread_ops on:
commit: cb57b28051ef1d84e7cb14db4e1ab99b4f33b4b5 ("[PATCH 16/19] sched/fair: Exclude processes with many threads from cache-aware scheduling")
url: https://github.com/intel-lab-lkp/linux/commits/Tim-Chen/sched-fair-Add-infrastructure-for-cache-aware-load-balancing/20251012-022248
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 45b7f780739a3145aeef24d2dfa02517a6c82ed6
patch link: https://lore.kernel.org/all/637cdb8ab11b1b978d697ed744cc402d32443ecc.1760206683.git.tim.c.chen@linux.intel.com/
patch subject: [PATCH 16/19] sched/fair: Exclude processes with many threads from cache-aware scheduling
testcase: will-it-scale
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 48 threads 2 sockets Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (Ivy Bridge-EP) with 64G memory parameters:
nr_task: 100% mode: thread test: tlb_flush2 cpufreq_governor: performance
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20251023/[email protected]
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/thread/100%/debian-13-x86_64-20250902.cgz/lkp-ivb-2ep2/tlb_flush2/will-it-scale
commit:
4ac141e433 ("sched/fair: Respect LLC preference in task migration and detach") cb57b28051 ("sched/fair: Exclude processes with many threads from cache-aware scheduling")
4ac141e4330723c0 cb57b28051ef1d84e7cb14db4e1
---------------- ---------------------------
%stddev %change %stddev
\ | \
1482496 -2.1% 1451299 will-it-scale.48.threads
30884 -2.1% 30235 will-it-scale.per_thread_ops
1482496 -2.1% 1451299 will-it-scale.workload
4.447e+08 -2.1% 4.355e+08 proc-vmstat.numa_hit
4.447e+08 -2.1% 4.355e+08 proc-vmstat.numa_local
4.447e+08 -2.1% 4.354e+08 proc-vmstat.pgalloc_normal
8.884e+08 -2.1% 8.698e+08 proc-vmstat.pgfault
4.446e+08 -2.1% 4.353e+08 proc-vmstat.pgfree
6.446e+09 -2.0% 6.318e+09 perf-stat.i.branch-instructions
1.462e+08 -1.4% 1.441e+08 perf-stat.i.branch-misses
1.467e+08 -1.6% 1.444e+08 perf-stat.i.cache-misses
7.692e+08 -1.4% 7.587e+08 perf-stat.i.cache-references
101348 ± 2% +3.6% 104965 perf-stat.i.context-switches
4.14 +1.9% 4.22 perf-stat.i.cpi
883.41 +1.4% 896.20 perf-stat.i.cycles-between-cache-misses
3.083e+10 -2.0% 3.022e+10 perf-stat.i.instructions
0.24 -1.8% 0.24 perf-stat.i.ipc
124.71 -2.0% 122.18 perf-stat.i.metric.K/sec
2944589 -2.1% 2882055 perf-stat.i.minor-faults
2944589 -2.1% 2882055 perf-stat.i.page-faults
4.17 +1.9% 4.25 perf-stat.overall.cpi
876.76 +1.5% 889.96 perf-stat.overall.cycles-between-cache-misses
0.24 -1.8% 0.24 perf-stat.overall.ipc
6.417e+09 -2.0% 6.29e+09 perf-stat.ps.branch-instructions
1.455e+08 -1.4% 1.434e+08 perf-stat.ps.branch-misses
1.46e+08 -1.6% 1.436e+08 perf-stat.ps.cache-misses
7.653e+08 -1.4% 7.549e+08 perf-stat.ps.cache-references
100692 ± 2% +3.6% 104309 perf-stat.ps.context-switches
3.069e+10 -2.0% 3.008e+10 perf-stat.ps.instructions
2931887 -2.1% 2869944 perf-stat.ps.minor-faults
2931887 -2.1% 2869944 perf-stat.ps.page-faults
9.273e+12 -1.9% 9.096e+12 perf-stat.total.instructions
62.03 -1.8 60.18 perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.flush_tlb_mm_range.tlb_finish_mmu.do_madvise.__x64_sys_madvise
63.66 -1.8 61.82 perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.tlb_finish_mmu.do_madvise.__x64_sys_madvise.do_syscall_64
61.19 -1.8 59.36 perf-profile.calltrace.cycles-pp.smp_call_function_many_cond.on_each_cpu_cond_mask.flush_tlb_mm_range.tlb_finish_mmu.do_madvise
65.49 -1.7 63.79 perf-profile.calltrace.cycles-pp.tlb_finish_mmu.do_madvise.__x64_sys_madvise.do_syscall_64.entry_SYSCALL_64_after_hwframe
75.54 -1.5 74.02 perf-profile.calltrace.cycles-pp.__madvise
71.89 -1.5 70.41 perf-profile.calltrace.cycles-pp.__x64_sys_madvise.do_syscall_64.entry_SYSCALL_64_after_hwframe.__madvise
72.40 -1.5 70.92 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__madvise
71.83 -1.5 70.35 perf-profile.calltrace.cycles-pp.do_madvise.__x64_sys_madvise.do_syscall_64.entry_SYSCALL_64_after_hwframe.__madvise
72.35 -1.5 70.87 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__madvise
15.31 -0.6 14.70 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.smp_call_function_many_cond.on_each_cpu_cond_mask.flush_tlb_mm_range.tlb_finish_mmu
12.04 -0.5 11.52 perf-profile.calltrace.cycles-pp.sysvec_call_function.asm_sysvec_call_function.smp_call_function_many_cond.on_each_cpu_cond_mask.flush_tlb_mm_range
10.97 -0.5 10.47 perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.smp_call_function_many_cond
11.08 -0.5 10.58 perf-profile.calltrace.cycles-pp.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.smp_call_function_many_cond.on_each_cpu_cond_mask
4.36 -0.2 4.15 perf-profile.calltrace.cycles-pp.flush_tlb_func.__flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function
4.53 -0.2 4.34 perf-profile.calltrace.cycles-pp.llist_reverse_order.__flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function 2.02 ± 2% -0.1 1.95 perf-profile.calltrace.cycles-pp.lock_vma_under_rcu.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.testcase
1.47 -0.1 1.42 perf-profile.calltrace.cycles-pp.folio_add_lru.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
0.83 ± 2% -0.0 0.80 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.testcase
0.65 ± 2% -0.0 0.62 perf-profile.calltrace.cycles-pp.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.do_user_addr_fault.exc_page_fault
0.73 ± 2% -0.0 0.70 perf-profile.calltrace.cycles-pp.sysvec_call_function.asm_sysvec_call_function.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
0.64 ± 2% -0.0 0.61 ± 2% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.do_user_addr_fault
0.73 -0.0 0.71 perf-profile.calltrace.cycles-pp.__sysvec_call_function.sysvec_call_function.asm_sysvec_call_function.testcase
0.84 -0.0 0.82 perf-profile.calltrace.cycles-pp.clear_page_erms.prep_new_page.get_page_from_freelist.__alloc_frozen_pages_noprof.alloc_pages_mpol
1.65 +0.0 1.68 perf-profile.calltrace.cycles-pp.__alloc_frozen_pages_noprof.alloc_pages_mpol.vma_alloc_folio_noprof.alloc_anon_folio.do_anonymous_page
1.92 +0.0 1.96 perf-profile.calltrace.cycles-pp.vma_alloc_folio_noprof.alloc_anon_folio.do_anonymous_page.__handle_mm_fault.handle_mm_fault
1.79 +0.0 1.83 perf-profile.calltrace.cycles-pp.alloc_pages_mpol.vma_alloc_folio_noprof.alloc_anon_folio.do_anonymous_page.__handle_mm_fault
0.92 ± 3% +0.1 1.04 perf-profile.calltrace.cycles-pp.tlb_gather_mmu.do_madvise.__x64_sys_madvise.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.83 ± 6% +0.2 3.04 ± 2% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
7.06 +0.4 7.48 perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
6.25 +0.4 6.70 perf-profile.calltrace.cycles-pp.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
2.48 +0.5 3.02 perf-profile.calltrace.cycles-pp.alloc_anon_folio.do_anonymous_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
0.00 +0.7 0.74 ± 5% perf-profile.calltrace.cycles-pp.get_mem_cgroup_from_mm.__mem_cgroup_charge.alloc_anon_folio.do_anonymous_page.__handle_mm_fault
0.00 +0.9 0.94 ± 4% perf-profile.calltrace.cycles-pp.__mem_cgroup_charge.alloc_anon_folio.do_anonymous_page.__handle_mm_fault.handle_mm_fault
19.08 +1.3 20.36 perf-profile.calltrace.cycles-pp.testcase
14.17 +1.3 15.46 perf-profile.calltrace.cycles-pp.asm_exc_page_fault.testcase
12.74 +1.4 14.10 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.testcase
12.49 +1.4 13.85 perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.testcase
7.97 +1.4 9.38 perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.testcase
62.25 -1.9 60.39 perf-profile.children.cycles-pp.smp_call_function_many_cond
62.26 -1.9 60.40 perf-profile.children.cycles-pp.on_each_cpu_cond_mask
63.94 -1.8 62.08 perf-profile.children.cycles-pp.flush_tlb_mm_range
65.76 -1.7 64.06 perf-profile.children.cycles-pp.tlb_finish_mmu
75.72 -1.5 74.19 perf-profile.children.cycles-pp.__madvise
73.51 -1.5 72.02 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
73.48 -1.5 71.99 perf-profile.children.cycles-pp.do_syscall_64
71.90 -1.5 70.41 perf-profile.children.cycles-pp.__x64_sys_madvise
71.85 -1.5 70.36 perf-profile.children.cycles-pp.do_madvise
18.46 -0.3 18.12 perf-profile.children.cycles-pp.__flush_smp_call_function_queue
19.79 -0.3 19.47 perf-profile.children.cycles-pp.sysvec_call_function
22.68 -0.3 22.36 perf-profile.children.cycles-pp.asm_sysvec_call_function
17.97 -0.3 17.65 perf-profile.children.cycles-pp.__sysvec_call_function
7.64 -0.2 7.46 perf-profile.children.cycles-pp.flush_tlb_func
7.49 -0.1 7.39 perf-profile.children.cycles-pp.llist_reverse_order
1.47 -0.1 1.42 perf-profile.children.cycles-pp.folio_add_lru
1.99 -0.0 1.94 perf-profile.children.cycles-pp.__pte_offset_map_lock
1.84 -0.0 1.80 perf-profile.children.cycles-pp._raw_spin_lock
1.31 -0.0 1.27 perf-profile.children.cycles-pp.folio_batch_move_lru
0.93 -0.0 0.90 perf-profile.children.cycles-pp.error_entry
0.42 -0.0 0.40 perf-profile.children.cycles-pp.vms_clear_ptes
0.89 -0.0 0.87 perf-profile.children.cycles-pp.clear_page_erms
0.94 -0.0 0.92 perf-profile.children.cycles-pp.prep_new_page
1.66 +0.0 1.69 perf-profile.children.cycles-pp.__alloc_frozen_pages_noprof
1.80 +0.0 1.84 perf-profile.children.cycles-pp.alloc_pages_mpol
0.00 +0.1 0.05 perf-profile.children.cycles-pp.__pi_memset
0.96 ± 3% +0.1 1.08 perf-profile.children.cycles-pp.tlb_gather_mmu
2.90 ± 6% +0.2 3.10 perf-profile.children.cycles-pp.intel_idle
3.23 ± 5% +0.2 3.45 perf-profile.children.cycles-pp.cpuidle_enter
3.32 ± 5% +0.2 3.54 perf-profile.children.cycles-pp.cpuidle_idle_call
7.09 +0.4 7.51 perf-profile.children.cycles-pp.__handle_mm_fault
6.27 +0.4 6.72 perf-profile.children.cycles-pp.do_anonymous_page
0.43 ± 6% +0.5 0.94 ± 4% perf-profile.children.cycles-pp.__mem_cgroup_charge
0.25 ± 11% +0.5 0.76 ± 5% perf-profile.children.cycles-pp.get_mem_cgroup_from_mm
2.49 +0.5 3.03 perf-profile.children.cycles-pp.alloc_anon_folio
19.67 +1.3 20.94 perf-profile.children.cycles-pp.testcase
14.47 +1.3 15.77 perf-profile.children.cycles-pp.asm_exc_page_fault
12.76 +1.4 14.12 perf-profile.children.cycles-pp.exc_page_fault
12.60 +1.4 13.96 perf-profile.children.cycles-pp.do_user_addr_fault
7.99 +1.5 9.44 perf-profile.children.cycles-pp.handle_mm_fault
42.94 -1.2 41.71 perf-profile.self.cycles-pp.smp_call_function_many_cond
6.02 -0.2 5.87 perf-profile.self.cycles-pp.flush_tlb_func
7.46 -0.1 7.36 perf-profile.self.cycles-pp.llist_reverse_order
1.44 -0.0 1.40 perf-profile.self.cycles-pp.lock_vma_under_rcu
0.88 -0.0 0.85 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.91 -0.0 0.88 perf-profile.self.cycles-pp.error_entry
0.07 +0.0 0.08 ± 4% perf-profile.self.cycles-pp.get_page_from_freelist
0.76 ± 2% +0.1 0.86 perf-profile.self.cycles-pp.tlb_gather_mmu
1.10 ± 5% +0.1 1.24 ± 2% perf-profile.self.cycles-pp.tlb_finish_mmu
2.90 ± 6% +0.2 3.10 perf-profile.self.cycles-pp.intel_idle
0.20 ± 10% +0.4 0.62 ± 6% perf-profile.self.cycles-pp.get_mem_cgroup_from_mm
0.15 ± 4% +0.8 1.00 ± 3% perf-profile.self.cycles-pp.handle_mm_fault
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
PATCH 17/19 sched/fair: Disable cache aware scheduling for processes with high thread counts
In reply to Tim ChenFrom: Chen Yu <[email protected]>
If the number of active threads within the process exceeds the number of Cores(divided by SMTs number) in the LLC, do not enable cache-aware scheduling. This is because there is a risk of cache contention within the preferred LLC when too many threads are present.
--- kernel/sched/fair.c | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)
kernel/sched/fair.c +25 -2
@@ -1240,6 +1240,18 @@ static inline int pref_llc_idx(struct task_struct *p)return llc_idx (p->preferred_llc); } +static bool exceed_llc_nr (struct mm_struct *mm,int cpu) +{ +int smt_nr =1 ; + +#ifdef CONFIG_SCHED_SMT +if (sched_smt_active ()) + smt_nr =cpumask_weight (cpu_smt_mask (cpu)); +#endif + +return ((mm->nr_running_avg * smt_nr) >per_cpu (sd_llc_size, cpu)); +} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) {int pref_llc;
@@ -1385,10 +1397,12 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)/* * If this task hasn't hit task_cache_work() for a while, or it -* has only 1 thread, invalidate its preferred state. +* has only 1 thread, or has too many active threads, invalidate +* its preferred state. */ if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || -get_nr_threads (p) <=1 ) { +get_nr_threads (p) <=1 || +exceed_llc_nr (mm,cpu_of (rq))) {if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; }
@@ -1467,6 +1481,11 @@ static void __no_profile task_cache_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; +if (get_nr_threads (p) <=1 ) { + mm->mm_sched_cpu =-1 ; +return ; + } +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL ))return ;
@@ -9826,6 +9845,10 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,if (cpu <0 ||cpus_share_cache (src_cpu, dst_cpu))return mig_unrestricted; +/* skip cache aware load balance for single/too many threads */ +if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu)) +return mig_unrestricted; +if (cpus_share_cache (dst_cpu, cpu)) to_pref = true;else if (cpus_share_cache (src_cpu, cpu))
Signature · 1 line
2.32.0
Re: [PATCH 17/19] sched/fair: Disable cache aware scheduling for processes with high thread counts
Quoted 24 lines · Tim Chen
From: Chen Yu <[email protected]>
If the number of active threads within the process exceeds the number of Cores(divided by SMTs number) in the LLC, do not enable cache-aware scheduling. This is because there is a risk of cache contention within the preferred LLC when too many threads are present.
Reported-by K Prateek Nayak <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)kernel/sched/fair.c +0
-0@@ -1240,6 +1240,18 @@ static inline int pref_llc_idx(struct task_struct *p)return llc_idx (p->preferred_llc); }
kernel/sched/fair.c static inline int pref_llc_idx(struct task_struct *p) · line 1252
Quoted 10 lines
kernel/sched/fair.c +10
-0+static bool exceed_llc_nr (struct mm_struct *mm,int cpu) +{ +int smt_nr =1 ; + +#ifdef CONFIG_SCHED_SMT +if (sched_smt_active ()) + smt_nr =cpumask_weight (cpu_smt_mask (cpu)); +#endif + +return ((mm->nr_running_avg * smt_nr) >per_cpu (sd_llc_size, cpu));
In Power10 and Power11 that has SMT8 and LLC size of 4, this would disable cache aware scheduling even for one thread.
Also, llc_overload_pct already ensures the load on the preferred LLC doesn't exceed certain capacity. Why is this exceed_llc_nr() check needed? Won't the existing overload_pct naturally prevent excessive task aggregation by blocking migrations when the destination LLC reaches ~50% utilization?
Thanks,
Madadi Vineeth Reddy
Quoted 43 lines
kernel/sched/fair.c +15
-2+} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) {int pref_llc;@@ -1385,10 +1397,12 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)/* * If this task hasn't hit task_cache_work() for a while, or it -* has only 1 thread, invalidate its preferred state. +* has only 1 thread, or has too many active threads, invalidate +* its preferred state. */ if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || -get_nr_threads (p) <=1 ) { +get_nr_threads (p) <=1 || +exceed_llc_nr (mm,cpu_of (rq))) {if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; }@@ -1467,6 +1481,11 @@ static void __no_profile task_cache_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; +if (get_nr_threads (p) <=1 ) { + mm->mm_sched_cpu =-1 ; +return ; + } +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL ))return ;@@ -9826,6 +9845,10 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,if (cpu <0 ||cpus_share_cache (src_cpu, dst_cpu))return mig_unrestricted; +/* skip cache aware load balance for single/too many threads */ +if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu)) +return mig_unrestricted; +if (cpus_share_cache (dst_cpu, cpu)) to_pref = true;else if (cpus_share_cache (src_cpu, cpu))
Re: [PATCH 17/19] sched/fair: Disable cache aware scheduling for processes with high thread counts
Quoted 25 lines · Madadi Vineeth Reddy
Quoted 24 lines · Tim Chen
From: Chen Yu <[email protected]>
If the number of active threads within the process exceeds the number of Cores(divided by SMTs number) in the LLC, do not enable cache-aware scheduling. This is because there is a risk of cache contention within the preferred LLC when too many threads are present.
Reported-by K Prateek Nayak <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- kernel/sched/fair.c | 27 +++++++++++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-)kernel/sched/fair.c +0
-0@@ -1240,6 +1240,18 @@ static inline int pref_llc_idx(struct task_struct *p)return llc_idx (p->preferred_llc); }
Quoted 14 lines
kernel/sched/fair.c static inline int pref_llc_idx(struct task_struct *p) · line 1252
Quoted 10 lines
kernel/sched/fair.c +10
-0+static bool exceed_llc_nr (struct mm_struct *mm,int cpu) +{ +int smt_nr =1 ; + +#ifdef CONFIG_SCHED_SMT +if (sched_smt_active ()) + smt_nr =cpumask_weight (cpu_smt_mask (cpu)); +#endif + +return ((mm->nr_running_avg * smt_nr) >per_cpu (sd_llc_size, cpu));In Power10 and Power11 that has SMT8 and LLC size of 4, this would disable cache aware scheduling even for one thread.
Using smt_nr was mainly due to concerns about introducing regressions on Power, as discussed in v3
https://lore.kernel.org/all/[email protected]/
and
https://lore.kernel.org/all/[email protected]/
It seems that aggregating tasks on an LLC with many SMT threads/smaller LLC size would pose a risk of cache contention. Additionally, with patch [19/19], users can tune /sys/kernel/debug/sched/llc_aggr_tolerance to adjust the threshold:
return ((mm->nr_running_avg * smt_nr) > (scale * per_cpu(sd_llc_size, cpu)));
Quoted 5 lines
Also,
llc_overload_pctalready ensures the load on the preferred LLC doesn't exceed certain capacity. Why is thisexceed_llc_nr()check needed? Won't the existingoverload_pctnaturally prevent excessive task aggregation by blocking migrations when the destination LLC reaches ~50% utilization?
Using exceed_llc_nr() was because some short-duration tasks could
generate low utilization but still cause cache contention (for
some reason, the util_avg cannot track that properly), such as schbench. Therefore, we inhibit task aggregation for a large number of active threads.
thanks,
Chenyu
PATCH 18/19 sched/fair: Avoid cache-aware scheduling for memory-heavy processes
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Prateek and Tingyin reported that memory-intensive workloads (such as stream) can saturate memory bandwidth and caches on the preferred LLC when sched_cache aggregates too many threads.
To mitigate this, estimate a process's memory footprint by comparing its RSS (anonymous and shared pages) to the size of the LLC. If RSS exceeds the LLC size, skip cache-aware scheduling.
Note that RSS is only an approximation of the memory footprint. By default, the comparison is strict, but a later patch will allow users to provide a hint to adjust this threshold.
According to the test from Adam, some systems do not have shared L3 but with shared L2 as clusters. In this case, the L2 becomes the LLC[1].
Link[1]: https://lore.kernel.org/all/[email protected]/
--- include/linux/cacheinfo.h | 21 ++++++++++------ kernel/sched/fair.c | 51 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 61 insertions(+), 11 deletions(-)
include/linux/cacheinfo.h +13 -8
@@ -113,18 +113,11 @@ int acpi_get_cache_info(unsigned int cpu,const struct attribute_group *cache_get_priv_group (struct cacheinfo *this_leaf); -/* -* Get the cacheinfo structure for the cache associated with @cpu at -* level @level. -* cpuhp lock must be held. -*/ -static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu,int level) +static inline struct cacheinfo *_get_cpu_cacheinfo_level (int cpu,int level) {struct cpu_cacheinfo *ci =get_cpu_cacheinfo (cpu);int i; -lockdep_assert_cpus_held (); -for (i =0 ; i < ci->num_leaves; i++) {if (ci->info_list[i].level == level) {if (ci->info_list[i].attributes &CACHE_ID )
@@ -136,6 +129,18 @@ static inline struct cacheinfo *get_cpu_cacheinfo_level(int cpu, int level)return NULL ; } +/* +* Get the cacheinfo structure for the cache associated with @cpu at +* level @level. +* cpuhp lock must be held. +*/ +static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu,int level) +{ +lockdep_assert_cpus_held (); + +return _get_cpu_cacheinfo_level (cpu, level); +} +/* * Get the id of the cache associated with @cpu at level @level. * cpuhp lock must be held.
kernel/sched/fair.c +48 -3
75 lines not on this page · raw message
Signature · 1 line
2.32.0
kernel test robot
build botRe: [PATCH 18/19] sched/fair: Avoid cache-aware scheduling for memory-heavy processes
kernel test robot noticed "UBSAN:array-index-out-of-bounds_in_drivers/base/cacheinfo.c" on:
Hello,
kernel test robot noticed "UBSAN:array-index-out-of-bounds_in_drivers/base/cacheinfo.c" on:
commit: e8b871200f11decae96692a3f5b385cdc25af231 ("[PATCH 18/19] sched/fair: Avoid cache-aware scheduling for memory-heavy processes")
url: https://github.com/intel-lab-lkp/linux/commits/Tim-Chen/sched-fair-Add-infrastructure-for-cache-aware-load-balancing/20251012-022248
base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git 45b7f780739a3145aeef24d2dfa02517a6c82ed6
patch link: https://lore.kernel.org/all/00da49fd590b95baad0525660bda4c0ba178243d.1760206683.git.tim.c.chen@linux.intel.com/
patch subject: [PATCH 18/19] sched/fair: Avoid cache-aware scheduling for memory-heavy processes
in testcase: boot
config: i386-randconfig-003-20251012
compiler: clang-20
test machine: qemu-system-i386 -enable-kvm -cpu SandyBridge -smp 2 -m 4G
(please refer to attached dmesg/kmsg for entire log/backtrace)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-lkp/[email protected]
[ 12.549731][ T83] ------------[ cut here ]------------ [ 12.550388][ T83] UBSAN: array-index-out-of-bounds in drivers/base/cacheinfo.c:37:9 [ 12.551060][ T83] index 4294967295 is out of range for type 'unsigned long[8]' [ 12.551580][ T83] CPU: 0 UID: 0 PID: 83 Comm: systemd-journal Not tainted 6.17.0-rc4-00035-ge8b871200f11 #1 PREEMPTLAZY [ 12.551585][ T83] Call Trace: [ 12.551588][ T83] __dump_stack (lib/dump_stack.c:95) [ 12.551594][ T83] dump_stack_lvl (lib/dump_stack.c:123) [ 12.551601][ T83] ubsan_epilogue.llvm.16751680356772289369 (lib/dump_stack.c:129 lib/ubsan.c:233) [ 12.551607][ T83] __ubsan_handle_out_of_bounds (lib/ubsan.c:?) [ 12.551621][ T83] get_cpu_cacheinfo (drivers/base/cacheinfo.c:?) [ 12.551625][ T83] exceed_llc_capacity (include/linux/cacheinfo.h:? kernel/sched/fair.c:1256) [ 12.551632][ T83] task_cache_work.llvm.12119588225164800824 (kernel/sched/fair.c:1527) [ 12.551637][ T83] ? task_work_run (kernel/task_work.c:?) [ 12.551641][ T83] ? _raw_spin_unlock_irq (arch/x86/include/asm/irqflags.h:42 arch/x86/include/asm/irqflags.h:119 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 12.551644][ T83] ? __this_cpu_preempt_check (lib/smp_processor_id.c:65) [ 12.551648][ T83] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4472) [ 12.551650][ T83] ? _raw_spin_unlock_irq (arch/x86/include/asm/irqflags.h:42 arch/x86/include/asm/irqflags.h:119 include/linux/spinlock_api_smp.h:159 kernel/locking/spinlock.c:202) [ 12.551652][ T83] ? task_work_run (kernel/task_work.c:?) [ 12.551655][ T83] ? trace_hardirqs_on (kernel/trace/trace_preemptirq.c:80) [ 12.551662][ T83] task_work_run (kernel/task_work.c:229) [ 12.551668][ T83] resume_user_mode_work (include/linux/resume_user_mode.h:?) [ 12.551673][ T83] irqentry_exit_to_user_mode (kernel/entry/common.c:53 include/linux/irq-entry-common.h:225 kernel/entry/common.c:73) [ 12.551676][ T83] ? sysvec_call_function_single (arch/x86/kernel/apic/apic.c:1050) [ 12.551681][ T83] irqentry_exit (kernel/entry/common.c:210) [ 12.551684][ T83] sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1050) [ 12.551689][ T83] handle_exception (arch/x86/entry/entry_32.S:1055) [ 12.551691][ T83] EIP: 0x3764e8f0 [ 12.551694][ T83] Code: 00 00 89 c2 eb d2 65 c7 05 28 02 00 00 ff ff ff ff 65 a1 08 00 00 00 f0 83 88 84 00 00 00 10 65 a1 80 00 00 00 e8 f0 02 00 00 <8b> 44 24 04 8b 54 24 08 89 10 8b 54 24 0c 89 50 04 65 8b 15 7c 00 All code ======== 0: 00 00 add %al,(%rax) 2: 89 c2 mov %eax,%edx 4: eb d2 jmp 0xffffffffffffffd8 6: 65 c7 05 28 02 00 00 movl $0xffffffff,%gs:0x228(%rip) # 0x239 d: ff ff ff ff 11: 65 a1 08 00 00 00 f0 movabs %gs:0x848883f000000008,%eax 18: 83 88 84 1b: 00 00 add %al,(%rax) 1d: 00 10 add %dl,(%rax) 1f: 65 a1 80 00 00 00 e8 movabs %gs:0x2f0e800000080,%eax 26: f0 02 00 29:* 00 8b 44 24 04 8b add %cl,-0x74fbdbbc(%rbx) <-- trapping instruction 2f: 54 push %rsp 30: 24 08 and $0x8,%al 32: 89 10 mov %edx,(%rax) 34: 8b 54 24 0c mov 0xc(%rsp),%edx 38: 89 50 04 mov %edx,0x4(%rax) 3b: 65 gs 3c: 8b .byte 0x8b 3d: 15 .byte 0x15 3e: 7c 00 jl 0x40
Code starting with the faulting instruction
===========================================
0: 8b 44 24 04 mov 0x4(%rsp),%eax 4: 8b 54 24 08 mov 0x8(%rsp),%edx 8: 89 10 mov %edx,(%rax) a: 8b 54 24 0c mov 0xc(%rsp),%edx e: 89 50 04 mov %edx,0x4(%rax) 11: 65 gs 12: 8b .byte 0x8b 13: 15 .byte 0x15 14: 7c 00 jl 0x16 [ 12.551696][ T83] EAX: 3fdacc1c EBX: 36a49ba8 ECX: 36a49d64 EDX: 00000001 [ 12.551697][ T83] ESI: 00000000 EDI: 36a49b40 EBP: 00000000 ESP: 3fdacbdc [ 12.551699][ T83] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00200212 [ 12.551708][ T83] ? sysvec_call_function_single (arch/x86/kernel/apic/apic.c:1050) [ 12.551717][ T83] ---[ end trace ]---
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20251015/[email protected]
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Re: [PATCH 18/19] sched/fair: Avoid cache-aware scheduling for memory-heavy processes
Quoted 5 lines · kernel test robot
Hello,
kernel test robot noticed "UBSAN:array-index-out-of-bounds_in_drivers/base/cacheinfo.c" on:
[snip]
Quoted 11 lines
[ 12.549731][ T83] ------------[ cut here ]------------ [ 12.550388][ T83] UBSAN: array-index-out-of-bounds in drivers/base/cacheinfo.c:37:9 [ 12.551060][ T83] index 4294967295 is out of range for type 'unsigned long[8]' [ 12.551580][ T83] CPU: 0 UID: 0 PID: 83 Comm: systemd-journal Not tainted 6.17.0-rc4-00035-ge8b871200f11 #1 PREEMPTLAZY [ 12.551585][ T83] Call Trace: [ 12.551588][ T83] __dump_stack (lib/dump_stack.c:95) [ 12.551594][ T83] dump_stack_lvl (lib/dump_stack.c:123) [ 12.551601][ T83] ubsan_epilogue.llvm.16751680356772289369 (lib/dump_stack.c:129 lib/ubsan.c:233) [ 12.551607][ T83] __ubsan_handle_out_of_bounds (lib/ubsan.c:?) [ 12.551621][ T83] get_cpu_cacheinfo (drivers/base/cacheinfo.c:?) [ 12.551625][ T83] exceed_llc_capacity (include/linux/cacheinfo.h:? kernel/sched/fair.c:1256)
Thanks 0day! It seems that for some reason the assignment of curr_cpu = task_cpu(p); was after the check exceed_llc_capacity(), will fix this.
thanks,
Chenyu
PATCH 19/19 sched/fair: Add user control to adjust the tolerance of cache-aware scheduling
In reply to Tim ChenFrom: Chen Yu <[email protected]>
With sched_cache enabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.
Introduce /sys/kernel/debug/sched/llc_aggr_tolerance to let users control how strictly RSS limits aggregation. Values range from 0 to 100:
- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.
For example, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped
(784GB = (1 + (99 - 1) * 256) * 32MB).
Similarly, /sys/kernel/debug/sched/llc_aggr_tolerance also controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.
For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with nr_running > 8 are skipped.
- llc_aggr_tolerance=99 -> tasks with nr_running > 785 are skipped
785 = (1 + (99 - 1) * 8).
--- kernel/sched/debug.c | 56 ++++++++++++++++++++++++++++++-- kernel/sched/fair.c | 76 ++++++++++++++++++++++++++++++++++++++++---- kernel/sched/sched.h | 3 ++ 3 files changed, 126 insertions(+), 9 deletions(-)
kernel/sched/debug.c +54 -2
@@ -169,6 +169,50 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int percent; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &percent)) \ + return -EINVAL; \ + if (percent > 100) \ + return -EINVAL; \ + llc_##name = percent; \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct); +SCHED_CACHE_CREATE_CONTROL (imb_pct); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance); +#endif /* SCHED_CACHE */ +static ssize_t sched_scaling_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) {
@@ -524,8 +568,16 @@ static __init int sched_init_debug(void)#endif /* CONFIG_NUMA_BALANCING */ #ifdef CONFIG_SCHED_CACHE -debugfs_create_u32 ("llc_overload_pct" ,0644 , debugfs_sched, &llc_overload_pct); -debugfs_create_u32 ("llc_imb_pct" ,0644 , debugfs_sched, &llc_imb_pct); +debugfs_create_file ("llc_overload_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_overload_pct); +debugfs_create_file ("llc_imb_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_imb_pct); +debugfs_create_file ("llc_aggr_tolerance" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_tolerance); +debugfs_create_u32 ("llc_epoch_period" ,0644 , debugfs_sched, + &llc_epoch_period); +debugfs_create_u32 ("llc_epoch_affinity_timeout" ,0644 , debugfs_sched, + &llc_epoch_affinity_timeout);#endif debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);
kernel/sched/fair.c +69 -7
126 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -2810,6 +2810,9 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout;extern struct static_key_false sched_cache_allowed;#endif
Signature · 1 line
2.32.0
Re: [PATCH 19/19] sched/fair: Add user control to adjust the tolerance of cache-aware scheduling
On Sat, Oct 11, 2025 at 11:24:56AM -0700, Tim Chen wrote:
... ...
Quoted 5 lines
kernel/sched/fair.c +5
-0+static inline int get_sched_cache_cap_scale (void ) +{ +return (llc_overload_pct / cpu_smt_num_threads); +} +
... ...
kernel/sched/fair.c static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_ · line 9815
Quoted 7 lines
kernel/sched/fair.c +1
-1@@ -9749,7 +9811,7 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_* (default: ~50%) */ #define fits_llc_capacity (util, max) \ - ((util) * 100 < (max) * llc_overload_pct) + ((util) * 100 < (max) * get_sched_cache_cap_scale())
With this change, fits_llc_capacity() would be false if util is just 1/4 of max(which is llc's total cpu capacity), is this intended?
Re: [PATCH 19/19] sched/fair: Add user control to adjust the tolerance of cache-aware scheduling
Quoted 8 lines · Aaron Lu
On Sat, Oct 11, 2025 at 11:24:56AM -0700, Tim Chen wrote:
... ...Quoted 5 lines
kernel/sched/fair.c +5
-0+static inline int get_sched_cache_cap_scale (void ) +{ +return (llc_overload_pct / cpu_smt_num_threads); +} +... ...
Quoted 10 lines
kernel/sched/fair.c static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_ · line 9815
Quoted 7 lines
kernel/sched/fair.c +1
-1@@ -9749,7 +9811,7 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_* (default: ~50%) */ #define fits_llc_capacity (util, max) \ - ((util) * 100 < (max) * llc_overload_pct) + ((util) * 100 < (max) * get_sched_cache_cap_scale())With this change,
fits_llc_capacity()would be false if util is just 1/4 of max(which is llc's total cpu capacity), is this intended?
Yes, it was changed to this because we want to avoid performance regressions in some systems with a large number of SMTs per core-aggressive task aggregation
is harmful to those systems. However, upon further thought, since we
have a
user-space knob to control how aggressively users want to enable task aggregation, we can try removing cpu_smt_num_threads and let users decide. I'll do some tests to check the impact.
thanks,
Chenyu
Hi Tim,
Thanks for the patch.
Quoted 5 lines · Tim Chen
There had been 4 RFC postings of this patch set. We've incorporated the feedbacks and comments and now would like to post this patch set for consideration of inclusion to mainline. The patches are based on the original patch proposed by Peter[1].
[snip]
Quoted 30 lines
The following tunables control under
/sys/kernel/debug/sched/control the behavior of cache aware scheduling:1.
llc_aggr_toleranceControls how aggressive we aggregate tasks to their preferred LLC, based on a process's RSS size and number of running threads. Processes that have smaller memory footprint and fewer number of tasks will benefit better from aggregation. Varies between 0 to 100 0: Cache aware scheduling is disabled 1: Process with RSS greater than LLC size,
or running threads more than number of cpu cores/LLC skip aggregation
100: Aggressive; a process's threads are aggregated regardless of
RSS or running threads.
For example, with a 32MB L3 cache 8 cores in L3:llc_aggr_tolerance=1 -> process with RSS > 32MB, or nr_running_avg > 8 are skipped. llc_aggr_tolerance=99 -> process with RSS > 784GB or nr_running_avg > 785 are skipped. 784GB = (1 + (99 - 1) * 256) * 32MB. 785 = (1 + (99 - 1) * 8).Currently this knob is a global control. Considering that different workloads have different requirements for task consolidation, it would be ideal to introduce per process control for this knob via prctl in the future.
2.
llc_overload_pct,llc_imb_pct
We'll always try to move a task to its preferred LLC if an LLC's average core utilization is belowllc_overload_pct(default to 50%). Otherwise, the utilization of preferred LLC has to be not more thanllc_imb_pct(default to 20%) to move a task to it. This is to prevent overloading on the preferred LLC.
Quoted 6 lines
3.
llc_epoch_period
Controls how often the scheduler collect LLC occupancy of a process (default to 10 msec)4.
llc_epoch_affinity_timeout
Detect that if a process has not run forllc_epoch_affinity_timeout(default to 50 msec), it loses its cache preference.
How are these default values arrived at? Is it based on some theory or based on the results of the runs?
Quoted 182 lines
Test results:
The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.The second test platform is a AMD Milan. There are 2 Nodes and 64 CPUs per node. Each node has 8 CCXs and each CCX has 8 CPUs.
The third test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.
[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when there is 1 group with different number of fd pairs(threads) within this process. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows ~10% throughput improvement. Other micro-workloads did not show much difference.Milan:
No obvious difference is observed so far.Genoa:
ChaCha20-xiangshan shows 44% throughput improvement.[Sapphire Rapids details]
[hackbench]
Hackbench show overall improvement when there is only 1 group, with different number of fd(pairs). This is the expected behavior because this test scenario would benefit from cache aware load balance most. Other number of groups shows not much difference(using default fd = 20).groups baseline
sched_cacheMin 1 37.5960 ( 0.00%) 26.4340 ( 29.69%) Min 3 38.7050 ( 0.00%) 38.6920 ( 0.03%) Min 5 39.4550 ( 0.00%) 38.6280 ( 2.10%) Min 7 51.4270 ( 0.00%) 50.6790 ( 1.45%) Min 12 62.8540 ( 0.00%) 63.6590 ( -1.28%) Min 16 74.0160 ( 0.00%) 74.7480 ( -0.99%) Amean 1 38.4768 ( 0.00%) 26.7146 * 30.57%* Amean 3 39.0750 ( 0.00%) 39.5586 ( -1.24%) Amean 5 41.5178 ( 0.00%) 41.2766 ( 0.58%) Amean 7 52.1164 ( 0.00%) 51.5152 ( 1.15%) Amean 12 63.9052 ( 0.00%) 64.0420 ( -0.21%) Amean 16 74.5812 ( 0.00%) 75.4318 ( -1.14%) BAmean-99 1 38.2027 ( 0.00%) 26.5500 ( 30.50%) BAmean-99 3 38.8725 ( 0.00%) 39.2225 ( -0.90%) BAmean-99 5 41.1898 ( 0.00%) 41.0037 ( 0.45%) BAmean-99 7 51.8645 ( 0.00%) 51.4453 ( 0.81%) BAmean-99 12 63.6317 ( 0.00%) 63.9307 ( -0.47%) BAmean-99 16 74.4528 ( 0.00%) 75.2113 ( -1.02%)[schbench]
Wakeup Latencies 99.0th improvement is observed.threads baseline sched_cache change 1 13.80(1.10) 14.80(2.86) -7.25% 2 12.00(1.00) 8.00(2.12) +33.33% 4 9.00(0.00) 5.60(0.89) +37.78% 8 9.00(0.00) 6.40(1.14) +28.89% 16 9.20(0.45) 6.20(0.84) +32.61% 32 9.60(0.55) 7.00(0.71) +27.08% 64 10.80(0.45) 8.40(0.55) +22.22% 128 12.60(0.55) 11.40(0.55) +9.52% 239 14.00(0.00) 14.20(0.45) -1.43%[stream]
No much difference is observed.
baseline scGB/sec copy-2 35.00 ( 0.00%) 34.79 ( -0.60%) GB/sec scale-2 24.04 ( 0.00%) 23.90 ( -0.58%) GB/sec add-2 28.98 ( 0.00%) 28.92 ( -0.22%) GB/sec triad-2 28.32 ( 0.00%) 28.31 ( -0.04%)[netperf]
No much difference is observed(consider the stdev).
nr_pairsnetperf netperfHmean 60 1023.44 ( 0.00%) 1021.87 ( -0.15%) BHmean-99 60 1023.78 ( 0.00%) 1022.22 ( -0.15%) Hmean 120 792.09 ( 0.00%) 793.75 ( 0.21%) BHmean-99 120 792.36 ( 0.00%) 794.04 ( 0.21%) Hmean 180 513.42 ( 0.00%) 513.53 ( 0.02%) BHmean-99 180 513.81 ( 0.00%) 513.80 ( -0.00%) Hmean 240 387.09 ( 0.00%) 387.33 ( 0.06%) BHmean-99 240 387.18 ( 0.00%) 387.45 ( 0.07%) Hmean 300 316.04 ( 0.00%) 315.68 ( -0.12%) BHmean-99 300 316.12 ( 0.00%) 315.77 ( -0.11%) Hmean 360 496.38 ( 0.00%) 455.49 ( -8.24%) BHmean-99 360 499.88 ( 0.00%) 458.17 ( -8.34%) Hmean 420 497.32 ( 0.00%) 501.84 ( 0.91%) BHmean-99 420 499.90 ( 0.00%) 504.56 ( 0.93%) Hmean 480 417.62 ( 0.00%) 432.25 ( 3.50%) BHmean-99 480 419.96 ( 0.00%) 434.43 ( 3.45%)In above case of 360 pairs, although there is a performance drop of 8.24%, the corresponding:
HCoeffVar 360 23.78 ( 0.00%) 29.52 ( -24.15%)
shows that the regression is within the run-to-run variance.[Milan details]
default settings:
[hackbench]Min 1 50.8170 ( 0.00%) 51.1890 ( -0.73%) Min 3 59.3610 ( 0.00%) 58.6080 ( 1.27%) Min 5 94.9760 ( 0.00%) 96.0210 ( -1.10%) Min 7 123.3270 ( 0.00%) 124.1680 ( -0.68%) Min 12 179.2000 ( 0.00%) 181.8390 ( -1.47%) Min 16 238.8680 ( 0.00%) 242.6390 ( -1.58%) Amean 1 51.6614 ( 0.00%) 51.3630 ( 0.58%) Amean 3 60.1886 ( 0.00%) 59.4542 ( 1.22%) Amean 5 95.7602 ( 0.00%) 96.8338 ( -1.12%) Amean 7 124.0332 ( 0.00%) 124.4406 ( -0.33%) Amean 12 181.0324 ( 0.00%) 182.9220 ( -1.04%) Amean 16 239.5556 ( 0.00%) 243.3556 * -1.59%* BAmean-99 1 51.5335 ( 0.00%) 51.3338 ( 0.39%) BAmean-99 3 59.7848 ( 0.00%) 59.0958 ( 1.15%) BAmean-99 5 95.6698 ( 0.00%) 96.5450 ( -0.91%) BAmean-99 7 123.8478 ( 0.00%) 124.3760 ( -0.43%) BAmean-99 12 180.8035 ( 0.00%) 182.5135 ( -0.95%) BAmean-99 16 239.1933 ( 0.00%) 243.0570 ( -1.62%)[schbench]
threads baseline sched_cache change 1 12.00(2.00) 11.00(0.71) +8.33% 2 12.40(0.89) 13.80(0.84) -11.29% 4 14.20(0.45) 14.80(0.45) -4.23% 8 16.00(0.00) 15.80(0.45) +1.25% 16 16.00(0.00) 16.00(0.71) 0.00% 32 19.40(0.55) 18.60(0.55) +4.12% 63 22.20(0.45) 23.20(0.45) -4.50%[stream]
No obvious difference is found.
export STREAM_SIZE=$((128000000))baseline sched_cache GB/sec copy-16 726.48 ( 0.00%) 715.60 ( -1.50%) GB/sec scale-16 577.71 ( 0.00%) 577.03 ( -0.12%) GB/sec add-16 678.85 ( 0.00%) 672.87 ( -0.88%) GB/sec triad-16 735.52 ( 0.00%) 729.05 ( -0.88%)[netperf]
No much difference is observed.
nr_pairsbaselinesched_cacheHmean 32 755.98 ( 0.00%) 755.17 ( -0.11%) BHmean-99 32 756.42 ( 0.00%) 755.40 ( -0.13%) Hmean 64 677.38 ( 0.00%) 669.75 ( -1.13%) BHmean-99 64 677.50 ( 0.00%) 669.86 ( -1.13%) Hmean 96 498.52 ( 0.00%) 496.73 ( -0.36%) BHmean-99 96 498.69 ( 0.00%) 496.93 ( -0.35%) Hmean 128 604.38 ( 0.00%) 604.22 ( -0.03%) BHmean-99 128 604.87 ( 0.00%) 604.87 ( 0.00%) Hmean 160 471.67 ( 0.00%) 468.29 ( -0.72%) BHmean-99 160 474.34 ( 0.00%) 471.05 ( -0.69%) Hmean 192 381.18 ( 0.00%) 384.88 ( 0.97%) BHmean-99 192 383.30 ( 0.00%) 386.82 ( 0.92%) Hmean 224 327.79 ( 0.00%) 326.05 ( -0.53%) BHmean-99 224 329.85 ( 0.00%) 327.87 ( -0.60%) Hmean 256 284.61 ( 0.00%) 300.52 ( 5.59%) BHmean-99 256 286.41 ( 0.00%) 302.06 ( 5.47%)[Genoa details]
[ChaCha20-xiangshan]
ChaCha20-xiangshan is a simple benchmark using a static build of an 8-thread Verilator of XiangShan(RISC-V). The README file can be found here[2]. The score depends on how aggressive the user set the/sys/kernel/debug/sched/llc_aggr_tolerance. Using the default values, there is no much difference observed. While setting the/sys/kernel/debug/sched/llc_aggr_toleranceto 100, 44% improvment is observed.
Quoted 7 lines
baseline:
Host time spent: 50,868ms
sched_cache:
Host time spent: 28,349msThe time has been reduced by 44%.
Milan showed no improvement across all benchmarks, which could be due to the CCX topology (8 CCXs × 8 CPUs) where the LLC domain is too small for this optimization to be effective. Moreover there could be overhead due to additional computations.
ChaCha20-xiangshan improvement in Genoa when llc_aggr_tolerance is set to 100 seems due to having relatively lesser thread count. Please provide the numbers with default values too. Would like to know numbers on varying loads.
In Power 10 and Power 11, the LLC size is 4 threads which is even smaller. Not expecting improvements here but will run some workloads and share the data.
Not gone through the entire series yet but are the situations like say in two NUMA system, if a task's preferred LLC is on the wrong NUMA node for its memory, which takes precedence?
Also, what about the workloads that don't share data like stress-ng? It will be good to make sure that most other workloads don't suffer. As mentioned, per process knob for llc_aggr_tolerance could help.
Thanks,
Madadi Vineeth Reddy
Quoted 70 lines
Thanks to everyone who participated and provided valuable suggestions for the previous versions. Your comments and tests on the latest version are also greatly appreciated in advance.
Tim
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://github.com/yu-chen-surf/chacha20-xiangshan/blob/master/README.eng.md
RFC v4:
[3] https://lore.kernel.org/all/[email protected]/RFC v3
[4] https://lore.kernel.org/all/[email protected]/RFC v2:
[5] https://lore.kernel.org/lkml/[email protected]/Chen Yu (7):
sched/fair: Record per-LLC utilization to guide cache-aware scheduling
decisions
sched/fair: Introduce helper functions to enforce LLC migration policy sched/fair: Introduce a static key to enable cache aware only for
multi LLCs
sched/fair: Exclude processes with many threads from cache-aware
scheduling
sched/fair: Disable cache aware scheduling for processes with high
thread counts
sched/fair: Avoid cache-aware scheduling for memory-heavy processes sched/fair: Add user control to adjust the tolerance of cache-aware
schedulingPeter Zijlstra (Intel) (1):
sched/fair: Add infrastructure for cache-aware load balancingTim Chen (11):
sched/fair: Add LLC index mapping for CPUs
sched/fair: Assign preferred LLC ID to processes sched/fair: Track LLC-preferred tasks per runqueue sched/fair: Introduce per runqueue task LLC preference counter sched/fair: Count tasks prefering each LLC in a sched group sched/fair: Prioritize tasks preferring destination LLC during
balancing
sched/fair: Identify busiestsched_groupfor LLC-aware load balancing sched/fair: Addmigrate_llc_taskmigration type for cache-aware
balancing
sched/fair: Handle moving single tasks to/from their preferred LLC sched/fair: Consider LLC preference when selecting tasks for load
balancing
sched/fair: Respect LLC preference in task migration and detachinclude/linux/cacheinfo.h | 21 +- include/linux/mm_types.h | 45 ++ include/linux/sched.h | 5 + include/linux/sched/topology.h | 4 + include/linux/threads.h | 10 + init/Kconfig | 20 + init/init_task.c | 3 + kernel/fork.c | 6 + kernel/sched/core.c | 18 + kernel/sched/debug.c | 56 ++ kernel/sched/fair.c | 1022 +++++++++++++++++++++++++++++++- kernel/sched/features.h | 1 + kernel/sched/sched.h | 27 + kernel/sched/topology.c | 61 +- 14 files changed, 1283 insertions(+), 16 deletions(-)
Quoted 45 lines · Madadi Vineeth Reddy
Hi Tim,
Thanks for the patch.Quoted 5 lines · Tim Chen
There had been 4 RFC postings of this patch set. We've incorporated the feedbacks and comments and now would like to post this patch set for consideration of inclusion to mainline. The patches are based on the original patch proposed by Peter[1].
[snip]
Quoted 33 lines
The following tunables control under
/sys/kernel/debug/sched/control the behavior of cache aware scheduling:1.
llc_aggr_toleranceControls how aggressive we aggregate tasks to their preferred LLC, based on a process's RSS size and number of running threads. Processes that have smaller memory footprint and fewer number of tasks will benefit better from aggregation. Varies between 0 to 100 0: Cache aware scheduling is disabled 1: Process with RSS greater than LLC size,
or running threads more than number of cpu cores/LLC skip aggregation
100: Aggressive; a process's threads are aggregated regardless of
RSS or running threads.
For example, with a 32MB L3 cache 8 cores in L3:llc_aggr_tolerance=1 -> process with RSS > 32MB, or nr_running_avg > 8 are skipped. llc_aggr_tolerance=99 -> process with RSS > 784GB or nr_running_avg > 785 are skipped. 784GB = (1 + (99 - 1) * 256) * 32MB. 785 = (1 + (99 - 1) * 8).Currently this knob is a global control. Considering that different workloads have different requirements for task consolidation, it would be ideal to introduce per process control for this knob via prctl in the future.
2.
llc_overload_pct,llc_imb_pct
We'll always try to move a task to its preferred LLC if an LLC's average core utilization is belowllc_overload_pct(default to 50%). Otherwise, the utilization of preferred LLC has to be not more thanllc_imb_pct(default to 20%) to move a task to it. This is to prevent overloading on the preferred LLC.3.
llc_epoch_period
Controls how often the scheduler collect LLC occupancy of a process (default to 10 msec)
Quoted 6 lines
Quoted 3 lines
4.
llc_epoch_affinity_timeout
Detect that if a process has not run forllc_epoch_affinity_timeout(default to 50 msec), it loses its cache preference.How are these default values arrived at? Is it based on some theory or based on the results of the runs?
Right now the default value of llc_aggr_tolerance is fairly conservative. We make sure that we don't cause regressions to workloads we tested.
Knobs like llc_overload_pct, llc_imb_pct are actually chosen from Len's Yogini micro-benchmark experiments we did that gave good aggregation without overloading LLC.
llc_epoch_period and llc_epoch_affinity_timeout are from Peter's original patch that seem to work fairly well so we left it as is.
Quoted 191 lines
Quoted 183 lines
Test results:
The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.The second test platform is a AMD Milan. There are 2 Nodes and 64 CPUs per node. Each node has 8 CCXs and each CCX has 8 CPUs.
The third test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.
[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when there is 1 group with different number of fd pairs(threads) within this process. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows ~10% throughput improvement. Other micro-workloads did not show much difference.Milan:
No obvious difference is observed so far.Genoa:
ChaCha20-xiangshan shows 44% throughput improvement.[Sapphire Rapids details]
[hackbench]
Hackbench show overall improvement when there is only 1 group, with different number of fd(pairs). This is the expected behavior because this test scenario would benefit from cache aware load balance most. Other number of groups shows not much difference(using default fd = 20).groups baseline
sched_cacheMin 1 37.5960 ( 0.00%) 26.4340 ( 29.69%) Min 3 38.7050 ( 0.00%) 38.6920 ( 0.03%) Min 5 39.4550 ( 0.00%) 38.6280 ( 2.10%) Min 7 51.4270 ( 0.00%) 50.6790 ( 1.45%) Min 12 62.8540 ( 0.00%) 63.6590 ( -1.28%) Min 16 74.0160 ( 0.00%) 74.7480 ( -0.99%) Amean 1 38.4768 ( 0.00%) 26.7146 * 30.57%* Amean 3 39.0750 ( 0.00%) 39.5586 ( -1.24%) Amean 5 41.5178 ( 0.00%) 41.2766 ( 0.58%) Amean 7 52.1164 ( 0.00%) 51.5152 ( 1.15%) Amean 12 63.9052 ( 0.00%) 64.0420 ( -0.21%) Amean 16 74.5812 ( 0.00%) 75.4318 ( -1.14%) BAmean-99 1 38.2027 ( 0.00%) 26.5500 ( 30.50%) BAmean-99 3 38.8725 ( 0.00%) 39.2225 ( -0.90%) BAmean-99 5 41.1898 ( 0.00%) 41.0037 ( 0.45%) BAmean-99 7 51.8645 ( 0.00%) 51.4453 ( 0.81%) BAmean-99 12 63.6317 ( 0.00%) 63.9307 ( -0.47%) BAmean-99 16 74.4528 ( 0.00%) 75.2113 ( -1.02%)[schbench]
Wakeup Latencies 99.0th improvement is observed.threads baseline sched_cache change 1 13.80(1.10) 14.80(2.86) -7.25% 2 12.00(1.00) 8.00(2.12) +33.33% 4 9.00(0.00) 5.60(0.89) +37.78% 8 9.00(0.00) 6.40(1.14) +28.89% 16 9.20(0.45) 6.20(0.84) +32.61% 32 9.60(0.55) 7.00(0.71) +27.08% 64 10.80(0.45) 8.40(0.55) +22.22% 128 12.60(0.55) 11.40(0.55) +9.52% 239 14.00(0.00) 14.20(0.45) -1.43%[stream]
No much difference is observed.
baseline scGB/sec copy-2 35.00 ( 0.00%) 34.79 ( -0.60%) GB/sec scale-2 24.04 ( 0.00%) 23.90 ( -0.58%) GB/sec add-2 28.98 ( 0.00%) 28.92 ( -0.22%) GB/sec triad-2 28.32 ( 0.00%) 28.31 ( -0.04%)[netperf]
No much difference is observed(consider the stdev).
nr_pairsnetperf netperfHmean 60 1023.44 ( 0.00%) 1021.87 ( -0.15%) BHmean-99 60 1023.78 ( 0.00%) 1022.22 ( -0.15%) Hmean 120 792.09 ( 0.00%) 793.75 ( 0.21%) BHmean-99 120 792.36 ( 0.00%) 794.04 ( 0.21%) Hmean 180 513.42 ( 0.00%) 513.53 ( 0.02%) BHmean-99 180 513.81 ( 0.00%) 513.80 ( -0.00%) Hmean 240 387.09 ( 0.00%) 387.33 ( 0.06%) BHmean-99 240 387.18 ( 0.00%) 387.45 ( 0.07%) Hmean 300 316.04 ( 0.00%) 315.68 ( -0.12%) BHmean-99 300 316.12 ( 0.00%) 315.77 ( -0.11%) Hmean 360 496.38 ( 0.00%) 455.49 ( -8.24%) BHmean-99 360 499.88 ( 0.00%) 458.17 ( -8.34%) Hmean 420 497.32 ( 0.00%) 501.84 ( 0.91%) BHmean-99 420 499.90 ( 0.00%) 504.56 ( 0.93%) Hmean 480 417.62 ( 0.00%) 432.25 ( 3.50%) BHmean-99 480 419.96 ( 0.00%) 434.43 ( 3.45%)In above case of 360 pairs, although there is a performance drop of 8.24%, the corresponding:
HCoeffVar 360 23.78 ( 0.00%) 29.52 ( -24.15%)
shows that the regression is within the run-to-run variance.[Milan details]
default settings:
[hackbench]Min 1 50.8170 ( 0.00%) 51.1890 ( -0.73%) Min 3 59.3610 ( 0.00%) 58.6080 ( 1.27%) Min 5 94.9760 ( 0.00%) 96.0210 ( -1.10%) Min 7 123.3270 ( 0.00%) 124.1680 ( -0.68%) Min 12 179.2000 ( 0.00%) 181.8390 ( -1.47%) Min 16 238.8680 ( 0.00%) 242.6390 ( -1.58%) Amean 1 51.6614 ( 0.00%) 51.3630 ( 0.58%) Amean 3 60.1886 ( 0.00%) 59.4542 ( 1.22%) Amean 5 95.7602 ( 0.00%) 96.8338 ( -1.12%) Amean 7 124.0332 ( 0.00%) 124.4406 ( -0.33%) Amean 12 181.0324 ( 0.00%) 182.9220 ( -1.04%) Amean 16 239.5556 ( 0.00%) 243.3556 * -1.59%* BAmean-99 1 51.5335 ( 0.00%) 51.3338 ( 0.39%) BAmean-99 3 59.7848 ( 0.00%) 59.0958 ( 1.15%) BAmean-99 5 95.6698 ( 0.00%) 96.5450 ( -0.91%) BAmean-99 7 123.8478 ( 0.00%) 124.3760 ( -0.43%) BAmean-99 12 180.8035 ( 0.00%) 182.5135 ( -0.95%) BAmean-99 16 239.1933 ( 0.00%) 243.0570 ( -1.62%)[schbench]
threads baseline sched_cache change 1 12.00(2.00) 11.00(0.71) +8.33% 2 12.40(0.89) 13.80(0.84) -11.29% 4 14.20(0.45) 14.80(0.45) -4.23% 8 16.00(0.00) 15.80(0.45) +1.25% 16 16.00(0.00) 16.00(0.71) 0.00% 32 19.40(0.55) 18.60(0.55) +4.12% 63 22.20(0.45) 23.20(0.45) -4.50%[stream]
No obvious difference is found.
export STREAM_SIZE=$((128000000))baseline sched_cache GB/sec copy-16 726.48 ( 0.00%) 715.60 ( -1.50%) GB/sec scale-16 577.71 ( 0.00%) 577.03 ( -0.12%) GB/sec add-16 678.85 ( 0.00%) 672.87 ( -0.88%) GB/sec triad-16 735.52 ( 0.00%) 729.05 ( -0.88%)[netperf]
No much difference is observed.
nr_pairsbaselinesched_cacheHmean 32 755.98 ( 0.00%) 755.17 ( -0.11%) BHmean-99 32 756.42 ( 0.00%) 755.40 ( -0.13%) Hmean 64 677.38 ( 0.00%) 669.75 ( -1.13%) BHmean-99 64 677.50 ( 0.00%) 669.86 ( -1.13%) Hmean 96 498.52 ( 0.00%) 496.73 ( -0.36%) BHmean-99 96 498.69 ( 0.00%) 496.93 ( -0.35%) Hmean 128 604.38 ( 0.00%) 604.22 ( -0.03%) BHmean-99 128 604.87 ( 0.00%) 604.87 ( 0.00%) Hmean 160 471.67 ( 0.00%) 468.29 ( -0.72%) BHmean-99 160 474.34 ( 0.00%) 471.05 ( -0.69%) Hmean 192 381.18 ( 0.00%) 384.88 ( 0.97%) BHmean-99 192 383.30 ( 0.00%) 386.82 ( 0.92%) Hmean 224 327.79 ( 0.00%) 326.05 ( -0.53%) BHmean-99 224 329.85 ( 0.00%) 327.87 ( -0.60%) Hmean 256 284.61 ( 0.00%) 300.52 ( 5.59%) BHmean-99 256 286.41 ( 0.00%) 302.06 ( 5.47%)[Genoa details]
[ChaCha20-xiangshan]
ChaCha20-xiangshan is a simple benchmark using a static build of an 8-thread Verilator of XiangShan(RISC-V). The README file can be found here[2]. The score depends on how aggressive the user set the/sys/kernel/debug/sched/llc_aggr_tolerance. Using the default values, there is no much difference observed. While setting the/sys/kernel/debug/sched/llc_aggr_toleranceto 100, 44% improvment is observed.Quoted 7 lines
baseline:
Host time spent: 50,868ms
sched_cache:
Host time spent: 28,349msThe time has been reduced by 44%.
Quoted 8 lines
Milan showed no improvement across all benchmarks, which could be due to the CCX topology (8 CCXs × 8 CPUs) where the LLC domain is too small for this optimization to be effective. Moreover there could be overhead due to additional computations.
ChaCha20-xiangshan improvement in Genoa when
llc_aggr_toleranceis set to 100 seems due to having relatively lesser thread count. Please provide the numbers with default values too. Would like to know numbers on varying loads.
I'll ask Chen Yu who did the Xiangshan experiments if he has those numbers.
Quoted 7 lines
In Power 10 and Power 11, the LLC size is 4 threads which is even smaller. Not expecting improvements here but will run some workloads and share the data.
Not gone through the entire series yet but are the situations like say in two NUMA system, if a task's preferred LLC is on the wrong NUMA node for its memory, which takes precedence?
We take preferred NUMA node in the consideration but we do not force task to go to the preferred node.
I remembered initially we limited the consideration to only LLCs in the preferred node. But we encountered regressions in hackbench and schbench, because when the preferred node don't have any occupancy resulting in preferred LLC to be set to -1 (no preference), and resulted in extra task migrations. And also the preferred node for hackbench and schbench was volatile
as they have small memory footprint. Chen Yu, please chime in if there were other reasons you remembered.
We'll need to revisit this part of the code to take care of such corner case. I think ideally we should move tasks to the least loaded LLC in the preferred node (even if no LLCs have occupancy in the preferred node), as long as preferred NUMA node don't changes too often.
Quoted 3 lines
Also, what about the workloads that don't share data like stress-ng?
We can test those. Ideally the controls to prevent over aggregation to preferred LLC would keep stress-ng happy.
Quoted 3 lines
It will
be good to make sure that most other workloads don't suffer. As mentioned, per process knob forllc_aggr_tolerancecould help.
Agree. We are planning to add per process knob for the next version. One thought is to use prctl. Any other suggestions are welcome.
Tim
Quoted 75 lines
Thanks,
Madadi Vineeth ReddyQuoted 70 lines
Thanks to everyone who participated and provided valuable suggestions for the previous versions. Your comments and tests on the latest version are also greatly appreciated in advance.
Tim
[1] https://lore.kernel.org/lkml/[email protected]/
[2] https://github.com/yu-chen-surf/chacha20-xiangshan/blob/master/README.eng.md
RFC v4:
[3] https://lore.kernel.org/all/[email protected]/RFC v3
[4] https://lore.kernel.org/all/[email protected]/RFC v2:
[5] https://lore.kernel.org/lkml/[email protected]/Chen Yu (7):
sched/fair: Record per-LLC utilization to guide cache-aware scheduling
decisions
sched/fair: Introduce helper functions to enforce LLC migration policy sched/fair: Introduce a static key to enable cache aware only for
multi LLCs
sched/fair: Exclude processes with many threads from cache-aware
scheduling
sched/fair: Disable cache aware scheduling for processes with high
thread counts
sched/fair: Avoid cache-aware scheduling for memory-heavy processes sched/fair: Add user control to adjust the tolerance of cache-aware
schedulingPeter Zijlstra (Intel) (1):
sched/fair: Add infrastructure for cache-aware load balancingTim Chen (11):
sched/fair: Add LLC index mapping for CPUs
sched/fair: Assign preferred LLC ID to processes sched/fair: Track LLC-preferred tasks per runqueue sched/fair: Introduce per runqueue task LLC preference counter sched/fair: Count tasks prefering each LLC in a sched group sched/fair: Prioritize tasks preferring destination LLC during
balancing
sched/fair: Identify busiestsched_groupfor LLC-aware load balancing sched/fair: Addmigrate_llc_taskmigration type for cache-aware
balancing
sched/fair: Handle moving single tasks to/from their preferred LLC sched/fair: Consider LLC preference when selecting tasks for load
balancing
sched/fair: Respect LLC preference in task migration and detachinclude/linux/cacheinfo.h | 21 +- include/linux/mm_types.h | 45 ++ include/linux/sched.h | 5 + include/linux/sched/topology.h | 4 + include/linux/threads.h | 10 + init/Kconfig | 20 + init/init_task.c | 3 + kernel/fork.c | 6 + kernel/sched/core.c | 18 + kernel/sched/debug.c | 56 ++ kernel/sched/fair.c | 1022 +++++++++++++++++++++++++++++++- kernel/sched/features.h | 1 + kernel/sched/sched.h | 27 + kernel/sched/topology.c | 61 +- 14 files changed, 1283 insertions(+), 16 deletions(-)
Quoted 5 lines · Tim Chen
Quoted 4 lines · Madadi Vineeth Reddy
Hi Tim,
Thanks for the patch.On 11/10/25 23:54, Tim Chen wrote:
[snip]
Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
[Genoa details]
[ChaCha20-xiangshan]
ChaCha20-xiangshan is a simple benchmark using a static build of an 8-thread Verilator of XiangShan(RISC-V). The README file can be found here[2]. The score depends on how aggressive the user set the/sys/kernel/debug/sched/llc_aggr_tolerance. Using the default values, there is no much difference observed. While setting the/sys/kernel/debug/sched/llc_aggr_toleranceto 100, 44% improvment is observed.Quoted 7 lines
baseline:
Host time spent: 50,868ms
sched_cache:
Host time spent: 28,349msThe time has been reduced by 44%.
Milan showed no improvement across all benchmarks, which could be due to the CCX topology (8 CCXs × 8 CPUs) where the LLC domain is too small for this optimization to be effective. Moreover there could be overhead due to additional computations.
Quoted 6 lines
Quoted 3 lines
ChaCha20-xiangshan improvement in Genoa when
llc_aggr_toleranceis set to 100 seems due to having relatively lesser thread count. Please provide the numbers with default values too. Would like to know numbers on varying loads.I'll ask Chen Yu who did the Xiangshan experiments if he has those numbers.
Madadi, do you mean the performance score number or active thread number
when llc_aggr_tolerance is set to 1(default)?
The score is around with sched_cache and llc_aggr_tolerance set to 1. The active number is 128 per process, and there are 8 processes when launching the benchmark. I suppose the 128 comes from the number of online CPUs. Please let me know if you need more data.
Cced Yangyu who's the author of this benchmark.
ls -l /proc/14460/task/ | grep -c '^d'
128
Quoted 11 lines
Quoted 7 lines
In Power 10 and Power 11, the LLC size is 4 threads which is even smaller. Not expecting improvements here but will run some workloads and share the data.
Not gone through the entire series yet but are the situations like say in two NUMA system, if a task's preferred LLC is on the wrong NUMA node for its memory, which takes precedence?
We take preferred NUMA node in the consideration but we do not force task to go to the preferred node.
Quoted 8 lines
I remembered initially we limited the consideration to only LLCs in the preferred node. But we encountered regressions in hackbench and schbench, because when the preferred node don't have any occupancy resulting in preferred LLC to be set to -1 (no preference), and resulted in extra task migrations. And also the preferred node for hackbench and schbench was volatile
as they have small memory footprint. Chen Yu, please chime in if there were other reasons you remembered.
Since the preferred NUMA node is per task, while the preferred LLC is per process, scanning only the current task's preferred node would lead to cross-node migration. This is because the process's preferred LLC may not reside within the current task's preferred node. Such a scenario could leave curr_m_a_occ at 0, and any LLC with an occupancy > 0 would then trigger a preferred LLC switch.
Quoted 6 lines
We'll need to revisit this part of the code to take care of such corner case. I think ideally we should move tasks to the least loaded LLC in the preferred node (even if no LLCs have occupancy in the preferred node), as long as preferred NUMA node don't changes too often.
Then we might need to introduce a new member in mm_struct to store the old occupancy, curr_m_a_occ, so that we can reliably compare the old and new occupancy - to avoid the 0 value of curr_m_a_occ.
Quoted 3 lines
Quoted 3 lines
Also, what about the workloads that don't share data like stress-ng?
The stream is single process stressing the memory without any share data, we did not observe any difference on stream. We can launch more tests on stress-ng.
thanks,
Chenyu>
Quoted 10 lines
We can test those. Ideally the controls to prevent over aggregation to preferred LLC would keep stress-ng happy.
Quoted 3 lines
It will
be good to make sure that most other workloads don't suffer. As mentioned, per process knob forllc_aggr_tolerancecould help.Agree. We are planning to add per process knob for the next version. One thought is to use prctl. Any other suggestions are welcome.
Quoted 39 lines · Yu C
Quoted 5 lines · Tim Chen
Quoted 4 lines · Madadi Vineeth Reddy
Hi Tim,
Thanks for the patch.On 11/10/25 23:54, Tim Chen wrote:
[snip]
Quoted 23 lines
Quoted 23 lines
Quoted 10 lines
[Genoa details]
[ChaCha20-xiangshan]
ChaCha20-xiangshan is a simple benchmark using a static build of an 8-thread Verilator of XiangShan(RISC-V). The README file can be found here[2]. The score depends on how aggressive the user set the/sys/kernel/debug/sched/llc_aggr_tolerance. Using the default values, there is no much difference observed. While setting the/sys/kernel/debug/sched/llc_aggr_toleranceto 100, 44% improvment is observed.Quoted 7 lines
baseline:
Host time spent: 50,868ms
sched_cache:
Host time spent: 28,349msThe time has been reduced by 44%.
Milan showed no improvement across all benchmarks, which could be due to the CCX topology (8 CCXs × 8 CPUs) where the LLC domain is too small for this optimization to be effective. Moreover there could be overhead due to additional computations.
Quoted 6 lines
Quoted 3 lines
ChaCha20-xiangshan improvement in Genoa when
llc_aggr_toleranceis set to 100 seems due to having relatively lesser thread count. Please provide the numbers with default values too. Would like to know numbers on varying loads.I'll ask Chen Yu who did the Xiangshan experiments if he has those numbers.
Quoted 8 lines
Madadi, do you mean the performance score number or active thread number when
llc_aggr_toleranceis set to 1(default)?
The score is around withsched_cacheandllc_aggr_toleranceset to 1. The active number is 128 per process, and there are 8 processes when launching the benchmark. I suppose the 128 comes from the number of online CPUs. Please let me know if you need more data.Cced Yangyu who's the author of this benchmark.
I mean the benchmark result with default value of llc_aggr_tolerance on Genoa in comparison to baseline. Knowing number of threads also helps to understand the impact.
Quoted 24 lines
ls -l
/proc/14460/task/| grep -c '^d'
128Quoted 11 lines
Quoted 7 lines
In Power 10 and Power 11, the LLC size is 4 threads which is even smaller. Not expecting improvements here but will run some workloads and share the data.
Not gone through the entire series yet but are the situations like say in two NUMA system, if a task's preferred LLC is on the wrong NUMA node for its memory, which takes precedence?
We take preferred NUMA node in the consideration but we do not force task to go to the preferred node.
Quoted 8 lines
I remembered initially we limited the consideration to only LLCs in the preferred node. But we encountered regressions in hackbench and schbench, because when the preferred node don't have any occupancy resulting in preferred LLC to be set to -1 (no preference), and resulted in extra task migrations. And also the preferred node for hackbench and schbench was volatile
as they have small memory footprint. Chen Yu, please chime in if there were other reasons you remembered.
Quoted 6 lines
Since the preferred NUMA node is per task, while the preferred LLC is per process, scanning only the current task's preferred node would lead to cross-node migration. This is because the process's preferred LLC may not reside within the current task's preferred node. Such a scenario could leave
curr_m_a_occat 0, and any LLC with an occupancy > 0 would then trigger a preferred LLC switch.
Understood. Thanks for the context.
Quoted 13 lines
Quoted 7 lines
We'll need to revisit this part of the code to take care of such corner case. I think ideally we should move tasks to the least loaded LLC in the preferred node (even if no LLCs have occupancy in the preferred node), as long as preferred NUMA node don't changes too often.
Then we might need to introduce a new member in
mm_structto store the old occupancy,curr_m_a_occ, so that we can reliably compare the old and new occupancy - to avoid the 0 value ofcurr_m_a_occ.Quoted 1 line
Quoted 1 line
Quoted 7 lines
Quoted 2 lines
Quoted 2 lines
Also, what about the workloads that don't share data like stress-ng?
The stream is single process stressing the memory without any share data, we did not observe any difference on stream. We can launch more tests on stress-ng.
That would be helpful.
Thanks,
Madadi Vineeth Reddy
Quoted 13 lines
thanks,
Chenyu>Quoted 10 lines
We can test those. Ideally the controls to prevent over aggregation to preferred LLC would keep stress-ng happy.
Quoted 3 lines
It will
be good to make sure that most other workloads don't suffer. As mentioned, per process knob forllc_aggr_tolerancecould help.Agree. We are planning to add per process knob for the next version. One thought is to use prctl. Any other suggestions are welcome.
Quoted 7 lines · Madadi Vineeth Reddy
Quoted 6 lines · Yu C
Quoted 5 lines · Tim Chen
Quoted 4 lines · Madadi Vineeth Reddy
Hi Tim,
Thanks for the patch.On 11/10/25 23:54, Tim Chen wrote:
[snip]
Quoted 14 lines
Quoted 14 lines
Quoted 6 lines
Quoted 3 lines
ChaCha20-xiangshan improvement in Genoa when
llc_aggr_toleranceis set to 100 seems due to having relatively lesser thread count. Please provide the numbers with default values too. Would like to know numbers on varying loads.I'll ask Chen Yu who did the Xiangshan experiments if he has those numbers.
Madadi, do you mean the performance score number or active thread number
whenllc_aggr_toleranceis set to 1(default)?
The score is around withsched_cacheandllc_aggr_toleranceset to 1. The active number is 128 per process, and there are 8 processes when launching the benchmark. I suppose the 128 comes from the number of online CPUs. Please let me know if you need more data.
Quoted 6 lines
Quoted 1 line
Cced Yangyu who's the author of this benchmark.
I mean the benchmark result with default value of
llc_aggr_toleranceon Genoa in comparison to baseline. Knowing number of threads also helps to understand the impact.
OK. Here are the full test script and corresponding data:
pepc pstates config --governor performance
pepc pstates config --turbo off
pepc cstates config --disable C2
echo 1 > /proc/sys/kernel/numa_balancing
echo NO_SCHED_CACHE > /sys/kernel/debug/sched/features make run
sleep 5
sync
echo SCHED_CACHE > /sys/kernel/debug/sched/features echo 50 > /sys/kernel/debug/sched/llc_overload_pct echo 1 > /sys/kernel/debug/sched/llc_aggr_tolerance make run
echo SCHED_CACHE > /sys/kernel/debug/sched/features
# to encourage task aggregation as much as possible echo 100 > /sys/kernel/debug/sched/llc_overload_pct echo 100 > /sys/kernel/debug/sched/llc_aggr_tolerance make run
# ./launch.sh
Host time spent: 51,323ms //baseline
Host time spent: 51,741ms //sched_cache default Host time spent: 27,934ms //sched_cache aggressive
thanks,
Chenyu