Cache aware scheduling
Subsystems (5)
v2 cover Cache aware scheduling
This patch series introduces infrastructure for cache-aware load balancing, with the goal of co-locating tasks that share data on the same Last Level Cache (LLC) domain. By improving cache locality, the scheduler can reduce cache bouncing and cache misses, ultimately improving data access efficiency. The design builds on the initial prototype from Peter [1].
In this initial implementation, threads within the same process are treated as entities that are likely to share data. During load balancing, the scheduler attempts to aggregate these threads onto the same LLC domain whenever possible.
We would like to thank everyone who provided feedbacks on the v1 series[1]. Most of the comments have been addressed in this revision. Several broader suggestions surfaced during review, and we believe they are best approached in follow-up work once the foundational cache-aware scheduling infrastructure is merged:
1. **Generalizing task grouping beyond processes.**
While v2 focuses on grouping threads within a single process, other classes of workloads naturally share data and could benefit from LLC co-location, such as:
a) Tasks from different processes that operate on shared data.
b) Tasks belonging to the same NUMA group.
c) Tasks with strong waker/wakee relationships.
d) User-defined groups via cgroups or other user interfaces.
2. **Configurable cache-aware scheduling policies.**
The current iteration implements a global cache-aware scheduling policy. Future work may introduce per-process or per-task-group policies, exposed through prctl() or other mechanisms.
**v2 Changes:**
1. Align NUMA balancing and cache affinity by prioritizing NUMA balancing when their decisions differ.
2. Dynamically resize per-LLC statistics structures based on the LLC size.
3. Switch to a contiguous LLC-ID space so these IDs can be used directly as array indices for LLC statistics.
4. Add clarification comments.
5. Add 3 debug patches (not meant for merging).
6. Other changes to address feedbacks from review of v1 patch set (see individual patch change log).
Test results:
The patch series was applied and tested on v6.18-rc7.
See: https://github.com/timcchen1298/linux/commits/cache_aware_v2
The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.
The second test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.
hackbench/schbench/netperf/stream/stress-ng/chacha20 were launched on these two platforms.
[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when the number of different active threads is below the capacity of a LLC. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows good throughput improvement.
Genoa:
ChaCha20-xiangshan shows huge throughput improvement. No obvious difference is observed in hackbench/schbench /netperf/stream/stress-ng.
Phoronix has tested v1 and shows good improvements in 33 cases[2].
Detail:
Due to length constraints, only part of the data is presented.
Sapphire Rapids:
hackbench thread pipes
baseline sched_cache
groups
Amean 1 38.8224 ( 0.00%) 26.4582 * 31.85%* Amean 3 38.2358 ( 0.00%) 38.0758 ( 0.42%) Amean 5 40.7282 ( 0.00%) 41.1568 ( -1.05%) Amean 7 51.1720 ( 0.00%) 50.6646 ( 0.99%) Amean 12 63.1562 ( 0.00%) 63.3516 ( -0.31%) Amean 16 73.9584 ( 0.00%) 75.5596 ( -2.17%) Max 1 39.4140 ( 0.00%) 26.7590 ( 32.11%) Max 3 40.8310 ( 0.00%) 39.8000 ( 2.53%) Max 5 42.2150 ( 0.00%) 42.4860 ( -0.64%) Max 7 52.1800 ( 0.00%) 51.9370 ( 0.47%) Max 12 63.9430 ( 0.00%) 64.2820 ( -0.53%) Max 16 74.3710 ( 0.00%) 76.4170 ( -2.75%)
further test hackbench using other number of fds:
case fd groups baseline(std%) compare%( std%) threads-pipe-2 1-groups 1.00 ( 1.25) +38.52 ( 1.33) threads-pipe-2 2-groups 1.00 ( 12.52) +12.74 ( 1.31) threads-pipe-2 4-groups 1.00 ( 7.91) +12.29 ( 1.86) threads-pipe-4 1-groups 1.00 ( 0.55) +34.99 ( 0.45) threads-pipe-4 2-groups 1.00 ( 16.00) +27.32 ( 0.75) threads-pipe-4 4-groups 1.00 ( 17.37) +25.75 ( 0.20) threads-pipe-8 1-groups 1.00 ( 0.74) +27.13 ( 0.44) threads-pipe-8 2-groups 1.00 ( 8.82) +23.79 ( 0.32) threads-pipe-8 4-groups 1.00 ( 1.30) +27.64 ( 0.51) threads-pipe-16 1-groups 1.00 ( 1.03) +30.55 ( 0.27) threads-pipe-16 2-groups 1.00 ( 6.43) +29.52 ( 0.20) threads-pipe-16 4-groups 1.00 ( 1.36) -1.85 ( 1.43) threads-pipe-20 1-groups 1.00 ( 0.45) +30.88 ( 0.42) threads-pipe-20 2-groups 1.00 ( 1.95) -0.81 ( 5.84) threads-pipe-20 4-groups 1.00 ( 2.09) -1.77 ( 7.57)
stream:
baseline sched_cache
GB/sec copy-2 36.48 ( 0.00%) 36.55 ( 0.18%) GB/sec scale-2 36.83 ( 0.00%) 36.97 ( 0.38%) GB/sec add-2 37.92 ( 0.00%) 38.03 ( 0.31%) GB/sec triad-2 37.83 ( 0.00%) 37.97 ( 0.37%)
stress-ng context switch:
baseline sched_cache
Min context-1 2957.81 ( 0.00%) 2966.17 ( 0.28%) Min context-2 5931.68 ( 0.00%) 5930.17 ( -0.03%) Min context-4 11874.20 ( 0.00%) 11875.68 ( 0.01%) Min context-8 23755.30 ( 0.00%) 23762.43 ( 0.03%) Min context-16 47535.14 ( 0.00%) 47526.46 ( -0.02%) Min context-32 95078.66 ( 0.00%) 94356.39 ( -0.76%) Min context-64 190074.62 ( 0.00%) 190042.93 ( -0.02%) Min context-128 371107.12 ( 0.00%) 371008.10 ( -0.03%) Min context-256 578443.73 ( 0.00%) 579037.86 ( 0.10%) Min context-480 580203.34 ( 0.00%) 580499.43 ( 0.05%) Hmean context-1 2964.59 ( 0.00%) 2967.69 ( 0.10%) Hmean context-2 5936.41 ( 0.00%) 5935.51 ( -0.02%) Hmean context-4 11879.56 ( 0.00%) 11881.70 ( 0.02%) Hmean context-8 23771.92 ( 0.00%) 23770.28 ( -0.01%) Hmean context-16 47552.23 ( 0.00%) 47538.01 ( -0.03%) Hmean context-32 95102.67 ( 0.00%) 94969.43 ( -0.14%) Hmean context-64 190129.74 ( 0.00%) 190088.68 ( -0.02%) Hmean context-128 371291.95 ( 0.00%) 371114.82 ( -0.05%) Hmean context-256 578907.96 ( 0.00%) 579338.99 ( 0.07%) Hmean context-480 580541.78 ( 0.00%) 580726.13 ( 0.03%) Max context-1 2967.93 ( 0.00%) 2968.90 ( 0.03%) Max context-2 5942.37 ( 0.00%) 5940.40 ( -0.03%) Max context-4 11885.25 ( 0.00%) 11886.43 ( 0.01%) Max context-8 23784.17 ( 0.00%) 23783.31 ( -0.00%) Max context-16 47576.84 ( 0.00%) 47561.42 ( -0.03%) Max context-32 95139.03 ( 0.00%) 95094.86 ( -0.05%) Max context-64 190180.08 ( 0.00%) 190123.31 ( -0.03%) Max context-128 371451.73 ( 0.00%) 371240.25 ( -0.06%) Max context-256 579355.24 ( 0.00%) 579731.37 ( 0.06%) Max context-480 580750.44 ( 0.00%) 581118.33 ( 0.06%) BHmean-50 context-1 2966.80 ( 0.00%) 2968.82 ( 0.07%) BHmean-50 context-2 5939.32 ( 0.00%) 5939.49 ( 0.00%) BHmean-50 context-4 11883.02 ( 0.00%) 11886.08 ( 0.03%) BHmean-50 context-8 23778.40 ( 0.00%) 23775.90 ( -0.01%) BHmean-50 context-16 47568.31 ( 0.00%) 47546.19 ( -0.05%) BHmean-50 context-32 95125.84 ( 0.00%) 95087.06 ( -0.04%) BHmean-50 context-64 190165.37 ( 0.00%) 190117.94 ( -0.02%) BHmean-50 context-128 371405.28 ( 0.00%) 371168.75 ( -0.06%) BHmean-50 context-256 579137.11 ( 0.00%) 579609.35 ( 0.08%) BHmean-50 context-480 580646.72 ( 0.00%) 580920.46 ( 0.05%) BHmean-95 context-1 2965.72 ( 0.00%) 2967.94 ( 0.07%) BHmean-95 context-2 5937.20 ( 0.00%) 5936.40 ( -0.01%) BHmean-95 context-4 11880.45 ( 0.00%) 11882.71 ( 0.02%) BHmean-95 context-8 23774.69 ( 0.00%) 23771.59 ( -0.01%) BHmean-95 context-16 47555.08 ( 0.00%) 47539.93 ( -0.03%) BHmean-95 context-32 95106.67 ( 0.00%) 95072.38 ( -0.04%) BHmean-95 context-64 190138.93 ( 0.00%) 190096.30 ( -0.02%) BHmean-95 context-128 371322.78 ( 0.00%) 371132.61 ( -0.05%) BHmean-95 context-256 578985.41 ( 0.00%) 579389.21 ( 0.07%) BHmean-95 context-480 580598.22 ( 0.00%) 580763.93 ( 0.03%) BHmean-99 context-1 2965.72 ( 0.00%) 2967.94 ( 0.07%) BHmean-99 context-2 5937.20 ( 0.00%) 5936.40 ( -0.01%) BHmean-99 context-4 11880.45 ( 0.00%) 11882.71 ( 0.02%) BHmean-99 context-8 23774.69 ( 0.00%) 23771.59 ( -0.01%) BHmean-99 context-16 47555.08 ( 0.00%) 47539.93 ( -0.03%) BHmean-99 context-32 95106.67 ( 0.00%) 95072.38 ( -0.04%) BHmean-99 context-64 190138.93 ( 0.00%) 190096.30 ( -0.02%) BHmean-99 context-128 371322.78 ( 0.00%) 371132.61 ( -0.05%) BHmean-99 context-256 578985.41 ( 0.00%) 579389.21 ( 0.07%) BHmean-99 context-480 580598.22 ( 0.00%) 580763.93 ( 0.03%)
schbench thread = 1
Metric Base (mean±std) Compare (mean±std) Change
-------------------------------------------------------------------------------------
Wakeup Latencies 99.0th 10.71(0.76) 9.86(1.46) +7.94% Request Latencies 99.0th 4036.00(6.53) 4054.29(10.03) -0.45% RPS 50.0th 267.29(0.49) 266.86(0.38) -0.16% Average RPS 268.42(0.16) 267.86(0.31) -0.21%
schbench thread = 2
Metric Base (mean±std) Compare (mean±std) Change
-------------------------------------------------------------------------------------
Wakeup Latencies 99.0th 11.43(1.13) 8.00(2.00) +30.01% Request Latencies 99.0th 4007.43(34.52) 3967.43(70.03) +1.00% RPS 50.0th 536.71(0.76) 536.14(1.57) -0.11% Average RPS 536.59(0.55) 535.33(1.34) -0.23%
schbench thread = 4
Metric Base (mean±std) Compare (mean±std) Change
-------------------------------------------------------------------------------------
Wakeup Latencies 99.0th 9.57(0.79) 6.14(1.46) +35.84% Request Latencies 99.0th 3789.14(31.47) 3810.86(48.97) -0.57% RPS 50.0th 1074.00(0.00) 1073.43(2.76) -0.05% Average RPS 1075.03(1.07) 1072.93(2.13) -0.20%
schbench thread = 8
Metric Base (mean±std) Compare (mean±std) Change
-------------------------------------------------------------------------------------
Wakeup Latencies 99.0th 9.29(0.49) 6.57(1.81) +29.28% Request Latencies 99.0th 3756.00(19.60) 3769.71(23.87) -0.37% RPS 50.0th 2152.57(4.28) 2152.57(4.28) 0.00% Average RPS 2151.07(2.71) 2150.58(3.41) -0.02%
schbench thread = 16
Metric Base (mean±std) Compare (mean±std) Change
-------------------------------------------------------------------------------------
Wakeup Latencies 99.0th 9.43(0.53) 6.86(0.90) +27.25% Request Latencies 99.0th 3780.00(32.98) 3774.29(11.04) +0.15% RPS 50.0th 4305.14(8.55) 4307.43(7.81) +0.05% Average RPS 4303.47(5.74) 4301.71(4.35) -0.04%
schbench thread = 32
Metric Base (mean±std) Compare (mean±std) Change
-------------------------------------------------------------------------------------
Wakeup Latencies 99.0th 10.14(0.38) 6.86(0.69) +32.35% Request Latencies 99.0th 3764.00(21.66) 3806.29(32.24) -1.12% RPS 50.0th 8624.00(0.00) 8619.43(12.09) -0.05% Average RPS 8607.36(5.29) 8602.69(7.08) -0.05%
schbench thread = 64
Metric Base (mean±std) Compare (mean±std) Change
-------------------------------------------------------------------------------------
Wakeup Latencies 99.0th 11.71(0.49) 8.43(1.81) +28.01% Request Latencies 99.0th 3796.00(62.48) 3860.25(147.35) -1.69% RPS 50.0th 17238.86(24.19) 16411.43(88.95) -4.80% Average RPS 17209.02(10.18) 16389.73(100.27) -4.76%
schbench thread = 128
Metric Base (mean±std) Compare (mean±std) Change
-------------------------------------------------------------------------------------
Wakeup Latencies 99.0th 13.29(0.49) 12.00(0.00) +9.71% Request Latencies 99.0th 7893.71(11.04) 7909.71(17.10) -0.20% RPS 50.0th 32013.71(194.52) 32068.57(50.35) +0.17% Average RPS 31762.03(238.18) 31884.81(300.85) +0.39%
schbench thread = 239
Metric Base (mean±std) Compare (mean±std) Change
-------------------------------------------------------------------------------------
Wakeup Latencies 99.0th 13.29(0.49) 14.43(0.53) -8.58% Request Latencies 99.0th 8174.86(8.55) 8244.57(12.09) -0.85% RPS 50.0th 30624.00(0.00) 30614.86(24.19) -0.03% Average RPS 30695.86(11.03) 30673.35(17.31) -0.07%
chacha20:
baseline:
Host time spent: 66,320mssched_cache:
Host time spent: 53,859ms
Time reduced by 18%, throughput increased by 23%
Genoa:
chacha20
baseline:
Host time spent: 51,848mssched_cache:
Host time spent: 28,439ms
Time reduced by 45%, throughput increased by 82%
[1] https://lore.kernel.org/all/[email protected]/
[2] https://www.phoronix.com/review/cache-aware-scheduling-amd-turin
Chen Yu (10):
sched/cache: Record per-LLC utilization to guide cache-aware
scheduling decisions
sched/cache: Introduce helper functions to enforce LLC migration
policy
sched/cache: Introduce sched_cache_present to enable cache aware
scheduling for multi LLCs NUMA node
sched/cache: Record the number of active threads per process for
cache-aware scheduling
sched/cache: Disable cache aware scheduling for processes with high
thread counts
sched/cache: Avoid cache-aware scheduling for memory-heavy processes sched/cache: Add user control to adjust the parameters of cache-aware
scheduling
-- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware
load balancing
-- DO NOT APPLY!!! -- sched/cache/debug: Add ftrace to track the load
balance statistics
-- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy
for each process via proc fs
Peter Zijlstra (Intel) (1):
sched/cache: Introduce infrastructure for cache-aware load balancing
Tim Chen (12):
sched/cache: Make LLC id continuous
sched/cache: Assign preferred LLC ID to processes sched/cache: Track LLC-preferred tasks per runqueue sched/cache: Introduce per runqueue task LLC preference counter sched/cache: Calculate the per runqueue task LLC preference sched/cache: Count tasks prefering destination LLC in a sched group sched/cache: Check local_group only once in update_sg_lb_stats() sched/cache: Prioritize tasks preferring destination LLC during
balancing
sched/cache: Add migrate_llc_task migration type for cache-aware
balancing
sched/cache: Handle moving single tasks to/from their preferred LLC sched/cache: Consider LLC preference when selecting tasks for load
balancing
sched/cache: Respect LLC preference in task migration and detach
fs/proc/base.c | 22 + include/linux/cacheinfo.h | 21 +- include/linux/mm_types.h | 60 ++ include/linux/sched.h | 19 + include/linux/sched/topology.h | 5 + include/trace/events/sched.h | 31 + init/Kconfig | 11 + init/init_task.c | 4 + kernel/fork.c | 6 + kernel/sched/core.c | 12 + kernel/sched/debug.c | 62 ++ kernel/sched/fair.c | 1034 +++++++++++++++++++++++++++++++- kernel/sched/sched.h | 39 ++ kernel/sched/stats.c | 5 +- kernel/sched/topology.c | 239 +++++++- 15 files changed, 1543 insertions(+), 27 deletions(-)
Signature · 1 line
2.32.0
PATCH v2 1/23 sched/cache: Introduce infrastructure for cache-aware load balancing
From: "Peter Zijlstra (Intel)" <[email protected]>
Adds infrastructure to enable cache-aware load balancing, which improves cache locality by grouping tasks that share resources within the same cache domain. This reduces cache misses and improves overall data access efficiency.
In this initial implementation, threads belonging to the same process are treated as entities that likely share working sets. The mechanism tracks per-process CPU occupancy across cache domains and attempts to migrate threads toward cache-hot domains where their process already has active threads, thereby enhancing locality.
This provides a basic model for cache affinity. While the current code targets the last-level cache (LLC), the approach could be extended to other domain types such as clusters (L2) or node-internal groupings.
At present, the mechanism selects the CPU within an LLC that has the highest recent runtime. Subsequent patches in this series will use this information in the load-balancing path to guide task placement toward preferred LLCs.
In the future, more advanced policies could be integrated through NUMA balancing-for example, migrating a task to its preferred LLC when spare capacity exists, or swapping tasks across LLCs to improve cache affinity. Grouping of tasks could also be generalized from that of a process to be that of a NUMA group, or be user configurable.
---
Notes:
v1->v2:
Restore the original CPU scan to cover all online CPUs, rather than scanning within the preferred NUMA node. (Peter Zijlstra)
Use rq->curr instead of rq->donor. (K Prateek Nayak)
Minor fix in task_tick_cache() to use
if (mm->mm_sched_epoch >= rq->cpu_epoch)
to avoid mm_sched_epoch going backwards.
include/linux/mm_types.h | 44 +++++++ include/linux/sched.h | 11 ++ init/Kconfig | 11 ++ kernel/fork.c | 6 + kernel/sched/core.c | 6 + kernel/sched/fair.c | 258 +++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 8 ++ 7 files changed, 344 insertions(+)
include/linux/mm_types.h +44 -0
@@ -939,6 +939,11 @@ typedef struct {DECLARE_BITMAP (__mm_flags,NUM_MM_FLAG_BITS ); }__private mm_flags_t; +struct mm_sched { +u64 runtime; +unsigned long epoch; +}; +struct kioctx_table ;struct iommu_mm_data ;struct mm_struct {
@@ -1029,6 +1034,17 @@ struct mm_struct {*/ raw_spinlock_t cpus_allowed_lock;#endif +#ifdef CONFIG_SCHED_CACHE +/* +* Track per-cpu-per-process occupancy as a proxy for cache residency. +* See account_mm_sched() and ... +*/ +struct mm_sched __percpu *pcpu_sched; +raw_spinlock_t mm_sched_lock; +unsigned long mm_sched_epoch; +int mm_sched_cpu; +#endif +#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */ #endif
@@ -1487,6 +1503,34 @@ static inline unsigned int mm_cid_size(void)static inline void mm_set_cpus_allowed (struct mm_struct *mm,const struct cpumask *cpumask) { }#endif /* CONFIG_SCHED_MM_CID */ +#ifdef CONFIG_SCHED_CACHE +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *pcpu_sched); + +static inline int mm_alloc_sched_noprof (struct mm_struct *mm) +{ + struct mm_sched __percpu *pcpu_sched = alloc_percpu_noprof(struct mm_sched); + +if (!pcpu_sched) +return -ENOMEM ; + +mm_init_sched (mm, pcpu_sched); +return 0 ; +} + +#define mm_alloc_sched (...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) + +static inline void mm_destroy_sched (struct mm_struct *mm) +{ +free_percpu (mm->pcpu_sched); + mm->pcpu_sched =NULL ; +} +#else /* !CONFIG_SCHED_CACHE */ + +static inline int mm_alloc_sched (struct mm_struct *mm) {return 0 ; } +static inline void mm_destroy_sched (struct mm_struct *mm) { } + +#endif /* CONFIG_SCHED_CACHE */ +struct mmu_gather ;extern void tlb_gather_mmu (struct mmu_gather *tlb,struct mm_struct *mm);extern void tlb_gather_mmu_fullmm (struct mmu_gather *tlb,struct mm_struct *mm);
include/linux/sched.h +11 -0
@@ -1406,6 +1406,10 @@ struct task_struct {unsigned long numa_pages_migrated;#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +struct callback_head cache_work; +#endif +#ifdef CONFIG_RSEQ struct rseq __user *rseq;u32 rseq_len;
@@ -2428,4 +2432,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable()) +#ifdef CONFIG_SCHED_CACHE +static inline bool sched_cache_enabled (void ) +{ +return false; +} +#endif +#endif
init/Kconfig +11 -0
@@ -983,6 +983,17 @@ config NUMA_BALANCING This system will be inactive on UMA systems. +config SCHED_CACHE + bool "Cache aware load balance" + default y + depends on SMP + help + When enabled, the scheduler will attempt to aggregate tasks from + the same process onto a single Last Level Cache (LLC) domain when + possible. This improves cache locality by keeping tasks that share + resources within the same cache domain, reducing cache misses and + lowering data access latency. + config NUMA_BALANCING_DEFAULT_ENABLED bool "Automatically enable NUMA aware memory/task placement" default y
kernel/fork.c +6 -0
@@ -680,6 +680,7 @@ void __mmdrop(struct mm_struct *mm)cleanup_lazy_tlbs (mm);WARN_ON_ONCE (mm == current->active_mm); +mm_destroy_sched (mm);mm_free_pgd (mm);mm_free_id (mm);destroy_context (mm);
@@ -1083,6 +1084,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,if (mm_alloc_cid (mm, p)) goto fail_cid; +if (mm_alloc_sched (mm)) + goto fail_sched; +if (percpu_counter_init_many (mm->rss_stat,0 ,GFP_KERNEL_ACCOUNT ,NR_MM_COUNTERS )) goto fail_pcpu;
@@ -1092,6 +1096,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,return mm; fail_pcpu: +mm_destroy_sched (mm); +fail_sched:mm_destroy_cid (mm); fail_cid:destroy_context (mm);
kernel/sched/core.c +6 -0
@@ -4488,6 +4488,7 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p) p->wake_entry.u_flags =CSD_TYPE_TTWU ; p->migration_pending =NULL ;init_sched_mm_cid (p); +init_sched_mm (p); }DEFINE_STATIC_KEY_FALSE (sched_numa_balancing);
@@ -8791,6 +8792,11 @@ void __init sched_init(void) rq->core_cookie =0UL ;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spin_lock_init (&rq->cpu_epoch_lock); + rq->cpu_epoch_next = jiffies; +#endif +zalloc_cpumask_var_node (&rq->scratch_mask,GFP_KERNEL ,cpu_to_node (i)); }
kernel/sched/fair.c +258 -0
282 lines not on this page · raw message
kernel/sched/sched.h +8 -0
@@ -1194,6 +1194,12 @@ struct rq {u64 clock_pelt_idle_copy;u64 clock_idle_copy;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spinlock_t cpu_epoch_lock____cacheline_aligned ; +u64 cpu_runtime; +unsigned long cpu_epoch; +unsigned long cpu_epoch_next; +#endif atomic_t nr_iowait;
@@ -3819,6 +3825,8 @@ static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) { }static inline void init_sched_mm_cid (struct task_struct *t) { }#endif /* !CONFIG_SCHED_MM_CID */ +extern void init_sched_mm (struct task_struct *p); +extern u64 avg_vruntime (struct cfs_rq *cfs_rq);extern int entity_eligible (struct cfs_rq *cfs_rq,struct sched_entity *se); static inline
Signature · 1 line
2.32.0
PATCH v2 2/23 sched/cache: Record per-LLC utilization to guide cache-aware scheduling decisions
In reply to Tim ChenFrom: Chen Yu <[email protected]>
When a system becomes busy and a process’s preferred LLC is saturated with too many threads, tasks within that LLC migrate frequently. These in LLC migrations introduce latency and degrade performance. To avoid this, task aggregation should be suppressed when the preferred LLC is overloaded, which requires a metric to indicate LLC utilization.
Record per LLC utilization/cpu capacity during periodic load balancing. These statistics will be used in later patches to decide whether tasks should be aggregated into their preferred LLC.
---
Notes:
v1->v2:
Refine the comments in record_sg_llc_stats().(Peter Zijlstra).
include/linux/sched/topology.h | 4 ++ kernel/sched/fair.c | 69 ++++++++++++++++++++++++++++++++++ 2 files changed, 73 insertions(+)
include/linux/sched/topology.h +4 -0
@@ -68,6 +68,10 @@ struct sched_domain_shared {atomic_t nr_busy_cpus;int has_idle_cores;int nr_idle_scan; +#ifdef CONFIG_SCHED_CACHE +unsigned long util_avg; +unsigned long capacity____cacheline_aligned_in_smp ; +#endif };struct sched_domain {
kernel/sched/fair.c +69 -0
87 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v2 3/23 sched/cache: Introduce helper functions to enforce LLC migration policy
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Cache-aware scheduling aggregates threads onto their preferred LLC, mainly through load balancing. When the preferred LLC becomes saturated, more threads are still placed there, increasing latency. A mechanism is needed to limit aggregation so that the preferred LLC does not become overloaded.
Introduce helper functions can_migrate_llc() and can_migrate_llc_task() to enforce the LLC migration policy:
1. Aggregate a task to its preferred LLC if both source and destination LLCs are not too busy (<50% utilization),
or if doing so will not leave the preferred LLC much more imbalanced than the non-preferred one (>20% utilization difference, similar to imbalance_pct of the LLC domain).
2. Allow moving a task from overloaded preferred LLC to a non preferred LLC if this will not cause the non preferred LLC to become too imbalanced to cause a later migration back.
3. If both LLCs are too busy, let the generic load balance to spread the tasks.
Further (hysteresis)action could be taken in the future to prevent tasks from being migrated into and out of the preferred LLC frequently (back and forth): the threshold for migrating a task out of its preferred LLC should be higher than that for migrating it into the LLC.
Since aggregation tends to make the preferred LLC busier than others, the imbalance tolerance is controlled by llc_imb_pct. If set to 0, tasks may still aggregate to the preferred LLC as long as it is not more utilized than the source LLC, preserving the preference.
---
Notes:
v1->v2:
No change.kernel/sched/fair.c | 153 +++++++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 5 ++ 2 files changed, 158 insertions(+)
kernel/sched/fair.c +153 -0
@@ -1205,6 +1205,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)#define EPOCH_PERIOD (HZ / 100)/* 10 ms */ #define EPOCH_LLC_AFFINITY_TIMEOUT 5/* 50 ms */ +__read_mostly unsigned int llc_overload_pct =50 ; +__read_mostly unsigned int llc_imb_pct =20 ; +static int llc_id (int cpu) {if (cpu <0 )
@@ -9623,6 +9626,27 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_ }#ifdef CONFIG_SCHED_CACHE +/* +* The margin used when comparing LLC utilization with CPU capacity. +* Parameter llc_overload_pct determines the LLC load level where +* active LLC aggregation is done. +* Derived from fits_capacity(). +* +* (default: ~50%) +*/ +#define fits_llc_capacity (util, max) \ + ((util) * 100 < (max) * llc_overload_pct) + +/* +* The margin used when comparing utilization. +* is 'util1' noticeably greater than 'util2' +* Derived from capacity_greater(). +* Bias is in perentage. +*/ +/* Allows dst util to be bigger than src util by up to bias percent */ +#define util_greater (util1, util2) \ + ((util1) * 100 > (util2) * (100 + llc_imb_pct)) +/* Called from load balancing paths with rcu_read_lock held */ static __maybe_unused bool get_llc_stats (int cpu,unsigned long *util,unsigned long *cap)
@@ -9638,6 +9662,135 @@ static __maybe_unused bool get_llc_stats(int cpu, unsigned long *util, return true; } + +/* +* Decision matrix according to the LLC utilization. To +* decide whether we can do task aggregation across LLC. +* +* By default, 50% is the threshold to treat the LLC as busy, +* and 20% is the utilization imbalance percentage to decide +* if the preferred LLC is busier than the non-preferred LLC. +* The hysteresis is used to avoid task bouncing between the +* preferred LLC and the non-preferred LLC. +* +* 1. moving towards the preferred LLC, dst is the preferred +* LLC, src is not. +* +* src \ dst 30% 40% 50% 60% +* 30% Y Y Y N +* 40% Y Y Y Y +* 50% Y Y G G +* 60% Y Y G G +* +* 2. moving out of the preferred LLC, src is the preferred +* LLC, dst is not: +* +* src \ dst 30% 40% 50% 60% +* 30% N N N N +* 40% N N N N +* 50% N N G G +* 60% Y N G G +* +* src : src_util +* dst : dst_util +* Y : Yes, migrate +* N : No, do not migrate +* G : let the Generic load balance to even the load. +* +* The intention is that if both LLCs are quite busy, cache aware +* load balance should not be performed, and generic load balance +* should take effect. However, if one is busy and the other is not, +* the preferred LLC capacity(50%) and imbalance criteria(20%) should +* be considered to determine whether LLC aggregation should be +* performed to bias the load towards the preferred LLC. +*/ + +/* migration decision, 3 states are orthogonal. */ +enum llc_mig { + mig_forbid =0 ,/* N: Don't migrate task, respect LLC preference */ + mig_llc,/* Y: Do LLC preference based migration */ + mig_unrestricted/* G: Don't restrict generic load balance migration */ +}; + +/* +* Check if task can be moved from the source LLC to the +* destination LLC without breaking cache aware preferrence. +* src_cpu and dst_cpu are arbitrary CPUs within the source +* and destination LLCs, respectively. +*/ +static enum llc_mig can_migrate_llc (int src_cpu,int dst_cpu, +unsigned long tsk_util, +bool to_pref) +{ +unsigned long src_util, dst_util, src_cap, dst_cap; + +if (!get_llc_stats (src_cpu, &src_util, &src_cap) || + !get_llc_stats (dst_cpu, &dst_util, &dst_cap)) +return mig_unrestricted; + +if (!fits_llc_capacity (dst_util, dst_cap) && + !fits_llc_capacity (src_util, src_cap)) +return mig_unrestricted; + + src_util = src_util < tsk_util ?0 : src_util - tsk_util; + dst_util = dst_util + tsk_util; +if (to_pref) { +/* +* llc_imb_pct is the imbalance allowed between +* preferred LLC and non-preferred LLC. +* Don't migrate if we will get preferred LLC too +* heavily loaded and if the dest is much busier +* than the src, in which case migration will +* increase the imbalance too much. +*/ +if (!fits_llc_capacity (dst_util, dst_cap) && +util_greater (dst_util, src_util)) +return mig_forbid; + }else { +/* +* Don't migrate if we will leave preferred LLC +* too idle, or if this migration leads to the +* non-preferred LLC falls within sysctl_aggr_imb percent +* of preferred LLC, leading to migration again +* back to preferred LLC. +*/ +if (fits_llc_capacity (src_util, src_cap) || + !util_greater (src_util, dst_util)) +return mig_forbid; + } +return mig_llc; +} + +/* +* Check if task p can migrate from source LLC to +* destination LLC in terms of cache aware load balance. +*/ +static __maybe_unused enum llc_mig can_migrate_llc_task (int src_cpu,int dst_cpu, +struct task_struct *p) +{ +struct mm_struct *mm; +bool to_pref; +int cpu; + + mm = p->mm; +if (!mm) +return mig_unrestricted; + + cpu = mm->mm_sched_cpu; +if (cpu <0 ||cpus_share_cache (src_cpu, dst_cpu)) +return mig_unrestricted; + +if (cpus_share_cache (dst_cpu, cpu)) + to_pref = true; +else if (cpus_share_cache (src_cpu, cpu)) + to_pref = false; +else +return mig_unrestricted; + +return can_migrate_llc (src_cpu, dst_cpu, +task_util (p), to_pref); +} +#else static inline bool get_llc_stats (int cpu,unsigned long *util,unsigned long *cap)
kernel/sched/sched.h +5 -0
@@ -2828,6 +2828,11 @@ extern unsigned int sysctl_numa_balancing_scan_period_max;extern unsigned int sysctl_numa_balancing_scan_size;extern unsigned int sysctl_numa_balancing_hot_threshold; +#ifdef CONFIG_SCHED_CACHE +extern unsigned int llc_overload_pct; +extern unsigned int llc_imb_pct; +#endif +#ifdef CONFIG_SCHED_HRTICK /*
Signature · 1 line
2.32.0
PATCH v2 4/23 sched/cache: Make LLC id continuous
In reply to Tim ChenIntroduce an index mapping between CPUs and their LLCs. This provides a continuous per LLC index needed for cache-aware load balancing in later patches.
The existing per_cpu llc_id usually points to the first CPU of the LLC domain, which is sparse and unsuitable as an array index. Using llc_id directly would waste memory.
With the new mapping, CPUs in the same LLC share a continuous id:
per_cpu(llc_id, CPU=0...15) = 0
per_cpu(llc_id, CPU=16...31) = 1
per_cpu(llc_id, CPU=32...47) = 2
...
---
Notes:
v1->v2:
Convert the static LLC id to be allocated sequentially as LLCs are discovered, and replace the old sd_llc_id. (Peter Zijlstra)
kernel/sched/fair.c | 9 ++++++- kernel/sched/sched.h | 1 + kernel/sched/topology.c | 60 +++++++++++++++++++++++++++++++++++++++-- 3 files changed, 67 insertions(+), 3 deletions(-)
kernel/sched/fair.c +8 -1
@@ -1210,10 +1210,17 @@ __read_mostly unsigned int llc_imb_pct = 20;static int llc_id (int cpu) { +int llc; +if (cpu <0 )return -1 ; -return per_cpu (sd_llc_id, cpu); + llc =per_cpu (sd_llc_id, cpu); +/* avoid race with cpu hotplug */ +if (unlikely (llc >= max_llcs)) +return -1 ; + +return llc; }void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)
kernel/sched/sched.h +1 -0
@@ -2075,6 +2075,7 @@ DECLARE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);extern struct static_key_false sched_asym_cpucapacity;extern struct static_key_false sched_cluster_active; +extern int max_llcs; static __always_inline bool sched_asym_cpucap_active(void) {
kernel/sched/topology.c +58 -2
90 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v2 5/23 sched/cache: Assign preferred LLC ID to processes
In reply to Tim ChenWith cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar to numa_preferred_nid in NUMA balancing.
---
Notes:
v1->v2: Align preferred LLC with NUMA balancing's preferred node.
include/linux/sched.h | 1 + init/init_task.c | 3 +++ kernel/sched/fair.c | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+)
include/linux/sched.h +1 -0
@@ -1408,6 +1408,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; +int preferred_llc;#endif #ifdef CONFIG_RSEQ
init/init_task.c +3 -0
@@ -191,6 +191,9 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .numa_group = NULL, .numa_faults = NULL,#endif +#ifdef CONFIG_SCHED_CACHE + .preferred_llc = -1, +#endif #if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS ) .kasan_depth = 1,#endif
kernel/sched/fair.c +18 -0
@@ -1300,6 +1300,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched;unsigned long epoch; +int mm_sched_llc =-1 ;if (!sched_cache_enabled ())return ;
@@ -1330,6 +1331,23 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; } + +if (mm->mm_sched_cpu !=-1 ) { + mm_sched_llc =llc_id (mm->mm_sched_cpu); + +#ifdef CONFIG_NUMA_BALANCING +/* +* Don't assign preferred LLC if it +* conflicts with NUMA balancing. +*/ +if (p->numa_preferred_nid >=0 && +cpu_to_node (mm->mm_sched_cpu) != p->numa_preferred_nid) + mm_sched_llc =-1 ; +#endif + } + +if (p->preferred_llc != mm_sched_llc) + p->preferred_llc = mm_sched_llc; }static void task_tick_cache (struct rq *rq,struct task_struct *p)
Signature · 1 line
2.32.0
PATCH v2 6/23 sched/cache: Track LLC-preferred tasks per runqueue
In reply to Tim ChenFor each runqueue, track the number of tasks with an LLC preference and how many of them are running on their preferred LLC. This mirrors nr_numa_running and nr_preferred_running for NUMA balancing, and will be used by cache-aware load balancing in later patches.
---
Notes:
v1->v2: Invoke task_of() once and reuse its result afterwards.
(Peter Zijlstra)
Remove hacky reset_llc_stats() and introduce sched_llc_active flag to properly pair enqueue/dequeue statistics update (Peter Zijlstra, K Prateek Nayak)
include/linux/sched.h | 2 ++ init/init_task.c | 1 + kernel/sched/core.c | 5 ++++ kernel/sched/fair.c | 60 ++++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 6 +++++ 5 files changed, 71 insertions(+), 3 deletions(-)
include/linux/sched.h +2 -0
@@ -1408,6 +1408,8 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; +/*the p is currently refcounted in a rq's preferred llc stats*/ +bool sched_llc_active;int preferred_llc;#endif
init/init_task.c +1 -0
@@ -192,6 +192,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .numa_faults = NULL,#endif #ifdef CONFIG_SCHED_CACHE + .sched_llc_active = false, .preferred_llc = -1,#endif #if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS )
kernel/sched/core.c +5 -0
@@ -531,6 +531,11 @@ void __trace_set_current_state(int state_value) }EXPORT_SYMBOL (__trace_set_current_state); +int task_llc (const struct task_struct *p) +{ +return per_cpu (sd_llc_id,task_cpu (p)); +} +/* * Serialization rules: *
kernel/sched/fair.c +57 -3
99 lines not on this page · raw message
kernel/sched/sched.h +6 -0
@@ -1126,6 +1126,10 @@ struct rq {unsigned int nr_preferred_running;unsigned int numa_migrate_on;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc_running; +unsigned int nr_llc_running; +#endif #ifdef CONFIG_NO_HZ_COMMON unsigned long last_blocked_load_update_tick;unsigned int has_blocked_load;
@@ -1980,6 +1984,8 @@ init_numa_balancing(u64 clone_flags, struct task_struct *p)#endif /* !CONFIG_NUMA_BALANCING */ +int task_llc (const struct task_struct *p); + static inline void queue_balance_callback(struct rq *rq, struct balance_callback *head,
Signature · 1 line
2.32.0
PATCH v2 7/23 sched/cache: Introduce per runqueue task LLC preference counter
In reply to Tim ChenEach runqueue is assigned an array where each element tracks the number of tasks preferring a given LLC, indexed from 0 to max_llcs - 1.
For example, rq->nr_pref_llc[3] = 2 signifies that there are 2 tasks on this runqueue which prefer to run within LLC3.
The load balancer can use this information to identify busy runqueues and migrate tasks to their preferred LLC domains. This array will be reallocated at runtime if the number of LLCs increases due to CPU hotplug. Only extending the buffer(rather than shrinking it) is supported to simplify the implementation.
Introduce the buffer allocation mechanism, and the statistics will be calculated in the subsequent patch.
---
Notes:
v1->v2:
Remove static allocation of per runqueue LLC preference arrays.
Allocate array size to the actual number of LLCs online. (Peter Zijlstra, Madadi Vineeth Reddy)
kernel/sched/core.c | 1 + kernel/sched/sched.h | 1 + kernel/sched/topology.c | 117 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 118 insertions(+), 1 deletion(-)
kernel/sched/core.c +1 -0
@@ -8800,6 +8800,7 @@ void __init sched_init(void)#ifdef CONFIG_SCHED_CACHE raw_spin_lock_init (&rq->cpu_epoch_lock); rq->cpu_epoch_next = jiffies; + rq->nr_pref_llc =NULL ;#endif zalloc_cpumask_var_node (&rq->scratch_mask,GFP_KERNEL ,cpu_to_node (i));
kernel/sched/sched.h +1 -0
@@ -1129,6 +1129,7 @@ struct rq {#ifdef CONFIG_SCHED_CACHE unsigned int nr_pref_llc_running;unsigned int nr_llc_running; +unsigned int *nr_pref_llc;#endif #ifdef CONFIG_NO_HZ_COMMON unsigned long last_blocked_load_update_tick;
kernel/sched/topology.c +116 -1
137 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v2 8/23 sched/cache: Calculate the per runqueue task LLC preference
In reply to Tim ChenCalculate the number of tasks' LLC preferences for each runqueue. This statistic is computed during task enqueue and dequeue operations, and is used by the cache-aware load balancing.
---
Notes:
v1->v2: Split from previous patch for easier review.
kernel/sched/fair.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
kernel/sched/fair.c +4 -2
@@ -1231,11 +1231,12 @@ static void account_llc_enqueue(struct rq *rq, struct task_struct *p)return ; pref_llc = p->preferred_llc; -if (pref_llc <0 ) +if (pref_llc <0 || pref_llc >= max_llcs)return ; rq->nr_llc_running++; rq->nr_pref_llc_running += (pref_llc ==task_llc (p)); + rq->nr_pref_llc[pref_llc]++; p->sched_llc_active = true; }
@@ -1252,11 +1253,12 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p)return ; pref_llc = p->preferred_llc; -if (pref_llc <0 ) +if (pref_llc <0 || pref_llc >= max_llcs)return ; rq->nr_llc_running--; rq->nr_pref_llc_running -= (pref_llc ==task_llc (p)); + rq->nr_pref_llc[pref_llc]--; p->sched_llc_active = false; }
Signature · 1 line
2.32.0
PATCH v2 9/23 sched/cache: Count tasks prefering destination LLC in a sched group
In reply to Tim ChenDuring LLC load balancing, tabulate the number of tasks on each runqueue that prefer the LLC contains the env->dst_cpu in a sched group.
For example, consider a system with 4 LLC sched groups (LLC0 to LLC3) balancing towards LLC3. LLC0 has 3 tasks preferring LLC3, LLC1 has 2, and LLC2 has 1. LLC0, having the most tasks preferring LLC3, is selected as the busiest source to pick tasks from.
Within a source LLC, the total number of tasks preferring a destination LLC is computed by summing counts across all CPUs in that LLC. For instance, if LLC0 has CPU0 with 2 tasks and CPU1 with 1 task preferring LLC3, the total for LLC0 is 3.
These statistics allow the load balancer to choose tasks from source sched groups that best match their preferred LLCs.
---
Notes:
v1->v2:
Convert nr_pref_llc array in sg_lb_stats to a single variable as only the dst LLC stat is needed. (K Prateek Nayak)
kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
kernel/sched/fair.c +12 -0
@@ -10445,6 +10445,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc; +#endif };/*
@@ -10912,6 +10915,9 @@ static inline void update_sg_lb_stats(struct lb_env *env, {int i, nr_running, local_group, sd_flags = env->sd->flags;bool balancing_at_rd = !env->sd->parent; +#ifdef CONFIG_SCHED_CACHE +int dst_llc =llc_id (env->dst_cpu); +#endif memset (sgs,0 ,sizeof (*sgs));
@@ -10932,6 +10938,12 @@ static inline void update_sg_lb_stats(struct lb_env *env,if (cpu_overutilized (i)) *sg_overutilized =1 ; +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled () &&llc_id (i) != dst_llc && + dst_llc >=0 ) + sgs->nr_pref_llc += rq->nr_pref_llc[dst_llc]; +#endif +/* * No need to call idle_cpu() if nr_running is not 0 */
Signature · 1 line
2.32.0
PATCH v2 10/23 sched/cache: Check local_group only once in update_sg_lb_stats()
In reply to Tim ChenThere is no need to check the local group twice for both group_asym_packing and group_smt_balance. Adjust the code to facilitate future checks for group types (cache-aware load balancing) as well.
No functional changes are expected.
---
Notes:
v1->v2:
New code cleanup patch. (Peter Zijlstra)
kernel/sched/fair.c | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-)
kernel/sched/fair.c +10 -8
@@ -10984,14 +10984,16 @@ static inline void update_sg_lb_stats(struct lb_env *env, sgs->group_weight = group->group_weight; -/* Check if dst CPU is idle and preferred to this group */ -if (!local_group && env->idle && sgs->sum_h_nr_running && -sched_group_asym (env, sgs, group)) - sgs->group_asym_packing =1 ; - -/* Check for loaded SMT group to be balanced to dst CPU */ -if (!local_group &&smt_balance (env, sgs, group)) - sgs->group_smt_balance =1 ; +if (!local_group) { +/* Check if dst CPU is idle and preferred to this group */ +if (env->idle && sgs->sum_h_nr_running && +sched_group_asym (env, sgs, group)) + sgs->group_asym_packing =1 ; + +/* Check for loaded SMT group to be balanced to dst CPU */ +if (smt_balance (env, sgs, group)) + sgs->group_smt_balance =1 ; + } sgs->group_type =group_classify (env->sd->imbalance_pct, group, sgs);
Signature · 1 line
2.32.0
PATCH v2 11/23 sched/cache: Prioritize tasks preferring destination LLC during balancing
In reply to Tim ChenDuring LLC load balancing, first check for tasks that prefer the destination LLC and balance them to it before others.
Mark source sched groups containing tasks preferring non local LLCs with the group_llc_balance flag. This ensures the load balancer later pulls or pushes these tasks toward their preferred LLCs.
The load balancer selects the busiest sched_group and migrates tasks to less busy groups to distribute load across CPUs.
With cache-aware scheduling enabled, the busiest sched_group is the one with most tasks preferring the destination LLC. If the group has the llc_balance flag set, cache aware load balancing is triggered.
Introduce the helper function update_llc_busiest() to identify the sched_group with the most tasks preferring the destination LLC.
---
Notes:
v1->v2:
Fix comparison in can_migrate_llc(), which uses an uninitialized env->src_cpu. Use the candidate group's first CPU instead. (Aaron Lu)
Fix a race condition during bootup with build_sched_domains(), where the per-cpu(sd_llc_id) is reset to -1. (lkp/0day) Put the set of group_llc_balance and the usage of it into 1 patch. (Peter Zijlstra)
Change group_llc_balance priority to be lower than group_overloaded and embed it into normal load balance path. (Peter Zijlstra)
Remove the sched group's SD_SHARE_LLC check in llc_balance(), because we should allow tasks migration across NUMA nodes to their preferred LLC, where the domain does not have SD_SHARE_LLC flag.
kernel/sched/fair.c | 66 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-)
kernel/sched/fair.c +65 -1
@@ -9531,6 +9531,11 @@ enum group_type {* from balancing the load across the system. */ group_imbalanced, +/* +* There are tasks running on non-preferred LLC, possible to move +* them to their preferred LLC without creating too much imbalance. +*/ + group_llc_balance,/* * The CPU is overloaded and can't provide expected CPU cycles to all * tasks.
@@ -10440,6 +10445,7 @@ struct sg_lb_stats {enum group_type group_type;unsigned int group_asym_packing;/* Tasks should be moved to preferred CPU */ unsigned int group_smt_balance;/* Task on busy SMT be moved */ +unsigned int group_llc_balance;/* Tasks should be moved to preferred LLC */ unsigned long group_misfit_task_load;/* A CPU has a task too big for its capacity */ #ifdef CONFIG_NUMA_BALANCING unsigned int nr_numa_running;
@@ -10698,6 +10704,9 @@ group_type group_classify(unsigned int imbalance_pct,if (group_is_overloaded (imbalance_pct, sgs))return group_overloaded; +if (sgs->group_llc_balance) +return group_llc_balance; +if (sg_imbalanced (group))return group_imbalanced;
@@ -10890,11 +10899,55 @@ static void record_sg_llc_stats(struct lb_env *env,if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity)) WRITE_ONCE(sd_share->capacity, sgs->group_capacity); } + +/* +* Do LLC balance on sched group that contains LLC, and have tasks preferring +* to run on LLC in idle dst_cpu. +*/ +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +if (!sched_cache_enabled ()) +return false; + +if (env->sd->flags &SD_SHARE_LLC ) +return false; + +if (sgs->nr_pref_llc && +can_migrate_llc (cpumask_first (sched_group_span (group)), + env->dst_cpu,0 , true) == mig_llc) +return true; + +return false; +} + +static bool update_llc_busiest (struct lb_env *env, +struct sg_lb_stats *busiest, +struct sg_lb_stats *sgs) +{ +/* +* There are more tasks that want to run on dst_cpu's LLC. +*/ +return sgs->nr_pref_llc > busiest->nr_pref_llc; +}#else static inline void record_sg_llc_stats (struct lb_env *env,struct sg_lb_stats *sgs,struct sched_group *group) { } + +static inline bool llc_balance (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +return false; +} + +static bool update_llc_busiest (struct lb_env *env, +struct sg_lb_stats *busiest, +struct sg_lb_stats *sgs) +{ +return false; +}#endif /**
@@ -10993,6 +11046,10 @@ static inline void update_sg_lb_stats(struct lb_env *env,/* Check for loaded SMT group to be balanced to dst CPU */ if (smt_balance (env, sgs, group)) sgs->group_smt_balance =1 ; + +/* Check for tasks in this group can be moved to their preferred LLC */ +if (llc_balance (env, sgs, group)) + sgs->group_llc_balance =1 ; } sgs->group_type =group_classify (env->sd->imbalance_pct, group, sgs);
@@ -11056,6 +11113,10 @@ static bool update_sd_pick_busiest(struct lb_env *env,/* Select the overloaded group with highest avg_load. */ return sgs->avg_load > busiest->avg_load; +case group_llc_balance: +/* Select the group with most tasks preferring dst LLC */ +return update_llc_busiest (env, busiest, sgs); +case group_imbalanced:/* * Select the 1st imbalanced group as we don't have any way to
@@ -11318,6 +11379,7 @@ static bool update_pick_idlest(struct sched_group *idlest,return false;break ; +case group_llc_balance:case group_imbalanced:case group_asym_packing:case group_smt_balance:
@@ -11450,6 +11512,7 @@ sched_balance_find_dst_group(struct sched_domain *sd, struct task_struct *p, intreturn NULL ;break ; +case group_llc_balance:case group_imbalanced:case group_asym_packing:case group_smt_balance:
@@ -11949,7 +12012,8 @@ static struct sched_group *sched_balance_find_src_group(struct lb_env *env)* group's child domain. */ if (sds.prefer_sibling && local->group_type == group_has_spare && -sibling_imbalance (env, &sds, busiest, local) >1 ) + (busiest->group_type == group_llc_balance || +sibling_imbalance (env, &sds, busiest, local) >1 )) goto force_balance;if (busiest->group_type != group_overloaded) {
Signature · 1 line
2.32.0
PATCH v2 12/23 sched/cache: Add migrate_llc_task migration type for cache-aware balancing
In reply to Tim ChenIntroduce a new migration type, migrate_llc_task, to support cache-aware load balancing.
After identifying the busiest sched_group (having the most tasks preferring the destination LLC), mark migrations with this type. During load balancing, each runqueue in the busiest sched_group is examined, and the runqueue with the highest number of tasks preferring the destination CPU is selected as the busiest runqueue.
---
Notes:
v1->v2: Remove unnecessary cpus_share_cache() check in
sched_balance_find_src_rq() (K Prateek Nayak)
kernel/sched/fair.c | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-)
kernel/sched/fair.c +31 -1
@@ -9547,7 +9547,8 @@ enum migration_type { migrate_load =0 , migrate_util, migrate_task, - migrate_misfit + migrate_misfit, + migrate_llc_task };#define LBF_ALL_PINNED 0x01
@@ -10134,6 +10135,10 @@ static int detach_tasks(struct lb_env *env) env->imbalance -= util;break ; +case migrate_llc_task: + env->imbalance--; +break ; +case migrate_task: env->imbalance--;break ;
@@ -11766,6 +11771,15 @@ static inline void calculate_imbalance(struct lb_env *env, struct sd_lb_stats *sreturn ; } +#ifdef CONFIG_SCHED_CACHE +if (busiest->group_type == group_llc_balance) { +/* Move a task that prefer local LLC */ + env->migration_type = migrate_llc_task; + env->imbalance =1 ; +return ; + } +#endif +if (busiest->group_type == group_imbalanced) {/* * In the group_imb case we cannot rely on group-wide averages
@@ -12073,6 +12087,10 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env,struct rq *busiest =NULL , *rq;unsigned long busiest_util =0 , busiest_load =0 , busiest_capacity =1 ;unsigned int busiest_nr =0 ; +#ifdef CONFIG_SCHED_CACHE +unsigned int busiest_pref_llc =0 ; +int dst_llc; +#endif int i;for_each_cpu_and (i,sched_group_span (group), env->cpus) {
@@ -12181,6 +12199,16 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env, }break ; +case migrate_llc_task: +#ifdef CONFIG_SCHED_CACHE + dst_llc =llc_id (env->dst_cpu); +if (dst_llc >=0 && + busiest_pref_llc < rq->nr_pref_llc[dst_llc]) { + busiest_pref_llc = rq->nr_pref_llc[dst_llc]; + busiest = rq; + } +#endif +break ;case migrate_task:if (busiest_nr < nr_running) { busiest_nr = nr_running;
@@ -12363,6 +12391,8 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sdcase migrate_misfit:__schedstat_add (sd->lb_imbalance_misfit[idle], env->imbalance);break ; +case migrate_llc_task: +break ; } }
Signature · 1 line
2.32.0
PATCH v2 13/23 sched/cache: Handle moving single tasks to/from their preferred LLC
In reply to Tim ChenIf the busiest runqueue has only one task, active balancing may be invoked to move it. However, before migration, check whether the task is running on its preferred LLC.
Do not move a lone task to another LLC if it would move the task away from its preferred LLC or cause excessive imbalance between LLCs.
---
Notes:
v1->v2: Remove uneeded preferred LLC migration check from
active_load_balance_cpu_stop().kernel/sched/fair.c | 51 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 50 insertions(+), 1 deletion(-)
kernel/sched/fair.c +50 -1
@@ -9878,12 +9878,57 @@ static __maybe_unused enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu task_util(p), to_pref); } +/* +* Check if active load balance breaks LLC locality in +* terms of cache aware load balance. +*/ +static inline bool +break_llc_locality (struct lb_env *env) +{ +if (!sched_cache_enabled ()) +return false; + +if (cpus_share_cache (env->src_cpu, env->dst_cpu)) +return false; +/* +* All tasks prefer to stay on their current CPU. +* Do not pull a task from its preferred CPU if: +* 1. It is the only task running there; OR +* 2. Migrating it away from its preferred LLC would violate +* the cache-aware scheduling policy. +*/ +if (env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable) { +unsigned long util =0 ; +struct task_struct *cur; + +if (env->src_rq->nr_running <=1 ) +return true; + +rcu_read_lock (); + cur =rcu_dereference (env->src_rq->curr); +if (cur) + util =task_util (cur); +rcu_read_unlock (); + +if (can_migrate_llc (env->src_cpu, env->dst_cpu, + util, false) == mig_forbid) +return true; + } + +return false; +}#else static inline bool get_llc_stats (int cpu,unsigned long *util,unsigned long *cap) {return false; } + +static inline bool +break_llc_locality (struct lb_env *env) +{ +return false; +}#endif /* * can_migrate_task - may task p from runqueue rq be migrated to this_cpu?
@@ -12279,6 +12324,9 @@ static int need_active_balance(struct lb_env *env) {struct sched_domain *sd = env->sd; +if (break_llc_locality (env)) +return 0 ; +if (asym_active_balance (env))return 1 ;
@@ -12298,7 +12346,8 @@ static int need_active_balance(struct lb_env *env)return 1 ; } -if (env->migration_type == migrate_misfit) +if (env->migration_type == migrate_misfit || + env->migration_type == migrate_llc_task)return 1 ;return 0 ;
Signature · 1 line
2.32.0
PATCH v2 14/23 sched/cache: Consider LLC preference when selecting tasks for load balancing
In reply to Tim ChenCurrently, task selection from the busiest runqueue ignores LLC preferences. Reorder tasks in the busiest queue to prioritize selection as follows:
1. Tasks preferring the destination CPU's LLC
2. Tasks with no LLC preference
3. Tasks preferring an LLC different from their current one
4. Tasks preferring the LLC they are currently on
This improves the likelihood that tasks are migrated to their preferred LLC.
---
Notes:
v1->v2: No change.
kernel/sched/fair.c | 66 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-)
kernel/sched/fair.c +65 -1
@@ -10092,6 +10092,68 @@ static struct task_struct *detach_one_task(struct lb_env *env)return NULL ; } +#ifdef CONFIG_SCHED_CACHE +/* +* Prepare lists to detach tasks in the following order: +* 1. tasks that prefer dst cpu's LLC +* 2. tasks that have no preference in LLC +* 3. tasks that prefer LLC other than the ones they are on +* 4. tasks that prefer the LLC that they are currently on. +*/ +static struct list_head +*order_tasks_by_llc (struct lb_env *env,struct list_head *tasks) +{ +struct task_struct *p; +LIST_HEAD (pref_old_llc); +LIST_HEAD (pref_new_llc); +LIST_HEAD (no_pref_llc); +LIST_HEAD (pref_other_llc); + +if (!sched_cache_enabled ()) +return tasks; + +if (cpus_share_cache (env->dst_cpu, env->src_cpu)) +return tasks; + +while (!list_empty (tasks)) { + p = list_last_entry(tasks, struct task_struct, se.group_node); + +if (p->preferred_llc ==llc_id (env->dst_cpu)) { +list_move (&p->se.group_node, &pref_new_llc); +continue ; + } + +if (p->preferred_llc ==llc_id (env->src_cpu)) { +list_move (&p->se.group_node, &pref_old_llc); +continue ; + } + +if (p->preferred_llc ==-1 ) { +list_move (&p->se.group_node, &no_pref_llc); +continue ; + } + +list_move (&p->se.group_node, &pref_other_llc); + } + +/* +* We detach tasks from list tail in detach tasks. Put tasks +* to be chosen first at end of list. +*/ +list_splice (&pref_new_llc, tasks); +list_splice (&no_pref_llc, tasks); +list_splice (&pref_other_llc, tasks); +list_splice (&pref_old_llc, tasks); +return tasks; +} +#else +static inline struct list_head +*order_tasks_by_llc (struct lb_env *env,struct list_head *tasks) +{ +return tasks; +} +#endif +/* * detach_tasks() -- tries to detach up to imbalance load/util/tasks from * busiest_rq, as part of a balancing operation within domain "sd".
@@ -10100,7 +10162,7 @@ static struct task_struct *detach_one_task(struct lb_env *env)*/ static int detach_tasks (struct lb_env *env) { -struct list_head *tasks = &env->src_rq->cfs_tasks; +struct list_head *tasks;unsigned long util, load;struct task_struct *p;int detached =0 ;
@@ -10119,6 +10181,8 @@ static int detach_tasks(struct lb_env *env)if (env->imbalance <=0 )return 0 ; + tasks =order_tasks_by_llc (env, &env->src_rq->cfs_tasks); +while (!list_empty (tasks)) {/* * We don't want to steal all, otherwise we may be treated likewise,
Signature · 1 line
2.32.0
PATCH v2 15/23 sched/cache: Respect LLC preference in task migration and detach
In reply to Tim ChenDuring the final step of load balancing, can_migrate_task() now considers a task's LLC preference before moving it out of its preferred LLC.
Additionally, add checks in detach_tasks() to prevent selecting tasks that prefer their current LLC.
---
Notes:
v1->v2: Leave out tasks under core scheduling from the cache aware
load balance. (K Prateek Nayak)
Reduce the degree of honoring preferred_llc in detach_tasks(). If certain conditions are met, stop migrating tasks that prefer their current LLC and instead continue load balancing from other busiest runqueues. (K Prateek Nayak)
kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++++++++-- kernel/sched/sched.h | 13 +++++++++ 2 files changed, 74 insertions(+), 2 deletions(-)
kernel/sched/fair.c +61 -2
@@ -9852,8 +9852,8 @@ static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu,* Check if task p can migrate from source LLC to * destination LLC in terms of cache aware load balance. */ -static __maybe_unused enum llc_mig can_migrate_llc_task (int src_cpu,int dst_cpu, -struct task_struct *p) +static enum llc_mig can_migrate_llc_task (int src_cpu,int dst_cpu, +struct task_struct *p) {struct mm_struct *mm;bool to_pref;
@@ -10025,6 +10025,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)if (env->flags &LBF_ACTIVE_LB )return 1 ; +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled () && +can_migrate_llc_task (env->src_cpu, env->dst_cpu, p) == mig_forbid && + !task_has_sched_core (p)) +return 0 ; +#endif + degrades =migrate_degrades_locality (p, env);if (!degrades) hot =task_hot (p, env);
@@ -10146,12 +10153,55 @@ static struct list_headlist_splice (&pref_old_llc, tasks);return tasks; } + +static bool stop_migrate_src_rq (struct task_struct *p, +struct lb_env *env, +int detached) +{ +if (!sched_cache_enabled () || p->preferred_llc ==-1 || +cpus_share_cache (env->src_cpu, env->dst_cpu) || + env->sd->nr_balance_failed) +return false; + +/* +* Stop migration for the src_rq and pull from a +* different busy runqueue in the following cases: +* +* 1. Trying to migrate task to its preferred +* LLC, but the chosen task does not prefer dest +* LLC - case 3 in order_tasks_by_llc(). This violates +* the goal of migrate_llc_task. However, we should +* stop detaching only if some tasks have been detached +* and the imbalance has been mitigated. +* +* 2. Don't detach more tasks if the remaining tasks want +* to stay. We know the remaining tasks all prefer the +* current LLC, because after order_tasks_by_llc(), the +* tasks that prefer the current LLC are the least favored +* candidates to be migrated out. +*/ +if (env->migration_type == migrate_llc_task && + detached &&llc_id (env->dst_cpu) != p->preferred_llc) +return true; + +if (llc_id (env->src_cpu) == p->preferred_llc) +return true; + +return false; +}#else static inline struct list_head *order_tasks_by_llc (struct lb_env *env,struct list_head *tasks) {return tasks; } + +static bool stop_migrate_src_rq (struct task_struct *p, +struct lb_env *env, +int detached) +{ +return false; +}#endif /*
@@ -10205,6 +10255,15 @@ static int detach_tasks(struct lb_env *env) p = list_last_entry(tasks, struct task_struct, se.group_node); +/* +* Check if detaching current src_rq should be stopped, because +* doing so would break cache aware load balance. If we stop +* here, the env->flags has LBF_ALL_PINNED, which would cause +* the load balance to pull from another busy runqueue. +*/ +if (stop_migrate_src_rq (p, env, detached)) +break ; +if (!can_migrate_task (p, env)) goto next;
kernel/sched/sched.h +13 -0
@@ -1485,6 +1485,14 @@ extern void sched_core_dequeue(struct rq *rq, struct task_struct *p, int flags);extern void sched_core_get (void );extern void sched_core_put (void ); +static inline bool task_has_sched_core (struct task_struct *p) +{ +if (sched_core_disabled ()) +return false; + +return !!p->core_cookie; +} +#else /* !CONFIG_SCHED_CORE: */ static inline bool sched_core_enabled (struct rq *rq)
@@ -1524,6 +1532,11 @@ static inline bool sched_group_cookie_match(struct rq *rq, return true; } +static inline bool task_has_sched_core (struct task_struct *p) +{ +return false; +} +#endif /* !CONFIG_SCHED_CORE */ #ifdef CONFIG_RT_GROUP_SCHED
Signature · 1 line
2.32.0
PATCH v2 16/23 sched/cache: Introduce sched_cache_present to enable cache aware scheduling for multi LLCs NUMA node
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Cache-aware load balancing should only be enabled if there are more than 1 LLCs within 1 NUMA node. sched_cache_present is introduced to indicate whether this platform supports this topology.
---
Notes:
v1->v2:
Use flag sched_cache_present to indicate whether a platform supports cache aware scheduling. Change this flag from staic key. There should be only 1 static key to control the cache aware scheduling. (Peter Zijlstra)
kernel/sched/topology.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)
kernel/sched/topology.c +15 -5
@@ -24,6 +24,8 @@ int max_llcs;#ifdef CONFIG_SCHED_CACHE +static bool sched_cache_present; +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) {unsigned int *new =NULL ;
@@ -54,7 +56,7 @@ static void populate_new_pref_llcs(unsigned int *old, unsigned int *new) new[i] = old[i]; } -static int resize_llc_pref (void ) +static int resize_llc_pref (bool has_multi_llcs) {unsigned int *__percpu *tmp_llc_pref;int i, ret =0 ;
@@ -102,6 +104,11 @@ static int resize_llc_pref(void)rq_unlock_irqrestore (rq, &rf); } +if (has_multi_llcs) { + sched_cache_present = true; +pr_info_once ("Cache aware load balance is enabled on the platform.\n" ); + } + release_old:/* * Load balance is done under rcu_lock.
@@ -124,7 +131,7 @@ static int resize_llc_pref(void)#else -static int resize_llc_pref (void ) +static int resize_llc_pref (bool has_multi_llcs) { max_llcs = new_max_llcs;return 0 ;
@@ -2644,6 +2651,7 @@ static int build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attr) { enum s_alloc alloc_state = sa_none; +bool has_multi_llcs = false;struct sched_domain *sd;struct s_data d;struct rq *rq =NULL ;
@@ -2736,10 +2744,12 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att* between LLCs and memory channels. */ nr_llcs = sd->span_weight / child->span_weight; -if (nr_llcs ==1 ) +if (nr_llcs ==1 ) { imb = sd->span_weight >>3 ; -else + }else { imb = nr_llcs; + has_multi_llcs = true; + } imb =max (1U , imb); sd->imb_numa_nr = imb;
@@ -2787,7 +2797,7 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (has_cluster)static_branch_inc_cpuslocked (&sched_cluster_active); -resize_llc_pref (); +resize_llc_pref (has_multi_llcs);if (rq && sched_debug_verbose)pr_info ("root domain span: %*pbl\n" ,cpumask_pr_args (cpu_map));
Signature · 1 line
2.32.0
PATCH v2 17/23 sched/cache: Record the number of active threads per process for cache-aware scheduling
In reply to Tim ChenFrom: Chen Yu <[email protected]>
A performance regression was observed by Prateek when running hackbench with many threads per process (high fd count). To avoid this, processes with a large number of active threads are excluded from cache-aware scheduling.
With sched_cache enabled, record the number of active threads in each process during the periodic task_cache_work(). While iterating over CPUs, if the currently running task belongs to the same process as the task that launched task_cache_work(), increment the active thread count.
This number will be used by subsequent patch to inhibit cache aware load balance.
---
Notes:
v1->v2: No change.
include/linux/mm_types.h | 1 + kernel/sched/fair.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)
include/linux/mm_types.h +1 -0
@@ -1043,6 +1043,7 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; +u64 nr_running_avg____cacheline_aligned_in_smp ;#endif #ifdef CONFIG_MMU
kernel/sched/fair.c +9 -2
@@ -1421,11 +1421,11 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p) static void __no_profile task_cache_work(struct callback_head *work) { -struct task_struct *p = current; +struct task_struct *p = current, *cur;struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ;unsigned long curr_m_a_occ =0 ; -int cpu, m_a_cpu =-1 ; +int cpu, m_a_cpu =-1 , nr_running =0 ;cpumask_var_t cpus;WARN_ON_ONCE (work != &p->cache_work);
@@ -1458,6 +1458,12 @@ static void __no_profile task_cache_work(struct callback_head *work) m_occ = occ; m_cpu = i; } +rcu_read_lock (); + cur =rcu_dereference (cpu_rq (i)->curr); +if (cur && !(cur->flags & (PF_EXITING |PF_KTHREAD )) && + cur->mm == mm) + nr_running++; +rcu_read_unlock (); }/*
@@ -1501,6 +1507,7 @@ static void __no_profile task_cache_work(struct callback_head *work) mm->mm_sched_cpu = m_a_cpu; } +update_avg (&mm->nr_running_avg, nr_running);free_cpumask_var (cpus); }
Signature · 1 line
2.32.0
PATCH v2 18/23 sched/cache: Disable cache aware scheduling for processes with high thread counts
In reply to Tim ChenFrom: Chen Yu <[email protected]>
If the number of active threads within the process exceeds the number of Cores(divided by SMTs number) in the LLC, do not enable cache-aware scheduling. This is because there is a risk of cache contention within the preferred LLC when too many threads are present.
---
Notes:
v1->v2: No change.
kernel/sched/fair.c | 29 +++++++++++++++++++++++++++-- 1 file changed, 27 insertions(+), 2 deletions(-)
kernel/sched/fair.c +27 -2
@@ -1223,6 +1223,18 @@ static int llc_id(int cpu)return llc; } +static bool exceed_llc_nr (struct mm_struct *mm,int cpu) +{ +int smt_nr =1 ; + +#ifdef CONFIG_SCHED_SMT +if (sched_smt_active ()) + smt_nr =cpumask_weight (cpu_smt_mask (cpu)); +#endif + +return ((mm->nr_running_avg * smt_nr) >per_cpu (sd_llc_size, cpu)); +} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) {int pref_llc;
@@ -1365,10 +1377,12 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)/* * If this task hasn't hit task_cache_work() for a while, or it -* has only 1 thread, invalidate its preferred state. +* has only 1 thread, or has too many active threads, invalidate +* its preferred state. */ if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || -get_nr_threads (p) <=1 ) { +get_nr_threads (p) <=1 || +exceed_llc_nr (mm,cpu_of (rq))) {if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; }
@@ -1435,6 +1449,13 @@ static void __no_profile task_cache_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; +if (get_nr_threads (p) <=1 ) { +if (mm->mm_sched_cpu !=-1 ) + mm->mm_sched_cpu =-1 ; + +return ; + } +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL ))return ;
@@ -9874,6 +9895,10 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,if (cpu <0 ||cpus_share_cache (src_cpu, dst_cpu))return mig_unrestricted; +/* skip cache aware load balance for single/too many threads */ +if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu)) +return mig_unrestricted; +if (cpus_share_cache (dst_cpu, cpu)) to_pref = true;else if (cpus_share_cache (src_cpu, cpu))
Signature · 1 line
2.32.0
PATCH v2 19/23 sched/cache: Avoid cache-aware scheduling for memory-heavy processes
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Prateek and Tingyin reported that memory-intensive workloads (such as stream) can saturate memory bandwidth and caches on the preferred LLC when sched_cache aggregates too many threads.
To mitigate this, estimate a process's memory footprint by comparing its RSS (anonymous and shared pages) to the size of the LLC. If RSS exceeds the LLC size, skip cache-aware scheduling.
Note that RSS is only an approximation of the memory footprint. By default, the comparison is strict, but a later patch will allow users to provide a hint to adjust this threshold.
According to the test from Adam, some systems do not have shared L3 but with shared L2 as clusters. In this case, the L2 becomes the LLC[1].
Link[1]: https://lore.kernel.org/all/[email protected]/
---
Notes:
v1->v2: Assigned curr_cpu in task_cache_work() before checking
exceed_llc_capacity(mm, curr_cpu) to avoid out-of-bound
access.(lkp/0day)include/linux/cacheinfo.h | 21 ++++++++++------- kernel/sched/fair.c | 49 +++++++++++++++++++++++++++++++++++---- 2 files changed, 57 insertions(+), 13 deletions(-)
include/linux/cacheinfo.h +13 -8
@@ -113,18 +113,11 @@ int acpi_get_cache_info(unsigned int cpu,const struct attribute_group *cache_get_priv_group (struct cacheinfo *this_leaf); -/* -* Get the cacheinfo structure for the cache associated with @cpu at -* level @level. -* cpuhp lock must be held. -*/ -static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu,int level) +static inline struct cacheinfo *_get_cpu_cacheinfo_level (int cpu,int level) {struct cpu_cacheinfo *ci =get_cpu_cacheinfo (cpu);int i; -lockdep_assert_cpus_held (); -for (i =0 ; i < ci->num_leaves; i++) {if (ci->info_list[i].level == level) {if (ci->info_list[i].attributes &CACHE_ID )
@@ -136,6 +129,18 @@ static inline struct cacheinfo *get_cpu_cacheinfo_level(int cpu, int level)return NULL ; } +/* +* Get the cacheinfo structure for the cache associated with @cpu at +* level @level. +* cpuhp lock must be held. +*/ +static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu,int level) +{ +lockdep_assert_cpus_held (); + +return _get_cpu_cacheinfo_level (cpu, level); +} +/* * Get the id of the cache associated with @cpu at level @level. * cpuhp lock must be held.
kernel/sched/fair.c +44 -5
79 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v2 20/23 sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1) llc_enabledllc_enabled acts as the primary switch - users can toggle it to enable or disable cache aware load balancing.
(2) llc_aggr_tolerance
With sched_cache enabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.
Introduce /sys/kernel/debug/sched/llc_aggr_tolerance to let users control how strictly RSS limits aggregation. Values range from 0 to 100:
- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.
For example, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped
(784GB = (1 + (99 - 1) * 256) * 32MB).
Similarly, /sys/kernel/debug/sched/llc_aggr_tolerance also controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.
For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with nr_running > 8 are skipped.
- llc_aggr_tolerance=99 -> tasks with nr_running > 785 are skipped
785 = (1 + (99 - 1) * 8).
(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides, llc_epoch_period and llc_epoch_affinity_timeout are also turned into tunable.
---
Notes:
v1->v2: Remove the smt_nr check in fits_llc_capacity().
(Aaron Lu)include/linux/sched.h | 4 ++- kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++ kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- 5 files changed, 178 insertions(+), 10 deletions(-)
include/linux/sched.h +3 -1
@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif
kernel/sched/debug.c +62 -0
74 lines not on this page · raw message
kernel/sched/fair.c +56 -7
108 lines not on this page · raw message
kernel/sched/sched.h +5 -0
@@ -2852,6 +2852,11 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_enabled; +void sched_cache_set (bool locked);#endif #ifdef CONFIG_SCHED_HRTICK
kernel/sched/topology.c +52 -2
78 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH v2 21/23 -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Debug patch only.
With cache-aware load balancing enabled, statistics related to its activity are exposed via /proc/schedstat and debugfs. For instance, if users want to verify metrics like the number of exceeding RSS and nr_running limits, they can filter the output of /sys/kernel/debug/sched/debug and compute the required statistics manually:
llc_exceed_cap SUM: 6llc_exceed_nr SUM: 4531
Furthermore, these statistics exposed in /proc/schedstats can be queried manually or via perf sched stats[1] with minor modifications.
--- include/linux/sched/topology.h | 1 + kernel/sched/fair.c | 1 + kernel/sched/stats.c | 5 +++-- 3 files changed, 5 insertions(+), 2 deletions(-)
include/linux/sched/topology.h +1 -0
@@ -108,6 +108,7 @@ struct sched_domain {unsigned int lb_imbalance_util[CPU_MAX_IDLE_TYPES ];unsigned int lb_imbalance_task[CPU_MAX_IDLE_TYPES ];unsigned int lb_imbalance_misfit[CPU_MAX_IDLE_TYPES ]; +unsigned int lb_imbalance_llc[CPU_MAX_IDLE_TYPES ];unsigned int lb_gained[CPU_MAX_IDLE_TYPES ];unsigned int lb_hot_gained[CPU_MAX_IDLE_TYPES ];unsigned int lb_nobusyg[CPU_MAX_IDLE_TYPES ];
kernel/sched/fair.c +1 -0
@@ -12684,6 +12684,7 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sd__schedstat_add (sd->lb_imbalance_misfit[idle], env->imbalance);break ;case migrate_llc_task: +__schedstat_add (sd->lb_imbalance_llc[idle], env->imbalance);break ; } }
kernel/sched/stats.c +3 -2
@@ -104,7 +104,7 @@ void __update_stats_enqueue_sleeper(struct rq *rq, struct task_struct *p,* Bump this up when changing the output format or the meaning of an existing * format, so that tools can adapt (or abort) */ -#define SCHEDSTAT_VERSION 17 +#define SCHEDSTAT_VERSION 18static int show_schedstat (struct seq_file *seq,void *v) {
@@ -139,7 +139,7 @@ static int show_schedstat(struct seq_file *seq, void *v)seq_printf (seq,"domain%d %s %*pb" , dcount++, sd->name,cpumask_pr_args (sched_domain_span (sd)));for (itype =0 ; itype <CPU_MAX_IDLE_TYPES ; itype++) { - seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u", + seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u %u", sd->lb_count[itype], sd->lb_balanced[itype], sd->lb_failed[itype],
@@ -147,6 +147,7 @@ static int show_schedstat(struct seq_file *seq, void *v) sd->lb_imbalance_util[itype], sd->lb_imbalance_task[itype], sd->lb_imbalance_misfit[itype], + sd->lb_imbalance_llc[itype], sd->lb_gained[itype], sd->lb_hot_gained[itype], sd->lb_nobusyq[itype],
Signature · 1 line
2.32.0
PATCH v2 22/23 -- DO NOT APPLY!!! -- sched/cache/debug: Add ftrace to track the load balance statistics
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Debug patch only.
The user leverages this trace event (via bpftrace, etc)to monitor the cache aware load balance activity - whether the tasks are moved to their preferred LLC, or moved out of their preferred LLC.
--- include/trace/events/sched.h | 31 +++++++++++++++++++++++++++++++ kernel/sched/fair.c | 10 ++++++++++ 2 files changed, 41 insertions(+)
include/trace/events/sched.h +31 -0
@@ -10,6 +10,37 @@#include <linux/tracepoint.h> #include <linux/binfmts.h> +TRACE_EVENT (sched_attach_task, + + TP_PROTO(struct task_struct *t, int pref_cpu, int pref_llc, + int attach_cpu, int attach_llc), + +TP_ARGS (t, pref_cpu, pref_llc, attach_cpu, attach_llc), + +TP_STRUCT__entry ( + __array( char, comm, TASK_COMM_LEN ) + __field( pid_t, pid ) + __field( int, pref_cpu ) + __field( int, pref_llc ) + __field( int, attach_cpu ) +__field ( int, attach_llc ) + ), + +TP_fast_assign ( + memcpy(__entry->comm, t->comm, TASK_COMM_LEN); + __entry->pid = t->pid; + __entry->pref_cpu = pref_cpu; + __entry->pref_llc = pref_llc; + __entry->attach_cpu = attach_cpu; + __entry->attach_llc = attach_llc; + ), + +TP_printk ("comm=%s pid=%d pref_cpu=%d pref_llc=%d attach_cpu=%d attach_llc=%d" , + __entry->comm, __entry->pid, + __entry->pref_cpu, __entry->pref_llc, + __entry->attach_cpu, __entry->attach_llc) +); +/* * Tracepoint for calling kthread_stop, performed to end a kthread: */
kernel/sched/fair.c +10 -0
@@ -10487,6 +10487,16 @@ static void attach_task(struct rq *rq, struct task_struct *p) {lockdep_assert_rq_held (rq); +#ifdef CONFIG_SCHED_CACHE +if (p->mm) { +int pref_cpu = p->mm->mm_sched_cpu; + +trace_sched_attach_task (p, + pref_cpu, + pref_cpu !=-1 ?llc_id (pref_cpu) :-1 , +cpu_of (rq),llc_id (cpu_of (rq))); + } +#endif WARN_ON_ONCE (task_rq (p) != rq);activate_task (rq, p,ENQUEUE_NOCLOCK );wakeup_preempt (rq, p,0 );
Signature · 1 line
2.32.0
PATCH v2 23/23 -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Debug patch only.
Show the per-LLC occupancy in /proc/{PID}/schedstat, with each column corresponding to one LLC. This can be used to verify if the cache-aware load balancer works as expected by aggregating threads onto dedicated LLCs.
Suppose there are 2 LLCs and the sampling duration is 10 seconds:
Enable the cache aware load balance:
0 12281 <--- LLC0 residency delta is 0, LLC1 is 12 seconds 0 18881
0 16217
disable the cache aware load balance:
6497 15802
9299 5435
17811 8278
--- fs/proc/base.c | 22 ++++++++++++++++++++++ include/linux/mm_types.h | 19 +++++++++++++++++-- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 80 insertions(+), 4 deletions(-)
fs/proc/base.c +22 -0
@@ -518,6 +518,28 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, (unsigned long long )task->se.sum_exec_runtime, (unsigned long long )task->sched_info.run_delay, task->sched_info.pcount); +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +struct mm_struct *mm = task->mm; +u64 *llc_runtime; + +if (!mm) +return 0 ; + + llc_runtime =kcalloc (max_llcs,sizeof (u64),GFP_KERNEL ); +if (!llc_runtime) +return 0 ; + +if (get_mm_per_llc_runtime (task, llc_runtime)) + goto out; + +for (int i =0 ; i < max_llcs; i++) +seq_printf (m,"%llu " , llc_runtime[i]); +seq_puts (m,"\n" ); +out: +kfree (llc_runtime); + } +#endif return 0 ; }
include/linux/mm_types.h +17 -2
@@ -944,6 +944,10 @@ struct mm_sched {unsigned long epoch; }; +struct mm_time { +u64 runtime_ns; +}; +struct kioctx_table ;struct iommu_mm_data ;struct mm_struct {
@@ -1040,6 +1044,7 @@ struct mm_struct {* See account_mm_sched() and ... */ struct mm_sched __percpu *pcpu_sched; +struct mm_time __percpu *pcpu_time;raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu;
@@ -1505,16 +1510,24 @@ static inline void mm_set_cpus_allowed(struct mm_struct *mm, const struct cpumas#endif /* CONFIG_SCHED_MM_CID */ #ifdef CONFIG_SCHED_CACHE -void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *pcpu_sched); +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *pcpu_sched, +struct mm_time __percpu *pcpu_time);static inline int mm_alloc_sched_noprof (struct mm_struct *mm) { struct mm_sched __percpu *pcpu_sched = alloc_percpu_noprof(struct mm_sched); +struct mm_time __percpu *pcpu_time;if (!pcpu_sched)return -ENOMEM ; -mm_init_sched (mm, pcpu_sched); + pcpu_time = alloc_percpu_noprof(struct mm_time); +if (!pcpu_time) { +free_percpu (mm->pcpu_sched); +return -ENOMEM ; + } + +mm_init_sched (mm, pcpu_sched, pcpu_time);return 0 ; }
@@ -1523,7 +1536,9 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)static inline void mm_destroy_sched (struct mm_struct *mm) {free_percpu (mm->pcpu_sched); +free_percpu (mm->pcpu_time); mm->pcpu_sched =NULL ; + mm->pcpu_time =NULL ; }#else /* !CONFIG_SCHED_CACHE */
include/linux/sched.h +3 -0
@@ -2442,6 +2442,9 @@ static inline bool sched_cache_enabled(void) {return static_branch_unlikely (&sched_cache_on); } + +int get_mm_per_llc_runtime (struct task_struct *p,u64 *buf); +extern int max_llcs;#endif #endif
kernel/sched/fair.c +38 -2
@@ -1355,16 +1355,19 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p) p->sched_llc_active = false; } -void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched, +struct mm_time __percpu *_pcpu_time) {unsigned long epoch;int i;for_each_possible_cpu (i) {struct mm_sched *pcpu_sched =per_cpu_ptr (_pcpu_sched, i); +struct mm_time *pcpu_time =per_cpu_ptr (_pcpu_time, i);struct rq *rq =cpu_rq (i); pcpu_sched->runtime =0 ; + pcpu_time->runtime_ns =0 ; pcpu_sched->epoch = rq->cpu_epoch; epoch = rq->cpu_epoch; }
@@ -1379,6 +1382,8 @@ void mm_init_sched(struct mm_struct *mm, struct mm_sched __percpu *_pcpu_sched)* the readers may get invalid mm_sched_epoch, etc. */ smp_store_release (&mm->pcpu_sched, _pcpu_sched); +/* same as above */ +smp_store_release (&mm->pcpu_time, _pcpu_time); }/* because why would C be fully specified */
@@ -1428,11 +1433,39 @@ static unsigned long __no_profile fraction_mm_sched(struct rq *rq, struct mm_schstatic unsigned int task_running_on_cpu (int cpu,struct task_struct *p); +/* p->pi_lock is hold */ +int get_mm_per_llc_runtime (struct task_struct *p,u64 *buf) +{ +struct mm_struct *mm = p->mm; +struct mm_time *pcpu_time; +int cpu; + +if (!mm) +return -EINVAL ; + +rcu_read_lock (); +for_each_online_cpu (cpu) { +int llc =llc_id (cpu); +u64 runtime_ms; + +if (llc <0 ) +continue ; + + pcpu_time =per_cpu_ptr (mm->pcpu_time, cpu); + runtime_ms =div_u64 (pcpu_time->runtime_ns,NSEC_PER_MSEC ); + buf[llc] += runtime_ms; + } +rcu_read_unlock (); + +return 0 ; +} +static inline void account_mm_sched (struct rq *rq,struct task_struct *p,s64 delta_exec) {struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched; +struct mm_time *pcpu_time;unsigned long epoch;int mm_sched_llc =-1 ;
@@ -1444,14 +1477,17 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)/* * init_task and kthreads don't having mm */ -if (!mm || !mm->pcpu_sched) +if (!mm || !mm->pcpu_sched || !mm->pcpu_time)return ; pcpu_sched =per_cpu_ptr (p->mm->pcpu_sched,cpu_of (rq)); + pcpu_time =per_cpu_ptr (p->mm->pcpu_time,cpu_of (rq)); scoped_guard (raw_spinlock, &rq->cpu_epoch_lock) {__update_mm_sched (rq, pcpu_sched); pcpu_sched->runtime += delta_exec; +/* pure runtime without decay */ + pcpu_time->runtime_ns += delta_exec; rq->cpu_runtime += delta_exec; epoch = rq->cpu_epoch; }
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH v2 01/23] sched/cache: Introduce infrastructure for cache-aware load balancing
In reply to Tim ChenQuoted 8 lines · Tim Chen
Minor fix in
task_tick_cache()to use
if (mm->mm_sched_epoch>=rq->cpu_epoch)
to avoidmm_sched_epochgoing backwards.kernel/sched/fair.c +4
-0+static void task_tick_cache (struct rq *rq,struct task_struct *p) +{ +struct callback_head *work = &p->cache_work; +struct mm_struct *mm = p->mm;
kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1338
Quoted 10 lines
kernel/sched/fair.c +10
-0+ +if (!sched_cache_enabled ()) +return ; + +if (!mm || !mm->pcpu_sched) +return ; + +/* avoid moving backwards */ +if (mm->mm_sched_epoch >= rq->cpu_epoch) +return ;
IIRC this was supposed to be able to wrap; which then means you should write it like:
if ((mm->mm_sched_epoch - rq->cpu_epoch) >= 0) return;
or somesuch.
Quoted 8 lines
kernel/sched/fair.c +8
-0+ +guard (raw_spinlock)(&mm->mm_sched_lock); + +if (work->next == work) { +task_work_add (p, work,TWA_RESUME ); +WRITE_ONCE (mm->mm_sched_epoch, rq->cpu_epoch); + } +}
Peter Zijlstra
maintainerRe: [PATCH v2 02/23] sched/cache: Record per-LLC utilization to guide cache-aware scheduling decisions
In reply to Tim Cheninclude/linux/sched/topology.h struct sched_domain_shared { · line 73
Quoted 11 lines · Tim Chen
include/linux/sched/topology.h +3
-0@@ -68,6 +68,10 @@ struct sched_domain_shared {atomic_t nr_busy_cpus;int has_idle_cores;int nr_idle_scan; +#ifdef CONFIG_SCHED_CACHE +unsigned long util_avg; +unsigned long capacity____cacheline_aligned_in_smp ;
This cacheline annotation confuses me, see below.
Quoted 15 lines
include/linux/sched/topology.h +1
-0+#endif };struct sched_domain {kernel/sched/fair.c +3
-0@@ -9622,6 +9622,29 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_return 0 ; } +#ifdef CONFIG_SCHED_CACHE +/* Called from load balancing paths with rcu_read_lock held */ +static __maybe_unused bool get_llc_stats(int cpu, unsigned long *util,
kernel/sched/fair.c static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_ · line 9637
Quoted 10 lines
kernel/sched/fair.c +10
-0+ unsigned long *cap) +{ +struct sched_domain_shared *sd_share; + + sd_share =rcu_dereference (per_cpu (sd_llc_shared, cpu)); +if (!sd_share) +return false; + + *util =READ_ONCE (sd_share->util_avg); + *cap =READ_ONCE (sd_share->capacity);
You placed capacity on a separate line, forcing the above to be 2 distinct lines. That seems... sub-optimal?
Quoted 44 lines
kernel/sched/fair.c +37
-0+ + return true; +} +#else +static inline bool get_llc_stats (int cpu,unsigned long *util, +unsigned long *cap) +{ +return false; +} +#endif /* * can_migrate_task - may task p from runqueue rq be migrated to this_cpu? */ @@ -10592,6 +10615,51 @@ sched_reduced_capacity(struct rq *rq, struct sched_domain *sd)return check_cpu_capacity (rq, sd); } +#ifdef CONFIG_SCHED_CACHE +/* +* Record the statistics for this scheduler group for later +* use. These values guide load balancing on aggregating tasks +* to a LLC. +*/ +static void record_sg_llc_stats (struct lb_env *env, +struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain_shared *sd_share; + +if (!sched_cache_enabled () || env->idle ==CPU_NEWLY_IDLE ) +return ; + +/* Only care about sched domain spanning multiple LLCs */ +if (env->sd->child !=rcu_dereference (per_cpu (sd_llc, env->dst_cpu))) +return ; + +/* +* At this point we know this group spans a LLC domain. +* Record the statistic of this group in its corresponding +* shared LLC domain. +* Note: sd_share cannot be obtained via sd->child->shared, because +* it refers to the domain that covers the local group, while +* sd_share could represent any of the LLC group. +*/
kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10654
Quoted 10 lines
kernel/sched/fair.c +10
-0+ sd_share =rcu_dereference (per_cpu (sd_llc_shared, +cpumask_first (sched_group_span (group)))); +if (!sd_share) +return ; + +if (READ_ONCE (sd_share->util_avg) != sgs->group_util) +WRITE_ONCE (sd_share->util_avg, sgs->group_util); + +if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity)) +WRITE_ONCE (sd_share->capacity, sgs->group_capacity);
And same here.
Quoted 7 lines
kernel/sched/fair.c +7
-0+} +#else +static inline void record_sg_llc_stats (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +} +#endif
Peter Zijlstra
maintainerRe: [PATCH v2 04/23] sched/cache: Make LLC id continuous
In reply to Tim ChenQuoted 65 lines · Tim Chen
kernel/sched/fair.c +46
-0@@ -1210,10 +1210,17 @@ __read_mostly unsigned int llc_imb_pct = 20;static int llc_id (int cpu) { +int llc; +if (cpu <0 )return -1 ; + llc =per_cpu (sd_llc_id, cpu); +/* avoid race with cpu hotplug */ +if (unlikely (llc >= max_llcs)) +return -1 ; + +return llc; }void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)@@ -668,6 +670,55 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_cluster_active); +/* +* Assign continuous llc id for the CPU, and return +* the assigned llc id. +*/ +static int update_llc_id (struct sched_domain *sd, +int cpu) +{ +int id =per_cpu (sd_llc_id, cpu), i; + +if (id >=0 ) +return id; + +if (sd) { +/* Look for any assigned id and reuse it.*/ +for_each_cpu (i,sched_domain_span (sd)) { + id =per_cpu (sd_llc_id, i); + +if (id >=0 ) { +per_cpu (sd_llc_id, cpu) = id; +return id; + } + } + } + +/* +* When 1. there is no id assigned to this LLC domain, +* or 2. the sd is NULL, we reach here. +* Consider the following scenario, +* CPU0~CPU95 are in the node0, CPU96~CPU191 are +* in the node1. During bootup, maxcpus=96 is +* appended. +* case 1: When running cpu_attach_domain(CPU24) +* during boot up, CPU24 is the first CPU in its +* non-NULL LLC domain. However, +* its corresponding llc id has not been assigned yet. +* +* case 2: After boot up, the CPU100 is brought up +* via sysfs manually. As a result, CPU100 has only a
kernel/sched/fair.c DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); · line 720
Quoted 10 lines
kernel/sched/fair.c +10
-0+* Numa domain attached, because CPU100 is the only CPU +* of a sched domain, all its bottom domains are degenerated. +* The LLC domain pointer sd is NULL for CPU100. +* +* For both cases, we want to increase the number of LLCs. +*/ +per_cpu (sd_llc_id, cpu) = max_llcs++; + +return per_cpu (sd_llc_id, cpu); +}
I'm not sure I follow. So partition_sched_domains() first calls detach_destroy_domains() on the old set, and then build_sched_domains() on the new set.
Do detach_destroy_domain() will do:
cpu_attach_domain(NULL,..);
That is, it will explicitly attach the NULL sched_domain to a CPU. At which point I feel update_llc_id() should be returning -1, no?
Then later, build_sched_domains() will set a !NULL sched_domain, at which point update_llc_id() can set a real value.
This should then also get rid of that weird max_llcs check in llc_id(), right?
Peter Zijlstra
maintainerRe: [PATCH v2 05/23] sched/cache: Assign preferred LLC ID to processes
In reply to Tim ChenQuoted 29 lines · Tim Chen
With cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar to
numa_preferred_nidin NUMA balancing.kernel/sched/fair.c +9
-0@@ -1300,6 +1300,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched;unsigned long epoch; +int mm_sched_llc =-1 ;if (!sched_cache_enabled ())return ;@@ -1330,6 +1331,23 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; } + +if (mm->mm_sched_cpu !=-1 ) { + mm_sched_llc =llc_id (mm->mm_sched_cpu); + +#ifdef CONFIG_NUMA_BALANCING +/* +* Don't assign preferred LLC if it +* conflicts with NUMA balancing.
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1351
Quoted 10 lines
kernel/sched/fair.c +9
-0+*/ +if (p->numa_preferred_nid >=0 && +cpu_to_node (mm->mm_sched_cpu) != p->numa_preferred_nid) + mm_sched_llc =-1 ; +#endif + } + +if (p->preferred_llc != mm_sched_llc) + p->preferred_llc = mm_sched_llc; }
This can of course still happen when sched_setnuma() gets called. I'm thinking it is not much of an issue because we expect this thing to get called fairly regularly -- at a higher rate than sched_setnuma() at least -- and thus the conflict only exists for a short period of time?
If so, that would make for a good comment.
Additionally, we could of course search for the busiest LLC inside the node, instead of setting -1. Again, that could live as a comment for future work.
Peter Zijlstra
maintainerRe: [PATCH v2 06/23] sched/cache: Track LLC-preferred tasks per runqueue
In reply to Tim Cheninclude/linux/sched.h struct task_struct { · line 1411
Quoted 3 lines · Tim Chen
#ifdef
CONFIG_SCHED_CACHEinclude/linux/sched.h +1
-0struct callback_head cache_work; +/*the p is currently refcounted in a rq's preferred llc stats*/
Shall we have spaces after and before the comment marks?
Also, comment confuses me, I don't see get_task_struct() / put_task_struct() usage. Did you mean something else with refcount?
Quoted 5 lines
kernel/sched/fair.c +2
-0+bool sched_llc_active;int preferred_llc;#endif +static void account_llc_dequeue (struct rq *rq,struct task_struct *p)
kernel/sched/fair.c static int llc_id(int cpu) · line 1252
Quoted 10 lines
kernel/sched/fair.c +10
-0+{ +int pref_llc; + +/* +* Borrow the uc_se->active from uclamp_rq_inc_id(), +* uclamp_rq_dec_id() to avoid the unbalanced calculation +* of rq statistics. +*/ +if (unlikely (!p->sched_llc_active)) +return ;
Another very confusing comment; what? Also, can you please explain (in the new comment) how we get here without having llc_active set?
Quoted 9 lines
kernel/sched/fair.c +9
-0+ + pref_llc = p->preferred_llc; +if (pref_llc <0 ) +return ; + + rq->nr_llc_running--; + rq->nr_pref_llc_running -= (pref_llc ==task_llc (p)); + p->sched_llc_active = false; +}
Peter Zijlstra
maintainerRe: [PATCH v2 07/23] sched/cache: Introduce per runqueue task LLC preference counter
In reply to Tim ChenQuoted 64 lines · Tim Chen
kernel/sched/topology.c +64
-0+#ifdef CONFIG_SCHED_CACHE + +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) +{ +unsigned int *new =NULL ; + + new =kcalloc (new_max_llcs,sizeof (unsigned int ), +GFP_KERNEL | __GFP_NOWARN); + +if (!new) { + *gc =NULL ; + }else { +/* +* Place old entry in garbage collector +* for later disposal. +*/ + *gc = old; + } +return new; +} + +static void populate_new_pref_llcs (unsigned int *old,unsigned int *new) +{ +int i; + +if (!old) +return ; + +for (i =0 ; i < max_llcs; i++) + new[i] = old[i]; +} + +static int resize_llc_pref (void ) +{ +unsigned int *__percpu *tmp_llc_pref; +int i, ret =0 ; + +if (new_max_llcs <= max_llcs) +return 0 ; + +/* +* Allocate temp percpu pointer for old llc_pref, +* which will be released after switching to the +* new buffer. +*/ + tmp_llc_pref = alloc_percpu_noprof(unsigned int *); +if (!tmp_llc_pref) +return -ENOMEM ; + +for_each_present_cpu (i) + *per_cpu_ptr (tmp_llc_pref, i) =NULL ; + +/* +* Resize the per rq nr_pref_llc buffer and +* switch to this new buffer. +*/ +for_each_present_cpu (i) { +struct rq_flags rf; +unsigned int *new; +struct rq *rq; + + rq =cpu_rq (i); + new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i)); +if (!new) {
kernel/sched/topology.c void sched_domains_mutex_unlock(void) · line 98
Quoted 10 lines
kernel/sched/topology.c +10
-0+ ret = -ENOMEM ; + + goto release_old; + } + +/* +* Locking rq ensures that rq->nr_pref_llc values +* don't change with new task enqueue/dequeue +* when we repopulate the newly enlarged array. +*/
guard(rq_lock_irq)(rq);
Notably, this cannot be with IRQs disabled, as you're doing allocations.
Quoted 15 lines
kernel/sched/topology.c +15
-0+rq_lock_irqsave (rq, &rf); +populate_new_pref_llcs (rq->nr_pref_llc, new); + rq->nr_pref_llc = new; +rq_unlock_irqrestore (rq, &rf); + } + +release_old: +/* +* Load balance is done under rcu_lock. +* Wait for load balance before and during resizing to +* be done. They may refer to old nr_pref_llc[] +* that hasn't been resized. +*/ +synchronize_rcu (); +for_each_present_cpu (i)
kernel/sched/topology.c void sched_domains_mutex_unlock(void) · line 123
Quoted 10 lines
kernel/sched/topology.c +10
-0+kfree (*per_cpu_ptr (tmp_llc_pref, i)); + +free_percpu (tmp_llc_pref); + +/* succeed and update */ +if (!ret) + max_llcs = new_max_llcs; + +return ret; +}
I think you need at least cpus_read_lock(), because present_cpu is dynamic -- but I'm not quite sure what lock is used to serialize it.
Re: [PATCH v2 01/23] sched/cache: Introduce infrastructure for cache-aware load balancing
In reply to Peter ZijlstraQuoted 24 lines · Peter Zijlstra
Quoted 8 lines · Tim Chen
Minor fix in
task_tick_cache()to use
if (mm->mm_sched_epoch>=rq->cpu_epoch)
to avoidmm_sched_epochgoing backwards.kernel/sched/fair.c +4
-0+static void task_tick_cache (struct rq *rq,struct task_struct *p) +{ +struct callback_head *work = &p->cache_work; +struct mm_struct *mm = p->mm;kernel/sched/fair.c static s64 update_se(struct rq *rq, struct sched_entity *se) · line 1338
Quoted 10 lines
kernel/sched/fair.c +10
-0+ +if (!sched_cache_enabled ()) +return ; + +if (!mm || !mm->pcpu_sched) +return ; + +/* avoid moving backwards */ +if (mm->mm_sched_epoch >= rq->cpu_epoch) +return ;IIRC this was supposed to be able to wrap; which then means you should write it like:
Quoted 4 lines
if ((mm->mm_sched_epoch - rq->cpu_epoch) >= 0) return;or somesuch.
Okay. Got it.
Tim
Quoted 9 lines
Quoted 9 lines
kernel/sched/fair.c +8
-0+ +guard (raw_spinlock)(&mm->mm_sched_lock); + +if (work->next == work) { +task_work_add (p, work,TWA_RESUME ); +WRITE_ONCE (mm->mm_sched_epoch, rq->cpu_epoch); + } +}
Re: [PATCH v2 05/23] sched/cache: Assign preferred LLC ID to processes
In reply to Peter ZijlstraQuoted 46 lines · Peter Zijlstra
Quoted 29 lines · Tim Chen
With cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar to
numa_preferred_nidin NUMA balancing.kernel/sched/fair.c +9
-0@@ -1300,6 +1300,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched;unsigned long epoch; +int mm_sched_llc =-1 ;if (!sched_cache_enabled ())return ;@@ -1330,6 +1331,23 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; } + +if (mm->mm_sched_cpu !=-1 ) { + mm_sched_llc =llc_id (mm->mm_sched_cpu); + +#ifdef CONFIG_NUMA_BALANCING +/* +* Don't assign preferred LLC if it +* conflicts with NUMA balancing. kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1351
Quoted 10 lines
kernel/sched/fair.c +9
-0+*/ +if (p->numa_preferred_nid >=0 && +cpu_to_node (mm->mm_sched_cpu) != p->numa_preferred_nid) + mm_sched_llc =-1 ; +#endif + } + +if (p->preferred_llc != mm_sched_llc) + p->preferred_llc = mm_sched_llc; }This can of course still happen when
sched_setnuma()gets called. I'm thinking it is not much of an issue because we expect this thing to get called fairly regularly -- at a higher rate thansched_setnuma()at least -- and thus the conflict only exists for a short period of time?
Quoted 1 line
If so, that would make for a good comment.
Sure. Will do.
Quoted 4 lines
Additionally, we could of course search for the busiest LLC inside the node, instead of setting -1. Again, that could live as a comment for future work.
A potential issue with scanning only the preferred node of a single task is that tasks within the same process may have different preferred nodes. For example, task 1 may prefer one node, while tasks 2…n prefer another. If we base the busiest-LLC scan solely on task 1’s preference, we may ignore the preferences of tasks 2…n. Consequently, constraining the preferred LLC according to task 1’s node can interfere with NUMA balancing for the rest of the process. This problem does not arise when all tasks being aggregated belong to the same numa_group, since they will share the same preferred node.
Tim
Re: [PATCH v2 06/23] sched/cache: Track LLC-preferred tasks per runqueue
In reply to Peter ZijlstraQuoted 11 lines · Peter Zijlstra
include/linux/sched.h struct task_struct { · line 1411
Quoted 3 lines · Tim Chen
#ifdef
CONFIG_SCHED_CACHEinclude/linux/sched.h +1
-0struct callback_head cache_work; +/*the p is currently refcounted in a rq's preferred llc stats*/ Shall we have spaces after and before the comment marks?
Also, comment confuses me, I don't see
get_task_struct()/put_task_struct()usage. Did you mean something else with refcount?
It is the accounting for number of tasks preferring a certain LLC on a runqueue during enqueue/dequeue, or when a task's LLC preference changes, by account_llc_enqueue() and account_llc_dequeue()
How about change he comment to
/* LLC preference accounting should be done in dequeue */
Quoted 5 lines
Quoted 4 lines
include/linux/sched.h +1
-0+bool sched_llc_active;int preferred_llc;#endif
Quoted 14 lines
kernel/sched/fair.c static int llc_id(int cpu) · line 1252
Quoted 11 lines
kernel/sched/fair.c +11
-0+static void account_llc_dequeue (struct rq *rq,struct task_struct *p) +{ +int pref_llc; + +/* +* Borrow the uc_se->active from uclamp_rq_inc_id(), +* uclamp_rq_dec_id() to avoid the unbalanced calculation +* of rq statistics. +*/ +if (unlikely (!p->sched_llc_active)) +return ;Another very confusing comment; what? Also, can you please explain (in the new comment) how we get here without having
llc_activeset?
The comment meant to say that we are using a similar mechanism as accounting done in uc_se->active from uclamp_rq_inc_id(). I agree that it confuses more than making things clearer.
How about the following comment to make things clearer:
/*
* Cache aware scheduling was active when the task was enqueued.
* Admin has disabled cache aware scheduling before task was dequeued
* but the accounting has to be kept straight in case cache aware scheduling
* is re-enabled.
*/
Quoted 10 lines
Quoted 10 lines
kernel/sched/fair.c +9
-0+ + pref_llc = p->preferred_llc; +if (pref_llc <0 ) +return ; + + rq->nr_llc_running--; + rq->nr_pref_llc_running -= (pref_llc ==task_llc (p)); + p->sched_llc_active = false; +}
Re: [PATCH v2 07/23] sched/cache: Introduce per runqueue task LLC preference counter
In reply to Peter ZijlstraQuoted 79 lines · Peter Zijlstra
Quoted 64 lines · Tim Chen
kernel/sched/topology.c +64
-0+#ifdef CONFIG_SCHED_CACHE + +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) +{ +unsigned int *new =NULL ; + + new =kcalloc (new_max_llcs,sizeof (unsigned int ), +GFP_KERNEL | __GFP_NOWARN); + +if (!new) { + *gc =NULL ; + }else { +/* +* Place old entry in garbage collector +* for later disposal. +*/ + *gc = old; + } +return new; +} + +static void populate_new_pref_llcs (unsigned int *old,unsigned int *new) +{ +int i; + +if (!old) +return ; + +for (i =0 ; i < max_llcs; i++) + new[i] = old[i]; +} + +static int resize_llc_pref (void ) +{ +unsigned int *__percpu *tmp_llc_pref; +int i, ret =0 ; + +if (new_max_llcs <= max_llcs) +return 0 ; + +/* +* Allocate temp percpu pointer for old llc_pref, +* which will be released after switching to the +* new buffer. +*/ + tmp_llc_pref = alloc_percpu_noprof(unsigned int *); +if (!tmp_llc_pref) +return -ENOMEM ; + +for_each_present_cpu (i) + *per_cpu_ptr (tmp_llc_pref, i) =NULL ; + +/* +* Resize the per rq nr_pref_llc buffer and +* switch to this new buffer. +*/ +for_each_present_cpu (i) { +struct rq_flags rf; +unsigned int *new; +struct rq *rq; + + rq =cpu_rq (i); + new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i)); +if (!new) {kernel/sched/topology.c void sched_domains_mutex_unlock(void) · line 98
Quoted 10 lines
kernel/sched/topology.c +10
-0+ ret = -ENOMEM ; + + goto release_old; + } + +/* +* Locking rq ensures that rq->nr_pref_llc values +* don't change with new task enqueue/dequeue +* when we repopulate the newly enlarged array. +*/ guard(rq_lock_irq)(rq);
Quoted 1 line
Notably, this cannot be with IRQs disabled, as you're doing allocations.
Okay.
Quoted 21 lines
Quoted 21 lines
kernel/sched/topology.c +20
-0+rq_lock_irqsave (rq, &rf); +populate_new_pref_llcs (rq->nr_pref_llc, new); + rq->nr_pref_llc = new; +rq_unlock_irqrestore (rq, &rf); + } + +release_old: +/* +* Load balance is done under rcu_lock. +* Wait for load balance before and during resizing to +* be done. They may refer to old nr_pref_llc[] +* that hasn't been resized. +*/ +synchronize_rcu (); +for_each_present_cpu (i) +kfree (*per_cpu_ptr (tmp_llc_pref, i)); + +free_percpu (tmp_llc_pref); + +/* succeed and update */
Quoted 8 lines
kernel/sched/topology.c void sched_domains_mutex_unlock(void) · line 123
Quoted 5 lines
kernel/sched/topology.c +5
-0+if (!ret) + max_llcs = new_max_llcs; + +return ret; +}I think you need at least
cpus_read_lock(), becausepresent_cpuis dynamic -- but I'm not quite sure what lock is used to serialize it.
Let me check on what is the right lock for making sure present_cpu is not chenged. Thanks.
Tim
Peter Zijlstra
maintainerRe: [PATCH v2 01/23] sched/cache: Introduce infrastructure for cache-aware load balancing
In reply to Tim ChenReview
Quoted 3 lines · Tim Chen
(quoted diff) +2
-0+static unsigned long __no_profile fraction_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched) +static void __no_profile task_cache_work(struct callback_head *work)
What's with the random __no_profile things?
Peter Zijlstra
maintainerRe: [PATCH v2 06/23] sched/cache: Track LLC-preferred tasks per runqueue
In reply to Tim ChenQuoted 19 lines · Tim Chen
Quoted 6 lines
Quoted 6 lines
kernel/sched/fair.c +6
-0+static void account_llc_dequeue (struct rq *rq,struct task_struct *p) +{ +int pref_llc; + +/* +* Borrow the uc_se->active from uclamp_rq_inc_id(), Quoted 8 lines
kernel/sched/fair.c static int llc_id(int cpu) · line 1252
Quoted 5 lines
kernel/sched/fair.c +5
-0+* uclamp_rq_dec_id() to avoid the unbalanced calculation +* of rq statistics. +*/ +if (unlikely (!p->sched_llc_active)) +return ;Another very confusing comment; what? Also, can you please explain (in the new comment) how we get here without having
llc_activeset?The comment meant to say that we are using a similar mechanism as accounting done in
uc_se->activefromuclamp_rq_inc_id(). I agree that it confuses more than making things clearer.
Quoted 8 lines
How about the following comment to make things clearer:
/*
* Cache aware scheduling was active when the task was enqueued.
* Admin has disabled cache aware scheduling before task was dequeued
* but the accounting has to be kept straight in case cache aware scheduling
* is re-enabled.
*/
Is having that sched_cache_enabled() test worth it? account_numa_{en,de}queue() don't seem to have any of this.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
kernel/sched/fair.c +8
-0+ pref_llc = p->preferred_llc; +if (pref_llc <0 ) +return ; + + rq->nr_llc_running--; + rq->nr_pref_llc_running -= (pref_llc ==task_llc (p)); + p->sched_llc_active = false; +}
Peter Zijlstra
maintainerRe: [PATCH v2 07/23] sched/cache: Introduce per runqueue task LLC preference counter
In reply to Tim ChenQuoted 69 lines · Tim Chen
kernel/sched/topology.c +67
-0+static int resize_llc_pref (void ) +{ +unsigned int *__percpu *tmp_llc_pref; +int i, ret =0 ; + +if (new_max_llcs <= max_llcs) +return 0 ; + +/* +* Allocate temp percpu pointer for old llc_pref, +* which will be released after switching to the +* new buffer. +*/ + tmp_llc_pref = alloc_percpu_noprof(unsigned int *); +if (!tmp_llc_pref) +return -ENOMEM ; + +for_each_present_cpu (i) + *per_cpu_ptr (tmp_llc_pref, i) =NULL ; + +/* +* Resize the per rq nr_pref_llc buffer and +* switch to this new buffer. +*/ +for_each_present_cpu (i) { +struct rq_flags rf; +unsigned int *new; +struct rq *rq; + + rq =cpu_rq (i); + new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i)); +if (!new) { + ret = -ENOMEM ; + + goto release_old; + } + +/* +* Locking rq ensures that rq->nr_pref_llc values +* don't change with new task enqueue/dequeue +* when we repopulate the newly enlarged array. +*/ +rq_lock_irqsave (rq, &rf); +populate_new_pref_llcs (rq->nr_pref_llc, new); + rq->nr_pref_llc = new; +rq_unlock_irqrestore (rq, &rf); + } + +release_old: +/* +* Load balance is done under rcu_lock. +* Wait for load balance before and during resizing to +* be done. They may refer to old nr_pref_llc[] +* that hasn't been resized. +*/ +synchronize_rcu (); +for_each_present_cpu (i) +kfree (*per_cpu_ptr (tmp_llc_pref, i)); + +free_percpu (tmp_llc_pref); + +/* succeed and update */ +if (!ret) + max_llcs = new_max_llcs; + +return ret; +}
kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2793
Quoted 7 lines
kernel/sched/topology.c +2
-0@@ -2674,6 +2787,8 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (has_cluster)static_branch_inc_cpuslocked (&sched_cluster_active); +resize_llc_pref (); +if (rq && sched_debug_verbose)pr_info ("root domain span: %*pbl\n" ,cpumask_pr_args (cpu_map));
I suspect people will hate on you for that synchronize_rcu() in there.
Specifically, we do build_sched_domain() for every CPU brought online, this means booting 512 CPUs now includes 512 sync_rcu()s.
Worse, IIRC sync_rcu() is O(n) (or worse -- could be n*ln(n)) in number of CPUs, so the total thing will be O(n^2) (or worse) for bringing CPUs online.
Peter Zijlstra
maintainerRe: [PATCH v2 07/23] sched/cache: Introduce per runqueue task LLC preference counter
In reply to Tim ChenQuoted 57 lines · Tim Chen
kernel/sched/topology.c +57
-0+static int resize_llc_pref (void ) +{ +unsigned int *__percpu *tmp_llc_pref; +int i, ret =0 ; + +if (new_max_llcs <= max_llcs) +return 0 ; + +/* +* Allocate temp percpu pointer for old llc_pref, +* which will be released after switching to the +* new buffer. +*/ + tmp_llc_pref = alloc_percpu_noprof(unsigned int *); +if (!tmp_llc_pref) +return -ENOMEM ; + +for_each_present_cpu (i) + *per_cpu_ptr (tmp_llc_pref, i) =NULL ; + +/* +* Resize the per rq nr_pref_llc buffer and +* switch to this new buffer. +*/ +for_each_present_cpu (i) { +struct rq_flags rf; +unsigned int *new; +struct rq *rq; + + rq =cpu_rq (i); + new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i)); +if (!new) { + ret = -ENOMEM ; + + goto release_old; + } + +/* +* Locking rq ensures that rq->nr_pref_llc values +* don't change with new task enqueue/dequeue +* when we repopulate the newly enlarged array. +*/ +rq_lock_irqsave (rq, &rf); +populate_new_pref_llcs (rq->nr_pref_llc, new); + rq->nr_pref_llc = new; +rq_unlock_irqrestore (rq, &rf); + } + +release_old: +/* +* Load balance is done under rcu_lock. +* Wait for load balance before and during resizing to +* be done. They may refer to old nr_pref_llc[] +* that hasn't been resized. +*/ +synchronize_rcu (); +for_each_present_cpu (i)
kernel/sched/topology.c void sched_domains_mutex_unlock(void) · line 123
Quoted 10 lines
kernel/sched/topology.c +10
-0+kfree (*per_cpu_ptr (tmp_llc_pref, i)); + +free_percpu (tmp_llc_pref); + +/* succeed and update */ +if (!ret) + max_llcs = new_max_llcs; + +return ret; +}
Would it perhaps be easier to stick this thing in rq->sd rather than in rq->nr_pref_llc. That way it automagically switches with the 'new' domain. And then, with a bit of care, a singe load-balance pass should see a consistent view (there should not be reloads of rq->sd -- which will be a bit of an audit I suppose).
Peter Zijlstra
maintainerRe: [PATCH v2 09/23] sched/cache: Count tasks prefering destination LLC in a sched group
In reply to Tim ChenQuoted 33 lines · Tim Chen
During LLC load balancing, tabulate the number of tasks on each runqueue that prefer the LLC contains the
env->dst_cpuin a sched group.For example, consider a system with 4 LLC sched groups (LLC0 to LLC3) balancing towards LLC3. LLC0 has 3 tasks preferring LLC3, LLC1 has 2, and LLC2 has 1. LLC0, having the most tasks preferring LLC3, is selected as the busiest source to pick tasks from.
Within a source LLC, the total number of tasks preferring a destination LLC is computed by summing counts across all CPUs in that LLC. For instance, if LLC0 has CPU0 with 2 tasks and CPU1 with 1 task preferring LLC3, the total for LLC0 is 3.
These statistics allow the load balancer to choose tasks from source sched groups that best match their preferred LLCs.
Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Convertnr_pref_llcarray insg_lb_statsto a single variable as only the dst LLC stat is needed. (K Prateek Nayak)kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)kernel/sched/fair.c +0
-0
kernel/sched/fair.c struct sg_lb_stats { · line 10450
Quoted 6 lines
kernel/sched/fair.c +3
-0@@ -10445,6 +10445,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc; +#endif
At this point I have to note that rq->nr_pref_llc seems like a horrible misnomer, for it being an array, and not an actual number like the naming suggests.
Peter Zijlstra
maintainerRe: [PATCH v2 12/23] sched/cache: Add migrate_llc_task migration type for cache-aware balancing
In reply to Tim ChenQuoted 44 lines · Tim Chen
kernel/sched/fair.c +16
-1@@ -9547,7 +9547,8 @@ enum migration_type { migrate_load =0 , migrate_util, migrate_task, - migrate_misfit + migrate_misfit, + migrate_llc_task };#define LBF_ALL_PINNED 0x01@@ -10134,6 +10135,10 @@ static int detach_tasks(struct lb_env *env) env->imbalance -= util;break ; +case migrate_llc_task: + env->imbalance--; +break ; +case migrate_task: env->imbalance--;break ;@@ -12181,6 +12199,16 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env, }break ; +case migrate_llc_task: +#ifdef CONFIG_SCHED_CACHE + dst_llc =llc_id (env->dst_cpu); +if (dst_llc >=0 && + busiest_pref_llc < rq->nr_pref_llc[dst_llc]) { + busiest_pref_llc = rq->nr_pref_llc[dst_llc]; + busiest = rq; + } +#endif +break ;case migrate_task:if (busiest_nr < nr_running) { busiest_nr = nr_running;
kernel/sched/fair.c static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sd · line 12397
Quoted 7 lines
kernel/sched/fair.c +2
-0@@ -12363,6 +12391,8 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sdcase migrate_misfit:__schedstat_add (sd->lb_imbalance_misfit[idle], env->imbalance);break ; +case migrate_llc_task: +break ; } }
The enum and all switch statements had the same order; you wrecked it!
Re: [PATCH v2 01/23] sched/cache: Introduce infrastructure for cache-aware load balancing
In reply to Peter ZijlstraQuoted 7 lines · Peter Zijlstra
Review
Quoted 3 lines · Tim Chen
(quoted diff) +2
-0+static unsigned long __no_profile fraction_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched) +static void __no_profile task_cache_work(struct callback_head *work)What's with the random
__no_profilethings?
In the early version without this tag we got some error reports by gcov. We will check if this issue still exists and do some investigations.
thanks,
Chenyu
Re: [PATCH v2 02/23] sched/cache: Record per-LLC utilization to guide cache-aware scheduling decisions
In reply to Peter ZijlstraQuoted 28 lines · Peter Zijlstra
include/linux/sched/topology.h struct sched_domain_shared { · line 73
Quoted 11 lines · Tim Chen
include/linux/sched/topology.h +3
-0@@ -68,6 +68,10 @@ struct sched_domain_shared {atomic_t nr_busy_cpus;int has_idle_cores;int nr_idle_scan; +#ifdef CONFIG_SCHED_CACHE +unsigned long util_avg; +unsigned long capacity____cacheline_aligned_in_smp ;This cacheline annotation confuses me, see below.
Quoted 12 lines
include/linux/sched/topology.h +1
-0+#endif };struct sched_domain {kernel/sched/fair.c +0
-0@@ -9622,6 +9622,29 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_return 0 ; }
Quoted 17 lines
kernel/sched/fair.c static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_ · line 9637
Quoted 13 lines
kernel/sched/fair.c +13
-0+#ifdef CONFIG_SCHED_CACHE +/* Called from load balancing paths with rcu_read_lock held */ +static __maybe_unused bool get_llc_stats (int cpu,unsigned long *util, +unsigned long *cap) +{ +struct sched_domain_shared *sd_share; + + sd_share =rcu_dereference (per_cpu (sd_llc_shared, cpu)); +if (!sd_share) +return false; + + *util =READ_ONCE (sd_share->util_avg); + *cap =READ_ONCE (sd_share->capacity);You placed capacity on a separate line, forcing the above to be 2 distinct lines. That seems... sub-optimal?
The reason capacity was placed in a separate cache line is that writes to capacity are not very frequent(cpu hotplug should not happen too-frequently), while writes to util_avg tend to be relatively frequent(changes frequently). If capacity and util_avg were placed in the same cache line, I’m thinking writes to util_avg might invalidate the entire cache line. This could potentially cause cache misses when capacity is read elsewhere, which might lead to false sharing?
thanks,
Chenyu
Quoted 64 lines
Quoted 44 lines
kernel/sched/fair.c +37
-0+ + return true; +} +#else +static inline bool get_llc_stats (int cpu,unsigned long *util, +unsigned long *cap) +{ +return false; +} +#endif /* * can_migrate_task - may task p from runqueue rq be migrated to this_cpu? */ @@ -10592,6 +10615,51 @@ sched_reduced_capacity(struct rq *rq, struct sched_domain *sd)return check_cpu_capacity (rq, sd); } +#ifdef CONFIG_SCHED_CACHE +/* +* Record the statistics for this scheduler group for later +* use. These values guide load balancing on aggregating tasks +* to a LLC. +*/ +static void record_sg_llc_stats (struct lb_env *env, +struct sg_lb_stats *sgs, +struct sched_group *group) +{ +struct sched_domain_shared *sd_share; + +if (!sched_cache_enabled () || env->idle ==CPU_NEWLY_IDLE ) +return ; + +/* Only care about sched domain spanning multiple LLCs */ +if (env->sd->child !=rcu_dereference (per_cpu (sd_llc, env->dst_cpu))) +return ; + +/* +* At this point we know this group spans a LLC domain. +* Record the statistic of this group in its corresponding +* shared LLC domain. +* Note: sd_share cannot be obtained via sd->child->shared, because +* it refers to the domain that covers the local group, while +* sd_share could represent any of the LLC group. +*/ kernel/sched/fair.c sched_reduced_capacity(struct rq *rq, struct sched_domain *sd) · line 10654
Quoted 10 lines
kernel/sched/fair.c +10
-0+ sd_share =rcu_dereference (per_cpu (sd_llc_shared, +cpumask_first (sched_group_span (group)))); +if (!sd_share) +return ; + +if (READ_ONCE (sd_share->util_avg) != sgs->group_util) +WRITE_ONCE (sd_share->util_avg, sgs->group_util); + +if (unlikely (READ_ONCE (sd_share->capacity) != sgs->group_capacity)) +WRITE_ONCE (sd_share->capacity, sgs->group_capacity);And same here.
Quoted 7 lines
kernel/sched/fair.c +7
-0+} +#else +static inline void record_sg_llc_stats (struct lb_env *env,struct sg_lb_stats *sgs, +struct sched_group *group) +{ +} +#endif
Re: [PATCH v2 09/23] sched/cache: Count tasks prefering destination LLC in a sched group
In reply to Peter ZijlstraQuoted 29 lines · Peter Zijlstra
Quoted 28 lines · Tim Chen
During LLC load balancing, tabulate the number of tasks on each runqueue that prefer the LLC contains the
env->dst_cpuin a sched group.For example, consider a system with 4 LLC sched groups (LLC0 to LLC3) balancing towards LLC3. LLC0 has 3 tasks preferring LLC3, LLC1 has 2, and LLC2 has 1. LLC0, having the most tasks preferring LLC3, is selected as the busiest source to pick tasks from.
Within a source LLC, the total number of tasks preferring a destination LLC is computed by summing counts across all CPUs in that LLC. For instance, if LLC0 has CPU0 with 2 tasks and CPU1 with 1 task preferring LLC3, the total for LLC0 is 3.
These statistics allow the load balancer to choose tasks from source sched groups that best match their preferred LLCs.
Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Convertnr_pref_llcarray insg_lb_statsto a single variable as only the dst LLC stat is needed. (K Prateek Nayak)kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
Quoted 15 lines
kernel/sched/fair.c struct sg_lb_stats { · line 10450
Quoted 11 lines
kernel/sched/fair.c +3
-0@@ -10445,6 +10445,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc; +#endif At this point I have to note that
rq->nr_pref_llcseems like a horrible misnomer, for it being an array, and not an actual number like the naming suggests.
In the v2 it seems that rq->nr_pref_llc is not an array anymore, it indicates the number of tasks that want to be migrated to the env->dst_cpu (dst_llc), because these tasks' preferred LLC are env->dst_cpu(dst_llc). Maybe renaming it to rq->nr_pref_dst_llc?
thanks,
Chenyu
Peter Zijlstra
maintainerRe: [PATCH v2 01/23] sched/cache: Introduce infrastructure for cache-aware load balancing
In reply to Chen, Yu CQuoted 11 lines · Yu C
Quoted 7 lines · Peter Zijlstra
Review
Quoted 3 lines · Tim Chen
(quoted diff) +2
-0+static unsigned long __no_profile fraction_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched) +static void __no_profile task_cache_work(struct callback_head *work)What's with the random
__no_profilethings?In the early version without this tag we got some error reports by gcov. We will check if this issue still exists and do some investigations.
That would be weird, nothing in the scheduler has anything like it. So yeah, please see if you can find out where that came from.
Peter Zijlstra
maintainerRe: [PATCH v2 02/23] sched/cache: Record per-LLC utilization to guide cache-aware scheduling decisions
In reply to Chen, Yu CQuoted 45 lines · Yu C
Quoted 34 lines · Peter Zijlstra
include/linux/sched/topology.h struct sched_domain_shared { · line 73
Quoted 11 lines · Tim Chen
include/linux/sched/topology.h +3
-0@@ -68,6 +68,10 @@ struct sched_domain_shared {atomic_t nr_busy_cpus;int has_idle_cores;int nr_idle_scan; +#ifdef CONFIG_SCHED_CACHE +unsigned long util_avg; +unsigned long capacity____cacheline_aligned_in_smp ;This cacheline annotation confuses me, see below.
Quoted 18 lines
include/linux/sched/topology.h +1
-0+#endif };struct sched_domain {kernel/sched/fair.c +8
-0@@ -9622,6 +9622,29 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_return 0 ; } +#ifdef CONFIG_SCHED_CACHE +/* Called from load balancing paths with rcu_read_lock held */ +static __maybe_unused bool get_llc_stats (int cpu,unsigned long *util, +unsigned long *cap) +{ +struct sched_domain_shared *sd_share; + + sd_share =rcu_dereference (per_cpu (sd_llc_shared, cpu));Quoted 9 lines
kernel/sched/fair.c static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_ · line 9637
Quoted 5 lines
kernel/sched/fair.c +5
-0+if (!sd_share) +return false; + + *util =READ_ONCE (sd_share->util_avg); + *cap =READ_ONCE (sd_share->capacity);You placed capacity on a separate line, forcing the above to be 2 distinct lines. That seems... sub-optimal?
Quoted 8 lines
The reason capacity was placed in a separate cache line is that writes to capacity are not very frequent(cpu hotplug should not happen too-frequently), while writes to
util_avgtend to be relatively frequent(changes frequently). If capacity andutil_avgwere placed in the same cache line, I’m thinking writes toutil_avgmight invalidate the entire cache line. This could potentially cause cache misses when capacity is read elsewhere, which might lead to false sharing?
But its introduced here and already read/written together. Is this not premature optimization?
Peter Zijlstra
maintainerRe: [PATCH v2 09/23] sched/cache: Count tasks prefering destination LLC in a sched group
In reply to Chen, Yu CQuoted 46 lines · Yu C
Quoted 29 lines · Peter Zijlstra
Quoted 28 lines · Tim Chen
During LLC load balancing, tabulate the number of tasks on each runqueue that prefer the LLC contains the
env->dst_cpuin a sched group.For example, consider a system with 4 LLC sched groups (LLC0 to LLC3) balancing towards LLC3. LLC0 has 3 tasks preferring LLC3, LLC1 has 2, and LLC2 has 1. LLC0, having the most tasks preferring LLC3, is selected as the busiest source to pick tasks from.
Within a source LLC, the total number of tasks preferring a destination LLC is computed by summing counts across all CPUs in that LLC. For instance, if LLC0 has CPU0 with 2 tasks and CPU1 with 1 task preferring LLC3, the total for LLC0 is 3.
These statistics allow the load balancer to choose tasks from source sched groups that best match their preferred LLCs.
Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Convertnr_pref_llcarray insg_lb_statsto a single variable as only the dst LLC stat is needed. (K Prateek Nayak)kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)Quoted 15 lines
kernel/sched/fair.c struct sg_lb_stats { · line 10450
Quoted 11 lines
kernel/sched/fair.c +3
-0@@ -10445,6 +10445,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc; +#endif At this point I have to note that
rq->nr_pref_llcseems like a horrible misnomer, for it being an array, and not an actual number like the naming suggests.
Quoted 1 line
In the v2 it seems that
rq->nr_pref_llcis not an array anymore, it
From two patches ago:
+ unsigned int *nr_pref_llc;
Its a pointer of some sort.
Quoted 5 lines
indicates
the number of tasks that want to be migrated to theenv->dst_cpu(dst_llc), because
these tasks' preferred LLC areenv->dst_cpu(dst_llc). Maybe renaming it torq->nr_pref_dst_llc?
Like I said in:
https://lkml.kernel.org/r/[email protected]
it might make sense to put it in struct sched_domain instead of struct rq, since then you can allocate and swap it right along with the rest of the domain tree.
Peter Zijlstra
maintainerRe: [PATCH v2 14/23] sched/cache: Consider LLC preference when selecting tasks for load balancing
In reply to Tim ChenQuoted 85 lines · Tim Chen
Currently, task selection from the busiest runqueue ignores LLC preferences. Reorder tasks in the busiest queue to prioritize selection as follows:
1. Tasks preferring the destination CPU's LLC
2. Tasks with no LLC preference
3. Tasks preferring an LLC different from their current one
4. Tasks preferring the LLC they are currently onThis improves the likelihood that tasks are migrated to their preferred LLC.
Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: No change.kernel/sched/fair.c | 66 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 65 insertions(+), 1 deletion(-)kernel/sched/fair.c +54
-0@@ -10092,6 +10092,68 @@ static struct task_struct *detach_one_task(struct lb_env *env)return NULL ; } +#ifdef CONFIG_SCHED_CACHE +/* +* Prepare lists to detach tasks in the following order: +* 1. tasks that prefer dst cpu's LLC +* 2. tasks that have no preference in LLC +* 3. tasks that prefer LLC other than the ones they are on +* 4. tasks that prefer the LLC that they are currently on. +*/ +static struct list_head +*order_tasks_by_llc (struct lb_env *env,struct list_head *tasks) +{ +struct task_struct *p; +LIST_HEAD (pref_old_llc); +LIST_HEAD (pref_new_llc); +LIST_HEAD (no_pref_llc); +LIST_HEAD (pref_other_llc); + +if (!sched_cache_enabled ()) +return tasks; + +if (cpus_share_cache (env->dst_cpu, env->src_cpu)) +return tasks; + +while (!list_empty (tasks)) { + p = list_last_entry(tasks, struct task_struct, se.group_node); + +if (p->preferred_llc ==llc_id (env->dst_cpu)) { +list_move (&p->se.group_node, &pref_new_llc); +continue ; + } + +if (p->preferred_llc ==llc_id (env->src_cpu)) { +list_move (&p->se.group_node, &pref_old_llc); +continue ; + } + +if (p->preferred_llc ==-1 ) { +list_move (&p->se.group_node, &no_pref_llc); +continue ; + } + +list_move (&p->se.group_node, &pref_other_llc); + } + +/* +* We detach tasks from list tail in detach tasks. Put tasks +* to be chosen first at end of list. +*/ +list_splice (&pref_new_llc, tasks); +list_splice (&no_pref_llc, tasks); +list_splice (&pref_other_llc, tasks); +list_splice (&pref_old_llc, tasks); +return tasks; +}
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10188
Quoted 8 lines
kernel/sched/fair.c +2
-0@@ -10119,6 +10181,8 @@ static int detach_tasks(struct lb_env *env)if (env->imbalance <=0 )return 0 ; + tasks =order_tasks_by_llc (env, &env->src_rq->cfs_tasks); +while (!list_empty (tasks)) {/* * We don't want to steal all, otherwise we may be treated likewise,
Humrph. So NUMA balancing does this differently. It skips over the tasks that would degrade locality in can_migrate_task(); and only if nr_balanced_failed is high enough do we ignore that.
Also, if there are a significant number of tasks on the list, this gets in the way of things like loop_break, since it does this sort unconditionally.
Bah, this feels like there is a sane way to integrate all this, but it seems to escape me at the moment. I'll ponder it a bit more.
Peter Zijlstra
maintainerRe: [PATCH v2 15/23] sched/cache: Respect LLC preference in task migration and detach
In reply to Tim Chenkernel/sched/fair.c int can_migrate_task(struct task_struct *p, struct lb_env *env) · line 10033
Quoted 10 lines · Tim Chen
kernel/sched/fair.c +6
-0@@ -10025,6 +10025,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)if (env->flags &LBF_ACTIVE_LB )return 1 ; +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled () && +can_migrate_llc_task (env->src_cpu, env->dst_cpu, p) == mig_forbid && + !task_has_sched_core (p)) +return 0 ; +#endif
This seems wrong:
- it does not let nr_balance_failed override things;
- it takes precedence over migrate_degrade_locality(); you really want to migrate towards the preferred NUMA node over staying on your LLC.
That is, this really wants to be done after migrate_degrades_locality() and only if degrades == 0 or something.
Quoted 6 lines
kernel/sched/fair.c +0
-0degrades =migrate_degrades_locality (p, env);if (!degrades) hot =task_hot (p, env);@@ -10146,12 +10153,55 @@ static struct list_headlist_splice (&pref_old_llc, tasks);return tasks;
kernel/sched/fair.c static struct list_head · line 10164
Quoted 10 lines
kernel/sched/fair.c +9
-0} + +static bool stop_migrate_src_rq (struct task_struct *p, +struct lb_env *env, +int detached) +{ +if (!sched_cache_enabled () || p->preferred_llc ==-1 || +cpus_share_cache (env->src_cpu, env->dst_cpu) || + env->sd->nr_balance_failed) +return false;
But you are allowing nr_balance_failed to override things here.
Quoted 16 lines
kernel/sched/fair.c +16
-0+/* +* Stop migration for the src_rq and pull from a +* different busy runqueue in the following cases: +* +* 1. Trying to migrate task to its preferred +* LLC, but the chosen task does not prefer dest +* LLC - case 3 in order_tasks_by_llc(). This violates +* the goal of migrate_llc_task. However, we should +* stop detaching only if some tasks have been detached +* and the imbalance has been mitigated. +* +* 2. Don't detach more tasks if the remaining tasks want +* to stay. We know the remaining tasks all prefer the +* current LLC, because after order_tasks_by_llc(), the +* tasks that prefer the current LLC are the least favored +* candidates to be migrated out.
kernel/sched/fair.c static struct list_head · line 10191
Quoted 10 lines
kernel/sched/fair.c +10
-0+*/ +if (env->migration_type == migrate_llc_task && + detached &&llc_id (env->dst_cpu) != p->preferred_llc) +return true; + +if (llc_id (env->src_cpu) == p->preferred_llc) +return true; + +return false; +}
Also, I think we have a problem with nr_balance_failed, cache_nice_tries is 1 for SHARE_LLC; this means for failed=0 we ignore:
- ineligible tasks
- llc fail
- node-degrading / hot
and then the very next round, we do all of them at once, without much grading.
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10262
Quoted 9 lines
kernel/sched/fair.c +5
-0@@ -10205,6 +10255,15 @@ static int detach_tasks(struct lb_env *env) p = list_last_entry(tasks, struct task_struct, se.group_node); +/* +* Check if detaching current src_rq should be stopped, because +* doing so would break cache aware load balance. If we stop +* here, the env->flags has LBF_ALL_PINNED, which would cause +* the load balance to pull from another busy runqueue.
Uhh, can_migrate_task() will clear that ALL_PINNED thing if we've found at least one task before getting here.
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10265
Quoted 3 lines
kernel/sched/fair.c +3
-0+*/ +if (stop_migrate_src_rq (p, env, detached)) +break ;
Perhaps split cfs_tasks into multiple lists from the get-go? That avoids this sorting.
Peter Zijlstra
maintainerRe: [PATCH v2 16/23] sched/cache: Introduce sched_cache_present to enable cache aware scheduling for multi LLCs NUMA node
In reply to Tim ChenQuoted 27 lines · Tim Chen
From: Chen Yu <[email protected]>
Cache-aware load balancing should only be enabled if there are more than 1 LLCs within 1 NUMA node.
sched_cache_presentis introduced to indicate whether this platform supports this topology.Suggested-by Libo Chen <[email protected]>Suggested-by Adam Li <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Use flagsched_cache_presentto indicate whether a platform supports cache aware scheduling. Change this flag from staic key. There should be only 1 static key to control the cache aware scheduling. (Peter Zijlstra)kernel/sched/topology.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)kernel/sched/topology.c +0
-0
kernel/sched/topology.c int max_llcs; · line 27
Quoted 4 lines
kernel/sched/topology.c +1
-0@@ -24,6 +24,8 @@ int max_llcs;#ifdef CONFIG_SCHED_CACHE +static bool sched_cache_present;
sched_energy_presentsched_asym_cpucapacitysched_cluster_activesched_smt_present
are all static keys tied to the current topology, why break the streak and make this a boolean?
Peter Zijlstra
maintainerRe: [PATCH v2 17/23] sched/cache: Record the number of active threads per process for cache-aware scheduling
In reply to Tim ChenQuoted 32 lines · Tim Chen
From: Chen Yu <[email protected]>
A performance regression was observed by Prateek when running hackbench with many threads per process (high fd count). To avoid this, processes with a large number of active threads are excluded from cache-aware scheduling.
With
sched_cacheenabled, record the number of active threads in each process during the periodictask_cache_work(). While iterating over CPUs, if the currently running task belongs to the same process as the task that launchedtask_cache_work(), increment the active thread count.This number will be used by subsequent patch to inhibit cache aware load balance.
Suggested-by K Prateek Nayak <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: No change.include/linux/mm_types.h | 1 + kernel/sched/fair.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)include/linux/mm_types.h +0
-0
include/linux/mm_types.h struct mm_struct { · line 1046
Quoted 4 lines
include/linux/mm_types.h +1
-0@@ -1043,6 +1043,7 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; +u64 nr_running_avg____cacheline_aligned_in_smp ;
This is unlikely to do what you hope it does, it will place this variable on a new cacheline, but will not ensure this variable is the only one in that line. Notably ogtables_bytes (the next field in this structure) will share the line.
It might all be less dodgy if you stick these here fields in their own structure, a little like mm_mm_cid or so.
Quoted 22 lines
include/linux/mm_types.h +0
-0#endif #ifdef CONFIG_MMU kernel/sched/fair.c +2
-2@@ -1421,11 +1421,11 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p) static void __no_profile task_cache_work(struct callback_head *work) { -struct task_struct *p = current; +struct task_struct *p = current, *cur;struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ;unsigned long curr_m_a_occ =0 ; -int cpu, m_a_cpu =-1 ; +int cpu, m_a_cpu =-1 , nr_running =0 ;cpumask_var_t cpus;WARN_ON_ONCE (work != &p->cache_work);
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1460
Quoted 3 lines
kernel/sched/fair.c +0
-0@@ -1458,6 +1458,12 @@ static void __no_profile task_cache_work(struct callback_head *work) m_occ = occ; m_cpu = i; }
guard(rcu)();
Quoted 10 lines
kernel/sched/fair.c +6
-0+rcu_read_lock (); + cur =rcu_dereference (cpu_rq (i)->curr); +if (cur && !(cur->flags & (PF_EXITING |PF_KTHREAD )) && + cur->mm == mm) + nr_running++; +rcu_read_unlock (); }/*
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1512
Quoted 6 lines
kernel/sched/fair.c +1
-0@@ -1501,6 +1507,7 @@ static void __no_profile task_cache_work(struct callback_head *work) mm->mm_sched_cpu = m_a_cpu; } +update_avg (&mm->nr_running_avg, nr_running);free_cpumask_var (cpus); }
Its a wee bit weird to introduce nr_running_avg without its user. Makes it hard to see what's what.
Peter Zijlstra
maintainerRe: [PATCH v2 16/23] sched/cache: Introduce sched_cache_present to enable cache aware scheduling for multi LLCs NUMA node
In reply to Peter ZijlstraQuoted 38 lines · Peter Zijlstra
Quoted 27 lines · Tim Chen
From: Chen Yu <[email protected]>
Cache-aware load balancing should only be enabled if there are more than 1 LLCs within 1 NUMA node.
sched_cache_presentis introduced to indicate whether this platform supports this topology.Suggested-by Libo Chen <[email protected]>Suggested-by Adam Li <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Use flagsched_cache_presentto indicate whether a platform supports cache aware scheduling. Change this flag from staic key. There should be only 1 static key to control the cache aware scheduling. (Peter Zijlstra)kernel/sched/topology.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)kernel/sched/topology.c +0
-0kernel/sched/topology.c int max_llcs; · line 27
Quoted 4 lines
kernel/sched/topology.c +1
-0@@ -24,6 +24,8 @@ int max_llcs;#ifdef CONFIG_SCHED_CACHE +static bool sched_cache_present;
sched_energy_presentsched_asym_cpucapacitysched_cluster_activesched_smt_present
Quoted 2 lines
are all static keys tied to the current topology, why break the streak and make this a boolean?
Also, patch doesn't use sched_cache_present at all, so perhaps just drop it on the floor entirely?
Peter Zijlstra
maintainerRe: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Tim ChenQuoted 37 lines · Tim Chen
include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +12
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \
kernel/sched/debug.c static const struct file_operations sched_feat_fops = { · line 193
Quoted 10 lines
kernel/sched/debug.c +8
-0+ buf[cnt] ='\0' ; \ +if (kstrtouint (buf,10 , &val)) \ +return -EINVAL ; \ +if (val > (max)) \ +return -EINVAL ; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \
Oh gawd :-(
Please just write out all the various write methods and use kstrtoul_from_user() and kstrtobool_from_user() where applicable.
Quoted 25 lines
kernel/sched/debug.c +25
-0+ *ppos += cnt; \ +return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 );
Re: [PATCH v2 07/23] sched/cache: Introduce per runqueue task LLC preference counter
In reply to Peter ZijlstraQuoted 81 lines · Peter Zijlstra
Quoted 69 lines · Tim Chen
kernel/sched/topology.c +67
-0+static int resize_llc_pref (void ) +{ +unsigned int *__percpu *tmp_llc_pref; +int i, ret =0 ; + +if (new_max_llcs <= max_llcs) +return 0 ; + +/* +* Allocate temp percpu pointer for old llc_pref, +* which will be released after switching to the +* new buffer. +*/ + tmp_llc_pref = alloc_percpu_noprof(unsigned int *); +if (!tmp_llc_pref) +return -ENOMEM ; + +for_each_present_cpu (i) + *per_cpu_ptr (tmp_llc_pref, i) =NULL ; + +/* +* Resize the per rq nr_pref_llc buffer and +* switch to this new buffer. +*/ +for_each_present_cpu (i) { +struct rq_flags rf; +unsigned int *new; +struct rq *rq; + + rq =cpu_rq (i); + new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i)); +if (!new) { + ret = -ENOMEM ; + + goto release_old; + } + +/* +* Locking rq ensures that rq->nr_pref_llc values +* don't change with new task enqueue/dequeue +* when we repopulate the newly enlarged array. +*/ +rq_lock_irqsave (rq, &rf); +populate_new_pref_llcs (rq->nr_pref_llc, new); + rq->nr_pref_llc = new; +rq_unlock_irqrestore (rq, &rf); + } + +release_old: +/* +* Load balance is done under rcu_lock. +* Wait for load balance before and during resizing to +* be done. They may refer to old nr_pref_llc[] +* that hasn't been resized. +*/ +synchronize_rcu (); +for_each_present_cpu (i) +kfree (*per_cpu_ptr (tmp_llc_pref, i)); + +free_percpu (tmp_llc_pref); + +/* succeed and update */ +if (!ret) + max_llcs = new_max_llcs; + +return ret; +}kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2793
Quoted 7 lines
kernel/sched/topology.c +2
-0@@ -2674,6 +2787,8 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (has_cluster)static_branch_inc_cpuslocked (&sched_cluster_active); +resize_llc_pref (); +if (rq && sched_debug_verbose)pr_info ("root domain span: %*pbl\n" ,cpumask_pr_args (cpu_map));I suspect people will hate on you for that
synchronize_rcu()in there.
Quoted 7 lines
Specifically, we do
build_sched_domain()for every CPU brought online, this means booting 512 CPUs now includes 512 sync_rcu()s. Worse, IIRCsync_rcu()is O(n) (or worse -- could be n*ln(n)) in number of CPUs, so the total thing will be O(n^2) (or worse) for bringing CPUs online.
Though we only do sychronize_rcu in resize_llc_pref() when we encounter a new LLC, and need a larger array of LLCs, and not on
every CPU. That said, I agree that free is better done in a RCU call back to avoid scynchronize_rcu overhead.
Tim
Re: [PATCH v2 07/23] sched/cache: Introduce per runqueue task LLC preference counter
In reply to Peter ZijlstraQuoted 67 lines · Peter Zijlstra
Quoted 65 lines · Tim Chen
kernel/sched/topology.c +65
-0+static int resize_llc_pref (void ) +{ +unsigned int *__percpu *tmp_llc_pref; +int i, ret =0 ; + +if (new_max_llcs <= max_llcs) +return 0 ; + +/* +* Allocate temp percpu pointer for old llc_pref, +* which will be released after switching to the +* new buffer. +*/ + tmp_llc_pref = alloc_percpu_noprof(unsigned int *); +if (!tmp_llc_pref) +return -ENOMEM ; + +for_each_present_cpu (i) + *per_cpu_ptr (tmp_llc_pref, i) =NULL ; + +/* +* Resize the per rq nr_pref_llc buffer and +* switch to this new buffer. +*/ +for_each_present_cpu (i) { +struct rq_flags rf; +unsigned int *new; +struct rq *rq; + + rq =cpu_rq (i); + new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i)); +if (!new) { + ret = -ENOMEM ; + + goto release_old; + } + +/* +* Locking rq ensures that rq->nr_pref_llc values +* don't change with new task enqueue/dequeue +* when we repopulate the newly enlarged array. +*/ +rq_lock_irqsave (rq, &rf); +populate_new_pref_llcs (rq->nr_pref_llc, new); + rq->nr_pref_llc = new; +rq_unlock_irqrestore (rq, &rf); + } + +release_old: +/* +* Load balance is done under rcu_lock. +* Wait for load balance before and during resizing to +* be done. They may refer to old nr_pref_llc[] +* that hasn't been resized. +*/ +synchronize_rcu (); +for_each_present_cpu (i) +kfree (*per_cpu_ptr (tmp_llc_pref, i)); + +free_percpu (tmp_llc_pref); + +/* succeed and update */ +if (!ret) + max_llcs = new_max_llcs; +
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return ret; +}Would it perhaps be easier to stick this thing in
rq->sdrather than inrq->nr_pref_llc. That way it automagically switches with the 'new' domain. And then, with a bit of care, a singe load-balance pass should see a consistent view (there should not be reloads ofrq->sd-- which will be a bit of an audit I suppose).
We need nr_pref_llc information at the runqueue level because the load balancer must identify which specific rq has the largest number of tasks that prefer a given destination LLC. If we move the counter to the LLC’s sd level, we would only know the aggregate number of tasks in the entire LLC that prefer that destination—not which rq they reside on. Without per-rq counts, we would not be able to select the correct source rq to pull tasks from.
The only way this could work at the LLC-sd level is if all CPUs within the LLC shared a single runqueue, which is not the case today.
Let me know if I understand your comments correctly.
Tim
Re: [PATCH v2 09/23] sched/cache: Count tasks prefering destination LLC in a sched group
In reply to Peter ZijlstraQuoted 47 lines · Peter Zijlstra
Quoted 46 lines · Yu C
Quoted 44 lines · Peter Zijlstra
Quoted 33 lines · Tim Chen
During LLC load balancing, tabulate the number of tasks on each runqueue that prefer the LLC contains the
env->dst_cpuin a sched group.For example, consider a system with 4 LLC sched groups (LLC0 to LLC3) balancing towards LLC3. LLC0 has 3 tasks preferring LLC3, LLC1 has 2, and LLC2 has 1. LLC0, having the most tasks preferring LLC3, is selected as the busiest source to pick tasks from.
Within a source LLC, the total number of tasks preferring a destination LLC is computed by summing counts across all CPUs in that LLC. For instance, if LLC0 has CPU0 with 2 tasks and CPU1 with 1 task preferring LLC3, the total for LLC0 is 3.
These statistics allow the load balancer to choose tasks from source sched groups that best match their preferred LLCs.
Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Convertnr_pref_llcarray insg_lb_statsto a single variable as only the dst LLC stat is needed. (K Prateek Nayak)kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)kernel/sched/fair.c +0
-0kernel/sched/fair.c struct sg_lb_stats { · line 10450
Quoted 6 lines
kernel/sched/fair.c +3
-0@@ -10445,6 +10445,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc; +#endif At this point I have to note that
rq->nr_pref_llcseems like a horrible misnomer, for it being an array, and not an actual number like the naming suggests.
Quoted 7 lines
Quoted 1 line
In the v2 it seems that
rq->nr_pref_llcis not an array anymore, itFrom two patches ago:
+ unsigned int
*nr_pref_llc;Its a pointer of some sort.
Perhaps I should used a different name here when I update this patch for v2.
rq->nr_pref_llc[] is an array as it records the number of tasks preferring each LLC.
Howeversgs->nr_pref_llc is a single number representing the number of tasks preferring the current domain preferring the destination LLC.
Sorry for using the same name that may have created this confusion.
Quoted 8 lines
Quoted 7 lines
indicates
the number of tasks that want to be migrated to theenv->dst_cpu(dst_llc), because
these tasks' preferred LLC areenv->dst_cpu(dst_llc). Maybe renaming it torq->nr_pref_dst_llc?
Quoted 7 lines
Like I said in:
https://lkml.kernel.org/r/[email protected]
it might make sense to put it in struct
sched_domaininstead of struct rq, since then you can allocate and swap it right along with the rest of the domain tree.
Sent a separate reply to that comment to clarify why I think we need nr_pref_llc[] per run queue.
Tim
Re: [PATCH v2 09/23] sched/cache: Count tasks prefering destination LLC in a sched group
In reply to Peter ZijlstraQuoted 47 lines · Peter Zijlstra
Quoted 46 lines · Yu C
Quoted 44 lines · Peter Zijlstra
Quoted 33 lines · Tim Chen
During LLC load balancing, tabulate the number of tasks on each runqueue that prefer the LLC contains the
env->dst_cpuin a sched group.For example, consider a system with 4 LLC sched groups (LLC0 to LLC3) balancing towards LLC3. LLC0 has 3 tasks preferring LLC3, LLC1 has 2, and LLC2 has 1. LLC0, having the most tasks preferring LLC3, is selected as the busiest source to pick tasks from.
Within a source LLC, the total number of tasks preferring a destination LLC is computed by summing counts across all CPUs in that LLC. For instance, if LLC0 has CPU0 with 2 tasks and CPU1 with 1 task preferring LLC3, the total for LLC0 is 3.
These statistics allow the load balancer to choose tasks from source sched groups that best match their preferred LLCs.
Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Convertnr_pref_llcarray insg_lb_statsto a single variable as only the dst LLC stat is needed. (K Prateek Nayak)kernel/sched/fair.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)kernel/sched/fair.c +0
-0kernel/sched/fair.c struct sg_lb_stats { · line 10450
Quoted 6 lines
kernel/sched/fair.c +3
-0@@ -10445,6 +10445,9 @@ struct sg_lb_stats {unsigned int nr_numa_running;unsigned int nr_preferred_running;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc; +#endif At this point I have to note that
rq->nr_pref_llcseems like a horrible misnomer, for it being an array, and not an actual number like the naming suggests.
Quoted 9 lines
Quoted 1 line
In the v2 it seems that
rq->nr_pref_llcis not an array anymore, itFrom two patches ago:
+ unsigned int
*nr_pref_llc;Its a pointer of some sort.
Ah I see, I thought it was the variable in the sgs structure.
Quoted 6 lines
Quoted 5 lines
indicates
the number of tasks that want to be migrated to theenv->dst_cpu(dst_llc), because
these tasks' preferred LLC areenv->dst_cpu(dst_llc). Maybe renaming it torq->nr_pref_dst_llc?
Quoted 7 lines
Like I said in:
https://lkml.kernel.org/r/[email protected]it might make sense to put it in struct
sched_domaininstead of struct rq, since then you can allocate and swap it right along with the rest of the domain tree.
I'll think more about this. Currently the per cpu rq's nr_pref_llc is used to identify the "busiest" runqueue. The busiest runqueue has most threads wanted to be migrated to llc_id(env->dst_cpu), because the threads' preferred LLC is there - in this way, the migration success ratio to the preferred LLC would be higher without breaking the imbalance too much IMHO. So we might have to track the per cpu rq's statistics during enqueue/dequeue. If we put it in the domain, not sure how to track that.
Thanks,
Chenyu
Re: [PATCH v2 02/23] sched/cache: Record per-LLC utilization to guide cache-aware scheduling decisions
In reply to Peter ZijlstraQuoted 55 lines · Peter Zijlstra
Quoted 45 lines · Yu C
Quoted 34 lines · Peter Zijlstra
include/linux/sched/topology.h struct sched_domain_shared { · line 73
Quoted 11 lines · Tim Chen
include/linux/sched/topology.h +3
-0@@ -68,6 +68,10 @@ struct sched_domain_shared {atomic_t nr_busy_cpus;int has_idle_cores;int nr_idle_scan; +#ifdef CONFIG_SCHED_CACHE +unsigned long util_avg; +unsigned long capacity____cacheline_aligned_in_smp ;This cacheline annotation confuses me, see below.
Quoted 18 lines
include/linux/sched/topology.h +1
-0+#endif };struct sched_domain {kernel/sched/fair.c +8
-0@@ -9622,6 +9622,29 @@ static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_return 0 ; } +#ifdef CONFIG_SCHED_CACHE +/* Called from load balancing paths with rcu_read_lock held */ +static __maybe_unused bool get_llc_stats (int cpu,unsigned long *util, +unsigned long *cap) +{ +struct sched_domain_shared *sd_share; + + sd_share =rcu_dereference (per_cpu (sd_llc_shared, cpu));Quoted 9 lines
kernel/sched/fair.c static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_ · line 9637
Quoted 5 lines
kernel/sched/fair.c +5
-0+if (!sd_share) +return false; + + *util =READ_ONCE (sd_share->util_avg); + *cap =READ_ONCE (sd_share->capacity);You placed capacity on a separate line, forcing the above to be 2 distinct lines. That seems... sub-optimal?
Quoted 8 lines
The reason capacity was placed in a separate cache line is that writes to capacity are not very frequent(cpu hotplug should not happen too-frequently), while writes to
util_avgtend to be relatively frequent(changes frequently). If capacity andutil_avgwere placed in the same cache line, I’m thinking writes toutil_avgmight invalidate the entire cache line. This could potentially cause cache misses when capacity is read elsewhere, which might lead to false sharing?
Quoted 2 lines
But its introduced here and already read/written together. Is this not premature optimization?
I see, since they are read together, there could be a pre-load of adjacent cachelines I suppose, I'll remove this cache alignment and check if there is any performance impact.
Thanks,
Chenyu
Re: [PATCH v2 01/23] sched/cache: Introduce infrastructure for cache-aware load balancing
In reply to Tim ChenHi, Peter, Chen Yu and Tim:
Quoted 73 lines · Tim Chen
From: "Peter Zijlstra (Intel)" <[email protected]>
Adds infrastructure to enable cache-aware load balancing, which improves cache locality by grouping tasks that share resources within the same cache domain. This reduces cache misses and improves overall data access efficiency.
In this initial implementation, threads belonging to the same process are treated as entities that likely share working sets. The mechanism tracks per-process CPU occupancy across cache domains and attempts to migrate threads toward cache-hot domains where their process already has active threads, thereby enhancing locality.
This provides a basic model for cache affinity. While the current code targets the last-level cache (LLC), the approach could be extended to other domain types such as clusters (L2) or node-internal groupings.
At present, the mechanism selects the CPU within an LLC that has the highest recent runtime. Subsequent patches in this series will use this information in the load-balancing path to guide task placement toward preferred LLCs.
In the future, more advanced policies could be integrated through NUMA balancing-for example, migrating a task to its preferred LLC when spare capacity exists, or swapping tasks across LLCs to improve cache affinity. Grouping of tasks could also be generalized from that of a process to be that of a NUMA group, or be user configurable.
Originally-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Restore the original CPU scan to cover all online CPUs, rather than scanning within the preferred NUMA node. (Peter Zijlstra)Use
rq->currinstead ofrq->donor. (K Prateek Nayak)Minor fix in
task_tick_cache()to use
if (mm->mm_sched_epoch>=rq->cpu_epoch)
to avoidmm_sched_epochgoing backwards.include/linux/mm_types.h | 44 +++++++ include/linux/sched.h | 11 ++ init/Kconfig | 11 ++ kernel/fork.c | 6 + kernel/sched/core.c | 6 + kernel/sched/fair.c | 258 +++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 8 ++ 7 files changed, 344 insertions(+)include/linux/mm_types.h +5
-0@@ -939,6 +939,11 @@ typedef struct {DECLARE_BITMAP (__mm_flags,NUM_MM_FLAG_BITS ); }__private mm_flags_t; +struct mm_sched { +u64 runtime; +unsigned long epoch; +}; +struct kioctx_table ;struct iommu_mm_data ;struct mm_struct {@@ -1029,6 +1034,17 @@ struct mm_struct {*/ raw_spinlock_t cpus_allowed_lock;
include/linux/mm_types.h struct mm_struct { · line 1045
Quoted 10 lines
include/linux/mm_types.h +9
-0#endif +#ifdef CONFIG_SCHED_CACHE +/* +* Track per-cpu-per-process occupancy as a proxy for cache residency. +* See account_mm_sched() and ... +*/ +struct mm_sched __percpu *pcpu_sched; +raw_spinlock_t mm_sched_lock; +unsigned long mm_sched_epoch; +int mm_sched_cpu;
As we discussed earlier,I continue to believe that dedicating 'mm_sched_cpu' to handle the aggregated hotspots of all threads is inappropriate, as the multiple threads lack a necessary correlation in our real application.
So, I was wondering if we could put this variable into struct task_struct, That allows us to better monitor the hotspot CPU of each thread, despite some details needing consideration.
Quoted 460 lines
include/linux/mm_types.h +30
-0+#endif +#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */ #endif @@ -1487,6 +1503,34 @@ static inline unsigned int mm_cid_size(void)static inline void mm_set_cpus_allowed (struct mm_struct *mm,const struct cpumask *cpumask) { }#endif /* CONFIG_SCHED_MM_CID */ +#ifdef CONFIG_SCHED_CACHE +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *pcpu_sched); + +static inline int mm_alloc_sched_noprof (struct mm_struct *mm) +{ + struct mm_sched __percpu *pcpu_sched = alloc_percpu_noprof(struct mm_sched); + +if (!pcpu_sched) +return -ENOMEM ; + +mm_init_sched (mm, pcpu_sched); +return 0 ; +} + +#define mm_alloc_sched (...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) + +static inline void mm_destroy_sched (struct mm_struct *mm) +{ +free_percpu (mm->pcpu_sched); + mm->pcpu_sched =NULL ; +} +#else /* !CONFIG_SCHED_CACHE */ + +static inline int mm_alloc_sched (struct mm_struct *mm) {return 0 ; } +static inline void mm_destroy_sched (struct mm_struct *mm) { } + +#endif /* CONFIG_SCHED_CACHE */ +struct mmu_gather ;extern void tlb_gather_mmu (struct mmu_gather *tlb,struct mm_struct *mm);extern void tlb_gather_mmu_fullmm (struct mmu_gather *tlb,struct mm_struct *mm);include/linux/sched.h +11
-0@@ -1406,6 +1406,10 @@ struct task_struct {unsigned long numa_pages_migrated;#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +struct callback_head cache_work; +#endif +#ifdef CONFIG_RSEQ struct rseq __user *rseq;u32 rseq_len;@@ -2428,4 +2432,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable()) +#ifdef CONFIG_SCHED_CACHE +static inline bool sched_cache_enabled (void ) +{ +return false; +} +#endif +#endif init/Kconfig +11
-0@@ -983,6 +983,17 @@ config NUMA_BALANCING This system will be inactive on UMA systems. +config SCHED_CACHE + bool "Cache aware load balance" + default y + depends on SMP + help + When enabled, the scheduler will attempt to aggregate tasks from + the same process onto a single Last Level Cache (LLC) domain when + possible. This improves cache locality by keeping tasks that share + resources within the same cache domain, reducing cache misses and + lowering data access latency. + config NUMA_BALANCING_DEFAULT_ENABLED bool "Automatically enable NUMA aware memory/task placement" default ykernel/fork.c +6
-0@@ -680,6 +680,7 @@ void __mmdrop(struct mm_struct *mm)cleanup_lazy_tlbs (mm);WARN_ON_ONCE (mm == current->active_mm); +mm_destroy_sched (mm);mm_free_pgd (mm);mm_free_id (mm);destroy_context (mm);@@ -1083,6 +1084,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,if (mm_alloc_cid (mm, p)) goto fail_cid; +if (mm_alloc_sched (mm)) + goto fail_sched; +if (percpu_counter_init_many (mm->rss_stat,0 ,GFP_KERNEL_ACCOUNT ,NR_MM_COUNTERS )) goto fail_pcpu;@@ -1092,6 +1096,8 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,return mm; fail_pcpu: +mm_destroy_sched (mm); +fail_sched:mm_destroy_cid (mm); fail_cid:destroy_context (mm);kernel/sched/core.c +6
-0@@ -4488,6 +4488,7 @@ static void __sched_fork(u64 clone_flags, struct task_struct *p) p->wake_entry.u_flags =CSD_TYPE_TTWU ; p->migration_pending =NULL ;init_sched_mm_cid (p); +init_sched_mm (p); }DEFINE_STATIC_KEY_FALSE (sched_numa_balancing);@@ -8791,6 +8792,11 @@ void __init sched_init(void) rq->core_cookie =0UL ;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spin_lock_init (&rq->cpu_epoch_lock); + rq->cpu_epoch_next = jiffies; +#endif +zalloc_cpumask_var_node (&rq->scratch_mask,GFP_KERNEL ,cpu_to_node (i)); }kernel/sched/fair.c +258
-0@@ -1152,6 +1152,8 @@ void post_init_entity_util_avg(struct task_struct *p) sa->runnable_avg = sa->util_avg; } +static inline void account_mm_sched (struct rq *rq,struct task_struct *p,s64 delta_exec); +static s64 update_se (struct rq *rq,struct sched_entity *se) {u64 now =rq_clock_task (rq);@@ -1174,6 +1176,7 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)trace_sched_stat_runtime (running, delta_exec);account_group_exec_runtime (running, delta_exec); +account_mm_sched (rq, running, delta_exec);/* cgroup time is always accounted against the donor */ cgroup_account_cputime (donor, delta_exec);@@ -1193,6 +1196,259 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)return delta_exec; } +#ifdef CONFIG_SCHED_CACHE + +/* +* XXX numbers come from a place the sun don't shine -- probably wants to be SD +* tunable or so. +*/ +#define EPOCH_PERIOD (HZ / 100)/* 10 ms */ +#define EPOCH_LLC_AFFINITY_TIMEOUT 5/* 50 ms */ + +static int llc_id (int cpu) +{ +if (cpu <0 ) +return -1 ; + +return per_cpu (sd_llc_id, cpu); +} + +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) +{ +unsigned long epoch; +int i; + +for_each_possible_cpu (i) { +struct mm_sched *pcpu_sched =per_cpu_ptr (_pcpu_sched, i); +struct rq *rq =cpu_rq (i); + + pcpu_sched->runtime =0 ; + pcpu_sched->epoch = rq->cpu_epoch; + epoch = rq->cpu_epoch; + } + +raw_spin_lock_init (&mm->mm_sched_lock); + mm->mm_sched_epoch = epoch; + mm->mm_sched_cpu =-1 ; + +/* +* The update to mm->pcpu_sched should not be reordered +* before initialization to mm's other fields, in case +* the readers may get invalid mm_sched_epoch, etc. +*/ +smp_store_release (&mm->pcpu_sched, _pcpu_sched); +} + +/* because why would C be fully specified */ +static __always_inline void __shr_u64(u64 *val, unsigned int n) +{ +if (n >=64 ) { + *val =0 ; +return ; + } + *val >>= n; +} + +static inline void __update_mm_sched (struct rq *rq,struct mm_sched *pcpu_sched) +{ +lockdep_assert_held (&rq->cpu_epoch_lock); + +unsigned long n, now = jiffies; +long delta = now - rq->cpu_epoch_next; + +if (delta >0 ) { + n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + rq->cpu_epoch += n; + rq->cpu_epoch_next += n *EPOCH_PERIOD ; +__shr_u64 (&rq->cpu_runtime, n); + } + + n = rq->cpu_epoch - pcpu_sched->epoch; +if (n) { + pcpu_sched->epoch += n; +__shr_u64 (&pcpu_sched->runtime, n); + } +} + +static unsigned long __no_profile fraction_mm_sched (struct rq *rq,struct mm_sched *pcpu_sched) +{ +guard (raw_spinlock_irqsave)(&rq->cpu_epoch_lock); + +__update_mm_sched (rq, pcpu_sched); + +/* +* Runtime is a geometric series (r=0.5) and as such will sum to twice +* the accumulation period, this means the multiplcation here should +* not overflow. +*/ +return div64_u64 (NICE_0_LOAD * pcpu_sched->runtime, rq->cpu_runtime +1 ); +} + +static inline +void account_mm_sched (struct rq *rq,struct task_struct *p,s64 delta_exec) +{ +struct mm_struct *mm = p->mm; +struct mm_sched *pcpu_sched; +unsigned long epoch; + +if (!sched_cache_enabled ()) +return ; + +if (p->sched_class != &fair_sched_class) +return ; +/* +* init_task and kthreads don't having mm +*/ +if (!mm || !mm->pcpu_sched) +return ; + + pcpu_sched =per_cpu_ptr (p->mm->pcpu_sched,cpu_of (rq)); + + scoped_guard (raw_spinlock, &rq->cpu_epoch_lock) { +__update_mm_sched (rq, pcpu_sched); + pcpu_sched->runtime += delta_exec; + rq->cpu_runtime += delta_exec; + epoch = rq->cpu_epoch; + } + +/* +* If this task hasn't hit task_cache_work() for a while, or it +* has only 1 thread, invalidate its preferred state. +*/ +if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || +get_nr_threads (p) <=1 ) { +if (mm->mm_sched_cpu !=-1 ) + mm->mm_sched_cpu =-1 ; + } +} + +static void task_tick_cache (struct rq *rq,struct task_struct *p) +{ +struct callback_head *work = &p->cache_work; +struct mm_struct *mm = p->mm; + +if (!sched_cache_enabled ()) +return ; + +if (!mm || !mm->pcpu_sched) +return ; + +/* avoid moving backwards */ +if (mm->mm_sched_epoch >= rq->cpu_epoch) +return ; + +guard (raw_spinlock)(&mm->mm_sched_lock); + +if (work->next == work) { +task_work_add (p, work,TWA_RESUME ); +WRITE_ONCE (mm->mm_sched_epoch, rq->cpu_epoch); + } +} + +static void __no_profile task_cache_work(struct callback_head *work) +{ +struct task_struct *p = current; +struct mm_struct *mm = p->mm; +unsigned long m_a_occ =0 ; +unsigned long curr_m_a_occ =0 ; +int cpu, m_a_cpu =-1 ; +cpumask_var_t cpus; + +WARN_ON_ONCE (work != &p->cache_work); + + work->next = work; + +if (p->flags &PF_EXITING ) +return ; + +if (!zalloc_cpumask_var (&cpus,GFP_KERNEL )) +return ; + +scoped_guard (cpus_read_lock) { +cpumask_copy (cpus, cpu_online_mask); + +for_each_cpu (cpu, cpus) { +/* XXX sched_cluster_active */ +struct sched_domain *sd =per_cpu (sd_llc, cpu); +unsigned long occ, m_occ =0 , a_occ =0 ; +int m_cpu =-1 , i; + +if (!sd) +continue ; + +for_each_cpu (i,sched_domain_span (sd)) { + occ =fraction_mm_sched (cpu_rq (i), +per_cpu_ptr (mm->pcpu_sched, i)); + a_occ += occ; +if (occ > m_occ) { + m_occ = occ; + m_cpu = i; + } + } + +/* +* Compare the accumulated occupancy of each LLC. The +* reason for using accumulated occupancy rather than average +* per CPU occupancy is that it works better in asymmetric LLC +* scenarios. +* For example, if there are 2 threads in a 4CPU LLC and 3 +* threads in an 8CPU LLC, it might be better to choose the one +* with 3 threads. However, this would not be the case if the +* occupancy is divided by the number of CPUs in an LLC (i.e., +* if average per CPU occupancy is used). +* Besides, NUMA balancing fault statistics behave similarly: +* the total number of faults per node is compared rather than +* the average number of faults per CPU. This strategy is also +* followed here. +*/ +if (a_occ > m_a_occ) { + m_a_occ = a_occ; + m_a_cpu = m_cpu; + } + +if (llc_id (cpu) ==llc_id (mm->mm_sched_cpu)) + curr_m_a_occ = a_occ; + +cpumask_andnot (cpus, cpus,sched_domain_span (sd)); + } + } + +if (m_a_occ > (2 * curr_m_a_occ)) { +/* +* Avoid switching mm_sched_cpu too fast. +* The reason to choose 2X is because: +* 1. It is better to keep the preferred LLC stable, +* rather than changing it frequently and cause migrations +* 2. 2X means the new preferred LLC has at least 1 more +* busy CPU than the old one(200% vs 100%, eg) +* 3. 2X is chosen based on test results, as it delivers +* the optimal performance gain so far. +*/ + mm->mm_sched_cpu = m_a_cpu; + } + +free_cpumask_var (cpus); +} + +void init_sched_mm (struct task_struct *p) +{ +struct callback_head *work = &p->cache_work; + +init_task_work (work, task_cache_work); + work->next = work; +} + +#else + +static inline void account_mm_sched (struct rq *rq,struct task_struct *p, +s64 delta_exec) { } + +void init_sched_mm (struct task_struct *p) { } + +static void task_tick_cache (struct rq *rq,struct task_struct *p) { } + +#endif +/* * Used by other classes to account runtime. */ @@ -13124,6 +13380,8 @@ static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)if (static_branch_unlikely (&sched_numa_balancing))task_tick_numa (rq, curr); +task_tick_cache (rq, curr); +update_misfit_status (curr, rq);check_update_overutilized_status (task_rq (curr));kernel/sched/sched.h +8
-0@@ -1194,6 +1194,12 @@ struct rq {u64 clock_pelt_idle_copy;u64 clock_idle_copy;#endif +#ifdef CONFIG_SCHED_CACHE +raw_spinlock_t cpu_epoch_lock____cacheline_aligned ; +u64 cpu_runtime; +unsigned long cpu_epoch; +unsigned long cpu_epoch_next; +#endif atomic_t nr_iowait;@@ -3819,6 +3825,8 @@ static inline void task_tick_mm_cid(struct rq *rq, struct task_struct *curr) { }static inline void init_sched_mm_cid (struct task_struct *t) { }#endif /* !CONFIG_SCHED_MM_CID */ +extern void init_sched_mm (struct task_struct *p); +extern u64 avg_vruntime (struct cfs_rq *cfs_rq);extern int entity_eligible (struct cfs_rq *cfs_rq,struct sched_entity *se); static inline
Peter Zijlstra
maintainerRe: [PATCH v2 07/23] sched/cache: Introduce per runqueue task LLC preference counter
In reply to Tim ChenQuoted 16 lines · Tim Chen
Quoted 7 lines · Peter Zijlstra
On Wed, Dec 03, 2025 at 03:07:26PM -0800, Tim Chen wrote:
Would it perhaps be easier to stick this thing in
rq->sdrather than inrq->nr_pref_llc. That way it automagically switches with the 'new' domain. And then, with a bit of care, a singe load-balance pass should see a consistent view (there should not be reloads ofrq->sd-- which will be a bit of an audit I suppose).We need
nr_pref_llcinformation at the runqueue level because the load balancer must identify which specific rq has the largest number of tasks that prefer a given destination LLC. If we move the counter to the LLC’s sd level, we would only know the aggregate number of tasks in the entire LLC that prefer that destination—not which rq they reside on. Without per-rq counts, we would not be able to select the correct source rq to pull tasks from.
Quoted 4 lines
The only way this could work at the LLC-sd level is if all CPUs within the LLC shared a single runqueue, which is not the case today.
Let me know if I understand your comments correctly.
So the sched_domain instances are per-cpu (hence the need for sched_domain_shared). So irrespective of what level you stick them at (I was thinking the bottom most, but it really doesn't matter) they will be per CPU.
Re: [PATCH v2 05/23] sched/cache: Assign preferred LLC ID to processes
In reply to Tim ChenQuoted 58 lines · Tim Chen
With cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar to
numa_preferred_nidin NUMA balancing.Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: Align preferred LLC with NUMA balancing's preferred node.include/linux/sched.h | 1 + init/init_task.c | 3 +++ kernel/sched/fair.c | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+)include/linux/sched.h +1
-0@@ -1408,6 +1408,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; +int preferred_llc;#endif #ifdef CONFIG_RSEQ init/init_task.c +3
-0@@ -191,6 +191,9 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .numa_group = NULL, .numa_faults = NULL,#endif +#ifdef CONFIG_SCHED_CACHE + .preferred_llc = -1, +#endif #if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS ) .kasan_depth = 1,#endif kernel/sched/fair.c +1
-0@@ -1300,6 +1300,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched;unsigned long epoch; +int mm_sched_llc =-1 ;if (!sched_cache_enabled ())return ;@@ -1330,6 +1331,23 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; }
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1343
Quoted 10 lines
kernel/sched/fair.c +10
-0+ +if (mm->mm_sched_cpu !=-1 ) { + mm_sched_llc =llc_id (mm->mm_sched_cpu); + +#ifdef CONFIG_NUMA_BALANCING +/* +* Don't assign preferred LLC if it +* conflicts with NUMA balancing. +*/ + if (p->numa_preferred_nid >= 0 &&
I wonder if the restriction here shouldn't be so strict. In Mel Gorman's patch (e496132ebedd sched/fair: Adjust the allowed NUMA imbalance when SD_NUMA spans multiple LLCs), the value of the 'imb_numa_nr' is checked to determine if |SD_NUMA| imbalance is allowed. Could we use this same check to decide whether or not to perform a cross-numa migration?
Quoted 10 lines
kernel/sched/fair.c +7
-0+ cpu_to_node(mm->mm_sched_cpu) != p->numa_preferred_nid) + mm_sched_llc =-1 ; +#endif + } + +if (p->preferred_llc != mm_sched_llc) + p->preferred_llc = mm_sched_llc; }static void task_tick_cache (struct rq *rq,struct task_struct *p)
Re: [PATCH v2 07/23] sched/cache: Introduce per runqueue task LLC preference counter
In reply to Peter ZijlstraQuoted 20 lines · Peter Zijlstra
Quoted 19 lines · Tim Chen
Quoted 7 lines · Peter Zijlstra
On Wed, Dec 03, 2025 at 03:07:26PM -0800, Tim Chen wrote:
Would it perhaps be easier to stick this thing in
rq->sdrather than inrq->nr_pref_llc. That way it automagically switches with the 'new' domain. And then, with a bit of care, a singe load-balance pass should see a consistent view (there should not be reloads ofrq->sd-- which will be a bit of an audit I suppose).We need
nr_pref_llcinformation at the runqueue level because the load balancer must identify which specific rq has the largest number of tasks that prefer a given destination LLC. If we move the counter to the LLC’s sd level, we would only know the aggregate number of tasks in the entire LLC that prefer that destination—not which rq they reside on. Without per-rq counts, we would not be able to select the correct source rq to pull tasks from.The only way this could work at the LLC-sd level is if all CPUs within the LLC shared a single runqueue, which is not the case today.
Quoted 7 lines
Quoted 1 line
Let me know if I understand your comments correctly.
So the
sched_domaininstances are per-cpu (hence the need forsched_domain_shared). So irrespective of what level you stick them at (I was thinking the bottom most, but it really doesn't matter) they will be per CPU.
Okay, I see what you're saying. Will update code accordingly.
Tim
Re: [PATCH v2 05/23] sched/cache: Assign preferred LLC ID to processes
In reply to Vern HaoQuoted 67 lines · Vern Hao
Quoted 66 lines · Tim Chen
With cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar to
numa_preferred_nidin NUMA balancing.Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: Align preferred LLC with NUMA balancing's preferred node.include/linux/sched.h | 1 + init/init_task.c | 3 +++ kernel/sched/fair.c | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+)include/linux/sched.h +1
-0@@ -1408,6 +1408,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; +int preferred_llc;#endif #ifdef CONFIG_RSEQ init/init_task.c +3
-0@@ -191,6 +191,9 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .numa_group = NULL, .numa_faults = NULL,#endif +#ifdef CONFIG_SCHED_CACHE + .preferred_llc = -1, +#endif #if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS ) .kasan_depth = 1,#endif kernel/sched/fair.c +9
-0@@ -1300,6 +1300,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched;unsigned long epoch; +int mm_sched_llc =-1 ;if (!sched_cache_enabled ())return ;@@ -1330,6 +1331,23 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; } + +if (mm->mm_sched_cpu !=-1 ) { + mm_sched_llc =llc_id (mm->mm_sched_cpu); + +#ifdef CONFIG_NUMA_BALANCING +/* +* Don't assign preferred LLC if it +* conflicts with NUMA balancing.
Quoted 8 lines
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1343
Quoted 2 lines
kernel/sched/fair.c +2
-0+*/ + if (p->numa_preferred_nid >= 0 &&I wonder if the restriction here shouldn't be so strict. In Mel Gorman's patch (
e496132ebeddsched/fair: Adjust the allowed NUMA imbalance whenSD_NUMAspans multiple LLCs), the value of the 'imb_numa_nr' is checked to determine if |SD_NUMA| imbalance is allowed. Could we use this same check to decide whether or not to perform a cross-numa migration?
If we set the preferred LLC that's in a different node other than the preferred node, the preferred LLC is going to fight with NUMA balancing and bounce tasks back and forth between nodes. NUMA locality is going to affect performance more so we'll let NUMA preference take precedence.
Tim
Quoted 11 lines
Quoted 11 lines
kernel/sched/fair.c +7
-0+ cpu_to_node(mm->mm_sched_cpu) != p->numa_preferred_nid) + mm_sched_llc =-1 ; +#endif + } + +if (p->preferred_llc != mm_sched_llc) + p->preferred_llc = mm_sched_llc; }static void task_tick_cache (struct rq *rq,struct task_struct *p)
Re: [PATCH v2 04/23] sched/cache: Make LLC id continuous
In reply to Peter ZijlstraQuoted 86 lines · Peter Zijlstra
Quoted 65 lines · Tim Chen
kernel/sched/fair.c +46
-0@@ -1210,10 +1210,17 @@ __read_mostly unsigned int llc_imb_pct = 20;static int llc_id (int cpu) { +int llc; +if (cpu <0 )return -1 ; + llc =per_cpu (sd_llc_id, cpu); +/* avoid race with cpu hotplug */ +if (unlikely (llc >= max_llcs)) +return -1 ; + +return llc; }void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)@@ -668,6 +670,55 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_cluster_active); +/* +* Assign continuous llc id for the CPU, and return +* the assigned llc id. +*/ +static int update_llc_id (struct sched_domain *sd, +int cpu) +{ +int id =per_cpu (sd_llc_id, cpu), i; + +if (id >=0 ) +return id; + +if (sd) { +/* Look for any assigned id and reuse it.*/ +for_each_cpu (i,sched_domain_span (sd)) { + id =per_cpu (sd_llc_id, i); + +if (id >=0 ) { +per_cpu (sd_llc_id, cpu) = id; +return id; + } + } + } + +/* +* When 1. there is no id assigned to this LLC domain, +* or 2. the sd is NULL, we reach here. +* Consider the following scenario, +* CPU0~CPU95 are in the node0, CPU96~CPU191 are +* in the node1. During bootup, maxcpus=96 is +* appended. +* case 1: When running cpu_attach_domain(CPU24) +* during boot up, CPU24 is the first CPU in its +* non-NULL LLC domain. However, +* its corresponding llc id has not been assigned yet. +* +* case 2: After boot up, the CPU100 is brought up +* via sysfs manually. As a result, CPU100 has only a kernel/sched/fair.c DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); · line 720
Quoted 10 lines
kernel/sched/fair.c +10
-0+* Numa domain attached, because CPU100 is the only CPU +* of a sched domain, all its bottom domains are degenerated. +* The LLC domain pointer sd is NULL for CPU100. +* +* For both cases, we want to increase the number of LLCs. +*/ +per_cpu (sd_llc_id, cpu) = max_llcs++; + +return per_cpu (sd_llc_id, cpu); +}I'm not sure I follow. So
partition_sched_domains()first callsdetach_destroy_domains()on the old set, and thenbuild_sched_domains()on the new set.Do
detach_destroy_domain()will do:cpu_attach_domain(NULL,..);
Quoted 8 lines
That is, it will explicitly attach the NULL
sched_domainto a CPU. At which point I feelupdate_llc_id()should be returning -1, no?Then later,
build_sched_domains()will set a !NULLsched_domain, at which pointupdate_llc_id()can set a real value.This should then also get rid of that weird
max_llcscheck inllc_id(), right?
Thanks for pointing this out. Yes, we should take care of the attachment of NULL sd. Will update the code accordingly.
Tim
Re: [PATCH v2 06/23] sched/cache: Track LLC-preferred tasks per runqueue
In reply to Peter ZijlstraQuoted 30 lines · Peter Zijlstra
Quoted 19 lines · Tim Chen
Quoted 6 lines
Quoted 6 lines
kernel/sched/fair.c +6
-0+static void account_llc_dequeue (struct rq *rq,struct task_struct *p) +{ +int pref_llc; + +/* +* Borrow the uc_se->active from uclamp_rq_inc_id(), Quoted 8 lines
kernel/sched/fair.c static int llc_id(int cpu) · line 1252
Quoted 5 lines
kernel/sched/fair.c +5
-0+* uclamp_rq_dec_id() to avoid the unbalanced calculation +* of rq statistics. +*/ +if (unlikely (!p->sched_llc_active)) +return ;Another very confusing comment; what? Also, can you please explain (in the new comment) how we get here without having
llc_activeset?The comment meant to say that we are using a similar mechanism as accounting done in
uc_se->activefromuclamp_rq_inc_id(). I agree that it confuses more than making things clearer.Quoted 8 lines
How about the following comment to make things clearer:
/*
* Cache aware scheduling was active when the task was enqueued.
* Admin has disabled cache aware scheduling before task was dequeued
* but the accounting has to be kept straight in case cache aware scheduling
* is re-enabled.
*/
Quoted 4 lines
Is having that
sched_cache_enabled()test worth it? account_numa_{en,de}queue() don't seem to have any of this.
OK, I think we can remove the sched_cache_enabled() check and make the account_llc_{en,de}queue() depending on CONFIG_SCHED_CACHE, so the sched_llc_active can be removed.
thanks,
Chenyu
Re: [PATCH v2 12/23] sched/cache: Add migrate_llc_task migration type for cache-aware balancing
In reply to Peter ZijlstraQuoted 46 lines · Peter Zijlstra
Quoted 45 lines · Tim Chen
kernel/sched/fair.c +16
-1@@ -9547,7 +9547,8 @@ enum migration_type { migrate_load =0 , migrate_util, migrate_task, - migrate_misfit + migrate_misfit, + migrate_llc_task };#define LBF_ALL_PINNED 0x01@@ -10134,6 +10135,10 @@ static int detach_tasks(struct lb_env *env) env->imbalance -= util;break ; +case migrate_llc_task: + env->imbalance--; +break ; +case migrate_task: env->imbalance--;break ;@@ -12181,6 +12199,16 @@ static struct rq *sched_balance_find_src_rq(struct lb_env *env, }break ; +case migrate_llc_task: +#ifdef CONFIG_SCHED_CACHE + dst_llc =llc_id (env->dst_cpu); +if (dst_llc >=0 && + busiest_pref_llc < rq->nr_pref_llc[dst_llc]) { + busiest_pref_llc = rq->nr_pref_llc[dst_llc]; + busiest = rq; + } +#endif +break ;case migrate_task:if (busiest_nr < nr_running) { busiest_nr = nr_running;@@ -12363,6 +12391,8 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sdcase migrate_misfit:
Quoted 8 lines
kernel/sched/fair.c static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sd · line 12397
Quoted 6 lines
kernel/sched/fair.c +2
-0__schedstat_add (sd->lb_imbalance_misfit[idle], env->imbalance);break ; +case migrate_llc_task: +break ; } }The enum and all switch statements had the same order; you wrecked it!
OK, will fix the order.
Thanks,
Chenyu
Re: [PATCH v2 04/23] sched/cache: Make LLC id continuous
In reply to Tim ChenQuoted 87 lines · Tim Chen
Quoted 86 lines · Peter Zijlstra
Quoted 65 lines · Tim Chen
kernel/sched/fair.c +46
-0@@ -1210,10 +1210,17 @@ __read_mostly unsigned int llc_imb_pct = 20;static int llc_id (int cpu) { +int llc; +if (cpu <0 )return -1 ; + llc =per_cpu (sd_llc_id, cpu); +/* avoid race with cpu hotplug */ +if (unlikely (llc >= max_llcs)) +return -1 ; + +return llc; }void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)@@ -668,6 +670,55 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_cluster_active); +/* +* Assign continuous llc id for the CPU, and return +* the assigned llc id. +*/ +static int update_llc_id (struct sched_domain *sd, +int cpu) +{ +int id =per_cpu (sd_llc_id, cpu), i; + +if (id >=0 ) +return id; + +if (sd) { +/* Look for any assigned id and reuse it.*/ +for_each_cpu (i,sched_domain_span (sd)) { + id =per_cpu (sd_llc_id, i); + +if (id >=0 ) { +per_cpu (sd_llc_id, cpu) = id; +return id; + } + } + } + +/* +* When 1. there is no id assigned to this LLC domain, +* or 2. the sd is NULL, we reach here. +* Consider the following scenario, +* CPU0~CPU95 are in the node0, CPU96~CPU191 are +* in the node1. During bootup, maxcpus=96 is +* appended. +* case 1: When running cpu_attach_domain(CPU24) +* during boot up, CPU24 is the first CPU in its +* non-NULL LLC domain. However, +* its corresponding llc id has not been assigned yet. +* +* case 2: After boot up, the CPU100 is brought up +* via sysfs manually. As a result, CPU100 has only a kernel/sched/fair.c DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); · line 720
Quoted 10 lines
kernel/sched/fair.c +10
-0+* Numa domain attached, because CPU100 is the only CPU +* of a sched domain, all its bottom domains are degenerated. +* The LLC domain pointer sd is NULL for CPU100. +* +* For both cases, we want to increase the number of LLCs. +*/ +per_cpu (sd_llc_id, cpu) = max_llcs++; + +return per_cpu (sd_llc_id, cpu); +}I'm not sure I follow. So
partition_sched_domains()first callsdetach_destroy_domains()on the old set, and thenbuild_sched_domains()on the new set.Do
detach_destroy_domain()will do:cpu_attach_domain(NULL,..);
Quoted 8 lines
Quoted 8 lines
That is, it will explicitly attach the NULL
sched_domainto a CPU. At which point I feelupdate_llc_id()should be returning -1, no?Then later,
build_sched_domains()will set a !NULLsched_domain, at which pointupdate_llc_id()can set a real value.This should then also get rid of that weird
max_llcscheck inllc_id(), right?
The check for max_llcs was intended to prevent out-of-bounds access to rq->nr_pref_llc[] at multiple points in the code.
Since dst_llc = llc_id(env->dst_cpu); — and while the LLC ID for the
CPU is updated in update_llc_id(), this update occurs before we reallocate the nr_pref_llc buffer — dst_llc may end up exceeding the bounds of the
original nr_pref_llc buffer.
For this reason, we added a check if (dst_llc > max_llc) in llc_id() when attempting to access rq->nr_pref_llc[dst_llc].
However, I agree that the max_llc check seems to not properly integrated into the current patch: it should instead be placed in the 7th patch, as this would better illustrate the rationale for the max_llc check here:
sched/cache: Introduce per runqueue task LLC preference counter
In the 7th patch, we actually increment new_max_llcs rather than max_llcs — meaning max_llcs always represents the "old" number of LLCs. As a result, there is a race window between extending the rq->nr_pref_llc buffer and updating max_llcs.
@@ -714,7 +827,7 @@ static int update_llc_id(struct sched_domain *sd,
*
* For both cases, we want to increase the number of LLCs. */
- per_cpu(sd_llc_id, cpu) = max_llcs++;
+ per_cpu(sd_llc_id, cpu) = new_max_llcs++;
return per_cpu(sd_llc_id, cpu);
}
Quoted 3 lines
Thanks for pointing this out. Yes, we should take care of the attachment of NULL sd. Will update the code accordingly.
My understanding is that, if the sd is NULL, it is either because invoked by detach_destroy_domain() for the old set, or by case 2 mentioned in above comments:
Say, CPU0-CPU95 are online during bootup, the boot command line is maxcpus=96. Later after bootup, the user wants to bring up CPU100, the LLC domain for CPU100 is NULL in this case(due to sd generation), and a new LLC should be detected.
That is to say, when we reach update_llc_id(), there could be 2 reasons for NULL sd. For the detach_destroy_domain() case, update_llc_id() should return a valid id without increasing the max_llcs, because of
if (id >= 0)
return id;And for the latter, the max_llcs should be increased. Let me double check on this.
thanks,
Chenyu
Quoted 1 line
Tim
Re: [PATCH v2 01/23] sched/cache: Introduce infrastructure for cache-aware load balancing
In reply to Vern HaoQuoted 86 lines · Vern Hao
Hi, Peter, Chen Yu and Tim:
Quoted 74 lines · Tim Chen
From: "Peter Zijlstra (Intel)" <[email protected]>
Adds infrastructure to enable cache-aware load balancing, which improves cache locality by grouping tasks that share resources within the same cache domain. This reduces cache misses and improves overall data access efficiency.
In this initial implementation, threads belonging to the same process are treated as entities that likely share working sets. The mechanism tracks per-process CPU occupancy across cache domains and attempts to migrate threads toward cache-hot domains where their process already has active threads, thereby enhancing locality.
This provides a basic model for cache affinity. While the current code targets the last-level cache (LLC), the approach could be extended to other domain types such as clusters (L2) or node-internal groupings.
At present, the mechanism selects the CPU within an LLC that has the highest recent runtime. Subsequent patches in this series will use this information in the load-balancing path to guide task placement toward preferred LLCs.
In the future, more advanced policies could be integrated through NUMA balancing-for example, migrating a task to its preferred LLC when spare capacity exists, or swapping tasks across LLCs to improve cache affinity. Grouping of tasks could also be generalized from that of a process to be that of a NUMA group, or be user configurable.
Originally-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Restore the original CPU scan to cover all online CPUs, rather than scanning within the preferred NUMA node. (Peter Zijlstra)
Userq->currinstead ofrq->donor. (K Prateek Nayak) Minor fix intask_tick_cache()to use
if (mm->mm_sched_epoch>=rq->cpu_epoch)
to avoidmm_sched_epochgoing backwards.
include/linux/mm_types.h| 44 +++++++ include/linux/sched.h | 11 ++
init/Kconfig | 11 ++
kernel/fork.c | 6 +
kernel/sched/core.c | 6 +
kernel/sched/fair.c | 258 +++++++++++++++++++++++++++++++++++++++ kernel/sched/sched.h | 8 ++7 files changed, 344 insertions(+)include/linux/mm_types.h +8
-0@@ -939,6 +939,11 @@ typedef struct { DECLARE_BITMAP(__mm_flags, NUM_MM_FLAG_BITS); } __private mm_flags_t; +struct mm_sched { + u64 runtime; + unsigned long epoch; +}; + struct kioctx_table; struct iommu_mm_data; struct mm_struct {@@ -1029,6 +1034,17 @@ struct mm_struct {*/ raw_spinlock_t cpus_allowed_lock; #endif +#ifdef CONFIG_SCHED_CACHE + /* +* Track per-cpu-per-process occupancy as a proxy for cache Quoted 7 lines
residency.
+ * Seeaccount_mm_sched()and ...
+ */
+ structmm_sched__percpu*pcpu_sched;
+raw_spinlock_tmm_sched_lock;
+ unsigned longmm_sched_epoch;
+ intmm_sched_cpu;As we discussed earlier,I continue to believe that dedicating '
mm_sched_cpu' to handle the aggregated hotspots of all threads is inappropriate, as the multiple threads lack a necessary correlation in our real application.
Quoted 2 lines
So, I was wondering if we could put this variable into struct
task_struct, That allows us to better monitor the hotspot CPU of each thread, despite some details needing consideration.
I suppose you are suggesting a fine-grained control for a set of tasks. Process-scope aggregation could be a start as the default strategy( conservative, benefit multi-thread workloads that share data per process, not introduce regression).
On top of that, I wonder if we could provide task-scope control like sched_setattr(), similar to core-scheduling cookie mechanism, for users that want aggressive aggregation. But before doing that, we need a mechanism that that leverages a monitor system(like PMU) to figure out if putting these tasks together would bring benefit(if I understand Steven's suggestion correctly on LPC), or detection tasks that share resource, then maybe leverage QOS interfaces to enable the cache-aware aggregation(something Qias mentioned on the LPC).
thanks,
Chenyu
Re: [PATCH v2 15/23] sched/cache: Respect LLC preference in task migration and detach
In reply to Peter ZijlstraQuoted 18 lines · Peter Zijlstra
kernel/sched/fair.c int can_migrate_task(struct task_struct *p, struct lb_env *env) · line 10033
Quoted 10 lines · Tim Chen
kernel/sched/fair.c +6
-0@@ -10025,6 +10025,13 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)if (env->flags &LBF_ACTIVE_LB )return 1 ; +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled () && +can_migrate_llc_task (env->src_cpu, env->dst_cpu, p) == mig_forbid && + !task_has_sched_core (p)) +return 0 ; +#endif This seems wrong:
- it does not letnr_balance_failedoverride things;
- it takes precedence overmigrate_degrade_locality(); you really want to migrate towards the preferred NUMA node over staying on your LLC.
Quoted 3 lines
That is, this really wants to be done after
migrate_degrades_locality()and only if degrades == 0 or something.
OK, will fix it.
Quoted 49 lines
Quoted 6 lines
kernel/sched/fair.c +0
-0degrades =migrate_degrades_locality (p, env);if (!degrades) hot =task_hot (p, env);@@ -10146,12 +10153,55 @@ static struct list_headlist_splice (&pref_old_llc, tasks);return tasks;kernel/sched/fair.c static struct list_head · line 10164
Quoted 10 lines
kernel/sched/fair.c +9
-0} + +static bool stop_migrate_src_rq (struct task_struct *p, +struct lb_env *env, +int detached) +{ +if (!sched_cache_enabled () || p->preferred_llc ==-1 || +cpus_share_cache (env->src_cpu, env->dst_cpu) || + env->sd->nr_balance_failed) +return false;But you are allowing
nr_balance_failedto override things here.Quoted 16 lines
kernel/sched/fair.c +16
-0+/* +* Stop migration for the src_rq and pull from a +* different busy runqueue in the following cases: +* +* 1. Trying to migrate task to its preferred +* LLC, but the chosen task does not prefer dest +* LLC - case 3 in order_tasks_by_llc(). This violates +* the goal of migrate_llc_task. However, we should +* stop detaching only if some tasks have been detached +* and the imbalance has been mitigated. +* +* 2. Don't detach more tasks if the remaining tasks want +* to stay. We know the remaining tasks all prefer the +* current LLC, because after order_tasks_by_llc(), the +* tasks that prefer the current LLC are the least favored +* candidates to be migrated out. kernel/sched/fair.c static struct list_head · line 10191
Quoted 10 lines
kernel/sched/fair.c +10
-0+*/ +if (env->migration_type == migrate_llc_task && + detached &&llc_id (env->dst_cpu) != p->preferred_llc) +return true; + +if (llc_id (env->src_cpu) == p->preferred_llc) +return true; + +return false; +}Also, I think we have a problem with
nr_balance_failed,cache_nice_triesis 1 forSHARE_LLC; this means for failed=0 we ignore:
Quoted 7 lines
- ineligible tasks
- llc fail
- node-degrading / hotand then the very next round, we do all of them at once, without much grading.
Do you mean we can set different thresholds for the different scenarios you mentioned above, so as to avoid migrating tasks at the same time in detach_tasks()?
For example,
ineligible tasks check:
if (env->sd->nr_balance_failed > env->sd->cache_nice_tries)
can_migrate;
llc fail check:
if (env->sd->nr_balance_failed > env->sd->cache_nice_tries + 1)
can_migrate;
node-degrading/hot check:
if (env->sd->nr_balance_failed > env->sd->cache_nice_tries + 2)
can_migrate;
Quoted 4 lines
Quoted 4 lines
kernel/sched/fair.c +0
-0@@ -10205,6 +10255,15 @@ static int detach_tasks(struct lb_env *env) p = list_last_entry(tasks, struct task_struct, se.group_node);
Quoted 9 lines
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10262
Quoted 5 lines
kernel/sched/fair.c +5
-0+/* +* Check if detaching current src_rq should be stopped, because +* doing so would break cache aware load balance. If we stop +* here, the env->flags has LBF_ALL_PINNED, which would cause +* the load balance to pull from another busy runqueue. Uhh,
can_migrate_task()will clear thatALL_PINNEDthing if we've found at least one task before getting here.
One problem is that, LBF_ALL_PINNED was cleared before migrate_degrades_locality()/can_migrate_llc_task() in detach_tasks(). I suppose we want to keep LBF_ALL_PINNED() if can_migrate_llc_task(break llc locality) failed.
Quoted 7 lines
kernel/sched/fair.c static int detach_tasks(struct lb_env *env) · line 10265
Quoted 3 lines
kernel/sched/fair.c +3
-0+*/ +if (stop_migrate_src_rq (p, env, detached)) +break ;Perhaps split
cfs_tasksinto multiple lists from the get-go? That avoids this sorting.
Will check with Tim on this.
thanks,
Chenyu
Re: [PATCH v2 16/23] sched/cache: Introduce sched_cache_present to enable cache aware scheduling for multi LLCs NUMA node
In reply to Peter ZijlstraQuoted 38 lines · Peter Zijlstra
Quoted 27 lines · Tim Chen
From: Chen Yu <[email protected]>
Cache-aware load balancing should only be enabled if there are more than 1 LLCs within 1 NUMA node.
sched_cache_presentis introduced to indicate whether this platform supports this topology.Suggested-by Libo Chen <[email protected]>Suggested-by Adam Li <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Use flagsched_cache_presentto indicate whether a platform supports cache aware scheduling. Change this flag from staic key. There should be only 1 static key to control the cache aware scheduling. (Peter Zijlstra)kernel/sched/topology.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)kernel/sched/topology.c +0
-0kernel/sched/topology.c int max_llcs; · line 27
Quoted 4 lines
kernel/sched/topology.c +1
-0@@ -24,6 +24,8 @@ int max_llcs;#ifdef CONFIG_SCHED_CACHE +static bool sched_cache_present;
sched_energy_presentsched_asym_cpucapacitysched_cluster_activesched_smt_present
Quoted 2 lines
are all static keys tied to the current topology, why break the streak and make this a boolean?
OK, will convert it into a key.
thanks,
Chenyu
Re: [PATCH v2 16/23] sched/cache: Introduce sched_cache_present to enable cache aware scheduling for multi LLCs NUMA node
In reply to Peter ZijlstraQuoted 39 lines · Peter Zijlstra
Quoted 38 lines · Peter Zijlstra
Quoted 27 lines · Tim Chen
From: Chen Yu <[email protected]>
Cache-aware load balancing should only be enabled if there are more than 1 LLCs within 1 NUMA node.
sched_cache_presentis introduced to indicate whether this platform supports this topology.Suggested-by Libo Chen <[email protected]>Suggested-by Adam Li <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Use flagsched_cache_presentto indicate whether a platform supports cache aware scheduling. Change this flag from staic key. There should be only 1 static key to control the cache aware scheduling. (Peter Zijlstra)kernel/sched/topology.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-)kernel/sched/topology.c +0
-0kernel/sched/topology.c int max_llcs; · line 27
Quoted 4 lines
kernel/sched/topology.c +1
-0@@ -24,6 +24,8 @@ int max_llcs;#ifdef CONFIG_SCHED_CACHE +static bool sched_cache_present;
sched_energy_presentsched_asym_cpucapacitysched_cluster_activesched_smt_present
Quoted 5 lines
Quoted 2 lines
are all static keys tied to the current topology, why break the streak and make this a boolean?
Also, patch doesn't use
sched_cache_presentat all, so perhaps just drop it on the floor entirely?
The sched_cache_present flag is used in a subsequent patch:
"[20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling" This flag is used to check whether the user is eligible to enable cache-aware scheduling. I will try to place the declaration and usage of this flag together.
thanks,
Chenyu
Re: [PATCH v2 17/23] sched/cache: Record the number of active threads per process for cache-aware scheduling
In reply to Peter ZijlstraQuoted 43 lines · Peter Zijlstra
Quoted 32 lines · Tim Chen
From: Chen Yu <[email protected]>
A performance regression was observed by Prateek when running hackbench with many threads per process (high fd count). To avoid this, processes with a large number of active threads are excluded from cache-aware scheduling.
With
sched_cacheenabled, record the number of active threads in each process during the periodictask_cache_work(). While iterating over CPUs, if the currently running task belongs to the same process as the task that launchedtask_cache_work(), increment the active thread count.This number will be used by subsequent patch to inhibit cache aware load balance.
Suggested-by K Prateek Nayak <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: No change.include/linux/mm_types.h | 1 + kernel/sched/fair.c | 11 +++++++++-- 2 files changed, 10 insertions(+), 2 deletions(-)include/linux/mm_types.h +0
-0include/linux/mm_types.h struct mm_struct { · line 1046
Quoted 4 lines
include/linux/mm_types.h +1
-0@@ -1043,6 +1043,7 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; +u64 nr_running_avg____cacheline_aligned_in_smp ;This is unlikely to do what you hope it does, it will place this variable on a new cacheline, but will not ensure this variable is the only one in that line. Notably
ogtables_bytes(the next field in this structure) will share the line.
Quoted 3 lines
It might all be less dodgy if you stick these here fields in their own structure, a little like
mm_mm_cidor so.
Got it, will do.
Quoted 20 lines
Quoted 20 lines
include/linux/mm_types.h +0
-0#endif #ifdef CONFIG_MMU kernel/sched/fair.c +2
-2@@ -1421,11 +1421,11 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p) static void __no_profile task_cache_work(struct callback_head *work) { -struct task_struct *p = current; +struct task_struct *p = current, *cur;struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ;unsigned long curr_m_a_occ =0 ; -int cpu, m_a_cpu =-1 ; +int cpu, m_a_cpu =-1 , nr_running =0 ;cpumask_var_t cpus;
Quoted 8 lines
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1460
Quoted 5 lines
kernel/sched/fair.c +0
-0WARN_ON_ONCE (work != &p->cache_work);@@ -1458,6 +1458,12 @@ static void __no_profile task_cache_work(struct callback_head *work) m_occ = occ; m_cpu = i; }guard(rcu)();
OK.
Quoted 13 lines
Quoted 13 lines
kernel/sched/fair.c +6
-0+rcu_read_lock (); + cur =rcu_dereference (cpu_rq (i)->curr); +if (cur && !(cur->flags & (PF_EXITING |PF_KTHREAD )) && + cur->mm == mm) + nr_running++; +rcu_read_unlock (); }/* @@ -1501,6 +1507,7 @@ static void __no_profile task_cache_work(struct callback_head *work) mm->mm_sched_cpu = m_a_cpu; }
Quoted 6 lines
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1512
Quoted 3 lines
kernel/sched/fair.c +1
-0+update_avg (&mm->nr_running_avg, nr_running);free_cpumask_var (cpus); }Its a wee bit weird to introduce
nr_running_avgwithout its user. Makes it hard to see what's what.
OK, will put the user together.
thanks,
Chenyu
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Peter ZijlstraQuoted 52 lines · Peter Zijlstra
Quoted 37 lines · Tim Chen
include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +12
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \kernel/sched/debug.c static const struct file_operations sched_feat_fops = { · line 193
Quoted 10 lines
kernel/sched/debug.c +8
-0+ buf[cnt] ='\0' ; \ +if (kstrtouint (buf,10 , &val)) \ +return -EINVAL ; \ +if (val > (max)) \ +return -EINVAL ; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \Oh gawd :-(
Quoted 3 lines
Please just write out all the various write methods and use
kstrtoul_from_user()andkstrtobool_from_user()where applicable.
OK, will do.
thanks,
Chenyu
Re: [PATCH v2 04/23] sched/cache: Make LLC id continuous
In reply to Chen, Yu CQuoted 142 lines · Yu C
Quoted 87 lines · Tim Chen
Quoted 86 lines · Peter Zijlstra
Quoted 65 lines · Tim Chen
kernel/sched/fair.c +46
-0@@ -1210,10 +1210,17 @@ __read_mostly unsigned int llc_imb_pct = 20;static int llc_id (int cpu) { +int llc; +if (cpu <0 )return -1 ; + llc =per_cpu (sd_llc_id, cpu); +/* avoid race with cpu hotplug */ +if (unlikely (llc >= max_llcs)) +return -1 ; + +return llc; }void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)@@ -668,6 +670,55 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_cluster_active); +/* +* Assign continuous llc id for the CPU, and return +* the assigned llc id. +*/ +static int update_llc_id (struct sched_domain *sd, +int cpu) +{ +int id =per_cpu (sd_llc_id, cpu), i; + +if (id >=0 ) +return id; + +if (sd) { +/* Look for any assigned id and reuse it.*/ +for_each_cpu (i,sched_domain_span (sd)) { + id =per_cpu (sd_llc_id, i); + +if (id >=0 ) { +per_cpu (sd_llc_id, cpu) = id; +return id; + } + } + } + +/* +* When 1. there is no id assigned to this LLC domain, +* or 2. the sd is NULL, we reach here. +* Consider the following scenario, +* CPU0~CPU95 are in the node0, CPU96~CPU191 are +* in the node1. During bootup, maxcpus=96 is +* appended. +* case 1: When running cpu_attach_domain(CPU24) +* during boot up, CPU24 is the first CPU in its +* non-NULL LLC domain. However, +* its corresponding llc id has not been assigned yet. +* +* case 2: After boot up, the CPU100 is brought up +* via sysfs manually. As a result, CPU100 has only a kernel/sched/fair.c DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); · line 720
Quoted 10 lines
kernel/sched/fair.c +10
-0+* Numa domain attached, because CPU100 is the only CPU +* of a sched domain, all its bottom domains are degenerated. +* The LLC domain pointer sd is NULL for CPU100. +* +* For both cases, we want to increase the number of LLCs. +*/ +per_cpu (sd_llc_id, cpu) = max_llcs++; + +return per_cpu (sd_llc_id, cpu); +}I'm not sure I follow. So
partition_sched_domains()first callsdetach_destroy_domains()on the old set, and thenbuild_sched_domains()on the new set.Do
detach_destroy_domain()will do:cpu_attach_domain(NULL,..);Quoted 8 lines
Quoted 8 lines
That is, it will explicitly attach the NULL
sched_domainto a CPU. At which point I feelupdate_llc_id()should be returning -1, no?Then later,
build_sched_domains()will set a !NULLsched_domain, at which pointupdate_llc_id()can set a real value.This should then also get rid of that weird
max_llcscheck inllc_id(), right?The check for
max_llcswas intended to prevent out-of-bounds access to rq->nr_pref_llc[] at multiple points in the code.
Sincedst_llc=llc_id(env->dst_cpu); — and while the LLC ID for the
CPU is updated inupdate_llc_id(), this update occurs before we reallocate thenr_pref_llcbuffer —dst_llcmay end up exceeding the bounds of the
originalnr_pref_llcbuffer.For this reason, we added a check if (
dst_llc>max_llc) inllc_id()when attempting to access rq->nr_pref_llc[dst_llc].However, I agree that the
max_llccheck seems to not properly integrated into the current patch: it should instead be placed in the 7th patch, as this would better illustrate the rationale for themax_llccheck here:
sched/cache: Introduce per runqueue task LLC preference counterIn the 7th patch, we actually increment
new_max_llcsrather thanmax_llcs— meaningmax_llcsalways represents the "old" number of LLCs. As a result, there is a race window between extending therq->nr_pref_llcbuffer and updatingmax_llcs.(quoted diff) +1
-1@@ -714,7 +827,7 @@ static int update_llc_id(struct sched_domain *sd, * * For both cases, we want to increase the number of LLCs. */ - per_cpu(sd_llc_id, cpu) = max_llcs++; + per_cpu(sd_llc_id, cpu) = new_max_llcs++; return per_cpu(sd_llc_id, cpu); }Quoted 5 lines
Thanks for pointing this out. Yes, we should take care of the attachment of NULL sd. Will update the code accordingly.
My understanding is that, if the sd is NULL, it is either because invoked by
detach_destroy_domain()for the old set, or by case 2 mentioned in above comments:
Say, CPU0-CPU95 are online during bootup, the boot command line is maxcpus=96.
Later after bootup, the user wants to bring up CPU100, the LLC domain for CPU100 is NULL in this case(due to sd generation), and a new LLC should be detected.
Quoted 7 lines
That is to say, when we reach
update_llc_id(), there could be 2 reasons for NULL sd. For thedetach_destroy_domain()case,update_llc_id()should return a valid id without increasing themax_llcs, because ofif (id >= 0) return id;And for the latter, the
max_llcsshould be increased. Let me double check on this.
The issue is we could offline all CPUs in a LLC and online them later. In the current code, we will assign their ids all to -1. So on attach of CPUs again, we'll be assigning a new LLC. I think the proper thing to do is not to assign llc id of the offlined cpu (the case where sd == NULL) and keep the original llc id assigned. Then we should be okay and not increase max_llcs.
Tim
Quoted 6 lines
thanks,
ChenyuQuoted 1 line
Tim
Re: [PATCH v2 01/23] sched/cache: Introduce infrastructure for cache-aware load balancing
In reply to a message not in the archiveQuoted 5 lines · Vern Hao
snip
Quoted 68 lines
Quoted 4 lines
include/linux/mm_types.h +4
-0+struct mm_sched __percpu *pcpu_sched; +raw_spinlock_t mm_sched_lock; +unsigned long mm_sched_epoch; +int mm_sched_cpu;Quoted 32 lines
Quoted 6 lines
As we discussed earlier,I continue to believe that dedicating '
mm_sched_cpu' to handle the aggregated hotspots of all threads is inappropriate, as the multiple threads lack a necessary correlation in our real application.
So, I was wondering if we could put this variable into structtask_struct, That allows us to better monitor the hotspot CPU of each thread, despite some details needing consideration.
Vern,
The stat is related to the group of threads(tasks) that we will like to group together in an LLC. In current implementation, it is per process and hence the placement in mm struct.
Later, we could change the grouping to other criteria (e.g. cgroup, or numa_group). In that case the stats may be associated with other data struct that represent the group. As I mentioned in the cover letter, that would be another exercise and we'll like to get the basic grouping by process to be merged first before considering other kinds of grouping.
Tim
Quoted 1 line
Re: [PATCH v2 07/23] sched/cache: Introduce per runqueue task LLC preference counter
In reply to Peter ZijlstraQuoted 20 lines · Peter Zijlstra
Quoted 19 lines · Tim Chen
Quoted 7 lines · Peter Zijlstra
On Wed, Dec 03, 2025 at 03:07:26PM -0800, Tim Chen wrote:
Would it perhaps be easier to stick this thing in
rq->sdrather than inrq->nr_pref_llc. That way it automagically switches with the 'new' domain. And then, with a bit of care, a singe load-balance pass should see a consistent view (there should not be reloads ofrq->sd-- which will be a bit of an audit I suppose).We need
nr_pref_llcinformation at the runqueue level because the load balancer must identify which specific rq has the largest number of tasks that prefer a given destination LLC. If we move the counter to the LLC’s sd level, we would only know the aggregate number of tasks in the entire LLC that prefer that destination—not which rq they reside on. Without per-rq counts, we would not be able to select the correct source rq to pull tasks from.The only way this could work at the LLC-sd level is if all CPUs within the LLC shared a single runqueue, which is not the case today.
Quoted 6 lines
Quoted 1 line
Let me know if I understand your comments correctly.
So the
sched_domaininstances are per-cpu (hence the need forsched_domain_shared). So irrespective of what level you stick them at (I was thinking the bottom most, but it really doesn't matter) they will be per CPU.
One side effect of that is when rebuild_sched_domains() got triggered, all rq->sd is getting reallocated. So we'll lose the old LLC preferences till we have time to re-sample process occupancy. I think it is okay as long as the call to rebuild_sched_domains() too frequently. Is this assumption correct?
Tim
Re: [PATCH v2 01/23] sched/cache: Introduce infrastructure for cache-aware load balancing
In reply to Chen, Yu CQuoted 88 lines · Yu C
Quoted 87 lines · Vern Hao
Hi, Peter, Chen Yu and Tim:
Quoted 75 lines · Tim Chen
From: "Peter Zijlstra (Intel)" <[email protected]>
Adds infrastructure to enable cache-aware load balancing, which improves cache locality by grouping tasks that share resources within the same cache domain. This reduces cache misses and improves overall data access efficiency.
In this initial implementation, threads belonging to the same process are treated as entities that likely share working sets. The mechanism tracks per-process CPU occupancy across cache domains and attempts to migrate threads toward cache-hot domains where their process already has active threads, thereby enhancing locality.
This provides a basic model for cache affinity. While the current code targets the last-level cache (LLC), the approach could be extended to other domain types such as clusters (L2) or node-internal groupings.
At present, the mechanism selects the CPU within an LLC that has the highest recent runtime. Subsequent patches in this series will use this information in the load-balancing path to guide task placement toward preferred LLCs.
In the future, more advanced policies could be integrated through NUMA balancing-for example, migrating a task to its preferred LLC when spare capacity exists, or swapping tasks across LLCs to improve cache affinity. Grouping of tasks could also be generalized from that of a process to be that of a NUMA group, or be user configurable.
Originally-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2:
Restore the original CPU scan to cover all online CPUs, rather than scanning within the preferred NUMA node. (Peter Zijlstra)
Userq->currinstead ofrq->donor. (K Prateek Nayak) Minor fix intask_tick_cache()to use
if (mm->mm_sched_epoch>=rq->cpu_epoch)
to avoidmm_sched_epochgoing backwards.
include/linux/mm_types.h| 44 +++++++
include/linux/sched.h | 11 ++
init/Kconfig | 11 ++
kernel/fork.c | 6 +
kernel/sched/core.c | 6 +
kernel/sched/fair.c | 258
+++++++++++++++++++++++++++++++++++++++
kernel/sched/sched.h | 8 ++7 files changed, 344 insertions(+)include/linux/mm_types.h +8
-0@@ -939,6 +939,11 @@ typedef struct { DECLARE_BITMAP(__mm_flags, NUM_MM_FLAG_BITS); } __private mm_flags_t; +struct mm_sched { + u64 runtime; + unsigned long epoch; +}; + struct kioctx_table; struct iommu_mm_data; struct mm_struct {@@ -1029,6 +1034,17 @@ struct mm_struct {*/ raw_spinlock_t cpus_allowed_lock; #endif +#ifdef CONFIG_SCHED_CACHE + /* +* Track per-cpu-per-process occupancy as a proxy for cache Quoted 7 lines
residency.
+ * Seeaccount_mm_sched()and ...
+ */
+ structmm_sched__percpu*pcpu_sched;
+raw_spinlock_tmm_sched_lock;
+ unsigned longmm_sched_epoch;
+ intmm_sched_cpu;As we discussed earlier,I continue to believe that dedicating '
mm_sched_cpu' to handle the aggregated hotspots of all threads is inappropriate, as the multiple threads lack a necessary correlation in our real application.
Quoted 7 lines
Quoted 2 lines
So, I was wondering if we could put this variable into struct
task_struct, That allows us to better monitor the hotspot CPU of each thread, despite some details needing consideration.I suppose you are suggesting a fine-grained control for a set of tasks. Process-scope aggregation could be a start as the default strategy( conservative, benefit multi-thread workloads that share data per process, not introduce regression).
Yes, in our real-world business scenarios at Tencent, I have indeed encountered this issue where multiple threads are divided into several categories to handle different transactions, so they are not share the hot data, the 'mm_sched_cpu' does not represent all of their task, so add a control interface such as cgroup or others will be a good idea.
Quoted 5 lines
On top of that, I wonder if we could provide task-scope control like
sched_setattr(), similar to core-scheduling cookie mechanism, for users that want aggressive aggregation. But before doing that, we need a mechanism that that leverages a monitor system(like PMU) to figure out
There will maybe a trouble, If the environment is running on a VM, We could use tags to differentiate these tasks and do some tests to verify the performance difference between unifying the |mm_sched_cpu| and not unifying.
Quoted 8 lines
if putting these tasks together would bring benefit(if I understand Steven's suggestion correctly on LPC), or detection tasks that share resource, then maybe leverage QOS interfaces to enable the cache-aware aggregation(something Qias mentioned on the LPC).
thanks,
Chenyu
Re: [PATCH v2 04/23] sched/cache: Make LLC id continuous
In reply to Tim ChenQuoted 151 lines · Tim Chen
Quoted 142 lines · Yu C
Quoted 87 lines · Tim Chen
Quoted 86 lines · Peter Zijlstra
Quoted 65 lines · Tim Chen
kernel/sched/fair.c +46
-0@@ -1210,10 +1210,17 @@ __read_mostly unsigned int llc_imb_pct = 20;static int llc_id (int cpu) { +int llc; +if (cpu <0 )return -1 ; + llc =per_cpu (sd_llc_id, cpu); +/* avoid race with cpu hotplug */ +if (unlikely (llc >= max_llcs)) +return -1 ; + +return llc; }void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched)@@ -668,6 +670,55 @@ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_asym_cpucapacity);DEFINE_STATIC_KEY_FALSE (sched_cluster_active); +/* +* Assign continuous llc id for the CPU, and return +* the assigned llc id. +*/ +static int update_llc_id (struct sched_domain *sd, +int cpu) +{ +int id =per_cpu (sd_llc_id, cpu), i; + +if (id >=0 ) +return id; + +if (sd) { +/* Look for any assigned id and reuse it.*/ +for_each_cpu (i,sched_domain_span (sd)) { + id =per_cpu (sd_llc_id, i); + +if (id >=0 ) { +per_cpu (sd_llc_id, cpu) = id; +return id; + } + } + } + +/* +* When 1. there is no id assigned to this LLC domain, +* or 2. the sd is NULL, we reach here. +* Consider the following scenario, +* CPU0~CPU95 are in the node0, CPU96~CPU191 are +* in the node1. During bootup, maxcpus=96 is +* appended. +* case 1: When running cpu_attach_domain(CPU24) +* during boot up, CPU24 is the first CPU in its +* non-NULL LLC domain. However, +* its corresponding llc id has not been assigned yet. +* +* case 2: After boot up, the CPU100 is brought up +* via sysfs manually. As a result, CPU100 has only a kernel/sched/fair.c DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity); · line 720
Quoted 10 lines
kernel/sched/fair.c +10
-0+* Numa domain attached, because CPU100 is the only CPU +* of a sched domain, all its bottom domains are degenerated. +* The LLC domain pointer sd is NULL for CPU100. +* +* For both cases, we want to increase the number of LLCs. +*/ +per_cpu (sd_llc_id, cpu) = max_llcs++; + +return per_cpu (sd_llc_id, cpu); +}I'm not sure I follow. So
partition_sched_domains()first callsdetach_destroy_domains()on the old set, and thenbuild_sched_domains()on the new set.Do
detach_destroy_domain()will do:cpu_attach_domain(NULL,..);Quoted 8 lines
Quoted 8 lines
That is, it will explicitly attach the NULL
sched_domainto a CPU. At which point I feelupdate_llc_id()should be returning -1, no?Then later,
build_sched_domains()will set a !NULLsched_domain, at which pointupdate_llc_id()can set a real value.This should then also get rid of that weird
max_llcscheck inllc_id(), right?The check for
max_llcswas intended to prevent out-of-bounds access to rq->nr_pref_llc[] at multiple points in the code.
Sincedst_llc=llc_id(env->dst_cpu); — and while the LLC ID for the
CPU is updated inupdate_llc_id(), this update occurs before we reallocate thenr_pref_llcbuffer —dst_llcmay end up exceeding the bounds of the
originalnr_pref_llcbuffer.For this reason, we added a check if (
dst_llc>max_llc) inllc_id()when attempting to access rq->nr_pref_llc[dst_llc].However, I agree that the
max_llccheck seems to not properly integrated into the current patch: it should instead be placed in the 7th patch, as this would better illustrate the rationale for themax_llccheck here:
sched/cache: Introduce per runqueue task LLC preference counterIn the 7th patch, we actually increment
new_max_llcsrather thanmax_llcs— meaningmax_llcsalways represents the "old" number of LLCs. As a result, there is a race window between extending therq->nr_pref_llcbuffer and updatingmax_llcs.(quoted diff) +1
-1@@ -714,7 +827,7 @@ static int update_llc_id(struct sched_domain *sd, * * For both cases, we want to increase the number of LLCs. */ - per_cpu(sd_llc_id, cpu) = max_llcs++; + per_cpu(sd_llc_id, cpu) = new_max_llcs++; return per_cpu(sd_llc_id, cpu); }Quoted 5 lines
Thanks for pointing this out. Yes, we should take care of the attachment of NULL sd. Will update the code accordingly.
My understanding is that, if the sd is NULL, it is either because invoked by
detach_destroy_domain()for the old set, or by case 2 mentioned in above comments:
Say, CPU0-CPU95 are online during bootup, the boot command line is maxcpus=96.
Later after bootup, the user wants to bring up CPU100, the LLC domain for CPU100 is NULL in this case(due to sd generation), and a new LLC should be detected.Quoted 7 lines
That is to say, when we reach
update_llc_id(), there could be 2 reasons for NULL sd. For thedetach_destroy_domain()case,update_llc_id()should return a valid id without increasing themax_llcs, because ofif (id >= 0) return id;And for the latter, the
max_llcsshould be increased. Let me double check on this.
Quoted 2 lines
The issue is we could offline all CPUs in a LLC and online them later. In the current code, we will assign their ids all to -1.
I suppose we don't reset the ids in current implementation, only the first scan of LLCs will reset/initialize the ids to -1 in build_sched_domains()?
if (!max_llcs) { //max_llcs is initialized to 0 during bootup
for_each_possible_cpu(i)
per_cpu(sd_llc_id, i) = -1;
}Quoted 6 lines
So on attach
of CPUs again, we'll be assigning a new LLC. I think the proper thing to do is not to assign llc id of the offlined cpu (the case where sd == NULL) and keep the original llc id assigned. Then we should be okay and not increasemax_llcs.
This is the current implementation because we don't assign new ids to CPUs that already have an id(no matter it is offline/online).
thanks,
Chenyu
Re: [PATCH v2 17/23] sched/cache: Record the number of active threads per process for cache-aware scheduling
In reply to Tim Chenkernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1510
Quoted 5 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -1501,6 +1507,7 @@ static void __no_profile task_cache_work(struct callback_head *work) mm->mm_sched_cpu = m_a_cpu; } +update_avg (&mm->nr_running_avg, nr_running);
update_avg() doesn't appear to deal with small numbers well and can have an error as large as 7, e.g. when nr_running < 8, nr_running_avg will always be 0 and when nr_running >= 8 && < 16, nr_running_avg will be 1 - 8, etc.
AMD Genoa has 8 cores per LLC and this will break exceed_llc_nr() there.
Quoted 2 lines
kernel/sched/fair.c +0
-0free_cpumask_var (cpus); }
Re: [PATCH v2 23/23] -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs
In reply to Tim ChenQuoted 44 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
Show the per-LLC occupancy in
/proc/{PID}/schedstat, with each column corresponding to one LLC. This can be used to verify if the cache-aware load balancer works as expected by aggregating threads onto dedicated LLCs.Suppose there are 2 LLCs and the sampling duration is 10 seconds:
Enable the cache aware load balance:
0 12281 <--- LLC0 residency delta is 0, LLC1 is 12 seconds 0 18881
0 16217disable the cache aware load balance:
6497 15802
9299 5435
17811 8278Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- fs/proc/base.c | 22 ++++++++++++++++++++++ include/linux/mm_types.h | 19 +++++++++++++++++-- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 80 insertions(+), 4 deletions(-)fs/proc/base.c +7
-0@@ -518,6 +518,28 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, (unsigned long long )task->se.sum_exec_runtime, (unsigned long long )task->sched_info.run_delay, task->sched_info.pcount); +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +struct mm_struct *mm = task->mm; +u64 *llc_runtime; + +if (!mm) +return 0 ;
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 537
Quoted 10 lines
fs/proc/base.c +10
-0+ + llc_runtime =kcalloc (max_llcs,sizeof (u64),GFP_KERNEL ); +if (!llc_runtime) +return 0 ; + +if (get_mm_per_llc_runtime (task, llc_runtime)) + goto out; + +for (int i =0 ; i < max_llcs; i++) +seq_printf (m,"%llu " , llc_runtime[i]);
I feel it is better to also mark the current preferred LLC of this process so that I can know how well it works.
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 545
Quoted 8 lines
fs/proc/base.c +5
-0+seq_puts (m,"\n" ); +out: + kfree(llc_runtime); + } +#endif return 0 ; }
BTW, is there a way to tell if a process is being taken care of by 'cache aware scheduling' or it's blocked due to its huge rss or having too many threads?
I used below debug code to get these info through schedstat, but maybe I missed something and there is a simpler method?
fs/proc/base.c +12 -1
@@ -505,6 +505,7 @@ static int proc_pid_stack(struct seq_file *m, struct pid_namespace *ns,#endif #ifdef CONFIG_SCHED_INFO +DECLARE_PER_CPU (int, sd_llc_id);/* * Provides /proc/PID/schedstat */
@@ -522,6 +523,7 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,if (sched_cache_enabled ()) {struct mm_struct *mm = task->mm;u64 *llc_runtime; +int mm_sched_llc;if (!mm)return 0 ;
@@ -533,8 +535,17 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,if (get_mm_per_llc_runtime (task, llc_runtime)) goto out; +if (mm->mm_sched_cpu ==-1 ) + mm_sched_llc =-1 ; +else + mm_sched_llc =per_cpu (sd_llc_id, mm->mm_sched_cpu); + +seq_printf (m,"%llu 0x%x\n" , mm->nr_running_avg, mm->mm_sched_flags);for (int i =0 ; i < max_llcs; i++) -seq_printf (m,"%llu " , llc_runtime[i]); +seq_printf (m,"%s%s%llu " , + i == task->preferred_llc ?"*" :"" , + i == mm_sched_llc ?"?" :"" , + llc_runtime[i]);seq_puts (m,"\n" ); out:kfree (llc_runtime);
include/linux/mm_types.h +1 -0
@@ -1048,6 +1048,7 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; +int mm_sched_flags;u64 nr_running_avg____cacheline_aligned_in_smp ;#endif
kernel/sched/fair.c +25 -3
@@ -1237,12 +1237,20 @@ static inline int get_sched_cache_scale(int mul)return (1 + (llc_aggr_tolerance -1 ) * mul); } +#define MM_SCHED_EXCEED_LLC_CAPACITY 1 +#define MM_SCHED_NO_CACHE_INFO 2 +#define MM_SCHED_EXCEED_LLC_NR 4 +#define MM_SCHED_NR_THREADS 8 +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) {unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; + mm->mm_sched_flags &= ~MM_SCHED_NO_CACHE_INFO ; + mm->mm_sched_flags &= ~MM_SCHED_EXCEED_LLC_CAPACITY ; +/* * get_cpu_cacheinfo_level() can not be used * because it requires the cpu_hotplug_lock
@@ -1257,8 +1265,10 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)* L2 becomes the LLC. */ ci =_get_cpu_cacheinfo_level (cpu,2 ); -if (!ci) +if (!ci) { + mm->mm_sched_flags |=MM_SCHED_NO_CACHE_INFO ;return true; + } } llc = ci->size;
@@ -1283,13 +1293,20 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)if (scale ==INT_MAX )return false; -return ((llc * scale) <= (rss *PAGE_SIZE )); +if ((llc * scale) <= (rss *PAGE_SIZE )) { + mm->mm_sched_flags |=MM_SCHED_EXCEED_LLC_CAPACITY ; +return true; + } + +return false; }static bool exceed_llc_nr (struct mm_struct *mm,int cpu) {int smt_nr =1 , scale; + mm->mm_sched_flags &= ~MM_SCHED_EXCEED_LLC_NR ; +#ifdef CONFIG_SCHED_SMT if (sched_smt_active ()) smt_nr =cpumask_weight (cpu_smt_mask (cpu));
@@ -1313,7 +1330,12 @@ static bool exceed_llc_nr(struct mm_struct *mm, int cpu)if (scale ==INT_MAX )return false; -return ((mm->nr_running_avg * smt_nr) > (scale *per_cpu (sd_llc_size, cpu))); +if ((mm->nr_running_avg * smt_nr) > (scale *per_cpu (sd_llc_size, cpu))) { + mm->mm_sched_flags |=MM_SCHED_EXCEED_LLC_NR ; +return true; + } + +return false; }static void account_llc_enqueue (struct rq *rq,struct task_struct *p)
Re: [PATCH v2 06/23] sched/cache: Track LLC-preferred tasks per runqueue
In reply to Tim ChenQuoted 120 lines · Tim Chen
For each runqueue, track the number of tasks with an LLC preference and how many of them are running on their preferred LLC. This mirrors
nr_numa_runningandnr_preferred_runningfor NUMA balancing, and will be used by cache-aware load balancing in later patches.Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: Invoketask_of()once and reuse its result afterwards.
(Peter Zijlstra)
Remove hackyreset_llc_stats()and introducesched_llc_activeflag to properly pair enqueue/dequeue statistics update (Peter Zijlstra, K Prateek Nayak)include/linux/sched.h | 2 ++ init/init_task.c | 1 + kernel/sched/core.c | 5 ++++ kernel/sched/fair.c | 60 ++++++++++++++++++++++++++++++++++++++++--- kernel/sched/sched.h | 6 +++++ 5 files changed, 71 insertions(+), 3 deletions(-)include/linux/sched.h +2
-0@@ -1408,6 +1408,8 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; +/*the p is currently refcounted in a rq's preferred llc stats*/ +bool sched_llc_active;int preferred_llc;#endif init/init_task.c +1
-0@@ -192,6 +192,7 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .numa_faults = NULL,#endif #ifdef CONFIG_SCHED_CACHE + .sched_llc_active = false, .preferred_llc = -1,#endif #if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS )kernel/sched/core.c +5
-0@@ -531,6 +531,11 @@ void __trace_set_current_state(int state_value) }EXPORT_SYMBOL (__trace_set_current_state); +int task_llc (const struct task_struct *p) +{ +return per_cpu (sd_llc_id,task_cpu (p)); +} +/* * Serialization rules: * kernel/sched/fair.c +39
-0@@ -1223,6 +1223,43 @@ static int llc_id(int cpu)return llc; } +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) +{ +int pref_llc; + +if (!sched_cache_enabled ()) +return ; + + pref_llc = p->preferred_llc; +if (pref_llc <0 ) +return ; + + rq->nr_llc_running++; + rq->nr_pref_llc_running += (pref_llc ==task_llc (p)); + p->sched_llc_active = true; +} + +static void account_llc_dequeue (struct rq *rq,struct task_struct *p) +{ +int pref_llc; + +/* +* Borrow the uc_se->active from uclamp_rq_inc_id(), +* uclamp_rq_dec_id() to avoid the unbalanced calculation +* of rq statistics. +*/ +if (unlikely (!p->sched_llc_active)) +return ; + + pref_llc = p->preferred_llc; +if (pref_llc <0 ) +return ; + + rq->nr_llc_running--; + rq->nr_pref_llc_running -= (pref_llc ==task_llc (p)); + p->sched_llc_active = false; +} +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) {unsigned long epoch;@@ -1294,6 +1331,8 @@ static unsigned long __no_profile fraction_mm_sched(struct rq *rq, struct mm_schreturn div64_u64 (NICE_0_LOAD * pcpu_sched->runtime, rq->cpu_runtime +1 ); } +static unsigned int task_running_on_cpu (int cpu,struct task_struct *p); +static inline void account_mm_sched (struct rq *rq,struct task_struct *p,s64 delta_exec) {
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1390
Quoted 7 lines
kernel/sched/fair.c +3
-1@@ -1346,8 +1385,13 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)#endif } -if (p->preferred_llc != mm_sched_llc) +/* task not on rq accounted later in account_entity_enqueue() */ +if (task_running_on_cpu (rq->cpu, p) && + p->preferred_llc != mm_sched_llc) {
#ifdef CONFIG_NUMA_BALANCING
/*
* Don't assign preferred LLC if it
* conflicts with NUMA balancing.
*/
if (p->numa_preferred_nid >= 0 && cpu_to_node(mm->mm_sched_cpu) != p->numa_preferred_nid) mm_sched_llc = -1;
#endif
}
/* task not on rq accounted later in account_entity_enqueue() */ if (task_running_on_cpu(rq->cpu, p) && p->preferred_llc != mm_sched_llc) { account_llc_dequeue(rq, p);
p->preferred_llc = mm_sched_llc; account_llc_enqueue(rq, p);
}
I am a little concerned that there might be cases where both |p->preferred_llc| and |mm_sched_llc| are equal to -1 at this point.", Is it necessary to add a check here?
Quoted 68 lines
kernel/sched/fair.c +15
-2+account_llc_dequeue (rq, p); p->preferred_llc = mm_sched_llc; +account_llc_enqueue (rq, p); + } }static void task_tick_cache (struct rq *rq,struct task_struct *p)@@ -1475,6 +1519,10 @@ void init_sched_mm(struct task_struct *p) { }static void task_tick_cache (struct rq *rq,struct task_struct *p) { } +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) {} + +static void account_llc_dequeue (struct rq *rq,struct task_struct *p) {} +#endif /* @@ -3965,9 +4013,11 @@ account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se) {update_load_add (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) { +struct task_struct *p =task_of (se);struct rq *rq =rq_of (cfs_rq); -account_numa_enqueue (rq,task_of (se)); +account_numa_enqueue (rq, p); +account_llc_enqueue (rq, p);list_add (&se->group_node, &rq->cfs_tasks); } cfs_rq->nr_queued++;@@ -3978,7 +4028,11 @@ account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se) {update_load_sub (&cfs_rq->load, se->load.weight);if (entity_is_task (se)) { -account_numa_dequeue (rq_of (cfs_rq),task_of (se)); +struct task_struct *p =task_of (se); +struct rq *rq =rq_of (cfs_rq); + +account_numa_dequeue (rq, p); +account_llc_dequeue (rq, p);list_del_init (&se->group_node); } cfs_rq->nr_queued--;kernel/sched/sched.h +6
-0@@ -1126,6 +1126,10 @@ struct rq {unsigned int nr_preferred_running;unsigned int numa_migrate_on;#endif +#ifdef CONFIG_SCHED_CACHE +unsigned int nr_pref_llc_running; +unsigned int nr_llc_running; +#endif #ifdef CONFIG_NO_HZ_COMMON unsigned long last_blocked_load_update_tick;unsigned int has_blocked_load;@@ -1980,6 +1984,8 @@ init_numa_balancing(u64 clone_flags, struct task_struct *p)#endif /* !CONFIG_NUMA_BALANCING */ +int task_llc (const struct task_struct *p); + static inline void queue_balance_callback(struct rq *rq, struct balance_callback *head,
Re: [PATCH v2 06/23] sched/cache: Track LLC-preferred tasks per runqueue
Quoted 29 lines · Vern Hao
Quoted 27 lines · Tim Chen
(quoted diff) +3
-1@@ -1346,8 +1385,13 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) #endif } - if (p->preferred_llc != mm_sched_llc) + /* task not on rq accounted later in account_entity_enqueue() */ + if (task_running_on_cpu(rq->cpu, p) && + p->preferred_llc != mm_sched_llc) { #ifdef CONFIG_NUMA_BALANCING /* * Don't assign preferred LLC if it * conflicts with NUMA balancing. */ if (p->numa_preferred_nid >= 0 && cpu_to_node(mm->mm_sched_cpu) != p- >numa_preferred_nid) mm_sched_llc = -1;#endif
}/* task not on rq accounted later in
account_entity_enqueue()*/ if (task_running_on_cpu(rq->cpu, p)&&p->preferred_llc!=mm_sched_llc) {account_llc_dequeue(rq, p);
p->preferred_llc=mm_sched_llc;account_llc_enqueue(rq, p);
Quoted 6 lines
Quoted 2 lines
}
I am a little concerned that there might be cases where both |p-
Quoted 1 line
preferred_llc| and |mm_sched_llc| are equal to -1 at this point.", Is
it necessary to add a check here?
Are you concerned about the mismatch between the percpu runqueue values of nr_pref_llc_running, nr_pref_llc, and nr_llc_running? This should not be an issue, because account_llc_dequeue() and account_llc_enqueue() are always invoked together in account_mm_sched(). If p->preferred_llc = mm_sched_llc = -1, account_llc_dequeue/enqueue() will not be invoked, so it is still paired. Please let me know if I understand your comments correctly.
thanks,
Chenyu
Re: [PATCH v2 17/23] sched/cache: Record the number of active threads per process for cache-aware scheduling
In reply to Aaron LuQuoted 5 lines · Aaron Lu
Quoted 4 lines · Tim Chen
kernel/sched/fair.c +0
-0@@ -1501,6 +1507,7 @@ static void __no_profile task_cache_work(struct callback_head *work) mm->mm_sched_cpu = m_a_cpu; }
Quoted 9 lines
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1510
Quoted 1 line
kernel/sched/fair.c +1
-0+update_avg (&mm->nr_running_avg, nr_running);
update_avg()doesn't appear to deal with small numbers well and can have an error as large as 7, e.g. whennr_running< 8,nr_running_avgwill always be 0 and whennr_running>= 8 && < 16,nr_running_avgwill be 1 - 8, etc.AMD Genoa has 8 cores per LLC and this will break
exceed_llc_nr()there.
Ah, you are right, thanks for pointing this out, dividing by 8 would make convergence slow for small LLC system. Maybe consider the number of Cores in the LLC, the smaller the number is, the more we should honor the diff between two invoking of update_avg()?
static inline void sched_cache_update_avg(u64 *avg, u64 sample)
{
s64 diff = sample - *avg;
u32 divisor = clamp_t(u32, nr_cores_llc/4, 2, 8);
*avg += diff / divisor;
}For <=8 cores per LLC, the divisor is 2,
for 16 cores per LLC, the divisor is 4,
for >=32 cores per LLC, the divisor is 8
Thanks,
Chenyu
Quoted 2 lines
Quoted 2 lines
kernel/sched/fair.c +0
-0free_cpumask_var (cpus); }
Re: [PATCH v2 23/23] -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs
In reply to Aaron LuQuoted 50 lines · Aaron Lu
Quoted 49 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
Show the per-LLC occupancy in
/proc/{PID}/schedstat, with each column corresponding to one LLC. This can be used to verify if the cache-aware load balancer works as expected by aggregating threads onto dedicated LLCs.Suppose there are 2 LLCs and the sampling duration is 10 seconds:
Enable the cache aware load balance:
0 12281 <--- LLC0 residency delta is 0, LLC1 is 12 seconds 0 18881
0 16217disable the cache aware load balance:
6497 15802
9299 5435
17811 8278Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- fs/proc/base.c | 22 ++++++++++++++++++++++ include/linux/mm_types.h | 19 +++++++++++++++++-- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 80 insertions(+), 4 deletions(-)fs/proc/base.c +12
-0@@ -518,6 +518,28 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, (unsigned long long )task->se.sum_exec_runtime, (unsigned long long )task->sched_info.run_delay, task->sched_info.pcount); +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +struct mm_struct *mm = task->mm; +u64 *llc_runtime; + +if (!mm) +return 0 ; + + llc_runtime =kcalloc (max_llcs,sizeof (u64),GFP_KERNEL ); +if (!llc_runtime) +return 0 ; +
Quoted 9 lines
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 537
Quoted 5 lines
fs/proc/base.c +5
-0+if (get_mm_per_llc_runtime (task, llc_runtime)) + goto out; + +for (int i =0 ; i < max_llcs; i++) +seq_printf (m,"%llu " , llc_runtime[i]);I feel it is better to also mark the current preferred LLC of this process so that I can know how well it works.
Sure.
Quoted 37 lines
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 545
Quoted 8 lines
fs/proc/base.c +5
-0+seq_puts (m,"\n" ); +out: + kfree(llc_runtime); + } +#endif return 0 ; }BTW, is there a way to tell if a process is being taken care of by 'cache aware scheduling' or it's blocked due to its huge rss or having too many threads?
I used below debug code to get these info through schedstat, but maybe I missed something and there is a simpler method?
fs/proc/base.c +2
-0@@ -505,6 +505,7 @@ static int proc_pid_stack(struct seq_file *m, struct pid_namespace *ns,#endif #ifdef CONFIG_SCHED_INFO +DECLARE_PER_CPU (int, sd_llc_id);/* * Provides /proc/PID/schedstat */ @@ -522,6 +523,7 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,if (sched_cache_enabled ()) {struct mm_struct *mm = task->mm;u64 *llc_runtime; +int mm_sched_llc;if (!mm)return 0 ;
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 541
Quoted 7 lines
fs/proc/base.c +4
-0@@ -533,8 +535,17 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,if (get_mm_per_llc_runtime (task, llc_runtime)) goto out; +if (mm->mm_sched_cpu ==-1 ) + mm_sched_llc =-1 ; +else + mm_sched_llc =per_cpu (sd_llc_id, mm->mm_sched_cpu);
We can use llc_id(mm->mm_sched_cpu).
Quoted 35 lines
fs/proc/base.c +6
-1+ +seq_printf (m,"%llu 0x%x\n" , mm->nr_running_avg, mm->mm_sched_flags);for (int i =0 ; i < max_llcs; i++) -seq_printf (m,"%llu " , llc_runtime[i]); +seq_printf (m,"%s%s%llu " , + i == task->preferred_llc ?"*" :"" , + i == mm_sched_llc ?"?" :"" , + llc_runtime[i]);seq_puts (m,"\n" ); out:kfree (llc_runtime);include/linux/mm_types.h +1
-0@@ -1048,6 +1048,7 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; +int mm_sched_flags;u64 nr_running_avg____cacheline_aligned_in_smp ;#endif kernel/sched/fair.c +4
-0@@ -1237,12 +1237,20 @@ static inline int get_sched_cache_scale(int mul)return (1 + (llc_aggr_tolerance -1 ) * mul); } +#define MM_SCHED_EXCEED_LLC_CAPACITY 1 +#define MM_SCHED_NO_CACHE_INFO 2 +#define MM_SCHED_EXCEED_LLC_NR 4 +#define MM_SCHED_NR_THREADS 8
kernel/sched/fair.c static inline int get_sched_cache_scale(int mul) · line 1253
Quoted 10 lines
kernel/sched/fair.c +4
-0+static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) {unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; + mm->mm_sched_flags &= ~MM_SCHED_NO_CACHE_INFO ; + mm->mm_sched_flags &= ~MM_SCHED_EXCEED_LLC_CAPACITY ; +
Maybe we can do some read-comparison before writing the flags, previously we found that writing the per-process mm struct is very expensive, so maybe avoid writing to it as much as possible.
I'll fold your changes and do the test. Thanks!
Thanks,
Chenyu
Re: [PATCH v2 19/23] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
In reply to Tim ChenQuoted 9 lines · Tim Chen
From: Chen Yu <[email protected]>
Prateek and Tingyin reported that memory-intensive workloads (such as stream) can saturate memory bandwidth and caches on the preferred LLC when
sched_cacheaggregates too many threads.To mitigate this, estimate a process's memory footprint by comparing its RSS (anonymous and shared pages) to the size of the LLC. If RSS exceeds the LLC size, skip cache-aware scheduling.
Restricting RSS prevents many applications from benefiting from this optimization. I believe this restriction should be lifted. For memory-intensive workloads, the optimization may simply yield no gains, but it certainly shouldn't make performance worse. We need to further refine this logic.
Quoted 87 lines
Note that RSS is only an approximation of the memory footprint. By default, the comparison is strict, but a later patch will allow users to provide a hint to adjust this threshold.
According to the test from Adam, some systems do not have shared L3 but with shared L2 as clusters. In this case, the L2 becomes the LLC[1].
Link[1]: https://lore.kernel.org/all/[email protected]/
Co-developed-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: Assigned curr_cpu in task_cache_work() before checking exceed_llc_capacity(mm, curr_cpu) to avoid out-of-bound access.(lkp/0day)include/linux/cacheinfo.h | 21 ++++++++++------- kernel/sched/fair.c | 49 +++++++++++++++++++++++++++++++++++---- 2 files changed, 57 insertions(+), 13 deletions(-)include/linux/cacheinfo.h +13
-8@@ -113,18 +113,11 @@ int acpi_get_cache_info(unsigned int cpu,const struct attribute_group *cache_get_priv_group (struct cacheinfo *this_leaf); -/* -* Get the cacheinfo structure for the cache associated with @cpu at -* level @level. -* cpuhp lock must be held. -*/ -static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu,int level) +static inline struct cacheinfo *_get_cpu_cacheinfo_level (int cpu,int level) {struct cpu_cacheinfo *ci =get_cpu_cacheinfo (cpu);int i; -lockdep_assert_cpus_held (); -for (i =0 ; i < ci->num_leaves; i++) {if (ci->info_list[i].level == level) {if (ci->info_list[i].attributes &CACHE_ID )@@ -136,6 +129,18 @@ static inline struct cacheinfo *get_cpu_cacheinfo_level(int cpu, int level)return NULL ; } +/* +* Get the cacheinfo structure for the cache associated with @cpu at +* level @level. +* cpuhp lock must be held. +*/ +static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu,int level) +{ +lockdep_assert_cpus_held (); + +return _get_cpu_cacheinfo_level (cpu, level); +} +/* * Get the id of the cache associated with @cpu at level @level. * cpuhp lock must be held. kernel/sched/fair.c +13
-0@@ -1223,6 +1223,38 @@ static int llc_id(int cpu)return llc; } +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) +{ +struct cacheinfo *ci; +unsigned long rss; +unsigned int llc; + +/* +* get_cpu_cacheinfo_level() can not be used +* because it requires the cpu_hotplug_lock +* to be held. Use _get_cpu_cacheinfo_level() +* directly because the 'cpu' can not be +* offlined at the moment. +*/
kernel/sched/fair.c static int llc_id(int cpu) · line 1248
Quoted 10 lines
kernel/sched/fair.c +10
-0+ ci =_get_cpu_cacheinfo_level (cpu,3 ); +if (!ci) { +/* +* On system without L3 but with shared L2, +* L2 becomes the LLC. +*/ + ci =_get_cpu_cacheinfo_level (cpu,2 ); +if (!ci) +return true; + }
Is there must call it one by one for get llc size? a static variable instead in building sched domain?
Quoted 57 lines
kernel/sched/fair.c +21
-5+ + llc = ci->size; + + rss =get_mm_counter (mm,MM_ANONPAGES ) + +get_mm_counter (mm,MM_SHMEMPAGES ); + +return (llc <= (rss *PAGE_SIZE )); +} +static bool exceed_llc_nr (struct mm_struct *mm,int cpu) {int smt_nr =1 ;@@ -1382,7 +1414,8 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)*/ if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT ||get_nr_threads (p) <=1 || -exceed_llc_nr (mm,cpu_of (rq))) { +exceed_llc_nr (mm,cpu_of (rq)) || +exceed_llc_capacity (mm,cpu_of (rq))) {if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; }@@ -1439,7 +1472,7 @@ static void __no_profile task_cache_work(struct callback_head *work)struct mm_struct *mm = p->mm;unsigned long m_a_occ =0 ;unsigned long curr_m_a_occ =0 ; -int cpu, m_a_cpu =-1 , nr_running =0 ; +int cpu, m_a_cpu =-1 , nr_running =0 , curr_cpu;cpumask_var_t cpus;WARN_ON_ONCE (work != &p->cache_work);@@ -1449,7 +1482,9 @@ static void __no_profile task_cache_work(struct callback_head *work)if (p->flags &PF_EXITING )return ; -if (get_nr_threads (p) <=1 ) { + curr_cpu =task_cpu (p); +if (get_nr_threads (p) <=1 || +exceed_llc_capacity (mm, curr_cpu)) {if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ;@@ -9895,8 +9930,12 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,if (cpu <0 ||cpus_share_cache (src_cpu, dst_cpu))return mig_unrestricted; -/* skip cache aware load balance for single/too many threads */ -if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu)) +/* +* Skip cache aware load balance for single/too many threads +* or large footprint. +*/ +if (get_nr_threads (p) <=1 ||exceed_llc_nr (mm, dst_cpu) || +exceed_llc_capacity (mm, dst_cpu))return mig_unrestricted;if (cpus_share_cache (dst_cpu, cpu))
Re: [PATCH v2 19/23] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
Quoted 4 lines · Vern Hao
Quoted 2 lines · Tim Chen
From: Chen Yu <[email protected]>
Quoted 8 lines
Quoted 7 lines
Prateek and Tingyin reported that memory-intensive workloads (such as stream) can saturate memory bandwidth and caches on the preferred LLC when
sched_cacheaggregates too many threads.To mitigate this, estimate a process's memory footprint by comparing its RSS (anonymous and shared pages) to the size of the LLC. If RSS exceeds the LLC size, skip cache-aware scheduling.
Restricting RSS prevents many applications from benefiting from this optimization. I believe this restriction should be lifted. For memory- intensive workloads, the optimization may simply yield no gains, but it certainly shouldn't make performance worse. We need to further refine this logic.
Memory-intensive workloads may trigger performance regressions when memory bandwidth(from L3 cache to memory controller) is saturated due to task aggregation on single LLC. We have seen this issue in stream benchmark runs in previous version.
Patch 23 introduces a debugfs knob llc_aggr_tolerance that lets userspace tune the scale factor. This allows memory-intensive workloads to perform task aggregation when their footprint is small and the administrator considers it safe. As you noted in another patch, fine-grained control would improve flexibility—and this can be addressed in future iterations.
Quoted 89 lines
Quoted 89 lines
Note that RSS is only an approximation of the memory footprint. By default, the comparison is strict, but a later patch will allow users to provide a hint to adjust this threshold.
According to the test from Adam, some systems do not have shared L3 but with shared L2 as clusters. In this case, the L2 becomes the LLC[1].
Link[1]: https://lore.kernel.org/all/3cb6ebc7-a2fd-42b3-8739- [email protected]/
Co-developed-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: Assignedcurr_cpuintask_cache_work()before checkingexceed_llc_capacity(mm, curr_cpu)to avoid out-of-bound access.(lkp/0day)
include/linux/cacheinfo.h| 21 ++++++++++------- kernel/sched/fair.c | 49 +++++++++++++++++++++++++++++++++++----2 files changed, 57 insertions(+), 13 deletions(-)include/linux/cacheinfo.h +13
-8@@ -113,18 +113,11 @@ int acpi_get_cache_info(unsigned int cpu, const struct attribute_group *cache_get_priv_group(struct cacheinfo *this_leaf); -/* -* Get the cacheinfo structure for the cache associated with @cpu at -* level @level. -* cpuhp lock must be held. -*/ -static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu,int level) +static inline struct cacheinfo *_get_cpu_cacheinfo_level(int cpu, int level) { struct cpu_cacheinfo *ci = get_cpu_cacheinfo(cpu); int i; - lockdep_assert_cpus_held(); - for (i = 0; i < ci->num_leaves; i++) { if (ci->info_list[i].level == level) { if (ci->info_list[i].attributes & CACHE_ID)@@ -136,6 +129,18 @@ static inline struct cacheinfo *get_cpu_cacheinfo_level(int cpu, int level) return NULL; } +/* +* Get the cacheinfo structure for the cache associated with @cpu at +* level @level. +* cpuhp lock must be held. +*/ +static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu,int level) +{ + lockdep_assert_cpus_held(); + + return _get_cpu_cacheinfo_level(cpu, level); +} + /** Get the id of the cache associated with @cpu at level @level. * cpuhp lock must be held.
kernel/sched/fair.c +16
-0@@ -1223,6 +1223,38 @@ static int llc_id(int cpu) return llc; } +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) +{ + struct cacheinfo *ci; + unsigned long rss; + unsigned int llc; + + /* + * get_cpu_cacheinfo_level() can not be used + * because it requires the cpu_hotplug_lock + * to be held. Use _get_cpu_cacheinfo_level() + * directly because the 'cpu' can not be + * offlined at the moment. + */ + ci = _get_cpu_cacheinfo_level(cpu, 3); + if (!ci) { + /*
Quoted 8 lines
kernel/sched/fair.c static int llc_id(int cpu) · line 1248
Quoted 7 lines
kernel/sched/fair.c +7
-0+* On system without L3 but with shared L2, +* L2 becomes the LLC. +*/ + ci = _get_cpu_cacheinfo_level(cpu, 2); + if (!ci) + return true; + }Is there must call it one by one for get llc size? a static variable instead in building sched domain?
I suppose you suggested introducing a per-CPU variable, like percpu(sd_llc_bytes, cpu), or something similar to struct cpuinfo_x86.x86_cache_size. I am not sure if the community would endorse introducing this variable, given that sched_cache would be its only user. We can leave this as an open question.
thanks,
Chenyu
Re: [PATCH v2 19/23] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
Quoted 9 lines · Yu C
Quoted 8 lines · Vern Hao
Quoted 6 lines · Tim Chen
From: Chen Yu <[email protected]>
Prateek and Tingyin reported that memory-intensive workloads (such as stream) can saturate memory bandwidth and caches on the preferred LLC when
sched_cacheaggregates too many threads.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
To mitigate this, estimate a process's memory footprint by comparing its RSS (anonymous and shared pages) to the size of the LLC. If RSS exceeds the LLC size, skip cache-aware scheduling.
Restricting RSS prevents many applications from benefiting from this optimization. I believe this restriction should be lifted. For memory- intensive workloads, the optimization may simply yield no gains, but it certainly shouldn't make performance worse. We need to further refine this logic.
Memory-intensive workloads may trigger performance regressions when memory bandwidth(from L3 cache to memory controller) is saturated due
RSS size and bandwidth saturation are not necessarily linked, In my view, the optimization should be robust enough that it doesn't cause a noticeable drop in performance, no matter how large the RSS is. We need to have a more profound discussion on this.
Quoted 115 lines
to task aggregation on single LLC. We have seen this issue in stream benchmark runs in previous version.
Patch 23 introduces a debugfs knob
llc_aggr_tolerancethat lets userspace tune the scale factor. This allows memory-intensive workloads to perform task aggregation when their footprint is small and the administrator considers it safe. As you noted in another patch, fine-grained control would improve flexibility—and this can be addressed in future iterations.Quoted 90 lines
Quoted 90 lines
Note that RSS is only an approximation of the memory footprint. By default, the comparison is strict, but a later patch will allow users to provide a hint to adjust this threshold.
According to the test from Adam, some systems do not have shared L3 but with shared L2 as clusters. In this case, the L2 becomes the LLC[1].
Link[1]: https://lore.kernel.org/all/3cb6ebc7-a2fd-42b3-8739- [email protected]/
Co-developed-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: Assignedcurr_cpuintask_cache_work()before checkingexceed_llc_capacity(mm, curr_cpu)to avoid out-of-bound access.(lkp/0day)
include/linux/cacheinfo.h| 21 ++++++++++------- kernel/sched/fair.c | 49
+++++++++++++++++++++++++++++++++++----2 files changed, 57 insertions(+), 13 deletions(-)include/linux/cacheinfo.h +13
-8@@ -113,18 +113,11 @@ int acpi_get_cache_info(unsigned int cpu, const struct attribute_group *cache_get_priv_group(struct cacheinfo *this_leaf); -/* -* Get the cacheinfo structure for the cache associated with @cpu at -* level @level. -* cpuhp lock must be held. -*/ -static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu, int level) +static inline struct cacheinfo *_get_cpu_cacheinfo_level (int cpu,int level) { struct cpu_cacheinfo *ci = get_cpu_cacheinfo(cpu); int i; - lockdep_assert_cpus_held(); - for (i = 0; i < ci->num_leaves; i++) { if (ci->info_list[i].level == level) { if (ci->info_list[i].attributes & CACHE_ID)@@ -136,6 +129,18 @@ static inline struct cacheinfo *get_cpu_cacheinfo_level(int cpu, int level) return NULL; } +/* +* Get the cacheinfo structure for the cache associated with @cpu at +* level @level. +* cpuhp lock must be held. +*/ +static inline struct cacheinfo *get_cpu_cacheinfo_level (int cpu,int level) +{ + lockdep_assert_cpus_held(); + + return _get_cpu_cacheinfo_level(cpu, level); +} + /** Get the id of the cache associated with @cpu at level @level. * cpuhp lock must be held.
kernel/sched/fair.c +16
-0@@ -1223,6 +1223,38 @@ static int llc_id(int cpu) return llc; } +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) +{ + struct cacheinfo *ci; + unsigned long rss; + unsigned int llc; + + /* + * get_cpu_cacheinfo_level() can not be used + * because it requires the cpu_hotplug_lock + * to be held. Use _get_cpu_cacheinfo_level() + * directly because the 'cpu' can not be + * offlined at the moment. + */ + ci = _get_cpu_cacheinfo_level(cpu, 3); + if (!ci) { + /*Quoted 8 lines
kernel/sched/fair.c static int llc_id(int cpu) · line 1248
Quoted 7 lines
kernel/sched/fair.c +7
-0+* On system without L3 but with shared L2, +* L2 becomes the LLC. +*/ + ci = _get_cpu_cacheinfo_level(cpu, 2); + if (!ci) + return true; + }Is there must call it one by one for get llc size? a static variable instead in building sched domain?
I suppose you suggested introducing a per-CPU variable, like
percpu(sd_llc_bytes, cpu), or something similar to struct cpuinfo_x86.x86_cache_size. I am not sure if the community would endorse introducing this variable, given thatsched_cachewould be its only user. We can leave this as an open question.thanks,
Chenyu
K Prateek Nayak
reviewerRe: [PATCH v2 19/23] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
Hello Vern,
Quoted 11 lines · Vern Hao
Quoted 9 lines · Yu C
Quoted 8 lines · Vern Hao
Quoted 6 lines · Tim Chen
From: Chen Yu <[email protected]>
Prateek and Tingyin reported that memory-intensive workloads (such as stream) can saturate memory bandwidth and caches on the preferred LLC when
sched_cacheaggregates too many threads.
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
To mitigate this, estimate a process's memory footprint by comparing its RSS (anonymous and shared pages) to the size of the LLC. If RSS exceeds the LLC size, skip cache-aware scheduling.
Restricting RSS prevents many applications from benefiting from this optimization. I believe this restriction should be lifted. For memory- intensive workloads, the optimization may simply yield no gains, but it certainly shouldn't make performance worse. We need to further refine this logic.
Memory-intensive workloads may trigger performance regressions when memory bandwidth(from L3 cache to memory controller) is saturated due
RSS size and bandwidth saturation are not necessarily linked, In my view, the optimization should be robust enough that it doesn't cause a noticeable drop in performance, no matter how large the RSS is.
Easier said than done. I agree RSS size is not a clear indication of bandwidth saturation. With NUMA Balancing enabled, we can use the hinting faults to estimate the working set and make decisions but for systems that do not have NUMA, short of programming some performance counters, there is no real way to estimate the working set.
Hinting faults are known to cause overheads so enabling them without NUMA can cause noticeable overheads with no real benefits.
Quoted 1 line
We need to have a more profound discussion on this.
What do you have in mind?
From where I stand, having the RSS based bailout for now won't make things worse for these tasks with huge memory reserves and when we can all agree on some generic method to estimate the working set of a task, we can always add it into exceed_llc_capacity().
Signature · 7 lines
Thanks and Regards, Prateek "Rome wasn't built in a day but they were laying bricks every hour. You don't have to build everything you want today, just lay a brick." - James Clear
On Wed, Dec 03, 2025 at 03:07:19PM -0800, Tim Chen wrote:
... ...
Quoted 23 lines
Test results:
The patch series was applied and tested on v6.18-rc7.
See: https://github.com/timcchen1298/linux/commits/cache_aware_v2The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.
The second test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.
hackbench/schbench/netperf/stream/stress-ng/chacha20 were launched on these two platforms.
[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when the number of different active threads is below the capacity of a LLC. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows good throughput improvement.
Quoted 3 lines
Genoa:
ChaCha20-xiangshan shows huge throughput improvement. No obvious difference is observed in hackbench/schbench
I think for small task number hackbench run, there should be some improvement.
I tried thread/pipe/2fds/1group, i.e. 4 tasks on Genoa:
./hackbench -T -f 2 -g 1 -p -l 2000000
And I noticed performance improved a lot:
(Result in seconds, less is better)
llc_off llc_on diff
time 4.755±1.6% 2.684±6.25% +43.6%
llc_off means /sys/kernel/debug/sched/llc_enabled set to 0 while llc_on means /sys/kernel/debug/sched/llc_enabled set to 1, other tunnables are left unchanged.
Turbo is disabled and cpufreq set to performance.
I also tried redis and noticed when I set io-threads to 4 in redis.conf, there is also some improvement on AMD Genoa:
llc_off manual diff llc_on diff
throughput 1536727±0% 1737619±0% +13.1% 1737720±0% +13.1%
Client cmdline:
numactl -N 1 redis-benchmark --threads 4 -t set -r 100000 -P 16 -n 10000000 server cmdline: numactl -N 0 redis-server ./redis.conf
I also tried to manually bind all tasks of redis server to a single LLC to see if this workload benefits from aggregation and that's what manual
means: taskset -c 8-15,200-207 redis-server ./redis.conf
According to the result, I think this 'cache aware scheduling' works as expected in that its performance is the same as manual binding; and they all beat llc_off.
Re: [PATCH v2 17/23] sched/cache: Record the number of active threads per process for cache-aware scheduling
In reply to Chen, Yu CQuoted 12 lines · Yu C
Quoted 11 lines · Aaron Lu
kernel/sched/fair.c static void __no_profile task_cache_work(struct callback_head *work) · line 1509
Quoted 4 lines · Tim Chen
kernel/sched/fair.c +1
-0@@ -1501,6 +1507,7 @@ static void __no_profile task_cache_work(struct callback_head *work) mm->mm_sched_cpu = m_a_cpu; } +update_avg (&mm->nr_running_avg, nr_running);
update_avg()doesn't appear to deal with small numbers well and can have an error as large as 7, e.g. whennr_running< 8,nr_running_avgwill always be 0 and whennr_running>= 8 && < 16,nr_running_avgwill be 1 - 8, etc.
Quoted 5 lines
Quoted 2 lines
AMD Genoa has 8 cores per LLC and this will break
exceed_llc_nr()there.Ah, you are right, thanks for pointing this out, dividing by 8 would make convergence slow for small LLC system. Maybe consider the number of Cores
Not just slow but the error is too large for a small LLC.
Quoted 11 lines
in the LLC, the smaller the number is, the more we should honor the diff between two invoking of
update_avg()?static inline void
sched_cache_update_avg(u64 *avg, u64 sample){ s64 diff = sample - *avg; u32 divisor = clamp_t(u32, nr_cores_llc/4, 2, 8); *avg += diff / divisor; }
Quoted 3 lines
For <=8 cores per LLC, the divisor is 2,
for 16 cores per LLC, the divisor is 4,
for >=32 cores per LLC, the divisor is 8
Yeah I guess it works. The error can be as large as 'divisor - 1' but since this avg is an estimate, it may be OK.
Re: [PATCH v2 05/23] sched/cache: Assign preferred LLC ID to processes
In reply to Tim ChenQuoted 75 lines · Tim Chen
Quoted 66 lines · Vern Hao
Quoted 65 lines · Tim Chen
With cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar to
numa_preferred_nidin NUMA balancing.Signed-off-by Tim Chen <[email protected]>---
Notes:
v1->v2: Align preferred LLC with NUMA balancing's preferred node.include/linux/sched.h | 1 + init/init_task.c | 3 +++ kernel/sched/fair.c | 18 ++++++++++++++++++ 3 files changed, 22 insertions(+)include/linux/sched.h +1
-0@@ -1408,6 +1408,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; +int preferred_llc;#endif #ifdef CONFIG_RSEQ init/init_task.c +3
-0@@ -191,6 +191,9 @@ struct task_struct init_task __aligned(L1_CACHE_BYTES) = { .numa_group = NULL, .numa_faults = NULL,#endif +#ifdef CONFIG_SCHED_CACHE + .preferred_llc = -1, +#endif #if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS ) .kasan_depth = 1,#endif kernel/sched/fair.c +8
-0@@ -1300,6 +1300,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched;unsigned long epoch; +int mm_sched_llc =-1 ;if (!sched_cache_enabled ())return ;@@ -1330,6 +1331,23 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; } + +if (mm->mm_sched_cpu !=-1 ) { + mm_sched_llc =llc_id (mm->mm_sched_cpu); + +#ifdef CONFIG_NUMA_BALANCING +/* +* Don't assign preferred LLC if it Quoted 8 lines
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1343
Quoted 3 lines
kernel/sched/fair.c +3
-0+* conflicts with NUMA balancing. +*/ + if (p->numa_preferred_nid >= 0 &&I wonder if the restriction here shouldn't be so strict. In Mel Gorman's patch (
e496132ebeddsched/fair: Adjust the allowed NUMA imbalance whenSD_NUMAspans multiple LLCs), the value of the 'imb_numa_nr' is checked to determine if |SD_NUMA| imbalance is allowed. Could we use this same check to decide whether or not to perform a cross-numa migration?
Quoted 4 lines
If we set the preferred LLC that's in a different node other than the preferred node, the preferred LLC is going to fight with NUMA balancing and bounce tasks back and forth between nodes. NUMA locality is going to affect performance more so we'll let NUMA preference take precedence.
I might not have explained myself clearly. I’m questioning whether we need to integrate an imbalance check into the 'sgs->group_type|' is ’group_has_spare|' scenario, like Mel’s patch, to refine our llc migration decisions.
just like this: 8 cpus in one LLC, LLC-A has 6 tasks, LLC-B has 2 tasks, if LLC-A has task_a need to migrate to LLC_B, how to deal it ?
Quoted 12 lines
Tim
Quoted 10 lines
Quoted 10 lines
kernel/sched/fair.c +7
-0+ cpu_to_node(mm->mm_sched_cpu) != p->numa_preferred_nid) + mm_sched_llc =-1 ; +#endif + } + +if (p->preferred_llc != mm_sched_llc) + p->preferred_llc = mm_sched_llc; }static void task_tick_cache (struct rq *rq,struct task_struct *p)
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Tim ChenQuoted 180 lines · Tim Chen
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.For example, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped(784GB = (1 + (99 - 1) * 256) * 32MB).Similarly,
/sys/kernel/debug/sched/llc_aggr_tolerancealso controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with
nr_running> 8 are skipped.
- llc_aggr_tolerance=99 -> tasks withnr_running> 785 are skipped785 = (1 + (99 - 1) * 8).(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides,llc_epoch_periodandllc_epoch_affinity_timeoutare also turned into tunable.Suggested-by K Prateek Nayak <[email protected]>Suggested-by Madadi Vineeth Reddy <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Suggested-by Tingyin Duan <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>---
Notes:
v1->v2: Remove the smt_nr check in fits_llc_capacity(). (Aaron Lu)include/linux/sched.h | 4 ++- kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++ kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- 5 files changed, 178 insertions(+), 10 deletions(-)include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +62
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &val)) \ + return -EINVAL; \ + if (val > (max)) \ + return -EINVAL; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 ); +#endif /* SCHED_CACHE */ +static ssize_t sched_scaling_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) {@@ -523,6 +570,21 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +debugfs_create_file ("llc_overload_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_overload_pct); +debugfs_create_file ("llc_imb_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_imb_pct); +debugfs_create_file ("llc_aggr_tolerance" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_tolerance); +debugfs_create_file ("llc_enabled" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_enabled); +debugfs_create_u32 ("llc_epoch_period" ,0644 , debugfs_sched, + &llc_epoch_period); +debugfs_create_u32 ("llc_epoch_affinity_timeout" ,0644 , debugfs_sched, + &llc_epoch_affinity_timeout); +#endif +debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();kernel/sched/fair.c +3
-0@@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +__read_mostly unsigned int llc_aggr_tolerance =1 ; +__read_mostly unsigned int llc_epoch_period =EPOCH_PERIOD ; +__read_mostly unsigned int llc_epoch_affinity_timeout =EPOCH_LLC_AFFINITY_TIMEOUT ;static int llc_id (int cpu) {
kernel/sched/fair.c static int llc_id(int cpu) · line 1234
Quoted 9 lines
kernel/sched/fair.c +6
-0@@ -1223,11 +1226,22 @@ static int llc_id(int cpu)return llc; } +static inline int get_sched_cache_scale (int mul) +{ +if (!llc_aggr_tolerance) +return 0 ; + +if (llc_aggr_tolerance ==100 )
the range of llc_aggr_tolerance is [0, 100], so a little bug here? maybe check if (llc_aggr_tolerance >= 100)
and if llc_aggr_tolerance = 0, the func return 0, it means exceed_llc_capacity & exceed_llc_nr always true, there maybeinconsistent to have this value set while |llc_enable=1| is set.
Quoted 195 lines
kernel/sched/fair.c +47
-7+return INT_MAX ; + +return (1 + (llc_aggr_tolerance -1 ) * mul); +} +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) { +unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; -unsigned int llc;/* * get_cpu_cacheinfo_level() can not be used @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) rss =get_mm_counter (mm,MM_ANONPAGES ) +get_mm_counter (mm,MM_SHMEMPAGES ); -return (llc <= (rss *PAGE_SIZE )); +/* +* Scale the LLC size by 256*llc_aggr_tolerance +* and compare it to the task's RSS size. +* +* Suppose the L3 size is 32MB. If the +* llc_aggr_tolerance is 1: +* When the RSS is larger than 32MB, the process +* is regarded as exceeding the LLC capacity. If +* the llc_aggr_tolerance is 99: +* When the RSS is larger than 784GB, the process +* is regarded as exceeding the LLC capacity because: +* 784GB = (1 + (99 - 1) * 256) * 32MB +*/ + scale =get_sched_cache_scale (256 ); +if (scale ==INT_MAX ) +return false; + +return ((llc * scale) <= (rss *PAGE_SIZE )); }static bool exceed_llc_nr (struct mm_struct *mm,int cpu) { -int smt_nr =1 ; +int smt_nr =1 , scale;#ifdef CONFIG_SCHED_SMT if (sched_smt_active ()) smt_nr =cpumask_weight (cpu_smt_mask (cpu));#endif +/* +* Scale the Core number in a LLC by llc_aggr_tolerance +* and compare it to the task's active threads. +* +* Suppose the number of Cores in LLC is 8. +* Every core has 2 SMTs. +* If the llc_aggr_tolerance is 1: When the +* nr_running is larger than 8, the process +* is regarded as exceeding the LLC capacity. +* If the llc_aggr_tolerance is 99: +* When the nr_running is larger than 785, +* the process is regarded as exceeding +* the LLC capacity: +* 785 = 1 + (99 - 1) * 8 +*/ + scale =get_sched_cache_scale (1 ); +if (scale ==INT_MAX ) +return false; -return ((mm->nr_running_avg * smt_nr) >per_cpu (sd_llc_size, cpu)); +return ((mm->nr_running_avg * smt_nr) > (scale *per_cpu (sd_llc_size, cpu))); }static void account_llc_enqueue (struct rq *rq,struct task_struct *p)@@ -1350,9 +1399,9 @@ static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)long delta = now - rq->cpu_epoch_next;if (delta >0 ) { - n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + n = (delta + llc_epoch_period -1 ) / llc_epoch_period; rq->cpu_epoch += n; - rq->cpu_epoch_next += n *EPOCH_PERIOD ; + rq->cpu_epoch_next += n * llc_epoch_period;__shr_u64 (&rq->cpu_runtime, n); }@@ -1412,7 +1461,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* has only 1 thread, or has too many active threads, invalidate * its preferred state. */ -if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || +if (epoch -READ_ONCE (mm->mm_sched_epoch) > llc_epoch_affinity_timeout ||get_nr_threads (p) <=1 ||exceed_llc_nr (mm,cpu_of (rq)) ||exceed_llc_capacity (mm,cpu_of (rq))) {kernel/sched/sched.h +5
-0@@ -2852,6 +2852,11 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_enabled; +void sched_cache_set (bool locked);#endif #ifdef CONFIG_SCHED_HRTICK kernel/sched/topology.c +52
-2@@ -26,6 +26,49 @@ int max_llcs;static bool sched_cache_present; +unsigned int llc_enabled =1 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_on); + +/* +* Enable/disable cache aware scheduling according to +* user input and the presence of hardware support. +*/ +static void _sched_cache_set (bool enable,bool locked) +{ +if (enable) { +if (locked) +static_branch_enable_cpuslocked (&sched_cache_on); +else +static_branch_enable (&sched_cache_on); + }else { +if (locked) +static_branch_disable_cpuslocked (&sched_cache_on); +else +static_branch_disable (&sched_cache_on); + } +} + +void sched_cache_set (bool locked) +{ +/* hardware does not support */ +if (!sched_cache_present) { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + +return ; + } + +/* user wants it or not ?*/ +if (llc_enabled) { +if (!static_branch_likely (&sched_cache_on)) +_sched_cache_set (true, locked); + + }else { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + } +} +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) {unsigned int *new =NULL ;@@ -70,8 +113,12 @@ static int resize_llc_pref(bool has_multi_llcs)* new buffer. */ tmp_llc_pref = alloc_percpu_noprof(unsigned int *); -if (!tmp_llc_pref) -return -ENOMEM ; +if (!tmp_llc_pref) { + sched_cache_present = false; + ret = -ENOMEM ; + + goto out; + }for_each_present_cpu (i) *per_cpu_ptr (tmp_llc_pref, i) =NULL ;@@ -89,6 +136,7 @@ static int resize_llc_pref(bool has_multi_llcs) new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i));if (!new) { ret = -ENOMEM ; + sched_cache_present = false; goto release_old; }@@ -126,6 +174,8 @@ static int resize_llc_pref(bool has_multi_llcs)if (!ret) max_llcs = new_max_llcs; +out: +sched_cache_set (true);return ret; }
Re: [PATCH v2 21/23] -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing
In reply to Tim ChenQuoted 13 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
With cache-aware load balancing enabled, statistics related to its activity are exposed via
/proc/schedstatand debugfs. For instance, if users want to verify metrics like the number of exceeding RSS andnr_runninglimits, they can filter the output of/sys/kernel/debug/sched/debugand compute the required statistics manually:
Quoted 6 lines
llc_exceed_capSUM: 6llc_exceed_nrSUM: 4531Furthermore, these statistics exposed in
/proc/schedstatscan be queried manually or via perf sched stats[1] with minor modifications.
Hi Tim,
This patch looks great, especially for multithread Verilator workloads on clustered LLC (like AMD EPYC). I'm discussing with Verilator upstream to disable automatic userspace affinity assignment in Verilator if such feature exist [1]. During the discussion, I think there should be a way for userspace software to detect if such a feature exists. Could we expose it in /proc/schedstats to allow userspace software to detect such a feature? We can just use this patch and remove the "DO NOT APPLY" tag.
[1] https://github.com/verilator/verilator/issues/6826#issuecomment-3671287551
Thanks,
Yangyu Chen
Quoted 66 lines
Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
include/linux/sched/topology.h | 1 + kernel/sched/fair.c | 1 + kernel/sched/stats.c | 5 +++-- 3 files changed, 5 insertions(+), 2 deletions(-)include/linux/sched/topology.h +1
-0@@ -108,6 +108,7 @@ struct sched_domain {unsigned int lb_imbalance_util[CPU_MAX_IDLE_TYPES ];unsigned int lb_imbalance_task[CPU_MAX_IDLE_TYPES ];unsigned int lb_imbalance_misfit[CPU_MAX_IDLE_TYPES ]; +unsigned int lb_imbalance_llc[CPU_MAX_IDLE_TYPES ];unsigned int lb_gained[CPU_MAX_IDLE_TYPES ];unsigned int lb_hot_gained[CPU_MAX_IDLE_TYPES ];unsigned int lb_nobusyg[CPU_MAX_IDLE_TYPES ];kernel/sched/fair.c +0
-0@@ -12684,6 +12684,7 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sd__schedstat_add (sd->lb_imbalance_misfit[idle], env->imbalance);break ;case
migrate_llc_task:
+__schedstat_add(sd->lb_imbalance_llc[idle], env->imbalance); break;
}
}kernel/sched/stats.c +3
-2@@ -104,7 +104,7 @@ void __update_stats_enqueue_sleeper(struct rq *rq, struct task_struct *p,* Bump this up when changing the output format or the meaning of an existing * format, so that tools can adapt (or abort) */ -#define SCHEDSTAT_VERSION 17 +#define SCHEDSTAT_VERSION 18static int show_schedstat (struct seq_file *seq,void *v) {@@ -139,7 +139,7 @@ static int show_schedstat(struct seq_file *seq, void *v)seq_printf (seq,"domain%d %s %*pb" , dcount++, sd->name,cpumask_pr_args (sched_domain_span (sd)));for (itype =0 ; itype <CPU_MAX_IDLE_TYPES ; itype++) { - seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u", + seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u %u", sd->lb_count[itype], sd->lb_balanced[itype], sd->lb_failed[itype],@@ -147,6 +147,7 @@ static int show_schedstat(struct seq_file *seq, void *v) sd->lb_imbalance_util[itype], sd->lb_imbalance_task[itype], sd->lb_imbalance_misfit[itype], + sd->lb_imbalance_llc[itype], sd->lb_gained[itype], sd->lb_hot_gained[itype], sd->lb_nobusyq[itype],Signature · 1 line
2.32.0
Re: [PATCH v2 19/23] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
In reply to K Prateek NayakQuoted 32 lines · K Prateek Nayak
Hello Vern,
Quoted 11 lines · Vern Hao
Quoted 9 lines · Yu C
Quoted 8 lines · Vern Hao
Quoted 6 lines · Tim Chen
From: Chen Yu <[email protected]>
Prateek and Tingyin reported that memory-intensive workloads (such as stream) can saturate memory bandwidth and caches on the preferred LLC when
sched_cacheaggregates too many threads.Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
To mitigate this, estimate a process's memory footprint by comparing its RSS (anonymous and shared pages) to the size of the LLC. If RSS exceeds the LLC size, skip cache-aware scheduling.
Restricting RSS prevents many applications from benefiting from this optimization. I believe this restriction should be lifted. For memory- intensive workloads, the optimization may simply yield no gains, but it certainly shouldn't make performance worse. We need to further refine this logic.
Memory-intensive workloads may trigger performance regressions when memory bandwidth(from L3 cache to memory controller) is saturated due
RSS size and bandwidth saturation are not necessarily linked, In my view, the optimization should be robust enough that it doesn't cause a noticeable drop in performance, no matter how large the RSS is.
Easier said than done. I agree RSS size is not a clear indication of bandwidth saturation. With NUMA Balancing enabled, we can use the hinting faults to estimate the working set and make decisions but for systems that do not have NUMA, short of programming some performance counters, there is no real way to estimate the working set.
Hinting faults are known to cause overheads so enabling them without NUMA can cause noticeable overheads with no real benefits.
Quoted 9 lines
Quoted 1 line
We need to have a more profound discussion on this.
What do you have in mind?
From where I stand, having the RSS based bailout for now won't make
things worse for these tasks with huge memory reserves and when we can all agree on some generic method to estimate the working set of a task, we can always add it intoexceed_llc_capacity().
Prateek, thanks very much for the practical callouts - using RSS seems to be the best trade-off we can go with for now. Vern, I get your point about the concern between RSS and actual memory footprint. However, detecting the working set doesn’t seem to be accurate or generic in kernel space - even with NUMA fault statistics sampling. One reliable way I can think of to
detect the working set is in user space, via resctrl (Intel RDT, AMD QoS,
Arm MPAM). So maybe we can leverage that information to implement fine-grained control on a per-process or per-task basis later.
thanks,
Chenyu
Quoted 58 lines · Aaron Lu
On Wed, Dec 03, 2025 at 03:07:19PM -0800, Tim Chen wrote:
... ...Quoted 23 lines
Test results:
The patch series was applied and tested on v6.18-rc7.
See: https://github.com/timcchen1298/linux/commits/cache_aware_v2The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.
The second test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.
hackbench/schbench/netperf/stream/stress-ng/chacha20 were launched on these two platforms.
[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when the number of different active threads is below the capacity of a LLC. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows good throughput improvement.Quoted 3 lines
Genoa:
ChaCha20-xiangshan shows huge throughput improvement. No obvious difference is observed in hackbench/schbenchI think for small task number hackbench run, there should be some improvement.
I tried thread/pipe/2fds/1group, i.e. 4 tasks on Genoa:
./hackbench -T -f 2 -g 1 -p -l 2000000
And I noticed performance improved a lot:
(Result in seconds, less is better)llc_off llc_on diff time 4.755±1.6% 2.684±6.25% +43.6%
llc_offmeans/sys/kernel/debug/sched/llc_enabledset to 0 whilellc_onmeans/sys/kernel/debug/sched/llc_enabledset to 1, other tunnables are left unchanged.
Turbo is disabled and cpufreq set to performance.I also tried redis and noticed when I set io-threads to 4 in redis.conf, there is also some improvement on AMD Genoa:
llc_off manual diff llc_on diff throughput 1536727±0% 1737619±0% +13.1% 1737720±0% +13.1%Client cmdline:
numactl -N 1 redis-benchmark --threads 4 -t set -r 100000 -P 16 -n 10000000 server cmdline: numactl -N 0 redis-server ./redis.conf
I also tried to manually bind all tasks of redis server to a single LLC to see if this workload benefits from aggregation and that's what manual
means: taskset -c 8-15,200-207 redis-server ./redis.conf
Quoted 3 lines
According to the result, I think this 'cache aware scheduling' works as expected in that its performance is the same as manual binding; and they all beat
llc_off.
Thanks Aaron for the test!
thanks,
Chenyu
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Vern HaoQuoted 180 lines · Vern Hao
Quoted 178 lines · Tim Chen
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.For example, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped (784GB = (1 + (99 - 1) * 256) * 32MB).Similarly,
/sys/kernel/debug/sched/llc_aggr_tolerancealso controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with
nr_running> 8 are skipped.
- llc_aggr_tolerance=99 -> tasks withnr_running> 785 are skipped 785 = (1 + (99 - 1) * 8).(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides,llc_epoch_periodandllc_epoch_affinity_timeoutare also turned into tunable.Suggested-by K Prateek Nayak <[email protected]>Suggested-by Madadi Vineeth Reddy <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Suggested-by Tingyin Duan <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>---
Notes:
v1->v2: Remove thesmt_nrcheck infits_llc_capacity(). (Aaron Lu)include/linux/sched.h | 4 ++-
kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++
kernel/sched/topology.c| 54 +++++++++++++++++++++++++++++++++--5 files changed, 178 insertions(+), 10 deletions(-)include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable()) #ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); + static inline bool sched_cache_enabled(void) { - return false; + return static_branch_unlikely(&sched_cache_on); } #endifkernel/sched/debug.c +62
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &val)) \ + return -EINVAL; \ + if (val > (max)) \ + return -EINVAL; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 ); +#endif /* SCHED_CACHE */ + static ssize_t sched_scaling_write(struct file *filp, const char __user *ubuf, size_t cnt, loff_t *ppos) {@@ -523,6 +570,21 @@ static __init int sched_init_debug(void) debugfs_create_u32("hot_threshold_ms", 0644, numa, &sysctl_numa_balancing_hot_threshold); #endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE + debugfs_create_file("llc_overload_pct", 0644, debugfs_sched, NULL, + &sched_cache_fops_overload_pct); + debugfs_create_file("llc_imb_pct", 0644, debugfs_sched, NULL, + &sched_cache_fops_imb_pct); + debugfs_create_file("llc_aggr_tolerance", 0644, debugfs_sched, NULL, + &sched_cache_fops_aggr_tolerance); + debugfs_create_file("llc_enabled", 0644, debugfs_sched, NULL, + &sched_cache_fops_enabled); + debugfs_create_u32("llc_epoch_period", 0644, debugfs_sched, + &llc_epoch_period); + debugfs_create_u32("llc_epoch_affinity_timeout", 0644, debugfs_sched, + &llc_epoch_affinity_timeout); +#endif + debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops); debugfs_fair_server_init();kernel/sched/fair.c +3
-0@@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se) __read_mostly unsigned int llc_overload_pct = 50; __read_mostly unsigned int llc_imb_pct = 20; +__read_mostly unsigned int llc_aggr_tolerance = 1; +__read_mostly unsigned int llc_epoch_period = EPOCH_PERIOD; +__read_mostly unsigned int llc_epoch_affinity_timeout =EPOCH_LLC_AFFINITY_TIMEOUT ; static int llc_id(int cpu) {@@ -1223,11 +1226,22 @@ static int llc_id(int cpu) return llc;
Quoted 8 lines
kernel/sched/fair.c static int llc_id(int cpu) · line 1234
Quoted 7 lines
kernel/sched/fair.c +6
-0} +static inline int get_sched_cache_scale (int mul) +{ + if (!llc_aggr_tolerance) + return 0; + + if (llc_aggr_tolerance == 100)the range of
llc_aggr_toleranceis [0, 100], so a little bug here? maybe check if (llc_aggr_tolerance>= 100)
I thought llc_aggr_tolerance should not exceed 100, in sched_cache_write_aggr_tolerance(), if the input value is higher than max, it will return invalid:
return -EINVAL;
I did a double check on this:
root@vm:/sys/kernel/debug/sched# echo 100 > llc_aggr_tolerance root@vm:/sys/kernel/debug/sched# echo 101 > llc_aggr_tolerance bash: echo: write error: Invalid argument
Quoted 4 lines
and if
llc_aggr_tolerance= 0, the func return 0, it meansexceed_llc_capacity&exceed_llc_nralways true, there may be inconsistent to have this value set while |llc_enable=1| is set.
If the llc_aggr_tolerance is 0, the cache aware scheduling is supposed to be disabled - that is, exceed_llc_capacity() always returns true -> the process is not eligible for cache aware scheduling.
thanks,
Chenyu
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Vern HaoQuoted 8 lines · Vern Hao
Quoted 1 line · Tim Chen
From: Chen Yu <[email protected]>
the range of
llc_aggr_toleranceis [0, 100], so a little bug here? maybe check if (llc_aggr_tolerance>= 100)and if
llc_aggr_tolerance= 0, the func return 0, it meansexceed_llc_capacity&exceed_llc_nralways true, there maybeinconsistent to have this value set while |llc_enable=1| is set.
I see your point. The original idea was that llc_aggr_tolerance and llc_enable work together (independently) to determine whether cache-aware scheduling should be enabled. That is to say, llc_enable was not supposed be used as an indicator for users to query whether the actual cache-aware scheduling is enabled. Let me check if we can reset llc_enable if llc_aggr_tolerance is 0.
thanks,
Chenyu
Re: [PATCH v2 21/23] -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing
In reply to Yangyu ChenQuoted 22 lines · Yangyu Chen
Quoted 13 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
With cache-aware load balancing enabled, statistics related to its activity are exposed via
/proc/schedstatand debugfs. For instance, if users want to verify metrics like the number of exceeding RSS andnr_runninglimits, they can filter the output of/sys/kernel/debug/sched/debugand compute the required statistics manually:Quoted 6 lines
llc_exceed_capSUM: 6llc_exceed_nrSUM: 4531Furthermore, these statistics exposed in
/proc/schedstatscan be queried manually or via perf sched stats[1] with minor modifications.Hi Tim,
Quoted 9 lines
This patch looks great, especially for multithread Verilator workloads on clustered LLC (like AMD EPYC). I'm discussing with Verilator upstream to disable automatic userspace affinity assignment in Verilator if such feature exist [1]. During the discussion, I think there should be a way for userspace software to detect if such a feature exists. Could we expose it in
/proc/schedstatsto allow userspace software to detect such a feature? We can just use this patch and remove the "DO NOT APPLY" tag.
Thanks for the test Yangyu. Does /sys/kernel/debug/sched/llc_enabled work for you?
Anyway we can try to include /proc/schedstats as the formal one in the next version.
Thanks,
Chenyu
Quoted 7 lines
[1] https://github.com/verilator/verilator/issues/6826#issuecomment-3671287551
Thanks,
Yangyu Chen
Re: [PATCH v2 21/23] -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing
Quoted 33 lines · Yu C
Quoted 18 lines · Yangyu Chen
Quoted 12 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
With cache-aware load balancing enabled, statistics related to its activity are exposed via
/proc/schedstatand debugfs. For instance, if users want to verify metrics like the number of exceeding RSS andnr_runninglimits, they can filter the output of/sys/kernel/debug/sched/debugand compute the required statistics manually:Quoted 6 lines
llc_exceed_capSUM: 6llc_exceed_nrSUM: 4531Furthermore, these statistics exposed in
/proc/schedstatscan be queried manually or via perf sched stats[1] with minor modifications.Quoted 9 lines
Hi Tim,
This patch looks great, especially for multithread Verilator workloads on clustered LLC (like AMD EPYC). I'm discussing with Verilator upstream to disable automatic userspace affinity assignment in Verilator if such feature exist [1]. During the discussion, I think there should be a way for userspace software to detect if such a feature exists. Could we expose it in/proc/schedstatsto allow userspace software to detect such a feature? We can just use this patch and remove the "DO NOT APPLY" tag.
Quoted 2 lines
Thanks for the test Yangyu. Does
/sys/kernel/debug/sched/llc_enabledwork for you?
It requires debugfs being mounted with enough permissions. It’s not feasible for normal user-space software without root permission.
Thanks,
Yangyu Chen
Quoted 9 lines
Anyway we can try to include
/proc/schedstatsas the formal one in the next version.Thanks,
ChenyuQuoted 3 lines
[1] https://github.com/verilator/verilator/issues/6826#issuecomment-3671287551
Thanks,
Yangyu Chen
Re: [PATCH v2 19/23] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
In reply to K Prateek NayakQuoted 16 lines · K Prateek Nayak
Hello Vern,
Quoted 13 lines · Vern Hao
Quoted 12 lines · Yu C
Quoted 11 lines · Vern Hao
Quoted 9 lines · Tim Chen
From: Chen Yu <[email protected]>
Prateek and Tingyin reported that memory-intensive workloads (such as stream) can saturate memory bandwidth and caches on the preferred LLC when
sched_cacheaggregates too many threads.To mitigate this, estimate a process's memory footprint by comparing its RSS (anonymous and shared pages) to the size of the LLC. If RSS exceeds the LLC size, skip cache-aware scheduling.
Restricting RSS prevents many applications from benefiting from this optimization. I believe this restriction should be lifted. For memory- intensive workloads, the optimization may simply yield no gains, but it certainly shouldn't make performance worse. We need to further refine this logic.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
Memory-intensive workloads may trigger performance regressions when memory bandwidth(from L3 cache to memory controller) is saturated due
RSS size and bandwidth saturation are not necessarily linked, In my view, the optimization should be robust enough that it doesn't cause a noticeable drop in performance, no matter how large the RSS is.
Easier said than done. I agree RSS size is not a clear indication of bandwidth saturation. With NUMA Balancing enabled, we can use the hinting faults to estimate the working set and make decisions but for systems that do not have NUMA, short of programming some performance counters, there is no real way to estimate the working set.
I see the challenge, but the reality is that many production workloads have large memory footprints and deserve to see performance gains as well. In my testing with Chen Yu on STREAM, it's intriguing that the performance is fine without |llc_enable| but drops significantly once it's turned on.I sincerely hope this situation can be optimized; otherwise, we won't be able to utilize these optimizations in large-memory scenarios.
Quoted 6 lines
Hinting faults are known to cause overheads so enabling them without NUMA can cause noticeable overheads with no real benefits.
Quoted 1 line
We need to have a more profound discussion on this.
What do you have in mind?
I am wondering if we could address this through alternative approaches, such as reducing the migration frequency or preventing excessive task stacking within a single LLC. Of course, defining the right metrics to evaluate these conditions remains a significant challenge.
Quoted 6 lines
From where I stand, having the RSS based bailout for now won't make
things worse for these tasks with huge memory reserves and when we can all agree on some generic method to estimate the working set of a task, we can always add it intoexceed_llc_capacity().
Re: [PATCH v2 19/23] sched/cache: Avoid cache-aware scheduling for memory-heavy processes
In reply to Chen, Yu CQuoted 43 lines · Yu C
Quoted 32 lines · K Prateek Nayak
Hello Vern,
Quoted 11 lines · Vern Hao
Quoted 9 lines · Yu C
Quoted 8 lines · Vern Hao
Quoted 6 lines · Tim Chen
From: Chen Yu <[email protected]>
Prateek and Tingyin reported that memory-intensive workloads (such as stream) can saturate memory bandwidth and caches on the preferred LLC when
sched_cacheaggregates too many threads.Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
To mitigate this, estimate a process's memory footprint by comparing its RSS (anonymous and shared pages) to the size of the LLC. If RSS exceeds the LLC size, skip cache-aware scheduling.
Restricting RSS prevents many applications from benefiting from this optimization. I believe this restriction should be lifted. For memory- intensive workloads, the optimization may simply yield no gains, but it certainly shouldn't make performance worse. We need to further refine this logic.
Memory-intensive workloads may trigger performance regressions when memory bandwidth(from L3 cache to memory controller) is saturated due
RSS size and bandwidth saturation are not necessarily linked, In my view, the optimization should be robust enough that it doesn't cause a noticeable drop in performance, no matter how large the RSS is.
Easier said than done. I agree RSS size is not a clear indication of bandwidth saturation. With NUMA Balancing enabled, we can use the hinting faults to estimate the working set and make decisions but for systems that do not have NUMA, short of programming some performance counters, there is no real way to estimate the working set.
Hinting faults are known to cause overheads so enabling them without NUMA can cause noticeable overheads with no real benefits.
Quoted 9 lines
Quoted 1 line
We need to have a more profound discussion on this.
What do you have in mind?
From where I stand, having the RSS based bailout for now won't make things worse for these tasks with huge memory reserves and when we can all agree on some generic method to estimate the working set of a task, we can always add it into
exceed_llc_capacity().
Quoted 8 lines
Prateek, thanks very much for the practical callouts - using RSS seems to be the best trade-off we can go with for now. Vern, I get your point about the concern between RSS and actual memory footprint. However, detecting the working set doesn’t seem to be accurate or generic in kernel space - even with NUMA fault statistics sampling. One reliable way I can think of to detect the working set is in user space, via resctrl (Intel RDT, AMD QoS, Arm MPAM). So maybe we can leverage that information to implement fine-grained control on a per-process or per-task basis later.
OK, I agree, thanks.
Quoted 3 lines
thanks,
Chenyu
K Prateek Nayak
reviewerRe: [PATCH v2 04/23] sched/cache: Make LLC id continuous
In reply to Tim ChenHello Tim, Chenyu,
Quoted 13 lines · Tim Chen
(quoted diff) +13
-0+/* + * Assign continuous llc id for the CPU, and return + * the assigned llc id. + */ +static int update_llc_id(struct sched_domain *sd, + int cpu) +{ + int id = per_cpu(sd_llc_id, cpu), i; + + if (id >= 0) + return id; + + if (sd) {
Review
Quoted 10 lines
(quoted diff) +10
-0+ /* Look for any assigned id and reuse it.*/ + for_each_cpu(i, sched_domain_span(sd)) { + id = per_cpu(sd_llc_id, i); + + if (id >= 0) { + per_cpu(sd_llc_id, cpu) = id; + return id; + } + } + }
I don't really like tying this down to the sched_domain span since partition and other weirdness can cause the max_llc count to go unnecessarily high. The tl->mask() (from sched_domain_topology_level) should give the mask considering all online CPUs and not bothering about cpusets.
How about something like:
kernel/sched/core.c +12 -0
@@ -8270,6 +8270,18 @@ static void cpuset_cpu_active(void)static void cpuset_cpu_inactive (unsigned int cpu) {if (!cpuhp_tasks_frozen) { +/* +* This is necessary since offline CPUs are +* taken out of the tl->mask() and a newly +* onlined CPU in same LLC will not realize +* whether it should reuse the LLC ID owned +* by an offline CPU without knowing the +* LLC association. +* +* Safe to release the reference if this is +* the last CPU in the LLC going offline. +*/ +sched_domain_free_llc_id (cpu);cpuset_update_active_cpus (); }else { num_cpus_frozen++;
kernel/sched/debug.c +1 -0
@@ -631,6 +631,7 @@ void update_sched_domain_debugfs(void) i++; } +debugfs_create_u32 ("llc_id" ,0444 , d_cpu, (u32 *)per_cpu_ptr (&sd_llc_id, cpu));__cpumask_clear_cpu (cpu, sd_sysctl_cpus); } }
kernel/sched/sched.h +1 -0
@@ -2142,6 +2142,7 @@ extern int group_balance_cpu(struct sched_group *sg);extern void update_sched_domain_debugfs (void );extern void dirty_sched_domain_sysctl (int cpu); +void sched_domain_free_llc_id (int cpu);extern int sched_update_scaling (void );
kernel/sched/topology.c +102 -4
145 lines not on this page · raw message
---
AFAICT, "sd_llc_id" isn't compared across different partitions so having the CPUs that are actually associated with same physical LLC but across different partitions sharing the same "sd_llc_id" shouldn't be a problem.
Thoughts?
Signature · 2 lines
Thanks and Regards, Prateek
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Tim ChenQuoted 22 lines · Tim Chen
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.
Quoted 8 lines
Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.
Hi Chen Yu and Tim Chen,
Maybe we should have something like prctl(PR_LLC_AGGR_TOLERANCE, 100).
I have tested this version of the patch on my EPYC Milan 7V13 (7763 variant) physical machine, with 32M LLC for each 8-core CCX. I found that I need to tune "llc_aggr_tolerance" to 100, else I can't get cache-aware scheduling to work on Verilated [1] XiangShan [2] running the chacha20 [3] as I mentioned before [4].
But if I set it to 100, I will lose some performance on stream copy benchmarks since the bandwidth is limited per CCX. Thus, I think we should have a new prctl to let userspace software hint the kernel that this task can be bound by latency between cores, and should use this feature no matter the RSS exceed the LLC size.
I finally have an EPYC Milan at home today. I'm glad to test this patch further. And I'm very willing to have this thus we can have a fast verilator[1] without numactl manually.
[1] https://github.com/verilator/verilator
[2] https://github.com/OpenXiangShan/Xiangshan
[3] https://github.com/cyyself/chacha20-xiangshan
[4] https://lore.kernel.org/lkml/[email protected]/
Thanks,
Yangyu Chen
Quoted 360 lines
For exaImple, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped (784GB = (1 + (99 - 1) * 256) * 32MB).Similarly,
/sys/kernel/debug/sched/llc_aggr_tolerancealso controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with
nr_running> 8 are skipped.
- llc_aggr_tolerance=99 -> tasks withnr_running> 785 are skipped 785 = (1 + (99 - 1) * 8).(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides,llc_epoch_periodandllc_epoch_affinity_timeoutare also turned into tunable.Suggested-by K Prateek Nayak <[email protected]>Suggested-by Madadi Vineeth Reddy <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Suggested-by Tingyin Duan <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>---
Notes:
v1->v2: Remove the smt_nr check in fits_llc_capacity(). (Aaron Lu)include/linux/sched.h | 4 ++- kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++ kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- 5 files changed, 178 insertions(+), 10 deletions(-)include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +62
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &val)) \ + return -EINVAL; \ + if (val > (max)) \ + return -EINVAL; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 ); +#endif /* SCHED_CACHE */ +static ssize_t sched_scaling_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) {@@ -523,6 +570,21 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +debugfs_create_file ("llc_overload_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_overload_pct); +debugfs_create_file ("llc_imb_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_imb_pct); +debugfs_create_file ("llc_aggr_tolerance" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_tolerance); +debugfs_create_file ("llc_enabled" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_enabled); +debugfs_create_u32 ("llc_epoch_period" ,0644 , debugfs_sched, + &llc_epoch_period); +debugfs_create_u32 ("llc_epoch_affinity_timeout" ,0644 , debugfs_sched, + &llc_epoch_affinity_timeout); +#endif +debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();kernel/sched/fair.c +15
-1@@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +__read_mostly unsigned int llc_aggr_tolerance =1 ; +__read_mostly unsigned int llc_epoch_period =EPOCH_PERIOD ; +__read_mostly unsigned int llc_epoch_affinity_timeout =EPOCH_LLC_AFFINITY_TIMEOUT ;static int llc_id (int cpu) {@@ -1223,11 +1226,22 @@ static int llc_id(int cpu)return llc; } +static inline int get_sched_cache_scale (int mul) +{ +if (!llc_aggr_tolerance) +return 0 ; + +if (llc_aggr_tolerance ==100 ) +return INT_MAX ; + +return (1 + (llc_aggr_tolerance -1 ) * mul); +} +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) { +unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; -unsigned int llc;/* * get_cpu_cacheinfo_level() can not be used @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)rss =
get_mm_counter(mm, MM_ANONPAGES)+get_mm_counter(mm, MM_SHMEMPAGES);- return (llc <= (rss *
PAGE_SIZE));
+ /*
+ * Scale the LLC size by 256*llc_aggr_tolerance
+ * and compare it to the task's RSS size.
+ *
+ * Suppose the L3 size is 32MB. If the
+ *llc_aggr_toleranceis 1:
+ * When the RSS is larger than 32MB, the process
+ * is regarded as exceeding the LLC capacity. If
+ * thellc_aggr_toleranceis 99:
+ * When the RSS is larger than 784GB, the process
+ * is regarded as exceeding the LLC capacity because:
+ * 784GB = (1 + (99 - 1) * 256) * 32MB
+ */
+ scale =get_sched_cache_scale(256);
+ if (scale ==INT_MAX)
+ return false;
+
+ return ((llc * scale) <= (rss *PAGE_SIZE)); }static bool
exceed_llc_nr(struct mm_struct *mm, int cpu){
- intsmt_nr= 1;
+ intsmt_nr= 1, scale;#ifdef
CONFIG_SCHED_SMT
if (sched_smt_active())smt_nr= cpumask_weight(cpu_smt_mask(cpu)); #endif
+ /*
+ * Scale the Core number in a LLC byllc_aggr_tolerance
+ * and compare it to the task's active threads.
+ *
+ * Suppose the number of Cores in LLC is 8.
+ * Every core has 2 SMTs.
+ * If thellc_aggr_toleranceis 1: When the
+ *nr_runningis larger than 8, the process
+ * is regarded as exceeding the LLC capacity.
+ * If thellc_aggr_toleranceis 99:
+ * When thenr_runningis larger than 785,
+ * the process is regarded as exceeding
+ * the LLC capacity:
+ * 785 = 1 + (99 - 1) * 8
+ */
+ scale =get_sched_cache_scale(1);
+ if (scale ==INT_MAX)
+ return false;- return ((
mm->nr_running_avg*smt_nr) >per_cpu(sd_llc_size, cpu));
+ return ((mm->nr_running_avg*smt_nr) > (scale *per_cpu(sd_llc_size, cpu))); }static void
account_llc_enqueue(struct rq *rq, struct task_struct *p)kernel/sched/fair.c +3
-3@@ -1350,9 +1399,9 @@ static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)long delta = now - rq->cpu_epoch_next;if (delta >0 ) { - n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + n = (delta + llc_epoch_period -1 ) / llc_epoch_period; rq->cpu_epoch += n; - rq->cpu_epoch_next += n *EPOCH_PERIOD ; + rq->cpu_epoch_next += n * llc_epoch_period;__shr_u64 (&rq->cpu_runtime, n); }@@ -1412,7 +1461,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* has only 1 thread, or has too many active threads, invalidate * its preferred state. */ -if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || +if (epoch -READ_ONCE (mm->mm_sched_epoch) > llc_epoch_affinity_timeout ||get_nr_threads (p) <=1 ||exceed_llc_nr (mm,cpu_of (rq)) ||exceed_llc_capacity (mm,cpu_of (rq))) {kernel/sched/sched.h +5
-0@@ -2852,6 +2852,11 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_enabled; +void sched_cache_set (bool locked);#endif #ifdef CONFIG_SCHED_HRTICK kernel/sched/topology.c +52
-2@@ -26,6 +26,49 @@ int max_llcs;static bool sched_cache_present; +unsigned int llc_enabled =1 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_on); + +/* +* Enable/disable cache aware scheduling according to +* user input and the presence of hardware support. +*/ +static void _sched_cache_set (bool enable,bool locked) +{ +if (enable) { +if (locked) +static_branch_enable_cpuslocked (&sched_cache_on); +else +static_branch_enable (&sched_cache_on); + }else { +if (locked) +static_branch_disable_cpuslocked (&sched_cache_on); +else +static_branch_disable (&sched_cache_on); + } +} + +void sched_cache_set (bool locked) +{ +/* hardware does not support */ +if (!sched_cache_present) { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + +return ; + } + +/* user wants it or not ?*/ +if (llc_enabled) { +if (!static_branch_likely (&sched_cache_on)) +_sched_cache_set (true, locked); + + }else { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + } +} +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) {unsigned int *new =NULL ;@@ -70,8 +113,12 @@ static int resize_llc_pref(bool has_multi_llcs)* new buffer. */ tmp_llc_pref = alloc_percpu_noprof(unsigned int *); -if (!tmp_llc_pref) -return -ENOMEM ; +if (!tmp_llc_pref) { + sched_cache_present = false; + ret = -ENOMEM ; + + goto out; + }for_each_present_cpu (i) *per_cpu_ptr (tmp_llc_pref, i) =NULL ;@@ -89,6 +136,7 @@ static int resize_llc_pref(bool has_multi_llcs) new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i));if (!new) { ret = -ENOMEM ; + sched_cache_present = false; goto release_old; }@@ -126,6 +174,8 @@ static int resize_llc_pref(bool has_multi_llcs)if (!ret) max_llcs = new_max_llcs; +out: +sched_cache_set (true);return ret; }Signature · 1 line
2.32.0
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
Quoted 33 lines · Yangyu Chen
Quoted 21 lines · Tim Chen
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Quoted 8 lines
Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.
Quoted 6 lines
Hi Chen Yu and Tim Chen,
Maybe we should have something like
prctl(PR_LLC_AGGR_TOLERANCE, 100).I have tested this version of the patch on my EPYC Milan 7V13 (7763 variant) physical machine, with 32M LLC for each 8-core CCX. I found that I need to tune "
llc_aggr_tolerance" to 100, else I can't get cache-aware scheduling to work on Verilated [1] XiangShan [2] running the chacha20 [3] as I mentioned before [4].
In addition, I have investigated why this happens. And finally I realized that's because that workload observed 35596 kB RssAnon on my EPYC Milan Machine, slightly exceeding the LLC size (32M). I have tested it on an EPYC Genoa cloud server with the correct core / cache hierarchy in ACPI table, that shows 31700 kB RssAnon, thus fitting in LLC. I have no idea why my result shows higher RssAnon, since they both run Debian Trixie with the exact same kernel and same executable. But it reminds me we should have a userspace API for that.
Thanks,
Yangyu Chen
Quoted 780 lines
But if I set it to 100, I will lose some performance on stream copy benchmarks since the bandwidth is limited per CCX. Thus, I think we should have a new prctl to let userspace software hint the kernel that this task can be bound by latency between cores, and should use this feature no matter the RSS exceed the LLC size.
I finally have an EPYC Milan at home today. I'm glad to test this patch further. And I'm very willing to have this thus we can have a fast verilator[1] without numactl manually.
[1] https://github.com/verilator/verilator
[2] https://github.com/OpenXiangShan/Xiangshan
[3] https://github.com/cyyself/chacha20-xiangshan
[4] https://lore.kernel.org/lkml/[email protected]/Thanks,
Yangyu ChenQuoted 389 lines
For exaImple, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped (784GB = (1 + (99 - 1) * 256) * 32MB).Similarly,
/sys/kernel/debug/sched/llc_aggr_tolerancealso controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with
nr_running> 8 are skipped.
- llc_aggr_tolerance=99 -> tasks withnr_running> 785 are skipped 785 = (1 + (99 - 1) * 8).(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides,llc_epoch_periodandllc_epoch_affinity_timeoutare also turned into tunable.Suggested-by K Prateek Nayak <[email protected]>Suggested-by Madadi Vineeth Reddy <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Suggested-by Tingyin Duan <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>---
Notes:
v1->v2: Remove thesmt_nrcheck infits_llc_capacity().
(Aaron Lu)include/linux/sched.h | 4 ++- kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++ kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- 5 files changed, 178 insertions(+), 10 deletions(-)include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +62
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &val)) \ + return -EINVAL; \ + if (val > (max)) \ + return -EINVAL; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 ); +#endif /* SCHED_CACHE */ +static ssize_t sched_scaling_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) {@@ -523,6 +570,21 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +debugfs_create_file ("llc_overload_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_overload_pct); +debugfs_create_file ("llc_imb_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_imb_pct); +debugfs_create_file ("llc_aggr_tolerance" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_tolerance); +debugfs_create_file ("llc_enabled" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_enabled); +debugfs_create_u32 ("llc_epoch_period" ,0644 , debugfs_sched, + &llc_epoch_period); +debugfs_create_u32 ("llc_epoch_affinity_timeout" ,0644 , debugfs_sched, + &llc_epoch_affinity_timeout); +#endif +debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();kernel/sched/fair.c +15
-1@@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +__read_mostly unsigned int llc_aggr_tolerance =1 ; +__read_mostly unsigned int llc_epoch_period =EPOCH_PERIOD ; +__read_mostly unsigned int llc_epoch_affinity_timeout =EPOCH_LLC_AFFINITY_TIMEOUT ;static int llc_id (int cpu) {@@ -1223,11 +1226,22 @@ static int llc_id(int cpu)return llc; } +static inline int get_sched_cache_scale (int mul) +{ +if (!llc_aggr_tolerance) +return 0 ; + +if (llc_aggr_tolerance ==100 ) +return INT_MAX ; + +return (1 + (llc_aggr_tolerance -1 ) * mul); +} +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) { +unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; -unsigned int llc;/* * get_cpu_cacheinfo_level() can not be used @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)rss =
get_mm_counter(mm, MM_ANONPAGES)+get_mm_counter(mm, MM_SHMEMPAGES);- return (llc <= (rss *
PAGE_SIZE));
+ /*
+ * Scale the LLC size by 256*llc_aggr_tolerance
+ * and compare it to the task's RSS size.
+ *
+ * Suppose the L3 size is 32MB. If the
+ *llc_aggr_toleranceis 1:
+ * When the RSS is larger than 32MB, the process
+ * is regarded as exceeding the LLC capacity. If
+ * thellc_aggr_toleranceis 99:
+ * When the RSS is larger than 784GB, the process
+ * is regarded as exceeding the LLC capacity because:
+ * 784GB = (1 + (99 - 1) * 256) * 32MB
+ */
+ scale =get_sched_cache_scale(256);
+ if (scale ==INT_MAX)
+ return false;
+
+ return ((llc * scale) <= (rss *PAGE_SIZE)); }static bool
exceed_llc_nr(struct mm_struct *mm, int cpu){
- intsmt_nr= 1;
+ intsmt_nr= 1, scale;#ifdef
CONFIG_SCHED_SMT
if (sched_smt_active())smt_nr= cpumask_weight(cpu_smt_mask(cpu)); #endif
+ /*
+ * Scale the Core number in a LLC byllc_aggr_tolerance
+ * and compare it to the task's active threads.
+ *
+ * Suppose the number of Cores in LLC is 8.
+ * Every core has 2 SMTs.
+ * If thellc_aggr_toleranceis 1: When the
+ *nr_runningis larger than 8, the process
+ * is regarded as exceeding the LLC capacity.
+ * If thellc_aggr_toleranceis 99:
+ * When thenr_runningis larger than 785,
+ * the process is regarded as exceeding
+ * the LLC capacity:
+ * 785 = 1 + (99 - 1) * 8
+ */
+ scale =get_sched_cache_scale(1);
+ if (scale ==INT_MAX)
+ return false;- return ((
mm->nr_running_avg*smt_nr) >per_cpu(sd_llc_size, cpu));
+ return ((mm->nr_running_avg*smt_nr) > (scale *per_cpu(sd_llc_size, cpu))); }static void
account_llc_enqueue(struct rq *rq, struct task_struct *p)kernel/sched/fair.c +3
-3@@ -1350,9 +1399,9 @@ static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)long delta = now - rq->cpu_epoch_next;if (delta >0 ) { - n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + n = (delta + llc_epoch_period -1 ) / llc_epoch_period; rq->cpu_epoch += n; - rq->cpu_epoch_next += n *EPOCH_PERIOD ; + rq->cpu_epoch_next += n * llc_epoch_period;__shr_u64 (&rq->cpu_runtime, n); }@@ -1412,7 +1461,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* has only 1 thread, or has too many active threads, invalidate * its preferred state. */ -if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || +if (epoch -READ_ONCE (mm->mm_sched_epoch) > llc_epoch_affinity_timeout ||get_nr_threads (p) <=1 ||exceed_llc_nr (mm,cpu_of (rq)) ||exceed_llc_capacity (mm,cpu_of (rq))) {kernel/sched/sched.h +5
-0@@ -2852,6 +2852,11 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_enabled; +void sched_cache_set (bool locked);#endif #ifdef CONFIG_SCHED_HRTICK kernel/sched/topology.c +52
-2@@ -26,6 +26,49 @@ int max_llcs;static bool sched_cache_present; +unsigned int llc_enabled =1 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_on); + +/* +* Enable/disable cache aware scheduling according to +* user input and the presence of hardware support. +*/ +static void _sched_cache_set (bool enable,bool locked) +{ +if (enable) { +if (locked) +static_branch_enable_cpuslocked (&sched_cache_on); +else +static_branch_enable (&sched_cache_on); + }else { +if (locked) +static_branch_disable_cpuslocked (&sched_cache_on); +else +static_branch_disable (&sched_cache_on); + } +} + +void sched_cache_set (bool locked) +{ +/* hardware does not support */ +if (!sched_cache_present) { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + +return ; + } + +/* user wants it or not ?*/ +if (llc_enabled) { +if (!static_branch_likely (&sched_cache_on)) +_sched_cache_set (true, locked); + + }else { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + } +} +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) {unsigned int *new =NULL ;@@ -70,8 +113,12 @@ static int resize_llc_pref(bool has_multi_llcs)* new buffer. */ tmp_llc_pref = alloc_percpu_noprof(unsigned int *); -if (!tmp_llc_pref) -return -ENOMEM ; +if (!tmp_llc_pref) { + sched_cache_present = false; + ret = -ENOMEM ; + + goto out; + }for_each_present_cpu (i) *per_cpu_ptr (tmp_llc_pref, i) =NULL ;@@ -89,6 +136,7 @@ static int resize_llc_pref(bool has_multi_llcs) new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i));if (!new) { ret = -ENOMEM ; + sched_cache_present = false; goto release_old; }@@ -126,6 +174,8 @@ static int resize_llc_pref(bool has_multi_llcs)if (!ret) max_llcs = new_max_llcs; +out: +sched_cache_set (true);return ret; }Signature · 1 line
2.32.0On 4 Dec 2025, at 07:07, Tim Chen <[email protected]> wrote:
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.Hi Chen Yu and Tim Chen,
Maybe we should have something like
prctl(PR_LLC_AGGR_TOLERANCE, 100).I have tested this version of the patch on my EPYC Milan 7V13 (7763 variant) physical machine, with 32M LLC for each 8-core CCX. I found that I need to tune "
llc_aggr_tolerance" to 100, else I can't get cache-aware scheduling to work on Verilated [1] XiangShan [2] running the chacha20 [3] as I mentioned before [4].But if I set it to 100, I will lose some performance on stream copy benchmarks since the bandwidth is limited per CCX. Thus, I think we should have a new prctl to let userspace software hint the kernel that this task can be bound by latency between cores, and should use this feature no matter the RSS exceed the LLC size.
I finally have an EPYC Milan at home today. I'm glad to test this patch further. And I'm very willing to have this thus we can have a fast verilator[1] without numactl manually.
[1] https://github.com/verilator/verilator
[2] https://github.com/OpenXiangShan/Xiangshan
[3] https://github.com/cyyself/chacha20-xiangshan
[4] https://lore.kernel.org/lkml/[email protected]/Thanks,
Yangyu ChenQuoted 360 lines
For exaImple, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped (784GB = (1 + (99 - 1) * 256) * 32MB).Similarly,
/sys/kernel/debug/sched/llc_aggr_tolerancealso controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with
nr_running> 8 are skipped.
- llc_aggr_tolerance=99 -> tasks withnr_running> 785 are skipped 785 = (1 + (99 - 1) * 8).(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides,llc_epoch_periodandllc_epoch_affinity_timeoutare also turned into tunable.Suggested-by K Prateek Nayak <[email protected]>Suggested-by Madadi Vineeth Reddy <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Suggested-by Tingyin Duan <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>---
Notes:
v1->v2: Remove thesmt_nrcheck infits_llc_capacity().
(Aaron Lu)include/linux/sched.h | 4 ++- kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++ kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- 5 files changed, 178 insertions(+), 10 deletions(-)include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +62
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &val)) \ + return -EINVAL; \ + if (val > (max)) \ + return -EINVAL; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 ); +#endif /* SCHED_CACHE */ +static ssize_t sched_scaling_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) {@@ -523,6 +570,21 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +debugfs_create_file ("llc_overload_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_overload_pct); +debugfs_create_file ("llc_imb_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_imb_pct); +debugfs_create_file ("llc_aggr_tolerance" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_tolerance); +debugfs_create_file ("llc_enabled" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_enabled); +debugfs_create_u32 ("llc_epoch_period" ,0644 , debugfs_sched, + &llc_epoch_period); +debugfs_create_u32 ("llc_epoch_affinity_timeout" ,0644 , debugfs_sched, + &llc_epoch_affinity_timeout); +#endif +debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();kernel/sched/fair.c +15
-1@@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +__read_mostly unsigned int llc_aggr_tolerance =1 ; +__read_mostly unsigned int llc_epoch_period =EPOCH_PERIOD ; +__read_mostly unsigned int llc_epoch_affinity_timeout =EPOCH_LLC_AFFINITY_TIMEOUT ;static int llc_id (int cpu) {@@ -1223,11 +1226,22 @@ static int llc_id(int cpu)return llc; } +static inline int get_sched_cache_scale (int mul) +{ +if (!llc_aggr_tolerance) +return 0 ; + +if (llc_aggr_tolerance ==100 ) +return INT_MAX ; + +return (1 + (llc_aggr_tolerance -1 ) * mul); +} +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) { +unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; -unsigned int llc;/* * get_cpu_cacheinfo_level() can not be used @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)rss =
get_mm_counter(mm, MM_ANONPAGES)+get_mm_counter(mm, MM_SHMEMPAGES);- return (llc <= (rss *
PAGE_SIZE));
+ /*
+ * Scale the LLC size by 256*llc_aggr_tolerance
+ * and compare it to the task's RSS size.
+ *
+ * Suppose the L3 size is 32MB. If the
+ *llc_aggr_toleranceis 1:
+ * When the RSS is larger than 32MB, the process
+ * is regarded as exceeding the LLC capacity. If
+ * thellc_aggr_toleranceis 99:
+ * When the RSS is larger than 784GB, the process
+ * is regarded as exceeding the LLC capacity because:
+ * 784GB = (1 + (99 - 1) * 256) * 32MB
+ */
+ scale =get_sched_cache_scale(256);
+ if (scale ==INT_MAX)
+ return false;
+
+ return ((llc * scale) <= (rss *PAGE_SIZE)); }static bool
exceed_llc_nr(struct mm_struct *mm, int cpu){
- intsmt_nr= 1;
+ intsmt_nr= 1, scale;#ifdef
CONFIG_SCHED_SMT
if (sched_smt_active())smt_nr= cpumask_weight(cpu_smt_mask(cpu)); #endif
+ /*
+ * Scale the Core number in a LLC byllc_aggr_tolerance
+ * and compare it to the task's active threads.
+ *
+ * Suppose the number of Cores in LLC is 8.
+ * Every core has 2 SMTs.
+ * If thellc_aggr_toleranceis 1: When the
+ *nr_runningis larger than 8, the process
+ * is regarded as exceeding the LLC capacity.
+ * If thellc_aggr_toleranceis 99:
+ * When thenr_runningis larger than 785,
+ * the process is regarded as exceeding
+ * the LLC capacity:
+ * 785 = 1 + (99 - 1) * 8
+ */
+ scale =get_sched_cache_scale(1);
+ if (scale ==INT_MAX)
+ return false;- return ((
mm->nr_running_avg*smt_nr) >per_cpu(sd_llc_size, cpu));
+ return ((mm->nr_running_avg*smt_nr) > (scale *per_cpu(sd_llc_size, cpu))); }static void
account_llc_enqueue(struct rq *rq, struct task_struct *p)kernel/sched/fair.c +3
-3@@ -1350,9 +1399,9 @@ static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)long delta = now - rq->cpu_epoch_next;if (delta >0 ) { - n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + n = (delta + llc_epoch_period -1 ) / llc_epoch_period; rq->cpu_epoch += n; - rq->cpu_epoch_next += n *EPOCH_PERIOD ; + rq->cpu_epoch_next += n * llc_epoch_period;__shr_u64 (&rq->cpu_runtime, n); }@@ -1412,7 +1461,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* has only 1 thread, or has too many active threads, invalidate * its preferred state. */ -if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || +if (epoch -READ_ONCE (mm->mm_sched_epoch) > llc_epoch_affinity_timeout ||get_nr_threads (p) <=1 ||exceed_llc_nr (mm,cpu_of (rq)) ||exceed_llc_capacity (mm,cpu_of (rq))) {kernel/sched/sched.h +5
-0@@ -2852,6 +2852,11 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_enabled; +void sched_cache_set (bool locked);#endif #ifdef CONFIG_SCHED_HRTICK kernel/sched/topology.c +52
-2@@ -26,6 +26,49 @@ int max_llcs;static bool sched_cache_present; +unsigned int llc_enabled =1 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_on); + +/* +* Enable/disable cache aware scheduling according to +* user input and the presence of hardware support. +*/ +static void _sched_cache_set (bool enable,bool locked) +{ +if (enable) { +if (locked) +static_branch_enable_cpuslocked (&sched_cache_on); +else +static_branch_enable (&sched_cache_on); + }else { +if (locked) +static_branch_disable_cpuslocked (&sched_cache_on); +else +static_branch_disable (&sched_cache_on); + } +} + +void sched_cache_set (bool locked) +{ +/* hardware does not support */ +if (!sched_cache_present) { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + +return ; + } + +/* user wants it or not ?*/ +if (llc_enabled) { +if (!static_branch_likely (&sched_cache_on)) +_sched_cache_set (true, locked); + + }else { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + } +} +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) {unsigned int *new =NULL ;@@ -70,8 +113,12 @@ static int resize_llc_pref(bool has_multi_llcs)* new buffer. */ tmp_llc_pref = alloc_percpu_noprof(unsigned int *); -if (!tmp_llc_pref) -return -ENOMEM ; +if (!tmp_llc_pref) { + sched_cache_present = false; + ret = -ENOMEM ; + + goto out; + }for_each_present_cpu (i) *per_cpu_ptr (tmp_llc_pref, i) =NULL ;@@ -89,6 +136,7 @@ static int resize_llc_pref(bool has_multi_llcs) new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i));if (!new) { ret = -ENOMEM ; + sched_cache_present = false; goto release_old; }@@ -126,6 +174,8 @@ static int resize_llc_pref(bool has_multi_llcs)if (!ret) max_llcs = new_max_llcs; +out: +sched_cache_set (true);return ret; }Signature · 1 line
2.32.0
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
Quoted 52 lines · Yangyu Chen
Quoted 32 lines · Yangyu Chen
Quoted 21 lines · Tim Chen
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Quoted 8 lines
Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.Quoted 6 lines
Hi Chen Yu and Tim Chen,
Maybe we should have something like
prctl(PR_LLC_AGGR_TOLERANCE, 100).I have tested this version of the patch on my EPYC Milan 7V13 (7763 variant) physical machine, with 32M LLC for each 8-core CCX. I found that I need to tune "
llc_aggr_tolerance" to 100, else I can't get cache-aware scheduling to work on Verilated [1] XiangShan [2] running the chacha20 [3] as I mentioned before [4].In addition, I have investigated why this happens. And finally I realized that's because that workload observed 35596 kB RssAnon on my EPYC Milan Machine, slightly exceeding the LLC size (32M). I have tested it on an EPYC Genoa cloud server with the correct core / cache hierarchy in ACPI table, that shows 31700 kB RssAnon, thus fitting in LLC. I have no idea why my result shows higher RssAnon, since they both run Debian Trixie with the exact same kernel and same executable. But it reminds me we should have a userspace API for that.
Quoted 3 lines
Thanks,
Yangyu Chen
In addition, during profiling the verilator, I found that if scheduled to SMTs, it will result in poor performance. Thus, I think we should separate the control for rss size with the SMT scale.
It's notable that rss size is not the actual memory footprint. It would be better if we could measure the l2_miss event or l3_miss event to measure the l3 hit rate. Just for future work.
I'm willing to provide a patch for such a prctl. But I'm busy these days, maybe I can have the time to do that after one week.
Thanks,
Yangyu chen
Quoted 1,613 lines
Quoted 813 lines
But if I set it to 100, I will lose some performance on stream copy benchmarks since the bandwidth is limited per CCX. Thus, I think we should have a new prctl to let userspace software hint the kernel that this task can be bound by latency between cores, and should use this feature no matter the RSS exceed the LLC size.
I finally have an EPYC Milan at home today. I'm glad to test this patch further. And I'm very willing to have this thus we can have a fast verilator[1] without numactl manually.
[1] https://github.com/verilator/verilator
[2] https://github.com/OpenXiangShan/Xiangshan
[3] https://github.com/cyyself/chacha20-xiangshan
[4] https://lore.kernel.org/lkml/[email protected]/Thanks,
Yangyu ChenQuoted 389 lines
For exaImple, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped (784GB = (1 + (99 - 1) * 256) * 32MB).Similarly,
/sys/kernel/debug/sched/llc_aggr_tolerancealso controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with
nr_running> 8 are skipped.
- llc_aggr_tolerance=99 -> tasks withnr_running> 785 are skipped 785 = (1 + (99 - 1) * 8).(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides,llc_epoch_periodandllc_epoch_affinity_timeoutare also turned into tunable.Suggested-by K Prateek Nayak <[email protected]>Suggested-by Madadi Vineeth Reddy <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Suggested-by Tingyin Duan <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>---
Notes:
v1->v2: Remove thesmt_nrcheck infits_llc_capacity().
(Aaron Lu)include/linux/sched.h | 4 ++- kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++ kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- 5 files changed, 178 insertions(+), 10 deletions(-)include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +62
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &val)) \ + return -EINVAL; \ + if (val > (max)) \ + return -EINVAL; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 ); +#endif /* SCHED_CACHE */ +static ssize_t sched_scaling_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) {@@ -523,6 +570,21 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +debugfs_create_file ("llc_overload_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_overload_pct); +debugfs_create_file ("llc_imb_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_imb_pct); +debugfs_create_file ("llc_aggr_tolerance" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_tolerance); +debugfs_create_file ("llc_enabled" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_enabled); +debugfs_create_u32 ("llc_epoch_period" ,0644 , debugfs_sched, + &llc_epoch_period); +debugfs_create_u32 ("llc_epoch_affinity_timeout" ,0644 , debugfs_sched, + &llc_epoch_affinity_timeout); +#endif +debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();kernel/sched/fair.c +15
-1@@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +__read_mostly unsigned int llc_aggr_tolerance =1 ; +__read_mostly unsigned int llc_epoch_period =EPOCH_PERIOD ; +__read_mostly unsigned int llc_epoch_affinity_timeout =EPOCH_LLC_AFFINITY_TIMEOUT ;static int llc_id (int cpu) {@@ -1223,11 +1226,22 @@ static int llc_id(int cpu)return llc; } +static inline int get_sched_cache_scale (int mul) +{ +if (!llc_aggr_tolerance) +return 0 ; + +if (llc_aggr_tolerance ==100 ) +return INT_MAX ; + +return (1 + (llc_aggr_tolerance -1 ) * mul); +} +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) { +unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; -unsigned int llc;/* * get_cpu_cacheinfo_level() can not be used @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)rss =
get_mm_counter(mm, MM_ANONPAGES)+get_mm_counter(mm, MM_SHMEMPAGES);- return (llc <= (rss *
PAGE_SIZE));
+ /*
+ * Scale the LLC size by 256*llc_aggr_tolerance
+ * and compare it to the task's RSS size.
+ *
+ * Suppose the L3 size is 32MB. If the
+ *llc_aggr_toleranceis 1:
+ * When the RSS is larger than 32MB, the process
+ * is regarded as exceeding the LLC capacity. If
+ * thellc_aggr_toleranceis 99:
+ * When the RSS is larger than 784GB, the process
+ * is regarded as exceeding the LLC capacity because:
+ * 784GB = (1 + (99 - 1) * 256) * 32MB
+ */
+ scale =get_sched_cache_scale(256);
+ if (scale ==INT_MAX)
+ return false;
+
+ return ((llc * scale) <= (rss *PAGE_SIZE)); }static bool
exceed_llc_nr(struct mm_struct *mm, int cpu){
- intsmt_nr= 1;
+ intsmt_nr= 1, scale;#ifdef
CONFIG_SCHED_SMT
if (sched_smt_active())smt_nr= cpumask_weight(cpu_smt_mask(cpu)); #endif
+ /*
+ * Scale the Core number in a LLC byllc_aggr_tolerance
+ * and compare it to the task's active threads.
+ *
+ * Suppose the number of Cores in LLC is 8.
+ * Every core has 2 SMTs.
+ * If thellc_aggr_toleranceis 1: When the
+ *nr_runningis larger than 8, the process
+ * is regarded as exceeding the LLC capacity.
+ * If thellc_aggr_toleranceis 99:
+ * When thenr_runningis larger than 785,
+ * the process is regarded as exceeding
+ * the LLC capacity:
+ * 785 = 1 + (99 - 1) * 8
+ */
+ scale =get_sched_cache_scale(1);
+ if (scale ==INT_MAX)
+ return false;- return ((
mm->nr_running_avg*smt_nr) >per_cpu(sd_llc_size, cpu));
+ return ((mm->nr_running_avg*smt_nr) > (scale *per_cpu(sd_llc_size, cpu))); }static void
account_llc_enqueue(struct rq *rq, struct task_struct *p)kernel/sched/fair.c +3
-3@@ -1350,9 +1399,9 @@ static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)long delta = now - rq->cpu_epoch_next;if (delta >0 ) { - n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + n = (delta + llc_epoch_period -1 ) / llc_epoch_period; rq->cpu_epoch += n; - rq->cpu_epoch_next += n *EPOCH_PERIOD ; + rq->cpu_epoch_next += n * llc_epoch_period;__shr_u64 (&rq->cpu_runtime, n); }@@ -1412,7 +1461,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* has only 1 thread, or has too many active threads, invalidate * its preferred state. */ -if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || +if (epoch -READ_ONCE (mm->mm_sched_epoch) > llc_epoch_affinity_timeout ||get_nr_threads (p) <=1 ||exceed_llc_nr (mm,cpu_of (rq)) ||exceed_llc_capacity (mm,cpu_of (rq))) {kernel/sched/sched.h +5
-0@@ -2852,6 +2852,11 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_enabled; +void sched_cache_set (bool locked);#endif #ifdef CONFIG_SCHED_HRTICK kernel/sched/topology.c +52
-2@@ -26,6 +26,49 @@ int max_llcs;static bool sched_cache_present; +unsigned int llc_enabled =1 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_on); + +/* +* Enable/disable cache aware scheduling according to +* user input and the presence of hardware support. +*/ +static void _sched_cache_set (bool enable,bool locked) +{ +if (enable) { +if (locked) +static_branch_enable_cpuslocked (&sched_cache_on); +else +static_branch_enable (&sched_cache_on); + }else { +if (locked) +static_branch_disable_cpuslocked (&sched_cache_on); +else +static_branch_disable (&sched_cache_on); + } +} + +void sched_cache_set (bool locked) +{ +/* hardware does not support */ +if (!sched_cache_present) { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + +return ; + } + +/* user wants it or not ?*/ +if (llc_enabled) { +if (!static_branch_likely (&sched_cache_on)) +_sched_cache_set (true, locked); + + }else { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + } +} +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) {unsigned int *new =NULL ;@@ -70,8 +113,12 @@ static int resize_llc_pref(bool has_multi_llcs)* new buffer. */ tmp_llc_pref = alloc_percpu_noprof(unsigned int *); -if (!tmp_llc_pref) -return -ENOMEM ; +if (!tmp_llc_pref) { + sched_cache_present = false; + ret = -ENOMEM ; + + goto out; + }for_each_present_cpu (i) *per_cpu_ptr (tmp_llc_pref, i) =NULL ;@@ -89,6 +136,7 @@ static int resize_llc_pref(bool has_multi_llcs) new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i));if (!new) { ret = -ENOMEM ; + sched_cache_present = false; goto release_old; }@@ -126,6 +174,8 @@ static int resize_llc_pref(bool has_multi_llcs)if (!ret) max_llcs = new_max_llcs; +out: +sched_cache_set (true);return ret; }Signature · 1 line
2.32.0On 4 Dec 2025, at 07:07, Tim Chen <[email protected]> wrote:
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.Hi Chen Yu and Tim Chen,
Maybe we should have something like
prctl(PR_LLC_AGGR_TOLERANCE, 100).I have tested this version of the patch on my EPYC Milan 7V13 (7763 variant) physical machine, with 32M LLC for each 8-core CCX. I found that I need to tune "
llc_aggr_tolerance" to 100, else I can't get cache-aware scheduling to work on Verilated [1] XiangShan [2] running the chacha20 [3] as I mentioned before [4].But if I set it to 100, I will lose some performance on stream copy benchmarks since the bandwidth is limited per CCX. Thus, I think we should have a new prctl to let userspace software hint the kernel that this task can be bound by latency between cores, and should use this feature no matter the RSS exceed the LLC size.
I finally have an EPYC Milan at home today. I'm glad to test this patch further. And I'm very willing to have this thus we can have a fast verilator[1] without numactl manually.
[1] https://github.com/verilator/verilator
[2] https://github.com/OpenXiangShan/Xiangshan
[3] https://github.com/cyyself/chacha20-xiangshan
[4] https://lore.kernel.org/lkml/[email protected]/Thanks,
Yangyu ChenQuoted 360 lines
For exaImple, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped (784GB = (1 + (99 - 1) * 256) * 32MB).Similarly,
/sys/kernel/debug/sched/llc_aggr_tolerancealso controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with
nr_running> 8 are skipped.
- llc_aggr_tolerance=99 -> tasks withnr_running> 785 are skipped 785 = (1 + (99 - 1) * 8).(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides,llc_epoch_periodandllc_epoch_affinity_timeoutare also turned into tunable.Suggested-by K Prateek Nayak <[email protected]>Suggested-by Madadi Vineeth Reddy <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Suggested-by Tingyin Duan <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>---
Notes:
v1->v2: Remove thesmt_nrcheck infits_llc_capacity().
(Aaron Lu)include/linux/sched.h | 4 ++- kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++ kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- 5 files changed, 178 insertions(+), 10 deletions(-)include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +62
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &val)) \ + return -EINVAL; \ + if (val > (max)) \ + return -EINVAL; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 ); +#endif /* SCHED_CACHE */ +static ssize_t sched_scaling_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) {@@ -523,6 +570,21 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +debugfs_create_file ("llc_overload_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_overload_pct); +debugfs_create_file ("llc_imb_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_imb_pct); +debugfs_create_file ("llc_aggr_tolerance" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_tolerance); +debugfs_create_file ("llc_enabled" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_enabled); +debugfs_create_u32 ("llc_epoch_period" ,0644 , debugfs_sched, + &llc_epoch_period); +debugfs_create_u32 ("llc_epoch_affinity_timeout" ,0644 , debugfs_sched, + &llc_epoch_affinity_timeout); +#endif +debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();kernel/sched/fair.c +15
-1@@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +__read_mostly unsigned int llc_aggr_tolerance =1 ; +__read_mostly unsigned int llc_epoch_period =EPOCH_PERIOD ; +__read_mostly unsigned int llc_epoch_affinity_timeout =EPOCH_LLC_AFFINITY_TIMEOUT ;static int llc_id (int cpu) {@@ -1223,11 +1226,22 @@ static int llc_id(int cpu)return llc; } +static inline int get_sched_cache_scale (int mul) +{ +if (!llc_aggr_tolerance) +return 0 ; + +if (llc_aggr_tolerance ==100 ) +return INT_MAX ; + +return (1 + (llc_aggr_tolerance -1 ) * mul); +} +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) { +unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; -unsigned int llc;/* * get_cpu_cacheinfo_level() can not be used @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)rss =
get_mm_counter(mm, MM_ANONPAGES)+get_mm_counter(mm, MM_SHMEMPAGES);- return (llc <= (rss *
PAGE_SIZE));
+ /*
+ * Scale the LLC size by 256*llc_aggr_tolerance
+ * and compare it to the task's RSS size.
+ *
+ * Suppose the L3 size is 32MB. If the
+ *llc_aggr_toleranceis 1:
+ * When the RSS is larger than 32MB, the process
+ * is regarded as exceeding the LLC capacity. If
+ * thellc_aggr_toleranceis 99:
+ * When the RSS is larger than 784GB, the process
+ * is regarded as exceeding the LLC capacity because:
+ * 784GB = (1 + (99 - 1) * 256) * 32MB
+ */
+ scale =get_sched_cache_scale(256);
+ if (scale ==INT_MAX)
+ return false;
+
+ return ((llc * scale) <= (rss *PAGE_SIZE)); }static bool
exceed_llc_nr(struct mm_struct *mm, int cpu){
- intsmt_nr= 1;
+ intsmt_nr= 1, scale;#ifdef
CONFIG_SCHED_SMT
if (sched_smt_active())smt_nr= cpumask_weight(cpu_smt_mask(cpu)); #endif
+ /*
+ * Scale the Core number in a LLC byllc_aggr_tolerance
+ * and compare it to the task's active threads.
+ *
+ * Suppose the number of Cores in LLC is 8.
+ * Every core has 2 SMTs.
+ * If thellc_aggr_toleranceis 1: When the
+ *nr_runningis larger than 8, the process
+ * is regarded as exceeding the LLC capacity.
+ * If thellc_aggr_toleranceis 99:
+ * When thenr_runningis larger than 785,
+ * the process is regarded as exceeding
+ * the LLC capacity:
+ * 785 = 1 + (99 - 1) * 8
+ */
+ scale =get_sched_cache_scale(1);
+ if (scale ==INT_MAX)
+ return false;- return ((
mm->nr_running_avg*smt_nr) >per_cpu(sd_llc_size, cpu));
+ return ((mm->nr_running_avg*smt_nr) > (scale *per_cpu(sd_llc_size, cpu))); }static void
account_llc_enqueue(struct rq *rq, struct task_struct *p)kernel/sched/fair.c +3
-3@@ -1350,9 +1399,9 @@ static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)long delta = now - rq->cpu_epoch_next;if (delta >0 ) { - n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + n = (delta + llc_epoch_period -1 ) / llc_epoch_period; rq->cpu_epoch += n; - rq->cpu_epoch_next += n *EPOCH_PERIOD ; + rq->cpu_epoch_next += n * llc_epoch_period;__shr_u64 (&rq->cpu_runtime, n); }@@ -1412,7 +1461,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* has only 1 thread, or has too many active threads, invalidate * its preferred state. */ -if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || +if (epoch -READ_ONCE (mm->mm_sched_epoch) > llc_epoch_affinity_timeout ||get_nr_threads (p) <=1 ||exceed_llc_nr (mm,cpu_of (rq)) ||exceed_llc_capacity (mm,cpu_of (rq))) {kernel/sched/sched.h +5
-0@@ -2852,6 +2852,11 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_enabled; +void sched_cache_set (bool locked);#endif #ifdef CONFIG_SCHED_HRTICK kernel/sched/topology.c +52
-2@@ -26,6 +26,49 @@ int max_llcs;static bool sched_cache_present; +unsigned int llc_enabled =1 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_on); + +/* +* Enable/disable cache aware scheduling according to +* user input and the presence of hardware support. +*/ +static void _sched_cache_set (bool enable,bool locked) +{ +if (enable) { +if (locked) +static_branch_enable_cpuslocked (&sched_cache_on); +else +static_branch_enable (&sched_cache_on); + }else { +if (locked) +static_branch_disable_cpuslocked (&sched_cache_on); +else +static_branch_disable (&sched_cache_on); + } +} + +void sched_cache_set (bool locked) +{ +/* hardware does not support */ +if (!sched_cache_present) { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + +return ; + } + +/* user wants it or not ?*/ +if (llc_enabled) { +if (!static_branch_likely (&sched_cache_on)) +_sched_cache_set (true, locked); + + }else { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + } +} +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) {unsigned int *new =NULL ;@@ -70,8 +113,12 @@ static int resize_llc_pref(bool has_multi_llcs)* new buffer. */ tmp_llc_pref = alloc_percpu_noprof(unsigned int *); -if (!tmp_llc_pref) -return -ENOMEM ; +if (!tmp_llc_pref) { + sched_cache_present = false; + ret = -ENOMEM ; + + goto out; + }for_each_present_cpu (i) *per_cpu_ptr (tmp_llc_pref, i) =NULL ;@@ -89,6 +136,7 @@ static int resize_llc_pref(bool has_multi_llcs) new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i));if (!new) { ret = -ENOMEM ; + sched_cache_present = false; goto release_old; }@@ -126,6 +174,8 @@ static int resize_llc_pref(bool has_multi_llcs)if (!ret) max_llcs = new_max_llcs; +out: +sched_cache_set (true);return ret; }Signature · 1 line
2.32.0Quoted 20 lines · Yangyu Chen
On 4 Dec 2025, at 07:07, Tim Chen <[email protected]> wrote:
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Quoted 8 lines
Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.Quoted 6 lines
Hi Chen Yu and Tim Chen,
Maybe we should have something like
prctl(PR_LLC_AGGR_TOLERANCE, 100).I have tested this version of the patch on my EPYC Milan 7V13 (7763 variant) physical machine, with 32M LLC for each 8-core CCX. I found that I need to tune "
llc_aggr_tolerance" to 100, else I can't get cache-aware scheduling to work on Verilated [1] XiangShan [2] running the chacha20 [3] as I mentioned before [4].In addition, I have investigated why this happens. And finally I realized that's because that workload observed 35596 kB RssAnon on my EPYC Milan Machine, slightly exceeding the LLC size (32M). I have tested it on an EPYC Genoa cloud server with the correct core / cache hierarchy in ACPI table, that shows 31700 kB RssAnon, thus fitting in LLC. I have no idea why my result shows higher RssAnon, since they both run Debian Trixie with the exact same kernel and same executable. But it reminds me we should have a userspace API for that.
Thanks,
Yangyu ChenQuoted 780 lines
But if I set it to 100, I will lose some performance on stream copy benchmarks since the bandwidth is limited per CCX. Thus, I think we should have a new prctl to let userspace software hint the kernel that this task can be bound by latency between cores, and should use this feature no matter the RSS exceed the LLC size.
I finally have an EPYC Milan at home today. I'm glad to test this patch further. And I'm very willing to have this thus we can have a fast verilator[1] without numactl manually.
[1] https://github.com/verilator/verilator
[2] https://github.com/OpenXiangShan/Xiangshan
[3] https://github.com/cyyself/chacha20-xiangshan
[4] https://lore.kernel.org/lkml/[email protected]/Thanks,
Yangyu ChenQuoted 389 lines
For exaImple, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped (784GB = (1 + (99 - 1) * 256) * 32MB).Similarly,
/sys/kernel/debug/sched/llc_aggr_tolerancealso controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with
nr_running> 8 are skipped.
- llc_aggr_tolerance=99 -> tasks withnr_running> 785 are skipped 785 = (1 + (99 - 1) * 8).(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides,llc_epoch_periodandllc_epoch_affinity_timeoutare also turned into tunable.Suggested-by K Prateek Nayak <[email protected]>Suggested-by Madadi Vineeth Reddy <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Suggested-by Tingyin Duan <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>---
Notes:
v1->v2: Remove thesmt_nrcheck infits_llc_capacity().
(Aaron Lu)include/linux/sched.h | 4 ++- kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++ kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- 5 files changed, 178 insertions(+), 10 deletions(-)include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +62
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &val)) \ + return -EINVAL; \ + if (val > (max)) \ + return -EINVAL; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 ); +#endif /* SCHED_CACHE */ +static ssize_t sched_scaling_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) {@@ -523,6 +570,21 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +debugfs_create_file ("llc_overload_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_overload_pct); +debugfs_create_file ("llc_imb_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_imb_pct); +debugfs_create_file ("llc_aggr_tolerance" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_tolerance); +debugfs_create_file ("llc_enabled" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_enabled); +debugfs_create_u32 ("llc_epoch_period" ,0644 , debugfs_sched, + &llc_epoch_period); +debugfs_create_u32 ("llc_epoch_affinity_timeout" ,0644 , debugfs_sched, + &llc_epoch_affinity_timeout); +#endif +debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();kernel/sched/fair.c +15
-1@@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +__read_mostly unsigned int llc_aggr_tolerance =1 ; +__read_mostly unsigned int llc_epoch_period =EPOCH_PERIOD ; +__read_mostly unsigned int llc_epoch_affinity_timeout =EPOCH_LLC_AFFINITY_TIMEOUT ;static int llc_id (int cpu) {@@ -1223,11 +1226,22 @@ static int llc_id(int cpu)return llc; } +static inline int get_sched_cache_scale (int mul) +{ +if (!llc_aggr_tolerance) +return 0 ; + +if (llc_aggr_tolerance ==100 ) +return INT_MAX ; + +return (1 + (llc_aggr_tolerance -1 ) * mul); +} +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) { +unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; -unsigned int llc;/* * get_cpu_cacheinfo_level() can not be used @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)rss =
get_mm_counter(mm, MM_ANONPAGES)+get_mm_counter(mm, MM_SHMEMPAGES);- return (llc <= (rss *
PAGE_SIZE));
+ /*
+ * Scale the LLC size by 256*llc_aggr_tolerance
+ * and compare it to the task's RSS size.
+ *
+ * Suppose the L3 size is 32MB. If the
+ *llc_aggr_toleranceis 1:
+ * When the RSS is larger than 32MB, the process
+ * is regarded as exceeding the LLC capacity. If
+ * thellc_aggr_toleranceis 99:
+ * When the RSS is larger than 784GB, the process
+ * is regarded as exceeding the LLC capacity because:
+ * 784GB = (1 + (99 - 1) * 256) * 32MB
+ */
+ scale =get_sched_cache_scale(256);
+ if (scale ==INT_MAX)
+ return false;
+
+ return ((llc * scale) <= (rss *PAGE_SIZE)); }static bool
exceed_llc_nr(struct mm_struct *mm, int cpu){
- intsmt_nr= 1;
+ intsmt_nr= 1, scale;#ifdef
CONFIG_SCHED_SMT
if (sched_smt_active())smt_nr= cpumask_weight(cpu_smt_mask(cpu)); #endif
+ /*
+ * Scale the Core number in a LLC byllc_aggr_tolerance
+ * and compare it to the task's active threads.
+ *
+ * Suppose the number of Cores in LLC is 8.
+ * Every core has 2 SMTs.
+ * If thellc_aggr_toleranceis 1: When the
+ *nr_runningis larger than 8, the process
+ * is regarded as exceeding the LLC capacity.
+ * If thellc_aggr_toleranceis 99:
+ * When thenr_runningis larger than 785,
+ * the process is regarded as exceeding
+ * the LLC capacity:
+ * 785 = 1 + (99 - 1) * 8
+ */
+ scale =get_sched_cache_scale(1);
+ if (scale ==INT_MAX)
+ return false;- return ((
mm->nr_running_avg*smt_nr) >per_cpu(sd_llc_size, cpu));
+ return ((mm->nr_running_avg*smt_nr) > (scale *per_cpu(sd_llc_size, cpu))); }static void
account_llc_enqueue(struct rq *rq, struct task_struct *p)kernel/sched/fair.c +3
-3@@ -1350,9 +1399,9 @@ static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)long delta = now - rq->cpu_epoch_next;if (delta >0 ) { - n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + n = (delta + llc_epoch_period -1 ) / llc_epoch_period; rq->cpu_epoch += n; - rq->cpu_epoch_next += n *EPOCH_PERIOD ; + rq->cpu_epoch_next += n * llc_epoch_period;__shr_u64 (&rq->cpu_runtime, n); }@@ -1412,7 +1461,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* has only 1 thread, or has too many active threads, invalidate * its preferred state. */ -if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || +if (epoch -READ_ONCE (mm->mm_sched_epoch) > llc_epoch_affinity_timeout ||get_nr_threads (p) <=1 ||exceed_llc_nr (mm,cpu_of (rq)) ||exceed_llc_capacity (mm,cpu_of (rq))) {kernel/sched/sched.h +5
-0@@ -2852,6 +2852,11 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_enabled; +void sched_cache_set (bool locked);#endif #ifdef CONFIG_SCHED_HRTICK kernel/sched/topology.c +52
-2@@ -26,6 +26,49 @@ int max_llcs;static bool sched_cache_present; +unsigned int llc_enabled =1 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_on); + +/* +* Enable/disable cache aware scheduling according to +* user input and the presence of hardware support. +*/ +static void _sched_cache_set (bool enable,bool locked) +{ +if (enable) { +if (locked) +static_branch_enable_cpuslocked (&sched_cache_on); +else +static_branch_enable (&sched_cache_on); + }else { +if (locked) +static_branch_disable_cpuslocked (&sched_cache_on); +else +static_branch_disable (&sched_cache_on); + } +} + +void sched_cache_set (bool locked) +{ +/* hardware does not support */ +if (!sched_cache_present) { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + +return ; + } + +/* user wants it or not ?*/ +if (llc_enabled) { +if (!static_branch_likely (&sched_cache_on)) +_sched_cache_set (true, locked); + + }else { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + } +} +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) {unsigned int *new =NULL ;@@ -70,8 +113,12 @@ static int resize_llc_pref(bool has_multi_llcs)* new buffer. */ tmp_llc_pref = alloc_percpu_noprof(unsigned int *); -if (!tmp_llc_pref) -return -ENOMEM ; +if (!tmp_llc_pref) { + sched_cache_present = false; + ret = -ENOMEM ; + + goto out; + }for_each_present_cpu (i) *per_cpu_ptr (tmp_llc_pref, i) =NULL ;@@ -89,6 +136,7 @@ static int resize_llc_pref(bool has_multi_llcs) new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i));if (!new) { ret = -ENOMEM ; + sched_cache_present = false; goto release_old; }@@ -126,6 +174,8 @@ static int resize_llc_pref(bool has_multi_llcs)if (!ret) max_llcs = new_max_llcs; +out: +sched_cache_set (true);return ret; }Signature · 1 line
2.32.0On 4 Dec 2025, at 07:07, Tim Chen <[email protected]> wrote:
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.Hi Chen Yu and Tim Chen,
Maybe we should have something like
prctl(PR_LLC_AGGR_TOLERANCE, 100).I have tested this version of the patch on my EPYC Milan 7V13 (7763 variant) physical machine, with 32M LLC for each 8-core CCX. I found that I need to tune "
llc_aggr_tolerance" to 100, else I can't get cache-aware scheduling to work on Verilated [1] XiangShan [2] running the chacha20 [3] as I mentioned before [4].But if I set it to 100, I will lose some performance on stream copy benchmarks since the bandwidth is limited per CCX. Thus, I think we should have a new prctl to let userspace software hint the kernel that this task can be bound by latency between cores, and should use this feature no matter the RSS exceed the LLC size.
I finally have an EPYC Milan at home today. I'm glad to test this patch further. And I'm very willing to have this thus we can have a fast verilator[1] without numactl manually.
[1] https://github.com/verilator/verilator
[2] https://github.com/OpenXiangShan/Xiangshan
[3] https://github.com/cyyself/chacha20-xiangshan
[4] https://lore.kernel.org/lkml/[email protected]/Thanks,
Yangyu ChenQuoted 360 lines
For exaImple, with a 32MB L3 cache:
- llc_aggr_tolerance=1 -> tasks with RSS > 32MB are skipped.
- llc_aggr_tolerance=99 -> tasks with RSS > 784GB are skipped (784GB = (1 + (99 - 1) * 256) * 32MB).Similarly,
/sys/kernel/debug/sched/llc_aggr_tolerancealso controls how strictly the number of active threads is considered when doing cache aware load balance. The number of SMTs is also considered. High SMT counts reduce the aggregation capacity, preventing excessive task aggregation on SMT-heavy systems like Power10/Power11.For example, with 8 Cores/16 CPUs in a L3:
- llc_aggr_tolerance=1 -> tasks with
nr_running> 8 are skipped.
- llc_aggr_tolerance=99 -> tasks withnr_running> 785 are skipped 785 = (1 + (99 - 1) * 8).(3) llc_epoch_period/llc_epoch_affinity_timeout
Besides,llc_epoch_periodandllc_epoch_affinity_timeoutare also turned into tunable.Suggested-by K Prateek Nayak <[email protected]>Suggested-by Madadi Vineeth Reddy <[email protected]>Suggested-by Shrikanth Hegde <[email protected]>Suggested-by Tingyin Duan <[email protected]>Co-developed-by Tim Chen <[email protected]>Signed-off-by Tim Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>---
Notes:
v1->v2: Remove thesmt_nrcheck infits_llc_capacity().
(Aaron Lu)include/linux/sched.h | 4 ++- kernel/sched/debug.c | 62 ++++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 63 ++++++++++++++++++++++++++++++++++++----- kernel/sched/sched.h | 5 ++++ kernel/sched/topology.c | 54 +++++++++++++++++++++++++++++++++-- 5 files changed, 178 insertions(+), 10 deletions(-)include/linux/sched.h +3
-1@@ -2436,9 +2436,11 @@ extern void migrate_enable(void); DEFINE_LOCK_GUARD_0(migrate, migrate_disable(), migrate_enable())#ifdef CONFIG_SCHED_CACHE +DECLARE_STATIC_KEY_FALSE (sched_cache_on); +static inline bool sched_cache_enabled (void ) { -return false; +return static_branch_unlikely (&sched_cache_on); }#endif kernel/sched/debug.c +62
-0@@ -169,6 +169,53 @@ static const struct file_operations sched_feat_fops = { .release = single_release, }; +#ifdef CONFIG_SCHED_CACHE +#define SCHED_CACHE_CREATE_CONTROL (name, max) \ +static ssize_t sched_cache_write_##name(struct file *filp, \ + const char __user *ubuf, \ + size_t cnt, loff_t *ppos) \ +{ \ + char buf[16]; \ + unsigned int val; \ + if (cnt > 15) \ + cnt = 15; \ + if (copy_from_user(&buf, ubuf, cnt)) \ + return -EFAULT; \ + buf[cnt] = '\0'; \ + if (kstrtouint(buf, 10, &val)) \ + return -EINVAL; \ + if (val > (max)) \ + return -EINVAL; \ + llc_##name = val; \ + if (!strcmp(#name, "enabled")) \ + sched_cache_set(false); \ + *ppos += cnt; \ + return cnt; \ +} \ +static int sched_cache_show_##name(struct seq_file *m, void *v) \ +{ \ + seq_printf(m, "%d\n", llc_##name); \ + return 0; \ +} \ +static int sched_cache_open_##name(struct inode *inode, \ + struct file *filp) \ +{ \ + return single_open(filp, sched_cache_show_##name, NULL); \ +} \ +static const struct file_operations sched_cache_fops_##name = { \ + .open = sched_cache_open_##name, \ + .write = sched_cache_write_##name, \ + .read = seq_read, \ + .llseek = seq_lseek, \ + .release = single_release, \ +} + +SCHED_CACHE_CREATE_CONTROL (overload_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (imb_pct,100 ); +SCHED_CACHE_CREATE_CONTROL (aggr_tolerance,100 ); +SCHED_CACHE_CREATE_CONTROL (enabled,1 ); +#endif /* SCHED_CACHE */ +static ssize_t sched_scaling_write (struct file *filp,const char __user *ubuf,size_t cnt,loff_t *ppos) {@@ -523,6 +570,21 @@ static __init int sched_init_debug(void)debugfs_create_u32 ("hot_threshold_ms" ,0644 , numa, &sysctl_numa_balancing_hot_threshold);#endif /* CONFIG_NUMA_BALANCING */ +#ifdef CONFIG_SCHED_CACHE +debugfs_create_file ("llc_overload_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_overload_pct); +debugfs_create_file ("llc_imb_pct" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_imb_pct); +debugfs_create_file ("llc_aggr_tolerance" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_aggr_tolerance); +debugfs_create_file ("llc_enabled" ,0644 , debugfs_sched,NULL , + &sched_cache_fops_enabled); +debugfs_create_u32 ("llc_epoch_period" ,0644 , debugfs_sched, + &llc_epoch_period); +debugfs_create_u32 ("llc_epoch_affinity_timeout" ,0644 , debugfs_sched, + &llc_epoch_affinity_timeout); +#endif +debugfs_create_file ("debug" ,0444 , debugfs_sched,NULL , &sched_debug_fops);debugfs_fair_server_init ();kernel/sched/fair.c +15
-1@@ -1207,6 +1207,9 @@ static s64 update_se(struct rq *rq, struct sched_entity *se)__read_mostly unsigned int llc_overload_pct =50 ;__read_mostly unsigned int llc_imb_pct =20 ; +__read_mostly unsigned int llc_aggr_tolerance =1 ; +__read_mostly unsigned int llc_epoch_period =EPOCH_PERIOD ; +__read_mostly unsigned int llc_epoch_affinity_timeout =EPOCH_LLC_AFFINITY_TIMEOUT ;static int llc_id (int cpu) {@@ -1223,11 +1226,22 @@ static int llc_id(int cpu)return llc; } +static inline int get_sched_cache_scale (int mul) +{ +if (!llc_aggr_tolerance) +return 0 ; + +if (llc_aggr_tolerance ==100 ) +return INT_MAX ; + +return (1 + (llc_aggr_tolerance -1 ) * mul); +} +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) { +unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; -unsigned int llc;/* * get_cpu_cacheinfo_level() can not be used @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)rss =
get_mm_counter(mm, MM_ANONPAGES)+get_mm_counter(mm, MM_SHMEMPAGES);- return (llc <= (rss *
PAGE_SIZE));
+ /*
+ * Scale the LLC size by 256*llc_aggr_tolerance
+ * and compare it to the task's RSS size.
+ *
+ * Suppose the L3 size is 32MB. If the
+ *llc_aggr_toleranceis 1:
+ * When the RSS is larger than 32MB, the process
+ * is regarded as exceeding the LLC capacity. If
+ * thellc_aggr_toleranceis 99:
+ * When the RSS is larger than 784GB, the process
+ * is regarded as exceeding the LLC capacity because:
+ * 784GB = (1 + (99 - 1) * 256) * 32MB
+ */
+ scale =get_sched_cache_scale(256);
+ if (scale ==INT_MAX)
+ return false;
+
+ return ((llc * scale) <= (rss *PAGE_SIZE)); }static bool
exceed_llc_nr(struct mm_struct *mm, int cpu){
- intsmt_nr= 1;
+ intsmt_nr= 1, scale;#ifdef
CONFIG_SCHED_SMT
if (sched_smt_active())smt_nr= cpumask_weight(cpu_smt_mask(cpu)); #endif
+ /*
+ * Scale the Core number in a LLC byllc_aggr_tolerance
+ * and compare it to the task's active threads.
+ *
+ * Suppose the number of Cores in LLC is 8.
+ * Every core has 2 SMTs.
+ * If thellc_aggr_toleranceis 1: When the
+ *nr_runningis larger than 8, the process
+ * is regarded as exceeding the LLC capacity.
+ * If thellc_aggr_toleranceis 99:
+ * When thenr_runningis larger than 785,
+ * the process is regarded as exceeding
+ * the LLC capacity:
+ * 785 = 1 + (99 - 1) * 8
+ */
+ scale =get_sched_cache_scale(1);
+ if (scale ==INT_MAX)
+ return false;- return ((
mm->nr_running_avg*smt_nr) >per_cpu(sd_llc_size, cpu));
+ return ((mm->nr_running_avg*smt_nr) > (scale *per_cpu(sd_llc_size, cpu))); }static void
account_llc_enqueue(struct rq *rq, struct task_struct *p)kernel/sched/fair.c +3
-3@@ -1350,9 +1399,9 @@ static inline void __update_mm_sched(struct rq *rq, struct mm_sched *pcpu_sched)long delta = now - rq->cpu_epoch_next;if (delta >0 ) { - n = (delta +EPOCH_PERIOD -1 ) /EPOCH_PERIOD ; + n = (delta + llc_epoch_period -1 ) / llc_epoch_period; rq->cpu_epoch += n; - rq->cpu_epoch_next += n *EPOCH_PERIOD ; + rq->cpu_epoch_next += n * llc_epoch_period;__shr_u64 (&rq->cpu_runtime, n); }@@ -1412,7 +1461,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* has only 1 thread, or has too many active threads, invalidate * its preferred state. */ -if (epoch -READ_ONCE (mm->mm_sched_epoch) >EPOCH_LLC_AFFINITY_TIMEOUT || +if (epoch -READ_ONCE (mm->mm_sched_epoch) > llc_epoch_affinity_timeout ||get_nr_threads (p) <=1 ||exceed_llc_nr (mm,cpu_of (rq)) ||exceed_llc_capacity (mm,cpu_of (rq))) {kernel/sched/sched.h +5
-0@@ -2852,6 +2852,11 @@ extern unsigned int sysctl_numa_balancing_hot_threshold;#ifdef CONFIG_SCHED_CACHE extern unsigned int llc_overload_pct;extern unsigned int llc_imb_pct; +extern unsigned int llc_aggr_tolerance; +extern unsigned int llc_epoch_period; +extern unsigned int llc_epoch_affinity_timeout; +extern unsigned int llc_enabled; +void sched_cache_set (bool locked);#endif #ifdef CONFIG_SCHED_HRTICK kernel/sched/topology.c +52
-2@@ -26,6 +26,49 @@ int max_llcs;static bool sched_cache_present; +unsigned int llc_enabled =1 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_on); + +/* +* Enable/disable cache aware scheduling according to +* user input and the presence of hardware support. +*/ +static void _sched_cache_set (bool enable,bool locked) +{ +if (enable) { +if (locked) +static_branch_enable_cpuslocked (&sched_cache_on); +else +static_branch_enable (&sched_cache_on); + }else { +if (locked) +static_branch_disable_cpuslocked (&sched_cache_on); +else +static_branch_disable (&sched_cache_on); + } +} + +void sched_cache_set (bool locked) +{ +/* hardware does not support */ +if (!sched_cache_present) { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + +return ; + } + +/* user wants it or not ?*/ +if (llc_enabled) { +if (!static_branch_likely (&sched_cache_on)) +_sched_cache_set (true, locked); + + }else { +if (static_branch_likely (&sched_cache_on)) +_sched_cache_set (false, locked); + } +} +static unsigned int *alloc_new_pref_llcs (unsigned int *old,unsigned int **gc) {unsigned int *new =NULL ;@@ -70,8 +113,12 @@ static int resize_llc_pref(bool has_multi_llcs)* new buffer. */ tmp_llc_pref = alloc_percpu_noprof(unsigned int *); -if (!tmp_llc_pref) -return -ENOMEM ; +if (!tmp_llc_pref) { + sched_cache_present = false; + ret = -ENOMEM ; + + goto out; + }for_each_present_cpu (i) *per_cpu_ptr (tmp_llc_pref, i) =NULL ;@@ -89,6 +136,7 @@ static int resize_llc_pref(bool has_multi_llcs) new =alloc_new_pref_llcs (rq->nr_pref_llc,per_cpu_ptr (tmp_llc_pref, i));if (!new) { ret = -ENOMEM ; + sched_cache_present = false; goto release_old; }@@ -126,6 +174,8 @@ static int resize_llc_pref(bool has_multi_llcs)if (!ret) max_llcs = new_max_llcs; +out: +sched_cache_set (true);return ret; }Signature · 1 line
2.32.0
Re: [PATCH v2 04/23] sched/cache: Make LLC id continuous
In reply to K Prateek NayakHello Prateek,
Quoted 24 lines · K Prateek Nayak
Hello Tim, Chenyu,
Quoted 21 lines · Tim Chen
(quoted diff) +21
-0+/* + * Assign continuous llc id for the CPU, and return + * the assigned llc id. + */ +static int update_llc_id(struct sched_domain *sd, + int cpu) +{ + int id = per_cpu(sd_llc_id, cpu), i; + + if (id >= 0) + return id; + + if (sd) { + /* Look for any assigned id and reuse it.*/ + for_each_cpu(i, sched_domain_span(sd)) { + id = per_cpu(sd_llc_id, i); + + if (id >= 0) { + per_cpu(sd_llc_id, cpu) = id; + return id; + }
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ } + }I don't really like tying this down to the
sched_domainspan since partition and other weirdness can cause themax_llccount to go unnecessarily high. Thetl->mask()(fromsched_domain_topology_level) should give the mask considering all online CPUs and not bothering about cpusets.
OK, using the topology_level's mask (tl's mask) should allow us to
skip the cpuset partition. I just wanted to check if your concern
is about the excessive number of sd_llc_ids caused by the cpuset?
I was under the impression that without this patch, llc_ids are unique across different partitions.
For example, on vanilla kernel without cache_aware, suppose 1 LLC has CPU0,1,2,3. Before partition, all CPUs have the same llc_id 0. Then create a new partition, mkdir -p /sys/fs/cgroup/cgroup0
echo "3" > /sys/fs/cgroup/cgroup0/cpuset.cpus
echo root > /sys/fs/cgroup/cgroup0/cpuset.cpus.partition CPU0,1,2 share llc_id 0, and CPU3 has a dedicated llc_id 3. Do you suggest to let CPU3 reuse llc_id 0, so as to save more llc_id space?
Quoted 13 lines
How about something like:
kernel/sched/core.c +2
-0@@ -8270,6 +8270,18 @@ static void cpuset_cpu_active(void)static void cpuset_cpu_inactive (unsigned int cpu) {if (!cpuhp_tasks_frozen) { +/* +* This is necessary since offline CPUs are
kernel/sched/core.c static void cpuset_cpu_active(void) · line 8284
Quoted 10 lines
kernel/sched/core.c +10
-0+* taken out of the tl->mask() and a newly +* onlined CPU in same LLC will not realize +* whether it should reuse the LLC ID owned +* by an offline CPU without knowing the +* LLC association. +* +* Safe to release the reference if this is +* the last CPU in the LLC going offline. +*/ +sched_domain_free_llc_id (cpu);
I'm OK with replacing the domain based cpumask by the topology_level mask, just wondering whether re-using the llc_id would increase the risk of race condition - it is possible that, a CPU has different llc_ids before/after online/offline. Can we assign/reserve a "static" llc_id for each CPU, whether it is online or offline? In this way, we don't need to worry about the data synchronization when using llc_id(). For example, I can think of adjusting the data in percpu nr_pref_llc[max_llcs] on every CPU whenever a CPU gets offline/online.
Quoted 63 lines
kernel/sched/core.c +0
-0cpuset_update_active_cpus (); }else { num_cpus_frozen++;kernel/sched/debug.c +1
-0@@ -631,6 +631,7 @@ void update_sched_domain_debugfs(void) i++; } +debugfs_create_u32 ("llc_id" ,0444 , d_cpu, (u32 *)per_cpu_ptr (&sd_llc_id, cpu));__cpumask_clear_cpu (cpu, sd_sysctl_cpus); } }kernel/sched/sched.h +1
-0@@ -2142,6 +2142,7 @@ extern int group_balance_cpu(struct sched_group *sg);extern void update_sched_domain_debugfs (void );extern void dirty_sched_domain_sysctl (int cpu); +void sched_domain_free_llc_id (int cpu);extern int sched_update_scaling (void );kernel/sched/topology.c +28
-0@@ -20,6 +20,46 @@ void sched_domains_mutex_unlock(void)/* Protected by sched_domains_mutex: */ static cpumask_var_t sched_domains_tmpmask;static cpumask_var_t sched_domains_tmpmask2; +static cpumask_var_t sched_llc_id_alloc_mask; +DEFINE_PER_CPU (int, sd_llc_id) =-1 ; +static int max_llcs =0 ; + +static inline int sched_domain_alloc_llc_id (void ) +{ +int llc_id; + +lockdep_assert_held (&sched_domains_mutex); + + llc_id =cpumask_first_zero (sched_llc_id_alloc_mask); +BUG_ON ((unsigned int )llc_id >= nr_cpumask_bits); +cpumask_set_cpu (llc_id, sched_llc_id_alloc_mask); + ++max_llcs; + +return llc_id; +} + +void sched_domain_free_llc_id (int cpu) +{ +int i, llc_id =per_cpu (sd_llc_id, cpu); +bool found = false; + +lockdep_assert_cpus_held ();/* For cpu_active_mask. */ +guard (mutex)(&sched_domains_mutex); + +per_cpu (sd_llc_id, cpu) =-1 ; +for_each_cpu (i, cpu_active_mask) {
kernel/sched/topology.c void sched_domains_mutex_unlock(void) · line 60
Quoted 10 lines
kernel/sched/topology.c +10
-0+if (per_cpu (sd_llc_id, i) == llc_id) { + found = true; +break ; + } + } + +/* Allow future hotplugs to claim this ID */ +if (!found) { +cpumask_clear_cpu (llc_id, sched_llc_id_alloc_mask); + --max_llcs;
Maybe only allow increasing the value of max_llcs when a new LLC is detected. That says, max_llcs represents the total number of LLCs that have ever been detected, even if some of the corresponding CPUs have been taken offline via runtime hotplug. In this way, the data synchronization might be simpler, maybe trade additional memory space for code simplicity?
Quoted 76 lines
kernel/sched/topology.c +41
-4+ } +}static int __init sched_debug_setup (char *str) {@@ -658,7 +698,6 @@ static void destroy_sched_domains(struct sched_domain *sd)*/ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc);DEFINE_PER_CPU (int, sd_llc_size); -DEFINE_PER_CPU (int, sd_llc_id);DEFINE_PER_CPU (int, sd_share_id); DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa);@@ -684,7 +723,6 @@ static void update_top_cache_domain(int cpu)rcu_assign_pointer (per_cpu (sd_llc, cpu), sd);per_cpu (sd_llc_size, cpu) = size; -per_cpu (sd_llc_id, cpu) = id;rcu_assign_pointer (per_cpu (sd_llc_shared, cpu), sds); sd =lowest_flag_domain (cpu,SD_CLUSTER );@@ -2567,10 +2605,35 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att/* Set up domains for CPUs specified by the cpu_map: */ for_each_cpu (i, cpu_map) { -struct sched_domain_topology_level *tl; +struct sched_domain_topology_level *tl, *tl_llc =NULL ; +bool done = false; sd =NULL ;for_each_sd_topology (tl) { +int flags =0 ; + +if (tl->sd_flags) + flags = (*tl->sd_flags)(); + +if (flags &SD_SHARE_LLC ) { + tl_llc = tl; + +/* +* Entire cpu_map has been covered. We are +* traversing only to find the highest +* SD_SHARE_LLC level. +*/ +if (done) +continue ; + } + +/* +* Since SD_SHARE_LLC is SDF_SHARED_CHILD, we can +* safely break out if the entire cpu_map has been +* covered by a child domain. +*/ +if (done) +break ; sd =build_sched_domain (tl, cpu_map, attr, sd, i);@@ -2579,7 +2642,41 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attif (tl == sched_domain_topology) *per_cpu_ptr (d.sd, i) = sd;if (cpumask_equal (cpu_map,sched_domain_span (sd))) -break ; + done = true; + } + +/* First time visiting this CPU. Assign the llc_id. */ +if (per_cpu (sd_llc_id, i) ==-1 ) { +int j, llc_id =-1 ; + +/* +* In case there are no SD_SHARE_LLC domains, +* each CPU gets its own llc_id. Find the first +* free bit on the mask and use it. +*/ +if (!tl_llc) {
kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2667
Quoted 10 lines
kernel/sched/topology.c +10
-0+per_cpu (sd_llc_id, i) =sched_domain_alloc_llc_id (); +continue ; + } + +/* +* Visit all the CPUs of the LLC irrespective of the +* partition constraints and find if any of them have +* a valid llc_id. +*/ +for_each_cpu (j, tl_llc->mask (tl, i)) {
This is doable, we can use tl rather than domain's mask to share llc_id among partitions.
Quoted 25 lines
kernel/sched/topology.c +13
-1+ llc_id =per_cpu (sd_llc_id, j); + +/* Found a valid llc_id for CPU's LLC. */ +if (llc_id !=-1 ) +break ; + } + +/* Valid llc_id not found. Allocate a new one. */ +if (llc_id ==-1 ) + llc_id =sched_domain_alloc_llc_id (); + +per_cpu (sd_llc_id, i) = llc_id; } }@@ -2759,6 +2856,7 @@ int __init sched_init_domains(const struct cpumask *cpu_map)zalloc_cpumask_var (&sched_domains_tmpmask,GFP_KERNEL );zalloc_cpumask_var (&sched_domains_tmpmask2,GFP_KERNEL ); +zalloc_cpumask_var (&sched_llc_id_alloc_mask,GFP_KERNEL );zalloc_cpumask_var (&fallback_doms,GFP_KERNEL );arch_update_cpu_topology (); ---
Quoted 6 lines
AFAICT, "
sd_llc_id" isn't compared across different partitions so having the CPUs that are actually associated with same physical LLC but across different partitions sharing the same "sd_llc_id" shouldn't be a problem.Thoughts?
This means cpus_share_resources(int this_cpu, int that_cpu)
should be invoked when this_cpu and that_cpu belong to the same
partition.
In this way, we do not alter the context of cpus_share_resources(). We can conduct an audit of the places where cpus_share_resources() is used.
Happy holidays,
Chenyu
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Yangyu ChenQuoted 53 lines · Yangyu Chen
Quoted 42 lines · Yangyu Chen
Quoted 32 lines · Yangyu Chen
Quoted 21 lines · Tim Chen
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Quoted 8 lines
Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.Quoted 6 lines
Hi Chen Yu and Tim Chen,
Maybe we should have something like
prctl(PR_LLC_AGGR_TOLERANCE, 100).I have tested this version of the patch on my EPYC Milan 7V13 (7763 variant) physical machine, with 32M LLC for each 8-core CCX. I found that I need to tune "
llc_aggr_tolerance" to 100, else I can't get cache-aware scheduling to work on Verilated [1] XiangShan [2] running the chacha20 [3] as I mentioned before [4].Quoted 10 lines
In addition, I have investigated why this happens. And finally I realized that's because that workload observed 35596 kB RssAnon on my EPYC Milan Machine, slightly exceeding the LLC size (32M). I have tested it on an EPYC Genoa cloud server with the correct core / cache hierarchy in ACPI table, that shows 31700 kB RssAnon, thus fitting in LLC. I have no idea why my result shows higher RssAnon, since they both run Debian Trixie with the exact same kernel and same executable. But it reminds me we should have a userspace API for that.
Quoted 4 lines
In addition, during profiling the verilator, I found that if scheduled to SMTs, it will result in poor performance. Thus, I think we should separate the control for rss size with the SMT scale.
Thanks for the investigation. Could you elaborate a little more about scheduled to SMTs? Do you mean, if every CPU(SMT) in the LLC has 1 running task, then the performance is impacted? I thought we have exceed_llc_nr() to check the smt to avoid this?
Quoted 4 lines
It's notable that rss size is not the actual memory footprint. It would be better if we could measure the
l2_missevent orl3_missevent to measure the l3 hit rate. Just for future work.
Yes, in user space, we can collect PMUs events/memory bandwidth via resctrl to decide whether to set task attributes.
Quoted 3 lines
I'm willing to provide a patch for such a prctl. But I'm busy these days, maybe I can have the time to do that after one week.
Sure. We haven't yet decided which interface we can leverage. Also, Qais is working on QOS interface[1] - maybe we can build on his work.
[1] https://lore.kernel.org/all/[email protected]/
thanks,
Chenyu
K Prateek Nayak
reviewerRe: [PATCH v2 04/23] sched/cache: Make LLC id continuous
In reply to Chen, Yu CHello Chenyu,
Quoted 36 lines · Yu C
Hello Prateek,
Quoted 24 lines · K Prateek Nayak
Hello Tim, Chenyu,
Quoted 21 lines · Tim Chen
(quoted diff) +21
-0+/* + * Assign continuous llc id for the CPU, and return + * the assigned llc id. + */ +static int update_llc_id(struct sched_domain *sd, + int cpu) +{ + int id = per_cpu(sd_llc_id, cpu), i; + + if (id >= 0) + return id; + + if (sd) { + /* Look for any assigned id and reuse it.*/ + for_each_cpu(i, sched_domain_span(sd)) { + id = per_cpu(sd_llc_id, i); + + if (id >= 0) { + per_cpu(sd_llc_id, cpu) = id; + return id; + }Quoted 8 lines
Quoted 2 lines
+ }
+ }I don't really like tying this down to the
sched_domainspan since partition and other weirdness can cause themax_llccount to go unnecessarily high. Thetl->mask()(fromsched_domain_topology_level) should give the mask considering all online CPUs and not bothering about cpusets.
Quoted 3 lines
OK, using the topology_level's mask (tl's mask) should allow us to skip the cpuset partition. I just wanted to check if your concern is about the excessive number of
sd_llc_idscaused by the cpuset?
Yes. Basically all cases where sched_domain_span() isn't covering the entire llc_span - even true for isolated partitions.
Quoted 4 lines
I was under the impression that without this patch,
llc_idsare unique across different partitions.
Quoted 9 lines
For example, on vanilla kernel without
cache_aware, suppose 1 LLC has CPU0,1,2,3. Before partition, all CPUs have the samellc_id0. Then create a new partition, mkdir -p/sys/fs/cgroup/cgroup0
echo "3" >/sys/fs/cgroup/cgroup0/cpuset.cpus
echo root >/sys/fs/cgroup/cgroup0/cpuset.cpus.partitionCPU0,1,2 sharellc_id0, and CPU3 has a dedicatedllc_id3. Do you suggest to let CPU3 reusellc_id0, so as to save morellc_idspace?
Yes. And I think it is logical. Load balancing doesn't happen across partitions so sd_llc_id reflecting the ID of the physical LLC shouldn't be a problem.
Quoted 26 lines
Quoted 14 lines
How about something like:
kernel/sched/core.c +2
-0@@ -8270,6 +8270,18 @@ static void cpuset_cpu_active(void) static void cpuset_cpu_inactive(unsigned int cpu) { if (!cpuhp_tasks_frozen) { + /* +* This is necessary since offline CPUs are kernel/sched/core.c static void cpuset_cpu_active(void) · line 8284
Quoted 10 lines
kernel/sched/core.c +10
-0+* taken out of the tl->mask() and a newly +* onlined CPU in same LLC will not realize +* whether it should reuse the LLC ID owned +* by an offline CPU without knowing the +* LLC association. +* +* Safe to release the reference if this is +* the last CPU in the LLC going offline. +*/ + sched_domain_free_llc_id(cpu);I'm OK with replacing the domain based cpumask by the
topology_level
Quoted 8 lines
mask, just wondering whether re-using the
llc_idwould increase the risk of race condition - it is possible that, a CPU has differentllc_idsbefore/after online/offline. Can we assign/reserve a "static"llc_idfor each CPU, whether it is online or offline? In this way, we don't need to worry about the data synchronization when usingllc_id(). For example, I can think of adjusting the data in percpu nr_pref_llc[max_llcs] on every CPU whenever a CPU gets offline/online.
So I was thinking of of expanding the rq->nr_pref_llc[] if the max_llc increases but leave it as is if the number of LLCs decreases. That way we don't have to worry about the dereferencing past the array boundary.
We can also have a wrapper like:
struct nr_llc_stats {
int nr_llcs;
struct rcu_head rcu;
int *nr_pref_llc;
}And re-allocate and attach it in rq_attach_root() during sd rebuild. That way, RCU read-side can always grab a reference to it, enqueue / dequeue don't need to care since it cannot change under rq_lock, and partition can use call_rcu() to free the old ones up.
Quoted 69 lines
Quoted 69 lines
kernel/sched/core.c +0
-0cpuset_update_active_cpus(); } else { num_cpus_frozen++;kernel/sched/debug.c +0
-0@@ -631,6 +631,7 @@ void update_sched_domain_debugfs(void) i++; } + debugfs_create_u32("llc_id", 0444, d_cpu, (u32 *)per_cpu_ptr(&sd_llc_id, cpu)); __cpumask_clear_cpu(cpu, sd_sysctl_cpus); } }kernel/sched/sched.h +1
-0@@ -2142,6 +2142,7 @@ extern int group_balance_cpu(struct sched_group *sg); extern void update_sched_domain_debugfs(void); extern void dirty_sched_domain_sysctl(int cpu); +void sched_domain_free_llc_id (int cpu); extern int sched_update_scaling(void);diff --git
a/kernel/sched/topology.cb/kernel/sched/topology.cindexcf643a5ddedd..d6e134767f30100644kernel/sched/topology.c +37
-0@@ -20,6 +20,46 @@ void sched_domains_mutex_unlock(void) /* Protected by sched_domains_mutex: */ static cpumask_var_t sched_domains_tmpmask; static cpumask_var_t sched_domains_tmpmask2; +static cpumask_var_t sched_llc_id_alloc_mask; +DEFINE_PER_CPU (int, sd_llc_id) =-1 ; +static int max_llcs =0 ; + +static inline int sched_domain_alloc_llc_id (void ) +{ + int llc_id; + + lockdep_assert_held(&sched_domains_mutex); + + llc_id = cpumask_first_zero(sched_llc_id_alloc_mask); + BUG_ON((unsigned int)llc_id >= nr_cpumask_bits); + cpumask_set_cpu(llc_id, sched_llc_id_alloc_mask); + ++max_llcs; + + return llc_id; +} + +void sched_domain_free_llc_id (int cpu) +{ + int i, llc_id = per_cpu(sd_llc_id, cpu); + bool found = false; + + lockdep_assert_cpus_held(); /* For cpu_active_mask. */ + guard(mutex)(&sched_domains_mutex); + + per_cpu(sd_llc_id, cpu) = -1; + for_each_cpu(i, cpu_active_mask) { + if (per_cpu(sd_llc_id, i) == llc_id) { + found = true; + break; + } + } + + /* Allow future hotplugs to claim this ID */ + if (!found) { + cpumask_clear_cpu(llc_id, sched_llc_id_alloc_mask);
Quoted 8 lines
kernel/sched/topology.c void sched_domains_mutex_unlock(void) · line 60
Quoted 1 line
kernel/sched/topology.c +1
-0+ --max_llcs;Maybe only allow increasing the value of
max_llcswhen a new LLC is detected. That says,max_llcsrepresents the total number of LLCs that have ever been detected, even if some of the corresponding CPUs have been taken offline via runtime hotplug. In this way, the data synchronization might be simpler, maybe trade additional memory space for code simplicity?
Ack. That "struct nr_llc_stats" might be over-engineering. I don't mind working on it later after this goes in.
Quoted 110 lines
Quoted 70 lines
kernel/sched/topology.c +28
-3+ } +} static int __init sched_debug_setup(char *str) {@@ -658,7 +698,6 @@ static void destroy_sched_domains(struct sched_domain *sd)*/ DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc); DEFINE_PER_CPU(int, sd_llc_size); -DEFINE_PER_CPU (int, sd_llc_id); DEFINE_PER_CPU(int, sd_share_id); DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared); DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa);@@ -684,7 +723,6 @@ static void update_top_cache_domain(int cpu) rcu_assign_pointer(per_cpu(sd_llc, cpu), sd); per_cpu(sd_llc_size, cpu) = size; - per_cpu(sd_llc_id, cpu) = id; rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds); sd = lowest_flag_domain(cpu, SD_CLUSTER);@@ -2567,10 +2605,35 @@ build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att /* Set up domains for CPUs specified by the cpu_map: */ for_each_cpu(i, cpu_map) { - struct sched_domain_topology_level *tl; + struct sched_domain_topology_level *tl, *tl_llc = NULL; + bool done = false; sd = NULL; for_each_sd_topology(tl) { + int flags = 0; + + if (tl->sd_flags) + flags = (*tl->sd_flags)(); + + if (flags & SD_SHARE_LLC) { + tl_llc = tl; + + /* + * Entire cpu_map has been covered. We are + * traversing only to find the highest + * SD_SHARE_LLC level. + */ + if (done) + continue; + } + + /* + * Since SD_SHARE_LLC is SDF_SHARED_CHILD, we can + * safely break out if the entire cpu_map has been + * covered by a child domain. + */ + if (done) + break; sd = build_sched_domain(tl, cpu_map, attr, sd, i);@@ -2579,7 +2642,41 @@ build_sched_domains(const struct cpumask
*cpu_map, structsched_domain_attr*att if (tl ==sched_domain_topology)
*per_cpu_ptr(d.sd, i) = sd;
if (cpumask_equal(cpu_map, sched_domain_span(sd)))
- break;
+ done = true;
+ }kernel/sched/topology.c +11
-0+ + /* First time visiting this CPU. Assign the llc_id. */ + if (per_cpu(sd_llc_id, i) == -1) { + int j, llc_id = -1; + + /* + * In case there are no SD_SHARE_LLC domains, + * each CPU gets its own llc_id. Find the first + * free bit on the mask and use it. + */ + if (!tl_llc) {kernel/sched/topology.c build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *att · line 2667
Quoted 10 lines
kernel/sched/topology.c +10
-0+ per_cpu(sd_llc_id, i) = sched_domain_alloc_llc_id(); + continue; + } + + /* + * Visit all the CPUs of the LLC irrespective of the + * partition constraints and find if any of them have + * a valid llc_id. + */ + for_each_cpu(j, tl_llc->mask(tl, i)) {This is doable, we can use tl rather than domain's mask to share
llc_idamong partitions.Quoted 26 lines
+
llc_id=per_cpu(sd_llc_id, j);kernel/sched/topology.c +11
-0+ + /* Found a valid llc_id for CPU's LLC. */ + if (llc_id != -1) + break; + } + + /* Valid llc_id not found. Allocate a new one. */ + if (llc_id == -1) + llc_id = sched_domain_alloc_llc_id(); + + per_cpu(sd_llc_id, i) = llc_id;}
}
@@ -2759,6 +2856,7 @@ int__initsched_init_domains(const struct cpumask *cpu_map)zalloc_cpumask_var(&sched_domains_tmpmask, GFP_KERNEL);zalloc_cpumask_var(&sched_domains_tmpmask2, GFP_KERNEL);
+zalloc_cpumask_var(&sched_llc_id_alloc_mask, GFP_KERNEL);zalloc_cpumask_var(&fallback_doms, GFP_KERNEL);arch_update_cpu_topology();
---AFAICT, "
sd_llc_id" isn't compared across different partitions so having the CPUs that are actually associated with same physical LLC but across different partitions sharing the same "sd_llc_id" shouldn't be a problem.
Quoted 7 lines
Quoted 2 lines
Thoughts?
This means
cpus_share_resources(int this_cpu, int that_cpu)should be invoked whenthis_cpuandthat_cpubelong to the same partition. In this way, we do not alter the context ofcpus_share_resources(). We can conduct an audit of the places wherecpus_share_resources()is used.
Only case I can think of is a task wakes up after partitioning and it's wake cpu from a different partition is mistaken to share the LLC as the current CPU - but the task cannot actually run on that old CPU and it'll have to take the select_fallback_rq() path if prev_cpu was selected during wake_affine().
I don't think it will be such a common occurence to cause an issue and even without that wake_affine() could still the prev_cpu if current CPU is busy or via wake_affine_weight().
Signature · 3 lines
Happy holidays! Thanks and Regards, Prateek
Re: [PATCH v2 04/23] sched/cache: Make LLC id continuous
Quoted 9 lines · K Prateek Nayak
Hello Chenyu,
Quoted 6 lines · Yu C
Hello Prateek,
Quoted 3 lines · K Prateek Nayak
Hello Tim, Chenyu,
On 12/4/2025 4:37 AM, Tim Chen wrote:
[snip]
Quoted 10 lines
Quoted 9 lines
I'm OK with replacing the domain based cpumask by the
topology_levelmask, just wondering whether re-using thellc_idwould increase the risk of race condition - it is possible that, a CPU has differentllc_idsbefore/after online/offline. Can we assign/reserve a "static"llc_idfor each CPU, whether it is online or offline? In this way, we don't need to worry about the data synchronization when usingllc_id(). For example, I can think of adjusting the data in percpu nr_pref_llc[max_llcs] on every CPU whenever a CPU gets offline/online.
Quoted 5 lines
So I was thinking of of expanding the rq->nr_pref_llc[] if the
max_llcincreases but leave it as is if the number of LLCs decreases. That way we don't have to worry about the dereferencing past the array boundary.
Sure, we can do in this way.
Quoted 8 lines
We can also have a wrapper like:
struct nr_llc_stats { int nr_llcs; struct rcu_head rcu; int *nr_pref_llc; }
Quoted 6 lines
And re-allocate and attach it in
rq_attach_root()during sd rebuild. That way, RCU read-side can always grab a reference to it, enqueue / dequeue don't need to care since it cannot change underrq_lock, and partition can usecall_rcu()to free the old ones up.
OK, doing it in this direction(and Peter also suggested something like this in the domain)
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
cpuset_update_active_cpus();
} else {
[snip]
Quoted 8 lines
Quoted 8 lines
Quoted 6 lines
AFAICT, "
sd_llc_id" isn't compared across different partitions so having the CPUs that are actually associated with same physical LLC but across different partitions sharing the same "sd_llc_id" shouldn't be a problem.Thoughts?
This means
cpus_share_resources(int this_cpu, int that_cpu)
Actually I was about to say cpus_share_cache().
Quoted 11 lines
Quoted 3 lines
should be invoked when
this_cpuandthat_cpubelong to the same partition.
In this way, we do not alter the context ofcpus_share_resources(). We can conduct an audit of the places wherecpus_share_resources()is used.Only case I can think of is a task wakes up after partitioning and it's wake cpu from a different partition is mistaken to share the LLC as the current CPU - but the task cannot actually run on that old CPU and it'll have to take the
select_fallback_rq()path ifprev_cpuwas selected duringwake_affine().
OK, make sense.
Actually, prev_cpu might not be chosen by select_task_rq_fair()-> select_idle_sibling(), because fast path select_idle_sibling()
is expected to be triggered when prev_cpu and the current cpu are in the
same domain in select_task_rq_fair():cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))
sd = NULL; //wake affine
curr cpu and prev_cpu are in different partitions, they
are not in the same domains.
Quoted 4 lines
I don't think it will be such a common occurence to cause an issue and even without that
wake_affine()could still theprev_cpuif current CPU is busy or viawake_affine_weight().
I realized that sched_cache has added cpus_share_cache() in several places, most of which should be related to load balancing, which should not be a problem if llc_id is shared among partitions. I'll double check.
thanks,
Chenyu
Re: [PATCH v2 05/23] sched/cache: Assign preferred LLC ID to processes
In reply to Vern HaoQuoted 9 lines · Vern Hao
Quoted 7 lines · Tim Chen
Quoted 6 lines · Vern Hao
Quoted 5 lines · Tim Chen
With cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar to
numa_preferred_nidin NUMA balancing.
[snip]
Quoted 15 lines
Quoted 15 lines
Quoted 15 lines
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1343
Quoted 10 lines
kernel/sched/fair.c +10
-0+ + if (mm->mm_sched_cpu != -1) { + mm_sched_llc = llc_id(mm->mm_sched_cpu); + +#ifdef CONFIG_NUMA_BALANCING + /* + * Don't assign preferred LLC if it + * conflicts with NUMA balancing. + */ + if (p->numa_preferred_nid >= 0 &&I wonder if the restriction here shouldn't be so strict. In Mel Gorman's patch (
e496132ebeddsched/fair: Adjust the allowed NUMA imbalance whenSD_NUMAspans multiple LLCs), the value of the 'imb_numa_nr' is checked to determine if |SD_NUMA| imbalance is allowed. Could we use this same check to decide whether or not to perform a cross-numa migration?
Quoted 9 lines
Quoted 4 lines
If we set the preferred LLC that's in a different node other than the preferred node, the preferred LLC is going to fight with NUMA balancing and bounce tasks back and forth between nodes. NUMA locality is going to affect performance more so we'll let NUMA preference take precedence.
I might not have explained myself clearly. I’m questioning whether we need to integrate an imbalance check into the 'sgs->group_type|' is ’group_has_spare|' scenario, like Mel’s patch, to refine our llc migration decisions.
just like this: 8 cpus in one LLC, LLC-A has 6 tasks, LLC-B has 2 tasks, if LLC-A has
task_aneed to migrate toLLC_B, how to deal it ?
If LLC_B is the preferred LLC of task_a, and if the average utilization of LLC_B has not reached 50%, task_a will be moved to LLC_B. If LLC_A is the preferred LLC of task_a, then if LLC_A has not reached 50%, task_a will not be migrated to LLC_B. There are some comments around can_migrate_llc(), which describe the decision matrix for migration.
thanks,
Chenyu
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Chen, Yu CQuoted 55 lines · Yu C
Quoted 50 lines · Yangyu Chen
Quoted 50 lines · Yangyu Chen
Quoted 32 lines · Yangyu Chen
Quoted 21 lines · Tim Chen
From: Chen Yu <[email protected]>
Introduce a set of debugfs knobs to control the enabling of and parameters for cache-aware load balancing.
(1)
llc_enabledllc_enabledacts as the primary switch - users can toggle it to enable or disable cache aware load balancing.(2)
llc_aggr_tolerance
Withsched_cacheenabled, the scheduler uses a process's RSS as a proxy for its LLC footprint to determine if aggregating tasks on the preferred LLC could cause cache contention. If RSS exceeds the LLC size, aggregation is skipped. Some workloads with large RSS but small actual memory footprints may still benefit from aggregation. Since the kernel cannot efficiently track per-task cache usage (resctrl is user-space only), userspace can provide a more accurate hint.Quoted 8 lines
Introduce
/sys/kernel/debug/sched/llc_aggr_toleranceto let users control how strictly RSS limits aggregation. Values range from 0 to 100:- 0: Cache-aware scheduling is disabled.
- 1: Strict; tasks with RSS larger than LLC size are skipped.
- 100: Aggressive; tasks are aggregated regardless of RSS.Quoted 6 lines
Hi Chen Yu and Tim Chen,
Maybe we should have something like
prctl(PR_LLC_AGGR_TOLERANCE, 100).I have tested this version of the patch on my EPYC Milan 7V13 (7763 variant) physical machine, with 32M LLC for each 8-core CCX. I found that I need to tune "
llc_aggr_tolerance" to 100, else I can't get cache-aware scheduling to work on Verilated [1] XiangShan [2] running the chacha20 [3] as I mentioned before [4].In addition, I have investigated why this happens. And finally I realized that's because that workload observed 35596 kB RssAnon on my EPYC Milan Machine, slightly exceeding the LLC size (32M). I have tested it on an EPYC Genoa cloud server with the correct core / cache hierarchy in ACPI table, that shows 31700 kB RssAnon, thus fitting in LLC. I have no idea why my result shows higher RssAnon, since they both run Debian Trixie with the exact same kernel and same executable. But it reminds me we should have a userspace API for that.
Quoted 8 lines
Quoted 3 lines
In addition, during profiling the verilator, I found that if scheduled to SMTs, it will result in poor performance. Thus, I think we should separate the control for rss size with the SMT scale.
Thanks for the investigation. Could you elaborate a little more about scheduled to SMTs? Do you mean, if every CPU(SMT) in the LLC has 1 running task, then the performance is impacted? I thought we have
exceed_llc_nr()to check the smt to avoid this?
The verilator can specify the number of threads being used for the RTL simulator during compilation. And it cannot be changed at runtime since it will do static partitioning. Thus, I didn't mean if there is another thread being scheduled to a SMT in the LLC and we got poor performance. I mean that the users can allow the verilator to use more threads larger than the LLC capacity. But I have tested your case, on my observation with the recent version of XiangShan
+ Verilator + LLVM21 with an 8-thread emulator, it shows 41%(30% for 1-thread) and 62%(39% for 1-thread) performance degradation on Raptor Lake and EPYC Milan if another 8 threads are running with a simple loop. But I think that's only a datapoint. Since both Raptor Lake and Zen 5 will statically partition the ROB in the CPU backend, and such workloads will suffer a lot of data cache misses since they have a very huge instruction footprint. I think SMT performance is not easy to characterize across different microarchitectures and workloads, but one thing for sure is that I didn't come across a situation where a 16-thread emulator on an EPYC machine scheduled to 1-CCX with 2-SMT is better than 2-CCX with only 1-SMT. That’s why I think we should split this two user controls, one for RSS and one for the number of threads.
Thanks,
Yangyu Chen
Quoted 19 lines
Quoted 4 lines
It's notable that rss size is not the actual memory footprint. It would be better if we could measure the
l2_missevent orl3_missevent to measure the l3 hit rate. Just for future work.Yes, in user space, we can collect PMUs events/memory bandwidth via resctrl to decide whether to set task attributes.
Quoted 2 lines
I'm willing to provide a patch for such a prctl. But I'm busy these days, maybe I can have the time to do that after one week.
Sure. We haven't yet decided which interface we can leverage. Also, Qais is working on QOS interface[1] - maybe we can build on his work.
[1] https://lore.kernel.org/all/[email protected]/
thanks,
Chenyu
K Prateek Nayak
reviewerRe: [PATCH v2 04/23] sched/cache: Make LLC id continuous
In reply to Chen, Yu CHello Chenyu,
Quoted 8 lines · Yu C
Quoted 7 lines
Only case I can think of is a task wakes up after partitioning and it's wake cpu from a different partition is mistaken to share the LLC as the current CPU - but the task cannot actually run on that old CPU and it'll have to take the
select_fallback_rq()path ifprev_cpuwas selected duringwake_affine().
Quoted 7 lines
OK, make sense.
Actually,prev_cpumight not be chosen by select_task_rq_fair()->select_idle_sibling(), because fast pathselect_idle_sibling()is expected to be triggered whenprev_cpuand the current cpu are in the same domain inselect_task_rq_fair():cpumask_test_cpu(prev_cpu, sched_domain_span(tmp))
sd = NULL; //wake affine
Then again, there are cases where want_affine is false, and "new_cpu" is initialized to the prev_cpu and we continue down to select_idle_sibling() since "tmp->flags & sd_flag" is always false - WF_TTWU matches with SD_BALANCE_WAKE but no domain sets it anymore afaict.
Again, going through the fallback selection path should be rare (once after partition on wakeup) and shouldn't cause any problems for most real-world scenarios.
Quoted 8 lines
curr cpu and
prev_cpuare in different partitions, they are not in the same domains.Quoted 4 lines
I don't think it will be such a common occurence to cause an issue and even without that
wake_affine()could still theprev_cpuif current CPU is busy or viawake_affine_weight().
Quoted 4 lines
I realized that
sched_cachehas addedcpus_share_cache()in several places, most of which should be related to load balancing, which should not be a problem ifllc_idis shared among partitions. I'll double check.
Thank you!
Signature · 3 lines
Happy Holidays! Thanks and Regards, Prateek
Re: [PATCH v2 05/23] sched/cache: Assign preferred LLC ID to processes
In reply to Tim ChenHi Tim,
Quoted 27 lines · Tim Chen
With cache-aware scheduling enabled, each task is assigned a preferred LLC ID. This allows quick identification of the LLC domain where the task prefers to run, similar to
numa_preferred_nidin NUMA balancing.Signed-off-by Tim Chen <[email protected]>---
kernel/sched/fair.c +3
-0@@ -1300,6 +1300,7 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched;unsigned long epoch; +int mm_sched_llc =-1 ;if (!sched_cache_enabled ())return ;@@ -1330,6 +1331,23 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)if (mm->mm_sched_cpu !=-1 ) mm->mm_sched_cpu =-1 ; } + +if (mm->mm_sched_cpu !=-1 ) {
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1345
Quoted 10 lines
kernel/sched/fair.c +10
-0+ mm_sched_llc =llc_id (mm->mm_sched_cpu); + +#ifdef CONFIG_NUMA_BALANCING +/* +* Don't assign preferred LLC if it +* conflicts with NUMA balancing. +*/ +if (p->numa_preferred_nid >=0 && +cpu_to_node (mm->mm_sched_cpu) != p->numa_preferred_nid) + mm_sched_llc =-1 ;
Scenario: NUMA balancing is initially enabled and numa_preferred_nid is set. After disabling NUMA balancing, numa_preferred_nid is not cleared, leading to unintended impact of NUMA balancing on cache-aware scheduling. Should we check if NUMA balancing is disabled before checking numa_preferred_nid?
Thanks
Jianyong Wu
Quoted 8 lines
kernel/sched/fair.c +5
-0+#endif + } + +if (p->preferred_llc != mm_sched_llc) + p->preferred_llc = mm_sched_llc; }static void task_tick_cache (struct rq *rq,struct task_struct *p)
Re: [PATCH v2 05/23] sched/cache: Assign preferred LLC ID to processes
Quoted 18 lines · Jianyong Wu
Hi Tim,
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1345
Quoted 12 lines · Tim Chen
kernel/sched/fair.c +12
-0+ + if (mm->mm_sched_cpu != -1) { + mm_sched_llc = llc_id(mm->mm_sched_cpu); + +#ifdef CONFIG_NUMA_BALANCING + /* + * Don't assign preferred LLC if it + * conflicts with NUMA balancing. + */ + if (p->numa_preferred_nid >= 0 && + cpu_to_node(mm->mm_sched_cpu) != p->numa_preferred_nid) + mm_sched_llc = -1;Scenario: NUMA balancing is initially enabled and
numa_preferred_nidis set. After disabling NUMA balancing,numa_preferred_nidis not cleared, leading to unintended impact of NUMA balancing on cache-aware scheduling. Should we check if NUMA balancing is disabled before checkingnuma_preferred_nid?
Good point, sched_numa_balancing should be checked here.
thanks,
Chenyu
Quoted 3 lines
Thanks
Jianyong Wu
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Tim ChenHello Tim, Chen Yu,
Quoted 25 lines
static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) {kernel/sched/fair.c +11
-2+unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; -unsigned int llc;/* * get_cpu_cacheinfo_level() can not be used @@ -1252,19 +1266,54 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) rss =get_mm_counter (mm,MM_ANONPAGES ) +get_mm_counter (mm,MM_SHMEMPAGES ); -return (llc <= (rss *PAGE_SIZE )); +/* +* Scale the LLC size by 256*llc_aggr_tolerance +* and compare it to the task's RSS size. +* +* Suppose the L3 size is 32MB. If the +* llc_aggr_tolerance is 1: +* When the RSS is larger than 32MB, the process +* is regarded as exceeding the LLC capacity. If +* the llc_aggr_tolerance is 99: +* When the RSS is larger than 784GB, the process
kernel/sched/fair.c static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) · line 1288
Quoted 10 lines
kernel/sched/fair.c +8
-0+* is regarded as exceeding the LLC capacity because: +* 784GB = (1 + (99 - 1) * 256) * 32MB +*/ + scale =get_sched_cache_scale (256 ); +if (scale ==INT_MAX ) +return false; + +return ((llc * scale) <= (rss *PAGE_SIZE )); }
Suppose the LLC is 16 MB (which is 16777216 bytes). If we set llc_aggr_tolerance to 99, the scale value will be 25089. When calculating 16777216 * 25089, the result is 420923572224, which exceeds 2^32 and thus causes an integer overflow.
Thanks
Jianyong