Cache aware scheduling
Subsystems (5)
Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling
In reply to Yangyu ChenHi Yangyu,
Quoted 3 lines · Yangyu Chen
On 4/12/2025 07:07, Tim Chen wrote:
[ ... ]
Quoted 8 lines
Quoted 1 line
+ scale =
get_sched_cache_scale(256);Hi Tim Chen and Chen Yu,
There's an integer overflow here. Since the unit of LLC size is bytes, you have a 256-scale unit. For a typical LLC size of 32M, you calculate 32M multiplied by 256, which equals 8GB. This value exceeds the maximum integer value (2GB,
INT_MAX), resulting in an integer overflow.I think such function should be use u64. Below is my patch:
Thanks very much for the investigation, Jianyong previously also mentioned this issue
https://lore.kernel.org/all/SI2PR04MB49317BA503E9C8A381956D6AE38CA@SI2PR04MB4931.apcprd04.prod.outlook.com/
we will fix the issue accordingly.
Quoted 15 lines
kernel/sched/fair.c +1
-1@@ -1226,20 +1226,20 @@ static int llc_id(int cpu) return llc; } -static inline int get_sched_cache_scale (int mul) +static inline u64 get_sched_cache_scale (int mul) { if (!llc_aggr_tolerance) return 0; if (llc_aggr_tolerance == 100)
kernel/sched/fair.c static int llc_id(int cpu) · line 1242
Quoted 10 lines
kernel/sched/fair.c +2
-2- return INT_MAX; + return ULLONG_MAX; return (1 + (llc_aggr_tolerance - 1) * mul); } static bool exceed_llc_capacity(struct mm_struct *mm, int cpu) { - unsigned int llc, scale; + unsigned long long llc, scale;
I suppose we only need to change llc to u64, and not change scale, because (llc * scale) would be converted to u64 anyway.
thanks,
Chenyu
PATCH v2 21/23 -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Debug patch only.
With cache-aware load balancing enabled, statistics related to its activity are exposed via /proc/schedstat and debugfs. For instance, if users want to verify metrics like the number of exceeding RSS and nr_running limits, they can filter the output of /sys/kernel/debug/sched/debug and compute the required statistics manually:
llc_exceed_cap SUM: 6llc_exceed_nr SUM: 4531
Furthermore, these statistics exposed in /proc/schedstats can be queried manually or via perf sched stats[1] with minor modifications.
--- include/linux/sched/topology.h | 1 + kernel/sched/fair.c | 1 + kernel/sched/stats.c | 5 +++-- 3 files changed, 5 insertions(+), 2 deletions(-)
include/linux/sched/topology.h +1 -0
@@ -108,6 +108,7 @@ struct sched_domain {unsigned int lb_imbalance_util[CPU_MAX_IDLE_TYPES ];unsigned int lb_imbalance_task[CPU_MAX_IDLE_TYPES ];unsigned int lb_imbalance_misfit[CPU_MAX_IDLE_TYPES ]; +unsigned int lb_imbalance_llc[CPU_MAX_IDLE_TYPES ];unsigned int lb_gained[CPU_MAX_IDLE_TYPES ];unsigned int lb_hot_gained[CPU_MAX_IDLE_TYPES ];unsigned int lb_nobusyg[CPU_MAX_IDLE_TYPES ];
kernel/sched/fair.c +1 -0
@@ -12684,6 +12684,7 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sd__schedstat_add (sd->lb_imbalance_misfit[idle], env->imbalance);break ;case migrate_llc_task: +__schedstat_add (sd->lb_imbalance_llc[idle], env->imbalance);break ; } }
kernel/sched/stats.c +3 -2
@@ -104,7 +104,7 @@ void __update_stats_enqueue_sleeper(struct rq *rq, struct task_struct *p,* Bump this up when changing the output format or the meaning of an existing * format, so that tools can adapt (or abort) */ -#define SCHEDSTAT_VERSION 17 +#define SCHEDSTAT_VERSION 18static int show_schedstat (struct seq_file *seq,void *v) {
@@ -139,7 +139,7 @@ static int show_schedstat(struct seq_file *seq, void *v)seq_printf (seq,"domain%d %s %*pb" , dcount++, sd->name,cpumask_pr_args (sched_domain_span (sd)));for (itype =0 ; itype <CPU_MAX_IDLE_TYPES ; itype++) { - seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u", + seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u %u", sd->lb_count[itype], sd->lb_balanced[itype], sd->lb_failed[itype],
@@ -147,6 +147,7 @@ static int show_schedstat(struct seq_file *seq, void *v) sd->lb_imbalance_util[itype], sd->lb_imbalance_task[itype], sd->lb_imbalance_misfit[itype], + sd->lb_imbalance_llc[itype], sd->lb_gained[itype], sd->lb_hot_gained[itype], sd->lb_nobusyq[itype],
Signature · 1 line
2.32.0
Re: [PATCH v2 21/23] -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing
Quoted 13 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
With cache-aware load balancing enabled, statistics related to its activity are exposed via
/proc/schedstatand debugfs. For instance, if users want to verify metrics like the number of exceeding RSS andnr_runninglimits, they can filter the output of/sys/kernel/debug/sched/debugand compute the required statistics manually:
Quoted 6 lines
llc_exceed_capSUM: 6llc_exceed_nrSUM: 4531Furthermore, these statistics exposed in
/proc/schedstatscan be queried manually or via perf sched stats[1] with minor modifications.
Hi Tim,
This patch looks great, especially for multithread Verilator workloads on clustered LLC (like AMD EPYC). I'm discussing with Verilator upstream to disable automatic userspace affinity assignment in Verilator if such feature exist [1]. During the discussion, I think there should be a way for userspace software to detect if such a feature exists. Could we expose it in /proc/schedstats to allow userspace software to detect such a feature? We can just use this patch and remove the "DO NOT APPLY" tag.
[1] https://github.com/verilator/verilator/issues/6826#issuecomment-3671287551
Thanks,
Yangyu Chen
Quoted 66 lines
Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
include/linux/sched/topology.h | 1 + kernel/sched/fair.c | 1 + kernel/sched/stats.c | 5 +++-- 3 files changed, 5 insertions(+), 2 deletions(-)include/linux/sched/topology.h +1
-0@@ -108,6 +108,7 @@ struct sched_domain {unsigned int lb_imbalance_util[CPU_MAX_IDLE_TYPES ];unsigned int lb_imbalance_task[CPU_MAX_IDLE_TYPES ];unsigned int lb_imbalance_misfit[CPU_MAX_IDLE_TYPES ]; +unsigned int lb_imbalance_llc[CPU_MAX_IDLE_TYPES ];unsigned int lb_gained[CPU_MAX_IDLE_TYPES ];unsigned int lb_hot_gained[CPU_MAX_IDLE_TYPES ];unsigned int lb_nobusyg[CPU_MAX_IDLE_TYPES ];kernel/sched/fair.c +0
-0@@ -12684,6 +12684,7 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sd__schedstat_add (sd->lb_imbalance_misfit[idle], env->imbalance);break ;case
migrate_llc_task:
+__schedstat_add(sd->lb_imbalance_llc[idle], env->imbalance); break;
}
}kernel/sched/stats.c +3
-2@@ -104,7 +104,7 @@ void __update_stats_enqueue_sleeper(struct rq *rq, struct task_struct *p,* Bump this up when changing the output format or the meaning of an existing * format, so that tools can adapt (or abort) */ -#define SCHEDSTAT_VERSION 17 +#define SCHEDSTAT_VERSION 18static int show_schedstat (struct seq_file *seq,void *v) {@@ -139,7 +139,7 @@ static int show_schedstat(struct seq_file *seq, void *v)seq_printf (seq,"domain%d %s %*pb" , dcount++, sd->name,cpumask_pr_args (sched_domain_span (sd)));for (itype =0 ; itype <CPU_MAX_IDLE_TYPES ; itype++) { - seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u", + seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u %u", sd->lb_count[itype], sd->lb_balanced[itype], sd->lb_failed[itype],@@ -147,6 +147,7 @@ static int show_schedstat(struct seq_file *seq, void *v) sd->lb_imbalance_util[itype], sd->lb_imbalance_task[itype], sd->lb_imbalance_misfit[itype], + sd->lb_imbalance_llc[itype], sd->lb_gained[itype], sd->lb_hot_gained[itype], sd->lb_nobusyq[itype],Signature · 1 line
2.32.0
Re: [PATCH v2 21/23] -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing
Quoted 22 lines · Yangyu Chen
Quoted 13 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
With cache-aware load balancing enabled, statistics related to its activity are exposed via
/proc/schedstatand debugfs. For instance, if users want to verify metrics like the number of exceeding RSS andnr_runninglimits, they can filter the output of/sys/kernel/debug/sched/debugand compute the required statistics manually:Quoted 6 lines
llc_exceed_capSUM: 6llc_exceed_nrSUM: 4531Furthermore, these statistics exposed in
/proc/schedstatscan be queried manually or via perf sched stats[1] with minor modifications.Hi Tim,
Quoted 9 lines
This patch looks great, especially for multithread Verilator workloads on clustered LLC (like AMD EPYC). I'm discussing with Verilator upstream to disable automatic userspace affinity assignment in Verilator if such feature exist [1]. During the discussion, I think there should be a way for userspace software to detect if such a feature exists. Could we expose it in
/proc/schedstatsto allow userspace software to detect such a feature? We can just use this patch and remove the "DO NOT APPLY" tag.
Thanks for the test Yangyu. Does /sys/kernel/debug/sched/llc_enabled work for you?
Anyway we can try to include /proc/schedstats as the formal one in the next version.
Thanks,
Chenyu
Quoted 7 lines
[1] https://github.com/verilator/verilator/issues/6826#issuecomment-3671287551
Thanks,
Yangyu Chen
Re: [PATCH v2 21/23] -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing
Quoted 33 lines · Yu C
Quoted 18 lines · Yangyu Chen
Quoted 12 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
With cache-aware load balancing enabled, statistics related to its activity are exposed via
/proc/schedstatand debugfs. For instance, if users want to verify metrics like the number of exceeding RSS andnr_runninglimits, they can filter the output of/sys/kernel/debug/sched/debugand compute the required statistics manually:Quoted 6 lines
llc_exceed_capSUM: 6llc_exceed_nrSUM: 4531Furthermore, these statistics exposed in
/proc/schedstatscan be queried manually or via perf sched stats[1] with minor modifications.Quoted 9 lines
Hi Tim,
This patch looks great, especially for multithread Verilator workloads on clustered LLC (like AMD EPYC). I'm discussing with Verilator upstream to disable automatic userspace affinity assignment in Verilator if such feature exist [1]. During the discussion, I think there should be a way for userspace software to detect if such a feature exists. Could we expose it in/proc/schedstatsto allow userspace software to detect such a feature? We can just use this patch and remove the "DO NOT APPLY" tag.
Quoted 2 lines
Thanks for the test Yangyu. Does
/sys/kernel/debug/sched/llc_enabledwork for you?
It requires debugfs being mounted with enough permissions. It’s not feasible for normal user-space software without root permission.
Thanks,
Yangyu Chen
Quoted 9 lines
Anyway we can try to include
/proc/schedstatsas the formal one in the next version.Thanks,
ChenyuQuoted 3 lines
[1] https://github.com/verilator/verilator/issues/6826#issuecomment-3671287551
Thanks,
Yangyu Chen
PATCH v2 22/23 -- DO NOT APPLY!!! -- sched/cache/debug: Add ftrace to track the load balance statistics
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Debug patch only.
The user leverages this trace event (via bpftrace, etc)to monitor the cache aware load balance activity - whether the tasks are moved to their preferred LLC, or moved out of their preferred LLC.
--- include/trace/events/sched.h | 31 +++++++++++++++++++++++++++++++ kernel/sched/fair.c | 10 ++++++++++ 2 files changed, 41 insertions(+)
include/trace/events/sched.h +31 -0
@@ -10,6 +10,37 @@#include <linux/tracepoint.h> #include <linux/binfmts.h> +TRACE_EVENT (sched_attach_task, + + TP_PROTO(struct task_struct *t, int pref_cpu, int pref_llc, + int attach_cpu, int attach_llc), + +TP_ARGS (t, pref_cpu, pref_llc, attach_cpu, attach_llc), + +TP_STRUCT__entry ( + __array( char, comm, TASK_COMM_LEN ) + __field( pid_t, pid ) + __field( int, pref_cpu ) + __field( int, pref_llc ) + __field( int, attach_cpu ) +__field ( int, attach_llc ) + ), + +TP_fast_assign ( + memcpy(__entry->comm, t->comm, TASK_COMM_LEN); + __entry->pid = t->pid; + __entry->pref_cpu = pref_cpu; + __entry->pref_llc = pref_llc; + __entry->attach_cpu = attach_cpu; + __entry->attach_llc = attach_llc; + ), + +TP_printk ("comm=%s pid=%d pref_cpu=%d pref_llc=%d attach_cpu=%d attach_llc=%d" , + __entry->comm, __entry->pid, + __entry->pref_cpu, __entry->pref_llc, + __entry->attach_cpu, __entry->attach_llc) +); +/* * Tracepoint for calling kthread_stop, performed to end a kthread: */
kernel/sched/fair.c +10 -0
@@ -10487,6 +10487,16 @@ static void attach_task(struct rq *rq, struct task_struct *p) {lockdep_assert_rq_held (rq); +#ifdef CONFIG_SCHED_CACHE +if (p->mm) { +int pref_cpu = p->mm->mm_sched_cpu; + +trace_sched_attach_task (p, + pref_cpu, + pref_cpu !=-1 ?llc_id (pref_cpu) :-1 , +cpu_of (rq),llc_id (cpu_of (rq))); + } +#endif WARN_ON_ONCE (task_rq (p) != rq);activate_task (rq, p,ENQUEUE_NOCLOCK );wakeup_preempt (rq, p,0 );
Signature · 1 line
2.32.0
PATCH v2 23/23 -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs
In reply to Tim ChenFrom: Chen Yu <[email protected]>
Debug patch only.
Show the per-LLC occupancy in /proc/{PID}/schedstat, with each column corresponding to one LLC. This can be used to verify if the cache-aware load balancer works as expected by aggregating threads onto dedicated LLCs.
Suppose there are 2 LLCs and the sampling duration is 10 seconds:
Enable the cache aware load balance:
0 12281 <--- LLC0 residency delta is 0, LLC1 is 12 seconds 0 18881
0 16217
disable the cache aware load balance:
6497 15802
9299 5435
17811 8278
--- fs/proc/base.c | 22 ++++++++++++++++++++++ include/linux/mm_types.h | 19 +++++++++++++++++-- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 80 insertions(+), 4 deletions(-)
fs/proc/base.c +22 -0
@@ -518,6 +518,28 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, (unsigned long long )task->se.sum_exec_runtime, (unsigned long long )task->sched_info.run_delay, task->sched_info.pcount); +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +struct mm_struct *mm = task->mm; +u64 *llc_runtime; + +if (!mm) +return 0 ; + + llc_runtime =kcalloc (max_llcs,sizeof (u64),GFP_KERNEL ); +if (!llc_runtime) +return 0 ; + +if (get_mm_per_llc_runtime (task, llc_runtime)) + goto out; + +for (int i =0 ; i < max_llcs; i++) +seq_printf (m,"%llu " , llc_runtime[i]); +seq_puts (m,"\n" ); +out: +kfree (llc_runtime); + } +#endif return 0 ; }
include/linux/mm_types.h +17 -2
@@ -944,6 +944,10 @@ struct mm_sched {unsigned long epoch; }; +struct mm_time { +u64 runtime_ns; +}; +struct kioctx_table ;struct iommu_mm_data ;struct mm_struct {
@@ -1040,6 +1044,7 @@ struct mm_struct {* See account_mm_sched() and ... */ struct mm_sched __percpu *pcpu_sched; +struct mm_time __percpu *pcpu_time;raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu;
@@ -1505,16 +1510,24 @@ static inline void mm_set_cpus_allowed(struct mm_struct *mm, const struct cpumas#endif /* CONFIG_SCHED_MM_CID */ #ifdef CONFIG_SCHED_CACHE -void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *pcpu_sched); +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *pcpu_sched, +struct mm_time __percpu *pcpu_time);static inline int mm_alloc_sched_noprof (struct mm_struct *mm) { struct mm_sched __percpu *pcpu_sched = alloc_percpu_noprof(struct mm_sched); +struct mm_time __percpu *pcpu_time;if (!pcpu_sched)return -ENOMEM ; -mm_init_sched (mm, pcpu_sched); + pcpu_time = alloc_percpu_noprof(struct mm_time); +if (!pcpu_time) { +free_percpu (mm->pcpu_sched); +return -ENOMEM ; + } + +mm_init_sched (mm, pcpu_sched, pcpu_time);return 0 ; }
@@ -1523,7 +1536,9 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)static inline void mm_destroy_sched (struct mm_struct *mm) {free_percpu (mm->pcpu_sched); +free_percpu (mm->pcpu_time); mm->pcpu_sched =NULL ; + mm->pcpu_time =NULL ; }#else /* !CONFIG_SCHED_CACHE */
include/linux/sched.h +3 -0
@@ -2442,6 +2442,9 @@ static inline bool sched_cache_enabled(void) {return static_branch_unlikely (&sched_cache_on); } + +int get_mm_per_llc_runtime (struct task_struct *p,u64 *buf); +extern int max_llcs;#endif #endif
kernel/sched/fair.c +38 -2
@@ -1355,16 +1355,19 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p) p->sched_llc_active = false; } -void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched) +void mm_init_sched (struct mm_struct *mm,struct mm_sched __percpu *_pcpu_sched, +struct mm_time __percpu *_pcpu_time) {unsigned long epoch;int i;for_each_possible_cpu (i) {struct mm_sched *pcpu_sched =per_cpu_ptr (_pcpu_sched, i); +struct mm_time *pcpu_time =per_cpu_ptr (_pcpu_time, i);struct rq *rq =cpu_rq (i); pcpu_sched->runtime =0 ; + pcpu_time->runtime_ns =0 ; pcpu_sched->epoch = rq->cpu_epoch; epoch = rq->cpu_epoch; }
@@ -1379,6 +1382,8 @@ void mm_init_sched(struct mm_struct *mm, struct mm_sched __percpu *_pcpu_sched)* the readers may get invalid mm_sched_epoch, etc. */ smp_store_release (&mm->pcpu_sched, _pcpu_sched); +/* same as above */ +smp_store_release (&mm->pcpu_time, _pcpu_time); }/* because why would C be fully specified */
@@ -1428,11 +1433,39 @@ static unsigned long __no_profile fraction_mm_sched(struct rq *rq, struct mm_schstatic unsigned int task_running_on_cpu (int cpu,struct task_struct *p); +/* p->pi_lock is hold */ +int get_mm_per_llc_runtime (struct task_struct *p,u64 *buf) +{ +struct mm_struct *mm = p->mm; +struct mm_time *pcpu_time; +int cpu; + +if (!mm) +return -EINVAL ; + +rcu_read_lock (); +for_each_online_cpu (cpu) { +int llc =llc_id (cpu); +u64 runtime_ms; + +if (llc <0 ) +continue ; + + pcpu_time =per_cpu_ptr (mm->pcpu_time, cpu); + runtime_ms =div_u64 (pcpu_time->runtime_ns,NSEC_PER_MSEC ); + buf[llc] += runtime_ms; + } +rcu_read_unlock (); + +return 0 ; +} +static inline void account_mm_sched (struct rq *rq,struct task_struct *p,s64 delta_exec) {struct mm_struct *mm = p->mm;struct mm_sched *pcpu_sched; +struct mm_time *pcpu_time;unsigned long epoch;int mm_sched_llc =-1 ;
@@ -1444,14 +1477,17 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)/* * init_task and kthreads don't having mm */ -if (!mm || !mm->pcpu_sched) +if (!mm || !mm->pcpu_sched || !mm->pcpu_time)return ; pcpu_sched =per_cpu_ptr (p->mm->pcpu_sched,cpu_of (rq)); + pcpu_time =per_cpu_ptr (p->mm->pcpu_time,cpu_of (rq)); scoped_guard (raw_spinlock, &rq->cpu_epoch_lock) {__update_mm_sched (rq, pcpu_sched); pcpu_sched->runtime += delta_exec; +/* pure runtime without decay */ + pcpu_time->runtime_ns += delta_exec; rq->cpu_runtime += delta_exec; epoch = rq->cpu_epoch; }
Signature · 1 line
2.32.0
Re: [PATCH v2 23/23] -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs
Quoted 44 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
Show the per-LLC occupancy in
/proc/{PID}/schedstat, with each column corresponding to one LLC. This can be used to verify if the cache-aware load balancer works as expected by aggregating threads onto dedicated LLCs.Suppose there are 2 LLCs and the sampling duration is 10 seconds:
Enable the cache aware load balance:
0 12281 <--- LLC0 residency delta is 0, LLC1 is 12 seconds 0 18881
0 16217disable the cache aware load balance:
6497 15802
9299 5435
17811 8278Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- fs/proc/base.c | 22 ++++++++++++++++++++++ include/linux/mm_types.h | 19 +++++++++++++++++-- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 80 insertions(+), 4 deletions(-)fs/proc/base.c +7
-0@@ -518,6 +518,28 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, (unsigned long long )task->se.sum_exec_runtime, (unsigned long long )task->sched_info.run_delay, task->sched_info.pcount); +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +struct mm_struct *mm = task->mm; +u64 *llc_runtime; + +if (!mm) +return 0 ;
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 537
Quoted 10 lines
fs/proc/base.c +10
-0+ + llc_runtime =kcalloc (max_llcs,sizeof (u64),GFP_KERNEL ); +if (!llc_runtime) +return 0 ; + +if (get_mm_per_llc_runtime (task, llc_runtime)) + goto out; + +for (int i =0 ; i < max_llcs; i++) +seq_printf (m,"%llu " , llc_runtime[i]);
I feel it is better to also mark the current preferred LLC of this process so that I can know how well it works.
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 545
Quoted 8 lines
fs/proc/base.c +5
-0+seq_puts (m,"\n" ); +out: + kfree(llc_runtime); + } +#endif return 0 ; }
BTW, is there a way to tell if a process is being taken care of by 'cache aware scheduling' or it's blocked due to its huge rss or having too many threads?
I used below debug code to get these info through schedstat, but maybe I missed something and there is a simpler method?
fs/proc/base.c +12 -1
@@ -505,6 +505,7 @@ static int proc_pid_stack(struct seq_file *m, struct pid_namespace *ns,#endif #ifdef CONFIG_SCHED_INFO +DECLARE_PER_CPU (int, sd_llc_id);/* * Provides /proc/PID/schedstat */
@@ -522,6 +523,7 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,if (sched_cache_enabled ()) {struct mm_struct *mm = task->mm;u64 *llc_runtime; +int mm_sched_llc;if (!mm)return 0 ;
@@ -533,8 +535,17 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,if (get_mm_per_llc_runtime (task, llc_runtime)) goto out; +if (mm->mm_sched_cpu ==-1 ) + mm_sched_llc =-1 ; +else + mm_sched_llc =per_cpu (sd_llc_id, mm->mm_sched_cpu); + +seq_printf (m,"%llu 0x%x\n" , mm->nr_running_avg, mm->mm_sched_flags);for (int i =0 ; i < max_llcs; i++) -seq_printf (m,"%llu " , llc_runtime[i]); +seq_printf (m,"%s%s%llu " , + i == task->preferred_llc ?"*" :"" , + i == mm_sched_llc ?"?" :"" , + llc_runtime[i]);seq_puts (m,"\n" ); out:kfree (llc_runtime);
include/linux/mm_types.h +1 -0
@@ -1048,6 +1048,7 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; +int mm_sched_flags;u64 nr_running_avg____cacheline_aligned_in_smp ;#endif
kernel/sched/fair.c +25 -3
@@ -1237,12 +1237,20 @@ static inline int get_sched_cache_scale(int mul)return (1 + (llc_aggr_tolerance -1 ) * mul); } +#define MM_SCHED_EXCEED_LLC_CAPACITY 1 +#define MM_SCHED_NO_CACHE_INFO 2 +#define MM_SCHED_EXCEED_LLC_NR 4 +#define MM_SCHED_NR_THREADS 8 +static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) {unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; + mm->mm_sched_flags &= ~MM_SCHED_NO_CACHE_INFO ; + mm->mm_sched_flags &= ~MM_SCHED_EXCEED_LLC_CAPACITY ; +/* * get_cpu_cacheinfo_level() can not be used * because it requires the cpu_hotplug_lock
@@ -1257,8 +1265,10 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)* L2 becomes the LLC. */ ci =_get_cpu_cacheinfo_level (cpu,2 ); -if (!ci) +if (!ci) { + mm->mm_sched_flags |=MM_SCHED_NO_CACHE_INFO ;return true; + } } llc = ci->size;
@@ -1283,13 +1293,20 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)if (scale ==INT_MAX )return false; -return ((llc * scale) <= (rss *PAGE_SIZE )); +if ((llc * scale) <= (rss *PAGE_SIZE )) { + mm->mm_sched_flags |=MM_SCHED_EXCEED_LLC_CAPACITY ; +return true; + } + +return false; }static bool exceed_llc_nr (struct mm_struct *mm,int cpu) {int smt_nr =1 , scale; + mm->mm_sched_flags &= ~MM_SCHED_EXCEED_LLC_NR ; +#ifdef CONFIG_SCHED_SMT if (sched_smt_active ()) smt_nr =cpumask_weight (cpu_smt_mask (cpu));
@@ -1313,7 +1330,12 @@ static bool exceed_llc_nr(struct mm_struct *mm, int cpu)if (scale ==INT_MAX )return false; -return ((mm->nr_running_avg * smt_nr) > (scale *per_cpu (sd_llc_size, cpu))); +if ((mm->nr_running_avg * smt_nr) > (scale *per_cpu (sd_llc_size, cpu))) { + mm->mm_sched_flags |=MM_SCHED_EXCEED_LLC_NR ; +return true; + } + +return false; }static void account_llc_enqueue (struct rq *rq,struct task_struct *p)
Re: [PATCH v2 23/23] -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs
Quoted 50 lines · Aaron Lu
Quoted 49 lines · Tim Chen
From: Chen Yu <[email protected]>
Debug patch only.
Show the per-LLC occupancy in
/proc/{PID}/schedstat, with each column corresponding to one LLC. This can be used to verify if the cache-aware load balancer works as expected by aggregating threads onto dedicated LLCs.Suppose there are 2 LLCs and the sampling duration is 10 seconds:
Enable the cache aware load balance:
0 12281 <--- LLC0 residency delta is 0, LLC1 is 12 seconds 0 18881
0 16217disable the cache aware load balance:
6497 15802
9299 5435
17811 8278Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>--- fs/proc/base.c | 22 ++++++++++++++++++++++ include/linux/mm_types.h | 19 +++++++++++++++++-- include/linux/sched.h | 3 +++ kernel/sched/fair.c | 40 ++++++++++++++++++++++++++++++++++++++-- 4 files changed, 80 insertions(+), 4 deletions(-)fs/proc/base.c +12
-0@@ -518,6 +518,28 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, (unsigned long long )task->se.sum_exec_runtime, (unsigned long long )task->sched_info.run_delay, task->sched_info.pcount); +#ifdef CONFIG_SCHED_CACHE +if (sched_cache_enabled ()) { +struct mm_struct *mm = task->mm; +u64 *llc_runtime; + +if (!mm) +return 0 ; + + llc_runtime =kcalloc (max_llcs,sizeof (u64),GFP_KERNEL ); +if (!llc_runtime) +return 0 ; +
Quoted 9 lines
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 537
Quoted 5 lines
fs/proc/base.c +5
-0+if (get_mm_per_llc_runtime (task, llc_runtime)) + goto out; + +for (int i =0 ; i < max_llcs; i++) +seq_printf (m,"%llu " , llc_runtime[i]);I feel it is better to also mark the current preferred LLC of this process so that I can know how well it works.
Sure.
Quoted 37 lines
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 545
Quoted 8 lines
fs/proc/base.c +5
-0+seq_puts (m,"\n" ); +out: + kfree(llc_runtime); + } +#endif return 0 ; }BTW, is there a way to tell if a process is being taken care of by 'cache aware scheduling' or it's blocked due to its huge rss or having too many threads?
I used below debug code to get these info through schedstat, but maybe I missed something and there is a simpler method?
fs/proc/base.c +2
-0@@ -505,6 +505,7 @@ static int proc_pid_stack(struct seq_file *m, struct pid_namespace *ns,#endif #ifdef CONFIG_SCHED_INFO +DECLARE_PER_CPU (int, sd_llc_id);/* * Provides /proc/PID/schedstat */ @@ -522,6 +523,7 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,if (sched_cache_enabled ()) {struct mm_struct *mm = task->mm;u64 *llc_runtime; +int mm_sched_llc;if (!mm)return 0 ;
fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 541
Quoted 7 lines
fs/proc/base.c +4
-0@@ -533,8 +535,17 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,if (get_mm_per_llc_runtime (task, llc_runtime)) goto out; +if (mm->mm_sched_cpu ==-1 ) + mm_sched_llc =-1 ; +else + mm_sched_llc =per_cpu (sd_llc_id, mm->mm_sched_cpu);
We can use llc_id(mm->mm_sched_cpu).
Quoted 35 lines
fs/proc/base.c +6
-1+ +seq_printf (m,"%llu 0x%x\n" , mm->nr_running_avg, mm->mm_sched_flags);for (int i =0 ; i < max_llcs; i++) -seq_printf (m,"%llu " , llc_runtime[i]); +seq_printf (m,"%s%s%llu " , + i == task->preferred_llc ?"*" :"" , + i == mm_sched_llc ?"?" :"" , + llc_runtime[i]);seq_puts (m,"\n" ); out:kfree (llc_runtime);include/linux/mm_types.h +1
-0@@ -1048,6 +1048,7 @@ struct mm_struct {raw_spinlock_t mm_sched_lock;unsigned long mm_sched_epoch;int mm_sched_cpu; +int mm_sched_flags;u64 nr_running_avg____cacheline_aligned_in_smp ;#endif kernel/sched/fair.c +4
-0@@ -1237,12 +1237,20 @@ static inline int get_sched_cache_scale(int mul)return (1 + (llc_aggr_tolerance -1 ) * mul); } +#define MM_SCHED_EXCEED_LLC_CAPACITY 1 +#define MM_SCHED_NO_CACHE_INFO 2 +#define MM_SCHED_EXCEED_LLC_NR 4 +#define MM_SCHED_NR_THREADS 8
kernel/sched/fair.c static inline int get_sched_cache_scale(int mul) · line 1253
Quoted 10 lines
kernel/sched/fair.c +4
-0+static bool exceed_llc_capacity (struct mm_struct *mm,int cpu) {unsigned int llc, scale;struct cacheinfo *ci;unsigned long rss; + mm->mm_sched_flags &= ~MM_SCHED_NO_CACHE_INFO ; + mm->mm_sched_flags &= ~MM_SCHED_EXCEED_LLC_CAPACITY ; +
Maybe we can do some read-comparison before writing the flags, previously we found that writing the per-process mm struct is very expensive, so maybe avoid writing to it as much as possible.
I'll fold your changes and do the test. Thanks!
Thanks,
Chenyu
On Wed, Dec 03, 2025 at 03:07:19PM -0800, Tim Chen wrote:
... ...
Quoted 23 lines
Test results:
The patch series was applied and tested on v6.18-rc7.
See: https://github.com/timcchen1298/linux/commits/cache_aware_v2The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.
The second test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.
hackbench/schbench/netperf/stream/stress-ng/chacha20 were launched on these two platforms.
[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when the number of different active threads is below the capacity of a LLC. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows good throughput improvement.
Quoted 3 lines
Genoa:
ChaCha20-xiangshan shows huge throughput improvement. No obvious difference is observed in hackbench/schbench
I think for small task number hackbench run, there should be some improvement.
I tried thread/pipe/2fds/1group, i.e. 4 tasks on Genoa:
./hackbench -T -f 2 -g 1 -p -l 2000000
And I noticed performance improved a lot:
(Result in seconds, less is better)
llc_off llc_on diff
time 4.755±1.6% 2.684±6.25% +43.6%
llc_off means /sys/kernel/debug/sched/llc_enabled set to 0 while llc_on means /sys/kernel/debug/sched/llc_enabled set to 1, other tunnables are left unchanged.
Turbo is disabled and cpufreq set to performance.
I also tried redis and noticed when I set io-threads to 4 in redis.conf, there is also some improvement on AMD Genoa:
llc_off manual diff llc_on diff
throughput 1536727±0% 1737619±0% +13.1% 1737720±0% +13.1%
Client cmdline:
numactl -N 1 redis-benchmark --threads 4 -t set -r 100000 -P 16 -n 10000000 server cmdline: numactl -N 0 redis-server ./redis.conf
I also tried to manually bind all tasks of redis server to a single LLC to see if this workload benefits from aggregation and that's what manual
means: taskset -c 8-15,200-207 redis-server ./redis.conf
According to the result, I think this 'cache aware scheduling' works as expected in that its performance is the same as manual binding; and they all beat llc_off.
Quoted 58 lines · Aaron Lu
On Wed, Dec 03, 2025 at 03:07:19PM -0800, Tim Chen wrote:
... ...Quoted 23 lines
Test results:
The patch series was applied and tested on v6.18-rc7.
See: https://github.com/timcchen1298/linux/commits/cache_aware_v2The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.
The second test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.
hackbench/schbench/netperf/stream/stress-ng/chacha20 were launched on these two platforms.
[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when the number of different active threads is below the capacity of a LLC. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows good throughput improvement.Quoted 3 lines
Genoa:
ChaCha20-xiangshan shows huge throughput improvement. No obvious difference is observed in hackbench/schbenchI think for small task number hackbench run, there should be some improvement.
I tried thread/pipe/2fds/1group, i.e. 4 tasks on Genoa:
./hackbench -T -f 2 -g 1 -p -l 2000000
And I noticed performance improved a lot:
(Result in seconds, less is better)llc_off llc_on diff time 4.755±1.6% 2.684±6.25% +43.6%
llc_offmeans/sys/kernel/debug/sched/llc_enabledset to 0 whilellc_onmeans/sys/kernel/debug/sched/llc_enabledset to 1, other tunnables are left unchanged.
Turbo is disabled and cpufreq set to performance.I also tried redis and noticed when I set io-threads to 4 in redis.conf, there is also some improvement on AMD Genoa:
llc_off manual diff llc_on diff throughput 1536727±0% 1737619±0% +13.1% 1737720±0% +13.1%Client cmdline:
numactl -N 1 redis-benchmark --threads 4 -t set -r 100000 -P 16 -n 10000000 server cmdline: numactl -N 0 redis-server ./redis.conf
I also tried to manually bind all tasks of redis server to a single LLC to see if this workload benefits from aggregation and that's what manual
means: taskset -c 8-15,200-207 redis-server ./redis.conf
Quoted 3 lines
According to the result, I think this 'cache aware scheduling' works as expected in that its performance is the same as manual binding; and they all beat
llc_off.
Thanks Aaron for the test!
thanks,
Chenyu