Skip to contentSkip to conversation

Cache aware scheduling

PATCH In review Tim Chen 8 participants 120 messages Dec 3, 2025 – Jan 22 1 message not in the archive Page 2 of 2

Subsystems (5)
lore
Share & tools
lore
#110 lore raw

Re: [PATCH v2 20/23] sched/cache: Add user control to adjust the parameters of cache-aware scheduling

In reply to Yangyu Chen

Hi Yangyu,

Quoted 3 lines · Yangyu Chen

On 4/12/2025 07:07, Tim Chen wrote:

[ ... ]

Quoted 8 lines
Quoted 1 line

+    scale = get_sched_cache_scale(256);

Hi Tim Chen and Chen Yu,

There's an integer overflow here. Since the unit of LLC size is bytes, you have a 256-scale unit. For a typical LLC size of 32M, you calculate 32M multiplied by 256, which equals 8GB. This value exceeds the maximum integer value (2GB, INT_MAX), resulting in an integer overflow.

I think such function should be use u64. Below is my patch:

Thanks very much for the investigation, Jianyong previously also mentioned this issue
https://lore.kernel.org/all/SI2PR04MB49317BA503E9C8A381956D6AE38CA@SI2PR04MB4931.apcprd04.prod.outlook.com/
we will fix the issue accordingly.

Quoted 15 lines
kernel/sched/fair.c +1 -1
index 205208f061bb..bcafb3c2b369 100644
@@ -1226,20 +1226,20 @@ static int llc_id(int cpu)
      return llc;
  }
 
-static inline int get_sched_cache_scale(int mul)
+static inline u64 get_sched_cache_scale(int mul)
  {
      if (!llc_aggr_tolerance)
          return 0;
 
      if (llc_aggr_tolerance == 100)

kernel/sched/fair.c static int llc_id(int cpu) · line 1242

Quoted 10 lines
kernel/sched/fair.c +2 -2
-        return INT_MAX;
+        return ULLONG_MAX;
 
      return (1 + (llc_aggr_tolerance - 1) * mul);
  }
 
  static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)
  {
-    unsigned int llc, scale;
+    unsigned long long llc, scale;

I suppose we only need to change llc to u64, and not change scale, because (llc * scale) would be converted to u64 anyway.

thanks,
Chenyu

author
#111 lore raw

PATCH v2 21/23 -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing

In reply to Tim Chen

From: Chen Yu <[email protected]>

Debug patch only.

With cache-aware load balancing enabled, statistics related to its activity are exposed via /proc/schedstat and debugfs. For instance, if users want to verify metrics like the number of exceeding RSS and nr_running limits, they can filter the output of /sys/kernel/debug/sched/debug and compute the required statistics manually:

llc_exceed_cap SUM: 6
llc_exceed_nr SUM: 4531

Furthermore, these statistics exposed in /proc/schedstats can be queried manually or via perf sched stats[1] with minor modifications.

---
 include/linux/sched/topology.h | 1 +
 kernel/sched/fair.c            | 1 +
 kernel/sched/stats.c           | 5 +++--
 3 files changed, 5 insertions(+), 2 deletions(-)
include/linux/sched/topology.h +1 -0
index 0ba4697d74ba..8702c1e731a0 100644
@@ -108,6 +108,7 @@ struct sched_domain {
 	unsigned int lb_imbalance_util[CPU_MAX_IDLE_TYPES];
 	unsigned int lb_imbalance_task[CPU_MAX_IDLE_TYPES];
 	unsigned int lb_imbalance_misfit[CPU_MAX_IDLE_TYPES];
+	unsigned int lb_imbalance_llc[CPU_MAX_IDLE_TYPES];
 	unsigned int lb_gained[CPU_MAX_IDLE_TYPES];
 	unsigned int lb_hot_gained[CPU_MAX_IDLE_TYPES];
 	unsigned int lb_nobusyg[CPU_MAX_IDLE_TYPES];
kernel/sched/fair.c +1 -0
index a2e2d6742481..742e455b093e 100644
@@ -12684,6 +12684,7 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sd
 		__schedstat_add(sd->lb_imbalance_misfit[idle], env->imbalance);
 		break;
 	case migrate_llc_task:
+		__schedstat_add(sd->lb_imbalance_llc[idle], env->imbalance);
 		break;
 	}
 }
kernel/sched/stats.c +3 -2
index d1c9429a4ac5..3736f6102261 100644
@@ -104,7 +104,7 @@ void __update_stats_enqueue_sleeper(struct rq *rq, struct task_struct *p,
  * Bump this up when changing the output format or the meaning of an existing
  * format, so that tools can adapt (or abort)
  */
-#define SCHEDSTAT_VERSION 17
+#define SCHEDSTAT_VERSION 18
 
 static int show_schedstat(struct seq_file *seq, void *v)
 {
@@ -139,7 +139,7 @@ static int show_schedstat(struct seq_file *seq, void *v)
 			seq_printf(seq, "domain%d %s %*pb", dcount++, sd->name,
 				   cpumask_pr_args(sched_domain_span(sd)));
 			for (itype = 0; itype < CPU_MAX_IDLE_TYPES; itype++) {
-				seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u",
+				seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u %u",
 				    sd->lb_count[itype],
 				    sd->lb_balanced[itype],
 				    sd->lb_failed[itype],
@@ -147,6 +147,7 @@ static int show_schedstat(struct seq_file *seq, void *v)
 				    sd->lb_imbalance_util[itype],
 				    sd->lb_imbalance_task[itype],
 				    sd->lb_imbalance_misfit[itype],
+				    sd->lb_imbalance_llc[itype],
 				    sd->lb_gained[itype],
 				    sd->lb_hot_gained[itype],
 				    sd->lb_nobusyq[itype],
Signature · 1 line
2.32.0
#112 lore raw

Re: [PATCH v2 21/23] -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing

Quoted 13 lines · Tim Chen

From: Chen Yu <[email protected]>

Debug patch only.

With cache-aware load balancing enabled, statistics related to its activity are exposed via /proc/schedstat and debugfs. For instance, if users want to verify metrics like the number of exceeding RSS and nr_running limits, they can filter the output of /sys/kernel/debug/sched/debug and compute the required statistics manually:

Quoted 6 lines

llc_exceed_cap SUM: 6
llc_exceed_nr SUM: 4531

Furthermore, these statistics exposed in /proc/schedstats can be queried manually or via perf sched stats[1] with minor modifications.

Hi Tim,

This patch looks great, especially for multithread Verilator workloads on clustered LLC (like AMD EPYC). I'm discussing with Verilator upstream to disable automatic userspace affinity assignment in Verilator if such feature exist [1]. During the discussion, I think there should be a way for userspace software to detect if such a feature exists. Could we expose it in /proc/schedstats to allow userspace software to detect such a feature? We can just use this patch and remove the "DO NOT APPLY" tag.

[1] https://github.com/verilator/verilator/issues/6826#issuecomment-3671287551

Thanks,
Yangyu Chen

Quoted 66 lines

---

include/linux/sched/topology.h | 1 +
kernel/sched/fair.c            | 1 +
kernel/sched/stats.c           | 5 +++--
3 files changed, 5 insertions(+), 2 deletions(-)
include/linux/sched/topology.h +1 -0
index 0ba4697d74ba..8702c1e731a0 100644
@@ -108,6 +108,7 @@ struct sched_domain {
 unsigned int lb_imbalance_util[CPU_MAX_IDLE_TYPES];
 unsigned int lb_imbalance_task[CPU_MAX_IDLE_TYPES];
 unsigned int lb_imbalance_misfit[CPU_MAX_IDLE_TYPES];
+ unsigned int lb_imbalance_llc[CPU_MAX_IDLE_TYPES];
 unsigned int lb_gained[CPU_MAX_IDLE_TYPES];
 unsigned int lb_hot_gained[CPU_MAX_IDLE_TYPES];
 unsigned int lb_nobusyg[CPU_MAX_IDLE_TYPES];
kernel/sched/fair.c +0 -0
index a2e2d6742481..742e455b093e 100644
@@ -12684,6 +12684,7 @@ static void update_lb_imbalance_stat(struct lb_env *env, struct sched_domain *sd
 __schedstat_add(sd->lb_imbalance_misfit[idle], env->imbalance);
 break;

case migrate_llc_task:
+ __schedstat_add(sd->lb_imbalance_llc[idle], env->imbalance); break;
}
}

kernel/sched/stats.c +3 -2
index d1c9429a4ac5..3736f6102261 100644
@@ -104,7 +104,7 @@ void __update_stats_enqueue_sleeper(struct rq *rq, struct task_struct *p,
 * Bump this up when changing the output format or the meaning of an existing
 * format, so that tools can adapt (or abort)
 */
-#define SCHEDSTAT_VERSION 17
+#define SCHEDSTAT_VERSION 18
 
 static int show_schedstat(struct seq_file *seq, void *v)
 {
@@ -139,7 +139,7 @@ static int show_schedstat(struct seq_file *seq, void *v)
 seq_printf(seq, "domain%d %s %*pb", dcount++, sd->name,
  cpumask_pr_args(sched_domain_span(sd)));
 for (itype = 0; itype < CPU_MAX_IDLE_TYPES; itype++) {
- seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u",
+ seq_printf(seq, " %u %u %u %u %u %u %u %u %u %u %u %u",
   sd->lb_count[itype],
   sd->lb_balanced[itype],
   sd->lb_failed[itype],
@@ -147,6 +147,7 @@ static int show_schedstat(struct seq_file *seq, void *v)
   sd->lb_imbalance_util[itype],
   sd->lb_imbalance_task[itype],
   sd->lb_imbalance_misfit[itype],
+    sd->lb_imbalance_llc[itype],
   sd->lb_gained[itype],
   sd->lb_hot_gained[itype],
   sd->lb_nobusyq[itype],
Signature · 1 line
2.32.0
#113 lore raw

Re: [PATCH v2 21/23] -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing

Quoted 22 lines · Yangyu Chen
Quoted 13 lines · Tim Chen

From: Chen Yu <[email protected]>

Debug patch only.

With cache-aware load balancing enabled, statistics related to its activity are exposed via /proc/schedstat and debugfs. For instance, if users want to verify metrics like the number of exceeding RSS and nr_running limits, they can filter the output of /sys/kernel/debug/sched/debug and compute the required statistics manually:

Quoted 6 lines

llc_exceed_cap SUM: 6
llc_exceed_nr SUM: 4531

Furthermore, these statistics exposed in /proc/schedstats can be queried manually or via perf sched stats[1] with minor modifications.

Hi Tim,

Quoted 9 lines

This patch looks great, especially for multithread Verilator workloads on clustered LLC (like AMD EPYC). I'm discussing with Verilator upstream to disable automatic userspace affinity assignment in Verilator if such feature exist [1]. During the discussion, I think there should be a way for userspace software to detect if such a feature exists. Could we expose it in /proc/schedstats to allow userspace software to detect such a feature? We can just use this patch and remove the "DO NOT APPLY" tag.

Thanks for the test Yangyu. Does /sys/kernel/debug/sched/llc_enabled work for you?
Anyway we can try to include /proc/schedstats as the formal one in the next version.

Thanks,
Chenyu

Quoted 7 lines

[1] https://github.com/verilator/verilator/issues/6826#issuecomment-3671287551

Thanks,
Yangyu Chen

#114 lore raw

Re: [PATCH v2 21/23] -- DO NOT APPLY!!! -- sched/cache/stats: Add schedstat for cache aware load balancing

Quoted 33 lines · Yu C
Quoted 18 lines · Yangyu Chen
Quoted 12 lines · Tim Chen

From: Chen Yu <[email protected]>

Debug patch only.

With cache-aware load balancing enabled, statistics related to its activity are exposed via /proc/schedstat and debugfs. For instance, if users want to verify metrics like the number of exceeding RSS and nr_running limits, they can filter the output of /sys/kernel/debug/sched/debug and compute the required statistics manually:

Quoted 6 lines

llc_exceed_cap SUM: 6
llc_exceed_nr SUM: 4531

Furthermore, these statistics exposed in /proc/schedstats can be queried manually or via perf sched stats[1] with minor modifications.

Quoted 9 lines

Hi Tim,
This patch looks great, especially for multithread Verilator workloads on clustered LLC (like AMD EPYC). I'm discussing with Verilator upstream to disable automatic userspace affinity assignment in Verilator if such feature exist [1]. During the discussion, I think there should be a way for userspace software to detect if such a feature exists. Could we expose it in /proc/schedstats to allow userspace software to detect such a feature? We can just use this patch and remove the "DO NOT APPLY" tag.

Quoted 2 lines

Thanks for the test Yangyu. Does /sys/kernel/debug/sched/llc_enabled work for you?

It requires debugfs being mounted with enough permissions. It’s not feasible for normal user-space software without root permission.

Thanks,
Yangyu Chen

Quoted 9 lines

Anyway we can try to include /proc/schedstats as the formal one in the next version.

Thanks,
Chenyu

Quoted 3 lines

[1] https://github.com/verilator/verilator/issues/6826#issuecomment-3671287551
Thanks,
Yangyu Chen

author
#115 lore raw

PATCH v2 22/23 -- DO NOT APPLY!!! -- sched/cache/debug: Add ftrace to track the load balance statistics

In reply to Tim Chen

From: Chen Yu <[email protected]>

Debug patch only.

The user leverages this trace event (via bpftrace, etc)to monitor the cache aware load balance activity - whether the tasks are moved to their preferred LLC, or moved out of their preferred LLC.

---
 include/trace/events/sched.h | 31 +++++++++++++++++++++++++++++++
 kernel/sched/fair.c          | 10 ++++++++++
 2 files changed, 41 insertions(+)
include/trace/events/sched.h +31 -0
index 7b2645b50e78..bd03f49f7e3c 100644
@@ -10,6 +10,37 @@
 #include <linux/tracepoint.h>
 #include <linux/binfmts.h>
 
+TRACE_EVENT(sched_attach_task,
+
+	TP_PROTO(struct task_struct *t, int pref_cpu, int pref_llc,
+		 int attach_cpu, int attach_llc),
+
+	TP_ARGS(t, pref_cpu, pref_llc, attach_cpu, attach_llc),
+
+	TP_STRUCT__entry(
+			__array(	char,	comm,	TASK_COMM_LEN	)
+			__field(	pid_t,	pid			)
+			__field(	int,	pref_cpu		)
+			__field(	int,	pref_llc		)
+			__field(	int,	attach_cpu		)
+			__field(	int,	attach_llc		)
+	),
+
+	TP_fast_assign(
+		      memcpy(__entry->comm, t->comm, TASK_COMM_LEN);
+		      __entry->pid	= t->pid;
+		      __entry->pref_cpu	= pref_cpu;
+		      __entry->pref_llc	= pref_llc;
+		      __entry->attach_cpu	= attach_cpu;
+		      __entry->attach_llc	= attach_llc;
+	),
+
+	TP_printk("comm=%s pid=%d pref_cpu=%d pref_llc=%d attach_cpu=%d attach_llc=%d",
+		  __entry->comm, __entry->pid,
+		  __entry->pref_cpu, __entry->pref_llc,
+		  __entry->attach_cpu, __entry->attach_llc)
+);
+
 /*
  * Tracepoint for calling kthread_stop, performed to end a kthread:
  */
kernel/sched/fair.c +10 -0
index 742e455b093e..e47b4096f0a6 100644
@@ -10487,6 +10487,16 @@ static void attach_task(struct rq *rq, struct task_struct *p)
 {
 	lockdep_assert_rq_held(rq);
 
+#ifdef CONFIG_SCHED_CACHE
+	if (p->mm) {
+		int pref_cpu = p->mm->mm_sched_cpu;
+
+		trace_sched_attach_task(p,
+					pref_cpu,
+					pref_cpu != -1 ? llc_id(pref_cpu) : -1,
+					cpu_of(rq), llc_id(cpu_of(rq)));
+	}
+#endif
 	WARN_ON_ONCE(task_rq(p) != rq);
 	activate_task(rq, p, ENQUEUE_NOCLOCK);
 	wakeup_preempt(rq, p, 0);
Signature · 1 line
2.32.0
author
#116 lore raw

PATCH v2 23/23 -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs

In reply to Tim Chen

From: Chen Yu <[email protected]>

Debug patch only.

Show the per-LLC occupancy in /proc/{PID}/schedstat, with each column corresponding to one LLC. This can be used to verify if the cache-aware load balancer works as expected by aggregating threads onto dedicated LLCs.

Suppose there are 2 LLCs and the sampling duration is 10 seconds:

Enable the cache aware load balance:
0 12281 <--- LLC0 residency delta is 0, LLC1 is 12 seconds 0 18881
0 16217

disable the cache aware load balance:
6497 15802
9299 5435
17811 8278

---
 fs/proc/base.c           | 22 ++++++++++++++++++++++
 include/linux/mm_types.h | 19 +++++++++++++++++--
 include/linux/sched.h    |  3 +++
 kernel/sched/fair.c      | 40 ++++++++++++++++++++++++++++++++++++++--
 4 files changed, 80 insertions(+), 4 deletions(-)
fs/proc/base.c +22 -0
index 6299878e3d97..f4be96f4bd01 100644
@@ -518,6 +518,28 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,
 		   (unsigned long long)task->se.sum_exec_runtime,
 		   (unsigned long long)task->sched_info.run_delay,
 		   task->sched_info.pcount);
+#ifdef CONFIG_SCHED_CACHE
+	if (sched_cache_enabled()) {
+		struct mm_struct *mm = task->mm;
+		u64 *llc_runtime;
+
+		if (!mm)
+			return 0;
+
+		llc_runtime = kcalloc(max_llcs, sizeof(u64), GFP_KERNEL);
+		if (!llc_runtime)
+			return 0;
+
+		if (get_mm_per_llc_runtime(task, llc_runtime))
+			goto out;
+
+		for (int i = 0; i < max_llcs; i++)
+			seq_printf(m, "%llu ", llc_runtime[i]);
+		seq_puts(m, "\n");
+out:
+		kfree(llc_runtime);
+	}
+#endif
 
 	return 0;
 }
include/linux/mm_types.h +17 -2
index 04743983de4d..255c22be7312 100644
@@ -944,6 +944,10 @@ struct mm_sched {
 	unsigned long epoch;
 };
 
+struct mm_time {
+	u64 runtime_ns;
+};
+
 struct kioctx_table;
 struct iommu_mm_data;
 struct mm_struct {
@@ -1040,6 +1044,7 @@ struct mm_struct {
 		 * See account_mm_sched() and ...
 		 */
 		struct mm_sched __percpu *pcpu_sched;
+		struct mm_time __percpu *pcpu_time;
 		raw_spinlock_t mm_sched_lock;
 		unsigned long mm_sched_epoch;
 		int mm_sched_cpu;
@@ -1505,16 +1510,24 @@ static inline void mm_set_cpus_allowed(struct mm_struct *mm, const struct cpumas
 #endif /* CONFIG_SCHED_MM_CID */
 
 #ifdef CONFIG_SCHED_CACHE
-void mm_init_sched(struct mm_struct *mm, struct mm_sched __percpu *pcpu_sched);
+void mm_init_sched(struct mm_struct *mm, struct mm_sched __percpu *pcpu_sched,
+		   struct mm_time __percpu *pcpu_time);
 
 static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
 {
 	struct mm_sched __percpu *pcpu_sched = alloc_percpu_noprof(struct mm_sched);
+	struct mm_time __percpu *pcpu_time;
 
 	if (!pcpu_sched)
 		return -ENOMEM;
 
-	mm_init_sched(mm, pcpu_sched);
+	pcpu_time = alloc_percpu_noprof(struct mm_time);
+	if (!pcpu_time) {
+		free_percpu(mm->pcpu_sched);
+		return -ENOMEM;
+	}
+
+	mm_init_sched(mm, pcpu_sched, pcpu_time);
 	return 0;
 }
 
@@ -1523,7 +1536,9 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
 static inline void mm_destroy_sched(struct mm_struct *mm)
 {
 	free_percpu(mm->pcpu_sched);
+	free_percpu(mm->pcpu_time);
 	mm->pcpu_sched = NULL;
+	mm->pcpu_time = NULL;
 }
 #else /* !CONFIG_SCHED_CACHE */
 
include/linux/sched.h +3 -0
index 95bf080bbbf0..875ac3f4208b 100644
@@ -2442,6 +2442,9 @@ static inline bool sched_cache_enabled(void)
 {
 	return static_branch_unlikely(&sched_cache_on);
 }
+
+int get_mm_per_llc_runtime(struct task_struct *p, u64 *buf);
+extern int max_llcs;
 #endif
 
 #endif
kernel/sched/fair.c +38 -2
index e47b4096f0a6..205208f061bb 100644
@@ -1355,16 +1355,19 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p)
 	p->sched_llc_active = false;
 }
 
-void mm_init_sched(struct mm_struct *mm, struct mm_sched __percpu *_pcpu_sched)
+void mm_init_sched(struct mm_struct *mm, struct mm_sched __percpu *_pcpu_sched,
+		   struct mm_time __percpu *_pcpu_time)
 {
 	unsigned long epoch;
 	int i;
 
 	for_each_possible_cpu(i) {
 		struct mm_sched *pcpu_sched = per_cpu_ptr(_pcpu_sched, i);
+		struct mm_time *pcpu_time = per_cpu_ptr(_pcpu_time, i);
 		struct rq *rq = cpu_rq(i);
 
 		pcpu_sched->runtime = 0;
+		pcpu_time->runtime_ns = 0;
 		pcpu_sched->epoch = rq->cpu_epoch;
 		epoch = rq->cpu_epoch;
 	}
@@ -1379,6 +1382,8 @@ void mm_init_sched(struct mm_struct *mm, struct mm_sched __percpu *_pcpu_sched)
 	 * the readers may get invalid mm_sched_epoch, etc.
 	 */
 	smp_store_release(&mm->pcpu_sched, _pcpu_sched);
+	/* same as above */
+	smp_store_release(&mm->pcpu_time, _pcpu_time);
 }
 
 /* because why would C be fully specified */
@@ -1428,11 +1433,39 @@ static unsigned long __no_profile fraction_mm_sched(struct rq *rq, struct mm_sch
 
 static unsigned int task_running_on_cpu(int cpu, struct task_struct *p);
 
+/* p->pi_lock is hold */
+int get_mm_per_llc_runtime(struct task_struct *p, u64 *buf)
+{
+	struct mm_struct *mm = p->mm;
+	struct mm_time *pcpu_time;
+	int cpu;
+
+	if (!mm)
+		return -EINVAL;
+
+	rcu_read_lock();
+	for_each_online_cpu(cpu) {
+		int llc = llc_id(cpu);
+		u64 runtime_ms;
+
+		if (llc < 0)
+			continue;
+
+		pcpu_time = per_cpu_ptr(mm->pcpu_time, cpu);
+		runtime_ms = div_u64(pcpu_time->runtime_ns, NSEC_PER_MSEC);
+		buf[llc] += runtime_ms;
+	}
+	rcu_read_unlock();
+
+	return 0;
+}
+
 static inline
 void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 {
 	struct mm_struct *mm = p->mm;
 	struct mm_sched *pcpu_sched;
+	struct mm_time *pcpu_time;
 	unsigned long epoch;
 	int mm_sched_llc = -1;
 
@@ -1444,14 +1477,17 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)
 	/*
 	 * init_task and kthreads don't having mm
 	 */
-	if (!mm || !mm->pcpu_sched)
+	if (!mm || !mm->pcpu_sched || !mm->pcpu_time)
 		return;
 
 	pcpu_sched = per_cpu_ptr(p->mm->pcpu_sched, cpu_of(rq));
+	pcpu_time = per_cpu_ptr(p->mm->pcpu_time, cpu_of(rq));
 
 	scoped_guard (raw_spinlock, &rq->cpu_epoch_lock) {
 		__update_mm_sched(rq, pcpu_sched);
 		pcpu_sched->runtime += delta_exec;
+		/* pure runtime without decay */
+		pcpu_time->runtime_ns += delta_exec;
 		rq->cpu_runtime += delta_exec;
 		epoch = rq->cpu_epoch;
 	}
Signature · 1 line
2.32.0
#117 lore raw

Re: [PATCH v2 23/23] -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs

Quoted 44 lines · Tim Chen

From: Chen Yu <[email protected]>

Debug patch only.

Show the per-LLC occupancy in /proc/{PID}/schedstat, with each column corresponding to one LLC. This can be used to verify if the cache-aware load balancer works as expected by aggregating threads onto dedicated LLCs.

Suppose there are 2 LLCs and the sampling duration is 10 seconds:

Enable the cache aware load balance:
0 12281 <--- LLC0 residency delta is 0, LLC1 is 12 seconds 0 18881
0 16217

disable the cache aware load balance:
6497 15802
9299 5435
17811 8278

---
 fs/proc/base.c           | 22 ++++++++++++++++++++++
 include/linux/mm_types.h | 19 +++++++++++++++++--
 include/linux/sched.h    |  3 +++
 kernel/sched/fair.c      | 40 ++++++++++++++++++++++++++++++++++++++--
 4 files changed, 80 insertions(+), 4 deletions(-)
fs/proc/base.c +7 -0
index 6299878e3d97..f4be96f4bd01 100644
@@ -518,6 +518,28 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,
 		   (unsigned long long)task->se.sum_exec_runtime,
 		   (unsigned long long)task->sched_info.run_delay,
 		   task->sched_info.pcount);
+#ifdef CONFIG_SCHED_CACHE
+	if (sched_cache_enabled()) {
+		struct mm_struct *mm = task->mm;
+		u64 *llc_runtime;
+
+		if (!mm)
+			return 0;

fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 537

Quoted 10 lines
fs/proc/base.c +10 -0
+
+		llc_runtime = kcalloc(max_llcs, sizeof(u64), GFP_KERNEL);
+		if (!llc_runtime)
+			return 0;
+
+		if (get_mm_per_llc_runtime(task, llc_runtime))
+			goto out;
+
+		for (int i = 0; i < max_llcs; i++)
+			seq_printf(m, "%llu ", llc_runtime[i]);

I feel it is better to also mark the current preferred LLC of this process so that I can know how well it works.

fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 545

Quoted 8 lines
fs/proc/base.c +5 -0
+		seq_puts(m, "\n");
+out:
+		kfree(llc_runtime);
+	}
+#endif
 
 	return 0;
 }

BTW, is there a way to tell if a process is being taken care of by 'cache aware scheduling' or it's blocked due to its huge rss or having too many threads?

I used below debug code to get these info through schedstat, but maybe I missed something and there is a simpler method?

fs/proc/base.c +12 -1
index f4be96f4bd015..c709a1a1bd867 100644
@@ -505,6 +505,7 @@ static int proc_pid_stack(struct seq_file *m, struct pid_namespace *ns,
 #endif
 
 #ifdef CONFIG_SCHED_INFO
+DECLARE_PER_CPU(int, sd_llc_id);
 /*
  * Provides /proc/PID/schedstat
  */
@@ -522,6 +523,7 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,
 	if (sched_cache_enabled()) {
 		struct mm_struct *mm = task->mm;
 		u64 *llc_runtime;
+		int mm_sched_llc;
 
 		if (!mm)
 			return 0;
@@ -533,8 +535,17 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,
 		if (get_mm_per_llc_runtime(task, llc_runtime))
 			goto out;
 
+		if (mm->mm_sched_cpu == -1)
+			mm_sched_llc = -1;
+		else
+			mm_sched_llc = per_cpu(sd_llc_id, mm->mm_sched_cpu);
+
+		seq_printf(m, "%llu 0x%x\n", mm->nr_running_avg, mm->mm_sched_flags);
 		for (int i = 0; i < max_llcs; i++)
-			seq_printf(m, "%llu ", llc_runtime[i]);
+			seq_printf(m, "%s%s%llu ",
+				   i == task->preferred_llc ? "*" : "",
+				   i == mm_sched_llc ? "?" : "",
+				   llc_runtime[i]);
 		seq_puts(m, "\n");
 out:
 		kfree(llc_runtime);
include/linux/mm_types.h +1 -0
index 255c22be7312f..06bb106d1b724 100644
@@ -1048,6 +1048,7 @@ struct mm_struct {
 		raw_spinlock_t mm_sched_lock;
 		unsigned long mm_sched_epoch;
 		int mm_sched_cpu;
+		int mm_sched_flags;
 		u64 nr_running_avg ____cacheline_aligned_in_smp;
 #endif
 
kernel/sched/fair.c +25 -3
index 205208f061bb3..ab1cdba65d389 100644
@@ -1237,12 +1237,20 @@ static inline int get_sched_cache_scale(int mul)
 	return (1 + (llc_aggr_tolerance - 1) * mul);
 }
 
+#define MM_SCHED_EXCEED_LLC_CAPACITY	1
+#define MM_SCHED_NO_CACHE_INFO		2
+#define MM_SCHED_EXCEED_LLC_NR		4
+#define MM_SCHED_NR_THREADS		8
+
 static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)
 {
 	unsigned int llc, scale;
 	struct cacheinfo *ci;
 	unsigned long rss;
 
+	mm->mm_sched_flags &= ~MM_SCHED_NO_CACHE_INFO;
+	mm->mm_sched_flags &= ~MM_SCHED_EXCEED_LLC_CAPACITY;
+
 	/*
 	 * get_cpu_cacheinfo_level() can not be used
 	 * because it requires the cpu_hotplug_lock
@@ -1257,8 +1265,10 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)
 		 * L2 becomes the LLC.
 		 */
 		ci = _get_cpu_cacheinfo_level(cpu, 2);
-		if (!ci)
+		if (!ci) {
+			mm->mm_sched_flags |= MM_SCHED_NO_CACHE_INFO;
 			return true;
+		}
 	}
 
 	llc = ci->size;
@@ -1283,13 +1293,20 @@ static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)
 	if (scale == INT_MAX)
 		return false;
 
-	return ((llc * scale) <= (rss * PAGE_SIZE));
+	if ((llc * scale) <= (rss * PAGE_SIZE)) {
+		mm->mm_sched_flags |= MM_SCHED_EXCEED_LLC_CAPACITY;
+		return true;
+	}
+
+	return false;
 }
 
 static bool exceed_llc_nr(struct mm_struct *mm, int cpu)
 {
 	int smt_nr = 1, scale;
 
+	mm->mm_sched_flags &= ~MM_SCHED_EXCEED_LLC_NR;
+
 #ifdef CONFIG_SCHED_SMT
 	if (sched_smt_active())
 		smt_nr = cpumask_weight(cpu_smt_mask(cpu));
@@ -1313,7 +1330,12 @@ static bool exceed_llc_nr(struct mm_struct *mm, int cpu)
 	if (scale == INT_MAX)
 		return false;
 
-	return ((mm->nr_running_avg * smt_nr) > (scale * per_cpu(sd_llc_size, cpu)));
+	if ((mm->nr_running_avg * smt_nr) > (scale * per_cpu(sd_llc_size, cpu))) {
+		mm->mm_sched_flags |= MM_SCHED_EXCEED_LLC_NR;
+		return true;
+	}
+
+	return false;
 }
 
 static void account_llc_enqueue(struct rq *rq, struct task_struct *p)
#118 lore raw

Re: [PATCH v2 23/23] -- DO NOT APPLY!!! -- sched/cache/debug: Display the per LLC occupancy for each process via proc fs

Quoted 50 lines · Aaron Lu
Quoted 49 lines · Tim Chen

From: Chen Yu <[email protected]>

Debug patch only.

Show the per-LLC occupancy in /proc/{PID}/schedstat, with each column corresponding to one LLC. This can be used to verify if the cache-aware load balancer works as expected by aggregating threads onto dedicated LLCs.

Suppose there are 2 LLCs and the sampling duration is 10 seconds:

Enable the cache aware load balance:
0 12281 <--- LLC0 residency delta is 0, LLC1 is 12 seconds 0 18881
0 16217

disable the cache aware load balance:
6497 15802
9299 5435
17811 8278

---
  fs/proc/base.c           | 22 ++++++++++++++++++++++
  include/linux/mm_types.h | 19 +++++++++++++++++--
  include/linux/sched.h    |  3 +++
  kernel/sched/fair.c      | 40 ++++++++++++++++++++++++++++++++++++++--
  4 files changed, 80 insertions(+), 4 deletions(-)
fs/proc/base.c +12 -0
index 6299878e3d97..f4be96f4bd01 100644
@@ -518,6 +518,28 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,
  		   (unsigned long long)task->se.sum_exec_runtime,
  		   (unsigned long long)task->sched_info.run_delay,
  		   task->sched_info.pcount);
+#ifdef CONFIG_SCHED_CACHE
+	if (sched_cache_enabled()) {
+		struct mm_struct *mm = task->mm;
+		u64 *llc_runtime;
+
+		if (!mm)
+			return 0;
+
+		llc_runtime = kcalloc(max_llcs, sizeof(u64), GFP_KERNEL);
+		if (!llc_runtime)
+			return 0;
+
Quoted 9 lines

fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 537

Quoted 5 lines
fs/proc/base.c +5 -0
+		if (get_mm_per_llc_runtime(task, llc_runtime))
+			goto out;
+
+		for (int i = 0; i < max_llcs; i++)
+			seq_printf(m, "%llu ", llc_runtime[i]);

I feel it is better to also mark the current preferred LLC of this process so that I can know how well it works.

Sure.

Quoted 37 lines

fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 545

Quoted 8 lines
fs/proc/base.c +5 -0
+		seq_puts(m, "\n");
+out:
+		kfree(llc_runtime);
+	}
+#endif
 
  	return 0;
  }

BTW, is there a way to tell if a process is being taken care of by 'cache aware scheduling' or it's blocked due to its huge rss or having too many threads?

I used below debug code to get these info through schedstat, but maybe I missed something and there is a simpler method?

fs/proc/base.c +2 -0
index f4be96f4bd015..c709a1a1bd867 100644
@@ -505,6 +505,7 @@ static int proc_pid_stack(struct seq_file *m, struct pid_namespace *ns,
  #endif
 
  #ifdef CONFIG_SCHED_INFO
+DECLARE_PER_CPU(int, sd_llc_id);
  /*
   * Provides /proc/PID/schedstat
   */
@@ -522,6 +523,7 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,
  	if (sched_cache_enabled()) {
  		struct mm_struct *mm = task->mm;
  		u64 *llc_runtime;
+		int mm_sched_llc;
 
  		if (!mm)
  			return 0;

fs/proc/base.c static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns, · line 541

Quoted 7 lines
fs/proc/base.c +4 -0
@@ -533,8 +535,17 @@ static int proc_pid_schedstat(struct seq_file *m, struct pid_namespace *ns,
  		if (get_mm_per_llc_runtime(task, llc_runtime))
  			goto out;
 
+		if (mm->mm_sched_cpu == -1)
+			mm_sched_llc = -1;
+		else
+			mm_sched_llc = per_cpu(sd_llc_id, mm->mm_sched_cpu);

We can use llc_id(mm->mm_sched_cpu).

Quoted 35 lines
fs/proc/base.c +6 -1
+
+		seq_printf(m, "%llu 0x%x\n", mm->nr_running_avg, mm->mm_sched_flags);
  		for (int i = 0; i < max_llcs; i++)
-			seq_printf(m, "%llu ", llc_runtime[i]);
+			seq_printf(m, "%s%s%llu ",
+				   i == task->preferred_llc ? "*" : "",
+				   i == mm_sched_llc ? "?" : "",
+				   llc_runtime[i]);
  		seq_puts(m, "\n");
  out:
  		kfree(llc_runtime);
include/linux/mm_types.h +1 -0
index 255c22be7312f..06bb106d1b724 100644
@@ -1048,6 +1048,7 @@ struct mm_struct {
  		raw_spinlock_t mm_sched_lock;
  		unsigned long mm_sched_epoch;
  		int mm_sched_cpu;
+		int mm_sched_flags;
  		u64 nr_running_avg ____cacheline_aligned_in_smp;
  #endif
 
kernel/sched/fair.c +4 -0
index 205208f061bb3..ab1cdba65d389 100644
@@ -1237,12 +1237,20 @@ static inline int get_sched_cache_scale(int mul)
  	return (1 + (llc_aggr_tolerance - 1) * mul);
  }
 
+#define MM_SCHED_EXCEED_LLC_CAPACITY	1
+#define MM_SCHED_NO_CACHE_INFO		2
+#define MM_SCHED_EXCEED_LLC_NR		4
+#define MM_SCHED_NR_THREADS		8

kernel/sched/fair.c static inline int get_sched_cache_scale(int mul) · line 1253

Quoted 10 lines
kernel/sched/fair.c +4 -0
+
  static bool exceed_llc_capacity(struct mm_struct *mm, int cpu)
  {
  	unsigned int llc, scale;
  	struct cacheinfo *ci;
  	unsigned long rss;
 
+	mm->mm_sched_flags &= ~MM_SCHED_NO_CACHE_INFO;
+	mm->mm_sched_flags &= ~MM_SCHED_EXCEED_LLC_CAPACITY;
+

Maybe we can do some read-comparison before writing the flags, previously we found that writing the per-process mm struct is very expensive, so maybe avoid writing to it as much as possible.

I'll fold your changes and do the test. Thanks!

Thanks,
Chenyu

#119 lore raw
In reply to Tim Chen

On Wed, Dec 03, 2025 at 03:07:19PM -0800, Tim Chen wrote:
... ...

Quoted 23 lines

Test results:

The patch series was applied and tested on v6.18-rc7.
See: https://github.com/timcchen1298/linux/commits/cache_aware_v2

The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.

The second test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.

hackbench/schbench/netperf/stream/stress-ng/chacha20 were launched on these two platforms.

[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when the number of different active threads is below the capacity of a LLC. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows good throughput improvement.

Quoted 3 lines

Genoa:
ChaCha20-xiangshan shows huge throughput improvement. No obvious difference is observed in hackbench/schbench

I think for small task number hackbench run, there should be some improvement.

I tried thread/pipe/2fds/1group, i.e. 4 tasks on Genoa:
./hackbench -T -f 2 -g 1 -p -l 2000000
And I noticed performance improved a lot:
(Result in seconds, less is better)

llc_off llc_on diff
time 4.755±1.6% 2.684±6.25% +43.6%

llc_off means /sys/kernel/debug/sched/llc_enabled set to 0 while llc_on means /sys/kernel/debug/sched/llc_enabled set to 1, other tunnables are left unchanged.
Turbo is disabled and cpufreq set to performance.

I also tried redis and noticed when I set io-threads to 4 in redis.conf, there is also some improvement on AMD Genoa:

llc_off manual diff llc_on diff
throughput 1536727±0% 1737619±0% +13.1% 1737720±0% +13.1%

Client cmdline:
numactl -N 1 redis-benchmark --threads 4 -t set -r 100000 -P 16 -n 10000000 server cmdline: numactl -N 0 redis-server ./redis.conf
I also tried to manually bind all tasks of redis server to a single LLC to see if this workload benefits from aggregation and that's what manual
means: taskset -c 8-15,200-207 redis-server ./redis.conf

According to the result, I think this 'cache aware scheduling' works as expected in that its performance is the same as manual binding; and they all beat llc_off.

#120 lore raw
Quoted 58 lines · Aaron Lu

On Wed, Dec 03, 2025 at 03:07:19PM -0800, Tim Chen wrote:
... ...

Quoted 23 lines

Test results:

The patch series was applied and tested on v6.18-rc7.
See: https://github.com/timcchen1298/linux/commits/cache_aware_v2

The first test platform is a 2 socket Intel Sapphire Rapids with 30 cores per socket. The DRAM interleaving is enabled in the BIOS so it essential has one NUMA node with two last level caches. There are 60 CPUs associated with each last level cache.

The second test platform is a AMD Genoa. There are 4 Nodes and 32 CPUs per node. Each node has 2 CCXs and each CCX has 16 CPUs.

hackbench/schbench/netperf/stream/stress-ng/chacha20 were launched on these two platforms.

[TL;DR]
Sappire Rapids:
hackbench shows significant improvement when the number of different active threads is below the capacity of a LLC. schbench shows overall wakeup latency improvement. ChaCha20-xiangshan shows good throughput improvement.

Quoted 3 lines

Genoa:
ChaCha20-xiangshan shows huge throughput improvement. No obvious difference is observed in hackbench/schbench

I think for small task number hackbench run, there should be some improvement.

I tried thread/pipe/2fds/1group, i.e. 4 tasks on Genoa:
./hackbench -T -f 2 -g 1 -p -l 2000000
And I noticed performance improved a lot:
(Result in seconds, less is better)

        llc_off       llc_on          diff
time   4.755±1.6%    2.684±6.25%    +43.6%

llc_off means /sys/kernel/debug/sched/llc_enabled set to 0 while llc_on means /sys/kernel/debug/sched/llc_enabled set to 1, other tunnables are left unchanged.
Turbo is disabled and cpufreq set to performance.

I also tried redis and noticed when I set io-threads to 4 in redis.conf, there is also some improvement on AMD Genoa:

                  llc_off        manual      diff     llc_on      diff
throughput      1536727±0%     1737619±0%  +13.1%   1737720±0%  +13.1%

Client cmdline:
numactl -N 1 redis-benchmark --threads 4 -t set -r 100000 -P 16 -n 10000000 server cmdline: numactl -N 0 redis-server ./redis.conf
I also tried to manually bind all tasks of redis server to a single LLC to see if this workload benefits from aggregation and that's what manual
means: taskset -c 8-15,200-207 redis-server ./redis.conf

Quoted 3 lines

According to the result, I think this 'cache aware scheduling' works as expected in that its performance is the same as manual binding; and they all beat llc_off.

Thanks Aaron for the test!

thanks,
Chenyu