sched/cache: Per-task control of cache aware scheduling via prctl
Subsystems (6)
cover sched/cache: Per-task control of cache aware scheduling via prctl
Hi all,
Cache aware scheduling today groups tasks by their mm: the LLC aggregation target lives in mm_struct, so the address space is the unit of grouping. That works, but in some scenarios that is too coarse and too eager, and the only knob we have over it is a single system-wide debugfs switch.
It's too coarse because plenty of workloads share data across cooperating *processes* rather than threads - a database with a process per connection, a browser with a renderer per site, a server and its worker helpers. They pass data through shm or pipes and would love to be pulled onto the same LLC, but they never share an mm, so today they can't be. And it's too eager in the other direction: a process whose threads don't actually share anything gets aggregated anyway, just because they happen to sit in one address space.
So the core idea of this series is simple: allow other groupings than the mm, make the grouping an object in its own right, and let user space say "put these tasks together" explicitly.
What the series does
====================
Patches 1-3 are pure preparation, no behavior change. They turn the per-mm statistics into a standalone, refcounted sched_cache_group, reach it from task_struct instead of from p->mm, and pull the allocation out into a helper. Once the group stands on its own, membership no longer has to follow the mm address space.
Patch 4 adds the actual interface:
int prctl(PR_SCHED_CACHE, subop, pid, arg4, pid_type);
PR_SCHED_CACHE_GET read back a task's (obfuscated) cookie id
PR_SCHED_CACHE_CREATE create a fresh group and install it
PR_SCHED_CACHE_SHARE_FROM copy the group of arg4 onto pid
PR_SCHED_CACHE_DISABLE opt this group out of LLC aggregation
PR_SCHED_CACHE_ENABLE opt it back in
'pid' is the task the operation applies to, with 0 meaning the caller, and 'pid_type' picks whether we touch just the thread, the whole thread group, or the process group. Permissions follow core scheduling: every task we touch has to pass ptrace_may_access(PTRACE_MODE_READ_REALCREDS), and for the group scopes we check *all* of them before changing *any* of them, so the operation is all-or-nothing.
Two subops need a word on their fourth argument:
- GET writes the cookie id to arg4, which is a u64 __user * and must be 8-byte aligned (misaligned pointers are rejected with -EINVAL). As in core scheduling the value handed out is an obfuscated hash of the kernel object, not a real pointer or a user-supplied number.
- SHARE_FROM reads arg4 as the *source* pid and copies its group onto 'pid'. CREATE is deliberately not idempotent - it always allocates a new group - so a caller that wants many tasks in one group does CREATE once and SHARE_FROM for the rest.
Patches 5-6 build on that. Patch 5 wires up DISABLE/ENABLE, and patch 6 turns the old debugfs boolean into a THP-style always/advise/never mode so that the system policy and the per-task hint compose the same way THP's do:
'always' aggregates regardless of the per-task hint, 'advise' honors it, 'never' turns the whole thing off. The write side still accepts the old 0/1 (and y/n, on/off) spelling so existing scripts don't break; only the read format grows to "[always] advise never".
Patch 7 documents the prctl and debugfs interface for cache aware scheduling in kerneldoc.
Relation to sched QoS
=====================
This patch set is also inspired by Qais' sched QoS tool for finer-grained control:
https://lore.kernel.org/lkml/20260415000910.2h5misvwc45bdumu@airbuntu/
https://github.com/qais-yousef/schedqos
There are two deliberate differences from the sched QoS interface:
1. The cookie is owned by the kernel. User space never invents or passes a cookie value - it only names tasks by pid and asks the kernel to create a group or link one task's group to another. That mirrors core scheduling, where PR_SCHED_CORE_GET only hands back an obfuscated id. It also relieves userspace the burden of cookie lifecycle management and avoiding duplicate cookies between separate entities doing the grouping. We think that it will make schedqos easier to implement.
The cookie value is only meant to identify whether two tasks belong in the same cache scheduling group.
2. It's prctl() rather than sched_setattr(). Group membership isn't really an attribute value, and core scheduling already set a prctl-shaped precedent for "put these tasks together", so we reuse its subop layout, pid_type scoping and ptrace_may_access() permission model.
On the schedqos side the change would live in apply_thread_qos(): create the group once per app instance and let the rest of the app join it - roughly:
if (!appi->grouped) {
/* CREATE is not idempotent, do it once per instance */
err = prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_CREATE,
tgid, 0, PIDTYPE_TGID);
appi->grouped = !err;
} else {
prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM,
pid, tgid, PIDTYPE_PID);
}Yangyu Chen also proposed exposing per-process parameters via prctl:
https://lore.kernel.org/all/[email protected]/
That fits naturally on top of this if people agree the direction is right.
And in theory cgroup could offer per-cgroup cache aware scheduling with the same mechanism - but we'd want input from the cgroup maintainers before going anywhere near that.
Still to do
===========
- Documentation for the new prctl (a prctl(2) man-page update).
- A selftest under tools/testing/selftests exercising the subops and the permission checks - likewise TODO.
Feedbacks very welcome, especially on the interface shape (prctl vs. a QoS attribute), the kernel-owned-cookie choice, and whether the always/advise/ never policy composition is the right model.
The series applies on v7.2-rc6.
Tim Chen and Chen Yu
Chen Yu (1):
sched/cache: Extend the enabled debugfs to more modes
Tim Chen (6):
sched/cache: Decouple sched_cache_group from mm sched/cache: Introduce task_struct->sched_cache_grp sched/cache: Extract sched_cache_alloc_group() helper sched/cache: Add prctl to manage per process cache scheduling groups sched/cache: Allow a process to enable cache aware scheduling via
prctl
sched/cache: Documentation: document the PR_SCHED_CACHE prctl
.../admin-guide/kernel-parameters.txt | 3 + Documentation/scheduler/index.rst | 1 + Documentation/scheduler/sched-cache.rst | 175 ++++++++ fs/exec.c | 24 ++ include/linux/mm_types.h | 15 +- include/linux/sched.h | 18 +- include/uapi/linux/prctl.h | 9 + kernel/exit.c | 35 +- kernel/fork.c | 38 ++ kernel/sched/build_utility.c | 4 + kernel/sched/cache_sched.c | 374 ++++++++++++++++++ kernel/sched/debug.c | 63 ++- kernel/sched/fair.c | 188 +++++---- kernel/sched/sched.h | 24 +- kernel/sched/topology.c | 32 +- kernel/sys.c | 5 + 16 files changed, 895 insertions(+), 113 deletions(-) create mode 100644 Documentation/scheduler/sched-cache.rst create mode 100644 kernel/sched/cache_sched.c
Signature · 1 line
2.32.0
RFC PATCH 1/7 sched/cache: Decouple sched_cache_group from mm
Currently the sched cache grouping is by mm. But in the future, the grouping could be by cgroup, cookie group, numa_group or others.
Rename sched_cache_stat to sched_cache_group and turn it into a refcounted object allocated from mm_struct.
The mm_struct now holds a pointer (sched_cache_grp) to this object instead of embedding it.
Introduce kernel/sched/cache_sched.c to host the cache aware scheduling helpers and define sched_cache_group_put() there.
Meanwhile skip kthreads in account_mm_sched(), consistently with task_tick_cache().
--- include/linux/mm_types.h | 15 ++--- include/linux/sched.h | 8 ++- kernel/exit.c | 6 +- kernel/sched/build_utility.c | 4 ++ kernel/sched/cache_sched.c | 20 +++++++ kernel/sched/fair.c | 110 +++++++++++++++++++++-------------- 6 files changed, 105 insertions(+), 58 deletions(-) create mode 100644 kernel/sched/cache_sched.c
include/linux/mm_types.h +5 -10
@@ -1211,7 +1211,7 @@ struct mm_struct {struct mm_mm_cid mm_cid;/* sched_cache related statistics */ -struct sched_cache_stat sc_stat; +struct sched_cache_group *sched_cache_grp;#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */ #endif
@@ -1609,8 +1609,9 @@ static inline unsigned int mm_cid_size(void)#endif /* CONFIG_SCHED_MM_CID */ #ifdef CONFIG_SCHED_CACHE -void mm_init_sched (struct mm_struct *mm, -struct sched_cache_time __percpu *pcpu_sched); +int mm_init_sched (struct mm_struct *mm, +struct sched_cache_time __percpu *pcpu_sched); +void mm_destroy_sched (struct mm_struct *mm);static inline int mm_alloc_sched_noprof (struct mm_struct *mm) {
@@ -1620,17 +1621,11 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)if (!pcpu_sched)return -ENOMEM ; -mm_init_sched (mm, pcpu_sched); - return 0; + return mm_init_sched(mm, pcpu_sched); }#define mm_alloc_sched (...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) -static inline void mm_destroy_sched (struct mm_struct *mm) -{ -free_percpu (mm->sc_stat.pcpu_sched); - mm->sc_stat.pcpu_sched =NULL ; -}#else /* !CONFIG_SCHED_CACHE */ static inline int mm_alloc_sched (struct mm_struct *mm) {return 0 ; }
include/linux/sched.h +6 -2
@@ -2390,7 +2390,7 @@ struct sched_cache_time {unsigned long epoch; }; -struct sched_cache_stat { +struct sched_cache_group {struct sched_cache_time __percpu *pcpu_sched;raw_spinlock_t lock;unsigned long epoch;
@@ -2398,11 +2398,15 @@ struct sched_cache_stat {unsigned long next_scan;unsigned long footprint;int cpu; +refcount_t refcnt; + struct rcu_head rcu; }____cacheline_aligned_in_smp ; +void sched_cache_group_put (struct sched_cache_group *grp); +#else -struct sched_cache_stat { }; +struct sched_cache_group { };#endif
kernel/exit.c +3 -3
@@ -561,12 +561,12 @@ static void exit_mm_sched_cache(struct mm_struct *mm)return ;/* * No lock protection due to performance considerations. -* Make sure mm->sc_stat.footprint does not become +* Make sure the group footprint does not become * negative. */ - fp =READ_ONCE (mm->sc_stat.footprint); + fp =READ_ONCE (mm->sched_cache_grp->footprint); sub =min (fp, current->total_numa_faults); - WRITE_ONCE(mm->sc_stat.footprint, fp - sub); + WRITE_ONCE(mm->sched_cache_grp->footprint, fp - sub); }#else static inline void exit_mm_sched_cache (struct mm_struct *mm)
kernel/sched/build_utility.c +4 -0
@@ -89,6 +89,10 @@# include "core_sched.c" #endif +#ifdef CONFIG_SCHED_CACHE +# include "cache_sched.c" +#endif +#ifdef CONFIG_PSI # include "psi.c" #endif
kernel/sched/cache_sched.c new file +20 -0
@@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include "sched.h" + +static void sched_cache_group_free_rcu (struct rcu_head *rcu) +{ +struct sched_cache_group *grp = + container_of(rcu, struct sched_cache_group, rcu); + +/* free_percpu() may be called from atomic context. */ +free_percpu (grp->pcpu_sched); +kfree (grp); +} + +void sched_cache_group_put (struct sched_cache_group *grp) +{ +if (!grp || !refcount_dec_and_test (&grp->refcnt)) +return ; + +call_rcu (&grp->rcu, sched_cache_group_free_rcu); +}
kernel/sched/fair.c +67 -43
254 lines not on this page · raw message
Signature · 1 line
2.32.0
RFC PATCH 2/7 sched/cache: Introduce task_struct->sched_cache_grp
In reply to Tim ChenAdd a sched_cache_grp pointer to task_struct so that scheduler code can access the cache group directly via the task, without going through mm->sched_cache_grp. This decouples the scheduler's hot-path accesses from the mm_struct.
Each task holds its own refcount on the sched_cache_group, separate from the reference held by its mm_struct. The reference is acquired in copy_mm() (fork) and exec_mmap() (exec), and released in exit_mm().
Convert all scheduler code in fair.c and exit.c to use p->sched_cache_grp instead of p->mm->sched_cache_grp.
Add sched_cache_group_get() to kernel/sched/cache_sched.c.
--- fs/exec.c | 14 ++++ include/linux/sched.h | 3 + kernel/exit.c | 26 +++++-- kernel/fork.c | 23 +++++++ kernel/sched/cache_sched.c | 19 ++++++ kernel/sched/fair.c | 135 ++++++++++++++++++++----------------- kernel/sched/sched.h | 3 + 7 files changed, 157 insertions(+), 66 deletions(-)
fs/exec.c +14 -0
@@ -880,6 +880,20 @@ static int exec_mmap(struct linux_binprm *bprm) active_mm = tsk->active_mm; tsk->active_mm = mm; tsk->mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +struct sched_cache_group *old_grp, *new_grp; + + old_grp =rcu_dereference_protected (tsk->sched_cache_grp, true); + +/* Acquire the reference before publishing the pointer. */ + new_grp =sched_cache_group_get (mm->sched_cache_grp); + +rcu_assign_pointer (tsk->sched_cache_grp, new_grp); +if (old_grp) +sched_cache_group_put (old_grp); + } +#endif mm_init_cid (mm, tsk); exec_state =task_exec_state_replace (tsk, exec_state);/*
include/linux/sched.h +3 -0
@@ -1422,6 +1422,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work;int preferred_llc; +struct sched_cache_group __rcu *sched_cache_grp;/* 1: task was enqueued to its preferred LLC, 0 otherwise */ int pref_llc_queued;#endif
@@ -2403,6 +2404,8 @@ struct sched_cache_group { } ____cacheline_aligned_in_smp;void sched_cache_group_put (struct sched_cache_group *grp); +struct sched_cache_group *sched_cache_group_get (struct sched_cache_group *grp); +struct sched_cache_group *task_cache_group_get (struct task_struct *p);#else
kernel/exit.c +20 -6
@@ -553,23 +553,25 @@ void mm_update_next_owner(struct mm_struct *mm)* Subtract the memory footprint of the current task from * mm. */ -static void exit_mm_sched_cache (struct mm_struct *mm) +static void exit_mm_sched_cache (void ) { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true);unsigned long fp, sub; -if (!current->total_numa_faults) +if (!grp || !current->total_numa_faults)return ;/* * No lock protection due to performance considerations. * Make sure the group footprint does not become * negative. */ - fp =READ_ONCE (mm->sched_cache_grp->footprint); + fp =READ_ONCE (grp->footprint); sub =min (fp, current->total_numa_faults); -WRITE_ONCE (mm->sched_cache_grp->footprint, fp - sub); +WRITE_ONCE (grp->footprint, fp - sub); }#else -static inline void exit_mm_sched_cache (struct mm_struct *mm) +static inline void exit_mm_sched_cache (void ) { }#endif /* CONFIG_SCHED_CACHE CONFIG_NUMA_BALANCING */
@@ -586,7 +588,19 @@ static void exit_mm(void)if (!mm)return ; -exit_mm_sched_cache (mm); +exit_mm_sched_cache (); + +#ifdef CONFIG_SCHED_CACHE + { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true); + +rcu_assign_pointer (current->sched_cache_grp,NULL ); + +if (grp) +sched_cache_group_put (grp); + } +#endif mmap_read_lock (mm);mmgrab_lazy_tlb (mm);
kernel/fork.c +23 -0
@@ -1599,6 +1599,19 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk) tsk->mm = mm; tsk->active_mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +/* +* A task holds its own reference on the group, separate from +* the reference held by its mm_struct. Acquire it before +* publishing the pointer. +*/ +struct sched_cache_group *grp = +sched_cache_group_get (mm->sched_cache_grp); + +rcu_assign_pointer (tsk->sched_cache_grp, grp); + } +#endif return 0 ; }
@@ -2581,6 +2594,16 @@ __latent_entropy struct task_struct *copy_process( bad_fork_cleanup_namespaces:exit_nsproxy_namespaces (p); bad_fork_cleanup_mm: +#ifdef CONFIG_SCHED_CACHE +/* +* copy_mm() took a task reference on the cache group; a failed fork +* never reaches exit_mm(), so release it here to avoid leaking the +* group and its per-CPU buffer. +*/ +sched_cache_group_put (rcu_dereference_protected (p->sched_cache_grp, true)); +RCU_INIT_POINTER (p->sched_cache_grp,NULL ); +#endif +if (p->mm) {mm_clear_owner (p->mm, p);mmput (p->mm);
kernel/sched/cache_sched.c +19 -0
@@ -1,6 +1,25 @@// SPDX-License-Identifier: GPL-2.0-only #include "sched.h" +struct sched_cache_group *sched_cache_group_get (struct sched_cache_group *grp) +{ +/* +* refcount_inc_not_zero() is the acquire primitive for lockless +* (RCU) lookups; plain refcount_inc() would scribble the count if +* it already reached zero. Return NULL in that case. +*/ +if (grp && !refcount_inc_not_zero (&grp->refcnt)) + grp =NULL ; + +return grp; +} + +struct sched_cache_group *task_cache_group_get (struct task_struct *p) +{ +guard (rcu)(); +return sched_cache_group_get (rcu_dereference (p->sched_cache_grp)); +} +static void sched_cache_group_free_rcu (struct rcu_head *rcu) { struct sched_cache_group *grp =
kernel/sched/fair.c +75 -60
311 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -4106,6 +4106,9 @@ static inline bool sched_cache_enabled(void)return static_branch_unlikely (&sched_cache_active); } +DEFINE_FREE(sched_cache_group_put, struct sched_cache_group *, +sched_cache_group_put (_T)); +extern void sched_cache_active_set (void );#endif
Signature · 1 line
2.32.0
RFC PATCH 3/7 sched/cache: Extract sched_cache_alloc_group() helper
In reply to Tim ChenExtract sched_cache_alloc_group() into kernel/sched/cache_sched.c as a shared helper.
Convert mm_init_sched() to use the new helper instead of open-coding the allocation and initialization. This prepares for a second caller in the upcoming prctl CREATE path.
No functional change.
--- include/linux/sched.h | 2 ++ kernel/sched/cache_sched.c | 46 ++++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 32 ++------------------------ 3 files changed, 50 insertions(+), 30 deletions(-)
include/linux/sched.h +2 -0
@@ -2406,6 +2406,8 @@ struct sched_cache_group {void sched_cache_group_put (struct sched_cache_group *grp);struct sched_cache_group *sched_cache_group_get (struct sched_cache_group *grp);struct sched_cache_group *task_cache_group_get (struct task_struct *p); +struct sched_cache_group * +sched_cache_alloc_group (struct sched_cache_time __percpu *pcpu_sched);#else
kernel/sched/cache_sched.c +46 -0
49 lines not on this page · raw message
kernel/sched/fair.c +2 -30
@@ -1575,40 +1575,12 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p)int mm_init_sched (struct mm_struct *mm,struct sched_cache_time __percpu *_pcpu_sched) { -struct sched_cache_group *grp; -unsigned long epoch =0 ; -int i; +struct sched_cache_group *grp =sched_cache_alloc_group (_pcpu_sched); - grp =kzalloc_obj (*grp); -if (!grp) { -free_percpu (_pcpu_sched); +if (!grp)return -ENOMEM ; - } - -for_each_possible_cpu (i) { -struct sched_cache_time *pcpu_sched =per_cpu_ptr (_pcpu_sched, i); -struct rq *rq =cpu_rq (i); - - pcpu_sched->runtime =0 ; -/* a slightly stale cpu epoch is acceptible */ - pcpu_sched->epoch = rq->cpu_epoch; - epoch = rq->cpu_epoch; - } -raw_spin_lock_init (&grp->lock); - grp->epoch = epoch; - grp->cpu =-1 ; - grp->next_scan = jiffies; - grp->nr_running_avg =0 ; - grp->footprint =0 ; -refcount_set (&grp->refcnt,1 ); mm->sched_cache_grp = grp; -/* -* The update to grp->pcpu_sched should not be reordered -* before initialization to grp's other fields, in case -* the readers may get invalid mm_sched_epoch, etc. -*/ -smp_store_release (&grp->pcpu_sched, _pcpu_sched);return 0 ; }
Signature · 1 line
2.32.0
RFC PATCH 4/7 sched/cache: Add prctl to manage per process cache scheduling groups
In reply to Tim ChenDerived from core scheduling's prctl interface.
Implement the controls for cache aware scheduling on a process:
int prctl(PR_SCHED_CACHE, unsigned long subop, pid_t pid,
unsigned long arg4, unsigned long type);
pid argument: the PID of the task the operation applies to, that is the destination of the group assignment. 0 means "the calling task."
pid_type : PIDTYPE_PID targets the single thread, PIDTYPE_TGID the whole thread group and PIDTYPE_PGID the process group of that task. PR_SCHED_CACHE_GET only accepts PIDTYPE_PID.
arg4 argument: the address to store the cookie for PR_SCHED_CACHE_GET, the pid to take the group from for PR_SCHED_CACHE_SHARE_FROM, where 0 means "the calling task".
The second argument (subop) selects one of the following operations:
1. PR_SCHED_CACHE_GET
Retrieve the cache aware scheduling cookie of a task.
u64 cookie;
prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_GET, pid, (unsigned long)&cookie,
PIDTYPE_PID);
2. PR_SCHED_CACHE_CREATE
Create a new cache scheduling group and assign it to the target pid,
prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_CREATE, pid, 0, PIDTYPE_TGID);
3. PR_SCHED_CACHE_SHARE_FROM
Copy the group of the task given in arg4 to the target task.
/* pull 1234's group onto the caller */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM, 0, 1234, PIDTYPE_PID);
/* push the caller's group onto the thread group of 1234 */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM, 1234, 0, PIDTYPE_TGID);
/* let the thread group of 1234 share 5678's group */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_SHARE_FROM, 1234, 5678, PIDTYPE_TGID);
Unlike the core scheduling's PR_SCHED_CORE_SHARE_FROM and PR_SCHED_CORE_SHARE_TO, there is no need to involve the current task: a daemon(scheqos, eg) can handle two tasks without joining their group, which would pollute the group's per LLC occupancy statistics.
--- fs/exec.c | 18 ++- include/linux/sched.h | 4 + include/uapi/linux/prctl.h | 7 + kernel/exit.c | 13 +- kernel/fork.c | 25 +++- kernel/sched/cache_sched.c | 274 +++++++++++++++++++++++++++++++++++++ kernel/sched/fair.c | 8 +- kernel/sys.c | 5 + 8 files changed, 341 insertions(+), 13 deletions(-)
fs/exec.c +14 -4
@@ -884,12 +884,22 @@ static int exec_mmap(struct linux_binprm *bprm) {struct sched_cache_group *old_grp, *new_grp; - old_grp =rcu_dereference_protected (tsk->sched_cache_grp, true); - -/* Acquire the reference before publishing the pointer. */ +/* +* Acquire the reference before publishing the pointer: once +* tsk->sched_cache_grp is visible, a concurrent +* prctl(PR_SCHED_CACHE) writer may pick the group up as its +* old_grp and drop a reference we have not taken yet. +* +* pi_lock serializes the exchange against such a writer. IRQs +* are already disabled here, so a plain raw_spin_lock() +* suffices. +*/ new_grp =sched_cache_group_get (mm->sched_cache_grp); -rcu_assign_pointer (tsk->sched_cache_grp, new_grp); +raw_spin_lock (&tsk->pi_lock); + old_grp =sched_cache_grp_replace (tsk, new_grp); +raw_spin_unlock (&tsk->pi_lock); +if (old_grp)sched_cache_group_put (old_grp); }
include/linux/sched.h +4 -0
@@ -2407,7 +2407,11 @@ void sched_cache_group_put(struct sched_cache_group *grp);struct sched_cache_group *sched_cache_group_get (struct sched_cache_group *grp);struct sched_cache_group *task_cache_group_get (struct task_struct *p);struct sched_cache_group * +sched_cache_grp_replace (struct task_struct *p,struct sched_cache_group *grp); +struct sched_cache_group *sched_cache_alloc_group (struct sched_cache_time __percpu *pcpu_sched); +int sched_cache_prctl (int option,unsigned long arg2,unsigned long arg3, +unsigned long arg4,unsigned long arg5);#else
include/uapi/linux/prctl.h +7 -0
@@ -416,4 +416,11 @@ struct prctl_mm_map {# define PR_CFI_DISABLE _BITUL(1)# define PR_CFI_LOCK _BITUL(2) +/* Cache-aware scheduling */ +#define PR_SCHED_CACHE 82 +# define PR_SCHED_CACHE_GET 0 +# define PR_SCHED_CACHE_CREATE 1 +# define PR_SCHED_CACHE_SHARE_FROM 2 +# define PR_SCHED_CACHE_MAX 3 +#endif /* _LINUX_PRCTL_H */
kernel/exit.c +10 -3
@@ -592,10 +592,17 @@ static void exit_mm(void)#ifdef CONFIG_SCHED_CACHE { -struct sched_cache_group *grp = -rcu_dereference_protected (current->sched_cache_grp, true); +struct sched_cache_group *grp; +unsigned long flags; -rcu_assign_pointer (current->sched_cache_grp,NULL ); +/* +* pi_lock serializes the clear against a concurrent +* prctl(PR_SCHED_CACHE) writer targeting this task, so the +* reference is dropped exactly once. +*/ +raw_spin_lock_irqsave (¤t->pi_lock, flags); + grp =sched_cache_grp_replace (current,NULL ); +raw_spin_unlock_irqrestore (¤t->pi_lock, flags);if (grp)sched_cache_group_put (grp);
kernel/fork.c +20 -5
@@ -1601,13 +1601,28 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk) tsk->active_mm = mm;#ifdef CONFIG_SCHED_CACHE { +struct sched_cache_group *grp; +/* -* A task holds its own reference on the group, separate from -* the reference held by its mm_struct. Acquire it before -* publishing the pointer. +* For CLONE_VM (threads): inherit the parent's +* sched_cache_grp, which may be a cookie-based group different +* from mm->sched_cache_grp. For kernel thread, don't assign +* it a sched cache group. +* +* For new processes (own mm): use the new mm's group. The mm +* has just been created and is not visible to anyone else, so +* there is no race against a concurrent writer and the mm's +* own reference keeps the group alive. Take the refcount +* directly, without RCU. */ -struct sched_cache_group *grp = -sched_cache_group_get (mm->sched_cache_grp); +if (clone_flags &CLONE_VM ) { +if (tsk->flags &PF_KTHREAD ) + grp =NULL ; +else + grp =task_cache_group_get (current); + }else { + grp =sched_cache_group_get (mm->sched_cache_grp); + }rcu_assign_pointer (tsk->sched_cache_grp, grp); }
kernel/sched/cache_sched.c +274 -0
277 lines not on this page · raw message
kernel/sched/fair.c +7 -1
@@ -1484,7 +1484,13 @@ static bool invalid_llc_nr(struct sched_cache_group *grp, struct task_struct *p, {int scale; -if (get_nr_threads (p) <=1 ) +/* +* Single threaded process that is not grouped with other threads +* do not need cache aware scheduling. +* A single-threaded process has a refcount of 2: from the mm and task +* respectively. +*/ +if (refcount_read (&grp->refcnt) <=2 &&get_nr_threads (p) <=1 )return true;/*
kernel/sys.c +5 -0
@@ -2907,6 +2907,11 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,if (arg3 &PR_CFI_LOCK && !(arg3 &PR_CFI_DISABLE )) error =arch_prctl_lock_branch_landing_pad_state (me);break ; +#ifdef CONFIG_SCHED_CACHE +case PR_SCHED_CACHE : + error =sched_cache_prctl (option, arg2, arg3, arg4, arg5); +break ; +#endif default :trace_task_prctl_unknown (option, arg2, arg3, arg4, arg5); error = -EINVAL ;
Signature · 1 line
2.32.0
RFC PATCH 5/7 sched/cache: Allow a process to enable cache aware scheduling via prctl
In reply to Tim ChenAdd the PR_SCHED_CACHE_ENABLE and PR_SCHED_CACHE_DISABLE subops to the PR_SCHED_CACHE prctl interface, allowing a process to turn cache aware scheduling on or off.
int prctl(PR_SCHED_CACHE, unsigned long subop, pid_t pid,
unsigned long cookie, unsigned long type);
/* disable cache aware scheduling for this task */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_DISABLE, 0, 0, PIDTYPE_PID);
/* enable cache aware scheduling for this task */ prctl(PR_SCHED_CACHE, PR_SCHED_CACHE_ENABLE, 0, 0, PIDTYPE_PID);
pid 0 targets the calling task; otherwise the group of the given pid is used. Both return -ENOENT when the task has no cache scheduling group.
--- include/linux/sched.h | 1 + include/uapi/linux/prctl.h | 4 +++- kernel/sched/cache_sched.c | 15 +++++++++++++++ kernel/sched/fair.c | 6 ++++++ 4 files changed, 25 insertions(+), 1 deletion(-)
include/linux/sched.h +1 -0
@@ -2399,6 +2399,7 @@ struct sched_cache_group {unsigned long next_scan;unsigned long footprint;int cpu; +int enabled;refcount_t refcnt;struct rcu_head rcu; }____cacheline_aligned_in_smp ;
include/uapi/linux/prctl.h +3 -1
@@ -421,6 +421,8 @@ struct prctl_mm_map {# define PR_SCHED_CACHE_GET 0# define PR_SCHED_CACHE_CREATE 1# define PR_SCHED_CACHE_SHARE_FROM 2 -# define PR_SCHED_CACHE_MAX 3 +# define PR_SCHED_CACHE_DISABLE 3 +# define PR_SCHED_CACHE_ENABLE 4 +# define PR_SCHED_CACHE_MAX 5#endif /* _LINUX_PRCTL_H */
kernel/sched/cache_sched.c +15 -0
@@ -60,6 +60,7 @@ static void sched_cache_group_init(struct sched_cache_group *grp, grp->next_scan = jiffies; grp->nr_running_avg =0 ; grp->footprint =0 ; + grp->enabled =1 ;refcount_set (&grp->refcnt,1 );/* * The update to grp->pcpu_sched should not be reordered
@@ -261,6 +262,20 @@ int sched_cache_prctl(int option, unsigned long arg2, unsigned long arg3, }switch (arg2) { +case PR_SCHED_CACHE_DISABLE : +case PR_SCHED_CACHE_ENABLE : +/* +* Setting a single task is OK, because the sched_cache_group is +* shared by multiple tasks, setting one equals to setting all. +*/ + grp =task_cache_group_get (dst); +if (!grp) { + err = -ENOENT ; + goto out_task; + } +WRITE_ONCE (grp->enabled, arg2 ==PR_SCHED_CACHE_ENABLE ); + + goto out_group;case PR_SCHED_CACHE_GET : {unsigned long id =0 ;
kernel/sched/fair.c +6 -0
@@ -1652,6 +1652,9 @@ static int get_pref_llc(struct task_struct *p, struct sched_cache_group *grp)if (!grp)return -1 ; +if (!READ_ONCE (grp->enabled)) +return -1 ; + mm_sched_cpu =READ_ONCE (grp->cpu);if (mm_sched_cpu !=-1 ) { mm_sched_llc =llc_id (mm_sched_cpu);
@@ -1743,6 +1746,9 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p) !grp->pcpu_sched)return ; +if (!READ_ONCE (grp->enabled)) +return ; + epoch = rq->cpu_epoch;/* avoid moving backwards */ if (time_after_eq (grp->epoch, epoch))
Signature · 1 line
2.32.0
RFC PATCH 6/7 sched/cache: Extend the enabled debugfs to more modes
In reply to Tim ChenFrom: Chen Yu <[email protected]>
A knob named "enabled" under debugfs llc_balancing is used to allow the user to turn off and on the cache aware scheduling at runtime. Since there is fine-gain control for per task cache-aware scheduling control via prctl, enhance the "enabled" knob in to a three-value mode, derived from the design of THP:
always, advise, never.
The design matrix is as followed:
prctl ENABLE prctl DISABLE
always Y Y advise Y N never N N
Y: cache aware scheduling for the task is enabled
N: cache aware scheduling for the task is disabled
The default mode is "always".
For backward compatibility, writing "1" or "0" is still accepted and is treated as "always" or "never" respectively.
The same modes can also be selected at boot time via the sched_cache= kernel command line parameter.
--- .../admin-guide/kernel-parameters.txt | 3 + kernel/sched/cache_sched.c | 2 +- kernel/sched/debug.c | 63 +++++++++++++++++-- kernel/sched/fair.c | 7 ++- kernel/sched/sched.h | 21 ++++++- kernel/sched/topology.c | 32 ++++++++-- 6 files changed, 112 insertions(+), 16 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +3 -0
@@ -6751,6 +6751,9 @@ Kernel parameters solution to mutex-based priority inversion. Format: <bool> + sched_cache= [KNL] Set the cache aware scheduling mode. + Format: { "always" | "advise" | "never" } + sched_verbose [KNL,EARLY] Enables verbose scheduler debug messages. schedstats= [KNL,X86] Enable or disable scheduled statistics.
kernel/sched/cache_sched.c +1 -1
@@ -60,7 +60,7 @@ static void sched_cache_group_init(struct sched_cache_group *grp, grp->next_scan = jiffies; grp->nr_running_avg =0 ; grp->footprint =0 ; - grp->enabled =1 ; + grp->enabled =0 ;refcount_set (&grp->refcnt,1 );/* * The update to grp->pcpu_sched should not be reordered
kernel/sched/debug.c +57 -6
83 lines not on this page · raw message
kernel/sched/fair.c +5 -2
@@ -1652,7 +1652,7 @@ static int get_pref_llc(struct task_struct *p, struct sched_cache_group *grp)if (!grp)return -1 ; -if (!READ_ONCE (grp->enabled)) +if (!sched_cache_group_enabled (grp))return -1 ; mm_sched_cpu =READ_ONCE (grp->cpu);
@@ -1746,7 +1746,7 @@ static void task_tick_cache(struct rq *rq, struct task_struct *p) !grp->pcpu_sched)return ; -if (!READ_ONCE (grp->enabled)) +if (!sched_cache_group_enabled (grp))return ; epoch = rq->cpu_epoch;
@@ -10601,6 +10601,9 @@ static enum llc_mig can_migrate_llc_task(int src_cpu, int dst_cpu,if (!grp)return mig_unrestricted; +if (!sched_cache_group_enabled (grp)) +return mig_unrestricted; + cpu =READ_ONCE (grp->cpu);if (cpu <0 ||cpus_share_cache (src_cpu, dst_cpu))return mig_unrestricted;
kernel/sched/sched.h +20 -1
@@ -4092,9 +4092,17 @@ static inline void mm_cid_switch_to(struct task_struct *prev, struct task_struct#endif /* !CONFIG_SCHED_MM_CID */ #ifdef CONFIG_SCHED_CACHE +enum sc_modes { +SC_ENABLED_ALWAYS =0 , +SC_ENABLED_ADVISE =1 , +SC_ENABLED_NEVER =2 , +SC_ENABLED_NR , +}; +DECLARE_STATIC_KEY_FALSE (sched_cache_present);DECLARE_STATIC_KEY_FALSE (sched_cache_active); -extern int sysctl_sched_cache_user; +DECLARE_STATIC_KEY_FALSE (sched_cache_adv); +extern int sysctl_sched_cache_mode;extern unsigned int llc_aggr_tolerance;extern unsigned int llc_epoch_period;extern unsigned int llc_epoch_affinity_timeout;
@@ -4106,6 +4114,17 @@ static inline bool sched_cache_enabled(void)return static_branch_unlikely (&sched_cache_active); } +static inline bool sched_cache_group_enabled (struct sched_cache_group *grp) +{ +if (!static_branch_unlikely (&sched_cache_active)) +return false; + +if (!static_branch_likely (&sched_cache_adv)) +return true; + +return READ_ONCE (grp->enabled); +} + DEFINE_FREE(sched_cache_group_put, struct sched_cache_group *,sched_cache_group_put (_T));
kernel/sched/topology.c +26 -6
@@ -853,8 +853,12 @@ DEFINE_STATIC_KEY_FALSE(sched_cache_present);* is active, used by the scheduler. */ DEFINE_STATIC_KEY_FALSE (sched_cache_active); -/* user wants cache aware scheduling [0 or 1] */ -int sysctl_sched_cache_user =1 ; +DEFINE_STATIC_KEY_FALSE (sched_cache_adv); +/* +* User provided cache aware scheduling modes: +* always, advise, never. +*/ +int sysctl_sched_cache_mode =SC_ENABLED_ALWAYS ;/* * Get the effective LLC size in bytes that @cpu's bottom sched_domain
@@ -952,6 +956,7 @@ static void _sched_cache_active_set(void)/* hardware does not support */ if (!static_branch_likely (&sched_cache_present)) {static_branch_disable_cpuslocked (&sched_cache_active); +static_branch_disable_cpuslocked (&sched_cache_adv);if (sched_debug ())pr_info ("%s: cache aware scheduling not supported on this platform\n" , __func__);return ;
@@ -963,14 +968,29 @@ static void _sched_cache_active_set(void)* It is not in the critical path, leave as-is * for now. */ -if (sysctl_sched_cache_user) { +switch (READ_ONCE (sysctl_sched_cache_mode)) { +case SC_ENABLED_ALWAYS : +static_branch_disable_cpuslocked (&sched_cache_adv);static_branch_enable_cpuslocked (&sched_cache_active);if (sched_debug ()) -pr_info ("%s: enabling cache aware scheduling\n" , __func__); - }else { +pr_info ("%s: cache aware scheduling switch to [always]\n" , __func__); +break ; +case SC_ENABLED_ADVISE : +static_branch_enable_cpuslocked (&sched_cache_adv); +static_branch_enable_cpuslocked (&sched_cache_active); +if (sched_debug ()) +pr_info ("%s: cache aware scheduling switch to [advise]\n" , __func__); +break ; +case SC_ENABLED_NEVER :static_branch_disable_cpuslocked (&sched_cache_active); +static_branch_disable_cpuslocked (&sched_cache_adv);if (sched_debug ()) -pr_info ("%s: disabling cache aware scheduling\n" , __func__); +pr_info ("%s: cache aware scheduling switch to [never]\n" , __func__); +break ; +default : +if (sched_debug ()) +pr_info ("%s: Invalid cache aware scheduling mode\n" , __func__); +break ; } }
Signature · 1 line
2.32.0
RFC PATCH 7/7 sched/cache: Documentation: document the PR_SCHED_CACHE prctl
In reply to Tim ChenAdd Documentation/scheduler/sched-cache.rst describing cache aware scheduling and the PR_SCHED_CACHE prctl() interface: the GET/CREATE/SHARE_FROM/DISABLE/ENABLE sub-operations, their arguments, the ptrace_may_access() permission model, the return values and how the per-task hint composes with the always/advise/never debugfs policy.
Hook it into the scheduler documentation toctree.
--- Documentation/scheduler/index.rst | 1 + Documentation/scheduler/sched-cache.rst | 175 ++++++++++++++++++++++++ 2 files changed, 176 insertions(+) create mode 100644 Documentation/scheduler/sched-cache.rst
Documentation/scheduler/index.rst +1 -0
@@ -23,5 +23,6 @@ Scheduler sched-stats sched-ext sched-debug + sched-cache text_files
Documentation/scheduler/sched-cache.rst new file +175 -0
175 lines not on this page · raw message
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerQuoted 3 lines · Tim Chen
Feedbacks very welcome, especially on the interface shape (prctl vs. a QoS attribute), the kernel-owned-cookie choice, and whether the always/advise/ never policy composition is the right model.
Who would be using this -- what workload prompted you do do this etc.
Hi Peter,
Quoted 11 lines · Peter Zijlstra
Subject: Re: [RFC PATCH 0/7] sched/cache: Per-task control of cache aware
scheduling via prctlQuoted 3 lines · Tim Chen
Feedbacks very welcome, especially on the interface shape (prctl vs. a QoS attribute), the kernel-owned-cookie choice, and whether the always/advise/ never policy composition is the right model.
Who would be using this -- what workload prompted you do do this etc.
One motivation is that some cloud users would like finer-grained control over cache‑aware scheduling. Vern Hao from Tencent previously asked about this:
https://lore.kernel.org/all/[email protected]/
and mentioned that, in their production environment, threads within the same process do not always share data. On the other hand, it is possible that within one process there are two thread groups, A and B. Threads in group A share data with each other, while threads in group B do not. Typically, in Vern's environment, group A and group B are cgroups. Group A usually runs memory‑intensive workloads, such as KV‑cache related ones, and such workloads have intensive data sharing among themselves, so they would like to enable cache‑aware scheduling separately. Furthermore, since group A is memory‑intensive, the default cache‑aware scheduling threshold might reject aggregation because group A's memory footprint is high. As a result, group A has a requirement to turn the threshold parameter separately.
thanks,
Chenyu
Quoted 20 lines · Chen Yu
Hi Peter,
Quoted 11 lines · Peter Zijlstra
Subject: Re: [RFC PATCH 0/7] sched/cache: Per-task control of cache aware
scheduling via prctlQuoted 3 lines · Tim Chen
Feedbacks very welcome, especially on the interface shape (prctl vs. a QoS attribute), the kernel-owned-cookie choice, and whether the always/advise/ never policy composition is the right model.
Who would be using this -- what workload prompted you do do this etc.
One motivation is that some cloud users would like finer-grained control over cache‑aware scheduling. Vern Hao from Tencent previously asked about this:
Quoted 10 lines
and mentioned that, in their production environment, threads within the same process do not always share data. On the other hand, it is possible that within one process there are two thread groups, A and B. Threads in group A share data with each other, while threads in group B do not. Typically, in Vern's environment, group A and group B are cgroups. Group A usually runs memory‑intensive workloads, such as KV‑cache related ones, and such workloads have intensive data sharing among themselves, so they would like to enable cache‑aware scheduling separately. Furthermore, since group A is memory‑intensive, the default cache‑aware scheduling threshold might reject aggregation because group A's memory footprint is high. As a result, group A has a requirement to turn the threshold parameter separately.
I also remembered in discussions with Vern, His usage scenario has processes each comprising of threads doing different functions, like one thread responsible for database lookup, one for encryption/decryption and one for file IO ...etc. So the threads in different processes performing similar function has more common data and perform better when grouped together.
Also in separate discussions with Qais, he has also mentioned that for his environment, tasks in the same process may not share data.
https://lore.kernel.org/lkml/20260219140828.a7pyzupun7lsdw34@airbuntu/ :
Quoted 6 lines
Quoted 2 lines
This initial implementation treats threads within the same process as entities that are likely to share data. During load balancing, the
This is a very aggressive assumption. From what I've seen, only few tasks truly share data. Lumping everything in a process together is an easy way to classify, but I think we can do better.
So this series is an attempt to address such cases where grouping tasks by other criteria than mm makes sense.
Tim
Hi Tim/Peter.
I have been trying to catch up. I still have to read and might have missed some conversation details. So please bear with me for silly questions.
Quoted 16 lines · Tim Chen
Hi all,
Cache aware scheduling today groups tasks by their mm: the LLC aggregation target lives in
mm_struct, so the address space is the unit of grouping. That works, but in some scenarios that is too coarse and too eager, and the only knob we have over it is a single system-wide debugfs switch.It's too coarse because plenty of workloads share data across cooperating *processes* rather than threads - a database with a process per connection, a browser with a renderer per site, a server and its worker helpers. They pass data through shm or pipes and would love to be pulled onto the same LLC, but they never share an mm, so today they can't be. And it's too eager in the other direction: a process whose threads don't actually share anything gets aggregated anyway, just because they happen to sit in one address space.
Quoted 3 lines
So the core idea of this series is simple: allow other groupings than the mm, make the grouping an object in its own right, and let user space say "put these tasks together" explicitly.
So, As you said, this is effectively asking user to make the decision.
But what tools do user space have today to make effective decisions? Application changes could turn out to be tricky to do and how an application developer will know whether to group them together or not? What's guidance there?
Can the grouping be done post the application started running? Like any option that says these pid's are to be bundled into one group?
I remember you guys discussed about cgroup and decided it is not a good option. That argument is still holds?
Quoted 5 lines · Shrikanth Hegde
Hi Tim/Peter.
I have been trying to catch up. I still have to read and might have missed some conversation details. So please bear with me for silly questions.
Thanks for taking a look. You questions are helpful for providing the context of why this series was proposed.
Quoted 18 lines · Tim Chen
Quoted 16 lines
Hi all,
Cache aware scheduling today groups tasks by their mm: the LLC aggregation target lives in
mm_struct, so the address space is the unit of grouping. That works, but in some scenarios that is too coarse and too eager, and the only knob we have over it is a single system-wide debugfs switch.It's too coarse because plenty of workloads share data across cooperating *processes* rather than threads - a database with a process per connection, a browser with a renderer per site, a server and its worker helpers. They pass data through shm or pipes and would love to be pulled onto the same LLC, but they never share an mm, so today they can't be. And it's too eager in the other direction: a process whose threads don't actually share anything gets aggregated anyway, just because they happen to sit in one address space.
Quoted 5 lines
Quoted 3 lines
So the core idea of this series is simple: allow other groupings than the mm, make the grouping an object in its own right, and let user space say "put these tasks together" explicitly.
So, As you said, this is effectively asking user to make the decision.
By default, tasks are grouped by process and that make sense in many cases. But sometimes the users have information about task characteristics that they wish to group tasks in other ways.
In our discussions with Vern Hao from Tencent, they have multiple processes in their workload, where some tasks in a process is responsible for database access, some for encryption, and some dealing with disk access. Those tasks across processes with similar function share more data than tasks in a process for their applications.
Another scenario is grouping processes with shared memory together.
Quoted 2 lines
But what tools do user space have today to make effective decisions?
As in the example above, this is for users who know about their workload characteristics and wish to group their tasks in other way than the default process grouping.
Also if people identify via perf c2c that tasks
Quoted 3 lines
Application changes could turn out to be tricky to do and how an application developer will know whether to group them together or not? What's guidance there?
No changes is required on application. An admin or a separate daemon can use prctl to group tasks together by sepcifying the pids pair of tasks to be grouped. Please see the PR_SCHED_CACHE_SHARE_FROM operation in patch 7 of the documentation.
Quoted 3 lines
Can the grouping be done post the application started running? Like any option that says these pid's are to be bundled into one group?
Yes.
Quoted 3 lines
I remember you guys discussed about cgroup and decided it is not a good option. That argument is still holds?
I think there is no strong case to support that tasks sharing
data necessarily belong in a cgroup. Using cgroup wouldn't cover all the use cases we want. With the proposed prctl based interface in this series,
the administrator can easily group the processes in a cgroup together if that makes sense. We also would rather not disturb the cgroup interface unnecessarily.
Tim