Fixes for cache aware scheduling
cover Fixes for cache aware scheduling
Greg,
Here is a series of patches backported to v7.2.8 (commit 9a66fdc0d7fd) to fix cache aware scheduling.
The original commit ids in upstream are listed as below.
Upstream commit ids are also listed at the beginning of commit log of each patch:
commit f0d243a96f26 ("sched/fair: Avoid creating misfits during cache-aware balancing")
commit 0d6526f82c3c ("sched/cache: Keep nr_pref_llc_running in the runnable domain, to fix LLC mis-scheduling bug") commit d6013e2465d9 ("sched/cache: Honor migrate_llc_task semantics in active load balance, to fix LLC mis-scheduling bug") commit 28f9c0e0a0b9 ("sched/cache: Decouple sched_cache_group from mm to fix UAF")
commit b636fef85bda ("sched/cache: Introduce task_struct->sched_cache_grp to fix UAF")
commit 65efcccddc83 ("sched/cache: Skip kernel threads for cache aware scheduling to rubustify the code") commit 3cb0243767fd ("sched/cache: Refresh LLC capacity across CPU hotplug, to fix capacity underestimation bug")
Thanks.
Tim
Chen Yu (1):
sched/cache: Skip kernel threads for cache aware scheduling to
rubustify the code
Davi Chaves Azevedo (1):
sched/cache: Refresh LLC capacity across CPU hotplug, to fix capacity
underestimation bug
Lu Wang (1):
sched/cache: Honor migrate_llc_task semantics in active load balance,
to fix LLC mis-scheduling bug
Tim Chen (4):
sched/fair: Avoid creating misfits during cache-aware balancing sched/cache: Keep nr_pref_llc_running in the runnable domain, to fix
LLC mis-scheduling bug
sched/cache: Decouple sched_cache_group from mm to fix UAF sched/cache: Introduce task_struct->sched_cache_grp to fix UAF
drivers/base/cacheinfo.c | 11 +- fs/exec.c | 1 + include/linux/mm_types.h | 15 +- include/linux/sched.h | 20 +- include/linux/sched/topology.h | 4 +- kernel/exit.c | 28 +- kernel/fork.c | 2 + kernel/sched/fair.c | 467 +++++++++++++++++++++++++++------ kernel/sched/topology.c | 22 +- 9 files changed, 432 insertions(+), 138 deletions(-)
Signature · 1 line
2.32.0
PATCH 1/7 sched/fair: Avoid creating misfits during cache-aware balancing
[ Upstream commit f0d243a96f2684ad771d678767d17972cf840bd7 ]
Cache-aware load balancing biases tasks toward their preferred LLC. On asymmetric CPU capacity systems (e.g. big.LITTLE) the destination LLC may contain CPUs that are too small to run the task. Pulling the task there turns it into a misfit, trading a cache-locality gain for a capacity loss that's more detrimental to performance.
Guard both cache-aware migration entry points against this:
- can_migrate_llc_task(): forbid the LLC migration when the task fits its source CPU but would not fit the destination CPU.
- alb_break_llc(): veto the active balance under the same condition so the runnable task is not pushed onto a CPU that cannot accommodate it.
Both checks are gated with checks for hybrid processors, so symmetric systems are unaffected. Tasks that already do not fit their source CPU are left to the existing LLC policy, since the move cannot make their fitness worse (this also preserves misfit up-migration to bigger CPUs).
Additionally, if there are misfit tasks found in the load balancing classification phase, prioritize misfit task migrations over LLC load aggregation on asymmetric systems. A better fitting CPU will boost performance more than better cache locality.
Stable-dep-of: d6013e2465d9 ("sched/cache: Honor migrate_llc_task semantics in active load balance, to fix LLC mis-scheduling bug")
--- kernel/sched/fair.c | 50 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 45 insertions(+), 5 deletions(-)
kernel/sched/fair.c +45 -5
@@ -10575,17 +10575,40 @@ static enum llc_mig can_migrate_llc(int src_cpu, int dst_cpu,return mig_llc; } +static inline bool task_misfits_asym_cpu (struct lb_env *env,struct task_struct *p) +{ +/* +* On asymmetric CPU capacity domains, do not let cache-aware +* balancing pull the task onto a destination CPU that cannot +* accommodate it. Doing so would turn the task into a misfit on +* the destination, trading a cache-locality gain for a capacity +* loss. If the task already does not fit its source CPU, the move +* cannot make things worse, so let the LLC preference decide. +*/ +if ((env->sd->flags &SD_ASYM_CPUCAPACITY ) && p && + !task_fits_cpu (p, env->dst_cpu) && +task_fits_cpu (p, env->src_cpu)) +return true; + +return false; +} +/* * Check if task p can migrate from source LLC to * destination LLC in terms of cache aware load balance. */ -static enum llc_mig can_migrate_llc_task (int src_cpu,int dst_cpu, +static enum llc_mig can_migrate_llc_task (struct lb_env *env,struct task_struct *p) {struct mm_struct *mm;bool to_pref; -int cpu; +int cpu, src_cpu, dst_cpu; + +if (task_misfits_asym_cpu (env, p)) +return mig_forbid; + src_cpu = env->src_cpu; + dst_cpu = env->dst_cpu; mm = p->mm;if (!mm)return mig_unrestricted;
@@ -10642,6 +10665,14 @@ alb_break_llc(struct lb_env *env)unsigned long util =0 ;struct task_struct *cur; +/* +* Migrating misfit tasks from current CPU +* to CPU with a better fit. +* Prioritize that over LLC preference. +*/ +if (env->migration_type == migrate_misfit) +return false; +if (env->src_rq->nr_running <=1 )return true;
@@ -10649,7 +10680,8 @@ alb_break_llc(struct lb_env *env)if (cur && cur->sched_class == &fair_sched_class) util =task_util (cur); -if (can_migrate_llc (env->src_cpu, env->dst_cpu, +if (task_misfits_asym_cpu (env, cur) || +can_migrate_llc (env->src_cpu, env->dst_cpu, util, false) == mig_forbid)return true; }
@@ -10689,8 +10721,7 @@ static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env) READ_ONCE(p->preferred_llc) != llc_id(env->dst_cpu))return true; -if (can_migrate_llc_task (env->src_cpu, - env->dst_cpu, p) != mig_forbid) +if (can_migrate_llc_task (env, p) != mig_forbid)return false;return true;
@@ -11753,6 +11784,15 @@ static inline bool llc_balance(struct lb_env *env, struct sg_lb_stats *sgs,if (env->sd->flags &SD_SHARE_LLC )return false; +/* +* On asymmetric domains, group_misfit_task_load +* should be prioritized to move tasks to CPU that fit them +* over aggregating tasks to their preferred LLC. +*/ +if ((env->sd->flags &SD_ASYM_CPUCAPACITY ) && + sgs->group_misfit_task_load) +return false; +/* * Skip cache aware tagging if nr_balanced_failed is sufficiently high. * Threshold of cache_nice_tries is set to 1 higher than nr_balance_failed
Signature · 1 line
2.32.0
PATCH 2/7 sched/cache: Keep nr_pref_llc_running in the runnable domain, to fix LLC mis-scheduling bug
In reply to Tim Chen[ Upstream commit 0d6526f82c3cdefcca47f73f5fc08dc6f335eac6 ]
alb_break_llc() decides whether to break LLC preference during active load balance. It does so by testing that every runnable fair task on the source rq prefers its LLC:
env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable
But the two counters cover different sets. nr_pref_llc_running is updated in account_llc_enqueue()/account_llc_dequeue(), next to cfs_rq->nr_queued, so it follows queued tasks. h_nr_runnable is updated in set_delayed()/ clear_delayed() and drops delay-dequeued tasks.
So under DELAY_DEQUEUE, a preferring task that goes to sleep stays counted in nr_pref_llc_running while h_nr_runnable falls. The equality then breaks, alb_break_llc() returns false, and active balance is free to pull a task off its preferred LLC. Active balance only moves runnable tasks, and this is the only LLC check it consults: once the stopper runs, LBF_ACTIVE_LB skips the per-task test in can_migrate_task(). The runnable set is the one we want.
Fix it on the counter side. A task should be counted in nr_pref_llc_running exactly while it is both queued on its preferred LLC (pref_llc_queued) and runnable (!sched_delayed). Define that membership once in task_pref_llc_runnable(), and adjust the counter only through pref_llc_running_inc()/pref_llc_running_dec() from the four sites that change either input: account_llc_enqueue(), account_llc_dequeue(), set_delayed() and clear_delayed(). Gating every update on the same predicate keeps the delay, wake and dequeue paths from double-counting or underflowing; see the comments at those sites for the ordering.
nr_llc_running and sd->llc_counts are not touched and stay on queued semantics.
--- kernel/sched/fair.c | 63 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 58 insertions(+), 5 deletions(-)
kernel/sched/fair.c +58 -5
@@ -1507,6 +1507,28 @@ static bool invalid_llc_nr(struct mm_struct *mm, struct task_struct *p, (scale * per_cpu(sd_llc_size, cpu))); } +/* +* A task counts in nr_pref_llc_running while it is queued on its preferred +* LLC (pref_llc_queued) and runnable (!sched_delayed), keeping the counter in +* the runnable domain so alb_break_llc() can compare it with h_nr_runnable. +*/ +static bool task_pref_llc_runnable (struct task_struct *p) +{ +return p->pref_llc_queued && !p->se.sched_delayed; +} + +static void pref_llc_running_inc (struct rq *rq,struct task_struct *p) +{ +if (task_pref_llc_runnable (p)) + rq->nr_pref_llc_running++; +} + +static void pref_llc_running_dec (struct rq *rq,struct task_struct *p) +{ +if (task_pref_llc_runnable (p)) + rq->nr_pref_llc_running--; +} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) {int pref_llc, pref_llc_queued;
@@ -1518,7 +1540,6 @@ static void account_llc_enqueue(struct rq *rq, struct task_struct *p) pref_llc_queued = (pref_llc ==task_llc (p)); rq->nr_llc_running++; - rq->nr_pref_llc_running += pref_llc_queued;/* * Record whether p is enqueued on its preferred
@@ -1536,6 +1557,9 @@ static void account_llc_enqueue(struct rq *rq, struct task_struct *p)*/ p->pref_llc_queued = pref_llc_queued; +/* Skipped while delayed; clear_delayed() adds it back on wake. */ +pref_llc_running_inc (rq, p); + sd =rcu_dereference_all (rq->sd);if (sd && (unsigned int )pref_llc < sd->llc_max) sd->llc_counts[pref_llc]++;
@@ -1552,7 +1576,12 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p) rq->nr_llc_running--;if (p->pref_llc_queued) { - rq->nr_pref_llc_running--; +/* +* Skipped if still delayed (set_delayed() already removed it); +* clearing pref_llc_queued below also stops clear_delayed() +* from re-adding it. +*/ +pref_llc_running_dec (rq, p);/* * Update the status in case * other logic might query
@@ -1956,6 +1985,7 @@ void init_sched_mm(struct task_struct *p)* polluting account_llc_enqueue(). */ p->preferred_llc =-1 ; + p->pref_llc_queued = 0; }#else /* CONFIG_SCHED_CACHE */
@@ -1977,6 +2007,10 @@ static void account_llc_enqueue(struct rq *rq, struct task_struct *p) {}static void account_llc_dequeue (struct rq *rq,struct task_struct *p) {} +static void pref_llc_running_inc (struct rq *rq,struct task_struct *p) {} + +static void pref_llc_running_dec (struct rq *rq,struct task_struct *p) {} +#endif /* CONFIG_SCHED_CACHE */ /*
@@ -6202,15 +6236,27 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);static void set_delayed (struct sched_entity *se) { - se->sched_delayed =1 ; -/* * Delayed se of cfs_rq have no tasks queued on them. * Do not adjust h_nr_runnable since dequeue_entities() * will account it for blocked tasks. +* +* This check can be removed because when flat pick +* patches get merged as only task can get delayed, +* same for clear_delayed(). */ -if (!entity_is_task (se)) +if (!entity_is_task (se)) { + se->sched_delayed =1 ;return ; + } + +/* +* Drop a task leaving the runnable set. +* Needs to be called before sched_delayed is set. +* clear_delayed() mirrors this after clearing the flag. +*/ +pref_llc_running_dec (rq_of (cfs_rq_of (se)),task_of (se)); + se->sched_delayed =1 ;for_each_sched_entity (se) {struct cfs_rq *cfs_rq =cfs_rq_of (se);
@@ -6232,6 +6278,13 @@ static void clear_delayed(struct sched_entity *se)if (!entity_is_task (se))return ; +/* +* Re-add on wake, after sched_delayed is cleared. On a final delayed +* dequeue account_llc_dequeue() already cleared pref_llc_queued, so +* this does nothing. +*/ +pref_llc_running_inc (rq_of (cfs_rq_of (se)),task_of (se)); +for_each_sched_entity (se) {struct cfs_rq *cfs_rq =cfs_rq_of (se);
Signature · 1 line
2.32.0
PATCH 3/7 sched/cache: Honor migrate_llc_task semantics in active load balance, to fix LLC mis-scheduling bug
In reply to Tim ChenFrom: Lu Wang <[email protected]>
[ Upstream commit d6013e2465d98d524b030a81c1223882a1bb7e4c ]
Cache aware scheduling introduced the migrate_llc_task migration type to direct tasks toward their preferred LLC, but its semantics can be lost when passive load balance falls back to active load balance (ALB). This may allow ALB to select a candidate whose preferred LLC does not match the destination, moving it away from its preferred LLC.
Example scenario:
src_rq has two runnable tasks, p1 and p2. p1 prefers dst_rq (dst_llc), while p2 prefers src_rq (src_llc). In this case, migrate_llc_task is set because src_rq has at least one task, p1, that wants to migrate to dst_rq. In ALB, can_migrate_task() finds p2 and returns true for it, thus moving p2 out of its preferred LLC.
Solution:
The CPU stopper in ALB constructs a fresh lb_env that does not inherit migration_type from the passive load-balance pass. Two approaches are possible:
(a) Add a new member to struct rq so ALB can inherit migrate_llc_task from the passive LB that triggered it.
(b) Define a new flag LBF_ACTIVE_LB_LLC and select the stopper callback at kick time to preserve the migration semantics across the asynchronous boundary.
We choose (b) because it avoids passing migration_type through the stopper, which would affect the meaning of migration_type for delayed-dequeue tasks.
--- kernel/sched/fair.c | 57 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 51 insertions(+), 6 deletions(-)
kernel/sched/fair.c +51 -6
@@ -10319,6 +10319,7 @@ enum migration_type {#define LBF_SOME_PINNED 0x08#define LBF_ACTIVE_LB 0x10#define LBF_LLC_PINNED 0x20 +#define LBF_ACTIVE_LB_LLC 0x40struct lb_env {struct sched_domain *sd;
@@ -10742,6 +10743,21 @@ alb_break_llc(struct lb_env *env)return false; } +/* +* Returns true if p's preferred LLC does not match the destination CPU +* under migrate_llc_task semantics. Passive LB passes migrate_llc_task +* in env->migration_type, while active LB carries LBF_ACTIVE_LB_LLC in +* env->flags to avoid overwriting env->migration_type. +*/ +static inline bool +migrate_llc_task_wrong_dst (struct task_struct *p,struct lb_env *env) +{ +return sched_cache_enabled () && + (env->migration_type == migrate_llc_task || + env->flags &LBF_ACTIVE_LB_LLC ) && +READ_ONCE (p->preferred_llc) !=llc_id (env->dst_cpu); +} +/* * Check if migrating task p from env->src_cpu to * env->dst_cpu breaks LLC localiy.
@@ -10770,8 +10786,7 @@ static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env)* run on env->dst_cpu, skip the tasks do not prefer * env->dst_cpu, and find the one that prefers. */ -if (env->migration_type == migrate_llc_task && -READ_ONCE (p->preferred_llc) !=llc_id (env->dst_cpu)) +if (migrate_llc_task_wrong_dst (p, env))return true;if (can_migrate_llc_task (env, p) != mig_forbid)
@@ -10793,6 +10808,12 @@ alb_break_llc(struct lb_env *env)return false; } +static inline bool +migrate_llc_task_wrong_dst (struct task_struct *p,struct lb_env *env) +{ +return false; +} +static inline bool migrate_degrades_llc (struct task_struct *p,struct lb_env *env) {
@@ -10892,7 +10913,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* 4) too many balance attempts have failed. */ if (env->flags &LBF_ACTIVE_LB ) -return 1 ; +return !migrate_llc_task_wrong_dst (p, env); degrades =migrate_degrades_locality (p, env);if (!degrades) {
@@ -13263,6 +13284,20 @@ static int need_active_balance(struct lb_env *env) }static int active_load_balance_cpu_stop (void *data); +static int active_load_balance_llc_cpu_stop (void *data); + +/* +* migration_type is checked elsewhere to decide migration policy, so +* it shouldn't be repurposed just to flag an LLC-directed active +* balance across the stopper. Pick the callback here instead. +*/ +static inline cpu_stop_fn_t alb_stop_fn (struct lb_env *env) +{ +if (env->migration_type == migrate_llc_task) +return active_load_balance_llc_cpu_stop; + +return active_load_balance_cpu_stop; +}static int should_we_balance (struct lb_env *env) {
@@ -13599,7 +13634,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq,raw_spin_rq_unlock_irqrestore (busiest, flags);if (active_balance) {stop_one_cpu_nowait (cpu_of (busiest), - active_load_balance_cpu_stop, busiest, +alb_stop_fn (&env), busiest, &busiest->active_balance_work); }preempt_enable ();
@@ -13710,7 +13745,7 @@ update_next_balance(struct sched_domain *sd, unsigned long *next_balance)* least 1 task to be running on each physical CPU where possible, and * avoids physical / logical imbalances. */ -static int active_load_balance_cpu_stop (void *data) +static int __active_load_balance_cpu_stop (void *data,unsigned int lb_flags) {struct rq *busiest_rq = data;int busiest_cpu =cpu_of (busiest_rq);
@@ -13760,7 +13795,7 @@ static int active_load_balance_cpu_stop(void *data) .src_cpu = busiest_rq->cpu, .src_rq = busiest_rq, .idle = CPU_IDLE, - .flags = LBF_ACTIVE_LB, + .flags = LBF_ACTIVE_LB | lb_flags, };schedstat_inc (sd->alb_count);
@@ -13788,6 +13823,16 @@ static int active_load_balance_cpu_stop(void *data)return 0 ; } +static int active_load_balance_cpu_stop (void *data) +{ +return __active_load_balance_cpu_stop (data,0 ); +} + +static int active_load_balance_llc_cpu_stop (void *data) +{ +return __active_load_balance_cpu_stop (data,LBF_ACTIVE_LB_LLC ); +} +/* * Scale the max sched_balance_rq interval with the number of CPUs in the system. * This trades load-balance latency on larger machines for less cross talk.
Signature · 1 line
2.32.0
PATCH 4/7 sched/cache: Decouple sched_cache_group from mm to fix UAF
In reply to Tim Chen[ Upstream commit 28f9c0e0a0b94c5d3e1b634db545f6e1f94858c5 ]
Currently the sched cache grouping is by mm and the scheduling statistics sched_cache_stat lives in the mm structure. This ties the life cycle of scheduling stats with mm.
In account_mm_sched(), the scheduling stats are accessed by task->mm->sc_stat. However, a task may be switching mm on one CPU when another CPU is running account_mm_sched(), and possibly accessing the old mm that was freed. This problem was found when running tests with KASAN by Hyunwoo:
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Instead of serializing the mm access by introducing extra acquisition of rq lock in the mm free path, extract sched_cache_stat from mm_struct, rename it as sched_cache_group and manage its life cycle apart from mm_struct with its own ref counting. This allows us in the next patch access sched_cache_group directly from task, and add a refcount on sched_cache_group when a task links to it. This prevents the use after free issue when accessing stale and released old mm and its sched cache stat a task switches to a new mm while account_mm_sched() is done elsewhere.
The other benefit of this restructure is in the future, the grouping of tasks to a LLC would have the flexibility to be associated with a user defined grouping, or cgroup, cookie group, numa_group or others instead of just with a single mm address space.
Rename sched_cache_stat to sched_cache_group and turn it into a refcounted object allocated from mm_struct. The mm_struct now holds a pointer (sched_cache_grp) to this object instead of embedding it.
--- include/linux/mm_types.h | 15 ++-- include/linux/sched.h | 6 +- kernel/exit.c | 11 ++- kernel/sched/fair.c | 173 ++++++++++++++++++++++++++++----------- 4 files changed, 144 insertions(+), 61 deletions(-)
include/linux/mm_types.h +5 -10
@@ -1211,7 +1211,7 @@ struct mm_struct {struct mm_mm_cid mm_cid;/* sched_cache related statistics */ -struct sched_cache_stat sc_stat; +struct sched_cache_group *sched_cache_grp;#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */ #endif
@@ -1609,8 +1609,9 @@ static inline unsigned int mm_cid_size(void)#endif /* CONFIG_SCHED_MM_CID */ #ifdef CONFIG_SCHED_CACHE -void mm_init_sched (struct mm_struct *mm, -struct sched_cache_time __percpu *pcpu_sched); +int mm_init_sched (struct mm_struct *mm, +struct sched_cache_time __percpu *pcpu_sched); +void mm_destroy_sched (struct mm_struct *mm);static inline int mm_alloc_sched_noprof (struct mm_struct *mm) {
@@ -1620,17 +1621,11 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)if (!pcpu_sched)return -ENOMEM ; -mm_init_sched (mm, pcpu_sched); - return 0; + return mm_init_sched(mm, pcpu_sched); }#define mm_alloc_sched (...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) -static inline void mm_destroy_sched (struct mm_struct *mm) -{ -free_percpu (mm->sc_stat.pcpu_sched); - mm->sc_stat.pcpu_sched =NULL ; -}#else /* !CONFIG_SCHED_CACHE */ static inline int mm_alloc_sched (struct mm_struct *mm) {return 0 ; }
include/linux/sched.h +4 -2
@@ -2398,7 +2398,7 @@ struct sched_cache_time {unsigned long epoch; }; -struct sched_cache_stat { +struct sched_cache_group {struct sched_cache_time __percpu *pcpu_sched;raw_spinlock_t lock;unsigned long epoch;
@@ -2406,11 +2406,13 @@ struct sched_cache_stat {unsigned long next_scan;unsigned long footprint;int cpu; +refcount_t refcnt; + struct rcu_head rcu; }____cacheline_aligned_in_smp ;#else -struct sched_cache_stat { }; +struct sched_cache_group { };#endif
kernel/exit.c +8 -3
@@ -559,18 +559,23 @@ void mm_update_next_owner(struct mm_struct *mm)*/ static void exit_mm_sched_cache (struct mm_struct *mm) { +struct sched_cache_group *grp;unsigned long fp, sub;if (!current->total_numa_faults)return ;/* * No lock protection due to performance considerations. -* Make sure mm->sc_stat.footprint does not become +* Make sure the group footprint does not become * negative. */ - fp =READ_ONCE (mm->sc_stat.footprint); + grp =READ_ONCE (mm->sched_cache_grp); +if (!grp) +return ; + + fp =READ_ONCE (grp->footprint); sub =min (fp, current->total_numa_faults); -WRITE_ONCE (mm->sc_stat.footprint, fp - sub); +WRITE_ONCE (grp->footprint, fp - sub); }#else static inline void exit_mm_sched_cache (struct mm_struct *mm)
kernel/sched/fair.c +127 -46
367 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH 5/7 sched/cache: Introduce task_struct->sched_cache_grp to fix UAF
In reply to Tim Chen[ Upstream commit b636fef85bda7d1bab9c0a45067ab1508d79d946 ]
Add a sched_cache_grp pointer to task_struct so that scheduler code can access the cache group directly via the task, without going through mm->sched_cache_grp. This decouples the scheduler's hot-path accesses from the mm_struct.
Each task holds its own refcount on the sched_cache_group, separate from the reference held by its mm_struct. The reference is acquired in copy_mm() (fork) and exec_mmap() (exec), and released in exit_mm(). This fixes the use-after-free when account_mm_sched() reaches the group through a task whose mm is being switched, as reported by Hyunwoo:
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Convert all scheduler code in fair.c and exit.c to use p->sched_cache_grp instead of p->mm->sched_cache_grp.
Keep the fork/exec/exit reference management out of the generic mm
paths: add sched_cache_fork(), sched_cache_fork_cleanup(), sched_cache_exec_mmap() and sched_cache_exit_mm() in kernel/sched/cache_sched.c (with empty stubs for !CONFIG_SCHED_CACHE), so fs/exec.c, kernel/fork.c and kernel/exit.c each call one helper instead of open-coding the refcounting under #ifdef. Also add sched_cache_group_get() and task_cache_group_get().
--- fs/exec.c | 1 + include/linux/sched.h | 14 +++ kernel/exit.c | 33 +------ kernel/fork.c | 2 + kernel/sched/fair.c | 196 +++++++++++++++++++++++++++++------------- 5 files changed, 154 insertions(+), 92 deletions(-)
fs/exec.c +1 -0
@@ -882,6 +882,7 @@ static int exec_mmap(struct linux_binprm *bprm) active_mm = tsk->active_mm; tsk->active_mm = mm; tsk->mm = mm; +sched_cache_exec_mmap (tsk, mm);mm_init_cid (mm, tsk); exec_state =task_exec_state_replace (tsk, exec_state);/*
include/linux/sched.h +14 -0
@@ -1421,6 +1421,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work; +struct sched_cache_group __rcu *sched_cache_grp;int preferred_llc;/* 1: task was enqueued to its preferred LLC, 0 otherwise */ int pref_llc_queued;
@@ -2410,10 +2411,23 @@ struct sched_cache_group {struct rcu_head rcu; } ____cacheline_aligned_in_smp; +struct sched_cache_group *sched_cache_group_get (struct sched_cache_group *grp); +struct sched_cache_group *task_cache_group_get (struct task_struct *p); + +void sched_cache_fork (struct task_struct *p); +void sched_cache_fork_cleanup (struct task_struct *p); +void sched_cache_exec_mmap (struct task_struct *p,struct mm_struct *mm); +void sched_cache_exit_mm (struct task_struct *p); +#else struct sched_cache_group { }; +static inline void sched_cache_fork (struct task_struct *p) { } +static inline void sched_cache_fork_cleanup (struct task_struct *p) { } +static inline void sched_cache_exec_mmap (struct task_struct *p,struct mm_struct *mm) { } +static inline void sched_cache_exit_mm (struct task_struct *p) { } +#endif #ifndef MODULE
kernel/exit.c +1 -32
@@ -552,37 +552,6 @@ void mm_update_next_owner(struct mm_struct *mm) }#endif /* CONFIG_MEMCG */ -#if defined(CONFIG_SCHED_CACHE ) && defined(CONFIG_NUMA_BALANCING ) -/* -* Subtract the memory footprint of the current task from -* mm. -*/ -static void exit_mm_sched_cache (struct mm_struct *mm) -{ -struct sched_cache_group *grp; -unsigned long fp, sub; - -if (!current->total_numa_faults) -return ; -/* -* No lock protection due to performance considerations. -* Make sure the group footprint does not become -* negative. -*/ - grp =READ_ONCE (mm->sched_cache_grp); -if (!grp) -return ; - - fp =READ_ONCE (grp->footprint); - sub =min (fp, current->total_numa_faults); -WRITE_ONCE (grp->footprint, fp - sub); -} -#else -static inline void exit_mm_sched_cache (struct mm_struct *mm) -{ -} -#endif /* CONFIG_SCHED_CACHE CONFIG_NUMA_BALANCING */ -/* * Turn us into a lazy TLB process if we * aren't already..
@@ -595,7 +564,7 @@ static void exit_mm(void)if (!mm)return ; -exit_mm_sched_cache (mm); +sched_cache_exit_mm (current);mmap_read_lock (mm);mmgrab_lazy_tlb (mm);
kernel/fork.c +2 -0
@@ -1598,6 +1598,7 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk) tsk->mm = mm; tsk->active_mm = mm; +sched_cache_fork (tsk);return 0 ; }
@@ -2583,6 +2584,7 @@ __latent_entropy struct task_struct *copy_process( bad_fork_cleanup_namespaces:exit_nsproxy_namespaces (p); bad_fork_cleanup_mm: +sched_cache_fork_cleanup (p);if (p->mm) {mm_clear_owner (p->mm, p);mmput (p->mm);
kernel/sched/fair.c +136 -60
347 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH 6/7 sched/cache: Skip kernel threads for cache aware scheduling to rubustify the code
In reply to Tim ChenFrom: Chen Yu <[email protected]>
[ Upstream commit 65efcccddc83d6a19e8a2e2a6117811e39e589bf ]
Kernel thread should not be covered by cache aware scheduling as it borrows the statistics from the user space thread. Filter the kernel thread in account_mm_sched().
In theory a kernel thread does not have any valid cache group, so !grp should gate the kernel thread. Add the PF_KTHREAD check explicitly here for safety reasons, to guard against future modifications and to pair with task_tick_cache().
--- kernel/sched/fair.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
kernel/sched/fair.c +8 -2
@@ -1868,8 +1868,14 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)/* * init_task, kthreads and user thread created * by user_mode_thread() don't have a cache group. -*/ -if (!grp || !grp->pcpu_sched) +* In theory a kernel thread does not have any valid +* cache group, because sched_cache_fork() is not +* invoked for a kernel thread - !grp should gate the +* kernel thread. Use the PF_KTHREAD check explicitly +* here for safety reasons, to guard against future +* modifications and to pair with task_tick_cache(). +*/ +if (p->flags &PF_KTHREAD || !grp || !grp->pcpu_sched)return ; pcpu_sched =per_cpu_ptr (grp->pcpu_sched,cpu_of (rq));
Signature · 1 line
2.32.0
PATCH 7/7 sched/cache: Refresh LLC capacity across CPU hotplug, to fix capacity underestimation bug
In reply to Tim ChenFrom: Davi Chaves Azevedo <[email protected]>
[ Upstream commit 3cb0243767fd033bdce95f4f1b5882172a2f8119 ]
The scheduler scales LLC capacity by the fraction of cache-sharing CPUs covered by a domain:
llc_bytes = cache_size * span_weight / shared_weight
During CPU teardown, sched_cpu_deactivate() rebuilds scheduler domains before cacheinfo_cpu_pre_down() removes the CPU from shared_cpu_map. The new domains therefore use the old sharing weight. The later call to sched_update_llc_bytes() looks up the departing CPU's sd_llc, which has already been detached, and returns without correcting the surviving CPUs.
On a Ryzen 5 7535U with twelve logical CPUs sharing a 16 MiB LLC, offlining one SMT sibling left the remaining CPUs with:
llc_bytes = floor(16777216 * 11 / 12) = 15379114 bytes
The correct capacity is still 16777216 bytes. On systems with active cache-aware scheduling, an underestimated capacity can cause exceed_llc_capacity() to reject aggregation for a process whose footprint would fit. Unchanged cpuset partitions sharing the physical cache can also retain stale capacity when a CPU comes online in another partition.
Pass the cache-sharing mask already retained by cacheinfo to the scheduler update. Refresh every surviving CPU using its own LLC domain so that each partition receives the correct share. This also preserves the correction needed as cache-sharing maps grow during boot.
Keep the existing CPU-hotplug and scheduler-domain synchronization. The update remains on the hotplug path; no steady-state scheduling operation or persistent allocation is added.
--- drivers/base/cacheinfo.c | 11 ++++++----- include/linux/sched/topology.h | 4 ++-- kernel/sched/topology.c | 22 +++++++++++++--------- 3 files changed, 21 insertions(+), 16 deletions(-)
drivers/base/cacheinfo.c +6 -5
@@ -1040,9 +1040,10 @@ static int cacheinfo_cpu_online(unsigned int cpu) rc =cache_add_dev (cpu);if (rc) goto err; -if (cpu_map_shared_cache (true, cpu, &cpu_map)) +if (cpu_map_shared_cache (true, cpu, &cpu_map)) {update_per_cpu_data_slice_size (true, cpu, cpu_map); -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; err:free_cache_attributes (cpu);
@@ -1059,10 +1060,10 @@ static int cacheinfo_cpu_pre_down(unsigned int cpu)cpu_cache_sysfs_exit (cpu);free_cache_attributes (cpu); -if (nr_shared >1 ) +if (nr_shared >1 ) {update_per_cpu_data_slice_size (false, cpu, cpu_map); - -sched_update_llc_bytes (cpu); +sched_update_llc_bytes (cpu_map); + }return 0 ; }
include/linux/sched/topology.h +2 -2
@@ -281,9 +281,9 @@ static inline int task_node(const struct task_struct *p) }#ifdef CONFIG_SCHED_CACHE -extern void sched_update_llc_bytes (unsigned int cpu); +extern void sched_update_llc_bytes (const struct cpumask *cpus);#else -static inline void sched_update_llc_bytes (unsigned int cpu) { } +static inline void sched_update_llc_bytes (const struct cpumask *cpus) { }#endif #endif /* _LINUX_SCHED_TOPOLOGY_H */
kernel/sched/topology.c +13 -9
@@ -985,8 +985,8 @@ void sched_cache_active_set(void) }/* -* Update the bottom sched_domain's llc_bytes for @cpu and all its -* LLC siblings. Called from cacheinfo_cpu_online() or +* Update the bottom sched_domain's llc_bytes for @cpus sharing a physical +* LLC. Called from cacheinfo_cpu_online() or * cacheinfo_cpu_pre_down() with cpu hotplug lock held. * * Note: get_effective_llc_bytes() returns 0 on PowerPC.
@@ -996,17 +996,13 @@ void sched_cache_active_set(void)* and does not populates the per-CPU struct cpu_cacheinfo array * that get_cpu_cacheinfo_llc() reads. */ -void sched_update_llc_bytes (unsigned int cpu) +void sched_update_llc_bytes (const struct cpumask *cpus) {struct sched_domain *sd, *sdp;unsigned int i;sched_domains_mutex_lock (); - sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, cpu)); -if (!sdp) - goto unlock; -/* * ci->shared_cpu_map is built incrementally as CPUs come * online, so the first CPU in an LLC initially sees
@@ -1014,14 +1010,22 @@ void sched_update_llc_bytes(unsigned int cpu)* get_effective_llc_bytes(). Re-evaluating every LLC * sibling on each online event corrects this once the full * shared_cpu_map is known. +* +* The departing CPU's domains have already been detached when +* cacheinfo removes it. Use the surviving cache siblings instead. +* They may belong to different cpuset partitions, so use each CPU's +* own LLC domain to scale its share of the physical cache. */ -for_each_cpu (i,sched_domain_span (sdp)) { +for_each_cpu (i, cpus) { + sdp =rcu_dereference_sched_domain (per_cpu (sd_llc, i)); +if (!sdp) +continue ; + sd =rcu_dereference_sched_domain (cpu_rq (i)->sd);if (sd) sd->llc_bytes =get_effective_llc_bytes (i, sdp); } -unlock:sched_domains_mutex_unlock (); }
Signature · 1 line
2.32.0
Quoted 2 lines
Here is a series of patches backported to v7.2.8 (commit
9a66fdc0d7fd) to fix cache aware scheduling.
Queued for 7.2, thanks. Patches 4 and 7 were already queued.
Signature · 2 lines
Thanks, Sasha