sched/cache: Fix use after free mm access in account_mm_sched()
Subsystems (5)
cover sched/cache: Fix use after free mm access in account_mm_sched()
Hyunwoo Kim reported a KASAN use-after-free in account_mm_sched():
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Cache-aware scheduling keeps its per-address-space statistics inside the mm:
struct mm_struct {
...
struct sched_cache_stat sc_stat;
};so their lifetime is the mm's lifetime. mm_alloc_sched() allocates sc_stat.pcpu_sched from mm_init(), and mm_destroy_sched() frees it from __mmdrop().
The scheduler, though, reaches that object from contexts that hold no reference on the mm:
- account_mm_sched() runs from update_curr() holding only the rq lock,
and dereferences rq->curr->mm.
- the load-balance predicates - can_migrate_llc_task() -> invalid_llc_nr() / exceed_llc_capacity() - and the task_cache_work() LLC occupancy scan read p->mm of *remote* tasks.
Nothing on those paths keeps the mm alive, so an exit, or an exec_mmap() installing a new mm, can free pcpu_sched underneath a concurrent reader.
Serializing the two sides - taking the rq lock in the mm teardown path - would put a scheduler lock in the middle of __mmdrop(), which is a lot of coupling to pay for a statistics object. Give the object its own lifetime instead.
Patch 1 lifts sched_cache_stat out of mm_struct, renames it sched_cache_group, and turns it into a refcounted object freed via call_rcu(); the mm now merely points at it.
Patch 2 gives every task its own reference in task_struct->sched_cache_grp - taken in copy_mm() and exec_mmap(), dropped in exit_mm() - and converts the scheduler to read p->sched_cache_grp rather than p->mm->sc_stat. Readers access the sched_cache_grp without worry that it was freed as it has a ref count on the object.
The two patches are one fix. Patch 1 does not stand alone, so they need to be applied, and backported, as a pair.
A welcome side effect of the decoupling is that the group is no longer welded to an address space, so a later series can key it on a cgroup, a core-scheduling cookie or a numa_group instead of on a single mm.
These are the first two patches of the cache-aware prctl RFC series
https://lore.kernel.org/lkml/[email protected]/
reposted on their own with the changelogs rewritten and minor updates around the use-after-free, so that they can be considered ahead of the rest of that series. Hyunwoo confirmed the splat is gone; his Tested-by is on both patches.
Tim
---
Tim Chen (2):
sched/cache: Decouple sched_cache_group from mm sched/cache: Introduce task_struct->sched_cache_grp
fs/exec.c | 14 +++ include/linux/mm_types.h | 15 +-- include/linux/sched.h | 11 +- kernel/exit.c | 28 ++++-- kernel/fork.c | 23 +++++ kernel/sched/build_utility.c | 4 + kernel/sched/cache_sched.c | 39 ++++++++ kernel/sched/fair.c | 188 ++++++++++++++++++++++------------- kernel/sched/sched.h | 3 + 9 files changed, 239 insertions(+), 86 deletions(-) create mode 100644 kernel/sched/cache_sched.c
Signature · 1 line
2.32.0
PATCH 1/2 sched/cache: Decouple sched_cache_group from mm
Currently the sched cache grouping is by mm and the scheduling statistics sched_cache_stat lives in the mm structure. This ties the life cycle of scheduling stats with mm.
In account_mm_sched(), the scheduling stats are accessed by task->mm->sc_stat. However, a task may be switching mm on one CPU when another CPU is running account_mm_sched(), and possibly accessing the old mm that was freed. This problem was found when running tests with KASAN by Hyunwoo. https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Instead of serializing the mm access by introducing extra acquisition of rq lock in the mm free path, extract sched_cache_stat from mm_struct, rename it as sched_cache_group and manage its life cycle apart from mm_struct with its own ref counting. Access sched_cache_group directly from a task instead of having to go through a task's mm. This allows us to later add a refcount on sched_cache_group when a task links to it, preventing the use after free issue when accessing stale and released old mm and its sched cache stat a task switches to a new mm while account_mm_sched() is done elsewhere.
The other benefit of this restructure is in the future, the grouping of tasks to a LLC would have the flexibility to be associated with cgroup, cookie group, numa_group or others instead of just with a single mm address space.
Rename sched_cache_stat to sched_cache_group and turn it into a refcounted object allocated from mm_struct. The mm_struct now holds a pointer (sched_cache_grp) to this object instead of embedding it.
Introduce kernel/sched/cache_sched.c to host the cache aware scheduling helpers and define sched_cache_group_put() there.
Meanwhile skip kthreads in account_mm_sched(), consistent with task_tick_cache().
--- include/linux/mm_types.h | 15 ++--- include/linux/sched.h | 8 ++- kernel/exit.c | 6 +- kernel/sched/build_utility.c | 4 ++ kernel/sched/cache_sched.c | 20 ++++++ kernel/sched/fair.c | 118 ++++++++++++++++++++++------------- 6 files changed, 113 insertions(+), 58 deletions(-) create mode 100644 kernel/sched/cache_sched.c
include/linux/mm_types.h +5 -10
@@ -1226,7 +1226,7 @@ struct mm_struct {struct mm_mm_cid mm_cid;/* sched_cache related statistics */ -struct sched_cache_stat sc_stat; +struct sched_cache_group *sched_cache_grp;#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */ #endif
@@ -1624,8 +1624,9 @@ static inline unsigned int mm_cid_size(void)#endif /* CONFIG_SCHED_MM_CID */ #ifdef CONFIG_SCHED_CACHE -void mm_init_sched (struct mm_struct *mm, -struct sched_cache_time __percpu *pcpu_sched); +int mm_init_sched (struct mm_struct *mm, +struct sched_cache_time __percpu *pcpu_sched); +void mm_destroy_sched (struct mm_struct *mm);static inline int mm_alloc_sched_noprof (struct mm_struct *mm) {
@@ -1635,17 +1636,11 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)if (!pcpu_sched)return -ENOMEM ; -mm_init_sched (mm, pcpu_sched); - return 0; + return mm_init_sched(mm, pcpu_sched); }#define mm_alloc_sched (...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) -static inline void mm_destroy_sched (struct mm_struct *mm) -{ -free_percpu (mm->sc_stat.pcpu_sched); - mm->sc_stat.pcpu_sched =NULL ; -}#else /* !CONFIG_SCHED_CACHE */ static inline int mm_alloc_sched (struct mm_struct *mm) {return 0 ; }
include/linux/sched.h +6 -2
@@ -2405,7 +2405,7 @@ struct sched_cache_time {unsigned long epoch; }; -struct sched_cache_stat { +struct sched_cache_group {struct sched_cache_time __percpu *pcpu_sched;raw_spinlock_t lock;unsigned long epoch;
@@ -2413,11 +2413,15 @@ struct sched_cache_stat {unsigned long next_scan;unsigned long footprint;int cpu; +refcount_t refcnt; + struct rcu_head rcu; }____cacheline_aligned_in_smp ; +void sched_cache_group_put (struct sched_cache_group *grp); +#else -struct sched_cache_stat { }; +struct sched_cache_group { };#endif
kernel/exit.c +3 -3
@@ -560,12 +560,12 @@ static void exit_mm_sched_cache(struct mm_struct *mm)return ;/* * No lock protection due to performance considerations. -* Make sure mm->sc_stat.footprint does not become +* Make sure the group footprint does not become * negative. */ - fp =READ_ONCE (mm->sc_stat.footprint); + fp =READ_ONCE (mm->sched_cache_grp->footprint); sub =min (fp, current->total_numa_faults); - WRITE_ONCE(mm->sc_stat.footprint, fp - sub); + WRITE_ONCE(mm->sched_cache_grp->footprint, fp - sub); }#else static inline void exit_mm_sched_cache (struct mm_struct *mm)
kernel/sched/build_utility.c +4 -0
@@ -89,6 +89,10 @@# include "core_sched.c" #endif +#ifdef CONFIG_SCHED_CACHE +# include "cache_sched.c" +#endif +#ifdef CONFIG_PSI # include "psi.c" #endif
kernel/sched/cache_sched.c new file +20 -0
@@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include "sched.h" + +static void sched_cache_group_free_rcu (struct rcu_head *rcu) +{ +struct sched_cache_group *grp = + container_of(rcu, struct sched_cache_group, rcu); + +/* free_percpu() may be called from atomic context. */ +free_percpu (grp->pcpu_sched); +kfree (grp); +} + +void sched_cache_group_put (struct sched_cache_group *grp) +{ +if (!grp || !refcount_dec_and_test (&grp->refcnt)) +return ; + +call_rcu (&grp->rcu, sched_cache_group_free_rcu); +}
kernel/sched/fair.c +75 -43
262 lines not on this page · raw message
Signature · 1 line
2.32.0
PATCH 2/2 sched/cache: Introduce task_struct->sched_cache_grp
In reply to Tim ChenAdd a sched_cache_grp pointer to task_struct so that scheduler code can access the cache group directly via the task, without going through mm->sched_cache_grp. This decouples the scheduler's hot-path accesses from the mm_struct.
Each task holds its own refcount on the sched_cache_group, separate from the reference held by its mm_struct. The reference is acquired in copy_mm() (fork) and exec_mmap() (exec), and released in exit_mm(). This fixes use after free problem when accessing sched_cache_grp in account_mm_sched() via mm as reported in
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Convert all scheduler code in fair.c and exit.c to use p->sched_cache_grp instead of p->mm->sched_cache_grp.
Add sched_cache_group_get() to kernel/sched/cache_sched.c.
--- fs/exec.c | 14 ++++ include/linux/sched.h | 3 + kernel/exit.c | 26 +++++-- kernel/fork.c | 23 ++++++ kernel/sched/cache_sched.c | 19 +++++ kernel/sched/fair.c | 142 +++++++++++++++++++++---------------- kernel/sched/sched.h | 3 + 7 files changed, 164 insertions(+), 66 deletions(-)
fs/exec.c +14 -0
@@ -882,6 +882,20 @@ static int exec_mmap(struct linux_binprm *bprm) active_mm = tsk->active_mm; tsk->active_mm = mm; tsk->mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +struct sched_cache_group *old_grp, *new_grp; + + old_grp =rcu_dereference_protected (tsk->sched_cache_grp, true); + +/* Acquire the reference before publishing the pointer. */ + new_grp =sched_cache_group_get (mm->sched_cache_grp); + +rcu_assign_pointer (tsk->sched_cache_grp, new_grp); +if (old_grp) +sched_cache_group_put (old_grp); + } +#endif mm_init_cid (mm, tsk); exec_state =task_exec_state_replace (tsk, exec_state);/*
include/linux/sched.h +3 -0
@@ -1434,6 +1434,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work;int preferred_llc; +struct sched_cache_group __rcu *sched_cache_grp;/* 1: task was enqueued to its preferred LLC, 0 otherwise */ int pref_llc_queued;#endif
@@ -2418,6 +2419,8 @@ struct sched_cache_group { } ____cacheline_aligned_in_smp;void sched_cache_group_put (struct sched_cache_group *grp); +struct sched_cache_group *sched_cache_group_get (struct sched_cache_group *grp); +struct sched_cache_group *task_cache_group_get (struct task_struct *p);#else
kernel/exit.c +20 -6
@@ -552,23 +552,25 @@ void mm_update_next_owner(struct mm_struct *mm)* Subtract the memory footprint of the current task from * mm. */ -static void exit_mm_sched_cache (struct mm_struct *mm) +static void exit_mm_sched_cache (void ) { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true);unsigned long fp, sub; -if (!current->total_numa_faults) +if (!grp || !current->total_numa_faults)return ;/* * No lock protection due to performance considerations. * Make sure the group footprint does not become * negative. */ - fp =READ_ONCE (mm->sched_cache_grp->footprint); + fp =READ_ONCE (grp->footprint); sub =min (fp, current->total_numa_faults); -WRITE_ONCE (mm->sched_cache_grp->footprint, fp - sub); +WRITE_ONCE (grp->footprint, fp - sub); }#else -static inline void exit_mm_sched_cache (struct mm_struct *mm) +static inline void exit_mm_sched_cache (void ) { }#endif /* CONFIG_SCHED_CACHE CONFIG_NUMA_BALANCING */
@@ -585,7 +587,19 @@ static void exit_mm(void)if (!mm)return ; -exit_mm_sched_cache (mm); +exit_mm_sched_cache (); + +#ifdef CONFIG_SCHED_CACHE + { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true); + +rcu_assign_pointer (current->sched_cache_grp,NULL ); + +if (grp) +sched_cache_group_put (grp); + } +#endif mmap_read_lock (mm);mmgrab_lazy_tlb (mm);
kernel/fork.c +23 -0
@@ -1599,6 +1599,19 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk) tsk->mm = mm; tsk->active_mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +/* +* A task holds its own reference on the group, separate from +* the reference held by its mm_struct. Acquire it before +* publishing the pointer. +*/ +struct sched_cache_group *grp = +sched_cache_group_get (mm->sched_cache_grp); + +rcu_assign_pointer (tsk->sched_cache_grp, grp); + } +#endif return 0 ; }
@@ -2599,6 +2612,16 @@ __latent_entropy struct task_struct *copy_process( bad_fork_cleanup_namespaces:exit_nsproxy_namespaces (p); bad_fork_cleanup_mm: +#ifdef CONFIG_SCHED_CACHE +/* +* copy_mm() took a task reference on the cache group; a failed fork +* never reaches exit_mm(), so release it here to avoid leaking the +* group and its per-CPU buffer. +*/ +sched_cache_group_put (rcu_dereference_protected (p->sched_cache_grp, true)); +RCU_INIT_POINTER (p->sched_cache_grp,NULL ); +#endif +if (p->mm) {mm_clear_owner (p->mm, p);mmput (p->mm);
kernel/sched/cache_sched.c +19 -0
@@ -1,6 +1,25 @@// SPDX-License-Identifier: GPL-2.0-only #include "sched.h" +struct sched_cache_group *sched_cache_group_get (struct sched_cache_group *grp) +{ +/* +* refcount_inc_not_zero() is the acquire primitive for lockless +* (RCU) lookups; plain refcount_inc() would scribble the count if +* it already reached zero. Return NULL in that case. +*/ +if (grp && !refcount_inc_not_zero (&grp->refcnt)) + grp =NULL ; + +return grp; +} + +struct sched_cache_group *task_cache_group_get (struct task_struct *p) +{ +guard (rcu)(); +return sched_cache_group_get (rcu_dereference (p->sched_cache_grp)); +} +static void sched_cache_group_free_rcu (struct rcu_head *rcu) { struct sched_cache_group *grp =
kernel/sched/fair.c +82 -60
324 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -4145,6 +4145,9 @@ static inline bool sched_cache_enabled(void)return static_branch_unlikely (&sched_cache_active); } +DEFINE_FREE(sched_cache_group_put, struct sched_cache_group *, +sched_cache_group_put (_T)); +extern void sched_cache_active_set (void );#endif
Signature · 1 line
2.32.0