Skip to contentSkip to conversation

sched/cache: Fix use after free mm access in account_mm_sched()

PATCH New Tim Chen 1 participant 3 messages Sep 2

Subsystems (5)
lore
Share & tools
lore

Tim Chen

author
#0 lore raw

cover sched/cache: Fix use after free mm access in account_mm_sched()

Hyunwoo Kim reported a KASAN use-after-free in account_mm_sched():

https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/

Cache-aware scheduling keeps its per-address-space statistics inside the mm:

        struct mm_struct {
                ...
                struct sched_cache_stat sc_stat;
        };

so their lifetime is the mm's lifetime. mm_alloc_sched() allocates sc_stat.pcpu_sched from mm_init(), and mm_destroy_sched() frees it from __mmdrop().

The scheduler, though, reaches that object from contexts that hold no reference on the mm:

- account_mm_sched() runs from update_curr() holding only the rq lock,

    and dereferences rq->curr->mm.

- the load-balance predicates - can_migrate_llc_task() -> invalid_llc_nr() / exceed_llc_capacity() - and the task_cache_work() LLC occupancy scan read p->mm of *remote* tasks.

Nothing on those paths keeps the mm alive, so an exit, or an exec_mmap() installing a new mm, can free pcpu_sched underneath a concurrent reader.

Serializing the two sides - taking the rq lock in the mm teardown path - would put a scheduler lock in the middle of __mmdrop(), which is a lot of coupling to pay for a statistics object. Give the object its own lifetime instead.

Patch 1 lifts sched_cache_stat out of mm_struct, renames it sched_cache_group, and turns it into a refcounted object freed via call_rcu(); the mm now merely points at it.

Patch 2 gives every task its own reference in task_struct->sched_cache_grp - taken in copy_mm() and exec_mmap(), dropped in exit_mm() - and converts the scheduler to read p->sched_cache_grp rather than p->mm->sc_stat. Readers access the sched_cache_grp without worry that it was freed as it has a ref count on the object.

The two patches are one fix. Patch 1 does not stand alone, so they need to be applied, and backported, as a pair.

A welcome side effect of the decoupling is that the group is no longer welded to an address space, so a later series can key it on a cgroup, a core-scheduling cookie or a numa_group instead of on a single mm.

These are the first two patches of the cache-aware prctl RFC series

https://lore.kernel.org/lkml/[email protected]/

reposted on their own with the changelogs rewritten and minor updates around the use-after-free, so that they can be considered ahead of the rest of that series. Hyunwoo confirmed the splat is gone; his Tested-by is on both patches.

Tim

---

Tim Chen (2):
sched/cache: Decouple sched_cache_group from mm sched/cache: Introduce task_struct->sched_cache_grp

 fs/exec.c                    |  14 +++
 include/linux/mm_types.h     |  15 +--
 include/linux/sched.h        |  11 +-
 kernel/exit.c                |  28 ++++--
 kernel/fork.c                |  23 +++++
 kernel/sched/build_utility.c |   4 +
 kernel/sched/cache_sched.c   |  39 ++++++++
 kernel/sched/fair.c          | 188 ++++++++++++++++++++++-------------
 kernel/sched/sched.h         |   3 +
 9 files changed, 239 insertions(+), 86 deletions(-)
 create mode 100644 kernel/sched/cache_sched.c
Signature · 1 line
2.32.0
author
#1 lore raw

PATCH 1/2 sched/cache: Decouple sched_cache_group from mm

Currently the sched cache grouping is by mm and the scheduling statistics sched_cache_stat lives in the mm structure. This ties the life cycle of scheduling stats with mm.

In account_mm_sched(), the scheduling stats are accessed by task->mm->sc_stat. However, a task may be switching mm on one CPU when another CPU is running account_mm_sched(), and possibly accessing the old mm that was freed. This problem was found when running tests with KASAN by Hyunwoo. https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/

Instead of serializing the mm access by introducing extra acquisition of rq lock in the mm free path, extract sched_cache_stat from mm_struct, rename it as sched_cache_group and manage its life cycle apart from mm_struct with its own ref counting. Access sched_cache_group directly from a task instead of having to go through a task's mm. This allows us to later add a refcount on sched_cache_group when a task links to it, preventing the use after free issue when accessing stale and released old mm and its sched cache stat a task switches to a new mm while account_mm_sched() is done elsewhere.

The other benefit of this restructure is in the future, the grouping of tasks to a LLC would have the flexibility to be associated with cgroup, cookie group, numa_group or others instead of just with a single mm address space.

Rename sched_cache_stat to sched_cache_group and turn it into a refcounted object allocated from mm_struct. The mm_struct now holds a pointer (sched_cache_grp) to this object instead of embedding it.

Introduce kernel/sched/cache_sched.c to host the cache aware scheduling helpers and define sched_cache_group_put() there.

Meanwhile skip kthreads in account_mm_sched(), consistent with task_tick_cache().

Fixes df0d98475954 ("sched/cache: Introduce infrastructure for cache-aware load balancing")
---
 include/linux/mm_types.h     |  15 ++---
 include/linux/sched.h        |   8 ++-
 kernel/exit.c                |   6 +-
 kernel/sched/build_utility.c |   4 ++
 kernel/sched/cache_sched.c   |  20 ++++++
 kernel/sched/fair.c          | 118 ++++++++++++++++++++++-------------
 6 files changed, 113 insertions(+), 58 deletions(-)
 create mode 100644 kernel/sched/cache_sched.c
include/linux/mm_types.h +5 -10
index 6d815f6440c9..f3e5a2fadbe5 100644
@@ -1226,7 +1226,7 @@ struct mm_struct {
 		struct mm_mm_cid mm_cid;
 
 		/* sched_cache related statistics */
-		struct sched_cache_stat sc_stat;
+		struct sched_cache_group *sched_cache_grp;
 #ifdef CONFIG_MMU
 		atomic_long_t pgtables_bytes;	/* size of all page tables */
 #endif
@@ -1624,8 +1624,9 @@ static inline unsigned int mm_cid_size(void)
 #endif /* CONFIG_SCHED_MM_CID */
 
 #ifdef CONFIG_SCHED_CACHE
-void mm_init_sched(struct mm_struct *mm,
-		   struct sched_cache_time __percpu *pcpu_sched);
+int mm_init_sched(struct mm_struct *mm,
+		  struct sched_cache_time __percpu *pcpu_sched);
+void mm_destroy_sched(struct mm_struct *mm);
 
 static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
 {
@@ -1635,17 +1636,11 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)
 	if (!pcpu_sched)
 		return -ENOMEM;
 
-	mm_init_sched(mm, pcpu_sched);
-	return 0;
+	return mm_init_sched(mm, pcpu_sched);
 }
 
 #define mm_alloc_sched(...)	alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__))
 
-static inline void mm_destroy_sched(struct mm_struct *mm)
-{
-	free_percpu(mm->sc_stat.pcpu_sched);
-	mm->sc_stat.pcpu_sched = NULL;
-}
 #else /* !CONFIG_SCHED_CACHE */
 
 static inline int mm_alloc_sched(struct mm_struct *mm) { return 0; }
include/linux/sched.h +6 -2
index 8b3d47a325cc..1f254364f216 100644
@@ -2405,7 +2405,7 @@ struct sched_cache_time {
 	unsigned long epoch;
 };
 
-struct sched_cache_stat {
+struct sched_cache_group {
 	struct sched_cache_time __percpu *pcpu_sched;
 	raw_spinlock_t lock;
 	unsigned long epoch;
@@ -2413,11 +2413,15 @@ struct sched_cache_stat {
 	unsigned long next_scan;
 	unsigned long footprint;
 	int cpu;
+	refcount_t refcnt;
+	struct rcu_head rcu;
 } ____cacheline_aligned_in_smp;
 
+void sched_cache_group_put(struct sched_cache_group *grp);
+
 #else
 
-struct sched_cache_stat { };
+struct sched_cache_group { };
 
 #endif
 
kernel/exit.c +3 -3
index 97686af89501..006edcc0c2c5 100644
@@ -560,12 +560,12 @@ static void exit_mm_sched_cache(struct mm_struct *mm)
 		return;
 	/*
 	 * No lock protection due to performance considerations.
-	 * Make sure mm->sc_stat.footprint does not become
+	 * Make sure the group footprint does not become
 	 * negative.
 	 */
-	fp = READ_ONCE(mm->sc_stat.footprint);
+	fp = READ_ONCE(mm->sched_cache_grp->footprint);
 	sub = min(fp, current->total_numa_faults);
-	WRITE_ONCE(mm->sc_stat.footprint, fp - sub);
+	WRITE_ONCE(mm->sched_cache_grp->footprint, fp - sub);
 }
 #else
 static inline void exit_mm_sched_cache(struct mm_struct *mm)
kernel/sched/build_utility.c +4 -0
index e2cf3b08d4e9..24202893b262 100644
@@ -89,6 +89,10 @@
 # include "core_sched.c"
 #endif
 
+#ifdef CONFIG_SCHED_CACHE
+# include "cache_sched.c"
+#endif
+
 #ifdef CONFIG_PSI
 # include "psi.c"
 #endif
kernel/sched/cache_sched.c new file +20 -0
new file mode 100644
index 000000000000..d492df55f9d5
@@ -0,0 +1,20 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include "sched.h"
+
+static void sched_cache_group_free_rcu(struct rcu_head *rcu)
+{
+	struct sched_cache_group *grp =
+		container_of(rcu, struct sched_cache_group, rcu);
+
+	/* free_percpu() may be called from atomic context. */
+	free_percpu(grp->pcpu_sched);
+	kfree(grp);
+}
+
+void sched_cache_group_put(struct sched_cache_group *grp)
+{
+	if (!grp || !refcount_dec_and_test(&grp->refcnt))
+		return;
+
+	call_rcu(&grp->rcu, sched_cache_group_free_rcu);
+}
kernel/sched/fair.c +75 -43

262 lines not on this page · raw message

Signature · 1 line
2.32.0
author
#2 lore raw

PATCH 2/2 sched/cache: Introduce task_struct->sched_cache_grp

In reply to Tim Chen

Add a sched_cache_grp pointer to task_struct so that scheduler code can access the cache group directly via the task, without going through mm->sched_cache_grp. This decouples the scheduler's hot-path accesses from the mm_struct.

Each task holds its own refcount on the sched_cache_group, separate from the reference held by its mm_struct. The reference is acquired in copy_mm() (fork) and exec_mmap() (exec), and released in exit_mm(). This fixes use after free problem when accessing sched_cache_grp in account_mm_sched() via mm as reported in
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/

Convert all scheduler code in fair.c and exit.c to use p->sched_cache_grp instead of p->mm->sched_cache_grp.

Add sched_cache_group_get() to kernel/sched/cache_sched.c.

Fixes df0d98475954 ("sched/cache: Introduce infrastructure for cache-aware load balancing")
---
 fs/exec.c                  |  14 ++++
 include/linux/sched.h      |   3 +
 kernel/exit.c              |  26 +++++--
 kernel/fork.c              |  23 ++++++
 kernel/sched/cache_sched.c |  19 +++++
 kernel/sched/fair.c        | 142 +++++++++++++++++++++----------------
 kernel/sched/sched.h       |   3 +
 7 files changed, 164 insertions(+), 66 deletions(-)
fs/exec.c +14 -0
index 745f6eb5279e..7a8a9954343e 100644
@@ -882,6 +882,20 @@ static int exec_mmap(struct linux_binprm *bprm)
 	active_mm = tsk->active_mm;
 	tsk->active_mm = mm;
 	tsk->mm = mm;
+#ifdef CONFIG_SCHED_CACHE
+	{
+		struct sched_cache_group *old_grp, *new_grp;
+
+		old_grp = rcu_dereference_protected(tsk->sched_cache_grp, true);
+
+		/* Acquire the reference before publishing the pointer. */
+		new_grp = sched_cache_group_get(mm->sched_cache_grp);
+
+		rcu_assign_pointer(tsk->sched_cache_grp, new_grp);
+		if (old_grp)
+			sched_cache_group_put(old_grp);
+	}
+#endif
 	mm_init_cid(mm, tsk);
 	exec_state = task_exec_state_replace(tsk, exec_state);
 	/*
include/linux/sched.h +3 -0
index 1f254364f216..cab8e89b1462 100644
@@ -1434,6 +1434,7 @@ struct task_struct {
 #ifdef CONFIG_SCHED_CACHE
 	struct callback_head		cache_work;
 	int				preferred_llc;
+	struct sched_cache_group __rcu	*sched_cache_grp;
 	/* 1: task was enqueued to its preferred LLC, 0 otherwise */
 	int				pref_llc_queued;
 #endif
@@ -2418,6 +2419,8 @@ struct sched_cache_group {
 } ____cacheline_aligned_in_smp;
 
 void sched_cache_group_put(struct sched_cache_group *grp);
+struct sched_cache_group *sched_cache_group_get(struct sched_cache_group *grp);
+struct sched_cache_group *task_cache_group_get(struct task_struct *p);
 
 #else
 
kernel/exit.c +20 -6
index 006edcc0c2c5..442535778ce1 100644
@@ -552,23 +552,25 @@ void mm_update_next_owner(struct mm_struct *mm)
  * Subtract the memory footprint of the current task from
  * mm.
  */
-static void exit_mm_sched_cache(struct mm_struct *mm)
+static void exit_mm_sched_cache(void)
 {
+	struct sched_cache_group *grp =
+		rcu_dereference_protected(current->sched_cache_grp, true);
 	unsigned long fp, sub;
 
-	if (!current->total_numa_faults)
+	if (!grp || !current->total_numa_faults)
 		return;
 	/*
 	 * No lock protection due to performance considerations.
 	 * Make sure the group footprint does not become
 	 * negative.
 	 */
-	fp = READ_ONCE(mm->sched_cache_grp->footprint);
+	fp = READ_ONCE(grp->footprint);
 	sub = min(fp, current->total_numa_faults);
-	WRITE_ONCE(mm->sched_cache_grp->footprint, fp - sub);
+	WRITE_ONCE(grp->footprint, fp - sub);
 }
 #else
-static inline void exit_mm_sched_cache(struct mm_struct *mm)
+static inline void exit_mm_sched_cache(void)
 {
 }
 #endif /* CONFIG_SCHED_CACHE CONFIG_NUMA_BALANCING */
@@ -585,7 +587,19 @@ static void exit_mm(void)
 	if (!mm)
 		return;
 
-	exit_mm_sched_cache(mm);
+	exit_mm_sched_cache();
+
+#ifdef CONFIG_SCHED_CACHE
+	{
+		struct sched_cache_group *grp =
+			rcu_dereference_protected(current->sched_cache_grp, true);
+
+		rcu_assign_pointer(current->sched_cache_grp, NULL);
+
+		if (grp)
+			sched_cache_group_put(grp);
+	}
+#endif
 
 	mmap_read_lock(mm);
 	mmgrab_lazy_tlb(mm);
kernel/fork.c +23 -0
index 416758c8a3d4..2e79548cb7c1 100644
@@ -1599,6 +1599,19 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk)
 
 	tsk->mm = mm;
 	tsk->active_mm = mm;
+#ifdef CONFIG_SCHED_CACHE
+	{
+		/*
+		 * A task holds its own reference on the group, separate from
+		 * the reference held by its mm_struct. Acquire it before
+		 * publishing the pointer.
+		 */
+		struct sched_cache_group *grp =
+			sched_cache_group_get(mm->sched_cache_grp);
+
+		rcu_assign_pointer(tsk->sched_cache_grp, grp);
+	}
+#endif
 	return 0;
 }
 
@@ -2599,6 +2612,16 @@ __latent_entropy struct task_struct *copy_process(
 bad_fork_cleanup_namespaces:
 	exit_nsproxy_namespaces(p);
 bad_fork_cleanup_mm:
+#ifdef CONFIG_SCHED_CACHE
+	/*
+	 * copy_mm() took a task reference on the cache group; a failed fork
+	 * never reaches exit_mm(), so release it here to avoid leaking the
+	 * group and its per-CPU buffer.
+	 */
+	sched_cache_group_put(rcu_dereference_protected(p->sched_cache_grp, true));
+	RCU_INIT_POINTER(p->sched_cache_grp, NULL);
+#endif
+
 	if (p->mm) {
 		mm_clear_owner(p->mm, p);
 		mmput(p->mm);
kernel/sched/cache_sched.c +19 -0
index d492df55f9d5..99d07e1e067c 100644
@@ -1,6 +1,25 @@
 // SPDX-License-Identifier: GPL-2.0-only
 #include "sched.h"
 
+struct sched_cache_group *sched_cache_group_get(struct sched_cache_group *grp)
+{
+	/*
+	 * refcount_inc_not_zero() is the acquire primitive for lockless
+	 * (RCU) lookups; plain refcount_inc() would scribble the count if
+	 * it already reached zero. Return NULL in that case.
+	 */
+	if (grp && !refcount_inc_not_zero(&grp->refcnt))
+		grp = NULL;
+
+	return grp;
+}
+
+struct sched_cache_group *task_cache_group_get(struct task_struct *p)
+{
+	guard(rcu)();
+	return sched_cache_group_get(rcu_dereference(p->sched_cache_grp));
+}
+
 static void sched_cache_group_free_rcu(struct rcu_head *rcu)
 {
 	struct sched_cache_group *grp =
kernel/sched/fair.c +82 -60

324 lines not on this page · raw message

kernel/sched/sched.h +3 -0
index e656c7059bf8..8b67af28a471 100644
@@ -4145,6 +4145,9 @@ static inline bool sched_cache_enabled(void)
 	return static_branch_unlikely(&sched_cache_active);
 }
 
+DEFINE_FREE(sched_cache_group_put, struct sched_cache_group *,
+	    sched_cache_group_put(_T));
+
 extern void sched_cache_active_set(void);
 
 #endif
Signature · 1 line
2.32.0