sched/cache: Fixes for cache aware scheduling
Subsystems (5)
cover sched/cache: Fixes for cache aware scheduling
Hi all,
Cache aware scheduling went in for v7.2 and people have found a few things wrong with it since. We collect the fixes in this series so it is easier to track. Two keep tasks from
being stranded outside, or yanked away from, their preferred LLC; two fix a use after free. Patches 1-2 stand alone, 3 and 4 go together.
Patch 1: alb_break_llc() compares nr_pref_llc_running with cfs.h_nr_runnable, but those count different sets - one follows queued tasks, the other drops delay-dequeued ones. With DELAY_DEQUEUE the equality stops holding and active balance pulls a task off its preferred LLC. So fix the counter. Reported by Zhan Xusheng:
https://lore.kernel.org/lkml/[email protected]/
Patch 2 (Lu Wang): the stopper doing active load balance builds a fresh lb_env that doesn't inherit migration_type, so can_migrate_task() can move a task *out* of its preferred LLC. A new LBF_ACTIVE_LB_LLC flag and picking the stopper callback at kick time keep the intent; passing migration_type through the stopper would muddy delayed dequeue. v4:
https://lore.kernel.org/lkml/[email protected]/
Patches 3-4 are for the use after free Hyunwoo Kim caught with KASAN:
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/account_mm_sched() reaches the stats via p->mm->sc_stat, but a task can be switching mm on one CPU while another is inside account_mm_sched(), so the mm and the stats inside it can go away underneath. Locking the rq in the mm free path felt like the wrong trade, so patch 4 pulls sched_cache_stat out of mm_struct into a refcounted, RCU freed sched_cache_group - just moving code - and patch 5 does the real fix:
each task takes its own reference (copy_mm(), exec_mmap(), dropped in exit_mm()), so the group outlives any mm switch. Same Fixes: tag and Hyunwoo's Tested-by on both; they want to go in together.
Nice side effect: the group no longer follows the address space, so a user defined group, or cgroup or numa_group could own it later. These are also the grouping by prctl RFC's first two patches, sent here so the fix isn't held up by that discussion.
BTW, there are two other issues in discussion currently and need a bit more work:
1. Incorrect donor context being passed to task_tick_cache().
https://lore.kernel.org/lkml/[email protected]/
It is currently under discussion and is not included in this series.
2. Cache aware scheduling interfering with ITMT.
https://lore.kernel.org/lkml/[email protected]/
https://lore.kernel.org/lkml/[email protected]/
Applies on sched/urgent branch.
Tim Chen and Chen Yu
Lu Wang (1):
sched/cache: Honor migrate_llc_task semantics in active load balance
Tim Chen (3):
sched/cache: Keep nr_pref_llc_running in the runnable domain sched/cache: Decouple sched_cache_group from mm sched/cache: Introduce task_struct->sched_cache_grp
fs/exec.c | 14 ++ include/linux/mm_types.h | 15 +- include/linux/sched.h | 11 +- kernel/exit.c | 28 +++- kernel/fork.c | 23 +++ kernel/sched/build_utility.c | 4 + kernel/sched/cache_sched.c | 39 +++++ kernel/sched/fair.c | 297 ++++++++++++++++++++++++++--------- kernel/sched/sched.h | 3 + 9 files changed, 340 insertions(+), 94 deletions(-) create mode 100644 kernel/sched/cache_sched.c
Signature · 1 line
2.32.0
PATCH 1/4 sched/cache: Keep nr_pref_llc_running in the runnable domain
alb_break_llc() decides whether to break LLC preference during active load balance. It does so by testing that every runnable fair task on the source rq prefers its LLC:
env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable
But the two counters cover different sets. nr_pref_llc_running is updated in account_llc_enqueue()/account_llc_dequeue(), next to cfs_rq->nr_queued, so it follows queued tasks. h_nr_runnable is updated in set_delayed()/ clear_delayed() and drops delay-dequeued tasks.
So under DELAY_DEQUEUE, a preferring task that goes to sleep stays counted in nr_pref_llc_running while h_nr_runnable falls. The equality then breaks, alb_break_llc() returns false, and active balance is free to pull a task off its preferred LLC. Active balance only moves runnable tasks, and this is the only LLC check it consults: once the stopper runs, LBF_ACTIVE_LB skips the per-task test in can_migrate_task(). The runnable set is the one we want.
Fix it on the counter side. A task should be counted in nr_pref_llc_running exactly while it is both queued on its preferred LLC (pref_llc_queued) and runnable (!sched_delayed). Define that membership once in task_pref_llc_runnable(), and adjust the counter only through pref_llc_running_inc()/pref_llc_running_dec() from the four sites that change either input: account_llc_enqueue(), account_llc_dequeue(), set_delayed() and clear_delayed(). Gating every update on the same predicate keeps the delay, wake and dequeue paths from double-counting or underflowing; see the comments at those sites for the ordering.
nr_llc_running and sd->llc_counts are not touched and stay on queued semantics.
--- kernel/sched/fair.c | 52 +++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-)
kernel/sched/fair.c +50 -2
@@ -1551,6 +1551,28 @@ static bool invalid_llc_nr(struct mm_struct *mm, struct task_struct *p, (scale * per_cpu(sd_llc_size, cpu))); } +/* +* A task counts in nr_pref_llc_running while it is queued on its preferred +* LLC (pref_llc_queued) and runnable (!sched_delayed), keeping the counter in +* the runnable domain so alb_break_llc() can compare it with h_nr_runnable. +*/ +static bool task_pref_llc_runnable (struct task_struct *p) +{ +return p->pref_llc_queued && !p->se.sched_delayed; +} + +static void pref_llc_running_inc (struct rq *rq,struct task_struct *p) +{ +if (task_pref_llc_runnable (p)) + rq->nr_pref_llc_running++; +} + +static void pref_llc_running_dec (struct rq *rq,struct task_struct *p) +{ +if (task_pref_llc_runnable (p)) + rq->nr_pref_llc_running--; +} +static void account_llc_enqueue (struct rq *rq,struct task_struct *p) {int pref_llc, pref_llc_queued;
@@ -1562,7 +1584,6 @@ static void account_llc_enqueue(struct rq *rq, struct task_struct *p) pref_llc_queued = (pref_llc ==task_llc (p)); rq->nr_llc_running++; - rq->nr_pref_llc_running += pref_llc_queued;/* * Record whether p is enqueued on its preferred
@@ -1580,6 +1601,9 @@ static void account_llc_enqueue(struct rq *rq, struct task_struct *p)*/ p->pref_llc_queued = pref_llc_queued; +/* Skipped while delayed; clear_delayed() adds it back on wake. */ +pref_llc_running_inc (rq, p); + sd =rcu_dereference_all (rq->sd);if (sd && (unsigned int )pref_llc < sd->llc_max) sd->llc_counts[pref_llc]++;
@@ -1596,7 +1620,12 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p) rq->nr_llc_running--;if (p->pref_llc_queued) { - rq->nr_pref_llc_running--; +/* +* Skipped if still delayed (set_delayed() already removed it); +* clearing pref_llc_queued below also stops clear_delayed() +* from re-adding it. +*/ +pref_llc_running_dec (rq, p);/* * Update the status in case * other logic might query
@@ -2021,6 +2050,10 @@ static void account_llc_enqueue(struct rq *rq, struct task_struct *p) {}static void account_llc_dequeue (struct rq *rq,struct task_struct *p) {} +static void pref_llc_running_inc (struct rq *rq,struct task_struct *p) {} + +static void pref_llc_running_dec (struct rq *rq,struct task_struct *p) {} +#endif /* CONFIG_SCHED_CACHE */ /*
@@ -6395,6 +6428,14 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);static void set_delayed (struct sched_entity *se) { +/* +* Drop a task leaving the runnable set. Must run before sched_delayed +* is set, or task_pref_llc_runnable() would already exclude it; +* clear_delayed() mirrors this after clearing the flag. +*/ +if (entity_is_task (se)) +pref_llc_running_dec (rq_of (cfs_rq_of (se)),task_of (se)); + se->sched_delayed =1 ;/*
@@ -6425,6 +6466,13 @@ static void clear_delayed(struct sched_entity *se)if (!entity_is_task (se))return ; +/* +* Re-add on wake, after sched_delayed is cleared. On a final delayed +* dequeue account_llc_dequeue() already cleared pref_llc_queued, so +* this does nothing. +*/ +pref_llc_running_inc (rq_of (cfs_rq_of (se)),task_of (se)); +for_each_sched_entity (se) {struct cfs_rq *cfs_rq =cfs_rq_of (se);
Signature · 1 line
2.32.0
Re: [PATCH 1/4] sched/cache: Keep nr_pref_llc_running in the runnable domain
Hello :>,
Quoted 29 lines
alb_break_llc()decides whether to break LLC preference during active load balance. It does so by testing that every runnable fair task on the source rq prefers its LLC:env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnableBut the two counters cover different sets.
nr_pref_llc_runningis updated in account_llc_enqueue()/account_llc_dequeue(), next tocfs_rq->nr_queued, so it follows queued tasks.h_nr_runnableis updated in set_delayed()/clear_delayed()and drops delay-dequeued tasks.So under
DELAY_DEQUEUE, a preferring task that goes to sleep stays counted innr_pref_llc_runningwhileh_nr_runnablefalls. The equality then breaks,alb_break_llc()returns false, and active balance is free to pull a task off its preferred LLC. Active balance only moves runnable tasks, and this is the only LLC check it consults: once the stopper runs,LBF_ACTIVE_LBskips the per-task test incan_migrate_task(). The runnable set is the one we want.Fix it on the counter side. A task should be counted in
nr_pref_llc_runningexactly while it is both queued on its preferred LLC (pref_llc_queued) and runnable (!sched_delayed). Define that membership once intask_pref_llc_runnable(), and adjust the counter only through pref_llc_running_inc()/pref_llc_running_dec() from the four sites that change either input:account_llc_enqueue(),account_llc_dequeue(),set_delayed()andclear_delayed(). Gating every update on the same predicate keeps the delay, wake and dequeue paths from double-counting or underflowing; see the comments at those sites for the ordering.
Quoted 2 lines
nr_llc_runningandsd->llc_countsare not touched and stay on queued semantics.
I have one question tho, can't we combine the checks with h_nr_runnable? On the paper if we are updating h_nr_runnable we could check if the nr_pref_llc_running can be updated and update it if the condition is right. Because, every nr_pref_llc_running enters h_nr_runnable while not every h_nr_runnable enters nr_pref_llc_running.
Why instead we just check the nr_pref_llc_running's conditions on task_pref_llc_runnable() and call these dec and inc functions after the h_nr_runnable updates. Wouldn't it be clear that way? If possible?
Thanks,
Kayra :_:
Re: [PATCH 1/4] sched/cache: Keep nr_pref_llc_running in the runnable domain
Quoted 34 lines · Kayra Cizmeci
Hello :>,
Quoted 29 lines
alb_break_llc()decides whether to break LLC preference during active load balance. It does so by testing that every runnable fair task on the source rq prefers its LLC:env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnableBut the two counters cover different sets.
nr_pref_llc_runningis updated in account_llc_enqueue()/account_llc_dequeue(), next tocfs_rq->nr_queued, so it follows queued tasks.h_nr_runnableis updated in set_delayed()/clear_delayed()and drops delay-dequeued tasks.So under
DELAY_DEQUEUE, a preferring task that goes to sleep stays counted innr_pref_llc_runningwhileh_nr_runnablefalls. The equality then breaks,alb_break_llc()returns false, and active balance is free to pull a task off its preferred LLC. Active balance only moves runnable tasks, and this is the only LLC check it consults: once the stopper runs,LBF_ACTIVE_LBskips the per-task test incan_migrate_task(). The runnable set is the one we want.Fix it on the counter side. A task should be counted in
nr_pref_llc_runningexactly while it is both queued on its preferred LLC (pref_llc_queued) and runnable (!sched_delayed). Define that membership once intask_pref_llc_runnable(), and adjust the counter only through pref_llc_running_inc()/pref_llc_running_dec() from the four sites that change either input:account_llc_enqueue(),account_llc_dequeue(),set_delayed()andclear_delayed(). Gating every update on the same predicate keeps the delay, wake and dequeue paths from double-counting or underflowing; see the comments at those sites for the ordering.Quoted 2 lines
nr_llc_runningandsd->llc_countsare not touched and stay on queued semantics.
Quoted 8 lines
I have one question tho, can't we combine the checks with
h_nr_runnable? On the paper if we are updatingh_nr_runnablewe could check if thenr_pref_llc_runningcan be updated and update it if the condition is right. Because, everynr_pref_llc_runningentersh_nr_runnablewhile not everyh_nr_runnableentersnr_pref_llc_running.Why instead we just check the nr_pref_llc_running's conditions on
task_pref_llc_runnable()and call these dec and inc functions after theh_nr_runnableupdates. Wouldn't it be clear that way? If possible?
Yes, nr_pref_llc_running is a subset of h_nr_runnable.
We have to keep nr_pref_llc_running accounting apart from h_nr_runnable in set_delayed(). Note that in set_delayed(), pref_llc_running_dec() has to run while the task still looks runnable, that is before se->sched_delayed = 1, because task_pref_llc_runnable()
gates on !sched_delayed. h_nr_runnable is decremented after the flag is
set:
if (entity_is_task(se))
pref_llc_running_dec(...); /* sched_delayed still 0 */
se->sched_delayed = 1;
...
for_each_sched_entity(se)
cfs_rq->h_nr_runnable--; /* sched_delayed already 1 */So moving the accounting next to (or after) the h_nr_runnable update would make task_pref_llc_runnable() return false and skip the decrement, leaving nr_pref_llc_running too high.
clear_delayed() happens to be safe either way, since it clears sched_delayed first, but keeping the two symmetric and calling inc/dec explicitly at each site is what lets the single task_pref_llc_runnable() predicate stay the one source of truth.
There is also a scope difference: h_nr_runnable is per-cfs_rq and updated at every level of the hierarchy in the for_each_sched_entity() loop, while nr_pref_llc_running is a per-rq scalar updated once per task - which is why the dec sits before the loop, not inside it.
Thanks.
Tim
Re: [PATCH 1/4] sched/cache: Keep nr_pref_llc_running in the runnable domain
Hello Tim,
Quoted 3 lines
So moving the accounting next to (or after) the
h_nr_runnableupdate would maketask_pref_llc_runnable()return false and skip the decrement, leavingnr_pref_llc_runningtoo high.
What I really wanted wasn't getting the accounting next to or after the h_nr_runnable. If we are updating h_nr_runnable in some way that means we don't need its check since it's already getting updated. And if it's getting updated that means on that branch we know how our check should behave since we are a subset of it. We can skip the delayed check on that way since we are trying to behave as h_nr_runnable's subset.
Quoted 6 lines
if (entity_is_task(se))
pref_llc_running_dec(...); /* sched_delayed still 0 */
se->sched_delayed = 1;
...
for_each_sched_entity(se)
cfs_rq->h_nr_runnable--; /* sched_delayed already 1 */For example:
In this code the h_nr_runnable is updated the same way regarding what is sched_delayed. That means if we want to behave as a subset of it, we don't need the check delayed, since we check the delayed to be a subset but if h_nr_runnable is decreasing/increasing we should look into our checks.
My head hurts. Please notify if I'm wrong.
Thanks,
Kayra
Re: [PATCH 1/4] sched/cache: Keep nr_pref_llc_running in the runnable domain
Hi Kayra,
Quoted 20 lines · Kayra Cizmeci
Hello Tim,
Quoted 3 lines
So moving the accounting next to (or after) the
h_nr_runnableupdate would maketask_pref_llc_runnable()return false and skip the decrement, leavingnr_pref_llc_runningtoo high.What I really wanted wasn't getting the accounting next to or after the
h_nr_runnable. If we are updatingh_nr_runnablein some way that means we don't need its check since it's already getting updated. And if it's getting updated that means on that branch we know how our check should behave since we are a subset of it. We can skip the delayed check on that way since we are trying to behave as h_nr_runnable's subset.Quoted 6 lines
if (entity_is_task(se)) pref_llc_running_dec(...); /* sched_delayed still 0 */ se->sched_delayed = 1; ... for_each_sched_entity(se) cfs_rq->h_nr_runnable--; /* sched_delayed already 1 */
Quoted 7 lines
For example:
In this code the
h_nr_runnableis updated the same way regarding what issched_delayed. That means if we want to behave as a subset of it, we don't need the check delayed, since we check the delayed to be a subset but ifh_nr_runnableis decreasing/increasing we should look into our checks.
I had a try according to your suggestion. It seems that the code becomes more complex and brings more headache :-( due to several corner cases. The current version is a simpler version with less code IMO. But I agree it looks a little hard to catch up with, so I added some comments around account_llc_dequeue() and adjusted the code sequence of clear_delayed() to make it easier to understand. Tim, could you please help check if this makes sense?
kernel/sched/fair.c +58 -13
@@ -1609,8 +1609,58 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p)if (p->pref_llc_queued) {/* * Skipped if still delayed (set_delayed() already removed it); -* clearing pref_llc_queued below also stops clear_delayed() -* from re-adding it. +* +* The nr_pref_llc_running varies with h_nr_runnable, it involves +* set_delayed(), clear_delayed(), account_llc_enqueued() and account_llc_dequeue(). +* The following shows two typical cases of how nr_pref_llc_running is maintained +* during enqueue/dequeue. +* +* case 1 - wakeup a delayed task +* +* CPU0 CPU1 +* __dequeue_task [fake dequeue] +* set_delayed +* rq0->nr_pref_llc_running-- +* p->se.sched_delayed = 1 +* +* try_to_wake_up(p) +* enqueue_task_fair +* requeue_delayed_entity +* clear_delayed +* rq0->nr_pref_llc_running++ +* +* +* case 2 - LB for delayed task +* +* CPU0 CPU1 +* __dequeue_task [fake dequeue] +* set_delayed +* rq0->nr_pref_llc_running-- +* p->se.sched_delayed = 1 +* +* +* --------- load balance --------- +* detach_task(p, rq0, migrate_load) +* account_llc_dequeue +* ** DO-NOT-DECREASE ** +* rq0->nr_pref_llc_running +* +* attach_task(p, rq1) +* account_llc_enqueue +* ** DO-NOT-INCREASE ** +* rq1->nr_pref_llc_running +* +* pick_eevdf +* __dequeue_task [real dequeue] +* account_llc_dequeue(DEQUEUE_DELAYED) +* ** DO-NOT-DECREASE ** +* rq1->nr_pref_llc_running +* p->pref_llc_queued = 0; +* +* clear_delayed +* p->se.sched_delayed = 0; +* ** DO-NOT-INCREASE as pref_llc_queued=0 ** +* rq1->nr_pref_llc_running */ pref_llc_running_dec (rq, p);/*
@@ -6415,23 +6465,18 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);static void set_delayed (struct sched_entity *se) { -/* -* Drop a task leaving the runnable set. Must run before sched_delayed -* is set, or task_pref_llc_runnable() would already exclude it; -* clear_delayed() mirrors this after clearing the flag. -*/ -if (entity_is_task (se)) -pref_llc_running_dec (rq_of (cfs_rq_of (se)),task_of (se)); - - se->sched_delayed =1 ; -/* * Delayed se of cfs_rq have no tasks queued on them. * Do not adjust h_nr_runnable since __dequeue_task() * will account it for blocked tasks. */ -if (!entity_is_task (se)) +if (!entity_is_task (se)) { + se->sched_delayed =1 ;return ; + } + +pref_llc_running_dec (rq_of (cfs_rq_of (se)),task_of (se)); + se->sched_delayed =1 ;for_each_sched_entity (se) {struct cfs_rq *cfs_rq =cfs_rq_of (se);
Signature · 1 line
2.43.0
Re: [PATCH 1/4] sched/cache: Keep nr_pref_llc_running in the runnable domain
Hello Chen,
Quoted 7 lines
Hi Kayra,
I had a try according to your suggestion. It seems that the code becomes more complex and brings more headache :-( due to several corner cases. The current version is a simpler version with less code IMO. But I agree it looks a little hard to catch up with, so I added some comments around
account_llc_dequeue()and adjusted the code sequence ofclear_delayed()to make it easier to understand. Tim, could you please help check if this makes sense?
I can't read your magnificent comment right now, I'm in a hurry. Sorry :-(. (But I will :>)
Quoted 20 lines
kernel/sched/fair.c +0
-11@@ -6415,23 +6465,18 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);static void set_delayed (struct sched_entity *se) { -/* -* Drop a task leaving the runnable set. Must run before sched_delayed -* is set, or task_pref_llc_runnable() would already exclude it; -* clear_delayed() mirrors this after clearing the flag. -*/ -if (entity_is_task (se)) -pref_llc_running_dec (rq_of (cfs_rq_of (se)),task_of (se)); - - se->sched_delayed =1 ; -/* * Delayed se of cfs_rq have no tasks queued on them. * Do not adjust h_nr_runnable since __dequeue_task() * will account it for blocked tasks. */ -if (!entity_is_task (se))
kernel/sched/fair.c static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq); · line 6482
Quoted 10 lines
kernel/sched/fair.c +6
-0+if (!entity_is_task (se)) { + se->sched_delayed =1 ;return ; + } + +pref_llc_running_dec (rq_of (cfs_rq_of (se)),task_of (se)); + se->sched_delayed =1 ;for_each_sched_entity (se) {struct cfs_rq *cfs_rq =cfs_rq_of (se);
Yeah, I think this is cleaner.
I didn't read the comment tho :>.
Thanks,
Kayra :--)
Re: [PATCH 1/4] sched/cache: Keep nr_pref_llc_running in the runnable domain
In reply to Chen YuQuoted 44 lines · Chen Yu
Hi Kayra,
Quoted 20 lines · Kayra Cizmeci
Hello Tim,
Quoted 3 lines
So moving the accounting next to (or after) the
h_nr_runnableupdate would maketask_pref_llc_runnable()return false and skip the decrement, leavingnr_pref_llc_runningtoo high.What I really wanted wasn't getting the accounting next to or after the
h_nr_runnable. If we are updatingh_nr_runnablein some way that means we don't need its check since it's already getting updated. And if it's getting updated that means on that branch we know how our check should behave since we are a subset of it. We can skip the delayed check on that way since we are trying to behave as h_nr_runnable's subset.Quoted 6 lines
if (entity_is_task(se)) pref_llc_running_dec(...); /* sched_delayed still 0 */ se->sched_delayed = 1; ... for_each_sched_entity(se) cfs_rq->h_nr_runnable--; /* sched_delayed already 1 */Quoted 7 lines
For example:
In this code the
h_nr_runnableis updated the same way regarding what issched_delayed. That means if we want to behave as a subset of it, we don't need the check delayed, since we check the delayed to be a subset but ifh_nr_runnableis decreasing/increasing we should look into our checks.I had a try according to your suggestion. It seems that the code becomes more complex and brings more headache :-( due to several corner cases. The current version is a simpler version with less code IMO. But I agree it looks a little hard to catch up with, so I added some comments around
account_llc_dequeue()and adjusted the code sequence ofclear_delayed()to make it easier to understand. Tim, could you please help check if this makes sense?kernel/sched/fair.c +0
-0@@ -1609,8 +1609,58 @@ static void account_llc_dequeue(struct rq *rq, struct task_struct *p)if (p->pref_llc_queued) {/*
kernel/sched/fair.c static void account_llc_dequeue(struct rq *rq, struct task_struct *p) · line 1618
Quoted 10 lines
kernel/sched/fair.c +7
-2* Skipped if still delayed (set_delayed() already removed it); -* clearing pref_llc_queued below also stops clear_delayed() -* from re-adding it. +* +* The nr_pref_llc_running varies with h_nr_runnable, it involves +* set_delayed(), clear_delayed(), account_llc_enqueued() and account_llc_dequeue(). +* The following shows two typical cases of how nr_pref_llc_running is maintained +* during enqueue/dequeue. +* +* case 1 - wakeup a delayed task
I assume both CPU0 and CPU1 in preferred LLC, and task woken on CPU1. Should say so.
kernel/sched/fair.c static void account_llc_dequeue(struct rq *rq, struct task_struct *p) · line 1630
Quoted 12 lines
kernel/sched/fair.c +12
-0+* +* CPU0 CPU1 +* __dequeue_task [fake dequeue] +* set_delayed +* rq0->nr_pref_llc_running-- +* p->se.sched_delayed = 1 +* +* try_to_wake_up(p) +* enqueue_task_fair +* requeue_delayed_entity +* clear_delayed +* rq0->nr_pref_llc_running++
rq1->nr_pref_llc_running++
kernel/sched/fair.c static void account_llc_dequeue(struct rq *rq, struct task_struct *p) · line 1633
Quoted 3 lines
kernel/sched/fair.c +3
-0+* +* +* case 2 - LB for delayed task
LB for delayed task from a preferred LLC to a non-preferred LLC.
kernel/sched/fair.c static void account_llc_dequeue(struct rq *rq, struct task_struct *p) · line 1646
Quoted 13 lines
kernel/sched/fair.c +13
-0+* +* CPU0 CPU1 +* __dequeue_task [fake dequeue] +* set_delayed +* rq0->nr_pref_llc_running-- +* p->se.sched_delayed = 1 +* +* +* --------- load balance --------- +* detach_task(p, rq0, migrate_load) +* account_llc_dequeue +* ** DO-NOT-DECREASE ** +* rq0->nr_pref_llc_running
The above block should be under CPU 0
Quoted 8 lines
kernel/sched/fair.c +8
-0+* +* attach_task(p, rq1) +* account_llc_enqueue +* ** DO-NOT-INCREASE ** +* rq1->nr_pref_llc_running +* +* pick_eevdf +* __dequeue_task [real dequeue]
kernel/sched/fair.c static void account_llc_dequeue(struct rq *rq, struct task_struct *p) · line 1664
Quoted 10 lines
kernel/sched/fair.c +9
-0+* account_llc_dequeue(DEQUEUE_DELAYED) +* ** DO-NOT-DECREASE ** +* rq1->nr_pref_llc_running +* p->pref_llc_queued = 0; +* +* clear_delayed +* p->se.sched_delayed = 0; +* ** DO-NOT-INCREASE as pref_llc_queued=0 ** +* rq1->nr_pref_llc_running */
Thanks for describing the operation scenario.
However the above cases are not comprehensive and a bit long for comment.
Quoted 19 lines
kernel/sched/fair.c +0
-10pref_llc_running_dec (rq, p);/* @@ -6415,23 +6465,18 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);static void set_delayed (struct sched_entity *se) { -/* -* Drop a task leaving the runnable set. Must run before sched_delayed -* is set, or task_pref_llc_runnable() would already exclude it; -* clear_delayed() mirrors this after clearing the flag. -*/ -if (entity_is_task (se)) -pref_llc_running_dec (rq_of (cfs_rq_of (se)),task_of (se)); - - se->sched_delayed =1 ; -/* * Delayed se of cfs_rq have no tasks queued on them. * Do not adjust h_nr_runnable since __dequeue_task()
kernel/sched/fair.c static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq); · line 6479
Quoted 10 lines
kernel/sched/fair.c +6
-1* will account it for blocked tasks. */ -if (!entity_is_task (se)) +if (!entity_is_task (se)) { + se->sched_delayed =1 ;return ; + } + +pref_llc_running_dec (rq_of (cfs_rq_of (se)),task_of (se)); + se->sched_delayed =1 ;
This change looks good and make the code cleaner.
Tim
Quoted 3 lines
kernel/sched/fair.c +0
-0for_each_sched_entity (se) {struct cfs_rq *cfs_rq =cfs_rq_of (se);
Re: [PATCH 1/4] sched/cache: Keep nr_pref_llc_running in the runnable domain
In reply to Tim ChenHello Tim,
I thought about the older messages and I think it's really not worth it. I mean, even if I'm right on 1 call site, the others won't behave the same.
And I think this version is cleaner anyways.
Only thing I could see was that on 1/4.
Feel free to add:
Now I'm going to sleep. Ah..
Peter Zijlstra
maintainerRe: [PATCH 1/4] sched/cache: Keep nr_pref_llc_running in the runnable domain
In reply to Tim ChenQuoted 29 lines · Tim Chen
alb_break_llc()decides whether to break LLC preference during active load balance. It does so by testing that every runnable fair task on the source rq prefers its LLC:env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnableBut the two counters cover different sets.
nr_pref_llc_runningis updated in account_llc_enqueue()/account_llc_dequeue(), next tocfs_rq->nr_queued, so it follows queued tasks.h_nr_runnableis updated in set_delayed()/clear_delayed()and drops delay-dequeued tasks.So under
DELAY_DEQUEUE, a preferring task that goes to sleep stays counted innr_pref_llc_runningwhileh_nr_runnablefalls. The equality then breaks,alb_break_llc()returns false, and active balance is free to pull a task off its preferred LLC. Active balance only moves runnable tasks, and this is the only LLC check it consults: once the stopper runs,LBF_ACTIVE_LBskips the per-task test incan_migrate_task(). The runnable set is the one we want.Fix it on the counter side. A task should be counted in
nr_pref_llc_runningexactly while it is both queued on its preferred LLC (pref_llc_queued) and runnable (!sched_delayed). Define that membership once intask_pref_llc_runnable(), and adjust the counter only through pref_llc_running_inc()/pref_llc_running_dec() from the four sites that change either input:account_llc_enqueue(),account_llc_dequeue(),set_delayed()andclear_delayed(). Gating every update on the same predicate keeps the delay, wake and dequeue paths from double-counting or underflowing; see the comments at those sites for the ordering.
Quoted 2 lines
nr_llc_runningandsd->llc_countsare not touched and stay on queued semantics.
Needs a fixes tag.
Quoted 11 lines
Reported-by Zhan Xusheng <[email protected]>Suggested-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>---
kernel/sched/fair.c +1
-0@@ -6395,6 +6428,14 @@ static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);static void set_delayed (struct sched_entity *se) { +/*
kernel/sched/fair.c static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq); · line 6441
Quoted 10 lines
kernel/sched/fair.c +7
-0+* Drop a task leaving the runnable set. Must run before sched_delayed +* is set, or task_pref_llc_runnable() would already exclude it; +* clear_delayed() mirrors this after clearing the flag. +*/ +if (entity_is_task (se)) +pref_llc_running_dec (rq_of (cfs_rq_of (se)),task_of (se)); + se->sched_delayed =1 ;/*
With flat pick stuff only task can get delayed. Now, I think we merged the cache aware stuff last cycle, so it we backport this (see that Fixes thing) then this must stay. But then also please make a note to take out this condition again later.
Re: [PATCH 1/4] sched/cache: Keep nr_pref_llc_running in the runnable domain
Quoted 1 line
Needs a fixes tag.
Hi Peter and Tim,
The Fixes tag should be something like:
This is the commit that adds this:env->src_rq->nr_pref_llc_running == env->src_rq->cfs.h_nr_runnable thing. It causes the issue so, you can use this, Tim.
Hope this wasn't a unnecessary message. I wanted to say this since I already found the commit and didn't want you guys to search for it again.
Thanks,
Kayra :>
PATCH 2/4 sched/cache: Honor migrate_llc_task semantics in active load balance
In reply to Tim ChenFrom: Lu Wang <[email protected]>
CAS introduced the migrate_llc_task migration type to direct tasks toward their preferred LLC, but its semantics can be lost when passive load balance falls back to active load balance. This may allow ALB to select a candidate whose preferred LLC does not match the destination, moving it away from its preferred LLC.
Example scenario:
src_rq has two runnable tasks, p1 and p2. p1 prefers dst_rq (dst_llc), while p2 prefers src_rq (src_llc). In this case, migrate_llc_task is set because src_rq has at least one task, p1, that wants to migrate to dst_rq. In ALB, can_migrate_task() finds p2 and returns true for it, thus moving p2 out of its preferred LLC.
Solution:
The CPU stopper in ALB constructs a fresh lb_env that does not inherit migration_type from the passive load-balance pass. Two approaches are possible:
(a) Add a new member to struct rq so ALB can inherit migrate_llc_task from the passive LB that triggered it.
(b) Define a new flag LBF_ACTIVE_LB_LLC and select the stopper callback at kick time to preserve the migration semantics across the asynchronous boundary.
We choose (b) because it avoids passing migration_type through the stopper, which would affect the meaning of migration_type for delayed-dequeue tasks.
--- kernel/sched/fair.c | 57 ++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 51 insertions(+), 6 deletions(-)
kernel/sched/fair.c +51 -6
@@ -10448,6 +10448,7 @@ enum migration_type {#define LBF_SOME_PINNED 0x08#define LBF_ACTIVE_LB 0x10#define LBF_LLC_PINNED 0x20 +#define LBF_ACTIVE_LB_LLC 0x40struct lb_env {struct sched_domain *sd;
@@ -10866,6 +10867,21 @@ alb_break_llc(struct lb_env *env)return false; } +/* +* Returns true if p's preferred LLC does not match the destination CPU +* under migrate_llc_task semantics. Passive LB passes migrate_llc_task +* in env->migration_type, while active LB carries LBF_ACTIVE_LB_LLC in +* env->flags to avoid overwriting env->migration_type. +*/ +static inline bool +migrate_llc_task_wrong_dst (struct task_struct *p,struct lb_env *env) +{ +return sched_cache_enabled () && + (env->migration_type == migrate_llc_task || + env->flags &LBF_ACTIVE_LB_LLC ) && +READ_ONCE (p->preferred_llc) !=llc_id (env->dst_cpu); +} +/* * Check if migrating task p from env->src_cpu to * env->dst_cpu breaks LLC localiy.
@@ -10894,8 +10910,7 @@ static bool migrate_degrades_llc(struct task_struct *p, struct lb_env *env)* run on env->dst_cpu, skip the tasks do not prefer * env->dst_cpu, and find the one that prefers. */ -if (env->migration_type == migrate_llc_task && -READ_ONCE (p->preferred_llc) !=llc_id (env->dst_cpu)) +if (migrate_llc_task_wrong_dst (p, env))return true;if (can_migrate_llc_task (env, p) != mig_forbid)
@@ -10917,6 +10932,12 @@ alb_break_llc(struct lb_env *env)return false; } +static inline bool +migrate_llc_task_wrong_dst (struct task_struct *p,struct lb_env *env) +{ +return false; +} +static inline bool migrate_degrades_llc (struct task_struct *p,struct lb_env *env) {
@@ -11016,7 +11037,7 @@ int can_migrate_task(struct task_struct *p, struct lb_env *env)* 4) too many balance attempts have failed. */ if (env->flags &LBF_ACTIVE_LB ) -return 1 ; +return !migrate_llc_task_wrong_dst (p, env); degrades =migrate_degrades_locality (p, env);if (!degrades) {
@@ -13415,6 +13436,20 @@ static int need_active_balance(struct lb_env *env) }static int active_load_balance_cpu_stop (void *data); +static int active_load_balance_llc_cpu_stop (void *data); + +/* +* migration_type is checked elsewhere to decide migration policy, so +* it shouldn't be repurposed just to flag an LLC-directed active +* balance across the stopper. Pick the callback here instead. +*/ +static inline cpu_stop_fn_t alb_stop_fn (struct lb_env *env) +{ +if (env->migration_type == migrate_llc_task) +return active_load_balance_llc_cpu_stop; + +return active_load_balance_cpu_stop; +}static int should_we_balance (struct lb_env *env) {
@@ -13760,7 +13795,7 @@ static int sched_balance_rq(int this_cpu, struct rq *this_rq, }if (active_balance) {stop_one_cpu_nowait (cpu_of (busiest), - active_load_balance_cpu_stop, busiest, +alb_stop_fn (&env), busiest, &busiest->active_balance_work); }preempt_enable ();
@@ -13865,7 +13900,7 @@ update_next_balance(struct sched_domain *sd, unsigned long *next_balance)* least 1 task to be running on each physical CPU where possible, and * avoids physical / logical imbalances. */ -static int active_load_balance_cpu_stop (void *data) +static int __active_load_balance_cpu_stop (void *data,unsigned int lb_flags) {struct rq *busiest_rq = data;int busiest_cpu =cpu_of (busiest_rq);
@@ -13915,7 +13950,7 @@ static int active_load_balance_cpu_stop(void *data) .src_cpu = busiest_rq->cpu, .src_rq = busiest_rq, .idle = CPU_IDLE, - .flags = LBF_ACTIVE_LB, + .flags = LBF_ACTIVE_LB | lb_flags, };schedstat_inc (sd->alb_count);
@@ -13943,6 +13978,16 @@ static int active_load_balance_cpu_stop(void *data)return 0 ; } +static int active_load_balance_cpu_stop (void *data) +{ +return __active_load_balance_cpu_stop (data,0 ); +} + +static int active_load_balance_llc_cpu_stop (void *data) +{ +return __active_load_balance_cpu_stop (data,LBF_ACTIVE_LB_LLC ); +} +/* * Scale the max sched_balance_rq interval with the number of CPUs in the system. * This trades load-balance latency on larger machines for less cross talk.
Signature · 1 line
2.32.0
PATCH 3/4 sched/cache: Decouple sched_cache_group from mm
In reply to Tim ChenCurrently the sched cache grouping is by mm and the scheduling statistics sched_cache_stat lives in the mm structure. This ties the life cycle of scheduling stats with mm.
In account_mm_sched(), the scheduling stats are accessed by task->mm->sc_stat. However, a task may be switching mm on one CPU when another CPU is running account_mm_sched(), and possibly accessing the old mm that was freed. This problem was found when running tests with KASAN by Hyunwoo. https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Instead of serializing the mm access by introducing extra acquisition of rq lock in the mm free path, extract sched_cache_stat from mm_struct, rename it as sched_cache_group and manage its life cycle apart from mm_struct with its own ref counting. This allows us in the next patch access sched_cache_group directly from task, and add a refcount on sched_cache_group when a task links to it. This prevents the use after free issue when accessing stale and released old mm and its sched cache stat a task switches to a new mm while account_mm_sched() is done elsewhere.
The other benefit of this restructure is in the future, the grouping of tasks to a LLC would have the flexibility to be associated with a user defined grouping, or cgroup, cookie group, numa_group or others instead of just with a single mm address space.
Rename sched_cache_stat to sched_cache_group and turn it into a refcounted object allocated from mm_struct. The mm_struct now holds a pointer (sched_cache_grp) to this object instead of embedding it.
Introduce kernel/sched/cache_sched.c to host the cache aware scheduling helpers and define sched_cache_group_put() there.
Meanwhile skip kthreads in account_mm_sched(), consistent with task_tick_cache().
--- include/linux/mm_types.h | 15 ++--- include/linux/sched.h | 8 ++- kernel/exit.c | 6 +- kernel/sched/build_utility.c | 4 ++ kernel/sched/cache_sched.c | 20 ++++++ kernel/sched/fair.c | 118 ++++++++++++++++++++++------------- 6 files changed, 113 insertions(+), 58 deletions(-) create mode 100644 kernel/sched/cache_sched.c
include/linux/mm_types.h +5 -10
@@ -1226,7 +1226,7 @@ struct mm_struct {struct mm_mm_cid mm_cid;/* sched_cache related statistics */ -struct sched_cache_stat sc_stat; +struct sched_cache_group *sched_cache_grp;#ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */ #endif
@@ -1624,8 +1624,9 @@ static inline unsigned int mm_cid_size(void)#endif /* CONFIG_SCHED_MM_CID */ #ifdef CONFIG_SCHED_CACHE -void mm_init_sched (struct mm_struct *mm, -struct sched_cache_time __percpu *pcpu_sched); +int mm_init_sched (struct mm_struct *mm, +struct sched_cache_time __percpu *pcpu_sched); +void mm_destroy_sched (struct mm_struct *mm);static inline int mm_alloc_sched_noprof (struct mm_struct *mm) {
@@ -1635,17 +1636,11 @@ static inline int mm_alloc_sched_noprof(struct mm_struct *mm)if (!pcpu_sched)return -ENOMEM ; -mm_init_sched (mm, pcpu_sched); - return 0; + return mm_init_sched(mm, pcpu_sched); }#define mm_alloc_sched (...) alloc_hooks(mm_alloc_sched_noprof(__VA_ARGS__)) -static inline void mm_destroy_sched (struct mm_struct *mm) -{ -free_percpu (mm->sc_stat.pcpu_sched); - mm->sc_stat.pcpu_sched =NULL ; -}#else /* !CONFIG_SCHED_CACHE */ static inline int mm_alloc_sched (struct mm_struct *mm) {return 0 ; }
include/linux/sched.h +6 -2
@@ -2405,7 +2405,7 @@ struct sched_cache_time {unsigned long epoch; }; -struct sched_cache_stat { +struct sched_cache_group {struct sched_cache_time __percpu *pcpu_sched;raw_spinlock_t lock;unsigned long epoch;
@@ -2413,11 +2413,15 @@ struct sched_cache_stat {unsigned long next_scan;unsigned long footprint;int cpu; +refcount_t refcnt; + struct rcu_head rcu; }____cacheline_aligned_in_smp ; +void sched_cache_group_put (struct sched_cache_group *grp); +#else -struct sched_cache_stat { }; +struct sched_cache_group { };#endif
kernel/exit.c +3 -3
@@ -560,12 +560,12 @@ static void exit_mm_sched_cache(struct mm_struct *mm)return ;/* * No lock protection due to performance considerations. -* Make sure mm->sc_stat.footprint does not become +* Make sure the group footprint does not become * negative. */ - fp =READ_ONCE (mm->sc_stat.footprint); + fp =READ_ONCE (mm->sched_cache_grp->footprint); sub =min (fp, current->total_numa_faults); - WRITE_ONCE(mm->sc_stat.footprint, fp - sub); + WRITE_ONCE(mm->sched_cache_grp->footprint, fp - sub); }#else static inline void exit_mm_sched_cache (struct mm_struct *mm)
kernel/sched/build_utility.c +4 -0
@@ -89,6 +89,10 @@# include "core_sched.c" #endif +#ifdef CONFIG_SCHED_CACHE +# include "cache_sched.c" +#endif +#ifdef CONFIG_PSI # include "psi.c" #endif
kernel/sched/cache_sched.c new file +20 -0
@@ -0,0 +1,20 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include "sched.h" + +static void sched_cache_group_free_rcu (struct rcu_head *rcu) +{ +struct sched_cache_group *grp = + container_of(rcu, struct sched_cache_group, rcu); + +/* free_percpu() may be called from atomic context. */ +free_percpu (grp->pcpu_sched); +kfree (grp); +} + +void sched_cache_group_put (struct sched_cache_group *grp) +{ +if (!grp || !refcount_dec_and_test (&grp->refcnt)) +return ; + +call_rcu (&grp->rcu, sched_cache_group_free_rcu); +}
kernel/sched/fair.c +75 -43
262 lines not on this page · raw message
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 3/4] sched/cache: Decouple sched_cache_group from mm
kernel/sched/cache_sched.c line 9
Quoted 6 lines · Tim Chen
kernel/sched/cache_sched.c new file +6
-0+static void sched_cache_group_free_rcu (struct rcu_head *rcu) +{ +struct sched_cache_group *grp = + container_of(rcu, struct sched_cache_group, rcu); + +/* free_percpu() may be called from atomic context. */
That comment is misleading at best. This is rcu-free context. And free_percpu() is not allowed from actual atomic context on RT.
Quoted 40 lines
kernel/sched/cache_sched.c new file +11
-0+free_percpu (grp->pcpu_sched); +kfree (grp); +} + +void sched_cache_group_put (struct sched_cache_group *grp) +{ +if (!grp || !refcount_dec_and_test (&grp->refcnt)) +return ; + +call_rcu (&grp->rcu, sched_cache_group_free_rcu); +}kernel/sched/fair.c +9
-8@@ -1669,18 +1677,35 @@ void mm_init_sched(struct mm_struct *mm, epoch = rq->cpu_epoch; } -raw_spin_lock_init (&mm->sc_stat.lock); - mm->sc_stat.epoch = epoch; - mm->sc_stat.cpu =-1 ; - mm->sc_stat.next_scan = jiffies; - mm->sc_stat.nr_running_avg =0 ; - mm->sc_stat.footprint =0 ; +raw_spin_lock_init (&grp->lock); + grp->epoch = epoch; + grp->cpu =-1 ; + grp->next_scan = jiffies; + grp->nr_running_avg =0 ; + grp->footprint =0 ; +refcount_set (&grp->refcnt,1 );/* -* The update to mm->sc_stat should not be reordered -* before initialization to mm's other fields, in case +* The update to grp->pcpu_sched should not be reordered +* before initialization to grp's other fields, in case * the readers may get invalid mm_sched_epoch, etc. */
kernel/sched/fair.c void mm_init_sched(struct mm_struct *mm, · line 1700
Quoted 10 lines
kernel/sched/fair.c +9
-1-smp_store_release (&mm->sc_stat.pcpu_sched, _pcpu_sched); +smp_store_release (&grp->pcpu_sched, _pcpu_sched); +/* +* Publish the group last. Not every reader qualifies it by +* grp->pcpu_sched - can_migrate_llc_task() only checks that the +* pointer is non-NULL before reading grp->footprint and +* grp->nr_running_avg - so a reachable group must already be +* fully initialized. +*/ + mm->sched_cache_grp = grp;
If it is a publish it needs to be store-release.
Quoted 21 lines
kernel/sched/fair.c +9
-1+return 0 ; +} + +void mm_destroy_sched (struct mm_struct *mm) +{ +if (mm->sched_cache_grp) +sched_cache_group_put (mm->sched_cache_grp); + mm->sched_cache_grp =NULL ; }/* because why would C be fully specified */ @@ -1738,7 +1763,7 @@ static int get_pref_llc(struct task_struct *p, struct mm_struct *mm)if (!mm)return -1 ; - mm_sched_cpu =READ_ONCE (mm->sc_stat.cpu); + mm_sched_cpu =READ_ONCE (mm->sched_cache_grp->cpu);if (mm_sched_cpu !=-1 ) { mm_sched_llc =llc_id (mm_sched_cpu);
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1811
Quoted 6 lines
kernel/sched/fair.c +3
-0@@ -1781,11 +1806,15 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)/* * init_task, kthreads and user thread created * by user_mode_thread() don't have mm. +* +* A kthread can temporarily adopt an mm via kthread_use_mm(), +* so p->mm alone does not imply a user task.
This seems like a related but distinct fix, no?
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1815
Quoted 5 lines
kernel/sched/fair.c +2
-1*/ -if (!mm || !mm->sc_stat.pcpu_sched) +if (!mm || p->flags &PF_KTHREAD || !mm->sched_cache_grp || + !mm->sched_cache_grp->pcpu_sched)return ;
Is this susceptible to TOCTOU ?
Quoted 11 lines
kernel/sched/fair.c +1
-2- pcpu_sched =per_cpu_ptr (mm->sc_stat.pcpu_sched,cpu_of (rq)); + pcpu_sched =per_cpu_ptr (mm->sched_cache_grp->pcpu_sched,cpu_of (rq)); scoped_guard (raw_spinlock, &rq->cpu_epoch_lock) {__update_mm_sched (rq, pcpu_sched);@@ -1798,11 +1827,11 @@ void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec)* If this process hasn't hit task_cache_work() for a while invalidate * its preferred state. */ - if ((long)(epoch - READ_ONCE(mm->sc_stat.epoch)) > llc_epoch_affinity_timeout ||
kernel/sched/fair.c void account_mm_sched(struct rq *rq, struct task_struct *p, s64 delta_exec) · line 1837
Quoted 10 lines
kernel/sched/fair.c +3
-2+if ((long )(epoch -READ_ONCE (mm->sched_cache_grp->epoch)) > llc_epoch_affinity_timeout ||invalid_llc_nr (mm, p,cpu_of (rq)) ||exceed_llc_capacity (mm,cpu_of (rq))) { -if (READ_ONCE (mm->sc_stat.cpu) !=-1 ) -WRITE_ONCE (mm->sc_stat.cpu,-1 ); +if (READ_ONCE (mm->sched_cache_grp->cpu) !=-1 ) +WRITE_ONCE (mm->sched_cache_grp->cpu,-1 ); } mm_sched_llc =get_pref_llc (p, mm);
Perhaps it makes sense to have a local:
struct sched_cache_group *scg = READ_ONCE(mm->sched_cache_grp);
because as is, the compiler is free to keep re-loading that. I mean, dumb, but allowed. Also the local variable will shorten some of those long expressions.
Somewhat applicable to the rest of the patch too, where it makes sense and all that.
PATCH 4/4 sched/cache: Introduce task_struct->sched_cache_grp
In reply to Tim ChenAdd a sched_cache_grp pointer to task_struct so that scheduler code can access the cache group directly via the task, without going through mm->sched_cache_grp. This decouples the scheduler's hot-path accesses from the mm_struct.
Each task holds its own refcount on the sched_cache_group, separate from the reference held by its mm_struct. The reference is acquired in copy_mm() (fork) and exec_mmap() (exec), and released in exit_mm(). This fixes use after free problem when accessing sched_cache_grp in account_mm_sched() via mm as reported in
https://lore.kernel.org/lkml/apPb-Dr4nPYuHQOK@v4bel/
Convert all scheduler code in fair.c and exit.c to use p->sched_cache_grp instead of p->mm->sched_cache_grp.
Add sched_cache_group_get() to kernel/sched/cache_sched.c.
--- fs/exec.c | 14 ++++ include/linux/sched.h | 3 + kernel/exit.c | 26 +++++-- kernel/fork.c | 23 ++++++ kernel/sched/cache_sched.c | 19 +++++ kernel/sched/fair.c | 142 +++++++++++++++++++++---------------- kernel/sched/sched.h | 3 + 7 files changed, 164 insertions(+), 66 deletions(-)
fs/exec.c +14 -0
@@ -882,6 +882,20 @@ static int exec_mmap(struct linux_binprm *bprm) active_mm = tsk->active_mm; tsk->active_mm = mm; tsk->mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +struct sched_cache_group *old_grp, *new_grp; + + old_grp =rcu_dereference_protected (tsk->sched_cache_grp, true); + +/* Acquire the reference before publishing the pointer. */ + new_grp =sched_cache_group_get (mm->sched_cache_grp); + +rcu_assign_pointer (tsk->sched_cache_grp, new_grp); +if (old_grp) +sched_cache_group_put (old_grp); + } +#endif mm_init_cid (mm, tsk); exec_state =task_exec_state_replace (tsk, exec_state);/*
include/linux/sched.h +3 -0
@@ -1434,6 +1434,7 @@ struct task_struct {#ifdef CONFIG_SCHED_CACHE struct callback_head cache_work;int preferred_llc; +struct sched_cache_group __rcu *sched_cache_grp;/* 1: task was enqueued to its preferred LLC, 0 otherwise */ int pref_llc_queued;#endif
@@ -2418,6 +2419,8 @@ struct sched_cache_group { } ____cacheline_aligned_in_smp;void sched_cache_group_put (struct sched_cache_group *grp); +struct sched_cache_group *sched_cache_group_get (struct sched_cache_group *grp); +struct sched_cache_group *task_cache_group_get (struct task_struct *p);#else
kernel/exit.c +20 -6
@@ -552,23 +552,25 @@ void mm_update_next_owner(struct mm_struct *mm)* Subtract the memory footprint of the current task from * mm. */ -static void exit_mm_sched_cache (struct mm_struct *mm) +static void exit_mm_sched_cache (void ) { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true);unsigned long fp, sub; -if (!current->total_numa_faults) +if (!grp || !current->total_numa_faults)return ;/* * No lock protection due to performance considerations. * Make sure the group footprint does not become * negative. */ - fp =READ_ONCE (mm->sched_cache_grp->footprint); + fp =READ_ONCE (grp->footprint); sub =min (fp, current->total_numa_faults); -WRITE_ONCE (mm->sched_cache_grp->footprint, fp - sub); +WRITE_ONCE (grp->footprint, fp - sub); }#else -static inline void exit_mm_sched_cache (struct mm_struct *mm) +static inline void exit_mm_sched_cache (void ) { }#endif /* CONFIG_SCHED_CACHE CONFIG_NUMA_BALANCING */
@@ -585,7 +587,19 @@ static void exit_mm(void)if (!mm)return ; -exit_mm_sched_cache (mm); +exit_mm_sched_cache (); + +#ifdef CONFIG_SCHED_CACHE + { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true); + +rcu_assign_pointer (current->sched_cache_grp,NULL ); + +if (grp) +sched_cache_group_put (grp); + } +#endif mmap_read_lock (mm);mmgrab_lazy_tlb (mm);
kernel/fork.c +23 -0
@@ -1599,6 +1599,19 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk) tsk->mm = mm; tsk->active_mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +/* +* A task holds its own reference on the group, separate from +* the reference held by its mm_struct. Acquire it before +* publishing the pointer. +*/ +struct sched_cache_group *grp = +sched_cache_group_get (mm->sched_cache_grp); + +rcu_assign_pointer (tsk->sched_cache_grp, grp); + } +#endif return 0 ; }
@@ -2599,6 +2612,16 @@ __latent_entropy struct task_struct *copy_process( bad_fork_cleanup_namespaces:exit_nsproxy_namespaces (p); bad_fork_cleanup_mm: +#ifdef CONFIG_SCHED_CACHE +/* +* copy_mm() took a task reference on the cache group; a failed fork +* never reaches exit_mm(), so release it here to avoid leaking the +* group and its per-CPU buffer. +*/ +sched_cache_group_put (rcu_dereference_protected (p->sched_cache_grp, true)); +RCU_INIT_POINTER (p->sched_cache_grp,NULL ); +#endif +if (p->mm) {mm_clear_owner (p->mm, p);mmput (p->mm);
kernel/sched/cache_sched.c +19 -0
@@ -1,6 +1,25 @@// SPDX-License-Identifier: GPL-2.0-only #include "sched.h" +struct sched_cache_group *sched_cache_group_get (struct sched_cache_group *grp) +{ +/* +* refcount_inc_not_zero() is the acquire primitive for lockless +* (RCU) lookups; plain refcount_inc() would scribble the count if +* it already reached zero. Return NULL in that case. +*/ +if (grp && !refcount_inc_not_zero (&grp->refcnt)) + grp =NULL ; + +return grp; +} + +struct sched_cache_group *task_cache_group_get (struct task_struct *p) +{ +guard (rcu)(); +return sched_cache_group_get (rcu_dereference (p->sched_cache_grp)); +} +static void sched_cache_group_free_rcu (struct rcu_head *rcu) { struct sched_cache_group *grp =
kernel/sched/fair.c +82 -60
327 lines not on this page · raw message
kernel/sched/sched.h +3 -0
@@ -4145,6 +4145,9 @@ static inline bool sched_cache_enabled(void)return static_branch_unlikely (&sched_cache_active); } +DEFINE_FREE(sched_cache_group_put, struct sched_cache_group *, +sched_cache_group_put (_T)); +extern void sched_cache_active_set (void );#endif
Signature · 1 line
2.32.0
Peter Zijlstra
maintainerRe: [PATCH 4/4] sched/cache: Introduce task_struct->sched_cache_grp
Quoted 3 lines · Tim Chen
Co-developed-by Chen Yu <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>
:-(
Quoted 22 lines
--- fs/exec.c | 14 ++++ include/linux/sched.h | 3 + kernel/exit.c | 26 +++++-- kernel/fork.c | 23 ++++++ kernel/sched/cache_sched.c | 19 +++++ kernel/sched/fair.c | 142 +++++++++++++++++++++---------------- kernel/sched/sched.h | 3 + 7 files changed, 164 insertions(+), 66 deletions(-)fs/exec.c +4
-0@@ -882,6 +882,20 @@ static int exec_mmap(struct linux_binprm *bprm) active_mm = tsk->active_mm; tsk->active_mm = mm; tsk->mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +struct sched_cache_group *old_grp, *new_grp; +
fs/exec.c static int exec_mmap(struct linux_binprm *bprm) · line 898
Quoted 10 lines
fs/exec.c +10
-0+ old_grp =rcu_dereference_protected (tsk->sched_cache_grp, true); + +/* Acquire the reference before publishing the pointer. */ + new_grp =sched_cache_group_get (mm->sched_cache_grp); + +rcu_assign_pointer (tsk->sched_cache_grp, new_grp); +if (old_grp) + sched_cache_group_put(old_grp); + } +#endif
Guys no! This is horrific crap. This is not how we do things and I would have expected you all to know this.
Have you heard of this new fangled thing called a function?
Imagine all of those being just:
sched_cache_exec_mmap(tsk, mm);
Also: rcu_dereference_protected(.c = true) is another offence, that's just wrong.
Quoted 43 lines
kernel/exit.c +10
-6@@ -552,23 +552,25 @@ void mm_update_next_owner(struct mm_struct *mm)* Subtract the memory footprint of the current task from * mm. */ -static void exit_mm_sched_cache (struct mm_struct *mm) +static void exit_mm_sched_cache (void ) { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true);unsigned long fp, sub; -if (!current->total_numa_faults) +if (!grp || !current->total_numa_faults)return ;/* * No lock protection due to performance considerations. * Make sure the group footprint does not become * negative. */ - fp =READ_ONCE (mm->sched_cache_grp->footprint); + fp =READ_ONCE (grp->footprint); sub =min (fp, current->total_numa_faults); -WRITE_ONCE (mm->sched_cache_grp->footprint, fp - sub); +WRITE_ONCE (grp->footprint, fp - sub); }#else -static inline void exit_mm_sched_cache (struct mm_struct *mm) +static inline void exit_mm_sched_cache (void ) { }#endif /* CONFIG_SCHED_CACHE CONFIG_NUMA_BALANCING */ @@ -585,7 +587,19 @@ static void exit_mm(void)if (!mm)return ; -exit_mm_sched_cache (mm); +exit_mm_sched_cache (); + +#ifdef CONFIG_SCHED_CACHE
kernel/exit.c static void exit_mm(void) · line 602
Quoted 10 lines
kernel/exit.c +10
-0+ { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true); + +rcu_assign_pointer (current->sched_cache_grp,NULL ); + +if (grp) +sched_cache_group_put (grp); + } +#endif
Seriously, WTF ?!
Quoted 14 lines
kernel/exit.c +0
-0mmap_read_lock (mm);mmgrab_lazy_tlb (mm);kernel/fork.c +3
-0@@ -1599,6 +1599,19 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk) tsk->mm = mm; tsk->active_mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +/*
kernel/fork.c static int copy_mm(u64 clone_flags, struct task_struct *tsk) · line 1614
Quoted 10 lines
kernel/fork.c +10
-0+* A task holds its own reference on the group, separate from +* the reference held by its mm_struct. Acquire it before +* publishing the pointer. +*/ +struct sched_cache_group *grp = +sched_cache_group_get (mm->sched_cache_grp); + + rcu_assign_pointer(tsk->sched_cache_grp, grp); + } +#endif
And again.
Quoted 10 lines
kernel/fork.c +3
-0return 0 ; }@@ -2599,6 +2612,16 @@ __latent_entropy struct task_struct *copy_process( bad_fork_cleanup_namespaces:exit_nsproxy_namespaces (p); bad_fork_cleanup_mm: +#ifdef CONFIG_SCHED_CACHE +/* +* copy_mm() took a task reference on the cache group; a failed fork
kernel/fork.c __latent_entropy struct task_struct *copy_process( · line 2627
Quoted 10 lines
kernel/fork.c +7
-0+* never reaches exit_mm(), so release it here to avoid leaking the +* group and its per-CPU buffer. +*/ +sched_cache_group_put (rcu_dereference_protected (p->sched_cache_grp, true)); +RCU_INIT_POINTER (p->sched_cache_grp,NULL ); +#endif +if (p->mm) {mm_clear_owner (p->mm, p);mmput (p->mm);
Drugs, it must be drugs and lots of it :-(
Please, try again.
Re: [PATCH 4/4] sched/cache: Introduce task_struct->sched_cache_grp
Quoted 152 lines · Peter Zijlstra
Quoted 3 lines · Tim Chen
Co-developed-by Chen Yu <[email protected]>Signed-off-by Chen Yu <[email protected]>Signed-off-by Tim Chen <[email protected]>:-(
Quoted 22 lines
--- fs/exec.c | 14 ++++ include/linux/sched.h | 3 + kernel/exit.c | 26 +++++-- kernel/fork.c | 23 ++++++ kernel/sched/cache_sched.c | 19 +++++ kernel/sched/fair.c | 142 +++++++++++++++++++++---------------- kernel/sched/sched.h | 3 + 7 files changed, 164 insertions(+), 66 deletions(-)fs/exec.c +4
-0@@ -882,6 +882,20 @@ static int exec_mmap(struct linux_binprm *bprm) active_mm = tsk->active_mm; tsk->active_mm = mm; tsk->mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +struct sched_cache_group *old_grp, *new_grp; +fs/exec.c static int exec_mmap(struct linux_binprm *bprm) · line 898
Quoted 10 lines
fs/exec.c +10
-0+ old_grp =rcu_dereference_protected (tsk->sched_cache_grp, true); + +/* Acquire the reference before publishing the pointer. */ + new_grp =sched_cache_group_get (mm->sched_cache_grp); + +rcu_assign_pointer (tsk->sched_cache_grp, new_grp); +if (old_grp) + sched_cache_group_put(old_grp); + } +#endif Guys no! This is horrific crap. This is not how we do things and I would have expected you all to know this.
Have you heard of this new fangled thing called a function?
Imagine all of those being just:
sched_cache_exec_mmap(tsk, mm);Also:
rcu_dereference_protected(.c = true)is another offence, that's just wrong.Quoted 43 lines
kernel/exit.c +10
-6@@ -552,23 +552,25 @@ void mm_update_next_owner(struct mm_struct *mm)* Subtract the memory footprint of the current task from * mm. */ -static void exit_mm_sched_cache (struct mm_struct *mm) +static void exit_mm_sched_cache (void ) { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true);unsigned long fp, sub; -if (!current->total_numa_faults) +if (!grp || !current->total_numa_faults)return ;/* * No lock protection due to performance considerations. * Make sure the group footprint does not become * negative. */ - fp =READ_ONCE (mm->sched_cache_grp->footprint); + fp =READ_ONCE (grp->footprint); sub =min (fp, current->total_numa_faults); -WRITE_ONCE (mm->sched_cache_grp->footprint, fp - sub); +WRITE_ONCE (grp->footprint, fp - sub); }#else -static inline void exit_mm_sched_cache (struct mm_struct *mm) +static inline void exit_mm_sched_cache (void ) { }#endif /* CONFIG_SCHED_CACHE CONFIG_NUMA_BALANCING */ @@ -585,7 +587,19 @@ static void exit_mm(void)if (!mm)return ; -exit_mm_sched_cache (mm); +exit_mm_sched_cache (); + +#ifdef CONFIG_SCHED_CACHE kernel/exit.c static void exit_mm(void) · line 602
Quoted 10 lines
kernel/exit.c +10
-0+ { +struct sched_cache_group *grp = +rcu_dereference_protected (current->sched_cache_grp, true); + +rcu_assign_pointer (current->sched_cache_grp,NULL ); + +if (grp) +sched_cache_group_put (grp); + } +#endif Seriously, WTF ?!
Quoted 14 lines
kernel/exit.c +0
-0mmap_read_lock (mm);mmgrab_lazy_tlb (mm);kernel/fork.c +3
-0@@ -1599,6 +1599,19 @@ static int copy_mm(u64 clone_flags, struct task_struct *tsk) tsk->mm = mm; tsk->active_mm = mm; +#ifdef CONFIG_SCHED_CACHE + { +/* kernel/fork.c static int copy_mm(u64 clone_flags, struct task_struct *tsk) · line 1614
Quoted 10 lines
kernel/fork.c +10
-0+* A task holds its own reference on the group, separate from +* the reference held by its mm_struct. Acquire it before +* publishing the pointer. +*/ +struct sched_cache_group *grp = +sched_cache_group_get (mm->sched_cache_grp); + + rcu_assign_pointer(tsk->sched_cache_grp, grp); + } +#endif And again.
Quoted 14 lines
kernel/fork.c +7
-0return 0 ; }@@ -2599,6 +2612,16 @@ __latent_entropy struct task_struct *copy_process( bad_fork_cleanup_namespaces:exit_nsproxy_namespaces (p); bad_fork_cleanup_mm: +#ifdef CONFIG_SCHED_CACHE +/* +* copy_mm() took a task reference on the cache group; a failed fork +* never reaches exit_mm(), so release it here to avoid leaking the +* group and its per-CPU buffer. +*/ +sched_cache_group_put (rcu_dereference_protected (p->sched_cache_grp, true));
Quoted 10 lines
kernel/fork.c __latent_entropy struct task_struct *copy_process( · line 2627
Quoted 6 lines
kernel/fork.c +3
-0+RCU_INIT_POINTER (p->sched_cache_grp,NULL ); +#endif +if (p->mm) {mm_clear_owner (p->mm, p);mmput (p->mm);Drugs, it must be drugs and lots of it :-(
Sorry for the warts in this version. Will clean it up and send an update.
Tim
Re: [PATCH 4/4] sched/cache: Introduce task_struct->sched_cache_grp
Quoted 9 lines · Tim Chen
Quoted 8 lines · Peter Zijlstra
fs/exec.c static int exec_mmap(struct linux_binprm *bprm) · line 884
Quoted 4 lines · Tim Chen
fs/exec.c +3
-0@@ -882,6 +882,20 @@ static int exec_mmap(struct linux_binprm *bprm) + + old_grp =rcu_dereference_protected (tsk->sched_cache_grp, true); +Also:
rcu_dereference_protected(.c = true)is another offence, that's just wrong.
Got it, will fix it.
If I understand correctly, the "c" should never pass an explicit const value? The original idea was that, in exec_mmap(), the current running task is the only writer to change tsk->sched_cache_grp, so the condition is true,
To fix it, it might be turned into the following for now:rcu_dereference_protected(tsk->sched_cache_grp, tsk == current); like deref_curr_numa_group().
In the future if task tagging is supported, the race condition of multiple updates would require a pi_lock for this task, and it will become:rcu_dereference_protected(tsk->sched_cache_grp, lockdep_is_held(&p->pi_lock)); like what we have done in the prctl based cache aware scheduling patch set:
https://lore.kernel.org/lkml/50fe2db1a62ea2376a87d0c14778b1ff456d11ec.1787955777.git.tim.c.chen@linux.intel.com/
thanks,
Chenyu
Peter Zijlstra
maintainerRe: [PATCH 4/4] sched/cache: Introduce task_struct->sched_cache_grp
Quoted 11 lines · Chen Yu
Quoted 9 lines · Tim Chen
Quoted 8 lines · Peter Zijlstra
fs/exec.c static int exec_mmap(struct linux_binprm *bprm) · line 884
Quoted 4 lines · Tim Chen
fs/exec.c +3
-0@@ -882,6 +882,20 @@ static int exec_mmap(struct linux_binprm *bprm) + + old_grp =rcu_dereference_protected (tsk->sched_cache_grp, true); +Also:
rcu_dereference_protected(.c = true)is another offence, that's just wrong.
Quoted 2 lines
Got it, will fix it.
If I understand correctly, the "c" should never pass an explicit const value?
Right, that is a bit of an anti-pattern. The 'c' thing is there to validate your claim of this in fact being an update side, eg. you holding the write lock and all that.
Quoted 7 lines
The original idea was that, in
exec_mmap(), the current running task is the only writer to changetsk->sched_cache_grp, so the condition is true,To fix it, it might be turned into the following for now:
rcu_dereference_protected(tsk->sched_cache_grp, tsk == current); likederef_curr_numa_group().
Quoted 5 lines
In the future if task tagging is supported, the race condition of multiple updates would require a
pi_lockfor this task, and it will become:rcu_dereference_protected(tsk->sched_cache_grp, lockdep_is_held(&p->pi_lock)); like what we have done in the prctl based cache aware scheduling patch set:
https://lore.kernel.org/lkml/50fe2db1a62ea2376a87d0c14778b1ff456d11ec.1787955777.git.tim.c.chen@linux.intel.com/
Right, it might also make sense to have a helper, like:
#define rcu_deref_sched_cache_grp(tsk) \ rcu_dereference_protected((tsk)->sched_cache_grp, (tsk) == current)
such that you don't have to keep repeating this thing over and over. Also consider if rcu_dereference_protected() makes sense, I tend to unconditionally use rcu_dereference_*check().
The only architecture where it really makes a difference is Alpha, and nobody cares about that anymore.