sched: Run task_mm_cid_work in batches to lower latency
Subsystems (7)
v2 cover sched: Run task_mm_cid_work in batches to lower latency
This V2 of [1] is a continuation of [2] but using a simpler approach. The task_mm_cid_work runs as a task_work returning to userspace and causes a non-negligible scheduling latency, mostly due to its iterations over all cores.
Split the work into several batches, each call to task_mm_cid_work will not run for all cpus but just for a configurable number of cpus. Next runs will pick up where the previous left off.
The mechanism that avoids running too frequently (100ms) is enforced only when finishing all cpus, that is when starting from 0.
Also improve the predictability of the scan on short running tasks by running it from rseq_handle_notify_resume, which runs it on every task switch (similar behaviour to [2]), the same workaround on the tick for long running tasks seen in [2] was ported also here.
Patch 1 add support for prev_sum_exec_runtime to the RT, deadline and sched_ext classes as it is supported by fair, this is required to avoid calling rseq_preempt on tick if the runtime is below a threshold.
Patch 2 moves the directly calls task_mm_cid_work instead of relying on a task_work, necessary to avoid rseq_handle_notify_resume being called twice while enqueuing a task_work.
Patch 3 splits the work into batches.
Patch 4 adds a selftest to validate the functionality of the task_mm_cid_work (i.e. to compact the mm_cids).
Changes since V1 [1]:
* Use cpu_possible_mask in scan.
* Make sure batches have the same number of CPUs also if mask is sparse.
* Run the task on rseq_handle_notify_resume as in [2] but call directly.
* Schedule the work and mm_cid update on tick for long running tasks.
* Fix condition for need_scan only on first batch.
* Change RSEQ_CID_SCAN_BATCH default to be a power of 2.
* Rebase selftest on [2].
* Increase the selftest timeout on large systems.
To: Mathieu Desnoyers <[email protected]>
To: Peter Zijlstra <[email protected]>
To: Ingo Molnar <[email protected]>
[1] - https://lore.kernel.org/lkml/[email protected]
[2] - https://lore.kernel.org/lkml/[email protected]
Gabriele Monaco (4):
sched: Add prev_sum_exec_runtime support for RT, DL and SCX classes
rseq: Run the mm_cid_compaction from rseq_handle_notify_resume()
sched: Compact RSEQ concurrency IDs in batches selftests/rseq: Add test for mm_cid compaction
include/linux/mm.h | 2 + include/linux/mm_types.h | 26 +++ include/linux/sched.h | 2 +- init/Kconfig | 12 ++ kernel/rseq.c | 2 + kernel/sched/core.c | 92 ++++++-- kernel/sched/deadline.c | 1 + kernel/sched/ext.c | 1 + kernel/sched/rt.c | 1 + kernel/sched/sched.h | 2 + tools/testing/selftests/rseq/.gitignore | 1 + tools/testing/selftests/rseq/Makefile | 2 +- .../selftests/rseq/mm_cid_compaction_test.c | 204 ++++++++++++++++++ 13 files changed, 323 insertions(+), 25 deletions(-) create mode 100644 tools/testing/selftests/rseq/mm_cid_compaction_test.c
base-commit: 155a3c003e555a7300d156a5252c004c392ec6b0
Signature · 1 line
2.50.1
PATCH v2 1/4 sched: Add prev_sum_exec_runtime support for RT, DL and SCX classes
The fair scheduling class relies on prev_sum_exec_runtime to compute the duration of the task's runtime since it was last scheduled. This value is currently not required by other scheduling classes but can be useful to understand long running tasks and take certain actions (e.g. during a scheduler tick).
Add support for prev_sum_exec_runtime to the RT, deadline and sched_ext classes by simply assigning the sum_exec_runtime at each set_next_task.
--- kernel/sched/deadline.c | 1 + kernel/sched/ext.c | 1 + kernel/sched/rt.c | 1 + 3 files changed, 3 insertions(+)
kernel/sched/deadline.c +1 -0
@@ -2389,6 +2389,7 @@ static void set_next_task_dl(struct rq *rq, struct task_struct *p, bool first) p->se.exec_start =rq_clock_task (rq);if (on_dl_rq (&p->dl))update_stats_wait_end_dl (dl_rq, dl_se); + p->se.prev_sum_exec_runtime = p->se.sum_exec_runtime;/* You can't push away the running task */ dequeue_pushable_dl_task (rq, p);
kernel/sched/ext.c +1 -0
@@ -3255,6 +3255,7 @@ static void set_next_task_scx(struct rq *rq, struct task_struct *p, bool first) } p->se.exec_start =rq_clock_task (rq); + p->se.prev_sum_exec_runtime = p->se.sum_exec_runtime;/* see dequeue_task_scx() on why we skip when !QUEUED */ if (SCX_HAS_OP (sch, running) && (p->scx.flags &SCX_TASK_QUEUED ))
kernel/sched/rt.c +1 -0
@@ -1693,6 +1693,7 @@ static inline void set_next_task_rt(struct rq *rq, struct task_struct *p, bool f p->se.exec_start =rq_clock_task (rq);if (on_rt_rq (&p->rt))update_stats_wait_end_rt (rt_rq, rt_se); + p->se.prev_sum_exec_runtime = p->se.sum_exec_runtime;/* The running task is never eligible for pushing */ dequeue_pushable_task (rq, p);
Signature · 1 line
2.50.1
PATCH v2 2/4 rseq: Run the mm_cid_compaction from rseq_handle_notify_resume()
In reply to Gabriele MonacoCurrently the mm_cid_compaction is triggered by the scheduler tick and runs in a task_work, behaviour is more unpredictable with periodic tasks with short runtime, which may rarely run during a tick.
Run the mm_cid_compaction from the rseq_handle_notify_resume() call, which runs from resume_user_mode_work. Since the context is the same where the task_work would run, skip this step and call the compaction function directly.
The compaction function still exits prematurely in case the scan is not required, that is when the pseudo-period of 100ms did not elapse.
Keep a tick handler used for long running tasks that are never preempted (i.e. that never call rseq_handle_notify_resume), which triggers a compaction and mm_cid update only in that case.
--- include/linux/mm.h | 2 ++ include/linux/mm_types.h | 11 ++++++++ include/linux/sched.h | 2 +- kernel/rseq.c | 2 ++ kernel/sched/core.c | 55 +++++++++++++++++++++++++--------------- kernel/sched/sched.h | 2 ++ 6 files changed, 53 insertions(+), 21 deletions(-)
include/linux/mm.h +2 -0
@@ -2294,6 +2294,7 @@ void sched_mm_cid_before_execve(struct task_struct *t);void sched_mm_cid_after_execve (struct task_struct *t);void sched_mm_cid_fork (struct task_struct *t);void sched_mm_cid_exit_signals (struct task_struct *t); +void task_mm_cid_work (struct task_struct *t);static inline int task_mm_cid (struct task_struct *t) {return t->mm_cid;
@@ -2303,6 +2304,7 @@ static inline void sched_mm_cid_before_execve(struct task_struct *t) { }static inline void sched_mm_cid_after_execve (struct task_struct *t) { }static inline void sched_mm_cid_fork (struct task_struct *t) { }static inline void sched_mm_cid_exit_signals (struct task_struct *t) { } +static inline void task_mm_cid_work (struct task_struct *t) { }static inline int task_mm_cid (struct task_struct *t) {/*
include/linux/mm_types.h +11 -0
@@ -1420,6 +1420,13 @@ static inline void mm_set_cpus_allowed(struct mm_struct *mm, const struct cpumasWRITE_ONCE (mm->nr_cpus_allowed,cpumask_weight (mm_allowed)); raw_spin_unlock(&mm->cpus_allowed_lock); } + +static inline bool mm_cid_needs_scan (struct mm_struct *mm) +{ +if (!mm) +return false; +return time_after (jiffies,READ_ONCE (mm->mm_cid_next_scan)); +}#else /* CONFIG_SCHED_MM_CID */ static inline void mm_init_cid (struct mm_struct *mm,struct task_struct *p) { }static inline int mm_alloc_cid (struct mm_struct *mm,struct task_struct *p) {return 0 ; }
@@ -1430,6 +1437,10 @@ static inline unsigned int mm_cid_size(void) return 0; }static inline void mm_set_cpus_allowed (struct mm_struct *mm,const struct cpumask *cpumask) { } +static inline bool mm_cid_needs_scan (struct mm_struct *mm) +{ +return false; +}#endif /* CONFIG_SCHED_MM_CID */ struct mmu_gather ;
include/linux/sched.h +1 -1
@@ -1428,7 +1428,7 @@ struct task_struct {int last_mm_cid;/* Most recent cid in mm */ int migrate_from_cpu;int mm_cid_active;/* Whether cid bitmap is active */ -struct callback_head cid_work; +unsigned long last_cid_reset;/* Time of last reset in jiffies */ #endif struct tlbflush_unmap_batch tlb_ubc;
kernel/rseq.c +2 -0
@@ -441,6 +441,8 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs) }if (unlikely (rseq_update_cpu_node_id (t))) goto error; +/* The mm_cid compaction returns prematurely if scan is not needed. */ +task_mm_cid_work (t);return ; error:
kernel/sched/core.c +35 -20
80 lines not on this page · raw message
kernel/sched/sched.h +2 -0
@@ -3606,6 +3606,7 @@ extern const char *preempt_modes[];#define SCHED_MM_CID_PERIOD_NS (100ULL * 1000000)/* 100ms */ #define MM_CID_SCAN_DELAY 100/* 100ms */ +#define RSEQ_UNPREEMPTED_THRESHOLD SCHED_MM_CID_PERIOD_NSextern raw_spinlock_t cid_lock;extern int use_cid_lock;
@@ -3809,6 +3810,7 @@ static inline int mm_cid_get(struct rq *rq, struct task_struct *t,int cid;lockdep_assert_rq_held (rq); + t->last_cid_reset = jiffies; cpumask =mm_cidmask (mm); cid =__this_cpu_read (pcpu_cid->cid);if (mm_cid_is_valid (cid)) {
Signature · 1 line
2.50.1
Mathieu Desnoyers
maintainerRe: [PATCH v2 2/4] rseq: Run the mm_cid_compaction from rseq_handle_notify_resume()
Quoted 11 lines · Gabriele Monaco
Currently the
mm_cid_compactionis triggered by the scheduler tick and runs in atask_work, behaviour is more unpredictable with periodic tasks with short runtime, which may rarely run during a tick.Run the
mm_cid_compactionfrom therseq_handle_notify_resume()call, which runs fromresume_user_mode_work. Since the context is the same where thetask_workwould run, skip this step and call the compaction function directly.
The compaction function still exits prematurely in case the scan is not required, that is when the pseudo-period of 100ms did not elapse.
Quoted 3 lines
Keep a tick handler used for long running tasks that are never preempted (i.e. that never call
rseq_handle_notify_resume), which triggers a compaction andmm_cidupdate only in that case.
Your approach looks good, but please note that this will probably need to be rebased on top of the rseq rework from Thomas Gleixner.
Latest version can be found here:
https://lore.kernel.org/lkml/[email protected]/
Thanks,
Mathieu
Quoted 192 lines
Signed-off-by Gabriele Monaco <[email protected]>--- include/linux/mm.h | 2 ++ include/linux/mm_types.h | 11 ++++++++ include/linux/sched.h | 2 +- kernel/rseq.c | 2 ++ kernel/sched/core.c | 55 +++++++++++++++++++++++++--------------- kernel/sched/sched.h | 2 ++ 6 files changed, 53 insertions(+), 21 deletions(-)include/linux/mm.h +2
-0@@ -2294,6 +2294,7 @@ void sched_mm_cid_before_execve(struct task_struct *t);void sched_mm_cid_after_execve (struct task_struct *t);void sched_mm_cid_fork (struct task_struct *t);void sched_mm_cid_exit_signals (struct task_struct *t); +void task_mm_cid_work (struct task_struct *t);static inline int task_mm_cid (struct task_struct *t) {return t->mm_cid;@@ -2303,6 +2304,7 @@ static inline void sched_mm_cid_before_execve(struct task_struct *t) { }static inline void sched_mm_cid_after_execve (struct task_struct *t) { }static inline void sched_mm_cid_fork (struct task_struct *t) { }static inline void sched_mm_cid_exit_signals (struct task_struct *t) { } +static inline void task_mm_cid_work (struct task_struct *t) { }static inline int task_mm_cid (struct task_struct *t) {/* include/linux/mm_types.h +11
-0@@ -1420,6 +1420,13 @@ static inline void mm_set_cpus_allowed(struct mm_struct *mm, const struct cpumasWRITE_ONCE (mm->nr_cpus_allowed,cpumask_weight (mm_allowed)); raw_spin_unlock(&mm->cpus_allowed_lock); } + +static inline bool mm_cid_needs_scan (struct mm_struct *mm) +{ +if (!mm) +return false; +return time_after (jiffies,READ_ONCE (mm->mm_cid_next_scan)); +}#else /* CONFIG_SCHED_MM_CID */ static inline void mm_init_cid (struct mm_struct *mm,struct task_struct *p) { }static inline int mm_alloc_cid (struct mm_struct *mm,struct task_struct *p) {return 0 ; }@@ -1430,6 +1437,10 @@ static inline unsigned int mm_cid_size(void) return 0; }static inline void mm_set_cpus_allowed (struct mm_struct *mm,const struct cpumask *cpumask) { } +static inline bool mm_cid_needs_scan (struct mm_struct *mm) +{ +return false; +}#endif /* CONFIG_SCHED_MM_CID */ struct mmu_gather ;include/linux/sched.h +1
-1@@ -1428,7 +1428,7 @@ struct task_struct {int last_mm_cid;/* Most recent cid in mm */ int migrate_from_cpu;int mm_cid_active;/* Whether cid bitmap is active */ -struct callback_head cid_work; +unsigned long last_cid_reset;/* Time of last reset in jiffies */ #endif struct tlbflush_unmap_batch tlb_ubc;kernel/rseq.c +2
-0@@ -441,6 +441,8 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs) }if (unlikely (rseq_update_cpu_node_id (t))) goto error; +/* The mm_cid compaction returns prematurely if scan is not needed. */ +task_mm_cid_work (t);return ; error:kernel/sched/core.c +35
-20@@ -10589,22 +10589,13 @@ static void sched_mm_cid_remote_clear_weight(struct mm_struct *mm, int cpu,sched_mm_cid_remote_clear (mm, pcpu_cid, cpu); } -static void task_mm_cid_work (struct callback_head *work) +void task_mm_cid_work (struct task_struct *t) {unsigned long now = jiffies, old_scan, next_scan; -struct task_struct *t = current;struct cpumask *cidmask; -struct mm_struct *mm;int weight, cpu; +struct mm_struct *mm = t->mm; - WARN_ON_ONCE(t != container_of(work, struct task_struct, cid_work)); - - work->next = work;/* Prevent double-add */ -if (t->flags &PF_EXITING ) -return ; - mm = t->mm; -if (!mm) -return ; old_scan =READ_ONCE (mm->mm_cid_next_scan); next_scan = now +msecs_to_jiffies (MM_CID_SCAN_DELAY );if (!old_scan) {@@ -10643,23 +10634,47 @@ void init_sched_mm_cid(struct task_struct *t)if (mm_users ==1 ) mm->mm_cid_next_scan = jiffies +msecs_to_jiffies (MM_CID_SCAN_DELAY ); } - t->cid_work.next = &t->cid_work;/* Protect against double add */ -init_task_work (&t->cid_work, task_mm_cid_work); }void task_tick_mm_cid (struct rq *rq,struct task_struct *curr) { -struct callback_head *work = &curr->cid_work; -unsigned long now = jiffies; +u64 rtime = curr->se.sum_exec_runtime - curr->se.prev_sum_exec_runtime; +/* +* If a task is running unpreempted for a long time, it won't get its +* mm_cid compacted and won't update its mm_cid value after a +* compaction occurs. +* For such a task, this function does two things: +* A) trigger the mm_cid recompaction, +* B) trigger an update of the task's rseq->mm_cid field at some point +* after recompaction, so it can get a mm_cid value closer to 0. +* A change in the mm_cid triggers an rseq_preempt. +* +* B occurs once after the compaction work completes, neither A nor B +* run as long as the compaction work is pending, the task is exiting +* or is not a userspace task. +*/ if (!curr->mm || (curr->flags & (PF_EXITING |PF_KTHREAD )) || - work->next != work) +test_tsk_thread_flag (curr,TIF_NOTIFY_RESUME ))return ; -if (time_before (now,READ_ONCE (curr->mm->mm_cid_next_scan))) +if (rtime <RSEQ_UNPREEMPTED_THRESHOLD )return ; - -/* No page allocation under rq lock */ -task_work_add (curr, work,TWA_RESUME ); +if (mm_cid_needs_scan (curr->mm)) { +/* Trigger mm_cid recompaction */ +rseq_set_notify_resume (curr); + }else if (time_after (jiffies, curr->last_cid_reset + +msecs_to_jiffies (MM_CID_SCAN_DELAY ))) { +/* Update mm_cid field */ +int old_cid = curr->mm_cid; + +if (!curr->mm_cid_active) +return ; +mm_cid_snapshot_time (rq, curr->mm); +mm_cid_put_lazy (curr); + curr->last_mm_cid = curr->mm_cid =mm_cid_get (rq, curr, curr->mm); +if (old_cid != curr->mm_cid) +rseq_preempt (curr); + } }void sched_mm_cid_exit_signals (struct task_struct *t)kernel/sched/sched.h +2
-0@@ -3606,6 +3606,7 @@ extern const char *preempt_modes[];#define SCHED_MM_CID_PERIOD_NS (100ULL * 1000000)/* 100ms */ #define MM_CID_SCAN_DELAY 100/* 100ms */ +#define RSEQ_UNPREEMPTED_THRESHOLD SCHED_MM_CID_PERIOD_NSextern raw_spinlock_t cid_lock;extern int use_cid_lock;@@ -3809,6 +3810,7 @@ static inline int mm_cid_get(struct rq *rq, struct task_struct *t,int cid;lockdep_assert_rq_held (rq); + t->last_cid_reset = jiffies; cpumask =mm_cidmask (mm); cid =__this_cpu_read (pcpu_cid->cid);if (mm_cid_is_valid (cid)) {
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Re: [PATCH v2 2/4] rseq: Run the mm_cid_compaction from rseq_handle_notify_resume()
Quoted 23 lines · Mathieu Desnoyers
Quoted 17 lines · Gabriele Monaco
Currently the
mm_cid_compactionis triggered by the scheduler tick and
runs in atask_work, behaviour is more unpredictable with periodic tasks
with short runtime, which may rarely run during a tick.Run the
mm_cid_compactionfrom therseq_handle_notify_resume()call,
which runs fromresume_user_mode_work. Since the context is the same
where thetask_workwould run, skip this step and call the compaction
function directly.
The compaction function still exits prematurely in case the scan is not
required, that is when the pseudo-period of 100ms did not elapse.Quoted 4 lines
Keep a tick handler used for long running tasks that are never preempted
(i.e. that never callrseq_handle_notify_resume), which triggers a compaction andmm_cidupdate only in that case.
Quoted 7 lines
Your approach looks good, but please note that this will probably need to be rebased on top of the rseq rework from Thomas Gleixner.
Latest version can be found here:
Mmh that's quite a large one, thanks for sharing!
I'm going to have a look but it might make sense to wait until that's included, I guess.
Thanks,
Gabriele
Quoted 232 lines
Thanks,
Mathieu
Quoted 227 lines
Signed-off-by Gabriele Monaco <[email protected]>---
include/linux/mm.h | 2 ++include/linux/mm_types.h| 11 ++++++++ include/linux/sched.h | 2 +-
kernel/rseq.c | 2 ++
kernel/sched/core.c | 55 +++++++++++++++++++++++++----------
-----
kernel/sched/sched.h | 2 ++6 files changed, 53 insertions(+), 21 deletions(-)include/linux/mm.h +1
-0@@ -2294,6 +2294,7 @@ void sched_mm_cid_before_execve(struct task_struct *t); void sched_mm_cid_after_execve(struct task_struct *t); void sched_mm_cid_fork(struct task_struct *t); void sched_mm_cid_exit_signals(struct task_struct *t); +void task_mm_cid_work (struct task_struct *t); static inline int task_mm_cid(struct task_struct *t) {return
t->mm_cid;include/linux/mm.h +0
-0@@ -2303,6 +2304,7 @@ static inline void sched_mm_cid_before_execve(struct task_struct *t) { }static inline void
sched_mm_cid_after_execve(struct task_struct *t){ }
static inline voidsched_mm_cid_fork(struct task_struct *t){ } static inline voidsched_mm_cid_exit_signals(struct task_struct *t){ }
+static inline voidtask_mm_cid_work(struct task_struct *t){ } static inline inttask_mm_cid(struct task_struct *t){
/*include/linux/mm_types.h +0
-0@@ -1420,6 +1420,13 @@ static inline voidmm_set_cpus_allowed(struct
mm_struct*mm, const struct cpumasWRITE_ONCE(mm->nr_cpus_allowed, cpumask_weight(mm_allowed));raw_spin_unlock(&mm->cpus_allowed_lock);
}include/linux/mm_types.h +6
-0+ +static inline bool mm_cid_needs_scan (struct mm_struct *mm) +{ +if (!mm) +return false; + return time_after(jiffies, READ_ONCE(mm-Quoted 1 line
mm_cid_next_scan));+}
#else /*CONFIG_SCHED_MM_CID*/
static inline voidmm_init_cid(struct mm_struct *mm, struct task_struct *p){ }
static inline intmm_alloc_cid(struct mm_struct *mm, struct task_struct *p){ return 0; }include/linux/mm_types.h +4
-0@@ -1430,6 +1437,10 @@ static inline unsigned int mm_cid_size(void) return 0; } static inline void mm_set_cpus_allowed(struct mm_struct *mm,const struct cpumask *cpumask) { } +static inline bool mm_cid_needs_scan (struct mm_struct *mm) +{ +return false; +} #endif /* CONFIG_SCHED_MM_CID */
structmmu_gather;include/linux/sched.h +0
-0@@ -1428,7 +1428,7 @@ struct task_struct {int
last_mm_cid; /* Most
recent cid in mm */
intmigrate_from_cpu;
intmm_cid_active; /* Whether
cid bitmap is active */include/linux/sched.h +1
-1-struct callback_head cid_work; +unsigned long last_cid_reset;/* Time of last reset in jiffies */
#endif
structtlbflush_unmap_batchtlb_ubc;kernel/rseq.c +1
-0@@ -441,6 +441,8 @@ void __rseq_handle_notify_resume(struct ksignal *ksig, struct pt_regs *regs) } if (unlikely(rseq_update_cpu_node_id(t))) goto error; +/* The mm_cid compaction returns prematurely if scan is not needed. */
kernel/rseq.c +1
-0+task_mm_cid_work (t);return;
error:kernel/sched/core.c +0
-0@@ -10589,22 +10589,13 @@ static void sched_mm_cid_remote_clear_weight(struct mm_struct *mm, int cpu, sched_mm_cid_remote_clear(mm, pcpu_cid, cpu); }kernel/sched/core.c +1
-1-static void task_mm_cid_work (struct callback_head *work) +void task_mm_cid_work (struct task_struct *t){
unsigned long now = jiffies,old_scan,next_scan;kernel/sched/core.c +0
-1-struct task_struct *t = current;struct cpumask *cidmask;
kernel/sched/core.c +0
-1-struct mm_struct *mm;int weight, cpu;
kernel/sched/core.c +1
-0+struct mm_struct *mm = t->mm;kernel/sched/core.c +0
-1- WARN_ON_ONCE(t != container_of(work, struct task_struct,
cid_work));kernel/sched/core.c +0
-7- - work->next = work;/* Prevent double-add */ -if (t->flags &PF_EXITING ) -return ; - mm = t->mm; -if (!mm) -return ;
old_scan=READ_ONCE(mm->mm_cid_next_scan);next_scan= now +msecs_to_jiffies(MM_CID_SCAN_DELAY); if (!old_scan) {kernel/sched/core.c +0
-0@@ -10643,23 +10634,47 @@ void init_sched_mm_cid(struct task_struct *t) if (mm_users == 1)
mm->mm_cid_next_scan= jiffies +msecs_to_jiffies(MM_CID_SCAN_DELAY);
}kernel/sched/core.c +0
-1- t->cid_work.next = &t->cid_work;/* Protect against double add */
kernel/sched/core.c +0
-1-init_task_work (&t->cid_work, task_mm_cid_work);}
voidtask_tick_mm_cid(struct rq *rq, struct task_struct *curr){kernel/sched/core.c +1
-2-struct callback_head *work = &curr->cid_work; -unsigned long now = jiffies; + u64 rtime = curr->se.sum_exec_runtime - curr-Quoted 1 line
se.prev_sum_exec_runtime;
kernel/sched/core.c +2
-0+/* +* If a task is running unpreempted for a long time, it won't get its
kernel/sched/core.c +1
-0+* mm_cid compacted and won't update its mm_cid value after a
kernel/sched/core.c +4
-0+* compaction occurs. +* For such a task, this function does two things: +* A) trigger the mm_cid recompaction, +* B) trigger an update of the task's rseq->mm_cid field at some point
kernel/sched/core.c +1
-0+* after recompaction, so it can get a mm_cid value closer to 0.
kernel/sched/core.c +3
-0+* A change in the mm_cid triggers an rseq_preempt. +* +* B occurs once after the compaction work completes, neither A nor B
kernel/sched/core.c +1
-0+* run as long as the compaction work is pending, the task is exiting
kernel/sched/core.c +2
-0+* or is not a userspace task. +*/ if (!curr->mm || (
curr->flags& (PF_EXITING|PF_KTHREAD)) ||kernel/sched/core.c +1
-1- work->next != work) + test_tsk_thread_flag(curr, TIF_NOTIFY_RESUME))return;
kernel/sched/core.c +0
-1- if (time_before(now, READ_ONCE(curr->mm-Quoted 1 line
mm_cid_next_scan)))kernel/sched/core.c +1
-0+if (rtime <RSEQ_UNPREEMPTED_THRESHOLD )return;
kernel/sched/core.c +5
-3- -/* No page allocation under rq lock */ -task_work_add (curr, work,TWA_RESUME ); +if (mm_cid_needs_scan (curr->mm)) { +/* Trigger mm_cid recompaction */ +rseq_set_notify_resume (curr); + } else if (time_after(jiffies, curr->last_cid_reset + +
msecs_to_jiffies(MM_CID_SCAN_DELAY))) {kernel/sched/core.c +8
-0+/* Update mm_cid field */ +int old_cid = curr->mm_cid; + +if (!curr->mm_cid_active) +return ; +mm_cid_snapshot_time (rq, curr->mm); +mm_cid_put_lazy (curr); + curr->last_mm_cid = curr->mm_cid = mm_cid_get(rq,curr,
curr->mm);kernel/sched/core.c +3
-0+if (old_cid != curr->mm_cid) +rseq_preempt (curr); + }}
voidsched_mm_cid_exit_signals(struct task_struct *t)kernel/sched/sched.h +0
-0@@ -3606,6 +3606,7 @@ extern const char *preempt_modes[];
#defineSCHED_MM_CID_PERIOD_NS(100ULL * 1000000) /*
100ms */
#defineMM_CID_SCAN_DELAY100 /* 100ms
*/
+#defineRSEQ_UNPREEMPTED_THRESHOLDSCHED_MM_CID_PERIOD_NS
externraw_spinlock_tcid_lock;
extern intuse_cid_lock;kernel/sched/sched.h +0
-0@@ -3809,6 +3810,7 @@ static inline int mm_cid_get(struct rq *rq,struct task_struct *t, int cid;
lockdep_assert_rq_held(rq);kernel/sched/sched.h +1
-0+ t->last_cid_reset = jiffies;cpumask =
mm_cidmask(mm);
cid =__this_cpu_read(pcpu_cid->cid);
if (mm_cid_is_valid(cid)) {
Re: [PATCH v2 2/4] rseq: Run the mm_cid_compaction from rseq_handle_notify_resume()
In reply to Mathieu DesnoyersQuoted 6 lines · Mathieu Desnoyers
Your approach looks good, but please note that this will probably need to be rebased on top of the rseq rework from Thomas Gleixner.
Latest version can be found here:
I rebased and adapted the patches on the V4 of that series.
To get close functionality I went back to the task_work and I'm scheduling it from switches (rseq_sched_switch_event).
Quick recap:
My series tries to reduce the latency caused by task_mm_cid_work on many-CPU systems. While at it, it improves reliability for bursty tasks that can miss the tick.
It reduces the latency by splitting the work in batches. This requires more reliability as compaction now needs more runs, which is achieved enqueuing on switches instead of ticks.
While this solution works, my doubt is whether running something there is still acceptable, considering Thomas' effort is going in the opposite direction.
My tests don't show any significant performance difference, but I'd gladly try different workloads.
Any thoughts on this?
If the approach still looks reasonable I can submit a proper series for review.
You can find the series at:
git://git.kernel.org/pub/scm/linux/kernel/git/gmonaco/linux.git mm_cid_batches_rebased
Thanks,
Gabriele
Re: [PATCH v2 2/4] rseq: Run the mm_cid_compaction from rseq_handle_notify_resume()
Quoted 12 lines · Gabriele Monaco
On Tue, 2025-08-26 at 14:01 -0400, Mathieu Desnoyers wrote:
To get close functionality I went back to thetask_workand I'm scheduling it from switches (rseq_sched_switch_event).Quick recap:
My series tries to reduce the latency caused bytask_mm_cid_workon many-CPU systems. While at it, it improves reliability for bursty tasks that can miss the tick.
It reduces the latency by splitting the work in batches. This requires more reliability as compaction now needs more runs, which is achieved enqueuing on switches instead of ticks.
Quoted 2 lines
While this solution works, my doubt is whether running something there is still acceptable, considering Thomas' effort is going in the opposite direction.
The current overhead of RSEQ is way too high. People have reported 3% regressions just because glibc uses RSEQ now. So that needs to be addressed. Moving the RSEQ fastpath to the last point before going back to user space is the correct thing to do.
mm cid compaction is a related but orthogonal problem. I just skimmed your patches and I'm really not convinced that this is the right approach to solve the problem.
The main issue with task_mm_cid_work() is that it iterates over all possible CPUs twice. Your batching just makes the impact smaller, but it does not really try to change the overall approach to this.
Looking at all the related pieces, the whole CID management is in my opinion way too complex and overengineered.
The basic goals of this are to:
1) keep the CID of a task stable when possible
2) limit the CID space to min(mm->nr_cpus_allowed, mm->mm_users)
with the requirement that:
3) The CID of a task running on a CPU cannot be changed.
IOW, the only point where the CID of a task can be modified is schedule().
The current implementation with all the bells and whistels including this scan magic was introduced due to a performance regression of the initial naive implementation which used a per MM lock to manage the CIDs. IOW, this went from truly simple to overly complex in one go.
None of this is required in my opinion because of #3.
If a task is scheduled in then the CID is already today picked below the limit provided by #2. That limit is racy, but there is not much what can be done about that without adding locking overhead. It's not a real issue. Let's look at the possible relevant scenarios:
A) Limit is unchanged
Allocating a CID in switch_to() is guaranteed to succeed
B) Limit is decreased after calculating the maximum value
Allocating a CID in switch_to() is guaranteed to succeed and the allocated CID which might be occacionally above the limit stays in use until the task schedules again.
C) Limit is increased after calculating the maximum value
Allocating a CID in switch_to() can fail transiently in this
scenario:
CPU0 CPU1 CPU2
// max = 2
max = min(nr_allowed, mm_users);
// current CID = 0
fork()
mm_users++;
...
switch_to(new_task)
// max = 3
max = min(nr_allowed, mm_users);
cid = 1; // observes CID 0, 1 in use
// and has to repeat
In theory this might result in an endless retry loop, but I claim that this is a purely academic problem unless proven otherwise.
Now, how to achieve #1 and keep the #2 constraint with the transient exception of the #B case above?
1) Each task keeps track of the current and the last allocated CID. When
scheduling out t->mm_cid is invalidated and t->mm_last_cid is kept.
2) Each MM keeps track of the last CID which was used on a CPU
3) The in-use CIDs of each MM are kept in a bitmap
When a task is scheduled in the CID allocation does:
A) Try to get tsk->mm_last_cid, if it still fits into the limit.
B) Try to get mm->pcpu_cid[cpu], if it still fits into the limit.
C) Allocate a free bit in mm->mm_cidmask
I've implemented a rough sketch of that on top of:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git rseq/perf
I picked that as a base because it removes all the other RSEQ noise and allowed me to instrument the whole thing better. For your conveniance the result is also in git:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git rseq/cid
That sketch holds up in initial testing pretty well. Quite some of the random test cases I threw at it showed slight improvements, but none of them degraded. Just for the record:
Full kernel build: 92.65s -> 92.31s (average of ten iterations) RSEQ selftests: 540.67s -> 538.62s (average of ten iterations)
Not really impresive, but the latencies of the task work are obviously gone. I know that Mathieu hates using the selftests for reference, but the numbers speak for themself no matter what.
I'm definitely not claiming that it will hold up in real wider testing, but those initial results tell me that this is definitely more worthwhile to explore further than trying to apply hacky bandaids to the existing implementation.
There are some rough edges in it:
- the above #A -> #B -> #C allocation order might be not ideal.
- the initial last_mm_cid assignment in init_sched_mm_cid() is just a quick hack to avoid that a newly forked task will take over a currently unused but previously in use CID.
But those are all solvable problems, which just need more thoughts than a couple of hours spent with initial reasoning.
What's really stunning given that those couple of hours of hacking on it did not end up in a obvious major regression is the resulting diffstat:
include/linux/mm_types.h | 53 ---- kernel/fork.c | 5 kernel/sched/core.c | 513 +---------------------------------------------- kernel/sched/sched.h | 297 ++++----------------------- 4 files changed, 72 insertions(+), 796 deletions(-)
Yet another datapoint for my long standing claim that KISS is the most important engineering principle.
Feel free to disagree with that, but then please prove me wrong.
Thanks,
tglx
---
Subject: sched: Simplify MM CID management
From: Thomas Gleixner <[email protected]>
Date: Mon, 29 Sep 2025 23:45:28 +0200
Add content here...
--- include/linux/mm_types.h | 53 ---- kernel/fork.c | 5 kernel/sched/core.c | 513 +---------------------------------------------- kernel/sched/sched.h | 301 ++++----------------------- 4 files changed, 74 insertions(+), 798 deletions(-)
include/linux/mm_types.h +2 -51
@@ -919,13 +919,9 @@ struct vm_area_struct {#define vma_policy (vma) NULL#endif -#ifdef CONFIG_SCHED_MM_CID struct mm_cid { -u64 time; -int cid; -int recent_cid; +unsigned int cid; }; -#endif struct kioctx_table ;struct iommu_mm_data ;
@@ -988,12 +984,6 @@ struct mm_struct {* runqueue locks. */ struct mm_cid __percpu *pcpu_cid; -/* -* @mm_cid_next_scan: Next mm_cid scan (in jiffies). -* -* When the next mm_cid scan is due (in jiffies). -*/ -unsigned long mm_cid_next_scan;/** * @nr_cpus_allowed: Number of CPUs allowed for mm. *
@@ -1002,14 +992,6 @@ struct mm_struct {*/ unsigned int nr_cpus_allowed;/** -* @max_nr_cid: Maximum number of allowed concurrency -* IDs allocated. -* -* Track the highest number of allowed concurrency IDs -* allocated for the mm. -*/ -atomic_t max_nr_cid; -/** * @cpus_allowed_lock: Lock protecting mm cpus_allowed. * * Provide mutual exclusion for mm cpus_allowed and
@@ -1320,35 +1302,7 @@ static inline void vma_iter_init(struct#ifdef CONFIG_SCHED_MM_CID -enum mm_cid_state { -MM_CID_UNSET =-1U ,/* Unset state has lazy_put flag set. */ -MM_CID_LAZY_PUT = (1U <<31 ), -}; - -static inline bool mm_cid_is_unset (int cid) -{ -return cid ==MM_CID_UNSET ; -} - -static inline bool mm_cid_is_lazy_put (int cid) -{ -return !mm_cid_is_unset (cid) && (cid &MM_CID_LAZY_PUT ); -} - -static inline bool mm_cid_is_valid (int cid) -{ -return !(cid &MM_CID_LAZY_PUT ); -} - -static inline int mm_cid_set_lazy_put (int cid) -{ -return cid |MM_CID_LAZY_PUT ; -} - -static inline int mm_cid_clear_lazy_put (int cid) -{ -return cid & ~MM_CID_LAZY_PUT ; -} +#define MM_CID_UNSET (~0U)/* * mm_cpus_allowed: Union of all mm's threads allowed CPUs.
@@ -1381,11 +1335,8 @@ static inline void mm_init_cid(struct mmstruct mm_cid *pcpu_cid =per_cpu_ptr (mm->pcpu_cid, i); pcpu_cid->cid =MM_CID_UNSET ; - pcpu_cid->recent_cid =MM_CID_UNSET ; - pcpu_cid->time =0 ; } mm->nr_cpus_allowed = p->nr_cpus_allowed; -atomic_set (&mm->max_nr_cid,0 );raw_spin_lock_init (&mm->cpus_allowed_lock);cpumask_copy (mm_cpus_allowed (mm), &p->cpus_mask);cpumask_clear (mm_cidmask (mm));
kernel/fork.c +2 -3
@@ -956,10 +956,9 @@ static struct task_struct *dup_task_stru#endif #ifdef CONFIG_SCHED_MM_CID - tsk->mm_cid =-1 ; - tsk->last_mm_cid =-1 ; + tsk->mm_cid =MM_CID_UNSET ; + tsk->last_mm_cid =MM_CID_UNSET ; tsk->mm_cid_active =0 ; - tsk->migrate_from_cpu =-1 ;#endif return tsk;
kernel/sched/core.c +17 -496
568 lines not on this page · raw message
kernel/sched/sched.h +53 -248
341 lines not on this page · raw message
Re: [PATCH v2 2/4] rseq: Run the mm_cid_compaction from rseq_handle_notify_resume()
Quoted 6 lines · Thomas Gleixner
The current overhead of RSEQ is way too high. People have reported 3% regressions just because glibc uses RSEQ now. So that needs to be addressed. Moving the RSEQ fastpath to the last point before going back to user space is the correct thing to do.
Quoted 7 lines
mm cid compaction is a related but orthogonal problem. I just skimmed your patches and I'm really not convinced that this is the right approach to solve the problem.
The main issue with
task_mm_cid_work()is that it iterates over all possible CPUs twice. Your batching just makes the impact smaller, but it does not really try to change the overall approach to this.
Thanks for the thorough explanation and sketch implementation. As I'm not really confident with rseq I didn't dare changing the compaction logic too much, but clearly your approach of getting compact cid by construction seems the way to go.
I confirm your patch passes the selftest in 4/4 of this series and, obviously, removes the latency I was observing.
Thanks,
Gabriele
Re: [PATCH v2 2/4] rseq: Run the mm_cid_compaction from rseq_handle_notify_resume()
In reply to Thomas GleixnerQuoted 9 lines · Thomas Gleixner
On Wed, Sep 24 2025 at 17:22, Gabriele Monaco wrote:
I'm definitely not claiming that it will hold up in real wider testing, but those initial results tell me that this is definitely more worthwhile to explore further than trying to apply hacky bandaids to the existing implementation.There are some rough edges in it:
- the above #A -> #B -> #C allocation order might be not ideal.
And actually it is not for the case where there is a massive back to back scheduling of threads which share the mm because that makes the cidmask operations contended. Thanks to Mathieu and Peter for pointing it out.
Though it actually needs a micro benchmark to see it in perf. Neither hackbench in threaded mode nor other obnoxious test cases show it.
But after trying a few approaches and utter frustration I decided to ignore it for a while and then look at it with a fresh mind again. The solution was there in plain sight as the current implementation does something similar in the middle of the maze:
If the scheduled out (previous) and the scheduled in (next) task share the mm, then just inheriting the CID from the previous task w/o touching the CID mask makes that prominent cpumask fiddling in perf top go away completely.
With that the overhead of __schedule() in perf top actually becomes smaller than with the original base implementation (rseq/perf branch). That's on bare metal with hyperthreading enabled.
Intel SKL AMD Zen3 CPUs: 112 256 base: 5.1% __schedule 4.4% __schedule
patch-v1: 10.2% __schedule 11.2% __schedule
patch-v2: 4.9% __schedule 3.8% __schedule
Cool, right?
But then I really have reached the limits of my performance testing abilities. Any benchmark I threw at the lot and especially a simple malloc comparison benchmark:
https://github.com/f18m/malloc-benchmarks
which runs the benchmark magic against various malloc libraries, gives me incomprehensible results, which vary from run to run on the same kernel and end up in both directions of improvement or regression.
The malloc-benchmark seems to amplify the improvements/regressions around the scalability points which are on the mainline base line already, so I assume that's some weak spot of those libraries.
Anyway, I've applied the delta patch below to my devel branch:
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git rseq/cid
I'd really appreciate help from people who actually have the relevant benchmarks and know how to do actually reproducible evaluations and comparisons.
The interesting evaluation points are:
1) upstream Linus
and then the commits in
git://git.kernel.org/pub/scm/linux/kernel/git/tglx/devel.git rseq/cid
2) 1822acbae2c9 ("rseq: Switch to TIF_RSEQ if supported")
which is the big rseq rework
3) a769dbb7a22a ("sched: Simplify MM CID management")
the v1 hack
4) abd407e145f2 ("sched: More CID hackery")
the below
Thanks,
tglx
---
Subject: sched: More CID hackery
From: Thomas Gleixner <[email protected]>
Date: Wed, 01 Oct 2025 21:38:48 +0200
Add content here...
--- include/linux/mm_types.h | 41 ++++++++++++++++++++++------------------- include/linux/sched.h | 4 +--- kernel/fork.c | 1 + kernel/sched/core.c | 36 +++++++++++++++++++----------------- kernel/sched/sched.h | 33 +++++++++++++++++++++++++-------- kernel/signal.c | 4 ++-- 6 files changed, 70 insertions(+), 49 deletions(-)
include/linux/mm_types.h +22 -19
@@ -919,7 +919,7 @@ struct vm_area_struct {#define vma_policy (vma) NULL#endif -struct mm_cid { +struct mm_cid_pcpu {unsigned int cid; };
@@ -977,27 +977,23 @@ struct mm_struct {#ifdef CONFIG_SCHED_MM_CID /** -* @pcpu_cid: Per-cpu current cid. -* -* Keep track of the currently allocated mm_cid for each cpu. -* The per-cpu mm_cid values are serialized by their respective -* runqueue locks. +* @pcpu_cid: Per-cpu CID storage */ -struct mm_cid __percpu *pcpu_cid; +struct mm_cid_pcpu __percpu *pcpu_cid;/** * @nr_cpus_allowed: Number of CPUs allowed for mm. * -* Number of CPUs allowed in the union of all mm's -* threads allowed CPUs. +* Number of CPUs allowed in the union of all mm's threads +* allowed CPUs. It only grows, but can never shrink. */ unsigned int nr_cpus_allowed; +unsigned int mm_cid_users; +unsigned int mm_max_cid; +/** -* @cpus_allowed_lock: Lock protecting mm cpus_allowed. -* -* Provide mutual exclusion for mm cpus_allowed and -* mm nr_cpus_allowed updates. +* @mm_cid_lock: Lock protecting the above */ -raw_spinlock_t cpus_allowed_lock; +raw_spinlock_t mm_cid_lock;#endif #ifdef CONFIG_MMU atomic_long_t pgtables_bytes;/* size of all page tables */
@@ -1332,19 +1328,19 @@ static inline void mm_init_cid(struct mmint i;for_each_possible_cpu (i) { -struct mm_cid *pcpu_cid =per_cpu_ptr (mm->pcpu_cid, i); +struct mm_cid_pcpu *pcpu_cid =per_cpu_ptr (mm->pcpu_cid, i); pcpu_cid->cid =MM_CID_UNSET ; } mm->nr_cpus_allowed = p->nr_cpus_allowed; -raw_spin_lock_init (&mm->cpus_allowed_lock); +raw_spin_lock_init (&mm->mm_cid_lock);cpumask_copy (mm_cpus_allowed (mm), &p->cpus_mask);cpumask_clear (mm_cidmask (mm)); }static inline int mm_alloc_cid_noprof (struct mm_struct *mm,struct task_struct *p) { - mm->pcpu_cid = alloc_percpu_noprof(struct mm_cid); + mm->pcpu_cid = alloc_percpu_noprof(struct mm_cid_pcpu);if (!mm->pcpu_cid)return -ENOMEM ;mm_init_cid (mm, p);
@@ -1363,6 +1359,13 @@ static inline unsigned int mm_cid_size(vreturn 2 *cpumask_size ();/* mm_cpus_allowed(), mm_cidmask(). */ } +static inline void mm_update_max_cid (struct mm_struct *mm) +{ +unsigned int max_cid =min (mm->nr_cpus_allowed, mm->mm_cid_users); + +WRITE_ONCE (mm->mm_max_cid, max_cid); +} +static inline void mm_set_cpus_allowed (struct mm_struct *mm,const struct cpumask *cpumask) {struct cpumask *mm_allowed =mm_cpus_allowed (mm);
@@ -1370,10 +1373,10 @@ static inline void mm_set_cpus_allowed(sif (!mm)return ;/* The mm_cpus_allowed is the union of each thread allowed CPUs masks. */ -raw_spin_lock (&mm->cpus_allowed_lock); +guard (raw_spinlock)(&mm->mm_cid_lock);cpumask_or (mm_allowed, mm_allowed, cpumask);WRITE_ONCE (mm->nr_cpus_allowed,cpumask_weight (mm_allowed)); - raw_spin_unlock(&mm->cpus_allowed_lock); + mm_update_max_cid(mm); }#else /* CONFIG_SCHED_MM_CID */ static inline void mm_init_cid (struct mm_struct *mm,struct task_struct *p) { }
include/linux/sched.h +1 -3
@@ -1405,9 +1405,7 @@ struct task_struct {#ifdef CONFIG_SCHED_MM_CID int mm_cid;/* Current cid in mm */ int last_mm_cid;/* Most recent cid in mm */ -int migrate_from_cpu;int mm_cid_active;/* Whether cid bitmap is active */ -struct callback_head cid_work;#endif struct tlbflush_unmap_batch tlb_ubc;
@@ -2299,7 +2297,7 @@ static __always_inline void alloc_tag_revoid sched_mm_cid_before_execve (struct task_struct *t);void sched_mm_cid_after_execve (struct task_struct *t);void sched_mm_cid_fork (struct task_struct *t); -void sched_mm_cid_exit_signals (struct task_struct *t); +void sched_mm_cid_exit (struct task_struct *t);static inline int task_mm_cid (struct task_struct *t) {return t->mm_cid;
kernel/fork.c +1 -0
@@ -2449,6 +2449,7 @@ static bool need_futex_hash_allocate_defexit_task_namespaces (p); bad_fork_cleanup_mm:if (p->mm) { +sched_mm_cid_exit (p);mm_clear_owner (p->mm, p);mmput (p->mm); }
kernel/sched/core.c +19 -17
@@ -10408,44 +10408,46 @@ void call_trace_sched_update_nr_running(* When a task exits, the MM CID held by the task is not longer required as * the task cannot return to user space. */ -void sched_mm_cid_exit_signals (struct task_struct *t) +void sched_mm_cid_exit (struct task_struct *t) {struct mm_struct *mm = t->mm;if (!mm || !t->mm_cid_active)return ; -guard (preempt)(); +guard (raw_spinlock_irq)(&mm->mm_cid_lock); t->mm_cid_active =0 ;if (t->mm_cid !=MM_CID_UNSET ) {cpumask_clear_cpu (t->mm_cid,mm_cidmask (mm)); t->mm_cid =MM_CID_UNSET ; } + mm->mm_cid_users--; +mm_update_max_cid (mm); } -/* Deactivate MM CID allocation across execve() */ -void sched_mm_cid_before_execve (struct task_struct *t) -{ -sched_mm_cid_exit_signals (t); -} - -/* Reactivate MM CID after successful execve() */ -void sched_mm_cid_after_execve (struct task_struct *t) +void sched_mm_cid_fork (struct task_struct *t) {struct mm_struct *mm = t->mm; -if (!mm) -return ; +WARN_ON_ONCE (!mm || t->mm_cid !=MM_CID_UNSET ); -guard (preempt)(); +guard (raw_spinlock_irq)(&mm->mm_cid_lock); t->mm_cid_active =1 ; -mm_cid_select (t); + mm->mm_cid_users++; +mm_update_max_cid (mm); } -void sched_mm_cid_fork (struct task_struct *t) +/* Deactivate MM CID allocation across execve() */ +void sched_mm_cid_before_execve (struct task_struct *t) { -WARN_ON_ONCE (!t->mm || t->mm_cid !=MM_CID_UNSET ); - t->mm_cid_active =1 ; +sched_mm_cid_exit (t); +} + +/* Reactivate MM CID after successful execve() */ +void sched_mm_cid_after_execve (struct task_struct *t) +{ +if (t->mm) +sched_mm_cid_fork (t); }#endif /* CONFIG_SCHED_MM_CID */
kernel/sched/sched.h +25 -8
@@ -3518,7 +3518,7 @@ static inline void init_sched_mm_cid(strreturn ;/* Preset last_mm_cid */ - max_cid =min_t (int,READ_ONCE (mm->nr_cpus_allowed),atomic_read (&mm->mm_users)); + max_cid =READ_ONCE (mm->mm_max_cid); t->last_mm_cid = max_cid -1 ; }
@@ -3531,7 +3531,7 @@ static inline bool __mm_cid_get(struct tif (cpumask_test_and_set_cpu (cid,mm_cidmask (mm)))return false; t->mm_cid = t->last_mm_cid = cid; -__this_cpu_write (mm->pcpu_cid->cid, cid); +__this_cpu_write (t->mm->pcpu_cid->cid, cid);return true; }
@@ -3540,7 +3540,7 @@ static inline bool mm_cid_get(struct tasstruct mm_struct *mm = t->mm;unsigned int max_cid; - max_cid =min_t (int,READ_ONCE (mm->nr_cpus_allowed),atomic_read (&mm->mm_users)); + max_cid =READ_ONCE (mm->mm_max_cid);/* Try to reuse the last CID of this task */ if (__mm_cid_get (t, t->last_mm_cid, max_cid))
@@ -3568,14 +3568,31 @@ static inline void mm_cid_select(struct } } -static inline void switch_mm_cid (struct task_struct *prev,struct task_struct *next) +static inline bool mm_cid_inherit (struct task_struct *prev,struct task_struct *next) { -if (prev->mm_cid_active) { -if (prev->mm_cid !=MM_CID_UNSET ) -cpumask_clear_cpu (prev->mm_cid,mm_cidmask (prev->mm)); - prev->mm_cid =MM_CID_UNSET ; +unsigned int cid = prev->mm_cid; + +if (!prev->mm_cid_active || cid ==MM_CID_UNSET ) +return false; + + prev->mm_cid =MM_CID_UNSET ; + +if (!next->mm_cid_active || prev->mm != next->mm || + cid >=READ_ONCE (prev->mm->mm_max_cid)) { +cpumask_clear_cpu (cid,mm_cidmask (prev->mm)); +return false; } + next->mm_cid = next->last_mm_cid = cid; +rseq_sched_set_task_mm_cid (next, cid); +return true; +} + +static inline void switch_mm_cid (struct task_struct *prev,struct task_struct *next) +{ +if (mm_cid_inherit (prev, next)) +return ; +if (next->mm_cid_active) {mm_cid_select (next);rseq_sched_set_task_mm_cid (next, next->mm_cid);
kernel/signal.c +2 -2
@@ -3125,7 +3125,7 @@ void exit_signals(struct task_struct *tscgroup_threadgroup_change_begin (tsk);if (thread_group_empty (tsk) || (tsk->signal->flags &SIGNAL_GROUP_EXIT )) { -sched_mm_cid_exit_signals (tsk); +sched_mm_cid_exit (tsk); tsk->flags |=PF_EXITING ;cgroup_threadgroup_change_end (tsk);return ;
@@ -3136,7 +3136,7 @@ void exit_signals(struct task_struct *ts* From now this task is not visible for group-wide signals, * see wants_signal(), do_signal_stop(). */ -sched_mm_cid_exit_signals (tsk); +sched_mm_cid_exit (tsk); tsk->flags |=PF_EXITING ;cgroup_threadgroup_change_end (tsk);
PATCH v2 3/4 sched: Compact RSEQ concurrency IDs in batches
In reply to Gabriele MonacoCurrently, task_mm_cid_work() is called from resume_user_mode_work(). This can delay the execution of the corresponding thread for the entire duration of the function, negatively affecting the response in case of real time tasks.
In practice, we observe task_mm_cid_work increasing the latency of 30-35us on a 128 cores system, this order of magnitude is meaningful under PREEMPT_RT.
Run the task_mm_cid_work in batches of up to CONFIG_RSEQ_CID_SCAN_BATCH CPUs, this reduces the duration of the delay for each scan.
The task_mm_cid_work contains a mechanism to avoid running more frequently than every 100ms. Keep this pseudo-periodicity only on complete scans.
This means each call to task_mm_cid_work returns prematurely if the period did not elapse and a scan is not ongoing (i.e. the next batch to scan is not the first).
This way full scans are not excessively delayed while still keeping each run, and introduced latency, short.
--- include/linux/mm_types.h | 15 +++++++++++++++ init/Kconfig | 12 ++++++++++++ kernel/sched/core.c | 37 ++++++++++++++++++++++++++++++++++--- 3 files changed, 61 insertions(+), 3 deletions(-)
include/linux/mm_types.h +15 -0
@@ -995,6 +995,13 @@ struct mm_struct {* When the next mm_cid scan is due (in jiffies). */ unsigned long mm_cid_next_scan; +/* +* @mm_cid_scan_batch: Counter for batch used in the next scan. +* +* Scan in batches of CONFIG_RSEQ_CID_SCAN_BATCH. This field +* increments at each scan and reset when all batches are done. +*/ +unsigned int mm_cid_scan_batch;/** * @nr_cpus_allowed: Number of CPUs allowed for mm. *
@@ -1385,6 +1392,7 @@ static inline void mm_init_cid(struct mm_struct *mm, struct task_struct *p)raw_spin_lock_init (&mm->cpus_allowed_lock);cpumask_copy (mm_cpus_allowed (mm), &p->cpus_mask);cpumask_clear (mm_cidmask (mm)); + mm->mm_cid_scan_batch =0 ; }static inline int mm_alloc_cid_noprof (struct mm_struct *mm,struct task_struct *p)
@@ -1423,8 +1431,15 @@ static inline void mm_set_cpus_allowed(struct mm_struct *mm, const struct cpumasstatic inline bool mm_cid_needs_scan (struct mm_struct *mm) { +unsigned int next_batch; +if (!mm)return false; + next_batch =READ_ONCE (mm->mm_cid_scan_batch); +/* Always needs scan unless it's the first batch. */ +if (CONFIG_RSEQ_CID_SCAN_BATCH * next_batch <num_possible_cpus () && + next_batch) +return true;return time_after (jiffies,READ_ONCE (mm->mm_cid_next_scan)); }#else /* CONFIG_SCHED_MM_CID */
init/Kconfig +12 -0
@@ -1860,6 +1860,18 @@ config DEBUG_RSEQ If unsure, say N. +config RSEQ_CID_SCAN_BATCH + int "Number of CPUs to scan at every mm_cid compaction attempt" + range 1 NR_CPUS + default 8 + depends on SCHED_MM_CID + help + CPUs are scanned pseudo-periodically to compact the CID of each task, + this operation can take a longer amount of time on systems with many + CPUs, resulting in higher scheduling latency for the current task. + A higher value means the CID is compacted faster, but results in + higher scheduling latency. + config CACHESTAT_SYSCALL bool "Enable cachestat() system call" if EXPERT default y
kernel/sched/core.c +34 -3
@@ -10591,11 +10591,26 @@ static void sched_mm_cid_remote_clear_weight(struct mm_struct *mm, int cpu,void task_mm_cid_work (struct task_struct *t) { +int weight, cpu, from_cpu, this_batch, next_batch, idx;unsigned long now = jiffies, old_scan, next_scan;struct cpumask *cidmask; -int weight, cpu;struct mm_struct *mm = t->mm; +/* +* This function is called from __rseq_handle_notify_resume, which +* makes sure t is a user thread and is not exiting. +*/ + this_batch =READ_ONCE (mm->mm_cid_scan_batch); + next_batch = this_batch +1 ; + from_cpu =cpumask_nth (this_batch *CONFIG_RSEQ_CID_SCAN_BATCH , + cpu_possible_mask); +if (from_cpu >= nr_cpu_ids) { + from_cpu =0 ; + next_batch =1 ; + } +/* Delay scan only if we are done with all cpus. */ +if (from_cpu !=0 ) + goto cid_compact; old_scan =READ_ONCE (mm->mm_cid_next_scan); next_scan = now +msecs_to_jiffies (MM_CID_SCAN_DELAY );if (!old_scan) {
@@ -10611,17 +10626,33 @@ void task_mm_cid_work(struct task_struct *t)return ;if (!try_cmpxchg (&mm->mm_cid_next_scan, &old_scan, next_scan))return ; + +cid_compact: +if (!try_cmpxchg (&mm->mm_cid_scan_batch, &this_batch, next_batch)) +return ; cidmask =mm_cidmask (mm);/* Clear cids that were not recently used. */ -for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH ) +break ;sched_mm_cid_remote_clear_old (mm, cpu); + ++idx; + } weight =cpumask_weight (cidmask);/* * Clear cids that are greater or equal to the cidmask weight to * recompact it. */ -for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH ) +break ;sched_mm_cid_remote_clear_weight (mm, cpu, weight); + ++idx; + } }void init_sched_mm_cid (struct task_struct *t)
Signature · 1 line
2.50.1
Re: [PATCH v2 3/4] sched: Compact RSEQ concurrency IDs in batches
Quoted 12 lines · Gabriele Monaco
Currently,
task_mm_cid_work()is called fromresume_user_mode_work(). This can delay the execution of the corresponding thread for the entire duration of the function, negatively affecting the response in case of real time tasks.
In practice, we observetask_mm_cid_workincreasing the latency of 30-35us on a 128 cores system, this order of magnitude is meaningful underPREEMPT_RT.Run the
task_mm_cid_workin batches of up toCONFIG_RSEQ_CID_SCAN_BATCHCPUs, this reduces the duration of the delay for each scan.
Quoted 9 lines
The
task_mm_cid_workcontains a mechanism to avoid running more frequently than every 100ms. Keep this pseudo-periodicity only on complete scans.
This means each call totask_mm_cid_workreturns prematurely if the period did not elapse and a scan is not ongoing (i.e. the next batch to scan is not the first).
This way full scans are not excessively delayed while still keeping each run, and introduced latency, short.
Mathieu, would you have some time to look at this implementation?
Thanks,
Gabriele
Quoted 161 lines
Fixes 223baf9d17f2 ("sched: Fix performance regression introduced by
mm_cid")Signed-off-by Gabriele Monaco <[email protected]>---
include/linux/mm_types.h | 15 +++++++++++++++ init/Kconfig | 12 ++++++++++++ kernel/sched/core.c | 37 ++++++++++++++++++++++++++++++++++---3 files changed, 61 insertions(+), 3 deletions(-)include/linux/mm_types.h +2
-0@@ -995,6 +995,13 @@ struct mm_struct {* When the next mm_cid scan is due (in jiffies). */ unsigned long mm_cid_next_scan; +/* +* @mm_cid_scan_batch: Counter for batch used in the next scan.
include/linux/mm_types.h +2
-0+* +* Scan in batches of CONFIG_RSEQ_CID_SCAN_BATCH. This field
include/linux/mm_types.h +1
-0+* increments at each scan and reset when all batches are done.
include/linux/mm_types.h +2
-0+*/ +unsigned int mm_cid_scan_batch;/**
* @nr_cpus_allowed: Number of CPUs allowed for mm. *include/linux/mm_types.h +1
-0@@ -1385,6 +1392,7 @@ static inline void mm_init_cid(struct mm_struct *mm, struct task_struct *p) raw_spin_lock_init(&mm->cpus_allowed_lock); cpumask_copy(mm_cpus_allowed(mm), &p->cpus_mask); cpumask_clear(mm_cidmask(mm)); + mm->mm_cid_scan_batch =0 ; }
static inline intmm_alloc_cid_noprof(struct mm_struct *mm, struct task_struct *p)include/linux/mm_types.h +0
-0@@ -1423,8 +1431,15 @@ static inline void mm_set_cpus_allowed(struct
mm_struct*mm, const struct cpumas
static inline boolmm_cid_needs_scan(struct mm_struct *mm){include/linux/mm_types.h +2
-0+unsigned int next_batch; +if (!mm)
return false;include/linux/mm_types.h +3
-0+ next_batch =READ_ONCE (mm->mm_cid_scan_batch); +/* Always needs scan unless it's the first batch. */ + if (CONFIG_RSEQ_CID_SCAN_BATCH * next_batch <
num_possible_cpus()&&include/linux/mm_types.h +2
-0+ next_batch) +return true;return
time_after(jiffies, READ_ONCE(mm->mm_cid_next_scan)); }
#else /*CONFIG_SCHED_MM_CID*/init/Kconfig +0
-0@@ -1860,6 +1860,18 @@ config DEBUG_RSEQ
If unsure, say N.init/Kconfig +2
-0+config RSEQ_CID_SCAN_BATCH + int "Number of CPUs to scan at every mm_cid compactionattempt"
init/Kconfig +5
-0+ range 1 NR_CPUS + default 8 + depends on SCHED_MM_CID + help + CPUs are scanned pseudo-periodically to compact the CID ofeach task,
init/Kconfig +1
-0+ this operation can take a longer amount of time on systemswith many
init/Kconfig +1
-0+ CPUs, resulting in higher scheduling latency for thecurrent task.
init/Kconfig +1
-0+ A higher value means the CID is compacted faster, butresults in
init/Kconfig +2
-0+ higher scheduling latency. +config
CACHESTAT_SYSCALL
bool "Enablecachestat()system call" if EXPERT default ykernel/sched/core.c +0
-0@@ -10591,11 +10591,26 @@ static void sched_mm_cid_remote_clear_weight(struct mm_struct *mm, int cpu,
voidtask_mm_cid_work(struct task_struct *t)
{kernel/sched/core.c +1
-0+int weight, cpu, from_cpu, this_batch, next_batch, idx;unsigned long now = jiffies,
old_scan,next_scan; struct cpumask *cidmask;kernel/sched/core.c +0
-1-int weight, cpu;struct
mm_struct*mm =t->mm;kernel/sched/core.c +2
-0+/* +* This function is called from __rseq_handle_notify_resume, which
kernel/sched/core.c +5
-0+* makes sure t is a user thread and is not exiting. +*/ + this_batch =READ_ONCE (mm->mm_cid_scan_batch); + next_batch = this_batch +1 ; + from_cpu = cpumask_nth(this_batch *
CONFIG_RSEQ_CID_SCAN_BATCH,kernel/sched/core.c +8
-0+ cpu_possible_mask); +if (from_cpu >= nr_cpu_ids) { + from_cpu =0 ; + next_batch =1 ; + } +/* Delay scan only if we are done with all cpus. */ +if (from_cpu !=0 ) + goto cid_compact;
old_scan=READ_ONCE(mm->mm_cid_next_scan);next_scan= now +msecs_to_jiffies(MM_CID_SCAN_DELAY); if (!old_scan) {kernel/sched/core.c +11
-1@@ -10611,17 +10626,33 @@ void task_mm_cid_work(struct task_struct *t) return; if (!try_cmpxchg(&mm->mm_cid_next_scan, &old_scan, next_scan)) return; + +cid_compact: +if (!try_cmpxchg (&mm->mm_cid_scan_batch, &this_batch, next_batch)) +return ; cidmask = mm_cidmask(mm); /* Clear cids that were not recently used. */ -for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH ) +break ; sched_mm_cid_remote_clear_old(mm, cpu); + ++idx; + } weight = cpumask_weight(cidmask); /** Clear cids that are greater or equal to the cidmask weight to
* recompact it.
*/kernel/sched/core.c +5
-1-for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH ) +break ;
sched_mm_cid_remote_clear_weight(mm, cpu, weight);kernel/sched/core.c +2
-0+ ++idx; + }}
voidinit_sched_mm_cid(struct task_struct *t)
Mathieu Desnoyers
maintainerRe: [PATCH v2 3/4] sched: Compact RSEQ concurrency IDs in batches
Quoted 23 lines · Gabriele Monaco
Quoted 12 lines · Gabriele Monaco
Currently,
task_mm_cid_work()is called fromresume_user_mode_work(). This can delay the execution of the corresponding thread for the entire duration of the function, negatively affecting the response in case of real time tasks.
In practice, we observetask_mm_cid_workincreasing the latency of 30-35us on a 128 cores system, this order of magnitude is meaningful underPREEMPT_RT.Run the
task_mm_cid_workin batches of up toCONFIG_RSEQ_CID_SCAN_BATCHCPUs, this reduces the duration of the delay for each scan.Quoted 9 lines
The
task_mm_cid_workcontains a mechanism to avoid running more frequently than every 100ms. Keep this pseudo-periodicity only on complete scans.
This means each call totask_mm_cid_workreturns prematurely if the period did not elapse and a scan is not ongoing (i.e. the next batch to scan is not the first).
This way full scans are not excessively delayed while still keeping each run, and introduced latency, short.
Quoted 1 line
Mathieu, would you have some time to look at this implementation?
Hi Gabriele,
Please note that I am currently on vacation. I'll be back shortly before the end of August, but I'm afraid there are other tasks I need to focus on before I can get back to this. I'm adding this review to my todo list for September.
Thanks,
Mathieu
Quoted 166 lines
Thanks,
GabrieleQuoted 161 lines
Fixes 223baf9d17f2 ("sched: Fix performance regression introduced by
mm_cid")Signed-off-by Gabriele Monaco <[email protected]>--- include/linux/mm_types.h | 15 +++++++++++++++ init/Kconfig | 12 ++++++++++++ kernel/sched/core.c | 37 ++++++++++++++++++++++++++++++++++--- 3 files changed, 61 insertions(+), 3 deletions(-)include/linux/mm_types.h +2
-0@@ -995,6 +995,13 @@ struct mm_struct {* When the next mm_cid scan is due (in jiffies). */ unsigned long mm_cid_next_scan; +/* +* @mm_cid_scan_batch: Counter for batch used in the next scan.
include/linux/mm_types.h +2
-0+* +* Scan in batches of CONFIG_RSEQ_CID_SCAN_BATCH. This field
include/linux/mm_types.h +1
-0+* increments at each scan and reset when all batches are done.
include/linux/mm_types.h +3
-0+*/ +unsigned int mm_cid_scan_batch; /*** @nr_cpus_allowed: Number of CPUs allowed for mm. * @@ -1385,6 +1392,7 @@ static inline void mm_init_cid(struct mm_struct *mm, struct task_struct *p) raw_spin_lock_init(&mm->cpus_allowed_lock); cpumask_copy(mm_cpus_allowed(mm), &p->cpus_mask); cpumask_clear(mm_cidmask(mm)); + mm->mm_cid_scan_batch =0 ; }static inline int mm_alloc_cid_noprof(struct
mm_struct*mm, structtask_struct*p)include/linux/mm_types.h +0
-0@@ -1423,8 +1431,15 @@ static inline void mm_set_cpus_allowed(struct
mm_struct*mm, const struct cpumasstatic inline bool
mm_cid_needs_scan(struct mm_struct *mm){include/linux/mm_types.h +5
-0+unsigned int next_batch; + if (!mm) return false; + next_batch =READ_ONCE (mm->mm_cid_scan_batch); +/* Always needs scan unless it's the first batch. */ + if (CONFIG_RSEQ_CID_SCAN_BATCH * next_batch <
num_possible_cpus()&&include/linux/mm_types.h +2
-0+ next_batch) +return true; return time_after(jiffies, READ_ONCE(mm->mm_cid_next_scan)); } #else /* CONFIG_SCHED_MM_CID */init/Kconfig +2
-0@@ -1860,6 +1860,18 @@ config DEBUG_RSEQ If unsure, say N. +config RSEQ_CID_SCAN_BATCH + int "Number of CPUs to scan at every mm_cid compactionattempt"
init/Kconfig +5
-0+ range 1 NR_CPUS + default 8 + depends on SCHED_MM_CID + help + CPUs are scanned pseudo-periodically to compact the CID ofeach task,
init/Kconfig +1
-0+ this operation can take a longer amount of time on systemswith many
init/Kconfig +1
-0+ CPUs, resulting in higher scheduling latency for thecurrent task.
init/Kconfig +1
-0+ A higher value means the CID is compacted faster, butresults in
init/Kconfig +2
-0+ higher scheduling latency. + config CACHESTAT_SYSCALL bool "Enable cachestat() system call" if EXPERT default ykernel/sched/core.c +3
-1@@ -10591,11 +10591,26 @@ static void sched_mm_cid_remote_clear_weight(struct mm_struct *mm, int cpu, void task_mm_cid_work(struct task_struct *t) { + int weight, cpu, from_cpu, this_batch, next_batch, idx; unsigned long now = jiffies, old_scan, next_scan; struct cpumask *cidmask; -int weight, cpu; struct mm_struct *mm = t->mm; +/* +* This function is called from __rseq_handle_notify_resume, which
kernel/sched/core.c +5
-0+* makes sure t is a user thread and is not exiting. +*/ + this_batch =READ_ONCE (mm->mm_cid_scan_batch); + next_batch = this_batch +1 ; + from_cpu = cpumask_nth(this_batch *
CONFIG_RSEQ_CID_SCAN_BATCH,kernel/sched/core.c +19
-1+ cpu_possible_mask); +if (from_cpu >= nr_cpu_ids) { + from_cpu =0 ; + next_batch =1 ; + } +/* Delay scan only if we are done with all cpus. */ +if (from_cpu !=0 ) + goto cid_compact; old_scan = READ_ONCE(mm->mm_cid_next_scan); next_scan = now + msecs_to_jiffies(MM_CID_SCAN_DELAY); if (!old_scan) {@@ -10611,17 +10626,33 @@ void task_mm_cid_work(struct task_struct *t) return; if (!try_cmpxchg(&mm->mm_cid_next_scan, &old_scan, next_scan)) return; + +cid_compact: +if (!try_cmpxchg (&mm->mm_cid_scan_batch, &this_batch, next_batch)) +return ; cidmask = mm_cidmask(mm); /* Clear cids that were not recently used. */ -for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH ) +break ; sched_mm_cid_remote_clear_old(mm, cpu); + ++idx; + } weight = cpumask_weight(cidmask); /** Clear cids that are greater or equal to the cidmask weight to
* recompact it.
*/kernel/sched/core.c +7
-1-for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH ) +break ; sched_mm_cid_remote_clear_weight(mm, cpu, weight); + ++idx; + } } void init_sched_mm_cid(struct task_struct *t)
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Re: [PATCH v2 3/4] sched: Compact RSEQ concurrency IDs in batches
2025-08-06T16:57:51Z Mathieu Desnoyers <[email protected]>:
Quoted 23 lines · Gabriele Monaco
Quoted 22 lines · Gabriele Monaco
Quoted 21 lines
Currently,
task_mm_cid_work()is called fromresume_user_mode_work(). This can delay the execution of the corresponding thread for the entire duration of the function, negatively affecting the response in case of real time tasks.
In practice, we observetask_mm_cid_workincreasing the latency of 30-35us on a 128 cores system, this order of magnitude is meaningful underPREEMPT_RT.Run the
task_mm_cid_workin batches of up toCONFIG_RSEQ_CID_SCAN_BATCHCPUs, this reduces the duration of the delay for each scan.The
task_mm_cid_workcontains a mechanism to avoid running more frequently than every 100ms. Keep this pseudo-periodicity only on complete scans.
This means each call totask_mm_cid_workreturns prematurely if the period did not elapse and a scan is not ongoing (i.e. the next batch to scan is not the first).
This way full scans are not excessively delayed while still keeping each run, and introduced latency, short.
Quoted 9 lines
Quoted 1 line
Mathieu, would you have some time to look at this implementation?
Hi Gabriele,
Please note that I am currently on vacation. I'll be back shortly before the end of August, but I'm afraid there are other tasks I need to focus on before I can get back to this. I'm adding this review to my todo list for September.
No problem, thanks for the update and enjoy your vacation!
Thanks,
Gabriele
Quoted 167 lines
Thanks,
Mathieu
Quoted 148 lines
Thanks,
GabrieleFixes 223baf9d17f2 ("sched: Fix performance regression introduced byQuoted 145 lines
mm_cid")Signed-off-by Gabriele Monaco <[email protected]>---
include/linux/mm_types.h | 15 +++++++++++++++ init/Kconfig | 12 ++++++++++++ kernel/sched/core.c | 37 ++++++++++++++++++++++++++++++++++---3 files changed, 61 insertions(+), 3 deletions(-)include/linux/mm_types.h +2
-0@@ -995,6 +995,13 @@ struct mm_struct {* When the next mm_cid scan is due (in jiffies). */ unsigned long mm_cid_next_scan; + /* +* @mm_cid_scan_batch: Counter for batch used in the next scan.
+ *
+ * Scan in batches ofCONFIG_RSEQ_CID_SCAN_BATCH. This field
+ * increments at each scan and reset when all batches are done.
+ */
+ unsigned intmm_cid_scan_batch;
/**
* @nr_cpus_allowed: Number of CPUs allowed for mm. *include/linux/mm_types.h +1
-0@@ -1385,6 +1392,7 @@ static inline void mm_init_cid(struct mm_struct *mm, struct task_struct *p) raw_spin_lock_init(&mm->cpus_allowed_lock); cpumask_copy(mm_cpus_allowed(mm), &p->cpus_mask); cpumask_clear(mm_cidmask(mm)); + mm->mm_cid_scan_batch = 0; }static inline int
mm_alloc_cid_noprof(struct mm_struct *mm, struct task_struct *p)include/linux/mm_types.h +0
-0@@ -1423,8 +1431,15 @@ static inline void mm_set_cpus_allowed(struct
mm_struct*mm, const struct cpumas
static inline boolmm_cid_needs_scan(struct mm_struct *mm){
+ unsigned intnext_batch;
+
if (!mm)
return false;
+next_batch=READ_ONCE(mm->mm_cid_scan_batch);
+ /* Always needs scan unless it's the first batch. */ + if (CONFIG_RSEQ_CID_SCAN_BATCH*next_batch<num_possible_cpus()&&
+next_batch)
+ return true;
returntime_after(jiffies, READ_ONCE(mm->mm_cid_next_scan)); }
#else /*CONFIG_SCHED_MM_CID*/init/Kconfig +0
-0@@ -1860,6 +1860,18 @@ config DEBUG_RSEQIf unsure, say N.
+configRSEQ_CID_SCAN_BATCH
+ int "Number of CPUs to scan at everymm_cidcompaction attempt"
+ range 1NR_CPUS
+ default 8
+ depends onSCHED_MM_CID
+ help
+ CPUs are scanned pseudo-periodically to compact the CID of each task,
+ this operation can take a longer amount of time on systems with many
+ CPUs, resulting in higher scheduling latency for the current task.
+ A higher value means the CID is compacted faster, but results in
+ higher scheduling latency.
+
configCACHESTAT_SYSCALL
bool "Enablecachestat()system call" if EXPERT default ykernel/sched/core.c +1
-1@@ -10591,11 +10591,26 @@ static void sched_mm_cid_remote_clear_weight(struct mm_struct *mm, int cpu, void task_mm_cid_work(struct task_struct *t) { + int weight, cpu, from_cpu, this_batch, next_batch, idx; unsigned long now = jiffies, old_scan, next_scan; struct cpumask *cidmask; - int weight, cpu; struct mm_struct *mm = t->mm;+ /*
+ * This function is called from__rseq_handle_notify_resume, which
+ * makes sure t is a user thread and is not exiting. + */
+this_batch=READ_ONCE(mm->mm_cid_scan_batch);
+next_batch=this_batch+ 1;
+from_cpu= cpumask_nth(this_batch *CONFIG_RSEQ_CID_SCAN_BATCH,
+cpu_possible_mask);
+ if (from_cpu>=nr_cpu_ids) {
+from_cpu= 0;
+next_batch= 1;
+ }
+ /* Delay scan only if we are done with all cpus. */ + if (from_cpu!= 0)
+ gotocid_compact;
old_scan=READ_ONCE(mm->mm_cid_next_scan);next_scan= now +msecs_to_jiffies(MM_CID_SCAN_DELAY); if (!old_scan) {kernel/sched/core.c +11
-1@@ -10611,17 +10626,33 @@ void task_mm_cid_work(struct task_struct *t) return; if (!try_cmpxchg(&mm->mm_cid_next_scan, &old_scan, next_scan)) return; + +cid_compact: + if (!try_cmpxchg(&mm->mm_cid_scan_batch, &this_batch, next_batch)) + return; cidmask = mm_cidmask(mm); /* Clear cids that were not recently used. */ - for_each_possible_cpu(cpu) + idx = 0; + cpu = from_cpu; + for_each_cpu_from(cpu, cpu_possible_mask) { + if (idx == CONFIG_RSEQ_CID_SCAN_BATCH) + break; sched_mm_cid_remote_clear_old(mm, cpu); + ++idx; + } weight = cpumask_weight(cidmask); /** Clear cids that are greater or equal to the cidmask weight to
* recompact it.
*/
-for_each_possible_cpu(cpu)
+ idx = 0;
+ cpu =from_cpu;Quoted 9 lines
Quoted 8 lines
+
for_each_cpu_from(cpu, cpu_possible_mask){ + if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH)
+ break;sched_mm_cid_remote_clear_weight(mm, cpu, weight); + ++idx;
+ }
}
voidinit_sched_mm_cid(struct task_struct *t)--
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com
Mathieu Desnoyers
maintainerRe: [PATCH v2 3/4] sched: Compact RSEQ concurrency IDs in batches
In reply to Gabriele MonacoQuoted 11 lines · Gabriele Monaco
Currently,
task_mm_cid_work()is called fromresume_user_mode_work(). This can delay the execution of the corresponding thread for the entire duration of the function, negatively affecting the response in case of real time tasks.
In practice, we observetask_mm_cid_workincreasing the latency of 30-35us on a 128 cores system, this order of magnitude is meaningful underPREEMPT_RT.Run the
task_mm_cid_workin batches of up toCONFIG_RSEQ_CID_SCAN_BATCHCPUs, this reduces the duration of the delay for each scan.
Quoted 8 lines
The
task_mm_cid_workcontains a mechanism to avoid running more frequently than every 100ms. Keep this pseudo-periodicity only on complete scans.
This means each call totask_mm_cid_workreturns prematurely if the period did not elapse and a scan is not ongoing (i.e. the next batch to scan is not the first).
This way full scans are not excessively delayed while still keeping each run, and introduced latency, short.
With your test hardware/workload as reference, do you have an idea of how many CPUs would be needed to require more than 100ms to iterate on all CPUs with the default scan batch size (8) ?
Quoted 111 lines
Fixes 223baf9d17f2 ("sched: Fix performance regression introduced by mm_cid")Signed-off-by Gabriele Monaco <[email protected]>--- include/linux/mm_types.h | 15 +++++++++++++++ init/Kconfig | 12 ++++++++++++ kernel/sched/core.c | 37 ++++++++++++++++++++++++++++++++++--- 3 files changed, 61 insertions(+), 3 deletions(-)include/linux/mm_types.h +15
-0@@ -995,6 +995,13 @@ struct mm_struct {* When the next mm_cid scan is due (in jiffies). */ unsigned long mm_cid_next_scan; +/* +* @mm_cid_scan_batch: Counter for batch used in the next scan. +* +* Scan in batches of CONFIG_RSEQ_CID_SCAN_BATCH. This field +* increments at each scan and reset when all batches are done. +*/ +unsigned int mm_cid_scan_batch;/** * @nr_cpus_allowed: Number of CPUs allowed for mm. * @@ -1385,6 +1392,7 @@ static inline void mm_init_cid(struct mm_struct *mm, struct task_struct *p)raw_spin_lock_init (&mm->cpus_allowed_lock);cpumask_copy (mm_cpus_allowed (mm), &p->cpus_mask);cpumask_clear (mm_cidmask (mm)); + mm->mm_cid_scan_batch =0 ; }static inline int mm_alloc_cid_noprof (struct mm_struct *mm,struct task_struct *p)@@ -1423,8 +1431,15 @@ static inline void mm_set_cpus_allowed(struct mm_struct *mm, const struct cpumasstatic inline bool mm_cid_needs_scan (struct mm_struct *mm) { +unsigned int next_batch; +if (!mm)return false; + next_batch =READ_ONCE (mm->mm_cid_scan_batch); +/* Always needs scan unless it's the first batch. */ +if (CONFIG_RSEQ_CID_SCAN_BATCH * next_batch <num_possible_cpus () && + next_batch) +return true;return time_after (jiffies,READ_ONCE (mm->mm_cid_next_scan)); }#else /* CONFIG_SCHED_MM_CID */ init/Kconfig +12
-0@@ -1860,6 +1860,18 @@ config DEBUG_RSEQ If unsure, say N. +config RSEQ_CID_SCAN_BATCH + int "Number of CPUs to scan at every mm_cid compaction attempt" + range 1 NR_CPUS + default 8 + depends on SCHED_MM_CID + help + CPUs are scanned pseudo-periodically to compact the CID of each task, + this operation can take a longer amount of time on systems with many + CPUs, resulting in higher scheduling latency for the current task. + A higher value means the CID is compacted faster, but results in + higher scheduling latency. + config CACHESTAT_SYSCALL bool "Enable cachestat() system call" if EXPERT default ykernel/sched/core.c +17
-1@@ -10591,11 +10591,26 @@ static void sched_mm_cid_remote_clear_weight(struct mm_struct *mm, int cpu,void task_mm_cid_work (struct task_struct *t) { +int weight, cpu, from_cpu, this_batch, next_batch, idx;unsigned long now = jiffies, old_scan, next_scan;struct cpumask *cidmask; -int weight, cpu;struct mm_struct *mm = t->mm; +/* +* This function is called from __rseq_handle_notify_resume, which +* makes sure t is a user thread and is not exiting. +*/ + this_batch =READ_ONCE (mm->mm_cid_scan_batch); + next_batch = this_batch +1 ; + from_cpu =cpumask_nth (this_batch *CONFIG_RSEQ_CID_SCAN_BATCH , + cpu_possible_mask); +if (from_cpu >= nr_cpu_ids) { + from_cpu =0 ; + next_batch =1 ; + } +/* Delay scan only if we are done with all cpus. */ +if (from_cpu !=0 ) + goto cid_compact; old_scan =READ_ONCE (mm->mm_cid_next_scan); next_scan = now +msecs_to_jiffies (MM_CID_SCAN_DELAY );if (!old_scan) {@@ -10611,17 +10626,33 @@ void task_mm_cid_work(struct task_struct *t)return ;if (!try_cmpxchg (&mm->mm_cid_next_scan, &old_scan, next_scan))return ; +
kernel/sched/core.c void task_mm_cid_work(struct task_struct *t) · line 10638
Quoted 10 lines
kernel/sched/core.c +7
-1+cid_compact: +if (!try_cmpxchg (&mm->mm_cid_scan_batch, &this_batch, next_batch)) +return ; cidmask =mm_cidmask (mm);/* Clear cids that were not recently used. */ -for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH )
could do "if (idx++ == CONFIG_RSEQ_CID_SCAN_BATCH)"
kernel/sched/core.c void task_mm_cid_work(struct task_struct *t) · line 10641
Quoted 3 lines
kernel/sched/core.c +2
-0+break ;sched_mm_cid_remote_clear_old (mm, cpu); + ++idx;
and remove this ^
kernel/sched/core.c void task_mm_cid_work(struct task_struct *t) · line 10651
Quoted 11 lines
kernel/sched/core.c +5
-1+ } weight =cpumask_weight (cidmask);/* * Clear cids that are greater or equal to the cidmask weight to * recompact it. */ -for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH )
Likewise.
kernel/sched/core.c void task_mm_cid_work(struct task_struct *t) · line 10654
Quoted 3 lines
kernel/sched/core.c +2
-0+break ;sched_mm_cid_remote_clear_weight (mm, cpu, weight); + ++idx;
Likewise.
Thanks,
Mathieu
Quoted 4 lines
kernel/sched/core.c +1
-0+ } }void init_sched_mm_cid (struct task_struct *t)
Signature · 3 lines
Mathieu Desnoyers EfficiOS Inc. https://www.efficios.com
Re: [PATCH v2 3/4] sched: Compact RSEQ concurrency IDs in batches
Quoted 23 lines · Mathieu Desnoyers
Quoted 13 lines · Gabriele Monaco
Currently,
task_mm_cid_work()is called fromresume_user_mode_work().
This can delay the execution of the corresponding thread for the entire duration of the function, negatively affecting the response in case of real time tasks.
In practice, we observetask_mm_cid_workincreasing the latency of 30-35us on a 128 cores system, this order of magnitude is meaningful underPREEMPT_RT.Run the
task_mm_cid_workin batches of up toCONFIG_RSEQ_CID_SCAN_BATCHCPUs, this reduces the duration of the delay for each scan.Quoted 8 lines
The
task_mm_cid_workcontains a mechanism to avoid running more frequently than every 100ms. Keep this pseudo-periodicity only on complete scans.
This means each call totask_mm_cid_workreturns prematurely if the period did not elapse and a scan is not ongoing (i.e. the next batch to scan is not the first).
This way full scans are not excessively delayed while still keeping each run, and introduced latency, short.
Quoted 3 lines
With your test hardware/workload as reference, do you have an idea of how many CPUs would be needed to require more than 100ms to iterate on all CPUs with the default scan batch size (8) ?
As you guessed, this is strongly dependent on the workload, where workloads with less threads are more likely to take longer. I used cyclictest (threads with 100us period) and hackbench (processes) on a 128 CPUs machine and measured the time to complete the scan (16 iterations) as well as the time between non-complete scans (not delayed by 100ms):
cyclictest: delay 0-400 us , complete scan 1.5-2 ms
hackbench: delay 5us - 3ms , complete scan 1.5-15 ms
So to answer your question, in the observed worst case for hackbench, it would take more than 800 CPUs to reach the 100ms limit.
That said, the problematic latency was observed on a full scan (128 CPUs), so perhaps the default of 8 is a bit too conservative and could easily be doubled.
Measurements showed these durations for each call to task_mm_cid_scan:
batch size 8: 1-11 us (majority below 10) batch size 16: 3-16 us (majority below 10) batch size 32: 10-21 us (majority above 15)
20 us is considered a relevant latency on this machine, so 16 seems a good tradeoff for a batch size to me.
I'm going to include those numbers in the next iteration of the series.
...
Quoted 35 lines
kernel/sched/core.c void task_mm_cid_work(struct task_struct *t) · line 10638
Quoted 11 lines
(quoted diff) +2
-0+cid_compact: + if (!try_cmpxchg(&mm->mm_cid_scan_batch, &this_batch,
next_batch))kernel/sched/core.c +1
-0+return ;cidmask =
mm_cidmask(mm);
/* Clear cids that were not recently used. */kernel/sched/core.c +4
-1-for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH )could do "if (idx++ ==
CONFIG_RSEQ_CID_SCAN_BATCH)"Review
Quoted 3 lines
(quoted diff) +1
-0+ break;
sched_mm_cid_remote_clear_old(mm, cpu);(quoted diff) +1
-0+ ++idx;and remove this ^
kernel/sched/core.c void task_mm_cid_work(struct task_struct *t) · line 10638
Quoted 12 lines
(quoted diff) +1
-0+ }weight =
cpumask_weight(cidmask);
/*
* Clear cids that are greater or equal to the cidmask weight to
* recompact it.
*/kernel/sched/core.c +4
-1-for_each_possible_cpu (cpu) + idx =0 ; + cpu = from_cpu; +for_each_cpu_from (cpu, cpu_possible_mask) { +if (idx ==CONFIG_RSEQ_CID_SCAN_BATCH )Likewise.
Quoted 5 lines
Review
Quoted 3 lines
(quoted diff) +1
-0+ break;
sched_mm_cid_remote_clear_weight(mm, cpu, weight);(quoted diff) +1
-0+ ++idx;Likewise.
Sure, will do.
Thanks,
Gabriele
PATCH v2 4/4 selftests/rseq: Add test for mm_cid compaction
In reply to Gabriele MonacoA task in the kernel (task_mm_cid_work) runs somewhat periodically to compact the mm_cid for each process. Add a test to validate that it runs correctly and timely.
The test spawns 1 thread pinned to each CPU, then each thread, including the main one, runs in short bursts for some time. During this period, the mm_cids should be spanning all numbers between 0 and nproc.
At the end of this phase, a thread with high enough mm_cid (>= nproc/2) is selected to be the new leader, all other threads terminate.
After some time, the only running thread should see 0 as mm_cid, if that doesn't happen, the compaction mechanism didn't work and the test fails.
The test never fails if only 1 core is available, in which case, we cannot test anything as the only available mm_cid is 0.
--- tools/testing/selftests/rseq/.gitignore | 1 + tools/testing/selftests/rseq/Makefile | 2 +- .../selftests/rseq/mm_cid_compaction_test.c | 204 ++++++++++++++++++ 3 files changed, 206 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/rseq/mm_cid_compaction_test.c
tools/testing/selftests/rseq/.gitignore +1 -0
@@ -3,6 +3,7 @@ basic_percpu_ops_test basic_percpu_ops_mm_cid_test basic_test basic_rseq_op_test +mm_cid_compaction_test param_test param_test_benchmark param_test_compare_twice
tools/testing/selftests/rseq/Makefile +1 -1
@@ -17,7 +17,7 @@ OVERRIDE_TARGETS = 1 TEST_GEN_PROGS = basic_test basic_percpu_ops_test basic_percpu_ops_mm_cid_test param_test \ param_test_benchmark param_test_compare_twice param_test_mm_cid \ param_test_mm_cid_benchmark param_test_mm_cid_compare_twice \ - syscall_errors_test + syscall_errors_test mm_cid_compaction_test TEST_GEN_PROGS_EXTENDED = librseq.so
tools/testing/selftests/rseq/mm_cid_compaction_test.c new file +204 -0
204 lines not on this page · raw message
Signature · 1 line
2.50.1