sched/numa: Fix NULL pointer access to mm_struct durng task swap
PATCH 1/1 sched/numa: Fix NULL pointer access to mm_struct durng task swap
It was reported that after Commit ad6b26b6a0a7 ("sched/numa: add statistics of numa balance task"), a NULL pointer exception[1] occurs when accessing p->mm. The following race condition was found to trigger this bug: After a swap task candidate is chosen during NUMA balancing, its mm_struct is released due to task exit. Later, when the task swapping is performed, p->mm is NULL, which causes the problem:
CPU0 CPU1
:
...task_numa_migrate
task_numa_find_cpu
task_numa_compare
# a normal task p is chosen
env->best_task = p
# p exit:
exit_signals(p);
p->flags |= PF_EXITING
exit_mm
p->mm = NULL;
migrate_swap_stop
__migrate_swap_task((arg->src_task, arg->dst_cpu)
count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULLFix this issue by checking if the task has the PF_EXITING flag set in migrate_swap_stop(). If it does, skip updating the memcg events. Additionally, log a warning if p->mm is NULL to facilitate future debugging.
--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
kernel/sched/core.c +8 -1
@@ -3364,7 +3364,14 @@ static void __migrate_swap_task(struct task_struct *p, int cpu) {__schedstat_inc (p->stats.numa_task_swapped);count_vm_numa_event (NUMA_TASK_SWAP ); -count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); +/* exiting task has NULL mm */ +if (!(p->flags &PF_EXITING )) { +WARN_ONCE (!p->mm,"swap task %d %s %x has no mm\n" , + p->pid, p->comm, p->flags); + +if (p->mm) +count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); + }if (task_on_rq_queued (p)) {struct rq *src_rq, *dst_rq;
Signature · 1 line
2.25.1
Quoted 41 lines · On Thu, 3 Jul 2025 00:32:47 +0800 Chen Yu <[email protected]> wrote:
It was reported that after Commit
ad6b26b6a0a7("sched/numa: add statistics of numa balance task"), a NULL pointer exception[1] occurs when accessingp->mm. The following race condition was found to trigger this bug: After a swap task candidate is chosen during NUMA balancing, itsmm_structis released due to task exit. Later, when the task swapping is performed,p->mmis NULL, which causes the problem:CPU0 CPU1
:
...task_numa_migrate
task_numa_find_cputask_numa_compare # a normal task p is chosen env->best_task = p # p exit: exit_signals(p); p->flags |= PF_EXITING exit_mm p->mm = NULL; migrate_swap_stop __migrate_swap_task((arg->src_task, arg->dst_cpu) count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULLFix this issue by checking if the task has the
PF_EXITINGflag set inmigrate_swap_stop(). If it does, skip updating the memcg events. Additionally, log a warning ifp->mmis NULL to facilitate future debugging....
kernel/sched/core.c +0
-0@@ -3364,7 +3364,14 @@ static void __migrate_swap_task(struct task_struct *p, int cpu) {__schedstat_inc (p->stats.numa_task_swapped);
kernel/sched/core.c static void __migrate_swap_task(struct task_struct *p, int cpu) · line 3374
Quoted 10 lines
kernel/sched/core.c +8
-1count_vm_numa_event (NUMA_TASK_SWAP ); -count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); +/* exiting task has NULL mm */ +if (!(p->flags &PF_EXITING )) { +WARN_ONCE (!p->mm,"swap task %d %s %x has no mm\n" , + p->pid, p->comm, p->flags); + +if (p->mm) +count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); + }
I don't think we should warn on a condition which is known to occur and which we successfully handle. What action can anyone take upon that warning?
Which means the change might as well become
+ /* comment goes here */
+ if (p->mm)
+ count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);
But is that a real fix? Can the other thread call exit(), set PF_EXITING and null its p->mm right between the above two lines? After the p->mm test and before the count_memcg_event_mm() call?
IOW, there needs to be some locking in place to stabilize p->mm throughout the p->mm test and the count_memcg_event_mm() call?
Hi Andrew,
Quoted 64 lines · Andrew Morton
Quoted 41 lines · On Thu, 3 Jul 2025 00:32:47 +0800 Chen Yu <[email protected]> wrote:
It was reported that after Commit
ad6b26b6a0a7("sched/numa: add statistics of numa balance task"), a NULL pointer exception[1] occurs when accessingp->mm. The following race condition was found to trigger this bug: After a swap task candidate is chosen during NUMA balancing, itsmm_structis released due to task exit. Later, when the task swapping is performed,p->mmis NULL, which causes the problem:CPU0 CPU1
:
...task_numa_migratetask_numa_find_cpu task_numa_compare # a normal task p is chosen env->best_task = p # p exit: exit_signals(p); p->flags |= PF_EXITING exit_mm p->mm = NULL; migrate_swap_stop __migrate_swap_task((arg->src_task, arg->dst_cpu) count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULLFix this issue by checking if the task has the
PF_EXITINGflag set inmigrate_swap_stop(). If it does, skip updating the memcg events. Additionally, log a warning ifp->mmis NULL to facilitate future debugging....
kernel/sched/core.c +0
-0@@ -3364,7 +3364,14 @@ static void __migrate_swap_task(struct task_struct *p, int cpu) {__schedstat_inc (p->stats.numa_task_swapped);kernel/sched/core.c static void __migrate_swap_task(struct task_struct *p, int cpu) · line 3374
Quoted 10 lines
kernel/sched/core.c +8
-1count_vm_numa_event (NUMA_TASK_SWAP ); -count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); +/* exiting task has NULL mm */ +if (!(p->flags &PF_EXITING )) { +WARN_ONCE (!p->mm,"swap task %d %s %x has no mm\n" , + p->pid, p->comm, p->flags); + +if (p->mm) +count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); + }I don't think we should warn on a condition which is known to occur and which we successfully handle. What action can anyone take upon that warning?
Which means the change might as well become
kernel/sched/core.c +3
-0+/* comment goes here */ +if (p->mm) +count_memcg_event_mm (p->mm,NUMA_TASK_SWAP );
Quoted 4 lines
But is that a real fix? Can the other thread call
exit(), setPF_EXITINGand null itsp->mmright between the above two lines? After thep->mmtest and before thecount_memcg_event_mm()call?
Thank you for the scenario description. Right, there is still a race condition although the p->pi_lock is hold when reaching here, the p->alloc_lock should be hold to access p->mm.
Quoted 2 lines
IOW, there needs to be some locking in place to stabilize
p->mmthroughout thep->mmtest and thecount_memcg_event_mm()call?
Yes, Michal and Peter have given some feedback on how to do this.
thanks,
Chenyu
Quoted 50 lines · Chen Yu
It was reported that after Commit
ad6b26b6a0a7("sched/numa: add statistics of numa balance task"), a NULL pointer exception[1] occurs when accessingp->mm. The following race condition was found to trigger this bug: After a swap task candidate is chosen during NUMA balancing, itsmm_structis released due to task exit. Later, when the task swapping is performed,p->mmis NULL, which causes the problem:CPU0 CPU1
:
...task_numa_migrate
task_numa_find_cputask_numa_compare # a normal task p is chosen env->best_task = p # p exit: exit_signals(p); p->flags |= PF_EXITING exit_mm p->mm = NULL; migrate_swap_stop __migrate_swap_task((arg->src_task, arg->dst_cpu) count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULLFix this issue by checking if the task has the
PF_EXITINGflag set inmigrate_swap_stop(). If it does, skip updating the memcg events. Additionally, log a warning ifp->mmis NULL to facilitate future debugging.Fixes ad6b26b6a0a7 ("sched/numa: add statistics of numa balance task")Reported-by Jirka Hladky <[email protected]>Closes https://lore.kernel.org/all/CAE4VaGBLJxpd=NeRJXpSCuw=REhC5LWJpC29kDy-Zh2ZDyzQZA@mail.gmail.com/ open hereReported-by Srikanth Aithal <[email protected]>Reported-by Suneeth D <[email protected]>Suggested-by Libo Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)kernel/sched/core.c +0
-0
kernel/sched/core.c static void __migrate_swap_task(struct task_struct *p, int cpu) · line 3370
Quoted 8 lines
kernel/sched/core.c +4
-1@@ -3364,7 +3364,14 @@ static void __migrate_swap_task(struct task_struct *p, int cpu) {__schedstat_inc (p->stats.numa_task_swapped);count_vm_numa_event (NUMA_TASK_SWAP ); -count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); +/* exiting task has NULL mm */ +if (!(p->flags &PF_EXITING )) { +WARN_ONCE (!p->mm,"swap task %d %s %x has no mm\n" , + p->pid, p->comm, p->flags);
As Andrew already said this is not really acceptable because this is very likely too easy to trigger and a) you do not want logs flooded with warnings and also there are setups with panic_on_warn configured and for those this would be a fatal situation without any good reason.
kernel/sched/core.c static void __migrate_swap_task(struct task_struct *p, int cpu) · line 3374
Quoted 4 lines
kernel/sched/core.c +4
-0+ +if (p->mm) +count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); + }
Why are you testing for p->mm here? Isn't PF_EXITING test sufficient? A robust way to guarantee non-NULL mm against races when a task is exiting is find_lock_task_mm. Probably too heavy weight for this path.
Quoted 6 lines
kernel/sched/core.c +0
-0if (task_on_rq_queued (p)) {struct rq *src_rq, *dst_rq;Signature · 1 line
2.25.1
Signature · 2 lines
Michal Hocko SUSE Labs
Hi Michal,
Quoted 56 lines · Michal Hocko
Quoted 55 lines · Chen Yu
It was reported that after Commit
ad6b26b6a0a7("sched/numa: add statistics of numa balance task"), a NULL pointer exception[1] occurs when accessingp->mm. The following race condition was found to trigger this bug: After a swap task candidate is chosen during NUMA balancing, itsmm_structis released due to task exit. Later, when the task swapping is performed,p->mmis NULL, which causes the problem:CPU0 CPU1
:
...task_numa_migratetask_numa_find_cpu task_numa_compare # a normal task p is chosen env->best_task = p # p exit: exit_signals(p); p->flags |= PF_EXITING exit_mm p->mm = NULL; migrate_swap_stop __migrate_swap_task((arg->src_task, arg->dst_cpu) count_memcg_event_mm(p->mm, NUMA_TASK_SWAP)# p->mm is NULLFix this issue by checking if the task has the
PF_EXITINGflag set inmigrate_swap_stop(). If it does, skip updating the memcg events. Additionally, log a warning ifp->mmis NULL to facilitate future debugging.Fixes ad6b26b6a0a7 ("sched/numa: add statistics of numa balance task")Reported-by Jirka Hladky <[email protected]>Closes https://lore.kernel.org/all/CAE4VaGBLJxpd=NeRJXpSCuw=REhC5LWJpC29kDy-Zh2ZDyzQZA@mail.gmail.com/ open hereReported-by Srikanth Aithal <[email protected]>Reported-by Suneeth D <[email protected]>Suggested-by Libo Chen <[email protected]>Signed-off-by Chen Yu <[email protected]>--- kernel/sched/core.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)kernel/sched/core.c +1
-1@@ -3364,7 +3364,14 @@ static void __migrate_swap_task(struct task_struct *p, int cpu) {__schedstat_inc (p->stats.numa_task_swapped);count_vm_numa_event (NUMA_TASK_SWAP ); -count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); +/* exiting task has NULL mm */
Quoted 9 lines
kernel/sched/core.c static void __migrate_swap_task(struct task_struct *p, int cpu) · line 3370
Quoted 3 lines
kernel/sched/core.c +3
-0+if (!(p->flags &PF_EXITING )) { +WARN_ONCE (!p->mm,"swap task %d %s %x has no mm\n" , + p->pid, p->comm, p->flags);As Andrew already said this is not really acceptable because this is very likely too easy to trigger and a) you do not want logs flooded with warnings and also there are setups with
panic_on_warnconfigured and for those this would be a fatal situation without any good reason.
OK, got it, thanks for pointing it out.
Quoted 8 lines
kernel/sched/core.c static void __migrate_swap_task(struct task_struct *p, int cpu) · line 3374
Quoted 4 lines
kernel/sched/core.c +4
-0+ +if (p->mm) +count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); + }Why are you testing for
p->mmhere? Isn'tPF_EXITINGtest sufficient? A robust way to guarantee non-NULL mm against races when a task is exiting isfind_lock_task_mm. Probably too heavy weight for this path.
I suppose we might only need to grab task_lock(p), check if its mm pointer is NULL. If yes, we skip the update of memcg event without scanning for a non-NULL mm within the process(as find_lock_task_mm() does)? If the mm is non-NULL, we update the memcg event with task_lock(p) hold and releases it later.
thanks,
Chenyu
Quoted 7 lines
Quoted 6 lines
kernel/sched/core.c +0
-0if (task_on_rq_queued (p)) {struct rq *src_rq, *dst_rq;Signature · 1 line
2.25.1
Quoted 4 lines · Yu C
Hi Michal,
Quoted 1 line · Michal Hocko
On Thu 03-07-25 00:32:47, Chen Yu wrote:
[...]
Quoted 9 lines
Quoted 8 lines
kernel/sched/core.c static void __migrate_swap_task(struct task_struct *p, int cpu) · line 3374
Quoted 4 lines
kernel/sched/core.c +4
-0+ +if (p->mm) +count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); + }Why are you testing for
p->mmhere? Isn'tPF_EXITINGtest sufficient? A robust way to guarantee non-NULL mm against races when a task is exiting isfind_lock_task_mm. Probably too heavy weight for this path.
Quoted 5 lines
I suppose we might only need to grab
task_lock(p), check if its mm pointer is NULL. If yes, we skip the update of memcg event without scanning for a non-NULL mm within theprocess(as find_lock_task_mm() does)? If the mm is non-NULL, we update the memcg event withtask_lock(p)hold and releases it later.
Why not use find_lock_task_mm if task_lock is acceptable for this code path?
Signature · 2 lines
Michal Hocko SUSE Labs
Peter Zijlstra
maintainerQuoted 2 lines · Michal Hocko
Why not use
find_lock_task_mmiftask_lockis acceptable for this code path?
Having to take locks just to count a number nobody is ever going to look at is silly. Iterating the whole thread group (might be hundreds or thousands of tasks) just to maybe find one when racing with a fatal signal is even more silly.
Peter Zijlstra
maintainerQuoted 7 lines · Chen Yu
kernel/sched/core.c +0
-0@@ -3364,7 +3364,14 @@ static void __migrate_swap_task(struct task_struct *p, int cpu) {__schedstat_inc (p->stats.numa_task_swapped);
kernel/sched/core.c static void __migrate_swap_task(struct task_struct *p, int cpu) · line 3374
Quoted 10 lines
kernel/sched/core.c +8
-1count_vm_numa_event (NUMA_TASK_SWAP ); -count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); +/* exiting task has NULL mm */ +if (!(p->flags &PF_EXITING )) { +WARN_ONCE (!p->mm,"swap task %d %s %x has no mm\n" , + p->pid, p->comm, p->flags); + +if (p->mm) +count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); + }
Aside from the things already mentioned by Andrew and Michal; why not simply do something like:
include/linux/memcontrol.h +1 -1
@@ -969,7 +969,7 @@ static inline void count_memcg_events_mm(struct mm_struct *mm, {struct mem_cgroup *memcg; -if (mem_cgroup_disabled ()) +if (mem_cgroup_disabled () || !mm)return ;rcu_read_lock ();
Quoted 28 lines · Peter Zijlstra
Quoted 7 lines · Chen Yu
kernel/sched/core.c +0
-0@@ -3364,7 +3364,14 @@ static void __migrate_swap_task(struct task_struct *p, int cpu) {__schedstat_inc (p->stats.numa_task_swapped);kernel/sched/core.c static void __migrate_swap_task(struct task_struct *p, int cpu) · line 3374
Quoted 10 lines
kernel/sched/core.c +8
-1count_vm_numa_event (NUMA_TASK_SWAP ); -count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); +/* exiting task has NULL mm */ +if (!(p->flags &PF_EXITING )) { +WARN_ONCE (!p->mm,"swap task %d %s %x has no mm\n" , + p->pid, p->comm, p->flags); + +if (p->mm) +count_memcg_event_mm (p->mm,NUMA_TASK_SWAP ); + }Aside from the things already mentioned by Andrew and Michal; why not simply do something like:
include/linux/memcontrol.h +0
-0
include/linux/memcontrol.h static inline void count_memcg_events_mm(struct mm_struct *mm, · line 973
Quoted 6 lines
include/linux/memcontrol.h +1
-1@@ -969,7 +969,7 @@ static inline void count_memcg_events_mm(struct mm_struct *mm, {struct mem_cgroup *memcg; -if (mem_cgroup_disabled ()) +if (mem_cgroup_disabled () || !mm)return ;
This would imply mm check for all other users that know their mm is valid as they are operating on vma->mm or current task.
But thinking about this some more, this would be racy same as the PF_EXITING check. This is not my area but is this performance sensitive path that couldn't live with the proper find_lock_task_mm?
I do not see other race free way to deal with a remote task exit race.
Signature · 2 lines
Michal Hocko SUSE Labs
Peter Zijlstra
maintainerQuoted 3 lines · Michal Hocko
But thinking about this some more, this would be racy same as the
PF_EXITINGcheck. This is not my area but is this performance sensitive path that couldn't live with the properfind_lock_task_mm?
find_lock_task_mm() seems eminently unsuitable for accounting -- iterating the task group is insane.
Looking at this, the mm_struct lifetimes suck.. task_struct reference doesn't help, rcu doesn't help :-(
Also, whatever the solution it needs to be inside this count_memcg_*() nonsense, because nobody wants this overhead, esp. not for something daft like accounting.
My primary desire at this point is to just revert the patch that caused this. Accounting just isn't worth it. Esp. not since there is already a tracepoint in this path -- people that want to count crap can very well get their numbers from that.
Quoted 12 lines · Peter Zijlstra
Quoted 3 lines · Michal Hocko
But thinking about this some more, this would be racy same as the
PF_EXITINGcheck. This is not my area but is this performance sensitive path that couldn't live with the properfind_lock_task_mm?
find_lock_task_mm()seems eminently unsuitable for accounting -- iterating the task group is insane.Looking at this, the
mm_structlifetimes suck..task_structreference doesn't help, rcu doesn't help :-(
Quoted 8 lines
Also, whatever the solution it needs to be inside this count_memcg_*() nonsense, because nobody wants this overhead, esp. not for something daft like accounting.
My primary desire at this point is to just revert the patch that caused this. Accounting just isn't worth it. Esp. not since there is already a tracepoint in this path -- people that want to count crap can very well get their numbers from that.
I would tend to agree with this. Doing the accounting race free on a remote task is nasty and if this is a rare event that could be avoided then it should be just dropped than racy and oops prone.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 17 lines · Michal Hocko
Quoted 16 lines · Peter Zijlstra
Quoted 3 lines · Michal Hocko
But thinking about this some more, this would be racy same as the
PF_EXITINGcheck. This is not my area but is this performance sensitive path that couldn't live with the properfind_lock_task_mm?
find_lock_task_mm()seems eminently unsuitable for accounting -- iterating the task group is insane.Looking at this, the
mm_structlifetimes suck..task_structreference doesn't help, rcu doesn't help :-(Also, whatever the solution it needs to be inside this count_memcg_*() nonsense, because nobody wants this overhead, esp. not for something daft like accounting.
Quoted 9 lines
Quoted 4 lines
My primary desire at this point is to just revert the patch that caused this. Accounting just isn't worth it. Esp. not since there is already a tracepoint in this path -- people that want to count crap can very well get their numbers from that.
I would tend to agree with this. Doing the accounting race free on a remote task is nasty and if this is a rare event that could be avoided then it should be just dropped than racy and oops prone.
OK, Michal and Peter,
how about keeping the per task schedstat and drop the memcg statistics? The user can still get the per task information without having to filter the ftrace log.
thanks,
Chenyu
Quoted 28 lines · Yu C
Quoted 17 lines · Michal Hocko
Quoted 16 lines · Peter Zijlstra
Quoted 3 lines · Michal Hocko
But thinking about this some more, this would be racy same as the
PF_EXITINGcheck. This is not my area but is this performance sensitive path that couldn't live with the properfind_lock_task_mm?
find_lock_task_mm()seems eminently unsuitable for accounting -- iterating the task group is insane.Looking at this, the
mm_structlifetimes suck..task_structreference doesn't help, rcu doesn't help :-(Also, whatever the solution it needs to be inside this count_memcg_*() nonsense, because nobody wants this overhead, esp. not for something daft like accounting.
Quoted 9 lines
Quoted 4 lines
My primary desire at this point is to just revert the patch that caused this. Accounting just isn't worth it. Esp. not since there is already a tracepoint in this path -- people that want to count crap can very well get their numbers from that.
I would tend to agree with this. Doing the accounting race free on a remote task is nasty and if this is a rare event that could be avoided then it should be just dropped than racy and oops prone.
Quoted 5 lines
OK, Michal and Peter,
how about keeping the per task schedstat and drop the memcg statistics? The user can still get the per task information without having to filter the ftrace log.
I agree. The other parts, schedstat and vmstat, are still quite helpful. Also tracepoints are more expensive than counters once enabled, I think that's too much for just counting numbers.
Libo
Quoted 2 lines
thanks,
Chenyu
Peter Zijlstra
maintainerQuoted 3 lines · Libo Chen
I agree. The other parts, schedstat and vmstat, are still quite helpful. Also tracepoints are more expensive than counters once enabled, I think that's too much for just counting numbers.
I'm not generally a fan of eBPF, but supposedly it is really good for stuff like this.
Attaching to a tracepoint and distributing into cgroup buckets seems like it should be a trivial script.
Hi Peter,
Quoted 11 lines · Peter Zijlstra
Quoted 3 lines · Libo Chen
I agree. The other parts, schedstat and vmstat, are still quite helpful. Also tracepoints are more expensive than counters once enabled, I think that's too much for just counting numbers.
I'm not generally a fan of eBPF, but supposedly it is really good for stuff like this.
Attaching to a tracepoint and distributing into cgroup buckets seems like it should be a trivial script.
Yes, it is feasible to use eBPF. On the other hand, if some existing monitoring programs rely on /proc/{pid}/sched to observe the NUMA balancing metrics of processes, it might be helpful to include the NUMA migration/swap information in /proc/{pid}/sched. This approach can minimize the modifications needed for these monitoring programs, eliminating the need to add a new BPF script to obtain NUMA balancing statistics from different sources IMHO.
thanks,
Chenyu
Peter Zijlstra
maintainerQuoted 15 lines · Yu C
Hi Peter,
Quoted 11 lines · Peter Zijlstra
Quoted 3 lines · Libo Chen
I agree. The other parts, schedstat and vmstat, are still quite helpful. Also tracepoints are more expensive than counters once enabled, I think that's too much for just counting numbers.
I'm not generally a fan of eBPF, but supposedly it is really good for stuff like this.
Attaching to a tracepoint and distributing into cgroup buckets seems like it should be a trivial script.
Quoted 7 lines
Yes, it is feasible to use eBPF. On the other hand, if some existing monitoring programs rely on
/proc/{pid}/schedto observe the NUMA balancing metrics of processes, it might be helpful to include the NUMA migration/swap information in/proc/{pid}/sched. This approach can minimize the modifications needed for these monitoring programs, eliminating the need to add a new BPF script to obtain NUMA balancing statistics from different sources IMHO.
Maybe...
The thing is, most of the time the effort spend on collecting all these numbers is wasted energy since nobody ever looks at them.
Sometimes we're stuck with ABI, like the proc files you mentioned. We can't readily remove them, stuff would break. But does that mean we should endlessly add to them just because convenient?
Ideally I would strip out all the statistics and accounting crap and make sure we have tracepoints (not trace-events) covering all the needed spots, and then maybe just maybe have a few kernel modules that hook into those tracepoints to provide the legacy interfaces.
That way, only the people that care get to pay the overhead of actually collecting the numbers.
One can dream I suppose... :-)
Quoted 24 lines · Peter Zijlstra
Quoted 15 lines · Yu C
Hi Peter,
Quoted 11 lines · Peter Zijlstra
Quoted 3 lines · Libo Chen
I agree. The other parts, schedstat and vmstat, are still quite helpful. Also tracepoints are more expensive than counters once enabled, I think that's too much for just counting numbers.
I'm not generally a fan of eBPF, but supposedly it is really good for stuff like this.
Attaching to a tracepoint and distributing into cgroup buckets seems like it should be a trivial script.
Quoted 7 lines
Yes, it is feasible to use eBPF. On the other hand, if some existing monitoring programs rely on
/proc/{pid}/schedto observe the NUMA balancing metrics of processes, it might be helpful to include the NUMA migration/swap information in/proc/{pid}/sched. This approach can minimize the modifications needed for these monitoring programs, eliminating the need to add a new BPF script to obtain NUMA balancing statistics from different sources IMHO.
Quoted 5 lines
Maybe...
The thing is, most of the time the effort spend on collecting all these numbers is wasted energy since nobody ever looks at them.
As for per-task NUMA balancing activity itself, we found it useful for debugging when trying to ensure that cache-aware load balancing coexists properly with NUMA balancing.
Quoted 4 lines
Sometimes we're stuck with ABI, like the proc files you mentioned. We can't readily remove them, stuff would break. But does that mean we should endlessly add to them just because convenient?
Quoted 8 lines
Ideally I would strip out all the statistics and accounting crap and> make sure we have tracepoints (not trace-events) covering all the needed spots, and then maybe just maybe have a few kernel modules that hook
into those tracepoints to provide the legacy interfaces.That way, only the people that care get to pay the overhead of actually collecting the numbers.
One can dream I suppose... :-)
I see.
If I understand correctly, it's generally not recommended to add new items under /proc. Users are recommended to use tracepoints/events when trying to collect the statistics, and something like schedstat_inc() should be avoided. Is the per-task data an exception? We recently exposed task's slice via /proc/pid/sched : D
thanks,
Chenyu
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Libo Chen
I agree. The other parts, schedstat and vmstat, are still quite helpful. Also tracepoints are more expensive than counters once enabled, I think that's too much for just counting numbers.
I'm not generally a fan of eBPF, but supposedly it is really good for stuff like this.
Yeah but not nearly as good as, for example, __schedstat_inc(var) which probably only takes a few CPU cycles if var is in the right place. eBPF is gonna take a whole bunch of sequences to even get to updating an eBPF map which itself is much more expensive than __schedstat_inc(var).
For one, __migrate_swap_task() happens when dst node is fully busy (most likely src node is full as well), so the overhead of ebpf could be quite noticeable.
Libo
Quoted 2 lines
Attaching to a tracepoint and distributing into cgroup buckets seems like it should be a trivial script.
Peter Zijlstra
maintainerQuoted 13 lines · Libo Chen
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Libo Chen
I agree. The other parts, schedstat and vmstat, are still quite helpful. Also tracepoints are more expensive than counters once enabled, I think that's too much for just counting numbers.
I'm not generally a fan of eBPF, but supposedly it is really good for stuff like this.
Quoted 8 lines
Yeah but not nearly as good as, for example,
__schedstat_inc(var)which probably only takes a few CPU cycles if var is in the right place. eBPF is gonna take a whole bunch of sequences to even get to updating an eBPF map which itself is much more expensive than__schedstat_inc(var).For one,
__migrate_swap_task()happens when dst node is fully busy (most likely src node is full as well), so the overhead of ebpf could be quite noticeable.
But that overhead is only paid if you actually care about the numbers; most people don't.
We already stick static branches in many of the accounting paths -- because we know they hurt.
But look at this:
__schedstat_inc(p->stats.numa_task_swapped);
count_vm_numa_event(NUMA_TASK_SWAP);
count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);that is _3_ different counters, 3 cachelines touched. For what?
Would not a single:
trace_numa_task_swap_tp(p);
be much saner? It translates into a single no-op; no lines touched. Only when someone wants the numbers do we attach to the tracepoint and start collecting things.
Is the collecting more expensive; maybe. But the rest of us will be better of, no?
Quoted 39 lines · Peter Zijlstra
Quoted 13 lines · Libo Chen
Quoted 9 lines · Peter Zijlstra
Quoted 3 lines · Libo Chen
I agree. The other parts, schedstat and vmstat, are still quite helpful. Also tracepoints are more expensive than counters once enabled, I think that's too much for just counting numbers.
I'm not generally a fan of eBPF, but supposedly it is really good for stuff like this.
Quoted 8 lines
Yeah but not nearly as good as, for example,
__schedstat_inc(var)which probably only takes a few CPU cycles if var is in the right place. eBPF is gonna take a whole bunch of sequences to even get to updating an eBPF map which itself is much more expensive than__schedstat_inc(var).For one,
__migrate_swap_task()happens when dst node is fully busy (most likely src node is full as well), so the overhead of ebpf could be quite noticeable.But that overhead is only paid if you actually care about the numbers; most people don't.
We already stick static branches in many of the accounting paths -- because we know they hurt.
But look at this:
__schedstat_inc(p->stats.numa_task_swapped); count_vm_numa_event(NUMA_TASK_SWAP); count_memcg_event_mm(p->mm, NUMA_TASK_SWAP);that is _3_ different counters, 3 cachelines touched. For what?
Would not a single:
Quoted 8 lines
trace_numa_task_swap_tp(p);be much saner? It translates into a single no-op; no lines touched. Only when someone wants the numbers do we attach to the tracepoint and start collecting things.
Is the collecting more expensive; maybe. But the rest of us will be better of, no?
Probably not as bad as you may think. Systems with one NUMA node or NUMA balancing disabled (which will be most of the machines) won't be affected by this at all , task_numa_migrate() is also ratelimited so it doesn't get touched nearly as often as most of other scheduler events.
If this is on a really hot and critical path that most of us have to take, such as wakeup, I won't argue with you at all. I don't want to be too persistent here, it's fine to use eBPF with the existing tracepoints. I just think this is convenient and doesn't really hurt those who don't care about these numbers.
Peter Zijlstra
maintainerQuoted 10 lines · Libo Chen
Probably not as bad as you may think. Systems with one NUMA node or NUMA balancing disabled (which will be most of the machines) won't be affected by this at all ,
task_numa_migrate()is also ratelimited so it doesn't get touched nearly as often as most of other scheduler events.If this is on a really hot and critical path that most of us have to take, such as wakeup, I won't argue with you at all. I don't want to be too persistent here, it's fine to use eBPF with the existing tracepoints. I just think this is convenient and doesn't really hurt those who don't care about these numbers.
Its not about this one path per-se, more a general rant on the 'merit' of endlessly adding statistics and accounting to code.