Task stuck on global_ctx_data_rwsem
Namhyung Kim
maintainerHello,
I got a report that a task is stuck in perf_event_exit_task() waiting for global_ctx_data_rwsem. On large systems, it'd have performance issues when it grabs the lock to iterate all threads in the system to allocate the context data. And it'd block task exit path which is problematic especially under memory pressure.
perf_event_open
perf_event_alloc
attach_perf_ctx_data
attach_global_ctx_data
percpu_down_write (global_ctx_data_rwsem)
for_each_process_thread
alloc_task_ctx_data
do_exit
perf_event_exit_task
percpu_down_read (global_ctx_data_rwsem)
I think attach_global_ctx_data() should skip tasks with PF_EXITING and it'd be nice if perf_event_exit_task() could release the ctx_data unconditionally. But I'm not sure how to synchronize them properly.
Any thoughts?
Thanks,
Namhyung
Namhyung Kim
maintainerRe: [BUG] perf/core: Task stuck on global_ctx_data_rwsem
Added a subject prefix and CC LKML.
Thanks,
Namhyung
Quoted 28 lines · Namhyung Kim
Hello,
I got a report that a task is stuck in
perf_event_exit_task()waiting forglobal_ctx_data_rwsem. On large systems, it'd have performance issues when it grabs the lock to iterate all threads in the system to allocate the context data. And it'd block task exit path which is problematic especially under memory pressure.
perf_event_open
perf_event_alloc
attach_perf_ctx_data
attach_global_ctx_data
percpu_down_write(global_ctx_data_rwsem)
for_each_process_thread
alloc_task_ctx_data
do_exit
perf_event_exit_task
percpu_down_read(global_ctx_data_rwsem)I think
attach_global_ctx_data()should skip tasks withPF_EXITINGand it'd be nice ifperf_event_exit_task()could release thectx_dataunconditionally. But I'm not sure how to synchronize them properly.Any thoughts?
Thanks,
Namhyung
Namhyung Kim
maintainerRe: [BUG] perf/core: Task stuck on global_ctx_data_rwsem
Hello,
Quoted 20 lines · Namhyung Kim
Quoted 19 lines · Namhyung Kim
Hello,
I got a report that a task is stuck in
perf_event_exit_task()waiting forglobal_ctx_data_rwsem. On large systems, it'd have performance issues when it grabs the lock to iterate all threads in the system to allocate the context data. And it'd block task exit path which is problematic especially under memory pressure.
perf_event_open
perf_event_alloc
attach_perf_ctx_data
attach_global_ctx_data
percpu_down_write(global_ctx_data_rwsem)
for_each_process_thread
alloc_task_ctx_data
do_exit
perf_event_exit_task
percpu_down_read(global_ctx_data_rwsem)
Quoted 5 lines
Quoted 5 lines
I think
attach_global_ctx_data()should skip tasks withPF_EXITINGand it'd be nice ifperf_event_exit_task()could release thectx_dataunconditionally. But I'm not sure how to synchronize them properly.Any thoughts?
I'm curious if this makes any sense.. I feel like it needs to check the flag again before allocation.
Thanks,
Namhyung
kernel/events/core.c +2 -1
@@ -5469,6 +5469,8 @@ attach_global_ctx_data(struct kmem_cache *ctx_cache)/* Allocate everything */ scoped_guard (rcu) {for_each_process_thread (g, p) { +if (p->flags &PF_EXITING ) +continue ; cd =rcu_dereference (p->perf_ctx_data);if (cd && !cd->global) { cd->global =1 ;
@@ -14563,7 +14565,6 @@ void perf_event_exit_task(struct task_struct *task)/* * Detach the perf_ctx_data for the system-wide event. */ -guard (percpu_read)(&global_ctx_data_rwsem);detach_task_ctx_data (task); }
Peter Zijlstra
maintainerRe: [BUG] perf/core: Task stuck on global_ctx_data_rwsem
Quoted 41 lines · Namhyung Kim
Hello,
Quoted 20 lines · Namhyung Kim
Quoted 19 lines · Namhyung Kim
Hello,
I got a report that a task is stuck in
perf_event_exit_task()waiting forglobal_ctx_data_rwsem. On large systems, it'd have performance issues when it grabs the lock to iterate all threads in the system to allocate the context data. And it'd block task exit path which is problematic especially under memory pressure.
perf_event_open
perf_event_alloc
attach_perf_ctx_data
attach_global_ctx_data
percpu_down_write(global_ctx_data_rwsem)
for_each_process_thread
alloc_task_ctx_data
do_exit
perf_event_exit_task
percpu_down_read(global_ctx_data_rwsem)Quoted 5 lines
Quoted 5 lines
I think
attach_global_ctx_data()should skip tasks withPF_EXITINGand it'd be nice ifperf_event_exit_task()could release thectx_dataunconditionally. But I'm not sure how to synchronize them properly.Any thoughts?
I'm curious if this makes any sense.. I feel like it needs to check the flag again before allocation.
Thanks,
Namhyungkernel/events/core.c +0
-0
kernel/events/core.c attach_global_ctx_data(struct kmem_cache *ctx_cache) · line 5476
Quoted 8 lines
kernel/events/core.c +2
-0@@ -5469,6 +5469,8 @@ attach_global_ctx_data(struct kmem_cache *ctx_cache)/* Allocate everything */ scoped_guard (rcu) {for_each_process_thread (g, p) { +if (p->flags &PF_EXITING ) +continue ; cd =rcu_dereference (p->perf_ctx_data);if (cd && !cd->global) { cd->global =1 ;
I suppose this makes sense.
kernel/events/core.c void perf_event_exit_task(struct task_struct *task) · line 14569
Quoted 7 lines
kernel/events/core.c +0
-1@@ -14563,7 +14565,6 @@ void perf_event_exit_task(struct task_struct *task)/* * Detach the perf_ctx_data for the system-wide event. */ -guard (percpu_read)(&global_ctx_data_rwsem);detach_task_ctx_data (task); }
This would need a comment; something like:
/* * This can be done without holding global_ctx_data_rwsem * because this is done after setting PF_EXITING such that * attach_global_ctx_data() will skip over this task. */ WARN_ON_ONCE(!(task->flags & PF_EXITING))
But yes, I suppose this can do. The question is however, how do you get into this predicament to begin with? Are you creating and destroying a lot of global LBR events or something?
Would it make sense to delay detach_global_ctx_data() for a second or so? That is, what is your event creation pattern?
Namhyung Kim
maintainerRe: [BUG] perf/core: Task stuck on global_ctx_data_rwsem
Quoted 70 lines · Peter Zijlstra
Quoted 41 lines · Namhyung Kim
Hello,
Quoted 20 lines · Namhyung Kim
Quoted 19 lines · Namhyung Kim
Hello,
I got a report that a task is stuck in
perf_event_exit_task()waiting forglobal_ctx_data_rwsem. On large systems, it'd have performance issues when it grabs the lock to iterate all threads in the system to allocate the context data. And it'd block task exit path which is problematic especially under memory pressure.
perf_event_open
perf_event_alloc
attach_perf_ctx_data
attach_global_ctx_data
percpu_down_write(global_ctx_data_rwsem)
for_each_process_thread
alloc_task_ctx_data
do_exit
perf_event_exit_task
percpu_down_read(global_ctx_data_rwsem)Quoted 5 lines
Quoted 5 lines
I think
attach_global_ctx_data()should skip tasks withPF_EXITINGand it'd be nice ifperf_event_exit_task()could release thectx_dataunconditionally. But I'm not sure how to synchronize them properly.Any thoughts?
I'm curious if this makes any sense.. I feel like it needs to check the flag again before allocation.
Thanks,
Namhyungkernel/events/core.c +0
-0kernel/events/core.c attach_global_ctx_data(struct kmem_cache *ctx_cache) · line 5476
Quoted 8 lines
kernel/events/core.c +2
-0@@ -5469,6 +5469,8 @@ attach_global_ctx_data(struct kmem_cache *ctx_cache)/* Allocate everything */ scoped_guard (rcu) {for_each_process_thread (g, p) { +if (p->flags &PF_EXITING ) +continue ; cd =rcu_dereference (p->perf_ctx_data);if (cd && !cd->global) { cd->global =1 ;I suppose this makes sense.
kernel/events/core.c void perf_event_exit_task(struct task_struct *task) · line 14569
Quoted 7 lines
kernel/events/core.c +0
-1@@ -14563,7 +14565,6 @@ void perf_event_exit_task(struct task_struct *task)/* * Detach the perf_ctx_data for the system-wide event. */ -guard (percpu_read)(&global_ctx_data_rwsem);detach_task_ctx_data (task); }This would need a comment; something like:
/* * This can be done without holding global_ctx_data_rwsem * because this is done after setting PF_EXITING such that * attach_global_ctx_data() will skip over this task. */ WARN_ON_ONCE(!(task->flags & PF_EXITING))
Quoted 3 lines
But yes, I suppose this can do. The question is however, how do you get into this predicament to begin with? Are you creating and destroying a lot of global LBR events or something?
I think it's just because there are too many tasks in the system like O(100K). And any thread going to exit needs to wait for attach_global_ctx_data() to finish the iteration over every task.
Quoted 3 lines
Would it make sense to delay
detach_global_ctx_data()for a second or so? That is, what is your event creation pattern?
I don't think it has a special pattern, but I'm curious how we can handle a race like below.
attach_global_ctx_data
check p->flags & PF_EXITING
do_exit
(preemption) set PF_EXITING
detach_task_ctx_data()
check p->perf_ctx_data
attach_task_ctx_data() ---> memory leakThanks,
Namhyung
Peter Zijlstra
maintainerRe: [BUG] perf/core: Task stuck on global_ctx_data_rwsem
Quoted 7 lines · Namhyung Kim
Quoted 3 lines
But yes, I suppose this can do. The question is however, how do you get into this predicament to begin with? Are you creating and destroying a lot of global LBR events or something?
I think it's just because there are too many tasks in the system like O(100K). And any thread going to exit needs to wait for
attach_global_ctx_data()to finish the iteration over every task.
OMG, so many tasks ...
Quoted 6 lines
Quoted 2 lines
Would it make sense to delay
detach_global_ctx_data()for a second or so? That is, what is your event creation pattern?I don't think it has a special pattern, but I'm curious how we can handle a race like below.
Quoted 7 lines
attach_global_ctx_datacheck p->flags & PF_EXITING do_exit (preemption) set PF_EXITING detach_task_ctx_data() check p->perf_ctx_data attach_task_ctx_data() ---> memory leak
Oh right. Something like so perhaps?
---
kernel/events/core.c +17 -2
@@ -5421,9 +5421,19 @@ attach_task_ctx_data(struct task_struct *task, struct kmem_cache *ctx_cache,return -ENOMEM ;for (;;) { -if (try_cmpxchg ((struct perf_ctx_data **)&task->perf_ctx_data, &old, cd)) { +if (try_cmpxchg (&task->perf_ctx_data, &old, cd)) {if (old)perf_free_ctx_data_rcu (old); +/* +* try_cmpxchg() pairs with try_cmpxchg() from +* detach_task_ctx_data() such that +* if we race with perf_event_exit_task(), we must +* observe PF_EXITING. +*/ +if (task->flags &PF_EXITING ) { + task->perf_ctx_data =NULL ; +perf_free_ctx_data_rcu (cd); + }return 0 ; }
@@ -5469,6 +5479,8 @@ attach_global_ctx_data(struct kmem_cache *ctx_cache)/* Allocate everything */ scoped_guard (rcu) {for_each_process_thread (g, p) { +if (p->flags &PF_EXITING ) +continue ; cd =rcu_dereference (p->perf_ctx_data);if (cd && !cd->global) { cd->global =1 ;
@@ -14568,8 +14580,11 @@ void perf_event_exit_task(struct task_struct *task)/* * Detach the perf_ctx_data for the system-wide event. +* +* Done without holding global_ctx_data_rwsem; typically +* attach_global_ctx_data() will skip over this task, but otherwise +* attach_task_ctx_data() will observe PF_EXITING. */ -guard (percpu_read)(&global_ctx_data_rwsem);detach_task_ctx_data (task); }
Peter Zijlstra
maintainerRe: [BUG] perf/core: Task stuck on global_ctx_data_rwsem
Quoted 40 lines · Peter Zijlstra
Quoted 7 lines · Namhyung Kim
Quoted 3 lines
But yes, I suppose this can do. The question is however, how do you get into this predicament to begin with? Are you creating and destroying a lot of global LBR events or something?
I think it's just because there are too many tasks in the system like O(100K). And any thread going to exit needs to wait for
attach_global_ctx_data()to finish the iteration over every task.OMG, so many tasks ...
Quoted 6 lines
Quoted 2 lines
Would it make sense to delay
detach_global_ctx_data()for a second or so? That is, what is your event creation pattern?I don't think it has a special pattern, but I'm curious how we can handle a race like below.
Quoted 7 lines
attach_global_ctx_datacheck p->flags & PF_EXITING do_exit (preemption) set PF_EXITING detach_task_ctx_data() check p->perf_ctx_data attach_task_ctx_data() ---> memory leakOh right. Something like so perhaps?
---
kernel/events/core.c +1
-1@@ -5421,9 +5421,19 @@ attach_task_ctx_data(struct task_struct *task, struct kmem_cache *ctx_cache,return -ENOMEM ;for (;;) { -if (try_cmpxchg ((struct perf_ctx_data **)&task->perf_ctx_data, &old, cd)) { +if (try_cmpxchg (&task->perf_ctx_data, &old, cd)) {if (old)
kernel/events/core.c attach_task_ctx_data(struct task_struct *task, struct kmem_cache *ctx_cache, · line 5435
Quoted 10 lines
kernel/events/core.c +9
-0perf_free_ctx_data_rcu (old); +/* +* try_cmpxchg() pairs with try_cmpxchg() from +* detach_task_ctx_data() such that +* if we race with perf_event_exit_task(), we must +* observe PF_EXITING. +*/ +if (task->flags &PF_EXITING ) { + task->perf_ctx_data =NULL ; +perf_free_ctx_data_rcu (cd);
Ugh and now it can race and do a double free, another try_cmpxchg() is needed here.
Quoted 26 lines
kernel/events/core.c +7
-1+ }return 0 ; }@@ -5469,6 +5479,8 @@ attach_global_ctx_data(struct kmem_cache *ctx_cache)/* Allocate everything */ scoped_guard (rcu) {for_each_process_thread (g, p) { +if (p->flags &PF_EXITING ) +continue ; cd =rcu_dereference (p->perf_ctx_data);if (cd && !cd->global) { cd->global =1 ;@@ -14568,8 +14580,11 @@ void perf_event_exit_task(struct task_struct *task)/* * Detach the perf_ctx_data for the system-wide event. +* +* Done without holding global_ctx_data_rwsem; typically +* attach_global_ctx_data() will skip over this task, but otherwise +* attach_task_ctx_data() will observe PF_EXITING. */ -guard (percpu_read)(&global_ctx_data_rwsem);detach_task_ctx_data (task); }
Namhyung Kim
maintainerRe: [BUG] perf/core: Task stuck on global_ctx_data_rwsem
Quoted 46 lines · Peter Zijlstra
Quoted 45 lines · Peter Zijlstra
Quoted 7 lines · Namhyung Kim
Quoted 3 lines
But yes, I suppose this can do. The question is however, how do you get into this predicament to begin with? Are you creating and destroying a lot of global LBR events or something?
I think it's just because there are too many tasks in the system like O(100K). And any thread going to exit needs to wait for
attach_global_ctx_data()to finish the iteration over every task.OMG, so many tasks ...
Quoted 6 lines
Quoted 2 lines
Would it make sense to delay
detach_global_ctx_data()for a second or so? That is, what is your event creation pattern?I don't think it has a special pattern, but I'm curious how we can handle a race like below.
Quoted 7 lines
attach_global_ctx_datacheck p->flags & PF_EXITING do_exit (preemption) set PF_EXITING detach_task_ctx_data() check p->perf_ctx_data attach_task_ctx_data() ---> memory leakOh right. Something like so perhaps?
---
kernel/events/core.c +5
-1@@ -5421,9 +5421,19 @@ attach_task_ctx_data(struct task_struct *task, struct kmem_cache *ctx_cache,return -ENOMEM ;for (;;) { -if (try_cmpxchg ((struct perf_ctx_data **)&task->perf_ctx_data, &old, cd)) { +if (try_cmpxchg (&task->perf_ctx_data, &old, cd)) {if (old)perf_free_ctx_data_rcu (old); +/* +* try_cmpxchg() pairs with try_cmpxchg() from +* detach_task_ctx_data() such that +* if we race with perf_event_exit_task(), we must
Quoted 8 lines
kernel/events/core.c attach_task_ctx_data(struct task_struct *task, struct kmem_cache *ctx_cache, · line 5435
Quoted 5 lines
kernel/events/core.c +5
-0+* observe PF_EXITING. +*/ +if (task->flags &PF_EXITING ) { + task->perf_ctx_data =NULL ; +perf_free_ctx_data_rcu (cd);Ugh and now it can race and do a double free, another
try_cmpxchg()is needed here.
Thanks! Something like this?
Namhyung
kernel/events/core.c +18 -2
@@ -5421,9 +5421,20 @@ attach_task_ctx_data(struct task_struct *task, struct kmem_cache *ctx_cache,return -ENOMEM ;for (;;) { -if (try_cmpxchg ((struct perf_ctx_data **)&task->perf_ctx_data, &old, cd)) { +if (try_cmpxchg (&task->perf_ctx_data, &old, cd)) {if (old)perf_free_ctx_data_rcu (old); +/* +* try_cmpxchg() pairs with try_cmpxchg() from +* detach_task_ctx_data() such that +* if we race with perf_event_exit_task(), we must +* observe PF_EXITING. +*/ +if (task->flags &PF_EXITING ) { +/* detach_task_ctx_data() may free it already */ +if (try_cmpxchg (&task->perf_ctx_data, &cd,NULL )) +perf_free_ctx_data_rcu (cd); + }return 0 ; }
@@ -5469,6 +5480,8 @@ attach_global_ctx_data(struct kmem_cache *ctx_cache)/* Allocate everything */ scoped_guard (rcu) {for_each_process_thread (g, p) { +if (p->flags &PF_EXITING ) +continue ; cd =rcu_dereference (p->perf_ctx_data);if (cd && !cd->global) { cd->global =1 ;
@@ -14562,8 +14575,11 @@ void perf_event_exit_task(struct task_struct *task)/* * Detach the perf_ctx_data for the system-wide event. +* +* Done without holding global_ctx_data_rwsem; typically +* attach_global_ctx_data() will skip over this task, but otherwise +* attach_task_ctx_data() will observe PF_EXITING. */ -guard (percpu_read)(&global_ctx_data_rwsem);detach_task_ctx_data (task); }