perf: Make perf_pmu_unregister() usable
Subsystems (4)
Peter Zijlstra
maintainerv2 cover perf: Make perf_pmu_unregister() usable
Hi
New version of the perf_pmu_unregister() rework. A fair number of bugs squashed. Much thanks to Ravi for writing a stress tester. The robots also managed to find a few problems.
Peter Zijlstra
maintainerPATCH v2 1/24 lockdep: Fix might_fault()
Turns out that commit 9ec23531fd48 ("sched/preempt, mm/fault: Trigger might_sleep() in might_fault() with disabled pagefaults") accidentally (and unnessecarily) put the lockdep part of __might_fault() under CONFIG_DEBUG_ATOMIC_SLEEP.
--- mm/memory.c | 2 -- 1 file changed, 2 deletions(-)
mm/memory.c +0 -2
@@ -6695,10 +6695,8 @@ void __might_fault(const char *file, intif (pagefault_disabled ())return ;__might_sleep (file, line); -#if defined(CONFIG_DEBUG_ATOMIC_SLEEP )if (current->mm) might_lock_read(¤t->mm->mmap_lock); -#endif }EXPORT_SYMBOL (__might_fault);#endif
Re: [PATCH v2 01/24] lockdep: Fix might_fault()
Quoted 7 lines · Peter Zijlstra
Turns out that commit
9ec23531fd48("sched/preempt, mm/fault: Triggermight_sleep()inmight_fault()with disabled pagefaults") accidentally (and unnessecarily) put the lockdep part of__might_fault()underCONFIG_DEBUG_ATOMIC_SLEEP.Cc David Hildenbrand <[email protected]>Fixes 9ec23531fd48 ("sched/preempt, mm/fault: Trigger might_sleep() in might_fault() with disabled pagefaults")
Heh, that was 10 years ago, part one of my first not-kvm-specific patch series.
Very likely it was not by accident, but I don't have any memory on that part :)
Probably I actually wanted this to be
#if defined(CONFIG_PROVE_LOCKING)
such that the function looked more similar to pre-662bbcb2747c
Quoted 20 lines
Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- mm/memory.c | 2 -- 1 file changed, 2 deletions(-)mm/memory.c +0
-2@@ -6695,10 +6695,8 @@ void __might_fault(const char *file, intif (pagefault_disabled ())return ;__might_sleep (file, line); -#if defined(CONFIG_DEBUG_ATOMIC_SLEEP )if (current->mm) might_lock_read(¤t->mm->mmap_lock); -#endif }EXPORT_SYMBOL (__might_fault);#endif
Signature · 3 lines
Cheers, David / dhildenb
Peter Zijlstra
maintainerPATCH v2 2/24 perf: Ensure bpf_perf_link path is properly serialized
In reply to Peter ZijlstraRavi reported that the bpf_perf_link_attach() usage of perf_event_set_bpf_prog() is not serialized by ctx->mutex, unlike the PERF_EVENT_IOC_SET_BPF case.
--- kernel/events/core.c | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-)
kernel/events/core.c +30 -4
@@ -6001,6 +6001,9 @@ static int perf_event_set_output(structstatic int perf_event_set_filter (struct perf_event *event,void __user *arg);static int perf_copy_attr (struct perf_event_attr __user *uattr,struct perf_event_attr *attr); +static int __perf_event_set_bpf_prog (struct perf_event *event, +struct bpf_prog *prog, +u64 bpf_cookie);static long _perf_ioctl (struct perf_event *event,unsigned int cmd,unsigned long arg) {
@@ -6063,7 +6066,7 @@ static long _perf_ioctl(struct perf_evenif (IS_ERR (prog))return PTR_ERR (prog); - err =perf_event_set_bpf_prog (event, prog,0 ); + err =__perf_event_set_bpf_prog (event, prog,0 );if (err) {bpf_prog_put (prog);return err;
@@ -10790,8 +10793,9 @@ static inline bool perf_event_is_tracingreturn false; } -int perf_event_set_bpf_prog (struct perf_event *event,struct bpf_prog *prog, -u64 bpf_cookie) +static int __perf_event_set_bpf_prog (struct perf_event *event, +struct bpf_prog *prog, +u64 bpf_cookie) {bool is_kprobe, is_uprobe, is_tracepoint, is_syscall_tp;
@@ -10829,6 +10833,20 @@ int perf_event_set_bpf_prog(struct perf_return perf_event_attach_bpf_prog (event, prog, bpf_cookie); } +int perf_event_set_bpf_prog (struct perf_event *event, +struct bpf_prog *prog, +u64 bpf_cookie) +{ +struct perf_event_context *ctx; +int ret; + + ctx =perf_event_ctx_lock (event); + ret =__perf_event_set_bpf_prog (event, prog, bpf_cookie); +perf_event_ctx_unlock (event, ctx); + +return ret; +} +void perf_event_free_bpf_prog (struct perf_event *event) {if (!perf_event_is_tracing (event)) {
@@ -10848,7 +10866,15 @@ static void perf_event_free_filter(struc { } -int perf_event_set_bpf_prog (struct perf_event *event,struct bpf_prog *prog, +static int __perf_event_set_bpf_prog (struct perf_event *event, +struct bpf_prog *prog, +u64 bpf_cookie) +{ +return -ENOENT ; +} + +int perf_event_set_bpf_prog (struct perf_event *event, +struct bpf_prog *prog,u64 bpf_cookie) {return -ENOENT ;
Peter Zijlstra
maintainerPATCH v2 3/24 perf: Simplify child event tear-down
In reply to Peter ZijlstraCurrently perf_event_release_kernel() will iterate the child events and attempt tear-down. However, it removes them from the child_list using list_move(), notably skipping the state management done by perf_child_detach().
Crucially, it fails to clear PERF_ATTACH_CHILD, which opens the door for a concurrent perf_remove_from_context() to race.
This way child_list management stays fully serialized using child_mutex.
--- kernel/events/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/events/core.c +2 -2
@@ -5526,8 +5526,8 @@ int perf_event_release_kernel(struct per tmp =list_first_entry_or_null (&event->child_list, struct perf_event, child_list);if (tmp == child) { -perf_remove_from_context (child,DETACH_GROUP ); -list_move (&child->child_list, &free_list); +perf_remove_from_context (child,DETACH_GROUP |DETACH_CHILD ); +list_add (&child->child_list, &free_list);/* * This matches the refcount bump in inherit_event(); * this can't be the last reference.
Peter Zijlstra
maintainerPATCH v2 4/24 perf: Simplify perf_event_free_task() wait
In reply to Peter Zijlstra--- kernel/events/core.c | 32 ++++++++------------------------ 1 file changed, 8 insertions(+), 24 deletions(-)
kernel/events/core.c +8 -24
@@ -1223,8 +1223,14 @@ static void put_ctx(struct perf_event_coif (refcount_dec_and_test (&ctx->refcount)) {if (ctx->parent_ctx)put_ctx (ctx->parent_ctx); -if (ctx->task && ctx->task !=TASK_TOMBSTONE ) -put_task_struct (ctx->task); +if (ctx->task) { +if (ctx->task ==TASK_TOMBSTONE ) { +smp_mb ();/* pairs with wait_var_event() */ +wake_up_var (&ctx->refcount); + }else { +put_task_struct (ctx->task); + } + }call_rcu (&ctx->rcu_head, free_ctx); } }
@@ -5492,8 +5498,6 @@ int perf_event_release_kernel(struct per again:mutex_lock (&event->child_mutex);list_for_each_entry (child, &event->child_list, child_list) { -void *var =NULL ; -/* * Cannot change, child events are not migrated, see the * comment with perf_event_ctx_lock_nested().
@@ -5533,39 +5537,19 @@ int perf_event_release_kernel(struct per* this can't be the last reference. */ put_event (event); - }else { - var = &ctx->refcount; }mutex_unlock (&event->child_mutex);mutex_unlock (&ctx->mutex);put_ctx (ctx); -if (var) { -/* -* If perf_event_free_task() has deleted all events from the -* ctx while the child_mutex got released above, make sure to -* notify about the preceding put_ctx(). -*/ -smp_mb ();/* pairs with wait_var_event() */ -wake_up_var (var); - } goto again; }mutex_unlock (&event->child_mutex);list_for_each_entry_safe (child, tmp, &free_list, child_list) { -void *var = &child->ctx->refcount; -list_del (&child->child_list);free_event (child); - -/* -* Wake any perf_event_free_task() waiting for this event to be -* freed. -*/ -smp_mb ();/* pairs with wait_var_event() */ -wake_up_var (var); } no_ctx:
Peter Zijlstra
maintainerPATCH v2 5/24 perf: Simplify perf_event_release_kernel()
In reply to Peter ZijlstraGet rid of the free_list...
--- kernel/events/core.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-)
kernel/events/core.c +5 -7
@@ -5462,7 +5462,6 @@ int perf_event_release_kernel(struct per {struct perf_event_context *ctx = event->ctx;struct perf_event *child, *tmp; -LIST_HEAD (free_list);/* * If we got here through err_alloc: free_event(event); we will not
@@ -5531,27 +5530,26 @@ int perf_event_release_kernel(struct per struct perf_event, child_list);if (tmp == child) {perf_remove_from_context (child,DETACH_GROUP |DETACH_CHILD ); -list_add (&child->child_list, &free_list);/* * This matches the refcount bump in inherit_event(); * this can't be the last reference. */ put_event (event); + }else { + child =NULL ; }mutex_unlock (&event->child_mutex);mutex_unlock (&ctx->mutex); + +if (child) +free_event (child);put_ctx (ctx); goto again; }mutex_unlock (&event->child_mutex); -list_for_each_entry_safe (child, tmp, &free_list, child_list) { -list_del (&child->child_list); -free_event (child); - } - no_ctx:put_event (event);/* Must be the 'last' reference */ return 0 ;
Peter Zijlstra
maintainerPATCH v2 6/24 perf: Fix pmus_lock vs pmus_srcu ordering
In reply to Peter ZijlstraCommit a63fbed776c7 ("perf/tracing/cpuhotplug: Fix locking order") placed pmus_lock inside pmus_srcu, this makes perf_pmu_unregister() trip lockdep.
Move the locking about such that only pmu_idr and pmus (list) are modified while holding pmus_lock. This avoids doing synchronize_srcu() while holding pmus_lock and all is well again.
--- kernel/events/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
kernel/events/core.c +2 -2
@@ -11836,6 +11836,8 @@ void perf_pmu_unregister(struct pmu *pmu {mutex_lock (&pmus_lock);list_del_rcu (&pmu->entry); +idr_remove (&pmu_idr, pmu->type); +mutex_unlock (&pmus_lock);/* * We dereference the pmu list under both SRCU and regular RCU, so
@@ -11845,7 +11847,6 @@ void perf_pmu_unregister(struct pmu *pmusynchronize_rcu ();free_percpu (pmu->pmu_disable_count); -idr_remove (&pmu_idr, pmu->type);if (pmu_bus_running && pmu->dev && pmu->dev !=PMU_NULL_DEV ) {if (pmu->nr_addr_filters)device_remove_file (pmu->dev, &dev_attr_nr_addr_filters);
@@ -11853,7 +11854,6 @@ void perf_pmu_unregister(struct pmu *pmuput_device (pmu->dev); }free_pmu_context (pmu); -mutex_unlock (&pmus_lock); }EXPORT_SYMBOL_GPL (perf_pmu_unregister);
Re: [PATCH v2 06/24] perf: Fix pmus_lock vs pmus_srcu ordering
Quoted 5 lines · Peter Zijlstra
Commit
a63fbed776c7("perf/tracing/cpuhotplug: Fix locking order") placedpmus_lockinsidepmus_srcu, this makesperf_pmu_unregister()trip lockdep.Move the locking about such that only
pmu_idrand pmus (list) are
s/about/around/ ?
Quoted 5 lines
modified while holding
pmus_lock. This avoids doingsynchronize_srcu()while holdingpmus_lockand all is well again.Fixes a63fbed776c7 ("perf/tracing/cpuhotplug: Fix locking order")Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Without this patch, I occasionally see the locking issue below, but it doesn't always happen. I'm running with this patch and haven't seen the issue since.
[ 268.195731] ======================================================
[ 268.201945] WARNING: possible circular locking dependency detected
[ 268.208165] 6.14.0-rc4-xe+ #59 Tainted: G U
[ 268.213777] ------------------------------------------------------
[ 268.219988] kmod-unbind/2509 is trying to acquire lock:
[ 268.225238] ffffffff8b40b310 (&pmus_srcu){.+.+}-{0:0}, at: __synchronize_srcu+0x21/0x270
[ 268.233374]
but task is already holding lock:
[ 268.239241] ffffffff86381de8 (pmus_lock){+.+.}-{3:3}, at: perf_pmu_unregister+0x25/0x2f0
[ 268.247372]
which lock already depends on the new lock.[ 268.255577]
the existing dependency chain (in reverse order) is:
[ 268.263101]
-> #1 (pmus_lock){+.+.}-{3:3}:
[ 268.268717] __mutex_lock+0x1a4/0x1300
[ 268.273019] mutex_lock_nested+0x1b/0x30
[ 268.277489] perf_swevent_init+0x15d/0x610
[ 268.282142] perf_try_init_event+0x111/0xba0
[ 268.286971] perf_event_alloc+0x1024/0x3eb0
[ 268.291703] __do_sys_perf_event_open+0x3b0/0x2460
[ 268.297046] __x64_sys_perf_event_open+0xbe/0x160
[ 268.302305] x64_sys_call+0x1eb0/0x2650
[ 268.306689] do_syscall_64+0x91/0x180
[ 268.310918] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 268.316524]
-> #0 (&pmus_srcu){.+.+}-{0:0}:
[ 268.322215] __lock_acquire+0x3339/0x6980
[ 268.326777] lock_sync+0xec/0x180
[ 268.330645] __synchronize_srcu+0xaf/0x270
[ 268.335291] synchronize_srcu+0x64/0x2b0
[ 268.339776] perf_pmu_unregister+0xed/0x2f0
[ 268.344523] xe_pmu_unregister+0x94/0xf0 [xe]
[ 268.349604] devm_action_release+0x49/0x80
[ 268.354257] devres_release_all+0x180/0x240
[ 268.358996] device_unbind_cleanup+0x1b/0x1b0
[ 268.363910] device_release_driver_internal+0x44a/0x5b0
[ 268.369693] device_driver_detach+0x36/0x50
[ 268.374435] unbind_store+0xe0/0x100
[ 268.378571] drv_attr_store+0x6a/0xc0
[ 268.382782] sysfs_kf_write+0x106/0x160
[ 268.387176] kernfs_fop_write_iter+0x359/0x550
[ 268.392180] vfs_write+0x64c/0x1030
[ 268.396234] ksys_write+0x11a/0x240
[ 268.400284] __x64_sys_write+0x72/0xc0
[ 268.404587] x64_sys_call+0x2c4/0x2650
[ 268.408885] do_syscall_64+0x91/0x180
[ 268.413102] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 268.418708]
other info that might help us debug this:[ 268.426746] Possible unsafe locking scenario:
[ 268.432715] CPU0 CPU1
[ 268.437277] ---- ----
[ 268.441842] lock(pmus_lock);
[ 268.444932] lock(&pmus_srcu);
[ 268.450633] lock(pmus_lock);
[ 268.456243] sync(&pmus_srcu);
[ 268.459419]
*** DEADLOCK ***[ 268.465375] 4 locks held by kmod-unbind/2509:
[ 268.469779] #0: ffff888135689420 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x11a/0x240
[ 268.477911] #1: ffff8881107aee88 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x221/0x550
[ 268.486829] #2: ffff888111cc41b0 (&dev->mutex){....}-{3:3}, at: device_release_driver_internal+0xa2/0x5b0
[ 268.496547] #3: ffffffff86381de8 (pmus_lock){+.+.}-{3:3}, at: perf_pmu_unregister+0x25/0x2f0Lucas De Marchi
Quoted 33 lines
---
kernel/events/core.c| 4 ++--1 file changed, 2 insertions(+), 2 deletions(-)kernel/events/core.c +2
-2@@ -11836,6 +11836,8 @@ void perf_pmu_unregister(struct pmu *pmu {mutex_lock (&pmus_lock);list_del_rcu (&pmu->entry); +idr_remove (&pmu_idr, pmu->type); +mutex_unlock (&pmus_lock);/* * We dereference the pmu list under both SRCU and regular RCU, so @@ -11845,7 +11847,6 @@ void perf_pmu_unregister(struct pmu *pmusynchronize_rcu ();free_percpu (pmu->pmu_disable_count); -idr_remove (&pmu_idr, pmu->type);if (pmu_bus_running && pmu->dev && pmu->dev !=PMU_NULL_DEV ) {if (pmu->nr_addr_filters)device_remove_file (pmu->dev, &dev_attr_nr_addr_filters);@@ -11853,7 +11854,6 @@ void perf_pmu_unregister(struct pmu *pmuput_device (pmu->dev); }free_pmu_context (pmu); -mutex_unlock (&pmus_lock); }EXPORT_SYMBOL_GPL (perf_pmu_unregister);
Peter Zijlstra
maintainerPATCH v2 7/24 perf: Fix perf_pmu_register() vs perf_init_event()
In reply to Peter ZijlstraThere is a fairly obvious race between perf_init_event() doing idr_find() and perf_pmu_register() doing idr_alloc() with an incompletely initialized pmu pointer.
Avoid by doing idr_alloc() on a NULL pointer to register the id, and swizzling the real pmu pointer at the end using idr_replace().
Also making sure to not set pmu members after publishing the pmu, duh.
[ introduce idr_cmpxchg() in order to better handle the idr_replace()
error case -- if it were to return an unexpected pointer, it will already have replaced the value and there is no going back. ]
--- kernel/events/core.c | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-)
kernel/events/core.c +26 -2
@@ -11739,6 +11739,21 @@ static int pmu_dev_alloc(struct pmu *pmustatic struct lock_class_key cpuctx_mutex;static struct lock_class_key cpuctx_lock; +static bool idr_cmpxchg (struct idr *idr,unsigned long id,void *old,void *new) +{ +void *tmp, *val =idr_find (idr, id); + +if (val != old) +return false; + + tmp =idr_replace (idr, new, id); +if (IS_ERR (tmp)) +return false; + +WARN_ON_ONCE (tmp != val); +return true; +} +int perf_pmu_register (struct pmu *pmu,const char *name,int type) {int cpu, ret, max =PERF_TYPE_MAX ;
@@ -11765,7 +11780,7 @@ int perf_pmu_register(struct pmu *pmu, cif (type >=0 ) max = type; - ret =idr_alloc (&pmu_idr, pmu, max,0 ,GFP_KERNEL ); + ret =idr_alloc (&pmu_idr,NULL , max,0 ,GFP_KERNEL );if (ret <0 ) goto free_pdc;
@@ -11773,6 +11788,7 @@ int perf_pmu_register(struct pmu *pmu, c type = ret; pmu->type = type; +atomic_set (&pmu->exclusive_cnt,0 );if (pmu_bus_running && !pmu->dev) { ret =pmu_dev_alloc (pmu);
@@ -11821,14 +11837,22 @@ int perf_pmu_register(struct pmu *pmu, cif (!pmu->event_idx) pmu->event_idx = perf_event_idx_default; +/* +* Now that the PMU is complete, make it visible to perf_try_init_event(). +*/ +if (!idr_cmpxchg (&pmu_idr, pmu->type,NULL , pmu)) + goto free_context;list_add_rcu (&pmu->entry, &pmus); -atomic_set (&pmu->exclusive_cnt,0 ); + ret =0 ; unlock:mutex_unlock (&pmus_lock);return ret; +free_context: +free_percpu (pmu->cpu_pmu_context); + free_dev:if (pmu->dev && pmu->dev !=PMU_NULL_DEV ) {device_del (pmu->dev);
Peter Zijlstra
maintainerPATCH v2 8/24 perf: Cleanup perf_try_init_event()
In reply to Peter ZijlstraMake sure that perf_try_init_event() doesn't leave event->pmu nor event->destroy set on failure.
--- kernel/events/core.c | 69 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 41 insertions(+), 28 deletions(-)
kernel/events/core.c +41 -28
@@ -12019,38 +12019,51 @@ static int perf_try_init_event(struct pmif (ctx)perf_event_ctx_unlock (event->group_leader, ctx); -if (!ret) { -if (!(pmu->capabilities &PERF_PMU_CAP_EXTENDED_REGS ) && -has_extended_regs (event)) - ret = -EOPNOTSUPP ; - -if (pmu->capabilities &PERF_PMU_CAP_NO_EXCLUDE && -event_has_any_exclude_flag (event)) - ret = -EINVAL ; - -if (pmu->scope !=PERF_PMU_SCOPE_NONE && event->cpu >=0 ) { -const struct cpumask *cpumask =perf_scope_cpu_topology_cpumask (pmu->scope, event->cpu); -struct cpumask *pmu_cpumask =perf_scope_cpumask (pmu->scope); -int cpu; - -if (pmu_cpumask && cpumask) { - cpu =cpumask_any_and (pmu_cpumask, cpumask); -if (cpu >= nr_cpu_ids) - ret = -ENODEV ; -else - event->event_caps |=PERF_EV_CAP_READ_SCOPE ; - }else { - ret = -ENODEV ; - } - } +if (ret) + goto err_pmu; -if (ret && event->destroy) - event->destroy (event); +if (!(pmu->capabilities &PERF_PMU_CAP_EXTENDED_REGS ) && +has_extended_regs (event)) { + ret = -EOPNOTSUPP ; + goto err_destroy; } -if (ret) -module_put (pmu->module); +if (pmu->capabilities &PERF_PMU_CAP_NO_EXCLUDE && +event_has_any_exclude_flag (event)) { + ret = -EINVAL ; + goto err_destroy; + } + +if (pmu->scope !=PERF_PMU_SCOPE_NONE && event->cpu >=0 ) { +const struct cpumask *cpumask; +struct cpumask *pmu_cpumask; +int cpu; + + cpumask =perf_scope_cpu_topology_cpumask (pmu->scope, event->cpu); + pmu_cpumask =perf_scope_cpumask (pmu->scope); + + ret = -ENODEV ; +if (!pmu_cpumask || !cpumask) + goto err_destroy; + + cpu =cpumask_any_and (pmu_cpumask, cpumask); +if (cpu >= nr_cpu_ids) + goto err_destroy; + + event->event_caps |=PERF_EV_CAP_READ_SCOPE ; + } + +return 0 ; + +err_destroy: +if (event->destroy) { + event->destroy (event); + event->destroy =NULL ; + } +err_pmu: + event->pmu =NULL ; +module_put (pmu->module);return ret; }
[tip: perf/core] perf/core: Clean up perf_try_init_event()
The following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: da02f54e81db2f7bf6af9d1d0cfc5b41ec6d0dcb Gitweb: https://git.kernel.org/tip/da02f54e81db2f7bf6af9d1d0cfc5b41ec6d0dcb Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 05 Feb 2025 11:21:28 +01:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Wed, 05 Mar 2025 12:13:59 +01:00
perf/core: Clean up perf_try_init_event()
Make sure that perf_try_init_event() doesn't leave event->pmu nor event->destroy set on failure.
--- kernel/events/core.c | 65 +++++++++++++++++++++++++------------------ 1 file changed, 38 insertions(+), 27 deletions(-)
kernel/events/core.c +38 -27
@@ -12109,40 +12109,51 @@ static int perf_try_init_event(struct pmu *pmu, struct perf_event *event)if (ctx)perf_event_ctx_unlock (event->group_leader, ctx); -if (!ret) { -if (!(pmu->capabilities &PERF_PMU_CAP_EXTENDED_REGS ) && -has_extended_regs (event)) - ret = -EOPNOTSUPP ; +if (ret) + goto err_pmu; -if (pmu->capabilities &PERF_PMU_CAP_NO_EXCLUDE && -event_has_any_exclude_flag (event)) - ret = -EINVAL ; +if (!(pmu->capabilities &PERF_PMU_CAP_EXTENDED_REGS ) && +has_extended_regs (event)) { + ret = -EOPNOTSUPP ; + goto err_destroy; + } -if (pmu->scope !=PERF_PMU_SCOPE_NONE && event->cpu >=0 ) { -const struct cpumask *cpumask =perf_scope_cpu_topology_cpumask (pmu->scope, event->cpu); -struct cpumask *pmu_cpumask =perf_scope_cpumask (pmu->scope); -int cpu; - -if (pmu_cpumask && cpumask) { - cpu =cpumask_any_and (pmu_cpumask, cpumask); -if (cpu >= nr_cpu_ids) - ret = -ENODEV ; -else - event->event_caps |=PERF_EV_CAP_READ_SCOPE ; - }else { - ret = -ENODEV ; - } - } +if (pmu->capabilities &PERF_PMU_CAP_NO_EXCLUDE && +event_has_any_exclude_flag (event)) { + ret = -EINVAL ; + goto err_destroy; + } -if (ret && event->destroy) - event->destroy (event); +if (pmu->scope !=PERF_PMU_SCOPE_NONE && event->cpu >=0 ) { +const struct cpumask *cpumask; +struct cpumask *pmu_cpumask; +int cpu; + + cpumask =perf_scope_cpu_topology_cpumask (pmu->scope, event->cpu); + pmu_cpumask =perf_scope_cpumask (pmu->scope); + + ret = -ENODEV ; +if (!pmu_cpumask || !cpumask) + goto err_destroy; + + cpu =cpumask_any_and (pmu_cpumask, cpumask); +if (cpu >= nr_cpu_ids) + goto err_destroy; + + event->event_caps |=PERF_EV_CAP_READ_SCOPE ; } -if (ret) { - event->pmu =NULL ; -module_put (pmu->module); +return 0 ; + +err_destroy: +if (event->destroy) { + event->destroy (event); + event->destroy =NULL ; } +err_pmu: + event->pmu =NULL ; +module_put (pmu->module);return ret; }
Peter Zijlstra
maintainerPATCH v2 9/24 perf: Simplify perf_event_alloc() error path
In reply to Peter ZijlstraThe error cleanup sequence in perf_event_alloc() is a subset of the existing _free_event() function (it must of course be).
Split this out into __free_event() and simplify the error path.
--- include/linux/perf_event.h | 16 +++-- kernel/events/core.c | 134 ++++++++++++++++++++++----------------------- 2 files changed, 75 insertions(+), 75 deletions(-)
include/linux/perf_event.h +9 -7
@@ -673,13 +673,15 @@ struct swevent_hlist {struct rcu_head rcu_head; }; -#define PERF_ATTACH_CONTEXT 0x01 -#define PERF_ATTACH_GROUP 0x02 -#define PERF_ATTACH_TASK 0x04 -#define PERF_ATTACH_TASK_DATA 0x08 -#define PERF_ATTACH_ITRACE 0x10 -#define PERF_ATTACH_SCHED_CB 0x20 -#define PERF_ATTACH_CHILD 0x40 +#define PERF_ATTACH_CONTEXT 0x0001 +#define PERF_ATTACH_GROUP 0x0002 +#define PERF_ATTACH_TASK 0x0004 +#define PERF_ATTACH_TASK_DATA 0x0008 +#define PERF_ATTACH_ITRACE 0x0010 +#define PERF_ATTACH_SCHED_CB 0x0020 +#define PERF_ATTACH_CHILD 0x0040 +#define PERF_ATTACH_EXCLUSIVE 0x0080 +#define PERF_ATTACH_CALLCHAIN 0x0100struct bpf_prog ;struct perf_cgroup ;
kernel/events/core.c +66 -68
247 lines not on this page · raw message
Peter Zijlstra
maintainerPATCH v2 10/24 perf: Simplify perf_pmu_register() error path
In reply to Peter ZijlstraThe error path of perf_pmu_register() is of course very similar to a subset of perf_pmu_unregister(). Extract this common part in perf_pmu_free() and simplify things.
--- kernel/events/core.c | 67 ++++++++++++++++++++++----------------------------- 1 file changed, 30 insertions(+), 37 deletions(-)
kernel/events/core.c +30 -37
@@ -11605,11 +11605,6 @@ static int perf_event_idx_default(structreturn 0 ; } -static void free_pmu_context (struct pmu *pmu) -{ -free_percpu (pmu->cpu_pmu_context); -} -/* * Let userspace know that this PMU supports address range filtering: */
@@ -11815,6 +11810,7 @@ static int pmu_dev_alloc(struct pmu *pmu free_dev:put_device (pmu->dev); + pmu->dev =NULL ; goto out; }
@@ -11836,25 +11832,38 @@ static bool idr_cmpxchg(struct idr *idr,return true; } +static void perf_pmu_free (struct pmu *pmu) +{ +free_percpu (pmu->pmu_disable_count); +if (pmu_bus_running && pmu->dev && pmu->dev !=PMU_NULL_DEV ) { +if (pmu->nr_addr_filters) +device_remove_file (pmu->dev, &dev_attr_nr_addr_filters); +device_del (pmu->dev); +put_device (pmu->dev); + } +free_percpu (pmu->cpu_pmu_context); +} +int perf_pmu_register (struct pmu *pmu,const char *name,int type) {int cpu, ret, max =PERF_TYPE_MAX ; + pmu->type =-1 ; +mutex_lock (&pmus_lock); ret = -ENOMEM ; pmu->pmu_disable_count =alloc_percpu (int);if (!pmu->pmu_disable_count) goto unlock; - pmu->type =-1 ;if (WARN_ONCE (!name,"Can not register anonymous pmu.\n" )) { ret = -EINVAL ; - goto free_pdc; + goto free; }if (WARN_ONCE (pmu->scope >=PERF_PMU_MAX_SCOPE ,"Can not register a pmu with an invalid scope.\n" )) { ret = -EINVAL ; - goto free_pdc; + goto free; } pmu->name = name;
@@ -11864,24 +11873,23 @@ int perf_pmu_register(struct pmu *pmu, c ret =idr_alloc (&pmu_idr,NULL , max,0 ,GFP_KERNEL );if (ret <0 ) - goto free_pdc; + goto free;WARN_ON (type >=0 && ret != type); - type = ret; - pmu->type = type; + pmu->type = ret;atomic_set (&pmu->exclusive_cnt,0 );if (pmu_bus_running && !pmu->dev) { ret =pmu_dev_alloc (pmu);if (ret) - goto free_idr; + goto free; } ret = -ENOMEM ; pmu->cpu_pmu_context = alloc_percpu(struct perf_cpu_pmu_context);if (!pmu->cpu_pmu_context) - goto free_dev; + goto free;for_each_possible_cpu (cpu) {struct perf_cpu_pmu_context *cpc;
@@ -11922,8 +11930,10 @@ int perf_pmu_register(struct pmu *pmu, c/* * Now that the PMU is complete, make it visible to perf_try_init_event(). */ -if (!idr_cmpxchg (&pmu_idr, pmu->type,NULL , pmu)) - goto free_context; +if (!idr_cmpxchg (&pmu_idr, pmu->type,NULL , pmu)) { + ret = -EINVAL ; + goto free; + }list_add_rcu (&pmu->entry, &pmus); ret =0 ;
@@ -11932,20 +11942,10 @@ int perf_pmu_register(struct pmu *pmu, creturn ret; -free_context: -free_percpu (pmu->cpu_pmu_context); - -free_dev: -if (pmu->dev && pmu->dev !=PMU_NULL_DEV ) { -device_del (pmu->dev); -put_device (pmu->dev); - } - -free_idr: -idr_remove (&pmu_idr, pmu->type); - -free_pdc: -free_percpu (pmu->pmu_disable_count); +free: +if (pmu->type >=0 ) +idr_remove (&pmu_idr, pmu->type); +perf_pmu_free (pmu); goto unlock; }EXPORT_SYMBOL_GPL (perf_pmu_register);
@@ -11964,14 +11964,7 @@ void perf_pmu_unregister(struct pmu *pmusynchronize_srcu (&pmus_srcu);synchronize_rcu (); -free_percpu (pmu->pmu_disable_count); -if (pmu_bus_running && pmu->dev && pmu->dev !=PMU_NULL_DEV ) { -if (pmu->nr_addr_filters) -device_remove_file (pmu->dev, &dev_attr_nr_addr_filters); -device_del (pmu->dev); -put_device (pmu->dev); - } -free_pmu_context (pmu); +perf_pmu_free (pmu); }EXPORT_SYMBOL_GPL (perf_pmu_unregister);
Peter Zijlstra
maintainerPATCH v2 11/24 perf: Simplify perf_pmu_register()
In reply to Peter ZijlstraUsing the previously introduced perf_pmu_free() and a new IDR helper, simplify the perf_pmu_register error paths.
--- include/linux/idr.h | 17 ++++++++++++ kernel/events/core.c | 71 ++++++++++++++++++++------------------------------- 2 files changed, 46 insertions(+), 42 deletions(-)
include/linux/idr.h +17 -0
@@ -15,6 +15,7 @@#include <linux/radix-tree.h> #include <linux/gfp.h> #include <linux/percpu.h> +#include <linux/cleanup.h> struct idr {struct radix_tree_root idr_rt;
@@ -124,6 +125,22 @@ void *idr_get_next_ul(struct idr *, unsivoid *idr_replace (struct idr *,void *,unsigned long id);void idr_destroy (struct idr *); +struct __class_idr { +struct idr *idr; +int id; +}; + +#define idr_null ((struct __class_idr){ NULL, -1 }) +#define take_idr_id (id) __get_and_null(id, idr_null) + +DEFINE_CLASS(idr_alloc, struct __class_idr, + if (_T.id >= 0) idr_remove(_T.idr, _T.id), + ((struct __class_idr ){ + .idr = idr, + .id =idr_alloc (idr, ptr, start, end, gfp), + }), + struct idr *idr, void *ptr, int start, int end, gfp_t gfp); +/** * idr_init_base() - Initialise an IDR. * @idr: IDR handle.
kernel/events/core.c +29 -42
113 lines not on this page · raw message
Peter Zijlstra
maintainerPATCH v2 12/24 perf: Simplify perf_init_event()
In reply to Peter Zijlstra--- kernel/events/core.c | 31 ++++++++++++------------------- 1 file changed, 12 insertions(+), 19 deletions(-)
kernel/events/core.c +12 -19
@@ -11942,10 +11942,10 @@ static int perf_try_init_event(struct pmstatic struct pmu *perf_init_event (struct perf_event *event) {bool extended_type = false; -int idx, type, ret;struct pmu *pmu; +int type, ret; - idx =srcu_read_lock (&pmus_srcu); +guard (srcu)(&pmus_srcu);/* * Save original type before calling pmu->event_init() since certain
@@ -11958,7 +11958,7 @@ static struct pmu *perf_init_event(struc pmu = event->parent->pmu; ret =perf_try_init_event (pmu, event);if (!ret) - goto unlock; +return pmu; }/*
@@ -11977,13 +11977,12 @@ static struct pmu *perf_init_event(struc } again: -rcu_read_lock (); - pmu =idr_find (&pmu_idr, type); -rcu_read_unlock (); +scoped_guard (rcu) + pmu =idr_find (&pmu_idr, type);if (pmu) {if (event->attr.type != type && type !=PERF_TYPE_RAW && !(pmu->capabilities &PERF_PMU_CAP_EXTENDED_HW_TYPE )) - goto fail; +return ERR_PTR (-ENOENT ); ret =perf_try_init_event (pmu, event);if (ret == -ENOENT && event->attr.type != type && !extended_type) {
@@ -11992,27 +11991,21 @@ static struct pmu *perf_init_event(struc }if (ret) - pmu =ERR_PTR (ret); +return ERR_PTR (ret); - goto unlock; +return pmu; }list_for_each_entry_rcu (pmu, &pmus, entry,lockdep_is_held (&pmus_srcu)) { ret =perf_try_init_event (pmu, event);if (!ret) - goto unlock; +return pmu; -if (ret != -ENOENT ) { - pmu =ERR_PTR (ret); - goto unlock; - } +if (ret != -ENOENT ) +return ERR_PTR (ret); } -fail: - pmu =ERR_PTR (-ENOENT ); -unlock: -srcu_read_unlock (&pmus_srcu, idx); -return pmu; +return ERR_PTR (-ENOENT ); }static void attach_sb_event (struct perf_event *event)
Peter Zijlstra
maintainerPATCH v2 13/24 perf: Simplify perf_event_alloc()
In reply to Peter ZijlstraUsing the previous simplifications, transition perf_event_alloc() to the cleanup way of things -- reducing error path magic.
--- kernel/events/core.c | 60 +++++++++++++++++++-------------------------------- 1 file changed, 23 insertions(+), 37 deletions(-)
kernel/events/core.c +22 -37
@@ -5360,6 +5361,8 @@ static void __free_event(struct perf_evecall_rcu (&event->rcu_head, free_event_rcu); } +DEFINE_FREE(__free_event, struct perf_event *, if (_T) __free_event(_T)) + /* vs perf_event_alloc() success */static void _free_event (struct perf_event *event) {
@@ -12238,7 +12241,6 @@ perf_event_alloc(struct perf_event_attr void *context, int cgroup_fd) {struct pmu *pmu; -struct perf_event *event;struct hw_perf_event *hwc;long err = -EINVAL ;int node;
@@ -12253,8 +12255,8 @@ perf_event_alloc(struct perf_event_attr } node = (cpu >=0 ) ?cpu_to_node (cpu) :-1 ; - event =kmem_cache_alloc_node (perf_event_cache,GFP_KERNEL | __GFP_ZERO, - node); + struct perf_event *event __free(__free_event) = +kmem_cache_alloc_node (perf_event_cache,GFP_KERNEL | __GFP_ZERO, node);if (!event)return ERR_PTR (-ENOMEM );
@@ -12361,65 +12363,53 @@ perf_event_alloc(struct perf_event_attr* See perf_output_read(). */ if (has_inherit_and_sample_read (attr) && !(attr->sample_type &PERF_SAMPLE_TID )) - goto err; +return ERR_PTR (-EINVAL );if (!has_branch_stack (event)) event->attr.branch_sample_type =0 ; pmu =perf_init_event (event); -if (IS_ERR (pmu)) { - err =PTR_ERR (pmu); - goto err; - } +if (IS_ERR (pmu)) +return (void *)pmu;/* * Disallow uncore-task events. Similarly, disallow uncore-cgroup * events (they don't make sense as the cgroup will be different * on other CPUs in the uncore mask). */ -if (pmu->task_ctx_nr == perf_invalid_context && (task || cgroup_fd !=-1 )) { - err = -EINVAL ; - goto err; - } +if (pmu->task_ctx_nr == perf_invalid_context && (task || cgroup_fd !=-1 )) +return ERR_PTR (-EINVAL );if (event->attr.aux_output && (!(pmu->capabilities &PERF_PMU_CAP_AUX_OUTPUT ) || - event->attr.aux_pause || event->attr.aux_resume)) { - err = -EOPNOTSUPP ; - goto err; - } + event->attr.aux_pause || event->attr.aux_resume)) +return ERR_PTR (-EOPNOTSUPP ); -if (event->attr.aux_pause && event->attr.aux_resume) { - err = -EINVAL ; - goto err; - } +if (event->attr.aux_pause && event->attr.aux_resume) +return ERR_PTR (-EINVAL );if (event->attr.aux_start_paused) { -if (!(pmu->capabilities &PERF_PMU_CAP_AUX_PAUSE )) { - err = -EOPNOTSUPP ; - goto err; - } +if (!(pmu->capabilities &PERF_PMU_CAP_AUX_PAUSE )) +return ERR_PTR (-EOPNOTSUPP ); event->hw.aux_paused =1 ; }if (cgroup_fd !=-1 ) { err =perf_cgroup_connect (cgroup_fd, event, attr, group_leader);if (err) - goto err; +return ERR_PTR (err); } err =exclusive_event_init (event);if (err) - goto err; +return ERR_PTR (err);if (has_addr_filter (event)) { event->addr_filter_ranges =kcalloc (pmu->nr_addr_filters,sizeof (struct perf_addr_filter_range ),GFP_KERNEL ); -if (!event->addr_filter_ranges) { - err = -ENOMEM ; - goto err; - } +if (!event->addr_filter_ranges) +return ERR_PTR (-ENOMEM );/* * Clone the parent's vma offsets: they are valid until exec()
@@ -12443,23 +12433,19 @@ perf_event_alloc(struct perf_event_attrif (event->attr.sample_type &PERF_SAMPLE_CALLCHAIN ) { err =get_callchain_buffers (attr->sample_max_stack);if (err) - goto err; +return ERR_PTR (err); event->attach_state |=PERF_ATTACH_CALLCHAIN ; } } err =security_perf_event_alloc (event);if (err) - goto err; +return ERR_PTR (err);/* symmetric to unaccount_event() in _free_event() */ account_event (event); -return event; - -err: -__free_event (event); -return ERR_PTR (err); +return_ptr (event); } static int perf_copy_attr(struct perf_event_attr __user *uattr,
Peter Zijlstra
maintainerPATCH v2 14/24 perf: Merge pmu_disable_count into cpu_pmu_context
In reply to Peter ZijlstraBecause it makes no sense to have two per-cpu allocations per pmu.
--- include/linux/perf_event.h | 2 +- kernel/events/core.c | 12 ++++-------- 2 files changed, 5 insertions(+), 9 deletions(-)
include/linux/perf_event.h +1 -1
@@ -343,7 +343,6 @@ struct pmu {*/ unsigned int scope; -int __percpu *pmu_disable_count;struct perf_cpu_pmu_context __percpu *cpu_pmu_context;atomic_t exclusive_cnt;/* < 0: cpu; > 0: tsk */ int task_ctx_nr;
@@ -1031,6 +1030,7 @@ struct perf_cpu_pmu_context {int active_oncpu;int exclusive; +int pmu_disable_count;raw_spinlock_t hrtimer_lock;struct hrtimer hrtimer;
kernel/events/core.c +4 -8
@@ -1174,21 +1174,22 @@ static int perf_mux_hrtimer_restart_ipi(void perf_pmu_disable (struct pmu *pmu) { -int *count =this_cpu_ptr (pmu->pmu_disable_count); +int *count = &this_cpu_ptr (pmu->cpu_pmu_context)->pmu_disable_count;if (!(*count)++) pmu->pmu_disable (pmu); }void perf_pmu_enable (struct pmu *pmu) { -int *count =this_cpu_ptr (pmu->pmu_disable_count); +int *count = &this_cpu_ptr (pmu->cpu_pmu_context)->pmu_disable_count;if (!--(*count)) pmu->pmu_enable (pmu); }static void perf_assert_pmu_disabled (struct pmu *pmu) { -WARN_ON_ONCE (*this_cpu_ptr (pmu->pmu_disable_count) ==0 ); +int *count = &this_cpu_ptr (pmu->cpu_pmu_context)->pmu_disable_count; +WARN_ON_ONCE (*count ==0 ); }static inline void perf_pmu_read (struct perf_event *event)
@@ -11860,7 +11861,6 @@ static bool idr_cmpxchg(struct idr *idr,static void perf_pmu_free (struct pmu *pmu) { -free_percpu (pmu->pmu_disable_count);if (pmu_bus_running && pmu->dev && pmu->dev !=PMU_NULL_DEV ) {if (pmu->nr_addr_filters)device_remove_file (pmu->dev, &dev_attr_nr_addr_filters);
@@ -11879,10 +11879,6 @@ int perf_pmu_register(struct pmu *_pmu, struct pmu *pmu __free(pmu_unregister) = _pmu;guard (mutex)(&pmus_lock); - pmu->pmu_disable_count =alloc_percpu (int); -if (!pmu->pmu_disable_count) -return -ENOMEM ; -if (WARN_ONCE (!name,"Can not register anonymous pmu.\n" ))return -EINVAL ;
Peter Zijlstra
maintainerPATCH v2 15/24 perf: Add this_cpc() helper
In reply to Peter ZijlstraAs a preparation for adding yet another indirection.
--- kernel/events/core.c | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-)
kernel/events/core.c +18 -16
@@ -1176,23 +1176,28 @@ static int perf_mux_hrtimer_restart_ipi(return perf_mux_hrtimer_restart (arg); } +static __always_inline struct perf_cpu_pmu_context *this_cpc(struct pmu *pmu) +{ +return this_cpu_ptr (pmu->cpu_pmu_context); +} +void perf_pmu_disable (struct pmu *pmu) { -int *count = &this_cpu_ptr (pmu->cpu_pmu_context)->pmu_disable_count; +int *count = &this_cpc (pmu)->pmu_disable_count;if (!(*count)++) pmu->pmu_disable (pmu); }void perf_pmu_enable (struct pmu *pmu) { -int *count = &this_cpu_ptr (pmu->cpu_pmu_context)->pmu_disable_count; +int *count = &this_cpc (pmu)->pmu_disable_count;if (!--(*count)) pmu->pmu_enable (pmu); }static void perf_assert_pmu_disabled (struct pmu *pmu) { -int *count = &this_cpu_ptr (pmu->cpu_pmu_context)->pmu_disable_count; +int *count = &this_cpc (pmu)->pmu_disable_count;WARN_ON_ONCE (*count ==0 ); }
@@ -2304,7 +2309,7 @@ static void event_sched_out(struct perf_event *event, struct perf_event_context *ctx) { struct perf_event_pmu_context *epc = event->pmu_ctx; -struct perf_cpu_pmu_context *cpc =this_cpu_ptr (epc->pmu->cpu_pmu_context); +struct perf_cpu_pmu_context *cpc =this_cpc (epc->pmu);enum perf_event_state state =PERF_EVENT_STATE_INACTIVE ;// XXX cpc serialization, probably per-cpu IRQ disabled
@@ -2445,9 +2450,8 @@ __perf_remove_from_context(struct perf_e pmu_ctx->rotate_necessary =0 ;if (ctx->task && ctx->is_active) { -struct perf_cpu_pmu_context *cpc; +struct perf_cpu_pmu_context *cpc =this_cpc (pmu_ctx->pmu); - cpc =this_cpu_ptr (pmu_ctx->pmu->cpu_pmu_context);WARN_ON_ONCE (cpc->task_epc && cpc->task_epc != pmu_ctx); cpc->task_epc =NULL ; }
@@ -2585,7 +2589,7 @@ static int event_sched_in(struct perf_event *event, struct perf_event_context *ctx) { struct perf_event_pmu_context *epc = event->pmu_ctx; -struct perf_cpu_pmu_context *cpc =this_cpu_ptr (epc->pmu->cpu_pmu_context); +struct perf_cpu_pmu_context *cpc =this_cpc (epc->pmu);int ret =0 ;WARN_ON_ONCE (event->ctx != ctx);
@@ -2692,7 +2696,7 @@ group_sched_in(struct perf_event *group_static int group_can_go_on (struct perf_event *event,int can_add_hw) {struct perf_event_pmu_context *epc = event->pmu_ctx; -struct perf_cpu_pmu_context *cpc =this_cpu_ptr (epc->pmu->cpu_pmu_context); +struct perf_cpu_pmu_context *cpc =this_cpc (epc->pmu);/* * Groups consisting entirely of software events can always go on.
@@ -3315,9 +3319,8 @@ static void __pmu_ctx_sched_out(struct pstruct pmu *pmu = pmu_ctx->pmu;if (ctx->task && !(ctx->is_active &EVENT_ALL )) { -struct perf_cpu_pmu_context *cpc; +struct perf_cpu_pmu_context *cpc =this_cpc (pmu); - cpc =this_cpu_ptr (pmu->cpu_pmu_context);WARN_ON_ONCE (cpc->task_epc && cpc->task_epc != pmu_ctx); cpc->task_epc =NULL ; }
@@ -3565,7 +3568,7 @@ static void perf_ctx_sched_task_cb(strucstruct perf_cpu_pmu_context *cpc;list_for_each_entry (pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { - cpc =this_cpu_ptr (pmu_ctx->pmu->cpu_pmu_context); + cpc =this_cpc (pmu_ctx->pmu);if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task) pmu_ctx->pmu->sched_task (pmu_ctx, sched_in);
@@ -3674,7 +3677,7 @@ static DEFINE_PER_CPU(int, perf_sched_cbvoid perf_sched_cb_dec (struct pmu *pmu) { -struct perf_cpu_pmu_context *cpc =this_cpu_ptr (pmu->cpu_pmu_context); +struct perf_cpu_pmu_context *cpc =this_cpc (pmu);this_cpu_dec (perf_sched_cb_usages);barrier ();
@@ -3686,7 +3689,7 @@ void perf_sched_cb_dec(struct pmu *pmu)void perf_sched_cb_inc (struct pmu *pmu) { -struct perf_cpu_pmu_context *cpc =this_cpu_ptr (pmu->cpu_pmu_context); +struct perf_cpu_pmu_context *cpc =this_cpc (pmu);if (!cpc->sched_cb_usage++)list_add (&cpc->sched_cb_entry,this_cpu_ptr (&sched_cb_list));
@@ -3810,7 +3813,7 @@ static void __link_epc(struct perf_eventif (!pmu_ctx->ctx->task)return ; - cpc =this_cpu_ptr (pmu_ctx->pmu->cpu_pmu_context); + cpc =this_cpc (pmu_ctx->pmu);WARN_ON_ONCE (cpc->task_epc && cpc->task_epc != pmu_ctx); cpc->task_epc = pmu_ctx; }
@@ -3939,10 +3942,9 @@ static int merge_sched_in(struct perf_evperf_cgroup_event_disable (event, ctx);perf_event_set_state (event,PERF_EVENT_STATE_ERROR ); }else { -struct perf_cpu_pmu_context *cpc; +struct perf_cpu_pmu_context *cpc =this_cpc (event->pmu_ctx->pmu); event->pmu_ctx->rotate_necessary =1 ; - cpc =this_cpu_ptr (event->pmu_ctx->pmu->cpu_pmu_context);perf_mux_hrtimer_restart (cpc);group_update_userpage (event); }
Peter Zijlstra
maintainerPATCH v2 16/24 perf: Detach perf_cpu_pmu_context and pmu lifetimes
In reply to Peter ZijlstraIn prepration for being able to unregister a pmu with existing events, it becomes important to detach struct perf_cpu_pmu_context lifetimes from that of struct pmu.
Notably perf_cpu_pmu_context embeds a perf_event_pmu_context that can stay referenced until the last event goes.
--- include/linux/perf_event.h | 4 +-- kernel/events/core.c | 56 +++++++++++++++++++++++++++++++++++++-------- 2 files changed, 49 insertions(+), 11 deletions(-)
include/linux/perf_event.h +2 -2
@@ -336,7 +336,7 @@ struct pmu {*/ unsigned int scope; -struct perf_cpu_pmu_context __percpu *cpu_pmu_context; +struct perf_cpu_pmu_context __percpu **cpu_pmu_context;atomic_t exclusive_cnt;/* < 0: cpu; > 0: tsk */ int task_ctx_nr;int hrtimer_interval_ms;
@@ -901,7 +901,7 @@ struct perf_event_pmu_context {struct list_head pinned_active;struct list_head flexible_active; -/* Used to avoid freeing per-cpu perf_event_pmu_context */ +/* Used to identify the per-cpu perf_event_pmu_context */ unsigned int embedded : 1;unsigned int nr_events;
kernel/events/core.c +47 -9
107 lines not on this page · raw message
Peter Zijlstra
maintainerPATCH v2 17/24 perf: Introduce perf_free_addr_filters()
In reply to Peter ZijlstraReplace _free_event()'s use of perf_addr_filters_splice()s use with an explicit perf_free_addr_filters() with the explicit propery that it is able to be called a second time without ill effect.
Most notable, referencing event->pmu must be avoided when there are no filters left (from eg a previous call).
--- kernel/events/core.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
kernel/events/core.c +13 -3
@@ -5307,8 +5307,7 @@ static bool exclusive_event_installable(return true; } -static void perf_addr_filters_splice (struct perf_event *event, -struct list_head *head); +static void perf_free_addr_filters (struct perf_event *event);static void perf_pending_task_sync (struct perf_event *event) {
@@ -5407,7 +5406,7 @@ static void _free_event(struct perf_even }perf_event_free_bpf_prog (event); -perf_addr_filters_splice (event,NULL ); +perf_free_addr_filters (event);__free_event (event); }
@@ -10880,6 +10882,17 @@ static void perf_addr_filters_splice(strfree_filters_list (&list); } +static void perf_free_addr_filters (struct perf_event *event) +{ +/* +* Used during free paths, there is no concurrency. +*/ +if (list_empty (&event->addr_filters.list)) +return ; + +perf_addr_filters_splice (event,NULL ); +} +/* * Scan through mm's vmas and see if one of them matches the * @filter; if so, adjust filter's address range.
Peter Zijlstra
maintainerPATCH v2 18/24 perf: Robustify perf_event_free_bpf_prog()
In reply to Peter ZijlstraEnsure perf_event_free_bpf_prog() is safe to call a second time; notably without making any references to event->pmu when there is no prog left.
--- kernel/events/core.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
kernel/events/core.c +3 -0
@@ -10782,6 +10781,9 @@ int perf_event_set_bpf_prog(struct perf_void perf_event_free_bpf_prog (struct perf_event *event) { +if (!event->prog) +return ; +if (!perf_event_is_tracing (event)) {perf_event_free_bpf_handler (event);return ;
Peter Zijlstra
maintainerPATCH v2 19/24 perf: Simplify perf_mmap() control flow
In reply to Peter Zijlstra if (c) {
X1;
} else {
Y;
goto l;
} X2;
l:
into:
if (c) {
X1;
X2;
} else {
Y;
}--- kernel/events/core.c | 73 ++++++++++++++++++++++++--------------------------- 1 file changed, 35 insertions(+), 38 deletions(-)
kernel/events/core.c +35 -38
@@ -6664,6 +6664,41 @@ static int perf_mmap(struct file *file,if (vma->vm_pgoff ==0 ) { nr_pages = (vma_size /PAGE_SIZE ) -1 ; + +/* +* If we have rb pages ensure they're a power-of-two number, so we +* can do bitmasks instead of modulo. +*/ +if (nr_pages !=0 && !is_power_of_2 (nr_pages)) +return -EINVAL ; + +if (vma_size !=PAGE_SIZE * (1 + nr_pages)) +return -EINVAL ; + +WARN_ON_ONCE (event->ctx->parent_ctx); +again: +mutex_lock (&event->mmap_mutex); +if (event->rb) { +if (data_page_nr (event->rb) != nr_pages) { + ret = -EINVAL ; + goto unlock; + } + +if (!atomic_inc_not_zero (&event->rb->mmap_count)) { +/* +* Raced against perf_mmap_close(); remove the +* event and try again. +*/ +ring_buffer_attach (event,NULL ); +mutex_unlock (&event->mmap_mutex); + goto again; + } + + rb = event->rb; + goto unlock; + } + + user_extra = nr_pages +1 ; }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already
@@ -6723,47 +6758,9 @@ static int perf_mmap(struct file *file,atomic_set (&rb->aux_mmap_count,1 ); user_extra = nr_pages; - - goto accounting; } -/* -* If we have rb pages ensure they're a power-of-two number, so we -* can do bitmasks instead of modulo. -*/ -if (nr_pages !=0 && !is_power_of_2 (nr_pages)) -return -EINVAL ; - -if (vma_size !=PAGE_SIZE * (1 + nr_pages)) -return -EINVAL ; - -WARN_ON_ONCE (event->ctx->parent_ctx); -again: -mutex_lock (&event->mmap_mutex); -if (event->rb) { -if (data_page_nr (event->rb) != nr_pages) { - ret = -EINVAL ; - goto unlock; - } - -if (!atomic_inc_not_zero (&event->rb->mmap_count)) { -/* -* Raced against perf_mmap_close(); remove the -* event and try again. -*/ -ring_buffer_attach (event,NULL ); -mutex_unlock (&event->mmap_mutex); - goto again; - } -/* We need the rb to map pages. */ - rb = event->rb; - goto unlock; - } - - user_extra = nr_pages +1 ; - -accounting: user_lock_limit = sysctl_perf_event_mlock >> (PAGE_SHIFT -10 );/*
Re: [PATCH v2 19/24] perf: Simplify perf_mmap() control flow
Hi Peter,
Minor nit below:
kernel/events/core.c static int perf_mmap(struct file *file, · line 6696
Quoted 10 lines
kernel/events/core.c +10
-0+if (!atomic_inc_not_zero (&event->rb->mmap_count)) { +/* +* Raced against perf_mmap_close(); remove the +* event and try again. +*/ +ring_buffer_attach (event,NULL ); +mutex_unlock (&event->mmap_mutex); + goto again; + } +
here ...
Quoted 36 lines
kernel/events/core.c +5
-22+ rb = event->rb; + goto unlock; + } + + user_extra = nr_pages +1 ; }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already @@ -6723,47 +6758,9 @@ static int perf_mmap(struct file *file,atomic_set (&rb->aux_mmap_count,1 ); user_extra = nr_pages; - - goto accounting; } -/* -* If we have rb pages ensure they're a power-of-two number, so we -* can do bitmasks instead of modulo. -*/ -if (nr_pages !=0 && !is_power_of_2 (nr_pages)) -return -EINVAL ; - -if (vma_size !=PAGE_SIZE * (1 + nr_pages)) -return -EINVAL ; - -WARN_ON_ONCE (event->ctx->parent_ctx); -again: -mutex_lock (&event->mmap_mutex); -if (event->rb) { -if (data_page_nr (event->rb) != nr_pages) { - ret = -EINVAL ; - goto unlock; - } - -if (!atomic_inc_not_zero (&event->rb->mmap_count)) {
kernel/events/core.c static int perf_mmap(struct file *file, · line 6763
Quoted 10 lines
kernel/events/core.c +0
-9-/* -* Raced against perf_mmap_close(); remove the -* event and try again. -*/ -ring_buffer_attach (event,NULL ); -mutex_unlock (&event->mmap_mutex); - goto again; - } -/* We need the rb to map pages. */
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This comment should also go up. Keeping it here is misleading.
kernel/events/core.c static int perf_mmap(struct file *file, · line 6766
Quoted 10 lines
kernel/events/core.c +0
-7- rb = event->rb; - goto unlock; - } - - user_extra = nr_pages +1 ; - -accounting: user_lock_limit = sysctl_perf_event_mlock >> (PAGE_SHIFT -10 );/*
Thanks,
Ravi
Re: [PATCH v2 19/24] perf: Simplify perf_mmap() control flow
Quoted 65 lines · Ravi Bangoria
Hi Peter,
Minor nit below:
kernel/events/core.c static int perf_mmap(struct file *file, · line 6696
Quoted 10 lines
kernel/events/core.c +10
-0+if (!atomic_inc_not_zero (&event->rb->mmap_count)) { +/* +* Raced against perf_mmap_close(); remove the +* event and try again. +*/ +ring_buffer_attach (event,NULL ); +mutex_unlock (&event->mmap_mutex); + goto again; + } +here ...
Quoted 36 lines
kernel/events/core.c +5
-22+ rb = event->rb; + goto unlock; + } + + user_extra = nr_pages +1 ; }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already @@ -6723,47 +6758,9 @@ static int perf_mmap(struct file *file,atomic_set (&rb->aux_mmap_count,1 ); user_extra = nr_pages; - - goto accounting; } -/* -* If we have rb pages ensure they're a power-of-two number, so we -* can do bitmasks instead of modulo. -*/ -if (nr_pages !=0 && !is_power_of_2 (nr_pages)) -return -EINVAL ; - -if (vma_size !=PAGE_SIZE * (1 + nr_pages)) -return -EINVAL ; - -WARN_ON_ONCE (event->ctx->parent_ctx); -again: -mutex_lock (&event->mmap_mutex); -if (event->rb) { -if (data_page_nr (event->rb) != nr_pages) { - ret = -EINVAL ; - goto unlock; - } - -if (!atomic_inc_not_zero (&event->rb->mmap_count)) {kernel/events/core.c static int perf_mmap(struct file *file, · line 6763
Quoted 10 lines
kernel/events/core.c +0
-9-/* -* Raced against perf_mmap_close(); remove the -* event and try again. -*/ -ring_buffer_attach (event,NULL ); -mutex_unlock (&event->mmap_mutex); - goto again; - } -/* We need the rb to map pages. */ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 1 line
This comment should also go up. Keeping it here is misleading.
Yeah, I did that in the conflict resolution (which conflicted in this very area), so the end result in tip:perf/core (or tip:master) should be fine as-is, correct?
Thanks,
Ingo
Re: [PATCH v2 19/24] perf: Simplify perf_mmap() control flow
Hi Ingo,
Quoted 9 lines
Quoted 5 lines
Review
Quoted 2 lines
(quoted diff) +0
-1- /* We need the rb to map pages. */^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^This comment should also go up. Keeping it here is misleading.
Yeah, I did that in the conflict resolution (which conflicted in this very area), so the end result in tip:perf/core (or tip:master) should be fine as-is, correct?
Yes, tip:perf/core looks good.
Thanks,
Ravi
Re: [PATCH v2 19/24] perf: Simplify perf_mmap() control flow
Quoted 5 lines · Ravi Bangoria
Hi Ingo,
Quoted 3 lines
Quoted 3 lines
Quoted 2 lines
(quoted diff) +0
-1- /* We need the rb to map pages. */
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^This comment should also go up. Keeping it here is misleading.
Yeah, I did that in the conflict resolution (which conflicted in this very area), so the end result in tip:perf/core (or tip:master) should be fine as-is, correct?
Yes, tip:perf/core looks good.
Great, thank you for checking!
I also picked up:
bfd33e88addd perf/core: Fix perf_mmap() failure path
and have added your Reviewed-by tags to the series.
Thanks,
Ingo
Peter Zijlstra
maintainerPATCH v2 20/24 perf: Fix perf_mmap() failure path
In reply to Peter ZijlstraWhen f_ops->mmap() returns failure, m_ops->close() is *not* called.
--- kernel/events/core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/events/core.c +1 -1
@@ -6852,7 +6852,7 @@ static int perf_mmap(struct file *file,if (!ret) ret =map_range (rb, vma); -if (event->pmu->event_mapped) +if (!ret && event->pmu->event_mapped) event->pmu->event_mapped (event, vma->vm_mm);return ret;
Peter Zijlstra
maintainerPATCH v2 21/24 perf: Further simplify perf_mmap()
In reply to Peter Zijlstra--- kernel/events/core.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-)
kernel/events/core.c +10 -11
@@ -6661,9 +6661,18 @@ static int perf_mmap(struct file *file,return ret; vma_size = vma->vm_end - vma->vm_start; + nr_pages = vma_size /PAGE_SIZE ; + +if (nr_pages >INT_MAX ) +return -ENOMEM ; + +if (vma_size !=PAGE_SIZE * nr_pages) +return -EINVAL ; + + user_extra = nr_pages;if (vma->vm_pgoff ==0 ) { - nr_pages = (vma_size /PAGE_SIZE ) -1 ; + nr_pages -=1 ;/* * If we have rb pages ensure they're a power-of-two number, so we
@@ -6672,9 +6681,6 @@ static int perf_mmap(struct file *file,if (nr_pages !=0 && !is_power_of_2 (nr_pages))return -EINVAL ; -if (vma_size !=PAGE_SIZE * (1 + nr_pages)) -return -EINVAL ; -WARN_ON_ONCE (event->ctx->parent_ctx); again:mutex_lock (&event->mmap_mutex);
@@ -6697,8 +6703,6 @@ static int perf_mmap(struct file *file, rb = event->rb; goto unlock; } - - user_extra = nr_pages +1 ; }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already
@@ -6710,10 +6714,6 @@ static int perf_mmap(struct file *file,if (!event->rb)return -EINVAL ; - nr_pages = vma_size /PAGE_SIZE ; -if (nr_pages >INT_MAX ) -return -ENOMEM ; -mutex_lock (&event->mmap_mutex); ret = -EINVAL ;
@@ -6757,7 +6757,6 @@ static int perf_mmap(struct file *file, }atomic_set (&rb->aux_mmap_count,1 ); - user_extra = nr_pages; }/* We need the rb to map pages. */
Peter Zijlstra
maintainerPATCH v2 22/24 perf: Remove retry loop from perf_mmap()
In reply to Peter ZijlstraAFAICT there is no actual benefit from the mutex drop on re-try. The 'worst' case scenario is that we instantly re-gain the mutex without perf_mmap_close() getting it. So might as well make that the normal case.
Reflow the code to make the ring buffer detach case naturally flow into the no ring buffer case.
--- kernel/events/core.c | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-)
kernel/events/core.c +14 -9
@@ -6667,27 +6667,32 @@ static int perf_mmap(struct file *file,return -EINVAL ;WARN_ON_ONCE (event->ctx->parent_ctx); -again:mutex_lock (&event->mmap_mutex); +if (event->rb) {if (data_page_nr (event->rb) != nr_pages) { ret = -EINVAL ; goto unlock; } -if (!atomic_inc_not_zero (&event->rb->mmap_count)) { +if (atomic_inc_not_zero (&event->rb->mmap_count)) {/* -* Raced against perf_mmap_close(); remove the -* event and try again. +* Success -- managed to mmap() the same buffer +* multiple times. */ -ring_buffer_attach (event,NULL ); -mutex_unlock (&event->mmap_mutex); - goto again; + rb = event->rb; + ret =0 ; + goto unlock; } - rb = event->rb; - goto unlock; +/* +* Raced against perf_mmap_close()'s +* atomic_dec_and_mutex_lock() remove the +* event and continue as if !event->rb +*/ +ring_buffer_attach (event,NULL ); } + }else {/* * AUX area mapping: if rb->aux_nr_pages != 0, it's already
Peter Zijlstra
maintainerPATCH v2 23/24 perf: Lift event->mmap_mutex in perf_mmap()
In reply to Peter ZijlstraThis puts 'all' of perf_mmap() under single event->mmap_mutex.
--- kernel/events/core.c | 20 ++++++++------------ 1 file changed, 8 insertions(+), 12 deletions(-)
kernel/events/core.c +8 -12
@@ -6643,7 +6643,7 @@ static int perf_mmap(struct file *file,unsigned long vma_size;unsigned long nr_pages;long user_extra =0 , extra =0 ; -int ret =0 , flags =0 ; +int ret, flags =0 ;/* * Don't allow mmap() of inherited per-task counters. This would
@@ -6671,6 +6671,9 @@ static int perf_mmap(struct file *file, user_extra = nr_pages; +mutex_lock (&event->mmap_mutex); + ret = -EINVAL ; +if (vma->vm_pgoff ==0 ) { nr_pages -=1 ;
@@ -6679,16 +6682,13 @@ static int perf_mmap(struct file *file,* can do bitmasks instead of modulo. */ if (nr_pages !=0 && !is_power_of_2 (nr_pages)) -return -EINVAL ; + goto unlock;WARN_ON_ONCE (event->ctx->parent_ctx); -mutex_lock (&event->mmap_mutex);if (event->rb) { -if (data_page_nr (event->rb) != nr_pages) { - ret = -EINVAL ; +if (data_page_nr (event->rb) != nr_pages) goto unlock; - }if (atomic_inc_not_zero (&event->rb->mmap_count)) {/*
@@ -6716,12 +6716,6 @@ static int perf_mmap(struct file *file,*/ u64 aux_offset, aux_size; -if (!event->rb) -return -EINVAL ; - -mutex_lock (&event->mmap_mutex); - ret = -EINVAL ; - rb = event->rb;if (!rb) goto aux_unlock;
@@ -6832,6 +6826,8 @@ static int perf_mmap(struct file *file, rb->aux_mmap_locked = extra; } + ret =0 ; + unlock:if (!ret) {atomic_long_add (user_extra, &user->locked_vm);
Peter Zijlstra
maintainerPATCH v2 24/24 perf: Make perf_pmu_unregister() useable
In reply to Peter ZijlstraPreviously it was only safe to call perf_pmu_unregister() if there were no active events of that pmu around -- which was impossible to guarantee since it races all sorts against perf_init_event().
Rework the whole thing by:
- keeping track of all events for a given pmu
- 'hiding' the pmu from perf_init_event()
- waiting for the appropriate (s)rcu grace periods such that all prior references to the PMU will be completed
- detaching all still existing events of that pmu (see first point) and moving them to a new REVOKED state.
- actually freeing the pmu data.
Where notably the new REVOKED state must inhibit all event actions from reaching code that wants to use event->pmu.
--- include/linux/perf_event.h | 15 +- kernel/events/core.c | 266 ++++++++++++++++++++++++++++++++++++++++----- 2 files changed, 249 insertions(+), 32 deletions(-)
include/linux/perf_event.h +10 -5
@@ -325,6 +325,9 @@ struct perf_output_handle;struct pmu {struct list_head entry; +spinlock_t events_lock; +struct list_head events; +struct module *module;struct device *dev;struct device *parent;
@@ -632,9 +635,10 @@ struct perf_addr_filter_range {* enum perf_event_state - the states of an event: */ enum perf_event_state { -PERF_EVENT_STATE_DEAD =-4 , -PERF_EVENT_STATE_EXIT =-3 , -PERF_EVENT_STATE_ERROR =-2 , +PERF_EVENT_STATE_DEAD =-5 , +PERF_EVENT_STATE_REVOKED =-4 ,/* pmu gone, must not touch */ +PERF_EVENT_STATE_EXIT =-3 ,/* task died, still inherit */ +PERF_EVENT_STATE_ERROR =-2 ,/* scheduling error, can enable */ PERF_EVENT_STATE_OFF =-1 ,PERF_EVENT_STATE_INACTIVE =0 ,PERF_EVENT_STATE_ACTIVE =1 ,
@@ -875,6 +879,7 @@ struct perf_event {void *security;#endif struct list_head sb_list; +struct list_head pmu_list;/* * Certain events gets forwarded to another pmu internally by over-
@@ -1126,7 +1131,7 @@ extern void perf_aux_output_flag(structextern void perf_event_itrace_started (struct perf_event *event);extern int perf_pmu_register (struct pmu *pmu,const char *name,int type); -extern void perf_pmu_unregister (struct pmu *pmu); +extern int perf_pmu_unregister (struct pmu *pmu);extern void __perf_event_task_sched_in (struct task_struct *prev,struct task_struct *task);
@@ -1737,7 +1742,7 @@ static inline bool needs_branch_stack(ststatic inline bool has_aux (struct perf_event *event) { -return event->pmu->setup_aux; +return event->pmu && event->pmu->setup_aux; }static inline bool has_aux_action (struct perf_event *event)
kernel/events/core.c +239 -27
498 lines not on this page · raw message
Re: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Hi Peter,
include/linux/perf_event.h static inline bool needs_branch_stack(st · line 1746
Quoted 7 lines
include/linux/perf_event.h +1
-1@@ -1737,7 +1742,7 @@ static inline bool needs_branch_stack(ststatic inline bool has_aux (struct perf_event *event) { -return event->pmu->setup_aux; +return event->pmu && event->pmu->setup_aux; }
this ...
Quoted 10 lines
kernel/events/core.c +1
-1@@ -6412,7 +6444,7 @@ static void perf_mmap_open(struct vm_areif (vma->vm_pgoff)atomic_inc (&event->rb->aux_mmap_count); -if (event->pmu->event_mapped) +if (event->pmu && event->pmu->event_mapped) event->pmu->event_mapped (event, vma->vm_mm); }
kernel/events/core.c static void perf_mmap_close(struct vm_ar · line 6472
Quoted 7 lines
kernel/events/core.c +2
-1@@ -6435,7 +6467,8 @@ static void perf_mmap_close(struct vm_arunsigned long size =perf_data_size (rb);bool detach_rest = false; -if (event->pmu->event_unmapped) +/* FIXIES vs perf_pmu_unregister() */ +if (event->pmu && event->pmu->event_unmapped) event->pmu->event_unmapped (event, vma->vm_mm);
these two ...
kernel/events/core.c static int perf_mmap(struct file *file, · line 6884
Quoted 7 lines
kernel/events/core.c +1
-1@@ -6837,7 +6880,7 @@ static int perf_mmap(struct file *file,if (!ret) ret =map_range (rb, vma); -if (!ret && event->pmu->event_mapped) +if (!ret && event->pmu && event->pmu->event_mapped) event->pmu->event_mapped (event, vma->vm_mm);
... and this one.
There is no serialization at all these places. i.e. event->pmu can become NULL in between two checks.
CPU0: event->pmu = NULL
|
V
CPU1: if (event->pmu && event->pmu->blahblah())Thanks,
Ravi
Peter Zijlstra
maintainerRe: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Quoted 11 lines · Ravi Bangoria
Hi Peter,
Review static inline bool needs_branch_stack(st · line 1746
Quoted 7 lines
(quoted diff) +1
-1@@ -1737,7 +1742,7 @@ static inline bool needs_branch_stack(st static inline bool has_aux(struct perf_event *event) { - return event->pmu->setup_aux; + return event->pmu && event->pmu->setup_aux; }this ...
Bah, I'll try and trace that.
Quoted 24 lines
Quoted 10 lines
(quoted diff) +1
-1@@ -6412,7 +6444,7 @@ static void perf_mmap_open(struct vm_are if (vma->vm_pgoff) atomic_inc(&event->rb->aux_mmap_count); - if (event->pmu->event_mapped) + if (event->pmu && event->pmu->event_mapped) event->pmu->event_mapped(event, vma->vm_mm); }Review static void perf_mmap_close(struct vm_ar · line 6472
Quoted 7 lines
(quoted diff) +2
-1@@ -6435,7 +6467,8 @@ static void perf_mmap_close(struct vm_ar unsigned long size = perf_data_size(rb); bool detach_rest = false; - if (event->pmu->event_unmapped) + /* FIXIES vs perf_pmu_unregister() */ + if (event->pmu && event->pmu->event_unmapped) event->pmu->event_unmapped(event, vma->vm_mm);these two ...
Quoted 4 lines
(quoted diff) +0
-0@@ -6837,7 +6880,7 @@ static int perf_mmap(struct file *file, if (!ret) ret = map_range(rb, vma);
Quoted 5 lines
Review static int perf_mmap(struct file *file, · line 6884
Quoted 3 lines
(quoted diff) +1
-1- if (!ret && event->pmu->event_mapped) + if (!ret && event->pmu && event->pmu->event_mapped) event->pmu->event_mapped(event, vma->vm_mm);... and this one.
These are relatively simple to fix, something like so. This relies on the fact that if there's mapped functions the whole unregister thing won't happen.
---
kernel/events/core.c +22 -6
@@ -6432,9 +6432,22 @@ void ring_buffer_put(struct perf_buffercall_rcu (&rb->rcu_head, rb_free_rcu); } +typedef void (*mapped_f )(struct perf_event *event,struct mm_struct *mm); + +#define get_mapped (event, func) \ +({ struct pmu *pmu; \ + mapped_f f = NULL; \ + guard(rcu)(); \ + pmu = READ_ONCE(event->pmu); \ + if (pmu) \ + f = pmu->func; \ + f; \ +}) +static void perf_mmap_open (struct vm_area_struct *vma) {struct perf_event *event = vma->vm_file->private_data; +mapped_f mapped =get_mapped (event, event_mapped);atomic_inc (&event->mmap_count);atomic_inc (&event->rb->mmap_count);
@@ -6442,8 +6455,8 @@ static void perf_mmap_open(struct vm_areif (vma->vm_pgoff)atomic_inc (&event->rb->aux_mmap_count); -if (event->pmu && event->pmu->event_mapped) - event->pmu->event_mapped (event, vma->vm_mm); +if (mapped) +mapped (event, vma->vm_mm); }static void perf_pmu_output_stop (struct perf_event *event);
@@ -6459,6 +6472,7 @@ static void perf_pmu_output_stop(structstatic void perf_mmap_close (struct vm_area_struct *vma) {struct perf_event *event = vma->vm_file->private_data; +mapped_f unmapped =get_mapped (event, event_unmapped);struct perf_buffer *rb =ring_buffer_get (event);struct user_struct *mmap_user = rb->mmap_user;int mmap_locked = rb->mmap_locked;
@@ -6466,8 +6480,8 @@ static void perf_mmap_close(struct vm_arbool detach_rest = false;/* FIXIES vs perf_pmu_unregister() */ -if (event->pmu && event->pmu->event_unmapped) - event->pmu->event_unmapped (event, vma->vm_mm); +if (unmapped) +unmapped (event, vma->vm_mm);/* * The AUX buffer is strictly a sub-buffer, serialize using aux_mutex
@@ -6660,6 +6674,7 @@ static int perf_mmap(struct file *file,unsigned long nr_pages;long user_extra =0 , extra =0 ;int ret, flags =0 ; +mapped_f mapped;/* * Don't allow mmap() of inherited per-task counters. This would
@@ -6878,8 +6893,9 @@ static int perf_mmap(struct file *file,if (!ret) ret =map_range (rb, vma); -if (!ret && event->pmu && event->pmu->event_mapped) - event->pmu->event_mapped (event, vma->vm_mm); + mapped =get_mapped (event, event_mapped); +if (mapped) +mapped (event, vma->vm_mm);return ret; }
Re: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
In reply to Peter ZijlstraQuoted 5 lines
kernel/events/core.c +0
-1@@ -13467,7 +13665,13 @@ perf_event_exit_event(struct perf_event* Kick perf_poll() for is_event_hup(); */ perf_event_wakeup (parent_event); -free_event (event);
kernel/events/core.c perf_event_exit_event(struct perf_event · line 13677
Quoted 10 lines
kernel/events/core.c +7
-0+/* +* pmu_detach_event() will have an extra refcount. +*/ +if (revoke) +put_event (event); +else +free_event (event);put_event (parent_event);return ; }
There is a race between do_exit() and perf_pmu_unregister():
Assume: a Parent process with 'event1' and a Child process with an inherited 'event2'.
CPU 1 CPU 2
Child process exiting ...
1 do_exit()
2 perf_event_exit_task()
3 perf_event_exit_task_context()
4 mutex_lock(&child_ctx->mutex);
5 list_for_each_entry_safe(&child_ctx->event_list)
6 /* picks event2. event2->refcount is 1 */
7 perf_pmu_unregister()
8 pmu_detach_events()
9 pmu_get_event(pmu) /* Picks event2 */
10 atomic_long_inc_not_zero(&event->refcount) /* event2 */
11 /* event2->refcount becomes 2 */
12 pmu_detach_event() /* event2 */
13 perf_event_ctx_lock(); /* event2 */
14 /* event2->refcount became 2 (by CPU 2) */ .
15 perf_event_exit_event() /* event2 */ . /* CPU 1 is holding ctx->mutex of
16 if (parent_event) { /* true */ . child process. Waiting ... */
17 if (revoke) /* false */ .
18 else .
19 free_event(); /* event2 */ .
20 WARN() .^^^^^^
This race manifests as:
unexpected event refcount: 2; ptr=00000000c6ca2049
WARNING: CPU: 57 PID: 9729 at kernel/events/core.c:5439 free_event+0x48/0x60 RIP: 0010:free_event+0x48/0x60 Call Trace: ? free_event+0x48/0x60 perf_event_exit_event.isra.0+0x60/0xf0 perf_event_exit_task+0x1e1/0x280 do_exit+0x327/0xb00
The race continues... now, with the stale child2->parent pointer:
CPU 1 CPU 2
15 perf_event_exit_event() /* event2 */ . /* CPU 1 is holding ctx->mutex of
16 if (parent_event) { /* true */ . child process. Waiting ... */
17 if (revoke) /* false */ .
18 else .
19 free_event(); /* event2 */ .
20 WARN() .
21 put_event(parent_event); /* event1 */ .
22 /* event1->refcount becomes 1 */ .
23 } .
24 +- mutex_unlock(&child_ctx->mutex); . /* Acquired child's ctx->mutex */
25 __pmu_detach_event() /* event2 */ 26 perf_event_exit_event() /* event2 */
27 if (parent_event) { /* true, BUT FALSE POSITIVE. */
28 if (revoke) /* true */
29 put_event(); /* event2 */
30 /* event2->refcount becomes 1 */
31 put_event(parent_event); /* event1 */
32 /* event1->refcount becomes 0 */
^^^^^^^^^^^^^^^^^^^^^^^^^^
This can manifest as some random bug. I guess, perf_event_exit_event() should also check for (event->attach_state & PERF_ATTACH_CHILD) when event->parent != NULL.
Thanks,
Ravi
Peter Zijlstra
maintainerRe: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Quoted 81 lines · Ravi Bangoria
Quoted 5 lines
(quoted diff) +0
-1@@ -13467,7 +13665,13 @@ perf_event_exit_event(struct perf_event * Kick perf_poll() for is_event_hup(); */ perf_event_wakeup(parent_event); - free_event(event);Review perf_event_exit_event(struct perf_event · line 13677
Quoted 10 lines
(quoted diff) +7
-0+ /* + * pmu_detach_event() will have an extra refcount. + */ + if (revoke) + put_event(event); + else + free_event(event); put_event(parent_event); return; }There is a race between
do_exit()andperf_pmu_unregister():Assume: a Parent process with 'event1' and a Child process with an inherited 'event2'.
CPU 1 CPU 2
Child process exiting ...
1do_exit()
2perf_event_exit_task()
3perf_event_exit_task_context()4 mutex_lock(&child_ctx->mutex); 5 list_for_each_entry_safe(&child_ctx->event_list) 6 /* picks event2. event2->refcount is 1 */ 7 perf_pmu_unregister() 8 pmu_detach_events() 9 pmu_get_event(pmu) /* Picks event2 */ 10 atomic_long_inc_not_zero(&event->refcount) /* event2 */ 11 /* event2->refcount becomes 2 */ 12 pmu_detach_event() /* event2 */ 13 perf_event_ctx_lock(); /* event2 */ 14 /* event2->refcount became 2 (by CPU 2) */ . 15 perf_event_exit_event() /* event2 */ . /* CPU 1 is holding ctx->mutex of 16 if (parent_event) { /* true */ . child process. Waiting ... */ 17 if (revoke) /* false */ . 18 else . 19 free_event(); /* event2 */ . 20 WARN() .^^^^^^This race manifests as:
unexpected event refcount: 2; ptr=00000000c6ca2049
WARNING: CPU: 57 PID: 9729 at kernel/events/core.c:5439 free_event+0x48/0x60 RIP: 0010:free_event+0x48/0x60 Call Trace: ? free_event+0x48/0x60 perf_event_exit_event.isra.0+0x60/0xf0 perf_event_exit_task+0x1e1/0x280 do_exit+0x327/0xb00The race continues... now, with the stale
child2->parentpointer:CPU 1 CPU 2
15 perf_event_exit_event() /* event2 */ . /* CPU 1 is holding ctx->mutex of 16 if (parent_event) { /* true */ . child process. Waiting ... */ 17 if (revoke) /* false */ . 18 else . 19 free_event(); /* event2 */ . 20 WARN() . 21 put_event(parent_event); /* event1 */ . 22 /* event1->refcount becomes 1 */ . 23 } . 24 +- mutex_unlock(&child_ctx->mutex); . /* Acquired child's ctx->mutex */ 25 __pmu_detach_event() /* event2 */26
perf_event_exit_event()/* event2 */
27 if (parent_event) { /* true, BUT FALSE POSITIVE. */
28 if (revoke) /* true */
29put_event(); /* event2 */
30 /*event2->refcountbecomes 1 */
31put_event(parent_event); /* event1 */
32 /*event1->refcountbecomes 0 */
Quoted 5 lines
^^^^^^^^^^^^^^^^^^^^^^^^^^This can manifest as some random bug. I guess,
perf_event_exit_event()should also check for (event->attach_state&PERF_ATTACH_CHILD) whenevent->parent!= NULL.
Does this work?
kernel/events/core.c +1 -0
@@ -2303,6 +2303,7 @@ static void perf_child_detach(struct persync_child_event (event);list_del_init (&event->child_list); + event->parent =NULL ; }static bool is_orphaned_event (struct perf_event *event)
Re: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
kernel/events/core.c static void perf_child_detach(struct per · line 2309
Quoted 12 lines
Does this work?
kernel/events/core.c +1
-0@@ -2303,6 +2303,7 @@ static void perf_child_detach(struct persync_child_event (event);list_del_init (&event->child_list); + event->parent =NULL ; }static bool is_orphaned_event (struct perf_event *event)
Apparently not, it ends up with:
------------[ cut here ]------------ WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:281 event_function+0xd2/0xf0 Modules linked in: [...] CPU: 145 UID: 1002 PID: 5459 Comm: perf_fuzzer Kdump: loaded Not tainted 6.14.0-rc1-pmu-unregister+ #244 Hardware name: AMD Corporation RUBY/RUBY, BIOS RRR1009C 07/21/2023 RIP: 0010:event_function+0xd2/0xf0 Code: [...] RSP: 0018:ffa000000647fac8 EFLAGS: 00010046 RAX: 0000000000000000 RBX: ff1100100d4b0ee0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: ffa000000647faf0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffa000000647fbb8 R13: 0000000000000000 R14: ff11000104cc9e00 R15: ff110001216fcec0 FS: 000070acabcae740(0000) GS:ff1100100d480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffff600000 CR3: 000000011d920005 CR4: 0000000000f71ff0 PKRU: 55555554 Call Trace: <TASK> ? show_regs+0x6c/0x80 ? __warn+0x8d/0x150 ? event_function+0xd2/0xf0 ? report_bug+0x182/0x1b0 ? handle_bug+0x6e/0xb0 ? exc_invalid_op+0x18/0x80 ? asm_exc_invalid_op+0x1b/0x20 ? event_function+0xd2/0xf0 ? event_function+0x3d/0xf0 remote_function+0x4f/0x70 ? __pfx_remote_function+0x10/0x10 generic_exec_single+0x7f/0x160 smp_call_function_single+0x110/0x160 ? __pfx_remote_function+0x10/0x10 ? ktime_get_ts64+0x49/0x150 event_function_call+0x98/0x1d0 ? __pfx___perf_event_disable+0x10/0x10 ? __pfx___perf_event_disable+0x10/0x10 ? __pfx_event_function+0x10/0x10 ? __pfx__perf_event_disable+0x10/0x10 _perf_event_disable+0x41/0x70 perf_event_for_each_child+0x40/0x90 ? __pfx__perf_event_disable+0x10/0x10 _perf_ioctl+0xac/0xb00 ? srso_alias_return_thunk+0x5/0xfbef5 ? place_entity+0xa7/0x170 ? srso_alias_return_thunk+0x5/0xfbef5 ? __x2apic_send_IPI_dest+0x32/0x50 ? srso_alias_return_thunk+0x5/0xfbef5 ? x2apic_send_IPI+0x26/0x40 ? srso_alias_return_thunk+0x5/0xfbef5 ? native_send_call_func_single_ipi+0x13/0x20 ? srso_alias_return_thunk+0x5/0xfbef5 ? __smp_call_single_queue+0xf7/0x180 ? srso_alias_return_thunk+0x5/0xfbef5 ? generic_exec_single+0x38/0x160 ? srso_alias_return_thunk+0x5/0xfbef5 ? smp_call_function_single_async+0x22/0x50 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? mutex_lock+0x12/0x50 perf_ioctl+0x45/0x80 __x64_sys_ioctl+0xa7/0xe0 x64_sys_call+0x131e/0x2650 do_syscall_64+0x7e/0x170 ? srso_alias_return_thunk+0x5/0xfbef5 ? rcu_core+0x1d1/0x3a0 ? srso_alias_return_thunk+0x5/0xfbef5 ? rcu_core_si+0xe/0x20 ? srso_alias_return_thunk+0x5/0xfbef5 ? handle_softirqs+0xe7/0x330 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? irqentry_exit_to_user_mode+0x43/0x250 ? srso_alias_return_thunk+0x5/0xfbef5 ? irqentry_exit+0x43/0x50 ? srso_alias_return_thunk+0x5/0xfbef5 ? sysvec_apic_timer_interrupt+0x4f/0xc0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x70acabb24ded Code: [...] RSP: 002b:00007fffb3dd4a40 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 00007fffb3dd6f28 RCX: 000070acabb24ded RDX: 0000000066f26a61 RSI: 0000000000002401 RDI: 0000000000000013 RBP: 00007fffb3dd4a90 R08: 000070acabc03084 R09: 000070acabc030a0 R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000001 R13: 0000000000000000 R14: 00005e38a9ec5b60 R15: 000070acabd00000 </TASK> ---[ end trace 0000000000000000 ]--- ------------[ cut here ]------------ WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:286 event_function+0xd6/0xf0 Modules linked in: [...] CPU: 145 UID: 1002 PID: 5459 Comm: perf_fuzzer Kdump: loaded Tainted: G W 6.14.0-rc1-pmu-unregister+ #244 Tainted: [W]=WARN Hardware name: AMD Corporation RUBY/RUBY, BIOS RRR1009C 07/21/2023 RIP: 0010:event_function+0xd6/0xf0 Code: [...] RSP: 0018:ffa000000647fac8 EFLAGS: 00010086 RAX: 0000000000000000 RBX: ff1100100d4b0ee0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: ffa000000647faf0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffa000000647fbb8 R13: 0000000000000000 R14: ff11000104cc9e00 R15: ff110001216fcec0 FS: 000070acabcae740(0000) GS:ff1100100d480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffff600000 CR3: 000000011d920005 CR4: 0000000000f71ff0 PKRU: 55555554 Call Trace: <TASK> ? show_regs+0x6c/0x80 ? __warn+0x8d/0x150 ? event_function+0xd6/0xf0 ? report_bug+0x182/0x1b0 ? handle_bug+0x6e/0xb0 ? exc_invalid_op+0x18/0x80 ? asm_exc_invalid_op+0x1b/0x20 ? event_function+0xd6/0xf0 ? event_function+0x3d/0xf0 remote_function+0x4f/0x70 ? __pfx_remote_function+0x10/0x10 generic_exec_single+0x7f/0x160 smp_call_function_single+0x110/0x160 ? __pfx_remote_function+0x10/0x10 ? ktime_get_ts64+0x49/0x150 event_function_call+0x98/0x1d0 ? __pfx___perf_event_disable+0x10/0x10 ? __pfx___perf_event_disable+0x10/0x10 ? __pfx_event_function+0x10/0x10 ? __pfx__perf_event_disable+0x10/0x10 _perf_event_disable+0x41/0x70 perf_event_for_each_child+0x40/0x90 ? __pfx__perf_event_disable+0x10/0x10 _perf_ioctl+0xac/0xb00 ? srso_alias_return_thunk+0x5/0xfbef5 ? place_entity+0xa7/0x170 ? srso_alias_return_thunk+0x5/0xfbef5 ? __x2apic_send_IPI_dest+0x32/0x50 ? srso_alias_return_thunk+0x5/0xfbef5 ? x2apic_send_IPI+0x26/0x40 ? srso_alias_return_thunk+0x5/0xfbef5 ? native_send_call_func_single_ipi+0x13/0x20 ? srso_alias_return_thunk+0x5/0xfbef5 ? __smp_call_single_queue+0xf7/0x180 ? srso_alias_return_thunk+0x5/0xfbef5 ? generic_exec_single+0x38/0x160 ? srso_alias_return_thunk+0x5/0xfbef5 ? smp_call_function_single_async+0x22/0x50 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? mutex_lock+0x12/0x50 perf_ioctl+0x45/0x80 __x64_sys_ioctl+0xa7/0xe0 x64_sys_call+0x131e/0x2650 do_syscall_64+0x7e/0x170 ? srso_alias_return_thunk+0x5/0xfbef5 ? rcu_core+0x1d1/0x3a0 ? srso_alias_return_thunk+0x5/0xfbef5 ? rcu_core_si+0xe/0x20 ? srso_alias_return_thunk+0x5/0xfbef5 ? handle_softirqs+0xe7/0x330 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? irqentry_exit_to_user_mode+0x43/0x250 ? srso_alias_return_thunk+0x5/0xfbef5 ? irqentry_exit+0x43/0x50 ? srso_alias_return_thunk+0x5/0xfbef5 ? sysvec_apic_timer_interrupt+0x4f/0xc0 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x70acabb24ded Code: [...] RSP: 002b:00007fffb3dd4a40 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 00007fffb3dd6f28 RCX: 000070acabb24ded RDX: 0000000066f26a61 RSI: 0000000000002401 RDI: 0000000000000013 RBP: 00007fffb3dd4a90 R08: 000070acabc03084 R09: 000070acabc030a0 R10: 0000000000000008 R11: 0000000000000246 R12: 0000000000000001 R13: 0000000000000000 R14: 00005e38a9ec5b60 R15: 000070acabd00000 </TASK> ---[ end trace 0000000000000000 ]--- watchdog: BUG: soft lockup - CPU#88 stuck for 26s! [perf_fuzzer:5740] Modules linked in: [...] CPU: 88 UID: 1002 PID: 5740 Comm: perf_fuzzer Kdump: loaded Tainted: G W 6.14.0-rc1-pmu-unregister+ #244 Tainted: [W]=WARN Hardware name: AMD Corporation RUBY/RUBY, BIOS RRR1009C 07/21/2023 RIP: 0010:smp_call_function_single+0x11f/0x160 Code: [...] RSP: 0000:ffa0000004b57ba0 EFLAGS: 00000202 RAX: 0000000000000000 RBX: ff11000153ad8000 RCX: 0000000000000000 RDX: 0000000000000011 RSI: 0000000000000000 RDI: 0000000000000000 RBP: ffa0000004b57be8 R08: 0000000000000000 R09: ffa0000004b57c20 R10: 0000000000000001 R11: ffffffff81621ef0 R12: ff11000104cc9e00 R13: ffa0000004b57c08 R14: ff1100013732af40 R15: ff1100100b830ee0 FS: 0000000000000000(0000) GS:ff1100100b800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00005e38a9ea8930 CR3: 0000000003440004 CR4: 0000000000f71ef0 PKRU: 55555554 Call Trace: <IRQ> ? show_regs+0x6c/0x80 ? watchdog_timer_fn+0x22b/0x2d0 ? __pfx_watchdog_timer_fn+0x10/0x10 ? __hrtimer_run_queues+0x112/0x2a0 ? clockevents_program_event+0xc1/0x150 ? hrtimer_interrupt+0xfd/0x260 ? __sysvec_apic_timer_interrupt+0x56/0x130 ? sysvec_apic_timer_interrupt+0x93/0xc0 </IRQ> <TASK> ? asm_sysvec_apic_timer_interrupt+0x1b/0x20 ? __pfx_remote_function+0x10/0x10 ? smp_call_function_single+0x11f/0x160 ? __pfx_remote_function+0x10/0x10 ? event_function_call+0x115/0x1d0 ? srso_alias_return_thunk+0x5/0xfbef5 event_function_call+0x98/0x1d0 ? __pfx___perf_remove_from_context+0x10/0x10 ? __pfx___perf_remove_from_context+0x10/0x10 ? __pfx_event_function+0x10/0x10 perf_remove_from_context+0x46/0xa0 perf_event_release_kernel+0x1f3/0x210 perf_release+0x12/0x20 __fput+0xed/0x2d0 ____fput+0x15/0x20 task_work_run+0x60/0xa0 do_exit+0x317/0xb00 ? srso_alias_return_thunk+0x5/0xfbef5 do_group_exit+0x34/0x90 get_signal+0x934/0x940 arch_do_signal_or_restart+0x2f/0x250 ? srso_alias_return_thunk+0x5/0xfbef5 ? event_function+0xa4/0xf0 ? __pfx_remote_function+0x10/0x10 ? srso_alias_return_thunk+0x5/0xfbef5 ? remote_function+0x4f/0x70 ? srso_alias_return_thunk+0x5/0xfbef5 irqentry_exit_to_user_mode+0x1e0/0x250 irqentry_exit+0x43/0x50 sysvec_reschedule_ipi+0x5d/0x100 asm_sysvec_reschedule_ipi+0x1b/0x20 RIP: 0033:0x5e38a9eb87dd Code: Unable to access opcode bytes at 0x5e38a9eb87b3. RSP: 002b:00007fffb3dd4ae8 EFLAGS: 00000202 RAX: 0000000000001553 RBX: 00007fffb3dd6f28 RCX: 00000000000020da RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 RBP: 00007fffb3dd4b00 R08: 0000000000000001 R09: 000070acabcae740 R10: 000070acabd010b8 R11: 0000000000000246 R12: 0000000000000001 R13: 0000000000000000 R14: 00005e38a9ec5b60 R15: 000070acabd00000 </TASK>
Something like below instead? I haven't tested it thoroughly though.
---
kernel/events/core.c +18 -13
@@ -13645,20 +13645,25 @@ perf_event_exit_event(struct perf_event *event,unsigned long detach_flags =DETACH_EXIT ;if (parent_event) { -/* -* Do not destroy the 'original' grouping; because of the -* context switch optimization the original events could've -* ended up in a random child task. -* -* If we were to destroy the original group, all group related -* operations would cease to function properly after this -* random child dies. -* -* Do destroy all inherited groups, we don't care about those -* and being thorough is better. -*/ - detach_flags |=DETACH_GROUP |DETACH_CHILD ;mutex_lock (&parent_event->child_mutex); +if (event->attach_state &PERF_ATTACH_CHILD ) { +/* +* Do not destroy the 'original' grouping; because of the +* context switch optimization the original events could've +* ended up in a random child task. +* +* If we were to destroy the original group, all group related +* operations would cease to function properly after this +* random child dies. +* +* Do destroy all inherited groups, we don't care about those +* and being thorough is better. +*/ + detach_flags |=DETACH_GROUP |DETACH_CHILD ; + }else { +mutex_unlock (&parent_event->child_mutex); + parent_event =NULL ; + } }if (revoke)
---
Thanks,
Ravi
Peter Zijlstra
maintainerRe: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Quoted 11 lines · Ravi Bangoria
Quoted 11 lines
Does this work?
kernel/events/core.c +1
-0@@ -2303,6 +2303,7 @@ static void perf_child_detach(struct persync_child_event (event);list_del_init (&event->child_list); + event->parent =NULL ; }
Quoted 6 lines
kernel/events/core.c static void perf_child_detach(struct per · line 2309
Quoted 1 line
kernel/events/core.c +0
-0static bool is_orphaned_event (struct perf_event *event)Apparently not, it ends up with:
------------[ cut here ]------------ WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:281 event_function+0xd2/0xf0
Durr, do you have an updated test case?
Quoted 34 lines
Something like below instead? I haven't tested it thoroughly though.
---
kernel/events/core.c +9
-13@@ -13645,20 +13645,25 @@ perf_event_exit_event(struct perf_event *event,unsigned long detach_flags =DETACH_EXIT ;if (parent_event) { -/* -* Do not destroy the 'original' grouping; because of the -* context switch optimization the original events could've -* ended up in a random child task. -* -* If we were to destroy the original group, all group related -* operations would cease to function properly after this -* random child dies. -* -* Do destroy all inherited groups, we don't care about those -* and being thorough is better. -*/ - detach_flags |=DETACH_GROUP |DETACH_CHILD ;mutex_lock (&parent_event->child_mutex); +if (event->attach_state &PERF_ATTACH_CHILD ) { +/* +* Do not destroy the 'original' grouping; because of the +* context switch optimization the original events could've +* ended up in a random child task. +* +* If we were to destroy the original group, all group related +* operations would cease to function properly after this +* random child dies.
kernel/events/core.c perf_event_exit_event(struct perf_event *event, · line 13667
Quoted 10 lines
kernel/events/core.c +9
-0+* +* Do destroy all inherited groups, we don't care about those +* and being thorough is better. +*/ + detach_flags |=DETACH_GROUP |DETACH_CHILD ; + }else { +mutex_unlock (&parent_event->child_mutex); + parent_event =NULL ; + } }
Yeah, that might do, but not really nice. But perhaps its the best we can do. I'll give it some thought.
Re: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Ravi Bangoria
Quoted 11 lines
Does this work?
kernel/events/core.c +1
-0@@ -2303,6 +2303,7 @@ static void perf_child_detach(struct persync_child_event (event);list_del_init (&event->child_list); + event->parent =NULL ; }
Quoted 8 lines
Quoted 6 lines
kernel/events/core.c static void perf_child_detach(struct per · line 2309
Quoted 1 line
kernel/events/core.c +0
-0static bool is_orphaned_event (struct perf_event *event)Apparently not, it ends up with:
------------[ cut here ]------------ WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:281 event_function+0xd2/0xf0Durr, do you have an updated test case?
This one triggered with just perf_fuzzer run (without my tinypmu test).
But in general, to test the whole series, I ran perf_fuzzer simultaneously with my tinypmu tests:
term1~$ echo -1 | sudo tee /proc/sys/kernel/perf_event_paranoid; while true; do ./perf_fuzzer; done term2~$ cd ~/tinypmu; make clean; make; while true; do sudo bash tinypmu-u.sh; done
Thanks,
Ravi
Peter Zijlstra
maintainerRe: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
In reply to Ravi BangoriaQuoted 14 lines · Ravi Bangoria
Apparently not, it ends up with:
------------[ cut here ]------------ WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:281 event_function+0xd2/0xf0 WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:286 event_function+0xd6/0xf0 remote_function+0x4f/0x70 generic_exec_single+0x7f/0x160 smp_call_function_single+0x110/0x160 event_function_call+0x98/0x1d0 _perf_event_disable+0x41/0x70 perf_event_for_each_child+0x40/0x90 _perf_ioctl+0xac/0xb00 perf_ioctl+0x45/0x80
Took me a long while trying to blame this on the 'event->parent = NULL;', but AFAICT this is a new, unrelated issue.
What I think happens is this perf_ioctl(DISABLE) vs pmu_detach_events() race, where the crux is that perf_ioctl() path does not take event2->mutex which allows the following interleave:
event1 <---> ctx1 | ^ child_list | | parent v | event2 <---> ctx2
perf_ioctl()
perf_event_ctx_lock(event1)
get_ctx(ctx1)
mutex_lock(ctx1->mutex)
_perf_ioctk()
perf_event_for_each_child()
mutex_lock(event1->child_mutex)
_perf_event_disable(event1)
_perf_event_disable(event2)
raw_spin_lock_irq(ctx2->lock)
raw_spin_unlock_irq()
event_function_call(event2, __perf_event_disable)
task_function_call()
<IPI __perf_event_disable>pmu_detach_events()
event2 = pmu_get_event() <-- inc(event2->refcount); pmu_detach_event(event2)
perf_event_ctx_lock(event2)
get_ctx(ctx2)
mutex_lock(ctx2->lock)
__pmu_detach_event()
perf_event_exit_event()
mutex_lock(event1->child_mutex)
perf_remove_from_context(event2, EXIT|GROUP|CHILD|REVOKE)
lockdep_assert_held(ctx2->mutex)
raw_spin_lock_irq(ctx2->lock)
raw_spin_unlock_irq(ctx2->lock)
event_function_call(event2, __perf_remove_from_context)
task_function_call(event_function)
<IPI __perf_remove_from_context>
remote_function()
event_function()
perf_ctx_lock(cpuctx, ctx2)
raw_spin_lock(ctx2->lock)
__perf_remove_from_context(event2)
event_sched_out()
perf_group_detach()
perf_child_detach()
list_del_event()
event->state = REVOKED;
cpc->task_epc = NULL; // event2 is last
ctx->is_active = 0; <--.
cpuctx->task_ctx = NULL; |
|
|
<IPI __perf_event_disable> |
remote_function() |
event_function() |
perf_ctx_lock(cpuctx, ctx2) |
raw_spin_lock(ctx2->lock) |
|
WARN_ON_ONCE(!ctx2->is_active) -'
WARN_ON_ONCE(cpuctx->task_ctx != ctx2)Still trying to work out how best to avoid this.
Re: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Hi Peter,
Quoted 15 lines
Quoted 14 lines
Apparently not, it ends up with:
------------[ cut here ]------------ WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:281 event_function+0xd2/0xf0 WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:286 event_function+0xd6/0xf0 remote_function+0x4f/0x70 generic_exec_single+0x7f/0x160 smp_call_function_single+0x110/0x160 event_function_call+0x98/0x1d0 _perf_event_disable+0x41/0x70 perf_event_for_each_child+0x40/0x90 _perf_ioctl+0xac/0xb00 perf_ioctl+0x45/0x80
Quoted 6 lines
Took me a long while trying to blame this on the '
event->parent= NULL;', but AFAICT this is a new, unrelated issue.What I think happens is this
perf_ioctl(DISABLE)vspmu_detach_events()race, where the crux is thatperf_ioctl()path does not takeevent2->mutexwhich allows the following interleave:
This one was only with perf_fuzzer, so pmu_detach_events() code path was not invoked.
Thanks,
Ravi
Re: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Quoted 20 lines · Ravi Bangoria
Hi Peter,
Quoted 18 lines
Quoted 14 lines
Apparently not, it ends up with:
------------[ cut here ]------------ WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:281 event_function+0xd2/0xf0 WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:286 event_function+0xd6/0xf0 remote_function+0x4f/0x70 generic_exec_single+0x7f/0x160 smp_call_function_single+0x110/0x160 event_function_call+0x98/0x1d0 _perf_event_disable+0x41/0x70 perf_event_for_each_child+0x40/0x90 _perf_ioctl+0xac/0xb00 perf_ioctl+0x45/0x80Took me a long while trying to blame this on the '
event->parent= NULL;', but AFAICT this is a new, unrelated issue.
Quoted 6 lines
Quoted 3 lines
What I think happens is this
perf_ioctl(DISABLE)vspmu_detach_events()race, where the crux is thatperf_ioctl()path does not takeevent2->mutexwhich allows the following interleave:This one was only with
perf_fuzzer, sopmu_detach_events()code path was not invoked.
I think the issue is, unaccount_event() gets called for the child event after the child is detached. Since event->parent is NULL, unaccount_event() abruptly corrupts the perf_sched_work.
I haven't verified it. Will do it tomorrow.
Thanks,
Ravi
Re: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Hi Peter,
Quoted 22 lines
Quoted 22 lines
Quoted 21 lines
Quoted 14 lines
Apparently not, it ends up with:
------------[ cut here ]------------ WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:281 event_function+0xd2/0xf0 WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:286 event_function+0xd6/0xf0 remote_function+0x4f/0x70 generic_exec_single+0x7f/0x160 smp_call_function_single+0x110/0x160 event_function_call+0x98/0x1d0 _perf_event_disable+0x41/0x70 perf_event_for_each_child+0x40/0x90 _perf_ioctl+0xac/0xb00 perf_ioctl+0x45/0x80Took me a long while trying to blame this on the '
event->parent= NULL;', but AFAICT this is a new, unrelated issue.What I think happens is this
perf_ioctl(DISABLE)vspmu_detach_events()race, where the crux is thatperf_ioctl()path does not takeevent2->mutexwhich allows the following interleave:
Quoted 6 lines
Quoted 2 lines
This one was only with
perf_fuzzer, sopmu_detach_events()code path was not invoked.I think the issue is,
unaccount_event()gets called for the child event after the child is detached. Sinceevent->parentis NULL,unaccount_event()abruptly corrupts theperf_sched_work.
A different manifestation of the same underlying issue (perf_sched_work is getting corrupted because of event->parent = NULL):
WARNING: CPU: 0 PID: 5799 at kernel/events/core.c:286 event_function+0xd6/0xf0 CPU: 0 UID: 1002 PID: 5799 Comm: perf_fuzzer Kdump: loaded Not tainted 6.14.0-rc1-pmu-unregister+ #263 RIP: 0010:event_function+0xd6/0xf0 RSP: 0018:ffa0000006a87828 EFLAGS: 00010086 RAX: 0000000000000007 RBX: ff11001008830ee0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: ffa0000006a87850 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: ffa0000006a87918 R13: 0000000000000000 R14: ff1100010e347c00 R15: ff110001226c9a40 FS: 0000000000000000(0000) GS:ff11001008800000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000595fed7d3e12 CR3: 000000016954c005 CR4: 0000000000f71ef0 PKRU: 55555554 Call Trace: <TASK> ? event_function+0xd6/0xf0 ? event_function+0x3d/0xf0 remote_function+0x4c/0x70 generic_exec_single+0x7c/0x160 smp_call_function_single+0x110/0x160 event_function_call+0x98/0x1d0 perf_remove_from_context+0x46/0xa0 perf_event_release_kernel+0x1f3/0x210 perf_release+0x12/0x20
With the above trace as reference, something like below is what is happening. (Code paths are based on my understanding of traces, I've not verified each an every bit :P)
Assume:
task1: event1
task2 (child of task1): event2 (child of event1)
task3: event3
CPU 1 CPU 2 CPU 3 CPU 4
/* task3 sched in */
perf_event_task_sched_in()
if (static_branch_unlikely(&perf_sched_events)) /* true */
__perf_event_task_sched_in()
... task2:
perf_event_exit_event() /* event2 */
event->parent = NULL /* by perf_perf_child_detach() */
free_event()
_free_event()
unaccount_event()
if (event->parent) /* false */
return; /* won't get executed */
perf_sched_work <== OFF ------.
|
| /* task3 sched out */
| perf_event_task_sched_out()
'-----> if (static_branch_unlikely(&perf_sched_events)) /* false */
| /* __perf_event_task_sched_out() won't get called */
|
| /* task3 sched in */
| perf_event_task_sched_in()
'--------------------------------------------------------------------------> if (static_branch_unlikely(&perf_sched_events)) /* false */
/* __perf_event_task_sched_in() won't get called
So cpuctx->task_ctx will remains NULL; */
close(event3)
perf_release()
perf_event_release_kernel()
perf_remove_from_context()
event_function_call()
/* IPI to CPU 3 ---> */
/* ---> IPI from CPU 4 */
event_function()
if (ctx->task) {
/* task_ctx is NULL whereas ctx is !NULL */
WARN_ON_ONCE(task_ctx != ctx);
^^^^^^^^^^^^IIUC, event->parent is the only way to detect whether the event is a child or not, so it makes sense to preserve the ->parent pointer even after the child is detached.
Thanks,
Ravi
Re: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Quoted 25 lines
Quoted 25 lines
Quoted 24 lines
Quoted 15 lines
Quoted 14 lines
Apparently not, it ends up with:
------------[ cut here ]------------ WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:281 event_function+0xd2/0xf0 WARNING: CPU: 145 PID: 5459 at kernel/events/core.c:286 event_function+0xd6/0xf0 remote_function+0x4f/0x70 generic_exec_single+0x7f/0x160 smp_call_function_single+0x110/0x160 event_function_call+0x98/0x1d0 _perf_event_disable+0x41/0x70 perf_event_for_each_child+0x40/0x90 _perf_ioctl+0xac/0xb00 perf_ioctl+0x45/0x80Quoted 6 lines
Took me a long while trying to blame this on the '
event->parent= NULL;', but AFAICT this is a new, unrelated issue.What I think happens is this
perf_ioctl(DISABLE)vspmu_detach_events()race, where the crux is thatperf_ioctl()path does not takeevent2->mutexwhich allows the following interleave:This one was only with
perf_fuzzer, sopmu_detach_events()code path was not invoked.
Quoted 6 lines
Quoted 3 lines
I think the issue is,
unaccount_event()gets called for the child event after the child is detached. Sinceevent->parentis NULL,unaccount_event()abruptly corrupts theperf_sched_work.A different manifestation of the same underlying issue (
perf_sched_workis getting corrupted because ofevent->parent= NULL):
Duh! I meant perf_sched_events gets corrupted by invoking perf_sched_work.
Thanks,
Ravi
Re: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
In reply to Peter ZijlstraQuoted 16 lines · Peter Zijlstra
Previously it was only safe to call
perf_pmu_unregister()if there were no active events of that pmu around -- which was impossible to guarantee since it races all sorts againstperf_init_event().Rework the whole thing by:
- keeping track of all events for a given pmu
- 'hiding' the pmu from
perf_init_event()- waiting for the appropriate (s)rcu grace periods such that all prior references to the PMU will be completed
- detaching all still existing events of that pmu (see first point) and moving them to a new REVOKED state.
Quoted 6 lines
- actually freeing the pmu data.
Where notably the new REVOKED state must inhibit all event actions from reaching code that wants to use
event->pmu.Signed-off-by Peter Zijlstra (Intel) <[email protected]>
Another race between perf_event_init_task() failure path and perf_pmu_unregister():
CPU 1 CPU 2
perf_event_init_task()
perf_event_free_task()
perf_free_event(event1)
/* event1->refcount is 1 */
perf_pmu_unregister()
pmu_detach_events()
pmu_get_event(pmu) /* Picks event1 */
atomic_long_inc_not_zero(&event1->refcount) /* event1 */
/* event1->refcount became 2 (by CPU 2) */
free_event(event1)
WARN()Thanks,
Ravi
Peter Zijlstra
maintainerRe: [PATCH v2 24/24] perf: Make perf_pmu_unregister() useable
Quoted 30 lines · Ravi Bangoria
Quoted 16 lines · Peter Zijlstra
Previously it was only safe to call
perf_pmu_unregister()if there were no active events of that pmu around -- which was impossible to guarantee since it races all sorts againstperf_init_event().Rework the whole thing by:
- keeping track of all events for a given pmu
- 'hiding' the pmu from
perf_init_event()- waiting for the appropriate (s)rcu grace periods such that all prior references to the PMU will be completed
- detaching all still existing events of that pmu (see first point) and moving them to a new REVOKED state.
Quoted 6 lines
- actually freeing the pmu data.
Where notably the new REVOKED state must inhibit all event actions from reaching code that wants to use
event->pmu.Signed-off-by Peter Zijlstra (Intel) <[email protected]>Another race between
perf_event_init_task()failure path andperf_pmu_unregister():CPU 1 CPU 2
perf_event_init_task()
Quoted 10 lines
perf_event_free_task()
perf_free_event(event1)
/* event1->refcount is 1 */
perf_pmu_unregister()
pmu_detach_events()
pmu_get_event(pmu) /* Picks event1 */
atomic_long_inc_not_zero(&event1->refcount) /* event1 */
/* event1->refcount became 2 (by CPU 2) */
free_event(event1)
WARN()I've unified perf_event_free_task() and perf_event_exit_task(). This makes both use perf_event_exit_event() and as such should no longer have this free_event().
Ingo, Peter,
Quoted 5 lines · Peter Zijlstra
Hi
New version of the
perf_pmu_unregister()rework. A fair number of bugs squashed. Much thanks to Ravi for writing a stress tester. The robots also managed to find a few problems.
I've seen more issues (in addition to what I've already reported) with v2 as well (I didn't get time to debug those so I didn't report them here). However, those additional issues appear only with pmu unregister stress test + perf_fuzzer and IMO pmu unregister is a rare event and I shouldn't block the inclusion of the series. So, with current set of issues addressed:
Thanks,
Ravi