perf_counter: Fix race in attaching counters to tasks and exiting
PATCH v4 1/1 perf_counter: Fix race in attaching counters to tasks and exiting
Commit 564c2b21 ("perf_counter: Optimize context switch between identical inherited contexts") introduced a race where it is possible that a counter being attached to a task could get attached to the wrong task, if the task is one that has inherited its context from another task via fork. This happens because the optimized context switch could switch the context to another task after find_get_context has read task->perf_counter_ctxp. In fact, it's possible that the context could then get freed, if the other task then exits.
This fixes the problem by protecting both the context switch and the critical code in find_get_context with spinlocks. The context switch locks the cxt->lock of both the outgoing and incoming contexts before swapping them. That means that once code such as find_get_context has obtained the spinlock for the context associated with a task, the context can't get swapped to another task. However, the context may have been swapped in the interval between reading task->perf_counter_ctxp and getting the lock, so it is necessary to check and retry.
To make sure that none of the contexts being looked at in find_get_context can get freed, this changes the context freeing code to use RCU. Thus an rcu_read_lock() is sufficient to ensure that no contexts can get freed. This part of the patch is lifted from a patch posted by Peter Zijlstra.
This also adds a check to make sure that we can't add a counter to a task that is exiting.
There is also a race between perf_counter_exit_task and find_get_context; this solves the race by moving the get_ctx that was in perf_counter_alloc into the locked region in find_get_context, so that once find_get_context has got the context for a task, it won't get freed even if the task calls perf_counter_exit_task. It doesn't matter if new top-level (non-inherited) counters get attached to the context after perf_counter_exit_task has detached the context from the task. They will just stay there and never get scheduled in until the counters' fds get closed, and then perf_release will remove them from the context and eventually free the context.
With this, we are now doing the unclone in find_get_context rather than when a counter was added to or removed from a context (actually, we were missing the unclone_ctx() call when adding a counter to a context). We don't need to unclone when removing a counter from a context because we have no way to remove a counter from a cloned context.
This also takes out the smp_wmb() in find_get_context, which Peter Zijlstra pointed out was unnecessary because the cmpxchg implies a full barrier anyway.
---
v4: added a missing get_task_struct in perf_counter_init_task
include/linux/perf_counter.h | 5 +- kernel/perf_counter.c | 204 +++++++++++++++++++++++++++++------------ 2 files changed, 147 insertions(+), 62 deletions(-)
include/linux/perf_counter.h +3 -2
@@ -541,8 +541,9 @@ struct perf_counter_context {* been cloned (inherited) from a common ancestor. */ struct perf_counter_context *parent_ctx; -u32 parent_gen; -u32 generation; +u64 parent_gen; +u64 generation; +struct rcu_head rcu_head; };/**
kernel/perf_counter.c +144 -60
360 lines not on this page · raw message
Signature · 1 line
1.6.0.4
Looks good, few nits below.
kernel/perf_counter.c static void get_ctx(struct perf_counter_context *ctx) · line 114
Quoted 9 lines · Paul Mackerras
kernel/perf_counter.c +9
-0+static void free_ctx (struct rcu_head *head) +{ +struct perf_counter_context *ctx; + + ctx = container_of(head, struct perf_counter_context, rcu_head); +if (ctx->task) +put_task_struct (ctx->task); +kfree (ctx); +}
The poor task will now have to wait for yet another RCU grace period, but yeah :-)
Quoted 22 lines
kernel/perf_counter.c +17
-0@@ -932,18 +950,39 @@ void perf_counter_task_sched_out(struct task_struct *task,return ;update_context_time (ctx); + +rcu_read_lock (); + parent =rcu_dereference (ctx->parent_ctx); next_ctx = next->perf_counter_ctxp; +if (parent && next_ctx && +rcu_dereference (next_ctx->parent_ctx) == parent) { +/* +* Looks like the two contexts are clones, so we might be +* able to optimize the context switch. We lock both +* contexts and check that they are clones under the +* lock (including re-checking that neither has been +* uncloned in the meantime). It doesn't matter which +* order we take the locks because no other cpu could +* be trying to lock both of these tasks. +*/ +spin_lock (&ctx->lock); +spin_lock (&next_ctx->lock); +if (context_equiv (ctx, next_ctx)) {
kernel/perf_counter.c void perf_counter_task_sched_out(struct task_struct *task, · line 980
Quoted 10 lines
kernel/perf_counter.c +9
-0+ task->perf_counter_ctxp = next_ctx; + next->perf_counter_ctxp = ctx; + ctx->task = next; + next_ctx->task = task; + do_switch =0 ; + } +spin_unlock (&next_ctx->lock); +spin_unlock (&ctx->lock); } +rcu_read_unlock ();
Looks good, although lockdep will complain, the proper annotation would be spin_lock_nested_lock(&ctx->lock, &rq->lock), but since we don't have access to the rq here, we should change the second spin_lock, to:
spin_lock_nested(&next_ctx->lock, SINGLE_DEPTH_NESTING);
Quoted 56 lines
kernel/perf_counter.c +47
-0@@ -1265,37 +1300,78 @@ static struct perf_counter_context *find_get_context(pid_t pid, int cpu)if (!task)return ERR_PTR (-ESRCH ); +/* +* Can't attach counters to a dying task. +*/ + err = -ESRCH ; +if (task->flags &PF_EXITING ) + goto errout; +/* Reuse ptrace permission checks for now. */ + err = -EACCES ; +if (!ptrace_may_access (task,PTRACE_MODE_READ )) + goto errout; + +rcu_read_lock (); + retry: + ctx =rcu_dereference (task->perf_counter_ctxp); +if (ctx) { +/* +* If this context is a clone of another, it might +* get swapped for another underneath us by +* perf_counter_task_sched_out, though the +* rcu_read_lock() protects us from any context +* getting freed. Lock the context and check if it +* got swapped before we could get the lock, and retry +* if so. If we locked the right context, then it +* can't get swapped on us any more and we can +* unclone it if necessary. +* Once it's not a clone things will be stable. +*/ +spin_lock (&ctx->lock); +if (ctx !=rcu_dereference (task->perf_counter_ctxp)) { +spin_unlock (&ctx->lock); + goto retry; + } + parent_ctx = ctx->parent_ctx; +if (parent_ctx) { +put_ctx (parent_ctx); + ctx->parent_ctx =NULL ;/* no longer a clone */ + } + ++ctx->generation; +/* +* Get an extra reference before dropping the lock so that +* this context won't get freed if the task exits. +*/ +get_ctx (ctx); +spin_unlock (&ctx->lock); } +rcu_read_unlock ();if (!ctx) { ctx =kmalloc (sizeof (struct perf_counter_context ),GFP_KERNEL ); + err = -ENOMEM ; +if (!ctx)
kernel/perf_counter.c static struct perf_counter_context *find_get_context(pid_t pid, int cpu) · line 1364
Quoted 10 lines
kernel/perf_counter.c +4
-0+ goto errout;__perf_counter_init_context (ctx, task); +get_ctx (ctx); +if (cmpxchg (&task->perf_counter_ctxp,NULL , ctx)) {/* * We raced with some other task; use * the context they set. */ kfree (ctx); + goto retry;
You jump into an rcu_read_lock() section there.
Quoted 13 lines
kernel/perf_counter.c +6
-0} +get_task_struct (task); } +put_task_struct (task);return ctx; + + errout: +put_task_struct (task); +return ERR_PTR (err); }static void free_counter_rcu (struct rcu_head *head)
Quoted 2 lines · Peter Zijlstra
The poor task will now have to wait for yet another RCU grace period, but yeah :-)
I moved the put_task_struct into put_ctx, inside the if, so the poor task won't get impatient. :)
Quoted 5 lines
Looks good, although lockdep will complain, the proper annotation would be
spin_lock_nested_lock(&ctx->lock, &rq->lock), but since we don't have access to the rq here, we should change the secondspin_lock, to:spin_lock_nested(&next_ctx->lock, SINGLE_DEPTH_NESTING);
Ah OK, thanks.
Quoted 1 line
You jump into an
rcu_read_lock()section there.
Oops. I'll fix that. I also need to use spin_[un]lock_irq in there. New version coming. Thanks for the review.
Paul.