Skip to contentSkip to conversation

perf: Finer grained full dynticks handling

PATCH In review Frederic Weisbecker 5 participants 49 messages Jul 23, 2013 – Aug 2, 2013 perf events

lore
Share & tools
lore
#0 lore raw

cover perf: Finer grained full dynticks handling

Hi,

This patchset inspires from Peterz patch in http://marc.info/?l=linux-kernel&m=137155182121860
to make perf retain the timer tick less often, ie: only when there are freq events or when one throttles.

The main purpose is to make the lockup detector work with full dynticks, as I suspect that distros want to enable both...

You can fetch from:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
perf/nohz

Thanks,
Frederic
---

Frederic Weisbecker (8):
perf: Fix branch stack refcount leak on callchain init failure
perf: Sanitize get_callchain_buffer()
perf: Gather event accounting code
perf: Split per cpu event accounting code
perf: Migrate per cpu event accounting
perf: Account freq events per cpu
perf: Finer grained full dynticks kick
watchdog: Remove hack to make full dynticks working

 kernel/events/callchain.c |    2 +
 kernel/events/core.c      |  203 ++++++++++++++++++++++++++++-----------------
 kernel/watchdog.c         |    8 --
 3 files changed, 128 insertions(+), 85 deletions(-)
#1 lore raw

PATCH 1/8 perf: Fix branch stack refcount leak on callchain init failure

On callchain buffers allocation failure, free_event() is called and all the accounting performed in perf_event_alloc() for that event is cancelled.

But if the event has branch stack sampling, it is unaccounted as well from the branch stack sampling events refcounts.

This is a bug because this accounting is performed after the callchain buffer allocation. As a result, the branch stack sampling events refcount can become negative.

To fix this, move the branch stack event accounting before the callchain buffer allocation.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)
kernel/events/core.c +6 -6
index 5e2bce9..7ffb81e 100644
@@ -6566,6 +6566,12 @@ done:
 			atomic_inc(&nr_comm_events);
 		if (event->attr.task)
 			atomic_inc(&nr_task_events);
+		if (has_branch_stack(event)) {
+			static_key_slow_inc(&perf_sched_events.key);
+			if (!(event->attach_state & PERF_ATTACH_TASK))
+				atomic_inc(&per_cpu(perf_branch_stack_events,
+						    event->cpu));
+		}
 		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) {
 			err = get_callchain_buffers();
 			if (err) {
@@ -6573,12 +6579,6 @@ done:
 				return ERR_PTR(err);
 			}
 		}
-		if (has_branch_stack(event)) {
-			static_key_slow_inc(&perf_sched_events.key);
-			if (!(event->attach_state & PERF_ATTACH_TASK))
-				atomic_inc(&per_cpu(perf_branch_stack_events,
-						    event->cpu));
-		}
 	}
 
 	return event;
Signature · 1 line
1.7.5.4
#2 lore raw

[tip:perf/core] perf: Fix branch stack refcount leak on callchain init failure

perf: Fix branch stack refcount leak on callchain init failure
Commit-ID:  6050cb0b0b366092d1383bc23d7b16cd26db00f0
Gitweb:     http://git.kernel.org/tip/6050cb0b0b366092d1383bc23d7b16cd26db00f0
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Tue, 23 Jul 2013 02:30:59 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 30 Jul 2013 22:22:58 +0200

perf: Fix branch stack refcount leak on callchain init failure

On callchain buffers allocation failure, free_event() is called and all the accounting performed in perf_event_alloc() for that event is cancelled.

But if the event has branch stack sampling, it is unaccounted as well from the branch stack sampling events refcounts.

This is a bug because this accounting is performed after the callchain buffer allocation. As a result, the branch stack sampling events refcount can become negative.

To fix this, move the branch stack event accounting before the callchain buffer allocation.

---
 kernel/events/core.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
kernel/events/core.c +6 -6
index 1274114..f35aa7e 100644
@@ -6567,6 +6567,12 @@ done:
 			atomic_inc(&nr_comm_events);
 		if (event->attr.task)
 			atomic_inc(&nr_task_events);
+		if (has_branch_stack(event)) {
+			static_key_slow_inc(&perf_sched_events.key);
+			if (!(event->attach_state & PERF_ATTACH_TASK))
+				atomic_inc(&per_cpu(perf_branch_stack_events,
+						    event->cpu));
+		}
 		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) {
 			err = get_callchain_buffers();
 			if (err) {
@@ -6574,12 +6580,6 @@ done:
 				return ERR_PTR(err);
 			}
 		}
-		if (has_branch_stack(event)) {
-			static_key_slow_inc(&perf_sched_events.key);
-			if (!(event->attach_state & PERF_ATTACH_TASK))
-				atomic_inc(&per_cpu(perf_branch_stack_events,
-						    event->cpu));
-		}
 	}
 
 	return event;
#3 lore raw

PATCH 2/8 perf: Sanitize get_callchain_buffer()

In reply to Frederic Weisbecker

In case of allocation failure, get_callchain_buffer() keeps the refcount incremented for the current event.

As a result, when get_callchain_buffers() returns an error, we must cleanup what it did by cancelling its last refcount with a call to put_callchain_buffers().

This is a hack in order to be able to call free_event() after that failure.

The original purpose of that was to simplify the failure path. But this error handling is actually counter intuitive, ugly and not very easy to follow because one expect to see the resources used to perform a service to be cleaned by the callee if case of failure, not by the caller.

So lets clean this up by cancelling the refcount from get_callchain_buffer() in case of failure. And correctly free the event accordingly in perf_event_alloc().

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/callchain.c |    2 ++
 kernel/events/core.c      |   41 +++++++++++++++++++++--------------------
 2 files changed, 23 insertions(+), 20 deletions(-)
kernel/events/callchain.c +2 -0
index c772061..76a8bc5 100644
@@ -117,6 +117,8 @@ int get_callchain_buffers(void)
 	err = alloc_callchain_buffers();
 exit:
 	mutex_unlock(&callchain_mutex);
+	if (err)
+		atomic_dec(&nr_callchain_events);
 
 	return err;
 }
kernel/events/core.c +21 -20

69 lines not on this page · raw message

Signature · 1 line
1.7.5.4
#4 lore raw

[tip:perf/core] perf: Sanitize get_callchain_buffer()

perf: Sanitize get_callchain_buffer()
Commit-ID:  90983b16078ab0fdc58f0dab3e8e3da79c9579a2
Gitweb:     http://git.kernel.org/tip/90983b16078ab0fdc58f0dab3e8e3da79c9579a2
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Tue, 23 Jul 2013 02:31:00 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 30 Jul 2013 22:29:12 +0200

perf: Sanitize get_callchain_buffer()

In case of allocation failure, get_callchain_buffer() keeps the refcount incremented for the current event.

As a result, when get_callchain_buffers() returns an error, we must cleanup what it did by cancelling its last refcount with a call to put_callchain_buffers().

This is a hack in order to be able to call free_event() after that failure.

The original purpose of that was to simplify the failure path. But this error handling is actually counter intuitive, ugly and not very easy to follow because one expect to see the resources used to perform a service to be cleaned by the callee if case of failure, not by the caller.

So lets clean this up by cancelling the refcount from get_callchain_buffer() in case of failure. And correctly free the event accordingly in perf_event_alloc().

---
 kernel/events/callchain.c |  2 ++
 kernel/events/core.c      | 41 +++++++++++++++++++++--------------------
 2 files changed, 23 insertions(+), 20 deletions(-)
kernel/events/callchain.c +2 -0
index c772061..76a8bc5 100644
@@ -117,6 +117,8 @@ int get_callchain_buffers(void)
 	err = alloc_callchain_buffers();
 exit:
 	mutex_unlock(&callchain_mutex);
+	if (err)
+		atomic_dec(&nr_callchain_events);
 
 	return err;
 }
kernel/events/core.c +21 -20

69 lines not on this page · raw message

#5 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

In reply to Frederic Weisbecker
Quoted 37 lines · Frederic Weisbecker

In case of allocation failure, get_callchain_buffer() keeps the refcount incremented for the current event.

As a result, when get_callchain_buffers() returns an error, we must cleanup what it did by cancelling its last refcount with a call to put_callchain_buffers().

This is a hack in order to be able to call free_event() after that failure.

The original purpose of that was to simplify the failure path. But this error handling is actually counter intuitive, ugly and not very easy to follow because one expect to see the resources used to perform a service to be cleaned by the callee if case of failure, not by the caller.

So lets clean this up by cancelling the refcount from get_callchain_buffer() in case of failure. And correctly free the event accordingly in perf_event_alloc().

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/callchain.c |    2 ++
 kernel/events/core.c      |   41 +++++++++++++++++++++--------------------
 2 files changed, 23 insertions(+), 20 deletions(-)
kernel/events/callchain.c +0 -0
index c772061..76a8bc5 100644

kernel/events/callchain.c int get_callchain_buffers(void) · line 121

Quoted 5 lines
kernel/events/callchain.c +2 -0
@@ -117,6 +117,8 @@ int get_callchain_buffers(void)
 	err = alloc_callchain_buffers();
 exit:
 	mutex_unlock(&callchain_mutex);
+	if (err)
+		atomic_dec(&nr_callchain_events);

shouldn't we touch this under above lock?

also that above hunk decrements nr_callchain_events also for following case:

        count = atomic_inc_return(&nr_callchain_events);
        if (WARN_ON_ONCE(count < 1)) {
                err = -EINVAL;
                goto exit;
        }

seems like it screws the count

jirka

#6 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

Quoted 33 lines · Jiri Olsa
Quoted 32 lines · Frederic Weisbecker

In case of allocation failure, get_callchain_buffer() keeps the refcount incremented for the current event.

As a result, when get_callchain_buffers() returns an error, we must cleanup what it did by cancelling its last refcount with a call to put_callchain_buffers().

This is a hack in order to be able to call free_event() after that failure.

The original purpose of that was to simplify the failure path. But this error handling is actually counter intuitive, ugly and not very easy to follow because one expect to see the resources used to perform a service to be cleaned by the callee if case of failure, not by the caller.

So lets clean this up by cancelling the refcount from get_callchain_buffer() in case of failure. And correctly free the event accordingly in perf_event_alloc().

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/callchain.c |    2 ++
 kernel/events/core.c      |   41 +++++++++++++++++++++--------------------
 2 files changed, 23 insertions(+), 20 deletions(-)
Quoted 12 lines

kernel/events/callchain.c int get_callchain_buffers(void) · line 121

Quoted 10 lines
kernel/events/callchain.c +2 -0
index c772061..76a8bc5 100644
@@ -117,6 +117,8 @@ int get_callchain_buffers(void)
 	err = alloc_callchain_buffers();
 exit:
 	mutex_unlock(&callchain_mutex);
+	if (err)
+		atomic_dec(&nr_callchain_events);

shouldn't we touch this under above lock?

Right, we should move that under the lock, or another user of the callchains may see that we failed the allocation and simply give up instead of retrying.

Quoted 11 lines

also that above hunk decrements nr_callchain_events also for following case:

        count = atomic_inc_return(&nr_callchain_events);
        if (WARN_ON_ONCE(count < 1)) {
                err = -EINVAL;
                goto exit;
        }

seems like it screws the count

I'm not sure what you mean here. You mean that it could be negative when the dec is done outside the lock?

Quoted 2 lines

jirka

#7 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

On Thu, Aug 01, 2013 at 03:28:34PM +0200, Frederic Weisbecker wrote:

SNIP

Quoted 8 lines
Quoted 8 lines

also for following case:

        count = atomic_inc_return(&nr_callchain_events);
        if (WARN_ON_ONCE(count < 1)) {
                err = -EINVAL;
                goto exit;
        }
Quoted 4 lines
Quoted 1 line

seems like it screws the count

I'm not sure what you mean here. You mean that it could be negative when the dec is done outside the lock?

yes

#8 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

Quoted 12 lines · Jiri Olsa

On Thu, Aug 01, 2013 at 03:28:34PM +0200, Frederic Weisbecker wrote:

SNIP

Quoted 8 lines
Quoted 8 lines

also for following case:

        count = atomic_inc_return(&nr_callchain_events);
        if (WARN_ON_ONCE(count < 1)) {
                err = -EINVAL;
                goto exit;
        }
Quoted 7 lines
Quoted 4 lines
Quoted 1 line

seems like it screws the count

I'm not sure what you mean here. You mean that it could be negative when the dec is done outside the lock?

yes

I don't understand why.

#9 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

Quoted 15 lines · Frederic Weisbecker
Quoted 14 lines · Jiri Olsa

On Thu, Aug 01, 2013 at 03:28:34PM +0200, Frederic Weisbecker wrote:

SNIP

Quoted 10 lines
Quoted 9 lines

also for following case:

        count = atomic_inc_return(&nr_callchain_events);
        if (WARN_ON_ONCE(count < 1)) {
                err = -EINVAL;
                goto exit;
        }

seems like it screws the count

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

I'm not sure what you mean here. You mean that it could be negative when the dec is done outside the lock?

yes

I don't understand why.

ah ok, with the count being 0 the nr_callchain_events still increments right? then it's ok..

jirka

#10 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

Quoted 19 lines · Jiri Olsa
Quoted 18 lines · Frederic Weisbecker
Quoted 17 lines · Jiri Olsa

On Thu, Aug 01, 2013 at 03:28:34PM +0200, Frederic Weisbecker wrote:

SNIP

Quoted 12 lines
Quoted 9 lines

also for following case:

        count = atomic_inc_return(&nr_callchain_events);
        if (WARN_ON_ONCE(count < 1)) {
                err = -EINVAL;
                goto exit;
        }

seems like it screws the count

I'm not sure what you mean here. You mean that it could be negative when the dec is done outside the lock?

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

yes

I don't understand why.

ah ok, with the count being 0 the nr_callchain_events still increments right? then it's ok..

Yep. But I still need to move the dec into the lock though.

Thanks for catching this!

#11 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

In reply to Frederic Weisbecker

On Tue, Jul 23, 2013 at 02:31:00AM +0200, Frederic Weisbecker wrote:
SNIP

Quoted 17 lines
kernel/events/core.c +0 -7
 		if (event->attach_state & PERF_ATTACH_TASK)
 			static_key_slow_inc(&perf_sched_events.key);
 		if (event->attr.mmap || event->attr.mmap_data)
@@ -6572,16 +6570,19 @@ done:
 				atomic_inc(&per_cpu(perf_branch_stack_events,
 						    event->cpu));
 		}
-		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) {
-			err = get_callchain_buffers();
-			if (err) {
-				free_event(event);
-				return ERR_PTR(err);
-			}
-		}
 	}
 
 	return event;

kernel/events/core.c done: · line 6585

Quoted 10 lines
kernel/events/core.c +10 -0
+
+err_pmu:
+	if (event->destroy)
+		event->destroy(event);
+err_ns:
+	if (event->ns)
+		put_pid_ns(event->ns);
+	kfree(event);
+
+	return ERR_PTR(err);

could we call __free_filter(event) here?

jirka

#12 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

Quoted 19 lines · Jiri Olsa

On Tue, Jul 23, 2013 at 02:31:00AM +0200, Frederic Weisbecker wrote:
SNIP

Quoted 16 lines
kernel/events/core.c +0 -7
 		if (event->attach_state & PERF_ATTACH_TASK)
 			static_key_slow_inc(&perf_sched_events.key);
 		if (event->attr.mmap || event->attr.mmap_data)
@@ -6572,16 +6570,19 @@ done:
 				atomic_inc(&per_cpu(perf_branch_stack_events,
 						    event->cpu));
 		}
-		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) {
-			err = get_callchain_buffers();
-			if (err) {
-				free_event(event);
-				return ERR_PTR(err);
-			}
-		}
 	}
 
Quoted 13 lines

kernel/events/core.c done: · line 6585

Quoted 11 lines
kernel/events/core.c +10 -0
 	return event;
+
+err_pmu:
+	if (event->destroy)
+		event->destroy(event);
+err_ns:
+	if (event->ns)
+		put_pid_ns(event->ns);
+	kfree(event);
+
+	return ERR_PTR(err);

could we call __free_filter(event) here?

Hmm, the filters are installed from ioctl time so there shouldn't be any yet. But there should be an exception with inherited events. I fail to find where the filter is inherited though. Do we actually inherit those?

thanks.

#13 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

Quoted 34 lines · Frederic Weisbecker
Quoted 19 lines · Jiri Olsa

On Tue, Jul 23, 2013 at 02:31:00AM +0200, Frederic Weisbecker wrote:
SNIP

Quoted 16 lines
kernel/events/core.c +0 -7
 		if (event->attach_state & PERF_ATTACH_TASK)
 			static_key_slow_inc(&perf_sched_events.key);
 		if (event->attr.mmap || event->attr.mmap_data)
@@ -6572,16 +6570,19 @@ done:
 				atomic_inc(&per_cpu(perf_branch_stack_events,
 						    event->cpu));
 		}
-		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) {
-			err = get_callchain_buffers();
-			if (err) {
-				free_event(event);
-				return ERR_PTR(err);
-			}
-		}
 	}
 
Quoted 13 lines

kernel/events/core.c done: · line 6585

Quoted 11 lines
kernel/events/core.c +10 -0
 	return event;
+
+err_pmu:
+	if (event->destroy)
+		event->destroy(event);
+err_ns:
+	if (event->ns)
+		put_pid_ns(event->ns);
+	kfree(event);
+
+	return ERR_PTR(err);

could we call __free_filter(event) here?

Quoted 3 lines

Hmm, the filters are installed from ioctl time so there shouldn't be any yet. But there should be an exception with inherited events. I fail to find where the filter is inherited though. Do we actually inherit those?

ouch.. last I checked was freeing filter before writing this... :)

what I meant was the __free_event(event)

sorry

jirka

#14 lore raw

Re: [PATCH 2/8] perf: Sanitize get_callchain_buffer()

Quoted 35 lines · Jiri Olsa
Quoted 34 lines · Frederic Weisbecker
Quoted 32 lines · Jiri Olsa

On Tue, Jul 23, 2013 at 02:31:00AM +0200, Frederic Weisbecker wrote:
SNIP

Quoted 17 lines
kernel/events/core.c +0 -7
 		if (event->attach_state & PERF_ATTACH_TASK)
 			static_key_slow_inc(&perf_sched_events.key);
 		if (event->attr.mmap || event->attr.mmap_data)
@@ -6572,16 +6570,19 @@ done:
 				atomic_inc(&per_cpu(perf_branch_stack_events,
 						    event->cpu));
 		}
-		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN) {
-			err = get_callchain_buffers();
-			if (err) {
-				free_event(event);
-				return ERR_PTR(err);
-			}
-		}
 	}
 
 	return event;

kernel/events/core.c done: · line 6585

Quoted 10 lines
kernel/events/core.c +10 -0
+
+err_pmu:
+	if (event->destroy)
+		event->destroy(event);
+err_ns:
+	if (event->ns)
+		put_pid_ns(event->ns);
+	kfree(event);
+
+	return ERR_PTR(err);

could we call __free_filter(event) here?

Quoted 7 lines
Quoted 3 lines

Hmm, the filters are installed from ioctl time so there shouldn't be any yet. But there should be an exception with inherited events. I fail to find where the filter is inherited though. Do we actually inherit those?

ouch.. last I checked was freeing filter before writing this... :)

what I meant was the __free_event(event)

free_event() doesn't work either because we want several level of rollback depending of where the error triggered:

   +err_pmu:
           if (event->destroy)
                 event->destroy(event);
   +err_ns:
           if (event->ns)
                 put_pid_ns(event->ns);
           kfree(event);

return ERR_PTR(err)

If we fail after pmu init we want to call destroy, free pid ns and the event. If we fail before the pmu init, we want to only free pid ns and the event, ...

_free_event() does the whole in any case, which is not what we want.

But...

OTOH it might work due to the if (event->destroy) and if (event->ns) before freeing the resource associated.

So may be I can replace the labels with a single call to __free_event() after all as it checks what needs to be freed. What do you think?

#15 lore raw

PATCH 3/8 perf: Gather event accounting code

In reply to Frederic Weisbecker

Gather all the event accounting code to a single place, once all the prerequisites are completed. This simplifies the refcounting.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |   79 +++++++++++++++++++++++++++++--------------------
 1 files changed, 47 insertions(+), 32 deletions(-)
kernel/events/core.c +47 -32
index c5f435f..3bb73af 100644
@@ -3128,6 +3128,21 @@ static void free_event_rcu(struct rcu_head *head)
 static void ring_buffer_put(struct ring_buffer *rb);
 static void ring_buffer_detach(struct perf_event *event, struct ring_buffer *rb);
 
+static void __free_event(struct perf_event *event)
+{
+	if (!event->parent) {
+		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN)
+			put_callchain_buffers();
+	}
+
+	if (event->destroy)
+		event->destroy(event);
+
+	if (event->ctx)
+		put_ctx(event->ctx);
+
+	call_rcu(&event->rcu_head, free_event_rcu);
+}
 static void free_event(struct perf_event *event)
 {
 	irq_work_sync(&event->pending);
@@ -3141,8 +3156,6 @@ static void free_event(struct perf_event *event)
 			atomic_dec(&nr_comm_events);
 		if (event->attr.task)
 			atomic_dec(&nr_task_events);
-		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN)
-			put_callchain_buffers();
 		if (is_cgroup_event(event)) {
 			atomic_dec(&per_cpu(perf_cgroup_events, event->cpu));
 			static_key_slow_dec_deferred(&perf_sched_events);
@@ -3180,13 +3193,8 @@ static void free_event(struct perf_event *event)
 	if (is_cgroup_event(event))
 		perf_detach_cgroup(event);
 
-	if (event->destroy)
-		event->destroy(event);
-
-	if (event->ctx)
-		put_ctx(event->ctx);
 
-	call_rcu(&event->rcu_head, free_event_rcu);
+	__free_event(event);
 }
 
 int perf_event_release_kernel(struct perf_event *event)
@@ -6442,6 +6450,29 @@ unlock:
 	return pmu;
 }
 
+static void account_event(struct perf_event *event)
+{
+	if (event->attach_state & PERF_ATTACH_TASK)
+		static_key_slow_inc(&perf_sched_events.key);
+	if (event->attr.mmap || event->attr.mmap_data)
+		atomic_inc(&nr_mmap_events);
+	if (event->attr.comm)
+		atomic_inc(&nr_comm_events);
+	if (event->attr.task)
+		atomic_inc(&nr_task_events);
+	if (has_branch_stack(event)) {
+		static_key_slow_inc(&perf_sched_events.key);
+		if (!(event->attach_state & PERF_ATTACH_TASK))
+			atomic_inc(&per_cpu(perf_branch_stack_events,
+					    event->cpu));
+	}
+
+	if (is_cgroup_event(event)) {
+		atomic_inc(&per_cpu(perf_cgroup_events, event->cpu));
+		static_key_slow_inc(&perf_sched_events.key);
+	}
+}
+
 /*
  * Allocate and initialize a event structure
  */
@@ -6555,21 +6586,6 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
 			if (err)
 				goto err_pmu;
 		}
-
-		if (event->attach_state & PERF_ATTACH_TASK)
-			static_key_slow_inc(&perf_sched_events.key);
-		if (event->attr.mmap || event->attr.mmap_data)
-			atomic_inc(&nr_mmap_events);
-		if (event->attr.comm)
-			atomic_inc(&nr_comm_events);
-		if (event->attr.task)
-			atomic_inc(&nr_task_events);
-		if (has_branch_stack(event)) {
-			static_key_slow_inc(&perf_sched_events.key);
-			if (!(event->attach_state & PERF_ATTACH_TASK))
-				atomic_inc(&per_cpu(perf_branch_stack_events,
-						    event->cpu));
-		}
 	}
 
 	return event;
@@ -6864,17 +6880,14 @@ SYSCALL_DEFINE5(perf_event_open,
 
 	if (flags & PERF_FLAG_PID_CGROUP) {
 		err = perf_cgroup_connect(pid, event, &attr, group_leader);
-		if (err)
-			goto err_alloc;
-		/*
-		 * one more event:
-		 * - that has cgroup constraint on event->cpu
-		 * - that may need work on context switch
-		 */
-		atomic_inc(&per_cpu(perf_cgroup_events, event->cpu));
-		static_key_slow_inc(&perf_sched_events.key);
+		if (err) {
+			__free_event(event);
+			goto err_task;
+		}
 	}
 
+	account_event(event);
+
 	/*
 	 * Special case software events and allow them to be part of
 	 * any hardware group.
@@ -7070,6 +7083,8 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,
 		goto err;
 	}
 
+	account_event(event);
+
 	ctx = find_get_context(event->pmu, task, cpu);
 	if (IS_ERR(ctx)) {
 		err = PTR_ERR(ctx);
Signature · 1 line
1.7.5.4
#16 lore raw

[tip:perf/core] perf: Factor out event accounting code to account_event()/__free_event()

perf: Factor out event accounting code to account_event()/__free_event()
Commit-ID:  766d6c076928191d75ad5b0d0f58f52b1e7682d8
Gitweb:     http://git.kernel.org/tip/766d6c076928191d75ad5b0d0f58f52b1e7682d8
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Tue, 23 Jul 2013 02:31:01 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 30 Jul 2013 22:29:12 +0200

perf: Factor out event accounting code to account_event()/__free_event()

Gather all the event accounting code to a single place, once all the prerequisites are completed. This simplifies the refcounting.

---
 kernel/events/core.c | 79 +++++++++++++++++++++++++++++++---------------------
 1 file changed, 47 insertions(+), 32 deletions(-)
kernel/events/core.c +47 -32
index 3b99862..158fd57 100644
@@ -3128,6 +3128,21 @@ static void free_event_rcu(struct rcu_head *head)
 static void ring_buffer_put(struct ring_buffer *rb);
 static void ring_buffer_detach(struct perf_event *event, struct ring_buffer *rb);
 
+static void __free_event(struct perf_event *event)
+{
+	if (!event->parent) {
+		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN)
+			put_callchain_buffers();
+	}
+
+	if (event->destroy)
+		event->destroy(event);
+
+	if (event->ctx)
+		put_ctx(event->ctx);
+
+	call_rcu(&event->rcu_head, free_event_rcu);
+}
 static void free_event(struct perf_event *event)
 {
 	irq_work_sync(&event->pending);
@@ -3141,8 +3156,6 @@ static void free_event(struct perf_event *event)
 			atomic_dec(&nr_comm_events);
 		if (event->attr.task)
 			atomic_dec(&nr_task_events);
-		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN)
-			put_callchain_buffers();
 		if (is_cgroup_event(event)) {
 			atomic_dec(&per_cpu(perf_cgroup_events, event->cpu));
 			static_key_slow_dec_deferred(&perf_sched_events);
@@ -3180,13 +3193,8 @@ static void free_event(struct perf_event *event)
 	if (is_cgroup_event(event))
 		perf_detach_cgroup(event);
 
-	if (event->destroy)
-		event->destroy(event);
-
-	if (event->ctx)
-		put_ctx(event->ctx);
 
-	call_rcu(&event->rcu_head, free_event_rcu);
+	__free_event(event);
 }
 
 int perf_event_release_kernel(struct perf_event *event)
@@ -6443,6 +6451,29 @@ unlock:
 	return pmu;
 }
 
+static void account_event(struct perf_event *event)
+{
+	if (event->attach_state & PERF_ATTACH_TASK)
+		static_key_slow_inc(&perf_sched_events.key);
+	if (event->attr.mmap || event->attr.mmap_data)
+		atomic_inc(&nr_mmap_events);
+	if (event->attr.comm)
+		atomic_inc(&nr_comm_events);
+	if (event->attr.task)
+		atomic_inc(&nr_task_events);
+	if (has_branch_stack(event)) {
+		static_key_slow_inc(&perf_sched_events.key);
+		if (!(event->attach_state & PERF_ATTACH_TASK))
+			atomic_inc(&per_cpu(perf_branch_stack_events,
+					    event->cpu));
+	}
+
+	if (is_cgroup_event(event)) {
+		atomic_inc(&per_cpu(perf_cgroup_events, event->cpu));
+		static_key_slow_inc(&perf_sched_events.key);
+	}
+}
+
 /*
  * Allocate and initialize a event structure
  */
@@ -6556,21 +6587,6 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
 			if (err)
 				goto err_pmu;
 		}
-
-		if (event->attach_state & PERF_ATTACH_TASK)
-			static_key_slow_inc(&perf_sched_events.key);
-		if (event->attr.mmap || event->attr.mmap_data)
-			atomic_inc(&nr_mmap_events);
-		if (event->attr.comm)
-			atomic_inc(&nr_comm_events);
-		if (event->attr.task)
-			atomic_inc(&nr_task_events);
-		if (has_branch_stack(event)) {
-			static_key_slow_inc(&perf_sched_events.key);
-			if (!(event->attach_state & PERF_ATTACH_TASK))
-				atomic_inc(&per_cpu(perf_branch_stack_events,
-						    event->cpu));
-		}
 	}
 
 	return event;
@@ -6865,17 +6881,14 @@ SYSCALL_DEFINE5(perf_event_open,
 
 	if (flags & PERF_FLAG_PID_CGROUP) {
 		err = perf_cgroup_connect(pid, event, &attr, group_leader);
-		if (err)
-			goto err_alloc;
-		/*
-		 * one more event:
-		 * - that has cgroup constraint on event->cpu
-		 * - that may need work on context switch
-		 */
-		atomic_inc(&per_cpu(perf_cgroup_events, event->cpu));
-		static_key_slow_inc(&perf_sched_events.key);
+		if (err) {
+			__free_event(event);
+			goto err_task;
+		}
 	}
 
+	account_event(event);
+
 	/*
 	 * Special case software events and allow them to be part of
 	 * any hardware group.
@@ -7071,6 +7084,8 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu,
 		goto err;
 	}
 
+	account_event(event);
+
 	ctx = find_get_context(event->pmu, task, cpu);
 	if (IS_ERR(ctx)) {
 		err = PTR_ERR(ctx);
#17 lore raw

Re: [PATCH 3/8] perf: Gather event accounting code

In reply to Frederic Weisbecker
Quoted 31 lines · Frederic Weisbecker

Gather all the event accounting code to a single place, once all the prerequisites are completed. This simplifies the refcounting.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |   79 +++++++++++++++++++++++++++++--------------------
 1 files changed, 47 insertions(+), 32 deletions(-)
kernel/events/core.c +7 -0
index c5f435f..3bb73af 100644
@@ -3128,6 +3128,21 @@ static void free_event_rcu(struct rcu_head *head)
 static void ring_buffer_put(struct ring_buffer *rb);
 static void ring_buffer_detach(struct perf_event *event, struct ring_buffer *rb);
 
+static void __free_event(struct perf_event *event)
+{
+	if (!event->parent) {
+		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN)
+			put_callchain_buffers();
+	}
+

kernel/events/core.c static void free_event_rcu(struct rcu_head *head) · line 3147

Quoted 10 lines
kernel/events/core.c +8 -0
+	if (event->destroy)
+		event->destroy(event);
+
+	if (event->ctx)
+		put_ctx(event->ctx);
+
+	call_rcu(&event->rcu_head, free_event_rcu);
+}
 static void free_event(struct perf_event *event)
 {

nitpick, missing nl between functions

Quoted 12 lines
kernel/events/core.c +0 -2
 	irq_work_sync(&event->pending);
@@ -3141,8 +3156,6 @@ static void free_event(struct perf_event *event)
 			atomic_dec(&nr_comm_events);
 		if (event->attr.task)
 			atomic_dec(&nr_task_events);
-		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN)
-			put_callchain_buffers();
 		if (is_cgroup_event(event)) {
 			atomic_dec(&per_cpu(perf_cgroup_events, event->cpu));
 			static_key_slow_dec_deferred(&perf_sched_events);
@@ -3180,13 +3193,8 @@ static void free_event(struct perf_event *event)
 	if (is_cgroup_event(event))

kernel/events/core.c static void free_event(struct perf_event *event) · line 3197

Quoted 10 lines
kernel/events/core.c +1 -6
 		perf_detach_cgroup(event);
 
-	if (event->destroy)
-		event->destroy(event);
-
-	if (event->ctx)
-		put_ctx(event->ctx);
 
-	call_rcu(&event->rcu_head, free_event_rcu);
+	__free_event(event);

nitpick, extra nl above

#18 lore raw

Re: [PATCH 3/8] perf: Gather event accounting code

Quoted 65 lines · Jiri Olsa
Quoted 31 lines · Frederic Weisbecker

Gather all the event accounting code to a single place, once all the prerequisites are completed. This simplifies the refcounting.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |   79 +++++++++++++++++++++++++++++--------------------
 1 files changed, 47 insertions(+), 32 deletions(-)
kernel/events/core.c +7 -0
index c5f435f..3bb73af 100644
@@ -3128,6 +3128,21 @@ static void free_event_rcu(struct rcu_head *head)
 static void ring_buffer_put(struct ring_buffer *rb);
 static void ring_buffer_detach(struct perf_event *event, struct ring_buffer *rb);
 
+static void __free_event(struct perf_event *event)
+{
+	if (!event->parent) {
+		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN)
+			put_callchain_buffers();
+	}
+

kernel/events/core.c static void free_event_rcu(struct rcu_head *head) · line 3147

Quoted 10 lines
kernel/events/core.c +8 -0
+	if (event->destroy)
+		event->destroy(event);
+
+	if (event->ctx)
+		put_ctx(event->ctx);
+
+	call_rcu(&event->rcu_head, free_event_rcu);
+}
 static void free_event(struct perf_event *event)
 {

nitpick, missing nl between functions

Quoted 20 lines
kernel/events/core.c +0 -7
 	irq_work_sync(&event->pending);
@@ -3141,8 +3156,6 @@ static void free_event(struct perf_event *event)
 			atomic_dec(&nr_comm_events);
 		if (event->attr.task)
 			atomic_dec(&nr_task_events);
-		if (event->attr.sample_type & PERF_SAMPLE_CALLCHAIN)
-			put_callchain_buffers();
 		if (is_cgroup_event(event)) {
 			atomic_dec(&per_cpu(perf_cgroup_events, event->cpu));
 			static_key_slow_dec_deferred(&perf_sched_events);
@@ -3180,13 +3193,8 @@ static void free_event(struct perf_event *event)
 	if (is_cgroup_event(event))
 		perf_detach_cgroup(event);
 
-	if (event->destroy)
-		event->destroy(event);
-
-	if (event->ctx)
-		put_ctx(event->ctx);
 
Quoted 4 lines

kernel/events/core.c static void free_event(struct perf_event *event) · line 3197

Quoted 2 lines
kernel/events/core.c +1 -1
-	call_rcu(&event->rcu_head, free_event_rcu);
+	__free_event(event);

nitpick, extra nl above

Ok, I'll fix, thanks!

#19 lore raw

PATCH 4/8 perf: Split per cpu event accounting code

In reply to Frederic Weisbecker

Split the per-cpu accounting part of the event
accounting code.

This way we can use the per-cpu handling seperately. This is going to be used by to fix the event migration code accounting.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |   87 +++++++++++++++++++++++++++++++------------------
 1 files changed, 55 insertions(+), 32 deletions(-)
kernel/events/core.c +55 -32
index 3bb73af..fd820a4 100644
@@ -3128,6 +3128,40 @@ static void free_event_rcu(struct rcu_head *head)
 static void ring_buffer_put(struct ring_buffer *rb);
 static void ring_buffer_detach(struct perf_event *event, struct ring_buffer *rb);
 
+static void unaccount_event_cpu(struct perf_event *event, int cpu)
+{
+	if (event->parent)
+		return;
+
+	if (has_branch_stack(event)) {
+		if (!(event->attach_state & PERF_ATTACH_TASK))
+			atomic_dec(&per_cpu(perf_branch_stack_events, cpu));
+	}
+	if (is_cgroup_event(event))
+		atomic_dec(&per_cpu(perf_cgroup_events, cpu));
+}
+
+static void unaccount_event(struct perf_event *event)
+{
+	if (event->parent)
+		return;
+
+	if (event->attach_state & PERF_ATTACH_TASK)
+		static_key_slow_dec_deferred(&perf_sched_events);
+	if (event->attr.mmap || event->attr.mmap_data)
+		atomic_dec(&nr_mmap_events);
+	if (event->attr.comm)
+		atomic_dec(&nr_comm_events);
+	if (event->attr.task)
+		atomic_dec(&nr_task_events);
+	if (is_cgroup_event(event))
+		static_key_slow_dec_deferred(&perf_sched_events);
+	if (has_branch_stack(event))
+		static_key_slow_dec_deferred(&perf_sched_events);
+
+	unaccount_event_cpu(event, event->cpu);
+}
+
 static void __free_event(struct perf_event *event)
 {
 	if (!event->parent) {
@@ -3147,29 +3181,7 @@ static void free_event(struct perf_event *event)
 {
 	irq_work_sync(&event->pending);
 
-	if (!event->parent) {
-		if (event->attach_state & PERF_ATTACH_TASK)
-			static_key_slow_dec_deferred(&perf_sched_events);
-		if (event->attr.mmap || event->attr.mmap_data)
-			atomic_dec(&nr_mmap_events);
-		if (event->attr.comm)
-			atomic_dec(&nr_comm_events);
-		if (event->attr.task)
-			atomic_dec(&nr_task_events);
-		if (is_cgroup_event(event)) {
-			atomic_dec(&per_cpu(perf_cgroup_events, event->cpu));
-			static_key_slow_dec_deferred(&perf_sched_events);
-		}
-
-		if (has_branch_stack(event)) {
-			static_key_slow_dec_deferred(&perf_sched_events);
-			/* is system-wide event */
-			if (!(event->attach_state & PERF_ATTACH_TASK)) {
-				atomic_dec(&per_cpu(perf_branch_stack_events,
-						    event->cpu));
-			}
-		}
-	}
+	unaccount_event(event);
 
 	if (event->rb) {
 		struct ring_buffer *rb;
@@ -6450,8 +6462,24 @@ unlock:
 	return pmu;
 }
 
+static void account_event_cpu(struct perf_event *event, int cpu)
+{
+	if (event->parent)
+		return;
+
+	if (has_branch_stack(event)) {
+		if (!(event->attach_state & PERF_ATTACH_TASK))
+			atomic_inc(&per_cpu(perf_branch_stack_events, cpu));
+	}
+	if (is_cgroup_event(event))
+		atomic_inc(&per_cpu(perf_cgroup_events, cpu));
+}
+
 static void account_event(struct perf_event *event)
 {
+	if (event->parent)
+		return;
+
 	if (event->attach_state & PERF_ATTACH_TASK)
 		static_key_slow_inc(&perf_sched_events.key);
 	if (event->attr.mmap || event->attr.mmap_data)
@@ -6460,17 +6488,12 @@ static void account_event(struct perf_event *event)
 		atomic_inc(&nr_comm_events);
 	if (event->attr.task)
 		atomic_inc(&nr_task_events);
-	if (has_branch_stack(event)) {
+	if (has_branch_stack(event))
 		static_key_slow_inc(&perf_sched_events.key);
-		if (!(event->attach_state & PERF_ATTACH_TASK))
-			atomic_inc(&per_cpu(perf_branch_stack_events,
-					    event->cpu));
-	}
-
-	if (is_cgroup_event(event)) {
-		atomic_inc(&per_cpu(perf_cgroup_events, event->cpu));
+	if (is_cgroup_event(event))
 		static_key_slow_inc(&perf_sched_events.key);
-	}
+
+	account_event_cpu(event, event->cpu);
 }
 
 /*
Signature · 1 line
1.7.5.4
#20 lore raw

[tip:perf/core] perf: Split the per-cpu accounting part of the event accounting code

perf: Split the per-cpu accounting part of the event accounting code
Commit-ID:  4beb31f3657348a8b702dd014d01c520e522012f
Gitweb:     http://git.kernel.org/tip/4beb31f3657348a8b702dd014d01c520e522012f
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Tue, 23 Jul 2013 02:31:02 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 30 Jul 2013 22:29:13 +0200

perf: Split the per-cpu accounting part of the event accounting code

This way we can use the per-cpu handling seperately. This is going to be used by to fix the event migration code accounting.

---
 kernel/events/core.c | 87 +++++++++++++++++++++++++++++++++-------------------
 1 file changed, 55 insertions(+), 32 deletions(-)
kernel/events/core.c +55 -32
index 158fd57..3a4b73a 100644
@@ -3128,6 +3128,40 @@ static void free_event_rcu(struct rcu_head *head)
 static void ring_buffer_put(struct ring_buffer *rb);
 static void ring_buffer_detach(struct perf_event *event, struct ring_buffer *rb);
 
+static void unaccount_event_cpu(struct perf_event *event, int cpu)
+{
+	if (event->parent)
+		return;
+
+	if (has_branch_stack(event)) {
+		if (!(event->attach_state & PERF_ATTACH_TASK))
+			atomic_dec(&per_cpu(perf_branch_stack_events, cpu));
+	}
+	if (is_cgroup_event(event))
+		atomic_dec(&per_cpu(perf_cgroup_events, cpu));
+}
+
+static void unaccount_event(struct perf_event *event)
+{
+	if (event->parent)
+		return;
+
+	if (event->attach_state & PERF_ATTACH_TASK)
+		static_key_slow_dec_deferred(&perf_sched_events);
+	if (event->attr.mmap || event->attr.mmap_data)
+		atomic_dec(&nr_mmap_events);
+	if (event->attr.comm)
+		atomic_dec(&nr_comm_events);
+	if (event->attr.task)
+		atomic_dec(&nr_task_events);
+	if (is_cgroup_event(event))
+		static_key_slow_dec_deferred(&perf_sched_events);
+	if (has_branch_stack(event))
+		static_key_slow_dec_deferred(&perf_sched_events);
+
+	unaccount_event_cpu(event, event->cpu);
+}
+
 static void __free_event(struct perf_event *event)
 {
 	if (!event->parent) {
@@ -3147,29 +3181,7 @@ static void free_event(struct perf_event *event)
 {
 	irq_work_sync(&event->pending);
 
-	if (!event->parent) {
-		if (event->attach_state & PERF_ATTACH_TASK)
-			static_key_slow_dec_deferred(&perf_sched_events);
-		if (event->attr.mmap || event->attr.mmap_data)
-			atomic_dec(&nr_mmap_events);
-		if (event->attr.comm)
-			atomic_dec(&nr_comm_events);
-		if (event->attr.task)
-			atomic_dec(&nr_task_events);
-		if (is_cgroup_event(event)) {
-			atomic_dec(&per_cpu(perf_cgroup_events, event->cpu));
-			static_key_slow_dec_deferred(&perf_sched_events);
-		}
-
-		if (has_branch_stack(event)) {
-			static_key_slow_dec_deferred(&perf_sched_events);
-			/* is system-wide event */
-			if (!(event->attach_state & PERF_ATTACH_TASK)) {
-				atomic_dec(&per_cpu(perf_branch_stack_events,
-						    event->cpu));
-			}
-		}
-	}
+	unaccount_event(event);
 
 	if (event->rb) {
 		struct ring_buffer *rb;
@@ -6451,8 +6463,24 @@ unlock:
 	return pmu;
 }
 
+static void account_event_cpu(struct perf_event *event, int cpu)
+{
+	if (event->parent)
+		return;
+
+	if (has_branch_stack(event)) {
+		if (!(event->attach_state & PERF_ATTACH_TASK))
+			atomic_inc(&per_cpu(perf_branch_stack_events, cpu));
+	}
+	if (is_cgroup_event(event))
+		atomic_inc(&per_cpu(perf_cgroup_events, cpu));
+}
+
 static void account_event(struct perf_event *event)
 {
+	if (event->parent)
+		return;
+
 	if (event->attach_state & PERF_ATTACH_TASK)
 		static_key_slow_inc(&perf_sched_events.key);
 	if (event->attr.mmap || event->attr.mmap_data)
@@ -6461,17 +6489,12 @@ static void account_event(struct perf_event *event)
 		atomic_inc(&nr_comm_events);
 	if (event->attr.task)
 		atomic_inc(&nr_task_events);
-	if (has_branch_stack(event)) {
+	if (has_branch_stack(event))
 		static_key_slow_inc(&perf_sched_events.key);
-		if (!(event->attach_state & PERF_ATTACH_TASK))
-			atomic_inc(&per_cpu(perf_branch_stack_events,
-					    event->cpu));
-	}
-
-	if (is_cgroup_event(event)) {
-		atomic_inc(&per_cpu(perf_cgroup_events, event->cpu));
+	if (is_cgroup_event(event))
 		static_key_slow_inc(&perf_sched_events.key);
-	}
+
+	account_event_cpu(event, event->cpu);
 }
 
 /*
#21 lore raw

PATCH 5/8 perf: Migrate per cpu event accounting

In reply to Frederic Weisbecker

When an event is migrated, move the event per-cpu accounting accordingly so that branch stack and cgroup events work correctly on the new CPU.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
kernel/events/core.c +2 -0
index fd820a4..b40c3db 100644
@@ -7144,6 +7144,7 @@ void perf_pmu_migrate_context(struct pmu *pmu, int src_cpu, int dst_cpu)
 	list_for_each_entry_safe(event, tmp, &src_ctx->event_list,
 				 event_entry) {
 		perf_remove_from_context(event);
+		unaccount_event_cpu(event, src_cpu);
 		put_ctx(src_ctx);
 		list_add(&event->event_entry, &events);
 	}
@@ -7156,6 +7157,7 @@ void perf_pmu_migrate_context(struct pmu *pmu, int src_cpu, int dst_cpu)
 		list_del(&event->event_entry);
 		if (event->state >= PERF_EVENT_STATE_OFF)
 			event->state = PERF_EVENT_STATE_INACTIVE;
+		account_event_cpu(event, dst_cpu);
 		perf_install_in_context(dst_ctx, event, dst_cpu);
 		get_ctx(dst_ctx);
 	}
Signature · 1 line
1.7.5.4
#22 lore raw

[tip:perf/core] perf: Migrate per cpu event accounting

perf: Migrate per cpu event accounting
Commit-ID:  9a545de019b536771feefb76f85e5038b65c2190
Gitweb:     http://git.kernel.org/tip/9a545de019b536771feefb76f85e5038b65c2190
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Tue, 23 Jul 2013 02:31:03 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 30 Jul 2013 22:29:14 +0200

perf: Migrate per cpu event accounting

When an event is migrated, move the event per-cpu accounting accordingly so that branch stack and cgroup events work correctly on the new CPU.

---
 kernel/events/core.c | 2 ++
 1 file changed, 2 insertions(+)
kernel/events/core.c +2 -0
index 3a4b73a..63bdec9 100644
@@ -7145,6 +7145,7 @@ void perf_pmu_migrate_context(struct pmu *pmu, int src_cpu, int dst_cpu)
 	list_for_each_entry_safe(event, tmp, &src_ctx->event_list,
 				 event_entry) {
 		perf_remove_from_context(event);
+		unaccount_event_cpu(event, src_cpu);
 		put_ctx(src_ctx);
 		list_add(&event->event_entry, &events);
 	}
@@ -7157,6 +7158,7 @@ void perf_pmu_migrate_context(struct pmu *pmu, int src_cpu, int dst_cpu)
 		list_del(&event->event_entry);
 		if (event->state >= PERF_EVENT_STATE_OFF)
 			event->state = PERF_EVENT_STATE_INACTIVE;
+		account_event_cpu(event, dst_cpu);
 		perf_install_in_context(dst_ctx, event, dst_cpu);
 		get_ctx(dst_ctx);
 	}
#23 lore raw

PATCH 6/8 perf: Account freq events per cpu

In reply to Frederic Weisbecker

This is going to be used by the full dynticks subsystem as a finer-grained information to know when to keep and when to stop the tick.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/events/core.c +7 -0
index b40c3db..f9bd39b 100644
@@ -141,6 +141,7 @@ enum event_type_t {
 struct static_key_deferred perf_sched_events __read_mostly;
 static DEFINE_PER_CPU(atomic_t, perf_cgroup_events);
 static DEFINE_PER_CPU(atomic_t, perf_branch_stack_events);
+static DEFINE_PER_CPU(atomic_t, perf_freq_events);
 
 static atomic_t nr_mmap_events __read_mostly;
 static atomic_t nr_comm_events __read_mostly;
@@ -3139,6 +3140,9 @@ static void unaccount_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_dec(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_dec(&per_cpu(perf_freq_events, cpu));
 }
 
 static void unaccount_event(struct perf_event *event)
@@ -6473,6 +6477,9 @@ static void account_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_inc(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_inc(&per_cpu(perf_freq_events, cpu));
 }
 
 static void account_event(struct perf_event *event)
Signature · 1 line
1.7.5.4
#24 lore raw

[tip:perf/core] perf: Account freq events per cpu

perf: Account freq events per cpu
Commit-ID:  ba8a75c16e292c0a3a87406a77508cbbc6cf4ee2
Gitweb:     http://git.kernel.org/tip/ba8a75c16e292c0a3a87406a77508cbbc6cf4ee2
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Tue, 23 Jul 2013 02:31:04 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 30 Jul 2013 22:29:14 +0200

perf: Account freq events per cpu

This is going to be used by the full dynticks subsystem as a finer-grained information to know when to keep and when to stop the tick.

---
 kernel/events/core.c | 7 +++++++
 1 file changed, 7 insertions(+)
kernel/events/core.c +7 -0
index 63bdec9..3fe385a 100644
@@ -141,6 +141,7 @@ enum event_type_t {
 struct static_key_deferred perf_sched_events __read_mostly;
 static DEFINE_PER_CPU(atomic_t, perf_cgroup_events);
 static DEFINE_PER_CPU(atomic_t, perf_branch_stack_events);
+static DEFINE_PER_CPU(atomic_t, perf_freq_events);
 
 static atomic_t nr_mmap_events __read_mostly;
 static atomic_t nr_comm_events __read_mostly;
@@ -3139,6 +3140,9 @@ static void unaccount_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_dec(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_dec(&per_cpu(perf_freq_events, cpu));
 }
 
 static void unaccount_event(struct perf_event *event)
@@ -6474,6 +6478,9 @@ static void account_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_inc(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_inc(&per_cpu(perf_freq_events, cpu));
 }
 
 static void account_event(struct perf_event *event)
#25 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

In reply to Frederic Weisbecker
Quoted 39 lines · Frederic Weisbecker

This is going to be used by the full dynticks subsystem as a finer-grained information to know when to keep and when to stop the tick.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/events/core.c +4 -0
index b40c3db..f9bd39b 100644
@@ -141,6 +141,7 @@ enum event_type_t {
 struct static_key_deferred perf_sched_events __read_mostly;
 static DEFINE_PER_CPU(atomic_t, perf_cgroup_events);
 static DEFINE_PER_CPU(atomic_t, perf_branch_stack_events);
+static DEFINE_PER_CPU(atomic_t, perf_freq_events);
 
 static atomic_t nr_mmap_events __read_mostly;
 static atomic_t nr_comm_events __read_mostly;
@@ -3139,6 +3140,9 @@ static void unaccount_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_dec(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_dec(&per_cpu(perf_freq_events, cpu));
 }
 
 static void unaccount_event(struct perf_event *event)

kernel/events/core.c static void account_event_cpu(struct perf_event *event, int cpu) · line 6482

Quoted 6 lines
kernel/events/core.c +3 -0
@@ -6473,6 +6477,9 @@ static void account_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_inc(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_inc(&per_cpu(perf_freq_events, cpu));

cpu could be -1 in here.. getting:

[  178.901881] BUG: unable to handle kernel paging request at 0000000f001cf45a
[  178.901989] IP: [<ffffffff8110c32b>] account_event_cpu+0xbb/0xd0
[  178.901989] PGD 751a5067 PUD 0 
[  178.901989] Oops: 0002 [#2] PREEMPT SMP 
[  178.901989] Modules linked in:
[  178.901989] CPU: 0 PID: 1309 Comm: perf Tainted: G      D W    3.11.0-rc3+ #281
[  178.901989] Hardware name: Intel Corporation Montevina platform/To be filled by O.E.M., BIOS AMVACRB1.86C.0066.B00.0805070703 05/07/2008
[  178.901989] task: ffff88007519e5c0 ti: ffff880074c08000 task.ti: ffff880074c08000
[  178.901989] RIP: 0010:[<ffffffff8110c32b>]  [<ffffffff8110c32b>] account_event_cpu+0xbb/0xd0
[  178.901989] RSP: 0018:ffff880074c09e38  EFLAGS: 00010216
[  178.901989] RAX: 0000000f001cf45a RBX: ffff880074c1a800 RCX: 0000000000100000
[  178.901989] RDX: 000000000000002c RSI: 000000000000002c RDI: ffff880000120400
[  178.901989] RBP: ffff880074c09e48 R08: 0000000000000000 R09: 00000000ffffffff
[  178.901989] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffffffffff
[  178.901989] R13: ffff880074c0e640 R14: 00000000ffffffff R15: 0000000000000000
[  178.901989] FS:  00007fc670738980(0000) GS:ffff88007a600000(0000) knlGS:0000000000000000
[  178.901989] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  178.901989] CR2: 0000000f001cf45a CR3: 00000000751a8000 CR4: 00000000000407f0
[  178.901989] Stack:
[  178.901989]  ffff880074c1a800 ffff880074c09ee8 ffff880074c09e68 ffffffff8110c3b6
[  178.901989]  00000000ffffffff ffff880074c1a800 ffff880074c09f78 ffffffff81113ae8
[  178.901989]  0000000000000000 0000000000000001 ffff880074c09eb8 ffffffff00000060
[  178.901989] Call Trace:
[  178.901989]  [<ffffffff8110c3b6>] account_event.part.66+0x76/0xa0
[  178.901989]  [<ffffffff81113ae8>] SyS_perf_event_open+0x678/0xdf0
[  178.901989]  [<ffffffff8100ed42>] ? syscall_trace_leave+0xb2/0x240
[  178.901989]  [<ffffffff81641512>] tracesys+0xd0/0xd5
[  178.901989] Code: d4 48 c7 c0 28 f4 1c 00 48 03 04 d5 80 ad ce 81 f0 ff 00 f6 83 c9 00 00 00 04 74 a7 48 c7 c0 4c f4 1c 00 4a 03 04 e5 80 ad ce 81 <f0> ff 00 5b 41 5c 5d c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 
[  178.901989] RIP  [<ffffffff8110c32b>] account_event_cpu+0xbb/0xd0
[  178.901989]  RSP <ffff880074c09e38>
[  178.901989] CR2: 0000000f001cf45a
[  179.122749] ---[ end trace 6f2f4f69b01368fb ]---

jirka

#26 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

Quoted 73 lines · Jiri Olsa
Quoted 39 lines · Frederic Weisbecker

This is going to be used by the full dynticks subsystem as a finer-grained information to know when to keep and when to stop the tick.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/events/core.c +4 -0
index b40c3db..f9bd39b 100644
@@ -141,6 +141,7 @@ enum event_type_t {
 struct static_key_deferred perf_sched_events __read_mostly;
 static DEFINE_PER_CPU(atomic_t, perf_cgroup_events);
 static DEFINE_PER_CPU(atomic_t, perf_branch_stack_events);
+static DEFINE_PER_CPU(atomic_t, perf_freq_events);
 
 static atomic_t nr_mmap_events __read_mostly;
 static atomic_t nr_comm_events __read_mostly;
@@ -3139,6 +3140,9 @@ static void unaccount_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_dec(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_dec(&per_cpu(perf_freq_events, cpu));
 }
 
 static void unaccount_event(struct perf_event *event)

kernel/events/core.c static void account_event_cpu(struct perf_event *event, int cpu) · line 6482

Quoted 6 lines
kernel/events/core.c +3 -0
@@ -6473,6 +6477,9 @@ static void account_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_inc(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_inc(&per_cpu(perf_freq_events, cpu));

cpu could be -1 in here.. getting:

[  178.901881] BUG: unable to handle kernel paging request at 0000000f001cf45a
[  178.901989] IP: [<ffffffff8110c32b>] account_event_cpu+0xbb/0xd0
[  178.901989] PGD 751a5067 PUD 0 
[  178.901989] Oops: 0002 [#2] PREEMPT SMP 
[  178.901989] Modules linked in:
[  178.901989] CPU: 0 PID: 1309 Comm: perf Tainted: G      D W    3.11.0-rc3+ #281
[  178.901989] Hardware name: Intel Corporation Montevina platform/To be filled by O.E.M., BIOS AMVACRB1.86C.0066.B00.0805070703 05/07/2008
[  178.901989] task: ffff88007519e5c0 ti: ffff880074c08000 task.ti: ffff880074c08000
[  178.901989] RIP: 0010:[<ffffffff8110c32b>]  [<ffffffff8110c32b>] account_event_cpu+0xbb/0xd0
[  178.901989] RSP: 0018:ffff880074c09e38  EFLAGS: 00010216
[  178.901989] RAX: 0000000f001cf45a RBX: ffff880074c1a800 RCX: 0000000000100000
[  178.901989] RDX: 000000000000002c RSI: 000000000000002c RDI: ffff880000120400
[  178.901989] RBP: ffff880074c09e48 R08: 0000000000000000 R09: 00000000ffffffff
[  178.901989] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffffffffff
[  178.901989] R13: ffff880074c0e640 R14: 00000000ffffffff R15: 0000000000000000
[  178.901989] FS:  00007fc670738980(0000) GS:ffff88007a600000(0000) knlGS:0000000000000000
[  178.901989] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  178.901989] CR2: 0000000f001cf45a CR3: 00000000751a8000 CR4: 00000000000407f0
[  178.901989] Stack:
[  178.901989]  ffff880074c1a800 ffff880074c09ee8 ffff880074c09e68 ffffffff8110c3b6
[  178.901989]  00000000ffffffff ffff880074c1a800 ffff880074c09f78 ffffffff81113ae8
[  178.901989]  0000000000000000 0000000000000001 ffff880074c09eb8 ffffffff00000060
[  178.901989] Call Trace:
[  178.901989]  [<ffffffff8110c3b6>] account_event.part.66+0x76/0xa0
Quoted 8 lines
[  178.901989]  [<ffffffff81113ae8>] SyS_perf_event_open+0x678/0xdf0
[  178.901989]  [<ffffffff8100ed42>] ? syscall_trace_leave+0xb2/0x240
[  178.901989]  [<ffffffff81641512>] tracesys+0xd0/0xd5
[  178.901989] Code: d4 48 c7 c0 28 f4 1c 00 48 03 04 d5 80 ad ce 81 f0 ff 00 f6 83 c9 00 00 00 04 74 a7 48 c7 c0 4c f4 1c 00 4a 03 04 e5 80 ad ce 81 <f0> ff 00 5b 41 5c 5d c3 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 
[  178.901989] RIP  [<ffffffff8110c32b>] account_event_cpu+0xbb/0xd0
[  178.901989]  RSP <ffff880074c09e38>
[  178.901989] CR2: 0000000f001cf45a
[  179.122749] ---[ end trace 6f2f4f69b01368fb ]---

the testcase was:

[root@dhcp-26-214 tracing]# cat
...
[root@dhcp-26-214 perf]# ./perf record -p pgrep cat Killed

jirka

maintainer
#27 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

In reply to Jiri Olsa
Quoted 38 lines · Jiri Olsa
Quoted 37 lines · Frederic Weisbecker

This is going to be used by the full dynticks subsystem as a finer-grained information to know when to keep and when to stop the tick.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/events/core.c +4 -0
index b40c3db..f9bd39b 100644
@@ -141,6 +141,7 @@ enum event_type_t {
 struct static_key_deferred perf_sched_events __read_mostly;
 static DEFINE_PER_CPU(atomic_t, perf_cgroup_events);
 static DEFINE_PER_CPU(atomic_t, perf_branch_stack_events);
+static DEFINE_PER_CPU(atomic_t, perf_freq_events);
 
 static atomic_t nr_mmap_events __read_mostly;
 static atomic_t nr_comm_events __read_mostly;
@@ -3139,6 +3140,9 @@ static void unaccount_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_dec(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_dec(&per_cpu(perf_freq_events, cpu));
 }
 
Quoted 10 lines

kernel/events/core.c static void account_event_cpu(struct perf_event *event, int cpu) · line 6482

Quoted 8 lines
kernel/events/core.c +3 -0
 static void unaccount_event(struct perf_event *event)
@@ -6473,6 +6477,9 @@ static void account_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_inc(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_inc(&per_cpu(perf_freq_events, cpu));

cpu could be -1 in here.. getting:

Ho humm, right you are.

So we have:

static void account_event_cpu(struct perf_event *event, int cpu)

{
	if (event->parent)
		return;

	if (has_branch_stack(event)) {
		if (!(event->attach_state & PERF_ATTACH_TASK))
			atomic_inc(&per_cpu(perf_branch_stack_events, cpu));
	}
	if (is_cgroup_event(event))
		atomic_inc(&per_cpu(perf_cgroup_events, cpu));

	if (event->attr.freq)
		atomic_inc(&per_cpu(perf_freq_events, cpu));
}

Where the freq thing is new and shiney, but we already had the other two. Of those, cgroup events must be per cpu so that should be good, the branch_stack thing tests ATTACH_TASK, which should also be good, but leaves me wonder wth they do for those that are attached to tasks.

But yes, the frequency thing is borken.

maintainer
#28 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

Quoted 2 lines · Peter Zijlstra

the branch_stack thing tests ATTACH_TASK, which should also be good, but leaves me wonder wth they do for those that are attached to tasks.

Ah found it, in intel_pmu_lbr_enable() we test cpuc->lbr_context against event->ctx and wipe the LBR if they differ, that should indeed avoid the leak but isn't sufficient to guarantee sane data as the context swap for inherited counters can foil this.

#29 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

In reply to Peter Zijlstra
Quoted 70 lines · Peter Zijlstra
Quoted 38 lines · Jiri Olsa
Quoted 37 lines · Frederic Weisbecker

This is going to be used by the full dynticks subsystem as a finer-grained information to know when to keep and when to stop the tick.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/events/core.c +4 -0
index b40c3db..f9bd39b 100644
@@ -141,6 +141,7 @@ enum event_type_t {
 struct static_key_deferred perf_sched_events __read_mostly;
 static DEFINE_PER_CPU(atomic_t, perf_cgroup_events);
 static DEFINE_PER_CPU(atomic_t, perf_branch_stack_events);
+static DEFINE_PER_CPU(atomic_t, perf_freq_events);
 
 static atomic_t nr_mmap_events __read_mostly;
 static atomic_t nr_comm_events __read_mostly;
@@ -3139,6 +3140,9 @@ static void unaccount_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_dec(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_dec(&per_cpu(perf_freq_events, cpu));
 }
 
Quoted 10 lines

kernel/events/core.c static void account_event_cpu(struct perf_event *event, int cpu) · line 6482

Quoted 8 lines
kernel/events/core.c +3 -0
 static void unaccount_event(struct perf_event *event)
@@ -6473,6 +6477,9 @@ static void account_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_inc(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_inc(&per_cpu(perf_freq_events, cpu));

cpu could be -1 in here.. getting:

Ho humm, right you are.

So we have:

static void account_event_cpu(struct perf_event *event, int cpu)

{
	if (event->parent)
		return;

	if (has_branch_stack(event)) {
		if (!(event->attach_state & PERF_ATTACH_TASK))
			atomic_inc(&per_cpu(perf_branch_stack_events, cpu));
	}
	if (is_cgroup_event(event))
		atomic_inc(&per_cpu(perf_cgroup_events, cpu));

	if (event->attr.freq)
		atomic_inc(&per_cpu(perf_freq_events, cpu));
}
Quoted 4 lines

Where the freq thing is new and shiney, but we already had the other two. Of those, cgroup events must be per cpu so that should be good, the branch_stack thing tests ATTACH_TASK, which should also be good, but leaves me wonder wth they do for those that are attached to tasks.

cgroup is cpu only:

SYSCALL(..

        if ((flags & PERF_FLAG_PID_CGROUP) && (pid == -1 || cpu == -1))
                return -EINVAL;

jirka

maintainer
#30 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

Quoted 4 lines · Jiri Olsa
Quoted 1 line · Peter Zijlstra

two. Of those, cgroup events must be per cpu so that should be good,

cgroup is cpu only:

jah! :-)

#31 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

In reply to Peter Zijlstra
Quoted 70 lines · Peter Zijlstra
Quoted 38 lines · Jiri Olsa
Quoted 37 lines · Frederic Weisbecker

This is going to be used by the full dynticks subsystem as a finer-grained information to know when to keep and when to stop the tick.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)
kernel/events/core.c +4 -0
index b40c3db..f9bd39b 100644
@@ -141,6 +141,7 @@ enum event_type_t {
 struct static_key_deferred perf_sched_events __read_mostly;
 static DEFINE_PER_CPU(atomic_t, perf_cgroup_events);
 static DEFINE_PER_CPU(atomic_t, perf_branch_stack_events);
+static DEFINE_PER_CPU(atomic_t, perf_freq_events);
 
 static atomic_t nr_mmap_events __read_mostly;
 static atomic_t nr_comm_events __read_mostly;
@@ -3139,6 +3140,9 @@ static void unaccount_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_dec(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_dec(&per_cpu(perf_freq_events, cpu));
 }
 
Quoted 10 lines

kernel/events/core.c static void account_event_cpu(struct perf_event *event, int cpu) · line 6482

Quoted 8 lines
kernel/events/core.c +3 -0
 static void unaccount_event(struct perf_event *event)
@@ -6473,6 +6477,9 @@ static void account_event_cpu(struct perf_event *event, int cpu)
 	}
 	if (is_cgroup_event(event))
 		atomic_inc(&per_cpu(perf_cgroup_events, cpu));
+
+	if (event->attr.freq)
+		atomic_inc(&per_cpu(perf_freq_events, cpu));

cpu could be -1 in here.. getting:

Ho humm, right you are.

So we have:

static void account_event_cpu(struct perf_event *event, int cpu)

{
	if (event->parent)
		return;

	if (has_branch_stack(event)) {
		if (!(event->attach_state & PERF_ATTACH_TASK))
			atomic_inc(&per_cpu(perf_branch_stack_events, cpu));
	}
	if (is_cgroup_event(event))
		atomic_inc(&per_cpu(perf_cgroup_events, cpu));

	if (event->attr.freq)
		atomic_inc(&per_cpu(perf_freq_events, cpu));
}
Quoted 6 lines

Where the freq thing is new and shiney, but we already had the other two. Of those, cgroup events must be per cpu so that should be good, the branch_stack thing tests ATTACH_TASK, which should also be good, but leaves me wonder wth they do for those that are attached to tasks.

But yes, the frequency thing is borken.

Aie, so the freq thing, I can either account to all CPUs (inc to all and send an IPI to all), or when the event scheds in/out. Probably we should do the former to avoid sending an IPI at all context switches.

maintainer
#32 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

Quoted 7 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra

Where the freq thing is new and shiney, but we already had the other two. Of those, cgroup events must be per cpu so that should be good, the branch_stack thing tests ATTACH_TASK, which should also be good, but leaves me wonder wth they do for those that are attached to tasks.

Quoted 5 lines
Quoted 1 line

But yes, the frequency thing is borken.

Aie, so the freq thing, I can either account to all CPUs (inc to all and send an IPI to all), or when the event scheds in/out. Probably we should do the former to avoid sending an IPI at all context switches.

Yeah, just go with a single global state for now..

maintainer
#33 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra

Where the freq thing is new and shiney, but we already had the other two. Of those, cgroup events must be per cpu so that should be good, the branch_stack thing tests ATTACH_TASK, which should also be good, but leaves me wonder wth they do for those that are attached to tasks.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

But yes, the frequency thing is borken.

Aie, so the freq thing, I can either account to all CPUs (inc to all and send an IPI to all), or when the event scheds in/out. Probably we should do the former to avoid sending an IPI at all context switches.

Yeah, just go with a single global state for now..

The perf default is to create inherited counter, which are per cpu anyway. So we'll not loose much.

#34 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

Quoted 11 lines · Peter Zijlstra
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Frederic Weisbecker
Quoted 7 lines · Peter Zijlstra

Where the freq thing is new and shiney, but we already had the other two. Of those, cgroup events must be per cpu so that should be good, the branch_stack thing tests ATTACH_TASK, which should also be good, but leaves me wonder wth they do for those that are attached to tasks.

But yes, the frequency thing is borken.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

Aie, so the freq thing, I can either account to all CPUs (inc to all and send an IPI to all), or when the event scheds in/out. Probably we should do the former to avoid sending an IPI at all context switches.

Yeah, just go with a single global state for now..

The perf default is to create inherited counter, which are per cpu anyway. So we'll not loose much.

So you mean that I keep the per cpu state when event->cpu != -1 and also have a global counter for the others. Right?

maintainer
#35 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

Quoted 7 lines · Frederic Weisbecker
Quoted 4 lines
Quoted 1 line

Yeah, just go with a single global state for now..

The perf default is to create inherited counter, which are per cpu anyway. So we'll not loose much.

So you mean that I keep the per cpu state when event->cpu != -1 and also have a global counter for the others. Right?

No, only a single global counter. The fact that perf, by default, creates a counter per cpu, means that there's little effective difference between a single global counter and per-cpu counters.

#36 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Frederic Weisbecker
Quoted 4 lines
Quoted 1 line

Yeah, just go with a single global state for now..

The perf default is to create inherited counter, which are per cpu anyway. So we'll not loose much.

Quoted 6 lines
Quoted 2 lines

So you mean that I keep the per cpu state when event->cpu != -1 and also have a global counter for the others. Right?

No, only a single global counter. The fact that perf, by default, creates a counter per cpu, means that there's little effective difference between a single global counter and per-cpu counters.

Good point! I just included that in the changelog.

Thanks.

#37 lore raw

Re: [PATCH 6/8] perf: Account freq events per cpu

In reply to Peter Zijlstra
Quoted 8 lines · Peter Zijlstra
Quoted 7 lines · Frederic Weisbecker
Quoted 6 lines · Peter Zijlstra

Where the freq thing is new and shiney, but we already had the other two. Of those, cgroup events must be per cpu so that should be good, the branch_stack thing tests ATTACH_TASK, which should also be good, but leaves me wonder wth they do for those that are attached to tasks.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

But yes, the frequency thing is borken.

Aie, so the freq thing, I can either account to all CPUs (inc to all and send an IPI to all), or when the event scheds in/out. Probably we should do the former to avoid sending an IPI at all context switches.

Yeah, just go with a single global state for now..

A single global counter? Ok sounds good.

#38 lore raw

PATCH 7/8 perf: Finer grained full dynticks kick

In reply to Frederic Weisbecker

Currently the full dynticks subsystem keep the tick alive as long as there are perf events running.

This prevents the tick from being stopped as long as features such that the lockup detectors are running. As a temporary fix, the lockup detector is disabled by default when full dynticks is built but this is not a long term viable solution.

To fix this, only keep the tick alive when an event configured with a frequency rather than a period is running on the CPU, or when an event throttles on the CPU.

These are the only purposes of the perf tick, especially now that the rotation of flexible events is handled from a seperate hrtimer. The tick can be shutdown the rest of the time.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
---
 kernel/events/core.c |   17 +++++++++--------
 1 files changed, 9 insertions(+), 8 deletions(-)
kernel/events/core.c +9 -8
index f9bd39b..61d70b7 100644
@@ -870,12 +870,8 @@ static void perf_pmu_rotate_start(struct pmu *pmu)
 
 	WARN_ON(!irqs_disabled());
 
-	if (list_empty(&cpuctx->rotation_list)) {
-		int was_empty = list_empty(head);
+	if (list_empty(&cpuctx->rotation_list))
 		list_add(&cpuctx->rotation_list, head);
-		if (was_empty)
-			tick_nohz_full_kick();
-	}
 }
 
 static void get_ctx(struct perf_event_context *ctx)
@@ -1875,6 +1871,9 @@ static int  __perf_install_in_context(void *info)
 	perf_pmu_enable(cpuctx->ctx.pmu);
 	perf_ctx_unlock(cpuctx, task_ctx);
 
+	if (atomic_read(&__get_cpu_var(perf_freq_events)))
+		tick_nohz_full_kick();
+
 	return 0;
 }
 
@@ -2812,10 +2811,11 @@ done:
 #ifdef CONFIG_NO_HZ_FULL
 bool perf_event_can_stop_tick(void)
 {
-	if (list_empty(&__get_cpu_var(rotation_list)))
-		return true;
-	else
+	if (atomic_read(&__get_cpu_var(perf_freq_events)) ||
+	    __this_cpu_read(perf_throttled_count))
 		return false;
+	else
+		return true;
 }
 #endif
 
@@ -5201,6 +5201,7 @@ static int __perf_event_overflow(struct perf_event *event,
 			__this_cpu_inc(perf_throttled_count);
 			hwc->interrupts = MAX_INTERRUPTS;
 			perf_log_throttle(event, 0);
+			tick_nohz_full_kick();
 			ret = 1;
 		}
 	}
Signature · 1 line
1.7.5.4
#39 lore raw

[tip:perf/core] perf: Implement finer grained full dynticks kick

perf: Implement finer grained full dynticks kick
Commit-ID:  d84153d6c96f61aa06429586284639f32debf03e
Gitweb:     http://git.kernel.org/tip/d84153d6c96f61aa06429586284639f32debf03e
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Tue, 23 Jul 2013 02:31:05 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 30 Jul 2013 22:29:15 +0200

perf: Implement finer grained full dynticks kick

Currently the full dynticks subsystem keep the tick alive as long as there are perf events running.

This prevents the tick from being stopped as long as features such that the lockup detectors are running. As a temporary fix, the lockup detector is disabled by default when full dynticks is built but this is not a long term viable solution.

To fix this, only keep the tick alive when an event configured with a frequency rather than a period is running on the CPU, or when an event throttles on the CPU.

These are the only purposes of the perf tick, especially now that the rotation of flexible events is handled from a seperate hrtimer. The tick can be shutdown the rest of the time.

---
 kernel/events/core.c | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)
kernel/events/core.c +9 -8
index 3fe385a..916cf1f 100644
@@ -870,12 +870,8 @@ static void perf_pmu_rotate_start(struct pmu *pmu)
 
 	WARN_ON(!irqs_disabled());
 
-	if (list_empty(&cpuctx->rotation_list)) {
-		int was_empty = list_empty(head);
+	if (list_empty(&cpuctx->rotation_list))
 		list_add(&cpuctx->rotation_list, head);
-		if (was_empty)
-			tick_nohz_full_kick();
-	}
 }
 
 static void get_ctx(struct perf_event_context *ctx)
@@ -1875,6 +1871,9 @@ static int  __perf_install_in_context(void *info)
 	perf_pmu_enable(cpuctx->ctx.pmu);
 	perf_ctx_unlock(cpuctx, task_ctx);
 
+	if (atomic_read(&__get_cpu_var(perf_freq_events)))
+		tick_nohz_full_kick();
+
 	return 0;
 }
 
@@ -2812,10 +2811,11 @@ done:
 #ifdef CONFIG_NO_HZ_FULL
 bool perf_event_can_stop_tick(void)
 {
-	if (list_empty(&__get_cpu_var(rotation_list)))
-		return true;
-	else
+	if (atomic_read(&__get_cpu_var(perf_freq_events)) ||
+	    __this_cpu_read(perf_throttled_count))
 		return false;
+	else
+		return true;
 }
 #endif
 
@@ -5202,6 +5202,7 @@ static int __perf_event_overflow(struct perf_event *event,
 			__this_cpu_inc(perf_throttled_count);
 			hwc->interrupts = MAX_INTERRUPTS;
 			perf_log_throttle(event, 0);
+			tick_nohz_full_kick();
 			ret = 1;
 		}
 	}
#40 lore raw

PATCH 8/8 watchdog: Remove hack to make full dynticks working

In reply to Frederic Weisbecker

A perf event can be used without forcing the tick to stay alive if it doesn't use a frequency but a sample period and if it doesn't throttle (raise storm of events).

Since the lockup detector neither use a perf event frequency nor should ever throttle due to its high period, it can now run concurrently with the full dynticks feature.

Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Don Zickus <[email protected]>
Cc Srivatsa S. Bhat <[email protected]>
Cc Anish Singh <[email protected]>
---
 kernel/watchdog.c |    8 --------
 1 files changed, 0 insertions(+), 8 deletions(-)
kernel/watchdog.c +0 -8
index 1241d8c..51c4f34 100644
@@ -553,14 +553,6 @@ void __init lockup_detector_init(void)
 {
 	set_sample_period();
 
-#ifdef CONFIG_NO_HZ_FULL
-	if (watchdog_user_enabled) {
-		watchdog_user_enabled = 0;
-		pr_warning("Disabled lockup detectors by default for full dynticks\n");
-		pr_warning("You can reactivate it with 'sysctl -w kernel.watchdog=1'\n");
-	}
-#endif
-
 	if (watchdog_user_enabled)
 		watchdog_enable_all_cpus();
 }
Signature · 1 line
1.7.5.4
Acked-by on v1 8/8
#41 lore raw

Re: [PATCH 8/8] watchdog: Remove hack to make full dynticks working

Quoted 7 lines · Frederic Weisbecker

A perf event can be used without forcing the tick to stay alive if it doesn't use a frequency but a sample period and if it doesn't throttle (raise storm of events).

Since the lockup detector neither use a perf event frequency nor should ever throttle due to its high period, it can now run concurrently with the full dynticks feature.

Thanks. Dumb question, I keep wondering if the lockup detector would be better or worse off if it used the perf event frequency as opposed to using a sample period? The idea is it could follow the varying cpu frequencies better (and probably simplify some of the code too).

Quoted 37 lines
Cc Jiri Olsa <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Namhyung Kim <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Stephane Eranian <[email protected]>
Cc Don Zickus <[email protected]>
Cc Srivatsa S. Bhat <[email protected]>
Cc Anish Singh <[email protected]>
---
 kernel/watchdog.c |    8 --------
 1 files changed, 0 insertions(+), 8 deletions(-)
kernel/watchdog.c +0 -8
index 1241d8c..51c4f34 100644
@@ -553,14 +553,6 @@ void __init lockup_detector_init(void)
 {
 	set_sample_period();
 
-#ifdef CONFIG_NO_HZ_FULL
-	if (watchdog_user_enabled) {
-		watchdog_user_enabled = 0;
-		pr_warning("Disabled lockup detectors by default for full dynticks\n");
-		pr_warning("You can reactivate it with 'sysctl -w kernel.watchdog=1'\n");
-	}
-#endif
-
 	if (watchdog_user_enabled)
 		watchdog_enable_all_cpus();
 }
Signature · 1 line
1.7.5.4
#42 lore raw

Re: [PATCH 8/8] watchdog: Remove hack to make full dynticks working

Quoted 9 lines · Don Zickus
Quoted 7 lines · Frederic Weisbecker

A perf event can be used without forcing the tick to stay alive if it doesn't use a frequency but a sample period and if it doesn't throttle (raise storm of events).

Since the lockup detector neither use a perf event frequency nor should ever throttle due to its high period, it can now run concurrently with the full dynticks feature.

Quoted 6 lines

Thanks. Dumb question, I keep wondering if the lockup detector would be better or worse off if it used the perf event frequency as opposed to using a sample period? The idea is it could follow the varying cpu frequencies better (and probably simplify some of the code too).

Thanks!

IIRC we tried that and I believe the issue was that perf did not support frequency below 1. So probably the limitation was that it had to fire at least once per sec.

Also if you do that, we fall into the full dynticks problem again.

maintainer
#43 lore raw

Re: [PATCH 8/8] watchdog: Remove hack to make full dynticks working

In reply to Don Zickus
Quoted 5 lines · Don Zickus
Quoted 4 lines · Frederic Weisbecker

A perf event can be used without forcing the tick to stay alive if it doesn't use a frequency but a sample period and if it doesn't throttle (raise storm of events).

Quoted 8 lines
Quoted 3 lines

Since the lockup detector neither use a perf event frequency nor should ever throttle due to its high period, it can now run concurrently with the full dynticks feature.

Thanks. Dumb question, I keep wondering if the lockup detector would be better or worse off if it used the perf event frequency as opposed to using a sample period? The idea is it could follow the varying cpu frequencies better (and probably simplify some of the code too).

Right, trouble is that someone didn't consider fractional frequencies when writing the interface :/ Lowest we can go is 1 Hz while we'd want something like 0.1 Hz or smaller.

Also, like the above says, that would interfere with the nohz efforts as perf needs the tick to re-compute those frequency thingies.

#44 lore raw

[tip:perf/core] watchdog: Make it work under full dynticks

In reply to Frederic Weisbecker
watchdog: Make it work under full dynticks
Commit-ID:  93786a5f6aeb9c032c1c240246c5aabcf457b38f
Gitweb:     http://git.kernel.org/tip/93786a5f6aeb9c032c1c240246c5aabcf457b38f
Author:     Frederic Weisbecker <[email protected]>
AuthorDate: Tue, 23 Jul 2013 02:31:06 +0200
Committer:  Ingo Molnar <[email protected]>
CommitDate: Tue, 30 Jul 2013 22:29:15 +0200

watchdog: Make it work under full dynticks

A perf event can be used without forcing the tick to stay alive if it doesn't use a frequency but a sample period and if it doesn't throttle (raise storm of events).

Since the lockup detector neither use a perf event frequency nor should ever throttle due to its high period, it can now run concurrently with the full dynticks feature.

So remove the hack that disabled the watchdog.

---
 kernel/watchdog.c | 8 --------
 1 file changed, 8 deletions(-)
kernel/watchdog.c +0 -8
index 1241d8c..51c4f34 100644
@@ -553,14 +553,6 @@ void __init lockup_detector_init(void)
 {
 	set_sample_period();
 
-#ifdef CONFIG_NO_HZ_FULL
-	if (watchdog_user_enabled) {
-		watchdog_user_enabled = 0;
-		pr_warning("Disabled lockup detectors by default for full dynticks\n");
-		pr_warning("You can reactivate it with 'sysctl -w kernel.watchdog=1'\n");
-	}
-#endif
-
 	if (watchdog_user_enabled)
 		watchdog_enable_all_cpus();
 }
maintainer Acked-by on the v1 series
#45 lore raw
In reply to Frederic Weisbecker
Quoted 10 lines · Frederic Weisbecker

Hi,

This patchset inspires from Peterz patch in http://marc.info/?l=linux-kernel&m=137155182121860
to make perf retain the timer tick less often, ie: only when there are freq events or when one throttles.

The main purpose is to make the lockup detector work with full dynticks, as I suspect that distros want to enable both...

Quoted 7 lines

You can fetch from:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
perf/nohz

Thanks,
Frederic

Awesome, looks good, thanks Frederic!

#46 lore raw
Quoted 16 lines · Peter Zijlstra
Quoted 15 lines · Frederic Weisbecker

Hi,

This patchset inspires from Peterz patch in http://marc.info/?l=linux-kernel&m=137155182121860
to make perf retain the timer tick less often, ie: only when there are freq events or when one throttles.

The main purpose is to make the lockup detector work with full dynticks, as I suspect that distros want to enable both...

You can fetch from:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
perf/nohz

Quoted 6 lines
Quoted 2 lines

Thanks,
Frederic

Awesome, looks good, thanks Frederic!

Great! What do you prefer? Do you want to take these or should I do a pull request to Ingo directly?

Thanks!

maintainer
#47 lore raw
Quoted 6 lines · Frederic Weisbecker
Quoted 3 lines

Awesome, looks good, thanks Frederic!

Great! What do you prefer? Do you want to take these or should I do a pull request to Ingo directly?

I had intended Ingo to pick these up; but then I remembered he's playing road warrior atm. So I stuck them in my queue so as not to loose them.

#48 lore raw
Quoted 10 lines · Peter Zijlstra
Quoted 6 lines · Frederic Weisbecker
Quoted 3 lines

Awesome, looks good, thanks Frederic!

Great! What do you prefer? Do you want to take these or should I do a pull request to Ingo directly?

I had intended Ingo to pick these up; but then I remembered he's playing road warrior atm. So I stuck them in my queue so as not to loose them.

Thanks!