Skip to contentSkip to conversation

perf/core: Small event scheduling changes

RFC PATCH Superseded Frederic Weisbecker 3 participants 12 messages Nov 8, 2009 – Nov 10, 2009 perf events

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover perf/core: Small event scheduling changes

Hi,

This is an rfc patchset, only compile tested just to ensure I'm taking a good direction before going ahead.

This is intended to rework a bit the perf event scheduling to guarantee a real priority of the pinned events over the volatile ones. This patchset handles such priority on task tick time only. But if the idea is agreed, I could expand that to every task event sched-in calls to guarantee the priority in every event rescheduling time.

Thanks.

Frederic Weisbecker (4):
perf/core: split context's event group list into pinned and
non-pinned lists
perf/core: Optimize a bit rotate_ctx()
perf/core: Split up pinned and non pinned processing perf/core: Schedule every pinned events before the the non-pinned

 include/linux/perf_event.h |    3 +-
 kernel/perf_event.c        |  283 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 212 insertions(+), 74 deletions(-)
#1 lore raw

RFC PATCH 1/4 perf/core: split context's event group list into pinned and non-pinned lists

Split-up struct perf_event_context::group_list into pinned_grp_list and volatile_grp_list (non-pinned).

This first appears to be useless as it duplicates various loops around the group list handlings.

But it scales better in the fast-path in perf_sched_in(). We don't anymore iterate twice through the entire list to separate pinned and non-pinned scheduling. Instead we interate through two distinct lists.

The another desired effect is that it makes easier the distinct scheduling rules for both.

Cc Peter Zijlstra <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 include/linux/perf_event.h |    3 +-
 kernel/perf_event.c        |  177 +++++++++++++++++++++++++++++++-------------
 2 files changed, 127 insertions(+), 53 deletions(-)
include/linux/perf_event.h +2 -1
index 6ff7c3b..659351c 100644
@@ -662,7 +662,8 @@ struct perf_event_context {
 	 */
 	struct mutex			mutex;
 
-	struct list_head		group_list;
+	struct list_head		pinned_grp_list;
+	struct list_head		volatile_grp_list;
 	struct list_head		event_list;
 	int				nr_events;
 	int				nr_active;
kernel/perf_event.c +125 -52

285 lines not on this page · raw message

Signature · 1 line
1.6.2.3
#2 lore raw

RFC PATCH 2/4 perf/core: Optimize a bit rotate_ctx()

In reply to Frederic Weisbecker

Don't use list_for_each_entry() just to swap the first and last entry in the list.

Instead, use a direct list->next derefencing. This saves up a useless prefetch.

Cc Peter Zijlstra <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/perf_event.c |   16 +++++++++-------
 1 files changed, 9 insertions(+), 7 deletions(-)
kernel/perf_event.c +9 -7
index b3a31c8..0432c1c 100644
@@ -1432,7 +1432,7 @@ static void perf_ctx_adjust_freq(struct perf_event_context *ctx)
  */
 static void rotate_ctx(struct perf_event_context *ctx)
 {
-	struct perf_event *event;
+	struct list_head *list;
 
 	if (!ctx->nr_events)
 		return;
@@ -1442,15 +1442,17 @@ static void rotate_ctx(struct perf_event_context *ctx)
 	 * Rotate the first entry last (works just fine for group events too):
 	 */
 	perf_disable();
-	list_for_each_entry(event, &ctx->pinned_grp_list, group_entry) {
-		list_move_tail(&event->group_entry, &ctx->pinned_grp_list);
-		break;
+
+	if (!list_empty(&ctx->pinned_grp_list)) {
+		list = ctx->pinned_grp_list.next;
+		list_move_tail(list, &ctx->pinned_grp_list);
 	}
 
-	list_for_each_entry(event, &ctx->volatile_grp_list, group_entry) {
-		list_move_tail(&event->group_entry, &ctx->volatile_grp_list);
-		break;
+	if (!list_empty(&ctx->volatile_grp_list)) {
+		list = ctx->volatile_grp_list.next;
+		list_move_tail(list, &ctx->volatile_grp_list);
 	}
+
 	perf_enable();
 
 	spin_unlock(&ctx->lock);
Signature · 1 line
1.6.2.3
#3 lore raw

RFC PATCH 3/4 perf/core: Split up pinned and non pinned processing

In reply to Frederic Weisbecker

Split up pinned and non-pinned events processing in two helpers so that it's more flexible to handle them seperately.

Cc Peter Zijlstra <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/perf_event.c |   51 +++++++++++++++++++++++++++++++++++----------------
 1 files changed, 35 insertions(+), 16 deletions(-)
kernel/perf_event.c +35 -16
index 0432c1c..50f2997 100644
@@ -1246,25 +1246,11 @@ static void perf_event_cpu_sched_out(struct perf_cpu_context *cpuctx)
 }
 
 static void
-__perf_event_sched_in(struct perf_event_context *ctx,
-			struct perf_cpu_context *cpuctx, int cpu)
+__perf_event_sched_in_pinned(struct perf_event_context *ctx,
+			     struct perf_cpu_context *cpuctx, int cpu)
 {
 	struct perf_event *event;
-	int can_add_hw = 1;
-
-	spin_lock(&ctx->lock);
-	ctx->is_active = 1;
-	if (likely(!ctx->nr_events))
-		goto out;
-
-	ctx->timestamp = perf_clock();
-
-	perf_disable();
 
-	/*
-	 * First go through the list and put on any pinned groups
-	 * in order to give them the best chance of going on.
-	 */
 	list_for_each_entry(event, &ctx->pinned_grp_list, group_entry) {
 		if (event->state <= PERF_EVENT_STATE_OFF)
 			continue;
@@ -1283,6 +1269,14 @@ __perf_event_sched_in(struct perf_event_context *ctx,
 			event->state = PERF_EVENT_STATE_ERROR;
 		}
 	}
+}
+
+static void
+__perf_event_sched_in_volatile(struct perf_event_context *ctx,
+			       struct perf_cpu_context *cpuctx, int cpu)
+{
+	int can_add_hw = 1;
+	struct perf_event *event;
 
 	list_for_each_entry(event, &ctx->volatile_grp_list, group_entry) {
 		/*
@@ -1303,6 +1297,31 @@ __perf_event_sched_in(struct perf_event_context *ctx,
 			if (group_sched_in(event, cpuctx, ctx, cpu))
 				can_add_hw = 0;
 	}
+}
+
+static void
+__perf_event_sched_in(struct perf_event_context *ctx,
+			struct perf_cpu_context *cpuctx, int cpu)
+{
+	spin_lock(&ctx->lock);
+	ctx->is_active = 1;
+	if (likely(!ctx->nr_events))
+		goto out;
+
+	ctx->timestamp = perf_clock();
+
+	perf_disable();
+
+	/*
+	 * First go through the list and put on any pinned groups
+	 * in order to give them the best chance of going on.
+	 */
+	__perf_event_sched_in_pinned(ctx, cpuctx, cpu);
+
+	/* Then handle the non-pinned groups */
+	__perf_event_sched_in_volatile(ctx, cpuctx, cpu);
+
+
 	perf_enable();
  out:
 	spin_unlock(&ctx->lock);
Signature · 1 line
1.6.2.3
#4 lore raw

Re: [RFC PATCH 3/4] perf/core: Split up pinned and non pinned processing

kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1276

Quoted 6 lines · Frederic Weisbecker

Split up pinned and non-pinned events processing in two helpers so that it's more flexible to handle them seperately.

kernel/perf_event.c +3 -0
+static void
+__perf_event_sched_in_volatile(struct perf_event_context *ctx,
+			       struct perf_cpu_context *cpuctx, int cpu)

Small naming suggestion: 'volatile' is a C keyword and rarely used outside of that context in the kernel, which makes this function name a bit confusing.

So instead of pinned/volatile, a pinned/flexible naming would be more readable, i.e. __perf_event_sched_in_flexible() or so.

Also, most of the static functions in kernel/perf_event.c could lose their perf_event_ prefix - we already know it's a perf thing, right? That will shorten quite a few function names there.

These functions would turn into __sched_in_pinned()/__sched_in_flexible().

Agreed?

Ingo

#5 lore raw

Re: [RFC PATCH 3/4] perf/core: Split up pinned and non pinned processing

Quoted 14 lines · Ingo Molnar

kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1276

Quoted 6 lines · Frederic Weisbecker

Split up pinned and non-pinned events processing in two helpers so that it's more flexible to handle them seperately.

kernel/perf_event.c +3 -0
+static void
+__perf_event_sched_in_volatile(struct perf_event_context *ctx,
+			       struct perf_cpu_context *cpuctx, int cpu)

Small naming suggestion: 'volatile' is a C keyword and rarely used outside of that context in the kernel, which makes this function name a bit confusing.

Quoted 2 lines

So instead of pinned/volatile, a pinned/flexible naming would be more readable, i.e. __perf_event_sched_in_flexible() or so.

Right, also that makes it consistent with the hw-breakpoint constraints naming.

Quoted 7 lines

Also, most of the static functions in kernel/perf_event.c could lose their perf_event_ prefix - we already know it's a perf thing, right? That will shorten quite a few function names there.

These functions would turn into __sched_in_pinned()/__sched_in_flexible().

Agreed?

Totally.

I'll prepare a new iteration, thanks!

#6 lore raw

RFC PATCH 4/4 perf/core: Schedule every pinned events before the the non-pinned

In reply to Frederic Weisbecker

Currently, the order to schedule events is as follows:

- cpu context pinned events
- cpu context non-pinned events
- task context pinned events
- task context non-pinned events

What we want instead is to schedule every pinned events first because those have a higher priority.

This is what does this patch in each task tick. If the approach is agreed, we may want to expand this to task-only sched in (where the cpu events are not sched out), fork, exec, etc... So that we guarantee the pinned priority every time the task is scheduled in.

Cc Peter Zijlstra <[email protected]>
Cc Arnaldo Carvalho de Melo <[email protected]>
Cc Mike Galbraith <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Thomas Gleixner <[email protected]>
---
 kernel/perf_event.c |   49 ++++++++++++++++++++++++++++++++++++++++++++++---
 1 files changed, 46 insertions(+), 3 deletions(-)
kernel/perf_event.c +46 -3
index 50f2997..f32aec4 100644
@@ -1327,6 +1327,41 @@ __perf_event_sched_in(struct perf_event_context *ctx,
 	spin_unlock(&ctx->lock);
 }
 
+static void
+__perf_event_sched_in_all(struct perf_event_context *ctx,
+			  struct perf_cpu_context *cpuctx, int cpu)
+{
+	struct perf_event_context *cpu_ctx = &cpuctx->ctx;
+
+	/* May require different classes between cpu and task lock */
+	spin_lock(&cpu_ctx->lock);
+	spin_lock(&ctx->lock);
+	cpu_ctx->is_active = ctx->is_active = 1;
+
+	ctx->timestamp = cpu_ctx->timestamp = perf_clock();
+
+	perf_disable();
+
+	if (cpu_ctx->nr_events)
+		__perf_event_sched_in_pinned(cpu_ctx, cpuctx, cpu);
+
+	if (ctx->nr_events)
+		__perf_event_sched_in_pinned(cpu_ctx, cpuctx, cpu);
+
+	if (cpu_ctx->nr_events)
+		__perf_event_sched_in_volatile(cpu_ctx, cpuctx, cpu);
+
+	if (ctx->nr_events)
+		__perf_event_sched_in_volatile(cpu_ctx, cpuctx, cpu);
+
+	cpuctx->task_ctx = ctx;
+
+	perf_enable();
+
+	spin_unlock(&ctx->lock);
+	spin_lock(&cpu_ctx->lock);
+}
+
 /*
  * Called from scheduler to add the events of the current task
  * with interrupts disabled.
@@ -1477,6 +1512,16 @@ static void rotate_ctx(struct perf_event_context *ctx)
 	spin_unlock(&ctx->lock);
 }
 
+static void
+perf_event_sched_in_all(struct perf_event_context *ctx,
+			struct perf_cpu_context *cpuctx, int cpu)
+{
+	if (!ctx || ctx == cpuctx->task_ctx)
+		perf_event_cpu_sched_in(cpuctx, cpu);
+	else
+		__perf_event_sched_in_all(ctx, cpuctx, cpu);
+}
+
 void perf_event_task_tick(struct task_struct *curr, int cpu)
 {
 	struct perf_cpu_context *cpuctx;
@@ -1500,9 +1545,7 @@ void perf_event_task_tick(struct task_struct *curr, int cpu)
 	if (ctx)
 		rotate_ctx(ctx);
 
-	perf_event_cpu_sched_in(cpuctx, cpu);
-	if (ctx)
-		perf_event_task_sched_in(curr, cpu);
+	perf_event_sched_in_all(ctx, cpuctx, cpu);
 }
 
 static void __perf_event_enable_on_exec(struct perf_event *event,
Signature · 1 line
1.6.2.3
maintainer
#7 lore raw

Re: [RFC PATCH 4/4] perf/core: Schedule every pinned events before the the non-pinned

kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1338

Quoted 9 lines · Frederic Weisbecker
kernel/perf_event.c +9 -0
+static void
+__perf_event_sched_in_all(struct perf_event_context *ctx,
+			  struct perf_cpu_context *cpuctx, int cpu)
+{
+	struct perf_event_context *cpu_ctx = &cpuctx->ctx;
+
+	/* May require different classes between cpu and task lock */
+	spin_lock(&cpu_ctx->lock);
+	spin_lock(&ctx->lock);

Would be good to know for sure, running with lockdep enabled ought to tell you that pretty quick ;-)

Quoted 14 lines
kernel/perf_event.c +14 -0
+	cpu_ctx->is_active = ctx->is_active = 1;
+
+	ctx->timestamp = cpu_ctx->timestamp = perf_clock();
+
+	perf_disable();
+
+	if (cpu_ctx->nr_events)
+		__perf_event_sched_in_pinned(cpu_ctx, cpuctx, cpu);
+
+	if (ctx->nr_events)
+		__perf_event_sched_in_pinned(cpu_ctx, cpuctx, cpu);
+
+	if (cpu_ctx->nr_events)
+		__perf_event_sched_in_volatile(cpu_ctx, cpuctx, cpu);

kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1362

Quoted 10 lines
kernel/perf_event.c +10 -0
+
+	if (ctx->nr_events)
+		__perf_event_sched_in_volatile(cpu_ctx, cpuctx, cpu);
+
+	cpuctx->task_ctx = ctx;
+
+	perf_enable();
+
+	spin_unlock(&ctx->lock);
+	spin_lock(&cpu_ctx->lock);

I'm pretty sure that ought to be spin_unlock() ;-)

Quoted 1 line

+}

Like Ingo I don't really like the volatile name.

Can't we simply have 2 lists per cpu a pinned and normal list, and first schedule all the pinned and RR the normal events?

I guess one could implement that by adding the task context events to the cpu context events on sched_in and removing them on sched_out. That would clear up a lot of funny scheduling details.

#8 lore raw

Re: [RFC PATCH 4/4] perf/core: Schedule every pinned events before the the non-pinned

Quoted 14 lines · Peter Zijlstra

kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1338

Quoted 9 lines · Frederic Weisbecker
kernel/perf_event.c +9 -0
+static void
+__perf_event_sched_in_all(struct perf_event_context *ctx,
+			  struct perf_cpu_context *cpuctx, int cpu)
+{
+	struct perf_event_context *cpu_ctx = &cpuctx->ctx;
+
+	/* May require different classes between cpu and task lock */
+	spin_lock(&cpu_ctx->lock);
+	spin_lock(&ctx->lock);

Would be good to know for sure, running with lockdep enabled ought to tell you that pretty quick ;-)

That's about sure I guess :)
I just wanted to take care of that after your comments.

Quoted 18 lines
Quoted 18 lines
kernel/perf_event.c +18 -0
+	cpu_ctx->is_active = ctx->is_active = 1;
+
+	ctx->timestamp = cpu_ctx->timestamp = perf_clock();
+
+	perf_disable();
+
+	if (cpu_ctx->nr_events)
+		__perf_event_sched_in_pinned(cpu_ctx, cpuctx, cpu);
+
+	if (ctx->nr_events)
+		__perf_event_sched_in_pinned(cpu_ctx, cpuctx, cpu);
+
+	if (cpu_ctx->nr_events)
+		__perf_event_sched_in_volatile(cpu_ctx, cpuctx, cpu);
+
+	if (ctx->nr_events)
+		__perf_event_sched_in_volatile(cpu_ctx, cpuctx, cpu);
+
Quoted 8 lines

kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1362

Quoted 6 lines
kernel/perf_event.c +6 -0
+	cpuctx->task_ctx = ctx;
+
+	perf_enable();
+
+	spin_unlock(&ctx->lock);
+	spin_lock(&cpu_ctx->lock);

I'm pretty sure that ought to be spin_unlock() ;-)

Indeed :)

Quoted 11 lines
Quoted 1 line

+}

Like Ingo I don't really like the volatile name.

Can't we simply have 2 lists per cpu a pinned and normal list, and first schedule all the pinned and RR the normal events?

I guess one could implement that by adding the task context events to the cpu context events on sched_in and removing them on sched_out. That would clear up a lot of funny scheduling details.

I thought about doing that, but didn't expand the idea that much, because of the list manipulation that induces.

But you're right, that would be be indeed more proper. I can just save the "real" cpu event group tail in the struct perf_cpu_context so that I can keep track of the real state and (un)glue the queues easily.

Yeah, I'll try that, thanks!

#9 lore raw
In reply to Frederic Weisbecker
Quoted 20 lines · Frederic Weisbecker

Hi,

This is an rfc patchset, only compile tested just to ensure I'm taking a good direction before going ahead.

This is intended to rework a bit the perf event scheduling to guarantee a real priority of the pinned events over the volatile ones. This patchset handles such priority on task tick time only. But if the idea is agreed, I could expand that to every task event sched-in calls to guarantee the priority in every event rescheduling time.

Thanks.

Frederic Weisbecker (4):
perf/core: split context's event group list into pinned and
non-pinned lists
perf/core: Optimize a bit rotate_ctx()
perf/core: Split up pinned and non pinned processing perf/core: Schedule every pinned events before the the non-pinned

Quoted 3 lines
 include/linux/perf_event.h |    3 +-
 kernel/perf_event.c        |  283 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 212 insertions(+), 74 deletions(-)

Sans the small naming suggestions i had, the general principle looks good to me - it's a nice restructuring of the various scheduling rules we have for events.

Ingo

#10 lore raw
Quoted 27 lines · Ingo Molnar
Quoted 20 lines · Frederic Weisbecker

Hi,

This is an rfc patchset, only compile tested just to ensure I'm taking a good direction before going ahead.

This is intended to rework a bit the perf event scheduling to guarantee a real priority of the pinned events over the volatile ones. This patchset handles such priority on task tick time only. But if the idea is agreed, I could expand that to every task event sched-in calls to guarantee the priority in every event rescheduling time.

Thanks.

Frederic Weisbecker (4):
perf/core: split context's event group list into pinned and
non-pinned lists
perf/core: Optimize a bit rotate_ctx()
perf/core: Split up pinned and non pinned processing perf/core: Schedule every pinned events before the the non-pinned

Quoted 3 lines
 include/linux/perf_event.h |    3 +-
 kernel/perf_event.c        |  283 ++++++++++++++++++++++++++++++++-----------
 2 files changed, 212 insertions(+), 74 deletions(-)
Quoted 5 lines

Sans the small naming suggestions i had, the general principle looks good to me - it's a nice restructuring of the various scheduling rules we have for events.

Ingo

Thanks.

With this draft, it makes the pinned priority more consistent with its purpose but it doesn't yet bring the full pinned over flexible priority determinism.

It does apply the priority in tick time, while we round robin. I did that there first so that it covers most of the events rescheduling actions and also it doesn't bring much more overhead over the previous layout (in theory), it just changes the order.

I'll also try to expand the priority constraint each time we sched in a task: when we schedule a new task that belongs to a new context, we don't schedule out/in the cpu context but that will be needed if we want the full priority determinism.

Anyway, I'll do that progressively.

Frederic.

#11 lore raw
Quoted 12 lines · Frederic Weisbecker

Thanks.

With this draft, it makes the pinned priority more consistent with its purpose but it doesn't yet bring the full pinned over flexible priority determinism.

It does apply the priority in tick time, while we round robin. I did that there first so that it covers most of the events rescheduling actions and also it doesn't bring much more overhead over the previous layout (in theory), it just changes the order.

Quoted 4 lines

I'll also try to expand the priority constraint each time we sched in a task: when we schedule a new task that belongs to a new context, we don't schedule out/in the cpu context but that will be needed if we want the full priority determinism.

To lower the overhead at non-tick time, we could even just reschedule the cpu flexible events. Anyway...

Quoted 5 lines

Anyway, I'll do that progressively.

Frederic.