perf/core: Small event scheduling changes
cover perf/core: Small event scheduling changes
Hi,
This is an rfc patchset, only compile tested just to ensure I'm taking a good direction before going ahead.
This is intended to rework a bit the perf event scheduling to guarantee a real priority of the pinned events over the volatile ones. This patchset handles such priority on task tick time only. But if the idea is agreed, I could expand that to every task event sched-in calls to guarantee the priority in every event rescheduling time.
Thanks.
Frederic Weisbecker (4):
perf/core: split context's event group list into pinned and
non-pinned lists
perf/core: Optimize a bit rotate_ctx()
perf/core: Split up pinned and non pinned processing perf/core: Schedule every pinned events before the the non-pinned
include/linux/perf_event.h | 3 +- kernel/perf_event.c | 283 ++++++++++++++++++++++++++++++++----------- 2 files changed, 212 insertions(+), 74 deletions(-)
RFC PATCH 1/4 perf/core: split context's event group list into pinned and non-pinned lists
Split-up struct perf_event_context::group_list into pinned_grp_list and volatile_grp_list (non-pinned).
This first appears to be useless as it duplicates various loops around the group list handlings.
But it scales better in the fast-path in perf_sched_in(). We don't anymore iterate twice through the entire list to separate pinned and non-pinned scheduling. Instead we interate through two distinct lists.
The another desired effect is that it makes easier the distinct scheduling rules for both.
--- include/linux/perf_event.h | 3 +- kernel/perf_event.c | 177 +++++++++++++++++++++++++++++++------------- 2 files changed, 127 insertions(+), 53 deletions(-)
include/linux/perf_event.h +2 -1
@@ -662,7 +662,8 @@ struct perf_event_context {*/ struct mutex mutex; -struct list_head group_list; +struct list_head pinned_grp_list; +struct list_head volatile_grp_list;struct list_head event_list;int nr_events;int nr_active;
kernel/perf_event.c +125 -52
285 lines not on this page · raw message
Signature · 1 line
1.6.2.3
RFC PATCH 2/4 perf/core: Optimize a bit rotate_ctx()
In reply to Frederic WeisbeckerDon't use list_for_each_entry() just to swap the first and last entry in the list.
Instead, use a direct list->next derefencing. This saves up a useless prefetch.
--- kernel/perf_event.c | 16 +++++++++------- 1 files changed, 9 insertions(+), 7 deletions(-)
kernel/perf_event.c +9 -7
@@ -1432,7 +1432,7 @@ static void perf_ctx_adjust_freq(struct perf_event_context *ctx)*/ static void rotate_ctx (struct perf_event_context *ctx) { -struct perf_event *event; +struct list_head *list;if (!ctx->nr_events)return ;
@@ -1442,15 +1442,17 @@ static void rotate_ctx(struct perf_event_context *ctx)* Rotate the first entry last (works just fine for group events too): */ perf_disable (); -list_for_each_entry (event, &ctx->pinned_grp_list, group_entry) { -list_move_tail (&event->group_entry, &ctx->pinned_grp_list); -break ; + +if (!list_empty (&ctx->pinned_grp_list)) { + list = ctx->pinned_grp_list.next; +list_move_tail (list, &ctx->pinned_grp_list); } -list_for_each_entry (event, &ctx->volatile_grp_list, group_entry) { -list_move_tail (&event->group_entry, &ctx->volatile_grp_list); -break ; +if (!list_empty (&ctx->volatile_grp_list)) { + list = ctx->volatile_grp_list.next; +list_move_tail (list, &ctx->volatile_grp_list); } +perf_enable ();spin_unlock (&ctx->lock);
Signature · 1 line
1.6.2.3
RFC PATCH 3/4 perf/core: Split up pinned and non pinned processing
In reply to Frederic WeisbeckerSplit up pinned and non-pinned events processing in two helpers so that it's more flexible to handle them seperately.
--- kernel/perf_event.c | 51 +++++++++++++++++++++++++++++++++++---------------- 1 files changed, 35 insertions(+), 16 deletions(-)
kernel/perf_event.c +35 -16
@@ -1246,25 +1246,11 @@ static void perf_event_cpu_sched_out(struct perf_cpu_context *cpuctx) }static void -__perf_event_sched_in (struct perf_event_context *ctx, -struct perf_cpu_context *cpuctx,int cpu) +__perf_event_sched_in_pinned (struct perf_event_context *ctx, +struct perf_cpu_context *cpuctx,int cpu) {struct perf_event *event; -int can_add_hw =1 ; - -spin_lock (&ctx->lock); - ctx->is_active =1 ; -if (likely (!ctx->nr_events)) - goto out; - - ctx->timestamp =perf_clock (); - -perf_disable (); -/* -* First go through the list and put on any pinned groups -* in order to give them the best chance of going on. -*/ list_for_each_entry (event, &ctx->pinned_grp_list, group_entry) {if (event->state <=PERF_EVENT_STATE_OFF )continue ;
@@ -1283,6 +1269,14 @@ __perf_event_sched_in(struct perf_event_context *ctx, event->state =PERF_EVENT_STATE_ERROR ; } } +} + +static void +__perf_event_sched_in_volatile (struct perf_event_context *ctx, +struct perf_cpu_context *cpuctx,int cpu) +{ +int can_add_hw =1 ; +struct perf_event *event;list_for_each_entry (event, &ctx->volatile_grp_list, group_entry) {/*
@@ -1303,6 +1297,31 @@ __perf_event_sched_in(struct perf_event_context *ctx,if (group_sched_in (event, cpuctx, ctx, cpu)) can_add_hw =0 ; } +} + +static void +__perf_event_sched_in (struct perf_event_context *ctx, +struct perf_cpu_context *cpuctx,int cpu) +{ +spin_lock (&ctx->lock); + ctx->is_active =1 ; +if (likely (!ctx->nr_events)) + goto out; + + ctx->timestamp =perf_clock (); + +perf_disable (); + +/* +* First go through the list and put on any pinned groups +* in order to give them the best chance of going on. +*/ +__perf_event_sched_in_pinned (ctx, cpuctx, cpu); + +/* Then handle the non-pinned groups */ +__perf_event_sched_in_volatile (ctx, cpuctx, cpu); + +perf_enable (); out:spin_unlock (&ctx->lock);
Signature · 1 line
1.6.2.3
Re: [RFC PATCH 3/4] perf/core: Split up pinned and non pinned processing
kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1276
Quoted 6 lines · Frederic Weisbecker
Split up pinned and non-pinned events processing in two helpers so that it's more flexible to handle them seperately.
kernel/perf_event.c +3
-0+static void +__perf_event_sched_in_volatile (struct perf_event_context *ctx, +struct perf_cpu_context *cpuctx,int cpu)
Small naming suggestion: 'volatile' is a C keyword and rarely used outside of that context in the kernel, which makes this function name a bit confusing.
So instead of pinned/volatile, a pinned/flexible naming would be more readable, i.e. __perf_event_sched_in_flexible() or so.
Also, most of the static functions in kernel/perf_event.c could lose their perf_event_ prefix - we already know it's a perf thing, right? That will shorten quite a few function names there.
These functions would turn into __sched_in_pinned()/__sched_in_flexible().
Agreed?
Ingo
Re: [RFC PATCH 3/4] perf/core: Split up pinned and non pinned processing
Quoted 14 lines · Ingo Molnar
kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1276
Quoted 6 lines · Frederic Weisbecker
Split up pinned and non-pinned events processing in two helpers so that it's more flexible to handle them seperately.
kernel/perf_event.c +3
-0+static void +__perf_event_sched_in_volatile (struct perf_event_context *ctx, +struct perf_cpu_context *cpuctx,int cpu)Small naming suggestion: 'volatile' is a C keyword and rarely used outside of that context in the kernel, which makes this function name a bit confusing.
Quoted 2 lines
So instead of pinned/volatile, a pinned/flexible naming would be more readable, i.e.
__perf_event_sched_in_flexible()or so.
Right, also that makes it consistent with the hw-breakpoint constraints naming.
Quoted 7 lines
Also, most of the static functions in
kernel/perf_event.ccould lose theirperf_event_prefix - we already know it's a perf thing, right? That will shorten quite a few function names there.These functions would turn into __sched_in_pinned()/__sched_in_flexible().
Agreed?
Totally.
I'll prepare a new iteration, thanks!
RFC PATCH 4/4 perf/core: Schedule every pinned events before the the non-pinned
In reply to Frederic WeisbeckerCurrently, the order to schedule events is as follows:
- cpu context pinned events
- cpu context non-pinned events
- task context pinned events
- task context non-pinned events
What we want instead is to schedule every pinned events first because those have a higher priority.
This is what does this patch in each task tick. If the approach is agreed, we may want to expand this to task-only sched in (where the cpu events are not sched out), fork, exec, etc... So that we guarantee the pinned priority every time the task is scheduled in.
--- kernel/perf_event.c | 49 ++++++++++++++++++++++++++++++++++++++++++++++--- 1 files changed, 46 insertions(+), 3 deletions(-)
kernel/perf_event.c +46 -3
@@ -1327,6 +1327,41 @@ __perf_event_sched_in(struct perf_event_context *ctx,spin_unlock (&ctx->lock); } +static void +__perf_event_sched_in_all (struct perf_event_context *ctx, +struct perf_cpu_context *cpuctx,int cpu) +{ +struct perf_event_context *cpu_ctx = &cpuctx->ctx; + +/* May require different classes between cpu and task lock */ +spin_lock (&cpu_ctx->lock); +spin_lock (&ctx->lock); + cpu_ctx->is_active = ctx->is_active =1 ; + + ctx->timestamp = cpu_ctx->timestamp =perf_clock (); + +perf_disable (); + +if (cpu_ctx->nr_events) +__perf_event_sched_in_pinned (cpu_ctx, cpuctx, cpu); + +if (ctx->nr_events) +__perf_event_sched_in_pinned (cpu_ctx, cpuctx, cpu); + +if (cpu_ctx->nr_events) +__perf_event_sched_in_volatile (cpu_ctx, cpuctx, cpu); + +if (ctx->nr_events) +__perf_event_sched_in_volatile (cpu_ctx, cpuctx, cpu); + + cpuctx->task_ctx = ctx; + +perf_enable (); + +spin_unlock (&ctx->lock); +spin_lock (&cpu_ctx->lock); +} +/* * Called from scheduler to add the events of the current task * with interrupts disabled.
@@ -1477,6 +1512,16 @@ static void rotate_ctx(struct perf_event_context *ctx)spin_unlock (&ctx->lock); } +static void +perf_event_sched_in_all (struct perf_event_context *ctx, +struct perf_cpu_context *cpuctx,int cpu) +{ +if (!ctx || ctx == cpuctx->task_ctx) +perf_event_cpu_sched_in (cpuctx, cpu); +else +__perf_event_sched_in_all (ctx, cpuctx, cpu); +} +void perf_event_task_tick (struct task_struct *curr,int cpu) {struct perf_cpu_context *cpuctx;
@@ -1500,9 +1545,7 @@ void perf_event_task_tick(struct task_struct *curr, int cpu)if (ctx)rotate_ctx (ctx); -perf_event_cpu_sched_in (cpuctx, cpu); -if (ctx) -perf_event_task_sched_in (curr, cpu); +perf_event_sched_in_all (ctx, cpuctx, cpu); } static void __perf_event_enable_on_exec(struct perf_event *event,
Signature · 1 line
1.6.2.3
Peter Zijlstra
maintainerRe: [RFC PATCH 4/4] perf/core: Schedule every pinned events before the the non-pinned
kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1338
Quoted 9 lines · Frederic Weisbecker
kernel/perf_event.c +9
-0+static void +__perf_event_sched_in_all (struct perf_event_context *ctx, +struct perf_cpu_context *cpuctx,int cpu) +{ +struct perf_event_context *cpu_ctx = &cpuctx->ctx; + +/* May require different classes between cpu and task lock */ +spin_lock (&cpu_ctx->lock); +spin_lock (&ctx->lock);
Would be good to know for sure, running with lockdep enabled ought to tell you that pretty quick ;-)
Quoted 14 lines
kernel/perf_event.c +14
-0+ cpu_ctx->is_active = ctx->is_active =1 ; + + ctx->timestamp = cpu_ctx->timestamp =perf_clock (); + +perf_disable (); + +if (cpu_ctx->nr_events) +__perf_event_sched_in_pinned (cpu_ctx, cpuctx, cpu); + +if (ctx->nr_events) +__perf_event_sched_in_pinned (cpu_ctx, cpuctx, cpu); + +if (cpu_ctx->nr_events) +__perf_event_sched_in_volatile (cpu_ctx, cpuctx, cpu);
kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1362
Quoted 10 lines
kernel/perf_event.c +10
-0+ +if (ctx->nr_events) +__perf_event_sched_in_volatile (cpu_ctx, cpuctx, cpu); + + cpuctx->task_ctx = ctx; + +perf_enable (); + +spin_unlock (&ctx->lock); +spin_lock (&cpu_ctx->lock);
I'm pretty sure that ought to be spin_unlock() ;-)
Quoted 1 line
+}
Like Ingo I don't really like the volatile name.
Can't we simply have 2 lists per cpu a pinned and normal list, and first schedule all the pinned and RR the normal events?
I guess one could implement that by adding the task context events to the cpu context events on sched_in and removing them on sched_out. That would clear up a lot of funny scheduling details.
Re: [RFC PATCH 4/4] perf/core: Schedule every pinned events before the the non-pinned
Quoted 14 lines · Peter Zijlstra
kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1338
Quoted 9 lines · Frederic Weisbecker
kernel/perf_event.c +9
-0+static void +__perf_event_sched_in_all (struct perf_event_context *ctx, +struct perf_cpu_context *cpuctx,int cpu) +{ +struct perf_event_context *cpu_ctx = &cpuctx->ctx; + +/* May require different classes between cpu and task lock */ +spin_lock (&cpu_ctx->lock); +spin_lock (&ctx->lock);Would be good to know for sure, running with lockdep enabled ought to tell you that pretty quick ;-)
That's about sure I guess :)
I just wanted to take care of that after your comments.
Quoted 18 lines
Quoted 18 lines
kernel/perf_event.c +18
-0+ cpu_ctx->is_active = ctx->is_active =1 ; + + ctx->timestamp = cpu_ctx->timestamp =perf_clock (); + +perf_disable (); + +if (cpu_ctx->nr_events) +__perf_event_sched_in_pinned (cpu_ctx, cpuctx, cpu); + +if (ctx->nr_events) +__perf_event_sched_in_pinned (cpu_ctx, cpuctx, cpu); + +if (cpu_ctx->nr_events) +__perf_event_sched_in_volatile (cpu_ctx, cpuctx, cpu); + +if (ctx->nr_events) +__perf_event_sched_in_volatile (cpu_ctx, cpuctx, cpu); +
Quoted 8 lines
kernel/perf_event.c __perf_event_sched_in(struct perf_event_context *ctx, · line 1362
Quoted 6 lines
kernel/perf_event.c +6
-0+ cpuctx->task_ctx = ctx; + +perf_enable (); + +spin_unlock (&ctx->lock); +spin_lock (&cpu_ctx->lock);I'm pretty sure that ought to be
spin_unlock();-)
Indeed :)
Quoted 11 lines
Quoted 1 line
+}
Like Ingo I don't really like the volatile name.
Can't we simply have 2 lists per cpu a pinned and normal list, and first schedule all the pinned and RR the normal events?
I guess one could implement that by adding the task context events to the cpu context events on
sched_inand removing them onsched_out. That would clear up a lot of funny scheduling details.
I thought about doing that, but didn't expand the idea that much, because of the list manipulation that induces.
But you're right, that would be be indeed more proper. I can just save the "real" cpu event group tail in the struct perf_cpu_context so that I can keep track of the real state and (un)glue the queues easily.
Yeah, I'll try that, thanks!
Quoted 20 lines · Frederic Weisbecker
Hi,
This is an rfc patchset, only compile tested just to ensure I'm taking a good direction before going ahead.
This is intended to rework a bit the perf event scheduling to guarantee a real priority of the pinned events over the volatile ones. This patchset handles such priority on task tick time only. But if the idea is agreed, I could expand that to every task event sched-in calls to guarantee the priority in every event rescheduling time.
Thanks.
Frederic Weisbecker (4):
perf/core: split context's event group list into pinned and
non-pinned lists
perf/core: Optimize a bitrotate_ctx()
perf/core: Split up pinned and non pinned processing perf/core: Schedule every pinned events before the the non-pinned
Quoted 3 lines
include/linux/perf_event.h | 3 +- kernel/perf_event.c | 283 ++++++++++++++++++++++++++++++++----------- 2 files changed, 212 insertions(+), 74 deletions(-)
Sans the small naming suggestions i had, the general principle looks good to me - it's a nice restructuring of the various scheduling rules we have for events.
Ingo
Quoted 27 lines · Ingo Molnar
Quoted 20 lines · Frederic Weisbecker
Hi,
This is an rfc patchset, only compile tested just to ensure I'm taking a good direction before going ahead.
This is intended to rework a bit the perf event scheduling to guarantee a real priority of the pinned events over the volatile ones. This patchset handles such priority on task tick time only. But if the idea is agreed, I could expand that to every task event sched-in calls to guarantee the priority in every event rescheduling time.
Thanks.
Frederic Weisbecker (4):
perf/core: split context's event group list into pinned and
non-pinned lists
perf/core: Optimize a bitrotate_ctx()
perf/core: Split up pinned and non pinned processing perf/core: Schedule every pinned events before the the non-pinnedQuoted 3 lines
include/linux/perf_event.h | 3 +- kernel/perf_event.c | 283 ++++++++++++++++++++++++++++++++----------- 2 files changed, 212 insertions(+), 74 deletions(-)
Quoted 5 lines
Sans the small naming suggestions i had, the general principle looks good to me - it's a nice restructuring of the various scheduling rules we have for events.
Ingo
Thanks.
With this draft, it makes the pinned priority more consistent with its purpose but it doesn't yet bring the full pinned over flexible priority determinism.
It does apply the priority in tick time, while we round robin. I did that there first so that it covers most of the events rescheduling actions and also it doesn't bring much more overhead over the previous layout (in theory), it just changes the order.
I'll also try to expand the priority constraint each time we sched in a task: when we schedule a new task that belongs to a new context, we don't schedule out/in the cpu context but that will be needed if we want the full priority determinism.
Anyway, I'll do that progressively.
Frederic.
Quoted 12 lines · Frederic Weisbecker
Thanks.
With this draft, it makes the pinned priority more consistent with its purpose but it doesn't yet bring the full pinned over flexible priority determinism.
It does apply the priority in tick time, while we round robin. I did that there first so that it covers most of the events rescheduling actions and also it doesn't bring much more overhead over the previous layout (in theory), it just changes the order.
Quoted 4 lines
I'll also try to expand the priority constraint each time we sched in a task: when we schedule a new task that belongs to a new context, we don't schedule out/in the cpu context but that will be needed if we want the full priority determinism.
To lower the overhead at non-tick time, we could even just reschedule the cpu flexible events. Anyway...
Quoted 5 lines
Anyway, I'll do that progressively.
Frederic.