perf: Sharing PMU counters across compatible events
v3 cover perf: Sharing PMU counters across compatible events
Chnages v2 -> v3:
1. Simplified event_pmu_add() as Jiri Olsa suggested
Changes v1 -> v2:
1. Fixed a bug reported by 0day kernel testing robot
This is to follow up earlier discussion on sharing hardware PMU counters across compatible events:
https://marc.info/?t=151213803600016
https://marc.info/?t=152547569200001
This version limits PMU sharing to events within same ctx. As a result, compatible events are not evaluated during task schedule. This vesion also limits PMU sharing to hardware counting events. From our discussion with internal users, this is sufficient for most cases.
This version introduces virtual master event. The virtual master event does require careful handling. But it is makes the logic of event add/del cleaner.
Song Liu (1):
perf: Sharing PMU counters across compatible events
include/linux/perf_event.h | 61 ++++++++ kernel/events/core.c | 281 +++++++++++++++++++++++++++++++++++-- 2 files changed, 332 insertions(+), 10 deletions(-)
--
2.17.1
PATCH v3 1/1 perf: Sharing PMU counters across compatible events
This patch tries to enable PMU sharing. To make perf event scheduling fast, we use special data structures.
An array of "struct perf_event_dup" is added to the perf_event_context, to remember all the duplicated events under this ctx. All the events under this ctx has a "dup_id" pointing to its perf_event_dup. Compatible events under the same ctx share the same perf_event_dup. The following figure shows a simplified version of the data structure.
ctx -> perf_event_dup -> master
^
|
perf_event /|
|
perf_event /Connection among perf_event and perf_event_dup are built when events are added or removed from the ctx. So these are not on the critical path of schedule or perf_rotate_context().
On the critical paths (add, del read), sharing PMU counters doesn't increase the complexity. Helper functions event_pmu_[add|del|read]() are introduced to cover these cases. All these functions have O(1) time complexity.
We allocate a separate perf_event for perf_event_dup->master. This needs extra attention, because perf_event_alloc() may sleep. To allocate the master event properly, a new pointer, tmp_master, is added to perf_event. tmp_master carries a separate perf_event into list_[add|del]_event(). The master event has valid ->ctx and holds ctx->refcount.
Details about the handling of the master event is added to include/linux/perf_event.h, before struct perf_event_dup.
--- include/linux/perf_event.h | 61 ++++++++ kernel/events/core.c | 281 +++++++++++++++++++++++++++++++++++-- 2 files changed, 332 insertions(+), 10 deletions(-)
include/linux/perf_event.h +61 -0
@@ -695,6 +695,12 @@ struct perf_event {#endif struct list_head sb_list; + +/* for PMU sharing */ +struct perf_event *tmp_master; +u64 dup_id; +u64 dup_base_count; +u64 dup_base_child_count;#endif /* CONFIG_PERF_EVENTS */ };
@@ -704,6 +710,58 @@ struct perf_event_groups {u64 index; }; +/* +* Sharing PMU across compatible events +* +* If two perf_events in the same perf_event_context are counting same +* hardware events (instructions, cycles, etc.), they could share the +* hardware PMU counter. +* +* When a perf_event is added to the ctx (list_add_event), it is compared +* against other events in the ctx. If they can share the PMU counter, +* a perf_event_dup is allocated to represent the sharing. +* +* Each perf_event_dup has a virtual master event, which is called by +* pmu->add() and pmu->del(). We cannot call perf_event_alloc() in +* list_add_event(), so it is allocated and carried by event->tmp_master +* into list_add_event(). +* +* Virtual master in different cases/paths: +* +* < I > perf_event_open() -> close() path: +* +* 1. Allocated by perf_event_alloc() in sys_perf_event_open(); +* 2. event->tmp_master->ctx assigned in perf_install_in_context(); +* 3.a. if used by ctx->dup_events, freed in perf_event_release_kernel(); +* 3.b. if not used by ctx->dup_events, freed in perf_event_open(). +* +* < II > inherit_event() path: +* +* 1. Allocated by perf_event_alloc() in inherit_event(); +* 2. tmp_master->ctx assigned in inherit_event(); +* 3.a. if used by ctx->dup_events, freed in perf_event_release_kernel(); +* 3.b. if not used by ctx->dup_events, freed in inherit_event(). +* +* < III > perf_pmu_migrate_context() path: +* all dup_events removed during migration (no sharing after the move). +* +* < IV > perf_event_create_kernel_counter() path: +* not supported yet. +*/ +struct perf_event_dup { +/* +* master event being called by pmu->add() and pmu->del(). +* This event is allocated with perf_event_alloc(). When +* attached to a ctx, this event should hold ctx->refcount. +*/ +struct perf_event *master; +/* number of events in the ctx that shares the master */ +int total_event_count; +/* number of active events of the master */ +int active_event_count; +}; + +#define MAX_PERF_EVENT_DUP_PER_CTX 4/** * struct perf_event_context - event context structure *
@@ -759,6 +817,9 @@ struct perf_event_context {#endif void *task_ctx_data;/* pmu specific data */ struct rcu_head rcu_head; + +/* for PMU sharing. array is needed for O(1) access */ +struct perf_event_dup dup_events[MAX_PERF_EVENT_DUP_PER_CTX ]; };/*
kernel/events/core.c +271 -10
440 lines not on this page · raw message
Signature · 1 line
2.17.1
Hi Song,
Quoted 9 lines · Song Liu
This patch tries to enable PMU sharing. To make perf event scheduling fast, we use special data structures.
An array of "struct
perf_event_dup" is added to theperf_event_context, to remember all the duplicated events under this ctx. All the events under this ctx has a "dup_id" pointing to itsperf_event_dup. Compatible events under the same ctx share the sameperf_event_dup. The following figure shows a simplified version of the data structure.
Quoted 7 lines
ctx -> perf_event_dup -> master
^
|
perf_event /|
|
perf_event /I've not gone through the patch in detail, but I was specifically interested in scenarios where one perf instance is counting event systemwide and thus other perf instance fails to count the same event for a specific workload because that event can be counted in one hw counter only.
Ex: https://lkml.org/lkml/2018/3/12/1011
Seems this patch does not solve this issue. Please let me know if I'm missing anything.
Thanks,
Ravi
Hi Ravi,
Quoted 28 lines · Ravi Bangoria
Hi Song,
Quoted 9 lines · Song Liu
This patch tries to enable PMU sharing. To make perf event scheduling fast, we use special data structures.
An array of "struct
perf_event_dup" is added to theperf_event_context, to remember all the duplicated events under this ctx. All the events under this ctx has a "dup_id" pointing to itsperf_event_dup. Compatible events under the same ctx share the sameperf_event_dup. The following figure shows a simplified version of the data structure.Quoted 7 lines
ctx -> perf_event_dup -> master ^ | perf_event /| | perf_event /I've not gone through the patch in detail, but I was specifically interested in scenarios where one perf instance is counting event systemwide and thus other perf instance fails to count the same event for a specific workload because that event can be counted in one hw counter only.
Quoted 5 lines
Ex: https://lkml.org/lkml/2018/3/12/1011
Seems this patch does not solve this issue. Please let me know if I'm missing anything.
In this case, unfortunately, these two events cannot share the same counter, because one of them is in cpu ctx; while the other belongs to the task ctx. They have to go through the rotation, that each event counts 50% of the time. However, if you have 2 events in cpu ctx and 2 events in task ctx on the same counter, this patch will help each event to count 50% of time, instead of 25%.
Another potential solution is to create a cgroup for the workload, and attach perf event to the cgroup. Since cgroup events are added to the cpu ctx, they can share counters with the system wide events.
I made this trade-off for O(1) time context switch. If we share hw counter between cpu ctx and task ctx, we have to do linear time comparison to identify events that can share the counter.
Thanks,
Song
Quoted 40 lines · Song Liu
Hi Ravi,
Quoted 28 lines · Ravi Bangoria
Hi Song,
Quoted 9 lines · Song Liu
This patch tries to enable PMU sharing. To make perf event scheduling fast, we use special data structures.
An array of "struct
perf_event_dup" is added to theperf_event_context, to remember all the duplicated events under this ctx. All the events under this ctx has a "dup_id" pointing to itsperf_event_dup. Compatible events under the same ctx share the sameperf_event_dup. The following figure shows a simplified version of the data structure.Quoted 7 lines
ctx -> perf_event_dup -> master ^ | perf_event /| | perf_event /I've not gone through the patch in detail, but I was specifically interested in scenarios where one perf instance is counting event systemwide and thus other perf instance fails to count the same event for a specific workload because that event can be counted in one hw counter only.
Quoted 5 lines
Ex: https://lkml.org/lkml/2018/3/12/1011
Seems this patch does not solve this issue. Please let me know if I'm missing anything.
Quoted 6 lines
In this case, unfortunately, these two events cannot share the same counter, because one of them is in cpu ctx; while the other belongs to the task ctx. They have to go through the rotation, that each event counts 50% of the time. However, if you have 2 events in cpu ctx and 2 events in task ctx on the same counter, this patch will help each event to count 50% of time, instead of 25%.
I was wrong here. The CPU flexible events have higher priority than task flexible events, so the per task event will never get scheduled.
On the other hand, I think it is possible to put per task event on per CPU ctx, and handle the event just like per cgroup events. Once we get this part work, system wide events will be able to share PMU with per task events. Let me try prototype it.
Thanks,
Song