perf: Store active software events in a hashlist
PATCH v3 1/1 perf: Store active software events in a hashlist
Each time a software event triggers, we need to walk through the entire list of events from the current cpu and task contexts to retrieve a running perf event that matches.
We also need to check a matching perf event is actually counting.
This walk is wasteful and makes the event fast path scaling down with a growing number of events running on the same contexts.
To solve this, we store the running perf events in a hashlist to get an immediate access to them against their type:event_id when they trigger.
v2: - Fix SWEVENT_HLIST_SIZE definition (and re-learn some basic
maths along the way)
- Only allocate hlist for online cpus, but keep track of the refcount on offline possible cpus too, so that we allocate it if needed when it becomes online.
- Drop the kref use as it's not adapted to our tricks anymore.
v3: - Fix bad refcount check (address instead of value). Thanks to
Eric Dumazet who spotted this.
- While exiting cpu, move the hlist release out of the IPI path to lock the hlist mutex sanely.
--- include/linux/perf_event.h | 12 ++ kernel/perf_event.c | 246 ++++++++++++++++++++++++++++++++----------- 2 files changed, 195 insertions(+), 63 deletions(-)
include/linux/perf_event.h +12 -0
@@ -589,6 +589,14 @@ enum perf_group_flag { PERF_GROUP_SOFTWARE = 0x1, }; +#define SWEVENT_HLIST_BITS 8 +#define SWEVENT_HLIST_SIZE (1 << SWEVENT_HLIST_BITS) + +struct swevent_hlist { +struct hlist_head heads[SWEVENT_HLIST_SIZE ]; +struct rcu_head rcu_head; +}; +/** * struct perf_event - performance event kernel representation: */
@@ -597,6 +605,7 @@ struct perf_event {struct list_head group_entry;struct list_head event_entry;struct list_head sibling_list; +struct hlist_node hlist_entry;int nr_siblings;int group_flags;struct perf_event *group_leader;
@@ -744,6 +753,9 @@ struct perf_cpu_context {int active_oncpu;int max_pertask;int exclusive; +struct swevent_hlist *swevent_hlist; +struct mutex hlist_mutex; +int hlist_refcount;/* * Recursion avoidance:
kernel/perf_event.c +183 -63
347 lines not on this page · raw message
Signature · 1 line
1.6.2.3
Peter Zijlstra
maintainerQuoted 6 lines · Frederic Weisbecker
Each time a software event triggers, we need to walk through the entire list of events from the current cpu and task contexts to retrieve a running perf event that matches.
We also need to check a matching perf event is actually counting.
Quoted 7 lines
This walk is wasteful and makes the event fast path scaling down with a growing number of events running on the same contexts.
To solve this, we store the running perf events in a hashlist to get an immediate access to them against their type:event_id when they trigger.
Looks good, thanks Frederic!