perf/core: addressing 4x slowdown during per-process profiling of STREAM benchmark on Intel Xeon Phi

Hi,

This patch set v8 addresses feedback captured on review of v7. For more background details please refer to v7 and earlier.

Specifically in this version mux switch is renamed to rotation switch and the value of rotation switch may be the one of defined by ROTATION_DISABLED or ROTATION_ENABLED macros.

ctx->rotate_disable bool flag is renamed to ctx->rotation and also employs ROTATION_DISABLE and ROTATION_ENABLED macros.

pinned_group_sched_in() and flexible_group_sched_in() API are introduced to consolidate code enabling a whole group from pinned and flexible groups appropriately.

Patches in the set are expected to be applied one after another in the mentioned order and they are logically split into two parts to simplify the review process.

The patch set was successfully tested over a night using perf_fuzzer tool.

Thanks,
Alexey

---
Alexey Budankov (2):
perf/core: use rb trees for pinned/flexible groups
perf/core: add rotation switch to skip to the current CPU's events list on mux interrupt

 include/linux/perf_event.h |  21 ++-
 kernel/events/core.c       | 456 ++++++++++++++++++++++++++++++++++-----------
 2 files changed, 362 insertions(+), 115 deletions(-)

This patch moves event groups into rb tree sorted by CPU, so that multiplexing hrtimer interrupt handler would be able skipping to the current CPU's list and ignore groups allocated for the other CPUs.

New API for manipulating event groups in the trees is implemented as well as adoption on the API in the current implementation.

---
 include/linux/perf_event.h |  19 +++-
 kernel/events/core.c       | 220 ++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 202 insertions(+), 37 deletions(-)

index b14095b..cc07904 100644
@@ -572,7 +572,20 @@ struct perf_event {
 	 */
 	struct list_head		group_entry;
 	struct list_head		sibling_list;
-
+	/*
+	 * Node on the pinned or flexible tree located at the event context;
+	 * the node may be empty in case its event is not directly attached
+	 * to the tree but to group_list list of the event directly
+	 * attached to the tree;
+	 */
+	struct rb_node			group_node;
+	/*
+	 * List keeps groups allocated for the same cpu;
+	 * the list may be empty in case its event is not directly
+	 * attached to the tree but to group_list list of the event directly
+	 * attached to the tree;
+	 */
+	struct list_head		group_list;
 	/*
 	 * We need storage to track the entries in perf_pmu_migrate_context; we
 	 * cannot use the event_entry because of RCU and we want to keep the

@@ -741,8 +754,8 @@ struct perf_event_context {
 	struct mutex			mutex;
 
 	struct list_head		active_ctx_list;
-	struct list_head		pinned_groups;
-	struct list_head		flexible_groups;
+	struct rb_root			pinned_groups;
+	struct rb_root			flexible_groups;
 	struct list_head		event_list;
 	int				nr_events;
 	int				nr_active;

This patch implements rotation switch that triggers skipping to the current CPU's events list at mulitplexing hrtimer interrupt handler as well as adoption of the switch in the existing implementation.

The value of rotation switch may be the one of defined by ROTATION_DISABLED or ROTATION_ENABLED macros.

ctx->rotate_disable bool flag is renamed to ctx->rotation and also employs ROTATION_DISABLE and ROTATION_ENABLED macros.

perf_event_groups_iterate_cpu() API is introduced to implement iteration through the certain CPU groups list skipping groups allocated for the other CPUs.

pinned_group_sched_in() and flexible_group_sched_in() API are introduced to consolidate code enabling the whole group from pinned and flexible groups appropriately.

---
 include/linux/perf_event.h |   2 +-
 kernel/events/core.c       | 272 +++++++++++++++++++++++++++++----------------
 2 files changed, 178 insertions(+), 96 deletions(-)

index cc07904..b41ce8f 100644
@@ -762,7 +762,7 @@ struct perf_event_context {
 	int				is_active;
 	int				nr_stat;
 	int				nr_freq;
-	int				rotate_disable;
+	int				rotation;
 	atomic_t			refcount;
 	struct task_struct		*task;