Skip to contentSkip to conversation

Optimize cgroup context switch

PATCH New [email protected] 1 participant 5 messages May 15, 2019 perf events

lore
Share & tools
lore
#0 lore raw

v2 cover Optimize cgroup context switch

From: Kan Liang <[email protected]>

Changes since V1:
- Add new event_type to indicate cgroup only switch
Add cgrp_event_type to track event type of a cgroup
Extend ctx_pinned/flexible_sched_in and struct sched_in_data to pass the event_type
- If the new cgroup has pinned events, schedule out all flexible events before sched in all events.
- Add macro and helper function to replace duplicated content in patch 1
- Add new RB tree keys, cgrp_id and cgrp_group_index, for cgroup. Now, cgrp_id is the same as css subsys-unique ID.
- Add per-cpu pinned/flexible_event in perf_cgroup to track the left most event for a cgroup.
- Add per-cpu rotated_event in perf_cgroup to handle multiplexing. Disable fast path for multiplexing.
- Support hierarchies
- Update test result. Test with different hierarchy.

On systems with very high context switch rates between cgroups, there are high overhead using cgroup perf.

Current codes have two issues.
- System-wide events are mistakenly switched in cgroup context switch. It causes system-wide events miscounting, and brings avoidable overhead.
Patch 1 fixes the issue.
- The cgroup context switch sched_in is low efficient.
All cgroup events share the same per-cpu pinned/flexible groups. The RB trees for pinned/flexible groups don't understand cgroup. Current code has to traverse all events, and use event_filter_match() to filter the events for specific cgroup.
Patch 2-4 adds a fast path for cgroup context switch sched_in by training the RB tree to understand cgroup. The extra filtering can be avoided.

Here is test with 6 child cgroups (sibling cgroups), 1 parent cgroup and system-wide events.
A specjbb benchmark is running in each child cgroup.
The perf command is as below.
perf stat -e cycles,instructions -e cycles,instructions -e cycles,instructions -e cycles,instructions
-e cycles,instructions -e cycles,instructions
-e cycles,instructions -e cycles,instructions
-G cgroup1,cgroup1,cgroup2,cgroup2,cgroup3,cgroup3
-G cgroup4,cgroup4,cgroup5,cgroup5,cgroup6,cgroup6
-G cgroup_parent,cgroup_parent
-a -e cycles,instructions -I 1000

The average RT (Response Time) reported from specjbb is used as key performance metrics. (The lower the better)

RT(us) Overhead

Baseline (no perf stat):                4286.9
Use cgroup perf, no patches:            4537.1                5.84%
Use cgroup perf, apply patch 1:         4440.7                3.59%
Use cgroup perf, apple all patches:     4403.5                2.72%

Kan Liang (4):
perf: Fix system-wide events miscounting during cgroup monitoring
perf: Add filter_match() as a parameter for pinned/flexible_sched_in() perf cgroup: Add new RB tree keys for cgroup
perf cgroup: Add fast path for cgroup switch

 include/linux/perf_event.h |   6 +
 kernel/events/core.c       | 427 ++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 394 insertions(+), 39 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH v2 1/4 perf: Fix system-wide events miscounting during cgroup monitoring

From: Kan Liang <[email protected]>

When counting system-wide events and cgroup events simultaneously, the value of system-wide events are miscounting. For example,

perf stat -e cycles,instructions -e cycles,instructions -G
cgroup1,cgroup1,cgroup2,cgroup2 -a -e cycles,instructions -I 1000

     1.096265502     12,375,398,872      cycles              cgroup1
     1.096265502      8,396,184,503      instructions        cgroup1
 #    0.10  insn per cycle
     1.096265502    109,609,027,112      cycles              cgroup2
     1.096265502     11,533,690,148      instructions        cgroup2
 #    0.14  insn per cycle
     1.096265502    121,672,937,058      cycles
     1.096265502     19,331,496,727      instructions               #

0.24 insn per cycle

The events are identical events for system-wide and cgroup. The value of system-wide events is less than the sum of cgroup events, which is wrong.

Both system-wide and cgroup are per-cpu. They share the same cpuctx groups, cpuctx->flexible_groups/pinned_groups.
In context switch, cgroup switch tries to schedule all the events from the cpuctx groups. The unmatched cgroup events can be filtered by its event->cgrp. However, system-wide events, which event->cgrp is NULL, are unconditionally switched. So the small period between the prev cgroup sched_out and the new cgroup sched_in will be missed for system-wide events.

Add new event type EVENT_CGROUP_FLEXIBLE_ONLY, EVENT_CGROUP_PINNED_ONLY, and EVENT_CGROUP_ALL_ONLY.
- EVENT_FLEXIBLE | EVENT_CGROUP_FLEXIBLE_ONLY: Only switch cgroup events from EVENT_FLEXIBLE groups.
- EVENT_PINNED | EVENT_CGROUP_PINNED_ONLY: Only switch cgroup events from EVENT_PINNED groups.
- EVENT_ALL | EVENT_CGROUP_ALL_ONLY: Only switch cgroup events from both EVENT_FLEXIBLE and EVENT_PINNED groups.
For cgroup schedule out, only cgroup events are scheduled out now. For cgroup schedule in, to keep the priority order (cpu pinned, cpu flexible), the event type of the new cgroup has to be checked. If the new cgroup has pinned events, the flexible system-wide events have to be scheduled out before all events schedule in, which give the pinned events the best chance to be scheduled.
Otherwise, only cgroup events are scheduled in.

To track the event type in a cgroup, add cgrp_event_type for cgroup. The event type of the cgroup and its ancestor is stored.

Fixes e5d1367f17ba ("perf: Add cgroup support")
---
 include/linux/perf_event.h |   1 +
 kernel/events/core.c       | 119 ++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 108 insertions(+), 12 deletions(-)
include/linux/perf_event.h +1 -0
index e47ef76..3f12937 100644
@@ -836,6 +836,7 @@ struct perf_cgroup_info {
 struct perf_cgroup {
 	struct cgroup_subsys_state	css;
 	struct perf_cgroup_info	__percpu *info;
+	int				cgrp_event_type;
 };
 
 /*
kernel/events/core.c +107 -12

235 lines not on this page · raw message

Signature · 1 line
2.7.4
#2 lore raw

PATCH v2 2/4 perf: Add filter_match() as a parameter for pinned/flexible_sched_in()

In reply to [email protected]

From: Kan Liang <[email protected]>

A fast path will be introduced in the following patches to speed up the cgroup events sched in, which only needs a simpler filter_match().

Add filter_match() as a parameter for pinned/flexible_sched_in().

No functional change.

---
 kernel/events/core.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
kernel/events/core.c +9 -6
index e7ca0474..a3885e68 100644
@@ -3316,7 +3316,8 @@ static void cpu_ctx_sched_out(struct perf_cpu_context *cpuctx,
 }
 
 static int visit_groups_merge(struct perf_event_groups *groups, int cpu,
-			      int (*func)(struct perf_event *, void *), void *data)
+			      int (*func)(struct perf_event *, void *, int (*)(struct perf_event *)),
+			      void *data)
 {
 	struct perf_event **evt, *evt1, *evt2;
 	int ret;
@@ -3336,7 +3337,7 @@ static int visit_groups_merge(struct perf_event_groups *groups, int cpu,
 			evt = &evt2;
 		}
 
-		ret = func(*evt, data);
+		ret = func(*evt, data, event_filter_match);
 		if (ret)
 			return ret;
 
@@ -3353,7 +3354,8 @@ struct sched_in_data {
 	enum event_type_t event_type;
 };
 
-static int pinned_sched_in(struct perf_event *event, void *data)
+static int pinned_sched_in(struct perf_event *event, void *data,
+			   int (*filter_match)(struct perf_event *))
 {
 	struct sched_in_data *sid = data;
 
@@ -3363,7 +3365,7 @@ static int pinned_sched_in(struct perf_event *event, void *data)
 	if (perf_cgroup_skip_switch(sid->event_type, event, true))
 		return 0;
 
-	if (!event_filter_match(event))
+	if (!filter_match(event))
 		return 0;
 
 	if (group_can_go_on(event, sid->cpuctx, sid->can_add_hw)) {
@@ -3381,7 +3383,8 @@ static int pinned_sched_in(struct perf_event *event, void *data)
 	return 0;
 }
 
-static int flexible_sched_in(struct perf_event *event, void *data)
+static int flexible_sched_in(struct perf_event *event, void *data,
+			     int (*filter_match)(struct perf_event *))
 {
 	struct sched_in_data *sid = data;
 
@@ -3391,7 +3394,7 @@ static int flexible_sched_in(struct perf_event *event, void *data)
 	if (perf_cgroup_skip_switch(sid->event_type, event, false))
 		return 0;
 
-	if (!event_filter_match(event))
+	if (!filter_match(event))
 		return 0;
 
 	if (group_can_go_on(event, sid->cpuctx, sid->can_add_hw)) {
Signature · 1 line
2.7.4
#3 lore raw

PATCH v2 3/4 perf cgroup: Add new RB tree keys for cgroup

In reply to [email protected]

From: Kan Liang <[email protected]>

Current RB tree for pinned/flexible groups doesn't take cgroup into account. All events on a given CPU will be fed to pinned/flexible_sched_in(), which relies on perf_cgroup_match() to filter the events for a specific cgroup. The method has high overhead, especially in frequent context switch with several events and cgroups involved.

Add new RB tree keys, cgrp_id and cgrp_group_index, for cgroup. The unique cgrp_id (the same as css subsys-unique ID) is used to indicate a cgroup. Events in the same cgroup has the same cgrp_id. The cgrp_id is always zero for non-cgroup case. There is no functional change for non-cgroup case.
The cgrp_group_index is used for multiplexing. The rotated events of a cgroup has the same cgrp_group_index, which equals to the (group_index -1) of the first rotated events.
The non-cgroup events, e.g. system-wide events, are treated as special cgroups. The cgrp_group_index is also updated in multiplexing.

Add percpu pinned/flexible_event in perf_cgroup to track the left most event for a cgroup, which will be used later to fast access the event of a given cgroup.
Add percpu rotated_event to track the rotated events of a cgroup.

Add perf_event_groups_first_cgroup() to find the left most event for a given cgroup ID and cgrp_group_index on a given CPU.

---
 include/linux/perf_event.h |   5 ++
 kernel/events/core.c       | 217 ++++++++++++++++++++++++++++++++++++++++++---
 2 files changed, 210 insertions(+), 12 deletions(-)
include/linux/perf_event.h +5 -0
index 3f12937..800bf62 100644
@@ -703,6 +703,8 @@ struct perf_event {
 
 #ifdef CONFIG_CGROUP_PERF
 	struct perf_cgroup		*cgrp; /* cgroup event is attach to */
+	u64				cgrp_id; /* perf cgroup ID */
+	u64				cgrp_group_index;
 #endif
 
 	struct list_head		sb_list;
@@ -837,6 +839,9 @@ struct perf_cgroup {
 	struct cgroup_subsys_state	css;
 	struct perf_cgroup_info	__percpu *info;
 	int				cgrp_event_type;
+	struct perf_event * __percpu	*pinned_event;
+	struct perf_event * __percpu	*flexible_event;
+	struct perf_event * __percpu	*rotated_event;
 };
 
 /*
kernel/events/core.c +205 -12

306 lines not on this page · raw message

Signature · 1 line
2.7.4
#4 lore raw

PATCH v2 4/4 perf cgroup: Add fast path for cgroup switch

In reply to [email protected]

From: Kan Liang <[email protected]>

Generic visit_groups_merge() is used in cgroup context switch to sched in cgroup events, which has high overhead especially in frequent context switch with several events and cgroups involved. Because it feeds all events on a given CPU to pinned/flexible_sched_in() regardless the cgroup.

Add a fast path to only feed the specific cgroup events to pinned/flexible_sched_in() in cgroup context switch for non-multiplexing case.

Don't need event_filter_match() to filter cgroup and CPU in fast path. Only pmu_filter_match() is enough.

Don't need to specially handle system-wide event for fast path. Move it to slow path.

---
 kernel/events/core.c | 92 ++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 75 insertions(+), 17 deletions(-)
kernel/events/core.c +75 -17
index 6891c74..67b0135 100644
@@ -1780,6 +1780,20 @@ perf_event_groups_first_cgroup(struct perf_event_groups *groups,
 	return match;
 }
 
+static struct perf_event *
+perf_event_groups_next_cgroup(struct perf_event *event)
+{
+	struct perf_event *next;
+
+	next = rb_entry_safe(rb_next(&event->group_node), typeof(*event), group_node);
+	if (next && (next->cpu == event->cpu) &&
+	    (next->cgrp_group_index == event->cgrp_group_index) &&
+	    (next->cgrp_id == event->cgrp_id))
+		return next;
+
+	return NULL;
+}
+
 static void
 perf_event_groups_insert(struct perf_event_groups *groups,
 			 struct perf_event *event)
@@ -3464,13 +3478,69 @@ static void cpu_ctx_sched_out(struct perf_cpu_context *cpuctx,
 	ctx_sched_out(&cpuctx->ctx, cpuctx, event_type);
 }
 
+struct sched_in_data {
+	struct perf_event_context *ctx;
+	struct perf_cpu_context *cpuctx;
+	int can_add_hw;
+	enum event_type_t event_type;
+};
+
+#ifdef CONFIG_CGROUP_PERF
+
+static void cgroup_visit_groups(struct perf_event *evt, void *data,
+				int (*func)(struct perf_event *, void *, int (*)(struct perf_event *)))
+{
+	while (evt) {
+		if (func(evt, (void *)data, pmu_filter_match))
+			break;
+		evt = perf_event_groups_next_cgroup(evt);
+	}
+}
+
+static int cgroup_visit_groups_merge(int cpu, void *data,
+				     int (*func)(struct perf_event *, void *, int (*)(struct perf_event *)))
+{
+	struct sched_in_data *sid = data;
+	struct cgroup_subsys_state *css;
+	struct perf_cgroup *cgrp;
+	struct perf_event *evt, *rotated_evt = NULL;
+
+	for (css = &sid->cpuctx->cgrp->css; css; css = css->parent) {
+		/* root cgroup doesn't have events */
+		if (css->id == 1)
+			return 0;
+
+		cgrp = container_of(css, struct perf_cgroup, css);
+		/* Only visit groups when the cgroup has events */
+		if (cgrp->cgrp_event_type & sid->event_type) {
+			if (CGROUP_PINNED(sid->event_type))
+				evt = *per_cpu_ptr(cgrp->pinned_event, cpu);
+			else {
+				evt = *per_cpu_ptr(cgrp->flexible_event, cpu);
+				rotated_evt = *per_cpu_ptr(cgrp->rotated_event, cpu);
+			}
+			cgroup_visit_groups(evt, data, func);
+			cgroup_visit_groups(rotated_evt, data, func);
+		}
+	}
+
+	return 0;
+}
+#endif
+
 static int visit_groups_merge(struct perf_event_groups *groups, int cpu,
 			      int (*func)(struct perf_event *, void *, int (*)(struct perf_event *)),
 			      void *data)
 {
 	struct perf_event **evt, *evt1, *evt2;
+	struct sched_in_data *sid = data;
 	int ret;
 
+#ifdef CONFIG_CGROUP_PERF
+	/* fast path for cgroup switch, not support multiplexing */
+	if ((sid->event_type) && !sid->cpuctx->hrtimer_active)
+		return cgroup_visit_groups_merge(cpu, data, func);
+#endif
 	evt1 = perf_event_groups_first(groups, -1);
 	evt2 = perf_event_groups_first(groups, cpu);
 
@@ -3486,23 +3556,17 @@ static int visit_groups_merge(struct perf_event_groups *groups, int cpu,
 			evt = &evt2;
 		}
 
-		ret = func(*evt, data, event_filter_match);
-		if (ret)
-			return ret;
-
+		if (!perf_cgroup_skip_switch(sid->event_type, *evt, CGROUP_PINNED(sid->event_type))) {
+			ret = func(*evt, data, event_filter_match);
+			if (ret)
+				return ret;
+		}
 		*evt = perf_event_groups_next(*evt);
 	}
 
 	return 0;
 }
 
-struct sched_in_data {
-	struct perf_event_context *ctx;
-	struct perf_cpu_context *cpuctx;
-	int can_add_hw;
-	enum event_type_t event_type;
-};
-
 static int pinned_sched_in(struct perf_event *event, void *data,
 			   int (*filter_match)(struct perf_event *))
 {
@@ -3511,9 +3575,6 @@ static int pinned_sched_in(struct perf_event *event, void *data,
 	if (event->state <= PERF_EVENT_STATE_OFF)
 		return 0;
 
-	if (perf_cgroup_skip_switch(sid->event_type, event, true))
-		return 0;
-
 	if (!filter_match(event))
 		return 0;
 
@@ -3540,9 +3601,6 @@ static int flexible_sched_in(struct perf_event *event, void *data,
 	if (event->state <= PERF_EVENT_STATE_OFF)
 		return 0;
 
-	if (perf_cgroup_skip_switch(sid->event_type, event, false))
-		return 0;
-
 	if (!filter_match(event))
 		return 0;
 
Signature · 1 line
2.7.4