Skip to contentSkip to conversation

perf_events: improve Intel event scheduling

PATCH In review Stephane Eranian 4 participants 19 messages Oct 19, 2009 – Jan 7, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 perf_events: improve Intel event scheduling

This patch improves Intel event scheduling by maximizing the use of PMU registers regardless of the order in which events are submitted.

The algorithm takes into account the list of counter constraints for each event. It assigns events to counters from the most constrained, i.e., works on only one counter, to the least constrained, i.e., works on any counter.

Fixed counter events and the BTS special event are also handled via this algorithm which is designed to be fairly generic.

The patch also updates the validation of an event to use the scheduling algorithm. This will cause early failure in perf_event_open().

---
 arch/x86/include/asm/perf_event.h |    6 +-
 arch/x86/kernel/cpu/perf_event.c  |  497 +++++++++++++++++++++++--------------
 2 files changed, 318 insertions(+), 185 deletions(-)
arch/x86/include/asm/perf_event.h +5 -1
index 8d9f854..7c737af 100644
@@ -26,7 +26,9 @@
 /*
  * Includes eventsel and unit mask as well:
  */
-#define ARCH_PERFMON_EVENT_MASK				    0xffff
+#define ARCH_PERFMON_EVENTSEL_EVENT_MASK		    0x00ff
+#define ARCH_PERFMON_EVENTSEL_UNIT_MASK			    0xff00
+#define ARCH_PERFMON_EVENT_MASK				    (ARCH_PERFMON_EVENTSEL_UNIT_MASK|ARCH_PERFMON_EVENTSEL_EVENT_MASK)
 
 /*
  * filter mask to validate fixed counter events.
@@ -38,6 +40,8 @@
  *  The any-thread option is supported starting with v3.
  */
 #define ARCH_PERFMON_EVENT_FILTER_MASK			0xff840000
+#define ARCH_PERFMON_FIXED_EVENT_MASK (ARCH_PERFMON_EVENT_FILTER_MASK|ARCH_PERFMON_EVENT_MASK)
+
 
 #define ARCH_PERFMON_UNHALTED_CORE_CYCLES_SEL		      0x3c
 #define ARCH_PERFMON_UNHALTED_CORE_CYCLES_UMASK		(0x00 << 8)
arch/x86/kernel/cpu/perf_event.c +313 -184

616 lines not on this page · raw message

Signature · 1 line
1.5.4.3
maintainer
#1 lore raw

Sorry on the delay.

In general I'd very much like to see some of this generalized because I think Sparc64 has very similar 'simple' constraints, I'll have to defer to Paul on how much of this could possibly be re-used for powerpc.

Quoted 9 lines · Stephane Eranian
 arch/x86/include/asm/perf_event.h |    6 +-
 arch/x86/kernel/cpu/perf_event.c  |  497 +++++++++++++++++++++++--------------
 2 files changed, 318 insertions(+), 185 deletions(-)
arch/x86/include/asm/perf_event.h +0 -0
index 8d9f854..7c737af 100644

arch/x86/include/asm/perf_event.h line 31

Quoted 7 lines
arch/x86/include/asm/perf_event.h +3 -1
@@ -26,7 +26,9 @@
 /*
  * Includes eventsel and unit mask as well:
  */
-#define ARCH_PERFMON_EVENT_MASK				    0xffff
+#define ARCH_PERFMON_EVENTSEL_EVENT_MASK		    0x00ff
+#define ARCH_PERFMON_EVENTSEL_UNIT_MASK			    0xff00
+#define ARCH_PERFMON_EVENT_MASK				    (ARCH_PERFMON_EVENTSEL_UNIT_MASK|ARCH_PERFMON_EVENTSEL_EVENT_MASK)

There's various forms of the above throughout the code, it would be nice not to add more..

And in general this patch has way too many >80 lines and other style nits, but those can be fixed up easily enough I guess.

arch/x86/kernel/cpu/perf_event.c struct debug_store { · line 72

Quoted 10 lines
arch/x86/kernel/cpu/perf_event.c +1 -0
index 2e20bca..0f96c51 100644
@@ -68,6 +69,15 @@ struct debug_store {
 	u64	pebs_event_reset[MAX_PEBS_EVENTS];
 };
 
+#define BITS_TO_U64(nr) DIV_ROUND_UP(nr, BITS_PER_BYTE * sizeof(u64))

Do we need this, is it realistic to expect X86_PMC_IDX_MAX to be anything else than 64?

arch/x86/kernel/cpu/perf_event.c struct cpu_hw_events { · line 98

Quoted 7 lines
arch/x86/kernel/cpu/perf_event.c +5 -2
-#define EVENT_CONSTRAINT(c, m) { .code = (c), .idxmsk[0] = (m) }
-#define EVENT_CONSTRAINT_END  { .code = 0, .idxmsk[0] = 0 }
+#define EVENT_CONSTRAINT(c, n, w, m) { \
+	.code = (c),	\
+	.mask = (m), 	\
+	.weight = (w),	\
+	.idxmsk[0] = (n) }

If not, we can do away with the weight argument here and use hweight64() which should reduce to a compile time constant.

arch/x86/kernel/cpu/perf_event.c static DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events) = { · line 140

Quoted 6 lines
arch/x86/kernel/cpu/perf_event.c +1 -1
@@ -124,7 +137,7 @@ static DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events) = {
        .enabled = 1,
 };
 
-static const struct event_constraint *event_constraints;
+static struct event_constraint *event_constraints;

I'm thinking this ought to live in x86_pmu, or possible, if we can generalize this enough, in pmu.

arch/x86/kernel/cpu/perf_event.c static const u64 intel_perfmon_event_map[] = · line 213

Quoted 2 lines
arch/x86/kernel/cpu/perf_event.c +2 -0
+static struct event_constraint intel_core_event_constraints[] =
+{

Inconsistent style with below:

Quoted 5 lines
arch/x86/kernel/cpu/perf_event.c +5 -0
+static struct event_constraint intel_nehalem_event_constraints[] = {
+	EVENT_CONSTRAINT(0xc0, (0x3|(1ULL<<32)), 3, ARCH_PERFMON_FIXED_EVENT_MASK), /* INSTRUCTIONS_RETIRED */
+	EVENT_CONSTRAINT(0x3c, (0x3|(1ULL<<33)), 3, ARCH_PERFMON_FIXED_EVENT_MASK), /* UNHALTED_CORE_CYCLES */
+	EVENT_CONSTRAINT(0x40, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* L1D_CACHE_LD */
+	EVENT_CONSTRAINT(0x41, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* L1D_CACHE_ST */

arch/x86/kernel/cpu/perf_event.c static const u64 intel_perfmon_event_map[] = · line 242

Quoted 10 lines
arch/x86/kernel/cpu/perf_event.c +8 -0
+	EVENT_CONSTRAINT(0x42, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* L1D_CACHE_LOCK */
+	EVENT_CONSTRAINT(0x43, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* L1D_ALL_REF */
+	EVENT_CONSTRAINT(0x4e, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* L1D_PREFETCH */
+	EVENT_CONSTRAINT(0x4c, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* LOAD_HIT_PRE */
+	EVENT_CONSTRAINT(0x51, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* L1D */
+	EVENT_CONSTRAINT(0x52, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* L1D_CACHE_PREFETCH_LOCK_FB_HIT */
+	EVENT_CONSTRAINT(0x53, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* L1D_CACHE_LOCK_FB_HIT */
+	EVENT_CONSTRAINT(0xc5, 0x3, 2, ARCH_PERFMON_EVENTSEL_EVENT_MASK), /* CACHE_LOCK_CYCLES */
 	EVENT_CONSTRAINT_END
 };

Would be nice to get rid of that EVENT_MASK part, maybe write EVENT_CONSTRAINT like:

.mask = ARCH_PERFMON_EVENTSEL_EVENT_MASK | ((n)>>32 ? ARCH_PERFMON_FIXED_MASK : 0),

Which ought to work for Intel based things, AMD will need a different base event mask.

Quoted 5 lines
arch/x86/kernel/cpu/perf_event.c +1 -0
@@ -1120,9 +1136,15 @@ static void amd_pmu_disable_all(void)
 
 void hw_perf_disable(void)
 {
+	struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);

arch/x86/kernel/cpu/perf_event.c static void amd_pmu_disable_all(void) · line 1148

Quoted 10 lines
arch/x86/kernel/cpu/perf_event.c +6 -1
+
 	if (!x86_pmu_initialized())
 		return;
-	return x86_pmu.disable_all();
+
+	if (cpuc->enabled)
+		cpuc->n_events = 0;
+
+	x86_pmu.disable_all();
 }

Right, so I cannot directly see the above being correct. You fully erase the PMU state when we disable it, but things like single counter add/remove doesn't reprogram the full PMU afaict.

The powerpc code has n_added, which indicates a delta algorithm is used there.

arch/x86/kernel/cpu/perf_event.c perf_event_nmi_handler(struct notifier_block *self, · line 1898

Quoted 13 lines
arch/x86/kernel/cpu/perf_event.c +13 -0
+static struct event_constraint *intel_get_event_constraints(struct perf_event *event)
+{
+	struct event_constraint *c;
+
+	c = intel_special_constraints(event);
+	if (c)
+		return c;
+
+	if (event_constraints)
+		for_each_event_constraint(c, event_constraints) {
+			if ((event->hw.config & c->mask) == c->code)
+				return c;
+		}

This wants extra { }, since its a multi-line stmt.

arch/x86/kernel/cpu/perf_event.c perf_event_nmi_handler(struct notifier_block *self, · line 1906

Quoted 7 lines
arch/x86/kernel/cpu/perf_event.c +7 -0
+	return NULL;
+}
+
+static struct event_constraint *amd_get_event_constraints(struct perf_event *event)
+{
+	return NULL;
+}

I guess we'll need to fill that out a bit more, but that can be another patch.

Quoted 106 lines
arch/x86/kernel/cpu/perf_event.c +106 -0
+static int schedule_events(struct cpu_hw_events *cpuhw, int n, bool assign)
+{
+	int i, j , w, num, lim;
+	int weight, wmax;
+	struct event_constraint *c;
+	unsigned long used_mask[BITS_TO_LONGS(X86_PMC_IDX_MAX)];
+	int assignments[X86_PMC_IDX_MAX];
+	struct hw_perf_event *hwc;
+
+	bitmap_zero(used_mask, X86_PMC_IDX_MAX);
+
+	/*
+	 * weight = number of possible counters
+	 *
+	 * 1    = most constrained, only works on one counter
+	 * wmax = least constrained, works on 1 fixed counter
+	 *        or any generic counter
+	 *
+	 * assign events to counters starting with most
+	 * constrained events.
+	 */
+	wmax = 1 + x86_pmu.num_events;
+	num = n;
+	for(w=1; num && w <= wmax; w++) {
+
+		/* for each event */
+		for(i=0; i < n; i++) {
+			c = cpuhw->constraints[i];
+			hwc = &cpuhw->event_list[i]->hw;
+
+			weight = c ? c->weight : x86_pmu.num_events;
+			if (weight != w)
+				continue;
+
+			/*
+			 * try to reuse previous assignment
+			 *
+			 * This is possible despite the fact that
+			 * events or events order may have changed.
+			 *
+			 * What matters is the level of constraints
+			 * of an event and this is constant for now.
+			 *
+			 * This is possible also because we always
+			 * scan from most to least constrained. Thus,
+			 * if a counter can be reused, it means no,
+			 * more constrained events, needed it. And
+			 * next events will either compete for it
+			 * (which cannot be solved anyway) or they
+			 * have fewer constraints, and they can use
+			 * another counter.
+			 */
+			j = hwc->idx;
+			if (j != -1 && !test_bit(j, used_mask))
+				goto skip;
+
+			if (c) {
+				lim = X86_PMC_IDX_MAX;
+				for_each_bit(j, (unsigned long *)c->idxmsk, lim)
+					if (!test_bit(j, used_mask))
+						break;
+
+			} else  {
+				lim = x86_pmu.num_events;
+				/*
+				 * fixed counter events have necessarily a
+				 * constraint thus we come here only for
+				 * generic counters and thus we limit the
+				 * scan to those
+				 */
+				j = find_first_zero_bit(used_mask, lim);
+			}
+			if (j == lim)
+				return -EAGAIN;
+skip:
+			set_bit(j, used_mask);
+			assignments[i] = j;
+			num--;
+		}
+	}
+	if (num)
+		return -ENOSPC;
+
+	/* just simulate scheduling */
+	if (!assign)
+		return 0;
+
+	/*
+	 * commit assignments
+	 */
+	for(i=0; i < n; i++) {
+		hwc = &cpuhw->event_list[i]->hw;
+
+		hwc->idx = assignments[i];
+
+		set_bit(hwc->idx, cpuhw->used_mask);
+
+		if (hwc->idx == X86_PMC_IDX_FIXED_BTS) {
+			hwc->config_base = 0;
+			hwc->event_base	= 0;
+		} else if (hwc->idx >= X86_PMC_IDX_FIXED) {
+			hwc->config_base = MSR_ARCH_PERFMON_FIXED_CTR_CTRL;
+			/*
+			 * We set it so that event_base + idx in wrmsr/rdmsr maps to
+			 * MSR_ARCH_PERFMON_FIXED_CTR0 ... CTR2:
+			 */

arch/x86/kernel/cpu/perf_event.c perf_event_nmi_handler(struct notifier_block *self, · line 2023

Quoted 10 lines
arch/x86/kernel/cpu/perf_event.c +10 -0
+			hwc->event_base =
+				MSR_ARCH_PERFMON_FIXED_CTR0 - X86_PMC_IDX_FIXED;
+		} else {
+			hwc->config_base = x86_pmu.eventsel;
+			hwc->event_base  = x86_pmu.perfctr;
+		}
+	}
+	cpuhw->n_events = n;
+	return 0;
+}

Straight forward O(n^2) algorithm looking for the best match, seems good from a single read -- will go over it again on another day to find more details.

arch/x86/kernel/cpu/perf_event.c perf_event_nmi_handler(struct notifier_block *self, · line 2035

Quoted 11 lines
arch/x86/kernel/cpu/perf_event.c +11 -0
+static int collect_events(struct cpu_hw_events *cpuhw, struct perf_event *leader)
+{
+	struct perf_event *event;
+	int n, max_count;
+
+	max_count = x86_pmu.num_events + x86_pmu.num_events_fixed;
+
+	/* current number of events already accepted */
+	n = cpuhw->n_events;
+
+	if (!is_software_event(leader)) {

With things like the hw-breakpoint stuff also growing a pmu ! is_software() isn't strong enough, something like:

static inline int is_x86_event(struct perf_event *event)

{
	return event->pmu == &pmu;
}

Should work though.

Quoted 36 lines
arch/x86/kernel/cpu/perf_event.c +36 -0
+		if (n >= max_count)
+			return -ENOSPC;
+		cpuhw->constraints[n] = x86_pmu.get_event_constraints(leader);
+		cpuhw->event_list[n] = leader;
+		n++;
+	}
+
+	list_for_each_entry(event, &leader->sibling_list, group_entry) {
+		if (is_software_event(event) ||
+		    event->state == PERF_EVENT_STATE_OFF)
+			continue;
+
+		if (n >= max_count)
+			return -ENOSPC;
+
+		cpuhw->constraints[n] = x86_pmu.get_event_constraints(event);
+		cpuhw->event_list[n] = event;
+		n++;
+	}
+	return n;
+}
+
+/*
+ * Called to enable a whole group of events.
+ * Returns 1 if the group was enabled, or -EAGAIN if it could not be.
+ * Assumes the caller has disabled interrupts and has
+ * frozen the PMU with hw_perf_save_disable.
+ */
+int hw_perf_group_sched_in(struct perf_event *leader,
+	       struct perf_cpu_context *cpuctx,
+	       struct perf_event_context *ctx, int cpu)
+{
+	struct cpu_hw_events *cpuhw = &per_cpu(cpu_hw_events, cpu);
+	int n, ret;
+
+	n = collect_events(cpuhw, leader);

arch/x86/kernel/cpu/perf_event.c perf_event_nmi_handler(struct notifier_block *self, · line 2081

Quoted 10 lines
arch/x86/kernel/cpu/perf_event.c +10 -0
+	if (n < 0)
+		return n;
+
+	ret = schedule_events(cpuhw, n, true);
+	if (ret)
+		return ret;
+
+	/* 0 means successful and enable each event in caller */
+	return 0;
+}

This is where powerpc does n_added += n, and it delays the schedule_events() bit to hw_perf_enable() conditional on n_added > 0. When you remove events it simply keeps the current layout and disables the one.

arch/x86/kernel/cpu/perf_event.c static int intel_pmu_init(void) · line 2252

Quoted 10 lines
arch/x86/kernel/cpu/perf_event.c +1 -1
@@ -2123,8 +2245,8 @@ static int intel_pmu_init(void)
 		memcpy(hw_cache_event_ids, core2_hw_cache_event_ids,
 		       sizeof(hw_cache_event_ids));
 
-		pr_cont("Core2 events, ");
 		event_constraints = intel_core_event_constraints;
+		pr_cont("Core2 events, ");
 		break;
 	default:
 	case 26:

Not that I object to the above change, but it seems out of place in this patch.

Quoted 36 lines
arch/x86/kernel/cpu/perf_event.c +22 -6
+/*
+ * validate a single event group
+ *
+ * validation include:
+ * 	- check events are compatible which each other
+ * 	- events do not compete for the same counter
+ * 	- number of events <= number of counters
+ *
+ * validation ensures the group can be loaded onto the
+ * PMU if it were the only group available.
+ */
 static int validate_group(struct perf_event *event)
 {
-	struct perf_event *sibling, *leader = event->group_leader;
+	struct perf_event *leader = event->group_leader;
 	struct cpu_hw_events fake_pmu;
+	int n, ret;
 
 	memset(&fake_pmu, 0, sizeof(fake_pmu));
 
-	if (!validate_event(&fake_pmu, leader))
+	/*
+	 * the event is not yet connected with its
+	 * siblings thererfore we must first collect
+	 * existing siblings, then add the new event
+	 * before we can simulate the scheduling
+	 */
+	n = collect_events(&fake_pmu, leader);
+	if (n < 0)
 		return -ENOSPC;
 
-	list_for_each_entry(sibling, &leader->sibling_list, group_entry) {
-		if (!validate_event(&fake_pmu, sibling))
-			return -ENOSPC;
-	}
+	fake_pmu.n_events = n;

arch/x86/kernel/cpu/perf_event.c static const struct pmu pmu = { · line 2386

Quoted 10 lines
arch/x86/kernel/cpu/perf_event.c +4 -2
 
-	if (!validate_event(&fake_pmu, event))
+	n = collect_events(&fake_pmu, event);
+	if (n < 0)
 		return -ENOSPC;
 
-	return 0;
+	ret = schedule_events(&fake_pmu, n, false);
+	return ret ? -ENOSPC : 0;
 }

This seems good.

#2 lore raw
Quoted 4 lines · Peter Zijlstra

In general I'd very much like to see some of this generalized because I think Sparc64 has very similar 'simple' constraints, I'll have to defer to Paul on how much of this could possibly be re-used for powerpc.

Let's wait until we have all X86 constraint scheduling code. There is way more needed. I have a first cut on the AMD64 constraints but there is an issue in the generic/arch specific API that needs to be flushed out first (see below).

Quoted 8 lines · Stephane Eranian
Quoted 7 lines
arch/x86/include/asm/perf_event.h +0 -0
index 8d9f854..7c737af 100644
@@ -26,7 +26,9 @@
  /*
   * Includes eventsel and unit mask as well:
Quoted 9 lines

arch/x86/include/asm/perf_event.h line 31

Quoted 5 lines
arch/x86/include/asm/perf_event.h +3 -1
   */
-#define ARCH_PERFMON_EVENT_MASK                                  0xffff
+#define ARCH_PERFMON_EVENTSEL_EVENT_MASK                 0x00ff
+#define ARCH_PERFMON_EVENTSEL_UNIT_MASK                          0xff00
+#define ARCH_PERFMON_EVENT_MASK                                  (ARCH_PERFMON_EVENTSEL_UNIT_MASK|ARCH_PERFMON_EVENTSEL_EVENT_MASK)

There's various forms of the above throughout the code, it would be nice not to add more..

I will simplify this.

Quoted 6 lines
Quoted 5 lines
arch/x86/kernel/cpu/perf_event.c +0 -0
index 2e20bca..0f96c51 100644
Quoted 9 lines

arch/x86/kernel/cpu/perf_event.c struct debug_store { · line 72

Quoted 5 lines
arch/x86/kernel/cpu/perf_event.c +1 -0
@@ -68,6 +69,15 @@ struct debug_store {
       u64     pebs_event_reset[MAX_PEBS_EVENTS];
  };
 
+#define BITS_TO_U64(nr) DIV_ROUND_UP(nr, BITS_PER_BYTE * sizeof(u64))

Do we need this, is it realistic to expect X86_PMC_IDX_MAX to be anything else than 64?

The issue had to do with i386 mode where long are 32 bits < 64. And in particular with the initializer .idxmsk[0] = V. In the future we may exceed 32 registers. That means the initializer would have to change. But I guess we have quite some ways before this case is reached. So I will revert all of this to unsigned long.

Quoted 11 lines

arch/x86/kernel/cpu/perf_event.c struct cpu_hw_events { · line 98

Quoted 7 lines
arch/x86/kernel/cpu/perf_event.c +5 -2
-#define EVENT_CONSTRAINT(c, m) { .code = (c), .idxmsk[0] = (m) }
-#define EVENT_CONSTRAINT_END  { .code = 0, .idxmsk[0] = 0 }
+#define EVENT_CONSTRAINT(c, n, w, m) { \
+     .code = (c),    \
+     .mask = (m),    \
+     .weight = (w),  \
+     .idxmsk[0] = (n) }

If not, we can do away with the weight argument here and use hweight64() which should reduce to a compile time constant.

I have dropped weight and I use bitmap_weight() to recompute, with all the inlining
it should come out to be a simple popcnt instruction.

Quoted 9 lines

arch/x86/kernel/cpu/perf_event.c static DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events) = { · line 140

Quoted 6 lines
arch/x86/kernel/cpu/perf_event.c +1 -1
@@ -124,7 +137,7 @@ static DEFINE_PER_CPU(struct cpu_hw_events, cpu_hw_events) = {
         .enabled = 1,
  };
 
-static const struct event_constraint *event_constraints;
+static struct event_constraint *event_constraints;

I'm thinking this ought to live in x86_pmu, or possible, if we can generalize this enough, in pmu.

True.

Quoted 5 lines

Inconsistent style with below:

arch/x86/kernel/cpu/perf_event.c static const u64 intel_perfmon_event_map[] = · line 229

Quoted 2 lines
arch/x86/kernel/cpu/perf_event.c +2 -0
+static struct event_constraint intel_nehalem_event_constraints[] = {
+     EVENT_CONSTRAINT(0xc0, (0x3|(1ULL<<32)), 3, ARCH_PERFMON_FIXED_EVENT_MASK), /* INSTRUCTIONS_RETIRED */
Quoted 8 lines

Would be nice to get rid of that EVENT_MASK part, maybe write EVENT_CONSTRAINT like:

 .mask = ARCH_PERFMON_EVENTSEL_EVENT_MASK | ((n)>>32 ? ARCH_PERFMON_FIXED_MASK : 0),

Which ought to work for Intel based things, AMD will need a different base event mask.

Will try to clean this up. AMD does not need a mask, it is a completely different kind of constraints.

Quoted 11 lines
Quoted 11 lines
arch/x86/kernel/cpu/perf_event.c +4 -1
@@ -1120,9 +1136,15 @@ static void amd_pmu_disable_all(void)
 
  void hw_perf_disable(void)
  {
+     struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);
+
       if (!x86_pmu_initialized())
               return;
-     return x86_pmu.disable_all();
+
+     if (cpuc->enabled)
Quoted 9 lines

arch/x86/kernel/cpu/perf_event.c static void amd_pmu_disable_all(void) · line 1148

Quoted 4 lines
arch/x86/kernel/cpu/perf_event.c +3 -0
+             cpuc->n_events = 0;
+
+     x86_pmu.disable_all();
  }

Right, so I cannot directly see the above being correct. You fully erase the PMU state when we disable it, but things like single counter add/remove doesn't reprogram the full PMU afaict.

Here is the key point. There is no clear API that says 'this is the final stop for this event group, i.e., next start will be with a different group'. In other words,
there is no signal that the 'resource' can be freed. This is a showstopper for the
AMD constraints. The generic code needs to signal that this is the final stop for
the event group, so the machine specific layer can release the registers. The hw_perf_disable() is used in many places and does not indicate the same thing as what I just described.

Quoted 15 lines

arch/x86/kernel/cpu/perf_event.c perf_event_nmi_handler(struct notifier_block *self, · line 2035

Quoted 11 lines
arch/x86/kernel/cpu/perf_event.c +11 -0
+static int collect_events(struct cpu_hw_events *cpuhw, struct perf_event *leader)
+{
+     struct perf_event *event;
+     int n, max_count;
+
+     max_count = x86_pmu.num_events + x86_pmu.num_events_fixed;
+
+     /* current number of events already accepted */
+     n = cpuhw->n_events;
+
+     if (!is_software_event(leader)) {

With things like the hw-breakpoint stuff also growing a pmu ! is_software() isn't strong enough, something like:

Quoted 5 lines

static inline int is_x86_event(struct perf_event *event) {
       return event->pmu == &pmu;
}

Agreed, I am using something like this now.

Quoted 15 lines
Quoted 15 lines
arch/x86/kernel/cpu/perf_event.c +15 -0
+int hw_perf_group_sched_in(struct perf_event *leader,
+            struct perf_cpu_context *cpuctx,
+            struct perf_event_context *ctx, int cpu)
+{
+     struct cpu_hw_events *cpuhw = &per_cpu(cpu_hw_events, cpu);
+     int n, ret;
+
+     n = collect_events(cpuhw, leader);
+     if (n < 0)
+             return n;
+
+     ret = schedule_events(cpuhw, n, true);
+     if (ret)
+             return ret;
+
Quoted 9 lines
Quoted 3 lines

+     /* 0 means successful and enable each event in caller */
+     return 0;
+}

This is where powerpc does n_added += n, and it delays the schedule_events() bit to hw_perf_enable() conditional on n_added > 0. When you remove events it simply keeps the current layout and disables the one.

Will look into this.

Quoted 4 lines
Quoted 4 lines
arch/x86/kernel/cpu/perf_event.c +0 -0
@@ -2123,8 +2245,8 @@ static int intel_pmu_init(void)
               memcpy(hw_cache_event_ids, core2_hw_cache_event_ids,
                      sizeof(hw_cache_event_ids));
 
Quoted 10 lines
Quoted 6 lines
arch/x86/kernel/cpu/perf_event.c +1 -1
-             pr_cont("Core2 events, ");
               event_constraints = intel_core_event_constraints;
+             pr_cont("Core2 events, ");
               break;

      default:
      case 26:

Not that I object to the above change, but it seems out of place in this patch.

Will remove that.

#3 lore raw
Quoted 38 lines
Quoted 38 lines
Quoted 6 lines
arch/x86/kernel/cpu/perf_event.c +1 -0
@@ -1120,9 +1136,15 @@ static void amd_pmu_disable_all(void)
 
  void hw_perf_disable(void)
  {
+     struct cpu_hw_events *cpuc = &__get_cpu_var(cpu_hw_events);

arch/x86/kernel/cpu/perf_event.c static void amd_pmu_disable_all(void) · line 1148

Quoted 10 lines
arch/x86/kernel/cpu/perf_event.c +6 -1
+
       if (!x86_pmu_initialized())
               return;
-     return x86_pmu.disable_all();
+
+     if (cpuc->enabled)
+             cpuc->n_events = 0;
+
+     x86_pmu.disable_all();
  }

Right, so I cannot directly see the above being correct. You fully erase the PMU state when we disable it, but things like single counter add/remove doesn't reprogram the full PMU afaict.

Quoted 16 lines
arch/x86/kernel/cpu/perf_event.c +16 -0
+int hw_perf_group_sched_in(struct perf_event *leader,
+            struct perf_cpu_context *cpuctx,
+            struct perf_event_context *ctx, int cpu)
+{
+     struct cpu_hw_events *cpuhw = &per_cpu(cpu_hw_events, cpu);
+     int n, ret;
+
+     n = collect_events(cpuhw, leader);
+     if (n < 0)
+             return n;
+
+     ret = schedule_events(cpuhw, n, true);
+     if (ret)
+             return ret;
+
+     /* 0 means successful and enable each event in caller */
Quoted 8 lines
Quoted 8 lines
Quoted 2 lines

+     return 0;
+}

This is where powerpc does n_added += n, and it delays the schedule_events() bit to hw_perf_enable() conditional on n_added > 0. When you remove events it simply keeps the current layout and disables the one.

There is a major difference between PPC and X86 here. PPC has a centralized register to control start/stop. This register uses bitmask to enable or disable counters. Thus, in hw_perf_enable(), if n_added=0, then you just need to
use the pre-computed bitmask. Otherwise, you need to recompute the bitmask to include the new registers. The assignment of events and validation is done in hw_group_sched_in().

In X86, assignment and validation is done in hw_group_sched_in(). Activation is done individually for each counter. There is no centralized register used here, thus
no bitmask to update.

Disabling a counter does not trigger a complete reschedule of events. This happens
only when hw_group_sched_in() is called.

The n_events = 0 in hw_perf_disable() is used to signal that something is changing.
It should not be here but here. The problem is that hw_group_sched_in() needs a way
to know that it is called for a completely new series of group scheduling so it can
discard any previous assignment. This goes back to the issue I raised in my previous
email. You could add a parameter to hw_group_sched_in() that would indicate this is
the first group. that would cause n_events =0 and the function would start accumulating
events for the new scheduling period.

maintainer
#4 lore raw
Quoted 11 lines · stephane eranian

There is a major difference between PPC and X86 here. PPC has a centralized register to control start/stop. This register uses bitmask to enable or disable counters. Thus, in hw_perf_enable(), if n_added=0, then you just need to use the pre-computed bitmask. Otherwise, you need to recompute the bitmask to include the new registers. The assignment of events and validation is done in hw_group_sched_in().

In X86, assignment and validation is done in hw_group_sched_in(). Activation is done individually for each counter. There is no centralized register used here, thus no bitmask to update.

intel core2 has the global control reg, but for all intents and purposes the perf_enable/disable calls emulate this global enable/disable.

Quoted 12 lines

Disabling a counter does not trigger a complete reschedule of events. This happens only when hw_group_sched_in() is called.

The n_events = 0 in hw_perf_disable() is used to signal that something is changing. It should not be here but here. The problem is that hw_group_sched_in() needs a way to know that it is called for a completely new series of group scheduling so it can discard any previous assignment. This goes back to the issue I raised in my previous email. You could add a parameter to hw_group_sched_in() that would indicate this is the first group. that would cause n_events =0 and the function would start accumulating events for the new scheduling period.

I'm not really seeing the problem here...

perf_disable() <-- shut down the full pmu

pmu->disable() <-- hey someone got removed (easy free the reg) pmu->enable() <-- hey someone got added (harder, check constraints)

hw_perf_group_sched_in() <-- hey a full group got added

                              (better than multiple ->enable)

perf_enable() <-- re-enable pmu

So ->disable() is used to track freeing, ->enable is used to add individual counters, check constraints etc..

hw_perf_group_sched_in() is used to optimize the full group enable.

Afaict that is what power does (Paul?) and that should I think be sufficient to track x86 as well.

Since sched_in() is balanced with sched_out(), the ->disable() calls should provide the required information as to the occupation of the pmu. I don't see the need for more hooks.

Paul, could you comment, since you did all this for power?

#5 lore raw

Hi,

[Repost because of HTML]

Quoted 42 lines · Peter Zijlstra
Quoted 11 lines · stephane eranian

There is a major difference between PPC and X86 here. PPC has a centralized register to control start/stop. This register  uses bitmask to enable or disable counters. Thus, in hw_perf_enable(), if n_added=0, then you just need to use the pre-computed bitmask. Otherwise, you need to recompute the bitmask to include the new registers. The assignment of events and validation is done in hw_group_sched_in().

In X86, assignment and validation is done in hw_group_sched_in(). Activation is done individually for each counter. There is no centralized register used here, thus no bitmask to update.

intel core2 has the global control reg, but for all intents and purposes the perf_enable/disable calls emulate this global enable/disable.

Quoted 12 lines

Disabling a counter does not trigger a complete reschedule of events. This happens only when hw_group_sched_in() is called.

The n_events = 0 in hw_perf_disable() is used to signal that something is changing. It should not be here but here. The problem is that hw_group_sched_in() needs a way to know that it is called for a completely new series of group scheduling so it can discard any previous assignment. This goes back to the issue I raised in my previous email. You could add a parameter to hw_group_sched_in() that would indicate this is the first group. that would cause n_events =0 and the function would start accumulating events for the new scheduling period.

I'm not really seeing the problem here...

 perf_disable() <-- shut down the full pmu

 pmu->disable() <-- hey someone got removed (easy free the reg) pmu->enable()  <-- hey someone got added (harder, check constraints)

 hw_perf_group_sched_in() <-- hey a full group got added (better than multiple ->enable)

Quoted 8 lines

 perf_enable() <-- re-enable pmu

So ->disable() is used to track freeing, ->enable is used to add individual counters, check constraints etc..

hw_perf_group_sched_in() is used to optimize the full group enable.

Does that mean that after a disable() I can assume that there won't be an enable() without a group_sched_in()?

I suspect not. In fact, there is a counter-example in perf_ctx_adjust_freq().

Quoted 9 lines

Afaict that is what power does (Paul?) and that should I think be sufficient to track x86 as well.

Since sched_in() is balanced with sched_out(), the ->disable() calls should provide the required information as to the occupation of the pmu. I don't see the need for more hooks.

Paul, could you comment, since you did all this for power?

maintainer
#6 lore raw
Quoted 14 lines · Stephane Eranian
Quoted 14 lines

perf_disable() <-- shut down the full pmu

pmu->disable() <-- hey someone got removed (easy free the reg) pmu->enable() <-- hey someone got added (harder, check constraints)

hw_perf_group_sched_in() <-- hey a full group got added

                             (better than multiple ->enable)

perf_enable() <-- re-enable pmu

So ->disable() is used to track freeing, ->enable is used to add individual counters, check constraints etc..

Quoted 7 lines
Quoted 2 lines

hw_perf_group_sched_in() is used to optimize the full group enable.

Does that mean that after a disable() I can assume that there won't be an enable() without a group_sched_in()?

I suspect not. In fact, there is a counter-example in perf_ctx_adjust_freq().

Why would that be a problem?

A perf_disable() will simply disable the pmu, but not alter its configuration, perf_enable() will program the pmu and re-enable it (with the obvious optimization that if the current state and the new state match we can skip the actual programming).

If a ->disable() was observed between it will simply not re-enable that particular counter, that is it will remove that counters' config, and perf_enable() can do a smart reprogram by simply no re-enabling that counter.

If an ->enable() or hw_perf_group_sched_in() was observed in between, you have to recompute the full state in order to validate the availability, if that fails, no state change, if it succeeds you have a new pmu state and perf_enable() will program that.

In the case of perf_ctx_adjust_freq():

	if (!interrupts) {
		perf_disable();
		event->pmu->disable(event);
		atomic64_set(&hwc->period_left, 0);
		event->pmu->enable(event);
		perf_enable();
	}

You'll very likely end up with the same state you had before, but if not who cares -- its supposed to be an unlikely case.

That is, the ->disable() will clear the cpu_hw_events::used_mask bit. The ->enable() will want to place the counter on its last know reg, which (not so very surprisingly) will be available, hence it will trivially succeed, depending on its smarts it might or might not find that the 'new' counter's config is identical to the current hw state, if not it might set a cpu_hw_event::reprogram state.

perf_enable() will, when it sees cpu_hw_event::reprogram, re-write the pmu config and re-enable the whole thing (global ctrl reg, or iterate individual EN bits).

#7 lore raw
Quoted 36 lines · Peter Zijlstra
Quoted 14 lines · Stephane Eranian
Quoted 14 lines

 perf_disable() <-- shut down the full pmu

 pmu->disable() <-- hey someone got removed (easy free the reg) pmu->enable()  <-- hey someone got added (harder, check constraints)

 hw_perf_group_sched_in() <-- hey a full group got added (better than multiple ->enable)

 perf_enable() <-- re-enable pmu

So ->disable() is used to track freeing, ->enable is used to add individual counters, check constraints etc..

Quoted 7 lines
Quoted 2 lines

hw_perf_group_sched_in() is used to optimize the full group enable.

Does that mean that after a disable() I can assume that there won't be an enable() without a group_sched_in()?

I suspect not. In fact, there is a counter-example in perf_ctx_adjust_freq().

Why would that be a problem?

A perf_disable() will simply disable the pmu, but not alter its configuration, perf_enable() will program the pmu and re-enable it (with the obvious optimization that if the current state and the new state match we can skip the actual programming).

If a ->disable() was observed between it will simply not re-enable that particular counter, that is it will remove that counters' config, and perf_enable() can do a smart reprogram by simply no re-enabling that counter.

Quoted 5 lines

If an ->enable() or hw_perf_group_sched_in() was observed in between, you have to recompute the full state in order to validate the availability, if that fails, no state change, if it succeeds you have a new pmu state and perf_enable() will program that.

Ok, so what you are suggesting is that the assignment is actually done incrementally in ->enable(). hw_group_sched_in() would simply validate that a single group is sane (i.e., can be scheduled if it was alone).

Quoted 10 lines

In the case of perf_ctx_adjust_freq():

       if (!interrupts) {
               perf_disable(); event->pmu->disable(event); atomic64_set(&hwc->period_left, 0); event->pmu->enable(event); perf_enable();
       }

Quoted 6 lines

You'll very likely end up with the same state you had before, but if not who cares -- its supposed to be an unlikely case.

That is, the ->disable() will clear the cpu_hw_events::used_mask bit. The ->enable() will want to place the counter on its last know reg, which (not so very surprisingly) will be available, hence it will

Not in the case I am looking at on AMD. The counter you need to grab depends on what is going on on the other cores on the socket. So there is no guarantee that you will get the same. Something similar exists on Nehalem but it has to do with HT.

Quoted 7 lines

trivially succeed, depending on its smarts it might or might not find that the 'new' counter's config is identical to the current hw state, if not it might set a cpu_hw_event::reprogram state.

perf_enable() will, when it sees cpu_hw_event::reprogram, re-write the pmu config and re-enable the whole thing (global ctrl reg, or iterate individual EN bits).

maintainer
#8 lore raw
Quoted 43 lines · Stephane Eranian
Quoted 36 lines · Peter Zijlstra
Quoted 14 lines · Stephane Eranian
Quoted 14 lines

perf_disable() <-- shut down the full pmu

pmu->disable() <-- hey someone got removed (easy free the reg) pmu->enable() <-- hey someone got added (harder, check constraints)

hw_perf_group_sched_in() <-- hey a full group got added

                             (better than multiple ->enable)

perf_enable() <-- re-enable pmu

So ->disable() is used to track freeing, ->enable is used to add individual counters, check constraints etc..

Quoted 7 lines
Quoted 2 lines

hw_perf_group_sched_in() is used to optimize the full group enable.

Does that mean that after a disable() I can assume that there won't be an enable() without a group_sched_in()?

I suspect not. In fact, there is a counter-example in perf_ctx_adjust_freq().

Why would that be a problem?

A perf_disable() will simply disable the pmu, but not alter its configuration, perf_enable() will program the pmu and re-enable it (with the obvious optimization that if the current state and the new state match we can skip the actual programming).

If a ->disable() was observed between it will simply not re-enable that particular counter, that is it will remove that counters' config, and perf_enable() can do a smart reprogram by simply no re-enabling that counter.

Quoted 5 lines

If an ->enable() or hw_perf_group_sched_in() was observed in between, you have to recompute the full state in order to validate the availability, if that fails, no state change, if it succeeds you have a new pmu state and perf_enable() will program that.

Quoted 3 lines

Ok, so what you are suggesting is that the assignment is actually done incrementally in ->enable(). hw_group_sched_in() would simply validate that a single group is sane (i.e., can be scheduled if it was alone).

hw_perf_group_sched_in() can be used to do a whole group at once (see how a !0 return value will short-circuit the whole incremental group buildup), but yes ->enable() is incremental.

Quoted 13 lines
Quoted 13 lines

In the case of perf_ctx_adjust_freq():

       if (!interrupts) {
               perf_disable();
               event->pmu->disable(event);
               atomic64_set(&hwc->period_left, 0);
               event->pmu->enable(event);
               perf_enable();
       }

You'll very likely end up with the same state you had before, but if not who cares -- its supposed to be an unlikely case.

Quoted 8 lines
Quoted 3 lines

That is, the ->disable() will clear the cpu_hw_events::used_mask bit. The ->enable() will want to place the counter on its last know reg, which (not so very surprisingly) will be available, hence it will

Not in the case I am looking at on AMD. The counter you need to grab depends on what is going on on the other cores on the socket. So there is no guarantee that you will get the same. Something similar exists on Nehalem but it has to do with HT.

Well the above code is assumed correct, so we need to make it true, which should not be too hard to do.

If there are cross cpu constraints (AMD has per node constraints IIRC) then we need a structure that describes these (a per node structure for AND, a per core structure for HT), if we take a lock on this structure the first time we touch it in a perf_disable() region, and release it on perf_enable(), then we ensure those constraints remain invariant.

With something like that in place the above code will still be valid, since the act of removing the counter will freeze all needed state to re-instate it.

Does that make sense?

#9 lore raw
In reply to Stephane Eranian
Quoted 3 lines · Stephane Eranian

Ok, so what you are suggesting is that the assignment is actually done incrementally in ->enable(). hw_group_sched_in() would simply validate that a single group is sane (i.e., can be scheduled if it was alone).

No, hw_group_sched_in needs to validate that this group can go on along with everything else that has already been enabled. But as I have said, if you have the complete list of enabled events to hand, that's not hard.

On the other hand, hw_perf_event_init does need to validate that a single group is sane by itself.

Paul.

#10 lore raw
In reply to Peter Zijlstra
Quoted 14 lines · Peter Zijlstra

I'm not really seeing the problem here...

perf_disable() <-- shut down the full pmu

pmu->disable() <-- hey someone got removed (easy free the reg) pmu->enable() <-- hey someone got added (harder, check constraints)

hw_perf_group_sched_in() <-- hey a full group got added

                              (better than multiple ->enable)

perf_enable() <-- re-enable pmu

Quoted 7 lines

So ->disable() is used to track freeing, ->enable is used to add individual counters, check constraints etc..

hw_perf_group_sched_in() is used to optimize the full group enable.

Afaict that is what power does (Paul?) and that should I think be sufficient to track x86 as well.

That sounds right to me.

Quoted 5 lines

Since sched_in() is balanced with sched_out(), the ->disable() calls should provide the required information as to the occupation of the pmu. I don't see the need for more hooks.

Paul, could you comment, since you did all this for power?

On powerpc we maintain a list of currently enabled events in the arch code. Does x86 do that as well?

If you have the list (or array) of events easily accessible, it's relatively easy to check whether the whole set is feasible at any point, without worrying about which events were recently added. The perf_event structure has a spot where the arch code can store which PMU register is used for that event, so you can easily optimize the case where the event doesn't move.

Like you, I'm not seeing where the difficulty lies. Perhaps Stephane could give us a detailed example if he still thinks there's a difficulty.

Paul.

#11 lore raw

Paul,

So if I understand what both of you are saying, it seems that event scheduling has to take place in the pmu->enable() callback which is per-event.

In the case of X86, you can chose to do a best-effort scheduling, i.e., only assign
the new event if there is a compatible free counter. That would be incremental.

But the better solution would be to re-examine the whole situation and potentially
move existing enabled events around to free a counter if the new event is more constrained. That would require stopping the PMU, rewriting config and data registers
and re-enabling the PMU. This latter solution is the only possibility to avoid ordering
side effects, i.e., the assignment of events to counters depends on the order in which
events are created (or enabled).

The register can be considered freed by pmu->disable() if scheduling takes place in pmu->enable().

Quoted 1 line

From what Paul was saying about hw_perf_group_sched_in(), it seems like this

function can be used to check if a new group is compatible with the existing enabled events. Compatible means that there is a possible assignment of events to counters.

As for the n_added logic, it seems like perf_disable() resets n_added to zero. N_added is incremented in pmu->enable(), i.e., add one event, or the hw_perf_group_sched_in(), i.e., add a whole group. Scheduling is based on n_events. The point of n_added is to verify whether something needs to be done, i.e., event scheduling, if an event or group was added between perf_disable()
and perf_enable(). In pmu->disable(), the list of enabled events is compacted and
n_events is decremented.

Did I get this right?

All the enable and disable calls can be called from NMI interrupt context and thus must be very careful with locks.

Quoted 48 lines · Paul Mackerras
Quoted 14 lines · Peter Zijlstra

I'm not really seeing the problem here...

 perf_disable() <-- shut down the full pmu

 pmu->disable() <-- hey someone got removed (easy free the reg) pmu->enable()  <-- hey someone got added (harder, check constraints)

 hw_perf_group_sched_in() <-- hey a full group got added (better than multiple ->enable)

 perf_enable() <-- re-enable pmu

Quoted 7 lines

So ->disable() is used to track freeing, ->enable is used to add individual counters, check constraints etc..

hw_perf_group_sched_in() is used to optimize the full group enable.

Afaict that is what power does (Paul?) and that should I think be sufficient to track x86 as well.

That sounds right to me.

Quoted 5 lines

Since sched_in() is balanced with sched_out(), the ->disable() calls should provide the required information as to the occupation of the pmu. I don't see the need for more hooks.

Paul, could you comment, since you did all this for power?

On powerpc we maintain a list of currently enabled events in the arch code.  Does x86 do that as well?

If you have the list (or array) of events easily accessible, it's relatively easy to check whether the whole set is feasible at any point, without worrying about which events were recently added.  The perf_event structure has a spot where the arch code can store which PMU register is used for that event, so you can easily optimize the case where the event doesn't move.

Like you, I'm not seeing where the difficulty lies.  Perhaps Stephane could give us a detailed example if he still thinks there's a difficulty.

Paul.

Signature · 8 lines
Stephane Eranian  | EMEA Software Engineering
Google France | 38 avenue de l'Opéra | 75002 Paris
Tel : +33 (0) 1 42 68 53 00
This email may be confidential or privileged. If you received this
communication by mistake, please
don't forward it to anyone else, please erase all copies and
attachments, and please let me know that
it went to the wrong person. Thanks
#12 lore raw

Stephane,

Quoted 3 lines

So if I understand what both of you are saying, it seems that event scheduling has to take place in the pmu->enable() callback which is per-event.

How much you have to do in the pmu->enable() function depends on whether the PMU is already stopped (via a call hw_perf_disable()) or not. If it is already stopped you don't have to do the full event scheduling computation, you only have to do enough to work out if the current set of events plus the event being enabled is feasible, and record the event for later. In other words, you can defer the actual scheduling until hw_perf_enable() time. Most calls to the pmu->enable() function are with the PMU already stopped, so it's a worthwhile optimization.

Quoted 4 lines

In the case of X86, you can chose to do a best-effort scheduling, i.e., only assign
the new event if there is a compatible free counter. That would be incremental.

Quoted 10 lines

But the better solution would be to re-examine the whole situation and potentially
move existing enabled events around to free a counter if the new event is more constrained. That would require stopping the PMU, rewriting config and data registers
and re-enabling the PMU. This latter solution is the only possibility to avoid ordering
side effects, i.e., the assignment of events to counters depends on the order in which
events are created (or enabled).

On powerpc, the pmu->enable() function always stops the PMU if it wasn't already stopped. That simplifies the code a lot because it means that I can do all the actual event scheduling (i.e. deciding on which event goes on which counter and working out PMU control register values) in hw_perf_enable().

Quoted 7 lines

The register can be considered freed by pmu->disable() if scheduling takes place in pmu->enable().

Quoted 1 line

From what Paul was saying about hw_perf_group_sched_in(), it seems like this

function can be used to check if a new group is compatible with the existing enabled events. Compatible means that there is a possible assignment of events to counters.

The hw_perf_group_sched_in() function is an optimization so I can add all the counters in the group to the set under consideration and then do just one feasibility check, rather than adding each counter in the group in turn and doing the feasibility check for each one.

Quoted 11 lines

As for the n_added logic, it seems like perf_disable() resets n_added to zero. N_added is incremented in pmu->enable(), i.e., add one event, or the hw_perf_group_sched_in(), i.e., add a whole group. Scheduling is based on n_events. The point of n_added is to verify whether something needs to be done, i.e., event scheduling, if an event or group was added between perf_disable()
and perf_enable(). In pmu->disable(), the list of enabled events is compacted and
n_events is decremented.

Did I get this right?

The main point of n_added was so that hw_perf_enable() could know whether the current set of events is a subset of the last set. If it is a subset, the scheduling decisions are much simpler.

Quoted 2 lines

All the enable and disable calls can be called from NMI interrupt context and thus must be very careful with locks.

I didn't think the pmu->enable() and pmu->disable() functions could be called from NMI context. Also, I would expect that if hw_perf_enable and hw_perf_disable are called from NMI context, that the calls would be balanced. That simplifies things a lot.

Paul.

maintainer
#13 lore raw
Quoted 3 lines · Paul Mackerras

On powerpc, the pmu->enable() function always stops the PMU if it wasn't already stopped.

I just looked at the generic code and it looks like kernel/perf_event.c always calls perf_disable() before calling ->enable().

maintainer
#14 lore raw
In reply to Paul Mackerras
Quoted 6 lines · Paul Mackerras
Quoted 3 lines

All the enable and disable calls can be called from NMI interrupt context and thus must be very careful with locks.

I didn't think the pmu->enable() and pmu->disable() functions could be called from NMI context.

I don't think they're called from NMI context either, most certainly not from the generic code.

The x86 calls the raw disable from nmi to throttle the counter, but all that (should) do is disable that counter, which is limited to a single msr write. After that it schedules a full disable by sending a self-ipi.

#15 lore raw

Hi,

Ok, so I made some progress yesterday on all of this.

The key elements are:
- pmu->enable() is always called from generic with PMU disabled
- pmu->disable() is called with PMU possibly enabled
- hw_perf_group_sched_in() is always called with PMU disabled

I got the n_added logic working now on X86.

I noticed the difference in pmu->enabled() between Power and X86. On PPC, you disable the whole PMU. On X86, that's not the case.

Now, I do the scheduling in hw_perf_enable(). Just like on PPC, I also move events around if their register assignment has changed. It is not quite working yet. I must have something wrong with the read and rewrite code.

I will experiment with pmu->enable(). Given the key elements above, I think Paul is right, all scheduling can be deferred until hw_perf_enable().

But there is a catch. I noticed that hw_perf_enable() is void. In other words, it
means that if scheduling fails, you won't notice. This is not a problem on PPC but will be on AMD64. That's because the scheduling depends on what goes on on the other cores on the socket. In other words, things can change between pmu->enable()/hw_perf_group_sched_in() and hw_perf_enable(). Unless we lock something down in between.

Quoted 18 lines · Peter Zijlstra
Quoted 6 lines · Paul Mackerras
Quoted 3 lines

All the enable and disable calls can be called from NMI interrupt context and thus must be very careful with locks.

I didn't think the pmu->enable() and pmu->disable() functions could be called from NMI context.

I don't think they're called from NMI context either, most certainly not from the generic code.

The x86 calls the raw disable from nmi to throttle the counter, but all that (should) do is disable that counter, which is limited to a single msr write. After that it schedules a full disable by sending a self-ipi.

Signature · 8 lines
Stephane Eranian  | EMEA Software Engineering
Google France | 38 avenue de l'Opéra | 75002 Paris
Tel : +33 (0) 1 42 68 53 00
This email may be confidential or privileged. If you received this
communication by mistake, please
don't forward it to anyone else, please erase all copies and
attachments, and please let me know that
it went to the wrong person. Thanks
maintainer
#16 lore raw
Quoted 21 lines · Stephane Eranian

Ok, so I made some progress yesterday on all of this.

The key elements are:
- pmu->enable() is always called from generic with PMU disabled
- pmu->disable() is called with PMU possibly enabled
- hw_perf_group_sched_in() is always called with PMU disabled

I got the n_added logic working now on X86.

I noticed the difference in pmu->enabled() between Power and X86. On PPC, you disable the whole PMU. On X86, that's not the case.

Now, I do the scheduling in hw_perf_enable(). Just like on PPC, I also move events around if their register assignment has changed. It is not quite working yet. I must have something wrong with the read and rewrite code.

I will experiment with pmu->enable(). Given the key elements above, I think Paul is right, all scheduling can be deferred until hw_perf_enable().

Quoted 7 lines

But there is a catch. I noticed that hw_perf_enable() is void. In other words, it
means that if scheduling fails, you won't notice. This is not a problem on PPC but will be on AMD64. That's because the scheduling depends on what goes on on the other cores on the socket. In other words, things can change between pmu->enable()/hw_perf_group_sched_in() and hw_perf_enable(). Unless we lock something down in between.

You have to lock stuff, you can't fail hw_perf_enable() because at that point we've lost all track of what failed.

#17 lore raw
In reply to stephane eranian
Quoted 7 lines · stephane eranian

There is a major difference between PPC and X86 here. PPC has a centralized register to control start/stop. This register uses bitmask to enable or disable counters. Thus, in hw_perf_enable(), if n_added=0, then you just need to
use the pre-computed bitmask. Otherwise, you need to recompute the bitmask to include the new registers. The assignment of events and validation is done in hw_group_sched_in().

That's not entirely accurate. Yes there is a global start/stop bit, but there isn't a bitmask to enable or disable counters. There is a selector bitfield for each counter (except the limited-function counters) and you can set the selector to the 'count nothing' value if you don't want a particular counter to count.

Validation is done in hw_group_sched_in() but not the assignment of events to counters. That's done in hw_perf_enable(), via the model-specific ppmu->compute_mmcr() call.

Quoted 5 lines

In X86, assignment and validation is done in hw_group_sched_in(). Activation is done individually for each counter. There is no centralized register used here, thus
no bitmask to update.

Quoted 7 lines

Disabling a counter does not trigger a complete reschedule of events. This happens
only when hw_group_sched_in() is called.

The n_events = 0 in hw_perf_disable() is used to signal that something is changing.
It should not be here but here.

The meaning of "It should not be here but here" is quite unclear to me.

Quoted 11 lines

The problem is that
hw_group_sched_in() needs a way
to know that it is called for a completely new series of group scheduling so it can
discard any previous assignment. This goes back to the issue I raised in my previous
email. You could add a parameter to hw_group_sched_in() that would indicate this is
the first group. that would cause n_events =0 and the function would start accumulating
events for the new scheduling period.

I don't think hw_group_sched_in is ever called for a completely new series of group scheduling. If you have per-cpu counters active, they don't get scheduled out and in again with each task switch. So you will tend to get a hw_pmu_disable call, then a series of disable calls for the per-task events for the old task, then a series of hw_group_sched_in calls for the per-task events for the new task, then a hw_pmu_enable call.

On powerpc we maintain an array with pointers to all the currently active events. That makes it easy to know at hw_pmu_enable() time what events need to be put on the PMU. Also it means that at hw_group_sched_in time you can look at the whole set of events, including the ones just added, to see if it's feasible to put them all on. At that point we just check feasibility, which is quite quick and easy using the bit-vector encoding of constraints. The bit-vector encoding lets us represent multiple constraints of various forms in one pair of 64-bit values per event. We can express constraints such as "you can have at most N events in a class X" or "you can't have events in all of classes A, B, C and D" or "control register bitfield X must be set to Y", and then check that a set of events satisfies all the constraints with some simple integer arithmetic. I don't know exactly what constraints you have on x86 but I would be surprised if you couldn't handle them the same way.

Paul.

#18 lore raw
In reply to stephane eranian
Quoted 12 lines · stephane eranian
Quoted 12 lines

arch/x86/kernel/cpu/perf_event.c struct debug_store { · line 72

Quoted 8 lines
arch/x86/kernel/cpu/perf_event.c +1 -0
@@ -68,6 +69,15 @@ struct debug_store {
       u64     pebs_event_reset[MAX_PEBS_EVENTS];
  };
 
+#define BITS_TO_U64(nr) DIV_ROUND_UP(nr, BITS_PER_BYTE * sizeof(u64))

Do we need this, is it realistic to expect X86_PMC_IDX_MAX to be anything else than 64?

Quoted 6 lines

The issue had to do with i386 mode where long are 32 bits  < 64. And in particular with the initializer .idxmsk[0] = V. In the future we may exceed 32 registers. That means the initializer would have to change. But I guess we have quite some ways before this case is reached. So I will revert all of this to unsigned long.

Well, in fact we have to use u64 because you are already using register indexes > 32, e.g., for the Intel fixed counters which start at position 32. Using unsigned long would make the static initializer uglier in this case.