Skip to contentSkip to conversation

perf: Implement Nehalem uncore pmu

RFC PATCH Superseded Lin Ming 5 participants 33 messages Nov 21, 2010 – Dec 2, 2010

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Lin Ming

author
#0 lore raw

RFC PATCH v2 2/3 perf: Implement Nehalem uncore pmu

For the background of Nehalem uncore pmu, see Intel SDM Volume 3B "30.6.2 Performance Monitoring Facility in the Uncore"

1. data structure

struct node_hw_events {
        struct perf_event *events[X86_PMC_IDX_MAX];
        int n_events;
        struct spinlock lock;
};

struct node_hw_events is the per node structure.
"lock" protects add/delete events to uncore pmu.

TODO: this need to be converted to per-socket since "uncore" refers to subsystems in the physical processor package that are shared by multiple processor cores.

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

---
 arch/x86/include/asm/msr-index.h              |    1 +
 arch/x86/include/asm/perf_event.h             |    5 +
 arch/x86/kernel/cpu/Makefile                  |    1 +
 arch/x86/kernel/cpu/perf_event.c              |    6 +-
 arch/x86/kernel/cpu/perf_event_intel_uncore.c |  475 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_uncore.h |   87 +++++
 include/linux/perf_event.h                    |    1 +
 7 files changed, 571 insertions(+), 5 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.h
arch/x86/include/asm/msr-index.h +1 -0
index 6b89f5e..a1cc40b 100644
@@ -81,6 +81,7 @@
 #define DEBUGCTLMSR_BTS_OFF_OS		(1UL <<  9)
 #define DEBUGCTLMSR_BTS_OFF_USR		(1UL << 10)
 #define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI	(1UL << 11)
+#define DEBUGCTLMSR_ENABLE_UNCORE_PMI	(1UL << 13)
 
 #define MSR_IA32_MC0_CTL		0x00000400
 #define MSR_IA32_MC0_STATUS		0x00000401
arch/x86/include/asm/perf_event.h +5 -0
index 550e26b..9572166 100644
@@ -75,6 +75,10 @@ union cpuid10_edx {
 	unsigned int full;
 };
 
+struct pmu_nmi_state {
+	unsigned int	marked;
+	int		handled;
+};
 
 /*
  * Fixed-purpose performance events:
@@ -127,6 +131,7 @@ union cpuid10_edx {
 #ifdef CONFIG_PERF_EVENTS
 extern void init_hw_perf_events(void);
 extern void perf_events_lapic_init(void);
+extern void init_uncore_pmu(void);
 
 #define PERF_EVENT_INDEX_OFFSET			0
 
arch/x86/kernel/cpu/Makefile +1 -0
index 3f0ebe4..db4bf99 100644
@@ -27,6 +27,7 @@ obj-$(CONFIG_CPU_SUP_TRANSMETA_32)	+= transmeta.o
 obj-$(CONFIG_CPU_SUP_UMC_32)		+= umc.o
 
 obj-$(CONFIG_PERF_EVENTS)		+= perf_event.o
+obj-$(CONFIG_PERF_EVENTS)		+= perf_event_intel_uncore.o
 
 obj-$(CONFIG_X86_MCE)			+= mcheck/
 obj-$(CONFIG_MTRR)			+= mtrr/
arch/x86/kernel/cpu/perf_event.c +1 -5
index cb16b9c..81517bc 100644
@@ -1196,11 +1196,6 @@ void perf_events_lapic_init(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 }
 
-struct pmu_nmi_state {
-	unsigned int	marked;
-	int		handled;
-};
-
 static DEFINE_PER_CPU(struct pmu_nmi_state, pmu_nmi);
 
 static int __kprobes
@@ -1348,6 +1343,7 @@ void __init init_hw_perf_events(void)
 
 	switch (boot_cpu_data.x86_vendor) {
 	case X86_VENDOR_INTEL:
+		init_uncore_pmu();
 		err = intel_pmu_init();
 		break;
 	case X86_VENDOR_AMD:
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +475 -0

475 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +87 -0

87 lines not on this page · raw message

include/linux/perf_event.h +1 -0
index 40150f3..7acc40c 100644
@@ -32,6 +32,7 @@ enum perf_type_id {
 	PERF_TYPE_HW_CACHE			= 3,
 	PERF_TYPE_RAW				= 4,
 	PERF_TYPE_BREAKPOINT			= 5,
+	PERF_TYPE_UNCORE			= 6,
 
 	PERF_TYPE_MAX,				/* non-ABI */
 };
Signature · 1 line
1.7.2.3
#1 lore raw
Quoted 7 lines

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3

Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2 -0
+
+static struct node_hw_events *uncore_events[MAX_NUMNODES];

Don't declare static arrays with MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar (e.g. with alloc_percpu)

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 149

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5 -0
+	/*
+	 * The hw event starts counting from this event offset,
+	 * mark it to be able to extra future deltas:
+	 */
+	local64_set(&hwc->prev_count, (u64)-left);

Your use of local* seems dubious. That is only valid if it's really all on the same CPU. Is that really true?

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 187

Quoted 3 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3 -0
+static int uncore_pmu_add(struct perf_event *event, int flags)
+{
+	int node = numa_node_id();

this should be still package id

Review

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3 -0
+	/* Check CPUID signatures: 06_1AH, 06_1EH, 06_1FH */
+	model = eax.split.model | (eax.split.ext_model << 4);
+	if (eax.split.family != 6 || (model != 0x1A && model != 0x1E && model !=

0x1F))

(quoted diff) +1 -0
+		return;

You can just get that from boot_cpu_data, no need to call cpuid

arch/x86/kernel/cpu/perf_event_intel_uncore.h line 13

Quoted 13 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +13 -0
+#include <linux/perf_event.h>
+#include <linux/capability.h>
+#include <linux/notifier.h>
+#include <linux/hardirq.h>
+#include <linux/kprobes.h>
+#include <linux/module.h>
+#include <linux/kdebug.h>
+#include <linux/sched.h>
+#include <linux/uaccess.h>
+#include <linux/slab.h>
+#include <linux/highmem.h>
+#include <linux/cpu.h>
+#include <linux/bitops.h>

Do you really need all these includes?

-Andi

author
#2 lore raw
Quoted 11 lines · Andi Kleen
Quoted 6 lines

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

OK, I can use on ordinary interrupt handler here.

Quoted 6 lines

In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.

Quoted 6 lines

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3

Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2 -0
+
+static struct node_hw_events *uncore_events[MAX_NUMNODES];

Don't declare static arrays with MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar (e.g. with alloc_percpu)

I really need is a per physical cpu data here, is alloc_percpu enough?

Quoted 9 lines

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 149

Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5 -0
+	/*
+	 * The hw event starts counting from this event offset,
+	 * mark it to be able to extra future deltas:
+	 */
+	local64_set(&hwc->prev_count, (u64)-left);

Your use of local* seems dubious. That is only valid if it's really all on the same CPU. Is that really true?

Good catch! That is not true.

The interrupt handler is running on one core and the data(hwc->prev_count) maybe on another core.

Any idea to set this cross-core data?

Quoted 6 lines

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 187

Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3 -0
+static int uncore_pmu_add(struct perf_event *event, int flags)
+{
+	int node = numa_node_id();

this should be still package id

Understand, this is in my TODO.

Quoted 8 lines

Review

Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3 -0
+	/* Check CPUID signatures: 06_1AH, 06_1EH, 06_1FH */
+	model = eax.split.model | (eax.split.ext_model << 4);
+	if (eax.split.family != 6 || (model != 0x1A && model != 0x1E && model !=

0x1F))

(quoted diff) +1 -0
+		return;

You can just get that from boot_cpu_data, no need to call cpuid

Nice, will use it.

Quoted 16 lines
Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +3 -0
+#include <linux/perf_event.h>
+#include <linux/capability.h>
+#include <linux/notifier.h>

arch/x86/kernel/cpu/perf_event_intel_uncore.h line 13

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +10 -0
+#include <linux/hardirq.h>
+#include <linux/kprobes.h>
+#include <linux/module.h>
+#include <linux/kdebug.h>
+#include <linux/sched.h>
+#include <linux/uaccess.h>
+#include <linux/slab.h>
+#include <linux/highmem.h>
+#include <linux/cpu.h>
+#include <linux/bitops.h>

Do you really need all these includes?

Only

#include <linux/perf_event.h>
#include <linux/kprobes.h>
#include <linux/hardirq.h>
#include <linux/slab.h>

are needed.

Thanks for the comments.
Lin Ming

#3 lore raw

BTW another thing I noticed that if you ever add opcode/address matching you'll need to add the new parameters for the address at least to the input perf_event structure. opcode could in theory be encoded in the upper 32bits like offcore does, but address needs to be extra. It's only a small incremental step, but may make this more useful.

Quoted 6 lines
Quoted 4 lines

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

OK, I can use on ordinary interrupt handler here.

You'll need to allocate a vector, it shouldn't be too difficult.

Quoted 14 lines
Quoted 6 lines

In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.

Quoted 7 lines

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3

Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2 -0
+
+static struct node_hw_events *uncore_events[MAX_NUMNODES];

Don't declare static arrays with MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar
(e.g. with alloc_percpu)

Quoted 1 line

I really need is a per physical cpu data here, is alloc_percpu enough?

If you use a per cpu array then each CPU can carry a pointer to its per socket data structure.

This could use a similar scheme as the per core data I submitted recently.

Quoted 2 lines

Any idea to set this cross-core data?

s/local/atomic/

But if it's just stores/loads without read-modify-write you can just use normal stores.

Quoted 9 lines
Quoted 7 lines

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 187

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3 -0
+static int uncore_pmu_add(struct perf_event *event, int flags)
+{
+	int node = numa_node_id();

this should be still package id

Understand, this is in my TODO.

With the per cpu pointer scheme you likely don't even need it, just check the topology at set up time (similar as in my patch, just using the package)

-Andi

#4 lore raw
In reply to Lin Ming
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

Quoted 6 lines
Quoted 4 lines

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

OK, I can use on ordinary interrupt handler here.

Does the hardware actually allow using a different interrupt source?

Quoted 6 lines
Quoted 6 lines

In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.

Quoted 8 lines
Quoted 6 lines

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3

Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2 -0
+
+static struct node_hw_events *uncore_events[MAX_NUMNODES];

Don't declare static arrays with MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar (e.g. with alloc_percpu)

I really need is a per physical cpu data here, is alloc_percpu enough?

Nah, simply manually allocate bits using kmalloc_node(), that's something I still need to fix in Andi's patches as well.

Quoted 9 lines
Quoted 8 lines

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 149

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5 -0
+	/*
+	 * The hw event starts counting from this event offset,
+	 * mark it to be able to extra future deltas:
+	 */
+	local64_set(&hwc->prev_count, (u64)-left);

Your use of local* seems dubious. That is only valid if it's really all on the same CPU. Is that really true?

Quoted 6 lines

Good catch! That is not true.

The interrupt handler is running on one core and the data(hwc->prev_count) maybe on another core.

Any idea to set this cross-core data?

IIRC you can steer the uncore interrupts (it has a mask somewhere) simply steer everything to the first cpu in the nodemask?

#5 lore raw
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

Quoted 9 lines
Quoted 6 lines
Quoted 4 lines

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

OK, I can use on ordinary interrupt handler here.

Does the hardware actually allow using a different interrupt source?

It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.

author
#6 lore raw
Quoted 15 lines · Stephane Eranian
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Lin Ming
Quoted 11 lines · Andi Kleen
Quoted 6 lines

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

OK, I can use on ordinary interrupt handler here.

Does the hardware actually allow using a different interrupt source?

It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.

If only counting(perf stat) makes sense for uncore events, do we still need an interrupt handler?

48 bits counter is not that easy to overflow in practice.

#7 lore raw
Quoted 20 lines · Lin Ming
Quoted 19 lines · Stephane Eranian
Quoted 18 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

Quoted 6 lines
Quoted 4 lines

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

OK, I can use on ordinary interrupt handler here.

Does the hardware actually allow using a different interrupt source?

Quoted 7 lines
Quoted 4 lines

It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.

If only counting(perf stat) makes sense for uncore events, do we still need an interrupt handler?

Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package, I bet there are people who will run just one task on a package as well.

Just because it doesn't make sense in general doesn't mean there isn't anybody who'd want to do it and actually knows wth he's doing.

Quoted 1 line

48 bits counter is not that easy to overflow in practice.

Still..

author
#8 lore raw
Quoted 34 lines · Peter Zijlstra
Quoted 20 lines · Lin Ming
Quoted 19 lines · Stephane Eranian
Quoted 18 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

Quoted 6 lines
Quoted 4 lines

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

OK, I can use on ordinary interrupt handler here.

Does the hardware actually allow using a different interrupt source?

Quoted 7 lines
Quoted 4 lines

It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.

If only counting(perf stat) makes sense for uncore events, do we still need an interrupt handler?

Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package, I bet there are people who will run just one task on a package as well.

Quoted 6 lines

Just because it doesn't make sense in general doesn't mean there isn't anybody who'd want to do it and actually knows wth he's doing.

Quoted 1 line

48 bits counter is not that easy to overflow in practice.

Still..

OK, will do more tests, then send out a new version.

Thanks.

#9 lore raw
In reply to Peter Zijlstra
Quoted 34 lines · Peter Zijlstra
Quoted 20 lines · Lin Ming
Quoted 19 lines · Stephane Eranian
Quoted 18 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

Quoted 6 lines
Quoted 4 lines

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

OK, I can use on ordinary interrupt handler here.

Does the hardware actually allow using a different interrupt source?

Quoted 7 lines
Quoted 4 lines

It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.

If only counting(perf stat) makes sense for uncore events, do we still need an interrupt handler?

Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package, I bet there are people who will run just one task on a package as well.

Quoted 7 lines

Just because it doesn't make sense in general doesn't mean there isn't anybody who'd want to do it and actually knows wth he's doing.

Quoted 1 line

48 bits counter is not that easy to overflow in practice.

Still..

Agreed.

#10 lore raw
In reply to Peter Zijlstra
Quoted 3 lines

Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package,

Assuming they don't have any active PCI devices either.

Quoted 2 lines

I bet there are people who will run just one task on a package as well.

In that case the sampling has a 1/NUM-CPU-THREADS-IN-PACKAGE chance to report the right task (or actually somewhat less because the measurement skew for uncore is much higher than for normal events)

Really for per core measurements using the OFFCORE events is much better.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#11 lore raw
Quoted 6 lines · Andi Kleen
Quoted 3 lines

Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package,

Assuming they don't have any active PCI devices either.

Good point.

Quoted 9 lines
Quoted 2 lines

I bet there are people who will run just one task on a package as well.

In that case the sampling has a 1/NUM-CPU-THREADS-IN-PACKAGE chance to report the right task (or actually somewhat less because the measurement skew for uncore is much higher than for normal events)

Really for per core measurements using the OFFCORE events is much better.

yes, OFFCORE_RESPONSE is much more useful.

author
#12 lore raw
In reply to Peter Zijlstra
Quoted 31 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines

2. Uncore pmu NMI handling

All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.

Quoted 6 lines
Quoted 4 lines

Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.

OK, I can use on ordinary interrupt handler here.

Does the hardware actually allow using a different interrupt source?

Quoted 13 lines
Quoted 12 lines

In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3

Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2 -0
+
+static struct node_hw_events *uncore_events[MAX_NUMNODES];

Don't declare static arrays with MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar (e.g. with alloc_percpu)

Quoted 4 lines
Quoted 1 line

I really need is a per physical cpu data here, is alloc_percpu enough?

Nah, simply manually allocate bits using kmalloc_node(), that's something I still need to fix in Andi's patches as well.

I'm writing this like AMD NB events allocation.

Thanks,
Lin Ming

Quoted 22 lines
Quoted 10 lines
Quoted 9 lines

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 149

Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5 -0
+	/*
+	 * The hw event starts counting from this event offset,
+	 * mark it to be able to extra future deltas:
+	 */
+	local64_set(&hwc->prev_count, (u64)-left);

Your use of local* seems dubious. That is only valid if it's really all on the same CPU. Is that really true?

Quoted 6 lines

Good catch! That is not true.

The interrupt handler is running on one core and the data(hwc->prev_count) maybe on another core.

Any idea to set this cross-core data?

IIRC you can steer the uncore interrupts (it has a mask somewhere) simply steer everything to the first cpu in the nodemask?

#13 lore raw
In reply to Lin Ming

Lin,

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48

Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8 -0
+static void uncore_pmu_enable_all(void)
+{
+       u64 ctrl;
+
+       /*
+        * (0xFULL << 48): 1 of the 4 cores can receive NMI each time
+        * but we don't know which core will receive the NMI when overflow happens
+        */

That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true in uncore_handle_irq().

I also think that if you want all processors to receive the interrupts, then the mask should
be 0xff when HT is on. The manual is rather obscure on this, but it does make sense.

Quoted 12 lines

+       ctrl = ((1 << UNCORE_NUM_GENERAL_COUNTERS) - 1) | (0xFULL << 48);
+       ctrl |= MSR_UNCORE_PERF_GLOBAL_CTRL_EN_FC0;

arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +10 -0
+
+       /*
+        * Freeze the uncore pmu on overflow of any uncore counter.
+        * This makes unocre NMI handling easier.
+        */
+       ctrl |= MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_FRZ;
+
+       wrmsrl(MSR_UNCORE_PERF_GLOBAL_CTRL, ctrl);
+}
+
author
#14 lore raw
Quoted 12 lines · Stephane Eranian

Lin,

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48

Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8 -0
+static void uncore_pmu_enable_all(void)
+{
+       u64 ctrl;
+
+       /*
+        * (0xFULL << 48): 1 of the 4 cores can receive NMI each time
+        * but we don't know which core will receive the NMI when overflow happens
+        */
Quoted 8 lines

That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true in uncore_handle_irq().

I thought it's a broadcast too in the v1 patches, let me double check it.

Quoted 5 lines

I also think that if you want all processors to receive the interrupts, then the mask should
be 0xff when HT is on. The manual is rather obscure on this, but it does make sense.

Kernel panics if 0xff is set, but it maybe bugs of my code.

Anyway, is it told in some errata that the mask should be 0xff when HT is on?

Thanks,
Lin Ming

Quoted 14 lines
Quoted 14 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +12 -0
+       ctrl = ((1 << UNCORE_NUM_GENERAL_COUNTERS) - 1) | (0xFULL << 48);
+       ctrl |= MSR_UNCORE_PERF_GLOBAL_CTRL_EN_FC0;
+
+       /*
+        * Freeze the uncore pmu on overflow of any uncore counter.
+        * This makes unocre NMI handling easier.
+        */
+       ctrl |= MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_FRZ;
+
+       wrmsrl(MSR_UNCORE_PERF_GLOBAL_CTRL, ctrl);
+}
+
#15 lore raw
In reply to Stephane Eranian
Quoted 12 lines · Stephane Eranian

Lin,

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48

Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8 -0
+static void uncore_pmu_enable_all(void)
+{
+       u64 ctrl;
+
+       /*
+        * (0xFULL << 48): 1 of the 4 cores can receive NMI each time
+        * but we don't know which core will receive the NMI when overflow happens
+        */
Quoted 8 lines

That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true in uncore_handle_irq().

Stephane,

The interrupt model is strange, it behaves differently when HT on/off.

If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.

If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.

Did you ever observe this?

Quoted 5 lines

I also think that if you want all processors to receive the interrupts, then the mask should
be 0xff when HT is on. The manual is rather obscure on this, but it does make sense.

I tried to set the mask 0xff when HT is on, but kernel panics, because the reserve bits are set.

Thanks,
Lin Ming

Quoted 15 lines
Quoted 4 lines

+       ctrl = ((1 << UNCORE_NUM_GENERAL_COUNTERS) - 1) | (0xFULL << 48);
+       ctrl |= MSR_UNCORE_PERF_GLOBAL_CTRL_EN_FC0;

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 60

Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +10 -0
+
+       /*
+        * Freeze the uncore pmu on overflow of any uncore counter.
+        * This makes unocre NMI handling easier.
+        */
+       ctrl |= MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_FRZ;
+
+       wrmsrl(MSR_UNCORE_PERF_GLOBAL_CTRL, ctrl);
+}
+

--

#16 lore raw
Quoted 22 lines · Lin Ming
Quoted 12 lines · Stephane Eranian

Lin,

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48

Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8 -0
+static void uncore_pmu_enable_all(void)
+{
+       u64 ctrl;
+
+       /*
+        * (0xFULL << 48): 1 of the 4 cores can receive NMI each time
+        * but we don't know which core will receive the NMI when overflow happens
+        */
Quoted 8 lines

That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true in uncore_handle_irq().

Quoted 6 lines

Stephane,

The interrupt model is strange, it behaves differently when HT on/off.

If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.

That's if yo set the mask to 0xf, right?

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Quoted 6 lines

If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.

Did you ever observe this?

No because I never set more than one bit in the mask.

Quoted 2 lines

I tried to set the mask 0xff when HT is on, but kernel panics, because the reserve bits are set.

Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.

Was that on Nehalem or Westmere?

#17 lore raw
Quoted 23 lines · Stephane Eranian
Quoted 22 lines · Lin Ming
Quoted 20 lines · Stephane Eranian

Lin,

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48

Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8 -0
+static void uncore_pmu_enable_all(void)
+{
+       u64 ctrl;
+
+       /*
+        * (0xFULL << 48): 1 of the 4 cores can receive NMI each time
+        * but we don't know which core will receive the NMI when overflow happens
+        */

That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true in uncore_handle_irq().

Quoted 7 lines
Quoted 6 lines

Stephane,

The interrupt model is strange, it behaves differently when HT on/off.

If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.

That's if yo set the mask to 0xf, right?

Right.

Quoted 5 lines

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

This is what my upcoming v3 patches doing.

Quoted 9 lines
Quoted 7 lines

If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.

Did you ever observe this?

No because I never set more than one bit in the mask.

Quoted 7 lines
Quoted 2 lines

I tried to set the mask 0xff when HT is on, but kernel panics, because the reserve bits are set.

Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.

Was that on Nehalem or Westmere?

Nehalem.

Thanks,
Lin Ming

#18 lore raw
In reply to Stephane Eranian

Lin,

Looked at the perfmon code, and it seems the mask is actual cores, not threads:

                rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
                val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id);
                wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);

That seems to imply both threads will get the interrupt.

In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.

On the cores where no uncore was programmed, then both threads will have a spurious interrupt.

That brings up back to the 'spurious interrupt' issue and the 'NMI Dazed' message
that Don tried to eliminate. Now we have a new situation where we will get interrupt
with no work to do, so the perf_event will pass the interrupt onto the next subsystem
and eventually we will get the 'dazed' message. I am just guessing here....

Quoted 52 lines · Stephane Eranian
Quoted 22 lines · Lin Ming
Quoted 12 lines · Stephane Eranian

Lin,

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48

Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8 -0
+static void uncore_pmu_enable_all(void)
+{
+       u64 ctrl;
+
+       /*
+        * (0xFULL << 48): 1 of the 4 cores can receive NMI each time
+        * but we don't know which core will receive the NMI when overflow happens
+        */
Quoted 8 lines

That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true in uncore_handle_irq().

Quoted 6 lines

Stephane,

The interrupt model is strange, it behaves differently when HT on/off.

If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.

That's if yo set the mask to 0xf, right?

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Quoted 6 lines

If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.

Did you ever observe this?

No because I never set more than one bit in the mask.

Quoted 2 lines

I tried to set the mask 0xff when HT is on, but kernel panics, because the reserve bits are set.

Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.

Was that on Nehalem or Westmere?

#19 lore raw
Quoted 8 lines · Stephane Eranian

Lin,

Looked at the perfmon code, and it seems the mask is actual cores, not threads:
               rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
               val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id); wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);

Quoted 7 lines

That seems to imply both threads will get the interrupt.

In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.

On the cores where no uncore was programmed, then both threads will have a spurious interrupt.

But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.

I'll do more tests to verify this.

Quoted 8 lines

That brings up back to the 'spurious interrupt' issue and the 'NMI Dazed' message
that Don tried to eliminate. Now we have a new situation where we will get interrupt
with no work to do, so the perf_event will pass the interrupt onto the next subsystem
and eventually we will get the 'dazed' message. I am just guessing here....

Add Don.

Thanks,
Lin Ming

Quoted 56 lines · Stephane Eranian
Quoted 52 lines · Lin Ming
Quoted 22 lines · Stephane Eranian
Quoted 12 lines

Lin,

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48

Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8 -0
+static void uncore_pmu_enable_all(void)
+{
+       u64 ctrl;
+
+       /*
+        * (0xFULL << 48): 1 of the 4 cores can receive NMI each time
+        * but we don't know which core will receive the NMI when overflow happens
+        */
Quoted 8 lines

That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true in uncore_handle_irq().

Quoted 6 lines

Stephane,

The interrupt model is strange, it behaves differently when HT on/off.

If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.

That's if yo set the mask to 0xf, right?

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Quoted 6 lines

If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.

Did you ever observe this?

No because I never set more than one bit in the mask.

Quoted 2 lines

I tried to set the mask 0xff when HT is on, but kernel panics, because the reserve bits are set.

Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.

Was that on Nehalem or Westmere?

#20 lore raw
Quoted 14 lines · Lin Ming
Quoted 13 lines · Stephane Eranian

Lin,

Looked at the perfmon code, and it seems the mask is actual cores, not threads:
               rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
               val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id); wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);

That seems to imply both threads will get the interrupt.

In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.

Quoted 8 lines
Quoted 2 lines

On the cores where no uncore was programmed, then both threads will have a spurious interrupt.

But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.

The MSR_NHM_UNC_GLOBAL_CTRL is per socket not per core.

Quoted 1 line

I'll do more tests to verify this.

In your tests, are your programming the same uncore event across all CPUs? If so then you may have a race condition setting the MSR because it read-modify-write.

What about you program only one uncore event from one CPU?

Quoted 72 lines
Quoted 9 lines

That brings up back to the 'spurious interrupt' issue and the 'NMI Dazed' message
that Don tried to eliminate. Now we have a new situation where we will get interrupt
with no work to do, so the perf_event will pass the interrupt onto the next subsystem
and eventually we will get the 'dazed' message. I am just guessing here....

Add Don.

Thanks,
Lin Ming

Quoted 56 lines · Stephane Eranian
Quoted 52 lines · Lin Ming
Quoted 22 lines · Stephane Eranian
Quoted 12 lines

Lin,

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48

Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8 -0
+static void uncore_pmu_enable_all(void)
+{
+       u64 ctrl;
+
+       /*
+        * (0xFULL << 48): 1 of the 4 cores can receive NMI each time
+        * but we don't know which core will receive the NMI when overflow happens
+        */
Quoted 8 lines

That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true in uncore_handle_irq().

Quoted 6 lines

Stephane,

The interrupt model is strange, it behaves differently when HT on/off.

If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.

That's if yo set the mask to 0xf, right?

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Quoted 6 lines

If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.

Did you ever observe this?

No because I never set more than one bit in the mask.

Quoted 2 lines

I tried to set the mask 0xff when HT is on, but kernel panics, because the reserve bits are set.

Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.

Was that on Nehalem or Westmere?

author
#21 lore raw
Quoted 18 lines · Stephane Eranian
Quoted 17 lines · Lin Ming
Quoted 15 lines · Stephane Eranian

Lin,

Looked at the perfmon code, and it seems the mask is actual cores, not threads:

               rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
               val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id);
               wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);

That seems to imply both threads will get the interrupt.

In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.

On the cores where no uncore was programmed, then both threads will have a spurious interrupt.

Quoted 6 lines
Quoted 5 lines

But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.

The MSR_NHM_UNC_GLOBAL_CTRL is per socket not per core.

Understood.

Quoted 8 lines
Quoted 2 lines

I'll do more tests to verify this.

In your tests, are your programming the same uncore event across all CPUs? If so then you may have a race condition setting the MSR because it read-modify-write.

What about you program only one uncore event from one CPU?

This is what I tested, programming only one uncore event from one CPU. When HT is off, all four cores in the socket receive the interrupt. When HT is on, only the 2 threads in one of the four cores receive the interrupt.

#22 lore raw
Quoted 36 lines · Lin Ming
Quoted 18 lines · Stephane Eranian
Quoted 17 lines · Lin Ming
Quoted 15 lines · Stephane Eranian

Lin,

Looked at the perfmon code, and it seems the mask is actual cores, not threads:
               rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
               val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id); wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);

That seems to imply both threads will get the interrupt.

In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.

On the cores where no uncore was programmed, then both threads will have a spurious interrupt.

Quoted 6 lines
Quoted 5 lines

But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.

The MSR_NHM_UNC_GLOBAL_CTRL is per socket not per core.

Understood.

Quoted 7 lines
Quoted 2 lines

I'll do more tests to verify this.

In your tests, are your programming the same uncore event across all CPUs? If so then you may have a race condition setting the MSR because it read-modify-write.

Quoted 5 lines
Quoted 1 line

What about you program only one uncore event from one CPU?

This is what I tested, programming only one uncore event from one CPU.

When HT is off, all four cores in the socket receive the interrupt.

If the value of the MSR is 0xf << 48?

Quoted 2 lines

When HT is on, only the 2 threads in one of the four cores receive the interrupt.

Something is not right here. Next week, I may be able to run some tests on a Nehalem using perfmon to compare. Could you also send me your latest uncore patch against tip-x86?
Thanks.

author
#23 lore raw
Quoted 37 lines · Stephane Eranian
Quoted 36 lines · Lin Ming
Quoted 18 lines · Stephane Eranian
Quoted 17 lines · Lin Ming
Quoted 15 lines · Stephane Eranian

Lin,

Looked at the perfmon code, and it seems the mask is actual cores, not threads:

               rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
               val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id);
               wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);

That seems to imply both threads will get the interrupt.

In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.

On the cores where no uncore was programmed, then both threads will have a spurious interrupt.

Quoted 6 lines
Quoted 5 lines

But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.

The MSR_NHM_UNC_GLOBAL_CTRL is per socket not per core.

Understood.

Quoted 7 lines
Quoted 2 lines

I'll do more tests to verify this.

In your tests, are your programming the same uncore event across all CPUs? If so then you may have a race condition setting the MSR because it read-modify-write.

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

What about you program only one uncore event from one CPU?

This is what I tested, programming only one uncore event from one CPU.

When HT is off, all four cores in the socket receive the interrupt.

If the value of the MSR is 0xf << 48?

Yes, the EN_PMI_CORE* bits are set to 0xf.

Quoted 7 lines
Quoted 3 lines

When HT is on, only the 2 threads in one of the four cores receive the interrupt.

Something is not right here. Next week, I may be able to run some tests on a Nehalem using perfmon to compare. Could you also send me your latest uncore patch against tip-x86?
Thanks.

I just send it out.
http://lkml.org/lkml/2010/12/2/4

Thanks,
Lin Ming

#24 lore raw
In reply to Stephane Eranian
Quoted 4 lines · Stephane Eranian

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Not really, you can redirect all these events to the first online cpu of the node.

You can re-write event->cpu in pmu::event_init(), and register cpu hotplug notifiers to migrate the state around.

#25 lore raw
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Quoted 6 lines

Not really, you can redirect all these events to the first online cpu of the node.

You can re-write event->cpu in pmu::event_init(), and register cpu hotplug notifiers to migrate the state around.

I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.

#26 lore raw
Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Not really, you can redirect all these events to the first online cpu of the node.

Quoted 6 lines
Quoted 3 lines

You can re-write event->cpu in pmu::event_init(), and register cpu hotplug notifiers to migrate the state around.

I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.

Well, its either that or keeping a node wide state like we do for AMD and serialize everything from there.

And I'm not sure what's most expensive, steering the interrupt to one core only, or broadcasting every interrupt, I'd favour the first approach.

The whole thing is a node-wide resource, so the user needs to think in nodes anyway, we already do a cpu->node mapping for identifying the thing.

#27 lore raw
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Not really, you can redirect all these events to the first online cpu of the node.

Quoted 6 lines
Quoted 3 lines

You can re-write event->cpu in pmu::event_init(), and register cpu hotplug notifiers to migrate the state around.

I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.

Quoted 6 lines

Well, its either that or keeping a node wide state like we do for AMD and serialize everything from there.

And I'm not sure what's most expensive, steering the interrupt to one core only, or broadcasting every interrupt, I'd favour the first approach.

I think the one core-only approach will limit the spurious interrupt aspect. In perfmon, that's how I had it setup. The first CPU where uncore is accessed owns the uncore PMU for the socket, thus all interrupts are routed there. What you are proposing is the same. Now you can chose you hardcode which is the default core to handle this, or (better) you use the first core that accesses uncore.

Quoted 5 lines

The whole thing is a node-wide resource, so the user needs to think in nodes anyway, we already do a cpu->node mapping for identifying the thing.

Agreed.

#28 lore raw
Quoted 35 lines · Stephane Eranian
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Not really, you can redirect all these events to the first online cpu of the node.

Quoted 6 lines
Quoted 3 lines

You can re-write event->cpu in pmu::event_init(), and register cpu hotplug notifiers to migrate the state around.

I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.

Quoted 6 lines

Well, its either that or keeping a node wide state like we do for AMD and serialize everything from there.

And I'm not sure what's most expensive, steering the interrupt to one core only, or broadcasting every interrupt, I'd favour the first approach.

I think the one core-only approach will limit the spurious interrupt aspect. In perfmon, that's how I had it setup. The first CPU where uncore is accessed owns the uncore PMU for the socket, thus all interrupts are routed there. What you are proposing is the same. Now you can chose you hardcode which is the default core to handle this, or (better) you use the first core that accesses uncore.

Quoted 1 line
Quoted 6 lines
Quoted 4 lines

The whole thing is a node-wide resource, so the user needs to think in nodes anyway, we already do a cpu->node mapping for identifying the thing.

Agreed.

Hi, all

Thanks for all the comments.
I'm on travel Nov 27 to Nov 30.

I'll address the comments when I'm back.

Thanks,
Lin Ming

author
#29 lore raw
In reply to Peter Zijlstra
Quoted 22 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.

Not really, you can redirect all these events to the first online cpu of the node.

Quoted 6 lines
Quoted 3 lines

You can re-write event->cpu in pmu::event_init(), and register cpu hotplug notifiers to migrate the state around.

I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.

Well, its either that or keeping a node wide state like we do for AMD and serialize everything from there.

Quoted 7 lines

And I'm not sure what's most expensive, steering the interrupt to one core only, or broadcasting every interrupt, I'd favour the first approach.

The whole thing is a node-wide resource, so the user needs to think in nodes anyway, we already do a cpu->node mapping for identifying the thing.

How about a new sub-command for node-wide events statistics?

perf node -n <node> -e <event>?

#30 lore raw
Quoted 3 lines · Lin Ming

How about a new sub-command for node-wide events statistics?

perf node -n <node> -e <event>?

Maybe as a very slim wrapper around perf stat, but personally I wouldn't care.

#31 lore raw
In reply to Lin Ming
Quoted 3 lines

How about a new sub-command for node-wide events statistics?

perf node -n <node> -e <event>?

Seems like the best option to me (and not allowing the uncore events for the other commands)

But I don't like "node" because in a non NUMA kernel you won't have nodes, but this is still useful. Or with NUMA emulation you may have nodes that doesn't match the sockets.

Maybe perf package or perf socket ?

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
#32 lore raw
Quoted 3 lines · Andi Kleen

Seems like the best option to me (and not allowing the uncore events for the other commands)

Andi, I don't care what you think, that's simply not going to happen.