perf: Implement Nehalem uncore pmu
Subsystems (2)
RFC PATCH v2 2/3 perf: Implement Nehalem uncore pmu
For the background of Nehalem uncore pmu, see Intel SDM Volume 3B "30.6.2 Performance Monitoring Facility in the Uncore"
1. data structure
struct node_hw_events {
struct perf_event *events[X86_PMC_IDX_MAX];
int n_events;
struct spinlock lock;
};struct node_hw_events is the per node structure.
"lock" protects add/delete events to uncore pmu.
TODO: this need to be converted to per-socket since "uncore" refers to subsystems in the physical processor package that are shared by multiple processor cores.
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
--- arch/x86/include/asm/msr-index.h | 1 + arch/x86/include/asm/perf_event.h | 5 + arch/x86/kernel/cpu/Makefile | 1 + arch/x86/kernel/cpu/perf_event.c | 6 +- arch/x86/kernel/cpu/perf_event_intel_uncore.c | 475 +++++++++++++++++++++++++ arch/x86/kernel/cpu/perf_event_intel_uncore.h | 87 +++++ include/linux/perf_event.h | 1 + 7 files changed, 571 insertions(+), 5 deletions(-) create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.c create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.h
arch/x86/include/asm/msr-index.h +1 -0
@@ -81,6 +81,7 @@#define DEBUGCTLMSR_BTS_OFF_OS (1UL << 9)#define DEBUGCTLMSR_BTS_OFF_USR (1UL << 10)#define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI (1UL << 11) +#define DEBUGCTLMSR_ENABLE_UNCORE_PMI (1UL << 13)#define MSR_IA32_MC0_CTL 0x00000400#define MSR_IA32_MC0_STATUS 0x00000401
arch/x86/include/asm/perf_event.h +5 -0
@@ -75,6 +75,10 @@ union cpuid10_edx {unsigned int full; }; +struct pmu_nmi_state { +unsigned int marked; +int handled; +};/* * Fixed-purpose performance events:
@@ -127,6 +131,7 @@ union cpuid10_edx {#ifdef CONFIG_PERF_EVENTS extern void init_hw_perf_events (void );extern void perf_events_lapic_init (void ); +extern void init_uncore_pmu (void );#define PERF_EVENT_INDEX_OFFSET 0
arch/x86/kernel/cpu/Makefile +1 -0
@@ -27,6 +27,7 @@ obj-$(CONFIG_CPU_SUP_TRANSMETA_32) += transmeta.o obj-$(CONFIG_CPU_SUP_UMC_32) += umc.o obj-$(CONFIG_PERF_EVENTS) += perf_event.o +obj-$(CONFIG_PERF_EVENTS) += perf_event_intel_uncore.o obj-$(CONFIG_X86_MCE) += mcheck/ obj-$(CONFIG_MTRR) += mtrr/
arch/x86/kernel/cpu/perf_event.c +1 -5
@@ -1196,11 +1196,6 @@ void perf_events_lapic_init(void)apic_write (APIC_LVTPC ,APIC_DM_NMI ); } -struct pmu_nmi_state { -unsigned int marked; -int handled; -}; - static DEFINE_PER_CPU(struct pmu_nmi_state, pmu_nmi); static int __kprobes
@@ -1348,6 +1343,7 @@ void __init init_hw_perf_events(void)switch (boot_cpu_data.x86_vendor) {case X86_VENDOR_INTEL : +init_uncore_pmu (); err =intel_pmu_init ();break ;case X86_VENDOR_AMD :
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +475 -0
475 lines not on this page · raw message
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +87 -0
87 lines not on this page · raw message
include/linux/perf_event.h +1 -0
@@ -32,6 +32,7 @@ enum perf_type_id {PERF_TYPE_HW_CACHE =3 ,PERF_TYPE_RAW =4 ,PERF_TYPE_BREAKPOINT =5 , +PERF_TYPE_UNCORE =6 , PERF_TYPE_MAX, /* non-ABI */ };
Signature · 1 line
1.7.2.3
Quoted 7 lines
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2
-0+ +static struct node_hw_events *uncore_events[MAX_NUMNODES ];
Don't declare static arrays with MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar (e.g. with alloc_percpu)
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 149
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5
-0+/* +* The hw event starts counting from this event offset, +* mark it to be able to extra future deltas: +*/ +local64_set (&hwc->prev_count, (u64)-left);
Your use of local* seems dubious. That is only valid if it's really all on the same CPU. Is that really true?
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 187
Quoted 3 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3
-0+static int uncore_pmu_add (struct perf_event *event,int flags) +{ +int node =numa_node_id ();
this should be still package id
Review
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3
-0+/* Check CPUID signatures: 06_1AH, 06_1EH, 06_1FH */ + model = eax.split.model | (eax.split.ext_model <<4 ); + if (eax.split.family != 6 || (model != 0x1A && model != 0x1E && model !=
0x1F))(quoted diff) +1
-0+ return;
You can just get that from boot_cpu_data, no need to call cpuid
arch/x86/kernel/cpu/perf_event_intel_uncore.h line 13
Quoted 13 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +13
-0+#include <linux/perf_event.h> +#include <linux/capability.h> +#include <linux/notifier.h> +#include <linux/hardirq.h> +#include <linux/kprobes.h> +#include <linux/module.h> +#include <linux/kdebug.h> +#include <linux/sched.h> +#include <linux/uaccess.h> +#include <linux/slab.h> +#include <linux/highmem.h> +#include <linux/cpu.h> +#include <linux/bitops.h>
Do you really need all these includes?
-Andi
Quoted 11 lines · Andi Kleen
Quoted 6 lines
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
OK, I can use on ordinary interrupt handler here.
Quoted 6 lines
In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.
Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2
-0+ +static struct node_hw_events *uncore_events[MAX_NUMNODES ];Don't declare static arrays with
MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar (e.g. withalloc_percpu)
I really need is a per physical cpu data here, is alloc_percpu enough?
Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 149
Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5
-0+/* +* The hw event starts counting from this event offset, +* mark it to be able to extra future deltas: +*/ +local64_set (&hwc->prev_count, (u64)-left);Your use of local* seems dubious. That is only valid if it's really all on the same CPU. Is that really true?
Good catch! That is not true.
The interrupt handler is running on one core and the data(hwc->prev_count) maybe on another core.
Any idea to set this cross-core data?
Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 187
Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3
-0+static int uncore_pmu_add (struct perf_event *event,int flags) +{ +int node =numa_node_id ();this should be still package id
Understand, this is in my TODO.
Quoted 8 lines
Review
Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3
-0+/* Check CPUID signatures: 06_1AH, 06_1EH, 06_1FH */ + model = eax.split.model | (eax.split.ext_model <<4 ); + if (eax.split.family != 6 || (model != 0x1A && model != 0x1E && model !=
0x1F))(quoted diff) +1
-0+ return;You can just get that from
boot_cpu_data, no need to call cpuid
Nice, will use it.
Quoted 16 lines
Quoted 4 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +3
-0+#include <linux/perf_event.h> +#include <linux/capability.h> +#include <linux/notifier.h> arch/x86/kernel/cpu/perf_event_intel_uncore.h line 13
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +10
-0+#include <linux/hardirq.h> +#include <linux/kprobes.h> +#include <linux/module.h> +#include <linux/kdebug.h> +#include <linux/sched.h> +#include <linux/uaccess.h> +#include <linux/slab.h> +#include <linux/highmem.h> +#include <linux/cpu.h> +#include <linux/bitops.h> Do you really need all these includes?
Only
#include <linux/perf_event.h>
#include <linux/kprobes.h>
#include <linux/hardirq.h>
#include <linux/slab.h>
are needed.
Thanks for the comments.
Lin Ming
BTW another thing I noticed that if you ever add opcode/address matching you'll need to add the new parameters for the address at least to the input perf_event structure. opcode could in theory be encoded in the upper 32bits like offcore does, but address needs to be extra. It's only a small incremental step, but may make this more useful.
Quoted 6 lines
Quoted 4 lines
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
OK, I can use on ordinary interrupt handler here.
You'll need to allocate a vector, it shouldn't be too difficult.
Quoted 14 lines
Quoted 6 lines
In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.
Quoted 7 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2
-0+ +static struct node_hw_events *uncore_events[MAX_NUMNODES ];Don't declare static arrays with
MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar
(e.g. withalloc_percpu)
Quoted 1 line
I really need is a per physical cpu data here, is
alloc_percpuenough?
If you use a per cpu array then each CPU can carry a pointer to its per socket data structure.
This could use a similar scheme as the per core data I submitted recently.
Quoted 2 lines
Any idea to set this cross-core data?
s/local/atomic/
But if it's just stores/loads without read-modify-write you can just use normal stores.
Quoted 9 lines
Quoted 7 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 187
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +3
-0+static int uncore_pmu_add (struct perf_event *event,int flags) +{ +int node =numa_node_id ();this should be still package id
Understand, this is in my TODO.
With the per cpu pointer scheme you likely don't even need it, just check the topology at set up time (similar as in my patch, just using the package)
-Andi
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
Quoted 6 lines
Quoted 4 lines
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
OK, I can use on ordinary interrupt handler here.
Does the hardware actually allow using a different interrupt source?
Quoted 6 lines
Quoted 6 lines
In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.
Quoted 8 lines
Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2
-0+ +static struct node_hw_events *uncore_events[MAX_NUMNODES ];Don't declare static arrays with
MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar (e.g. withalloc_percpu)I really need is a per physical cpu data here, is
alloc_percpuenough?
Nah, simply manually allocate bits using kmalloc_node(), that's something I still need to fix in Andi's patches as well.
Quoted 9 lines
Quoted 8 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 149
Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5
-0+/* +* The hw event starts counting from this event offset, +* mark it to be able to extra future deltas: +*/ +local64_set (&hwc->prev_count, (u64)-left);Your use of local* seems dubious. That is only valid if it's really all on the same CPU. Is that really true?
Quoted 6 lines
Good catch! That is not true.
The interrupt handler is running on one core and the
data(hwc->prev_count)maybe on another core.Any idea to set this cross-core data?
IIRC you can steer the uncore interrupts (it has a mask somewhere) simply steer everything to the first cpu in the nodemask?
Quoted 9 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
Quoted 9 lines
Quoted 6 lines
Quoted 4 lines
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
OK, I can use on ordinary interrupt handler here.
Does the hardware actually allow using a different interrupt source?
It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.
Quoted 15 lines · Stephane Eranian
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Lin Ming
Quoted 11 lines · Andi Kleen
Quoted 6 lines
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
OK, I can use on ordinary interrupt handler here.
Does the hardware actually allow using a different interrupt source?
It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via
IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.
If only counting(perf stat) makes sense for uncore events, do we still need an interrupt handler?
48 bits counter is not that easy to overflow in practice.
Quoted 20 lines · Lin Ming
Quoted 19 lines · Stephane Eranian
Quoted 18 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
Quoted 6 lines
Quoted 4 lines
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
OK, I can use on ordinary interrupt handler here.
Does the hardware actually allow using a different interrupt source?
Quoted 7 lines
Quoted 4 lines
It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via
IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.If only counting(perf stat) makes sense for uncore events, do we still need an interrupt handler?
Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package, I bet there are people who will run just one task on a package as well.
Just because it doesn't make sense in general doesn't mean there isn't anybody who'd want to do it and actually knows wth he's doing.
Quoted 1 line
48 bits counter is not that easy to overflow in practice.
Still..
Quoted 34 lines · Peter Zijlstra
Quoted 20 lines · Lin Ming
Quoted 19 lines · Stephane Eranian
Quoted 18 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
Quoted 6 lines
Quoted 4 lines
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
OK, I can use on ordinary interrupt handler here.
Does the hardware actually allow using a different interrupt source?
Quoted 7 lines
Quoted 4 lines
It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via
IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.If only counting(perf stat) makes sense for uncore events, do we still need an interrupt handler?
Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package, I bet there are people who will run just one task on a package as well.
Quoted 6 lines
Just because it doesn't make sense in general doesn't mean there isn't anybody who'd want to do it and actually knows wth he's doing.
Quoted 1 line
48 bits counter is not that easy to overflow in practice.
Still..
OK, will do more tests, then send out a new version.
Thanks.
Quoted 34 lines · Peter Zijlstra
Quoted 20 lines · Lin Ming
Quoted 19 lines · Stephane Eranian
Quoted 18 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
Quoted 6 lines
Quoted 4 lines
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
OK, I can use on ordinary interrupt handler here.
Does the hardware actually allow using a different interrupt source?
Quoted 7 lines
Quoted 4 lines
It does not. It's using whatever you've programmed into the APIC LVT vector, AFAIK. Uncore interrupt mode is enabled via
IA32_DEBUGCTL. Regarless of sampling or not, you need the interrupt to virtualize the counters to 64 bits.If only counting(perf stat) makes sense for uncore events, do we still need an interrupt handler?
Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package, I bet there are people who will run just one task on a package as well.
Quoted 7 lines
Just because it doesn't make sense in general doesn't mean there isn't anybody who'd want to do it and actually knows wth he's doing.
Quoted 1 line
48 bits counter is not that easy to overflow in practice.
Still..
Agreed.
Quoted 3 lines
Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package,
Assuming they don't have any active PCI devices either.
Quoted 2 lines
I bet there are people who will run just one task on a package as well.
In that case the sampling has a 1/NUM-CPU-THREADS-IN-PACKAGE chance to report the right task (or actually somewhat less because the measurement skew for uncore is much higher than for normal events)
Really for per core measurements using the OFFCORE events is much better.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 6 lines · Andi Kleen
Quoted 3 lines
Yep, I see no reason to dis-allow sampling. Sure its hard to make sense of it, but since there are people who offline all but one cpu of a package,
Assuming they don't have any active PCI devices either.
Good point.
Quoted 9 lines
Quoted 2 lines
I bet there are people who will run just one task on a package as well.
In that case the sampling has a 1/NUM-CPU-THREADS-IN-PACKAGE chance to report the right task (or actually somewhat less because the measurement skew for uncore is much higher than for normal events)
Really for per core measurements using the OFFCORE events is much better.
yes, OFFCORE_RESPONSE is much more useful.
Quoted 31 lines · Peter Zijlstra
Quoted 8 lines · Lin Ming
Quoted 7 lines · Andi Kleen
Quoted 6 lines
2. Uncore pmu NMI handling
All the 4 cores are programmed to receive uncore counter overflow interrupt. The NMI handler(running on 1 of the 4 cores) handle all counters enabled by all 4 cores.
Quoted 6 lines
Quoted 4 lines
Really for uncore monitoring there is no need to use an NMI handler. You can't profile a core anyways, so you can just delay the reporting a little bit. It may simplify the code to not use one here and just use an ordinary handler.
OK, I can use on ordinary interrupt handler here.
Does the hardware actually allow using a different interrupt source?
Quoted 13 lines
Quoted 12 lines
In general since there is already much trouble with overloaded NMI events avoiding new NMIs is a good idea.
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 3
Quoted 2 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +2
-0+ +static struct node_hw_events *uncore_events[MAX_NUMNODES ];Don't declare static arrays with
MAX_NUMNODES, that number can be very large and cause unnecessary bloat. Better use per CPU data or similar (e.g. withalloc_percpu)
Quoted 4 lines
Quoted 1 line
I really need is a per physical cpu data here, is
alloc_percpuenough?Nah, simply manually allocate bits using
kmalloc_node(), that's something I still need to fix in Andi's patches as well.
I'm writing this like AMD NB events allocation.
Thanks,
Lin Ming
Quoted 22 lines
Quoted 10 lines
Quoted 9 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 149
Quoted 6 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5
-0+/* +* The hw event starts counting from this event offset, +* mark it to be able to extra future deltas: +*/ +local64_set (&hwc->prev_count, (u64)-left);Your use of local* seems dubious. That is only valid if it's really all on the same CPU. Is that really true?
Quoted 6 lines
Good catch! That is not true.
The interrupt handler is running on one core and the
data(hwc->prev_count)maybe on another core.Any idea to set this cross-core data?
IIRC you can steer the uncore interrupts (it has a mask somewhere) simply steer everything to the first cpu in the nodemask?
Lin,
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48
Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8
-0+static void uncore_pmu_enable_all (void ) +{ + u64 ctrl; + + /* +* (0xFULL << 48): 1 of the 4 cores can receive NMI each time +* but we don't know which core will receive the NMI when overflow happens +*/
That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true in uncore_handle_irq().
I also think that if you want all processors to receive the interrupts, then the mask should
be 0xff when HT is on. The manual is rather obscure on this, but it does make sense.
Quoted 12 lines
+ ctrl = ((1 <<
UNCORE_NUM_GENERAL_COUNTERS) - 1) | (0xFULL<< 48);
+ ctrl |=MSR_UNCORE_PERF_GLOBAL_CTRL_EN_FC0;arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +10
-0+ + /* + * Freeze the uncore pmu on overflow of any uncore counter. + * This makes unocre NMI handling easier. + */ + ctrl |= MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_FRZ; + + wrmsrl(MSR_UNCORE_PERF_GLOBAL_CTRL, ctrl); +} +
Quoted 12 lines · Stephane Eranian
Lin,
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48
Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8
-0+static void uncore_pmu_enable_all (void ) +{ +u64 ctrl; + +/* +* (0xFULL << 48): 1 of the 4 cores can receive NMI each time +* but we don't know which core will receive the NMI when overflow happens +*/
Quoted 8 lines
That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true inuncore_handle_irq().
I thought it's a broadcast too in the v1 patches, let me double check it.
Quoted 5 lines
I also think that if you want all processors to receive the interrupts, then the mask should
be0xffwhen HT is on. The manual is rather obscure on this, but it does make sense.
Kernel panics if 0xff is set, but it maybe bugs of my code.
Anyway, is it told in some errata that the mask should be 0xff when HT is on?
Thanks,
Lin Ming
Quoted 14 lines
Quoted 14 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +12
-0+ ctrl = ((1 <<UNCORE_NUM_GENERAL_COUNTERS ) -1 ) | (0xFULL <<48 ); + ctrl |=MSR_UNCORE_PERF_GLOBAL_CTRL_EN_FC0 ; + +/* +* Freeze the uncore pmu on overflow of any uncore counter. +* This makes unocre NMI handling easier. +*/ + ctrl |=MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_FRZ ; + +wrmsrl (MSR_UNCORE_PERF_GLOBAL_CTRL , ctrl); +} +
Quoted 12 lines · Stephane Eranian
Lin,
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48
Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8
-0+static void uncore_pmu_enable_all (void ) +{ + u64 ctrl; + + /* +* (0xFULL << 48): 1 of the 4 cores can receive NMI each time +* but we don't know which core will receive the NMI when overflow happens +*/
Quoted 8 lines
That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true inuncore_handle_irq().
Stephane,
The interrupt model is strange, it behaves differently when HT on/off.
If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.
If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.
Did you ever observe this?
Quoted 5 lines
I also think that if you want all processors to receive the interrupts, then the mask should
be0xffwhen HT is on. The manual is rather obscure on this, but it does make sense.
I tried to set the mask 0xff when HT is on, but kernel panics, because the reserve bits are set.
Thanks,
Lin Ming
Quoted 15 lines
Quoted 4 lines
+ ctrl = ((1 <<
UNCORE_NUM_GENERAL_COUNTERS) - 1) | (0xFULL<< 48);
+ ctrl |=MSR_UNCORE_PERF_GLOBAL_CTRL_EN_FC0;arch/x86/kernel/cpu/perf_event_intel_uncore.c line 60
Quoted 10 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +10
-0+ + /* + * Freeze the uncore pmu on overflow of any uncore counter. + * This makes unocre NMI handling easier. + */ + ctrl |= MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_FRZ; + + wrmsrl(MSR_UNCORE_PERF_GLOBAL_CTRL, ctrl); +} +--
Quoted 22 lines · Lin Ming
Quoted 12 lines · Stephane Eranian
Lin,
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48
Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8
-0+static void uncore_pmu_enable_all (void ) +{ + u64 ctrl; + + /* +* (0xFULL << 48): 1 of the 4 cores can receive NMI each time +* but we don't know which core will receive the NMI when overflow happens +*/ Quoted 8 lines
That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true inuncore_handle_irq().
Quoted 6 lines
Stephane,
The interrupt model is strange, it behaves differently when HT on/off.
If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.
That's if yo set the mask to 0xf, right?
In the perf_event model, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.
Quoted 6 lines
If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.Did you ever observe this?
No because I never set more than one bit in the mask.
Quoted 2 lines
I tried to set the mask
0xffwhen HT is on, but kernel panics, because the reserve bits are set.
Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.
Was that on Nehalem or Westmere?
Quoted 23 lines · Stephane Eranian
Quoted 22 lines · Lin Ming
Quoted 20 lines · Stephane Eranian
Lin,
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48
Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8
-0+static void uncore_pmu_enable_all (void ) +{ + u64 ctrl; + + /* +* (0xFULL << 48): 1 of the 4 cores can receive NMI each time +* but we don't know which core will receive the NMI when overflow happens +*/ That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true inuncore_handle_irq().
Quoted 7 lines
Quoted 6 lines
Stephane,
The interrupt model is strange, it behaves differently when HT on/off.
If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.
That's if yo set the mask to
0xf, right?
Right.
Quoted 5 lines
In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.
This is what my upcoming v3 patches doing.
Quoted 9 lines
Quoted 7 lines
If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.Did you ever observe this?
No because I never set more than one bit in the mask.
Quoted 7 lines
Quoted 2 lines
I tried to set the mask
0xffwhen HT is on, but kernel panics, because the reserve bits are set.Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.
Was that on Nehalem or Westmere?
Nehalem.
Thanks,
Lin Ming
Lin,
Looked at the perfmon code, and it seems the mask is actual cores, not threads:
rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id);
wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);That seems to imply both threads will get the interrupt.
In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.
On the cores where no uncore was programmed, then both threads will have a spurious interrupt.
That brings up back to the 'spurious interrupt' issue and the 'NMI Dazed' message
that Don tried to eliminate. Now we have a new situation where we will get interrupt
with no work to do, so the perf_event will pass the interrupt onto the next subsystem
and eventually we will get the 'dazed' message. I am just guessing here....
Quoted 52 lines · Stephane Eranian
Quoted 22 lines · Lin Ming
Quoted 12 lines · Stephane Eranian
Lin,
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48
Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8
-0+static void uncore_pmu_enable_all (void ) +{ + u64 ctrl; + + /* +* (0xFULL << 48): 1 of the 4 cores can receive NMI each time +* but we don't know which core will receive the NMI when overflow happens +*/ Quoted 8 lines
That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true inuncore_handle_irq().Quoted 6 lines
Stephane,
The interrupt model is strange, it behaves differently when HT on/off.
If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.
That's if yo set the mask to
0xf, right?In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.Quoted 6 lines
If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.Did you ever observe this?
No because I never set more than one bit in the mask.
Quoted 2 lines
I tried to set the mask
0xffwhen HT is on, but kernel panics, because the reserve bits are set.Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.
Was that on Nehalem or Westmere?
Quoted 8 lines · Stephane Eranian
Lin,
Looked at the perfmon code, and it seems the mask is actual cores, not threads:
rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id);wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
Quoted 7 lines
That seems to imply both threads will get the interrupt.
In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.
On the cores where no uncore was programmed, then both threads will have a spurious interrupt.
But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.
I'll do more tests to verify this.
Quoted 8 lines
That brings up back to the 'spurious interrupt' issue and the 'NMI Dazed' message
that Don tried to eliminate. Now we have a new situation where we will get interrupt
with no work to do, so theperf_eventwill pass the interrupt onto the next subsystem
and eventually we will get the 'dazed' message. I am just guessing here....
Add Don.
Thanks,
Lin Ming
Quoted 56 lines · Stephane Eranian
Quoted 52 lines · Lin Ming
Quoted 22 lines · Stephane Eranian
Quoted 12 lines
Lin,
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48
Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8
-0+static void uncore_pmu_enable_all (void ) +{ + u64 ctrl; + + /* +* (0xFULL << 48): 1 of the 4 cores can receive NMI each time +* but we don't know which core will receive the NMI when overflow happens +*/ Quoted 8 lines
That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true inuncore_handle_irq().Quoted 6 lines
Stephane,
The interrupt model is strange, it behaves differently when HT on/off.
If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.
That's if yo set the mask to
0xf, right?In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.Quoted 6 lines
If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.Did you ever observe this?
No because I never set more than one bit in the mask.
Quoted 2 lines
I tried to set the mask
0xffwhen HT is on, but kernel panics, because the reserve bits are set.Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.
Was that on Nehalem or Westmere?
Quoted 14 lines · Lin Ming
Quoted 13 lines · Stephane Eranian
Lin,
Looked at the perfmon code, and it seems the mask is actual cores, not threads:
rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id);wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);That seems to imply both threads will get the interrupt.
In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.
Quoted 8 lines
Quoted 2 lines
On the cores where no uncore was programmed, then both threads will have a spurious interrupt.
But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.
The MSR_NHM_UNC_GLOBAL_CTRL is per socket not per core.
Quoted 1 line
I'll do more tests to verify this.
In your tests, are your programming the same uncore event across all CPUs? If so then you may have a race condition setting the MSR because it read-modify-write.
What about you program only one uncore event from one CPU?
Quoted 72 lines
Quoted 9 lines
That brings up back to the 'spurious interrupt' issue and the 'NMI Dazed' message
that Don tried to eliminate. Now we have a new situation where we will get interrupt
with no work to do, so theperf_eventwill pass the interrupt onto the next subsystem
and eventually we will get the 'dazed' message. I am just guessing here....Add Don.
Thanks,
Lin MingQuoted 56 lines · Stephane Eranian
Quoted 52 lines · Lin Ming
Quoted 22 lines · Stephane Eranian
Quoted 12 lines
Lin,
arch/x86/kernel/cpu/perf_event_intel_uncore.c line 48
Quoted 8 lines · Lin Ming
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +8
-0+static void uncore_pmu_enable_all (void ) +{ + u64 ctrl; + + /* +* (0xFULL << 48): 1 of the 4 cores can receive NMI each time +* but we don't know which core will receive the NMI when overflow happens +*/ Quoted 8 lines
That does not sound right. If you set bit 48-51 to 1, then all 4 cores will receive EVERY
interrupt, i.e., it's a broadcast. That seems to contradict your
comment: 1 of the 4. Unless
you meant, they all get the interrupt and one will handle it, the other will find nothing to
process. But I don't see the atomic op that would make this true inuncore_handle_irq().Quoted 6 lines
Stephane,
The interrupt model is strange, it behaves differently when HT on/off.
If HT is off, all 4 cores will receive every interrupt, i.e., it's a broadcast.
That's if yo set the mask to
0xf, right?In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.Quoted 6 lines
If HT is on, only 1 of the 4 cores will receive the interrupt(both Threads in that core receive the interrupt),
and it can't be determined which core will receive the interrupt.Did you ever observe this?
No because I never set more than one bit in the mask.
Quoted 2 lines
I tried to set the mask
0xffwhen HT is on, but kernel panics, because the reserve bits are set.Let me check on this. It would seem to imply that in HT mode, both threads necessarily receive the interrupts.
Was that on Nehalem or Westmere?
Quoted 18 lines · Stephane Eranian
Quoted 17 lines · Lin Ming
Quoted 15 lines · Stephane Eranian
Lin,
Looked at the perfmon code, and it seems the mask is actual cores, not threads:
rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val); val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id); wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);That seems to imply both threads will get the interrupt.
In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.
On the cores where no uncore was programmed, then both threads will have a spurious interrupt.
Quoted 6 lines
Quoted 5 lines
But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.The
MSR_NHM_UNC_GLOBAL_CTRLis per socket not per core.
Understood.
Quoted 8 lines
Quoted 2 lines
I'll do more tests to verify this.
In your tests, are your programming the same uncore event across all CPUs? If so then you may have a race condition setting the MSR because it read-modify-write.
What about you program only one uncore event from one CPU?
This is what I tested, programming only one uncore event from one CPU. When HT is off, all four cores in the socket receive the interrupt. When HT is on, only the 2 threads in one of the four cores receive the interrupt.
Quoted 36 lines · Lin Ming
Quoted 18 lines · Stephane Eranian
Quoted 17 lines · Lin Ming
Quoted 15 lines · Stephane Eranian
Lin,
Looked at the perfmon code, and it seems the mask is actual cores, not threads:
rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);
val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id);wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);That seems to imply both threads will get the interrupt.
In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.
On the cores where no uncore was programmed, then both threads will have a spurious interrupt.
Quoted 6 lines
Quoted 5 lines
But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.The
MSR_NHM_UNC_GLOBAL_CTRLis per socket not per core.Understood.
Quoted 7 lines
Quoted 2 lines
I'll do more tests to verify this.
In your tests, are your programming the same uncore event across all CPUs? If so then you may have a race condition setting the MSR because it read-modify-write.
Quoted 5 lines
Quoted 1 line
What about you program only one uncore event from one CPU?
This is what I tested, programming only one uncore event from one CPU.
When HT is off, all four cores in the socket receive the interrupt.
If the value of the MSR is 0xf << 48?
Quoted 2 lines
When HT is on, only the 2 threads in one of the four cores receive the interrupt.
Something is not right here. Next week, I may be able to run some tests on a Nehalem using perfmon to compare. Could you also send me your latest uncore patch against tip-x86?
Thanks.
Quoted 37 lines · Stephane Eranian
Quoted 36 lines · Lin Ming
Quoted 18 lines · Stephane Eranian
Quoted 17 lines · Lin Ming
Quoted 15 lines · Stephane Eranian
Lin,
Looked at the perfmon code, and it seems the mask is actual cores, not threads:
rdmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val); val |= 1ULL << (48 + cpu_data(smp_processor_id()).cpu_core_id); wrmsrl(MSR_NHM_UNC_GLOBAL_CTRL, val);That seems to imply both threads will get the interrupt.
In the the overflowed event was programmed from on of the two threads, that means one will process the overflow, the other will get spurious.
On the cores where no uncore was programmed, then both threads will have a spurious interrupt.
Quoted 6 lines
Quoted 5 lines
But in my test, if HT is on, only the 2 theads in one of the four cores will receive the interrupt. Even worse, we don't know which core will receive the interrupt
when overflow happens.The
MSR_NHM_UNC_GLOBAL_CTRLis per socket not per core.Understood.
Quoted 7 lines
Quoted 2 lines
I'll do more tests to verify this.
In your tests, are your programming the same uncore event across all CPUs? If so then you may have a race condition setting the MSR because it read-modify-write.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
What about you program only one uncore event from one CPU?
This is what I tested, programming only one uncore event from one CPU.
When HT is off, all four cores in the socket receive the interrupt.
If the value of the MSR is
0xf<< 48?
Yes, the EN_PMI_CORE* bits are set to 0xf.
Quoted 7 lines
Quoted 3 lines
When HT is on, only the 2 threads in one of the four cores receive the interrupt.
Something is not right here. Next week, I may be able to run some tests on a Nehalem using perfmon to compare. Could you also send me your latest uncore patch against tip-x86?
Thanks.
I just send it out.
http://lkml.org/lkml/2010/12/2/4
Thanks,
Lin Ming
Quoted 4 lines · Stephane Eranian
In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.
Not really, you can redirect all these events to the first online cpu of the node.
You can re-write event->cpu in pmu::event_init(), and register cpu hotplug notifiers to migrate the state around.
Quoted 7 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.
Quoted 6 lines
Not really, you can redirect all these events to the first online cpu of the node.
You can re-write
event->cpuinpmu::event_init(), and register cpu hotplug notifiers to migrate the state around.
I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.
Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.Not really, you can redirect all these events to the first online cpu of the node.
Quoted 6 lines
Quoted 3 lines
You can re-write
event->cpuinpmu::event_init(), and register cpu hotplug notifiers to migrate the state around.I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.
Well, its either that or keeping a node wide state like we do for AMD and serialize everything from there.
And I'm not sure what's most expensive, steering the interrupt to one core only, or broadcasting every interrupt, I'd favour the first approach.
The whole thing is a node-wide resource, so the user needs to think in nodes anyway, we already do a cpu->node mapping for identifying the thing.
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.Not really, you can redirect all these events to the first online cpu of the node.
Quoted 6 lines
Quoted 3 lines
You can re-write
event->cpuinpmu::event_init(), and register cpu hotplug notifiers to migrate the state around.I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.
Quoted 6 lines
Well, its either that or keeping a node wide state like we do for AMD and serialize everything from there.
And I'm not sure what's most expensive, steering the interrupt to one core only, or broadcasting every interrupt, I'd favour the first approach.
I think the one core-only approach will limit the spurious interrupt aspect. In perfmon, that's how I had it setup. The first CPU where uncore is accessed owns the uncore PMU for the socket, thus all interrupts are routed there. What you are proposing is the same. Now you can chose you hardcode which is the default core to handle this, or (better) you use the first core that accesses uncore.
Quoted 5 lines
The whole thing is a node-wide resource, so the user needs to think in nodes anyway, we already do a
cpu->nodemapping for identifying the thing.
Agreed.
Quoted 35 lines · Stephane Eranian
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.Not really, you can redirect all these events to the first online cpu of the node.
Quoted 6 lines
Quoted 3 lines
You can re-write
event->cpuinpmu::event_init(), and register cpu hotplug notifiers to migrate the state around.I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.
Quoted 6 lines
Well, its either that or keeping a node wide state like we do for AMD and serialize everything from there.
And I'm not sure what's most expensive, steering the interrupt to one core only, or broadcasting every interrupt, I'd favour the first approach.
I think the one core-only approach will limit the spurious interrupt aspect. In perfmon, that's how I had it setup. The first CPU where uncore is accessed owns the uncore PMU for the socket, thus all interrupts are routed there. What you are proposing is the same. Now you can chose you hardcode which is the default core to handle this, or (better) you use the first core that accesses uncore.
Quoted 1 line
Quoted 6 lines
Quoted 4 lines
The whole thing is a node-wide resource, so the user needs to think in nodes anyway, we already do a
cpu->nodemapping for identifying the thing.Agreed.
Hi, all
Thanks for all the comments.
I'm on travel Nov 27 to Nov 30.
I'll address the comments when I'm back.
Thanks,
Lin Ming
Quoted 22 lines · Peter Zijlstra
Quoted 11 lines · Stephane Eranian
Quoted 10 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian
In the
perf_eventmodel, given that any one of the 4 cores can be used to program uncore events, you have no choice but to broadcast to all 4 cores. Each has to demultiplex and figure out which of its counters have overflowed.Not really, you can redirect all these events to the first online cpu of the node.
Quoted 6 lines
Quoted 3 lines
You can re-write
event->cpuinpmu::event_init(), and register cpu hotplug notifiers to migrate the state around.I am sure you could. But then the user thinks the event is controlled from CPUx when it's actually from CPUz. I am sure it can work but that's confusing, especially interrupt-wise.
Well, its either that or keeping a node wide state like we do for AMD and serialize everything from there.
Quoted 7 lines
And I'm not sure what's most expensive, steering the interrupt to one core only, or broadcasting every interrupt, I'd favour the first approach.
The whole thing is a node-wide resource, so the user needs to think in nodes anyway, we already do a
cpu->nodemapping for identifying the thing.
How about a new sub-command for node-wide events statistics?
perf node -n <node> -e <event>?
Quoted 3 lines · Lin Ming
How about a new sub-command for node-wide events statistics?
perf node -n <node> -e <event>?
Maybe as a very slim wrapper around perf stat, but personally I wouldn't care.
Quoted 3 lines
How about a new sub-command for node-wide events statistics?
perf node -n <node> -e <event>?
Seems like the best option to me (and not allowing the uncore events for the other commands)
But I don't like "node" because in a non NUMA kernel you won't have nodes, but this is still useful. Or with NUMA emulation you may have nodes that doesn't match the sockets.
Maybe perf package or perf socket ?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 3 lines · Andi Kleen
Seems like the best option to me (and not allowing the uncore events for the other commands)
Andi, I don't care what you think, that's simply not going to happen.