Skip to contentSkip to conversation

perf: Implement Nehalem uncore pmu

PATCH Superseded Lin Ming 4 participants 8 messages Nov 2, 2010 – Nov 9, 2010

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Lin Ming

author
#0 lore raw

PATCH 2/3 perf: Implement Nehalem uncore pmu

For the background of Nehalem uncore pmu, see Intel SDM Volume 3B "30.6.2 Performance Monitoring Facility in the Uncore"

1. data structure

struct node_hw_events {
        struct perf_event *events[UNCORE_NUM_COUNTERS];
        int n_events;
        struct spinlock lock;
        int enabled;
};

struct node_hw_events is the per node structure.
"lock" protects add/delete events to uncore pmu.

struct uncore_cpu_hw_events {
        unsigned long active_mask[BITS_TO_LONGS(UNCORE_NUM_COUNTERS)];
};

struct uncore_cpu_hw_events is the per logical cpu structure. "active_mask" represents the counters used by the cpu.
For example, if bit 3, 6 are set for cpuX, then it means uncore counter 3 and 6 are used by cpuX.

2. Uncore pmu NMI handling

Every core in the socket can be programmed to receive uncore counter overflow interrupt.

In this draft implementation, each core handles the overflow interrupt caused by the counters with bit set in "active_mask".

---
 arch/x86/include/asm/msr-index.h              |    1 +
 arch/x86/kernel/cpu/perf_event.c              |    7 +-
 arch/x86/kernel/cpu/perf_event_intel_uncore.c |  280 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_uncore.h |   80 +++++++
 4 files changed, 367 insertions(+), 1 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.h
arch/x86/include/asm/msr-index.h +1 -0
index 3ea3dc4..816fb4b 100644
@@ -81,6 +81,7 @@
 #define DEBUGCTLMSR_BTS_OFF_OS		(1UL <<  9)
 #define DEBUGCTLMSR_BTS_OFF_USR		(1UL << 10)
 #define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI	(1UL << 11)
+#define DEBUGCTLMSR_ENABLE_UNCORE_PMI	(1UL << 13)
 
 #define MSR_IA32_MC0_CTL		0x00000400
 #define MSR_IA32_MC0_STATUS		0x00000401
arch/x86/kernel/cpu/perf_event.c +6 -1
index 7cea0f4..cca07b4 100644
@@ -1215,6 +1215,8 @@ struct pmu_nmi_state {
 
 static DEFINE_PER_CPU(struct pmu_nmi_state, pmu_nmi);
 
+static int uncore_pmu_handle_irq(struct pt_regs *regs);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
 			 unsigned long cmd, void *__args)
@@ -1249,7 +1251,8 @@ perf_event_nmi_handler(struct notifier_block *self,
 
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 
-	handled = x86_pmu.handle_irq(args->regs);
+	handled = uncore_pmu_handle_irq(args->regs);
+	handled += x86_pmu.handle_irq(args->regs);
 	if (!handled)
 		return NOTIFY_DONE;
 
@@ -1305,6 +1308,7 @@ x86_get_event_constraints(struct cpu_hw_events *cpuc, struct perf_event *event)
 #include "perf_event_intel_lbr.c"
 #include "perf_event_intel_ds.c"
 #include "perf_event_intel.c"
+#include "perf_event_intel_uncore.c"
 
 static int __cpuinit
 x86_pmu_notifier(struct notifier_block *self, unsigned long action, void *hcpu)
@@ -1360,6 +1364,7 @@ void __init init_hw_perf_events(void)
 
 	switch (boot_cpu_data.x86_vendor) {
 	case X86_VENDOR_INTEL:
+		init_uncore_pmu();
 		err = intel_pmu_init();
 		break;
 	case X86_VENDOR_AMD:
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +280 -0

280 lines not on this page · raw message

arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +80 -0

80 lines not on this page · raw message

Signature · 1 line
1.7.1
#1 lore raw

Lin,

Quoted 15 lines · Lin Ming

For the background of Nehalem uncore pmu, see Intel SDM Volume 3B "30.6.2 Performance Monitoring Facility in the Uncore"

1. data structure

struct node_hw_events {
       struct perf_event *events[UNCORE_NUM_COUNTERS]; int n_events;
       struct spinlock lock;
       int enabled;
};

struct node_hw_events is the per node structure.
"lock" protects add/delete events to uncore pmu.

Quoted 9 lines

struct uncore_cpu_hw_events {
       unsigned long active_mask[BITS_TO_LONGS(UNCORE_NUM_COUNTERS)]; };

struct uncore_cpu_hw_events is the per logical cpu structure. "active_mask" represents the counters used by the cpu.
For example, if bit 3, 6 are set for cpuX, then it means uncore counter 3 and 6 are used by cpuX.

I would advise you allocate your uncore_events[] table dynamically using kmalloc_node(). That way you avoid unnecessary remote memory accesses.

Furthermore, the patch is missing support for the fixed uncore counter. It is very useful as it allows measuring some reference cycles at the socket level. You have 8+1 counter total. You need to define some encoding for UNC_CPU_CLK.

Another tidbit is about handling of exclude_*. Uncore PMU does measure at all priv level all the time. I think it would be wise to fail if the user passes
anything different from exclude_* = 0. Otherwise you think you are measuring at some levels but in reality you are not.

Quoted 8 lines

2. Uncore pmu NMI handling

Every core in the socket can be programmed to receive uncore counter overflow interrupt.

In this draft implementation, each core handles the overflow interrupt caused by the counters with bit set in "active_mask".

Seems like in your model, interrupting all cores is the only solution given that you can program uncore events from any cores on the socket.

Quoted 439 lines

---
 arch/x86/include/asm/msr-index.h              |    1 + arch/x86/kernel/cpu/perf_event.c              |    7 +- arch/x86/kernel/cpu/perf_event_intel_uncore.c |  280 +++++++++++++++++++++++++ arch/x86/kernel/cpu/perf_event_intel_uncore.h |   80 +++++++

 4 files changed, 367 insertions(+), 1 deletions(-)

 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.c create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.h

arch/x86/include/asm/msr-index.h +1 -0
index 3ea3dc4..816fb4b 100644
@@ -81,6 +81,7 @@
  #define DEBUGCTLMSR_BTS_OFF_OS         (1UL <<  9)
  #define DEBUGCTLMSR_BTS_OFF_USR                (1UL << 10)
  #define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI (1UL << 11)
+#define DEBUGCTLMSR_ENABLE_UNCORE_PMI  (1UL << 13)
 
  #define MSR_IA32_MC0_CTL               0x00000400
  #define MSR_IA32_MC0_STATUS            0x00000401
arch/x86/kernel/cpu/perf_event.c +2 -0
index 7cea0f4..cca07b4 100644
@@ -1215,6 +1215,8 @@ struct pmu_nmi_state {
 
  static DEFINE_PER_CPU(struct pmu_nmi_state, pmu_nmi);
 
+static int uncore_pmu_handle_irq(struct pt_regs *regs);
+

 static int __kprobes perf_event_nmi_handler(struct notifier_block *self, unsigned long cmd, void *__args)

arch/x86/kernel/cpu/perf_event.c +3 -1
@@ -1249,7 +1251,8 @@ perf_event_nmi_handler(struct notifier_block *self,
 
        apic_write(APIC_LVTPC, APIC_DM_NMI);
 
-       handled = x86_pmu.handle_irq(args->regs);
+       handled = uncore_pmu_handle_irq(args->regs);
+       handled += x86_pmu.handle_irq(args->regs);
        if (!handled)
                return NOTIFY_DONE;
 
@@ -1305,6 +1308,7 @@ x86_get_event_constraints(struct cpu_hw_events *cpuc, struct perf_event *event)
  #include "perf_event_intel_lbr.c"
  #include "perf_event_intel_ds.c"
  #include "perf_event_intel.c"
+#include "perf_event_intel_uncore.c"
 

 static int __cpuinit
 x86_pmu_notifier(struct notifier_block *self, unsigned long action, void *hcpu)

arch/x86/kernel/cpu/perf_event.c +0 -0
@@ -1360,6 +1364,7 @@ void __init init_hw_perf_events(void)
 
        switch (boot_cpu_data.x86_vendor) {

       case X86_VENDOR_INTEL:
+               init_uncore_pmu();
               err = intel_pmu_init();
               break;
       case X86_VENDOR_AMD:

arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +280 -0
new file mode 100644
index 0000000..fafa0f9
@@ -0,0 +1,280 @@
+#include "perf_event_intel_uncore.h"
+
+static struct node_hw_events uncore_events[MAX_NUMNODES];
+static DEFINE_PER_CPU(struct uncore_cpu_hw_events, uncore_cpu_hw_events);
+static bool uncore_pmu_initialized;
+
+static void uncore_pmu_enable_event(struct perf_event *event)
+{
+       struct hw_perf_event *hwc = &event->hw;
+
+       wrmsrl(hwc->config_base + hwc->idx, hwc->config | UNCORE_EVENTSEL_ENABLE);
+}
+
+static void uncore_pmu_disable_event(struct perf_event *event)
+{
+       struct hw_perf_event *hwc = &event->hw;
+
+       wrmsrl(hwc->config_base + hwc->idx, hwc->config);
+}
+
+static void uncore_pmu_disable_events(void)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       int node = numa_node_id();
+       int bit;
+
+       for_each_set_bit(bit, cpuc->active_mask, UNCORE_NUM_COUNTERS)
+               uncore_pmu_disable_event(uncore_events[node].events[bit]);
+}
+
+static void uncore_pmu_enable_events(void)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       int node = numa_node_id();
+       int bit;
+
+       for_each_set_bit(bit, cpuc->active_mask, UNCORE_NUM_COUNTERS)
+               uncore_pmu_disable_event(uncore_events[node].events[bit]);
+}
+
+static void uncore_pmu_global_enable(void)
+{
+       u64 ctrl;
+
+       /* (0xFULL << 48): all 4 cores will receive NMI */
+       ctrl = ((1 << UNCORE_NUM_COUNTERS) - 1) | (0xFULL << 48);
+
+       wrmsrl(MSR_UNCORE_PERF_GLOBAL_CTRL, ctrl);
+}
+
+static void uncore_perf_event_destroy(struct perf_event *event)
+{
+       atomic_dec(&active_events);
+}
+
+static int uncore_pmu_event_init(struct perf_event *event)
+{
+       struct hw_perf_event *hwc = &event->hw;
+
+       if (!uncore_pmu_initialized)
+               return -ENOENT;
+
+       switch (event->attr.type) {
+       case PERF_TYPE_UNCORE:
+               break;
+
+       default:
+               return -ENOENT;
+       }
+
+       atomic_inc(&active_events);
+
+       event->destroy = uncore_perf_event_destroy;
+
+       hwc->idx = -1;
+       hwc->config = (event->attr.config & UNCORE_RAW_EVENT_MASK) | UNCORE_EVENTSEL_PMI;
+       hwc->config_base = MSR_UNCORE_PERFEVTSEL0;
+       hwc->event_base = MSR_UNCORE_PMC0;
+
+       return 0;
+}
+
+static void uncore_pmu_start(struct perf_event *event, int flags)
+{
+       if (flags & PERF_EF_RELOAD)
+               x86_perf_event_set_period(event);
+
+       uncore_pmu_enable_event(event);
+
+       perf_event_update_userpage(event);
+}
+
+static void uncore_pmu_stop(struct perf_event *event, int flags)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       struct hw_perf_event *hwc = &event->hw;
+
+       if (__test_and_clear_bit(hwc->idx, cpuc->active_mask))
+               uncore_pmu_disable_event(event);
+
+       if (flags & PERF_EF_UPDATE)
+               x86_perf_event_update(event, UNCORE_CNTVAL_BITS);
+}
+
+static int uncore_pmu_add(struct perf_event *event, int flags)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       int node = numa_node_id();
+       int ret = 1;
+       int i;
+       u64 ctrl;
+
+       spin_lock(&uncore_events[node].lock);
+
+       for (i = 0; i < UNCORE_NUM_COUNTERS; i++) {
+               if (!uncore_events[node].events[i]) {
+                       uncore_events[node].events[i] = event;
+                       uncore_events[node].n_events++;
+
+                       event->hw.idx = i;
+                       __set_bit(i, cpuc->active_mask);
+                       if (flags & PERF_EF_START)
+                               uncore_pmu_start(event, PERF_EF_RELOAD);
+                       ret = 0;
+                       break;
+               }
+       }
+
+       /*
+        * PMI delivery due to an uncore counter overflow is enabled by
+        * setting IA32_DEBUG_CTL.Offcore_PMI_EN to 1.
+        */
+       if (uncore_events[node].n_events == 1) {
+               rdmsrl(MSR_IA32_DEBUGCTLMSR, ctrl);
+               wrmsrl(MSR_IA32_DEBUGCTLMSR, ctrl | DEBUGCTLMSR_ENABLE_UNCORE_PMI);
+       }
+
+       if (unlikely(!uncore_events[node].enabled)) {
+               uncore_pmu_global_enable();
+               uncore_events[node].enabled = 1;
+       }
+
+       spin_unlock(&uncore_events[node].lock);
+
+       return ret;
+}
+
+static void uncore_pmu_del(struct perf_event *event, int flags)
+{
+       int node = numa_node_id();
+       struct hw_perf_event *hwc = &event->hw;
+       int i;
+
+       spin_lock(&uncore_events[node].lock);
+
+       for (i = 0; i < UNCORE_NUM_COUNTERS; i++) {
+               if (uncore_events[node].events[i] == event) {
+                       uncore_events[node].events[hwc->idx] = NULL;
+                       uncore_events[node].n_events--;
+
+                       uncore_pmu_stop(event, PERF_EF_UPDATE);
+                       break;
+               }
+       }
+
+       spin_unlock(&uncore_events[node].lock);
+}
+
+static void uncore_pmu_read(struct perf_event *event)
+{
+       x86_perf_event_update(event, UNCORE_CNTVAL_BITS);
+}
+
+static struct pmu uncore_pmu = {
+       .event_init     = uncore_pmu_event_init,
+       .add            = uncore_pmu_add,
+       .del            = uncore_pmu_del,
+       .start          = uncore_pmu_start,
+       .stop           = uncore_pmu_stop,
+       .read           = uncore_pmu_read,
+};
+
+
+static inline u64 uncore_pmu_get_status(void)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       u64 status;
+
+       rdmsrl(MSR_UNCORE_PERF_GLOBAL_STATUS, status);
+
+       return status & (*(u64 *)cpuc->active_mask |
+               MSR_UNCORE_PERF_GLOBAL_STATUS_OVF_PMI | MSR_UNCORE_PERF_GLOBAL_STATUS_CHG);
+}
+
+static inline void uncore_pmu_ack_status(u64 ack)
+{
+       wrmsrl(MSR_UNCORE_PERF_GLOBAL_OVF_CTRL, ack);
+}
+
+static int uncore_pmu_save_and_restart(struct perf_event *event)
+{
+       x86_perf_event_update(event, UNCORE_CNTVAL_BITS);
+       return x86_perf_event_set_period(event);
+}
+
+int uncore_pmu_handle_irq(struct pt_regs *regs)
+{
+       struct perf_sample_data data;
+       struct node_hw_events *uncore_node;
+       int node;
+       int bit;
+       u64 status;
+       int handled = 0;
+
+       perf_sample_data_init(&data, 0);
+
+       node = numa_node_id();
+       uncore_node = &uncore_events[node];
+
+       status = uncore_pmu_get_status();
+       if (!status) {
+               apic_write(APIC_LVTPC, APIC_DM_NMI);
+
+               return 1;
+       }
+
+       uncore_pmu_disable_events();
+again:
+       uncore_pmu_ack_status(status);
+
+       for_each_set_bit(bit, (unsigned long *)&status, UNCORE_NUM_COUNTERS) {
+               struct perf_event *event = uncore_node->events[bit];
+
+               handled++;
+
+               if (!uncore_pmu_save_and_restart(event))
+                       continue;
+
+               data.period = event->hw.last_period;
+
+               if (perf_event_overflow(event, 1, &data, regs))
+                       uncore_pmu_stop(event, 0);
+       }
+
+       /*
+        * Repeat if there is more work to be done:
+        */
+       status = uncore_pmu_get_status();
+       if (status)
+               goto again;
+
+       uncore_pmu_enable_events();
+       return handled;
+}
+
+void __init init_uncore_pmu(void)
+{
+       union cpuid01_eax eax;
+       unsigned int unused;
+       unsigned int model;
+       int i;
+
+       if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+               return;
+
+       cpuid(1, &eax.full, &unused, &unused, &unused);
+
+       /* Check CPUID signatures: 06_1AH, 06_1EH, 06_1FH */
+       model = eax.split.model | (eax.split.ext_model << 4);
+       if (eax.split.family != 6 || (model != 0x1A && model != 0x1E && model != 0x1F))
+               return;
+
+       pr_cont("Nehalem uncore pmu, \n");
+
+       for (i = 0; i < MAX_NUMNODES; i++)
+               spin_lock_init(&uncore_events[i].lock);
+
+       perf_pmu_register(&uncore_pmu);
+       uncore_pmu_initialized = true;
+}
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +80 -1
new file mode 100644
index 0000000..33b9b5e
@@ -0,0 +1,80 @@
+#include <linux/perf_event.h>
+#include <linux/capability.h>
+#include <linux/notifier.h>
+#include <linux/hardirq.h>
+#include <linux/kprobes.h>
+#include <linux/module.h>
+#include <linux/kdebug.h>
+#include <linux/sched.h>
+#include <linux/uaccess.h>
+#include <linux/slab.h>
+#include <linux/highmem.h>
+#include <linux/cpu.h>
+#include <linux/bitops.h>
+
+#include <asm/apic.h>
+#include <asm/stacktrace.h>
+#include <asm/nmi.h>
+#include <asm/compat.h>
+
+#define MSR_UNCORE_PERF_GLOBAL_CTRL    0x391
+#define MSR_UNCORE_PERF_GLOBAL_STATUS  0x392
+#define MSR_UNCORE_PERF_GLOBAL_OVF_CTRL        0x393
+#define MSR_UNCORE_FIXED_CTR0          0x394
+#define MSR_UNCORE_FIXED_CTR_CTRL      0x395
+#define MSR_UNCORE_ADDR_OPCODE_MATCH   0x396
+
+#define MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_CORE0 (1ULL << 48)
+#define MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_FRZ (1ULL << 63)
+
+#define MSR_UNCORE_PERF_GLOBAL_STATUS_OVF_PMI  (1ULL << 61)
+#define MSR_UNCORE_PERF_GLOBAL_STATUS_CHG       (1ULL << 63)
+
+#define MSR_UNCORE_PMC0                        0x3b0
+
+#define MSR_UNCORE_PERFEVTSEL0         0x3c0
+
+#define UNCORE_EVENTSEL_EVENT                  0x000000FFULL
+#define UNCORE_EVENTSEL_UMASK                  0x0000FF00ULL
+#define UNCORE_EVENTSEL_OCC_CTR_RST            (1ULL << 17)
+#define UNCORE_EVENTSEL_EDGE                   (1ULL << 18)
+#define UNCORE_EVENTSEL_PMI                    (1ULL << 20)
+#define UNCORE_EVENTSEL_ENABLE                 (1ULL << 22)
+#define UNCORE_EVENTSEL_INV                    (1ULL << 23)
+#define UNCORE_EVENTSEL_CMASK                  0xFF000000ULL
+
+#define UNCORE_RAW_EVENT_MASK          \
+       (UNCORE_EVENTSEL_EVENT |        \
+        UNCORE_EVENTSEL_UMASK |        \
+        UNCORE_EVENTSEL_EDGE  |        \
+        UNCORE_EVENTSEL_INV   |        \
+        UNCORE_EVENTSEL_CMASK)
+
+#define UNCORE_CNTVAL_BITS     48
+
+#define UNCORE_NUM_COUNTERS 8
+
+union cpuid01_eax {
+       struct {
+               unsigned int stepping:4;
+               unsigned int model:4;
+               unsigned int family:4;
+               unsigned int type:2;
+               unsigned int reserve:2;
+               unsigned int ext_model:4;
+               unsigned int ext_family:4;
+       } split;
+       unsigned int full;
+};
+
+struct node_hw_events {
+       struct perf_event *events[UNCORE_NUM_COUNTERS]; /* in counter order */
+       int n_events;
+       struct spinlock lock;
+       int enabled;
+};
+
+struct uncore_cpu_hw_events {
+       unsigned long active_mask[BITS_TO_LONGS(UNCORE_NUM_COUNTERS)];
+};
+
--

1.7.1

author
#2 lore raw
Quoted 22 lines · Stephane Eranian

Lin,

Quoted 19 lines · Lin Ming

For the background of Nehalem uncore pmu, see Intel SDM Volume 3B "30.6.2 Performance Monitoring Facility in the Uncore"

1. data structure

struct node_hw_events {
       struct perf_event *events[UNCORE_NUM_COUNTERS];
       int n_events;
       struct spinlock lock;
       int enabled;
};

struct node_hw_events is the per node structure.
"lock" protects add/delete events to uncore pmu.

struct uncore_cpu_hw_events {
       unsigned long active_mask[BITS_TO_LONGS(UNCORE_NUM_COUNTERS)];
};
Quoted 8 lines
Quoted 5 lines

struct uncore_cpu_hw_events is the per logical cpu structure. "active_mask" represents the counters used by the cpu.
For example, if bit 3, 6 are set for cpuX, then it means uncore counter 3 and 6 are used by cpuX.

I would advise you allocate your uncore_events[] table dynamically using kmalloc_node(). That way you avoid unnecessary remote memory accesses.

Good point. Will do this.

Quoted 3 lines

Furthermore, the patch is missing support for the fixed uncore counter. It is very useful as it allows measuring some reference cycles at the socket

Yes, I'll add fixed uncore counter and possibly also the uncore address/opcode match thing.

Quoted 2 lines

level. You have 8+1 counter total. You need to define some encoding for UNC_CPU_CLK.

Could you explain a bit more? What's the encoding for UNC_CPU_CLK?

Quoted 6 lines

Another tidbit is about handling of exclude_*. Uncore PMU does measure at all priv level all the time. I think it would be wise to fail if the user passes
anything different from exclude_* = 0. Otherwise you think you are measuring at some levels but in reality you are not.

Yes, will do this.

Quoted 4 lines
Quoted 4 lines

2. Uncore pmu NMI handling

Quoted 8 lines
Quoted 6 lines

Every core in the socket can be programmed to receive uncore counter overflow interrupt.

In this draft implementation, each core handles the overflow interrupt caused by the counters with bit set in "active_mask".

Seems like in your model, interrupting all cores is the only solution given that you can program uncore events from any cores on the socket.

Do you see some potential problem with this model?

And do you have some ideas about the issue I mentioned in PATCH 0/3? Copy it here.

4. Issues

How to eliminate the duplicate counter values accumulated by multi child processes on the same socket?

perf stat -e ru0101 -- make -j4

Assume the 4 "make" child processes are running on the same socket and counting uncore raw event "0101", and the counter value read by them are val0, val1, val2, val3.

Then the final counter result given by "perf stat" will be "val0 + val1
+ val2 + val3".

But this is obvious wrong, because the uncore counter is shared by all cores in the socket, so the final result should not be accumulated.

Many thanks,
Lin Ming

Quoted 441 lines
Quoted 441 lines

---
 arch/x86/include/asm/msr-index.h              |    1 +
 arch/x86/kernel/cpu/perf_event.c              |    7 +-
 arch/x86/kernel/cpu/perf_event_intel_uncore.c |  280 +++++++++++++++++++++++++
 arch/x86/kernel/cpu/perf_event_intel_uncore.h |   80 +++++++
 4 files changed, 367 insertions(+), 1 deletions(-)
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.c
 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.h
arch/x86/include/asm/msr-index.h +1 -0
index 3ea3dc4..816fb4b 100644
@@ -81,6 +81,7 @@
 #define DEBUGCTLMSR_BTS_OFF_OS         (1UL <<  9)
 #define DEBUGCTLMSR_BTS_OFF_USR                (1UL << 10)
 #define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI (1UL << 11)
+#define DEBUGCTLMSR_ENABLE_UNCORE_PMI  (1UL << 13)
 
 #define MSR_IA32_MC0_CTL               0x00000400
 #define MSR_IA32_MC0_STATUS            0x00000401
arch/x86/kernel/cpu/perf_event.c +6 -1
index 7cea0f4..cca07b4 100644
@@ -1215,6 +1215,8 @@ struct pmu_nmi_state {
 
 static DEFINE_PER_CPU(struct pmu_nmi_state, pmu_nmi);
 
+static int uncore_pmu_handle_irq(struct pt_regs *regs);
+
 static int __kprobes
 perf_event_nmi_handler(struct notifier_block *self,
                        unsigned long cmd, void *__args)
@@ -1249,7 +1251,8 @@ perf_event_nmi_handler(struct notifier_block *self,
 
       apic_write(APIC_LVTPC, APIC_DM_NMI);
 
-       handled = x86_pmu.handle_irq(args->regs);
+       handled = uncore_pmu_handle_irq(args->regs);
+       handled += x86_pmu.handle_irq(args->regs);
       if (!handled)
               return NOTIFY_DONE;
 
@@ -1305,6 +1308,7 @@ x86_get_event_constraints(struct cpu_hw_events *cpuc, struct perf_event *event)
 #include "perf_event_intel_lbr.c"
 #include "perf_event_intel_ds.c"
 #include "perf_event_intel.c"
+#include "perf_event_intel_uncore.c"
 
 static int __cpuinit
 x86_pmu_notifier(struct notifier_block *self, unsigned long action, void *hcpu)
@@ -1360,6 +1364,7 @@ void __init init_hw_perf_events(void)
 
       switch (boot_cpu_data.x86_vendor) {
       case X86_VENDOR_INTEL:
+               init_uncore_pmu();
               err = intel_pmu_init();
               break;
       case X86_VENDOR_AMD:
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +280 -0
new file mode 100644
index 0000000..fafa0f9
@@ -0,0 +1,280 @@
+#include "perf_event_intel_uncore.h"
+
+static struct node_hw_events uncore_events[MAX_NUMNODES];
+static DEFINE_PER_CPU(struct uncore_cpu_hw_events, uncore_cpu_hw_events);
+static bool uncore_pmu_initialized;
+
+static void uncore_pmu_enable_event(struct perf_event *event)
+{
+       struct hw_perf_event *hwc = &event->hw;
+
+       wrmsrl(hwc->config_base + hwc->idx, hwc->config | UNCORE_EVENTSEL_ENABLE);
+}
+
+static void uncore_pmu_disable_event(struct perf_event *event)
+{
+       struct hw_perf_event *hwc = &event->hw;
+
+       wrmsrl(hwc->config_base + hwc->idx, hwc->config);
+}
+
+static void uncore_pmu_disable_events(void)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       int node = numa_node_id();
+       int bit;
+
+       for_each_set_bit(bit, cpuc->active_mask, UNCORE_NUM_COUNTERS)
+               uncore_pmu_disable_event(uncore_events[node].events[bit]);
+}
+
+static void uncore_pmu_enable_events(void)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       int node = numa_node_id();
+       int bit;
+
+       for_each_set_bit(bit, cpuc->active_mask, UNCORE_NUM_COUNTERS)
+               uncore_pmu_disable_event(uncore_events[node].events[bit]);
+}
+
+static void uncore_pmu_global_enable(void)
+{
+       u64 ctrl;
+
+       /* (0xFULL << 48): all 4 cores will receive NMI */
+       ctrl = ((1 << UNCORE_NUM_COUNTERS) - 1) | (0xFULL << 48);
+
+       wrmsrl(MSR_UNCORE_PERF_GLOBAL_CTRL, ctrl);
+}
+
+static void uncore_perf_event_destroy(struct perf_event *event)
+{
+       atomic_dec(&active_events);
+}
+
+static int uncore_pmu_event_init(struct perf_event *event)
+{
+       struct hw_perf_event *hwc = &event->hw;
+
+       if (!uncore_pmu_initialized)
+               return -ENOENT;
+
+       switch (event->attr.type) {
+       case PERF_TYPE_UNCORE:
+               break;
+
+       default:
+               return -ENOENT;
+       }
+
+       atomic_inc(&active_events);
+
+       event->destroy = uncore_perf_event_destroy;
+
+       hwc->idx = -1;
+       hwc->config = (event->attr.config & UNCORE_RAW_EVENT_MASK) | UNCORE_EVENTSEL_PMI;
+       hwc->config_base = MSR_UNCORE_PERFEVTSEL0;
+       hwc->event_base = MSR_UNCORE_PMC0;
+
+       return 0;
+}
+
+static void uncore_pmu_start(struct perf_event *event, int flags)
+{
+       if (flags & PERF_EF_RELOAD)
+               x86_perf_event_set_period(event);
+
+       uncore_pmu_enable_event(event);
+
+       perf_event_update_userpage(event);
+}
+
+static void uncore_pmu_stop(struct perf_event *event, int flags)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       struct hw_perf_event *hwc = &event->hw;
+
+       if (__test_and_clear_bit(hwc->idx, cpuc->active_mask))
+               uncore_pmu_disable_event(event);
+
+       if (flags & PERF_EF_UPDATE)
+               x86_perf_event_update(event, UNCORE_CNTVAL_BITS);
+}
+
+static int uncore_pmu_add(struct perf_event *event, int flags)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       int node = numa_node_id();
+       int ret = 1;
+       int i;
+       u64 ctrl;
+
+       spin_lock(&uncore_events[node].lock);
+
+       for (i = 0; i < UNCORE_NUM_COUNTERS; i++) {
+               if (!uncore_events[node].events[i]) {
+                       uncore_events[node].events[i] = event;
+                       uncore_events[node].n_events++;
+
+                       event->hw.idx = i;
+                       __set_bit(i, cpuc->active_mask);
+                       if (flags & PERF_EF_START)
+                               uncore_pmu_start(event, PERF_EF_RELOAD);
+                       ret = 0;
+                       break;
+               }
+       }
+
+       /*
+        * PMI delivery due to an uncore counter overflow is enabled by
+        * setting IA32_DEBUG_CTL.Offcore_PMI_EN to 1.
+        */
+       if (uncore_events[node].n_events == 1) {
+               rdmsrl(MSR_IA32_DEBUGCTLMSR, ctrl);
+               wrmsrl(MSR_IA32_DEBUGCTLMSR, ctrl | DEBUGCTLMSR_ENABLE_UNCORE_PMI);
+       }
+
+       if (unlikely(!uncore_events[node].enabled)) {
+               uncore_pmu_global_enable();
+               uncore_events[node].enabled = 1;
+       }
+
+       spin_unlock(&uncore_events[node].lock);
+
+       return ret;
+}
+
+static void uncore_pmu_del(struct perf_event *event, int flags)
+{
+       int node = numa_node_id();
+       struct hw_perf_event *hwc = &event->hw;
+       int i;
+
+       spin_lock(&uncore_events[node].lock);
+
+       for (i = 0; i < UNCORE_NUM_COUNTERS; i++) {
+               if (uncore_events[node].events[i] == event) {
+                       uncore_events[node].events[hwc->idx] = NULL;
+                       uncore_events[node].n_events--;
+
+                       uncore_pmu_stop(event, PERF_EF_UPDATE);
+                       break;
+               }
+       }
+
+       spin_unlock(&uncore_events[node].lock);
+}
+
+static void uncore_pmu_read(struct perf_event *event)
+{
+       x86_perf_event_update(event, UNCORE_CNTVAL_BITS);
+}
+
+static struct pmu uncore_pmu = {
+       .event_init     = uncore_pmu_event_init,
+       .add            = uncore_pmu_add,
+       .del            = uncore_pmu_del,
+       .start          = uncore_pmu_start,
+       .stop           = uncore_pmu_stop,
+       .read           = uncore_pmu_read,
+};
+
+
+static inline u64 uncore_pmu_get_status(void)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       u64 status;
+
+       rdmsrl(MSR_UNCORE_PERF_GLOBAL_STATUS, status);
+
+       return status & (*(u64 *)cpuc->active_mask |
+               MSR_UNCORE_PERF_GLOBAL_STATUS_OVF_PMI | MSR_UNCORE_PERF_GLOBAL_STATUS_CHG);
+}
+
+static inline void uncore_pmu_ack_status(u64 ack)
+{
+       wrmsrl(MSR_UNCORE_PERF_GLOBAL_OVF_CTRL, ack);
+}
+
+static int uncore_pmu_save_and_restart(struct perf_event *event)
+{
+       x86_perf_event_update(event, UNCORE_CNTVAL_BITS);
+       return x86_perf_event_set_period(event);
+}
+
+int uncore_pmu_handle_irq(struct pt_regs *regs)
+{
+       struct perf_sample_data data;
+       struct node_hw_events *uncore_node;
+       int node;
+       int bit;
+       u64 status;
+       int handled = 0;
+
+       perf_sample_data_init(&data, 0);
+
+       node = numa_node_id();
+       uncore_node = &uncore_events[node];
+
+       status = uncore_pmu_get_status();
+       if (!status) {
+               apic_write(APIC_LVTPC, APIC_DM_NMI);
+
+               return 1;
+       }
+
+       uncore_pmu_disable_events();
+again:
+       uncore_pmu_ack_status(status);
+
+       for_each_set_bit(bit, (unsigned long *)&status, UNCORE_NUM_COUNTERS) {
+               struct perf_event *event = uncore_node->events[bit];
+
+               handled++;
+
+               if (!uncore_pmu_save_and_restart(event))
+                       continue;
+
+               data.period = event->hw.last_period;
+
+               if (perf_event_overflow(event, 1, &data, regs))
+                       uncore_pmu_stop(event, 0);
+       }
+
+       /*
+        * Repeat if there is more work to be done:
+        */
+       status = uncore_pmu_get_status();
+       if (status)
+               goto again;
+
+       uncore_pmu_enable_events();
+       return handled;
+}
+
+void __init init_uncore_pmu(void)
+{
+       union cpuid01_eax eax;
+       unsigned int unused;
+       unsigned int model;
+       int i;
+
+       if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+               return;
+
+       cpuid(1, &eax.full, &unused, &unused, &unused);
+
+       /* Check CPUID signatures: 06_1AH, 06_1EH, 06_1FH */
+       model = eax.split.model | (eax.split.ext_model << 4);
+       if (eax.split.family != 6 || (model != 0x1A && model != 0x1E && model != 0x1F))
+               return;
+
+       pr_cont("Nehalem uncore pmu, \n");
+
+       for (i = 0; i < MAX_NUMNODES; i++)
+               spin_lock_init(&uncore_events[i].lock);
+
+       perf_pmu_register(&uncore_pmu);
+       uncore_pmu_initialized = true;
+}
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +80 -1
new file mode 100644
index 0000000..33b9b5e
@@ -0,0 +1,80 @@
+#include <linux/perf_event.h>
+#include <linux/capability.h>
+#include <linux/notifier.h>
+#include <linux/hardirq.h>
+#include <linux/kprobes.h>
+#include <linux/module.h>
+#include <linux/kdebug.h>
+#include <linux/sched.h>
+#include <linux/uaccess.h>
+#include <linux/slab.h>
+#include <linux/highmem.h>
+#include <linux/cpu.h>
+#include <linux/bitops.h>
+
+#include <asm/apic.h>
+#include <asm/stacktrace.h>
+#include <asm/nmi.h>
+#include <asm/compat.h>
+
+#define MSR_UNCORE_PERF_GLOBAL_CTRL    0x391
+#define MSR_UNCORE_PERF_GLOBAL_STATUS  0x392
+#define MSR_UNCORE_PERF_GLOBAL_OVF_CTRL        0x393
+#define MSR_UNCORE_FIXED_CTR0          0x394
+#define MSR_UNCORE_FIXED_CTR_CTRL      0x395
+#define MSR_UNCORE_ADDR_OPCODE_MATCH   0x396
+
+#define MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_CORE0 (1ULL << 48)
+#define MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_FRZ (1ULL << 63)
+
+#define MSR_UNCORE_PERF_GLOBAL_STATUS_OVF_PMI  (1ULL << 61)
+#define MSR_UNCORE_PERF_GLOBAL_STATUS_CHG       (1ULL << 63)
+
+#define MSR_UNCORE_PMC0                        0x3b0
+
+#define MSR_UNCORE_PERFEVTSEL0         0x3c0
+
+#define UNCORE_EVENTSEL_EVENT                  0x000000FFULL
+#define UNCORE_EVENTSEL_UMASK                  0x0000FF00ULL
+#define UNCORE_EVENTSEL_OCC_CTR_RST            (1ULL << 17)
+#define UNCORE_EVENTSEL_EDGE                   (1ULL << 18)
+#define UNCORE_EVENTSEL_PMI                    (1ULL << 20)
+#define UNCORE_EVENTSEL_ENABLE                 (1ULL << 22)
+#define UNCORE_EVENTSEL_INV                    (1ULL << 23)
+#define UNCORE_EVENTSEL_CMASK                  0xFF000000ULL
+
+#define UNCORE_RAW_EVENT_MASK          \
+       (UNCORE_EVENTSEL_EVENT |        \
+        UNCORE_EVENTSEL_UMASK |        \
+        UNCORE_EVENTSEL_EDGE  |        \
+        UNCORE_EVENTSEL_INV   |        \
+        UNCORE_EVENTSEL_CMASK)
+
+#define UNCORE_CNTVAL_BITS     48
+
+#define UNCORE_NUM_COUNTERS 8
+
+union cpuid01_eax {
+       struct {
+               unsigned int stepping:4;
+               unsigned int model:4;
+               unsigned int family:4;
+               unsigned int type:2;
+               unsigned int reserve:2;
+               unsigned int ext_model:4;
+               unsigned int ext_family:4;
+       } split;
+       unsigned int full;
+};
+
+struct node_hw_events {
+       struct perf_event *events[UNCORE_NUM_COUNTERS]; /* in counter order */
+       int n_events;
+       struct spinlock lock;
+       int enabled;
+};
+
+struct uncore_cpu_hw_events {
+       unsigned long active_mask[BITS_TO_LONGS(UNCORE_NUM_COUNTERS)];
+};
+
--

1.7.1

#3 lore raw
Quoted 32 lines · Lin Ming
Quoted 22 lines · Stephane Eranian

Lin,

Quoted 19 lines · Lin Ming

For the background of Nehalem uncore pmu, see Intel SDM Volume 3B "30.6.2 Performance Monitoring Facility in the Uncore"

1. data structure

struct node_hw_events {
       struct perf_event *events[UNCORE_NUM_COUNTERS]; int n_events;
       struct spinlock lock;
       int enabled;
};

struct node_hw_events is the per node structure.
"lock" protects add/delete events to uncore pmu.

struct uncore_cpu_hw_events {
       unsigned long active_mask[BITS_TO_LONGS(UNCORE_NUM_COUNTERS)]; };

Quoted 8 lines
Quoted 5 lines

struct uncore_cpu_hw_events is the per logical cpu structure. "active_mask" represents the counters used by the cpu.
For example, if bit 3, 6 are set for cpuX, then it means uncore counter 3 and 6 are used by cpuX.

I would advise you allocate your uncore_events[] table dynamically using kmalloc_node(). That way you avoid unnecessary remote memory accesses.

Quoted 9 lines

Good point. Will do this.

Quoted 3 lines

Furthermore, the patch is missing support for the fixed uncore counter. It is very useful as it allows measuring some reference cycles at the socket

Yes, I'll add fixed uncore counter and possibly also the uncore address/opcode match thing.

Good.

Quoted 5 lines
Quoted 2 lines

level. You have 8+1 counter total. You need to define some encoding for UNC_CPU_CLK.

Could you explain a bit more? What's the encoding for UNC_CPU_CLK?

Well, there is no encoding given it is a fixed counter. This is similar to the issue that exists today with core PMU and unhalted_reference_cycles vs unhalted_core_cycles for the fixed counters. Unfortunately, today you cannot name unhalted_reference_cycles. I'd like to have it because in many situations it is better than unhalted_core_cycles.

You need to pick an encoding for this event so it can be named and passed in attr.config.

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

2. Uncore pmu NMI handling

Every core in the socket can be programmed to receive uncore counter overflow interrupt.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

In this draft implementation, each core handles the overflow interrupt caused by the counters with bit set in "active_mask".

Seems like in your model, interrupting all cores is the only solution given that you can program uncore events from any cores on the socket.

Do you see some potential problem with this model?

You are interrupting other CPUs for potentially nothing. So you incur some overhead.

But it may also be interesting to snapshot the IP across all CPUs to determine where they all are. In other words, use uncore PMU to get a global view of the cores and that's where UNC_CPU_CLK comes in handy.

Quoted 7 lines

And do you have some ideas about the issue I mentioned in PATCH 0/3? Copy it here.

4. Issues

How to eliminate the duplicate counter values accumulated by multi child processes on the same socket?

I think using uncore PMU to measure per-thread is pretty much useless. Maybe it should not even be allowed. There is no way you can correlate the counts you're getting to a place in your program. Or put differently, sampling in per-thread mode using uncore is useless.

Given how perf works in system-wide mode, i.e., automatic aggregation you'll have the problem regardless of the mode. I think for uncore, you need to measure from only one core per socket. That's the only way to get the meaningful counts. You can do this explicitly using the -C mode. I believe the tool should print a warning or refuse to do it if you just pass -a (system-wide mode).

Quoted 460 lines

perf stat -e ru0101 -- make -j4

Assume the 4 "make" child processes are running on the same socket and counting uncore raw event "0101", and the counter value read by them are val0, val1, val2, val3.

Then the final counter result given by "perf stat" will be "val0 + val1
+ val2 + val3".

But this is obvious wrong, because the uncore counter is shared by all cores in the socket, so the final result should not be accumulated.

Many thanks,
Lin Ming

Quoted 441 lines
Quoted 441 lines

---
 arch/x86/include/asm/msr-index.h              |    1 + arch/x86/kernel/cpu/perf_event.c              |    7 +- arch/x86/kernel/cpu/perf_event_intel_uncore.c |  280 +++++++++++++++++++++++++ arch/x86/kernel/cpu/perf_event_intel_uncore.h |   80 +++++++

 4 files changed, 367 insertions(+), 1 deletions(-)

 create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.c create mode 100644 arch/x86/kernel/cpu/perf_event_intel_uncore.h

arch/x86/include/asm/msr-index.h +1 -0
index 3ea3dc4..816fb4b 100644
@@ -81,6 +81,7 @@
  #define DEBUGCTLMSR_BTS_OFF_OS         (1UL <<  9)
  #define DEBUGCTLMSR_BTS_OFF_USR                (1UL << 10)
  #define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI (1UL << 11)
+#define DEBUGCTLMSR_ENABLE_UNCORE_PMI  (1UL << 13)
 
  #define MSR_IA32_MC0_CTL               0x00000400
  #define MSR_IA32_MC0_STATUS            0x00000401
arch/x86/kernel/cpu/perf_event.c +2 -0
index 7cea0f4..cca07b4 100644
@@ -1215,6 +1215,8 @@ struct pmu_nmi_state {
 
  static DEFINE_PER_CPU(struct pmu_nmi_state, pmu_nmi);
 
+static int uncore_pmu_handle_irq(struct pt_regs *regs);
+

 static int __kprobes perf_event_nmi_handler(struct notifier_block *self, unsigned long cmd, void *__args)

arch/x86/kernel/cpu/perf_event.c +3 -1
@@ -1249,7 +1251,8 @@ perf_event_nmi_handler(struct notifier_block *self,
 
        apic_write(APIC_LVTPC, APIC_DM_NMI);
 
-       handled = x86_pmu.handle_irq(args->regs);
+       handled = uncore_pmu_handle_irq(args->regs);
+       handled += x86_pmu.handle_irq(args->regs);
        if (!handled)
                return NOTIFY_DONE;
 
@@ -1305,6 +1308,7 @@ x86_get_event_constraints(struct cpu_hw_events *cpuc, struct perf_event *event)
  #include "perf_event_intel_lbr.c"
  #include "perf_event_intel_ds.c"
  #include "perf_event_intel.c"
+#include "perf_event_intel_uncore.c"
 

 static int __cpuinit
 x86_pmu_notifier(struct notifier_block *self, unsigned long action, void *hcpu)

arch/x86/kernel/cpu/perf_event.c +0 -0
@@ -1360,6 +1364,7 @@ void __init init_hw_perf_events(void)
 
        switch (boot_cpu_data.x86_vendor) {

       case X86_VENDOR_INTEL:
+               init_uncore_pmu();
               err = intel_pmu_init();
               break;
       case X86_VENDOR_AMD:

arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +280 -0
new file mode 100644
index 0000000..fafa0f9
@@ -0,0 +1,280 @@
+#include "perf_event_intel_uncore.h"
+
+static struct node_hw_events uncore_events[MAX_NUMNODES];
+static DEFINE_PER_CPU(struct uncore_cpu_hw_events, uncore_cpu_hw_events);
+static bool uncore_pmu_initialized;
+
+static void uncore_pmu_enable_event(struct perf_event *event)
+{
+       struct hw_perf_event *hwc = &event->hw;
+
+       wrmsrl(hwc->config_base + hwc->idx, hwc->config | UNCORE_EVENTSEL_ENABLE);
+}
+
+static void uncore_pmu_disable_event(struct perf_event *event)
+{
+       struct hw_perf_event *hwc = &event->hw;
+
+       wrmsrl(hwc->config_base + hwc->idx, hwc->config);
+}
+
+static void uncore_pmu_disable_events(void)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       int node = numa_node_id();
+       int bit;
+
+       for_each_set_bit(bit, cpuc->active_mask, UNCORE_NUM_COUNTERS)
+               uncore_pmu_disable_event(uncore_events[node].events[bit]);
+}
+
+static void uncore_pmu_enable_events(void)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       int node = numa_node_id();
+       int bit;
+
+       for_each_set_bit(bit, cpuc->active_mask, UNCORE_NUM_COUNTERS)
+               uncore_pmu_disable_event(uncore_events[node].events[bit]);
+}
+
+static void uncore_pmu_global_enable(void)
+{
+       u64 ctrl;
+
+       /* (0xFULL << 48): all 4 cores will receive NMI */
+       ctrl = ((1 << UNCORE_NUM_COUNTERS) - 1) | (0xFULL << 48);
+
+       wrmsrl(MSR_UNCORE_PERF_GLOBAL_CTRL, ctrl);
+}
+
+static void uncore_perf_event_destroy(struct perf_event *event)
+{
+       atomic_dec(&active_events);
+}
+
+static int uncore_pmu_event_init(struct perf_event *event)
+{
+       struct hw_perf_event *hwc = &event->hw;
+
+       if (!uncore_pmu_initialized)
+               return -ENOENT;
+
+       switch (event->attr.type) {
+       case PERF_TYPE_UNCORE:
+               break;
+
+       default:
+               return -ENOENT;
+       }
+
+       atomic_inc(&active_events);
+
+       event->destroy = uncore_perf_event_destroy;
+
+       hwc->idx = -1;
+       hwc->config = (event->attr.config & UNCORE_RAW_EVENT_MASK) | UNCORE_EVENTSEL_PMI;
+       hwc->config_base = MSR_UNCORE_PERFEVTSEL0;
+       hwc->event_base = MSR_UNCORE_PMC0;
+
+       return 0;
+}
+
+static void uncore_pmu_start(struct perf_event *event, int flags)
+{
+       if (flags & PERF_EF_RELOAD)
+               x86_perf_event_set_period(event);
+
+       uncore_pmu_enable_event(event);
+
+       perf_event_update_userpage(event);
+}
+
+static void uncore_pmu_stop(struct perf_event *event, int flags)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       struct hw_perf_event *hwc = &event->hw;
+
+       if (__test_and_clear_bit(hwc->idx, cpuc->active_mask))
+               uncore_pmu_disable_event(event);
+
+       if (flags & PERF_EF_UPDATE)
+               x86_perf_event_update(event, UNCORE_CNTVAL_BITS);
+}
+
+static int uncore_pmu_add(struct perf_event *event, int flags)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       int node = numa_node_id();
+       int ret = 1;
+       int i;
+       u64 ctrl;
+
+       spin_lock(&uncore_events[node].lock);
+
+       for (i = 0; i < UNCORE_NUM_COUNTERS; i++) {
+               if (!uncore_events[node].events[i]) {
+                       uncore_events[node].events[i] = event;
+                       uncore_events[node].n_events++;
+
+                       event->hw.idx = i;
+                       __set_bit(i, cpuc->active_mask);
+                       if (flags & PERF_EF_START)
+                               uncore_pmu_start(event, PERF_EF_RELOAD);
+                       ret = 0;
+                       break;
+               }
+       }
+
+       /*
+        * PMI delivery due to an uncore counter overflow is enabled by
+        * setting IA32_DEBUG_CTL.Offcore_PMI_EN to 1.
+        */
+       if (uncore_events[node].n_events == 1) {
+               rdmsrl(MSR_IA32_DEBUGCTLMSR, ctrl);
+               wrmsrl(MSR_IA32_DEBUGCTLMSR, ctrl | DEBUGCTLMSR_ENABLE_UNCORE_PMI);
+       }
+
+       if (unlikely(!uncore_events[node].enabled)) {
+               uncore_pmu_global_enable();
+               uncore_events[node].enabled = 1;
+       }
+
+       spin_unlock(&uncore_events[node].lock);
+
+       return ret;
+}
+
+static void uncore_pmu_del(struct perf_event *event, int flags)
+{
+       int node = numa_node_id();
+       struct hw_perf_event *hwc = &event->hw;
+       int i;
+
+       spin_lock(&uncore_events[node].lock);
+
+       for (i = 0; i < UNCORE_NUM_COUNTERS; i++) {
+               if (uncore_events[node].events[i] == event) {
+                       uncore_events[node].events[hwc->idx] = NULL;
+                       uncore_events[node].n_events--;
+
+                       uncore_pmu_stop(event, PERF_EF_UPDATE);
+                       break;
+               }
+       }
+
+       spin_unlock(&uncore_events[node].lock);
+}
+
+static void uncore_pmu_read(struct perf_event *event)
+{
+       x86_perf_event_update(event, UNCORE_CNTVAL_BITS);
+}
+
+static struct pmu uncore_pmu = {
+       .event_init     = uncore_pmu_event_init,
+       .add            = uncore_pmu_add,
+       .del            = uncore_pmu_del,
+       .start          = uncore_pmu_start,
+       .stop           = uncore_pmu_stop,
+       .read           = uncore_pmu_read,
+};
+
+
+static inline u64 uncore_pmu_get_status(void)
+{
+       struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+       u64 status;
+
+       rdmsrl(MSR_UNCORE_PERF_GLOBAL_STATUS, status);
+
+       return status & (*(u64 *)cpuc->active_mask |
+               MSR_UNCORE_PERF_GLOBAL_STATUS_OVF_PMI | MSR_UNCORE_PERF_GLOBAL_STATUS_CHG);
+}
+
+static inline void uncore_pmu_ack_status(u64 ack)
+{
+       wrmsrl(MSR_UNCORE_PERF_GLOBAL_OVF_CTRL, ack);
+}
+
+static int uncore_pmu_save_and_restart(struct perf_event *event)
+{
+       x86_perf_event_update(event, UNCORE_CNTVAL_BITS);
+       return x86_perf_event_set_period(event);
+}
+
+int uncore_pmu_handle_irq(struct pt_regs *regs)
+{
+       struct perf_sample_data data;
+       struct node_hw_events *uncore_node;
+       int node;
+       int bit;
+       u64 status;
+       int handled = 0;
+
+       perf_sample_data_init(&data, 0);
+
+       node = numa_node_id();
+       uncore_node = &uncore_events[node];
+
+       status = uncore_pmu_get_status();
+       if (!status) {
+               apic_write(APIC_LVTPC, APIC_DM_NMI);
+
+               return 1;
+       }
+
+       uncore_pmu_disable_events();
+again:
+       uncore_pmu_ack_status(status);
+
+       for_each_set_bit(bit, (unsigned long *)&status, UNCORE_NUM_COUNTERS) {
+               struct perf_event *event = uncore_node->events[bit];
+
+               handled++;
+
+               if (!uncore_pmu_save_and_restart(event))
+                       continue;
+
+               data.period = event->hw.last_period;
+
+               if (perf_event_overflow(event, 1, &data, regs))
+                       uncore_pmu_stop(event, 0);
+       }
+
+       /*
+        * Repeat if there is more work to be done:
+        */
+       status = uncore_pmu_get_status();
+       if (status)
+               goto again;
+
+       uncore_pmu_enable_events();
+       return handled;
+}
+
+void __init init_uncore_pmu(void)
+{
+       union cpuid01_eax eax;
+       unsigned int unused;
+       unsigned int model;
+       int i;
+
+       if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL)
+               return;
+
+       cpuid(1, &eax.full, &unused, &unused, &unused);
+
+       /* Check CPUID signatures: 06_1AH, 06_1EH, 06_1FH */
+       model = eax.split.model | (eax.split.ext_model << 4);
+       if (eax.split.family != 6 || (model != 0x1A && model != 0x1E && model != 0x1F))
+               return;
+
+       pr_cont("Nehalem uncore pmu, \n");
+
+       for (i = 0; i < MAX_NUMNODES; i++)
+               spin_lock_init(&uncore_events[i].lock);
+
+       perf_pmu_register(&uncore_pmu);
+       uncore_pmu_initialized = true;
+}
arch/x86/kernel/cpu/perf_event_intel_uncore.h new file +80 -1
new file mode 100644
index 0000000..33b9b5e
@@ -0,0 +1,80 @@
+#include <linux/perf_event.h>
+#include <linux/capability.h>
+#include <linux/notifier.h>
+#include <linux/hardirq.h>
+#include <linux/kprobes.h>
+#include <linux/module.h>
+#include <linux/kdebug.h>
+#include <linux/sched.h>
+#include <linux/uaccess.h>
+#include <linux/slab.h>
+#include <linux/highmem.h>
+#include <linux/cpu.h>
+#include <linux/bitops.h>
+
+#include <asm/apic.h>
+#include <asm/stacktrace.h>
+#include <asm/nmi.h>
+#include <asm/compat.h>
+
+#define MSR_UNCORE_PERF_GLOBAL_CTRL    0x391
+#define MSR_UNCORE_PERF_GLOBAL_STATUS  0x392
+#define MSR_UNCORE_PERF_GLOBAL_OVF_CTRL        0x393
+#define MSR_UNCORE_FIXED_CTR0          0x394
+#define MSR_UNCORE_FIXED_CTR_CTRL      0x395
+#define MSR_UNCORE_ADDR_OPCODE_MATCH   0x396
+
+#define MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_CORE0 (1ULL << 48)
+#define MSR_UNCORE_PERF_GLOBAL_CTRL_PMI_FRZ (1ULL << 63)
+
+#define MSR_UNCORE_PERF_GLOBAL_STATUS_OVF_PMI  (1ULL << 61)
+#define MSR_UNCORE_PERF_GLOBAL_STATUS_CHG       (1ULL << 63)
+
+#define MSR_UNCORE_PMC0                        0x3b0
+
+#define MSR_UNCORE_PERFEVTSEL0         0x3c0
+
+#define UNCORE_EVENTSEL_EVENT                  0x000000FFULL
+#define UNCORE_EVENTSEL_UMASK                  0x0000FF00ULL
+#define UNCORE_EVENTSEL_OCC_CTR_RST            (1ULL << 17)
+#define UNCORE_EVENTSEL_EDGE                   (1ULL << 18)
+#define UNCORE_EVENTSEL_PMI                    (1ULL << 20)
+#define UNCORE_EVENTSEL_ENABLE                 (1ULL << 22)
+#define UNCORE_EVENTSEL_INV                    (1ULL << 23)
+#define UNCORE_EVENTSEL_CMASK                  0xFF000000ULL
+
+#define UNCORE_RAW_EVENT_MASK          \
+       (UNCORE_EVENTSEL_EVENT |        \
+        UNCORE_EVENTSEL_UMASK |        \
+        UNCORE_EVENTSEL_EDGE  |        \
+        UNCORE_EVENTSEL_INV   |        \
+        UNCORE_EVENTSEL_CMASK)
+
+#define UNCORE_CNTVAL_BITS     48
+
+#define UNCORE_NUM_COUNTERS 8
+
+union cpuid01_eax {
+       struct {
+               unsigned int stepping:4;
+               unsigned int model:4;
+               unsigned int family:4;
+               unsigned int type:2;
+               unsigned int reserve:2;
+               unsigned int ext_model:4;
+               unsigned int ext_family:4;
+       } split;
+       unsigned int full;
+};
+
+struct node_hw_events {
+       struct perf_event *events[UNCORE_NUM_COUNTERS]; /* in counter order */
+       int n_events;
+       struct spinlock lock;
+       int enabled;
+};
+
+struct uncore_cpu_hw_events {
+       unsigned long active_mask[BITS_TO_LONGS(UNCORE_NUM_COUNTERS)];
+};
+
--

1.7.1

#4 lore raw
Quoted 4 lines · Stephane Eranian

I think using uncore PMU to measure per-thread is pretty much useless. Maybe it should not even be allowed. There is no way you can correlate the counts you're getting to a place in your program. Or put differently, sampling in per-thread mode using uncore is useless.

Right, per-task uncore event should not be allowed. I think it as simple as setting pmu::task_ctx_nr = perf_invalid_context.

author
#5 lore raw
Quoted 8 lines · Peter Zijlstra
Quoted 4 lines · Stephane Eranian

I think using uncore PMU to measure per-thread is pretty much useless. Maybe it should not even be allowed. There is no way you can correlate the counts you're getting to a place in your program. Or put differently, sampling in per-thread mode using uncore is useless.

Right, per-task uncore event should not be allowed. I think it as simple as setting pmu::task_ctx_nr = perf_invalid_context.

Thanks for the comments from you and Stephane.
I'll update the patches.

Lin Ming

#6 lore raw
In reply to Lin Ming

Lin Ming <[email protected]> writes:

Looking at a an old patch.

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 24

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5 -0
+
+static void uncore_pmu_disable_events(void)
+{
+	struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+	int node = numa_node_id();

Using numa_node_id() here implies this can't be used when NUMA is turned off. Better use the package id.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only.
author
#7 lore raw
Quoted 4 lines · Andi Kleen

Lin Ming <[email protected]> writes:

Looking at a an old patch.

Quoted 8 lines

arch/x86/kernel/cpu/perf_event_intel_uncore.c line 24

Quoted 5 lines
arch/x86/kernel/cpu/perf_event_intel_uncore.c new file +5 -0
+
+static void uncore_pmu_disable_events(void)
+{
+	struct uncore_cpu_hw_events *cpuc = &__get_cpu_var(uncore_cpu_hw_events);
+	int node = numa_node_id();

Using numa_node_id() here implies this can't be used when NUMA is turned off. Better use the package id.

Right, will fix it with topology_physical_package_id.

Thanks.

Quoted 3 lines

-Andi