Performance Counters for Linux, v4

We are pleased to announce the v4 release of our performance counters subsystem implementation. The kernel changes can be picked up from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git perfcounters/core

(also in the master branch. There's also a kernel patch attached
below.)

The biggest new feature in this release is the implementation of "performance counter inheritance" for the per task counters: the ability to extend performance counters to cover the execution of child tasks too, transparently and automatically - following them to other CPUs.

This can be used to monitor a hierarchy of tasks without stopping them (or impacting them in any observable way), and extending that monitoring to all child tasks as well.

We've written a new utility: 'timec', which takes advantage of this new kernel capability:

http://redhat.com/~mingo/perfcounters/timec.c

'timec' works like /usr/bin/time, but it extends the dimension of "time" with all the metrics that hardware and software performance counters are able to capture.

For example, a full kernel build's statistics on a 16-way x86 box are:

  $ timec -e -5,-4,-3,1,2,3,5 make -j32 bzImage

Performance counter stats for 'make':

         142420.882  task clock ticks     (millisecs)

            9951033  pagefaults           (events)
             302628  context switches     (events)
              57810  CPU migrations       (events)
       208439082509  instructions         (events)
          657918810  cache references     (events)
          120243697  cache misses         (events)
         3134162468  branch misses        (events)

The CPU elapsed time metric is also a lot more accurate and finegrained than /usr/bin/time is able to offer. (the counter measures nanoseconds - it is displayed by timec up to microsecond resolution.)

Or here are the stats for a 'hackbench' run that creates and tears down a web of 1000 tasks within 3 seconds:

  $ timec -e -5,-4,-3,0,1,3,5 ./hackbench 50

Time: 3.073

Performance counter stats for './hackbench':

         23340.852  task clock ticks     (millisecs)

             13173  CPU migrations       (events)
             45992  context switches     (events)
             80053  pagefaults           (events)
       57653910508  cycles               (events)
       43456427830  instructions         (events)
          93659475  cache misses         (events)
         901367477 branch misses (events)

These stats include the summed up counter stats of all the sub-tasks propagated back and added up back to the original counter - without slowing down the workload. (i could not measure a clearly observable impact on even context-switchy workloads)

Another change in v4 is the addition of more types of "software counters":

        PERF_COUNT_CPU_CLOCK            = -1,
        PERF_COUNT_TASK_CLOCK           = -2,
        PERF_COUNT_PAGE_FAULTS          = -3,
        PERF_COUNT_CONTEXT_SWITCHES     = -4,
        PERF_COUNT_CPU_MIGRATIONS       = -5,

The page-faults and context-switches per task metric is now implemented, plus nr-of-cpu-migrations scheduler metric as well. These sw counters are available on all CPUs, regardless of hardware perf-counter support.

There's a new KernelTop profiler update as well:

http://redhat.com/~mingo/perfcounters/kerneltop.c

There's a high-pass filter useful to filter out rare entries from weighted histograms (--filter):

------------------------------------------------------------------------------
KernelTop: 9152 irqs/sec [NMI, cache-misses/cache-refs], (all, 16 CPUs)
------------------------------------------------------------------------------

  weight     events         RIP          kernel function
  ______     ______   ________________   _______________

     2.2      17028 - ffffffff802b3f3f : check_bytes_and_report
     1.4       8630 - ffffffff802b466c : init_object
     1.3        718 - ffffffff802b4031 : check_object
     1.2       2050 - ffffffff8037ae10 : copy_user_generic_string!
     1.2       6631 - ffffffff802b3d7e : slab_pad_check
     1.2       1078 - ffffffff8037ea9a : _raw_spin_lock
     1.1        864 - ffffffff802b6250 : kfree
     1.1        948 - ffffffff802b55e6 : new_slab
     1.0       1001 - ffffffff8055b1c2 : unix_stream_recvmsg
     1.0       6266 - ffffffff802b3b20 : on_freelist
     1.0        588 - ffffffff802b596d : __slab_free
     1.0        956 - ffffffff802602a2 : lock_release
     0.8        571 - ffffffff802b3ead : check_slab
     0.5       3353 - ffffffff8025e8c8 : __lock_acquire

Also a number of bugfixes and small tweaks are included i this release.

There's a handful of updates to the core perfcounter code as well, see the Git history for details.

Hardware support is still limited to Core2 and later CPUs - but the design is not limited to these CPU types - help is welcome with more CPU drivers :-)

Thanks,

Ingo, Thomas

------------------>

The latest perfcounters/core git tree can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git perfcounters/core

------------------>
Ingo Molnar (27):
performance counters: documentation
performance counters: x86 support
x86, perfcounters: read out MSR_CORE_PERF_GLOBAL_STATUS with counters disabled
perfcounters: select ANON_INODES
perfcounters, x86: simplify disable/enable of counters perfcounters, x86: clean up debug code
perfcounters: consolidate global-disable codepaths
perf counters: restructure the API
perf counters: add support for group counters
perf counters: group counter, fixes
perf counters: hw driver API
perf counters: implement PERF_COUNT_CPU_CLOCK
perf counters: consolidate hw_perf save/restore APIs
perf counters: implement PERF_COUNT_TASK_CLOCK
perf counters: add prctl interface to disable/enable counters perf counters: clean up state transitions
perf counters: update docs
x86: implement atomic64_t on 32-bit
perfcounters: restructure x86 counter math
perfcounters: implement "counter inheritance"
perfcounters: fix task clock counter
perfcounters: add context switch counter
perfcounters: add task migrations counter
perfcounters: add nr-of-faults counter
perfcounters: fix non-intel-perfmon CPUs
perfcounters, x86: fix sw counters on non-PMC CPUs
perfcounters: fix lapic initialization

Thomas Gleixner (4):
performance counters: core code
perf counters: protect them against CSTATE transitions perf counters: clean up 'raw' type API
perf counters: expand use of counter->event

 Documentation/perf-counters.txt                |  147 +++
 arch/x86/Kconfig                               |    1 +
 arch/x86/ia32/ia32entry.S                      |    3 +-
 arch/x86/include/asm/atomic_32.h               |  218 ++++
 arch/x86/include/asm/hardirq_32.h              |    1 +
 arch/x86/include/asm/hw_irq.h                  |    2 +
 arch/x86/include/asm/intel_arch_perfmon.h      |   34 +-
 arch/x86/include/asm/irq_vectors.h             |    5 +
 arch/x86/include/asm/mach-default/entry_arch.h |    5 +
 arch/x86/include/asm/pda.h                     |    1 +
 arch/x86/include/asm/thread_info.h             |    4 +-
 arch/x86/include/asm/unistd_32.h               |    1 +
 arch/x86/include/asm/unistd_64.h               |    3 +-
 arch/x86/kernel/apic.c                         |    2 +
 arch/x86/kernel/cpu/Makefile                   |   12 +-
 arch/x86/kernel/cpu/common.c                   |    2 +
 arch/x86/kernel/cpu/perf_counter.c             |  587 +++++++++
 arch/x86/kernel/entry_64.S                     |    5 +
 arch/x86/kernel/irq.c                          |    5 +
 arch/x86/kernel/irqinit_32.c                   |    3 +
 arch/x86/kernel/irqinit_64.c                   |    5 +
 arch/x86/kernel/signal.c                       |    7 +-
 arch/x86/kernel/syscall_table_32.S             |    1 +
 drivers/acpi/processor_idle.c                  |    8 +
 drivers/char/sysrq.c                           |    2 +
 include/linux/perf_counter.h                   |  257 ++++
 include/linux/prctl.h                          |    3 +
 include/linux/sched.h                          |   12 +-
 include/linux/syscalls.h                       |    8 +
 init/Kconfig                                   |   30 +
 kernel/Makefile                                |    1 +
 kernel/exit.c                                  |    7 +-
 kernel/fork.c                                  |    1 +
 kernel/perf_counter.c                          | 1542 ++++++++++++++++++++++++
 kernel/sched.c                                 |   31 +-
 kernel/sys.c                                   |    7 +
 kernel/sys_ni.c                                |    3 +
 37 files changed, 2939 insertions(+), 27 deletions(-)
 create mode 100644 Documentation/perf-counters.txt
 create mode 100644 arch/x86/kernel/cpu/perf_counter.c
 create mode 100644 include/linux/perf_counter.h
 create mode 100644 kernel/perf_counter.c

new file mode 100644
index 0000000..fddd321
@@ -0,0 +1,147 @@
+
+Performance Counters for Linux
+------------------------------
+
+Performance counters are special hardware registers available on most modern
+CPUs. These registers count the number of certain types of hw events: such
+as instructions executed, cachemisses suffered, or branches mis-predicted -
+without slowing down the kernel or applications. These registers can also
+trigger interrupts when a threshold number of events have passed - and can
+thus be used to profile the code that runs on that CPU.
+
+The Linux Performance Counter subsystem provides an abstraction of these
+hardware capabilities. It provides per task and per CPU counters, counter
+groups, and it provides event capabilities on top of those.
+
+Performance counters are accessed via special file descriptors.
+There's one file descriptor per virtual counter used.
+
+The special file descriptor is opened via the perf_counter_open()
+system call:
+
+   int sys_perf_counter_open(struct perf_counter_hw_event *hw_event_uptr,
+			     pid_t pid, int cpu, int group_fd);
+
+The syscall returns the new fd. The fd can be used via the normal
+VFS system calls: read() can be used to read the counter, fcntl()
+can be used to set the blocking mode, etc.
+
+Multiple counters can be kept open at a time, and the counters
+can be poll()ed.
+
+When creating a new counter fd, 'perf_counter_hw_event' is:
+
+/*
+ * Hardware event to monitor via a performance monitoring counter:
+ */
+struct perf_counter_hw_event {
+	s64			type;
+
+	u64			irq_period;
+	u32			record_type;
+
+	u32			disabled     :  1, /* off by default */
+				nmi	     :  1, /* NMI sampling   */
+				raw	     :  1, /* raw event type */
+				__reserved_1 : 29;
+
+	u64			__reserved_2;
+};
+
+/*
+ * Generalized performance counter event types, used by the hw_event.type
+ * parameter of the sys_perf_counter_open() syscall:
+ */
+enum hw_event_types {
+	/*
+	 * Common hardware events, generalized by the kernel:
+	 */
+	PERF_COUNT_CYCLES		=  0,
+	PERF_COUNT_INSTRUCTIONS		=  1,
+	PERF_COUNT_CACHE_REFERENCES	=  2,
+	PERF_COUNT_CACHE_MISSES		=  3,
+	PERF_COUNT_BRANCH_INSTRUCTIONS	=  4,
+	PERF_COUNT_BRANCH_MISSES	=  5,
+
+	/*
+	 * Special "software" counters provided by the kernel, even if
+	 * the hardware does not support performance counters. These
+	 * counters measure various physical and sw events of the
+	 * kernel (and allow the profiling of them as well):
+	 */
+	PERF_COUNT_CPU_CLOCK		= -1,
+	PERF_COUNT_TASK_CLOCK		= -2,
+	/*
+	 * Future software events:
+	 */
+	/* PERF_COUNT_PAGE_FAULTS	= -3,
+	   PERF_COUNT_CONTEXT_SWITCHES	= -4, */
+};
+
+These are standardized types of events that work uniformly on all CPUs
+that implements Performance Counters support under Linux. If a CPU is
+not able to count branch-misses, then the system call will return
+-EINVAL.
+
+More hw_event_types are supported as well, but they are CPU
+specific and are enumerated via /sys on a per CPU basis. Raw hw event
+types can be passed in under hw_event.type if hw_event.raw is 1.
+For example, to count "External bus cycles while bus lock signal asserted"
+events on Intel Core CPUs, pass in a 0x4064 event type value and set
+hw_event.raw to 1.
+
+'record_type' is the type of data that a read() will provide for the
+counter, and it can be one of:
+
+/*
+ * IRQ-notification data record type:
+ */
+enum perf_counter_record_type {
+	PERF_RECORD_SIMPLE		=  0,
+	PERF_RECORD_IRQ			=  1,
+	PERF_RECORD_GROUP		=  2,
+};
+
+a "simple" counter is one that counts hardware events and allows
+them to be read out into a u64 count value. (read() returns 8 on
+a successful read of a simple counter.)
+
+An "irq" counter is one that will also provide an IRQ context information:
+the IP of the interrupted context. In this case read() will return
+the 8-byte counter value, plus the Instruction Pointer address of the
+interrupted context.
+
+The parameter 'hw_event_period' is the number of events before waking up
+a read() that is blocked on a counter fd. Zero value means a non-blocking
+counter.
+
+The 'pid' parameter allows the counter to be specific to a task:
+
+ pid == 0: if the pid parameter is zero, the counter is attached to the
+ current task.
+
+ pid > 0: the counter is attached to a specific task (if the current task
+ has sufficient privilege to do so)
+
+ pid < 0: all tasks are counted (per cpu counters)
+
+The 'cpu' parameter allows a counter to be made specific to a full
+CPU:
+
+ cpu >= 0: the counter is restricted to a specific CPU
+ cpu == -1: the counter counts on all CPUs
+
+(Note: the combination of 'pid == -1' and 'cpu == -1' is not valid.)
+
+A 'pid > 0' and 'cpu == -1' counter is a per task counter that counts
+events of that task and 'follows' that task to whatever CPU the task
+gets schedule to. Per task counters can be created by any user, for
+their own tasks.
+
+A 'pid == -1' and 'cpu == x' counter is a per CPU counter that counts
+all events on CPU-x. Per CPU counters need CAP_SYS_ADMIN privilege.
+
+Group counters are created by passing in a group_fd of another counter.
+Groups are scheduled at once and can be used with PERF_RECORD_GROUP
+to record multi-dimensional timestamps.
+

index d4d4cb7..fe94490 100644
@@ -643,6 +643,7 @@ config X86_UP_IOAPIC
 config X86_LOCAL_APIC
 	def_bool y
 	depends on X86_64 || (X86_32 && (X86_UP_APIC || (SMP && !X86_VOYAGER) || X86_GENERICARCH))
+	select HAVE_PERF_COUNTERS if (!M386 && !M486)
 
 config X86_IO_APIC
 	def_bool y

index 256b00b..3c14ed0 100644
@@ -823,7 +823,8 @@ ia32_sys_call_table:
 	.quad compat_sys_signalfd4
 	.quad sys_eventfd2
 	.quad sys_epoll_create1
-	.quad sys_dup3			/* 330 */
+	.quad sys_dup3				/* 330 */
 	.quad sys_pipe2
 	.quad sys_inotify_init1
+	.quad sys_perf_counter_open
 ia32_syscall_end:

index cf7954d..7a07897 100644
@@ -9,6 +9,7 @@ typedef struct {
 	unsigned long idle_timestamp;
 	unsigned int __nmi_count;	/* arch dependent */
 	unsigned int apic_timer_irqs;	/* arch dependent */
+	unsigned int apic_perf_irqs;	/* arch dependent */
 	unsigned int irq0_irqs;
 	unsigned int irq_resched_count;
 	unsigned int irq_call_count;

index 8de644b..aa93e53 100644
@@ -30,6 +30,8 @@
 /* Interrupt handlers registered during init_IRQ */
 extern void apic_timer_interrupt(void);
 extern void error_interrupt(void);
+extern void perf_counter_interrupt(void);
+
 extern void spurious_interrupt(void);
 extern void thermal_interrupt(void);
 extern void reschedule_interrupt(void);

index fa0fd06..71598a9 100644
@@ -1,22 +1,24 @@
 #ifndef _ASM_X86_INTEL_ARCH_PERFMON_H
 #define _ASM_X86_INTEL_ARCH_PERFMON_H
 
-#define MSR_ARCH_PERFMON_PERFCTR0		0xc1
-#define MSR_ARCH_PERFMON_PERFCTR1		0xc2
+#define MSR_ARCH_PERFMON_PERFCTR0			      0xc1
+#define MSR_ARCH_PERFMON_PERFCTR1			      0xc2
 
-#define MSR_ARCH_PERFMON_EVENTSEL0		0x186
-#define MSR_ARCH_PERFMON_EVENTSEL1		0x187
+#define MSR_ARCH_PERFMON_EVENTSEL0			     0x186
+#define MSR_ARCH_PERFMON_EVENTSEL1			     0x187
 
-#define ARCH_PERFMON_EVENTSEL0_ENABLE	(1 << 22)
-#define ARCH_PERFMON_EVENTSEL_INT	(1 << 20)
-#define ARCH_PERFMON_EVENTSEL_OS	(1 << 17)
-#define ARCH_PERFMON_EVENTSEL_USR	(1 << 16)
+#define ARCH_PERFMON_EVENTSEL0_ENABLE			  (1 << 22)
+#define ARCH_PERFMON_EVENTSEL_INT			  (1 << 20)
+#define ARCH_PERFMON_EVENTSEL_OS			  (1 << 17)
+#define ARCH_PERFMON_EVENTSEL_USR			  (1 << 16)
 
-#define ARCH_PERFMON_UNHALTED_CORE_CYCLES_SEL	(0x3c)
-#define ARCH_PERFMON_UNHALTED_CORE_CYCLES_UMASK	(0x00 << 8)
-#define ARCH_PERFMON_UNHALTED_CORE_CYCLES_INDEX (0)
+#define ARCH_PERFMON_UNHALTED_CORE_CYCLES_SEL		      0x3c
+#define ARCH_PERFMON_UNHALTED_CORE_CYCLES_UMASK		(0x00 << 8)
+#define ARCH_PERFMON_UNHALTED_CORE_CYCLES_INDEX 		 0
 #define ARCH_PERFMON_UNHALTED_CORE_CYCLES_PRESENT \
-	(1 << (ARCH_PERFMON_UNHALTED_CORE_CYCLES_INDEX))
+		(1 << (ARCH_PERFMON_UNHALTED_CORE_CYCLES_INDEX))
+
+#define ARCH_PERFMON_BRANCH_MISSES_RETIRED			 6
 
 union cpuid10_eax {
 	struct {

@@ -28,4 +30,12 @@ union cpuid10_eax {
 	unsigned int full;
 };
 
+#ifdef CONFIG_PERF_COUNTERS
+extern void init_hw_perf_counters(void);
+extern void perf_counters_lapic_init(int nmi);
+#else
+static inline void init_hw_perf_counters(void)		{ }
+static inline void perf_counters_lapic_init(int nmi)	{ }
+#endif
+
 #endif /* _ASM_X86_INTEL_ARCH_PERFMON_H */

index 0005adb..b8d277f 100644
@@ -87,6 +87,11 @@
 #define LOCAL_TIMER_VECTOR	0xef
 
 /*
+ * Performance monitoring interrupt vector:
+ */
+#define LOCAL_PERF_VECTOR	0xee
+
+/*
  * First APIC vector available to drivers: (vectors 0x30-0xee) we
  * start at 0x31(0x41) to spread out vectors evenly between priority
  * levels. (0x80 is the syscall vector)

index 6b1add8..ad31e5d 100644
@@ -25,10 +25,15 @@ BUILD_INTERRUPT(irq_move_cleanup_interrupt,IRQ_MOVE_CLEANUP_VECTOR)
  * a much simpler SMP time architecture:
  */
 #ifdef CONFIG_X86_LOCAL_APIC
+
 BUILD_INTERRUPT(apic_timer_interrupt,LOCAL_TIMER_VECTOR)
 BUILD_INTERRUPT(error_interrupt,ERROR_APIC_VECTOR)
 BUILD_INTERRUPT(spurious_interrupt,SPURIOUS_APIC_VECTOR)
 
+#ifdef CONFIG_PERF_COUNTERS
+BUILD_INTERRUPT(perf_counter_interrupt, LOCAL_PERF_VECTOR)
+#endif
+
 #ifdef CONFIG_X86_MCE_P4THERMAL
 BUILD_INTERRUPT(thermal_interrupt,THERMAL_APIC_VECTOR)
 #endif

index 2fbfff8..90a8d9d 100644
@@ -30,6 +30,7 @@ struct x8664_pda {
 	short isidle;
 	struct mm_struct *active_mm;
 	unsigned apic_timer_irqs;
+	unsigned apic_perf_irqs;
 	unsigned irq0_irqs;
 	unsigned irq_resched_count;
 	unsigned irq_call_count;

index e44d379..810bf26 100644
@@ -80,6 +80,7 @@ struct thread_info {
 #define TIF_SYSCALL_AUDIT	7	/* syscall auditing active */
 #define TIF_SECCOMP		8	/* secure computing */
 #define TIF_MCE_NOTIFY		10	/* notify userspace of an MCE */
+#define TIF_PERF_COUNTERS	11	/* notify perf counter work */
 #define TIF_NOTSC		16	/* TSC is not accessible in userland */
 #define TIF_IA32		17	/* 32bit process */
 #define TIF_FORK		18	/* ret_from_fork */

@@ -103,6 +104,7 @@ struct thread_info {
 #define _TIF_SYSCALL_AUDIT	(1 << TIF_SYSCALL_AUDIT)
 #define _TIF_SECCOMP		(1 << TIF_SECCOMP)
 #define _TIF_MCE_NOTIFY		(1 << TIF_MCE_NOTIFY)
+#define _TIF_PERF_COUNTERS	(1 << TIF_PERF_COUNTERS)
 #define _TIF_NOTSC		(1 << TIF_NOTSC)
 #define _TIF_IA32		(1 << TIF_IA32)
 #define _TIF_FORK		(1 << TIF_FORK)

@@ -135,7 +137,7 @@ struct thread_info {
 
 /* Only used for 64 bit */
 #define _TIF_DO_NOTIFY_MASK						\
-	(_TIF_SIGPENDING|_TIF_MCE_NOTIFY|_TIF_NOTIFY_RESUME)
+	(_TIF_SIGPENDING|_TIF_MCE_NOTIFY|_TIF_PERF_COUNTERS|_TIF_NOTIFY_RESUME)
 
 /* flags to check in __switch_to() */
 #define _TIF_WORK_CTXSW							\

index f2bba78..7e47658 100644
@@ -338,6 +338,7 @@
 #define __NR_dup3		330
 #define __NR_pipe2		331
 #define __NR_inotify_init1	332
+#define __NR_perf_counter_open	333
 
 #ifdef __KERNEL__
 

index d2e415e..53025fe 100644
@@ -653,7 +653,8 @@ __SYSCALL(__NR_dup3, sys_dup3)
 __SYSCALL(__NR_pipe2, sys_pipe2)
 #define __NR_inotify_init1			294
 __SYSCALL(__NR_inotify_init1, sys_inotify_init1)
-
+#define __NR_perf_counter_open		295
+__SYSCALL(__NR_perf_counter_open, sys_perf_counter_open)
 
 #ifndef __NO_STUBS
 #define __ARCH_WANT_OLD_READDIR

index 1771dd7..0579ec1 100644
@@ -31,6 +31,7 @@
 #include <linux/dmi.h>
 #include <linux/dmar.h>
 
+#include <asm/intel_arch_perfmon.h>
 #include <asm/atomic.h>
 #include <asm/smp.h>
 #include <asm/mtrr.h>

@@ -1143,6 +1144,7 @@ void __cpuinit setup_local_APIC(void)
 		apic_write(APIC_ESR, 0);
 	}
 #endif
+	perf_counters_lapic_init(0);
 
 	preempt_disable();
 

index 82ec607..89e5336 100644
@@ -1,5 +1,5 @@
 #
-# Makefile for x86-compatible CPU details and quirks
+# Makefile for x86-compatible CPU details, features and quirks
 #
 
 obj-y			:= intel_cacheinfo.o addon_cpuid_features.o

@@ -16,11 +16,13 @@ obj-$(CONFIG_CPU_SUP_CENTAUR_64)	+= centaur_64.o
 obj-$(CONFIG_CPU_SUP_TRANSMETA_32)	+= transmeta.o
 obj-$(CONFIG_CPU_SUP_UMC_32)		+= umc.o
 
-obj-$(CONFIG_X86_MCE)	+= mcheck/
-obj-$(CONFIG_MTRR)	+= mtrr/
-obj-$(CONFIG_CPU_FREQ)	+= cpufreq/
+obj-$(CONFIG_PERF_COUNTERS)		+= perf_counter.o
 
-obj-$(CONFIG_X86_LOCAL_APIC) += perfctr-watchdog.o
+obj-$(CONFIG_X86_MCE)			+= mcheck/
+obj-$(CONFIG_MTRR)			+= mtrr/
+obj-$(CONFIG_CPU_FREQ)			+= cpufreq/
+
+obj-$(CONFIG_X86_LOCAL_APIC)		+= perfctr-watchdog.o
 
 quiet_cmd_mkcapflags = MKCAP   $@
       cmd_mkcapflags = $(PERL) $(srctree)/$(src)/mkcapflags.pl $< $@

index b9c9ea0..4461011 100644
@@ -17,6 +17,7 @@
 #include <asm/mmu_context.h>
 #include <asm/mtrr.h>
 #include <asm/mce.h>
+#include <asm/intel_arch_perfmon.h>
 #include <asm/pat.h>
 #include <asm/asm.h>
 #include <asm/numa.h>

@@ -750,6 +751,7 @@ void __init identify_boot_cpu(void)
 #else
 	vgetcpu_set_mode();
 #endif
+	init_hw_perf_counters();
 }
 
 void __cpuinit identify_secondary_cpu(struct cpuinfo_x86 *c)

index 3194636..fc013cf 100644
@@ -984,6 +984,11 @@ apicinterrupt ERROR_APIC_VECTOR \
 apicinterrupt SPURIOUS_APIC_VECTOR \
 	spurious_interrupt smp_spurious_interrupt
 
+#ifdef CONFIG_PERF_COUNTERS
+apicinterrupt LOCAL_PERF_VECTOR \
+	perf_counter_interrupt smp_perf_counter_interrupt
+#endif
+
 /*
  * Exception entry points.
  */

index d1d4dc5..d92bc71 100644
@@ -56,6 +56,10 @@ static int show_other_interrupts(struct seq_file *p)
 	for_each_online_cpu(j)
 		seq_printf(p, "%10u ", irq_stats(j)->apic_timer_irqs);
 	seq_printf(p, "  Local timer interrupts\n");
+	seq_printf(p, "CNT: ");
+	for_each_online_cpu(j)
+		seq_printf(p, "%10u ", irq_stats(j)->apic_perf_irqs);
+	seq_printf(p, "  Performance counter interrupts\n");
 #endif
 #ifdef CONFIG_SMP
 	seq_printf(p, "RES: ");

@@ -160,6 +164,7 @@ u64 arch_irq_stat_cpu(unsigned int cpu)
 
 #ifdef CONFIG_X86_LOCAL_APIC
 	sum += irq_stats(cpu)->apic_timer_irqs;
+	sum += irq_stats(cpu)->apic_perf_irqs;
 #endif
 #ifdef CONFIG_SMP
 	sum += irq_stats(cpu)->irq_resched_count;

index 607db63..6a33b5e 100644
@@ -160,6 +160,9 @@ void __init native_init_IRQ(void)
 	/* IPI vectors for APIC spurious and error interrupts */
 	alloc_intr_gate(SPURIOUS_APIC_VECTOR, spurious_interrupt);
 	alloc_intr_gate(ERROR_APIC_VECTOR, error_interrupt);
+# ifdef CONFIG_PERF_COUNTERS
+	alloc_intr_gate(LOCAL_PERF_VECTOR, perf_counter_interrupt);
+# endif
 #endif
 
 #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86_MCE_P4THERMAL)

index 8670b3c..91d785c 100644
@@ -138,6 +138,11 @@ static void __init apic_intr_init(void)
 	/* IPI vectors for APIC spurious and error interrupts */
 	alloc_intr_gate(SPURIOUS_APIC_VECTOR, spurious_interrupt);
 	alloc_intr_gate(ERROR_APIC_VECTOR, error_interrupt);
+
+	/* Performance monitoring interrupt: */
+#ifdef CONFIG_PERF_COUNTERS
+	alloc_intr_gate(LOCAL_PERF_VECTOR, perf_counter_interrupt);
+#endif
 }
 
 void __init native_init_IRQ(void)

index b1cc6da..dee553c 100644
@@ -6,7 +6,7 @@
  *  2000-06-20  Pentium III FXSR, SSE support by Gareth Hughes
  *  2000-2002   x86-64 support by Andi Kleen
  */
-
+#include <linux/perf_counter.h>
 #include <linux/sched.h>
 #include <linux/mm.h>
 #include <linux/smp.h>

@@ -891,6 +891,11 @@ do_notify_resume(struct pt_regs *regs, void *unused, __u32 thread_info_flags)
 		tracehook_notify_resume(regs);
 	}
 
+	if (thread_info_flags & _TIF_PERF_COUNTERS) {
+		clear_thread_flag(TIF_PERF_COUNTERS);
+		perf_counter_notify(regs);
+	}
+
 #ifdef CONFIG_X86_32
 	clear_thread_flag(TIF_IRET);
 #endif /* CONFIG_X86_32 */

index d44395f..496726d 100644
@@ -332,3 +332,4 @@ ENTRY(sys_call_table)
 	.long sys_dup3			/* 330 */
 	.long sys_pipe2
 	.long sys_inotify_init1
+	.long sys_perf_counter_open

index 5f8d746..a3e66a3 100644
@@ -270,8 +270,11 @@ static atomic_t c3_cpu_count;
 /* Common C-state entry for C2, C3, .. */
 static void acpi_cstate_enter(struct acpi_processor_cx *cstate)
 {
+	u64 perf_flags;
+
 	/* Don't trace irqs off for idle */
 	stop_critical_timings();
+	perf_flags = hw_perf_save_disable();
 	if (cstate->entry_method == ACPI_CSTATE_FFH) {
 		/* Call into architectural FFH based C-state */
 		acpi_processor_ffh_cstate_enter(cstate);

@@ -284,6 +287,7 @@ static void acpi_cstate_enter(struct acpi_processor_cx *cstate)
 		   gets asserted in time to freeze execution properly. */
 		unused = inl(acpi_gbl_FADT.xpm_timer_block.address);
 	}
+	hw_perf_restore(perf_flags);
 	start_critical_timings();
 }
 #endif /* !CONFIG_CPU_IDLE */

@@ -1425,8 +1429,11 @@ static inline void acpi_idle_update_bm_rld(struct acpi_processor *pr,
  */
 static inline void acpi_idle_do_entry(struct acpi_processor_cx *cx)
 {
+	u64 pctrl;
+
 	/* Don't trace irqs off for idle */
 	stop_critical_timings();
+	pctrl = hw_perf_save_disable();
 	if (cx->entry_method == ACPI_CSTATE_FFH) {
 		/* Call into architectural FFH based C-state */
 		acpi_processor_ffh_cstate_enter(cx);

@@ -1441,6 +1448,7 @@ static inline void acpi_idle_do_entry(struct acpi_processor_cx *cx)
 		   gets asserted in time to freeze execution properly. */
 		unused = inl(acpi_gbl_FADT.xpm_timer_block.address);
 	}
+	hw_perf_restore(pctrl);
 	start_critical_timings();
 }
 

index ce0d9da..52146c2 100644
@@ -25,6 +25,7 @@
 #include <linux/kbd_kern.h>
 #include <linux/proc_fs.h>
 #include <linux/quotaops.h>
+#include <linux/perf_counter.h>
 #include <linux/kernel.h>
 #include <linux/module.h>
 #include <linux/suspend.h>

@@ -244,6 +245,7 @@ static void sysrq_handle_showregs(int key, struct tty_struct *tty)
 	struct pt_regs *regs = get_irq_regs();
 	if (regs)
 		show_regs(regs);
+	perf_counter_print_debug();
 }
 static struct sysrq_key_op sysrq_showregs_op = {
 	.handler	= sysrq_handle_showregs,

index 48d887e..b00df4c 100644
@@ -85,4 +85,7 @@
 #define PR_SET_TIMERSLACK 29
 #define PR_GET_TIMERSLACK 30
 
+#define PR_TASK_PERF_COUNTERS_DISABLE		31
+#define PR_TASK_PERF_COUNTERS_ENABLE		32
+
 #endif /* _LINUX_PRCTL_H */

index 55e30d1..2e15be8 100644
@@ -71,6 +71,7 @@ struct sched_param {
 #include <linux/fs_struct.h>
 #include <linux/compiler.h>
 #include <linux/completion.h>
+#include <linux/perf_counter.h>
 #include <linux/pid.h>
 #include <linux/percpu.h>
 #include <linux/topology.h>

@@ -1013,6 +1014,8 @@ struct sched_entity {
 	u64			last_wakeup;
 	u64			avg_overlap;
 
+	u64			nr_migrations;
+
 #ifdef CONFIG_SCHEDSTATS
 	u64			wait_start;
 	u64			wait_max;

@@ -1028,7 +1031,6 @@ struct sched_entity {
 	u64			exec_max;
 	u64			slice_max;
 
-	u64			nr_migrations;
 	u64			nr_migrations_cold;
 	u64			nr_failed_migrations_affine;
 	u64			nr_failed_migrations_running;

@@ -1326,6 +1328,7 @@ struct task_struct {
 	struct list_head pi_state_list;
 	struct futex_pi_state *pi_state_cache;
 #endif
+	struct perf_counter_context perf_counter_ctx;
 #ifdef CONFIG_NUMA
 	struct mempolicy *mempolicy;
 	short il_next;

@@ -2285,6 +2288,13 @@ static inline void inc_syscw(struct task_struct *tsk)
 #define TASK_SIZE_OF(tsk)	TASK_SIZE
 #endif
 
+/*
+ * Call the function if the target task is executing on a CPU right now:
+ */
+extern void task_oncpu_function_call(struct task_struct *p,
+				     void (*func) (void *info), void *info);
+
+
 #ifdef CONFIG_MM_OWNER
 extern void mm_update_next_owner(struct mm_struct *mm);
 extern void mm_init_owner(struct mm_struct *mm, struct task_struct *p);

index 04fb47b..a549678 100644
@@ -54,6 +54,7 @@ struct compat_stat;
 struct compat_timeval;
 struct robust_list_head;
 struct getcpu_cache;
+struct perf_counter_hw_event;
 
 #include <linux/types.h>
 #include <linux/aio_abi.h>

@@ -624,4 +625,11 @@ asmlinkage long sys_fallocate(int fd, int mode, loff_t offset, loff_t len);
 
 int kernel_execve(const char *filename, char *const argv[], char *const envp[]);
 
+
+asmlinkage int sys_perf_counter_open(
+
+	struct perf_counter_hw_event	*hw_event_uptr		__user,
+	pid_t				pid,
+	int				cpu,
+	int				group_fd);
 #endif

index f763762..7d147a3 100644
@@ -732,6 +732,36 @@ config AIO
           by some high performance threaded applications. Disabling
           this option saves about 7k.
 
+config HAVE_PERF_COUNTERS
+	bool
+
+menu "Performance Counters"
+
+config PERF_COUNTERS
+	bool "Kernel Performance Counters"
+	depends on HAVE_PERF_COUNTERS
+	default y
+	select ANON_INODES
+	help
+	  Enable kernel support for performance counter hardware.
+
+	  Performance counters are special hardware registers available
+	  on most modern CPUs. These registers count the number of certain
+	  types of hw events: such as instructions executed, cachemisses
+	  suffered, or branches mis-predicted - without slowing down the
+	  kernel or applications. These registers can also trigger interrupts
+	  when a threshold number of events have passed - and can thus be
+	  used to profile the code that runs on that CPU.
+
+	  The Linux Performance Counter subsystem provides an abstraction of
+	  these hardware capabilities, available via a system call. It
+	  provides per task and per CPU counters, and it provides event
+	  capabilities on top of those.
+
+	  Say Y if unsure.
+
+endmenu
+
 config VM_EVENT_COUNTERS
 	default y
 	bool "Enable VM event counters for /proc/vmstat" if EMBEDDED

index 19fad00..1f184a1 100644
@@ -89,6 +89,7 @@ obj-$(CONFIG_HAVE_GENERIC_DMA_COHERENT) += dma-coherent.o
 obj-$(CONFIG_FUNCTION_TRACER) += trace/
 obj-$(CONFIG_TRACING) += trace/
 obj-$(CONFIG_SMP) += sched_cpupri.o
+obj-$(CONFIG_PERF_COUNTERS) += perf_counter.o
 
 ifneq ($(CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

index 2d8be7e..d336c90 100644
@@ -1093,11 +1093,12 @@ NORET_TYPE void do_exit(long code)
 	mpol_put(tsk->mempolicy);
 	tsk->mempolicy = NULL;
 #endif
-#ifdef CONFIG_FUTEX
 	/*
-	 * This must happen late, after the PID is not
-	 * hashed anymore:
+	 * These must happen late, after the PID is not
+	 * hashed anymore, but still at a point that may sleep:
 	 */
+	perf_counter_exit_task(tsk);
+#ifdef CONFIG_FUTEX
 	if (unlikely(!list_empty(&tsk->pi_state_list)))
 		exit_pi_state_list(tsk);
 	if (unlikely(current->pi_state_cache))

index 495da2e..e207860 100644
@@ -978,6 +978,7 @@ static struct task_struct *copy_process(unsigned long clone_flags,
 		goto fork_out;
 
 	rt_mutex_init_task(p);
+	perf_counter_init_task(p);
 
 #ifdef CONFIG_PROVE_LOCKING
 	DEBUG_LOCKS_WARN_ON(!p->hardirqs_enabled);

index e4bb1dd..382cfdb 100644

index 31deba8..0f66633 100644

index e14a232..4be8bbc 100644

Part of the diff is not shown here. View the raw message.

Looking at the code, I am wondering what you are planning to do to support machines that have constraints on what sets of events can be counted simultaneously. Currently you have the core code calling counter->hw_ops->hw_perf_counter_enable which can't return an error. The core expects it to be able to add any counter regardless of what event it's counting, subject only to a maximum number of counters. I assume you're going to change that.

I think the core should put together a list of counters and counter groups that it would like to have on the PMU simultaneously and then make one call to the arch layer to ask if that is possible. That could either return success or failure. If it returns failure then the core needs to ask for something less, or something different. I'm not sure how the core should choose what to ask for instead, though.

Paul.

I think the constraint set should be applied when we add to a group, if when we add a counter to the group, the result isn't schedulable anymore, we should fail the group addition - and thereby the counter creation.

This would leave us with groups that are always schedulable in an atomic fashion.

possible) (ungrouped counter is a group of one), this means that with the above addition you'd have the needed control over things.

If you need things to be atomic, create a single group, if you're fine with RR time-sharing, create multiple.

This seems to leave a hole where multiple monitors collide and create multiple groups unaware of each-other - could we plug this hole with a group attribute?

I agree that if adding a counter to a group results in that group not being schedulable any more, we should fail the addition.

I think we need a "full-time" attribute for counters and groups that says "I need to be on the whole time", where "whole time" means whenever the task is running, for a per-task counter or group, or continuously for per-cpu counters/groups.

With that, the core can check at creation or enable time and return an error if it can't fit all the full-time counters and groups on, or if there is any part-time counter/group that would never get to go on when all the full-time counters/groups are on. (And I guess creation or enabling of a part-time counter or group should fail if it would never be able to go on.)

There could be a "whole-PMU" group attribute, that says "I need raw access to the PMU with no other counters scheduled", which would allow monitoring programs to use arcane PMU features that the kernel doesn't necessarily know about. But I think that's a bit different from what you're talking about.

The perf counter subsystem will, in Ingo's design, naturally try to schedule as many counters and groups on as it can. Given a list of counters/groups, it could start with the first and keep on trying to add counters or groups while it can, essentially trying all possible combinations until it either fills up all the hardware counters or exhausts the possible combinations. If it moves all the counters/groups that do fit on up to the head of the list, and then rotates them to the back of the list when the timeslice expires, that would probably be OK. In fact the computation about what set of counters/groups to put on should be done when adding/removing a counter/group and when the timeslice expires, rather than at context switch time. (I'm talking about the list of part-time counters/groups here, of course.)

Paul.

yeah, something like that sounds sensible.

The principle is this: as long as it's an add-on attribute with a default value of zero, so that normal usage does not need to care about it, it's a reasonable extension. What we want to avoid is burdening the normal case with the weird cases. (and none of the things you mentioned seem to be in that category so we are fine.)

Ingo

One issue is that PMU counts can cover more than one CPU. One example for this are the Uncore events on Nehalem (which cover a whole socket) or when you are in AnyThreads monitoring mode (then you get events from both SMT siblings in a core)

With that you would need to examine other CPU's state at context switch time. Probably not a good idea for scalability.

-Andi

[email protected]

Over time, it seems clear that we will see multi-core processor designs with increasingly large uncore/nest facilities, so this could become more and more of an issue.

- Corey

Corey Ashford
Software Engineer
IBM Linux Technology Center, Linux Toolchain
Beaverton, OR
503-578-3507
[email protected]

Ingo, I'll add my voice to the chorus here. To reiterate the point, some PMUs count events that are external to the processor cores, and these events cannot be attributed to any one particular CPU -- and certainly not to a particular pid. The current interface has a restriction that the user cannot pass -1 for both pid and cpu. But it seems to me that's exactly what would be needed for such off-core events. Can this feature fit in with the current interface or is some sort of extension needed?

Thanks.
-Maynard

They fit in just fine, they just will have constraints that dont allow their scheduling in a conflicting way. I.e. you'll only be able to occupy it from a single counter per physical package - but otherwise it still behaves like a normal counter if you define a single such counter per physical package, as a percpu counter. (they dont make much sense as task counters)

Btw., those kind of constraints make them quite noisy and hard to interpret as well - because they summarize per physical package characteristics (of up to 8 logical CPUs on Nehalem for example) and cannot be tied to tasks easily.

I'm not dismissing them entirely: they do give an overview of "all stuff that happens" at that level, and they do show a few things that is obviously tied to the 'uncore' of a CPU (the cache, the memory interlink, etc.), which cannot be provided by the normal PMCs - but they are too highlevel to really be useful for finegrained analysis and for eventing.

In any case, despite their limitations they can still be provided just fine, and any limitations they have is an inherent limitation of those hardware counters, not of the perfcounters framework.

Ingo

Those nest events still get counted on counters that are in the CPU core, right? So that sounds like they can be counted by one or more per-cpu perf_counter instances. That means that you're measuring them across all processes. Does it make any sense to try to attribute those events to individual processes? How would one do that?

Clearly, something has to know enough about the system topology to know how many counters are needed and which (virtual) cpus they should be on. At present that defaults to userspace, but we could extend perf_counters to handle it in the kernel by adding 'core' and 'node' specifiers to the hw_event structure (assuming a three-level node / core / cpu hierarchy for the system structure).

Paul.

Nope, Nehalem uncore counters are separate per socket. The uncore has its own counters.

You can program them to interrupt some fixed CPU thread (but it's not necessarily the thread who caused the event) or all CPU threads on the socket.

I found these semantics quite hard to fit into oprofile too because it also has too many per cpu events assumptions.

You can't (except by forcing only a single thread to run on the socket). The hardware doesn't know otherwise.

perfmon3 has a concept of counters which are system wide. While that's also not a 100% match on a multi socket system it's better than assigning it to some random CPU.

Hmm you want to create new virtual cpus for this?

-Andi

[email protected] -- Speaking for myself only.

That's what PAPI does.
In userspace.
Using libpfm.
Before counting anything.
On linux, AIX, Windows, Cray...
Talking to perfmon, perfctr, our own drivers, and maybe someday even the linux performance counter subsystem.
- d

Paul,

Who would want not to be the only owner of the PMU?

Especially when you know that if you don't, you may be sharing it with another user who may perturb your measurement, for instance because of heavy sampling while you are just counting. It is a tradeoff between flexibility and accuracy. Which one is more important?

Does this machine have sufficient hardware counters to count those four hardware events at the same time? Or were those counters timeshared onto 1 or 2 hardware counters? If it's the latter, are those counts from half or a quarter of the total execution?

Paul.

Hello

I see a large (2300 instruction) fixed overhead when measuring retired instruction count using the "timec" command compared to the "pfmon" tool that comes with perfmon3 (the pfmon tool has essentially no overhead when doing aggragate counts).

Is this an inherent weakness with the new proposed performance counter infrastructure?

I wanted to compare perfmon3 against Ingo's proposed performance counter infrastructure. This is on a Core2 Q6600 (the only machine I have that supports Ingo's codebase).

For perfmon3 comparison, it's the same machine running 2.6.27.4 patched with the appropriate full (not stripped-down) perfmon3 patchset available from perfmon2.sf.net.

All code for these tests can be had from:
http://www.csl.cornell.edu/~vince/projects/perf_counter/

#
# 100 instruction test
#

Testing with a 100 instruction assembly program:

# perfmon3

tasse:~/assembly_tests% pfmon -e INSTRUCTIONS_RETIRED ./100_insns 100 INSTRUCTIONS_RETIRED

# Ingo

tasse:~/assembly_tests% ./timec -e 1 ./100_insns

Performance counter stats for './100_insns':

0.762 task clock ticks (millisecs)

2446 instructions (events)

As we can see, timec overcounts by a lot! Is it 24x, or a fixed value?

#
# 8 billion instruction comparison
#

# perfmon3

tasse:~/assembly_tests% time pfmon -e INSTRUCTIONS_RETIRED ./8B_insns 8000000440 INSTRUCTIONS_RETIRED
1.77s user 0.00s system 100% cpu 1.771 total

Note that on almost all x86 chips that any hardware interrupt that occurs adds an extra retired instruction to the total count (some AMD engineers told me this is probably due to some artifact
due to long pipelines and how the microcode changes user/kernel flag).

So you see that in 1.77s we acccumulate 1.77s*250Hz timer interrupts which is 442.5 which is roughly the extra instructions we see.

(for more info on sources of non-determinism in instruction counting
with performance counters see the paper here:
http://www.csl.cornell.edu/~vince/papers/iiswc08 )

# ingo

tasse:~/assembly_tests% ./timec -e 1 ./8B_insns

Performance counter stats for './8B_insns':

1743.446 task clock ticks (millisecs)

8000002799 instructions (events)

So it turns out the overhead isn't 24x, but is actually a fixed 2300 or so.

Still, that's overhead perfmon does not have.

Will this be fixed, or is it an inherent limitation of the new proposal?

Vince

Hello

I'm trying a more complicated benchmark and getting even stranger results.

This is still on the Q6600 machine

The benchmark does a loop, reading some memory. It should have roughly:
12295 instructions
4096 memory loads
4096 branches

perfmon3 is close on all of these stats, and this is consistent across runs with a small variation (+/- 3 or so).

The timec program returns 0 (!) for all of the stats except retired instruction count! And with certain combinations of counters I get 0 for all counts. No error messages are printed.

Is this expected behavior?

The test program can be had from:
http://www.csl.cornell.edu/~vince/projects/perf_counter/

Details below:

#
# Perfmon results
#

# First, trying to read all 5 events at once fails, only 4 counters # avail

tasse:~/assembly_tests% pfmon -e INSTRUCTIONS_RETIRED,BRANCH_INSTRUCTIONS_RETIRED,L1D_ALL_CACHE_REF,MEM_LOAD_RETIRED:L1D_MISS ./read_test cannot configure events: set0 events incompatible or too many events

# Cache results are close to expected, L1D looks a little high

tasse:~/assembly_tests% pfmon -e INSTRUCTIONS_RETIRED,L1D_ALL_CACHE_REF,MEM_LOAD_RETIRED:L1D_MISS ./read_test 12299 INSTRUCTIONS_RETIRED
4164 L1D_ALL_CACHE_REF
4 MEM_LOAD_RETIRED:L1D_MISS

# Branch results. Close to what they should be, though a bit higher # than expected.

tasse:~/assembly_tests% pfmon -e INSTRUCTIONS_RETIRED,BRANCH_INSTRUCTIONS_RETIRED,MISPREDICTED_BRANCH_RETIRED ./read_test 12299 INSTRUCTIONS_RETIRED
4102 BRANCH_INSTRUCTIONS_RETIRED
1 MISPREDICTED_BRANCH_RETIRED

#

# performance counter v4
#

# Including all stats gives no errors, but gives no results

# either

tasse:~/assembly_tests% ./timec -e 0 -e 1 -e 2 -e 3 -e 4 -e 5 ./read_test

Performance counter stats for './read_test':

0.716 task clock ticks (millisecs)

85049 cycles (events)
0 instructions (events)
0 cache references (events)
0 cache misses (events)
0 branches (events)
0 branch misses (events)

#
# If I include the cycles count, I consistently get 0 # for all counts???

tasse:~/assembly_tests% ./timec -e 0 -e 1 -e 2 -e 3 ./read_test

Performance counter stats for './read_test':

0.520 task clock ticks (millisecs)

73833 cycles (events)
0 instructions (events)
0 cache references (events)
0 cache misses (events)

#
# If I drop the cycles count, I get an instruction count
# with a value 2300 too high (see previous e-mail)
# And really low cache values.

tasse:~/assembly_tests% ./timec -e 1 -e 2 -e 3 ./read_test

Performance counter stats for './read_test':

0.723 task clock ticks (millisecs)

14644 instructions (events)
8 cache references (events)
0 cache misses (events)

#
# And the branch stats don't work either
#

tasse:~/assembly_tests% ./timec -e 1 -e 4 -e 5 ./read_test

Performance counter stats for './read_test':

0.711 task clock ticks (millisecs)

14643 instructions (events)
0 branches (events)
0 branch misses (events)

When you have more software counters than hardware counters, the kernel will be time-slicing the counters, but because your program doesn't run for very long, it's possible that only the first set will ever get to count anything.

That doesn't seem to explain all the 0 values (since the kernel should be able to use at least 2 hardware counters on your machine, I expect) but it might be part of the explanation.

Paul.

Looks like timec will be counting the fork() and execvp() system calls that are used to run your executable, as well as the executable itself. The fork() overhead could be removed fairly easily I think, the execvp would be hard to get rid of without using ptrace() - and the use of ptrace was one of the things that Ingo et al. objected to in perfmon3.

Paul.

Paul, I think your analysis is correct. This is likely what is happening.

Not that timec could not use ptrace() to block the task from executing its first instruction, but you'd still have a problem because of prctl(ENABLE) which applies to the current task, not another task, unless I am mistaken.

Prctl() looks odd to me because you have all those supposedly independent file descriptors to identify events you want to measure, but they are not used to start/stop. If you are attaching to multiple tasks at the same time which you can do with the current API, you may not necessarily want to start/stop all counters at the same time.

Looks like prctl() is not what we want after all...

I don't think using ptrace in this case is a big issue - aside from the fact that ptrace is crap in that you'd not be able to timec from a debugger context :-(

The biggest objection to using ptrace was that ptrace was needed _during_ the execution of the monitored load, thereby distorting the load.

This case is different in that it would be used to start off the load.

Still it would be good if we could find another (elegant) way to fix this.

Also, I'm pretty sure the regular 'time' suffers the very same issue and counts the exec syscall as well - I saw that when I tinkered with the execve argument code.

Furthermore, I think output of tools such as time and now timec are most relevant when compared between runs - that is, the change in values between runs, not the absolute values as such. At least, that's what I usually do:

time ./foo

tinker with foo.c

time ./foo

if time2 < time1 :-)
else :-(

That's doesn't do you any good when comparing results across different machines, or even different kernels on the same machine.

perfmon shows that good results can be had, even if it's not the cleanest way in the world. It would be a shame to lose that.

Small micro-benchmarks like this are important. You can't always trust the performance counters to work, so being able to sanity check them with exact test-cases is critical. Otherwise you might just be measuring nonsense.

And while it might be able to subtract the exec() overhead for something like retired instructions, it gets a lot more complicated when you have something like cache bus snoops or branch mispredicts where it's hard to tell what comes from the program and what is overhead from the monitoring infrastructure.

Vince

Just for the record, I didn't write the quoted paragraph, Peter Zijlstra did.

Paul.

Hmm, if I timec some setuid program, what happens?

Performance counters seem like great tool to pull secret keys out of other processes :-).
Pavel

(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

yes, i already had a quick look at that a few days ago when i implemented counter inheritance (for different reasons) and couldnt find the cleanest place to put the exec() flushing into so i procrastinated that a bit :)

if you worry about _that_ angle you also have to:

- turn off the cycle counter

- turn off precise utimes

- plus you have to forbid SMT CPUs as well. On HT a task could co-schedule with your setuid task and observe its timing characteristics via its _own_ behavior. (which is impacted by whatever is running on another SMT/HT thread.)

the real exec() worry are: active, IRQ driven samples/events. Not possible yet via the current iteration of counter inheritance (hence my procrastination) - but it makes sense and that's why i was looking at the exec() angle.

and that will flush simple counters too, removing your theoretical attack angle as well.

So how about the patch below?

Ingo

--------------->
Subject: perfcounters: flush on setuid exec
From: Ingo Molnar <[email protected]>
Date: Tue Dec 16 13:40:44 CET 2008

Pavel Machek pointed out that performance counters should be flushed when crossing protection domains on setuid execution.

---
 fs/exec.c |    8 ++++++++
 1 file changed, 8 insertions(+)

Index: linux/fs/exec.c
===================================================================
@@ -33,6 +33,7 @@
 #include <linux/string.h>
 #include <linux/init.h>
 #include <linux/pagemap.h>
+#include <linux/perf_counter.h>
 #include <linux/highmem.h>
 #include <linux/spinlock.h>
 #include <linux/key.h>

@@ -1015,6 +1016,13 @@ int flush_old_exec(struct linux_binprm *
 		set_dumpable(current->mm, suid_dumpable);
 	}
 
+	/*
+	 * Flush performance counters when crossing a
+	 * security domain:
+	 */
+	if (!get_dumpable(current->mm))
+		perf_counter_exit_task(current);
+
 	/* An exec changes our domain. We are no longer part of the thread
 	   group */
 

Probably good idea, yes.

Yes, SMT is evil.

Thanks!

(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

HT got added back to Nehalem, so SMT is coming to you in every future x86 CPU. It brings a serious performance win, so nobody will turn off SMT threading in practice. If SMT worries you, it needs explicit partitioning of security-relevant processing to different physical CPUs, via cgroups/cpusets/etc.

find below the final commit, thanks Pavel.

Ingo

------------>

From: Ingo Molnar <[email protected]>
Date: Tue, 16 Dec 2008 13:40:44 +0100
Subject: [PATCH] perfcounters: flush on setuid exec

Pavel Machek pointed out that performance counters should be flushed when crossing protection domains on setuid execution.

---
 fs/exec.c |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)

index ec5df9a..d5165d8 100644
@@ -33,6 +33,7 @@
 #include <linux/string.h>
 #include <linux/init.h>
 #include <linux/pagemap.h>
+#include <linux/perf_counter.h>
 #include <linux/highmem.h>
 #include <linux/spinlock.h>
 #include <linux/key.h>

@@ -1017,6 +1018,13 @@ int flush_old_exec(struct linux_binprm * bprm)
 		set_dumpable(current->mm, suid_dumpable);
 	}
 
+	/*
+	 * Flush performance counters when crossing a
+	 * security domain:
+	 */
+	if (!get_dumpable(current->mm))
+		perf_counter_exit_task(current);
+
 	/* An exec changes our domain. We are no longer part of the thread
 	   group */
 

and/or you use properly implemented crypto code (see Bruce Schneider's books). The timing "problem" isn't really SMT specific. If you have improperly implemented crypto (eg crypto code where the code paths and not just the data payload are key dependent) then on any system with more than one (logical) processor there is interference that an attacker can use.

The only possible answer is to use proper implementation; turning off HT may make you feel good but you go from shoddy crypto for which there is some internet papers on how to crack it, to shoddy crypto for which the same papers apply ;)

Arjan van de Ven 	Intel Open Source Technology Centre
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Fortunately, Intel is not the only x86 vendor :-).

I guess we should refuse to run threads from different uids on one physical core...

It is not only timing. If attacker has access to detailed cache miss statistics, things are easier for him... I probably should review the books, but even if code paths are key-independend (hard), you'll get timing differences due to [data] cache misses...?

Ok, this is getting off-topic.

Pavel

(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Could people fix this perfctl-devel list thingy to not generate silly bounces - otherwise I'm going to drop it from the CC on further emails.

---
The reason it is being held:

Too many recipients to the message

Ingo Molnar wrote on Sun, Dec 14, 2008 at 10:28:29PM +0100:

Does this come with a PAPI frontend or is one planned?

I picked the old perfctr mainly because I need PAPI. Outside-process measurement doesn't do for my application, I need subdivisions in my code.

Thanks
Martin

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Martin Cracauer <[email protected]>   http://www.cons.org/cracauer/

I'm trying to evaluate this new proposal for the kind of workloads I use performance counters for, and even the simplest tests don't work.

I'm trying to do a simple aggragate count for some benchmarks here using timec and I'm getting poor results.

Are any of the problems I'm reporting going to be fixed?

In any case, I was testing aggregate counts on a longer running benchmark, this time equake from the spec2k benchmark suite, still on the q6600.

If I only count retired instructions, I get consistent results:

timec -e 1

      119175255369  instructions         (events)
      119175255561  instructions         (events)
      119175255383  instructions         (events)

however the minute I add another count, say cycles so I can calculate CPI/IPC the results for instructions are suddenly off by 33%.

Needless to say, perfmon can handle reading both cycles and instructions at the same time.

timec -e 0, -e 1

       91758816320  cycles               (events)
       79428247907  instructions         (events)

       91849140396  cycles               (events)
       79449560742  instructions         (events)

It gets worse when trying to look at cache statistics:

timec -e 1 -e 2 -e 3

       59611457943  instructions         (events)
        1872499771  cache references     (events)
          97471971  cache misses         (events)

       59601907232  instructions         (events)
        1871766376  cache references     (events)
          97435199  cache misses         (events)

and so on

timec -e1 -e2 -e3 -e4

       47671703285  instructions         (events)
        1498246999  cache references     (events)
          77838085  cache misses         (events)
        3394839360  branches             (events)

       47666131604  instructions         (events)
        1497069685  cache references     (events)
          78065325  cache misses         (events)
        3393244879  branches             (events)

So apparently this performance counter infrastructure will always be useless for trying to get plain aggregate counts? It's the simplest case to get right, so it makes me wonder about the design of the rest of the infrastructure.

Vince

Your test case demonstrates that scaling is missing from the current version of Performance Counters for Linux.

When each set of events is scheduled onto a set of hardware event counters, in order to scale the results properly, a cycles counter needs to be included in each set as well.

When the counts are read up, the counts from each set need to be scaled by a factor of (total cycles)/(cycles in that set)

This is something that can be handled by perfmon3 (full) because set multiplexing is explicitly programmed, not transparent as it is in Ingo's current code. In perfmon3, the set switching can be determined by events counter overflow, as well as time.

In common with both perfmon3 and Ingo's solution is that as more and more events are scheduled onto the same set of hardware registers, the accuracy drops and has to be compensated with longer run times.

Another source of error is that if the sets are rotated across the hardware at a fixed periodic rate, if there's any correlation between that rate and what's going on in the program being analyzed, the results will be dubious. Ideally, you'd want to have some sort of pseudo-random set switching rate to mitigate this sort of problem.

If Ingo could make some sort of provision for including a cycles count in every set, and then transparently performing the scaling, that would make it easier to use. As it stands now, I don't think there's any way to recover the needed scaling information, because you cannot tell what events are in what sets and how many cycles are associated with each set.

- Corey

Corey Ashford
Software Engineer
IBM Linux Technology Center, Linux Toolchain
Beaverton, OR
503-578-3507
[email protected]

There seems to be some confusion.

I want aggregate instruction count. I do not want any sort of scaling or sampling.

When I cound retired instructions and cycles, I want the full counts for those. The q6600 definitely has more than 2 counters available, so it should be able to give me exact aggregate counts for those counters. No sampling or scaling should be involved.

Is it not possible to get raw, aggregate count with Ingo's infrastructure? The documentation is vague on this.

Vince

PAPI also has the concept of event presets to hide some of the event selection details. The following URL lists the presets and which processors they are supported on:

http://icl.cs.utk.edu/projects/papi/presets.html

Looking through the PAPI preset events can see lots of variation due to the difference in what precisely the performance monitoring hardware support within the processor. The following events are defined in include/linux/perf_counter.h. It would be helpful to state what is meant by of the following events:

PERF_COUNT_CYCLES
PERF_COUNT_INSTRUCTIONS
PERF_COUNT_CACHE_REFERENCES
PERF_COUNT_CACHE_MISSES
PERF_COUNT_BRANCH_INSTRUCTIONS
PERF_COUNT_BRANCH_MISSES

PERF_COUNT_CYCLES and PERF_COUNT_INSTRUCTIONS

Is this the cpu clock rate to compute clocks per instruction (CPI)? On some processors there are several possible sources of "cycles":

Reference clock frequency (fixed frequency, e.g. always 2.2GHz) Cycles of processor subject to frequency changes and halts

Is PERF_COUNT_INSTRUCTIONS the instructions actually retired by the processor and would be used with PERF_COUNT_CYCLES to estimate CPI? In the case of a SMT (symetric multi-threaded) processor these are going to be kept on a per-virtual-CPU basis?

PERF_COUNT_CACHE_REFERENCES and PERF_COUNT_CACHE_MISSES

PERF_COUNT_CACHE_REFERENCES and PERF_COUNT_CACHE_MISSES are not single monolitic events on many processors. There are multiple cache levels. The L1 cache most processors have separate instruction and data caches and require multiple counters to implement. Would these refer to the last level of cache before memory and just be used to compute the hit/miss rate for that last level? Some processors in the same family have L2 and some processors have L3 cache. The setup code would need to distinguish between these processor variants.

What memory references and misses are included and excluded in cache operation counts? TLB accesses? Cache eviction/snooping operations?

PERF_COUNT_BRANCH_INSTRUCTIONS and PERF_COUNT_BRANCH_MISSES

Assume the PERF_COUNT_BRANCH_INSTRUCTIONS and PERF_COUNT_BRANCH_MISSES refer only to retired instructions are are used to compute branch misprediction ratio. Speculative instructions don't count for these numbers.

Any thought to including events that are not in the Intel architected events such as ITLB/DTLB accesses and misses?

-Will

The difference between L1 and L3 caches can be huge (in some cases two orders of magnitude). With that I'm not sure a single cache miss/hit event even makes any sense.

On a modern CPU with L2 and L3 caches as soon as you fall out of the L2 you're going to perform more poorly on parallel workloads.

With that you really have to distingush some levels.

-Andi

[email protected]

Confirmed. I have a code for which I'd like to know whether it fits into at least L2 or even L1.

Samuel

The point was that L1 and L2/L3 behave so differently in practice due to their latency/bandwidth differences (typically L1 is more like a slower register file while L3 is more like faster DRAM) that it's unclear that any generic event not defining what it means is any useful.

I could see the point of generic FLC/LLC cache events perhaps, but not of a single one lumping these two together.

But although with multi core it looks like 3 level hierarchies are becoming more and more common and also more important to know about.

-Andi

[email protected]

Ok that makes sense.

It doesn't really obscure it (although there are some doubts it can express many of the more powerful/complicated features of modern PMUs), but provides a set of generalized standard events in addition, plus a "raw mode". The current list of events as of v4 is:

+       PERF_COUNT_CYCLES               =  0,
+       PERF_COUNT_INSTRUCTIONS         =  1,
+       PERF_COUNT_CACHE_REFERENCES     =  2,
+       PERF_COUNT_CACHE_MISSES         =  3,
+       PERF_COUNT_BRANCH_INSTRUCTIONS  =  4,
+       PERF_COUNT_BRANCH_MISSES        =  5,

I think cache_references/misses is not well defined enough to be useful.

The Intel architectural perfmon (which is a standard set of event supported over a range of Intel x86 micro architectures) has similar events defined as hitting the LLC (last level cache). With that it makes some sense.

-Andi

[email protected]

Machines that support virtualization are becoming very common. How is this performance monitoring support going to work with virtualization (e.g. KVM)? Having the performance counters only work on physical machines would be pretty limiting.

-Will

On Power machines, the PMU counter registers are virtualized by the hypervisor, at the request of the OS, on a per-cpu/per-partition basis.
So this issue is handled transparently on Power anyway.

- Corey

Corey Ashford
Software Engineer
IBM Linux Technology Center, Linux Toolchain
Beaverton, OR
503-578-3507
[email protected]