Skip to contentSkip to conversation

linux-2.6.28-rc2 regression : oprofile doesnt work anymore ?

DISCUSSION Eric Dumazet 6 participants 18 messages Oct 30, 2008 – Nov 7, 2008 x86

lore
Share & tools
lore
#0 lore raw

Hi

I tried to start my favorite tool, and discovered it wont fly on my HP BL460C G1

It samples few events then, seems to stop (no more NMI), other than that, machine still running correctly.

(8 cpus like that one

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
stepping : 6
cpu MHz : 3000.253
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority
bogomips : 6000.50
clflush size : 64
power management:

It is working correctly on 2.6.27

I tried a git bisect but quickly found a point where machine was not useable... (partly because of the problem with the VPD problem on 5708S (PCI: Limit VPD length for Broadcom 5708S)

# opcontrol --deinit
Stopping profiling.
Killing daemon.
# rm -rf /var/lib/oprofile/samples/current
# opcontrol --vmlinux=vmlinux --start
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
 samples|      %|
------------------
      17 80.9524 vmlinux
       2  9.5238 libc-2.3.4.so
       1  4.7619 bash
       1  4.7619 ld-2.3.4.so
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
 samples|      %|
------------------
      17 80.9524 vmlinux
       2  9.5238 libc-2.3.4.so
       1  4.7619 bash
       1  4.7619 ld-2.3.4.so
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
 samples|      %|
------------------
      17 80.9524 vmlinux
       2  9.5238 libc-2.3.4.so
       1  4.7619 bash
       1  4.7619 ld-2.3.4.so
# grep NMI /proc/interrupts
NMI:          8          2          2          6         11          7          6          5   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:          8          2          2          6         11          7          6          5   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:          8          2          2          6         11          7          6          5   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:          8          2          2          6         11          7          6          5   Non-maskable interrupts
#1 lore raw

(adding some more cc's)

Quoted 91 lines · Eric Dumazet

I tried to start my favorite tool, and discovered it wont fly on my HP BL460C G1

It samples few events then, seems to stop (no more NMI), other than that, machine
still running correctly.

(8 cpus like that one

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
stepping : 6
cpu MHz : 3000.253
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority
bogomips : 6000.50
clflush size : 64
power management:

It is working correctly on 2.6.27

I tried a git bisect but quickly found a point where machine was not useable...
(partly because of the problem with the VPD problem on 5708S (PCI: Limit VPD length for Broadcom 5708S)

# opcontrol --deinit
Stopping profiling.
Killing daemon.
# rm -rf /var/lib/oprofile/samples/current
# opcontrol --vmlinux=vmlinux --start
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit
mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
 samples|      %|
------------------
     17 80.9524 vmlinux
      2  9.5238 libc-2.3.4.so
      1  4.7619 bash
      1  4.7619 ld-2.3.4.so
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit
mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
 samples|      %|
------------------
     17 80.9524 vmlinux
      2  9.5238 libc-2.3.4.so
      1  4.7619 bash
      1  4.7619 ld-2.3.4.so
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit
mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
 samples|      %|
------------------
     17 80.9524 vmlinux
      2  9.5238 libc-2.3.4.so
      1  4.7619 bash
      1  4.7619 ld-2.3.4.so
Quoted 12 lines
# grep NMI /proc/interrupts
NMI:          8          2          2          6         11          7
   6          5   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:          8          2          2          6         11          7
   6          5   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:          8          2          2          6         11          7
   6          5   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:          8          2          2          6         11          7
   6          5   Non-maskable interrupts

I'm seeing the same thing on my 2-way x86-64. Here's my /proc/cpuinfo in case it helps:

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz
stepping : 6
cpu MHz : 2000.000
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 3989.97
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz
stepping : 6
cpu MHz : 2000.000
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 3989.97
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:

#16 lore raw
In reply to Eric Dumazet

Hi Eric (and related people touching the code),

Just to confirm your observations (as you have a ? in subj);
oprofile is also broken for me on kernel 2.6.28-rc2.

(Related people Cc'ed
git log v2.6.27.. drivers/oprofile/ | grep Author: | sort |uniq)

Quoted 2 lines · Eric Dumazet

I tried to start my favorite tool, and discovered it wont fly on my HP BL460C G1

My hardware: HP-proliant DL380-G5
CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz

Quoted 2 lines

It samples few events then, seems to stop (no more NMI), other than that, machine still running correctly.

Same observation.

Quoted 1 line

It is working correctly on 2.6.27

Same observation.

#17 lore raw
In reply to Eric Dumazet

Eric,

Quoted 8 lines · Eric Dumazet

Hi

I tried to start my favorite tool, and discovered it wont fly on my HP BL460C G1

It samples few events then, seems to stop (no more NMI), other than that, machine
still running correctly.

hmm, I can not reproduce this on my system. I am receiving samples and the number is increasing. Andi, could you test this on some of your systems?

-Robert

# opcontrol --dump; opreport 
CPU: AMD64 family10, speed 2100 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
  samples|      %|
------------------
    70897 98.6970 vmlinux
      467  0.6501 libc-2.6.1.so
      219  0.3049 bash
      141  0.1963 ld-2.6.1.so
       43  0.0599 oprofiled
       15  0.0209 gawk-3.1.5
       15  0.0209 opreport
        9  0.0125 libstdc++.so.6.0.8
        8  0.0111 grep
        5  0.0070 sshd
        4  0.0056 libcrypto.so.0.9.8
        2  0.0028 rm
        1  0.0014 cat
        1  0.0014 dirname
        1  0.0014 ls
        1  0.0014 mkdir
        1  0.0014 id
        1  0.0014 ophelp
        1  0.0014 libbfd-2.18.so
        1  0.0014 ntpd
# grep NMI /proc/interrupts 
NMI:      20357      14272      20151       9967      20140      13961      16291      20826   Non-maskable interrupts
# grep NMI /proc/interrupts 
NMI:      20627      14458      20398      10080      20391      14204      16444      21016   Non-maskable interrupts
# cat /proc/cpuinfo
processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 16
model		: 2
model name	: Quad-Core AMD Opteron(tm) Processor 2352
stepping	: 3
cpu MHz		: 2100.000
cache size	: 512 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 5
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good pni monitor cx16 lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs
bogomips	: 4189.26
TLB size	: 1024 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
# uname -a
Linux paloma 2.6.28-rc2-oprofile-x86_64-standard #1 SMP Sat Nov 1 01:28:08 CET 2008 x86_64 Quad-Core AMD Opteron(tm) Processor 2352 AuthenticAMD GNU/Linux
Signature · 3 lines
Advanced Micro Devices, Inc.
Operating System Research Center
email: [email protected]
#2 lore raw
In reply to Pekka Enberg

Andi,

your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.

Thanks,

-Robert

Linux woglinde 2.6.27-oprofile-x86_64-standard-03189-g5951290 #5 SMP Tue Nov 4 18:16:41 CET 2008 x86_64 Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz GenuineIntel GNU/Linux IBS Fetch not supported - ignored
IBS OP not supported - ignored
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
Oprofile started
Daemon running: pid 5845
Separate options: none
vmlinux file: none
Image filter: none
Call-graph depth: 0
number of samples #1: 11
Stopping profiling.
Killing daemon.
number of samples #2: 11

Linux woglinde 2.6.27-oprofile-x86_64-standard-03185-gc493756 #6 SMP Tue Nov 4 19:05:13 CET 2008 x86_64 Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz GenuineIntel GNU/Linux IBS Fetch not supported - ignored
IBS OP not supported - ignored
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
Oprofile started
Daemon running: pid 5263
Separate options: none
vmlinux file: none
Image filter: none
Call-graph depth: 0
number of samples #1: 25
Stopping profiling.
Killing daemon.
number of samples #2: 64
Increasing sample number

commit 59512900baab03c5629f2ff5efad1d5d4e682ece

Author: Andi Kleen <[email protected]>
Date:   Mon Sep 29 22:23:33 2008 +0200

oprofile: discover counters for op ppro too

Discover number of counters for all family 6 models even when not in arch perfmon mode.

 arch/x86/oprofile/op_model_ppro.c |    8 +++++---
 arch/x86/oprofile/op_x86_model.h  |    2 +-
 2 files changed, 6 insertions(+), 4 deletions(-)

commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8

Author: Andi Kleen <[email protected]>
Date:   Mon Aug 18 14:50:31 2008 +0200

oprofile: Implement Intel architectural perfmon support

Newer Intel CPUs (Core1+) have support for architectural events described in CPUID 0xA. See the IA32 SDM Vol3b.18 for details.

The advantage of this is that it can be done without knowing about the specific CPU, because the CPU describes by itself what performance events are supported. This is only a fallback because only a limited set of 6 events are supported. This allows to do profiling on Nehalem and on Atom systems (later not tested)

This patch implements support for that in oprofile's Intel Family 6 profiling module. It also has the advantage of supporting an arbitary number of events now as reported by the CPU. Also allow arbitary counter widths >32bit while we're at it.

Requires a patched oprofile userland to support the new architecture.

v2: update for latest oprofile tree
remove force_arch_perfmon

 arch/x86/oprofile/nmi_int.c       |   23 ++++++--
 arch/x86/oprofile/op_model_ppro.c |  104 +++++++++++++++++++++++++++++--------
 arch/x86/oprofile/op_x86_model.h  |    3 +
 3 files changed, 102 insertions(+), 28 deletions(-)

commit f645f6406463a01869c50844befc76d528971690

Author: Andi Kleen <[email protected]>
Date:   Wed Aug 20 17:22:02 2008 +0200

oprofile: Don't report Nehalem as core_2

This essentially reverts Linus' earlier 4b9f12a3779c548b68bc9af7d94030868ad3aa1b commit. Nehalem is not core_2, so it shouldn't be reported as such. However with the earlier arch perfmon patch it will fall back to arch perfmon mode now, so there is no need to fake it as core_2. The only drawback is that Linus will need to patch the arch perfmon support into his oprofile binary now, but I think he can do that.

 arch/x86/oprofile/nmi_int.c |    3 ---
 1 files changed, 0 insertions(+), 3 deletions(-)

commit 5d4488027d9cf3161c71566dfabb116bf69ab4d9

Author: Andi Kleen <[email protected]>
Date:   Mon Aug 18 14:49:47 2008 +0200

oprofile: drop const in num counters field

allow to modify it at runtime

 arch/x86/oprofile/op_x86_model.h |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
Signature · 3 lines
Advanced Micro Devices, Inc.
Operating System Research Center
email: [email protected]
#3 lore raw

Hi Robert,

Quoted 2 lines · Robert Richter

your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.

Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?

Pekka

Quoted 1 line

From 37d362c0262006be6a5d6a72b5993834afa776d7 Mon Sep 17 00:00:00 2001

From: Pekka Enberg <[email protected]>
Date: Tue, 4 Nov 2008 21:17:47 +0200
Subject: [PATCH] oprofile: make perfmon nethalem only

Commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8 ("oprofile: Implement Intel architectural perfmon support") added perfmon support to newer Intel CPUs (Core1+). The problem there is that it requires a patched oprofile userland to work properly.

Fix that up by making oprofile perfmon support nethalem only.

---
 arch/x86/oprofile/nmi_int.c       |    7 +++++++
 arch/x86/oprofile/op_model_ppro.c |    4 +++-
 2 files changed, 10 insertions(+), 1 deletions(-)
arch/x86/oprofile/nmi_int.c +7 -0
index 022cd41..1f4448e 100644
@@ -426,8 +426,15 @@ static int __init ppro_init(char **cpu_type)
 
 static int __init arch_perfmon_init(char **cpu_type)
 {
+	__u8 cpu_model = boot_cpu_data.x86_model;
+
 	if (!cpu_has_arch_perfmon)
 		return 0;
+
+	/* Nethalem only */
+	if (cpu_model != 26)
+		return 0;
+
 	*cpu_type = "i386/arch_perfmon";
 	model = &op_arch_perfmon_spec;
 	arch_perfmon_setup_counters();
arch/x86/oprofile/op_model_ppro.c +3 -1
index 0620d6d..1c91ba2 100644
@@ -66,6 +66,7 @@ static void ppro_fill_in_addresses(struct op_msrs * const msrs)
 
 static void ppro_setup_ctrs(struct op_msrs const * const msrs)
 {
+	__u8 cpu_model = boot_cpu_data.x86_model;
 	unsigned int low, high;
 	int i;
 
@@ -76,7 +77,8 @@ static void ppro_setup_ctrs(struct op_msrs const * const msrs)
 			return;
 	}
 
-	if (cpu_has_arch_perfmon) {
+	/* Nethalem only */
+	if (cpu_has_arch_perfmon && cpu_model == 26) {
 		union cpuid10_eax eax;
 		eax.full = cpuid_eax(0xa);
 		if (counter_width < eax.split.bit_width)
Signature · 1 line
1.5.3.7
#15 lore raw
In reply to Robert Richter
Quoted 4 lines · Robert Richter

Andi,

your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.

I'll take a look but it'll be some time because my test setup is still not fully functional after moving.

-Andi

author Tested-by
#4 lore raw
In reply to Pekka J Enberg
Quoted 7 lines · Pekka J Enberg

Hi Robert,

Quoted 2 lines · Robert Richter

your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.

Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?

Thanks a lot Pekka, this patch restores oprofile on my HP BL460c G1

Quoted 63 lines

Pekka

Quoted 1 line

From 37d362c0262006be6a5d6a72b5993834afa776d7 Mon Sep 17 00:00:00 2001

From: Pekka Enberg <[email protected]>
Date: Tue, 4 Nov 2008 21:17:47 +0200
Subject: [PATCH] oprofile: make perfmon nethalem only

Commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8 ("oprofile: Implement Intel architectural perfmon support") added perfmon support to newer Intel CPUs (Core1+). The problem there is that it requires a patched oprofile userland to work properly.

Fix that up by making oprofile perfmon support nethalem only.

---
 arch/x86/oprofile/nmi_int.c       |    7 +++++++
 arch/x86/oprofile/op_model_ppro.c |    4 +++-
 2 files changed, 10 insertions(+), 1 deletions(-)
arch/x86/oprofile/nmi_int.c +7 -0
index 022cd41..1f4448e 100644
@@ -426,8 +426,15 @@ static int __init ppro_init(char **cpu_type)
 
 static int __init arch_perfmon_init(char **cpu_type)
 {
+	__u8 cpu_model = boot_cpu_data.x86_model;
+
 	if (!cpu_has_arch_perfmon)
 		return 0;
+
+	/* Nethalem only */
+	if (cpu_model != 26)
+		return 0;
+
 	*cpu_type = "i386/arch_perfmon";
 	model = &op_arch_perfmon_spec;
 	arch_perfmon_setup_counters();
arch/x86/oprofile/op_model_ppro.c +3 -1
index 0620d6d..1c91ba2 100644
@@ -66,6 +66,7 @@ static void ppro_fill_in_addresses(struct op_msrs * const msrs)
 
 static void ppro_setup_ctrs(struct op_msrs const * const msrs)
 {
+	__u8 cpu_model = boot_cpu_data.x86_model;
 	unsigned int low, high;
 	int i;
 
@@ -76,7 +77,8 @@ static void ppro_setup_ctrs(struct op_msrs const * const msrs)
 			return;
 	}
 
-	if (cpu_has_arch_perfmon) {
+	/* Nethalem only */
+	if (cpu_has_arch_perfmon && cpu_model == 26) {
 		union cpuid10_eax eax;
 		eax.full = cpuid_eax(0xa);
 		if (counter_width < eax.split.bit_width)
#14 lore raw
In reply to Pekka J Enberg
Quoted 7 lines · Pekka J Enberg

Hi Robert,

Quoted 2 lines · Robert Richter

your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.

Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?

The patch is not the correct fix to whatever the problem is. arch perfmon is not Nehalem specific.

-Andi

#5 lore raw
In reply to Eric Dumazet
Quoted 4 lines · Eric Dumazet
Quoted 3 lines · Pekka J Enberg

Hi Robert,

On Tue, 4 Nov 2008, Robert Richter wrote:

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.

Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?

Thanks a lot Pekka, this patch restores oprofile on my HP BL460c G1

Update :

After a while, no more NMI are delivered. No anymore samples.

# grep NMI /proc/interrupts
NMI:     214337    2004224     183702    2121878     641711     793553      18841     306364   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:     214337    2004224     183702    2121878     641711     793553      18841     306364   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:     214337    2004224     183702    2121878     641711     793553      18841     306364   Non-maskable interrupts
[root@svivoipvnx001 /opt/src/linux-2.6]$
# ps aux|grep oprof
root      5336  0.4  0.0  3512 1704 ?        S    07:02   0:01 /usr/local/bin/oprofiled --session-dir=/var/lib/oprofile --separate-lib=0 --separate-kernel=0 --separate-thread=0 --separate-cpu=0 --events=CPU_CLK_UNHALTED:60:0:10000:0:1:1, --vmlinux=vmlinux --kernel-range=c02000a0,c045e9c1
# opcontrol --stop
Stopping profiling.
# opcontrol --start
Profiler running.
# grep NMI /proc/interrupts
NMI:     214337    2004224     183702    2121878     641711     793553      18841     306364   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:     214337    2004224     183702    2121878     641711     793553      18841     306364   Non-maskable interrupts
Quoted 69 lines
Quoted 66 lines

Pekka

Quoted 1 line

From 37d362c0262006be6a5d6a72b5993834afa776d7 Mon Sep 17 00:00:00 2001

From: Pekka Enberg <[email protected]>
Date: Tue, 4 Nov 2008 21:17:47 +0200
Subject: [PATCH] oprofile: make perfmon nethalem only

Commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8 ("oprofile: Implement Intel architectural perfmon support") added perfmon support to newer Intel CPUs (Core1+). The problem there is that it requires a patched oprofile userland to work properly.

Fix that up by making oprofile perfmon support nethalem only.

---
 arch/x86/oprofile/nmi_int.c       |    7 +++++++
 arch/x86/oprofile/op_model_ppro.c |    4 +++-
 2 files changed, 10 insertions(+), 1 deletions(-)
arch/x86/oprofile/nmi_int.c +7 -0
index 022cd41..1f4448e 100644
@@ -426,8 +426,15 @@ static int __init ppro_init(char **cpu_type)
 
 static int __init arch_perfmon_init(char **cpu_type)
 {
+    __u8 cpu_model = boot_cpu_data.x86_model;
+
     if (!cpu_has_arch_perfmon)
         return 0;
+
+    /* Nethalem only */
+    if (cpu_model != 26)
+        return 0;
+
     *cpu_type = "i386/arch_perfmon";
     model = &op_arch_perfmon_spec;
     arch_perfmon_setup_counters();
arch/x86/oprofile/op_model_ppro.c +3 -1
index 0620d6d..1c91ba2 100644
@@ -66,6 +66,7 @@ static void ppro_fill_in_addresses(struct op_msrs * 
 const msrs)
 
 static void ppro_setup_ctrs(struct op_msrs const * const msrs)
 {
+    __u8 cpu_model = boot_cpu_data.x86_model;
     unsigned int low, high;
     int i;
@@ -76,7 +77,8 @@ static void ppro_setup_ctrs(struct op_msrs const * 
 const msrs)
             return;
     }
 
-    if (cpu_has_arch_perfmon) {
+    /* Nethalem only */
+    if (cpu_has_arch_perfmon && cpu_model == 26) {
         union cpuid10_eax eax;
         eax.full = cpuid_eax(0xa);

if (counter_width < eax.split.bit_width)

#6 lore raw
Quoted 10 lines · Eric Dumazet
Quoted 9 lines · Eric Dumazet
Quoted 7 lines · Pekka J Enberg

Hi Robert,

Quoted 2 lines · Robert Richter

your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.

Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?

Quoted 8 lines
Quoted 3 lines

Thanks a lot Pekka, this patch restores oprofile on my HP BL460c G1

Update :

After a while, no more NMI are delivered. No anymore samples.

Ingo asked me to test a revert of b99170288421c79f0c2efa8b33e26e65f4bb7fb8 (oprofile: Implement Intel architectural perfmon support)

Same problem than previous patch.

(doing a tbench 8 in background)

oprofile works a few seconds, then no more NMI / samples

# opcontrol --vmlinux=vmlinux --start
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.
# grep NMI /proc/interrupts
NMI:    3102535    1009518     733819    2088857    1414146     703880    1229563     473603   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:    3490492    1009518     767878    2212917    1812791    1654826    2151141     473603   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:    3490492    1009518     767878    2212917    1812791    2188653    2671979     473603   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:    3490492    1009518     767878    2212917    1812791    2612789    2671979     473603   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:    3490492    1009518     767878    2212917    1812791    2612789    2671979     473603   Non-maskable interrupts
# opcontrol --deinit
Stopping profiling.
Killing daemon.
# grep NMI /proc/interrupts
NMI:    3490492    1009518     767878    2212917    1812791    2612789    2671979     473603   Non-maskable interrupts
# opcontrol --vmlinux=vmlinux --start
Using 2.6+ OProfile kernel interface.
Reading module info.
Using log file /var/lib/oprofile/samples/oprofiled.log
Daemon started.
Profiler running.
# grep NMI /proc/interrupts
NMI:    3730357    1265822    1174545    2485635    2257881    3082318    3078445     931068   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:    3730357    1265822    1573509    2485635    2469701    3512810    3477233    1143043   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:    3730357    1265822    2069149    2485635    2469701    3512810    3511350    1143043   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:    3730357    1265822    2069149    2485635    2469701    3512810    3511350    1143043   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:    3730357    1265822    2069149    2485635    2469701    3512810    3511350    1143043   Non-maskable interrupts
# grep . /dev/oprofile/*/* /dev/oprofile/*
/dev/oprofile/0/count:10000
/dev/oprofile/0/enabled:1
/dev/oprofile/0/event:60
/dev/oprofile/0/kernel:1
/dev/oprofile/0/unit_mask:0
/dev/oprofile/0/user:1
/dev/oprofile/1/count:0
/dev/oprofile/1/enabled:0
/dev/oprofile/1/event:0
/dev/oprofile/1/kernel:0
/dev/oprofile/1/unit_mask:0
/dev/oprofile/1/user:0
/dev/oprofile/stats/bt_lost_no_mapping:0
/dev/oprofile/stats/event_lost_overflow:83466
/dev/oprofile/stats/sample_lost_no_mapping:19174
/dev/oprofile/stats/sample_lost_no_mm:0
/dev/oprofile/backtrace_depth:0
grep: /dev/oprofile/buffer: Device or resource busy
/dev/oprofile/buffer_size:131072
/dev/oprofile/buffer_watershed:32768
/dev/oprofile/cpu_buffer_size:8192
/dev/oprofile/cpu_type:i386/core_2
grep: /dev/oprofile/dump: Invalid argument
/dev/oprofile/enable:1
/dev/oprofile/pointer_size:4

I tried with bigger buffer_size, same problem

opcontrol --vmlinux=vmlinux --cpu-buffer-size=65536 --buffer-size=1048576 --start

# grep . /dev/oprofile/*/* /dev/oprofile/*
/dev/oprofile/0/count:10000
/dev/oprofile/0/enabled:1
/dev/oprofile/0/event:60
/dev/oprofile/0/kernel:1
/dev/oprofile/0/unit_mask:0
/dev/oprofile/0/user:1
/dev/oprofile/1/count:0
/dev/oprofile/1/enabled:0
/dev/oprofile/1/event:0
/dev/oprofile/1/kernel:0
/dev/oprofile/1/unit_mask:0
/dev/oprofile/1/user:0
/dev/oprofile/stats/bt_lost_no_mapping:0
/dev/oprofile/stats/event_lost_overflow:1465392
/dev/oprofile/stats/sample_lost_no_mapping:117498
/dev/oprofile/stats/sample_lost_no_mm:20
/dev/oprofile/backtrace_depth:0
grep: /dev/oprofile/buffer: Device or resource busy
/dev/oprofile/buffer_size:1048576
/dev/oprofile/buffer_watershed:32768
/dev/oprofile/cpu_buffer_size:65536
/dev/oprofile/cpu_type:i386/core_2
grep: /dev/oprofile/dump: Invalid argument
/dev/oprofile/enable:1
/dev/oprofile/pointer_size:4
#7 lore raw

Can you please test this patch and see if it solves the problem?

Thanks,
-Andi

---

oprofile: Fix p6 counter overflow check

Fix the counter overflow check for CPUs with counter width > 32

I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.

---
 arch/x86/oprofile/op_model_ppro.c |    9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)
arch/x86/oprofile/op_model_ppro.c +4 -5
Index: linux-2.6.28-rc3-test/arch/x86/oprofile/op_model_ppro.c
===================================================================
@@ -27,8 +27,7 @@
 static int counter_width = 32;
 
 #define CTR_IS_RESERVED(msrs, c) (msrs->counters[(c)].addr ? 1 : 0)
-#define CTR_READ(l, h, msrs, c) do {rdmsr(msrs->counters[(c)].addr, (l), (h)); } while (0)
-#define CTR_OVERFLOWED(n) (!((n) & (1U<<(counter_width-1))))
+#define CTR_OVERFLOWED(n) (!((n) & (1ULL<<(counter_width-1))))
 
 #define CTRL_IS_RESERVED(msrs, c) (msrs->controls[(c)].addr ? 1 : 0)
 #define CTRL_READ(l, h, msrs, c) do {rdmsr((msrs->controls[(c)].addr), (l), (h)); } while (0)
@@ -124,14 +123,14 @@
 static int ppro_check_ctrs(struct pt_regs * const regs,
 			   struct op_msrs const * const msrs)
 {
-	unsigned int low, high;
+	u64 val;
 	int i;
 
 	for (i = 0 ; i < num_counters; ++i) {
 		if (!reset_value[i])
 			continue;
-		CTR_READ(low, high, msrs, i);
-		if (CTR_OVERFLOWED(low)) {
+		rdmsrl(msrs->counters[i].addr, val);
+		if (CTR_OVERFLOWED(val)) {
 			oprofile_add_sample(regs, i);
 			wrmsrl(msrs->counters[i].addr, -reset_value[i]);
 		}
Signature · 1 line
[email protected]
#8 lore raw
Quoted 9 lines · Andi Kleen

Can you please test this patch and see if it solves the problem?

Thanks,
-Andi

---

oprofile: Fix p6 counter overflow check

Quoted 8 lines

Fix the counter overflow check for CPUs with counter width > 32

I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.

Thanks Andi

Unfortunatly, there is still a freeze of NMIS after a while.

# grep NMI /proc/interrupts
NMI:     127869     439091     905577       6158     809531      54720    4402881    2411582   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:     294473     765839    1174243     282249    1190866     382543    4747495    2463270   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:     294473     899262    1174243     282249    1282407     515973    5139226    2463270   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:     294473     899262    1174243     282249    1282407     515973    5187081    2463270   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:     294473     899262    1174243     282249    1282407     515973    5187081    2463270   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:     294473     899262    1174243     282249    1282407     515973    5187081    2463270   Non-maskable interrupts
# grep NMI /proc/interrupts
NMI:     294473     899262    1174243     282249    1282407     515973    5187081    2463270   Non-maskable interrupts
#
# grep . /dev/oprofile/*/*
/dev/oprofile/0/count:10000
/dev/oprofile/0/enabled:1
/dev/oprofile/0/event:60
/dev/oprofile/0/kernel:1
/dev/oprofile/0/unit_mask:0
/dev/oprofile/0/user:1
/dev/oprofile/1/count:0
/dev/oprofile/1/enabled:0
/dev/oprofile/1/event:0
/dev/oprofile/1/kernel:0
/dev/oprofile/1/unit_mask:0
/dev/oprofile/1/user:0
/dev/oprofile/stats/bt_lost_no_mapping:0
/dev/oprofile/stats/event_lost_overflow:376941
/dev/oprofile/stats/sample_lost_no_mapping:43358
/dev/oprofile/stats/sample_lost_no_mm:26
# grep . /dev/oprofile/*
/dev/oprofile/backtrace_depth:0
grep: /dev/oprofile/buffer: Device or resource busy
/dev/oprofile/buffer_size:1048576
/dev/oprofile/buffer_watershed:32768
/dev/oprofile/cpu_buffer_size:65536
/dev/oprofile/cpu_type:i386/core_2
grep: /dev/oprofile/dump: Invalid argument
/dev/oprofile/enable:1
/dev/oprofile/pointer_size:4
Quoted 38 lines
---
 arch/x86/oprofile/op_model_ppro.c |    9 ++++-----
 1 file changed, 4 insertions(+), 5 deletions(-)

Index: linux-2.6.28-rc3-test/arch/x86/oprofile/op_model_ppro.c
===================================================================

arch/x86/oprofile/op_model_ppro.c +4 -5
@@ -27,8 +27,7 @@
 static int counter_width = 32;
 
 #define CTR_IS_RESERVED(msrs, c) (msrs->counters[(c)].addr ? 1 : 0)
-#define CTR_READ(l, h, msrs, c) do {rdmsr(msrs->counters[(c)].addr, (l), (h)); } while (0)
-#define CTR_OVERFLOWED(n) (!((n) & (1U<<(counter_width-1))))
+#define CTR_OVERFLOWED(n) (!((n) & (1ULL<<(counter_width-1))))
 
 #define CTRL_IS_RESERVED(msrs, c) (msrs->controls[(c)].addr ? 1 : 0)
 #define CTRL_READ(l, h, msrs, c) do {rdmsr((msrs->controls[(c)].addr), (l), (h)); } while (0)
@@ -124,14 +123,14 @@
 static int ppro_check_ctrs(struct pt_regs * const regs,
 			   struct op_msrs const * const msrs)
 {
-	unsigned int low, high;
+	u64 val;
 	int i;
 
 	for (i = 0 ; i < num_counters; ++i) {
 		if (!reset_value[i])
 			continue;
-		CTR_READ(low, high, msrs, i);
-		if (CTR_OVERFLOWED(low)) {
+		rdmsrl(msrs->counters[i].addr, val);
+		if (CTR_OVERFLOWED(val)) {
 			oprofile_add_sample(regs, i);
 			wrmsrl(msrs->counters[i].addr, -reset_value[i]);
 		}
#9 lore raw
Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Andi Kleen

Can you please test this patch and see if it solves the problem?

Thanks,
-Andi

---

oprofile: Fix p6 counter overflow check

Fix the counter overflow check for CPUs with counter width > 32

I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.

Quoted 6 lines
Quoted 2 lines

Thanks Andi

Unfortunatly, there is still a freeze of NMIS after a while.

How long is the while?

I don't see this here on a Merom when I let it run for 10 minutes or so.

You might need to bisect it.

-Andi

#10 lore raw
Quoted 17 lines · Andi Kleen
Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Andi Kleen

Can you please test this patch and see if it solves the problem?

Thanks,
-Andi

---

oprofile: Fix p6 counter overflow check

Fix the counter overflow check for CPUs with counter width > 32

I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Thanks Andi

Unfortunatly, there is still a freeze of NMIS after a while.

How long is the while?

Less than 20 seconds if fully loaded with "tbench 8" on my 8 cpus machine.

With your patch, many events are handled by oprofile (about 10.000.000 last time I tried) before freeze.

Without your patch, freeze was very very fast, no more than 10.000 NMI...

Quoted 4 lines

I don't see this here on a Merom when I let it run for 10 minutes or so.

You might need to bisect it.

Too many bugs during bisection unfortunatly, machine doesnt boot :(

#13 lore raw
In reply to Andi Kleen
Quoted 10 lines · Andi Kleen

Can you please test this patch and see if it solves the problem?

Thanks,
-Andi

---

oprofile: Fix p6 counter overflow check

Quoted 7 lines

Fix the counter overflow check for CPUs with counter width > 32

I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.

Andi,

your patch fixes the initial bug that stops sampling immediately after starting. The issue Eric mentions seems to be a different problem. I am also not able to reproduce this. Your patch on top of rc3 continues sampling at least for several minutes.

I will apply your fix.

Thanks Andi,

-Robert

Signature · 3 lines
Advanced Micro Devices, Inc.
Operating System Research Center
email: [email protected]
#11 lore raw
In reply to Eric Dumazet
Quoted 20 lines · Eric Dumazet
Quoted 19 lines · Andi Kleen
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Andi Kleen

Can you please test this patch and see if it solves the problem?

Thanks,
-Andi

---

oprofile: Fix p6 counter overflow check

Fix the counter overflow check for CPUs with counter width > 32

I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Thanks Andi

Unfortunatly, there is still a freeze of NMIS after a while.

How long is the while?

Less than 20 seconds if fully loaded with "tbench 8" on my 8 cpus machine.

I don't have tbench on the test box, but when I load it with a couple of while true ; do true ; done loops it also continues sampling.

Are you sure it worked in 2.6.27?

-Andi

#12 lore raw
Quoted 24 lines · Andi Kleen
Quoted 23 lines · Eric Dumazet
Quoted 22 lines · Andi Kleen
Quoted 21 lines · Eric Dumazet
Quoted 9 lines · Andi Kleen

Can you please test this patch and see if it solves the problem?

Thanks,
-Andi

---

oprofile: Fix p6 counter overflow check

Quoted 8 lines

Fix the counter overflow check for CPUs with counter width > 32

I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.

Thanks Andi

Unfortunatly, there is still a freeze of NMIS after a while.

Quoted 8 lines
Quoted 3 lines
Quoted 2 lines

How long is the while?

Less than 20 seconds if fully loaded with "tbench 8" on my 8 cpus machine.

I don't have tbench on the test box, but when I load it with a couple of while true ; do true ; done loops it also continues sampling.

Are you sure it worked in 2.6.27?

You are right... I got same problem with 2.6.27.4, it seems...

$ grep NMI /proc/interrupts
NMI:     423689      79703    1576236     580137    1405669    1187909    2923037    5163299   Non-maskable interrupts
$ grep NMI /proc/interrupts
NMI:     423689      79703    1576236     580137    1405669    1187909    2923037    5163299   Non-maskable interrupts

I'll try to find the faulty commit.

Thank you