linux-2.6.28-rc2 regression : oprofile doesnt work anymore ?
Hi
I tried to start my favorite tool, and discovered it wont fly on my HP BL460C G1
It samples few events then, seems to stop (no more NMI), other than that, machine still running correctly.
(8 cpus like that one
processor : 0vendor_id : GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
stepping : 6
cpu MHz : 3000.253
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0fdiv_bug : nohlt_bug : nof00f_bug : nocoma_bug : no
fpu : yesfpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lm constant_tsc arch_perfmon pebs bts pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 lahf_lm tpr_shadow vnmi flexpriority
bogomips : 6000.50
clflush size : 64
power management:
It is working correctly on 2.6.27
I tried a git bisect but quickly found a point where machine was not useable... (partly because of the problem with the VPD problem on 5708S (PCI: Limit VPD length for Broadcom 5708S)
# opcontrol --deinit Stopping profiling. Killing daemon. # rm -rf /var/lib/oprofile/samples/current # opcontrol --vmlinux=vmlinux --start Using 2.6+ OProfile kernel interface. Reading module info. Using log file /var/lib/oprofile/samples/oprofiled.log Daemon started. Profiler running.
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
samples| %|
------------------
17 80.9524 vmlinux
2 9.5238 libc-2.3.4.so
1 4.7619 bash
1 4.7619 ld-2.3.4.so
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
samples| %|
------------------
17 80.9524 vmlinux
2 9.5238 libc-2.3.4.so
1 4.7619 bash
1 4.7619 ld-2.3.4.so
# opcontrol --dump; opreport
CPU: Core 2, speed 3000.25 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000
CPU_CLK_UNHALT...|
samples| %|
------------------
17 80.9524 vmlinux
2 9.5238 libc-2.3.4.so
1 4.7619 bash
1 4.7619 ld-2.3.4.so# grep NMI /proc/interrupts NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts
(adding some more cc's)
Quoted 91 lines · Eric Dumazet
I tried to start my favorite tool, and discovered it wont fly on my HP BL460C G1
It samples few events then, seems to stop (no more NMI), other than that, machine
still running correctly.(8 cpus like that one
processor : 0
vendor_id: GenuineIntel
cpu family : 6
model : 23
model name : Intel(R) Xeon(R) CPU E5450 @ 3.00GHz
stepping : 6
cpu MHz : 3000.253
cache size : 6144 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0fdiv_bug: nohlt_bug: nof00f_bug: nocoma_bug: no
fpu : yesfpu_exception: yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe lmconstant_tscarch_perfmonpebs bts pni dtes64 monitords_cplvmx est tm2 ssse3 cx16 xtpr pdcm dcasse4_1lahf_lmtpr_shadowvnmi flexpriority
bogomips : 6000.50
clflush size : 64
power management:It is working correctly on 2.6.27
I tried a git bisect but quickly found a point where machine was not useable...
(partly because of the problem with the VPD problem on 5708S (PCI: Limit VPD length for Broadcom 5708S)# opcontrol --deinit Stopping profiling. Killing daemon. # rm -rf /var/lib/oprofile/samples/current # opcontrol --vmlinux=vmlinux --start Using 2.6+ OProfile kernel interface. Reading module info. Using log file /var/lib/oprofile/samples/oprofiled.log Daemon started. Profiler running.# opcontrol --dump; opreport CPU: Core 2, speed 3000.25 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000 CPU_CLK_UNHALT...| samples| %| ------------------ 17 80.9524 vmlinux 2 9.5238 libc-2.3.4.so 1 4.7619 bash 1 4.7619 ld-2.3.4.so # opcontrol --dump; opreport CPU: Core 2, speed 3000.25 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000 CPU_CLK_UNHALT...| samples| %| ------------------ 17 80.9524 vmlinux 2 9.5238 libc-2.3.4.so 1 4.7619 bash 1 4.7619 ld-2.3.4.so # opcontrol --dump; opreport CPU: Core 2, speed 3000.25 MHz (estimated) Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 10000 CPU_CLK_UNHALT...| samples| %| ------------------ 17 80.9524 vmlinux 2 9.5238 libc-2.3.4.so 1 4.7619 bash 1 4.7619 ld-2.3.4.so
Quoted 12 lines
# grep NMI /proc/interrupts NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 8 2 2 6 11 7 6 5 Non-maskable interrupts
I'm seeing the same thing on my 2-way x86-64. Here's my /proc/cpuinfo in case it helps:
processor : 0vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz
stepping : 6
cpu MHz : 2000.000
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yesfpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 3989.97
clflush size : 64cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 1vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Core(TM)2 CPU T7200 @ 2.00GHz
stepping : 6
cpu MHz : 2000.000
cache size : 4096 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yesfpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm tpr_shadow
bogomips : 3989.97
clflush size : 64cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
Andi,
your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.
Thanks,
-Robert
Linux woglinde 2.6.27-oprofile-x86_64-standard-03189-g5951290 #5 SMP Tue Nov 4 18:16:41 CET 2008 x86_64 Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz GenuineIntel GNU/Linux IBS Fetch not supported - ignored
IBS OP not supported - ignored
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
Oprofile started
Daemon running: pid 5845
Separate options: none
vmlinux file: none
Image filter: none
Call-graph depth: 0
number of samples #1: 11
Stopping profiling.
Killing daemon.
number of samples #2: 11
Linux woglinde 2.6.27-oprofile-x86_64-standard-03185-gc493756 #6 SMP Tue Nov 4 19:05:13 CET 2008 x86_64 Intel(R) Core(TM)2 CPU 6300 @ 1.86GHz GenuineIntel GNU/Linux IBS Fetch not supported - ignored
IBS OP not supported - ignored
Using default event: CPU_CLK_UNHALTED:100000:0:1:1
Daemon started.
Profiler running.
Oprofile started
Daemon running: pid 5263
Separate options: none
vmlinux file: none
Image filter: none
Call-graph depth: 0
number of samples #1: 25
Stopping profiling.
Killing daemon.
number of samples #2: 64
Increasing sample number
commit 59512900baab03c5629f2ff5efad1d5d4e682ece
Author: Andi Kleen <[email protected]> Date: Mon Sep 29 22:23:33 2008 +0200
oprofile: discover counters for op ppro too
Discover number of counters for all family 6 models even when not in arch perfmon mode.
arch/x86/oprofile/op_model_ppro.c | 8 +++++--- arch/x86/oprofile/op_x86_model.h | 2 +- 2 files changed, 6 insertions(+), 4 deletions(-)
commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8
Author: Andi Kleen <[email protected]> Date: Mon Aug 18 14:50:31 2008 +0200
oprofile: Implement Intel architectural perfmon support
Newer Intel CPUs (Core1+) have support for architectural events described in CPUID 0xA. See the IA32 SDM Vol3b.18 for details.
The advantage of this is that it can be done without knowing about the specific CPU, because the CPU describes by itself what performance events are supported. This is only a fallback because only a limited set of 6 events are supported. This allows to do profiling on Nehalem and on Atom systems (later not tested)
This patch implements support for that in oprofile's Intel Family 6 profiling module. It also has the advantage of supporting an arbitary number of events now as reported by the CPU. Also allow arbitary counter widths >32bit while we're at it.
Requires a patched oprofile userland to support the new architecture.
v2: update for latest oprofile tree
remove force_arch_perfmon
arch/x86/oprofile/nmi_int.c | 23 ++++++-- arch/x86/oprofile/op_model_ppro.c | 104 +++++++++++++++++++++++++++++-------- arch/x86/oprofile/op_x86_model.h | 3 + 3 files changed, 102 insertions(+), 28 deletions(-)
commit f645f6406463a01869c50844befc76d528971690
Author: Andi Kleen <[email protected]> Date: Wed Aug 20 17:22:02 2008 +0200
oprofile: Don't report Nehalem as core_2
This essentially reverts Linus' earlier 4b9f12a3779c548b68bc9af7d94030868ad3aa1b commit. Nehalem is not core_2, so it shouldn't be reported as such. However with the earlier arch perfmon patch it will fall back to arch perfmon mode now, so there is no need to fake it as core_2. The only drawback is that Linus will need to patch the arch perfmon support into his oprofile binary now, but I think he can do that.
arch/x86/oprofile/nmi_int.c | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-)
commit 5d4488027d9cf3161c71566dfabb116bf69ab4d9
Author: Andi Kleen <[email protected]> Date: Mon Aug 18 14:49:47 2008 +0200
oprofile: drop const in num counters field
allow to modify it at runtime
arch/x86/oprofile/op_x86_model.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
Signature · 3 lines
Advanced Micro Devices, Inc. Operating System Research Center email: [email protected]
Hi Robert,
Quoted 2 lines · Robert Richter
your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.
Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?
Pekka
Quoted 1 line
From
37d362c0262006be6a5d6a72b5993834afa776d7Mon Sep 17 00:00:00 2001
From: Pekka Enberg <[email protected]>
Date: Tue, 4 Nov 2008 21:17:47 +0200
Subject: [PATCH] oprofile: make perfmon nethalem only
Commit b99170288421c79f0c2efa8b33e26e65f4bb7fb8 ("oprofile: Implement Intel architectural perfmon support") added perfmon support to newer Intel CPUs (Core1+). The problem there is that it requires a patched oprofile userland to work properly.
Fix that up by making oprofile perfmon support nethalem only.
--- arch/x86/oprofile/nmi_int.c | 7 +++++++ arch/x86/oprofile/op_model_ppro.c | 4 +++- 2 files changed, 10 insertions(+), 1 deletions(-)
arch/x86/oprofile/nmi_int.c +7 -0
@@ -426,8 +426,15 @@ static int __init ppro_init(char **cpu_type)static int __init arch_perfmon_init (char **cpu_type) { +__u8 cpu_model = boot_cpu_data.x86_model; +if (!cpu_has_arch_perfmon)return 0 ; + +/* Nethalem only */ +if (cpu_model !=26 ) +return 0 ; + *cpu_type ="i386/arch_perfmon" ; model = &op_arch_perfmon_spec;arch_perfmon_setup_counters ();
arch/x86/oprofile/op_model_ppro.c +3 -1
@@ -66,6 +66,7 @@ static void ppro_fill_in_addresses(struct op_msrs * const msrs)static void ppro_setup_ctrs (struct op_msrs const *const msrs) { +__u8 cpu_model = boot_cpu_data.x86_model;unsigned int low, high;int i;
@@ -76,7 +77,8 @@ static void ppro_setup_ctrs(struct op_msrs const * const msrs)return ; } -if (cpu_has_arch_perfmon) { +/* Nethalem only */ +if (cpu_has_arch_perfmon && cpu_model ==26 ) {union cpuid10_eax eax; eax.full =cpuid_eax (0xa );if (counter_width < eax.split.bit_width)
Signature · 1 line
1.5.3.7
Eric Dumazet
Tested-byQuoted 7 lines · Pekka J Enberg
Hi Robert,
Quoted 2 lines · Robert Richter
your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.
Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?
Thanks a lot Pekka, this patch restores oprofile on my HP BL460c G1
Quoted 63 lines
Pekka
Quoted 1 line
From
37d362c0262006be6a5d6a72b5993834afa776d7Mon Sep 17 00:00:00 2001From: Pekka Enberg <[email protected]>
Date: Tue, 4 Nov 2008 21:17:47 +0200
Subject: [PATCH] oprofile: make perfmon nethalem onlyCommit
b99170288421c79f0c2efa8b33e26e65f4bb7fb8("oprofile: Implement Intel architectural perfmon support") added perfmon support to newer Intel CPUs (Core1+). The problem there is that it requires a patched oprofile userland to work properly.Fix that up by making oprofile perfmon support nethalem only.
Signed-off-by Pekka Enberg <[email protected]>--- arch/x86/oprofile/nmi_int.c | 7 +++++++ arch/x86/oprofile/op_model_ppro.c | 4 +++- 2 files changed, 10 insertions(+), 1 deletions(-)arch/x86/oprofile/nmi_int.c +7
-0@@ -426,8 +426,15 @@ static int __init ppro_init(char **cpu_type)static int __init arch_perfmon_init (char **cpu_type) { +__u8 cpu_model = boot_cpu_data.x86_model; +if (!cpu_has_arch_perfmon)return 0 ; + +/* Nethalem only */ +if (cpu_model !=26 ) +return 0 ; + *cpu_type ="i386/arch_perfmon" ; model = &op_arch_perfmon_spec;arch_perfmon_setup_counters ();arch/x86/oprofile/op_model_ppro.c +3
-1@@ -66,6 +66,7 @@ static void ppro_fill_in_addresses(struct op_msrs * const msrs)static void ppro_setup_ctrs (struct op_msrs const *const msrs) { +__u8 cpu_model = boot_cpu_data.x86_model;unsigned int low, high;int i;@@ -76,7 +77,8 @@ static void ppro_setup_ctrs(struct op_msrs const * const msrs)return ; } -if (cpu_has_arch_perfmon) { +/* Nethalem only */ +if (cpu_has_arch_perfmon && cpu_model ==26 ) {union cpuid10_eax eax; eax.full =cpuid_eax (0xa );if (counter_width < eax.split.bit_width)
Quoted 4 lines · Eric Dumazet
Quoted 3 lines · Pekka J Enberg
Hi Robert,
On Tue, 4 Nov 2008, Robert Richter wrote:
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.
Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?
Thanks a lot Pekka, this patch restores oprofile on my HP BL460c G1
Tested-by Eric Dumazet <[email protected]>
Update :
After a while, no more NMI are delivered. No anymore samples.
# grep NMI /proc/interrupts NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts [root@svivoipvnx001 /opt/src/linux-2.6]$
# ps aux|grep oprof root 5336 0.4 0.0 3512 1704 ? S 07:02 0:01 /usr/local/bin/oprofiled --session-dir=/var/lib/oprofile --separate-lib=0 --separate-kernel=0 --separate-thread=0 --separate-cpu=0 --events=CPU_CLK_UNHALTED:60:0:10000:0:1:1, --vmlinux=vmlinux --kernel-range=c02000a0,c045e9c1
# opcontrol --stop Stopping profiling. # opcontrol --start Profiler running. # grep NMI /proc/interrupts NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 214337 2004224 183702 2121878 641711 793553 18841 306364 Non-maskable interrupts
Quoted 69 lines
Quoted 66 lines
Pekka
Quoted 1 line
From
37d362c0262006be6a5d6a72b5993834afa776d7Mon Sep 17 00:00:00 2001From: Pekka Enberg <[email protected]>
Date: Tue, 4 Nov 2008 21:17:47 +0200
Subject: [PATCH] oprofile: make perfmon nethalem onlyCommit
b99170288421c79f0c2efa8b33e26e65f4bb7fb8("oprofile: Implement Intel architectural perfmon support") added perfmon support to newer Intel CPUs (Core1+). The problem there is that it requires a patched oprofile userland to work properly.Fix that up by making oprofile perfmon support nethalem only.
Signed-off-by Pekka Enberg <[email protected]>--- arch/x86/oprofile/nmi_int.c | 7 +++++++ arch/x86/oprofile/op_model_ppro.c | 4 +++- 2 files changed, 10 insertions(+), 1 deletions(-)arch/x86/oprofile/nmi_int.c +7
-0@@ -426,8 +426,15 @@ static int __init ppro_init(char **cpu_type)static int __init arch_perfmon_init (char **cpu_type) { +__u8 cpu_model = boot_cpu_data.x86_model; +if (!cpu_has_arch_perfmon)return 0 ; + +/* Nethalem only */ +if (cpu_model !=26 ) +return 0 ; + *cpu_type ="i386/arch_perfmon" ; model = &op_arch_perfmon_spec;arch_perfmon_setup_counters ();arch/x86/oprofile/op_model_ppro.c +3
-1@@ -66,6 +66,7 @@ static void ppro_fill_in_addresses(struct op_msrs * const msrs)static void ppro_setup_ctrs (struct op_msrs const *const msrs) { +__u8 cpu_model = boot_cpu_data.x86_model;unsigned int low, high;int i;@@ -76,7 +77,8 @@ static void ppro_setup_ctrs(struct op_msrs const * const msrs)return ; } -if (cpu_has_arch_perfmon) { +/* Nethalem only */ +if (cpu_has_arch_perfmon && cpu_model ==26 ) {union cpuid10_eax eax; eax.full =cpuid_eax (0xa );if (
counter_width< eax.split.bit_width)
Quoted 10 lines · Eric Dumazet
Quoted 9 lines · Eric Dumazet
Quoted 7 lines · Pekka J Enberg
Hi Robert,
Quoted 2 lines · Robert Richter
your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.
Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?
Quoted 8 lines
Quoted 3 lines
Thanks a lot Pekka, this patch restores oprofile on my HP BL460c G1
Tested-by Eric Dumazet <[email protected]>Update :
After a while, no more NMI are delivered. No anymore samples.
Ingo asked me to test a revert of b99170288421c79f0c2efa8b33e26e65f4bb7fb8 (oprofile: Implement Intel architectural perfmon support)
Same problem than previous patch.
(doing a tbench 8 in background)
oprofile works a few seconds, then no more NMI / samples
# opcontrol --vmlinux=vmlinux --start Using 2.6+ OProfile kernel interface. Reading module info. Using log file /var/lib/oprofile/samples/oprofiled.log Daemon started. Profiler running. # grep NMI /proc/interrupts NMI: 3102535 1009518 733819 2088857 1414146 703880 1229563 473603 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 3490492 1009518 767878 2212917 1812791 1654826 2151141 473603 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 3490492 1009518 767878 2212917 1812791 2188653 2671979 473603 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 3490492 1009518 767878 2212917 1812791 2612789 2671979 473603 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 3490492 1009518 767878 2212917 1812791 2612789 2671979 473603 Non-maskable interrupts # opcontrol --deinit Stopping profiling. Killing daemon. # grep NMI /proc/interrupts NMI: 3490492 1009518 767878 2212917 1812791 2612789 2671979 473603 Non-maskable interrupts # opcontrol --vmlinux=vmlinux --start Using 2.6+ OProfile kernel interface. Reading module info. Using log file /var/lib/oprofile/samples/oprofiled.log Daemon started. Profiler running. # grep NMI /proc/interrupts NMI: 3730357 1265822 1174545 2485635 2257881 3082318 3078445 931068 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 3730357 1265822 1573509 2485635 2469701 3512810 3477233 1143043 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 3730357 1265822 2069149 2485635 2469701 3512810 3511350 1143043 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 3730357 1265822 2069149 2485635 2469701 3512810 3511350 1143043 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 3730357 1265822 2069149 2485635 2469701 3512810 3511350 1143043 Non-maskable interrupts
# grep . /dev/oprofile/*/* /dev/oprofile/* /dev/oprofile/0/count:10000 /dev/oprofile/0/enabled:1 /dev/oprofile/0/event:60 /dev/oprofile/0/kernel:1 /dev/oprofile/0/unit_mask:0 /dev/oprofile/0/user:1 /dev/oprofile/1/count:0 /dev/oprofile/1/enabled:0 /dev/oprofile/1/event:0 /dev/oprofile/1/kernel:0 /dev/oprofile/1/unit_mask:0 /dev/oprofile/1/user:0 /dev/oprofile/stats/bt_lost_no_mapping:0 /dev/oprofile/stats/event_lost_overflow:83466 /dev/oprofile/stats/sample_lost_no_mapping:19174 /dev/oprofile/stats/sample_lost_no_mm:0 /dev/oprofile/backtrace_depth:0 grep: /dev/oprofile/buffer: Device or resource busy /dev/oprofile/buffer_size:131072 /dev/oprofile/buffer_watershed:32768 /dev/oprofile/cpu_buffer_size:8192 /dev/oprofile/cpu_type:i386/core_2 grep: /dev/oprofile/dump: Invalid argument /dev/oprofile/enable:1 /dev/oprofile/pointer_size:4
I tried with bigger buffer_size, same problem
opcontrol --vmlinux=vmlinux --cpu-buffer-size=65536 --buffer-size=1048576 --start
# grep . /dev/oprofile/*/* /dev/oprofile/* /dev/oprofile/0/count:10000 /dev/oprofile/0/enabled:1 /dev/oprofile/0/event:60 /dev/oprofile/0/kernel:1 /dev/oprofile/0/unit_mask:0 /dev/oprofile/0/user:1 /dev/oprofile/1/count:0 /dev/oprofile/1/enabled:0 /dev/oprofile/1/event:0 /dev/oprofile/1/kernel:0 /dev/oprofile/1/unit_mask:0 /dev/oprofile/1/user:0 /dev/oprofile/stats/bt_lost_no_mapping:0 /dev/oprofile/stats/event_lost_overflow:1465392 /dev/oprofile/stats/sample_lost_no_mapping:117498 /dev/oprofile/stats/sample_lost_no_mm:20 /dev/oprofile/backtrace_depth:0 grep: /dev/oprofile/buffer: Device or resource busy /dev/oprofile/buffer_size:1048576 /dev/oprofile/buffer_watershed:32768 /dev/oprofile/cpu_buffer_size:65536 /dev/oprofile/cpu_type:i386/core_2 grep: /dev/oprofile/dump: Invalid argument /dev/oprofile/enable:1 /dev/oprofile/pointer_size:4
Can you please test this patch and see if it solves the problem?
Thanks,
-Andi
---
oprofile: Fix p6 counter overflow check
Fix the counter overflow check for CPUs with counter width > 32
I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.
--- arch/x86/oprofile/op_model_ppro.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
arch/x86/oprofile/op_model_ppro.c +4 -5
@@ -27,8 +27,7 @@static int counter_width =32 ;#define CTR_IS_RESERVED (msrs, c) (msrs->counters[(c)].addr ? 1 : 0) -#define CTR_READ (l, h, msrs, c) do {rdmsr(msrs->counters[(c)].addr, (l), (h)); } while (0) -#define CTR_OVERFLOWED (n) (!((n) & (1U<<(counter_width-1)))) +#define CTR_OVERFLOWED (n) (!((n) & (1ULL<<(counter_width-1))))#define CTRL_IS_RESERVED (msrs, c) (msrs->controls[(c)].addr ? 1 : 0)#define CTRL_READ (l, h, msrs, c) do {rdmsr((msrs->controls[(c)].addr), (l), (h)); } while (0)
@@ -124,14 +123,14 @@static int ppro_check_ctrs (struct pt_regs *const regs,struct op_msrs const *const msrs) { -unsigned int low, high; +u64 val;int i;for (i =0 ; i < num_counters; ++i) {if (!reset_value[i])continue ; -CTR_READ (low, high, msrs, i); -if (CTR_OVERFLOWED (low)) { +rdmsrl (msrs->counters[i].addr, val); +if (CTR_OVERFLOWED (val)) {oprofile_add_sample (regs, i);wrmsrl (msrs->counters[i].addr, -reset_value[i]); }
Signature · 1 line
[email protected]
Quoted 9 lines · Andi Kleen
Can you please test this patch and see if it solves the problem?
Thanks,
-Andi---
oprofile: Fix p6 counter overflow check
Quoted 8 lines
Fix the counter overflow check for CPUs with counter width > 32
I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.
Signed-off-by Andi Kleen <[email protected]>
Thanks Andi
Unfortunatly, there is still a freeze of NMIS after a while.
# grep NMI /proc/interrupts NMI: 127869 439091 905577 6158 809531 54720 4402881 2411582 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 294473 765839 1174243 282249 1190866 382543 4747495 2463270 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 294473 899262 1174243 282249 1282407 515973 5139226 2463270 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 294473 899262 1174243 282249 1282407 515973 5187081 2463270 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 294473 899262 1174243 282249 1282407 515973 5187081 2463270 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 294473 899262 1174243 282249 1282407 515973 5187081 2463270 Non-maskable interrupts # grep NMI /proc/interrupts NMI: 294473 899262 1174243 282249 1282407 515973 5187081 2463270 Non-maskable interrupts #
# grep . /dev/oprofile/*/* /dev/oprofile/0/count:10000 /dev/oprofile/0/enabled:1 /dev/oprofile/0/event:60 /dev/oprofile/0/kernel:1 /dev/oprofile/0/unit_mask:0 /dev/oprofile/0/user:1 /dev/oprofile/1/count:0 /dev/oprofile/1/enabled:0 /dev/oprofile/1/event:0 /dev/oprofile/1/kernel:0 /dev/oprofile/1/unit_mask:0 /dev/oprofile/1/user:0 /dev/oprofile/stats/bt_lost_no_mapping:0 /dev/oprofile/stats/event_lost_overflow:376941 /dev/oprofile/stats/sample_lost_no_mapping:43358 /dev/oprofile/stats/sample_lost_no_mm:26
# grep . /dev/oprofile/* /dev/oprofile/backtrace_depth:0 grep: /dev/oprofile/buffer: Device or resource busy /dev/oprofile/buffer_size:1048576 /dev/oprofile/buffer_watershed:32768 /dev/oprofile/cpu_buffer_size:65536 /dev/oprofile/cpu_type:i386/core_2 grep: /dev/oprofile/dump: Invalid argument /dev/oprofile/enable:1 /dev/oprofile/pointer_size:4
Quoted 38 lines
--- arch/x86/oprofile/op_model_ppro.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)Index:
linux-2.6.28-rc3-test/arch/x86/oprofile/op_model_ppro.c
===================================================================arch/x86/oprofile/op_model_ppro.c +4
-5@@ -27,8 +27,7 @@static int counter_width =32 ;#define CTR_IS_RESERVED (msrs, c) (msrs->counters[(c)].addr ? 1 : 0) -#define CTR_READ (l, h, msrs, c) do {rdmsr(msrs->counters[(c)].addr, (l), (h)); } while (0) -#define CTR_OVERFLOWED (n) (!((n) & (1U<<(counter_width-1)))) +#define CTR_OVERFLOWED (n) (!((n) & (1ULL<<(counter_width-1))))#define CTRL_IS_RESERVED (msrs, c) (msrs->controls[(c)].addr ? 1 : 0)#define CTRL_READ (l, h, msrs, c) do {rdmsr((msrs->controls[(c)].addr), (l), (h)); } while (0)@@ -124,14 +123,14 @@static int ppro_check_ctrs (struct pt_regs *const regs,struct op_msrs const *const msrs) { -unsigned int low, high; +u64 val;int i;for (i =0 ; i < num_counters; ++i) {if (!reset_value[i])continue ; -CTR_READ (low, high, msrs, i); -if (CTR_OVERFLOWED (low)) { +rdmsrl (msrs->counters[i].addr, val); +if (CTR_OVERFLOWED (val)) {oprofile_add_sample (regs, i);wrmsrl (msrs->counters[i].addr, -reset_value[i]); }
Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Andi Kleen
Can you please test this patch and see if it solves the problem?
Thanks,
-Andi---
oprofile: Fix p6 counter overflow check
Fix the counter overflow check for CPUs with counter width > 32
I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.
Quoted 6 lines
Quoted 2 lines
Signed-off-by Andi Kleen <[email protected]>Thanks Andi
Unfortunatly, there is still a freeze of NMIS after a while.
How long is the while?
I don't see this here on a Merom when I let it run for 10 minutes or so.
You might need to bisect it.
-Andi
Quoted 17 lines · Andi Kleen
Quoted 16 lines · Eric Dumazet
Quoted 15 lines · Andi Kleen
Can you please test this patch and see if it solves the problem?
Thanks,
-Andi---
oprofile: Fix p6 counter overflow check
Fix the counter overflow check for CPUs with counter width > 32
I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Signed-off-by Andi Kleen <[email protected]>Thanks Andi
Unfortunatly, there is still a freeze of NMIS after a while.
How long is the while?
Less than 20 seconds if fully loaded with "tbench 8" on my 8 cpus machine.
With your patch, many events are handled by oprofile (about 10.000.000 last time I tried) before freeze.
Without your patch, freeze was very very fast, no more than 10.000 NMI...
Quoted 4 lines
I don't see this here on a Merom when I let it run for 10 minutes or so.
You might need to bisect it.
Too many bugs during bisection unfortunatly, machine doesnt boot :(
Quoted 20 lines · Eric Dumazet
Quoted 19 lines · Andi Kleen
Quoted 18 lines · Eric Dumazet
Quoted 17 lines · Andi Kleen
Can you please test this patch and see if it solves the problem?
Thanks,
-Andi---
oprofile: Fix p6 counter overflow check
Fix the counter overflow check for CPUs with counter width > 32
I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.
Signed-off-by Andi Kleen <[email protected]>
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Thanks Andi
Unfortunatly, there is still a freeze of NMIS after a while.
How long is the while?
Less than 20 seconds if fully loaded with "tbench 8" on my 8 cpus machine.
I don't have tbench on the test box, but when I load it with a couple of while true ; do true ; done loops it also continues sampling.
Are you sure it worked in 2.6.27?
-Andi
Quoted 24 lines · Andi Kleen
Quoted 23 lines · Eric Dumazet
Quoted 22 lines · Andi Kleen
Quoted 21 lines · Eric Dumazet
Quoted 9 lines · Andi Kleen
Can you please test this patch and see if it solves the problem?
Thanks,
-Andi---
oprofile: Fix p6 counter overflow check
Quoted 8 lines
Fix the counter overflow check for CPUs with counter width > 32
I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.
Signed-off-by Andi Kleen <[email protected]>Thanks Andi
Unfortunatly, there is still a freeze of NMIS after a while.
Quoted 8 lines
Quoted 3 lines
Quoted 2 lines
How long is the while?
Less than 20 seconds if fully loaded with "tbench 8" on my 8 cpus machine.
I don't have tbench on the test box, but when I load it with a couple of while true ; do true ; done loops it also continues sampling.
Are you sure it worked in 2.6.27?
You are right... I got same problem with 2.6.27.4, it seems...
$ grep NMI /proc/interrupts NMI: 423689 79703 1576236 580137 1405669 1187909 2923037 5163299 Non-maskable interrupts $ grep NMI /proc/interrupts NMI: 423689 79703 1576236 580137 1405669 1187909 2923037 5163299 Non-maskable interrupts
I'll try to find the faulty commit.
Thank you
Quoted 10 lines · Andi Kleen
Can you please test this patch and see if it solves the problem?
Thanks,
-Andi---
oprofile: Fix p6 counter overflow check
Quoted 7 lines
Fix the counter overflow check for CPUs with counter width > 32
I had a similar change in a different patch that I didn't submit and I didn't notice the problem earlier because it was always tested together.
Signed-off-by Andi Kleen <[email protected]>
Andi,
your patch fixes the initial bug that stops sampling immediately after starting. The issue Eric mentions seems to be a different problem. I am also not able to reproduce this. Your patch on top of rc3 continues sampling at least for several minutes.
I will apply your fix.
Thanks Andi,
-Robert
Signature · 3 lines
Advanced Micro Devices, Inc. Operating System Research Center email: [email protected]
Quoted 7 lines · Pekka J Enberg
Hi Robert,
Quoted 2 lines · Robert Richter
your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.
Unfortunately I don't have my machine with me right now so I can't test this, but don't we want something like this? Andi?
The patch is not the correct fix to whatever the problem is. arch perfmon is not Nehalem specific.
-Andi
Quoted 4 lines · Robert Richter
Andi,
your patchset breaks oprofile on Intel Core 2 cpus. I could bisect the bug for c493756..5951290. Please take a look at this and send me a fix.
I'll take a look but it'll be some time because my test setup is still not fully functional after moving.
-Andi
Hi Eric (and related people touching the code),
Just to confirm your observations (as you have a ? in subj);
oprofile is also broken for me on kernel 2.6.28-rc2.
(Related people Cc'ed
git log v2.6.27.. drivers/oprofile/ | grep Author: | sort |uniq)
Quoted 2 lines · Eric Dumazet
I tried to start my favorite tool, and discovered it wont fly on my HP BL460C G1
My hardware: HP-proliant DL380-G5
CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz
Quoted 2 lines
It samples few events then, seems to stop (no more NMI), other than that, machine still running correctly.
Same observation.
Quoted 1 line
It is working correctly on 2.6.27
Same observation.
Eric,
Quoted 8 lines · Eric Dumazet
Hi
I tried to start my favorite tool, and discovered it wont fly on my HP BL460C G1
It samples few events then, seems to stop (no more NMI), other than that, machine
still running correctly.
hmm, I can not reproduce this on my system. I am receiving samples and the number is increasing. Andi, could you test this on some of your systems?
-Robert
# opcontrol --dump; opreport
CPU: AMD64 family10, speed 2100 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Cycles outside of halt state) with a unit mask of 0x00 (No unit mask) count 100000
CPU_CLK_UNHALT...|
samples| %|
------------------
70897 98.6970 vmlinux
467 0.6501 libc-2.6.1.so
219 0.3049 bash
141 0.1963 ld-2.6.1.so
43 0.0599 oprofiled
15 0.0209 gawk-3.1.5
15 0.0209 opreport
9 0.0125 libstdc++.so.6.0.8
8 0.0111 grep
5 0.0070 sshd
4 0.0056 libcrypto.so.0.9.8
2 0.0028 rm
1 0.0014 cat
1 0.0014 dirname
1 0.0014 ls
1 0.0014 mkdir
1 0.0014 id
1 0.0014 ophelp
1 0.0014 libbfd-2.18.so
1 0.0014 ntpd
# grep NMI /proc/interrupts
NMI: 20357 14272 20151 9967 20140 13961 16291 20826 Non-maskable interrupts
# grep NMI /proc/interrupts
NMI: 20627 14458 20398 10080 20391 14204 16444 21016 Non-maskable interrupts
# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 2
model name : Quad-Core AMD Opteron(tm) Processor 2352
stepping : 3
cpu MHz : 2100.000
cache size : 512 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good pni monitor cx16 lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs
bogomips : 4189.26
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate
# uname -a
Linux paloma 2.6.28-rc2-oprofile-x86_64-standard #1 SMP Sat Nov 1 01:28:08 CET 2008 x86_64 Quad-Core AMD Opteron(tm) Processor 2352 AuthenticAMD GNU/LinuxSignature · 3 lines
Advanced Micro Devices, Inc. Operating System Research Center email: [email protected]