Skip to contentSkip to conversation

Oops while running systemtap on the p6 machine against the kernel version 2.6.36-rc7-git3

DISCUSSION divya 2 participants 4 messages Oct 14, 2010 – Nov 3, 2010

lore
Share & tools
lore

divya

author
#0 lore raw

While running systemtap tests on the p6 machine , against the kernel version 2.6.36-rc7-git3 Oops occured , here are the call trace

 BUG: spinlock bad magic on CPU#6, stapio/20398
-- 0:conmux-control -- time-stamp -- Oct/13/10  2:49:18 --res
 lock: c000000000fcfa18, .magic: 00000000, .owner:<none>/-1, .owner_cpu: 0
Call Trace:
[c0000001effbfab0] [c000000000011934] .show_stack+0x6c/0x16c (unreliable)
[c0000001effbfb60] [c0000000002c9274] .spin_bug+0xb0/0xd4
[c0000001effbfbf0] [c0000000002c953c] .do_raw_spin_lock+0x48/0x184
[c0000001effbfc90] [c00000000054af78] ._raw_spin_lock+0x10/0x24
[c0000001effbfd00] [d000000003015908] .__stp_time_timer_callback+0x94/0x13c [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798]
[c0000001effbfdc0] [c00000000009c2f8] .run_timer_softirq+0x1d8/0x2a8
[c0000001effbfec0] [c0000000000952d0] .__do_softirq+0xe4/0x1b4
[c0000001effbff90] [c00000000002a7a8] .call_do_softirq+0x14/0x24
[c00000010af1f560] [c00000000000dde4] .do_softirq+0x88/0xf0
[c00000010af1f600] [c000000000095030] .irq_exit+0x50/0xac
[c00000010af1f680] [c000000000027660] .timer_interrupt+0x110/0x13c
[c00000010af1f710] [c000000000003718] decrementer_common+0x118/0x180
--- Exception: 901 at .smp_call_function_many+0x284/0x2d0
    LR = .smp_call_function_many+0x268/0x2d0
[c00000010af1fae0] [c0000000000c55c4] .smp_call_function+0x3c/0x54
[c00000010af1fb60] [c000000000094cdc] .on_each_cpu+0x24/0x84
[c00000010af1fc00] [d000000003016738] ._stp_ctl_write_cmd+0x3b0/0x9c8 [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798]
[c00000010af1fce0] [c0000000001585f4] .vfs_write+0xd0/0x1b8
[c00000010af1fd80] [c0000000001587e4] .SyS_write+0x58/0xa0
[c00000010af1fe30] [c0000000000085b4] syscall_exit+0x0/0x40
------------[ cut here ]------------
kernel BUG at kernel/timer.c:681!
Oops: Exception in kernel mode, sig: 5 [#1]
SMP NR_CPUS=1024 NUMA pSeries
last sysfs file: /sys/module/stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798/sections/__param
Modules linked in: stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798 ipv6 fuse loop dm_mod ibmveth sg sr_mod cdrom sd_mod crc_t10dif ibmvscsic scsi_transport_srp scsi_tgt scsi_mod [last unloaded: stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798]
NIP: c00000000009d090 LR: d00000000301597c CTR: c00000000009cfb0
REGS: c0000001effbf9d0 TRAP: 0700   Not tainted  (2.6.36-rc7-git3-autotest)
MSR: 8000000000029032<EE,ME,CE,IR,DR>   CR: 28000482  XER: 00000002

TASK = c000000103422410[20398] 'stapio' THREAD: c00000010af1c000 CPU: 6
GPR00: 0000000000000001 c0000001effbfc50 c000000000a31660 c000000000fcfa48
GPR04: 000000010000c8da 0000000000000070 0000000000000002 0000000000000000
GPR08: ffffffffffffffff c000000000ac7bf8 0000000000000006 c00000000009cfb0
GPR12: d000000003017090 c00000000f600f00 0000000010008fd8 0000000010008ff8
GPR16: 0000000000000000 c000000000a91180 0000000000000000 0000000000000001
GPR20: c00000010edb9030 c00000010edb9430 c00000010edb9830 c00000010edb9c30
GPR24: 0000000000000001 001167a5b590dc41 c000000000fcfa18 000000010000c8da
GPR28: c000000000fcfa48 d00000000301597c c0000000009a6470 000000010000c8da
NIP [c00000000009d090] .mod_timer+0xe0/0x24c
LR [d00000000301597c] .__stp_time_timer_callback+0x108/0x13c [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798]

Call Trace:

[c0000001effbfc50] [c0000001effbfd00] 0xc0000001effbfd00 (unreliable)
[c0000001effbfd00] [d00000000301597c] .__stp_time_timer_callback+0x108/0x13c [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798] [c0000001effbfdc0] [c00000000009c2f8] .run_timer_softirq+0x1d8/0x2a8
[c0000001effbfec0] [c0000000000952d0] .__do_softirq+0xe4/0x1b4
[c0000001effbff90] [c00000000002a7a8] .call_do_softirq+0x14/0x24
[c00000010af1f560] [c00000000000dde4] .do_softirq+0x88/0xf0
[c00000010af1f600] [c000000000095030] .irq_exit+0x50/0xac
[c00000010af1f680] [c000000000027660] .timer_interrupt+0x110/0x13c
[c00000010af1f710] [c000000000003718] decrementer_common+0x118/0x180
--- Exception: 901 at .smp_call_function_many+0x284/0x2d0

    LR = .smp_call_function_many+0x268/0x2d0

[c00000010af1fae0] [c0000000000c55c4] .smp_call_function+0x3c/0x54
[c00000010af1fb60] [c000000000094cdc] .on_each_cpu+0x24/0x84
[c00000010af1fc00] [d000000003016738] ._stp_ctl_write_cmd+0x3b0/0x9c8 [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798] [c00000010af1fce0] [c0000000001585f4] .vfs_write+0xd0/0x1b8
[c00000010af1fd80] [c0000000001587e4] .SyS_write+0x58/0xa0
[c00000010af1fe30] [c0000000000085b4] syscall_exit+0x0/0x40
Instruction dump:
7ffb4878 f9210070 e93e8078 80090000 2f800000 419e0010 7fa4eb78 7f83e378
4bfffcb1 e81c0020 7c000074 7800d182<0b000000> 7f83e378 38810070 3b400000

Kernel panic - not syncing: Fatal exception in interrupt
Call Trace:
[c0000001effbf5b0] [c000000000011934] .show_stack+0x6c/0x16c (unreliable)
[c0000001effbf660] [c000000000552094] .panic+0x9c/0x204
[c0000001effbf700] [c000000000028974] .die+0x268/0x2ac
[c0000001effbf7a0] [c000000000028ca8] ._exception+0x88/0x174
[c0000001effbf960] [c000000000004f8c] program_check_common+0x10c/0x180
--- Exception: 700 at .mod_timer+0xe0/0x24c
    LR = .__stp_time_timer_callback+0x108/0x13c [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798]

[c0000001effbfc50] [c0000001effbfd00] 0xc0000001effbfd00 (unreliable)
[c0000001effbfd00] [d00000000301597c] .__stp_time_timer_callback+0x108/0x13c [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798] [c0000001effbfdc0] [c00000000009c2f8] .run_timer_softirq+0x1d8/0x2a8
[c0000001effbfec0] [c0000000000952d0] .__do_softirq+0xe4/0x1b4
[c0000001effbff90] [c00000000002a7a8] .call_do_softirq+0x14/0x24
[c00000010af1f560] [c00000000000dde4] .do_softirq+0x88/0xf0
[c00000010af1f600] [c000000000095030] .irq_exit+0x50/0xac
[c00000010af1f680] [c000000000027660] .timer_interrupt+0x110/0x13c
[c00000010af1f710] [c000000000003718] decrementer_common+0x118/0x180
--- Exception: 901 at .smp_call_function_many+0x284/0x2d0

    LR = .smp_call_function_many+0x268/0x2d0

[c00000010af1fae0] [c0000000000c55c4] .smp_call_function+0x3c/0x54
[c00000010af1fb60] [c000000000094cdc] .on_each_cpu+0x24/0x84
[c00000010af1fc00] [d000000003016738] ._stp_ctl_write_cmd+0x3b0/0x9c8 [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798] [c00000010af1fce0] [c0000000001585f4] .vfs_write+0xd0/0x1b8
[c00000010af1fd80] [c0000000001587e4] .SyS_write+0x58/0xa0
[c00000010af1fe30] [c0000000000085b4] syscall_exit+0x0/0x40
Rebooting in 180 seconds..-- 0:conmux-control -- time-stamp -- Oct/13/10 2:51:18 --

Thanks
Divya

#1 lore raw
Quoted 2 lines · divya

While running systemtap tests on the p6 machine , against the kernel version 2.6.36-rc7-git3 Oops occured , here are the call trace

Did the oops happen during a systemtap module startup vs. operation vs. shutdown? stap -V version string?

Quoted 12 lines
 BUG: spinlock bad magic on CPU#6, stapio/20398
-- 0:conmux-control -- time-stamp -- Oct/13/10  2:49:18 --res
 lock: c000000000fcfa18, .magic: 00000000, .owner:<none>/-1, .owner_cpu: 0
Call Trace:
[c0000001effbfab0] [c000000000011934] .show_stack+0x6c/0x16c (unreliable)
[c0000001effbfb60] [c0000000002c9274] .spin_bug+0xb0/0xd4
[c0000001effbfbf0] [c0000000002c953c] .do_raw_spin_lock+0x48/0x184
[c0000001effbfc90] [c00000000054af78] ._raw_spin_lock+0x10/0x24
[c0000001effbfd00] [d000000003015908] .__stp_time_timer_callback+0x94/0x13c [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798]
[...]
kernel BUG at kernel/timer.c:681!
Oops: Exception in kernel mode, sig: 5 [#1]
Quoted 5 lines

SMP NR_CPUS=1024 NUMA pSeries
[...]
[c0000001effbfc50] [c0000001effbfd00] 0xc0000001effbfd00 (unreliable)
[c0000001effbfd00] [d00000000301597c] .__stp_time_timer_callback+0x108/0x13c [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798] [c0000001effbfdc0] [c00000000009c2f8] .run_timer_softirq+0x1d8/0x2a8

We have had occasional problems in the past with something like this:
http://sourceware.org/PR10651, but it never was tracked down to a systemtap bug per se, as opposed to suspicions that the kernel was not satisfying one of its guarantees w.r.t. del_timer_sync().

- FChE

author
#2 lore raw
Quoted 29 lines · Frank Ch. Eigler
Quoted 3 lines · divya

While running systemtap tests on the p6 machine , against the kernel version 2.6.36-rc7-git3 Oops occured , here are the call trace

Did the oops happen during a systemtap module startup vs. operation vs. shutdown? stap -V version string?

Quoted 12 lines
  BUG: spinlock bad magic on CPU#6, stapio/20398
-- 0:conmux-control -- time-stamp -- Oct/13/10  2:49:18 --res
  lock: c000000000fcfa18, .magic: 00000000, .owner:<none>/-1, .owner_cpu: 0
Call Trace:
[c0000001effbfab0] [c000000000011934] .show_stack+0x6c/0x16c (unreliable)
[c0000001effbfb60] [c0000000002c9274] .spin_bug+0xb0/0xd4
[c0000001effbfbf0] [c0000000002c953c] .do_raw_spin_lock+0x48/0x184
[c0000001effbfc90] [c00000000054af78] ._raw_spin_lock+0x10/0x24
[c0000001effbfd00] [d000000003015908] .__stp_time_timer_callback+0x94/0x13c [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798]
[...]
kernel BUG at kernel/timer.c:681!
Oops: Exception in kernel mode, sig: 5 [#1]
Quoted 6 lines

SMP NR_CPUS=1024 NUMA pSeries
[...]
[c0000001effbfc50] [c0000001effbfd00] 0xc0000001effbfd00 (unreliable)
[c0000001effbfd00] [d00000000301597c] .__stp_time_timer_callback+0x108/0x13c [stap_75ce6f84d34f8665c9a6b8e27fb9ea95_818798] [c0000001effbfdc0] [c00000000009c2f8] .run_timer_softirq+0x1d8/0x2a8

Quoted 7 lines

We have had occasional problems in the past with something like this:
http://sourceware.org/PR10651, but it never was tracked down to a systemtap bug per se, as opposed to suspicions that the kernel was not satisfying one of its guarantees w.r.t. del_timer_sync().

- FChE

Sorry Frank for the late reply
The Oops occured during the execution of systemtap tests.

stap version being :
Snapshot: 89e2abb

Attached are the systemtap test execution logs

Thanks
Divya

10/13 15:06:44 DEBUG|     utils:0053| Running 'which gcc'
10/13 15:06:44 DEBUG|     utils:0085| /usr/bin/gcc
10/13 15:06:44 INFO |      test:0256| Test started. Number of iterations: 1
10/13 15:06:44 INFO |      test:0259| Executing iteration 1 of 1
10/13 15:06:44 DEBUG|     utils:0053| Running 'PATH=/usr/local/autobench/autotest/deps/systemtap/systemtap/bin:/usr/local/autobench/autotest/deps/dejagnu/dejagnu/bin:$PATH make installcheck'
10/13 15:06:44 DEBUG|     utils:0085| make  check-DEJAGNU RUNTESTFLAGS=" --tool_opts \'install \'"
10/13 15:06:44 DEBUG|     utils:0085| make[1]: Entering directory `/usr/local/autobench/autotest/tests/systemtap/test'
10/13 15:06:44 DEBUG|     utils:0085| srcdir=`CDPATH="${ZSH_VERSION+.}:" && cd /usr/local/autobench/autotest/tests/systemtap/src/testsuite && pwd`; export srcdir; \
10/13 15:06:44 DEBUG|     utils:0085| 	EXPECT=expect; export EXPECT; \
10/13 15:06:44 DEBUG|     utils:0085| 	runtest="env SYSTEMTAP_TESTAPPS= SYSTEMTAP_RUNTIME=/usr/local/autobench/autotest/deps/systemtap/systemtap/share/systemtap/runtime SYSTEMTAP_TAPSET=/usr/local/autobench/autotest/deps/systemtap/systemtap/share/systemtap/tapset LD_LIBRARY_PATH=/usr/local/autobench/autotest/deps/systemtap/systemtap/lib/systemtap CRASH_LIBDIR=/usr/local/autobench/autotest/deps/systemtap/systemtap/lib/systemtap PATH=/usr/local/autobench/autotest/deps/systemtap/systemtap/bin:$PATH SYSTEMTAP_PATH=/usr/local/autobench/autotest/deps/systemtap/systemtap/bin SYSTEMTAP_INCLUDES=/usr/local/autobench/autotest/deps/systemtap/systemtap/include  PKGLIBDIR=/usr/local/autobench/autotest/deps/systemtap/systemtap/libexec/systemtap /usr/local/autobench/autotest/tests/systemtap/src/testsuite/execrc runtest"; \
10/13 15:06:44 DEBUG|     utils:0085| 	if /bin/sh -c "$runtest --version" > /dev/null 2>&1; then \
10/13 15:06:44 DEBUG|     utils:0085| 	  exit_status=0; l='systemtap'; for tool in $l; do \
10/13 15:06:44 DEBUG|     utils:0085| 	    if $runtest  --tool $tool --tool_opts \'\' --srcdir $srcdir --tool_opts \'install \'; \
10/13 15:06:44 DEBUG|     utils:0085| 	    then :; else exit_status=1; fi; \
10/13 15:06:44 DEBUG|     utils:0085| 	  done; \
10/13 15:06:44 DEBUG|     utils:0085| 	else echo "WARNING: could not find \`runtest'" 1>&2; :;\
10/13 15:06:44 DEBUG|     utils:0085| 	fi; \
10/13 15:06:44 DEBUG|     utils:0085| 	exit $exit_status
10/13 15:06:45 ERROR|     utils:0085| WARNING: Couldn't find the global config file.
10/13 15:06:45 DEBUG|     utils:0085| kernel location: 
10/13 15:06:45 DEBUG|     utils:0085| kernel version: 2.6.36-rc7-git3-autotest
10/13 15:06:45 DEBUG|     utils:0085| systemtap location: /usr/local/autobench/autotest/deps/systemtap/systemtap/bin/stap
10/13 15:06:45 DEBUG|     utils:0085| systemtap version: version 1.4/0.148 non-git sources
10/13 15:06:45 DEBUG|     utils:0085| gcc location: /usr/bin/gcc
10/13 15:06:45 DEBUG|     utils:0085| gcc version: gcc (SUSE Linux) 4.3.2 [gcc-4_3-branch revision 141291]
10/13 15:06:46 DEBUG|     utils:0085| Test Run By root on Wed Oct 13 15:06:46 2010
10/13 15:06:46 DEBUG|     utils:0085| Native configuration is powerpc64-unknown-linux-gnu
10/13 15:06:46 DEBUG|     utils:0085| 
10/13 15:06:46 DEBUG|     utils:0085| 		=== systemtap tests ===
10/13 15:06:46 DEBUG|     utils:0085| 
10/13 15:06:46 DEBUG|     utils:0085| Schedule of variations:
10/13 15:06:46 DEBUG|     utils:0085|     unix
10/13 15:06:46 DEBUG|     utils:0085| 
10/13 15:06:46 DEBUG|     utils:0085| Running target unix
10/13 15:06:46 DEBUG|     utils:0085| Using /usr/local/autobench/autotest/deps/dejagnu/dejagnu/share/dejagnu/baseboards/unix.exp as board description file for target.
10/13 15:06:46 DEBUG|     utils:0085| Using /usr/local/autobench/autotest/deps/dejagnu/dejagnu/share/dejagnu/config/unix.exp as generic interface file for target.
10/13 15:06:46 DEBUG|     utils:0085| Using /usr/local/autobench/autotest/tests/systemtap/src/testsuite/config/unix.exp as tool-and-target-specific interface file.
10/13 15:06:46 DEBUG|     utils:0085| 
10/13 15:06:46 DEBUG|     utils:0085| Host: Linux p55alp2 2.6.36-rc7-git3-autotest #1 SMP Wed Oct 13 14:46:58 IST 2010 ppc64 ppc64 ppc64 GNU/Linux
10/13 15:06:46 DEBUG|     utils:0085| Snapshot: 89e2abb
10/13 15:06:46 DEBUG|     utils:0085| GCC: 4.3.2 [gcc (SUSE Linux) 4.3.2 [gcc-4_3-branch revision 141291]]
10/13 15:06:46 DEBUG|     utils:0085| Distro: SUSE Linux Enterprise Server 11 (ppc64)
10/13 15:06:46 DEBUG|     utils:0085| 
10/13 15:06:46 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap/notest.exp ...
10/13 15:06:46 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.apps/mysql.exp ...
10/13 15:06:46 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.apps/postgres.exp ...
10/13 15:06:46 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.apps/tcl.exp ...
10/13 15:06:46 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.apps/xulrunner.exp ...
10/13 15:06:46 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/add.exp ...
10/13 15:06:58 DEBUG|     utils:0085| FAIL: systemtap.base/add.stp startup (eof)
10/13 15:06:58 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/alias-condition.exp ...
10/13 15:07:06 DEBUG|     utils:0085| FAIL: systemtap.base/alias-condition.stp startup (eof)
10/13 15:07:06 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/alias_tapset.exp ...
10/13 15:07:10 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/alternatives.exp ...
10/13 15:07:13 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/arith.exp ...
10/13 15:07:21 DEBUG|     utils:0085| FAIL: arith
10/13 15:07:21 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/arith_limits.exp ...
10/13 15:07:30 DEBUG|     utils:0085| FAIL: arith_limits
10/13 15:07:30 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/array_size.exp ...
10/13 15:07:43 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/array_string.exp ...
10/13 15:07:49 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/atomic.exp ...
10/13 15:08:32 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/backtrace.exp ...
10/13 15:09:03 DEBUG|     utils:0085| FAIL: backtrace (0 0)
10/13 15:09:27 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/badkprobe.exp ...
10/13 15:10:23 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/be_order.exp ...
10/13 15:10:29 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/beginenderror.exp ...
10/13 15:10:36 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/bench.exp ...
10/13 15:10:38 DEBUG|     utils:0085| FAIL: bench (0)
10/13 15:10:38 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/bitfield.exp ...
10/13 15:10:47 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/bz10078.exp ...
10/13 15:10:48 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/bz5274.exp ...
10/13 15:10:48 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/bz6850.exp ...
10/13 15:10:49 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/cache.exp ...
10/13 15:12:14 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/cast.exp ...
10/13 15:12:23 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/cmd_parse.exp ...
10/13 15:12:40 DEBUG|     utils:0085| FAIL: cmd_parse9 eof
10/13 15:12:42 DEBUG|     utils:0085| FAIL: cmd_parse10 eof
10/13 15:12:44 DEBUG|     utils:0085| FAIL: cmd_parse11 eof
10/13 15:12:46 DEBUG|     utils:0085| FAIL: cmd_parse12 eof
10/13 15:13:05 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/const_value.exp ...
10/13 15:13:06 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/control_limits.exp ...
10/13 15:13:30 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/crash.exp ...
10/13 15:13:30 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/ctime.exp ...
10/13 15:13:36 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/cu-decl.exp ...
10/13 15:13:36 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/cxxclass.exp ...
10/13 15:13:37 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/debugpath.exp ...
10/13 15:13:40 DEBUG|     utils:0085| FAIL: debugpath-good (eof)
10/13 15:13:40 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/deref.exp ...
10/13 15:13:47 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/deref2.exp ...
10/13 15:13:47 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/div0.exp ...
10/13 15:13:49 DEBUG|     utils:0085| FAIL: systemtap.base/div0.stp startup (eof)
10/13 15:13:49 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/dtrace.exp ...
10/13 15:13:51 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/equal.exp ...
10/13 15:13:56 DEBUG|     utils:0085| FAIL: systemtap.base/equal.stp startup (eof)
10/13 15:13:56 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/error_fn.exp ...
10/13 15:14:02 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/externalvar.exp ...
10/13 15:14:02 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/finloop2.exp ...
10/13 15:14:08 DEBUG|     utils:0085| FAIL: systemtap.base/finloop2.stp startup (eof)
10/13 15:14:08 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/flightrec1.exp ...
10/13 15:14:13 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/flightrec2.exp ...
10/13 15:14:26 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/flightrec3.exp ...
10/13 15:14:45 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/flightrec4.exp ...
10/13 15:14:54 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/flightrec5.exp ...
10/13 15:15:09 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/foreach_value.exp ...
10/13 15:15:16 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/func_alias.exp ...
10/13 15:15:18 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/global_end.exp ...
10/13 15:15:34 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/global_init.exp ...
10/13 15:15:36 DEBUG|     utils:0085| FAIL: systemtap.base/global_init.stp startup (eof)
10/13 15:15:36 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/global_opt.exp ...
10/13 15:15:38 DEBUG|     utils:0085| FAIL: global_opt: unexpected EOF
10/13 15:15:38 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/global_stat.exp ...
10/13 15:15:44 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/global_vars.exp ...
10/13 15:15:46 DEBUG|     utils:0085| FAIL: systemtap.base/global_vars.stp startup (eof)
10/13 15:15:46 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/gtod.exp ...
10/13 15:16:06 DEBUG|     utils:0085| FAIL: gtod (0)
10/13 15:16:06 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/if.exp ...
10/13 15:16:11 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/inc.exp ...
10/13 15:16:13 DEBUG|     utils:0085| FAIL: systemtap.base/inc.stp startup (eof)
10/13 15:16:13 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/inlinedvars.exp ...
10/13 15:16:15 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/itrace.exp ...
10/13 15:16:16 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/kfunct.exp ...
10/13 15:16:21 DEBUG|     utils:0085| FAIL: systemtap.base/kfunct.stp startup (eof)
10/13 15:16:21 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/kmodule.exp ...
10/13 15:16:25 DEBUG|     utils:0085| FAIL: systemtap.base/kmodule.stp startup (eof)
10/13 15:16:25 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/kprobes.exp ...
10/13 15:16:31 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/kretprobe-vars.exp ...
10/13 15:16:37 DEBUG|     utils:0085| FAIL: systemtap.base/kretprobe-vars.stp startup (eof)
10/13 15:16:37 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/labels.exp ...
10/13 15:16:37 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/limits.exp ...
10/13 15:16:43 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/logical_and.exp ...
10/13 15:16:44 DEBUG|     utils:0085| FAIL: systemtap.base/logical_and.stp startup (eof)
10/13 15:16:44 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/marker.exp ...
10/13 15:16:45 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/maxactive.exp ...
10/13 15:16:50 DEBUG|     utils:0085| FAIL: MAXACTIVE01 startup (eof)
10/13 15:16:55 DEBUG|     utils:0085| FAIL: MAXACTIVE02 startup (eof)
10/13 15:16:55 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/maxmemory.exp ...
10/13 15:17:14 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/modargs.exp ...
10/13 15:17:16 DEBUG|     utils:0085| FAIL: modargs compilation: /tmp/stapbqFK0D/stap_b3c6d56a1ee2d88b4eee11c3a17a03c8_1022.c:594: error: 'param_ops_int64_t' undeclared here (not in a function)
10/13 15:17:16 DEBUG|     utils:0085| make[3]: *** [/tmp/stapbqFK0D/stap_b3c6d56a1ee2d88b4eee11c3a17a03c8_1022.o] Error 1
10/13 15:17:16 DEBUG|     utils:0085| make[2]: *** [_module_/tmp/stapbqFK0D] Error 2
10/13 15:17:16 DEBUG|     utils:0085| Pass 4: compilation failed.  Try again with another '--vp 0001' option.
10/13 15:17:16 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/not.exp ...
10/13 15:17:18 DEBUG|     utils:0085| FAIL: systemtap.base/not.stp startup (eof)
10/13 15:17:18 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/onoffprobe.exp ...
10/13 15:17:24 DEBUG|     utils:0085| FAIL: conditional probes (0)
10/13 15:17:24 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/optim.exp ...
10/13 15:17:30 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/optim_arridx.exp ...
10/13 15:17:30 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/optim_voidstmt.exp ...
10/13 15:17:36 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/optionalprobe.exp ...
10/13 15:17:40 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/overcatcher.exp ...
10/13 15:17:46 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/overflow_error.exp ...
10/13 15:17:48 DEBUG|     utils:0085| FAIL: overflow_error (0)
10/13 15:17:48 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/overload.exp ...
10/13 15:18:09 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/pointer_array.exp ...
10/13 15:18:20 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/poll_map.exp ...
10/13 15:18:28 DEBUG|     utils:0085| FAIL: poll_map (0)
10/13 15:18:28 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/pp.exp ...
10/13 15:18:50 DEBUG|     utils:0085| Running /usr/local/autobench/autotest/tests/systemtap/src/testsuite/systemtap.base/pr10854.exp ...
#3 lore raw
In reply to divya
Quoted 7 lines · divya

While running systemtap tests on the p6 machine , against the kernel version 2.6.36-rc7-git3 Oops occured , here are the call trace

 BUG: spinlock bad magic on CPU#6, stapio/20398

-- 0:conmux-control -- time-stamp -- Oct/13/10 2:49:18 --res
lock: c000000000fcfa18, .magic: 00000000, .owner:<none>/-1, .owner_cpu: 0
[...]

jistone committed some timing-related changes last night. Would you mind trying a new build on your ppc box?

It seems as through there is a concurrency problem with the timer shutdown process. If the problem still occurs, we might need to instrument that part of the kernel and/or runtime code to figure it out.

- FChE