Skip to contentSkip to conversation

tracing fixes

GIT PULL Ingo Molnar 2 participants 6 messages Jul 17, 2008 – Jul 19, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Linus,

Please pull the latest tracing fixes git tree from:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git tracing-fixes-for-linus

Thanks,

Ingo

------------------>
Ingo Molnar (4):
ftrace: fix merge buglet
ftrace: fix lockup with MAXSMP
ftrace: do not trace scheduler functions
ftrace: do not trace library functions

 arch/x86/kernel/Makefile |    3 +--
 kernel/Makefile          |    3 +--
 lib/Makefile             |   14 +++++---------
 3 files changed, 7 insertions(+), 13 deletions(-)
arch/x86/kernel/Makefile +1 -2
index 5112c84..da14061 100644
@@ -8,8 +8,7 @@ CPPFLAGS_vmlinux.lds += -U$(UTS_MACHINE)
 
 ifdef CONFIG_FTRACE
 # Do not profile debug utilities
-CFLAGS_REMOVE_tsc_64.o = -pg
-CFLAGS_REMOVE_tsc_32.o = -pg
+CFLAGS_REMOVE_tsc.o = -pg
 CFLAGS_REMOVE_rtc.o = -pg
 endif
 
kernel/Makefile +1 -2
index 0a7ed83..985ddb7 100644
@@ -11,8 +11,6 @@ obj-y     = sched.o fork.o exec_domain.o panic.o printk.o profile.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o pm_qos_params.o sched_clock.o
 
-CFLAGS_REMOVE_sched.o = -mno-spe
-
 ifdef CONFIG_FTRACE
 # Do not trace debug files and internal ftrace files
 CFLAGS_REMOVE_lockdep.o = -pg
@@ -21,6 +19,7 @@ CFLAGS_REMOVE_mutex-debug.o = -pg
 CFLAGS_REMOVE_rtmutex-debug.o = -pg
 CFLAGS_REMOVE_cgroup-debug.o = -pg
 CFLAGS_REMOVE_sched_clock.o = -pg
+CFLAGS_REMOVE_sched.o = -mno-spe -pg
 endif
 
 obj-$(CONFIG_SYSCTL_SYSCALL_CHECK) += sysctl_check.o
lib/Makefile +5 -9
index 2c62a9c..818c4d4 100644
@@ -2,21 +2,17 @@
 # Makefile for some libs needed in the kernel.
 #
 
+ifdef CONFIG_FTRACE
+ORIG_CFLAGS := $(KBUILD_CFLAGS)
+KBUILD_CFLAGS = $(subst -pg,,$(ORIG_CFLAGS))
+endif
+
 lib-y := ctype.o string.o vsprintf.o cmdline.o \
 	 rbtree.o radix-tree.o dump_stack.o \
 	 idr.o int_sqrt.o extable.o prio_tree.o \
 	 sha1.o irq_regs.o reciprocal_div.o argv_split.o \
 	 proportions.o prio_heap.o ratelimit.o
 
-ifdef CONFIG_FTRACE
-# Do not profile string.o, since it may be used in early boot or vdso
-CFLAGS_REMOVE_string.o = -pg
-# Also do not profile any debug utilities
-CFLAGS_REMOVE_spinlock_debug.o = -pg
-CFLAGS_REMOVE_list_debug.o = -pg
-CFLAGS_REMOVE_debugobjects.o = -pg
-endif
-
 lib-$(CONFIG_MMU) += ioremap.o
 lib-$(CONFIG_SMP) += cpumask.o
 
#1 lore raw
Quoted 7 lines · Ingo Molnar

Ingo Molnar (4):
ftrace: fix merge buglet
ftrace: fix lockup with MAXSMP
ftrace: do not trace scheduler functions
ftrace: do not trace library functions

[...]

Quoted 12 lines
kernel/Makefile +0 -2
@@ -11,8 +11,6 @@ obj-y     = sched.o fork.o exec_domain.o panic.o printk.o profile.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o pm_qos_params.o sched_clock.o
 
-CFLAGS_REMOVE_sched.o = -mno-spe
-
 ifdef CONFIG_FTRACE
 # Do not trace debug files and internal ftrace files
 CFLAGS_REMOVE_lockdep.o = -pg

kernel/Makefile CFLAGS_REMOVE_mutex-debug.o = -pg · line 24

Quoted 6 lines
kernel/Makefile +1 -0
@@ -21,6 +19,7 @@ CFLAGS_REMOVE_mutex-debug.o = -pg
 CFLAGS_REMOVE_rtmutex-debug.o = -pg
 CFLAGS_REMOVE_cgroup-debug.o = -pg
 CFLAGS_REMOVE_sched_clock.o = -pg
+CFLAGS_REMOVE_sched.o = -mno-spe -pg
 endif
 

Ingo,

Why not trace the scheduler functions? I found a lot of useful information from seeing what functions are being called (namely the latencies caused by the fair scheduler balancing). Not being able to trace sched.c seems to keep a lot of useful data from being accessed.

also, is the '-mno-spe' save when ftrace is not configured?

-- Steve

#2 lore raw
Quoted 2 lines · Steven Rostedt

also, is the '-mno-spe' save when ftrace is not configured?

s/save/safe/

-- Steve

#3 lore raw
In reply to Steven Rostedt
Quoted 31 lines · Steven Rostedt
Quoted 7 lines · Ingo Molnar

Ingo Molnar (4):
ftrace: fix merge buglet
ftrace: fix lockup with MAXSMP
ftrace: do not trace scheduler functions
ftrace: do not trace library functions

[...]

Quoted 12 lines
kernel/Makefile +0 -2
@@ -11,8 +11,6 @@ obj-y     = sched.o fork.o exec_domain.o panic.o printk.o profile.o \
 	    hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \
 	    notifier.o ksysfs.o pm_qos_params.o sched_clock.o
 
-CFLAGS_REMOVE_sched.o = -mno-spe
-
 ifdef CONFIG_FTRACE
 # Do not trace debug files and internal ftrace files
 CFLAGS_REMOVE_lockdep.o = -pg

kernel/Makefile CFLAGS_REMOVE_mutex-debug.o = -pg · line 24

Quoted 6 lines
kernel/Makefile +1 -0
@@ -21,6 +19,7 @@ CFLAGS_REMOVE_mutex-debug.o = -pg
 CFLAGS_REMOVE_rtmutex-debug.o = -pg
 CFLAGS_REMOVE_cgroup-debug.o = -pg
 CFLAGS_REMOVE_sched_clock.o = -pg
+CFLAGS_REMOVE_sched.o = -mno-spe -pg
 endif
 

Ingo,

Quoted 4 lines

Why not trace the scheduler functions? I found a lot of useful information from seeing what functions are being called (namely the latencies caused by the fair scheduler balancing). Not being able to trace sched.c seems to keep a lot of useful data from being accessed.

i agree in general, but it was causing lockups with:

      http://redhat.com/~mingo/misc/config-Thu_Jul_17_13_34_52_CEST_2008

note the MAXSMP in the config which sets NR_CPUS to 4096:

      CONFIG_NR_CPUS=4096

our randconfig testing stumbled on it. That is a debug helper to "tune up the kernel for as large systems as possible" and can bring in regressions not normally seen.

after i spent a good 4 hours on figuring out the lib/*.o details i didnt have the stamina to find the exact reason within sched.o :-)

One thing that needs looking at is that ftrace's self-recursion checks are not as robust as they used to be, and this is a recent regression (as in: last 1-2 weeks). Why do we have to exclude tsc.o from tracing for example? Why isnt cpu_clock() called inside a recursion-protected section? Why are all the trace function callbacks called outside of recursion checks? Why arent ftrace lockups debuggable via the NMI watchdog + early printk? I think it would be more robust to do a recursion check ASAP.

Quoted 1 line

also, is the '-mno-spe' safe when ftrace is not configured?

Why was the -mno-spe added exactly? I havent seen it explained in the commit that added its removal:

| commit 6ec562328fda585be2d7f472cfac99d3b44d362a
| Author: Steven Rostedt <[email protected]>
| Date:   Wed May 14 21:30:30 2008 -0400
|
|    ftrace: use the new kbuild CFLAGS_REMOVE for kernel directory

it talks about a cleanup but also adds -mno-spe removal that wasnt there before. This seems to be a powerpc special and the exact context is not clear to me.

Ingo

#4 lore raw
Quoted 16 lines · Ingo Molnar
Quoted 11 lines
Quoted 4 lines

CFLAGS_REMOVE_sched_clock.o = -pg
+CFLAGS_REMOVE_sched.o = -mno-spe -pg
endif

Ingo,

Why not trace the scheduler functions? I found a lot of useful information from seeing what functions are being called (namely the latencies caused by the fair scheduler balancing). Not being able to trace sched.c seems to keep a lot of useful data from being accessed.

i agree in general, but it was causing lockups with:

      http://redhat.com/~mingo/misc/config-Thu_Jul_17_13_34_52_CEST_2008
Quoted 7 lines

note the MAXSMP in the config which sets NR_CPUS to 4096:

      CONFIG_NR_CPUS=4096

our randconfig testing stumbled on it. That is a debug helper to "tune up the kernel for as large systems as possible" and can bring in regressions not normally seen.

ok, figured it out today: the lockups were due to the NMI watchdog and a missing NMI protection in cpu_clock(). I've reactivated the topic that solves this problem area and it all works fine now.

the sched.o change probably made a difference just because it reduced the cross section between the NMI watchdog and the scheduler, making lockups less likely during the ftrace self-test. I'll revert it once the tracing/nmisafe is upstream.

Ingo

#5 lore raw
Quoted 8 lines · Ingo Molnar

ok, figured it out today: the lockups were due to the NMI watchdog and a missing NMI protection in cpu_clock(). I've reactivated the topic that solves this problem area and it all works fine now.

the sched.o change probably made a difference just because it reduced the cross section between the NMI watchdog and the scheduler, making lockups less likely during the ftrace self-test. I'll revert it once the tracing/nmisafe is upstream.

Ingo,

Thanks a lot for looking into this.

-- Steve