tracing fixes
Subsystems (2)
Linus,
Please pull the latest tracing fixes git tree from:
git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git tracing-fixes-for-linus
Thanks,
Ingo
------------------>
Ingo Molnar (4):
ftrace: fix merge buglet
ftrace: fix lockup with MAXSMP
ftrace: do not trace scheduler functions
ftrace: do not trace library functions
arch/x86/kernel/Makefile | 3 +-- kernel/Makefile | 3 +-- lib/Makefile | 14 +++++--------- 3 files changed, 7 insertions(+), 13 deletions(-)
arch/x86/kernel/Makefile +1 -2
@@ -8,8 +8,7 @@ CPPFLAGS_vmlinux.lds += -U$(UTS_MACHINE) ifdef CONFIG_FTRACE # Do not profile debug utilities -CFLAGS_REMOVE_tsc_64.o = -pg -CFLAGS_REMOVE_tsc_32.o = -pg +CFLAGS_REMOVE_tsc.o = -pg CFLAGS_REMOVE_rtc.o = -pg endif
kernel/Makefile +1 -2
@@ -11,8 +11,6 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o profile.o \ hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o -CFLAGS_REMOVE_sched.o = -mno-spe - ifdef CONFIG_FTRACE # Do not trace debug files and internal ftrace files CFLAGS_REMOVE_lockdep.o = -pg
@@ -21,6 +19,7 @@ CFLAGS_REMOVE_mutex-debug.o = -pg CFLAGS_REMOVE_rtmutex-debug.o = -pg CFLAGS_REMOVE_cgroup-debug.o = -pg CFLAGS_REMOVE_sched_clock.o = -pg +CFLAGS_REMOVE_sched.o = -mno-spe -pg endif obj-$(CONFIG_SYSCTL_SYSCALL_CHECK) += sysctl_check.o
lib/Makefile +5 -9
@@ -2,21 +2,17 @@ # Makefile for some libs needed in the kernel. # +ifdef CONFIG_FTRACE +ORIG_CFLAGS := $(KBUILD_CFLAGS) +KBUILD_CFLAGS = $(subst -pg,,$(ORIG_CFLAGS)) +endif + lib-y := ctype.o string.o vsprintf.o cmdline.o \ rbtree.o radix-tree.o dump_stack.o \ idr.o int_sqrt.o extable.o prio_tree.o \ sha1.o irq_regs.o reciprocal_div.o argv_split.o \ proportions.o prio_heap.o ratelimit.o -ifdef CONFIG_FTRACE -# Do not profile string.o, since it may be used in early boot or vdso -CFLAGS_REMOVE_string.o = -pg -# Also do not profile any debug utilities -CFLAGS_REMOVE_spinlock_debug.o = -pg -CFLAGS_REMOVE_list_debug.o = -pg -CFLAGS_REMOVE_debugobjects.o = -pg -endif - lib-$(CONFIG_MMU) += ioremap.o lib-$(CONFIG_SMP) += cpumask.o
Quoted 7 lines · Ingo Molnar
Ingo Molnar (4):
ftrace: fix merge buglet
ftrace: fix lockup with MAXSMP
ftrace: do not trace scheduler functions
ftrace: do not trace library functions
[...]
Quoted 12 lines
kernel/Makefile +0
-2@@ -11,8 +11,6 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o profile.o \ hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o -CFLAGS_REMOVE_sched.o = -mno-spe - ifdef CONFIG_FTRACE # Do not trace debug files and internal ftrace files CFLAGS_REMOVE_lockdep.o = -pg
kernel/Makefile CFLAGS_REMOVE_mutex-debug.o = -pg · line 24
Quoted 6 lines
kernel/Makefile +1
-0@@ -21,6 +19,7 @@ CFLAGS_REMOVE_mutex-debug.o = -pg CFLAGS_REMOVE_rtmutex-debug.o = -pg CFLAGS_REMOVE_cgroup-debug.o = -pg CFLAGS_REMOVE_sched_clock.o = -pg +CFLAGS_REMOVE_sched.o = -mno-spe -pg endif
Ingo,
Why not trace the scheduler functions? I found a lot of useful information from seeing what functions are being called (namely the latencies caused by the fair scheduler balancing). Not being able to trace sched.c seems to keep a lot of useful data from being accessed.
also, is the '-mno-spe' save when ftrace is not configured?
-- Steve
Quoted 2 lines · Steven Rostedt
also, is the '-mno-spe' save when ftrace is not configured?
s/save/safe/
-- Steve
Quoted 31 lines · Steven Rostedt
Quoted 7 lines · Ingo Molnar
Ingo Molnar (4):
ftrace: fix merge buglet
ftrace: fix lockup with MAXSMP
ftrace: do not trace scheduler functions
ftrace: do not trace library functions[...]
Quoted 12 lines
kernel/Makefile +0
-2@@ -11,8 +11,6 @@ obj-y = sched.o fork.o exec_domain.o panic.o printk.o profile.o \ hrtimer.o rwsem.o nsproxy.o srcu.o semaphore.o \ notifier.o ksysfs.o pm_qos_params.o sched_clock.o -CFLAGS_REMOVE_sched.o = -mno-spe - ifdef CONFIG_FTRACE # Do not trace debug files and internal ftrace files CFLAGS_REMOVE_lockdep.o = -pgkernel/Makefile CFLAGS_REMOVE_mutex-debug.o = -pg · line 24
Quoted 6 lines
kernel/Makefile +1
-0@@ -21,6 +19,7 @@ CFLAGS_REMOVE_mutex-debug.o = -pg CFLAGS_REMOVE_rtmutex-debug.o = -pg CFLAGS_REMOVE_cgroup-debug.o = -pg CFLAGS_REMOVE_sched_clock.o = -pg +CFLAGS_REMOVE_sched.o = -mno-spe -pg endifIngo,
Quoted 4 lines
Why not trace the scheduler functions? I found a lot of useful information from seeing what functions are being called (namely the latencies caused by the fair scheduler balancing). Not being able to trace
sched.cseems to keep a lot of useful data from being accessed.
i agree in general, but it was causing lockups with:
http://redhat.com/~mingo/misc/config-Thu_Jul_17_13_34_52_CEST_2008
note the MAXSMP in the config which sets NR_CPUS to 4096:
CONFIG_NR_CPUS=4096
our randconfig testing stumbled on it. That is a debug helper to "tune up the kernel for as large systems as possible" and can bring in regressions not normally seen.
after i spent a good 4 hours on figuring out the lib/*.o details i didnt have the stamina to find the exact reason within sched.o :-)
One thing that needs looking at is that ftrace's self-recursion checks are not as robust as they used to be, and this is a recent regression (as in: last 1-2 weeks). Why do we have to exclude tsc.o from tracing for example? Why isnt cpu_clock() called inside a recursion-protected section? Why are all the trace function callbacks called outside of recursion checks? Why arent ftrace lockups debuggable via the NMI watchdog + early printk? I think it would be more robust to do a recursion check ASAP.
Quoted 1 line
also, is the '-mno-spe' safe when ftrace is not configured?
Why was the -mno-spe added exactly? I havent seen it explained in the commit that added its removal:
| commit 6ec562328fda585be2d7f472cfac99d3b44d362a | Author: Steven Rostedt <[email protected]> | Date: Wed May 14 21:30:30 2008 -0400 | | ftrace: use the new kbuild CFLAGS_REMOVE for kernel directory
it talks about a cleanup but also adds -mno-spe removal that wasnt there before. This seems to be a powerpc special and the exact context is not clear to me.
Ingo
Quoted 16 lines · Ingo Molnar
Quoted 11 lines
Quoted 4 lines
CFLAGS_REMOVE_sched_clock.o = -pg
+CFLAGS_REMOVE_sched.o = -mno-spe -pg
endifIngo,
Why not trace the scheduler functions? I found a lot of useful information from seeing what functions are being called (namely the latencies caused by the fair scheduler balancing). Not being able to trace
sched.cseems to keep a lot of useful data from being accessed.i agree in general, but it was causing lockups with:
http://redhat.com/~mingo/misc/config-Thu_Jul_17_13_34_52_CEST_2008
Quoted 7 lines
note the MAXSMP in the config which sets
NR_CPUSto 4096:CONFIG_NR_CPUS=4096our randconfig testing stumbled on it. That is a debug helper to "tune up the kernel for as large systems as possible" and can bring in regressions not normally seen.
ok, figured it out today: the lockups were due to the NMI watchdog and a missing NMI protection in cpu_clock(). I've reactivated the topic that solves this problem area and it all works fine now.
the sched.o change probably made a difference just because it reduced the cross section between the NMI watchdog and the scheduler, making lockups less likely during the ftrace self-test. I'll revert it once the tracing/nmisafe is upstream.
Ingo
Quoted 8 lines · Ingo Molnar
ok, figured it out today: the lockups were due to the NMI watchdog and a missing NMI protection in
cpu_clock(). I've reactivated the topic that solves this problem area and it all works fine now.the sched.o change probably made a difference just because it reduced the cross section between the NMI watchdog and the scheduler, making lockups less likely during the ftrace self-test. I'll revert it once the tracing/nmisafe is upstream.
Ingo,
Thanks a lot for looking into this.
-- Steve