LTT for 2.5.46 1/10: Core definitions

Apparently Linus doesn't see what this patch buys Linux. Since I can't personally convince him otherwise, having written LTT myself, here it is in the hope that others on the list actually find it of some use.

A couple of years back I was attending USENIX where an important kernel developer (who shall remain nameless) was giving a lecture. During the break I showed him a session of LTT. His comment explains, to me at least, much of the attitude I've seen towards this tool from some key kernel developers. Upon seeing the thing, he said: "Well, that's entertaining." For sure many kernel developers don't see the point of this tool because they themselves already understand the kernel's behavior, and consequently, that of much of the rest of the system very well, which is expected. Nonetheless, LTT makes a world of a difference to users, developers and system administrators who's priority is to understand how their own applications' dynamic behavior is influenced by the rest of the system. For these folks, LTT buys much more than a simple distraction.

Parts 1, 2 and 3 of the patch set are the core. The rest are the actual trace statements and can be gradually applied.

Stress test results on UP and SMP available here:
http://marc.theaimsgroup.com/?l=linux-kernel&m=103573710926859&w=2

This set of patches works with version 0.9.6pre2 of the user tools but you will need to edit Daemon/TraceDaemon.c and change __NR_trace to 258 instead of 254.

The complete patch is available from here:
http://opersys.com/ftp/pub/LTT/ExtraPatches/patch-ltt-linux-2.5.46-vanilla-021104-2.2.bz2

D: This patch includes all the definitions (headers/configs/makefiles)
D: required by the trace subsystem.

diffstat:

 MAINTAINERS                 |    7 
 include/asm-generic/trace.h |   86 ++++
 include/linux/trace.h       |  930 ++++++++++++++++++++++++++++++++++++++++++++
 init/Kconfig                |   26 +
 kernel/Makefile             |    3 
 5 files changed, 1051 insertions(+), 1 deletion(-)

diff -urpN linux-2.5.46/MAINTAINERS linux-2.5.46-ltt/MAINTAINERS
@@ -996,6 +996,13 @@ W:	http://linuxppc64.org
 L:	[email protected]
 S:	Supported
 
+LINUX TRACE TOOLKIT
+P:	Karim Yaghmour
+M:	[email protected]
+W:	http://www.opersys.com/LTT
+L:	[email protected]
+S:	Maintained
+
 LOGICAL DISK MANAGER SUPPORT (LDM, Windows 2000/XP Dynamic Disks)
 P:	Richard Russon (FlatCap)
 M:	[email protected]

diff -urpN linux-2.5.46/init/Kconfig linux-2.5.46-ltt/init/Kconfig
@@ -98,6 +98,32 @@ config SYSCTL
 	  building a kernel for install/rescue disks or your system is very
 	  limited in memory.
 
+config TRACE
+	bool "Tracing support"
+	---help---
+	  It is possible for the kernel to log important events to a trace
+	  facility. Doing so, enables the use of the generated traces in order
+	  to reconstruct the dynamic behavior of the kernel, and hence the
+	  whole system.
+
+	  The tracing process contains 4 parts :
+	      1) The logging of events by key parts of the kernel.
+	      2) The tracer that keeps the events in a data buffer.
+	      3) A trace daemon that interacts with the tracer and is
+	         notified every time there is a certain quantity of data to
+	         read from the tracer.
+	      4) A trace event data decoder that reads the accumulated data
+	         and formats it in a human-readable format.
+
+	  If you say Y, the first two components will be built into the kernel.
+	  Critical parts of the kernel will call upon the kernel tracing
+	  function. The data is then recorded by the tracer if a trace daemon
+	  is running in user-space and has issued a "start" command.
+
+	  For more information on kernel tracing, the trace daemon or the event
+	  decoder, please check the following address :
+	       http://www.opersys.com/LTT
+
 endmenu
 
 

diff -urpN linux-2.5.46/kernel/Makefile linux-2.5.46-ltt/kernel/Makefile
@@ -4,7 +4,7 @@
 
 export-objs = signal.o sys.o kmod.o workqueue.o ksyms.o pm.o exec_domain.o \
 		printk.o platform.o suspend.o dma.o module.o cpufreq.o \
-		profile.o rcupdate.o
+		profile.o rcupdate.o trace.o
 
 obj-y     = sched.o fork.o exec_domain.o panic.o printk.o profile.o \
 	    module.o exit.o itimer.o time.o softirq.o resource.o \

@@ -21,6 +21,7 @@ obj-$(CONFIG_PM) += pm.o
 obj-$(CONFIG_CPU_FREQ) += cpufreq.o
 obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
 obj-$(CONFIG_SOFTWARE_SUSPEND) += suspend.o
+obj-$(CONFIG_TRACE) += trace.o
 
 ifneq ($(CONFIG_IA64),y)
 # According to Alan Modra <[email protected]>, the -fno-omit-frame-pointer is

I've had LTT in my own tree for a while now. I appreciate all of the work you've done.

If anyone wants to use it without applying each LTT patch, my tree is at http://members.verizon.net/~vze2j9fk

But beware, this is my own tree that I play with. It has all of kprobes and LTT right now. It usually also has LKCD, but right now it's out because I couldn't do the merge from the last version (2.5.44.) It'll be back as soon an updated LKCD is available.

Also as Karim noted, you'll have to recompile the LTT userland tools to change the number of syscalls. You may also have to upgrade your binutils for the initramfs merge.

Skip

I am quite interested in having LTT as a configurable option in the Linux kernel. My company (the National Center for Atmospheric Research) uses networks of Linux computers to process data from weather radars. Occasionally, we've had unexplained performance problems where the system is slow to respond, even though the process load is low on each computer and no process is "hogging" the CPU. LTT would be extremely valuable to help us diagnose such problems, since we could see the interaction of our processing and radar display tasks with the kernel and the NFS daemons.

Also, we build Linux based data acquisition systems containing signal processing cards. LTT would really help us tune the performance of these systems, since we could see how quickly our processes are scheduled in response to interrupts from the signal processing cards.

Please, include LTT in the Linux kernel. Karim and others have demonstrated that LTT has no impact on kernel performance if it is not configured, and minimal impact even when tracing is enabled. I've used a commercial product (Stethoscope - sold by Real Time Innovations) when doing real-time programming on VxWorks, and found it quite valuable. If LTT is included in the Linux kernel, Linux will be much more appealing to the real-time and embedded programming community.