Skip to contentSkip to conversation

utrace

PATCH In review Roland McGrath 15 participants 53 messages Mar 21, 2009 – Mar 31, 2009

Subsystems (5)
lore
Share & tools
lore
#0 lore raw

cover utrace

utrace is a new kernel-side API for kernel modules, intended to make it tractable to work on novel ways to trace and debug user-mode tasks.

These patches apply to the current Linus tree (v2.6.29-rc8-241-g65c2449). The first two should apply fine on the -tip tree as well, and we will be glad to rebase the set to whichever tree. Frank has another version of the ftrace patch (3/3) that works for -tip. The utrace patches don't touch anything unless you set a new kconfig option (still marked EXPERIMENTAL), and so are quite safe in that regard.

utrace cannot be enabled without CONFIG_HAVE_ARCH_TRACEHOOK and the arch details it indicates. If your arch does not have it yet, its maintainers will have to work on that. The details are in the comments in arch/Kconfig.

The first patch makes a small update to one of the tracehook.h interfaces that we needed for utrace. It moves code a little but does not change any of the logic in the existing code.

The second patch adds the utrace kernel API (if CONFIG_UTRACE=y is set). There is no change at all without the config option, and with it there is no effect on anything at all until a kernel module using the utrace API is loaded. There is detailed documentation on the API in DocBook form.

The third patch is an ftrace widget based on utrace, by Frank Eigler. Frank will follow up on any issues about that patch.

Thanks,
Roland

#1 lore raw

PATCH 1/3 signals: tracehook_notify_jctl change

This changes tracehook_notify_jctl() so it's called with the siglock held, and changes its argument and return value definition. These clean-ups make it a better fit for what new tracing hooks need to check.

Tracing needs the siglock here, held from the time TASK_STOPPED was set, to avoid potential SIGCONT races if it wants to allow any blocking in its tracing hooks.

This also folds the finish_stop() function into its caller do_signal_stop(). The function is short, called only once and only unconditionally. It aids readability to fold it in.

---
 include/linux/tracehook.h |   25 ++++++++++------
 kernel/signal.c           |   69 +++++++++++++++++++++++----------------------
 2 files changed, 51 insertions(+), 43 deletions(-)
include/linux/tracehook.h +16 -9
index 6186a78..b622498 100644  
@@ -1,7 +1,7 @@
 /*
  * Tracing hooks
  *
- * Copyright (C) 2008 Red Hat, Inc.  All rights reserved.
+ * Copyright (C) 2008-2009 Red Hat, Inc.  All rights reserved.
  *
  * This copyrighted material is made available to anyone wishing to use,
  * modify, copy, or redistribute it subject to the terms and conditions
@@ -469,22 +469,29 @@ static inline int tracehook_get_signal(s
 
 /**
  * tracehook_notify_jctl - report about job control stop/continue
- * @notify:		nonzero if this is the last thread in the group to stop
+ * @notify:		zero, %CLD_STOPPED or %CLD_CONTINUED
  * @why:		%CLD_STOPPED or %CLD_CONTINUED
  *
  * This is called when we might call do_notify_parent_cldstop().
- * It's called when about to stop for job control; we are already in
- * %TASK_STOPPED state, about to call schedule().  It's also called when
- * a delayed %CLD_STOPPED or %CLD_CONTINUED report is ready to be made.
  *
- * Return nonzero to generate a %SIGCHLD with @why, which is
- * normal if @notify is nonzero.
+ * @notify is zero if we would not ordinarily send a %SIGCHLD,
+ * or is the %CLD_STOPPED or %CLD_CONTINUED .si_code for %SIGCHLD.
  *
- * Called with no locks held.
+ * @why is %CLD_STOPPED when about to stop for job control;
+ * we are already in %TASK_STOPPED state, about to call schedule().
+ * It might also be that we have just exited (check %PF_EXITING),
+ * but need to report that a group-wide stop is complete.
+ *
+ * @why is %CLD_CONTINUED when waking up after job control stop and
+ * ready to make a delayed @notify report.
+ *
+ * Return the %CLD_* value for %SIGCHLD, or zero to generate no signal.
+ *
+ * Called with the siglock held.
  */
 static inline int tracehook_notify_jctl(int notify, int why)
 {
-	return notify || (current->ptrace & PT_PTRACED);
+	return notify ?: (current->ptrace & PT_PTRACED) ? why : 0;
 }
 
 #define DEATH_REAP			-1
kernel/signal.c +35 -34

113 lines not on this page · raw message

#2 lore raw

PATCH 2/3 utrace core

In reply to Roland McGrath

This adds the utrace facility, a new modular interface in the kernel for implementing user thread tracing and debugging. This fits on top of the tracehook_* layer, so the new code is well-isolated.

The new interface is in <linux/utrace.h> and the DocBook utrace book describes it. It allows for multiple separate tracing engines to work in parallel without interfering with each other. Higher-level tracing facilities can be implemented as loadable kernel modules using this layer.

The new facility is made optional under CONFIG_UTRACE. When this is not enabled, no new code is added. It can only be enabled on machines that have all the prerequisites and select CONFIG_HAVE_ARCH_TRACEHOOK.

In this initial version, utrace and ptrace do not play together at all. If ptrace is attached to a thread, the attach calls in the utrace kernel API return -EBUSY. If utrace is attached to a thread, the PTRACE_ATTACH or PTRACE_TRACEME request will return EBUSY to userland. The old ptrace code is otherwise unchanged and nothing using ptrace should be affected by this patch as long as utrace is not used at the same time. In the future we can clean up the ptrace implementation and rework it to use the utrace API.

---
 Documentation/DocBook/Makefile    |    2 +-
 Documentation/DocBook/utrace.tmpl |  571 +++++++++
 fs/proc/array.c                   |    3 +
 include/linux/init_task.h         |    1 +
 include/linux/sched.h             |    6 +
 include/linux/tracehook.h         |   50 +-
 include/linux/utrace.h            |  692 +++++++++++
 include/linux/utrace_struct.h     |   58 +
 init/Kconfig                      |    9 +
 kernel/Makefile                   |    1 +
 kernel/ptrace.c                   |   18 +-
 kernel/utrace.c                   | 2348 +++++++++++++++++++++++++++++++++++++
 12 files changed, 3756 insertions(+), 3 deletions(-)
Documentation/DocBook/Makefile +1 -1
index 1462ed8..f5da1b4 100644  
@@ -9,7 +9,7 @@
 DOCBOOKS := z8530book.xml mcabook.xml device-drivers.xml \
 	    kernel-hacking.xml kernel-locking.xml deviceiobook.xml \
 	    procfs-guide.xml writing_usb_driver.xml networking.xml \
-	    kernel-api.xml filesystems.xml lsm.xml usb.xml kgdb.xml \
+	    kernel-api.xml filesystems.xml lsm.xml usb.xml kgdb.xml utrace.xml \
 	    gadget.xml libata.xml mtdnand.xml librs.xml rapidio.xml \
 	    genericirq.xml s390-drivers.xml uio-howto.xml scsi.xml \
 	    mac80211.xml debugobjects.xml sh.xml regulator.xml
Documentation/DocBook/utrace.tmpl new file +571 -0

571 lines not on this page · raw message

fs/proc/array.c +3 -0
index 7e4877d..0c683ed 100644  
@@ -81,6 +81,7 @@
 #include <linux/seq_file.h>
 #include <linux/pid_namespace.h>
 #include <linux/tracehook.h>
+#include <linux/utrace.h>
 
 #include <asm/pgtable.h>
 #include <asm/processor.h>
@@ -187,6 +188,8 @@ static inline void task_state(struct seq
 		cred->uid, cred->euid, cred->suid, cred->fsuid,
 		cred->gid, cred->egid, cred->sgid, cred->fsgid);
 
+	task_utrace_proc_status(m, p);
+
 	task_lock(p);
 	if (p->files)
 		fdt = files_fdtable(p->files);
include/linux/init_task.h +1 -0
index e752d97..39eebc8 100644  
@@ -181,6 +181,7 @@ extern struct cred init_cred;
 		[PIDTYPE_SID]  = INIT_PID_LINK(PIDTYPE_SID),		\
 	},								\
 	.dirties = INIT_PROP_LOCAL_SINGLE(dirties),			\
+	INIT_UTRACE(tsk)						\
 	INIT_IDS							\
 	INIT_TRACE_IRQFLAGS						\
 	INIT_LOCKDEP							\
include/linux/sched.h +6 -0
index 011db2f..786ef2d 100644  
@@ -59,6 +59,7 @@ struct sched_param {
 #include <linux/errno.h>
 #include <linux/nodemask.h>
 #include <linux/mm_types.h>
+#include <linux/utrace_struct.h>
 
 #include <asm/system.h>
 #include <asm/page.h>
@@ -1287,6 +1288,11 @@ struct task_struct {
 #endif
 	seccomp_t seccomp;
 
+#ifdef CONFIG_UTRACE
+	struct utrace utrace;
+	unsigned long utrace_flags;
+#endif
+
 /* Thread group tracking */
    	u32 parent_exec_id;
    	u32 self_exec_id;
include/linux/tracehook.h +49 -1

173 lines not on this page · raw message

include/linux/utrace.h new file +692 -0

692 lines not on this page · raw message

include/linux/utrace_struct.h new file +58 -0

58 lines not on this page · raw message

init/Kconfig +9 -0
index 6a5c5fe..4b5ab3e 100644  
@@ -1060,6 +1060,15 @@ config STOP_MACHINE
 	help
 	  Need stop_machine() primitive.
 
+menuconfig UTRACE
+	bool "Infrastructure for tracing and debugging user processes"
+	depends on EXPERIMENTAL
+	depends on HAVE_ARCH_TRACEHOOK
+	help
+	  Enable the utrace process tracing interface.  This is an internal
+	  kernel interface exported to kernel modules, to track events in
+	  user threads, extract and change user thread state.
+
 source "block/Kconfig"
 
 config PREEMPT_NOTIFIERS
kernel/Makefile +1 -0
index e4791b3..7bff724 100644  
@@ -68,6 +68,7 @@ obj-$(CONFIG_IKCONFIG) += configs.o
 obj-$(CONFIG_RESOURCE_COUNTERS) += res_counter.o
 obj-$(CONFIG_STOP_MACHINE) += stop_machine.o
 obj-$(CONFIG_KPROBES_SANITY_TEST) += test_kprobes.o
+obj-$(CONFIG_UTRACE) += utrace.o
 obj-$(CONFIG_AUDIT) += audit.o auditfilter.o
 obj-$(CONFIG_AUDITSYSCALL) += auditsc.o
 obj-$(CONFIG_AUDIT_TREE) += audit_tree.o
kernel/ptrace.c +17 -1
index c9cf48b..41e9542 100644  
@@ -16,6 +16,7 @@
 #include <linux/pagemap.h>
 #include <linux/smp_lock.h>
 #include <linux/ptrace.h>
+#include <linux/utrace.h>
 #include <linux/security.h>
 #include <linux/signal.h>
 #include <linux/audit.h>
@@ -172,6 +173,14 @@ bool ptrace_may_access(struct task_struc
 	return (!err ? true : false);
 }
 
+/*
+ * For experimental use of utrace, exclude ptrace on the same task.
+ */
+static inline bool exclude_ptrace(struct task_struct *task)
+{
+	return unlikely(!!task_utrace_flags(task));
+}
+
 int ptrace_attach(struct task_struct *task)
 {
 	int retval;
@@ -210,6 +219,11 @@ repeat:
 		goto repeat;
 	}
 
+	if (exclude_ptrace(task)) {
+		retval = -EBUSY;
+		goto bad;
+	}
+
 	if (!task->mm)
 		goto bad;
 	/* the same process cannot be attached many times */
@@ -515,7 +529,9 @@ int ptrace_traceme(void)
 	 */
 repeat:
 	task_lock(current);
-	if (!(current->ptrace & PT_PTRACED)) {
+	if (exclude_ptrace(current)) {
+		ret = -EBUSY;
+	} else if (!(current->ptrace & PT_PTRACED)) {
 		/*
 		 * See ptrace_attach() comments about the locking here.
 		 */
kernel/utrace.c new file +2348 -0

1,394 lines not on this page · raw message

The rest of this diff is not shown. View the raw message.

Part of the diff is not shown here. View the raw message.

#3 lore raw

Re: [PATCH 2/3] utrace core

Quoted 14 lines · On Fri, 20 Mar 2009 18:41:40 -0700 (PDT) Roland McGrath <[email protected]>…

This adds the utrace facility, a new modular interface in the kernel for implementing user thread tracing and debugging. This fits on top of the tracehook_* layer, so the new code is well-isolated.

The new interface is in <linux/utrace.h> and the DocBook utrace book describes it. It allows for multiple separate tracing engines to work in parallel without interfering with each other. Higher-level tracing facilities can be implemented as loadable kernel modules using this layer.

The new facility is made optional under CONFIG_UTRACE. When this is not enabled, no new code is added. It can only be enabled on machines that have all the prerequisites and select CONFIG_HAVE_ARCH_TRACEHOOK.

Quoted 8 lines

In this initial version, utrace and ptrace do not play together at all. If ptrace is attached to a thread, the attach calls in the utrace kernel API return -EBUSY. If utrace is attached to a thread, the PTRACE_ATTACH or PTRACE_TRACEME request will return EBUSY to userland. The old ptrace code is otherwise unchanged and nothing using ptrace should be affected by this patch as long as utrace is not used at the same time. In the future we can clean up the ptrace implementation and rework it to use the utrace API.

I'd be interested in seeing a bit of discussion regarding the overall value of utrace - it has been quite a while since it floated past.

I assume that redoing ptrace to be a client of utrace _will_ happen, and that this is merely a cleanup exercise with no new user-visible features?

The "prototype utrace-ftrace interface" seems to be more a cool toy rather than a serious new kernel feature (yes?)

If so, what are the new killer utrace clients which would justify all these changes?

Also, is it still the case that RH are shipping utrace? If so, for what reasons and what benefits are users seeing from it?

And I recall that there were real problems wiring up the Feb 2007 version of utrace to the ARM architecture. Have those issues been resolved? Are any problems expected for any architectures?

Thanks.

#4 lore raw

Re: [PATCH 2/3] utrace core

Tracing does not mean only debug. Some tracing facilities can be used for virtualization. For example User-Mode Linux is based on ptrace.

I have a prototype of kernel module for virtualization (kmview) based on utrace. Using kmview (module+VMM) it is possible for a user (not root) to mount a filesystem just for a process (or a hierarchy of processes), or it is possible for some processes to use different networking stacks or virtual devices. It is something like user-mode containers. kmview provides the same features of umview, based on ptrace, in a (very) faster way. (umview is in Debian lenny,squeeze,sid if you want to test it)

*Utrace is really what I wanted* to support kmview (apart from
some minor issues about the support of nested virtualizations).
Other virtualizations now based on ptrace could move part of their implementation at kernel level by utrace and several speedups become possible.
For example kmview is a partial virtual machine monitor: some system calls are forwarded to the kernel, some others virtualized.
When a user mounts a filesystem, all the system calls which use pathnames inside the mountpoint subtree get virtualized while the others are forwarded to the kernel.
With utrace the kmview kernel module handles many system calls at kernel level. I mean, if an "open" system call was sent to the kernel because the path is outside the virtualized part of the file system, all the system calls on the same file descriptors can be forwarded to the kernel without any request to the VMM at user level. This is just one example of speedup, several others are possible.

Other virtualizations like user-mode linux or fakeroot-ng could use utrace to speedup their virtualization, too.

As far as I have seen, systemtap is a wonderful tool for debugging, expecially for kernel debugging but it has not been designed for virtualization. Ptrace provide a standard set of features and all the implementations of VMM must be in userland. Utrace provides the flexibility to split a VMM and move part of it to a kernel module.

Utrace provides a unified interface to kernel modules for tracing/virtualization. kmview can be implemented as a client of utrace or by spreading code around the kernel and like kmview other virtualizations based on ptrace could need to move some of their logic to the kernel to speedup their execution.
These VMMs will use utrace based modules instead of kernel patches.

renzo

Quoted 18 lines · Andrew Morton

I'd be interested in seeing a bit of discussion regarding the overall value of utrace - it has been quite a while since it floated past.

I assume that redoing ptrace to be a client of utrace _will_ happen, and that this is merely a cleanup exercise with no new user-visible features?

The "prototype utrace-ftrace interface" seems to be more a cool toy rather than a serious new kernel feature (yes?)

If so, what are the new killer utrace clients which would justify all these changes?

Also, is it still the case that RH are shipping utrace? If so, for what reasons and what benefits are users seeing from it?

And I recall that there were real problems wiring up the Feb 2007 version of utrace to the ARM architecture. Have those issues been resolved? Are any problems expected for any architectures?

#5 lore raw

Re: [PATCH 2/3] utrace core

Quoted 6 lines · Renzo Davoli

Tracing does not mean only debug. Some tracing facilities can be used for virtualization. For example User-Mode Linux is based on ptrace.

I have a prototype of kernel module for virtualization (kmview) based on utrace. [...]

Hm, i cannot find the source code. Can it be downloaded from somewhere?

Ingo

#6 lore raw

Re: [PATCH 2/3] utrace core

Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Renzo Davoli

Tracing does not mean only debug. Some tracing facilities can be used for virtualization. For example User-Mode Linux is based on ptrace.

Quoted 5 lines
Quoted 2 lines

I have a prototype of kernel module for virtualization (kmview) based on utrace. [...]

Hm, i cannot find the source code. Can it be downloaded from somewhere?

Sure! kmview is not included in our Debian packages yet as it relies on (still) non mainstream features (utrace), but the code is available on our view-os svn repository.

Check out:
svn co https://view-os.svn.sourceforge.net/svnroot/view-os view-os

More specifically to browse the code/specifications:
The kmview device protocol is here:
http://wiki.virtualsquare.org/index.php/KMview_module_interface_specifications
The kernel module itself is here:
http://view-os.svn.sourceforge.net/viewvc/view-os/trunk/kmview-kernel-module/
The VMM userland application share most of the code with umview, the source code for both is here:
http://view-os.svn.sourceforge.net/viewvc/view-os/trunk/xmview-os/xmview/

kmview kernel module (current version) needs the following patches:
utrace
http://www.mail-archive.com/[email protected]/msg00654.html
http://www.mail-archive.com/[email protected]/msg00655.html
I am trying to keep everything up to date, but the whole stuff is evolving in a quite fast way.

Everything has been released under GPLv2.

renzo

#7 lore raw

Re: [PATCH 2/3] utrace core

Quoted 18 lines · Renzo Davoli
Quoted 7 lines · Ingo Molnar
Quoted 4 lines · Renzo Davoli

Tracing does not mean only debug. Some tracing facilities can be used for virtualization. For example User-Mode Linux is based on ptrace.

Quoted 5 lines
Quoted 2 lines

I have a prototype of kernel module for virtualization (kmview) based on utrace. [...]

Hm, i cannot find the source code. Can it be downloaded from somewhere?

Sure! kmview is not included in our Debian packages yet as it relies on (still) non mainstream features (utrace), but the code is available on our view-os svn repository.

Quoted 8 lines

Check out:
svn co https://view-os.svn.sourceforge.net/svnroot/view-os view-os

More specifically to browse the code/specifications:
The kmview device protocol is here:
http://wiki.virtualsquare.org/index.php/KMview_module_interface_specifications
The kernel module itself is here:
http://view-os.svn.sourceforge.net/viewvc/view-os/trunk/kmview-kernel-module/

Looks really interesting.

That's btw. what i see as the biggest value of utrace: it's a comprehesive, all-encompassing framework all around process state events and process state manipulation.

Utrace came from Frysk (generic debugger), but the fact that you were able to build a completely unanticipated usecase and virtualization module on top of it is a very good sign of a robust and complete design. I'm impressed.

Ingo

#8 lore raw

Re: [PATCH 2/3] utrace core

Quoted 3 lines

That's btw. what i see as the biggest value of utrace: it's a comprehesive, all-encompassing framework all around process state events and process state manipulation.

Me too!

And while we're on the btw's, I want to let everyone know that Ingo is the one who came up with the name "utrace". I had only completely dismal ideas for names, and nothing but the philosophy, "For the love of God, anything but [a-z]trace!" So that's one tiny piece of the whole mess that you can't blame on me. (Yes, I do believe I would be killed if we changed it again now.) ;-) ;-) ;-)

Quoted 4 lines

Utrace came from Frysk (generic debugger), but the fact that you were able to build a completely unanticipated usecase and virtualization module on top of it is a very good sign of a robust and complete design. I'm impressed.

Um, thanks, I guess. The antecedents of your statement are not really accurate, but I'll take the consequent as a compliment! :-)

In fact, utrace came from my experience of maintaining the old ptrace code. Nor was this particular use "completely unanticipated".

I was not aware of Renzo or his work before he got in touch about making use of utrace. But my imagined list of vaporware always included "specialized engines for UML or other syscall-interception type things". (e.g. seccomp is trivial with no need for per-arch asm work.) I swear, a third of the people who ever came to me complaining about ptrace being so hard to work with were doing things that to me are all "syscall interception and/or tracking", whether for some security-minded purpose or something more virtualization-like. Surely for many of those cases, it was really the wrong way to solve the problem they were tackling. Seems it's just the next stop after someone talks you out of LD_PRELOAD. But who am I to say? It was quite clear that people really wanted easier ways to experiment with doing this sort of thing.

That said, I certainly have always hoped for completely unanticipated uses. (I will readily admit to succumbing to "Build it and they will come" mentality. I'm sure flames about my deep character flaws, moral turpitude, and dubious lineage will follow. The history of my career will show that I was not striving for the appearance of cogent planning.)

I hatched the essential design of utrace when I'd recently spent a whole lot of time fixing the innards of ptrace and a whole lot of time helping userland implementors of debuggers and the like figure out how to work with ptrace (and hearing their complaints about it). At the same time, the group I'm in (still) was contemplating both the implementation issues of a generic debugger, how to make it tractable to work up to far smarter debuggers, and also the design of what became systemtap.

It was clear to me that this whole space of problems and potential features would be an open-ended area where different approaches would need to be hashed out, and that there would not be one "ptrace killer" feature that would be the right fit for all uses. It has long been clear that the threshold of effort was far too high for people to experiment and innovate in this area. Hence the plan to make a new platform that lowered that threshold at least closer to "pretty easy" from "intractable", staying about as simple as what both brings that threshold down enough and lets unrelated developments in these things coexist well on the system.

Thanks,
Roland

#9 lore raw

Re: [PATCH 2/3] utrace core

In reply to Andrew Morton
Quoted 2 lines

I'd be interested in seeing a bit of discussion regarding the overall value of utrace - it has been quite a while since it floated past.

Me too!

Quoted 2 lines

I assume that redoing ptrace to be a client of utrace _will_ happen, and that this is merely a cleanup exercise with no new user-visible features?

Yes. It's my expectation that Oleg and I will do that clean-up in several small stages, in the not-too-distant future. I think more of that work has to do with making the ptrace data structures clean and sane than with utrace details.

Quoted 2 lines

The "prototype utrace-ftrace interface" seems to be more a cool toy rather than a serious new kernel feature (yes?)

I don't personally have any experience with either Frank's utrace-ftrace widget or with using any ftrace-based things to debug user programs. I would guess it is more of a demonstration than a tool people will be using in the long run.

Quoted 2 lines

If so, what are the new killer utrace clients which would justify all these changes?

I hope I can leave those examples to the people who will write them. utrace will be a failure if it only serves to underlie the things I want to implement or can think up. My intent is to open up this area of new feature generation to the people who are killer hackers, but have been daunted or turned off by the prospect of becoming killer ptrace innards hackers. (Only Oleg seems to have taken to that opportunity, or perhaps he expected to wind up doing it as little as I did.)

Quoted 2 lines

Also, is it still the case that RH are shipping utrace? If so, for what reasons and what benefits are users seeing from it?

Fedora Rawhide has this current code, yes. The people trying to develop new features using utrace certainly like having it there. (There really truly are people who like to build novel kernel modules without compiling their own kernels from scratch.) I won't try to speak for them or their users.

Quoted 3 lines

And I recall that there were real problems wiring up the Feb 2007 version of utrace to the ARM architecture. Have those issues been resolved? Are any problems expected for any architectures?

That was a misimpression. There were never real problems for ARM, only misunderstandings. I'm sure the way I tried to stage the changes at that time contributed to those misunderstandings arising as they did. Since then, all the arch requirements have been distilled into the HAVE_ARCH_TRACEHOOK set that is already merged for several architectures. It is in the hands of each arch maintainer to update their code to meet the HAVE_ARCH_TRACEHOOK requirements (I'm glad to give advice when asked), and there is no porting work that is actually specific to utrace itself. (You just can't turn it on without HAVE_ARCH_TRACEHOOK.) Of course it is never all that unlikely that some bits of the generic code will get some new tweaks brought to light by making it work with a particular arch. To my knowledge, the strangest arch for cleaning up any of this stuff has always been ia64, and sparc second; those arch maintainers have already done the HAVE_ARCH_TRACEHOOK work.

Thanks,
Roland

#10 lore raw

Re: [PATCH 2/3] utrace core

In reply to Andrew Morton
Quoted 28 lines · Andrew Morton
Quoted 14 lines · On Fri, 20 Mar 2009 18:41:40 -0700 (PDT) Roland McGrath <[email protected]>…

This adds the utrace facility, a new modular interface in the kernel for implementing user thread tracing and debugging.  This fits on top of the tracehook_* layer, so the new code is well-isolated.

The new interface is in <linux/utrace.h> and the DocBook utrace book describes it.  It allows for multiple separate tracing engines to work in parallel without interfering with each other.  Higher-level tracing facilities can be implemented as loadable kernel modules using this layer.

The new facility is made optional under CONFIG_UTRACE. When this is not enabled, no new code is added. It can only be enabled on machines that have all the prerequisites and select CONFIG_HAVE_ARCH_TRACEHOOK.

Quoted 8 lines

In this initial version, utrace and ptrace do not play together at all. If ptrace is attached to a thread, the attach calls in the utrace kernel API return -EBUSY.  If utrace is attached to a thread, the PTRACE_ATTACH or PTRACE_TRACEME request will return EBUSY to userland.  The old ptrace code is otherwise unchanged and nothing using ptrace should be affected by this patch as long as utrace is not used at the same time.  In the future we can clean up the ptrace implementation and rework it to use the utrace API.

I'd be interested in seeing a bit of discussion regarding the overall value of utrace - it has been quite a while since it floated past.

Quoted 8 lines

I assume that redoing ptrace to be a client of utrace _will_ happen, and that this is merely a cleanup exercise with no new user-visible features?

The "prototype utrace-ftrace interface" seems to be more a cool toy rather than a serious new kernel feature (yes?)

If so, what are the new killer utrace clients which would justify all these changes?

It looks like utrace could provide a nice way to do low latency tracing of userspace processes via a hardware interface (e.g. JTAG or custom trace hardware). The only way to do that at present is to scatter bits of instrumentation throughout the kernel.

I would like to see utrace merged so I can work on that type of feature.

#11 lore raw

PATCH 3/3 utrace-based ftrace "process" engine, v2

In reply to Roland McGrath

From: Frank Ch. Eigler <[email protected]>

This is v2 of the prototype utrace-ftrace interface. This code is based on Roland McGrath's utrace API, which provides programmatic hooks to the in-tree tracehook layer. This new patch interfaces many of those events to ftrace, as configured by a small number of debugfs controls. Here's the /debugfs/tracing/process_trace_README:

process event tracer mini-HOWTO

1. Select process hierarchy to monitor. Other processes will be completely unaffected. Leave at 0 for system-wide tracing. % echo NNN > process_follow_pid

2. Determine which process event traces are potentially desired. syscall and signal tracing slow down monitored processes. % echo 0 > process_trace_{syscalls,signals,lifecycle}

3. Add any final uid- or taskcomm-based filtering. Non-matching processes will skip trace messages, but will still be slowed. % echo NNN > process_trace_uid_filter # -1: unrestricted % echo ls > process_trace_taskcomm_filter # empty: unrestricted

4. Start tracing.
% echo process > current_tracer

5. Examine trace.
% cat trace

6. Stop tracing.
% echo nop > current_tracer

---
 include/linux/processtrace.h |   41 +++
 kernel/trace/Kconfig         |    9 +
 kernel/trace/Makefile        |    1 +
 kernel/trace/trace.h         |    8 +
 kernel/trace/trace_process.c |  601 ++++++++++++++++++++++++++++++++++++++++++
 5 files changed, 660 insertions(+), 0 deletions(-)
include/linux/processtrace.h new file +41 -0
new file mode 100644
index ...f2b7d94 100644  
@@ -0,0 +1,41 @@
+#ifndef PROCESSTRACE_H
+#define PROCESSTRACE_H
+
+#include <linux/types.h>
+#include <linux/list.h>
+
+struct process_trace_entry {
+	unsigned char opcode;	/* one of _UTRACE_EVENT_* */
+	char comm[TASK_COMM_LEN]; /* XXX: should be in/via trace_entry */
+	union {
+		struct {
+			pid_t child;
+			unsigned long flags;
+		} trace_clone;
+		struct {
+			long code;
+		} trace_exit;
+		struct {
+		} trace_exec;
+		struct {
+			int si_signo;
+			int si_errno;
+			int si_code;
+		} trace_signal;
+		struct {
+			long callno;
+			unsigned long args[6];
+		} trace_syscall_entry;
+		struct {
+			long rc;
+			long error;
+		} trace_syscall_exit;
+	};
+};
+
+/* in kernel/trace/trace_process.c */
+
+extern void enable_process_trace(void);
+extern void disable_process_trace(void);
+
+#endif /* PROCESSTRACE_H */
kernel/trace/Kconfig +9 -0
index 34e707e..8a92d6f 100644  
@@ -150,6 +150,15 @@ config CONTEXT_SWITCH_TRACER
 	  This tracer gets called from the context switch and records
 	  all switching of tasks.
 
+config PROCESS_TRACER
+	bool "Trace process events via utrace"
+	depends on DEBUG_KERNEL
+	select TRACING
+	select UTRACE
+	help
+	  This tracer provides trace records from process events
+	  accessible to utrace: lifecycle, system calls, and signals.
+
 config BOOT_TRACER
 	bool "Trace boot initcalls"
 	depends on DEBUG_KERNEL
kernel/trace/Makefile +1 -0
index 349d5a9..a774db2 100644  
@@ -33,5 +33,6 @@ obj-$(CONFIG_FUNCTION_GRAPH_TRACER) += t
 obj-$(CONFIG_TRACE_BRANCH_PROFILING) += trace_branch.o
 obj-$(CONFIG_HW_BRANCH_TRACER) += trace_hw_branches.o
 obj-$(CONFIG_POWER_TRACER) += trace_power.o
+obj-$(CONFIG_PROCESS_TRACER) += trace_process.o
 
 libftrace-y := ftrace.o
kernel/trace/trace.h +8 -0
index 4d3d381..c4d2e7f 100644  
@@ -7,6 +7,7 @@
 #include <linux/clocksource.h>
 #include <linux/ring_buffer.h>
 #include <linux/mmiotrace.h>
+#include <linux/processtrace.h>
 #include <linux/ftrace.h>
 #include <trace/boot.h>
 
@@ -30,6 +31,7 @@ enum trace_type {
 	TRACE_USER_STACK,
 	TRACE_HW_BRANCHES,
 	TRACE_POWER,
+        TRACE_PROCESS,
 
 	__TRACE_LAST_TYPE
 };
@@ -170,6 +172,11 @@ struct trace_power {
 	struct power_trace	state_data;
 };
 
+struct trace_process {
+	struct trace_entry		ent;
+	struct process_trace_entry	event;
+};
+
 /*
  * trace_flag_type is an enumeration that holds different
  * states when a trace occurs. These are:
@@ -280,6 +287,7 @@ extern void __ftrace_bad_type(void);
 			  TRACE_GRAPH_RET);		\
 		IF_ASSIGN(var, ent, struct hw_branch_entry, TRACE_HW_BRANCHES);\
  		IF_ASSIGN(var, ent, struct trace_power, TRACE_POWER); \
+		IF_ASSIGN(var, ent, struct trace_process, TRACE_PROCESS); \
 		__ftrace_bad_type();					\
 	} while (0)
 
kernel/trace/trace_process.c new file +601 -0

601 lines not on this page · raw message

#12 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 8 lines · Roland McGrath

From: Frank Ch. Eigler <[email protected]>

This is v2 of the prototype utrace-ftrace interface. This code is based on Roland McGrath's utrace API, which provides programmatic hooks to the in-tree tracehook layer. This new patch interfaces many of those events to ftrace, as configured by a small number of debugfs controls. Here's the /debugfs/tracing/process_trace_README:

Please submit changes/enhancements to kernel/trace/* to the tracing tree maintainers (Steve and me) for review, testing and integration.

Please also post patches against the latest tracing tree:

http://people.redhat.com/mingo/tip.git/README

As this patch does not apply:

Applying patch patches/utrace-based-ftrace-process-engine-v2.patch
patching file include/linux/processtrace.h
patching file kernel/trace/Kconfig
Hunk #1 succeeded at 186 with fuzz 2 (offset 36 lines).
patching file kernel/trace/Makefile
Hunk #1 FAILED at 33.
1 out of 1 hunk FAILED -- rejects in file kernel/trace/Makefile
patching file kernel/trace/trace.h
Hunk #1 succeeded at 7 with fuzz 1.
Hunk #2 FAILED at 31.
Hunk #3 succeeded at 215 with fuzz 2 (offset 43 lines).
Hunk #4 FAILED at 330.
2 out of 4 hunks FAILED -- rejects in file kernel/trace/trace.h
patching file kernel/trace/trace_process.c
Patch patches/utrace-based-ftrace-process-engine-v2.patch does not apply (enforce with -f)

Thanks,

Ingo

#13 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 12 lines · On Sat, 21 Mar 2009 08:43:01 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 8 lines · Roland McGrath

From: Frank Ch. Eigler <[email protected]>

This is v2 of the prototype utrace-ftrace interface. This code is based on Roland McGrath's utrace API, which provides programmatic hooks to the in-tree tracehook layer. This new patch interfaces many of those events to ftrace, as configured by a small number of debugfs controls. Here's the /debugfs/tracing/process_trace_README:

Quoted 6 lines

Please submit changes/enhancements to kernel/trace/* to the tracing tree maintainers (Steve and me) for review, testing and integration.

Please also post patches against the latest tracing tree:

http://people.redhat.com/mingo/tip.git/README

uhm, this patch depends on the (large) utrace patch, which is not kernel/trace material.

Quoted 17 lines

As this patch does not apply:

Applying patch patches/utrace-based-ftrace-process-engine-v2.patch
patching file include/linux/processtrace.h
patching file kernel/trace/Kconfig
Hunk #1 succeeded at 186 with fuzz 2 (offset 36 lines).
patching file kernel/trace/Makefile
Hunk #1 FAILED at 33.
1 out of 1 hunk FAILED -- rejects in file kernel/trace/Makefile
patching file kernel/trace/trace.h
Hunk #1 succeeded at 7 with fuzz 1.
Hunk #2 FAILED at 31.
Hunk #3 succeeded at 215 with fuzz 2 (offset 43 lines).
Hunk #4 FAILED at 330.
2 out of 4 hunks FAILED -- rejects in file kernel/trace/trace.h
patching file kernel/trace/trace_process.c
Patch patches/utrace-based-ftrace-process-engine-v2.patch does not apply (enforce with -f)

The rejects are trivial.

#14 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 17 lines · Andrew Morton
Quoted 15 lines · On Sat, 21 Mar 2009 08:43:01 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 8 lines · Roland McGrath

From: Frank Ch. Eigler <[email protected]>

This is v2 of the prototype utrace-ftrace interface. This code is based on Roland McGrath's utrace API, which provides programmatic hooks to the in-tree tracehook layer. This new patch interfaces many of those events to ftrace, as configured by a small number of debugfs controls. Here's the /debugfs/tracing/process_trace_README:

Please submit changes/enhancements to kernel/trace/* to the tracing tree maintainers (Steve and me) for review, testing and integration.

Quoted 6 lines
Quoted 3 lines

Please also post patches against the latest tracing tree:

http://people.redhat.com/mingo/tip.git/README

uhm, this patch depends on the (large) utrace patch, which is not kernel/trace material.

The thing is, utrace crashes in Fedora have dominated kerneloops.org for many months, so i'm not sure what to make of the idea of posting a 4000+ lines of core kernel code patchset on the last day of the development cycle, a posting that has carefully avoided the Cc:-ing of affected maintainers ;-)

Utrace is very much tracing material - without the ftrace plugin the whole utrace machinery is just something that provides a _ton_ of hooks to something entirely external: SystemTap mainly.

kernel/utrace.c should probably be introduced as kernel/trace/utrace.c not kernel/utrace.c. It also overlaps pending work in the tracing tree and cooperation would be nice and desired.

The ftrace/utrace plugin is the only real connection utrace has to the mainline kernel, so proper review by the tracing folks and cooperation with the tracing folks is very much needed for the whole thing.

Ingo

#15 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 33 lines · On Sat, 21 Mar 2009 10:12:35 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 17 lines · Andrew Morton
Quoted 15 lines · On Sat, 21 Mar 2009 08:43:01 +0100 Ingo Molnar <[email protected]> wrote:
Quoted 8 lines · Roland McGrath

From: Frank Ch. Eigler <[email protected]>

This is v2 of the prototype utrace-ftrace interface. This code is based on Roland McGrath's utrace API, which provides programmatic hooks to the in-tree tracehook layer. This new patch interfaces many of those events to ftrace, as configured by a small number of debugfs controls. Here's the /debugfs/tracing/process_trace_README:

Please submit changes/enhancements to kernel/trace/* to the tracing tree maintainers (Steve and me) for review, testing and integration.

Quoted 6 lines
Quoted 3 lines

Please also post patches against the latest tracing tree:

http://people.redhat.com/mingo/tip.git/README

uhm, this patch depends on the (large) utrace patch, which is not kernel/trace material.

The thing is, utrace crashes in Fedora have dominated kerneloops.org for many months, so i'm not sure what to make of the idea of posting a 4000+ lines of core kernel code patchset on the last day of the development cycle, a posting that has carefully avoided the Cc:-ing of affected maintainers ;-)

Quoted 3 lines

Utrace is very much tracing material - without the ftrace plugin the whole utrace machinery is just something that provides a _ton_ of hooks to something entirely external: SystemTap mainly.

Roland's changelogs don't mention systemtap at all afacit.

That was, umm, major information lossage.

Quoted 8 lines

kernel/utrace.c should probably be introduced as kernel/trace/utrace.c not kernel/utrace.c. It also overlaps pending work in the tracing tree and cooperation would be nice and desired.

The ftrace/utrace plugin is the only real connection utrace has to the mainline kernel, so proper review by the tracing folks and cooperation with the tracing folks is very much needed for the whole thing.

Actually it seems that the whole utrace-ftrace thing is a big distraction and could/should just be omitted. This is a systemtap feature and should be viewed as such.

This is all a bit weird.

#16 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Hi -

Quoted 7 lines · Andrew Morton

[...]

Quoted 3 lines

Utrace is very much tracing material - without the ftrace plugin the whole utrace machinery is just something that provides a _ton_ of hooks to something entirely external: SystemTap mainly.

Roland's changelogs don't mention systemtap at all afacit. That was, umm, major information lossage.

There have been many mixed messages from LKML on the topic - sometimes mentioning systemtap is forbidden, other times necessary. Sorry about that.

There are several non-systemtap clients in existence or under development. You've may have heard of the ptrace cleanup, a multi-client ptrace replacement, an on-the-fly core dumper, the ftrace widget, user-space probes. All of these should have somewhat compelling non-systemtap uses, if that's an important criterion.

Quoted 3 lines

Actually it seems that the whole utrace-ftrace thing is a big distraction and could/should just be omitted. This is a systemtap feature and should be viewed as such. [...]

utrace is a better way to perform user thread management than what is there now, and the utrace-ftrace widget shows how to *hook* thread events such as syscalls in a lighter weight / more managed way than the first one proposed. (That's one reason we've been participating in the ftrace discussions.) Of course it can be made to use the fine syscall pretty-printing code recently added.

- FChE

#17 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 8 lines · On Sat, 21 Mar 2009 07:51:41 -0400 "Frank Ch. Eigler" <[email protected]> wrote:

Hi -

Quoted 5 lines · Andrew Morton

[...]

Quoted 3 lines

Utrace is very much tracing material - without the ftrace plugin the whole utrace machinery is just something that provides a _ton_ of hooks to something entirely external: SystemTap mainly.

Quoted 6 lines
Quoted 2 lines

Roland's changelogs don't mention systemtap at all afacit. That was, umm, major information lossage.

There have been many mixed messages from LKML on the topic - sometimes mentioning systemtap is forbidden, other times necessary. Sorry about that.

heh. We all love systemtap and want it to get better.

Quoted 5 lines

There are several non-systemtap clients in existence or under development. You've may have heard of the ptrace cleanup, a multi-client ptrace replacement, an on-the-fly core dumper, the ftrace widget, user-space probes. All of these should have somewhat compelling non-systemtap uses, if that's an important criterion.

Well I dunno. You guys are closer to this than I am, but I'd have thought that systemtap is the main game here, and most/all of the above is just fluff.

IOW, "this helps systemtap" is sufficient reason for merging a kernel change. For sufficiently large values of "help", and sufficiently small values of "eww", of course.

I have strong memories of being traumatised by reading the uprobes code. What's the story on all of that nowadays?

Quoted 11 lines
Quoted 4 lines

Actually it seems that the whole utrace-ftrace thing is a big distraction and could/should just be omitted. This is a systemtap feature and should be viewed as such. [...]

utrace is a better way to perform user thread management than what is there now, and the utrace-ftrace widget shows how to *hook* thread events such as syscalls in a lighter weight / more managed way than the first one proposed. (That's one reason we've been participating in the ftrace discussions.) Of course it can be made to use the fine syscall pretty-printing code recently added.

eh. Boring. Let's fix systemtap?

#18 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Hi -

Quoted 6 lines · Andrew Morton

[...]

Quoted 3 lines

There have been many mixed messages from LKML on the topic - sometimes mentioning systemtap is forbidden, other times necessary. Sorry about that.

heh. We all love systemtap and want it to get better.

Great!

Quoted 3 lines

[...]
I have strong memories of being traumatised by reading the uprobes code. What's the story on all of that nowadays?

uprobes, being a layer upon utrace that provides a kprobes-like breakpointing API for user threads, is being refactored into several parts. I don't know about the aesthetics of it all, but I believe the general future plan is this:

One piece would perform machine code analysis (to classify instructions for ideal/safe placement of breakpoints or for code patching), and another thin layer that uses this and utrace to manage user-space breakpoints. (Systemtap would interface at this point.) Then a user-space syscallish interface could come along to expose this to a super-ptrace client (to speed up gdb; perhaps to allow multiple debuggers). Plus one might as well add an ftrace-engine for it (directly analogous to the recent kprobe-based one that ftrace people found "cool".)

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

Actually it seems that the whole utrace-ftrace thing is a big distraction and could/should just be omitted. This is a systemtap feature and should be viewed as such. [...]

Quoted 8 lines
Quoted 6 lines

utrace is a better way to perform user thread management than what is there now, and the utrace-ftrace widget shows how to *hook* thread events such as syscalls in a lighter weight / more managed way than the first one proposed. (That's one reason we've been participating in the ftrace discussions.) Of course it can be made to use the fine syscall pretty-printing code recently added.

eh. Boring. Let's fix systemtap?

There are several constituencies here, some of which find the above exciting. That's OK and we'd like to help them too.

- FChE

#19 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Andrew Morton
Quoted 1 line · Andrew Morton

[...] Let's fix systemtap?

Yes, it needs to be fixed.

The main issue i see is that no kernel developer i work with on a daily basis uses SystemTap - and i work with a lot of people. Yes, i could perhaps name two or three people from lkml using it, but its average penetration amongst kernel folks is essentially zero.

Was any critical analysis done why that penetration is so absymally low for a tool with such a promise and with years of availability, and what are the measures planned to address those problems?

To me personally there are two big direct usability issues with SystemTap:

1) It relies on DEBUG_INFO for any reasonable level of utility. Yes, it will limp along otherwise as well, but most of the actual novel capabilities depend on debuginfo. Which is an acceptable constraint for enterprise usage where kernels are switched every few months and having a debuginfo package is not a big issue. Not acceptable for upstream kernel development. It also puts way too trust into the compiler generating 1GB+ of debuginfo correctly. I want to be able to rely on tools all the time and thus i want tools to have some really simple and predictable foundations.

2) It's not upstream and folks using it seem to insist on not having it upstream ;-) This 'distance' to upstream seems to have grown during the past few years - instead of shrinking. As a result it simply does not matter and there's no know-how and no visibility of it upstream.

If these fundamental problems are addressed then i'd even argue for the totality of SystemTap to be aimed upstreamed (including the scripting language, etc.), because for something this fundamental there's just no good reason not to have a turn-key solution there.

Plus then there should be a (steadily growing) library of utility scripts in the kernel proper as well.

Anything less does not make much sense IMO. Having a separate tool will reduce efficiency, increases the latency of fixes and enhancements and creates ABI-like expectations - which are all counter-productive to good instrumentation.

These are the aspects of SystemTap that i have to say were never done right, and these are the aspects of SystemTap that need to change most. Putting utrace upstream now will just make it more convenient to have SystemTap as a separate entity - without any of the benefits. Do we want to do that? Maybe, but we could do better i think.

Ingo

#20 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 4 lines · On Sábado 21 Marzo 2009 16:45:01 Ingo Molnar escribió:

The main issue i see is that no kernel developer i work with on a daily basis uses SystemTap - and i work with a lot of people. Yes, i could perhaps name two or three people from lkml using it, but its average penetration amongst kernel folks is essentially zero.

What about userspace developers? People always talks of systemtap as a kernel thing, but my (humble) impression is that kernel hackers don't seem to need it that much (maybe for the same reasons they didn't a kernel debugger ;), but userspace developers do. There're many userspace projects that offer optional compile options to enable dtrace probes (some people like apple even ship executables of python, perl and ruby with probes by default). There're several firefox hackers that switched to dtrace-capable systems just because the dtrace-javascript probes enabled them to debug javashit code in ways they weren't able in linux or windows.

In my humble opinion a better development environment for linux userspace programmers is way more important than whether kernel hackers like systemtap or not. So maybe the discussion should be less about "does it help kernel hackers?" and more about "does it help userspace hackers?". My 2¢...

#21 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 18 lines · Diego Calleja
Quoted 4 lines · On Sábado 21 Marzo 2009 16:45:01 Ingo Molnar escribió:

The main issue i see is that no kernel developer i work with on a daily basis uses SystemTap - and i work with a lot of people. Yes, i could perhaps name two or three people from lkml using it, but its average penetration amongst kernel folks is essentially zero.

What about userspace developers? People always talks of systemtap as a kernel thing, but my (humble) impression is that kernel hackers don't seem to need it that much (maybe for the same reasons they didn't a kernel debugger ;), but userspace developers do. There're many userspace projects that offer optional compile options to enable dtrace probes (some people like apple even ship executables of python, perl and ruby with probes by default). There're several firefox hackers that switched to dtrace-capable systems just because the dtrace-javascript probes enabled them to debug javashit code in ways they weren't able in linux or windows.

Quoted 5 lines

In my humble opinion a better development environment for linux userspace programmers is way more important than whether kernel hackers like systemtap or not. So maybe the discussion should be less about "does it help kernel hackers?" and more about "does it help userspace hackers?". My 2¢...

Well, i consider kernel development to be just another form of software development, so i dont subscribe to the view that it is intrinsically different. (Yes, the kernel has many unique aspects - but most software projects have unique aspects.)

In terms of development methodology and tools, in fact i claim that the kernel workflow and style of development can be applied to most user-space software projects with great success.

So ... if a new development tool is apparently not (yet?) suited to a very large and sanely developed software project like the Linux kernel, i dont take that as an encouraging sign.

Also, there's practical aspects: the kernel is what we know best so if we can make it work well for the kernel, hopes are that other large projects can use it too. If we _only_ make the tool good for non-kernel purposes, who else will fix it for the kernel? The icentive to fix it for the kernel will be significantly lower.

Ingo

#22 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Ingo Molnar
Quoted 3 lines · On Sat, 21 Mar 2009 16:45:01 +0100 Ingo Molnar <[email protected]> wrote:

[...]

useful, thanks.

Quoted 4 lines

Putting utrace upstream now will just make it more convenient to have SystemTap as a separate entity - without any of the benefits. Do we want to do that? Maybe, but we could do better i think.

It would not be good to merge a large kernel feature which kernel developers and testers cannot test, and regression test.

If testing utrace against its main application requires installation of a complete enterprise distro from a distro which the particular developer might not prefer to use then that's quite a problem.

So it is desirable for this reason (and, I suspect, for other reasons) that systemtap (or a part thereof) be dragged out in some standalone form which is usable by random mortals.

IOW: I agree.

#23 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Hi -

Quoted 3 lines · Andrew Morton

[...]
It would not be good to merge a large kernel feature which kernel developers and testers cannot test, and regression test.

It does not. Other kernel self-sufficient utrace clients are on their way, and of course one was just (re)posted.

Quoted 2 lines

If testing utrace against its main application requires installation of a complete enterprise distro from a distro [...]

This has *never* been a requirement.

- FChE

#24 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 5 lines · Frank Ch. Eigler
Quoted 3 lines

If testing utrace against its main application requires installation of a complete enterprise distro from a distro [...]

This has *never* been a requirement.

You guys are getting off a tangent.

Let's go back to the post that started this all.

Quoted 5 lines

The thing is, utrace crashes in Fedora have dominated kerneloops.org for many months, so i'm not sure what to make of the idea of posting a 4000+ lines of core kernel code patchset on the last day of the development cycle, a posting that has carefully avoided the Cc:-ing of affected maintainers ;-)

.. and dammit, I agree 100%. If utrace really shows up in _any_ way on kerneloops.org, then I think THE ENTIRE DISCUSSION in this thread is moot.

I'm not going to take known-bad crap. It's that simple. Don't bother posting it, don't bother discussing it, don't bother making excuses for it.

Linus

#25 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Hi -

Quoted 6 lines · Linus Torvalds

[...]

Quoted 2 lines

The thing is, utrace crashes in Fedora have dominated kerneloops.org for many months [...]

.. and dammit, I agree 100%. If utrace really shows up in _any_ way on kerneloops.org, then I think THE ENTIRE DISCUSSION in this thread is moot.

There was a short span of time during last fall, when Roland was on vacation. That bug (in 2.6.26.3) was fixed during the kernel summit. So this is a six-month obsolete grievance.

- FChE

#26 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 11 lines · Frank Ch. Eigler
Quoted 6 lines · Linus Torvalds

[...]

Quoted 2 lines

The thing is, utrace crashes in Fedora have dominated kerneloops.org for many months [...]

.. and dammit, I agree 100%. If utrace really shows up in _any_ way on kerneloops.org, then I think THE ENTIRE DISCUSSION in this thread is moot.

There was a short span of time during last fall, when Roland was on vacation. That bug (in 2.6.26.3) was fixed during the kernel summit. So this is a six-month obsolete grievance.

struct task_struct::utrace became embedded struct. This is good and should remove quite a few of utrace bugs. Better late than never.

However, "rewrite-ptrace-via-utrace" patch was omitted, so almost noone can easily see by how much situation improved.

I see this patch was dropped in Fedora.

Will ptrace(2) will be rewritten through utrace?

#27 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Hi -

Quoted 3 lines · Alexey Dobriyan

[...]
struct task_struct::utrace became embedded struct. This is good and should remove quite a few of utrace bugs. Better late than never.

Yeah.

Quoted 3 lines

However, "rewrite-ptrace-via-utrace" patch was omitted, so almost noone can easily see by how much situation improved. [...] Will ptrace(2) will be rewritten through utrace?

Yes, I believe that is Roland's intent. I believe it was separated from the current suite of patches for staging purposes, to merge the most solid code up first. The code is available from the utrace git tree in the utrace-ptrace branch.

- FChE

#28 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 9 lines · Frank Ch. Eigler

Hi -

Quoted 3 lines · Alexey Dobriyan

[...]
struct task_struct::utrace became embedded struct. This is good and should remove quite a few of utrace bugs. Better late than never.

Yeah.

Quoted 8 lines
Quoted 3 lines

However, "rewrite-ptrace-via-utrace" patch was omitted, so almost noone can easily see by how much situation improved. [...] Will ptrace(2) will be rewritten through utrace?

Yes, I believe that is Roland's intent. I believe it was separated from the current suite of patches for staging purposes, to merge the most solid code up first. The code is available from the utrace git tree in the utrace-ptrace branch.

i think they should be submitted together.

Here's the histogram of utrace bugs on kerneloops.org:

  2.6.27.5          1 x
  2.6.27.15         1 x
  2.6.27.12         2 x
  2.6.27-rc4        2 x
  2.6.26.6          1 x
  2.6.26.5         43 x
  2.6.26.3       1102 x
  2.6.26.2          2 x
  2.6.26.1          3 x
  2.6.26            1 x
  2.6.25            3 x

That peak in 2.6.26.3 is what i referred to. The latest F10 kernel rpm is kernel-2.6.27.12-170.2.5.fc10, and it does include the utrace-ptrace engine as well:

  # grep UTRACE /boot/config-2.6.27.19-170.2.35.fc10.i686
  CONFIG_UTRACE=y
  CONFIG_UTRACE_PTRACE=y

So the bug i referred to was fixed and the bug count has gone down - but still we have the utrace core submission here without any (tested) mainline kernel usage of the core code.

My suggestion would be to:

- submit the ptrace-on-utrace engine as well (with Oleg's signoff?)

- perhaps also submit with a well-tested ftrace plugin that tries to utilize _all_ aspects of utrace and ftrace (and hence gives good and continuous burn-in testing via the ftrace bootup self-tests, etc.)

ideally we want both, because:

- tracing corner-case bugs tend to be found much faster than ptrace corner case bugs - partly because tracing is much more invasive when activated system-wide.

- ptrace-over-utrace on the other hand utilizes utrace more deeply than passive tracing ever can. (for example UML does full, active virtualization via ptrace - this depth of functional utrace usage is not possible via a tracing plugin.)

And i think the ptrace-via-utrace engine is actually fully ready, just perhaps it was not submitted out of caution to keep the logistics simple.

So i do think we've still got a shot at merging it, in this merge window.

Ingo

#29 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 3 lines

And i think the ptrace-via-utrace engine is actually fully ready, just perhaps it was not submitted out of caution to keep the logistics simple.

That's not so. There is a clumsy prototype version. Much of the work to do it properly is really just plain ptrace clean-up and not specifically about using utrace. Oleg and I are ready to work on it as soon as our time is not monopolized by trying to get the core utrace code to be accepted.

This ptrace work really buys nothing with immediate pay-off at all. It's a real shame if its lack keeps people from actually looking at utrace itself. (This has been a long conversation so far with zero discussion of the code.) A collaboration with focus on what new things can be built, rather than on reasons not to let the foundations be poured, would be a lovely thing.

Thanks,
Roland

#30 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Frank Ch. Eigler
Quoted 4 lines

Yes, I believe that is Roland's intent. I believe it was separated from the current suite of patches for staging purposes, to merge the most solid code up first. The code is available from the utrace git tree in the utrace-ptrace branch.

More than just "staging". The utrace-ptrace code there today is really not very nice to look at, and it's not ready for prime time. As has been mentioned, it is a "pure clean-up exercise". As such, it's not the top priority. It also didn't seem to me like much of an argument for merging
utrace: "Look, more code and now it still does the same thing!"

Instead, I figured we should merge utrace in a way that lets everybody beat on it for new features and hash out details, without immediate risk of regressions in ptrace (which are severely annoying to everyone when they happen). The proper clean-ups for ptrace can proceed in parallel with work using utrace for things that are actually new and interesting, and at least the first half of that clean-up work is orthogonal to utrace.

This seems like the normal way that new optional CONFIG_FOOBAR features (marked EXPERIMENTAL, even) are handled. We don't jump over ourselves to make existing crucial code paths subject to a new subsystem that is getting its first rounds of shake-out.

Thanks,
Roland

#31 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Linus Torvalds
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Frank Ch. Eigler
Quoted 3 lines

If testing utrace against its main application requires installation of a complete enterprise distro from a distro [...]

This has *never* been a requirement.

You guys are getting off a tangent.

Let's go back to the post that started this all.

Quoted 5 lines

The thing is, utrace crashes in Fedora have dominated kerneloops.org for many months, so i'm not sure what to make of the idea of posting a 4000+ lines of core kernel code patchset on the last day of the development cycle, a posting that has carefully avoided the Cc:-ing of affected maintainers ;-)

Quoted 7 lines

.. and dammit, I agree 100%. If utrace really shows up in _any_ way on kerneloops.org, then I think THE ENTIRE DISCUSSION in this thread is moot.

I'm not going to take known-bad crap. It's that simple. Don't bother posting it, don't bother discussing it, don't bother making excuses for it.

The kerneloops stats on utrace crashes are way down currently, after that peak last fall. So i didnt want to suggest that it's known-broken now - i only wanted to point out that it's a known-risky area and that the submission of it should involve the affected maintainers/developers.

Regarding current stability, Roland, Frank, is the utrace patch in latest (today's) Fedora rawhide:

-rw-r--r-- 1 root root 176555 2009-01-08 05:42 linux-2.6-utrace.patch

a bug fixed equivalent of the utrace bits that crashed in the 2.6.26.3 kernel? In that case it is certainly known-good.

Or is it a slimmed-down version?

The ptrace bits and signoffs from Oleg and Alexey would certainly help (me) in trusting it. (I've Cc:-ed Oleg and Alexey)

The ftrace bits could certainly be staged to go in via the tracing tree (in .31 or so) after the utrace-core+ptrace bits went upstream.

Ingo

#32 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 2 lines · Ingo Molnar

The ptrace bits and signoffs from Oleg and Alexey would certainly help (me) in trusting it. (I've Cc:-ed Oleg and Alexey)

The further utrace stays away from mainline, the better. That's from my experience with this code.

But let's see how ptrace(2) rewrite will look like because this is the biggest thing that matters. All those cool virtual machines, fancy tracers and what not aren't even comparable.

Right now with ptrace(2) rewrite the following is easily triggerable:

WARNING: at kernel/ptrace.c:515 ptrace_report_signal+0x2c1/0x2d0()
Pid: 4814, comm: exe Not tainted 2.6.29-rc8-utrace #1
Call Trace:
 [<c0126df1>] warn_slowpath+0x81/0xa0
 [<c014c359>] ? validate_chain+0xe9/0x1150
 [<c014d606>] ? __lock_acquire+0x246/0xa50
 [<c0232959>] ? __delay+0x9/0x10
 [<c014b8eb>] ? mark_held_locks+0x6b/0x80
 [<c03d3dd2>] ? _spin_unlock_irq+0x22/0x50
 [<c012fdd1>] ptrace_report_signal+0x2c1/0x2d0
 [<c012fb10>] ? ptrace_report_signal+0x0/0x2d0
 [<c0154a79>] utrace_get_signal+0x1c9/0x660
 [<c0135478>] get_signal_to_deliver+0x288/0x330
 [<c01029e9>] do_notify_resume+0xb9/0x890
 [<c017edd2>] ? cache_free_debugcheck+0x232/0x2f0
 [<c014957b>] ? trace_hardirqs_off+0xb/0x10
 [<c03d3d79>] ? _spin_unlock_irqrestore+0x39/0x70
 [<c01015a0>] ? sys_execve+0x40/0x60
 [<c017f139>] ? kmem_cache_free+0x89/0xc0
 [<c014baad>] ? trace_hardirqs_on_caller+0xfd/0x190
 [<c014bb4b>] ? trace_hardirqs_on+0xb/0x10
 [<c010340a>] work_notifysig+0x13/0x19

It looks like WARN_ON is just bogus, but who knows.

maintainer
#33 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 4 lines · Alexey Dobriyan

Right now with ptrace(2) rewrite the following is easily triggerable:

WARNING: at kernel/ptrace.c:515 ptrace_report_signal+0x2c1/0x2d0()

Yes, ptrace-over-utrace needs more work. But your message looks as if utrace core is buggy, imho this is a bit unfair ;)

As Roland said, ptrace-over-utrace is not ready yet. If you mean that utrace core should not be merged alone - this is another story.

But personally I understand why Roland sends utrace core before changing ptrace.

Oleg.

#34 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 9 lines · Oleg Nesterov

Yes, ptrace-over-utrace needs more work. But your message looks as if utrace core is buggy, imho this is a bit unfair ;)

As Roland said, ptrace-over-utrace is not ready yet. If you mean that utrace core should not be merged alone - this is another story.

But personally I understand why Roland sends utrace core before changing ptrace.

Yes, but if it's going to be merged this during 2.6.x cycle, we need to have a user for the kernel interface along with the new kernel interface. This is true for any body trying to add some new infrastructure to the kernel; you have to have an in-tree user of said interface.

I mean, if some device manufacturer were to go to Red Hat's kernel team, and say, "we need this interface for our uber expensive RDMA interface card", and there was no in-kernel user for the interface, we know what Red Hat would tell that device manufacturer, right? So why is the SystemTap team trying to get an exception for utrace? It just seems a little hypocritical.

So what about the ftrace user of utrace? Is that ready to be merged?

- Ted

#35 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

So we need to work out what to do about utrace and I feel a need to hit the reset button on all this. Largely because I've forgotten everything and it was all confusing anyway.

Could those who object to utrace please pipe up and summarise their reasons?

Just to kick the can down the road a bit I merged the first two patches. The ftrace patch merged about as (un)successfully as one would expect.

#36 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Hi -

Quoted 2 lines · Andrew Morton

So we need to work out what to do about utrace and I feel a need to hit the reset button on all this. [...]

Thanks.

Quoted 1 line

[...] The ftrace patch merged about as (un)successfully as one would

A new version against -tip is coming by in a few days.

- FChE

#37 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Andrew Morton
Quoted 3 lines · Andrew Morton

So we need to work out what to do about utrace and I feel a need to hit the reset button on all this. Largely because I've forgotten everything and it was all confusing anyway.

Right, from my POV something like utrace is desirable, since its basically a huge multiplexer for the debugger state, eventually allowing us to have multiple debuggers attached to the same process.

So in that respect its a very nice feature.

Quoted 2 lines

Could those who object to utrace please pipe up and summarise their reasons?

Christoph used to have an opinion on this matter, so I've added him to the CC.

Last time when I looked at the code, it needed a bit more care and comments wrt lifetimes and such. I know Roland has done a lot on that front -- so I'll need to re-inspect.

As to in-kernel users, currently we only have ptrace, and no full conversion to utrace is in a mergeable shape afaik.

UML (Jeff CC'ed) might want to use this.

I know the Systemtap people need this (fche). But that isn't really moving towards mainline any time soon afaict.

Then there is this little thing called frysk which uses it, no idea what kind of kernel space that needs, nor where it lives -- or for that matter, wth it really does ;-)

Anyway, long story short, once people have had a little time to go over the code, and a few in-kernel users are lined-up, I think we should consider merging it.

maintainer
#38 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 24 lines · Peter Zijlstra
Quoted 3 lines · Andrew Morton

So we need to work out what to do about utrace and I feel a need to hit the reset button on all this. Largely because I've forgotten everything and it was all confusing anyway.

Right, from my POV something like utrace is desirable, since its basically a huge multiplexer for the debugger state, eventually allowing us to have multiple debuggers attached to the same process.

So in that respect its a very nice feature.

Quoted 2 lines

Could those who object to utrace please pipe up and summarise their reasons?

Christoph used to have an opinion on this matter, so I've added him to the CC.

Last time when I looked at the code, it needed a bit more care and comments wrt lifetimes and such. I know Roland has done a lot on that front -- so I'll need to re-inspect.

As to in-kernel users, currently we only have ptrace, and no full conversion to utrace is in a mergeable shape afaik.

Quoted 8 lines

UML (Jeff CC'ed) might want to use this.

I know the Systemtap people need this (fche). But that isn't really moving towards mainline any time soon afaict.

Then there is this little thing called frysk which uses it, no idea what kind of kernel space that needs, nor where it lives -- or for that matter, wth it really does ;-)

And Frank reminded me we have an ftrace tracer that utilizes utrace.

Quoted 3 lines

Anyway, long story short, once people have had a little time to go over the code, and a few in-kernel users are lined-up, I think we should consider merging it.

#39 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Peter Zijlstra

Hi -

Quoted 4 lines · Peter Zijlstra

[...] Right, from my POV something like utrace is desirable, since its basically a huge multiplexer for the debugger state, eventually allowing us to have multiple debuggers attached to the same process. [...]

Right.

Quoted 3 lines

Then there is this little thing called frysk which uses it, no idea what kind of kernel space that needs, nor where it lives -- or for that matter, wth it really does ;-)

Frysk was to be a first user of such an improved ptrace(2) API in order to do the sort of background / multiply-connected debugging, but that project has been on indefinite hold for about a year. Instead, there are experiments under way to extend gdb's backend for that capability.

- FChE

#40 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Peter Zijlstra
Quoted 5 lines · Peter Zijlstra
Quoted 2 lines

Could those who object to utrace please pipe up and summarise their reasons?

Christoph used to have an opinion on this matter, so I've added him to the CC.

I've never objected utrace per see, quite contrary I think it's a useful abstraction. I did have objection over various implementation details which should be sorted out now (have to take a look again to make sure).

I do have a really large objection of merging the current messy double ptrace implementation. If current utrace based ptrace isn't 100% ready there's absolutely no point in merging it. Other user would be even better, e.g. the seccomp rewrite.

#41 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 3 lines

I do have a really large objection of merging the current messy double ptrace implementation. If current utrace based ptrace isn't 100% ready there's absolutely no point in merging it.

There is no "current" utrace-ptrace implementation. I haven't proposed one for merging. When one is ready and working, we can discuss its actual technical details then.

Quoted 1 line

Other user would be even better, e.g. the seccomp rewrite.

The seccomp rewrite is a very simple user for which I have a prototype patch. (It needs testing, but that should be easy enough.) The only real complexity there is in deciding how to merge those changes. Its components are:

* clean up Kconfig
* remove old arch/asm hooks
** mips
** powerpc
** sh
** sparc
** x86
* replace kernel/seccomp.c with utrace-based one

Except for the first one, doing it in small incremental changes would leave some intermediate states with no seccomp feature usable in the tree. (And, of course, CONFIG_SECCOMP will require CONFIG_UTRACE thereafter.) Please advise on how many pieces to slice it into and how to stage the merging.

Thanks,
Roland

#42 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Ingo Molnar

Hi -

Quoted 10 lines · Ingo Molnar

[...]
To me personally there are two big direct usability issues with SystemTap:

1) It relies on DEBUG_INFO for any reasonable level of utility. Yes, it will limp along otherwise as well, but most of the actual novel capabilities depend on debuginfo. Which is an acceptable constraint for enterprise usage where kernels are switched every few months and having a debuginfo package is not a big issue. Not acceptable for upstream kernel development.

In my own limited kernel-building experience, I find the debuginfo data conveniently and instantly available after every "make". Can you elaborate how is it harder for you to incidentally make it than for someone to download it?

Quoted 4 lines

It also puts way too trust into the compiler generating 1GB+ of debuginfo correctly. I want to be able to rely on tools all the time and thus i want tools to have some really simple and predictable foundations.

Well, the data has to come from *somewhere*. We know several shortcomings (and have staff working on gcc debuginfo improvements), but there is little alternative. If not from the compiler, where are you going to get detailed type/structure layouts? Stack slot to variable mappings? Statement-level PC addresses? Unwind data?

Quoted 3 lines

2) It's not upstream and folks using it seem to insist on not having it upstream ;-) This 'distance' to upstream seems to have grown during the past few years - instead of shrinking. [...]

Considering our upstream-bound assistance with foundation technologies like markers, tracepoints, kprobes, utrace, and several other bits, this does not seem entirely fair.

Quoted 3 lines

If these fundamental problems are addressed then i'd even argue for the totality of SystemTap to be aimed upstreamed (including the scripting language, etc.), [...]

If consensus on this were plausible, we could seriously discuss it.

But I don't buy the package-deal that utrace must not attempt merging on its own merits, just because it makes systemtap (as it is today) useful to more people.

- FChE

#43 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 14 lines · Frank Ch. Eigler

Hi -

Quoted 10 lines · Ingo Molnar

[...]
To me personally there are two big direct usability issues with SystemTap:

1) It relies on DEBUG_INFO for any reasonable level of utility. Yes, it will limp along otherwise as well, but most of the actual novel capabilities depend on debuginfo. Which is an acceptable constraint for enterprise usage where kernels are switched every few months and having a debuginfo package is not a big issue. Not acceptable for upstream kernel development.

Quoted 4 lines

In my own limited kernel-building experience, I find the debuginfo data conveniently and instantly available after every "make". Can you elaborate how is it harder for you to incidentally make it than for someone to download it?

Four reasons:

1)

I have CONFIG_DEBUG_INFO turned off in 99.9% of my kernel builds, because it slows down the kernel build times significantly:

without: 4343.31 user 416.39 system 6:09.97 elapsed 1286%CPU
with: 4871.07 user 501.90 system 7:43.22 elapsed 1159 %CPU

( x86 allyesconfig. On an obscenely overpowered Nehalem box
with 12 GB of RAM. )

2)

When the kernel build becomes IO-bound, for example when i build over a distcc cluster (which is how i generally build my kernels) - or when others with less RAM build a debuginfo kernel, the ratio becomes even worse:

  without:   870.36 user 292.79 system 3:32.10 elapsed  548% CPU
  with:      929.65 user 384.55 system 8:28.70 elapsed  258% CPU

3)

Another metric. Here's an x86 defconfig (i.e. fairly regular config
- not allyesconfig) build's size:

with: 1645 MB
without: 211 MB

Try to build 1.6 GB of dirty data on ext3 and run into an fsync() in your editor ... you'll sit there twiddling thumbs for a minute or more.

4)

Or yet another metric - Linux distro package overhead. Try installing a debuginfo package:

 # yum install kernel-debuginfo

==========================================
Package Arch Version
==========================================
Installing:
kernel-debuginfo x86_64 2.6.29-0.258.rc8.git2.fc11
rawhide-debuginfo 294 M
Installing for dependencies:
kernel-debuginfo-common x86_64 2.6.29-0.258.rc8.git2.fc11
rawhide-debuginfo 35 M

Total download size: 329 M

That size of a _compressed_ debuginfo kernel package is obscene. We can fit 4 years of full Linux kernel Git history into that size - 60,000+ commits, full metadata and full 20 million lines of code flux included!

Uncompressed it blows up to gigabytes of on-disk data.

And this download has to be repeated for _every_ minor kernel upgrade.

So when i come into a situation where i could use some debugging help ... i'd have to rebuild the kernel with DEBUG_INFO=y and i'll always notice when i have a debuginfo kernel because it's inconvenient.

The solution?)

Dunno - but i definitely think we should think bigger:

The fundamental disconnect i believe seems to come from the fact that most user-space projects are relatively small, so debuginfo bloat is a secondary issue there.

But for a project with the size of the kernel, even for moderate builds (not allyesconfig), it's a _much_ bigger deal. This has been known for a long time and the situation has become worse over the last two years, not better. (last time i checked the debuginfo package overhead it was below 150 MB)

A few random ideas:

Instead of trying to cache 2+GB of debuginfo for a 50 MB kernel source repo (+50 MB of genuine .o output) - just to be able to debug one or two source files [which is the typical scope of a debugging session], why not build debuginfo on the fly, when a debugging session requires it? Rarely do we need debuginfo for more than a fraction of the whole kernel.

( Yes, it needs a few smarts like knowing the SHA1 of the source
code module that a particular kernel portion got built with, to make sure the debuginfo is fresh and relevant - but nothing major. )

I mean, lets _use_ the fact that we have source code available, more intelligently. It takes zero time to build detailed debuginfo for a portion of a tree.

If 'download debuginfo' can be replaced with: 'have a recent Git repository of the distro kernel source', we'll have a _much_ more efficient use of resources all around.

Ingo

#44 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 11 lines · Ingo Molnar

Total download size: 329 M

That size of a _compressed_ debuginfo kernel package is obscene. We can fit 4 years of full Linux kernel Git history into that size
- 60,000+ commits, full metadata and full 20 million lines of code flux included!

Uncompressed it blows up to gigabytes of on-disk data.

And this download has to be repeated for _every_ minor kernel upgrade.

Have to correct my memories about how many commits the kernel repo
has: 132,019 commits. That massive history fits into 298 MB. (!)

Ingo

#45 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Ingo Molnar

Hi -

Quoted 6 lines · Ingo Molnar

[...]

Quoted 4 lines

In my own limited kernel-building experience, I find the debuginfo data conveniently and instantly available after every "make". Can you elaborate how is it harder for you to incidentally make it than for someone to download it?

Quoted 6 lines

Four reasons:

1)

I have CONFIG_DEBUG_INFO turned off in 99.9% of my kernel builds, because it slows down the kernel build times significantly: [...]

OK, 15% longer time.

Quoted 5 lines

2)

When the kernel build becomes IO-bound [...]

  without:   870.36 user 292.79 system 3:32.10 elapsed  548% CPU
  with:      929.65 user 384.55 system 8:28.70 elapsed  258% CPU

OK, lots of network traffic.

Quoted 4 lines

3) [...]
Try to build 1.6 GB of dirty data on ext3 and run into an fsync() in your editor ... you'll sit there twiddling thumbs for a minute or more.

Now don't go blaming us for ext3 & fsync & not having a low enough /proc/sys/vm/dirty_background_ratio.

Quoted 3 lines

4)
Or yet another metric - Linux distro package overhead. Try installing a debuginfo package: [...]

Same as 3).

Quoted 2 lines

And this download has to be repeated for _every_ minor kernel upgrade.

Actually, no. If you just want to run the newly built kernel somewhere else on your network, you can run a systemtap compile server on your build machine, and let the systemtap network client do ~RPCs to get at the data.

Quoted 7 lines

The solution?)

Dunno - but i definitely think we should think bigger:

The fundamental disconnect i believe seems to come from the fact that most user-space projects are relatively small, so debuginfo bloat is a secondary issue there.

(Well, the fedora debuginfo archive shows a couple of packages of similar or greater heft than the kernel - openoffice.org, qt3, ...)

Quoted 8 lines

A few random ideas:

[...] why not build debuginfo on the fly, when a debugging session requires it? Rarely do we need debuginfo for more than a fraction of the whole kernel. [...]
I mean, lets _use_ the fact that we have source code available, more intelligently. It takes zero time to build detailed debuginfo for a portion of a tree. [...]

Something like that might be made to work.

Note that this backs away from earlier claims that we can make do without debuginfo, or that the compiler can't be trusted to build the stuff. We all agree it'd be nice if made it better and made a little less.

- FChE

#46 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 4 lines · Frank Ch. Eigler
Quoted 2 lines

I have CONFIG_DEBUG_INFO turned off in 99.9% of my kernel builds, because it slows down the kernel build times significantly: [...]

OK, 15% longer time.

It's way more than that if you don't have tons of memory and excessive amounts of CPU power.

Quoted 7 lines
Quoted 5 lines

2)

When the kernel build becomes IO-bound [...]

  without:   870.36 user 292.79 system 3:32.10 elapsed  548% CPU
  with:      929.65 user 384.55 system 8:28.70 elapsed  258% CPU

OK, lots of network traffic.

This is the _normal_ situation for a debug info build. If it's not network traffic (distcc), it's just disk IO. Have you tried it on a laptop? Ingo is not the only one that turns off DEBUG_INFO in disgust. It's totally useless for any sane kernel developer - the costs are excessive.

Adn that's totally ignoring the disk usage of multiple debug info kernels.

Quoted 4 lines

Note that this backs away from earlier claims that we can make do without debuginfo, or that the compiler can't be trusted to build the stuff. We all agree it'd be nice if made it better and made a little less.

Gaah. I'd wish you all agreed that DEBUG_INFO is just TOTALLY UNREALISTIC.

Linus

#47 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Andrew Morton
Quoted 3 lines

Well I dunno. You guys are closer to this than I am, but I'd have thought that systemtap is the main game here, and most/all of the above is just fluff.

That is certainly not true for me. It is true that I hear plenty from systemtap developers, users, and boosters wanting utrace to be merged. But all that "fluff" you dismiss out of hand is what I would really like to see become reality. Pretty much the only people who ever tell me they would hack on those things are the ones who say, "I'm looking forward to utrace getting merged in so I can try to write something."

Quoted 1 line

eh. Boring. [...]

Since it's boring to you, it must be so boring to everyone that they have no interest in a platform they can use to do exciting things with. Great. Silly me trying to enable collaboration to produce things less boring than I'm capable of myself. Clearly there is no need for any such thing. Sorry I'm so out of touch, but I just thought it was cool.

Thanks,
Roland

#48 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Andrew Morton
Quoted 4 lines · Andrew Morton
Quoted 1 line · On Sat, 21 Mar 2009 07:51:41 -0400 "Frank Ch. Eigler" <[email protected]> wrote:

On Sat, Mar 21, 2009 at 04:19:54AM -0700, Andrew Morton wrote:

I have strong memories of being traumatised by reading the uprobes code.

That was a long time ago wasn't it? :-)

That approach was a carry over from an implementation from dprobes that used readdir hooks. Yes, that was not the most elegant approach, as such has long been shelved.

Quoted 1 line

What's the story on all of that nowadays?

Utrace makes implementing uprobes more cleaner. We have a prototype that implements uprobes over utrace. Its per process, doesn't use any in-kernel hooks, etc. It currently has a kprobes like interface (needs a kernel module), but it shouldn't be difficult to adapt it to use utrace's user interfaces (syscalls?) when those come around. The current generation of uprobes that has all the bells and whistles can be found at
http://sources.redhat.com/git/gitweb.cgi?p=systemtap.git;a=tree;f=runtime/uprobes2

However, there are aspects of the current uprobes that can be useful to any other userspace tracer: instruction analysis, breakpoint insertion and removal, single-stepping support. With these layered on top of utrace, building userspace debug/trace tools that depend on utrace should be easier, outside of ptrace.

Work is currently on to factor these layers out. The intention is to upstream all the bits required for userspace tracing once utrace gets in, along with an easy to use interface for userspace developers (a /proc or /debugfs interface?) -- one should be able to use it on its own or with SystemTap, whatever they prefer. Details are still hazy at the moment.

But, utrace is the foundation to do all of that.

Ananth

#49 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 28 lines · On Tue, 24 Mar 2009 10:59:26 +0530 Ananth N Mavinakayanahalli…
Quoted 4 lines · Andrew Morton
Quoted 1 line · On Sat, 21 Mar 2009 07:51:41 -0400 "Frank Ch. Eigler" <[email protected]> wrote:

On Sat, Mar 21, 2009 at 04:19:54AM -0700, Andrew Morton wrote:

I have strong memories of being traumatised by reading the uprobes code.

That was a long time ago wasn't it? :-)

That approach was a carry over from an implementation from dprobes that used readdir hooks. Yes, that was not the most elegant approach, as such has long been shelved.

Quoted 1 line

What's the story on all of that nowadays?

Utrace makes implementing uprobes more cleaner. We have a prototype that implements uprobes over utrace. Its per process, doesn't use any in-kernel hooks, etc. It currently has a kprobes like interface (needs a kernel module), but it shouldn't be difficult to adapt it to use utrace's user interfaces (syscalls?) when those come around. The current generation of uprobes that has all the bells and whistles can be found at
http://sources.redhat.com/git/gitweb.cgi?p=systemtap.git;a=tree;f=runtime/uprobes2

However, there are aspects of the current uprobes that can be useful to any other userspace tracer: instruction analysis, breakpoint insertion and removal, single-stepping support. With these layered on top of utrace, building userspace debug/trace tools that depend on utrace should be easier, outside of ptrace.

Quoted 9 lines

Work is currently on to factor these layers out. The intention is to upstream all the bits required for userspace tracing once utrace gets in, along with an easy to use interface for userspace developers (a /proc or /debugfs interface?) -- one should be able to use it on its own or with SystemTap, whatever they prefer. Details are still hazy at the moment.

But, utrace is the foundation to do all of that.

The sticking point was uprobes's modification of live pagecache. We said "ick, COW the pages" and you said "too expensive". And there things remained.

Is that all going to happen again?

#50 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 37 lines · Andrew Morton
Quoted 35 lines · On Tue, 24 Mar 2009 10:59:26 +0530 Ananth N Mavinakayanahalli…
Quoted 4 lines · Andrew Morton
Quoted 1 line · On Sat, 21 Mar 2009 07:51:41 -0400 "Frank Ch. Eigler" <[email protected]> wrote:

On Sat, Mar 21, 2009 at 04:19:54AM -0700, Andrew Morton wrote:

I have strong memories of being traumatised by reading the uprobes code.

That was a long time ago wasn't it? :-)

That approach was a carry over from an implementation from dprobes that used readdir hooks. Yes, that was not the most elegant approach, as such has long been shelved.

Quoted 1 line

What's the story on all of that nowadays?

Utrace makes implementing uprobes more cleaner. We have a prototype that implements uprobes over utrace. Its per process, doesn't use any in-kernel hooks, etc. It currently has a kprobes like interface (needs a kernel module), but it shouldn't be difficult to adapt it to use utrace's user interfaces (syscalls?) when those come around. The current generation of uprobes that has all the bells and whistles can be found at
http://sources.redhat.com/git/gitweb.cgi?p=systemtap.git;a=tree;f=runtime/uprobes2

However, there are aspects of the current uprobes that can be useful to any other userspace tracer: instruction analysis, breakpoint insertion and removal, single-stepping support. With these layered on top of utrace, building userspace debug/trace tools that depend on utrace should be easier, outside of ptrace.

Work is currently on to factor these layers out. The intention is to upstream all the bits required for userspace tracing once utrace gets in, along with an easy to use interface for userspace developers (a /proc or /debugfs interface?) -- one should be able to use it on its own or with SystemTap, whatever they prefer. Details are still hazy at the moment.

Quoted 8 lines
Quoted 2 lines

But, utrace is the foundation to do all of that.

The sticking point was uprobes's modification of live pagecache. We said "ick, COW the pages" and you said "too expensive". And there things remained.

Is that all going to happen again?

No. All modifications are via access_process_vm().

Ananth

#51 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

In reply to Ingo Molnar
Quoted 3 lines

kernel/utrace.c should probably be introduced as kernel/trace/utrace.c not kernel/utrace.c. It also overlaps pending work in the tracing tree and cooperation would be nice and desired.

Of course I would like to cooperate with everyone. And of course it does not really matter much where a source file lives. But IMHO utrace really does not fit in with the kernel/trace/ code much at all. Sure, its hooks can be used by tracer implementations that use CONFIG_TRACING stuff. But it is a general API about user thread state. It belongs in kernel/trace/ "naturally" far less than, say, kprobes. utrace will in future be used to implement userland features (ptrace et al) that are just aspects of the basics of what an operating system does: mediate userland for userland. Those uses will have nothing to do with "kernel tracing".

Thanks,
Roland

#52 lore raw

Re: [PATCH 3/3] utrace-based ftrace "process" engine, v2

Quoted 4 lines · Roland McGrath
Quoted 3 lines

kernel/utrace.c should probably be introduced as kernel/trace/utrace.c not kernel/utrace.c. It also overlaps pending work in the tracing tree and cooperation would be nice and desired.

Quoted 11 lines

Of course I would like to cooperate with everyone. And of course it does not really matter much where a source file lives. But IMHO utrace really does not fit in with the kernel/trace/ code much at all. Sure, its hooks can be used by tracer implementations that use CONFIG_TRACING stuff. But it is a general API about user thread state. It belongs in kernel/trace/ "naturally" far less than, say, kprobes. utrace will in future be used to implement userland features (ptrace et al) that are just aspects of the basics of what an operating system does: mediate userland for userland. Those uses will have nothing to do with "kernel tracing".

But it is fitting if you think of kernel/trace/ as kernel/instrumentation/.

The virtualization-alike uses for utrace are in essence using system call instrumentation callbacks to inject extra functionality into the system. That's possible not because it's primarily geared at doing that, but because the instrumentation callbacks are generic and complete enough. It's still correct to think of it as an instrumentation tool and to maintain it as such. That also makes it clear that none of these APIs are to be regarded permanent ABIs.

Anyway ... placement is no big deal, and kernel/utrace.c is certainly a good way of avoiding the tracing tree ;-)

Ingo