Skip to contentSkip to conversation

LTTng-core (basic tracing infrastructure) 0.5.108

PATCH In review Mathieu Desnoyers 24 participants 277 messages Sep 14, 2006 – Sep 20, 2006 1 message not in the archive kprobes Page 2 of 2

lore
Share & tools
lore
#270 lore raw
In reply to Michel Dagenais
Quoted 10 lines · Michel Dagenais
Quoted 6 lines

the question is: what is more maintainance, hundreds of static tracepoints (with long parameter lists) all around the (core) kernel, or hundreds of detached dynamic rules that need an update every now and then? [but of which most would still be usable even if some of them "broke"] To me the answer is clear: having hundreds of tracepoints _within_ the source code is higher cost. But please prove me wrong :-)

Actually I rarely find that any of the 70 000 printk is such a huge nuisance to code readability. They may even help understand what is going on in a code area you are less familiar with.

i disagree. Consider the following example from LTT:

int sock_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)

 {
         struct kiocb iocb;
         struct sock_iocb siocb;
         int ret;

         trace_socket_sendmsg(sock, sock->sk->sk_family,
                 sock->sk->sk_type,
                 sock->sk->sk_protocol,
                 size);

         init_sync_kiocb(&iocb, NULL);
         iocb.private = &siocb;
         ret = __sock_sendmsg(&iocb, sock, msg, size);
         if (-EIOCBQUEUED == ret)
                 ret = wait_on_sync_kiocb(&iocb);
         return ret;

}

what do the 5 extra lines introduced by trace_socket_sendmsg() tell us? Nothing. They mostly just duplicate the information i already have from the function declaration. They obscure the clear view of the function:

int sock_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)

 {
         struct kiocb iocb;
         struct sock_iocb siocb;
         int ret;

         init_sync_kiocb(&iocb, NULL);
         iocb.private = &siocb;
         ret = __sock_sendmsg(&iocb, sock, msg, size);
         if (-EIOCBQUEUED == ret)
                 ret = wait_on_sync_kiocb(&iocb);
         return ret;

}

the resulting visual and structural redundancy hurts.

Ingo

#271 lore raw
Quoted 1 line

i disagree. Consider the following example from LTT:

...

Quoted 4 lines
         trace_socket_sendmsg(sock, sock->sk->sk_family,
                 sock->sk->sk_type,
                 sock->sk->sk_protocol,
                 size);

...

Quoted 1 line

what do the 5 extra lines introduced by trace_socket_sendmsg() tell us? Nothing. They mostly just duplicate the information i already have from the function declaration. They obscure the clear view of the function:

...

Quoted 1 line

the resulting visual and structural redundancy hurts.

Couldn't that be easily fixed by just doing

	trace_socket_sendmsg(sock, size);

and have it work out which esoteric parts of the sock we want to trace, and which we don't? Is much less visually invasive, and gives the same effect.

M.

#272 lore raw
Quoted 14 lines · Martin Bligh
Quoted 1 line

i disagree. Consider the following example from LTT:

...

Quoted 4 lines
        trace_socket_sendmsg(sock, sock->sk->sk_family,
                sock->sk->sk_type,
                sock->sk->sk_protocol,
                size);

...

Quoted 3 lines

what do the 5 extra lines introduced by trace_socket_sendmsg() tell us? Nothing. They mostly just duplicate the information i already have from the function declaration. They obscure the clear view of the function:

...

Quoted 1 line

the resulting visual and structural redundancy hurts.

Quoted 7 lines

Couldn't that be easily fixed by just doing

	trace_socket_sendmsg(sock, size);

and have it work out which esoteric parts of the sock we want to trace, and which we don't? Is much less visually invasive, and gives the same effect.

yeah, visual impact is everything. The best that Frank and me came up with is:

	_(socket_sendmsg, sock, size);

we could quickly learn to visually skip over lines like that, they have a pretty unique geometric form . While if it's called:

	trace_socket_sendmsg(sock, size);

it always looks like a function call in the corner of the eye and attracts attention.

the '_()' macro is defined as:

	#define _(x,y,z) STAP_MARK(x,y,z)

(STAP_MARK is an existing SystemTap helper to insert static tracepoints into the kernel.)

but the other property of dynamic tracing is still very important too:
we have the technological freedom to remove static tracepoints, if we decide so. With static tracers, once they are in the tree, we are stuck with these APIs.

Ingo

#273 lore raw
In reply to Martin J. Bligh
Quoted 1 line · Martin J. Bligh

2. You can get zero overhead by CONFIG'ing things out.

IOW, no distro will enable it by default to avoid the overhead, making it useless for lots of real-world working systems where you need to guess what's hapenning to software running real workloads that can't just be stopped.

I guess there's no problem in having both LTT and Kprobes merged in the main tree at the same time. But Kprobes + systemtap will get enabled and used by distros massively just because users can start using it inmediately, without recompiling or installing extra kernels and rebooting. There're cases where distros may want to enable automatic tracing in every boot and only on boot but that don't like to suffer from an extra performance hit after booting...

I'm not meaning that LTT sucks and doesn't have advantages and that doesn't deserve being merged/used, it just looks like kprobes+systemtap will get way more real-world users no matter how much you discuss here

#274 lore raw
Quoted 1 line · grundig

IOW, no distro will enable it by default to avoid the overhead,

Please bear in mind that this is an implementation issue. As I've explained elsewhere, there are ways to implement this where even compiled-in static tracepoints have practically no cost at all -- being noops until enabling. Thereby being no justification for not actually shipping with such built kernels and, therefore, no reason why tools such as ltt can't real-world usage.

Karim

#275 lore raw
In reply to Martin J. Bligh
Quoted 2 lines · Martin J. Bligh

[...] What would be really nice is one trace infrastructure, that allowed both static and dynamic tracepoints

We in systemtap land hope to encounter *some* static tracepoint structure, perhaps like the one I presented at OLS, via which systemtap could become your unified static+dynamic "infrastructure". Even in that universe, using LTT-derived code for high-performance tracing is within the realm of reason.

Quoted 2 lines

without all the awk-style language crap that seems to come with systemtap.

I'm sorry to hear you dislike the scripting language. But that's okay, you Real Men can embed literal C code inside systemtap scripts to do the Real Work, and leave to systemtap only sundry duties such as probe placement and removal.

- FChE

#276 lore raw
Quoted 10 lines · Frank Ch. Eigler
Quoted 2 lines · Martin J. Bligh

without all the awk-style language crap that seems to come with systemtap.

I'm sorry to hear you dislike the scripting language. But that's okay, you Real Men can embed literal C code inside systemtap scripts to do the Real Work, and leave to systemtap only sundry duties such as probe placement and removal.

There are also a couple of projects within SystemTap that provide trace like functionality without the need to use the SystemTap language. In the case of LKET, we've tried to make this as simple as possible by predefining probe points using the SystemTap language and embedded C code, but from a users perspective all he really need to do is just invoke a simple script like:

#! stap
process_snapshot() {}
addevent.tskdispatch.cpuidle {}
addevent.process {}
addevent.syscall.entry { printf ("%4b", $flags) }
addevent.syscall.exit {}
addevent.tskdispatch.cpuidle {}

The data can later be analyses in user-space with what ever method you like. The developer instrumenting the probe point needs to know the Systemtap language, but the user of the trace just need to know which events are available to him.

We also plan to do static tracing once SystemTap supports static markers. This may not be the perfect solution, but I'm interested in knowing how we can get there.

-JRS

#277 lore raw
In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar
Quoted 7 lines · Mathieu Desnoyers

Following an advice Christoph gave me this summer, submitting a smaller, easier to review patch should make everybody happier. Here is a stripped down version of LTTng : I removed everything that would make the code review reluctant (especially kernel instrumentation and kernel state dump module). I plan to release this "core" version every few LTTng releases and post it to LKML.

Quoted 5 lines
Quoted 1 line

Comments and reviews are very welcome.

i have one very fundamental question: why should we do this source-intrusive method of adding tracepoints instead of the dynamic, unintrusive (and thus zero-overhead) KProbes+SystemTap method?

Coming a little late to this thread because I've been travelling the last three weeks I'll answer here before wading through hundreds of mails.

I'll categorize tracing methods into a few categories:

a) static and in-inline

These are tracepoints directly in the kernel source, always compiled in (or under a CONFIG option). We have various ad-hoc tracers of this type already in the kernel, e.g. blktrace or xfs's ktrace

b) dynamic and in-line (markers)

These are in-line but normally don't do anything in the code except of maybe adding a nop. We currently don't support this at all.

c) dynamic and out-of-line

These are mainained as external modules or things that need to be translated to modules. We have various low-level mechanisms to implement the hooking up of those currently (*probes) but no other infratsurcture in the kernel to help with those. There's an external project, systemtap which supports probes like those but has a bunch of problems:

- it doesn't allow writing scripts in C but only in some odd scripting language
- it doesn't actually put support code into the kernel tree but keeps it separate, not allowing to keep probes with the kernel either. In addition it also needs quite frequent updates because it has to poke deep into kernel internals by it's nature.

So what's the right way of tracing for us? I'd say a pretty clear all three,
and most importantly we need to have a common infrastrucuture for all of those.

The most important bit we need right now is a reliable framework to transfer trace data to userspace - one we have that we support a) and a subset of
b) above. LTT might be that missing bit, but I'd need to look at the actual patches to see if it's suitable. b) is something people have talked about a lot and we've seen lots of prototypes, in my eyes it's the second priority.

But even after that the way we support c) is very rudimentary - we need helpers to look at data, put probes at points outside of function entry/ return we needs things like a dwarf parser, an so on.

I think the systemtap approach of the external package is the very last thing we need. Unlike you said elsewhere having the tracepoints externally does not eliminitate maintaince overhead - it shifts it to someone else. Shifting maintaince overhead to someone else is a valid concept in the linux kernel development, we do this all the time for things we don't care about. I think it's fundamentally wrong for traces, though. Traces are very important for debugging complex problems, and I've grown very tired of maintaining all my ad-hoc scripts. Having them in the kernel tree or traces static in it's nature inline would allow and force kernel developers to always keept it uptodate with it's changes.