malloc() tracing in perf?
Quoted 44 lines · Peter Zijlstra
Quoted 11 lines · Pekka Enberg
Hi,
It's me again :-).
I have a little user-space application that is pretty memory hungry and I want to understand why. I started to google around for a memory profiler or a
malloc()tracer but didn't seem to find anything really useful.But then it hit me, why can't I have kmemtrace + perf but for user-space? Something like the "Malloc Trace" shown here:
Does this sound like something that could/should be part of "perf"? How would all this work anyway? Can we intercept
malloc()andfree()somehow? Where would the data be pushed? Am I just going perf-crazy and trying to turn it into a swiss army knife because it's so easy to use?-)OK, you just trod into a wasp's nest :-)
That sounds like uprobes, the equivalent of kprobes but for userspace.
I seem to have heard people are working on such a thing, but I can't seem to find a single LKML post with 'uprobe' in the subject in the past two years or something (except for MTUprobe) -- so I guess its not really going anywhere any fast.
Now doing probes on userspace is hard because you need to know more about the userspace bits than a kernel really ought to be interested in.
Then again, the only way to extract bits from userspace is to stop it -- now one could do that using [pu]trace and have some monitoring app prod at it like any other debugger would, and I think this is the approach suggested by some (hch iirc).
Others seem to think we ought to stuff all this into the kernel, I can only imagine the pain that will cause, since you need to teach the kernel about these instrumentation sites' context, so I can only imagine it'd be through a kernel module interface much like system-tap does (they would be doing the in-kernel bit).
Then there are the tracer folks who also want to collect userspace traces. Some have proposed a
sys_trace()call, others want to play silly games withmmap()and then there is the uprobe idea. Others (tglx and me) proposed letting userspace log itself and post-merge all the various trace buffers to get a complete picture.
Quoted 6 lines
Anyway, like you say, it has uses (potentially very powerful ones), Sun/Apple do it with Dtrace, Linux wants it but I don't think we quite agreed on how to do it :-)
And here I see LKML isn't on the CC list, perhaps we should?
Oh sure, lets do that, half of the people seem to be on the CC anyway now :-).
Pekka
Quoted 26 lines · Pekka Enberg
Hi Peter,
Quoted 12 lines · Pekka Enberg
Quoted 12 lines
Hi,
It's me again :-).
I have a little user-space application that is pretty memory hungry and I want to understand why. I started to google around for a memory profiler or a
malloc()tracer but didn't seem to find anything really useful.But then it hit me, why can't I have kmemtrace + perf but for user-space? Something like the "Malloc Trace" shown here:
Does this sound like something that could/should be part of "perf"? How would all this work anyway? Can we intercept
malloc()andfree()somehow? Where would the data be pushed? Am I just going perf-crazy and trying to turn it into a swiss army knife because it's so easy to use?-)Quoted 8 lines · Peter Zijlstra
OK, you just trod into a wasp's nest :-)
That sounds like uprobes, the equivalent of kprobes but for userspace.
I seem to have heard people are working on such a thing, but I can't seem to find a single LKML post with 'uprobe' in the subject in the past two years or something (except for MTUprobe) -- so I guess its not really going anywhere any fast.
Quoted 5 lines
[snip]
Right. Are dynamic tracepoints what we want for
malloc()andfree()interception, though? I guess we can just do generic "userspace function called" events and record the passed parameters. Then "perf memreport" can just go find all themalloc()andfree()calls and construct a memory profile out of it?That said, I'd also be interested in wiring my userspace VM garbage collector to emit similar alloc and free events as well (because perf already has a JIT symbol map) which makes me think we want generic "userspace allocated memory" events.
Argh, I dropped LKML from the CC. Sorry for the duplicate email!
Pekka
Quoted 43 lines · Pekka Enberg
Quoted 42 lines · Pekka Enberg
Hi Peter,
Quoted 20 lines · Pekka Enberg
Quoted 20 lines
Hi,
It's me again :-).
I have a little user-space application that is pretty memory hungry and I want to understand why. I started to google around for a memory profiler or a
malloc()tracer but didn't seem to find anything really useful.But then it hit me, why can't I have kmemtrace + perf but for user-space? Something like the "Malloc Trace" shown here:
Does this sound like something that could/should be part of "perf"? How would all this work anyway? Can we intercept
malloc()andfree()somehow? Where would the data be pushed? Am I just going perf-crazy and trying to turn it into a swiss army knife because it's so easy to use?-)Quoted 8 lines · Peter Zijlstra
OK, you just trod into a wasp's nest :-)
That sounds like uprobes, the equivalent of kprobes but for userspace.
I seem to have heard people are working on such a thing, but I can't seem to find a single LKML post with 'uprobe' in the subject in the past two years or something (except for MTUprobe) -- so I guess its not really going anywhere any fast.
[snip]
Right. Are dynamic tracepoints what we want for
malloc()andfree()interception, though? I guess we can just do generic "userspace function called" events and record the passed parameters. Then "perf memreport" can just go find all themalloc()andfree()calls and construct a memory profile out of it?
Quoted 4 lines
Quoted 4 lines
That said, I'd also be interested in wiring my userspace VM garbage collector to emit similar alloc and free events as well (because perf already has a JIT symbol map) which makes me think we want generic "userspace allocated memory" events.
Right, there is the distinction between static and dynamic tracepoints, and I'm not sure how that'll work out. Most people who've looked at the subject matter are on CC, so I'm hoping someone will expand.
That said, we could simply LD_PRELOAD malloc and free (new and delete) like any other memory debugger out there already does, that's not rocket science (in fact, I've done so on many an occasion).
I remember mpatrol being a hackable code base, but I just looked it up and its LGPL-v3 so I'm not sure we should be borrowing code from there. Maybe there is another such creature out there we can borrow code from.
I've got similar ideas for your JIT stuff, you now open-code the tmp file writing, ideally I'd like to push that into a library and have you link against something with weak stubs and let perf LD_PRELOAD real code which will do whatever is needed. That'd rid you of having to know about where and what to write.
Anyway, malloc/free are easy, but like you say, you might be interested in other functions which aren't as easy.
Hi -
Quoted 8 lines · Peter Zijlstra
[...]
Quoted 3 lines
But then it hit me, why can't I have kmemtrace + perf but for user-space? Something like the "Malloc Trace" shown here:
[...]I seem to have heard people are working on such a thing, but I can't seem to find a single LKML post with 'uprobe' in the subject in the past two years [...]
That work is ongoing, and being discussed on [email protected], since it is a prerequisite. One of the widgets that is being proposed for it is an ftrace engine frontend for uprobes, paralleling the one mhiramat wrote for kprobes.
Quoted 5 lines
Now doing probes on userspace is hard because you need to know more about the userspace bits than a kernel really ought to be interested in. [...] Anyway, like you say, it has uses (potentially very powerful ones), Sun/Apple do it with Dtrace, Linux wants it but I don't think we quite agreed on how to do it :-)
While these deliberations are ongoing, you can use systemtap. Probing random places in userspace is about as casual as probing the kernel:
# stap -e '
probe process("/lib/libc.so.6").function("malloc").return,
process("/lib/libc.so.6").function("free").return
{ println(probfunc()," ",$$parms," ",$$return) }
' -c 'ls' # -c 'CMD ARGS'__libc_malloc bytes=0x238 return=0x1524010__libc_malloc bytes=0x238 return=0x1524010__libc_malloc bytes=0x78 return=0x1524250__libc_malloc bytes=0xa return=0x15242d0__libc_free mem=0x1524250__libc_free mem=0x1524010__libc_malloc bytes=0x64 return=0x1524010__libc_malloc bytes=0x16 return=0x1524080__libc_free mem=0x1524010__libc_malloc bytes=0x64 return=0x1524010
[...]__libc_free mem=0x1524490__libc_free mem=0x1524500__libc_free mem=0x15245e0__libc_free mem=0x1524620__libc_free mem=0x1524430__libc_free mem=0x1524570__libc_free mem=0x1524660__libc_free mem=0x15246d0__libc_free mem=0x15242d0
You can google some other success stories or ask for more help at [email protected].
- FChE
Quoted 8 lines · Frank Ch. Eigler
Hi -
Quoted 4 lines · Peter Zijlstra
[...]
Quoted 3 lines
But then it hit me, why can't I have kmemtrace + perf but for user-space? Something like the "Malloc Trace" shown here:
[...]
Quoted 6 lines
Quoted 3 lines
I seem to have heard people are working on such a thing, but I can't seem to find a single LKML post with 'uprobe' in the subject in the past two years [...]
That work is ongoing, and being discussed on [email protected], since it is a prerequisite.
Still hiding the discussion and the design never helped anybody.
You might think it ready and then post it to LKML only to have it ripped apart -- have the discussion openly please.
Quoted 8 lines
Quoted 5 lines
Now doing probes on userspace is hard because you need to know more about the userspace bits than a kernel really ought to be interested in. [...] Anyway, like you say, it has uses (potentially very powerful ones), Sun/Apple do it with Dtrace, Linux wants it but I don't think we quite agreed on how to do it :-)
While these deliberations are ongoing, you can use systemtap. Probing random places in userspace is about as casual as probing the kernel:
Right, but that still doesn't tell us anything on how you're doing that, does it?
Hi -
Quoted 5 lines · Peter Zijlstra
[...]
Quoted 2 lines
That work is ongoing, and being discussed on [email protected], since it is a prerequisite.
Still hiding the discussion and the design never helped anybody.
I guess "hiding" is a matter of opinion.
Quoted 5 lines
Quoted 2 lines
While these deliberations are ongoing, you can use systemtap. Probing random places in userspace is about as casual as probing the kernel:
Right, but that still doesn't tell us anything on how you're doing that, does it?
Since you asked... the probe process("/lib64/libc.so.6") points systemtap to a shared library file, whose symbol table & debug data gives us information about what functions & parameters are available. Among other things, we record a shared-library base-relative address for the function.
At run time, we monitor the entire system (or just a given process if -x PID/-c CMD was specified) to see when that shared library gets loaded. (This in turn is done with a utrace-based hook of the mmap syscall - see "task_finder" in our sources). When it's loaded, we can finally compute a run-time address for the probed malloc/free functions. At this point we insert a uprobe (using the uprobes module API, which also relies on utrace.) Should the shared library become unmapped, or the application exit, the uprobe will be removed.
When/if the uprobe eventually hits, we run the compiled probe handler. As usual, it can do lots of stuff beside just simple tracing, but in the case of that statement (println ($$parms ...)) that's all we do. $$parms was expanded at compilation time to the list of variables, and instructions about how to fetch their values at run time. So now those instructions are run, their values formatted, and the tracing message sent to our buffer.
This overview skims over issues related to return probes, tracing buffer manipulations, and much other stuff.
All the code for this is hidden in plain sight in every systemtap release, so please feel free to refer to that and/or ask more detailed questions.
- FChE
Quoted 10 lines · Frank Ch. Eigler
Hi -
Quoted 5 lines · Peter Zijlstra
[...]
Quoted 2 lines
That work is ongoing, and being discussed on [email protected], since it is a prerequisite.
Still hiding the discussion and the design never helped anybody.
I guess "hiding" is a matter of opinion.
Yeah in plain sight where nobody looks, really I thought the idea was to get utrace upstream, this means posting to linux-kernel. Going on LKML posts I'd say the utrace project was dead. Same goes for uprobe.
I've told this before and I'll say it again, post to LKML.
Quoted 12 lines
Quoted 5 lines
Quoted 2 lines
While these deliberations are ongoing, you can use systemtap. Probing random places in userspace is about as casual as probing the kernel:
Right, but that still doesn't tell us anything on how you're doing that, does it?
Since you asked... the probe
process("/lib64/libc.so.6")points systemtap to a shared library file, whose symbol table & debug data gives us information about what functions & parameters are available. Among other things, we record a shared-library base-relative address for the function.
Quoted 4 lines
At run time, we monitor the entire system (or just a given process if -x PID/-c CMD was specified) to see when that shared library gets loaded. (This in turn is done with a utrace-based hook of the mmap syscall - see "
task_finder" in our sources).
Does this also iterate the already existing tasks to find if it was already mmap()ed?
Quoted 2 lines
This overview skims over issues related to return probes, tracing buffer manipulations, and much other stuff.
Right, so you basically read the (dwarf2) debug info for a particular lib/symbol and generate a kernel module that knows about that and then insert that into the kernel to act as a uprobe handler?
Uprobe will then do the code rewrite on mmap() time to insert a trap much like kprobe does? What if its a JITed code section and the JIT rewrites it? Will uprobes detect that?
Quoted 3 lines
All the code for this is hidden in plain sight in every systemtap release, so please feel free to refer to that and/or ask more detailed questions.
I thought the goal was to get stuff upstream, but if you want to continue living in la-la land and not bother with upstream Linux then so be it I guess :-(
Quoted 44 lines · * Peter Zijlstra ([email protected]) wrote:
Quoted 12 lines · Pekka Enberg
Hi,
It's me again :-).
I have a little user-space application that is pretty memory hungry and I want to understand why. I started to google around for a memory profiler or a
malloc()tracer but didn't seem to find anything really useful.But then it hit me, why can't I have kmemtrace + perf but for user-space? Something like the "Malloc Trace" shown here:
Quoted 6 lines
Does this sound like something that could/should be part of "perf"? How would all this work anyway? Can we intercept
malloc()andfree()somehow? Where would the data be pushed? Am I just going perf-crazy and trying to turn it into a swiss army knife because it's so easy to use?-)OK, you just trod into a wasp's nest :-)
That sounds like uprobes, the equivalent of kprobes but for userspace.
I seem to have heard people are working on such a thing, but I can't seem to find a single LKML post with 'uprobe' in the subject in the past two years or something (except for MTUprobe) -- so I guess its not really going anywhere any fast.
Now doing probes on userspace is hard because you need to know more about the userspace bits than a kernel really ought to be interested in.
Then again, the only way to extract bits from userspace is to stop it -- now one could do that using [pu]trace and have some monitoring app prod at it like any other debugger would, and I think this is the approach suggested by some (hch iirc).
Others seem to think we ought to stuff all this into the kernel, I can only imagine the pain that will cause, since you need to teach the kernel about these instrumentation sites' context, so I can only imagine it'd be through a kernel module interface much like system-tap does (they would be doing the in-kernel bit).
Quoted 6 lines
Then there are the tracer folks who also want to collect userspace traces. Some have proposed a
sys_trace()call, others want to play silly games withmmap()and then there is the uprobe idea. Others (tglx and me) proposed letting userspace log itself and post-merge all the various trace buffers to get a complete picture.
Hi !
Since we are comparing solutions, here is the path we are taking in LTTng. We choose to let userspace log itself and merge the kernel/userspace trace streams at post-processing. Past winter, Pierre-Marc has been working on a port of the LTTng kernel tracer to userspace. We named it "ust" library (for userspace tracing). Although it still requires some polishing, it should be a generally usable state (currently only on x86 with synchronized TSCs).
The plan is to gradually instrument userspace libraries and applications with tracepoints, just like we are currently doing in the kernel.
The main difference between our approach and Dtrace is that Dtrace requires a trap when userspace instrumentation is hit (yeah, even for static instrumentation, isn't it a shame ?), adding a huge performance overhead (especially on x86). Using tracepoints and staying in userspace to log data helps keeping the performance overhead low.
For more info on lib ust :
Combined Tracing of the Kernel and Applications with LTTng at Linux Symposium 2009 (presentation)
http://lttng.org/sites/default/files/presentation.pdf
Access to the git tree:
http://git.dorsal.polymtl.ca/?p=ust.git
(please note that we are about 99.9% done in asking permissions or rewriting code to distribute under LGPL. I think we only wait for 1-2 bits of code to be rewritten. Pierre-Marc could tell more about it. Until then, it's temporarily distributed under GPLv2.)
Eventually, parts of this ust library could be integrated to the libc, so its usage should become more transparent that it currently is.
Eventually, the only dependency ust would have on the kernel is to require a userspace trace clock mapped in a vDSO, synchronized with the kernel trace clock.
Feedback is welcome,
Thanks,
Mathieu
Quoted 8 lines
Anyway, like you say, it has uses (potentially very powerful ones), Sun/Apple do it with Dtrace, Linux wants it but I don't think we quite agreed on how to do it :-)
And here I see LKML isn't on the CC list, perhaps we should?
Signature · 2 lines
Mathieu Desnoyers OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Frank Ch. Eigler
Hi -
Quoted 5 lines · Peter Zijlstra
[...]
Quoted 2 lines
That work is ongoing, and being discussed on [email protected], since it is a prerequisite.
Still hiding the discussion and the design never helped anybody.
Quoted 7 lines
Quoted 1 line
I guess "hiding" is a matter of opinion.
Yeah in plain sight where nobody looks, really I thought the idea was to get utrace upstream, this means posting to linux-kernel. Going on LKML posts I'd say the utrace project was dead. Same goes for uprobe.
I've told this before and I'll say it again, post to LKML.
I second this. _Any_ kernel development that is new must be discussed on LKML. Sure, you can do the initial prototypes on a smaller mailing list, but once you have that prototype ready, it should go to LKML. This was the problem with Xen. The -rt patch was developed on LKML and even though there's a lot more to get in, a lot already has. The reason is that all development was done on the kernel mailing list.
-- Steve
Hi -
Quoted 3 lines · Peter Zijlstra
[...] Yeah in plain sight where nobody looks, really I thought the idea was to get utrace upstream, this means posting to linux-kernel. Going on LKML posts [...]
I don't wish to get into arguing about how much lkml posting is appropriate, or to editorialize about what happened when some of this code did actually get posted. I'm simply discussing the technology. It's never been hard to find.
Quoted 3 lines
[...]
Does this also iterate the already existing tasks to find if it was already mmap()ed?
Yes.
Quoted 6 lines
Quoted 2 lines
This overview skims over issues related to return probes, tracing buffer manipulations, and much other stuff.
Right, so you basically read the (dwarf2) debug info for a particular lib/symbol and generate a kernel module that knows about that and then insert that into the kernel to act as a uprobe handler?
Yes.
Quoted 3 lines
Uprobe will then do the code rewrite on
mmap()time to insert a trap much like kprobe does? What if its a JITed code section and the JIT rewrites it? Will uprobes detect that?
You mean if the uprobe is made to refer to code that was created by a JIT? That does not happen in the preceding scenario, since the uprobes addresses were based on prior generated debugging data.
Putting uprobes into JIT code such as LLVM or JVM code is something for the future.
Quoted 7 lines
Quoted 3 lines
All the code for this is hidden in plain sight in every systemtap release, so please feel free to refer to that and/or ask more detailed questions.
I thought the goal was to get stuff upstream, but if you want to continue living in la-la land and not bother with upstream Linux then so be it I guess :-(
It's not that simple, and you know it.
- FChE