utrace: new modular infrastructure for user debug/tracing
In-Reply-To: <[email protected]>
Quoted 3 lines · Roland McGrath
I have been working on for a while, and imagining for much longer, replacing ptrace from the ground up. This is what I've come up with so far, and I'm looking for some reactions on the direction.
At least three different sets of people want to extend the syscall tracing. Jeff Dike posted a patch that lets you supply a bitmask of syscalls to trace. Renzo Davoli posted one that lets you decide, after trapping entrance to a syscall, whether to skip the trap that would normally be done on exit from the same call. Charles P. Wright also had a similar patch. I think this needs to be done at the utrace level -- a tracing engine couldn't add that on its own (could it?)
Renzo Davoli also posted a patch to allow "batching" of ptrace requests and Systemptap really needs this, too. AFAICT this can be done by writing a custom engine.
And BTW patches 1 and 2 never made it to the list. The ones on your server (http://redhat.com/~roland/utrace/) don't apply cleanly due to whitespace damage but that can be fixed by stripping trailing whitespace from the kernel files patch(1) complains about.
Signature · 1 line
Chuck
Quoted 3 lines · Chuck Ebbert
[...] Renzo Davoli also posted a patch to allow "batching" of ptrace requests and Systemptap really needs this, too. AFAICT this can be done by writing a custom engine. [...]
Indeed, I like what I see (at least those parts I understand) in roland's utrace code. One missed opportunity bit appears to be any new support for something like per-thread breakpoints.
If I correctly understand how gdb etc. work, a hit breakpoint involves stoppage of all other threads of a process, then the breakpoint instruction is replaced by the original one, then the thread is single-stepped, then the breakpoint is put back, then finally all threads are resumed. Could utrace API provide short-lived per-thread page copies to execute the single-stepped original instruction out of, and avoid stopping & resuming all other threads?
- FChE
Quoted 7 lines
At least three different sets of people want to extend the syscall tracing. Jeff Dike posted a patch that lets you supply a bitmask of syscalls to trace. Renzo Davoli posted one that lets you decide, after trapping entrance to a syscall, whether to skip the trap that would normally be done on exit from the same call. Charles P. Wright also had a similar patch. I think this needs to be done at the utrace level -- a tracing engine couldn't add that on its own (could it?)
Indeed I do think a tracing engine can and should do those things itself. An engine's report_syscall callback can look in a bitmap that it maintains itself and decide to return quickly without doing anything. That callback is only a function call or two away from where a core-level check in a bitmap would be. It's my expectation that the overhead of an engine getting a callback and bailing out quickly will not be troublesome. (I'd guess the biggest slowdown might be just from taking the slow path in the assembly necessary to do any kind of syscall tracing. To optimally avoid that would require checking the bitmap in the assembly code, and I tend to doubt that optimization would be worth its maintenance burden.) If the overhead inherent in engines doing their own filtering proves burdensome, then we could add new calls to the utrace core for setting certain special kinds of automatic filters like a syscall bitmap. But that is an optimization for later; if it proves desireable, it fits cleanly as a compatible extension of the current interface.
The latter feature, of deciding on a syscall-entry event whether ot not to see the next syscall-exit event, is already easy to implement in an engine and about as optimally as any specialized utrace core addition might be. An engine's report_syscall_entry callback can store a flag that is checked by its report_syscall_exit callback, and that is pretty simple. If you don't want to use a flag bit of your own somewhere (engine->data or a data structure whose pointer you store there), there is a second way that is pretty straightforward as well. Each syscall-exit event is preceded by a syscall-entry event (except when you've just attached to a threa already in the middle of a syscall). So your report_syscall_entry callback call utrace_set_flags every time to change the set of events of interest to include or exclude the syscall-exit event, knowing each new setting will only affect the immediate corresponding syscall exit and you'll be called again to decide afresh for the next syscall.
Quoted 3 lines
Renzo Davoli also posted a patch to allow "batching" of ptrace requests and Systemptap really needs this, too. AFAICT this can be done by writing a custom engine.
That's the idea. Batching is one of several obvious parts of a decent user-level interface (in contrast to the whole ptrace model). The purpose of the utrace layer is to make it tractable to go and write these things, and to experiment with many of them, small and large.
Quoted 1 line
And BTW patches 1 and 2 never made it to the list.
The patches are rather large and might have been discarded by the mailing list software because of that. Having noticed the 40k limit on mailed patches in Documentation/SubmittingPatches, perhaps I shouldn't be posting them directly.
Quoted 3 lines
The ones on your server (http://redhat.com/~roland/utrace/) don't apply cleanly due to whitespace damage but that can be fixed by stripping trailing whitespace from the kernel files patch(1) complains about.
Sorry about that. (You can also apply them fine with patch -l, which is why I managed not to notice the problem myself the first time around.) I've fixed the way I generate the patches, and I've verified that the new patches now my site do apply to 2.6.17.
Thanks,
Roland
Quoted 11 lines · Frank Ch. Eigler
Indeed, I like what I see (at least those parts I understand) in roland's utrace code. One missed opportunity bit appears to be any new support for something like per-thread breakpoints.
If I correctly understand how gdb etc. work, a hit breakpoint involves stoppage of all other threads of a process, then the breakpoint instruction is replaced by the original one, then the thread is single-stepped, then the breakpoint is put back, then finally all threads are resumed. Could utrace API provide short-lived per-thread page copies to execute the single-stepped original instruction out of, and avoid stopping & resuming all other threads?
FYI: I talked with Roland about this a while ago, and got the impression that he was interested in implementing it, but wanted to get utrace going first. It doesn't really relate to utrace; but it would need a new interface however it was implemented.
GDB really does crave this feature.
Signature · 2 lines
Daniel Jacobowitz CodeSourcery