Skip to contentSkip to conversation

Full syscall argument decode in "perf trace"

RFC Denys Vlasenko 2 participants 2 messages Sep 17, 2013

lore
Share & tools
lore
#0 lore raw

Hi,

I'm trying to figure out how to extend "perf trace".

Currently, it shows syscall names and arguments, and only them. Meaning that syscalls such as open(2) are shown as:

    open(filename: 140736118412184, flags: 0, mode: 140736118403776) = 3

The problem is, of course, that user wants to see the filename per se, not the address of its first byte.

To improve that, we need to fetch the pointed-to data.
There are two approaches to this: extending "raw_syscalls:sys_{enter,exit}" tracepoint so that it returns this data, or selectively stopping the traced process when it reaches the thacepoint.

First solution is attractive performance-wise, but requires a lot of new code: *ALL* syscalls will need to know which arguments are pointers, how large their pointed-to data structures are, and (remember readv and friends!) some of pointed-to structures themselves contain pointers which reference even more data.

If we want to go this way, do we want to encode all this knowledge in kernel? If yes, how? If no, in what form userspace (perf trace) would configure the tracepoint wrt which syscalls' arguments to copy to trace buffer?

The second solution is to pause traced process, let "perf trace" to fetch its data (e.g. via process_vm_readv(2)) and unpause it.

The dead-simple approach ("pause on every sys_{enter,exit}") would be no faster than strace. To make any sense, as a minimum the pausing needs to be conditional: there is no need to stop on syscalls which do not have indirect data (e.g. close(2), dup2(2)...).

Optimizing further, we can choose a few typical syscalls such as [f]stat(2), write(2), and apply solution #1 ("dump data to trace buffer and don't pause") to them.
For example, fstat(fd, &statbuf) does not need to stop on sys_enter at all, and needs to only copy the fixed number of bytes of statbuf to trace buffer on exit to avoid the need to pause.

If we want to go this way, how do you guys think this should be implemented?

IIUC tracepoints weren't meant to be able to influence execution,
the "pause the current process when tracepoint
is triggered" is a new feature. Does it look acceptable?
How to go about implementing it? Something like an ad-hoc extension field in struct perf_event_attr to enable it?
Specifically, a new field or flag can enable this:
perf_event_open -> perf_event_alloc(... overflow_handler_which_conditionally_stops_current ...)

The "pausing", what it should be, exactly? In the ancient times, strace chose to simply use SIGSTOP for similar needs, and it ended up interfering with tracing real SIGSTOPs. I guess we don't want to repeat that. Then, how? More specifically: when "perf trace" will read trace buffer and see "process FOO paused in sys_exit from readv", how it should kick process FOO to unpause it?

**end of brain dump**

Comments? Suggestions?

#1 lore raw

Em Tue, Sep 17, 2013 at 05:10:55PM +0200, Denys Vlasenko escreveu:

Quoted 7 lines

I'm trying to figure out how to extend "perf trace".

Currently, it shows syscall names and arguments, and only them. Meaning that syscalls such as open(2) are shown as:

    open(filename: 140736118412184, flags: 0, mode: 140736118403776) = 3
Quoted 7 lines

The problem is, of course, that user wants to see the filename per se, not the address of its first byte.

To improve that, we need to fetch the pointed-to data.
There are two approaches to this: extending "raw_syscalls:sys_{enter,exit}" tracepoint so that it returns this data, or selectively stopping the traced process when it reaches the thacepoint.

We don't want to stop the process at all, this is one of the major advantages of 'perf trace' over 'strace'.

Look at the tmp.perf/trace2 branch in my git repo, tglx and Ingo added a tracepoint to vfs_getname to use that.

Quoted 5 lines

First solution is attractive performance-wise, but requires a lot of new code: *ALL* syscalls will need to know which arguments are pointers, how large their pointed-to data structures are, and (remember readv and friends!) some of pointed-to structures themselves contain pointers which reference even more data.

Well, we can look at DWARF to get the function signatures, types, librarize 'perf probe' and insert probes in the syscalls we want decoding.

That for the cases where we don't have a tracepoint or when adding a new tracepoint is not an option.

And this all with what we have in the kernel right now.

Also for 'perf trace' look at my perf/core branch, where we have more syscall arg beautifiers and the machinery that is getting in place to allow that.

Longer term we could have something like dtrace's CTF to have a more compact type only ELF section that always go with the kernel, like we have CFI in binaries these days.

- Arnaldo