Skip to contentSkip to conversation

perf sampling library for LTTng-UST

RFC Staffan Tjernstrom 2 participants 2 messages Nov 17, 2016 – Nov 18, 2016 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Whilst I'm not competent to comment on the implementation, I really like the functionality - it's going to be a life-saver for me.

The need to sample perf counters from user space programs is huge in my world.

Quoted 27 lines

Hello!

I did a small shared library for profiling code using perf sampling and LTTng-UST:

https://github.com/giraldeau/lttng-ust/tree/sampling/liblttng-ust-sampling

It works by preloading the library when executing a program. Inside the library constructor, a perf counter is created and samples are saved inside the SIGIO handler using an LTTng-UST tracepoint. The call stack is obtained using libunwind, and thus it works even without frame pointers and with unmodified executables.

Preliminary overhead measure with the default sampling period of 1E4 for cpu cycles is about 8.5%, or 2.2us per event. On my machine, about 38k samples per second are generated. This figure is obtained when compiling libunwind without signal re-entrance support.

Genevieve Bastien did a nice view in TraceCompass to load this trace and display the corresponding call graph view.

http://secretaire.dorsal.polymtl.ca/~gbastien/screenshots/lttng_sampling_callstack.png

The counter is hard coded now, but it's just a prototype to demonstrate the concept. I would find it very cool to see such feature in LTTng. What do you think?

Quoted 8 lines

Cheers,

Francis
_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

#2 lore raw
In reply to a message not in the archive
Quoted 22 lines · CET Francis Giraldeau

Hello!

I did a small shared library for profiling code using perf sampling and LTTng-UST:

https://github.com/giraldeau/lttng-ust/tree/sampling/liblttng-ust-sampling

It works by preloading the library when executing a program. Inside the library constructor, a perf counter is created and samples are saved inside the SIGIO handler using an LTTng-UST tracepoint. The call stack is obtained using libunwind, and thus it works even without frame pointers and with unmodified executables.

Preliminary overhead measure with the default sampling period of 1E4 for cpu cycles is about 8.5%, or 2.2us per event. On my machine, about 38k samples per second are generated. This figure is obtained when compiling libunwind without signal re-entrance support.

Genevieve Bastien did a nice view in TraceCompass to load this trace and display the corresponding call graph view.

Quoted 6 lines

http://secretaire.dorsal.polymtl.ca/~gbastien/screenshots/lttng_sampling_ca
llstack.png

The counter is hard coded now, but it's just a prototype to demonstrate the concept. I would find it very cool to see such feature in LTTng. What do you think?

The image times out for me, i.e. I cannot load it.

I very much like the idea, as it would easily allow to combine LTTNG and perf. The performance impact is pretty bad though. Can't this be done differently, such that you reuse whatever perf record uses internally? That one has a far smaller overhead, even when using libunwind/libdw for DWARF based unwinding (--call-graph dwarf).

In general, I see perf being really good for profiling, whereas LTTNG seems to be really good for tracing. Both somehow support the other side, but not as nicely. I would welcome if the two projects would start to collaborate more deeply.

From my POV, I'd like to:

- trace most of the kernel stuff, most notably scheduler, page faults, syscalls, ...
- trace all UST points
- sample CPU

The latter two usually only for a single process, but sometimes multiple ones. LTTNG gives me the first two points, and perf gives me the latter.

Bye

Signature · 4 lines
Milian Wolff | [email protected] | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

_______________________________________________
lttng-dev mailing list
[email protected]
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev