Sampling of non-C-like stacks with eBPF and perf_events?
Hi all,
I have recently been exploring the possibility of using a BPF_PROG_TYPE_PERF_EVENT program to implement stack sampling for languages which do not use the platform's %sp for their stack pointer (in my case, GHC/Haskell [1], which on x86-64 uses %rbp for its stack pointer). Specifically, the idea is to use a sampling perf_events session with an eBPF overflow handler which locates the currently-running thread's stack and records it in the sample ringbuffer (see [2] for my current attempt). At this point I only care about user-space samples.
However, I quickly ran up against the fact that perf_event's stack sampling logic (namely perf_output_sample_ustack) is called from an IRQ context. This appears to preclude use of a sleepable BPF program, which would be necessary to use bpf_copy_from_user. Indeed, the fact that the usual stack sampling logic uses copy_from_user_inatomic rather than copy_from_user suggests that this isn't a safe context for sleeping.
So, I'm at this point a bit unclear on how to proceed. I can see a few possible directions forward, although none are particularly enticing:
* Add a bpf_copy_from_user_atomic helper, which can be called from a non-sleepable context like a perf_events overflow handler. This would take the same set_fs() and pagefault_disable() precautions as perf_output_sample_ustack to ensure that the access is safe and aborts on fault.
* Introduce a new BPF program type, BPF_PROG_TYPE_PERF_EVENT_STACK_LOCATOR, which can be invoked by perf_output_sample_ustack to locate the stack to be sampled.
Do either of these ideas sound upstreamable? Perhaps there are other ideas on how to attack this general problem? I do not believe Haskell is alone in its struggle with the current inflexibility of stack sampling; the JVM introduced framepointer support specifically to allow callgraph sampling; however, dedicating a register and code to this seems like an unfortunate compromise, especially on x86-64 where registers are already fairly precious.
Any thoughts or suggestions would be greatly appreciated.
Cheers,
- Ben
[1] https://www.haskell.org/ghc/
[2] https://gitlab.haskell.org/bgamari/hs-bpf-prof/
Quoted 41 lines · Ben Gamari
Hi all,
I have recently been exploring the possibility of using a
BPF_PROG_TYPE_PERF_EVENTprogram to implement stack sampling for languages which do not use the platform's %sp for their stack pointer (in my case, GHC/Haskell [1], which on x86-64 uses %rbp for its stack pointer). Specifically, the idea is to use a samplingperf_eventssession with an eBPF overflow handler which locates the currently-running thread's stack and records it in the sample ringbuffer (see [2] for my current attempt). At this point I only care about user-space samples.However, I quickly ran up against the fact that perf_event's stack sampling logic (namely
perf_output_sample_ustack) is called from an IRQ context. This appears to preclude use of a sleepable BPF program, which would be necessary to usebpf_copy_from_user. Indeed, the fact that the usual stack sampling logic usescopy_from_user_inatomicrather thancopy_from_usersuggests that this isn't a safe context for sleeping.So, I'm at this point a bit unclear on how to proceed. I can see a few possible directions forward, although none are particularly enticing:
* Add a
bpf_copy_from_user_atomichelper, which can be called from a non-sleepable context like aperf_eventsoverflow handler. This would take the sameset_fs()andpagefault_disable()precautions asperf_output_sample_ustackto ensure that the access is safe and aborts on fault.* Introduce a new BPF program type,
BPF_PROG_TYPE_PERF_EVENT_STACK_LOCATOR, which can be invoked byperf_output_sample_ustackto locate the stack to be sampled.Do either of these ideas sound upstreamable? Perhaps there are other ideas on how to attack this general problem? I do not believe Haskell is alone in its struggle with the current inflexibility of stack sampling; the JVM introduced framepointer support specifically to allow callgraph sampling; however, dedicating a register and code to this seems like an unfortunate compromise, especially on x86-64 where registers are already fairly precious.
Quoted 1 line
Any thoughts or suggestions would be greatly appreciated.
Hi Ben,
if you're sampling the stack trace of the current process there is no need for copy_from_user and sleepable. The memory with the stack trace unlikely was paged out. So normal bpf_probe_read_user() will work fine.
This approach was used to implement 'pyperf'. It walks python stack traces:
https://github.com/iovisor/bcc/tree/master/examples/cpp/pyperf
What you're trying to do for haskel sounds very similar.
Hi Alexei and Ben,
This sounds awesome! Somewhat off-topic, I wonder if we could include the pyperf and ghc support in regular perf? I think there is an assumption that these languages are a minority concern, but I think everyone would benefit from being packaged with perf, being kept in sync with how the APIs evolve, code reuse, etc.
Thanks,
Ian
Quoted 56 lines · Alexei Starovoitov
Quoted 41 lines · Ben Gamari
Hi all,
I have recently been exploring the possibility of using a
BPF_PROG_TYPE_PERF_EVENTprogram to implement stack sampling for languages which do not use the platform's %sp for their stack pointer (in my case, GHC/Haskell [1], which on x86-64 uses %rbp for its stack pointer). Specifically, the idea is to use a samplingperf_eventssession with an eBPF overflow handler which locates the currently-running thread's stack and records it in the sample ringbuffer (see [2] for my current attempt). At this point I only care about user-space samples.However, I quickly ran up against the fact that perf_event's stack sampling logic (namely
perf_output_sample_ustack) is called from an IRQ context. This appears to preclude use of a sleepable BPF program, which would be necessary to usebpf_copy_from_user. Indeed, the fact that the usual stack sampling logic usescopy_from_user_inatomicrather thancopy_from_usersuggests that this isn't a safe context for sleeping.So, I'm at this point a bit unclear on how to proceed. I can see a few possible directions forward, although none are particularly enticing:
* Add a
bpf_copy_from_user_atomichelper, which can be called from a non-sleepable context like aperf_eventsoverflow handler. This would take the sameset_fs()andpagefault_disable()precautions asperf_output_sample_ustackto ensure that the access is safe and aborts on fault.* Introduce a new BPF program type,
BPF_PROG_TYPE_PERF_EVENT_STACK_LOCATOR, which can be invoked byperf_output_sample_ustackto locate the stack to be sampled.Do either of these ideas sound upstreamable? Perhaps there are other ideas on how to attack this general problem? I do not believe Haskell is alone in its struggle with the current inflexibility of stack sampling; the JVM introduced framepointer support specifically to allow callgraph sampling; however, dedicating a register and code to this seems like an unfortunate compromise, especially on x86-64 where registers are already fairly precious.
Quoted 1 line
Any thoughts or suggestions would be greatly appreciated.
Hi Ben,
if you're sampling the stack trace of the current process there is no need for
copy_from_userand sleepable. The memory with the stack trace unlikely was paged out. So normalbpf_probe_read_user()will work fine.This approach was used to implement 'pyperf'. It walks python stack traces:
https://github.com/iovisor/bcc/tree/master/examples/cpp/pyperf
What you're trying to do for haskel sounds very similar.
Just FYI I have plans to make a BPF variant of Austin
(https://github.com/P403n1x87/austin) as an alternative to the ptrace-based austinp variant
(https://github.com/P403n1x87/austin#native-frame-stack). This will then do what pyperf does, plus all of Austin's other features, like GC sampling etc...
Quoted 70 lines · Ian Rogers
Hi Alexei and Ben,
This sounds awesome! Somewhat off-topic, I wonder if we could include the pyperf and ghc support in regular perf? I think there is an assumption that these languages are a minority concern, but I think everyone would benefit from being packaged with perf, being kept in sync with how the APIs evolve, code reuse, etc.
Thanks,
IanQuoted 56 lines · Alexei Starovoitov
Quoted 41 lines · Ben Gamari
Hi all,
I have recently been exploring the possibility of using a
BPF_PROG_TYPE_PERF_EVENTprogram to implement stack sampling for languages which do not use the platform's %sp for their stack pointer (in my case, GHC/Haskell [1], which on x86-64 uses %rbp for its stack pointer). Specifically, the idea is to use a samplingperf_eventssession with an eBPF overflow handler which locates the currently-running thread's stack and records it in the sample ringbuffer (see [2] for my current attempt). At this point I only care about user-space samples.However, I quickly ran up against the fact that perf_event's stack sampling logic (namely
perf_output_sample_ustack) is called from an IRQ context. This appears to preclude use of a sleepable BPF program, which would be necessary to usebpf_copy_from_user. Indeed, the fact that the usual stack sampling logic usescopy_from_user_inatomicrather thancopy_from_usersuggests that this isn't a safe context for sleeping.So, I'm at this point a bit unclear on how to proceed. I can see a few possible directions forward, although none are particularly enticing:
* Add a
bpf_copy_from_user_atomichelper, which can be called from a non-sleepable context like aperf_eventsoverflow handler. This would take the sameset_fs()andpagefault_disable()precautions asperf_output_sample_ustackto ensure that the access is safe and aborts on fault.* Introduce a new BPF program type,
BPF_PROG_TYPE_PERF_EVENT_STACK_LOCATOR, which can be invoked byperf_output_sample_ustackto locate the stack to be sampled.Do either of these ideas sound upstreamable? Perhaps there are other ideas on how to attack this general problem? I do not believe Haskell is alone in its struggle with the current inflexibility of stack sampling; the JVM introduced framepointer support specifically to allow callgraph sampling; however, dedicating a register and code to this seems like an unfortunate compromise, especially on x86-64 where registers are already fairly precious.
Quoted 1 line
Any thoughts or suggestions would be greatly appreciated.
Hi Ben,
if you're sampling the stack trace of the current process there is no need for
copy_from_userand sleepable. The memory with the stack trace unlikely was paged out. So normalbpf_probe_read_user()will work fine.This approach was used to implement 'pyperf'. It walks python stack traces:
https://github.com/iovisor/bcc/tree/master/examples/cpp/pyperf
What you're trying to do for haskel sounds very similar.
Signature · 6 lines
"Egli è scritto in lingua matematica, e i caratteri son triangoli, cerchi, ed altre figure geometriche, senza i quali mezzi è impossibile a intenderne umanamente parola; senza questi è un aggirarsi vanamente per un oscuro laberinto." -- G. Galilei, Il saggiatore.