perf call stacks on 32bit ARM v7

Hello Jean, others.

Can someone please clarify the requirements for getting perf to properly unwind the call stack on 32bit ARM v7?

Looking at [1], it seems that I need either (a) frame pointers, or (b) DWARF debug information. Because (a) is often not available, and (b) is too large for small flash drives on embedded - how do I use perf with split debug information files? I.e. I have tries to record on the arm board using DWARF:

arm-v7$ perf record --call-graph dwarf ...

Then transferred the perf.data file over to my host machine. Perf archive said that no build-id's could be found, so I'm not using that. Instead, I try to ask perf to find the split debug packes using symfs:

x86-64$ perf report --symfs ... -g graph

But that does not work and I'm not seeing any backtraces. Stracing the report, I don't see it even trying to access files - how can I debug this and figure out what I'm supposed to be using?

Hi Milian, Wangnan,

I am pretty busy this week. Let me check and come back to you next week.

PS: did you check the detailed instructions from the Linaro wiki pages? These are linked from the Fosdem presentation slides.

Quoted 7 lines · Milian Wolff

Quoted 6 lines

Hello Jean, others.

Can someone please clarify the requirements for getting perf to properly unwind the call stack on 32bit ARM v7?

Looking at [1], it seems that I need either (a) frame pointers, or (b)

DWARF

Quoted 1 line

Quoted 1 line

debug information. Because (a) is often not available, and (b) is too

large

Quoted 2 lines

Quoted 2 lines

for small flash drives on embedded - how do I use perf with split debug information files? I.e. I have tries to record on the arm board using

DWARF:

Quoted 3 lines

Quoted 3 lines

arm-v7$ perf record --call-graph dwarf ...

Then transferred the perf.data file over to my host machine. Perf

archive said

Quoted 1 line

Quoted 1 line

that no build-id's could be found, so I'm not using that. Instead, I try

to

Quoted 5 lines

Quoted 5 lines

ask perf to find the split debug packes using symfs:

x86-64$ perf report --symfs ... -g graph

But that does not work and I'm not seeing any backtraces. Stracing the

report,

Quoted 1 line

Quoted 1 line

I don't see it even trying to access files - how can I debug this and

figure

Quoted 17 lines

Quoted 1 line

out what I'm supposed to be using?

Unfortunaetly, currently perf only supports corss decoding dwarf for x86_64, x86_32 and arm64. ARM32 is not on the list.

Please see:

http://www.spinics.net/lists/kernel/msg2266293.html

and

tools/perf/util/unwind-libunwind.c

I think adding ARM32 support should not be very hard on perf side. Add He Kuang to the CC list, he is the author of this patch set.

Thank you.

Quoted 3 lines · On Donnerstag, 29. September 2016 09:39:51 CEST Jean Pihet wrote:

Hi Milian, Wangnan,

I am pretty busy this week. Let me check and come back to you next week.

Sure, take your time - much appreciated!

Quoted 2 lines

PS: did you check the detailed instructions from the Linaro wiki pages? These are linked from the Fosdem presentation slides.

You mean this page, right:

https://wiki.linaro.org/KenWerner/Sandbox/libunwind

That does not mention perf at all, and only talks about libunwind. Note how your talk, and the wiki, say .exidx unwinding using libunwind works. But your slides also explicitly say that this mechanism is not supported by perf, yet. I would like to know the reason for that, or whether this is outdated information.

Thanks!

Quoted 34 lines · Wangnan (F)

Quoted 7 lines · Milian Wolff

Quoted 6 lines

Hello Jean, others.

Can someone please clarify the requirements for getting perf to properly unwind the call stack on 32bit ARM v7?

Looking at [1], it seems that I need either (a) frame pointers, or (b)

DWARF

Quoted 1 line

Quoted 1 line

debug information. Because (a) is often not available, and (b) is too

large

Quoted 2 lines

Quoted 2 lines

for small flash drives on embedded - how do I use perf with split debug information files? I.e. I have tries to record on the arm board using

DWARF:

Quoted 3 lines

Quoted 3 lines

arm-v7$ perf record --call-graph dwarf ...

Then transferred the perf.data file over to my host machine. Perf

archive said

Quoted 1 line

Quoted 1 line

that no build-id's could be found, so I'm not using that. Instead, I try

to

Quoted 4 lines

Quoted 4 lines

ask perf to find the split debug packes using symfs:

x86-64$ perf report --symfs ... -g graph

Quoted 25 lines

Quoted 1 line

Quoted 1 line

But that does not work and I'm not seeing any backtraces. Stracing the

report,

Quoted 1 line

Quoted 1 line

I don't see it even trying to access files - how can I debug this and

figure

Quoted 17 lines

Quoted 1 line

out what I'm supposed to be using?

Unfortunaetly, currently perf only supports corss decoding dwarf for x86_64, x86_32 and arm64. ARM32 is not on the list.

Please see:

http://www.spinics.net/lists/kernel/msg2266293.html

and

tools/perf/util/unwind-libunwind.c

I think adding ARM32 support should not be very hard on perf side. Add He Kuang to the CC list, he is the author of this patch set.

Thank you.

--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

Hi,

On Thu, Sep 29, 2016 at 12:33 PM, Milian Wolff <[email protected]>

Quoted 5 lines · On Donnerstag, 29. September 2016 09:39:51 CEST Jean Pihet wrote:

Quoted 3 lines

Hi Milian, Wangnan,

I am pretty busy this week. Let me check and come back to you next week.

Quoted 8 lines

Sure, take your time - much appreciated!

Quoted 2 lines

PS: did you check the detailed instructions from the Linaro wiki pages? These are linked from the Fosdem presentation slides.

You mean this page, right:

https://wiki.linaro.org/KenWerner/Sandbox/libunwind

I mean
https://wiki.linaro.org/LEG/Engineering/TOOLS/perf-callstack-unwinding
where the details about installation, compilation etc. are found.

In short, DWARF unwinding on ARMv7 should work out of the box. You need libunwind, the correct kernel config options and perf installed. As you are mentioning the downside is the size of the generated trace which can be a limitation on embedded systems. Some parameters can be used to control the size of the generated trace (-F).

Quoted 6 lines

That does not mention perf at all, and only talks about libunwind. Note how
your talk, and the wiki, say .exidx unwinding using libunwind works. But your slides also explicitly say that this mechanism is not supported by perf, yet. I would like to know the reason for that, or whether this is outdated information.

I do not recall the details but .exidx is not supported on ARM, because of the compiler that does not generate the info in the binaries (you can check the ELF sections for it)

or the kernel does not unwind the info.

Quoted 1 line

Thanks!

I hope this helps!

Quoted 3 lines · On Freitag, 30. September 2016 09:32:44 CEST Jean Pihet wrote:

Hi,

On Thu, Sep 29, 2016 at 12:33 PM, Milian Wolff <[email protected]>

Quoted 18 lines · wrote:

Quoted 5 lines · On Donnerstag, 29. September 2016 09:39:51 CEST Jean Pihet wrote:

Quoted 3 lines

Hi Milian, Wangnan,

I am pretty busy this week. Let me check and come back to you next week.

Quoted 8 lines

Sure, take your time - much appreciated!

Quoted 2 lines

PS: did you check the detailed instructions from the Linaro wiki pages? These are linked from the Fosdem presentation slides.

You mean this page, right:

https://wiki.linaro.org/KenWerner/Sandbox/libunwind

I mean
https://wiki.linaro.org/LEG/Engineering/TOOLS/perf-callstack-unwinding
where the details about installation, compilation etc. are found.

Quoted 5 lines

In short, DWARF unwinding on ARMv7 should work out of the box. You need libunwind, the correct kernel config options and perf installed. As you are mentioning the downside is the size of the generated trace which can be a limitation on embedded systems. Some parameters can be used to control the size of the generated trace (-F).

No, I mean the size of the application code when debug symbols are added:

tmp$ du -hs libQt5Core.so.5.7.1
63M libQt5Core.so.5.7.1
tmp$ strip libQt5Core.so.5.7.1
tmp$ du -hs libQt5Core.so.5.7.1
4.8M libQt5Core.so.5.7.1

And this is just one library involved of many in the applications I have to deal with regularly. I just had a case where the target platform had about 50MB storage space free. I resorted to connecting an USB drive to load the application code with debug symbols from. But this changes the performance behavior of the application, as the code is not loaded anymore from the often excruciatingly slow on-board flash drive.

Quoted 11 lines

Quoted 7 lines

That does not mention perf at all, and only talks about libunwind. Note how
your talk, and the wiki, say .exidx unwinding using libunwind works. But your slides also explicitly say that this mechanism is not supported by perf, yet. I would like to know the reason for that, or whether this is outdated information.

I do not recall the details but .exidx is not supported on ARM, because of the compiler that does not generate the info in the binaries (you can check the ELF sections for it)

If I compile with -funwind-tables, then I do get the .exidx section, see this link I posted earlier:

https://wiki.linaro.org/KenWerner/Sandbox/libunwind

Quoted 1 line

or the kernel does not unwind the info.

That sounds more like the culprit - can someone confirm, that:

- for frame-pointer based unwinding, perf unwinds the stacks in kernel space
- whereas dwarf-based unwinding copies the stack together with the samples into the perf.data, and does the unwinding later on

So one would need to either

- add .exidx unwinding support into the kernel
- copy whatever is required into perf.data and do the unwinding later on can someone clarify what is actually required to do that?

Quoted 4 lines

Quoted 2 lines

Thanks!

I hope this helps!

Cheers

--
Milian Wolff | [email protected] | Software Engineer KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts

Quoted 40 lines · Milian Wolff

Quoted 3 lines · On Freitag, 30. September 2016 09:32:44 CEST Jean Pihet wrote:

Hi,

On Thu, Sep 29, 2016 at 12:33 PM, Milian Wolff <[email protected]>

Quoted 20 lines · wrote:

Quoted 8 lines · On Donnerstag, 29. September 2016 09:39:51 CEST Jean Pihet wrote:

Quoted 4 lines

Hi Milian, Wangnan,

I am pretty busy this week. Let me check and come back to you next week.

Sure, take your time - much appreciated!

Quoted 7 lines

Quoted 3 lines

PS: did you check the detailed instructions from the Linaro wiki pages?
These are linked from the Fosdem presentation slides.

You mean this page, right:

https://wiki.linaro.org/KenWerner/Sandbox/libunwind

I mean
https://wiki.linaro.org/LEG/Engineering/TOOLS/perf-callstack-unwinding
where the details about installation, compilation etc. are found.

Quoted 5 lines

In short, DWARF unwinding on ARMv7 should work out of the box. You need libunwind, the correct kernel config options and perf installed. As you are mentioning the downside is the size of the generated trace which can be a limitation on embedded systems. Some parameters can be used to control the size of the generated trace (-F).

No, I mean the size of the application code when debug symbols are added:

tmp$ du -hs libQt5Core.so.5.7.1
63M libQt5Core.so.5.7.1
tmp$ strip libQt5Core.so.5.7.1
tmp$ du -hs libQt5Core.so.5.7.1
4.8M libQt5Core.so.5.7.1

Quoted 7 lines

And this is just one library involved of many in the applications I have to
deal with regularly. I just had a case where the target platform had about 50MB storage space free. I resorted to connecting an USB drive to load the application code with debug symbols from. But this changes the performance behavior of the application, as the code is not loaded anymore from the often excruciatingly slow on-board flash drive.

That is correct. You need some space for the debug symbols and also for the trace generation (I am using the RAM for speed reasons).

Quoted 17 lines

Quoted 11 lines

Quoted 7 lines

That does not mention perf at all, and only talks about libunwind. Note how
your talk, and the wiki, say .exidx unwinding using libunwind works. But
your slides also explicitly say that this mechanism is not supported by perf, yet. I would like to know the reason for that, or whether this is outdated information.

I do not recall the details but .exidx is not supported on ARM, because of the compiler that does not generate the info in the binaries (you can check the ELF sections for it)

If I compile with -funwind-tables, then I do get the .exidx section, see this link I posted earlier:

https://wiki.linaro.org/KenWerner/Sandbox/libunwind

Quoted 7 lines

Quoted 1 line

or the kernel does not unwind the info.

That sounds more like the culprit - can someone confirm, that:

- for frame-pointer based unwinding, perf unwinds the stacks in kernel space - whereas dwarf-based unwinding copies the stack together with the samples into the perf.data, and does the unwinding later on

Correct! This dwarf unwinding method generates a lot of trace data. Also the real time trace decoding is not possible.

Quoted 5 lines

So one would need to either

- add .exidx unwinding support into the kernel
- copy whatever is required into perf.data and do the unwinding later on

Quoted 1 line

can someone clarify what is actually required to do that?

That requires a more extensive analysis.

As you know I am a freelance consultant and could work on that topic, as I did for Linaro before.