addr2line performance
Hi.
https://bugzilla.kernel.org/show_bug.cgi?id=218996#c1 mentions "there are proposals to move this logic to being from a library again. I think the addr2line processing attempted is valiant but ultimately using a library will be best" (this was when LLVM addr2line support was already present),
Obviously the test patch mentioned in the above bug doesn't address the underlying performance issue, only better classifying a timeout error.
We have a customer bug complaining about the performance of addr2line. I'm seeing cases where it's taking over 20 seconds wall clock to resolve certain addresses which obviously causes the io_timer to trip but the performance of the success path isn't great either.
Obviously BFD isn't an option and (for various reasons) we can't use LLVM for this specific distro so I'm looking at the performance of addr2line.
Can someone point me to the prior "using a library" discussion referenced in the above bug or whatever the current thinking is. I couldn't find anything searching the archive.
Thanks in advance!
Tony
Signature · 2 lines
Tony Jones SUSE Kernel Performance Team
Quoted 18 lines · Tony Jones
Hi.
https://bugzilla.kernel.org/show_bug.cgi?id=218996#c1 mentions "there are proposals to move this logic to being from a library again. I think the addr2line processing attempted is valiant but ultimately using a library will be best" (this was when LLVM addr2line support was already present),
Obviously the test patch mentioned in the above bug doesn't address the underlying performance issue, only better classifying a timeout error.
We have a customer bug complaining about the performance of addr2line. I'm seeing cases where it's taking over 20 seconds wall clock to resolve certain addresses which obviously causes the
io_timerto trip but the performance of the success path isn't great either.
Quoted 7 lines
Obviously BFD isn't an option and (for various reasons) we can't use LLVM for this specific distro so I'm looking at the performance of addr2line.
Can someone point me to the prior "using a library" discussion referenced in the above bug or whatever the current thinking is. I couldn't find anything searching the archive.
I'm not sure there is an explicit discussion rather than discussion on individual patch sets. It is something that comes up when fixing the text parsing of addr2line, something that is particularly problematic because of inlining. It comes up wrt removing non-distributable stuff, most recently in:
https://lore.kernel.org/lkml/[email protected]/
Wrt the bug, the --addr2line option for perf script is merged:
https://lore.kernel.org/r/[email protected]
There has also been clean up into how the addr2line implementation is picked, see:
https://web.git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git/tree/tools/perf/util/srcline.c?h=tmp.perf-tools-next#n117
which came from:
https://lore.kernel.org/lkml/[email protected]/
Wrt dependencies, I have been pushing these patches that use dlopen for libLLVM mean installing libLLVM can be done separately from building perf against it:
https://lore.kernel.org/lkml/[email protected]/
So if addr2line is the problem, perhaps use the command line option to switch to the llvm version. If that's not fast enough, and you can't distribute linked against libLLVM, some feedback on the dlopen libLLVM changes would be appreciated!
Thanks,
Ian
Quoted 7 lines
Thanks in advance!
Tony
--
Tony Jones
SUSE Kernel Performance Team
Quoted 3 lines · Ian Rogers
I'm not sure there is an explicit discussion rather than discussion on individual patch sets.
Thanks for the reply. I thought the original comment in that BZ was talking about plans to remove the current forking of an "addr2line" process and communicating via pipe with some library implementation. Of course what that library would be if not BFD/LLVM was my question. Clearly this was a misread on my part.
I'd seen the patches related to dlopen'ing llvm, backporting this is a possibility since we don't want a rpm requires.
Cheers!
Tony
Signature · 2 lines
Tony Jones SUSE Kernel Performance Team
Quoted 6 lines · Tony Jones
Quoted 3 lines · Ian Rogers
I'm not sure there is an explicit discussion rather than discussion on individual patch sets.
Quoted 8 lines
Thanks for the reply. I thought the original comment in that BZ was talking about plans to remove the current forking of an "addr2line" process and communicating via pipe with some library implementation. Of course what that library would be if not BFD/LLVM was my question. Clearly this was a misread on my part.
I'd seen the patches related to dlopen'ing llvm, backporting this is a possibility since we don't want a rpm required.
That's great to know and motivates the dlopen work! I can resend a rebase of that series, would it be okay to add an Acked-by tag[1] to the series saying it'd be useful for SuSE ?
Thanks,
Ian
[1] https://www.kernel.org/doc/Documentation/process/submitting-patches.rst
Quoted 7 lines
Cheers!
Tony
--
Tony Jones
SUSE Kernel Performance Team
Hello,
Quoted 17 lines · Tony Jones
Hi.
https://bugzilla.kernel.org/show_bug.cgi?id=218996#c1 mentions "there are proposals to move this logic to being from a library again. I think the addr2line processing attempted is valiant but ultimately using a library will be best" (this was when LLVM addr2line support was already present),
Obviously the test patch mentioned in the above bug doesn't address the underlying performance issue, only better classifying a timeout error.
We have a customer bug complaining about the performance of addr2line. I'm seeing cases where it's taking over 20 seconds wall clock to resolve certain addresses which obviously causes the
io_timerto trip but the performance of the success path isn't great either.
Quoted 7 lines
Obviously BFD isn't an option and (for various reasons) we can't use LLVM for this specific distro so I'm looking at the performance of addr2line.
Can someone point me to the prior "using a library" discussion referenced in the above bug or whatever the current thinking is. I couldn't find anything searching the archive.
I think we can just use libdw. I tried it in the past but never finished. :(
Thanks,
Namhyung
Quoted 2 lines · Namhyung Kim
I think we can just use libdw. I tried it in the past but never finished. :(
This would be interesting.
I asked our toolchain folks if they were aware of libraries (other than libllvm and libbfd) that could replace the addr2line+pipe (and were license compatible) but I've not heard back from them yet.
I'm happy to look into implementing this if you lack the time. LMK.
Tony
Signature · 2 lines
Tony Jones SUSE Kernel Performance Team
Hello,
Quoted 5 lines · Tony Jones
Quoted 2 lines · Namhyung Kim
I think we can just use libdw. I tried it in the past but never finished. :(
Quoted 7 lines
This would be interesting.
I asked our toolchain folks if they were aware of libraries (other than libllvm and libbfd) that could replace the addr2line+pipe (and were license compatible) but I've not heard back from them yet.
I'm happy to look into implementing this if you lack the time. LMK.
Sure, it'd be great if you can work on it. I think it's basically dwarf_lineaddr() followed by dwarf_linesrc() and dwarf_lineno() but you also need to check inlined functions.
And probably you want to cache dbginfo of recent binaries to speed up addr2line processing for the same DSOs.
Thanks,
Namhyung