State of "perf: Add a new sort order: SORT_INCLUSIVE"
Hi Namhyung,
Frederic Weisbecker and Arnaldo Carvalho de Melo told me on IRC that you were working to forward-port a patch that adds a new sort order to perf report, SORT_INCLUSIVE.
That will be useful for me and I was wondering if you are still working on that or if there is a newer version than v6:
http://thread.gmane.org/gmane.linux.kernel.perf.user/882
Maybe you have a newer version of it not sent to the list ? Or do you plan to work on it anytime soon ?
If there is any branch to test, please let me know :)
Thanks a lot,
Rodrigo
Hi Rodrigo,
Quoted 9 lines · Rodrigo Campos
Hi Namhyung,
Frederic Weisbecker and Arnaldo Carvalho de Melo told me on IRC that you were working to forward-port a patch that adds a new sort order to perf report,
SORT_INCLUSIVE.That will be useful for me and I was wondering if you are still working on that or if there is a newer version than v6:
Quoted 7 lines
http://thread.gmane.org/gmane.linux.kernel.perf.user/882
Maybe you have a newer version of it not sent to the list ? Or do you plan to work on it anytime soon ?
If there is any branch to test, please let me know :)
Yes I have a patch series for that. But it's not a new sort order but a new command line option --culumate. I don't think it should be a new sort order since it affects only how it counts period value on each sample not how samples are sorted/grouped.
It's about a year since I sent this series to list. I'll work on it and send it to the list soon. But before that I have to re-read what's the Frederic's concern - IIRC it's about consolidating code in perf report that does similar things on branch stack.
Frederic, can you remember what was the problem?
Anyway, You can find the series and discussion on the link below:
https://lkml.org/lkml/2012/9/13/81
Thanks,
Namhyung
Quoted 15 lines · Namhyung Kim
Hi Rodrigo,
Quoted 11 lines · Rodrigo Campos
That will be useful for me and I was wondering if you are still working on that or if there is a newer version than v6:
http://thread.gmane.org/gmane.linux.kernel.perf.user/882
Maybe you have a newer version of it not sent to the list ? Or do you plan to work on it anytime soon ?
If there is any branch to test, please let me know :)
Quoted 7 lines
Yes I have a patch series for that. But it's not a new sort order but a new command line option --culumate. I don't think it should be a new sort order since it affects only how it counts period value on each sample not how samples are sorted/grouped.
It's about a year since I sent this series to list. I'll work on it and send it to the list soon. But before that I have to re-read what's the
Great, thanks a lot!
Quoted 8 lines
Frederic's concern - IIRC it's about consolidating code in perf report that does similar things on branch stack.
Frederic, can you remember what was the problem?
Anyway, You can find the series and discussion on the link below:
I've read the cover letter for that series and probably because I don't know about perf internals I have a question: How will "--culumate" interact with "--sort=dso" for example ?
I mean, is it possible for that to show more than 100% ? (if you add all the 93.35% in your example in the cover letter, or something similar). Or "--culumate --sort=dso" will just group together all entries that have a dso in the call chain ?
Sorry if the question was too obvious if I knew about perf internals :S
Thanks a lot,
Rodrigo
Quoted 5 lines · Rodrigo Campos
Quoted 3 lines · Namhyung Kim
Anyway, You can find the series and discussion on the link below:
Quoted 8 lines
I've read the cover letter for that series and probably because I don't know about perf internals I have a question: How will "--culumate" interact with "--sort=dso" for example ?
I mean, is it possible for that to show more than 100% ? (if you add all the 93.35% in your example in the cover letter, or something similar). Or "--culumate --sort=dso" will just group together all entries that have a dso in the call chain ?
Hmm.. I think --cumulate option is only meaningful when sort order includes symbol. Maybe I can add support for --sort=dso case as well but not sure it's worth. Do you think it's really needed?
Quoted 2 lines
Sorry if the question was too obvious if I knew about perf internals :S
No need to sorry about it. :)
Thanks,
Namhyung
Quoted 10 lines · Namhyung Kim
Quoted 9 lines · Rodrigo Campos
Quoted 3 lines · Namhyung Kim
Anyway, You can find the series and discussion on the link below:
I've read the cover letter for that series and probably because I don't know about perf internals I have a question: How will "--culumate" interact with "--sort=dso" for example ?
Quoted 8 lines
Quoted 4 lines
I mean, is it possible for that to show more than 100% ? (if you add all the 93.35% in your example in the cover letter, or something similar). Or "--culumate --sort=dso" will just group together all entries that have a dso in the call chain ?
Hmm.. I think --cumulate option is only meaningful when sort order includes symbol. Maybe I can add support for --sort=dso case as well but not sure it's worth. Do you think it's really needed?
I don't know if it is *needed*, but that was what I need :)
I mean, what I was looking for is a way to know the percentage spent when some symbol in the call chain belongs to a DSO. Like group all samples that have a symbol of a DSO in the callchain, and know the percentage for that group. Something like the "inclusive" time spent inside a library.
Quoted 3 lines
Quoted 1 line
Sorry if the question was too obvious if I knew about perf internals :S
No need to sorry about it. :)
Thanks =)
Thanks again,
Rodrigo
Quoted 16 lines · Rodrigo Campos
Quoted 15 lines · Namhyung Kim
Quoted 13 lines · Rodrigo Campos
Quoted 3 lines · Namhyung Kim
Anyway, You can find the series and discussion on the link below:
https://lkml.org/lkml/2012/9/13/81I've read the cover letter for that series and probably because I don't know about perf internals I have a question: How will "--culumate" interact with "--sort=dso" for example ?
I mean, is it possible for that to show more than 100% ? (if you add all the 93.35% in your example in the cover letter, or something similar). Or "--culumate --sort=dso" will just group together all entries that have a dso in the call chain ?
Quoted 5 lines
Quoted 3 lines
Hmm.. I think --cumulate option is only meaningful when sort order includes symbol. Maybe I can add support for --sort=dso case as well but not sure it's worth. Do you think it's really needed?
I don't know if it is *needed*, but that was what I need :)
I suspect that users will find creative ways of using these options to solve real world problems and we shouldn't restrict usage any more than we need to to protect against obvious bugs/crashes.
Also, what's the reasoning for --cumulate not being an option under perf record -g ..,<order>?
In order to integrate perf record -b and --cumulate, we'll have to sort out the underlying infrastructure for processing callgraphs and branch stacks. I think the main roadblock here is that one is statistical and on many CPUs incomplete (only top N branches are reported).
Given that there are clear use cases in production involving complex callgraphs, I'm for getting this support in first and then reconciling the differences with perf record -b later.
-Arun
Hi Arun,
Quoted 21 lines · Arun Sharma
Quoted 20 lines · Rodrigo Campos
Quoted 18 lines · Namhyung Kim
Quoted 5 lines · Rodrigo Campos
Quoted 3 lines · Namhyung Kim
Anyway, You can find the series and discussion on the link below:
https://lkml.org/lkml/2012/9/13/81Quoted 8 lines
I've read the cover letter for that series and probably because I don't know about perf internals I have a question: How will "--culumate" interact with "--sort=dso" for example ?
I mean, is it possible for that to show more than 100% ? (if you add all the 93.35% in your example in the cover letter, or something similar). Or "--culumate --sort=dso" will just group together all entries that have a dso in the call chain ?
Hmm.. I think --cumulate option is only meaningful when sort order includes symbol. Maybe I can add support for --sort=dso case as well but not sure it's worth. Do you think it's really needed?
Quoted 8 lines
Quoted 1 line
I don't know if it is *needed*, but that was what I need :)
I suspect that users will find creative ways of using these options to solve real world problems and we shouldn't restrict usage any more than we need to to protect against obvious bugs/crashes.
Also, what's the reasoning for --cumulate not being an option under perf record -g ..,<order>?
Sorry, I cannot understand you. The 'perf record' just saves sample data (and callchains) from the ring-buffer. All the processing happens in 'perf report'. I can't see what you expect from the 'perf record --cumulate'. Am I missing something?
Quoted 10 lines
In order to integrate perf record -b and --cumulate, we'll have to sort out the underlying infrastructure for processing callgraphs and branch stacks. I think the main roadblock here is that one is statistical and on many CPUs incomplete (only top N branches are reported).
Given that there are clear use cases in production involving complex callgraphs, I'm for getting this support in first and then reconciling the differences with perf record -b later.
I think what Frederic said is that the code de-duplication of 'perf report' side. The branch stack and --cumulate are different - branch stack concentrates on the branch itself but --cumulate uses callchains to find parents and give some credit to them as side information.
Thanks,
Namhyung
On 10/28/13 8:11 PM, Namhyung Kim wrote:
Hey Namhyung:
Quoted 8 lines
Quoted 3 lines
Also, what's the reasoning for --cumulate not being an option under perf record -g ..,<order>?
Sorry, I cannot understand you. The 'perf record' just saves sample data (and callchains) from the ring-buffer. All the processing happens in 'perf report'. I can't see what you expect from the 'perf record --cumulate'. Am I missing something?
Yes - I meant to say perf report -g :)
Quoted 1 line
-g [type,min[,limit],order]
Specifically, along with callee, caller, we could have a third option. Or we could have a new type (graph, fractal, cumulative).
Quoted 8 lines
Quoted 3 lines
Given that there are clear use cases in production involving complex callgraphs, I'm for getting this support in first and then reconciling the differences with perf record -b later.
I think what Frederic said is that the code de-duplication of 'perf report' side. The branch stack and --cumulate are different - branch stack concentrates on the branch itself but --cumulate uses callchains to find parents and give some credit to them as side information.
Me too. I brought it up with Stephane at some point in the last year or so and there wasn't an obvious way to de-duplicate because of these differences.
-Arun
Quoted 8 lines · Arun Sharma
On 10/28/13 8:11 PM, Namhyung Kim wrote:
Hey Namhyung:
Quoted 4 lines
Quoted 3 lines
Also, what's the reasoning for --cumulate not being an option under perf record -g ..,<order>?
Quoted 6 lines
Quoted 4 lines
Sorry, I cannot understand you. The 'perf record' just saves sample data (and callchains) from the ring-buffer. All the processing happens in 'perf report'. I can't see what you expect from the 'perf record --cumulate'. Am I missing something?
Yes - I meant to say perf report -g :)
:)
Quoted 5 lines
Quoted 2 lines
-g [type,min[,limit],order]
Specifically, along with callee, caller, we could have a third option. Or we could have a new type (graph, fractal, cumulative).
That's also fine by me. But I added --cumulate since it's quite different from other callchain behaviors.
If we go with -g option, I'd like add it as a new type.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Given that there are clear use cases in production involving complex callgraphs, I'm for getting this support in first and then reconciling the differences with perf record -b later.
Quoted 8 lines
Quoted 4 lines
I think what Frederic said is that the code de-duplication of 'perf report' side. The branch stack and --cumulate are different - branch stack concentrates on the branch itself but --cumulate uses callchains to find parents and give some credit to them as side information.
Me too. I brought it up with Stephane at some point in the last year or so and there wasn't an obvious way to de-duplicate because of these differences.
Yeah, looking at the code, I can hardly find how I can do it. :-/
Thanks,
Namhyung
Quoted 24 lines · Arun Sharma
On 10/28/13 8:11 PM, Namhyung Kim wrote:
Hey Namhyung:
Quoted 8 lines
Quoted 3 lines
Also, what's the reasoning for --cumulate not being an option under perf record -g ..,<order>?
Sorry, I cannot understand you. The 'perf record' just saves sample data (and callchains) from the ring-buffer. All the processing happens in 'perf report'. I can't see what you expect from the 'perf record --cumulate'. Am I missing something?
Yes - I meant to say perf report -g :)
Quoted 1 line
-g [type,min[,limit],order]
Specifically, along with callee, caller, we could have a third option. Or we could have a new type (graph, fractal, cumulative).
Quoted 4 lines
Quoted 3 lines
Given that there are clear use cases in production involving complex callgraphs, I'm for getting this support in first and then reconciling the differences with perf record -b later.
Quoted 8 lines
Quoted 4 lines
I think what Frederic said is that the code de-duplication of 'perf report' side. The branch stack and --cumulate are different - branch stack concentrates on the branch itself but --cumulate uses callchains to find parents and give some credit to them as side information.
Me too. I brought it up with Stephane at some point in the last year or so and there wasn't an obvious way to de-duplicate because of these differences.
I agree that the interface is debatable. It could be -g ...,cumulative, expand -b, or whatever. But the backend is the same: perf_report__add_branch_hist_entry should be shared 80%.
2013/10/28 Namhyung Kim <[email protected]>:
Quoted 20 lines
Hi Rodrigo,
Quoted 9 lines · Rodrigo Campos
Hi Namhyung,
Frederic Weisbecker and Arnaldo Carvalho de Melo told me on IRC that you were working to forward-port a patch that adds a new sort order to perf report,
SORT_INCLUSIVE.That will be useful for me and I was wondering if you are still working on that or if there is a newer version than v6:
Quoted 7 lines
http://thread.gmane.org/gmane.linux.kernel.perf.user/882
Maybe you have a newer version of it not sent to the list ? Or do you plan to work on it anytime soon ?
If there is any branch to test, please let me know :)
Quoted 7 lines
Yes I have a patch series for that. But it's not a new sort order but a new command line option --culumate. I don't think it should be a new sort order since it affects only how it counts period value on each sample not how samples are sorted/grouped.
It's about a year since I sent this series to list. I'll work on it and send it to the list soon.
Thanks a lot!
Quoted 9 lines
But before that I have to re-read what's the
Frederic's concern - IIRC it's about consolidating code in perf report that does similar things on branch stack.Frederic, can you remember what was the problem?
Anyway, You can find the series and discussion on the link below:
Right so my concern was that it should be an extension of "perf report
- b" rather than an ad-hoc. Cumulative callchains is basically what perf report -b does but with callchains. And since callchains are branches, this should be a perfect fit.
Also I don't remember if perf report -b adds branches to the hist with a period of 1 of with the period of the referring event. I remember there was an issue a long time ago there. May be it was fixed. Anyway IMO the period of 1 doesn't make sense, branches should be added with the period of the event.
Thanks.