Profiling support?
Looking in the Makefile, I just find support for coverage reports using gcov. Whatever is there with "profile" in it seems to be for profile-based compilation rather than using gprof.
Now since I've managed to push most of the runtime for basic git-blame operation out of blame.c proper, it becomes important to figure out where most of the remaining runtime (a sizable part of that being system time) is being spent. Loop counts like that provided by gcov (or am I missing something here?) are not helpful for that, I think I rather need the kind of per-function breakdown that gprof provides.
Is there a reason there are no prewired recipes or advice for using gprof on git? Is there a way to get the work done, namely seeing the actual distribution of call times (rather than iterations) using gcov so that this is not necessary?
Signature · 1 line
David Kastrup
Quoted 12 lines · David Kastrup
Looking in the
Makefile, I just find support for coverage reports using gcov. Whatever is there with "profile" in it seems to be for profile-based compilation rather than using gprof.Now since I've managed to push most of the runtime for basic git-blame operation out of
blame.cproper, it becomes important to figure out where most of the remaining runtime (a sizable part of that being system time) is being spent. Loop counts like that provided by gcov (or am I missing something here?) are not helpful for that, I think I rather need the kind of per-function breakdown that gprof provides.
Quoted 4 lines
Is there a reason there are no prewired recipes or advice for using gprof on git? Is there a way to get the work done, namely seeing the actual distribution of call times (rather than iterations) using gcov so that this is not necessary?
Would perf help? No changes required, and almost no overhead, I think.
Signature · 1 line
Duy
Quoted 13 lines · Duy Nguyen
Quoted 12 lines · David Kastrup
Looking in the
Makefile, I just find support for coverage reports using gcov. Whatever is there with "profile" in it seems to be for profile-based compilation rather than using gprof.Now since I've managed to push most of the runtime for basic git-blame operation out of
blame.cproper, it becomes important to figure out where most of the remaining runtime (a sizable part of that being system time) is being spent. Loop counts like that provided by gcov (or am I missing something here?) are not helpful for that, I think I rather need the kind of per-function breakdown that gprof provides.
Quoted 6 lines
Quoted 4 lines
Is there a reason there are no prewired recipes or advice for using gprof on git? Is there a way to get the work done, namely seeing the actual distribution of call times (rather than iterations) using gcov so that this is not necessary?
Would perf help? No changes required, and almost no overhead, I think.
Not useful. It would be probably nice for nailing down the performance gains when the work is finished so that future regressions will be noticeable. It's reasonable easy to create a test case that will take hours with the current git-blame and would finish in seconds with the improved one.
But it's not useful at all for figuring out the hotspots within the git-blame binary.
I made do with something like
make CFLAGS=-pg LDFLAGS=-pg
but it is sort of annoying that the required "make clean" apparently also cleans out the coverage files: for sensible finding of bad stuff one needs them as well.
At any rate, I'll probably figure out something eventually. No idea whether I'll get around to writing some useful instructions. At the current point of time, it would appear that a large part of the remaining user time (about half) is spent in xdl_hash_record so x86_64 architectures already benefit from XDL_FAST_HASH (which seems to hurt more than it would help with my i686). So finding a good fast hash function would likely help. The current hash function _and_ the XDL_FAST_HASH replacement used on x86_64 are a drag here because they are not easily split into a word-unaligned, a (typically long and thus most performance-relevant) word-aligned, and another word-unaligned part in a manner that allows calculating the same hash for different alignments. Something like a CRC lends itself much better to that, but since its basic operation is more complex on a general-purpose CPU, it's not likely to result in a net win.
In assembly language, add-and-multiply operations modulo 2^32-1 are pretty easy to do and lend themselves well to considering alignment at the end, but in C, access to mixed precision multiplications and the carry flag are rather awkward.
Signature · 1 line
David Kastrup
Quoted 22 lines · David Kastrup
Quoted 13 lines · Duy Nguyen
Quoted 12 lines · David Kastrup
Looking in the
Makefile, I just find support for coverage reports using gcov. Whatever is there with "profile" in it seems to be for profile-based compilation rather than using gprof.Now since I've managed to push most of the runtime for basic git-blame operation out of
blame.cproper, it becomes important to figure out where most of the remaining runtime (a sizable part of that being system time) is being spent. Loop counts like that provided by gcov (or am I missing something here?) are not helpful for that, I think I rather need the kind of per-function breakdown that gprof provides.Quoted 6 lines
Quoted 4 lines
Is there a reason there are no prewired recipes or advice for using gprof on git? Is there a way to get the work done, namely seeing the actual distribution of call times (rather than iterations) using gcov so that this is not necessary?
Would perf help? No changes required, and almost no overhead, I think.
Quoted 8 lines
Not useful. It would be probably nice for nailing down the performance gains when the work is finished so that future regressions will be noticeable. It's reasonable easy to create a test case that will take hours with the current git-blame and would finish in seconds with the improved one.
But it's not useful at all for figuring out the hotspots within the git-blame binary.
I would have thought the annotation described at [1] is exactly what you're looking for, isn't it?
Alternatively, I've had some success with callgrind and kCachegrind in the past.
[1] https://perf.wiki.kernel.org/index.php/Tutorial#Source_level_analysis_with_perf_annotate
Quoted 14 lines · John Keeping
Quoted 4 lines · David Kastrup
Quoted 1 line · Duy Nguyen
Would perf help? No changes required, and almost no overhead, I think.
Quoted 8 lines
Not useful. It would be probably nice for nailing down the performance gains when the work is finished so that future regressions will be noticeable. It's reasonable easy to create a test case that will take hours with the current git-blame and would finish in seconds with the improved one.
But it's not useful at all for figuring out the hotspots within the git-blame binary.
Quoted 8 lines
I would have thought the annotation described at [1] is exactly what you're looking for, isn't it?
Alternatively, I've had some success with callgrind and kCachegrind in the past.
[1]
https://perf.wiki.kernel.org/index.php/Tutorial#Source_level_analysis_with_perf_annotate
Misunderstanding on my part. I thought this was about the "make perf" Makefile target. I'll have to take a look at what the perf utility does.
Thanks for the clarification.
Signature · 1 line
David Kastrup
Quoted 22 lines · David Kastrup
Quoted 20 lines · John Keeping
Quoted 4 lines · David Kastrup
Quoted 1 line · Duy Nguyen
Would perf help? No changes required, and almost no overhead, I think.
Quoted 8 lines
Not useful. It would be probably nice for nailing down the performance gains when the work is finished so that future regressions will be noticeable. It's reasonable easy to create a test case that will take hours with the current git-blame and would finish in seconds with the improved one.
But it's not useful at all for figuring out the hotspots within the git-blame binary.
I would have thought the annotation described at [1] is exactly what you're looking for, isn't it?
Alternatively, I've had some success with callgrind and kCachegrind in the past.
Quoted 8 lines
Quoted 2 lines
[1]
https://perf.wiki.kernel.org/index.php/Tutorial#Source_level_analysis_with_perf_annotateMisunderstanding on my part. I thought this was about the "make perf"
Makefiletarget. I'll have to take a look at what the perf utility does.Thanks for the clarification.
Looks indeed like it could be useful. In the currently working code from me (last of the complex patch series), significant time is spent in moving and allocating memory. It might make sense not to continue ignoring the patch I proposed to get rid of the absurd realloc series in builtin/blame.c. While it may not be responsible for all of avoidable allocation churn, it's still large and untypical enough to mask further culprits.
Signature · 1 line
David Kastrup
Quoted 3 lines · David Kastrup
Looking in the
Makefile, I just find support for coverage reports using gcov. Whatever is there with "profile" in it seems to be for profile-based compilation rather than using gprof.
[...]
Quoted 4 lines
Is there a reason there are no prewired recipes or advice for using gprof on git? Is there a way to get the work done, namely seeing the actual distribution of call times (rather than iterations) using gcov so that this is not necessary?
No reason I'm aware of, other than that nobody ever wrote it.
Note that I wouldn't exactly be surprised if the gcov targets had bitrotted without anyone noticing. I haven't heard of any heavy users. I originally wrote them to do some basic test coverage analysis, but that's about it.
Signature · 2 lines
Thomas Rast [email protected]
Quoted 5 lines · Thomas Rast
Quoted 3 lines · David Kastrup
Looking in the
Makefile, I just find support for coverage reports using gcov. Whatever is there with "profile" in it seems to be for profile-based compilation rather than using gprof.
Quoted 7 lines
[...]
Quoted 4 lines
Is there a reason there are no prewired recipes or advice for using gprof on git? Is there a way to get the work done, namely seeing the actual distribution of call times (rather than iterations) using gcov so that this is not necessary?
No reason I'm aware of, other than that nobody ever wrote it.
A solid testing/benchmarking framework would quite seem like a useful GSoC project as it would make it easy for casual programmers to dip their feet into their personal bottlenecks, and it would make it much easier to find worthwhile hotspots for future projects taking the challenge of speeding up core and/or specific operations.
Quoted 4 lines
Note that I wouldn't exactly be surprised if the gcov targets had bitrotted without anyone noticing. I haven't heard of any heavy users. I originally wrote them to do some basic test coverage analysis, but that's about it.
I've managed to make use of the outer sandwich layers: the prepare and the evaluate stuff. I ran my own tests for benchmarking though.
It's enfuriatingly hard to get the stuff in the right directories and with the right options, so that was already more helpful than it probably should have been.
Signature · 1 line
David Kastrup
Quoted 21 lines · David Kastrup
Quoted 5 lines · Thomas Rast
Quoted 3 lines · David Kastrup
Looking in the
Makefile, I just find support for coverage reports using gcov. Whatever is there with "profile" in it seems to be for profile-based compilation rather than using gprof.Quoted 7 lines
[...]
Quoted 4 lines
Is there a reason there are no prewired recipes or advice for using gprof on git? Is there a way to get the work done, namely seeing the actual distribution of call times (rather than iterations) using gcov so that this is not necessary?
No reason I'm aware of, other than that nobody ever wrote it.
A solid testing/benchmarking framework would quite seem like a useful GSoC project as it would make it easy for casual programmers to dip their feet into their personal bottlenecks, and it would make it much easier to find worthwhile hotspots for future projects taking the challenge of speeding up core and/or specific operations.
Quoted 7 lines
Quoted 4 lines
Note that I wouldn't exactly be surprised if the gcov targets had bitrotted without anyone noticing. I haven't heard of any heavy users. I originally wrote them to do some basic test coverage analysis, but that's about it.
I've managed to make use of the outer sandwich layers: the prepare and the evaluate stuff. I ran my own tests for benchmarking though.
Umm, are we even discussing the same thing here?
Are you saying you ran profiling-instrumented code under the t/perf/ support code? Sounds nice...
Signature · 2 lines
Thomas Rast [email protected]
Quoted 28 lines · Thomas Rast
Quoted 26 lines · David Kastrup
Quoted 5 lines · Thomas Rast
Quoted 3 lines · David Kastrup
Looking in the
Makefile, I just find support for coverage reports using gcov. Whatever is there with "profile" in it seems to be for profile-based compilation rather than using gprof.Quoted 7 lines
[...]
Quoted 4 lines
Is there a reason there are no prewired recipes or advice for using gprof on git? Is there a way to get the work done, namely seeing the actual distribution of call times (rather than iterations) using gcov so that this is not necessary?
No reason I'm aware of, other than that nobody ever wrote it.
A solid testing/benchmarking framework would quite seem like a useful GSoC project as it would make it easy for casual programmers to dip their feet into their personal bottlenecks, and it would make it much easier to find worthwhile hotspots for future projects taking the challenge of speeding up core and/or specific operations.
Quoted 4 lines
Note that I wouldn't exactly be surprised if the gcov targets had bitrotted without anyone noticing. I haven't heard of any heavy users. I originally wrote them to do some basic test coverage analysis, but that's about it.
Quoted 4 lines
Quoted 2 lines
I've managed to make use of the outer sandwich layers: the prepare and the evaluate stuff. I ran my own tests for benchmarking though.
Umm, are we even discussing the same thing here?
Maybe not.
Quoted 2 lines
Are you saying you ran profiling-instrumented code under the t/perf/ support code?
No. I used the toplevel Makefile targets coverage-clean, coverage-compile and, after running my own problematic test cases, coverage-report. I did not use the coverage-test target, nor did I venture into t/perf/ in any other way.
Signature · 1 line
David Kastrup