LMbench as gcc performance regression test?
http://cs.nmu.edu/~benchmark/ has an interesting little graph of LMBench results vs. Linux kernel version, all done with the same compiler.
Has anyone seen a similar graph showing LMBench results vs. gcc version, all done with the same Linux kernel?
And does everyone agree that's a meaningful way to compare the performance of code generated by different compilers?
I happen to have a number of versions of gcc handy, and was considering making such a graph, but was hoping somebody else had already done it.
(There seems to be large variations in successive runs of LMBench when I try it, so it may take me a bit of work to get repeatable results.)
Thanks,
Dan
Signature · 3 lines
Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045
Quoted 3 lines · Dan Kegel
(There seems to be large variations in successive runs of LMBench when I try it, so it may take me a bit of work to get repeatable results.)
Other than the context switch part or anything based on it, that shouldn't be true, it should be very stable.
I'm pretty convinced that the variations are due to different pages being allocated and the result cache contention makes things bounce.
Signature · 2 lines
--- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Quoted 7 lines · Larry McVoy
Quoted 3 lines · Dan Kegel
(There seems to be large variations in successive runs of LMBench when I try it, so it may take me a bit of work to get repeatable results.)
Quoted 5 lines
Other than the context switch part or anything based on it, that shouldn't be true, it should be very stable.
I'm pretty convinced that the variations are due to different pages being allocated and the result cache contention makes things bounce.
Or an idiot running the benchmark. We really do have to rule that out first.
- Dan
Signature · 3 lines
Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045
Quoted 4 lines · Dan Kegel
http://cs.nmu.edu/~benchmark/ has an interesting little graph of LMBench results vs. Linux kernel version, all done with the same compiler.
Quoted 8 lines
Has anyone seen a similar graph showing LMBench results vs. gcc version, all done with the same Linux kernel?
And does everyone agree that's a meaningful way to compare the performance of code generated by different compilers?I happen to have a number of versions of gcc handy, and was considering making such a graph, but was hoping somebody else had already done it.
It's been a while since I looked at lmbench but: why do you think this would be useful? It's a system and kernel benchmark; I doubt optimization makes much difference at all.
Signature · 2 lines
Daniel Jacobowitz MontaVista Software Debian GNU/Linux Developer
Quoted 6 lines · Daniel Jacobowitz
Quoted 4 lines · Dan Kegel
http://cs.nmu.edu/~benchmark/ has an interesting little graph of LMBench results vs. Linux kernel version, all done with the same compiler.
Quoted 8 lines
Quoted 4 lines
Has anyone seen a similar graph showing LMBench results vs. gcc version, all done with the same Linux kernel?
And does everyone agree that's a meaningful way to compare the performance of code generated by different compilers?It's been a while since I looked at lmbench but: why do you think this would be useful? It's a system and kernel benchmark; I doubt optimization makes much difference at all.
I need to make sure that moving to a newer compiler for our kernel will cause no performance regressions. Before bothering to bring up a real-world networking application and measuring its performance under the new compiler, it seems sensible to use a couple microbenchmarks to verify that identifiable parts of the system have
not degraded in performance.
I myself am quite convinced I need to move to a newer compiler, since I keep running into problems building various things with old compilers, but my users are very conservative and skeptical; I have to build a solid case for updating. Hence the insane amount of time I spent figuring out and documenting how to build and test the various versions of gcc and glibc (http://kegel.com/crosstool),
and then understanding the regression test failures.
- Dan
Signature · 3 lines
Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045
Quoted 2 lines · Dan Kegel
I need to make sure that moving to a newer compiler for our kernel will cause no performance regressions.
Perhaps people think that you mean to compile LMbench w/ different GCC's and maybe what you really mean is compile the same kernel with different GCC's and measure that.
Signature · 2 lines
--- Larry McVoy lm at bitmover.com http://www.bitmover.com/lm
Quoted 8 lines
http://cs.nmu.edu/~benchmark/ has an interesting little graph of LMBench results vs. Linux kernel version, all done with the same compiler.
Has anyone seen a similar graph showing LMBench results vs. gcc version, all done with the same Linux kernel?
And does everyone agree that's a meaningful way to compare the performance of code generated by different compilers?
I've done similar things with kernbench before (always using 2.95 to run the test, but comparing kernels compiled with gcc 2.95 vs 3.2 vs 3.3, and -Os vs -O2, etc). Summary was that 3.x takes *much* longer to compile the kernel, and produces worse code (though 3.3 is almost back up to the performance of 2.95, and is better than 3.2). -O2 is better than -Os, at least on a machine with 2MB L2 cache. Search the archives for results I posted if you want, but I never bothered graphing them.
Quoted 7 lines
I happen to have a number of versions of gcc handy, and was considering making such a graph, but was hoping somebody else had already done it.
(There seems to be large variations in successive runs of LMBench when I try it, so it may take me a bit of work to get repeatable results.)
I'd just throw away any of the subtests that give you > 1% variations, deriving anything meaning from crap data is hard (and dubious). I have similar problems with some of the results in lmbench - Larry suggested setting "ENOUGH=" or something, which helped a few tests, but most of them still aren't stable enough to be useful to me.
I'd also use something "bigger" than just a microbenchmark - you need to exercise a realistic set of the kernel functions in order to see space vs time tradeoffs, etc. If you want to see whether it's faster for you, you need a benchmark that simulates roughly what you do with the machine (ie a system-level benchmark).
M.
Quoted 5 lines · Larry McVoy
Here is some background, pick a benchmark and play with it and see if you can convince yourself of anything. The basic idea is to run the benchmark TRIES times for $ENOUGH milliseconds. TRIES is set to an odd number in
bench.hbecause we sort the results and take the midpoint and print that as the result.
It seems lat_pipe never does any median smoothing; it always sets TRIES to 1. However, at least on the fairly quiet embedded system I'm testing on, smoothing samples taken within a single run wouldn't make
a huge difference. Any smoothing you get with that would be swamped by the fact that lat_pipe's result has a bimodal distribution only one of whose peaks shows up in any one run.
This sure sounds like the kind of thing page coloring is
supposed to solve; has anyone observed page coloring improving the repeatability of the lat_pipe benchmark?
(There's no median smoothing in lat_pipe.c, I think, because it passes a value >= 1000000 as the 2nd arg of BENCH:
BENCH(doit(p2[0], p1[1]), SHORT);
BENCH computes the number of samples to take the median of as
__N = (get_enough(1000000) <= 100000) ? TRIES : 1;get_enough() will always return at least what it is passed, thus __N will always be 1. It sure was whenever I printed it out, too. This seems to be the case for the following tests:bw_pipe bw_tcp bw_unixlat_fcntl lat_fifo lat_pipe lat_rpc lat_tcp lat_udp lat_unix)
- Dan
Signature · 3 lines
Dan Kegel http://www.kegel.com http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=78045