Not as much ccache win as I expected
I'm running an automated test which does numerous compiles of the Linux kernel. One of the things I do is create a localversion file at the root of the kernel source tree with a unique identifier that I use later on in testing.
I started using ccache to improve the performance of my builds, but found that the hit rate on the cache was not very good.
$ ccache -s cache directory /home/tbird/.ccache cache hit 74416 cache miss 59400 called for link 87252 compile failed 21 not a C/C++ file 143449 no input file 49336 files in cache 42844 cache size 1.8 Gbytes max cache size 2.0 Gbytes
Thinking that the problem might be having a unique version for every build (and that this change flowed to every file via the version.h file), I tried building without this change. I saw an improvement, but not much.
Is there anything else obvious which is prevents ccache from working well with a kernel build (that is, anything else that would, for otherwise identical C files with a similar build, cause a difference?)
Any tips would be appreciated.
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================
Tim Bird @ Fri, 13 Jun 2008 12:06:05 -0700:
Quoted 4 lines
I'm running an automated test which does numerous compiles of the Linux kernel. One of the things I do is create a localversion file at the root of the kernel source tree with a unique identifier that I use later on in testing.
And what kinds of source/kconfig changes are made for every build? (any versions, e.g. localversion, .version, aren't important, they are for modules ko and vmlinux, afaik)
[...]
Quoted 4 lines
Is there anything else obvious which is prevents ccache from working well with a kernel build (that is, anything else that would, for otherwise identical C files with a similar build, cause a difference?)
kbuild is ccache on itself. Every *.o.cmd is kind of info ccache hashes (except things like stderr, gcc version) to check repeated rebuilds. Also kconfig<->kbuild link via header magic may confuse general-purpose ccache.
For rebuilds of the same codebase, it's better to use separate kbuild object output directories.
Quoted 1 line
Any tips would be appreciated.
Just my handwaving, but test with couple core config symbols toggling, shows only one ccache hit.
olecom@flower:/mnt/zdev0/blinux$ CCACHE_DIR=_ccache/ ccache -s cache directory _ccache/ cache hit 1133 cache miss 1141 called for link 28 not a C/C++ file 64 no input file 282 files in cache 2282 cache size 15.7 Mbytes max cache size 976.6 Mbytes olecom@flower:/mnt/zdev0/blinux$ make menuconfig # toggle olecom@flower:/mnt/zdev0/blinux$ diff -u1 .config.old .config --- .config.old 2008-06-13 22:52:37.000000000 +0200 +++ .config 2008-06-13 23:01:07.000000000 +0200 @@ -3,3 +3,3 @@ # Linux kernel version: 2.6.24 -# Fri Jun 13 22:52:37 2008 +# Fri Jun 13 23:01:07 2008 # @@ -67,3 +67,4 @@ CONFIG_POSIX_MQUEUE=y -# CONFIG_BSD_PROCESS_ACCT is not set +CONFIG_BSD_PROCESS_ACCT=y +CONFIG_BSD_PROCESS_ACCT_V3=y # CONFIG_TASKSTATS is not set olecom@flower:/mnt/zdev0/blinux$ olecom@flower:/mnt/zdev0/blinux$ CCACHE_DIR=_ccache/ ccache -s cache directory _ccache/ cache hit 1134 cache miss 1879 called for link 33 not a C/C++ file 79 no input file 401 files in cache 3758 cache size 28.1 Mbytes max cache size 976.6 Mbytes olecom@flower:/mnt/zdev0/blinux$
Signature · 4 lines
sed 'sed && sh + olecom = love' << '' -o--=O`C #oo'L O <___=E M
Quoted 1 line · Oleg Verych
And what kinds of source/kconfig changes are made for every build?
I start with a baseline config for an embedded board, then alter, one at a time, individual config items related to kernel size. No source changes are made.
I do full removal of the kernel source tree and build area before the start of each test.
Quoted 2 lines
(any versions, e.g. localversion, .version, aren't important, they are for modules ko and vmlinux, afaik)
Ok - this is helpful.
Quoted 3 lines
kbuild is
ccacheon itself. Every *.o.cmd is kind of infoccachehashes (except things like stderr, gcc version) to check repeated rebuilds.
Yeah, I'm pretty impressed with how well kbuild avoids rebuilding stuff in the first place.
Maybe I should just be grateful for any ccache hits I get.
Thanks,
-- Tim
=============================
Tim Bird
Architecture Group Chair, CE Linux Forum Senior Staff Engineer, Sony Corporation of America
=============================
Quoted 2 lines · Tim Bird
Maybe I should just be grateful for any ccache hits I get.
ccache's usefulness depends on your workload. If you make a change to include/linux/fs.h, close to 100% of the kernel is rebuilt, with or without ccache. But when you revert that change, the build time differs dramatically. Without ccache, fs.h was simply changed again and everything is rebuild. With ccache, there are hits for the old version and all is pulled from the cache - provided you have allotted enough disk for it.
If you never revert to an old version or do some equivalent operation, ccache can even be a net loss. On a fast machine, the additional disk accesses are easily more expensive than the minimal cpu gains.
Jörn
Signature · 8 lines
Public Domain - Free as in Beer General Public - Free as in Speech BSD License - Free as in Enterprise Shared Source - Free as in "Work will make you..." -- To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html