ccache: Separate out into its own class
PATCH 1/1 ccache: Separate out into its own class
Currently, ccache is used if it is present. When building from scratch it gives no performance improvement and creates a ton of empty directories even when its not in use.
This change moves ccache support to a bbclass file which the user can choose to enable. This should make builds more determinstic and make it easier/clearer to the end user when its being used and when it is not.
---
meta/classes/base.bbclass +1 -1
@@ -271,7 +271,7 @@ python base_eventhandler() { } addtask configure after do_patch -do_configure[dirs] = "${CCACHE_DIR} ${S} ${B}" +do_configure[dirs] = "${S} ${B}" do_configure[deptask] = "do_populate_sysroot" base_do_configure() { :
meta/classes/ccache.bbclass renamed from dev/null +5 -0
@@ -0,0 +1,5 @@ +CCACHE = "${@bb.which(d.getVar('PATH', True), 'ccache') and 'ccache '}" +export CCACHE_DIR = "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}" + +do_configure[dirs] =+ "${CCACHE_DIR}" +do_kernel_configme[dirs] =+ "${CCACHE_DIR}"
meta/classes/kernel-yocto.bbclass +1 -1
@@ -161,7 +161,7 @@ do_kernel_checkout[dirs] = "${S}" addtask kernel_checkout before do_patch after do_unpack -do_kernel_configme[dirs] = "${CCACHE_DIR} ${S} ${B}" +do_kernel_configme[dirs] = "${S} ${B}" do_kernel_configme() { echo "[INFO] doing kernel configme"
meta/conf/bitbake.conf +2 -3
@@ -409,10 +409,9 @@ export PATH # Build utility info. ################################################################## -CCACHE = "${@bb.which(d.getVar('PATH', True), 'ccache') and 'ccache '}" +CCACHE ??= "" TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}" -export CCACHE_DIR = "${TMPDIR}/ccache/${MULTIMACH_HOST_SYS}/${PN}" export CC = "${CCACHE}${HOST_PREFIX}gcc ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}" export CXX = "${CCACHE}${HOST_PREFIX}g++ ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}" export F77 = "${CCACHE}${HOST_PREFIX}g77 ${HOST_CC_ARCH}${TOOLCHAIN_OPTIONS}"
@@ -746,7 +745,7 @@ BB_CONSOLELOG ?= "${TMPDIR}/cooker.log.${DATETIME}" # Setup our default hash policy BB_SIGNATURE_HANDLER ?= "OEBasic" -BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE CCACHE_DIR EXTERNAL_TOOLCHAIN" +BB_HASHBASE_WHITELIST ?= "TMPDIR FILE PATH PWD BB_TASKHASH BBPATH DL_DIR SSTATE_DIR THISDIR FILESEXTRAPATHS FILE_DIRNAME HOME LOGNAME SHELL TERM USER FILESPATH STAGING_DIR_HOST STAGING_DIR_TARGET COREBASE PRSERV_HOST PRSERV_PORT PRSERV_DUMPDIR PRSERV_DUMPFILE PRSERV_LOCKDOWN PARALLEL_MAKE CCACHE_DIR EXTERNAL_TOOLCHAIN CCACHE" BB_HASHCONFIG_WHITELIST ?= "${BB_HASHBASE_WHITELIST} DATE TIME SESSION_MANAGER DBUS_SESSION_BUS_ADDRESS SSH_AGENT_PID XDG_SESSION_COOKIE SSH_AUTH_SOCK XAUTHORITY" MLPREFIX ??= ""
Quoted 7 lines · Richard Purdie
Currently, ccache is used if it is present. When building from scratch it gives no performance improvement and creates a ton of empty directories even when its not in use.
This change moves ccache support to a bbclass file which the user can choose to enable. This should make builds more determinstic and make it easier/clearer to the end user when its being used and when it is not.
Are the sstate sums the same with and without the use of the new bbclass ccache?
It is not clear to me that this is or is not the case based on this single commit header.
Jason.
Quoted 5 lines · Jason Wessel
Quoted 4 lines · Richard Purdie
Currently, ccache is used if it is present. When building from scratch it gives no performance improvement and creates a ton of empty directories even when its not in use.
Quoted 7 lines
Quoted 3 lines
This change moves ccache support to a bbclass file which the user can choose to enable. This should make builds more determinstic and make it easier/clearer to the end user when its being used and when it is not.
Are the sstate sums the same with and without the use of the new bbclass ccache?
It is not clear to me that this is or is not the case based on this single commit header.
There is a problem with the existing sstate checksums since CCACHE itself is not being excluded, only CCACHE_DIR so you'd end up with sstate built using ccache and sstate built without using cccache with different checksums. This change unifies those to have matching sstate checksums.
So with and without use of the class, the checksums should now be the same.
Cheers,
Richard
Richard Purdie <[email protected]>
Quoted 2 lines · writes:
Currently, ccache is used if it is present. When building from scratch it gives no performance improvement
Here, I have a 10% cache hit ratio in a scratch build:
cache hit (direct) 2841
cache hit (preprocessed) 1370
cache miss 39957
called for link 3369
called for preprocessing 3207
multiple source files 8
compile failed 2531
preprocessor error 432 couldn't find the compiler 111 bad compiler arguments 468 unsupported source language 532
autoconf compile/link 6613
unsupported compiler option 132
no input file 2919
files in cache 94205
cache size 2.0 Gbytes
max cache size 3.0 Gbytes
Enrico
Quoted 20 lines · Enrico Scholz
Richard Purdie <[email protected]>
Quoted 2 lines · writes:
Currently, ccache is used if it is present. When building from scratch it gives no performance improvement
Here, I have a 10% cache hit ratio in a scratch build:
cache hit (direct) 2841
cache hit (preprocessed) 1370
cache miss 39957
called for link 3369
called for preprocessing 3207
multiple source files 8
compile failed 2531preprocessor error 432 couldn't find the compiler 111 bad compiler arguments 468 unsupported source language 532
Quoted 7 lines
autoconf compile/link 6613
unsupported compiler option 132
no input file 2919
files in cache 94205
cache size 2.0 Gbytes
max cache size 3.0 Gbytes
That's good data, thanks for sharing it. In any measurement I've made, it doesn't seem to help the wall time for the build though :(
We're not removing the ability to use ccache, just changing the way its enabled which all things considered I still think is a good move...
Cheers,
Richard