The design document for ccache-native

-------- Original Message --------

Quoted 28 lines · Saul Wold

Quoted 27 lines · wenzong fan

Hi Folks,

Please help me to review the design document for ccache-native, and I also have two questions about it, any answers or suggestions are appreciated.

* Feature name: ccache-native
Priority: P3; M2
Owner: Wenzong Fan
Summary: Integrate ccache-native to yocto

* Description:
Bitbake supports the 'CCACHE Mechanism', but 'ccache' hasn't been included by poky/yocto, just add it as a native tool.

* Usage:
Build ccache as a native tool by default and enable it for speeding target packages build.

* Implementation:
1) Copy bb file from OE upstream to:
meta/recipes-devtools/ccache/

2) Update bb file to get the latest ccache_3.1.5 and split the single bb file to:
'ccache_3.1.5.bb', 'ccache.inc'

Quoted 7 lines

Quoted 2 lines

3) Enable ccache in the native tools building.

You will need to have it be a dependency pretty early on in the build. Additionally, this is a bit a new part to this task, we want to have the default CCACHE_DIR for the build default to a directory in TMPDIR instead of the user's home directory. This will mean setting an environment variable somewhere early also.

There is a little more detail on:

https://wiki.yoctoproject.org/wiki/Yocto_1.1_Schedule

Specifically, "c) Set CCACHE on a per recipe basis. need to figure out whether ccache data can be shared and under what circumstances."

so something like adding:

export CCACHE_DIR = "${TMPDIR}/ccache/${TARGET_SYS}/${PN}"

to bitbake.conf with a bit more thought into working out the right components to add to the variable.

Cheers,

Richard

Quoted 43 lines · wenzong fan

-------- Original Message --------

Quoted 28 lines · Saul Wold

Quoted 27 lines · wenzong fan

Hi Folks,

Please help me to review the design document for ccache-native, and I also have two questions about it, any answers or suggestions are appreciated.

* Feature name: ccache-native
Priority: P3; M2
Owner: Wenzong Fan
Summary: Integrate ccache-native to yocto

* Description:
Bitbake supports the 'CCACHE Mechanism', but 'ccache' hasn't been included by poky/yocto, just add it as a native tool.

* Usage:
Build ccache as a native tool by default and enable it for speeding target packages build.

* Implementation:
1) Copy bb file from OE upstream to:
meta/recipes-devtools/ccache/

2) Update bb file to get the latest ccache_3.1.5 and split the single bb file to:
'ccache_3.1.5.bb', 'ccache.inc'

Quoted 7 lines

Quoted 2 lines

3) Enable ccache in the native tools building.

You will need to have it be a dependency pretty early on in the build. Additionally, this is a bit a new part to this task, we want to have the default CCACHE_DIR for the build default to a directory in TMPDIR instead of the user's home directory. This will mean setting an environment variable somewhere early also.

There is a little more detail on:

Quoted 9 lines

https://wiki.yoctoproject.org/wiki/Yocto_1.1_Schedule

Specifically, "c) Set CCACHE on a per recipe basis. need to figure out whether ccache data can be shared and under what circumstances."

so something like adding:

export CCACHE_DIR = "${TMPDIR}/ccache/${TARGET_SYS}/${PN}"

I think that set CCACHE_DIR on per recipe basis would degrade hit efficiency,
the following ccache data can be shared if they they use the same CCACHE_DIR,
but if we set CACHE_DIR on a per recipe basis, then they can't be shared:

1) Most pkg's configure will run "gcc foo.c" for checking the C compiler, these compiling are the same between different pkgs at most time.

2) Some recipes' compiling are similar, for example: gcc-cross, gcc-corss-initial and gcc-cross-intermediate.

I think that:

export CCACHE_DIR = "${TMPDIR}/ccache/${TARGET_SYS}/"

would be better.

// Robert

Quoted 10 lines

to bitbake.conf with a bit more thought into working out the right components to add to the variable.

Cheers,

Richard

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

I fear that the use of ccache is inherently risky with OE. Given the (relatively common) case where the user blows away their TMPDIR in order to get a full, clean rebuild after an update to the toolchain is make, it is possible that ccache will erroneously re-use an object created by the earlier version of the toolchain.

While imperfect, I would suggest that we would do better if we would somehow embed the PV (or something like that) for the toolchain into the CCACHE_DIR, thus ensuring that we don't risk the re-use of old objects when the toolchain is updated.

(Frankly, given my experiences with it, I'd prefer we just disable ccache entirely with OE.)

Quoted 67 lines · Robert Yang

Quoted 43 lines · wenzong fan

-------- Original Message --------

Quoted 28 lines · Saul Wold

Quoted 27 lines · wenzong fan

Hi Folks,

Please help me to review the design document for ccache-native, and I also have two questions about it, any answers or suggestions are appreciated.

* Feature name: ccache-native
Priority: P3; M2
Owner: Wenzong Fan
Summary: Integrate ccache-native to yocto

* Description:
Bitbake supports the 'CCACHE Mechanism', but 'ccache' hasn't been included by poky/yocto, just add it as a native tool.

* Usage:
Build ccache as a native tool by default and enable it for speeding target packages build.

* Implementation:
1) Copy bb file from OE upstream to:
meta/recipes-devtools/ccache/

2) Update bb file to get the latest ccache_3.1.5 and split the single bb file to:
'ccache_3.1.5.bb', 'ccache.inc'

Quoted 7 lines

Quoted 2 lines

3) Enable ccache in the native tools building.

You will need to have it be a dependency pretty early on in the build. Additionally, this is a bit a new part to this task, we want to have the default CCACHE_DIR for the build default to a directory in TMPDIR instead of the user's home directory. This will mean setting an environment variable somewhere early also.

There is a little more detail on:

Quoted 9 lines

https://wiki.yoctoproject.org/wiki/Yocto_1.1_Schedule

Specifically, "c) Set CCACHE on a per recipe basis. need to figure out whether ccache data can be shared and under what circumstances."

so something like adding:

export CCACHE_DIR = "${TMPDIR}/ccache/${TARGET_SYS}/${PN}"

I think that set CCACHE_DIR on per recipe basis would degrade hit efficiency,
the following ccache data can be shared if they they use the same CCACHE_DIR,
but if we set CACHE_DIR on a per recipe basis, then they can't be shared:

1) Most pkg's configure will run "gcc foo.c" for checking the C compiler, these compiling are the same between different pkgs at most time.

2) Some recipes' compiling are similar, for example: gcc-cross, gcc-corss-initial and gcc-cross-intermediate.

Quoted 5 lines

I think that:

export CCACHE_DIR = "${TMPDIR}/ccache/${TARGET_SYS}/"

would be better.

I fear that the use of ccache is inherently risky with OE. Given the (relatively common) case where the user blows away their TMPDIR in order

to get a full, clean rebuild after an update to the toolchain is make, it is possible that ccache will erroneously re-use an object created by the earlier version of the toolchain.

While imperfect, I would suggest that we would do better if we would somehow embed the PV (or something like that) for the toolchain into the CCACHE_DIR, thus ensuring that we don't risk the re-use of old objects when the toolchain is updated.

(Frankly, given my experiences with it, I'd prefer we just disable ccache entirely with OE.)

-Mike (mwester)

Quoted 13 lines

// Robert

Quoted 10 lines

to bitbake.conf with a bit more thought into working out the right components to add to the variable.

Cheers,

Richard

Quoted 6 lines

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core

_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core