ccache.bbclass: Refactor it to make it more reliable

== Series Details ==

Series: ccache.bbclass: Refactor it to make it more reliable (rev3)
Revision: 3
URL : https://patchwork.openembedded.org/series/15276/
State : failure

== Summary ==

Thank you for submitting this patch series to OpenEmbedded Core. This is an automated response. Several tests have been executed on the proposed series by patchtest resulting in the following failures:

* Issue LIC_FILES_CHKSUM changed on target ccache but there is no "License-Update" tag in commit message [test_lic_files_chksum_modified_not_mentioned]
Suggested fix Include "License-Update: <description>" into the commit message with a brief description
Current checksum file://LICENSE.adoc;md5=2722abeaf1750dbf175b9491112374e1
New checksum file://LICENSE.adoc;md5=39f34d6ff4df51ed718fd7d243fc4e51

If you believe any of these test results are incorrect, please reply to the mailing list ([email protected]) raising your concerns. Otherwise we would appreciate you correcting the issues and submitting a new version of the patchset if applicable. Please ensure you add/increment the version number when sending the new version (i.e. [PATCH] -> [PATCH v2] -> [PATCH v3] -> ...).

---

Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

* V3
- Make cache files can be shared between different build dirs
- Add task do_cleanccache
- Make cmake work with ccache
- Upgrade ccache to 3.5, this version has a debug mode, which is useful for figuring out why a cache file is not used.

* V2
- Fix comments from Martin Jansa and Martin Hundebøll.
- Fix RP's comments:
1) Add comments for ccache related code in base.bbclass
2) Remove all ccache-native's dependencies so that all recipes can depend on and use it.
- Update test_ccache_tool in oeqa

* V1
Initial version

The following changes since commit 20aea61385e1a53ac245353899277ba20104ed2f:

gcc: Drop 7.3 since 8.2 is working fine for us (2018-12-15 17:10:44 +0000)

are available in the git repository at:

git://git.openembedded.org/openembedded-core-contrib rbt/ccache
http://cgit.openembedded.org/openembedded-core-contrib/log/?h=rbt/ccache

Robert Yang (11):
ccache-native: Remove dependencies
ccache: 3.4.2 -> 3.5
ccache.bbclass: Refactor it to make it more reliable ccache.bbclass: Make it can be shared between different builds ccache.bbclass: Add task do_cleanccache ccache.bbclass: Set CCACHE_CONFIGPATH to fix race issues bitbake.conf: Enable -fdebug-prefix-map for nativesdk goarch.bbclass: Set CCACHE_DISABLE
cmake.bbclass: Make it work with ccache apr/openssl10: Enable ccache for them oeqa/selftest/cases: Update test_ccache_tool

  meta/classes/ccache.bbclass                        | 63 ++++++++++++++++++++--
  meta/classes/cmake.bbclass                         | 21 +++++---
  meta/classes/goarch.bbclass                        |  4 ++
  meta/conf/bitbake.conf                             |  8 +--
  meta/conf/ccache.conf                              |  2 +
  meta/conf/layer.conf                               |  1 -
  meta/lib/oe/utils.py                               |  3 ++
  meta/lib/oeqa/selftest/cases/buildoptions.py       | 10 ++--
  .../openssl/openssl10_1.0.2q.bb                    |  3 --
  meta/recipes-devtools/ccache/ccache.inc            | 13 ++++-
  meta/recipes-devtools/ccache/ccache_3.4.2.bb       | 11 ----
  meta/recipes-devtools/ccache/ccache_3.5.bb         | 11 ++++
  meta/recipes-graphics/piglit/piglit_git.bb         |  3 ++
  meta/recipes-graphics/vulkan/vulkan_1.1.73.0.bb    |  3 ++
  meta/recipes-sato/webkit/webkitgtk_2.22.4.bb       |  3 ++
  meta/recipes-support/apr/apr_1.6.5.bb              |  5 --
  16 files changed, 122 insertions(+), 42 deletions(-)

create mode 100644 meta/conf/ccache.conf
delete mode 100644 meta/recipes-devtools/ccache/ccache_3.4.2.bb create mode 100644 meta/recipes-devtools/ccache/ccache_3.5.bb

The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses

ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is

in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccahe is installed on the host:

   CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except ccache-native itself, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe

rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear.

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/ccache.bbclass | 25 ++++++++++++++++++++++---
  meta/conf/bitbake.conf      |  6 +-----
  meta/conf/layer.conf        |  1 -
  meta/lib/oe/utils.py        |  3 +++
  4 files changed, 26 insertions(+), 9 deletions(-)

meta/classes/ccache.bbclass +22 -3

index 9609020..59e1022 100644
@@ -1,4 +1,14 @@
-CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
+#
+# Usage:
+# - Enable ccache
+#   Add the following line to a conffile such as conf/local.conf:
+#   INHERIT += "ccache"
+#
+# - Disable ccache for a recipe
+#   Add the following line to the recipe if it can't be built with ccache:
+#   CCACHE_DISABLE = '1'
+#
+
  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
 
  # We need to stop ccache considering the current directory or the

@@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
  # ${PV} or ${PR} change.
  export CCACHE_NOHASHDIR ?= "1"
 
-DEPENDS_append_class-target = " ccache-native"
-DEPENDS[vardepvalueexclude] = " ccache-native"
+python() {
+    """
+    Enable ccache for the recipe
+    """
+    pn = d.getVar('PN')
+    # quilt-native doesn't need ccache since no c files
+    if not (pn in ('ccache-native', 'quilt-native') or
+            bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))):
+        d.appendVar('DEPENDS', ' ccache-native')
+        d.setVar('CCACHE', 'ccache ')
+}

meta/conf/bitbake.conf +1 -5

index 6480062..c88d66c 100644
@@ -492,7 +492,7 @@ HOSTTOOLS += " \
  HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}"
 
  # Link to these if present
-HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
+HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
 
  # Temporary add few more detected in bitbake world
  HOSTTOOLS_NONFATAL += "join nl size yes zcat"

@@ -504,10 +504,6 @@ HOSTTOOLS_NONFATAL += "bzr"
  HOSTTOOLS_NONFATAL += "scp"
 
  CCACHE ??= ""
-# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and
-# autogen sets HOME=/dev/null so in certain situations builds can fail.
-# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host.
-export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache"
 
  TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}"
 

meta/conf/layer.conf +0 -1

index c67dec4..4141d6c 100644
@@ -49,7 +49,6 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \
    *->quilt-native \
    *->subversion-native \
    *->git-native \
-  *->ccache-native \
    *->icecc-create-env-native \
    gcc-cross-${TARGET_ARCH}->virtual/${TARGET_PREFIX}libc-for-gcc \
    gcc-cross-${TARGET_ARCH}->linux-libc-headers \

meta/lib/oe/utils.py +3 -0

index 29b4115..ee6f0e6 100644
@@ -363,6 +363,9 @@ def host_gcc_version(d, taskcontextonly=False):
          return
 
      compiler = d.getVar("BUILD_CC")
+    # Get rid of ccache since it is not present when parsing.
+    if compiler.startswith('ccache '):
+        compiler = compiler[7:]
      try:
          env = os.environ.copy()
          env["PATH"] = d.getVar("PATH")

A couple of typos and grammar improvements, otherwise a good improvement for iterative development of a package/image where sstate isn't useful.

Quoted 7 lines · Robert Yang

The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses

* host's ccache

Quoted 4 lines

   ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is

* and be hard to debug

Quoted 5 lines

   in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccahe is installed on the host:

* no ccache

Quoted 8 lines

   CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except ccache-native itself, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe

* disable ccache

../Randy

Quoted 96 lines

rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear.

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/ccache.bbclass | 25 ++++++++++++++++++++++--- meta/conf/bitbake.conf      |  6 +----- meta/conf/layer.conf        |  1 - meta/lib/oe/utils.py        |  3 +++

  4 files changed, 26 insertions(+), 9 deletions(-)

meta/classes/ccache.bbclass +11 -1

index 9609020..59e1022 100644
@@ -1,4 +1,14 @@
-CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
+#
+# Usage:
+# - Enable ccache
+#   Add the following line to a conffile such as conf/local.conf:
+#   INHERIT += "ccache"
+#
+# - Disable ccache for a recipe
+#   Add the following line to the recipe if it can't be built with ccache:
+#   CCACHE_DISABLE = '1'
+#
+

  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}" # We need to stop ccache considering the current directory or the

meta/classes/ccache.bbclass +0 -0

@@ -7,5 +17,14 @@ export CCACHE_DIR ?= 

"${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
  # ${PV} or ${PR} change.
  export CCACHE_NOHASHDIR ?= "1"

meta/classes/ccache.bbclass +11 -2

-DEPENDS_append_class-target = " ccache-native"
-DEPENDS[vardepvalueexclude] = " ccache-native"
+python() {
+    """
+    Enable ccache for the recipe
+    """
+    pn = d.getVar('PN')
+    # quilt-native doesn't need ccache since no c files
+    if not (pn in ('ccache-native', 'quilt-native') or
+            bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))):
+        d.appendVar('DEPENDS', ' ccache-native')
+        d.setVar('CCACHE', 'ccache ')
+}

meta/conf/bitbake.conf +0 -0

index 6480062..c88d66c 100644
@@ -492,7 +492,7 @@ HOSTTOOLS += " \

  HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}" # Link to these if present
-HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp
socat ssh sudo"
+HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh
sudo"
  # Temporary add few more detected in bitbake world
  HOSTTOOLS_NONFATAL += "join nl size yes zcat"

meta/conf/bitbake.conf +0 -0

@@ -504,10 +504,6 @@ HOSTTOOLS_NONFATAL += "bzr"

  HOSTTOOLS_NONFATAL += "scp"
  CCACHE ??= ""

meta/conf/bitbake.conf +0 -4

-# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and
-# autogen sets HOME=/dev/null so in certain situations builds can fail.
-# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host.
-export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache"

  TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_TARGET}"

meta/conf/layer.conf +0 -1

index c67dec4..4141d6c 100644
@@ -49,7 +49,6 @@ SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS += " \
     *->quilt-native \
     *->subversion-native \
     *->git-native \
-  *->ccache-native \
     *->icecc-create-env-native \
     gcc-cross-${TARGET_ARCH}->virtual/${TARGET_PREFIX}libc-for-gcc \
     gcc-cross-${TARGET_ARCH}->linux-libc-headers \

meta/lib/oe/utils.py +0 -0

index 29b4115..ee6f0e6 100644
@@ -363,6 +363,9 @@ def host_gcc_version(d, taskcontextonly=False):

          return
      compiler = d.getVar("BUILD_CC")
+    # Get rid of ccache since it is not present when parsing. +    if compiler.startswith('ccache '):
+        compiler = compiler[7:]
      try:
          env = os.environ.copy()
          env["PATH"] = d.getVar("PATH")

Hi Randy,

I'm sorry for the types, and thanks for the correction. I found that it has
been merged, I will be more careful.

Quoted 5 lines · Robert Yang

Hi Randy,

I'm sorry for the types, and thanks for the correction. I found that it has
been merged, I will be more careful.

I did tweak the commit message!

Cheers,

Richard