ccache.bbclass: Refactor it to make it more reliable

The work well now.

Signed-off-by Robert Yang <[email protected]>

---
 meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb | 3 ---
 meta/recipes-support/apr/apr_1.6.5.bb                 | 1 -
 2 files changed, 4 deletions(-)

meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb +0 -3

index 6518dac..1986cf2 100644
@@ -76,9 +76,6 @@ EXTRA_OEMAKE = "${@bb.utils.contains('PACKAGECONFIG', 'manpages', '', 'OE_DISABL
 
 export OE_LDFLAGS = "${LDFLAGS}"
 
-# openssl fails with ccache: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12810
-CCACHE = ""
-
 TERMIO ?= "-DTERMIO"
 TERMIO_libc-musl = "-DTERMIOS"
 

meta/recipes-support/apr/apr_1.6.5.bb +0 -0

index c2f5900..ed93a7e 100644

meta/recipes-support/apr/apr_1.6.5.bb +0 -0

 PACKAGECONFIG ??= "${@bb.utils.filter('DISTRO_FEATURES', 'ipv6', d)}"
 PACKAGECONFIG[ipv6] = "--enable-ipv6,--disable-ipv6,"

Signature · 7 lines

2.10.2

-- 
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccahe is installed on the host:

   CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except ccache-native's dependencies, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear.

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
  meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
  meta/conf/layer.conf                    |  1 -
  meta/lib/oe/utils.py                    |  3 +++
  meta/recipes-devtools/ccache/ccache.inc |  2 ++
  5 files changed, 43 insertions(+), 9 deletions(-)

meta/lib/oe/utils.py +0 -0

      try:
          env = os.environ.copy()
          env["PATH"] = d.getVar("PATH")

meta/recipes-devtools/ccache/ccache.inc +2 -0

index 6566328..259c196 100644
@@ -9,6 +9,8 @@ LICENSE = "GPLv3+"
 
  DEPENDS = "zlib"
 
+DEPENDS_class-native = "${CCACHE_NATIVE_DEPENDS}"
+
  SRC_URI = "https://download.samba.org/pub/${BPN}/${BP}.tar.xz"
 
  inherit autotools

The code is in setup_hosttools_dir(), it searches ccache in PATH in reverse order, I can't figure out what it is used for, maybe it is aimed to use host's ccache rather than recipe-sysroot- native/usr/bin/ccache, but the later one isn't in original PATH, so remove the code.

Signed-off-by Robert Yang <[email protected]>

---
 meta/classes/base.bbclass | 2 --
 1 file changed, 2 deletions(-)

The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccahe is installed on the host:
CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except ccache-native's dependencies, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear.

Signed-off-by Robert Yang <[email protected]>

---
 meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
 meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
 meta/conf/layer.conf                    |  1 -
 meta/lib/oe/utils.py                    |  3 +++
 meta/recipes-devtools/ccache/ccache.inc |  2 ++
 5 files changed, 43 insertions(+), 9 deletions(-)

meta/classes/ccache.bbclass +22 -3

index 9609020..f4eefce 100644
@@ -1,4 +1,14 @@
-CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
+#
+# Usage:
+# - Enable ccache
+#   Add the following line to a conffile such as conf/local.conf:
+#   INHERIT += "ccache"
+#
+# - Disable ccache for a recipe
+#   Add the following line to the recipe if it can't be built with ccache:
+#   CCACHE_DISABLE = '1'
+#
+
 export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
 
 # We need to stop ccache considering the current directory or the

@@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
 # ${PV} or ${PR} change.
 export CCACHE_NOHASHDIR ?= "1"
 
-DEPENDS_append_class-target = " ccache-native"
-DEPENDS[vardepvalueexclude] = " ccache-native"
+python() {
+    """
+    Enable ccache for the recipe if is not a dependency of ccache-native
+    """
+    ccache_native_deps = d.getVar('CCACHE_NATIVE_DEPENDS') or ''
+    ccache_native_deps += 'ccache-native'
+    if not (d.getVar('PN') in ccache_native_deps.split() or \
+            bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))):
+        d.appendVar('DEPENDS', ' ccache-native')
+        d.setVar('CCACHE', 'ccache ')
+}

meta/conf/bitbake.conf +11 -1

index 6480062..cb1dba8 100644
@@ -492,7 +492,7 @@ HOSTTOOLS += " \
 HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}"
 
 # Link to these if present
-HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
+HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
 
 # Temporary add few more detected in bitbake world
 HOSTTOOLS_NONFATAL += "join nl size yes zcat"

@@ -503,11 +503,22 @@ HOSTTOOLS_NONFATAL += "bzr"
 # Used by ssh fetcher
 HOSTTOOLS_NONFATAL += "scp"
 
+# This is used by both ccache.bbclass and ccahe-native recipe, these
+# recipes can't use ccache since ccache-native depends on them,
+# otherwise, there would be loop dependencies.
+CCACHE_NATIVE_DEPENDS ?= " \
+    gnu-config-native \
+    autoconf-native \
+    quilt-native \
+    zlib-native \
+    gettext-minimal-native \
+    texinfo-dummy-native \

Quoted 22 lines · Robert Yang

The work well now.

Signed-off-by Robert Yang <[email protected]>

---
  meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb | 3 ---
  meta/recipes-support/apr/apr_1.6.5.bb                 | 1 -
  2 files changed, 4 deletions(-)

meta/recipes-connectivity/openssl/openssl10_1.0.2q.bb +0 -3

index 6518dac..1986cf2 100644
@@ -76,9 +76,6 @@ EXTRA_OEMAKE = "${@bb.utils.contains('PACKAGECONFIG', 'manpages', '', 'OE_DISABL
 
  export OE_LDFLAGS = "${LDFLAGS}"
 
-# openssl fails with ccache: https://bugzilla.yoctoproject.org/show_bug.cgi?id=12810
-CCACHE = ""
-
  TERMIO ?= "-DTERMIO"
  TERMIO_libc-musl = "-DTERMIOS"
 

meta/recipes-support/apr/apr_1.6.5.bb CACHED_CONFIGUREVARS += "ac_cv_file__dev_zero=yes" · line 43

Quoted 9 lines

meta/recipes-support/apr/apr_1.6.5.bb +0 -1

index c2f5900..ed93a7e 100644
@@ -40,7 +40,6 @@ CACHED_CONFIGUREVARS += "ac_cv_file__dev_zero=yes"
  # Otherwise libtool fails to compile apr-utils
  # x86_64-linux-libtool: compile: unable to infer tagged configuration
  # x86_64-linux-libtool:   error: specify a tag with '--tag'
-CCACHE = ""

Remove the comment as well.

Quoted 11 lines

meta/recipes-support/apr/apr_1.6.5.bb +0 -0

  PACKAGECONFIG ??= "${@bb.utils.filter('DISTRO_FEATURES', 'ipv6', d)}"
  PACKAGECONFIG[ipv6] = "--enable-ipv6,--disable-ipv6,"

Signature · 7 lines

2.10.2

-- 
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Hi Robert,

Just a small nit-pick below..

Quoted 38 lines · Robert Yang

The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccahe is installed on the host:
   CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except ccache-native's dependencies, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear.

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++--- meta/conf/bitbake.conf                  | 21 ++++++++++++++++----- meta/conf/layer.conf                    |  1 - meta/lib/oe/utils.py                    |  3 +++ meta/recipes-devtools/ccache/ccache.inc |  2 ++

  5 files changed, 43 insertions(+), 9 deletions(-)

<snip>

Quoted 10 lines

meta/lib/oe/utils.py +0 -0

index 8a584d6..9c41705 100644
@@ -363,6 +363,9 @@ def host_gcc_version(d, taskcontextonly=False):

          return
      compiler = d.getVar("BUILD_CC")
+    # Get rid of ccache since it is not present when parsing. +    if compiler.startswith('ccache '):
+        compiler = compiler[7:]

You can avoid the if-condition and hard-coded 7-value by doing:

  # Get rid of ccache since it is no present when parsing. compiler = d.getVar("BUILD_CC").lstrip("ccache-")

// Martin

Quoted 14 lines

      try:
          env = os.environ.copy()
          env["PATH"] = d.getVar("PATH")

meta/recipes-devtools/ccache/ccache.inc +0 -0

index 6566328..259c196 100644
@@ -9,6 +9,8 @@ LICENSE = "GPLv3+"

  DEPENDS = "zlib"

meta/recipes-devtools/ccache/ccache.inc +2 -0

+DEPENDS_class-native = "${CCACHE_NATIVE_DEPENDS}"
+

  SRC_URI = "https://download.samba.org/pub/${BPN}/${BP}.tar.xz" inherit autotools

Quoted 10 lines · Robert Yang

The code is in setup_hosttools_dir(), it searches ccache in PATH in reverse order, I can't figure out what it is used for, maybe it is aimed to use host's ccache rather than recipe-sysroot- native/usr/bin/ccache, but the later one isn't in original PATH, so remove the code.

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/base.bbclass | 2 --
  1 file changed, 2 deletions(-)

We should document it and I think its still needed.

Some distros have a "gcc" which automatically wraps ccache behind the scenes. This code lets us find the real gcc and bypass ccache.

There would be oe-selftest failures on such a distro if we didn't do that.

Cheers,

Richard

Quoted 107 lines · Robert Yang

The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccahe is installed on the host:

   CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except ccache-native's dependencies, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear.

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++---
  meta/conf/bitbake.conf                  | 21 ++++++++++++++++-----
  meta/conf/layer.conf                    |  1 -
  meta/lib/oe/utils.py                    |  3 +++
  meta/recipes-devtools/ccache/ccache.inc |  2 ++
  5 files changed, 43 insertions(+), 9 deletions(-)

meta/classes/ccache.bbclass +22 -3

index 9609020..f4eefce 100644
@@ -1,4 +1,14 @@
-CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"
+#
+# Usage:
+# - Enable ccache
+#   Add the following line to a conffile such as conf/local.conf:
+#   INHERIT += "ccache"
+#
+# - Disable ccache for a recipe
+#   Add the following line to the recipe if it can't be built with ccache:
+#   CCACHE_DISABLE = '1'
+#
+
  export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
 
  # We need to stop ccache considering the current directory or the

@@ -7,5 +17,14 @@ export CCACHE_DIR ?= "${TMPDIR}/ccache/${MULTIMACH_TARGET_SYS}/${PN}"
  # ${PV} or ${PR} change.
  export CCACHE_NOHASHDIR ?= "1"
 
-DEPENDS_append_class-target = " ccache-native"
-DEPENDS[vardepvalueexclude] = " ccache-native"
+python() {
+    """
+    Enable ccache for the recipe if is not a dependency of ccache-native
+    """
+    ccache_native_deps = d.getVar('CCACHE_NATIVE_DEPENDS') or ''
+    ccache_native_deps += 'ccache-native'
+    if not (d.getVar('PN') in ccache_native_deps.split() or \
+            bb.utils.to_boolean(d.getVar('CCACHE_DISABLE'))):
+        d.appendVar('DEPENDS', ' ccache-native')
+        d.setVar('CCACHE', 'ccache ')
+}

meta/conf/bitbake.conf +16 -1

index 6480062..cb1dba8 100644
@@ -492,7 +492,7 @@ HOSTTOOLS += " \
  HOSTTOOLS += "${@'ip ping ps scp ssh stty' if (bb.utils.contains_any('IMAGE_CLASSES', 'testimage testsdk', True, False, d) or any(x in (d.getVar("BBINCLUDED") or "") for x in ["testimage.bbclass", "testsdk.bbclass"])) else ''}"
 
  # Link to these if present
-HOSTTOOLS_NONFATAL += "aws ccache gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
+HOSTTOOLS_NONFATAL += "aws gcc-ar gpg ld.bfd ld.gold nc pigz sftp socat ssh sudo"
 
  # Temporary add few more detected in bitbake world
  HOSTTOOLS_NONFATAL += "join nl size yes zcat"

@@ -503,11 +503,22 @@ HOSTTOOLS_NONFATAL += "bzr"
  # Used by ssh fetcher
  HOSTTOOLS_NONFATAL += "scp"
 
+# This is used by both ccache.bbclass and ccahe-native recipe, these
+# recipes can't use ccache since ccache-native depends on them,
+# otherwise, there would be loop dependencies.
+CCACHE_NATIVE_DEPENDS ?= " \
+    gnu-config-native \
+    autoconf-native \
+    quilt-native \
+    zlib-native \
+    gettext-minimal-native \
+    texinfo-dummy-native \
+    m4-native \
+    libtool-native \
+    automake-native \
+    xz-native \
+"

meta/conf/bitbake.conf HOSTTOOLS_NONFATAL += "bzr" · line 510

Quoted 5 lines

meta/conf/bitbake.conf +0 -4

  CCACHE ??= ""
-# ccache < 3.1.10 will create CCACHE_DIR on startup even if disabled, and
-# autogen sets HOME=/dev/null so in certain situations builds can fail.
-# Explicitly export CCACHE_DIR until we can assume ccache >3.1.10 on the host.
-export CCACHE_DIR ??= "${@os.getenv('HOME')}/.ccache"

Hardcoding a list of ccache-native dependencies is horrible and I do not want to do that.

Also, I want to stop the build up of unneeded 'stuff' in bitbake.conf. We should start making better use of inc files.

Please also ensure that the oe-selftest ccache tests continue to work after these changes. I haven't checked specifically here but we've had a lot of trouble with that before.

Cheers,

Richard

Quoted 15 lines · Richard Purdie

Quoted 9 lines · Robert Yang

The code is in setup_hosttools_dir(), it searches ccache in PATH in
reverse order, I can't figure out what it is used for, maybe it is
aimed to use host's ccache rather than recipe-sysroot- native/usr/bin/ccache, but the later one isn't in original PATH, so
remove the code.

Quoted 4 lines

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/base.bbclass | 2 --
  1 file changed, 2 deletions(-)

Quoted 5 lines

We should document it and I think its still needed.

Some distros have a "gcc" which automatically wraps ccache behind the
scenes. This code lets us find the real gcc and bypass ccache.

Ah, thanks for the explanations, I got it now. I think that we don't need the
code any more since we don't use host's ccache, and I have removed ccache from
HOSTTOOLS.

I will update commit message and send a V2.

Quoted 29 lines · Robert Yang

Quoted 15 lines · Richard Purdie

Quoted 9 lines · Robert Yang

The code is in setup_hosttools_dir(), it searches ccache in PATH in
reverse order, I can't figure out what it is used for, maybe it is
aimed to use host's ccache rather than recipe-sysroot- native/usr/bin/ccache, but the later one isn't in original PATH, so
remove the code.

Quoted 4 lines

Signed-off-by Robert Yang <[email protected]>

---
   meta/classes/base.bbclass | 2 --
   1 file changed, 2 deletions(-)

Quoted 5 lines

We should document it and I think its still needed.

Some distros have a "gcc" which automatically wraps ccache behind the
scenes. This code lets us find the real gcc and bypass ccache.

Ah, thanks for the explanations, I got it now. I think that we don't need the
code any more since we don't use host's ccache, and I have removed ccache from
HOSTTOOLS.

Quoted 1 line

I will update commit message and send a V2.

No, that isn't correct :(

The problem is that "gcc" can be linked directly to ccache by the host distro. This means you end up using the host ccache just by calling "gcc". This code is still needed if we want to control ccache and be able to disable it and use ccache-native for determinism.

Cheers,

Richard

Quoted 10 lines · On 12/11/18 5:44 PM, [email protected] wrote:

Quoted 1 line · Robert Yang

On 12/10/18 6:29 PM, Richard Purdie wrote:

No, that isn't correct :(

The problem is that "gcc" can be linked directly to ccache by the host
distro. This means you end up using the host ccache just by calling "gcc". This code is still needed if we want to control ccache and be
able to disable it and use ccache-native for determinism.

I did a test:

$ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
I got errors with/without the code:

$ bitbake -p
bb.BBHandledException
ERROR: Error running gcc  --version:

$ ls -l tmp/hosttools/gcc
tmp/hosttools/gcc -> /folk/lyang1/bin/gcc

$ ls -l /folk/lyang1/bin/gcc
/folk/lyang1/bin/gcc -> /usr/bin/ccache

It worked well if I run gcc --version manually:

$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
$ echo $?
0

There must be something wrong in such a case. I will fix this later.

Let's get back to he code:

srctool = bb.utils.which(path, tool, executable=True) if "ccache" in srctool:
srctool = bb.utils.which(path, tool,
executable=True, direction=1)
if srctool:
os.symlink(srctool, desttool)

* If tool == "gcc", then srctool would be gcc, too.
* If tool == "ccache", then srctool would be ccache, too.

So I think that the code doesn't help when gcc links to ccache ?

Quoted 48 lines · Robert Yang

Quoted 11 lines · On 12/11/18 5:44 PM, [email protected] wrote:

Quoted 2 lines · Robert Yang

On 12/10/18 6:29 PM, Richard Purdie wrote:

No, that isn't correct :(

The problem is that "gcc" can be linked directly to ccache by the host
distro. This means you end up using the host ccache just by calling "gcc". This code is still needed if we want to control ccache and be
able to disable it and use ccache-native for determinism.

I did a test:

$ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
I got errors with/without the code:

$ bitbake -p
bb.BBHandledException
ERROR: Error running gcc  --version:

$ ls -l tmp/hosttools/gcc
tmp/hosttools/gcc -> /folk/lyang1/bin/gcc

$ ls -l /folk/lyang1/bin/gcc
/folk/lyang1/bin/gcc -> /usr/bin/ccache

It worked well if I run gcc --version manually:

$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
$ echo $?
0

There must be something wrong in such a case. I will fix this later.

Let's get back to he code:

srctool = bb.utils.which(path, tool, executable=True) if "ccache" in srctool:
srctool = bb.utils.which(path, tool,
executable=True, direction=1)
if srctool:
os.symlink(srctool, desttool)

Quoted 4 lines

* If tool == "gcc", then srctool would be gcc, too.
* If tool == "ccache", then srctool would be ccache, too.

So I think that the code doesn't help when gcc links to ccache ?

https://www.unix.com/man-page/suse/1/ccache/

When the distros we tested on did this they placed it in something like bin/ccache/ so the code detects the "ccache" element of the path returned by which() (something like /usr/bin/ccache/gcc but I don't remember exactly).

Try using /folk/lyang1/ccache/ or /folk/lyang1/bin/ccache/ in your example.

Cheers,

Richard

Quoted 46 lines · Robert Yang

Quoted 10 lines · On 12/11/18 5:44 PM, [email protected] wrote:

Quoted 1 line · Robert Yang

On 12/10/18 6:29 PM, Richard Purdie wrote:

No, that isn't correct :(

The problem is that "gcc" can be linked directly to ccache by the host
distro. This means you end up using the host ccache just by calling "gcc". This code is still needed if we want to control ccache and be
able to disable it and use ccache-native for determinism.

I did a test:

$ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
I got errors with/without the code:

$ bitbake -p
bb.BBHandledException
ERROR: Error running gcc  --version:

$ ls -l tmp/hosttools/gcc
tmp/hosttools/gcc -> /folk/lyang1/bin/gcc

$ ls -l /folk/lyang1/bin/gcc
/folk/lyang1/bin/gcc -> /usr/bin/ccache

It worked well if I run gcc --version manually:

$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
$ echo $?
0

There must be something wrong in such a case. I will fix this later.

Let's get back to he code:

srctool = bb.utils.which(path, tool, executable=True) if "ccache" in srctool:
srctool = bb.utils.which(path, tool,
executable=True, direction=1)
if srctool:
os.symlink(srctool, desttool)

Quoted 4 lines

* If tool == "gcc", then srctool would be gcc, too.
* If tool == "ccache", then srctool would be ccache, too.

So I think that the code doesn't help when gcc links to ccache ?

https://www.unix.com/man-page/suse/1/ccache/

When the distros we tested on did this they placed it in something like bin/ccache/ so the code detects the "ccache" element of the path returned by which() (something like /usr/bin/ccache/gcc but I don't remember exactly).

Try using /folk/lyang1/ccache/ or /folk/lyang1/bin/ccache/ in your example.

Cheers,

Richard

Hi Robert,

Just a small nit-pick below..

Quoted 38 lines · Robert Yang

The previous ccache.bbclass has the following problems:
- It uses host's ccache for native recipes, but this may not work on some hosts, for example, it nerver works on my Ubuntu 14.04.4, there are always build failures (m4-native failed at do_configure, and others will also be failed if I disable CCACHE for m4-native)

- native/nativesdk/cross/crosssdk recipes use host's ccahe, but target uses ccache-native, this may confuse user.

- The target recipes may use both host's ccache and ccache-native, this may cause unexpected problems and hard to debug. This is because ccache-native is in SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS, so ccache-native may not be present when rebuild target recipes, and then it would use hosttools/ccache, but the previous ccache files were generated by ccache-native.

- Target recipes can't use ccache when no ccahe is installed on the host:
   CCACHE = "${@bb.utils.which(d.getVar('PATH'), 'ccache') and 'ccache '}"

After refactored:
All types recipes (native, target and others) will use ccache-native except ccache-native's dependencies, host's cache won't be used any more. It is more reliable now, which will work everywhere when ccache-native can be built.

And now we need use "CCACHE_DISABLE = '1'" to disable ccahe for the recipe rather than "CCACHE = ''" since we set CCACHE in anonymous function, and d.getVar('CCACHE') works after "CCACHE ??=" which is set in bitbake.conf, so we can't check whether CCACHE is set or not in anonymous function since it is always set. Use CCACHE_DISABLE to disable it would be more clear.

Signed-off-by Robert Yang <[email protected]>

---
  meta/classes/ccache.bbclass             | 25 ++++++++++++++++++++++--- meta/conf/bitbake.conf                  | 21 ++++++++++++++++----- meta/conf/layer.conf                    |  1 - meta/lib/oe/utils.py                    |  3 +++ meta/recipes-devtools/ccache/ccache.inc |  2 ++

  5 files changed, 43 insertions(+), 9 deletions(-)

<snip>

Quoted 10 lines

meta/lib/oe/utils.py +0 -0

index 8a584d6..9c41705 100644
@@ -363,6 +363,9 @@ def host_gcc_version(d, taskcontextonly=False):

          return
      compiler = d.getVar("BUILD_CC")
+    # Get rid of ccache since it is not present when parsing. +    if compiler.startswith('ccache '):
+        compiler = compiler[7:]

You can avoid the if-condition and hard-coded 7-value by doing:

  # Get rid of ccache since it is no present when parsing. compiler = d.getVar("BUILD_CC").lstrip("ccache-")

// Martin

Quoted 14 lines

      try:
          env = os.environ.copy()
          env["PATH"] = d.getVar("PATH")

meta/recipes-devtools/ccache/ccache.inc +0 -0

index 6566328..259c196 100644
@@ -9,6 +9,8 @@ LICENSE = "GPLv3+"

  DEPENDS = "zlib"

meta/recipes-devtools/ccache/ccache.inc +2 -0

+DEPENDS_class-native = "${CCACHE_NATIVE_DEPENDS}"
+

  SRC_URI = "https://download.samba.org/pub/${BPN}/${BP}.tar.xz" inherit autotools

Hi RP,

Quoted 1 line · On 12/11/18 5:44 PM, [email protected] wrote:

On Tue, 2018-12-11 at 17:24 +0800, Robert Yang wrote:

I did a test:

$ ln -s /usr/bin/ccache /folk/lyang1/bin/gcc
I got errors with/without the code:

$ bitbake -p
bb.BBHandledException
ERROR: Error running gcc  --version:

$ ls -l tmp/hosttools/gcc
tmp/hosttools/gcc -> /folk/lyang1/bin/gcc

$ ls -l /folk/lyang1/bin/gcc
/folk/lyang1/bin/gcc -> /usr/bin/ccache

It worked well if I run gcc --version manually:

$ gcc --version
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
$ echo $?
0

There must be something wrong in such a case. I will fix this later.

Let's get back to he code:

            srctool = bb.utils.which(path, tool, executable=True)
            if "ccache" in srctool:
                srctool = bb.utils.which(path, tool, executable=True, direction=1) if srctool:
                os.symlink(srctool, desttool)

* If tool == "gcc", then srctool would be gcc, too.
* If tool == "ccache", then srctool would be ccache, too.

So I think that the code doesn't help when gcc links to ccache ?

// Robert

Quoted 8 lines

Cheers,

Richard