New: Finding non-relative paths in the ccache

https://bugs.busybox.net/show_bug.cgi?id=10536

Bug ID: 10536
Summary: Finding non-relative paths in the ccache
Product: buildroot
Version: unspecified
Hardware: All
OS: Linux
Status: NEW
Severity: normal
Priority: P5
Component: Other
Assignee: unassigned at buildroot.uclibc.org
Reporter: mhirota at impinj.com
CC: buildroot at uclibc.org
Target Milestone: ---

It appears that non-relative paths are leaking into the buildroot ccache despite the default BR2_CCACHE_USE_BASEDIR=y.

Here are my steps to demonstrate this:

    $ git clone git://git.buildroot.net/buildroot
    $ cd buildroot
    $ make zynq_zybo_defconfig
    $ make menuconfig
    $ # Build options -->
    $ #   [*] Enable compiler cache
    $ #   < Save > and < Exit > back to prompt
    $ make

At this point, I grep the ccache for anything that starts with "/home":

    $ grep -r --include "*.d" "^\/home" $HOME/.buildroot-ccache/ | wc -l
    647

I was expecting this to be zero.

Just to confirm some assumptions:

    $ grep -r --include "*.d" "^\.\." $HOME/.buildroot-ccache/ | wc -l
    60840
    $ grep CCACHE .config
    BR2_CCACHE=y
    BR2_CCACHE_DIR="$(HOME)/.buildroot-ccache"
    BR2_CCACHE_INITIAL_SETUP=""
    BR2_CCACHE_USE_BASEDIR=y
    # BR2_PACKAGE_CCACHE is not set

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #1 from Trent Piepho <[email protected]> --- Examining the build of objects that have non-relative paths into the buildroot output directory in ccache shows how they are getting created. Here's an example from host-uboot-tools:

/work/build-system/output/r700/host/bin/ccache /usr/lib64/ccache/gcc
-Wp,-MD,tools/.mkimage.o.d -O2 -I/work/build-system/output/r700/host/include -include ./include/libfdt_env.h -idirafterinclude -idirafter./arch//include -I./lib/libfdt -I./tools -DUSE_HOSTCC -D__KERNEL_STRICT_NAMES -D_GNU_SOURCE -c -o tools/mkimage.o tools/mkimage.c

Since this is building a host binary it doesn't not call the buildroot toolchain wrapper. The toolchain wrapper is what sets basedir for ccache and so ccache will not make absolute paths into the buildroot output directory tree relative.

I think in order to see this one needs to build a host binary, have the package's build system for the host binary use buildroot's cccache, have it produce an auto-dependency file (-MD, etc), and have the code use a header that in the buildroot output tree, i.e. some part of a host library package.

Perhaps this could be fixed if CCACHE_BASEDIR was injected in a way that also managed to apply it to host compilations?

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #2 from Trent Piepho <[email protected]> ---
This ends up being a big problem if multiple buildroot output directories are used. For example:

make O=first some_defconfig ; make O=first all rm -rf first
make O=second some_defconfig ; make O=second all

The build in second fails because ccache will use cached dependency files from the first build, and these cached files reference headers with a "*/first/*" path and those headers no longer exist.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

Thomas Petazzoni <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
Assignee|unassigned at buildroot.uclibc |arnout at mind.be
|.org |

--- Comment #3 from Thomas Petazzoni <[email protected]> --- Arnout, since you're looking after all the ccache related topics, could you have a look at this? Thanks!

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #4 from Arnout Vandecappelle <[email protected]> --- Does the build really fail in that second example? Since the first/ directory is in the command-line arguments during the build and it doesn't match BR2_CCACHE_DIR, ccache should just treat it as a cache miss. So indeed, ccache doesn't work for host packages unless the output directory is the same.

We could actually easily export CCACHE_BASEDIR from make, so that it works also for host packages. We could perhaps even remove it from the toolchain wrapper then, though it's probably convenient that the same basedir is used when building outside of Buildroot.

That said, this problem will still exist when a BR2_HOST_DIR is specified that points outside of the output directory. In that case we'd need two different 'base directories', which is obviously not possible...

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #5 from Mark Hirota <[email protected]> ---
Arnout: yes, we have seen a failure as Trent describes, but with 2 similar
configs: r700 and r700_rescue.

PATH="/mnt/work/output/r700/host/bin:/mnt/work/output/r700/host/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" /usr/bin/make -j5 -C /mnt/work/output/r700/build/uboot-tools-2017.09 CROSS_COMPILE="/mnt/work/output/r700/host/bin/arm-buildroot-linux-gnueabihf-"
CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -O3
-fstack-protector-strong" LDFLAGS=""
STRIP=/mnt/work/output/r700/host/bin/arm-buildroot-linux-gnueabihf-strip
CROSS_BUILD_TOOLS=y tools-only

  CHK     include/config/uboot.release
  CHK     include/generated/timestamp_autogenerated.h
  HOSTCC  scripts/basic/fixdep
  UPD     include/generated/timestamp_autogenerated.h
  UPD     include/config/uboot.release
  CHK     include/generated/version_autogenerated.h
  UPD     include/generated/version_autogenerated.h
  HOSTCC  tools/mkenvimage.o
  WRAP    tools/lib/crc32.c
  HOSTCC  tools/os_support.o
  HOSTCC  tools/aisimage.o
  HOSTCC  tools/atmelimage.o
  WRAP    tools/common/bootm.c
  HOSTCC  tools/default_image.o

fixdep: error opening config file:
/mnt/work/output/r700_rescue/host/lib/gcc/arm-buildroot-linux-gnueabihf/7.2.0/include/stddef.h:
No such file or directory

Our build automation performs clean builds, but shares a buildroot-ccache across them. This type of failure forces us to build the projects on separate build hosts to avoid hitting it.

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #6 from Arnout Vandecappelle <[email protected]> ---
Hm, this build failure doesn't look like it will be solved with CCACHE_BASEDIR. Indeed, it is the *target* ccache'd compiler that generated a *.d file that apparently still has a full path in it. I don't see how this could happen...

Possibly the issue is that this is a path that is embedded in the specs file, and ccache will only substitute paths that get passed on the command line (through -I etc.).

Needs more debugging...

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #7 from Mark Hirota <[email protected]> ---
Arnout: I believe the issue is with ccache 3.3.4. I see in the 3.3.5 release notes (https://ccache.samba.org/releasenotes.html#_ccache_3_3_5) the following:

Should the ccache version in buildroot be updated to 3.3.5?

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #8 from Arnout Vandecappelle <[email protected]> ---
Good catch! Can you try updating ccache, check if that fixes things for you, and if so send the patch to the list with a reference to this bug in the commit log?

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

Peter Korsgaard <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

         Resolution|---                         |FIXED
             Status|NEW                         |RESOLVED

--- Comment #9 from Peter Korsgaard <[email protected]> --- Fixed in git by bumping ccache to 3.3.5

You are receiving this mail because:
You are on the CC list for the bug.

https://bugs.busybox.net/show_bug.cgi?id=10536

--- Comment #10 from Trent Piepho <[email protected]> ---
I seems like Arnout should be right about host compiles in different directories not sharing ccache. Since CCACHE_BASEDIR is not set, the paths should be absolute and different directories would be a cache miss.

But here's some testing with host-uboot-tools. First a build from a cleared and reset ccache:
cache hit (direct) 0
cache hit (preprocessed) 0
cache miss 58
cache hit rate 0.00 %
called for link 8
cleanups performed 0
files in cache 171

As expected, all misses. 58 files.

Building again from the same output dir:
cache hit (direct) 53
cache hit (preprocessed) 5
cache miss 58
cache hit rate 50.00 %
called for link 16
cleanups performed 0
files in cache 176

Exactly 58 new hits, as expected, everything in cache. But 5 new files, odd.

Now the tricky one, building from a different output dir:
cache hit (direct) 53
cache hit (preprocessed) 15
cache miss 106
cache hit rate 39.08 %
called for link 24
cleanups performed 0
files in cache 331

These should be all misses, right? But there were 10 hits in preprocessed mode! Which is fair, if the preprocessed code ends up being exactly the same. But just because the c code is the same doesn't mean the auto-dependency file is the same (absolute paths in the .d file differ!). If ccache tries to use the cached .d file with absolute paths on this supposed preprocessed hit, it will fail.

You are receiving this mail because:
You are on the CC list for the bug.