Internal toolchain wrapper & ccache support
Subsystems (2)
v2 cover Internal toolchain wrapper & ccache support
This series introduces two features that were discussed on the list and in the Buildroot Summer Camp: a wrapper for the internal toolchain, and fixing ccache. See below why the two are combined in one series.
[Note: to make review easier, all the potentially contentious stuff from the individual patches is repeated here. So for the first few iterations, it's probably enough to reply to just this e-mail.]
The idea of a wrapper for the internal toolchain has been floating around for a while now. It allows us to make sure that BR2_TARGET_OPTIMIZATION and BR2_TARGET_LDFLAGS are passed down to the compiler and linker even if the package build system doesn't use CFLAGS and LDFLAGS. This is something which is not tested in the autobuilders (they never set these options) so probably there are packages out there that are not built with these flags.
A second reason for the wrapper is to be able to use the unsafe path check also for internal toolchains without patching gcc.
More recently, and additional motivation for an internal toolchain wrapper was discovered: it makes the support for a per-package staging directory significantly simpler. See the discussion in [1][2].
And finally, the wrapper makes the ccache handling simpler, as explained below.
To create the wrapper for the internal toolchain, this series refactors as much as possible with the external toolchain. Therefore, the first few patch moves things around a bit as an enabler of the internal toolchain wrapper. The same patch also does some related cleanups.
The second patch adds the toolchain wrapper for the internal toolchain. It uses the same toolchain-wrapper.c as the external toolchain. The wrapper is created both for gcc-initial and gcc-final. It is rebuilt for gcc-final, which is not exactly
The real gcc, g++, etc. executables are moved to gcc.real etc. and a symlink is created to toolchain-wrapper. This requires the wrapper code to be extended with an additional BR_CROSS_PATH_SUFFIX to find the real executable. I could have reused the same approach of the external toolchain, installing in /opt, but that would require copying a bunch files around as well and overall would be much more invasive, so I opted for this simpler change.
The creation of the wrapper symlinks is covered by a fairly complex case-condition which is similar to the external toolchain wrapper. I didn't refactor these two commands, because there are suble differences that would make the refactored code very complicated.
The third patch implements the second use case: it removes the now-redundant gcc patches that add the unsafe path check. This check is now handled by the toolchain wrapper. Note that the patches for binutils are still needed, since ld isn't wrapped. We actually should also wrap cpp and ld to cover the first use case.
The first use case (passing TARGET_CFLAGS through the wrapper) isn't implemented yet because it is a bit complicated, because some packages do _not_ want TARGET_CFLAGS, e.g. gcc and linux. So some infrastructure needs to be added that allows overriding TARGET_CFLAGS.
The third use case should be handled by Fabio in his per-package staging series.
The fourth use case brings us to the ccache problem.
The problem with our current ccache is that we disable hashing of the compiler executable itself. If we don't do that, using ccache is pointless since we always rebuild the compiler (for the external toolchain we rebuild the wrapper) so the cache would always miss. But in the current situation, if a user changes the compiler configuration (which would result in the compiler generating different object files than before) and does 'make clean all', ccache may in fact reuse object files from the previous run. This rarely gives problems, because
(1) the cache expires quite quickly (it's only 1GB by default),
(2) radically changing compiler options will cause cache misses because different header files are used,
(3) many compiler changes (e.g. changing -mtune) have little practical effect because the resulting code is usually still compatible,
(4) we currently don't use CCACHE_BASEDIR, and almost all object files will contain an absolute path (e.g. in debug info), so when building in a different directory, most of it will miss,
(5) we do mostly build test, and many of the potential problems only appear at runtime.
Still, when ccache _does_ use the wrong cached object files, the effects are really weird and hard to debug. Also, we want reproducible builds and obviously the above makes builds non-reproducible. So we have a FAQ entry that warns against using ccache and tells the user to clear the cache in case of problems.
To fix the situation, this series sets the CCACHE_COMPILERCHECK environment variable to a hash of the compiler options. I already analyzed the options that should be checked a while ago in [3].
Unfortunately, it's not so simple, since for the host compiler we should not use the same hash: on the one hand, it's possible that the host compiler has changed while the target compiler options are still the same, and on the other hand when we change the target compiler options, the cached host object files are still valid.
My first attempt was to create a ccache wrapper that sets the CCACHE_COMPILERCHECK environment variable, and compiles it differently for host and target (i.e. use ccache-host for host compilation, and ccache-target for target compilation). However, it was very difficult to insert the calculation of the target compiler hash correctly in the dependency chain. The calculation of the hash really belongs to the toolchain build, but ccache is already built before everything else. It's much simpler, instead, to use the toolchain wrapper and insert ccache in there. That's why these two features (internal toolchain wrapper and ccache fix) are combined in a single series.
The ccache part of the series starts with moving the ccache handling into the toolchain wrapper. This allows a lot of follow-up patches to remove ccache handling in individual packages, including some patches to handle the fact that CC has multiple words. These patches were tested by doing a ccache-enabled build of the affected packages, then cleaning them, then rebuilding, and checking the evolution of the ccache stats while building. If the number of cache hits is increasing significantly faster than the number of cache misses, all is well.
Moving the ccache handling into the toolchain wrapper has one nasty side effect: when you change the value of BR2_CCACHE, this will have no effect unless host-gcc-final is rebuilt. Of course, we don't really support the incremental build scenario so it's not such a big deal. Still, I've added a patch (marked RFC) to support this scenario, by exporting a BR_NO_CCACHE that tells the wrapper to skip ccache. Note that this works only to turn ccache off, but I think that's the most important use case.
Once ccache moves to the wrapper, we can fix the compilercheck. All of that is done in a single commit - I didn't think splitting it up was really worthwhile. It reverts the default compilercheck to 'mtime' like it is in upstream, and it adds an option to the toolchain wrapper to set the CCACHE_COMPILERCHECK environment variable. The hash is calculated based on the config options passed to gcc and on the sources and patches.
A final cleanup step, also marked RFC, is to support changing the output directory. Many of the build commands have a -I argument pointing somewhere in the staging directory, so when you rebuild in a different directory, this will mot match and the cache isn't used. ccache has the CCACHE_BASEDIR potion to deal with that, but it's not bullet-proof so I added a Kconfig option for it.
It would be really nice if someone else could take over this series, since it is rather 'deep' stuff and if someone else really gets their hands dirty on this code, there's a better chance of catching some issues that I overlooked.
Changelog:
v2: - Rebased on master.
- Updated comments to remove the 'external' from the wrapper.
- Include BR2_GLOBAL_PATCH_DIR in the wrapper hash; also use $(wildcard ...) for all patches.
- Remove --enable-poison-system-directories from gcc-final configure.
- Explain in commit message why gcc-initial also needs the wrapper.
[1] http://lists.busybox.net/pipermail/buildroot/2015-March/121005.html
[2] http://lists.busybox.net/pipermail/buildroot/2015-June/131051.html
[3] http://lists.busybox.net/pipermail/buildroot/2015-April/127341.html
The following changes since commit 0611258aeae3b2e311b1c8fbebd337c9567830fb:
package/freerdp: needs C++ (2015-10-04 14:16:27 +0200)
are available in the git repository at:
https://github.com/arnout/buildroot.git toolchain-wrapper-2
for you to fetch changes up to 11995e8efc5b34d78a45a1ea44bcb9cbdc4e53d3:
[RFC] ccache: support changing the output directory (2015-10-04 13:23:38 +0100)
----------------------------------------------------------------
Arnout Vandecappelle (Essensium/Mind) (18):
toolchain-external: move wrapper to toolchain directory
gcc: use toolchain wrapper
gcc: remove unsafe patch check (poison system dirs) patch
infra: move ccache handling to the toolchain wrapper
perl: Remove ccache handling
imx-lib: remove now-redundant ccache handling
imx-vpu: remove now-redundant ccache handling
linux: remove now-redundant ccache handling
uboot: remove now-redundant ccache handling
barebox: remove now-redundant ccache handling
cryptodev-linux: remove now-redundant fix-ccache-compile patch
qt5base: remove now-redundant ccache handling
package-cmake: remove now-redundant target ccache support
qpid-proton: remove now-redundant ccache handling patch Makefile.in: remove now-unused TARGET_CC/CXX_NOCCACHE
toolchain-wrapper: support change of BR2_CCACHE
ccache: use mtime for external toolchain, CONF_OPTS for internal toolchain [RFC] ccache: support changing the output directory
Config.in | 27 +++
Makefile | 2 +
boot/barebox/barebox.mk | 3 +-
boot/uboot/uboot.mk | 2 +-
docs/manual/ccache-support.txt | 20 ++
linux/linux.mk | 2 +-
package/Makefile.in | 8 -
package/ccache/ccache.mk | 5 -
.../cryptodev-linux/0002-fix-ccache-compile.patch | 20 --
package/freescale-imx/imx-lib/imx-lib.mk | 2 +-
package/freescale-imx/imx-vpu/imx-vpu.mk | 2 +-
.../4.7.4/910-gcc-poison-system-directories.patch | 207 -------------------
.../4.8.5/910-gcc-poison-system-directories.patch | 207 -------------------
.../4.9.3/910-gcc-poison-system-directories.patch | 207 -------------------
.../5.2.0/200-gcc-poison-system-directories.patch | 207 -------------------
.../910-gcc-poison-system-directories.patch | 221 ---------------------
package/gcc/gcc-final/gcc-final.mk | 14 +-
package/gcc/gcc-initial/gcc-initial.mk | 4 +
package/gcc/gcc.mk | 65 +++++-
package/perl/perl.mk | 5 +-
package/pkg-cmake.mk | 5 +-
...fix-C-compiler-detection-with-_ARG1-_ARG2.patch | 52 -----
package/qt5/qt5base/0002-mkspecs-files.patch | 6 +-
.../qt5/qt5base/0009-fix-build-with-ccache.patch | 49 -----
package/qt5/qt5base/qt5base.mk | 1 -
support/misc/toolchainfile.cmake.in | 32 +--
toolchain/toolchain-external/toolchain-external.mk | 55 ++---
...ext-toolchain-wrapper.c => toolchain-wrapper.c} | 51 ++++-
toolchain/toolchain-wrapper.mk | 34 ++++
29 files changed, 231 insertions(+), 1284 deletions(-)
delete mode 100644 package/cryptodev-linux/0002-fix-ccache-compile.patch
delete mode 100644 package/gcc/4.7.4/910-gcc-poison-system-directories.patch
delete mode 100644 package/gcc/4.8.5/910-gcc-poison-system-directories.patch
delete mode 100644 package/gcc/4.9.3/910-gcc-poison-system-directories.patch
delete mode 100644 package/gcc/5.2.0/200-gcc-poison-system-directories.patch
delete mode 100644 package/gcc/arc-2015.06/910-gcc-poison-system-directories.patch
delete mode 100644 package/qpid-proton/0001-proton-c-fix-C-compiler-detection-with-_ARG1-_ARG2.patch
delete mode 100644 package/qt5/qt5base/0009-fix-build-with-ccache.patch
rename toolchain/{toolchain-external/ext-toolchain-wrapper.c => toolchain-wrapper.c} (81%)
create mode 100644 toolchain/toolchain-wrapper.mkPATCH v2 1/18 toolchain-external: move wrapper to toolchain directory
The toolchain wrapper will be reused for the internal toolchain, so it belongs in the toolchain directory. Also, the ext- prefix is removed from it. The build commands are moved to a new toolchain-wrapper.mk.
The wrapper arguments that are also relevant for the internal toolchain wrapper are moved to toolchain-wrapper.mk, the rest stays in toolchain-external.mk.
While we're at it, move the building of the toolchain wrapper to the build step of toolchain-external. There is no specific reason to do this, other than that it fits better semantically. Also remove the MESSAGE call, otherwise we'd see:
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
toolchain-external undefined Building toolchain-external undefined Building toolchain wrapper
/usr/bin/gcc ...
Having an extra "Building toolchain wrapper' message is pointless.
The useless condition on $(BR2_TARGET_OPTIMIZATION) is removed. It was always true because it wasn't qstrip'ped first, so clearly it works without that condition as well.
Also rewrapped some comments and removed the 'external' reference.
---
v2: Remove references to 'external' in comments [Romain]
Rewrap some comments [Arnout]
---
toolchain/toolchain-external/toolchain-external.mk | 55 +++++++---------------
...ext-toolchain-wrapper.c => toolchain-wrapper.c} | 6 +--
toolchain/toolchain-wrapper.mk | 26 ++++++++++
3 files changed, 47 insertions(+), 40 deletions(-)
rename toolchain/{toolchain-external/ext-toolchain-wrapper.c => toolchain-wrapper.c} (96%)
create mode 100644 toolchain/toolchain-wrapper.mktoolchain/toolchain-external/toolchain-external.mk +18 -37
@@ -153,15 +153,14 @@ TOOLCHAIN_EXTERNAL_CROSS = $(TOOLCHAIN_EXTERNAL_BIN)/$(TOOLCHAIN_EXTERNAL_PREFIX TOOLCHAIN_EXTERNAL_CC = $(TOOLCHAIN_EXTERNAL_CROSS)gcc TOOLCHAIN_EXTERNAL_CXX = $(TOOLCHAIN_EXTERNAL_CROSS)g++ TOOLCHAIN_EXTERNAL_READELF = $(TOOLCHAIN_EXTERNAL_CROSS)readelf -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS = -DBR_SYSROOT='"$(STAGING_SUBDIR)"' ifeq ($(filter $(HOST_DIR)/%,$(TOOLCHAIN_EXTERNAL_BIN)),) # TOOLCHAIN_EXTERNAL_BIN points outside HOST_DIR => absolute path -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += \ +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += \ -DBR_CROSS_PATH_ABS='"$(TOOLCHAIN_EXTERNAL_BIN)"' else # TOOLCHAIN_EXTERNAL_BIN points inside HOST_DIR => relative path -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += \ +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += \ -DBR_CROSS_PATH_REL='"$(TOOLCHAIN_EXTERNAL_BIN:$(HOST_DIR)/%=%)"' endif
@@ -180,59 +179,54 @@ CC_TARGET_MODE_ := $(call qstrip,$(BR2_GCC_TARGET_MODE)) # to select the right multilib variant ifeq ($(BR2_x86_64),y) TOOLCHAIN_EXTERNAL_CFLAGS += -m64 -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_64 +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_64 endif ifneq ($(CC_TARGET_ARCH_),) TOOLCHAIN_EXTERNAL_CFLAGS += -march=$(CC_TARGET_ARCH_) -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_ARCH='"$(CC_TARGET_ARCH_)"' +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_ARCH='"$(CC_TARGET_ARCH_)"' endif ifneq ($(CC_TARGET_CPU_),) TOOLCHAIN_EXTERNAL_CFLAGS += -mcpu=$(CC_TARGET_CPU_) -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_CPU='"$(CC_TARGET_CPU_)"' +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_CPU='"$(CC_TARGET_CPU_)"' endif ifneq ($(CC_TARGET_ABI_),) TOOLCHAIN_EXTERNAL_CFLAGS += -mabi=$(CC_TARGET_ABI_) -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_ABI='"$(CC_TARGET_ABI_)"' +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_ABI='"$(CC_TARGET_ABI_)"' endif ifneq ($(CC_TARGET_FPU_),) TOOLCHAIN_EXTERNAL_CFLAGS += -mfpu=$(CC_TARGET_FPU_) -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_FPU='"$(CC_TARGET_FPU_)"' +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_FPU='"$(CC_TARGET_FPU_)"' endif ifneq ($(CC_TARGET_FLOAT_ABI_),) TOOLCHAIN_EXTERNAL_CFLAGS += -mfloat-abi=$(CC_TARGET_FLOAT_ABI_) -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_FLOAT_ABI='"$(CC_TARGET_FLOAT_ABI_)"' +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_FLOAT_ABI='"$(CC_TARGET_FLOAT_ABI_)"' endif ifneq ($(CC_TARGET_MODE_),) TOOLCHAIN_EXTERNAL_CFLAGS += -m$(CC_TARGET_MODE_) -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_MODE='"$(CC_TARGET_MODE_)"' +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_MODE='"$(CC_TARGET_MODE_)"' endif ifeq ($(BR2_BINFMT_FLAT),y) TOOLCHAIN_EXTERNAL_CFLAGS += -Wl,-elf2flt -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_BINFMT_FLAT +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_BINFMT_FLAT endif ifeq ($(BR2_mipsel)$(BR2_mips64el),y) -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_MIPS_TARGET_LITTLE_ENDIAN +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_MIPS_TARGET_LITTLE_ENDIAN TOOLCHAIN_EXTERNAL_CFLAGS += -EL endif ifeq ($(BR2_mips)$(BR2_mips64),y) -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_MIPS_TARGET_BIG_ENDIAN +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_MIPS_TARGET_BIG_ENDIAN TOOLCHAIN_EXTERNAL_CFLAGS += -EB endif ifeq ($(BR2_arceb),y) -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_ARC_TARGET_BIG_ENDIAN +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_ARC_TARGET_BIG_ENDIAN TOOLCHAIN_EXTERNAL_CFLAGS += -EB endif -ifneq ($(BR2_TARGET_OPTIMIZATION),) + TOOLCHAIN_EXTERNAL_CFLAGS += $(call qstrip,$(BR2_TARGET_OPTIMIZATION)) -# We create a list like '"-mfoo", "-mbar", "-mbarfoo"' so that each -# flag is a separate argument when used in execv() by the external -# toolchain wrapper. -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_ADDITIONAL_CFLAGS='$(foreach f,$(call qstrip,$(BR2_TARGET_OPTIMIZATION)),"$(f)",)' -endif ifeq ($(BR2_SOFT_FLOAT),y) TOOLCHAIN_EXTERNAL_CFLAGS += -msoft-float -TOOLCHAIN_EXTERNAL_WRAPPER_ARGS += -DBR_SOFTFLOAT=1 +TOOLCHAIN_EXTERNAL_TOOLCHAIN_WRAPPER_ARGS += -DBR_SOFTFLOAT=1 endif # The Linaro ARMhf toolchain expects the libraries in
@@ -688,15 +682,6 @@ define TOOLCHAIN_EXTERNAL_INSTALL_BFIN_FLAT endef endif -# We use --hash-style=both to increase the compatibility of -# the generated binary with older platforms, except for MIPS, -# where the only acceptable hash style is 'sysv' -ifeq ($(findstring mips,$(HOSTARCH)),mips) -TOOLCHAIN_EXTERNAL_WRAPPER_HASH_STYLE = sysv -else -TOOLCHAIN_EXTERNAL_WRAPPER_HASH_STYLE = both -endif - # Build toolchain wrapper for preprocessor, C and C++ compiler and setup # symlinks for everything else. Skip gdb symlink when we are building our # own gdb to prevent two gdb's in output/host/usr/bin.
@@ -707,8 +692,6 @@ endif # match the *cc-* pattern. Therefore, an additional case is added for *-ar, # *-ranlib and *-nm. define TOOLCHAIN_EXTERNAL_INSTALL_WRAPPER - $(Q)$(call MESSAGE,"Building ext-toolchain wrapper") - $(Q)mkdir -p $(HOST_DIR)/usr/bin $(Q)cd $(HOST_DIR)/usr/bin; \ for i in $(TOOLCHAIN_EXTERNAL_CROSS)*; do \ base=$${i##*/}; \
@@ -717,7 +700,7 @@ define TOOLCHAIN_EXTERNAL_INSTALL_WRAPPER ln -sf $$(echo $$i | sed 's%^$(HOST_DIR)%../..%') .; \ ;; \ *cc|*cc-*|*++|*++-*|*cpp) \ - ln -sf ext-toolchain-wrapper $$base; \ + ln -sf toolchain-wrapper $$base; \ ;; \ *gdb|*gdbtui) \ if test "$(BR2_PACKAGE_HOST_GDB)" != "y"; then \
@@ -729,10 +712,6 @@ define TOOLCHAIN_EXTERNAL_INSTALL_WRAPPER ;; \ esac; \ done - $(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_EXTERNAL_WRAPPER_ARGS) \ - -s -Wl,--hash-style=$(TOOLCHAIN_EXTERNAL_WRAPPER_HASH_STYLE) \ - toolchain/toolchain-external/ext-toolchain-wrapper.c \ - -o $(HOST_DIR)/usr/bin/ext-toolchain-wrapper endef # This sed magic is taken from Linux headers_install.sh script.
@@ -771,6 +750,8 @@ define TOOLCHAIN_EXTERNAL_FIXUP_UCLIBCNG_LDSO fi endef +TOOLCHAIN_EXTERNAL_BUILD_CMDS = $(TOOLCHAIN_BUILD_WRAPPER) + define TOOLCHAIN_EXTERNAL_INSTALL_STAGING_CMDS $(TOOLCHAIN_EXTERNAL_INSTALL_SYSROOT_LIBS) $(TOOLCHAIN_EXTERNAL_INSTALL_WRAPPER)
toolchain/toolchain-wrapper.c renamed from toolchain/toolchain-external/ext-toolchain-wrapper.c +3 -3
@@ -1,7 +1,7 @@/** -* Buildroot wrapper for external toolchains. This simply executes the real -* toolchain with a number of arguments (sysroot/arch/..) hardcoded, -* to ensure the external toolchain uses the correct configuration. +* Buildroot wrapper for toolchains. This simply executes the real toolchain +* with a number of arguments (sysroot/arch/..) hardcoded, to ensure the +* toolchain uses the correct configuration. * The hardcoded path arguments are defined relative to the actual location * of the binary. *
toolchain/toolchain-wrapper.mk new file +26 -0
@@ -0,0 +1,26 @@ +# This file contains the definition of the toolchain wrapper build commands + +# We use --hash-style=both to increase the compatibility of the generated +# binary with older platforms, except for MIPS, where the only acceptable +# hash style is 'sysv' +ifeq ($(findstring mips,$(HOSTARCH)),mips) +TOOLCHAIN_WRAPPER_HASH_STYLE = sysv +else +TOOLCHAIN_WRAPPER_HASH_STYLE = both +endif + +TOOLCHAIN_WRAPPER_ARGS = $($(PKG)_TOOLCHAIN_WRAPPER_ARGS) +TOOLCHAIN_WRAPPER_ARGS += -DBR_SYSROOT='"$(STAGING_SUBDIR)"' + +# We create a list like '"-mfoo", "-mbar", "-mbarfoo"' so that each flag is a +# separate argument when used in execv() by the toolchain wrapper. +TOOLCHAIN_WRAPPER_ARGS += -DBR_ADDITIONAL_CFLAGS='$(foreach f,$(call qstrip,$(BR2_TARGET_OPTIMIZATION)),"$(f)",)' + +# For simplicity, build directly into the install location +define TOOLCHAIN_BUILD_WRAPPER + $(Q)mkdir -p $(HOST_DIR)/usr/bin + $(HOSTCC) $(HOST_CFLAGS) $(TOOLCHAIN_WRAPPER_ARGS) \ + -s -Wl,--hash-style=$(TOOLCHAIN_WRAPPER_HASH_STYLE) \ + toolchain/toolchain-wrapper.c \ + -o $(HOST_DIR)/usr/bin/toolchain-wrapper +endef
Signature · 1 line
2.5.3
PATCH v2 2/18 gcc: use toolchain wrapper
In reply to Arnout Vandecappelle (Essensium/Mind)We have a toolchain wrapper for external toolchain, but it is also beneficial for internal toolchains, for the following reasons:
1. It can make sure that BR2_TARGET_OPTIMIZATION is passed to the compiler even if a package's build system doesn't honor CFLAGS.
2. It allows us to do the unsafe path check (i.e. -I/usr/include) without patching gcc.
3. It makes it simpler to implement building each package with a separate staging directory (per-package staging).
4. It makes it simpler to implement a compiler hash check for ccache.
The wrapper is reused from the external toolchain. A third CROSS_PATH_ option is added to the wrapper: in this case, the real executable is in the same directory, with the extension .real.
The creation of the simple symlinks is merged with the creation of the wrapper symlinks, otherwise part of the -gcc-ar handling logic would have to be repeated.
The complex case-condition could be refactored with the one for the external toolchain, but then it becomes even more complex because they each have special corner cases. For example, the internal toolchain has to handle *.real to avoid creating an extra indirection after host-gcc-{final,initial}-rebuild.
Instead of creating the .real files, it would also have been possible to install the internal toolchain in $(HOST_DIR)/opt, similar to what we do for the external toolchain. However, then we would also have to copy things to the sysroot and do more of the magic that the external toolchain is doing. So keeping it in $(HOST_DIR)/usr/bin is much simpler.
Note that gcc-initial has to be wrapped as well, because it is used for building libc and we want to apply the same magic when building libc.
---
v2: - Update commit message to explain why gcc-initial has to be wrapped
[Romain]
- Remove redundant newline [Romain]
--- package/gcc/gcc-final/gcc-final.mk | 13 +++++-------- package/gcc/gcc-initial/gcc-initial.mk | 4 ++++ package/gcc/gcc.mk | 34 ++++++++++++++++++++++++++++++++++ toolchain/toolchain-wrapper.c | 4 +++- 4 files changed, 46 insertions(+), 9 deletions(-)
package/gcc/gcc-final/gcc-final.mk +5 -8
@@ -96,14 +96,11 @@ endef HOST_GCC_FINAL_POST_INSTALL_HOOKS += HOST_GCC_FINAL_CREATE_CC_SYMLINKS -# Create <arch>-linux-<tool> symlinks -define HOST_GCC_FINAL_CREATE_SIMPLE_SYMLINKS - (cd $(HOST_DIR)/usr/bin; for i in $(GNU_TARGET_NAME)-*; do \ - ln -snf $$i $(ARCH)-linux$${i##$(GNU_TARGET_NAME)}; \ - done) -endef - -HOST_GCC_FINAL_POST_INSTALL_HOOKS += HOST_GCC_FINAL_CREATE_SIMPLE_SYMLINKS +HOST_GCC_FINAL_TOOLCHAIN_WRAPPER_ARGS += $(HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS) +HOST_GCC_FINAL_POST_BUILD_HOOKS += TOOLCHAIN_BUILD_WRAPPER +# Note: this must be done after CREATE_CC_SYMLINKS, otherwise the +# -cc symlink to the wrapper is not created. +HOST_GCC_FINAL_POST_INSTALL_HOOKS += HOST_GCC_INSTALL_WRAPPER_AND_SIMPLE_SYMLINKS # In gcc 4.7.x, the ARM EABIhf library loader path for (e)glibc was not # correct, so we create a symbolic link to make things work
package/gcc/gcc-initial/gcc-initial.mk +4 -0
@@ -62,4 +62,8 @@ HOST_GCC_INITIAL_MAKE_OPTS += all-target-libgcc HOST_GCC_INITIAL_INSTALL_OPTS += install-target-libgcc endif +HOST_GCC_INITIAL_TOOLCHAIN_WRAPPER_ARGS += $(HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS) +HOST_GCC_INITIAL_POST_BUILD_HOOKS += TOOLCHAIN_BUILD_WRAPPER +HOST_GCC_INITIAL_POST_INSTALL_HOOKS += HOST_GCC_INSTALL_WRAPPER_AND_SIMPLE_SYMLINKS + $(eval $(host-autotools-package))
package/gcc/gcc.mk +34 -0
@@ -235,4 +235,38 @@ HOST_GCC_COMMON_CONF_OPTS += \ --with-long-double-128 endif +HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CROSS_PATH_SUFFIX='".real"' + +# The LTO support in gcc creates wrappers for ar, ranlib and nm which load +# the lto plugin. These wrappers are called *-gcc-ar, *-gcc-ranlib, and +# *-gcc-nm and should be used instead of the real programs when -flto is +# used. However, we should not add the toolchain wrapper for them, and they +# match the *cc-* pattern. Therefore, an additional case is added for *-ar, +# *-ranlib and *-nm. +# Avoid that a .real is symlinked a second time. +# Also create <arch>-linux-<tool> symlinks. +define HOST_GCC_INSTALL_WRAPPER_AND_SIMPLE_SYMLINKS + $(Q)cd $(HOST_DIR)/usr/bin; \ + for i in $(GNU_TARGET_NAME)-*; do \ + case "$$i" in \ + *.real) \ + ;; \ + *-ar|*-ranlib|*-nm) \ + ln -snf $$i $(ARCH)-linux$${i##$(GNU_TARGET_NAME)}; \ + ;; \ + *cc|*cc-*|*++|*++-*|*cpp) \ + rm -f $$i.real; \ + mv $$i $$i.real; \ + ln -sf toolchain-wrapper $$i; \ + ln -sf toolchain-wrapper $(ARCH)-linux$${i##$(GNU_TARGET_NAME)}; \ + ln -snf $$i.real $(ARCH)-linux$${i##$(GNU_TARGET_NAME)}.real; \ + ;; \ + *) \ + ln -snf $$i $(ARCH)-linux$${i##$(GNU_TARGET_NAME)}; \ + ;; \ + esac; \ + done + +endef + include $(sort $(wildcard package/gcc/*/*.mk))
toolchain/toolchain-wrapper.c +3 -1
@@ -138,8 +138,10 @@ int main(int argc, char **argv)/* Fill in the relative paths */ #ifdef BR_CROSS_PATH_REL ret =snprintf (path,sizeof (path),"%s/" BR_CROSS_PATH_REL "/%s" , absbasedir, basename); -#else /* BR_CROSS_PATH_ABS */ +#elif BR_CROSS_PATH_ABS ret =snprintf (path,sizeof (path),BR_CROSS_PATH_ABS "/%s" , basename); +#else /* BR_CROSS_PATH_SUFFIX */ + ret =snprintf (path,sizeof (path),"%s/usr/bin/%s" BR_CROSS_PATH_SUFFIX , absbasedir, basename);#endif if (ret >=sizeof (path)) {perror (__FILE__": overflow" );
Signature · 1 line
2.5.3
PATCH v2 3/18 gcc: remove unsafe patch check (poison system dirs) patch
In reply to Arnout Vandecappelle (Essensium/Mind)Now that the calls to gcc always pass through the toolchain wrapper, it is no longer necessary to patch gcc to support poisoning.
This does have the disadvantage that there is no unsafe path check for libc, libgcc and libstdc++ (all of these are built before the wrapper exists). But we can assume that the toolchain components themselves should be pretty safe.
---
Verified that host-gcc-{initial,final}-patch still work for all toolchains.
Verified that the unsafe path check still works by hacking a package Makefile with an additional -I/usr/include
v2: Remove --enable-poison-system-directories from gcc-final configure
commands [Romain]
--- .../4.7.4/910-gcc-poison-system-directories.patch | 207 ------------------- .../4.8.5/910-gcc-poison-system-directories.patch | 207 ------------------- .../4.9.3/910-gcc-poison-system-directories.patch | 207 ------------------- .../5.2.0/200-gcc-poison-system-directories.patch | 207 ------------------- .../910-gcc-poison-system-directories.patch | 221 --------------------- package/gcc/gcc-final/gcc-final.mk | 1 - 6 files changed, 1050 deletions(-) delete mode 100644 package/gcc/4.7.4/910-gcc-poison-system-directories.patch delete mode 100644 package/gcc/4.8.5/910-gcc-poison-system-directories.patch delete mode 100644 package/gcc/4.9.3/910-gcc-poison-system-directories.patch delete mode 100644 package/gcc/5.2.0/200-gcc-poison-system-directories.patch delete mode 100644 package/gcc/arc-2015.06/910-gcc-poison-system-directories.patch
package/gcc/4.7.4/910-gcc-poison-system-directories.patch deleted +0 -207
@@ -1,207 +0,0 @@ -From 160397ef3c3331099af028f1b8d3e085b07d88ad Mon Sep 17 00:00:00 2001 -From: Khem Raj <[email protected]> -Date: Fri, 29 Mar 2013 08:59:00 +0400 -Subject: [PATCH 16/35] gcc: poison-system-directories - -Adapted to Buildroot and gcc 4.7.4 by Thomas Petazzoni, especially the -addition of the BR_COMPILER_PARANOID_UNSAFE_PATH environment variable. - -Signed-off-by: Khem Raj <[email protected]> -Signed-off-by: Thomas Petazzoni <[email protected]> - -Upstream-Status: Inappropriate [distribution: codesourcery] ---- - gcc/Makefile.in | 2 +- - gcc/common.opt | 4 ++++ - gcc/config.in | 6 ++++++ - gcc/configure | 20 ++++++++++++++++++-- - gcc/configure.ac | 10 ++++++++++ - gcc/doc/invoke.texi | 9 +++++++++ - gcc/gcc.c | 2 ++ - gcc/incpath.c | 19 +++++++++++++++++++ - 8 files changed, 69 insertions(+), 3 deletions(-) - -Index: b/gcc/common.opt -=================================================================== ---- a/gcc/common.opt -+++ b/gcc/common.opt -@@ -585,6 +585,10 @@ - Common Var(warn_padded) Warning - Warn when padding is required to align structure members - -+Wpoison-system-directories -+Common Var(flag_poison_system_directories) Init(1) Warning -+Warn for -I and -L options using system directories if cross compiling -+ - Wshadow - Common Var(warn_shadow) Warning - Warn when one local variable shadows another -Index: b/gcc/config.in -=================================================================== ---- a/gcc/config.in -+++ b/gcc/config.in -@@ -144,6 +144,12 @@ - #endif - - -+/* Define to warn for use of native system header directories */ -+#ifndef USED_FOR_TARGET -+#undef ENABLE_POISON_SYSTEM_DIRECTORIES -+#endif -+ -+ - /* Define if you want all operations on RTL (the basic data structure of the - optimizer and back end) to be checked for dynamic type safety at runtime. - This is quite expensive. */ -Index: b/gcc/configure -=================================================================== ---- a/gcc/configure -+++ b/gcc/configure -@@ -918,6 +918,7 @@ - with_system_zlib - enable_maintainer_mode - enable_version_specific_runtime_libs -+enable_poison_system_directories - enable_plugin - enable_libquadmath_support - with_linker_hash_style -@@ -1632,6 +1633,8 @@ - --enable-version-specific-runtime-libs - specify that runtime libraries should be installed - in a compiler-specific directory -+ --enable-poison-system-directories -+ warn for use of native system header directories - --enable-plugin enable plugin support - --disable-libquadmath-support - disable libquadmath support for Fortran -@@ -27186,6 +27189,19 @@ - fi - - -+# Check whether --enable-poison-system-directories was given. -+if test "${enable_poison_system_directories+set}" = set; then : -+ enableval=$enable_poison_system_directories; -+else -+ enable_poison_system_directories=no -+fi -+ -+if test "x${enable_poison_system_directories}" = "xyes"; then -+ -+$as_echo "#define ENABLE_POISON_SYSTEM_DIRECTORIES 1" >>confdefs.h -+ -+fi -+ - # Substitute configuration variables - - -Index: b/gcc/configure.ac -=================================================================== ---- a/gcc/configure.ac -+++ b/gcc/configure.ac -@@ -5037,6 +5037,16 @@ - [specify that runtime libraries should be - installed in a compiler-specific directory])]) - -+AC_ARG_ENABLE([poison-system-directories], -+ AS_HELP_STRING([--enable-poison-system-directories], -+ [warn for use of native system header directories]),, -+ [enable_poison_system_directories=no]) -+if test "x${enable_poison_system_directories}" = "xyes"; then -+ AC_DEFINE([ENABLE_POISON_SYSTEM_DIRECTORIES], -+ [1], -+ [Define to warn for use of native system header directories]) -+fi -+ - # Substitute configuration variables - AC_SUBST(subdirs) - AC_SUBST(srcdir) -Index: b/gcc/doc/invoke.texi -=================================================================== ---- a/gcc/doc/invoke.texi -+++ b/gcc/doc/invoke.texi -@@ -260,6 +260,7 @@ - -Woverlength-strings -Wpacked -Wpacked-bitfield-compat -Wpadded @gol - -Wparentheses -Wpedantic-ms-format -Wno-pedantic-ms-format @gol - -Wpointer-arith -Wno-pointer-to-int-cast @gol -+-Wno-poison-system-directories @gol - -Wredundant-decls @gol - -Wreturn-type -Wsequence-point -Wshadow @gol - -Wsign-compare -Wsign-conversion -Wstack-protector @gol -@@ -3880,6 +3881,14 @@ - for most targets, it is made up of code and thus requires the stack - to be made executable in order for the program to work properly. - -+ at item -Wno-poison-system-directories -+ at opindex Wno-poison-system-directories -+Do not warn for @option{-I} or @option{-L} options using system -+directories such as @file{/usr/include} when cross compiling. This -+option is intended for use in chroot environments when such -+directories contain the correct headers and libraries for the target -+system rather than the host. -+ - @item -Wfloat-equal - @opindex Wfloat-equal - @opindex Wno-float-equal -Index: b/gcc/gcc.c -=================================================================== ---- a/gcc/gcc.c -+++ b/gcc/gcc.c -@@ -674,6 +674,8 @@ - %{flto} %{flto=*} %l " LINK_PIE_SPEC \ - "%X %{o*} %{e*} %{N} %{n} %{r}\ - %{s} %{t} %{u*} %{z} %{Z} %{!nostdlib:%{!nostartfiles:%S}}\ -+ %{Wno-poison-system-directories:--no-poison-system-directories}\ -+ %{Werror=poison-system-directories:--error-poison-system-directories}\ - %{static:} %{L*} %(mfwrap) %(link_libgcc) %o\ - %{fopenmp|ftree-parallelize-loops=*:%:include(libgomp.spec)%(link_gomp)}\ - %{fgnu-tm:%:include(libitm.spec)%(link_itm)}\ -@@ -3907,6 +3909,12 @@ - gcc_assert (!compare_debug_opt); - } - -+ temp = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH"); -+ if (temp && strlen(temp) > 0) -+ { -+ save_switch("-Werror=poison-system-directories", 0, NULL, false); -+ } -+ - /* Set up the search paths. We add directories that we expect to - contain GNU Toolchain components before directories specified by - the machine description so that we will find GNU components (like -Index: b/gcc/incpath.c -=================================================================== ---- a/gcc/incpath.c -+++ b/gcc/incpath.c -@@ -30,6 +30,7 @@ - #include "intl.h" - #include "incpath.h" - #include "cppdefault.h" -+#include "diagnostic-core.h" - - /* Microsoft Windows does not natively support inodes. - VMS has non-numeric inodes. */ -@@ -373,6 +374,24 @@ - } - fprintf (stderr, _("End of search list.\n")); - } -+ -+#ifdef ENABLE_POISON_SYSTEM_DIRECTORIES -+ if (flag_poison_system_directories) -+ { -+ struct cpp_dir *p; -+ -+ for (p = heads[QUOTE]; p; p = p->next) -+ { -+ if ((!strncmp (p->name, "/usr/include", 12)) -+ || (!strncmp (p->name, "/usr/local/include", 18)) -+ || (!strncmp (p->name, "/usr/X11R6/include", 18))) -+ warning (OPT_Wpoison_system_directories, -+ "include location \"%s\" is unsafe for " -+ "cross-compilation", -+ p->name); -+ } -+ } -+#endif - } - - /* Use given -I paths for #include "..." but not #include <...>, and
package/gcc/4.8.5/910-gcc-poison-system-directories.patch deleted +0 -207
207 lines not on this page · raw message
package/gcc/4.9.3/910-gcc-poison-system-directories.patch deleted +0 -207
207 lines not on this page · raw message
package/gcc/5.2.0/200-gcc-poison-system-directories.patch deleted +0 -207
207 lines not on this page · raw message
package/gcc/arc-2015.06/910-gcc-poison-system-directories.patch deleted +0 -221
221 lines not on this page · raw message
package/gcc/gcc-final/gcc-final.mk +0 -1
@@ -50,7 +50,6 @@ GCC_FINAL_CROSS_LANGUAGES = $(subst $(space),$(comma),$(GCC_FINAL_CROSS_LANGUAGE HOST_GCC_FINAL_CONF_OPTS = \ $(HOST_GCC_COMMON_CONF_OPTS) \ --enable-languages=$(GCC_FINAL_CROSS_LANGUAGES) \ - --enable-poison-system-directories \ --with-build-time-tools=$(HOST_DIR)/usr/$(GNU_TARGET_NAME)/bin HOST_GCC_FINAL_GCC_LIB_DIR = $(HOST_DIR)/usr/$(GNU_TARGET_NAME)/lib*
Signature · 1 line
2.5.3
PATCH v2 4/18 infra: move ccache handling to the toolchain wrapper
In reply to Arnout Vandecappelle (Essensium/Mind)Since we always have a toolchain wrapper now, we can move the ccache call to the toolchain wrapper.
The hostcc ccache handling obviously stays.
The global addition of ccache to TARGET_CC/CXX is removed, but many individual packages and infras still add it. This means we have a chain like this: ccache -> toolchain-wrapper -> ccache -> gcc However, this is fairly harmless: for cache misses, the inner ccache just adds overhead and for cache hits, the inner ccache is never called. Later patches will remove these redundant ccache calls.
As a side effect, perl now supports ccache as well.
---
v2: No change
--- package/Makefile.in | 5 ----- toolchain/toolchain-wrapper.c | 5 ++++- toolchain/toolchain-wrapper.mk | 4 ++++ 3 files changed, 8 insertions(+), 6 deletions(-)
package/Makefile.in +0 -5
@@ -189,11 +189,6 @@ TARGET_OBJDUMP = $(TARGET_CROSS)objdump TARGET_CC_NOCCACHE := $(TARGET_CC) TARGET_CXX_NOCCACHE := $(TARGET_CXX) -ifeq ($(BR2_CCACHE),y) -TARGET_CC := $(CCACHE) $(TARGET_CC) -TARGET_CXX := $(CCACHE) $(TARGET_CXX) -endif - ifeq ($(BR2_STRIP_strip),y) STRIP_STRIP_DEBUG := --strip-debug STRIP_STRIP_UNNEEDED := --strip-unneeded
toolchain/toolchain-wrapper.c +4 -1
@@ -40,6 +40,9 @@ static char sysroot[PATH_MAX];#define EXCLUSIVE_ARGS 3 static char *predef_args[] = { +#ifdef BR_CCACHE +BR_CCACHE , +#endif path,"--sysroot" , sysroot, #ifdef BR_ABI
@@ -251,7 +254,7 @@ int main(int argc, char **argv) } } -if (execv (path, args)) +if (execv (args[0 ], args))perror (path);free (args);
toolchain/toolchain-wrapper.mk +4 -0
@@ -16,6 +16,10 @@ TOOLCHAIN_WRAPPER_ARGS += -DBR_SYSROOT='"$(STAGING_SUBDIR)"' # separate argument when used in execv() by the toolchain wrapper. TOOLCHAIN_WRAPPER_ARGS += -DBR_ADDITIONAL_CFLAGS='$(foreach f,$(call qstrip,$(BR2_TARGET_OPTIMIZATION)),"$(f)",)' +ifeq ($(BR2_CCACHE),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE='"$(CCACHE)"' +endif + # For simplicity, build directly into the install location define TOOLCHAIN_BUILD_WRAPPER $(Q)mkdir -p $(HOST_DIR)/usr/bin
Signature · 1 line
2.5.3
PATCH v2 4/18 infra: move ccache handling to the toolchain wrapper
Quoted 23 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle (Essensium/Mind) <[email protected]> writes:
Since we always have a toolchain wrapper now, we can move the ccache call to the toolchain wrapper.
The hostcc ccache handling obviously stays.
The global addition of ccache to TARGET_CC/CXX is removed, but many individual packages and infras still add it. This means we have a chain like this: ccache -> toolchain-wrapper -> ccache -> gcc However, this is fairly harmless: for cache misses, the inner ccache just adds overhead and for cache hits, the inner ccache is never called. Later patches will remove these redundant ccache calls.
As a side effect, perl now supports ccache as well.
Cc Danomi Manchego <[email protected]>Cc K?roly Kasza <[email protected]>Reviewed-by Romain Naour <[email protected]>+++
b/toolchain/toolchain-wrapper.mk
toolchain/toolchain-wrapper.mk TOOLCHAIN_WRAPPER_ARGS += -DBR_SYSROOT='"$(STAGING_SUBDIR)"' · line 21
Quoted 6 lines
toolchain/toolchain-wrapper.mk +3
-0@@ -16,6 +16,10 @@ TOOLCHAIN_WRAPPER_ARGS += -DBR_SYSROOT='"$(STAGING_SUBDIR)"' # separate argument when used in execv() by the toolchain wrapper. TOOLCHAIN_WRAPPER_ARGS += -DBR_ADDITIONAL_CFLAGS='$(foreach f,$(call qstrip,$(BR2_TARGET_OPTIMIZATION)),"$(f)",)' +ifeq ($(BR2_CCACHE),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE='"$(CCACHE)"' +endif
As I just mentioned IRL, CCACHE is $(HOST_DIR)/usr/bin/ccache, E.G the absolute path to ccache - So hardcoding this in the wrapper isn't really nice for relocatable SDK.
Signature · 1 line
Bye, Peter Korsgaard
PATCH v3 1/1 infra: move ccache handling to the toolchain wrapper
In reply to Arnout Vandecappelle (Essensium/Mind)Since we always have a toolchain wrapper now, we can move the ccache call to the toolchain wrapper.
The hostcc ccache handling obviously stays.
The global addition of ccache to TARGET_CC/CXX is removed, but many individual packages and infras still add it. This means we have a chain like this: ccache -> toolchain-wrapper -> ccache -> gcc However, this is fairly harmless: for cache misses, the inner ccache just adds overhead and for cache hits, the inner ccache is never called. Later patches will remove these redundant ccache calls.
As a side effect, perl now supports ccache as well.
---
v3: Don't use absolute path for BR_CCACHE; instead, use the abspath
that is anyway already calculated by the wrapper. [Peter]
v2: No change
--- package/Makefile.in | 5 ----- toolchain/toolchain-wrapper.c | 15 ++++++++++++++- toolchain/toolchain-wrapper.mk | 4 ++++ 3 files changed, 18 insertions(+), 6 deletions(-)
package/Makefile.in +0 -5
@@ -189,11 +189,6 @@ TARGET_OBJDUMP = $(TARGET_CROSS)objdump TARGET_CC_NOCCACHE := $(TARGET_CC) TARGET_CXX_NOCCACHE := $(TARGET_CXX) -ifeq ($(BR2_CCACHE),y) -TARGET_CC := $(CCACHE) $(TARGET_CC) -TARGET_CXX := $(CCACHE) $(TARGET_CXX) -endif - ifeq ($(BR2_STRIP_strip),y) STRIP_STRIP_DEBUG := --strip-debug STRIP_STRIP_UNNEEDED := --strip-unneeded
toolchain/toolchain-wrapper.c +14 -1
@@ -23,6 +23,9 @@#include <stdlib.h> #include <errno.h> +#ifdef BR_CCACHE +static char ccache_path[PATH_MAX ]; +#endif static char path[PATH_MAX ];static char sysroot[PATH_MAX ];
@@ -40,6 +43,9 @@ static char sysroot[PATH_MAX];#define EXCLUSIVE_ARGS 3 static char *predef_args[] = { +#ifdef BR_CCACHE + ccache_path, +#endif path,"--sysroot" , sysroot, #ifdef BR_ABI
@@ -147,6 +153,13 @@ int main(int argc, char **argv)perror (__FILE__": overflow" );return 3 ; } +#ifdef BR_CCACHE + ret =snprintf (ccache_path,sizeof (ccache_path),"%s/usr/bin/ccache" , absbasedir); +if (ret >=sizeof (ccache_path)) { +perror (__FILE__": overflow" ); +return 3 ; + } +#endif ret =snprintf (sysroot,sizeof (sysroot),"%s/" BR_SYSROOT , absbasedir);if (ret >=sizeof (sysroot)) {perror (__FILE__": overflow" );
@@ -251,7 +264,7 @@ int main(int argc, char **argv) } } -if (execv (path, args)) +if (execv (args[0 ], args))perror (path);free (args);
toolchain/toolchain-wrapper.mk +4 -0
@@ -16,6 +16,10 @@ TOOLCHAIN_WRAPPER_ARGS += -DBR_SYSROOT='"$(STAGING_SUBDIR)"' # separate argument when used in execv() by the toolchain wrapper. TOOLCHAIN_WRAPPER_ARGS += -DBR_ADDITIONAL_CFLAGS='$(foreach f,$(call qstrip,$(BR2_TARGET_OPTIMIZATION)),"$(f)",)' +ifeq ($(BR2_CCACHE),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE +endif + # For simplicity, build directly into the install location define TOOLCHAIN_BUILD_WRAPPER $(Q)mkdir -p $(HOST_DIR)/usr/bin
Signature · 1 line
2.5.3
PATCH v2 5/18 perl: Remove ccache handling
In reply to Arnout Vandecappelle (Essensium/Mind)Now the ccache handling has moved to the toolchain wrapper, it is no longer necessary to pass TARGET_CC_NOCCACHE.
---
v2: No change
--- package/perl/perl.mk | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
package/perl/perl.mk +2 -3
@@ -45,13 +45,12 @@ PERL_DEPENDENCIES += gdbm endif # We have to override LD, because an external multilib toolchain ld is not -# wrapped to provide the required sysroot options. We also can't use ccache -# because the configure script doesn't support it. +# wrapped to provide the required sysroot options. PERL_CONF_OPTS = \ --target=$(GNU_TARGET_NAME) \ --target-tools-prefix=$(TARGET_CROSS) \ --prefix=/usr \ - -Dld="$(TARGET_CC_NOCCACHE)" \ + -Dld="$(TARGET_CC)" \ -Dccflags="$(TARGET_CFLAGS)" \ -Dldflags="$(TARGET_LDFLAGS) -lm" \ -Dmydomain="" \
Signature · 1 line
2.5.3
PATCH v2 6/18 imx-lib: remove now-redundant ccache handling
In reply to Arnout Vandecappelle (Essensium/Mind)---
v2: No change
--- package/freescale-imx/imx-lib/imx-lib.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/freescale-imx/imx-lib/imx-lib.mk +1 -1
@@ -21,7 +21,7 @@ IMX_LIB_INCLUDE = \ IMX_LIB_MAKE_ENV = \ $(TARGET_MAKE_ENV) \ $(TARGET_CONFIGURE_OPTS) \ - CROSS_COMPILE="$(CCACHE) $(TARGET_CROSS)" \ + CROSS_COMPILE="$(TARGET_CROSS)" \ PLATFORM=$(BR2_PACKAGE_FREESCALE_IMX_PLATFORM) \ INCLUDE="$(IMX_LIB_INCLUDE)"
Signature · 1 line
2.5.3
PATCH v2 7/18 imx-vpu: remove now-redundant ccache handling
In reply to Arnout Vandecappelle (Essensium/Mind)---
v2: Rebased on master
--- package/freescale-imx/imx-vpu/imx-vpu.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
package/freescale-imx/imx-vpu/imx-vpu.mk +1 -1
@@ -13,7 +13,7 @@ IMX_VPU_INSTALL_STAGING = YES IMX_VPU_MAKE_ENV = \ $(TARGET_MAKE_ENV) \ $(TARGET_CONFIGURE_OPTS) \ - CROSS_COMPILE="$(CCACHE) $(TARGET_CROSS)" \ + CROSS_COMPILE="$(TARGET_CROSS)" \ PLATFORM=$(BR2_PACKAGE_FREESCALE_IMX_PLATFORM) IMX_VPU_LICENSE = Freescale Semiconductor Software License Agreement
Signature · 1 line
2.5.3
PATCH v2 8/18 linux: remove now-redundant ccache handling
In reply to Arnout Vandecappelle (Essensium/Mind)---
v2: No change
--- linux/linux.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
linux/linux.mk +1 -1
@@ -67,7 +67,7 @@ LINUX_MAKE_FLAGS = \ HOSTCFLAGS="$(HOSTCFLAGS)" \ ARCH=$(KERNEL_ARCH) \ INSTALL_MOD_PATH=$(TARGET_DIR) \ - CROSS_COMPILE="$(CCACHE) $(TARGET_CROSS)" \ + CROSS_COMPILE="$(TARGET_CROSS)" \ DEPMOD=$(HOST_DIR)/sbin/depmod LINUX_MAKE_ENV = \
Signature · 1 line
2.5.3
PATCH v2 9/18 uboot: remove now-redundant ccache handling
In reply to Arnout Vandecappelle (Essensium/Mind)---
v2: No change
--- boot/uboot/uboot.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
boot/uboot/uboot.mk +1 -1
@@ -74,7 +74,7 @@ endif UBOOT_ARCH = $(KERNEL_ARCH) UBOOT_MAKE_OPTS += \ - CROSS_COMPILE="$(CCACHE) $(TARGET_CROSS)" \ + CROSS_COMPILE="$(TARGET_CROSS)" \ ARCH=$(UBOOT_ARCH) # Helper function to fill the U-Boot config.h file.
Signature · 1 line
2.5.3
PATCH v2 10/18 barebox: remove now-redundant ccache handling
In reply to Arnout Vandecappelle (Essensium/Mind)---
v2: No change
--- boot/barebox/barebox.mk | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
boot/barebox/barebox.mk +1 -2
@@ -52,8 +52,7 @@ else BAREBOX_ARCH = $(KERNEL_ARCH) endif -BAREBOX_MAKE_FLAGS = ARCH=$(BAREBOX_ARCH) CROSS_COMPILE="$(CCACHE) \ - $(TARGET_CROSS)" +BAREBOX_MAKE_FLAGS = ARCH=$(BAREBOX_ARCH) CROSS_COMPILE="$(TARGET_CROSS)" BAREBOX_MAKE_ENV = $(TARGET_MAKE_ENV) ifeq ($(BR2_TARGET_BAREBOX_USE_DEFCONFIG),y)
Signature · 1 line
2.5.3
PATCH v2 11/18 cryptodev-linux: remove now-redundant fix-ccache-compile patch
In reply to Arnout Vandecappelle (Essensium/Mind)---
v2: No change
--- .../cryptodev-linux/0002-fix-ccache-compile.patch | 20 -------------------- 1 file changed, 20 deletions(-) delete mode 100644 package/cryptodev-linux/0002-fix-ccache-compile.patch
package/cryptodev-linux/0002-fix-ccache-compile.patch deleted +0 -20
@@ -1,20 +0,0 @@ -Fix compilation with ccache enabled - -When ccache is enabled the CROSS_COMPILE variable contains a space, so -it must be properly quoted. - -Signed-off-by: Doug Kehn <[email protected]> - -Index: cryptodev-linux-1.7/Makefile -=================================================================== ---- cryptodev-linux-1.7.orig/Makefile -+++ cryptodev-linux-1.7/Makefile -@@ -18,7 +18,7 @@ ifneq (${ARCH},) - KERNEL_MAKE_OPTS += ARCH=${ARCH} - endif - ifneq (${CROSS_COMPILE},) --KERNEL_MAKE_OPTS += CROSS_COMPILE=${CROSS_COMPILE} -+KERNEL_MAKE_OPTS += CROSS_COMPILE="${CROSS_COMPILE}" - endif - - build: version.h
Signature · 1 line
2.5.3
PATCH v2 12/18 qt5base: remove now-redundant ccache handling
In reply to Arnout Vandecappelle (Essensium/Mind)---
Note that ccache was not and still is not used for the host tools (e.g. qmake). This commit only affects the target builds.
v2: No change
--- package/qt5/qt5base/0002-mkspecs-files.patch | 6 ++- .../qt5/qt5base/0009-fix-build-with-ccache.patch | 49 ---------------------- package/qt5/qt5base/qt5base.mk | 1 - 3 files changed, 4 insertions(+), 52 deletions(-) delete mode 100644 package/qt5/qt5base/0009-fix-build-with-ccache.patch
package/qt5/qt5base/0002-mkspecs-files.patch +4 -2
@@ -9,6 +9,8 @@ allows us to easily pass the cross-compiler paths and flags from our qt5.mk. Signed-off-by: Thomas Petazzoni <[email protected]> +[Arnout: remove ccache support] +Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <[email protected]> Index: b/mkspecs/devices/linux-buildroot-g++/qmake.conf ===================================================================
@@ -18,8 +20,8 @@ Index: b/mkspecs/devices/linux-buildroot-g++/qmake.conf +include(../common/linux_device_pre.conf) + +# modifications to g++-unix.conf -+QMAKE_CC = $${BR_CCACHE} $${CROSS_COMPILE}gcc -+QMAKE_CXX = $${BR_CCACHE} $${CROSS_COMPILE}g++ ++QMAKE_CC = $${CROSS_COMPILE}gcc ++QMAKE_CXX = $${CROSS_COMPILE}g++ + +#modifications to gcc-base.conf +QMAKE_CFLAGS += $${BR_COMPILER_CFLAGS}
package/qt5/qt5base/0009-fix-build-with-ccache.patch deleted +0 -49
49 lines not on this page · raw message
package/qt5/qt5base/qt5base.mk +0 -1
@@ -189,7 +189,6 @@ define QT5BASE_CONFIGURE_CMDS -nomake tests \ -device buildroot \ -device-option CROSS_COMPILE="$(TARGET_CROSS)" \ - -device-option BR_CCACHE="$(CCACHE)" \ -device-option BR_COMPILER_CFLAGS="$(TARGET_CFLAGS) $(QT5BASE_EXTRA_CFLAGS)" \ -device-option BR_COMPILER_CXXFLAGS="$(TARGET_CXXFLAGS) $(QT5BASE_EXTRA_CFLAGS)" \ $(QT5BASE_CONFIGURE_OPTS) \
Signature · 1 line
2.5.3
PATCH v2 13/18 package-cmake: remove now-redundant target ccache support
In reply to Arnout Vandecappelle (Essensium/Mind)All the complexity with the different ways that CMAKE_C_COMPILER and CMAKE_C_COMPILER_ARG1 can be set are no longer needed, it's all handled by the toolchain wrapper now.
Note that it is still necessary to handle this for the host build.
---
v2: Rebased on master
--- package/pkg-cmake.mk | 5 ++--- support/misc/toolchainfile.cmake.in | 32 ++------------------------------ 2 files changed, 4 insertions(+), 33 deletions(-)
package/pkg-cmake.mk +2 -3
@@ -98,7 +98,6 @@ define $(2)_CONFIGURE_CMDS -DBUILD_TESTS=OFF \ -DBUILD_TESTING=OFF \ -DBUILD_SHARED_LIBS=$$(if $$(BR2_STATIC_LIBS),OFF,ON) \ - -DUSE_CCACHE=$$(if $$(BR2_CCACHE),ON,OFF) \ $$(CMAKE_QUIET) \ $$($$(PKG)_CONF_OPTS) \ )
@@ -245,8 +244,8 @@ $(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake: -e 's#@@TARGET_CFLAGS@@#$(call qstrip,$(TARGET_CFLAGS))#' \ -e 's#@@TARGET_CXXFLAGS@@#$(call qstrip,$(TARGET_CXXFLAGS))#' \ -e 's#@@TARGET_LDFLAGS@@#$(call qstrip,$(TARGET_LDFLAGS))#' \ - -e 's#@@TARGET_CC_NOCCACHE@@#$(subst $(HOST_DIR)/,,$(call qstrip,$(TARGET_CC_NOCCACHE)))#' \ - -e 's#@@TARGET_CXX_NOCCACHE@@#$(subst $(HOST_DIR)/,,$(call qstrip,$(TARGET_CXX_NOCCACHE)))#' \ + -e 's#@@TARGET_CC@@#$(subst $(HOST_DIR)/,,$(call qstrip,$(TARGET_CC)))#' \ + -e 's#@@TARGET_CXX@@#$(subst $(HOST_DIR)/,,$(call qstrip,$(TARGET_CXX)))#' \ -e 's#@@CMAKE_SYSTEM_PROCESSOR@@#$(call qstrip,$(CMAKE_SYSTEM_PROCESSOR))#' \ $(TOPDIR)/support/misc/toolchainfile.cmake.in \ > $@
support/misc/toolchainfile.cmake.in +2 -30
@@ -27,33 +27,5 @@ set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY) set(ENV{PKG_CONFIG_SYSROOT_DIR} "${RELOCATED_HOST_DIR}/@@STAGING_SUBDIR@@") # This toolchain file can be used both inside and outside Buildroot. -# * When used inside Buildroot, ccache support is explicitly driven using the -# USE_CCACHE variable. -# * When used outside Buildroot (i.e. when USE_CCACHE is not defined), ccache -# support is automatically enabled if the ccache program is available. -if(DEFINED USE_CCACHE) - if(USE_CCACHE) - set(CMAKE_ASM_COMPILER "${RELOCATED_HOST_DIR}/@@TARGET_CC_NOCCACHE@@") - set(CMAKE_C_COMPILER "${RELOCATED_HOST_DIR}/usr/bin/ccache") - set(CMAKE_CXX_COMPILER "${RELOCATED_HOST_DIR}/usr/bin/ccache") - set(CMAKE_C_COMPILER_ARG1 "${RELOCATED_HOST_DIR}/@@TARGET_CC_NOCCACHE@@") - set(CMAKE_CXX_COMPILER_ARG1 "${RELOCATED_HOST_DIR}/@@TARGET_CXX_NOCCACHE@@") - else() - set(CMAKE_C_COMPILER "${RELOCATED_HOST_DIR}/@@TARGET_CC_NOCCACHE@@") - set(CMAKE_CXX_COMPILER "${RELOCATED_HOST_DIR}/@@TARGET_CXX_NOCCACHE@@") - endif() -else() - find_program(CCACHE ccache HINTS "${RELOCATED_HOST_DIR}/usr/bin") - if(CCACHE) - set(CMAKE_ASM_COMPILER "${RELOCATED_HOST_DIR}/@@TARGET_CC_NOCCACHE@@") - set(CMAKE_C_COMPILER "${CCACHE}") - set(CMAKE_CXX_COMPILER "${CCACHE}") - set(CMAKE_C_COMPILER_ARG1 "${RELOCATED_HOST_DIR}/@@TARGET_CC_NOCCACHE@@") - set(CMAKE_CXX_COMPILER_ARG1 "${RELOCATED_HOST_DIR}/@@TARGET_CXX_NOCCACHE@@") - message(STATUS "ccache program has been found and will be used for the build.") - message(STATUS " To disable ccache, add -DUSE_CCACHE=OFF on the cmake command line.") - else() - set(CMAKE_C_COMPILER "${RELOCATED_HOST_DIR}/@@TARGET_CC_NOCCACHE@@") - set(CMAKE_CXX_COMPILER "${RELOCATED_HOST_DIR}/@@TARGET_CXX_NOCCACHE@@") - endif() -endif() +set(CMAKE_C_COMPILER "${RELOCATED_HOST_DIR}/@@TARGET_CC@@") +set(CMAKE_CXX_COMPILER "${RELOCATED_HOST_DIR}/@@TARGET_CXX@@")
Signature · 1 line
2.5.3
PATCH v2 14/18 qpid-proton: remove now-redundant ccache handling patch
In reply to Arnout Vandecappelle (Essensium/Mind)Although this patch is still relevant for upstream, we don't need it anymore in buildoort because we no longer have _ARG1.
Note that it would be relevant again if we would add host-qpid-proton. However, the 0.10 release already has the fix.
---
v2: No change
--- ...fix-C-compiler-detection-with-_ARG1-_ARG2.patch | 52 ---------------------- 1 file changed, 52 deletions(-) delete mode 100644 package/qpid-proton/0001-proton-c-fix-C-compiler-detection-with-_ARG1-_ARG2.patch
package/qpid-proton/0001-proton-c-fix-C-compiler-detection-with-_ARG1-_ARG2.patch deleted +0 -52
@@ -1,52 +0,0 @@ -From f24be9ae9ab01c7507a366e9d5529c444f0b7edd Mon Sep 17 00:00:00 2001 -From: Luca Ceresoli <[email protected]> -Date: Fri, 10 Jul 2015 10:13:47 +0200 -Subject: [PATCH] proton-c: fix C compiler detection with _ARG1/_ARG2 - -The C compiler commandline in CMake is composed by the concatenation of -CMAKE_C_COMPILER + CMAKE_C_COMPILER_ARG1 + CMAKE_C_COMPILER_ARG2. - -In most use cases the two additional argument variables are empty, thus -CMAKE_C_COMPILER can be used without any noticeable difference. - -The Buildroot embedded Linux build system [0], however, optionally exploits the -CMAKE_C_COMPILER_ARG1 variable to speed up the cross-compilation of CMake-based -packages using ccache. It does so by setting [1]: - - CMAKE_C_COMPILER = /path/to/ccache - CMAKE_C_COMPILER_ARG1 = /path/to/cross-gcc - -This works fine with other CMake-based packages, but proton-c's CMakeLists.txt -calls gcc to extract the compiler version. It does so by calling -"${CMAKE_C_COMPILER} -dumpversion", without honoring the two extra arguments. -Within Buildroot with ccache enabled, this means calling -"/path/to/ccache -dumpversion", which fails with the error: - - ccache: invalid option -- 'd' - -Fix the compiler check by adding the two arguments. - -[0] http://buildroot.net/ -[1] http://git.buildroot.net/buildroot/tree/support/misc/toolchainfile.cmake.in?id=2015.05 - -Signed-off-by: Luca Ceresoli <[email protected]> ---- - proton-c/CMakeLists.txt | 2 +- - 1 file changed, 1 insertion(+), 1 deletion(-) - -diff --git a/proton-c/CMakeLists.txt b/proton-c/CMakeLists.txt -index 93449a9..8c31a89 100644 ---- a/proton-c/CMakeLists.txt -+++ b/proton-c/CMakeLists.txt -@@ -218,7 +218,7 @@ if (CMAKE_COMPILER_IS_GNUCC) - set (COMPILE_LANGUAGE_FLAGS "-std=c99") - set (COMPILE_PLATFORM_FLAGS "-std=gnu99") - -- execute_process(COMMAND ${CMAKE_C_COMPILER} -dumpversion OUTPUT_VARIABLE GCC_VERSION -+ execute_process(COMMAND ${CMAKE_C_COMPILER} ${CMAKE_C_COMPILER_ARG1} ${CMAKE_C_COMPILER_ARG2} -dumpversion OUTPUT_VARIABLE GCC_VERSION - OUTPUT_STRIP_TRAILING_WHITESPACE) - if (${GCC_VERSION} VERSION_LESS "4.3.0") - # Only a concern if contibuting code back. --- -1.9.1 -
Signature · 1 line
2.5.3
PATCH v2 15/18 Makefile.in: remove now-unused TARGET_CC/CXX_NOCCACHE
In reply to Arnout Vandecappelle (Essensium/Mind)---
v2: No change
--- package/Makefile.in | 3 --- 1 file changed, 3 deletions(-)
package/Makefile.in +0 -3
@@ -186,9 +186,6 @@ TARGET_READELF = $(TARGET_CROSS)readelf TARGET_OBJCOPY = $(TARGET_CROSS)objcopy TARGET_OBJDUMP = $(TARGET_CROSS)objdump -TARGET_CC_NOCCACHE := $(TARGET_CC) -TARGET_CXX_NOCCACHE := $(TARGET_CXX) - ifeq ($(BR2_STRIP_strip),y) STRIP_STRIP_DEBUG := --strip-debug STRIP_STRIP_UNNEEDED := --strip-unneeded
Signature · 1 line
2.5.3
PATCH v2 16/18 toolchain-wrapper: support change of BR2_CCACHE
In reply to Arnout Vandecappelle (Essensium/Mind)By moving the ccache call to the toolchain wrapper, the following scenario no longer works:
make foo-dirclean all BR2_CCACHE=
That's a sometimes useful call to check if some failure is perhaps caused by ccache.
We can enable this scenario again by exporting BR_NO_CCACHE when BR2_CCACHE is not set, and by handling this in the toolchain wrapper.
---
v2: No change, but no longer RFC.
--- Makefile | 2 ++ toolchain/toolchain-wrapper.c | 15 +++++++++++---- 2 files changed, 13 insertions(+), 4 deletions(-)
Makefile +2 -0
@@ -375,6 +375,8 @@ BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR)) export BR_CACHE_DIR HOSTCC := $(CCACHE) $(HOSTCC) HOSTCXX := $(CCACHE) $(HOSTCXX) +else +export BR_NO_CCACHE endif # Scripts in support/ or post-build scripts may need to reference
toolchain/toolchain-wrapper.c +11 -4
@@ -95,7 +95,7 @@ static void check_unsafe_path(const char *path, int paranoid)int main (int argc,char **argv) { -char **args, **cur; +char **args, **cur, **exec_args;char *relbasedir, *absbasedir;char *progpath = argv[0 ];char *basename;
@@ -237,6 +237,13 @@ int main(int argc, char **argv)/* finish with NULL termination */ *cur =NULL ; + exec_args = args; +#ifdef BR_CCACHE +if (getenv ("BR_NO_CCACHE" )) +/* Skip the ccache call */ + exec_args++; +#endif +/* Debug the wrapper to see actual arguments passed to * the compiler: * unset, empty, or 0: do not trace
@@ -247,14 +254,14 @@ int main(int argc, char **argv) debug =atoi (env_debug);if (debug >0 ) {fprintf (stderr,"Toolchain wrapper executing:" ); -for (i =0 ; args[i]; i++) +for (i =0 ; exec_args[i]; i++)fprintf (stderr,"%s'%s'" , - (debug ==2 ) ?"\n " :" " , args[i]); + (debug ==2 ) ?"\n " :" " , exec_args[i]);fprintf (stderr,"\n" ); } } -if (execv (args[0 ], args)) +if (execv (exec_args[0 ], exec_args))perror (path);free (args);
Signature · 1 line
2.5.3
PATCH v2 16/18 toolchain-wrapper: support change of BR2_CCACHE
Arnout,
Le 04/10/2015 14:28, Arnout Vandecappelle (Essensium/Mind) a ?crit :
Quoted 5 lines
By moving the ccache call to the toolchain wrapper, the following scenario no longer works:
make foo-dirclean all BR2_CCACHE=
Quoted 7 lines
That's a sometimes useful call to check if some failure is perhaps caused by ccache.
We can enable this scenario again by exporting
BR_NO_CCACHEwhenBR2_CCACHEis not set, and by handling this in the toolchain wrapper.
I tried this patch with the whole series applied (I removed the hash to ease the reading)
$ make O=test/ccache/ busybox BR2_DEBUG_WRAPPER=2
Toolchain wrapper executing:CCACHE_COMPILERCHECK=string:[hash]
CCACHE_BASEDIR=/home/naourr/git/buildroot/test/ccache
'/home/naourr/git/buildroot/test/ccache/host/usr/bin/ccache'
'/home/naourr/git/buildroot/test/ccache/host/usr/bin/i686-buildroot-linux-uclibc-gcc.real'
'--sysroot'
'/home/naourr/git/buildroot/test/ccache/host/usr/i686-buildroot-linux-uclibc/sysroot'
'-E'
'-xc'
'-'$ make O=test/ccache/ busybox BR2_DEBUG_WRAPPER=2 BR2_CCACHE=
Toolchain wrapper executing:CCACHE_COMPILERCHECK=string:[hash]
CCACHE_BASEDIR=/home/naourr/git/buildroot/test/ccache
'/home/naourr/git/buildroot/test/ccache/host/usr/bin/i686-buildroot-linux-uclibc-gcc.real'
'--sysroot'
'/home/naourr/git/buildroot/test/ccache/host/usr/i686-buildroot-linux-uclibc/sysroot'
'-E'
'-xc'
'-'Best regards,
Romain
Quoted 66 lines
---
v2: No change, but no longer RFC.--- Makefile | 2 ++ toolchain/toolchain-wrapper.c | 15 +++++++++++---- 2 files changed, 13 insertions(+), 4 deletions(-)Makefile +2
-0@@ -375,6 +375,8 @@ BR_CACHE_DIR = $(call qstrip,$(BR2_CCACHE_DIR)) export BR_CACHE_DIR HOSTCC := $(CCACHE) $(HOSTCC) HOSTCXX := $(CCACHE) $(HOSTCXX) +else +export BR_NO_CCACHE endif # Scripts in support/ or post-build scripts may need to referencetoolchain/toolchain-wrapper.c +11
-4@@ -95,7 +95,7 @@ static void check_unsafe_path(const char *path, int paranoid)int main (int argc,char **argv) { -char **args, **cur; +char **args, **cur, **exec_args;char *relbasedir, *absbasedir;char *progpath = argv[0 ];char *basename;@@ -237,6 +237,13 @@ int main(int argc, char **argv)/* finish with NULL termination */ *cur =NULL ; + exec_args = args; +#ifdef BR_CCACHE +if (getenv ("BR_NO_CCACHE" )) +/* Skip the ccache call */ + exec_args++; +#endif +/* Debug the wrapper to see actual arguments passed to * the compiler: * unset, empty, or 0: do not trace @@ -247,14 +254,14 @@ int main(int argc, char **argv) debug =atoi (env_debug);if (debug >0 ) {fprintf (stderr,"Toolchain wrapper executing:" ); -for (i =0 ; args[i]; i++) +for (i =0 ; exec_args[i]; i++)fprintf (stderr,"%s'%s'" , - (debug ==2 ) ?"\n " :" " , args[i]); + (debug ==2 ) ?"\n " :" " , exec_args[i]);fprintf (stderr,"\n" ); } } -if (execv (args[0 ], args)) +if (execv (exec_args[0 ], exec_args))perror (path);free (args);
PATCH v2 17/18 ccache: use mtime for external toolchain, CONF_OPTS for internal toolchain
In reply to Arnout Vandecappelle (Essensium/Mind)Our current ccache disables hashing of the compiler executable itself, because using the default 'mtime' doesn't work in buildroot: we always rebuild the compiler, so the mtime is always different, so the cache always misses.
However, in the current situation, if a user changes the compiler configuration (which would result in the compiler generating different object files than before) and does 'make clean all', ccache may in fact reuse object files from the previous run. This rarely gives problems, because
(1) the cache expires quite quickly (it's only 1GB by default),
(2) radically changing compiler options will cause cache misses because different header files are used,
(3) many compiler changes (e.g. changing -mtune) have little practical effect because the resulting code is usually still compatible,
(4) we currently don't use CCACHE_BASEDIR, and almost all object files will contain an absolute path (e.g. in debug info), so when building in a different directory, most of it will miss,
(5) we do mostly build test, and many of the potential problems only appear at runtime.
Still, when ccache _does_ use the wrong cached object files, the effects are really weird and hard to debug. Also, we want reproducible builds and obviously the above makes builds non-reproducible. So we have a FAQ entry that warns against using ccache and tells the user to clear the cache in case of problems.
Now that ccache is called from the toolchain wrapper, it is in fact possible to at least use the 'mtime' compiler hash for the external toolchain and for the host-gcc. Indeed, in this case, the compiler executable comes from a tarball so the mtime will be a good reference for it state. Therefore, the patch (sed script) that changes the default from 'mtime' to 'none' is removed.
For the internal toolchain, we can do better by providing a hash of the relevant toolchain options. We are only interested in things that affect the compiler itself, because ccache also processes the header files and it doesn't look at libraries because it doesn't cache the link step, just compilation. Everything that affects the compiler itself can nicely be summarised in $(HOST_GCC_FINAL_CONF_OPTS). Of course, also the compiler source itself is relevant, so the source tarball and all the patches are included in the hash. For this purpose, a new HOST_GCC_XTENSA_OVERLAY_TAR is introduced.
The following procedure tests the ccache behaviour:
Use this defconfig:
BR2_arm=y
BR2_CCACHE=y
make
readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
-> Tag_CPU_name: "ARM926EJ-S"
Now make menuconfig, change variant into BR2_cortex_a9
make clean; make
readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
-> Tag_CPU_name: "ARM926EJ-S"
should be "Cortex-A9"
After this commit, it is "Cortex-A9".
---
v2: - Add BR2_GLOBAL_PATCH_DIR to the patches used to generate the hash.
[Samuel]
- Use $(wildcard ...) for the patches, so missing patch dir (e.g. for OVERRIDE_SRCDIR) doesn't lead to an error. [Arnout]
- Remove redundant newline [Romain]
--- package/ccache/ccache.mk | 5 ----- package/gcc/gcc.mk | 29 +++++++++++++++++++++++++++-- toolchain/toolchain-wrapper.c | 12 ++++++++++++ 3 files changed, 39 insertions(+), 7 deletions(-)
package/ccache/ccache.mk +0 -5
@@ -26,15 +26,10 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib # is already used by autotargets for the ccache package. # BR_CACHE_DIR is exported by Makefile based on config option # BR2_CCACHE_DIR. -# - ccache shouldn't use the compiler binary mtime to detect a change in -# the compiler, because in the context of Buildroot, that completely -# defeats the purpose of ccache. Of course, that leaves the user -# responsible for purging its cache when the compiler changes. # - Change hard-coded last-ditch default to match path in .config, to avoid # the need to specify BR_CACHE_DIR when invoking ccache directly. define HOST_CCACHE_PATCH_CONFIGURATION sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c - sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c endef
package/gcc/gcc.mk +27 -2
@@ -23,9 +23,10 @@ GCC_SOURCE ?= gcc-$(GCC_VERSION).tar.bz2 # Xtensa special hook # +HOST_GCC_XTENSA_OVERLAY_TAR = $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,$(BR2_XTENSA_CORE_NAME)).tar + define HOST_GCC_XTENSA_OVERLAY_EXTRACT - tar xf $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,\ - $(BR2_XTENSA_CORE_NAME)).tar -C $(@D) --strip-components=1 gcc + tar xf $(HOST_GCC_XTENSA_OVERLAY_TAR) -C $(@D) --strip-components=1 gcc endef #
@@ -237,6 +238,30 @@ endif HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CROSS_PATH_SUFFIX='".real"' +ifeq ($(BR2_CCACHE),y) +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE) +# Cfr. PATCH_BASE_DIRS in .stamp_patched, but we catch both versioned and +# unversioned patches unconditionally +HOST_GCC_COMMON_CCACHE_HASH_FILES += \ + $(wildcard \ + package/gcc/$(GCC_VERSION)/*.patch \ + $(addsuffix $((PKG)_RAWNAME)/$(GCC_VERSION)/*.patch,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR))) \ + $(addsuffix $((PKG)_RAWNAME)/*.patch,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)))) +ifeq ($(BR2_xtensa),y) +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(HOST_GCC_XTENSA_OVERLAY_TAR) +endif +ifeq ($(ARCH),powerpc) +ifneq ($(BR2_SOFT_FLOAT),) +HOST_GCC_COMMON_CCACHE_HASH_FILES += package/gcc/$(GCC_VERSION)/1000-powerpc-link-with-math-lib.patch.conditional +endif +endif + +HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE_HASH=\"`\ + printf '%s' $($(PKG)_CONF_OPTS) $(GCC_VERSION) \ + | sha256sum - $(HOST_GCC_COMMON_CCACHE_HASH_FILES) \ + | cut -c -64 | tr -d '\n'`\" +endif # BR2_CCACHE + # The LTO support in gcc creates wrappers for ar, ranlib and nm which load # the lto plugin. These wrappers are called *-gcc-ar, *-gcc-ranlib, and # *-gcc-nm and should be used instead of the real programs when -flto is
toolchain/toolchain-wrapper.c +12 -0
@@ -254,6 +254,10 @@ int main(int argc, char **argv) debug =atoi (env_debug);if (debug >0 ) {fprintf (stderr,"Toolchain wrapper executing:" ); +#ifdef BR_CCACHE_HASH +fprintf (stderr,"CCACHE_COMPILERCHECK=string:" BR_CCACHE_HASH "%s" , + (debug ==2 ) ?"\n " :" " ); +#endif for (i =0 ; exec_args[i]; i++)fprintf (stderr,"%s'%s'" , (debug ==2 ) ?"\n " :" " , exec_args[i]);
@@ -261,6 +265,14 @@ int main(int argc, char **argv) } } +#ifdef BR_CCACHE_HASH +/* Allow compilercheck to be overridden through the environment */ +if (setenv ("CCACHE_COMPILERCHECK" ,"string:" BR_CCACHE_HASH ,0 )) { +perror (__FILE__": Failed to set CCACHE_COMPILERCHECK" ); +return 3 ; + } +#endif +if (execv (exec_args[0 ], exec_args))perror (path);
Signature · 1 line
2.5.3
PATCH v2 17/18 ccache: use mtime for external toolchain, CONF_OPTS for internal toolchain
Arnout,
Le 04/10/2015 14:28, Arnout Vandecappelle (Essensium/Mind) a ?crit :
Quoted 152 lines
Our current ccache disables hashing of the compiler executable itself, because using the default 'mtime' doesn't work in buildroot: we always rebuild the compiler, so the mtime is always different, so the cache always misses.
However, in the current situation, if a user changes the compiler configuration (which would result in the compiler generating different object files than before) and does 'make clean all', ccache may in fact reuse object files from the previous run. This rarely gives problems, because
(1) the cache expires quite quickly (it's only 1GB by default),
(2) radically changing compiler options will cause cache misses because different header files are used,
(3) many compiler changes (e.g. changing -mtune) have little practical effect because the resulting code is usually still compatible,
(4) we currently don't useCCACHE_BASEDIR, and almost all object files will contain an absolute path (e.g. in debug info), so when building in a different directory, most of it will miss,
(5) we do mostly build test, and many of the potential problems only appear at runtime.
Still, when ccache _does_ use the wrong cached object files, the effects are really weird and hard to debug. Also, we want reproducible builds and obviously the above makes builds non-reproducible. So we have a FAQ entry that warns against using ccache and tells the user to clear the cache in case of problems.Now that ccache is called from the toolchain wrapper, it is in fact possible to at least use the 'mtime' compiler hash for the external toolchain and for the host-gcc. Indeed, in this case, the compiler executable comes from a tarball so the mtime will be a good reference for it state. Therefore, the patch (sed script) that changes the default from 'mtime' to 'none' is removed.
For the internal toolchain, we can do better by providing a hash of the relevant toolchain options. We are only interested in things that affect the compiler itself, because ccache also processes the header files and it doesn't look at libraries because it doesn't cache the link step, just compilation. Everything that affects the compiler itself can nicely be summarised in $(HOST_GCC_FINAL_CONF_OPTS). Of course, also the compiler source itself is relevant, so the source tarball and all the patches are included in the hash. For this purpose, a new
HOST_GCC_XTENSA_OVERLAY_TARis introduced.The following procedure tests the ccache behaviour:
Use this defconfig:
BR2_arm=y
BR2_CCACHE=ymake
readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
->Tag_CPU_name: "ARM926EJ-S"Now make menuconfig, change variant into
BR2_cortex_a9make clean; make
readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
->Tag_CPU_name: "ARM926EJ-S"
should be "Cortex-A9"After this commit, it is "Cortex-A9".
Cc Danomi Manchego <[email protected]>Cc K?roly Kasza <[email protected]>Cc Samuel Martin <[email protected]>Cc Romain Naour <[email protected]>---
v2: - AddBR2_GLOBAL_PATCH_DIRto the patches used to generate the hash.
[Samuel]
- Use $(wildcard ...) for the patches, so missing patch dir (e.g. forOVERRIDE_SRCDIR) doesn't lead to an error. [Arnout]
- Remove redundant newline [Romain]--- package/ccache/ccache.mk | 5 ----- package/gcc/gcc.mk | 29 +++++++++++++++++++++++++++-- toolchain/toolchain-wrapper.c | 12 ++++++++++++ 3 files changed, 39 insertions(+), 7 deletions(-)package/ccache/ccache.mk +0
-5@@ -26,15 +26,10 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib # is already used by autotargets for the ccache package. # BR_CACHE_DIR is exported by Makefile based on config option # BR2_CCACHE_DIR. -# - ccache shouldn't use the compiler binary mtime to detect a change in -# the compiler, because in the context of Buildroot, that completely -# defeats the purpose of ccache. Of course, that leaves the user -# responsible for purging its cache when the compiler changes. # - Change hard-coded last-ditch default to match path in .config, to avoid # the need to specify BR_CACHE_DIR when invoking ccache directly. define HOST_CCACHE_PATCH_CONFIGURATION sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c - sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c endefpackage/gcc/gcc.mk +27
-2@@ -23,9 +23,10 @@ GCC_SOURCE ?= gcc-$(GCC_VERSION).tar.bz2 # Xtensa special hook # +HOST_GCC_XTENSA_OVERLAY_TAR = $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,$(BR2_XTENSA_CORE_NAME)).tar + define HOST_GCC_XTENSA_OVERLAY_EXTRACT - tar xf $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,\ - $(BR2_XTENSA_CORE_NAME)).tar -C $(@D) --strip-components=1 gcc + tar xf $(HOST_GCC_XTENSA_OVERLAY_TAR) -C $(@D) --strip-components=1 gcc endef #@@ -237,6 +238,30 @@ endif HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CROSS_PATH_SUFFIX='".real"' +ifeq ($(BR2_CCACHE),y) +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE) +# Cfr. PATCH_BASE_DIRS in .stamp_patched, but we catch both versioned and +# unversioned patches unconditionally +HOST_GCC_COMMON_CCACHE_HASH_FILES += \ + $(wildcard \ + package/gcc/$(GCC_VERSION)/*.patch \ + $(addsuffix $((PKG)_RAWNAME)/$(GCC_VERSION)/*.patch,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR))) \ + $(addsuffix $((PKG)_RAWNAME)/*.patch,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)))) +ifeq ($(BR2_xtensa),y) +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(HOST_GCC_XTENSA_OVERLAY_TAR) +endif +ifeq ($(ARCH),powerpc) +ifneq ($(BR2_SOFT_FLOAT),) +HOST_GCC_COMMON_CCACHE_HASH_FILES += package/gcc/$(GCC_VERSION)/1000-powerpc-link-with-math-lib.patch.conditional +endif +endif + +HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE_HASH=\"`\ + printf '%s' $($(PKG)_CONF_OPTS) $(GCC_VERSION) \ + | sha256sum - $(HOST_GCC_COMMON_CCACHE_HASH_FILES) \ + | cut -c -64 | tr -d '\n'`\" +endif # BR2_CCACHE + # The LTO support in gcc creates wrappers for ar, ranlib and nm which load # the lto plugin. These wrappers are called *-gcc-ar, *-gcc-ranlib, and # *-gcc-nm and should be used instead of the real programs when -flto istoolchain/toolchain-wrapper.c +0
-0
toolchain/toolchain-wrapper.c int main(int argc, char **argv) · line 260
Quoted 7 lines
toolchain/toolchain-wrapper.c +4
-0@@ -254,6 +254,10 @@ int main(int argc, char **argv) debug =atoi (env_debug);if (debug >0 ) {fprintf (stderr,"Toolchain wrapper executing:" ); +#ifdef BR_CCACHE_HASH +fprintf (stderr,"CCACHE_COMPILERCHECK=string:" BR_CCACHE_HASH "%s" , + (debug ==2 ) ?"\n " :" " ); +#endif
There is probably a formatting issue here when debug==2:
Toolchain wrapper executing:CCACHE_COMPILERCHECK=string:[hash]
CCACHE_BASEDIR=/home/naourr/git/buildroot/test/ccache
'/home/naourr/git/buildroot/test/ccache/host/usr/bin/ccache'
'/home/naourr/git/buildroot/test/ccache/host/usr/bin/i686-buildroot-linux-uclibc-gcc.real'
'--sysroot'
The fprintf line should look-like:
fprintf(stderr, "%sCCACHE_COMPILERCHECK=string:'" BR_CCACHE_HASH "'",
(debug == 2) ? "\n " : " ");
Toolchain wrapper executing:
CCACHE_COMPILERCHECK='string:[hash]' CCACHE_BASEDIR='/home/naourr/git/buildroot/test/ccache'
'/home/naourr/git/buildroot/test/ccache/host/usr/bin/i686-buildroot-linux-uclibc-gcc.real'
'--sysroot'
'/home/naourr/git/buildroot/test/ccache/host/usr/i686-buildroot-linux-uclibc/sysroot'
'-E'
'-xc'
'-'toolchain/toolchain-wrapper.c int main(int argc, char **argv)
Quoted 4 lines
toolchain/toolchain-wrapper.c +0
-0for (i =0 ; exec_args[i]; i++)fprintf (stderr,"%s'%s'" , (debug ==2 ) ?"\n " :" " , exec_args[i]);@@ -261,6 +265,14 @@ int main(int argc, char **argv)
Same here.
Best regards,
Romain
Quoted 15 lines
toolchain/toolchain-wrapper.c +8
-0} } +#ifdef BR_CCACHE_HASH +/* Allow compilercheck to be overridden through the environment */ +if (setenv ("CCACHE_COMPILERCHECK" ,"string:" BR_CCACHE_HASH ,0 )) { +perror (__FILE__": Failed to set CCACHE_COMPILERCHECK" ); +return 3 ; + } +#endif +if (execv (exec_args[0 ], exec_args))perror (path);
PATCH v3 1/1 ccache: use mtime for external toolchain, CONF_OPTS for internal toolchain
In reply to Arnout Vandecappelle (Essensium/Mind)Our current ccache disables hashing of the compiler executable itself, because using the default 'mtime' doesn't work in buildroot: we always rebuild the compiler, so the mtime is always different, so the cache always misses.
However, in the current situation, if a user changes the compiler configuration (which would result in the compiler generating different object files than before) and does 'make clean all', ccache may in fact reuse object files from the previous run. This rarely gives problems, because
(1) the cache expires quite quickly (it's only 1GB by default),
(2) radically changing compiler options will cause cache misses because different header files are used,
(3) many compiler changes (e.g. changing -mtune) have little practical effect because the resulting code is usually still compatible,
(4) we currently don't use CCACHE_BASEDIR, and almost all object files will contain an absolute path (e.g. in debug info), so when building in a different directory, most of it will miss,
(5) we do mostly build test, and many of the potential problems only appear at runtime.
Still, when ccache _does_ use the wrong cached object files, the effects are really weird and hard to debug. Also, we want reproducible builds and obviously the above makes builds non-reproducible. So we have a FAQ entry that warns against using ccache and tells the user to clear the cache in case of problems.
Now that ccache is called from the toolchain wrapper, it is in fact possible to at least use the 'mtime' compiler hash for the external toolchain and for the host-gcc. Indeed, in this case, the compiler executable comes from a tarball so the mtime will be a good reference for it state. Therefore, the patch (sed script) that changes the default from 'mtime' to 'none' is removed.
For the internal toolchain, we can do better by providing a hash of the relevant toolchain options. We are only interested in things that affect the compiler itself, because ccache also processes the header files and it doesn't look at libraries because it doesn't cache the link step, just compilation. Everything that affects the compiler itself can nicely be summarised in $(HOST_GCC_FINAL_CONF_OPTS). Of course, also the compiler source itself is relevant, so the source tarball and all the patches are included in the hash. For this purpose, a new HOST_GCC_XTENSA_OVERLAY_TAR is introduced.
The following procedure tests the ccache behaviour:
Use this defconfig:
BR2_arm=y
BR2_CCACHE=y
make
readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
-> Tag_CPU_name: "ARM926EJ-S"
Now make menuconfig, change variant into BR2_cortex_a9
make clean; make
readelf -A output/build/uclibc-1.0.6/libc/signal/signal.os
-> Tag_CPU_name: "ARM926EJ-S"
should be "Cortex-A9"
After this commit, it is "Cortex-A9".
---
v3: - Fix format string of BR2_DEBUG_WRAPPER. [Romain]
v2: - Add BR2_GLOBAL_PATCH_DIR to the patches used to generate the hash.
[Samuel]
- Use $(wildcard ...) for the patches, so missing patch dir (e.g. for OVERRIDE_SRCDIR) doesn't lead to an error. [Arnout]
- Remove redundant newline [Romain]
--- package/ccache/ccache.mk | 5 ----- package/gcc/gcc.mk | 29 +++++++++++++++++++++++++++-- toolchain/toolchain-wrapper.c | 12 ++++++++++++ 3 files changed, 39 insertions(+), 7 deletions(-)
package/ccache/ccache.mk +0 -5
@@ -26,15 +26,10 @@ HOST_CCACHE_CONF_OPTS += --with-bundled-zlib # is already used by autotargets for the ccache package. # BR_CACHE_DIR is exported by Makefile based on config option # BR2_CCACHE_DIR. -# - ccache shouldn't use the compiler binary mtime to detect a change in -# the compiler, because in the context of Buildroot, that completely -# defeats the purpose of ccache. Of course, that leaves the user -# responsible for purging its cache when the compiler changes. # - Change hard-coded last-ditch default to match path in .config, to avoid # the need to specify BR_CACHE_DIR when invoking ccache directly. define HOST_CCACHE_PATCH_CONFIGURATION sed -i 's,getenv("CCACHE_DIR"),getenv("BR_CACHE_DIR"),' $(@D)/ccache.c - sed -i 's,conf->compiler_check = x_strdup("mtime"),conf->compiler_check = x_strdup("none"),' $(@D)/conf.c sed -i 's,"%s/.ccache","$(BR_CACHE_DIR)",' $(@D)/conf.c endef
package/gcc/gcc.mk +27 -2
@@ -23,9 +23,10 @@ GCC_SOURCE ?= gcc-$(GCC_VERSION).tar.bz2 # Xtensa special hook # +HOST_GCC_XTENSA_OVERLAY_TAR = $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,$(BR2_XTENSA_CORE_NAME)).tar + define HOST_GCC_XTENSA_OVERLAY_EXTRACT - tar xf $(BR2_XTENSA_OVERLAY_DIR)/xtensa_$(call qstrip,\ - $(BR2_XTENSA_CORE_NAME)).tar -C $(@D) --strip-components=1 gcc + tar xf $(HOST_GCC_XTENSA_OVERLAY_TAR) -C $(@D) --strip-components=1 gcc endef #
@@ -237,6 +238,30 @@ endif HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CROSS_PATH_SUFFIX='".real"' +ifeq ($(BR2_CCACHE),y) +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(DL_DIR)/$(GCC_SOURCE) +# Cfr. PATCH_BASE_DIRS in .stamp_patched, but we catch both versioned and +# unversioned patches unconditionally +HOST_GCC_COMMON_CCACHE_HASH_FILES += \ + $(wildcard \ + package/gcc/$(GCC_VERSION)/*.patch \ + $(addsuffix $((PKG)_RAWNAME)/$(GCC_VERSION)/*.patch,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR))) \ + $(addsuffix $((PKG)_RAWNAME)/*.patch,$(call qstrip,$(BR2_GLOBAL_PATCH_DIR)))) +ifeq ($(BR2_xtensa),y) +HOST_GCC_COMMON_CCACHE_HASH_FILES += $(HOST_GCC_XTENSA_OVERLAY_TAR) +endif +ifeq ($(ARCH),powerpc) +ifneq ($(BR2_SOFT_FLOAT),) +HOST_GCC_COMMON_CCACHE_HASH_FILES += package/gcc/$(GCC_VERSION)/1000-powerpc-link-with-math-lib.patch.conditional +endif +endif + +HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE_HASH=\"`\ + printf '%s' $($(PKG)_CONF_OPTS) $(GCC_VERSION) \ + | sha256sum - $(HOST_GCC_COMMON_CCACHE_HASH_FILES) \ + | cut -c -64 | tr -d '\n'`\" +endif # BR2_CCACHE + # The LTO support in gcc creates wrappers for ar, ranlib and nm which load # the lto plugin. These wrappers are called *-gcc-ar, *-gcc-ranlib, and # *-gcc-nm and should be used instead of the real programs when -flto is
toolchain/toolchain-wrapper.c +12 -0
@@ -264,6 +264,10 @@ int main(int argc, char **argv) debug =atoi (env_debug);if (debug >0 ) {fprintf (stderr,"Toolchain wrapper executing:" ); +#ifdef BR_CCACHE_HASH +fprintf (stderr,"%sCCACHE_COMPILERCHECK='string:" BR_CCACHE_HASH "'" , + (debug ==2 ) ?"\n " :" " ); +#endif for (i =0 ; exec_args[i]; i++)fprintf (stderr,"%s'%s'" , (debug ==2 ) ?"\n " :" " , exec_args[i]);
@@ -271,6 +275,14 @@ int main(int argc, char **argv) } } +#ifdef BR_CCACHE_HASH +/* Allow compilercheck to be overridden through the environment */ +if (setenv ("CCACHE_COMPILERCHECK" ,"string:" BR_CCACHE_HASH ,0 )) { +perror (__FILE__": Failed to set CCACHE_COMPILERCHECK" ); +return 3 ; + } +#endif +if (execv (exec_args[0 ], exec_args))perror (path);
Signature · 1 line
2.5.3
RFC PATCH v2 18/18 ccache: support changing the output directory
In reply to Arnout Vandecappelle (Essensium/Mind)When building in a different output directory than the original build, there will currently be a lot of ccache misses because in many cases there is some -I/... absolute path in the compilation. ccache has an option CCACHE_BASEDIR to substitute absolute paths with relative paths, so they wil be the same in the hash (and in the output).
Since there are some disadvantages to this path rewriting, it is made optional as BR2_CCACHE_USE_BASEDIR. It defaults to y because the usefulness of ccache is severely reduced without this option.
In addition to CCACHE_BASEDIR, we also substitute away the occurences of $(HOST_DIR) in the calculation of the compiler hash. This is done regardless of the setting of BR2_CCACHE_USE_BASEDIR because it's quite harmless.
---
Perhaps we should use $(HOST_DIR) instead of $(BASE_DIR) as CCACHE_BASEDIR, because most of the time an absolute path will point into the host dir. Not always, though (e.g. references to LINUX_DIR). And most people will have $(HOST_DIR) inside $(BASE_DIR) so the latter works well for them.
It's hard to be sure with ccache, but in my tests I still had the impression that there were less cache hits when rebuilding in a different output directory than when rebuilding in the same directory. Oh well...
--- Config.in | 27 +++++++++++++++++++++++++++ docs/manual/ccache-support.txt | 20 ++++++++++++++++++++ package/gcc/gcc.mk | 4 +++- toolchain/toolchain-wrapper.c | 11 +++++++++++ toolchain/toolchain-wrapper.mk | 4 ++++ 5 files changed, 65 insertions(+), 1 deletion(-)
Config.in +27 -0
@@ -286,6 +286,33 @@ config BR2_CCACHE_INITIAL_SETUP These initial settings are applied after ccache has been compiled. +config BR2_CCACHE_USE_BASEDIR + bool "Use relative paths" + default y + help + Allow ccache to convert absolute paths within the output + directory into relative paths. + + During the build, many -I include directives are given with + an absolute path. These absolute paths end up in the hashes + that are computed by ccache. Therefore, when you build from a + different directory, the hash will be different and the + cached object will not be used. + + To improve cache performance, set this option to y. This + allows ccache to rewrite absolute paths within the output + directory into relative paths. Note that only paths within + the output directory will be rewritten; therefore, if you + change BR2_HOST_DIR to point outside the output directory and + subsequently move it to a different location, this will lead + to cache misses. + + This option has as a result that the debug information in the + object files also has only relative paths. Therefore, make + sure you cd to the build directory before starting gdb. See + the section "COMPILING IN DIFFERENT DIRECTORIES" in the + ccache manual for more information. + endif config BR2_DEPRECATED
docs/manual/ccache-support.txt +20 -0
@@ -33,3 +33,23 @@ make CCACHE_OPTIONS="--max-size=5G" ccache-options # zero statistics counters make CCACHE_OPTIONS="--zero-stats" ccache-options ----------------- + ++ccache+ makes a hash of the source files and of the compiler options. +If a compiler option is different, the cached object file will not be +used. Many compiler options, however, contain an absolute path to the +staging directory. Because of this, building in a different output +directory would lead to many cache misses. + +To avoid this issue, buildroot has the +Use relative paths+ option +(+BR2_CCACHE_USE_BASEDIR+). This will rewrite all absolute paths that +point inside the output directory into relative paths. Thus, changing +the output directory no longer leads to cache misses. + +A disadvantage of the relative paths is that they also end up to be +relative paths in the object file. Therefore, for example, the debugger +will no longer find the file, unless you cd to the output directory +first. + +See https://ccache.samba.org/manual.html#_compiling_in_different_directories[the +ccache manual's section on "Compiling in different directories"] for +more details about this rewriting of absolute paths.
package/gcc/gcc.mk +3 -1
@@ -256,8 +256,10 @@ HOST_GCC_COMMON_CCACHE_HASH_FILES += package/gcc/$(GCC_VERSION)/1000-powerpc-lin endif endif +# _CONF_OPTS contains some references to the absolute path of $(HOST_DIR), +# so substitute those away. HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE_HASH=\"`\ - printf '%s' $($(PKG)_CONF_OPTS) $(GCC_VERSION) \ + printf '%s' $(subst $(HOST_DIR), at HOST_DIR@,$($(PKG)_CONF_OPTS)) \ | sha256sum - $(HOST_GCC_COMMON_CCACHE_HASH_FILES) \ | cut -c -64 | tr -d '\n'`\" endif # BR2_CCACHE
toolchain/toolchain-wrapper.c +11 -0
@@ -258,6 +258,10 @@ int main(int argc, char **argv)fprintf (stderr,"CCACHE_COMPILERCHECK=string:" BR_CCACHE_HASH "%s" , (debug ==2 ) ?"\n " :" " );#endif +#ifdef BR_CCACHE_BASEDIR +fprintf (stderr,"CCACHE_BASEDIR=" BR_CCACHE_BASEDIR "%s" , + (debug ==2 ) ?"\n " :" " ); +#endif for (i =0 ; exec_args[i]; i++)fprintf (stderr,"%s'%s'" , (debug ==2 ) ?"\n " :" " , exec_args[i]);
@@ -272,6 +276,13 @@ int main(int argc, char **argv) return 3; }#endif +#ifdef BR_CCACHE_BASEDIR +/* Allow compilercheck to be overridden through the environment */ +if (setenv ("CCACHE_BASEDIR" ,BR_CCACHE_BASEDIR ,0 )) { +perror (__FILE__": Failed to set CCACHE_BASEDIR" ); +return 3 ; + } +#endif if (execv (exec_args[0 ], exec_args))perror (path);
toolchain/toolchain-wrapper.mk +4 -0
@@ -20,6 +20,10 @@ ifeq ($(BR2_CCACHE),y) TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE='"$(CCACHE)"' endif +ifeq ($(BR2_CCACHE_USE_BASEDIR),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE_BASEDIR='"$(BASE_DIR)"' +endif + # For simplicity, build directly into the install location define TOOLCHAIN_BUILD_WRAPPER $(Q)mkdir -p $(HOST_DIR)/usr/bin
Signature · 1 line
2.5.3
RFC PATCH v3 1/1 ccache: support changing the output directory
When building in a different output directory than the original build, there will currently be a lot of ccache misses because in many cases there is some -I/... absolute path in the compilation. ccache has an option CCACHE_BASEDIR to substitute absolute paths with relative paths, so they wil be the same in the hash (and in the output).
Since there are some disadvantages to this path rewriting, it is made optional as BR2_CCACHE_USE_BASEDIR. It defaults to y because the usefulness of ccache is severely reduced without this option.
In addition to CCACHE_BASEDIR, we also substitute away the occurences of $(HOST_DIR) in the calculation of the compiler hash. This is done regardless of the setting of BR2_CCACHE_USE_BASEDIR because it's quite harmless.
---
Perhaps we should use $(HOST_DIR) instead of $(BASE_DIR) as CCACHE_BASEDIR, because most of the time an absolute path will point into the host dir. Not always, though (e.g. references to LINUX_DIR). And most people will have $(HOST_DIR) inside $(BASE_DIR) so the latter works well for them.
It's hard to be sure with ccache, but in my tests I still had the impression that there were less cache hits when rebuilding in a different output directory than when rebuilding in the same directory. Oh well...
v3: - Fix format string of BR2_DEBUG_WRAPPER. [Romain]
--- Config.in | 27 +++++++++++++++++++++++++++ docs/manual/ccache-support.txt | 20 ++++++++++++++++++++ package/gcc/gcc.mk | 4 +++- toolchain/toolchain-wrapper.c | 11 +++++++++++ toolchain/toolchain-wrapper.mk | 4 ++++ 5 files changed, 65 insertions(+), 1 deletion(-)
Config.in +27 -0
@@ -286,6 +286,33 @@ config BR2_CCACHE_INITIAL_SETUP These initial settings are applied after ccache has been compiled. +config BR2_CCACHE_USE_BASEDIR + bool "Use relative paths" + default y + help + Allow ccache to convert absolute paths within the output + directory into relative paths. + + During the build, many -I include directives are given with + an absolute path. These absolute paths end up in the hashes + that are computed by ccache. Therefore, when you build from a + different directory, the hash will be different and the + cached object will not be used. + + To improve cache performance, set this option to y. This + allows ccache to rewrite absolute paths within the output + directory into relative paths. Note that only paths within + the output directory will be rewritten; therefore, if you + change BR2_HOST_DIR to point outside the output directory and + subsequently move it to a different location, this will lead + to cache misses. + + This option has as a result that the debug information in the + object files also has only relative paths. Therefore, make + sure you cd to the build directory before starting gdb. See + the section "COMPILING IN DIFFERENT DIRECTORIES" in the + ccache manual for more information. + endif config BR2_DEPRECATED
docs/manual/ccache-support.txt +20 -0
@@ -33,3 +33,23 @@ make CCACHE_OPTIONS="--max-size=5G" ccache-options # zero statistics counters make CCACHE_OPTIONS="--zero-stats" ccache-options ----------------- + ++ccache+ makes a hash of the source files and of the compiler options. +If a compiler option is different, the cached object file will not be +used. Many compiler options, however, contain an absolute path to the +staging directory. Because of this, building in a different output +directory would lead to many cache misses. + +To avoid this issue, buildroot has the +Use relative paths+ option +(+BR2_CCACHE_USE_BASEDIR+). This will rewrite all absolute paths that +point inside the output directory into relative paths. Thus, changing +the output directory no longer leads to cache misses. + +A disadvantage of the relative paths is that they also end up to be +relative paths in the object file. Therefore, for example, the debugger +will no longer find the file, unless you cd to the output directory +first. + +See https://ccache.samba.org/manual.html#_compiling_in_different_directories[the +ccache manual's section on "Compiling in different directories"] for +more details about this rewriting of absolute paths.
package/gcc/gcc.mk +3 -1
@@ -256,8 +256,10 @@ HOST_GCC_COMMON_CCACHE_HASH_FILES += package/gcc/$(GCC_VERSION)/1000-powerpc-lin endif endif +# _CONF_OPTS contains some references to the absolute path of $(HOST_DIR), +# so substitute those away. HOST_GCC_COMMON_TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE_HASH=\"`\ - printf '%s' $($(PKG)_CONF_OPTS) $(GCC_VERSION) \ + printf '%s' $(subst $(HOST_DIR), at HOST_DIR@,$($(PKG)_CONF_OPTS)) \ | sha256sum - $(HOST_GCC_COMMON_CCACHE_HASH_FILES) \ | cut -c -64 | tr -d '\n'`\" endif # BR2_CCACHE
toolchain/toolchain-wrapper.c +11 -0
@@ -268,6 +268,10 @@ int main(int argc, char **argv)fprintf (stderr,"%sCCACHE_COMPILERCHECK='string:" BR_CCACHE_HASH "'" , (debug ==2 ) ?"\n " :" " );#endif +#ifdef BR_CCACHE_BASEDIR +fprintf (stderr,"%sCCACHE_BASEDIR='" BR_CCACHE_BASEDIR "'" , + (debug ==2 ) ?"\n " :" " ); +#endif for (i =0 ; exec_args[i]; i++)fprintf (stderr,"%s'%s'" , (debug ==2 ) ?"\n " :" " , exec_args[i]);
@@ -282,6 +286,13 @@ int main(int argc, char **argv) return 3; }#endif +#ifdef BR_CCACHE_BASEDIR +/* Allow compilercheck to be overridden through the environment */ +if (setenv ("CCACHE_BASEDIR" ,BR_CCACHE_BASEDIR ,0 )) { +perror (__FILE__": Failed to set CCACHE_BASEDIR" ); +return 3 ; + } +#endif if (execv (exec_args[0 ], exec_args))perror (path);
toolchain/toolchain-wrapper.mk +4 -0
@@ -20,6 +20,10 @@ ifeq ($(BR2_CCACHE),y) TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE endif +ifeq ($(BR2_CCACHE_USE_BASEDIR),y) +TOOLCHAIN_WRAPPER_ARGS += -DBR_CCACHE_BASEDIR='"$(BASE_DIR)"' +endif + # For simplicity, build directly into the install location define TOOLCHAIN_BUILD_WRAPPER $(Q)mkdir -p $(HOST_DIR)/usr/bin
Signature · 1 line
2.5.3
v2 cover Internal toolchain wrapper & ccache support
In reply to Arnout Vandecappelle (Essensium/Mind)Quoted 9 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Arnout" == Arnout Vandecappelle (Essensium/Mind) <[email protected]> writes:
This series introduces two features that were discussed on the list and in the Buildroot Summer Camp: a wrapper for the internal toolchain, and fixing ccache. See below why the two are combined in one series.
[Note: to make review easier, all the potentially contentious stuff from the individual patches is repeated here. So for the first few iterations, it's probably enough to reply to just this e-mail.]
Committed entire series, including the v3 versions of the 3 patches, thanks!
Signature · 1 line
Bye, Peter Korsgaard