Pull request for branch for-2011.02/fix-ccache-support
Hello,
[ Sorry, previous pull request had some mistakes. ]
Here is a patch that reworks the ccache support to make it work and make it useful. It is now a "Build options" rather than a "Toolchain option". I have tested it with both internal toolchain and external toolchain. The principle is simply to build ccache as early as possible in the build process, and then use it to cache host and target compiled files.
The ccache cache is kept in ~/.buildroot-ccache/, so that it can be shared between different builds.
I'm not using the symlink technique that was used in the past, but rather prefixing all invocation to host gcc and target gcc with "$(HOST_DIR)/usr/bin/ccache".
Here are some speed-up results :
*) First test on my laptop (dual core, 2 GB of RAM), using an ARM
external toolchain, building Busybox and a couple of packages.
-> ccache disabled: 5 minutes 29 seconds
-> ccache enabled, cold cache: 5 minutes 47 seconds -> ccache enabled, hot cache: 3 minutes 40 seconds
The test with "cold cache" is done after removing ~/.buildroot-ccache/. The test with "hot cache" is done after cleaning up the Buildroot build tree, but keeping the cache of the previous build.
Here we see that ccache has a little cost, but that once the cache is hot from previous builds, the build speed-up is interesting.
*) Second test on my build server (quad core i7, 12 GB of RAM), using
an ARM internal toolchain, building Busybox and an ext2 filesystem.
-> ccache disabled: 7 minutes 10 seconds
-> ccache enabled, cold cache: 5 minutes 43 seconds
-> ccache enabled, hot cache: 4 minutes 24 seconds
In this case we see that even when the cache is cold, ccache is useful. It's probably because gcc gets recompiled twice during the toolchain build process.
Regards,
Thomas
The following changes since commit 7e9c3a387820154fd1355f23c2669072c0c3a5f7:
Peter Korsgaard (1):
docs/news.html: add 2010.11 announce link
are available in the git repository at:
git://git.busybox.net/~tpetazzoni/git/buildroot for-2011.02/fix-ccache-support
Thomas Petazzoni (1):
ccache: rework ccache management
Config.in | 8 ++ Makefile | 27 +++- docs/buildroot.html | 26 ++++ package/Config.in | 2 +- package/Makefile.in | 10 ++- package/ccache/Config.in | 5 + package/ccache/ccache.mk | 32 +++++ toolchain/ccache/Config.in | 16 --- toolchain/ccache/Config.in.2 | 13 -- toolchain/ccache/ccache.mk | 196 ------------------------------ toolchain/dependencies/dependencies.sh | 2 +- toolchain/toolchain-buildroot.mk | 1 - toolchain/toolchain-buildroot/Config.in | 1 - toolchain/toolchain-crosstool-ng.mk | 1 - toolchain/toolchain-external.mk | 1 - toolchain/toolchain-external/ext-tool.mk | 2 +- 16 files changed, 103 insertions(+), 240 deletions(-) create mode 100644 package/ccache/Config.in create mode 100644 package/ccache/ccache.mk delete mode 100644 toolchain/ccache/Config.in delete mode 100644 toolchain/ccache/Config.in.2 delete mode 100644 toolchain/ccache/ccache.mk
Thanks,
Signature · 1 line
Thomas Petazzoni
PATCH 1/1 ccache: rework ccache management
* ccache is now a normal package (both for the host and the target).
* ccache option is now part of the "Build options" menu. It will automatically build ccache for the host before building anything, and will use it to cache builds for both host compilations and target compilations.
* bump ccache to 3.1.3
--- Config.in | 8 ++ Makefile | 27 +++- docs/buildroot.html | 26 ++++ package/Config.in | 2 +- package/Makefile.in | 10 ++- package/ccache/Config.in | 5 + package/ccache/ccache.mk | 32 +++++ toolchain/ccache/Config.in | 16 --- toolchain/ccache/Config.in.2 | 13 -- toolchain/ccache/ccache.mk | 196 ------------------------------ toolchain/dependencies/dependencies.sh | 2 +- toolchain/toolchain-buildroot.mk | 1 - toolchain/toolchain-buildroot/Config.in | 1 - toolchain/toolchain-crosstool-ng.mk | 1 - toolchain/toolchain-external.mk | 1 - toolchain/toolchain-external/ext-tool.mk | 2 +- 16 files changed, 103 insertions(+), 240 deletions(-) create mode 100644 package/ccache/Config.in create mode 100644 package/ccache/ccache.mk delete mode 100644 toolchain/ccache/Config.in delete mode 100644 toolchain/ccache/Config.in.2 delete mode 100644 toolchain/ccache/ccache.mk
Config.in +8 -0
@@ -94,6 +94,14 @@ config BR2_JLEVEL help Number of jobs to run simultaneously +config BR2_CCACHE + bool "Enable compiler cache" + help + This option will enable the use of ccache, a compiler + cache. It will cache the result of previous builds to speed + up future builds. The cache is stored in + $HOME/.buildroot-ccache. + config BR2_DEPRECATED bool "Show packages that are deprecated or obsolete" help
Makefile +20 -7
@@ -125,10 +125,13 @@ HOSTAS:=as endif ifndef HOSTCC HOSTCC:=gcc -else +HOSTCC:=$(shell which $(HOSTCC) || type -p $(HOSTCC) || echo gcc) +HOSTCC_NOCCACHE:=$(HOSTCC) endif ifndef HOSTCXX HOSTCXX:=g++ +HOSTCXX:=$(shell which $(HOSTCXX) || type -p $(HOSTCXX) || echo g++) +HOSTCXX_NOCCACHE:=$(HOSTCXX) endif ifndef HOSTFC HOSTFC:=gfortran
@@ -147,15 +150,13 @@ HOSTNM:=nm endif HOSTAR:=$(shell which $(HOSTAR) || type -p $(HOSTAR) || echo ar) HOSTAS:=$(shell which $(HOSTAS) || type -p $(HOSTAS) || echo as) -HOSTCC:=$(shell which $(HOSTCC) || type -p $(HOSTCC) || echo gcc) -HOSTCXX:=$(shell which $(HOSTCXX) || type -p $(HOSTCXX) || echo g++) HOSTFC:=$(shell which $(HOSTLD) || type -p $(HOSTLD) || echo || which g77 || type -p g77 || echo gfortran) HOSTCPP:=$(shell which $(HOSTCPP) || type -p $(HOSTCPP) || echo cpp) HOSTLD:=$(shell which $(HOSTLD) || type -p $(HOSTLD) || echo ld) HOSTLN:=$(shell which $(HOSTLN) || type -p $(HOSTLN) || echo ln) HOSTNM:=$(shell which $(HOSTNM) || type -p $(HOSTNM) || echo nm) -export HOSTAR HOSTAS HOSTCC HOSTCXX HOSTFC HOSTLD +export HOSTAR HOSTAS HOSTCC HOSTCXX HOSTFC HOSTLD HOSTCC_NOCCACHE # bash prints the name of the directory on 'cd <dir>' if CDPATH is # set, so unset it here to not cause problems. Notice that the export
@@ -231,10 +232,15 @@ PREFERRED_LIB_FLAGS:=--enable-static --enable-shared # along with the packages to build for the target. # ############################################################## + +ifeq ($(BR2_CCACHE),y) +BASE_TARGETS += host-ccache +endif + ifeq ($(BR2_TOOLCHAIN_BUILDROOT),y) -BASE_TARGETS:=uclibc-configured binutils cross_compiler uclibc-target-utils kernel-headers +BASE_TARGETS += uclibc-configured binutils cross_compiler uclibc-target-utils kernel-headers else -BASE_TARGETS:=uclibc +BASE_TARGETS += uclibc endif TARGETS:=
@@ -276,6 +282,13 @@ TARGET_SKELETON=$(TOPDIR)/fs/skeleton BR2_DEPENDS_DIR=$(BUILD_DIR)/buildroot-config +ifeq ($(BR2_CCACHE),y) +CCACHE:=$(HOST_DIR)/usr/bin/ccache +CCACHE_CACHE_DIR=$(HOME)/.buildroot-ccache +HOSTCC := $(CCACHE) $(HOSTCC) +HOSTCXX := $(CCACHE) $(HOSTCXX) +endif + include toolchain/Makefile.in include package/Makefile.in
@@ -480,7 +493,7 @@ export HOSTCFLAGS $(BUILD_DIR)/buildroot-config/%onf: mkdir -p $(@D)/lxdialog - $(MAKE) CC="$(HOSTCC)" obj=$(@D) -C $(CONFIG) -f Makefile.br $(@F) + $(MAKE) CC="$(HOSTCC_NOCCACHE)" HOSTCC="$(HOSTCC_NOCCACHE)" obj=$(@D) -C $(CONFIG) -f Makefile.br $(@F) COMMON_CONFIG_ENV = \ KCONFIG_AUTOCONFIG=$(BUILD_DIR)/buildroot-config/auto.conf \
docs/buildroot.html +26 -0
@@ -29,6 +29,7 @@ <li><a href="#buildroot_innards">How Buildroot works</a></li> <li><a href="#using_toolchain">Using the uClibc toolchain outside Buildroot</a></li> <li><a href="#external_toolchain">Use an external toolchain</a></li> + <li><a href="#ccache-support">Using <code>ccache</code> in Buildroot</li> <li><a href="#downloaded_packages">Location of downloaded packages</a></li> <li><a href="#add_packages">Adding new packages to Buildroot</a></li> <li><a href="#board_support">Creating your own board support</a></li>
@@ -681,6 +682,31 @@ endif Build options -> Toolchain and header file location</code> options. This could be useful if the toolchain must be shared with other users.</p> + <h2 id="ccache-support">Using <code>ccache</code> in Buildroot</h2> + + <p><a href="http://ccache.samba.org">ccache</a> is a compiler + cache. It stores the object files resulting from each compilation + process, and is able to skip future compilation of the same source + file (with same compiler and same arguments) by using the + pre-existing object files. When doing almost identical builds from + scratch a number of times, it can divide the build time by a + factor of two.</p> + + <p><code>ccache</code> support is integrated in Buildroot. You + just have to enable <code>Enable compiler cache</code> + in <code>Build options</code>. This will automatically build + <code>ccache</code> and use it for every host and target + compilation.</p> + + <p>The cache is located + in <code>$HOME/.buildroot-ccache</code>. It is stored outside of + Buildroot output directory so that it can be shared by separate + Buildroot builds. If you want to get rid of the cache, simply + remove this directory.</p> + + <p>You can get statistics on the cache (its size, number of hits, + misses, etc.) by running <code>make ccache-stats</code>.</p> + <h2 id="downloaded_packages">Location of downloaded packages</h2> <p>It might be useful to know that the various tarballs that are
package/Config.in +1 -1
@@ -35,7 +35,7 @@ source "package/autoconf/Config.in" source "package/automake/Config.in" source "package/bison/Config.in" source "package/bsdiff/Config.in" -source "toolchain/ccache/Config.in.2" +source "package/ccache/Config.in" if BR2_PACKAGE_BUSYBOX_SHOW_OTHERS source "package/coreutils/Config.in" endif
package/Makefile.in +9 -1
@@ -142,6 +142,14 @@ TARGET_RANLIB = $(TARGET_CROSS)ranlib TARGET_OBJCOPY = $(TARGET_CROSS)objcopy TARGET_OBJDUMP = $(TARGET_CROSS)objdump +TARGET_CC_NOCCACHE := $(TARGET_CC) +TARGET_CXX_NOCCACHE := $(TARGET_CXX) + +ifeq ($(BR2_CCACHE),y) +TARGET_CC := $(CCACHE) $(TARGET_CC) +TARGET_CXX := $(CCACHE) $(TARGET_CXX) +endif + ifeq ($(BR2_STRIP_strip),y) STRIP_DISCARD_ALL:=--discard-all STRIP_STRIP_UNNEEDED:=--strip-unneeded
@@ -175,7 +183,7 @@ HOST_LDFLAGS += -L$(HOST_DIR)/lib -L$(HOST_DIR)/usr/lib -Wl,-rpath,$(HOST_DIR)/ HOST_PATH=$(HOST_DIR)/bin:$(HOST_DIR)/usr/bin:$(PATH) # hostcc version as an integer - E.G. 4.3.2 => 432 -HOSTCC_VERSION:=$(shell $(HOSTCC) --version | \ +HOSTCC_VERSION:=$(shell $(HOSTCC_NOCCACHE) --version | \ sed -n 's/^.* \([0-9]*\)\.\([0-9]*\)\.\([0-9]*\)[ ]*.*$$/\1\2\3/p') TARGET_CONFIGURE_OPTS=PATH=$(TARGET_PATH) \
package/ccache/Config.in new file +5 -0
@@ -0,0 +1,5 @@ +config BR2_PACKAGE_CCACHE + bool "ccache" + help + Compiler Cache + http://samba.org/ftp/ccache
package/ccache/ccache.mk new file +32 -0
@@ -0,0 +1,32 @@ +############################################################# +# +# ccache +# +############################################################# + +CCACHE_VERSION = 3.1.3 +CCACHE_SITE = http://samba.org/ftp/ccache +CCACHE_SOURCE = ccache-$(CCACHE_VERSION).tar.bz2 + +# When ccache is being built for the host, ccache is not yet +# available, so we have to use the special C compiler without the +# cache. +HOST_CCACHE_CONF_ENV = \ + CC="$(HOSTCC_NOCCACHE)" + +# We directly hardcode the cache location into the binary, as it is +# much easier to handle than passing an environment variable. +define HOST_CCACHE_FIX_CCACHE_DIR + sed -i 's,getenv("CCACHE_DIR"),"$(CCACHE_CACHE_DIR)",' $(@D)/ccache.c +endef + +HOST_CCACHE_POST_CONFIGURE_HOOKS += \ + HOST_CCACHE_FIX_CCACHE_DIR + +$(eval $(call AUTOTARGETS,package,ccache)) +$(eval $(call AUTOTARGETS,package,ccache,host)) + +ifeq ($(BR2_CCACHE),y) +ccache-stats: host-ccache + $(Q)$(CCACHE) -s +endif
toolchain/ccache/Config.in deleted +0 -16
@@ -1,16 +0,0 @@ -# - -comment "Ccache Options" - -config BR2_CCACHE - bool "Enable ccache support?" - help - Enable ccache support? - -config BR2_CCACHE_DIR - string "ccache dir location?" - depends on BR2_CCACHE - default "$(TOOLCHAIN_DIR)/ccache-$(CCACHE_VER)/cache" - help - Where ccache should store cached files. -
toolchain/ccache/Config.in.2 deleted +0 -13
@@ -1,13 +0,0 @@ -# - -config BR2_PACKAGE_CCACHE_TARGET - bool "ccache" - depends on BR2_PACKAGE_GCC_TARGET - help - ccache is a compiler cache. It speeds up recompilation of - C/C++ code by caching previous compiles and detecting when - the same compile is being done again. - - http://ccache.samba.org - -
toolchain/ccache/ccache.mk deleted +0 -196
196 lines not on this page · raw message
toolchain/dependencies/dependencies.sh +1 -1
@@ -79,7 +79,7 @@ if [ $MAKE_MAJOR -lt 3 ] || [ $MAKE_MAJOR -eq 3 -a $MAKE_MINOR -lt 81 ] ; then fi; # Check host gcc -COMPILER=$(which $HOSTCC 2> /dev/null) +COMPILER=$(which $HOSTCC_NOCCACHE 2> /dev/null) if [ -z "$COMPILER" ] ; then COMPILER=$(which cc 2> /dev/null) fi;
toolchain/toolchain-buildroot.mk +0 -1
@@ -2,7 +2,6 @@ include toolchain/dependencies/dependencies.mk include toolchain/binutils/binutils.mk -include toolchain/ccache/ccache.mk include toolchain/elf2flt/elf2flt.mk include toolchain/gcc/gcc-uclibc-4.x.mk include toolchain/gdb/gdb.mk
toolchain/toolchain-buildroot/Config.in +0 -1
@@ -5,5 +5,4 @@ source "toolchain/kernel-headers/Config.in" source "toolchain/uClibc/Config.in" source "toolchain/binutils/Config.in" source "toolchain/gcc/Config.in" -source "toolchain/ccache/Config.in" endif
toolchain/toolchain-crosstool-ng.mk +0 -1
@@ -3,7 +3,6 @@ # Explicit ordering: include toolchain/helpers.mk include toolchain/binutils/binutils.mk -include toolchain/ccache/ccache.mk include toolchain/dependencies/dependencies.mk include toolchain/elf2flt/elf2flt.mk include toolchain/gcc/gcc-uclibc-4.x.mk
toolchain/toolchain-external.mk +0 -1
@@ -2,7 +2,6 @@ include toolchain/helpers.mk include toolchain/binutils/binutils.mk -include toolchain/ccache/ccache.mk include toolchain/dependencies/dependencies.mk include toolchain/elf2flt/elf2flt.mk include toolchain/gcc/gcc-uclibc-4.x.mk
toolchain/toolchain-external/ext-tool.mk +1 -1
@@ -62,7 +62,7 @@ endif # ! no threads # could select a multilib variant as we want the "main" sysroot, which # contains all variants of the C library in the case of multilib # toolchains. -TARGET_CC_NO_SYSROOT=$(filter-out --sysroot=%,$(TARGET_CC)) +TARGET_CC_NO_SYSROOT=$(filter-out --sysroot=%,$(TARGET_CC_NOCCACHE)) SYSROOT_DIR=$(shell $(TARGET_CC_NO_SYSROOT) -print-sysroot 2>/dev/null) ifeq ($(SYSROOT_DIR),) SYSROOT_DIR=$(shell readlink -f $$(LANG=C $(TARGET_CC_NO_SYSROOT) -print-file-name=libc.a) |sed -r -e 's:usr/lib/libc\.a::;')
Signature · 1 line
1.7.0.4
PATCH 1/1 ccache: rework ccache management
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Hi,
Thomas> * ccache is now a normal package (both for the host and the target).
Besides the general comments, I have a few specific ones as well:
Thomas> +++ b/docs/buildroot.html
Thomas> @@ -681,6 +682,31 @@ endif
Thomas> Build options -> Toolchain and header file location</code> options.
Thomas> This could be useful if the toolchain must be shared with other users.</p>
Thomas> + <h2 id="ccache-support">Using <code>ccache</code> in Buildroot</h2>
Thomas> +
Thomas> + <p><a href="http://ccache.samba.org">ccache</a> is a compiler Thomas> + cache. It stores the object files resulting from each compilation Thomas> + process, and is able to skip future compilation of the same source Thomas> + file (with same compiler and same arguments) by using the Thomas> + pre-existing object files. When doing almost identical builds from Thomas> + scratch a number of times, it can divide the build time by a Thomas> + factor of two.</p>
The improvement depends quite a lot on the configuration / use case, so I think we should simpluy write something like 'it can significantly speed up the build process.' rather than specifically saying x2.
Thomas> +++ b/package/ccache/Config.in
Thomas> @@ -0,0 +1,5 @@
Thomas> +config BR2_PACKAGE_CCACHE
Thomas> + bool "ccache"
Thomas> + help
Thomas> + Compiler Cache
Thomas> + http://samba.org/ftp/ccache
Thomas> \ No newline at end of file
You forgot the missing newline here and for the .mk. We normally have an empty line before the website. The official website is afaik also
http://ccache.samba.org, rather than this download URL.
Doesn't ccache need zlib?
Signature · 1 line
Bye, Peter Korsgaard
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Thomas> Hello,
Thomas> [ Sorry, previous pull request had some mistakes. ]
Thomas> Here is a patch that reworks the ccache support to make it work and Thomas> make it useful. It is now a "Build options" rather than a "Toolchain Thomas> option". I have tested it with both internal toolchain and external Thomas> toolchain. The principle is simply to build ccache as early as Thomas> possible in the build process, and then use it to cache host and Thomas> target compiled files.
Thanks, looks interesting.
Thomas> The ccache cache is kept in ~/.buildroot-ccache/, so that it can be Thomas> shared between different builds.
Why here and not in the default ~/.ccache? Is the ~/.ccache directory content ccache-version dependent?
Thomas> I'm not using the symlink technique that was used in the past, but Thomas> rather prefixing all invocation to host gcc and target gcc with Thomas> "$(HOST_DIR)/usr/bin/ccache".
Is that working everywhere? I remember we had some problems back when we added --sysroot= to TARGET_CC. The qt package in particular is stripping the --sysroot argument because of this.
Signature · 1 line
Bye, Peter Korsgaard
Hi,
Quoted 1 line · Peter Korsgaard
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
[snip]
Quoted 7 lines
?Thomas> I'm not using the symlink technique that was used in the past, but ?Thomas> rather prefixing all invocation to host gcc and target gcc with ?Thomas> "$(HOST_DIR)/usr/bin/ccache".
Is that working everywhere? I remember we had some problems back when we added --sysroot= to
TARGET_CC. The qt package in particular is stripping the --sysroot argument because of this.
The cdrkit package also strips off --sysroot because its build system CMake doesn't like it when --sysroot is part of the compiler path.
Actually, I am currently working on a CMAKETARGETS macro (similar to AUTOTARGETS) and decided to write a patch that takes --sysroot out of TARGET_CC and move it to the {C,CXX}FLAGS variables, to make CMake happy. I've been meaning to post the patch series but I don't feel it's ready to be merged yet. How do the Buildroot developers feel about
a) moving --sysroot from TARGET_CC to {C,CXX}FLAGS
b) add CMAKETARGETS infrastructure
If this sounds interesting I can post the patches for review.
Best regards,
Bj?rn Forsman
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Bj?rn" == Bj?rn Forsman <[email protected]> writes:
Hi,
Bj?rn> Actually, I am currently working on a CMAKETARGETS macro (similar to Bj?rn> AUTOTARGETS) and decided to write a patch that takes --sysroot out of Bj?rn> TARGET_CC and move it to the {C,CXX}FLAGS variables, to make CMake Bj?rn> happy. I've been meaning to post the patch series but I don't feel Bj?rn> it's ready to be merged yet. How do the Buildroot developers feel Bj?rn> about
Bj?rn> a) moving --sysroot from TARGET_CC to {C,CXX}FLAGS
It afaik got moved in the first place to fix issues with external toolchains (E.G. where sysroot is important) and various configure scripts not using CFLAGS, so I don't think this will work.
Perhaps we should simply bite the bullet and add simple shell wrappers in HOST_DIR / STAGING_DIR for gcc/g++/ld which add --sysroot when we're using an external toolchain, and then get rid of --sysroot in TARGET_CC.
Bj?rn> b) add CMAKETARGETS infrastructure
Sounds good!
Bj?rn> If this sounds interesting I can post the patches for review.
Yes, please.
Signature · 1 line
Bye, Peter Korsgaard
Quoted 3 lines · On Mon, 06 Dec 2010 16:08:17 +0100 Peter Korsgaard <[email protected]> wrote:
It afaik got moved in the first place to fix issues with external toolchains (E.G. where sysroot is important) and various configure scripts not using CFLAGS, so I don't think this will work.
Yes.
Quoted 4 lines
Perhaps we should simply bite the bullet and add simple shell wrappers in
HOST_DIR/STAGING_DIRfor gcc/g++/ld which add --sysroot when we're using an external toolchain, and then get rid of --sysroot inTARGET_CC.
Maybe. But for the moment, I think doing the little mungling done in cdrkit.mk that fixes all CMake-based packages is quite reasonable.
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Hello,
Quoted 2 lines · On Mon, 6 Dec 2010 10:44:40 +0100 Bj?rn Forsman <[email protected]> wrote:
The cdrkit package also strips off --sysroot because its build system CMake doesn't like it when --sysroot is part of the compiler path.
Yes.
Quoted 8 lines
Actually, I am currently working on a CMAKETARGETS macro (similar to AUTOTARGETS) and decided to write a patch that takes --sysroot out of
TARGET_CCand move it to the {C,CXX}FLAGS variables, to make CMake happy. I've been meaning to post the patch series but I don't feel it's ready to be merged yet. How do the Buildroot developers feel abouta) moving --sysroot from
TARGET_CCto {C,CXX}FLAGS
No, please don't do that :-)
There are other packages that need this. Seeefb1d8d3f40281645c178c150d992601c8042c1a.
commit efb1d8d3f40281645c178c150d992601c8042c1a
Author: Thomas Petazzoni <[email protected]> Date: Fri Jun 25 15:04:08 2010 +0200
Cleanup TARGET_CONFIGURE_OPTS
The definition of CC, LD, GCC, CPP, CXX and FC shouldn't contain the CFLAGS/LDFLAGS/CXXFLAGS, those should be passed through the appropriate variables.
However, the --sysroot option is a particular case here: it needs to be part of the CC/LD/GCC/etc. definitions otherwise libtool strips it from the CFLAGS/LDFLAGS.
The current choice has been fairly hard to get to, we made a number of different tries before finding a solution that seems to work relatively fine for most packages. Therefore, I'd prefer to keep the TARGET_CC/TARGET_CFLAGS as they are today, even if the CMAKETARGETS infrastructure needs to do the same kind of mungling as cdrkit.mk is doing.
If you really want to be convinced that having --sysroot in TARGET_CFLAGS causes problems, I can probably trigger a particular configuration to exhibit the issue.
Quoted 1 line
b) add CMAKETARGETS infrastructure
Yes, that would be great. We're likely to see more CMake-based packages in the future.
Note that some people in the Armadeus project are working on a different thing related to CMake and Buildroot: generate automatically a CMake toolchain description file, to make it easy to build CMake-based applications outside of Buildroot, using the toolchain and staging directory of Buildroot.
Regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Hi Thomas,
2010/12/6 Thomas Petazzoni <[email protected]>:
Quoted 41 lines
Hello,
Quoted 8 lines · On Mon, 6 Dec 2010 10:44:40 +0100 Bj?rn Forsman <[email protected]> wrote:
Actually, I am currently working on a CMAKETARGETS macro (similar to AUTOTARGETS) and decided to write a patch that takes --sysroot out of
TARGET_CCand move it to the {C,CXX}FLAGS variables, to make CMake happy. I've been meaning to post the patch series but I don't feel it's ready to be merged yet. How do the Buildroot developers feel abouta) moving --sysroot from
TARGET_CCto {C,CXX}FLAGSNo, please don't do that :-)
There are other packages that need this. See
efb1d8d3f40281645c178c150d992601c8042c1a.commit
efb1d8d3f40281645c178c150d992601c8042c1a
Author: Thomas Petazzoni <[email protected]>
Date: ? Fri Jun 25 15:04:08 2010 +0200? ?Cleanup
TARGET_CONFIGURE_OPTS? ?The definition of CC, LD, GCC, CPP, CXX and FC shouldn't contain the ? ?CFLAGS/LDFLAGS/CXXFLAGS, those should be passed through the ? ?appropriate variables.
? ?However, the --sysroot option is a particular case here: it needs to ? ?be part of the CC/LD/GCC/etc. definitions otherwise libtool strips it ? ?from the CFLAGS/LDFLAGS.
? ?Signed-off-by: Thomas Petazzoni <[email protected]>
The current choice has been fairly hard to get to, we made a number of different tries before finding a solution that seems to work relatively fine for most packages. Therefore, I'd prefer to keep the TARGET_CC/TARGET_CFLAGS as they are today, even if the CMAKETARGETS infrastructure needs to do the same kind of mungling as cdrkit.mk is doing.
Quoted 3 lines
If you really want to be convinced that having --sysroot in
TARGET_CFLAGScauses problems, I can probably trigger a particular configuration to exhibit the issue.
Are many packages affected? If many, I'll just take your word for it and leave TARGET_CC and xFLAGS variables be, but if there is only a few packages affected, maybe we can fix them instead?
Quoted 10 lines
Quoted 1 line
b) add CMAKETARGETS infrastructure
Yes, that would be great. We're likely to see more CMake-based packages in the future.
Note that some people in the Armadeus project are working on a different thing related to CMake and Buildroot: generate automatically a CMake toolchain description file, to make it easy to build CMake-based applications outside of Buildroot, using the toolchain and staging directory of Buildroot.
Yes, I'd like to get this CMake toolchain file feature into Buildroot as well. How about appending a toolchain file target to $(2)_DEPENDENCIES in Makefile.cmake.in? And place the toolchain file in output/?
Best regards,
Bj?rn Forsman
Hello,
Quoted 1 line · On Tue, 7 Dec 2010 01:04:41 +0100 Bj?rn Forsman <[email protected]> wrote:
Are many packages affected?
I don't remember.
Quoted 3 lines
If many, I'll just take your word for it and leave
TARGET_CCand xFLAGS variables be, but if there is only a few packages affected, maybe we can fix them instead?
The problem was related to libtool. And unfortunately "fix problem" and "libtool" in the same sentence is fairly hard to have :-)
Honestly, I've spent quite a bit of time at this moment trying to find the solution for the --sysroot option that would work with most packages, without requiring to do horrible things. As things are working today, I'm not really willing to spend more time on this, but if others want, I of course won't prevent them from doing so. I'll just ask for some heavy testing to be done before accepting changes on this :-)
Quoted 3 lines
Yes, I'd like to get this CMake toolchain file feature into Buildroot as well. How about appending a toolchain file target to $(2)_DEPENDENCIES in Makefile.cmake.in?
Hum, why ? Do you want to use the toolchain file even to build the CMake-based packages in Buildroot ? Or just provide it for users to build external softwares outside of Buildroot ?
Quoted 1 line
And place the toolchain file in output/?
As it should be part of the SDK, it should probably be located in $(HOST_DIR) somewhere. But I'm not really clear yet on what the organization of the SDK should be. In the mean time, just assume you can put it in $(O), moving it to some other place won't be hard.
Thanks!
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Hi Thomas,
2010/12/7 Thomas Petazzoni <[email protected]>:
Quoted 16 lines
Hello,
Quoted 1 line · On Tue, 7 Dec 2010 01:04:41 +0100 Bj?rn Forsman <[email protected]> wrote:
Are many packages affected?
I don't remember.
Quoted 3 lines
If many, I'll just take your word for it and leave
TARGET_CCand xFLAGS variables be, but if there is only a few packages affected, maybe we can fix them instead?The problem was related to libtool. And unfortunately "fix problem" and "libtool" in the same sentence is fairly hard to have :-)
Quoted 7 lines
Honestly, I've spent quite a bit of time at this moment trying to find the solution for the --sysroot option that would work with most packages, without requiring to do horrible things. As things are working today, I'm not really willing to spend more time on this, but if others want, I of course won't prevent them from doing so. I'll just ask for some heavy testing to be done before accepting changes on this :-)
Ok. I'll just leave --sysroot the way it is.
Quoted 7 lines
Quoted 3 lines
Yes, I'd like to get this CMake toolchain file feature into Buildroot as well. How about appending a toolchain file target to $(2)_DEPENDENCIES in Makefile.cmake.in?
Hum, why ? Do you want to use the toolchain file even to build the CMake-based packages in Buildroot ? Or just provide it for users to build external softwares outside of Buildroot ?
I haven't really decided whether to use the toolchain file internally or not. If using it internally reduces code duplication, then maybe yes :-) And adding the target to $(2)_DEPENDENCIES in Makefile.cmake.in was just the first place that I thought of.
I guess the toolchain file should be generated earlier, and be generated even if the user doesn't select any CMake packages inside Buildroot, but I don't know where the right 'hook' in Buildroot would be for that. Maybe when the .config file is written?
Quoted 6 lines
Quoted 1 line
And place the toolchain file in output/?
As it should be part of the SDK, it should probably be located in $(HOST_DIR) somewhere. But I'm not really clear yet on what the organization of the SDK should be. In the mean time, just assume you can put it in $(O), moving it to some other place won't be hard.
Ok. So until the SDK is ready I'll just put the toolchain file in $(O) and don't use it internally so it will be easy to change later.
I'll post the patch series once I've rebased.
Best regards,
Bj?rn Forsman
Hello,
Quoted 5 lines · On Sun, 05 Dec 2010 23:40:14 +0100 Peter Korsgaard <[email protected]> wrote:
Thomas> The ccache cache is kept in ~/.buildroot-ccache/, so that it can be Thomas> shared between different builds.
Why here and not in the default ~/.ccache? Is the ~/.ccache directory content ccache-version dependent?
I don't know how ccache-version dependent are the contents of the cache directory. In the previous ccache integration, the cache was inside Buildroot build directory (which I thought was stupid since you then couldn't share the cache between different Buildroot builds), but I kept the idea of having a Buildroot-specific location for the cache.
I don't have strong opinion/arguments on that, so if you say we should use the default cache location, I'll just do it.
Quoted 1 line
Is that working everywhere?
Everywhere I don't know, I obviously haven't compiled all our packages with ccache support enabled.
Quoted 3 lines
I remember we had some problems back when we added --sysroot= to
TARGET_CC. The qt package in particular is stripping the --sysroot argument because of this.
I just tried Qt, and it built fine. It does not use ccache for the parts compiled for the host (since we don't tell Qt about $(HOSTCXX)), but it definitely uses the cache for parts compiled for the target.
Here are the results for a Busybox + Qt build, with a CodeSourcery glibc ARM external toolchain.
Cold cache
==========
real 7m41.319s user 37m53.620s sys 1m31.660s
Hot cache
=========
real 3m4.738s user 5m34.480s sys 0m36.160s
And in the hot cache case, a quite significant time is spent rebuilding the host tools, as ccache is not used there. So we could probably speed this up a bit further.
I am not strongly advocating the usage of the "ccache /path/to/gcc" solution compared to the symbolic link solutions, but if the first solution works, I find it better because: 1) it's nicer and 2) it's easier to implement with external toolchains.
Regards,
Thomas
Signature · 4 lines
Thomas Petazzoni, Free Electrons Kernel, drivers, real-time and embedded Linux development, consulting, training and support. http://free-electrons.com
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Thomas" == Thomas Petazzoni <[email protected]> writes:
Hi,
Quoted 2 lines
Quoted 2 lines
Why here and not in the default ~/.ccache? Is the ~/.ccache directory content ccache-version dependent?
Thomas> I don't know how ccache-version dependent are the contents of Thomas> the cache directory. In the previous ccache integration, the Thomas> cache was inside Buildroot build directory (which I thought was Thomas> stupid since you then couldn't share the cache between Thomas> different Buildroot builds), but I kept the idea of having a Thomas> Buildroot-specific location for the cache.
Thomas> I don't have strong opinion/arguments on that, so if you say we Thomas> should use the default cache location, I'll just do it.
Ok, me neither - Just keep it where it is. It just wasn't clear to me from the commit message if there was any deeper reason behind it.
Quoted 3 lines
Quoted 3 lines
I remember we had some problems back when we added --sysroot= to
TARGET_CC. The qt package in particular is stripping the --sysroot argument because of this.
Thomas> I just tried Qt, and it built fine. It does not use ccache for the Thomas> parts compiled for the host (since we don't tell Qt about $(HOSTCXX)), Thomas> but it definitely uses the cache for parts compiled for the target.
Ok, thanks.
Thomas> Here are the results for a Busybox + Qt build, with a Thomas> CodeSourcery glibc ARM external toolchain.
Thomas> Cold cache
Thomas> ==========
Thomas> real 7m41.319s Thomas> user 37m53.620s Thomas> sys 1m31.660s
Thomas> Hot cache
Thomas> =========
Thomas> real 3m4.738s Thomas> user 5m34.480s Thomas> sys 0m36.160s
Nice!
Thomas> I am not strongly advocating the usage of the "ccache /path/to/gcc" Thomas> solution compared to the symbolic link solutions, but if the first Thomas> solution works, I find it better because: 1) it's nicer and 2) it's Thomas> easier to implement with external toolchains.
Ok, fine by me if it works.
Care to send a new pull request with the few issues I pointed out fixed, then I'll commit it?
Signature · 1 line
Bye, Peter Korsgaard