Don't build host-cmake if it is available on the build host

PATCH New Luca Ceresoli 2 participants 5 messages Feb 1, 2016 – Feb 5, 2016

Share & tools

Luca Ceresoli

author

2016-02-01 21:44 UTC #0 lore raw

PATCH 1/1 Don't build host-cmake if it is available on the build host

Currently all cmake packages depend on host-cmake. Unfortunately host-cmake takes a long time to configure and build: almost 7 minutes on a dual-core i5 with SSD. The time does not change even with ccache enabled.

Indeed, building host-ccache is not very useful if it is already installed on the build host: it is supposed to be quite portable, and the only patch we have for cmake seems to only affect target-cmake.

We avoid building host-cmake if cmake is already available on the host using a technique similar to the one used for host-tar and host-xzcat.

First, we leverage the existing infrastructure in support/dependencies/dependencies.mk to set CMAKE to either "cmake" or "$(HOST_DIR)/usr/bin/cmake". In the latter case we also set BUILD_HOST_CMAKE = YES meaning we need to build host-cmake.

Then in pkg-cmake.mk we launch $(CMAKE) instead of $(HOST_DIR)/usr/bin/cmake.

Finally, we do skip adding the dependency on host-cmake for all cmake packages when $(BUILD_HOST_CMAKE) != YES.

Unlike what we do for host-tar and host-xzcat, for host-cmake we do not add host-cmake to DEPENDENCIES_HOST_PREREQ. Otherwise host-cmake would be a dependency for _any_ package when it's not installed on the host, even if no cmake package is selected.

Signed-off-by Luca Ceresoli <[email protected]>

Cc Samuel Martin <[email protected]>

Cc Davide Viti <[email protected]>

---
 package/pkg-cmake.mk                     |  6 ++++--
 support/dependencies/check-host-cmake.mk |  6 ++++++
 support/dependencies/check-host-cmake.sh | 14 ++++++++++++++
 3 files changed, 24 insertions(+), 2 deletions(-)
 create mode 100644 support/dependencies/check-host-cmake.mk
 create mode 100755 support/dependencies/check-host-cmake.sh

package/pkg-cmake.mk +4 -2

index 81dcfcc..e3bd603 100644
@@ -85,7 +85,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$($$(PKG)_CONF_ENV) $$(CMAKE) $$($$(PKG)_SRCDIR) \
 		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
 		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),Debug,Release) \
 		-DCMAKE_INSTALL_PREFIX="/usr" \

@@ -110,7 +110,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	$$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$(CMAKE) $$($$(PKG)_SRCDIR) \
 		-DCMAKE_INSTALL_SO_NO_EXE=0 \
 		-DCMAKE_FIND_ROOT_PATH="$$(HOST_DIR)" \
 		-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM="BOTH" \

@@ -149,7 +149,9 @@ $(2)_DEPENDENCIES ?= $$(filter-out host-skeleton host-toolchain $(1),\
 	$$(patsubst host-host-%,host-%,$$(addprefix host-,$$($(3)_DEPENDENCIES))))
 endif
 
+ifeq ($$(BUILD_HOST_CMAKE),YES)
 $(2)_DEPENDENCIES += host-cmake
+endif
 
 #
 # Build step. Only define it if not already defined by the package .mk

support/dependencies/check-host-cmake.mk new file +6 -0

new file mode 100644
index 0000000..fe16322
@@ -0,0 +1,6 @@
+CMAKE ?= cmake
+
+ifeq (,$(call suitable-host-package,cmake,$(CMAKE)))
+BUILD_HOST_CMAKE = YES
+CMAKE = $(HOST_DIR)/usr/bin/cmake
+endif

support/dependencies/check-host-cmake.sh new file +14 -0

new file mode 100755
index 0000000..76a81e0
@@ -0,0 +1,14 @@
+#!/bin/sh
+
+candidate="$1"
+
+cmake=`which $candidate`
+if [ ! -x "$cmake" ]; then
+	cmake=`which cmake`
+	if [ ! -x "$cmake" ]; then
+		# echo nothing: no suitable cmake found
+		exit 1
+	fi
+fi
+
+echo $cmake

Signature · 1 line

1.9.1

Arnout Vandecappelle

2016-02-02 08:44 UTC #1 lore raw

PATCH 1/1 Don't build host-cmake if it is available on the build host

Quoted 6 lines · Luca Ceresoli

Currently all cmake packages depend on host-cmake. Unfortunately host-cmake takes a long time to configure and build: almost 7 minutes on a dual-core i5 with SSD. The time does not change even with ccache enabled.
Indeed, building host-ccache is not very useful if it is already

host-ccache -> host-cmake

Quoted 14 lines

installed on the build host: it is supposed to be quite portable, and the only patch we have for cmake seems to only affect target-cmake.
We avoid building host-cmake if cmake is already available on the host using a technique similar to the one used for host-tar and host-xzcat.
First, we leverage the existing infrastructure in support/dependencies/dependencies.mk to set CMAKE to either "cmake" or "$(HOST_DIR)/usr/bin/cmake". In the latter case we also set BUILD_HOST_CMAKE = YES meaning we need to build host-cmake.
Then in pkg-cmake.mk we launch $(CMAKE) instead of $(HOST_DIR)/usr/bin/cmake.

Quoted 7 lines

Finally, we do skip adding the dependency on host-cmake for all cmake packages when $(BUILD_HOST_CMAKE) != YES.
Unlike what we do for host-tar and host-xzcat, for host-cmake we do not add host-cmake to DEPENDENCIES_HOST_PREREQ. Otherwise host-cmake would be a dependency for _any_ package when it's not installed on the host, even if no cmake package is selected.

Nice and long commit message, thanks!

Quoted 62 lines

Signed-off-by Luca Ceresoli <[email protected]>

Cc Samuel Martin <[email protected]>

Cc Davide Viti <[email protected]>

---
 package/pkg-cmake.mk                     |  6 ++++--
 support/dependencies/check-host-cmake.mk |  6 ++++++
 support/dependencies/check-host-cmake.sh | 14 ++++++++++++++
 3 files changed, 24 insertions(+), 2 deletions(-)
 create mode 100644 support/dependencies/check-host-cmake.mk
 create mode 100755 support/dependencies/check-host-cmake.sh

package/pkg-cmake.mk +4 -2

index 81dcfcc..e3bd603 100644
@@ -85,7 +85,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$($$(PKG)_CONF_ENV) $$(CMAKE) $$($$(PKG)_SRCDIR) \
 		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
 		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),Debug,Release) \
 		-DCMAKE_INSTALL_PREFIX="/usr" \

@@ -110,7 +110,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	$$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$(CMAKE) $$($$(PKG)_SRCDIR) \
 		-DCMAKE_INSTALL_SO_NO_EXE=0 \
 		-DCMAKE_FIND_ROOT_PATH="$$(HOST_DIR)" \
 		-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM="BOTH" \

@@ -149,7 +149,9 @@ $(2)_DEPENDENCIES ?= $$(filter-out host-skeleton host-toolchain $(1),\
 	$$(patsubst host-host-%,host-%,$$(addprefix host-,$$($(3)_DEPENDENCIES))))
 endif
 
+ifeq ($$(BUILD_HOST_CMAKE),YES)
 $(2)_DEPENDENCIES += host-cmake
+endif
 
 #
 # Build step. Only define it if not already defined by the package .mk

support/dependencies/check-host-cmake.mk new file +6 -0

new file mode 100644
index 0000000..fe16322
@@ -0,0 +1,6 @@
+CMAKE ?= cmake
+
+ifeq (,$(call suitable-host-package,cmake,$(CMAKE)))
+BUILD_HOST_CMAKE = YES
+CMAKE = $(HOST_DIR)/usr/bin/cmake
+endif

support/dependencies/check-host-cmake.sh new file +0 -0

new file mode 100755
index 0000000..76a81e0

support/dependencies/check-host-cmake.sh line 5

Quoted 5 lines

support/dependencies/check-host-cmake.sh new file +5 -0
@@ -0,0 +1,14 @@
+#!/bin/sh
+
+candidate="$1"
+
+cmake=`which $candidate`

$candidate could be an absolute path, in which case which returns nothing.

support/dependencies/check-host-cmake.sh line 7

Quoted 2 lines

support/dependencies/check-host-cmake.sh new file +2 -0
+if [ ! -x "$cmake" ]; then
+	cmake=`which cmake`

Since there is already CMAKE ?= cmake in the caller, this is pointless.

How about a

for cmake in "$cmake" "$( which "$cmake" )"; do
...

instead?

support/dependencies/check-host-cmake.sh line 12

Quoted 5 lines

support/dependencies/check-host-cmake.sh new file +5 -0
+	if [ ! -x "$cmake" ]; then
+		# echo nothing: no suitable cmake found
+		exit 1
+	fi
+fi

I have the feeling that there should also be a version check here, though I
don't know which version we should use.

Regards,
Arnout

Quoted 3 lines

support/dependencies/check-host-cmake.sh new file +2 -0
+
+echo $cmake

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

Luca Ceresoli

author

2016-02-02 09:01 UTC #2 lore raw

PATCH 1/1 Don't build host-cmake if it is available on the build host

Hi Arnout,

thanks for the review.

Unfortunately this patch is not working, at least for qjson with qt4.

qjson fails with:

CMake Error in src/CMakeLists.txt:
Imported target "Qt4::QtCore" includes non-existent path

"QT_MKSPECS_DIR-NOTFOUND/default"

in its INTERFACE_INCLUDE_DIRECTORIES. Possible reasons include:

* The path was deleted, renamed, or moved to another location.

* An install or uninstall procedure did not complete successfully.

* The installation package was faulty and references files it does not provide.

I am investigating right now.

Below my replies to your comments.

Quoted 9 lines · Arnout Vandecappelle

Quoted 6 lines · Luca Ceresoli
Currently all cmake packages depend on host-cmake. Unfortunately host-cmake takes a long time to configure and build: almost 7 minutes on a dual-core i5 with SSD. The time does not change even with ccache enabled.
Indeed, building host-ccache is not very useful if it is already
host-ccache -> host-cmake

Oops. :)

Quoted 100 lines

Quoted 15 lines
installed on the build host: it is supposed to be quite portable, and the only patch we have for cmake seems to only affect target-cmake.
We avoid building host-cmake if cmake is already available on the host using a technique similar to the one used for host-tar and host-xzcat.
First, we leverage the existing infrastructure in support/dependencies/dependencies.mk to set CMAKE to either "cmake" or "$(HOST_DIR)/usr/bin/cmake". In the latter case we also set BUILD_HOST_CMAKE = YES meaning we need to build host-cmake.
Then in pkg-cmake.mk we launch $(CMAKE) instead of $(HOST_DIR)/usr/bin/cmake.
Quoted 7 lines
Finally, we do skip adding the dependency on host-cmake for all cmake packages when $(BUILD_HOST_CMAKE) != YES.
Unlike what we do for host-tar and host-xzcat, for host-cmake we do not add host-cmake to DEPENDENCIES_HOST_PREREQ. Otherwise host-cmake would be a dependency for _any_ package when it's not installed on the host, even if no cmake package is selected.
Nice and long commit message, thanks!
Quoted 62 lines
Signed-off-by Luca Ceresoli <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Davide Viti <[email protected]>
---
  package/pkg-cmake.mk                     |  6 ++++--
  support/dependencies/check-host-cmake.mk |  6 ++++++
  support/dependencies/check-host-cmake.sh | 14 ++++++++++++++
  3 files changed, 24 insertions(+), 2 deletions(-)
create mode 100644 support/dependencies/check-host-cmake.mk create mode 100755 support/dependencies/check-host-cmake.sh
package/pkg-cmake.mk +4 -2
index 81dcfcc..e3bd603 100644
@@ -85,7 +85,7 @@ define $(2)_CONFIGURE_CMDS
  	cd $$($$(PKG)_BUILDDIR) && \
  	rm -f CMakeCache.txt && \
  	PATH=$$(BR_PATH) \
-	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$($$(PKG)_CONF_ENV) $$(CMAKE) $$($$(PKG)_SRCDIR) \
  		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
  		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),Debug,Release) \
  		-DCMAKE_INSTALL_PREFIX="/usr" \
@@ -110,7 +110,7 @@ define $(2)_CONFIGURE_CMDS
  	cd $$($$(PKG)_BUILDDIR) && \
  	rm -f CMakeCache.txt && \
  	PATH=$$(BR_PATH) \
-	$$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$(CMAKE) $$($$(PKG)_SRCDIR) \
  		-DCMAKE_INSTALL_SO_NO_EXE=0 \
  		-DCMAKE_FIND_ROOT_PATH="$$(HOST_DIR)" \
  		-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM="BOTH" \
@@ -149,7 +149,9 @@ $(2)_DEPENDENCIES ?= $$(filter-out host-skeleton host-toolchain $(1),\
  	$$(patsubst host-host-%,host-%,$$(addprefix host-,$$($(3)_DEPENDENCIES))))
  endif
 
+ifeq ($$(BUILD_HOST_CMAKE),YES)
  $(2)_DEPENDENCIES += host-cmake
+endif
 
  #
  # Build step. Only define it if not already defined by the package .mk
support/dependencies/check-host-cmake.mk new file +6 -0
new file mode 100644
index 0000000..fe16322
@@ -0,0 +1,6 @@
+CMAKE ?= cmake
+
+ifeq (,$(call suitable-host-package,cmake,$(CMAKE)))
+BUILD_HOST_CMAKE = YES
+CMAKE = $(HOST_DIR)/usr/bin/cmake
+endif
support/dependencies/check-host-cmake.sh new file +0 -0
new file mode 100755
index 0000000..76a81e0
support/dependencies/check-host-cmake.sh line 5
Quoted 5 lines
support/dependencies/check-host-cmake.sh new file +5 -0
@@ -0,0 +1,14 @@
+#!/bin/sh
+
+candidate="$1"
+
+cmake=`which $candidate`
$candidate could be an absolute path, in which case which returns nothing.
support/dependencies/check-host-cmake.sh line 7
Quoted 2 lines
support/dependencies/check-host-cmake.sh new file +2 -0
+if [ ! -x "$cmake" ]; then
+	cmake=`which cmake`
Since there is already CMAKE ?= cmake in the caller, this is pointless.

Quoted 6 lines

How about a
for cmake in "$cmake" "$( which "$cmake" )"; do
...
instead?

Your suggestions are absolutely correct. I shamelessly copied other check-host-*.sh scripts and just concentrated on having the thing working. Seems like thos other script need a cleanup now. I'll have a look (if/after I can make this thing working).

Quoted 9 lines

support/dependencies/check-host-cmake.sh line 12
Quoted 6 lines
support/dependencies/check-host-cmake.sh new file +5 -0
+	if [ ! -x "$cmake" ]; then
+		# echo nothing: no suitable cmake found
+		exit 1
+	fi
+fi
I have the feeling that there should also be a version check here, though I
don't know which version we should use.

Right. My idea is to download all the cmake packages that are currently in Buildroot, find the highest version mentioned in their CMAKE_MINIMUM_REQUIRED() statement and check for that version in the script.

Signature · 1 line

Luca

Luca Ceresoli

author

2016-02-04 13:37 UTC #3 lore raw

PATCH 1/1 Don't build host-cmake if it is available on the build host

Hi Arnout,

Quoted 5 lines · Luca Ceresoli

Hi Arnout,
thanks for the review.
Unfortunately this patch is not working, at least for qjson with qt4.

It's fixed now (thanks to Samuel Martin for the help). So it's now time for the refinements.

[...]

Quoted 18 lines

Quoted 18 lines
Quoted 18 lines
new file mode 100644
index 0000000..fe16322
support/dependencies/check-host-cmake.mk new file +6 -0
@@ -0,0 +1,6 @@
+CMAKE ?= cmake
+
+ifeq (,$(call suitable-host-package,cmake,$(CMAKE)))
+BUILD_HOST_CMAKE = YES
+CMAKE = $(HOST_DIR)/usr/bin/cmake
+endif
support/dependencies/check-host-cmake.sh new file +0 -0
new file mode 100755
index 0000000..76a81e0
@@ -0,0 +1,14 @@

Quoted 8 lines

Quoted 8 lines
support/dependencies/check-host-cmake.sh line 5
Quoted 5 lines
support/dependencies/check-host-cmake.sh new file +5 -0
+#!/bin/sh
+
+candidate="$1"
+
+cmake=`which $candidate`
$candidate could be an absolute path, in which case which returns
nothing.

Not sure I got what you mean, sorry. which + absolute path returns the absolute path itself, if it exists:

$ which cmake
/usr/bin/cmake
$ which /usr/bin/cmake
/usr/bin/cmake
$ which /not/quite/cmake
$

And the which(1) manpage confirms this is correct.

Quoted 4 lines

Quoted 4 lines
support/dependencies/check-host-cmake.sh line 7
Quoted 2 lines
support/dependencies/check-host-cmake.sh new file +2 -0
+if [ ! -x "$cmake" ]; then
+    cmake=`which cmake`
Since there is already CMAKE ?= cmake in the caller, this is pointless.

Aah, yes. Removing the second if will still allow to override CMAKE ('make CMAKE=/my/custom/cmake qjson') but if it does not exist it will not search for one in the path. Sounds good.

Quoted 6 lines

Quoted 6 lines
How about a
for cmake in "$cmake" "$( which "$cmake" )"; do
...
instead?

Given the above discussion about which(1), I think this is useless.

Quoted 6 lines

Quoted 6 lines
Quoted 5 lines
support/dependencies/check-host-cmake.sh new file +5 -0
+    if [ ! -x "$cmake" ]; then
+        # echo nothing: no suitable cmake found
+        exit 1
+    fi
+fi

Quoted 8 lines

Quoted 3 lines
I have the feeling that there should also be a version check here,
though I
don't know which version we should use.
Right. My idea is to download all the cmake packages that are currently in Buildroot, find the highest version mentioned in their CMAKE_MINIMUM_REQUIRED() statement and check for that version in the script.

I added it, will be in v2. This was absolutely needed to make it work on a Ubuntu 14.04 LTS host. More details will be in the commit log (which will be even longer, ooh yeah).

Signature · 1 line

Luca

Arnout Vandecappelle

2016-02-05 21:47 UTC #4 lore raw

PATCH 1/1 Don't build host-cmake if it is available on the build host

Quoted 12 lines · Luca Ceresoli

Hi Arnout,
Quoted 4 lines · Luca Ceresoli
Quoted 4 lines
Quoted 1 line
+cmake=which $candidate
$candidate could be an absolute path, in which case which returns
nothing.
Not sure I got what you mean, sorry. which + absolute path returns the absolute path itself, if it exists:

Quoted 9 lines

$ which cmake
/usr/bin/cmake
$ which /usr/bin/cmake
/usr/bin/cmake
$ which /not/quite/cmake
$
And the which(1) manpage confirms this is correct.

Oops, my bad, I tested it with a file that wasn't executable...

Regards,
Arnout

[snip]

Signature · 6 lines

Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF