Don't build host-cmake if it is available on the build host

PATCH Superseded Luca Ceresoli 4 participants 4 messages Jul 16, 2016 – Sep 5, 2016

Share & tools

Luca Ceresoli

author

2016-07-16 20:56 UTC #0 lore raw

PATCH v4 1/1 Don't build host-cmake if it is available on the build host

Currently all cmake packages depend on host-cmake. Unfortunately host-cmake takes a long time to configure and build: almost 7 minutes on a dual-core i5 with SSD. The time does not change even with ccache enabled.

Indeed, building host-cmake is avoidable if it is already installed on the build host: CMake is supposed to be quite portable, and the only patch in Buildroot for the CMake package seems to only affect target-cmake.

Thus we automatically skip building host-cmake and use the one on the system if:
- cmake is available on the system and
- it is recent enough.

First, we leverage the existing infrastructure in support/dependencies/dependencies.mk to find out whether there's a suitable cmake executable on the system. Its path can be passed in the BR2_CMAKE environment variable, otherwise it defaults to "cmake". If it is enabled, found and suitable then we set USE_SYSTEM_CMAKE. Otherwise we override BR2_CMAKE with "$(HOST_DIR)/usr/bin/cmake" to revert to the old behaviour.

Then in pkg-cmake.mk we launch $(BR2_CMAKE) instead of $(HOST_DIR)/usr/bin/cmake.

Finally, we skip adding the dependency on host-cmake for all cmake packages when $(USE_SYSTEM_CMAKE) = YES.

Unlike what we do for host-tar and host-xzcat, for host-cmake we do not add host-cmake to DEPENDENCIES_HOST_PREREQ. If we did, host-cmake would be a dependency for _any_ package when it's not installed on the host, even when no cmake package is selected.

check-host-cmake.sh requires CMake to be at least 3.0 to consider it suitable. This is because older versions are affected by the bug described and fixed in Buildroot in ef2c1970e4bf ("cmake: add patch to fix Qt mkspecs detection"). The bug was fixed in upstream CMake in version 3.0 [0].

Besides, among all the cmake packages currently in Buildroot, the highest version mentioned in cmake_minimum_required() is 3.0 (the grantlee package). Thus 3.0 should be enough to build all current packages. Of course, with the addition or bump of packages, the minimum required version will raise.

Tested on:
- Ubuntu 14.04 without CMake, with official CMake (2.8), PPA CMake (3.2)
- Ubuntu 15.10 without CMake, with official CMake (3.2)
- Ubuntu 16.04 without CMake, with official CMake (3.5)

[0] https://cmake.org/gitweb?p=cmake.git;h=e8b8b37ef6fef094940d3384df5a1d421b9fa568

Signed-off-by Luca Ceresoli <[email protected]>

Cc Samuel Martin <[email protected]>

Cc Davide Viti <[email protected]>

Cc Arnout Vandecappelle <[email protected]>

Cc Thomas Petazzoni <[email protected]>

---

Results:

On my build server, this patch reduced the build time for a batch of 16 Buildroot configurations (all building cmake-based packages) from 2h44m to 1h54m. Woah, it's a 30% saving!

Changes v3 -> v4:
- rename USE_SYSTEM_HOST_CMAKE -> USE_SYSTEM_CMAKE (Arnout)
- ditch BR2_TRY_SYSTEM_CMAKE and use system-cmake unconditionally if it is found and >= 3.0 (Arnout)
- rename BR2_HOST_CMAKE -> BR2_CMAKE since it can be either a host-cmake (built by Buildroot) or a system-cmake

Changes v2 -> v3:
- make this feature optional via the BR2_TRY_SYSTEM_CMAKE kconfig variable
- rename the CMAKE variable to BR2_HOST_CMAKE, so it's coherent with the naming of other variables overridable by the environment, e.g. BR2_DL_DIR
- invert the logic of the variable triggering the host-cmake
dependency: USE_SYSTEM_CMAKE instead of BUILD_HOST_CMAKE; needed to implement BR2_TRY_SYSTEM_CMAKE

Changes v1 -> v2:
- Require cmake >= 3.0. Fixes qjson failure (Luca, Samuel, Thomas)
- In check-host-cmake.sh only search $1, not "cmake" (Arnout)
- typo: host-ccache -> host-cmake (Arnout)

---
 package/pkg-cmake.mk                     |  6 ++++--
 support/dependencies/check-host-cmake.mk |  7 +++++++
 support/dependencies/check-host-cmake.sh | 30 ++++++++++++++++++++++++++++++
 3 files changed, 41 insertions(+), 2 deletions(-)
 create mode 100644 support/dependencies/check-host-cmake.mk
 create mode 100755 support/dependencies/check-host-cmake.sh

package/pkg-cmake.mk +4 -2

index 4c6dc6227036..d63fd321fedc 100644
@@ -85,7 +85,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$($$(PKG)_CONF_ENV) $$(BR2_CMAKE) $$($$(PKG)_SRCDIR) \
 		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
 		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),RelWithDebInfo,Release) \
 		-DCMAKE_INSTALL_PREFIX="/usr" \

@@ -110,7 +110,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	$$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$(BR2_CMAKE) $$($$(PKG)_SRCDIR) \
 		-DCMAKE_INSTALL_SO_NO_EXE=0 \
 		-DCMAKE_FIND_ROOT_PATH="$$(HOST_DIR)" \
 		-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM="BOTH" \

@@ -146,7 +146,9 @@ endif
 # primitives to find {C,LD}FLAGS, add it to the dependency list.
 $(2)_DEPENDENCIES += host-pkgconf
 
+ifneq ($$(USE_SYSTEM_CMAKE),YES)
 $(2)_DEPENDENCIES += host-cmake
+endif
 
 #
 # Build step. Only define it if not already defined by the package .mk

support/dependencies/check-host-cmake.mk new file +7 -0

new file mode 100644
index 000000000000..3005cfe68365
@@ -0,0 +1,7 @@
+BR2_CMAKE ?= cmake
+
+ifneq (,$(call suitable-host-package,cmake,$(BR2_CMAKE)))
+USE_SYSTEM_CMAKE = YES
+else
+BR2_CMAKE = $(HOST_DIR)/usr/bin/cmake
+endif

support/dependencies/check-host-cmake.sh new file +30 -0

new file mode 100755
index 000000000000..08de60c974e8
@@ -0,0 +1,30 @@
+#!/bin/sh
+
+candidate="$1"
+
+cmake=`which $candidate`
+if [ ! -x "$cmake" ]; then
+	# echo nothing: no suitable cmake found
+	exit 1
+fi
+
+version=`$cmake --version | head -n1 | cut -d\  -f3`
+major=`echo "$version" | cut -d. -f1`
+minor=`echo "$version" | cut -d. -f2`
+
+# Versions before 3.0 are affected by the bug described in
+# https://git.busybox.net/buildroot/commit/?id=ef2c1970e4bff3be3992014070392b0e6bc28bd2
+# and fixed in upstream CMake in version 3.0:
+# https://cmake.org/gitweb?p=cmake.git;h=e8b8b37ef6fef094940d3384df5a1d421b9fa568
+major_min=3
+minor_min=0
+if [ $major -gt $major_min ]; then
+	echo $cmake
+else
+	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
+		echo $cmake
+	else
+		# echo nothing: no suitable cmake found
+		exit 1
+	fi
+fi

Signature · 1 line

2.7.4

Romain Naour

2016-07-16 21:22 UTC #1 lore raw

PATCH v4 1/1 Don't build host-cmake if it is available on the build host

Hi Luca,

Le 16/07/2016 ? 22:56, Luca Ceresoli a ?crit :

Quoted 120 lines

Currently all cmake packages depend on host-cmake. Unfortunately host-cmake takes a long time to configure and build: almost 7 minutes on a dual-core i5 with SSD. The time does not change even with ccache enabled.
Indeed, building host-cmake is avoidable if it is already installed on the build host: CMake is supposed to be quite portable, and the only patch in Buildroot for the CMake package seems to only affect target-cmake.
Thus we automatically skip building host-cmake and use the one on the system if:
- cmake is available on the system and
- it is recent enough.
First, we leverage the existing infrastructure in support/dependencies/dependencies.mk to find out whether there's a suitable cmake executable on the system. Its path can be passed in the BR2_CMAKE environment variable, otherwise it defaults to "cmake". If it is enabled, found and suitable then we set USE_SYSTEM_CMAKE. Otherwise we override BR2_CMAKE with "$(HOST_DIR)/usr/bin/cmake" to revert to the old behaviour.
Then in pkg-cmake.mk we launch $(BR2_CMAKE) instead of $(HOST_DIR)/usr/bin/cmake.
Finally, we skip adding the dependency on host-cmake for all cmake packages when $(USE_SYSTEM_CMAKE) = YES.
Unlike what we do for host-tar and host-xzcat, for host-cmake we do not add host-cmake to DEPENDENCIES_HOST_PREREQ. If we did, host-cmake would be a dependency for _any_ package when it's not installed on the host, even when no cmake package is selected.
check-host-cmake.sh requires CMake to be at least 3.0 to consider it suitable. This is because older versions are affected by the bug described and fixed in Buildroot in ef2c1970e4bf ("cmake: add patch to fix Qt mkspecs detection"). The bug was fixed in upstream CMake in version 3.0 [0].
Besides, among all the cmake packages currently in Buildroot, the highest version mentioned in cmake_minimum_required() is 3.0 (the grantlee package). Thus 3.0 should be enough to build all current packages. Of course, with the addition or bump of packages, the minimum required version will raise.
Tested on:
- Ubuntu 14.04 without CMake, with official CMake (2.8), PPA CMake (3.2)
- Ubuntu 15.10 without CMake, with official CMake (3.2)
- Ubuntu 16.04 without CMake, with official CMake (3.5)
[0] https://cmake.org/gitweb?p=cmake.git;h=e8b8b37ef6fef094940d3384df5a1d421b9fa568
Signed-off-by Luca Ceresoli <[email protected]>
Cc Samuel Martin <[email protected]>
Cc Davide Viti <[email protected]>
Cc Arnout Vandecappelle <[email protected]>
Cc Thomas Petazzoni <[email protected]>
---
Results:
On my build server, this patch reduced the build time for a batch of 16 Buildroot configurations (all building cmake-based packages) from 2h44m to 1h54m. Woah, it's a 30% saving!
Changes v3 -> v4:
- rename USE_SYSTEM_HOST_CMAKE -> USE_SYSTEM_CMAKE (Arnout)
- ditch BR2_TRY_SYSTEM_CMAKE and use system-cmake unconditionally if it is found and >= 3.0 (Arnout)
- rename BR2_HOST_CMAKE -> BR2_CMAKE since it can be either a host-cmake (built by Buildroot) or a system-cmake
Changes v2 -> v3:
- make this feature optional via the BR2_TRY_SYSTEM_CMAKE kconfig variable
- rename the CMAKE variable to BR2_HOST_CMAKE, so it's coherent with the naming of other variables overridable by the environment, e.g. BR2_DL_DIR
- invert the logic of the variable triggering the host-cmake
dependency: USE_SYSTEM_CMAKE instead of BUILD_HOST_CMAKE; needed to implement BR2_TRY_SYSTEM_CMAKE
Changes v1 -> v2:
- Require cmake >= 3.0. Fixes qjson failure (Luca, Samuel, Thomas)
- In check-host-cmake.sh only search $1, not "cmake" (Arnout)
- typo: host-ccache -> host-cmake (Arnout)
---
 package/pkg-cmake.mk                     |  6 ++++--
 support/dependencies/check-host-cmake.mk |  7 +++++++
 support/dependencies/check-host-cmake.sh | 30 ++++++++++++++++++++++++++++++
 3 files changed, 41 insertions(+), 2 deletions(-)
 create mode 100644 support/dependencies/check-host-cmake.mk
 create mode 100755 support/dependencies/check-host-cmake.sh
package/pkg-cmake.mk +2 -2
index 4c6dc6227036..d63fd321fedc 100644
@@ -85,7 +85,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	$$($$(PKG)_CONF_ENV) $$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$($$(PKG)_CONF_ENV) $$(BR2_CMAKE) $$($$(PKG)_SRCDIR) \
 		-DCMAKE_TOOLCHAIN_FILE="$$(HOST_DIR)/usr/share/buildroot/toolchainfile.cmake" \
 		-DCMAKE_BUILD_TYPE=$$(if $$(BR2_ENABLE_DEBUG),RelWithDebInfo,Release) \
 		-DCMAKE_INSTALL_PREFIX="/usr" \
@@ -110,7 +110,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$($$(PKG)_BUILDDIR) && \
 	rm -f CMakeCache.txt && \
 	PATH=$$(BR_PATH) \
-	$$(HOST_DIR)/usr/bin/cmake $$($$(PKG)_SRCDIR) \
+	$$(BR2_CMAKE) $$($$(PKG)_SRCDIR) \
 		-DCMAKE_INSTALL_SO_NO_EXE=0 \
 		-DCMAKE_FIND_ROOT_PATH="$$(HOST_DIR)" \
 		-DCMAKE_FIND_ROOT_PATH_MODE_PROGRAM="BOTH" \

package/pkg-cmake.mk endif · line 149

Quoted 4 lines

package/pkg-cmake.mk +1 -0

@@ -146,7 +146,9 @@ endif
 # primitives to find {C,LD}FLAGS, add it to the dependency list.
 $(2)_DEPENDENCIES += host-pkgconf
 
+ifneq ($$(USE_SYSTEM_CMAKE),YES)

Maybe it would be better to use the "positive" logic instead.

Otherwise:

Reviewed-by Romain Naour <[email protected]>

[Tested on Fedora 23 with official CMake 3.4.3 with openpowerlink package selected]

Tested-by Romain Naour <[email protected]>

Best regards,
Romain

Quoted 55 lines

package/pkg-cmake.mk +1 -0

 $(2)_DEPENDENCIES += host-cmake
+endif
 
 #
 # Build step. Only define it if not already defined by the package .mk

support/dependencies/check-host-cmake.mk new file +7 -0

new file mode 100644
index 000000000000..3005cfe68365
@@ -0,0 +1,7 @@
+BR2_CMAKE ?= cmake
+
+ifneq (,$(call suitable-host-package,cmake,$(BR2_CMAKE)))
+USE_SYSTEM_CMAKE = YES
+else
+BR2_CMAKE = $(HOST_DIR)/usr/bin/cmake
+endif

support/dependencies/check-host-cmake.sh new file +30 -0

new file mode 100755
index 000000000000..08de60c974e8
@@ -0,0 +1,30 @@
+#!/bin/sh
+
+candidate="$1"
+
+cmake=`which $candidate`
+if [ ! -x "$cmake" ]; then
+	# echo nothing: no suitable cmake found
+	exit 1
+fi
+
+version=`$cmake --version | head -n1 | cut -d\  -f3`
+major=`echo "$version" | cut -d. -f1`
+minor=`echo "$version" | cut -d. -f2`
+
+# Versions before 3.0 are affected by the bug described in
+# https://git.busybox.net/buildroot/commit/?id=ef2c1970e4bff3be3992014070392b0e6bc28bd2
+# and fixed in upstream CMake in version 3.0:
+# https://cmake.org/gitweb?p=cmake.git;h=e8b8b37ef6fef094940d3384df5a1d421b9fa568
+major_min=3
+minor_min=0
+if [ $major -gt $major_min ]; then
+	echo $cmake
+else
+	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
+		echo $cmake
+	else
+		# echo nothing: no suitable cmake found
+		exit 1
+	fi
+fi

Ben Boeckel

2016-07-23 19:33 UTC #2 lore raw

PATCH v4 1/1 Don't build host-cmake if it is available on the build host

In reply to Luca Ceresoli

Quoted 4 lines · Luca Ceresoli

Currently all cmake packages depend on host-cmake. Unfortunately host-cmake takes a long time to configure and build: almost 7 minutes on a dual-core i5 with SSD. The time does not change even with ccache enabled.

FWIW, if Ninja is used to build CMake it is faster[1]. Though I don't see a way to use Ninja from a bootstrapped build?I'm not sure how much work that would be.

--Ben

[1]This is also true of pretty much any CMake-based project.

Yann E. MORIN

2016-09-05 10:00 UTC #3 lore raw

PATCH v4 1/1 Don't build host-cmake if it is available on the build host

In reply to Luca Ceresoli

On 2016-07-16 22:56 +0200, Luca Ceresoli spake thusly:

Quoted 10 lines

Currently all cmake packages depend on host-cmake. Unfortunately host-cmake takes a long time to configure and build: almost 7 minutes on a dual-core i5 with SSD. The time does not change even with ccache enabled.
Indeed, building host-cmake is avoidable if it is already installed on the build host: CMake is supposed to be quite portable, and the only patch in Buildroot for the CMake package seems to only affect target-cmake.

Quoted 4 lines

Thus we automatically skip building host-cmake and use the one on the system if:
- cmake is available on the system and
- it is recent enough.

[--SNIP--]

Quoted 1 line

Signed-off-by Luca Ceresoli <[email protected]>

[--SNIP--]

support/dependencies/check-host-cmake.mk line 7

Quoted 13 lines

support/dependencies/check-host-cmake.mk new file +7 -0

new file mode 100644
index 000000000000..3005cfe68365
@@ -0,0 +1,7 @@
+BR2_CMAKE ?= cmake
+
+ifneq (,$(call suitable-host-package,cmake,$(BR2_CMAKE)))
+USE_SYSTEM_CMAKE = YES
+else
+BR2_CMAKE = $(HOST_DIR)/usr/bin/cmake
+endif

I would simplify the test a little bit:

BR2_CMAKE ?= cmake
ifeq ($(call suitable-host-package,cmake,$(BR2_CMAKE)),) BR2_CMAKE = $(HOST_DIR)/usr/bin/cmake BR2_CMAKE_HOST_DEPENDENCY = host-cmake
endif

and then in package/pkg-cmake.mk:

    $(2)_DEPENDENCIES += $(BR2_CMAKE_HOST_DEPENDENCY)

Regards,
Yann E. MORIN.

Quoted 43 lines

support/dependencies/check-host-cmake.sh new file +30 -0

new file mode 100755
index 000000000000..08de60c974e8
@@ -0,0 +1,30 @@
+#!/bin/sh
+
+candidate="$1"
+
+cmake=`which $candidate`
+if [ ! -x "$cmake" ]; then
+	# echo nothing: no suitable cmake found
+	exit 1
+fi
+
+version=`$cmake --version | head -n1 | cut -d\  -f3`
+major=`echo "$version" | cut -d. -f1`
+minor=`echo "$version" | cut -d. -f2`
+
+# Versions before 3.0 are affected by the bug described in
+# https://git.busybox.net/buildroot/commit/?id=ef2c1970e4bff3be3992014070392b0e6bc28bd2
+# and fixed in upstream CMake in version 3.0:
+# https://cmake.org/gitweb?p=cmake.git;h=e8b8b37ef6fef094940d3384df5a1d421b9fa568
+major_min=3
+minor_min=0
+if [ $major -gt $major_min ]; then
+	echo $cmake
+else
+	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
+		echo $cmake
+	else
+		# echo nothing: no suitable cmake found
+		exit 1
+	fi
+fi

Signature · 6 lines

2.7.4

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 6 lines

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'