Skip to contentSkip to conversation

allow build infrastructure to pick up installed meson tool

PATCH Superseded Norbert Lange 3 participants 8 messages May 3, 2019 – May 8, 2019 Arm/Amlogic Meson SoC support

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 allow build infrastructure to pick up installed meson tool

From: Norbert Lange <[email protected]>

Automatically check for an available meson tool,
and use it aslond the version is fitting.

Some concerns are about being able to figure out the correct version for all available packages in buildroot. The min version could be set to version 1.0 to postpone the problem, but still have the infrastructure in place to allow users to override the version.

Currently host-ninja will still be always built.

---
 package/meson/meson.mk                   |  1 -
 package/pkg-meson.mk                     |  4 +--
 support/dependencies/check-host-meson.mk | 16 +++++++++
 support/dependencies/check-host-meson.sh | 45 ++++++++++++++++++++++++
 4 files changed, 63 insertions(+), 3 deletions(-)
 create mode 100644 support/dependencies/check-host-meson.mk
 create mode 100755 support/dependencies/check-host-meson.sh
package/meson/meson.mk +0 -1
index cf62b0ddde..2ce7a26ab5 100644
@@ -10,7 +10,6 @@ MESON_LICENSE = Apache-2.0
 MESON_LICENSE_FILES = COPYING
 MESON_SETUP_TYPE = setuptools
 
-HOST_MESON_DEPENDENCIES = host-ninja
 HOST_MESON_NEEDS_HOST_PYTHON = python3
 
 HOST_MESON_TARGET_ENDIAN = $(call LOWERCASE,$(BR2_ENDIAN))
package/pkg-meson.mk +2 -2
index 7f1838c09a..25e9bb814e 100644
@@ -25,7 +25,7 @@
 # $(HOST_DIR)/bin/python3 will not look for Meson modules in
 # $HOME/.local/lib/python3.x/site-packages
 #
-MESON		= PYTHONNOUSERSITE=y $(HOST_DIR)/bin/meson
+MESON		= PYTHONNOUSERSITE=y $(BR2_MESON)
 NINJA		= PYTHONNOUSERSITE=y $(HOST_DIR)/bin/ninja
 NINJA_OPTS	= $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
@@ -105,7 +105,7 @@ endef
 endif
 endif
 
-$(2)_DEPENDENCIES += host-meson
+$(2)_DEPENDENCIES += $(BR2_MESON_HOST_DEPENDENCY)
 
 #
 # Build step. Only define it if not already defined by the package .mk
support/dependencies/check-host-meson.mk new file +16 -0
new file mode 100644
index 0000000000..6c779b3e6d
@@ -0,0 +1,16 @@
+# Set this to either 0.49 or higher, depending on the highest minimum
+# version required by any of the packages bundled in Buildroot. If a
+# package is bumped or a new one added, and it requires a higher
+# version, our meson infra will catch it and build its own.
+#
+BR2_MESON_VERSION_MIN = 0.49
+
+BR2_MESON_CANDIDATES ?= meson
+BR2_MESON ?= $(call suitable-host-package,meson,\
+	$(BR2_MESON_VERSION_MIN) $(BR2_MESON_CANDIDATES))
+ifeq ($(BR2_MESON),)
+BR2_MESON = $(HOST_DIR)/bin/meson
+BR2_MESON_HOST_DEPENDENCY = host-meson host-ninja
+else
+BR2_MESON_HOST_DEPENDENCY = host-ninja
+endif
support/dependencies/check-host-meson.sh new file +45 -0

45 lines not on this page · raw message

--
2.20.1

#1 lore raw

PATCH 2/2 allow build infrastructure to pick up installed ninja tool

From: Norbert Lange <[email protected]>

If a fitting ninja tool is detected it will be used, otherwise the tool will be built from source.

Replace the fixed dependencies to host-ninja,
notably from the meson infrastructure.

---
 package/pkg-meson.mk                     |  2 +-
 support/dependencies/check-host-meson.mk |  4 +--
 support/dependencies/check-host-ninja.mk | 14 ++++++++
 support/dependencies/check-host-ninja.sh | 45 ++++++++++++++++++++++++
 4 files changed, 62 insertions(+), 3 deletions(-)
 create mode 100644 support/dependencies/check-host-ninja.mk
 create mode 100755 support/dependencies/check-host-ninja.sh
package/pkg-meson.mk +1 -1
index 25e9bb814e..c4cf682ee8 100644
@@ -26,7 +26,7 @@
 # $HOME/.local/lib/python3.x/site-packages
 #
 MESON		= PYTHONNOUSERSITE=y $(BR2_MESON)
-NINJA		= PYTHONNOUSERSITE=y $(HOST_DIR)/bin/ninja
+NINJA		= PYTHONNOUSERSITE=y $(BR2_NINJA)
 NINJA_OPTS	= $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
 ################################################################################
support/dependencies/check-host-meson.mk +2 -2
index 6c779b3e6d..36fc9129d1 100644
@@ -10,7 +10,7 @@ BR2_MESON ?= $(call suitable-host-package,meson,\
 	$(BR2_MESON_VERSION_MIN) $(BR2_MESON_CANDIDATES))
 ifeq ($(BR2_MESON),)
 BR2_MESON = $(HOST_DIR)/bin/meson
-BR2_MESON_HOST_DEPENDENCY = host-meson host-ninja
+BR2_MESON_HOST_DEPENDENCY = host-meson $(BR2_NINJA_HOST_DEPENDENCY)
 else
-BR2_MESON_HOST_DEPENDENCY = host-ninja
+BR2_MESON_HOST_DEPENDENCY = $(BR2_NINJA_HOST_DEPENDENCY)
 endif
support/dependencies/check-host-ninja.mk new file +14 -0
new file mode 100644
index 0000000000..6d89255ee5
@@ -0,0 +1,14 @@
+# Set this to either 1.8.2 or higher, depending on the highest minimum
+# version required by any of the packages bundled in Buildroot. If a
+# package is bumped or a new one added, and it requires a higher
+# version, our ninja infra will catch it and build its own.
+#
+BR2_NINJA_VERSION_MIN = 1.8.0
+
+BR2_NINJA_CANDIDATES ?= ninja
+BR2_NINJA ?= $(call suitable-host-package,ninja,\
+	$(BR2_NINJA_VERSION_MIN) $(BR2_NINJA_CANDIDATES))
+ifeq ($(BR2_NINJA),)
+BR2_NINJA = $(HOST_DIR)/bin/ninja
+BR2_NINJA_HOST_DEPENDENCY = host-ninja
+endif
support/dependencies/check-host-ninja.sh new file +45 -0

45 lines not on this page · raw message

--
2.20.1

#2 lore raw

PATCH 2/2 allow build infrastructure to pick up installed ninja tool

Hello Norbert,

Quoted 9 lines · On Fri, 3 May 2019 15:18:51 +0200 Norbert Lange <[email protected]> wrote:

From: Norbert Lange <[email protected]>

If a fitting ninja tool is detected it will be used, otherwise the tool will be built from source.

Replace the fixed dependencies to host-ninja,
notably from the meson infrastructure.

Since host-meson cannot be changed to use the system-provided meson, as discussed with Peter Seiderer on PATCH 1/2, I don't think it's really worth doing the effort for host-ninja, which requires the same thing as
host-meson: a host Python interpreter.

I think the approaches to reduce the build time are:

- Make sure we only need to build one of host-python or host-python3, and not both. This is what J?rg Krause is proposing in
http://patchwork.ozlabs.org/patch/1047119/.

- Or perhaps, try to go one step further, and see if we can use the system-provided Python instead of a Python built by Buildroot. But that would only work if the system provides Python 3, since Meson requires Python 3.

support/dependencies/check-host-ninja.mk line 4

Quoted 10 lines
support/dependencies/check-host-ninja.mk new file +4 -0
new file mode 100644
index 0000000000..6d89255ee5
@@ -0,0 +1,14 @@
+# Set this to either 1.8.2 or higher, depending on the highest minimum
+# version required by any of the packages bundled in Buildroot. If a
+# package is bumped or a new one added, and it requires a higher
+# version, our ninja infra will catch it and build its own.

I know this comment is in other .mk files in the same directory, but it doesn't make any sense. There is no such thing as a "ninja infra" in Buildroot, and there is nothing that will "catch it" and build its own. If we forget to update BR2_NINJA_VERSION_MIN to the oldest Ninja version that is acceptable by all packages using Ninja in Buildroot, we will never notice until a user hits the problem.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw

PATCH 2/2 allow build infrastructure to pick up installed ninja tool

Am Di., 7. Mai 2019 um 23:00 Uhr schrieb Thomas Petazzoni <[email protected]>:

Quoted 14 lines

Hello Norbert,

Quoted 8 lines · On Fri, 3 May 2019 15:18:51 +0200 Norbert Lange <[email protected]> wrote:

From: Norbert Lange <[email protected]>

If a fitting ninja tool is detected it will be used, otherwise the tool will be built from source.

Replace the fixed dependencies to host-ninja,
notably from the meson infrastructure.

Quoted 6 lines
Quoted 1 line

Since host-meson cannot be changed to use the system-provided meson, as discussed with Peter Seiderer on PATCH 1/2, I don't think it's really worth doing the effort for host-ninja, which requires the same thing as
host-meson: a host Python interpreter.

The effort is already done in the patch and not really a big one. Is there anything bad about
having some additional requirements matched?
But yes, most of the time is spent building python (twice).

Quoted 10 lines

I think the approaches to reduce the build time are:

- Make sure we only need to build one of host-python or host-python3, and not both. This is what J?rg Krause is proposing in
http://patchwork.ozlabs.org/patch/1047119/.

- Or perhaps, try to go one step further, and see if we can use the system-provided Python instead of a Python built by Buildroot. But that would only work if the system provides Python 3, since Meson requires Python 3.

maybe add some python-any dependency (resolves to either 2 or 3), which is enough for bootstrapping host-ninja?

Quoted 10 lines
Quoted 10 lines
support/dependencies/check-host-ninja.mk new file +3 -0
new file mode 100644
index 0000000000..6d89255ee5
@@ -0,0 +1,14 @@
+# Set this to either 1.8.2 or higher, depending on the highest minimum
+# version required by any of the packages bundled in Buildroot. If a
+# package is bumped or a new one added, and it requires a higher
Quoted 8 lines

support/dependencies/check-host-ninja.mk line 4

Quoted 1 line
support/dependencies/check-host-ninja.mk new file +1 -0
+# version, our ninja infra will catch it and build its own.

I know this comment is in other .mk files in the same directory, but it doesn't make any sense. There is no such thing as a "ninja infra" in Buildroot, and there is nothing that will "catch it" and build its own. If we forget to update BR2_NINJA_VERSION_MIN to the oldest Ninja version that is acceptable by all packages using Ninja in Buildroot, we will never notice until a user hits the problem.

That's something what your CI builder could catch.
packages break all the time with new compilers and libc's and I think ninja should be pretty much painless in comparison.
The meta-buildsystems like mesons should be able to check for a matching version (which might depend on the build, like C++20 modules will need some features in ninja)

Norbert.

#4 lore raw

PATCH 1/2 allow build infrastructure to pick up installed meson tool

In reply to Norbert Lange

Hello Norbert,

appreciating every patch which speeds up the buildroot build process ;-), but some nits, one minor comments and one major concern, see below...

Quoted 4 lines · Norbert Lange

From: Norbert Lange <[email protected]>

Automatically check for an available meson tool,
and use it aslond the version is fitting.

s/aslond/as long/

Quoted 4 lines

Some concerns are about being able to figure out the correct version for all available packages in buildroot. The min version could be set to version 1.0 to postpone the

You mean 1.0 to disable host meson usage?

Quoted 55 lines

problem, but still have the infrastructure in place to allow users to override the version.

Currently host-ninja will still be always built.

---
 package/meson/meson.mk                   |  1 -
 package/pkg-meson.mk                     |  4 +--
 support/dependencies/check-host-meson.mk | 16 +++++++++
 support/dependencies/check-host-meson.sh | 45 ++++++++++++++++++++++++
 4 files changed, 63 insertions(+), 3 deletions(-)
 create mode 100644 support/dependencies/check-host-meson.mk
 create mode 100755 support/dependencies/check-host-meson.sh
package/meson/meson.mk +0 -1
index cf62b0ddde..2ce7a26ab5 100644
@@ -10,7 +10,6 @@ MESON_LICENSE = Apache-2.0
 MESON_LICENSE_FILES = COPYING
 MESON_SETUP_TYPE = setuptools
 
-HOST_MESON_DEPENDENCIES = host-ninja
 HOST_MESON_NEEDS_HOST_PYTHON = python3
 
 HOST_MESON_TARGET_ENDIAN = $(call LOWERCASE,$(BR2_ENDIAN))
package/pkg-meson.mk +2 -2
index 7f1838c09a..25e9bb814e 100644
@@ -25,7 +25,7 @@
 # $(HOST_DIR)/bin/python3 will not look for Meson modules in
 # $HOME/.local/lib/python3.x/site-packages
 #
-MESON		= PYTHONNOUSERSITE=y $(HOST_DIR)/bin/meson
+MESON		= PYTHONNOUSERSITE=y $(BR2_MESON)
 NINJA		= PYTHONNOUSERSITE=y $(HOST_DIR)/bin/ninja
 NINJA_OPTS	= $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
@@ -105,7 +105,7 @@ endef
 endif
 endif
 
-$(2)_DEPENDENCIES += host-meson
+$(2)_DEPENDENCIES += $(BR2_MESON_HOST_DEPENDENCY)
 
 #
 # Build step. Only define it if not already defined by the package .mk
support/dependencies/check-host-meson.mk new file +0 -0
new file mode 100644
index 0000000000..6c779b3e6d

support/dependencies/check-host-meson.mk line 6

Quoted 6 lines
support/dependencies/check-host-meson.mk new file +6 -0
@@ -0,0 +1,16 @@
+# Set this to either 0.49 or higher, depending on the highest minimum
+# version required by any of the packages bundled in Buildroot. If a
+# package is bumped or a new one added, and it requires a higher
+# version, our meson infra will catch it and build its own.
+#
+BR2_MESON_VERSION_MIN = 0.49

Needs support for third version number, see latest meson bump to 0.50 and revert, but 0.50.1 worked ([1], [2], [3])...

The major concern is no distribution will ship meson with the still needed buildroot patch, see [4]...

Regards,
Peter

[1] https://git.buildroot.net/buildroot/commit/?id=114e9dcd28e1001b74689215ec669b8940dc3ea9
[2] https://git.buildroot.net/buildroot/commit/?id=c37b81af64dd8d6729325d8edbe633e6a186bb3d
[3] https://git.buildroot.net/buildroot/commit/?id=70fb5e610761dd6468a058c580acc3d4f81fe547
[4] https://git.buildroot.net/buildroot/tree/package/meson/0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch

Quoted 68 lines
support/dependencies/check-host-meson.mk new file +10 -0
+
+BR2_MESON_CANDIDATES ?= meson
+BR2_MESON ?= $(call suitable-host-package,meson,\
+	$(BR2_MESON_VERSION_MIN) $(BR2_MESON_CANDIDATES))
+ifeq ($(BR2_MESON),)
+BR2_MESON = $(HOST_DIR)/bin/meson
+BR2_MESON_HOST_DEPENDENCY = host-meson host-ninja
+else
+BR2_MESON_HOST_DEPENDENCY = host-ninja
+endif
support/dependencies/check-host-meson.sh new file +45 -1
new file mode 100755
index 0000000000..805fac9349
@@ -0,0 +1,45 @@
+#!/bin/sh
+
+# prevent shift error
+[ $# -lt 2 ] && exit 1
+
+split_version() {
+    local VARPREFIX
+    local NUMBERS
+    local major
+    local minor
+
+    VARPREFIX=$1
+    NUMBERS=$2
+
+    major=${NUMBERS%%\.*}
+    NUMBERS=${NUMBERS#$major}; NUMBERS=${NUMBERS#\.}
+    minor=${NUMBERS%%\.*}
+    NUMBERS=${NUMBERS#$minor}; NUMBERS=${NUMBERS#\.}
+
+    # ensure that missing values are 0
+    eval "${VARPREFIX}_major=\$major; ${VARPREFIX}_minor=\$((minor + 0)); ${VARPREFIX}_bugfix=\$((NUMBERS + 0));"
+}
+
+split_version req "$1"
+
+shift
+
+for candidate; do
+
+    # Try to locate the candidate. Discard it if not located.
+    meson=$(which "${candidate}" 2>/dev/null)
+    [ -n "${meson}" ] || continue
+
+    split_version cur "$("${meson}" --version)"
+
+    [ -n "${cur_major}" -a "${cur_major}" -ge "${req_major}" ] || continue
+    [ "${cur_major}" -gt "${req_major}" ] || [ "${cur_minor}" -ge "${req_minor}" ] || continue
+    [ "${cur_minor}" -gt "${req_minor}" ] || [ "${cur_bugfix}" -ge "${req_bugfix}" ] || continue
+
+    echo "${meson}"
+    exit
+done
+
+# echo nothing: no suitable meson found
+exit 1
--

2.20.1

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#5 lore raw

PATCH 1/2 allow build infrastructure to pick up installed meson tool

Am Fr., 3. Mai 2019 um 18:57 Uhr schrieb Peter Seiderer <[email protected]>:

Quoted 6 lines

Hello Norbert,

appreciating every patch which speeds up the buildroot build process ;-), but some nits, one minor comments and one major concern, see below...

Quoted 8 lines
Quoted 4 lines · Norbert Lange

From: Norbert Lange <[email protected]>

Automatically check for an available meson tool,
and use it aslond the version is fitting.

s/aslond/as long/

Ack

Quoted 6 lines
Quoted 4 lines

Some concerns are about being able to figure out the correct version for all available packages in buildroot. The min version could be set to version 1.0 to postpone the

You mean 1.0 to disable host meson usage?

Yes, so meson is always built like it is now, if this is too critical to enable by default.

Quoted 56 lines
Quoted 56 lines

problem, but still have the infrastructure in place to allow users to override the version.

Currently host-ninja will still be always built.

---
 package/meson/meson.mk                   |  1 -
 package/pkg-meson.mk                     |  4 +--
 support/dependencies/check-host-meson.mk | 16 +++++++++
 support/dependencies/check-host-meson.sh | 45 ++++++++++++++++++++++++
 4 files changed, 63 insertions(+), 3 deletions(-)
 create mode 100644 support/dependencies/check-host-meson.mk
 create mode 100755 support/dependencies/check-host-meson.sh
package/meson/meson.mk +0 -1
index cf62b0ddde..2ce7a26ab5 100644
@@ -10,7 +10,6 @@ MESON_LICENSE = Apache-2.0
 MESON_LICENSE_FILES = COPYING
 MESON_SETUP_TYPE = setuptools
 
-HOST_MESON_DEPENDENCIES = host-ninja
 HOST_MESON_NEEDS_HOST_PYTHON = python3
 
 HOST_MESON_TARGET_ENDIAN = $(call LOWERCASE,$(BR2_ENDIAN))
package/pkg-meson.mk +2 -2
index 7f1838c09a..25e9bb814e 100644
@@ -25,7 +25,7 @@
 # $(HOST_DIR)/bin/python3 will not look for Meson modules in
 # $HOME/.local/lib/python3.x/site-packages
 #
-MESON                = PYTHONNOUSERSITE=y $(HOST_DIR)/bin/meson
+MESON                = PYTHONNOUSERSITE=y $(BR2_MESON)
 NINJA                = PYTHONNOUSERSITE=y $(HOST_DIR)/bin/ninja
 NINJA_OPTS   = $(if $(VERBOSE),-v) -j$(PARALLEL_JOBS)
 
@@ -105,7 +105,7 @@ endef
 endif
 endif
 
-$(2)_DEPENDENCIES += host-meson
+$(2)_DEPENDENCIES += $(BR2_MESON_HOST_DEPENDENCY)
 
 #
 # Build step. Only define it if not already defined by the package .mk
support/dependencies/check-host-meson.mk new file +1 -0
new file mode 100644
index 0000000000..6c779b3e6d
@@ -0,0 +1,16 @@
+# Set this to either 0.49 or higher, depending on the highest minimum
Quoted 8 lines

support/dependencies/check-host-meson.mk line 6

Quoted 5 lines
support/dependencies/check-host-meson.mk new file +5 -0
+# version required by any of the packages bundled in Buildroot. If a
+# package is bumped or a new one added, and it requires a higher
+# version, our meson infra will catch it and build its own.
+#
+BR2_MESON_VERSION_MIN = 0.49

Needs support for third version number, see latest meson bump to 0.50 and revert, but 0.50.1 worked ([1], [2], [3])...

There is support for up to 3 version components in support/dependencies/check-host-meson.sh

Quoted 3 lines

The major concern is no distribution will ship meson with the still needed buildroot patch, see [4]...

Ok, news to me.
Whats the practical effect though, does rpath even work if you relocated buildroot?
Is it necessary if you set LD_LIBRARY_PATH?
(From my personal experience I was never happy with RPATH).

Quoted 3 lines

Regards,
Peter

Regards, Norbert

#6 lore raw

PATCH 1/2 allow build infrastructure to pick up installed meson tool

In reply to Peter Seiderer

Hello,

Quoted 2 lines · On Fri, 3 May 2019 18:57:21 +0200 Peter Seiderer <[email protected]> wrote:

The major concern is no distribution will ship meson with the still needed buildroot patch, see [4]...

Indeed, that makes it impractical/impossible to use a system-provided meson. If we want to get there at some point, the 0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch patch needs to be upstreamed in a form or another.

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#7 lore raw

PATCH 1/2 allow build infrastructure to pick up installed meson tool

Am Di., 7. Mai 2019 um 22:56 Uhr schrieb Thomas Petazzoni <[email protected]>:

Quoted 6 lines

Hello,

On Fri, 3 May 2019 18:57:21 +0200
Peter Seiderer <[email protected]> wrote:

Quoted 7 lines
Quoted 2 lines

The major concern is no distribution will ship meson with the still needed buildroot patch, see [4]...

Indeed, that makes it impractical/impossible to use a system-provided meson. If we want to get there at some point, the 0001-Only-fix-RPATH-if-install_rpath-is-not-empty.patch patch needs to be upstreamed in a form or another.

I see why you would not want to depend on a system provided meson, but it does work for target libraries.
So my proposal would be to incorporate the changes now, with min-version set to a value that always evaluates to false.

So a user like me can enable the system meson easily, even if its not officially supported or tested.

Norbert