dependencies: some improvements
PATCH v5 1/1 dependencies: some improvements
---
v4: add Acks by ThomasP and update after feedback
v5: add extra patch to fix download problem because dl dir is absent
Makefile | 10 +++++++++- package/Makefile.package.in | 12 ------------ package/tar/tar.mk | 12 ++++++++++++ support/dependencies/check-host-tar.mk | 9 +++++++++ support/dependencies/check-host-tar.sh | 35 +++++++++++++++++++++++++++++++++++ toolchain/dependencies/dependencies.mk | 12 +++++++++++- toolchain/dependencies/dependencies.sh | Bin toolchain/toolchain-buildroot.mk | 1 - toolchain/toolchain-crosstool-ng.mk | 1 - toolchain/toolchain-external.mk | 1 - 10 files changed, 76 insertions(+), 17 deletions(-)
PATCH v5 1/1 dependencies: move from toolchain/ to support/
As suggested by Arnout Vandecappelle, move toolchain/dependencies to support/dependencies, as it really is not toolchain-specific anymore.
--- Makefile | 2 ++ toolchain/dependencies/dependencies.mk | 2 +- toolchain/dependencies/dependencies.sh | Bin toolchain/toolchain-buildroot.mk | 1 - toolchain/toolchain-crosstool-ng.mk | 1 - toolchain/toolchain-external.mk | 1 - 6 files changed, 3 insertions(+), 4 deletions(-)
Makefile +2 -0
@@ -298,6 +298,8 @@ include package/Makefile.in all: world +include support/dependencies/dependencies.mk + # We also need the various per-package makefiles, which also add # each selected package to TARGETS if that package was selected # in the .config file.
support/dependencies/dependencies.mk renamed from toolchain/dependencies/dependencies.mk +1 -1
@@ -20,7 +20,7 @@ core-dependencies: @HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \ CONFIG_FILE="$(CONFIG_DIR)/.config" \ DL_TOOLS="$(DL_TOOLS)" \ - $(TOPDIR)/toolchain/dependencies/dependencies.sh + $(TOPDIR)/support/dependencies/dependencies.sh dependencies: core-dependencies $(DEPENDENCIES_HOST_PREREQ)
support/dependencies/dependencies.sh renamed from toolchain/dependencies/dependencies.sh +0 -0
toolchain/toolchain-buildroot.mk +0 -1
@@ -1,6 +1,5 @@ # Include files required for the internal toolchain backend -include toolchain/dependencies/dependencies.mk include toolchain/elf2flt/elf2flt.mk include toolchain/gcc/gcc-uclibc-4.x.mk include toolchain/gdb/gdb.mk
toolchain/toolchain-crosstool-ng.mk +0 -1
@@ -2,7 +2,6 @@ # Explicit ordering: include toolchain/helpers.mk -include toolchain/dependencies/dependencies.mk include toolchain/elf2flt/elf2flt.mk include toolchain/gcc/gcc-uclibc-4.x.mk include toolchain/gdb/gdb.mk
toolchain/toolchain-external.mk +0 -1
@@ -1,7 +1,6 @@ # Required includes for the external toolchain backend include toolchain/helpers.mk -include toolchain/dependencies/dependencies.mk include toolchain/elf2flt/elf2flt.mk include toolchain/gcc/gcc-uclibc-4.x.mk include toolchain/gdb/gdb.mk
PATCH v5 1/1 dependencies: check minimal make version early on
In reply to Thomas De SchampheleireAlthough support/dependencies/dependencies.sh checks for the version of make, this script doesn't get a chance to run if make encounters a syntax error as a result of being too old.
For example, the following syntax is only supported from make 3.81 onwards:
ifeq ($(BR2_TOOLCHAIN_BUILDROOT),y)
include toolchain/toolchain-buildroot.mk
else ifeq ($(BR2_TOOLCHAIN_EXTERNAL),y)
include toolchain/toolchain-external.mk
else ifeq ($(BR2_TOOLCHAIN_CTNG),y)
include toolchain/toolchain-crosstool-ng.mk
endif
This patch adds a check for the version of make very early in the Makefile, so that old make versions are handled gracefully.
--- Makefile | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
Makefile +6 -0
@@ -26,6 +26,12 @@ # Set and export the version string export BR2_VERSION:=2011.11-rc2 +# Check for minimal make version (note: this check will break at make 10.x) +MIN_MAKE_VERSION=3.81 +ifeq (,$(filter $(MIN_MAKE_VERSION),$(firstword $(sort $(MAKE_VERSION) $(MIN_MAKE_VERSION))))) +$(error You have make '$(MAKE_VERSION)' installed. GNU make >= $(MIN_MAKE_VERSION) is required) +endif + # This top-level Makefile can *not* be executed in parallel .NOTPARALLEL:
PATCH v5 1/1 dependencies: add function suitable-host-package
In reply to Thomas De SchampheleireSometimes, buildroot needs a certain host tool to do its job, e.g. tar. In many cases, we expect this tool to be present on the host system, but this is not always the case. Or maybe, the version on the host system is not suitable, and we need a more recent one.
In some of these cases, instead of bailing out, buildroot could build the package first (but only if the existing system package is not suitable).
To aid in detecting if a host package is suitable or not, this patch adds a function suitable-host-package. When called with parameter foo, it will execute check-host-foo.sh. This script should return either the path to the suitable host package, or the empty string if no suitable package can be found. The rules to determine whether something is suitable or not is left to check-host-foo.sh and depends on foo.
An example usage of suitable-host-package is:DEPENDENCIES_HOST_PREREQ += $(if $(call suitable-host-package,foo),,host-foo)
To avoid cluttering the existing dependencies.mk file, it includes any check-host-foo.mk file. These files can be used to hold appropriate dependency-related actions for foo.
v1 Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
v3: Add optional second parameter to suitable-host-package, to specify a
candidate tool.
support/dependencies/dependencies.mk | 10 ++++++++++ 1 files changed, 10 insertions(+), 0 deletions(-)
support/dependencies/dependencies.mk +10 -0
@@ -6,6 +6,16 @@ ###################################################################### DEPENDENCIES_HOST_PREREQ:= + +# suitable-host-pkg: calls check-host-$(1).sh shell script. Parameter (2) +# can be the candidate to be checked. If not present, the check-host-$(1).sh +# script should use 'which' to find a candidate. The script should return +# the path to the suitable host tool, or nothing if no suitable tool was found. +define suitable-host-package +$(shell support/dependencies/check-host-$(1).sh $(2)) +endef +-include support/dependencies/check-host-*.mk + ifeq ($(BR2_STRIP_sstrip),y) DEPENDENCIES_HOST_PREREQ+=host-sstrip endif
PATCH v5 1/1 Makefile: change order of dirs and dependencies
In reply to Thomas De SchampheleireIf during the dependencies step, a package needs to be downloaded, the download directory already has to be present. If not, the file will be downloaded under the name 'dl' instead of in the directory 'dl'.
This patch changes the order of dirs and dependencies in the world target to fix this.
--- Makefile | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
Makefile +1 -1
@@ -376,7 +376,7 @@ dirs: $(DL_DIR) $(TOOLCHAIN_DIR) $(BUILD prepare: $(BUILD_DIR)/buildroot-config/auto.conf -world: prepare dependencies dirs $(BASE_TARGETS) $(TARGETS_ALL) +world: prepare dirs dependencies $(BASE_TARGETS) $(TARGETS_ALL) $(O)/toolchainfile.cmake: @echo -en "\
PATCH v5 1/1 dependencies: build a host-tar if no suitable tar can be found
In reply to Thomas De SchampheleireSome toolchains, like the one built with buildroot itself, use hardlinks (for example to link between the c++ and g++ binary). Unpacking such a toolchain with the --strip-components options does not work correctly if the system tar is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15.
This patch checks for a suitable tar version (tar 1.17+) on the host system, and adds host-tar to the host dependencies if none can be found.
host-tar is download and extracted as cpio.gz instead of tar.gz, to prevent chicken-egg problem.
v1/v2 Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <[email protected]>
---
v3: remove chicken-egg problem, causing great simplification; update check-
host-tar.sh with Arnout's comments.
v4: use 'define' for HOST_TAR_EXTRACT_CMDS after suggestion by ThomasP
package/Makefile.package.in | 12 ------------ package/tar/tar.mk | 12 ++++++++++++ support/dependencies/check-host-tar.mk | 9 +++++++++ support/dependencies/check-host-tar.sh | 35 +++++++++++++++++++++++++++++++++++ 4 files changed, 56 insertions(+), 12 deletions(-)
package/Makefile.package.in +0 -12
@@ -235,18 +235,6 @@ define DOWNLOAD exit 1 endef -# Utility programs used to build packages -TAR ?= tar - -# Automatically detect tar --strip-path/components option -TAR_STRIP_COMPONENTS := \ - $(shell $(TAR) --help | grep strip-path > /dev/null ; \ - if test $$? = 0 ; then \ - echo '--strip-path' ; \ - else \ - echo '--strip-components' ; \ - fi) - # Needed for the foreach loops to loop over the list of hooks, so that # each hook call is properly separated by a newline. define sep
package/tar/tar.mk +12 -0
@@ -13,3 +13,15 @@ TAR_DEPENDENCIES += busybox endif $(eval $(call AUTOTARGETS)) + +# host-tar: use cpio.gz instead of tar.gz to prevent chicken-egg problem +# of needing tar to build tar. +HOST_TAR_SOURCE = tar-$(TAR_VERSION).cpio.gz +define HOST_TAR_EXTRACT_CMDS + mkdir -p $(HOST_TAR_DIR) + cd $(HOST_TAR_DIR) && \ + $(INFLATE.gz) $(DL_DIR)/$(HOST_TAR_SOURCE) | cpio -i + mv $(HOST_TAR_DIR)/tar-$(TAR_VERSION)/* $(HOST_TAR_DIR) + rmdir $(HOST_TAR_DIR)/tar-$(TAR_VERSION) +endef +$(eval $(call AUTOTARGETS,host))
support/dependencies/check-host-tar.mk new file +9 -0
@@ -0,0 +1,9 @@ +TAR ?= tar + +ifeq (,$(call suitable-host-package,tar,$(TAR))) + DEPENDENCIES_HOST_PREREQ += host-tar + TAR = $(HOST_DIR)/usr/bin/tar +endif + +# Since TAR is at least 1.17, it will certainly support --strip-components +TAR_STRIP_COMPONENTS = --strip-components
support/dependencies/check-host-tar.sh new file +35 -0
@@ -0,0 +1,35 @@ +#!/bin/sh + +candidate="$1" + +tar=`which $candidate` +if [ ! -x "$tar" ]; then + tar=`which tar` + if [ ! -x "$tar" ]; then + # echo nothing: no suitable tar found + exit 1 + fi +fi + +# Output of 'tar --version' examples: +# tar (GNU tar) 1.15.1 +# tar (GNU tar) 1.25 +version=`$tar --version | head -n 1 | sed 's/^.*\s\([0-9]\+\.\S\+\).*$/\1/'` +major=`echo "$version" | cut -d. -f1` +minor=`echo "$version" | cut -d. -f2` +bugfix=`echo "$version" | cut -d. -f3` + +# Minimal version = 1.17 (previous versions do not correctly unpack archives +# containing hard-links if the --strip-components option is used). +major_min=1 +minor_min=17 +if [ $major -gt $major_min ]; then + echo $tar +else + if [ $major -eq $major_min -a $minor -ge $minor_min ]; then + echo $tar + else + # echo nothing: no suitable tar found + exit 1 + fi +fi
PATCH v5 1/1 dependencies: build a host-tar if no suitable tar can be found
Quoted 6 lines · On Friday 16 December 2011 09:52:29 Thomas De Schampheleire wrote:
Some toolchains, like the one built with buildroot itself, use hardlinks (for example to link between the c++ and g++ binary). Unpacking such a toolchain with the --strip-components options does not work correctly if the system tar is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15.
Quoted 8 lines
This patch checks for a suitable tar version (tar 1.17+) on the host system, and adds host-tar to the host dependencies if none can be found.
host-tar is download and extracted as cpio.gz instead of tar.gz, to prevent chicken-egg problem.
Signed-off-by Thomas De Schampheleire <[email protected]>v1/v2 Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <[email protected]>
[snip]
Quoted 9 lines
package/tar/tar.mk +2
-0@@ -13,3 +13,15 @@ TAR_DEPENDENCIES += busybox endif $(eval $(call AUTOTARGETS)) + +# host-tar: use cpio.gz instead of tar.gz to prevent chicken-egg problem
package/tar/tar.mk TAR_DEPENDENCIES += busybox · line 27
Quoted 10 lines
package/tar/tar.mk +10
-0+# of needing tar to build tar. +HOST_TAR_SOURCE = tar-$(TAR_VERSION).cpio.gz +define HOST_TAR_EXTRACT_CMDS + mkdir -p $(HOST_TAR_DIR) + cd $(HOST_TAR_DIR) && \ + $(INFLATE.gz) $(DL_DIR)/$(HOST_TAR_SOURCE) | cpio -i + mv $(HOST_TAR_DIR)/tar-$(TAR_VERSION)/* $(HOST_TAR_DIR) + rmdir $(HOST_TAR_DIR)/tar-$(TAR_VERSION) +endef +$(eval $(call AUTOTARGETS,host))
It is indeed much simpler this way!
One small remark: we usually use $(@D) rather than $($(PKG)_DIR).
Quoted 44 lines
support/dependencies/check-host-tar.mk new file +9
-0@@ -0,0 +1,9 @@ +TAR ?= tar + +ifeq (,$(call suitable-host-package,tar,$(TAR))) + DEPENDENCIES_HOST_PREREQ += host-tar + TAR = $(HOST_DIR)/usr/bin/tar +endif + +# Since TAR is at least 1.17, it will certainly support --strip-components +TAR_STRIP_COMPONENTS = --strip-componentssupport/dependencies/check-host-tar.sh new file +25
-0@@ -0,0 +1,35 @@ +#!/bin/sh + +candidate="$1" + +tar=`which $candidate` +if [ ! -x "$tar" ]; then + tar=`which tar` + if [ ! -x "$tar" ]; then + # echo nothing: no suitable tar found + exit 1 + fi +fi + +# Output of 'tar --version' examples: +# tar (GNU tar) 1.15.1 +# tar (GNU tar) 1.25 +version=`$tar --version | head -n 1 | sed 's/^.*\s\([0-9]\+\.\S\+\).*$/\1/'` +major=`echo "$version" | cut -d. -f1` +minor=`echo "$version" | cut -d. -f2` +bugfix=`echo "$version" | cut -d. -f3` + +# Minimal version = 1.17 (previous versions do not correctly unpack archives +# containing hard-links if the --strip-components option is used). +major_min=1 +minor_min=17
support/dependencies/check-host-tar.sh line 35
Quoted 10 lines
support/dependencies/check-host-tar.sh new file +10
-0+if [ $major -gt $major_min ]; then + echo $tar +else + if [ $major -eq $major_min -a $minor -ge $minor_min ]; then + echo $tar + else + # echo nothing: no suitable tar found + exit 1 + fi +fi
You could use the same sorting trick that you use in patch 2/5, but then
using host sort:
version=echo -e '${MINIMAL_VERSION}\n${version}' | sort -n if [ "$version" != "${MINIMAL_VERSION}" ]; then
exit 1
else
echo $tar
fi
(completely untested, of course).
But don't count this as a requirement to get my Acked-by :-)
Regards,
Arnout
Signature · 6 lines
Arnout Vandecappelle arnout at mind be Senior Embedded Software Architect +32-16-286540 Essensium/Mind http://www.mind.be G.Geenslaan 9, 3001 Leuven, Belgium BE 872 984 063 RPR Leuven LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle GPG fingerprint: 7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F