dependencies: some improvements

This patch series makes some changes in the dependencies code.

dependencies: check core dependencies before anything else
dependencies: add function suitable-host-package
dependencies: build a host-tar if no suitable tar can be found
dependencies: remove unused lzma checking scripts dirs and dependencies should be executed before every package

Although comments on all patches would be fine, I'm mainly troubled with the last one ('dirs and dependencies...'). Please see the patch description there.

Currently, the dependencies target (that runs dependencies.sh) depends on DEPENDENCIES_HOST_PREREQ. This means that the dependencies listed in DEPENDENCIES_HOST_PREREQ (currently host-sstrip if sstrip is selected) are built *before* the dependencies.sh script is run.

As a result, if e.g. there is no gcc compiler present on the build system, the dependencies in DEPENDENCIES_HOST_PREREQ will fail to build, and buildroot will fail non-gracefully.

This patch makes sure that the DEPENDENCIES_HOST_PREREQ are checked *after* the dependencies.sh script, so that any problem in the build system is reported in a clean way by dependencies.sh.

---
 toolchain/dependencies/dependencies.mk |  6 ++++--
 1 files changed, 4 insertions(+), 2 deletions(-)

@@ -16,12 +16,14 @@ DL_TOOLS = \
 	$(findstring git,$(DL_TOOLS_DEPENDENCIES)) \
 	$(findstring bzr,$(DL_TOOLS_DEPENDENCIES))
 
-dependencies: $(DEPENDENCIES_HOST_PREREQ)
+core-dependencies:
 	@HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \
 		CONFIG_FILE="$(CONFIG_DIR)/.config" \
 		DL_TOOLS="$(DL_TOOLS)" \
 		$(TOPDIR)/toolchain/dependencies/dependencies.sh
 
+dependencies: core-dependencies $(DEPENDENCIES_HOST_PREREQ)
+
 dependencies-source:
 
 dependencies-clean:

@@ -35,5 +37,5 @@ dependencies-dirclean:
 # Toplevel Makefile options
 #
 #############################################################
-.PHONY: dependencies
+.PHONY: dependencies core-dependencies
 

Sometimes, buildroot needs a certain host tool to do its job, e.g. tar. In many cases, we expect this tool to be present on the host system, but this is not always the case. Or maybe, the version on the host system is not suitable, and we need a more recent one.

In some of these cases, instead of bailing out, buildroot could build the package first (but only if the existing system package is not suitable).

To aid in detecting if a host package is suitable or not, this patch adds a function suitable-host-package. When called with parameter foo, it will execute check-host-foo.sh, or if absent, make function check-host-foo. These should return either the path to the suitable host package, or the empty string if no suitable package can be found.
The rules to determine whether something is suitable or not is left to the check-host-foo(.sh) function and depends on foo.

An example usage of suitable-host-package is:
DEPENDENCIES_HOST_PREREQ += $(if $(call suitable-host-package,foo),,host-foo)

To avoid cluttering the existing dependencies.mk file, it includes any check-host-foo.mk file. These files can be used to hold the appropriate functions or definitions.

---
 toolchain/dependencies/dependencies.mk |  10 ++++++++++
 1 files changed, 10 insertions(+), 0 deletions(-)

@@ -6,6 +6,16 @@
 ######################################################################
 
 DEPENDENCIES_HOST_PREREQ:=
+
+# suitable-host-pkg: calls check-host-$(1).sh shell script if it exists,
+# otherwise calls check-host-$(1) make function. This script/function
+# should return the path to the suitable host tool, or nothing if no
+# suitable tool was found.
+define suitable-host-package
+$(if $(wildcard toolchain/dependencies/check-host-$(1).sh),$(shell toolchain/dependencies/check-host-$(1).sh),$(call check-host-$(1)))
+endef
+include toolchain/dependencies/check-host-*.mk
+
 ifeq ($(BR2_STRIP_sstrip),y)
 DEPENDENCIES_HOST_PREREQ+=host-sstrip
 endif

Some toolchains, like the one built with buildroot itself, use hardlinks (for example to link between the c++ and g++ binary). Unpacking such a toolchain with the --strip-components options does not work correctly if the system tar is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15.

This patch checks for a suitable tar version (tar 1.17+) on the host system, and adds host-tar to the host dependencies if none can be found.

TAR is redefined to HOST_TAR, except when extracting host-tar (this is a chicken-and-egg problem), so that all packages use the host-tar if no suitable tar was found.

Along with this patch, the definition of TAR_STRIP_COMPONENTS was moved to check-host-tar.mk.

---
 package/Makefile.package.in              |  12 ------------
 package/tar/tar.mk                       |   4 ++++
 toolchain/dependencies/check-host-tar.mk |  24 ++++++++++++++++++++++++
 toolchain/dependencies/check-host-tar.sh |  29 +++++++++++++++++++++++++++++
 4 files changed, 57 insertions(+), 12 deletions(-)

@@ -235,18 +235,6 @@ define DOWNLOAD
 	exit 1
 endef
 
-# Utility programs used to build packages
-TAR ?= tar
-
-# Automatically detect tar --strip-path/components option
-TAR_STRIP_COMPONENTS := \
-  $(shell $(TAR) --help | grep strip-path > /dev/null ; \
-  if test $$? = 0 ; then \
-   echo '--strip-path' ; \
-  else \
-   echo '--strip-components' ; \
-  fi)
-
 # Needed for the foreach loops to loop over the list of hooks, so that
 # each hook call is properly separated by a newline.
 define sep

@@ -8,3 +8,7 @@ TAR_VERSION = 1.26
 TAR_SITE = $(BR2_GNU_MIRROR)/tar
 
 $(eval $(call AUTOTARGETS))
+
+TAR := $(SYSTEM_TAR) # We need a real tar to extract tar
+$(eval $(call AUTOTARGETS,host))
+TAR := $(HOST_TAR)

new file mode 100644
@@ -0,0 +1,24 @@
+TAR ?= tar
+SYSTEM_TAR := $(TAR)
+
+ifneq (,$(call suitable-host-package,tar))
+  HOST_TAR := $(TAR)
+
+  # Automatically detect tar --strip-path/components option
+  # --strip-path was renamed to --strip-components in tar 1.15
+  TAR_STRIP_COMPONENTS := \
+    $(shell $(TAR) --help | grep strip-path > /dev/null ; \
+    if test $$? = 0 ; then \
+     echo '--strip-path' ; \
+    else \
+     echo '--strip-components' ; \
+    fi)
+
+else
+  DEPENDENCIES_HOST_PREREQ += host-tar
+  HOST_TAR := $(HOST_DIR)/usr/bin/tar
+  TAR_STRIP_COMPONENTS := --strip-components
+
+endif
+
+TAR := $(HOST_TAR)

new file mode 100755
@@ -0,0 +1,29 @@
+#!/bin/sh
+
+tar=`which tar`
+if [ -z "$tar" ]; then
+	return
+fi
+
+# Output of 'tar --version' examples:
+# tar (GNU tar) 1.15.1 
+# tar (GNU tar) 1.25
+version=`$tar --version | head -n 1 | sed 's/^.*\s\([0-9]\+\.\S\+\).*$/\1/'`
+major=`echo "$version" | cut -d. -f1`
+minor=`echo "$version" | cut -d. -f2`
+bugfix=`echo "$version" | cut -d. -f3`
+
+# Minimal version = 1.17 (previous versions do not correctly unpack archives
+# containing hard-links if the --strip-components option is used).
+major_min=1
+minor_min=17
+if [ $major -gt $major_min ]; then
+	echo $tar
+else
+	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
+		echo $tar
+	else
+		false
+		# echo nothing: no suitable tar found
+	fi
+fi

Files package/lzma/lzmacheck.sh and toolchain/dependencies/check-host-lzma.sh are present since the very beginning of buildroot, but do not appear to be used (anymore). Let's remove them.

---
 package/lzma/lzmacheck.sh                 |   9 ---------
 toolchain/dependencies/check-host-lzma.sh |  13 -------------
 2 files changed, 0 insertions(+), 22 deletions(-)

deleted file mode 100755
@@ -1,9 +0,0 @@
-#!/bin/sh
-
-bin=$(toolchain/dependencies/check-host-lzma.sh)
-if [ "x$bin" = "x" ] ; then
-  echo build-lzma-host-binary
-else
-  echo use-lzma-host-binary
-fi
-

deleted file mode 100755
@@ -1,13 +0,0 @@
-#!/bin/sh
-
-ok=""
-
-for bin in /usr/bin/lzma $LZMA
-do
-# TODO: add check for proper functionality here..
-  $bin --version > /dev/null 2>&1 && ok="$bin"
-  if test "x$ok" != "x" ; then
-    break
-  fi
-done
-echo "$ok"

This patch attempts to solve two problems:
1. download directory not created before downloading
If the first download is handled by wget, then the download directory will be created by wget itself and all is fine. However, if the first download is handled by another tool, e.g. git/bzr/hg/scp etc., then there is no automatic creation of this directory.

In most cases, you just execute 'make', and the 'dirs' target will be executed automatically. But, if you give an explicit target the first time, e.g. 'make ctng-menuconfig', there is no direct dependency to 'dirs'

A similar problem was detected and fixed earlier: see
'source' target should depend on 'dirs'
http://git.buildroot.net/buildroot/commit/?id=4eb982cf90f16927442f4e7692ce51dd5d21f37b
In that case, the fix was made specifically for the 'source' target, but in fact the problem is generic.

2. dirs not created and dependencies not met before packages are built The second problem is new. If there are dependencies that are needed to build packages, e.g. a dependency to host-tar, then these should be executed before trying to build any other package. Moreover, the build directory etc. should also be made before building any package. This is not always the case.

For example, ctng-menuconfig depends on 'host-gawk'. If you run ctng-menuconfig directly, host-gawk will be built first, but the general 'dirs' and 'dependencies' targets will be skipped! This is problematic in the mentioned 'tar' case.
The same would occur if you explicitly execute 'make foo' as first command.

To solve both problems, this patch changes the gentargets 'foo-source' target to explicitly depend on 'dirs' and 'dependencies'. This way, every package built with gentargets/autotargets/cmake will be built after dirs and dependencies.

But, certain problems remain:
* When a dependency package is not using gentargets.
* In some cases, this patch introduces a circular make dependency, because e.g. dependencies <-- host-tar
<any package> <-- dirs dependencies

Although make gracefully drops this circular dependency when present, it is not nice to have. Maybe we need to treat the 'tar' case in a special manner. For example by adding only a dependency to host-tar (and not 'dependencies' in general) to the extraction step for all packages, except for host-tar itself. Or, alternatively, we may decide that we won't support direct calls to 'make ctng-menuconfig' or similar, without the user having called 'make dependencies' first.

With respect to the downloads problem, we may also want to change it
differently: either let the download step depend on DL_DIR, or have the download helpers create the download directory first.

Any thoughts are welcome!

---
 package/Makefile.package.in |  2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

@@ -558,7 +558,7 @@ ifeq ($$($(2)_OVERRIDE_SRCDIR),)
 
 $(1)-depends:		$$($(2)_DEPENDENCIES)
 
-$(1)-source:		$$($(2)_TARGET_SOURCE)
+$(1)-source:		dirs dependencies $$($(2)_TARGET_SOURCE)
 else
 # In the package override case, the sequence of steps
 #  source, by rsyncing

@@ -571,7 +571,7 @@ else
 
 $(1)-rsync:		$$($(2)_TARGET_RSYNC)
 
-$(1)-source:		$$($(2)_TARGET_RSYNC_SOURCE)
+$(1)-source:		dirs dependencies $$($(2)_TARGET_RSYNC_SOURCE)
 endif
 
 $(1)-show-depends:

While you're at it, perhaps move the toolchain/dependencies directory to support/dependencies? And also it can be included from the top-level Makefile instead of the toolchain makefiles. Not that this is in any way related to this patch :-)

[ I just realized that I agree with Thomas DS's conclusion that we should use Reviewed-by, not Acked-by... So my previous Acks should be Reviews.]

Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111115/738a076b/attachment-0001.html>

Since tar >= 1.17 is required, this can be removed.

# finalize the value of SYSTEM_TAR.

Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

On Thursday 10 November 2011 19:32:02 Thomas De Schampheleire wrote:
[snip]

packets not using gentargets are deprecated. AFAICS there are only 14 left:
cups/cups.mk
customize/customize.mk
fis/fis.mk
games/doom-wad/doom-wads.mk
gettext/gettext.mk
java/concierge/concierge.mk
microperl/microperl.mk
netkitbase/netkitbase.mk
netkittelnet/netkittelnet.mk
newt/newt.mk
tinyhttpd/tinyhttpd.mk
ttcp/ttcp.mk
uemacs/uemacs.mk
vpnc/vpnc.mk
xfsprogs/xfsprogs.mk

The only worrysome one is gettext.

This is of course much more important.

For dirs it certainly isn't a problem.

For the dependencies, I'm actually thinking that a much better solution
would be to add a dependencies step at the first build, that checks for all potentially needed host packages. It creates a hostpackages.mk file that contains variable assignments for all potential dependencies. Specifying a rule for hostpackages.mk in the top-level Makefile will make sure that it is generated if it doesn't exist. This has the advantage that dependencies have to be checked only at the first build and not every single time. But it is a lot more work to implement of course...

Regards,
Arnout

Arnout Vandecappelle                               arnout@mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43

Thomas> Currently, the dependencies target (that runs dependencies.sh) Thomas> depends on DEPENDENCIES_HOST_PREREQ. This means that the Thomas> dependencies listed in DEPENDENCIES_HOST_PREREQ (currently Thomas> host-sstrip if sstrip is selected) are built *before* the Thomas> dependencies.sh script is run.

Committed, thanks.

Bye, Peter Korsgaard

Thomas> Files package/lzma/lzmacheck.sh and
Thomas> toolchain/dependencies/check-host-lzma.sh are present since the Thomas> very beginning of buildroot, but do not appear to be used Thomas> (anymore). Let's remove them.

Committed, thanks.

Bye, Peter Korsgaard