Skip to contentSkip to conversation

dependencies: some improvements

PATCH Superseded Thomas De Schampheleire 4 participants 20 messages Nov 23, 2011 – Dec 10, 2011 1 message not in the archive Kbuild

v2 of 6 Superseded Latest is v6 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 dependencies: some improvements

The original series had the following patches:
1 dependencies: check core dependencies before anything else 2 dependencies: add function suitable-host-package
3 dependencies: build a host-tar if no suitable tar can be found 4 dependencies: remove unused lzma checking scripts
5 dirs and dependencies should be executed before every package

Patches 1 and 4 are already merged (thanks Peter).

Patch 5 tried to fix some non-default scenarios, like running 'make ctng-menuconfig'
without having run 'make dependencies' or plain 'make' first. But, this attempt mainly suffered from the introduction of circular dependencies, which are not easy to solve. I therefore decided not to support these scenarios in this series.

Patch 2 and 3 are updated in this series based on the input from Arnout (thanks!)

Additionally, a new patch moves toolchain/dependencies to support/dependencies.

#1 lore raw

PATCH v2 1/1 dependencies: add function suitable-host-package

Sometimes, buildroot needs a certain host tool to do its job, e.g. tar. In many cases, we expect this tool to be present on the host system, but this is not always the case. Or maybe, the version on the host system is not suitable, and we need a more recent one.

In some of these cases, instead of bailing out, buildroot could build the package first (but only if the existing system package is not suitable).

To aid in detecting if a host package is suitable or not, this patch adds a function suitable-host-package. When called with parameter foo, it will execute check-host-foo.sh. This script should return either the path to the suitable host package, or the empty string if no suitable package can be found. The rules to determine whether something is suitable or not is left to check-host-foo.sh and depends on foo.

An example usage of suitable-host-package is:
DEPENDENCIES_HOST_PREREQ += $(if $(call suitable-host-package,foo),,host-foo)

To avoid cluttering the existing dependencies.mk file, it includes any check-host-foo.mk file. These files can be used to hold appropriate dependency-related actions for foo.

---
 toolchain/dependencies/dependencies.mk |  9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)
toolchain/dependencies/dependencies.mk +9 -0
@@ -6,6 +6,15 @@
 ######################################################################
 
 DEPENDENCIES_HOST_PREREQ:=
+
+# suitable-host-pkg: calls check-host-$(1).sh shell script, which
+# should return the path to the suitable host tool, or nothing if no
+# suitable tool was found.
+define suitable-host-package
+$(shell toolchain/dependencies/check-host-$(1).sh)
+endef
+include toolchain/dependencies/check-host-*.mk
+
 ifeq ($(BR2_STRIP_sstrip),y)
 DEPENDENCIES_HOST_PREREQ+=host-sstrip
 endif
#2 lore raw

PATCH v2 1/1 dependencies: add function suitable-host-package

toolchain/dependencies/dependencies.mk line 15

Quoted 6 lines · On Wednesday 23 November 2011 14:59:50 Thomas De Schampheleire wrote:
toolchain/dependencies/dependencies.mk +6 -0
+# suitable-host-pkg: calls check-host-$(1).sh shell script, which
+# should return the path to the suitable host tool, or nothing if no
+# suitable tool was found.
+define suitable-host-package
+$(shell toolchain/dependencies/check-host-$(1).sh)
+endef

I should have noticed this in the first review, but: If the function is
this simple, it becomes pointless. Especially because other check-host-* mk files may want to implement the checking in a different way, e.g. $(eval $(shell ...)).

Quoted 1 line

+include toolchain/dependencies/check-host-*.mk

And if this is the only thing remaining, perhaps it's better to merge
this patch with the next one :-)

Regards,
Arnout

Signature · 9 lines
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111206/4d32f6c6/attachment.html>
#3 lore raw

PATCH v2 1/1 dependencies: add function suitable-host-package

Hi Arnout,

Quoted 15 lines · Arnout Vandecappelle

toolchain/dependencies/dependencies.mk line 15

Quoted 11 lines · On Wednesday 23 November 2011 14:59:50 Thomas De Schampheleire wrote:
toolchain/dependencies/dependencies.mk +6 -0
+# suitable-host-pkg: calls check-host-$(1).sh shell script, which
 
+# should return the path to the suitable host tool, or nothing if no
 
+# suitable tool was found.
 
+define suitable-host-package
 
+$(shell toolchain/dependencies/check-host-$(1).sh)
 
+endef
Quoted 7 lines

I should have noticed this in the first review, but: If the function is

this simple, it becomes pointless. Especially because other check-host-*

mk files may want to implement the checking in a different way, e.g.

$(eval $(shell ...)).

I'm not sure if it's pointless. It provides a uniform way of checking for host packages, and in that sense becomes a sort of 'infrastructure'. Moreover, the syntax is simpler and does not require knowing the path to the correct check-host file. If anything changes, it only needs to change in one place.

Compare:
$(if $(call suitable-host-package,foo),,host-foo)
with
$(if $(shell toolchain/dependencies/check-host-foo.sh),,host-foo)

Quoted 11 lines
Quoted 3 lines

+include toolchain/dependencies/check-host-*.mk

And if this is the only thing remaining, perhaps it's better to merge

this patch with the next one :-)

Best regards,
Thomas

#4 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

In reply to Thomas De Schampheleire

Some toolchains, like the one built with buildroot itself, use hardlinks (for example to link between the c++ and g++ binary). Unpacking such a toolchain with the --strip-components options does not work correctly if the system tar is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15.

This patch checks for a suitable tar version (tar 1.17+) on the host system, and adds host-tar to the host dependencies if none can be found.

TAR is redefined to HOST_TAR, except when extracting host-tar (this is a chicken-and-egg problem), so that all packages use the host-tar if no suitable tar was found.

---
 package/Makefile.package.in              |  12 ------------
 package/tar/tar.mk                       |   6 ++++++
 toolchain/dependencies/check-host-tar.mk |  32 ++++++++++++++++++++++++++++++++
 toolchain/dependencies/check-host-tar.sh |  29 +++++++++++++++++++++++++++++
 4 files changed, 67 insertions(+), 12 deletions(-)
package/Makefile.package.in +0 -12
@@ -235,18 +235,6 @@ define DOWNLOAD
 	exit 1
 endef
 
-# Utility programs used to build packages
-TAR ?= tar
-
-# Automatically detect tar --strip-path/components option
-TAR_STRIP_COMPONENTS := \
-  $(shell $(TAR) --help | grep strip-path > /dev/null ; \
-  if test $$? = 0 ; then \
-   echo '--strip-path' ; \
-  else \
-   echo '--strip-components' ; \
-  fi)
-
 # Needed for the foreach loops to loop over the list of hooks, so that
 # each hook call is properly separated by a newline.
 define sep
package/tar/tar.mk +6 -0
@@ -13,3 +13,9 @@ TAR_DEPENDENCIES += busybox
 endif
 
 $(eval $(call AUTOTARGETS))
+
+TAR = $(SYSTEM_TAR) # We need a real tar to extract tar
+TAR_STRIP_COMPONENTS = $(SYSTEM_TAR_STRIP_COMPONENTS)
+$(eval $(call AUTOTARGETS,host))
+TAR = $(HOST_TAR)
+TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
toolchain/dependencies/check-host-tar.mk new file +32 -0
new file mode 100644
@@ -0,0 +1,32 @@
+TAR ?= tar
+
+# To extract tar, we always need a real (system) tar. Provide the necessary
+# definitions here to avoid cluttering package/tar/tar.mk .
+# TAR will be overridden again when the tar .mk file is included, so we must
+# finalize the value of SYSTEM_TAR by using :=
+SYSTEM_TAR := $(TAR)
+# Automatically detect tar --strip-path/components option
+# --strip-path was renamed to --strip-components in tar 1.15
+SYSTEM_TAR_STRIP_COMPONENTS := \
+    $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
+    if test $$? = 0 ; then \
+     echo '--strip-path' ; \
+    else \
+     echo '--strip-components' ; \
+    fi)
+
+# Determine correct HOST_TAR
+ifneq (,$(call suitable-host-package,tar))
+  HOST_TAR = $(SYSTEM_TAR)
+else
+  DEPENDENCIES_HOST_PREREQ += host-tar
+  HOST_TAR = $(HOST_DIR)/usr/bin/tar
+endif
+
+# Since HOST_TAR is at least 1.17, it will certainly support --strip-components
+HOST_TAR_STRIP_COMPONENTS = --strip-components
+
+# Set definitions used by all .mk files included after this one and before
+# package/tar/tar.mk
+TAR = $(HOST_TAR)
+TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
toolchain/dependencies/check-host-tar.sh new file +29 -0
new file mode 100755
@@ -0,0 +1,29 @@
+#!/bin/sh
+
+tar=`which tar`
+if [ -z "$tar" ]; then
+	return
+fi
+
+# Output of 'tar --version' examples:
+# tar (GNU tar) 1.15.1 
+# tar (GNU tar) 1.25
+version=`$tar --version | head -n 1 | sed 's/^.*\s\([0-9]\+\.\S\+\).*$/\1/'`
+major=`echo "$version" | cut -d. -f1`
+minor=`echo "$version" | cut -d. -f2`
+bugfix=`echo "$version" | cut -d. -f3`
+
+# Minimal version = 1.17 (previous versions do not correctly unpack archives
+# containing hard-links if the --strip-components option is used).
+major_min=1
+minor_min=17
+if [ $major -gt $major_min ]; then
+	echo $tar
+else
+	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
+		echo $tar
+	else
+		false
+		# echo nothing: no suitable tar found
+	fi
+fi
#5 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

Le Wed, 23 Nov 2011 14:59:59 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :

Quoted 6 lines

Some toolchains, like the one built with buildroot itself, use hardlinks (for example to link between the c++ and g++ binary). Unpacking such a toolchain with the --strip-components options does not work correctly if the system tar is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15.

Quoted 7 lines

This patch checks for a suitable tar version (tar 1.17+) on the host system, and adds host-tar to the host dependencies if none can be found.

TAR is redefined to HOST_TAR, except when extracting host-tar (this is a chicken-and-egg problem), so that all packages use the host-tar if no suitable tar was found.

I am still puzzled by the fact that tar is needed to extract this host-tar, so there's like a chicken-and-egg problem. I understand that the real host tar is used to extract the host-tar, which is then use to extract all other packages. But that sounds really nasty to me.

Thomas

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#6 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

Quoted 13 lines · Thomas Petazzoni

Le Wed, 23 Nov 2011 14:59:59 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :

Quoted 10 lines

Some toolchains, like the one built with buildroot itself, use hardlinks (for example to link between the c++ and g++ binary). Unpacking such a toolchain with the --strip-components options does not work correctly if the system tar is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15.

This patch checks for a suitable tar version (tar 1.17+) on the host system, and adds host-tar to the host dependencies if none can be found.

Quoted 9 lines
Quoted 3 lines

TAR is redefined to HOST_TAR, except when extracting host-tar (this is a chicken-and-egg problem), so that all packages use the host-tar if no suitable tar was found.

I am still puzzled by the fact that tar is needed to extract this host-tar, so there's like a chicken-and-egg problem. I understand that the real host tar is used to extract the host-tar, which is then use to extract all other packages. But that sounds really nasty to me.

Maybe:
ftp://ftp.gnu.org/gnu/tar/tar-1.26.cpio.gz

Then you only need the more stable cpio and gzip utilities. Even Busybox can provide those.

Mike

Quoted 1 line

Thomas

#7 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

In reply to Thomas Petazzoni
Quoted 13 lines · Thomas Petazzoni

Le Wed, 23 Nov 2011 14:59:59 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :

Quoted 10 lines

Some toolchains, like the one built with buildroot itself, use hardlinks (for example to link between the c++ and g++ binary). Unpacking such a toolchain with the --strip-components options does not work correctly if the system tar is too old (<1.17). Even recent releases of RedHat/CentOS still ship with tar 1.15.

This patch checks for a suitable tar version (tar 1.17+) on the host system, and adds host-tar to the host dependencies if none can be found.

Quoted 8 lines
Quoted 3 lines

TAR is redefined to HOST_TAR, except when extracting host-tar (this is a chicken-and-egg problem), so that all packages use the host-tar if no suitable tar was found.

I am still puzzled by the fact that tar is needed to extract this host-tar, so there's like a chicken-and-egg problem. I understand that the real host tar is used to extract the host-tar, which is then use to extract all other packages. But that sounds really nasty to me.

To summarize the problem about tar:
* some distributions (like RedHat/CentOS) still ship an old tar version (1.15) which is known to have problems when --tar-strip-components is used on tarballs with hardlinks.
* the problem was first seen when tar'ing up a buildroot toolchain and use it as an external toolchain later.
* since the problem is not restricted to toolchains, we decided to fix the problem in a common way, by redefining TAR.

I see several solutions, some more desirable than others:
1. don't change anything and force users of RedHat/CentOS to build their own tar that is recent enough. This is by far the easiest from a buildroot perspective, but least user-friendly.
2. use the proposed patch
2. don't download tar as .tar.gz/.tar.bz2, but rather as .cpio.gz (available on the official mirrors since 1.16.1). It requires cpio and gzip to be present, but that is already checked in dependencies.sh. We'd need to update gentargets to cater for .cpio, but that is doable.

Best regards,
Thomas

#8 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

In reply to Thomas Petazzoni
Quoted 8 lines · On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
Quoted 3 lines

TAR is redefined to HOST_TAR, except when extracting host-tar (this is a chicken-and-egg problem), so that all packages use the host-tar if no suitable tar was found.

I am still puzzled by the fact that tar is needed to extract this host-tar, so there's like a chicken-and-egg problem. I understand that the real host tar is used to extract the host-tar, which is then use to extract all other packages. But that sounds really nasty to me.

It's not so strange, it's like a compiler's canadian build. You need a
compiler to compile the compiler, right?

The way it is solved for tar is OK for me. I think adding support for
cpio archives would just complicate things, because then host-tar can't use GENTARGETS anymore.

Regards,
Arnout

Signature · 9 lines
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111206/d726b617/attachment-0001.html>
#9 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

Quoted 21 lines · Arnout Vandecappelle
Quoted 8 lines · On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
Quoted 5 lines

TAR is redefined to HOST_TAR, except when extracting host-tar (this

is a chicken-and-egg problem), so that all packages use the host-tar

if no suitable tar was found.

Quoted 7 lines

I am still puzzled by the fact that tar is needed to extract this

host-tar, so there's like a chicken-and-egg problem. I understand that

the real host tar is used to extract the host-tar, which is then use to

extract all other packages. But that sounds really nasty to me.

It's not so strange, it's like a compiler's canadian build. You need a

Quoted 8 lines

compiler to compile the compiler, right?

The way it is solved for tar is OK for me. I think adding support for

cpio archives would just complicate things, because then host-tar

can't use GENTARGETS anymore.

I don't think that's true. I started adding cpio support to gentargets. Essentially, here is the code that needs modification:

# Define extractors for different archive suffixes

INFLATE.bz2  = $(BZCAT)
INFLATE.gz   = $(ZCAT)
INFLATE.tbz  = $(BZCAT)
INFLATE.tbz2 = $(BZCAT)
INFLATE.tgz  = $(ZCAT)
INFLATE.xz   = $(XZCAT)
INFLATE.tar  = cat
$(2)_EXTRACT_CMDS ?= \
        $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
$(DL_DIR)/$$($(2)_SOURCE) | \
        $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)

Although the inflate step already handles multiple suffices, the extract does not yet.

My current approach is this:
+INFLATE.cpio = cat
+
+EXTRACT.tar = $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(1) $(TAR_OPTIONS) - +EXTRACT.cpio = ( mkdir -p $(1) && cd $(1) && cpio -i )

 $(2)_EXTRACT_CMDS ?= \
        $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))

$(DL_DIR)/$$($(2)_SOURCE) | \
- $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
+ $$(if $$(EXTRACT$$(suffix $$($(2)_SOURCE))),$$(call
EXTRACT$$(suffix $$($(2)_SOURCE)),$$($(2)_DIR)),$$(call EXTRACT$$(suffix $$(basename $$($(2)_SOURCE))),$$($(2)_DIR))))

The only thing I'm struggling with is that cpio does not have a --strip-components functionality. As a result, the package is extracted in output/build/host-tar-1.26/tar-1.26 which is not right. I don't directly know the name 'tar-1.26', because all I have is the string 'host-tar-1.26'. I see two approaches:
* either try to strip of 'host' from $(2)_DIR
* or do something like 'mv * .' or 'find -maxdepth 1 -type d | xargs mv {} .'

Suggestions are much appreciated...

Thanks,
Thomas

#10 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

Quoted 67 lines · Thomas De Schampheleire
Quoted 21 lines · Arnout Vandecappelle
Quoted 8 lines · On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
Quoted 5 lines

TAR is redefined to HOST_TAR, except when extracting host-tar (this

is a chicken-and-egg problem), so that all packages use the host-tar

if no suitable tar was found.

Quoted 7 lines

I am still puzzled by the fact that tar is needed to extract this

host-tar, so there's like a chicken-and-egg problem. I understand that

the real host tar is used to extract the host-tar, which is then use to

extract all other packages. But that sounds really nasty to me.

It's not so strange, it's like a compiler's canadian build. You need a

Quoted 8 lines

compiler to compile the compiler, right?

The way it is solved for tar is OK for me. I think adding support for

cpio archives would just complicate things, because then host-tar

can't use GENTARGETS anymore.

I don't think that's true. I started adding cpio support to gentargets. Essentially, here is the code that needs modification:

# Define extractors for different archive suffixes

INFLATE.bz2  = $(BZCAT)
INFLATE.gz   = $(ZCAT)
INFLATE.tbz  = $(BZCAT)
INFLATE.tbz2 = $(BZCAT)
INFLATE.tgz  = $(ZCAT)
INFLATE.xz   = $(XZCAT)
INFLATE.tar  = cat
$(2)_EXTRACT_CMDS ?= \
        $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
$(DL_DIR)/$$($(2)_SOURCE) | \
        $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)

Although the inflate step already handles multiple suffices, the extract does not yet.

My current approach is this:

(quoted diff) +4 -0
+INFLATE.cpio = cat
+
+EXTRACT.tar = $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(1) $(TAR_OPTIONS) -
+EXTRACT.cpio = ( mkdir -p $(1) && cd $(1) && cpio -i )
 
 $(2)_EXTRACT_CMDS ?= \
        $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))

$(DL_DIR)/$$($(2)_SOURCE) | \

(quoted diff) +1 -1
-       $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
+       $$(if $$(EXTRACT$$(suffix $$($(2)_SOURCE))),$$(call

EXTRACT$$(suffix $$($(2)_SOURCE)),$$($(2)_DIR)),$$(call EXTRACT$$(suffix $$(basename $$($(2)_SOURCE))),$$($(2)_DIR))))

Quoted 3 lines

The only thing I'm struggling with is that cpio does not have a --strip-components functionality.

Would: "--no-absolute-filenames" help on the cpio command?

If this will become a general purpose extract.cpio command, it should have that option even if it doesn't help in this case.

Many cpio archives do have absolute paths (like the initramFS archive) even if the GNU-TAR cpio archive does not.

Mike

Quoted 17 lines

As a result, the package is
extracted in output/build/host-tar-1.26/tar-1.26 which is not right. I don't directly know the name 'tar-1.26', because all I have is the string 'host-tar-1.26'. I see two approaches:
* either try to strip of 'host' from $(2)_DIR
* or do something like 'mv * .' or 'find -maxdepth 1 -type d | xargs mv {} .'

Suggestions are much appreciated...

Thanks,
Thomas
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#11 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

Quoted 50 lines · Michael S. Zick
Quoted 49 lines · Thomas De Schampheleire
Quoted 21 lines · Arnout Vandecappelle
Quoted 8 lines · On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
Quoted 5 lines

TAR is redefined to HOST_TAR, except when extracting host-tar (this

is a chicken-and-egg problem), so that all packages use the host-tar

if no suitable tar was found.

Quoted 7 lines

I am still puzzled by the fact that tar is needed to extract this

host-tar, so there's like a chicken-and-egg problem. I understand that

the real host tar is used to extract the host-tar, which is then use to

extract all other packages. But that sounds really nasty to me.

It's not so strange, it's like a compiler's canadian build. You need a

Quoted 8 lines

compiler to compile the compiler, right?

The way it is solved for tar is OK for me. I think adding support for

cpio archives would just complicate things, because then host-tar

can't use GENTARGETS anymore.

I don't think that's true. I started adding cpio support to gentargets. Essentially, here is the code that needs modification:

# Define extractors for different archive suffixes

INFLATE.bz2  = $(BZCAT)
INFLATE.gz   = $(ZCAT)
INFLATE.tbz  = $(BZCAT)
INFLATE.tbz2 = $(BZCAT)
INFLATE.tgz  = $(ZCAT)
INFLATE.xz   = $(XZCAT)
INFLATE.tar  = cat
$(2)_EXTRACT_CMDS ?= \
        $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))
$(DL_DIR)/$$($(2)_SOURCE) | \
        $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
Quoted 3 lines
Quoted 3 lines

Although the inflate step already handles multiple suffices, the extract does not yet.

Is the inflate step being used for tar archives?

The reason I ask is because GNU-tar (and BSD-tar) will auto-detect the compression method from the file and un-archive it without a prior de-compression.

In fact, if the "host tar" is BSD-tar, then you can unpack *.iso9660 as if it was an archive.

Mike

Quoted 53 lines
Quoted 15 lines

My current approach is this:

(quoted diff) +4 -0
+INFLATE.cpio = cat
+
+EXTRACT.tar = $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(1) $(TAR_OPTIONS) -
+EXTRACT.cpio = ( mkdir -p $(1) && cd $(1) && cpio -i )
 
 $(2)_EXTRACT_CMDS ?= \
        $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE)))

$(DL_DIR)/$$($(2)_SOURCE) | \

(quoted diff) +1 -1
-       $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
+       $$(if $$(EXTRACT$$(suffix $$($(2)_SOURCE))),$$(call

EXTRACT$$(suffix $$($(2)_SOURCE)),$$($(2)_DIR)),$$(call EXTRACT$$(suffix $$(basename $$($(2)_SOURCE))),$$($(2)_DIR))))

Quoted 3 lines

The only thing I'm struggling with is that cpio does not have a --strip-components functionality.

Would: "--no-absolute-filenames" help on the cpio command?

If this will become a general purpose extract.cpio command, it should have that option even if it doesn't help in this case.

Many cpio archives do have absolute paths (like the initramFS archive) even if the GNU-TAR cpio archive does not.

Mike

Quoted 7 lines

As a result, the package is
extracted in output/build/host-tar-1.26/tar-1.26 which is not right. I don't directly know the name 'tar-1.26', because all I have is the string 'host-tar-1.26'. I see two approaches:
* either try to strip of 'host' from $(2)_DIR
* or do something like 'mv * .' or 'find -maxdepth 1 -type d | xargs mv {} .'

Quoted 10 lines

Suggestions are much appreciated...

Thanks,
Thomas
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#12 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

Quoted 55 lines · Michael S. Zick
Quoted 50 lines · Michael S. Zick
Quoted 49 lines · Thomas De Schampheleire
Quoted 21 lines · Arnout Vandecappelle
Quoted 8 lines · On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
Quoted 5 lines

TAR is redefined to HOST_TAR, except when extracting host-tar (this

is a chicken-and-egg problem), so that all packages use the host-tar

if no suitable tar was found.

Quoted 7 lines

I am still puzzled by the fact that tar is needed to extract this

host-tar, so there's like a chicken-and-egg problem. I understand that

the real host tar is used to extract the host-tar, which is then use to

extract all other packages. But that sounds really nasty to me.

It's not so strange, it's like a compiler's canadian build. You need a

Quoted 8 lines

compiler to compile the compiler, right?

The way it is solved for tar is OK for me. I think adding support for

cpio archives would just complicate things, because then host-tar

can't use GENTARGETS anymore.

I don't think that's true. I started adding cpio support to gentargets. Essentially, here is the code that needs modification:

# Define extractors for different archive suffixes INFLATE.bz2 ?= $(BZCAT)
INFLATE.gz ? = $(ZCAT)
INFLATE.tbz ?= $(BZCAT)
INFLATE.tbz2 = $(BZCAT)
INFLATE.tgz ?= $(ZCAT)
INFLATE.xz ? = $(XZCAT)
INFLATE.tar ?= cat

$(2)_EXTRACT_CMDS ?= \
? ? ? ? $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $(DL_DIR)/$$($(2)_SOURCE) | \
? ? ? ? $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)

Quoted 3 lines
Quoted 3 lines

Although the inflate step already handles multiple suffices, the extract does not yet.

Quoted 6 lines

Is the inflate step being used for tar archives?

The reason I ask is because GNU-tar (and BSD-tar) will auto-detect the compression method from the file and un-archive it without a prior de-compression.

Yes, the inflate step is used for tar archives.
Suppose you have a file called foo.tar.gz, then the commands executed will be similar to: (left out some details)
zcat foo.tar.gz | tar -xf

For foo.tar.bz2, this will be:
bzcat foo.tar.bz2 | tar -xf

The tar step does not have to bother with compression.

Quoted 63 lines

In fact, if the "host tar" is BSD-tar, then you can unpack *.iso9660 as if it was an archive.

Mike

Quoted 47 lines
Quoted 15 lines

My current approach is this:

(quoted diff) +4 -0
+INFLATE.cpio = cat
+
+EXTRACT.tar = $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $(1) $(TAR_OPTIONS) -
+EXTRACT.cpio = ( mkdir -p $(1) && cd $(1) && cpio -i )

?$(2)_EXTRACT_CMDS ?= \
? ? ? ? $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $(DL_DIR)/$$($(2)_SOURCE) | \
- ? ? ? $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)
+ ? ? ? $$(if $$(EXTRACT$$(suffix $$($(2)_SOURCE))),$$(call EXTRACT$$(suffix $$($(2)_SOURCE)),$$($(2)_DIR)),$$(call EXTRACT$$(suffix $$(basename $$($(2)_SOURCE))),$$($(2)_DIR))))

Quoted 3 lines

The only thing I'm struggling with is that cpio does not have a --strip-components functionality.

Would: "--no-absolute-filenames" help on the cpio command?

If this will become a general purpose extract.cpio command, it should have that option even if it doesn't help in this case.

Many cpio archives do have absolute paths (like the initramFS archive) even if the GNU-TAR cpio archive does not.

Mike

Quoted 7 lines

As a result, the package is
extracted in output/build/host-tar-1.26/tar-1.26 which is not right. I don't directly know the name 'tar-1.26', because all I have is the string 'host-tar-1.26'. I see two approaches:
* either try to strip of 'host' from $(2)_DIR
* or do something like 'mv * .' or 'find -maxdepth 1 -type d | xargs mv {} .'

Quoted 10 lines

Suggestions are much appreciated...

Thanks,
Thomas
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Quoted 6 lines

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

A message that is not in the archive (sent off-list or not yet received): [email protected]

#14 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

Quoted 93 lines · Michael S. Zick
Quoted 56 lines · you
Quoted 55 lines · Michael S. Zick
Quoted 53 lines · Michael S. Zick
Quoted 52 lines · Thomas De Schampheleire
Quoted 21 lines · Arnout Vandecappelle
Quoted 8 lines · On Wednesday 23 November 2011 15:24:40 Thomas Petazzoni wrote:
Quoted 5 lines

TAR is redefined to HOST_TAR, except when extracting host-tar (this

is a chicken-and-egg problem), so that all packages use the host-tar

if no suitable tar was found.

Quoted 7 lines

I am still puzzled by the fact that tar is needed to extract this

host-tar, so there's like a chicken-and-egg problem. I understand that

the real host tar is used to extract the host-tar, which is then use to

extract all other packages. But that sounds really nasty to me.

It's not so strange, it's like a compiler's canadian build. You need a

Quoted 8 lines

compiler to compile the compiler, right?

The way it is solved for tar is OK for me. I think adding support for

cpio archives would just complicate things, because then host-tar

can't use GENTARGETS anymore.

I don't think that's true. I started adding cpio support to gentargets. Essentially, here is the code that needs modification:

# Define extractors for different archive suffixes INFLATE.bz2 ?= $(BZCAT)
INFLATE.gz ? = $(ZCAT)
INFLATE.tbz ?= $(BZCAT)
INFLATE.tbz2 = $(BZCAT)
INFLATE.tgz ?= $(ZCAT)
INFLATE.xz ? = $(XZCAT)
INFLATE.tar ?= cat

$(2)_EXTRACT_CMDS ?= \
? ? ? ? $$(if $$($(2)_SOURCE),$$(INFLATE$$(suffix $$($(2)_SOURCE))) $(DL_DIR)/$$($(2)_SOURCE) | \
? ? ? ? $(TAR) $(TAR_STRIP_COMPONENTS)=1 -C $$($(2)_DIR) $(TAR_OPTIONS) -)

Although the inflate step already handles multiple suffices, the extract does not yet.

Quoted 9 lines
Quoted 6 lines

Is the inflate step being used for tar archives?

The reason I ask is because GNU-tar (and BSD-tar) will auto-detect the compression method from the file and un-archive it without a prior de-compression.

Yes, the inflate step is used for tar archives.

My point was, it does not need to.
The INFLATE.* methods need only be available for use by things other than gnu-tar.

Without leaving out any of the details (except the src/dest ls check):

mszick at wolf466:~$ tar --create --file=foo.tar.gz *.txt

mszick at wolf466:~$ ls -l *.tar.gz
-rw-r--r-- 1 mszick mszick 501760 2011-12-09 08:02 foo.tar.gz

mszick at wolf466:~$ mkdir tmp
mszick at wolf466:~$ cd tmp

mszick at wolf466:~/tmp$ tar --extract --file=../foo.tar.gz

mszick at wolf466:~/tmp$ ls
- - - shows the expected files - - -

mszick at wolf466:~/tmp$ tar --version
tar (GNU tar) 1.20
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.

Quoted 2 lines

Which is a pretty old gnu-tar.
I think the auto-detect went in around 1.17 or so.

Thanks for the tip. I wasn't aware of this.

As long as tar supports all compression methods buildroot supports (currently .gz, .bz2 and .xz) then we should be able to remove the inflate step, I guess. I'll try to make a patch for this.

It doesn't help the cpio unpacking though, so it is more or less an independent patch.

Best regards,
Thomas

#15 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

In reply to Thomas De Schampheleire
Quoted 11 lines · On Wednesday 23 November 2011 14:59:59 Thomas De Schampheleire wrote:
toolchain/dependencies/check-host-tar.mk new file +6 -0
new file mode 100644
@@ -0,0 +1,32 @@
+TAR ?= tar
+
+# To extract tar, we always need a real (system) tar. Provide the necessary
+# definitions here to avoid cluttering package/tar/tar.mk .
+# TAR will be overridden again when the tar .mk file is included, so we must
+# finalize the value of SYSTEM_TAR by using :=

toolchain/dependencies/check-host-tar.mk line 16

Quoted 10 lines
toolchain/dependencies/check-host-tar.mk new file +10 -0
+SYSTEM_TAR := $(TAR)
+# Automatically detect tar --strip-path/components option
+# --strip-path was renamed to --strip-components in tar 1.15
+SYSTEM_TAR_STRIP_COMPONENTS := \
+    $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
+    if test $$? = 0 ; then \
+     echo '--strip-path' ; \
+    else \
+     echo '--strip-components' ; \
+    fi)

This can move inside the ifneq below, with a
SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS) in the else (and moving that one up of course).

Quoted 21 lines
toolchain/dependencies/check-host-tar.mk new file +16 -0
+
+# Determine correct HOST_TAR
+ifneq (,$(call suitable-host-package,tar))
+  HOST_TAR = $(SYSTEM_TAR)
+else
+  DEPENDENCIES_HOST_PREREQ += host-tar
+  HOST_TAR = $(HOST_DIR)/usr/bin/tar
+endif
+
+# Since HOST_TAR is at least 1.17, it will certainly support --strip-components
+HOST_TAR_STRIP_COMPONENTS = --strip-components
+
+# Set definitions used by all .mk files included after this one and before
+# package/tar/tar.mk
+TAR = $(HOST_TAR)
+TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
toolchain/dependencies/check-host-tar.sh new file +0 -0
new file mode 100755

toolchain/dependencies/check-host-tar.sh line 3

Quoted 3 lines
toolchain/dependencies/check-host-tar.sh new file +3 -0
@@ -0,0 +1,29 @@
+#!/bin/sh
+
+tar=`which tar`

This contradicts the TAR ?= tar in check-host-tar.mk: if the user provided
their own tar version, it isn't checked and host-tar will still be built... I guess $TAR should be given as an argument to this script.

toolchain/dependencies/check-host-tar.sh line 6

Quoted 3 lines
toolchain/dependencies/check-host-tar.sh new file +3 -0
+if [ -z "$tar" ]; then
+	return
+fi

This could be a [ ! -x "$tar" ]
and it should be exit 1, not return.

Regards,
Arnout

Quoted 25 lines
toolchain/dependencies/check-host-tar.sh new file +23 -0
+
+# Output of 'tar --version' examples:
+# tar (GNU tar) 1.15.1 
+# tar (GNU tar) 1.25
+version=`$tar --version | head -n 1 | sed 's/^.*\s\([0-9]\+\.\S\+\).*$/\1/'`
+major=`echo "$version" | cut -d. -f1`
+minor=`echo "$version" | cut -d. -f2`
+bugfix=`echo "$version" | cut -d. -f3`
+
+# Minimal version = 1.17 (previous versions do not correctly unpack archives
+# containing hard-links if the --strip-components option is used).
+major_min=1
+minor_min=17
+if [ $major -gt $major_min ]; then
+	echo $tar
+else
+	if [ $major -eq $major_min -a $minor -ge $minor_min ]; then
+		echo $tar
+	else
+		false
+		# echo nothing: no suitable tar found
+	fi
+fi
Signature · 9 lines
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111206/4fce0ba6/attachment.html>
#16 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

Hi Arnout,

(First of all: your sending mails in both plaintext and html, messes up my Gmail sending: extra white lines are added between each line. Am I the only one with this problem? Does anyone know a fix? (one fix would be to send pure plain text to this list, but I don't know if that's an option for you)

Quoted 45 lines · Arnout Vandecappelle
Quoted 42 lines · On Wednesday 23 November 2011 14:59:59 Thomas De Schampheleire wrote:
toolchain/dependencies/check-host-tar.mk +0 -0

new file mode 100644

--- /dev/null

+++ b/toolchain/dependencies/check-host-tar.mk

toolchain/dependencies/check-host-tar.mk +3 -0
@@ -0,0 +1,32 @@
 
+TAR ?= tar
 
+
 
+# To extract tar, we always need a real (system) tar. Provide the

necessary

toolchain/dependencies/check-host-tar.mk +2 -0
+# definitions here to avoid cluttering package/tar/tar.mk .
 
+# TAR will be overridden again when the tar .mk file is included, so we

must

toolchain/dependencies/check-host-tar.mk +10 -0
+# finalize the value of SYSTEM_TAR by using :=
 
+SYSTEM_TAR := $(TAR)
 
+# Automatically detect tar --strip-path/components option
 
+# --strip-path was renamed to --strip-components in tar 1.15
 
+SYSTEM_TAR_STRIP_COMPONENTS := \
 
+ $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
 
+ if test $$? = 0 ; then \
 
+ echo '--strip-path' ; \
 
+ else \
 
+ echo '--strip-components' ; \
Quoted 9 lines
Quoted 1 line

+ fi)

This can move inside the ifneq below, with a

SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS)

in the else (and moving that one up of course).

I don't think this is true.
For the chicken-egg problem in the current patch, you need SYSTEM_TAR (a real one) to build host-tar if needed. So in the 'else' case, you need the full SYSTEM_TAR_STRIP_COMPONENTS.

You could do the following: (maybe this is what you meant)

HOST_TAR_STRIP_COMPONENTS = --strip-components ifneq (,$(call suitable-host-package,tar))
HOST_TAR = $(SYSTEM_TAR) SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS)
else
DEPENDENCIES_HOST_PREREQ += host-tar
HOST_TAR = $(HOST_DIR)/usr/bin/tar

  SYSTEM_TAR_STRIP_COMPONENTS := \
    $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
    if test $$? = 0 ; then \
     echo '--strip-path' ; \
    else \
     echo '--strip-components' ; \
    fi)

endif

But to be honest, I don't think we'll need any of this complexity anymore with the cpio approach (see next mail).

Quoted 50 lines
Quoted 49 lines
toolchain/dependencies/check-host-tar.mk +13 -1
+
 
+# Determine correct HOST_TAR
 
+ifneq (,$(call suitable-host-package,tar))
 
+ HOST_TAR = $(SYSTEM_TAR)
 
+else
 
+ DEPENDENCIES_HOST_PREREQ += host-tar
 
+ HOST_TAR = $(HOST_DIR)/usr/bin/tar
 
+endif
 
+
 
+# Since HOST_TAR is at least 1.17, it will certainly support
--strip-components
 
+HOST_TAR_STRIP_COMPONENTS = --strip-components
 
+
 
+# Set definitions used by all .mk files included after this one and

before

toolchain/dependencies/check-host-tar.mk +3 -0
+# package/tar/tar.mk
 
+TAR = $(HOST_TAR)
 
+TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
toolchain/dependencies/check-host-tar.sh +0 -0

new file mode 100755

--- /dev/null

+++ b/toolchain/dependencies/check-host-tar.sh

toolchain/dependencies/check-host-tar.sh +2 -0
@@ -0,0 +1,29 @@
 
+#!/bin/sh
 
+
Quoted 8 lines
Quoted 1 line

+tar=which tar

This contradicts the TAR ?= tar in check-host-tar.mk: if the user provided

their own tar version, it isn't checked and host-tar will still be built...

I guess $TAR should be given as an argument to this script.

You're right, I missed this one.

Quoted 10 lines

toolchain/dependencies/check-host-tar.sh line 6

Quoted 7 lines
toolchain/dependencies/check-host-tar.sh +3 -0
+if [ -z "$tar" ]; then
 
+ return
 
+fi

This could be a [ ! -x "$tar" ]

ok

Quoted 2 lines

and it should be exit 1, not return.

Why? The convention is that the check-host-foo.sh script should give as output either nothing (no suitable package found) or the path to the suitable package. The return code is not checked.
Of course, it wouldn't hurt, but it's not needed for the implementation.

Thanks for your comments.

Best regards,
Thomas

#17 lore raw

PATCH v2 1/1 dependencies: build a host-tar if no suitable tar can be found

Quoted 8 lines · Thomas De Schampheleire

Hi Arnout,

(First of all: your sending mails in both plaintext and html, messes up my Gmail sending: extra white lines are added between each line. Am I the only one with this problem? Does anyone know a fix? (one fix would be to send pure plain text to this list, but I don't know if that's an option for you)

I see the same thing here.
He is using Kmail for an e-mail client, so "plain text only" is an option of his e-mail client.
(At least the version of Kmail I am sending this with has the option.)

Mike

Quoted 175 lines · Arnout Vandecappelle
Quoted 45 lines · On Wednesday 23 November 2011 14:59:59 Thomas De Schampheleire wrote:
Quoted 43 lines
toolchain/dependencies/check-host-tar.mk +0 -0

new file mode 100644

--- /dev/null

+++ b/toolchain/dependencies/check-host-tar.mk

toolchain/dependencies/check-host-tar.mk +3 -0
@@ -0,0 +1,32 @@
 
+TAR ?= tar
 
+
 
+# To extract tar, we always need a real (system) tar. Provide the

necessary

toolchain/dependencies/check-host-tar.mk +2 -0
+# definitions here to avoid cluttering package/tar/tar.mk .
 
+# TAR will be overridden again when the tar .mk file is included, so we

must

toolchain/dependencies/check-host-tar.mk +10 -0
+# finalize the value of SYSTEM_TAR by using :=
 
+SYSTEM_TAR := $(TAR)
 
+# Automatically detect tar --strip-path/components option
 
+# --strip-path was renamed to --strip-components in tar 1.15
 
+SYSTEM_TAR_STRIP_COMPONENTS := \
 
+ $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
 
+ if test $$? = 0 ; then \
 
+ echo '--strip-path' ; \
 
+ else \
 
+ echo '--strip-components' ; \
Quoted 9 lines
Quoted 1 line

+ fi)

This can move inside the ifneq below, with a

SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS)

in the else (and moving that one up of course).

I don't think this is true.
For the chicken-egg problem in the current patch, you need SYSTEM_TAR (a real one) to build host-tar if needed. So in the 'else' case, you need the full SYSTEM_TAR_STRIP_COMPONENTS.

You could do the following: (maybe this is what you meant)

HOST_TAR_STRIP_COMPONENTS = --strip-components ifneq (,$(call suitable-host-package,tar))
HOST_TAR = $(SYSTEM_TAR) SYSTEM_TAR_STRIP_COMPONENTS := $(HOST_TAR_STRIP_COMPONENTS)
else
DEPENDENCIES_HOST_PREREQ += host-tar
HOST_TAR = $(HOST_DIR)/usr/bin/tar

  SYSTEM_TAR_STRIP_COMPONENTS := \
    $(shell $(SYSTEM_TAR) --help | grep strip-path > /dev/null ; \
    if test $$? = 0 ; then \
     echo '--strip-path' ; \
    else \
     echo '--strip-components' ; \
    fi)

endif

But to be honest, I don't think we'll need any of this complexity anymore with the cpio approach (see next mail).

Quoted 50 lines
Quoted 49 lines
toolchain/dependencies/check-host-tar.mk +13 -1
+
 
+# Determine correct HOST_TAR
 
+ifneq (,$(call suitable-host-package,tar))
 
+ HOST_TAR = $(SYSTEM_TAR)
 
+else
 
+ DEPENDENCIES_HOST_PREREQ += host-tar
 
+ HOST_TAR = $(HOST_DIR)/usr/bin/tar
 
+endif
 
+
 
+# Since HOST_TAR is at least 1.17, it will certainly support
--strip-components
 
+HOST_TAR_STRIP_COMPONENTS = --strip-components
 
+
 
+# Set definitions used by all .mk files included after this one and

before

toolchain/dependencies/check-host-tar.mk +3 -0
+# package/tar/tar.mk
 
+TAR = $(HOST_TAR)
 
+TAR_STRIP_COMPONENTS = $(HOST_TAR_STRIP_COMPONENTS)
toolchain/dependencies/check-host-tar.sh +0 -0

new file mode 100755

--- /dev/null

+++ b/toolchain/dependencies/check-host-tar.sh

toolchain/dependencies/check-host-tar.sh +2 -0
@@ -0,0 +1,29 @@
 
+#!/bin/sh
 
+
Quoted 8 lines
Quoted 1 line

+tar=which tar

This contradicts the TAR ?= tar in check-host-tar.mk: if the user provided

their own tar version, it isn't checked and host-tar will still be built...

I guess $TAR should be given as an argument to this script.

You're right, I missed this one.

Quoted 10 lines

toolchain/dependencies/check-host-tar.sh line 6

Quoted 7 lines
toolchain/dependencies/check-host-tar.sh +3 -0
+if [ -z "$tar" ]; then
 
+ return
 
+fi

This could be a [ ! -x "$tar" ]

ok

Quoted 2 lines

and it should be exit 1, not return.

Why? The convention is that the check-host-foo.sh script should give as output either nothing (no suitable package found) or the path to the suitable package. The return code is not checked.
Of course, it wouldn't hurt, but it's not needed for the implementation.

Thanks for your comments.

Best regards,
Thomas
_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#18 lore raw

PATCH v2 1/1 Move toolchain/dependencies to support/dependencies

In reply to Thomas De Schampheleire

As suggested by Arnout Vandecappelle, move toolchain/dependencies to support/dependencies, as it really is not toolchain-specific anymore.

Makefile +2 -0
@@ -298,6 +298,8 @@ include package/Makefile.in
 
 all: world
 
+include support/dependencies/dependencies.mk
+
 # We also need the various per-package makefiles, which also add
 # each selected package to TARGETS if that package was selected
 # in the .config file.
support/dependencies/check-host-tar.mk renamed from toolchain/dependencies/check-host-tar.mk +0 -0
rename from toolchain/dependencies/check-host-tar.mk
rename to support/dependencies/check-host-tar.mk
support/dependencies/check-host-tar.sh renamed from toolchain/dependencies/check-host-tar.sh +0 -0
rename from toolchain/dependencies/check-host-tar.sh
rename to support/dependencies/check-host-tar.sh
support/dependencies/dependencies.mk renamed from toolchain/dependencies/dependencies.mk +3 -3
rename from toolchain/dependencies/dependencies.mk
rename to support/dependencies/dependencies.mk
@@ -11,9 +11,9 @@ DEPENDENCIES_HOST_PREREQ:=
 # should return the path to the suitable host tool, or nothing if no
 # suitable tool was found.
 define suitable-host-package
-$(shell toolchain/dependencies/check-host-$(1).sh)
+$(shell support/dependencies/check-host-$(1).sh)
 endef
-include toolchain/dependencies/check-host-*.mk
+include support/dependencies/check-host-*.mk
 
 ifeq ($(BR2_STRIP_sstrip),y)
 DEPENDENCIES_HOST_PREREQ+=host-sstrip
@@ -29,7 +29,7 @@ core-dependencies:
 	@HOSTCC="$(firstword $(HOSTCC))" MAKE="$(MAKE)" \
 		CONFIG_FILE="$(CONFIG_DIR)/.config" \
 		DL_TOOLS="$(DL_TOOLS)" \
-		$(TOPDIR)/toolchain/dependencies/dependencies.sh
+		$(TOPDIR)/support/dependencies/dependencies.sh
 
 dependencies: core-dependencies $(DEPENDENCIES_HOST_PREREQ)
 
support/dependencies/dependencies.sh renamed from toolchain/dependencies/dependencies.sh +0 -0
rename from toolchain/dependencies/dependencies.sh
rename to support/dependencies/dependencies.sh
toolchain/toolchain-buildroot.mk +0 -1
@@ -1,6 +1,5 @@
 # Include files required for the internal toolchain backend
 
-include toolchain/dependencies/dependencies.mk
 include toolchain/elf2flt/elf2flt.mk
 include toolchain/gcc/gcc-uclibc-4.x.mk
 include toolchain/gdb/gdb.mk
toolchain/toolchain-crosstool-ng.mk +0 -1
@@ -2,7 +2,6 @@
 
 # Explicit ordering:
 include toolchain/helpers.mk
-include toolchain/dependencies/dependencies.mk
 include toolchain/elf2flt/elf2flt.mk
 include toolchain/gcc/gcc-uclibc-4.x.mk
 include toolchain/gdb/gdb.mk
toolchain/toolchain-external.mk +0 -1
@@ -1,7 +1,6 @@
 # Required includes for the external toolchain backend
 
 include toolchain/helpers.mk
-include toolchain/dependencies/dependencies.mk
 include toolchain/elf2flt/elf2flt.mk
 include toolchain/gcc/gcc-uclibc-4.x.mk
 include toolchain/gdb/gdb.mk
#19 lore raw

PATCH v2 1/1 Move toolchain/dependencies to support/dependencies

Le Wed, 23 Nov 2011 15:00:02 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :

Quoted 4 lines

As suggested by Arnout Vandecappelle, move toolchain/dependencies to support/dependencies, as it really is not toolchain-specific anymore.

Signature · 4 lines
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com
#20 lore raw

PATCH v2 1/1 Move toolchain/dependencies to support/dependencies

Quoted 9 lines · On Wednesday 23 November 2011 15:25:26 Thomas Petazzoni wrote:

Le Wed, 23 Nov 2011 15:00:02 +0100,
Thomas De Schampheleire <[email protected]> a ?crit :

Quoted 4 lines

As suggested by Arnout Vandecappelle, move toolchain/dependencies to support/dependencies, as it really is not toolchain-specific anymore.

Signature · 9 lines
Arnout Vandecappelle                               arnout at mind be
Senior Embedded Software Architect                 +32-16-286540
Essensium/Mind                                     http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium                BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  31BB CF53 8660 6F88 345D  54CC A836 5879 20D7 CF43
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20111206/0b7d8209/attachment.html>