Skip to contentSkip to conversation

Ensure extractor dependencies are taken into account

PATCH New Thomas Petazzoni 2 participants 8 messages Dec 16, 2019 – Dec 17, 2019

lore
Share & tools
lore
#0 lore raw

v2 cover Ensure extractor dependencies are taken into account

Hello,

The starting point for this series is to fix the following build failure:

http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log

The problem is the following:

- The _SOURCE for gettext-tiny is a .tar.gz

- In addition to this _SOURCE tarball, we also extract the gettext-gnu source code, which is a .tar.xz

- Extracting .tar.xz may require building host-xzcat, is xzcat is not available system-wide

- The build fails because host-xzcat is not added in the dependencies of gettext-tiny. Indeed, while generic-package takes care of adding the extract dependencies needed for _SOURCE, it does not know about other extract dependencies.

This issue appears in
http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log
because BR2_PER_PACKAGE_DIRECTORIES=y and therefore any missing dependency is immediately visible. But in theory, it could appear without per-package support.

This series solves that by introducing an extractor-pkg-dependency that returns the list of packages that must be added to <pkg>_EXTRACT_DEPENDENCIES to extract the archive passed as argument. It is then used in package/pkg-generic.mk to replace the existing logic to calculate <pkg>_EXTRACT_DEPENDENCIES, and extended to take into account the extractor dependencies needed by _EXTRA_DOWNLOADS instead of just _SOURCE.

We also rename extractor-dependency to extractor-system-dependency to clarify its difference from extractor-pkg-dependency, and we rework the implementation of extractor-system-dependency.

Changes since v1:

- Instead of fixing manually individual packages, directly derive extractor dependencies of EXTRA_DOWNLOADS files inside the generic-package infrastructure. Suggested by Yann. This allowed to greatly removed the number of patches in the series.

- Added Reviewed-by from Romain Naour on patches 1 and 4.

Best regards,

Thomas Petazzoni

Thomas Petazzoni (5):
package/pkg-utils.mk: rename extractor-dependency to
extractor-system-dependency
package/pkg-utils.mk: add extractor-pkg-dependency macro package/pkg-utils.mk: rework implementation of
extractor-system-dependency
package/pkg-generic.mk: use extractor-pkg-dependency macro package/pkg-generic.mk: also apply extractor-pkg-dependency to
<pkg>_EXTRA_DOWNLOADS

 package/pkg-generic.mk                   | 14 ++++----------
 package/pkg-utils.mk                     | 24 ++++++++++++++++++------
 support/dependencies/check-host-lzip.mk  |  1 -
 support/dependencies/check-host-xzcat.mk |  1 -
 4 files changed, 22 insertions(+), 18 deletions(-)
Signature · 1 line
2.23.0
#1 lore raw

PATCH v2 1/5 package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency

The extractor-dependency macro returns which system-provided tools are needed to be able to extract the archive passed as argument. The result of this macro is added to DL_TOOLS_DEPENDENCIES so that the logic in support/dependencies/ verifies that the necessary tools are provided by the system.

However, we are going to add another macro, extractor-pkg-dependency, which says which Buildroot packages are needed to extract the archive passed as argument. Indeed, for those archive types, if the extractor is not provided system-wide, we build it as a host Buildroot package.

To clarify the distinction between the upcoming extractor-pkg-dependency and existing extractor-dependency, we rename the latter to extractor-system-dependency.

We take this opportunity to extend the documentation of this macro.

---
 package/pkg-generic.mk |  2 +-
 package/pkg-utils.mk   | 14 +++++++++-----
 2 files changed, 10 insertions(+), 6 deletions(-)
package/pkg-generic.mk +1 -1
index 3904a1c4db..fd1817a86c 100644
@@ -1098,7 +1098,7 @@ else ifeq ($$($(2)_SITE_METHOD),cvs)
 DL_TOOLS_DEPENDENCIES += cvs
 endif # SITE_METHOD
 
-DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
+DL_TOOLS_DEPENDENCIES += $$(call extractor-system-dependency,$$($(2)_SOURCE))
 
 # Ensure all virtual targets are PHONY. Listed alphabetically.
 .PHONY:	$(1) \
package/pkg-utils.mk +9 -5
index d38971caf6..0f916d81d8 100644
@@ -46,11 +46,15 @@ INFLATE.tar  = cat
 # suitable-extractor(filename): returns extractor based on suffix
 suitable-extractor = $(INFLATE$(suffix $(1)))
 
-# extractor-dependency(filename): returns extractor for 'filename' if the
-# extractor is a dependency. If we build the extractor return nothing.
-# $(firstword) is used here because the extractor can have arguments, like
-# ZCAT="gzip -d -c", and to check for the dependency we only want 'gzip'.
-extractor-dependency = $(firstword $(INFLATE$(filter-out \
+# extractor-system-dependency(filename): returns the name of the tool
+# needed to extract 'filename', and is meant to be used with
+# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
+# provided by the system Buildroot runs on.
+#
+# $(firstword) is used here because the extractor can have arguments,
+# like ZCAT="gzip -d -c", and to check for the dependency we only want
+# 'gzip'.
+extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
 	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
 
 # check-deprecated-variable -- throw an error on deprecated variables
Signature · 1 line
2.23.0
#2 lore raw

PATCH v2 1/5 package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency

Thomas, All,

On 2019-12-16 16:31 +0100, Thomas Petazzoni spake thusly:

Quoted 11 lines

The extractor-dependency macro returns which system-provided tools are needed to be able to extract the archive passed as argument. The result of this macro is added to DL_TOOLS_DEPENDENCIES so that the logic in support/dependencies/ verifies that the necessary tools are provided by the system.

However, we are going to add another macro, extractor-pkg-dependency, which says which Buildroot packages are needed to extract the archive passed as argument. Indeed, for those archive types, if the extractor is not provided system-wide, we build it as a host Buildroot package.

Quoted 8 lines

To clarify the distinction between the upcoming extractor-pkg-dependency and existing extractor-dependency, we rename the latter to extractor-system-dependency.

We take this opportunity to extend the documentation of this macro.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 46 lines
---
 package/pkg-generic.mk |  2 +-
 package/pkg-utils.mk   | 14 +++++++++-----
 2 files changed, 10 insertions(+), 6 deletions(-)
package/pkg-generic.mk +1 -1
index 3904a1c4db..fd1817a86c 100644
@@ -1098,7 +1098,7 @@ else ifeq ($$($(2)_SITE_METHOD),cvs)
 DL_TOOLS_DEPENDENCIES += cvs
 endif # SITE_METHOD
 
-DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE))
+DL_TOOLS_DEPENDENCIES += $$(call extractor-system-dependency,$$($(2)_SOURCE))
 
 # Ensure all virtual targets are PHONY. Listed alphabetically.
 .PHONY:	$(1) \
package/pkg-utils.mk +9 -5
index d38971caf6..0f916d81d8 100644
@@ -46,11 +46,15 @@ INFLATE.tar  = cat
 # suitable-extractor(filename): returns extractor based on suffix
 suitable-extractor = $(INFLATE$(suffix $(1)))
 
-# extractor-dependency(filename): returns extractor for 'filename' if the
-# extractor is a dependency. If we build the extractor return nothing.
-# $(firstword) is used here because the extractor can have arguments, like
-# ZCAT="gzip -d -c", and to check for the dependency we only want 'gzip'.
-extractor-dependency = $(firstword $(INFLATE$(filter-out \
+# extractor-system-dependency(filename): returns the name of the tool
+# needed to extract 'filename', and is meant to be used with
+# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
+# provided by the system Buildroot runs on.
+#
+# $(firstword) is used here because the extractor can have arguments,
+# like ZCAT="gzip -d -c", and to check for the dependency we only want
+# 'gzip'.
+extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
 	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
 
 # check-deprecated-variable -- throw an error on deprecated variables
Signature · 1 line
2.23.0
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH v2 2/5 package/pkg-utils.mk: add extractor-pkg-dependency macro

In reply to Thomas Petazzoni

To extract some archive types, if the extracting tool is not available on the system where Buildroot runs on, we build it using a Buildroot host package.

Such dependencies are currently explicitly handled by the inner-generic-package macro, but in fact we also need to handle them in all places where the "suitable-extractor" macro is invoked, and some packages invoke it directly. Otherwise, such packages may be missing a dependency to the appropriate host Buildroot package building the extracting tool they need. An example is gettext-tiny, whose source code is a gzip-compressed tarball, but in addition manually extracts a xz-compressed tarball.

This extractor-pkg-dependency macro will be used in follow-up commits to ensure all the packages that use suitable-extractor properly add the correct dependencies.

---
 package/pkg-utils.mk | 8 ++++++++
 1 file changed, 8 insertions(+)
package/pkg-utils.mk +8 -0
index 0f916d81d8..35a441e0aa 100644
@@ -46,6 +46,14 @@ INFLATE.tar  = cat
 # suitable-extractor(filename): returns extractor based on suffix
 suitable-extractor = $(INFLATE$(suffix $(1)))
 
+EXTRACTOR_PKG_DEPENDENCY.lzma = $(BR2_XZCAT_HOST_DEPENDENCY)
+EXTRACTOR_PKG_DEPENDENCY.xz   = $(BR2_XZCAT_HOST_DEPENDENCY)
+EXTRACTOR_PKG_DEPENDENCY.lz   = $(BR2_LZIP_HOST_DEPENDENCY)
+
+# extractor-pkg-dependency(filename): returns a Buildroot package
+# dependency needed to extract file based on suffix
+extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
+
 # extractor-system-dependency(filename): returns the name of the tool
 # needed to extract 'filename', and is meant to be used with
 # DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
Signature · 1 line
2.23.0
#4 lore raw

PATCH v2 3/5 package/pkg-utils.mk: rework implementation of extractor-system-dependency

In reply to Thomas Petazzoni

Now that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available, we can use them to implement extractor-system-dependency: if for a given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type> variable is empty, then it means we need the corresponding extractor tool to be provided by the system.

Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no longer used, so we can drop it from support/dependencies/.

---
 package/pkg-utils.mk                     | 4 ++--
 support/dependencies/check-host-lzip.mk  | 1 -
 support/dependencies/check-host-xzcat.mk | 1 -
 3 files changed, 2 insertions(+), 4 deletions(-)
package/pkg-utils.mk +2 -2
index 35a441e0aa..d324934dba 100644
@@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
 # $(firstword) is used here because the extractor can have arguments,
 # like ZCAT="gzip -d -c", and to check for the dependency we only want
 # 'gzip'.
-extractor-system-dependency = $(firstword $(INFLATE$(filter-out \
-	$(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1)))))
+extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
+	$(firstword $(INFLATE$(suffix $(1)))))
 
 # check-deprecated-variable -- throw an error on deprecated variables
 # example:
support/dependencies/check-host-lzip.mk +0 -1
index cdd784058c..98a4f42388 100644
@@ -1,5 +1,4 @@
 ifeq (,$(call suitable-host-package,lzip,$(LZCAT)))
 BR2_LZIP_HOST_DEPENDENCY = host-lzip
-EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz
 LZCAT = $(HOST_DIR)/bin/lzip -d -c
 endif
support/dependencies/check-host-xzcat.mk +0 -1
index e5d72f95dd..9a78a8f38a 100644
@@ -3,6 +3,5 @@
 
 ifeq (,$(call suitable-host-package,xzcat,$(XZCAT)))
 BR2_XZCAT_HOST_DEPENDENCY = host-xz
-EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma
 XZCAT = $(HOST_DIR)/bin/xzcat
 endif
Signature · 1 line
2.23.0
#5 lore raw

PATCH v2 4/5 package/pkg-generic.mk: use extractor-pkg-dependency macro

In reply to Thomas Petazzoni

Instead of manually calculating the EXTRACT_DEPENDENCIES value based on the archive extension, let's use the newly introduced extractor-pkg-dependency macro.

---
 package/pkg-generic.mk | 10 +---------
 1 file changed, 1 insertion(+), 9 deletions(-)
package/pkg-generic.mk +1 -9
index fd1817a86c..6c0105a1e1 100644
@@ -644,15 +644,7 @@ $(2)_EXTRACT_DEPENDENCIES += $$(BR2_TAR_HOST_DEPENDENCY)
 endif
 
 ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
-ifneq ($$(filter .xz .lzma,$$(suffix $$($(2)_SOURCE))),)
-$(2)_EXTRACT_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY)
-endif
-endif
-
-ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
-ifneq ($$(filter .lz,$$(suffix $$($(2)_SOURCE))),)
-$(2)_EXTRACT_DEPENDENCIES += $$(BR2_LZIP_HOST_DEPENDENCY)
-endif
+$(2)_EXTRACT_DEPENDENCIES += $$(call extractor-pkg-dependency,$$($(2)_SOURCE))
 endif
 
 ifeq ($$(BR2_CCACHE),y)
Signature · 1 line
2.23.0
#6 lore raw

PATCH v2 5/5 package/pkg-generic.mk: also apply extractor-pkg-dependency to <pkg>_EXTRA_DOWNLOADS

In reply to Thomas Petazzoni

For now, the extractor dependencies were only calculated for <pkg>_SOURCE, so if the package manually downloads another file using <pkg>_EXTRA_DOWNLOADS and then extracts it with $(call suitable-extractor), we are missing the corresponding dependency on the appropriate extracting tool.

Since the vast majority of <pkg>_EXTRA_DOWNLOADS are compressed files that will be uncompressed at build time, it makes sense to derive the corresponding extractor dependencies directly in the common package infrastructure, rather than having each and every package using <pkg>_EXTRA_DOWNLOADS making this effort.

On a system without xzcat, before this patch:

$ make printvars VARS=HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES
HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES=host-tar

After this patch:

$ make printvars VARS=HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES
HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES=host-tar    host-xz

This commit most notably fixes the build of host-gettext-tiny on systems without xzcat, and with per-package support enabled. Indeed, the main _SOURCE for gettext-tiny is a .gz file, but it has a .xz file in its _EXTRA_DOWNLOADS, which is then extracted. Except that xzcat being missing from the dependencies, it is not built.

Fixes:

http://autobuild.buildroot.net/results/83c6d47c06334bef27791a59bdd491b1de124c49/

---
 package/pkg-generic.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-generic.mk +3 -1
index 6c0105a1e1..268d999efb 100644
@@ -644,7 +644,9 @@ $(2)_EXTRACT_DEPENDENCIES += $$(BR2_TAR_HOST_DEPENDENCY)
 endif
 
 ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),)
-$(2)_EXTRACT_DEPENDENCIES += $$(call extractor-pkg-dependency,$$($(2)_SOURCE))
+$(2)_EXTRACT_DEPENDENCIES += \
+	$$(foreach dl,$$($(2)_ALL_DOWNLOADS),\
+		$$(call extractor-pkg-dependency,$$(notdir $$(dl))))
 endif
 
 ifeq ($$(BR2_CCACHE),y)
Signature · 1 line
2.23.0
#7 lore raw

v2 cover Ensure extractor dependencies are taken into account

In reply to Thomas Petazzoni

Thomas, All,

On 2019-12-16 16:31 +0100, Thomas Petazzoni spake thusly:

Quoted 4 lines

The starting point for this series is to fix the following build failure:

http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log

After a slow start yesterday evening, I now applied the rest of the series to master. Thanks!

Regards,
Yann E. MORIN.

Quoted 65 lines

The problem is the following:

- The _SOURCE for gettext-tiny is a .tar.gz

- In addition to this _SOURCE tarball, we also extract the gettext-gnu source code, which is a .tar.xz

- Extracting .tar.xz may require building host-xzcat, is xzcat is not available system-wide

- The build fails because host-xzcat is not added in the dependencies of gettext-tiny. Indeed, while generic-package takes care of adding the extract dependencies needed for _SOURCE, it does not know about other extract dependencies.

This issue appears in
http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log
because BR2_PER_PACKAGE_DIRECTORIES=y and therefore any missing dependency is immediately visible. But in theory, it could appear without per-package support.

This series solves that by introducing an extractor-pkg-dependency that returns the list of packages that must be added to <pkg>_EXTRACT_DEPENDENCIES to extract the archive passed as argument. It is then used in package/pkg-generic.mk to replace the existing logic to calculate <pkg>_EXTRACT_DEPENDENCIES, and extended to take into account the extractor dependencies needed by _EXTRA_DOWNLOADS instead of just _SOURCE.

We also rename extractor-dependency to extractor-system-dependency to clarify its difference from extractor-pkg-dependency, and we rework the implementation of extractor-system-dependency.

Changes since v1:

- Instead of fixing manually individual packages, directly derive extractor dependencies of EXTRA_DOWNLOADS files inside the generic-package infrastructure. Suggested by Yann. This allowed to greatly removed the number of patches in the series.

- Added Reviewed-by from Romain Naour on patches 1 and 4.

Best regards,

Thomas Petazzoni

Thomas Petazzoni (5):
package/pkg-utils.mk: rename extractor-dependency to
extractor-system-dependency
package/pkg-utils.mk: add extractor-pkg-dependency macro package/pkg-utils.mk: rework implementation of
extractor-system-dependency
package/pkg-generic.mk: use extractor-pkg-dependency macro package/pkg-generic.mk: also apply extractor-pkg-dependency to
<pkg>_EXTRA_DOWNLOADS

 package/pkg-generic.mk                   | 14 ++++----------
 package/pkg-utils.mk                     | 24 ++++++++++++++++++------
 support/dependencies/check-host-lzip.mk  |  1 -
 support/dependencies/check-host-xzcat.mk |  1 -
 4 files changed, 22 insertions(+), 18 deletions(-)
Signature · 1 line
2.23.0
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'