Ensure extractor dependencies are taken into account
v2 cover Ensure extractor dependencies are taken into account
Hello,
The starting point for this series is to fix the following build failure:
http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log
The problem is the following:
- The _SOURCE for gettext-tiny is a .tar.gz
- In addition to this _SOURCE tarball, we also extract the gettext-gnu source code, which is a .tar.xz
- Extracting .tar.xz may require building host-xzcat, is xzcat is not available system-wide
- The build fails because host-xzcat is not added in the dependencies of gettext-tiny. Indeed, while generic-package takes care of adding the extract dependencies needed for _SOURCE, it does not know about other extract dependencies.
This issue appears in
http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log
because BR2_PER_PACKAGE_DIRECTORIES=y and therefore any missing dependency is immediately visible. But in theory, it could appear without per-package support.
This series solves that by introducing an extractor-pkg-dependency that returns the list of packages that must be added to <pkg>_EXTRACT_DEPENDENCIES to extract the archive passed as argument. It is then used in package/pkg-generic.mk to replace the existing logic to calculate <pkg>_EXTRACT_DEPENDENCIES, and extended to take into account the extractor dependencies needed by _EXTRA_DOWNLOADS instead of just _SOURCE.
We also rename extractor-dependency to extractor-system-dependency to clarify its difference from extractor-pkg-dependency, and we rework the implementation of extractor-system-dependency.
Changes since v1:
- Instead of fixing manually individual packages, directly derive extractor dependencies of EXTRA_DOWNLOADS files inside the generic-package infrastructure. Suggested by Yann. This allowed to greatly removed the number of patches in the series.
- Added Reviewed-by from Romain Naour on patches 1 and 4.
Best regards,
Thomas Petazzoni
Thomas Petazzoni (5):
package/pkg-utils.mk: rename extractor-dependency to
extractor-system-dependency
package/pkg-utils.mk: add extractor-pkg-dependency macro package/pkg-utils.mk: rework implementation of
extractor-system-dependency
package/pkg-generic.mk: use extractor-pkg-dependency macro package/pkg-generic.mk: also apply extractor-pkg-dependency to
<pkg>_EXTRA_DOWNLOADS
package/pkg-generic.mk | 14 ++++---------- package/pkg-utils.mk | 24 ++++++++++++++++++------ support/dependencies/check-host-lzip.mk | 1 - support/dependencies/check-host-xzcat.mk | 1 - 4 files changed, 22 insertions(+), 18 deletions(-)
Signature · 1 line
2.23.0
PATCH v2 1/5 package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency
The extractor-dependency macro returns which system-provided tools are needed to be able to extract the archive passed as argument. The result of this macro is added to DL_TOOLS_DEPENDENCIES so that the logic in support/dependencies/ verifies that the necessary tools are provided by the system.
However, we are going to add another macro, extractor-pkg-dependency, which says which Buildroot packages are needed to extract the archive passed as argument. Indeed, for those archive types, if the extractor is not provided system-wide, we build it as a host Buildroot package.
To clarify the distinction between the upcoming extractor-pkg-dependency and existing extractor-dependency, we rename the latter to extractor-system-dependency.
We take this opportunity to extend the documentation of this macro.
--- package/pkg-generic.mk | 2 +- package/pkg-utils.mk | 14 +++++++++----- 2 files changed, 10 insertions(+), 6 deletions(-)
package/pkg-generic.mk +1 -1
@@ -1098,7 +1098,7 @@ else ifeq ($$($(2)_SITE_METHOD),cvs) DL_TOOLS_DEPENDENCIES += cvs endif # SITE_METHOD -DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) +DL_TOOLS_DEPENDENCIES += $$(call extractor-system-dependency,$$($(2)_SOURCE)) # Ensure all virtual targets are PHONY. Listed alphabetically. .PHONY: $(1) \
package/pkg-utils.mk +9 -5
@@ -46,11 +46,15 @@ INFLATE.tar = cat # suitable-extractor(filename): returns extractor based on suffix suitable-extractor = $(INFLATE$(suffix $(1))) -# extractor-dependency(filename): returns extractor for 'filename' if the -# extractor is a dependency. If we build the extractor return nothing. -# $(firstword) is used here because the extractor can have arguments, like -# ZCAT="gzip -d -c", and to check for the dependency we only want 'gzip'. -extractor-dependency = $(firstword $(INFLATE$(filter-out \ +# extractor-system-dependency(filename): returns the name of the tool +# needed to extract 'filename', and is meant to be used with +# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is +# provided by the system Buildroot runs on. +# +# $(firstword) is used here because the extractor can have arguments, +# like ZCAT="gzip -d -c", and to check for the dependency we only want +# 'gzip'. +extractor-system-dependency = $(firstword $(INFLATE$(filter-out \ $(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1))))) # check-deprecated-variable -- throw an error on deprecated variables
Signature · 1 line
2.23.0
Yann E. MORIN
AppliedPATCH v2 1/5 package/pkg-utils.mk: rename extractor-dependency to extractor-system-dependency
Thomas, All,
On 2019-12-16 16:31 +0100, Thomas Petazzoni spake thusly:
Quoted 11 lines
The extractor-dependency macro returns which system-provided tools are needed to be able to extract the archive passed as argument. The result of this macro is added to
DL_TOOLS_DEPENDENCIESso that the logic in support/dependencies/ verifies that the necessary tools are provided by the system.However, we are going to add another macro, extractor-pkg-dependency, which says which Buildroot packages are needed to extract the archive passed as argument. Indeed, for those archive types, if the extractor is not provided system-wide, we build it as a host Buildroot package.
Quoted 8 lines
To clarify the distinction between the upcoming extractor-pkg-dependency and existing extractor-dependency, we rename the latter to extractor-system-dependency.
We take this opportunity to extend the documentation of this macro.
Signed-off-by Thomas Petazzoni <[email protected]>Reviewed-by Romain Naour <[email protected]>
Applied to master, thanks.
Regards,
Yann E. MORIN.
Quoted 46 lines
--- package/pkg-generic.mk | 2 +- package/pkg-utils.mk | 14 +++++++++----- 2 files changed, 10 insertions(+), 6 deletions(-)package/pkg-generic.mk +1
-1@@ -1098,7 +1098,7 @@ else ifeq ($$($(2)_SITE_METHOD),cvs) DL_TOOLS_DEPENDENCIES += cvs endif # SITE_METHOD -DL_TOOLS_DEPENDENCIES += $$(call extractor-dependency,$$($(2)_SOURCE)) +DL_TOOLS_DEPENDENCIES += $$(call extractor-system-dependency,$$($(2)_SOURCE)) # Ensure all virtual targets are PHONY. Listed alphabetically. .PHONY: $(1) \package/pkg-utils.mk +9
-5@@ -46,11 +46,15 @@ INFLATE.tar = cat # suitable-extractor(filename): returns extractor based on suffix suitable-extractor = $(INFLATE$(suffix $(1))) -# extractor-dependency(filename): returns extractor for 'filename' if the -# extractor is a dependency. If we build the extractor return nothing. -# $(firstword) is used here because the extractor can have arguments, like -# ZCAT="gzip -d -c", and to check for the dependency we only want 'gzip'. -extractor-dependency = $(firstword $(INFLATE$(filter-out \ +# extractor-system-dependency(filename): returns the name of the tool +# needed to extract 'filename', and is meant to be used with +# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is +# provided by the system Buildroot runs on. +# +# $(firstword) is used here because the extractor can have arguments, +# like ZCAT="gzip -d -c", and to check for the dependency we only want +# 'gzip'. +extractor-system-dependency = $(firstword $(INFLATE$(filter-out \ $(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1))))) # check-deprecated-variable -- throw an error on deprecated variablesSignature · 1 line
2.23.0
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 2/5 package/pkg-utils.mk: add extractor-pkg-dependency macro
In reply to Thomas PetazzoniTo extract some archive types, if the extracting tool is not available on the system where Buildroot runs on, we build it using a Buildroot host package.
Such dependencies are currently explicitly handled by the inner-generic-package macro, but in fact we also need to handle them in all places where the "suitable-extractor" macro is invoked, and some packages invoke it directly. Otherwise, such packages may be missing a dependency to the appropriate host Buildroot package building the extracting tool they need. An example is gettext-tiny, whose source code is a gzip-compressed tarball, but in addition manually extracts a xz-compressed tarball.
This extractor-pkg-dependency macro will be used in follow-up commits to ensure all the packages that use suitable-extractor properly add the correct dependencies.
--- package/pkg-utils.mk | 8 ++++++++ 1 file changed, 8 insertions(+)
package/pkg-utils.mk +8 -0
@@ -46,6 +46,14 @@ INFLATE.tar = cat # suitable-extractor(filename): returns extractor based on suffix suitable-extractor = $(INFLATE$(suffix $(1))) +EXTRACTOR_PKG_DEPENDENCY.lzma = $(BR2_XZCAT_HOST_DEPENDENCY) +EXTRACTOR_PKG_DEPENDENCY.xz = $(BR2_XZCAT_HOST_DEPENDENCY) +EXTRACTOR_PKG_DEPENDENCY.lz = $(BR2_LZIP_HOST_DEPENDENCY) + +# extractor-pkg-dependency(filename): returns a Buildroot package +# dependency needed to extract file based on suffix +extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))) + # extractor-system-dependency(filename): returns the name of the tool # needed to extract 'filename', and is meant to be used with # DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
Signature · 1 line
2.23.0
PATCH v2 3/5 package/pkg-utils.mk: rework implementation of extractor-system-dependency
In reply to Thomas PetazzoniNow that we have the EXTRACTOR_PKG_DEPENDENCY.* variables available, we can use them to implement extractor-system-dependency: if for a given archive type, the corresponding EXTRACTOR_PKG_DEPENDENCY.<type> variable is empty, then it means we need the corresponding extractor tool to be provided by the system.
Following this, EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS is no longer used, so we can drop it from support/dependencies/.
--- package/pkg-utils.mk | 4 ++-- support/dependencies/check-host-lzip.mk | 1 - support/dependencies/check-host-xzcat.mk | 1 - 3 files changed, 2 insertions(+), 4 deletions(-)
package/pkg-utils.mk +2 -2
@@ -62,8 +62,8 @@ extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))) # $(firstword) is used here because the extractor can have arguments, # like ZCAT="gzip -d -c", and to check for the dependency we only want # 'gzip'. -extractor-system-dependency = $(firstword $(INFLATE$(filter-out \ - $(EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS),$(suffix $(1))))) +extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\ + $(firstword $(INFLATE$(suffix $(1))))) # check-deprecated-variable -- throw an error on deprecated variables # example:
support/dependencies/check-host-lzip.mk +0 -1
@@ -1,5 +1,4 @@ ifeq (,$(call suitable-host-package,lzip,$(LZCAT))) BR2_LZIP_HOST_DEPENDENCY = host-lzip -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .lz LZCAT = $(HOST_DIR)/bin/lzip -d -c endif
support/dependencies/check-host-xzcat.mk +0 -1
@@ -3,6 +3,5 @@ ifeq (,$(call suitable-host-package,xzcat,$(XZCAT))) BR2_XZCAT_HOST_DEPENDENCY = host-xz -EXTRACTOR_DEPENDENCY_PRECHECKED_EXTENSIONS += .xz .lzma XZCAT = $(HOST_DIR)/bin/xzcat endif
Signature · 1 line
2.23.0
PATCH v2 4/5 package/pkg-generic.mk: use extractor-pkg-dependency macro
In reply to Thomas PetazzoniInstead of manually calculating the EXTRACT_DEPENDENCIES value based on the archive extension, let's use the newly introduced extractor-pkg-dependency macro.
--- package/pkg-generic.mk | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-)
package/pkg-generic.mk +1 -9
@@ -644,15 +644,7 @@ $(2)_EXTRACT_DEPENDENCIES += $$(BR2_TAR_HOST_DEPENDENCY) endif ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),) -ifneq ($$(filter .xz .lzma,$$(suffix $$($(2)_SOURCE))),) -$(2)_EXTRACT_DEPENDENCIES += $$(BR2_XZCAT_HOST_DEPENDENCY) -endif -endif - -ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),) -ifneq ($$(filter .lz,$$(suffix $$($(2)_SOURCE))),) -$(2)_EXTRACT_DEPENDENCIES += $$(BR2_LZIP_HOST_DEPENDENCY) -endif +$(2)_EXTRACT_DEPENDENCIES += $$(call extractor-pkg-dependency,$$($(2)_SOURCE)) endif ifeq ($$(BR2_CCACHE),y)
Signature · 1 line
2.23.0
PATCH v2 5/5 package/pkg-generic.mk: also apply extractor-pkg-dependency to <pkg>_EXTRA_DOWNLOADS
In reply to Thomas PetazzoniFor now, the extractor dependencies were only calculated for <pkg>_SOURCE, so if the package manually downloads another file using <pkg>_EXTRA_DOWNLOADS and then extracts it with $(call suitable-extractor), we are missing the corresponding dependency on the appropriate extracting tool.
Since the vast majority of <pkg>_EXTRA_DOWNLOADS are compressed files that will be uncompressed at build time, it makes sense to derive the corresponding extractor dependencies directly in the common package infrastructure, rather than having each and every package using <pkg>_EXTRA_DOWNLOADS making this effort.
On a system without xzcat, before this patch:
$ make printvars VARS=HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES=host-tar
After this patch:
$ make printvars VARS=HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES HOST_GETTEXT_TINY_EXTRACT_DEPENDENCIES=host-tar host-xz
This commit most notably fixes the build of host-gettext-tiny on systems without xzcat, and with per-package support enabled. Indeed, the main _SOURCE for gettext-tiny is a .gz file, but it has a .xz file in its _EXTRA_DOWNLOADS, which is then extracted. Except that xzcat being missing from the dependencies, it is not built.
Fixes:
http://autobuild.buildroot.net/results/83c6d47c06334bef27791a59bdd491b1de124c49/
--- package/pkg-generic.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-generic.mk +3 -1
@@ -644,7 +644,9 @@ $(2)_EXTRACT_DEPENDENCIES += $$(BR2_TAR_HOST_DEPENDENCY) endif ifeq ($$(filter host-tar host-skeleton host-xz host-lzip host-fakedate,$(1)),) -$(2)_EXTRACT_DEPENDENCIES += $$(call extractor-pkg-dependency,$$($(2)_SOURCE)) +$(2)_EXTRACT_DEPENDENCIES += \ + $$(foreach dl,$$($(2)_ALL_DOWNLOADS),\ + $$(call extractor-pkg-dependency,$$(notdir $$(dl)))) endif ifeq ($$(BR2_CCACHE),y)
Signature · 1 line
2.23.0
v2 cover Ensure extractor dependencies are taken into account
In reply to Thomas PetazzoniThomas, All,
On 2019-12-16 16:31 +0100, Thomas Petazzoni spake thusly:
Quoted 4 lines
The starting point for this series is to fix the following build failure:
http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log
After a slow start yesterday evening, I now applied the rest of the series to master. Thanks!
Regards,
Yann E. MORIN.
Quoted 65 lines
The problem is the following:
- The
_SOURCEfor gettext-tiny is a .tar.gz- In addition to this
_SOURCEtarball, we also extract the gettext-gnu source code, which is a .tar.xz- Extracting .tar.xz may require building host-xzcat, is xzcat is not available system-wide
- The build fails because host-xzcat is not added in the dependencies of gettext-tiny. Indeed, while generic-package takes care of adding the extract dependencies needed for
_SOURCE, it does not know about other extract dependencies.This issue appears in
http://autobuild.buildroot.net/results/83c/83c6d47c06334bef27791a59bdd491b1de124c49/build-end.log
because BR2_PER_PACKAGE_DIRECTORIES=y and therefore any missing dependency is immediately visible. But in theory, it could appear without per-package support.This series solves that by introducing an extractor-pkg-dependency that returns the list of packages that must be added to <pkg>_EXTRACT_DEPENDENCIES to extract the archive passed as argument. It is then used in
package/pkg-generic.mkto replace the existing logic to calculate <pkg>_EXTRACT_DEPENDENCIES, and extended to take into account the extractor dependencies needed by_EXTRA_DOWNLOADSinstead of just_SOURCE.We also rename extractor-dependency to extractor-system-dependency to clarify its difference from extractor-pkg-dependency, and we rework the implementation of extractor-system-dependency.
Changes since v1:
- Instead of fixing manually individual packages, directly derive extractor dependencies of
EXTRA_DOWNLOADSfiles inside the generic-package infrastructure. Suggested by Yann. This allowed to greatly removed the number of patches in the series.- Added Reviewed-by from Romain Naour on patches 1 and 4.
Best regards,
Thomas Petazzoni
Thomas Petazzoni (5):
package/pkg-utils.mk: rename extractor-dependency to
extractor-system-dependency
package/pkg-utils.mk: add extractor-pkg-dependency macropackage/pkg-utils.mk: rework implementation of
extractor-system-dependency
package/pkg-generic.mk: use extractor-pkg-dependency macropackage/pkg-generic.mk: also apply extractor-pkg-dependency to
<pkg>_EXTRA_DOWNLOADSpackage/pkg-generic.mk | 14 ++++---------- package/pkg-utils.mk | 24 ++++++++++++++++++------ support/dependencies/check-host-lzip.mk | 1 - support/dependencies/check-host-xzcat.mk | 1 - 4 files changed, 22 insertions(+), 18 deletions(-)Signature · 1 line
2.23.0
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'