legal-info: save the external-toolchain source archive
v2 cover legal-info: save the external-toolchain source archive
Hi,
during the Buildroot Developer Days in October last year, prompted by Thomas De Schampheleire, we discussed a way to improve 'make legal-info' to collect the external toolchain source code.
The external toolchain is a special package because TOOLCHAIN_EXTERNAL_SITE/SOURCE points to a file that is indeed a binary archive. The "actual" source code is usually provided by toolchain vendors as a separate file.
During the BDD we stated [1]:
We'll need to add a FOO_ACTUAL_SOURCES to the infrastructure: if set (to a tarball URL) it forces 'make legal-info' to download that tarball and store in the tarball directory instead of the FOO_SOURCE tarball.
I submitted a first patchset in January [2], which got some comments and change requests by Arnout. One of them was to cleanup to trailing slash in all FOO_SITE variables, for which I submitted a separate patchset, now in v4 [3].
The two patchset are pretty much indepent anyway, and this one rebases seamlessly on top of the slash-removal one.
Now it's time to revive this patchset. Changes are quite minor, however.
The core change is in patch 1. I added two variables to the pkg-generic infra:FOO_ACTUAL_SOURCE_TARBALL and FOO_ACTUAL_SOURCE_SITE, which default to the value of the good old FOO_SOURCE and FOO_SITE. Then I use the new variables for the legal-info processing.
If these variables are overridden, which will be the case for external toolchains, they trigger an extra $(call DOWNLOAD,...) to fetch the actual package source tarball.
Patches 2 and 3 set the TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL to appropriate values in external-toolchain.mk for most toolchains. TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_SITE is never set because it happens to be equal to TOOLCHAIN_EXTERNAL_SITE in all the cases I implemented.
Patch 4 documents it all in the manual.
The current solution works fine according to all my tests.
There are a few improvements that might be done.
The first limitation it that there's no "actual" version of FOO_EXTRA_DOWNLOADS. Thus Blackfin toolchains, which use that feature, cannot be handled in a complete way. Arnout proposed FOO_ACTUAL_SOURCE to become a list of files rather than a single file. It should work, but I prefer to add it later and the the bulk of the changes up and running first.
The other is there's no provision for extracting the license files from the downloaded tarball, which would be a nice addition. I'd leave that for the future as well. It's a separate feature, after all.
That's all folks, here's the code. Be aware that some toolchains are not yet implemented: Blackfin for the reason stated above, musl because there seem to be no complete source code tarball.
Regards,
Luca
[1] http://www.elinux.org/Buildroot:DeveloperDaysELCE2014#State_of_legal-info_infrastructure.2C_improvements_to_be_made.3F
[2] http://lists.busybox.net/pipermail/buildroot/2015-January/116303.html
[3] http://lists.busybox.net/pipermail/buildroot/2015-October/140813.html
Luca Ceresoli (4):
legal-info: allow to declare the actual sources for binary packages
toolchain-external: mass-define actual source tarball for known
patterns
toolchain-external: define actual sources for arago toolchains docs/manual: document new actual source package variables
Makefile | 1 - docs/manual/adding-packages-generic.txt | 19 ++++++++++++++++++ package/pkg-generic.mk | 23 +++++++++++++++++----- toolchain/toolchain-external/toolchain-external.mk | 12 +++++++++++ 4 files changed, 49 insertions(+), 6 deletions(-)
Signature · 1 line
1.9.1
PATCH v2 1/4 legal-info: allow to declare the actual sources for binary packages
The FOO_SITE/FOO_SOURCE variables usually point to a tarball containing source code.
For the downloaded external toolchains this is not true, the "source" tarball actually contains binaries. This is fine for making Buildroot work, but for legal-info we really want to ship real source code, not binaries.
Luckily, some (hopefully all) toolchain vendors publish a downloadable tarball containing the source code counterpart for their binary packages.
Here we allow the user to declare the URL of this other tarball in the pair of variables FOO_ACTUAL_SOURCE_TARBALL (by default equal to FOO_SOURCE) and FOO_ACTUAL_SOURCE_SITE (by default equal to FOO_SITE). If the "actual source" package can be downloaded from the same directory as the binary package, then only FOO_ACTUAL_SOURCE_TARBALL needs to be set.
Note this change is not strictly toolchain-specific: it might be useful for other packages that happen to ship binaries in the same way.
---
Changes v1 -> v2:
- Fix warning text ($(1) instead of "the toolchain source code"), reported by Arnout
- Get rid of the trailing-slash-removal hack [as in $$($(2)_ACTUAL_SOURCE_SITE:/=)], there's another patchset to fix that.
- Fix a typo.
--- Makefile | 1 - package/pkg-generic.mk | 23 ++++++++++++++++++----- 2 files changed, 18 insertions(+), 6 deletions(-)
Makefile +0 -1
@@ -647,7 +647,6 @@ legal-info-prepare: $(LEGAL_INFO_DIR) @$(call legal-manifest,PACKAGE,VERSION,LICENSE,LICENSE FILES,SOURCE ARCHIVE,SOURCE SITE,HOST) @$(call legal-manifest,buildroot,$(BR2_VERSION_FULL),GPLv2+,COPYING,not saved,not saved,HOST) @$(call legal-warning,the Buildroot source code has not been saved) - @$(call legal-warning,the toolchain has not been saved) @cp $(BR2_CONFIG) $(LEGAL_INFO_DIR)/buildroot.config legal-info: dirs legal-info-clean legal-info-prepare $(foreach p,$(PACKAGES),$(p)-all-legal-info) \
package/pkg-generic.mk +18 -5
@@ -708,12 +708,18 @@ ifneq ($$($(2)_SITE_METHOD),local) ifneq ($$($(2)_SITE_METHOD),override) # Packages that have a tarball need it downloaded beforehand $(1)-legal-info: $(1)-source $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) -$(2)_MANIFEST_TARBALL = $$($(2)_SOURCE) -$(2)_MANIFEST_SITE = $$(call qstrip,$$($(2)_SITE)) endif endif endif +# If FOO_ACTUAL_SOURCE_TARBALL is explicitly defined, it means FOO_SOURCE is +# indeed a binary (e.g. external toolchain) and FOO_ACTUAL_SOURCE_TARBALL/_SITE +# point to the actual sources tarball. Use the actual sources for legal-info. +# For most packages the FOO_SITE/FOO_SOURCE pair points to real source code, +# so these are the defaults for FOO_ACTUAL_*. +$(2)_ACTUAL_SOURCE_TARBALL ?= $$($(2)_SOURCE) +$(2)_ACTUAL_SOURCE_SITE ?= $$(call qstrip,$$($(2)_SITE)) + # legal-info: produce legally relevant info. $(1)-legal-info: # Packages without a source are assumed to be part of Buildroot, skip them.
@@ -744,13 +750,20 @@ else # Other packages ifeq ($$($(2)_REDISTRIBUTE),YES) +ifeq ($$($(2)_ACTUAL_SOURCE_TARBALL),) + @$(call legal-warning,$(1): the source code has not been saved) +else +ifneq ($$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_SOURCE)) + $(call DOWNLOAD,$$($(2)_ACTUAL_SOURCE_SITE)/$$($(2)_ACTUAL_SOURCE_TARBALL)) +endif # Copy the source tarball (just hardlink if possible) - @cp -l $$(DL_DIR)/$$($(2)_SOURCE) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) 2>/dev/null || \ - cp $$(DL_DIR)/$$($(2)_SOURCE) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) + @cp -l $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) 2>/dev/null || \ + cp $$(DL_DIR)/$$($(2)_ACTUAL_SOURCE_TARBALL) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) +endif # ($$($(2)_ACTUAL_SOURCE_TARBALL),) endif # redistribute endif # other packages - @$$(call legal-manifest,$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_LICENSE),$$($(2)_MANIFEST_LICENSE_FILES),$$($(2)_MANIFEST_TARBALL),$$($(2)_MANIFEST_SITE),$$(call UPPERCASE,$(4))) + @$$(call legal-manifest,$$($(2)_RAWNAME),$$($(2)_VERSION),$$($(2)_LICENSE),$$($(2)_MANIFEST_LICENSE_FILES),$$($(2)_ACTUAL_SOURCE_TARBALL),$$($(2)_ACTUAL_SOURCE_SITE),$$(call UPPERCASE,$(4))) endif # ifneq ($$(call qstrip,$$($(2)_SOURCE)),) $$(foreach hook,$$($(2)_POST_LEGAL_INFO_HOOKS),$$(call $$(hook))$$(sep))
Signature · 1 line
1.9.1
PATCH v2 2/4 toolchain-external: mass-define actual source tarball for known patterns
In reply to Luca CeresoliFor some external toolchain vendors the actual source code URL can be simply derived from the binary file URL.
Here we obtain TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL for all Mentor and Linaro toolchains with a few $(subst) calls.
---
Changes v1 -> v2:
- Wrap a long line as much as possible (Arnout).
--- toolchain/toolchain-external/toolchain-external.mk | 10 ++++++++++ 1 file changed, 10 insertions(+)
toolchain/toolchain-external/toolchain-external.mk +10 -0
@@ -422,6 +422,16 @@ TOOLCHAIN_EXTERNAL_SOURCE = $(notdir $(call qstrip,$(BR2_TOOLCHAIN_EXTERNAL_URL) BR_NO_CHECK_HASH_FOR += $(TOOLCHAIN_EXTERNAL_SOURCE) endif +# Some toolchain vendors have a regular file naming pattern. +# For them, mass-define _ACTUAL_SOURCE_TARBALL based _SITE. +ifneq ($(findstring sourcery.mentor.com/public/gnu_toolchain,$(TOOLCHAIN_EXTERNAL_SITE)),) +TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL ?= \ + $(subst -i686-pc-linux-gnu.tar.bz2,.src.tar.bz2,$(subst -i686-pc-linux-gnu-i386-linux.tar.bz2,-i686-pc-linux-gnu.src.tar.bz2,$(TOOLCHAIN_EXTERNAL_SOURCE))) +else ifneq ($(findstring http://releases.linaro.org,$(TOOLCHAIN_EXTERNAL_SITE)),) +TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL ?= \ + $(subst _linux.tar.xz,_src.tar.bz2,$(TOOLCHAIN_EXTERNAL_SOURCE)) +endif + # In fact, we don't need to download the toolchain, since it is already # available on the system, so force the site and source to be empty so # that nothing will be downloaded/extracted.
Signature · 1 line
1.9.1
PATCH v2 3/4 toolchain-external: define actual sources for arago toolchains
In reply to Luca Ceresoli--- toolchain/toolchain-external/toolchain-external.mk | 2 ++ 1 file changed, 2 insertions(+)
toolchain/toolchain-external/toolchain-external.mk +2 -0
@@ -296,6 +296,7 @@ TOOLCHAIN_EXTERNAL_SOURCE = arm-2014.05-29-arm-none-linux-gnueabi-i686-pc-linux- else ifeq ($(BR2_TOOLCHAIN_EXTERNAL_ARAGO_ARMV7A_201109),y) TOOLCHAIN_EXTERNAL_SITE = http://software-dl.ti.com/sdoemb/sdoemb_public_sw/arago_toolchain/2011_09/exports TOOLCHAIN_EXTERNAL_SOURCE = arago-2011.09-armv7a-linux-gnueabi-sdk.tar.bz2 +TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL = arago-toolchain-2011.09-sources.tar.bz2 define TOOLCHAIN_EXTERNAL_FIXUP_CMDS mv $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/arago-2011.09/armv7a/* $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/ rm -rf $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/arago-2011.09/
@@ -303,6 +304,7 @@ endef else ifeq ($(BR2_TOOLCHAIN_EXTERNAL_ARAGO_ARMV5TE_201109),y) TOOLCHAIN_EXTERNAL_SITE = http://software-dl.ti.com/sdoemb/sdoemb_public_sw/arago_toolchain/2011_09/exports TOOLCHAIN_EXTERNAL_SOURCE = arago-2011.09-armv5te-linux-gnueabi-sdk.tar.bz2 +TOOLCHAIN_EXTERNAL_ACTUAL_SOURCE_TARBALL = arago-toolchain-2011.09-sources.tar.bz2 define TOOLCHAIN_EXTERNAL_FIXUP_CMDS mv $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/arago-2011.09/armv5te/* $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/ rm -rf $(TOOLCHAIN_EXTERNAL_INSTALL_DIR)/arago-2011.09/
Signature · 1 line
1.9.1
PATCH v2 4/4 docs/manual: document new actual source package variables
In reply to Luca Ceresoli--- docs/manual/adding-packages-generic.txt | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
docs/manual/adding-packages-generic.txt +19 -0
@@ -388,6 +388,25 @@ information is (assuming the package name is +libfoo+) : to let you know, and +not saved+ will appear in the +license files+ field of the manifest file for this package. +* +LIBFOO_ACTUAL_SOURCE_TARBALL+ only applies to packages whose + +LIBFOO_SITE+ / +LIBTOO_SOURCE+ pair points to an archive that does + not actually contain source code, but binary code. This a very + uncommon case, only known to apply to external toolchains which come + already compiled, although theoretically it might apply to other + packages. In such cases a separate tarball is usually available with + the actual source code. Set +LIBFOO_ACTUAL_SOURCE_TARBALL+ to the + name of the actual source code archive and Buildroot will download + it and use it when you run +make legal-info+ to collect + legally-relevant material. Note this file will not be downloaded + during regular builds nor by +make source+. + +* +LIBFOO_ACTUAL_SOURCE_SITE+ provides the location of the actual + source tarball. The default value is +LIBFOO_SITE+, so you don't + need to set this variable if the binary and source archives are + hosted on the same directory. If +LIBFOO_ACTUAL_SOURCE_TARBALL+ is + not set, it doesn't make sense to define + +LIBFOO_ACTUAL_SOURCE_SITE+. + * +LIBFOO_REDISTRIBUTE+ can be set to +YES+ (default) or +NO+ to indicate if the package source code is allowed to be redistributed. Set it to +NO+ for non-opensource packages: Buildroot will not save the source code for this
Signature · 1 line
1.9.1
v2 cover legal-info: save the external-toolchain source archive
In reply to Luca CeresoliDear Luca Ceresoli,
Quoted 6 lines · Luca Ceresoli
Luca Ceresoli (4):
legal-info: allow to declare the actual sources for binary packages
toolchain-external: mass-define actual source tarball for known
patterns
toolchain-external: define actual sources for arago toolchains docs/manual: document new actual source package variables
All four patches applied, with only the removal of unneeded warning in the first patch.
We noted several things during the review:
* The ifneq ($$($(2)_SITE_METHOD),override) test will always be false because SITE_METHOD is never set to "override". This part of the code should be reworked to simply test $(2)_OVERRIDE_SRCDIR instead:
when empty, it indicates that the packages is neither a "local" site-method package nor a package that has its source directory overridden.
* Maybe the fetching of the "actual source" could be deferred to a separate $(1)-legal-info-source target, to refactor with the existing dependency of $(1)-legal-info on $(1)-source.
Thanks!
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Free Electrons Embedded Linux, Kernel and Android engineering http://free-electrons.com