legal-info: extract packages to get license files (branch yem/legal)

From: "Yann E. MORIN" <[email protected]>

Hello All!

Here is a series that fixes the legal-info infra:
- extracts sources before copying license files
- saves license files even for no-redistribute packages
- cleans up to account for the changes (comment and helper rename)

Changes v5 -> v6:
- fix commit log about what is actually fixed (Luca)
- use a proper name for the renamed macro (Luca)
- fix rebasing after the $$ mess ;-)
- drop patch 5, it's no longer needed

Changes v4 -> v5:
- don't warn about missing license fioles for packages bundled with Buildroot (Luca)
- new patch 5/5 to display a proper error message for missing license files
- typoes

Changes v3 -> v4:
- save licens files for no-redistribute packages (Luca)
- download sources before copying to legal-info/sources/
- rename a legal-info helper

Changes v2 -> v3:
- fix manifest for no-redistribute, or overriden, or local packages

Regards,
Yann E. MORIN.

The following changes since commit 8d24201f4b943b7a1edd9a16c97155f1a2efda59:

imagemagick: bump to version 6.8.9-3 (2014-06-21 19:38:07 +0200)

are available in the git repository at:

git://gitorious.org/buildroot/buildroot.git yem/legal

for you to fetch changes up to 1b7669d8f5b489b6c8b2b3394ba1207c903a9e68:

legal-info: rename legal-warning-pkg-savednothing helper (2014-06-22 11:19:31 +0200)

----------------------------------------------------------------
Yann E. MORIN (4):
legal-info: extract even no-redistribute packages
legal-info: save license files even for local or overridden packages
legal-info: add a comment about what packages we save the tarballs of
legal-info: rename legal-warning-pkg-savednothing helper

 package/pkg-generic.mk | 39 +++++++++++++++++++++++++++------------
 package/pkg-utils.mk   |  4 ++--
 2 files changed, 29 insertions(+), 14 deletions(-)

.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

From: "Yann E. MORIN" <[email protected]>

Currently, if a package is marked _REDISTRIBUTE = NO, then legal-info will not try to extract it first.

If that package also declares some _LICENSE_FILES, legal-info fails if it is the only action we're trying to run:

    $ cat defconfig
    BR2_arm=y
    BR2_TOOLCHAIN_BUILDROOT_EGLIBC=y
    BR2_PACKAGE_LIBFSLCODEC=y
    $ make BR2_DEFCONFIG=$(pwd)/defconfig defconfig
    $ make legal-info
    [--SNIP--]
    cat: /home/ymorin/dev/buildroot/O/build/libfslcodec-3.5.7-1.0.0/EULA: No such file or directory

Fix this by always having legal-info extract the archives if one or more _LICENSE_FILES are specified.

We do this for all types of packages: overridden, local or 'normal' remote packages. Even though we do not save the sources for the overridden or local packages, we need to save their licensing info, so we need to extract them.

This implies that we now need only PKG-source, not PKG-extract anymore, as a dependency of legal-info for packages we want to save (ie. redistributable, non-local and non-overriden packages.)

---
Chamges v5 -> v6:
- fix commit log about the actual case that is being solved (Luca)
- fix the comment about needing only PKG-source (Luca)

Changes v4 -> v5:
- my usual s/pacakge/package/

Changes v3 -> v4:
- legal-info needs to depend on PKG-source when it needs to save a package's tarball

Changes v2 -> v3:
- don't include source URL of no-redistribute, or overriden, or local packages in the manifest

Changes v1 -> v2:
- this is not fixing the autobuilders failure it was written to fix, so remove the references to such build failures (Thomas P)
- also extract overriden and local packages (Fabio)

---
 package/pkg-generic.mk | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

index 457d873..5f2a27e 100644
@@ -620,11 +620,18 @@ $(2)_MANIFEST_LICENSE_FILES = $$($(2)_LICENSE_FILES)
 endif
 $(2)_MANIFEST_LICENSE_FILES ?= not saved
 
+# If the package declares _LICENSE_FILES, we need to extract it,
+# for overriden, local or normal remote packages alike, whether
+# we want to redistribute it or not.
+ifneq ($$($(2)_LICENSE_FILES),)
+$(1)-legal-info: $(1)-extract
+endif
+
 ifeq ($$($(2)_REDISTRIBUTE),YES)
 ifneq ($$($(2)_SITE_METHOD),local)
 ifneq ($$($(2)_SITE_METHOD),override)
-# Packages that have a tarball need it downloaded and extracted beforehand
-$(1)-legal-info: $(1)-extract $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))
+# Packages that have a tarball need it downloaded beforehand
+$(1)-legal-info: $(1)-source $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4)))
 $(2)_MANIFEST_TARBALL = $$($(2)_SOURCE)
 endif
 endif

1.8.3.2

Dear Yann E. MORIN,

Applied, thanks.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

From: "Yann E. MORIN" <[email protected]>

Even if we do not save the sources for local or overridden packages because it is too complex, we can still quite easily save the license files.

Also, having the license files is a very important part of complying with the licenses.

Move the copy of license files out of the non-local, non-overridden package case, but still in the case where packages have a _SOURCE defined, to avoid catching packages bundled in Buildroot (eg. mkpasswd et al.)

---
Changes v5 -> v6:
- typoes (Luca)
- fix the rebasing mess after the $$ conversion

Changes v4 -> v5:
- avoid warning for packages bundled in Buildroot (Luca)
- the change is not about non-redistributable packages, but about local and overriden packages

---
 package/pkg-generic.mk | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

index 5f2a27e..e8f2f9e 100644
@@ -644,6 +644,19 @@ $(1)-legal-info:
 	$$(foreach hook,$$($(2)_PRE_LEGAL_INFO_HOOKS),$$(call $$(hook))$$(sep))
 ifneq ($$(call qstrip,$$($(2)_SOURCE)),)
 
+# Save license files if defined
+# We save the license files for any kind of package: normal, local,
+# overridden, or non-redistributable alike.
+# The reason to save license files even for no-redistribute packages
+# is that the license still applies to the files distributed as part
+# of the rootfs, even if the sources are not themselves redistributed.
+ifeq ($$(call qstrip,$$($(2)_LICENSE_FILES)),)
+	@$$(call legal-license-nofiles,$$($(2)_RAWNAME),$$(call UPPERCASE,$(4)))
+	@$$(call legal-warning-pkg,$$($(2)_RAWNAME),cannot save license ($(2)_LICENSE_FILES not defined))
+else
+	@$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$($(2)_RAWNAME),$$(F),$$($(2)_DIR)/$$(F),$$(call UPPERCASE,$(4)))$$(sep))
+endif # license files
+
 ifeq ($$($(2)_SITE_METHOD),local)
 # Packages without a tarball: don't save and warn
 	@$$(call legal-warning-pkg-savednothing,$$($(2)_RAWNAME),local)

@@ -654,14 +667,6 @@ else ifneq ($$($(2)_OVERRIDE_SRCDIR),)
 else
 # Other packages
 
-# Save license files if defined
-ifeq ($$(call qstrip,$$($(2)_LICENSE_FILES)),)
-	@$$(call legal-license-nofiles,$$($(2)_RAWNAME),$$(call UPPERCASE,$(4)))
-	@$$(call legal-warning-pkg,$$($(2)_RAWNAME),cannot save license ($(2)_LICENSE_FILES not defined))
-else
-	@$$(foreach F,$$($(2)_LICENSE_FILES),$$(call legal-license-file,$$($(2)_RAWNAME),$$(F),$$($(2)_DIR)/$$(F),$$(call UPPERCASE,$(4)))$$(sep))
-endif # license files
-
 ifeq ($$($(2)_REDISTRIBUTE),YES)
 # Copy the source tarball (just hardlink if possible)
 	@cp -l $$(DL_DIR)/$$($(2)_SOURCE) $$(REDIST_SOURCES_DIR_$$(call UPPERCASE,$(4))) 2>/dev/null || \

1.8.3.2

Dear Yann E. MORIN,

Applied, thanks.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

From: "Yann E. MORIN" <[email protected]>

As the legal-info infra only (rightfully) saves the tarballs of packages that:
- we want to redistribute,
- and are not local,
- and are not overriden,

add a comment stating so.

This should clarify the code-block, which although trivial to read, was not easy to interpret without thinking thouroughly about it.

---
 package/pkg-generic.mk | 3 +++
 1 file changed, 3 insertions(+)

index e8f2f9e..55936d2 100644
@@ -627,6 +627,9 @@ ifneq ($$($(2)_LICENSE_FILES),)
 $(1)-legal-info: $(1)-extract
 endif
 
+# We only save the sources of packages we want to redistribute, that are
+# non-local, and non-overriden. So only store, in the manifest, the tarball
+# name of those packages.
 ifeq ($$($(2)_REDISTRIBUTE),YES)
 ifneq ($$($(2)_SITE_METHOD),local)
 ifneq ($$($(2)_SITE_METHOD),override)

1.8.3.2

Dear Yann E. MORIN,

Applied, thanks.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

From: "Yann E. MORIN" <[email protected]>

This helper was called when none of the sources or license files were saved.

Now we handle license files separately from the sources, this is no longer the case: they are only called when the sources are not saved.

Rename the handler and change the warning message accordingly.

---
Changes v5 -> v6:
- fix the silly new name of the macro (Luca)

---
 package/pkg-generic.mk | 4 ++--
 package/pkg-utils.mk   | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

index 55936d2..97c8416 100644
@@ -662,10 +662,10 @@ endif # license files
 
 ifeq ($$($(2)_SITE_METHOD),local)
 # Packages without a tarball: don't save and warn
-	@$$(call legal-warning-pkg-savednothing,$$($(2)_RAWNAME),local)
+	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),local)
 
 else ifneq ($$($(2)_OVERRIDE_SRCDIR),)
-	@$$(call legal-warning-pkg-savednothing,$$($(2)_RAWNAME),override)
+	@$$(call legal-warning-nosource,$$($(2)_RAWNAME),override)
 
 else
 # Other packages

index 163ef8a..59a5ab2 100644
@@ -101,8 +101,8 @@ define legal-warning-pkg # pkg, text
 	echo "WARNING: $(1): $(2)" >>$(LEGAL_WARNINGS)
 endef
 
-define legal-warning-pkg-savednothing # pkg, {local|override}
-	$(call legal-warning-pkg,$(1),sources and license files not saved ($(2) packages not handled))
+define legal-warning-nosource # pkg, {local|override}
+	$(call legal-warning-pkg,$(1),sources not saved ($(2) packages not handled))
 endef
 
 define legal-manifest # pkg, version, license, license-files, source, {HOST|TARGET}

1.8.3.2

Dear Yann E. MORIN,

Applied, thanks.

Thomas

Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com