LICENSE_FLAGS, a replacement for COMMERCIAL_LICENSE, v4

From: Tom Zanussi <[email protected]>

This patchset is a replacement for COMMERCIAL_LICENSE called LICENSE_FLAGS.

Please see the commit message for '[PATCH 1/5] base.bbclass: add support
for LICENSE_FLAGS' for an explanation of the LICENSE_FLAGS mechanism.

v4 changes, reflecting comments from Saul Wold:

- move the main functionality to license.bbclass as check_license_flags()
- keep the call to check_license_flags() in base.bbclass

v3 changes:

- add back an accidentally-stripped comment in PATCH 1.

v2 changes, reflecting comments from Phil Blundell and Paul Eggleton:

- This version converts all the existing packages listed in COMMERCIAL_LICENSE to the equivalent "commercial_${PN}" LICENSE_FLAGS. This allows each package to be added to or removed from the whitelist instead of the previously too-broad 'Commercial' flags for those packages.

- Changes all values to lowercase.

- The new commit message should explain the mechanism and how it can be used a little better.

For some background on these changes, the original proposal for the functionality covered by this replacement was drafted by Saul Wold - the relevant details of that proposal are copied below:

***

There has been some issues raised with the initial implementation of COMMERCIAL_LICENSE and we are looking for ways to address this. Currently COMMERCIAL_LICENSE (C_L) is defined in default-distrovars.conf to contain a list of packages that have additional license requirements when used commercially (such as royalty requirements, or acknowledging some type of commercial T&Cs). These packages are skipped during parsing.

It currently contains a number of Audio and Video packages that require additional licensing terms when used commercially. As we add additional layers, some of these layers want to add additional package to the C_L list, but how to easily enable them.

Since local.conf, where you would normally override things like this, is read in before base.bbclass, which contains tools like oe_filter_out() to modify lists, we can't use that mechanism.

That's the background, now for the proposal.

Do away with C_L and C_*_PLUGINS, move to a "Named Bit Flag" list in LICENSE_FLAGS, each recipe can then maintain their flags directly, instead of in a shared location like default-distrovars.conf.

LICENSE_FLAGS_WHITELIST would be set in local.conf with the values that are acceptable to include in this image, by default it would be blank.

Possible values for LICENSE_FLAGS could be:
- binary - provides some kind of binary with no source
- patent - provides a potential infringing item, that some may not want
- commercial - include recipes that may have commercial T&C
- commercial_${PN} - commercial licenses specific to ${PN}
- license_${PN} - include a recipe that has a specific license
- maybe similar or different than commercial_${PN}

***

[T&C = Terms and Conditions]

[NOTE: the above are only 'possible values' that particular license flags could take. The above are not proposals for specific flags that will be implemented - it's completely up to the package maintainers to define appropriate flags for their packages.]

Note that there's no policy attached to any of the above license types
- this is simply string-matching that can be used for the purpose of screening packages - if the strings match, the recipe gets in, if not, it doesn't i.e. during parsing, we would inspect the recipe's data for LICENSE_FLAGS and if it has a value then try to match against the WHITELIST - if it matches it gets added to the parsed list, if there is no match then it gets Skip_Package()'ed.

The following changes since commit 0f4d99d207b224bb9ce23de00a48f795ae20b3a0:
Richard Purdie (1):
multilib.conf: Clean up file and add missing entries for various dependencies

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib.git tzanussi/license-flags.v4
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/license-flags.v4

Tom Zanussi (5):
base.bbclass: add support for LICENSE_FLAGS
Add LICENSE_FLAGS to packages mentioned in COMMERCIAL_LICENSE base.bbclass: remove COMMERCIAL_LICENSE code default-distrovars.inc: remove COMMERCIAL_LICENSE et al documentation-audit.sh: remove COMMERCIAL_LICENSE warning

 meta/classes/base.bbclass                          |   12 ++++----
 meta/classes/license.bbclass                       |   30 ++++++++++++++++++++
 meta/conf/distro/include/default-distrovars.inc    |    5 ---
 .../gstreamer/gst-fluendo-mp3_0.10.16.bb           |    1 +
 .../gstreamer/gst-openmax_0.10.1.bb                |    1 +
 .../gstreamer/gst-plugins-ugly_0.10.18.bb          |    1 +
 meta/recipes-multimedia/lame/lame_3.99.3.bb        |    2 +
 meta/recipes-multimedia/libmad/libmad_0.15.1b.bb   |    1 +
 meta/recipes-multimedia/libomxil/libomxil_0.9.3.bb |    1 +
 meta/recipes-multimedia/mpeg2dec/mpeg2dec_0.4.1.bb |    1 +
 meta/recipes-qt/qt-apps/qmmp_0.5.2.bb              |    1 +
 scripts/contrib/documentation-audit.sh             |    3 +-
 12 files changed, 47 insertions(+), 12 deletions(-)

From: Tom Zanussi <[email protected]>

LICENSE_FLAGS are a per-recipe replacement for the COMMERCIAL_LICENSE mechanism.

In the COMMERCIAL_LICENSE mechanism, any package name mentioned in the global COMMERCIAL_LICENSE list is 'blacklisted' from being included in an image. To allow the blacklisted package into the image, the corresponding packages need to be removed from the COMMERCIAL_LICENSE list. This mechanism relies on a global list defined in default-distrovars.inc.

The LICENSE_FLAGS mechanism essentially implements the same thing but turns the global blacklist into a per-recipe whitelist. Any recipe can optionally define one or more 'license flags'; if defined, each of the license flags defined for a recipe must have matching entries in a global LICENSE_FLAGS_WHITELIST variable. Typically a recipe will have a single license flag specific to itself, which allows it to be individually toggled on and off. For example, a package named 'foo' might define a single license flag, 'commercial_foo':

LICENSE_FLAGS = "commercial_foo"

This says that in order for the foo package to be included in the image, the string 'commercial_foo' must appear in the LICENSE_FLAGS_WHITELIST variable:

LICENSE_FLAGS_WHITELIST = "commercial_foo"

Because the typical case is indeed to create LICENSE_FLAGS containing the package name, the LICENSE_FLAGS could just as well have been specified as:

LICENSE_FLAGS = "commercial_${PN}

which would pick up the package name automatically.

The mechanism has the word 'flags' in the name because although the typical case is to specify a single string to match as above, the user can add additional strings that might be thought of additional 'attributes' of a license that also need to be matched. This allows for the creation and specification of license categories that could be used to flexibly match sets of packages that match certain attributes without forcing them to all be specified individually. For example, a particular set of recipes that are typically used together might all contain a 'commercial_video' flag. Additionally, some of them might specify an additional 'binary' flag meaning that it's not possible to get the source for those packages. Specifying both 'commercial_video and binary' in the LICENSE_FLAGS_WHITELIST would allow them all to be pulled in, but if 'binary' was missing, it would only allow those packages that had source to be allowed in to the image.

The current behavior of COMMERCIAL_LICENSE is replicated as mentioned above by having the current set of COMMERCIAL_LICENSE flags implement their using LICENSE_FLAGS = "commercial_${PN}.

That being the case, the current COMMERCIAL_LICENSE can equivalently be specified in the new scheme by putting the below in local.conf:

# This is a list of packages that require a commercial license to ship

 # product. If shipped as part of an image these packages may have
 # implications so they are disabled by default.  To enable them,
 # un-comment the below as appropriate.
 #LICENSE_FLAGS_WHITELIST = "commercial_gst-fluendo-mp3 \
 #                           commercial_gst-openmax \
 #                           commercial_gst-plugins-ugly \
 #                           commercial_lame \
 #                           commercial_libmad \
 #                           commercial_libomxil \
 #                           commercial_mpeg2dec \
 #                           commercial_qmmp"

The above allows all of the current COMMERCIAL_LICENSE packages in - to disallow a particular package from appearing in the image, simply remove it from the whitelist.

---
 meta/classes/base.bbclass    |    7 +++++++
 meta/classes/license.bbclass |   30 ++++++++++++++++++++++++++++++
 2 files changed, 37 insertions(+), 0 deletions(-)

index f0c358e..085bb36 100644
@@ -349,6 +349,13 @@ python () {
     if license == "INVALID":
         bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
 
+    unmatched_license_flag = check_license_flags(d)
+    if unmatched_license_flag:
+        bb.debug(1, "Skipping %s because it has a restricted license (%s) not"
+             " whitelisted in LICENSE_FLAGS_WHITELIST" % (pn, unmatched_license_flag))
+        raise bb.parse.SkipPackage("because it has a restricted license (%s) not"
+             " whitelisted in LICENSE_FLAGS_WHITELIST" % unmatched_license_flag)
+
     commercial_license = " %s " % d.getVar('COMMERCIAL_LICENSE', 1)
     import re
     pnr = "[ \t]%s[ \t]" % pn.replace('+', "\+")

index 4b98e29..bc638fc 100644
@@ -282,6 +282,36 @@ def incompatible_license(d,dont_want_license):
 		return True
     return False
 
+
+def check_license_flags(d):
+    """
+    This function checks if a recipe has any LICENSE_FLAGs that aren't whitelisted.
+
+    If it does, it returns the first LICENSE_FLAG missing from the whitelist, or all the
+    the LICENSE_FLAGs if there is no whitelist.
+
+    If everything is is properly whitelisted, it returns None.
+    """
+
+    def all_license_flags_match(flags, whitelist):
+        """ Return first unmatched flag, None if all flags match """
+
+        for flag in flags.split():
+            if not flag in whitelist.split():
+                return flag
+        return None
+
+    license_flags = d.getVar('LICENSE_FLAGS', True)
+    if license_flags:
+        license_flags_whitelist = d.getVar('LICENSE_FLAGS_WHITELIST', True)
+        if not license_flags_whitelist:
+            return license_flags
+        unmatched_flag = all_license_flags_match(license_flags, license_flags_whitelist)
+        if unmatched_flag:
+            return unmatched_flag
+    return None
+
+
 SSTATETASKS += "do_populate_lic"
 do_populate_lic[sstate-name] = "populate-lic"
 do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"

1.7.0.4

From: Tom Zanussi <[email protected]>

Per-recipe LICENSE_FLAGS replace the global COMMERCIAL_LICENSE list; add LICENSE_FLAGS varables to each the recipes mentioned in that list:

- lame
- gst-fluendo-mp3
- gst-openmax
- gst-plugins-ugly
- libmad
- libomxil
- mpeg2dec
- qmmp

---
 .../gstreamer/gst-fluendo-mp3_0.10.16.bb           |    1 +
 .../gstreamer/gst-openmax_0.10.1.bb                |    1 +
 .../gstreamer/gst-plugins-ugly_0.10.18.bb          |    1 +
 meta/recipes-multimedia/lame/lame_3.99.3.bb        |    2 ++
 meta/recipes-multimedia/libmad/libmad_0.15.1b.bb   |    1 +
 meta/recipes-multimedia/libomxil/libomxil_0.9.3.bb |    1 +
 meta/recipes-multimedia/mpeg2dec/mpeg2dec_0.4.1.bb |    1 +
 meta/recipes-qt/qt-apps/qmmp_0.5.2.bb              |    1 +
 8 files changed, 9 insertions(+), 0 deletions(-)

index 5975513..2ce2993 100644
@@ -2,6 +2,7 @@ require gst-fluendo.inc
 
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://COPYING;md5=98326cbb1723a5a97e9b1db62e9faa05"
+LICENSE_FLAGS = "commercial_${PN}"
 
 acpaths = "-I ${S}/common/m4 -I ${S}/m4"
 

index 776ed1c..d2a56ce 100644
@@ -1,6 +1,7 @@
 DEPENDS = "gstreamer"
 RDEPENDS_${PN} = "libomxil"
 LICENSE = "LGPLv2.1"
+LICENSE_FLAGS = "commercial_${PN}"
 LIC_FILES_CHKSUM = "file://COPYING;md5=fbc093901857fcd118f065f900982c24 \
                     file://util/sem.h;beginline=1;endline=20;md5=accce5550d5583b839b441a0623f09fc"
 

index 2d7fa91..ff7e2d6 100644
@@ -1,6 +1,7 @@
 require gst-plugins.inc
 
 LICENSE = "GPLv2+ & LGPLv2.1+ & LGPLv2+"
+LICENSE_FLAGS = "commercial_${PN}"
 LIC_FILES_CHKSUM = "file://COPYING;md5=a6f89e2100d9b6cdffcea4f398e37343 \
                     file://gst/synaesthesia/synaescope.h;beginline=1;endline=20;md5=99f301df7b80490c6ff8305fcc712838 \
                     file://tests/check/elements/xingmux.c;beginline=1;endline=21;md5=4c771b8af188724855cb99cadd390068 \

index 3c42cc7..1e64b68 100644
@@ -3,6 +3,8 @@ HOMEPAGE = "http://sourceforge.net/projects/lame/files/lame/"
 BUGTRACKER = "http://sourceforge.net/tracker/?group_id=290&atid=100290"
 SECTION = "console/utils"
 LICENSE = "LGPLv2+"
+LICENSE_FLAGS = "commercial_${PN}"
+
 LIC_FILES_CHKSUM = "file://COPYING;md5=c46bda00ffbb0ba1dac22f8d087f54d9 \
                     file://include/lame.h;beginline=1;endline=20;md5=a2258182c593c398d15a48262130a92b
 PR = "r0"

index aec929c..7678b32 100644
@@ -2,6 +2,7 @@ DESCRIPTION = "MPEG Audio Decoder Library"
 HOMEPAGE = "http://sourceforge.net/projects/mad/"
 BUGTRACKER = "http://sourceforge.net/tracker/?group_id=12349&atid=112349"
 LICENSE = "GPLv2+"
+LICENSE_FLAGS = "commercial_${PN}"
 LIC_FILES_CHKSUM = "file://COPYING;md5=94d55d512a9ba36caa9b7df079bae19f \
 			file://COPYRIGHT;md5=8e55eb14894e782b84488d5a239bc23d \
 			file://version.h;beginline=1;endline=8;md5=aa07311dd39288d4349f28e1de516454"

index bb31c56..ad11013 100644
@@ -1,6 +1,7 @@
 DESCRIPTION = "Bellagio OpenMAX Integration Layer"
 HOMEPAGE = "http://omxil.sourceforge.net/"
 LICENSE = "LGPLv2.1+"
+LICENSE_FLAGS = "commercial_${PN}"
 LIC_FILES_CHKSUM = "file://COPYING;md5=ae6f0f4dbc7ac193b50f323a6ae191cb \
                     file://src/omxcore.h;beginline=1;endline=27;md5=806b1e5566c06486fe8e42b461e03a90"
 DEPENDS = "libvorbis libogg alsa-lib libmad"

index 351962f..ef59a55 100644
@@ -2,6 +2,7 @@ DESCRIPTION = "Library and test program for decoding mpeg-2 and mpeg-1 video str
 HOMEPAGE = "http://libmpeg2.sourceforge.net/"
 SECTION = "libs"
 LICENSE = "GPLv2+"
+LICENSE_FLAGS = "commercial_${PN}"
 LIC_FILES_CHKSUM = "file://COPYING;md5=94d55d512a9ba36caa9b7df079bae19f \
                     file://include/mpeg2.h;beginline=1;endline=22;md5=ead62602d4638329d3b5b86a55803154"
 

index b3ecc45..13a53c7 100644
@@ -1,6 +1,7 @@
 DESCRIPTION = "Qmmp (Qt-based Multimedia Player) is an audio-player, written with help of Qt library"
 HOMEPAGE = "http://qmmp.ylsoftware.com"
 LICENSE = "GPLv2"
+LICENSE_FLAGS = "commercial_${PN}"
 LIC_FILES_CHKSUM = "file://COPYING;md5=393a5ca445f6965873eca0259a17f833"
 SECTION = "multimedia"
 

1.7.0.4

From: Tom Zanussi <[email protected]>

The COMMERCIAL_LICENSE mechanism has been superseded by LICENSE_FLAGS so remove the code that implements COMMERCIAL_LICENSE.

---
 meta/classes/base.bbclass |    7 -------
 1 files changed, 0 insertions(+), 7 deletions(-)

index 085bb36..8898882 100644
@@ -356,13 +356,6 @@ python () {
         raise bb.parse.SkipPackage("because it has a restricted license (%s) not"
              " whitelisted in LICENSE_FLAGS_WHITELIST" % unmatched_license_flag)
 
-    commercial_license = " %s " % d.getVar('COMMERCIAL_LICENSE', 1)
-    import re
-    pnr = "[ \t]%s[ \t]" % pn.replace('+', "\+")
-    if commercial_license and re.search(pnr, commercial_license):
-        bb.debug(1, "Skipping %s because it's commercially licensed" % pn)
-        raise bb.parse.SkipPackage("because it may require a commercial license to ship in a product (listed in COMMERCIAL_LICENSE)")
-
     # If we're building a target package we need to use fakeroot (pseudo)
     # in order to capture permissions, owners, groups and special files
     if not bb.data.inherits_class('native', d) and not bb.data.inherits_class('cross', d):

1.7.0.4

From: Tom Zanussi <[email protected]>

The global COMMERCIAL_LICENSE mechanism has been obsoleted by per-recipe LICENSE_FLAGS, so remove the related variables.

---
 meta/conf/distro/include/default-distrovars.inc |    5 -----
 1 files changed, 0 insertions(+), 5 deletions(-)

index e1594f3..16b3108 100644
@@ -26,11 +26,6 @@ HOSTTOOLS_WHITELIST_GPLv3 ?= ""
 WHITELIST_GPLv3 ?= "less"
 LGPLv2_WHITELIST_GPLv3 ?= "libassuan gnutls libtasn1 libidn libgcc gcc-runtime" 
 
-# This is a list of packages that require a commercial license to ship
-# product. If shipped as part of an image these packages may have 
-# implications so they are disabled by default
-COMMERCIAL_LICENSE ?= "lame gst-fluendo-mp3 libmad mpeg2dec ffmpeg qmmp ${COMMERCIAL_LICENSE_DEPENDEES}"
-COMMERCIAL_LICENSE_DEPENDEES ?= "gst-plugins-ugly libomxil gst-openmax"
 COMMERCIAL_AUDIO_PLUGINS ?= ""
 # COMMERCIAL_AUDIO_PLUGINS ?= "gst-plugins-ugly-mad gst-plugins-ugly-mpegaudioparse"
 COMMERCIAL_VIDEO_PLUGINS ?= ""

1.7.0.4

From: Tom Zanussi <[email protected]>

COMMERCIAL_LICENSE no longer exists; the equivalent functionality is now has been replaced by LICENSE_FLAGS_WHITELIST, so replace the COMMERCIAL_LICENSE warning with a similarly equivalent warning.

---
 scripts/contrib/documentation-audit.sh |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

index 5070fee..5b66f03 100755
@@ -25,7 +25,8 @@ if [ -z "$BITBAKE" ]; then
 fi
 
 echo "REMINDER: you need to build for MACHINE=qemux86 or you won't get useful results"
-echo "REMINDER: you need to have COMMERCIAL_LICENSE = \"\" in local.conf or you'll get false positives"
+echo "REMINDER: you need to set LICENSE_FLAGS_WHITELIST appropriately in local.conf or "
+echo " you'll get false positives.  For example, LICENSE_FLAGS_WHITELIST = \"Commercial\""
 
 for pkg in `bitbake -s | awk '{ print \$1 }'`; do
 	if [[ "$pkg" == "Loading" || "$pkg" == "Loaded" ||

1.7.0.4