LICENSE_FLAGS, a replacement for COMMERCIAL_LICENSE, v5

From: Tom Zanussi <[email protected]>

This patchset is a replacement for COMMERCIAL_LICENSE called LICENSE_FLAGS.

Please see the commit message for '[PATCH 1/5] license.bbclass: add support
for LICENSE_FLAGS' for an explanation of the LICENSE_FLAGS mechanism.

v5 changes, reflecting comments from Richard Purdie:

- instead of having each recipe add _{PN} to its license flags, have the license code do that instead, and change the implementation to match
- update the commit descriptions and comments in the code to match

v4 changes, reflecting comments from Saul Wold:

- move the main functionality to license.bbclass as check_license_flags()
- keep the call to check_license_flags() in base.bbclass

v3 changes:

- add back an accidentally-stripped comment in PATCH 1.

v2 changes, reflecting comments from Phil Blundell and Paul Eggleton:

- This version converts all the existing packages listed in COMMERCIAL_LICENSE to the equivalent "commercial_${PN}" LICENSE_FLAGS. This allows each package to be added to or removed from the whitelist instead of the previously too-broad 'Commercial' flags for those packages.

- Changes all values to lowercase.

- The new commit message should explain the mechanism and how it can be used a little better.

For some background on these changes, the original proposal for the functionality covered by this replacement was drafted by Saul Wold - the relevant details of that proposal are copied below:

***

There has been some issues raised with the initial implementation of COMMERCIAL_LICENSE and we are looking for ways to address this. Currently COMMERCIAL_LICENSE (C_L) is defined in default-distrovars.conf to contain a list of packages that have additional license requirements when used commercially (such as royalty requirements, or acknowledging some type of commercial T&Cs). These packages are skipped during parsing.

It currently contains a number of Audio and Video packages that require additional licensing terms when used commercially. As we add additional layers, some of these layers want to add additional package to the C_L list, but how to easily enable them.

Since local.conf, where you would normally override things like this, is read in before base.bbclass, which contains tools like oe_filter_out() to modify lists, we can't use that mechanism.

That's the background, now for the proposal.

Do away with C_L and C_*_PLUGINS, move to a "Named Bit Flag" list in LICENSE_FLAGS, each recipe can then maintain their flags directly, instead of in a shared location like default-distrovars.conf.

LICENSE_FLAGS_WHITELIST would be set in local.conf with the values that are acceptable to include in this image, by default it would be blank.

Possible values for LICENSE_FLAGS could be:
- binary - provides some kind of binary with no source
- patent - provides a potential infringing item, that some may not want
- commercial - include recipes that may have commercial T&C
- commercial_${PN} - commercial licenses specific to ${PN}
- license_${PN} - include a recipe that has a specific license
- maybe similar or different than commercial_${PN}

***

[T&C = Terms and Conditions]

[NOTE: the above are only 'possible values' that particular license flags could take. The above are not proposals for specific flags that will be implemented - it's completely up to the package maintainers to define appropriate flags for their packages. Also the implementation now adds ${PN} automatically, so the specific potential values may be obsolete - the above is just for context.]

Note that there's no policy attached to any of the above license types
- this is simply string-matching that can be used for the purpose of screening packages - if the strings match, the recipe gets in, if not, it doesn't i.e. during parsing, we would inspect the recipe's data for LICENSE_FLAGS and if it has a value then try to match against the WHITELIST - if it matches it gets added to the parsed list, if there is no match then it gets Skip_Package()'ed.

The following changes since commit 967de59f35acef7fb258524973473f3d154e4a37:
Paul Eggleton (1):
buildhistory_analysis: include related fields in output

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib.git tzanussi/license-flags.v5
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/license-flags.v5

Tom Zanussi (5):
license.bbclass: add support for LICENSE_FLAGS
Add LICENSE_FLAGS to packages mentioned in COMMERCIAL_LICENSE base.bbclass: replace COMMERCIAL_LICENSE code with LICENSE_FLAGS code default-distrovars.inc: remove COMMERCIAL_LICENSE et al documentation-audit.sh: remove COMMERCIAL_LICENSE warning

 meta/classes/base.bbclass                          |   12 ++--
 meta/classes/license.bbclass                       |   63 ++++++++++++++++++++
 meta/conf/distro/include/default-distrovars.inc    |    5 --
 .../gstreamer/gst-fluendo-mp3_0.10.16.bb           |    1 +
 .../gstreamer/gst-openmax_0.10.1.bb                |    1 +
 .../gstreamer/gst-plugins-ugly_0.10.18.bb          |    1 +
 meta/recipes-multimedia/lame/lame_3.99.3.bb        |    2 +
 meta/recipes-multimedia/libmad/libmad_0.15.1b.bb   |    1 +
 meta/recipes-multimedia/libomxil/libomxil_0.9.3.bb |    1 +
 meta/recipes-multimedia/mpeg2dec/mpeg2dec_0.4.1.bb |    1 +
 meta/recipes-qt/qt-apps/qmmp_0.5.2.bb              |    1 +
 scripts/contrib/documentation-audit.sh             |    3 +-
 12 files changed, 80 insertions(+), 12 deletions(-)

From: Tom Zanussi <[email protected]>

LICENSE_FLAGS are a per-recipe replacement for the COMMERCIAL_LICENSE mechanism.

In the COMMERCIAL_LICENSE mechanism, any package name mentioned in the global COMMERCIAL_LICENSE list is 'blacklisted' from being included in an image. To allow the blacklisted package into the image, the corresponding packages need to be removed from the COMMERCIAL_LICENSE list. This mechanism relies on a global list defined in default-distrovars.inc.

The LICENSE_FLAGS mechanism essentially implements the same thing but turns the global blacklist into a per-recipe whitelist. Any recipe can optionally define one or more 'license flags'; if defined, each of the license flags defined for a recipe must have matching entries in a global LICENSE_FLAGS_WHITELIST variable.

The definition of 'matching' is simple, but there are a couple things users need to know in order to correctly and effectively use it. Before we test a flag against the whitelist, we append _${PN} to it, thus automatically making each LICENSE_FLAG recipe-specific. We then try to match that string against the whitelist. So if the user specifies LICENSE_FLAGS = 'commercial' for recipe 'foo', the string 'commercial_foo' should be specified in the whitelist in order for it to match.

However, the user can also broaden the match by putting any '_'-separated beginning subset of a LICENSE_FLAG in the whitelist, which will also match e.g. simply specifying 'commercial' in the whitelist would match any expanded LICENSE_FLAG starting with 'commercial' such as 'commercial_foo' and 'commercial_bar' which are the strings that would have been automatically generated if those recipes had simply specified LICENSE_FLAGS = 'commercial'

This allows for a range of specificity for the items in the whitelist, from more general to perfectly specific. So users have the choice of exhaustively enumerating each license flag in the whitelist to allow only those specific recipes into the image, or of using a more general string to pick up anything matching just the first component(s).

Note that this scheme works even if the flag already has _pn appended
- the extra _pn is redundant, but doesn't affect the outcome e.g. a license flag of 'commercial_1.2_foo' would turn into 'commercial_1.2_foo_foo' and would match both the general 'commercial' and the specific 'commercial_1.2_foo' as expected (it would also match commercial_1.2_foo_foo' and 'commercial_1.2', which don't make much sense as far as something a user would think of specifying in the whitelist). For a versioned string, the user could instead specify 'commercial_foo_1.2', which would turn into 'commercial_foo_1.2_foo', but which would as expected allow the user to pick up this package along with anything else 'commercial' by specifying 'commercial' in the whitelist, or anything with a 'commercial_foo' license regardless of version by using 'commercial_foo' in the whitelist, or 'commercial_foo_1.1' to be completely specific about package and version.

The current behavior of COMMERCIAL_LICENSE is replicated as mentioned above by having the current set of COMMERCIAL_LICENSE flags implemented using LICENSE_FLAGS = "commercial".

That being the case, the current COMMERCIAL_LICENSE can equivalently be specified in the new scheme by putting the below in local.conf:

# This is a list of packages that require a commercial license to ship

 # product. If shipped as part of an image these packages may have
 # implications so they are disabled by default.  To enable them,
 # un-comment the below as appropriate.
 #LICENSE_FLAGS_WHITELIST = "commercial_gst-fluendo-mp3 \
 #                           commercial_gst-openmax \
 #                           commercial_gst-plugins-ugly \
 #                           commercial_lame \
 #                           commercial_libmad \
 #                           commercial_libomxil \
 #                           commercial_mpeg2dec \
 #                           commercial_qmmp"

The above allows all of the current COMMERCIAL_LICENSE packages in - to disallow a particular package from appearing in the image, simply remove it from the whitelist. To allow them all in, you could also specify LICENSE_FLAGS_WHITELIST = "commercial".

---
 meta/classes/license.bbclass |   63 ++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 63 insertions(+), 0 deletions(-)

index 4b98e29..10a937b 100644
@@ -282,6 +282,69 @@ def incompatible_license(d,dont_want_license):
 		return True
     return False
 
+
+def check_license_flags(d):
+    """
+    This function checks if a recipe has any LICENSE_FLAGs that
+    aren't whitelisted.
+
+    If it does, it returns the first LICENSE_FLAG missing from the
+    whitelist, or all the LICENSE_FLAGs if there is no whitelist.
+
+    If everything is is properly whitelisted, it returns None.
+    """
+
+    def license_flag_matches(flag, whitelist, pn):
+        """
+        Return True if flag matches something in whitelist, None if not.
+
+        Before we test a flag against the whitelist, we append _${PN}
+        to it.  We then try to match that string against the
+        whitelist.  This covers the normal case, where we expect
+        LICENSE_FLAGS to be a simple string like 'commercial', which
+        the user typically matches exactly in the whitelist by
+        explicitly appending the package name e.g 'commercial_foo'.
+        If we fail the match however, we then split the flag across
+        '_' and append each fragment and test until we either match or
+        run out of fragments.
+        """
+        flag_pn = ("%s_%s" % (flag, pn))
+        for candidate in whitelist:
+            if flag_pn == candidate:
+                    return True
+
+        flag_cur = ""
+        flagments = flag_pn.split("_")
+        flagments.pop() # we've already tested the full string
+        for flagment in flagments:
+            if flag_cur:
+                flag_cur += "_"
+            flag_cur += flagment
+            for candidate in whitelist:
+                if flag_cur == candidate:
+                    return True
+        return False
+
+    def all_license_flags_match(license_flags, whitelist):
+        """ Return first unmatched flag, None if all flags match """
+        pn = d.getVar('PN', True)
+        split_whitelist = whitelist.split()
+        for flag in license_flags.split():
+            if not license_flag_matches(flag, split_whitelist, pn):
+                return flag
+        return None
+
+    license_flags = d.getVar('LICENSE_FLAGS', True)
+    if license_flags:
+        whitelist = d.getVar('LICENSE_FLAGS_WHITELIST', True)
+        if not whitelist:
+            return license_flags
+        unmatched_flag = all_license_flags_match(license_flags, whitelist)
+        if unmatched_flag:
+            return unmatched_flag
+    return None
+
+
 SSTATETASKS += "do_populate_lic"
 do_populate_lic[sstate-name] = "populate-lic"
 do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}"

1.7.0.4

From: Tom Zanussi <[email protected]>

Per-recipe LICENSE_FLAGS replace the global COMMERCIAL_LICENSE list; add LICENSE_FLAGS varables to each the recipes mentioned in that list:

- lame
- gst-fluendo-mp3
- gst-openmax
- gst-plugins-ugly
- libmad
- libomxil
- mpeg2dec
- qmmp

---
 .../gstreamer/gst-fluendo-mp3_0.10.16.bb           |    1 +
 .../gstreamer/gst-openmax_0.10.1.bb                |    1 +
 .../gstreamer/gst-plugins-ugly_0.10.18.bb          |    1 +
 meta/recipes-multimedia/lame/lame_3.99.3.bb        |    2 ++
 meta/recipes-multimedia/libmad/libmad_0.15.1b.bb   |    1 +
 meta/recipes-multimedia/libomxil/libomxil_0.9.3.bb |    1 +
 meta/recipes-multimedia/mpeg2dec/mpeg2dec_0.4.1.bb |    1 +
 meta/recipes-qt/qt-apps/qmmp_0.5.2.bb              |    1 +
 8 files changed, 9 insertions(+), 0 deletions(-)

index 5975513..3f8c67e 100644
@@ -2,6 +2,7 @@ require gst-fluendo.inc
 
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://COPYING;md5=98326cbb1723a5a97e9b1db62e9faa05"
+LICENSE_FLAGS = "commercial"
 
 acpaths = "-I ${S}/common/m4 -I ${S}/m4"
 

index b4fe52d..76c8135 100644
@@ -1,6 +1,7 @@
 DEPENDS = "gstreamer"
 RDEPENDS_${PN} = "libomxil"
 LICENSE = "LGPLv2.1"
+LICENSE_FLAGS = "commercial"
 LIC_FILES_CHKSUM = "file://COPYING;md5=fbc093901857fcd118f065f900982c24 \
                     file://util/sem.h;beginline=1;endline=20;md5=accce5550d5583b839b441a0623f09fc"
 

index 2d7fa91..72a8151 100644
@@ -1,6 +1,7 @@
 require gst-plugins.inc
 
 LICENSE = "GPLv2+ & LGPLv2.1+ & LGPLv2+"
+LICENSE_FLAGS = "commercial"
 LIC_FILES_CHKSUM = "file://COPYING;md5=a6f89e2100d9b6cdffcea4f398e37343 \
                     file://gst/synaesthesia/synaescope.h;beginline=1;endline=20;md5=99f301df7b80490c6ff8305fcc712838 \
                     file://tests/check/elements/xingmux.c;beginline=1;endline=21;md5=4c771b8af188724855cb99cadd390068 \

index 3c42cc7..4dc4e0a 100644
@@ -3,6 +3,8 @@ HOMEPAGE = "http://sourceforge.net/projects/lame/files/lame/"
 BUGTRACKER = "http://sourceforge.net/tracker/?group_id=290&atid=100290"
 SECTION = "console/utils"
 LICENSE = "LGPLv2+"
+LICENSE_FLAGS = "commercial"
+
 LIC_FILES_CHKSUM = "file://COPYING;md5=c46bda00ffbb0ba1dac22f8d087f54d9 \
                     file://include/lame.h;beginline=1;endline=20;md5=a2258182c593c398d15a48262130a92b
 PR = "r0"

index aec929c..e10b401 100644
@@ -2,6 +2,7 @@ DESCRIPTION = "MPEG Audio Decoder Library"
 HOMEPAGE = "http://sourceforge.net/projects/mad/"
 BUGTRACKER = "http://sourceforge.net/tracker/?group_id=12349&atid=112349"
 LICENSE = "GPLv2+"
+LICENSE_FLAGS = "commercial"
 LIC_FILES_CHKSUM = "file://COPYING;md5=94d55d512a9ba36caa9b7df079bae19f \
 			file://COPYRIGHT;md5=8e55eb14894e782b84488d5a239bc23d \
 			file://version.h;beginline=1;endline=8;md5=aa07311dd39288d4349f28e1de516454"

index bb31c56..ded0e1c 100644
@@ -1,6 +1,7 @@
 DESCRIPTION = "Bellagio OpenMAX Integration Layer"
 HOMEPAGE = "http://omxil.sourceforge.net/"
 LICENSE = "LGPLv2.1+"
+LICENSE_FLAGS = "commercial"
 LIC_FILES_CHKSUM = "file://COPYING;md5=ae6f0f4dbc7ac193b50f323a6ae191cb \
                     file://src/omxcore.h;beginline=1;endline=27;md5=806b1e5566c06486fe8e42b461e03a90"
 DEPENDS = "libvorbis libogg alsa-lib libmad"

index 351962f..7ff8a03 100644
@@ -2,6 +2,7 @@ DESCRIPTION = "Library and test program for decoding mpeg-2 and mpeg-1 video str
 HOMEPAGE = "http://libmpeg2.sourceforge.net/"
 SECTION = "libs"
 LICENSE = "GPLv2+"
+LICENSE_FLAGS = "commercial"
 LIC_FILES_CHKSUM = "file://COPYING;md5=94d55d512a9ba36caa9b7df079bae19f \
                     file://include/mpeg2.h;beginline=1;endline=22;md5=ead62602d4638329d3b5b86a55803154"
 

index b3ecc45..469351c 100644
@@ -1,6 +1,7 @@
 DESCRIPTION = "Qmmp (Qt-based Multimedia Player) is an audio-player, written with help of Qt library"
 HOMEPAGE = "http://qmmp.ylsoftware.com"
 LICENSE = "GPLv2"
+LICENSE_FLAGS = "commercial"
 LIC_FILES_CHKSUM = "file://COPYING;md5=393a5ca445f6965873eca0259a17f833"
 SECTION = "multimedia"
 

1.7.0.4

From: Tom Zanussi <[email protected]>

The COMMERCIAL_LICENSE mechanism has been superseded by LICENSE_FLAGS so remove the code that implements COMMERCIAL_LICENSE and replace it with the corresponding LICENSE_FLAGS version.

---
 meta/classes/base.bbclass |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

index 091e965..bb3c38d 100644
@@ -349,12 +349,12 @@ python () {
     if license == "INVALID":
         bb.fatal('This recipe does not have the LICENSE field set (%s)' % pn)
 
-    commercial_license = " %s " % d.getVar('COMMERCIAL_LICENSE', 1)
-    import re
-    pnr = "[ \t]%s[ \t]" % pn.replace('+', "\+")
-    if commercial_license and re.search(pnr, commercial_license):
-        bb.debug(1, "Skipping %s because it's commercially licensed" % pn)
-        raise bb.parse.SkipPackage("because it may require a commercial license to ship in a product (listed in COMMERCIAL_LICENSE)")
+    unmatched_license_flag = check_license_flags(d)
+    if unmatched_license_flag:
+        bb.debug(1, "Skipping %s because it has a restricted license not"
+             " whitelisted in LICENSE_FLAGS_WHITELIST" % pn)
+        raise bb.parse.SkipPackage("because it has a restricted license not"
+             " whitelisted in LICENSE_FLAGS_WHITELIST")
 
     # If we're building a target package we need to use fakeroot (pseudo)
     # in order to capture permissions, owners, groups and special files

1.7.0.4

From: Tom Zanussi <[email protected]>

The global COMMERCIAL_LICENSE mechanism has been obsoleted by per-recipe LICENSE_FLAGS, so remove the related variables.

---
 meta/conf/distro/include/default-distrovars.inc |    5 -----
 1 files changed, 0 insertions(+), 5 deletions(-)

index e1594f3..16b3108 100644
@@ -26,11 +26,6 @@ HOSTTOOLS_WHITELIST_GPLv3 ?= ""
 WHITELIST_GPLv3 ?= "less"
 LGPLv2_WHITELIST_GPLv3 ?= "libassuan gnutls libtasn1 libidn libgcc gcc-runtime" 
 
-# This is a list of packages that require a commercial license to ship
-# product. If shipped as part of an image these packages may have 
-# implications so they are disabled by default
-COMMERCIAL_LICENSE ?= "lame gst-fluendo-mp3 libmad mpeg2dec ffmpeg qmmp ${COMMERCIAL_LICENSE_DEPENDEES}"
-COMMERCIAL_LICENSE_DEPENDEES ?= "gst-plugins-ugly libomxil gst-openmax"
 COMMERCIAL_AUDIO_PLUGINS ?= ""
 # COMMERCIAL_AUDIO_PLUGINS ?= "gst-plugins-ugly-mad gst-plugins-ugly-mpegaudioparse"
 COMMERCIAL_VIDEO_PLUGINS ?= ""

1.7.0.4

From: Tom Zanussi <[email protected]>

COMMERCIAL_LICENSE no longer exists; the equivalent functionality is now has been replaced by LICENSE_FLAGS_WHITELIST, so replace the COMMERCIAL_LICENSE warning with a similarly equivalent warning.

---
 scripts/contrib/documentation-audit.sh |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

index 5070fee..5b66f03 100755
@@ -25,7 +25,8 @@ if [ -z "$BITBAKE" ]; then
 fi
 
 echo "REMINDER: you need to build for MACHINE=qemux86 or you won't get useful results"
-echo "REMINDER: you need to have COMMERCIAL_LICENSE = \"\" in local.conf or you'll get false positives"
+echo "REMINDER: you need to set LICENSE_FLAGS_WHITELIST appropriately in local.conf or "
+echo " you'll get false positives.  For example, LICENSE_FLAGS_WHITELIST = \"Commercial\""
 
 for pkg in `bitbake -s | awk '{ print \$1 }'`; do
 	if [[ "$pkg" == "Loading" || "$pkg" == "Loaded" ||

1.7.0.4

Merged into OE-Core

Thanks
Sau!