license_class: Added support for LICENSE_PRIORITY in manifest creation
cover license_class: Added support for LICENSE_PRIORITY in manifest creation
Now you can specify LICENSE_PRIOTIY that enables the distro to set the license preferences for manifest creation.
The major change is when you have OR'ed LICENSE only one is choose according LICENSE_PRIORITY preferences if LICENSE_PRIORITY isn't specified the left one is choosen.
The test was done building qemux86/core-image-sato with and without these set of patches and then run diff recursive under common-licenses created directory.
You can find a tarball with common-license directories, resulted diff and local.conf at:
https://bugzilla.yoctoproject.org/attachment.cgi?id=2223
Aníbal Limón (4):
license_class: Reimplemented manifest creation in python
license_class: Added support for INCOMPATIBLE_LICENSE into
license_create_manifest
license_class: Added LICENSE_PRIORITY support
license_class: Fix remove + trim in license_create_manifest.
meta/classes/license.bbclass | 250 ++++++++++++++++++++++++++++--------------- meta/conf/documentation.conf | 1 + 2 files changed, 164 insertions(+), 87 deletions(-)
Signature · 1 line
1.9.1
PATCH 1/4 license_class: Reimplemented manifest creation in python
Reimplemented license_manifest_create from shell to python, in order to use oe.license module for handle INCOMPATIBLE_LICENSE and add License prorities into OR's evaluation.
--- meta/classes/license.bbclass | 161 ++++++++++++++++++++++++------------------- 1 file changed, 90 insertions(+), 71 deletions(-)
meta/classes/license.bbclass +90 -71
@@ -25,78 +25,97 @@ python write_package_manifest() { 'w+').write(image_list_installed_packages(d)) } -license_create_manifest() { - # Test if BUILD_IMAGES_FROM_FEEDS is defined in env - if [ -n "${BUILD_IMAGES_FROM_FEEDS}" ]; then - exit 0 - fi - - INSTALLED_PKGS=`cat ${LICENSE_DIRECTORY}/${IMAGE_NAME}/package.manifest` - LICENSE_MANIFEST="${LICENSE_DIRECTORY}/${IMAGE_NAME}/license.manifest" - # remove existing license.manifest file - if [ -f ${LICENSE_MANIFEST} ]; then - rm ${LICENSE_MANIFEST} - fi - touch ${LICENSE_MANIFEST} - for pkg in ${INSTALLED_PKGS}; do - filename=`ls ${PKGDATA_DIR}/runtime-reverse/${pkg}| head -1` - pkged_pn="$(sed -n 's/^PN: //p' ${filename})" - - # check to see if the package name exists in the manifest. if so, bail. - if grep -q "^PACKAGE NAME: ${pkg}" ${LICENSE_MANIFEST}; then - continue - fi - - pkged_pv="$(sed -n 's/^PV: //p' ${filename})" - pkged_name="$(basename $(readlink ${filename}))" - pkged_lic="$(sed -n "/^LICENSE_${pkged_name}: /{ s/^LICENSE_${pkged_name}: //; s/[|&()*]/ /g; s/ */ /g; p }" ${filename})" - if [ -z ${pkged_lic} ]; then - # fallback checking value of LICENSE - pkged_lic="$(sed -n "/^LICENSE: /{ s/^LICENSE: //; s/[|&()*]/ /g; s/ */ /g; p }" ${filename})" - fi - - echo "PACKAGE NAME:" ${pkg} >> ${LICENSE_MANIFEST} - echo "PACKAGE VERSION:" ${pkged_pv} >> ${LICENSE_MANIFEST} - echo "RECIPE NAME:" ${pkged_pn} >> ${LICENSE_MANIFEST} - printf "LICENSE:" >> ${LICENSE_MANIFEST} - for lic in ${pkged_lic}; do - # to reference a license file trim trailing + symbol - if ! [ -e "${LICENSE_DIRECTORY}/${pkged_pn}/generic_${lic%+}" ]; then - bbwarn "The license listed ${lic} was not in the licenses collected for ${pkged_pn}" - fi - printf " ${lic}" >> ${LICENSE_MANIFEST} - done - printf "\n\n" >> ${LICENSE_MANIFEST} - done - - # Two options here: - # - Just copy the manifest - # - Copy the manifest and the license directories - # With both options set we see a .5 M increase in core-image-minimal - if [ "${COPY_LIC_MANIFEST}" = "1" ]; then - mkdir -p ${IMAGE_ROOTFS}/usr/share/common-licenses/ - cp ${LICENSE_MANIFEST} ${IMAGE_ROOTFS}/usr/share/common-licenses/license.manifest - if [ "${COPY_LIC_DIRS}" = "1" ]; then - for pkg in ${INSTALLED_PKGS}; do - mkdir -p ${IMAGE_ROOTFS}/usr/share/common-licenses/${pkg} - pkged_pn="$(oe-pkgdata-util lookup-recipe ${PKGDATA_DIR} ${pkg})" - for lic in `ls ${LICENSE_DIRECTORY}/${pkged_pn}`; do - # Really don't need to copy the generics as they're - # represented in the manifest and in the actual pkg licenses - # Doing so would make your image quite a bit larger - if [ "${lic#generic_}" = "${lic}" ]; then - cp ${LICENSE_DIRECTORY}/${pkged_pn}/${lic} ${IMAGE_ROOTFS}/usr/share/common-licenses/${pkg}/${lic} - else - if [ ! -f ${IMAGE_ROOTFS}/usr/share/common-licenses/${lic} ]; then - cp ${LICENSE_DIRECTORY}/${pkged_pn}/${lic} ${IMAGE_ROOTFS}/usr/share/common-licenses/ - fi - ln -sf ../${lic} ${IMAGE_ROOTFS}/usr/share/common-licenses/${pkg}/${lic} - fi - done - done - fi - fi +python license_create_manifest() { + import shutil + import re + build_images_from_feeds = d.getVar('BUILD_IMAGES_FROM_FEEDS', True) + if build_images_from_feeds == "1": + return 0 + + pkg_dic = {} + package_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + d.getVar('IMAGE_NAME', True), 'package.manifest') + with open(package_manifest, "r") as package_file: + pkg_list = package_file.read().split() + for pkg in pkg_list: + pkg_info = os.path.join(d.getVar('PKGDATA_DIR', True), + 'runtime-reverse', pkg) + with open(pkg_info, "r") as pkg_info_file: + pkg_dic[pkg] = {} + + pkg_lic_name = os.path.basename(os.readlink(pkg_info)) + + for line in pkg_info_file.read().split("\n"): + if re.match("^PN: .*$", line): + pkg_dic[pkg]["PN"] = re.search("PN: (.*)$", line).group(1) + + if re.match("^PV: .*$", line): + pkg_dic[pkg]["PV"] = re.search("PV: (.*)$", line).group(1) + + if re.match("^LICENSE_%s: (.*)$" % re.escape(pkg_lic_name) + , line): + pkg_dic[pkg]["LICENSE"] = re.search("^LICENSE_%s: (.*)$" + % pkg_lic_name, line).group(1) + elif re.match("^LICENSE: (.*)$", line): + pkg_dic[pkg]["LICENSE"] = re.search("^LICENSE: (.*)$", + line).group(1) + + license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + d.getVar('IMAGE_NAME', True), 'license.manifest') + with open(license_manifest, "w") as license_file: + for pkg in sorted(pkg_dic): + license_file.write("PACKAGE NAME: %s\n" % pkg) + license_file.write("PACKAGE VERSION: %s\n" % pkg_dic[pkg]["PV"]) + license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"]) + license_file.write("LICENSE:") + + licenses = re.sub('[|&()*]', '', pkg_dic[pkg]["LICENSE"]) + licenses = re.sub(' *', ' ', licenses) + for lic in licenses.split(): + lic = re.sub('\+', '', lic) + lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + pkg_dic[pkg]["PN"], "generic_%s" % lic) + if not os.path.exists(lic_file): + bb.warn("The license listed %s was not in the "\ + "licenses collected for recipe %s" + % (lic, pkg_dic[pkg]["PN"])) + license_file.write(" %s" % lic) + license_file.write("\n\n") + + # Two options here: + # - Just copy the manifest + # - Copy the manifest and the license directories + # With both options set we see a .5 M increase in core-image-minimal + copy_lic_manifest = d.getVar('COPY_LIC_MANIFEST', True) + copy_lic_dirs = d.getVar('COPY_LIC_DIRS', True) + if copy_lic_manifest == "1": + rootfs_license_dir = os.path.join(d.getVar('IMAGE_ROOTFS', 'True'), + 'usr', 'share', 'common-licenses') + os.makedirs(rootfs_license_dir) + rootfs_license_manifest = os.path.join(rootfs_license_dir, + 'license.manifest') + shutil.copyfile(license_manifest, rootfs_license_manifest) + + if copy_lic_dirs == "1": + for pkg in sorted(pkg_dic): + pkg_rootfs_license_dir = os.path.join(rootfs_license_dir, pkg) + os.makedirs(pkg_rootfs_license_dir) + pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + pkg_dic[pkg]["PN"]) + licenses = os.listdir(pkg_license_dir) + for lic in licenses: + rootfs_license = os.path.join(rootfs_license_dir, lic) + pkg_license = os.path.join(pkg_license_dir, lic) + pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic) + + if re.match("^generic_.*$", lic): + if not os.path.exists(pkg_rootfs_license): + shutil.copyfile(pkg_license, rootfs_license) + + os.symlink(os.path.join('..', lic), pkg_rootfs_license) + else: + shutil.copyfile(pkg_license, pkg_rootfs_license) } python do_populate_lic() {
Signature · 1 line
1.9.1
Re: [PATCH 1/4] license_class: Reimplemented manifest creation in python
Quoted 113 lines · Aníbal Limón
Reimplemented
license_manifest_createfrom shell to python, in order to use oe.license module for handleINCOMPATIBLE_LICENSEand add License prorities into OR's evaluation.Signed-off-by Aníbal Limón <[email protected]>--- meta/classes/license.bbclass | 161 ++++++++++++++++++++++++------------------- 1 file changed, 90 insertions(+), 71 deletions(-)meta/classes/license.bbclass +24
-71@@ -25,78 +25,97 @@ python write_package_manifest() { 'w+').write(image_list_installed_packages(d)) } -license_create_manifest() { - # Test if BUILD_IMAGES_FROM_FEEDS is defined in env - if [ -n "${BUILD_IMAGES_FROM_FEEDS}" ]; then - exit 0 - fi - - INSTALLED_PKGS=`cat ${LICENSE_DIRECTORY}/${IMAGE_NAME}/package.manifest` - LICENSE_MANIFEST="${LICENSE_DIRECTORY}/${IMAGE_NAME}/license.manifest" - # remove existing license.manifest file - if [ -f ${LICENSE_MANIFEST} ]; then - rm ${LICENSE_MANIFEST} - fi - touch ${LICENSE_MANIFEST} - for pkg in ${INSTALLED_PKGS}; do - filename=`ls ${PKGDATA_DIR}/runtime-reverse/${pkg}| head -1` - pkged_pn="$(sed -n 's/^PN: //p' ${filename})" - - # check to see if the package name exists in the manifest. if so, bail. - if grep -q "^PACKAGE NAME: ${pkg}" ${LICENSE_MANIFEST}; then - continue - fi - - pkged_pv="$(sed -n 's/^PV: //p' ${filename})" - pkged_name="$(basename $(readlink ${filename}))" - pkged_lic="$(sed -n "/^LICENSE_${pkged_name}: /{ s/^LICENSE_${pkged_name}: //; s/[|&()*]/ /g; s/ */ /g; p }" ${filename})" - if [ -z ${pkged_lic} ]; then - # fallback checking value of LICENSE - pkged_lic="$(sed -n "/^LICENSE: /{ s/^LICENSE: //; s/[|&()*]/ /g; s/ */ /g; p }" ${filename})" - fi - - echo "PACKAGE NAME:" ${pkg} >> ${LICENSE_MANIFEST} - echo "PACKAGE VERSION:" ${pkged_pv} >> ${LICENSE_MANIFEST} - echo "RECIPE NAME:" ${pkged_pn} >> ${LICENSE_MANIFEST} - printf "LICENSE:" >> ${LICENSE_MANIFEST} - for lic in ${pkged_lic}; do - # to reference a license file trim trailing + symbol - if ! [ -e "${LICENSE_DIRECTORY}/${pkged_pn}/generic_${lic%+}" ]; then - bbwarn "The license listed ${lic} was not in the licenses collected for ${pkged_pn}" - fi - printf " ${lic}" >> ${LICENSE_MANIFEST} - done - printf "\n\n" >> ${LICENSE_MANIFEST} - done - - # Two options here: - # - Just copy the manifest - # - Copy the manifest and the license directories - # With both options set we see a .5 M increase in core-image-minimal - if [ "${COPY_LIC_MANIFEST}" = "1" ]; then - mkdir -p ${IMAGE_ROOTFS}/usr/share/common-licenses/ - cp ${LICENSE_MANIFEST} ${IMAGE_ROOTFS}/usr/share/common-licenses/license.manifest - if [ "${COPY_LIC_DIRS}" = "1" ]; then - for pkg in ${INSTALLED_PKGS}; do - mkdir -p ${IMAGE_ROOTFS}/usr/share/common-licenses/${pkg} - pkged_pn="$(oe-pkgdata-util lookup-recipe ${PKGDATA_DIR} ${pkg})" - for lic in `ls ${LICENSE_DIRECTORY}/${pkged_pn}`; do - # Really don't need to copy the generics as they're - # represented in the manifest and in the actual pkg licenses - # Doing so would make your image quite a bit larger - if [ "${lic#generic_}" = "${lic}" ]; then - cp ${LICENSE_DIRECTORY}/${pkged_pn}/${lic} ${IMAGE_ROOTFS}/usr/share/common-licenses/${pkg}/${lic} - else - if [ ! -f ${IMAGE_ROOTFS}/usr/share/common-licenses/${lic} ]; then - cp ${LICENSE_DIRECTORY}/${pkged_pn}/${lic} ${IMAGE_ROOTFS}/usr/share/common-licenses/ - fi - ln -sf ../${lic} ${IMAGE_ROOTFS}/usr/share/common-licenses/${pkg}/${lic} - fi - done - done - fi - fi +python license_create_manifest() { + import shutil + import re + build_images_from_feeds = d.getVar('BUILD_IMAGES_FROM_FEEDS', True) + if build_images_from_feeds == "1": + return 0 + + pkg_dic = {} + package_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + d.getVar('IMAGE_NAME', True), 'package.manifest') + with open(package_manifest, "r") as package_file: + pkg_list = package_file.read().split() + for pkg in pkg_list: + pkg_info = os.path.join(d.getVar('PKGDATA_DIR', True), + 'runtime-reverse', pkg) + with open(pkg_info, "r") as pkg_info_file: + pkg_dic[pkg] = {} + + pkg_lic_name = os.path.basename(os.readlink(pkg_info)) + + for line in pkg_info_file.read().split("\n"): + if re.match("^PN: .*$", line): + pkg_dic[pkg]["PN"] = re.search("PN: (.*)$", line).group(1) +
meta/classes/license.bbclass python write_package_manifest() { · line 62
Quoted 10 lines
meta/classes/license.bbclass +10
-0+ if re.match("^PV: .*$", line): + pkg_dic[pkg]["PV"] = re.search("PV: (.*)$", line).group(1) + + if re.match("^LICENSE_%s: (.*)$" % re.escape(pkg_lic_name) + , line): + pkg_dic[pkg]["LICENSE"] = re.search("^LICENSE_%s: (.*)$" + % pkg_lic_name, line).group(1) + elif re.match("^LICENSE: (.*)$", line): + pkg_dic[pkg]["LICENSE"] = re.search("^LICENSE: (.*)$", + line).group(1)
Could you have a look at oe.packagedata.read_pkgdatafile(x) and see if the above could use that instead please?
Quoted 45 lines
meta/classes/license.bbclass +45
-0+ license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + d.getVar('IMAGE_NAME', True), 'license.manifest') + with open(license_manifest, "w") as license_file: + for pkg in sorted(pkg_dic): + license_file.write("PACKAGE NAME: %s\n" % pkg) + license_file.write("PACKAGE VERSION: %s\n" % pkg_dic[pkg]["PV"]) + license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"]) + license_file.write("LICENSE:") + + licenses = re.sub('[|&()*]', '', pkg_dic[pkg]["LICENSE"]) + licenses = re.sub(' *', ' ', licenses) + for lic in licenses.split(): + lic = re.sub('\+', '', lic) + lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + pkg_dic[pkg]["PN"], "generic_%s" % lic) + if not os.path.exists(lic_file): + bb.warn("The license listed %s was not in the "\ + "licenses collected for recipe %s" + % (lic, pkg_dic[pkg]["PN"])) + license_file.write(" %s" % lic) + license_file.write("\n\n") + + # Two options here: + # - Just copy the manifest + # - Copy the manifest and the license directories + # With both options set we see a .5 M increase in core-image-minimal + copy_lic_manifest = d.getVar('COPY_LIC_MANIFEST', True) + copy_lic_dirs = d.getVar('COPY_LIC_DIRS', True) + if copy_lic_manifest == "1": + rootfs_license_dir = os.path.join(d.getVar('IMAGE_ROOTFS', 'True'), + 'usr', 'share', 'common-licenses') + os.makedirs(rootfs_license_dir) + rootfs_license_manifest = os.path.join(rootfs_license_dir, + 'license.manifest') + shutil.copyfile(license_manifest, rootfs_license_manifest) + + if copy_lic_dirs == "1": + for pkg in sorted(pkg_dic): + pkg_rootfs_license_dir = os.path.join(rootfs_license_dir, pkg) + os.makedirs(pkg_rootfs_license_dir) + pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + pkg_dic[pkg]["PN"]) + licenses = os.listdir(pkg_license_dir) + for lic in licenses: + rootfs_license = os.path.join(rootfs_license_dir, lic)
meta/classes/license.bbclass python write_package_manifest() { · line 118
Quoted 10 lines
meta/classes/license.bbclass +10
-0+ pkg_license = os.path.join(pkg_license_dir, lic) + pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic) + + if re.match("^generic_.*$", lic): + if not os.path.exists(pkg_rootfs_license): + shutil.copyfile(pkg_license, rootfs_license) + + os.symlink(os.path.join('..', lic), pkg_rootfs_license) + else: + shutil.copyfile(pkg_license, pkg_rootfs_license)
One small tweak we could make here is to hardlink these files. There are several copies of many of them and it would reduce image size a bit. I appreciate the original doesn't but I've been doing this in other places and this is probably another we should put on the list...
Otherwise looks good, thanks.
Richard
Re: [PATCH 1/4] license_class: Reimplemented manifest creation in python
Richard,
I did the changes that you suggest and another small changes adding LIC_FILES_CHKSUM in some recipes to avoid warnings in manifest creation.
I put together into this branch for review,
https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/license
Best regards.
Quoted 194 lines · Richard Purdie
Quoted 113 lines · Aníbal Limón
Reimplemented
license_manifest_createfrom shell to python, in order to use oe.license module for handleINCOMPATIBLE_LICENSEand add License prorities into OR's evaluation.Signed-off-by Aníbal Limón <[email protected]>--- meta/classes/license.bbclass | 161 ++++++++++++++++++++++++------------------- 1 file changed, 90 insertions(+), 71 deletions(-)meta/classes/license.bbclass +24
-71@@ -25,78 +25,97 @@ python write_package_manifest() { 'w+').write(image_list_installed_packages(d)) } -license_create_manifest() { - # Test if BUILD_IMAGES_FROM_FEEDS is defined in env - if [ -n "${BUILD_IMAGES_FROM_FEEDS}" ]; then - exit 0 - fi - - INSTALLED_PKGS=`cat ${LICENSE_DIRECTORY}/${IMAGE_NAME}/package.manifest` - LICENSE_MANIFEST="${LICENSE_DIRECTORY}/${IMAGE_NAME}/license.manifest" - # remove existing license.manifest file - if [ -f ${LICENSE_MANIFEST} ]; then - rm ${LICENSE_MANIFEST} - fi - touch ${LICENSE_MANIFEST} - for pkg in ${INSTALLED_PKGS}; do - filename=`ls ${PKGDATA_DIR}/runtime-reverse/${pkg}| head -1` - pkged_pn="$(sed -n 's/^PN: //p' ${filename})" - - # check to see if the package name exists in the manifest. if so, bail. - if grep -q "^PACKAGE NAME: ${pkg}" ${LICENSE_MANIFEST}; then - continue - fi - - pkged_pv="$(sed -n 's/^PV: //p' ${filename})" - pkged_name="$(basename $(readlink ${filename}))" - pkged_lic="$(sed -n "/^LICENSE_${pkged_name}: /{ s/^LICENSE_${pkged_name}: //; s/[|&()*]/ /g; s/ */ /g; p }" ${filename})" - if [ -z ${pkged_lic} ]; then - # fallback checking value of LICENSE - pkged_lic="$(sed -n "/^LICENSE: /{ s/^LICENSE: //; s/[|&()*]/ /g; s/ */ /g; p }" ${filename})" - fi - - echo "PACKAGE NAME:" ${pkg} >> ${LICENSE_MANIFEST} - echo "PACKAGE VERSION:" ${pkged_pv} >> ${LICENSE_MANIFEST} - echo "RECIPE NAME:" ${pkged_pn} >> ${LICENSE_MANIFEST} - printf "LICENSE:" >> ${LICENSE_MANIFEST} - for lic in ${pkged_lic}; do - # to reference a license file trim trailing + symbol - if ! [ -e "${LICENSE_DIRECTORY}/${pkged_pn}/generic_${lic%+}" ]; then - bbwarn "The license listed ${lic} was not in the licenses collected for ${pkged_pn}" - fi - printf " ${lic}" >> ${LICENSE_MANIFEST} - done - printf "\n\n" >> ${LICENSE_MANIFEST} - done - - # Two options here: - # - Just copy the manifest - # - Copy the manifest and the license directories - # With both options set we see a .5 M increase in core-image-minimal - if [ "${COPY_LIC_MANIFEST}" = "1" ]; then - mkdir -p ${IMAGE_ROOTFS}/usr/share/common-licenses/ - cp ${LICENSE_MANIFEST} ${IMAGE_ROOTFS}/usr/share/common-licenses/license.manifest - if [ "${COPY_LIC_DIRS}" = "1" ]; then - for pkg in ${INSTALLED_PKGS}; do - mkdir -p ${IMAGE_ROOTFS}/usr/share/common-licenses/${pkg} - pkged_pn="$(oe-pkgdata-util lookup-recipe ${PKGDATA_DIR} ${pkg})" - for lic in `ls ${LICENSE_DIRECTORY}/${pkged_pn}`; do - # Really don't need to copy the generics as they're - # represented in the manifest and in the actual pkg licenses - # Doing so would make your image quite a bit larger - if [ "${lic#generic_}" = "${lic}" ]; then - cp ${LICENSE_DIRECTORY}/${pkged_pn}/${lic} ${IMAGE_ROOTFS}/usr/share/common-licenses/${pkg}/${lic} - else - if [ ! -f ${IMAGE_ROOTFS}/usr/share/common-licenses/${lic} ]; then - cp ${LICENSE_DIRECTORY}/${pkged_pn}/${lic} ${IMAGE_ROOTFS}/usr/share/common-licenses/ - fi - ln -sf ../${lic} ${IMAGE_ROOTFS}/usr/share/common-licenses/${pkg}/${lic} - fi - done - done - fi - fi +python license_create_manifest() { + import shutil + import re + build_images_from_feeds = d.getVar('BUILD_IMAGES_FROM_FEEDS', True) + if build_images_from_feeds == "1": + return 0 + + pkg_dic = {} + package_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + d.getVar('IMAGE_NAME', True), 'package.manifest') + with open(package_manifest, "r") as package_file: + pkg_list = package_file.read().split() + for pkg in pkg_list: + pkg_info = os.path.join(d.getVar('PKGDATA_DIR', True), + 'runtime-reverse', pkg) + with open(pkg_info, "r") as pkg_info_file: + pkg_dic[pkg] = {} + + pkg_lic_name = os.path.basename(os.readlink(pkg_info)) + + for line in pkg_info_file.read().split("\n"): + if re.match("^PN: .*$", line): + pkg_dic[pkg]["PN"] = re.search("PN: (.*)$", line).group(1) +meta/classes/license.bbclass python write_package_manifest() { · line 62
Quoted 10 lines
meta/classes/license.bbclass +10
-0+ if re.match("^PV: .*$", line): + pkg_dic[pkg]["PV"] = re.search("PV: (.*)$", line).group(1) + + if re.match("^LICENSE_%s: (.*)$" % re.escape(pkg_lic_name) + , line): + pkg_dic[pkg]["LICENSE"] = re.search("^LICENSE_%s: (.*)$" + % pkg_lic_name, line).group(1) + elif re.match("^LICENSE: (.*)$", line): + pkg_dic[pkg]["LICENSE"] = re.search("^LICENSE: (.*)$", + line).group(1)Could you have a look at
oe.packagedata.read_pkgdatafile(x)and see if the above could use that instead please?Quoted 45 lines
meta/classes/license.bbclass +45
-0+ license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + d.getVar('IMAGE_NAME', True), 'license.manifest') + with open(license_manifest, "w") as license_file: + for pkg in sorted(pkg_dic): + license_file.write("PACKAGE NAME: %s\n" % pkg) + license_file.write("PACKAGE VERSION: %s\n" % pkg_dic[pkg]["PV"]) + license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"]) + license_file.write("LICENSE:") + + licenses = re.sub('[|&()*]', '', pkg_dic[pkg]["LICENSE"]) + licenses = re.sub(' *', ' ', licenses) + for lic in licenses.split(): + lic = re.sub('\+', '', lic) + lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + pkg_dic[pkg]["PN"], "generic_%s" % lic) + if not os.path.exists(lic_file): + bb.warn("The license listed %s was not in the "\ + "licenses collected for recipe %s" + % (lic, pkg_dic[pkg]["PN"])) + license_file.write(" %s" % lic) + license_file.write("\n\n") + + # Two options here: + # - Just copy the manifest + # - Copy the manifest and the license directories + # With both options set we see a .5 M increase in core-image-minimal + copy_lic_manifest = d.getVar('COPY_LIC_MANIFEST', True) + copy_lic_dirs = d.getVar('COPY_LIC_DIRS', True) + if copy_lic_manifest == "1": + rootfs_license_dir = os.path.join(d.getVar('IMAGE_ROOTFS', 'True'), + 'usr', 'share', 'common-licenses') + os.makedirs(rootfs_license_dir) + rootfs_license_manifest = os.path.join(rootfs_license_dir, + 'license.manifest') + shutil.copyfile(license_manifest, rootfs_license_manifest) + + if copy_lic_dirs == "1": + for pkg in sorted(pkg_dic): + pkg_rootfs_license_dir = os.path.join(rootfs_license_dir, pkg) + os.makedirs(pkg_rootfs_license_dir) + pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + pkg_dic[pkg]["PN"]) + licenses = os.listdir(pkg_license_dir) + for lic in licenses: + rootfs_license = os.path.join(rootfs_license_dir, lic)meta/classes/license.bbclass python write_package_manifest() { · line 118
Quoted 10 lines
meta/classes/license.bbclass +10
-0+ pkg_license = os.path.join(pkg_license_dir, lic) + pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic) + + if re.match("^generic_.*$", lic): + if not os.path.exists(pkg_rootfs_license): + shutil.copyfile(pkg_license, rootfs_license) + + os.symlink(os.path.join('..', lic), pkg_rootfs_license) + else: + shutil.copyfile(pkg_license, pkg_rootfs_license)One small tweak we could make here is to hardlink these files. There are several copies of many of them and it would reduce image size a bit. I appreciate the original doesn't but I've been doing this in other places and this is probably another we should put on the list...
Otherwise looks good, thanks.
Richard
PATCH 2/4 license_class: Added support for INCOMPATIBLE_LICENSE into license_create_manifest
In reply to Aníbal LimónNow license_create_manifest handle INCOMPATIBLE_LICENSE to avoid put them into license.manifest and copy them into target image.
Generalized license_ok(license) to license_ok(bad_licenses, license) to avoid duplicate code.
[YOCTO #6765]
--- meta/classes/license.bbclass | 60 ++++++++++++++++++++++++++++++-------------- 1 file changed, 41 insertions(+), 19 deletions(-)
meta/classes/license.bbclass +41 -19
@@ -28,6 +28,15 @@ python write_package_manifest() { python license_create_manifest() { import shutil import re + import oe.license + + bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE", True) or "").split() + bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses) + + # Handles an "or" or two license sets provided by + # flattened_licenses(), pick one that works if possible. + def choose_lic_set(a, b): + return a if all(license_ok(bad_licenses, lic) for lic in a) else b build_images_from_feeds = d.getVar('BUILD_IMAGES_FROM_FEEDS', True) if build_images_from_feeds == "1":
@@ -70,9 +79,13 @@ python license_create_manifest() { license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"]) license_file.write("LICENSE:") - licenses = re.sub('[|&()*]', '', pkg_dic[pkg]["LICENSE"]) - licenses = re.sub(' *', ' ', licenses) - for lic in licenses.split(): + try: + licenses = oe.license.flattened_licenses(pkg_dic[pkg]["LICENSE"] + , choose_lic_set) + except oe.license.LicenseError as exc: + bb.fatal('%s: %s' % (d.getVar('P', True), exc)) + + for lic in licenses: lic = re.sub('\+', '', lic) lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY', True), pkg_dic[pkg]["PN"], "generic_%s" % lic)
@@ -110,11 +123,18 @@ python license_create_manifest() { pkg_rootfs_license = os.path.join(pkg_rootfs_license_dir, lic) if re.match("^generic_.*$", lic): + generic_lic = re.search("^generic_(.*)$", lic).group(1) + if not license_ok(bad_licenses, generic_lic): + continue + if not os.path.exists(pkg_rootfs_license): shutil.copyfile(pkg_license, rootfs_license) os.symlink(os.path.join('..', lic), pkg_rootfs_license) else: + if not license_ok(bad_licenses, lic): + continue + shutil.copyfile(pkg_license, pkg_rootfs_license) }
@@ -297,6 +317,21 @@ def canonical_license(d, license): """ return d.getVarFlag('SPDXLICENSEMAP', license, True) or license +def license_ok(dont_want_licenses, license): + import re + from fnmatch import fnmatchcase as fnmatch + for dwl in dont_want_licenses: + # If you want to exclude license named generically 'X', we + # surely want to exclude 'X+' as well. In consequence, we + # will exclude a trailing '+' character from LICENSE in + # case INCOMPATIBLE_LICENSE is not a 'X+' license. + lic = license + if not re.search('\+$', dwl): + lic = re.sub('\+', '', license) + if fnmatch(lic, dwl): + return False + return True + def incompatible_license(d, dont_want_licenses, package=None): """ This function checks if a recipe has only incompatible licenses. It also
@@ -305,34 +340,21 @@ def incompatible_license(d, dont_want_licenses, package=None): """ import re import oe.license - from fnmatch import fnmatchcase as fnmatch license = d.getVar("LICENSE_%s" % package, True) if package else None if not license: license = d.getVar('LICENSE', True) - def license_ok(license): - for dwl in dont_want_licenses: - # If you want to exclude license named generically 'X', we - # surely want to exclude 'X+' as well. In consequence, we - # will exclude a trailing '+' character from LICENSE in - # case INCOMPATIBLE_LICENSE is not a 'X+' license. - lic = license - if not re.search('\+$', dwl): - lic = re.sub('\+', '', license) - if fnmatch(lic, dwl): - return False - return True - # Handles an "or" or two license sets provided by # flattened_licenses(), pick one that works if possible. def choose_lic_set(a, b): - return a if all(license_ok(lic) for lic in a) else b + return a if all(license_ok(dont_want_licenses, lic) for lic in a) else b try: licenses = oe.license.flattened_licenses(license, choose_lic_set) except oe.license.LicenseError as exc: bb.fatal('%s: %s' % (d.getVar('P', True), exc)) - return any(not license_ok(canonical_license(d, l)) for l in licenses) + return any(not license_ok(dont_want_licenses,canonical_license(d, l)) + for l in licenses) def check_license_flags(d): """
Signature · 1 line
1.9.1
PATCH 3/4 license_class: Added LICENSE_PRIORITY support
In reply to Aníbal LimónNow you can specify LICENSE_PRIORITY for license manifest creation.
This means that if you have LICENSE expression with OR's only one license is selected based on LICENSE_PRIORITY and INCOMPATIBLE_LICENSE.
[YOCTO #6757]
--- meta/classes/license.bbclass | 47 ++++++++++++++++++++++++++++++++++++++------ meta/conf/documentation.conf | 1 + 2 files changed, 42 insertions(+), 6 deletions(-)
meta/classes/license.bbclass +41 -6
@@ -33,10 +33,45 @@ python license_create_manifest() { bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE", True) or "").split() bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses) + priority_licenses = (d.getVar("LICENSE_PRIORITY", True) or "").split() + priority_licenses = map(lambda l: canonical_license(d, l), priority_licenses) + priority_licenses.reverse() + + # Get license priority based on index, + # INCOMPATIBLE_LICENSE entries have negative weight. + # Licenses not listed have a weight of 0. + # LICENSE_PRIORITY entries have a positive weight. + def get_license_priority(license): + # If you want to exclude license named generically 'X', we + # surely want to exclude 'X+' as well. In consequence, we + # will exclude a trailing '+' character from LICENSE in + # case INCOMPATIBLE_LICENSE is not a 'X+' license. + for bl in bad_licenses: + lic = license + if not re.search('\+$', bl): + lic = re.sub('\+', '', license) + + if lic == bl: + return -1 * (bad_licenses.index(lic) + 1) + + if license in priority_licenses: + return priority_licenses.index(license) + 1 + else: + return 0 + # Handles an "or" or two license sets provided by - # flattened_licenses(), pick one that works if possible. - def choose_lic_set(a, b): - return a if all(license_ok(bad_licenses, lic) for lic in a) else b + # flattened_licenses(), pick one that works based on + # LICENSE_PRIORITY and INCOMPATIBLE_LICENSE. + def choose_lic_set(alpha, beta): + alpha_canonical = [canonical_license(d, l) for l in alpha] + beta_canonical = [canonical_license(d, l) for l in beta] + alpha_weight = sum(get_license_priority(a) for a in alpha_canonical) + beta_weight = sum(get_license_priority(b) for b in beta_canonical) + + if alpha_weight >= beta_weight: + return alpha + else: + return beta build_images_from_feeds = d.getVar('BUILD_IMAGES_FROM_FEEDS', True) if build_images_from_feeds == "1":
@@ -80,12 +115,12 @@ python license_create_manifest() { license_file.write("LICENSE:") try: - licenses = oe.license.flattened_licenses(pkg_dic[pkg]["LICENSE"] - , choose_lic_set) + pkg_dic[pkg]["FLATTENED_LICENSE"] = oe.license.flattened_licenses( + pkg_dic[pkg]["LICENSE"], choose_lic_set) except oe.license.LicenseError as exc: bb.fatal('%s: %s' % (d.getVar('P', True), exc)) - for lic in licenses: + for lic in pkg_dic[pkg]["FLATTENED_LICENSE"]: lic = re.sub('\+', '', lic) lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY', True), pkg_dic[pkg]["PN"], "generic_%s" % lic)
meta/conf/documentation.conf +1 -0
@@ -220,6 +220,7 @@ IMAGE_ROOTFS_SIZE[doc] = "Defines the size in Kbytes for the generated image." IMAGE_TYPES[doc] = "Specifies the complete list of supported image types by default." INC_PR[doc] = "Helps define the recipe revision for recipes that share a common include file." INCOMPATIBLE_LICENSE[doc] = "Specifies a space-separated list of license names (as they would appear in LICENSE) that should be excluded from the build." +LICENSE_PRIORITY[doc] = "Space separated list of licenses in priority order, highest to lowest." INHIBIT_DEFAULT_DEPS[doc] = "Prevents the default dependencies, namely the C compiler and standard C library (libc), from being added to DEPENDS." INHIBIT_PACKAGE_STRIP[doc] = "If set to "1", causes the build to not strip binaries in resulting packages." INHERIT[doc] = "Causes the named class to be inherited at this point during parsing. The variable is only valid in configuration files."
Signature · 1 line
1.9.1
PATCH 4/4 license_class: Fix remove + trim in license_create_manifest.
In reply to Aníbal Limón--- meta/classes/license.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/license.bbclass +2 -2
@@ -121,9 +121,9 @@ python license_create_manifest() { bb.fatal('%s: %s' % (d.getVar('P', True), exc)) for lic in pkg_dic[pkg]["FLATTENED_LICENSE"]: - lic = re.sub('\+', '', lic) lic_file = os.path.join(d.getVar('LICENSE_DIRECTORY', True), - pkg_dic[pkg]["PN"], "generic_%s" % lic) + pkg_dic[pkg]["PN"], "generic_%s" % + re.sub('\+', '', lic)) if not os.path.exists(lic_file): bb.warn("The license listed %s was not in the "\ "licenses collected for recipe %s"
Signature · 1 line
1.9.1
Hi Beth,
I know you're looking at rewriting some of license.bbclass in python - FYI here's an as-yet unmerged patchset from Aníbal that may contain some of that already:
Quoted 27 lines · On Wednesday 29 October 2014 12:34:12 Aníbal Limón wrote:
Now you can specify
LICENSE_PRIOTIYthat enables the distro to set the license preferences for manifest creation.The major change is when you have OR'ed LICENSE only one is choose according
LICENSE_PRIORITYpreferences ifLICENSE_PRIORITYisn't specified the left one is choosen.The test was done building qemux86/core-image-sato with and without these set of patches and then run diff recursive under common-licenses created directory.
You can find a tarball with common-license directories, resulted diff and local.conf at:
https://bugzilla.yoctoproject.org/attachment.cgi?id=2223
Aníbal Limón (4):
license_class: Reimplemented manifest creation in python
license_class: Added support forINCOMPATIBLE_LICENSEinto
license_create_manifest
license_class: AddedLICENSE_PRIORITYsupport
license_class: Fix remove + trim inlicense_create_manifest.meta/classes/license.bbclass | 250 ++++++++++++++++++++++++++++--------------- meta/conf/documentation.conf | 1 + 2 files changed, 164 insertions(+), 87 deletions(-)
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Hi,
Here is the contrib branch that contains the changes, some rebase is needed.
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/license
Kind regards,
alimon
Quoted 36 lines · Paul Eggleton
Hi Beth,
I know you're looking at rewriting some of license.bbclass in python - FYI here's an as-yet unmerged patchset from Aníbal that may contain some of that already:
Quoted 27 lines · On Wednesday 29 October 2014 12:34:12 Aníbal Limón wrote:
Now you can specify
LICENSE_PRIOTIYthat enables the distro to set the license preferences for manifest creation.The major change is when you have OR'ed LICENSE only one is choose according
LICENSE_PRIORITYpreferences ifLICENSE_PRIORITYisn't specified the left one is choosen.The test was done building qemux86/core-image-sato with and without these set of patches and then run diff recursive under common-licenses created directory.
You can find a tarball with common-license directories, resulted diff and local.conf at:
https://bugzilla.yoctoproject.org/attachment.cgi?id=2223
Aníbal Limón (4):
license_class: Reimplemented manifest creation in python
license_class: Added support forINCOMPATIBLE_LICENSEinto
license_create_manifest
license_class: AddedLICENSE_PRIORITYsupport
license_class: Fix remove + trim inlicense_create_manifest.meta/classes/license.bbclass | 250 ++++++++++++++++++++++++++++--------------- meta/conf/documentation.conf | 1 + 2 files changed, 164 insertions(+), 87 deletions(-)
Quoted 6 lines · Paul Eggleton
Hi Beth,
I know you're looking at rewriting some of license.bbclass in python - FYI here's an as-yet unmerged patchset from Aníbal that may contain some of that already:
Yeah, I'm almost finished with it. I'll look at what is there, but for now, can we avoid taking anything that touches manifest until I get a moment to look at it?
Quoted 35 lines · On Wednesday 29 October 2014 12:34:12 Aníbal Limón wrote:
Quoted 23 lines
Now you can specify
LICENSE_PRIOTIYthat enables the distro to set the license preferences for manifest creation.The major change is when you have OR'ed LICENSE only one is choose according
LICENSE_PRIORITYpreferences ifLICENSE_PRIORITYisn't specified the left one is choosen.The test was done building qemux86/core-image-sato with and without these set of patches and then run diff recursive under common-licenses created directory.
You can find a tarball with common-license directories, resulted diff and local.conf at:
https://bugzilla.yoctoproject.org/attachment.cgi?id=2223
Aníbal Limón (4):
license_class: Reimplemented manifest creation in python
license_class: Added support forINCOMPATIBLE_LICENSEinto
license_create_manifest
license_class: AddedLICENSE_PRIORITYsupport
license_class: Fix remove + trim inlicense_create_manifest.Quoted 4 lines
meta/classes/license.bbclass | 250 ++++++++++++++++++++++++++++--------------- meta/conf/documentation.conf | 1 + 2 files changed, 164 insertions(+), 87 deletions(-)--
Paul Eggleton
Intel Open Source Technology Centre
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release
Quoted 4 lines · Elizabeth
Quoted 2 lines · Paul Eggleton
Hi Beth,
Quoted 8 lines
Quoted 4 lines
I know you're looking at rewriting some of license.bbclass in python - FYI here's an as-yet unmerged patchset from Aníbal that may contain some of that already:
Yeah, I'm almost finished with it. I'll look at what is there, but for now, can we avoid taking anything that touches manifest until I get a moment to look at it?
Actually, this contains almost exactly what I was writing. So yes, I ACKing.
Quoted 43 lines
Quoted 32 lines · On Wednesday 29 October 2014 12:34:12 Aníbal Limón wrote:
Quoted 23 lines
Now you can specify
LICENSE_PRIOTIYthat enables the distro to set the license preferences for manifest creation.The major change is when you have OR'ed LICENSE only one is choose according
LICENSE_PRIORITYpreferences ifLICENSE_PRIORITYisn't specified the left one is choosen.The test was done building qemux86/core-image-sato with and without these set of patches and then run diff recursive under common-licenses created directory.
You can find a tarball with common-license directories, resulted diff and local.conf at:
https://bugzilla.yoctoproject.org/attachment.cgi?id=2223
Aníbal Limón (4):
license_class: Reimplemented manifest creation in python
license_class: Added support forINCOMPATIBLE_LICENSEinto
license_create_manifest
license_class: AddedLICENSE_PRIORITYsupport
license_class: Fix remove + trim inlicense_create_manifest.Quoted 4 lines
meta/classes/license.bbclass | 250 ++++++++++++++++++++++++++++--------------- meta/conf/documentation.conf | 1 + 2 files changed, 164 insertions(+), 87 deletions(-)Quoted 4 lines
--
Paul Eggleton
Intel Open Source Technology Centre--
Elizabeth Flanagan
Yocto Project
Build and Release
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release
Quoted 2 lines · Elizabeth
Actually, this contains almost exactly what I was writing. So yes, I ACKing.
Can someone post a rebased patch of what we're looking to merge please?
Cheers,
Richard
HI Richard,
I can do the rebase this change affects documentation that needs to describe LICENSE_PRIORITY feature.
Kind regards,
alimon
Quoted 10 lines · Richard Purdie
Quoted 2 lines · Elizabeth
Actually, this contains almost exactly what I was writing. So yes, I ACKing.
Can someone post a rebased patch of what we're looking to merge please?
Cheers,
Richard
Quoted 9 lines · On Friday 27 February 2015 10:27:15 Aníbal Limón wrote:
Quoted 5 lines · Richard Purdie
Quoted 2 lines · Elizabeth
Actually, this contains almost exactly what I was writing. So yes, I ACKing.
Can someone post a rebased patch of what we're looking to merge please?
I can do the rebase this change affects documentation that needs to describe
LICENSE_PRIORITYfeature.
Right, that would be a separate change to the yocto-docs repository and the best thing to do would be to talk to Scott R about that. For the moment we just need the rebased code changes.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre