Create image manifest
v2 cover Create image manifest
From: Mariano Lopez <[email protected]>
When building the rootfs the rootfs is created but the recipes that deployed files in the image doesn't show up. Usually these recipes include the bootloaders and the kernel.
With these patches a new manifest will be created that includes the deployed recipes that were in the deployed. The format of the manifest is different because the recipes doesn't install packages, instead the recipes deploy files into the image
[YOCTO #6772]
Changes in v2:
- Correct a typo in title of the last commit
The following changes since commit fc45deac89ef63ca1c44e763c38ced7dfd72cbe1:
build-appliance-image: Update to jethro head revision (2015-11-03 14:03:03 +0000)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib mariano/bug6772
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug6772
Mariano Lopez (6):
license.bbclass: Split license create manifest license.bbclass: Added get_boot_dependencies function license.bbclass: Added function get_deployed_dependencies license.bbclass: Added get_package_from_deployed license.bbclass: Add function get_deployed_files license.bbclass: Create image license manifest
meta/classes/license.bbclass | 219 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 200 insertions(+), 19 deletions(-)
Signature · 1 line
1.8.4.5
PATCH v2 1/6 license.bbclass: Split license create manifest
From: Mariano Lopez <[email protected]>
This changes moves the writing of the licenses to a separated function that could be called for other packages.
With these change it will be easier to reuse the writing of the license for the packages deployed but not installed in the rootfs.
[YOCTO #6772]
--- meta/classes/license.bbclass | 36 ++++++++++++++++++++++-------------- 1 file changed, 22 insertions(+), 14 deletions(-)
meta/classes/license.bbclass +22 -14
@@ -26,14 +26,9 @@ python write_package_manifest() { } python license_create_manifest() { - import re import oe.packagedata from oe.rootfs import image_list_installed_packages - bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE", True) or "").split() - bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses) - bad_licenses = expand_wildcard_licenses(d, bad_licenses) - build_images_from_feeds = d.getVar('BUILD_IMAGES_FROM_FEEDS', True) if build_images_from_feeds == "1": return 0
@@ -49,8 +44,18 @@ python license_create_manifest() { pkg_lic_name = "LICENSE_" + pkg_name pkg_dic[pkg_name]["LICENSE"] = pkg_dic[pkg_name][pkg_lic_name] - license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + rootfs_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), d.getVar('IMAGE_NAME', True), 'license.manifest') + write_license_files(d, rootfs_license_manifest, pkg_dic) +} + +def write_license_files(d, license_manifest, pkg_dic): + import re + + bad_licenses = (d.getVar("INCOMPATIBLE_LICENSE", True) or "").split() + bad_licenses = map(lambda l: canonical_license(d, l), bad_licenses) + bad_licenses = expand_wildcard_licenses(d, bad_licenses) + with open(license_manifest, "w") as license_file: for pkg in sorted(pkg_dic): if bad_licenses:
@@ -98,15 +103,16 @@ python license_create_manifest() { if copy_lic_manifest == "1": rootfs_license_dir = os.path.join(d.getVar('IMAGE_ROOTFS', 'True'), 'usr', 'share', 'common-licenses') - os.makedirs(rootfs_license_dir) + bb.utils.mkdirhier(rootfs_license_dir) rootfs_license_manifest = os.path.join(rootfs_license_dir, - 'license.manifest') - os.link(license_manifest, rootfs_license_manifest) + os.path.split(license_manifest)[1]) + if not os.path.exists(rootfs_license_manifest): + os.link(license_manifest, rootfs_license_manifest) if copy_lic_dirs == "1": for pkg in sorted(pkg_dic): pkg_rootfs_license_dir = os.path.join(rootfs_license_dir, pkg) - os.makedirs(pkg_rootfs_license_dir) + bb.utils.mkdirhier(pkg_rootfs_license_dir) pkg_license_dir = os.path.join(d.getVar('LICENSE_DIRECTORY', True), pkg_dic[pkg]["PN"]) licenses = os.listdir(pkg_license_dir)
@@ -124,14 +130,16 @@ python license_create_manifest() { if not os.path.exists(rootfs_license): os.link(pkg_license, rootfs_license) - os.symlink(os.path.join('..', lic), pkg_rootfs_license) + if not os.path.exists(pkg_rootfs_license): + os.symlink(os.path.join('..', lic), pkg_rootfs_license) else: - if oe.license.license_ok(canonical_license(d, - lic), bad_licenses) == False: + if (oe.license.license_ok(canonical_license(d, + lic), bad_licenses) == False or + os.path.exists(pkg_rootfs_license)): continue os.link(pkg_license, pkg_rootfs_license) -} + python do_populate_lic() { """
Signature · 1 line
1.8.4.5
PATCH v2 2/6 license.bbclass: Added get_boot_dependencies function
In reply to [email protected]From: Mariano Lopez <[email protected]>
This function gets the dependencies from the classes that create a boot image, this is required because sometimes the bootloader dependecy is in these classes. The current classes covered are bootimg and bootdirectdisk because these are the only clases that add dependencies.
[YOCTO #6772]
--- meta/classes/license.bbclass | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)
meta/classes/license.bbclass +31 -0
@@ -49,6 +49,37 @@ python license_create_manifest() { write_license_files(d, rootfs_license_manifest, pkg_dic) } +def get_boot_dependencies(d): + """ Return the dependencies from boot tasks """ + + depends = "" + boot_depends_string = "" + taskdepdata = d.getVar("BB_TASKDEPDATA", True) + # Only bootimg and bootdirectdisk include the depends flag + boot_tasks = ["do_bootimg", "do_bootdirectdisk",] + + for task in boot_tasks: + boot_depends_string = "%s %s" % (boot_depends_string, + d.getVarFlag(task, "depends", True) or "") + boot_depends = [dep.split(":")[0] for dep + in boot_depends_string.split() + if not dep.split(":")[0].endswith("-native")] + for dep in boot_depends: + pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True), dep) + # If the package and dependency name is the same + if os.path.exists(pkgs_file): + depends = "%s %s" % (depends, dep) + # We need to search for the provider of the dependency + else: + for taskdep in taskdepdata.itervalues(): + # The fifth field contains what the task provides + if dep in taskdep[4]: + pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True), taskdep[0]) + if os.path.isfile(pkgs_file): + depends = "%s %s" % (depends, taskdep[0]) + break + return depends + def write_license_files(d, license_manifest, pkg_dic): import re
Signature · 1 line
1.8.4.5
Re: [PATCHv2 2/6] license.bbclass: Added get_boot_dependencies function
Hi Mariano,
Quoted 43 lines · On Monday 09 November 2015 14:04:39 [email protected] wrote:
From: Mariano Lopez <[email protected]>
This function gets the dependencies from the classes that create a boot image, this is required because sometimes the bootloader dependecy is in these classes. The current classes covered are bootimg and bootdirectdisk because these are the only clases that add dependencies.
[YOCTO #6772]
Signed-off-by Mariano Lopez <[email protected]>--- meta/classes/license.bbclass | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+)meta/classes/license.bbclass +20
-0@@ -49,6 +49,37 @@ python license_create_manifest() { write_license_files(d, rootfs_license_manifest, pkg_dic) } +def get_boot_dependencies(d): + """ Return the dependencies from boot tasks """ + + depends = "" + boot_depends_string = "" + taskdepdata = d.getVar("BB_TASKDEPDATA", True) + # Only bootimg and bootdirectdisk include the depends flag + boot_tasks = ["do_bootimg", "do_bootdirectdisk",] + + for task in boot_tasks: + boot_depends_string = "%s %s" % (boot_depends_string, + d.getVarFlag(task, "depends", True) or "") + boot_depends = [dep.split(":")[0] for dep + in boot_depends_string.split() + if not dep.split(":")[0].endswith("-native")] + for dep in boot_depends: + pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True), dep) + # If the package and dependency name is the same + if os.path.exists(pkgs_file): + depends = "%s %s" % (depends, dep)
meta/classes/license.bbclass python license_create_manifest() { · line 81
Quoted 10 lines
meta/classes/license.bbclass +9
-0+ # We need to search for the provider of the dependency + else: + for taskdep in taskdepdata.itervalues(): + # The fifth field contains what the task provides + if dep in taskdep[4]: + pkgs_file = os.path.join(d.getVar('PKGDATA_DIR', True), taskdep[0]) + if os.path.isfile(pkgs_file): + depends = "%s %s" % (depends, taskdep[0]) + break
Can you explain what you're getting out of looking into PKGDATA_DIR here? I'm not sure this is correct.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
PATCH v2 3/6 license.bbclass: Added function get_deployed_dependencies
In reply to [email protected]From: Mariano Lopez <[email protected]>
This change introduce a new function to get the dependencies that were deployed into the image. It uses BB_TASKDEPDATA to get all the dependencies of the current task, so it is possible to get different packages depending at what point this function is called.
[YOCTO #6772]
--- meta/classes/license.bbclass | 46 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+)
meta/classes/license.bbclass +46 -0
@@ -49,6 +49,52 @@ python license_create_manifest() { write_license_files(d, rootfs_license_manifest, pkg_dic) } +def get_deployed_dependencies(d): + """ Get all the deployed dependencies of an image """ + + deploy = {} + # Get all the dependencies for the current task (rootfs). + # Also get EXTRA_IMAGEDEPENDS because the bootloader is + # usually in this var and not listed in rootfs. + # At last, get the dependencies from boot classes because + # it might contain the bootloader. + taskdata = d.getVar("BB_TASKDEPDATA", True) + depends = list(set([dep[0] for dep + in taskdata.itervalues() + if not dep[0].endswith("-native")])) + extra_depends = d.getVar("EXTRA_IMAGEDEPENDS", True) + boot_depends = get_boot_dependencies(d) + depends.extend(extra_depends.split()) + depends.extend(boot_depends.split()) + depends = list(set(depends)) + + # To check what was deployed it check the rootfs dependencies against + # the SSTATE_MANIFESTS for "deploy" task. + # The manifest file name contains the arch. Because we are not running + # in the package context it is necessary to check every arch used. + sstate_manifest_dir = d.getVar("SSTATE_MANIFESTS", True) + sstate_archs = d.getVar("SSTATE_ARCHS", True) + extra_archs = d.getVar("PACKAGE_EXTRA_ARCHS", True) + archs = list(set(("%s %s" % (sstate_archs, extra_archs)).split())) + for dep in depends: + # Some packages have an arch on their own, so we try that first. + special_arch = d.getVar("PACKAGE_ARCH_pn-%s" % dep, True) + if special_arch: + sstate_manifest_file = os.path.join(sstate_manifest_dir, + "manifest-%s-%s.deploy" % (special_arch, dep)) + if os.path.exists(sstate_manifest_file): + deploy[dep] = sstate_manifest_file + continue + + for arch in archs: + sstate_manifest_file = os.path.join(sstate_manifest_dir, + "manifest-%s-%s.deploy" % (arch, dep)) + if os.path.exists(sstate_manifest_file): + deploy[dep] = sstate_manifest_file + break + + return deploy + def get_boot_dependencies(d): """ Return the dependencies from boot tasks """
Signature · 1 line
1.8.4.5
PATCH v2 4/6 license.bbclass: Added get_package_from_deployed
In reply to [email protected]From: Mariano Lopez <[email protected]>
This function will verify if the dependency is a package by itself or if the dependency installed more packages. This is special useful with the kernels because the dependency could be "linux-yocto" that in fact installed other packages. This function will be useful to get the version of the packages.
[YOCTO #6772]
--- meta/classes/license.bbclass | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
meta/classes/license.bbclass +15 -0
@@ -95,6 +95,21 @@ def get_deployed_dependencies(d): return deploy +def get_package_from_deployed(d, dep): + """ Get just one package from a dependency """ + + import oe.packagedata + + # Only one package is neccesary to get the version + data_file = os.path.join(d.getVar("PKGDATA_DIR",True), + "runtime", dep) + if os.path.exists(data_file): + return dep + else: + pkg_file = os.path.join(d.getVar("PKGDATA_DIR",True), dep) + pkgdata = oe.packagedata.read_pkgdatafile(pkg_file) + return pkgdata["PACKAGES"].split()[0] + def get_boot_dependencies(d): """ Return the dependencies from boot tasks """
Signature · 1 line
1.8.4.5
PATCH v2 5/6 license.bbclass: Add function get_deployed_files
In reply to [email protected]From: Mariano Lopez <[email protected]>
This function will get the files that were deployed in the image using the sstate-control manifest file. This will give a better view of what was deployed in the image.
[YOCTO #6772]
--- meta/classes/license.bbclass | 13 +++++++++++++ 1 file changed, 13 insertions(+)
meta/classes/license.bbclass +13 -0
@@ -141,6 +141,19 @@ def get_boot_dependencies(d): break return depends +def get_deployed_files(d, man_file): + """ This will get the files deployed from the sstate manifest """ + + dep_files = "" + excluded_files = ["README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt"] + with open(man_file, "r") as manifest: + all_files = manifest.read() + for f in all_files.splitlines(): + if ((not (os.path.islink(f) or os.path.isdir(f))) and + not os.path.basename(f) in excluded_files): + dep_files = "%s %s" % (dep_files, os.path.basename(f)) + return dep_files + def write_license_files(d, license_manifest, pkg_dic): import re
Signature · 1 line
1.8.4.5
Re: [PATCHv2 5/6] license.bbclass: Add function get_deployed_files
Quoted 5 lines · On Monday 09 November 2015 14:04:42 [email protected] wrote:
From: Mariano Lopez <[email protected]>
This function will get the files that were deployed in the image using the sstate-control manifest file. This will give a better view of what was deployed in the image.
Not *in* the image, next to it.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
PATCH v2 6/6 license.bbclass: Create image license manifest
In reply to [email protected]From: Mariano Lopez <[email protected]>
This change adds the license_deployed_manifest function that will create the manifest for the packages deployed in the image but not installed in rootfs.
This new function was added to ROOTFS_POSTPROCESS_COMMAND so it will run after every rootfs task. Because of this it could run few times for a single build and get different dependencies. Sometimes this dependencies won't include all the deployed packages, in order to avoid missing licenses a tmp file is create during the build and deleted after the build (LICENSE_TMP_JSON).
This change also modify the write_license_files because the image manifest is different from the root manifest.
[YOCTO #6772]
--- meta/classes/license.bbclass | 78 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73 insertions(+), 5 deletions(-)
meta/classes/license.bbclass +73 -5
@@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean" LICENSE_CREATE_PACKAGE ??= "0" LICENSE_PACKAGE_SUFFIX ??= "-lic" LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/" +LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json" addtask populate_lic after do_patch before do_build do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"
@@ -49,6 +50,59 @@ python license_create_manifest() { write_license_files(d, rootfs_license_manifest, pkg_dic) } +python license_deployed_manifest_task() { + license_deployed_manifest(d) +} + +def license_deployed_manifest(d): + """ Write the license manifest for the deployed packages. + The deployed packages usually includes the bootloader + and extra files to boot the target. + """ + import json + + packages = "" + dep_dic = {} + pkg_dic = {} + info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime") + + # Sometimes the initramfs image is build and it doesn't have + # the boot dependencies. In order to overcome this it is + # necessary to get save previous dependencies found. This is + # why the json_file is used (and deleted once the build is done) + json_file = d.getVar("LICENSE_TMP_JSON", True) + json_lock = bb.utils.lockfile("%s.lock" % json_file) + if os.path.exists(json_file): + with open(json_file) as f: + pkg_dic = json.loads(f.read()) + + dep_dic = get_deployed_dependencies(d) + for dep in dep_dic.keys(): + # At least one package of the deployed dependency is needed + # to get the version. + pkg = get_package_from_deployed(d, dep) + if pkg and pkg not in pkg_dic.keys(): + data_file = os.path.join(info_dir, pkg) + pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file) + # It is necessary to mark this will be used for image manifest + pkg_dic[pkg]["IMAGE_MANIFEST"] = True + pkg_dic[pkg]["FILES"] = \ + get_deployed_files(d, dep_dic[dep]) + if not "LICENSE" in pkg_dic[pkg].keys(): + pkg_lic = "LICENSE_" + pkg + pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic] + + with open(json_file, "w") as f: + json.dump(pkg_dic, f, indent=4) + bb.utils.unlockfile(json_lock) + + # Because it might be called several times we lock the license file + image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + d.getVar('IMAGE_NAME', True), 'image_license.manifest') + manifest_lock = bb.utils.lockfile("%s.lock" % image_license_manifest) + write_license_files(d, image_license_manifest, pkg_dic) + bb.utils.unlockfile(manifest_lock) + def get_deployed_dependencies(d): """ Get all the deployed dependencies of an image """
@@ -154,6 +208,11 @@ def get_deployed_files(d, man_file): dep_files = "%s %s" % (dep_files, os.path.basename(f)) return dep_files +python license_delete_tmp_files () { + json_file = d.getVar("LICENSE_TMP_JSON", True) + os.remove(json_file) +} + def write_license_files(d, license_manifest, pkg_dic): import re
@@ -175,10 +234,18 @@ def write_license_files(d, license_manifest, pkg_dic): pkg_dic[pkg]["LICENSES"] = re.sub(' *', ' ', pkg_dic[pkg]["LICENSES"]) pkg_dic[pkg]["LICENSES"] = pkg_dic[pkg]["LICENSES"].split() - license_file.write("PACKAGE NAME: %s\n" % pkg) - license_file.write("PACKAGE VERSION: %s\n" % pkg_dic[pkg]["PV"]) - license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"]) - license_file.write("LICENSE: %s\n\n" % pkg_dic[pkg]["LICENSE"]) + if not "IMAGE_MANIFEST" in pkg_dic[pkg]: + # Rootfs manifest + license_file.write("PACKAGE NAME: %s\n" % pkg) + license_file.write("PACKAGE VERSION: %s\n" % pkg_dic[pkg]["PV"]) + license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"]) + license_file.write("LICENSE: %s\n\n" % pkg_dic[pkg]["LICENSE"]) + else: + # Image manifest + license_file.write("RECIPE NAME: %s\n" % pkg_dic[pkg]["PN"]) + license_file.write("VERSION: %s\n" % pkg_dic[pkg]["PV"]) + license_file.write("LICENSE: %s\n" % pkg_dic[pkg]["LICENSE"]) + license_file.write("FILES:%s\n\n" % pkg_dic[pkg]["FILES"]) # If the package doesn't contain any file, that is, its size is 0, the license # isn't relevant as far as the final image is concerned. So doing license check
@@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic" do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}" do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/" -ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest; license_create_manifest; " +ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest; license_create_manifest; license_deployed_manifest_task; " +IMAGE_POSTPROCESS_COMMAND_prepend = " license_delete_tmp_files; " do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}" do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"
Signature · 1 line
1.8.4.5
Re: [PATCHv2 6/6] license.bbclass: Create image license manifest
Quoted 5 lines · On Monday 09 November 2015 14:04:43 [email protected] wrote:
From: Mariano Lopez <[email protected]>
This change adds the
license_deployed_manifestfunction that will create the manifest for the packages deployed in the image but not installed in rootfs.
Again, not "in the image" but "next to the image". I'd suggest giving an example or two here so that it's clear what's being referred to.
Quoted 132 lines
This new function was added to
ROOTFS_POSTPROCESS_COMMANDso it will run after every rootfs task. Because of this it could run few times for a single build and get different dependencies. Sometimes this dependencies won't include all the deployed packages, in order to avoid missing licenses a tmp file is create during the build and deleted after the build (LICENSE_TMP_JSON).This change also modify the
write_license_filesbecause the image manifest is different from the root manifest.[YOCTO #6772]
Signed-off-by Mariano Lopez <[email protected]>--- meta/classes/license.bbclass | 78+++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73 insertions(+), 5 deletions(-)
meta/classes/license.bbclass +58
-2@@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean" LICENSE_CREATE_PACKAGE ??= "0" LICENSE_PACKAGE_SUFFIX ??= "-lic" LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/" +LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json" addtask populate_lic after do_patch before do_build do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"@@ -49,6 +50,59 @@ python license_create_manifest() { write_license_files(d, rootfs_license_manifest, pkg_dic) } +python license_deployed_manifest_task() { + license_deployed_manifest(d) +} + +def license_deployed_manifest(d): + """ Write the license manifest for the deployed packages. + The deployed packages usually includes the bootloader + and extra files to boot the target. + """ + import json + + packages = "" + dep_dic = {} + pkg_dic = {} + info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime") + + # Sometimes the initramfs image is build and it doesn't have + # the boot dependencies. In order to overcome this it is + # necessary to get save previous dependencies found. This is + # why the json_file is used (and deleted once the build is done) + json_file = d.getVar("LICENSE_TMP_JSON", True) + json_lock = bb.utils.lockfile("%s.lock" % json_file) + if os.path.exists(json_file): + with open(json_file) as f: + pkg_dic = json.loads(f.read()) + + dep_dic = get_deployed_dependencies(d) + for dep in dep_dic.keys(): + # At least one package of the deployed dependency is needed + # to get the version. + pkg = get_package_from_deployed(d, dep) + if pkg and pkg not in pkg_dic.keys(): + data_file = os.path.join(info_dir, pkg) + pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file) + # It is necessary to mark this will be used for image manifest + pkg_dic[pkg]["IMAGE_MANIFEST"] = True + pkg_dic[pkg]["FILES"] = \ + get_deployed_files(d, dep_dic[dep]) + if not "LICENSE" in pkg_dic[pkg].keys(): + pkg_lic = "LICENSE_" + pkg + pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic] + + with open(json_file, "w") as f: + json.dump(pkg_dic, f, indent=4) + bb.utils.unlockfile(json_lock) + + # Because it might be called several times we lock the license file + image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + d.getVar('IMAGE_NAME', True), 'image_license.manifest') + manifest_lock = bb.utils.lockfile("%s.lock" % image_license_manifest) + write_license_files(d, image_license_manifest, pkg_dic) + bb.utils.unlockfile(manifest_lock) + def get_deployed_dependencies(d): """ Get all the deployed dependencies of an image """@@ -154,6 +208,11 @@ def get_deployed_files(d, man_file): dep_files = "%s %s" % (dep_files, os.path.basename(f)) return dep_files +python license_delete_tmp_files () { + json_file = d.getVar("LICENSE_TMP_JSON", True) + os.remove(json_file) +} + def write_license_files(d, license_manifest, pkg_dic): import re@@ -175,10 +234,18 @@ def write_license_files(d, license_manifest, pkg_dic): pkg_dic[pkg]["LICENSES"] = re.sub(' *', ' ', pkg_dic[pkg]["LICENSES"]) pkg_dic[pkg]["LICENSES"] = pkg_dic[pkg]["LICENSES"].split() - license_file.write("PACKAGE NAME: %s\n" % pkg) - license_file.write("PACKAGE VERSION: %s\n" %pkg_dic[pkg]["PV"]) - license_file.write("RECIPE NAME: %s\n" %
pkg_dic[pkg]["PN"]) - license_file.write("LICENSE: %s\n\n" %
pkg_dic[pkg]["LICENSE"]) + if not "IMAGE_MANIFEST" in
pkg_dic[pkg]:meta/classes/license.bbclass +3
-0+ # Rootfs manifest + license_file.write("PACKAGE NAME: %s\n" % pkg) + license_file.write("PACKAGE VERSION: %s\n" %pkg_dic[pkg]["PV"]) + license_file.write("RECIPE NAME: %s\n"
% pkg_dic[pkg]["PN"]) + license_file.write("LICENSE: %s\n\n"
% pkg_dic[pkg]["LICENSE"]) + else:meta/classes/license.bbclass +2
-0+ # Image manifest + license_file.write("RECIPE NAME: %s\n" %pkg_dic[pkg]["PN"]) + license_file.write("VERSION: %s\n" %
pkg_dic[pkg]["PV"]) + license_file.write("LICENSE: %s\n" %
pkg_dic[pkg]["LICENSE"]) + license_file.write("FILES:%s\n\n"
% pkg_dic[pkg]["FILES"])# If the package doesn't contain any file, that is, its size is
0, the license # isn't relevant as far as the final image is concerned. So doing license check @@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic"
do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}" do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"
Quoted 8 lines
-ROOTFS_POSTPROCESS_COMMAND_prepend = "
write_package_manifest;license_create_manifest; " +ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest;license_create_manifest;license_deployed_manifest_task; " +IMAGE_POSTPROCESS_COMMAND_prepend = "license_delete_tmp_files; "do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}" do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"
A couple of things:
1) Please take care to use the appropriate terminology. "package" has a specific meaning; what we are dealing with here is files deployed directly from recipes, packages aren't involved (or shouldn't be).
2) You're looking into PKGDATA_DIR, as I mentioned in one of my other replies I don't think this is right. The risk is that if a recipe doesn't actually write out any packages, the manifest will be incomplete. We need a different approach here where pkgdata isn't involved at all. IIRC my earlier suggestion was to use what is written out by do_populate_lic, i.e. tmp/deploy/licenses/ - if that isn't practical we can have that task write out the LICENSE value to a separate file, and that will then work for any recipe.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Re: [PATCHv2 6/6] license.bbclass: Create image license manifest
Quoted 147 lines · Paul Eggleton
Quoted 5 lines · On Monday 09 November 2015 14:04:43 [email protected] wrote:
From: Mariano Lopez <[email protected]>
This change adds the
license_deployed_manifestfunction that will create the manifest for the packages deployed in the image but not installed in rootfs.Again, not "in the image" but "next to the image". I'd suggest giving an example or two here so that it's clear what's being referred to.
Quoted 138 lines
This new function was added to
ROOTFS_POSTPROCESS_COMMANDso it will run after every rootfs task. Because of this it could run few times for a single build and get different dependencies. Sometimes this dependencies won't include all the deployed packages, in order to avoid missing licenses a tmp file is create during the build and deleted after the build (LICENSE_TMP_JSON).This change also modify the
write_license_filesbecause the image manifest is different from the root manifest.[YOCTO #6772]
Signed-off-by Mariano Lopez <[email protected]>--- meta/classes/license.bbclass | 78+++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73 insertions(+), 5 deletions(-)
meta/classes/license.bbclass +58
-2@@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean" LICENSE_CREATE_PACKAGE ??= "0" LICENSE_PACKAGE_SUFFIX ??= "-lic" LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/" +LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json" addtask populate_lic after do_patch before do_build do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"@@ -49,6 +50,59 @@ python license_create_manifest() { write_license_files(d, rootfs_license_manifest, pkg_dic) } +python license_deployed_manifest_task() { + license_deployed_manifest(d) +} + +def license_deployed_manifest(d): + """ Write the license manifest for the deployed packages. + The deployed packages usually includes the bootloader + and extra files to boot the target. + """ + import json + + packages = "" + dep_dic = {} + pkg_dic = {} + info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime") + + # Sometimes the initramfs image is build and it doesn't have + # the boot dependencies. In order to overcome this it is + # necessary to get save previous dependencies found. This is + # why the json_file is used (and deleted once the build is done) + json_file = d.getVar("LICENSE_TMP_JSON", True) + json_lock = bb.utils.lockfile("%s.lock" % json_file) + if os.path.exists(json_file): + with open(json_file) as f: + pkg_dic = json.loads(f.read()) + + dep_dic = get_deployed_dependencies(d) + for dep in dep_dic.keys(): + # At least one package of the deployed dependency is needed + # to get the version. + pkg = get_package_from_deployed(d, dep) + if pkg and pkg not in pkg_dic.keys(): + data_file = os.path.join(info_dir, pkg) + pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file) + # It is necessary to mark this will be used for image manifest + pkg_dic[pkg]["IMAGE_MANIFEST"] = True + pkg_dic[pkg]["FILES"] = \ + get_deployed_files(d, dep_dic[dep]) + if not "LICENSE" in pkg_dic[pkg].keys(): + pkg_lic = "LICENSE_" + pkg + pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic] + + with open(json_file, "w") as f: + json.dump(pkg_dic, f, indent=4) + bb.utils.unlockfile(json_lock) + + # Because it might be called several times we lock the license file + image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY', True), + d.getVar('IMAGE_NAME', True), 'image_license.manifest') + manifest_lock = bb.utils.lockfile("%s.lock" % image_license_manifest) + write_license_files(d, image_license_manifest, pkg_dic) + bb.utils.unlockfile(manifest_lock) + def get_deployed_dependencies(d): """ Get all the deployed dependencies of an image """@@ -154,6 +208,11 @@ def get_deployed_files(d, man_file): dep_files = "%s %s" % (dep_files, os.path.basename(f)) return dep_files +python license_delete_tmp_files () { + json_file = d.getVar("LICENSE_TMP_JSON", True) + os.remove(json_file) +} + def write_license_files(d, license_manifest, pkg_dic): import re@@ -175,10 +234,18 @@ def write_license_files(d, license_manifest, pkg_dic): pkg_dic[pkg]["LICENSES"] = re.sub(' *', ' ', pkg_dic[pkg]["LICENSES"]) pkg_dic[pkg]["LICENSES"] = pkg_dic[pkg]["LICENSES"].split() - license_file.write("PACKAGE NAME: %s\n" % pkg) - license_file.write("PACKAGE VERSION: %s\n" %pkg_dic[pkg]["PV"]) - license_file.write("RECIPE NAME: %s\n" %
pkg_dic[pkg]["PN"]) - license_file.write("LICENSE: %s\n\n" %
pkg_dic[pkg]["LICENSE"]) + if not "IMAGE_MANIFEST" in
pkg_dic[pkg]:meta/classes/license.bbclass +3
-0+ # Rootfs manifest + license_file.write("PACKAGE NAME: %s\n" % pkg) + license_file.write("PACKAGE VERSION: %s\n" %pkg_dic[pkg]["PV"]) + license_file.write("RECIPE NAME: %s\n"
% pkg_dic[pkg]["PN"]) + license_file.write("LICENSE: %s\n\n"
% pkg_dic[pkg]["LICENSE"]) + else:meta/classes/license.bbclass +2
-0+ # Image manifest + license_file.write("RECIPE NAME: %s\n" %pkg_dic[pkg]["PN"]) + license_file.write("VERSION: %s\n" %
pkg_dic[pkg]["PV"]) + license_file.write("LICENSE: %s\n" %
pkg_dic[pkg]["LICENSE"]) + license_file.write("FILES:%s\n\n"
% pkg_dic[pkg]["FILES"])# If the package doesn't contain any file, that is, its size is
0, the license # isn't relevant as far as the final image is concerned. So doing license check @@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic"
do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}" do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"-ROOTFS_POSTPROCESS_COMMAND_prepend = "
write_package_manifest;license_create_manifest; " +ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest;license_create_manifest;license_deployed_manifest_task; " +IMAGE_POSTPROCESS_COMMAND_prepend = "license_delete_tmp_files; "
Quoted 7 lines
Quoted 2 lines
do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}" do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"A couple of things:
1) Please take care to use the appropriate terminology. "package" has a specific meaning; what we are dealing with here is files deployed directly from recipes, packages aren't involved (or shouldn't be).
The terminology here is used because I reused the code to write rootfs manifest. I can change the terminology of this function or I can write a function just for the image manifest.
Quoted 8 lines
2) You're looking into
PKGDATA_DIR, as I mentioned in one of my other replies I don't think this is right. The risk is that if a recipe doesn't actually write out any packages, the manifest will be incomplete. We need a different approach here where pkgdata isn't involved at all. IIRC my earlier suggestion was to use what is written out bydo_populate_lic, i.e. tmp/deploy/licenses/ - if that isn't practical we can have that task write out the LICENSE value to a separate file, and that will then work for any recipe.
The use of PKGDATA_DIR is for the version of the deployed files, getting the license from here was just because it was practical. Is there another way to get the version without using PKGDATA_DIR?
Quoted 4 lines
Cheers,
Paul
Regards,
Mariano Lopez
Re: [PATCHv2 6/6] license.bbclass: Create image license manifest
Quoted 175 lines · On Tuesday 10 November 2015 08:52:22 Mariano Lopez wrote:
Quoted 174 lines · Paul Eggleton
Quoted 5 lines · On Monday 09 November 2015 14:04:43 [email protected] wrote:
From: Mariano Lopez <[email protected]>
This change adds the
license_deployed_manifestfunction that will create the manifest for the packages deployed in the image but not installed in rootfs.Again, not "in the image" but "next to the image". I'd suggest giving an example or two here so that it's clear what's being referred to.
Quoted 153 lines
This new function was added to
ROOTFS_POSTPROCESS_COMMANDso it will run after every rootfs task. Because of this it could run few times for a single build and get different dependencies. Sometimes this dependencies won't include all the deployed packages, in order to avoid missing licenses a tmp file is create during the build and deleted after the build (LICENSE_TMP_JSON).This change also modify the
write_license_filesbecause the image manifest is different from the root manifest.[YOCTO #6772]
Signed-off-by Mariano Lopez <[email protected]>---
meta/classes/license.bbclass | 78+++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 73 insertions(+), 5 deletions(-)
meta/classes/license.bbclass +1
-0@@ -11,6 +11,7 @@ LICENSE_CREATE_PACKAGE[type] = "boolean" LICENSE_CREATE_PACKAGE ??= "0" LICENSE_PACKAGE_SUFFIX ??= "-lic" LICENSE_FILES_DIRECTORY ??= "${datadir}/licenses/" +LICENSE_TMP_JSON ?= "${LICENSE_DIRECTORY}/deploy_packages.json"addtask populate_lic after do_patch before do_build do_populate_lic[dirs] = "${LICSSTATEDIR}/${PN}"meta/classes/license.bbclass +35
-0@@ -49,6 +50,59 @@ python license_create_manifest() { write_license_files(d, rootfs_license_manifest, pkg_dic) } +python license_deployed_manifest_task() { + license_deployed_manifest(d) +} + +def license_deployed_manifest(d): + """ Write the license manifest for the deployed packages. + The deployed packages usually includes the bootloader + and extra files to boot the target. + """ + import json + + packages = "" + dep_dic = {} + pkg_dic = {} + info_dir = os.path.join(d.getVar("PKGDATA_DIR",True), "runtime") + + # Sometimes the initramfs image is build and it doesn't have + # the boot dependencies. In order to overcome this it is + # necessary to get save previous dependencies found. This is + # why the json_file is used (and deleted once the build is done) + json_file = d.getVar("LICENSE_TMP_JSON", True) + json_lock = bb.utils.lockfile("%s.lock" % json_file) + if os.path.exists(json_file): + with open(json_file) as f: + pkg_dic = json.loads(f.read()) + + dep_dic = get_deployed_dependencies(d) + for dep in dep_dic.keys(): + # At least one package of the deployed dependency is needed + # to get the version. + pkg = get_package_from_deployed(d, dep) + if pkg and pkg not in pkg_dic.keys(): + data_file = os.path.join(info_dir, pkg) + pkg_dic[pkg] = oe.packagedata.read_pkgdatafile(data_file) + # It is necessary to mark this will be used for imagemanifest
meta/classes/license.bbclass +13
-0+ pkg_dic[pkg]["IMAGE_MANIFEST"] = True + pkg_dic[pkg]["FILES"] = \ + get_deployed_files(d, dep_dic[dep]) + if not "LICENSE" in pkg_dic[pkg].keys(): + pkg_lic = "LICENSE_" + pkg + pkg_dic[pkg]["LICENSE"] = pkg_dic[pkg][pkg_lic] + + with open(json_file, "w") as f: + json.dump(pkg_dic, f, indent=4) + bb.utils.unlockfile(json_lock) + + # Because it might be called several times we lock the license file + image_license_manifest = os.path.join(d.getVar('LICENSE_DIRECTORY',True), +
d.getVar('IMAGE_NAME', True),
'image_license.manifest')
+manifest_lock= bb.utils.lockfile("%s.lock" %image_license_manifest)meta/classes/license.bbclass +8
-0+ write_license_files(d, image_license_manifest, pkg_dic) + bb.utils.unlockfile(manifest_lock) + def get_deployed_dependencies(d): """ Get all the deployed dependencies of an image """@@ -154,6 +208,11 @@ def get_deployed_files(d, man_file): dep_files = "%s %s" % (dep_files, os.path.basename(f)) return dep_files +python license_delete_tmp_files () { + json_file = d.getVar("LICENSE_TMP_JSON", True) + os.remove(json_file) +} + def write_license_files(d, license_manifest, pkg_dic):import re
meta/classes/license.bbclass +0
-0@@ -175,10 +234,18 @@ def write_license_files(d, license_manifest, pkg_dic): pkg_dic[pkg]["LICENSES"] = re.sub(' *', ' ',pkg_dic[pkg]["LICENSES"]) pkg_dic[pkg]["LICENSES"] = pkg_dic[pkg]["LICENSES"].split()
meta/classes/license.bbclass +0
-2- license_file.write("PACKAGE NAME: %s\n" % pkg) - license_file.write("PACKAGE VERSION: %s\n" %pkg_dic[pkg]["PV"]) - license_file.write("RECIPE NAME: %s\n" %
pkg_dic[pkg]["PN"]) - license_file.write("LICENSE: %s\n\n" %
pkg_dic[pkg]["LICENSE"]) + if not "IMAGE_MANIFEST" in
pkg_dic[pkg]:meta/classes/license.bbclass +3
-0+ # Rootfs manifest + license_file.write("PACKAGE NAME: %s\n" % pkg) + license_file.write("PACKAGE VERSION: %s\n" %pkg_dic[pkg]["PV"]) + license_file.write("RECIPE NAME:
%s\n"
% pkg_dic[pkg]["PN"]) + license_file.write("LICENSE:
%s\n\n"
% pkg_dic[pkg]["LICENSE"]) + else:meta/classes/license.bbclass +2
-0+ # Image manifest + license_file.write("RECIPE NAME: %s\n" %pkg_dic[pkg]["PN"]) + license_file.write("VERSION: %s\n" %
pkg_dic[pkg]["PV"]) + license_file.write("LICENSE: %s\n" %
pkg_dic[pkg]["LICENSE"]) +
license_file.write("FILES:%s\n\n"
% pkg_dic[pkg]["FILES"])# If the package doesn't contain any file, that is, its size is
0, the license # isn't relevant as far as the final image is concerned. So
doing license check @@ -586,7 +653,8 @@ SSTATETASKS += "do_populate_lic"do_populate_lic[sstate-inputdirs] = "${LICSSTATEDIR}" do_populate_lic[sstate-outputdirs] = "${LICENSE_DIRECTORY}/"Quoted 8 lines
-ROOTFS_POSTPROCESS_COMMAND_prepend = "
write_package_manifest;license_create_manifest; " +ROOTFS_POSTPROCESS_COMMAND_prepend = "write_package_manifest;license_create_manifest;license_deployed_manifest_task; " +IMAGE_POSTPROCESS_COMMAND_prepend = "license_delete_tmp_files; "do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}" do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}"A couple of things:
Quoted 7 lines
Quoted 3 lines
1) Please take care to use the appropriate terminology. "package" has a specific meaning; what we are dealing with here is files deployed directly from recipes, packages aren't involved (or shouldn't be).
The terminology here is used because I reused the code to write rootfs manifest. I can change the terminology of this function or I can write a function just for the image manifest.
Understood, but I was referring more to how you describe things in the commit message and elsewhere in the code.
Quoted 9 lines
Quoted 8 lines
2) You're looking into
PKGDATA_DIR, as I mentioned in one of my other replies I don't think this is right. The risk is that if a recipe doesn't actually write out any packages, the manifest will be incomplete. We need a different approach here where pkgdata isn't involved at all. IIRC my earlier suggestion was to use what is written out bydo_populate_lic, i.e. tmp/deploy/licenses/ - if that isn't practical we can have that task write out the LICENSE value to a separate file, and that will then work for any recipe.
Quoted 3 lines
The use of
PKGDATA_DIRis for the version of the deployed files, getting the license from here was just because it was practical. Is there another way to get the version without usingPKGDATA_DIR?
In the context you're in, I'm not sure there is. However, I'm tempted to say if we're writing out LICENSE to a separate file during do_populate_lic, we might as well do the same with PV.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre