Write out file manifest in do_unpack
cover Write out file manifest in do_unpack
This patch series implements a mechanism for do_unpack to write out a manifest of unpacked files, which is now possible due to the UNPACKDIR changes. This manifest can be consumed by downstream tasks to report the source files for the recipe; specifically SPDX can use this instead of the current behavior where it re-runs do_unpack on it's own.
Joshua Watt (2):
utils: Add write_files_manifest helper
classes-global/base: Add manifest to do_unpack
meta/classes-global/base.bbclass | 4 +- meta/lib/oe/utils.py | 67 ++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+), 1 deletion(-)
Signature · 1 line
2.43.2
RFC 1/2 utils: Add write_files_manifest helper
Adds a helper function to write out a JSON file with a manifest of files
--- meta/lib/oe/utils.py | 67 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 67 insertions(+)
meta/lib/oe/utils.py +67 -0
@@ -8,6 +8,9 @@ import subprocess import multiprocessing import traceback import errno +import json +import hashlib +import re def read_file(filename): try:
@@ -540,3 +543,67 @@ def touch(filename): # Handle read-only file systems gracefully if e.errno != errno.EROFS: raise e + + +def write_file_manifest(path, outfile, *, include_hash=True, extract_lic=False, ignore_dirs=[]): + LIC_REGEX = re.compile(rb'^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$', re.MULTILINE) + + manifest = { + "files": [], + "symlinks": [], + "dirs": [], + } + + for root, dirs, files in os.walk(path): + dirs[:] = [d for d in dirs if d not in ignore_dirs] + + for fn in files: + p = os.path.join(root, fn) + stat = os.lstat(p) + if os.path.islink(p): + manifest["symlinks"].append({ + "path": os.path.relpath(p, path), + "target": os.readlink(p), + "mode": stat.st_mode, + }) + continue + + data = { + "path": os.path.relpath(p, path), + "size": stat.st_size, + "mode": stat.st_mode, + } + + if include_hash or extract_lic: + h = hashlib.sha256() + with open(p, "rb") as f: + if extract_lic: + d = f.read(15000) + h.update(d) + + licenses = re.findall(LIC_REGEX, d) + if licenses: + data["licenses"] = [lic.decode('ascii') for lic in licenses] + + if include_hash: + while True: + d = f.read(4096) + if not d: + break + h.update(d) + + if include_hash: + data["sha256"] = h.hexdigest() + + manifest["files"].append(data) + + for dn in dirs: + p = os.path.join(root, dn) + stat = os.lstat(p) + manifest["dirs"].append({ + "path": os.path.relpath(p, path), + "mode": stat.st_mode, + }) + + with open(outfile, "w") as f: + json.dump(manifest, f)
Signature · 1 line
2.43.2
RFC 2/2 classes-global/base: Add manifest to do_unpack
In reply to Joshua WattWrites a manifest of files unpacked in do_unpack
--- meta/classes-global/base.bbclass | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes-global/base.bbclass +3 -1
@@ -177,10 +177,12 @@ python base_do_unpack() { try: fetcher = bb.fetch2.Fetch(src_uri, d) - fetcher.unpack(d.getVar('UNPACKDIR')) + fetcher.unpack(unpackdir) except bb.fetch2.BBFetchException as e: bb.fatal("Bitbake Fetcher Error: " + repr(e)) + oe.utils.write_file_manifest(unpackdir, workdir + "/unpack-manifest.json", extract_lic=True, ignore_dirs=[".git"]) + if basedir and os.path.exists(unpackdir + '/' + basedir): # Compatibility magic to ensure ${WORKDIR}/git and ${WORKDIR}/${BP} # as often used in S work as expected.
Signature · 1 line
2.43.2
If I understand it correctly, spdx uses patched source, so in order to be potentially used by spdx, the function should be called after do_patch
Regards,
Qi
Quoted 24 lines · Joshua Watt via lists.openembedded.org
This patch series implements a mechanism for
do_unpackto write out a manifest of unpacked files, which is now possible due to the UNPACKDIR changes. This manifest can be consumed by downstream tasks to report the source files for the recipe; specifically SPDX can use this instead of the current behavior where it re-runsdo_unpackon it's own.Joshua Watt (2):
utils: Addwrite_files_manifesthelper
classes-global/base: Add manifest todo_unpackmeta/classes-global/base.bbclass | 4 +- meta/lib/oe/utils.py | 67 ++++++++++++++++++++++++++++++++ 2 files changed, 70 insertions(+), 1 deletion(-)-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#200402): https://lists.openembedded.org/g/openembedded-core/message/200402
Mute This Topic: https://lists.openembedded.org/mt/106523630/3618072
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-