Skip to contentSkip to conversation

Write out file manifest in do_unpack

RFC In review Joshua Watt 2 participants 4 messages Jun 6, 2024 – Jun 7, 2024

lore
Share & tools
lore
#0 lore raw

cover Write out file manifest in do_unpack

This patch series implements a mechanism for do_unpack to write out a manifest of unpacked files, which is now possible due to the UNPACKDIR changes. This manifest can be consumed by downstream tasks to report the source files for the recipe; specifically SPDX can use this instead of the current behavior where it re-runs do_unpack on it's own.

Joshua Watt (2):
utils: Add write_files_manifest helper
classes-global/base: Add manifest to do_unpack

 meta/classes-global/base.bbclass |  4 +-
 meta/lib/oe/utils.py             | 67 ++++++++++++++++++++++++++++++++
 2 files changed, 70 insertions(+), 1 deletion(-)
Signature · 1 line
2.43.2
#1 lore raw

RFC 1/2 utils: Add write_files_manifest helper

Adds a helper function to write out a JSON file with a manifest of files

---
 meta/lib/oe/utils.py | 67 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)
meta/lib/oe/utils.py +67 -0
index 14a7d07ef01..5437e7a2d9f 100644
@@ -8,6 +8,9 @@ import subprocess
 import multiprocessing
 import traceback
 import errno
+import json
+import hashlib
+import re
 
 def read_file(filename):
     try:
@@ -540,3 +543,67 @@ def touch(filename):
         # Handle read-only file systems gracefully
         if e.errno != errno.EROFS:
             raise e
+
+
+def write_file_manifest(path, outfile, *, include_hash=True, extract_lic=False, ignore_dirs=[]):
+    LIC_REGEX = re.compile(rb'^\W*SPDX-License-Identifier:\s*([ \w\d.()+-]+?)(?:\s+\W*)?$', re.MULTILINE)
+
+    manifest = {
+        "files": [],
+        "symlinks": [],
+        "dirs": [],
+    }
+
+    for root, dirs, files in os.walk(path):
+        dirs[:] = [d for d in dirs if d not in ignore_dirs]
+
+        for fn in files:
+            p = os.path.join(root, fn)
+            stat = os.lstat(p)
+            if os.path.islink(p):
+                manifest["symlinks"].append({
+                    "path": os.path.relpath(p, path),
+                    "target": os.readlink(p),
+                    "mode": stat.st_mode,
+                })
+                continue
+
+            data = {
+                "path": os.path.relpath(p, path),
+                "size": stat.st_size,
+                "mode": stat.st_mode,
+            }
+
+            if include_hash or extract_lic:
+                h = hashlib.sha256()
+                with open(p, "rb") as f:
+                    if extract_lic:
+                        d = f.read(15000)
+                        h.update(d)
+
+                        licenses = re.findall(LIC_REGEX, d)
+                        if licenses:
+                            data["licenses"] = [lic.decode('ascii') for lic in licenses]
+
+                    if include_hash:
+                        while True:
+                            d = f.read(4096)
+                            if not d:
+                                break
+                            h.update(d)
+
+            if include_hash:
+                data["sha256"] = h.hexdigest()
+
+            manifest["files"].append(data)
+
+        for dn in dirs:
+            p = os.path.join(root, dn)
+            stat = os.lstat(p)
+            manifest["dirs"].append({
+                "path": os.path.relpath(p, path),
+                "mode": stat.st_mode,
+            })
+
+    with open(outfile, "w") as f:
+        json.dump(manifest, f)
Signature · 1 line
2.43.2
#2 lore raw

RFC 2/2 classes-global/base: Add manifest to do_unpack

In reply to Joshua Watt

Writes a manifest of files unpacked in do_unpack

---
 meta/classes-global/base.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/classes-global/base.bbclass +3 -1
index b6940bbb6ff..e543e206d08 100644
@@ -177,10 +177,12 @@ python base_do_unpack() {
 
     try:
         fetcher = bb.fetch2.Fetch(src_uri, d)
-        fetcher.unpack(d.getVar('UNPACKDIR'))
+        fetcher.unpack(unpackdir)
     except bb.fetch2.BBFetchException as e:
         bb.fatal("Bitbake Fetcher Error: " + repr(e))
 
+    oe.utils.write_file_manifest(unpackdir, workdir + "/unpack-manifest.json", extract_lic=True, ignore_dirs=[".git"])
+
     if basedir and os.path.exists(unpackdir + '/' + basedir):
         # Compatibility magic to ensure ${WORKDIR}/git and ${WORKDIR}/${BP}
         # as often used in S work as expected.
Signature · 1 line
2.43.2
#3 lore raw
In reply to Joshua Watt

If I understand it correctly, spdx uses patched source, so in order to be potentially used by spdx, the function should be called after do_patch

Regards,
Qi

Quoted 24 lines · Joshua Watt via lists.openembedded.org

This patch series implements a mechanism for do_unpack to write out a manifest of unpacked files, which is now possible due to the UNPACKDIR changes. This manifest can be consumed by downstream tasks to report the source files for the recipe; specifically SPDX can use this instead of the current behavior where it re-runs do_unpack on it's own.

Joshua Watt (2):
utils: Add write_files_manifest helper
classes-global/base: Add manifest to do_unpack

  meta/classes-global/base.bbclass |  4 +-
  meta/lib/oe/utils.py             | 67 ++++++++++++++++++++++++++++++++
  2 files changed, 70 insertions(+), 1 deletion(-)

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#200402): https://lists.openembedded.org/g/openembedded-core/message/200402
Mute This Topic: https://lists.openembedded.org/mt/106523630/3618072
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-