Skip to contentSkip to conversation

create-spdx-3.0: record component release date in SPDX output

PATCH In review [email protected] 2 participants 4 messages Sep 25 – Sep 28

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 create-spdx-3.0: record component release date in SPDX output

From: Daniel Turull <[email protected]>

Record each recipe's release date in the releaseTime property of its software_Package object, using the SOURCE_DATE_EPOCH already computed for reproducible builds.

Accuracy depends on how SOURCE_DATE_EPOCH was derived: exact for git-tagged recipes, best-effort for tarball/http(s) sources. Some Python sdists (e.g. cryptography, hypothesis, maturin) normalize all file mtimes to a fixed placeholder, so their releaseTime reflects packaging-tool behavior, not the real release date.

Tested with oe-selftest -r spdx, and with bitbake world --runall=do_create_spdx: 1011/1150 recipes got a releaseTime (range 1998-12-30 to 2026-09-17), 139 correctly had none.

AI-Generated: Uses Kiro with Claude Sonnet 5

---
v2:
- Dropped all options per Joshua's feedback; read SDE_FILE directly.
- Dropped the redundant else: delattr(recipe, "releaseTime") branch.
- Selftest compares against SDE_FILE content directly instead of SOURCE_DATE_EPOCH, which can diverge from it.
- Fixed a leak: recipes with no git checkout and no fetched source had SDE_FILE holding only SOURCE_DATE_EPOCH_FALLBACK, showing a bogus 2011-04-05T23:00:00Z releaseTime instead of none.

---
 meta/classes/create-spdx-3.0.bbclass |  2 +-
 meta/lib/oe/spdx30_tasks.py          | 27 +++++++++++++++
 meta/lib/oeqa/selftest/cases/spdx.py | 52 ++++++++++++++++++++++++++++
 3 files changed, 80 insertions(+), 1 deletion(-)
meta/classes/create-spdx-3.0.bbclass +1 -1
index 56fd01fd53..da413d19a3 100644
@@ -192,7 +192,7 @@ python do_create_recipe_spdx() {
     import oe.spdx30_tasks
     oe.spdx30_tasks.create_recipe_spdx(d)
 }
-addtask do_create_recipe_spdx
+addtask do_create_recipe_spdx after do_deploy_source_date_epoch
 
 SSTATETASKS += "do_create_recipe_spdx"
 do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}"
meta/lib/oe/spdx30_tasks.py +27 -0
index b6456a214a..46cc0c5711 100644
@@ -9,6 +9,7 @@ import oe.cve_check
 import oe.license
 import oe.packagedata
 import oe.patch
+import oe.reproducible
 import oe.sbom30
 import oe.sdk
 import oe.spdx30
@@ -36,6 +37,28 @@ def set_timestamp_now(d, o, prop):
         delattr(o, prop)
 
 
+def get_release_date(d):
+    """Resolve the release date to record in a recipe's releaseTime property.
+
+    Uses the SOURCE_DATE_EPOCH already determined for reproducible builds
+    (SDE_FILE), without falling back to SOURCE_DATE_EPOCH_FALLBACK, since a
+    fixed fallback timestamp is not a meaningful release date.
+
+    Returns a datetime, or None if no release date should be recorded.
+    """
+    sde_file = d.getVar("SDE_FILE")
+    if not sde_file or not os.path.isfile(sde_file):
+        return None
+
+    source_date_epoch = oe.reproducible.epochfile_read(sde_file, d)
+    if source_date_epoch == d.getVar("SOURCE_DATE_EPOCH_FALLBACK"):
+        # SDE_FILE exists but its value is only the fallback (e.g. no git
+        # checkout and no fetched source to derive a date from).
+        return None
+
+    return datetime.fromtimestamp(int(source_date_epoch), tz=timezone.utc)
+
+
 def add_license_expression(
     d, objset, license_expression, license_data, search_objsets=[]
 ):
@@ -633,6 +656,10 @@ def create_recipe_spdx(d):
     if val := d.getVar("DESCRIPTION"):
         recipe.description = val
 
+    release_date = get_release_date(d)
+    if release_date is not None:
+        recipe.releaseTime = release_date
+
     for cpe_id in oe.cve_check.get_cpe_ids(
         d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION")
     ):
meta/lib/oeqa/selftest/cases/spdx.py +52 -0

61 lines not on this page · raw message

#1 lore raw

PATCH v2 2/2 scripts/contrib: add spdx-release-date-report.py

From: Daniel Turull <[email protected]>

Reports each recipe's releaseTime (added by the create-spdx-3.0 patch) from either a DEPLOY_DIR_SPDX tree or a single merged image SBOM. Used to spot-check release dates across a build and surface recipes missing one; helped find the SOURCE_DATE_EPOCH_FALLBACK leak and archive-mtime issues fixed by the two preceding patches.

AI-Generated: Uses Kiro with Claude Sonnet 5

---
 scripts/contrib/spdx-release-date-report.py | 193 ++++++++++++++++++++
 1 file changed, 193 insertions(+)
 create mode 100755 scripts/contrib/spdx-release-date-report.py
scripts/contrib/spdx-release-date-report.py new file +193 -0
new file mode 100755
index 0000000000..be429409fc
@@ -0,0 +1,193 @@
+#! /usr/bin/env python3
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Author: Daniel Turull <[email protected]>
+#
+# Reports, per recipe, whether a valid releaseTime was recorded in SPDX
+# 3.0.1 output.
+#
+# AI-Generated: Uses Kiro (Claude)
+
+import argparse
+import csv
+import glob
+import json
+import logging
+import os
+import re
+import sys
+
+
+def load_jsonld_graph(path):
+    """Return the @graph list of a SPDX 3.0.1 JSON-LD document, or [] on error."""
+    try:
+        with open(path, "r", encoding="utf-8") as f:
+            data = json.load(f)
+    except (OSError, json.JSONDecodeError) as e:
+        logging.warning("Skipping %s: %s", path, e)
+        return []
+    return data.get("@graph", [])
+
+
+def collect_release_dates(deploy_dir):
+    """
+    Map recipe name -> releaseTime (or None) from all static/static-*.spdx.json
+    files found under deploy_dir.
+    """
+    release_dates = {}
+    pattern = os.path.join(deploy_dir, "**", "static", "static-*.spdx.json")
+    for path in sorted(glob.glob(pattern, recursive=True)):
+        for element in load_jsonld_graph(path):
+            if element.get("type") != "software_Package":
+                continue
+            name = element.get("name")
+            if not name:
+                continue
+            # First occurrence wins; the same recipe can appear for multiple
+            # arches (e.g. allarch vs machine-specific) with identical data.
+            release_dates.setdefault(name, element.get("releaseTime"))
+    return release_dates
+
+
+def _to_row(name, release_date):
+    return {
+        "recipe": name,
+        "release_date": release_date or "",
+        "valid_date": bool(release_date),
+    }
+
+
+def build_report(deploy_dir):
+    """
+    Build the report: [{"recipe": ..., "release_date": ..., "valid_date": bool}]
+    """
+    release_dates = collect_release_dates(deploy_dir)
+    return [_to_row(name, release_dates[name]) for name in sorted(release_dates)]
+
+
+# Per-recipe document namespace shared by all elements of that recipe in a
+# merged SBOM, e.g. http://spdx.org/spdxdocs/acl-<uuid>/<hash>/recipe/acl
+DOC_NAMESPACE_RE = re.compile(
+    r"^(https?://spdx\.org/spdxdocs/[^/]+-"
+    r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})/"
+)
+
+
+def doc_namespace(spdx_id):
+    """Extract the per-recipe document namespace from a SPDX ID, or None."""
+    match = DOC_NAMESPACE_RE.match(spdx_id or "")
+    return match.group(1) if match else None
+
+
+def build_report_from_file(spdx_file):
+    """
+    Build the same report as build_report(), but from a single merged SBOM
+    document (e.g. an image's *.rootfs.spdx.json) instead of a DEPLOY_DIR_SPDX
+    tree. Recipes are identified by their document namespace.
+    """
+    graph = load_jsonld_graph(spdx_file)
+
+    recipes = {}  # namespace -> {"name": ..., "release_date": ...}
+    for element in graph:
+        if element.get("type") != "software_Package":
+            continue
+
+        namespace = doc_namespace(element.get("spdxId"))
+        if namespace is None:
+            continue
+
+        if element.get("software_primaryPurpose") == "specification" and element.get("name"):
+            # The recipe itself, as opposed to its runtime package(s).
+            recipes[namespace] = {
+                "name": element["name"],
+                "release_date": element.get("releaseTime"),
+            }
+
+    report = [_to_row(info["name"], info["release_date"]) for info in recipes.values()]
+    report.sort(key=lambda r: r["recipe"])
+    return report
+
+
+def filter_rows(report, missing_only, sort_by_date=False):
+    rows = [r for r in report if not missing_only or not r["valid_date"]]
+    if sort_by_date:
+        rows.sort(key=lambda r: (not r["valid_date"], r["release_date"], r["recipe"]))
+    return rows
+
+
+def print_table(rows):
+    if not rows:
+        print("No matching recipes found.")
+        return
+
+    name_width = max(len("recipe"), *(len(r["recipe"]) for r in rows))
+    print(f"{'recipe':<{name_width}}  {'release_date':<21}  valid")
+    for r in rows:
+        print(f"{r['recipe']:<{name_width}}  {r['release_date']:<21}  {r['valid_date']}")
+
+
+def write_csv(rows, path):
+    with open(path, "w", newline="", encoding="utf-8") as f:
+        writer = csv.writer(f)
+        writer.writerow(["recipe", "release_date", "valid_date"])
+        for r in rows:
+            writer.writerow([r["recipe"], r["release_date"], r["valid_date"]])
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Report recipe release dates from SPDX 3.0.1 output"
+    )
+    parser.add_argument(
+        "--deploy-dir",
+        required=True,
+        help="Path to DEPLOY_DIR_SPDX (e.g. tmp/deploy/spdx/3.0.1) or to a "
+             "single merged image SBOM file (e.g. "
+             "tmp/deploy/images/<machine>/<image>.rootfs.spdx.json)",
+    )
+    parser.add_argument(
+        "--missing-only",
+        action="store_true",
+        help="Only report recipes without a valid release date",
+    )
+    parser.add_argument(
+        "--csv",
+        help="Write the report to a CSV file instead of only printing a table",
+    )
+    parser.add_argument(
+        "--sort-by-date",
+        action="store_true",
+        help="Sort output by release date instead of recipe name (missing dates last)",
+    )
+    args = parser.parse_args()
+
+    logging.basicConfig(format="[%(filename)s:%(lineno)d] %(message)s", level=logging.INFO)
+
+    if os.path.isdir(args.deploy_dir):
+        report = build_report(args.deploy_dir)
+    elif os.path.isfile(args.deploy_dir):
+        report = build_report_from_file(args.deploy_dir)
+    else:
+        parser.error(f"--deploy-dir {args.deploy_dir} does not exist")
+
+    total = len(report)
+    valid = sum(1 for r in report if r["valid_date"])
+    logging.info("Recipes with SPDX static data: %d", total)
+    logging.info("Recipes with a valid release date: %d", valid)
+    logging.info("Recipes missing a release date: %d", total - valid)
+
+    rows = filter_rows(report, args.missing_only, args.sort_by_date)
+    print_table(rows)
+
+    if args.csv:
+        write_csv(rows, args.csv)
+        logging.info("CSV report written to %s", args.csv)
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
#2 lore raw
In reply to [email protected]
Quoted 12 lines · Daniel Turull via lists.openembedded.org

From: Daniel Turull <[email protected]>

Record each recipe's release date in the releaseTime property of its software_Package object, using the SOURCE_DATE_EPOCH already computed for reproducible builds.

Accuracy depends on how SOURCE_DATE_EPOCH was derived: exact for git-tagged recipes, best-effort for tarball/http(s) sources. Some Python sdists (e.g. cryptography, hypothesis, maturin) normalize all file mtimes to a fixed placeholder, so their releaseTime reflects packaging-tool behavior, not the real release date.

Quoted 7 lines

Tested with oe-selftest -r spdx, and with bitbake world --runall=do_create_spdx: 1011/1150 recipes got a releaseTime (range 1998-12-30 to 2026-09-17), 139 correctly had none.

AI-Generated: Uses Kiro with Claude Sonnet 5

---

Hi Daniel,

Thanks for the new version, but it looks like the added test is failing:

2026-09-27 08:54:07,889 - oe-selftest - INFO - spdx.SPDX30Check.test_release_date_omitted_for_fallback_value (subunit.RemotedTestCase)
2026-09-27 08:54:07,890 - oe-selftest - INFO -  ... FAIL

Stderr:

2026-09-27 07:49:50,901 - oe-selftest - INFO - Adding: "include selftest.inc" in /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700487/conf/local.conf
2026-09-27 07:49:50,902 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2026-09-27 08:28:06,204 - oe-selftest - INFO - Found ANNOTATION2: ANNOTATION2=TestAnnotation2
2026-09-27 08:28:06,204 - oe-selftest - INFO - Found ANNOTATION1: ANNOTATION1=TestAnnotation1
2026-09-27 08:28:17,771 - oe-selftest - INFO - External references not found in SPDX output (defensive handling verified)
2026-09-27 08:51:21,993 - oe-selftest - INFO - The spdxId of gcc-16.2.0/README in build-gcc.spdx.json is http://spdx.org/spdxdocs/gcc-f2eaeb0d-b54b-53ba-899a-8c36c21139bf/8d0c5dabebec7cf453eedeadd0b28521a17ebf965ccdf6c437d3613676ca56dc/sourcefile/22
2026-09-27 08:54:07,890 - oe-selftest - INFO - 8: 51/64 792/807 (15.32s) (0 failed) (spdx.SPDX30Check.test_release_date_omitted_for_fallback_value)
2026-09-27 08:54:07,890 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 482, in test_release_date_omitted_for_fallback_value
    self.assertExists(sde_file)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)

AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700487/tmp/work/qemux86_64-poky-linux/packagegroup-base/1.0/source-date-epoch/__source_date_epoch.txt' does not exist

https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4933
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4740

Can you have a look at the issue?

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#3 lore raw
Quoted 41 lines · Mathieu Dubois-Briand
Quoted 12 lines · Daniel Turull via lists.openembedded.org

From: Daniel Turull <[email protected]>

Record each recipe's release date in the releaseTime property of its software_Package object, using the SOURCE_DATE_EPOCH already computed for reproducible builds.

Accuracy depends on how SOURCE_DATE_EPOCH was derived: exact for git-tagged recipes, best-effort for tarball/http(s) sources. Some Python sdists (e.g. cryptography, hypothesis, maturin) normalize all file mtimes to a fixed placeholder, so their releaseTime reflects packaging-tool behavior, not the real release date.

Quoted 7 lines

Tested with oe-selftest -r spdx, and with bitbake world --runall=do_create_spdx: 1011/1150 recipes got a releaseTime (range 1998-12-30 to 2026-09-17), 139 correctly had none.

AI-Generated: Uses Kiro with Claude Sonnet 5

---

Hi Daniel,

Thanks for the new version, but it looks like the added test is failing:

2026-09-27 08:54:07,889 - oe-selftest - INFO - spdx.SPDX30Check.test_release_date_omitted_for_fallback_value (subunit.RemotedTestCase)
2026-09-27 08:54:07,890 - oe-selftest - INFO -  ... FAIL

Stderr:

2026-09-27 07:49:50,901 - oe-selftest - INFO - Adding: "include selftest.inc" in /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700487/conf/local.conf
2026-09-27 07:49:50,902 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2026-09-27 08:28:06,204 - oe-selftest - INFO - Found ANNOTATION2: ANNOTATION2=TestAnnotation2
2026-09-27 08:28:06,204 - oe-selftest - INFO - Found ANNOTATION1: ANNOTATION1=TestAnnotation1
2026-09-27 08:28:17,771 - oe-selftest - INFO - External references not found in SPDX output (defensive handling verified)
2026-09-27 08:51:21,993 - oe-selftest - INFO - The spdxId of gcc-16.2.0/README in build-gcc.spdx.json is http://spdx.org/spdxdocs/gcc-f2eaeb0d-b54b-53ba-899a-8c36c21139bf/8d0c5dabebec7cf453eedeadd0b28521a17ebf965ccdf6c437d3613676ca56dc/sourcefile/22
2026-09-27 08:54:07,890 - oe-selftest - INFO - 8: 51/64 792/807 (15.32s) (0 failed) (spdx.SPDX30Check.test_release_date_omitted_for_fallback_value)
2026-09-27 08:54:07,890 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 482, in test_release_date_omitted_for_fallback_value
    self.assertExists(sde_file)
  File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists
    raise self.failureException(msg)
Quoted 6 lines

AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700487/tmp/work/qemux86_64-poky-linux/packagegroup-base/1.0/source-date-epoch/__source_date_epoch.txt' does not exist

https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4933
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4740

Can you have a look at the issue?

Sure, I'll take a look and provide a v3

Daniel

Quoted 4 lines

Thanks,
Mathieu