create-spdx-3.0: record component release date in SPDX output
PATCH v2 1/2 create-spdx-3.0: record component release date in SPDX output
From: Daniel Turull <[email protected]>
Record each recipe's release date in the releaseTime property of its software_Package object, using the SOURCE_DATE_EPOCH already computed for reproducible builds.
Accuracy depends on how SOURCE_DATE_EPOCH was derived: exact for git-tagged recipes, best-effort for tarball/http(s) sources. Some Python sdists (e.g. cryptography, hypothesis, maturin) normalize all file mtimes to a fixed placeholder, so their releaseTime reflects packaging-tool behavior, not the real release date.
Tested with oe-selftest -r spdx, and with bitbake world --runall=do_create_spdx: 1011/1150 recipes got a releaseTime (range 1998-12-30 to 2026-09-17), 139 correctly had none.
AI-Generated: Uses Kiro with Claude Sonnet 5
---
v2:
- Dropped all options per Joshua's feedback; read SDE_FILE directly.
- Dropped the redundant else: delattr(recipe, "releaseTime") branch.
- Selftest compares against SDE_FILE content directly instead of SOURCE_DATE_EPOCH, which can diverge from it.
- Fixed a leak: recipes with no git checkout and no fetched source had SDE_FILE holding only SOURCE_DATE_EPOCH_FALLBACK, showing a bogus 2011-04-05T23:00:00Z releaseTime instead of none.
--- meta/classes/create-spdx-3.0.bbclass | 2 +- meta/lib/oe/spdx30_tasks.py | 27 +++++++++++++++ meta/lib/oeqa/selftest/cases/spdx.py | 52 ++++++++++++++++++++++++++++ 3 files changed, 80 insertions(+), 1 deletion(-)
meta/classes/create-spdx-3.0.bbclass +1 -1
@@ -192,7 +192,7 @@ python do_create_recipe_spdx() { import oe.spdx30_tasks oe.spdx30_tasks.create_recipe_spdx(d) } -addtask do_create_recipe_spdx +addtask do_create_recipe_spdx after do_deploy_source_date_epoch SSTATETASKS += "do_create_recipe_spdx" do_create_recipe_spdx[sstate-inputdirs] = "${SPDXRECIPEDEPLOY}"
meta/lib/oe/spdx30_tasks.py +27 -0
@@ -9,6 +9,7 @@ import oe.cve_check import oe.license import oe.packagedata import oe.patch +import oe.reproducible import oe.sbom30 import oe.sdk import oe.spdx30
@@ -36,6 +37,28 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) +def get_release_date(d): + """Resolve the release date to record in a recipe's releaseTime property. + + Uses the SOURCE_DATE_EPOCH already determined for reproducible builds + (SDE_FILE), without falling back to SOURCE_DATE_EPOCH_FALLBACK, since a + fixed fallback timestamp is not a meaningful release date. + + Returns a datetime, or None if no release date should be recorded. + """ + sde_file = d.getVar("SDE_FILE") + if not sde_file or not os.path.isfile(sde_file): + return None + + source_date_epoch = oe.reproducible.epochfile_read(sde_file, d) + if source_date_epoch == d.getVar("SOURCE_DATE_EPOCH_FALLBACK"): + # SDE_FILE exists but its value is only the fallback (e.g. no git + # checkout and no fetched source to derive a date from). + return None + + return datetime.fromtimestamp(int(source_date_epoch), tz=timezone.utc) + + def add_license_expression( d, objset, license_expression, license_data, search_objsets=[] ):
@@ -633,6 +656,10 @@ def create_recipe_spdx(d): if val := d.getVar("DESCRIPTION"): recipe.description = val + release_date = get_release_date(d) + if release_date is not None: + recipe.releaseTime = release_date + for cpe_id in oe.cve_check.get_cpe_ids( d.getVar("CVE_PRODUCT"), d.getVar("CVE_VERSION") ):
meta/lib/oeqa/selftest/cases/spdx.py +52 -0
61 lines not on this page · raw message
PATCH v2 2/2 scripts/contrib: add spdx-release-date-report.py
From: Daniel Turull <[email protected]>
Reports each recipe's releaseTime (added by the create-spdx-3.0 patch) from either a DEPLOY_DIR_SPDX tree or a single merged image SBOM. Used to spot-check release dates across a build and surface recipes missing one; helped find the SOURCE_DATE_EPOCH_FALLBACK leak and archive-mtime issues fixed by the two preceding patches.
AI-Generated: Uses Kiro with Claude Sonnet 5
--- scripts/contrib/spdx-release-date-report.py | 193 ++++++++++++++++++++ 1 file changed, 193 insertions(+) create mode 100755 scripts/contrib/spdx-release-date-report.py
scripts/contrib/spdx-release-date-report.py new file +193 -0
@@ -0,0 +1,193 @@ +#! /usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Author: Daniel Turull <[email protected]> +# +# Reports, per recipe, whether a valid releaseTime was recorded in SPDX +# 3.0.1 output. +# +# AI-Generated: Uses Kiro (Claude) + +import argparse +import csv +import glob +import json +import logging +import os +import re +import sys + + +def load_jsonld_graph(path): + """Return the @graph list of a SPDX 3.0.1 JSON-LD document, or [] on error.""" + try: + with open(path, "r", encoding="utf-8") as f: + data = json.load(f) + except (OSError, json.JSONDecodeError) as e: + logging.warning("Skipping %s: %s", path, e) + return [] + return data.get("@graph", []) + + +def collect_release_dates(deploy_dir): + """ + Map recipe name -> releaseTime (or None) from all static/static-*.spdx.json + files found under deploy_dir. + """ + release_dates = {} + pattern = os.path.join(deploy_dir, "**", "static", "static-*.spdx.json") + for path in sorted(glob.glob(pattern, recursive=True)): + for element in load_jsonld_graph(path): + if element.get("type") != "software_Package": + continue + name = element.get("name") + if not name: + continue + # First occurrence wins; the same recipe can appear for multiple + # arches (e.g. allarch vs machine-specific) with identical data. + release_dates.setdefault(name, element.get("releaseTime")) + return release_dates + + +def _to_row(name, release_date): + return { + "recipe": name, + "release_date": release_date or "", + "valid_date": bool(release_date), + } + + +def build_report(deploy_dir): + """ + Build the report: [{"recipe": ..., "release_date": ..., "valid_date": bool}] + """ + release_dates = collect_release_dates(deploy_dir) + return [_to_row(name, release_dates[name]) for name in sorted(release_dates)] + + +# Per-recipe document namespace shared by all elements of that recipe in a +# merged SBOM, e.g. http://spdx.org/spdxdocs/acl-<uuid>/<hash>/recipe/acl +DOC_NAMESPACE_RE = re.compile( + r"^(https?://spdx\.org/spdxdocs/[^/]+-" + r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12})/" +) + + +def doc_namespace(spdx_id): + """Extract the per-recipe document namespace from a SPDX ID, or None.""" + match = DOC_NAMESPACE_RE.match(spdx_id or "") + return match.group(1) if match else None + + +def build_report_from_file(spdx_file): + """ + Build the same report as build_report(), but from a single merged SBOM + document (e.g. an image's *.rootfs.spdx.json) instead of a DEPLOY_DIR_SPDX + tree. Recipes are identified by their document namespace. + """ + graph = load_jsonld_graph(spdx_file) + + recipes = {} # namespace -> {"name": ..., "release_date": ...} + for element in graph: + if element.get("type") != "software_Package": + continue + + namespace = doc_namespace(element.get("spdxId")) + if namespace is None: + continue + + if element.get("software_primaryPurpose") == "specification" and element.get("name"): + # The recipe itself, as opposed to its runtime package(s). + recipes[namespace] = { + "name": element["name"], + "release_date": element.get("releaseTime"), + } + + report = [_to_row(info["name"], info["release_date"]) for info in recipes.values()] + report.sort(key=lambda r: r["recipe"]) + return report + + +def filter_rows(report, missing_only, sort_by_date=False): + rows = [r for r in report if not missing_only or not r["valid_date"]] + if sort_by_date: + rows.sort(key=lambda r: (not r["valid_date"], r["release_date"], r["recipe"])) + return rows + + +def print_table(rows): + if not rows: + print("No matching recipes found.") + return + + name_width = max(len("recipe"), *(len(r["recipe"]) for r in rows)) + print(f"{'recipe':<{name_width}} {'release_date':<21} valid") + for r in rows: + print(f"{r['recipe']:<{name_width}} {r['release_date']:<21} {r['valid_date']}") + + +def write_csv(rows, path): + with open(path, "w", newline="", encoding="utf-8") as f: + writer = csv.writer(f) + writer.writerow(["recipe", "release_date", "valid_date"]) + for r in rows: + writer.writerow([r["recipe"], r["release_date"], r["valid_date"]]) + + +def main(): + parser = argparse.ArgumentParser( + description="Report recipe release dates from SPDX 3.0.1 output" + ) + parser.add_argument( + "--deploy-dir", + required=True, + help="Path to DEPLOY_DIR_SPDX (e.g. tmp/deploy/spdx/3.0.1) or to a " + "single merged image SBOM file (e.g. " + "tmp/deploy/images/<machine>/<image>.rootfs.spdx.json)", + ) + parser.add_argument( + "--missing-only", + action="store_true", + help="Only report recipes without a valid release date", + ) + parser.add_argument( + "--csv", + help="Write the report to a CSV file instead of only printing a table", + ) + parser.add_argument( + "--sort-by-date", + action="store_true", + help="Sort output by release date instead of recipe name (missing dates last)", + ) + args = parser.parse_args() + + logging.basicConfig(format="[%(filename)s:%(lineno)d] %(message)s", level=logging.INFO) + + if os.path.isdir(args.deploy_dir): + report = build_report(args.deploy_dir) + elif os.path.isfile(args.deploy_dir): + report = build_report_from_file(args.deploy_dir) + else: + parser.error(f"--deploy-dir {args.deploy_dir} does not exist") + + total = len(report) + valid = sum(1 for r in report if r["valid_date"]) + logging.info("Recipes with SPDX static data: %d", total) + logging.info("Recipes with a valid release date: %d", valid) + logging.info("Recipes missing a release date: %d", total - valid) + + rows = filter_rows(report, args.missing_only, args.sort_by_date) + print_table(rows) + + if args.csv: + write_csv(rows, args.csv) + logging.info("CSV report written to %s", args.csv) + + return 0 + + +if __name__ == "__main__": + sys.exit(main())
Quoted 12 lines · Daniel Turull via lists.openembedded.org
From: Daniel Turull <[email protected]>
Record each recipe's release date in the releaseTime property of its
software_Packageobject, using theSOURCE_DATE_EPOCHalready computed for reproducible builds.Accuracy depends on how
SOURCE_DATE_EPOCHwas derived: exact for git-tagged recipes, best-effort for tarball/http(s) sources. Some Python sdists (e.g. cryptography, hypothesis, maturin) normalize all file mtimes to a fixed placeholder, so their releaseTime reflects packaging-tool behavior, not the real release date.
Quoted 7 lines
Tested with oe-selftest -r spdx, and with
bitbake world --runall=do_create_spdx: 1011/1150 recipes got a releaseTime (range 1998-12-30 to 2026-09-17), 139 correctly had none.AI-Generated: Uses Kiro with Claude Sonnet 5
Signed-off-by Daniel Turull <[email protected]>---
Hi Daniel,
Thanks for the new version, but it looks like the added test is failing:
2026-09-27 08:54:07,889 - oe-selftest - INFO - spdx.SPDX30Check.test_release_date_omitted_for_fallback_value (subunit.RemotedTestCase) 2026-09-27 08:54:07,890 - oe-selftest - INFO - ... FAIL
Stderr:
2026-09-27 07:49:50,901 - oe-selftest - INFO - Adding: "include selftest.inc" in /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700487/conf/local.conf 2026-09-27 07:49:50,902 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2026-09-27 08:28:06,204 - oe-selftest - INFO - Found ANNOTATION2: ANNOTATION2=TestAnnotation2 2026-09-27 08:28:06,204 - oe-selftest - INFO - Found ANNOTATION1: ANNOTATION1=TestAnnotation1 2026-09-27 08:28:17,771 - oe-selftest - INFO - External references not found in SPDX output (defensive handling verified) 2026-09-27 08:51:21,993 - oe-selftest - INFO - The spdxId of gcc-16.2.0/README in build-gcc.spdx.json is http://spdx.org/spdxdocs/gcc-f2eaeb0d-b54b-53ba-899a-8c36c21139bf/8d0c5dabebec7cf453eedeadd0b28521a17ebf965ccdf6c437d3613676ca56dc/sourcefile/22 2026-09-27 08:54:07,890 - oe-selftest - INFO - 8: 51/64 792/807 (15.32s) (0 failed) (spdx.SPDX30Check.test_release_date_omitted_for_fallback_value) 2026-09-27 08:54:07,890 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 482, in test_release_date_omitted_for_fallback_value self.assertExists(sde_file) File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists raise self.failureException(msg)
AssertionError: '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700487/tmp/work/qemux86_64-poky-linux/packagegroup-base/1.0/source-date-epoch/__source_date_epoch.txt' does not exist
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4933
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4740
Can you have a look at the issue?
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 41 lines · Mathieu Dubois-Briand
Quoted 12 lines · Daniel Turull via lists.openembedded.org
From: Daniel Turull <[email protected]>
Record each recipe's release date in the releaseTime property of its
software_Packageobject, using theSOURCE_DATE_EPOCHalready computed for reproducible builds.Accuracy depends on how
SOURCE_DATE_EPOCHwas derived: exact for git-tagged recipes, best-effort for tarball/http(s) sources. Some Python sdists (e.g. cryptography, hypothesis, maturin) normalize all file mtimes to a fixed placeholder, so their releaseTime reflects packaging-tool behavior, not the real release date.Quoted 7 lines
Tested with oe-selftest -r spdx, and with
bitbake world --runall=do_create_spdx: 1011/1150 recipes got a releaseTime (range 1998-12-30 to 2026-09-17), 139 correctly had none.AI-Generated: Uses Kiro with Claude Sonnet 5
Signed-off-by Daniel Turull <[email protected]>---
Hi Daniel,
Thanks for the new version, but it looks like the added test is failing:
2026-09-27 08:54:07,889 - oe-selftest - INFO - spdx.SPDX30Check.test_release_date_omitted_for_fallback_value (subunit.RemotedTestCase) 2026-09-27 08:54:07,890 - oe-selftest - INFO - ... FAILStderr:
2026-09-27 07:49:50,901 - oe-selftest - INFO - Adding: "include selftest.inc" in /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700487/conf/local.conf 2026-09-27 07:49:50,902 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf 2026-09-27 08:28:06,204 - oe-selftest - INFO - Found ANNOTATION2: ANNOTATION2=TestAnnotation2 2026-09-27 08:28:06,204 - oe-selftest - INFO - Found ANNOTATION1: ANNOTATION1=TestAnnotation1 2026-09-27 08:28:17,771 - oe-selftest - INFO - External references not found in SPDX output (defensive handling verified) 2026-09-27 08:51:21,993 - oe-selftest - INFO - The spdxId of gcc-16.2.0/README in build-gcc.spdx.json is http://spdx.org/spdxdocs/gcc-f2eaeb0d-b54b-53ba-899a-8c36c21139bf/8d0c5dabebec7cf453eedeadd0b28521a17ebf965ccdf6c437d3613676ca56dc/sourcefile/22 2026-09-27 08:54:07,890 - oe-selftest - INFO - 8: 51/64 792/807 (15.32s) (0 failed) (spdx.SPDX30Check.test_release_date_omitted_for_fallback_value) 2026-09-27 08:54:07,890 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 482, in test_release_date_omitted_for_fallback_value self.assertExists(sde_file) File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/case.py", line 249, in assertExists raise self.failureException(msg)
Quoted 6 lines
AssertionError: '
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-2700487/tmp/work/qemux86_64-poky-linux/packagegroup-base/1.0/source-date-epoch/__source_date_epoch.txt' does not existhttps://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4933
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4740Can you have a look at the issue?
Sure, I'll take a look and provide a v3
Daniel
Quoted 4 lines
Thanks,
Mathieu