spdx: add support to include releaseTime
v6 cover spdx: add support to include releaseTime
From: Daniel Turull <[email protected]>
We have a requirements to include release time of open source components in the SBOM. There is a field specific for that in spdx 3 spec.
https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Properties/releaseTime/
This can also be used to evaluate how old are some of the core components and decide if they need replacement.
The previous 2 versions did not have cover letter.
In v4 I kept the simpler logic to just check for the epoch date, even if we have multiple sources with different release dates. It was getting complicated and adding more code for git and tarfiles. I can do a follow up patch after this simpler version gets in, so we can fine tune it.
Tested with oe-selftest -r spdx
v6: fix first commit for __CACHED_SOURCE_DATE_EPOCH and use same function for getting epoch as SOURCE_DATE_EPOCH uses.
Daniel Turull (3):
oe/reproducible: exclude __CACHED_SOURCE_DATE_EPOCH from task hash create-spdx-3.0: record component release date in SPDX output scripts/contrib: add spdx-release-date-report.py
meta/lib/oe/reproducible.py | 3 + meta/lib/oe/spdx30_tasks.py | 25 +++ meta/lib/oeqa/selftest/cases/spdx.py | 42 +++++ scripts/contrib/spdx-release-date-report.py | 191 ++++++++++++++++++++ 4 files changed, 261 insertions(+) create mode 100755 scripts/contrib/spdx-release-date-report.py
PATCH v6 1/3 oe/reproducible: exclude __CACHED_SOURCE_DATE_EPOCH from task hash
From: Daniel Turull <[email protected]>
epochfile_read() memoizes its result into __CACHED_SOURCE_DATE_EPOCH via d.setVar(), so its runtime value leaks into the basehash of any task that calls it, causing a "metadata is not deterministic" error once SDE_FILE is populated.
Exclude it, since it is not meaningful build metadata, using the bb.parse.vardepsexclude decorator on the function itself.
AI-Generated: Uses Kiro with Claude Sonnet 5
---
This should fix the selftest errors reported by Mathieu
v6: use bb.parse.vardepsexclude as requested by Richard
--- meta/lib/oe/reproducible.py | 3 +++ 1 file changed, 3 insertions(+)
meta/lib/oe/reproducible.py +3 -0
@@ -156,6 +156,9 @@ def get_source_date_epoch(d, sourcedir): fixed_source_date_epoch(d) # Last resort ) +# epochfile_read() memoizes into __CACHED_SOURCE_DATE_EPOCH via d.setVar(), so +# its runtime value must not leak into any task's hash. [email protected]("__CACHED_SOURCE_DATE_EPOCH") def epochfile_read(epochfile, d): cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None) if cached and efile == epochfile:
PATCH v6 2/3 create-spdx-3.0: record component release date in SPDX output
In reply to [email protected]From: Daniel Turull <[email protected]>
Record each package's release date in the releaseTime property of its software_Package object, using the source date epoch already computed for reproducible builds.
Accuracy depends on how the source date epoch was derived: exact for git-tagged recipes, best-effort for tarball/http(s) sources. Some Python sdists (e.g. cryptography, hypothesis, maturin) normalize all file mtimes to a fixed placeholder, so their releaseTime reflects packaging-tool behavior, not the real release date.
AI-Generated: Uses Kiro with Claude Sonnet 5
---
v2:
- Dropped all options per Joshua's feedback; read SDE_FILE directly.
- Dropped the redundant else: delattr(recipe, "releaseTime") branch.
- Selftest compares against SDE_FILE content directly instead of SOURCE_DATE_EPOCH, which can diverge from it.
- Fixed a leak: recipes with no git checkout and no fetched source had SDE_FILE holding only SOURCE_DATE_EPOCH_FALLBACK, showing a bogus 2011-04-05T23:00:00Z releaseTime instead of none.
v3:
- Also run after do_unpack: do_deploy_source_date_epoch's setscene shortcut can skip it, leaving SOURCE_DATE_EPOCH unset.
- get_release_date() reads SOURCE_DATE_EPOCH again instead of SDE_FILE, now that they're guaranteed equivalent.
- test_release_date_source_date_epoch: switched to tar (base-files has S == UNPACKDIR and never gets a real SOURCE_DATE_EPOCH).
- Added test_release_date_omitted_for_fallback_value.
v4:
- Per Richard's feedback, moved releaseTime from do_create_recipe_spdx to do_create_spdx to keep the recipe-only task fetch-free.
- Per Joshua's suggestion added into each downloaded source software_Package
- Updated selftests accordingly.
v5:
- test_release_date_source_date_epoch: force do_unpack before reading SOURCE_DATE_EPOCH. Mathieu hit this on the autobuilder: with all tasks restored from sstate do_unpack never runs, so SDE_FILE is missing and SOURCE_DATE_EPOCH falls back, making the test compare the correct releaseTime against the fallback date.
v6:
- Per Joshua's feedback, get_release_date() now reads SDE_FILE via oe.reproducible.epochfile_read() instead of SOURCE_DATE_EPOCH, since a recipe can set SOURCE_DATE_EPOCH itself to override the computed value, and that override is not a release date.
- Added test_release_date_ignores_source_date_epoch_override.
--- meta/lib/oe/spdx30_tasks.py | 25 +++++++++++++++++ meta/lib/oeqa/selftest/cases/spdx.py | 42 ++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+)
meta/lib/oe/spdx30_tasks.py +25 -0
@@ -9,6 +9,7 @@ import oe.cve_check import oe.license import oe.packagedata import oe.patch +import oe.reproducible import oe.sbom30 import oe.sdk import oe.spdx30
@@ -36,6 +37,26 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) +def get_release_date(d): + """Resolve the release date to record in a package's releaseTime property. + + Reads SDE_FILE via oe.reproducible.epochfile_read() directly, rather than + SOURCE_DATE_EPOCH, since a recipe may set SOURCE_DATE_EPOCH itself to + override the computed value, and that override is not a release date. + + Omits the fallback value since it is not a meaningful release date. + + Returns a datetime, or None if no release date should be recorded. + """ + source_date_epoch = oe.reproducible.epochfile_read(d.getVar("SDE_FILE"), d) + if not source_date_epoch or source_date_epoch == d.getVar( + "SOURCE_DATE_EPOCH_FALLBACK" + ): + return None + + return datetime.fromtimestamp(int(source_date_epoch), tz=timezone.utc) + + def add_license_expression( d, objset, license_expression, license_data, search_objsets=[] ):
@@ -441,6 +462,7 @@ def _enrich_source_package(d, dl, fd, file_name, primary_purpose): def add_download_files(d, objset): inputs = set() + release_date = get_release_date(d) urls = d.getVar("SRC_URI").split() fetch = bb.fetch.Fetch(urls, d)
@@ -504,6 +526,9 @@ def add_download_files(d, objset): _enrich_source_package(d, dl, fd, file_name, primary_purpose) + if release_date is not None: + dl.releaseTime = release_date + if fd.method.supports_checksum(fd): for checksum_id in bb.fetch.CHECKSUM_LIST: if checksum_id not in oe.spdx30.HashAlgorithm.NAMED_INDIVIDUALS:
meta/lib/oeqa/selftest/cases/spdx.py +42 -0
51 lines not on this page · raw message
PATCH v6 3/3 scripts/contrib: add spdx-release-date-report.py
In reply to [email protected]From: Daniel Turull <[email protected]>
Reports each recipe's releaseTime (added by the create-spdx-3.0 patch) from either a DEPLOY_DIR_SPDX tree or a single merged image SBOM. Used to spot-check release dates across a build and surface recipes missing one; helped find the SOURCE_DATE_EPOCH_FALLBACK leak fixed by the preceding patch.
AI-Generated: Uses Kiro with Claude Sonnet 5
---
v4:
- Rework to match the new placement of the releaseTime.
--- scripts/contrib/spdx-release-date-report.py | 191 ++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100755 scripts/contrib/spdx-release-date-report.py
scripts/contrib/spdx-release-date-report.py new file +191 -0
@@ -0,0 +1,191 @@ +#! /usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Author: Daniel Turull <[email protected]> +# +# Reports, per recipe, whether a valid releaseTime was recorded in SPDX +# 3.0.1 output. +# +# AI-Generated: Uses Kiro (Claude) + +import argparse +import csv +import glob +import json +import logging +import os +import re +import sys + + +# Each do_create_spdx software_Package's spdxId is +# "<SPDX_NAMESPACE_PREFIX>/<PN>-<uuid5>/<unihash>/...", so the recipe name +# can be recovered directly from the id without walking relationships. +RECIPE_FROM_SPDX_ID_RE = re.compile( + r"^.*/([^/]+)-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/" +) + + +def recipe_name_from_spdx_id(spdx_id): + """Extract the PN that generated an element, from its spdxId, or None.""" + match = RECIPE_FROM_SPDX_ID_RE.match(spdx_id or "") + return match.group(1) if match else None + + +def load_jsonld_graph(path): + """Return the @graph list of a SPDX 3.0.1 JSON-LD document, or [] on error.""" + try: + with open(path, "r", encoding="utf-8") as f: + data = json.load(f) + except (OSError, json.JSONDecodeError) as e: + logging.warning("Skipping %s: %s", path, e) + return [] + return data.get("@graph", []) + + +def extract_release_dates(elements): + """ + Map recipe name -> releaseTime (or None), for each downloaded source + software_Package found in elements. The recipe name is recovered from + each package's own spdxId (see recipe_name_from_spdx_id). For a given + name, the first package with a releaseTime wins; if none have one, the + name still appears, mapped to None. + """ + release_dates = {} + for element in elements: + if element.get("type") != "software_Package": + continue + if not element.get("software_downloadLocation"): + continue + name = recipe_name_from_spdx_id(element.get("spdxId")) + if not name: + continue + release_dates.setdefault(name, None) + release_time = element.get("releaseTime") + if release_time and not release_dates[name]: + release_dates[name] = release_time + return release_dates + + +def iter_spdx_elements(path): + """Yield @graph elements from path: every build-*.spdx.json under path + if it's a DEPLOY_DIR_SPDX tree, or path's own @graph if it's a single + SPDX JSON-LD file (e.g. a merged image SBOM). + + Only build-*.spdx.json is read for a DEPLOY_DIR_SPDX tree: it's the + output of do_create_spdx, the only recipe task that calls + add_download_files and so the only one that can carry + software_downloadLocation/releaseTime. The other per-recipe documents + (static-*.spdx.json from the fetch-free do_create_recipe_spdx, and + package-*.spdx.json for binary packages) never have that data. + """ + if os.path.isdir(path): + build_pattern = os.path.join(path, "**", "builds", "build-*.spdx.json") + for build_path in sorted(glob.glob(build_pattern, recursive=True)): + yield from load_jsonld_graph(build_path) + else: + yield from load_jsonld_graph(path) + + +def _to_row(name, release_date): + return { + "recipe": name, + "release_date": release_date or "", + } + + +def build_report(path): + """ + Build the report: [{"recipe": ..., "release_date": ...}], from either a + DEPLOY_DIR_SPDX tree or a single merged image SBOM file. + """ + release_dates = extract_release_dates(iter_spdx_elements(path)) + return [_to_row(name, release_dates[name]) for name in sorted(release_dates)] + + +def drop_redundant_native(report): + """Drop foo-native rows when foo is also present in the report.""" + names = {r["recipe"] for r in report} + return [ + r + for r in report + if not (r["recipe"].endswith("-native") and r["recipe"][:-len("-native")] in names) + ] + + +def filter_rows(report, sort_by_date=False): + rows = list(report) + if sort_by_date: + rows.sort(key=lambda r: (not r["release_date"], r["release_date"], r["recipe"])) + return rows + + +def print_table(rows): + if not rows: + print("No matching recipes found.") + return + + name_width = max(len("recipe"), *(len(r["recipe"]) for r in rows)) + print(f"{'recipe':<{name_width}} release_date") + for r in rows: + print(f"{r['recipe']:<{name_width}} {r['release_date']}") + + +def write_csv(rows, path): + with open(path, "w", newline="", encoding="utf-8") as f: + writer = csv.writer(f) + writer.writerow(["recipe", "release_date"]) + for r in rows: + writer.writerow([r["recipe"], r["release_date"]]) + + +def main(): + parser = argparse.ArgumentParser( + description="Report recipe release dates from SPDX 3.0.1 output" + ) + parser.add_argument( + "path", + help="Path to DEPLOY_DIR_SPDX (e.g. tmp/deploy/spdx/3.0.1) or to a " + "single merged image SBOM file (e.g. " + "tmp/deploy/images/<machine>/<image>.rootfs.spdx.json)", + ) + parser.add_argument( + "--csv", + help="Write the report to a CSV file instead of only printing a table", + ) + parser.add_argument( + "--sort-by-date", + action="store_true", + help="Sort output by release date instead of recipe name (missing dates last)", + ) + args = parser.parse_args() + + logging.basicConfig(format="[%(filename)s:%(lineno)d] %(message)s", level=logging.INFO) + + if not os.path.isdir(args.path) and not os.path.isfile(args.path): + parser.error(f"{args.path} does not exist") + + report = build_report(args.path) + report = drop_redundant_native(report) + + total = len(report) + found = sum(1 for r in report if r["release_date"]) + logging.info("Recipes with SPDX package data: %d", total) + logging.info("Recipes with a release date: %d", found) + logging.info("Recipes missing a release date: %d", total - found) + + rows = filter_rows(report, args.sort_by_date) + print_table(rows) + + if args.csv: + write_csv(rows, args.csv) + logging.info("CSV report written to %s", args.csv) + + return 0 + + +if __name__ == "__main__": + sys.exit(main())