spdx: add support to include releaseTime
v4 cover spdx: add support to include releaseTime
From: Daniel Turull <[email protected]>
We have a requirements to include release time of open source components in the SBOM. There is a field specific for that in spdx 3 spec.
https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Properties/releaseTime/
This can also be used to evaluate how old are some of the core components and decide if they need replacement.
The previous 2 versions did not have cover letter.
In v4 I kept the simpler logic to just check for the epoch date, even if we have multiple sources with different release dates. It was getting complicated and adding more code for git and tarfiles. I can do a follow up patch after this simpler version gets in, so we can fine tune it.
Tested with oe-selftest -r spdx
Daniel Turull (3):
classes/base: exclude __CACHED_SOURCE_DATE_EPOCH from task hash create-spdx-3.0: record component release date in SPDX output scripts/contrib: add spdx-release-date-report.py
meta/classes-global/base.bbclass | 4 + meta/lib/oe/spdx30_tasks.py | 22 +++ meta/lib/oeqa/selftest/cases/spdx.py | 41 +++++ scripts/contrib/spdx-release-date-report.py | 191 ++++++++++++++++++++ 4 files changed, 258 insertions(+) create mode 100755 scripts/contrib/spdx-release-date-report.py
PATCH v4 1/3 classes/base: exclude __CACHED_SOURCE_DATE_EPOCH from task hash
From: Daniel Turull <[email protected]>
epochfile_read() memoizes its result into __CACHED_SOURCE_DATE_EPOCH via d.setVar(), so its runtime value leaks into the basehash of any task that calls it, causing a "metadata is not deterministic" error once SDE_FILE is populated.
Exclude it, since it is not meaningful build metadata.
AI-Generated: Uses Kiro with Claude Sonnet 5
---
This should fix the selftest errors reported by Mathieu
--- meta/classes-global/base.bbclass | 4 ++++ 1 file changed, 4 insertions(+)
meta/classes-global/base.bbclass +4 -0
@@ -249,6 +249,10 @@ do_unpack[postfuncs] += "create_source_date_epoch_stamp" def get_source_date_epoch_value(d): return oe.reproducible.epochfile_read(d.getVar('SDE_FILE'), d) +# epochfile_read() memoizes into this variable via d.setVar(), so its +# runtime value must not leak into any task's hash. +oe.reproducible.epochfile_read[vardepsexclude] = "__CACHED_SOURCE_DATE_EPOCH" + def get_layers_branch_rev(d): revisions = oe.buildcfg.get_layer_revisions(d) layers_branch_rev = ["%-20s = \"%s:%s\"" % (r[1], r[2], r[3]) for r in revisions]
PATCH v4 2/3 create-spdx-3.0: record component release date in SPDX output
In reply to [email protected]From: Daniel Turull <[email protected]>
Record each package's release date in the releaseTime property of its software_Package object, using the SOURCE_DATE_EPOCH already computed for reproducible builds.
Accuracy depends on how SOURCE_DATE_EPOCH was derived: exact for git-tagged recipes, best-effort for tarball/http(s) sources. Some Python sdists (e.g. cryptography, hypothesis, maturin) normalize all file mtimes to a fixed placeholder, so their releaseTime reflects packaging-tool behavior, not the real release date.
AI-Generated: Uses Kiro with Claude Sonnet 5
---
v2:
- Dropped all options per Joshua's feedback; read SDE_FILE directly.
- Dropped the redundant else: delattr(recipe, "releaseTime") branch.
- Selftest compares against SDE_FILE content directly instead of SOURCE_DATE_EPOCH, which can diverge from it.
- Fixed a leak: recipes with no git checkout and no fetched source had SDE_FILE holding only SOURCE_DATE_EPOCH_FALLBACK, showing a bogus 2011-04-05T23:00:00Z releaseTime instead of none.
v3:
- Also run after do_unpack: do_deploy_source_date_epoch's setscene shortcut can skip it, leaving SOURCE_DATE_EPOCH unset.
- get_release_date() reads SOURCE_DATE_EPOCH again instead of SDE_FILE, now that they're guaranteed equivalent.
- test_release_date_source_date_epoch: switched to tar (base-files has S == UNPACKDIR and never gets a real SOURCE_DATE_EPOCH).
- Added test_release_date_omitted_for_fallback_value.
v4:
- Per Richard's feedback, moved releaseTime from do_create_recipe_spdx to do_create_spdx to keep the recipe-only task fetch-free.
- Per Joshua's suggestion added into each downloaded source software_Package
- Updated selftests accordingly.
--- meta/lib/oe/spdx30_tasks.py | 22 +++++++++++++++ meta/lib/oeqa/selftest/cases/spdx.py | 41 ++++++++++++++++++++++++++++ 2 files changed, 63 insertions(+)
meta/lib/oe/spdx30_tasks.py +22 -0
@@ -36,6 +36,24 @@ def set_timestamp_now(d, o, prop): delattr(o, prop) +def get_release_date(d): + """Resolve the release date to record in a package's releaseTime property. + + Uses SOURCE_DATE_EPOCH, which by this point (do_create_spdx runs after + do_unpack and do_deploy_source_date_epoch) reflects the source mtimes. + Omits the fallback value since it is not a meaningful release date. + + Returns a datetime, or None if no release date should be recorded. + """ + source_date_epoch = d.getVar("SOURCE_DATE_EPOCH") + if not source_date_epoch or source_date_epoch == d.getVar( + "SOURCE_DATE_EPOCH_FALLBACK" + ): + return None + + return datetime.fromtimestamp(int(source_date_epoch), tz=timezone.utc) + + def add_license_expression( d, objset, license_expression, license_data, search_objsets=[] ):
@@ -441,6 +459,7 @@ def _enrich_source_package(d, dl, fd, file_name, primary_purpose): def add_download_files(d, objset): inputs = set() + release_date = get_release_date(d) urls = d.getVar("SRC_URI").split() fetch = bb.fetch.Fetch(urls, d)
@@ -504,6 +523,9 @@ def add_download_files(d, objset): _enrich_source_package(d, dl, fd, file_name, primary_purpose) + if release_date is not None: + dl.releaseTime = release_date + if fd.method.supports_checksum(fd): for checksum_id in bb.fetch.CHECKSUM_LIST: if checksum_id not in oe.spdx30.HashAlgorithm.NAMED_INDIVIDUALS:
meta/lib/oeqa/selftest/cases/spdx.py +41 -0
50 lines not on this page · raw message
PATCH v4 3/3 scripts/contrib: add spdx-release-date-report.py
In reply to [email protected]From: Daniel Turull <[email protected]>
Reports each recipe's releaseTime (added by the create-spdx-3.0 patch) from either a DEPLOY_DIR_SPDX tree or a single merged image SBOM. Used to spot-check release dates across a build and surface recipes missing one; helped find the SOURCE_DATE_EPOCH_FALLBACK leak fixed by the preceding patch.
AI-Generated: Uses Kiro with Claude Sonnet 5
---
v4:
- Rework to match the new placement of the releaseTime.
--- scripts/contrib/spdx-release-date-report.py | 191 ++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100755 scripts/contrib/spdx-release-date-report.py
scripts/contrib/spdx-release-date-report.py new file +191 -0
@@ -0,0 +1,191 @@ +#! /usr/bin/env python3 +# +# Copyright OpenEmbedded Contributors +# +# SPDX-License-Identifier: GPL-2.0-only +# +# Author: Daniel Turull <[email protected]> +# +# Reports, per recipe, whether a valid releaseTime was recorded in SPDX +# 3.0.1 output. +# +# AI-Generated: Uses Kiro (Claude) + +import argparse +import csv +import glob +import json +import logging +import os +import re +import sys + + +# Each do_create_spdx software_Package's spdxId is +# "<SPDX_NAMESPACE_PREFIX>/<PN>-<uuid5>/<unihash>/...", so the recipe name +# can be recovered directly from the id without walking relationships. +RECIPE_FROM_SPDX_ID_RE = re.compile( + r"^.*/([^/]+)-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/" +) + + +def recipe_name_from_spdx_id(spdx_id): + """Extract the PN that generated an element, from its spdxId, or None.""" + match = RECIPE_FROM_SPDX_ID_RE.match(spdx_id or "") + return match.group(1) if match else None + + +def load_jsonld_graph(path): + """Return the @graph list of a SPDX 3.0.1 JSON-LD document, or [] on error.""" + try: + with open(path, "r", encoding="utf-8") as f: + data = json.load(f) + except (OSError, json.JSONDecodeError) as e: + logging.warning("Skipping %s: %s", path, e) + return [] + return data.get("@graph", []) + + +def extract_release_dates(elements): + """ + Map recipe name -> releaseTime (or None), for each downloaded source + software_Package found in elements. The recipe name is recovered from + each package's own spdxId (see recipe_name_from_spdx_id). For a given + name, the first package with a releaseTime wins; if none have one, the + name still appears, mapped to None. + """ + release_dates = {} + for element in elements: + if element.get("type") != "software_Package": + continue + if not element.get("software_downloadLocation"): + continue + name = recipe_name_from_spdx_id(element.get("spdxId")) + if not name: + continue + release_dates.setdefault(name, None) + release_time = element.get("releaseTime") + if release_time and not release_dates[name]: + release_dates[name] = release_time + return release_dates + + +def iter_spdx_elements(path): + """Yield @graph elements from path: every build-*.spdx.json under path + if it's a DEPLOY_DIR_SPDX tree, or path's own @graph if it's a single + SPDX JSON-LD file (e.g. a merged image SBOM). + + Only build-*.spdx.json is read for a DEPLOY_DIR_SPDX tree: it's the + output of do_create_spdx, the only recipe task that calls + add_download_files and so the only one that can carry + software_downloadLocation/releaseTime. The other per-recipe documents + (static-*.spdx.json from the fetch-free do_create_recipe_spdx, and + package-*.spdx.json for binary packages) never have that data. + """ + if os.path.isdir(path): + build_pattern = os.path.join(path, "**", "builds", "build-*.spdx.json") + for build_path in sorted(glob.glob(build_pattern, recursive=True)): + yield from load_jsonld_graph(build_path) + else: + yield from load_jsonld_graph(path) + + +def _to_row(name, release_date): + return { + "recipe": name, + "release_date": release_date or "", + } + + +def build_report(path): + """ + Build the report: [{"recipe": ..., "release_date": ...}], from either a + DEPLOY_DIR_SPDX tree or a single merged image SBOM file. + """ + release_dates = extract_release_dates(iter_spdx_elements(path)) + return [_to_row(name, release_dates[name]) for name in sorted(release_dates)] + + +def drop_redundant_native(report): + """Drop foo-native rows when foo is also present in the report.""" + names = {r["recipe"] for r in report} + return [ + r + for r in report + if not (r["recipe"].endswith("-native") and r["recipe"][:-len("-native")] in names) + ] + + +def filter_rows(report, sort_by_date=False): + rows = list(report) + if sort_by_date: + rows.sort(key=lambda r: (not r["release_date"], r["release_date"], r["recipe"])) + return rows + + +def print_table(rows): + if not rows: + print("No matching recipes found.") + return + + name_width = max(len("recipe"), *(len(r["recipe"]) for r in rows)) + print(f"{'recipe':<{name_width}} release_date") + for r in rows: + print(f"{r['recipe']:<{name_width}} {r['release_date']}") + + +def write_csv(rows, path): + with open(path, "w", newline="", encoding="utf-8") as f: + writer = csv.writer(f) + writer.writerow(["recipe", "release_date"]) + for r in rows: + writer.writerow([r["recipe"], r["release_date"]]) + + +def main(): + parser = argparse.ArgumentParser( + description="Report recipe release dates from SPDX 3.0.1 output" + ) + parser.add_argument( + "path", + help="Path to DEPLOY_DIR_SPDX (e.g. tmp/deploy/spdx/3.0.1) or to a " + "single merged image SBOM file (e.g. " + "tmp/deploy/images/<machine>/<image>.rootfs.spdx.json)", + ) + parser.add_argument( + "--csv", + help="Write the report to a CSV file instead of only printing a table", + ) + parser.add_argument( + "--sort-by-date", + action="store_true", + help="Sort output by release date instead of recipe name (missing dates last)", + ) + args = parser.parse_args() + + logging.basicConfig(format="[%(filename)s:%(lineno)d] %(message)s", level=logging.INFO) + + if not os.path.isdir(args.path) and not os.path.isfile(args.path): + parser.error(f"{args.path} does not exist") + + report = build_report(args.path) + report = drop_redundant_native(report) + + total = len(report) + found = sum(1 for r in report if r["release_date"]) + logging.info("Recipes with SPDX package data: %d", total) + logging.info("Recipes with a release date: %d", found) + logging.info("Recipes missing a release date: %d", total - found) + + rows = filter_rows(report, args.sort_by_date) + print_table(rows) + + if args.csv: + write_csv(rows, args.csv) + logging.info("CSV report written to %s", args.csv) + + return 0 + + +if __name__ == "__main__": + sys.exit(main())
Quoted 24 lines · Daniel Turull via lists.openembedded.org
From: Daniel Turull <[email protected]>
We have a requirements to include release time of open source components in the SBOM. There is a field specific for that in spdx 3 spec.
https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Properties/releaseTime/
This can also be used to evaluate how old are some of the core components and decide if they need replacement.
The previous 2 versions did not have cover letter.
In v4 I kept the simpler logic to just check for the epoch date, even if we have multiple sources with different release dates. It was getting complicated and adding more code for git and tarfiles. I can do a follow up patch after this simpler version gets in, so we can fine tune it.
Tested with oe-selftest -r spdx
Daniel Turull (3):
classes/base: exclude__CACHED_SOURCE_DATE_EPOCHfrom task hash create-spdx-3.0: record component release date in SPDX outputscripts/contrib: addspdx-release-date-report.py
Quoted 6 lines
meta/classes-global/base.bbclass | 4 + meta/lib/oe/spdx30_tasks.py | 22 +++ meta/lib/oeqa/selftest/cases/spdx.py | 41 +++++ scripts/contrib/spdx-release-date-report.py | 191 ++++++++++++++++++++ 4 files changed, 258 insertions(+) create mode 100755 scripts/contrib/spdx-release-date-report.py
Hi Daniel,
It looks like one of the added tests is failing here:
2026-10-03 15:26:28,760 - oe-selftest - INFO - spdx.SPDX30Check.test_release_date_source_date_epoch (subunit.RemotedTestCase)
2026-10-03 15:26:28,761 - oe-selftest - INFO - ... FAIL
...
2026-10-03 15:26:28,761 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 473, in test_release_date_source_date_epoch
self.assertEqual(pkg.releaseTime, expected)
File "/usr/lib/python3.11/unittest/case.py", line 873, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/lib/python3.11/unittest/case.py", line 866, in _baseAssertEqual
raise self.failureException(msg)AssertionError: datetime.datetime(2023, 7, 18, 6, 55, 23, tzinfo=datetime.timezone.utc) != datetime.datetime(2011, 4, 5, 23, 0, tzinfo=datetime.timezone.utc)
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4997
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 46 lines · Mathieu Dubois-Briand
Quoted 24 lines · Daniel Turull via lists.openembedded.org
From: Daniel Turull <[email protected]>
We have a requirements to include release time of open source components in the SBOM. There is a field specific for that in spdx 3 spec.
https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Properties/releaseTime/
This can also be used to evaluate how old are some of the core components and decide if they need replacement.
The previous 2 versions did not have cover letter.
In v4 I kept the simpler logic to just check for the epoch date, even if we have multiple sources with different release dates. It was getting complicated and adding more code for git and tarfiles. I can do a follow up patch after this simpler version gets in, so we can fine tune it.
Tested with oe-selftest -r spdx
Daniel Turull (3):
classes/base: exclude__CACHED_SOURCE_DATE_EPOCHfrom task hash create-spdx-3.0: record component release date in SPDX outputscripts/contrib: addspdx-release-date-report.pyQuoted 6 lines
meta/classes-global/base.bbclass | 4 + meta/lib/oe/spdx30_tasks.py | 22 +++ meta/lib/oeqa/selftest/cases/spdx.py | 41 +++++ scripts/contrib/spdx-release-date-report.py | 191 ++++++++++++++++++++ 4 files changed, 258 insertions(+) create mode 100755 scripts/contrib/spdx-release-date-report.pyHi Daniel,
It looks like one of the added tests is failing here:
2026-10-03 15:26:28,760 - oe-selftest - INFO - spdx.SPDX30Check.test_release_date_source_date_epoch (subunit.RemotedTestCase) 2026-10-03 15:26:28,761 - oe-selftest - INFO - ... FAIL ... 2026-10-03 15:26:28,761 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/spdx.py", line 473, in test_release_date_source_date_epoch self.assertEqual(pkg.releaseTime, expected) File "/usr/lib/python3.11/unittest/case.py", line 873, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib/python3.11/unittest/case.py", line 866, in _baseAssertEqual raise self.failureException(msg)
Quoted 4 lines
AssertionError:
datetime.datetime(2023, 7, 18, 6, 55, 23, tzinfo=datetime.timezone.utc)!=datetime.datetime(2011, 4, 5, 23, 0, tzinfo=datetime.timezone.utc)https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4997
I'll need to investigate further.
I did a rerun and it works on my end. Maybe I choose the wrong recipe for the selftest. I'm suspecting some sstate issue. I'll keep you posted if I find the issue.
bitbake tar -c cleanall && oe-selftest -r spdx.SPDX30Check.test_release_date_source_date_epoch
Loading cache: 100% |############################################################################### ####################################| Time: 0:00:00
Loaded 0 entries from dependency cache.
Parsing recipes: 100% |############################################################################### ##################################| Time: 0:00:10
Parsing of 1044 .bb files complete (0 cached, 1044 parsed). 2099 targets, 58 skipped, 0 masked, 0 errors.
Removing 2 recipes from the allarch sysroot: 100% |############################################################################### ######| Time: 0:00:00
Removing 32 recipes from the qemux86_64 sysroot: 100% |############################################################################### ##| Time: 0:00:16
Removing 29 recipes from the x86-64-v3 sysroot: 100% |############################################################################### ###| Time: 0:00:01
Removing 18 recipes from the x86_64 sysroot: 100% |############################################################################### ######| Time: 0:00:08
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "2.19.1" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "universal" TARGET_SYS = "x86_64-oe-linux" MACHINE = "qemux86-64" SDKMACHINE = "x86_64" DISTRO = "nodistro" DISTRO_VERSION = "nodistro.0" TUNE_FEATURES = "m64 x86-64-v3"
meta
meta-selftest = "spdx-release:86f3f8bd5e3832411a8d4abb5661f77beb123511"
Sstate summary: Wanted 0 Local 0 Mirrors 0 Missed 0 Current 0 (0% match, 0% complete)
Initialising tasks: 100% |############################################################################### ###############################| Time: 0:00:00
NOTE: No setscene tasks
NOTE: Executing Tasks (spdx.SPDX30Check.test_release_date_source_date_epoch)
2026-10-05 07:05:09,855 - oe-selftest - INFO - ... ok 2026-10-05 07:05:25,192 - oe-selftest - INFO - --------------------------------- ------------------------------------- 2026-10-05 07:05:25,192 - oe-selftest - INFO - Ran 1 test in 109.248s 2026-10-05 07:05:25,192 - oe-selftest - INFO - OK 2026-10-05 07:05:28,492 - oe-selftest - INFO - RESULTS: 2026-10-05 07:05:28,492 - oe-selftest - INFO - RESULTS - spdx.SPDX30Check.test_release_date_source_date_epoch: PASSED (93.60s) 2026-10-05 07:05:28,492 - oe-selftest - INFO - SUMMARY: 2026-10-05 07:05:28,493 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 109.249s 2026-10-05 07:05:28,493 - oe-selftest - INFO - oe-selftest - OK - All required
tests passed (successes=1, skipped=0, failures=0, errors=0)
Best regards,
Daniel
Quoted 3 lines
Thanks,
Mathieu