spdx30: Fix two bugs in the source download enrichment
cover spdx30: Fix two bugs in the source download enrichment
Two fixes for the source download enrichment added in 9f1825e74d ("spdx30: Add Git version and PURL to source downloads"):
- _enrich_source_package() and _generate_git_purl() start with an underscore, so bitbake leaves them out of the do_create_spdx signature, and SPDX_GIT_PURL_MAPPINGS with them. Changing the mappings restored the previous SPDX from sstate.
- gitsm downloads got no version and no PURL, because only fd.type "git" was handled.
Patch 1 changes the do_create_spdx signature of every recipe once.
Patch 3 adds one selftest per fix. Each fails on master without its fix. With the series, all SPDX30Check selftests pass.
Stefano Tondo (3):
spdx30: Make source download enrichment part of the task signature
spdx30: Enrich gitsm source downloads like git ones oeqa/selftest: Add tests for source download enrichment
meta/lib/oe/spdx30_tasks.py | 10 +++--- meta/lib/oeqa/selftest/cases/spdx.py | 47 ++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+), 5 deletions(-)
Signature · 1 line
2.56.0
PATCH 1/3 spdx30: Make source download enrichment part of the task signature
From: Stefano Tondo <[email protected]>
bitbake does not track module functions whose names start with an underscore when it computes task signatures. _enrich_source_package() and _generate_git_purl() were therefore not part of the do_create_spdx signature, and neither was SPDX_GIT_PURL_MAPPINGS, which _generate_git_purl() reads. Changing the mappings or this code did not rerun do_create_spdx, and the previous SPDX was restored from sstate.
Rename both functions so that bitbake tracks them. This changes the do_create_spdx signature of every recipe once.
--- meta/lib/oe/spdx30_tasks.py | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
meta/lib/oe/spdx30_tasks.py +4 -4
@@ -358,7 +358,7 @@ def collect_dep_sources(dep_objsets, dest): index_sources_by_hash(e.to, dest) -def _generate_git_purl(d, download_location, srcrev): +def generate_git_purl(d, download_location, srcrev): """Generate a Package URL for a Git source from its download location. Parses the Git URL to identify the hosting service and generates the
@@ -416,7 +416,7 @@ def _generate_git_purl(d, download_location, srcrev): return None -def _enrich_source_package(d, dl, fd, file_name, primary_purpose): +def enrich_source_package(d, dl, fd, file_name, primary_purpose): """Enrich a Git source download package with version, PURL, and external refs. For Git sources, extracts the full SHA-1 from SRCREV as the version,
@@ -435,7 +435,7 @@ def _enrich_source_package(d, dl, fd, file_name, primary_purpose): # Generate PURL for Git hosting services download_location = getattr(dl, "software_downloadLocation", None) if version and download_location: - purl = _generate_git_purl(d, download_location, version) + purl = generate_git_purl(d, download_location, version) if version: dl.software_packageVersion = version
@@ -524,7 +524,7 @@ def add_download_files(d, objset): ) ) - _enrich_source_package(d, dl, fd, file_name, primary_purpose) + enrich_source_package(d, dl, fd, file_name, primary_purpose) if release_date is not None: dl.releaseTime = release_date
Signature · 1 line
2.56.0
PATCH 2/3 spdx30: Enrich gitsm source downloads like git ones
In reply to Stefano TondoFrom: Stefano Tondo <[email protected]>
The source download enrichment only handled fd.type "git", so gitsm downloads got the VCS external reference but neither the SRCREV as version nor the PURL. The gitsm fetcher is a git fetcher with submodule support and carries the same revision, so treat it the same.
--- meta/lib/oe/spdx30_tasks.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/lib/oe/spdx30_tasks.py +1 -1
@@ -426,7 +426,7 @@ def enrich_source_package(d, dl, fd, file_name, primary_purpose): version = None purl = None - if fd.type == "git": + if fd.type in ("git", "gitsm"): # Use full SHA-1 from fd.revision srcrev = getattr(fd, "revision", None) if srcrev and srcrev not in {"${AUTOREV}", "AUTOINC", "INVALID"}:
Signature · 1 line
2.56.0
PATCH 3/3 oeqa/selftest: Add tests for source download enrichment
In reply to Stefano TondoFrom: Stefano Tondo <[email protected]>
test_gitsm_source_version builds git-submodule-test, which uses gitsm, and checks that its source download carries the SRCREV as version.
test_git_purl_mappings_rerun_create_spdx builds fastfloat twice, first with the default mapping and then with github.com mapped to another PURL type, and checks that the second SPDX carries the new PURL. Before the enrichment became part of the task signature, the second build restored the first SPDX from sstate.
--- meta/lib/oeqa/selftest/cases/spdx.py | 47 ++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+)
meta/lib/oeqa/selftest/cases/spdx.py +47 -0
@@ -445,6 +445,53 @@ class SPDX30Check(SPDX3CheckBase, OESelftestTestCase): f"Version '{version}' for package '{name}' should contain digits" ) + def test_gitsm_source_version(self): + """Test that gitsm source downloads are enriched like git ones. + + git-submodule-test is fetched with gitsm://, so its source download + must carry the SRCREV as version. + """ + objset = self.check_recipe_spdx( + "git-submodule-test", + "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-git-submodule-test.spdx.json", + ) + + srcrev = get_bb_var("SRCREV", "git-submodule-test") + for pkg in objset.foreach_type(oe.spdx30.software_Package): + location = pkg.software_downloadLocation or "" + if location.startswith("git+https://git.yoctoproject.org/git-submodule-test"): + self.assertEqual(pkg.software_packageVersion, srcrev) + break + else: + self.fail("git-submodule-test gitsm source download not found") + + def test_git_purl_mappings_rerun_create_spdx(self): + """Test that SPDX_GIT_PURL_MAPPINGS is part of the do_create_spdx signature. + + fastfloat is fetched from github.com. Build it with the default + mapping, then map github.com to another PURL type: do_create_spdx + must run again instead of restoring the first result from sstate. + """ + spdx_path = "{DEPLOY_DIR_SPDX}/{SSTATE_PKGARCH}/builds/build-fastfloat.spdx.json" + location_prefix = "git+https://github.com/fastfloat/fast_float" + srcrev = get_bb_var("SRCREV", "fastfloat") + + def source_purl(objset): + for pkg in objset.foreach_type(oe.spdx30.software_Package): + if (pkg.software_downloadLocation or "").startswith(location_prefix): + return pkg.software_packageUrl + self.fail("fastfloat git source download not found") + + objset = self.check_recipe_spdx("fastfloat", spdx_path) + self.assertEqual(source_purl(objset), f"pkg:github/fastfloat/fast_float@{srcrev}") + + objset = self.check_recipe_spdx( + "fastfloat", + spdx_path, + extraconf='SPDX_GIT_PURL_MAPPINGS = "github.com:pkg:selftest"\n', + ) + self.assertEqual(source_purl(objset), f"pkg:selftest/fastfloat/fast_float@{srcrev}") + def test_release_date_source_date_epoch(self): """releaseTime should be derived from SDE_FILE via epochfile_read().
Signature · 1 line
2.56.0