spdx30: name, version and purl for dependency downloads

Hi all,

I would like your opinion on a direction for the SPDX output of recipes with language dependencies. I have working patches for it, described below, but I would rather agree on the approach before sending them.

Yocto already knows each language dependency at fetch time for the recipes that list them in SRC_URI (crate://, gomod://, gomodgit://, and npmsw:// through its shrinkwrap file). The SPDX output loses most of
that: the downloads are recorded, but without a name, version or package URL that a scanner can use, and dependencies fetched implicitly (git submodules, shrinkwrap dependencies) are not recorded at all.

Measured on current master (oe-core 3a3eef7afb, bitbake 0f33b68cb, meta-openembedded 33f74f5dc5), source downloads in the build SPDX of each recipe, today and with the patches:

                  today                   with the patches
  python3-bcrypt  42 crates, none with    all 42 with name,
                  version or purl         version and purl
  crucible        36 Go modules, none     all 36 with name,
                  with version or purl    version and purl
  p11-kit         gitsm, submodule        submodule recorded with
                  pkcs11-json missing     its own revision
  webserver-oob   2 downloads, the 70     72 downloads, 71 with
                  shrinkwrap packages     a purl

missing

webserver-oob is from TI's meta-tisdk, built on qemux86-64 without its am335x and meta-ti runtime dependencies and without one git.ti.com source that failed to clone, since neither oe-core nor meta-openembedded has an npmsw recipe. crucible also fetches 67 go.mod-only entries (;mod=1), which are not compiled in and stay without an identity.

Today a crate shows up as name "cipher-0.4.4.crate" with download location https://static.crates.io/crates/cipher/0.4.4/download, and a Go module as gomod://github.com/felixge/httpsnoop, with the version only in the file name "[email protected]". Consumers end up parsing URLs and file names per ecosystem to recover what the fetcher already knew.

I am aware of two earlier threads and would like to build on them, not reopen them:

- Joshua's "lib: spdx30_tasks: Use Fetch.expanded_urldata()" (March 2025), which stalled on an intermittent edk2-firmware gitsm failure on the autobuilder. I think I found the cause: the patch passes each expanded URL to oe.patch.patch_path(), which calls Fetch.localpath(url). For a URL that is not in SRC_URI, that creates a new FetchData with the recipe datastore, where the SRCREV_<submodule> that the gitsm fetcher sets in its own datastore copy does not exist. A recipe with only named SRCREVs, like edk2-firmware, then fails with "Please set a valid SRCREV". It reproduces every time with meta-selftest's git-submodule-test given a named SRCREV. My guess for the intermittency is that gitsm only lists the submodules when the clone is in DL_DIR.
- Stefan's "Concept for tightly coupled package manager" RFC (December 2024). My reading of that thread is that explicit SRC_URI lists generated in layers (go-mod-update-modules, cargo-update-recipe-crates) are the preferred model and that fetchers should stay thin. This proposal assumes that model and does not add any dependency resolution to bitbake.

The patches have two parts.

1. Record implicit downloads

add_download_files() also records what implicit_urldata() returns for each SRC_URI entry, but keeps the patch check on the SRC_URI entries, so implicit downloads are never set up again with the recipe datastore. The npmsw fetcher gets an implicit_urldata(), since it already builds the list of URLs internally (ud.proxy). The gitsm fetcher warns when the repository is not in DL_DIR, instead of silently returning no submodules.

2. Let the fetcher state the identity of what it downloads

FetchData gets a package_identity attribute, None by default and set in urldata_init() by the fetchers that know it:

     crate     ("cargo", "cipher", "0.4.4", {})
     gomod     ("golang", "github.com/felixge/httpsnoop", "v1.0.4", {})
     gomodgit  same as gomod
     npm       ("npm", "@scope/name", "1.2.3", {})
     npmsw     the same for each registry dependency

The last element holds purl qualifiers, so a crate or npm package from another registry gets a repository_url. go.mod-only entries get None. create-spdx-3.0 sets the name, software_packageVersion and software_packageUrl from it and builds the purl in oe-core, so bitbake does not need to know the purl specification.

I first had a FetchMethod method in mind, but npmsw fetches its dependencies through plain https URLs, so it has to attach the identity to each of them, and an attribute is enough for that. No URL syntax changes, unlike the dn=/dv= parameters in Stefan's series.

Both parts are small, about 35 lines in bitbake and 40 in oe-core plus tests. I tested them on master with new bitbake fetcher unit tests and new oe-selftest cases (submodules, including the named SRCREV case, and crate identity), which fail without the patches, and with the existing fetcher tests and the SPDX selftests closest to the change. Two bugs I found in the existing Git handling while doing this (gitsm skipped, changes not reaching the task signature) are fixed in a separate series, which the oe-core part builds on:
https://lore.kernel.org/openembedded-core/[email protected]/

Questions:

a) Is the fetcher the right place for the identity, or would you prefer oe-core to derive it from each fetcher's FetchData? The latter avoids touching bitbake, but for npmsw it means reading the internals of its proxy fetcher.

b) Should bitbake keep returning the parts and leave the purl to oe-core, as above, or return a purl directly?

c) For ecosystems without a fetcher (.NET/NuGet, Maven), would a fetcher plus an update class following go-mod-update-modules be welcome, or is that better kept in a layer?

d) Dependency scope. Identity alone does not give a correct SBOM: a lock file lists what may be needed on any platform. python3-bcrypt fetches wasi, wasip2 and wit-bindgen (WebAssembly only) and r-efi (UEFI), and with the patches they show up with a clean purl as if they were part of the product. As a follow-up I tried running cargo metadata --filter-platform ${RUST_HOST_SYS} in cargo.bbclass:
of the 42 crates, 25 are runtime, 12 only run at build time (build scripts, proc-macros) and 5 are not used on the target. I recorded them as LifecycleScopedRelationship dependsOn with scope runtime or build, and an annotation on the unused ones. Go (go list -deps) and npm (dev, optional, os and cpu in the shrinkwrap) could work the same way. Does that split sound right: declared scope from the fetcher where the lock file has it, observed scope from the language classes? And is there a preferred SPDX representation for "fetched but not built"?

Thanks,
Stefano

I'm afraid I am missing context, which is hardly surprising because I did no spdx work until now. What are the language dependencies? Does something require or recommend them? Is there a normative document that defines them? The email talks about how the implementation works, but says nothing about what the problem is.

Alex

Hi Alex,

You are right: I jumped straight into the how and never explained the why. Let me start from the problem.

Take crucible from meta-oe. It is a Go program, and besides its own code it compiles in 36 Go modules fetched with gomod://, one of them golang.org/x/net v0.39.0. Now imagine the next advisory for golang.org/x/net comes out, and whoever ships a device with crucible on it is asked: "Is your product affected?"

The SPDX that Yocto writes for the image should answer that, but it can't. The module is in there, but only as a downloaded file named "[email protected]" with the location gomod://golang.org/x/net: no version field and no package URL, so no scanner recognises it. cve-check does not see it either, because it only knows the CPEs of recipes. The answer ends up being someone reading file names by hand, for every advisory and every image.

That is what I meant by language dependencies: the packages of a language's own package manager that a recipe compiles in without having a recipe of their own. Rust crates (crate://), Go modules (gomod://) and npm packages (npmsw://) are the cases Yocto can fetch today. For the first two the problem is the one above: present, but not identifiable. Git submodules and the dependencies listed in an npm shrinkwrap file are worse off: they are not recorded at all.

Why this matters now, with the documents that ask for it:

- The EU Cyber Resilience Act (Regulation (EU) 2024/2847), Annex I, Part II, point 1, requires manufacturers to identify and document the components of their products, including by drawing up an SBOM in a commonly used, machine-readable format covering at the very least the top-level dependencies. It applies from December 2027 to products sold in the EU, so many Yocto users will have to deliver one.

- BSI TR-03183-2, the German technical guideline that spells out what such an SBOM contains, asks for the dependencies of everything that is delivered (section 5.1), each with a name, a version and an identifier such as a purl or CPE (section 5.2).

- The "2026 Minimum Elements for a Software Bill of Materials", published in July by CISA together with BSI, ANSSI and other agencies, asks for transitive dependencies and an identifier per component, and gives the example above as its reason: a recipient should be able to conclude that a new vulnerability does not affect them when the SBOM does not list the component.

Yocto already knows all of this at fetch time; the RFC is only about writing it into the SPDX it already produces, without changing how anything is fetched. I hope that makes the context clearer, and I am happy to go into more detail on any of it.

Thanks,
Stefano

Hello Stefano,

thanks for the detailed response. I need to look at code to give specific guidance, but I want to make a couple general points:

- do not make fetchers any more complex than they already are. The reason is that few people understand or have an interest in that code, even fewer are able to meaningfully review contributions to it. And still fewer actually show up to patch review meetings where patches are looked at. Rather try to do as much as possible in oe-core through a clear API.

- gitsm and npmsw are historical mistakes, however 'convenient' they may be. Doing internal magic just resulted in a ton of corner cases, maintenance headaches and (in case of npm) bitrot and lack of integrity protection that caused the whole thing to be disabled until that disabling forced someone to fix the issues. I would want to replace them with flat, explicit implementations.

Alex

We should revisit that and fix it as I think it exposes general bugs.

That said, I am thinking we should change the fetcher so all the submodule urls are included in SRC_URI in future and expanded_urldata() goes away. We need to update the other backends to match. I have no knowledge of npm so that one will be hard for me.

Once we do that, your problem becomes easier too I think. It also means I can then fix various mirror issues I'd like to change.

Yes, although just to be clear, I think bitbake needs to verify that the dependencies are all listed.

I think that the fetcher is the right place to do this as I would be very much against poking fetcher internals from OE-Core.

We need to declare and document the stage at which this is expected to work (i.e. before/after fetch, before/after unpack).

It should probably return structured data including the purl from some new API.

If the fetcher can conform to our APIs and roughly follow how rust/crates are working today, I think those would be fine, particularly if we have good tests too.

It sounds possibly right but I'd defer to Joshua on that one.

Cheers,

Richard

Hi Richard, hi Alex,

Thank you both for the feedback. Your replies point in the same direction, so I answer them together here.

I think these two points fit together. The fetchers already parse the name and the version of each dependency for their own use. The new API only returns that data as a documented structure, with the purl. It does not change the fetch or the unpack. oe-core then writes the SPDX from it and does not read any fetcher internals.

About the stage: crate, gomod and npmsw know the identity when they create the FetchData. That is before fetch, and it needs no network access. npmsw only reads its shrinkwrap file, which is a local file in the layer. gitsm is the exception, because it can list the submodules only from the clone. When the submodules are in SRC_URI, that exception goes away. So I propose to document "before fetch".

I agree, and I would like to help with npm. My idea is to follow cargo-update-recipe-crates and go-mod-update-modules: an update class writes one SRC_URI entry with a checksum for each package in the shrinkwrap file. Then bitbake checks that the list matches the shrinkwrap file, as you said in your reply. Is
that what you have in mind? If yes, I can prepare patches.

This also makes my RFC smaller. Its first part records implicit downloads. With flat SRC_URI lists, there are no implicit downloads left to record.

I can investigate this too. I have a reproducer that fails every time:
meta-selftest's git-submodule-test with a named SRCREV. The bug I found is in Fetch.localpath() for a URL that is not in SRC_URI. It creates a new FetchData with the recipe datastore, which does not have the SRCREV that gitsm set for the submodule. Is this the bug you mean, or do you know of others in this area? I can start from any cases you have.

Thanks, that is clear. I will come back to new fetchers only after we agree on the API.

Joshua, Richard deferred question d) to you. In short: a lock file lists packages for all platforms, so the fetcher downloads some dependencies that are never built for the target. I proposed to take the declared scope from the lock file in the fetcher, and the observed scope from the language classes (for example cargo metadata --filter-platform). Does that split work for you? And how do you want "fetched but not built" to look in SPDX 3.0?

For the next step: crate and gomod already list every dependency in SRC_URI, so the identity API can work for them now. Do you want me to send patches for these two, with tests, and add npm after the flattening? Or do you prefer that I wait for Joshua's answer first?

Thanks,
Stefano

I'm not sure if we can. The current support we have is that the things listed in SRC_URI are "build inputs" to the build, which is accurate (they are inputs, even if they aren't used). We don't currently have a mechanism to link the outputs of the build back to the SRC_URI inputs.

Perhaps, the better option would be to propagate the SRC_URI purls to the output packages, but apply some sort of (optional) filter that can remove a SRC_URI if it is not actually present in the output. That way, if we have positive confirmation that some input is not present in the output, we can remove its PURL from the output package (but, barring that, we include it).

Hi Joshua,

Thanks, that model makes sense to me. A purl stays unless there is proof that the dependency is not in the output. For vulnerability checks, that is the safer side.

The cargo prototype already works as that kind of filter. cargo metadata --filter-platform resolves the dependencies for the target. For python3-bcrypt, it gives the proof for 17 of the 42
crates: 5 are not used on the target, and 12 only run at build time (build scripts and proc-macros). None of them end up in the binary, so they lose their purl, and the 25 runtime crates keep it. For Go, go version -m on each binary lists the modules compiled into it, so it can give the same proof.

One question about the output side: how do you want the purls on the output package? I see two options:

1. Add them to the externalIdentifier list of the output package, the way set_purls() handles SPDX_PACKAGE_URLS today.
2. Link the output package to the source download elements, which carry the purls, with a relationship such as contains.

Option 1 is simpler. But an external identifier says what the package itself is, so a scanner can take python3-bcrypt for one of its crates. Option 2 keeps each dependency as its own element. I prefer option 2, unless you see a reason against it.

Thanks,
Stefano