spdx: Add software file externalRef support
cover spdx: Add software file externalRef support
Hi all,
When starting to test SPDX 3.0 in our projects, we noticed that it would be necessary to have more information for files fetched via the 'file://' protocol, such as the full path of the file or a URL with git information.
Our first idea was to use 'downloadLocation', but what I understand is that this is a package property, and files fetched from the layer are 'software_File' type. Looking at the SPDX spec, it appears we could use the 'ExternalRef' for this purpose.
The idea is to have two options to add this information: one to add the full path of a file, and another to add the git information 'git+https://host/repo@commit#path/to/file'. The information is added as an 'externalRef' and can be configured using these types:
https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Vocabularies/ExternalRefType/
When using the 'path' option, something like this is added:
```
"externalRef": [
{
"type": "ExternalRef",
"externalRefType": "sourceArtifact",
"locator": [
"/home/user/src/openembedded-core/meta/recipes-core/busybox/files/syslog"
]
}
],
```This option is non-reproducible, if the build path changes, the SPDX will be different.
And with the 'git' option:
```
"externalRef": [
{
"type": "ExternalRef",
"externalRefType": "sourceArtifact",
"locator": [
"git+https://git.openembedded.org/openembedded-core@ac5d9579a0db63b54bbebb5015de2ae860a462bf#meta/recipes-core/busybox/files/syslog"
]
}
],
```The implementation is not completely finished, but since there is already a thread on this subject,
https://lists.openembedded.org/g/openembedded-core/topic/thoughts_on_spdx_for_files/116135395,
I wanted to share my work and get opinions on how to improve this implementation.
My questions are:
Is the 'externalRef' the right way to add the information in the spdx file?
I'm using the 'choices' type, but this only works when inheriting typecheck.bbclass, and this bbclass is not inherited when using OE-Core with 'nodistro'. Can this 'choices' type be used here?
I still need to find a way to cache Git layer information to avoid calling the 'oe.buildcfg' function every time. Maybe it would be possible to use something like this:
https://git.openembedded.org/openembedded-core/tree/meta/classes/metadata_scm.bbclass
to get information at parsing time. However, this information is only needed when using SPDX_FILE_LOCATION with the git option, and for all layers. Any idea here?
For the git option, we need to get a git remote, but there can be more than one remote per layer, so we need a way to configure these remotes. In this first implementation, I'm assuming that all layers use the same remote, and the remote name can be configured, which fits our current use case.
Should I add a variable like 'SPDX_FILE_LOCATION_GIT_REMOTE_<layername> = "remote_name"' to set a specific remote for each layer? Would setting the git remote be sufficient to cover most cases?
Any feedback or suggestions would be appreciated.
Best regards,
Fabio
Fabio Berton (1):
spdx: Add software file externalRef support
meta/classes/create-spdx-3.0.bbclass | 24 +++++++ meta/lib/oe/sbom30.py | 14 ++++- meta/lib/oe/spdx30_tasks.py | 93 ++++++++++++++++++++++++++++ 3 files changed, 130 insertions(+), 1 deletion(-)
Signature · 1 line
2.51.1
RFC PATCH 1/1 spdx: Add software file externalRef support
From: Fabio Berton <[email protected]>
Add support for including external references in SPDX 3.0 file elements, allowing files fetched via file:// protocol to reference their source location via absolute path or git URL with commit hash.
Change add the following variables:
- SPDX_FILE_LOCATION variable with three options:
* 'none' (default) - disables external references
* 'path' - adds absolute file path as external reference
* 'git' - generates git URL with commit hash in format git+https://host/repo@commit#path/to/file
- SPDX_FILE_LOCATION_REF_TYPE control the external reference type (defaults to 'sourceArtifact')
- SPDX_FILE_LOCATION_GIT_REMOTE to specify which git remote to use when constructing git URLs (defaults to 'origin')
The git URL generation requires files to be in a git repository with a configured remote.
--- meta/classes/create-spdx-3.0.bbclass | 24 +++++++ meta/lib/oe/sbom30.py | 14 ++++- meta/lib/oe/spdx30_tasks.py | 93 ++++++++++++++++++++++++++++ 3 files changed, 130 insertions(+), 1 deletion(-)
meta/classes/create-spdx-3.0.bbclass +24 -0
@@ -122,6 +122,30 @@ SPDX_PACKAGE_URL[doc] = "Provides a place for the SPDX data creator to record \ the package URL string (in accordance with the Package URL specification) for \ a software Package." +SPDX_FILE_LOCATION ?= "none" +SPDX_FILE_LOCATION[type] = "choice" +SPDX_FILE_LOCATION[choices] = "none path git" +# TODO: Type validation only works when inheriting typecheck bbclass + +# Available options are listed here: +# https://spdx.github.io/spdx-spec/v3.0.1/model/Core/Vocabularies/ExternalRefType +SPDX_FILE_LOCATION_REF_TYPE ?= "sourceArtifact" + +# Use 'origin' remote as default, this option is only valid when +# SPDX_FILE_LOCATION is set to 'git' +SPDX_FILE_LOCATION_GIT_REMOTE ?= "origin" + +SPDX_FILE_LOCATION[doc] = "Controls whether and how to add external references \ + to source files in the SPDX SBOM. Valid options are: \ + 'none' (default) - disables external references \ + 'path' - adds the absolute file path as an external reference \ + 'git' - generates Git URLs with commit hash in the format \ + 'git+https://host/repo@commit#path/to/file' using the configured remote \ + (see SPDX_FILE_LOCATION_GIT_REMOTE). When using 'git', files must be in a \ + git repository with a configured remote. The external reference type is \ + controlled by SPDX_FILE_LOCATION_REF_TYPE (default 'sourceArtifact'). \ + NOTE: Using 'path' may result in non-reproducible SPDX output." + IMAGE_CLASSES:append = " create-spdx-image-3.0" SDK_CLASSES += "create-spdx-sdk-3.0"
meta/lib/oe/sbom30.py +13 -1
@@ -4,6 +4,7 @@ # SPDX-License-Identifier: GPL-2.0-only # +from collections import namedtuple from pathlib import Path import oe.spdx30
@@ -15,6 +16,9 @@ import os import oe.spdx_common from datetime import datetime, timezone +# Named tuple for external reference information +ExternalRef = namedtuple("ExternalRef", ["ref_type", "locator"]) + OE_SPDX_BASE = "https://rdf.openembedded.org/spdx/3.0/" VEX_VERSION = "1.0.0"
@@ -600,7 +604,7 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): ) spdx_file.extension.append(OELicenseScannedExtension()) - def new_file(self, _id, name, path, *, purposes=[]): + def new_file(self, _id, name, path, *, purposes=[], external_ref=None): sha256_hash = bb.utils.sha256_file(path) for f in self.by_sha256_hash.get(sha256_hash, []):
@@ -641,6 +645,14 @@ class ObjectSet(oe.spdx30.SHACLObjectSet): spdx_file.software_primaryPurpose = purposes[0] spdx_file.software_additionalPurpose = purposes[1:] + if external_ref: + spdx_file.externalRef.append( + oe.spdx30.ExternalRef( + externalRefType=external_ref.ref_type, + locator=[external_ref.locator], + ) + ) + spdx_file.verifiedUsing.append( oe.spdx30.Hash( algorithm=oe.spdx30.HashAlgorithm.sha256,
meta/lib/oe/spdx30_tasks.py +93 -0
122 lines not on this page · raw message
Signature · 1 line
2.51.1
Hello Fabio,
thanks for your comments and patch!
Quoted 4 lines · Fabio Berton
Our first idea was to use 'downloadLocation', but what I understand is that this is a package property, and files fetched from the layer are '
software_File' type. Looking at the SPDX spec, it appears we could use the 'ExternalRef' for this purpose.
I'm not to familiar with the SPDX spec yet, but adding individual files entries as ExternalRef instead of downloadLocation to a recipes spdx sounds reasonable.
I think in the long term adding a SPDXRef-Layer-xyz entry per layer with a downloadLocation pointing to the subpath of the layer inside a git repo. I'm not quite shure if it would be possible to formulate a dependency on a file contained within a different SPDXRef, e.g.
``` SPDXRef-Layer-xyz:recipes-core/base-files/base-files/fstab ```
or if we'd have to create a SPDXRef Item for each file within a layer in order to reference it properly. That would make it even more verbose.
The approach of having a layer as an independent SPDXRef would mean getting the git revision etc. for that layer would run only once per build and not per file:// entry in SRC_URI.
Quoted 3 lines
The idea is to have two options to add this information: one to add the full path of a file, and another to add the git information
IMO the full path to the file is unneeded information, if the file is solely available locally a NOASSERTION would be appropriate.
Quoted 4 lines
Should I add a variable like 'SPDX_FILE_LOCATION_GIT_REMOTE_<layername> = "
remote_name"' to set a specific remote for each layer? Would setting the git remote be sufficient to cover most cases?
In my experimentation I removed the per-layer setting again because tracking the vardeps for the do_create_spdx get's more complicated with per-layer variables.
Quoted 1 line
Sincerely
Daniel Wagenknecht
Quoted 18 lines · Daniel Wagenknecht
Hello Fabio,
thanks for your comments and patch!
Quoted 4 lines · Fabio Berton
Our first idea was to use 'downloadLocation', but what I understand is that this is a package property, and files fetched from the layer are '
software_File' type. Looking at the SPDX spec, it appears we could use the 'ExternalRef' for this purpose.I'm not to familiar with the SPDX spec yet, but adding individual files entries as
ExternalRefinstead ofdownloadLocationto a recipes spdx sounds reasonable.I think in the long term adding a
SPDXRef-Layer-xyzentry per layer with adownloadLocationpointing to the subpath of the layer inside a git repo. I'm not quite shure if it would be possible to formulate a dependency on a file contained within a different SPDXRef, e.g.
Quoted 6 lines
``` SPDXRef-Layer-xyz:recipes-core/base-files/base-files/fstab ```or if we'd have to create a SPDXRef Item for each file within a layer in order to reference it properly. That would make it even more verbose.
Hi Daniel,
Yes, we should have a way to get Git information at parser time to avoid calling for every file://. But I don't know exactly how to do this, because if we need to add a variable in all layers, and of course, we can't do this, we still need a fallback if the variable doesn't exist. In my case, we don't use OE-Core from https://git.openembedded.org/, we have all layers in an internal infrastructure, so we need to change all variables to point to our fork. My idea to use functions from 'oe.buildcfg' is to get Git information from the layer and not from variables, it doesn't matter if it's a fork or not. But I didn't cover the case where different remotes are used. I know that when using repo to manage Git repositories, it's common to use different remotes, e.g., https://github.com/Freescale/fsl-community-bsp-platform/blob/scarthgap/default.xml. Honestly, I don't know if adding a variable to set the "downloadLocation" will be better or not.
Quoted 10 lines
The approach of having a layer as an independent SPDXRef would mean getting the git revision etc. for that layer would run only once per build and not per
file://entry inSRC_URI.Quoted 3 lines
The idea is to have two options to add this information: one to add the full path of a file, and another to add the git information
IMO the full path to the file is unneeded information, if the file is solely available locally a
NOASSERTIONwould be appropriate.
The 'path' option is to not use the 'Git' information, e.g., when using a tarball and not a Git repo. The 'locator' will be '/home/user/src/openembedded-core/meta/recipes-core/busybox/files/syslog' instead of 'git+[https://git.openembedded.org/openembedded-core@ac5d9579a0db63b54bbebb5015de2ae860a462bf#meta/recipes-core/busybox/files/syslog](https://git.openembedded.org/openembedded-core@ac5d9579a0db63b54bbebb5015de2ae860a462bf#meta/recipes-core/busybox/files/syslog)'
Quoted 8 lines
Quoted 5 lines
Should I add a variable like 'SPDX_FILE_LOCATION_GIT_REMOTE_<layername> = "
remote_name"' to set a specific remote for each layer? Would setting the git remote be sufficient to cover most cases?In my experimentation I removed the per-layer setting again because tracking the
vardepsfor thedo_create_spdxget's more complicated with per-layer variables.
Uhmm, good point, I didn't think about vardeps.
Quoted 3 lines
Quoted 1 line
Sincerely
Daniel Wagenknecht