Skip to contentSkip to conversation

utils/generate-cyclonedx: fixup scp-style git sites

PATCH Applied Peter Korsgaard 2 participants 4 messages Jun 19 – Jun 26

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 utils/generate-cyclonedx: fixup scp-style git sites

Commit e8c54ffb3d ("utils/generate-cyclonedx: generate vcs externalReferences for source repos") added externalReferences to the source code of packages.

This unfortunately causes issues with packages (in br2-external) fetching from git using the scp-like syntax, E.G.:

FOO_SITE_METHOD = git
FOO_SITE = [email protected]:<project>/<repo>.git

Which ends up in the SBOM as:

[

  {
    "type": "vcs",
    "url": "[email protected]:<project>/<repo>.git",
    "comment": "git repository"

}
]

This (correctly) causes Dependency track to reject the SBOM import with:

{
"status": 400,
"title": "The uploaded BOM is invalid",
"detail": "Schema validation failed",
"errors": [
"$.components[2].externalReferences[0].url: does not match the iri-reference pattern must be a valid RFC 3987 IRI-reference", "$.components[2].externalReferences[0].url: does not match the iri-reference pattern must be a valid RFC 3987 IRI-reference", "$.components[2].externalReferences[0].url: does not match the regex pattern ^urn:cdx:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/[1-9][0-9]*$",
]
}

The CycloneDX spec indeed requires a URI:

The URI (URL or URN) to the external reference. External references are URIs and therefore can accept any URL scheme including https (RFC-7230), mailto (RFC-2368), tel (RFC-3966), and dns (RFC-4501)

https://cyclonedx.org/docs/1.6/json/#metadata_tools_oneOf_i0_components_items_externalReferences_items_url

The user@host:project/repo.git is a git-specific shorthand for a git-over-ssh URL. From man git-clone:

Git supports ssh, git, http, and https protocols (in addition, ftp and ftps can be used for fetching, but this is inefficient and deprecated; do not use them).

The native transport (i.e. git:// URL) does no authentication and should be used with caution on unsecured networks.

The following syntaxes may be used with them:

 •   ssh://[user@]host.xz[:port]/path/to/repo.git/
 •   git://host.xz[:port]/path/to/repo.git/
 •   http[s]://host.xz[:port]/path/to/repo.git/
 •   ftp[s]://host.xz[:port]/path/to/repo.git/

An alternative scp-like syntax may also be used with the ssh protocol:

• [user@]host.xz:path/to/repo.git/

So convert the scp-like syntax to ssh:// URLs in parse_uris() for spec compliance.

---
 utils/generate-cyclonedx | 6 ++++++
 1 file changed, 6 insertions(+)
utils/generate-cyclonedx +6 -0
index bade018cd4..df12ee84c0 100755
@@ -280,6 +280,12 @@ def parse_uris(uris: list[str]) -> Iterator[tuple[list[str], str]]:
         scheme, _, stripped_uri = uri.partition("+")
         if stripped_uri:
             parsed = urllib.parse.urlparse(stripped_uri)
+
+            # convert scp-style host:path site to ssh:// uri
+            if scheme == "git" and not parsed.scheme:
+                host, _, path = stripped_uri.partition(":")
+                stripped_uri = "ssh://" + host + "/" + path
+
             if parsed.hostname != "sources.buildroot.net":
                 yield scheme.split("|"), stripped_uri
 
Signature · 6 lines
2.47.3

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Acked-by Thomas Perale on v1 1/1
#1 lore raw

Hi Peter,

In reply of:

Quoted 57 lines

Commit e8c54ffb3d ("utils/generate-cyclonedx: generate vcs externalReferences for source repos") added externalReferences to the source code of packages.

This unfortunately causes issues with packages (in br2-external) fetching from git using the scp-like syntax, E.G.:

FOO_SITE_METHOD = git
FOO_SITE = [email protected]:<project>/<repo>.git

Which ends up in the SBOM as:

[

  {
    "type": "vcs",
    "url": "[email protected]:<project>/<repo>.git",
    "comment": "git repository"

}
]

This (correctly) causes Dependency track to reject the SBOM import with:

{
"status": 400,
"title": "The uploaded BOM is invalid",
"detail": "Schema validation failed",
"errors": [
"$.components[2].externalReferences[0].url: does not match the iri-reference pattern must be a valid RFC 3987 IRI-reference", "$.components[2].externalReferences[0].url: does not match the iri-reference pattern must be a valid RFC 3987 IRI-reference", "$.components[2].externalReferences[0].url: does not match the regex pattern ^urn:cdx:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/[1-9][0-9]*$",
]
}

The CycloneDX spec indeed requires a URI:

The URI (URL or URN) to the external reference. External references are URIs and therefore can accept any URL scheme including https (RFC-7230), mailto (RFC-2368), tel (RFC-3966), and dns (RFC-4501)

https://cyclonedx.org/docs/1.6/json/#metadata_tools_oneOf_i0_components_items_externalReferences_items_url

The user@host:project/repo.git is a git-specific shorthand for a git-over-ssh URL. From man git-clone:

Git supports ssh, git, http, and https protocols (in addition, ftp and ftps can be used for fetching, but this is inefficient and deprecated; do not use them).

The native transport (i.e. git:// URL) does no authentication and should be used with caution on unsecured networks.

The following syntaxes may be used with them:

 •   ssh://[user@]host.xz[:port]/path/to/repo.git/
 •   git://host.xz[:port]/path/to/repo.git/
 •   http[s]://host.xz[:port]/path/to/repo.git/
 •   ftp[s]://host.xz[:port]/path/to/repo.git/
Quoted 8 lines

An alternative scp-like syntax may also be used with the ssh protocol:

• [user@]host.xz:path/to/repo.git/

So convert the scp-like syntax to ssh:// URLs in parse_uris() for spec compliance.

I see that "downloads" section from "show-info" are not tested on the generate-cyclonedx tests yet. I will add this later.

Thanks for the detailled description !

Quoted 28 lines
---
 utils/generate-cyclonedx | 6 ++++++
 1 file changed, 6 insertions(+)
utils/generate-cyclonedx +6 -0
index bade018cd4..df12ee84c0 100755
@@ -280,6 +280,12 @@ def parse_uris(uris: list[str]) -> Iterator[tuple[list[str], str]]:
         scheme, _, stripped_uri = uri.partition("+")
         if stripped_uri:
             parsed = urllib.parse.urlparse(stripped_uri)
+
+            # convert scp-style host:path site to ssh:// uri
+            if scheme == "git" and not parsed.scheme:
+                host, _, path = stripped_uri.partition(":")
+                stripped_uri = "ssh://" + host + "/" + path
+
             if parsed.hostname != "sources.buildroot.net":
                 yield scheme.split("|"), stripped_uri
 
Signature · 6 lines
2.47.3

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#2 lore raw
In reply to Peter Korsgaard
Quoted 59 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Peter" == Peter Korsgaard <[email protected]> writes:

Commit e8c54ffb3d ("utils/generate-cyclonedx: generate vcs externalReferences for source repos") added externalReferences to the source code of packages.

This unfortunately causes issues with packages (in br2-external) fetching from git using the scp-like syntax, E.G.:

FOO_SITE_METHOD = git
FOO_SITE = [email protected]:<project>/<repo>.git

Which ends up in the SBOM as:

[

  {
    "type": "vcs",
    "url": "[email protected]:<project>/<repo>.git",
    "comment": "git repository"

}
]

This (correctly) causes Dependency track to reject the SBOM import with:

{
"status": 400,
"title": "The uploaded BOM is invalid",
"detail": "Schema validation failed",
"errors": [
"$.components[2].externalReferences[0].url: does not match the iri-reference pattern must be a valid RFC 3987 IRI-reference", "$.components[2].externalReferences[0].url: does not match the iri-reference pattern must be a valid RFC 3987 IRI-reference", "$.components[2].externalReferences[0].url: does not match the regex pattern ^urn:cdx:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/[1-9][0-9]*$",
]
}

The CycloneDX spec indeed requires a URI:

The URI (URL or URN) to the external reference. External references are URIs and therefore can accept any URL scheme including https (RFC-7230), mailto (RFC-2368), tel (RFC-3966), and dns (RFC-4501)

https://cyclonedx.org/docs/1.6/json/#metadata_tools_oneOf_i0_components_items_externalReferences_items_url

The user@host:project/repo.git is a git-specific shorthand for a git-over-ssh URL. From man git-clone:

Git supports ssh, git, http, and https protocols (in addition, ftp and ftps can be used for fetching, but this is inefficient and deprecated; do not use them).

The native transport (i.e. git:// URL) does no authentication and should be used with caution on unsecured networks.

The following syntaxes may be used with them:

 •   ssh://[user@]host.xz[:port]/path/to/repo.git/
 •   git://host.xz[:port]/path/to/repo.git/
 •   http[s]://host.xz[:port]/path/to/repo.git/
 •   ftp[s]://host.xz[:port]/path/to/repo.git/
Quoted 8 lines

An alternative scp-like syntax may also be used with the ssh protocol:

• [user@]host.xz:path/to/repo.git/

So convert the scp-like syntax to ssh:// URLs in parse_uris() for spec compliance.

Committed, thanks.

Signature · 5 lines
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw
In reply to Peter Korsgaard

In reply of:

Quoted 57 lines

Commit e8c54ffb3d ("utils/generate-cyclonedx: generate vcs externalReferences for source repos") added externalReferences to the source code of packages.

This unfortunately causes issues with packages (in br2-external) fetching from git using the scp-like syntax, E.G.:

FOO_SITE_METHOD = git
FOO_SITE = [email protected]:<project>/<repo>.git

Which ends up in the SBOM as:

[

  {
    "type": "vcs",
    "url": "[email protected]:<project>/<repo>.git",
    "comment": "git repository"

}
]

This (correctly) causes Dependency track to reject the SBOM import with:

{
"status": 400,
"title": "The uploaded BOM is invalid",
"detail": "Schema validation failed",
"errors": [
"$.components[2].externalReferences[0].url: does not match the iri-reference pattern must be a valid RFC 3987 IRI-reference", "$.components[2].externalReferences[0].url: does not match the iri-reference pattern must be a valid RFC 3987 IRI-reference", "$.components[2].externalReferences[0].url: does not match the regex pattern ^urn:cdx:[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/[1-9][0-9]*$",
]
}

The CycloneDX spec indeed requires a URI:

The URI (URL or URN) to the external reference. External references are URIs and therefore can accept any URL scheme including https (RFC-7230), mailto (RFC-2368), tel (RFC-3966), and dns (RFC-4501)

https://cyclonedx.org/docs/1.6/json/#metadata_tools_oneOf_i0_components_items_externalReferences_items_url

The user@host:project/repo.git is a git-specific shorthand for a git-over-ssh URL. From man git-clone:

Git supports ssh, git, http, and https protocols (in addition, ftp and ftps can be used for fetching, but this is inefficient and deprecated; do not use them).

The native transport (i.e. git:// URL) does no authentication and should be used with caution on unsecured networks.

The following syntaxes may be used with them:

 •   ssh://[user@]host.xz[:port]/path/to/repo.git/
 •   git://host.xz[:port]/path/to/repo.git/
 •   http[s]://host.xz[:port]/path/to/repo.git/
 •   ftp[s]://host.xz[:port]/path/to/repo.git/
Quoted 8 lines

An alternative scp-like syntax may also be used with the ssh protocol:

• [user@]host.xz:path/to/repo.git/

So convert the scp-like syntax to ssh:// URLs in parse_uris() for spec compliance.

Applied to 2025.02.x & 2026.05.x. Thanks

Quoted 28 lines
---
 utils/generate-cyclonedx | 6 ++++++
 1 file changed, 6 insertions(+)
utils/generate-cyclonedx +6 -0
index bade018cd4..df12ee84c0 100755
@@ -280,6 +280,12 @@ def parse_uris(uris: list[str]) -> Iterator[tuple[list[str], str]]:
         scheme, _, stripped_uri = uri.partition("+")
         if stripped_uri:
             parsed = urllib.parse.urlparse(stripped_uri)
+
+            # convert scp-style host:path site to ssh:// uri
+            if scheme == "git" and not parsed.scheme:
+                host, _, path = stripped_uri.partition(":")
+                stripped_uri = "ssh://" + host + "/" + path
+
             if parsed.hostname != "sources.buildroot.net":
                 yield scheme.split("|"), stripped_uri
 
Signature · 6 lines
2.47.3

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot