create-spdx: Get SPDX-License-Identifier from source
PATCH v2 1/1 create-spdx: Get SPDX-License-Identifier from source
This patch will read the begining of source files and try to find the SPDX-License-Identifier to populate the licenseInfoInFiles field for each source file. This does not populate licenseConcluded at this time, nor rolls it up to package level.
We read as binary file since some source code seem to have some binary characters, the license is then converted to ascii strings.
---
v2: Updated commit message, and fixed REGEX based on Peter's suggetion
meta/classes/create-spdx.bbclass | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
meta/classes/create-spdx.bbclass +23 -0
@@ -37,6 +37,24 @@ SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field for SPDX packages created f do_image_complete[depends] = "virtual/kernel:do_create_spdx" +def extract_licenses(filename): + import re + import oe.spdx + + lic_regex = re.compile(b'SPDX-License-Identifier:\s+([-A-Za-z\d. ]+)[ |\n|\r\n]*?') + + try: + with open(filename, 'rb') as f: + size = min(15000, os.stat(filename).st_size) + txt = f.read(size) + licenses = re.findall(lic_regex, txt) + if licenses: + ascii_licenses = [lic.decode('ascii') for lic in licenses] + return ascii_licenses + except Exception as e: + bb.warn(f"Exception reading {filename}: {e}") + return None + def get_doc_namespace(d, doc): import uuid namespace_uuid = uuid.uuid5(uuid.NAMESPACE_DNS, d.getVar("SPDX_UUID_NAMESPACE"))
@@ -232,6 +250,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv checksumValue=bb.utils.sha256_file(filepath), )) + if "SOURCE" in spdx_file.fileTypes: + extracted_lics = extract_licenses(filepath) + if extracted_lics: + spdx_file.licenseInfoInFiles = extracted_lics + doc.files.append(spdx_file) doc.add_relationship(spdx_pkg, "CONTAINS", spdx_file) spdx_pkg.hasFiles.append(spdx_file.SPDXID)
Signature · 1 line
2.31.1
Quoted 46 lines · Saul Wold
This patch will read the begining of source files and try to find the SPDX-License-Identifier to populate the licenseInfoInFiles field for each source file. This does not populate licenseConcluded at this time, nor rolls it up to package level.
We read as binary file since some source code seem to have some binary characters, the license is then converted to ascii strings.
Signed-off-by Saul Wold <[email protected]>---
v2: Updated commit message, and fixed REGEX based on Peter's suggetionmeta/classes/create-spdx.bbclass | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)meta/classes/create-spdx.bbclass +18
-0@@ -37,6 +37,24 @@ SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field for SPDX packages created f do_image_complete[depends] = "virtual/kernel:do_create_spdx" +def extract_licenses(filename): + import re + import oe.spdx + + lic_regex = re.compile(b'SPDX-License-Identifier:\s+([-A-Za-z\d. ]+)[ |\n|\r\n]*?') + + try: + with open(filename, 'rb') as f: + size = min(15000, os.stat(filename).st_size) + txt = f.read(size) + licenses = re.findall(lic_regex, txt) + if licenses: + ascii_licenses = [lic.decode('ascii') for lic in licenses] + return ascii_licenses + except Exception as e: + bb.warn(f"Exception reading {filename}: {e}") + return None + def get_doc_namespace(d, doc): import uuid namespace_uuid = uuid.uuid5(uuid.NAMESPACE_DNS, d.getVar("SPDX_UUID_NAMESPACE"))@@ -232,6 +250,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv checksumValue=bb.utils.sha256_file(filepath),
meta/classes/create-spdx.bbclass def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv · line 260
Quoted 10 lines
meta/classes/create-spdx.bbclass +5
-0)) + if "SOURCE" in spdx_file.fileTypes: + extracted_lics = extract_licenses(filepath) + if extracted_lics: + spdx_file.licenseInfoInFiles = extracted_lics + doc.files.append(spdx_file) doc.add_relationship(spdx_pkg, "CONTAINS", spdx_file) spdx_pkg.hasFiles.append(spdx_file.SPDXID)
IMO this seems like perhaps either going too far, or not far enough. If we go to the trouble to scan source files for explicit SPDX license declarations, but do not go as far as pattern detection like the meta-spdxscanner layer does with its use Scancode Toolkit
(https://github.com/nexB/scancode-toolkit), then it seems there's more potential for giving users a false impression as to the completeness of the resulting report/SBOM. Perhaps that can be handled by making it very clear that further scanning and auditing is still required in the hopefully forthcoming create-spdx.bbclass documentation, but I can imagine having to explain this to customers.
Scott
Quoted 60 lines · Scott Murray
Quoted 46 lines · Saul Wold
This patch will read the begining of source files and try to find the SPDX-License-Identifier to populate the licenseInfoInFiles field for each source file. This does not populate licenseConcluded at this time, nor rolls it up to package level.
We read as binary file since some source code seem to have some binary characters, the license is then converted to ascii strings.
Signed-off-by Saul Wold <[email protected]>---
v2: Updated commit message, and fixed REGEX based on Peter's suggetionmeta/classes/create-spdx.bbclass | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)meta/classes/create-spdx.bbclass +18
-0@@ -37,6 +37,24 @@ SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field for SPDX packages created f do_image_complete[depends] = "virtual/kernel:do_create_spdx" +def extract_licenses(filename): + import re + import oe.spdx + + lic_regex = re.compile(b'SPDX-License-Identifier:\s+([-A-Za-z\d. ]+)[ |\n|\r\n]*?') + + try: + with open(filename, 'rb') as f: + size = min(15000, os.stat(filename).st_size) + txt = f.read(size) + licenses = re.findall(lic_regex, txt) + if licenses: + ascii_licenses = [lic.decode('ascii') for lic in licenses] + return ascii_licenses + except Exception as e: + bb.warn(f"Exception reading {filename}: {e}") + return None + def get_doc_namespace(d, doc): import uuid namespace_uuid = uuid.uuid5(uuid.NAMESPACE_DNS, d.getVar("SPDX_UUID_NAMESPACE"))@@ -232,6 +250,11 @@ def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv checksumValue=bb.utils.sha256_file(filepath),meta/classes/create-spdx.bbclass def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv · line 260
Quoted 10 lines
meta/classes/create-spdx.bbclass +5
-0)) + if "SOURCE" in spdx_file.fileTypes: + extracted_lics = extract_licenses(filepath) + if extracted_lics: + spdx_file.licenseInfoInFiles = extracted_lics + doc.files.append(spdx_file) doc.add_relationship(spdx_pkg, "CONTAINS", spdx_file) spdx_pkg.hasFiles.append(spdx_file.SPDXID)IMO this seems like perhaps either going too far, or not far enough. If we go to the trouble to scan source files for explicit SPDX license
Quoted 8 lines
declarations, but do not go as far as pattern detection like the meta-spdxscanner layer does with its use Scancode Toolkit
(https://github.com/nexB/scancode-toolkit), then it seems there's more potential for giving users a false impression as to the completeness of the resulting report/SBOM. Perhaps that can be handled by making it very clear that further scanning and auditing is still required in the hopefully forthcoming create-spdx.bbclass documentation, but I can imagine having to explain this to customers.
Can you given an overview of what meta-spdxscanner does? I'm not quite clear what extra processing would be required here.
Quoted 14 lines
Scott
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161466): https://lists.openembedded.org/g/openembedded-core/message/161466
Mute This Topic: https://lists.openembedded.org/mt/88980079/3616693
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 76 lines · Joshua Watt
Quoted 65 lines · Scott Murray
Quoted 53 lines · Saul Wold
This patch will read the begining of source files and try to find the SPDX-License-Identifier to populate the licenseInfoInFiles field for each source file. This does not populate licenseConcluded at this time, nor rolls it up to package level.
We read as binary file since some source code seem to have some binary characters, the license is then converted to ascii strings.
Signed-off-by Saul Wold <[email protected]>---
v2: Updated commit message, and fixed REGEX based on Peter's suggetionmeta/classes/create-spdx.bbclass | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)meta/classes/create-spdx.bbclass +0
-0@@ -37,6 +37,24 @@ SPDX_SUPPLIER[doc] = "The SPDX PackageSupplier field forSPDX packages created f
do_image_complete[depends] = "virtual/kernel:do_create_spdx"
meta/classes/create-spdx.bbclass +5
-0+def extract_licenses(filename): + import re + import oe.spdx + + lic_regex = re.compile(b'SPDX-License-Identifier:\s+([-A-Za-z\d. ]+)[|\n|\r\n]*?')
meta/classes/create-spdx.bbclass +13
-0+ + try: + with open(filename, 'rb') as f: + size = min(15000, os.stat(filename).st_size) + txt = f.read(size) + licenses = re.findall(lic_regex, txt) + if licenses: + ascii_licenses = [lic.decode('ascii') for lic in licenses] + return ascii_licenses + except Exception as e: + bb.warn(f"Exception reading {filename}: {e}") + return None + def get_doc_namespace(d, doc): import uuid namespace_uuid = uuid.uuid5(uuid.NAMESPACE_DNS, d.getVar("SPDX_UUID_NAMESPACE"))@@ -232,6 +250,11 @@ def add_package_files(d, doc, spdx_pkg, topdir,
get_spdxid,get_types, *, archivchecksumValue=bb.utils.sha256_file(filepath), ))meta/classes/create-spdx.bbclass def add_package_files(d, doc, spdx_pkg, topdir, get_spdxid, get_types, *, archiv · line 260
Quoted 8 lines
meta/classes/create-spdx.bbclass +5
-0+ if "SOURCE" in spdx_file.fileTypes: + extracted_lics = extract_licenses(filepath) + if extracted_lics: + spdx_file.licenseInfoInFiles = extracted_lics + doc.files.append(spdx_file) doc.add_relationship(spdx_pkg, "CONTAINS", spdx_file) spdx_pkg.hasFiles.append(spdx_file.SPDXID)IMO this seems like perhaps either going too far, or not far enough. If we go to the trouble to scan source files for explicit SPDX license
Quoted 8 lines
declarations, but do not go as far as pattern detection like the meta-spdxscanner layer does with its use Scancode Toolkit
(https://github.com/nexB/scancode-toolkit), then it seems there's more potential for giving users a false impression as to the completeness of the resulting report/SBOM. Perhaps that can be handled by making it very clear that further scanning and auditing is still required in the hopefully forthcoming create-spdx.bbclass documentation, but I can imagine having to explain this to customers.
Quoted 2 lines
Can you given an overview of what meta-spdxscanner does? I'm not quite clear what extra processing would be required here.
Jan-Simon can talk to it better, as he's done some dev work on the layer and done tests with it against AGL (and the subsequent Fossology instance experimentation), but AFAIK for the actual scanning scancode-toolkit does pattern matching based license detection, so in theory it'll catch excerpts of or slightly modified versions of the licenses in its database, as opposed to just searching for SPDX-License-Identifier declarations. If everyone else is happy with the latter, I'm willing to believe I'm offbase in my concerns, but either way I do think the limitations are going to need to be documented so users (and their lawyers) are aware of them.
Scott
Hi,
Quoted 7 lines · Scott Murray
Jan-Simon can talk to it better, as he's done some dev work on the layer and done tests with it against AGL (and the subsequent Fossology instance experimentation), but AFAIK for the actual scanning scancode-toolkit does pattern matching based license detection, so in theory it'll catch excerpts of or slightly modified versions of the licenses in its database, as opposed to just searching for SPDX-License-Identifier declarations.
If I understand it correctly scancode-toolkit[1] claims to do this:
"ScanCode provides the most accurate license detection engine and does a full comparison (also known as diff or red line comparison) between a database of license texts and your code instead of relying only on approximate regex patterns or probabilistic search, edit distance or machine learning."
[1] https://github.com/nexB/scancode-toolkit
meta-spdxscanner can also create tarballs for each recipe and uploads them to a Fossology instance. Then Fossology takes care about license scanning and so on - and manual intervention is required.
A major issue here is, that the combined work (linking) is not taken into account at all, but it's all only per recipe.
Regards,
Robert
Signature · 11 lines
Robert Berger Embedded Software Evangelist Reliable Embedded Systems Consulting Training Engineering URL: https://www.reliableembeddedsystems.com Schedule a web meeting: https://calendly.com/reliableembeddedsystems/ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ --
Hi all
Quoted 13 lines
Quoted 2 lines
Can you given an overview of what meta-spdxscanner does? I'm not quite clear what extra processing would be required here.
Jan-Simon can talk to it better, as he's done some dev work on the layer and done tests with it against AGL (and the subsequent Fossology instance experimentation), but AFAIK for the actual scanning scancode-toolkit does pattern matching based license detection, so in theory it'll catch excerpts of or slightly modified versions of the licenses in its database, as opposed to just searching for SPDX-License-Identifier declarations. If everyone else is happy with the latter, I'm willing to believe I'm offbase in my concerns, but either way I do think the limitations are going to need to be documented so users (and their lawyers) are aware of them.
TLDR: meta-spdxscanner integrates with scanning tools. Either with fossology or scancode-tk. An upload to blackduck is also possible meanwhile.
Let's focus on fossology and scancode-tk.
a) fossology
Here we essentially integrate in the task chain and archive the sources after patching to upload them to a fossology instance. All the scanning/processing happens then on the server and after some time (a lot ! ;) ) we get a SPDX report back that we store alongside the package. This is a result of a scan, so it might catch licenses of files deep in the source tree that may not be declared in the recipe and so on.
Also, fossology offers then a webinterface for manual inspection and review. So this is a thorough but quite manual process. More for release work than daily or occasional stuff.
b) scancode-tk
scancode on the contrary will run on your host during the build and gather the data. It will write the spdx file out as well.
I think for us the interesting part would be to compare e.g. the scancode-tk scan from b) with what we have declared in the recipe.
Best,
JS
Hi,
Quoted 35 lines · Jan-Simon Möller
Hi all
Quoted 13 lines
Quoted 2 lines
Can you given an overview of what meta-spdxscanner does? I'm not quite clear what extra processing would be required here.
Jan-Simon can talk to it better, as he's done some dev work on the layer and done tests with it against AGL (and the subsequent Fossology instance experimentation), but AFAIK for the actual scanning scancode-toolkit does pattern matching based license detection, so in theory it'll catch excerpts of or slightly modified versions of the licenses in its database, as opposed to just searching for SPDX-License-Identifier declarations. If everyone else is happy with the latter, I'm willing to believe I'm offbase in my concerns, but either way I do think the limitations are going to need to be documented so users (and their lawyers) are aware of them.
TLDR: meta-spdxscanner integrates with scanning tools. Either with fossology or scancode-tk. An upload to blackduck is also possible meanwhile.
Let's focus on fossology and scancode-tk.
a) fossology
Here we essentially integrate in the task chain and archive the sources after patching to upload them to a fossology instance. All the scanning/processing happens then on the server and after some time (a lot ! ;) ) we get a SPDX report back that we store alongside the package. This is a result of a scan, so it might catch licenses of files deep in the source tree that may not be declared in the recipe and so on.
Also, fossology offers then a webinterface for manual inspection and review. So this is a thorough but quite manual process. More for release work than daily or occasional stuff.
Quoted 7 lines
b) scancode-tk
scancode on the contrary will run on your host during the build and gather the data. It will write the spdx file out as well.I think for us the interesting part would be to compare e.g. the scancode-tk scan from b) with what we have declared in the recipe.
I guess reports from both will be a superset of used licenses (and possibly copyright statements too) since the list of source files which are actually compiled is not known to these services.
Currently the source recipes which have multiple licenses including problematic ones, are not cleaned up for license compliance scan. E.g. GPLv3 licensed source code are not deleted at do_patch() time. Thus reports need to be manually adjusted.
Cheers,
-Mikko
Hi Mikko,
Quoted 3 lines
I guess reports from both will be a superset of used licenses (and possibly copyright statements too) since the list of source files which are actually compiled is not known to these services.
Yes, the input is the 'bare' patched source archives. Yes, you're right. We do only know the 'input' side of things aka all licenses of the source tree.
We do *not* know the 'output' side of things, yet. Aka what of these do end-up in the image.
But IMHO let's raise the bar step by step.
Quoted 4 lines
Currently the source recipes which have multiple licenses including problematic ones, are not cleaned up for license compliance scan. E.g. GPLv3 licensed source code are not deleted at
do_patch()time. Thus reports need to be manually adjusted.
Well, thats a different topic and should be discussed alongside meta-gplv2.
Best,
JS
Quoted 36 lines · Jan-Simon Moeller
Hi all
Quoted 13 lines
Quoted 2 lines
Can you given an overview of what meta-spdxscanner does? I'm not quite clear what extra processing would be required here.
Jan-Simon can talk to it better, as he's done some dev work on the layer and done tests with it against AGL (and the subsequent Fossology instance experimentation), but AFAIK for the actual scanning scancode-toolkit does pattern matching based license detection, so in theory it'll catch excerpts of or slightly modified versions of the licenses in its database, as opposed to just searching for SPDX-License-Identifier declarations. If everyone else is happy with the latter, I'm willing to believe I'm offbase in my concerns, but either way I do think the limitations are going to need to be documented so users (and their lawyers) are aware of them.
TLDR: meta-spdxscanner integrates with scanning tools. Either with fossology or scancode-tk. An upload to blackduck is also possible meanwhile.
Let's focus on fossology and scancode-tk.
a) fossology
Here we essentially integrate in the task chain and archive the sources after patching to upload them to a fossology instance. All the scanning/processing happens then on the server and after some time (a lot ! ;) ) we get a SPDX report back that we store alongside the package. This is a result of a scan, so it might catch licenses of files deep in the source tree that may not be declared in the recipe and so on.
Also, fossology offers then a webinterface for manual inspection and review. So this is a thorough but quite manual process. More for release work than daily or occasional stuff.
Quoted 7 lines
b) scancode-tk
scancode on the contrary will run on your host during the build and gather the data. It will write the spdx file out as well.I think for us the interesting part would be to compare e.g. the scancode-tk scan from b) with what we have declared in the recipe.
Excellent. Thanks for the synopsis.
Our perspective on all of the SPDX work we've done hasn't been to necessarily replace or replicate the behavior of these advanced licensing tools. Instead, we want to supplement them because there is a lot of information about recipes and packages that is pretty easy for us to know since we track it as metadata and are the ones actually *doing* the build. I don't know if we can claim to have a "complete" view of the SBoM, but I also don't think any single tool can provide a complete view. The amount of detail you want is going to be driven by you're requirements and some tools are simply better at providing different views than others. Our goal has always been to provide what we can (and especially what is hard for others to discover), and allow users to supplement with other tools of their choice.
The license scanning may not fall under the "hard to discover" for others (depending on how integrated they are to scan the build source), but it's certainly something we *can* provide and can be quite useful. I would expect that different scanning tools will generate different results anyway, so as long as we indicate that our scanning uses a simplistic method, I don't think including it is necessarily wrong or a bad idea. If you want a more detailed view, than some other tool should be used to supplement the SBoM.
I'm hopeful that someday our core SPDX work can make it a little easier on some of these tools so that they can "build" on our existing framework instead of replicating parts of it.
Quoted 10 lines
Best,
JS