repost: how to create a SPDX "notice file" from a build?
i asked about this a couple months ago but didn't see any replies,
so i'll ask again with a little more detail.
colleague wants to, from YP (actually petalinux but should be
irrelevant), some sort of SPDX "notice file", along the lines of what can be generated by black duck. it doesn't need to be identical, but it would be useful to at least have a first pass that people can look at and say what they want tweaked.
is there an example of how to (using the spdx.bbclass class file, i
assume) do something like this? thanks.
rday
Signature · 7 lines
========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca
Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================Quoted 11 lines · Robert P. J. Day
i asked about this a couple months ago but didn't see any replies,
so i'll ask again with a little more detail.colleague wants to, from YP (actually petalinux but should be
irrelevant), some sort of SPDX "notice file", along the lines of what can be generated by black duck. it doesn't need to be identical, but it would be useful to at least have a first pass that people can look at and say what they want tweaked.is there an example of how to (using the spdx.bbclass class file, i
assume) do something like this? thanks.
a lilttle more progress, and then an error -- i installed fossology
on my ubuntu 18.04 dev box, then added
INHERIT += "spdx"
to local.conf, and tried to build a simple qemuarmx86-64, and very quickly had a number of native packages fail with:
//////////// start //////////
DEBUG: Executing python function do_spdx
ERROR: Error executing a python function in exec_python_func() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:do_spdx(d)
0003:
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx
0079: foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False
0080: foss_command = "wget %s --post-file=%s %s"\
0081: % (foss_flags, info['tar_file'], foss_server)
0082: *** 0083: foss_result = run_fossology(foss_command, foss_full_spdx)
0084: if foss_result is not None:
0085: (foss_package_info, foss_file_info, foss_license_info) = foss_result
0086: spdx_file_info = create_spdx_doc(local_file_info, foss_file_info)
0087: ## write to cache
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology
0226: # Package info
0227: package_info = {}
0228: if full_spdx:
0229: # All mandatory, only one occurrence
*** 0230: package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0]
0231: package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0]
0232: package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0]
0233: # These may be more than one
0234: package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output)Exception: IndexError: list index out of range
ERROR: list index out of range
DEBUG: Python function do_spdx finished
ERROR: Function failed: do_spdx
/////////// end /////////
i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate the usage of spdx.bbclass?
rday
Signature · 7 lines
========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca
Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================Quoted 59 lines · On Fri, Nov 22, 2019 at 9:03 AM [email protected] <[email protected]>…
Quoted 11 lines · Robert P. J. Day
i asked about this a couple months ago but didn't see any replies,
so i'll ask again with a little more detail.colleague wants to, from YP (actually petalinux but should be
irrelevant), some sort of SPDX "notice file", along the lines of what can be generated by black duck. it doesn't need to be identical, but it would be useful to at least have a first pass that people can look at and say what they want tweaked.is there an example of how to (using the spdx.bbclass class file, i
assume) do something like this? thanks.a lilttle more progress, and then an error -- i installed fossology
on my ubuntu 18.04 dev box, then addedINHERIT += "spdx"
to local.conf, and tried to build a simple qemuarmx86-64, and very quickly had a number of native packages fail with:
//////////// start //////////
DEBUG: Executing python function
do_spdx
ERROR: Error executing a python function inexec_python_func()autogenerated:The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func()autogenerated', lineno: 2, function: <module>0001: *** 0002:do_spdx(d) 0003: File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx 0079: foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False 0080: foss_command = "wget %s --post-file=%s %s"\ 0081: % (foss_flags, info['tar_file'], foss_server) 0082:*** 0083:
foss_result=run_fossology(foss_command, foss_full_spdx)
0084: iffoss_resultis not None:
0085: (foss_package_info,foss_file_info,foss_license_info) =foss_result
0086:spdx_file_info=create_spdx_doc(local_file_info, foss_file_info)
0087: ## write to cache
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function:run_fossology0226: # Package info 0227: package_info = {} 0228: if full_spdx: 0229: # All mandatory, only one occurrence *** 0230: package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0] 0231: package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0] 0232: package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0] 0233: # These may be more than one 0234: package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output)Exception: IndexError: list index out of range
ERROR: list index out of range
DEBUG: Python functiondo_spdxfinished
ERROR: Function failed:do_spdx
Quoted 5 lines
/////////// end /////////
i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate the usage of spdx.bbclass?
it once generated SPDX manifest from patched sources for package and file level it has sadly regressed, so would need some love as you see above.
Quoted 20 lines
rday
--
========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.caTwitter: http://twitter.com/rpjday LinkedIn: http://ca.linkedin.com/in/rpjday========================================================================
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.View/Reply Online (#47387): https://lists.yoctoproject.org/g/yocto/message/47387
Mute This Topic: https://lists.yoctoproject.org/mt/61664060/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 57 lines · On 11/22/19 9:03 AM, [email protected] wrote:
Quoted 11 lines · Robert P. J. Day
i asked about this a couple months ago but didn't see any replies,
so i'll ask again with a little more detail.colleague wants to, from YP (actually petalinux but should be
irrelevant), some sort of SPDX "notice file", along the lines of what can be generated by black duck. it doesn't need to be identical, but it would be useful to at least have a first pass that people can look at and say what they want tweaked.is there an example of how to (using the spdx.bbclass class file, i
assume) do something like this? thanks.a lilttle more progress, and then an error -- i installed fossology
on my ubuntu 18.04 dev box, then addedINHERIT += "spdx"
to local.conf, and tried to build a simple qemuarmx86-64, and very quickly had a number of native packages fail with:
//////////// start //////////
DEBUG: Executing python function
do_spdx
ERROR: Error executing a python function inexec_python_func()autogenerated:The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func()autogenerated', lineno: 2, function: <module>0001: *** 0002:do_spdx(d) 0003: File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx 0079: foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False 0080: foss_command = "wget %s --post-file=%s %s"\ 0081: % (foss_flags, info['tar_file'], foss_server) 0082:*** 0083:
foss_result=run_fossology(foss_command, foss_full_spdx)
0084: iffoss_resultis not None:
0085: (foss_package_info,foss_file_info,foss_license_info) =foss_result
0086:spdx_file_info=create_spdx_doc(local_file_info, foss_file_info)
0087: ## write to cache
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function:run_fossology0226: # Package info 0227: package_info = {} 0228: if full_spdx: 0229: # All mandatory, only one occurrence *** 0230: package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0] 0231: package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0] 0232: package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0] 0233: # These may be more than one 0234: package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output)Exception: IndexError: list index out of range
ERROR: list index out of range
DEBUG: Python functiondo_spdxfinished
ERROR: Function failed:do_spdx
Quoted 5 lines
/////////// end /////////
i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate the usage of spdx.bbclass?
Would you mind opening a Yocto defect.
- armin
Quoted 11 lines
rday
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.View/Reply Online (#47387): https://lists.yoctoproject.org/g/yocto/message/47387
Mute This Topic: https://lists.yoctoproject.org/mt/61664060/1024635
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 6 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 3 lines · On 11/22/19 9:03 AM, [email protected] wrote:
On Fri, 22 Nov 2019, Robert P. J. Day wrote:
Quoted 8 lines
Quoted 6 lines
/////////// end /////////
i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?Would you mind opening a Yocto defect.
That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...
Cheers,
Richard
Quoted 14 lines · Richard Purdie
Quoted 13 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 9 lines · On 11/22/19 9:03 AM, [email protected] wrote:
On Fri, 22 Nov 2019, Robert P. J. Day wrote:
/////////// end /////////
i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?
Quoted 6 lines
Quoted 1 line
Would you mind opening a Yocto defect.
That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...
yeah, that the impression i got ... is there a newer alternative to
that, or just the standard license info that's been generated all this time?
rday
Signature · 7 lines
========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca
Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================Quoted 14 lines · Richard Purdie
Quoted 13 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 9 lines · On 11/22/19 9:03 AM, [email protected] wrote:
On Fri, 22 Nov 2019, Robert P. J. Day wrote:
/////////// end /////////
i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?
Quoted 6 lines
Quoted 1 line
Would you mind opening a Yocto defect.
That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...
I think the use-cases have changed over time, even though parts and pieces are still valid. There are really a few groups to consider.
1) (old case) someone is building a system and wants to construct SPDX files for the things they are building. Contacting, uploading, getting a report from fossology may still be the best way of doing this.
2) (new case) things could be shipped with prebuilt SPDX files (based on fossology run by the system, maintainer, an addon layer, OSV, etc..) In this case we would want to simply tie a recipe to an SPDX and be able to correlate them.
3) In either case, we have a list of SPDX files, but that doesn't meet Robert's question. Something needs to process these SPDX files and generate notice files and similar. To me this is an external tool, that could optionally be invoked at image creation time (or by the user directly.)
Further, a 4th case.. what is the license of the components I've actually deployed. I've wanted to do this for a long time, but using the dwarf debug information you can determine what files were actually used to construct the binaries in your images. From that you can go back to the SPDX files and correlated to exactly what was deployed including file level copyright, notice, and license requirements (not just recipe) and produce an incredibly accurate report. Add to this that SPDX has the ability for custom fields that can be used to track other IP issues like patents, legal concerns, etc. And you could construct a report in a form for the legal organization of a company to review prior to product shipment.
Right now, we have an old way to do 1, but it doesn't solve Robert's issue -- even if it DID work. and no way to do the rest (that I am aware of).
--Mark
Quoted 14 lines
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.View/Reply Online (#47394): https://lists.yoctoproject.org/g/yocto/message/47394
Mute This Topic: https://lists.yoctoproject.org/mt/61664060/3616948
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 14 lines · Richard Purdie
Quoted 13 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 9 lines · On 11/22/19 9:03 AM, [email protected] wrote:
On Fri, 22 Nov 2019, Robert P. J. Day wrote:
/////////// end /////////
i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?
Quoted 6 lines
Quoted 1 line
Would you mind opening a Yocto defect.
That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...
as one more followup to this, first, i was asked by a colleague
about the possibility of generating an "SPDX notice file," and as i don't know what the format of one of those things is, it's hard for me to know how to answer. it seems clear at this point that spdx.bbclass is not going to cut it, as richard points out.
more to the point, has anyone had to do something that seems to
resemble what i was asked about? i'm open to suggestions as to what *is* supported so i can try it out and hand it over and at least say, "is this close to what you're looking for?"
thanks for any pointers.
rday
Signature · 7 lines
========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca
Twitter: http://twitter.com/rpjday
LinkedIn: http://ca.linkedin.com/in/rpjday
========================================================================(resending, the list rejected the first reply)
Quoted 29 lines · On 11/24/19 4:11 AM, [email protected] wrote:
Quoted 14 lines · Richard Purdie
Quoted 13 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 9 lines · On 11/22/19 9:03 AM, [email protected] wrote:
On Fri, 22 Nov 2019, Robert P. J. Day wrote:
/////////// end /////////
i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?Quoted 6 lines
Quoted 1 line
Would you mind opening a Yocto defect.
That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...
as one more followup to this, first, i was asked by a colleague
about the possibility of generating an "SPDX notice file," and as i don't know what the format of one of those things is, it's hard for me to know how to answer. it seems clear at this point that spdx.bbclass is not going to cut it, as richard points out.
Quoted 4 lines
more to the point, has anyone had to do something that seems to
resemble what i was asked about? i'm open to suggestions as to what *is* supported so i can try it out and hand it over and at least say, "is this close to what you're looking for?"
I know individual companies have created their own tools to do this, but since it's both required and could include liability (if you get it wrong), I'm not aware of any open source way of doing this.
If there is one, it would likely be published as part of the SPDX project/mailing lists.
--Mark
Quoted 14 lines
thanks for any pointers.
rday
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.View/Reply Online (#47399): https://lists.yoctoproject.org/g/yocto/message/47399
Mute This Topic: https://lists.yoctoproject.org/mt/61664060/3616948
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-