Skip to contentSkip to conversation

repost: how to create a SPDX "notice file" from a build?

DISCUSSION [email protected] 5 participants 9 messages Nov 22, 2019 – Nov 24, 2019

lore
Share & tools
lore
#0 lore raw

i asked about this a couple months ago but didn't see any replies,
so i'll ask again with a little more detail.

colleague wants to, from YP (actually petalinux but should be
irrelevant), some sort of SPDX "notice file", along the lines of what can be generated by black duck. it doesn't need to be identical, but it would be useful to at least have a first pass that people can look at and say what they want tweaked.

is there an example of how to (using the spdx.bbclass class file, i
assume) do something like this? thanks.

rday

Signature · 7 lines
========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                         http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
#1 lore raw
Quoted 11 lines · Robert P. J. Day

i asked about this a couple months ago but didn't see any replies,
so i'll ask again with a little more detail.

colleague wants to, from YP (actually petalinux but should be
irrelevant), some sort of SPDX "notice file", along the lines of what can be generated by black duck. it doesn't need to be identical, but it would be useful to at least have a first pass that people can look at and say what they want tweaked.

is there an example of how to (using the spdx.bbclass class file, i
assume) do something like this? thanks.

a lilttle more progress, and then an error -- i installed fossology
on my ubuntu 18.04 dev box, then added

INHERIT += "spdx"

to local.conf, and tried to build a simple qemuarmx86-64, and very quickly had a number of native packages fail with:

//////////// start //////////

DEBUG: Executing python function do_spdx
ERROR: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_spdx(d)
     0003:
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx
     0079:        foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False
     0080:        foss_command = "wget %s --post-file=%s %s"\
     0081:            % (foss_flags, info['tar_file'], foss_server)
     0082:

*** 0083: foss_result = run_fossology(foss_command, foss_full_spdx)
0084: if foss_result is not None:
0085: (foss_package_info, foss_file_info, foss_license_info) = foss_result
0086: spdx_file_info = create_spdx_doc(local_file_info, foss_file_info)
0087: ## write to cache
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology

     0226:    # Package info
     0227:    package_info = {}
     0228:    if full_spdx:
     0229:        # All mandatory, only one occurrence
 *** 0230:        package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0]
     0231:        package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0]
     0232:        package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0]
     0233:        # These may be more than one
     0234:        package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output)

Exception: IndexError: list index out of range

ERROR: list index out of range
DEBUG: Python function do_spdx finished
ERROR: Function failed: do_spdx

/////////// end /////////

i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate the usage of spdx.bbclass?

rday

Signature · 7 lines
========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                         http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
#2 lore raw
Quoted 59 lines · On Fri, Nov 22, 2019 at 9:03 AM [email protected] <[email protected]>…
Quoted 11 lines · Robert P. J. Day

i asked about this a couple months ago but didn't see any replies,
so i'll ask again with a little more detail.

colleague wants to, from YP (actually petalinux but should be
irrelevant), some sort of SPDX "notice file", along the lines of what can be generated by black duck. it doesn't need to be identical, but it would be useful to at least have a first pass that people can look at and say what they want tweaked.

is there an example of how to (using the spdx.bbclass class file, i
assume) do something like this? thanks.

a lilttle more progress, and then an error -- i installed fossology
on my ubuntu 18.04 dev box, then added

INHERIT += "spdx"

to local.conf, and tried to build a simple qemuarmx86-64, and very quickly had a number of native packages fail with:

//////////// start //////////

DEBUG: Executing python function do_spdx
ERROR: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_spdx(d)
     0003:
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx
     0079:        foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False
     0080:        foss_command = "wget %s --post-file=%s %s"\
     0081:            % (foss_flags, info['tar_file'], foss_server)
     0082:

*** 0083: foss_result = run_fossology(foss_command, foss_full_spdx)
0084: if foss_result is not None:
0085: (foss_package_info, foss_file_info, foss_license_info) = foss_result
0086: spdx_file_info = create_spdx_doc(local_file_info, foss_file_info)
0087: ## write to cache
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology

     0226:    # Package info
     0227:    package_info = {}
     0228:    if full_spdx:
     0229:        # All mandatory, only one occurrence
 *** 0230:        package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0]
     0231:        package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0]
     0232:        package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0]
     0233:        # These may be more than one
     0234:        package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output)

Exception: IndexError: list index out of range

ERROR: list index out of range
DEBUG: Python function do_spdx finished
ERROR: Function failed: do_spdx

Quoted 5 lines

/////////// end /////////

i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate the usage of spdx.bbclass?

it once generated SPDX manifest from patched sources for package and file level it has sadly regressed, so would need some love as you see above.

Quoted 20 lines

rday

--

========================================================================
Robert P. J. Day Ottawa, Ontario, CANADA
http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday

========================================================================
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47387): https://lists.yoctoproject.org/g/yocto/message/47387
Mute This Topic: https://lists.yoctoproject.org/mt/61664060/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#3 lore raw
In reply to [email protected]
Quoted 57 lines · On 11/22/19 9:03 AM, [email protected] wrote:
Quoted 11 lines · Robert P. J. Day

i asked about this a couple months ago but didn't see any replies,
so i'll ask again with a little more detail.

colleague wants to, from YP (actually petalinux but should be
irrelevant), some sort of SPDX "notice file", along the lines of what can be generated by black duck. it doesn't need to be identical, but it would be useful to at least have a first pass that people can look at and say what they want tweaked.

is there an example of how to (using the spdx.bbclass class file, i
assume) do something like this? thanks.

a lilttle more progress, and then an error -- i installed fossology
on my ubuntu 18.04 dev box, then added

INHERIT += "spdx"

to local.conf, and tried to build a simple qemuarmx86-64, and very quickly had a number of native packages fail with:

//////////// start //////////

DEBUG: Executing python function do_spdx
ERROR: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>

     0001:
 *** 0002:do_spdx(d)
     0003:
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 83, function: do_spdx
     0079:        foss_full_spdx = d.getVar('FOSS_FULL_SPDX') == "true" or False
     0080:        foss_command = "wget %s --post-file=%s %s"\
     0081:            % (foss_flags, info['tar_file'], foss_server)
     0082:

*** 0083: foss_result = run_fossology(foss_command, foss_full_spdx)
0084: if foss_result is not None:
0085: (foss_package_info, foss_file_info, foss_license_info) = foss_result
0086: spdx_file_info = create_spdx_doc(local_file_info, foss_file_info)
0087: ## write to cache
File: '/home/rpjday/oe/dist/layers/poky/meta/classes/spdx.bbclass', lineno: 230, function: run_fossology

     0226:    # Package info
     0227:    package_info = {}
     0228:    if full_spdx:
     0229:        # All mandatory, only one occurrence
 *** 0230:        package_info['PackageCopyrightText'] = re.findall('PackageCopyrightText: (.*?</text>)', foss_output, re.S)[0]
     0231:        package_info['PackageLicenseDeclared'] = re.findall('PackageLicenseDeclared: (.*)', foss_output)[0]
     0232:        package_info['PackageLicenseConcluded'] = re.findall('PackageLicenseConcluded: (.*)', foss_output)[0]
     0233:        # These may be more than one
     0234:        package_info['PackageLicenseInfoFromFiles'] = re.findall('PackageLicenseInfoFromFiles: (.*)', foss_output)

Exception: IndexError: list index out of range

ERROR: list index out of range
DEBUG: Python function do_spdx finished
ERROR: Function failed: do_spdx

Quoted 5 lines

/////////// end /////////

i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate the usage of spdx.bbclass?

Would you mind opening a Yocto defect.

- armin

Quoted 11 lines

rday

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47387): https://lists.yoctoproject.org/g/yocto/message/47387
Mute This Topic: https://lists.yoctoproject.org/mt/61664060/1024635
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw
Quoted 6 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 3 lines · On 11/22/19 9:03 AM, [email protected] wrote:

On Fri, 22 Nov 2019, Robert P. J. Day wrote:

Quoted 8 lines
Quoted 6 lines

/////////// end /////////

i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?

Would you mind opening a Yocto defect.

That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...

Cheers,

Richard

#5 lore raw
Quoted 14 lines · Richard Purdie
Quoted 13 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 9 lines · On 11/22/19 9:03 AM, [email protected] wrote:

On Fri, 22 Nov 2019, Robert P. J. Day wrote:

/////////// end /////////

i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?

Quoted 6 lines
Quoted 1 line

Would you mind opening a Yocto defect.

That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...

yeah, that the impression i got ... is there a newer alternative to
that, or just the standard license info that's been generated all this time?

rday

Signature · 7 lines
========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                         http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
#6 lore raw
In reply to Richard Purdie
Quoted 14 lines · Richard Purdie
Quoted 13 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 9 lines · On 11/22/19 9:03 AM, [email protected] wrote:

On Fri, 22 Nov 2019, Robert P. J. Day wrote:

/////////// end /////////

i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?

Quoted 6 lines
Quoted 1 line

Would you mind opening a Yocto defect.

That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...

I think the use-cases have changed over time, even though parts and pieces are still valid. There are really a few groups to consider.

1) (old case) someone is building a system and wants to construct SPDX files for the things they are building. Contacting, uploading, getting a report from fossology may still be the best way of doing this.

2) (new case) things could be shipped with prebuilt SPDX files (based on fossology run by the system, maintainer, an addon layer, OSV, etc..) In this case we would want to simply tie a recipe to an SPDX and be able to correlate them.

3) In either case, we have a list of SPDX files, but that doesn't meet Robert's question. Something needs to process these SPDX files and generate notice files and similar. To me this is an external tool, that could optionally be invoked at image creation time (or by the user directly.)

Further, a 4th case.. what is the license of the components I've actually deployed. I've wanted to do this for a long time, but using the dwarf debug information you can determine what files were actually used to construct the binaries in your images. From that you can go back to the SPDX files and correlated to exactly what was deployed including file level copyright, notice, and license requirements (not just recipe) and produce an incredibly accurate report. Add to this that SPDX has the ability for custom fields that can be used to track other IP issues like patents, legal concerns, etc. And you could construct a report in a form for the legal organization of a company to review prior to product shipment.

Right now, we have an old way to do 1, but it doesn't solve Robert's issue -- even if it DID work. and no way to do the rest (that I am aware of).

--Mark

Quoted 14 lines

Cheers,

Richard

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47394): https://lists.yoctoproject.org/g/yocto/message/47394
Mute This Topic: https://lists.yoctoproject.org/mt/61664060/3616948
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#7 lore raw
In reply to Richard Purdie
Quoted 14 lines · Richard Purdie
Quoted 13 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 9 lines · On 11/22/19 9:03 AM, [email protected] wrote:

On Fri, 22 Nov 2019, Robert P. J. Day wrote:

/////////// end /////////

i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?

Quoted 6 lines
Quoted 1 line

Would you mind opening a Yocto defect.

That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...

as one more followup to this, first, i was asked by a colleague
about the possibility of generating an "SPDX notice file," and as i don't know what the format of one of those things is, it's hard for me to know how to answer. it seems clear at this point that spdx.bbclass is not going to cut it, as richard points out.

more to the point, has anyone had to do something that seems to
resemble what i was asked about? i'm open to suggestions as to what *is* supported so i can try it out and hand it over and at least say, "is this close to what you're looking for?"

thanks for any pointers.

rday

Signature · 7 lines
========================================================================
Robert P. J. Day                                 Ottawa, Ontario, CANADA
                         http://crashcourse.ca

Twitter:                                       http://twitter.com/rpjday
LinkedIn:                               http://ca.linkedin.com/in/rpjday
========================================================================
#8 lore raw

(resending, the list rejected the first reply)

Quoted 29 lines · On 11/24/19 4:11 AM, [email protected] wrote:
Quoted 14 lines · Richard Purdie
Quoted 13 lines · On Fri, 2019-11-22 at 09:59 -0800, akuster808 wrote:
Quoted 9 lines · On 11/22/19 9:03 AM, [email protected] wrote:

On Fri, 22 Nov 2019, Robert P. J. Day wrote:

/////////// end /////////

i have absolutely no idea what to think of this, and am open to
suggestions. does anyone have a working scenario to simply demonstrate
the usage of spdx.bbclass?

Quoted 6 lines
Quoted 1 line

Would you mind opening a Yocto defect.

That code hasn't been touched in a while and needs some serious attention. The underlying tools and processes have changed so much it may be a case of starting again and we should perhaps consider removing that class...

as one more followup to this, first, i was asked by a colleague
about the possibility of generating an "SPDX notice file," and as i don't know what the format of one of those things is, it's hard for me to know how to answer. it seems clear at this point that spdx.bbclass is not going to cut it, as richard points out.

Quoted 4 lines

more to the point, has anyone had to do something that seems to
resemble what i was asked about? i'm open to suggestions as to what *is* supported so i can try it out and hand it over and at least say, "is this close to what you're looking for?"

I know individual companies have created their own tools to do this, but since it's both required and could include liability (if you get it wrong), I'm not aware of any open source way of doing this.

If there is one, it would likely be published as part of the SPDX project/mailing lists.

--Mark

Quoted 14 lines

thanks for any pointers.

rday

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.

View/Reply Online (#47399): https://lists.yoctoproject.org/g/yocto/message/47399
Mute This Topic: https://lists.yoctoproject.org/mt/61664060/3616948
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-