Removing Github release SRC_URIs from oe-core recipes?
Hi all,
The TL;DR:
If no one objects, I would like to put into motion that we gradually move all oe-core recipes SRC_URIs from Github releases to git source code (archives) for better source code traceability.
Reasoning follows:
---
Currently, there are (at least¹) 35 recipes in poky that use Github release packages, rather than using the unmodified git source code (archives).
In my humble opinion, this is not a good idea, as the GitHub releases do not necessarily align with the content of the accompanying git repository, obscurifying the actual source code that goes into a yocto package.
Case in point, I am currently working on a (MIT licensed) bitbake layer for integrating Osselot (https://www.osselot.org/), a software license curation database into the bitbake build process, in the hope of collectively lessening the reoccuring work (and pain) of companies with strict open source software clearance requirements.
Osselot uses the original source code for point-releases (e.g. GitHub Source Code archive) for curating package license information on file- level and exporting the license information in SPDX format.
My bitbake integration would then attempt to match code in the S directory against license data within the SPDX file, verifying file equivalence by hash.
This is when I noticed a reoccurring pattern within multiple oe-core packages that rely on GitHub releases:
For example, the release download for libexpat
(https://github.com/libexpat/libexpat/releases/download/R_2_5_0/expat-2.5.0.tar.gz
) contains these additional files when compared to the corresponding tag in the git repository:
"expat_config.h.in",
"configure",
"Makefile.in",
"aclocal.m4",
"expat_config.h",
"m4/ltversion.m4",
"m4/ltoptions.m4",
"m4/libtool.m4",
"m4/lt~obsolete.m4",
"m4/ltsugar.m4",
"conftools/config.guess",
"conftools/config.sub",
"conftools/missing",
"conftools/install-sh",
"conftools/ar-lib",
"conftools/ltmain.sh",
"conftools/depcomp",
"conftools/compile",
"conftools/test-driver",
"xmlwf/Makefile.in",
"lib/Makefile.in"In this case the files are the result of pre-run autoconf (which from a openembedded perspective do not belong here in the first place, that's the "do_configure" tasks job), however theoretically the GitHub project maintainer could upload anything as a "GitHub release" that does not necessarily need to align with the actually git source code.
So not only does this make things complicated for me and/or the Osselot curators (either I am stuck with X amount of uncurated data, or curation efforts will have to be expanded to also include GitHub release archives) in my specific use-case, but it also reduces the confidence in the reproducibility. Who is to say that a build from the git sources produces the same output of a "magic" release package?
I believe that with the amount of work oe has already put into reproducible builds from scratch, it would be a shame if we wouldn't expand this effort make (at least) oe-core recipes build reproducible from git source.
What are your thoughts on this?
With best regards,
Jasper
---
¹: grep -rE "inherit.*github-releases" poky/**/*.bb | wc -l Currently, the GITHUB_BASE_URI variable used in this class defaults to "https://github.com/${BPN}/${BPN}/releases/".
Quoted 9 lines · Jasper Orschulko via lists.openembedded.org
If no one objects, I would like to put into motion that we gradually move all oe-core recipes
SRC_URIsfrom Github releases to git source code (archives) for better source code traceability.Reasoning follows:
---
Quoted 8 lines
Currently, there are (at least¹) 35 recipes in poky that use Github release packages, rather than using the unmodified git source code (archives).
In my humble opinion, this is not a good idea, as the GitHub releases do not necessarily align with the content of the accompanying git repository, obscurifying the actual source code that goes into a yocto package.
I don't understand this argument. It's an either-or situation: either the recipe takes the source code from release tarballs and never touches the git repository or vice versa. The two are never mixed. So if you need to trace the source code from the tarball, you trace it back to what is in the tarball, and stop there. That this source code also happens to be developed in git somewhere isn't relevant.
Quoted 6 lines
I believe that with the amount of work oe has already put into reproducible builds from scratch, it would be a shame if we wouldn't expand this effort make (at least) oe-core recipes build reproducible from git source.
What are your thoughts on this?
Firmly negative, I'm afraid. Release tarballs are not a github invention, for example pretty much all of GNU and GNOME software is released that way, it's just that those projects obscure their respective git repositories better (on savannah and gitlab respectively).
As 'old school' as they are, release tarballs are still an entirely valid way to consume source code from upstream, and in some cases upstreams prefer their users doing it that way. You need to fix the osselot tool to respect and support that.
I'm also curious about what osselot outputs that can't be done with a oe-core class directly? Is something missing in existing create-spdx classes? Here's for example what osselot provides for busybox, but I can't really make sense of it:
Alex
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 6 lines · Alexander Kanavin via lists.openembedded.org
I'm also curious about what osselot outputs that can't be done with a oe-core class directly? Is something missing in existing create-spdx classes? Here's for example what osselot provides for busybox, but I can't really make sense of it:
Okay, I've read the README file in that repo, and if i understood it right, the process is:
- run fossology
- have a human inspect the output, and correct it on a file by file basis (tremendous waste of time and limited developer resources even when done the 'open source way' if you ask me but whatevs)
- place the corrected output into the above repository
Do you really really need the 'human corrected' part of all this? It will never possibly cover all of the packages you need to ship and match all their versions. If not, then maybe using and improving this layer is a better way out?
https://git.yoctoproject.org/meta-spdxscanner/
Alex
Hi Jasper,
one thing to keep in mind is that for some repositories, downloading a release tarball is significantly faster than cloning the whole repository (since the tarball is much smaller than the git repository containing the entire git history).
Etienne
Quoted 42 lines · Alexander Kanavin
Quoted 6 lines · Alexander Kanavin via lists.openembedded.org
I'm also curious about what osselot outputs that can't be done with a oe-core class directly? Is something missing in existing create-spdx classes? Here's for example what osselot provides for busybox, but I can't really make sense of it:
Okay, I've read the README file in that repo, and if i understood it right, the process is:
- run fossology
- have a human inspect the output, and correct it on a file by file basis (tremendous waste of time and limited developer resources even when done the 'open source way' if you ask me but whatevs)
- place the corrected output into the above repositoryDo you really really need the 'human corrected' part of all this? It will never possibly cover all of the packages you need to ship and match all their versions. If not, then maybe using and improving this layer is a better way out?
Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#193634):
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_193634&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=r7zIQ1hCuaw4Wi_3pum0WMoAGtRcnP46ZIUSk5U39M06UdjBoG2FhzgXSrFqApMp&s=z3CFxWVTjcEstB_nG_JPATqWSmPa3VDD0K-AfVudff4&e=
Mute This Topic:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_103730186_7048771&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=r7zIQ1hCuaw4Wi_3pum0WMoAGtRcnP46ZIUSk5U39M06UdjBoG2FhzgXSrFqApMp&s=Hr3qkkZmag5Ih91KHUuxVDIqcVKHLvgZMAwazFeWlSk&e=
Group Owner: [email protected] Unsubscribe:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=r7zIQ1hCuaw4Wi_3pum0WMoAGtRcnP46ZIUSk5U39M06UdjBoG2FhzgXSrFqApMp&s=bkuz8C12_AoHFBh1nsC9EAgy1CUR7G5ymbxEdXabqA4&e=
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Hi Etienne,
GitHub also provides tarballs for the unmodified source code, e.g.:https://github.com/libexpat/libexpat/archive/refs/tags/R_2_5_0.tar.gz
This corresponds to the "Source Code" asset that Github automatically adds to releases, see:
https://github.com/libexpat/libexpat/releases/tag/R_2_5_0
So if download speeds are a concern, using the source archive tars would be a valid alternative.
Best,
Jasper
Quoted 57 lines · Etienne Cordonnier
Hi Jasper,
one thing to keep in mind is that for some repositories, downloading a release tarball is significantly faster than cloning the whole repository (since the tarball is much smaller than the git repository containing the entire git history).Etienne
Quoted 47 lines · Alexander Kanavin
On Mon, 15 Jan 2024 at 07:26, Alexander Kanavin via lists.openembedded.org <[email protected]>
Quoted 9 lines · wrote:
I'm also curious about what osselot outputs that can't be done with a
oe-core class directly? Is something missing in existing create- spdx
classes? Here's for example what osselot provides for busybox, but I
can't really make sense of it:Okay, I've read the README file in that repo, and if i understood it
right, the process is:
- run fossology
- have a human inspect the output, and correct it on a file by file basis (tremendous waste of time and limited developer resources even
when done the 'open source way' if you ask me but whatevs)
- place the corrected output into the above repositoryDo you really really need the 'human corrected' part of all this? It
will never possibly cover all of the packages you need to ship and match all their versions. If not, then maybe using and improving this
layer is a better way out?
https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_meta-2Dspdxscanner_&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=r7zIQ1hCuaw4Wi_3pum0WMoAGtRcnP46ZIUSk5U39M06UdjBoG2FhzgXSrFqApMp&s=TRpMNG5GBFN9HHhbGEuDcivc9L3M4YJoBiOCfnMpvgE&e=Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#193634):
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_193634&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=r7zIQ1hCuaw4Wi_3pum0WMoAGtRcnP46ZIUSk5U39M06UdjBoG2FhzgXSrFqApMp&s=z3CFxWVTjcEstB_nG_JPATqWSmPa3VDD0K-AfVudff4&e=
Mute This Topic:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_103730186_7048771&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=r7zIQ1hCuaw4Wi_3pum0WMoAGtRcnP46ZIUSk5U39M06UdjBoG2FhzgXSrFqApMp&s=Hr3qkkZmag5Ih91KHUuxVDIqcVKHLvgZMAwazFeWlSk&e=
Group Owner: [email protected] Unsubscribe:
https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIFaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsRih1DULJTKUkSGa66m67E&m=r7zIQ1hCuaw4Wi_3pum0WMoAGtRcnP46ZIUSk5U39M06UdjBoG2FhzgXSrFqApMp&s=bkuz8C12_AoHFBh1nsC9EAgy1CUR7G5ymbxEdXabqA4&e=
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 9 lines · Jasper Orschulko
GitHub also provides tarballs for the unmodified source code, e.g.:https://github.com/libexpat/libexpat/archive/refs/tags/R_2_5_0.tar.gz
This corresponds to the "Source Code" asset that Github automatically adds to releases, see:
https://github.com/libexpat/libexpat/releases/tag/R_2_5_0So if download speeds are a concern, using the source archive tars would be a valid alternative.
Sadly it wouldn't. Auto-generated github archives are known to be non-deterministic, and we even have a qa check to ensure no recipe is using them. I didn't raise this point because my objections are on the principle of using release tarballs, not this technicality.
As for the stuff that autotools-based tarballs add to the pristine git checkout, none of it goes into the target image, and by and large it's the same between all autotools-based projects, so you could probably special-case it into the reports or even the original data. But I would first raise the issue with osselot upstream.
Alex
Quoted 5 lines
Sadly it wouldn't. Auto-generated github archives are known to be non-deterministic, and we even have a qa check to ensure no recipe is using them. I didn't raise this point because my objections are on the
principle of using release tarballs, not this technicality.
Ah ok, that is interesting. Could you please elaborate on the non- deterministic nature of the github archives? This information could be vital for the Osselot project as well, as they currently use said archives as the data basis.
Quoted 11 lines · Jasper Orschulko
Quoted 6 lines
Sadly it wouldn't. Auto-generated github archives are known to be non-deterministic, and we even have a qa check to ensure no recipe is using them. I didn't raise this point because my objections are on the
principle of using release tarballs, not this technicality.Ah ok, that is interesting. Could you please elaborate on the non- deterministic nature of the github archives? This information could be vital for the Osselot project as well, as they currently use said archives as the data basis.
https://lwn.net/Articles/921787/ covers the subject well, I think.
Alex
Thanks, good to know! Well, that buries that idea then I guess... we will have to somehow deal with the release tarballs then I guess.
Quoted 20 lines · Alexander Kanavin
Quoted 9 lines · Jasper Orschulko
Quoted 8 lines
Sadly it wouldn't. Auto-generated github archives are known to be non-deterministic, and we even have a qa check to ensure no recipe is
using them. I didn't raise this point because my objections are on
the
principle of using release tarballs, not this technicality.Quoted 5 lines
Ah ok, that is interesting. Could you please elaborate on the non- deterministic nature of the github archives? This information could be
vital for the Osselot project as well, as they currently use said archives as the data basis.https://lwn.net/Articles/921787/ covers the subject well, I think.
Alex
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Alexander Kanavin via lists.openembedded.org
Quoted 7 lines · Jasper Orschulko
Quoted 6 lines
Sadly it wouldn't. Auto-generated github archives are known to be non-deterministic, and we even have a qa check to ensure no recipe is using them. I didn't raise this point because my objections are on the
principle of using release tarballs, not this technicality.
Quoted 6 lines
Quoted 4 lines
Ah ok, that is interesting. Could you please elaborate on the non- deterministic nature of the github archives? This information could be vital for the Osselot project as well, as they currently use said archives as the data basis.
https://lwn.net/Articles/921787/ covers the subject well, I think.
Oh, and before I forget: Alberto (in cc) has recently done some work in bitbake to enable file-level source tracing, presumably for exact same purpose as you are, so I'd suggest you discuss what kind of approach he envisions (or maybe has implemented).
Alex
Ah nice, that is definitely something we would be interested in! Alberto, would you be so kind to share some details?
Quoted 32 lines · Alexander Kanavin
On Mon, 15 Jan 2024 at 15:05, Alexander Kanavin via lists.openembedded.org <[email protected]>
Quoted 13 lines · wrote:
Quoted 10 lines · Jasper Orschulko
Quoted 9 lines
Sadly it wouldn't. Auto-generated github archives are known to be
non-deterministic, and we even have a qa check to ensure no recipe is
using them. I didn't raise this point because my objections are on
the
principle of using release tarballs, not this technicality.Quoted 8 lines
Quoted 6 lines
Ah ok, that is interesting. Could you please elaborate on the non-
deterministic nature of the github archives? This information could be
vital for the Osselot project as well, as they currently use said archives as the data basis.https://lwn.net/Articles/921787/ covers the subject well, I think.
Oh, and before I forget: Alberto (in cc) has recently done some work in bitbake to enable file-level source tracing, presumably for exact same purpose as you are, so I'd suggest you discuss what kind of approach he envisions (or maybe has implemented).
Alex
Alberto's commit I was referring to is:
https://git.yoctoproject.org/poky/commit/?h=master&id=ef3e46afd910d4b7727d42c4c18b501525c65695
You can also dig up the related dicussion on the bitbake list.
Alex
Quoted 37 lines · Jasper Orschulko
Ah nice, that is definitely something we would be interested in! Alberto, would you be so kind to share some details?
Quoted 32 lines · Alexander Kanavin
On Mon, 15 Jan 2024 at 15:05, Alexander Kanavin via lists.openembedded.org <[email protected]>
Quoted 13 lines · wrote:
Quoted 10 lines · Jasper Orschulko
Quoted 9 lines
Sadly it wouldn't. Auto-generated github archives are known to be
non-deterministic, and we even have a qa check to ensure no recipe is
using them. I didn't raise this point because my objections are on
the
principle of using release tarballs, not this technicality.Quoted 8 lines
Quoted 6 lines
Ah ok, that is interesting. Could you please elaborate on the non-
deterministic nature of the github archives? This information could be
vital for the Osselot project as well, as they currently use said archives as the data basis.https://lwn.net/Articles/921787/ covers the subject well, I think.
Oh, and before I forget: Alberto (in cc) has recently done some work in bitbake to enable file-level source tracing, presumably for exact same purpose as you are, so I'd suggest you discuss what kind of approach he envisions (or maybe has implemented).
Alex
Hi Alex,
Quoted 7 lines
Okay, I've read the README file in that repo, and if i understood it right, the process is:
- run fossology
- have a human inspect the output, and correct it on a file by file basis (tremendous waste of time and limited developer resources even when done the 'open source way' if you ask me but whatevs)
- place the corrected output into the above repository
Correct.
Quoted 1 line
Do you really really need the 'human corrected' part of all this?
Unfortunately we do need the "human corrected" part (I wish we wouldn't). We have industry customers (and in turn our legal department) that demand a license compliance report and clearing on a "per-file" basis. Currently available scanning tools are unfortunately way to unreliable for usage without any human interaction (believe me, we tried).
Will this be stupid amount of work? Yes. Is this compliance obligations gone mad? IMO, absolutely yes.
But unfortunately this seems to be the way the industry is heading. We as a supplier company are getting more and more requests of this sort from our clients (big players in the public transport and automotive industry) who won't "play" with us unless these obligations are fulfilled. I've also heared similar stories from other companies. (That's what you get when you let company lawyers go wild 😉️)
So I'm not happy with the situation but doing the best I can, given the requirements. In our case that means sharing our license clearings (which we have to do in any case) with the open source community, in the hope that other companies have similar challenges and that we can get some crowd-sourcing going.
Quoted 2 lines
It will never possibly cover all of the packages you need to ship and match all their versions.
So what we are currently striving for is covering all target relevant packages (aka without any special suffixes) of a "basic linux build" (aka image-core-minimal) for LTS releases (starting with kirkstone). Additionally, meta-ossselot will have logic for re-use of other package versions (so does osselot as a whole), limiting curation to the diff. I also have some ideas on dealing with openembedded patches added to point releases (currently another pain point).
Best,
Jasper
Quoted 40 lines · Jasper Orschulko via lists.openembedded.org
Hi Alex,
Quoted 7 lines
Okay, I've read the README file in that repo, and if i understood it right, the process is:
- run fossology
- have a human inspect the output, and correct it on a file by file basis (tremendous waste of time and limited developer resources even when done the 'open source way' if you ask me but whatevs)
- place the corrected output into the above repositoryCorrect.
Quoted 1 line
Do you really really need the 'human corrected' part of all this?
Unfortunately we do need the "human corrected" part (I wish we wouldn't). We have industry customers (and in turn our legal department) that demand a license compliance report and clearing on a "per-file" basis. Currently available scanning tools are unfortunately way to unreliable for usage without any human interaction (believe me, we tried).
Will this be stupid amount of work? Yes. Is this compliance obligations gone mad? IMO, absolutely yes.
But unfortunately this seems to be the way the industry is heading. We as a supplier company are getting more and more requests of this sort from our clients (big players in the public transport and automotive industry) who won't "play" with us unless these obligations are fulfilled. I've also heared similar stories from other companies. (That's what you get when you let company lawyers go wild 😉️)
So I'm not happy with the situation but doing the best I can, given the requirements. In our case that means sharing our license clearings (which we have to do in any case) with the open source community, in the hope that other companies have similar challenges and that we can get some crowd-sourcing going.
Quoted 2 lines
It will never possibly cover all of the packages you need to ship and match all their versions.
Quoted 7 lines
So what we are currently striving for is covering all target relevant packages (aka without any special suffixes) of a "basic linux build" (aka image-core-minimal) for LTS releases (starting with kirkstone). Additionally, meta-ossselot will have logic for re-use of other package versions (so does osselot as a whole), limiting curation to the diff. I also have some ideas on dealing with openembedded patches added to point releases (currently another pain point).
One note, when I worked on this a few years ago (pre-2020) each source file was correlated to an entry in our SPDX database. The licenses were manually reviewed and that was all attached to a file's hash.
This way it didn't matter if you were looking at an upstream tarball, an upstream git repository, or any other mode of getting the source code.
Tracking was effectively "human reviewed" SPDX was compared against the patched file manifest. Any sources that didn't match the checksum were flagged for review. A master database of checksums were also available to see if the file was duplicated in other repositories as well. This also allowed comparison of license/copyright and possibly patent data between different projects. (and ALSO allowed exceptions for generated m4, configure.in, etc files!)
This manual review work absolutely needs to be done for some parts of the world and some company requirements... but be careful not to make the work so complex it can't be completed.
--Mark
Quoted 13 lines
Best,
Jasper-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#193674): https://lists.openembedded.org/g/openembedded-core/message/193674
Mute This Topic: https://lists.openembedded.org/mt/103730186/3616948
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-