package/pkg-golang: download deps to vendor tree if not present
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
NOTE: This patch is a RFC and is not intended for merging in its current state. It is a naiive implementation of the "go mod vendor" download step as a post-extract hook, for early testing and demonstration of the desired effect. I don't yet know what a final implementation might look like.
Add a new hook to POST_EXTRACT_HOOKS for Go packages which will create the "vendor" directory structure under $(@D)/vendor with Go package deps by running the "go mod vendor" command.
This will download dependency sources and use $GOPATH/pkg as a caching directory for lookups and downloads.
Go specifies commit hashes OR version tags in go.mod, and lists source code checksums in go.sum. The Go module system has a robust security model for preventing MITM attacks or changed Git tags on dependencies through this checksumming and explicitly-specified versioning approach.
Reference: https://blog.golang.org/using-go-modules
--- package/pkg-golang.mk | 10 ++++++++++ 1 file changed, 10 insertions(+)
package/pkg-golang.mk +10 -0
@@ -98,6 +98,16 @@ endef $(2)_POST_EXTRACT_HOOKS += $(2)_APPLY_EXTRACT_GOMOD +# WIP - download dependencies with the Go tool if vendor does not exist. +define $(2)_DOWNLOAD_GOMOD + if [ ! -d $$(@D)/vendor ]; then \ + cd $$(@D); \ + go mod vendor; \ + fi +endef + +$(2)_POST_EXTRACT_HOOKS += $(2)_DOWNLOAD_GOMOD + # Build step. Only define it if not already defined by the package .mk # file. ifndef $(2)_BUILD_CMDS
Signature · 1 line
2.28.0
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Christian, All,
On 2020-08-30 23:23 -0700, Christian Stewart spake thusly:
Quoted 4 lines
NOTE: This patch is a RFC and is not intended for merging in its current state. It is a naiive implementation of the "go mod vendor" download step as a post-extract hook, for early testing and demonstration of the desired effect. I don't yet know what a final implementation might look like.
Thanks, that's a good starting point.
My proposal was that we introduce per-package managers download backends, which would typically do something like the following (pseudo-code):
#!/usr/bin/env bash
# This file is support/fownload/go
# Parse options:
actual_site_method="$(get_option '--actual-site-method')"
# Call actual download helper:
"${0%/*}/${actual_site_method}" "${@}" -o "${temp_tarball}"
# Populate the vendor:
tar xf "${temp_tarball}" -C "${temp_directory}"
cd "${temp_directory}/${package_name_version}"
go mod vendor
cd ..
tar czf "${final_tarball}" "${package_name_version}"(of course, the details would be a bit more complex, and would require that we pass the actual site method vi the download infra, but the idea is there)
What's your opinion on this?
See also the following mails from Thomas, which contain copies of some of the IRC discussions we had on the topic (about rust and cargo, but that's the same topic):
http://lists.busybox.net/pipermail/buildroot/2020-August/289895.html http://lists.busybox.net/pipermail/buildroot/2020-August/289894.html
Quoted 6 lines
Add a new hook to
POST_EXTRACT_HOOKSfor Go packages which will create the "vendor" directory structure under $(@D)/vendor with Go package deps by running the "go mod vendor" command.This will download dependency sources and use $GOPATH/pkg as a caching directory for lookups and downloads.
But that does the download at extract time, and we would like that we still be able to do:
$ make source
# Unplug network
$ makeAlso, the hook is registered in the infra (we can't do otherwise), so it means it would run after any hook registered by the package, while those hooks may expect the package to be fully available (.e. fully vendored).
Quoted 4 lines
Go specifies commit hashes OR version tags in go.mod, and lists source code checksums in go.sum. The Go module system has a robust security model for preventing MITM attacks or changed Git tags on dependencies through this checksumming and explicitly-specified versioning approach.
This is good, because supposedly that will allow us to generate reproducible archives, and have hashes for them (in foo.hash)
Regards,
Yann E. MORIN.
Quoted 31 lines
Reference: https://blog.golang.org/using-go-modules
Signed-off-by Christian Stewart <[email protected]>--- package/pkg-golang.mk | 10 ++++++++++ 1 file changed, 10 insertions(+)package/pkg-golang.mk +10
-0@@ -98,6 +98,16 @@ endef $(2)_POST_EXTRACT_HOOKS += $(2)_APPLY_EXTRACT_GOMOD +# WIP - download dependencies with the Go tool if vendor does not exist. +define $(2)_DOWNLOAD_GOMOD + if [ ! -d $$(@D)/vendor ]; then \ + cd $$(@D); \ + go mod vendor; \ + fi +endef + +$(2)_POST_EXTRACT_HOOKS += $(2)_DOWNLOAD_GOMOD + # Build step. Only define it if not already defined by the package .mk # file. ifndef $(2)_BUILD_CMDSSignature · 1 line
2.28.0
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Yann, All,
Quoted 31 lines · Yann E. MORIN
Christian, All,
On 2020-08-30 23:23 -0700, Christian Stewart spake thusly:
Quoted 4 lines
NOTE: This patch is a RFC and is not intended for merging in its current state. It is a naiive implementation of the "go mod vendor" download step as a post-extract hook, for early testing and demonstration of the desired effect. I don't yet know what a final implementation might look like.
Thanks, that's a good starting point.
My proposal was that we introduce per-package managers download backends, which would typically do something like the following (pseudo-code):
#!/usr/bin/env bash # This file is support/fownload/go # Parse options: actual_site_method="$(get_option '--actual-site-method')" # Call actual download helper: "${0%/*}/${actual_site_method}" "${@}" -o "${temp_tarball}" # Populate the vendor: tar xf "${temp_tarball}" -C "${temp_directory}" cd "${temp_directory}/${package_name_version}" go mod vendor cd .. tar czf "${final_tarball}" "${package_name_version}"
Quoted 5 lines
(of course, the details would be a bit more complex, and would require that we pass the actual site method vi the download infra, but the idea is there)
What's your opinion on this?
As we spoke about plenty on the IRC, I'm sure you know my opinion on this but I figure I mention it anyway: I believe we should split these recursive-requirements from the base tar. This should allow Proprietary applications and TPIP requirements to be captured while maintaining separation between them.
Now, to your point about "well, what if the repository has proprietary dependencies?". I think this is still a valid point, especially when looking at the case you mentioned of "proprietary app with commingled TPIP+proprietary requirements", but I believe we should be able to handle this at a buildroot level fairly reasonably.
I took a look at go mod, it seems to share a similar mechanism with cargo which allows us to pass local paths for dependencies. My proposition is to put the responsibility of whomever is adding a proprietary application, which has mixed dependencies, to instead split any proprietary modules into selectable options in buildroot, and use the standard depends mechanism. To enforce this, we could investigate using license-coalescing options of the package managers to find any proprietary dependencies and fail if they're found to be pointing to upstream URLs (and would be captured) with an error message clearly describing our intentions.
Quoted 15 lines
See also the following mails from Thomas, which contain copies of some of the IRC discussions we had on the topic (about rust and cargo, but that's the same topic):
http://lists.busybox.net/pipermail/buildroot/2020-August/289895.html http://lists.busybox.net/pipermail/buildroot/2020-August/289894.htmlQuoted 6 lines
Add a new hook to
POST_EXTRACT_HOOKSfor Go packages which will create the "vendor" directory structure under $(@D)/vendor with Go package deps by running the "go mod vendor" command.This will download dependency sources and use $GOPATH/pkg as a caching directory for lookups and downloads.
Quoted 6 lines
But that does the download at extract time, and we would like that we still be able to do:
$ make source # Unplug network $ make
In my (so far unshared) patchset, my solution to do this agrees with your suggestion above, by adding download-backend support for these package managers. I leveraged the implementation suggested previously by patrick[1] to use the cargo-dl step to then create two tarballs:
<package>-<ver>.tar.gz <- we're all familiar with
<package>-<ver>-vendor.tar.gz <- the TPIP portion
The reasons for splitting are shared above. I believe that patchset is a good initial direction, and I think those interested in this patchset who are unfamiliar with that one should give it a review.
Thanks,
Sam
1: http://patchwork.ozlabs.org/project/buildroot/list/?series=159771
Quoted 55 lines
Also, the hook is registered in the infra (we can't do otherwise), so it means it would run after any hook registered by the package, while those hooks may expect the package to be fully available (.e. fully vendored).
Quoted 4 lines
Go specifies commit hashes OR version tags in go.mod, and lists source code checksums in go.sum. The Go module system has a robust security model for preventing MITM attacks or changed Git tags on dependencies through this checksumming and explicitly-specified versioning approach.
This is good, because supposedly that will allow us to generate reproducible archives, and have hashes for them (in foo.hash)
Regards,
Yann E. MORIN.Quoted 29 lines
Reference: https://blog.golang.org/using-go-modules
Signed-off-by Christian Stewart <[email protected]>--- package/pkg-golang.mk | 10 ++++++++++ 1 file changed, 10 insertions(+)package/pkg-golang.mk +10
-1@@ -98,6 +98,16 @@ endef $(2)_POST_EXTRACT_HOOKS += $(2)_APPLY_EXTRACT_GOMOD +# WIP - download dependencies with the Go tool if vendor does not exist. +define $(2)_DOWNLOAD_GOMOD + if [ ! -d $$(@D)/vendor ]; then \ + cd $$(@D); \ + go mod vendor; \ + fi +endef + +$(2)_POST_EXTRACT_HOOKS += $(2)_DOWNLOAD_GOMOD + # Build step. Only define it if not already defined by the package .mk # file. ifndef $(2)_BUILD_CMDS --Quoted 2 lines
2.28.0
--
.-----------------.--------------------.------------------.--------------------.| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |'------------------------------^-------^------------------^--------------------'
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Hello,
Quoted 24 lines · On Thu, 3 Sep 2020 05:52:56 -0500 Sam Voss <[email protected]> wrote:
Quoted 24 lines
My proposal was that we introduce per-package managers download backends, which would typically do something like the following (pseudo-code):
#!/usr/bin/env bash # This file is support/fownload/go # Parse options: actual_site_method="$(get_option '--actual-site-method')" # Call actual download helper: "${0%/*}/${actual_site_method}" "${@}" -o "${temp_tarball}" # Populate the vendor: tar xf "${temp_tarball}" -C "${temp_directory}" cd "${temp_directory}/${package_name_version}" go mod vendor cd .. tar czf "${final_tarball}" "${package_name_version}"(of course, the details would be a bit more complex, and would require that we pass the actual site method vi the download infra, but the idea is there)
Quoted 7 lines
Quoted 1 line
What's your opinion on this?
As we spoke about plenty on the IRC, I'm sure you know my opinion on this but I figure I mention it anyway: I believe we should split these recursive-requirements from the base tar. This should allow Proprietary applications and TPIP requirements to be captured while maintaining separation between them.
For those tuning in: TPIP stands for (I believe) "Third Party Intellectual Property".
I think I agree with the idea of splitting into two tarballs the package source code itself from the source code of all its dependencies.
Quoted 2 lines
Now, to your point about "well, what if the repository has proprietary dependencies?".
That was indeed my question as well.
Quoted 5 lines
I think this is still a valid point, especially when looking at the case you mentioned of "proprietary app with commingled TPIP+proprietary requirements", but I believe we should be able to handle this at a buildroot level fairly reasonably.
Quoted 10 lines
I took a look at
go mod, it seems to share a similar mechanism with cargo which allows us to pass local paths for dependencies. My proposition is to put the responsibility of whomever is adding a proprietary application, which has mixed dependencies, to instead split any proprietary modules into selectable options in buildroot, and use the standard depends mechanism. To enforce this, we could investigate using license-coalescing options of the package managers to find any proprietary dependencies and fail if they're found to be pointing to upstream URLs (and would be captured) with an error message clearly describing our intentions.
This feels like a reasonable approach to me.
Question: for the dependencies, instead of having a single tarball for all of them, would there be a way of having separate tarballs for each dependency that gets pulled by "go mod" or "cargo" ? If so, then we could perhaps be able to teach the package which parts of the package are proprietary (including the proprietary dependencies) and which parts are open-source and under which license.
Note that as a first step, I am personally perfectly fine with what you are proposing. The above question is really just a question, not a request to implement something like that.
Quoted 11 lines
In my (so far unshared) patchset, my solution to do this agrees with your suggestion above, by adding download-backend support for these package managers. I leveraged the implementation suggested previously by patrick[1] to use the cargo-dl step to then create two tarballs:
<package>-<ver>.tar.gz <- we're all familiar with
<package>-<ver>-vendor.tar.gz <- the TPIP portionThe reasons for splitting are shared above. I believe that patchset is a good initial direction, and I think those interested in this patchset who are unfamiliar with that one should give it a review.
Could you submit your patch series ? Did you fix the issues that we pointed out in the review of Patrick's series ?
I think we have a good opportunity to try to solve this problem for both Cargo and Go at the same time, so that we at least see if there's a reasonably similar solution that can be used.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Hey Thomas,
Quoted 32 lines · Thomas Petazzoni
Hello,
Quoted 26 lines · On Thu, 3 Sep 2020 05:52:56 -0500 Sam Voss <[email protected]> wrote:
Quoted 25 lines
My proposal was that we introduce per-package managers download backends, which would typically do something like the following (pseudo-code):
#!/usr/bin/env bash # This file is support/fownload/go # Parse options: actual_site_method="$(get_option '--actual-site-method')" # Call actual download helper: "${0%/*}/${actual_site_method}" "${@}" -o "${temp_tarball}" # Populate the vendor: tar xf "${temp_tarball}" -C "${temp_directory}" cd "${temp_directory}/${package_name_version}" go mod vendor cd .. tar czf "${final_tarball}" "${package_name_version}"(of course, the details would be a bit more complex, and would require that we pass the actual site method vi the download infra, but the idea is there)
What's your opinion on this?
Quoted 8 lines
Quoted 5 lines
As we spoke about plenty on the IRC, I'm sure you know my opinion on this but I figure I mention it anyway: I believe we should split these recursive-requirements from the base tar. This should allow Proprietary applications and TPIP requirements to be captured while maintaining separation between them.
For those tuning in: TPIP stands for (I believe) "Third Party Intellectual Property".
Sorry, that is correct, I should have clarified. Thanks for mentioning it.
Quoted 25 lines
I think I agree with the idea of splitting into two tarballs the package source code itself from the source code of all its dependencies.
Quoted 2 lines
Now, to your point about "well, what if the repository has proprietary dependencies?".
That was indeed my question as well.
Quoted 5 lines
I think this is still a valid point, especially when looking at the case you mentioned of "proprietary app with commingled TPIP+proprietary requirements", but I believe we should be able to handle this at a buildroot level fairly reasonably.
Quoted 10 lines
I took a look at
go mod, it seems to share a similar mechanism with cargo which allows us to pass local paths for dependencies. My proposition is to put the responsibility of whomever is adding a proprietary application, which has mixed dependencies, to instead split any proprietary modules into selectable options in buildroot, and use the standard depends mechanism. To enforce this, we could investigate using license-coalescing options of the package managers to find any proprietary dependencies and fail if they're found to be pointing to upstream URLs (and would be captured) with an error message clearly describing our intentions.
Quoted 8 lines
This feels like a reasonable approach to me.
Question: for the dependencies, instead of having a single tarball for all of them, would there be a way of having separate tarballs for each dependency that gets pulled by "go mod" or "cargo" ? If so, then we could perhaps be able to teach the package which parts of the package are proprietary (including the proprietary dependencies) and which parts are open-source and under which license.
I don't exactly know, but to your point I know it pulls them into correctly-named folders, however no version string attached:
VENDOR/ws2_32-sys/Cargo.toml
Quoted 4 lines
Note that as a first step, I am personally perfectly fine with what you are proposing. The above question is really just a question, not a request to implement something like that.
Understood, I figured since I had a vendor tar I may as well take a look at what it produced.
Quoted 9 lines
Quoted 9 lines
In my (so far unshared) patchset, my solution to do this agrees with your suggestion above, by adding download-backend support for these package managers. I leveraged the implementation suggested previously by patrick[1] to use the cargo-dl step to then create two tarballs:
<package>-<ver>.tar.gz <- we're all familiar with
<package>-<ver>-vendor.tar.gz <- the TPIP portion
Quoted 6 lines
Quoted 3 lines
The reasons for splitting are shared above. I believe that patchset is a good initial direction, and I think those interested in this patchset who are unfamiliar with that one should give it a review.
Could you submit your patch series ? Did you fix the issues that we pointed out in the review of Patrick's series ?
I did not change Patrick's series at all, built a new change-set on top of it. I will have to go back and integrate them together, and I can take a look at the issues with him in the meantime.
Thanks,
Sam
Signature · 13 lines
Sam Voss | Sr. Software Engineer | Commercial Avionics COLLINS AEROSPACE 400 Collins Road NE, Cedar Rapids, Iowa 52498, USA Tel: +1 319 263 4039 sam.voss at collins.com | collinsaerospace.com CONFIDENTIALITY WARNING: This message may contain proprietary and/or privileged information of Collins Aerospace and its affiliated companies. If you are not the intended recipient, please 1) Do not disclose, copy, distribute or use this message or its contents. 2) Advise the sender by return email. 3) Delete all copies (including all attachments) from your computer. Your cooperation is greatly appreciated.
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Hello,
Quoted 11 lines · On Thu, 3 Sep 2020 08:01:27 -0500 Sam Voss <[email protected]> wrote:
Quoted 6 lines
Question: for the dependencies, instead of having a single tarball for all of them, would there be a way of having separate tarballs for each dependency that gets pulled by "go mod" or "cargo" ? If so, then we could perhaps be able to teach the package which parts of the package are proprietary (including the proprietary dependencies) and which parts are open-source and under which license.
I don't exactly know, but to your point I know it pulls them into correctly-named folders, however no version string attached:
VENDOR/ws2_32-sys/Cargo.toml
So potentially we could tarball independently each dependency, perhaps in a tarball named:
<pkgname>-<pkgversion>-<dependencyname>.tar.gz
where <dependencyname> here would be "ws2_32-sys".
The question is then how we can associate the licenses to each dependency. Again, I don't think it's worth tackling at this moment.
Quoted 6 lines
Quoted 2 lines
Could you submit your patch series ? Did you fix the issues that we pointed out in the review of Patrick's series ?
I did not change Patrick's series at all, built a new change-set on top of it. I will have to go back and integrate them together, and I can take a look at the issues with him in the meantime.
Yes, if you can re-integrate your improvements within the patches from Patrick and submit an updated series, it would be great.
Thanks,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
In reply to Thomas PetazzoniHi Thomas, Sam,
Thanks much for looking into this, and I'm excited at the idea of splitting vendor into a separate tarball.
In my mind the optimal approach would be to leverage the Go module system to produce a tarball with associated hash for every Go module that's imported by any of the selected Build targets, and vendor specifically those, excluding all else.
I can write the Go code to do this & prototype a proof of concept if this is the way we are interested in going. It would be IMO appropriate because every Go module is checksummed, will produce a guaranteed identical output tarball that we can add to the hash file, and the vendor tree would only contain the necessary packages for the build targets flagged by the buildroot config.
Quoted 10 lines · Thomas Petazzoni
Quoted 10 lines
I took a look at
go mod, it seems to share a similar mechanism with cargo which allows us to pass local paths for dependencies. My proposition is to put the responsibility of whomever is adding a proprietary application, which has mixed dependencies, to instead split any proprietary modules into selectable options in buildroot, and use the standard depends mechanism. To enforce this, we could investigate using license-coalescing options of the package managers to find any proprietary dependencies and fail if they're found to be pointing to upstream URLs (and would be captured) with an error message clearly describing our intentions.
This fits well with what I'm describing above. The Go code to produce / extract the tarballs for the dependency modules and the module metadata could also be used to produce the license check and legal output. In fact, we could even generate the upstream URLs from this, and all other metadata desired (even authors information and a dependency chart).
Quoted 6 lines
Question: for the dependencies, instead of having a single tarball for all of them, would there be a way of having separate tarballs for each dependency that gets pulled by "go mod" or "cargo" ? If so, then we could perhaps be able to teach the package which parts of the package are proprietary (including the proprietary dependencies) and which parts are open-source and under which license.
See above.
Quoted 3 lines
Note that as a first step, I am personally perfectly fine with what you are proposing. The above question is really just a question, not a request to implement something like that.
To me those two are the same, heh.
Quoted 3 lines
I think we have a good opportunity to try to solve this problem for both Cargo and Go at the same time, so that we at least see if there's a reasonably similar solution that can be used.
Since our host-go infrastructure is now up to par, we can integrate the tool I'm describing above as a host-go package with no external dependencies since it would only use the Go stdlib.
Best regards,
Christian Stewart
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
In reply to Sam VossSam, All,
On 2020-09-03 05:52 -0500, Sam Voss spake thusly:
Quoted 4 lines · Yann E. MORIN
Quoted 3 lines
My proposal was that we introduce per-package managers download backends, which would typically do something like the following (pseudo-code):
[--SNIP--]
Quoted 11 lines
As we spoke about plenty on the IRC, I'm sure you know my opinion on this but I figure I mention it anyway: I believe we should split these recursive-requirements from the base tar. This should allow Proprietary applications and TPIP requirements to be captured while maintaining separation between them.
Now, to your point about "well, what if the repository has proprietary dependencies?". I think this is still a valid point, especially when looking at the case you mentioned of "proprietary app with commingled TPIP+proprietary requirements", but I believe we should be able to handle this at a buildroot level fairly reasonably.
And as I explained on IRC (which I am going to repeat here so that it is archived in the list archives, or for all to see as well), I believe this dichotomy between main and vendored archives is incorrect.
What I understand is that, for proprietary applications, you do not want to provide the main tarball, but since those package managers make it so easy to depend on external, FLOSS components, you want to be able to provide those vendored FLOSS stuff to be compliant, without providing the code for your proprietary blurb.
Consider for example that your company develops two proprietary packages, foo and bar, which have the following dependencies, which will ultimately be vendored in those packages (names totally made up; licenses between brakcets; CORP means proprietary package):
- foo [CORP] depends on libssl [MIT]
- bar [CORP] depends on foo and libhttp [MIT]
So if you have a separate archive for bar's vendoring, it will include your foo proprietary package, and in the case of 'bar', the separation of archive will not help you properly separate the "TPIP" that you would have to distribute to be complaint.
And I don't think we could have a single, simple mechanism for that, except by splitting each and every vendored dependencies at all levels into their own archives.
And I doubt this would be something we would want to do.
Quoted 6 lines
I took a look at
go mod, it seems to share a similar mechanism with cargo which allows us to pass local paths for dependencies. My proposition is to put the responsibility of whomever is adding a proprietary application, which has mixed dependencies, to instead split any proprietary modules into selectable options in buildroot, and use the standard depends mechanism.
This is doomed to not work, because it relies on process. And since this is not enforce by the tool (more on that below), people will not realise that they have proprietary dependencies down to the n-th level.
Quoted 5 lines
To enforce this, we could
investigate using license-coalescing options of the package managers to find any proprietary dependencies and fail if they're found to be pointing to upstream URLs (and would be captured) with an error message clearly describing our intentions.
But that could also be a totally valid setup, where one would not care about that situation, if one builds a device for their own use, and that is never ever distributed; those people would not care about the licensing of their dependency hell.
Quoted 6 lines
In my (so far unshared) patchset, my solution to do this agrees with your suggestion above, by adding download-backend support for these package managers. I leveraged the implementation suggested previously by patrick[1] to use the cargo-dl step to then create two tarballs:
Quoted 6 lines
<package>-<ver>.tar.gz <- we're all familiar with
<package>-<ver>-vendor.tar.gz <- the TPIP portionThe reasons for splitting are shared above. I believe that patchset is a good initial direction, and I think those interested in this patchset who are unfamiliar with that one should give it a review.
Well, whether we go for one or two archives, I think posting that patchset would anyway be a good first step: indeed, switching to the other solution would be relatively trivial.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Hello,
Quoted 10 lines · On Thu, 3 Sep 2020 15:28:52 +0200 "Yann E. MORIN" <[email protected]>…
And as I explained on IRC (which I am going to repeat here so that it is archived in the list archives, or for all to see as well), I believe this dichotomy between main and vendored archives is incorrect.
What I understand is that, for proprietary applications, you do not want to provide the main tarball, but since those package managers make it so easy to depend on external, FLOSS components, you want to be able to provide those vendored FLOSS stuff to be compliant, without providing the code for your proprietary blurb.
Quoted 8 lines
Consider for example that your company develops two proprietary packages, foo and bar, which have the following dependencies, which will ultimately be vendored in those packages (names totally made up; licenses between brakcets; CORP means proprietary package):
- foo [CORP] depends on libssl [MIT]
- bar [CORP] depends on foo and libhttp [MIT]
In this case, Sam's suggestion is that "foo" should be packaged as a proper Buildroot package, and the "bar" package should use "foo" from the "foo" Buildroot package, rather than using the language-specific vendoring/module system to retrieve it.
In other words, Sam's suggestion is that if you have proprietary / non-redistributable bits of code used as dependencies in your Go/Rust application, you should go and create Buildroot packages for these bits.
Quoted 4 lines
So if you have a separate archive for bar's vendoring, it will include your foo proprietary package, and in the case of 'bar', the separation of archive will not help you properly separate the "TPIP" that you would have to distribute to be complaint.
See above. It was also covered in Sam's proposal.
Quoted 10 lines
Quoted 6 lines
I took a look at
go mod, it seems to share a similar mechanism with cargo which allows us to pass local paths for dependencies. My proposition is to put the responsibility of whomever is adding a proprietary application, which has mixed dependencies, to instead split any proprietary modules into selectable options in buildroot, and use the standard depends mechanism.This is doomed to not work, because it relies on process. And since this is not enforce by the tool (more on that below), people will not realise that they have proprietary dependencies down to the n-th level.
What alternative do you suggest ?
Quoted 10 lines
Quoted 5 lines
To enforce this, we could
investigate using license-coalescing options of the package managers to find any proprietary dependencies and fail if they're found to be pointing to upstream URLs (and would be captured) with an error message clearly describing our intentions.But that could also be a totally valid setup, where one would not care about that situation, if one builds a device for their own use, and that is never ever distributed; those people would not care about the licensing of their dependency hell.
Again, what do you suggest ?
I agree it's not ideal, but it is relatively reasonable, and in the absence of other clear solution and proposal, we're likely to opt for a "relatively reasonable" solution, even if not ideal.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Thomas, All,
On 2020-09-03 16:02 +0200, Thomas Petazzoni spake thusly:
Quoted 21 lines
Quoted 10 lines · On Thu, 3 Sep 2020 15:28:52 +0200 "Yann E. MORIN" <[email protected]>…
And as I explained on IRC (which I am going to repeat here so that it is archived in the list archives, or for all to see as well), I believe this dichotomy between main and vendored archives is incorrect.
What I understand is that, for proprietary applications, you do not want to provide the main tarball, but since those package managers make it so easy to depend on external, FLOSS components, you want to be able to provide those vendored FLOSS stuff to be compliant, without providing the code for your proprietary blurb.
Quoted 8 lines
Consider for example that your company develops two proprietary packages, foo and bar, which have the following dependencies, which will ultimately be vendored in those packages (names totally made up; licenses between brakcets; CORP means proprietary package):
- foo [CORP] depends on libssl [MIT]
- bar [CORP] depends on foo and libhttp [MIT]
Quoted 8 lines
In this case, Sam's suggestion is that "foo" should be packaged as a proper Buildroot package, and the "bar" package should use "foo" from the "foo" Buildroot package, rather than using the language-specific vendoring/module system to retrieve it.
In other words, Sam's suggestion is that if you have proprietary / non-redistributable bits of code used as dependencies in your Go/Rust application, you should go and create Buildroot packages for these bits.
But that only ever works if the bar package is only developped for use by Buildroot, and is not shared with other projects in $BIG_CORP.
The situation is not uncommon that a package is developped in some department (that has no idea what Buildroot is or even that it exists) and be re-used in anotehr department, which uses Buildroot and will be very sad that the package does not folow Buildroot packaging principles.
Quoted 10 lines
Quoted 9 lines
Quoted 6 lines
I took a look at
go mod, it seems to share a similar mechanism with cargo which allows us to pass local paths for dependencies. My proposition is to put the responsibility of whomever is adding a proprietary application, which has mixed dependencies, to instead split any proprietary modules into selectable options in buildroot, and use the standard depends mechanism.This is doomed to not work, because it relies on process. And since this is not enforce by the tool (more on that below), people will not realise that they have proprietary dependencies down to the n-th level.
What alternative do you suggest ?
That we do not create separate archives, at all. Vendored stuf is like any other bundled dependencies: they are part of the source archive.
Quoted 11 lines
Quoted 10 lines
Quoted 5 lines
To enforce this, we could
investigate using license-coalescing options of the package managers to find any proprietary dependencies and fail if they're found to be pointing to upstream URLs (and would be captured) with an error message clearly describing our intentions.But that could also be a totally valid setup, where one would not care about that situation, if one builds a device for their own use, and that is never ever distributed; those people would not care about the licensing of their dependency hell.
Quoted 5 lines
Again, what do you suggest ?
I agree it's not ideal, but it is relatively reasonable, and in the absence of other clear solution and proposal, we're likely to opt for a "relatively reasonable" solution, even if not ideal.
The best and simplest solution is to not create separate archives. That is a reasonable solution.
At least as a first step.
If, and I say "if", the need arises and *all* cases are properly covered, then we can further the infra to do the split (but I have a lot of doubts about the relevance of that).
However, there are a few things I forgot about in my previous comments (but the first two probably the same issue, just at two different places):
- at extract step, how do you know that you have to also extract the archive with the vendor stuff? Probably easy by looking if said archive exists. But then, if each dependency is stored in its own archive, how do you know which to extract? And then, if two deps at different levels have a depednency on another package, but at different versions?
- when you generate the legal-info/ directory, how do you know what to put in there for that package? You are back to the problem above, plus you would also want to ignore those vendored deps that are not redistributable, although we have no way in Buildroot to describe that either....
- why would we impose the policy that vendoring dependencies on other proprietary packages must use Buildroot packages (see the rationale I also gave, above, about code shared across multiple departments of a single company), but dependencies on FLOSS (or even other proprietary but redistributable packages) would be OK?
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Quoted 9 lines · On Thu, 3 Sep 2020 17:12:21 +0200 "Yann E. MORIN" <[email protected]>…
Quoted 8 lines
In this case, Sam's suggestion is that "foo" should be packaged as a proper Buildroot package, and the "bar" package should use "foo" from the "foo" Buildroot package, rather than using the language-specific vendoring/module system to retrieve it.
In other words, Sam's suggestion is that if you have proprietary / non-redistributable bits of code used as dependencies in your Go/Rust application, you should go and create Buildroot packages for these bits.
Quoted 7 lines
But that only ever works if the bar package is only developped for use by Buildroot, and is not shared with other projects in $BIG_CORP.
The situation is not uncommon that a package is developped in some department (that has no idea what Buildroot is or even that it exists) and be re-used in anotehr department, which uses Buildroot and will be very sad that the package does not folow Buildroot packaging principles.
What you mean is that "bar" (which needs "foo") would have to not use the standard Go/Cargo vendoring model to describe its dependency on "foo" so that the Go/Cargo vendoring tools don't fetch "foo". Correct ?
If yes, then indeed that's a fair point.
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
I took a look at
go mod, it seems to share a similar mechanism with cargo which allows us to pass local paths for dependencies. My proposition is to put the responsibility of whomever is adding a proprietary application, which has mixed dependencies, to instead split any proprietary modules into selectable options in buildroot, and use the standard depends mechanism.
Quoted 7 lines
Quoted 4 lines
Quoted 3 lines
This is doomed to not work, because it relies on process. And since this is not enforce by the tool (more on that below), people will not realise that they have proprietary dependencies down to the n-th level.
What alternative do you suggest ?
That we do not create separate archives, at all. Vendored stuf is like any other bundled dependencies: they are part of the source archive.
But then what is your proposal to handle this mixture of licensing/redistributability conditions between the core package source and its dependencies ?
Quoted 4 lines
If, and I say "if", the need arises and *all* cases are properly covered, then we can further the infra to do the split (but I have a lot of doubts about the relevance of that).
Quoted 8 lines
However, there are a few things I forgot about in my previous comments (but the first two probably the same issue, just at two different places):
- at extract step, how do you know that you have to also extract the archive with the vendor stuff? Probably easy by looking if said archive exists. But then, if each dependency is stored in its own archive, how do you know which to extract?
You would parse the go.mod file I suppose, but that doesn't give you indirect dependencies. Perhaps some Go tool can help with that ? But indeed, that's a good question.
Note that storing each dependency as a separate tarball was an idea I have thrown on the table, not something that was proposed by Sam.
Quoted 3 lines
And then, if two deps at
different levels have a depednency on another package, but at different versions?
I am not following you on this one.
Quoted 5 lines
- when you generate the legal-info/ directory, how do you know what to put in there for that package? You are back to the problem above, plus you would also want to ignore those vendored deps that are not redistributable, although we have no way in Buildroot to describe that either....
Yes, we would certainly need to extend somehow the LICENSE / LICENSE_FILES stuff to be able to distinguish which license / license files applies to what.
Quoted 5 lines
- why would we impose the policy that vendoring dependencies on other proprietary packages must use Buildroot packages (see the rationale I also gave, above, about code shared across multiple departments of a single company), but dependencies on FLOSS (or even other proprietary but redistributable packages) would be OK?
Well, this I can answer: the reason why we don't create one Buildroot package per Cargo or Go package is because there are zillions of such FLOSS packages. So we take the lazy route and use the Cargo/Go dependency management system. But since we have a problem with how to /not/ redistribute the few Proprietary apps/libs, an idea was to package them separately. So the reason is really: probably zillions of FLOSS dependencies, while just a handful of internal/proprietary dependencies.
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Thomas, All,
On 2020-09-03 18:13 +0200, Thomas Petazzoni spake thusly:
Quoted 2 lines
On Thu, 3 Sep 2020 17:12:21 +0200
"Yann E. MORIN" <[email protected]> wrote:
[--SNIP--]
Quoted 8 lines
Quoted 4 lines
The situation is not uncommon that a package is developped in some department (that has no idea what Buildroot is or even that it exists) and be re-used in anotehr department, which uses Buildroot and will be very sad that the package does not folow Buildroot packaging principles.
What you mean is that "bar" (which needs "foo") would have to not use the standard Go/Cargo vendoring model to describe its dependency on "foo" so that the Go/Cargo vendoring tools don't fetch "foo". Correct ?
Something the go/cargo/npm/composer/... folkds would have to answer. And if at all, I;d like that we get a consistent behaviour across all those package managers that have vendoring of dependencies. Indeed, it would be very disapointing to have one support it and nto the others. I.e. if we can't do separate tarball for one of them, we should not do for the others...
[--SNIP--]
Quoted 5 lines
Quoted 2 lines
That we do not create separate archives, at all. Vendored stuf is like any other bundled dependencies: they are part of the source archive.
But then what is your proposal to handle this mixture of licensing/redistributability conditions between the core package source and its dependencies ?
How do we handle it in other proprietary packages? We don't.
For example, let's say that Joe Developer Jr. in BIG CORP copies some LGPL-licensed code into their proprietary project, as a bundled library.
This is totally legit: the proprietary parts are not tainted (I hate that word, but have not better for now) by the LGPL stuff. Yet, they have to redistribute the LGPL stuff.
Buildroot currently offers noway for them to do that.
Vendoring is the same as bundling, except it happens at download time on our side, not on the upstream developper's machine.
Quoted 8 lines
Quoted 8 lines
If, and I say "if", the need arises and *all* cases are properly covered, then we can further the infra to do the split (but I have a lot of doubts about the relevance of that).
However, there are a few things I forgot about in my previous comments (but the first two probably the same issue, just at two different places):
Quoted 8 lines
Quoted 4 lines
- at extract step, how do you know that you have to also extract the archive with the vendor stuff? Probably easy by looking if said archive exists. But then, if each dependency is stored in its own archive, how do you know which to extract?
You would parse the go.mod file I suppose, but that doesn't give you indirect dependencies. Perhaps some Go tool can help with that ? But indeed, that's a good question.
And what about cargo? npm? php composer? Others? (what, there are others? ;-] )
Quoted 2 lines
Note that storing each dependency as a separate tarball was an idea I have thrown on the table, not something that was proposed by Sam.
Yes, but one or many, it does not change much...
Quoted 4 lines
Quoted 3 lines
And then, if two deps at
different levels have a depednency on another package, but at different versions?I am not following you on this one.
- foo vendors libssl version 42.27
- foo vendors libhttp version 1.2.3
- libhttp 1.2.3 vendors libssl 666.777
What version of libssl do we extract where?
I do not want to have to repeat the vendoring logic in Buildroot.
Also, I do not want that we have various level of vendoring support for the various package managers.
Quoted 9 lines
Quoted 5 lines
- when you generate the legal-info/ directory, how do you know what to put in there for that package? You are back to the problem above, plus you would also want to ignore those vendored deps that are not redistributable, although we have no way in Buildroot to describe that either....
Yes, we would certainly need to extend somehow the LICENSE /
LICENSE_FILESstuff to be able to distinguish which license / license files applies to what.
As I understand it, in fact, the problem is just about legal-info.
People do not want to redistribute their proprietary packages and the proprietary bits it vendors, but they want to redistribute the FLOSS bits they vendor.
So, if we jut concentrate on how we can help people do exactly that:
filter out the bits they do not want to redistribute?
One solution would be to have packages provide some legal-inf hooks, something like (e.g.: only keep files which names match the regexp):
FOO_LEGAL_INFO_FILTER_REGEXP = ^vendor/FLOSS/
Or whatever, that would be applied at the time the legal-info is generated.
Thoughts?
Note: I find that awfull, but less so than the split archives. Also, it is generic, so it can also be used by other packages like the hypotetical Joe Developer Jr. case explained above.
Quoted 6 lines
Quoted 5 lines
- why would we impose the policy that vendoring dependencies on other proprietary packages must use Buildroot packages (see the rationale I also gave, above, about code shared across multiple departments of a single company), but dependencies on FLOSS (or even other proprietary but redistributable packages) would be OK?
Quoted 8 lines
Well, this I can answer: the reason why we don't create one Buildroot package per Cargo or Go package is because there are zillions of such FLOSS packages. So we take the lazy route and use the Cargo/Go dependency management system. But since we have a problem with how to /not/ redistribute the few Proprietary apps/libs, an idea was to package them separately. So the reason is really: probably zillions of FLOSS dependencies, while just a handful of internal/proprietary dependencies.
Paint me unconvinced.
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Yann,
Quoted 6 lines · Yann E. MORIN
Something the go/cargo/npm/composer/... folkds would have to answer. And if at all, I;d like that we get a consistent behaviour across all those package managers that have vendoring of dependencies. Indeed, it would be very disapointing to have one support it and nto the others. I.e. if we can't do separate tarball for one of them, we should not do for the others...
I do not see in any way how this would not be possible, even for npm, you can quite easily query the package dependency graph and set things up properly.
Quoted 8 lines
For example, let's say that Joe Developer Jr. in BIG CORP copies some LGPL-licensed code into their proprietary project, as a bundled library.
This is totally legit: the proprietary parts are not tainted (I hate that word, but have not better for now) by the LGPL stuff. Yet, they have to redistribute the LGPL stuff.
Buildroot currently offers noway for them to do that.
In what world would a Buildroot package ever be added as an in-tree package with a proprietary library *copied into the source code tree* ??
Quoted 2 lines
Vendoring is the same as bundling, except it happens at download time on our side, not on the upstream developper's machine.
We're enforcing hash checks on these bundles. The format may not always be the same across versions. Storing the source code before it's extracted into a vendor tree is the only way to be sure that the hashes won't change between iterations of the package manager. It's also the only way to redistribute the source code packages for the libraries independently from the proprietary part, maintaining the hashes. It's the only way to deduplicate downloads of identical package versions, to do LICENSE checks on dependencies, etc etc etc.
I don't see an alternative option here. Just running the package manager and compressing the result is less convincing than writing some code to properly understand each dependency.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
- at extract step, how do you know that you have to also extract the archive with the vendor stuff? Probably easy by looking if said archive exists. But then, if each dependency is stored in its own archive, how do you know which to extract?
- Extract the main package
- Check the package.json or go.mod or cargo or whatever
- Extract the relevant stuff into a format the package manager understands
- Run the package manager from the language to assemble the "vendor"
tree in the source dir (maybe same step).
Quoted 3 lines
Quoted 3 lines
You would parse the go.mod file I suppose, but that doesn't give you indirect dependencies. Perhaps some Go tool can help with that ? But indeed, that's a good question.
?? go.mod handles indirect dependencies.
Quoted 2 lines
And what about cargo? npm? php composer? Others? (what, there are others? ;-] )
package.lock yarn.lock. Require it.
Quoted 4 lines
Quoted 2 lines
Note that storing each dependency as a separate tarball was an idea I have thrown on the table, not something that was proposed by Sam.
Yes, but one or many, it does not change much...
It changes a lot! As described above.
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
And then, if two deps at
different levels have a depednency on another package, but at different versions?I am not following you on this one.
Quoted 7 lines
- foo vendors libssl version 42.27
- foo vendors libhttp version 1.2.3
- libhttp 1.2.3 vendors libssl 666.777What version of libssl do we extract where?
I do not want to have to repeat the vendoring logic in Buildroot.
Why repeat it? Re-use it from the programming language! Not everything has to be in bash.
Quoted 2 lines
Also, I do not want that we have various level of vendoring support for the various package managers.
OK, so we implement it across the board, which language would not be able to support this?
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
- when you generate the legal-info/ directory, how do you know what to put in there for that package? You are back to the problem above, plus you would also want to ignore those vendored deps that are not redistributable, although we have no way in Buildroot to describe that either....
Use the license field in the package.json or wherever the specifiers exist, and if they aren't there, detect common LICENSE file names, if you can't find anything, fail.
Go has a few very robust license detector packages. (if desired).
Quoted 10 lines
So, if we jut concentrate on how we can help people do exactly that:
filter out the bits they do not want to redistribute?One solution would be to have packages provide some legal-inf hooks, something like (e.g.: only keep files which names match the regexp):
FOO_LEGAL_INFO_FILTER_REGEXP= ^vendor/FLOSS/Or whatever, that would be applied at the time the legal-info is generated.
How does this solve the problem? If I need to give the source tarballs away for dependencies, and it's all mixed into one massive tarball, you can't separate things out and keep the hashes the same
I thought the requirement was that you would be able to send someone the buildroot "dl" directory and be able to perform a build without network fetches.
Quoted 1 line
Paint me unconvinced.
What's the alternative?
Quoted 2 lines
Regards,
Yann E. MORIN.
best regards,
Christian Stewart
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Christian, All,
On 2020-09-03 12:40 -0700, Christian Stewart spake thusly:
Quoted 8 lines · Yann E. MORIN
Quoted 7 lines
For example, let's say that Joe Developer Jr. in BIG CORP copies some LGPL-licensed code into their proprietary project, as a bundled library.
This is totally legit: the proprietary parts are not tainted (I hate that word, but have not better for now) by the LGPL stuff. Yet, they have to redistribute the LGPL stuff.
Quoted 5 lines
Quoted 1 line
Buildroot currently offers noway for them to do that.
In what world would a Buildroot package ever be added as an in-tree package with a proprietary library *copied into the source code tree* ??
I never talked about upstream Buildroot.
But consider that people would take Buildroot, put it in git in their internal git server, and modify it by adding local packages to it. Or they could also use a br2-external tree.
The packages at stake here are non-public packages that people write as part of their day-time job in their company. It is totally possible that someone belives it would be "easier" to have the source of a dependency bundled in their proprietary package.
Which is *exactly* the case about a proprietary package vendoring a set of external libraries.
Quoted 7 lines
Quoted 2 lines
Vendoring is the same as bundling, except it happens at download time on our side, not on the upstream developper's machine.
We're enforcing hash checks on these bundles. The format may not always be the same across versions. Storing the source code before it's extracted into a vendor tree is the only way to be sure that the hashes won't change between iterations of the package manager.
If a package vendors unversioned dependencies, then indeed we can't add hashes for that package, because two builds may get different stuff for each such vendored dependency, like we don;t add hashes for stuff we know is not reproducible (bzr, cvs, hg for example).
Quoted 3 lines
It's
also the only way to redistribute the source code packages for the libraries independently from the proprietary part,
Except as I explained, it does not work in case the dependencies have dependencies to other proprietary packages, at an arbitrary depth...
Quoted 1 line
maintaining the hashes.
With my proposal, it would not be: there would be a single archive, for which we have hashes. Then when we call legal-info, the package filter is applied to generate a new archive in legal-info, which only contains the filterd files.
And in the output of legal-info, we do not store the hahes from the infra, we calculate the hashes already:
https://git.buildroot.org/buildroot/tree/Makefile#n870
... so we do not need to have hashes of download archives match the legal-info archives.
Quoted 2 lines
It's the only way to deduplicate downloads of identical package versions, to do LICENSE checks on dependencies, etc etc etc.
That would not de-duplicate, because the separate archives would end up in $(FOO_DL_DIR), which is $(BR2_DL_DIR)/foo/ anyway.
Quoted 3 lines
I don't see an alternative option here. Just running the package manager and compressing the result is less convincing than writing some code to properly understand each dependency.
Just running the package manager and compressing the result is the easiest and simplest solution, that will work reliably and consistently across all cases.
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
- at extract step, how do you know that you have to also extract the archive with the vendor stuff? Probably easy by looking if said archive exists. But then, if each dependency is stored in its own archive, how do you know which to extract?
- Extract the main package
- Check the package.json or go.mod or cargo or whatever
- Extract the relevant stuff into a format the package manager understands
This is what I mean by "reinventing the logic of the package managers". Because this one go.mod would refer to dependencies, that my have their won dependencies, so we'd have to look at the go.mod of teach dependencies, recursively... Well, I think the top-level go.mod has all the info, but what of the others?
Quoted 2 lines
- Run the package manager from the language to assemble the "vendor"
tree in the source dir (maybe same step).
go mod vendor
And that is all, I don't even need to look at go.mod and parse it to know where to extract things; or even where to download them from.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
You would parse the go.mod file I suppose, but that doesn't give you indirect dependencies. Perhaps some Go tool can help with that ? But indeed, that's a good question.
?? go.mod handles indirect dependencies.
Quoted 2 lines
And what about cargo? npm? php composer? Others? (what, there are others? ;-] )
package.lock yarn.lock. Require it.
I guess package.log is for npm. No idea what yarn is. Still, that's only two out of at least three...
Quoted 3 lines
Quoted 1 line
I do not want to have to repeat the vendoring logic in Buildroot.
Why repeat it? Re-use it from the programming language! Not everything has to be in bash.
It's not about the language; it's about the logic.
Quoted 4 lines
Quoted 2 lines
Also, I do not want that we have various level of vendoring support for the various package managers.
OK, so we implement it across the board, which language would not be able to support this?
npm is noptorious for having very bad behaviour wrt vendoring dependencies, for example (in my limited suffering^Wexperience packaging npm stuff, I have to admit)
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
- when you generate the legal-info/ directory, how do you know what to put in there for that package? You are back to the problem above, plus you would also want to ignore those vendored deps that are not redistributable, although we have no way in Buildroot to describe that either....
Use the license field in the package.json or wherever the specifiers exist, and if they aren't there, detect common LICENSE file names, if you can't find anything, fail.
How do we know that such or such vendored depednency has to be redsitributed?
But is license "(C) BIG CORP" a redistributable license or not?
Quoted 1 line
Go has a few very robust license detector packages. (if desired).
It is not only about detecting the license (which is however a very important step, indeed)m but it is about deciding whether to redistribute it or not.
If we assume that all vendored stuff is only FLOSS and can only be FLOSS, then that is OK: we redistribute everything that is vendored.
But that is not the cae: if a proprietary package vendors another proprietary package, how do we know that we should not redistribute that second package as well? Knowing the license name is *not* enough to decide; only a human can tell.
Quoted 6 lines
Quoted 6 lines
So, if we jut concentrate on how we can help people do exactly that:
filter out the bits they do not want to redistribute?One solution would be to have packages provide some legal-inf hooks, something like (e.g.: only keep files which names match the regexp):
Quoted 8 lines
Quoted 4 lines
FOO_LEGAL_INFO_FILTER_REGEXP= ^vendor/FLOSS/Or whatever, that would be applied at the time the legal-info is generated.
How does this solve the problem? If I need to give the source tarballs away for dependencies, and it's all mixed into one massive tarball, you can't separate things out and keep the hashes the same
It solves the problem that the legal-info/ directory only contains what you accept to redistribute.
Quoted 3 lines
I thought the requirement was that you would be able to send someone the buildroot "dl" directory and be able to perform a build without network fetches.
Wait, you are confusing the two: the content of dl/ which is used at build time, and from which we extract the sources that are built, and the content of legal-info/ which contains what you should provide to be in compliance with licenses terms.
Quoted 2 lines
Quoted 1 line
Paint me unconvinced.
What's the alternative?
Please re-review my proposal: the content of dl/ would always contains everything unmolested. It is only when calling 'make legal-info' that the filtering would be applied, and a new archive would be genrated with only the filter (or filtered-out) content. I.e. basically:
$ make legal-info
for pkg in PACKAGES:
if pkg.FOO_LEGAL_INFO_FILTER_REGEXP is not set:
copy dl/foo-version.tar.gz to legal-info/foo-version/foo-version.tar.gz
continue
extract dl/foo-version.tar.gz \
into temp-dir/ \
if file matches pkg.FOO_LEGAL_INFO_FILTER_REGEXP
create legal-info/foo-version/foo-version.tar.gz \
from temp-dir/Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Hi Yann,
Quoted 10 lines · Yann E. MORIN
On 2020-09-03 12:40 -0700, Christian Stewart spake thusly:
Quoted 3 lines
In what world would a Buildroot package ever be added as an in-tree package with a proprietary library *copied into the source code tree* ??
I never talked about upstream Buildroot.
But consider that people would take Buildroot, put it in git in their internal git server, and modify it by adding local packages to it. Or they could also use a br2-external tree.
People are going to take Buildroot and store a private copy of it on their internal server, make modifications?
All of this on a GPLv2 licensed project? I thought this wasn't legal?
For br2-external packages you can relax the LICENSE requirement.
Quoted 4 lines
The packages at stake here are non-public packages that people write as part of their day-time job in their company. It is totally possible that someone belives it would be "easier" to have the source of a dependency bundled in their proprietary package.
That's fine, but that would go into br2-external as you've said.
Quoted 2 lines
Which is *exactly* the case about a proprietary package vendoring a set of external libraries.
If proprietary package is importing some external libraries that may be permissively licensed, even requiring redistribution in source form, without the proprietary section - how do you redistribute those dependencies separately?
Quoted 9 lines
Quoted 4 lines
We're enforcing hash checks on these bundles. The format may not always be the same across versions. Storing the source code before it's extracted into a vendor tree is the only way to be sure that the hashes won't change between iterations of the package manager.
If a package vendors unversioned dependencies, then indeed we can't add hashes for that package, because two builds may get different stuff for each such vendored dependency, like we don;t add hashes for stuff we know is not reproducible (bzr, cvs, hg for example).
I don't understand what you're saying here. It should not be possible to have the package manager bring in arbitrary dependencies at build time. Buildroot builds are meant to produce the same output every time, right?
Quoted 6 lines
Quoted 3 lines
It's
also the only way to redistribute the source code packages for the libraries independently from the proprietary part,Except as I explained, it does not work in case the dependencies have dependencies to other proprietary packages, at an arbitrary depth...
I don't understand what you're saying here.
Package A (in buildroot) imports package B. Package B imports proprietary package C.
Result: three tarballs, package-a-1.0.1.tar.gz, package-b-0.0.5.tar.gz, package-c-fooversion.tar.gz.
Quoted 4 lines
With my proposal, it would not be: there would be a single archive, for which we have hashes. Then when we call legal-info, the package filter is applied to generate a new archive in legal-info, which only contains the filterd files.
Yes this is simpler but it won't work in every case. The vendor tree or the node_modules tree might have some minor things changed about it which will break the hash. Node-modules also often contains symlinks and binaries prepared for the particular host build system.
Quoted 7 lines
And in the output of legal-info, we do not store the hahes from the infra, we calculate the hashes already:
https://git.buildroot.org/buildroot/tree/Makefile#n870
... so we do not need to have hashes of download archives match the legal-info archives.
I don't agree that legal is the only thing that matters here, you also want to be sure that you'll have a Buildroot build that works every time, without internet access, if you have a full "make download" pass run in advance.
Quoted 5 lines
Quoted 2 lines
It's the only way to deduplicate downloads of identical package versions, to do LICENSE checks on dependencies, etc etc etc.
That would not de-duplicate, because the separate archives would end up in $(FOO_DL_DIR), which is $(BR2_DL_DIR)/foo/ anyway.
I don't understand what you're saying here. If I download package-c dependency at 1.0.4 it will be under - for example - $(BR2_DL_DIR)/go-modules/package-c/package-c-1.0.4.tar.gz. The deduplication is for package dependencies with identical versions and identical source URLs.
Quoted 3 lines
Just running the package manager and compressing the result is the easiest and simplest solution, that will work reliably and consistently across all cases.
I don't agree, there are tons of cases where simply compressing the result after running "npm install" or "go mod vendor" will not necessarily work.
You're also going to need to download tons of dependencies for features of the program that you may not even have enabled in your Buildroot config.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
- at extract step, how do you know that you have to also extract the archive with the vendor stuff? Probably easy by looking if said archive exists. But then, if each dependency is stored in its own archive, how do you know which to extract?
- Extract the main package
- Check the package.json or go.mod or cargo or whatever
- Extract the relevant stuff into a format the package manager understands
Quoted 5 lines
This is what I mean by "reinventing the logic of the package managers". Because this one go.mod would refer to dependencies, that my have their won dependencies, so we'd have to look at the go.mod of teach dependencies, recursively... Well, I think the top-level go.mod has all the info, but what of the others?
This is already implemented as a library in Go. You don't have to re-do it from scratch.
https://pkg.go.dev/golang.org/x/tools/go/packages?tab=doc
The top-level go.mod and go.sum have all information on transient and indirect dependencies.
Quoted 7 lines
Quoted 2 lines
- Run the package manager from the language to assemble the "vendor"
tree in the source dir (maybe same step).go mod vendor
And that is all, I don't even need to look at go.mod and parse it to know where to extract things; or even where to download them from.
And how is this better than running a Go program which understands how to download dependencies into the .tar.gz format that we expect, and to fetch them back again from that format into the Go module cache, and then the vendor/ tree?
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
You would parse the go.mod file I suppose, but that doesn't give you indirect dependencies. Perhaps some Go tool can help with that ? But indeed, that's a good question.
?? go.mod handles indirect dependencies.
Quoted 2 lines
And what about cargo? npm? php composer? Others? (what, there are others? ;-] )
package.lock yarn.lock. Require it.
I guess package.log is for npm. No idea what yarn is. Still, that's only two out of at least three...
Are you saying it's not possible to collect an index of indirect dependencies with those?
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I do not want to have to repeat the vendoring logic in Buildroot.
Why repeat it? Re-use it from the programming language! Not everything has to be in bash.
It's not about the language; it's about the logic.
I don't understand what you mean.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Also, I do not want that we have various level of vendoring support for the various package managers.
OK, so we implement it across the board, which language would not be able to support this?
npm is noptorious for having very bad behaviour wrt vendoring dependencies, for example (in my limited suffering^Wexperience packaging npm stuff, I have to admit)
And this is exactly why compressing the node_modules is not enough.
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
- when you generate the legal-info/ directory, how do you know what to put in there for that package? You are back to the problem above, plus you would also want to ignore those vendored deps that are not redistributable, although we have no way in Buildroot to describe that either....
Quoted 8 lines
Quoted 3 lines
Use the license field in the package.json or wherever the specifiers exist, and if they aren't there, detect common LICENSE file names, if you can't find anything, fail.
How do we know that such or such vendored depednency has to be redsitributed?
But is license "(C) BIG CORP" a redistributable license or not?
If you run "make source" it collects source for everything to produce the build, correct?
So, in this case we would collect everything needed for the build, scan LICENSE, if the package is in the Buildroot tree, fail if we don't recognize all the LICENSE files (allowing for manual override of course), and if it's in buildroot-ext, assume anything without LICENSE or with an unrecognized LICENSE is not redistributable and show a warning.
You wouldn't put anything proprietary into Buildroot proper since it's a GPLv2 project. It would be a extension package.
Quoted 6 lines
Quoted 1 line
Go has a few very robust license detector packages. (if desired).
It is not only about detecting the license (which is however a very important step, indeed)m but it is about deciding whether to redistribute it or not.
Quoted 7 lines
If we assume that all vendored stuff is only FLOSS and can only be FLOSS, then that is OK: we redistribute everything that is vendored.
But that is not the cae: if a proprietary package vendors another proprietary package, how do we know that we should not redistribute that second package as well? Knowing the license name is *not* enough to decide; only a human can tell.
OK, so you put a manual override to block things from being included in the redistributable... I still don't see how compressing the entire source tree after "npm install" or "go mod vendor" would address this, in this case you're going to unconditionally include all proprietary code into that package and redistribute it no matter what.
Quoted 11 lines
Quoted 11 lines
Quoted 10 lines
So, if we jut concentrate on how we can help people do exactly that:
filter out the bits they do not want to redistribute?One solution would be to have packages provide some legal-inf hooks, something like (e.g.: only keep files which names match the regexp):
FOO_LEGAL_INFO_FILTER_REGEXP= ^vendor/FLOSS/Or whatever, that would be applied at the time the legal-info is generated.
Quoted 6 lines
Quoted 3 lines
How does this solve the problem? If I need to give the source tarballs away for dependencies, and it's all mixed into one massive tarball, you can't separate things out and keep the hashes the same
It solves the problem that the legal-info/ directory only contains what you accept to redistribute.
This makes sense, probably for Go this would be better as a regexp on the package import path - for example, ^github.com/myprivateorg/ would deny all of the myprivateorg packages from inclusion in legal-info, this is quite similar to the GOPRIVATE variable in golang.
Quoted 8 lines
Quoted 3 lines
I thought the requirement was that you would be able to send someone the buildroot "dl" directory and be able to perform a build without network fetches.
Wait, you are confusing the two: the content of dl/ which is used at build time, and from which we extract the sources that are built, and the content of legal-info/ which contains what you should provide to be in compliance with licenses terms.
I agree, I think these are two things that are being mixed here.
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
Paint me unconvinced.
What's the alternative?
Please re-review my proposal: the content of dl/ would always contains everything unmolested. It is only when calling 'make legal-info' that the filtering would be applied, and a new archive would be genrated with only the filter (or filtered-out) content. I.e. basically:
Quoted 10 lines
$ make legal-info
for pkg in PACKAGES:
if pkg.FOO_LEGAL_INFO_FILTER_REGEXP is not set:
copy dl/foo-version.tar.gz to legal-info/foo-version/foo-version.tar.gz
continue
extract dl/foo-version.tar.gz \
into temp-dir/ \
if file matches pkg.FOO_LEGAL_INFO_FILTER_REGEXP
create legal-info/foo-version/foo-version.tar.gz \
from temp-dir/This handles legal-info but not "make download" or "make extract" or "make source".
Probably best approach is some combination of the two?
Best regards,
Christian
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Christian, All,
On 2020-09-03 14:47 -0700, Christian Stewart spake thusly:
Quoted 2 lines · Yann E. MORIN
Quoted 1 line
On 2020-09-03 12:40 -0700, Christian Stewart spake thusly:
[--SNIP--]
Quoted 4 lines
People are going to take Buildroot and store a private copy of it on their internal server, make modifications?
All of this on a GPLv2 licensed project? I thought this wasn't legal?
Of course this is totally legit. You do it all the time when you giot clone toyour local machine and do modifications, for example.
Quoted 1 line
For br2-external packages you can relax the LICENSE requirement.
But in the Buildroot infra, we do not treat packages from br2-external differently from in-tree packages.
Quoted 6 lines
Quoted 4 lines
The packages at stake here are non-public packages that people write as part of their day-time job in their company. It is totally possible that someone belives it would be "easier" to have the source of a dependency bundled in their proprietary package.
That's fine, but that would go into br2-external as you've said.
No.
Quoted 7 lines
Quoted 2 lines
Which is *exactly* the case about a proprietary package vendoring a set of external libraries.
If proprietary package is importing some external libraries that may be permissively licensed, even requiring redistribution in source form, without the proprietary section - how do you redistribute those dependencies separately?
Redistribution is the role of legal-info, see below how I suggest this is handled.
Quoted 10 lines
Quoted 9 lines
Quoted 4 lines
We're enforcing hash checks on these bundles. The format may not always be the same across versions. Storing the source code before it's extracted into a vendor tree is the only way to be sure that the hashes won't change between iterations of the package manager.
If a package vendors unversioned dependencies, then indeed we can't add hashes for that package, because two builds may get different stuff for each such vendored dependency, like we don;t add hashes for stuff we know is not reproducible (bzr, cvs, hg for example).
Quoted 4 lines
I don't understand what you're saying here. It should not be possible to have the package manager bring in arbitrary dependencies at build time. Buildroot builds are meant to produce the same output every time, right?
For example, dependencies in npm are loose, where yiou can say "I need package bar at version 1.x". So at some point, the 'x' in '1.x' will match the latest '1.1' and use that, but the next day, '1.2' might get released, and the 'x' would match that. So if bar 1.2 brings in a new dependency,or a new version of an existing dependency, two builds do not provide the same output and are thus not reproducible.
See:
https://docs.npmjs.com/files/package.json#dependencies
https://docs.npmjs.com/misc/semver
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
It's
also the only way to redistribute the source code packages for the libraries independently from the proprietary part,Except as I explained, it does not work in case the dependencies have dependencies to other proprietary packages, at an arbitrary depth...
I don't understand what you're saying here.
Package A (in buildroot) imports package B. Package B imports proprietary package C.
That is the other way around: the top-leval package is proprietary, and it imports FLOSS packages:
- foo is proprietary
- foo vendors bar
- bar is proprietary
- bar vendors buz, which is FLOSS (e.g. MIT, LGPL...)
- buz vendors ni
- ni is FLOSS...
- foo vendors doh
- doh is FLOSS...
- doh vendors bla
- bla is FLOSS...
Quoted 8 lines
Quoted 4 lines
With my proposal, it would not be: there would be a single archive, for which we have hashes. Then when we call legal-info, the package filter is applied to generate a new archive in legal-info, which only contains the filterd files.
Yes this is simpler but it won't work in every case. The vendor tree or the
node_modulestree might have some minor things changed about it which will break the hash.
Then if the package manager can not generate reproducible archives, we can't have hashes for it, period.
Quoted 2 lines
Node-modules also often contains symlinks
and binaries prepared for the particular host build system.
But those are created at build time, not at download time, right? Well, node is so awful, that I would not be surprised...
Quoted 5 lines
Quoted 5 lines
And in the output of legal-info, we do not store the hahes from the infra, we calculate the hashes already:
Quoted 7 lines
Quoted 2 lines
... so we do not need to have hashes of download archives match the legal-info archives.
I don't agree that legal is the only thing that matters here, you also want to be sure that you'll have a Buildroot build that works every time, without internet access, if you have a full "make download" pass run in advance.
Exactly the role of the dl/ directory; and that would contain the complete downloaded archives with all the vendored dependencies.
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
It's the only way to deduplicate downloads of identical package versions, to do LICENSE checks on dependencies, etc etc etc.
That would not de-duplicate, because the separate archives would end up in $(FOO_DL_DIR), which is $(BR2_DL_DIR)/foo/ anyway.
I don't understand what you're saying here. If I download package-c dependency at 1.0.4 it will be under - for example - $(BR2_DL_DIR)/go-modules/package-c/package-c-1.0.4.tar.gz. The deduplication is for package dependencies with identical versions and identical source URLs.
Ah, because you store all the go vendored dependencies "away" of the package.
I am not sure I like that, because it breaks the semantics of the dl/
directory: all the csource needed by one package is in dl/foo/ and the vendored dependencies *are* part of the package.
So I am absolutely not in favour of storing the go modules on the side.
Quoted 7 lines
Quoted 3 lines
Just running the package manager and compressing the result is the easiest and simplest solution, that will work reliably and consistently across all cases.
I don't agree, there are tons of cases where simply compressing the result after running "npm install" or "go mod vendor" will not necessarily work.
Why?
Quoted 3 lines
You're also going to need to download tons of dependencies for features of the program that you may not even have enabled in your Buildroot config.
So, when you download the linux kernel, there are tons of code for tons of features you don;t need.
Really, vendoring is, tome, exactly like bundling, except that the aggregation happens on the client side, at download time, rather than on the developpers machine that pushes a pre-aggregated "repository".
Quoted 15 lines
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
- at extract step, how do you know that you have to also extract the archive with the vendor stuff? Probably easy by looking if said archive exists. But then, if each dependency is stored in its own archive, how do you know which to extract?
- Extract the main package
- Check the package.json or go.mod or cargo or whatever
- Extract the relevant stuff into a format the package manager understandsQuoted 5 lines
This is what I mean by "reinventing the logic of the package managers". Because this one go.mod would refer to dependencies, that my have their won dependencies, so we'd have to look at the go.mod of teach dependencies, recursively... Well, I think the top-level go.mod has all the info, but what of the others?
Quoted 4 lines
This is already implemented as a library in Go. You don't have to re-do it from scratch.
But I don;t want we even have to deal with the internals of the go module stuff at all, that's what I'm saying.
I don;t want we have to write a script (in whatever language) that has to deal with the go module internals, to reproduce the logic.
go mod vendor is all that we should ever need to create the archives.
Quoted 2 lines
The top-level go.mod and go.sum have all information on transient and indirect dependencies.
Good for go. Last I had to play wit ^W^W suffer of npm stuff, that was not the case IIRC: a paclkage would only list its first-level dependencies, and the install would recurse into that... And since the versions of depenencies are floating, there is no way you can now what you'd need ahead of time.
So again, I want that we have a consistent handling of all the package managers, that they all behave the same.
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines
- Run the package manager from the language to assemble the "vendor"
tree in the source dir (maybe same step).go mod vendor
Quoted 7 lines
Quoted 2 lines
And that is all, I don't even need to look at go.mod and parse it to know where to extract things; or even where to download them from.
And how is this better than running a Go program which understands how to download dependencies into the .tar.gz format that we expect, and to fetch them back again from that format into the Go module cache, and then the vendor/ tree?
Because writing a go program (or any other language) is duplicating the logic of 'go mod vendor'.
Quoted 4 lines
Quoted 2 lines
I guess package.log is for npm. No idea what yarn is. Still, that's only two out of at least three...
Are you saying it's not possible to collect an index of indirect dependencies with those?
IIRC, for NPM, no. Or not trivially, or not reproducibly.
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
I do not want to have to repeat the vendoring logic in Buildroot.
Why repeat it? Re-use it from the programming language! Not everything has to be in bash.
It's not about the language; it's about the logic.
I don't understand what you mean.
The logic of vendoring.
For example, this could be a logic:
def fill_deps(dep_file):
for dep in file.read(dep_file):
download(dep)
fill_deps(dep.dependencies)
def main():
fill_deps(load_main_deps())I do not want we do that, because it will be a maintenance burden, and is duplicating the logic the package managers are there to cover in the first place!
Quoted 15 lines
Quoted 4 lines
How do we know that such or such vendored depednency has to be redsitributed?
But is license "(C) BIG CORP" a redistributable license or not?
If you run "make source" it collects source for everything to produce the build, correct?
So, in this case we would collect everything needed for the build, scan LICENSE, if the package is in the Buildroot tree, fail if we don't recognize all the LICENSE files (allowing for manual override of course), and if it's in buildroot-ext, assume anything without LICENSE or with an unrecognized LICENSE is not redistributable and show a warning.
Quoted 2 lines
You wouldn't put anything proprietary into Buildroot proper since it's a GPLv2 project. It would be a extension package.
We do have proprietary packages in Buidlroot:
boot/s500-bootloader/
package/armbian-firmware/
package/nvidia-driver/
package/wilc1000-firmware/And quite a few others...
IEANAL and all disclaimers... And no, this is not a violation of the GPLv2 at all: Buildroot is not a derived work of those, not are those a derived work of Buildroot.
(Note: I dropped the rest of the mail because I don't have time to reply to it right now, and I am afraid I would anyway re-hash what I already said...)
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Hi Yann,
Quoted 7 lines · Yann E. MORIN
On 2020-09-03 14:47 -0700, Christian Stewart spake thusly:
Quoted 3 lines · Yann E. MORIN
People are going to take Buildroot and store a private copy of it on their internal server, make modifications?
Of course this is totally legit. You do it all the time when you giot clone toyour local machine and do modifications, for example.
OK.
Quoted 2 lines
Redistribution is the role of legal-info, see below how I suggest this is handled.
This isn't really my domain (legal-info) and what you've suggested - picking the dependencies out of vendor/ - might work for this.
However, how do you know which version of the dependency is coming out of vendor/ ? I suppose you're going to redistribute my-package-vendor.tar.gz with anything proprietary excluded? So ultimately you redistribute 15 copies of the same thing?
Quoted 11 lines
Quoted 4 lines
I don't understand what you're saying here. It should not be possible to have the package manager bring in arbitrary dependencies at build time. Buildroot builds are meant to produce the same output every time, right?
For example, dependencies in npm are loose, where yiou can say "I need package bar at version 1.x". So at some point, the 'x' in '1.x' will match the latest '1.1' and use that, but the next day, '1.2' might get released, and the 'x' would match that. So if bar 1.2 brings in a new dependency,or a new version of an existing dependency, two builds do not provide the same output and are thus not reproducible.
We're really going to add packages to Buildroot which will have fuzzy dependencies and might bring in something different if Npm is having a bad day?
For third party packages, this makes sense - you wouldn't have a hash on it. But for packages in the Buildroot tree you would probably expect a hash + a lock file. The same goes for Go with go.mod and go.sum - without go.sum you can't be sure it will be the same every time, and should not have a hash there.
Even with package-lock.json, the node_modules will not necessarily produce the same hash every time, particularly across different OS versions. So, are you saying we're not going to put hashes on /anything/ that uses npm?
Go Modules is designed around always downloading the exact same dependencies, and our approach for that language can at least be built around that assumption (that it's going to use the go.sum every time to produce the same dependency output for most Buildroot in-tree packages).
Quoted 13 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
It's
also the only way to redistribute the source code packages for the libraries independently from the proprietary part,Except as I explained, it does not work in case the dependencies have dependencies to other proprietary packages, at an arbitrary depth...
Package A (in buildroot) imports package B. Package B imports proprietary package C.
That is the other way around: the top-leval package is proprietary, and it imports FLOSS packages:
Quoted 10 lines
- foo is proprietary
- foo vendors bar
- bar is proprietary
- bar vendors buz, which is FLOSS (e.g. MIT, LGPL...)
- buz vendors ni
- ni is FLOSS...
- foo vendors doh
- doh is FLOSS...
- doh vendors bla
- bla is FLOSS...
foo is proprietary - download to foo-version.tar.gz
foo selects bar - download to bar-version.tar.gz.
bar selects baz - download to baz-version.tar.gz
foo selects baz - we already have it in baz-version.tar.gz.
You can package them separately. Yes, you need to look into the first archive to know what the dependencies are. But, you need to do this with the vendoring anyway.
Quoted 6 lines
Quoted 3 lines
Yes this is simpler but it won't work in every case. The vendor tree or the
node_modulestree might have some minor things changed about it which will break the hash.Then if the package manager can not generate reproducible archives, we can't have hashes for it, period.
No hashes for node_modules across all node_modules packages? :(
Buildroot packages that are actually merged into mainline, I would expect to produce reproducible output and have hashes on their downloaded source code files. It doesn't make sense to have a mainline package which is fetching random stuff due to fuzzy semver specifiers.
For external or third party packages, those wouldn't have source code hashes, and that makes sense.
For Go at least it will always be possible to make a hashed set of source code archives. It's possible to download & compress each dependency independently for Go as well, and analyze the licenses and whatever else.
Quoted 5 lines
Quoted 2 lines
Node-modules also often contains symlinks
and binaries prepared for the particular host build system.But those are created at build time, not at download time, right? Well, node is so awful, that I would not be surprised...
No - download time - the post install hooks are run. For example, electron might download + extract a tarball with Chromium.
Quoted 7 lines
Quoted 4 lines
I don't agree that legal is the only thing that matters here, you also want to be sure that you'll have a Buildroot build that works every time, without internet access, if you have a full "make download" pass run in advance.
Exactly the role of the dl/ directory; and that would contain the complete downloaded archives with all the vendored dependencies.
You want to have a download archive for every Buildroot package which contains ALL the dependencies for the package, and make this the ONLY way to do this?
Even when node_modules won't necessarily work on some machines without running "npm install" again?
The storage space increase alone of storing the entire node_modules for multiple packages across potentially many versions is a reason to at least consider an alternative.
There's a way to split dependency source tarballs properly - even in Node - and even if it was impossible in Node, the limitations of Node shouldn't prevent us from doing this for a language that /can/ support it like Go.
I understand that the goal is to keep things as simple as possible and avoid adding any work for the maintainers, and indeed this makes sense.
Quoted 6 lines
Quoted 5 lines
I don't understand what you're saying here. If I download package-c dependency at 1.0.4 it will be under - for example - $(BR2_DL_DIR)/go-modules/package-c/package-c-1.0.4.tar.gz. The deduplication is for package dependencies with identical versions and identical source URLs.
Quoted 6 lines
Ah, because you store all the go vendored dependencies "away" of the package.
I am not sure I like that, because it breaks the semantics of the dl/
directory: all the csource needed by one package is in dl/foo/ and the vendored dependencies *are* part of the package.
We aren't actually running "go mod vendor" or "npm install" and storing the result in the tarballs (as far as I am aware) today.
Quoted 1 line
So I am absolutely not in favour of storing the go modules on the side.
Just to clarify, so I understand, the reasons not to are ?
- Currently we can say that if you extract a .tar.gz from dl it will
be buildable
- It's too hard to add code to manage dependencies
- Some packages don't use locking and are getting different deps everytime
And you're willing to take the downsides of mixing together all sorts of proprietary and FLOSS code into these .tar.gz download files, making it next to impossible to independently redistribute dependencies from proprietary Go packages (for example?) and increasing the size of the dl/ tree many times over?
Quoted 10 lines
Quoted 3 lines
You're also going to need to download tons of dependencies for features of the program that you may not even have enabled in your Buildroot config.
So, when you download the linux kernel, there are tons of code for tons of features you don;t need.
Really, vendoring is, tome, exactly like bundling, except that the aggregation happens on the client side, at download time, rather than on the developpers machine that pushes a pre-aggregated "repository".
It's entirely avoidable to store duplicate copies of the dependencies for Go programs many times over in vendor/ trees compressed into the .tar.gz with the root project.
Linux being a large project with tons of code you don't build, doesn't necessarily preclude making some alternative when possible.
Quoted 9 lines
Quoted 9 lines
Quoted 5 lines
This is what I mean by "reinventing the logic of the package managers". Because this one go.mod would refer to dependencies, that my have their won dependencies, so we'd have to look at the go.mod of teach dependencies, recursively... Well, I think the top-level go.mod has all the info, but what of the others?
This is already implemented as a library in Go. You don't have to re-do it from scratch.
Quoted 7 lines
But I don;t want we even have to deal with the internals of the go module stuff at all, that's what I'm saying.
I don;t want we have to write a script (in whatever language) that has to deal with the go module internals, to reproduce the logic.
The packages and tools make it easy to do this without getting into the internals, those are very high level APIs there.
I guess the only way to prove this particular point is to just write the scripts as a RFC?
Quoted 8 lines
Quoted 2 lines
The top-level go.mod and go.sum have all information on transient and indirect dependencies.
Good for go. Last I had to play wit ^W^W suffer of npm stuff, that was not the case IIRC: a paclkage would only list its first-level dependencies, and the install would recurse into that... And since the versions of depenencies are floating, there is no way you can now what you'd need ahead of time.
I was not aware that you would ever add a package to Buildroot which uses floating semver selections, and could get a different version between "make" executions. Please remind me to never build or install any of those packages :)
Quoted 2 lines
Because writing a go program (or any other language) is duplicating the logic of 'go mod vendor'.
But we would still be running "go mod vendor" - the idea is to pre-fill the Go modules cache from the Buildroot dl tree, and avoid having tarballs in the dl tree that contain source code from multiple projects simultaneously.
The goal is also to avoid breaking package source code download hashes after upgrading the tool, due to a change in the format of vendor/, to reduce source code tarball sizes, to make it easier to separate out proprietary and FLOSS components, and to ensure that the build is reproducible.
I'll build & submit a RFC prototype so that it's clearer what I'm actually suggesting here.
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines
I guess package.log is for npm. No idea what yarn is. Still, that's only two out of at least three...
Are you saying it's not possible to collect an index of indirect dependencies with those?
IIRC, for NPM, no. Or not trivially, or not reproducibly.
You absolutely can collect a manifest of dependencies reproducibly and easily with the package-lock.json.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
I do not want to have to repeat the vendoring logic in Buildroot.
Why repeat it? Re-use it from the programming language! Not everything has to be in bash.
It's not about the language; it's about the logic.
I don't understand what you mean.
The logic of vendoring.
All of which is implemented in these languages already in a format that is at a very high level and will require little to no "reinventing the wheel" from us. (at least, for Go)
Quoted 11 lines
Quoted 2 lines
You wouldn't put anything proprietary into Buildroot proper since it's a GPLv2 project. It would be a extension package.
We do have proprietary packages in Buidlroot:
boot/s500-bootloader/ package/armbian-firmware/ package/nvidia-driver/ package/wilc1000-firmware/And quite a few others...
OK, I see what you mean by proprietary.
Quoted 3 lines
(Note: I dropped the rest of the mail because I don't have time to reply to it right now, and I am afraid I would anyway re-hash what I already said...)
I'll get back to you all with a RFC patch for the Go approach. I can't speak for node but allowing Node to have unhashed fuzzy semver specifiers in Buildroot, is not something I can recommend, since it seems almost like a security issue.
Best regards,
Christian Stewart
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Hey everybody,
Was a long end of the week, and I finally had a minute to catch back up.
Quoted 1 line · Christian Stewart
[snip]
Quoted 10 lines
Quoted 2 lines
Redistribution is the role of legal-info, see below how I suggest this is handled.
This isn't really my domain (legal-info) and what you've suggested - picking the dependencies out of vendor/ - might work for this.
However, how do you know which version of the dependency is coming out of vendor/ ? I suppose you're going to redistribute my-package-vendor.tar.gz with anything proprietary excluded? So ultimately you redistribute 15 copies of the same thing?
I think that is an unfortunate consequence, yes, that we will have some overlap of released things if two top-level packages which share a similar dep chain. I think for now that should not be the focus of the topic, as it's an unfortunate side-effect disk space is cheap.
Quoted 17 lines
Quoted 6 lines
Quoted 5 lines
I don't understand what you're saying here. It should not be possible to have the package manager bring in arbitrary dependencies at build time. Buildroot builds are meant to produce the same output every time, right?
Quoted 6 lines
For example, dependencies in npm are loose, where yiou can say "I need package bar at version 1.x". So at some point, the 'x' in '1.x' will match the latest '1.1' and use that, but the next day, '1.2' might get released, and the 'x' would match that. So if bar 1.2 brings in a new dependency,or a new version of an existing dependency, two builds do not provide the same output and are thus not reproducible.
We're really going to add packages to Buildroot which will have fuzzy dependencies and might bring in something different if Npm is having a bad day?
Quoted 5 lines
For third party packages, this makes sense - you wouldn't have a hash on it. But for packages in the Buildroot tree you would probably expect a hash + a lock file. The same goes for Go with go.mod and go.sum - without go.sum you can't be sure it will be the same every time, and should not have a hash there.
Cargo+rust also uses a lock file, and the patchset proposed by Patrick uses the "lock" flag to disallow buildroot from "upgrading" any packages on the fly. It must use what is in the lock.
Quoted 6 lines
Even with package-lock.json, the
node_moduleswill not necessarily produce the same hash every time, particularly across different OS versions. So, are you saying we're not going to put hashes on /anything/ that uses npm?
Quoted 15 lines
Go Modules is designed around always downloading the exact same dependencies, and our approach for that language can at least be built around that assumption (that it's going to use the go.sum every time to produce the same dependency output for most Buildroot in-tree packages).
Quoted 9 lines
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
It's
also the only way to redistribute the source code packages for the libraries independently from the proprietary part,Except as I explained, it does not work in case the dependencies have dependencies to other proprietary packages, at an arbitrary depth...
Package A (in buildroot) imports package B. Package B imports proprietary package C.
I'm not sure if this makes sense, in a general sense. I'm not sure how we would anticipate this actually happening, where a FLOSS package grabs something proprietary.
And to that point, maybe proprietary isn't what we should necessarily be directly describing. In my mind it should be considered as non-openly distributable.
Quoted 19 lines
Quoted 4 lines
That is the other way around: the top-leval package is proprietary, and it imports FLOSS packages:
Quoted 10 lines
- foo is proprietary
- foo vendors bar
- bar is proprietary
- bar vendors buz, which is FLOSS (e.g. MIT, LGPL...)
- buz vendors ni
- ni is FLOSS...
- foo vendors doh
- doh is FLOSS...
- doh vendors bla
- bla is FLOSS...foo is proprietary - download to foo-version.tar.gz
foo selects bar - download to bar-version.tar.gz.
Quoted 7 lines
bar selects baz - download to baz-version.tar.gz
foo selects baz - we already have it in baz-version.tar.gz.
You can package them separately. Yes, you need to look into the first archive to know what the dependencies are. But, you need to do this with the vendoring anyway.
I don't think I'm following what you're getting at here - if you mean 'select' at a buildroot-level, I think it makes sense that both foo and bar have vendoring which include baz, because they both need it.
This goes back into what I said before, and while it isn't perfect I think is going to be a fact of our first iteration without making this so convoluted nobody is going to take it on leading to everybody carrying patchsets internally with an RFC thrown out to the mailing list and wishing whoever wants to handle it "good luck".
Quoted 9 lines
Quoted 7 lines
Quoted 4 lines
Yes this is simpler but it won't work in every case. The vendor tree or the
node_modulestree might have some minor things changed about it which will break the hash.Then if the package manager can not generate reproducible archives, we can't have hashes for it, period.
No hashes for
node_modulesacross allnode_modulespackages? :(
To my understanding, a lot of node is kind of "wild west" still, we had to pull stuff in through the "install node packages" string this year, and handle all vendoring ourselves.
Quoted 35 lines
Buildroot packages that are actually merged into mainline, I would expect to produce reproducible output and have hashes on their downloaded source code files. It doesn't make sense to have a mainline package which is fetching random stuff due to fuzzy semver specifiers.
For external or third party packages, those wouldn't have source code hashes, and that makes sense.
For Go at least it will always be possible to make a hashed set of source code archives. It's possible to download & compress each dependency independently for Go as well, and analyze the licenses and whatever else.
Quoted 5 lines
Quoted 2 lines
Node-modules also often contains symlinks
and binaries prepared for the particular host build system.But those are created at build time, not at download time, right? Well, node is so awful, that I would not be surprised...
No - download time - the post install hooks are run. For example, electron might download + extract a tarball with Chromium.
Quoted 7 lines
Quoted 4 lines
I don't agree that legal is the only thing that matters here, you also want to be sure that you'll have a Buildroot build that works every time, without internet access, if you have a full "make download" pass run in advance.
Exactly the role of the dl/ directory; and that would contain the complete downloaded archives with all the vendored dependencies.
You want to have a download archive for every Buildroot package which contains ALL the dependencies for the package, and make this the ONLY way to do this?
Quoted 6 lines
Even when
node_moduleswon't necessarily work on some machines without running "npm install" again?The storage space increase alone of storing the entire
node_modulesfor multiple packages across potentially many versions is a reason to at least consider an alternative.
What "reasonable" alternative do we have at this point?
Quoted 5 lines
There's a way to split dependency source tarballs properly - even in Node - and even if it was impossible in Node, the limitations of Node shouldn't prevent us from doing this for a language that /can/ support it like Go.
Agree. NPM is notoriously worse than other modern package managers.
Quoted 17 lines
I understand that the goal is to keep things as simple as possible and avoid adding any work for the maintainers, and indeed this makes sense.
Quoted 12 lines
Quoted 5 lines
I don't understand what you're saying here. If I download package-c dependency at 1.0.4 it will be under - for example - $(BR2_DL_DIR)/go-modules/package-c/package-c-1.0.4.tar.gz. The deduplication is for package dependencies with identical versions and identical source URLs.
Ah, because you store all the go vendored dependencies "away" of the package.
I am not sure I like that, because it breaks the semantics of the dl/
directory: all the csource needed by one package is in dl/foo/ and the vendored dependencies *are* part of the package.
Semantics are semantics though, and a deviation shouldn't be thrown out just due to them.
[snip]
Quoted 8 lines
The goal is also to avoid breaking package source code download hashes after upgrading the tool, due to a change in the format of vendor/, to reduce source code tarball sizes, to make it easier to separate out proprietary and FLOSS components, and to ensure that the build is reproducible.
I'll build & submit a RFC prototype so that it's clearer what I'm actually suggesting here.
Please reach out to me, I think we should be on reasonably-same footing on an RFC and we're going to want to leverage this for both Go and Rust.
Quoted 28 lines
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
I guess package.log is for npm. No idea what yarn is. Still, that's only two out of at least three...
Are you saying it's not possible to collect an index of indirect dependencies with those?
IIRC, for NPM, no. Or not trivially, or not reproducibly.
You absolutely can collect a manifest of dependencies reproducibly and easily with the package-lock.json.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
Quoted 3 lines
Quoted 1 line
I do not want to have to repeat the vendoring logic in Buildroot.
Why repeat it? Re-use it from the programming language! Not everything has to be in bash.
It's not about the language; it's about the logic.
I don't understand what you mean.
The logic of vendoring.
All of which is implemented in these languages already in a format that is at a very high level and will require little to no "reinventing the wheel" from us. (at least, for Go)
Quoted 5 lines
Quoted 2 lines
You wouldn't put anything proprietary into Buildroot proper since it's a GPLv2 project. It would be a extension package.
We do have proprietary packages in Buidlroot:
Quoted 8 lines
Quoted 6 lines
boot/s500-bootloader/ package/armbian-firmware/ package/nvidia-driver/ package/wilc1000-firmware/And quite a few others...
OK, I see what you mean by proprietary.
I actually meant to make this point earlier, but didn't. I think "proprietary" isn't necessarily what I meant to stir up here. In my case, and I believe others, my proprietary is non-distributable. Some proprietary is distributable.
This is where we go back to my original comments: I think keeping these packages out of the dep-chain should be the responsibility of the package owners. Maybe we don't even error out when we catch it happening, because to your point we may not care. Or add an option to explicitly override.
Sorry for snipping a lot out, but as with Christian I replied to everything that I felt wouldn't send us back in circles.
Thanks,
Sam
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
In reply to Yann E. MORINYann,
Quoted 8 lines · Yann E. MORIN
Ah, because you store all the go vendored dependencies "away" of the package.
I am not sure I like that, because it breaks the semantics of the dl/
directory: all the csource needed by one package is in dl/foo/ and the vendored dependencies *are* part of the package.So I am absolutely not in favour of storing the go modules on the side.
I've thought about this quite a bit and I think you're probably right, for a v1 implementation in Buildroot we most likely should do the approach of "go mod vendor" just before compressing the source file for "make source." It should work fine for Go (can't speak for node or others) because the vendor/ tree is always consistent.
Best regards,
Christian
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Quoted 10 lines · Christian Stewart
Yann,
Quoted 7 lines · Yann E. MORIN
Ah, because you store all the go vendored dependencies "away" of the package.
I am not sure I like that, because it breaks the semantics of the dl/
directory: all the csource needed by one package is in dl/foo/ and the vendored dependencies *are* part of the package.
Quoted 7 lines
Quoted 1 line
So I am absolutely not in favour of storing the go modules on the side.
I've thought about this quite a bit and I think you're probably right, for a v1 implementation in Buildroot we most likely should do the approach of "go mod vendor" just before compressing the source file for "make source." It should work fine for Go (can't speak for node or others) because the vendor/ tree is always consistent.
Now that apparently everyone is on the same page, I think it's good to keep
track of the conclusions, so I've created a wiki page [1].
If there is anything that needs further discussion, please continue that here
in the list. However, if there is something I just didn't write down correctly or clearly enough, simply update the wiki.
For your convenience, I've copied the wiki page below.
== Problem statement and requirements
Language package managers make it possible to express dependencies and make sure the correct version of those dependencies are used.
For Buildroot, the problem is to make sure those dependencies are downloaded. Sometimes the upstream package simply bundles all dependencies (this is called "vendored"), but sometimes they only have them by reference.
Buildroot needs to make sure that:
* the dependencies are downloaded;
* the dependencies are extracted in a way that they're accessible by the language's build system;
* everything is properly hashed so we are sure the correct tarballs were downloaded;
* it is possible to do "make source", unplug the network, and do a build;
* it is possible to use a BR2_PRIMARY_SITE and BR2_PRIMARY_SITE_ONLY;
* it is possible to use BR2_BACKUP_SITE;
* the information produced by "make legal-info" is correct.
An additional nice-to-have is to be able to split off the open source dependencies from the non-redistributable dependencies.
== Possible solutions
1. Add a download helper that downloads the base source, runs the package manager to vendor all dependencies, and creates a tarball of this bundle.
2. Add a download helper that downloads the base source, runs the package manager to vendor all dependencies, and creates a separate tarball containing only the dependencies.
3. Add a download helper that downloads the base source, runs the package manager to vendor all dependencies, splits off the non-redistributable dependencies, and creates a separate tarball for the open-source and the non-redistributable dependencies.
4. Add a download helper that downloads the base source, runs the package manager to get all dependencies, and creates a separate tarball for each individual dependency.
5. Solution 1 + extra legal-info infrastructure that makes it possible to strip off non-redistributable parts when generating the legal-info tarball.
Solution 1 is the simplest to implement. It is the approach taken by
http://patchwork.ozlabs.org/project/buildroot/list/?series=159771 for cargo.
Solution 2 is also easy to implement. However:
- the download infra has to "know" how to download the additional tarball (it should not be downloaded when using the normal method, but it should be downloaded for PRIMARY and BACKUP);
- the additional tarball has to be added to the hash file;
- the additional tarball has to be extracted as well.
Solution 3 is like solution 2, but times two.
Solution 4 is a lot more complicated to implement. It is probably doable to use the language itself to wrap the package manager (which typically has a library as well) to gather a list of all dependencies and URLs. Then each of them has to be downloaded and tarred separately. However, to support PRIMARY and BACKUP, we'll need to maintain this list inside Buildroot as well. In addition, we need a separate hash for each tarball.
Solution 5 is somewhat orthogonal to the rest. It has the nice advantage that it can also be applied to cases where the bundling is done without using a language package manager (e.g. for C/C++ code).
Regards,
Arnout
[1] https://elinux.org/Buildroot:Language_package_managers_and_dependencies
RFC PATCH 1/1 package/pkg-golang: download deps to vendor tree if not present
Hey Arnout,
Quoted 22 lines · Arnout Vandecappelle
Quoted 10 lines · Christian Stewart
Yann,
Quoted 7 lines · Yann E. MORIN
Ah, because you store all the go vendored dependencies "away" of the package.
I am not sure I like that, because it breaks the semantics of the dl/
directory: all the csource needed by one package is in dl/foo/ and the vendored dependencies *are* part of the package.Quoted 7 lines
Quoted 1 line
So I am absolutely not in favour of storing the go modules on the side.
I've thought about this quite a bit and I think you're probably right, for a v1 implementation in Buildroot we most likely should do the approach of "go mod vendor" just before compressing the source file for "make source." It should work fine for Go (can't speak for node or others) because the vendor/ tree is always consistent.
Quoted 6 lines
Now that apparently everyone is on the same page, I think it's good to keep
track of the conclusions, so I've created a wiki page [1].If there is anything that needs further discussion, please continue that here
in the list. However, if there is something I just didn't write down correctly or clearly enough, simply update the wiki.
Thanks for writing all of this down and creating a wiki for this. I think this helps clarify our options and will give us room for discussion.
Quoted 48 lines
For your convenience, I've copied the wiki page below.
== Problem statement and requirements
Language package managers make it possible to express dependencies and make sure the correct version of those dependencies are used.
For Buildroot, the problem is to make sure those dependencies are downloaded. Sometimes the upstream package simply bundles all dependencies (this is called "vendored"), but sometimes they only have them by reference.
Buildroot needs to make sure that:
* the dependencies are downloaded;
* the dependencies are extracted in a way that they're accessible by the language's build system;
* everything is properly hashed so we are sure the correct tarballs were downloaded;
* it is possible to do "make source", unplug the network, and do a build;
* it is possible to use aBR2_PRIMARY_SITEandBR2_PRIMARY_SITE_ONLY;
* it is possible to useBR2_BACKUP_SITE;
* the information produced by "make legal-info" is correct.An additional nice-to-have is to be able to split off the open source dependencies from the non-redistributable dependencies.
== Possible solutions
1. Add a download helper that downloads the base source, runs the package manager to vendor all dependencies, and creates a tarball of this bundle.
2. Add a download helper that downloads the base source, runs the package manager to vendor all dependencies, and creates a separate tarball containing only the dependencies.
3. Add a download helper that downloads the base source, runs the package manager to vendor all dependencies, splits off the non-redistributable dependencies, and creates a separate tarball for the open-source and the non-redistributable dependencies.
4. Add a download helper that downloads the base source, runs the package manager to get all dependencies, and creates a separate tarball for each individual dependency.
5. Solution 1 + extra legal-info infrastructure that makes it possible to strip off non-redistributable parts when generating the legal-info tarball.Solution 1 is the simplest to implement. It is the approach taken by
http://patchwork.ozlabs.org/project/buildroot/list/?series=159771 for cargo.
Quoted 7 lines
Solution 2 is also easy to implement. However:
- the download infra has to "know" how to download the additional tarball (it should not be downloaded when using the normal method, but it should be downloaded for PRIMARY and BACKUP);
- the additional tarball has to be added to the hash file;
- the additional tarball has to be extracted as well.
I sent an RFC last night, which created a very crude initial implementation of this, as I needed/wanted to leverage the atomic saves in support/download/dl-wrapper.
After briefly chatting with Christian, we believe this could very easily be leveraged by using go mod vendor instead of cargo vendor.
http://lists.busybox.net/pipermail/buildroot/2020-September/292211.html
Thanks,
Signature · 13 lines
Sam Voss | Sr. Software Engineer | Commercial Avionics COLLINS AEROSPACE 400 Collins Road NE, Cedar Rapids, Iowa 52498, USA Tel: +1 319 263 4039 sam.voss at collins.com | collinsaerospace.com CONFIDENTIALITY WARNING: This message may contain proprietary and/or privileged information of Collins Aerospace and its affiliated companies. If you are not the intended recipient, please 1) Do not disclose, copy, distribute or use this message or its contents. 2) Advise the sender by return email. 3) Delete all copies (including all attachments) from your computer. Your cooperation is greatly appreciated.