package/pkg-cargo.mk: Introduce the cargo package infrastructure.
PATCH v3 1/10 package/pkg-cargo.mk: Introduce the cargo package infrastructure.
In order to be package agnostic, the install phase is now using cargo instead of install.TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.
---
changes:
v2 -> v3 (Sam Voss)
- set TARGET_CONFIGURE_OPTS when running cargo
This is the V3 of the previously posted series :
http://patchwork.ozlabs.org/project/buildroot/list/?series=156166&state=%2A&archive=both
A lot of changes following the buildroot developer meeting's decisions from February 2020.
Needs to be applied on top of the rust 1.40 bump
--- package/Makefile.in | 1 + package/pkg-cargo.mk | 92 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) create mode 100644 package/pkg-cargo.mk
package/Makefile.in +1 -0
@@ -426,3 +426,4 @@ include package/pkg-kernel-module.mk include package/pkg-waf.mk include package/pkg-golang.mk include package/pkg-meson.mk +include package/pkg-cargo.mk
package/pkg-cargo.mk new file +92 -0
92 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 2/10 docs/manual/cargo: Update manual for cargo packages.
--- docs/manual/adding-packages-cargo.txt | 54 +++++---------------------- 1 file changed, 10 insertions(+), 44 deletions(-)
docs/manual/adding-packages-cargo.txt +10 -44
@@ -27,9 +27,9 @@ The +Config.in+ file of Cargo-based package 'foo' should contain: ==== Cargo-based package's +.mk+ file -Buildroot does not (yet) provide a dedicated package infrastructure for -Cargo-based packages. So, we will explain how to write a +.mk+ file for such a -package. Let's start with an example: +Buildroot provides a dedicated package infrastructure for Cargo-based packages. +So, we will explain how to write a +.mk+ file for such a package. Let's start +with an example: ------------------------------ 01: ################################################################################
@@ -44,53 +44,19 @@ package. Let's start with an example: 10: FOO_LICENSE = GPL-3.0+ 11: FOO_LICENSE_FILES = COPYING 12: -13: FOO_DEPENDENCIES = host-rustc -14: -15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo -16: FOO_CARGO_MODE = $(if $(BR2_ENABLE_DEBUG),debug,release) -17: -18: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE) -19: -20: FOO_CARGO_OPTS = \ -21: --$(FOO_CARGO_MODE) \ -22: --target=$(RUSTC_TARGET_NAME) \ -23: --manifest-path=$(@D)/Cargo.toml -24: -25: define FOO_BUILD_CMDS -26: $(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \ -27: cargo build $(FOO_CARGO_OPTS) -28: endef -29: -30: define FOO_INSTALL_TARGET_CMDS -31: $(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \ -32: $(TARGET_DIR)/usr/bin/foo -33: endef -34: -35: $(eval $(generic-package)) +13: $(eval $(cargo-package)) -------------------------------- The Makefile starts with the definition of the standard variables for package declaration (lines 7 to 11). -As seen in line 35, it is based on the -xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines -the variables required by this particular infrastructure, where Cargo is -invoked: +As seen in line 13, it is based on the cargo-package infrastructure. Cargo will +be invoked automatically by this infrastructure. The required dependencies of the +crate will be downloaded and the buildroot dependencies will also be set. -* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required - to configure the cross-compilation of the package are passed via - +FOO_CONF_OPTS+. - -* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on - the target. - -In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to -contain +host-cargo+. - -To sum it up, to add a new Cargo-based package, the Makefile example can be -copied verbatim then edited to replace all occurences of +FOO+ with the -uppercase name of the new package and update the values of the standard -variables. +It is still possible to define custom build commands or install commands (i.e. +with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS). +Those will then replace the commands from the cargo infrastructure. ==== About Dependencies Management
Signature · 1 line
2.17.1
PATCH v3 3/10 package/ripgrep: convert to cargo infrastructure
In reply to Patrick Havelange--- package/ripgrep/ripgrep.mk | 23 +---------------------- 1 file changed, 1 insertion(+), 22 deletions(-)
package/ripgrep/ripgrep.mk +1 -22
@@ -9,25 +9,4 @@ RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION)) RIPGREP_LICENSE = MIT RIPGREP_LICENSE_FILES = LICENSE-MIT -RIPGREP_DEPENDENCIES = host-rustc -RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo -RIPGREP_CARGO_MODE = $(if $(BR2_ENABLE_DEBUG),debug,release) - -RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_MODE) - -RIPGREP_CARGO_OPTS = \ - --$(RIPGREP_CARGO_MODE) \ - --target=$(RUSTC_TARGET_NAME) \ - --manifest-path=$(@D)/Cargo.toml - -define RIPGREP_BUILD_CMDS - $(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \ - cargo build $(RIPGREP_CARGO_OPTS) -endef - -define RIPGREP_INSTALL_TARGET_CMDS - $(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \ - $(TARGET_DIR)/usr/bin/rg -endef - -$(eval $(generic-package)) +$(eval $(cargo-package))
Signature · 1 line
2.17.1
PATCH v3 4/10 support/download/dl-wrapper: rework backend parsing
In reply to Patrick HavelangeThe point of this modification is to be more generic in the way the backends/options are parsed from the urls.
The '+' char serves as delimiter between the backends and the actual url.
The '|' char serves as separator between the different options or backends.
This will be useful later when more options or backends would be needed.
pkg-generic.mk now passes the optional FOO_PKGMGR value in case a package needs to use a second backend.
--- package/pkg-generic.mk | 2 +- support/download/dl-wrapper | 34 ++++++++++++++++++++++------------ 2 files changed, 23 insertions(+), 13 deletions(-)
package/pkg-generic.mk +1 -1
@@ -541,7 +541,7 @@ ifndef $(2)_PATCH endif $(2)_ALL_DOWNLOADS = \ - $$(if $$($(2)_SOURCE),$$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$($(2)_SOURCE)) \ + $$(if $$($(2)_SOURCE),$$($(2)_PKGMGR)$$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$($(2)_SOURCE)) \ $$(foreach p,$$($(2)_PATCH) $$($(2)_EXTRA_DOWNLOADS),\ $$(if $$(findstring ://,$$(p)),$$(p),\ $$($(2)_SITE_METHOD)+$$($(2)_SITE)/$$(p)))
support/download/dl-wrapper +22 -12
@@ -21,8 +21,8 @@ export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e" main() { local OPT OPTARG - local backend output hfile recurse quiet rc - local -a uris + local backend backends_str output hfile recurse quiet rc backend2 + local -a uris backends # Parse our options; anything after '--' is for the backend while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
@@ -85,18 +85,24 @@ main() { download_and_check=0 rc=1 for uri in "${uris[@]}"; do - backend_urlencode="${uri%%+*}" - backend="${backend_urlencode%|*}" - case "${backend}" in - git|svn|cvs|bzr|file|scp|hg) ;; - *) backend="wget" ;; - esac + urlencode="" + backend="wget" + backend2="" + + # Extract the backends list, separated by '+' from the actual url + backends_str="${uri%%+*}" + # make an array out of it, backends are '|' separated + IFS='|' read -r -a backends <<< "${backends_str}" + # parse each of them, set flags and set the real backend + for b in "${backends[@]}" ; do + case "${b}" in + urlencode) urlencode="${b}" ;; + git|svn|cvs|bzr|file|scp|hg) backend="${b}" ;; + # insert here supported second backends) backend2="${b}" ;; + esac + done uri=${uri#*+} - urlencode=${backend_urlencode#*|} - # urlencode must be "urlencode" - [ "${urlencode}" != "urlencode" ] && urlencode="" - # tmpd is a temporary directory in which backends may store # intermediate by-products of the download. # tmpf is the file in which the backends should put the downloaded
@@ -138,6 +144,10 @@ main() { # cd back to free the temp-dir, so we can remove it later cd "${OLDPWD}" + if [ -n "${backend2}" ] ; then + support/download/${backend2} "${tmpf}" "${tmpd}" "${old_dl_dir}" + fi + # Check if the downloaded file is sane, and matches the stored hashes # for that file if support/download/check-hash ${quiet} "${hfile}" "${tmpf}" "${output##*/}"; then
Signature · 1 line
2.17.1
PATCH v3 5/10 docs/manual/adding-packages-generic: update for new FOO_PKGMGR value
In reply to Patrick Havelange--- docs/manual/adding-packages-generic.txt | 7 +++++++ 1 file changed, 7 insertions(+)
docs/manual/adding-packages-generic.txt +7 -0
@@ -265,6 +265,13 @@ not and can not work as people would expect it should: +LIBFOO_SITE=/opt/software/libfoo.tar.gz+ + +LIBFOO_SITE=$(TOPDIR)/../src/libfoo+ +* +LIBFOO_PKGMGR+ may contain the name of the second backend used + to complete the sources tarball. This second backend is called after + the first archive has been generated (e.g. git) or simply downloaded + (wget). It can be useful in the case where some other files are + needed for the build and require to be downloaded by a specific tool. + The value of this variable must end with a '|' (serves as a separator). + * +LIBFOO_DL_OPTS+ is a space-separated list of additional options to pass to the downloader. Useful for retrieving documents with server-side checking for user logins and passwords, or to use a proxy.
Signature · 1 line
2.17.1
PATCH v3 6/10 package/pkg-cargo.mk: Introduce the cargo dl backend
In reply to Patrick HavelangeCargo is now a fully supported PKGMGR, automatically set for any package using the cargo infrastructure.
This effectively splits the download phase and the build phase.
The cargo backend will set CARGO_HOME inside DL_DIR during download. The cached files in CARGO_HOME permits to download only once the crates index and each dependencies.
During download phase, it will call cargo vendor to copy the dependencies inside the VENDOR directory inside the source archive.
A local cargo config is also inserted inside the archive in order to use the VENDOR dir during the build phase.
The build phase is forced to not query the online repository anymore and thus will be using the vendored dependencies from the tarball. This also permits to have offline builds.
--- package/pkg-cargo.mk | 10 ++++-- package/ripgrep/ripgrep.hash | 2 +- support/download/cargo | 65 ++++++++++++++++++++++++++++++++++++ support/download/dl-wrapper | 2 +- 4 files changed, 75 insertions(+), 4 deletions(-) create mode 100755 support/download/cargo
package/pkg-cargo.mk +8 -2
@@ -39,6 +39,8 @@ define inner-cargo-package # We need host-rustc to run cargo $(2)_DEPENDENCIES += host-rustc +$(2)_PKGMGR = cargo\| + $(2)_CARGO_ENV = \ CARGO_HOME=$(HOST_DIR)/share/cargo \ $(TARGET_CONFIGURE_OPTS)
@@ -55,11 +57,13 @@ endif # ifndef $(2)_BUILD_CMDS define $(2)_BUILD_CMDS + cd $$(@D) && \ $(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \ cargo build \ --$$($(2)_CARGO_MODE) \ $$($(2)_CARGO_TARGET_OPT) \ - --manifest-path $$(@D)/Cargo.toml + --manifest-path Cargo.toml \ + --offline endef endif
@@ -69,12 +73,14 @@ endif # ifndef $(2)_INSTALL_TARGET_CMDS define $(2)_INSTALL_TARGET_CMDS + cd $$(@D) && \ $(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \ cargo install \ --root $(TARGET_DIR)/usr/ \ --bins \ - --path $$(@D) \ + --path ./ \ $$($(2)_CARGO_TARGET_OPT) \ + --offline \ --force endef endif
package/ripgrep/ripgrep.hash +1 -1
@@ -1,3 +1,3 @@ # Locally calculated -sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz +sha256 cb895cff182740c219fefbbaaf903e60f005a4ac4688cac1e43e5fbbbccc5a94 ripgrep-0.8.1.tar.gz sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT
support/download/cargo new file +65 -0
65 lines not on this page · raw message
support/download/dl-wrapper +1 -1
@@ -98,7 +98,7 @@ main() { case "${b}" in urlencode) urlencode="${b}" ;; git|svn|cvs|bzr|file|scp|hg) backend="${b}" ;; - # insert here supported second backends) backend2="${b}" ;; + cargo) backend2="${b}" ;; esac done uri=${uri#*+}
Signature · 1 line
2.17.1
PATCH v3 6/10 package/pkg-cargo.mk: Introduce the cargo dl backend
Hello,
Quoted 10 lines · On Thu, 20 Feb 2020 17:01:15 +0100 Patrick Havelange…
Cargo is now a fully supported PKGMGR, automatically set for any package using the cargo infrastructure.
This effectively splits the download phase and the build phase.The cargo backend will set
CARGO_HOMEinsideDL_DIRduring download. The cached files inCARGO_HOMEpermits to download only once the crates index and each dependencies.
During download phase, it will call cargo vendor to copy the dependencies inside the VENDOR directory inside the source archive.
Quoted 7 lines
A local cargo config is also inserted inside the archive in order to use the VENDOR dir during the build phase.
The build phase is forced to not query the online repository anymore and thus will be using the vendored dependencies from the tarball. This also permits to have offline builds.Signed-off-by Patrick Havelange <[email protected]>
We looked at this patch series during the previous (virtual) Buildroot Developers Meeting in late July, and we forgot to give some feedback. I'll provide the raw IRC logs of the discussion we had, however, I see there's been more discussion tonight on the IRC channel on this topic.
[13:54:34] <kos_tom> wow, wow this series https://patchwork.ozlabs.org/project/buildroot/list/?series=159771 has some pretty weird download infrastructure changes [13:55:17] <y_morin> Yes, it scares me quite a bit...
[13:55:33] <y_morin> However, I think we already discussed that here some time ago.
[13:55:52] <y_morin> The proposal I did was:
[13:56:19] <y_morin> 1) introduce a cargo download backend
[13:56:47] <y_morin> 2) have the cargo backend call the real backend (git, wget, etc...) for the cargo package,
[13:57:21] <y_morin> 3) have the cargo backend call "cargo bundle" to complete the package archive.
[13:57:26] <y_morin> kos_tom: ^^^
[13:59:50] <kos_tom> y_morin: I'm not sure how the "cargo backend can call the real backend
[14:00:10] <kos_tom> y_morin: what the patch series does is call "cargo vendor" to download all the dependencies
[14:00:34] <kos_tom> and then it makes a tarball of the whole thing, i.e the actual package source code + all its cargo dependencies [14:00:48] <kos_tom> and that's what you get as a tarball for this package in DL_DIR
[14:01:09] <y_morin> kos_tom: It is easy for the cargo backend to call the adequate "real" backend: "${0%/*}/wget"
[14:01:30] <y_morin> We "just" need to be able to pass that info yto he cargo backend
[14:01:32] <kos_tom> y_morin: the cargo backend does not do the download of dependecnies by itself
[14:01:37] <kos_tom> it calls the "cargo vendor" command
[14:01:55] <y_morin> kos_tom: I don;t yet care about the depdnencies. I care about the main download for now.
[14:02:15] <y_morin> kos_tom: Hold on a sec, I'll post a simple script that shows that...
[14:02:46] <kos_tom> y_morin: look at the patch series
[14:02:52] <kos_tom> y_morin: it does use *two* backends
[14:02:56] <kos_tom> it uses the normal backend for the main download
[14:03:04] <y_morin> kos_tom: I know.
[14:03:08] <kos_tom> + an extra secondary backend, called "cargo" to download the dependencis
[14:03:11] <y_morin> And I am not OK with that.
[14:03:23] <y_morin> I am propiosing something else.
[14:03:40] <y_morin> I.e. to invert the logic
[14:03:48] <kos_tom> yeah, you are proposing that the "cargo" backend takes care of both the main download and the dependencies download [14:03:57] <kos_tom> and not have this concept of "secondary backend"
[14:04:02] <y_morin> Exactly.
[14:04:05] <kos_tom> Ok.
[14:04:15] <kos_tom> the question is: do we want this kind of download integration at all?
[14:04:22] <kos_tom> if we do, does it work for NodeJS ? PHP ?
[14:05:40] <y_morin> kos_tom: https://pastebin.com/c3K42Lb7
[14:06:03] <y_morin> kos_tom: That is also the idea. For go, it would be very similar too.
[14:06:19] <kos_tom> y_morin: but how do you pass the "real backend" if the backend is "cargo" ?
[14:06:22] <y_morin> And for npm and php, afaiu (which is not much), as well
[14:06:28] <kos_tom> i.e if you have a cargo package to fetch from Git, and another from https
[14:07:06] <y_morin> kos_tom: Pretty east. In the cargo infra: FOO_SITE_METHOD = cargo FOO_EXTRA_DLOPTS += --backend=$(whatever)
[14:07:22] <y_morin> kos_tom: Hmm...
[14:07:39] <y_morin> you mean "cargo vendor" can fetch from various locations?
[14:08:07] <y_morin> I meant: from various location usign various methods?
[14:08:21] <y_morin> It that is so, then my proposal is not good, indeed.
[14:08:55] <kos_tom> y_morin: no I meant the primary download
[14:09:11] <kos_tom> y_morin: ie. for the ripgrep package, the actual download of the ripgrep code
[14:09:44] <kos_tom> of the cargo series, I'm wondering if I should apply patches 1/2/3, which create the package infra, but not the download stuff [14:09:53] <y_morin> kos_tom: Ah, but the primary download is exactly that: FOO_EXTRA_DLOPTS += --backend=$(geturischeme $(FOO_SITE)) [14:10:08] <kos_tom> my main concern is that we have a single package that would use this infra right now, which doesn't make merging such a simple infra very convincing. [14:10:30] <kos_tom> y_morin: instead of an extra backend, can we do with a post-download hook ?
[14:10:54] <y_morin> kos_tom: I had a good reason not to use such a hook.
[14:11:06] <y_morin> kos_tom: I would have to dig my irc logs...
[14:11:54] <y_morin> kos_tom: Ah yes, I know...
[14:12:19] <y_morin> kos_tom: If you use a post-dl hook, it means the archive with just the main package is already in $(FOO_DL_DIR) [15:12:14] <kos_tom> y_morin: so can we try to conclude on the cargo download thing?
[15:13:08] <y_morin> kos_tom: I provided some info above, not sure you saw it...
[15:13:29] <y_morin> kos_tom: If you use a post-dl hook, it means the archive with just the main package is already in $(FOO_DL_DIR) [15:13:47] <kos_tom> y_morin: but is that a problem ?
[15:14:00] <kos_tom> I guess it is indeed simpler to have a single tarball that has the source code for everything, including dependencies [15:14:05] <y_morin> kos_tom: Yes it is, for a subsequent build: it would not match the hashes.
[15:14:35] <kos_tom> are the hashes anyway going to match when you see (from the "bat" example being discussed with Romain) that dependency versions anyway can change [15:14:40] <y_morin> kos_tom: That is the idea that the archive contains the main package *and* all its depednencies.
[15:14:56] <y_morin> For those package managers that allow pinning the version, we can get hashes of the archive, then
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v3 7/10 docs/manual/adding-packages-cargo: update doc for new infra
In reply to Patrick Havelange--- docs/manual/adding-packages-cargo.txt | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-)
docs/manual/adding-packages-cargo.txt +4 -12
@@ -61,16 +61,8 @@ Those will then replace the commands from the cargo infrastructure. ==== About Dependencies Management A crate can depend on other libraries from crates.io or git repositories, listed -in its Cargo.toml file. Before starting a build, Cargo usually downloads -automatically them. This step can also be performed independently, via the -+cargo fetch+ command. +in its Cargo.toml file. Before starting a build, the +cargo vendor+ command +will be run and will take care of downloading those dependencies. -Cargo maintains a local cache of the registry index and of git checkouts of the -crates, whose location is given by +$CARGO_HOME+. As seen in the package -Makefile example at line 15, this environment variable is set to -+$(HOST_DIR)/share/cargo+. - -This dependency download mechanism is not convenient when performing an offline -build, as Cargo will fail to fetch the dependencies. In that case, it is advised -to generate a tarball of the dependencies using the +cargo vendor+ and add it to -+FOO_EXTRA_DOWNLOADS+. +The final source tarball will contain all the required dependencies, effectively +permitting us to do an offline build.
Signature · 1 line
2.17.1
PATCH v3 8/10 package/ripgrep: bump to version 11.0.1
In reply to Patrick HavelangeThis bump is needed because the old cached archive of the tarball of ripgrep is still using the old format and can't be used anymore (as the expected content of the final archive has changed with the recent infrastructure changes for cargo packages).
--- package/ripgrep/ripgrep.hash | 2 +- package/ripgrep/ripgrep.mk | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
package/ripgrep/ripgrep.hash +1 -1
@@ -1,3 +1,3 @@ # Locally calculated -sha256 cb895cff182740c219fefbbaaf903e60f005a4ac4688cac1e43e5fbbbccc5a94 ripgrep-0.8.1.tar.gz +sha256 c02fe3077c95872175b098d002f872a619195436569a7dff443bbb605bc27474 ripgrep-11.0.1.tar.gz sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT
package/ripgrep/ripgrep.mk +1 -1
@@ -4,7 +4,7 @@ # ################################################################################ -RIPGREP_VERSION = 0.8.1 +RIPGREP_VERSION = 11.0.1 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION)) RIPGREP_LICENSE = MIT RIPGREP_LICENSE_FILES = LICENSE-MIT
Signature · 1 line
2.17.1
PATCH v3 9/10 package/ripgrep: add legal-info for dependencies
In reply to Patrick HavelangeAs the dependencies are now vendored in the tarball, we can extract them easily. Added a helper script to list them. Updated the .hash & .mk files with those new licenses.
--- package/ripgrep/find_vendor_licenses.sh | 6 + package/ripgrep/ripgrep.hash | 167 +++++++++++++++++++++++ package/ripgrep/ripgrep.mk | 4 +- package/ripgrep/vendor.inc | 169 ++++++++++++++++++++++++ 4 files changed, 345 insertions(+), 1 deletion(-) create mode 100755 package/ripgrep/find_vendor_licenses.sh create mode 100644 package/ripgrep/vendor.inc
package/ripgrep/find_vendor_licenses.sh new file +6 -0
@@ -0,0 +1,6 @@ +#!/bin/sh +# to run from the extracted final sources +find VENDOR/ -type f \ + \( -iname '*license*' -o -iname 'licence*' -o -iname 'copying*' -o -iname 'copyright*' \) \ + -a -not -name '*.a' | \ + sort
package/ripgrep/ripgrep.hash +167 -0
170 lines not on this page · raw message
package/ripgrep/ripgrep.mk +3 -1
@@ -4,9 +4,11 @@ # ################################################################################ +include package/ripgrep/vendor.inc + RIPGREP_VERSION = 11.0.1 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION)) RIPGREP_LICENSE = MIT -RIPGREP_LICENSE_FILES = LICENSE-MIT +RIPGREP_LICENSE_FILES = LICENSE-MIT $(RIPGREP_VENDOR_LICENSE_FILES) $(eval $(cargo-package))
package/ripgrep/vendor.inc new file +169 -0
169 lines not on this page · raw message
Signature · 1 line
2.17.1
PATCH v3 9/10 package/ripgrep: add legal-info for dependencies
Hello all,
Quoted 11 lines · On Thu, 20 Feb 2020 17:01:18 +0100 Patrick Havelange…
As the dependencies are now vendored in the tarball, we can extract them easily. Added a helper script to list them. Updated the .hash & .mk files with those new licenses.
Signed-off-by Patrick Havelange <[email protected]>--- package/ripgrep/find_vendor_licenses.sh | 6 + package/ripgrep/ripgrep.hash | 167 +++++++++++++++++++++++ package/ripgrep/ripgrep.mk | 4 +- package/ripgrep/vendor.inc | 169 ++++++++++++++++++++++++ 4 files changed, 345 insertions(+), 1 deletion(-)
At the latest virtual Buildroot Developers Meeting, Yann and I talked about this. I'll paste the complete IRC conversation below. In short, Yann wanted some way of dynamically grabbing the legal information at runtime, but our discussion made it clear (I believe) that this was not possible. Yann agreed that a static description of the licenses and licenses files (like you did) was the only option. However, Yann didn't like that the vendor licenses are described in a separate vendor.inc file. I personally don't mind too much, as we already do that for the qt5webengine package. Overall, I believe this aspect is fairly minor compared to the concerns about the download backend.
Here is the unedited IRC conversation:
[15:32:56] <y_morin> kos_tom: For cargo: I do not like the vendor.inc stuff at all...
[15:33:05] <kos_tom> y_morin: do we have a better proposal ?
[15:33:33] <y_morin> kos_tom: Yes: handle it in the cargo infra
[15:33:46] <y_morin> I.e.: do not let the packages do the inclusion by themselves.
[15:33:53] <kos_tom> how would you do that? we don't support "dynamic legal-info" currently
[15:35:29] <y_morin> And in the infra: include $(pkgdir)/vendor.inc $(1)_LICENSE += , $($(1)_VENDOR_LICENSE) $(1)_LICENSE_FILES += $($(1)_VENDOR_LICENSE_FILES)
[15:36:05] <y_morin> kos_tom: The trick with the license list, is that the 'space-comma' is properly replaced by a comma in the legal-infra already
[15:36:25] <kos_tom> mooh, https://github.com/facebook/rocksdb/commit/085e4dcf8173e91311fbf9037d8bc9393f254c6f exists, but "git show 085e4dcf8173e91311fbf9037d8bc9393f254c6f" in this repo says "no such commit" [15:36:54] <kos_tom> y_morin: but the value of this VENDOR_LICENSE variable would only be known after the package has been downloaded
[15:36:56] <y_morin> (*if* we can get the license list of the vensored stuff; having the license fiels would already be nice enough)
[15:37:04] <kos_tom> y_morin: how does the infra fills it in ?
[15:37:07] <y_morin> Ah, right/
[15:37:24] <kos_tom> that's what I called "support for dynamic legal-info"
[15:37:39] <y_morin> No, no, the infra does not fill in the variables, they are prepared as is.
[15:37:45] <y_morin> ... as is in the series.
[15:37:52] <kos_tom> aah, github says "This commit does not belong to any branch on this repository."
[15:38:04] <y_morin> Except it is not the package ./mk that does the include and aggregartion, but the cvargo infra
[15:38:09] <kos_tom> y_morin: ah, so it doesn't change much compared to the series
[15:38:20] <y_morin> No, not much.
[15:38:37] <y_morin> kos_tom: Except the series as is does not even catter for the list of licenses either.
[15:39:21] <y_morin> kos_tom: But:
[15:39:42] <y_morin> 1) I am not too fond of the vendor.inc to begin with; the info should be in the .mk
[15:40:00] <y_morin> 2) *if* we want to go with the vendor.inc, then let's handle it in the cargo-infra
[15:40:17] <y_morin> But TBH, I would really prefer option 1.
[15:40:40] <y_morin> There is no reason to single out cargo from other infras.
[15:41:36] <kos_tom> y_morin: we do have something similar as the vendor.inc already today
[15:41:56] <y_morin> Where?
[15:42:02] <kos_tom> enjoy package/qt5/qt5webengine/chromium-latest.inc
[15:42:23] <y_morin> Oh noes... :-(
[15:43:18] <y_morin> kos_tom: OK, so *if* we want to go that route, let's not let packages handle it, but let's handle it in the infra, and in pkg-generic at that.
[15:44:05] <kos_tom> hm, maybe this is something we should do long-term, but I'm not sure it's really something we should promote at the pkg-generic level today
[15:44:18] <kos_tom> we're only talking about a few cargo-related packages for now
[15:46:05] <y_morin> kos_tom: https://pastebin.com/AQfyKaSL
[15:46:18] <y_morin> I think it would not be much more complex that that.
[15:46:46] <kos_tom> doesn't account for per-version subdirectories
[15:47:04] <kos_tom> for qt5webengine, we use two have .inc files :-)
[15:47:09] <y_morin> Yes, yes... You are nit-picking. ;-)
[15:47:26] <kos_tom> to me, that is a minor detail in the whole story
[15:47:37] <kos_tom> the question is: do we accept to handle legal-info for cargo packages this way ?
[15:48:05] <y_morin> To me: no.
[15:48:21] <y_morin> I'd like chromuium stays the esception.
[15:49:19] <y_morin> I don;t think this is a problem that the .mk of cargo packages is just 99% legal-stuff. That's just the sad story there is to it.
[15:49:42] <y_morin> Also, I don't like that parts of the legal-info is in the .mk, and part in the vendor.inc.
[15:50:25] <y_morin> Unless we decide that we want to move the legal-info to is own file, like foo.mk foo.hash and now foo.legal
[15:50:45] <y_morin> But meh, I would keep everything in a single file, please.
[15:50:47] <kos_tom> the idea for qt5webengine is also that the .inc file is kind of auto-generated
[15:51:07] <y_morin> Havinf the legal-info in a side file means it fille bit-rot there ebecause no one will look/update it.
[15:51:37] <y_morin> kos_tom: So what? The legalinfo in scan-pypi or scan-cpan is also autogenerated the first time. Tyhen it is manually updated..
[15:51:58] <kos_tom> y_morin: no, it is again updated with scanpypi or scancpan
[15:52:05] <kos_tom> scanpypi and scancpan are not one-time scripts
[15:52:18] <Jacmet> kos_tom: yeah, I couldn't backport the gnutls security bump before we had a fix for uclibc
[15:53:34] <Kubu> kos_tom: ok, I'm able to build bat
[15:53:40] <kos_tom> Kubu: ah :)
[15:53:43] <Kubu> even with ARM toolchain
[15:58:44] <y_morin> kos_tom: So, even the generated packages are properly updated, so even that is not an excuse ti have the licensing info outside the .mk
[15:58:58] <y_morin> (sorry, I had a sigfood from the nano-tigers...)
[16:00:33] <y_morin> Kubu: When you're done with cargo/rust, mind diving back into elf2flt: https://patchwork.ozlabs.org/project/buildroot/patch/20200726094338.1276868-1-fontaine.fabrice at gmail.com/ ? ;-) [16:01:27] <Kubu> y_morin: well, there is no much change in this version compared to what we have currently
[16:02:00] <kos_tom> y_morin: again, to me inside/outside the .mk is a small detail
[16:02:21] <kos_tom> Lightsword: have you looked at the Lua 5.4 series from Fran?ois Perrad ?
[16:02:22] <y_morin> kos_tom: So I am not sure I understood your question...
[16:02:23] <Kubu> it's just about sync with with the tag added recently
[16:02:52] <kos_tom> y_morin: are we OK with handling legal-info like this, i.e it's quite manual. I anyway don't really see any other option.
[16:03:36] <y_morin> kos_tom: Still, not sure to understand... :-/
[16:04:30] <y_morin> kos_tom: In fact, we *could* very well have the list established at runtime.
[16:04:31] <kos_tom> y_morin: well, aside the location of the legal details, are we OK with the way it's done ?
[16:04:59] <y_morin> Since the lists are not used to generate Makefile code, they are only evaluated in shell context
[16:05:20] <kos_tom> ooh, but
[16:05:22] <y_morin> so we could do: FOO_LICENSE = some script to extract the list
[16:05:23] <kos_tom> we already have BR2_PACKAGE_LUAINTERPRETER_ABI_VERSION
[16:05:33] <kos_tom> which contains 5.1 for LuaJIT/Lua5.1 and 5.3 for Lua 5.3
[16:06:03] <y_morin> and FOO_LICENSE_FILES = some script to extract the list of files
[16:06:04] <kos_tom> y_morin: but then we could only evaluate the FOO_LICENSE variable after the download has taken place
[16:06:32] <y_morin> kos_tom: No, because the variable are evaluated when the corresponding make rule is generated.
[16:07:02] <y_morin> which is another rule than foo-legal-info
[16:07:32] <y_morin> So, at the time foo-lkegal-info is "started/run", the foo-extract and foo-patch are finished
[16:08:07] <y_morin> .. it should work (famous last words)
[16:08:23] <kos_tom> y_morin: well, if I do "make FOO_LICENSE printvars"
[16:08:47] <y_morin> That won;t work if the patckage is not extractd/patched, no. indeed.
[16:09:09] <y_morin> So that wouild bust show-info too
[16:09:12] <y_morin> So not good.
[16:10:01] <y_morin> We indeed do need a static list of licenses and files
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH v3 10/10 docs/manual/adding-packages-cargo: Update for legal-info
In reply to Patrick HavelangeAdded a bit of explanations regarding the vendored licenses.
--- docs/manual/adding-packages-cargo.txt | 45 ++++++++++++++++++++------- 1 file changed, 34 insertions(+), 11 deletions(-)
docs/manual/adding-packages-cargo.txt +34 -11
@@ -38,19 +38,25 @@ with an example: 04: # 05: ################################################################################ 06: -07: FOO_VERSION = 1.0 -08: FOO_SOURCE = foo-$(FOO_VERSION).tar.gz -09: FOO_SITE = http://www.foosoftware.org/download -10: FOO_LICENSE = GPL-3.0+ -11: FOO_LICENSE_FILES = COPYING -12: -13: $(eval $(cargo-package)) +07: include package/foo/vendor.inc +08: +09: FOO_VERSION = 1.0 +10: FOO_SOURCE = foo-$(FOO_VERSION).tar.gz +11: FOO_SITE = http://www.foosoftware.org/download +12: FOO_LICENSE = GPL-3.0+ +13: FOO_LICENSE_FILES = COPYING $(FOO_VENDOR_LICENSE_FILES) +14: +15: $(eval $(cargo-package)) -------------------------------- -The Makefile starts with the definition of the standard variables for package -declaration (lines 7 to 11). +The Makefile starts with the inclusion of a second file that +contains the vendored licenses, more info on that topic later. +Then it continues with the definition of the standard variables +for package declaration (lines 9 to 13). -As seen in line 13, it is based on the cargo-package infrastructure. Cargo will +FOO_VENDOR_LICENSE_FILES is defined in the previously included file vendor.inc + +As seen in line 15, it is based on the cargo-package infrastructure. Cargo will be invoked automatically by this infrastructure. The required dependencies of the crate will be downloaded and the buildroot dependencies will also be set.
@@ -58,7 +64,20 @@ It is still possible to define custom build commands or install commands (i.e. with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS). Those will then replace the commands from the cargo infrastructure. -==== About Dependencies Management +The file vendor.inc contains only one variable : + +------------------------------ +01: FOO_VENDOR_LICENSES_FILES = \ +02: VENDOR/bar/COPYING \ +03: VENDOR/blah/LICENSE-MIT \ +04: VENDOR/bleh/COPYING \ +05: +-------------------------------- + +It is recommended to generate the list via scripting. +A script example can be found in the ripgrep directory. + +==== About Dependencies Management and Licenses A crate can depend on other libraries from crates.io or git repositories, listed in its Cargo.toml file. Before starting a build, the +cargo vendor+ command
@@ -66,3 +85,7 @@ will be run and will take care of downloading those dependencies. The final source tarball will contain all the required dependencies, effectively permitting us to do an offline build. + +In order for the +legal-info+ target to work as expected, it is necessary to list +the licenses of the vendored dependencies. Each of those entries should be added +to the +.mk+ and +.hash+ files.
Signature · 1 line
2.17.1
PATCH v3 1/10 package/pkg-cargo.mk: Introduce the cargo package infrastructure.
In reply to Patrick HavelangeHi Patrick, Arnout, All,
Le 20/02/2020 ? 17:01, Patrick Havelange a ?crit?:
Quoted 74 lines
In order to be package agnostic, the install phase is now using cargo instead of install.
TARGET_CONFIGURE_OPTSis now also set when running cargo in order to support cross compiling C code within cargo.Signed-off-by Patrick Havelange <[email protected]>---
changes:
v2 -> v3 (Sam Voss)
- setTARGET_CONFIGURE_OPTSwhen running cargoThis is the V3 of the previously posted series :
http://patchwork.ozlabs.org/project/buildroot/list/?series=156166&state=%2A&archive=bothA lot of changes following the buildroot developer meeting's decisions from February 2020.
Needs to be applied on top of the rust 1.40 bump
--- package/Makefile.in | 1 + package/pkg-cargo.mk | 92 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 93 insertions(+) create mode 100644 package/pkg-cargo.mkpackage/Makefile.in +1
-0@@ -426,3 +426,4 @@ include package/pkg-kernel-module.mk include package/pkg-waf.mk include package/pkg-golang.mk include package/pkg-meson.mk +include package/pkg-cargo.mkpackage/pkg-cargo.mk new file +34
-0@@ -0,0 +1,92 @@ +################################################################################ +# Cargo package infrastructure +# +# This file implements an infrastructure that eases development of package +# .mk files for Cargo packages. It should be used for all packages that use +# Cargo as their build system. +# +# See the Buildroot documentation for details on the usage of this +# infrastructure +# +# In terms of implementation, this Cargo infrastructure requires the .mk file +# to only specify metadata information about the package: name, version, +# download URL, etc. +# +# We still allow the package .mk file to override what the different steps +# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined, +# it is used as the list of commands to perform to build the package, +# instead of the default Cargo behaviour. The package can also define some +# post operation hooks. +# +################################################################################ + +################################################################################ +# inner-cargo-package -- defines how the configuration, compilation and +# installation of a cargo package should be done, implements a few hooks +# to tune the build process for cargo specifities and calls the generic +# package infrastructure to generate the necessary make targets +# +# argument 1 is the lowercase package name +# argument 2 is the uppercase package name, including a HOST_ prefix +# for host packages +# argument 3 is the uppercase package name, without the HOST_ prefix +# for host packages +# argument 4 is the type (target or host)
package/pkg-cargo.mk line 44
Quoted 10 lines
package/pkg-cargo.mk new file +10
-0+################################################################################ + +define inner-cargo-package + +# We need host-rustc to run cargo +$(2)_DEPENDENCIES += host-rustc + +$(2)_CARGO_ENV = \ + CARGO_HOME=$(HOST_DIR)/share/cargo \ + $(TARGET_CONFIGURE_OPTS)
Like other package infrastructure, it would be useful to define $(2)_CONF_ENV to allow cargo based packages to modify cargo environment.
For example, we need to add PKG_CONFIG_ALLOW_CROSS=1 to use pkg-config while cross-compiling.
Quoted 8 lines
package/pkg-cargo.mk new file +8
-0+ +$(2)_CARGO_MODE = $(if $(BR2_ENABLE_DEBUG),debug,release) + +ifeq ($(4),target) + $(2)_CARGO_TARGET_OPT = --target $$(RUSTC_TARGET_NAME) +endif + +#
package/pkg-cargo.mk line 62
Quoted 10 lines
package/pkg-cargo.mk new file +10
-0+# Build step. Only define it if not already defined by the package .mk +# file. +# +ifndef $(2)_BUILD_CMDS +define $(2)_BUILD_CMDS + $(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \ + cargo build \ + --$$($(2)_CARGO_MODE) \ + $$($(2)_CARGO_TARGET_OPT) \ + --manifest-path $$(@D)/Cargo.toml
cargo support -q (quiet) option that can be added to BUILD_CMDS and INSTALL_TARGET_CMDS.
Quoted 6 lines
package/pkg-cargo.mk new file +6
-0+endef +endif + +# +# Target installation step. Only define it if not already defined by +# the package .mk file.
package/pkg-cargo.mk line 78
Quoted 10 lines
package/pkg-cargo.mk new file +10
-0+# +ifndef $(2)_INSTALL_TARGET_CMDS +define $(2)_INSTALL_TARGET_CMDS + $(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \ + cargo install \ + --root $(TARGET_DIR)/usr/ \ + --bins \ + --path $$(@D) \ + $$($(2)_CARGO_TARGET_OPT) \ + --force
While testing this infra with opcua [1], I get this error :
"found a virtual manifest at path/Cargo.toml instead of a package manifest"
We need to customize "cargo --path ./" to a package subdirectory containing another Cargo.toml with a package manifest.
But it seems we can provide only one path to cargo command line, but I need to install binaries from several subdirectories. We may need a loop here otherwise cargo packages should redefine _INSTALL_TARGET_CMDS.
[1] https://github.com/locka99/opcua
Best regards,
Romain
Quoted 15 lines
package/pkg-cargo.mk new file +14
-0+endef +endif + +# Call the generic package infrastructure to generate the necessary +# make targets +$(call inner-generic-package,$(1),$(2),$(3),$(4)) + +endef + +################################################################################ +# cargo-package -- the target generator macro for Cargo packages +################################################################################ + +cargo-package = $(call inner-cargo-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)