Support for Cargo and Go vendoring
v2 cover Support for Cargo and Go vendoring
Hello,
Here is a patch series that implements support for vendoring of Cargo and Go packages.
As you're aware, many Cargo and Go packages describe their dependencies in a very language-specific way, and expect such dependencies to be downloaded by their language-specific tools.
This series builds on top of the work from Patrick Havelange <[email protected]> who worked on the Cargo integration, but extends it to also cover the case of Go, showing that we have a solution solving both of these language-specific package managers. For the Go vendoring, I've also used the indication provided by Christian Stewart <[email protected]>.
Overall, the solution consists in a concept of "download post-process helper" is introduced in the download infrastructure. These are shell scripts that can be invoked right after the download is done, but before the tarball is hash-checked and stored in DL_DIR. The idea is that such "download post-process helpers" can run the language-specific vendoring logic to fetch the package dependencies. Thanks to this, the tarball in DL_DIR for a given package not only contains the package source code itself, but also the source code of its Cargo or Go dependencies. The tarball hash covers the entire source code, the complete tarball is cached in DL_DIR, in the primary site and backup site, and the build can fully be done offline.
Such "download post-process helpers" are registered by means of the <pkg>_DOWNLOAD_POST_PROCESS variable by the appropriate package infrastructure.
This series is also available at:
https://github.com/tpetazzoni/buildroot/commits/pkg-mgr-v2
More specifically, the series goes like this:
- PATCH 1: add the concept of download post-processing in the download helper support/download/dl-wrapper
- PATCH 2: add the <pkg>_DOWNLOAD_POST_PROCESS variable to the package infrastructure, which gets passed down to support/download/dl-wrapper
- PATCH 3: add the <pkg>_DL_ENV variable to the package infrastructure, which allows to pass extra variables in the download wrapper environment. This will be used to pass extra variables specific to Cargo/Go vendoring.
- PATCH 4: add a set of two helper shell functions that will be used by the post processing scripts
- PATCH 5: implements the Go vendoring itself
- PATCH 6: adds package/tinifier, which is one package that requires Go vendoring. I don't particularly need this package, it was just added because the other Go packages we have in Buildroot today don't need vendoring.
- PATCH 7: introduces a cargo-package infrastructure, for both target and host packages
- PATCH 8: documents the added cargo-package infrastructure
- PATCH 9 and 10: converts ripgrep and sentry-cli to the cargo-package and host-cargo-package infrastructures respecitively
- PATCH 11: implements the Cargo vendoring itself. Note that we need to bump the ripgrep and sentry-cli package at the same time, as with the vendoring their tarball contents change and therefore their hash changes
- PATCH 12: rewrite the cargo-package documentation about dependency management
Changes since v1:
- Add -modcacherw when calling Go to get files read/write instead of read-only, to make sure they can be deleted by a "make clean". Was reported by Ryan Barnett.
- Change the Cargo infrastructure to "cd" into the build directory before calling Cargo, instead of calling Cargo with the full path to Cargo.toml as this was causing issues. Was reported by Ryan Barnett, investigated and fixed by Patrick Havelange.
- Call "go mod init" in the go-post-process script if go.mod doesn't exist. Was reported by Christian Stewart, who also provided the solution.
- Add a license warning about Go/Cargo packages, as they might have vendored dependencies that are not taken into account by Buildroot licensing details. Was reported by Ryan Barnett, and Yann E. Morin suggested this simple solution.
Best regards,
Thomas
Patrick Havelange (3):
package/pkg-cargo.mk: introduce the cargo package infrastructure docs/manual/cargo: document the cargo-package infrastructure package/ripgrep: convert to cargo infrastructure
Thomas Petazzoni (9):
support/download/dl-wrapper: add concept of download post-processing package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable package/pkg-download.mk: add <pkg>_DL_ENV variable support/download/post-process-helpers: add helper function for post
process scripts
support/download/go-post-process: implement Go vendoring support package/tinifier: new package
package/sentry-cli: convert to host-cargo-package infrastructure support/download/cargo-post-process, package/pkg-cargo.mk: enable
vendoring for Cargo packages
docs/manual/adding-packages-cargo.txt: rewrite explanation about
dependency management
DEVELOPERS | 1 + docs/manual/adding-packages-cargo.txt | 106 ++++++++---------- package/Config.in | 1 + package/Makefile.in | 1 + package/pkg-cargo.mk | 149 ++++++++++++++++++++++++++ package/pkg-download.mk | 5 +- package/pkg-golang.mk | 20 ++-- package/ripgrep/ripgrep.hash | 2 +- package/ripgrep/ripgrep.mk | 30 +----- package/sentry-cli/sentry-cli.hash | 2 +- package/sentry-cli/sentry-cli.mk | 24 +---- package/tinifier/Config.in | 10 ++ package/tinifier/tinifier.hash | 3 + package/tinifier/tinifier.mk | 13 +++ support/download/cargo-post-process | 38 +++++++ support/download/dl-wrapper | 9 +- support/download/go-post-process | 35 ++++++ support/download/post-process-helpers | 30 ++++++ 18 files changed, 358 insertions(+), 121 deletions(-) create mode 100644 package/pkg-cargo.mk create mode 100644 package/tinifier/Config.in create mode 100644 package/tinifier/tinifier.hash create mode 100644 package/tinifier/tinifier.mk create mode 100755 support/download/cargo-post-process create mode 100755 support/download/go-post-process create mode 100644 support/download/post-process-helpers
Signature · 1 line
2.29.2
PATCH v2 1/12 support/download/dl-wrapper: add concept of download post-processing
In order to support package managers such as Cargo (Rust) or Go, we want to run some custom logic after the main download, but before packing the tarball and checking the hash.
To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then support/download/<something>-post-process will be called.
--- support/download/dl-wrapper | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
support/download/dl-wrapper +8 -1
@@ -25,7 +25,7 @@ main() { local -a uris # Parse our options; anything after '--' is for the backend - while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do + while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do case "${OPT}" in c) cset="${OPTARG}";; d) dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() { r) recurse="-r";; f) filename="${OPTARG}";; u) uris+=( "${OPTARG}" );; + p) post_process="${OPTARG}";; q) quiet="-q";; :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -135,6 +136,12 @@ main() { continue fi + if [ -n "${post_process}" ] ; then + ${OLDPWD}/support/download/${post_process}-post-process \ + -o "${tmpf}" \ + -n "${raw_base_name}" + fi + # cd back to free the temp-dir, so we can remove it later cd "${OLDPWD}"
Signature · 1 line
2.29.2
PATCH v2 2/12 package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable
In reply to Thomas PetazzoniThis will allow packages to register than a download post-processing is needed. Note that this variable is intentionally not documented: it is an internal variable meant to be set by package infrastructures, not directly by packages.
--- package/pkg-download.mk | 1 + 1 file changed, 1 insertion(+)
package/pkg-download.mk +1 -0
@@ -108,6 +108,7 @@ define DOWNLOAD -n '$($(2)_BASENAME_RAW)' \ -N '$($(2)_RAWNAME)' \ -o '$($(2)_DL_DIR)/$(notdir $(1))' \ + $(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \ $(if $($(2)_GIT_SUBMODULES),-r) \ $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \ $(QUIET) \
Signature · 1 line
2.29.2
PATCH v2 3/12 package/pkg-download.mk: add <pkg>_DL_ENV variable
In reply to Thomas PetazzoniThis variable can be used by package to pass extra environment variables to the download logic. It will be used for the Go/Cargo vendoring.
The <pkg>_DL_ENV variable is intentionally not documented: at this point, it is not meant to be used by packages directly, but only by package infrastructures.
--- package/pkg-download.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-download.mk +3 -1
@@ -99,7 +99,9 @@ endif define DOWNLOAD $(Q)mkdir -p $($(2)_DL_DIR) - $(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \ + $(Q)$(EXTRA_ENV) \ + $($(2)_DL_ENV) \ + flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \ -c '$($(2)_DL_VERSION)' \ -d '$($(2)_DL_DIR)' \ -D '$(DL_DIR)' \
Signature · 1 line
2.29.2
PATCH v2 3/12 package/pkg-download.mk: add <pkg>_DL_ENV variable
Thomas, All,
On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
Quoted 10 lines
This variable can be used by package to pass extra environment variables to the download logic. It will be used for the Go/Cargo vendoring.
The <pkg>_DL_ENV variable is intentionally not documented: at this point, it is not meant to be used by packages directly, but only by package infrastructures.
Suggested-by Yann E. MORIN <[email protected]>Signed-off-by Thomas Petazzoni <[email protected]>
Although it's not much off your chest on this series: applied to master, with a little tweak, see below...
Quoted 9 lines
--- package/pkg-download.mk | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)package/pkg-download.mk +0
-0
package/pkg-download.mk endif · line 104
Quoted 7 lines
package/pkg-download.mk +3
-1@@ -99,7 +99,9 @@ endif define DOWNLOAD $(Q)mkdir -p $($(2)_DL_DIR) - $(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \ + $(Q)$(EXTRA_ENV) \ + $($(2)_DL_ENV) \ + flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
I find this code to be ugly to look at. But I could not come up with something that is nice to look at. So I moved the two _ENV variables expansions to the same line, and it looks less-ugly now...
Thanks.
Regards,
Yann E. MORIN.
Quoted 10 lines
package/pkg-download.mk +0
-0-c '$($(2)_DL_VERSION)' \ -d '$($(2)_DL_DIR)' \ -D '$(DL_DIR)' \Signature · 6 lines
2.29.2 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 4/12 support/download/post-process-helpers: add helper function for post process scripts
In reply to Thomas Petazzonidownload post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an unpack() function and a repack() function.
--- support/download/post-process-helpers | 30 +++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 support/download/post-process-helpers
support/download/post-process-helpers new file +30 -0
@@ -0,0 +1,30 @@ + +unpack() { + dest="$1" + tarball="$2" + + mkdir ${dest} + tar -C ${dest} --strip-components=1 -xf ${tarball} +} + +repack() { + src="$1" + tarball="$2" + + # Generate the archive, sort with the C locale so that it is reproducible. + find "$(basename ${src})" -not -type d -print0 >files.list + LC_ALL=C sort -z <files.list >files.list.sorted + + # let's use a fixed hardcoded date to be reproducible + date="2020-02-06 01:02:03 +0000" + + # Create GNU-format tarballs, since that's the format of the tarballs on + # sources.buildroot.org and used in the *.hash files + tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \ + --owner=0 --group=0 --mtime="${date}" -T files.list.sorted + gzip -6 -n <new.tar >new.tar.gz + mv "${tarball}" "${tarball}".old + mv new.tar.gz "${tarball}" + rm "${tarball}".old + rm -rf ${src} +}
Signature · 1 line
2.29.2
PATCH v2 4/12 support/download/post-process-helpers: add helper function for post process scripts
Thomas, All,
On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
Quoted 4 lines
download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an
unpack()function and arepack()function.
You forgot to explain in the commit log why we need a hard-coded, fixed date.
However, I find it ugly that we resort to such an arbitrary constant. At the very least, I'd like we take a memorable value, like 1970-01-01 00:00:00 +0000.
However, I am afraid that tar would refuse to extract files to that date, whining about an "implausibly old time stamp" when extracting such files...
But see below for a proposal...
Quoted 12 lines
Signed-off-by Thomas Petazzoni <[email protected]>--- support/download/post-process-helpers | 30 +++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 support/download/post-process-helperssupport/download/post-process-helpers new file +0
-0
support/download/post-process-helpers line 4
Quoted 4 lines
support/download/post-process-helpers new file +4
-0@@ -0,0 +1,30 @@ + +unpack() { + dest="$1" + tarball="$2"
This is a shell script, variables are global. But since we run it with bash, we can declare such variable as local:
local dest="${1}"
local tarball="${2}"support/download/post-process-helpers line 7
Quoted 2 lines
support/download/post-process-helpers new file +2
-0+ mkdir ${dest} + tar -C ${dest} --strip-components=1 -xf ${tarball}
local one_file
one_file="$(find "${dest}" -type f |head -n1)"
touch -r "${one_file}" "${dest}.timestamp"support/download/post-process-helpers line 12
Quoted 5 lines
support/download/post-process-helpers new file +5
-0+} + +repack() { + src="$1" + tarball="$2"
Ditto: make them 'local' (this can be fixed when applying).
Also, I suppose that the 'dest' parameter of unpack is expected to be the 'src' parameter of repack, right?
support/download/post-process-helpers line 19
Quoted 6 lines
support/download/post-process-helpers new file +6
-0+ # Generate the archive, sort with the C locale so that it is reproducible. + find "$(basename ${src})" -not -type d -print0 >files.list + LC_ALL=C sort -z <files.list >files.list.sorted + + # let's use a fixed hardcoded date to be reproducible + date="2020-02-06 01:02:03 +0000"
Assuming that unpack's dest and repack's src are pointing to the same directory:
date="$(stat -c "${src}.timestamp")"support/download/post-process-helpers line 24
Quoted 4 lines
support/download/post-process-helpers new file +4
-0+ # Create GNU-format tarballs, since that's the format of the tarballs on + # sources.buildroot.org and used in the *.hash files + tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \ + --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
This will have to be changed according to my pending series to change to the PAX format, whish is even more reproducible.
support/download/post-process-helpers line 29
Quoted 5 lines
support/download/post-process-helpers new file +5
-0+ gzip -6 -n <new.tar >new.tar.gz + mv "${tarball}" "${tarball}".old + mv new.tar.gz "${tarball}" + rm "${tarball}".old + rm -rf ${src}
Quote all expanded variable, even "${src}". And also rm "${src}.timestamp", now.
Regards,
Yann E. MORIN.
Quoted 4 lines
support/download/post-process-helpers new file +1
-0+}Signature · 1 line
2.29.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 4/12 support/download/post-process-helpers: add helper function for post process scripts
Thomas, All,
A few mre tweaks I forgot about...
On 2020-12-19 18:39 +0100, Yann E. MORIN spake thusly:
Quoted 1 line
On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
[--SNIP--]
Quoted 1 line
Quoted 1 line
+ mkdir ${dest}
Using 'mkdir -p' would ensure that parent direcotries are created as needed.
Quoted 2 lines
one_file="$(find "${dest}" -type f |head -n1)"Of course, the output should be sorted in the C locale, to ensure the same file is always used as reference (and hence a better name for that file):
local ref_file
ref_file="$(find "${dest}" -type f |LC_ALL=C sort |head -n1)"
touch -r "${ref_file}" "${dest}.timestamp"[--SNIP--]
Quoted 5 lines
support/download/post-process-helpers line 19
Quoted 2 lines
support/download/post-process-helpers new file +2
-0+ # let's use a fixed hardcoded date to be reproducible + date="2020-02-06 01:02:03 +0000"Assuming that unpack's dest and repack's src are pointing to the same directory:
date="$(stat -c "${src}.timestamp")"
Meh.. I forgot the actual format:
date="$(stat -c '%y' "${src}.timestamp")"Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 4/12 support/download/post-process-helpers: add helper function for post process scripts
In reply to Thomas PetazzoniThomas, All,
On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:
Quoted 4 lines
download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an
unpack()function and arepack()function.
Sorry to come back yet once more on that one, but I may have found quite an issue. Basically, your code does this:
unpack() {
tar -C ${dest} --strip-components=1 -xf ${tarball}
}
repack() {
tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
--owner=0 --group=0 --mtime="${date}" -T files.list.sorted
gzip -6 -n <new.tar >new.tar.gz
}So, it takes a tarball in whatever copression scheme and extract it, but it will alwyas create a gzip-compressed tarball.
So, how would we hanfle a case where the upstream tarball is, say, a .tar.bz2 ? For example;
FOO_SOURCE = foo=1.2.3.tar.bz2
FOO_SITE = http://foo.example.net/download
FOO_SITE_METOD = wget
$(eval $(go-package))
Besides the fact that the packager would see in DL_DIR a .tar.gz tarball different from the .tar.bz2 he coded, the most problematic issue wil be that we would have to list the .tar.gz inthe .hash file. This will be very confusing to packagers and reviewers...
Furthermore, there is another discrepancy in the post-processing scripts. For example, the go post-processor does, basically:
if tarball already vendored:
exit leaving the tarball as-is
unpack
vendor
repack as .tar.gz
So some go tarballs will be left as they are upstream, while others will be re-encoded. This is going to be quite confusing...
I think the post-processing script should always re-encode the tarball, so that at least we have a consistent behaviour across the board. This is pretty trivial to come up with, at the exepense of a bit of overhead for those tarballs that are already vendored, but the majority of packages will probably be git clones that need to be vendored, so it is a small price to pay.
The big problem that's left, is how we handle the discrepancy between the tarball listed in _SOURCE and the one listed in the .hash file...
I have to admit I do not have an easy solution for that for now...
Especially since with the pending series about changing the way we assemble tarballs from git and svn: the tar format changes, and the compression changes...
Regards,
Yann E. MORIN.
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'
PATCH v2 5/12 support/download/go-post-process: implement Go vendoring support
In reply to Thomas PetazzoniThis commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure.
The -modcacherw flag is added to ensure that the Go cache is read/write, and can be deleted properly upon "make clean".
The <pkg>_LICENSE variable of golang packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.
The logic to generate go.mod when not available is moved to the download post-process helper, as it must be done before "go mod vendor" is executed. Also, "go mod init" is used instead of manually crafting go.mod. This was suggested by Christian Stewart <[email protected]>. The Go module name is passed down to go-post-process using the BR_GOMOD environment variable.
--- package/pkg-golang.mk | 20 ++++++++++-------- support/download/go-post-process | 35 ++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 9 deletions(-) create mode 100755 support/download/go-post-process
package/pkg-golang.mk +11 -9
@@ -42,12 +42,13 @@ define inner-golang-package $(2)_BUILD_OPTS += \ -ldflags "$$($(2)_LDFLAGS)" \ + -modcacherw \ -tags "$$($(2)_TAGS)" \ -trimpath \ -p $(PARALLEL_JOBS) # Target packages need the Go compiler on the host. -$(2)_DEPENDENCIES += host-go +$(2)_DOWNLOAD_DEPENDENCIES += host-go $(2)_BUILD_TARGETS ?= .
@@ -72,14 +73,15 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE)))) # If the go.mod file does not exist, one is written with this root path. $(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE) -# Generate a go.mod file if it doesn't exist. Note: Go is configured -# to use the "vendor" dir and not make network calls. -define $(2)_GEN_GOMOD - if [ ! -f $$(@D)/go.mod ]; then \ - printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \ - fi -endef -$(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD +$(2)_DOWNLOAD_POST_PROCESS = go +$(2)_DL_ENV = \ + $(HOST_GO_COMMON_ENV) \ + GOPROXY=direct \ + BR_GOMOD=$$($(2)_GOMOD) + +# Due to vendoring, it is pretty likely that not all licenses are +# listed in <pkg>_LICENSE. +$(2)_LICENSE += , vendored dependencies licenses probably not listed # Build step. Only define it if not already defined by the package .mk # file.
support/download/go-post-process new file +35 -0
@@ -0,0 +1,35 @@ +#!/usr/bin/env bash + +set -e + +. $(dirname $0)/post-process-helpers + +# Parse our options +while getopts "n:o:" OPT; do + case "${OPT}" in + o) output="${OPTARG}";; + n) base_name="${OPTARG}";; + :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; + \?) error "unknown option '%s'\n" "${OPTARG}";; + esac +done + +# Already vendored tarball, nothing to do +if tar tf ${output} | grep -q "^[^/]*/vendor" ; then + exit 0 +fi + +unpack ${base_name} ${output} + +# Do the Go vendoring +pushd ${base_name} > /dev/null + +# Generate go.mod if it doesn't exist +if [ ! -f go.mod ] && [ -n "${BR_GOMOD}" ]; then + go mod init ${BR_GOMOD} +fi + +go mod vendor -v -modcacherw +popd > /dev/null + +repack ${base_name} ${output}
Signature · 1 line
2.29.2
Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
Thomas, all,
support/download/go-post-process line 35
Quoted 11 lines · Thomas Petazzoni
The logic to generate go.mod when not available is moved to the download post-process helper, as it must be done before "go mod vendor" is executed. Also, "go mod init" is used instead of manually crafting go.mod. This was suggested by Christian Stewart <[email protected]>. The Go module name is passed down to go-post-process using the
BR_GOMODenvironment variable.support/download/go-post-process new file +4
-0+go mod vendor -v -modcacherw +popd > /dev/null + +repack ${base_name} ${output}
This works fine in most cases, however, this causes inconsistent hashes.
On my system, the hash is correct after re-pack. But on 2 of my users machines, the hash ends up different.
So something specific to the host machine must be influencing the hash of the repacked archive.
It's either something in "go mod vendor" that's not deterministic, or maybe host Go is being used instead of the one compiled by Buildroot, or perhaps it's not possible to always get deterministic hashes this way.
Can we possibly switch to hashing the download /before/ re-packing it, given that Go verifies the hashes in go.sum automatically as part of the "go mod vendor" step?
Go mod vendor will fail if any of the dependencies are different, the checksums are included in go.sum
Thanks & best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
Hello Christian,
Quoted 5 lines · On Thu, 29 Jul 2021 13:17:09 -0700 Christian Stewart <[email protected]> wrote:
This works fine in most cases, however, this causes inconsistent hashes.
On my system, the hash is correct after re-pack. But on 2 of my users machines, the hash ends up different.
Quoted 7 lines
So something specific to the host machine must be influencing the hash of the repacked archive.
It's either something in "go mod vendor" that's not deterministic, or maybe host Go is being used instead of the one compiled by Buildroot, or perhaps it's not possible to always get deterministic hashes this way.
Meh :-/
Quoted 3 lines
Can we possibly switch to hashing the download /before/ re-packing it, given that Go verifies the hashes in go.sum automatically as part of the "go mod vendor" step?
This is going to be difficult, because the whole principle of this patch series is to integrate the vendoring within the download step, i.e *before* the hash is verified. For example, the hash is verified even if you're just extracting, and not downloading.
One possibility that we had discussed in the past, but that Yann didn't like was to create one tarball for the upstream code, and another for the vendored dependencies.
Based on the difficulties, I would really like to understand a little bit why the archive is not deterministic. Could you compare the contents of the download stuff between your different machines, and the contents of the tarball, and see what changes?
Quoted 2 lines
Go mod vendor will fail if any of the dependencies are different, the checksums are included in go.sum
Is the presence of go.sum mandatory?
Best regards,
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
Christian, all,
Quoted 10 lines · Thomas Petazzoni
Hello Christian,
Quoted 5 lines · On Thu, 29 Jul 2021 13:17:09 -0700 Christian Stewart <[email protected]> wrote:
This works fine in most cases, however, this causes inconsistent hashes.
On my system, the hash is correct after re-pack. But on 2 of my users machines, the hash ends up different.
Quoted 9 lines
Quoted 7 lines
So something specific to the host machine must be influencing the hash of the repacked archive.
It's either something in "go mod vendor" that's not deterministic, or maybe host Go is being used instead of the one compiled by Buildroot, or perhaps it's not possible to always get deterministic hashes this way.
Meh :-/
I think Yann already calls out what may be going on in this patch https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
The tarball format will need to follow the same or similar format we use for other source packs by using the PAX format and trimming some metadata being included in the archive. GNU format tarballs are not guaranteed to be consistent across versions.
Signature · 9 lines
Vincent Fazio Embedded Software Engineer - ATS Extreme Engineering Solutions, Inc http://www.xes-inc.com _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
A message that is not in the archive (sent off-list or not yet received): CA+h8R2onPMjOuvC0U6iM8QbhuAQQ9=aQ-yB-rWQkCbhpxcdiHw@mail.gmail.com
Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
Christian, All,
On 2021-07-31 21:59 -0700, Christian Stewart spake thusly:
Quoted 6 lines
Quoted 2 lines · Vincent
I think Yann already calls out what may be going on in this patch
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/I think this is probably correct, the .tar.gz needs to have deterministic formatting.
Can you try usign the tar helper in support/download/helpers instead?
Quoted 4 lines
In this particular case the user was trying to build on Kali 2.
Doesn't buildroot compile host-gzip and host-tar for this? I'm wondering why it's not the same between host machines.
Since 2020.02, we are no longer building host-tar, unless for very old versions (i.e. before 1.27), thanks to some tar trickery vy Vincent. See:
cbe95b1a455b support/download: add helper to generate a reproducible archive
We also only build host-gzip if the host gzip is pigz (or missing).
Quoted 12 lines
Quoted 5 lines · Thomas Petazzoni
Quoted 1 line
This works fine in most cases, however, this causes inconsistent hashes.
Based on the difficulties, I would really like to understand a little bit why the archive is not deterministic. Could you compare the contents of the download stuff between your different machines, and the contents of the tarball, and see what changes?
The two versions are attached. The actual contents of the two are identical. But for some reason the formatting of embiggen-disk-bad.tar.gz is different:
Quoted 5 lines
00000030: 8232 63de 4282 92dd acff fb79 6740 eae2
Starting at "2dd acff" - in "embiggen-disk-good" it's
00000030: 8232 63de 4282 96dd acff fb79 6740 eae6
So, I zcat both archives, and the two tar are identical!
$ sha1sum *.tar
dbb8338ab4acaff1e232c67d3b46602d61114b53 embiggen-disk-bad.tar
dbb8338ab4acaff1e232c67d3b46602d61114b53 embiggen-disk-good.tarSo I also hashed the archives you provided, and they too are identical:
$ sha1sum *.tar.gz
bd887a60c4ca60e55d062067132ccab7d85e7d95 embiggen-disk-bad.tar.gz
bd887a60c4ca60e55d062067132ccab7d85e7d95 embiggen-disk-good.tar.gzDid you mess up when sending the archives?
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
A message that is not in the archive (sent off-list or not yet received): CA+h8R2pLN_aYiQ1vp+rTMUsQAcGT88fsAiCC-i9uJpK1y0r4rw@mail.gmail.com
Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
Christian, All,
On 2021-08-01 00:24 -0700, Christian Stewart spake thusly:
Quoted 8 lines · Yann E. MORIN
Quoted 5 lines
So I also hashed the archives you provided, and they too are identical:
$ sha1sum *.tar.gz bd887a60c4ca60e55d062067132ccab7d85e7d95 embiggen-disk-bad.tar.gz bd887a60c4ca60e55d062067132ccab7d85e7d95 embiggen-disk-good.tar.gz Did you mess up when sending the archives?Not sure how that got mixed up, attached is the correct embiggen-disk-bad.tar.gz:
So, what are the differences?
First, looking at the compressed archive is useless. We really need to look at the uncompressed archives to see what is going on.
$ gunzip embiggen-disk-good.tar.gz embiggen-disk-bad.tar.gz
$ hxedump -Cv embiggen-disk-good.tar >embiggen-disk-good.tar.hex
$ hxedump -Cv embiggen-disk-bad.tar >embiggen-disk-bad.tar.hex$ diff -du embiggen-disk-good.tar.hex embiggen-disk-bad.tar.hex >embiggen-disk.diff
$ diffstat <embiggen-disk.diff
embiggen-disk-bad.tar.hex | 52 26 26 0 +++++++++++++-------------
1 file changed, 26 insertions(+), 26 deletions(-)So, there aren't so many differences, in the end. What's about them, then? Let's see the first hunk (the others are very similar):
@@ -4,10 +4,10 @@
00000030 31 61 32 33 30 38 2f 43 4f 4e 54 52 49 42 55 54 |1a2308/CONTRIBUT|
00000040 49 4e 47 2e 6d 64 00 00 00 00 00 00 00 00 00 00 |ING.md..........|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
-00000060 00 00 00 00 30 30 30 30 36 34 34 00 30 30 30 30 |....0000644.0000|
+00000060 00 00 00 00 30 30 30 30 36 36 34 00 30 30 30 30 |....0000664.0000|
00000070 30 30 30 00 30 30 30 30 30 30 30 00 30 30 30 30 |000.0000000.0000|
00000080 30 30 30 31 33 34 37 00 31 33 36 31 36 36 36 32 |0001347.13616662|
-00000090 36 31 33 00 30 32 30 34 30 33 00 20 30 00 00 00 |613.020403. 0...|
+00000090 36 31 33 00 30 32 30 34 30 35 00 20 30 00 00 00 |613.020405. 0...|
000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|So, the first obvious delta: the access mode. In the 'good' archive, it is 0644 (rw-r--r--), while in the 'bad' archive, it is 0664 (rw-rw-r--).
But Buildroot is supposed to enforce the mode, see Makefile:
70: REQ_UMASK = 0022
...
77: ifneq ($(shell umask):$(CURDIR):$(O),$(REQ_UMASK):$(CANONICAL_CURDIR):$(CANONICAL_O))
...
83: _all:
84: @umask $(REQ_UMASK) && \
85: $(MAKE) -C $(CANONICAL_CURDIR) --no-print-directory \
86: $(MAKECMDGOALS) $(EXTRAMAKEARGS)So, how comes that the two have different modes?
The second change is a bit more curious. As per the GNU tar manual [0]:
offset 0x94 (148) is 'char chksum[8]'.
So we should expect an 8-byte field (still from [0]):
The chksum field represents the simple sum of all bytes in the header block. Each 8-bit byte in the header is added to an unsigned integer, initialized to zero [...]
OK, so because the mode is different, the delta from the mode is carried over to the chksum.
If we can find why the modes are different in your two cases, we can probably solve the issue.
QED.
Still, I would prefer that we do use the tarball helper, because then we would have a single location where we could solve this kind of problems...
[0] https://www.gnu.org/software/tar/manual/html_node/Standard.html
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] http://lists.busybox.net/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
Hi Yann, all,
Some users are still running into the issue that on their system the "go mod vendor" process is producing different .tar.gz hashes than on most other systems. Yann previously checked on this and found that the access modes are wrong in the "bad" archive:
Quoted 6 lines · Yann E. MORIN
So, what are the differences?
First, looking at the compressed archive is useless. We really need to look at the uncompressed archives to see what is going on.
$ gunzip embiggen-disk-good.tar.gz embiggen-disk-bad.tar.gz
[snip]
Quoted 13 lines
So, the first obvious delta: the access mode. In the 'good' archive, it is 0644 (rw-r--r--), while in the 'bad' archive, it is 0664 (rw-rw-r--).
But Buildroot is supposed to enforce the mode, see
Makefile:70: REQ_UMASK = 0022 ... 77: ifneq ($(shell umask):$(CURDIR):$(O),$(REQ_UMASK):$(CANONICAL_CURDIR):$(CANONICAL_O)) ... 83: _all: 84: @umask $(REQ_UMASK) && \ 85: $(MAKE) -C $(CANONICAL_CURDIR) --no-print-directory \ 86: $(MAKECMDGOALS) $(EXTRAMAKEARGS)
[snip]
Quoted 2 lines
If we can find why the modes are different in your two cases, we can probably solve the issue.
Does anyone know a suitable fix for this? I recall there might also have been some other places where this source of lack of determinism when making .tar.gz was causing issues.
Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support
Hi all,
Quoted 19 lines · Christian Stewart
Some users are still running into the issue that on their system the "go mod vendor" process is producing different .tar.gz hashes than on most other systems. Yann previously checked on this and found that the access modes are wrong in the "bad" archive:
Quoted 5 lines
So, the first obvious delta: the access mode. In the 'good' archive, it is 0644 (rw-r--r--), while in the 'bad' archive, it is 0664 (rw-rw-r--).
But Buildroot is supposed to enforce the mode, see
Makefile:Quoted 8 lines
70: REQ_UMASK = 0022 ... 77: ifneq ($(shell umask):$(CURDIR):$(O),$(REQ_UMASK):$(CANONICAL_CURDIR):$(CANONICAL_O)) ... 83: _all: 84: @umask $(REQ_UMASK) && \ 85: $(MAKE) -C $(CANONICAL_CURDIR) --no-print-directory \ 86: $(MAKECMDGOALS) $(EXTRAMAKEARGS)
Quoted 5 lines
[snip]
Quoted 2 lines
If we can find why the modes are different in your two cases, we can probably solve the issue.
I have adjusted the code from the "support for Go vendoring" patch, which I now see was never merged into mainline (yet),
The following now uses the "mk_tar_gz" helper, as Yann suggests to do:
(quoted diff) +0 -0
b/support/download/post-process-helpers
old mode 100644
new mode 100755
index bed8df2577..0517abcc24
support/download/post-process-helpers +3 -3
@@ -1,3 +1,5 @@ +# Source the mk_tar_gz helper. +. "${0%/*}/helpers" unpack() { dest="$1"
@@ -8,23 +10,23 @@ unpack() { } repack() { - src="$1" + base_name="$1" tarball="$2" - - # Generate the archive, sort with the C locale so that it is
reproducible.
- find "$(basename ${src})" -not -type d -print0 >files.list
- LC_ALL=C sort -z <files.list >files.list.sorted
+ wdir="$(pwd)"# let's use a fixed hardcoded date to be reproducible date="2020-02-06 01:02:03 +0000"
- # Create GNU-format tarballs, since that's the format of the
tarballs on
- # sources.buildroot.org and used in the *.hash files
- tar cf new.tar --null --verbatim-files-from --numeric-owner
--format=gnu \
- --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
- gzip -6 -n <new.tar >new.tar.gz
+ # use the helper to create a reproducible archive
+ # note: the current working dir contains base_name.
+ mk_tar_gz "${base_name}" "${base_name}" "${date}" "${wdir}/new.tar.gz"
+
+ # replace the old tarball
mv "${tarball}" "${tarball}".old
mv new.tar.gz "${tarball}"
rm "${tarball}".old
rm -rf ${src} }
+
+# Keep this line and the following as last lines in this file. +# vim: ft=bash
I think this might fix the issue, if so, shall I re-submit the go vendoring series with the adjustments?
Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 6/12 package/tinifier: new package
In reply to Thomas PetazzoniThis is a Go package that needs vendor modules to be downloaded at build time.
--- DEVELOPERS | 1 + package/Config.in | 1 + package/tinifier/Config.in | 10 ++++++++++ package/tinifier/tinifier.hash | 3 +++ package/tinifier/tinifier.mk | 13 +++++++++++++ 5 files changed, 28 insertions(+) create mode 100644 package/tinifier/Config.in create mode 100644 package/tinifier/tinifier.hash create mode 100644 package/tinifier/tinifier.mk
DEVELOPERS +1 -0
@@ -2590,6 +2590,7 @@ F: package/rtc-tools/ F: package/sam-ba/ F: package/scons/ F: package/squashfs/ +F: package/tinifier/ F: package/wayland/ F: package/weston/ F: support/testing/tests/boot/test_syslinux.py
package/Config.in +1 -0
@@ -298,6 +298,7 @@ comment "Graphic applications" source "package/rrdtool/Config.in" source "package/stellarium/Config.in" source "package/tesseract-ocr/Config.in" + source "package/tinifier/Config.in" comment "Graphic libraries" source "package/cegui/Config.in"
package/tinifier/Config.in new file +10 -0
@@ -0,0 +1,10 @@ +config BR2_PACKAGE_TINIFIER + bool "tinifier" + depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS + help + CLI tool for images compressing + + This tool uses tinypng.com API endpoint for compressing your + local jpg/png images (it supports parallel jobs). + + https://github.com/tarampampam/tinifier
package/tinifier/tinifier.hash new file +3 -0
@@ -0,0 +1,3 @@ +# Locally calculated +sha256 707a1d9e55aab8c83b65bb10f3ec8c3bc094d77cd576b266f821d9f39133ac3c tinifier-2.1.0.tar.gz +sha256 791d8fd993ace44d4d83e2f4820a64d5ad3e37412f029afad46d17a9259de2b6 LICENSE
package/tinifier/tinifier.mk new file +13 -0
@@ -0,0 +1,13 @@ +################################################################################ +# +# tinifier +# +################################################################################ + +TINIFIER_VERSION = 2.1.0 +TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION)) +TINIFIER_LICENSE = MIT +TINIFIER_LICENSE_FILES = LICENSE +TINIFIER_GOMOD = tinifier + +$(eval $(golang-package))
Signature · 1 line
2.29.2
PATCH v2 7/12 package/pkg-cargo.mk: introduce the cargo package infrastructure
In reply to Thomas PetazzoniFrom: Patrick Havelange <[email protected]>
In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.
--- package/Makefile.in | 1 + package/pkg-cargo.mk | 140 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 141 insertions(+) create mode 100644 package/pkg-cargo.mk
package/Makefile.in +1 -0
@@ -427,3 +427,4 @@ include package/pkg-waf.mk include package/pkg-golang.mk include package/pkg-meson.mk include package/pkg-qmake.mk +include package/pkg-cargo.mk
package/pkg-cargo.mk new file +140 -0
140 lines not on this page · raw message
Signature · 1 line
2.29.2
PATCH v2 8/12 docs/manual/cargo: document the cargo-package infrastructure
In reply to Thomas PetazzoniFrom: Patrick Havelange <[email protected]>
The Buildroot manual was already providing some details on how to integrate Cargo packages, and those details now need to be updated with a proper documentation for the cargo-package infrastructure.
--- docs/manual/adding-packages-cargo.txt | 88 ++++++++++++--------------- 1 file changed, 39 insertions(+), 49 deletions(-)
docs/manual/adding-packages-cargo.txt +39 -49
@@ -1,7 +1,7 @@ // -*- mode:doc; -*- // vim: set syntax=asciidoc: -=== Integration of Cargo-based packages +=== Infrastructure for Cargo-based packages Cargo is the package manager for the Rust programming language. It allows the user to build programs or libraries written in Rust, but it also downloads and
@@ -10,7 +10,7 @@ called "crates". [[cargo-package-tutorial]] -==== Cargo-based package's +Config.in+ file +==== +cargo-package+ tutorial The +Config.in+ file of Cargo-based package 'foo' should contain:
@@ -25,11 +25,7 @@ The +Config.in+ file of Cargo-based package 'foo' should contain: 08: http://foosoftware.org/foo/ --------------------------- -==== Cargo-based package's +.mk+ file - -Buildroot does not (yet) provide a dedicated package infrastructure for -Cargo-based packages. So, we will explain how to write a +.mk+ file for such a -package. Let's start with an example: +And the +.mk+ file for this package should contain: ------------------------------ 01: ################################################################################
@@ -44,52 +40,48 @@ package. Let's start with an example: 10: FOO_LICENSE = GPL-3.0+ 11: FOO_LICENSE_FILES = COPYING 12: -13: FOO_DEPENDENCIES = host-rustc -14: -15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo -16: -17: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE) -18: -19: FOO_CARGO_OPTS = \ -20: $(if $(BR2_ENABLE_DEBUG),,--release) \ -21: --target=$(RUSTC_TARGET_NAME) \ -22: --manifest-path=$(@D)/Cargo.toml -23: -24: define FOO_BUILD_CMDS -25: $(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \ -26: cargo build $(FOO_CARGO_OPTS) -27: endef -28: -29: define FOO_INSTALL_TARGET_CMDS -30: $(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \ -31: $(TARGET_DIR)/usr/bin/foo -32: endef -33: -34: $(eval $(generic-package)) +13: $(eval $(cargo-package)) -------------------------------- -The Makefile starts with the definition of the standard variables for package -declaration (lines 7 to 11). +The Makefile starts with the definition of the standard variables for +package declaration (lines 7 to 11). + +As seen in line 13, it is based on the +cargo-package+ +infrastructure. Cargo will be invoked automatically by this +infrastructure to build and install the package. + +It is still possible to define custom build commands or install +commands (i.e. with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS). +Those will then replace the commands from the cargo infrastructure. + +==== +cargo-package+ reference + +The main macros for the Cargo package infrastructure are ++cargo-package+ for target packages and +host-cargo-package+ for host +packages. + +Just like the generic infrastructure, the Cargo infrastructure works +by defining a number of variables before calling the +cargo-package+ +or +host-cargo-package+ macros. -As seen in line 34, it is based on the -xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines -the variables required by this particular infrastructure, where Cargo is -invoked: +First, all the package metadata information variables that exist in +the generic infrastructure also exist in the Cargo infrastructure: ++FOO_VERSION+, +FOO_SOURCE+, +FOO_PATCH+, +FOO_SITE+, ++FOO_DEPENDENCIES+, +FOO_LICENSE+, +FOO_LICENSE_FILES+, etc. -* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required - to configure the cross-compilation of the package are passed via - +FOO_CONF_OPTS+. +A few additional variables, specific to the Cargo infrastructure, can +also be defined. Many of them are only useful in very specific cases, +typical packages will therefore only use a few of them. -* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on - the target. +* +FOO_CARGO_ENV+ can be used to pass additional variables in the + environment of +cargo+ invocations. It used at both build and + installation time -In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to -contain +host-cargo+. +* +FOO_CARGO_BUILD_OPTS+ can be used to pass additional options to + +cargo+ at build time. -To sum it up, to add a new Cargo-based package, the Makefile example can be -copied verbatim then edited to replace all occurences of +FOO+ with the -uppercase name of the new package and update the values of the standard -variables. +* +FOO_CARGO_INSTALL_OPTS+ can be used to pass additional options to + +cargo+ at install time. ==== About Dependencies Management
@@ -99,9 +91,7 @@ automatically them. This step can also be performed independently, via the +cargo fetch+ command. Cargo maintains a local cache of the registry index and of git checkouts of the -crates, whose location is given by +$CARGO_HOME+. As seen in the package -Makefile example at line 15, this environment variable is set to -+$(HOST_DIR)/share/cargo+. +crates, whose location is given by +$CARGO_HOME+. This dependency download mechanism is not convenient when performing an offline build, as Cargo will fail to fetch the dependencies. In that case, it is advised
Signature · 1 line
2.29.2
PATCH v2 9/12 package/ripgrep: convert to cargo infrastructure
In reply to Thomas PetazzoniFrom: Patrick Havelange <[email protected]>
--- package/ripgrep/ripgrep.mk | 28 +--------------------------- 1 file changed, 1 insertion(+), 27 deletions(-)
package/ripgrep/ripgrep.mk +1 -27
@@ -9,30 +9,4 @@ RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION)) RIPGREP_LICENSE = MIT RIPGREP_LICENSE_FILES = LICENSE-MIT -RIPGREP_DEPENDENCIES = host-rustc -RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo - -RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_BIN_SUBDIR) - -RIPGREP_CARGO_OPTS = \ - --target=$(RUSTC_TARGET_NAME) \ - --manifest-path=$(@D)/Cargo.toml - -ifeq ($(BR2_ENABLE_DEBUG),y) -RIPGREP_CARGO_BIN_SUBDIR = debug -else -RIPGREP_CARGO_OPTS += --release -RIPGREP_CARGO_BIN_SUBDIR = release -endif - -define RIPGREP_BUILD_CMDS - $(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \ - cargo build $(RIPGREP_CARGO_OPTS) -endef - -define RIPGREP_INSTALL_TARGET_CMDS - $(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \ - $(TARGET_DIR)/usr/bin/rg -endef - -$(eval $(generic-package)) +$(eval $(cargo-package))
Signature · 1 line
2.29.2
PATCH v2 10/12 package/sentry-cli: convert to host-cargo-package infrastructure
In reply to Thomas Petazzoni--- package/sentry-cli/sentry-cli.mk | 22 ++-------------------- 1 file changed, 2 insertions(+), 20 deletions(-)
package/sentry-cli/sentry-cli.mk +2 -20
@@ -9,24 +9,6 @@ SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION)) SENTRY_CLI_LICENSE = BSD-3-clause SENTRY_CLI_LICENSE_FILES = LICENSE -HOST_SENTRY_CLI_DEPENDENCIES = host-rustc host-zlib +HOST_SENTRY_CLI_DEPENDENCIES = host-zlib -HOST_SENTRY_CLI_CARGO_ENV = \ - CARGO_HOME=$(HOST_DIR)/share/cargo \ - RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))" - -HOST_SENTRY_CLI_CARGO_OPTS = \ - --release \ - --manifest-path=$(@D)/Cargo.toml - -define HOST_SENTRY_CLI_BUILD_CMDS - $(HOST_MAKE_ENV) $(HOST_SENTRY_CLI_CARGO_ENV) \ - cargo build $(HOST_SENTRY_CLI_CARGO_OPTS) -endef - -define HOST_SENTRY_CLI_INSTALL_CMDS - $(INSTALL) -D -m 0755 $(@D)/target/release/sentry-cli \ - $(HOST_DIR)/bin/sentry-cli -endef - -$(eval $(host-generic-package)) +$(eval $(host-cargo-package))
Signature · 1 line
2.29.2
PATCH v2 11/12 support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages
In reply to Thomas PetazzoniThis commit adds support/download/cargo-post-process to perform the vendoring on Cargo packages, and enables it in package/pkg-cargo.mk.
Since it changes the contents of the tarballs for ripgrep and sentry-cli, it changes their hashes. To not have a different hash for the same version of ripgrep and sentry-cli, we bump their versions. It has to be done in the same commit as the Cargo vendoring to make the series bisectable.
The <pkg>_LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.
--- package/pkg-cargo.mk | 13 ++++++++-- package/ripgrep/ripgrep.hash | 2 +- package/ripgrep/ripgrep.mk | 2 +- package/sentry-cli/sentry-cli.hash | 2 +- package/sentry-cli/sentry-cli.mk | 2 +- support/download/cargo-post-process | 38 +++++++++++++++++++++++++++++ 6 files changed, 53 insertions(+), 6 deletions(-) create mode 100755 support/download/cargo-post-process
package/pkg-cargo.mk +11 -2
@@ -37,10 +37,17 @@ define inner-cargo-package # We need host-rustc to run cargo -$(2)_DEPENDENCIES += host-rustc +$(2)_DOWNLOAD_DEPENDENCIES += host-rustc $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo +$(2)_DOWNLOAD_POST_PROCESS = cargo +$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo + +# Due to vendoring, it is pretty likely that not all licenses are +# listed in <pkg>_LICENSE. +$(2)_LICENSE += , vendored dependencies licenses probably not listed + # Note: in all the steps below, we "cd" into the build directory to # execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the # manifest-path. Indeed while the latter seems to work, it in fact
@@ -50,7 +57,9 @@ $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo # directory, its configuration file will not be taken into account. # # Also, we pass: -# * --offline to prevent cargo from downloading anything +# * --offline to prevent cargo from downloading anything: all +# dependencies should have been built by the download post +# process logic # * --locked to force cargo to use the Cargo.lock file, which ensures # that a fixed set of dependency versions is used
package/ripgrep/ripgrep.hash +1 -1
@@ -1,3 +1,3 @@ # Locally calculated -sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz +sha256 80d7f07325f4d485c5e4baf061b0376b45a497ee7a1c540d7894057bb9ac3a59 ripgrep-12.1.1.tar.gz sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT
package/ripgrep/ripgrep.mk +1 -1
@@ -4,7 +4,7 @@ # ################################################################################ -RIPGREP_VERSION = 0.8.1 +RIPGREP_VERSION = 12.1.1 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION)) RIPGREP_LICENSE = MIT RIPGREP_LICENSE_FILES = LICENSE-MIT
package/sentry-cli/sentry-cli.hash +1 -1
@@ -1,3 +1,3 @@ # locally calculated -sha256 5d0f7acf6a139d1c1716b9a8ff76c8bfaab09104ba663c957bb9a5dba2ffbabd sentry-cli-1.57.0.tar.gz +sha256 a657dd1a46e3de044deff4c311b7276c5a9409582e707e6e6a78b0cf712591c4 sentry-cli-1.59.0.tar.gz sha256 9503def7b54ceb6e3cd182fd59bc05d3a30d7eae481e65aaba4b495133c83c14 LICENSE
package/sentry-cli/sentry-cli.mk +1 -1
@@ -4,7 +4,7 @@ # ################################################################################ -SENTRY_CLI_VERSION = 1.57.0 +SENTRY_CLI_VERSION = 1.59.0 SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION)) SENTRY_CLI_LICENSE = BSD-3-clause SENTRY_CLI_LICENSE_FILES = LICENSE
support/download/cargo-post-process new file +38 -0
@@ -0,0 +1,38 @@ +#!/usr/bin/env bash + +set -e + +. $(dirname $0)/post-process-helpers + +while getopts "n:o:" OPT; do + case "${OPT}" in + o) output="${OPTARG}";; + n) base_name="${OPTARG}";; + :) error "option '%s' expects a mandatory argument\n" "${OPTARG}";; + \?) error "unknown option '%s'\n" "${OPTARG}";; + esac +done + +# Already vendored tarball, nothing to do +if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then + exit 0 +fi + +unpack ${base_name} ${output} + +# Do the Cargo vendoring +pushd ${base_name} > /dev/null +cargo vendor --locked VENDOR +echo $? +# Create the local .cargo/config with vendor info +mkdir -p .cargo/ +cat <<EOF >.cargo/config +[source.crates-io] +replace-with = "vendored-sources" + +[source.vendored-sources] +directory = "VENDOR" +EOF +popd > /dev/null + +repack ${base_name} ${output}
Signature · 1 line
2.29.2
PATCH v2 12/12 docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management
In reply to Thomas PetazzoniNow that we have vendoring support for Cargo packages, let's rewrite the dependency management section in a more accurate way.
We drop the part about the local cache of the registry, because +CARGO_HOME+ in Buildroot points to $(HOST_DIR)/share/cargo, which is not shared between builds nor preserved accross builds, so its effect as a cache is limited.
--- docs/manual/adding-packages-cargo.txt | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-)
docs/manual/adding-packages-cargo.txt +10 -12
@@ -85,15 +85,13 @@ typical packages will therefore only use a few of them. ==== About Dependencies Management -A crate can depend on other libraries from crates.io or git repositories, listed -in its Cargo.toml file. Before starting a build, Cargo usually downloads -automatically them. This step can also be performed independently, via the -+cargo fetch+ command. - -Cargo maintains a local cache of the registry index and of git checkouts of the -crates, whose location is given by +$CARGO_HOME+. - -This dependency download mechanism is not convenient when performing an offline -build, as Cargo will fail to fetch the dependencies. In that case, it is advised -to generate a tarball of the dependencies using the +cargo vendor+ and add it to -+FOO_EXTRA_DOWNLOADS+. +A crate can depend on other libraries from crates.io or git +repositories, listed in its Cargo.toml file. Buildroot automatically +takes care of downloading such dependencies as part of the download +step of packages that use the +cargo-package+ infrastructure. Such +dependencies are then kept together with the package source code in +the tarball cached in Buildroot's +DL_DIR+, and therefore the hash of +the package's tarball includes such dependencies. + +This mechanism ensures that any change in the dependencies will be +detected, and allows the build to be performed completely offline.
Signature · 1 line
2.29.2