Skip to contentSkip to conversation

Support for Cargo and Go vendoring

PATCH Superseded Thomas Petazzoni 7 participants 38 messages Nov 19, 2020 – Dec 16, 2020

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Support for Cargo and Go vendoring

Hello,

Here is a patch series that implements support for vendoring of Cargo and Go packages.

As you're aware, many Cargo and Go packages describe their dependencies in a very language-specific way, and expect such dependencies to be downloaded by their language-specific tools.

This series builds on top of the work from Patrick Havelange <[email protected]> who worked on the Cargo integration, but extends it to also cover the case of Go, showing that we have a solution solving both of these language-specific package managers. For the Go vendoring, I've also used the indication provided by Christian Stewart <[email protected]>.

Overall, the solution consists in a concept of "download post-process helper" is introduced in the download infrastructure. These are shell scripts that can be invoked right after the download is done, but before the tarball is hash-checked and stored in DL_DIR. The idea is that such "download post-process helpers" can run the language-specific vendoring logic to fetch the package dependencies. Thanks to this, the tarball in DL_DIR for a given package not only contains the package source code itself, but also the source code of its Cargo or Go dependencies. The tarball hash covers the entire source code, the complete tarball is cached in DL_DIR, in the primary site and backup site, and the build can fully be done offline.

Such "download post-process helpers" are registered by means of the <pkg>_DOWNLOAD_POST_PROCESS variable by the appropriate package infrastructure.

This series is also available at:

https://github.com/tpetazzoni/buildroot/commits/pkg-mgr

More specifically, the series goes like this:

- PATCH 1: add the concept of download post-processing in the download helper support/download/dl-wrapper

- PATCH 2: add the <pkg>_DOWNLOAD_POST_PROCESS variable to the package infrastructure, which gets passed down to support/download/dl-wrapper

- PATCH 3: add the <pkg>_DL_ENV variable to the package infrastructure, which allows to pass extra variables in the download wrapper environment. This will be used to pass extra variables specific to Cargo/Go vendoring.

- PATCH 4: add a set of two helper shell functions that will be used by the post processing scripts

- PATCH 5: implements the Go vendoring itself

- PATCH 6: adds package/tinifier, which is one package that requires Go vendoring. I don't particularly need this package, it was just added because the other Go packages we have in Buildroot today don't need vendoring.

- PATCH 7: introduces a cargo-package infrastructure, for both target and host packages

- PATCH 8: documents the added cargo-package infrastructure

- PATCH 9 and 10: converts ripgrep and sentry-cli to the cargo-package and host-cargo-package infrastructures respecitively

- PATCH 11: implements the Cargo vendoring itself. Note that we need to bump the ripgrep and sentry-cli package at the same time, as with the vendoring their tarball contents change and therefore their hash changes

- PATCH 12: rewrite the cargo-package documentation about dependency management

Best regards,

Thomas

Patrick Havelange (3):
package/pkg-cargo.mk: introduce the cargo package infrastructure docs/manual/cargo: document the cargo-package infrastructure package/ripgrep: convert to cargo infrastructure

Thomas Petazzoni (9):
support/download/dl-wrapper: add concept of download post-processing package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable package/pkg-download.mk: add <pkg>_DL_ENV variable support/download/post-process-helpers: add helper function for post
process scripts
support/download/go-post-process: implement Go vendoring support package/tinifier: new package
package/sentry-cli: convert to host-cargo-package infrastructure support/download/cargo-post-process, package/pkg-cargo.mk: enable
vendoring for Cargo packages
docs/manual/adding-packages-cargo.txt: rewrite explanation about
dependency management

 DEVELOPERS                            |   1 +
 docs/manual/adding-packages-cargo.txt | 106 ++++++++++------------
 package/Config.in                     |   1 +
 package/Makefile.in                   |   1 +
 package/pkg-cargo.mk                  | 126 ++++++++++++++++++++++++++
 package/pkg-download.mk               |   5 +-
 package/pkg-golang.mk                 |   7 +-
 package/ripgrep/ripgrep.hash          |   2 +-
 package/ripgrep/ripgrep.mk            |  30 +-----
 package/sentry-cli/sentry-cli.hash    |   2 +-
 package/sentry-cli/sentry-cli.mk      |  24 +----
 package/tinifier/Config.in            |  10 ++
 package/tinifier/tinifier.hash        |   3 +
 package/tinifier/tinifier.mk          |  13 +++
 support/download/cargo-post-process   |  38 ++++++++
 support/download/dl-wrapper           |   9 +-
 support/download/go-post-process      |  29 ++++++
 support/download/post-process-helpers |  30 ++++++
 18 files changed, 324 insertions(+), 113 deletions(-)
 create mode 100644 package/pkg-cargo.mk
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
 create mode 100755 support/download/cargo-post-process
 create mode 100755 support/download/go-post-process
 create mode 100644 support/download/post-process-helpers
Signature · 1 line
2.28.0
#1 lore raw

PATCH 1/12 support/download/dl-wrapper: add concept of download post-processing

In order to support package managers such as Cargo (Rust) or Go, we want to run some custom logic after the main download, but before packing the tarball and checking the hash.

To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then support/download/<something>-post-process will be called.

---
 support/download/dl-wrapper | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
support/download/dl-wrapper +8 -1
index 3315bd410e..2d74554213 100755
@@ -25,7 +25,7 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() {
         r)  recurse="-r";;
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
+        p)  post_process="${OPTARG}";;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -135,6 +136,12 @@ main() {
             continue
         fi
 
+        if [ -n "${post_process}" ] ; then
+                ${OLDPWD}/support/download/${post_process}-post-process \
+                         -o "${tmpf}" \
+                         -n "${raw_base_name}"
+        fi
+
         # cd back to free the temp-dir, so we can remove it later
         cd "${OLDPWD}"
 
Signature · 1 line
2.28.0
#2 lore raw

PATCH 2/12 package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable

In reply to Thomas Petazzoni

This will allow packages to register than a download post-processing is needed. Note that this variable is intentionally not documented: it is an internal variable meant to be set by package infrastructures, not directly by packages.

---
 package/pkg-download.mk | 1 +
 1 file changed, 1 insertion(+)
package/pkg-download.mk +1 -0
index 951d2fb554..175b9d7dba 100644
@@ -108,6 +108,7 @@ define DOWNLOAD
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
Signature · 1 line
2.28.0
#3 lore raw

PATCH 3/12 package/pkg-download.mk: add <pkg>_DL_ENV variable

In reply to Thomas Petazzoni

This variable can be used by package to pass extra environment variables to the download logic. It will be used for the Go/Cargo vendoring.

The <pkg>_DL_ENV variable is intentionally not documented: at this point, it is not meant to be used by packages directly, but only by package infrastructures.

---
 package/pkg-download.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-download.mk +3 -1
index 175b9d7dba..c914d016e2 100644
@@ -99,7 +99,9 @@ endif
 
 define DOWNLOAD
 	$(Q)mkdir -p $($(2)_DL_DIR)
-	$(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
+	$(Q)$(EXTRA_ENV) \
+		$($(2)_DL_ENV) \
+		flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
 		-c '$($(2)_DL_VERSION)' \
 		-d '$($(2)_DL_DIR)' \
 		-D '$(DL_DIR)' \
Signature · 1 line
2.28.0
#4 lore raw

PATCH 4/12 support/download/post-process-helpers: add helper function for post process scripts

In reply to Thomas Petazzoni

download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an unpack() function and a repack() function.

---
 support/download/post-process-helpers | 30 +++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 support/download/post-process-helpers
support/download/post-process-helpers new file +30 -0
new file mode 100644
index 0000000000..bed8df2577
@@ -0,0 +1,30 @@
+
+unpack() {
+        dest="$1"
+        tarball="$2"
+
+        mkdir ${dest}
+        tar -C ${dest} --strip-components=1 -xf ${tarball}
+}
+
+repack() {
+        src="$1"
+        tarball="$2"
+
+        # Generate the archive, sort with the C locale so that it is reproducible.
+        find "$(basename ${src})" -not -type d -print0 >files.list
+        LC_ALL=C sort -z <files.list >files.list.sorted
+
+        # let's use a fixed hardcoded date to be reproducible
+        date="2020-02-06 01:02:03 +0000"
+
+        # Create GNU-format tarballs, since that's the format of the tarballs on
+        # sources.buildroot.org and used in the *.hash files
+        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
+            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
+        gzip -6 -n <new.tar >new.tar.gz
+        mv "${tarball}" "${tarball}".old
+        mv new.tar.gz "${tarball}"
+        rm "${tarball}".old
+        rm -rf ${src}
+}
Signature · 1 line
2.28.0
#5 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

In reply to Thomas Petazzoni

This commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure.

---
 package/pkg-golang.mk            |  7 ++++++-
 support/download/go-post-process | 29 +++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100755 support/download/go-post-process
package/pkg-golang.mk +6 -1
index 3813e1c406..150445bf17 100644
@@ -47,7 +47,7 @@ $(2)_BUILD_OPTS += \
 	-p $(PARALLEL_JOBS)
 
 # Target packages need the Go compiler on the host.
-$(2)_DEPENDENCIES += host-go
+$(2)_DOWNLOAD_DEPENDENCIES += host-go
 
 $(2)_BUILD_TARGETS ?= .
 
@@ -72,6 +72,11 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
 # If the go.mod file does not exist, one is written with this root path.
 $(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
 
+$(2)_DOWNLOAD_POST_PROCESS = go
+$(2)_DL_ENV = \
+	$(HOST_GO_COMMON_ENV) \
+	GOPROXY=direct
+
 # Generate a go.mod file if it doesn't exist. Note: Go is configured
 # to use the "vendor" dir and not make network calls.
 define $(2)_GEN_GOMOD
support/download/go-post-process new file +29 -0
new file mode 100755
index 0000000000..01073aee8b
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+# Parse our options
+while getopts "n:o:" OPT; do
+        case "${OPT}" in
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
+	exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Go vendoring
+pushd ${base_name} > /dev/null
+go mod vendor -v
+popd > /dev/null
+
+repack ${base_name} ${output}
Signature · 1 line
2.28.0
#6 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

Hi Thomas,

Quoted 26 lines · Thomas Petazzoni

This commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure.

+++ b/support/download/go-post-process

support/download/go-post-process new file +19 -0
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+# Parse our options
+while getopts "n:o:" OPT; do
+        case "${OPT}" in
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
+       exit 0

support/download/go-post-process line 29

Quoted 10 lines
support/download/go-post-process new file +10 -0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Go vendoring
+pushd ${base_name} > /dev/null
+go mod vendor -v
+popd > /dev/null
+
+repack ${base_name} ${output}

This looks good. There's only one problem - if the package does not come with a go.mod file (which is the case with nvidia-container-toolkit).

In this case we need to run the $(2)_GEN_GOMOD script before running the download post-process.

Today, this is written as so and creates a go.mod if it doesn't already exist:

# Generate a go.mod file if it doesn't exist. Note: Go is configured

# to use the "vendor" dir and not make network calls.
define $(2)_GEN_GOMOD
if [ ! -f $$(@D)/go.mod ]; then \
  printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \
fi
endef
$(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD

Otherwise it looks like it works well, with this one tweak.

Best regards,
Christian Stewart

#7 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

Hello Christian,

Thanks for the feedback!

Quoted 9 lines · On Wed, 25 Nov 2020 12:28:31 -0800 Christian Stewart <[email protected]> wrote:

This looks good. There's only one problem - if the package does not come with a go.mod file (which is the case with nvidia-container-toolkit).

In this case we need to run the $(2)_GEN_GOMOD script before running the download post-process.

Today, this is written as so and creates a go.mod if it doesn't already exist:

Quoted 8 lines

# Generate a go.mod file if it doesn't exist. Note: Go is configured

# to use the "vendor" dir and not make network calls.
define $(2)_GEN_GOMOD
if [ ! -f $$(@D)/go.mod ]; then \
  printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \
fi
endef
$(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD

Ah, good point. Do you have an example of such a package (publicly available)? That would help testing a solution.

I guess the best solution is to modify the POST_PATCH_HOOKS to be done by the download post-process script ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#8 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

Hi Thomas,

There are a number of packages that don't have go.mod files yet, but might have files from other package managers in the Go ecosystem.

Quoted 26 lines · Thomas Petazzoni

Hello Christian,

Thanks for the feedback!

Quoted 9 lines · On Wed, 25 Nov 2020 12:28:31 -0800 Christian Stewart <[email protected]> wrote:

This looks good. There's only one problem - if the package does not come with a go.mod file (which is the case with nvidia-container-toolkit).

In this case we need to run the $(2)_GEN_GOMOD script before running the download post-process.

Today, this is written as so and creates a go.mod if it doesn't already exist:

Quoted 8 lines

# Generate a go.mod file if it doesn't exist. Note: Go is configured

# to use the "vendor" dir and not make network calls.
define $(2)_GEN_GOMOD
if [ ! -f $$(@D)/go.mod ]; then \
  printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \
fi
endef
$(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD
Quoted 2 lines

Ah, good point. Do you have an example of such a package (publicly available)? That would help testing a solution.

Yes: https://github.com/NVIDIA/nvidia-container-runtime

Buildroot package:
https://github.com/skiffos/buildroot/commit/983f30d31c7fc8011fbfb57dd3abef1751c3e4d7

And some more like it in this tree:

https://github.com/skiffos/buildroot/tree/pkg-mgr

Quoted 3 lines

I guess the best solution is to modify the POST_PATCH_HOOKS to be done by the download post-process script ?

The go module system will actually convert those other formats faithfully into the go.mod and go.sum format.

If we add the patch that I sent in the previous email, which runs "go mod init" just before "go mod vendor," it will read the legacy formats in this step.

Alternatively we could do something which was rejected in one of my original go.mod support revisions - adding the go.mod and go.sum files into the Buildroot tree.

To generate these:

1. Clone the repo
2. "go mod init" and "go mod vendor"
3. Copy go.mod and go.sum to buildroot tree
4. These are copied (if exist) into the package just before running
"go mod vendor" in the post-process.

The reasoning behind this is to provide an exact version for all of the dependencies locked in the Buildroot tree so that the output is deterministic. We might get hash mismatches otherwise.

Best,
Christian

Quoted 6 lines

Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

#9 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

In reply to Christian Stewart

Hi Thomas,

Quoted 7 lines · Christian Stewart
Quoted 4 lines · Thomas Petazzoni

This commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure.

Quoted 6 lines

This looks good. There's only one problem - if the package does not come with a go.mod file (which is the case with nvidia-container-toolkit).

In this case we need to run the $(2)_GEN_GOMOD script before running the download post-process.

I was able to solve this with the following patch:

package/pkg-download.mk +1 -0
index c914d016e2..29462ebaa4 100644
@@ -112,6 +112,7 @@ define DOWNLOAD
  -o '$($(2)_DL_DIR)/$(notdir $(1))' \
  $(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
  $(if $($(2)_GIT_SUBMODULES),-r) \
+ $(if $($(2)_GOMOD),-g '$($(2)_GOMOD)') \
  $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
  $(QUIET) \
  -- \
support/download/dl-wrapper +4 -2
index 2d74554213..2fc530f24f 100755
@@ -17,7 +17,7 @@
 # We want to catch any unexpected failure, and exit immediately.
 set -e
 
-export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
+export BR_BACKEND_DL_GETOPTS=":hc:d:g:o:n:N:H:ru:qf:e"
 
 main() {
     local OPT OPTARG
@@ -25,11 +25,12 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
+    while getopts ":c:d:D:g:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
         D)  old_dl_dir="${OPTARG}";;
+        g)  gomod_init="${OPTARG}";;
         o)  output="${OPTARG}";;
         n)  raw_base_name="${OPTARG}";;
         N)  base_name="${OPTARG}";;
@@ -138,6 +139,7 @@ main() {
 
         if [ -n "${post_process}" ] ; then
                 ${OLDPWD}/support/download/${post_process}-post-process \
+                         -g "${gomod_init}" \
                          -o "${tmpf}" \
                          -n "${raw_base_name}"
         fi
support/download/go-post-process +6 -1
index 01073aee8b..796fa08e38 100755
@@ -5,8 +5,9 @@ set -e
 . $(dirname $0)/post-process-helpers
 
 # Parse our options
-while getopts "n:o:" OPT; do
+while getopts "n:g:o:" OPT; do
         case "${OPT}" in
+        g)  gomod_init="${OPTARG}";;
         o)  output="${OPTARG}";;
         n)  base_name="${OPTARG}";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
@@ -23,6 +24,10 @@ unpack ${base_name} ${output}
 
 # Do the Go vendoring
 pushd ${base_name} > /dev/null
+if [ -n "${gomod_init}" ]; then
+    # Note: does nothing if go.mod already exists.
+    go mod init ${gomod_init}
+fi
 go mod vendor -v
 popd > /dev/null
 

Not sure if this is the best way to pass the go module identifier into the script, but running "go mod init" will conditionally create the go.mod file if it doesn't already exist, just before running "go mod vendor."

Best,
Christian

#10 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

Hello,

Quoted 26 lines · On Wed, 25 Nov 2020 12:45:59 -0800 Christian Stewart <[email protected]> wrote:
package/pkg-download.mk +1 -0
index c914d016e2..29462ebaa4 100644
@@ -112,6 +112,7 @@ define DOWNLOAD
  -o '$($(2)_DL_DIR)/$(notdir $(1))' \
  $(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
  $(if $($(2)_GIT_SUBMODULES),-r) \
+ $(if $($(2)_GOMOD),-g '$($(2)_GOMOD)') \
  $(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
  $(QUIET) \
  -- \
support/download/dl-wrapper +1 -1
index 2d74554213..2fc530f24f 100755
@@ -17,7 +17,7 @@
 # We want to catch any unexpected failure, and exit immediately.
 set -e
 
-export BR_BACKEND_DL_GETOPTS=":hc:d:o:n:N:H:ru:qf:e"
+export BR_BACKEND_DL_GETOPTS=":hc:d:g:o:n:N:H:ru:qf:e"
 
 main() {
     local OPT OPTARG

support/download/dl-wrapper main() { · line 33

Quoted 10 lines
support/download/dl-wrapper +2 -1
@@ -25,11 +25,12 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
+    while getopts ":c:d:D:g:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
         D)  old_dl_dir="${OPTARG}";;
+        g)  gomod_init="${OPTARG}";;

Unfortunately, this is not really a very good solution, because the dl-wrapper is generic, and if we start adding language-specific and backend-specific options, it's really going to become a nightmare :-/

support/download/dl-wrapper main() {

Quoted 5 lines
support/download/dl-wrapper +3 -0
 # Do the Go vendoring
 pushd ${base_name} > /dev/null
+if [ -n "${gomod_init}" ]; then
+    # Note: does nothing if go.mod already exists.
+    go mod init ${gomod_init}

So this cannot simply be conditional on go.mod existing or not, because we need the name of the Go module ?

Best regards,

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#11 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

Hi Thomas,

Quoted 10 lines · Thomas Petazzoni

support/download/dl-wrapper main() { · line 33

Quoted 9 lines

# Parse our options; anything after '--' is for the backend

support/download/dl-wrapper +2 -1
-    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
+    while getopts ":c:d:D:g:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
         D)  old_dl_dir="${OPTARG}";;
+        g)  gomod_init="${OPTARG}";;
Quoted 3 lines

Unfortunately, this is not really a very good solution, because the dl-wrapper is generic, and if we start adding language-specific and backend-specific options, it's really going to become a nightmare :-/

Pass it as an environment variable? I don't see what the problem is aside from the optargs.

Quoted 8 lines

support/download/go-post-process unpack ${base_name} ${output} · line 29

Quoted 5 lines

# Do the Go vendoring
pushd ${base_name} > /dev/null

support/download/go-post-process +3 -0
+if [ -n "${gomod_init}" ]; then
+    # Note: does nothing if go.mod already exists.
+    go mod init ${gomod_init}

So this cannot simply be conditional on go.mod existing or not, because we need the name of the Go module ?

The "go mod init" command will actually do nothing if go.mod already exists.

We do need the name of the go module though in the case that go.mod does not exist.

Best regards,
Christian

#12 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

Thomas,

Quoted 4 lines · Christian Stewart
Quoted 2 lines

So this cannot simply be conditional on go.mod existing or not, because we need the name of the Go module ?

The "go mod init" command will actually do nothing if go.mod already exists.

Scratch that, "go mod init" will actually fail if go.mod already exists. So we should instead:

if [ ! -f go.mod ] && [ -n "${gomod_init}" ]; then

    go mod init ${gomod_init}

fi

Apologies for confusion on that.

Best,
Christian

#13 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

In reply to Thomas Petazzoni

Thomas, All,

I've tried out the new go vendoring support and I am running into an issue with cleaning the build directory when using this series. I've added a new package on your pkg-mgr branch called maddy with the following at package/maddy/maddy.mk:

MADDY_VERSION = 0.4.2
MADDY_SITE = $(call github,foxcpp,maddy,v$(MADDY_VERSION))

$(eval $(golang-package))

Then I run the following commands:

make maddy
make clean

The error I get is as follows:

rm -rf /home/ryan/projects/br/br-pkg-mgr/output/target
/home/ryan/projects/br/br-pkg-mgr/output/images
/home/ryan/projects/br/br-pkg-mgr/output/host \
/home/ryan/projects/br/br-pkg-mgr/output/build
/home/ryan/projects/br/br-pkg-mgr/output/staging \
/home/ryan/projects/br/br-pkg-mgr/output/legal-info
/home/ryan/projects/br/br-pkg-mgr/output/graphs
/home/ryan/projects/br/br-pkg-mgr/output/per-package
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/README.md':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_amd64.go':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_amd64.s':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_test.go':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash.go':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/go.mod':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhsum/xxhsum.go':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhsum/.gitignore':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe.go':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_other.go':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/LICENSE.txt':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/go.sum':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/.travis.yml':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_safe.go':
Permission denied
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go':
Permission denied

There are about 1000 more lines with this error. The only way I can do a clean is by running:

sudo rm -rf output/*

Here are the permissions on one of the files:

$ ls -al /home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go
-r--r--r-- 1 ryan ryan 402 Nov 25 15:01
/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go

I've tried that same package on the latest version of master and I do not see this error. With that same maddy.mk file on master I am able to successfully build the package and run make clean. I believe the error I see it is due to the vendoring of go modules. Before today, I've never build a go package so I'm not familiar with how those files could be have incorrect permissions.

Thanks,
-Ryan

Quoted 11 lines · Thomas Petazzoni

This commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure.

---
 package/pkg-golang.mk            |  7 ++++++-
 support/download/go-post-process | 29 +++++++++++++++++++++++++++++
 2 files changed, 35 insertions(+), 1 deletion(-)
 create mode 100755 support/download/go-post-process

[...]

#14 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

Hi Ryan, all,

Quoted 25 lines · Ryan Barnett

I've tried out the new go vendoring support and I am running into an issue with cleaning the build directory when using this series. I've added a new package on your pkg-mgr branch called maddy with the following at package/maddy/maddy.mk:

make maddy
make clean

The error I get is as follows:

rm -rf /home/ryan/projects/br/br-pkg-mgr/output/target /home/ryan/projects/br/br-pkg-mgr/output/images
/home/ryan/projects/br/br-pkg-mgr/output/host \
/home/ryan/projects/br/br-pkg-mgr/output/build
/home/ryan/projects/br/br-pkg-mgr/output/staging \
/home/ryan/projects/br/br-pkg-mgr/output/legal-info
/home/ryan/projects/br/br-pkg-mgr/output/graphs
/home/ryan/projects/br/br-pkg-mgr/output/per-package
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/README.md':
Permission denied

There are about 1000 more lines with this error. The only way I can do a clean is by running:

Quoted 7 lines

sudo rm -rf output/*

Here are the permissions on one of the files:

$ ls -al /home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go
-r--r--r-- 1 ryan ryan 402 Nov 25 15:01
/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/xxhash_unsafe_test.go

This is because the Go tool sets restrictive permissions on the gopath/pkg/mod tree to prevent source code tools from editing the dependencies code on the filesystem.

Relevant Go issues:

- https://github.com/golang/go/issues/27455

myitcv says: "This is, as @mark-rushakoff pointed out, working as intended. The integrity of pkg/mod/** is important. Hence it is not writable."

- https://github.com/golang/go/issues/31481

We can solve this by adding "-modcacherw" flag to the "go mod" commands. I will test this and send some more info.

Best regards,
Christian Stewart

#15 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

Hi Ryan, Thomas, all,

I have a modified version of the go-post-process patch with fixes here:

https://github.com/skiffos/buildroot/commit/5f31774f54402f07c1d83aa407430db5152e6de3

Quoted 11 lines · Christian Stewart
Quoted 10 lines · Ryan Barnett

I've tried out the new go vendoring support and I am running into an issue with cleaning the build directory when using this series. I've added a new package on your pkg-mgr branch called maddy with the following at package/maddy/maddy.mk:

make maddy
make clean

The error I get is as follows:

Quoted 11 lines
Quoted 11 lines

rm -rf /home/ryan/projects/br/br-pkg-mgr/output/target /home/ryan/projects/br/br-pkg-mgr/output/images
/home/ryan/projects/br/br-pkg-mgr/output/host \
/home/ryan/projects/br/br-pkg-mgr/output/build
/home/ryan/projects/br/br-pkg-mgr/output/staging \
/home/ryan/projects/br/br-pkg-mgr/output/legal-info
/home/ryan/projects/br/br-pkg-mgr/output/graphs
/home/ryan/projects/br/br-pkg-mgr/output/per-package
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/README.md':
Permission denied

Here is the fix for this issue specifically (using the modcacherw flag):

package/pkg-golang.mk +1 -0
index 150445bf17..b22720add6 100644
@@ -42,6 +42,7 @@ define inner-golang-package
 
 $(2)_BUILD_OPTS += \
  -ldflags "$$($(2)_LDFLAGS)" \
+ -modcacherw \
  -tags "$$($(2)_TAGS)" \
  -trimpath \
  -p $(PARALLEL_JOBS)
support/download/go-post-process +4 -2
index 8830ee56c1..1e5441f6e6 100755
@@ -24,10 +24,12 @@ unpack ${base_name} ${output}
 
 # Do the Go vendoring
 pushd ${base_name} > /dev/null
+# modcacherw option leaves directories in the module cache at their default
+# permissions rather than making them read-only.
 if [ ! -f go.mod ] && [ -n "${gomod_init}" ]; then
-    go mod init ${gomod_init}
+    go mod init -modcacherw ${gomod_init}
 fi
-go mod vendor -v
+go mod vendor -modcacherw -v
 popd > /dev/null
 
 repack ${base_name} ${output}

Best regards,
Christian Stewart

#16 lore raw

PATCH 5/12 support/download/go-post-process: implement Go vendoring support

Christian, Thomas,

Quoted 31 lines · Christian Stewart

Hi Ryan, Thomas, all,

I have a modified version of the go-post-process patch with fixes here:

https://github.com/skiffos/buildroot/commit/5f31774f54402f07c1d83aa407430db5152e6de3

Quoted 11 lines · Christian Stewart
Quoted 10 lines · Ryan Barnett

I've tried out the new go vendoring support and I am running into an issue with cleaning the build directory when using this series. I've added a new package on your pkg-mgr branch called maddy with the following at package/maddy/maddy.mk:

make maddy
make clean

The error I get is as follows:

Quoted 11 lines
Quoted 11 lines

rm -rf /home/ryan/projects/br/br-pkg-mgr/output/target /home/ryan/projects/br/br-pkg-mgr/output/images
/home/ryan/projects/br/br-pkg-mgr/output/host \
/home/ryan/projects/br/br-pkg-mgr/output/build
/home/ryan/projects/br/br-pkg-mgr/output/staging \
/home/ryan/projects/br/br-pkg-mgr/output/legal-info
/home/ryan/projects/br/br-pkg-mgr/output/graphs
/home/ryan/projects/br/br-pkg-mgr/output/per-package
rm: cannot remove '/home/ryan/projects/br/br-pkg-mgr/output/host/share/go-path/pkg/mod/github.com/cespare/xxhash/v2 at v2.1.1/README.md':
Permission denied

Quoted 1 line

Here is the fix for this issue specifically (using the modcacherw flag):

[...]

I manually added the '-modcacherw' flag to the pkg-golang.mk and go-post-process files and I was able to now successfully run 'make clean'

Thanks for the quick response and the fix.

-Ryan

#17 lore raw

PATCH 6/12 package/tinifier: new package

In reply to Thomas Petazzoni

This is a Go package that needs vendor modules to be downloaded at build time.

---
 DEVELOPERS                     |  1 +
 package/Config.in              |  1 +
 package/tinifier/Config.in     | 10 ++++++++++
 package/tinifier/tinifier.hash |  3 +++
 package/tinifier/tinifier.mk   | 13 +++++++++++++
 5 files changed, 28 insertions(+)
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
DEVELOPERS +1 -0
index d0c9c16423..eb97cfeac0 100644
@@ -2590,6 +2590,7 @@ F:	package/rtc-tools/
 F:	package/sam-ba/
 F:	package/scons/
 F:	package/squashfs/
+F:	package/tinifier/
 F:	package/wayland/
 F:	package/weston/
 F:	support/testing/tests/boot/test_syslinux.py
package/Config.in +1 -0
index d32a271113..a93959ada8 100644
@@ -298,6 +298,7 @@ comment "Graphic applications"
 	source "package/rrdtool/Config.in"
 	source "package/stellarium/Config.in"
 	source "package/tesseract-ocr/Config.in"
+	source "package/tinifier/Config.in"
 
 comment "Graphic libraries"
 	source "package/cegui/Config.in"
package/tinifier/Config.in new file +10 -0
new file mode 100644
index 0000000000..fbadfe6bd9
@@ -0,0 +1,10 @@
+config BR2_PACKAGE_TINIFIER
+	bool "tinifier"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	help
+	  CLI tool for images compressing
+
+	  This tool uses tinypng.com API endpoint for compressing your
+	  local jpg/png images (it supports parallel jobs).
+
+	  https://github.com/tarampampam/tinifier
package/tinifier/tinifier.hash new file +3 -0
new file mode 100644
index 0000000000..146700817b
@@ -0,0 +1,3 @@
+# Locally calculated
+sha256  707a1d9e55aab8c83b65bb10f3ec8c3bc094d77cd576b266f821d9f39133ac3c  tinifier-2.1.0.tar.gz
+sha256  791d8fd993ace44d4d83e2f4820a64d5ad3e37412f029afad46d17a9259de2b6  LICENSE
package/tinifier/tinifier.mk new file +13 -0
new file mode 100644
index 0000000000..b47d265a8e
@@ -0,0 +1,13 @@
+################################################################################
+#
+# tinifier
+#
+################################################################################
+
+TINIFIER_VERSION = 2.1.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
+TINIFIER_LICENSE = MIT
+TINIFIER_LICENSE_FILES = LICENSE
+TINIFIER_GOMOD = tinifier
+
+$(eval $(golang-package))
Signature · 1 line
2.28.0
#18 lore raw

PATCH 6/12 package/tinifier: new package

Thomas,

I've taken a preliminary look at your pkg-mgr patch series and will be providing additional feedback over the next few days as I have a chance to test it further.

Quoted 15 lines · Thomas Petazzoni

This is a Go package that needs vendor modules to be downloaded at build time.

---
 DEVELOPERS                     |  1 +
 package/Config.in              |  1 +
 package/tinifier/Config.in     | 10 ++++++++++
 package/tinifier/tinifier.hash |  3 +++
 package/tinifier/tinifier.mk   | 13 +++++++++++++
 5 files changed, 28 insertions(+)
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk

[...]

Quoted 6 lines
package/tinifier/tinifier.mk new file +0 -0
new file mode 100644
index 0000000000..b47d265a8e

package/tinifier/tinifier.mk line 10

Quoted 10 lines
package/tinifier/tinifier.mk new file +10 -0
@@ -0,0 +1,13 @@
+################################################################################
+#
+# tinifier
+#
+################################################################################
+
+TINIFIER_VERSION = 2.1.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
+TINIFIER_LICENSE = MIT
+TINIFIER_LICENSE_FILES = LICENSE

I took a look at the legal-info side in regards to downloading packages with the post-processing support. This has been discussed previously on the patch "[v3,09/10] package/ripgrep: add legal-info for dependencies":

https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/

When I ran 'make legal-info' for the tinifier package all that is mentioned in the 'manifest.csv' file for the package is:

"tinifier","2.1.0","MIT","LICENSE","tinifier-2.1.0.tar.gz","https://github.com/tarampampam/tinifier/archive/v2.1.0","skeleton-init-common
[unknown] skeleton-init-none [unknown] toolchain-external-bootlin
[unknown]"

This doesn't give any indication or warnings that dependencies were downloaded or that other open source license could be needed by including this package. As user of buildroot who may not have any knowledge in regards to 'go' or 'cargo'. When they see tinifier row in the manifest.csv file, they could just think that the tinifier package would only have/introduce the MIT license to their product. Which is not the case because it downloads the following vendor packages:

bou.ke/monkey v1.0.2
github.com/dustin/go-humanize v1.0.0
github.com/jessevdk/go-flags v1.4.1-0.20181221193153-c0795c8afcf4 github.com/json-iterator/go v1.1.10 github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.1 // indirect github.com/olekukonko/tablewriter v0.0.4 github.com/schollz/progressbar/v3 v3.3.3 github.com/stretchr/testify v1.6.1

I understand add/showing these license and how to exactly handle this are future additions. However, I re-read the section in regards to the "legal-info" in the buildroot manual I came across this:

Moreover, due to technical limitations, Buildroot does not produce some material that you will or may need, such as the toolchain source code for some of the external toolchains and the Buildroot source code itself. When you run +make legal-info+, Buildroot produces warnings in the +README+ file to inform you of relevant material that could not be saved.

So would it be possible to put a warning into the 'legal-info/README' file that not all the dependency licenses could be downloaded/added to manifest.csv file?

Maybe a hint could be given to go take a look at the tinifier/go.mod under the requires section to figure out the licenses. I think this could be a good temporary solution under a more "dynamic legal-info" infrastructure could be introduced. Would there be a way to detect if there are any vendor packages downloaded and then add a warning in the 'legal-info/README' file easily?

Thanks,
-Ryan Barnett

Quoted 5 lines
package/tinifier/tinifier.mk new file +3 -1
+TINIFIER_GOMOD = tinifier
+
+$(eval $(golang-package))
--

2.28.0

#19 lore raw

PATCH 6/12 package/tinifier: new package

Ryan, All,

On 2020-11-21 10:37 -0600, Ryan Barnett spake thusly:

Quoted 4 lines
Quoted 2 lines · Thomas Petazzoni

This is a Go package that needs vendor modules to be downloaded at build time.

[--SNIP--]

Quoted 22 lines
Quoted 6 lines
package/tinifier/tinifier.mk new file +0 -0
new file mode 100644
index 0000000000..b47d265a8e

package/tinifier/tinifier.mk line 10

Quoted 10 lines
package/tinifier/tinifier.mk new file +10 -0
@@ -0,0 +1,13 @@
+################################################################################
+#
+# tinifier
+#
+################################################################################
+
+TINIFIER_VERSION = 2.1.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
+TINIFIER_LICENSE = MIT
+TINIFIER_LICENSE_FILES = LICENSE

I took a look at the legal-info side in regards to downloading packages with the post-processing support. This has been discussed previously on the patch "[v3,09/10] package/ripgrep: add legal-info for dependencies":

Quoted 1 line

https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/

legal-info is also something Thomas and I discussed and IRC when he posted his series.

We know it is not perfect, but this can be extended in a followup series.

Quoted 10 lines

When I ran 'make legal-info' for the tinifier package all that is mentioned in the 'manifest.csv' file for the package is:

   "tinifier","2.1.0","MIT","LICENSE","tinifier-2.1.0.tar.gz","https://github.com/tarampampam/tinifier/archive/v2.1.0","skeleton-init-common

[unknown] skeleton-init-none [unknown] toolchain-external-bootlin [unknown]"

This doesn't give any indication or warnings that dependencies were downloaded or that other open source license could be needed by including this package.

To simplify the series, my position as a first step would be to extend the FOO_LICENSE list in the infra, with just a very short notice, something like:

    FOO_LICENSE += , vendored licenses not listed
Quoted 17 lines

As user of buildroot who may not have any
knowledge in regards to 'go' or 'cargo'. When they see tinifier row in the manifest.csv file, they could just think that the tinifier package would only have/introduce the MIT license to their product. Which is not the case because it downloads the following vendor packages:

bou.ke/monkey v1.0.2
github.com/dustin/go-humanize v1.0.0
github.com/jessevdk/go-flags v1.4.1-0.20181221193153-c0795c8afcf4 github.com/json-iterator/go v1.1.10 github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.1 // indirect github.com/olekukonko/tablewriter v0.0.4 github.com/schollz/progressbar/v3 v3.3.3 github.com/stretchr/testify v1.6.1

Quoted 2 lines

I understand add/showing these license and how to exactly handle this are future additions.

My idea is that the go/cargo/... infras would be responsible for providing "some kind of" post-legal-info hooks, so they can extend the licenses list and license files list as well.

But really, I would like to make that a next step, so that the technical side of the support for package managers can get in sooner rather than later.

If we can not in the end come up woth a satifying licensing report for those (or for some of those) package managers, we would at least have support for building them.

FTR, Thomas and I already adressed that issue quite a while ago, and we concluded that it was not so obvious as one may initially think (I'd have to dig my IRC logs to find the explanations...)

Quoted 9 lines

However, I re-read the section in regards to
the "legal-info" in the buildroot manual I came across this:

Moreover, due to technical limitations, Buildroot does not produce some material that you will or may need, such as the toolchain source code for some of the external toolchains and the Buildroot source code itself. When you run +make legal-info+, Buildroot produces warnings in the +README+ file to inform you of relevant material that could not be saved.

Quoted 3 lines

So would it be possible to put a warning into the 'legal-info/README' file that not all the dependency licenses could be downloaded/added to manifest.csv file?

With the added blurb I suggest above, I think we would be pretty much covered, no?

Regards,
Yann E. MORIN.

Quoted 19 lines

Maybe a hint could be given to go take a look at the tinifier/go.mod under the requires section to figure out the licenses. I think this could be a good temporary solution under a more "dynamic legal-info" infrastructure could be introduced. Would there be a way to detect if there are any vendor packages downloaded and then add a warning in the 'legal-info/README' file easily?

Thanks,
-Ryan Barnett

Quoted 5 lines
package/tinifier/tinifier.mk new file +3 -1
+TINIFIER_GOMOD = tinifier
+
+$(eval $(golang-package))
--

2.28.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#20 lore raw

PATCH 6/12 package/tinifier: new package

Yann,

Quoted 24 lines · Yann E. MORIN
Quoted 6 lines

I took a look at the legal-info side in regards to downloading packages with the post-processing support. This has been discussed previously on the patch "[v3,09/10] package/ripgrep: add legal-info for dependencies":

https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/

legal-info is also something Thomas and I discussed and IRC when he posted his series.

We know it is not perfect, but this can be extended in a followup series.

Quoted 2 lines

I understand add/showing these license and how to exactly handle this are future additions.

My idea is that the go/cargo/... infras would be responsible for providing "some kind of" post-legal-info hooks, so they can extend the licenses list and license files list as well.

But really, I would like to make that a next step, so that the technical side of the support for package managers can get in sooner rather than later.

Quoted 7 lines

If we can not in the end come up woth a satifying licensing report for those (or for some of those) package managers, we would at least have support for building them.

FTR, Thomas and I already adressed that issue quite a while ago, and we concluded that it was not so obvious as one may initially think (I'd have to dig my IRC logs to find the explanations...)

If you're open to building + running host Go code as part of this process, then it is straightforward to extract the module dependency graph and the Licenses.

If we're doing this bash + makefile purism, then this is definitely not easy.

Best,
Christian

#21 lore raw

PATCH 6/12 package/tinifier: new package

Christian, Yann,

Quoted 37 lines · Christian Stewart

Yann,

Quoted 24 lines · Yann E. MORIN
Quoted 6 lines

I took a look at the legal-info side in regards to downloading packages with the post-processing support. This has been discussed previously on the patch "[v3,09/10] package/ripgrep: add legal-info for dependencies":

https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/

legal-info is also something Thomas and I discussed and IRC when he posted his series.

We know it is not perfect, but this can be extended in a followup series.

Quoted 2 lines

I understand add/showing these license and how to exactly handle this are future additions.

My idea is that the go/cargo/... infras would be responsible for providing "some kind of" post-legal-info hooks, so they can extend the licenses list and license files list as well.

But really, I would like to make that a next step, so that the technical side of the support for package managers can get in sooner rather than later.

Quoted 7 lines

If we can not in the end come up woth a satifying licensing report for those (or for some of those) package managers, we would at least have support for building them.

FTR, Thomas and I already adressed that issue quite a while ago, and we concluded that it was not so obvious as one may initially think (I'd have to dig my IRC logs to find the explanations...)

Quoted 5 lines

If you're open to building + running host Go code as part of this process, then it is straightforward to extract the module dependency graph and the Licenses.

If we're doing this bash + makefile purism, then this is definitely not easy.

Same with cargo - unfortunately though for cargo it isn't baked in and you need to install a plugin to do it. Not the best...but possible.

Sam

#22 lore raw

PATCH 6/12 package/tinifier: new package

In reply to Yann E. MORIN

Yann, All,

Quoted 50 lines · Yann E. MORIN

Ryan, All,

On 2020-11-21 10:37 -0600, Ryan Barnett spake thusly:

Quoted 4 lines
Quoted 2 lines · Thomas Petazzoni

This is a Go package that needs vendor modules to be downloaded at build time.

[--SNIP--]

Quoted 22 lines
Quoted 6 lines
package/tinifier/tinifier.mk new file +0 -0
new file mode 100644
index 0000000000..b47d265a8e

package/tinifier/tinifier.mk line 10

Quoted 10 lines
package/tinifier/tinifier.mk new file +10 -0
@@ -0,0 +1,13 @@
+################################################################################
+#
+# tinifier
+#
+################################################################################
+
+TINIFIER_VERSION = 2.1.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
+TINIFIER_LICENSE = MIT
+TINIFIER_LICENSE_FILES = LICENSE

I took a look at the legal-info side in regards to downloading packages with the post-processing support. This has been discussed previously on the patch "[v3,09/10] package/ripgrep: add legal-info for dependencies":

Quoted 1 line

https://patchwork.ozlabs.org/project/buildroot/patch/20200220160119.3407-9-patrick.havelange at essensium.com/

legal-info is also something Thomas and I discussed and IRC when he posted his series.

We know it is not perfect, but this can be extended in a followup series.

Quoted 10 lines

When I ran 'make legal-info' for the tinifier package all that is mentioned in the 'manifest.csv' file for the package is:

   "tinifier","2.1.0","MIT","LICENSE","tinifier-2.1.0.tar.gz","https://github.com/tarampampam/tinifier/archive/v2.1.0","skeleton-init-common

[unknown] skeleton-init-none [unknown] toolchain-external-bootlin [unknown]"

This doesn't give any indication or warnings that dependencies were downloaded or that other open source license could be needed by including this package.

Quoted 5 lines

To simplify the series, my position as a first step would be to extend the FOO_LICENSE list in the infra, with just a very short notice, something like:

    FOO_LICENSE += , vendored licenses not listed

Adding this within the go/cargo infrastructure works for since it informs a user that the package has licenses not listed.

Would a note in the buildroot manual be useful under the legal-info section mention the go/cargo license limitations be useful as well?

Quoted 28 lines
Quoted 17 lines

As user of buildroot who may not have any
knowledge in regards to 'go' or 'cargo'. When they see tinifier row in the manifest.csv file, they could just think that the tinifier package would only have/introduce the MIT license to their product. Which is not the case because it downloads the following vendor packages:

bou.ke/monkey v1.0.2
github.com/dustin/go-humanize v1.0.0
github.com/jessevdk/go-flags v1.4.1-0.20181221193153-c0795c8afcf4 github.com/json-iterator/go v1.1.10 github.com/kami-zh/go-capturer v0.0.0-20171211120116-e492ea43421d github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect github.com/modern-go/reflect2 v1.0.1 // indirect github.com/olekukonko/tablewriter v0.0.4 github.com/schollz/progressbar/v3 v3.3.3 github.com/stretchr/testify v1.6.1

Quoted 2 lines

I understand add/showing these license and how to exactly handle this are future additions.

My idea is that the go/cargo/... infras would be responsible for providing "some kind of" post-legal-info hooks, so they can extend the licenses list and license files list as well.

But really, I would like to make that a next step, so that the technical side of the support for package managers can get in sooner rather than later.

Quoted 7 lines

If we can not in the end come up woth a satifying licensing report for those (or for some of those) package managers, we would at least have support for building them.

FTR, Thomas and I already adressed that issue quite a while ago, and we concluded that it was not so obvious as one may initially think (I'd have to dig my IRC logs to find the explanations...)

The patchworks link I provided in my original feedback email contains the IRC convo between you and Thomas.

Quoted 9 lines
Quoted 9 lines

However, I re-read the section in regards to
the "legal-info" in the buildroot manual I came across this:

Moreover, due to technical limitations, Buildroot does not produce some material that you will or may need, such as the toolchain source code for some of the external toolchains and the Buildroot source code itself. When you run +make legal-info+, Buildroot produces warnings in the +README+ file to inform you of relevant material that could not be saved.

Quoted 6 lines
Quoted 3 lines

So would it be possible to put a warning into the 'legal-info/README' file that not all the dependency licenses could be downloaded/added to manifest.csv file?

With the added blurb I suggest above, I think we would be pretty much covered, no?

Yes the suggested blurb in the go/cargo infrastructure is good with me.

Thanks,
-Ryan

#23 lore raw

PATCH 6/12 package/tinifier: new package

In reply to Yann E. MORIN

Hello,

Quoted 5 lines · On Sat, 21 Nov 2020 19:04:18 +0100 "Yann E. MORIN" <[email protected]>…

legal-info is also something Thomas and I discussed and IRC when he posted his series.

We know it is not perfect, but this can be extended in a followup series.

Right.

Quoted 12 lines
Quoted 11 lines

When I ran 'make legal-info' for the tinifier package all that is mentioned in the 'manifest.csv' file for the package is:

   "tinifier","2.1.0","MIT","LICENSE","tinifier-2.1.0.tar.gz","https://github.com/tarampampam/tinifier/archive/v2.1.0","skeleton-init-common

[unknown] skeleton-init-none [unknown] toolchain-external-bootlin [unknown]"

This doesn't give any indication or warnings that dependencies were downloaded or that other open source license could be needed by including this package.

Quoted 5 lines

To simplify the series, my position as a first step would be to extend the FOO_LICENSE list in the infra, with just a very short notice, something like:

    FOO_LICENSE += , vendored licenses not listed

The problem is that we do not know if the vendoring was done by Buildroot itself, or if vendored dependencies are provided directly in the upstream repository, so it's difficult to add this only if Buildroot has done the vendoring.

In addition, in both cases, the FOO_LICENSE of the package may very well be completely accurate, taking into account all vendored dependencies. Indeed, now that they are properly downloaded, nothing prevents from having correct FOO_LICENSE and FOO_LICENSE_FILES values for those packages.

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#24 lore raw

PATCH 7/12 package/pkg-cargo.mk: introduce the cargo package infrastructure

In reply to Thomas Petazzoni

From: Patrick Havelange <[email protected]>

In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

[Thomas: add support for host-cargo-package]
---
 package/Makefile.in  |   1 +
 package/pkg-cargo.mk | 123 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+)
 create mode 100644 package/pkg-cargo.mk
package/Makefile.in +1 -0
index 51f5cbce4f..2af123f36d 100644
@@ -427,3 +427,4 @@ include package/pkg-waf.mk
 include package/pkg-golang.mk
 include package/pkg-meson.mk
 include package/pkg-qmake.mk
+include package/pkg-cargo.mk
package/pkg-cargo.mk new file +123 -0

123 lines not on this page · raw message

Signature · 1 line
2.28.0
#25 lore raw

PATCH 7/12 package/pkg-cargo.mk: introduce the cargo package infrastructure

Thomas,

I've been testing out your pkg-mgr manager series branch over the last few days. I've run into an issue with the cargo package infrastructure in trying to build a package after doing a 'make clean'. The error is follows:

Quoted 1 line
Quoted 1 line
Quoted 1 line

ripgrep 12.1.1 Extracting

gzip -d -c /home/ryan/.br-dl/ripgrep/ripgrep-12.1.1.tar.gz | /home/ryan/projects/br/br-pkg-mgr/output/host/bin/tar --strip-components=1 -C
/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1 -xf -

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

ripgrep 12.1.1 Patching
ripgrep 12.1.1 Configuring
ripgrep 12.1.1 Building

PATH="/home/ryan/projects/br/br-pkg-mgr/output/host/bin:/home/ryan/projects/br/br-pkg-mgr/output/host/sbin:/home/ryan/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin" PATH="/home/ryan/projects/br/br-pkg-mgr/output/host/bin:/home/ryan/projects/br/br-pkg-mgr/output/host/sbin:/home/ryan/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin" AR="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-ar"
AS="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-as"
LD="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-ld"
NM="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-nm"
CC="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-gcc"
GCC="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-gcc"
CPP="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-cpp"
CXX="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-g++"
FC="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-gfortran"
F77="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-gfortran"
RANLIB="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-ranlib"
READELF="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-readelf"
STRIP="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-strip"
OBJCOPY="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-objcopy"
OBJDUMP="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-objdump"
AR_FOR_BUILD="/usr/bin/ar" AS_FOR_BUILD="/usr/bin/as"
CC_FOR_BUILD="/usr/bin/gcc" GCC_FOR_BUILD="/usr/bin/gcc"
CXX_FOR_BUILD="/usr/bin/g++" LD_FOR_BUILD="/usr/bin/ld"
CPPFLAGS_FOR_BUILD="-I/home/ryan/projects/br/br-pkg-mgr/output/host/include"
CFLAGS_FOR_BUILD="-O2
-I/home/ryan/projects/br/br-pkg-mgr/output/host/include"
CXXFLAGS_FOR_BUILD="-O2
-I/home/ryan/projects/br/br-pkg-mgr/output/host/include"
LDFLAGS_FOR_BUILD="-L/home/ryan/projects/br/br-pkg-mgr/output/host/lib
-Wl,-rpath,/home/ryan/projects/br/br-pkg-mgr/output/host/lib"
FCFLAGS_FOR_BUILD=""
DEFAULT_ASSEMBLER="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-as"
DEFAULT_LINKER="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/x86_64-linux-ld"
CPPFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE
-D_FILE_OFFSET_BITS=64" CFLAGS="-D_LARGEFILE_SOURCE
-D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os "
CXXFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE
-D_FILE_OFFSET_BITS=64 -Os " LDFLAGS="" FCFLAGS=" -Os " FFLAGS=" -Os
" PKG_CONFIG="/home/ryan/projects/br/br-pkg-mgr/output/host/bin/pkg-config"
STAGING_DIR="/home/ryan/projects/br/br-pkg-mgr/output/host/x86_64-buildroot-linux-gnu/sysroot"
INTLTOOL_PERL=/usr/bin/perl
CARGO_HOME=/home/ryan/projects/br/br-pkg-mgr/output/host/share/cargo
cargo build --offline --target x86_64-unknown-linux-gnu --release
--manifest-path
/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/Cargo.toml
--locked
error: no matching package named aho-corasick found
location searched: registry https://github.com/rust-lang/crates.io-index
required by package `globset v0.4.5
(/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/crates/globset)`
As a reminder, you're using offline mode (--offline) which can
sometimes cause surprising resolution failures, if this error is too
confusing you may wish to retry without the offline flag.
make[1]: *** [package/pkg-generic.mk:250:
/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/.stamp_built]
Error 101
make: *** [Makefile:84: _all] Error 2

I get this error by running the following commands:

make ripgrep
make clean ripgrep

The defconfig I used is:

BR2_x86_64=y
BR2_x86_corei7=y
BR2_TOOLCHAIN_EXTERNAL=y BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_X86_64_CORE_I7_GLIBC_STABLE=y BR2_INIT_NONE=y
# BR2_TARGET_ENABLE_ROOT_LOGIN is not set
# BR2_TARGET_GENERIC_GETTY is not set
# BR2_TARGET_GENERIC_REMOUNT_ROOTFS_RW is not set # BR2_PACKAGE_BUSYBOX is not set

I have BR2_DL_DIR configured as follows:

declare -x BR2_DL_DIR="~/.br-dl"

I can work around the issue by running the following commands:

rm -rf ~/.br-dl/ripgrep/
make clean ripgrep

So it appears that after the rippackage has been downloaded there is an issue with doing a clean build. I'm not familiar with cargo/rust builds and how the offline component works but it appears that the introduction of this cargo infrastructure and the downloading of vendor dependencies is not currently reproducible.

Let me know if you need any additional information to reproduce the problem.

Thanks,
-Ryan

Quoted 161 lines · Thomas Petazzoni

From: Patrick Havelange <[email protected]>

In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

[Thomas: add support for host-cargo-package]
---
 package/Makefile.in  |   1 +
 package/pkg-cargo.mk | 123 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 124 insertions(+)
 create mode 100644 package/pkg-cargo.mk
package/Makefile.in +1 -0
index 51f5cbce4f..2af123f36d 100644
@@ -427,3 +427,4 @@ include package/pkg-waf.mk
 include package/pkg-golang.mk
 include package/pkg-meson.mk
 include package/pkg-qmake.mk
+include package/pkg-cargo.mk
package/pkg-cargo.mk new file +123 -1
new file mode 100644
index 0000000000..52c237aa3c
@@ -0,0 +1,123 @@
+################################################################################
+# Cargo package infrastructure
+#
+# This file implements an infrastructure that eases development of package
+# .mk files for Cargo packages. It should be used for all packages that use
+# Cargo as their build system.
+#
+# See the Buildroot documentation for details on the usage of this
+# infrastructure
+#
+# In terms of implementation, this Cargo infrastructure requires the .mk file
+# to only specify metadata information about the package: name, version,
+# download URL, etc.
+#
+# We still allow the package .mk file to override what the different steps
+# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined,
+# it is used as the list of commands to perform to build the package,
+# instead of the default Cargo behaviour. The package can also define some
+# post operation hooks.
+#
+################################################################################
+
+################################################################################
+# inner-cargo-package -- defines how the configuration, compilation and
+# installation of a cargo package should be done, implements a few hooks
+# to tune the build process for cargo specifities and calls the generic
+# package infrastructure to generate the necessary make targets
+#
+#  argument 1 is the lowercase package name
+#  argument 2 is the uppercase package name, including a HOST_ prefix
+#             for host packages
+#  argument 3 is the uppercase package name, without the HOST_ prefix
+#             for host packages
+#  argument 4 is the type (target or host)
+################################################################################
+
+define inner-cargo-package
+
+# We need host-rustc to run cargo
+$(2)_DEPENDENCIES += host-rustc
+
+$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+#
+# Build step. Only define it if not already defined by the package .mk
+# file.
+#
+ifndef $(2)_BUILD_CMDS
+ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+       $$(TARGET_MAKE_ENV) \
+               $$(TARGET_CONFIGURE_OPTS) \
+               $$($(2)_CARGO_ENV) \
+               cargo build \
+                       --offline \
+                       --target $$(RUSTC_TARGET_NAME) \
+                       $$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
+                       --manifest-path $$(@D)/Cargo.toml \
+                       --locked \
+                       $$($(2)_CARGO_BUILD_OPTS)
+endef
+else # ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+       $$(HOST_MAKE_ENV) \
+               RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+               $$($(2)_CARGO_ENV) \
+               cargo build \
+                       --offline \
+                       --release \
+                       --manifest-path $$(@D)/Cargo.toml \
+                       --locked \
+                       $$($(2)_CARGO_BUILD_OPTS)
+endef
+endif # ifeq ($(4),target)
+endif # ifndef $(2)_BUILD_CMDS
+
+#
+# Target installation step. Only define it if not already defined by
+# the package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+       $$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
+               cargo install \
+                       --target $$(RUSTC_TARGET_NAME) \
+                       --offline \
+                       --root $$(TARGET_DIR)/usr/ \
+                       --bins \
+                       --path $$(@D) \
+                       --force \
+                       --locked \
+                       $$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+       $$(HOST_MAKE_ENV) \
+               RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+               $$($(2)_CARGO_ENV) \
+               cargo install \
+                       --offline \
+                       --root $$(HOST_DIR) \
+                       --bins \
+                       --path $$(@D) \
+                       --force \
+                       --locked \
+                       $$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+# Call the generic package infrastructure to generate the necessary
+# make targets
+$(call inner-generic-package,$(1),$(2),$(3),$(4))
+
+endef
+
+################################################################################
+# cargo-package -- the target generator macro for Cargo packages
+################################################################################
+
+cargo-package = $(call inner-cargo-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)
+host-cargo-package = $(call inner-cargo-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host)
--

2.28.0

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot

#26 lore raw

PATCH 7/12 package/pkg-cargo.mk: introduce the cargo package infrastructure

Hello Ryan,

Quoted 16 lines · On Wed, 25 Nov 2020 13:46:38 -0600 Ryan Barnett <[email protected]> wrote:

error: no matching package named aho-corasick found
location searched: registry https://github.com/rust-lang/crates.io-index required by package globset v0.4.5 (/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/crates/globset) As a reminder, you're using offline mode (--offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may wish to retry without the offline flag.
make[1]: *** [package/pkg-generic.mk:250:
/home/ryan/projects/br/br-pkg-mgr/output/build/ripgrep-12.1.1/.stamp_built] Error 101
make: *** [Makefile:84: _all] Error 2

I get this error by running the following commands:

make ripgrep
make clean ripgrep

[...]

Quoted 5 lines

So it appears that after the rippackage has been downloaded there is an issue with doing a clean build. I'm not familiar with cargo/rust builds and how the offline component works but it appears that the introduction of this cargo infrastructure and the downloading of vendor dependencies is not currently reproducible.

Thanks for the report, I'll try to have a look as time permits. I'm also not that familiar with the Cargo stuff, but we'll see if I can find something.

Thanks!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#27 lore raw

PATCH 1/1 package/pkg-cargo.mk: make sure .cargo/config is used

In reply to Thomas Petazzoni

The way cargo searches for config file is based (among other rules) on the current directory. This means that if cargo is started outside of a package directory, its configuration file will not be taken into account.
So just 'cd' into it before running cargo build/install

This fix the issue where the build is failing in offline mode and only the dl/package.tar.gz is available.

Without this, it would have worked only if the CARGO_HOME cache had been populated (by running cargo vendor for example).

---
 package/pkg-cargo.mk | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)
package/pkg-cargo.mk +8 -4
index 6653f64594..2047aa80cc 100644
@@ -51,6 +51,7 @@ $(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo
 ifndef $(2)_BUILD_CMDS
 ifeq ($(4),target)
 define $(2)_BUILD_CMDS
+	cd $$(@D) && \
 	$$(TARGET_MAKE_ENV) \
 		$$(TARGET_CONFIGURE_OPTS) \
 		$$($(2)_CARGO_ENV) \
@@ -58,19 +59,20 @@ define $(2)_BUILD_CMDS
 			--offline \
 			--target $$(RUSTC_TARGET_NAME) \
 			$$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
-			--manifest-path $$(@D)/Cargo.toml \
+			--manifest-path Cargo.toml \
 			--locked \
 			$$($(2)_CARGO_BUILD_OPTS)
 endef
 else # ifeq ($(4),target)
 define $(2)_BUILD_CMDS
+	cd $$(@D) && \
 	$$(HOST_MAKE_ENV) \
 		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
 		$$($(2)_CARGO_ENV) \
 		cargo build \
 			--offline \
 			--release \
-			--manifest-path $$(@D)/Cargo.toml \
+			--manifest-path Cargo.toml \
 			--locked \
 			$$($(2)_CARGO_BUILD_OPTS)
 endef
@@ -83,13 +85,14 @@ endif # ifndef $(2)_BUILD_CMDS
 #
 ifndef $(2)_INSTALL_TARGET_CMDS
 define $(2)_INSTALL_TARGET_CMDS
+	cd $$(@D) && \
 	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
 		cargo install \
 			--target $$(RUSTC_TARGET_NAME) \
 			--offline \
 			--root $$(TARGET_DIR)/usr/ \
 			--bins \
-			--path $$(@D) \
+			--path ./ \
 			--force \
 			--locked \
 			$$($(2)_CARGO_INSTALL_OPTS)
@@ -98,6 +101,7 @@ endif
 
 ifndef $(2)_INSTALL_CMDS
 define $(2)_INSTALL_CMDS
+	cd $$(@D) && \
 	$$(HOST_MAKE_ENV) \
 		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
 		$$($(2)_CARGO_ENV) \
@@ -105,7 +109,7 @@ define $(2)_INSTALL_CMDS
 			--offline \
 			--root $$(HOST_DIR) \
 			--bins \
-			--path $$(@D) \
+			--path ./ \
 			--force \
 			--locked \
 			$$($(2)_CARGO_INSTALL_OPTS)
Signature · 1 line
2.17.1
#28 lore raw

PATCH 1/1 package/pkg-cargo.mk: make sure .cargo/config is used

Hello Patrick,

Quoted 6 lines · On Thu, 10 Dec 2020 16:48:50 +0100 Patrick Havelange…

The way cargo searches for config file is based (among other rules) on the current directory. This means that if cargo is started outside of a package directory, its configuration file will not be taken into account.
So just 'cd' into it before running cargo build/install

Quoted 7 lines

This fix the issue where the build is failing in offline mode and only the dl/package.tar.gz is available.

Without this, it would have worked only if the CARGO_HOME cache had been populated (by running cargo vendor for example).

Thanks a lot! This does fix the "make ripgrep; make clean ripgrep" issue that Ryan reported. Also, for ripgrep, I no longer see anything being rebuilt during the install step, which is good.

*However*, for host-sentry-cli, even with this patch, quite a bit of stuff gets rebuilt during the install step. Here is how it goes:

Quoted 1 line
Quoted 1 line

host-sentry-cli 1.59.0 Extracting

gzip -d -c /home/thomas/dl/sentry-cli/sentry-cli-1.59.0.tar.gz | tar --strip-components=1 -C /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 -xf -

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

host-sentry-cli 1.59.0 Patching
host-sentry-cli 1.59.0 Configuring
host-sentry-cli 1.59.0 Building

cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo build --offline --release --manifest-path Cargo.toml --locked

Compiling libc v0.2.71
Compiling cfg-if v0.1.10
Compiling autocfg v1.0.0
Compiling proc-macro2 v1.0.18
Compiling unicode-xid v0.2.1
Compiling syn v1.0.33
Compiling serde_derive v1.0.114
Compiling serde v1.0.114
Compiling lazy_static v1.4.0
Compiling pkg-config v0.3.17
Compiling getrandom v0.1.14
Compiling memchr v2.3.3
Compiling ppv-lite86 v0.2.8
Compiling fallible-iterator v0.2.0
Compiling regex-syntax v0.6.18
Compiling byteorder v1.3.4
Compiling stable_deref_trait v1.1.1
Compiling adler32 v1.1.0
Compiling rand_core v0.4.2
Compiling autocfg v0.1.7
Compiling failure_derive v0.1.8
Compiling log v0.4.8
Compiling scopeguard v1.1.0
Compiling ryu v1.0.5
Compiling rustc-demangle v0.1.16
Compiling tinyvec v0.3.3
Compiling object v0.20.0
Compiling matches v0.1.8
Compiling semver-parser v0.7.0
Compiling maybe-uninit v2.0.0
Compiling itoa v0.4.6
Compiling serde_json v1.0.56
Compiling proc-macro2 v0.4.30
Compiling encoding_index_tests v0.1.4
Compiling typenum v1.12.0
Compiling unicode-xid v0.1.0
Compiling ucd-trie v0.1.3
Compiling crc32fast v1.2.0
Compiling smallvec v1.4.0
Compiling syn v0.15.44
Compiling maplit v1.0.2
Compiling unicode-width v0.1.8
Compiling adler v0.2.2
Compiling version_check v0.9.2
Compiling percent-encoding v2.1.0
Compiling bitflags v1.2.1
Compiling rayon-core v1.7.1
Compiling curl v0.4.30
Compiling plain v0.2.3
Compiling podio v0.1.7
Compiling ahash v0.3.8
Compiling siphasher v0.3.3
Compiling fnv v1.0.7
Compiling xml-rs v0.8.3
Compiling dmsort v1.0.0
Compiling new_debug_unreachable v1.0.4
Compiling openssl-probe v0.1.2
Compiling arc-swap v0.4.7
Compiling match_cfg v0.1.0
Compiling safemem v0.3.3
Compiling percent-encoding v1.0.1
Compiling lazycell v1.2.1
Compiling either v1.5.3
Compiling precomputed-hash v0.1.1
Compiling same-file v1.0.6
Compiling sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0) Compiling xdg v2.2.0
Compiling number_prefix v0.3.0
Compiling encode_unicode v0.3.6
Compiling httpdate v0.3.2
Compiling instant v0.1.5
Compiling if_chain v0.1.3
Compiling strsim v0.8.0
Compiling chardet v0.2.4
Compiling glob v0.3.0
Compiling dotenv v0.15.0
Compiling username v0.2.0
Compiling open v1.4.0
Compiling if_chain v1.0.0
Compiling thread_local v1.0.1
Compiling crossbeam-utils v0.7.2
Compiling num-traits v0.2.12
Compiling num-integer v0.1.43
Compiling memoffset v0.5.4
Compiling crossbeam-epoch v0.8.2
Compiling hashbrown v0.7.2
Compiling indexmap v1.4.0
Compiling rayon v1.3.1
Compiling gimli v0.21.0
Compiling miniz_oxide v0.3.7
Compiling rand_core v0.3.1
Compiling rand_jitter v0.1.4
Compiling rand_pcg v0.1.2
Compiling rand_chacha v0.1.1
Compiling rand v0.6.5
Compiling lock_api v0.3.4
Compiling unicode-normalization v0.1.13
Compiling unicode-bidi v0.3.4
Compiling semver v0.9.0
Compiling encoding-index-japanese v1.20141219.5
Compiling encoding-index-tradchinese v1.20141219.5
Compiling encoding-index-simpchinese v1.20141219.5
Compiling encoding-index-korean v1.20141219.5
Compiling encoding-index-singlebyte v1.20141219.5
Compiling pest v2.1.3
Compiling miniz_oxide v0.4.0
Compiling im v14.3.0
Compiling phf_shared v0.8.0
Compiling line-wrap v0.1.1
Compiling itertools v0.9.0
Compiling walkdir v2.3.1
Compiling app_dirs v1.2.1
Compiling rand_hc v0.1.0
Compiling rand_xorshift v0.1.1
Compiling rand_isaac v0.1.1
Compiling idna v0.2.0
Compiling idna v0.1.5
Compiling rustc_version v0.2.3
Compiling addr2line v0.12.2
Compiling encoding v0.2.33
Compiling pest_meta v2.1.3
Compiling time v0.1.43
Compiling parking_lot_core v0.7.2
Compiling memmap v0.7.0
Compiling socket2 v0.3.12
Compiling rand_os v0.1.3
Compiling term_size v0.3.2
Compiling num_cpus v1.13.0
Compiling termios v0.3.2
Compiling dirs v1.0.5
Compiling terminal_size v0.1.12
Compiling uname v0.1.1
Compiling atty v0.2.14
Compiling signal-hook-registry v1.2.0
Compiling dirs-sys v0.3.5
Compiling hostname v0.3.1
Compiling jobserver v0.1.21
Compiling quote v1.0.7
Compiling aho-corasick v0.7.13
Compiling csv-core v0.1.10
Compiling regex-automata v0.1.9
Compiling base64 v0.10.1
Compiling backtrace v0.3.49
Compiling url v1.7.2
Compiling scroll v0.9.2
Compiling sentry v0.18.1
Compiling quote v0.6.13
Compiling bitmaps v2.1.0
Compiling flate2 v1.0.16
Compiling crossbeam-queue v0.2.3
Compiling crossbeam-channel v0.4.2
Compiling rand_core v0.5.1
Compiling parking_lot v0.10.2
Compiling textwrap v0.11.0
Compiling term v0.5.2
Compiling signal-hook v0.1.16
Compiling dirs v2.0.2
Compiling cc v1.0.56
Compiling regex v1.3.9
Compiling sized-chunks v0.5.3
Compiling rand_chacha v0.2.2
Compiling rand_xoshiro v0.4.0
Compiling scheduled-thread-pool v0.2.4
Compiling clap v2.33.1
Compiling crossbeam-deque v0.7.3
Compiling openssl-src v111.10.0+1.1.1g
Compiling libz-sys v1.0.25
Compiling bzip2-sys v0.1.9+1.0.8
Compiling curl-sys v0.4.32+curl-7.70.0
Compiling brotli-sys v0.3.2
Compiling libgit2-sys v0.12.7+1.0.0
Compiling runas v0.2.1
Compiling backoff v0.1.6
Compiling console v0.11.3
Compiling java-properties v1.2.0
Compiling might-be-minified v0.3.0
Compiling rand v0.7.3
Compiling r2d2 v0.8.8
Compiling synstructure v0.12.4
Compiling pest_generator v2.1.3
Compiling scroll_derive v0.9.5
Compiling openssl-sys v0.9.58
Compiling indicatif v0.14.0
Compiling scroll_derive v0.10.2
Compiling dlv-list v0.2.2
Compiling pest_derive v2.1.0
Compiling scroll v0.10.1
Compiling ordered-multimap v0.2.4
Compiling failure v0.1.8
Compiling goblin v0.2.3
Compiling sha1 v0.6.0
Compiling bstr v0.2.13
Compiling chrono v0.4.11
Compiling url v2.1.1
Compiling string_cache v0.8.0
Compiling rust-ini v0.15.3
Compiling which v3.1.1
Compiling uuid v0.8.1
Compiling csv v1.1.3
Compiling globset v0.4.5
Compiling sourcemap v5.0.0
Compiling plist v0.5.5
Compiling anylog v0.5.0
Compiling elementtree v0.5.0
Compiling brotli2 v0.3.2
Compiling debugid v0.7.2
Compiling pdb v0.6.0
Compiling proguard v4.0.1
Compiling prettytable-rs v0.8.0
Compiling ignore v0.4.16
Compiling symbolic-common v7.5.0
Compiling sentry-types v0.14.1
Compiling bzip2 v0.3.3
Compiling zip v0.5.6
Compiling symbolic-debuginfo v7.5.0
Compiling symbolic v7.5.0
Compiling git2 v0.13.6
Finished release [optimized] target(s) in 6m 10s

Quoted 1 line
Quoted 1 line
Quoted 1 line

host-sentry-cli 1.59.0 Installing to host directory

cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo install --offline --root /home/thomas/projets/buildroot/outpu
t/host --bins --path ./ --force --locked
Installing sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
Compiling syn v1.0.33
Compiling synstructure v0.12.4
Compiling pest_generator v2.1.3
Compiling serde_derive v1.0.114
Compiling failure_derive v0.1.8
Compiling scroll_derive v0.10.2
Compiling pest_derive v2.1.0
Compiling failure v0.1.8
Compiling which v3.1.1
Compiling scroll v0.10.1
Compiling runas v0.2.1
Compiling goblin v0.2.3
Compiling serde v1.0.114
Compiling sha1 v0.6.0
Compiling serde_json v1.0.56
Compiling bstr v0.2.13
Compiling chrono v0.4.11
Compiling url v2.1.1
Compiling string_cache v0.8.0
Compiling uuid v0.8.1
Compiling csv v1.1.3
Compiling globset v0.4.5
Compiling sourcemap v5.0.0
Compiling plist v0.5.5
Compiling anylog v0.5.0
Compiling git2 v0.13.6
Compiling elementtree v0.5.0
Compiling debugid v0.7.2
Compiling pdb v0.6.0
Compiling proguard v4.0.1
Compiling prettytable-rs v0.8.0
Compiling ignore v0.4.16
Compiling symbolic-common v7.5.0
Compiling sentry-types v0.14.1
Compiling symbolic-debuginfo v7.5.0

    Building [====================================================>  ] 299/305: sentry-types, symbolic-debuginfo, ignore                                                                                                                    

Compiling symbolic v7.5.0
Compiling sentry v0.18.1
Compiling sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
Finished release [optimized] target(s) in 2m 35s
Installing /home/thomas/projets/buildroot/output/host/bin/sentry-cli
Installed package sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0) (executable sentry-cli)

Any idea ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#29 lore raw

PATCH 1/1 package/pkg-cargo.mk: make sure .cargo/config is used

Quoted 1 line · Thomas Petazzoni

Hello Patrick,

Hi,

Quoted 13 lines
Quoted 9 lines · On Thu, 10 Dec 2020 16:48:50 +0100 Patrick Havelange…

The way cargo searches for config file is based (among other rules) on the current directory. This means that if cargo is started outside of a package directory, its configuration file will not be taken into account.
So just 'cd' into it before running cargo build/install

This fix the issue where the build is failing in offline mode and only the dl/package.tar.gz is available.

Quoted 8 lines
Quoted 4 lines

Without this, it would have worked only if the CARGO_HOME cache had been populated (by running cargo vendor for example).

Thanks a lot! This does fix the "make ripgrep; make clean ripgrep" issue that Ryan reported. Also, for ripgrep, I no longer see anything being rebuilt during the install step, which is good.

glad it fixes those issues indeed.

Quoted 15 lines

*However*, for host-sentry-cli, even with this patch, quite a bit of stuff gets rebuilt during the install step. Here is how it goes:

Quoted 1 line
Quoted 1 line

host-sentry-cli 1.59.0 Extracting

gzip -d -c /home/thomas/dl/sentry-cli/sentry-cli-1.59.0.tar.gz | tar --strip-components=1 -C /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 -xf -

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

host-sentry-cli 1.59.0 Patching
host-sentry-cli 1.59.0 Configuring
host-sentry-cli 1.59.0 Building

cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo build --offline --release --manifest-path Cargo.toml --locked

    Compiling libc v0.2.71
    Compiling cfg-if v0.1.10
    Compiling autocfg v1.0.0
    Compiling proc-macro2 v1.0.18

[snip]

Quoted 43 lines

Compiling git2 v0.13.6
Finished release [optimized] target(s) in 6m 10s

Quoted 1 line
Quoted 1 line
Quoted 1 line

host-sentry-cli 1.59.0 Installing to host directory

cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo install --offline --root /home/thomas/projets/buildroot/outpu
t/host --bins --path ./ --force --locked
Installing sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
Compiling syn v1.0.33
Compiling synstructure v0.12.4
Compiling pest_generator v2.1.3
Compiling serde_derive v1.0.114
Compiling failure_derive v0.1.8
Compiling scroll_derive v0.10.2
Compiling pest_derive v2.1.0
Compiling failure v0.1.8
Compiling which v3.1.1
Compiling scroll v0.10.1
Compiling runas v0.2.1
Compiling goblin v0.2.3
Compiling serde v1.0.114
Compiling sha1 v0.6.0
Compiling serde_json v1.0.56
Compiling bstr v0.2.13
Compiling chrono v0.4.11
Compiling url v2.1.1
Compiling string_cache v0.8.0
Compiling uuid v0.8.1
Compiling csv v1.1.3
Compiling globset v0.4.5
Compiling sourcemap v5.0.0
Compiling plist v0.5.5
Compiling anylog v0.5.0
Compiling git2 v0.13.6
Compiling elementtree v0.5.0
Compiling debugid v0.7.2
Compiling pdb v0.6.0
Compiling proguard v4.0.1
Compiling prettytable-rs v0.8.0
Compiling ignore v0.4.16
Compiling symbolic-common v7.5.0
Compiling sentry-types v0.14.1
Compiling symbolic-debuginfo v7.5.0
Building [====================================================> ] 299/305: sentry-types, symbolic-debuginfo, ignore

Quoted 8 lines

Compiling symbolic v7.5.0
Compiling sentry v0.18.1
Compiling sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
Finished release [optimized] target(s) in 2m 35s
Installing /home/thomas/projets/buildroot/output/host/bin/sentry-cli
Installed package sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0) (executable sentry-cli)

Any idea ?

Sadly no. I've spent some time trying various combinations of flags for cargo build & cargo install, without luck.

I've found this bug-report though :
https://github.com/rust-lang/cargo/issues/8703 .

Since the first package being rebuild in this case is 'syn', I've looked at it and indeed it does make (non trivial) use of the [features] section inside its Cargo.toml. So maybe this can explain why syn is being rebuild (and also all similar packages and packages depending on those)? I'm absolutely not sure of that explanation though.

Quoted 3 lines

Thomas

Patrick H.

#30 lore raw

PATCH 1/1 package/pkg-cargo.mk: make sure .cargo/config is used

Hey Patrick, all,

Quoted 9 lines · Patrick Havelange
Quoted 1 line · Thomas Petazzoni

Hello Patrick,

Hi,

Quoted 3 lines

On Thu, 10 Dec 2020 16:48:50 +0100
Patrick Havelange <[email protected]> wrote:

[snip]

Quoted 74 lines
Quoted 9 lines

*However*, for host-sentry-cli, even with this patch, quite a bit of stuff gets rebuilt during the install step. Here is how it goes:

Quoted 1 line
Quoted 1 line

host-sentry-cli 1.59.0 Extracting

gzip -d -c /home/thomas/dl/sentry-cli/sentry-cli-1.59.0.tar.gz | tar --strip-components=1 -C /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 -xf -

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

host-sentry-cli 1.59.0 Patching
host-sentry-cli 1.59.0 Configuring
host-sentry-cli 1.59.0 Building

cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo build --offline --release --manifest-path Cargo.tom

l --locked

Quoted 5 lines
    Compiling libc v0.2.71
    Compiling cfg-if v0.1.10
    Compiling autocfg v1.0.0
    Compiling proc-macro2 v1.0.18

[snip]

Quoted 4 lines

Compiling git2 v0.13.6
Finished release [optimized] target(s) in 6m 10s

Quoted 1 line
Quoted 1 line
Quoted 1 line

host-sentry-cli 1.59.0 Installing to host directory

cd /home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0 && PATH="/home/thomas/projets/buildroot/output/host/bin:/home/thomas/projets/buildroot/output/host/sbin:/home/thomas/sys/bin:/home/thomas/.gem/ruby/2.1.0/bin:/usr/share/Modules/bin:/usr/lib64/ccache:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/thomas/.rvm/bin:/home/thomas/.rvm/bin:/home/thomas/.local/bin:/home/thomas/projets/git-wrappers" PKG_CONFIG="/home/thomas/projets/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/home/thomas/projets/buildroot/output/host/lib/pkgconfig:/home/thomas/projets/buildroot/output/host/share/pkgconfig" RUSTFLAGS="-C link-args=-L/home/thomas/projets/buildroot/output/host/lib -C link-args=-Wl,-rpath,/home/thomas/projets/buildroot/output/host/lib" CARGO_HOME=/home/thomas/projets/buildroot/output/host/share/cargo cargo install --offline --root /home/thomas/projets/build

root/outpu

Quoted 39 lines
  t/host --bins --path ./ --force --locked
   Installing sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
    Compiling syn v1.0.33
    Compiling synstructure v0.12.4
    Compiling pest_generator v2.1.3
    Compiling serde_derive v1.0.114
    Compiling failure_derive v0.1.8
    Compiling scroll_derive v0.10.2
    Compiling pest_derive v2.1.0
    Compiling failure v0.1.8
    Compiling which v3.1.1
    Compiling scroll v0.10.1
    Compiling runas v0.2.1
    Compiling goblin v0.2.3
    Compiling serde v1.0.114
    Compiling sha1 v0.6.0
    Compiling serde_json v1.0.56
    Compiling bstr v0.2.13
    Compiling chrono v0.4.11
    Compiling url v2.1.1
    Compiling string_cache v0.8.0
    Compiling uuid v0.8.1
    Compiling csv v1.1.3
    Compiling globset v0.4.5
    Compiling sourcemap v5.0.0
    Compiling plist v0.5.5
    Compiling anylog v0.5.0
    Compiling git2 v0.13.6
    Compiling elementtree v0.5.0
    Compiling debugid v0.7.2
    Compiling pdb v0.6.0
    Compiling proguard v4.0.1
    Compiling prettytable-rs v0.8.0
    Compiling ignore v0.4.16
    Compiling symbolic-common v7.5.0
    Compiling sentry-types v0.14.1
    Compiling symbolic-debuginfo v7.5.0
     Building [====================================================>  ] 299/305: sentry-types, symbolic-debuginfo, ignore
Quoted 8 lines

Compiling symbolic v7.5.0
Compiling sentry v0.18.1
Compiling sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0)
Finished release [optimized] target(s) in 2m 35s
Installing /home/thomas/projets/buildroot/output/host/bin/sentry-cli
Installed package sentry-cli v1.59.0 (/home/thomas/projets/buildroot/output/build/host-sentry-cli-1.59.0) (executable sentry-cli)

Any idea ?

Sadly no. I've spent some time trying various combinations of flags for cargo build & cargo install, without luck.

Quoted 8 lines

I've found this bug-report though :
https://github.com/rust-lang/cargo/issues/8703 .

Since the first package being rebuild in this case is 'syn', I've looked at it and indeed it does make (non trivial) use of the [features] section inside its Cargo.toml. So maybe this can explain why syn is being rebuild (and also all similar packages and packages depending on those)? I'm absolutely not sure of that explanation though.

I meant to mention this on the mailing thread but it fell between the cracks - I tested this without buildroot and had the same outcome. Seems like you already got to the root of "why" this happens, but just to show it's not buildroot-centric:

sentry-cli(master)? cargo build --locked --offline --release
Compiling libc v0.2.71
Compiling cfg-if v0.1.10
Compiling proc-macro2 v1.0.18
Compiling autocfg v1.0.0
Compiling unicode-xid v0.2.1
Compiling syn v1.0.33
Compiling serde_derive v1.0.114
Compiling serde v1.0.114
Compiling lazy_static v1.4.0
Compiling pkg-config v0.3.17
Compiling getrandom v0.1.14
Compiling memchr v2.3.3
Compiling ppv-lite86 v0.2.8
[snip compile logs]
Compiling symbolic-debuginfo v7.5.0
(https://github.com/getsentry/symbolic#075ee512)
Compiling symbolic v7.5.0 (https://github.com/getsentry/symbolic#075ee512)
Finished release [optimized] target(s) in 52.72s

sentry-cli(master)? cargo install --path . --root ./exroot --locked --offline --bins --force
Installing sentry-cli v1.60.1
(/accts/smvoss/projects/rust-buildroot/sentry-cli)
warning: package smallvec v1.4.0 in Cargo.lock is yanked in registry crates.io, consider running without --locked
warning: package socket2 v0.3.12 in Cargo.lock is yanked in registry crates.io, consider running without --locked
Compiling syn v1.0.33
Compiling synstructure v0.12.4
Compiling pest_generator v2.1.3
Compiling serde_derive v1.0.114
[snip compile logs]
Compiling sentry-cli v1.60.1
(/accts/smvoss/projects/rust-buildroot/sentry-cli)
Finished release [optimized] target(s) in 44.14s
Installing ./exroot/bin/sentry-cli
Installed package `sentry-cli v1.60.1
(/accts/smvoss/projects/rust-buildroot/sentry-cli) (executable sentry-cli`)
warning: be sure to add ./exroot/bin to your PATH to be able to run the installed binaries

Signature · 13 lines
Sam Voss | Sr. Software Engineer | Commercial Avionics
COLLINS AEROSPACE
400 Collins Road NE, Cedar Rapids, Iowa 52498, USA
Tel: +1 319 263 4039
sam.voss at collins.com | collinsaerospace.com

CONFIDENTIALITY WARNING: This message may contain proprietary and/or
privileged information of Collins Aerospace and its affiliated
companies. If you are not the intended recipient, please 1) Do not
disclose, copy, distribute or use this message or its contents. 2)
Advise the sender by return email. 3) Delete all copies (including all
attachments) from your computer. Your cooperation is greatly
appreciated.
#31 lore raw

PATCH 7/12 package/pkg-cargo.mk: introduce the cargo package infrastructure

In reply to Thomas Petazzoni

Hi Thomas,

Quoted 130 lines · Thomas Petazzoni

From: Patrick Havelange <[email protected]>

In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

[Thomas: add support for host-cargo-package]
---
  package/Makefile.in  |   1 +
  package/pkg-cargo.mk | 123 +++++++++++++++++++++++++++++++++++++++++++
  2 files changed, 124 insertions(+)

create mode 100644 package/pkg-cargo.mk

package/Makefile.in +1 -0
index 51f5cbce4f..2af123f36d 100644
@@ -427,3 +427,4 @@ include package/pkg-waf.mk
  include package/pkg-golang.mk
  include package/pkg-meson.mk
  include package/pkg-qmake.mk
+include package/pkg-cargo.mk
package/pkg-cargo.mk new file +100 -0
new file mode 100644
index 0000000000..52c237aa3c
@@ -0,0 +1,123 @@
+################################################################################
+# Cargo package infrastructure
+#
+# This file implements an infrastructure that eases development of package
+# .mk files for Cargo packages. It should be used for all packages that use
+# Cargo as their build system.
+#
+# See the Buildroot documentation for details on the usage of this
+# infrastructure
+#
+# In terms of implementation, this Cargo infrastructure requires the .mk file
+# to only specify metadata information about the package: name, version,
+# download URL, etc.
+#
+# We still allow the package .mk file to override what the different steps
+# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined,
+# it is used as the list of commands to perform to build the package,
+# instead of the default Cargo behaviour. The package can also define some
+# post operation hooks.
+#
+################################################################################
+
+################################################################################
+# inner-cargo-package -- defines how the configuration, compilation and
+# installation of a cargo package should be done, implements a few hooks
+# to tune the build process for cargo specifities and calls the generic
+# package infrastructure to generate the necessary make targets
+#
+#  argument 1 is the lowercase package name
+#  argument 2 is the uppercase package name, including a HOST_ prefix
+#             for host packages
+#  argument 3 is the uppercase package name, without the HOST_ prefix
+#             for host packages
+#  argument 4 is the type (target or host)
+################################################################################
+
+define inner-cargo-package
+
+# We need host-rustc to run cargo
+$(2)_DEPENDENCIES += host-rustc
+
+$(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+#
+# Build step. Only define it if not already defined by the package .mk
+# file.
+#
+ifndef $(2)_BUILD_CMDS
+ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	$$(TARGET_MAKE_ENV) \
+		$$(TARGET_CONFIGURE_OPTS) \
+		$$($(2)_CARGO_ENV) \
+		cargo build \
+			--offline \
+			--target $$(RUSTC_TARGET_NAME) \
+			$$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
+			--manifest-path $$(@D)/Cargo.toml \
+			--locked \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+else # ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \
+		cargo build \
+			--offline \
+			--release \
+			--manifest-path $$(@D)/Cargo.toml \
+			--locked \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+endif # ifeq ($(4),target)
+endif # ifndef $(2)_BUILD_CMDS
+
+#
+# Target installation step. Only define it if not already defined by
+# the package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
+		cargo install \
+			--target $$(RUSTC_TARGET_NAME) \
+			--offline \
+			--root $$(TARGET_DIR)/usr/ \
+			--bins \
+			--path $$(@D) \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \

package/pkg-cargo.mk line 110

Quoted 10 lines
package/pkg-cargo.mk new file +10 -0
+		cargo install \
+			--offline \
+			--root $$(HOST_DIR) \
+			--bins \
+			--path $$(@D) \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif

Nitpick: This is the only minor thing that bothers me a bit: the duplication for the host/target cases of the cargo command flags. Would it not be better if they were merged ?

BR,

Patrick H.

#32 lore raw

PATCH 7/12 package/pkg-cargo.mk: introduce the cargo package infrastructure

Hello,

Quoted 30 lines · On Wed, 16 Dec 2020 14:58:39 +0100 Patrick Havelange…
Quoted 30 lines
package/pkg-cargo.mk new file +30 -0
+#
+# Target installation step. Only define it if not already defined by
+# the package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
+		cargo install \
+			--target $$(RUSTC_TARGET_NAME) \
+			--offline \
+			--root $$(TARGET_DIR)/usr/ \
+			--bins \
+			--path $$(@D) \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \
+		cargo install \
+			--offline \
+			--root $$(HOST_DIR) \
+			--bins \
+			--path $$(@D) \
+			--force \
Quoted 8 lines

package/pkg-cargo.mk line 94

Quoted 4 lines
package/pkg-cargo.mk new file +4 -0
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif  

Nitpick: This is the only minor thing that bothers me a bit: the duplication for the host/target cases of the cargo command flags. Would it not be better if they were merged ?

I agree. In fact while re-using your code and extending it, I did quite a bit of back and forth in using variables to share options, not using variables. I was not sure myself. But OK, if you also feel like we should share those common options, I guess it's a sign that we should go with that!

Quoted 1 line

BR,

Is this "Best Regards", "BuildRoot" or both ? :-)

Cheers!

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#33 lore raw

PATCH 8/12 docs/manual/cargo: document the cargo-package infrastructure

In reply to Thomas Petazzoni

From: Patrick Havelange <[email protected]>

The Buildroot manual was already providing some details on how to integrate Cargo packages, and those details now need to be updated with a proper documentation for the cargo-package infrastructure.

[Thomas: numerous updates and extensions.]
---
 docs/manual/adding-packages-cargo.txt | 88 ++++++++++++---------------
 1 file changed, 39 insertions(+), 49 deletions(-)
docs/manual/adding-packages-cargo.txt +39 -49
index 8fcc80bcc6..c65a32f017 100644
@@ -1,7 +1,7 @@
 // -*- mode:doc; -*-
 // vim: set syntax=asciidoc:
 
-=== Integration of Cargo-based packages
+=== Infrastructure for Cargo-based packages
 
 Cargo is the package manager for the Rust programming language. It allows the
 user to build programs or libraries written in Rust, but it also downloads and
@@ -10,7 +10,7 @@ called "crates".
 
 [[cargo-package-tutorial]]
 
-==== Cargo-based package's +Config.in+ file
+==== +cargo-package+ tutorial
 
 The +Config.in+ file of Cargo-based package 'foo' should contain:
 
@@ -25,11 +25,7 @@ The +Config.in+ file of Cargo-based package 'foo' should contain:
 08: 	  http://foosoftware.org/foo/
 ---------------------------
 
-==== Cargo-based package's +.mk+ file
-
-Buildroot does not (yet) provide a dedicated package infrastructure for
-Cargo-based packages. So, we will explain how to write a +.mk+ file for such a
-package. Let's start with an example:
+And the +.mk+ file for this package should contain:
 
 ------------------------------
 01: ################################################################################
@@ -44,52 +40,48 @@ package. Let's start with an example:
 10: FOO_LICENSE = GPL-3.0+
 11: FOO_LICENSE_FILES = COPYING
 12:
-13: FOO_DEPENDENCIES = host-rustc
-14:
-15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-16:
-17: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE)
-18:
-19: FOO_CARGO_OPTS = \
-20: 	$(if $(BR2_ENABLE_DEBUG),,--release) \
-21: 	--target=$(RUSTC_TARGET_NAME) \
-22: 	--manifest-path=$(@D)/Cargo.toml
-23:
-24: define FOO_BUILD_CMDS
-25: 	$(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \
-26: 		cargo build $(FOO_CARGO_OPTS)
-27: endef
-28:
-29: define FOO_INSTALL_TARGET_CMDS
-30: 	$(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \
-31: 		$(TARGET_DIR)/usr/bin/foo
-32: endef
-33:
-34: $(eval $(generic-package))
+13: $(eval $(cargo-package))
 --------------------------------
 
-The Makefile starts with the definition of the standard variables for package
-declaration (lines 7 to 11).
+The Makefile starts with the definition of the standard variables for
+package declaration (lines 7 to 11).
+
+As seen in line 13, it is based on the +cargo-package+
+infrastructure. Cargo will be invoked automatically by this
+infrastructure to build and install the package.
+
+It is still possible to define custom build commands or install
+commands (i.e.  with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS).
+Those will then replace the commands from the cargo infrastructure.
+
+==== +cargo-package+ reference
+
+The main macros for the Cargo package infrastructure are
++cargo-package+ for target packages and +host-cargo-package+ for host
+packages.
+
+Just like the generic infrastructure, the Cargo infrastructure works
+by defining a number of variables before calling the +cargo-package+
+or +host-cargo-package+ macros.
 
-As seen in line 34, it is based on the
-xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines
-the variables required by this particular infrastructure, where Cargo is
-invoked:
+First, all the package metadata information variables that exist in
+the generic infrastructure also exist in the Cargo infrastructure:
++FOO_VERSION+, +FOO_SOURCE+, +FOO_PATCH+, +FOO_SITE+,
++FOO_DEPENDENCIES+, +FOO_LICENSE+, +FOO_LICENSE_FILES+, etc.
 
-* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required
-  to configure the cross-compilation of the package are passed via
-  +FOO_CONF_OPTS+.
+A few additional variables, specific to the Cargo infrastructure, can
+also be defined. Many of them are only useful in very specific cases,
+typical packages will therefore only use a few of them.
 
-* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on
-  the target.
+* +FOO_CARGO_ENV+ can be used to pass additional variables in the
+  environment of +cargo+ invocations. It used at both build and
+  installation time
 
-In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to
-contain +host-cargo+.
+* +FOO_CARGO_BUILD_OPTS+ can be used to pass additional options to
+  +cargo+ at build time.
 
-To sum it up, to add a new Cargo-based package, the Makefile example can be
-copied verbatim then edited to replace all occurences of +FOO+ with the
-uppercase name of the new package and update the values of the standard
-variables.
+* +FOO_CARGO_INSTALL_OPTS+ can be used to pass additional options to
+  +cargo+ at install time.
 
 ==== About Dependencies Management
 
@@ -99,9 +91,7 @@ automatically them. This step can also be performed independently, via the
 +cargo fetch+ command.
 
 Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+. As seen in the package
-Makefile example at line 15, this environment variable is set to
-+$(HOST_DIR)/share/cargo+.
+crates, whose location is given by +$CARGO_HOME+.
 
 This dependency download mechanism is not convenient when performing an offline
 build, as Cargo will fail to fetch the dependencies. In that case, it is advised
Signature · 1 line
2.28.0
#34 lore raw

PATCH 9/12 package/ripgrep: convert to cargo infrastructure

In reply to Thomas Petazzoni

From: Patrick Havelange <[email protected]>

---
 package/ripgrep/ripgrep.mk | 28 +---------------------------
 1 file changed, 1 insertion(+), 27 deletions(-)
package/ripgrep/ripgrep.mk +1 -27
index 9dd8d58de1..97e9e2ce5f 100644
@@ -9,30 +9,4 @@ RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
 RIPGREP_LICENSE = MIT
 RIPGREP_LICENSE_FILES = LICENSE-MIT
 
-RIPGREP_DEPENDENCIES = host-rustc
-RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-
-RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_BIN_SUBDIR)
-
-RIPGREP_CARGO_OPTS = \
-	--target=$(RUSTC_TARGET_NAME) \
-	--manifest-path=$(@D)/Cargo.toml
-
-ifeq ($(BR2_ENABLE_DEBUG),y)
-RIPGREP_CARGO_BIN_SUBDIR = debug
-else
-RIPGREP_CARGO_OPTS += --release
-RIPGREP_CARGO_BIN_SUBDIR = release
-endif
-
-define RIPGREP_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \
-		cargo build $(RIPGREP_CARGO_OPTS)
-endef
-
-define RIPGREP_INSTALL_TARGET_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \
-		$(TARGET_DIR)/usr/bin/rg
-endef
-
-$(eval $(generic-package))
+$(eval $(cargo-package))
Signature · 1 line
2.28.0
#35 lore raw

PATCH 10/12 package/sentry-cli: convert to host-cargo-package infrastructure

In reply to Thomas Petazzoni
---
 package/sentry-cli/sentry-cli.mk | 22 ++--------------------
 1 file changed, 2 insertions(+), 20 deletions(-)
package/sentry-cli/sentry-cli.mk +2 -20
index 7e53f16026..40ca22fdf9 100644
@@ -9,24 +9,6 @@ SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
 SENTRY_CLI_LICENSE = BSD-3-clause
 SENTRY_CLI_LICENSE_FILES = LICENSE
 
-HOST_SENTRY_CLI_DEPENDENCIES = host-rustc host-zlib
+HOST_SENTRY_CLI_DEPENDENCIES = host-zlib
 
-HOST_SENTRY_CLI_CARGO_ENV = \
-	CARGO_HOME=$(HOST_DIR)/share/cargo \
-	RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))"
-
-HOST_SENTRY_CLI_CARGO_OPTS = \
-	--release \
-	--manifest-path=$(@D)/Cargo.toml
-
-define HOST_SENTRY_CLI_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(HOST_SENTRY_CLI_CARGO_ENV) \
-		cargo build $(HOST_SENTRY_CLI_CARGO_OPTS)
-endef
-
-define HOST_SENTRY_CLI_INSTALL_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/target/release/sentry-cli \
-		$(HOST_DIR)/bin/sentry-cli
-endef
-
-$(eval $(host-generic-package))
+$(eval $(host-cargo-package))
Signature · 1 line
2.28.0
#36 lore raw

PATCH 11/12 support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages

In reply to Thomas Petazzoni

This commit adds support/download/cargo-post-process to perform the vendoring on Cargo packages, and enables it in package/pkg-cargo.mk.

Since it changes the contents of the tarballs for ripgrep and sentry-cli, it changes their hashes. To not have a different hash for the same version of ripgrep and sentry-cli, we bump their versions. It has to be done in the same commit as the Cargo vendoring to make the series bisectable.

---
 package/pkg-cargo.mk                |  5 +++-
 package/ripgrep/ripgrep.hash        |  2 +-
 package/ripgrep/ripgrep.mk          |  2 +-
 package/sentry-cli/sentry-cli.hash  |  2 +-
 package/sentry-cli/sentry-cli.mk    |  2 +-
 support/download/cargo-post-process | 38 +++++++++++++++++++++++++++++
 6 files changed, 46 insertions(+), 5 deletions(-)
 create mode 100755 support/download/cargo-post-process
package/pkg-cargo.mk +4 -1
index 52c237aa3c..6653f64594 100644
@@ -37,10 +37,13 @@
 define inner-cargo-package
 
 # We need host-rustc to run cargo
-$(2)_DEPENDENCIES += host-rustc
+$(2)_DOWNLOAD_DEPENDENCIES += host-rustc
 
 $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
 
+$(2)_DOWNLOAD_POST_PROCESS = cargo
+$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo
+
 #
 # Build step. Only define it if not already defined by the package .mk
 # file.
package/ripgrep/ripgrep.hash +1 -1
index 0841c0185c..81ac905d3d 100644
@@ -1,3 +1,3 @@
 # Locally calculated
-sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz
+sha256 80d7f07325f4d485c5e4baf061b0376b45a497ee7a1c540d7894057bb9ac3a59 ripgrep-12.1.1.tar.gz
 sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT
package/ripgrep/ripgrep.mk +1 -1
index 97e9e2ce5f..d21f88a6e8 100644
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-RIPGREP_VERSION = 0.8.1
+RIPGREP_VERSION = 12.1.1
 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
 RIPGREP_LICENSE = MIT
 RIPGREP_LICENSE_FILES = LICENSE-MIT
package/sentry-cli/sentry-cli.hash +1 -1
index 3b0733a276..63b0c812bc 100644
@@ -1,3 +1,3 @@
 # locally calculated
-sha256  5d0f7acf6a139d1c1716b9a8ff76c8bfaab09104ba663c957bb9a5dba2ffbabd  sentry-cli-1.57.0.tar.gz
+sha256  a657dd1a46e3de044deff4c311b7276c5a9409582e707e6e6a78b0cf712591c4  sentry-cli-1.59.0.tar.gz
 sha256  9503def7b54ceb6e3cd182fd59bc05d3a30d7eae481e65aaba4b495133c83c14  LICENSE
package/sentry-cli/sentry-cli.mk +1 -1
index 40ca22fdf9..58f5f1e325 100644
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-SENTRY_CLI_VERSION = 1.57.0
+SENTRY_CLI_VERSION = 1.59.0
 SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
 SENTRY_CLI_LICENSE = BSD-3-clause
 SENTRY_CLI_LICENSE_FILES = LICENSE
support/download/cargo-post-process new file +38 -0
new file mode 100755
index 0000000000..5081476385
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+while getopts "n:o:" OPT; do
+        case "${OPT}" in
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then
+        exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Cargo vendoring
+pushd ${base_name} > /dev/null
+cargo vendor --locked VENDOR
+echo $?
+# Create the local .cargo/config with vendor info
+mkdir -p .cargo/
+cat <<EOF >.cargo/config
+[source.crates-io]
+replace-with = "vendored-sources"
+
+[source.vendored-sources]
+directory = "VENDOR"
+EOF
+popd > /dev/null
+
+repack ${base_name} ${output}
Signature · 1 line
2.28.0
#37 lore raw

PATCH 12/12 docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management

In reply to Thomas Petazzoni

Now that we have vendoring support for Cargo packages, let's rewrite the dependency management section in a more accurate way.

We drop the part about the local cache of the registry, because +CARGO_HOME+ in Buildroot points to $(HOST_DIR)/share/cargo, which is not shared between builds nor preserved accross builds, so its effect as a cache is limited.

---
 docs/manual/adding-packages-cargo.txt | 22 ++++++++++------------
 1 file changed, 10 insertions(+), 12 deletions(-)
docs/manual/adding-packages-cargo.txt +10 -12
index c65a32f017..fb3e7d0780 100644
@@ -85,15 +85,13 @@ typical packages will therefore only use a few of them.
 
 ==== About Dependencies Management
 
-A crate can depend on other libraries from crates.io or git repositories, listed
-in its Cargo.toml file. Before starting a build, Cargo usually downloads
-automatically them. This step can also be performed independently, via the
-+cargo fetch+ command.
-
-Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+.
-
-This dependency download mechanism is not convenient when performing an offline
-build, as Cargo will fail to fetch the dependencies. In that case, it is advised
-to generate a tarball of the dependencies using the +cargo vendor+ and add it to
-+FOO_EXTRA_DOWNLOADS+.
+A crate can depend on other libraries from crates.io or git
+repositories, listed in its Cargo.toml file. Buildroot automatically
+takes care of downloading such dependencies as part of the download
+step of packages that use the +cargo-package+ infrastructure. Such
+dependencies are then kept together with the package source code in
+the tarball cached in Buildroot's +DL_DIR+, and therefore the hash of
+the package's tarball includes such dependencies.
+
+This mechanism ensures that any change in the dependencies will be
+detected, and allows the build to be performed completely offline.
Signature · 1 line
2.28.0