Skip to contentSkip to conversation

Support for Cargo and Go vendoring

PATCH Applied Thomas Petazzoni 6 participants 38 messages Jan 6, 2022 – Jan 9, 2022

lore
Share & tools
lore
#0 lore raw

v3 cover Support for Cargo and Go vendoring

Hello,

Believe it or not, but here is a new version of my patch series adding support for Go vendoring, as well as a Cargo package infrastructure that also supports vendoring.

This series builds on top of the work from Patrick Havelange <[email protected]> who worked on the Cargo integration, but extends it to also cover the case of Go, showing that we have a solution solving both of these language-specific package managers. For the Go vendoring, I've also used the indication provided by Christian Stewart <[email protected]>. I've also integrated some changes that Christian did in his series at:

https://patchwork.ozlabs.org/project/buildroot/list/?series=266310

Overall, the solution consists in a concept of "download post-process helper" is introduced in the download infrastructure. These are shell scripts that can be invoked right after the download is done, but before the tarball is hash-checked and stored in DL_DIR. The idea is that such "download post-process helpers" can run the language-specific vendoring logic to fetch the package dependencies. Thanks to this, the tarball in DL_DIR for a given package not only contains the package source code itself, but also the source code of its Cargo or Go dependencies. The tarball hash covers the entire source code, the complete tarball is cached in DL_DIR, in the primary site and backup site, and the build can fully be done offline.

Such "download post-process helpers" are registered by means of the <pkg>_DOWNLOAD_POST_PROCESS variable by the appropriate package infrastructure.

This new series is available at:

https://github.com/tpetazzoni/buildroot/tree/pkg-vendoring

A useful defconfig to test this series is:

BR2_aarch64=y
BR2_TOOLCHAIN_EXTERNAL=y BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_AARCH64_GLIBC_STABLE=y BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set BR2_PACKAGE_DELVE=y
BR2_PACKAGE_RIPGREP=y
BR2_PACKAGE_GOCRYPTFS=y BR2_PACKAGE_TINIFIER=y
BR2_PACKAGE_FLANNEL=y BR2_PACKAGE_BALENA_ENGINE=y BR2_PACKAGE_DOCKER_CLI=y BR2_PACKAGE_DOCKER_ENGINE=y BR2_PACKAGE_EMBIGGEN_DISK=y BR2_PACKAGE_MENDER=y
# BR2_TARGET_ROOTFS_TAR is not set BR2_PACKAGE_HOST_MENDER_ARTIFACT=y BR2_PACKAGE_HOST_SENTRY_CLI=y

It is useful because it builds:

- All currently existing packages that use the golang-package or host-golang-package infrastructures. Those existing packages did not need vendoring support.

- Builds the new tinifier, embiggen-disk and gocryptfs Go packages, which need vendoring.

- Builds ripgrep, which is a target Cargo package.

- Builds host-entry-cli, which is a host Cargo package.

Here are the main changes compared to v2 (posted December 2020):

- Use of the mk_tar_gz shell function to re-create the tarball after vendoring. This allows to benefit from the reproducible magic implemented by mk_tar_gz.

- post-process download helpers moved to support/download/helpers instead of having their own files.

- Usage of a proper timestamp for the re-created tarballs instead of a fixed hardcoded timestamp.

- Keep the creation of go.mod in golang-package, as it is needed for Go packages that are already vendored in their upstream tarball. The creation of go.mod in the Go post-process download script, used for vendoring, now uses the same method to create a minimal go.mod file.

- The Cargo infra is now added in one patch, with vendoring support, instead of adding it without vendoring support and add it after the fact.

- For the same reason, the Cargo infrastructure documentation now comes through a single patch.

- The ripgrep changes are modified to account for the ripgrep version bump.

- The tinifier package (a Go package that needs vendoring) was version bumped.

- The packages embiggen-disk and gocryptfs, which were submitted by Christian, are now part of this series.

- host-sentry-cli, which had been removed previously due to the lack of Cargo vendoring support, is now re-added by this series.

- Small formatting and coding style improvements.

Best regards,

Thomas Petazzoni

Christian Stewart via buildroot (2):
package/embiggen-disk: new package
package/gocryptfs: new package

Patrick Havelange (3):
package/pkg-cargo.mk: introduce the cargo package infrastructure docs/manual/cargo: document the cargo-package infrastructure package/ripgrep: convert to cargo infrastructure

Thomas Petazzoni (6):
support/download/dl-wrapper: add concept of download post-processing package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable support/download/post-process-helpers: add helper function for post
process scripts
package/pkg-golang.mk: implement Go vendoring support package/sentry-cli: re-add package
package/tinifier: new package

 DEVELOPERS                                    |   4 +
 docs/manual/adding-packages-cargo.txt         | 106 ++++++------
 package/Config.in                             |   3 +
 package/Config.in.host                        |   1 +
 package/Makefile.in                           |   1 +
 ...-Fix-resizing-of-dev-mmcblk0pN-paths.patch |  45 ++++++
 package/embiggen-disk/Config.in               |  19 +++
 package/embiggen-disk/embiggen-disk.hash      |   3 +
 package/embiggen-disk/embiggen-disk.mk        |  12 ++
 package/gocryptfs/Config.in                   |  12 ++
 package/gocryptfs/gocryptfs.hash              |   3 +
 package/gocryptfs/gocryptfs.mk                |  19 +++
 package/pkg-cargo.mk                          | 153 ++++++++++++++++++
 package/pkg-download.mk                       |   1 +
 package/pkg-golang.mk                         |  13 +-
 package/ripgrep/ripgrep.hash                  |   2 +-
 package/ripgrep/ripgrep.mk                    |  37 +----
 package/sentry-cli/Config.in.host             |  10 ++
 package/sentry-cli/sentry-cli.hash            |   3 +
 package/sentry-cli/sentry-cli.mk              |  14 ++
 package/tinifier/Config.in                    |  10 ++
 package/tinifier/tinifier.hash                |   3 +
 package/tinifier/tinifier.mk                  |  13 ++
 support/download/cargo-post-process           |  38 +++++
 support/download/dl-wrapper                   |   9 +-
 support/download/go-post-process              |  35 ++++
 support/download/helpers                      |  22 +++
 27 files changed, 498 insertions(+), 93 deletions(-)
 create mode 100644 package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch
 create mode 100644 package/embiggen-disk/Config.in
 create mode 100644 package/embiggen-disk/embiggen-disk.hash
 create mode 100644 package/embiggen-disk/embiggen-disk.mk
 create mode 100644 package/gocryptfs/Config.in
 create mode 100644 package/gocryptfs/gocryptfs.hash
 create mode 100644 package/gocryptfs/gocryptfs.mk
 create mode 100644 package/pkg-cargo.mk
 create mode 100644 package/sentry-cli/Config.in.host
 create mode 100644 package/sentry-cli/sentry-cli.hash
 create mode 100644 package/sentry-cli/sentry-cli.mk
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
 create mode 100755 support/download/cargo-post-process
 create mode 100755 support/download/go-post-process
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

PATCH v3 1/11 support/download/dl-wrapper: add concept of download post-processing

In order to support package managers such as Cargo (Rust) or Go, we want to run some custom logic after the main download, but before packing the tarball and checking the hash.

To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then support/download/<something>-post-process will be called.

---
 support/download/dl-wrapper | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
support/download/dl-wrapper +8 -1
index 3315bd410e..e6509e5671 100755
@@ -25,7 +25,7 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() {
         r)  recurse="-r";;
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
+        p)  post_process="${OPTARG}";;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -135,6 +136,12 @@ main() {
             continue
         fi
 
+        if [ -n "${post_process}" ] ; then
+            ${OLDPWD}/support/download/${post_process}-post-process \
+                -o "${tmpf}" \
+                -n "${raw_base_name}"
+        fi
+
         # cd back to free the temp-dir, so we can remove it later
         cd "${OLDPWD}"
 
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

Re: [Buildroot] [PATCH v3 01/11] support/download/dl-wrapper: add concept of download post-processing

Thomas, All,

On 2022-01-06 21:59 +0100, Thomas Petazzoni spake thusly:

Quoted 35 lines

In order to support package managers such as Cargo (Rust) or Go, we want to run some custom logic after the main download, but before packing the tarball and checking the hash.

To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then support/download/<something>-post-process will be called.

---
 support/download/dl-wrapper | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
support/download/dl-wrapper +2 -1
index 3315bd410e..e6509e5671 100755
@@ -25,7 +25,7 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() {
         r)  recurse="-r";;
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
+        p)  post_process="${OPTARG}";;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;

support/download/dl-wrapper main() { · line 140

Quoted 5 lines
support/download/dl-wrapper +2 -0
@@ -135,6 +136,12 @@ main() {
             continue
         fi
 
+        if [ -n "${post_process}" ] ; then
+            ${OLDPWD}/support/download/${post_process}-post-process \

I added double-quotes when expanding those variables.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 10 lines
support/download/dl-wrapper +4 -0
+                -o "${tmpf}" \
+                -n "${raw_base_name}"
+        fi
+
         # cd back to free the temp-dir, so we can remove it later
         cd "${OLDPWD}"
 
Signature · 1 line
2.33.1
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

PATCH v3 2/11 package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable

In reply to Thomas Petazzoni

This will allow packages to register than a download post-processing is needed. Note that this variable is intentionally not documented: it is an internal variable meant to be set by package infrastructures, not directly by packages.

---
 package/pkg-download.mk | 1 +
 1 file changed, 1 insertion(+)
package/pkg-download.mk +1 -0
index 2527ba5c60..66a415cce0 100644
@@ -117,6 +117,7 @@ define DOWNLOAD
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw

Re: [Buildroot] [PATCH v3 02/11] package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable

Thomas, All,

On 2022-01-06 21:59 +0100, Thomas Petazzoni spake thusly:

Quoted 6 lines

This will allow packages to register than a download post-processing is needed. Note that this variable is intentionally not documented: it is an internal variable meant to be set by package infrastructures, not directly by packages.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 19 lines
---
 package/pkg-download.mk | 1 +
 1 file changed, 1 insertion(+)
package/pkg-download.mk +1 -0
index 2527ba5c60..66a415cce0 100644
@@ -117,6 +117,7 @@ define DOWNLOAD
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
Signature · 1 line
2.33.1
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw

PATCH v3 3/11 support/download/post-process-helpers: add helper function for post process scripts

In reply to Thomas Petazzoni

download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an unpack() function and a repack() function.

---
 support/download/helpers | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
support/download/helpers +22 -0
index 40d5eea591..4c5eb466cc 100755
@@ -72,5 +72,27 @@ mk_tar_gz() {
     popd >/dev/null
 }
 
+post_process_unpack() {
+    local dest="${1}"
+    local tarball="${2}"
+    local one_file
+
+    mkdir "${dest}"
+    tar -C "${dest}" --strip-components=1 -xf "${tarball}"
+    one_file="$(find "${dest}" -type f -print0 |LC_ALL=C sort -z |head -z -n1 |tr -d "\0")"
+    touch -r "${one_file}" "${dest}.timestamp"
+}
+
+post_process_repack() {
+    local in_dir="${1}"
+    local base_dir="${2}"
+    local out="${3}"
+    local date
+
+    date="@$(stat -c '%Y' "${in_dir}/${base_dir}.timestamp")"
+
+    mk_tar_gz "${in_dir}/${base_dir}" "${base_dir}" "${date}" "${out}"
+}
+
 # Keep this line and the following as last lines in this file.
 # vim: ft=bash
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#6 lore raw

Re: [Buildroot] [PATCH v3 03/11] support/download/post-process-helpers: add helper function for post process scripts

Thomas, All,

On 2022-01-06 21:59 +0100, Thomas Petazzoni spake thusly:

Quoted 6 lines

download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an unpack() function and a repack() function.

Applied to master, thanks.

Regards,
Yann E. MORIN.

Quoted 39 lines
---
 support/download/helpers | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
support/download/helpers +22 -0
index 40d5eea591..4c5eb466cc 100755
@@ -72,5 +72,27 @@ mk_tar_gz() {
     popd >/dev/null
 }
 
+post_process_unpack() {
+    local dest="${1}"
+    local tarball="${2}"
+    local one_file
+
+    mkdir "${dest}"
+    tar -C "${dest}" --strip-components=1 -xf "${tarball}"
+    one_file="$(find "${dest}" -type f -print0 |LC_ALL=C sort -z |head -z -n1 |tr -d "\0")"
+    touch -r "${one_file}" "${dest}.timestamp"
+}
+
+post_process_repack() {
+    local in_dir="${1}"
+    local base_dir="${2}"
+    local out="${3}"
+    local date
+
+    date="@$(stat -c '%Y' "${in_dir}/${base_dir}.timestamp")"
+
+    mk_tar_gz "${in_dir}/${base_dir}" "${base_dir}" "${date}" "${out}"
+}
+
 # Keep this line and the following as last lines in this file.
 # vim: ft=bash
Signature · 1 line
2.33.1
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#7 lore raw

PATCH v3 4/11 package/pkg-golang.mk: implement Go vendoring support

In reply to Thomas Petazzoni

This commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure.

The -modcacherw flag is added to ensure that the Go cache is read/write, and can be deleted properly upon "make clean".

The <pkg>_LICENSE variable of golang packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

---
 package/pkg-golang.mk            | 13 +++++++++++-
 support/download/go-post-process | 35 ++++++++++++++++++++++++++++++++
 2 files changed, 47 insertions(+), 1 deletion(-)
 create mode 100755 support/download/go-post-process
package/pkg-golang.mk +12 -1
index d07242310d..35bcb1673b 100644
@@ -42,12 +42,13 @@ define inner-golang-package
 
 $(2)_BUILD_OPTS += \
 	-ldflags "$$($(2)_LDFLAGS)" \
+	-modcacherw \
 	-tags "$$($(2)_TAGS)" \
 	-trimpath \
 	-p $(PARALLEL_JOBS)
 
 # Target packages need the Go compiler on the host.
-$(2)_DEPENDENCIES += host-go
+$(2)_DOWNLOAD_DEPENDENCIES += host-go
 
 $(2)_BUILD_TARGETS ?= .
 
@@ -81,6 +82,16 @@ define $(2)_GEN_GOMOD
 endef
 $(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD
 
+$(2)_DOWNLOAD_POST_PROCESS = go
+$(2)_DL_ENV = \
+	$(HOST_GO_COMMON_ENV) \
+	GOPROXY=direct \
+	BR_GOMOD=$$($(2)_GOMOD)
+
+# Due to vendoring, it is pretty likely that not all licenses are
+# listed in <pkg>_LICENSE.
+$(2)_LICENSE += , vendored dependencies licenses probably not listed
+
 # Build step. Only define it if not already defined by the package .mk
 # file.
 ifndef $(2)_BUILD_CMDS
support/download/go-post-process new file +35 -0
new file mode 100755
index 0000000000..a6e74e6221
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -e
+
+. "${0%/*}/helpers"
+
+# Parse our options
+while getopts "n:o:" OPT; do
+    case "${OPT}" in
+    o)  output="${OPTARG}";;
+    n)  base_name="${OPTARG}";;
+    :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+    \?) error "unknown option '%s'\n" "${OPTARG}";;
+    esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
+    exit 0
+fi
+
+post_process_unpack ${base_name} ${output}
+
+# Do the Go vendoring
+pushd ${base_name} > /dev/null
+
+# Generate go.mod if it doesn't exist
+if [ ! -f go.mod ] && [ -n "${BR_GOMOD}" ]; then
+    printf "module ${BR_GOMOD}\n" > go.mod
+fi
+
+go mod vendor -v -modcacherw
+popd > /dev/null
+
+post_process_repack $(pwd) ${base_name} ${output}
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#8 lore raw

Re: [Buildroot] [PATCH v3 04/11] package/pkg-golang.mk: implement Go vendoring support

Hello Thomas,

Le 06/01/2022 à 21:59, Thomas Petazzoni a écrit :

Quoted 7 lines

This commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure.

The -modcacherw flag is added to ensure that the Go cache is read/write, and can be deleted properly upon "make clean".

Quoted 7 lines

The <pkg>_LICENSE variable of golang packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

---

I'm testing the golang infra with telegraf package and I got an error while downloading go modules:

"verifying [email protected]: checksum mismatch

	downloaded: h1:mRTLdljvxJNXPMMO9RSxf0PANDAqu/Tz+I6Dt6OjB28=
	go.sum:     h1:y4uFSAuOmeVhG3GCRa3/oH+ysePfO/+eGJNfd0Qa3d8=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum. The bits may have been replaced on the origin server, or an attacker may have intercepted the download attempt.

For more information, see 'go help module-auth'."

It seems there is something wrong in go.sum file from telegraf... but we are in download step not in patch step. It seems there is no way to fix locally (by patching) this problem.

This is not the first time that infludb/telegraf releases are broken due to change in git repo listed in go.sum.

Best regards,
Romain
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#9 lore raw

PATCH v3 5/11 package/pkg-cargo.mk: introduce the cargo package infrastructure

In reply to Thomas Petazzoni

From: Patrick Havelange <[email protected]>

In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

This commit also adds support/download/cargo-post-process to perform the vendoring on Cargo packages.

The <pkg>_LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

[Thomas: add support for host-cargo-package and vendoring]
---
 package/Makefile.in                 |   1 +
 package/pkg-cargo.mk                | 153 ++++++++++++++++++++++++++++
 support/download/cargo-post-process |  38 +++++++
 3 files changed, 192 insertions(+)
 create mode 100644 package/pkg-cargo.mk
 create mode 100755 support/download/cargo-post-process
package/Makefile.in +1 -0
index dae7a859fd..0ca2a5844e 100644
@@ -441,3 +441,4 @@ include package/pkg-waf.mk
 include package/pkg-golang.mk
 include package/pkg-meson.mk
 include package/pkg-qmake.mk
+include package/pkg-cargo.mk
package/pkg-cargo.mk new file +153 -0

153 lines not on this page · raw message

support/download/cargo-post-process new file +38 -0
new file mode 100755
index 0000000000..25dbbedbfa
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -e
+
+. "${0%/*}/helpers"
+
+while getopts "n:o:" OPT; do
+    case "${OPT}" in
+    o)  output="${OPTARG}";;
+    n)  base_name="${OPTARG}";;
+    :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+    \?) error "unknown option '%s'\n" "${OPTARG}";;
+    esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then
+    exit 0
+fi
+
+post_process_unpack ${base_name} ${output}
+
+# Do the Cargo vendoring
+pushd ${base_name} > /dev/null
+cargo vendor --locked VENDOR
+
+# Create the local .cargo/config with vendor info
+mkdir -p .cargo/
+cat <<EOF >.cargo/config
+[source.crates-io]
+replace-with = "vendored-sources"
+
+[source.vendored-sources]
+directory = "VENDOR"
+EOF
+popd > /dev/null
+
+post_process_repack $(pwd) ${base_name} ${output}
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#10 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

Quoted 76 lines · Thomas Petazzoni

From: Patrick Havelange <[email protected]>

In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

This commit also adds support/download/cargo-post-process to perform the vendoring on Cargo packages.

The <pkg>_LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

[Thomas: add support for host-cargo-package and vendoring]
---
 package/Makefile.in                 |   1 +
 package/pkg-cargo.mk                | 153 ++++++++++++++++++++++++++++
 support/download/cargo-post-process |  38 +++++++
 3 files changed, 192 insertions(+)
 create mode 100644 package/pkg-cargo.mk
 create mode 100755 support/download/cargo-post-process
package/Makefile.in +1 -0
index dae7a859fd..0ca2a5844e 100644
@@ -441,3 +441,4 @@ include package/pkg-waf.mk
 include package/pkg-golang.mk
 include package/pkg-meson.mk
 include package/pkg-qmake.mk
+include package/pkg-cargo.mk
package/pkg-cargo.mk new file +35 -0
new file mode 100644
index 0000000000..bf1436a86b
@@ -0,0 +1,153 @@
+################################################################################
+# Cargo package infrastructure
+#
+# This file implements an infrastructure that eases development of package
+# .mk files for Cargo packages. It should be used for all packages that use
+# Cargo as their build system.
+#
+# See the Buildroot documentation for details on the usage of this
+# infrastructure
+#
+# In terms of implementation, this Cargo infrastructure requires the .mk file
+# to only specify metadata information about the package: name, version,
+# download URL, etc.
+#
+# We still allow the package .mk file to override what the different steps
+# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined,
+# it is used as the list of commands to perform to build the package,
+# instead of the default Cargo behaviour. The package can also define some
+# post operation hooks.
+#
+################################################################################
+
+################################################################################
+# inner-cargo-package -- defines how the configuration, compilation and
+# installation of a cargo package should be done, implements a few hooks
+# to tune the build process for cargo specifities and calls the generic
+# package infrastructure to generate the necessary make targets
+#
+#  argument 1 is the lowercase package name
+#  argument 2 is the uppercase package name, including a HOST_ prefix
+#             for host packages
+#  argument 3 is the uppercase package name, without the HOST_ prefix
+#             for host packages
+#  argument 4 is the type (target or host)
+################################################################################

package/pkg-cargo.mk line 45

Quoted 10 lines
package/pkg-cargo.mk new file +10 -0
+
+define inner-cargo-package
+
+# We need host-rustc to run cargo
+$(2)_DOWNLOAD_DEPENDENCIES += host-rustc
+
+$(2)_CARGO_ENV += \
+       CARGO_HOME=$$(HOST_DIR)/share/cargo \
+       __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" \
+       CARGO_TARGET_APPLIES_TO_HOST="false"

Maybe want to set these additional options as well so that packages using cargo can
build with only the env variables set(in some cases at least I think), still probably
need to pass some command line args but this should reduce the amount at least:
RUSTC_TARGET_TRIPLE = $(subst -,_,$(call UPPERCASE,$(RUSTC_TARGET_NAME)))

PKG_RUST_CARGO_ENV = \
    CARGO_HOME=$(HOST_DIR)/share/cargo \
    CARGO_BUILD_TARGET=$(RUSTC_TARGET_NAME) \
    CARGO_INSTALL_ROOT=$(TARGET_DIR)/usr \
    CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc
HOST_PKG_RUST_CARGO_ENV = \
    CARGO_HOME=$(HOST_DIR)/share/cargo \
    CARGO_INSTALL_ROOT=$(HOST_DIR) \
    RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))"
Quoted 153 lines
package/pkg-cargo.mk new file +108 -0
+
+$(2)_DOWNLOAD_POST_PROCESS = cargo
+$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+# Due to vendoring, it is pretty likely that not all licenses are
+# listed in <pkg>_LICENSE.
+$(2)_LICENSE += , vendored dependencies licenses probably not listed
+
+# Note: in all the steps below, we "cd" into the build directory to
+# execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the
+# manifest-path. Indeed while the latter seems to work, it in fact
+# breaks in subtle ways as the way cargo searches for its
+# configuration file is based (among other rules) on the current
+# directory. This means that if cargo is started outside of a package
+# directory, its configuration file will not be taken into account.
+#
+# Also, we pass:
+#  * --offline to prevent cargo from downloading anything: all
+#    dependencies should have been built by the download post
+#    process logic
+#  * --locked to force cargo to use the Cargo.lock file, which ensures
+#    that a fixed set of dependency versions is used
+
+#
+# Build step. Only define it if not already defined by the package .mk
+# file.
+#
+ifndef $(2)_BUILD_CMDS
+ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+       cd $$(@D) && \
+       $$(TARGET_MAKE_ENV) \
+               $$(TARGET_CONFIGURE_OPTS) \
+               $$($(2)_CARGO_ENV) \
+               cargo build \
+                       --offline \
+                       --target $$(RUSTC_TARGET_NAME) \
+                       $$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
+                       --manifest-path Cargo.toml \
+                       --locked \
+                       -Z target-applies-to-host \
+                       $$($(2)_CARGO_BUILD_OPTS)
+endef
+else # ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+       cd $$(@D) && \
+       $$(HOST_MAKE_ENV) \
+               RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+               $$($(2)_CARGO_ENV) \
+               cargo build \
+                       --offline \
+                       --release \
+                       --manifest-path Cargo.toml \
+                       --locked \
+                       $$($(2)_CARGO_BUILD_OPTS)
+endef
+endif # ifeq ($(4),target)
+endif # ifndef $(2)_BUILD_CMDS
+
+#
+# Target installation step. Only define it if not already defined by
+# the package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+       cd $$(@D) && \
+       $$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
+               cargo install \
+                       --target $$(RUSTC_TARGET_NAME) \
+                       --offline \
+                       --root $$(TARGET_DIR)/usr/ \
+                       --bins \
+                       --path ./ \
+                       --force \
+                       --locked \
+                       $$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+       cd $$(@D) && \
+       $$(HOST_MAKE_ENV) \
+               RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+               $$($(2)_CARGO_ENV) \
+               cargo install \
+                       --offline \
+                       --root $$(HOST_DIR) \
+                       --bins \
+                       --path ./ \
+                       --force \
+                       --locked \
+                       $$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+# Call the generic package infrastructure to generate the necessary
+# make targets
+$(call inner-generic-package,$(1),$(2),$(3),$(4))
+
+endef
+
+################################################################################
+# cargo-package -- the target generator macro for Cargo packages
+################################################################################
+
+cargo-package = $(call inner-cargo-package,$(pkgname),$(call UPPERCASE,$(pkgname)),$(call UPPERCASE,$(pkgname)),target)
+host-cargo-package = $(call inner-cargo-package,host-$(pkgname),$(call UPPERCASE,host-$(pkgname)),$(call UPPERCASE,$(pkgname)),host)
support/download/cargo-post-process new file +38 -1
new file mode 100755
index 0000000000..25dbbedbfa
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -e
+
+. "${0%/*}/helpers"
+
+while getopts "n:o:" OPT; do
+    case "${OPT}" in
+    o)  output="${OPTARG}";;
+    n)  base_name="${OPTARG}";;
+    :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+    \?) error "unknown option '%s'\n" "${OPTARG}";;
+    esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then
+    exit 0
+fi
+
+post_process_unpack ${base_name} ${output}
+
+# Do the Cargo vendoring
+pushd ${base_name} > /dev/null
+cargo vendor --locked VENDOR
+
+# Create the local .cargo/config with vendor info
+mkdir -p .cargo/
+cat <<EOF >.cargo/config
+[source.crates-io]
+replace-with = "vendored-sources"
+
+[source.vendored-sources]
+directory = "VENDOR"
+EOF
+popd > /dev/null
+
+post_process_repack $(pwd) ${base_name} ${output}
--
Quoted 2 lines

2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#11 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

Hello James,

Quoted 8 lines · On Thu, 6 Jan 2022 19:12:51 -0600 James Hilliard <[email protected]>…

package/pkg-cargo.mk line 45

Quoted 4 lines

+$(2)_CARGO_ENV += \

package/pkg-cargo.mk new file +3 -0
+       CARGO_HOME=$$(HOST_DIR)/share/cargo \
+       __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" \
+       CARGO_TARGET_APPLIES_TO_HOST="false"  

Maybe want to set these additional options as well so that packages using cargo can
build with only the env variables set(in some cases at least I think),

Thanks for the suggestion. However, could you be more specific than "maybe" and "I think" ? The way you put it seems like we don't really know why those variables would be useful and what they do.

Quoted 3 lines

still probably
need to pass some command line args but this should reduce the amount at least:
RUSTC_TARGET_TRIPLE = $(subst -,_,$(call UPPERCASE,$(RUSTC_TARGET_NAME)))

Which command line arguments would be replaced by those variables?

Quoted 2 lines
PKG_RUST_CARGO_ENV = \
    CARGO_HOME=$(HOST_DIR)/share/cargo \

This one we already pass.

Quoted 1 line
    CARGO_BUILD_TARGET=$(RUSTC_TARGET_NAME) \

This one would replace the --target argument? Just to understand, what would be the advantage of the variable compared to the comand line argument?

Quoted 1 line
    CARGO_INSTALL_ROOT=$(TARGET_DIR)/usr \

This would replace --root $$(TARGET_DIR)/usr/ I suppose. Here as well, any benefit/drawback of arguments vs. variables?

Quoted 1 line
    CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc

How come things are working today with this one?

Quoted 3 lines
HOST_PKG_RUST_CARGO_ENV = \
    CARGO_HOME=$(HOST_DIR)/share/cargo \

This one already passed.

Quoted 1 line
    CARGO_INSTALL_ROOT=$(HOST_DIR) \

This would replace --root $$(HOST_DIR) I suppose.

Quoted 1 line
    RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))"

This is already passed, although replicated between the build and install steps of the host variant.

So essentially the only questions is:

- Advantage of variables vs. command line options

- Relevance of CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc

Thanks!

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#12 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

Quoted 28 lines · Thomas Petazzoni

Hello James,

Quoted 8 lines · On Thu, 6 Jan 2022 19:12:51 -0600 James Hilliard <[email protected]>…

package/pkg-cargo.mk line 45

Quoted 4 lines

+$(2)_CARGO_ENV += \

package/pkg-cargo.mk new file +3 -0
+       CARGO_HOME=$$(HOST_DIR)/share/cargo \
+       __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" \
+       CARGO_TARGET_APPLIES_TO_HOST="false"

Maybe want to set these additional options as well so that packages using cargo can
build with only the env variables set(in some cases at least I think),

Thanks for the suggestion. However, could you be more specific than "maybe" and "I think" ? The way you put it seems like we don't really know why those variables would be useful and what they do.

Quoted 3 lines

still probably
need to pass some command line args but this should reduce the amount at least:
RUSTC_TARGET_TRIPLE = $(subst -,_,$(call UPPERCASE,$(RUSTC_TARGET_NAME)))

Which command line arguments would be replaced by those variables?

Quoted 2 lines
PKG_RUST_CARGO_ENV = \
    CARGO_HOME=$(HOST_DIR)/share/cargo \
Quoted 7 lines

This one we already pass.

Quoted 1 line
    CARGO_BUILD_TARGET=$(RUSTC_TARGET_NAME) \

This one would replace the --target argument? Just to understand, what would be the advantage of the variable compared to the comand line argument?

Might be better for reusing this env for indirect cargo build invocations like those
done by python-setuptools-rust.

Quoted 5 lines
Quoted 2 lines
    CARGO_INSTALL_ROOT=$(TARGET_DIR)/usr \

This would replace --root $$(TARGET_DIR)/usr/ I suppose. Here as well, any benefit/drawback of arguments vs. variables?

Yeah, mostly for indict invocations as well.

Quoted 4 lines
Quoted 2 lines
    CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc

How come things are working today with this one?

I think this is currently set in the cargo config file.

Quoted 10 lines
Quoted 4 lines

HOST_PKG_RUST_CARGO_ENV = \
    CARGO_HOME=$(HOST_DIR)/share/cargo \

This one already passed.

Quoted 1 line
    CARGO_INSTALL_ROOT=$(HOST_DIR) \

This would replace --root $$(HOST_DIR) I suppose.

Yeah, sounds right.

Quoted 9 lines
Quoted 2 lines
    RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))"

This is already passed, although replicated between the build and install steps of the host variant.

So essentially the only questions is:

- Advantage of variables vs. command line options

Makes setting cargo variables when cargo is not directly invoked simpler.

Quoted 2 lines

- Relevance of CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc

Set via CARGO_HOME currently, but I think may allow the cargo config to not be required.

Quoted 8 lines

Thanks!

Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training
https://bootlin.com

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#13 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

Hello James,

Quoted 5 lines · On Fri, 7 Jan 2022 16:30:12 -0600 James Hilliard <[email protected]>…
Quoted 3 lines
Quoted 1 line
    CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc  

How come things are working today with this one?

I think this is currently set in the cargo config file.

Ah, right:

[target.@RUSTC_TARGET_NAME@]
linker = "@CROSS_PREFIX@gcc"

in package/rustc/cargo-config.in

Quoted 4 lines
Quoted 1 line

- Relevance of CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc

Set via CARGO_HOME currently, but I think may allow the cargo config to not be required.

I'm no sure why you say this is set via CARGO_HOME? Because the config file is stored in $(CARGO_HOME)/config ?

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#14 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

Quoted 19 lines · Thomas Petazzoni

Hello James,

Quoted 5 lines · On Fri, 7 Jan 2022 16:30:12 -0600 James Hilliard <[email protected]>…
Quoted 3 lines
Quoted 1 line
    CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc

How come things are working today with this one?

I think this is currently set in the cargo config file.

Ah, right:

[target.@RUSTC_TARGET_NAME@]
linker = "@CROSS_PREFIX@gcc"

in package/rustc/cargo-config.in

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

- Relevance of CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc

Set via CARGO_HOME currently, but I think may allow the cargo config to not be required.

I'm no sure why you say this is set via CARGO_HOME? Because the config file is stored in $(CARGO_HOME)/config ?

Yeah, that's what I mean.

Quoted 6 lines

Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training
https://bootlin.com

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#15 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

In reply to Thomas Petazzoni

Hi Thomas, James, all,

Quoted 8 lines · On Fri, 7 Jan 2022 23:59:59 +0100 Thomas Petazzoni…
Quoted 5 lines · On Fri, 7 Jan 2022 16:30:12 -0600 James Hilliard <[email protected]>…
Quoted 4 lines
Quoted 2 lines
    CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir

$(TARGET_CROSS))gcc

How come things are working today with this one?

Quoted 8 lines
Quoted 1 line

I think this is currently set in the cargo config file.

Ah, right:

[target.@RUSTC_TARGET_NAME@]
linker = "@CROSS_PREFIX@gcc"

in package/rustc/cargo-config.in

the config file has the problem that it is found via CARGO_HOME, which comes last in the search path, after searching upward in the directory hierarchy. When I used Rust under buildroot for the first time, a broken .cargo/config in my home directory which I had completely forgotten about caused the build to fail, and it took me a while to figure it out.

Environment variables take precedence over config files, so IMHO they are the better choice.

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines

- Relevance of
CARGO_TARGET_$(RUSTC_TARGET_TRIPLE)_LINKER=$(notdir $(TARGET_CROSS))gcc

Set via CARGO_HOME currently, but I think may allow the cargo config to not be required.

I'm no sure why you say this is set via CARGO_HOME? Because the config file is stored in $(CARGO_HOME)/config ?

Greetings

Jürgen

Signature · 9 lines
Jürgen Stuber <[email protected]>
http://www.jstuber.net/
Tel: +49-208-304 20 50
Mobil: +49-178-39 39 628
1B78 A579 E159 2A85 67BB  1314 C083 224B 0F9C DA21
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#16 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

In reply to Thomas Petazzoni

Hello Thomas, All,

Le 06/01/2022 à 21:59, Thomas Petazzoni a écrit :

Quoted 53 lines

From: Patrick Havelange <[email protected]>

In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

This commit also adds support/download/cargo-post-process to perform the vendoring on Cargo packages.

The <pkg>_LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

[Thomas: add support for host-cargo-package and vendoring]

---

package/pkg-cargo.mk new file +34 -0
+#
+# Build step. Only define it if not already defined by the package .mk
+# file.
+#
+ifndef $(2)_BUILD_CMDS
+ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	cd $$(@D) && \
+	$$(TARGET_MAKE_ENV) \
+		$$(TARGET_CONFIGURE_OPTS) \
+		$$($(2)_CARGO_ENV) \
+		cargo build \
+			--offline \
+			--target $$(RUSTC_TARGET_NAME) \
+			$$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
+			--manifest-path Cargo.toml \
+			--locked \
+			-Z target-applies-to-host \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+else # ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	cd $$(@D) && \
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \
+		cargo build \
+			--offline \
+			--release \
+			--manifest-path Cargo.toml \
+			--locked \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+endif # ifeq ($(4),target)

package/pkg-cargo.mk line 112

Quoted 10 lines
package/pkg-cargo.mk new file +10 -0
+endif # ifndef $(2)_BUILD_CMDS
+
+#
+# Target installation step. Only define it if not already defined by
+# the package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+	cd $$(@D) && \
+	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \

Removing "CC_$$(subst -,_,$$(RUSTC_TARGET_NAME))=$$(TARGET_CC) " [1] and adding $$(TARGET_CONFIGURE_OPTS) fixes the build of bat package.

The package is still rebuild during the install step but at least it build...

[1] http://lists.busybox.net/pipermail/buildroot/2022-January/632610.html

Best regards,
Romain

Quoted 20 lines
package/pkg-cargo.mk new file +19 -0
+		cargo install \
+			--target $$(RUSTC_TARGET_NAME) \
+			--offline \
+			--root $$(TARGET_DIR)/usr/ \
+			--bins \
+			--path ./ \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+	cd $$(@D) && \
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \
+		cargo install \

package/pkg-cargo.mk line 141

Quoted 10 lines
package/pkg-cargo.mk new file +10 -0
+			--offline \
+			--root $$(HOST_DIR) \
+			--bins \
+			--path ./ \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#17 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

In reply to Thomas Petazzoni

Le 06/01/2022 à 21:59, Thomas Petazzoni a écrit :

Quoted 8 lines

From: Patrick Havelange <[email protected]>

In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

This commit also adds support/download/cargo-post-process to perform the vendoring on Cargo packages.

Download post-processing does't work when we use _OVERRIDE_SRCDIR.

Maybe we have to make sure that all download of vendor modules are already present in the source directory used by _OVERRIDE_SRCDIR.

Best regards,
Romain

Quoted 9 lines

The <pkg>_LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

[Thomas: add support for host-cargo-package and vendoring]

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#18 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

Romain, All,

On 2022-01-07 11:26 +0100, Romain Naour spake thusly:

Quoted 10 lines

Le 06/01/2022 à 21:59, Thomas Petazzoni a écrit :

Quoted 7 lines

From: Patrick Havelange <[email protected]> In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

This commit also adds support/download/cargo-post-process to perform the vendoring on Cargo packages.

Download post-processing does't work when we use _OVERRIDE_SRCDIR.

This is indeed expected: _OVERRIDE_SRCDIR is skipping all the download, extract and patch steps.

Developers that are using an _OVERRIDE_SRCDIR are responsible for providing an source tree that is ready to be configured.

So, this is probably something that should be added to the manual, if it is not already there, or to emphasise it if already there:

When you are using OVERRIDE_SRCDIR, you are responsible for providing a source tree that is ready to be configured.

Care to send a patch, please?

Regards,
Yann E. MORIN.

Quoted 15 lines

Maybe we have to make sure that all download of vendor modules are already present in the source directory used by _OVERRIDE_SRCDIR.

Best regards,
Romain

Quoted 9 lines

The <pkg>_LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

[Thomas: add support for host-cargo-package and vendoring]
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#19 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

Le 07/01/2022 à 11:44, Yann E. MORIN a écrit :

Quoted 17 lines

Romain, All,

On 2022-01-07 11:26 +0100, Romain Naour spake thusly:

Quoted 10 lines

Le 06/01/2022 à 21:59, Thomas Petazzoni a écrit :

Quoted 7 lines

From: Patrick Havelange <[email protected]> In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

This commit also adds support/download/cargo-post-process to perform the vendoring on Cargo packages.

Download post-processing does't work when we use _OVERRIDE_SRCDIR.

This is indeed expected: _OVERRIDE_SRCDIR is skipping all the download, extract and patch steps.

Quoted 8 lines

Developers that are using an _OVERRIDE_SRCDIR are responsible for providing an source tree that is ready to be configured.

So, this is probably something that should be added to the manual, if it is not already there, or to emphasise it if already there:

When you are using OVERRIDE_SRCDIR, you are responsible for providing a source tree that is ready to be configured.

I did a try, It's not just adding vendor sources into package sources (in my case "bat" package).

We have to execute "cargo vendor" from "bat" sources and use CARGO_HOME used by Buildroot:

CARGO_HOME=$HOST_DIR/share/cargo cargo vendor

Do we want to document this method ?

Best regards,
Romain

Quoted 22 lines

Care to send a patch, please?

Regards,
Yann E. MORIN.

Quoted 15 lines

Maybe we have to make sure that all download of vendor modules are already present in the source directory used by _OVERRIDE_SRCDIR.

Best regards,
Romain

Quoted 9 lines

The <pkg>_LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

[Thomas: add support for host-cargo-package and vendoring]

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#20 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

In reply to Thomas Petazzoni

Hello,

Le 06/01/2022 à 21:59, Thomas Petazzoni a écrit :

Quoted 134 lines

From: Patrick Havelange <[email protected]>

In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

This commit also adds support/download/cargo-post-process to perform the vendoring on Cargo packages.

The <pkg>_LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

[Thomas: add support for host-cargo-package and vendoring]
---
 package/Makefile.in                 |   1 +
 package/pkg-cargo.mk                | 153 ++++++++++++++++++++++++++++
 support/download/cargo-post-process |  38 +++++++
 3 files changed, 192 insertions(+)
 create mode 100644 package/pkg-cargo.mk
 create mode 100755 support/download/cargo-post-process
package/Makefile.in +1 -0
index dae7a859fd..0ca2a5844e 100644
@@ -441,3 +441,4 @@ include package/pkg-waf.mk
 include package/pkg-golang.mk
 include package/pkg-meson.mk
 include package/pkg-qmake.mk
+include package/pkg-cargo.mk
package/pkg-cargo.mk new file +94 -0
new file mode 100644
index 0000000000..bf1436a86b
@@ -0,0 +1,153 @@
+################################################################################
+# Cargo package infrastructure
+#
+# This file implements an infrastructure that eases development of package
+# .mk files for Cargo packages. It should be used for all packages that use
+# Cargo as their build system.
+#
+# See the Buildroot documentation for details on the usage of this
+# infrastructure
+#
+# In terms of implementation, this Cargo infrastructure requires the .mk file
+# to only specify metadata information about the package: name, version,
+# download URL, etc.
+#
+# We still allow the package .mk file to override what the different steps
+# are doing, if needed. For example, if <PKG>_BUILD_CMDS is already defined,
+# it is used as the list of commands to perform to build the package,
+# instead of the default Cargo behaviour. The package can also define some
+# post operation hooks.
+#
+################################################################################
+
+################################################################################
+# inner-cargo-package -- defines how the configuration, compilation and
+# installation of a cargo package should be done, implements a few hooks
+# to tune the build process for cargo specifities and calls the generic
+# package infrastructure to generate the necessary make targets
+#
+#  argument 1 is the lowercase package name
+#  argument 2 is the uppercase package name, including a HOST_ prefix
+#             for host packages
+#  argument 3 is the uppercase package name, without the HOST_ prefix
+#             for host packages
+#  argument 4 is the type (target or host)
+################################################################################
+
+define inner-cargo-package
+
+# We need host-rustc to run cargo
+$(2)_DOWNLOAD_DEPENDENCIES += host-rustc
+
+$(2)_CARGO_ENV += \
+	CARGO_HOME=$$(HOST_DIR)/share/cargo \
+	__CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" \
+	CARGO_TARGET_APPLIES_TO_HOST="false"
+
+$(2)_DOWNLOAD_POST_PROCESS = cargo
+$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+# Due to vendoring, it is pretty likely that not all licenses are
+# listed in <pkg>_LICENSE.
+$(2)_LICENSE += , vendored dependencies licenses probably not listed
+
+# Note: in all the steps below, we "cd" into the build directory to
+# execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the
+# manifest-path. Indeed while the latter seems to work, it in fact
+# breaks in subtle ways as the way cargo searches for its
+# configuration file is based (among other rules) on the current
+# directory. This means that if cargo is started outside of a package
+# directory, its configuration file will not be taken into account.
+#
+# Also, we pass:
+#  * --offline to prevent cargo from downloading anything: all
+#    dependencies should have been built by the download post
+#    process logic
+#  * --locked to force cargo to use the Cargo.lock file, which ensures
+#    that a fixed set of dependency versions is used
+
+#
+# Build step. Only define it if not already defined by the package .mk
+# file.
+#
+ifndef $(2)_BUILD_CMDS
+ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	cd $$(@D) && \
+	$$(TARGET_MAKE_ENV) \
+		$$(TARGET_CONFIGURE_OPTS) \
+		$$($(2)_CARGO_ENV) \
+		cargo build \
+			--offline \
+			--target $$(RUSTC_TARGET_NAME) \
+			$$(if $$(BR2_ENABLE_DEBUG),--debug,--release) \
+			--manifest-path Cargo.toml \
+			--locked \
+			-Z target-applies-to-host \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+else # ifeq ($(4),target)
+define $(2)_BUILD_CMDS
+	cd $$(@D) && \
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \

package/pkg-cargo.mk line 104

Quoted 10 lines
package/pkg-cargo.mk new file +10 -0
+		cargo build \
+			--offline \
+			--release \
+			--manifest-path Cargo.toml \
+			--locked \
+			$$($(2)_CARGO_BUILD_OPTS)
+endef
+endif # ifeq ($(4),target)
+endif # ifndef $(2)_BUILD_CMDS
+

is _INSTALL_STAGING_CMDS is missing here ?

Rust package may wand to install *.rlib files [1] (similar to .a)

package/pkg-cargo.mk line 113

Quoted 9 lines
package/pkg-cargo.mk new file +9 -0
+#
+# Target installation step. Only define it if not already defined by
+# the package .mk file.
+#
+ifndef $(2)_INSTALL_TARGET_CMDS
+define $(2)_INSTALL_TARGET_CMDS
+	cd $$(@D) && \
+	$$(TARGET_MAKE_ENV) $$($(2)_CARGO_ENV) \
+		cargo install \

cargo install seems overingeneered and produce unwanted rebuild when packages are using cargo features [2].

meta-rust doesn't use cago install at all but use an cargo_do_install() function that use manual install commandes [3].

[1] https://doc.rust-lang.org/reference/linkage.html
[2] https://github.com/rust-lang/cargo/issues/8703
[3] https://github.com/meta-rust/meta-rust/blob/master/classes/cargo.bbclass#L55

Quoted 24 lines
package/pkg-cargo.mk new file +24 -0
+			--target $$(RUSTC_TARGET_NAME) \
+			--offline \
+			--root $$(TARGET_DIR)/usr/ \
+			--bins \
+			--path ./ \
+			--force \
+			--locked \
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+ifndef $(2)_INSTALL_CMDS
+define $(2)_INSTALL_CMDS
+	cd $$(@D) && \
+	$$(HOST_MAKE_ENV) \
+		RUSTFLAGS="$$(addprefix -C link-args=,$$(HOST_LDFLAGS))" \
+		$$($(2)_CARGO_ENV) \
+		cargo install \
+			--offline \
+			--root $$(HOST_DIR) \
+			--bins \
+			--path ./ \
+			--force \
+			--locked \

package/pkg-cargo.mk line 147

Quoted 10 lines
package/pkg-cargo.mk new file +10 -0
+			$$($(2)_CARGO_INSTALL_OPTS)
+endef
+endif
+
+# Call the generic package infrastructure to generate the necessary
+# make targets
+$(call inner-generic-package,$(1),$(2),$(3),$(4))
+
+endef
+

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#21 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

Hello,

Quoted 8 lines · On Fri, 7 Jan 2022 22:07:26 +0100 Romain Naour <[email protected]> wrote:

package/pkg-cargo.mk line 104

Quoted 4 lines
package/pkg-cargo.mk new file +4 -0
+endef
+endif # ifeq ($(4),target)
+endif # ifndef $(2)_BUILD_CMDS
+  

is _INSTALL_STAGING_CMDS is missing here ?

Rust package may wand to install *.rlib files [1] (similar to .a)

So far we haven't seen any package that needed staging installation, so I did not implement it (Yann asked a similar question today, live).

Quoted 2 lines

cargo install seems overingeneered and produce unwanted rebuild when packages are using cargo features [2].

Why do you say "over engineered" ? Buggy perhaps, over-engineered, I'm not sure.

Quoted 6 lines

meta-rust doesn't use cago install at all but use an cargo_do_install() function that use manual install commandes [3].

[1] https://doc.rust-lang.org/reference/linkage.html
[2] https://github.com/rust-lang/cargo/issues/8703
[3] https://github.com/meta-rust/meta-rust/blob/master/classes/cargo.bbclass#L55

I was wondering how they knew what to install and how/where, but that link explains it.

It would a lot simpler to be able to use "cargo install", but I agree those rebuilds during the install step are not very nice (and they are time consuming).

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#22 lore raw

Re: [Buildroot] [PATCH v3 05/11] package/pkg-cargo.mk: introduce the cargo package infrastructure

In reply to Romain Naour
Quoted 7 lines · On Fri, 7 Jan 2022 22:07:26 +0100 Romain Naour <[email protected]> wrote:

package/pkg-cargo.mk line 104

Quoted 3 lines
package/pkg-cargo.mk new file +3 -0
+endif # ifeq ($(4),target)
+endif # ifndef $(2)_BUILD_CMDS
+  

is _INSTALL_STAGING_CMDS is missing here ?

Rust package may wand to install *.rlib files [1] (similar to .a)

But will Cargo pick up .rlib files from STAGING_DIR ? I don't see how Cargo/Rust knows about STAGING_DIR. Aren't those .rlib files anyway installed in CARGO_HOME, and Cargo picks them up from here, or something like that?

Installing to STAGING_DIR is one thing, but what's important is to figure out if Cargo is really going to use what we install in STAGING_DIR.

Do we have a specific example where one package installs a .rlib and another package uses it?

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#23 lore raw

PATCH v3 6/11 docs/manual/cargo: document the cargo-package infrastructure

In reply to Thomas Petazzoni

From: Patrick Havelange <[email protected]>

The Buildroot manual was already providing some details on how to integrate Cargo packages, and those details now need to be updated with a proper documentation for the cargo-package infrastructure, as well as updates related to vendoring support, which affects how dependencies are handled.

[Thomas: numerous updates and extensions.]

docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management

Now that we have vendoring support for Cargo packages, let's rewrite the dependency management section in a more accurate way.

We drop the part about the local cache of the registry, because +CARGO_HOME+ in Buildroot points to $(HOST_DIR)/share/cargo, which is not shared between builds nor preserved accross builds, so its effect as a cache is limited.

---
 docs/manual/adding-packages-cargo.txt | 106 ++++++++++++--------------
 1 file changed, 47 insertions(+), 59 deletions(-)
docs/manual/adding-packages-cargo.txt +47 -59
index 8fcc80bcc6..fb3e7d0780 100644
@@ -1,7 +1,7 @@
 // -*- mode:doc; -*-
 // vim: set syntax=asciidoc:
 
-=== Integration of Cargo-based packages
+=== Infrastructure for Cargo-based packages
 
 Cargo is the package manager for the Rust programming language. It allows the
 user to build programs or libraries written in Rust, but it also downloads and
@@ -10,7 +10,7 @@ called "crates".
 
 [[cargo-package-tutorial]]
 
-==== Cargo-based package's +Config.in+ file
+==== +cargo-package+ tutorial
 
 The +Config.in+ file of Cargo-based package 'foo' should contain:
 
@@ -25,11 +25,7 @@ The +Config.in+ file of Cargo-based package 'foo' should contain:
 08: 	  http://foosoftware.org/foo/
 ---------------------------
 
-==== Cargo-based package's +.mk+ file
-
-Buildroot does not (yet) provide a dedicated package infrastructure for
-Cargo-based packages. So, we will explain how to write a +.mk+ file for such a
-package. Let's start with an example:
+And the +.mk+ file for this package should contain:
 
 ------------------------------
 01: ################################################################################
@@ -44,66 +40,58 @@ package. Let's start with an example:
 10: FOO_LICENSE = GPL-3.0+
 11: FOO_LICENSE_FILES = COPYING
 12:
-13: FOO_DEPENDENCIES = host-rustc
-14:
-15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-16:
-17: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE)
-18:
-19: FOO_CARGO_OPTS = \
-20: 	$(if $(BR2_ENABLE_DEBUG),,--release) \
-21: 	--target=$(RUSTC_TARGET_NAME) \
-22: 	--manifest-path=$(@D)/Cargo.toml
-23:
-24: define FOO_BUILD_CMDS
-25: 	$(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \
-26: 		cargo build $(FOO_CARGO_OPTS)
-27: endef
-28:
-29: define FOO_INSTALL_TARGET_CMDS
-30: 	$(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \
-31: 		$(TARGET_DIR)/usr/bin/foo
-32: endef
-33:
-34: $(eval $(generic-package))
+13: $(eval $(cargo-package))
 --------------------------------
 
-The Makefile starts with the definition of the standard variables for package
-declaration (lines 7 to 11).
+The Makefile starts with the definition of the standard variables for
+package declaration (lines 7 to 11).
 
-As seen in line 34, it is based on the
-xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines
-the variables required by this particular infrastructure, where Cargo is
-invoked:
+As seen in line 13, it is based on the +cargo-package+
+infrastructure. Cargo will be invoked automatically by this
+infrastructure to build and install the package.
 
-* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required
-  to configure the cross-compilation of the package are passed via
-  +FOO_CONF_OPTS+.
+It is still possible to define custom build commands or install
+commands (i.e.  with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS).
+Those will then replace the commands from the cargo infrastructure.
 
-* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on
-  the target.
+==== +cargo-package+ reference
 
-In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to
-contain +host-cargo+.
+The main macros for the Cargo package infrastructure are
++cargo-package+ for target packages and +host-cargo-package+ for host
+packages.
 
-To sum it up, to add a new Cargo-based package, the Makefile example can be
-copied verbatim then edited to replace all occurences of +FOO+ with the
-uppercase name of the new package and update the values of the standard
-variables.
+Just like the generic infrastructure, the Cargo infrastructure works
+by defining a number of variables before calling the +cargo-package+
+or +host-cargo-package+ macros.
 
-==== About Dependencies Management
+First, all the package metadata information variables that exist in
+the generic infrastructure also exist in the Cargo infrastructure:
++FOO_VERSION+, +FOO_SOURCE+, +FOO_PATCH+, +FOO_SITE+,
++FOO_DEPENDENCIES+, +FOO_LICENSE+, +FOO_LICENSE_FILES+, etc.
+
+A few additional variables, specific to the Cargo infrastructure, can
+also be defined. Many of them are only useful in very specific cases,
+typical packages will therefore only use a few of them.
+
+* +FOO_CARGO_ENV+ can be used to pass additional variables in the
+  environment of +cargo+ invocations. It used at both build and
+  installation time
 
-A crate can depend on other libraries from crates.io or git repositories, listed
-in its Cargo.toml file. Before starting a build, Cargo usually downloads
-automatically them. This step can also be performed independently, via the
-+cargo fetch+ command.
+* +FOO_CARGO_BUILD_OPTS+ can be used to pass additional options to
+  +cargo+ at build time.
+
+* +FOO_CARGO_INSTALL_OPTS+ can be used to pass additional options to
+  +cargo+ at install time.
+
+==== About Dependencies Management
 
-Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+. As seen in the package
-Makefile example at line 15, this environment variable is set to
-+$(HOST_DIR)/share/cargo+.
+A crate can depend on other libraries from crates.io or git
+repositories, listed in its Cargo.toml file. Buildroot automatically
+takes care of downloading such dependencies as part of the download
+step of packages that use the +cargo-package+ infrastructure. Such
+dependencies are then kept together with the package source code in
+the tarball cached in Buildroot's +DL_DIR+, and therefore the hash of
+the package's tarball includes such dependencies.
 
-This dependency download mechanism is not convenient when performing an offline
-build, as Cargo will fail to fetch the dependencies. In that case, it is advised
-to generate a tarball of the dependencies using the +cargo vendor+ and add it to
-+FOO_EXTRA_DOWNLOADS+.
+This mechanism ensures that any change in the dependencies will be
+detected, and allows the build to be performed completely offline.
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#24 lore raw

PATCH v3 7/11 package/ripgrep: convert to cargo infrastructure

In reply to Thomas Petazzoni

From: Patrick Havelange <[email protected]>

This commit converts the ripgrep package to the cargo infrastructure. This causes the vendoring mechanism to kick in, which changes the contents of the tarball. To avoid a clash on the hash, we change the version of ripgrep to the Git commit hash matching the 13.0.0 tag, so there are no functional changes, but the tarball name is different.

---
 package/ripgrep/ripgrep.hash |  2 +-
 package/ripgrep/ripgrep.mk   | 37 ++++++------------------------------
 2 files changed, 7 insertions(+), 32 deletions(-)
package/ripgrep/ripgrep.hash +1 -1
index b2f03a95e6..cd2dbedfb4 100644
@@ -1,3 +1,3 @@
 # Locally calculated
-sha256 0fb17aaf285b3eee8ddab17b833af1e190d73de317ff9648751ab0660d763ed2  ripgrep-13.0.0.tar.gz
+sha256 6f1d4a8b653ce48d59ad777288b1257cbda607db29db19d031b7e622c60526f8  ripgrep-af6b6c543b224d348a8876f0c06245d9ea7929c5.tar.gz
 sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f  LICENSE-MIT
package/ripgrep/ripgrep.mk +6 -31
index 7c2b79a4a7..d587441cfa 100644
@@ -4,7 +4,11 @@
 #
 ################################################################################
 
-RIPGREP_VERSION = 13.0.0
+# Same as 13.0.0, we use a Git commit hash because the hash of this
+# tarball changed when moving to the cargo-package infrastructure, and
+# we can't change the hash of existing tarball. Please switch back to
+# a Git tag at the next version bump.
+RIPGREP_VERSION = af6b6c543b224d348a8876f0c06245d9ea7929c5
 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
 RIPGREP_LICENSE = MIT
 RIPGREP_LICENSE_FILES = LICENSE-MIT
@@ -13,33 +17,4 @@ RIPGREP_CPE_ID_VENDOR = ripgrep_project
 # CVE only impacts ripgrep on Windows
 RIPGREP_IGNORE_CVES += CVE-2021-3013
 
-RIPGREP_DEPENDENCIES = host-rustc
-RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo \
-	__CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" \
-	CARGO_TARGET_APPLIES_TO_HOST="false"
-
-RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_BIN_SUBDIR)
-
-RIPGREP_CARGO_OPTS = \
-	-Z target-applies-to-host \
-	--target=$(RUSTC_TARGET_NAME) \
-	--manifest-path=$(@D)/Cargo.toml
-
-ifeq ($(BR2_ENABLE_RUNTIME_DEBUG),y)
-RIPGREP_CARGO_BIN_SUBDIR = debug
-else
-RIPGREP_CARGO_OPTS += --release
-RIPGREP_CARGO_BIN_SUBDIR = release
-endif
-
-define RIPGREP_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \
-		cargo build $(RIPGREP_CARGO_OPTS)
-endef
-
-define RIPGREP_INSTALL_TARGET_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \
-		$(TARGET_DIR)/usr/bin/rg
-endef
-
-$(eval $(generic-package))
+$(eval $(cargo-package))
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#25 lore raw

PATCH v3 8/11 package/sentry-cli: re-add package

In reply to Thomas Petazzoni

This reverts commit d19077677fc71fc8c8cd73945606f773d907a99a, but the package now uses the cargo-package infrastructure with its vendoring mechanism, which allows us to avoid the problems that caused the package to be removed in d19077677fc71fc8c8cd73945606f773d907a99a.

---
 DEVELOPERS                         |  1 +
 package/Config.in.host             |  1 +
 package/sentry-cli/Config.in.host  | 10 ++++++++++
 package/sentry-cli/sentry-cli.hash |  3 +++
 package/sentry-cli/sentry-cli.mk   | 14 ++++++++++++++
 5 files changed, 29 insertions(+)
 create mode 100644 package/sentry-cli/Config.in.host
 create mode 100644 package/sentry-cli/sentry-cli.hash
 create mode 100644 package/sentry-cli/sentry-cli.mk
DEVELOPERS +1 -0
index a52a8cfa38..55ea7805f7 100644
@@ -1508,6 +1508,7 @@ F:	package/python-schedule/
 F:	package/python-sentry-sdk/
 F:	package/python-websockets/
 F:	package/python-xlib/
+F:	package/sentry-cli/
 F:	package/sentry-native/
 F:	package/unclutter-xfixes/
 
package/Config.in.host +1 -0
index 7400da5894..b3e00bb482 100644
@@ -81,6 +81,7 @@ menu "Host utilities"
 	source "package/s6-rc/Config.in.host"
 	source "package/sam-ba/Config.in.host"
 	source "package/sdbusplus/Config.in.host"
+	source "package/sentry-cli/Config.in.host"
 	source "package/sloci-image/Config.in.host"
 	source "package/squashfs/Config.in.host"
 	source "package/sunxi-tools/Config.in.host"
package/sentry-cli/Config.in.host new file +10 -0
new file mode 100644
index 0000000000..3124015367
@@ -0,0 +1,10 @@
+config BR2_PACKAGE_HOST_SENTRY_CLI
+	bool "host sentry-cli"
+	depends on BR2_PACKAGE_HOST_RUSTC_ARCH_SUPPORTS
+	select BR2_PACKAGE_HOST_RUSTC
+	help
+	  Official Sentry command line interface
+
+	  For managing debug information files and source maps.
+
+	  https://docs.sentry.io/cli/
package/sentry-cli/sentry-cli.hash new file +3 -0
new file mode 100644
index 0000000000..73f818125f
@@ -0,0 +1,3 @@
+# locally calculated
+sha256  d687336379fd96c4ee73f24bd062d31672bd7453af53311b72977be3c8c78fef  sentry-cli-1.59.0.tar.gz
+sha256  9503def7b54ceb6e3cd182fd59bc05d3a30d7eae481e65aaba4b495133c83c14  LICENSE
package/sentry-cli/sentry-cli.mk new file +14 -0
new file mode 100644
index 0000000000..58f5f1e325
@@ -0,0 +1,14 @@
+################################################################################
+#
+# sentry-cli
+#
+################################################################################
+
+SENTRY_CLI_VERSION = 1.59.0
+SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
+SENTRY_CLI_LICENSE = BSD-3-clause
+SENTRY_CLI_LICENSE_FILES = LICENSE
+
+HOST_SENTRY_CLI_DEPENDENCIES = host-zlib
+
+$(eval $(host-cargo-package))
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#26 lore raw

Re: [Buildroot] [PATCH v3 08/11] package/sentry-cli: re-add package

Hi Thomas, all,

I get a build error with "make host-sentry-cli":

error: failed to run custom build command for libc v0.2.71

Caused by:
process didn't exit successfully:
/build/host-sentry-cli-1.59.0/target/release/build/libc-a39b7215be30da0c/build-script-build (exit status: 1)
--- stderr /build/host-sentry-cli-1.59.0/target/release/build/libc-a39b7215be30da0c/build-script-build:
/lib64/libc.so.6: version `GLIBC_2.34' not found (required by /build/host-sentry-cli-1.59.0/target/release/build/libc-a39b7215be30da0c/build-script-build)

I guess this is a rust-related issue? I think it's complaining about the glibc version of my build system.

On Thu, Jan 6, 2022 at 1:00 PM Thomas Petazzoni
<[email protected]> wrote:
###############################################

package/sentry-cli/sentry-cli.mk line 8

Quoted 2 lines
package/sentry-cli/sentry-cli.mk new file +2 -0
+SENTRY_CLI_VERSION = 1.59.0
+SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))

Thanks & best regards,
Christian
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#27 lore raw

Re: [Buildroot] [PATCH v3 08/11] package/sentry-cli: re-add package

Hello Christian,

Quoted 13 lines · On Thu, 6 Jan 2022 14:04:56 -0800 Christian Stewart <[email protected]> wrote:

I get a build error with "make host-sentry-cli":

error: failed to run custom build command for libc v0.2.71

Caused by:
process didn't exit successfully:
/build/host-sentry-cli-1.59.0/target/release/build/libc-a39b7215be30da0c/build-script-build (exit status: 1)
--- stderr /build/host-sentry-cli-1.59.0/target/release/build/libc-a39b7215be30da0c/build-script-build:
/lib64/libc.so.6: version `GLIBC_2.34' not found (required by /build/host-sentry-cli-1.59.0/target/release/build/libc-a39b7215be30da0c/build-script-build)

Quoted 2 lines

I guess this is a rust-related issue? I think it's complaining about the glibc version of my build system.

It build fine on two machines here. Could you provide more details about your host distribution? Perhaps provide a docker container that reproduces this?

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#28 lore raw

Re: [Buildroot] [PATCH v3 08/11] package/sentry-cli: re-add package

Hi Thomas,

Quoted 5 lines · Thomas Petazzoni
Quoted 2 lines · On Thu, 6 Jan 2022 14:04:56 -0800 Christian Stewart <[email protected]> wrote:

I get a build error with "make host-sentry-cli":

Quoted 7 lines
Quoted 3 lines

error: failed to run custom build command for libc v0.2.71

/lib64/libc.so.6: version `GLIBC_2.34' not found (required by

It build fine on two machines here. Could you provide more details about your host distribution? Perhaps provide a docker container that reproduces this?

I'm running an up-to-date Gentoo system at version 2.33.

Currently 2.34 is masked:

https://packages.gentoo.org/packages/sys-libs/glibc

I guess the host-rust packages expect at least glibc 2.34.

Best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#29 lore raw

Re: [Buildroot] [PATCH v3 08/11] package/sentry-cli: re-add package

Hello,

Quoted 7 lines · On Thu, 6 Jan 2022 14:38:06 -0800 Christian Stewart <[email protected]> wrote:

I'm running an up-to-date Gentoo system at version 2.33.

Currently 2.34 is masked:

https://packages.gentoo.org/packages/sys-libs/glibc

I guess the host-rust packages expect at least glibc 2.34.

Seems unlikely, because I just built the following configuration in the buildroot/base:20200814.2228 container, which is very old Debian with glibc 2.24 (yes 2.24).

BR2_aarch64=y
BR2_TOOLCHAIN_EXTERNAL=y BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_AARCH64_GLIBC_STABLE=y BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
# BR2_TARGET_ROOTFS_TAR is not set BR2_PACKAGE_HOST_SENTRY_CLI=y

$ dpkg -l | grep libc6
ii  libc6:amd64                2.24-11+deb9u1                 amd64        GNU C Library: Shared libraries
ii  libc6:i386                 2.24-11+deb9u1                 i386         GNU C Library: Shared libraries

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#30 lore raw

PATCH v3 9/11 package/tinifier: new package

In reply to Thomas Petazzoni

This is a Go package that needs vendor modules to be downloaded at build time.

---
 DEVELOPERS                     |  1 +
 package/Config.in              |  1 +
 package/tinifier/Config.in     | 10 ++++++++++
 package/tinifier/tinifier.hash |  3 +++
 package/tinifier/tinifier.mk   | 13 +++++++++++++
 5 files changed, 28 insertions(+)
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
DEVELOPERS +1 -0
index 55ea7805f7..774deb012f 100644
@@ -2765,6 +2765,7 @@ F:	package/rtc-tools/
 F:	package/sam-ba/
 F:	package/scons/
 F:	package/squashfs/
+F:	package/tinifier/
 F:	package/wayland/
 F:	package/weston/
 F:	support/testing/tests/boot/test_syslinux.py
package/Config.in +1 -0
index cb15574746..e47c49ade5 100644
@@ -314,6 +314,7 @@ comment "Graphic applications"
 	source "package/rrdtool/Config.in"
 	source "package/stellarium/Config.in"
 	source "package/tesseract-ocr/Config.in"
+	source "package/tinifier/Config.in"
 
 comment "Graphic libraries"
 	source "package/cegui/Config.in"
package/tinifier/Config.in new file +10 -0
new file mode 100644
index 0000000000..fbadfe6bd9
@@ -0,0 +1,10 @@
+config BR2_PACKAGE_TINIFIER
+	bool "tinifier"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	help
+	  CLI tool for images compressing
+
+	  This tool uses tinypng.com API endpoint for compressing your
+	  local jpg/png images (it supports parallel jobs).
+
+	  https://github.com/tarampampam/tinifier
package/tinifier/tinifier.hash new file +3 -0
new file mode 100644
index 0000000000..5681ed6a55
@@ -0,0 +1,3 @@
+# Locally calculated
+sha256  927e966f9d23ea7cc1942967a8fc16ef4c55c51bd096d44f54b418844a51aa12  tinifier-3.4.0.tar.gz
+sha256  791d8fd993ace44d4d83e2f4820a64d5ad3e37412f029afad46d17a9259de2b6  LICENSE
package/tinifier/tinifier.mk new file +13 -0
new file mode 100644
index 0000000000..d7ed6f7c93
@@ -0,0 +1,13 @@
+################################################################################
+#
+# tinifier
+#
+################################################################################
+
+TINIFIER_VERSION = 3.4.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
+TINIFIER_LICENSE = MIT
+TINIFIER_LICENSE_FILES = LICENSE
+TINIFIER_GOMOD = ./cmd/tinifier
+
+$(eval $(golang-package))
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Reviewed-by Christian Stewart on v3 9/11
#31 lore raw

Re: [Buildroot] [PATCH v3 09/11] package/tinifier: new package

Hi all,

package/tinifier/tinifier.mk line 8

Quoted 8 lines · Thomas Petazzoni

This is a Go package that needs vendor modules to be downloaded at build time.

---

package/tinifier/tinifier.mk new file +2 -0
+TINIFIER_VERSION = 3.4.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))

Build tested.

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#32 lore raw

PATCH v3 10/11 package/embiggen-disk: new package

In reply to Thomas Petazzoni

From: Christian Stewart via buildroot <[email protected]>

embiggen-disk is a tool to automatically resize disks to fill available space.

Patch submitted upstream: https://github.com/bradfitz/embiggen-disk/pull/13

Adds support for /dev/mmcblk0pN type paths.

---
 DEVELOPERS                                    |  1 +
 package/Config.in                             |  1 +
 ...-Fix-resizing-of-dev-mmcblk0pN-paths.patch | 45 +++++++++++++++++++
 package/embiggen-disk/Config.in               | 19 ++++++++
 package/embiggen-disk/embiggen-disk.hash      |  3 ++
 package/embiggen-disk/embiggen-disk.mk        | 12 +++++
 6 files changed, 81 insertions(+)
 create mode 100644 package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch
 create mode 100644 package/embiggen-disk/Config.in
 create mode 100644 package/embiggen-disk/embiggen-disk.hash
 create mode 100644 package/embiggen-disk/embiggen-disk.mk
DEVELOPERS +1 -0
index 774deb012f..2649cf4ac4 100644
@@ -518,6 +518,7 @@ F:	package/delve/
 F:	package/docker-cli/
 F:	package/docker-engine/
 F:	package/docker-proxy/
+F:	package/embiggen-disk/
 F:	package/fuse-overlayfs/
 F:	package/go/
 F:	package/mbpfan/
package/Config.in +1 -0
index e47c49ade5..34a36279ea 100644
@@ -2516,6 +2516,7 @@ menu "System tools"
 	source "package/earlyoom/Config.in"
 	source "package/efibootmgr/Config.in"
 	source "package/efivar/Config.in"
+	source "package/embiggen-disk/Config.in"
 	source "package/emlog/Config.in"
 	source "package/ftop/Config.in"
 	source "package/getent/Config.in"
package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch new file +45 -0

45 lines not on this page · raw message

package/embiggen-disk/Config.in new file +19 -0
new file mode 100644
index 0000000000..667d71c86e
@@ -0,0 +1,19 @@
+config BR2_PACKAGE_EMBIGGEN_DISK
+	bool "embiggen-disk"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	depends on BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
+	depends on BR2_TOOLCHAIN_HAS_THREADS
+	depends on !BR2_TOOLCHAIN_USES_UCLIBC # no fexecve
+	depends on BR2_USE_MMU # util-linux
+	select BR2_PACKAGE_UTIL_LINUX # sfdisk
+	select BR2_PACKAGE_UTIL_LINUX_BINARIES # sfdisk
+	help
+	  embiggen-disk is a tool to resize disk partitions at runtime.
+
+	  https://github.com/bradfitz/embiggen-disk
+
+comment "embiggen-disk needs a glibc or musl toolchain w/ threads"
+	depends on BR2_USE_MMU
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS && \
+		BR2_PACKAGE_HOST_GO_TARGET_CGO_LINKING_SUPPORTS
+	depends on !BR2_TOOLCHAIN_HAS_THREADS || BR2_TOOLCHAIN_USES_UCLIBC
package/embiggen-disk/embiggen-disk.hash new file +3 -0
new file mode 100644
index 0000000000..274958edc4
@@ -0,0 +1,3 @@
+# Locally computed
+sha256  5ef943b1f6468cf0eaf9d7e6ca22d8b5692cbb5b80f19d95468fb41b675b986c  embiggen-disk-c554fc1c93a4004ce0b6a3f69b0dabe0481a2308.tar.gz
+sha256  063aedec1652c5a05c2d04c40e032b932453142ee8ef7fd53c04a9acc127fc95  LICENSE
package/embiggen-disk/embiggen-disk.mk new file +12 -0
new file mode 100644
index 0000000000..5d5ac15ae8
@@ -0,0 +1,12 @@
+################################################################################
+#
+# embiggen-disk
+#
+################################################################################
+
+EMBIGGEN_DISK_VERSION = c554fc1c93a4004ce0b6a3f69b0dabe0481a2308
+EMBIGGEN_DISK_SITE = $(call github,bradfitz,embiggen-disk,$(EMBIGGEN_DISK_VERSION))
+EMBIGGEN_DISK_LICENSE = Apache-2.0
+EMBIGGEN_DISK_LICENSE_FILES = LICENSE
+
+$(eval $(golang-package))
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#33 lore raw

Re: [Buildroot] [PATCH v3 10/11] package/embiggen-disk: new package

Hi all,

package/embiggen-disk/embiggen-disk.mk line 8

Quoted 7 lines · Thomas Petazzoni

From: Christian Stewart via buildroot <[email protected]>

embiggen-disk is a tool to automatically resize disks to fill available space.

package/embiggen-disk/embiggen-disk.mk new file +2 -0
+EMBIGGEN_DISK_VERSION = c554fc1c93a4004ce0b6a3f69b0dabe0481a2308
+EMBIGGEN_DISK_SITE = $(call github,bradfitz,embiggen-disk,$(EMBIGGEN_DISK_VERSION))

The PR to fix mmcblk resizing was just merged (thanks bradfitz):

https://github.com/bradfitz/embiggen-disk/pull/13#event-5854783562

So the patch can be dropped and the VERSION changed to:

9e7b2fc7b99c4dece41a805489a6ca377ce55a62

Thanks & best,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#34 lore raw

PATCH v3 11/11 package/gocryptfs: new package

In reply to Thomas Petazzoni

From: Christian Stewart via buildroot <[email protected]>

Adds the gocryptfs encrypted FUSE filesystem.

Currently uses without_openssl build tag, to use the native Go cryptography. However, the package could be improved by conditionally enabling openssl if it is also configured to be built by Buildroot.

---
 DEVELOPERS                       |  1 +
 package/Config.in                |  1 +
 package/gocryptfs/Config.in      | 12 ++++++++++++
 package/gocryptfs/gocryptfs.hash |  3 +++
 package/gocryptfs/gocryptfs.mk   | 19 +++++++++++++++++++
 5 files changed, 36 insertions(+)
 create mode 100644 package/gocryptfs/Config.in
 create mode 100644 package/gocryptfs/gocryptfs.hash
 create mode 100644 package/gocryptfs/gocryptfs.mk
DEVELOPERS +1 -0
index 2649cf4ac4..bf58a20fc4 100644
@@ -521,6 +521,7 @@ F:	package/docker-proxy/
 F:	package/embiggen-disk/
 F:	package/fuse-overlayfs/
 F:	package/go/
+F:	package/gocryptfs/
 F:	package/mbpfan/
 F:	package/mosh/
 F:	package/pkg-golang.mk
package/Config.in +1 -0
index 34a36279ea..2946ce7a4b 100644
@@ -221,6 +221,7 @@ menu "Filesystem and flash utilities"
 	source "package/genext2fs/Config.in"
 	source "package/genpart/Config.in"
 	source "package/genromfs/Config.in"
+	source "package/gocryptfs/Config.in"
 	source "package/imx-usb-loader/Config.in"
 	source "package/mmc-utils/Config.in"
 	source "package/mtd/Config.in"
package/gocryptfs/Config.in new file +12 -0
new file mode 100644
index 0000000000..2630cafdcf
@@ -0,0 +1,12 @@
+config BR2_PACKAGE_GOCRYPTFS
+	bool "gocryptfs"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	depends on BR2_TOOLCHAIN_HAS_THREADS
+	help
+	  gocryptfs is an encrypted FUSE overlay filesystem.
+
+	  https://github.com/rfjakob/gocryptfs
+
+comment "gocryptfs needs a toolchain w/ threads"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	depends on !BR2_TOOLCHAIN_HAS_THREADS
package/gocryptfs/gocryptfs.hash new file +3 -0
new file mode 100644
index 0000000000..94487815f5
@@ -0,0 +1,3 @@
+# Locally computed
+sha256  4d89b76fe0b0c7218099f6f0ea6c1c87efe13927d95579a6ede82f20b6dbe48a  gocryptfs-2.2.0.tar.gz
+sha256  322a7e3b02cf18e38b7e6b18cafefb773df8676c65634b34e8a2beb931294a4b  LICENSE
package/gocryptfs/gocryptfs.mk new file +19 -0
new file mode 100644
index 0000000000..2a00e9a959
@@ -0,0 +1,19 @@
+################################################################################
+#
+# gocryptfs
+#
+################################################################################
+
+GOCRYPTFS_VERSION = 2.2.0
+GOCRYPTFS_SITE = $(call github,rfjakob,gocryptfs,v$(GOCRYPTFS_VERSION))
+GOCRYPTFS_LICENSE = MIT
+GOCRYPTFS_LICENSE_FILES = LICENSE
+
+GOCRYPTFS_GOMOD = github.com/rfjakob/gocryptfs/v2
+
+GOCRYPTFS_LDFLAGS = \
+	-X main.GitVersion=$(GOCRYPTFS_VERSION) \
+	-X main.GitVersionFuse=[vendored]
+GOCRYPTFS_TAGS = without_openssl
+
+$(eval $(golang-package))
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#35 lore raw
In reply to Thomas Petazzoni

Hi Thomas,

Your time spent on this series is much appreciated.

Quoted 3 lines · Thomas Petazzoni

Christian Stewart via buildroot (2):
package/embiggen-disk: new package
package/gocryptfs: new package

It'd be nice if the author on these commits could be edited to remove the "via buildroot"

... this was added by a Patchwork bug a while back.

Best regards,
Christian
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#36 lore raw
Quoted 4 lines · On Thu, 6 Jan 2022 14:26:13 -0800 Christian Stewart <[email protected]> wrote:

It'd be nice if the author on these commits could be edited to remove the "via buildroot"

... this was added by a Patchwork bug a while back.

ACK, will fix!

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#37 lore raw
In reply to Thomas Petazzoni

Hello,

Quoted 9 lines · On Thu, 6 Jan 2022 21:59:48 +0100 Thomas Petazzoni…

Christian Stewart via buildroot (2):
package/embiggen-disk: new package
package/gocryptfs: new package

Patrick Havelange (3):
package/pkg-cargo.mk: introduce the cargo package infrastructure docs/manual/cargo: document the cargo-package infrastructure package/ripgrep: convert to cargo infrastructure

Quoted 8 lines

Thomas Petazzoni (6):
support/download/dl-wrapper: add concept of download post-processing package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable support/download/post-process-helpers: add helper function for post
process scripts
package/pkg-golang.mk: implement Go vendoring support package/sentry-cli: re-add package
package/tinifier: new package

So, this series has been applied, with some additional tweaks based on your feedback + review from Yann and Arnout.

Here are the things that have *not* yet been taken into account, and left for future work:

* We still use "cargo install" even though that causes rebuilds with some packages. Either we switch to a manual installation, or we "wait" for some better solution from cargo maintainers.

* We don't use environment variables for everything, and some flags are passed through a cargo config file, which is not ideal.

* We only support cargo/go packages that use .tar.gz tarballs, because the repacking with mk_tar_gz assumes .tar.gz is used.

These 3 limitations have been captured in the TODO list at
https://elinux.org/Buildroot#Core_Buildroot_infrastructure. Patches welcome to implement these things of course, but we wanted to get the initial infrastructure in place and merged, so that we can build on top of it.

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot