Skip to contentSkip to conversation

Support for Cargo and Go vendoring

PATCH In review Thomas Petazzoni 5 participants 24 messages Dec 19, 2020 – Sep 19, 2021 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

v2 cover Support for Cargo and Go vendoring

Hello,

Here is a patch series that implements support for vendoring of Cargo and Go packages.

As you're aware, many Cargo and Go packages describe their dependencies in a very language-specific way, and expect such dependencies to be downloaded by their language-specific tools.

This series builds on top of the work from Patrick Havelange <[email protected]> who worked on the Cargo integration, but extends it to also cover the case of Go, showing that we have a solution solving both of these language-specific package managers. For the Go vendoring, I've also used the indication provided by Christian Stewart <[email protected]>.

Overall, the solution consists in a concept of "download post-process helper" is introduced in the download infrastructure. These are shell scripts that can be invoked right after the download is done, but before the tarball is hash-checked and stored in DL_DIR. The idea is that such "download post-process helpers" can run the language-specific vendoring logic to fetch the package dependencies. Thanks to this, the tarball in DL_DIR for a given package not only contains the package source code itself, but also the source code of its Cargo or Go dependencies. The tarball hash covers the entire source code, the complete tarball is cached in DL_DIR, in the primary site and backup site, and the build can fully be done offline.

Such "download post-process helpers" are registered by means of the <pkg>_DOWNLOAD_POST_PROCESS variable by the appropriate package infrastructure.

This series is also available at:

https://github.com/tpetazzoni/buildroot/commits/pkg-mgr-v2

More specifically, the series goes like this:

- PATCH 1: add the concept of download post-processing in the download helper support/download/dl-wrapper

- PATCH 2: add the <pkg>_DOWNLOAD_POST_PROCESS variable to the package infrastructure, which gets passed down to support/download/dl-wrapper

- PATCH 3: add the <pkg>_DL_ENV variable to the package infrastructure, which allows to pass extra variables in the download wrapper environment. This will be used to pass extra variables specific to Cargo/Go vendoring.

- PATCH 4: add a set of two helper shell functions that will be used by the post processing scripts

- PATCH 5: implements the Go vendoring itself

- PATCH 6: adds package/tinifier, which is one package that requires Go vendoring. I don't particularly need this package, it was just added because the other Go packages we have in Buildroot today don't need vendoring.

- PATCH 7: introduces a cargo-package infrastructure, for both target and host packages

- PATCH 8: documents the added cargo-package infrastructure

- PATCH 9 and 10: converts ripgrep and sentry-cli to the cargo-package and host-cargo-package infrastructures respecitively

- PATCH 11: implements the Cargo vendoring itself. Note that we need to bump the ripgrep and sentry-cli package at the same time, as with the vendoring their tarball contents change and therefore their hash changes

- PATCH 12: rewrite the cargo-package documentation about dependency management

Changes since v1:

- Add -modcacherw when calling Go to get files read/write instead of read-only, to make sure they can be deleted by a "make clean". Was reported by Ryan Barnett.

- Change the Cargo infrastructure to "cd" into the build directory before calling Cargo, instead of calling Cargo with the full path to Cargo.toml as this was causing issues. Was reported by Ryan Barnett, investigated and fixed by Patrick Havelange.

- Call "go mod init" in the go-post-process script if go.mod doesn't exist. Was reported by Christian Stewart, who also provided the solution.

- Add a license warning about Go/Cargo packages, as they might have vendored dependencies that are not taken into account by Buildroot licensing details. Was reported by Ryan Barnett, and Yann E. Morin suggested this simple solution.

Best regards,

Thomas

Patrick Havelange (3):
package/pkg-cargo.mk: introduce the cargo package infrastructure docs/manual/cargo: document the cargo-package infrastructure package/ripgrep: convert to cargo infrastructure

Thomas Petazzoni (9):
support/download/dl-wrapper: add concept of download post-processing package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable package/pkg-download.mk: add <pkg>_DL_ENV variable support/download/post-process-helpers: add helper function for post
process scripts
support/download/go-post-process: implement Go vendoring support package/tinifier: new package
package/sentry-cli: convert to host-cargo-package infrastructure support/download/cargo-post-process, package/pkg-cargo.mk: enable
vendoring for Cargo packages
docs/manual/adding-packages-cargo.txt: rewrite explanation about
dependency management

 DEVELOPERS                            |   1 +
 docs/manual/adding-packages-cargo.txt | 106 ++++++++----------
 package/Config.in                     |   1 +
 package/Makefile.in                   |   1 +
 package/pkg-cargo.mk                  | 149 ++++++++++++++++++++++++++
 package/pkg-download.mk               |   5 +-
 package/pkg-golang.mk                 |  20 ++--
 package/ripgrep/ripgrep.hash          |   2 +-
 package/ripgrep/ripgrep.mk            |  30 +-----
 package/sentry-cli/sentry-cli.hash    |   2 +-
 package/sentry-cli/sentry-cli.mk      |  24 +----
 package/tinifier/Config.in            |  10 ++
 package/tinifier/tinifier.hash        |   3 +
 package/tinifier/tinifier.mk          |  13 +++
 support/download/cargo-post-process   |  38 +++++++
 support/download/dl-wrapper           |   9 +-
 support/download/go-post-process      |  35 ++++++
 support/download/post-process-helpers |  30 ++++++
 18 files changed, 358 insertions(+), 121 deletions(-)
 create mode 100644 package/pkg-cargo.mk
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
 create mode 100755 support/download/cargo-post-process
 create mode 100755 support/download/go-post-process
 create mode 100644 support/download/post-process-helpers
Signature · 1 line
2.29.2
#1 lore raw

PATCH v2 1/12 support/download/dl-wrapper: add concept of download post-processing

In order to support package managers such as Cargo (Rust) or Go, we want to run some custom logic after the main download, but before packing the tarball and checking the hash.

To implement this, this commit introduces a concept of download
post-processing: if -p <something> is passed to the dl-wrapper, then support/download/<something>-post-process will be called.

---
 support/download/dl-wrapper | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
support/download/dl-wrapper +8 -1
index 3315bd410e..2d74554213 100755
@@ -25,7 +25,7 @@ main() {
     local -a uris
 
     # Parse our options; anything after '--' is for the backend
-    while getopts ":c:d:D:o:n:N:H:rf:u:q" OPT; do
+    while getopts ":c:d:D:o:n:N:H:rf:u:qp:" OPT; do
         case "${OPT}" in
         c)  cset="${OPTARG}";;
         d)  dl_dir="${OPTARG}";;
@@ -37,6 +37,7 @@ main() {
         r)  recurse="-r";;
         f)  filename="${OPTARG}";;
         u)  uris+=( "${OPTARG}" );;
+        p)  post_process="${OPTARG}";;
         q)  quiet="-q";;
         :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
         \?) error "unknown option '%s'\n" "${OPTARG}";;
@@ -135,6 +136,12 @@ main() {
             continue
         fi
 
+        if [ -n "${post_process}" ] ; then
+                ${OLDPWD}/support/download/${post_process}-post-process \
+                         -o "${tmpf}" \
+                         -n "${raw_base_name}"
+        fi
+
         # cd back to free the temp-dir, so we can remove it later
         cd "${OLDPWD}"
 
Signature · 1 line
2.29.2
#2 lore raw

PATCH v2 2/12 package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable

In reply to Thomas Petazzoni

This will allow packages to register than a download post-processing is needed. Note that this variable is intentionally not documented: it is an internal variable meant to be set by package infrastructures, not directly by packages.

---
 package/pkg-download.mk | 1 +
 1 file changed, 1 insertion(+)
package/pkg-download.mk +1 -0
index 951d2fb554..175b9d7dba 100644
@@ -108,6 +108,7 @@ define DOWNLOAD
 		-n '$($(2)_BASENAME_RAW)' \
 		-N '$($(2)_RAWNAME)' \
 		-o '$($(2)_DL_DIR)/$(notdir $(1))' \
+		$(if $($(2)_DOWNLOAD_POST_PROCESS),-p '$($(2)_DOWNLOAD_POST_PROCESS)') \
 		$(if $($(2)_GIT_SUBMODULES),-r) \
 		$(foreach uri,$(call DOWNLOAD_URIS,$(1),$(2)),-u $(uri)) \
 		$(QUIET) \
Signature · 1 line
2.29.2
#3 lore raw

PATCH v2 3/12 package/pkg-download.mk: add <pkg>_DL_ENV variable

In reply to Thomas Petazzoni

This variable can be used by package to pass extra environment variables to the download logic. It will be used for the Go/Cargo vendoring.

The <pkg>_DL_ENV variable is intentionally not documented: at this point, it is not meant to be used by packages directly, but only by package infrastructures.

---
 package/pkg-download.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-download.mk +3 -1
index 175b9d7dba..c914d016e2 100644
@@ -99,7 +99,9 @@ endif
 
 define DOWNLOAD
 	$(Q)mkdir -p $($(2)_DL_DIR)
-	$(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
+	$(Q)$(EXTRA_ENV) \
+		$($(2)_DL_ENV) \
+		flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
 		-c '$($(2)_DL_VERSION)' \
 		-d '$($(2)_DL_DIR)' \
 		-D '$(DL_DIR)' \
Signature · 1 line
2.29.2
#4 lore raw

PATCH v2 3/12 package/pkg-download.mk: add <pkg>_DL_ENV variable

Thomas, All,

On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:

Quoted 10 lines

This variable can be used by package to pass extra environment variables to the download logic. It will be used for the Go/Cargo vendoring.

The <pkg>_DL_ENV variable is intentionally not documented: at this point, it is not meant to be used by packages directly, but only by package infrastructures.

Although it's not much off your chest on this series: applied to master, with a little tweak, see below...

Quoted 9 lines
---
 package/pkg-download.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
package/pkg-download.mk +0 -0
index 175b9d7dba..c914d016e2 100644

package/pkg-download.mk endif · line 104

Quoted 7 lines
package/pkg-download.mk +3 -1
@@ -99,7 +99,9 @@ endif
 
 define DOWNLOAD
 	$(Q)mkdir -p $($(2)_DL_DIR)
-	$(Q)$(EXTRA_ENV) flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \
+	$(Q)$(EXTRA_ENV) \
+		$($(2)_DL_ENV) \
+		flock $($(2)_DL_DIR)/.lock $(DL_WRAPPER) \

I find this code to be ugly to look at. But I could not come up with something that is nice to look at. So I moved the two _ENV variables expansions to the same line, and it looks less-ugly now...

Thanks.

Regards,
Yann E. MORIN.

Quoted 10 lines
package/pkg-download.mk +0 -0
 		-c '$($(2)_DL_VERSION)' \
 		-d '$($(2)_DL_DIR)' \
 		-D '$(DL_DIR)' \
Signature · 6 lines
2.29.2

_______________________________________________
buildroot mailing list
buildroot at busybox.net
http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#5 lore raw

PATCH v2 4/12 support/download/post-process-helpers: add helper function for post process scripts

In reply to Thomas Petazzoni

download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an unpack() function and a repack() function.

---
 support/download/post-process-helpers | 30 +++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 support/download/post-process-helpers
support/download/post-process-helpers new file +30 -0
new file mode 100644
index 0000000000..bed8df2577
@@ -0,0 +1,30 @@
+
+unpack() {
+        dest="$1"
+        tarball="$2"
+
+        mkdir ${dest}
+        tar -C ${dest} --strip-components=1 -xf ${tarball}
+}
+
+repack() {
+        src="$1"
+        tarball="$2"
+
+        # Generate the archive, sort with the C locale so that it is reproducible.
+        find "$(basename ${src})" -not -type d -print0 >files.list
+        LC_ALL=C sort -z <files.list >files.list.sorted
+
+        # let's use a fixed hardcoded date to be reproducible
+        date="2020-02-06 01:02:03 +0000"
+
+        # Create GNU-format tarballs, since that's the format of the tarballs on
+        # sources.buildroot.org and used in the *.hash files
+        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
+            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
+        gzip -6 -n <new.tar >new.tar.gz
+        mv "${tarball}" "${tarball}".old
+        mv new.tar.gz "${tarball}"
+        rm "${tarball}".old
+        rm -rf ${src}
+}
Signature · 1 line
2.29.2
#6 lore raw

PATCH v2 4/12 support/download/post-process-helpers: add helper function for post process scripts

Thomas, All,

On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:

Quoted 4 lines

download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an unpack() function and a repack() function.

You forgot to explain in the commit log why we need a hard-coded, fixed date.

However, I find it ugly that we resort to such an arbitrary constant. At the very least, I'd like we take a memorable value, like 1970-01-01 00:00:00 +0000.

However, I am afraid that tar would refuse to extract files to that date, whining about an "implausibly old time stamp" when extracting such files...

But see below for a proposal...

Quoted 12 lines
---
 support/download/post-process-helpers | 30 +++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 support/download/post-process-helpers
support/download/post-process-helpers new file +0 -0
new file mode 100644
index 0000000000..bed8df2577

support/download/post-process-helpers line 4

Quoted 4 lines
support/download/post-process-helpers new file +4 -0
@@ -0,0 +1,30 @@
+
+unpack() {
+        dest="$1"
+        tarball="$2"

This is a shell script, variables are global. But since we run it with bash, we can declare such variable as local:

    local dest="${1}"
    local tarball="${2}"

support/download/post-process-helpers line 7

Quoted 2 lines
support/download/post-process-helpers new file +2 -0
+        mkdir ${dest}
+        tar -C ${dest} --strip-components=1 -xf ${tarball}
    local one_file

    one_file="$(find "${dest}" -type f |head -n1)"
    touch -r "${one_file}" "${dest}.timestamp"

support/download/post-process-helpers line 12

Quoted 5 lines
support/download/post-process-helpers new file +5 -0
+}
+
+repack() {
+        src="$1"
+        tarball="$2"

Ditto: make them 'local' (this can be fixed when applying).

Also, I suppose that the 'dest' parameter of unpack is expected to be the 'src' parameter of repack, right?

support/download/post-process-helpers line 19

Quoted 6 lines
support/download/post-process-helpers new file +6 -0
+        # Generate the archive, sort with the C locale so that it is reproducible.
+        find "$(basename ${src})" -not -type d -print0 >files.list
+        LC_ALL=C sort -z <files.list >files.list.sorted
+
+        # let's use a fixed hardcoded date to be reproducible
+        date="2020-02-06 01:02:03 +0000"

Assuming that unpack's dest and repack's src are pointing to the same directory:

        date="$(stat -c "${src}.timestamp")"

support/download/post-process-helpers line 24

Quoted 4 lines
support/download/post-process-helpers new file +4 -0
+        # Create GNU-format tarballs, since that's the format of the tarballs on
+        # sources.buildroot.org and used in the *.hash files
+        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
+            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted

This will have to be changed according to my pending series to change to the PAX format, whish is even more reproducible.

support/download/post-process-helpers line 29

Quoted 5 lines
support/download/post-process-helpers new file +5 -0
+        gzip -6 -n <new.tar >new.tar.gz
+        mv "${tarball}" "${tarball}".old
+        mv new.tar.gz "${tarball}"
+        rm "${tarball}".old
+        rm -rf ${src}

Quote all expanded variable, even "${src}". And also rm "${src}.timestamp", now.

Regards,
Yann E. MORIN.

Quoted 4 lines
support/download/post-process-helpers new file +1 -0
+}
Signature · 1 line
2.29.2
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#7 lore raw

PATCH v2 4/12 support/download/post-process-helpers: add helper function for post process scripts

Thomas, All,

A few mre tweaks I forgot about...

On 2020-12-19 18:39 +0100, Yann E. MORIN spake thusly:

Quoted 1 line

On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:

[--SNIP--]

Quoted 1 line
Quoted 1 line

+ mkdir ${dest}

Using 'mkdir -p' would ensure that parent direcotries are created as needed.

Quoted 2 lines
    one_file="$(find "${dest}" -type f |head -n1)"

Of course, the output should be sorted in the C locale, to ensure the same file is always used as reference (and hence a better name for that file):

    local ref_file
    ref_file="$(find "${dest}" -type f |LC_ALL=C sort |head -n1)"
    touch -r "${ref_file}" "${dest}.timestamp"

[--SNIP--]

Quoted 5 lines

support/download/post-process-helpers line 19

Quoted 2 lines
support/download/post-process-helpers new file +2 -0
+        # let's use a fixed hardcoded date to be reproducible
+        date="2020-02-06 01:02:03 +0000"

Assuming that unpack's dest and repack's src are pointing to the same directory:

        date="$(stat -c "${src}.timestamp")"

Meh.. I forgot the actual format:

    date="$(stat -c '%y' "${src}.timestamp")"

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#8 lore raw

PATCH v2 4/12 support/download/post-process-helpers: add helper function for post process scripts

In reply to Thomas Petazzoni

Thomas, All,

On 2020-12-19 16:35 +0100, Thomas Petazzoni spake thusly:

Quoted 4 lines

download post process scripts will often need to unpack the source code tarball, do some operation, and then repack it. In order to help with this, post-process-helpers provide an unpack() function and a repack() function.

Sorry to come back yet once more on that one, but I may have found quite an issue. Basically, your code does this:

    unpack() {
        tar -C ${dest} --strip-components=1 -xf ${tarball}
    }

    repack() {
        tar cf new.tar --null --verbatim-files-from --numeric-owner --format=gnu \
            --owner=0 --group=0 --mtime="${date}" -T files.list.sorted
        gzip -6 -n <new.tar >new.tar.gz
    }

So, it takes a tarball in whatever copression scheme and extract it, but it will alwyas create a gzip-compressed tarball.

So, how would we hanfle a case where the upstream tarball is, say, a .tar.bz2 ? For example;

FOO_SOURCE = foo=1.2.3.tar.bz2
FOO_SITE = http://foo.example.net/download
FOO_SITE_METOD = wget
$(eval $(go-package))

Besides the fact that the packager would see in DL_DIR a .tar.gz tarball different from the .tar.bz2 he coded, the most problematic issue wil be that we would have to list the .tar.gz inthe .hash file. This will be very confusing to packagers and reviewers...

Furthermore, there is another discrepancy in the post-processing scripts. For example, the go post-processor does, basically:

if tarball already vendored:
exit leaving the tarball as-is

unpack
vendor
repack as .tar.gz

So some go tarballs will be left as they are upstream, while others will be re-encoded. This is going to be quite confusing...

I think the post-processing script should always re-encode the tarball, so that at least we have a consistent behaviour across the board. This is pretty trivial to come up with, at the exepense of a bit of overhead for those tarballs that are already vendored, but the majority of packages will probably be git clones that need to be vendored, so it is a small price to pay.

The big problem that's left, is how we handle the discrepancy between the tarball listed in _SOURCE and the one listed in the .hash file...

I have to admit I do not have an easy solution for that for now...

Especially since with the pending series about changing the way we assemble tarballs from git and svn: the tar format changes, and the compression changes...

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#9 lore raw

PATCH v2 5/12 support/download/go-post-process: implement Go vendoring support

In reply to Thomas Petazzoni

This commit introduces the download post-process script support/download/go-post-process, and hooks it into the Go package infrastructure.

The -modcacherw flag is added to ensure that the Go cache is read/write, and can be deleted properly upon "make clean".

The <pkg>_LICENSE variable of golang packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

The logic to generate go.mod when not available is moved to the download post-process helper, as it must be done before "go mod vendor" is executed. Also, "go mod init" is used instead of manually crafting go.mod. This was suggested by Christian Stewart <[email protected]>. The Go module name is passed down to go-post-process using the BR_GOMOD environment variable.

---
 package/pkg-golang.mk            | 20 ++++++++++--------
 support/download/go-post-process | 35 ++++++++++++++++++++++++++++++++
 2 files changed, 46 insertions(+), 9 deletions(-)
 create mode 100755 support/download/go-post-process
package/pkg-golang.mk +11 -9
index d07242310d..00a74a4d18 100644
@@ -42,12 +42,13 @@ define inner-golang-package
 
 $(2)_BUILD_OPTS += \
 	-ldflags "$$($(2)_LDFLAGS)" \
+	-modcacherw \
 	-tags "$$($(2)_TAGS)" \
 	-trimpath \
 	-p $(PARALLEL_JOBS)
 
 # Target packages need the Go compiler on the host.
-$(2)_DEPENDENCIES += host-go
+$(2)_DOWNLOAD_DEPENDENCIES += host-go
 
 $(2)_BUILD_TARGETS ?= .
 
@@ -72,14 +73,15 @@ $(2)_SRC_SOFTWARE = $$(word 2,$$(subst /, ,$$(call notdomain,$$($(2)_SITE))))
 # If the go.mod file does not exist, one is written with this root path.
 $(2)_GOMOD ?= $$($(2)_SRC_DOMAIN)/$$($(2)_SRC_VENDOR)/$$($(2)_SRC_SOFTWARE)
 
-# Generate a go.mod file if it doesn't exist. Note: Go is configured
-# to use the "vendor" dir and not make network calls.
-define $(2)_GEN_GOMOD
-	if [ ! -f $$(@D)/go.mod ]; then \
-		printf "module $$($(2)_GOMOD)\n" > $$(@D)/go.mod; \
-	fi
-endef
-$(2)_POST_PATCH_HOOKS += $(2)_GEN_GOMOD
+$(2)_DOWNLOAD_POST_PROCESS = go
+$(2)_DL_ENV = \
+	$(HOST_GO_COMMON_ENV) \
+	GOPROXY=direct \
+	BR_GOMOD=$$($(2)_GOMOD)
+
+# Due to vendoring, it is pretty likely that not all licenses are
+# listed in <pkg>_LICENSE.
+$(2)_LICENSE += , vendored dependencies licenses probably not listed
 
 # Build step. Only define it if not already defined by the package .mk
 # file.
support/download/go-post-process new file +35 -0
new file mode 100755
index 0000000000..4ca3e9a710
@@ -0,0 +1,35 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+# Parse our options
+while getopts "n:o:" OPT; do
+        case "${OPT}" in
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/vendor" ; then
+	exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Go vendoring
+pushd ${base_name} > /dev/null
+
+# Generate go.mod if it doesn't exist
+if [ ! -f go.mod ] && [ -n "${BR_GOMOD}" ]; then
+	go mod init ${BR_GOMOD}
+fi
+
+go mod vendor -v -modcacherw
+popd > /dev/null
+
+repack ${base_name} ${output}
Signature · 1 line
2.29.2
#10 lore raw

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

Thomas, all,

support/download/go-post-process line 35

Quoted 11 lines · Thomas Petazzoni

The logic to generate go.mod when not available is moved to the download post-process helper, as it must be done before "go mod vendor" is executed. Also, "go mod init" is used instead of manually crafting go.mod. This was suggested by Christian Stewart <[email protected]>. The Go module name is passed down to go-post-process using the BR_GOMOD environment variable.

support/download/go-post-process new file +4 -0
+go mod vendor -v -modcacherw
+popd > /dev/null
+
+repack ${base_name} ${output}

This works fine in most cases, however, this causes inconsistent hashes.

On my system, the hash is correct after re-pack. But on 2 of my users machines, the hash ends up different.

So something specific to the host machine must be influencing the hash of the repacked archive.

It's either something in "go mod vendor" that's not deterministic, or maybe host Go is being used instead of the one compiled by Buildroot, or perhaps it's not possible to always get deterministic hashes this way.

Can we possibly switch to hashing the download /before/ re-packing it, given that Go verifies the hashes in go.sum automatically as part of the "go mod vendor" step?

Go mod vendor will fail if any of the dependencies are different, the checksums are included in go.sum

Thanks & best regards,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot

#11 lore raw

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

Hello Christian,

Quoted 5 lines · On Thu, 29 Jul 2021 13:17:09 -0700 Christian Stewart <[email protected]> wrote:

This works fine in most cases, however, this causes inconsistent hashes.

On my system, the hash is correct after re-pack. But on 2 of my users machines, the hash ends up different.

Quoted 7 lines

So something specific to the host machine must be influencing the hash of the repacked archive.

It's either something in "go mod vendor" that's not deterministic, or maybe host Go is being used instead of the one compiled by Buildroot, or perhaps it's not possible to always get deterministic hashes this way.

Meh :-/

Quoted 3 lines

Can we possibly switch to hashing the download /before/ re-packing it, given that Go verifies the hashes in go.sum automatically as part of the "go mod vendor" step?

This is going to be difficult, because the whole principle of this patch series is to integrate the vendoring within the download step, i.e *before* the hash is verified. For example, the hash is verified even if you're just extracting, and not downloading.

One possibility that we had discussed in the past, but that Yann didn't like was to create one tarball for the upstream code, and another for the vendored dependencies.

Based on the difficulties, I would really like to understand a little bit why the archive is not deterministic. Could you compare the contents of the download stuff between your different machines, and the contents of the tarball, and see what changes?

Quoted 2 lines

Go mod vendor will fail if any of the dependencies are different, the checksums are included in go.sum

Is the presence of go.sum mandatory?

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#12 lore raw

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

Christian, all,

Quoted 10 lines · Thomas Petazzoni

Hello Christian,

Quoted 5 lines · On Thu, 29 Jul 2021 13:17:09 -0700 Christian Stewart <[email protected]> wrote:

This works fine in most cases, however, this causes inconsistent hashes.

On my system, the hash is correct after re-pack. But on 2 of my users machines, the hash ends up different.

Quoted 9 lines
Quoted 7 lines

So something specific to the host machine must be influencing the hash of the repacked archive.

It's either something in "go mod vendor" that's not deterministic, or maybe host Go is being used instead of the one compiled by Buildroot, or perhaps it's not possible to always get deterministic hashes this way.

Meh :-/

I think Yann already calls out what may be going on in this patch https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

The tarball format will need to follow the same or similar format we use for other source packs by using the PAX format and trimming some metadata being included in the archive. GNU format tarballs are not guaranteed to be consistent across versions.

See: https://patchwork.ozlabs.org/project/buildroot/patch/bcb281853f0da8cd970446f4afed093b317dcc82.1609239666.git.yann.morin.1998@free.fr/

Signature · 9 lines
Vincent Fazio
Embedded Software Engineer - ATS
Extreme Engineering Solutions, Inc
http://www.xes-inc.com

_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot

A message that is not in the archive (sent off-list or not yet received): CA+h8R2onPMjOuvC0U6iM8QbhuAQQ9=aQ-yB-rWQkCbhpxcdiHw@mail.gmail.com

#14 lore raw

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

Christian, All,

On 2021-07-31 21:59 -0700, Christian Stewart spake thusly:

Quoted 6 lines
Quoted 2 lines · Vincent

I think Yann already calls out what may be going on in this patch
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

I think this is probably correct, the .tar.gz needs to have deterministic formatting.

Can you try usign the tar helper in support/download/helpers instead?

Quoted 4 lines

In this particular case the user was trying to build on Kali 2.

Doesn't buildroot compile host-gzip and host-tar for this? I'm wondering why it's not the same between host machines.

Since 2020.02, we are no longer building host-tar, unless for very old versions (i.e. before 1.27), thanks to some tar trickery vy Vincent. See:

cbe95b1a455b support/download: add helper to generate a reproducible archive

We also only build host-gzip if the host gzip is pigz (or missing).

Quoted 12 lines
Quoted 5 lines · Thomas Petazzoni
Quoted 1 line

This works fine in most cases, however, this causes inconsistent hashes.

Based on the difficulties, I would really like to understand a little bit why the archive is not deterministic. Could you compare the contents of the download stuff between your different machines, and the contents of the tarball, and see what changes?

The two versions are attached. The actual contents of the two are identical. But for some reason the formatting of embiggen-disk-bad.tar.gz is different:

Quoted 5 lines

00000030: 8232 63de 4282 92dd acff fb79 6740 eae2

Starting at "2dd acff" - in "embiggen-disk-good" it's

00000030: 8232 63de 4282 96dd acff fb79 6740 eae6

So, I zcat both archives, and the two tar are identical!

    $ sha1sum *.tar
    dbb8338ab4acaff1e232c67d3b46602d61114b53  embiggen-disk-bad.tar
    dbb8338ab4acaff1e232c67d3b46602d61114b53  embiggen-disk-good.tar

So I also hashed the archives you provided, and they too are identical:

    $ sha1sum *.tar.gz
    bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-bad.tar.gz
    bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-good.tar.gz

Did you mess up when sending the archives?

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot

A message that is not in the archive (sent off-list or not yet received): CA+h8R2pLN_aYiQ1vp+rTMUsQAcGT88fsAiCC-i9uJpK1y0r4rw@mail.gmail.com

#16 lore raw

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

Christian, All,

On 2021-08-01 00:24 -0700, Christian Stewart spake thusly:

Quoted 8 lines · Yann E. MORIN
Quoted 5 lines

So I also hashed the archives you provided, and they too are identical:

    $ sha1sum *.tar.gz
    bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-bad.tar.gz
    bd887a60c4ca60e55d062067132ccab7d85e7d95  embiggen-disk-good.tar.gz
Did you mess up when sending the archives?

Not sure how that got mixed up, attached is the correct embiggen-disk-bad.tar.gz:

So, what are the differences?

First, looking at the compressed archive is useless. We really need to look at the uncompressed archives to see what is going on.

    $ gunzip embiggen-disk-good.tar.gz embiggen-disk-bad.tar.gz
    $ hxedump -Cv embiggen-disk-good.tar >embiggen-disk-good.tar.hex
    $ hxedump -Cv embiggen-disk-bad.tar >embiggen-disk-bad.tar.hex
    $ diff -du embiggen-disk-good.tar.hex embiggen-disk-bad.tar.hex >embiggen-disk.diff
    $ diffstat <embiggen-disk.diff
     embiggen-disk-bad.tar.hex |   52    26    26     0 +++++++++++++-------------
     1 file changed, 26 insertions(+), 26 deletions(-)

So, there aren't so many differences, in the end. What's about them, then? Let's see the first hunk (the others are very similar):

    @@ -4,10 +4,10 @@
     00000030  31 61 32 33 30 38 2f 43  4f 4e 54 52 49 42 55 54 |1a2308/CONTRIBUT|
     00000040  49 4e 47 2e 6d 64 00 00  00 00 00 00 00 00 00 00 |ING.md..........|
     00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
    -00000060  00 00 00 00 30 30 30 30  36 34 34 00 30 30 30 30 |....0000644.0000|
    +00000060  00 00 00 00 30 30 30 30  36 36 34 00 30 30 30 30 |....0000664.0000|
     00000070  30 30 30 00 30 30 30 30  30 30 30 00 30 30 30 30 |000.0000000.0000|
     00000080  30 30 30 31 33 34 37 00  31 33 36 31 36 36 36 32 |0001347.13616662|
    -00000090  36 31 33 00 30 32 30 34  30 33 00 20 30 00 00 00 |613.020403. 0...|
    +00000090  36 31 33 00 30 32 30 34  30 35 00 20 30 00 00 00 |613.020405. 0...|
     000000a0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
     000000b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
     000000c0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|

So, the first obvious delta: the access mode. In the 'good' archive, it is 0644 (rw-r--r--), while in the 'bad' archive, it is 0664 (rw-rw-r--).

But Buildroot is supposed to enforce the mode, see Makefile:

    70: REQ_UMASK = 0022
    ...
    77: ifneq ($(shell umask):$(CURDIR):$(O),$(REQ_UMASK):$(CANONICAL_CURDIR):$(CANONICAL_O))
    ...
    83: _all:
    84:     @umask $(REQ_UMASK) && \
    85:         $(MAKE) -C $(CANONICAL_CURDIR) --no-print-directory \
    86:             $(MAKECMDGOALS) $(EXTRAMAKEARGS)

So, how comes that the two have different modes?

The second change is a bit more curious. As per the GNU tar manual [0]:
offset 0x94 (148) is 'char chksum[8]'.

So we should expect an 8-byte field (still from [0]):

The chksum field represents the simple sum of all bytes in the header block. Each 8-bit byte in the header is added to an unsigned integer, initialized to zero [...]

OK, so because the mode is different, the delta from the mode is carried over to the chksum.

If we can find why the modes are different in your two cases, we can probably solve the issue.

QED.

Still, I would prefer that we do use the tarball helper, because then we would have a single location where we could solve this kind of problems...

[0] https://www.gnu.org/software/tar/manual/html_node/Standard.html

Regards,
Yann E. MORIN.

Signature · 10 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
[email protected]
http://lists.busybox.net/mailman/listinfo/buildroot
#17 lore raw

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

Hi Yann, all,

Some users are still running into the issue that on their system the "go mod vendor" process is producing different .tar.gz hashes than on most other systems. Yann previously checked on this and found that the access modes are wrong in the "bad" archive:

Quoted 6 lines · Yann E. MORIN

So, what are the differences?

First, looking at the compressed archive is useless. We really need to look at the uncompressed archives to see what is going on.

    $ gunzip embiggen-disk-good.tar.gz embiggen-disk-bad.tar.gz

[snip]

Quoted 13 lines

So, the first obvious delta: the access mode. In the 'good' archive, it is 0644 (rw-r--r--), while in the 'bad' archive, it is 0664 (rw-rw-r--).

But Buildroot is supposed to enforce the mode, see Makefile:

    70: REQ_UMASK = 0022
    ...
    77: ifneq ($(shell umask):$(CURDIR):$(O),$(REQ_UMASK):$(CANONICAL_CURDIR):$(CANONICAL_O))
    ...
    83: _all:
    84:     @umask $(REQ_UMASK) && \
    85:         $(MAKE) -C $(CANONICAL_CURDIR) --no-print-directory \
    86:             $(MAKECMDGOALS) $(EXTRAMAKEARGS)

[snip]

Quoted 2 lines

If we can find why the modes are different in your two cases, we can probably solve the issue.

Does anyone know a suitable fix for this? I recall there might also have been some other places where this source of lack of determinism when making .tar.gz was causing issues.

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#18 lore raw

Re: [Buildroot] [PATCH v2 05/12] support/download/go-post-process: implement Go vendoring support

Hi all,

Quoted 19 lines · Christian Stewart

Some users are still running into the issue that on their system the "go mod vendor" process is producing different .tar.gz hashes than on most other systems. Yann previously checked on this and found that the access modes are wrong in the "bad" archive:

Quoted 5 lines

So, the first obvious delta: the access mode. In the 'good' archive, it is 0644 (rw-r--r--), while in the 'bad' archive, it is 0664 (rw-rw-r--).

But Buildroot is supposed to enforce the mode, see Makefile:

Quoted 8 lines
    70: REQ_UMASK = 0022
    ...
    77: ifneq ($(shell umask):$(CURDIR):$(O),$(REQ_UMASK):$(CANONICAL_CURDIR):$(CANONICAL_O))
    ...
    83: _all:
    84:     @umask $(REQ_UMASK) && \
    85:         $(MAKE) -C $(CANONICAL_CURDIR) --no-print-directory \
    86:             $(MAKECMDGOALS) $(EXTRAMAKEARGS)
Quoted 5 lines

[snip]

Quoted 2 lines

If we can find why the modes are different in your two cases, we can probably solve the issue.

I have adjusted the code from the "support for Go vendoring" patch, which I now see was never merged into mainline (yet),

The following now uses the "mk_tar_gz" helper, as Yann suggests to do:

(quoted diff) +0 -0

b/support/download/post-process-helpers
old mode 100644
new mode 100755
index bed8df2577..0517abcc24

support/download/post-process-helpers +3 -3
@@ -1,3 +1,5 @@
+# Source the mk_tar_gz helper.
+. "${0%/*}/helpers"
 
 unpack() {
         dest="$1"
@@ -8,23 +10,23 @@ unpack() {
 }
 
 repack() {
-        src="$1"
+        base_name="$1"
         tarball="$2"
-
-        # Generate the archive, sort with the C locale so that it is

reproducible.

-        find "$(basename ${src})" -not -type d -print0 >files.list
-        LC_ALL=C sort -z <files.list >files.list.sorted
+        wdir="$(pwd)"

# let's use a fixed hardcoded date to be reproducible date="2020-02-06 01:02:03 +0000"

- # Create GNU-format tarballs, since that's the format of the
tarballs on
- # sources.buildroot.org and used in the *.hash files
- tar cf new.tar --null --verbatim-files-from --numeric-owner
--format=gnu \
- --owner=0 --group=0 --mtime="${date}" -T files.list.sorted

-        gzip -6 -n <new.tar >new.tar.gz
+        # use the helper to create a reproducible archive
+        # note: the current working dir contains base_name.
+        mk_tar_gz "${base_name}" "${base_name}" "${date}" "${wdir}/new.tar.gz"
+
+        # replace the old tarball
         mv "${tarball}" "${tarball}".old
         mv new.tar.gz "${tarball}"
         rm "${tarball}".old
         rm -rf ${src}

}
+
+# Keep this line and the following as last lines in this file. +# vim: ft=bash

I think this might fix the issue, if so, shall I re-submit the go vendoring series with the adjustments?

Thanks,
Christian Stewart
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#19 lore raw

PATCH v2 6/12 package/tinifier: new package

In reply to Thomas Petazzoni

This is a Go package that needs vendor modules to be downloaded at build time.

---
 DEVELOPERS                     |  1 +
 package/Config.in              |  1 +
 package/tinifier/Config.in     | 10 ++++++++++
 package/tinifier/tinifier.hash |  3 +++
 package/tinifier/tinifier.mk   | 13 +++++++++++++
 5 files changed, 28 insertions(+)
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
DEVELOPERS +1 -0
index 9406f30cd5..1a7c22b718 100644
@@ -2590,6 +2590,7 @@ F:	package/rtc-tools/
 F:	package/sam-ba/
 F:	package/scons/
 F:	package/squashfs/
+F:	package/tinifier/
 F:	package/wayland/
 F:	package/weston/
 F:	support/testing/tests/boot/test_syslinux.py
package/Config.in +1 -0
index dc7139a49a..d5d8caef47 100644
@@ -298,6 +298,7 @@ comment "Graphic applications"
 	source "package/rrdtool/Config.in"
 	source "package/stellarium/Config.in"
 	source "package/tesseract-ocr/Config.in"
+	source "package/tinifier/Config.in"
 
 comment "Graphic libraries"
 	source "package/cegui/Config.in"
package/tinifier/Config.in new file +10 -0
new file mode 100644
index 0000000000..fbadfe6bd9
@@ -0,0 +1,10 @@
+config BR2_PACKAGE_TINIFIER
+	bool "tinifier"
+	depends on BR2_PACKAGE_HOST_GO_TARGET_ARCH_SUPPORTS
+	help
+	  CLI tool for images compressing
+
+	  This tool uses tinypng.com API endpoint for compressing your
+	  local jpg/png images (it supports parallel jobs).
+
+	  https://github.com/tarampampam/tinifier
package/tinifier/tinifier.hash new file +3 -0
new file mode 100644
index 0000000000..146700817b
@@ -0,0 +1,3 @@
+# Locally calculated
+sha256  707a1d9e55aab8c83b65bb10f3ec8c3bc094d77cd576b266f821d9f39133ac3c  tinifier-2.1.0.tar.gz
+sha256  791d8fd993ace44d4d83e2f4820a64d5ad3e37412f029afad46d17a9259de2b6  LICENSE
package/tinifier/tinifier.mk new file +13 -0
new file mode 100644
index 0000000000..b47d265a8e
@@ -0,0 +1,13 @@
+################################################################################
+#
+# tinifier
+#
+################################################################################
+
+TINIFIER_VERSION = 2.1.0
+TINIFIER_SITE = $(call github,tarampampam,tinifier,v$(TINIFIER_VERSION))
+TINIFIER_LICENSE = MIT
+TINIFIER_LICENSE_FILES = LICENSE
+TINIFIER_GOMOD = tinifier
+
+$(eval $(golang-package))
Signature · 1 line
2.29.2
#20 lore raw

PATCH v2 7/12 package/pkg-cargo.mk: introduce the cargo package infrastructure

In reply to Thomas Petazzoni

From: Patrick Havelange <[email protected]>

In order to be package agnostic, the install phase is now using cargo instead of install. TARGET_CONFIGURE_OPTS is now also set when running cargo in order to support cross compiling C code within cargo.

[Thomas: add support for host-cargo-package]
---
 package/Makefile.in  |   1 +
 package/pkg-cargo.mk | 140 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 141 insertions(+)
 create mode 100644 package/pkg-cargo.mk
package/Makefile.in +1 -0
index 51f5cbce4f..2af123f36d 100644
@@ -427,3 +427,4 @@ include package/pkg-waf.mk
 include package/pkg-golang.mk
 include package/pkg-meson.mk
 include package/pkg-qmake.mk
+include package/pkg-cargo.mk
package/pkg-cargo.mk new file +140 -0

140 lines not on this page · raw message

Signature · 1 line
2.29.2
#21 lore raw

PATCH v2 8/12 docs/manual/cargo: document the cargo-package infrastructure

In reply to Thomas Petazzoni

From: Patrick Havelange <[email protected]>

The Buildroot manual was already providing some details on how to integrate Cargo packages, and those details now need to be updated with a proper documentation for the cargo-package infrastructure.

[Thomas: numerous updates and extensions.]
---
 docs/manual/adding-packages-cargo.txt | 88 ++++++++++++---------------
 1 file changed, 39 insertions(+), 49 deletions(-)
docs/manual/adding-packages-cargo.txt +39 -49
index 8fcc80bcc6..c65a32f017 100644
@@ -1,7 +1,7 @@
 // -*- mode:doc; -*-
 // vim: set syntax=asciidoc:
 
-=== Integration of Cargo-based packages
+=== Infrastructure for Cargo-based packages
 
 Cargo is the package manager for the Rust programming language. It allows the
 user to build programs or libraries written in Rust, but it also downloads and
@@ -10,7 +10,7 @@ called "crates".
 
 [[cargo-package-tutorial]]
 
-==== Cargo-based package's +Config.in+ file
+==== +cargo-package+ tutorial
 
 The +Config.in+ file of Cargo-based package 'foo' should contain:
 
@@ -25,11 +25,7 @@ The +Config.in+ file of Cargo-based package 'foo' should contain:
 08: 	  http://foosoftware.org/foo/
 ---------------------------
 
-==== Cargo-based package's +.mk+ file
-
-Buildroot does not (yet) provide a dedicated package infrastructure for
-Cargo-based packages. So, we will explain how to write a +.mk+ file for such a
-package. Let's start with an example:
+And the +.mk+ file for this package should contain:
 
 ------------------------------
 01: ################################################################################
@@ -44,52 +40,48 @@ package. Let's start with an example:
 10: FOO_LICENSE = GPL-3.0+
 11: FOO_LICENSE_FILES = COPYING
 12:
-13: FOO_DEPENDENCIES = host-rustc
-14:
-15: FOO_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-16:
-17: FOO_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(FOO_CARGO_MODE)
-18:
-19: FOO_CARGO_OPTS = \
-20: 	$(if $(BR2_ENABLE_DEBUG),,--release) \
-21: 	--target=$(RUSTC_TARGET_NAME) \
-22: 	--manifest-path=$(@D)/Cargo.toml
-23:
-24: define FOO_BUILD_CMDS
-25: 	$(TARGET_MAKE_ENV) $(FOO_CARGO_ENV) \
-26: 		cargo build $(FOO_CARGO_OPTS)
-27: endef
-28:
-29: define FOO_INSTALL_TARGET_CMDS
-30: 	$(INSTALL) -D -m 0755 $(@D)/$(FOO_BIN_DIR)/foo \
-31: 		$(TARGET_DIR)/usr/bin/foo
-32: endef
-33:
-34: $(eval $(generic-package))
+13: $(eval $(cargo-package))
 --------------------------------
 
-The Makefile starts with the definition of the standard variables for package
-declaration (lines 7 to 11).
+The Makefile starts with the definition of the standard variables for
+package declaration (lines 7 to 11).
+
+As seen in line 13, it is based on the +cargo-package+
+infrastructure. Cargo will be invoked automatically by this
+infrastructure to build and install the package.
+
+It is still possible to define custom build commands or install
+commands (i.e.  with FOO_BUILD_CMDS and FOO_INSTALL_TARGET_CMDS).
+Those will then replace the commands from the cargo infrastructure.
+
+==== +cargo-package+ reference
+
+The main macros for the Cargo package infrastructure are
++cargo-package+ for target packages and +host-cargo-package+ for host
+packages.
+
+Just like the generic infrastructure, the Cargo infrastructure works
+by defining a number of variables before calling the +cargo-package+
+or +host-cargo-package+ macros.
 
-As seen in line 34, it is based on the
-xref:generic-package-tutorial[+generic-package+ infrastructure]. So, it defines
-the variables required by this particular infrastructure, where Cargo is
-invoked:
+First, all the package metadata information variables that exist in
+the generic infrastructure also exist in the Cargo infrastructure:
++FOO_VERSION+, +FOO_SOURCE+, +FOO_PATCH+, +FOO_SITE+,
++FOO_DEPENDENCIES+, +FOO_LICENSE+, +FOO_LICENSE_FILES+, etc.
 
-* +FOO_BUILD_CMDS+: Cargo is invoked to perform the build. The options required
-  to configure the cross-compilation of the package are passed via
-  +FOO_CONF_OPTS+.
+A few additional variables, specific to the Cargo infrastructure, can
+also be defined. Many of them are only useful in very specific cases,
+typical packages will therefore only use a few of them.
 
-* +FOO_INSTALL_TARGET_CMDS+: The binary executable generated is installed on
-  the target.
+* +FOO_CARGO_ENV+ can be used to pass additional variables in the
+  environment of +cargo+ invocations. It used at both build and
+  installation time
 
-In order to have Cargo available for the build, +FOO_DEPENDENCIES+ needs to
-contain +host-cargo+.
+* +FOO_CARGO_BUILD_OPTS+ can be used to pass additional options to
+  +cargo+ at build time.
 
-To sum it up, to add a new Cargo-based package, the Makefile example can be
-copied verbatim then edited to replace all occurences of +FOO+ with the
-uppercase name of the new package and update the values of the standard
-variables.
+* +FOO_CARGO_INSTALL_OPTS+ can be used to pass additional options to
+  +cargo+ at install time.
 
 ==== About Dependencies Management
 
@@ -99,9 +91,7 @@ automatically them. This step can also be performed independently, via the
 +cargo fetch+ command.
 
 Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+. As seen in the package
-Makefile example at line 15, this environment variable is set to
-+$(HOST_DIR)/share/cargo+.
+crates, whose location is given by +$CARGO_HOME+.
 
 This dependency download mechanism is not convenient when performing an offline
 build, as Cargo will fail to fetch the dependencies. In that case, it is advised
Signature · 1 line
2.29.2
#22 lore raw

PATCH v2 9/12 package/ripgrep: convert to cargo infrastructure

In reply to Thomas Petazzoni

From: Patrick Havelange <[email protected]>

---
 package/ripgrep/ripgrep.mk | 28 +---------------------------
 1 file changed, 1 insertion(+), 27 deletions(-)
package/ripgrep/ripgrep.mk +1 -27
index 9dd8d58de1..97e9e2ce5f 100644
@@ -9,30 +9,4 @@ RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
 RIPGREP_LICENSE = MIT
 RIPGREP_LICENSE_FILES = LICENSE-MIT
 
-RIPGREP_DEPENDENCIES = host-rustc
-RIPGREP_CARGO_ENV = CARGO_HOME=$(HOST_DIR)/share/cargo
-
-RIPGREP_BIN_DIR = target/$(RUSTC_TARGET_NAME)/$(RIPGREP_CARGO_BIN_SUBDIR)
-
-RIPGREP_CARGO_OPTS = \
-	--target=$(RUSTC_TARGET_NAME) \
-	--manifest-path=$(@D)/Cargo.toml
-
-ifeq ($(BR2_ENABLE_DEBUG),y)
-RIPGREP_CARGO_BIN_SUBDIR = debug
-else
-RIPGREP_CARGO_OPTS += --release
-RIPGREP_CARGO_BIN_SUBDIR = release
-endif
-
-define RIPGREP_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(RIPGREP_CARGO_ENV) \
-		cargo build $(RIPGREP_CARGO_OPTS)
-endef
-
-define RIPGREP_INSTALL_TARGET_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/$(RIPGREP_BIN_DIR)/rg \
-		$(TARGET_DIR)/usr/bin/rg
-endef
-
-$(eval $(generic-package))
+$(eval $(cargo-package))
Signature · 1 line
2.29.2
#23 lore raw

PATCH v2 10/12 package/sentry-cli: convert to host-cargo-package infrastructure

In reply to Thomas Petazzoni
---
 package/sentry-cli/sentry-cli.mk | 22 ++--------------------
 1 file changed, 2 insertions(+), 20 deletions(-)
package/sentry-cli/sentry-cli.mk +2 -20
index 7e53f16026..40ca22fdf9 100644
@@ -9,24 +9,6 @@ SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
 SENTRY_CLI_LICENSE = BSD-3-clause
 SENTRY_CLI_LICENSE_FILES = LICENSE
 
-HOST_SENTRY_CLI_DEPENDENCIES = host-rustc host-zlib
+HOST_SENTRY_CLI_DEPENDENCIES = host-zlib
 
-HOST_SENTRY_CLI_CARGO_ENV = \
-	CARGO_HOME=$(HOST_DIR)/share/cargo \
-	RUSTFLAGS="$(addprefix -C link-args=,$(HOST_LDFLAGS))"
-
-HOST_SENTRY_CLI_CARGO_OPTS = \
-	--release \
-	--manifest-path=$(@D)/Cargo.toml
-
-define HOST_SENTRY_CLI_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(HOST_SENTRY_CLI_CARGO_ENV) \
-		cargo build $(HOST_SENTRY_CLI_CARGO_OPTS)
-endef
-
-define HOST_SENTRY_CLI_INSTALL_CMDS
-	$(INSTALL) -D -m 0755 $(@D)/target/release/sentry-cli \
-		$(HOST_DIR)/bin/sentry-cli
-endef
-
-$(eval $(host-generic-package))
+$(eval $(host-cargo-package))
Signature · 1 line
2.29.2
#24 lore raw

PATCH v2 11/12 support/download/cargo-post-process, package/pkg-cargo.mk: enable vendoring for Cargo packages

In reply to Thomas Petazzoni

This commit adds support/download/cargo-post-process to perform the vendoring on Cargo packages, and enables it in package/pkg-cargo.mk.

Since it changes the contents of the tarballs for ripgrep and sentry-cli, it changes their hashes. To not have a different hash for the same version of ripgrep and sentry-cli, we bump their versions. It has to be done in the same commit as the Cargo vendoring to make the series bisectable.

The <pkg>_LICENSE variable of cargo packages is expanded with ", vendored dependencies licenses probably not listed" as currently for all packages, the licenses of the vendored dependencies are not taken into account.

---
 package/pkg-cargo.mk                | 13 ++++++++--
 package/ripgrep/ripgrep.hash        |  2 +-
 package/ripgrep/ripgrep.mk          |  2 +-
 package/sentry-cli/sentry-cli.hash  |  2 +-
 package/sentry-cli/sentry-cli.mk    |  2 +-
 support/download/cargo-post-process | 38 +++++++++++++++++++++++++++++
 6 files changed, 53 insertions(+), 6 deletions(-)
 create mode 100755 support/download/cargo-post-process
package/pkg-cargo.mk +11 -2
index 058c6756bb..371b8dd424 100644
@@ -37,10 +37,17 @@
 define inner-cargo-package
 
 # We need host-rustc to run cargo
-$(2)_DEPENDENCIES += host-rustc
+$(2)_DOWNLOAD_DEPENDENCIES += host-rustc
 
 $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
 
+$(2)_DOWNLOAD_POST_PROCESS = cargo
+$(2)_DL_ENV = CARGO_HOME=$$(HOST_DIR)/share/cargo
+
+# Due to vendoring, it is pretty likely that not all licenses are
+# listed in <pkg>_LICENSE.
+$(2)_LICENSE += , vendored dependencies licenses probably not listed
+
 # Note: in all the steps below, we "cd" into the build directory to
 # execute the "cargo" tool instead of passing $(@D)/Cargo.toml as the
 # manifest-path. Indeed while the latter seems to work, it in fact
@@ -50,7 +57,9 @@ $(2)_CARGO_ENV += CARGO_HOME=$$(HOST_DIR)/share/cargo
 # directory, its configuration file will not be taken into account.
 #
 # Also, we pass:
-#  * --offline to prevent cargo from downloading anything
+#  * --offline to prevent cargo from downloading anything: all
+#    dependencies should have been built by the download post
+#    process logic
 #  * --locked to force cargo to use the Cargo.lock file, which ensures
 #    that a fixed set of dependency versions is used
 
package/ripgrep/ripgrep.hash +1 -1
index 0841c0185c..81ac905d3d 100644
@@ -1,3 +1,3 @@
 # Locally calculated
-sha256 7035379fce0c1e32552e8ee528b92c3d01b8d3935ea31d26c51a73287be74bb3 ripgrep-0.8.1.tar.gz
+sha256 80d7f07325f4d485c5e4baf061b0376b45a497ee7a1c540d7894057bb9ac3a59 ripgrep-12.1.1.tar.gz
 sha256 0f96a83840e146e43c0ec96a22ec1f392e0680e6c1226e6f3ba87e0740af850f LICENSE-MIT
package/ripgrep/ripgrep.mk +1 -1
index 97e9e2ce5f..d21f88a6e8 100644
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-RIPGREP_VERSION = 0.8.1
+RIPGREP_VERSION = 12.1.1
 RIPGREP_SITE = $(call github,burntsushi,ripgrep,$(RIPGREP_VERSION))
 RIPGREP_LICENSE = MIT
 RIPGREP_LICENSE_FILES = LICENSE-MIT
package/sentry-cli/sentry-cli.hash +1 -1
index 3b0733a276..63b0c812bc 100644
@@ -1,3 +1,3 @@
 # locally calculated
-sha256  5d0f7acf6a139d1c1716b9a8ff76c8bfaab09104ba663c957bb9a5dba2ffbabd  sentry-cli-1.57.0.tar.gz
+sha256  a657dd1a46e3de044deff4c311b7276c5a9409582e707e6e6a78b0cf712591c4  sentry-cli-1.59.0.tar.gz
 sha256  9503def7b54ceb6e3cd182fd59bc05d3a30d7eae481e65aaba4b495133c83c14  LICENSE
package/sentry-cli/sentry-cli.mk +1 -1
index 40ca22fdf9..58f5f1e325 100644
@@ -4,7 +4,7 @@
 #
 ################################################################################
 
-SENTRY_CLI_VERSION = 1.57.0
+SENTRY_CLI_VERSION = 1.59.0
 SENTRY_CLI_SITE = $(call github,getsentry,sentry-cli,$(SENTRY_CLI_VERSION))
 SENTRY_CLI_LICENSE = BSD-3-clause
 SENTRY_CLI_LICENSE_FILES = LICENSE
support/download/cargo-post-process new file +38 -0
new file mode 100755
index 0000000000..5081476385
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -e
+
+. $(dirname $0)/post-process-helpers
+
+while getopts "n:o:" OPT; do
+        case "${OPT}" in
+        o)  output="${OPTARG}";;
+        n)  base_name="${OPTARG}";;
+        :)  error "option '%s' expects a mandatory argument\n" "${OPTARG}";;
+        \?) error "unknown option '%s'\n" "${OPTARG}";;
+        esac
+done
+
+# Already vendored tarball, nothing to do
+if tar tf ${output} | grep -q "^[^/]*/VENDOR" ; then
+        exit 0
+fi
+
+unpack ${base_name} ${output}
+
+# Do the Cargo vendoring
+pushd ${base_name} > /dev/null
+cargo vendor --locked VENDOR
+echo $?
+# Create the local .cargo/config with vendor info
+mkdir -p .cargo/
+cat <<EOF >.cargo/config
+[source.crates-io]
+replace-with = "vendored-sources"
+
+[source.vendored-sources]
+directory = "VENDOR"
+EOF
+popd > /dev/null
+
+repack ${base_name} ${output}
Signature · 1 line
2.29.2
#25 lore raw

PATCH v2 12/12 docs/manual/adding-packages-cargo.txt: rewrite explanation about dependency management

In reply to Thomas Petazzoni

Now that we have vendoring support for Cargo packages, let's rewrite the dependency management section in a more accurate way.

We drop the part about the local cache of the registry, because +CARGO_HOME+ in Buildroot points to $(HOST_DIR)/share/cargo, which is not shared between builds nor preserved accross builds, so its effect as a cache is limited.

---
 docs/manual/adding-packages-cargo.txt | 22 ++++++++++------------
 1 file changed, 10 insertions(+), 12 deletions(-)
docs/manual/adding-packages-cargo.txt +10 -12
index c65a32f017..fb3e7d0780 100644
@@ -85,15 +85,13 @@ typical packages will therefore only use a few of them.
 
 ==== About Dependencies Management
 
-A crate can depend on other libraries from crates.io or git repositories, listed
-in its Cargo.toml file. Before starting a build, Cargo usually downloads
-automatically them. This step can also be performed independently, via the
-+cargo fetch+ command.
-
-Cargo maintains a local cache of the registry index and of git checkouts of the
-crates, whose location is given by +$CARGO_HOME+.
-
-This dependency download mechanism is not convenient when performing an offline
-build, as Cargo will fail to fetch the dependencies. In that case, it is advised
-to generate a tarball of the dependencies using the +cargo vendor+ and add it to
-+FOO_EXTRA_DOWNLOADS+.
+A crate can depend on other libraries from crates.io or git
+repositories, listed in its Cargo.toml file. Buildroot automatically
+takes care of downloading such dependencies as part of the download
+step of packages that use the +cargo-package+ infrastructure. Such
+dependencies are then kept together with the package source code in
+the tarball cached in Buildroot's +DL_DIR+, and therefore the hash of
+the package's tarball includes such dependencies.
+
+This mechanism ensures that any change in the dependencies will be
+detected, and allows the build to be performed completely offline.
Signature · 1 line
2.29.2