Skip to content

Support for Cargo and Go vendoring

Thomas Petazzoni buildroot 1 version Applied

Status from this message: Applied to master, thanks.

Get series with b4
b4 am [email protected]

Versions newest first

  1. v3 Latest11 patches 26 repliesApplied

Patches in v3 Applied

#Subject and filesSizeRepliesApplied
1/11support/download/dl-wrapper: add concept of download post-processingsupport/download/dl-wrapper+8 -11 reply
2/11package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variablepackage/pkg-download.mk+1 -01 reply
3/11support/download/post-process-helpers: add helper function for post process scriptssupport/download/helpers+22 -01 reply
4/11package/pkg-golang.mk: implement Go vendoring supportpackage/pkg-golang.mk support/download/go-post-process+47 -11 reply
5/11package/pkg-cargo.mk: introduce the cargo package infrastructurepackage/Makefile.in package/pkg-cargo.mk support/download/cargo-post-process+192 -013 replies
6/11docs/manual/cargo: document the cargo-package infrastructuredocs/manual/adding-packages-cargo.txt+47 -590 replies
7/11package/ripgrep: convert to cargo infrastructurepackage/ripgrep/ripgrep.hash package/ripgrep/ripgrep.mk+7 -320 replies
8/11package/sentry-cli: re-add packageDEVELOPERS package/Config.in.host package/sentry-cli/Config.in.host +2 more+29 -04 replies
9/11package/tinifier: new packageDEVELOPERS package/Config.in package/tinifier/Config.in +2 more+28 -01 reply
10/11package/embiggen-disk: new packageDEVELOPERS package/Config.in package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch +3 more+81 -01 reply
11/11package/gocryptfs: new packageDEVELOPERS package/Config.in package/gocryptfs/Config.in +2 more+36 -00 replies

Reviews trailers per patch

1 of 11 patches has a Reviewed-by or Acked-by.

PatchChristian Stewart
1/11
2/11
3/11
4/11
5/11
6/11
7/11
8/11
9/11R-b
10/11
11/11

Trailers given in replies are filled; those carried in the patches from earlier versions are outlined with dashes.

Read the v3 thread

Cover letter [Buildroot] [PATCH v3 00/11] Support for Cargo and Go vendoring

Hello,

Believe it or not, but here is a new version of my patch series adding support for Go vendoring, as well as a Cargo package infrastructure that also supports vendoring.

This series builds on top of the work from Patrick Havelange <[email protected]> who worked on the Cargo integration, but extends it to also cover the case of Go, showing that we have a solution solving both of these language-specific package managers. For the Go vendoring, I've also used the indication provided by Christian Stewart <[email protected]>. I've also integrated some changes that Christian did in his series at:

https://patchwork.ozlabs.org/project/buildroot/list/?series=266310

Overall, the solution consists in a concept of "download post-process helper" is introduced in the download infrastructure. These are shell scripts that can be invoked right after the download is done, but before the tarball is hash-checked and stored in DL_DIR. The idea is that such "download post-process helpers" can run the language-specific vendoring logic to fetch the package dependencies. Thanks to this, the tarball in DL_DIR for a given package not only contains the package source code itself, but also the source code of its Cargo or Go dependencies. The tarball hash covers the entire source code, the complete tarball is cached in DL_DIR, in the primary site and backup site, and the build can fully be done offline.

Such "download post-process helpers" are registered by means of the <pkg>_DOWNLOAD_POST_PROCESS variable by the appropriate package infrastructure.

This new series is available at:

https://github.com/tpetazzoni/buildroot/tree/pkg-vendoring

A useful defconfig to test this series is:

BR2_aarch64=y
BR2_TOOLCHAIN_EXTERNAL=y BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_AARCH64_GLIBC_STABLE=y BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set BR2_PACKAGE_DELVE=y
BR2_PACKAGE_RIPGREP=y
BR2_PACKAGE_GOCRYPTFS=y BR2_PACKAGE_TINIFIER=y
BR2_PACKAGE_FLANNEL=y BR2_PACKAGE_BALENA_ENGINE=y BR2_PACKAGE_DOCKER_CLI=y BR2_PACKAGE_DOCKER_ENGINE=y BR2_PACKAGE_EMBIGGEN_DISK=y BR2_PACKAGE_MENDER=y
# BR2_TARGET_ROOTFS_TAR is not set BR2_PACKAGE_HOST_MENDER_ARTIFACT=y BR2_PACKAGE_HOST_SENTRY_CLI=y

It is useful because it builds:

- All currently existing packages that use the golang-package or host-golang-package infrastructures. Those existing packages did not need vendoring support.

- Builds the new tinifier, embiggen-disk and gocryptfs Go packages, which need vendoring.

- Builds ripgrep, which is a target Cargo package.

- Builds host-entry-cli, which is a host Cargo package.

Here are the main changes compared to v2 (posted December 2020):

- Use of the mk_tar_gz shell function to re-create the tarball after vendoring. This allows to benefit from the reproducible magic implemented by mk_tar_gz.

- post-process download helpers moved to support/download/helpers instead of having their own files.

- Usage of a proper timestamp for the re-created tarballs instead of a fixed hardcoded timestamp.

- Keep the creation of go.mod in golang-package, as it is needed for Go packages that are already vendored in their upstream tarball. The creation of go.mod in the Go post-process download script, used for vendoring, now uses the same method to create a minimal go.mod file.

- The Cargo infra is now added in one patch, with vendoring support, instead of adding it without vendoring support and add it after the fact.

- For the same reason, the Cargo infrastructure documentation now comes through a single patch.

- The ripgrep changes are modified to account for the ripgrep version bump.

- The tinifier package (a Go package that needs vendoring) was version bumped.

- The packages embiggen-disk and gocryptfs, which were submitted by Christian, are now part of this series.

- host-sentry-cli, which had been removed previously due to the lack of Cargo vendoring support, is now re-added by this series.

- Small formatting and coding style improvements.

Best regards,

Thomas Petazzoni

Christian Stewart via buildroot (2):
package/embiggen-disk: new package
package/gocryptfs: new package

Patrick Havelange (3):
package/pkg-cargo.mk: introduce the cargo package infrastructure docs/manual/cargo: document the cargo-package infrastructure package/ripgrep: convert to cargo infrastructure

Thomas Petazzoni (6):
support/download/dl-wrapper: add concept of download post-processing package/pkg-download.mk: add <pkg>_DOWNLOAD_POST_PROCESS variable support/download/post-process-helpers: add helper function for post
process scripts
package/pkg-golang.mk: implement Go vendoring support package/sentry-cli: re-add package
package/tinifier: new package

 DEVELOPERS                                    |   4 +
 docs/manual/adding-packages-cargo.txt         | 106 ++++++------
 package/Config.in                             |   3 +
 package/Config.in.host                        |   1 +
 package/Makefile.in                           |   1 +
 ...-Fix-resizing-of-dev-mmcblk0pN-paths.patch |  45 ++++++
 package/embiggen-disk/Config.in               |  19 +++
 package/embiggen-disk/embiggen-disk.hash      |   3 +
 package/embiggen-disk/embiggen-disk.mk        |  12 ++
 package/gocryptfs/Config.in                   |  12 ++
 package/gocryptfs/gocryptfs.hash              |   3 +
 package/gocryptfs/gocryptfs.mk                |  19 +++
 package/pkg-cargo.mk                          | 153 ++++++++++++++++++
 package/pkg-download.mk                       |   1 +
 package/pkg-golang.mk                         |  13 +-
 package/ripgrep/ripgrep.hash                  |   2 +-
 package/ripgrep/ripgrep.mk                    |  37 +----
 package/sentry-cli/Config.in.host             |  10 ++
 package/sentry-cli/sentry-cli.hash            |   3 +
 package/sentry-cli/sentry-cli.mk              |  14 ++
 package/tinifier/Config.in                    |  10 ++
 package/tinifier/tinifier.hash                |   3 +
 package/tinifier/tinifier.mk                  |  13 ++
 support/download/cargo-post-process           |  38 +++++
 support/download/dl-wrapper                   |   9 +-
 support/download/go-post-process              |  35 ++++
 support/download/helpers                      |  22 +++
 27 files changed, 498 insertions(+), 93 deletions(-)
 create mode 100644 package/embiggen-disk/0001-Fix-resizing-of-dev-mmcblk0pN-paths.patch
 create mode 100644 package/embiggen-disk/Config.in
 create mode 100644 package/embiggen-disk/embiggen-disk.hash
 create mode 100644 package/embiggen-disk/embiggen-disk.mk
 create mode 100644 package/gocryptfs/Config.in
 create mode 100644 package/gocryptfs/gocryptfs.hash
 create mode 100644 package/gocryptfs/gocryptfs.mk
 create mode 100644 package/pkg-cargo.mk
 create mode 100644 package/sentry-cli/Config.in.host
 create mode 100644 package/sentry-cli/sentry-cli.hash
 create mode 100644 package/sentry-cli/sentry-cli.mk
 create mode 100644 package/tinifier/Config.in
 create mode 100644 package/tinifier/tinifier.hash
 create mode 100644 package/tinifier/tinifier.mk
 create mode 100755 support/download/cargo-post-process
 create mode 100755 support/download/go-post-process
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot