Building cargo with per-package directories
Hello ?ric,
As you probably know, I'm working on support for per-package directories, which will allow to do top-level parallel in a reliable way. As part of this effort, I've started an autobuilder that uses per-package directories, and one of the package that is failing a lot is host-cargo.
It fails like this:
= note: /usr/bin/ld: cannot find -lhttp_parser
collect2: error: ld returned 1 exit status
error: aborting due to previous error
error: Could not compile cargo.
The problem is that while libhttp_parser gets built properly, cargo is not passing the appropriate -L flag to cargo's per-package host/lib directory. So the linker doesn't find the libhttp_parser library, and bails out.
I've tried to look into this, but really the cargo build system is very unique and weird, and I'm having a hard time finding my way. Do you think you could help me a bit on this topic ?
To reproduce, you need a machine where libhttp_parser is not installed system-wide (so either remove it from your system, or build in a chroot/docker/VM). Then, take this branch:
https://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v7-work
And build the following defconfig:
BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_ENABLE_VFP=y
BR2_PER_PACKAGE_DIRECTORIES=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-cortex-a9-glibc-2018.05.tar.bz2" BR2_TOOLCHAIN_EXTERNAL_GCC_7=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_16=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
# BR2_TARGET_ROOTFS_TAR is not set
BR2_PACKAGE_HOST_CARGO=y
You should get this failure:
http://code.bulix.org/fxjgrv-520980?raw
Thanks in advance for your support,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Hi!
Quoted 16 lines · Thomas Petazzoni
Hello ?ric,
As you probably know, I'm working on support for per-package directories, which will allow to do top-level parallel in a reliable way. As part of this effort, I've started an autobuilder that uses per-package directories, and one of the package that is failing a lot is host-cargo.
It fails like this:
= note:
/usr/bin/ld: cannot find -lhttp_parser
collect2: error: ld returned 1 exit statuserror: aborting due to previous error
error: Could not compilecargo.
Quoted 8 lines
The problem is that while
libhttp_parsergets built properly, cargo is not passing the appropriate -L flag to cargo's per-package host/lib directory. So the linker doesn't find thelibhttp_parserlibrary, and bails out.I've tried to look into this, but really the cargo build system is very unique and weird, and I'm having a hard time finding my way. Do you think you could help me a bit on this topic ?
Of course!
Quoted 27 lines
To reproduce, you need a machine where
libhttp_parseris not installed system-wide (so either remove it from your system, or build in a chroot/docker/VM). Then, take this branch:https://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v7-work
And build the following defconfig:
BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_ENABLE_VFP=y
BR2_PER_PACKAGE_DIRECTORIES=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-cortex-a9-glibc-2018.05.tar.bz2" BR2_TOOLCHAIN_EXTERNAL_GCC_7=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_16=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
#BR2_PACKAGE_BUSYBOXis not set
#BR2_TARGET_ROOTFS_TARis not set
BR2_PACKAGE_HOST_CARGO=y
Quoted 5 lines
You should get this failure:
http://code.bulix.org/fxjgrv-520980?raw
Thanks in advance for your support,
I added your clone as remote, checked out ppsh-v7-work branch and launched a Debian Stretch chroot. I'll keep you posted on the result.
Regards,
--
ELB
Quoted 56 lines · Eric Le Bihan
Hi!
Quoted 16 lines · Thomas Petazzoni
Hello ?ric,
As you probably know, I'm working on support for per-package directories, which will allow to do top-level parallel in a reliable way. As part of this effort, I've started an autobuilder that uses per-package directories, and one of the package that is failing a lot is host-cargo.
It fails like this:
= note:
/usr/bin/ld: cannot find -lhttp_parser
collect2: error: ld returned 1 exit statuserror: aborting due to previous error
error: Could not compilecargo.Quoted 8 lines
The problem is that while
libhttp_parsergets built properly, cargo is not passing the appropriate -L flag to cargo's per-package host/lib directory. So the linker doesn't find thelibhttp_parserlibrary, and bails out.I've tried to look into this, but really the cargo build system is very unique and weird, and I'm having a hard time finding my way. Do you think you could help me a bit on this topic ?
Of course!
Quoted 27 lines
To reproduce, you need a machine where
libhttp_parseris not installed system-wide (so either remove it from your system, or build in a chroot/docker/VM). Then, take this branch:https://git.bootlin.com/users/thomas-petazzoni/buildroot/log/?h=ppsh-v7-work
And build the following defconfig:
BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_ENABLE_VFP=y
BR2_PER_PACKAGE_DIRECTORIES=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM=y
BR2_TOOLCHAIN_EXTERNAL_DOWNLOAD=y BR2_TOOLCHAIN_EXTERNAL_URL="http://autobuild.buildroot.org/toolchains/tarballs/br-arm-cortex-a9-glibc-2018.05.tar.bz2" BR2_TOOLCHAIN_EXTERNAL_GCC_7=y
BR2_TOOLCHAIN_EXTERNAL_HEADERS_4_16=y
BR2_TOOLCHAIN_EXTERNAL_CUSTOM_GLIBC=y
BR2_TOOLCHAIN_EXTERNAL_CXX=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
#BR2_PACKAGE_BUSYBOXis not set
#BR2_TARGET_ROOTFS_TARis not set
BR2_PACKAGE_HOST_CARGO=y
Quoted 8 lines
Quoted 5 lines
You should get this failure:
http://code.bulix.org/fxjgrv-520980?raw
Thanks in advance for your support,
I added your clone as remote, checked out ppsh-v7-work branch and launched a Debian Stretch chroot. I'll keep you posted on the result.
The build ended as expected.
```
= note: /usr/bin/ld: cannot find -lhttp_parser
collect2: error: ld returned 1 exit status
```The library libhttp_parser.so is required by one of the dependencies of
cargo: libgit2-sys, which provides the Rust interface to libgit2.
There is trick in vendor/libgit2-sys/build.rs to handle libhttp_parser.so: if the pkgconfig file of libgit2 contains -lhttp_parser, then pass it to rustc via "cargo:rustc-link-lib=http_parser".
See "Outputs of the Build Script" in Cargo Book [1] for details.
We want to use per-package/host-cargo/host/lib/libhttp_parser.so. Adding "-Clink-arg=-L$(HOST_DIR)/lib" to the RUSTFLAGS environment variable solves the issue on my system:
``` diff --git a/package/cargo/cargo.mk b/package/cargo/cargo.mk index 1b1053ba41..891901578e 100644 --- a/package/cargo/cargo.mk +++ b/package/cargo/cargo.mk @@ -70,7 +70,7 @@ HOST_CARGO_SNAP_OPTS = \ $(if $(VERBOSE),--verbose) HOST_CARGO_ENV = \ - RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib" \ + RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib -Clink-arg=-L$(HOST_DIR)/lib" \ CARGO_HOME=$(HOST_CARGO_HOME) define HOST_CARGO_BUILD_CMDS ```
[1] https://doc.rust-lang.org/cargo/reference/build-scripts.html#outputs-of-the-build-script
Regards,
--
ELB
Hello Eric,
Quoted 26 lines · Eric Le Bihan
The build ended as expected.
``` = note: /usr/bin/ld: cannot find -lhttp_parser collect2: error: ld returned 1 exit status ```The library libhttp_parser.so is required by one of the dependencies of
cargo: libgit2-sys, which provides the Rust interface to libgit2.There is trick in
vendor/libgit2-sys/build.rsto handle libhttp_parser.so: if the pkgconfig file of libgit2 contains -lhttp_parser, then pass it to rustc via "cargo:rustc-link-lib=http_parser".See "Outputs of the Build Script" in Cargo Book [1] for details.
We want to use per-package/host-cargo/host/lib/libhttp_parser.so. Adding "-Clink-arg=-L$(HOST_DIR)/lib" to the RUSTFLAGS environment variable solves the issue on my system:
```
package/cargo/cargo.mk +0
-0
package/cargo/cargo.mk HOST_CARGO_SNAP_OPTS = \ · line 74
Quoted 6 lines
package/cargo/cargo.mk +1
-1@@ -70,7 +70,7 @@ HOST_CARGO_SNAP_OPTS = \ $(if $(VERBOSE),--verbose) HOST_CARGO_ENV = \ - RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib" \ + RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib -Clink-arg=-L$(HOST_DIR)/lib" \ CARGO_HOME=$(HOST_CARGO_HOME)
Thanks a lot for this investigation and solution. I will definitely have a look and see how it goes. Many, many thanks!
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Hello Eric,
Quoted 15 lines · Eric Le Bihan
package/cargo/cargo.mk +1
-1@@ -70,7 +70,7 @@ HOST_CARGO_SNAP_OPTS = \ $(if $(VERBOSE),--verbose) HOST_CARGO_ENV = \ - RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib" \ + RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib -Clink-arg=-L$(HOST_DIR)/lib" \ CARGO_HOME=$(HOST_CARGO_HOME) define HOST_CARGO_BUILD_CMDS```
Quoted 1 line
[1] https://doc.rust-lang.org/cargo/reference/build-scripts.html#outputs-of-the-build-script
Thanks a lot, I was finally able to test this, and confirm it works fine.
I propose to change it to:
RUSTFLAGS="$(addprefix -Clink-arg=,$(HOST_LDFLAGS))"
so that if we ever change HOST_LDFLAGS in the future, it will continue to work. What do you think ?
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 24 lines · Thomas Petazzoni
Hello Eric,
Quoted 15 lines · Eric Le Bihan
package/cargo/cargo.mk +1
-1@@ -70,7 +70,7 @@ HOST_CARGO_SNAP_OPTS = \ $(if $(VERBOSE),--verbose) HOST_CARGO_ENV = \ - RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib" \ + RUSTFLAGS="-Clink-arg=-Wl,-rpath,$(HOST_DIR)/lib -Clink-arg=-L$(HOST_DIR)/lib" \ CARGO_HOME=$(HOST_CARGO_HOME) define HOST_CARGO_BUILD_CMDS```
Quoted 1 line
[1] https://doc.rust-lang.org/cargo/reference/build-scripts.html#outputs-of-the-build-script
Thanks a lot, I was finally able to test this, and confirm it works fine.
Quoted 6 lines
I propose to change it to:
RUSTFLAGS="$(addprefix -Clink-arg=,$(HOST_LDFLAGS))"so that if we ever change
HOST_LDFLAGSin the future, it will continue to work. What do you think ?
I tested this change without issues, so I will post a patch to include it in the cargo package.
Regards,
--
ELB
Hello Eric,
Quoted 5 lines · Eric Le Bihan
Quoted 2 lines
so that if we ever change
HOST_LDFLAGSin the future, it will continue to work. What do you think ?I tested this change without issues, so I will post a patch to include it in the cargo package.
Thanks. In fact I already had a patch at
https://git.bootlin.com/users/thomas-petazzoni/buildroot/commit/?h=ppsh-v7-work&id=55c8e671d5130b22c72cedd194659c3dde5a6923,
but I will happily take an alternate patch if you have one.
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Quoted 7 lines · Thomas Petazzoni
Hello Eric,
Quoted 3 lines · Eric Le Bihan
Quoted 2 lines
so that if we ever change
HOST_LDFLAGSin the future, it will continue to work. What do you think ?
Quoted 6 lines
Quoted 2 lines
I tested this change without issues, so I will post a patch to include it in the cargo package.
Thanks. In fact I already had a patch at
https://git.bootlin.com/users/thomas-petazzoni/buildroot/commit/?h=ppsh-v7-work&id=55c8e671d5130b22c72cedd194659c3dde5a6923,
but I will happily take an alternate patch if you have one.
I do not have one yet, so your patch is fine.
Regards,
--
ELB