kbuild: host: use single executable for rustc -C linker
PATCH v7 1/1 kbuild: host: use single executable for rustc -C linker
rustc's -C linker= option expects a single executable path. When HOSTCC contains a wrapper (e.g. "ccache gcc"), passing -Clinker=$(HOSTCC) results in the shell splitting the value into multiple words, and rustc interprets the additional word as an input filename:
error: multiple input filenames provided ...
Generate a small wrapper script and pass it to -Clinker e.g.
``` #!/bin/sh exec ccache gcc "$@" ```
This fix should be general enough to address most if not all cases (incl. wrappers or subcommands) and avoids surprises of simpler fixes like just defaulting to gcc.
This avoids passing the user command as an environment variable as that would be more challenging to trace and debug shell expansions.
---
v6 -> v7:
- Always generate scripts/rustc-wrapper when CONFIG_RUST=y, including cases with no Rust host programs in scripts/.
- Use $(obj)/rustc-wrapper for host Rust programs, fixing builds outside scripts/, e.g. samples/rust/hostprogs/.
- remove leading exec (again) to allow leading env vars e.g. HOSTRUSTC_LD="VAR=VAL ccache gcc".
- Track, clean and ignore generated wrapper as needed.
v5 -> v6:
- Add fix to rust/Makefile as well (Yoann)
- Include script to .gitignore and make clean (Nicolas)
- Add back the outer exec to the command
v4 -> v5:
- Fix word splitting issues
- Remove unnecessary exec sh -c and simplify generated script
v3 -> v4:
- Use filechk instead of if_changed macro to regenerate script
- Remove trailing space at EOL
v2 -> v3:
- Scrap previous hacky approaches (e.g. using lastword) and go with a proper fix (Gary) which turned out not that complex to implement.
v1 -> v2:
- Rename HOSTRUSTC_LINKER to HOSTRUSTC_LD for consistency
- Introduce explicit HOSTRUSTC_LD override
- Warn when falling back due to multi-argument HOSTCC
- Error out if a user-specified HOSTRUSTC_LD is not an executable
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v5: https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/all/[email protected]/
--- Makefile | 3 ++- rust/Makefile | 8 +++++--- samples/rust/hostprogs/.gitignore | 1 + samples/rust/hostprogs/Makefile | 2 ++ scripts/.gitignore | 1 + scripts/Makefile | 3 +++ scripts/Makefile.host | 26 +++++++++++++++++++++++--- 7 files changed, 37 insertions(+), 7 deletions(-)
Makefile +2 -1
@@ -1651,7 +1651,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \ modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ compile_commands.json rust/test \ rust-project.json .vmlinux.objs .vmlinux.export.c \ - .builtin-dtbs-list .builtin-dtb.S + .builtin-dtbs-list .builtin-dtb.S \ + scripts/rustc-wrapper # Directories & files removed with 'make mrproper' MRPROPER_FILES += include/config include/generated \
rust/Makefile +5 -3
@@ -565,7 +565,7 @@ $(obj)/libsyn.rlib: $(src)/syn/lib.rs $(obj)/libquote.rlib FORCE quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@ cmd_rustc_procmacro = \ $(RUSTC_OR_CLIPPY) $(rust_common_flags) $(rustc_target_flags) \ - -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=scripts/rustc-wrapper \ -Clink-args='$(call escsq,$(KBUILD_PROCMACROLDFLAGS))' \ --emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \ --crate-type proc-macro -L$(objtree)/$(obj) \
@@ -576,12 +576,14 @@ quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@ $(obj)/$(libmacros_name): private rustc_target_flags = \ --extern proc_macro2 --extern quote --extern syn $(obj)/$(libmacros_name): $(src)/macros/lib.rs $(obj)/libproc_macro2.rlib \ - $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE + $(obj)/libquote.rlib $(obj)/libsyn.rlib \ + scripts/rustc-wrapper FORCE +$(call if_changed_dep,rustc_procmacro) $(obj)/$(libpin_init_internal_name): private rustc_target_flags = $(pin_init_internal-flags) $(obj)/$(libpin_init_internal_name): $(src)/pin-init/internal/src/lib.rs \ - $(obj)/libproc_macro2.rlib $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE + $(obj)/libproc_macro2.rlib $(obj)/libquote.rlib $(obj)/libsyn.rlib \ + scripts/rustc-wrapper FORCE +$(call if_changed_dep,rustc_procmacro) # `rustc` requires `-Zunstable-options` to use custom target specifications
samples/rust/hostprogs/.gitignore +1 -0
@@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 single +rustc-wrapper
samples/rust/hostprogs/Makefile +2 -0
@@ -3,3 +3,5 @@ hostprogs-always-y := single single-rust := y + +clean-files += rustc-wrapper
scripts/.gitignore +1 -0
@@ -6,6 +6,7 @@ /kallsyms /module.lds /recordmcount +/rustc-wrapper /rustdoc_test_builder /rustdoc_test_gen /sign-file
scripts/Makefile +3 -0
@@ -29,6 +29,9 @@ generate_rust_target-rust := y rustdoc_test_builder-rust := y rustdoc_test_gen-rust := y +always-$(CONFIG_RUST) += rustc-wrapper +clean-files += rustc-wrapper + HOSTCFLAGS_tracepoint-update.o = -I$(srctree)/tools/include HOSTCFLAGS_elf-parse.o = -I$(srctree)/tools/include HOSTCFLAGS_sorttable.o = -I$(srctree)/tools/include
scripts/Makefile.host +23 -3
@@ -87,11 +87,30 @@ hostcxx_flags = -Wp,-MMD,$(depfile) \ $(KBUILD_HOSTCXXFLAGS) $(HOST_EXTRACXXFLAGS) \ $(HOSTCXXFLAGS_$(target-stem).o) +# rustc's `-Clinker=` expects a single executable path, not a command line. +# `HOSTCC` may be a multi-word command when wrapped (e.g. "ccache gcc"), which +# would otherwise be split by the shell and mis-parsed by rustc. +# To work around this, we generate a wrapper script that forwards arguments to +# `HOSTRUSTC_LD` so that such commands can be used safely. +# +# Set `HOSTRUSTC_LD` for a different rustc linker command than `HOSTCC` +HOSTRUSTC_LD ?= $(HOSTCC) + +define filechk_rustc-wrapper + printf "%s\n" \ + '#!/bin/sh' \ + '$(call escsq,$(HOSTRUSTC_LD)) "$$@"' +endef + +$(obj)/rustc-wrapper: FORCE + $(call filechk,rustc-wrapper) + $(Q)chmod +x $@ + # `--out-dir` is required to avoid temporaries being created by `rustc` in the # current working directory, which may be not accessible in the out-of-tree # modules case. hostrust_flags = --out-dir $(dir $@) --emit=dep-info=$(depfile) \ - -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=$(obj)/rustc-wrapper \ -Clink-args='$(call escsq,$(KBUILD_HOSTLDFLAGS))' \ $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \ $(HOSTRUSTFLAGS_$(target-stem))
@@ -153,10 +172,11 @@ $(host-cxxobjs): $(obj)/%.o: $(obj)/%.cc FORCE quiet_cmd_host-rust = HOSTRUSTC $@ cmd_host-rust = \ $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $< -$(host-rust): $(obj)/%: $(src)/%.rs FORCE +$(host-rust): $(obj)/%: $(src)/%.rs $(obj)/rustc-wrapper FORCE +$(call if_changed_dep,host-rust) -targets += $(host-csingle) $(host-cmulti) $(host-cobjs) \ +targets += $(obj)/rustc-wrapper \ + $(host-csingle) $(host-cmulti) $(host-cobjs) \ $(host-cxxmulti) $(host-cxxobjs) $(host-rust) # %.lex.o <- %.lex.c <- %.l
Signature · 1 line
2.53.0
Quoted 19 lines · Mohamad Alsadhan
rustc's -C linker= option expects a single executable path. When HOSTCC contains a wrapper (e.g. "ccache gcc"), passing
-Clinker=$(HOSTCC)results in the shell splitting the value into multiple words, and rustc interprets the additional word as an input filename:error: multiple input filenames provided ...
Generate a small wrapper script and pass it to -Clinker e.g.
``` #!/bin/sh exec ccache gcc "$@" ```This fix should be general enough to address most if not all cases (incl. wrappers or subcommands) and avoids surprises of simpler fixes like just defaulting to gcc.
Quoted 6 lines
This avoids passing the user command as an environment variable as that would be more challenging to trace and debug shell expansions.
Suggested-by Yoann Congal <[email protected]>Signed-off-by Mohamad Alsadhan <[email protected]>
This does work on my ccache-enabled Yocto build, so:
Thanks!
Quoted 189 lines
---
v6 -> v7:
- Always generatescripts/rustc-wrapperwhenCONFIG_RUST=y, including cases with no Rust host programs inscripts/.
- Use$(obj)/rustc-wrapperfor host Rust programs, fixing builds outsidescripts/, e.g.samples/rust/hostprogs/.
- remove leading exec (again) to allow leading env vars e.g. HOSTRUSTC_LD="VAR=VAL ccache gcc".
- Track, clean and ignore generated wrapper as needed.v5 -> v6:
- Add fix torust/Makefileas well (Yoann)
- Include script to.gitignoreand make clean (Nicolas)
- Add back the outerexecto the commandv4 -> v5:
- Fix word splitting issues
- Remove unnecessaryexec sh -cand simplify generated scriptv3 -> v4:
- Use filechk instead ofif_changedmacro to regenerate script
- Remove trailing space at EOLv2 -> v3:
- Scrap previous hacky approaches (e.g. using lastword) and go with a proper fix (Gary) which turned out not that complex to implement.v1 -> v2:
- RenameHOSTRUSTC_LINKERtoHOSTRUSTC_LDfor consistency
- Introduce explicitHOSTRUSTC_LDoverride
- Warn when falling back due to multi-argument HOSTCC
- Error out if a user-specifiedHOSTRUSTC_LDis not an executablev1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v5: https://lore.kernel.org/all/[email protected]/
v6: https://lore.kernel.org/all/[email protected]/--- Makefile | 3 ++- rust/Makefile | 8 +++++--- samples/rust/hostprogs/.gitignore | 1 + samples/rust/hostprogs/Makefile | 2 ++ scripts/.gitignore | 1 + scripts/Makefile | 3 +++ scripts/Makefile.host | 26 +++++++++++++++++++++++--- 7 files changed, 37 insertions(+), 7 deletions(-)Makefile +2
-1@@ -1651,7 +1651,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \ modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ compile_commands.json rust/test \ rust-project.json .vmlinux.objs .vmlinux.export.c \ - .builtin-dtbs-list .builtin-dtb.S + .builtin-dtbs-list .builtin-dtb.S \ + scripts/rustc-wrapper # Directories & files removed with 'make mrproper' MRPROPER_FILES += include/config include/generated \rust/Makefile +5
-3@@ -565,7 +565,7 @@ $(obj)/libsyn.rlib: $(src)/syn/lib.rs $(obj)/libquote.rlib FORCE quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@ cmd_rustc_procmacro = \ $(RUSTC_OR_CLIPPY) $(rust_common_flags) $(rustc_target_flags) \ - -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=scripts/rustc-wrapper \ -Clink-args='$(call escsq,$(KBUILD_PROCMACROLDFLAGS))' \ --emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \ --crate-type proc-macro -L$(objtree)/$(obj) \@@ -576,12 +576,14 @@ quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@ $(obj)/$(libmacros_name): private rustc_target_flags = \ --extern proc_macro2 --extern quote --extern syn $(obj)/$(libmacros_name): $(src)/macros/lib.rs $(obj)/libproc_macro2.rlib \ - $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE + $(obj)/libquote.rlib $(obj)/libsyn.rlib \ + scripts/rustc-wrapper FORCE +$(call if_changed_dep,rustc_procmacro) $(obj)/$(libpin_init_internal_name): private rustc_target_flags = $(pin_init_internal-flags) $(obj)/$(libpin_init_internal_name): $(src)/pin-init/internal/src/lib.rs \ - $(obj)/libproc_macro2.rlib $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE + $(obj)/libproc_macro2.rlib $(obj)/libquote.rlib $(obj)/libsyn.rlib \ + scripts/rustc-wrapper FORCE +$(call if_changed_dep,rustc_procmacro) # `rustc` requires `-Zunstable-options` to use custom target specificationssamples/rust/hostprogs/.gitignore +1
-0@@ -1,3 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 single +rustc-wrappersamples/rust/hostprogs/Makefile +2
-0@@ -3,3 +3,5 @@ hostprogs-always-y := single single-rust := y + +clean-files += rustc-wrapperscripts/.gitignore +1
-0@@ -6,6 +6,7 @@ /kallsyms /module.lds /recordmcount +/rustc-wrapper /rustdoc_test_builder /rustdoc_test_gen /sign-filescripts/Makefile +3
-0@@ -29,6 +29,9 @@ generate_rust_target-rust := y rustdoc_test_builder-rust := y rustdoc_test_gen-rust := y +always-$(CONFIG_RUST) += rustc-wrapper +clean-files += rustc-wrapper + HOSTCFLAGS_tracepoint-update.o = -I$(srctree)/tools/include HOSTCFLAGS_elf-parse.o = -I$(srctree)/tools/include HOSTCFLAGS_sorttable.o = -I$(srctree)/tools/includescripts/Makefile.host +23
-3@@ -87,11 +87,30 @@ hostcxx_flags = -Wp,-MMD,$(depfile) \ $(KBUILD_HOSTCXXFLAGS) $(HOST_EXTRACXXFLAGS) \ $(HOSTCXXFLAGS_$(target-stem).o) +# rustc's `-Clinker=` expects a single executable path, not a command line. +# `HOSTCC` may be a multi-word command when wrapped (e.g. "ccache gcc"), which +# would otherwise be split by the shell and mis-parsed by rustc. +# To work around this, we generate a wrapper script that forwards arguments to +# `HOSTRUSTC_LD` so that such commands can be used safely. +# +# Set `HOSTRUSTC_LD` for a different rustc linker command than `HOSTCC` +HOSTRUSTC_LD ?= $(HOSTCC) + +define filechk_rustc-wrapper + printf "%s\n" \ + '#!/bin/sh' \ + '$(call escsq,$(HOSTRUSTC_LD)) "$$@"' +endef + +$(obj)/rustc-wrapper: FORCE + $(call filechk,rustc-wrapper) + $(Q)chmod +x $@ + # `--out-dir` is required to avoid temporaries being created by `rustc` in the # current working directory, which may be not accessible in the out-of-tree # modules case. hostrust_flags = --out-dir $(dir $@) --emit=dep-info=$(depfile) \ - -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=$(obj)/rustc-wrapper \ -Clink-args='$(call escsq,$(KBUILD_HOSTLDFLAGS))' \ $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \ $(HOSTRUSTFLAGS_$(target-stem))@@ -153,10 +172,11 @@ $(host-cxxobjs): $(obj)/%.o: $(obj)/%.cc FORCE quiet_cmd_host-rust = HOSTRUSTC $@ cmd_host-rust = \ $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $< -$(host-rust): $(obj)/%: $(src)/%.rs FORCE +$(host-rust): $(obj)/%: $(src)/%.rs $(obj)/rustc-wrapper FORCE +$(call if_changed_dep,host-rust) -targets += $(host-csingle) $(host-cmulti) $(host-cobjs) \ +targets += $(obj)/rustc-wrapper \ + $(host-csingle) $(host-cmulti) $(host-cobjs) \ $(host-cxxmulti) $(host-cxxobjs) $(host-rust) # %.lex.o <- %.lex.c <- %.l
Signature · 2 lines
Yoann Congal Smile ECS
Nathan Chancellor
maintainerrust/Makefile $(obj)/libsyn.rlib: $(src)/syn/lib.rs $(obj)/libquote.rlib FORCE · line 568
Quoted 10 lines · Mohamad Alsadhan
rust/Makefile +1
-1@@ -565,7 +565,7 @@ $(obj)/libsyn.rlib: $(src)/syn/lib.rs $(obj)/libquote.rlib FORCE quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@ cmd_rustc_procmacro = \ $(RUSTC_OR_CLIPPY) $(rust_common_flags) $(rustc_target_flags) \ - -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=scripts/rustc-wrapper \
I think this should use an explicit $(objtree)/ prefix to make it an absolute path.
scripts/Makefile.host hostcxx_flags = -Wp,-MMD,$(depfile) \ · line 113
Quoted 13 lines
scripts/Makefile.host +0
-0@@ -87,11 +87,30 @@ hostcxx_flags = -Wp,-MMD,$(depfile) \[ ... skip 22 lines ... ]
#--out-diris required to avoid temporaries being created byrustcin the# current working directory, which may be not accessible in the out-of-tree # modules case. hostrust_flags = --out-dir $(dir $@) --emit=dep-info=$(depfile) \scripts/Makefile.host +1
-1- -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=$(obj)/rustc-wrapper \
I have not been following this series too closely but it feels weird that multiple rustc-wrapper files would be generated when their contents should be the exact same. Would other host programs have to copy the changes you made to .gitignore and Makefile under samples/rust/hostprogs? What was the issue with attempting to unify these?
Signature · 2 lines
Cheers, Nathan
Thanks for your input, good points.
Quoted 6 lines · Nathan Chancellor
rust/Makefile $(obj)/libsyn.rlib: $(src)/syn/lib.rs $(obj)/libquote.rlib FORCE · line 568
Quoted 2 lines
rust/Makefile +1
-1- -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=scripts/rustc-wrapper \I think this should use an explicit $(objtree)/ prefix to make it an absolute path.
I changed the rustc-facing path to $(objtree)/scripts/rustc-wrapper in v8 for clarity. However, the make prerequisites stay as scripts/rustc-wrapper, because the rule is defined for that target name; using $(objtree)/... there breaks make dependency resolution.
Quoted 5 lines
I have not been following this series too closely but it feels weird that multiple rustc-wrapper files would be generated when their contents should be the exact same. Would other host programs have to copy the changes you made to .gitignore and
Makefileundersamples/rust/hostprogs? What was the issue with attempting to unify these?
I agree; per-directory wrappers were the wrong end state.
I have reworked v8 to use a single global wrapper under scripts/ for both rust/Makefile and generic host Rust builds.
The missing piece before was making sure that wrapper is always generated when Rust is enabled, even if no Rust hostprog in scripts/ would otherwise build it. With that fixed, the local ignore/clean changes are dropped.
v8 should follow soon. Apologies for this taking longer than needed, it was my first kbuild patch. The feedback and patience are very much appreciated.
--
Best,
mo