kbuild: host: use single executable for rustc -C linker
PATCH v6 1/1 kbuild: host: use single executable for rustc -C linker
rustc's -C linker= option expects a single executable path. When HOSTCC contains a wrapper (e.g. "ccache gcc"), passing -Clinker=$(HOSTCC) results in the shell splitting the value into multiple words, and rustc interprets the additional word as an input filename:
error: multiple input filenames provided ...
Generate a small wrapper script and pass it to -Clinker e.g.
``` #!/bin/sh exec ccache gcc "$@" ```
This fix should be general enough to address most if not all cases (incl. wrappers or subcommands) and avoids surprises of simpler fixes like just defaulting to gcc.
This avoids passing the user command as an environment variable as that would be more challenging to trace and debug shell expansions.
---
v5 -> v6:
- Add fix to rust/Makefile as well (Yoann)
- Include script to .gitignore and make clean (Nicolas)
- Add back the outer exec to the command
v4 -> v5:
- Fix word splitting issues
- Remove unnecessary exec sh -c and simplify generated script
v3 -> v4:
- Use filechk instead of if_changed macro to regenerate script
- Remove trailing space at EOL
v2 -> v3:
- Scrap previous hacky approaches (e.g. using lastword) and go with a proper fix (Gary) which turned out not that complex to implement.
v1 -> v2:
- Rename HOSTRUSTC_LINKER to HOSTRUSTC_LD for consistency
- Introduce explicit HOSTRUSTC_LD override
- Warn when falling back due to multi-argument HOSTCC
- Error out if a user-specified HOSTRUSTC_LD is not an executable
v1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v5: https://lore.kernel.org/all/[email protected]/
--- Makefile | 3 ++- rust/Makefile | 8 +++++--- scripts/.gitignore | 1 + scripts/Makefile.host | 23 +++++++++++++++++++++-- 4 files changed, 29 insertions(+), 6 deletions(-)
Makefile +2 -1
@@ -1651,7 +1651,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \ modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ compile_commands.json rust/test \ rust-project.json .vmlinux.objs .vmlinux.export.c \ - .builtin-dtbs-list .builtin-dtb.S + .builtin-dtbs-list .builtin-dtb.S \ + scripts/rustc-wrapper # Directories & files removed with 'make mrproper' MRPROPER_FILES += include/config include/generated \
rust/Makefile +5 -3
@@ -565,7 +565,7 @@ $(obj)/libsyn.rlib: $(src)/syn/lib.rs $(obj)/libquote.rlib FORCE quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@ cmd_rustc_procmacro = \ $(RUSTC_OR_CLIPPY) $(rust_common_flags) $(rustc_target_flags) \ - -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=scripts/rustc-wrapper \ -Clink-args='$(call escsq,$(KBUILD_PROCMACROLDFLAGS))' \ --emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \ --crate-type proc-macro -L$(objtree)/$(obj) \
@@ -576,12 +576,14 @@ quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@ $(obj)/$(libmacros_name): private rustc_target_flags = \ --extern proc_macro2 --extern quote --extern syn $(obj)/$(libmacros_name): $(src)/macros/lib.rs $(obj)/libproc_macro2.rlib \ - $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE + $(obj)/libquote.rlib $(obj)/libsyn.rlib \ + scripts/rustc-wrapper FORCE +$(call if_changed_dep,rustc_procmacro) $(obj)/$(libpin_init_internal_name): private rustc_target_flags = $(pin_init_internal-flags) $(obj)/$(libpin_init_internal_name): $(src)/pin-init/internal/src/lib.rs \ - $(obj)/libproc_macro2.rlib $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE + $(obj)/libproc_macro2.rlib $(obj)/libquote.rlib $(obj)/libsyn.rlib \ + scripts/rustc-wrapper FORCE +$(call if_changed_dep,rustc_procmacro) # `rustc` requires `-Zunstable-options` to use custom target specifications
scripts/.gitignore +1 -0
@@ -6,6 +6,7 @@ /kallsyms /module.lds /recordmcount +/rustc-wrapper /rustdoc_test_builder /rustdoc_test_gen /sign-file
scripts/Makefile.host +21 -2
@@ -87,11 +87,30 @@ hostcxx_flags = -Wp,-MMD,$(depfile) \ $(KBUILD_HOSTCXXFLAGS) $(HOST_EXTRACXXFLAGS) \ $(HOSTCXXFLAGS_$(target-stem).o) +# rustc's `-Clinker=` expects a single executable path, not a command line. +# `HOSTCC` may be a multi-word command when wrapped (e.g. "ccache gcc"), which +# would otherwise be split by the shell and mis-parsed by rustc. +# To work around this, we generate a wrapper script that forwards arguments to +# `HOSTRUSTC_LD` so that such commands can be used safely. +# +# Set `HOSTRUSTC_LD` for a different rustc linker command than `HOSTCC` +HOSTRUSTC_LD ?= $(HOSTCC) + +define filechk_rustc-wrapper + printf "%s\n" \ + '#!/bin/sh' \ + 'exec $(call escsq,$(HOSTRUSTC_LD)) "$$@"' +endef + +$(obj)/rustc-wrapper: FORCE + $(call filechk,rustc-wrapper) + $(Q)chmod +x $@ + # `--out-dir` is required to avoid temporaries being created by `rustc` in the # current working directory, which may be not accessible in the out-of-tree # modules case. hostrust_flags = --out-dir $(dir $@) --emit=dep-info=$(depfile) \ - -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=scripts/rustc-wrapper \ -Clink-args='$(call escsq,$(KBUILD_HOSTLDFLAGS))' \ $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \ $(HOSTRUSTFLAGS_$(target-stem))
@@ -153,7 +172,7 @@ $(host-cxxobjs): $(obj)/%.o: $(obj)/%.cc FORCE quiet_cmd_host-rust = HOSTRUSTC $@ cmd_host-rust = \ $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $< -$(host-rust): $(obj)/%: $(src)/%.rs FORCE +$(host-rust): $(obj)/%: $(src)/%.rs $(obj)/rustc-wrapper FORCE +$(call if_changed_dep,host-rust) targets += $(host-csingle) $(host-cmulti) $(host-cobjs) \
Signature · 1 line
2.52.0
Quoted 19 lines · Mohamad Alsadhan
rustc's -C linker= option expects a single executable path. When HOSTCC contains a wrapper (e.g. "ccache gcc"), passing
-Clinker=$(HOSTCC)results in the shell splitting the value into multiple words, and rustc interprets the additional word as an input filename:error: multiple input filenames provided ...
Generate a small wrapper script and pass it to -Clinker e.g.
``` #!/bin/sh exec ccache gcc "$@" ```This fix should be general enough to address most if not all cases (incl. wrappers or subcommands) and avoids surprises of simpler fixes like just defaulting to gcc.
Quoted 5 lines
This avoids passing the user command as an environment variable as that would be more challenging to trace and debug shell expansions.
I tested this patch in the original environnement for the issue : a ccache-enabled Yocto build. It works. Thanks!
Quoted 117 lines
Signed-off-by Mohamad Alsadhan <[email protected]>---
v5 -> v6:
- Add fix torust/Makefileas well (Yoann)
- Include script to.gitignoreand make clean (Nicolas)
- Add back the outerexecto the commandv4 -> v5:
- Fix word splitting issues
- Remove unnecessaryexec sh -cand simplify generated scriptv3 -> v4:
- Use filechk instead ofif_changedmacro to regenerate script
- Remove trailing space at EOLv2 -> v3:
- Scrap previous hacky approaches (e.g. using lastword) and go with a proper fix (Gary) which turned out not that complex to implement.v1 -> v2:
- RenameHOSTRUSTC_LINKERtoHOSTRUSTC_LDfor consistency
- Introduce explicitHOSTRUSTC_LDoverride
- Warn when falling back due to multi-argument HOSTCC
- Error out if a user-specifiedHOSTRUSTC_LDis not an executablev1: https://lore.kernel.org/all/[email protected]/
v2: https://lore.kernel.org/all/[email protected]/
v3: https://lore.kernel.org/all/[email protected]/
v4: https://lore.kernel.org/all/[email protected]/
v5: https://lore.kernel.org/all/[email protected]/--- Makefile | 3 ++- rust/Makefile | 8 +++++--- scripts/.gitignore | 1 + scripts/Makefile.host | 23 +++++++++++++++++++++-- 4 files changed, 29 insertions(+), 6 deletions(-)Makefile +2
-1@@ -1651,7 +1651,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \ modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \ compile_commands.json rust/test \ rust-project.json .vmlinux.objs .vmlinux.export.c \ - .builtin-dtbs-list .builtin-dtb.S + .builtin-dtbs-list .builtin-dtb.S \ + scripts/rustc-wrapper # Directories & files removed with 'make mrproper' MRPROPER_FILES += include/config include/generated \rust/Makefile +5
-3@@ -565,7 +565,7 @@ $(obj)/libsyn.rlib: $(src)/syn/lib.rs $(obj)/libquote.rlib FORCE quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@ cmd_rustc_procmacro = \ $(RUSTC_OR_CLIPPY) $(rust_common_flags) $(rustc_target_flags) \ - -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=scripts/rustc-wrapper \ -Clink-args='$(call escsq,$(KBUILD_PROCMACROLDFLAGS))' \ --emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \ --crate-type proc-macro -L$(objtree)/$(obj) \@@ -576,12 +576,14 @@ quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@ $(obj)/$(libmacros_name): private rustc_target_flags = \ --extern proc_macro2 --extern quote --extern syn $(obj)/$(libmacros_name): $(src)/macros/lib.rs $(obj)/libproc_macro2.rlib \ - $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE + $(obj)/libquote.rlib $(obj)/libsyn.rlib \ + scripts/rustc-wrapper FORCE +$(call if_changed_dep,rustc_procmacro) $(obj)/$(libpin_init_internal_name): private rustc_target_flags = $(pin_init_internal-flags) $(obj)/$(libpin_init_internal_name): $(src)/pin-init/internal/src/lib.rs \ - $(obj)/libproc_macro2.rlib $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE + $(obj)/libproc_macro2.rlib $(obj)/libquote.rlib $(obj)/libsyn.rlib \ + scripts/rustc-wrapper FORCE +$(call if_changed_dep,rustc_procmacro) # `rustc` requires `-Zunstable-options` to use custom target specificationsscripts/.gitignore +1
-0@@ -6,6 +6,7 @@ /kallsyms /module.lds /recordmcount +/rustc-wrapper /rustdoc_test_builder /rustdoc_test_gen /sign-filescripts/Makefile.host +15
-0@@ -87,11 +87,30 @@ hostcxx_flags = -Wp,-MMD,$(depfile) \ $(KBUILD_HOSTCXXFLAGS) $(HOST_EXTRACXXFLAGS) \ $(HOSTCXXFLAGS_$(target-stem).o) +# rustc's `-Clinker=` expects a single executable path, not a command line. +# `HOSTCC` may be a multi-word command when wrapped (e.g. "ccache gcc"), which +# would otherwise be split by the shell and mis-parsed by rustc. +# To work around this, we generate a wrapper script that forwards arguments to +# `HOSTRUSTC_LD` so that such commands can be used safely. +# +# Set `HOSTRUSTC_LD` for a different rustc linker command than `HOSTCC` +HOSTRUSTC_LD ?= $(HOSTCC) + +define filechk_rustc-wrapper + printf "%s\n" \ + '#!/bin/sh' \ + 'exec $(call escsq,$(HOSTRUSTC_LD)) "$$@"' +endef +
scripts/Makefile.host hostcxx_flags = -Wp,-MMD,$(depfile) \ · line 113
Quoted 10 lines
scripts/Makefile.host +5
-1+$(obj)/rustc-wrapper: FORCE + $(call filechk,rustc-wrapper) + $(Q)chmod +x $@ + # `--out-dir` is required to avoid temporaries being created by `rustc` in the # current working directory, which may be not accessible in the out-of-tree # modules case. hostrust_flags = --out-dir $(dir $@) --emit=dep-info=$(depfile) \ - -Clinker-flavor=gcc -Clinker=$(HOSTCC) \ + -Clinker-flavor=gcc -Clinker=scripts/rustc-wrapper \
Does using "scripts/" without a $(obj)/$(srctree)/... prefix always works? I know I've suggested this as a quick fix. But I did not think about it a lot.
Quoted 12 lines
scripts/Makefile.host +1
-1-Clink-args='$(call escsq,$(KBUILD_HOSTLDFLAGS))' \ $(KBUILD_HOSTRUSTFLAGS) $(HOST_EXTRARUSTFLAGS) \ $(HOSTRUSTFLAGS_$(target-stem))@@ -153,7 +172,7 @@ $(host-cxxobjs): $(obj)/%.o: $(obj)/%.cc FORCE quiet_cmd_host-rust = HOSTRUSTC $@ cmd_host-rust = \ $(HOSTRUSTC) $(hostrust_flags) --emit=link=$@ $< -$(host-rust): $(obj)/%: $(src)/%.rs FORCE +$(host-rust): $(obj)/%: $(src)/%.rs $(obj)/rustc-wrapper FORCE +$(call if_changed_dep,host-rust) targets += $(host-csingle) $(host-cmulti) $(host-cobjs) \
Signature · 2 lines
Yoann Congal Smile ECS
Apologies for getting back to you late, I was busy with other things and I wanted to double-check things before replying.
Quoted 5 lines · Yoann Congal
Does using "
scripts/" without a $(obj)/$(srctree)/... prefix always works? I know I've suggested this as a quick fix. But I did not think about it a lot.
You're in fact correct, it's not always correct. Thanks for pointing it out :)
I assumed it was fine since the top-level build runs the scripts pass before rust/, and existing Kbuild commands commonly refer to generated tools under scripts/ using paths relative to the object tree.
That misses some cases:
- On e.g. ARM64 with CONFIG_RUST_KERNEL_DOCTESTS=n, no Rust hostprog may be built in scripts/, so scripts/rustc-wrapper may never be generated before rust/Makefile needs it.
- scripts/Makefile.host is also used outside scripts/. For example, samples/rust/hostprogs/ generates samples/rust/hostprogs/rustc-wrapper, but the hardcoded path made rustc look for scripts/rustc-wrapper.
A v7 patch is on the way that should fix this issue, by always generating scripts/rustc-wrapper for CONFIG_RUST=y and using $(obj)/rustc-wrapper for generic host Rust builds (+ some little hardening and cleaning).
Best regards,
Mo
Nicolas Schier
maintainerQuoted 22 lines · Mohamad Alsadhan
rustc's -C linker= option expects a single executable path. When HOSTCC contains a wrapper (e.g. "ccache gcc"), passing
-Clinker=$(HOSTCC)results in the shell splitting the value into multiple words, and rustc interprets the additional word as an input filename:error: multiple input filenames provided ...
Generate a small wrapper script and pass it to -Clinker e.g.
``` #!/bin/sh exec ccache gcc "$@" ```This fix should be general enough to address most if not all cases (incl. wrappers or subcommands) and avoids surprises of simpler fixes like just defaulting to gcc.
This avoids passing the user command as an environment variable as that would be more challenging to trace and debug shell expansions.
Quoted 8 lines
Suggested-by Yoann Congal <[email protected]>Signed-off-by Mohamad Alsadhan <[email protected]>---
v5 -> v6:
- Add fix torust/Makefileas well (Yoann)
- Include script to.gitignoreand make clean (Nicolas)
- Add back the outerexecto the command
What is the reason for re-adding the 'exec'?
Signature · 1 line
Nicolas
Quoted 2 lines · Nicolas Schier
What is the reason for re-adding the 'exec'?
I was under the impression of that it won't hurt, unless I am missing something.
'exec' would replace the shell process entirely rather than leaving it alive waiting for the compiler to finish.
Best regards,
Mo