Skip to contentSkip to conversation

kbuild: pass jobserver to cmd_ld_vmlinux.o

PATCH In review Jiri Slaby 4 participants 7 messages Jul 19, 2022 – Sep 20, 2022 Kbuild

lore
Share & tools
lore

Jiri Slaby

author
#0 lore raw

PATCH v2 1/2 kbuild: pass jobserver to cmd_ld_vmlinux.o

Until the link-vmlinux.sh split (cf. the commit below), the linker was run with jobserver set in MAKEFLAGS. After the split, the command in Makefile.vmlinux_o is not prefixed by "+" anymore, so this information is lost.

Restore it as linkers working in parallel (esp. the LTO ones) make a use
of it. In this case, it is gcc producing this warning otherwise:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

Cc Sedat Dilek <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Michal Marek <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Signed-off-by: Martin Liska <[email protected]>
Fixes 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)

---

Notes:
[v2] only commit message updated

 scripts/Makefile.vmlinux_o | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/Makefile.vmlinux_o +1 -1
index 84019814f33f..886db0659d9c 100644
@@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
 endef
 
 vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
-	$(call if_changed_rule,ld_vmlinux.o)
+	+$(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
 
Signature · 1 line
2.37.1
author
#1 lore raw

PATCH v2 2/2 kbuild: lto: preserve MAKEFLAGS for module linking

From: Martin Liska <[email protected]>

Prefix command in makefile run in order to preserve access to jobserver. This is needed for gcc at least.

Fixes this warning:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

Cc Sedat Dilek <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Michal Marek <[email protected]>
Cc Nick Desaulniers <[email protected]>
Fixes 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)

---

Notes:
[v2] this is new in v2

 scripts/Makefile.build | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
scripts/Makefile.build +3 -3
index bb63374fb055..493f3c4e8461 100644
@@ -247,8 +247,8 @@ endef
 
 # Built-in and composite module parts
 $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
-	$(call if_changed_rule,cc_o_c)
-	$(call cmd,force_checksrc)
+	+$(call if_changed_rule,cc_o_c)
+	+$(call cmd,force_checksrc)
 
 # To make this rule robust against "Argument list too long" error,
 # ensure to add $(obj)/ prefix by a shell command.
@@ -457,7 +457,7 @@ endef
 $(multi-obj-m): objtool-enabled := $(delay-objtool)
 $(multi-obj-m): part-of-module := y
 $(multi-obj-m): %.o: %.mod FORCE
-	$(call if_changed_rule,ld_multi_m)
+	+$(call if_changed_rule,ld_multi_m)
 $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)
 
 targets := $(filter-out $(PHONY), $(targets))
Signature · 1 line
2.37.1
#2 lore raw

Re: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

Quoted 8 lines · Jiri Slaby

From: Martin Liska <[email protected]>

Prefix command in makefile run in order to preserve access to jobserver. This is needed for gcc at least.

Fixes this warning:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

What is lto-wrapper ?

Quoted 21 lines
Cc Sedat Dilek <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Michal Marek <[email protected]>
Cc Nick Desaulniers <[email protected]>
Fixes 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)

---

Notes:
[v2] this is new in v2

 scripts/Makefile.build | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
scripts/Makefile.build +0 -0
index bb63374fb055..493f3c4e8461 100644

scripts/Makefile.build endef · line 251

Quoted 7 lines
scripts/Makefile.build +2 -2
@@ -247,8 +247,8 @@ endef
 
 # Built-in and composite module parts
 $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
-       $(call if_changed_rule,cc_o_c)
-       $(call cmd,force_checksrc)
+       +$(call if_changed_rule,cc_o_c)
+       +$(call cmd,force_checksrc)

cmd_force_checksrc invokes sparse.
How is it related to GCC?

scripts/Makefile.build endef · line 461

Quoted 10 lines
scripts/Makefile.build +1 -1
 # To make this rule robust against "Argument list too long" error,
 # ensure to add $(obj)/ prefix by a shell command.
@@ -457,7 +457,7 @@ endef
 $(multi-obj-m): objtool-enabled := $(delay-objtool)
 $(multi-obj-m): part-of-module := y
 $(multi-obj-m): %.o: %.mod FORCE
-       $(call if_changed_rule,ld_multi_m)
+       +$(call if_changed_rule,ld_multi_m)
 $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)

cmd_ld_multi_m invokes $(LD) directly.
How is it related to gcc?

Quoted 5 lines
scripts/Makefile.build +0 -1
 targets := $(filter-out $(PHONY), $(targets))
--

2.37.1

Signature · 2 lines
Best Regards
Masahiro Yamada
#3 lore raw

Re: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

Quoted 4 lines · Masahiro Yamada
Quoted 3 lines · Jiri Slaby

From: Martin Liska <[email protected]>

Quoted 8 lines
Quoted 5 lines

Prefix command in makefile run in order to preserve access to jobserver. This is needed for gcc at least.

Fixes this warning:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

What is lto-wrapper ?

lto-wrapper is a part of gcc:
https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc

And is invoked during the link of LTO objects (also to link in parallel if jobserver is available).

I am leaving the unanswered questions below for Martin to answer.

Quoted 49 lines
Quoted 20 lines
Cc Sedat Dilek <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Michal Marek <[email protected]>
Cc Nick Desaulniers <[email protected]>
Fixes 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)

---

Notes:
[v2] this is new in v2

  scripts/Makefile.build | 6 +++---
  1 file changed, 3 insertions(+), 3 deletions(-)
scripts/Makefile.build +0 -0
index bb63374fb055..493f3c4e8461 100644

scripts/Makefile.build endef · line 251

Quoted 7 lines
scripts/Makefile.build +2 -2
@@ -247,8 +247,8 @@ endef
 
  # Built-in and composite module parts
  $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE
-       $(call if_changed_rule,cc_o_c)
-       $(call cmd,force_checksrc)
+       +$(call if_changed_rule,cc_o_c)
+       +$(call cmd,force_checksrc)

cmd_force_checksrc invokes sparse.
How is it related to GCC?

scripts/Makefile.build endef · line 461

Quoted 10 lines
scripts/Makefile.build +1 -1
  # To make this rule robust against "Argument list too long" error,
  # ensure to add $(obj)/ prefix by a shell command.
@@ -457,7 +457,7 @@ endef
  $(multi-obj-m): objtool-enabled := $(delay-objtool)
  $(multi-obj-m): part-of-module := y
  $(multi-obj-m): %.o: %.mod FORCE
-       $(call if_changed_rule,ld_multi_m)
+       +$(call if_changed_rule,ld_multi_m)
  $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)

cmd_ld_multi_m invokes $(LD) directly.
How is it related to gcc?

Signature · 2 lines
js
suse labs
#4 lore raw

Re: [PATCH v2 2/2] kbuild: lto: preserve MAKEFLAGS for module linking

Quoted 12 lines · Jiri Slaby
Quoted 11 lines · Masahiro Yamada
Quoted 8 lines · Jiri Slaby

From: Martin Liska <[email protected]>

Prefix command in makefile run in order to preserve access to jobserver. This is needed for gcc at least.

Fixes this warning:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

Quoted 6 lines
Quoted 1 line

What is lto-wrapper ?

lto-wrapper is a part of gcc:
https://gcc.gnu.org/git/?p=gcc.git;a=blob;f=gcc/lto-wrapper.cc

And is invoked during the link of LTO objects (also to link in parallel if jobserver is available).

Yes, it's a driver that is invoked by Linker when it decides to use LTO mode in GCC compiler. Later on the lto-wrapper connects to make's jobserver and runs in parallel. Note a recent [1] make implemented named pipes when it comes to communication with jobserver. And thus the '+' won't be needed in the future, but we need it for now.

Cheers,
Martin

[1] https://git.savannah.gnu.org/cgit/make.git/commit/?id=7ad2593b2d2bb5b9332f4444d8bf93ac6f958bc6

Quoted 55 lines

I am leaving the unanswered questions below for Martin to answer.

Quoted 49 lines
Quoted 22 lines
Cc Sedat Dilek <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Michal Marek <[email protected]>
Cc Nick Desaulniers <[email protected]>
Fixes 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)

---

Notes:
     [v2] this is new in v2

  scripts/Makefile.build | 6 +++---

  1 file changed, 3 insertions(+), 3 deletions(-)
scripts/Makefile.build +0 -0
index bb63374fb055..493f3c4e8461 100644
@@ -247,8 +247,8 @@ endef
 
   # Built-in and composite module parts
Quoted 5 lines

  $(obj)/%.o: $(src)/%.c $(recordmcount_source) FORCE -       $(call if_changed_rule,cc_o_c)
-       $(call cmd,force_checksrc)
+       +$(call if_changed_rule,cc_o_c)
+       +$(call cmd,force_checksrc)

cmd_force_checksrc invokes sparse.
How is it related to GCC?

Quoted 10 lines

  # To make this rule robust against "Argument list too long" error,

  # ensure to add $(obj)/ prefix by a shell command.
scripts/Makefile.build +0 -0
@@ -457,7 +457,7 @@ endef
   $(multi-obj-m): objtool-enabled := $(delay-objtool)

  $(multi-obj-m): part-of-module := y $(multi-obj-m): %.o: %.mod FORCE
-       $(call if_changed_rule,ld_multi_m)
+       +$(call if_changed_rule,ld_multi_m) $(call multi_depend, $(multi-obj-m), .o, -objs -y -m)

cmd_ld_multi_m invokes $(LD) directly.
How is it related to gcc?

#5 lore raw
In reply to Jiri Slaby
Quoted 17 lines · Jiri Slaby

Until the link-vmlinux.sh split (cf. the commit below), the linker was run with jobserver set in MAKEFLAGS. After the split, the command in Makefile.vmlinux_o is not prefixed by "+" anymore, so this information is lost.

Restore it as linkers working in parallel (esp. the LTO ones) make a use
of it. In this case, it is gcc producing this warning otherwise:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

Cc Sedat Dilek <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Michal Marek <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Signed-off-by: Martin Liska <[email protected]>
Fixes 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)
Quoted 4 lines

---

Notes:
[v2] only commit message updated

Did you read my previous email?

https://lore.kernel.org/linux-kbuild/CAK7LNAS46hrVh3FNiHHL8_Hqp85VgWnsHyjQ4t-TFH_H344NOg@mail.gmail.com/

Quoted 20 lines
 scripts/Makefile.vmlinux_o | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
scripts/Makefile.vmlinux_o +1 -2
index 84019814f33f..886db0659d9c 100644
@@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
 endef
 
 vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
-       $(call if_changed_rule,ld_vmlinux.o)
+       +$(call if_changed_rule,ld_vmlinux.o)
 
 targets += vmlinux.o
 
--

2.37.1

Signature · 2 lines
Best Regards
Masahiro Yamada
#6 lore raw
Quoted 20 lines · Masahiro Yamada
Quoted 19 lines · Jiri Slaby

Until the link-vmlinux.sh split (cf. the commit below), the linker was run with jobserver set in MAKEFLAGS. After the split, the command in Makefile.vmlinux_o is not prefixed by "+" anymore, so this information is lost.

Restore it as linkers working in parallel (esp. the LTO ones) make a use
of it. In this case, it is gcc producing this warning otherwise:
lto-wrapper: warning: jobserver is not available: ‘--jobserver-auth=’ is not present in ‘MAKEFLAGS’

Cc Sedat Dilek <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Michal Marek <[email protected]>
Cc Nick Desaulniers <[email protected]>
Cc Signed-off-by: Martin Liska <[email protected]>
Fixes 5d45950dfbb1 (kbuild: move vmlinux.o link to scripts/Makefile.vmlinux_o)

---

Quoted 7 lines
Quoted 2 lines

Notes:
[v2] only commit message updated

Did you read my previous email?

https://lore.kernel.org/linux-kbuild/CAK7LNAS46hrVh3FNiHHL8_Hqp85VgWnsHyjQ4t-TFH_H344NOg@mail.gmail.com/

Huh, I neither received this email nor the linked one to my inbox. @suse.cz somehow filters mails from you :/. Let me figure out (and use kernel.org account for now). (And get back to your comments too.)

Quoted 21 lines
Quoted 19 lines
  scripts/Makefile.vmlinux_o | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
scripts/Makefile.vmlinux_o +1 -2
index 84019814f33f..886db0659d9c 100644
@@ -65,7 +65,7 @@ define rule_ld_vmlinux.o
  endef
 
  vmlinux.o: $(initcalls-lds) $(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
-       $(call if_changed_rule,ld_vmlinux.o)
+       +$(call if_changed_rule,ld_vmlinux.o)
 
  targets += vmlinux.o
 
--

2.37.1

Signature · 1 line
js