Skip to contentSkip to conversation

kbuild: simplify cmd_mod

PATCH In review Jeff Johnson 4 participants 12 messages Jan 13, 2021 – Jan 19, 2021 Kbuild

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 kbuild: simplify cmd_mod

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units can exceed execv argument list resulting in E2BIG (Argument list too long) error.

Fix this by replacing shell 'echo > file' into a more native $(file op filename[,text]) option.

---
 scripts/Makefile.build | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
scripts/Makefile.build +5 -4
index 4c058f1..252b7d2 100644
@@ -279,10 +279,11 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE
 	$(call if_changed_rule,cc_o_c)
 	$(call cmd,force_checksrc)
 
-cmd_mod = { \
-	echo $(if $($*-objs)$($*-y)$($*-m), $(addprefix $(obj)/, $($*-objs) $($*-y) $($*-m)), $(@:.mod=.o)); \
-	$(undefined_syms) echo; \
-	} > $@
+cmd_mod = $(file >$@,\
+		$(if $($*-objs)$($*-y)$($*-m), \
+			$(addprefix $(obj)/, $($*-objs) $($*-y) $($*-m)), \
+			$(@:.mod=.o))) \
+	$(undefined_syms) echo >> $@
 
 $(obj)/%.mod: $(obj)/%.o FORCE
 	$(call if_changed,mod)
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#1 lore raw

PATCH 2/2 kbuild: handle excessively long argument lists

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---
 scripts/Makefile.build | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/Makefile.build +5 -1
index 252b7d2..d5ef345 100644
@@ -425,7 +425,11 @@ $(obj)/lib.a: $(lib-y) FORCE
 # module is turned into a multi object module, $^ will contain header file
 # dependencies recorded in the .*.cmd file.
 quiet_cmd_link_multi-m = LD [M]  $@
-      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ $(filter %.o,$^)
+      cmd_link_multi-m =					\
+	$(file >[email protected],$(filter %.o,$^))				\
+	$(LD) $(ld_flags) -r -o $@ @[email protected];			\
+	rm -f [email protected]
+endif
 
 $(multi-used-m): FORCE
 	$(call if_changed,link_multi-m)
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
#2 lore raw

Re: [PATCH 2/2] kbuild: handle excessively long argument lists

Quoted 21 lines

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---
 scripts/Makefile.build | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/Makefile.build +0 -0
index 252b7d2..d5ef345 100644

scripts/Makefile.build $(obj)/lib.a: $(lib-y) FORCE · line 432

Quoted 9 lines
scripts/Makefile.build +5 -1
@@ -425,7 +425,11 @@ $(obj)/lib.a: $(lib-y) FORCE
 # module is turned into a multi object module, $^ will contain header file
 # dependencies recorded in the .*.cmd file.
 quiet_cmd_link_multi-m = LD [M]  $@
-      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ $(filter %.o,$^)
+      cmd_link_multi-m =					\
+	$(file >[email protected],$(filter %.o,$^))				\
+	$(LD) $(ld_flags) -r -o $@ @[email protected];			\
+	rm -f [email protected]
+endif

Was this build tested?

$ make LLVM=1 LLVM_IAS=1 -j72 defconfig
scripts/Makefile.build:432: *** extraneous 'endif'.  Stop.
make: *** [Makefile:535: scripts_basic] Error 2

(Please cc me on v2)

Quoted 6 lines
scripts/Makefile.build +0 -1
 $(multi-used-m): FORCE
 	$(call if_changed,link_multi-m)
-- 

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

#3 lore raw

Re: [PATCH 2/2] kbuild: handle excessively long argument lists

Quoted 34 lines · Nick Desaulniers
Quoted 25 lines

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---
 scripts/Makefile.build | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
scripts/Makefile.build +0 -0
index 252b7d2..d5ef345 100644
@@ -425,7 +425,11 @@ $(obj)/lib.a: $(lib-y) FORCE
 # module is turned into a multi object module, $^ will contain header 

file

 # dependencies recorded in the .*.cmd file.
 quiet_cmd_link_multi-m = LD [M]  $@

scripts/Makefile.build $(obj)/lib.a: $(lib-y) FORCE · line 432

Quoted 6 lines
scripts/Makefile.build +5 -1
-      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ $(filter %.o,$^)
+      cmd_link_multi-m =					\
+	$(file >[email protected],$(filter %.o,$^))				\
+	$(LD) $(ld_flags) -r -o $@ @[email protected];			\
+	rm -f [email protected]
+endif

Was this build tested?

Quoted 5 lines
$ make LLVM=1 LLVM_IAS=1 -j72 defconfig
scripts/Makefile.build:432: *** extraneous 'endif'.  Stop.
make: *** [Makefile:535: scripts_basic] Error 2

(Please cc me on v2)

blush

It was tested on a workspace that also contains the Clang LTO series
https://patchwork.kernel.org/project/linux-kbuild/patch/[email protected]/

I messed up when trimming, will update in v2

#4 lore raw

PATCH v2 2/2 kbuild: handle excessively long argument lists

In reply to Jeff Johnson

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---

Changes in v2:
- Remove spurious endif

scripts/Makefile.build | 5 ++++-

 1 file changed, 4 insertions(+), 1 deletion(-)
scripts/Makefile.build +4 -1
index 252b7d2..787dca2 100644
@@ -425,7 +425,10 @@ $(obj)/lib.a: $(lib-y) FORCE
 # module is turned into a multi object module, $^ will contain header file
 # dependencies recorded in the .*.cmd file.
 quiet_cmd_link_multi-m = LD [M]  $@
-      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ $(filter %.o,$^)
+      cmd_link_multi-m =					\
+	$(file >[email protected],$(filter %.o,$^))				\
+	$(LD) $(ld_flags) -r -o $@ @[email protected];			\
+	rm -f [email protected]
 
 $(multi-used-m): FORCE
 	$(call if_changed,link_multi-m)
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum,
a Linux Foundation Collaborative Project
Reviewed-by Tested-by on v2 2/2
#5 lore raw

Re: [PATCH v2 2/2] kbuild: handle excessively long argument lists

Quoted 26 lines · Jeff Johnson

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---

Changes in v2:
- Remove spurious endif

scripts/Makefile.build | 5 ++++-

 1 file changed, 4 insertions(+), 1 deletion(-)
scripts/Makefile.build +0 -0
index 252b7d2..787dca2 100644

scripts/Makefile.build $(obj)/lib.a: $(lib-y) FORCE · line 431

Quoted 8 lines
scripts/Makefile.build +4 -1
@@ -425,7 +425,10 @@ $(obj)/lib.a: $(lib-y) FORCE
 # module is turned into a multi object module, $^ will contain header file
 # dependencies recorded in the .*.cmd file.
 quiet_cmd_link_multi-m = LD [M]  $@
-      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ $(filter %.o,$^)
+      cmd_link_multi-m =                                       \
+       $(file >[email protected],$(filter %.o,$^))                         \
+       $(LD) $(ld_flags) -r -o $@ @[email protected];                      \
+       rm -f [email protected]

The GNU Make docs linked above use an @ before the invocation of rm. I don't know what that's about, but that or even this patch doesn't affect my ability to build negatively. LGTM

Quoted 7 lines
scripts/Makefile.build +0 -1
 $(multi-used-m): FORCE
        $(call if_changed,link_multi-m)
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

Signature · 2 lines
Thanks,
~Nick Desaulniers
#6 lore raw

Re: [PATCH v2 2/2] kbuild: handle excessively long argument lists

Quoted 32 lines · Nick Desaulniers
Quoted 30 lines · Jeff Johnson

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---

Changes in v2:
- Remove spurious endif

scripts/Makefile.build | 5 ++++-

 1 file changed, 4 insertions(+), 1 deletion(-)
scripts/Makefile.build +0 -1
index 252b7d2..787dca2 100644
@@ -425,7 +425,10 @@ $(obj)/lib.a: $(lib-y) FORCE
 # module is turned into a multi object module, $^ will contain header file
 # dependencies recorded in the .*.cmd file.
 quiet_cmd_link_multi-m = LD [M]  $@
-      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ $(filter %.o,$^)
Quoted 8 lines

scripts/Makefile.build $(obj)/lib.a: $(lib-y) FORCE · line 431

Quoted 4 lines
scripts/Makefile.build +4 -0
+      cmd_link_multi-m =                                       \
+       $(file >[email protected],$(filter %.o,$^))                         \
+       $(LD) $(ld_flags) -r -o $@ @[email protected];                      \
+       rm -f [email protected]

The GNU Make docs linked above use an @ before the invocation of rm. I don't know what that's about, but that or even this patch doesn't affect my ability to build negatively. LGTM

See this:

https://www.gnu.org/software/make/manual/html_node/Echoing.html#Echoing

Quoted 15 lines
Quoted 7 lines
scripts/Makefile.build +0 -1
 $(multi-used-m): FORCE
        $(call if_changed,link_multi-m)
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

--
Thanks,
~Nick Desaulniers

Signature · 2 lines
Best Regards
Masahiro Yamada
#7 lore raw

Re: [PATCH v2 2/2] kbuild: handle excessively long argument lists

In reply to Jeff Johnson
Quoted 8 lines · Jeff Johnson

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

Quoted 6 lines

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---

First, is this a real problem?
If so, which module is exceeding the command line limit?

$(file ) is only supported by GNU Make 4.0 or later.

The current minimum version is GNU Make 3.81.

If we need this feature, Documentation/process/changes.rst must be updated.

But, more importantly, none of your patches
works correctly.

Since $(file ...) is evaluated into an empty string, your patches would break the Kbuild ability that detects the command changes.

Steps to reproduce the problem
------------------------------

[1] add a module foo that consists of
three objects foo1.o, foo2.o, foo3.o

For example, like follows:

obj-m += foo.o
foo-objs := foo1.o foo2.o foo3.o

[2] Run 'make modules'

You will get the module foo.

[3] Drop foo3.o from the module members

Change Makefile as follows:

obj-m += foo.o
foo-objs := foo1.o foo2.o

[4] Re-run 'make modules'

The current build system cleverly
notices the Makefile change, and
correctly rebuilds the foo module.

With your patch set applied,
the build system would not rebuild
the module.

Quoted 26 lines

Changes in v2:
- Remove spurious endif

scripts/Makefile.build | 5 ++++-

 1 file changed, 4 insertions(+), 1 deletion(-)
scripts/Makefile.build +4 -2
index 252b7d2..787dca2 100644
@@ -425,7 +425,10 @@ $(obj)/lib.a: $(lib-y) FORCE
 # module is turned into a multi object module, $^ will contain header file
 # dependencies recorded in the .*.cmd file.
 quiet_cmd_link_multi-m = LD [M]  $@
-      cmd_link_multi-m = $(LD) $(ld_flags) -r -o $@ $(filter %.o,$^)
+      cmd_link_multi-m =                                       \
+       $(file >[email protected],$(filter %.o,$^))                         \
+       $(LD) $(ld_flags) -r -o $@ @[email protected];                      \
+       rm -f [email protected]
 
 $(multi-used-m): FORCE
        $(call if_changed,link_multi-m)
--

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

Signature · 2 lines
Best Regards
Masahiro Yamada
#8 lore raw

Re: [PATCH v2 2/2] kbuild: handle excessively long argument lists

Quoted 21 lines · Masahiro Yamada
Quoted 8 lines · Jeff Johnson

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

Quoted 6 lines

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---

First, is this a real problem?
If so, which module is exceeding the command line limit?

Quoted 2 lines · Masahiro Yamada

First, is this a real problem?
If so, which module is exceeding the command line limit?

Mahesh & I appreciate all of the feedback.

The issue is seen in an Android environment with an out-of-tree driver. The combination of long path names and a large number of source files is leading to the issue.

Since Mahesh & I are not Kbuild gurus, is there an alternative solution to this issue?

Jeff

--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project

#9 lore raw

Re: [PATCH v2 2/2] kbuild: handle excessively long argument lists

Quoted 31 lines · On Sat, Jan 16, 2021 at 5:15 AM <[email protected]> wrote:
Quoted 22 lines · Masahiro Yamada
Quoted 8 lines · Jeff Johnson

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

Quoted 6 lines

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---

First, is this a real problem?
If so, which module is exceeding the command line limit?

Quoted 2 lines · Masahiro Yamada

First, is this a real problem?
If so, which module is exceeding the command line limit?

Mahesh & I appreciate all of the feedback.

Quoted 8 lines

The issue is seen in an Android environment with an out-of-tree driver. The combination of long path names and a large number of source files is leading to the issue.

Since Mahesh & I are not Kbuild gurus, is there an alternative solution to this issue?

Jeff

I see.

The support for out-of-tree modules
is not nice in this regard, but fixing it
would need many changes.

The long-term solution might be to upstream your driver, but it might not be possible.

One cheesy workaround might be to point the module path via a symbolic link.

Let's say your module is located in a very deep
directory,
/home/foo/long/long/.../path/to/your/module

make M=/home/foo/long/long/.../path/to/your/module modules

would fail due to the too long command line.

First, create a symbolic link as follows:

ln -s /home/foo/long/long/.../path/to/your/module mod_dir

Then, pass the symbolic link to M= option.

make M=mod_dir modules

Signature · 2 lines
Best Regards
Masahiro Yamada
#10 lore raw

Re: [PATCH v2 2/2] kbuild: handle excessively long argument lists

Quoted 39 lines · Masahiro Yamada
Quoted 38 lines · On Sat, Jan 16, 2021 at 5:15 AM <[email protected]> wrote:
Quoted 22 lines · Masahiro Yamada
Quoted 8 lines · Jeff Johnson

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units may be exceeding AR and LD command argument list. Handle this gracefully by writing the long argument list in a file. The command line options read from file are inserted in place of the original @file option.

Quoted 6 lines

The usage is well documented at
https://www.gnu.org/software/make/manual/html_node/File-Function.html

---

First, is this a real problem?
If so, which module is exceeding the command line limit?

Quoted 2 lines · Masahiro Yamada

First, is this a real problem?
If so, which module is exceeding the command line limit?

Mahesh & I appreciate all of the feedback.

The issue is seen in an Android environment with an out-of-tree driver. The combination of long path names and a large number of source files is leading to the issue.

Since Mahesh & I are not Kbuild gurus, is there an alternative solution to this issue?

Quoted 9 lines
Quoted 1 line

Jeff

I see.

The support for out-of-tree modules
is not nice in this regard, but fixing it
would need many changes.

Agree with that. I checked the same after your first comment on this thread. It needs changes in multiple layers of makefiles.

Quoted 25 lines

The long-term solution might be to upstream your driver, but it might not be possible.

One cheesy workaround might be to point the module path via a symbolic link.

Let's say your module is located in a very deep
directory,
/home/foo/long/long/.../path/to/your/module

make M=/home/foo/long/long/.../path/to/your/module modules

would fail due to the too long command line.

Quoted 8 lines

First, create a symbolic link as follows:

ln -s /home/foo/long/long/.../path/to/your/module mod_dir

Then, pass the symbolic link to M= option.

make M=mod_dir modules

Thanks for the suggestion. Earlier, we have used a similar workaround of using relative path instead of absolute to reduce the command line length.

What's your input on the following approach where we link object files in different stages to reduce the command line length.

./Makefile

 # foo.c is combined and final module.

obj-m = fooa.o foob.o fooc.o

 # link already combined object files
 fooc-y := fooa.o foob.o
 # combine into different groups
 fooa-y := foo1.o foo2.o
 foob-y := foo3.o foo4.o

....

Note: We need to add MODULE_LICENSE in every group.

make -C ../linux-kbuild M=/local/mnt2/workspace/dev/foo modules

  CC [M]  /local/mnt2/workspace/dev/foo/foo1.o
  CC [M]  /local/mnt2/workspace/dev/foo/foo2.o
  LD [M]  /local/mnt2/workspace/dev/foo/fooa.o
  CC [M]  /local/mnt2/workspace/dev/foo/foo3.o
  CC [M]  /local/mnt2/workspace/dev/foo/foo4.o
  LD [M]  /local/mnt2/workspace/dev/foo/foob.o
  LD [M]  /local/mnt2/workspace/dev/foo/fooc.o
  MODPOST /local/mnt2/workspace/dev/foo/Module.symvers
  CC [M]  /local/mnt2/workspace/dev/foo/fooa.mod.o
  LD [M]  /local/mnt2/workspace/dev/foo/fooa.ko
  CC [M]  /local/mnt2/workspace/dev/foo/foob.mod.o
  LD [M]  /local/mnt2/workspace/dev/foo/foob.ko
  CC [M]  /local/mnt2/workspace/dev/foo/fooc.mod.o
  LD [M]  /local/mnt2/workspace/dev/foo/fooc.ko
Reviewed-by Tested-by on v1 1/2
#11 lore raw
In reply to Jeff Johnson
Quoted 21 lines

From: Mahesh Kumar Kalikot Veetil <[email protected]>

Modules with a large number of compilation units can exceed execv argument list resulting in E2BIG (Argument list too long) error.

Fix this by replacing shell 'echo > file' into a more native $(file op filename[,text]) option.

---
 scripts/Makefile.build | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)
scripts/Makefile.build +0 -0
index 4c058f1..252b7d2 100644
@@ -279,10 +279,11 @@ $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE
 	$(call if_changed_rule,cc_o_c)
 	$(call cmd,force_checksrc)

scripts/Makefile.build $(obj)/%.o: $(src)/%.c $(recordmcount_source) $(objtool_dep) FORCE · line 286

Quoted 10 lines
scripts/Makefile.build +5 -4
 
-cmd_mod = { \
-	echo $(if $($*-objs)$($*-y)$($*-m), $(addprefix $(obj)/, $($*-objs) $($*-y) $($*-m)), $(@:.mod=.o)); \
-	$(undefined_syms) echo; \
-	} > $@
+cmd_mod = $(file >$@,\
+		$(if $($*-objs)$($*-y)$($*-m), \
+			$(addprefix $(obj)/, $($*-objs) $($*-y) $($*-m)), \
+			$(@:.mod=.o))) \
+	$(undefined_syms) echo >> $@

I find the indendation to be a readability improvement. Thanks for the patch.

Quoted 8 lines
scripts/Makefile.build +0 -1
 $(obj)/%.mod: $(obj)/%.o FORCE
 	$(call if_changed,mod)
-- 

The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project