Kbuild: Support nested composite objects
cover Kbuild: Support nested composite objects
This series was developed after discussion in https://lkml.org/lkml/2021/1/19/850
The motivation for this series is an out-of-tree module which contains a large number of source files. This causes Kbuild to exceed the maximum command line argument length when linking the files. Proposal here permits composite objects to contain other composite objects. This allows the driver to split linking into several steps and avoid the maximum command line length error.
Kbuild composite objects only supports one level of composite objects. That is, a composite object may only be composed of real compilable source files.
As a simple example, the following Kbuild description is now supported:
bar-a-y := a/bar0.o a/bar1.o
bar-b-y := b/bar2.o b/bar3.o
foo-objs := bar-a.o bar-b.o
obj-m += foo.o
Add such support by recursively searching for composite objects and listing them in $(multi-used-*) and $(real-obj-*).
Elliot Berman (2):
Kbuild: Make composite object searching more generic
Kbuild: Support nested composite objects
scripts/Makefile.lib | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
RFC 1/2 Kbuild: Make composite object searching more generic
Reduce repeated logic around expanding composite objects in preparation for later commit to support nested composite objects.
--- scripts/Makefile.lib | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
scripts/Makefile.lib +8 -4
@@ -56,15 +56,19 @@ else obj-y := $(filter-out %/, $(obj-y)) endif +# Expand $(foo-objs) $(foo-y) by calling $(call suffix-search,foo.o,-objs -y) +suffix-search = $(foreach s,$(2),$($(1:.o=$s))) # If $(foo-objs), $(foo-y), $(foo-m), or $(foo-) exists, foo.o is a composite object -multi-used-y := $(sort $(foreach m,$(obj-y), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-))), $(m)))) -multi-used-m := $(sort $(foreach m,$(obj-m), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)) $($(m:.o=-))), $(m)))) +multi-search = $(sort $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)), $(m)))) +multi-used-y := $(call multi-search,$(obj-y),-objs -y) +multi-used-m := $(call multi-search,$(obj-m),-objs -y -m) multi-used := $(multi-used-y) $(multi-used-m) # Replace multi-part objects by their individual parts, # including built-in.a from subdirectories -real-obj-y := $(foreach m, $(obj-y), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-))),$($(m:.o=-objs)) $($(m:.o=-y)),$(m))) -real-obj-m := $(foreach m, $(obj-m), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)) $($(m:.o=-))),$($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)),$(m))) +real-search = $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)),$(call suffix-search,$(m),$(2)),$(m))) +real-obj-y := $(call real-search, $(obj-y),-objs -y) +real-obj-m := $(call real-search, $(obj-m),-objs -y -m) always-y += $(always-m)
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
RFC 2/2 Kbuild: Support nested composite objects
In reply to Elliot BermanKbuild composite objects only supports one level of composite objects. That is, a composite object may only be composed of real compilable source files.
As a simple example, the following Kbuild description is now supported:
bar-a-y := a/bar0.o a/bar1.o
bar-b-y := b/bar2.o b/bar3.o
foo-objs := bar-a.o bar-b.o
obj-m += foo.o
Add such support by recursively searching for composite objects and listing them in $(multi-used-*) and $(real-obj-*).
--- scripts/Makefile.lib | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
scripts/Makefile.lib +10 -2
@@ -59,14 +59,22 @@ endif # Expand $(foo-objs) $(foo-y) by calling $(call suffix-search,foo.o,-objs -y) suffix-search = $(foreach s,$(2),$($(1:.o=$s))) # If $(foo-objs), $(foo-y), $(foo-m), or $(foo-) exists, foo.o is a composite object -multi-search = $(sort $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)), $(m)))) +# Do this recursively to find nested composite objects. For backwards compatibility, +# foo-y may contain foo.o bar.o . We shouldn't recurse further in this case. +multi-search = $(sort $(foreach m,$(1),$(if $(strip $(call suffix-search,$(m),$(2) -)),\ + $(m) $(call multi-search,$(filter-out $(m),$(call suffix-search,$(m),$(2))),$(2))))) multi-used-y := $(call multi-search,$(obj-y),-objs -y) multi-used-m := $(call multi-search,$(obj-m),-objs -y -m) multi-used := $(multi-used-y) $(multi-used-m) # Replace multi-part objects by their individual parts, # including built-in.a from subdirectories -real-search = $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)),$(call suffix-search,$(m),$(2)),$(m))) +# Recursively search for real files. For backwards compatibility, +# foo-y may contain foo.o bar.o . foo.o in this context is a real object, and +# shouldn't be recursed into. +real-search = $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)), \ + $(filter $(m),$(call suffix-search,$(m),$(2))) $(call real-search,$(filter-out $(m),$(call suffix-search,$(m),$(2))),$(2)),\ + $(m))) real-obj-y := $(call real-search, $(obj-y),-objs -y) real-obj-m := $(call real-search, $(obj-m),-objs -y -m)
Signature · 2 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Quoted 8 lines · Elliot Berman
This series was developed after discussion in https://lkml.org/lkml/2021/1/19/850
The motivation for this series is an out-of-tree module which contains a large number of source files. This causes
Kbuildto exceed the maximum command line argument length when linking the files. Proposal here permits composite objects to contain other composite objects. This allows the driver to split linking into several steps and avoid the maximum command line length error.
External modules often get the cold shoulder.
For example,
https://lore.kernel.org/patchwork/patch/1175556/#1372233
This problem has not been observed in the upstream tree.
I do not see a good reason to complicate the build infrastructure for some external modules.
Quoted 27 lines
Kbuildcomposite objects only supports one level of composite objects. That is, a composite object may only be composed of real compilable source files.As a simple example, the following
Kbuilddescription is now supported:bar-a-y :=
a/bar0.oa/bar1.o
bar-b-y :=b/bar2.ob/bar3.ofoo-objs := bar-a.o bar-b.o
obj-m += foo.o
Add such support by recursively searching for composite objects and listing them in $(multi-used-*) and $(real-obj-*).
Elliot Berman (2):
Kbuild: Make composite object searching more generic
Kbuild: Support nested composite objectsscripts/Makefile.lib | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)--
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
Signature · 2 lines
Best Regards Masahiro Yamada
Re: [RFC 1/2] Kbuild: Make composite object searching more generic
In reply to Elliot BermanQuoted 5 lines · Elliot Berman
Reduce repeated logic around expanding composite objects in preparation for later commit to support nested composite objects.
Signed-off-by Elliot Berman <[email protected]>
This patch is a nice clean-up
regardless of your main motivation in 2/2.
I removed the "in preparation ..." part.
Applied to linux-kbuild. Thanks.
Quoted 33 lines
--- scripts/Makefile.lib | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)scripts/Makefile.lib +8
-5@@ -56,15 +56,19 @@ else obj-y := $(filter-out %/, $(obj-y)) endif +# Expand $(foo-objs) $(foo-y) by calling $(call suffix-search,foo.o,-objs -y) +suffix-search = $(foreach s,$(2),$($(1:.o=$s))) # If $(foo-objs), $(foo-y), $(foo-m), or $(foo-) exists, foo.o is a composite object -multi-used-y := $(sort $(foreach m,$(obj-y), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-))), $(m)))) -multi-used-m := $(sort $(foreach m,$(obj-m), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)) $($(m:.o=-))), $(m)))) +multi-search = $(sort $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)), $(m)))) +multi-used-y := $(call multi-search,$(obj-y),-objs -y) +multi-used-m := $(call multi-search,$(obj-m),-objs -y -m) multi-used := $(multi-used-y) $(multi-used-m) # Replace multi-part objects by their individual parts, # including built-in.a from subdirectories -real-obj-y := $(foreach m, $(obj-y), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-))),$($(m:.o=-objs)) $($(m:.o=-y)),$(m))) -real-obj-m := $(foreach m, $(obj-m), $(if $(strip $($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)) $($(m:.o=-))),$($(m:.o=-objs)) $($(m:.o=-y)) $($(m:.o=-m)),$(m))) +real-search = $(foreach m,$(1), $(if $(strip $(call suffix-search,$(m),$(2) -)),$(call suffix-search,$(m),$(2)),$(m))) +real-obj-y := $(call real-search, $(obj-y),-objs -y) +real-obj-m := $(call real-search, $(obj-m),-objs -y -m) always-y += $(always-m) --
Quoted 3 lines
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project
--
Best Regards
Masahiro Yamada
Quoted 5 lines · Elliot Berman
This series was developed after discussion in https://lkml.org/lkml/2021/1/19/850
The motivation for this series is an out-of-tree module which contains a large number of source files. This causes
Kbuildto exceed the maximum command line argument length when linking the files. Proposal here permits composite objects
Please don't even try to bloat the kernel build system for this. Thanks!
Quoted 3 lines · PDT Elliot Berman
The motivation for this series is an out-of-tree module which contains a large number of source files. This causes
Kbuildto exceed the maximum command line argument length when linking the files. Proposal here
I think I am familiar with this module.
In addition to exceeding the argument length when linking, it will fail to compile due to 169 include directories being added to c_flags. Depending on root path, this can also overflow the single argument limit of 128 kB. Perhaps you have not triggered this yet? These patches here will not help with this issue. But see below for correct and incorrect fixes.
The fundamental problem is that the kernel build system is designed to be recursive, with one makefile per directory. This module has a single Kbuild of over 3000 lines for the entire tree.
Some of this is from this driver not using multiple modules for what is a giant driver (337 MB .ko file). It should be split into multiple modules, if only to manage complexity by creating public vs private interfaces through choosing what symbols to export or not. It would avoid merge conflicts on the single shared Kbuild file used tree-wide. Potential name collisions across the 100+ include directories could be avoided by not having a shared set of include flags for the entire tree.
But some I think is also a limitation of kbuild. Giant monolithic Kbuilds do not work well, but this is ok, because you are supposed to use a recursive make design with a Kbuild per directory. But only non-module code may be split across directories. This important design feature is not extended to modules. A module must be be contained in one directory or monolithic tree. Why should non-module code be allowed the complexity of a recursive build but code for a module is not?
I did split this driver. I'm not aware of any way for me to send patches to
Qualcomm. I'd have to maintain this change myself and that would be very
difficult.
So I worked around it without needing to modify the kernel build system and with a minimal patch to Qualcomm's code for me to maintain.
1st, use @file to inject the include directories. I didn't test well, but I think this does not break kbuild's ability to detect changes in command lines.
-ccflags-y += $(INCS)
+ccflags-y += @$(obj)/includes
+$(obj)/includes: FORCE
+ $(call if_changed,include_flag_file)
+
+quiet_cmd_include_flag_file = I-FLAGS $@ +cmd_include_flag_file = echo "$(INCS)" > $@
+
+# To make all objects depend on the file that contains the flags +$(addprefix $(obj)/,$(OBJS)): $(obj)/includes
The last line might be a bit hacky. I couldn't find any proper mechanism to manually add a dependency to object files. It seems like auto dependency discovered headers are the only ones supported.
Next, we must deal with ~550 object files being linked at once. The subject of this patch series.
-$(MODNAME)-y := $(OBJS)
+$(MODNAME)-y := blob1.o blob2.o blob3.o
+blob1-objs := $(wordlist 1, 200, $(OBJS))
+blob2-objs := $(wordlist 201, 400, $(OBJS))
+blob3-objs := $(wordlist 401, 700, $(OBJS))
+# Multiple levels of composite objects don't really work. There is no rule +# to create blobX.o unless we add them to the obj-m list, even though +# wlan-y lists them and will be used to create wlan.o from them. We'll get +# blobX.ko modules we don't actually want by doing this. +obj-$(CONFIG_QCA_CLD_WLAN) += blob1.o blob2.o blob3.o
The last line here is a total hack, but it does work. By added blob1.ko as a module to be built, a rule to build blob1.o is also created, which will then be used by wlan.o's dependency on blob1.o