kbuild: Restore ability to build out-of-tree modules
Subsystems (4)
PATCH 1/1 kbuild: Restore ability to build out-of-tree modules
From: Leon Romanovsky <[email protected]>
The out-of-tree modules are built without vmlinux target and request to recompile that target unconditionally causes to the following compilation error.
[root@server kernel]# make
<..>
make -f ./scripts/Makefile.modpost
make -f ./scripts/Makefile.modfinal
make[3]: *** No rule to make target 'vmlinux', needed by '/my_temp/out-of-tree-module/kernel/test.ko'. Stop. make[2]: *** [scripts/Makefile.modpost:117: __modpost] Error 2
make[1]: *** [Makefile:1703: modules] Error 2
make[1]: Leaving directory '/usr/src/kernels/5.10.0-rc5_for_upstream_base_2020_11_29_11_34'
make: *** [Makefile:80: modules] Error 2
As a solution separate between build paths that has vmlinux target and paths without.
---
Not proficient enough in Makefile, but it fixes the issue.
--- scripts/Makefile.modfinal | 5 +++++ scripts/Makefile.modpost | 4 ++++ 2 files changed, 9 insertions(+)
scripts/Makefile.modfinal +5 -0
@@ -48,9 +48,14 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check), \ $(cmd); \ printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:) +ifdef MODPOST_VMLINUX # Re-generate module BTFs if either module's .ko or vmlinux changed $(modules): %.ko: %.o %.mod.o scripts/module.lds vmlinux FORCE +$(call if_changed_except,ld_ko_o,vmlinux) +else +$(modules): %.ko: %.o %.mod.o scripts/module.lds FORCE + +$(call if_changed_except,ld_ko_o) +endif ifdef CONFIG_DEBUG_INFO_BTF_MODULES +$(if $(newer-prereqs),$(call cmd,btf_ko)) endif
scripts/Makefile.modpost +4 -0
@@ -114,8 +114,12 @@ targets += $(output-symdump) __modpost: $(output-symdump) ifneq ($(KBUILD_MODPOST_NOFINAL),1) +ifdef MODPOST_VMLINUX + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal MODPOST_VMLINUX=1 +else $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal endif +endif PHONY += FORCE FORCE:
--
2.28.0
Quoted 17 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
The out-of-tree modules are built without vmlinux target and request to recompile that target unconditionally causes to the following compilation error.
[root@server kernel]# make
<..>
make -f./scripts/Makefile.modpost
make -f./scripts/Makefile.modfinal
make[3]: *** No rule to make target 'vmlinux', needed by '/my_temp/out-of-tree-module/kernel/test.ko'. Stop. make[2]: *** [scripts/Makefile.modpost:117:__modpost] Error 2
make[1]: *** [Makefile:1703: modules] Error 2
make[1]: Leaving directory '/usr/src/kernels/5.10.0-rc5_for_upstream_base_2020_11_29_11_34'
make: *** [Makefile:80: modules] Error 2
Quoted 6 lines
As a solution separate between build paths that has vmlinux target and paths without.
Fixes 5f9ae91f7c0d ("kbuild: Build kernel module BTFs if BTF is enabled and pahole supports it")Reported-by Edward Srouji <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>---
e732b538f455 ("kbuild: Skip module BTF generation for out-of-tree external modules") ([0]) was supposed to take care of this. Did you try it?
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Quoted 45 lines
Not proficient enough in
Makefile, but it fixes the issue.--- scripts/Makefile.modfinal | 5 +++++ scripts/Makefile.modpost | 4 ++++ 2 files changed, 9 insertions(+)scripts/Makefile.modfinal +5
-0@@ -48,9 +48,14 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check), \ $(cmd); \ printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:) +ifdef MODPOST_VMLINUX # Re-generate module BTFs if either module's .ko or vmlinux changed $(modules): %.ko: %.o %.mod.o scripts/module.lds vmlinux FORCE +$(call if_changed_except,ld_ko_o,vmlinux) +else +$(modules): %.ko: %.o %.mod.o scripts/module.lds FORCE + +$(call if_changed_except,ld_ko_o) +endif ifdef CONFIG_DEBUG_INFO_BTF_MODULES +$(if $(newer-prereqs),$(call cmd,btf_ko)) endifscripts/Makefile.modpost +4
-1@@ -114,8 +114,12 @@ targets += $(output-symdump) __modpost: $(output-symdump) ifneq ($(KBUILD_MODPOST_NOFINAL),1) +ifdef MODPOST_VMLINUX + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal MODPOST_VMLINUX=1 +else $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal endif +endif PHONY += FORCE FORCE: --2.28.0
Quoted 20 lines · Andrii Nakryiko
Quoted 19 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
The out-of-tree modules are built without vmlinux target and request to recompile that target unconditionally causes to the following compilation error.
[root@server kernel]# make
<..>
make -f./scripts/Makefile.modpost
make -f./scripts/Makefile.modfinal
make[3]: *** No rule to make target 'vmlinux', needed by '/my_temp/out-of-tree-module/kernel/test.ko'. Stop. make[2]: *** [scripts/Makefile.modpost:117:__modpost] Error 2
make[1]: *** [Makefile:1703: modules] Error 2
make[1]: Leaving directory '/usr/src/kernels/5.10.0-rc5_for_upstream_base_2020_11_29_11_34'
make: *** [Makefile:80: modules] Error 2As a solution separate between build paths that has vmlinux target and paths without.
Quoted 8 lines
Quoted 4 lines
Fixes 5f9ae91f7c0d ("kbuild: Build kernel module BTFs if BTF is enabled and pahole supports it")Reported-by Edward Srouji <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>---
e732b538f455("kbuild: Skip module BTF generation for out-of-tree external modules") ([0]) was supposed to take care of this. Did you try it?
My tree doesn't have this patch yet, so my questions can be stupid:
1. Will it print "Skipping BTF generation for ... due to unavailability of vmlinux" line if my .config doesn't have "CONFIG_DEBUG_INFO_BTF_MODULES"? I hope it is not.
2. Reliance on existence of vmlinux can be problematic, no one promises us that "make clean" is called before and there are no other leftovers from previous builds.
And in general, the idea that such invasive change in build infrastructure came without any Ack from relevant maintainers doesn't look right to me.
Thanks
Quoted 49 lines
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Quoted 45 lines
Not proficient enough in
Makefile, but it fixes the issue.--- scripts/Makefile.modfinal | 5 +++++ scripts/Makefile.modpost | 4 ++++ 2 files changed, 9 insertions(+)scripts/Makefile.modfinal +5
-0@@ -48,9 +48,14 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check), \ $(cmd); \ printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:) +ifdef MODPOST_VMLINUX # Re-generate module BTFs if either module's .ko or vmlinux changed $(modules): %.ko: %.o %.mod.o scripts/module.lds vmlinux FORCE +$(call if_changed_except,ld_ko_o,vmlinux) +else +$(modules): %.ko: %.o %.mod.o scripts/module.lds FORCE + +$(call if_changed_except,ld_ko_o) +endif ifdef CONFIG_DEBUG_INFO_BTF_MODULES +$(if $(newer-prereqs),$(call cmd,btf_ko)) endifscripts/Makefile.modpost +4
-1@@ -114,8 +114,12 @@ targets += $(output-symdump) __modpost: $(output-symdump) ifneq ($(KBUILD_MODPOST_NOFINAL),1) +ifdef MODPOST_VMLINUX + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal MODPOST_VMLINUX=1 +else $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal endif +endif PHONY += FORCE FORCE: --2.28.0
Quoted 27 lines · Leon Romanovsky
Quoted 25 lines · Andrii Nakryiko
Quoted 23 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
The out-of-tree modules are built without vmlinux target and request to recompile that target unconditionally causes to the following compilation error.
[root@server kernel]# make
<..>
make -f./scripts/Makefile.modpost
make -f./scripts/Makefile.modfinal
make[3]: *** No rule to make target 'vmlinux', needed by '/my_temp/out-of-tree-module/kernel/test.ko'. Stop. make[2]: *** [scripts/Makefile.modpost:117:__modpost] Error 2
make[1]: *** [Makefile:1703: modules] Error 2
make[1]: Leaving directory '/usr/src/kernels/5.10.0-rc5_for_upstream_base_2020_11_29_11_34'
make: *** [Makefile:80: modules] Error 2As a solution separate between build paths that has vmlinux target and paths without.
Fixes 5f9ae91f7c0d ("kbuild: Build kernel module BTFs if BTF is enabled and pahole supports it")Reported-by Edward Srouji <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>---
Quoted 8 lines
Quoted 3 lines
e732b538f455("kbuild: Skip module BTF generation for out-of-tree external modules") ([0]) was supposed to take care of this. Did you try it?My tree doesn't have this patch yet, so my questions can be stupid:
1. Will it print "Skipping BTF generation for ... due to unavailability of vmlinux" line if my .config doesn't have "CONFIG_DEBUG_INFO_BTF_MODULES"? I hope it is not.
No, it shouldn't. cmd_btf_ko is only executed ifCONFIG_DEBUG_INFO_BTF_MODULES is set.
Quoted 3 lines
2. Reliance on existence of vmlinux can be problematic, no one promises us that "make clean" is called before and there are no other leftovers from previous builds.
In such a case, the worst thing that can happen would be that the kernel module will get BTF that doesn't match actual vmlinux BTF, and when attempted to load into the kernel BTF will be ignored (with a warning). It's not ideal, but I don't know how else we can handle this short of just not supporting BTF for out-of-tree modules, which a bunch of folks would be disappointed about, I think. I'm open to suggestions on how to do it better, though.
Quoted 55 lines
And in general, the idea that such invasive change in build infrastructure came without any Ack from relevant maintainers doesn't look right to me.
Thanks
Quoted 49 lines
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Quoted 45 lines
Not proficient enough in
Makefile, but it fixes the issue.--- scripts/Makefile.modfinal | 5 +++++ scripts/Makefile.modpost | 4 ++++ 2 files changed, 9 insertions(+)scripts/Makefile.modfinal +5
-0@@ -48,9 +48,14 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check), \ $(cmd); \ printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:) +ifdef MODPOST_VMLINUX # Re-generate module BTFs if either module's .ko or vmlinux changed $(modules): %.ko: %.o %.mod.o scripts/module.lds vmlinux FORCE +$(call if_changed_except,ld_ko_o,vmlinux) +else +$(modules): %.ko: %.o %.mod.o scripts/module.lds FORCE + +$(call if_changed_except,ld_ko_o) +endif ifdef CONFIG_DEBUG_INFO_BTF_MODULES +$(if $(newer-prereqs),$(call cmd,btf_ko)) endifscripts/Makefile.modpost +4
-1@@ -114,8 +114,12 @@ targets += $(output-symdump) __modpost: $(output-symdump) ifneq ($(KBUILD_MODPOST_NOFINAL),1) +ifdef MODPOST_VMLINUX + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal MODPOST_VMLINUX=1 +else $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal endif +endif PHONY += FORCE FORCE: --2.28.0
Quoted 44 lines · Andrii Nakryiko
Quoted 27 lines · Leon Romanovsky
Quoted 25 lines · Andrii Nakryiko
Quoted 23 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
The out-of-tree modules are built without vmlinux target and request to recompile that target unconditionally causes to the following compilation error.
[root@server kernel]# make
<..>
make -f./scripts/Makefile.modpost
make -f./scripts/Makefile.modfinal
make[3]: *** No rule to make target 'vmlinux', needed by '/my_temp/out-of-tree-module/kernel/test.ko'. Stop. make[2]: *** [scripts/Makefile.modpost:117:__modpost] Error 2
make[1]: *** [Makefile:1703: modules] Error 2
make[1]: Leaving directory '/usr/src/kernels/5.10.0-rc5_for_upstream_base_2020_11_29_11_34'
make: *** [Makefile:80: modules] Error 2As a solution separate between build paths that has vmlinux target and paths without.
Fixes 5f9ae91f7c0d ("kbuild: Build kernel module BTFs if BTF is enabled and pahole supports it")Reported-by Edward Srouji <[email protected]>Signed-off-by Leon Romanovsky <[email protected]>---
Quoted 8 lines
Quoted 3 lines
e732b538f455("kbuild: Skip module BTF generation for out-of-tree external modules") ([0]) was supposed to take care of this. Did you try it?My tree doesn't have this patch yet, so my questions can be stupid:
1. Will it print "Skipping BTF generation for ... due to unavailability of vmlinux" line if my .config doesn't have "CONFIG_DEBUG_INFO_BTF_MODULES"? I hope it is not.No, it shouldn't.
cmd_btf_kois only executed ifCONFIG_DEBUG_INFO_BTF_MODULESis set.Quoted 3 lines
2. Reliance on existence of vmlinux can be problematic, no one promises us that "make clean" is called before and there are no other leftovers from previous builds.
Quoted 7 lines
In such a case, the worst thing that can happen would be that the kernel module will get BTF that doesn't match actual vmlinux BTF, and when attempted to load into the kernel BTF will be ignored (with a warning). It's not ideal, but I don't know how else we can handle this short of just not supporting BTF for out-of-tree modules, which a bunch of folks would be disappointed about, I think. I'm open to suggestions on how to do it better, though.
You can rely on MODPOST_VMLINUX variable to differentiate between builds of modules vs. vmlinux. Proper message during kernel boot that vmlinux BTF doesn't match module can be enough, because in real world you won't load that recompiled vmlinux without "make install" first.
So right now, from out-of-tree modules perspective, you are testing against some pseudo-random file.
BTW, I tried your patch with and without vmlinux, it worked, so we are talking about better-to-fix and not must-to-fix thing.
Thanks
Quoted 57 lines
Quoted 57 lines
And in general, the idea that such invasive change in build infrastructure came without any Ack from relevant maintainers doesn't look right to me.
Thanks
Quoted 49 lines
[0] https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Quoted 45 lines
Not proficient enough in
Makefile, but it fixes the issue.--- scripts/Makefile.modfinal | 5 +++++ scripts/Makefile.modpost | 4 ++++ 2 files changed, 9 insertions(+)scripts/Makefile.modfinal +5
-0@@ -48,9 +48,14 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check), \ $(cmd); \ printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:) +ifdef MODPOST_VMLINUX # Re-generate module BTFs if either module's .ko or vmlinux changed $(modules): %.ko: %.o %.mod.o scripts/module.lds vmlinux FORCE +$(call if_changed_except,ld_ko_o,vmlinux) +else +$(modules): %.ko: %.o %.mod.o scripts/module.lds FORCE + +$(call if_changed_except,ld_ko_o) +endif ifdef CONFIG_DEBUG_INFO_BTF_MODULES +$(if $(newer-prereqs),$(call cmd,btf_ko)) endifscripts/Makefile.modpost +4
-1@@ -114,8 +114,12 @@ targets += $(output-symdump) __modpost: $(output-symdump) ifneq ($(KBUILD_MODPOST_NOFINAL),1) +ifdef MODPOST_VMLINUX + $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal MODPOST_VMLINUX=1 +else $(Q)$(MAKE) -f $(srctree)/scripts/Makefile.modfinal endif +endif PHONY += FORCE FORCE: --2.28.0