Skip to contentSkip to conversation

kbuild: move non-boot built-in DTBs to .rodata section

PATCH In review Masahiro Yamada 2 participants 4 messages Sep 23, 2024 – Sep 27, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH v3 1/2 kbuild: move non-boot built-in DTBs to .rodata section

Commit aab94339cd85 ("of: Add support for linking device tree blobs into vmlinux") introduced a mechanism to embed DTBs into vmlinux.

Initially, it was used for wrapping boot DTBs in arch/*/boot/dts/, but it is now reused for more generic purposes, such as testing.

Built-in DTBs are discarded because KERNEL_DTB() is part of INIT_DATA, as defined in include/asm-generic/vmlinux.lds.h.

This has not been an issue so far because OF unittests are triggered during boot, as defined by late_initcall(of_unittest).

However, the recent clk KUnit test additions have caused problems because KUnit can execute test suites after boot.

For example:

  # echo > /sys/kernel/debug/kunit/clk_register_clk_parent_data_device/run

This command triggers a stack trace because built-in DTBs have already been freed.

While it is possible to move such test suites from kunit_test_suites to kunit_test_init_section_suites, it would be preferable to avoid usage limitations.

This commit moves non-boot built-in DTBs to the .rodata section. Since these generic DTBs are looked up by name, they do not need to be placed in the special .dtb.init.rodata section.

Boot DTBs should remain in .dtb.init.rodata because the arch boot code generally does not know the DT name, thus it uses the __dtb_start symbol to locate it.

This separation also ensures that the __dtb_start symbol references the boot DTB. Currently, the .dtb.init.rodata is a mixture of both boot and non-boot DTBs. The __dtb_start symbol must be followed by the boot DTB, but we currently rely on the link order (i.e., the order in Makefiles), which is very fragile.

Fixes 5c9dd72d8385 ("of: Add a KUnit test for overlays and test managed APIs")
Fixes 5776526beb95 ("clk: Add KUnit tests for clk fixed rate basic type")
Fixes 274aff8711b2 ("clk: Add KUnit tests for clks registered with struct clk_parent_data")

---

Changes in v3:
- Move to .rodata section instead of .init.rodata

 scripts/Makefile.dtbs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
scripts/Makefile.dtbs +3 -1
index 46009d5f1486..3e81cca6f68a 100644
@@ -34,12 +34,14 @@ $(obj)/dtbs-list: $(dtb-y) FORCE
 # Assembly file to wrap dtb(o)
 # ---------------------------------------------------------------------------
 
+builtin-dtb-section = $(if $(filter arch/%, $(obj)),.dtb.init.rodata,.rodata)
+
 # Generate an assembly file to wrap the output of the device tree compiler
 quiet_cmd_wrap_S_dtb = WRAP    $@
       cmd_wrap_S_dtb = {								\
 		symbase=__$(patsubst .%,%,$(suffix $<))_$(subst -,_,$(notdir $*));	\
 		echo '\#include <asm-generic/vmlinux.lds.h>';				\
-		echo '.section .dtb.init.rodata,"a"';					\
+		echo '.section $(builtin-dtb-section),"a"';				\
 		echo '.balign STRUCT_ALIGNMENT';					\
 		echo ".global $${symbase}_begin";					\
 		echo "$${symbase}_begin:";						\
Signature · 1 line
2.43.0
#1 lore raw

PATCH v3 2/2 kbuild: add generic support for built-in boot DTBs

Some architectures embed boot DTBs in vmlinux. A potential issue for these architectures is a race condition during parallel builds because Kbuild descends into arch/*/boot/dts/ twice.

One build thread is initiated by the 'dtbs' target, which is a prerequisite of the 'all' target in the top-level Makefile:

ifdef CONFIG_OF_EARLY_FLATTREE
all: dtbs
endif

For architectures that support the built-in boot dtb, arch/*/boot/dts/ is visited also during the ordinary directory traversal in order to build obj-y objects that wrap DTBs.

Since these build threads are unaware of each other, they can run simultaneously during parallel builds.

This commit introduces a generic build rule to scripts/Makefile.vmlinux to support embedded boot DTBs in a race-free way. Architectures that want to use this rule need to select CONFIG_GENERIC_BUILTIN_DTB.

After the migration, Makefiles under arch/*/boot/dts/ will be visited only once to build only *.dtb files.

This change also aims to unify the CONFIG options used for built-in DTBs support. Currently, different architectures use different CONFIG options for the same purposes.

With this commit, the CONFIG options will be unified as follows:

- CONFIG_GENERIC_BUILTIN_DTB

This enables the generic rule for built-in boot DTBs. This will be renamed to CONFIG_BUILTIN_DTB after all architectures migrate to the generic rule.

- CONFIG_BUILTIN_DTB_NAME

This specifies the path to the embedded DTB.
(relative to arch/*/boot/dts/)

- CONFIG_BUILTIN_DTB_ALL

If this is enabled, all DTB files compiled under arch/*/boot/dts/ are embedded into vmlinux. Only used by MIPS.

---

(no changes since v1)

 Makefile                 |  7 ++++++-
 drivers/of/Kconfig       |  6 ++++++
 scripts/Makefile.vmlinux | 44 ++++++++++++++++++++++++++++++++++++++++
 scripts/link-vmlinux.sh  |  4 ++++
 4 files changed, 60 insertions(+), 1 deletion(-)
Makefile +6 -1
index 0d3d45a88a71..35b8392d2bef 100644
@@ -1413,6 +1413,10 @@ ifdef CONFIG_OF_EARLY_FLATTREE
 all: dtbs
 endif
 
+ifdef CONFIG_GENERIC_BUILTIN_DTB
+vmlinux: dtbs
+endif
+
 endif
 
 PHONY += scripts_dtc
@@ -1480,7 +1484,8 @@ CLEAN_FILES += vmlinux.symvers modules-only.symvers \
 	       modules.builtin modules.builtin.modinfo modules.nsdeps \
 	       modules.builtin.ranges vmlinux.o.map \
 	       compile_commands.json rust/test \
-	       rust-project.json .vmlinux.objs .vmlinux.export.c
+	       rust-project.json .vmlinux.objs .vmlinux.export.c \
+               .builtin-dtbs-list .builtin-dtb.S
 
 # Directories & files removed with 'make mrproper'
 MRPROPER_FILES += include/config include/generated          \
drivers/of/Kconfig +6 -0
index dd726c7056bf..5142e7d7fef8 100644
@@ -2,6 +2,12 @@
 config DTC
 	bool
 
+config GENERIC_BUILTIN_DTB
+	bool
+
+config BUILTIN_DTB_ALL
+	bool
+
 menuconfig OF
 	bool "Device Tree and Open Firmware support"
 	help
scripts/Makefile.vmlinux +44 -0

50 lines not on this page · raw message

scripts/link-vmlinux.sh +4 -0
index c27b4e969f20..bd196944e350 100755
@@ -68,6 +68,10 @@ vmlinux_link()
 		libs="${KBUILD_VMLINUX_LIBS}"
 	fi
 
+	if is_enabled CONFIG_GENERIC_BUILTIN_DTB; then
+		objs="${objs} .builtin-dtbs.o"
+	fi
+
 	if is_enabled CONFIG_MODULES; then
 		objs="${objs} .vmlinux.export.o"
 	fi
Signature · 1 line
2.43.0
maintainer Acked-by on v3 1/2
#2 lore raw
In reply to Masahiro Yamada
Quoted 44 lines · Masahiro Yamada

Commit aab94339cd85 ("of: Add support for linking device tree blobs into vmlinux") introduced a mechanism to embed DTBs into vmlinux.

Initially, it was used for wrapping boot DTBs in arch/*/boot/dts/, but it is now reused for more generic purposes, such as testing.

Built-in DTBs are discarded because KERNEL_DTB() is part of INIT_DATA, as defined in include/asm-generic/vmlinux.lds.h.

This has not been an issue so far because OF unittests are triggered during boot, as defined by late_initcall(of_unittest).

However, the recent clk KUnit test additions have caused problems because KUnit can execute test suites after boot.

For example:

  # echo > /sys/kernel/debug/kunit/clk_register_clk_parent_data_device/run

This command triggers a stack trace because built-in DTBs have already been freed.

While it is possible to move such test suites from kunit_test_suites to kunit_test_init_section_suites, it would be preferable to avoid usage limitations.

This commit moves non-boot built-in DTBs to the .rodata section. Since these generic DTBs are looked up by name, they do not need to be placed in the special .dtb.init.rodata section.

Boot DTBs should remain in .dtb.init.rodata because the arch boot code generally does not know the DT name, thus it uses the __dtb_start symbol to locate it.

This separation also ensures that the __dtb_start symbol references the boot DTB. Currently, the .dtb.init.rodata is a mixture of both boot and non-boot DTBs. The __dtb_start symbol must be followed by the boot DTB, but we currently rely on the link order (i.e., the order in Makefiles), which is very fragile.

Fixes 5c9dd72d8385 ("of: Add a KUnit test for overlays and test managed APIs")
Fixes 5776526beb95 ("clk: Add KUnit tests for clk fixed rate basic type")
Fixes 274aff8711b2 ("clk: Add KUnit tests for clks registered with struct clk_parent_data")
Quoted 7 lines

---

Changes in v3:
- Move to .rodata section instead of .init.rodata

 scripts/Makefile.dtbs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
#3 lore raw
In reply to Masahiro Yamada
Quoted 58 lines · Masahiro Yamada

Commit aab94339cd85 ("of: Add support for linking device tree blobs into vmlinux") introduced a mechanism to embed DTBs into vmlinux.

Initially, it was used for wrapping boot DTBs in arch/*/boot/dts/, but it is now reused for more generic purposes, such as testing.

Built-in DTBs are discarded because KERNEL_DTB() is part of INIT_DATA, as defined in include/asm-generic/vmlinux.lds.h.

This has not been an issue so far because OF unittests are triggered during boot, as defined by late_initcall(of_unittest).

However, the recent clk KUnit test additions have caused problems because KUnit can execute test suites after boot.

For example:

  # echo > /sys/kernel/debug/kunit/clk_register_clk_parent_data_device/run

This command triggers a stack trace because built-in DTBs have already been freed.

While it is possible to move such test suites from kunit_test_suites to kunit_test_init_section_suites, it would be preferable to avoid usage limitations.

This commit moves non-boot built-in DTBs to the .rodata section. Since these generic DTBs are looked up by name, they do not need to be placed in the special .dtb.init.rodata section.

Boot DTBs should remain in .dtb.init.rodata because the arch boot code generally does not know the DT name, thus it uses the __dtb_start symbol to locate it.

This separation also ensures that the __dtb_start symbol references the boot DTB. Currently, the .dtb.init.rodata is a mixture of both boot and non-boot DTBs. The __dtb_start symbol must be followed by the boot DTB, but we currently rely on the link order (i.e., the order in Makefiles), which is very fragile.

Fixes 5c9dd72d8385 ("of: Add a KUnit test for overlays and test managed APIs")
Fixes 5776526beb95 ("clk: Add KUnit tests for clk fixed rate basic type")
Fixes 274aff8711b2 ("clk: Add KUnit tests for clks registered with struct clk_parent_data")

---

Changes in v3:
- Move to .rodata section instead of .init.rodata

 scripts/Makefile.dtbs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
scripts/Makefile.dtbs +0 -0
index 46009d5f1486..3e81cca6f68a 100644

scripts/Makefile.dtbs $(obj)/dtbs-list: $(dtb-y) FORCE · line 37

Quoted 4 lines
scripts/Makefile.dtbs +1 -0
@@ -34,12 +34,14 @@ $(obj)/dtbs-list: $(dtb-y) FORCE
 # Assembly file to wrap dtb(o)
 # ---------------------------------------------------------------------------
 
+builtin-dtb-section = $(if $(filter arch/%, $(obj)),.dtb.init.rodata,.rodata)

For more precise matching, I will change as follows:

scripts/Makefile.dtbs +1 -1
index 3e81cca6f68a..8d56c0815f33 100644
@@ -34,7 +34,7 @@ $(obj)/dtbs-list: $(dtb-y) FORCE
 # Assembly file to wrap dtb(o)
 # ---------------------------------------------------------------------------
 
-builtin-dtb-section = $(if $(filter arch/%, $(obj)),.dtb.init.rodata,.rodata)
+builtin-dtb-section = $(if $(filter arch/$(SRCARCH)/boot/dts%,

$(obj)),.dtb.init.rodata,.rodata)

# Generate an assembly file to wrap the output of the device tree compiler
quiet_cmd_wrap_S_dtb = WRAP $@

Applied to linux-kbuild/fixes.

Quoted 14 lines
scripts/Makefile.dtbs +2 -2
+
 # Generate an assembly file to wrap the output of the device tree compiler
 quiet_cmd_wrap_S_dtb = WRAP    $@
       cmd_wrap_S_dtb = {                                                               \
                symbase=__$(patsubst .%,%,$(suffix $<))_$(subst -,_,$(notdir $*));      \
                echo '\#include <asm-generic/vmlinux.lds.h>';                           \
-               echo '.section .dtb.init.rodata,"a"';                                   \
+               echo '.section $(builtin-dtb-section),"a"';                             \
                echo '.balign STRUCT_ALIGNMENT';                                        \
                echo ".global $${symbase}_begin";                                       \
                echo "$${symbase}_begin:";                                              \
--

2.43.0

Signature · 2 lines
Best Regards
Masahiro Yamada