Skip to contentSkip to conversation

ARM: decompressor: simplify libfdt builds

PATCH Superseded Masahiro Yamada 4 participants 7 messages Apr 19, 2020 – Apr 23, 2020 Arm (32-bit)

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 ARM: decompressor: simplify libfdt builds

Copying source files during the build time may not end up with as clean code as expected.

lib/fdt*.c simply wrap scripts/dtc/libfdt/fdt*.c, and it works nicely. Let's follow this approach for the arm decompressor, too.

Add four wrappers, arch/arm/boot/compressed/fdt*.c and remove the Makefile messes. Another nice thing is we no longer need to maintain the own libfdt_env.h because the decompressor can include <linux/libfdt_env.h>.

There is a subtle problem when generated files are turned into check-in files.

When you are doing a rebuild of an existing object tree with O= option, there exists stale "shipped" copies that the old Makefile implementation created. The build system ends up with compiling the stale generated files because Make searches for prerequisites in the current directory, i.e. $(objtree) first, and then the directory listed in VPATH, i.e. $(srctree).

To mend this issue, I added the following code:

  ifdef building_out_of_srctree
  $(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
  endif

This will need to stay for a while because "git bisect" crossing this commit, otherwise, would result in a build error.

---

KernelVersion: v5.7-rc1

Changes in v4:
- rebased on v5.7-rc1

Changes in v3:
- remove stale generated

Changes in v2:
- fix build error

 arch/arm/boot/compressed/.gitignore     |  9 ------
 arch/arm/boot/compressed/Makefile       | 39 ++++++++++---------------
 arch/arm/boot/compressed/atags_to_fdt.c |  1 +
 arch/arm/boot/compressed/fdt.c          |  2 ++
 arch/arm/boot/compressed/fdt_ro.c       |  2 ++
 arch/arm/boot/compressed/fdt_rw.c       |  2 ++
 arch/arm/boot/compressed/fdt_wip.c      |  2 ++
 arch/arm/boot/compressed/libfdt_env.h   | 24 ---------------
 8 files changed, 25 insertions(+), 56 deletions(-)
 create mode 100644 arch/arm/boot/compressed/fdt.c
 create mode 100644 arch/arm/boot/compressed/fdt_ro.c
 create mode 100644 arch/arm/boot/compressed/fdt_rw.c
 create mode 100644 arch/arm/boot/compressed/fdt_wip.c
 delete mode 100644 arch/arm/boot/compressed/libfdt_env.h
arch/arm/boot/compressed/.gitignore +0 -9
index db05c6ef3e31..60606b0f378d 100644
@@ -7,12 +7,3 @@ hyp-stub.S
 piggy_data
 vmlinux
 vmlinux.lds
-
-# borrowed libfdt files
-fdt.c
-fdt.h
-fdt_ro.c
-fdt_rw.c
-fdt_wip.c
-libfdt.h
-libfdt_internal.h
arch/arm/boot/compressed/Makefile +16 -23
index 9c11e7490292..e5d58a49327e 100644
@@ -76,29 +76,31 @@ compress-$(CONFIG_KERNEL_LZMA) = lzma
 compress-$(CONFIG_KERNEL_XZ)   = xzkern
 compress-$(CONFIG_KERNEL_LZ4)  = lz4
 
-# Borrowed libfdt files for the ATAG compatibility mode
-
-libfdt		:= fdt_rw.c fdt_ro.c fdt_wip.c fdt.c
-libfdt_hdrs	:= fdt.h libfdt.h libfdt_internal.h
-
-libfdt_objs	:= $(addsuffix .o, $(basename $(libfdt)))
+ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
+libfdt_objs = fdt_rw.o fdt_ro.o fdt_wip.o fdt.o atags_to_fdt.o
 
-$(addprefix $(obj)/,$(libfdt) $(libfdt_hdrs)): $(obj)/%: $(srctree)/scripts/dtc/libfdt/%
-	$(call cmd,shipped)
+OBJS	+= $(libfdt_objs)
 
-$(addprefix $(obj)/,$(libfdt_objs) atags_to_fdt.o): \
-	$(addprefix $(obj)/,$(libfdt_hdrs))
+# -fstack-protector-strong triggers protection checks in this code,
+# but it is being used too early to link to meaningful stack_chk logic.
+nossp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
+$(foreach o, $(libfdt_objs), \
+	$(eval CFLAGS_$(o) := -I $(srctree)/scripts/dtc/libfdt $(nossp-flags-y)))
+
+# These were previously generated C files. When you are building the kernel
+# with O=, make sure to remove the stale files in the output tree. Otherwise,
+# the build system wrongly compiles the stale ones.
+ifdef building_out_of_srctree
+$(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
+endif
 
-ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
-OBJS	+= $(libfdt_objs) atags_to_fdt.o
 endif
 
 targets       := vmlinux vmlinux.lds piggy_data piggy.o \
 		 lib1funcs.o ashldi3.o bswapsdi2.o \
 		 head.o $(OBJS)
 
-clean-files += piggy_data lib1funcs.S ashldi3.S bswapsdi2.S \
-		$(libfdt) $(libfdt_hdrs) hyp-stub.S
+clean-files += piggy_data lib1funcs.S ashldi3.S bswapsdi2.S hyp-stub.S
 
 KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING
 
@@ -107,15 +109,6 @@ ORIG_CFLAGS := $(KBUILD_CFLAGS)
 KBUILD_CFLAGS = $(subst -pg, , $(ORIG_CFLAGS))
 endif
 
-# -fstack-protector-strong triggers protection checks in this code,
-# but it is being used too early to link to meaningful stack_chk logic.
-nossp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
-CFLAGS_atags_to_fdt.o := $(nossp-flags-y)
-CFLAGS_fdt.o := $(nossp-flags-y)
-CFLAGS_fdt_ro.o := $(nossp-flags-y)
-CFLAGS_fdt_rw.o := $(nossp-flags-y)
-CFLAGS_fdt_wip.o := $(nossp-flags-y)
-
 ccflags-y := -fpic $(call cc-option,-mno-single-pic-base,) -fno-builtin \
 	     -I$(obj) $(DISABLE_ARM_SSP_PER_TASK_PLUGIN)
 asflags-y := -DZIMAGE
arch/arm/boot/compressed/atags_to_fdt.c +1 -0
index 64c49747f8a3..8452753efebe 100644
@@ -1,4 +1,5 @@
 // SPDX-License-Identifier: GPL-2.0
+#include <linux/libfdt_env.h>
 #include <asm/setup.h>
 #include <libfdt.h>
 
arch/arm/boot/compressed/fdt.c new file +2 -0
new file mode 100644
index 000000000000..f8ea7a201ab1
@@ -0,0 +1,2 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include "../../../../lib/fdt.c"
arch/arm/boot/compressed/fdt_ro.c new file +2 -0
new file mode 100644
index 000000000000..93970a4ad5ae
@@ -0,0 +1,2 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include "../../../../lib/fdt_ro.c"
arch/arm/boot/compressed/fdt_rw.c new file +2 -0
new file mode 100644
index 000000000000..f7c6b8b7e01c
@@ -0,0 +1,2 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include "../../../../lib/fdt_rw.c"
arch/arm/boot/compressed/fdt_wip.c new file +2 -0
new file mode 100644
index 000000000000..048d2c7a088d
@@ -0,0 +1,2 @@
+// SPDX-License-Identifier: GPL-2.0-only
+#include "../../../../lib/fdt_wip.c"
arch/arm/boot/compressed/libfdt_env.h deleted +0 -24
deleted file mode 100644
index 6a0f1f524466..000000000000
@@ -1,24 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ARM_LIBFDT_ENV_H
-#define _ARM_LIBFDT_ENV_H
-
-#include <linux/limits.h>
-#include <linux/types.h>
-#include <linux/string.h>
-#include <asm/byteorder.h>
-
-#define INT32_MAX	S32_MAX
-#define UINT32_MAX	U32_MAX
-
-typedef __be16 fdt16_t;
-typedef __be32 fdt32_t;
-typedef __be64 fdt64_t;
-
-#define fdt16_to_cpu(x)		be16_to_cpu(x)
-#define cpu_to_fdt16(x)		cpu_to_be16(x)
-#define fdt32_to_cpu(x)		be32_to_cpu(x)
-#define cpu_to_fdt32(x)		cpu_to_be32(x)
-#define fdt64_to_cpu(x)		be64_to_cpu(x)
-#define cpu_to_fdt64(x)		cpu_to_be64(x)
-
-#endif
Signature · 1 line
2.25.1
#1 lore raw

Hi Yamada-san, Kees, Russell,

-CC RMK's patch system
+CC lakml

Quoted 23 lines · Masahiro Yamada

Copying source files during the build time may not end up with as clean code as expected.

lib/fdt*.c simply wrap scripts/dtc/libfdt/fdt*.c, and it works nicely. Let's follow this approach for the arm decompressor, too.

Add four wrappers, arch/arm/boot/compressed/fdt*.c and remove the Makefile messes. Another nice thing is we no longer need to maintain the own libfdt_env.h because the decompressor can include <linux/libfdt_env.h>.

There is a subtle problem when generated files are turned into check-in files.

When you are doing a rebuild of an existing object tree with O= option, there exists stale "shipped" copies that the old Makefile implementation created. The build system ends up with compiling the stale generated files because Make searches for prerequisites in the current directory, i.e. $(objtree) first, and then the directory listed in VPATH, i.e. $(srctree).

To mend this issue, I added the following code:

Quoted 8 lines
  ifdef building_out_of_srctree
  $(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
  endif

This will need to stay for a while because "git bisect" crossing this commit, otherwise, would result in a build error.

This is now commit 365a6327cd643eed ("ARM: 8968/1: decompressor:
simplify libfdt builds") in arm/for-next.

In light of reworking "[PATCH v5] ARM: boot: Obtain start of physical memory from DTB"[1] on top of this, which would conditionally add another source file to libfdt_objs, I have a few questions.

Quoted 5 lines
arch/arm/boot/compressed/Makefile +0 -0
@@ -76,29 +76,31 @@ compress-$(CONFIG_KERNEL_LZMA) = lzma
 compress-$(CONFIG_KERNEL_XZ)   = xzkern
 compress-$(CONFIG_KERNEL_LZ4)  = lz4

arch/arm/boot/compressed/Makefile compress-$(CONFIG_KERNEL_LZMA) = lzma · line 81

Quoted 10 lines
arch/arm/boot/compressed/Makefile +2 -6
 
-# Borrowed libfdt files for the ATAG compatibility mode
-
-libfdt         := fdt_rw.c fdt_ro.c fdt_wip.c fdt.c
-libfdt_hdrs    := fdt.h libfdt.h libfdt_internal.h
-
-libfdt_objs    := $(addsuffix .o, $(basename $(libfdt)))
+ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
+libfdt_objs = fdt_rw.o fdt_ro.o fdt_wip.o fdt.o atags_to_fdt.o
 

I guess the code below can be moved out of the ifeq block, as it doesn't really do anything if CONFIG_ARM_ATAG_DTB_COMPAT=n, and $(libfdt_objs) becomes empty?
If not, I think I'll have to add a new Kconfig symbol ARM_BOOT_LIBFDT, to be selected by ARM_ATAG_DTB_COMPAT and USE_OF.

arch/arm/boot/compressed/Makefile compress-$(CONFIG_KERNEL_LZMA) = lzma · line 88

Quoted 11 lines
arch/arm/boot/compressed/Makefile +6 -4
-$(addprefix $(obj)/,$(libfdt) $(libfdt_hdrs)): $(obj)/%: $(srctree)/scripts/dtc/libfdt/%
-       $(call cmd,shipped)
+OBJS   += $(libfdt_objs)
 
-$(addprefix $(obj)/,$(libfdt_objs) atags_to_fdt.o): \
-       $(addprefix $(obj)/,$(libfdt_hdrs))
+# -fstack-protector-strong triggers protection checks in this code,
+# but it is being used too early to link to meaningful stack_chk logic.
+nossp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
+$(foreach o, $(libfdt_objs), \
+       $(eval CFLAGS_$(o) := -I $(srctree)/scripts/dtc/libfdt $(nossp-flags-y)))

Is there a real reason this is only applied to a subset of the C object files, and not to all of them? Or have we been lucky so far, by not triggering the issue in decompressed.c, misc.c, and string.c (yet)?

Thanks!

Quoted 28 lines
arch/arm/boot/compressed/Makefile +8 -6
+
+# These were previously generated C files. When you are building the kernel
+# with O=, make sure to remove the stale files in the output tree. Otherwise,
+# the build system wrongly compiles the stale ones.
+ifdef building_out_of_srctree
+$(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
+endif
 
-ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
-OBJS   += $(libfdt_objs) atags_to_fdt.o
 endif
 
 targets       := vmlinux vmlinux.lds piggy_data piggy.o \
                 lib1funcs.o ashldi3.o bswapsdi2.o \
                 head.o $(OBJS)
 
-clean-files += piggy_data lib1funcs.S ashldi3.S bswapsdi2.S \
-               $(libfdt) $(libfdt_hdrs) hyp-stub.S
+clean-files += piggy_data lib1funcs.S ashldi3.S bswapsdi2.S hyp-stub.S
 
 KBUILD_CFLAGS += -DDISABLE_BRANCH_PROFILING
 
@@ -107,15 +109,6 @@ ORIG_CFLAGS := $(KBUILD_CFLAGS)
 KBUILD_CFLAGS = $(subst -pg, , $(ORIG_CFLAGS))
 endif
 
-# -fstack-protector-strong triggers protection checks in this code,
-# but it is being used too early to link to meaningful stack_chk logic.

arch/arm/boot/compressed/Makefile ORIG_CFLAGS := $(KBUILD_CFLAGS) · line 114

Quoted 10 lines
arch/arm/boot/compressed/Makefile +0 -7
-nossp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
-CFLAGS_atags_to_fdt.o := $(nossp-flags-y)
-CFLAGS_fdt.o := $(nossp-flags-y)
-CFLAGS_fdt_ro.o := $(nossp-flags-y)
-CFLAGS_fdt_rw.o := $(nossp-flags-y)
-CFLAGS_fdt_wip.o := $(nossp-flags-y)
-
 ccflags-y := -fpic $(call cc-option,-mno-single-pic-base,) -fno-builtin \
             -I$(obj) $(DISABLE_ARM_SSP_PER_TASK_PLUGIN)
 asflags-y := -DZIMAGE

[1] https://lore.kernel.org/r/[email protected]

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#2 lore raw
Quoted 71 lines · Geert Uytterhoeven

Hi Yamada-san, Kees, Russell,

(quoted diff) +1 -1
-CC RMK's patch system
+CC lakml
Quoted 23 lines · Masahiro Yamada

Copying source files during the build time may not end up with as clean code as expected.

lib/fdt*.c simply wrap scripts/dtc/libfdt/fdt*.c, and it works nicely. Let's follow this approach for the arm decompressor, too.

Add four wrappers, arch/arm/boot/compressed/fdt*.c and remove the Makefile messes. Another nice thing is we no longer need to maintain the own libfdt_env.h because the decompressor can include <linux/libfdt_env.h>.

There is a subtle problem when generated files are turned into check-in files.

When you are doing a rebuild of an existing object tree with O= option, there exists stale "shipped" copies that the old Makefile implementation created. The build system ends up with compiling the stale generated files because Make searches for prerequisites in the current directory, i.e. $(objtree) first, and then the directory listed in VPATH, i.e. $(srctree).

To mend this issue, I added the following code:

Quoted 8 lines
  ifdef building_out_of_srctree
  $(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
  endif

This will need to stay for a while because "git bisect" crossing this commit, otherwise, would result in a build error.

This is now commit 365a6327cd643eed ("ARM: 8968/1: decompressor:
simplify libfdt builds") in arm/for-next.

In light of reworking "[PATCH v5] ARM: boot: Obtain start of physical memory from DTB"[1] on top of this, which would conditionally add another source file to libfdt_objs, I have a few questions.

Quoted 5 lines
arch/arm/boot/compressed/Makefile +0 -0
@@ -76,29 +76,31 @@ compress-$(CONFIG_KERNEL_LZMA) = lzma
 compress-$(CONFIG_KERNEL_XZ)   = xzkern
 compress-$(CONFIG_KERNEL_LZ4)  = lz4

arch/arm/boot/compressed/Makefile compress-$(CONFIG_KERNEL_LZMA) = lzma · line 81

Quoted 10 lines
arch/arm/boot/compressed/Makefile +2 -6
 
-# Borrowed libfdt files for the ATAG compatibility mode
-
-libfdt         := fdt_rw.c fdt_ro.c fdt_wip.c fdt.c
-libfdt_hdrs    := fdt.h libfdt.h libfdt_internal.h
-
-libfdt_objs    := $(addsuffix .o, $(basename $(libfdt)))
+ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
+libfdt_objs = fdt_rw.o fdt_ro.o fdt_wip.o fdt.o atags_to_fdt.o
 

I guess the code below can be moved out of the ifeq block, as it doesn't really do anything if CONFIG_ARM_ATAG_DTB_COMPAT=n, and $(libfdt_objs) becomes empty?
If not, I think I'll have to add a new Kconfig symbol ARM_BOOT_LIBFDT, to be selected by ARM_ATAG_DTB_COMPAT and USE_OF.

Quoted 4 lines
arch/arm/boot/compressed/Makefile +1 -2
-$(addprefix $(obj)/,$(libfdt) $(libfdt_hdrs)): $(obj)/%: $(srctree)/scripts/dtc/libfdt/%
-       $(call cmd,shipped)
+OBJS   += $(libfdt_objs)
 
Quoted 11 lines

arch/arm/boot/compressed/Makefile compress-$(CONFIG_KERNEL_LZMA) = lzma · line 88

Quoted 7 lines
arch/arm/boot/compressed/Makefile +5 -2
-$(addprefix $(obj)/,$(libfdt_objs) atags_to_fdt.o): \
-       $(addprefix $(obj)/,$(libfdt_hdrs))
+# -fstack-protector-strong triggers protection checks in this code,
+# but it is being used too early to link to meaningful stack_chk logic.
+nossp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
+$(foreach o, $(libfdt_objs), \
+       $(eval CFLAGS_$(o) := -I $(srctree)/scripts/dtc/libfdt $(nossp-flags-y)))

Is there a real reason this is only applied to a subset of the C object files, and not to all of them? Or have we been lucky so far, by not triggering the issue in decompressed.c, misc.c, and string.c (yet)?

I don't remember the details. See commit 7f66cd3f5420, which came from Kees which introduced this.

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
#4 lore raw
In reply to Geert Uytterhoeven
Quoted 52 lines · Geert Uytterhoeven

Hi Yamada-san, Kees, Russell,

(quoted diff) +1 -1
-CC RMK's patch system
+CC lakml
Quoted 23 lines · Masahiro Yamada

Copying source files during the build time may not end up with as clean code as expected.

lib/fdt*.c simply wrap scripts/dtc/libfdt/fdt*.c, and it works nicely. Let's follow this approach for the arm decompressor, too.

Add four wrappers, arch/arm/boot/compressed/fdt*.c and remove the Makefile messes. Another nice thing is we no longer need to maintain the own libfdt_env.h because the decompressor can include <linux/libfdt_env.h>.

There is a subtle problem when generated files are turned into check-in files.

When you are doing a rebuild of an existing object tree with O= option, there exists stale "shipped" copies that the old Makefile implementation created. The build system ends up with compiling the stale generated files because Make searches for prerequisites in the current directory, i.e. $(objtree) first, and then the directory listed in VPATH, i.e. $(srctree).

To mend this issue, I added the following code:

Quoted 8 lines
  ifdef building_out_of_srctree
  $(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
  endif

This will need to stay for a while because "git bisect" crossing this commit, otherwise, would result in a build error.

This is now commit 365a6327cd643eed ("ARM: 8968/1: decompressor:
simplify libfdt builds") in arm/for-next.

In light of reworking "[PATCH v5] ARM: boot: Obtain start of physical memory from DTB"[1] on top of this, which would conditionally add another source file to libfdt_objs, I have a few questions.

Quoted 6 lines
arch/arm/boot/compressed/Makefile +0 -0
@@ -76,29 +76,31 @@ compress-$(CONFIG_KERNEL_LZMA) = lzma
 compress-$(CONFIG_KERNEL_XZ)   = xzkern
 compress-$(CONFIG_KERNEL_LZ4)  = lz4
 
Quoted 15 lines

arch/arm/boot/compressed/Makefile compress-$(CONFIG_KERNEL_LZMA) = lzma · line 81

Quoted 9 lines
arch/arm/boot/compressed/Makefile +2 -6
-# Borrowed libfdt files for the ATAG compatibility mode
-
-libfdt         := fdt_rw.c fdt_ro.c fdt_wip.c fdt.c
-libfdt_hdrs    := fdt.h libfdt.h libfdt_internal.h
-
-libfdt_objs    := $(addsuffix .o, $(basename $(libfdt)))
+ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
+libfdt_objs = fdt_rw.o fdt_ro.o fdt_wip.o fdt.o atags_to_fdt.o
 

I guess the code below can be moved out of the ifeq block, as it doesn't really do anything if CONFIG_ARM_ATAG_DTB_COMPAT=n, and $(libfdt_objs) becomes empty?
If not, I think I'll have to add a new Kconfig symbol ARM_BOOT_LIBFDT, to be selected by ARM_ATAG_DTB_COMPAT and USE_OF.

Right. We can narrow the ifeq block.
I did not know your on-going work.

If I had known your work adding one more file here, I would have written this part as follows:

------------------------------>8---------------------------------- libfdt_objs := fdt_rw.o fdt_ro.o fdt_wip.o fdt.o

ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
OBJS += $(libfdt_objs) atags_to_fdt.o
endif

# -fstack-protector-strong triggers protection checks in this code,

# but it is being used too early to link to meaningful stack_chk logic.
nossp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
$(foreach o, $(libfdt_objs) atags_to_fdt.o, \
$(eval CFLAGS_$(o) := -I $(srctree)/scripts/dtc/libfdt $(nossp-flags-y)))

# These were previously generated C files. When you are building the kernel

# with O=, make sure to remove the stale files in the output tree. Otherwise,
# the build system wrongly compiles the stale ones.
ifdef building_out_of_srctree
$(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
endif
-------------------------------------->8-----------------------------

So, how shall we move forward?

Leave the necessary Makefile change to Geert?

If Geert and Russell want to replace my patch,
I can send v5 with the code above.

Signature · 7 lines
Best Regards
Masahiro Yamada

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
#5 lore raw
Quoted 103 lines · Masahiro Yamada
Quoted 52 lines · Geert Uytterhoeven

Hi Yamada-san, Kees, Russell,

(quoted diff) +1 -1
-CC RMK's patch system
+CC lakml
Quoted 23 lines · Masahiro Yamada

Copying source files during the build time may not end up with as clean code as expected.

lib/fdt*.c simply wrap scripts/dtc/libfdt/fdt*.c, and it works nicely. Let's follow this approach for the arm decompressor, too.

Add four wrappers, arch/arm/boot/compressed/fdt*.c and remove the Makefile messes. Another nice thing is we no longer need to maintain the own libfdt_env.h because the decompressor can include <linux/libfdt_env.h>.

There is a subtle problem when generated files are turned into check-in files.

When you are doing a rebuild of an existing object tree with O= option, there exists stale "shipped" copies that the old Makefile implementation created. The build system ends up with compiling the stale generated files because Make searches for prerequisites in the current directory, i.e. $(objtree) first, and then the directory listed in VPATH, i.e. $(srctree).

To mend this issue, I added the following code:

Quoted 8 lines
  ifdef building_out_of_srctree
  $(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
  endif

This will need to stay for a while because "git bisect" crossing this commit, otherwise, would result in a build error.

This is now commit 365a6327cd643eed ("ARM: 8968/1: decompressor:
simplify libfdt builds") in arm/for-next.

In light of reworking "[PATCH v5] ARM: boot: Obtain start of physical memory from DTB"[1] on top of this, which would conditionally add another source file to libfdt_objs, I have a few questions.

Quoted 6 lines
arch/arm/boot/compressed/Makefile +0 -0
@@ -76,29 +76,31 @@ compress-$(CONFIG_KERNEL_LZMA) = lzma
 compress-$(CONFIG_KERNEL_XZ)   = xzkern
 compress-$(CONFIG_KERNEL_LZ4)  = lz4
 
Quoted 15 lines

arch/arm/boot/compressed/Makefile compress-$(CONFIG_KERNEL_LZMA) = lzma · line 81

Quoted 9 lines
arch/arm/boot/compressed/Makefile +2 -6
-# Borrowed libfdt files for the ATAG compatibility mode
-
-libfdt         := fdt_rw.c fdt_ro.c fdt_wip.c fdt.c
-libfdt_hdrs    := fdt.h libfdt.h libfdt_internal.h
-
-libfdt_objs    := $(addsuffix .o, $(basename $(libfdt)))
+ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
+libfdt_objs = fdt_rw.o fdt_ro.o fdt_wip.o fdt.o atags_to_fdt.o
 

I guess the code below can be moved out of the ifeq block, as it doesn't really do anything if CONFIG_ARM_ATAG_DTB_COMPAT=n, and $(libfdt_objs) becomes empty?
If not, I think I'll have to add a new Kconfig symbol ARM_BOOT_LIBFDT, to be selected by ARM_ATAG_DTB_COMPAT and USE_OF.

Right. We can narrow the ifeq block.
I did not know your on-going work.

If I had known your work adding one more file here, I would have written this part as follows:

------------------------------>8---------------------------------- libfdt_objs := fdt_rw.o fdt_ro.o fdt_wip.o fdt.o

ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
OBJS += $(libfdt_objs) atags_to_fdt.o
endif

# -fstack-protector-strong triggers protection checks in this code,

# but it is being used too early to link to meaningful stack_chk logic.
nossp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
$(foreach o, $(libfdt_objs) atags_to_fdt.o, \
$(eval CFLAGS_$(o) := -I $(srctree)/scripts/dtc/libfdt $(nossp-flags-y)))

# These were previously generated C files. When you are building the kernel

# with O=, make sure to remove the stale files in the output tree. Otherwise,
# the build system wrongly compiles the stale ones.
ifdef building_out_of_srctree
$(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
endif
-------------------------------------->8-----------------------------
Quoted 6 lines

So, how shall we move forward?

Leave the necessary Makefile change to Geert?

If Geert and Russell want to replace my patch,
I can send v5 with the code above.

I've no problem with replacing your patch...

Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 10.2Mbps down 587kbps up
#6 lore raw
In reply to Masahiro Yamada

Hi Yamada-san,

Quoted 86 lines · Masahiro Yamada
Quoted 46 lines · Geert Uytterhoeven
Quoted 23 lines · Masahiro Yamada

Copying source files during the build time may not end up with as clean code as expected.

lib/fdt*.c simply wrap scripts/dtc/libfdt/fdt*.c, and it works nicely. Let's follow this approach for the arm decompressor, too.

Add four wrappers, arch/arm/boot/compressed/fdt*.c and remove the Makefile messes. Another nice thing is we no longer need to maintain the own libfdt_env.h because the decompressor can include <linux/libfdt_env.h>.

There is a subtle problem when generated files are turned into check-in files.

When you are doing a rebuild of an existing object tree with O= option, there exists stale "shipped" copies that the old Makefile implementation created. The build system ends up with compiling the stale generated files because Make searches for prerequisites in the current directory, i.e. $(objtree) first, and then the directory listed in VPATH, i.e. $(srctree).

To mend this issue, I added the following code:

Quoted 8 lines
  ifdef building_out_of_srctree
  $(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
  endif

This will need to stay for a while because "git bisect" crossing this commit, otherwise, would result in a build error.

This is now commit 365a6327cd643eed ("ARM: 8968/1: decompressor:
simplify libfdt builds") in arm/for-next.

In light of reworking "[PATCH v5] ARM: boot: Obtain start of physical memory from DTB"[1] on top of this, which would conditionally add another source file to libfdt_objs, I have a few questions.

Quoted 6 lines
arch/arm/boot/compressed/Makefile +0 -0
@@ -76,29 +76,31 @@ compress-$(CONFIG_KERNEL_LZMA) = lzma
 compress-$(CONFIG_KERNEL_XZ)   = xzkern
 compress-$(CONFIG_KERNEL_LZ4)  = lz4
 
Quoted 15 lines

arch/arm/boot/compressed/Makefile compress-$(CONFIG_KERNEL_LZMA) = lzma · line 81

Quoted 9 lines
arch/arm/boot/compressed/Makefile +2 -6
-# Borrowed libfdt files for the ATAG compatibility mode
-
-libfdt         := fdt_rw.c fdt_ro.c fdt_wip.c fdt.c
-libfdt_hdrs    := fdt.h libfdt.h libfdt_internal.h
-
-libfdt_objs    := $(addsuffix .o, $(basename $(libfdt)))
+ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
+libfdt_objs = fdt_rw.o fdt_ro.o fdt_wip.o fdt.o atags_to_fdt.o
 

I guess the code below can be moved out of the ifeq block, as it doesn't really do anything if CONFIG_ARM_ATAG_DTB_COMPAT=n, and $(libfdt_objs) becomes empty?
If not, I think I'll have to add a new Kconfig symbol ARM_BOOT_LIBFDT, to be selected by ARM_ATAG_DTB_COMPAT and USE_OF.

Right. We can narrow the ifeq block.
I did not know your on-going work.

If I had known your work adding one more file here, I would have written this part as follows:

------------------------------>8---------------------------------- libfdt_objs := fdt_rw.o fdt_ro.o fdt_wip.o fdt.o

ifeq ($(CONFIG_ARM_ATAG_DTB_COMPAT),y)
OBJS += $(libfdt_objs) atags_to_fdt.o
endif

# -fstack-protector-strong triggers protection checks in this code,

# but it is being used too early to link to meaningful stack_chk logic.
nossp-flags-$(CONFIG_CC_HAS_STACKPROTECTOR_NONE) := -fno-stack-protector
$(foreach o, $(libfdt_objs) atags_to_fdt.o, \
$(eval CFLAGS_$(o) := -I $(srctree)/scripts/dtc/libfdt $(nossp-flags-y)))
Quoted 7 lines

# These were previously generated C files. When you are building the kernel

# with O=, make sure to remove the stale files in the output tree. Otherwise,
# the build system wrongly compiles the stale ones.
ifdef building_out_of_srctree
$(shell rm -f $(addprefix $(obj)/, fdt_rw.c fdt_ro.c fdt_wip.c fdt.c))
endif
-------------------------------------->8-----------------------------

Thanks, looks good to me!

Quoted 6 lines

So, how shall we move forward?

Leave the necessary Makefile change to Geert?

If Geert and Russell want to replace my patch,
I can send v5 with the code above.

I can fix it myself when rebasing my patch, unless you get to a v5 before me. I just wanted to find a good approach, to avoid delaying my patch.

Gr{oetje,eeting}s,

Geert

Signature · 5 lines
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- [email protected]

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds
#3 lore raw
In reply to Russell King - ARM Linux admin
Quoted 7 lines · Russell King - ARM Linux admin
Quoted 3 lines · Geert Uytterhoeven

Is there a real reason this is only applied to a subset of the C object files, and not to all of them? Or have we been lucky so far, by not triggering the issue in decompressed.c, misc.c, and string.c (yet)?

I don't remember the details. See commit 7f66cd3f5420, which came from Kees which introduced this.

Just to clarify: the original change was just removing it where it was detected in the then-current build. I was going for the least invasive change to the build system.

Signature · 1 line
Kees Cook