kbuild: switch to thin archives
Subsystems (6)
Nicholas Piggin
reviewerPATCH 1/1 kbuild: switch to thin archives
Switch from incremental build to thin archives for packaging built-in.o. binutils version must be bumped to 2.20. Proposed patch for 4.13.
---
Documentation/process/changes.rst | 9 ++++----- Makefile | 3 +++ arch/Kconfig | 6 ------ arch/powerpc/Kconfig | 8 -------- scripts/Makefile.build | 27 ++++++------------------- scripts/link-vmlinux.sh | 42 +++++++++------------------------------ 6 files changed, 22 insertions(+), 73 deletions(-)
Documentation/process/changes.rst +4 -5
@@ -31,7 +31,7 @@ you probably needn't concern yourself with isdn4k-utils. ====================== =============== ======================================== GNU C 3.2 gcc --version GNU make 3.81 make --version -binutils 2.12 ld -v +binutils 2.20 ld -v util-linux 2.10o fdformat --version module-init-tools 0.9.10 depmod -V e2fsprogs 1.41.4 e2fsck -V
@@ -75,10 +75,9 @@ You will need GNU make 3.81 or later to build the kernel. Binutils -------- -Linux on IA-32 has recently switched from using ``as86`` to using ``gas`` for -assembling the 16-bit boot code, removing the need for ``as86`` to compile -your kernel. This change does, however, mean that you need a recent -release of binutils. +The build system has recently switched to using thin archives (`ar T`) rather +than incremental linking (`ld -r`) for built-in.o intermediate steps. This +requires binutils 2.20 or newer. Perl ----
Makefile +3 -0
@@ -818,6 +818,9 @@ KBUILD_CFLAGS += $(call cc-option,-Werror=designated-init) # use the deterministic mode of AR if available KBUILD_ARFLAGS := $(call ar-option,D) +# Use thin archives. +KBUILD_ARFLAGS :=$(KBUILD_ARFLAGS)T + include scripts/Makefile.kasan include scripts/Makefile.extrawarn include scripts/Makefile.ubsan
arch/Kconfig +0 -6
@@ -510,12 +510,6 @@ config CC_STACKPROTECTOR_STRONG endchoice -config THIN_ARCHIVES - bool - help - Select this if the architecture wants to use thin archives - instead of ld -r to create the built-in.o files. - config LD_DEAD_CODE_DATA_ELIMINATION bool help
arch/powerpc/Kconfig +0 -8
@@ -472,14 +472,6 @@ config MPROFILE_KERNEL depends on PPC64 && CPU_LITTLE_ENDIAN def_bool !DISABLE_MPROFILE_KERNEL -config USE_THIN_ARCHIVES - bool "Build the kernel using thin archives" - default n - select THIN_ARCHIVES - help - Build the kernel using thin archives. - If you're unsure say N. - config IOMMU_HELPER def_bool PPC64
scripts/Makefile.build +6 -21
@@ -436,15 +436,9 @@ $(sort $(subdir-obj-y)): $(subdir-ym) ; # ifdef builtin-target -ifdef CONFIG_THIN_ARCHIVES - cmd_make_builtin = rm -f $@; $(AR) rcST$(KBUILD_ARFLAGS) - cmd_make_empty_builtin = rm -f $@; $(AR) rcST$(KBUILD_ARFLAGS) - quiet_cmd_link_o_target = AR $@ -else - cmd_make_builtin = $(LD) $(ld_flags) -r -o - cmd_make_empty_builtin = rm -f $@; $(AR) rcs$(KBUILD_ARFLAGS) - quiet_cmd_link_o_target = LD $@ -endif +cmd_make_builtin = rm -f $@; $(AR) rcS$(KBUILD_ARFLAGS) +cmd_make_empty_builtin = rm -f $@; $(AR) rcS$(KBUILD_ARFLAGS) +quiet_cmd_link_o_target = AR $@ # If the list of objects to link is empty, just create an empty built-in.o cmd_link_o_target = $(if $(strip $(obj-y)),\
@@ -477,11 +471,7 @@ $(modorder-target): $(subdir-ym) FORCE ifdef lib-target quiet_cmd_link_l_target = AR $@ -ifdef CONFIG_THIN_ARCHIVES - cmd_link_l_target = rm -f $@; $(AR) rcsT$(KBUILD_ARFLAGS) $@ $(lib-y) -else - cmd_link_l_target = rm -f $@; $(AR) rcs$(KBUILD_ARFLAGS) $@ $(lib-y) -endif +cmd_link_l_target = rm -f $@; $(AR) rcs$(KBUILD_ARFLAGS) $@ $(lib-y) $(lib-target): $(lib-y) FORCE $(call if_changed,link_l_target)
@@ -529,13 +519,8 @@ $($(subst $(obj)/,,$(@:.o=-m)))), $^) cmd_link_multi-link = $(LD) $(ld_flags) -r -o $@ $(link_multi_deps) $(cmd_secanalysis) -ifdef CONFIG_THIN_ARCHIVES - quiet_cmd_link_multi-y = AR $@ - cmd_link_multi-y = rm -f $@; $(AR) rcST$(KBUILD_ARFLAGS) $@ $(link_multi_deps) -else - quiet_cmd_link_multi-y = LD $@ - cmd_link_multi-y = $(cmd_link_multi-link) -endif +quiet_cmd_link_multi-y = AR $@ +cmd_link_multi-y = rm -f $@; $(AR) rcS$(KBUILD_ARFLAGS) $@ $(link_multi_deps) quiet_cmd_link_multi-m = LD [M] $@ cmd_link_multi-m = $(cmd_link_multi-link)
scripts/link-vmlinux.sh +9 -33
64 lines not on this page · raw message
Signature · 1 line
2.11.0
kbuild test robot
build boturl: https://github.com/0day-ci/linux/commits/Nicholas-Piggin/kbuild-switch-to-thin-archives/20170603-025436 config: parisc-c3000_defconfig (attached as .config) compiler: hppa-linux-gnu-gcc (Debian…
Hi Nicholas,
[auto build test ERROR on linus/master]
[also build test ERROR on v4.12-rc3 next-20170602]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Nicholas-Piggin/kbuild-switch-to-thin-archives/20170603-025436
config: parisc-c3000_defconfig (attached as .config)
compiler: hppa-linux-gnu-gcc (Debian 6.1.1-9) 6.1.1 20160705
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=parisc
All errors (new ones prefixed by >>):
/usr/lib/gcc-cross/hppa-linux-gnu/6/libgcc.a(_ucmpdi2.o): In function `__ucmpdi2':
>> (.text+0x0): multiple definition of `__ucmpdi2' arch/parisc/lib/ucmpdi2.o:(.text.__ucmpdi2+0x0): first defined here /usr/lib/gcc-cross/hppa-linux-gnu/6/libgcc.a(fptr.o): In function `__canonicalize_funcptr_for_compare': >> (.text+0x0): multiple definition of `__canonicalize_funcptr_for_compare'
arch/parisc/kernel/real2.o:(.text+0x278): first defined here
hppa-linux-gnu-ld: internal error ../../ld/ldlang.c 6366
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Nicholas Piggin
reviewerQuoted 5 lines · On Sat, 3 Jun 2017 04:57:21 +0800 kbuild test robot <[email protected]> wrote:
Hi Nicholas,
[auto build test ERROR on linus/master]
[also build test ERROR on v4.12-rc3 next-20170602]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
Thank you robot, can we try this one?
--- Documentation/process/changes.rst | 9 ++++----- Makefile | 3 +++ arch/Kconfig | 6 ------ arch/powerpc/Kconfig | 8 -------- arch/x86/um/vdso/Makefile | 2 +- scripts/Makefile.build | 27 ++++++------------------- scripts/link-vmlinux.sh | 42 +++++++++------------------------------ 7 files changed, 23 insertions(+), 74 deletions(-)
Documentation/process/changes.rst +4 -5
@@ -31,7 +31,7 @@ you probably needn't concern yourself with isdn4k-utils. ====================== =============== ======================================== GNU C 3.2 gcc --version GNU make 3.81 make --version -binutils 2.12 ld -v +binutils 2.20 ld -v util-linux 2.10o fdformat --version module-init-tools 0.9.10 depmod -V e2fsprogs 1.41.4 e2fsck -V
@@ -75,10 +75,9 @@ You will need GNU make 3.81 or later to build the kernel. Binutils -------- -Linux on IA-32 has recently switched from using ``as86`` to using ``gas`` for -assembling the 16-bit boot code, removing the need for ``as86`` to compile -your kernel. This change does, however, mean that you need a recent -release of binutils. +The build system has recently switched to using thin archives (`ar T`) rather +than incremental linking (`ld -r`) for built-in.o intermediate steps. This +requires binutils 2.20 or newer. Perl ----
Makefile +3 -0
@@ -818,6 +818,9 @@ KBUILD_CFLAGS += $(call cc-option,-Werror=designated-init) # use the deterministic mode of AR if available KBUILD_ARFLAGS := $(call ar-option,D) +# must use thin archives. +KBUILD_ARFLAGS :=$(KBUILD_ARFLAGS)T + include scripts/Makefile.kasan include scripts/Makefile.extrawarn include scripts/Makefile.ubsan
arch/Kconfig +0 -6
@@ -510,12 +510,6 @@ config CC_STACKPROTECTOR_STRONG endchoice -config THIN_ARCHIVES - bool - help - Select this if the architecture wants to use thin archives - instead of ld -r to create the built-in.o files. - config LD_DEAD_CODE_DATA_ELIMINATION bool help
arch/powerpc/Kconfig +0 -8
@@ -472,14 +472,6 @@ config MPROFILE_KERNEL depends on PPC64 && CPU_LITTLE_ENDIAN def_bool !DISABLE_MPROFILE_KERNEL -config USE_THIN_ARCHIVES - bool "Build the kernel using thin archives" - default n - select THIN_ARCHIVES - help - Build the kernel using thin archives. - If you're unsure say N. - config IOMMU_HELPER def_bool PPC64
arch/x86/um/vdso/Makefile +1 -1
@@ -53,7 +53,7 @@ CFLAGS_REMOVE_vdso-note.o = -pg -fprofile-arcs -ftest-coverage CFLAGS_REMOVE_um_vdso.o = -pg -fprofile-arcs -ftest-coverage targets += vdso-syms.lds -obj-$(VDSO64-y) += vdso-syms.lds +extra-$(VDSO64-y) += vdso-syms.lds # # Match symbols in the DSO that look like VDSO*; produce a file of constants.
scripts/Makefile.build +6 -21
@@ -436,15 +436,9 @@ $(sort $(subdir-obj-y)): $(subdir-ym) ; # ifdef builtin-target -ifdef CONFIG_THIN_ARCHIVES - cmd_make_builtin = rm -f $@; $(AR) rcST$(KBUILD_ARFLAGS) - cmd_make_empty_builtin = rm -f $@; $(AR) rcST$(KBUILD_ARFLAGS) - quiet_cmd_link_o_target = AR $@ -else - cmd_make_builtin = $(LD) $(ld_flags) -r -o - cmd_make_empty_builtin = rm -f $@; $(AR) rcs$(KBUILD_ARFLAGS) - quiet_cmd_link_o_target = LD $@ -endif +cmd_make_builtin = rm -f $@; $(AR) rcS$(KBUILD_ARFLAGS) +cmd_make_empty_builtin = rm -f $@; $(AR) rcS$(KBUILD_ARFLAGS) +quiet_cmd_link_o_target = AR $@ # If the list of objects to link is empty, just create an empty built-in.o cmd_link_o_target = $(if $(strip $(obj-y)),\
@@ -477,11 +471,7 @@ $(modorder-target): $(subdir-ym) FORCE ifdef lib-target quiet_cmd_link_l_target = AR $@ -ifdef CONFIG_THIN_ARCHIVES - cmd_link_l_target = rm -f $@; $(AR) rcsT$(KBUILD_ARFLAGS) $@ $(lib-y) -else - cmd_link_l_target = rm -f $@; $(AR) rcs$(KBUILD_ARFLAGS) $@ $(lib-y) -endif +cmd_link_l_target = rm -f $@; $(AR) rcs$(KBUILD_ARFLAGS) $@ $(lib-y) $(lib-target): $(lib-y) FORCE $(call if_changed,link_l_target)
@@ -529,13 +519,8 @@ $($(subst $(obj)/,,$(@:.o=-m)))), $^) cmd_link_multi-link = $(LD) $(ld_flags) -r -o $@ $(link_multi_deps) $(cmd_secanalysis) -ifdef CONFIG_THIN_ARCHIVES - quiet_cmd_link_multi-y = AR $@ - cmd_link_multi-y = rm -f $@; $(AR) rcST$(KBUILD_ARFLAGS) $@ $(link_multi_deps) -else - quiet_cmd_link_multi-y = LD $@ - cmd_link_multi-y = $(cmd_link_multi-link) -endif +quiet_cmd_link_multi-y = AR $@ +cmd_link_multi-y = rm -f $@; $(AR) rcS$(KBUILD_ARFLAGS) $@ $(link_multi_deps) quiet_cmd_link_multi-m = LD [M] $@ cmd_link_multi-m = $(cmd_link_multi-link)
scripts/link-vmlinux.sh +9 -33
64 lines not on this page · raw message
Signature · 1 line
2.11.0
kbuild test robot
build boturl: https://github.com/0day-ci/linux/commits/Nicholas-Piggin/kbuild-switch-to-thin-archives/20170603-025436 config: ia64-allnoconfig (attached as .config) compiler: ia64-linux-gcc (GCC) 6.2.0…
Hi Nicholas,
[auto build test ERROR on linus/master]
[also build test ERROR on v4.12-rc3 next-20170602]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Nicholas-Piggin/kbuild-switch-to-thin-archives/20170603-025436
config: ia64-allnoconfig (attached as .config)
compiler: ia64-linux-gcc (GCC) 6.2.0
reproduce:
wget https://raw.githubusercontent.com/01org/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# save the attached .config to linux build tree
make.cross ARCH=ia64
All errors (new ones prefixed by >>):
arch/ia64/kernel/entry.o: In function `ia64_work_processed_syscall':
>> (.text+0xa61): undefined reference to `__kernel_syscall_via_epc' arch/ia64/kernel/fsys.o: In function `fsys_bubble_down': >> (.text+0x551): undefined reference to `__kernel_syscall_via_break' arch/ia64/kernel/patch.o: In function `ia64_patch_gate': >> patch.c:(.init.text+0x3d1): undefined reference to `__start_gate_fsyscall_patchlist' >> patch.c:(.init.text+0x3e0): undefined reference to `__end_gate_fsyscall_patchlist' patch.c:(.init.text+0x3f1): undefined reference to `__start_gate_fsyscall_patchlist' patch.c:(.init.text+0x400): undefined reference to `__end_gate_fsyscall_patchlist' >> patch.c:(.init.text+0x4a0): undefined reference to `__start_gate_brl_fsys_bubble_down_patchlist' >> patch.c:(.init.text+0x4a1): undefined reference to `__end_gate_brl_fsys_bubble_down_patchlist' patch.c:(.init.text+0x4b1): undefined reference to `__start_gate_brl_fsys_bubble_down_patchlist' patch.c:(.init.text+0x4c0): undefined reference to `__end_gate_brl_fsys_bubble_down_patchlist' >> patch.c:(.init.text+0x560): undefined reference to `__end_gate_vtop_patchlist' >> patch.c:(.init.text+0x561): undefined reference to `__start_gate_vtop_patchlist' patch.c:(.init.text+0x570): undefined reference to `__end_gate_vtop_patchlist' patch.c:(.init.text+0x571): undefined reference to `__start_gate_vtop_patchlist' >> patch.c:(.init.text+0x580): undefined reference to `__end_gate_mckinley_e9_patchlist' >> patch.c:(.init.text+0x581): undefined reference to `__start_gate_mckinley_e9_patchlist' patch.c:(.init.text+0x590): undefined reference to `__end_gate_mckinley_e9_patchlist' patch.c:(.init.text+0x591): undefined reference to `__start_gate_mckinley_e9_patchlist' arch/ia64/kernel/signal.o: In function `ia64_do_signal': >> signal.c:(.text+0xf11): undefined reference to `__kernel_sigtramp'
signal.c:(.text+0xf40): undefined reference to `__kernel_sigtramp'
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
Quoted 4 lines · Nicholas Piggin
Switch from incremental build to thin archives for packaging built-in.o. binutils version must be bumped to 2.20. Proposed patch for 4.13.
Signed-off-by Nicholas Piggin <[email protected]>
You should explain any advantage this is supposed to have.
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
Nicholas Piggin
reviewerQuoted 7 lines · On Sun, 18 Jun 2017 12:04:26 +0200 Pavel Machek <[email protected]> wrote:
Quoted 4 lines · Nicholas Piggin
Switch from incremental build to thin archives for packaging built-in.o. binutils version must be bumped to 2.20. Proposed patch for 4.13.
Signed-off-by Nicholas Piggin <[email protected]>You should explain any advantage this is supposed to have.
Hi Pavel,
You're right, sorry for this, I was spamming the list with a quick hack to try to get 0day to pick up the patch.
Most of the discussion has been going on in other patch series (check linux-kbuild).
There are 3 main categories of advantages.
First is reduced size of intermediate artifacts in the build output tree.
Second is that incremental linking constrains the linker because it has to assemble each section in the output as a sequence of those section from the inputs. So the more incremental linking, the more you constrain the linker, and this can end up resulting in link fail with large kernels on some architectures.
Third is that some link-time optimizations are not compatible or not very optimal with incremental linking, whereas thin archives is much more amenable to such things.
Thanks,
Nick
Quoted 30 lines · Nicholas Piggin
Quoted 7 lines · On Sun, 18 Jun 2017 12:04:26 +0200 Pavel Machek <[email protected]> wrote:
Quoted 4 lines · Nicholas Piggin
Switch from incremental build to thin archives for packaging built-in.o. binutils version must be bumped to 2.20. Proposed patch for 4.13.
Signed-off-by Nicholas Piggin <[email protected]>You should explain any advantage this is supposed to have.
Hi Pavel,
You're right, sorry for this, I was spamming the list with a quick hack to try to get 0day to pick up the patch.
Most of the discussion has been going on in other patch series (check linux-kbuild).
There are 3 main categories of advantages.
First is reduced size of intermediate artifacts in the build output tree.
Second is that incremental linking constrains the linker because it has to assemble each section in the output as a sequence of those section from the inputs. So the more incremental linking, the more you constrain the linker, and this can end up resulting in link fail with large kernels on some architectures.
Quoted 3 lines
Third is that some link-time optimizations are not compatible or not very optimal with incremental linking, whereas thin archives is much more amenable to such things.
Thanks for explanation. I checked, and binutils version bumb will not affect me, but...
Pavel
Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek (cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html