arm64: fix Build ID if CONFIG_MODVERSIONS

Hi Greg,
Our internal rpm build for aarch64 started failing after merging
v5.15.60. The build actually succeeds, but packaging failed due to tools which extract the Build ID from vmlinux.

Expected:
readelf -n vmlinux

Displaying notes found in: .notes
Owner Data size Description
GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring)

    Build ID: ae5803ded69a15fb0a75905ec226a76cc1823d22

Linux 0x00000004 func
description data: 00 00 00 00
Linux 0x00000001 OPEN
description data: 00

Actual (post v5.15.60 merge):
readelf -n vmlinux
<no output>

Digging deeper...

Expected:
readelf --headers vmlinux | sed -ne '/Program Header/,$p' Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000010000 0xffff800008000000 0xffff800008000000

                 0x0000000002012a00 0x000000000208f724  RWE    0x10000
  NOTE           0x0000000001705948 0xffff8000096f5948 0xffff8000096f5948
                 0x0000000000000054 0x0000000000000054  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10

Section to Segment mapping:
Segment Sections...
00 .head.text .text .got.plt .rodata .pci_fixup __ksymtab __ksymtab_gpl __kcrctab __kcrctab_gpl __ksymtab_strings __param __modver __ex_table .notes .hyp.rodata .init.text .exit.text .altinstructions .init.data .data..percpu .hyp.data..percpu .hyp.reloc .rela.dyn .data __bug_table .mmuoff.data.write .mmuoff.data.read .pecoff_edata_padding .bss
01 .notes
02

Actual:
readelf --headers vmlinux | sed -ne '/Program Header/,$p' Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
LOAD 0x0000000000010000 0xffff800008000000 0xffff800008000000

                 0x0000000002012a00 0x000000000208f724  RWE    0x10000
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10

Section to Segment mapping:
Segment Sections...
00 .head.text .text .got.plt .rodata .pci_fixup __ksymtab __ksymtab_gpl __kcrctab __kcrctab_gpl __ksymtab_strings __param __modver __ex_table .notes .hyp.rodata .init.text .exit.text .altinstructions .init.data .data..percpu .hyp.data..percpu .hyp.reloc .rela.dyn .data __bug_table .mmuoff.data.write .mmuoff.data.read .pecoff_edata_padding .bss
01

readelf -n fails since NOTE segment is missing from vmlinux.

Why?

5.15 Commit: 4c7ee827da2c ("Makefile: link with -z noexecstack --no-warn-rwx-segments") surfaced this issue.

1. Reverting this patch fixed this issue.

2. Removing all .note.GNU-stack sections in cmd_modversions_S also worked:

index 17aa8ef2d52a..b95cfbb43cee 100644
@@ -379,6 +379,8 @@ cmd_modversions_S =								\
 	if $(OBJDUMP) -h $@ | grep -q __ksymtab; then				\
 		$(call cmd_gensymtypes_S,$(KBUILD_SYMTYPES),$(@:.o=.symtypes))	\
 		    > $(@D)/.tmp_$(@F:.o=.ver);					\
+		echo "SECTIONS { /DISCARD/ : { *(.note.GNU-stack) }}"		\
+		>> $(@D)/.tmp_$(@F:.o=.ver); 					\
 										\
 		$(LD) $(KBUILD_LDFLAGS) -r -o $(@D)/.tmp_$(@F) $@ 		\
 			-T $(@D)/.tmp_$(@F:.o=.ver);				\

3. But the simplest fix I found was to remove -z noexecstack just for head.S (this patch).

This issue exists, in my testing of arm64 builds, for stable versions 5.15.30+, 5.10.136+, and 5.4.210+
and reproduces with every attempted combination of gcc{11,12} ld{2.36,2.37,2.38,2.39}.

It can also be reproduced upstream by cherry-picking 0d362be5b142 on-top until
7b4537199a4a ("kbuild: link symbol CRCs at final link, removing CONFIG_MODULE_REL_CRCS")
which was part of merge
df202b452fe6 ("Merge tag 'kbuild-v5.19' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild")

It seems kernels prior to v5.4 and 5.19+ use differing KBUILD rules (with CONFIG_MODVERSIONS=y) for linking components of vmlinux, and in my testing these other versions did not encounter this issue.

Other architectures besides arm64 might also have similar bug/feature of ld. Though, x86_64 v5.15.60 worked as expected, so no further experiments were performed for x86_64.

arm64 repro test:

make ARCH=arm64 mrproper
cp arch/arm64/configs/defconfig ".config"
scripts/config -e MODVERSIONS
make ARCH=arm64 olddefconfig
make ARCH=arm64 -j16 vmlinux
readelf -n vmlinux | grep -F "Build ID:"

Please consider applying for 5.15, 5.10, and 5.4.

Regards,

--Tom

Tom Saeger (1):
arm64: fix Build ID if CONFIG_MODVERSIONS

 arch/arm64/kernel/Makefile | 5 +++++
 1 file changed, 5 insertions(+)

base-commit: e4a7232c917cd1b56d5b4fa9d7a23e3eabfecba0

2.38.1

Backport of: 0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments") breaks arm64 Build ID when CONFIG_MODVERSIONS=y.

CONFIG_MODVERSIONS adds extra tooling to calculate symbol versions. This kernel's KBUILD tooling uses both
relocatable (-r) and (-z noexecstack) to link head.o which results in ld adding a .note.GNU-stack section. Final linking of vmlinux should add a .NOTES segment containing the Build ID, but does NOT if head.o has a .note.GNU-stack section.

Selectively remove -z noexecstack from head.o's KBUILD_LDFLAGS to prevent .note.GNU-stack from being added to head.o. Final link of vmlinux then properly adds .NOTES segment containing Build ID that can be read using tools like 'readelf -n'.

---
 arch/arm64/kernel/Makefile | 5 +++++
 1 file changed, 5 insertions(+)

index 749e31475e41..8bdee0f655e2 100644
@@ -85,3 +85,8 @@ extra-y					+= $(head-y) vmlinux.lds
 ifeq ($(CONFIG_DEBUG_EFI),y)
 AFLAGS_head.o += -DVMLINUX_PATH="\"$(realpath $(objtree)/vmlinux)\""
 endif
+
+ifeq ($(CONFIG_MODVERSIONS),y)
+# filter-out -z noexecstack for head.S
+$(obj)/head.o: KBUILD_LDFLAGS := $(shell echo "$(KBUILD_LDFLAGS)" | sed -e 's/-z\s\+noexecstack//')
+endif

2.38.1

Why isn't this needed in Linus's tree?

And why not cc: everyone involved in this, I would need acks from maintainers to be able to accept this...

thanks,

greg k-h

0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments")

was merged after
7b4537199a4a ("kbuild: link symbol CRCs at final link, removing CONFIG_MODULE_REL_CRCS")

Linus's tree never had -z noexecstack with these same KBUILD rules.

Fair request.

Between ~5.3 and 5.19-rc1 cherry-picking
0d362be5b142 ("Makefile: link with -z noexecstack --no-warn-rwx-segments") and building arm64 with CONFIG_MODVERSIONS=y
results in vmlinux missing Build ID

head.S is compiled to head.o
head.o is linked (ld) with -r and -z noexecstack which adds .note.GNU-stack section in head.o head.o is then linked again with vmlinux (resulting vmlinux is missing .NOTE segment)

Can folks confirm/deny ld behavior is expected (arm64)?
And whether the above patch would be an acceptable fix for these kernel versions?

repro test in cover letter: https://lore.kernel.org/all/[email protected]/#r

Regards,
--Tom

Then it needs to say that, in detail, in this changelog please.

Please resend, no need for a cover letter for just one patch, and include all of the needed info in the changelog itself.

thanks,

greg k-h

If you remove -z noexecstack, aren't you just going to trigger warnings from BFD again?

At the least consider adding a fixes tag for 0d362be5b142, and note that stable doesn't have
7b4537199a4a, in the commit message.

Thanks,
~Nick Desaulniers

hmm, probably so, let me check