pahole -J usage in kernel scripts/link-vmlinux.sh
Hi BTF folks,
I have discovered some problems with pahole -J.
Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):
ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o
pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?
When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC. This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
In particular, there are two bugs:
* pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its sh_offset != sh_addr (mod max-page-size)
Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0 - [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096 + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096
* pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:
error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
This is very rare, though.
So I suggest:
* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections. Place the new non-SHF_ALLOC .BTF section at the end.
Then not rewriting PT_LOAD will be justified.
* Add a new option incorporating the current pahole -J + llvm-objcopy
Quoted 6 lines · Fangrui Song
Hi BTF folks,
I have discovered some problems with pahole -J.
Its usage in kernelscripts/link-vmlinux.shis like (make LLVM=1 bzImage):
Quoted 7 lines
ld.lld -m
elf_x86_64--emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T./arch/x86/kernel/vmlinux.lds--whole-archivearch/x86/kernel/head_64.oarch/x86/kernel/head64.oarch/x86/kernel/ebda.oarch/x86/kernel/platform-quirks.oinit/built-in.ausr/built-in.aarch/x86/built-in.akernel/built-in.acerts/built-in.amm/built-in.afs/built-in.aipc/built-in.asecurity/built-in.acrypto/built-in.ablock/built-in.alib/built-in.aarch/x86/lib/built-in.alib/lib.aarch/x86/lib/lib.adrivers/built-in.asound/built-in.anet/built-in.avirt/built-in.aarch/x86/pci/built-in.aarch/x86/power/built-in.aarch/x86/video/built-in.a--no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.opahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?
We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.
Quoted 4 lines
When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes
SHF_ALLOC.
I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:
$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF [13] .BTF_ids PROGBITS ffffffff822b6804 14b6804 000510 00 A 0 0 1 [43] .BTF PROGBITS 0000000000000000 18bc0e8c 4c8056 00 0 0 1
Quoted 18 lines
This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
In particular, there are two bugs:
* pahole does not respect max-page-size (
p_alignofPT_LOAD). See the .text section, itssh_offset!=sh_addr(mod max-page-size)Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0(quoted diff) +1
-1- [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096 + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096* pahole does not rewrite p_offset/p_filesz of
PT_LOADsegments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz ofPT_LOADsegments are not rewritten, the file offset range of .symtab may be within aPT_LOADrange. llvm-objcopy --strip-all considers .symtab as part of thePT_LOADand refuses --strip-all:
Quoted 8 lines
error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
This is very rare, though.
So I suggest:
* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite
sh_offsetof existing sections.
so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ
Quoted 3 lines
Place the new non-SHF_ALLOC .BTF section at the end. Then not rewriting
PT_LOADwill be justified.
* Add a new option incorporating the current pahole -J + llvm-objcopy
pahole is already using llvm-objcopy. Do you mean to add a mode where pahole will dump a separate ELF object file with just .BTF? Or something else?
Quoted 60 lines · Andrii Nakryiko
Quoted 6 lines · Fangrui Song
Hi BTF folks,
I have discovered some problems with pahole -J.
Its usage in kernelscripts/link-vmlinux.shis like (make LLVM=1 bzImage):Quoted 7 lines
ld.lld -m
elf_x86_64--emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T./arch/x86/kernel/vmlinux.lds--whole-archivearch/x86/kernel/head_64.oarch/x86/kernel/head64.oarch/x86/kernel/ebda.oarch/x86/kernel/platform-quirks.oinit/built-in.ausr/built-in.aarch/x86/built-in.akernel/built-in.acerts/built-in.amm/built-in.afs/built-in.aipc/built-in.asecurity/built-in.acrypto/built-in.ablock/built-in.alib/built-in.aarch/x86/lib/built-in.alib/lib.aarch/x86/lib/lib.adrivers/built-in.asound/built-in.anet/built-in.avirt/built-in.aarch/x86/pci/built-in.aarch/x86/power/built-in.aarch/x86/video/built-in.a--no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.opahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?
We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.
Quoted 4 lines
When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes
SHF_ALLOC.I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add
SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF [13] .BTF_ids PROGBITS ffffffff822b6804 14b6804 000510 00 A 0 0 1 [43] .BTF PROGBITS 0000000000000000 18bc0e8c 4c8056 00 0 0 1Quoted 23 lines
This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
In particular, there are two bugs:
* pahole does not respect max-page-size (
p_alignofPT_LOAD). See the .text section, itssh_offset!=sh_addr(mod max-page-size)Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0(quoted diff) +1
-1- [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096 + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096* pahole does not rewrite p_offset/p_filesz of
PT_LOADsegments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz ofPT_LOADsegments are not rewritten, the file offset range of .symtab may be within aPT_LOADrange. llvm-objcopy --strip-all considers .symtab as part of thePT_LOADand refuses --strip-all:error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
This is very rare, though.
Quoted 7 lines
Quoted 3 lines
So I suggest:
* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite
sh_offsetof existing sections.so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ
Here's one things we're seeing (note, we're using a kernel based on 4.15). Before we run pahole, we have this. Notice the offset of the '.text' section:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 200000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c 100169c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 17caf40
000000 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 17caf40
0004c0 00 A 0 0 1
After running pahole, we get this:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 001000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c e0269c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 11ecf40
3c1ebe 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 15aedfe
0004c0 00 A 0 0 1
The offset of '.text' changed. This is because elf_update decides what the offsets should be. Did the linker scripts change between 4.15 and top-of-tree to mark .BTF as non-allocatable?
-bw
Quoted 7 lines
Quoted 3 lines
Place the new non-SHF_ALLOC .BTF section at the end. Then not rewriting
PT_LOADwill be justified.
* Add a new option incorporating the current pahole -J + llvm-objcopypahole is already using llvm-objcopy. Do you mean to add a mode where pahole will dump a separate ELF object file with just .BTF? Or something else?
Quoted 100 lines · Bill Wendling
Quoted 60 lines · Andrii Nakryiko
Quoted 6 lines · Fangrui Song
Hi BTF folks,
I have discovered some problems with pahole -J.
Its usage in kernelscripts/link-vmlinux.shis like (make LLVM=1 bzImage):Quoted 7 lines
ld.lld -m
elf_x86_64--emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T./arch/x86/kernel/vmlinux.lds--whole-archivearch/x86/kernel/head_64.oarch/x86/kernel/head64.oarch/x86/kernel/ebda.oarch/x86/kernel/platform-quirks.oinit/built-in.ausr/built-in.aarch/x86/built-in.akernel/built-in.acerts/built-in.amm/built-in.afs/built-in.aipc/built-in.asecurity/built-in.acrypto/built-in.ablock/built-in.alib/built-in.aarch/x86/lib/built-in.alib/lib.aarch/x86/lib/lib.adrivers/built-in.asound/built-in.anet/built-in.avirt/built-in.aarch/x86/pci/built-in.aarch/x86/power/built-in.aarch/x86/video/built-in.a--no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.opahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?
We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.
Quoted 4 lines
When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes
SHF_ALLOC.I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add
SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF [13] .BTF_ids PROGBITS ffffffff822b6804 14b6804 000510 00 A 0 0 1 [43] .BTF PROGBITS 0000000000000000 18bc0e8c 4c8056 00 0 0 1Quoted 23 lines
This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
In particular, there are two bugs:
* pahole does not respect max-page-size (
p_alignofPT_LOAD). See the .text section, itssh_offset!=sh_addr(mod max-page-size)Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0(quoted diff) +1
-1- [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096 + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096* pahole does not rewrite p_offset/p_filesz of
PT_LOADsegments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz ofPT_LOADsegments are not rewritten, the file offset range of .symtab may be within aPT_LOADrange. llvm-objcopy --strip-all considers .symtab as part of thePT_LOADand refuses --strip-all:error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
This is very rare, though.
Quoted 7 lines
Quoted 3 lines
So I suggest:
* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite
sh_offsetof existing sections.so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ
Here's one things we're seeing (note, we're using a kernel based on 4.15). Before we run pahole, we have this. Notice the offset of the '.text' section:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITSffffffff81000000200000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTEffffffff81e0169c100169c
000024 00 A 0 0 4
...
[24] .BTF PROGBITSffffffff825caf4017caf40
000000 00 WA 0 0 1
[25] .BTF_ids PROGBITSffffffff825caf4017caf40
0004c0 00 A 0 0 1After running pahole, we get this:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITSffffffff81000000001000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTEffffffff81e0169ce0269c
000024 00 A 0 0 4
...
[24] .BTF PROGBITSffffffff825caf4011ecf40
3c1ebe 00 WA 0 0 1
[25] .BTF_ids PROGBITSffffffff825caf4015aedfe
0004c0 00 A 0 0 1
Quoted 3 lines
The offset of '.text' changed. This is because
elf_updatedecides what the offsets should be. Did the linker scripts change between 4.15 and top-of-tree to mark .BTF as non-allocatable?
maybe so, but .tmp_vmlinux.btf which gets .BTF is discarded and not used anymore. We only dump .BTF section into a separate ELF, which is linked into the final vmlinux image as the next step. So pahole doesn't rewrite the final binary.
Quoted 10 lines
-bw
Quoted 7 lines
Quoted 3 lines
Place the new non-SHF_ALLOC .BTF section at the end. Then not rewriting
PT_LOADwill be justified.
* Add a new option incorporating the current pahole -J + llvm-objcopypahole is already using llvm-objcopy. Do you mean to add a mode where pahole will dump a separate ELF object file with just .BTF? Or something else?
Quoted 102 lines · Andrii Nakryiko
Quoted 100 lines · Bill Wendling
Quoted 60 lines · Andrii Nakryiko
Quoted 6 lines · Fangrui Song
Hi BTF folks,
I have discovered some problems with pahole -J.
Its usage in kernelscripts/link-vmlinux.shis like (make LLVM=1 bzImage):Quoted 7 lines
ld.lld -m
elf_x86_64--emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T./arch/x86/kernel/vmlinux.lds--whole-archivearch/x86/kernel/head_64.oarch/x86/kernel/head64.oarch/x86/kernel/ebda.oarch/x86/kernel/platform-quirks.oinit/built-in.ausr/built-in.aarch/x86/built-in.akernel/built-in.acerts/built-in.amm/built-in.afs/built-in.aipc/built-in.asecurity/built-in.acrypto/built-in.ablock/built-in.alib/built-in.aarch/x86/lib/built-in.alib/lib.aarch/x86/lib/lib.adrivers/built-in.asound/built-in.anet/built-in.avirt/built-in.aarch/x86/pci/built-in.aarch/x86/power/built-in.aarch/x86/video/built-in.a--no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.opahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?
We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.
Quoted 4 lines
When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes
SHF_ALLOC.I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add
SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF [13] .BTF_ids PROGBITS ffffffff822b6804 14b6804 000510 00 A 0 0 1 [43] .BTF PROGBITS 0000000000000000 18bc0e8c 4c8056 00 0 0 1Quoted 23 lines
This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
In particular, there are two bugs:
* pahole does not respect max-page-size (
p_alignofPT_LOAD). See the .text section, itssh_offset!=sh_addr(mod max-page-size)Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0(quoted diff) +1
-1- [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096 + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096* pahole does not rewrite p_offset/p_filesz of
PT_LOADsegments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz ofPT_LOADsegments are not rewritten, the file offset range of .symtab may be within aPT_LOADrange. llvm-objcopy --strip-all considers .symtab as part of thePT_LOADand refuses --strip-all:error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
This is very rare, though.
Quoted 7 lines
Quoted 3 lines
So I suggest:
* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite
sh_offsetof existing sections.so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ
Here's one things we're seeing (note, we're using a kernel based on 4.15). Before we run pahole, we have this. Notice the offset of the '.text' section:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITSffffffff81000000200000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTEffffffff81e0169c100169c
000024 00 A 0 0 4
...
[24] .BTF PROGBITSffffffff825caf4017caf40
000000 00 WA 0 0 1
[25] .BTF_ids PROGBITSffffffff825caf4017caf40
0004c0 00 A 0 0 1After running pahole, we get this:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITSffffffff81000000001000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTEffffffff81e0169ce0269c
000024 00 A 0 0 4
...
[24] .BTF PROGBITSffffffff825caf4011ecf40
3c1ebe 00 WA 0 0 1
[25] .BTF_ids PROGBITSffffffff825caf4015aedfe
0004c0 00 A 0 0 1
Quoted 9 lines
Quoted 3 lines
The offset of '.text' changed. This is because
elf_updatedecides what the offsets should be. Did the linker scripts change between 4.15 and top-of-tree to mark .BTF as non-allocatable?maybe so, but .tmp_vmlinux.btf which gets .BTF is discarded and not used anymore. We only dump .BTF section into a separate ELF, which is linked into the final vmlinux image as the next step. So pahole doesn't rewrite the final binary.
Right. The problem is with the follow-up command:
${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \
--strip-all ${1} ${2} 2>/dev/nullThe file .tmp_vmlinux.btf isn't "correct" (see maskray's explanation), so this command fails to write a valid ELF file.
-bw
Quoted 114 lines · Bill Wendling
Quoted 102 lines · Andrii Nakryiko
Quoted 100 lines · Bill Wendling
Quoted 60 lines · Andrii Nakryiko
Quoted 6 lines · Fangrui Song
Hi BTF folks,
I have discovered some problems with pahole -J.
Its usage in kernelscripts/link-vmlinux.shis like (make LLVM=1 bzImage):Quoted 7 lines
ld.lld -m
elf_x86_64--emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T./arch/x86/kernel/vmlinux.lds--whole-archivearch/x86/kernel/head_64.oarch/x86/kernel/head64.oarch/x86/kernel/ebda.oarch/x86/kernel/platform-quirks.oinit/built-in.ausr/built-in.aarch/x86/built-in.akernel/built-in.acerts/built-in.amm/built-in.afs/built-in.aipc/built-in.asecurity/built-in.acrypto/built-in.ablock/built-in.alib/built-in.aarch/x86/lib/built-in.alib/lib.aarch/x86/lib/lib.adrivers/built-in.asound/built-in.anet/built-in.avirt/built-in.aarch/x86/pci/built-in.aarch/x86/power/built-in.aarch/x86/video/built-in.a--no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.opahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?
We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.
Quoted 4 lines
When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes
SHF_ALLOC.I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add
SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF [13] .BTF_ids PROGBITS ffffffff822b6804 14b6804 000510 00 A 0 0 1 [43] .BTF PROGBITS 0000000000000000 18bc0e8c 4c8056 00 0 0 1Quoted 23 lines
This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).
In particular, there are two bugs:
* pahole does not respect max-page-size (
p_alignofPT_LOAD). See the .text section, itssh_offset!=sh_addr(mod max-page-size)Section Headers:
[Nr] Name Type Address Off Size ES Flg Lk Inf Al [ 0] NULL 0000000000000000 000000 000000 00 0 0 0(quoted diff) +1
-1- [ 1] .text PROGBITS ffffffff81000000 200000 1003917 00 AX 0 0 4096 + [ 1] .text PROGBITS ffffffff81000000 001000 e0169a 00 AX 0 0 4096* pahole does not rewrite p_offset/p_filesz of
PT_LOADsegments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz ofPT_LOADsegments are not rewritten, the file offset range of .symtab may be within aPT_LOADrange. llvm-objcopy --strip-all considers .symtab as part of thePT_LOADand refuses --strip-all:error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'
This is very rare, though.
Quoted 7 lines
Quoted 3 lines
So I suggest:
* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite
sh_offsetof existing sections.so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ
Here's one things we're seeing (note, we're using a kernel based on 4.15). Before we run pahole, we have this. Notice the offset of the '.text' section:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITSffffffff81000000200000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTEffffffff81e0169c100169c
000024 00 A 0 0 4
...
[24] .BTF PROGBITSffffffff825caf4017caf40
000000 00 WA 0 0 1
[25] .BTF_ids PROGBITSffffffff825caf4017caf40
0004c0 00 A 0 0 1After running pahole, we get this:
[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITSffffffff81000000001000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTEffffffff81e0169ce0269c
000024 00 A 0 0 4
...
[24] .BTF PROGBITSffffffff825caf4011ecf40
3c1ebe 00 WA 0 0 1
[25] .BTF_ids PROGBITSffffffff825caf4015aedfe
0004c0 00 A 0 0 1Quoted 9 lines
Quoted 3 lines
The offset of '.text' changed. This is because
elf_updatedecides what the offsets should be. Did the linker scripts change between 4.15 and top-of-tree to mark .BTF as non-allocatable?maybe so, but .tmp_vmlinux.btf which gets .BTF is discarded and not used anymore. We only dump .BTF section into a separate ELF, which is linked into the final vmlinux image as the next step. So pahole doesn't rewrite the final binary.
Quoted 7 lines
Right. The problem is with the follow-up command:
${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \ --strip-all ${1} ${2} 2>/dev/nullThe file .tmp_vmlinux.btf isn't "correct" (see maskray's explanation), so this command fails to write a valid ELF file.
Oh, I see. Yeah, I didn't get that from the original email. So seems like llvm-objcopy that pahole invokes internally will corrupt that .tmp_vmlinux.btf file to the point of another objcopy not being able to handle it. Interesting.
But regardless, as I said, we already discussed this feature and think that it's a good addition. I'd like to be able to both get object file with just .BTF (for kernel linking process) and pure binary contents of .BTF section (for other, CO-RE-related reasons, especially for legacy systems). I think Arnaldo is onboard as well. So please feel free to send patches.
Quoted 2 lines
-bw