Skip to contentSkip to conversation

pahole -J usage in kernel scripts/link-vmlinux.sh

DISCUSSION Fangrui Song 3 participants 6 messages Mar 17, 2021 – Mar 20, 2021

lore
Share & tools
lore
#0 lore raw

Hi BTF folks,

I have discovered some problems with pahole -J.
Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):

ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o

pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?

When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC. This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).

In particular, there are two bugs:

* pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its sh_offset != sh_addr (mod max-page-size)

Section Headers:

   [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
   [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
-  [ 1] .text             PROGBITS        ffffffff81000000 200000 1003917 00  AX  0   0 4096
+  [ 1] .text             PROGBITS        ffffffff81000000 001000 e0169a 00  AX  0   0 4096

* pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:

error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'

This is very rare, though.

So I suggest:

* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections. Place the new non-SHF_ALLOC .BTF section at the end.
Then not rewriting PT_LOAD will be justified.
* Add a new option incorporating the current pahole -J + llvm-objcopy

#1 lore raw
Quoted 6 lines · Fangrui Song

Hi BTF folks,

I have discovered some problems with pahole -J.
Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):

Quoted 7 lines

ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o

pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?

We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.

Quoted 4 lines

When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.

I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:

$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
  [13] .BTF_ids          PROGBITS        ffffffff822b6804 14b6804
000510 00   A  0   0  1
  [43] .BTF              PROGBITS        0000000000000000 18bc0e8c
4c8056 00      0   0  1
Quoted 18 lines

This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).

In particular, there are two bugs:

* pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its sh_offset != sh_addr (mod max-page-size)

Section Headers:

    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
    [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
(quoted diff) +1 -1
-  [ 1] .text             PROGBITS        ffffffff81000000 200000 1003917 00  AX  0   0 4096
+  [ 1] .text             PROGBITS        ffffffff81000000 001000 e0169a 00  AX  0   0 4096

* pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:

Quoted 8 lines

error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'

This is very rare, though.

So I suggest:

* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.

so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ

Quoted 3 lines

Place the new non-SHF_ALLOC .BTF section at the end. Then not rewriting PT_LOAD will be justified.
* Add a new option incorporating the current pahole -J + llvm-objcopy

pahole is already using llvm-objcopy. Do you mean to add a mode where pahole will dump a separate ELF object file with just .BTF? Or something else?

#2 lore raw
Quoted 60 lines · Andrii Nakryiko
Quoted 6 lines · Fangrui Song

Hi BTF folks,

I have discovered some problems with pahole -J.
Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):

Quoted 7 lines

ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o

pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?

We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.

Quoted 4 lines

When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.

I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:

$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
  [13] .BTF_ids          PROGBITS        ffffffff822b6804 14b6804
000510 00   A  0   0  1
  [43] .BTF              PROGBITS        0000000000000000 18bc0e8c
4c8056 00      0   0  1
Quoted 23 lines

This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).

In particular, there are two bugs:

* pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its sh_offset != sh_addr (mod max-page-size)

Section Headers:

    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
    [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
(quoted diff) +1 -1
-  [ 1] .text             PROGBITS        ffffffff81000000 200000 1003917 00  AX  0   0 4096
+  [ 1] .text             PROGBITS        ffffffff81000000 001000 e0169a 00  AX  0   0 4096

* pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:

error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'

This is very rare, though.

Quoted 7 lines
Quoted 3 lines

So I suggest:

* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.

so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ

Here's one things we're seeing (note, we're using a kernel based on 4.15). Before we run pahole, we have this. Notice the offset of the '.text' section:

[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 200000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c 100169c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 17caf40
000000 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 17caf40
0004c0 00 A 0 0 1

After running pahole, we get this:

[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 001000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c e0269c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 11ecf40
3c1ebe 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 15aedfe
0004c0 00 A 0 0 1

The offset of '.text' changed. This is because elf_update decides what the offsets should be. Did the linker scripts change between 4.15 and top-of-tree to mark .BTF as non-allocatable?

-bw

Quoted 7 lines
Quoted 3 lines

Place the new non-SHF_ALLOC .BTF section at the end. Then not rewriting PT_LOAD will be justified.
* Add a new option incorporating the current pahole -J + llvm-objcopy

pahole is already using llvm-objcopy. Do you mean to add a mode where pahole will dump a separate ELF object file with just .BTF? Or something else?

#3 lore raw
Quoted 100 lines · Bill Wendling
Quoted 60 lines · Andrii Nakryiko
Quoted 6 lines · Fangrui Song

Hi BTF folks,

I have discovered some problems with pahole -J.
Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):

Quoted 7 lines

ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o

pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?

We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.

Quoted 4 lines

When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.

I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:

$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
  [13] .BTF_ids          PROGBITS        ffffffff822b6804 14b6804
000510 00   A  0   0  1
  [43] .BTF              PROGBITS        0000000000000000 18bc0e8c
4c8056 00      0   0  1
Quoted 23 lines

This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).

In particular, there are two bugs:

* pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its sh_offset != sh_addr (mod max-page-size)

Section Headers:

    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
    [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
(quoted diff) +1 -1
-  [ 1] .text             PROGBITS        ffffffff81000000 200000 1003917 00  AX  0   0 4096
+  [ 1] .text             PROGBITS        ffffffff81000000 001000 e0169a 00  AX  0   0 4096

* pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:

error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'

This is very rare, though.

Quoted 7 lines
Quoted 3 lines

So I suggest:

* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.

so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ

Here's one things we're seeing (note, we're using a kernel based on 4.15). Before we run pahole, we have this. Notice the offset of the '.text' section:

[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 200000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c 100169c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 17caf40
000000 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 17caf40
0004c0 00 A 0 0 1

After running pahole, we get this:

[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 001000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c e0269c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 11ecf40
3c1ebe 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 15aedfe
0004c0 00 A 0 0 1

Quoted 3 lines

The offset of '.text' changed. This is because elf_update decides what the offsets should be. Did the linker scripts change between 4.15 and top-of-tree to mark .BTF as non-allocatable?

maybe so, but .tmp_vmlinux.btf which gets .BTF is discarded and not used anymore. We only dump .BTF section into a separate ELF, which is linked into the final vmlinux image as the next step. So pahole doesn't rewrite the final binary.

Quoted 10 lines

-bw

Quoted 7 lines
Quoted 3 lines

Place the new non-SHF_ALLOC .BTF section at the end. Then not rewriting PT_LOAD will be justified.
* Add a new option incorporating the current pahole -J + llvm-objcopy

pahole is already using llvm-objcopy. Do you mean to add a mode where pahole will dump a separate ELF object file with just .BTF? Or something else?

#4 lore raw
Quoted 102 lines · Andrii Nakryiko
Quoted 100 lines · Bill Wendling
Quoted 60 lines · Andrii Nakryiko
Quoted 6 lines · Fangrui Song

Hi BTF folks,

I have discovered some problems with pahole -J.
Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):

Quoted 7 lines

ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o

pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?

We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.

Quoted 4 lines

When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.

I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:

$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
  [13] .BTF_ids          PROGBITS        ffffffff822b6804 14b6804
000510 00   A  0   0  1
  [43] .BTF              PROGBITS        0000000000000000 18bc0e8c
4c8056 00      0   0  1
Quoted 23 lines

This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).

In particular, there are two bugs:

* pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its sh_offset != sh_addr (mod max-page-size)

Section Headers:

    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
    [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
(quoted diff) +1 -1
-  [ 1] .text             PROGBITS        ffffffff81000000 200000 1003917 00  AX  0   0 4096
+  [ 1] .text             PROGBITS        ffffffff81000000 001000 e0169a 00  AX  0   0 4096

* pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:

error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'

This is very rare, though.

Quoted 7 lines
Quoted 3 lines

So I suggest:

* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.

so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ

Here's one things we're seeing (note, we're using a kernel based on 4.15). Before we run pahole, we have this. Notice the offset of the '.text' section:

[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 200000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c 100169c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 17caf40
000000 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 17caf40
0004c0 00 A 0 0 1

After running pahole, we get this:

[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 001000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c e0269c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 11ecf40
3c1ebe 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 15aedfe
0004c0 00 A 0 0 1

Quoted 9 lines
Quoted 3 lines

The offset of '.text' changed. This is because elf_update decides what the offsets should be. Did the linker scripts change between 4.15 and top-of-tree to mark .BTF as non-allocatable?

maybe so, but .tmp_vmlinux.btf which gets .BTF is discarded and not used anymore. We only dump .BTF section into a separate ELF, which is linked into the final vmlinux image as the next step. So pahole doesn't rewrite the final binary.

Right. The problem is with the follow-up command:

        ${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \
                --strip-all ${1} ${2} 2>/dev/null

The file .tmp_vmlinux.btf isn't "correct" (see maskray's explanation), so this command fails to write a valid ELF file.

-bw

#5 lore raw
Quoted 114 lines · Bill Wendling
Quoted 102 lines · Andrii Nakryiko
Quoted 100 lines · Bill Wendling
Quoted 60 lines · Andrii Nakryiko
Quoted 6 lines · Fangrui Song

Hi BTF folks,

I have discovered some problems with pahole -J.
Its usage in kernel scripts/link-vmlinux.sh is like (make LLVM=1 bzImage):

Quoted 7 lines

ld.lld -m elf_x86_64 --emit-relocs --discard-none -z max-page-size=0x200000 --build-id=sha1 --orphan-handling=warn -o .tmp_vmlinux.btf -T ./arch/x86/kernel/vmlinux.lds --whole-archive arch/x86/kernel/head_64.o arch/x86/kernel/head64.o arch/x86/kernel/ebda.o arch/x86/kernel/platform-quirks.o init/built-in.a usr/built-in.a arch/x86/built-in.a kernel/built-in.a certs/built-in.a mm/built-in.a fs/built-in.a ipc/built-in.a security/built-in.a crypto/built-in.a block/built-in.a lib/built-in.a arch/x86/lib/built-in.a lib/lib.a arch/x86/lib/lib.a drivers/built-in.a sound/built-in.a net/built-in.a virt/built-in.a arch/x86/pci/built-in.a arch/x86/power/built-in.a arch/x86/video/built-in.a --no-whole-archive --start-group --end-group pahole -J .tmp_vmlinux.btf
llvm-objcopy --only-section=.BTF --set-section-flags .BTF=alloc,readonly --strip-all .tmp_vmlinux.btf .btf.vmlinux.bin.o

pahole -J adds .BTF and rewrites .tmp_vmlinux.btf, then llvm-objcopy produces .btf.vmlinux.bin.o of just one section. Why doesn't pahole provide a command generating an object file with just the .BTF section?

We just recently discussed adding this. So the reason is that historically pahole never had this feature and no one bothered to add it.

Quoted 4 lines

When I contributed https://git.kernel.org/linus/90ceddcb495008ac8ba7a3dce297841efcd7d584 , I remember pahole at that time added a non-SHF_ALLOC .BTF , now (1.20) .BTF becomes SHF_ALLOC.

I don't think anything changed in 1.20 about how .BTF is added. pahole still uses the same llvm-objcopt and it doesn't add SHF_ALLOC. And I just double-checked that after running pahole -j .tmp_vmlinux.btf has non-allocatable .BTF section:

$ llvm-readelf -S ~/tmp/pahole-vmlinux-output.o | grep BTF
  [13] .BTF_ids          PROGBITS        ffffffff822b6804 14b6804
000510 00   A  0   0  1
  [43] .BTF              PROGBITS        0000000000000000 18bc0e8c
4c8056 00      0   0  1
Quoted 23 lines

This is problematic if pahole does not have full-fledged binary manipulation ability (objcopy,llvm-objcopy).

In particular, there are two bugs:

* pahole does not respect max-page-size (p_align of PT_LOAD). See the .text section, its sh_offset != sh_addr (mod max-page-size)

Section Headers:

    [Nr] Name              Type            Address          Off    Size   ES Flg Lk Inf Al
    [ 0]                   NULL            0000000000000000 000000 000000 00      0   0  0
(quoted diff) +1 -1
-  [ 1] .text             PROGBITS        ffffffff81000000 200000 1003917 00  AX  0   0 4096
+  [ 1] .text             PROGBITS        ffffffff81000000 001000 e0169a 00  AX  0   0 4096

* pahole does not rewrite p_offset/p_filesz of PT_LOAD segments.
Because of the first bug, pahole -J rewritten object file generally has small offsets.
If p_offset/p_filesz of PT_LOAD segments are not rewritten, the file offset range of .symtab may be within a PT_LOAD range. llvm-objcopy --strip-all considers .symtab as part of the PT_LOAD and refuses --strip-all:

error: '.tmp_vmlinux.btf': string table '.strtab' cannot be removed because it is referenced by the symbol table '.symtab'

This is very rare, though.

Quoted 7 lines
Quoted 3 lines

So I suggest:

* pahole -J: restore the previous non-SHF_ALLOC behavior. Don't rewrite sh_offset of existing sections.

so nothing changed (at least as of 1.20) about how pahole adds .BTF, so I'd like to understand why our observations differ

Here's one things we're seeing (note, we're using a kernel based on 4.15). Before we run pahole, we have this. Notice the offset of the '.text' section:

[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 200000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c 100169c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 17caf40
000000 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 17caf40
0004c0 00 A 0 0 1

After running pahole, we get this:

[ 0] NULL 0000000000000000 000000
000000 00 0 0 0
[ 1] .text PROGBITS ffffffff81000000 001000
e0169a 00 AX 0 0 4096
[ 2] .notes NOTE ffffffff81e0169c e0269c
000024 00 A 0 0 4
...
[24] .BTF PROGBITS ffffffff825caf40 11ecf40
3c1ebe 00 WA 0 0 1
[25] .BTF_ids PROGBITS ffffffff825caf40 15aedfe
0004c0 00 A 0 0 1

Quoted 9 lines
Quoted 3 lines

The offset of '.text' changed. This is because elf_update decides what the offsets should be. Did the linker scripts change between 4.15 and top-of-tree to mark .BTF as non-allocatable?

maybe so, but .tmp_vmlinux.btf which gets .BTF is discarded and not used anymore. We only dump .BTF section into a separate ELF, which is linked into the final vmlinux image as the next step. So pahole doesn't rewrite the final binary.

Quoted 7 lines

Right. The problem is with the follow-up command:

        ${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \
                --strip-all ${1} ${2} 2>/dev/null

The file .tmp_vmlinux.btf isn't "correct" (see maskray's explanation), so this command fails to write a valid ELF file.

Oh, I see. Yeah, I didn't get that from the original email. So seems like llvm-objcopy that pahole invokes internally will corrupt that .tmp_vmlinux.btf file to the point of another objcopy not being able to handle it. Interesting.

But regardless, as I said, we already discussed this feature and think that it's a good addition. I'd like to be able to both get object file with just .BTF (for kernel linking process) and pure binary contents of .BTF section (for other, CO-RE-related reasons, especially for legacy systems). I think Arnaldo is onboard as well. So please feel free to send patches.

Quoted 2 lines

-bw