Skip to contentSkip to conversation

docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

PATCH Superseded Yonghong Song 4 participants 9 messages May 22, 2021 – May 25, 2021

Subsystems (2)

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations

LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.

Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.

Cc John Fastabend <[email protected]>
Cc Lorenz Bauer <[email protected]>
---
 Documentation/bpf/index.rst      |   1 +
 Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
 2 files changed, 169 insertions(+)
 create mode 100644 Documentation/bpf/llvm_reloc.rst
Documentation/bpf/index.rst +1 -0
index a702f67dd45f..93e8cf12a6d4 100644
@@ -84,6 +84,7 @@ Other
    :maxdepth: 1
 
    ringbuf
+   llvm_reloc
 
 .. Links:
 .. _networking-filter: ../networking/filter.rst
Documentation/bpf/llvm_reloc.rst new file +168 -0

168 lines not on this page · raw message

Signature · 1 line
2.30.2
#1 lore raw
Quoted 6 lines · Yonghong Song

LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.

Daniel, John and Lorenz,

Could you help check how the new relocation scheme may impact you? libbpf has a similar issue and is fixed by
https://lore.kernel.org/bpf/[email protected]/
In most cases, you should just change relocation enum number, no relocation resolution is changed.

Please let me know. Thanks!

Quoted 14 lines

Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.

Cc John Fastabend <[email protected]>
Cc Lorenz Bauer <[email protected]>
---
  Documentation/bpf/index.rst      |   1 +
  Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
  2 files changed, 169 insertions(+)

create mode 100644 Documentation/bpf/llvm_reloc.rst

[..]

#2 lore raw
Quoted 10 lines · Yonghong Song

Daniel, John and Lorenz,

Could you help check how the new relocation scheme may impact you? libbpf has a similar issue and is fixed by
https://lore.kernel.org/bpf/[email protected]/
In most cases, you should just change relocation enum number, no relocation resolution is changed.

Please let me know. Thanks!

Thank you for the heads up :) cilium/ebpf currently doesn't look at relocation types at all for better or worse. We simply collect "well-known" sections like maps, programs, etc. and only process relocations for these. So your change won't break cilium/ebpf, but it makes me wonder whether we should check the relocation type.

Best
Lorenz

Signature · 4 lines
Lorenz Bauer  |  Systems Engineer
6th Floor, County Hall/The Riverside Building, SE1 7PB, UK

www.cloudflare.com
#3 lore raw
Quoted 10 lines · Lorenz Bauer
Quoted 9 lines · Yonghong Song

Daniel, John and Lorenz,

Could you help check how the new relocation scheme may impact you? libbpf has a similar issue and is fixed by

    https://lore.kernel.org/bpf/[email protected]/

In most cases, you should just change relocation enum number, no relocation resolution is changed.

Quoted 6 lines
Quoted 1 line

Please let me know. Thanks!

Thank you for the heads up :) cilium/ebpf currently doesn't look at relocation types at all for better or worse. We simply collect "well-known" sections like maps, programs, etc. and only process relocations for these. So your change won't break cilium/ebpf, but it

Thanks for confirmation. So yes, you should be fine though.

Quoted 1 line

makes me wonder whether we should check the relocation type.

If the library is used in a control environment, e.g., the object file is generated by llvm, bpftool linker, etc. You should be fine. But yes, checking relocation types will make the library more robust.

Quoted 4 lines

Best
Lorenz

#4 lore raw
In reply to Yonghong Song
Quoted 66 lines · Yonghong Song

LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.

Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.

Cc John Fastabend <[email protected]>
Cc Lorenz Bauer <[email protected]>
---
 Documentation/bpf/index.rst      |   1 +
 Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
 2 files changed, 169 insertions(+)
 create mode 100644 Documentation/bpf/llvm_reloc.rst
Documentation/bpf/index.rst +1 -0
index a702f67dd45f..93e8cf12a6d4 100644
@@ -84,6 +84,7 @@ Other
    :maxdepth: 1
 
    ringbuf
+   llvm_reloc
 
 .. Links:
 .. _networking-filter: ../networking/filter.rst
Documentation/bpf/llvm_reloc.rst new file +27 -0
new file mode 100644
index 000000000000..bc62bce591b1
@@ -0,0 +1,168 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+====================
+BPF LLVM Relocations
+====================
+
+This document describes LLVM BPF backend relocation types.
+
+Relocation Record
+=================
+
+LLVM BPF backend records each relocation with the following 16-byte
+ELF structure::
+
+  typedef struct
+  {
+    Elf64_Addr    r_offset;  // Offset from the beginning of section.
+    Elf64_Xword   r_info;    // Relocation type and symbol index.
+  } Elf64_Rel;
+
+For static function/variable references, the symbol often refers to
+the section itself which has a value of 0. To identify actual static
+function/variable, its section offset or some computation result
+based on section offset is written to the original insn/data buffer,
+which is called ``IA`` (implicit addend) below.  For global
+function/variables, the symbol refers to actual global and the implicit
+addend is 0.

Documentation/bpf/llvm_reloc.rst line 37

Quoted 10 lines
Documentation/bpf/llvm_reloc.rst new file +10 -0
+
+Different Relocation Types
+==========================
+
+Six relocation types are supported. The following is an overview and
+``S`` represents the value of the symbol in the symbol table::
+
+  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
+  0     R_BPF_NONE         None
+  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA

There are cases where we set all 64-bits of ld_imm64 (e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.

Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)

Quoted 37 lines
Documentation/bpf/llvm_reloc.rst new file +37 -0
+  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
+  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
+  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
+  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
+
+For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
+The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
+data bitsize is 32 (4 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
+The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
+bitsize is 64 (8 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
+But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
+``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
+is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
+to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
+become unusable by bcc and kernel.
+
+Type ``R_BPF_64_32`` is used for call instruction. The call target section
+offset is stored at ``r_offset + 4`` (32bit) and calculated as
+``(S + IA) / 8 - 1``.
+
+Examples
+========
+
+Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``
+and ``call`` instructions. For example::
+
+  __attribute__((noinline)) __attribute__((section("sec1")))
+  int gfunc(int a, int b) {
+    return a * b;
+  }
+  static __attribute__((noinline)) __attribute__((section("sec1")))

Documentation/bpf/llvm_reloc.rst line 84

Quoted 10 lines
Documentation/bpf/llvm_reloc.rst new file +10 -0
+  int lfunc(int a, int b) {
+    return a + b;
+  }
+  int global __attribute__((section("sec2")));
+  int test(int a, int b) {
+    return gfunc(a, b) +  lfunc(a, b) + global;
+  }
+
+Compiled with ``clang -target bpf -O2 -c test.c``, we will have
+following code with `llvm-objdump -d test.o``::

I recently learned about llvm-objdump -dr test.o, which shows relocations inline, it would be nice to use that output here.

Quoted 87 lines
Documentation/bpf/llvm_reloc.rst new file +84 -1
+
+  Disassembly of section .text:
+
+  0000000000000000 <test>:
+         0:       bf 26 00 00 00 00 00 00 r6 = r2
+         1:       bf 17 00 00 00 00 00 00 r7 = r1
+         2:       85 10 00 00 ff ff ff ff call -1
+         3:       bf 08 00 00 00 00 00 00 r8 = r0
+         4:       bf 71 00 00 00 00 00 00 r1 = r7
+         5:       bf 62 00 00 00 00 00 00 r2 = r6
+         6:       85 10 00 00 02 00 00 00 call 2
+         7:       0f 80 00 00 00 00 00 00 r0 += r8
+         8:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
+        10:       61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
+        11:       0f 10 00 00 00 00 00 00 r0 += r1
+        12:       95 00 00 00 00 00 00 00 exit
+
+  Disassembly of section sec1:
+
+  0000000000000000 <gfunc>:
+         0:       bf 20 00 00 00 00 00 00 r0 = r2
+         1:       2f 10 00 00 00 00 00 00 r0 *= r1
+         2:       95 00 00 00 00 00 00 00 exit
+
+  0000000000000018 <lfunc>:
+         3:       bf 20 00 00 00 00 00 00 r0 = r2
+         4:       0f 10 00 00 00 00 00 00 r0 += r1
+         5:       95 00 00 00 00 00 00 00 exit
+
+Three relocations are generated with ``llvm-readelf -r test.o``::
+
+  Relocation section '.rel.text' at offset 0x188 contains 3 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000010  000000040000000a R_BPF_64_32            0000000000000000 gfunc
+  0000000000000030  000000020000000a R_BPF_64_32            0000000000000000 sec1
+  0000000000000040  0000000600000001 R_BPF_64_64            0000000000000000 global
+
+The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,
+so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.
+The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section
+offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.
+
+The following is an example to show how R_BPF_64_ABS64 could be generated::
+
+  int global() { return 0; }
+  struct t { void *g; } gbl = { global };
+
+Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a
+relocation below in ``.data`` section with command
+``llvm-readelf -r test.o``::
+
+  Relocation section '.rel.data' at offset 0x458 contains 1 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000000  0000000700000002 R_BPF_64_ABS64         0000000000000000 global
+
+The relocation says the first 8-byte of ``.data`` section should be
+filled with address of ``global`` variable.
+
+With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of
+``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::
+
+  Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000006  0000000300000003 R_BPF_64_ABS32         0000000000000000 .debug_abbrev
+  000000000000000c  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  0000000000000012  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  0000000000000016  0000000600000003 R_BPF_64_ABS32         0000000000000000 .debug_line
+  000000000000001a  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  000000000000001e  0000000200000002 R_BPF_64_ABS64         0000000000000000 .text
+  000000000000002b  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  0000000000000037  0000000800000002 R_BPF_64_ABS64         0000000000000000 gbl
+  0000000000000040  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  ......
+
+The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
+
+  Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000084  0000000800000004 R_BPF_64_NODYLD32      0000000000000000 gbl
+
+  Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  000000000000002c  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
+  0000000000000040  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
--

2.30.2

#5 lore raw
Quoted 73 lines · Andrii Nakryiko
Quoted 72 lines · Yonghong Song

LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.

Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.

Cc John Fastabend <[email protected]>
Cc Lorenz Bauer <[email protected]>
---
  Documentation/bpf/index.rst      |   1 +
  Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
  2 files changed, 169 insertions(+)

create mode 100644 Documentation/bpf/llvm_reloc.rst

Documentation/bpf/index.rst +1 -0
index a702f67dd45f..93e8cf12a6d4 100644
@@ -84,6 +84,7 @@ Other
     :maxdepth: 1
 
     ringbuf
+   llvm_reloc
 
  .. Links:
  .. _networking-filter: ../networking/filter.rst
Documentation/bpf/llvm_reloc.rst new file +33 -0
new file mode 100644
index 000000000000..bc62bce591b1
@@ -0,0 +1,168 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+====================
+BPF LLVM Relocations
+====================
+
+This document describes LLVM BPF backend relocation types.
+
+Relocation Record
+=================
+
+LLVM BPF backend records each relocation with the following 16-byte
+ELF structure::
+
+  typedef struct
+  {
+    Elf64_Addr    r_offset;  // Offset from the beginning of section.
+    Elf64_Xword   r_info;    // Relocation type and symbol index.
+  } Elf64_Rel;
+
+For static function/variable references, the symbol often refers to
+the section itself which has a value of 0. To identify actual static
+function/variable, its section offset or some computation result
+based on section offset is written to the original insn/data buffer,
+which is called ``IA`` (implicit addend) below.  For global
+function/variables, the symbol refers to actual global and the implicit
+addend is 0.
+
+Different Relocation Types
+==========================
+
+Six relocation types are supported. The following is an overview and
+``S`` represents the value of the symbol in the symbol table::
Quoted 8 lines

Documentation/bpf/llvm_reloc.rst line 37

Quoted 4 lines
Documentation/bpf/llvm_reloc.rst new file +4 -0
+
+  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
+  0     R_BPF_NONE         None
+  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA

There are cases where we set all 64-bits of ld_imm64 (e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.

It is still R_BPF_64_64. In llvm, we have restriction that section offset must be <= UINT32_MAX, and that is why only 32bit is used to find the actual symbol in symbol table. 32bit permits 4GB section which should enough in practice for a bpf program.

libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.

The name is a little bit misleading, but it has become part of ABI and lives in /usr/include/elf.h and we are not able to change it any more.

Quoted 4 lines

Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)

We have llvm/test/CodeGen/BPF/reloc.ll and llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think I can enhance llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.

Quoted 43 lines
Quoted 43 lines
Documentation/bpf/llvm_reloc.rst new file +42 -0
+  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
+  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
+  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
+  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
+
+For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
+The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
+data bitsize is 32 (4 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
+The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
+bitsize is 64 (8 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
+But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
+``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
+is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
+to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
+become unusable by bcc and kernel.
+
+Type ``R_BPF_64_32`` is used for call instruction. The call target section
+offset is stored at ``r_offset + 4`` (32bit) and calculated as
+``(S + IA) / 8 - 1``.
+
+Examples
+========
+
+Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64``
+and ``call`` instructions. For example::
+
+  __attribute__((noinline)) __attribute__((section("sec1")))
+  int gfunc(int a, int b) {
+    return a * b;
+  }
+  static __attribute__((noinline)) __attribute__((section("sec1")))
+  int lfunc(int a, int b) {
+    return a + b;
+  }
+  int global __attribute__((section("sec2")));
+  int test(int a, int b) {
Quoted 8 lines

Documentation/bpf/llvm_reloc.rst line 84

Quoted 5 lines
Documentation/bpf/llvm_reloc.rst new file +5 -0
+    return gfunc(a, b) +  lfunc(a, b) + global;
+  }
+
+Compiled with ``clang -target bpf -O2 -c test.c``, we will have
+following code with `llvm-objdump -d test.o``::

I recently learned about llvm-objdump -dr test.o, which shows relocations inline, it would be nice to use that output here.

Yes, will do.

Quoted 88 lines
Quoted 88 lines
Documentation/bpf/llvm_reloc.rst new file +84 -1
+
+  Disassembly of section .text:
+
+  0000000000000000 <test>:
+         0:       bf 26 00 00 00 00 00 00 r6 = r2
+         1:       bf 17 00 00 00 00 00 00 r7 = r1
+         2:       85 10 00 00 ff ff ff ff call -1
+         3:       bf 08 00 00 00 00 00 00 r8 = r0
+         4:       bf 71 00 00 00 00 00 00 r1 = r7
+         5:       bf 62 00 00 00 00 00 00 r2 = r6
+         6:       85 10 00 00 02 00 00 00 call 2
+         7:       0f 80 00 00 00 00 00 00 r0 += r8
+         8:       18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll
+        10:       61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0)
+        11:       0f 10 00 00 00 00 00 00 r0 += r1
+        12:       95 00 00 00 00 00 00 00 exit
+
+  Disassembly of section sec1:
+
+  0000000000000000 <gfunc>:
+         0:       bf 20 00 00 00 00 00 00 r0 = r2
+         1:       2f 10 00 00 00 00 00 00 r0 *= r1
+         2:       95 00 00 00 00 00 00 00 exit
+
+  0000000000000018 <lfunc>:
+         3:       bf 20 00 00 00 00 00 00 r0 = r2
+         4:       0f 10 00 00 00 00 00 00 r0 += r1
+         5:       95 00 00 00 00 00 00 00 exit
+
+Three relocations are generated with ``llvm-readelf -r test.o``::
+
+  Relocation section '.rel.text' at offset 0x188 contains 3 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000010  000000040000000a R_BPF_64_32            0000000000000000 gfunc
+  0000000000000030  000000020000000a R_BPF_64_32            0000000000000000 sec1
+  0000000000000040  0000000600000001 R_BPF_64_64            0000000000000000 global
+
+The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0,
+so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``.
+The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section
+offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``.
+
+The following is an example to show how R_BPF_64_ABS64 could be generated::
+
+  int global() { return 0; }
+  struct t { void *g; } gbl = { global };
+
+Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a
+relocation below in ``.data`` section with command
+``llvm-readelf -r test.o``::
+
+  Relocation section '.rel.data' at offset 0x458 contains 1 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000000  0000000700000002 R_BPF_64_ABS64         0000000000000000 global
+
+The relocation says the first 8-byte of ``.data`` section should be
+filled with address of ``global`` variable.
+
+With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of
+``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations::
+
+  Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000006  0000000300000003 R_BPF_64_ABS32         0000000000000000 .debug_abbrev
+  000000000000000c  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  0000000000000012  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  0000000000000016  0000000600000003 R_BPF_64_ABS32         0000000000000000 .debug_line
+  000000000000001a  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  000000000000001e  0000000200000002 R_BPF_64_ABS64         0000000000000000 .text
+  000000000000002b  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  0000000000000037  0000000800000002 R_BPF_64_ABS64         0000000000000000 gbl
+  0000000000000040  0000000400000003 R_BPF_64_ABS32         0000000000000000 .debug_str
+  ......
+
+The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations::
+
+  Relocation section '.rel.BTF' at offset 0x538 contains 1 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  0000000000000084  0000000800000004 R_BPF_64_NODYLD32      0000000000000000 gbl
+
+  Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries:
+      Offset             Info             Type               Symbol's Value  Symbol's Name
+  000000000000002c  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
+  0000000000000040  0000000200000004 R_BPF_64_NODYLD32      0000000000000000 .text
--

2.30.2

#6 lore raw
Quoted 86 lines · Yonghong Song
Quoted 73 lines · Andrii Nakryiko
Quoted 72 lines · Yonghong Song

LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.

Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.

Cc John Fastabend <[email protected]>
Cc Lorenz Bauer <[email protected]>
---
  Documentation/bpf/index.rst      |   1 +
  Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
  2 files changed, 169 insertions(+)

create mode 100644 Documentation/bpf/llvm_reloc.rst

Documentation/bpf/index.rst +1 -0
index a702f67dd45f..93e8cf12a6d4 100644
@@ -84,6 +84,7 @@ Other
     :maxdepth: 1
 
     ringbuf
+   llvm_reloc
 
  .. Links:
  .. _networking-filter: ../networking/filter.rst
Documentation/bpf/llvm_reloc.rst new file +33 -0
new file mode 100644
index 000000000000..bc62bce591b1
@@ -0,0 +1,168 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+====================
+BPF LLVM Relocations
+====================
+
+This document describes LLVM BPF backend relocation types.
+
+Relocation Record
+=================
+
+LLVM BPF backend records each relocation with the following 16-byte
+ELF structure::
+
+  typedef struct
+  {
+    Elf64_Addr    r_offset;  // Offset from the beginning of section.
+    Elf64_Xword   r_info;    // Relocation type and symbol index.
+  } Elf64_Rel;
+
+For static function/variable references, the symbol often refers to
+the section itself which has a value of 0. To identify actual static
+function/variable, its section offset or some computation result
+based on section offset is written to the original insn/data buffer,
+which is called ``IA`` (implicit addend) below.  For global
+function/variables, the symbol refers to actual global and the implicit
+addend is 0.
+
+Different Relocation Types
+==========================
+
+Six relocation types are supported. The following is an overview and
+``S`` represents the value of the symbol in the symbol table::
Quoted 8 lines

Documentation/bpf/llvm_reloc.rst line 37

Quoted 4 lines
Documentation/bpf/llvm_reloc.rst new file +4 -0
+
+  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
+  0     R_BPF_NONE         None
+  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA

There are cases where we set all 64-bits of ld_imm64 (e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.

Quoted 7 lines

It is still R_BPF_64_64. In llvm, we have restriction that section offset must be <= UINT32_MAX, and that is why only 32bit is used to find the actual symbol in symbol table. 32bit permits 4GB section which should enough in practice for a bpf program.

libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.

Ok, sounds good.

Quoted 5 lines

The name is a little bit misleading, but it has become part of ABI and lives in /usr/include/elf.h and we are not able to change it any more.

Quoted 1 line
Quoted 9 lines
Quoted 3 lines

Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)

We have llvm/test/CodeGen/BPF/reloc.ll and llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think I can enhance
llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.

Great, thanks.

Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Documentation/bpf/llvm_reloc.rst new file +15 -0
+  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
+  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
+  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
+  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
+
+For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
+The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
+data bitsize is 32 (4 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
+The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
+bitsize is 64 (8 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+

[...]

#7 lore raw
In reply to Yonghong Song
Quoted 32 lines · Yonghong Song
Quoted 29 lines · Andrii Nakryiko
Quoted 28 lines · Yonghong Song

LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.

Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.

Cc John Fastabend <[email protected]>
Cc Lorenz Bauer <[email protected]>
---
  Documentation/bpf/index.rst      |   1 +
  Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
  2 files changed, 169 insertions(+)

create mode 100644 Documentation/bpf/llvm_reloc.rst

Documentation/bpf/index.rst +0 -0
index a702f67dd45f..93e8cf12a6d4 100644
@@ -84,6 +84,7 @@ Other
     :maxdepth: 1
 
Quoted 3 lines
Quoted 3 lines

Documentation/bpf/index.rst Other · line 88

Quoted 3 lines
Documentation/bpf/index.rst +1 -0
     ringbuf
+   llvm_reloc
 

Thanks Yonghong, I found this helpful. I still had to crack open llvm code though to follow along. A couple small suggestions below, may or may not be useful. Overall looks good.

Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Documentation/bpf/index.rst +0 -0
  .. Links:
  .. _networking-filter: ../networking/filter.rst
Documentation/bpf/llvm_reloc.rst new file +19 -0
new file mode 100644
index 000000000000..bc62bce591b1
@@ -0,0 +1,168 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+====================
+BPF LLVM Relocations
+====================
+
+This document describes LLVM BPF backend relocation types.
+
+Relocation Record
+=================
+
+LLVM BPF backend records each relocation with the following 16-byte
+ELF structure::
+
+  typedef struct
+  {
+    Elf64_Addr    r_offset;  // Offset from the beginning of section.
+    Elf64_Xword   r_info;    // Relocation type and symbol index.
+  } Elf64_Rel;
Quoted 8 lines
Quoted 8 lines

Documentation/bpf/llvm_reloc.rst line 27

Quoted 8 lines
Documentation/bpf/llvm_reloc.rst new file +8 -0
+
+For static function/variable references, the symbol often refers to
+the section itself which has a value of 0. To identify actual static
+function/variable, its section offset or some computation result
+based on section offset is written to the original insn/data buffer,
+which is called ``IA`` (implicit addend) below.  For global
+function/variables, the symbol refers to actual global and the implicit
+addend is 0.

Above was too terse for me to follow without looking into some clang examples. Maybe an example right here would help not sure? Maybe expand the text a bit? I don't have a really good suggestion.

Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Documentation/bpf/llvm_reloc.rst new file +6 -0
+
+Different Relocation Types
+==========================
+
+Six relocation types are supported. The following is an overview and
+``S`` represents the value of the symbol in the symbol table::
Quoted 8 lines

Documentation/bpf/llvm_reloc.rst line 37

Quoted 4 lines
Documentation/bpf/llvm_reloc.rst new file +4 -0
+
+  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
+  0     R_BPF_NONE         None
+  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA

There are cases where we set all 64-bits of ld_imm64 (e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.

Quoted 4 lines

It is still R_BPF_64_64. In llvm, we have restriction that section offset must be <= UINT32_MAX, and that is why only 32bit is used to find the actual symbol in symbol table. 32bit permits 4GB section which should enough in practice for a bpf program.

^^^ maybe add this note in the doc somewhere? I had similar questions.

Quoted 8 lines

libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.

The name is a little bit misleading, but it has become part of ABI and lives in /usr/include/elf.h and we are not able to change it any more.

Quoted 1 line
Quoted 8 lines
Quoted 3 lines

Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)

We have llvm/test/CodeGen/BPF/reloc.ll and llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think I can enhance
llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.

^^^ maybe cross-reference llvm tests from kernel docs side? I often look at these when I get something unexpected/unknown maybe others would find it helpful, but not know where to look?

Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Documentation/bpf/llvm_reloc.rst new file +21 -0
+  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
+  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
+  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
+  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
+
+For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
+The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
+data bitsize is 32 (4 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
+The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
+bitsize is 64 (8 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
+But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
+``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
+is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
+to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
+become unusable by bcc and kernel.
Quoted 8 lines
Quoted 8 lines

Documentation/bpf/llvm_reloc.rst line 66

Quoted 8 lines
Documentation/bpf/llvm_reloc.rst new file +8 -0
+
+Type ``R_BPF_64_32`` is used for call instruction. The call target section
+offset is stored at ``r_offset + 4`` (32bit) and calculated as
+``(S + IA) / 8 - 1``.
+
+Examples
+========
+

I liked the examples.

#8 lore raw
Quoted 72 lines · John Fastabend
Quoted 32 lines · Yonghong Song
Quoted 29 lines · Andrii Nakryiko
Quoted 28 lines · Yonghong Song

LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.

Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.

Cc John Fastabend <[email protected]>
Cc Lorenz Bauer <[email protected]>
---
   Documentation/bpf/index.rst      |   1 +
   Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++
   2 files changed, 169 insertions(+)

create mode 100644 Documentation/bpf/llvm_reloc.rst

Documentation/bpf/index.rst +0 -0
index a702f67dd45f..93e8cf12a6d4 100644
@@ -84,6 +84,7 @@ Other
      :maxdepth: 1
 
Quoted 3 lines
Quoted 3 lines

Documentation/bpf/index.rst Other · line 88

Quoted 3 lines
Documentation/bpf/index.rst +1 -0
      ringbuf
+   llvm_reloc
 

Thanks Yonghong, I found this helpful. I still had to crack open llvm code though to follow along. A couple small suggestions below, may or may not be useful. Overall looks good.

Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Documentation/bpf/index.rst +0 -0
   .. Links:
   .. _networking-filter: ../networking/filter.rst
Documentation/bpf/llvm_reloc.rst new file +23 -0
new file mode 100644
index 000000000000..bc62bce591b1
@@ -0,0 +1,168 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+====================
+BPF LLVM Relocations
+====================
+
+This document describes LLVM BPF backend relocation types.
+
+Relocation Record
+=================
+
+LLVM BPF backend records each relocation with the following 16-byte
+ELF structure::
+
+  typedef struct
+  {
+    Elf64_Addr    r_offset;  // Offset from the beginning of section.
+    Elf64_Xword   r_info;    // Relocation type and symbol index.
+  } Elf64_Rel;
+
+For static function/variable references, the symbol often refers to
+the section itself which has a value of 0. To identify actual static
+function/variable, its section offset or some computation result
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines

Documentation/bpf/llvm_reloc.rst line 27

Quoted 4 lines
Documentation/bpf/llvm_reloc.rst new file +4 -0
+based on section offset is written to the original insn/data buffer,
+which is called ``IA`` (implicit addend) below.  For global
+function/variables, the symbol refers to actual global and the implicit
+addend is 0.

Above was too terse for me to follow without looking into some clang examples. Maybe an example right here would help not sure? Maybe expand the text a bit? I don't have a really good suggestion.

Just send a new revision with an example. Hope it will make it easy to understand the above ``IA`` concept.

Quoted 16 lines
Quoted 16 lines
Quoted 15 lines

Documentation/bpf/llvm_reloc.rst line 37

Quoted 11 lines
Documentation/bpf/llvm_reloc.rst new file +10 -0
+
+Different Relocation Types
+==========================
+
+Six relocation types are supported. The following is an overview and
+``S`` represents the value of the symbol in the symbol table::
+
+  Enum  ELF Reloc Type     Description      BitSize  Offset        Calculation
+  0     R_BPF_NONE         None
+  1     R_BPF_64_64        ld_imm64 insn    32       r_offset + 4  S + IA

There are cases where we set all 64-bits of ld_imm64 (e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.

Quoted 6 lines
Quoted 4 lines

It is still R_BPF_64_64. In llvm, we have restriction that section offset must be <= UINT32_MAX, and that is why only 32bit is used to find the actual symbol in symbol table. 32bit permits 4GB section which should enough in practice for a bpf program.

^^^ maybe add this note in the doc somewhere? I had similar questions.

Added in the new revision.

Quoted 13 lines
Quoted 13 lines

libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.

The name is a little bit misleading, but it has become part of ABI and lives in /usr/include/elf.h and we are not able to change it any more.

Quoted 4 lines

Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)

Quoted 8 lines
Quoted 4 lines

We have llvm/test/CodeGen/BPF/reloc.ll and llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think I can enhance
llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.

^^^ maybe cross-reference llvm tests from kernel docs side? I often look at these when I get something unexpected/unknown maybe others would find it helpful, but not know where to look?

The llvm patch has not merged. We need to merge libbpf patch first. Otherwise, nightly libbpf CI will fail. But this doc includes a link to the LLVM patch and you can just go to that llvm patch to find
examples!

Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Documentation/bpf/llvm_reloc.rst new file +23 -0
+  2     R_BPF_64_ABS64     normal data      64       r_offset      S + IA
+  3     R_BPF_64_ABS32     normal data      32       r_offset      S + IA
+  4     R_BPF_64_NODYLD32  .BTF[.ext] data  32       r_offset      S + IA
+  10    R_BPF_64_32        call insn        32       r_offset + 4  (S + IA) / 8 - 1
+
+For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction.
+The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write
+data bitsize is 32 (4 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data.
+The actual to-be-relocated data is stored at ``r_offset`` and the read/write data
+bitsize is 64 (8 bytes). The relocation can be resolved with
+the symbol value plus implicit addend.
+
+Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data.
+But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and
+``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld``
+is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved
+to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext``
+become unusable by bcc and kernel.
+
+Type ``R_BPF_64_32`` is used for call instruction. The call target section
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines

Documentation/bpf/llvm_reloc.rst line 66

Quoted 6 lines
Documentation/bpf/llvm_reloc.rst new file +6 -0
+offset is stored at ``r_offset + 4`` (32bit) and calculated as
+``(S + IA) / 8 - 1``.
+
+Examples
+========
+

I liked the examples.

Great. Just added one more in the new revision!