docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
Subsystems (2)
PATCH 1/1 docs/bpf: add llvm_reloc.rst to explain llvm bpf relocations
LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.
Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.
--- Documentation/bpf/index.rst | 1 + Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+) create mode 100644 Documentation/bpf/llvm_reloc.rst
Documentation/bpf/index.rst +1 -0
@@ -84,6 +84,7 @@ Other :maxdepth: 1 ringbuf + llvm_reloc .. Links: .. _networking-filter: ../networking/filter.rst
Documentation/bpf/llvm_reloc.rst new file +168 -0
168 lines not on this page · raw message
Signature · 1 line
2.30.2
Quoted 6 lines · Yonghong Song
LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.
Daniel, John and Lorenz,
Could you help check how the new relocation scheme may impact you? libbpf has a similar issue and is fixed by
https://lore.kernel.org/bpf/[email protected]/
In most cases, you should just change relocation enum number, no relocation resolution is changed.
Please let me know. Thanks!
Quoted 14 lines
Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.
Cc John Fastabend <[email protected]>Cc Lorenz Bauer <[email protected]>Signed-off-by Yonghong Song <[email protected]>--- Documentation/bpf/index.rst | 1 + Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+)create mode 100644
Documentation/bpf/llvm_reloc.rst
[..]
Quoted 10 lines · Yonghong Song
Daniel, John and Lorenz,
Could you help check how the new relocation scheme may impact you? libbpf has a similar issue and is fixed by
https://lore.kernel.org/bpf/[email protected]/
In most cases, you should just change relocation enum number, no relocation resolution is changed.Please let me know. Thanks!
Thank you for the heads up :) cilium/ebpf currently doesn't look at relocation types at all for better or worse. We simply collect "well-known" sections like maps, programs, etc. and only process relocations for these. So your change won't break cilium/ebpf, but it makes me wonder whether we should check the relocation type.
Best
Lorenz
Signature · 4 lines
Lorenz Bauer | Systems Engineer 6th Floor, County Hall/The Riverside Building, SE1 7PB, UK www.cloudflare.com
Quoted 10 lines · Lorenz Bauer
Quoted 9 lines · Yonghong Song
Daniel, John and Lorenz,
Could you help check how the new relocation scheme may impact you? libbpf has a similar issue and is fixed by
https://lore.kernel.org/bpf/[email protected]/In most cases, you should just change relocation enum number, no relocation resolution is changed.
Quoted 6 lines
Quoted 1 line
Please let me know. Thanks!
Thank you for the heads up :) cilium/ebpf currently doesn't look at relocation types at all for better or worse. We simply collect "well-known" sections like maps, programs, etc. and only process relocations for these. So your change won't break cilium/ebpf, but it
Thanks for confirmation. So yes, you should be fine though.
Quoted 1 line
makes me wonder whether we should check the relocation type.
If the library is used in a control environment, e.g., the object file is generated by llvm, bpftool linker, etc. You should be fine. But yes, checking relocation types will make the library more robust.
Quoted 4 lines
Best
Lorenz
Quoted 66 lines · Yonghong Song
LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.
Cc John Fastabend <[email protected]>Cc Lorenz Bauer <[email protected]>Signed-off-by Yonghong Song <[email protected]>--- Documentation/bpf/index.rst | 1 + Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+) create mode 100644 Documentation/bpf/llvm_reloc.rstDocumentation/bpf/index.rst +1
-0@@ -84,6 +84,7 @@ Other :maxdepth: 1 ringbuf + llvm_reloc .. Links: .. _networking-filter: ../networking/filter.rstDocumentation/bpf/llvm_reloc.rst new file +27
-0@@ -0,0 +1,168 @@ +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) + +==================== +BPF LLVM Relocations +==================== + +This document describes LLVM BPF backend relocation types. + +Relocation Record +================= + +LLVM BPF backend records each relocation with the following 16-byte +ELF structure:: + + typedef struct + { + Elf64_Addr r_offset; // Offset from the beginning of section. + Elf64_Xword r_info; // Relocation type and symbol index. + } Elf64_Rel; + +For static function/variable references, the symbol often refers to +the section itself which has a value of 0. To identify actual static +function/variable, its section offset or some computation result +based on section offset is written to the original insn/data buffer, +which is called ``IA`` (implicit addend) below. For global +function/variables, the symbol refers to actual global and the implicit +addend is 0.
Documentation/bpf/llvm_reloc.rst line 37
Quoted 10 lines
Documentation/bpf/llvm_reloc.rst new file +10
-0+ +Different Relocation Types +========================== + +Six relocation types are supported. The following is an overview and +``S`` represents the value of the symbol in the symbol table:: + + Enum ELF Reloc Type Description BitSize Offset Calculation + 0 R_BPF_NONE None + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IA
There are cases where we set all 64-bits of ld_imm64 (e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)
Quoted 37 lines
Documentation/bpf/llvm_reloc.rst new file +37
-0+ 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1 + +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction. +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write +data bitsize is 32 (4 bytes). The relocation can be resolved with +the symbol value plus implicit addend. + +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data. +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data +bitsize is 64 (8 bytes). The relocation can be resolved with +the symbol value plus implicit addend. + +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data. +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld`` +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext`` +become unusable by bcc and kernel. + +Type ``R_BPF_64_32`` is used for call instruction. The call target section +offset is stored at ``r_offset + 4`` (32bit) and calculated as +``(S + IA) / 8 - 1``. + +Examples +======== + +Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64`` +and ``call`` instructions. For example:: + + __attribute__((noinline)) __attribute__((section("sec1"))) + int gfunc(int a, int b) { + return a * b; + } + static __attribute__((noinline)) __attribute__((section("sec1")))
Documentation/bpf/llvm_reloc.rst line 84
Quoted 10 lines
Documentation/bpf/llvm_reloc.rst new file +10
-0+ int lfunc(int a, int b) { + return a + b; + } + int global __attribute__((section("sec2"))); + int test(int a, int b) { + return gfunc(a, b) + lfunc(a, b) + global; + } + +Compiled with ``clang -target bpf -O2 -c test.c``, we will have +following code with `llvm-objdump -d test.o``::
I recently learned about llvm-objdump -dr test.o, which shows relocations inline, it would be nice to use that output here.
Quoted 87 lines
Documentation/bpf/llvm_reloc.rst new file +84
-1+ + Disassembly of section .text: + + 0000000000000000 <test>: + 0: bf 26 00 00 00 00 00 00 r6 = r2 + 1: bf 17 00 00 00 00 00 00 r7 = r1 + 2: 85 10 00 00 ff ff ff ff call -1 + 3: bf 08 00 00 00 00 00 00 r8 = r0 + 4: bf 71 00 00 00 00 00 00 r1 = r7 + 5: bf 62 00 00 00 00 00 00 r2 = r6 + 6: 85 10 00 00 02 00 00 00 call 2 + 7: 0f 80 00 00 00 00 00 00 r0 += r8 + 8: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll + 10: 61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0) + 11: 0f 10 00 00 00 00 00 00 r0 += r1 + 12: 95 00 00 00 00 00 00 00 exit + + Disassembly of section sec1: + + 0000000000000000 <gfunc>: + 0: bf 20 00 00 00 00 00 00 r0 = r2 + 1: 2f 10 00 00 00 00 00 00 r0 *= r1 + 2: 95 00 00 00 00 00 00 00 exit + + 0000000000000018 <lfunc>: + 3: bf 20 00 00 00 00 00 00 r0 = r2 + 4: 0f 10 00 00 00 00 00 00 r0 += r1 + 5: 95 00 00 00 00 00 00 00 exit + +Three relocations are generated with ``llvm-readelf -r test.o``:: + + Relocation section '.rel.text' at offset 0x188 contains 3 entries: + Offset Info Type Symbol's Value Symbol's Name + 0000000000000010 000000040000000a R_BPF_64_32 0000000000000000 gfunc + 0000000000000030 000000020000000a R_BPF_64_32 0000000000000000 sec1 + 0000000000000040 0000000600000001 R_BPF_64_64 0000000000000000 global + +The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0, +so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``. +The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section +offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``. + +The following is an example to show how R_BPF_64_ABS64 could be generated:: + + int global() { return 0; } + struct t { void *g; } gbl = { global }; + +Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a +relocation below in ``.data`` section with command +``llvm-readelf -r test.o``:: + + Relocation section '.rel.data' at offset 0x458 contains 1 entries: + Offset Info Type Symbol's Value Symbol's Name + 0000000000000000 0000000700000002 R_BPF_64_ABS64 0000000000000000 global + +The relocation says the first 8-byte of ``.data`` section should be +filled with address of ``global`` variable. + +With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of +``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations:: + + Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries: + Offset Info Type Symbol's Value Symbol's Name + 0000000000000006 0000000300000003 R_BPF_64_ABS32 0000000000000000 .debug_abbrev + 000000000000000c 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + 0000000000000012 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + 0000000000000016 0000000600000003 R_BPF_64_ABS32 0000000000000000 .debug_line + 000000000000001a 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + 000000000000001e 0000000200000002 R_BPF_64_ABS64 0000000000000000 .text + 000000000000002b 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + 0000000000000037 0000000800000002 R_BPF_64_ABS64 0000000000000000 gbl + 0000000000000040 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + ...... + +The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations:: + + Relocation section '.rel.BTF' at offset 0x538 contains 1 entries: + Offset Info Type Symbol's Value Symbol's Name + 0000000000000084 0000000800000004 R_BPF_64_NODYLD32 0000000000000000 gbl + + Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries: + Offset Info Type Symbol's Value Symbol's Name + 000000000000002c 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text + 0000000000000040 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text --2.30.2
Quoted 73 lines · Andrii Nakryiko
Quoted 72 lines · Yonghong Song
LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.
Cc John Fastabend <[email protected]>Cc Lorenz Bauer <[email protected]>Signed-off-by Yonghong Song <[email protected]>--- Documentation/bpf/index.rst | 1 + Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+)create mode 100644
Documentation/bpf/llvm_reloc.rstDocumentation/bpf/index.rst +1
-0@@ -84,6 +84,7 @@ Other :maxdepth: 1 ringbuf + llvm_reloc .. Links: .. _networking-filter: ../networking/filter.rstDocumentation/bpf/llvm_reloc.rst new file +33
-0@@ -0,0 +1,168 @@ +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) + +==================== +BPF LLVM Relocations +==================== + +This document describes LLVM BPF backend relocation types. + +Relocation Record +================= + +LLVM BPF backend records each relocation with the following 16-byte +ELF structure:: + + typedef struct + { + Elf64_Addr r_offset; // Offset from the beginning of section. + Elf64_Xword r_info; // Relocation type and symbol index. + } Elf64_Rel; + +For static function/variable references, the symbol often refers to +the section itself which has a value of 0. To identify actual static +function/variable, its section offset or some computation result +based on section offset is written to the original insn/data buffer, +which is called ``IA`` (implicit addend) below. For global +function/variables, the symbol refers to actual global and the implicit +addend is 0. + +Different Relocation Types +========================== + +Six relocation types are supported. The following is an overview and +``S`` represents the value of the symbol in the symbol table::
Quoted 8 lines
Documentation/bpf/llvm_reloc.rst line 37
Quoted 4 lines
Documentation/bpf/llvm_reloc.rst new file +4
-0+ + Enum ELF Reloc Type Description BitSize Offset Calculation + 0 R_BPF_NONE None + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IAThere are cases where we set all 64-bits of
ld_imm64(e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
It is still R_BPF_64_64. In llvm, we have restriction that section offset must be <= UINT32_MAX, and that is why only 32bit is used to find the actual symbol in symbol table. 32bit permits 4GB section which should enough in practice for a bpf program.
libbpf or tools can write to full 64bits of imm values of ld_imm64 insn.
The name is a little bit misleading, but it has become part of ABI and lives in /usr/include/elf.h and we are not able to change it any more.
Quoted 4 lines
Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)
We have llvm/test/CodeGen/BPF/reloc.ll and llvm/test/CodeGen/BPF/reloc-btf.ll covering R_BPF_64_ABS64. But I think I can enhance llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.
Quoted 43 lines
Quoted 43 lines
Documentation/bpf/llvm_reloc.rst new file +42
-0+ 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1 + +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction. +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write +data bitsize is 32 (4 bytes). The relocation can be resolved with +the symbol value plus implicit addend. + +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data. +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data +bitsize is 64 (8 bytes). The relocation can be resolved with +the symbol value plus implicit addend. + +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data. +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld`` +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext`` +become unusable by bcc and kernel. + +Type ``R_BPF_64_32`` is used for call instruction. The call target section +offset is stored at ``r_offset + 4`` (32bit) and calculated as +``(S + IA) / 8 - 1``. + +Examples +======== + +Types ``R_BPF_64_64`` and ``R_BPF_64_32`` are used to resolve ``ld_imm64`` +and ``call`` instructions. For example:: + + __attribute__((noinline)) __attribute__((section("sec1"))) + int gfunc(int a, int b) { + return a * b; + } + static __attribute__((noinline)) __attribute__((section("sec1"))) + int lfunc(int a, int b) { + return a + b; + } + int global __attribute__((section("sec2"))); + int test(int a, int b) {
Quoted 8 lines
Documentation/bpf/llvm_reloc.rst line 84
Quoted 5 lines
Documentation/bpf/llvm_reloc.rst new file +5
-0+ return gfunc(a, b) + lfunc(a, b) + global; + } + +Compiled with ``clang -target bpf -O2 -c test.c``, we will have +following code with `llvm-objdump -d test.o``::I recently learned about
llvm-objdump -dr test.o, which shows relocations inline, it would be nice to use that output here.
Yes, will do.
Quoted 88 lines
Quoted 88 lines
Documentation/bpf/llvm_reloc.rst new file +84
-1+ + Disassembly of section .text: + + 0000000000000000 <test>: + 0: bf 26 00 00 00 00 00 00 r6 = r2 + 1: bf 17 00 00 00 00 00 00 r7 = r1 + 2: 85 10 00 00 ff ff ff ff call -1 + 3: bf 08 00 00 00 00 00 00 r8 = r0 + 4: bf 71 00 00 00 00 00 00 r1 = r7 + 5: bf 62 00 00 00 00 00 00 r2 = r6 + 6: 85 10 00 00 02 00 00 00 call 2 + 7: 0f 80 00 00 00 00 00 00 r0 += r8 + 8: 18 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r1 = 0 ll + 10: 61 11 00 00 00 00 00 00 r1 = *(u32 *)(r1 + 0) + 11: 0f 10 00 00 00 00 00 00 r0 += r1 + 12: 95 00 00 00 00 00 00 00 exit + + Disassembly of section sec1: + + 0000000000000000 <gfunc>: + 0: bf 20 00 00 00 00 00 00 r0 = r2 + 1: 2f 10 00 00 00 00 00 00 r0 *= r1 + 2: 95 00 00 00 00 00 00 00 exit + + 0000000000000018 <lfunc>: + 3: bf 20 00 00 00 00 00 00 r0 = r2 + 4: 0f 10 00 00 00 00 00 00 r0 += r1 + 5: 95 00 00 00 00 00 00 00 exit + +Three relocations are generated with ``llvm-readelf -r test.o``:: + + Relocation section '.rel.text' at offset 0x188 contains 3 entries: + Offset Info Type Symbol's Value Symbol's Name + 0000000000000010 000000040000000a R_BPF_64_32 0000000000000000 gfunc + 0000000000000030 000000020000000a R_BPF_64_32 0000000000000000 sec1 + 0000000000000040 0000000600000001 R_BPF_64_64 0000000000000000 global + +The first relocation corresponds to ``gfunc(a, b)`` where ``gfunc`` has a value of 0, +so the ``call`` instruction offset is ``(0 + 0)/8 - 1 = -1``. +The second relocation corresponds to ``lfunc(a, b)`` where ``lfunc`` has a section +offset ``0x18``, so the ``call`` instruction offset is ``(0 + 0x18)/8 - 1 = 2``. + +The following is an example to show how R_BPF_64_ABS64 could be generated:: + + int global() { return 0; } + struct t { void *g; } gbl = { global }; + +Compiled with ``clang -target bpf -O2 -g -c test.c``, we will see a +relocation below in ``.data`` section with command +``llvm-readelf -r test.o``:: + + Relocation section '.rel.data' at offset 0x458 contains 1 entries: + Offset Info Type Symbol's Value Symbol's Name + 0000000000000000 0000000700000002 R_BPF_64_ABS64 0000000000000000 global + +The relocation says the first 8-byte of ``.data`` section should be +filled with address of ``global`` variable. + +With ``llvm-readelf`` output, we can see that dwarf sections have a bunch of +``R_BPF_64_ABS32`` and ``R_BPF_64_ABS64`` relocations:: + + Relocation section '.rel.debug_info' at offset 0x468 contains 13 entries: + Offset Info Type Symbol's Value Symbol's Name + 0000000000000006 0000000300000003 R_BPF_64_ABS32 0000000000000000 .debug_abbrev + 000000000000000c 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + 0000000000000012 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + 0000000000000016 0000000600000003 R_BPF_64_ABS32 0000000000000000 .debug_line + 000000000000001a 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + 000000000000001e 0000000200000002 R_BPF_64_ABS64 0000000000000000 .text + 000000000000002b 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + 0000000000000037 0000000800000002 R_BPF_64_ABS64 0000000000000000 gbl + 0000000000000040 0000000400000003 R_BPF_64_ABS32 0000000000000000 .debug_str + ...... + +The .BTF/.BTF.ext sections has R_BPF_64_NODYLD32 relocations:: + + Relocation section '.rel.BTF' at offset 0x538 contains 1 entries: + Offset Info Type Symbol's Value Symbol's Name + 0000000000000084 0000000800000004 R_BPF_64_NODYLD32 0000000000000000 gbl + + Relocation section '.rel.BTF.ext' at offset 0x548 contains 2 entries: + Offset Info Type Symbol's Value Symbol's Name + 000000000000002c 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text + 0000000000000040 0000000200000004 R_BPF_64_NODYLD32 0000000000000000 .text --2.30.2
Quoted 86 lines · Yonghong Song
Quoted 73 lines · Andrii Nakryiko
Quoted 72 lines · Yonghong Song
LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.
Cc John Fastabend <[email protected]>Cc Lorenz Bauer <[email protected]>Signed-off-by Yonghong Song <[email protected]>--- Documentation/bpf/index.rst | 1 + Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+)create mode 100644
Documentation/bpf/llvm_reloc.rstDocumentation/bpf/index.rst +1
-0@@ -84,6 +84,7 @@ Other :maxdepth: 1 ringbuf + llvm_reloc .. Links: .. _networking-filter: ../networking/filter.rstDocumentation/bpf/llvm_reloc.rst new file +33
-0@@ -0,0 +1,168 @@ +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) + +==================== +BPF LLVM Relocations +==================== + +This document describes LLVM BPF backend relocation types. + +Relocation Record +================= + +LLVM BPF backend records each relocation with the following 16-byte +ELF structure:: + + typedef struct + { + Elf64_Addr r_offset; // Offset from the beginning of section. + Elf64_Xword r_info; // Relocation type and symbol index. + } Elf64_Rel; + +For static function/variable references, the symbol often refers to +the section itself which has a value of 0. To identify actual static +function/variable, its section offset or some computation result +based on section offset is written to the original insn/data buffer, +which is called ``IA`` (implicit addend) below. For global +function/variables, the symbol refers to actual global and the implicit +addend is 0. + +Different Relocation Types +========================== + +Six relocation types are supported. The following is an overview and +``S`` represents the value of the symbol in the symbol table::Quoted 8 lines
Documentation/bpf/llvm_reloc.rst line 37
Quoted 4 lines
Documentation/bpf/llvm_reloc.rst new file +4
-0+ + Enum ELF Reloc Type Description BitSize Offset Calculation + 0 R_BPF_NONE None + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IAThere are cases where we set all 64-bits of
ld_imm64(e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
Quoted 7 lines
It is still
R_BPF_64_64. In llvm, we have restriction that section offset must be <=UINT32_MAX, and that is why only 32bit is used to find the actual symbol in symbol table. 32bit permits 4GB section which should enough in practice for a bpf program.libbpf or tools can write to full 64bits of imm values of
ld_imm64insn.
Ok, sounds good.
Quoted 5 lines
The name is a little bit misleading, but it has become part of ABI and lives in
/usr/include/elf.hand we are not able to change it any more.Quoted 1 line
Quoted 9 lines
Quoted 3 lines
Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)
We have llvm/test/CodeGen/BPF/reloc.ll and llvm/test/CodeGen/BPF/reloc-btf.ll covering
R_BPF_64_ABS64. But I think I can enhance
llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.
Great, thanks.
Quoted 16 lines
Quoted 16 lines
Quoted 16 lines
Documentation/bpf/llvm_reloc.rst new file +15
-0+ 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1 + +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction. +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write +data bitsize is 32 (4 bytes). The relocation can be resolved with +the symbol value plus implicit addend. + +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data. +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data +bitsize is 64 (8 bytes). The relocation can be resolved with +the symbol value plus implicit addend. +
[...]
Quoted 32 lines · Yonghong Song
Quoted 29 lines · Andrii Nakryiko
Quoted 28 lines · Yonghong Song
LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.
Cc John Fastabend <[email protected]>Cc Lorenz Bauer <[email protected]>Signed-off-by Yonghong Song <[email protected]>--- Documentation/bpf/index.rst | 1 + Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+)create mode 100644
Documentation/bpf/llvm_reloc.rstDocumentation/bpf/index.rst +0
-0@@ -84,6 +84,7 @@ Other :maxdepth: 1
Quoted 3 lines
Quoted 3 lines
Documentation/bpf/index.rst Other · line 88
Quoted 3 lines
Documentation/bpf/index.rst +1
-0ringbuf + llvm_reloc
Thanks Yonghong, I found this helpful. I still had to crack open llvm code though to follow along. A couple small suggestions below, may or may not be useful. Overall looks good.
Quoted 27 lines
Quoted 27 lines
Quoted 27 lines
Documentation/bpf/index.rst +0
-0.. Links: .. _networking-filter: ../networking/filter.rstDocumentation/bpf/llvm_reloc.rst new file +19
-0@@ -0,0 +1,168 @@ +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) + +==================== +BPF LLVM Relocations +==================== + +This document describes LLVM BPF backend relocation types. + +Relocation Record +================= + +LLVM BPF backend records each relocation with the following 16-byte +ELF structure:: + + typedef struct + { + Elf64_Addr r_offset; // Offset from the beginning of section. + Elf64_Xword r_info; // Relocation type and symbol index. + } Elf64_Rel;
Quoted 8 lines
Quoted 8 lines
Documentation/bpf/llvm_reloc.rst line 27
Quoted 8 lines
Documentation/bpf/llvm_reloc.rst new file +8
-0+ +For static function/variable references, the symbol often refers to +the section itself which has a value of 0. To identify actual static +function/variable, its section offset or some computation result +based on section offset is written to the original insn/data buffer, +which is called ``IA`` (implicit addend) below. For global +function/variables, the symbol refers to actual global and the implicit +addend is 0.
Above was too terse for me to follow without looking into some clang examples. Maybe an example right here would help not sure? Maybe expand the text a bit? I don't have a really good suggestion.
Quoted 15 lines
Quoted 6 lines
Quoted 6 lines
Documentation/bpf/llvm_reloc.rst new file +6
-0+ +Different Relocation Types +========================== + +Six relocation types are supported. The following is an overview and +``S`` represents the value of the symbol in the symbol table::Quoted 8 lines
Documentation/bpf/llvm_reloc.rst line 37
Quoted 4 lines
Documentation/bpf/llvm_reloc.rst new file +4
-0+ + Enum ELF Reloc Type Description BitSize Offset Calculation + 0 R_BPF_NONE None + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IAThere are cases where we set all 64-bits of
ld_imm64(e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
Quoted 4 lines
It is still
R_BPF_64_64. In llvm, we have restriction that section offset must be <=UINT32_MAX, and that is why only 32bit is used to find the actual symbol in symbol table. 32bit permits 4GB section which should enough in practice for a bpf program.
^^^ maybe add this note in the doc somewhere? I had similar questions.
Quoted 8 lines
libbpf or tools can write to full 64bits of imm values of
ld_imm64insn.The name is a little bit misleading, but it has become part of ABI and lives in
/usr/include/elf.hand we are not able to change it any more.Quoted 1 line
Quoted 8 lines
Quoted 3 lines
Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)
We have llvm/test/CodeGen/BPF/reloc.ll and llvm/test/CodeGen/BPF/reloc-btf.ll covering
R_BPF_64_ABS64. But I think I can enhance
llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.
^^^ maybe cross-reference llvm tests from kernel docs side? I often look at these when I get something unexpected/unknown maybe others would find it helpful, but not know where to look?
Quoted 23 lines
Quoted 23 lines
Quoted 23 lines
Documentation/bpf/llvm_reloc.rst new file +21
-0+ 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1 + +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction. +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write +data bitsize is 32 (4 bytes). The relocation can be resolved with +the symbol value plus implicit addend. + +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data. +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data +bitsize is 64 (8 bytes). The relocation can be resolved with +the symbol value plus implicit addend. + +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data. +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld`` +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext`` +become unusable by bcc and kernel.
Quoted 8 lines
Quoted 8 lines
Documentation/bpf/llvm_reloc.rst line 66
Quoted 8 lines
Documentation/bpf/llvm_reloc.rst new file +8
-0+ +Type ``R_BPF_64_32`` is used for call instruction. The call target section +offset is stored at ``r_offset + 4`` (32bit) and calculated as +``(S + IA) / 8 - 1``. + +Examples +======== +
I liked the examples.
Quoted 72 lines · John Fastabend
Quoted 32 lines · Yonghong Song
Quoted 29 lines · Andrii Nakryiko
Quoted 28 lines · Yonghong Song
LLVM upstream commit https://reviews.llvm.org/D102712
made some changes to bpf relocations to make them llvm linker lld friendly. The scope of existing relocations R_BPF_64_{64,32} is narrowed and new relocations R_BPF_64_{ABS32,ABS64,NODYLD32} are introduced.Let us add some documentation about llvm bpf relocations so people can understand how to resolve them properly in their respective tools.
Cc John Fastabend <[email protected]>Cc Lorenz Bauer <[email protected]>Signed-off-by Yonghong Song <[email protected]>--- Documentation/bpf/index.rst | 1 + Documentation/bpf/llvm_reloc.rst | 168 +++++++++++++++++++++++++++++++ 2 files changed, 169 insertions(+)create mode 100644
Documentation/bpf/llvm_reloc.rstDocumentation/bpf/index.rst +0
-0@@ -84,6 +84,7 @@ Other :maxdepth: 1Quoted 3 lines
Quoted 3 lines
Documentation/bpf/index.rst Other · line 88
Quoted 3 lines
Documentation/bpf/index.rst +1
-0ringbuf + llvm_relocThanks Yonghong, I found this helpful. I still had to crack open llvm code though to follow along. A couple small suggestions below, may or may not be useful. Overall looks good.
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Documentation/bpf/index.rst +0
-0.. Links: .. _networking-filter: ../networking/filter.rstDocumentation/bpf/llvm_reloc.rst new file +23
-0@@ -0,0 +1,168 @@ +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) + +==================== +BPF LLVM Relocations +==================== + +This document describes LLVM BPF backend relocation types. + +Relocation Record +================= + +LLVM BPF backend records each relocation with the following 16-byte +ELF structure:: + + typedef struct + { + Elf64_Addr r_offset; // Offset from the beginning of section. + Elf64_Xword r_info; // Relocation type and symbol index. + } Elf64_Rel; + +For static function/variable references, the symbol often refers to +the section itself which has a value of 0. To identify actual static +function/variable, its section offset or some computation result
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Documentation/bpf/llvm_reloc.rst line 27
Quoted 4 lines
Documentation/bpf/llvm_reloc.rst new file +4
-0+based on section offset is written to the original insn/data buffer, +which is called ``IA`` (implicit addend) below. For global +function/variables, the symbol refers to actual global and the implicit +addend is 0.Above was too terse for me to follow without looking into some clang examples. Maybe an example right here would help not sure? Maybe expand the text a bit? I don't have a really good suggestion.
Just send a new revision with an example. Hope it will make it easy to understand the above ``IA`` concept.
Quoted 16 lines
Quoted 16 lines
Quoted 15 lines
Documentation/bpf/llvm_reloc.rst line 37
Quoted 11 lines
Documentation/bpf/llvm_reloc.rst new file +10
-0+ +Different Relocation Types +========================== + +Six relocation types are supported. The following is an overview and +``S`` represents the value of the symbol in the symbol table:: + + Enum ELF Reloc Type Description BitSize Offset Calculation + 0 R_BPF_NONE None + 1 R_BPF_64_64 ld_imm64 insn 32 r_offset + 4 S + IAThere are cases where we set all 64-bits of
ld_imm64(e.g., extern ksym, global variables). Or those will be a different relocation now (R_BPF_64_ABS64?). If not, I think BitSize 64 is more correct here.
Quoted 6 lines
Quoted 4 lines
It is still
R_BPF_64_64. In llvm, we have restriction that section offset must be <=UINT32_MAX, and that is why only 32bit is used to find the actual symbol in symbol table. 32bit permits 4GB section which should enough in practice for a bpf program.^^^ maybe add this note in the doc somewhere? I had similar questions.
Added in the new revision.
Quoted 13 lines
Quoted 13 lines
libbpf or tools can write to full 64bits of imm values of
ld_imm64insn.The name is a little bit misleading, but it has become part of ABI and lives in
/usr/include/elf.hand we are not able to change it any more.Quoted 4 lines
Looking at LLVM diff I haven't found a test for global variables (at least I didn't realize it was there), so double-checking here (and it might be a good idea to have an explicit test for global variables?)
Quoted 8 lines
Quoted 4 lines
We have llvm/test/CodeGen/BPF/reloc.ll and llvm/test/CodeGen/BPF/reloc-btf.ll covering
R_BPF_64_ABS64. But I think I can enhance
llvm/test/CodeGen/BPF/reloc-2.ll to cover an explicit global variable case.^^^ maybe cross-reference llvm tests from kernel docs side? I often look at these when I get something unexpected/unknown maybe others would find it helpful, but not know where to look?
The llvm patch has not merged. We need to merge libbpf patch first. Otherwise, nightly libbpf CI will fail. But this doc includes a link to the LLVM patch and you can just go to that llvm patch to find
examples!
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Quoted 26 lines
Documentation/bpf/llvm_reloc.rst new file +23
-0+ 2 R_BPF_64_ABS64 normal data 64 r_offset S + IA + 3 R_BPF_64_ABS32 normal data 32 r_offset S + IA + 4 R_BPF_64_NODYLD32 .BTF[.ext] data 32 r_offset S + IA + 10 R_BPF_64_32 call insn 32 r_offset + 4 (S + IA) / 8 - 1 + +For example, ``R_BPF_64_64`` relocation type is used for ``ld_imm64`` instruction. +The actual to-be-relocated data is stored at ``r_offset + 4`` and the read/write +data bitsize is 32 (4 bytes). The relocation can be resolved with +the symbol value plus implicit addend. + +In another case, ``R_BPF_64_ABS64`` relocation type is used for normal 64-bit data. +The actual to-be-relocated data is stored at ``r_offset`` and the read/write data +bitsize is 64 (8 bytes). The relocation can be resolved with +the symbol value plus implicit addend. + +Both ``R_BPF_64_ABS32`` and ``R_BPF_64_NODYLD32`` types are for 32-bit data. +But ``R_BPF_64_NODYLD32`` specifically refers to relocations in ``.BTF`` and +``.BTF.ext`` sections. For cases like bcc where llvm ``ExecutionEngine RuntimeDyld`` +is involved, ``R_BPF_64_NODYLD32`` types of relocations should not be resolved +to actual function/variable address. Otherwise, ``.BTF`` and ``.BTF.ext`` +become unusable by bcc and kernel. + +Type ``R_BPF_64_32`` is used for call instruction. The call target section
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Documentation/bpf/llvm_reloc.rst line 66
Quoted 6 lines
Documentation/bpf/llvm_reloc.rst new file +6
-0+offset is stored at ``r_offset + 4`` (32bit) and calculated as +``(S + IA) / 8 - 1``. + +Examples +======== +I liked the examples.
Great. Just added one more in the new revision!