fix BTF usage on embedded systems
Subsystems (5)
cover fix BTF usage on embedded systems
Hello,
I've been experimenting with BPF and BTF on small, emebedded platforms requiring cross-compilation to varying archs, word-sizes, and endianness. These environments are not the most common for the majority of eBPF users, and have exposed multiple problems with basic functionality. This patch series addresses some of these issues.
Enabling BTF support in the kernel can sometimes result in sysfs export of /sys/kernel/btf/vmlinux as a zero-length file, which is still readable and seen to leak non-zero kernel data. Patch #1 adds a sanity-check to avoid this situation.
Small systems commonly enable LD_DEAD_CODE_DATA_ELIMINATION, which causes the .BTF section data to be incorrectly removed and can trigger the problem above. Patch #2 preserves the BTF data.
Even if BTF data is generated and embedded in the kernel, it may be encoded as non-native endianness due to another bug [1] currently being worked on. Patch #3 lets bpftool recognize the wrong BTF endianness rather than output a confusing/misleading ELF header error message.
Patches #1 and #2 were first developed for Linux 5.4.x and should be backported if possible. Feedback and suggestions for improvement are welcome!
Thanks,
Tony
[1] https://lore.kernel.org/bpf/CAPGftE8ipAacAnm9xMHFabXCL-XrCXGmOsX-Nsjvz9wnh3Zx-w@mail.gmail.com/
Tony Ambardar (3):
bpf: fix sysfs export of empty BTF section
bpf: prevent .BTF section elimination
libbpf: fix native endian assumption when parsing BTF
include/asm-generic/vmlinux.lds.h | 2 +- kernel/bpf/sysfs_btf.c | 6 +++--- tools/lib/bpf/btf.c | 6 ++++++ 3 files changed, 10 insertions(+), 4 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/3 bpf: fix sysfs export of empty BTF section
If BTF data is missing or removed from the ELF section it is still exported via sysfs as a zero-length file:
root@OpenWrt:/# ls -l /sys/kernel/btf/vmlinux -r--r--r-- 1 root root 0 Jul 18 02:59 /sys/kernel/btf/vmlinux
Moreover, reads from this file succeed and leak kernel data:
root@OpenWrt:/# hexdump -C /sys/kernel/btf/vmlinux|head -10 000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000cc0 00 00 00 00 00 00 00 00 00 00 00 00 80 83 b0 80 |................| 000cd0 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000ce0 00 00 00 00 00 00 00 00 00 00 00 00 57 ac 6e 9d |............W.n.| 000cf0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 002650 00 00 00 00 00 00 00 10 00 00 00 01 00 00 00 01 |................| 002660 80 82 9a c4 80 85 97 80 81 a9 51 68 00 00 00 02 |..........Qh....| 002670 80 25 44 dc 80 85 97 80 81 a9 50 24 81 ab c4 60 |.%D.......P$...`|
This situation was first observed with kernel 5.4.x, cross-compiled for a MIPS target system. Fix by adding a sanity-check for export of zero-length data sections.
--- kernel/bpf/sysfs_btf.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
kernel/bpf/sysfs_btf.c +3 -3
@@ -30,15 +30,15 @@ static struct kobject *btf_kobj;static int __init btf_vmlinux_init (void ) { -if (!__start_BTF) + bin_attr_btf_vmlinux.size = __stop_BTF - __start_BTF; + +if (!__start_BTF || bin_attr_btf_vmlinux.size ==0 )return 0 ; btf_kobj =kobject_create_and_add ("btf" , kernel_kobj);if (!btf_kobj)return -ENOMEM ; - bin_attr_btf_vmlinux.size = __stop_BTF - __start_BTF; -return sysfs_create_bin_file (btf_kobj, &bin_attr_btf_vmlinux); }
Signature · 1 line
2.25.1
RE: [PATCH bpf v1 1/3] bpf: fix sysfs export of empty BTF section
Quoted 43 lines · Tony Ambardar
If BTF data is missing or removed from the ELF section it is still exported via sysfs as a zero-length file:
root@OpenWrt:/# ls -l /sys/kernel/btf/vmlinux -r--r--r-- 1 root root 0 Jul 18 02:59 /sys/kernel/btf/vmlinuxMoreover, reads from this file succeed and leak kernel data:
root@OpenWrt:/# hexdump -C /sys/kernel/btf/vmlinux|head -10 000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000cc0 00 00 00 00 00 00 00 00 00 00 00 00 80 83 b0 80 |................| 000cd0 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000ce0 00 00 00 00 00 00 00 00 00 00 00 00 57 ac 6e 9d |............W.n.| 000cf0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 002650 00 00 00 00 00 00 00 10 00 00 00 01 00 00 00 01 |................| 002660 80 82 9a c4 80 85 97 80 81 a9 51 68 00 00 00 02 |..........Qh....| 002670 80 25 44 dc 80 85 97 80 81 a9 50 24 81 ab c4 60 |.%D.......P$...`|This situation was first observed with kernel 5.4.x, cross-compiled for a MIPS target system. Fix by adding a sanity-check for export of zero-length data sections.
Fixes 341dfcf8d78e ("btf: expose BTF info through sysfs")Signed-off-by Tony Ambardar <[email protected]>--- kernel/bpf/sysfs_btf.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)kernel/bpf/sysfs_btf.c +3
-1@@ -30,15 +30,15 @@ static struct kobject *btf_kobj;static int __init btf_vmlinux_init (void ) { -if (!__start_BTF) + bin_attr_btf_vmlinux.size = __stop_BTF - __start_BTF; + +if (!__start_BTF || bin_attr_btf_vmlinux.size ==0 )
kernel/bpf/sysfs_btf.c static struct kobject *btf_kobj; · line 43
Quoted 10 lines
kernel/bpf/sysfs_btf.c +0
-2return 0 ; btf_kobj =kobject_create_and_add ("btf" , kernel_kobj);if (!btf_kobj)return -ENOMEM ; - bin_attr_btf_vmlinux.size = __stop_BTF - __start_BTF; -return sysfs_create_bin_file (btf_kobj, &bin_attr_btf_vmlinux); }
Thanks LGTM.
Re: [PATCH bpf v1 1/3] bpf: fix sysfs export of empty BTF section
In reply to Tony AmbardarQuoted 21 lines · Tony Ambardar
If BTF data is missing or removed from the ELF section it is still exported via sysfs as a zero-length file:
root@OpenWrt:/# ls -l /sys/kernel/btf/vmlinux -r--r--r-- 1 root root 0 Jul 18 02:59 /sys/kernel/btf/vmlinuxMoreover, reads from this file succeed and leak kernel data:
root@OpenWrt:/# hexdump -C /sys/kernel/btf/vmlinux|head -10 000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 000cc0 00 00 00 00 00 00 00 00 00 00 00 00 80 83 b0 80 |................| 000cd0 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 000ce0 00 00 00 00 00 00 00 00 00 00 00 00 57 ac 6e 9d |............W.n.| 000cf0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| * 002650 00 00 00 00 00 00 00 10 00 00 00 01 00 00 00 01 |................| 002660 80 82 9a c4 80 85 97 80 81 a9 51 68 00 00 00 02 |..........Qh....| 002670 80 25 44 dc 80 85 97 80 81 a9 50 24 81 ab c4 60 |.%D.......P$...`|
Quoted 8 lines
This situation was first observed with kernel 5.4.x, cross-compiled for a MIPS target system. Fix by adding a sanity-check for export of zero-length data sections.
Fixes 341dfcf8d78e ("btf: expose BTF info through sysfs")Signed-off-by Tony Ambardar <[email protected]>---
Apparently sysfs infrastructure doesn't validate read position and size when bin_attribute's size is 0, and just expects read callback to handle such situation explicitly. Preventing sysfs entry from registering seems like a good solution. Thanks!
Quoted 29 lines
kernel/bpf/sysfs_btf.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)kernel/bpf/sysfs_btf.c +3
-4@@ -30,15 +30,15 @@ static struct kobject *btf_kobj;static int __init btf_vmlinux_init (void ) { -if (!__start_BTF) + bin_attr_btf_vmlinux.size = __stop_BTF - __start_BTF; + +if (!__start_BTF || bin_attr_btf_vmlinux.size ==0 )return 0 ; btf_kobj =kobject_create_and_add ("btf" , kernel_kobj);if (!btf_kobj)return -ENOMEM ; - bin_attr_btf_vmlinux.size = __stop_BTF - __start_BTF; -return sysfs_create_bin_file (btf_kobj, &bin_attr_btf_vmlinux); } --2.25.1
PATCH 2/3 bpf: prevent .BTF section elimination
In reply to Tony AmbardarSystems with memory or disk constraints often reduce the kernel footprint by configuring LD_DEAD_CODE_DATA_ELIMINATION. However, this can result in removal of any BTF information.
Use the KEEP() macro to preserve the BTF data as done with other important sections, while still allowing for smaller kernels.
--- include/asm-generic/vmlinux.lds.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
include/asm-generic/vmlinux.lds.h +1 -1
@@ -661,7 +661,7 @@#define BTF \ .BTF : AT(ADDR(.BTF) - LOAD_OFFSET) { \ __start_BTF = .; \ - *(.BTF) \ + KEEP(*(.BTF)) \ __stop_BTF = .; \ } \ . = ALIGN(4); \
Signature · 1 line
2.25.1
RE: [PATCH bpf v1 2/3] bpf: prevent .BTF section elimination
Quoted 30 lines · Tony Ambardar
Systems with memory or disk constraints often reduce the kernel footprint by configuring
LD_DEAD_CODE_DATA_ELIMINATION. However, this can result in removal of any BTF information.Use the
KEEP()macro to preserve the BTF data as done with other important sections, while still allowing for smaller kernels.Fixes 90ceddcb4950 ("bpf: Support llvm-objcopy for vmlinux BTF")Signed-off-by Tony Ambardar <[email protected]>--- include/asm-generic/vmlinux.lds.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)include/asm-generic/vmlinux.lds.h +1
-1@@ -661,7 +661,7 @@#define BTF \ .BTF : AT(ADDR(.BTF) - LOAD_OFFSET) { \ __start_BTF = .; \ - *(.BTF) \ + KEEP(*(.BTF)) \ __stop_BTF = .; \ } \ . = ALIGN(4); \Signature · 1 line
2.25.1
PATCH 3/3 libbpf: fix native endian assumption when parsing BTF
In reply to Tony AmbardarCode in btf__parse_raw() fails to detect raw BTF of non-native endianness and assumes it must be ELF data, which then fails to parse as ELF and yields a misleading error message:
root:/# bpftool btf dump file /sys/kernel/btf/vmlinux libbpf: failed to get EHDR from /sys/kernel/btf/vmlinux
For example, this could occur after cross-compiling a BTF-enabled kernel for a target with non-native endianness, which is currently unsupported.
Check for correct endianness and emit a clearer error message:
root:/# bpftool btf dump file /sys/kernel/btf/vmlinux libbpf: non-native BTF endianness is not supported
--- tools/lib/bpf/btf.c | 6 ++++++ 1 file changed, 6 insertions(+)
tools/lib/bpf/btf.c +6 -0
@@ -659,6 +659,12 @@ struct btf *btf__parse_raw(const char *path) err = -EIO ; goto err_out; } +if (magic ==__bswap_16 (BTF_MAGIC )) { +/* non-native endian raw BTF */ +pr_warn ("non-native BTF endianness is not supported\n" ); + err = -LIBBPF_ERRNO__ENDIAN ; + goto err_out; + }if (magic !=BTF_MAGIC ) {/* definitely not a raw BTF */ err = -EPROTO ;
Signature · 1 line
2.25.1
RE: [PATCH bpf v1 3/3] libbpf: fix native endian assumption when parsing BTF
Quoted 42 lines · Tony Ambardar
Code in
btf__parse_raw()fails to detect raw BTF of non-native endianness and assumes it must be ELF data, which then fails to parse as ELF and yields a misleading error message:root:/# bpftool btf dump file /sys/kernel/btf/vmlinux libbpf: failed to get EHDR from /sys/kernel/btf/vmlinuxFor example, this could occur after cross-compiling a BTF-enabled kernel for a target with non-native endianness, which is currently unsupported.
Check for correct endianness and emit a clearer error message:
root:/# bpftool btf dump file /sys/kernel/btf/vmlinux libbpf: non-native BTF endianness is not supportedFixes 94a1fedd63ed ("libbpf: Add btf__parse_raw() and generic btf__parse() APIs")Signed-off-by Tony Ambardar <[email protected]>--- tools/lib/bpf/btf.c | 6 ++++++ 1 file changed, 6 insertions(+)tools/lib/bpf/btf.c +6
-0@@ -659,6 +659,12 @@ struct btf *btf__parse_raw(const char *path) err = -EIO ; goto err_out; } +if (magic ==__bswap_16 (BTF_MAGIC )) { +/* non-native endian raw BTF */ +pr_warn ("non-native BTF endianness is not supported\n" ); + err = -LIBBPF_ERRNO__ENDIAN ; + goto err_out; + }if (magic !=BTF_MAGIC ) {/* definitely not a raw BTF */ err = -EPROTO ;Signature · 1 line
2.25.1
Quoted 37 lines · Tony Ambardar
Hello,
I've been experimenting with BPF and BTF on small, emebedded platforms requiring cross-compilation to varying archs, word-sizes, and endianness. These environments are not the most common for the majority of eBPF users, and have exposed multiple problems with basic functionality. This patch series addresses some of these issues.
Enabling BTF support in the kernel can sometimes result in sysfs export of
/sys/kernel/btf/vmlinuxas a zero-length file, which is still readable and seen to leak non-zero kernel data. Patch #1 adds a sanity-check to avoid this situation.Small systems commonly enable
LD_DEAD_CODE_DATA_ELIMINATION, which causes the .BTF section data to be incorrectly removed and can trigger the problem above. Patch #2 preserves the BTF data.Even if BTF data is generated and embedded in the kernel, it may be encoded as non-native endianness due to another bug [1] currently being worked on. Patch #3 lets bpftool recognize the wrong BTF endianness rather than output a confusing/misleading ELF header error message.
Patches #1 and #2 were first developed for Linux 5.4.x and should be backported if possible. Feedback and suggestions for improvement are welcome!
Thanks,
Tony[1] https://lore.kernel.org/bpf/CAPGftE8ipAacAnm9xMHFabXCL-XrCXGmOsX-Nsjvz9wnh3Zx-w@mail.gmail.com/
Tony Ambardar (3):
bpf: fix sysfs export of empty BTF section
bpf: prevent .BTF section elimination
libbpf: fix native endian assumption when parsing BTF
Quoted 8 lines
include/asm-generic/vmlinux.lds.h | 2 +- kernel/bpf/sysfs_btf.c | 6 +++--- tools/lib/bpf/btf.c | 6 ++++++ 3 files changed, 10 insertions(+), 4 deletions(-)--
2.25.1
All fixes look good:
Quoted 36 lines · Tony Ambardar
Hello,
I've been experimenting with BPF and BTF on small, emebedded platforms requiring cross-compilation to varying archs, word-sizes, and endianness. These environments are not the most common for the majority of eBPF users, and have exposed multiple problems with basic functionality. This patch series addresses some of these issues.
Enabling BTF support in the kernel can sometimes result in sysfs export of
/sys/kernel/btf/vmlinuxas a zero-length file, which is still readable and seen to leak non-zero kernel data. Patch #1 adds a sanity-check to avoid this situation.Small systems commonly enable
LD_DEAD_CODE_DATA_ELIMINATION, which causes the .BTF section data to be incorrectly removed and can trigger the problem above. Patch #2 preserves the BTF data.Even if BTF data is generated and embedded in the kernel, it may be encoded as non-native endianness due to another bug [1] currently being worked on. Patch #3 lets bpftool recognize the wrong BTF endianness rather than output a confusing/misleading ELF header error message.
Patches #1 and #2 were first developed for Linux 5.4.x and should be backported if possible. Feedback and suggestions for improvement are welcome!
Thanks,
Tony[1] https://lore.kernel.org/bpf/CAPGftE8ipAacAnm9xMHFabXCL-XrCXGmOsX-Nsjvz9wnh3Zx-w@mail.gmail.com/
Tony Ambardar (3):
bpf: fix sysfs export of empty BTF section
bpf: prevent .BTF section elimination
libbpf: fix native endian assumption when parsing BTF
Quoted 5 lines
include/asm-generic/vmlinux.lds.h | 2 +- kernel/bpf/sysfs_btf.c | 6 +++--- tools/lib/bpf/btf.c | 6 ++++++ 3 files changed, 10 insertions(+), 4 deletions(-)
Applied, thanks!