bpf: allow empty module BTFs
Subsystems (4)
Andrii Nakryiko
maintainerPATCH 1/2 bpf: allow empty module BTFs
Some modules don't declare any new types and end up with an empty BTF, containing only valid BTF header and no types or strings sections. This currently causes BTF validation error. There is nothing wrong with such BTF, so fix the issue by allowing module BTFs with no types or strings.
--- kernel/bpf/btf.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/bpf/btf.c +1 -1
@@ -4172,7 +4172,7 @@ static int btf_parse_hdr(struct btf_verifier_env *env)return -ENOTSUPP ; } -if (btf_data_size == hdr->hdr_len) { +if (!btf->base_btf && btf_data_size == hdr->hdr_len) {btf_verifier_log (env,"No data" );return -EINVAL ; }
Signature · 1 line
2.24.1
Andrii Nakryiko
maintainerPATCH 2/2 libbpf: allow loading empty BTFs
Empty BTFs do come up (e.g., simple kernel modules with no new types and strings, compared to the vmlinux BTF) and there is nothing technically wrong with them. So remove unnecessary check preventing loading empty BTFs.
--- tools/lib/bpf/btf.c | 5 ----- 1 file changed, 5 deletions(-)
tools/lib/bpf/btf.c +0 -5
@@ -240,11 +240,6 @@ static int btf_parse_hdr(struct btf *btf) } meta_left = btf->raw_size -sizeof (*hdr); -if (!meta_left) { -pr_debug ("BTF has no data\n" ); -return -EINVAL ; - } -if (meta_left < hdr->str_off + hdr->str_len) {pr_debug ("Invalid BTF total size:%u\n" , btf->raw_size);return -EINVAL ;
Signature · 1 line
2.24.1
Re: [PATCH bpf 2/2] libbpf: allow loading empty BTFs
Quoted 16 lines · Andrii Nakryiko
Empty BTFs do come up (e.g., simple kernel modules with no new types and strings, compared to the vmlinux BTF) and there is nothing technically wrong with them. So remove unnecessary check preventing loading empty BTFs.
Reported-by Christopher William Snowhill <[email protected]>Fixes ("d8123624506c libbpf: Fix BTF data layout checks and allow empty BTF")Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/btf.c | 5 ----- 1 file changed, 5 deletions(-)tools/lib/bpf/btf.c +0
-0
tools/lib/bpf/btf.c static int btf_parse_hdr(struct btf *btf) · line 246
Quoted 7 lines
tools/lib/bpf/btf.c +0
-4@@ -240,11 +240,6 @@ static int btf_parse_hdr(struct btf *btf) } meta_left = btf->raw_size -sizeof (*hdr); -if (!meta_left) { -pr_debug ("BTF has no data\n" ); -return -EINVAL ; - }
Previous kernel patch allows empty btf only if that btf is module (not base/vmlinux) btf. Here it seems we allow any empty non-module btf to be loaded into the kernel. In such cases, loading may fail? Maybe we should detect such cases in libbpf and error out instead of going to kernel and
get error back?
Quoted 5 lines
tools/lib/bpf/btf.c +0
-1-if (meta_left < hdr->str_off + hdr->str_len) {pr_debug ("Invalid BTF total size:%u\n" , btf->raw_size);return -EINVAL ;
Re: [PATCH bpf 2/2] libbpf: allow loading empty BTFs
Quoted 22 lines · Yonghong Song
Quoted 18 lines · Andrii Nakryiko
Empty BTFs do come up (e.g., simple kernel modules with no new types and strings, compared to the vmlinux BTF) and there is nothing technically wrong with them. So remove unnecessary check preventing loading empty BTFs.
Reported-by Christopher William Snowhill <[email protected]>Fixes ("d8123624506c libbpf: Fix BTF data layout checks and allow empty BTF")Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/btf.c | 5 ----- 1 file changed, 5 deletions(-)tools/lib/bpf/btf.c +0
-0@@ -240,11 +240,6 @@ static int btf_parse_hdr(struct btf *btf) }
Quoted 11 lines
tools/lib/bpf/btf.c static int btf_parse_hdr(struct btf *btf) · line 246
Quoted 5 lines
tools/lib/bpf/btf.c +0
-4meta_left = btf->raw_size -sizeof (*hdr); -if (!meta_left) { -pr_debug ("BTF has no data\n" ); -return -EINVAL ; - }Previous kernel patch allows empty btf only if that btf is module (not base/vmlinux) btf. Here it seems we allow any empty non-module btf to be loaded into the kernel. In such cases, loading may fail? Maybe we should detect such cases in libbpf and error out instead of going to kernel and get error back?
I did this consciously. Kernel is more strict, because there is no reasonable case when vmlinux BTF or BPF program's BTF can be empty (at least not that now we have FUNCs in BTF). But allowing libbpf to load empty BTF generically is helpful for bpftool, as one example, for inspection. If you do bpftool btf dump on empty BTF, it will just print nothing and you'll know that it's a valid (from BTF header perspective) BTF, just doesn't have any types (besides VOID). If we don't allow it, then we'll just get an error and then you'll have to do painful hex dumping and decoding to see what's wrong.
In practice, no BPF program's BTF should be empty, but if it is, the kernel will rightfully stop you. I don't think it's a common enough case for libbpf to handle.
Quoted 6 lines
Quoted 6 lines
tools/lib/bpf/btf.c +0
-1-if (meta_left < hdr->str_off + hdr->str_len) {pr_debug ("Invalid BTF total size:%u\n" , btf->raw_size);return -EINVAL ;
Re: [PATCH bpf 2/2] libbpf: allow loading empty BTFs
Quoted 35 lines · Andrii Nakryiko
Quoted 22 lines · Yonghong Song
Quoted 18 lines · Andrii Nakryiko
Empty BTFs do come up (e.g., simple kernel modules with no new types and strings, compared to the vmlinux BTF) and there is nothing technically wrong with them. So remove unnecessary check preventing loading empty BTFs.
Reported-by Christopher William Snowhill <[email protected]>Fixes ("d8123624506c libbpf: Fix BTF data layout checks and allow empty BTF")Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/btf.c | 5 ----- 1 file changed, 5 deletions(-)tools/lib/bpf/btf.c +0
-0@@ -240,11 +240,6 @@ static int btf_parse_hdr(struct btf *btf) }Quoted 11 lines
tools/lib/bpf/btf.c static int btf_parse_hdr(struct btf *btf) · line 246
Quoted 5 lines
tools/lib/bpf/btf.c +0
-4meta_left = btf->raw_size -sizeof (*hdr); -if (!meta_left) { -pr_debug ("BTF has no data\n" ); -return -EINVAL ; - }Previous kernel patch allows empty btf only if that btf is module (not base/vmlinux) btf. Here it seems we allow any empty non-module btf to be loaded into the kernel. In such cases, loading may fail? Maybe we should detect such cases in libbpf and error out instead of going to kernel and get error back?
Quoted 9 lines
I did this consciously. Kernel is more strict, because there is no reasonable case when vmlinux BTF or BPF program's BTF can be empty (at least not that now we have FUNCs in BTF). But allowing libbpf to load empty BTF generically is helpful for bpftool, as one example, for inspection. If you do
bpftool btf dumpon empty BTF, it will just print nothing and you'll know that it's a valid (from BTF header perspective) BTF, just doesn't have any types (besides VOID). If we don't allow it, then we'll just get an error and then you'll have to do painful hex dumping and decoding to see what's wrong.
It is totally okay to allow empty btf in libbpf. I just want to check
if this btf is going to be loaded into the kernel, right before it is loading whether libbpf could check whether it is a non-module empty btf or not, if it is, do not go to kernel.
Quoted 4 lines
In practice, no BPF program's BTF should be empty, but if it is, the kernel will rightfully stop you. I don't think it's a common enough case for libbpf to handle.
In general, libbpf should catch errors earlier if possible without going to kernel. This way, we can have better error messages for user. But I won't insist in this case as it is indeed really rare.
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
tools/lib/bpf/btf.c +0
-1-if (meta_left < hdr->str_off + hdr->str_len) {pr_debug ("Invalid BTF total size:%u\n" , btf->raw_size);return -EINVAL ;
Re: [PATCH bpf 2/2] libbpf: allow loading empty BTFs
Quoted 49 lines · Yonghong Song
Quoted 35 lines · Andrii Nakryiko
Quoted 22 lines · Yonghong Song
Quoted 18 lines · Andrii Nakryiko
Empty BTFs do come up (e.g., simple kernel modules with no new types and strings, compared to the vmlinux BTF) and there is nothing technically wrong with them. So remove unnecessary check preventing loading empty BTFs.
Reported-by Christopher William Snowhill <[email protected]>Fixes ("d8123624506c libbpf: Fix BTF data layout checks and allow empty BTF")Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/btf.c | 5 ----- 1 file changed, 5 deletions(-)tools/lib/bpf/btf.c +0
-0@@ -240,11 +240,6 @@ static int btf_parse_hdr(struct btf *btf) }Quoted 11 lines
tools/lib/bpf/btf.c static int btf_parse_hdr(struct btf *btf) · line 246
Quoted 5 lines
tools/lib/bpf/btf.c +0
-4meta_left = btf->raw_size -sizeof (*hdr); -if (!meta_left) { -pr_debug ("BTF has no data\n" ); -return -EINVAL ; - }Previous kernel patch allows empty btf only if that btf is module (not base/vmlinux) btf. Here it seems we allow any empty non-module btf to be loaded into the kernel. In such cases, loading may fail? Maybe we should detect such cases in libbpf and error out instead of going to kernel and get error back?
Quoted 9 lines
I did this consciously. Kernel is more strict, because there is no reasonable case when vmlinux BTF or BPF program's BTF can be empty (at least not that now we have FUNCs in BTF). But allowing libbpf to load empty BTF generically is helpful for bpftool, as one example, for inspection. If you do
bpftool btf dumpon empty BTF, it will just print nothing and you'll know that it's a valid (from BTF header perspective) BTF, just doesn't have any types (besides VOID). If we don't allow it, then we'll just get an error and then you'll have to do painful hex dumping and decoding to see what's wrong.
Quoted 4 lines
It is totally okay to allow empty btf in libbpf. I just want to check if this btf is going to be loaded into the kernel, right before it is loading whether libbpf could check whether it is a non-module empty btf or not, if it is, do not go to kernel.
Ok, I see what you are proposing. We can do that, but it's definitely separate from these bug fixes. But, to be honest, I wouldn't bother because libbpf will return BTF verification log with a very readable "No data" message in it.
Quoted 9 lines
Quoted 5 lines
In practice, no BPF program's BTF should be empty, but if it is, the kernel will rightfully stop you. I don't think it's a common enough case for libbpf to handle.
In general, libbpf should catch errors earlier if possible without going to kernel. This way, we can have better error messages for user. But I won't insist in this case as it is indeed really rare.
I wouldn't say in general. Rather in cases that commonly would cause confusion. I don't think libbpf should grow into a massive "let's double check everything before kernel" thing.
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
tools/lib/bpf/btf.c +0
-1-if (meta_left < hdr->str_off + hdr->str_len) {pr_debug ("Invalid BTF total size:%u\n" , btf->raw_size);return -EINVAL ;
Daniel Borkmann
maintainerRe: [PATCH bpf 2/2] libbpf: allow loading empty BTFs
Quoted 10 lines · Andrii Nakryiko
Quoted 9 lines · Yonghong Song
Quoted 8 lines · Andrii Nakryiko
Quoted 7 lines · Yonghong Song
Quoted 6 lines · Andrii Nakryiko
Empty BTFs do come up (e.g., simple kernel modules with no new types and strings, compared to the vmlinux BTF) and there is nothing technically wrong with them. So remove unnecessary check preventing loading empty BTFs.
Reported-by Christopher William Snowhill <[email protected]>Fixes ("d8123624506c libbpf: Fix BTF data layout checks and allow empty BTF")
Fixed up Fixes tag ^^^^^ while applying. ;-)
Quoted 39 lines
Quoted 34 lines
Quoted 24 lines
Quoted 12 lines
Quoted 12 lines
Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/btf.c | 5 ----- 1 file changed, 5 deletions(-)tools/lib/bpf/btf.c +0
-0@@ -240,11 +240,6 @@ static int btf_parse_hdr(struct btf *btf) }Quoted 11 lines
tools/lib/bpf/btf.c static int btf_parse_hdr(struct btf *btf) · line 246
Quoted 5 lines
tools/lib/bpf/btf.c +0
-4meta_left = btf->raw_size -sizeof (*hdr); -if (!meta_left) { -pr_debug ("BTF has no data\n" ); -return -EINVAL ; - }Previous kernel patch allows empty btf only if that btf is module (not base/vmlinux) btf. Here it seems we allow any empty non-module btf to be loaded into the kernel. In such cases, loading may fail? Maybe we should detect such cases in libbpf and error out instead of going to kernel and get error back?
Quoted 9 lines
I did this consciously. Kernel is more strict, because there is no reasonable case when vmlinux BTF or BPF program's BTF can be empty (at least not that now we have FUNCs in BTF). But allowing libbpf to load empty BTF generically is helpful for bpftool, as one example, for inspection. If you do
bpftool btf dumpon empty BTF, it will just print nothing and you'll know that it's a valid (from BTF header perspective) BTF, just doesn't have any types (besides VOID). If we don't allow it, then we'll just get an error and then you'll have to do painful hex dumping and decoding to see what's wrong.Quoted 4 lines
It is totally okay to allow empty btf in libbpf. I just want to check if this btf is going to be loaded into the kernel, right before it is loading whether libbpf could check whether it is a non-module empty btf or not, if it is, do not go to kernel.
Quoted 4 lines
Ok, I see what you are proposing. We can do that, but it's definitely separate from these bug fixes. But, to be honest, I wouldn't bother because libbpf will return BTF verification log with a very readable "No data" message in it.
Right, seems okay to me for this particular case given the user will be able to make some sense of it from the log.
Thanks,
Daniel
Re: [PATCH bpf 2/2] libbpf: allow loading empty BTFs
Quoted 6 lines · Daniel Borkmann
Quoted 4 lines · Andrii Nakryiko
Quoted 3 lines · Yonghong Song
Quoted 2 lines · Andrii Nakryiko
Quoted 1 line · Yonghong Song
On 1/9/21 11:03 PM, Andrii Nakryiko wrote:
Quoted 8 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Empty BTFs do come up (e.g., simple kernel modules with no new types and strings, compared to the vmlinux BTF) and there is nothing technically wrong with them. So remove unnecessary check preventing loading empty BTFs.
Reported-by Christopher William Snowhill <[email protected]>Fixes ("d8123624506c libbpf: Fix BTF data layout checks and allow empty BTF")Fixed up Fixes tag ^^^^^ while applying. ;-)
Oh the irony, eh? :) Thanks, Daniel!
Quoted 50 lines
Quoted 40 lines
Quoted 35 lines
Quoted 25 lines
Quoted 13 lines
Quoted 13 lines
Signed-off-by Andrii Nakryiko <[email protected]>--- tools/lib/bpf/btf.c | 5 ----- 1 file changed, 5 deletions(-)tools/lib/bpf/btf.c +0
-0@@ -240,11 +240,6 @@ static int btf_parse_hdr(struct btf *btf) }Quoted 11 lines
tools/lib/bpf/btf.c static int btf_parse_hdr(struct btf *btf) · line 246
Quoted 5 lines
tools/lib/bpf/btf.c +0
-4meta_left = btf->raw_size -sizeof (*hdr); -if (!meta_left) { -pr_debug ("BTF has no data\n" ); -return -EINVAL ; - }Previous kernel patch allows empty btf only if that btf is module (not base/vmlinux) btf. Here it seems we allow any empty non-module btf to be loaded into the kernel. In such cases, loading may fail? Maybe we should detect such cases in libbpf and error out instead of going to kernel and get error back?
Quoted 9 lines
I did this consciously. Kernel is more strict, because there is no reasonable case when vmlinux BTF or BPF program's BTF can be empty (at least not that now we have FUNCs in BTF). But allowing libbpf to load empty BTF generically is helpful for bpftool, as one example, for inspection. If you do
bpftool btf dumpon empty BTF, it will just print nothing and you'll know that it's a valid (from BTF header perspective) BTF, just doesn't have any types (besides VOID). If we don't allow it, then we'll just get an error and then you'll have to do painful hex dumping and decoding to see what's wrong.Quoted 4 lines
It is totally okay to allow empty btf in libbpf. I just want to check if this btf is going to be loaded into the kernel, right before it is loading whether libbpf could check whether it is a non-module empty btf or not, if it is, do not go to kernel.
Quoted 4 lines
Ok, I see what you are proposing. We can do that, but it's definitely separate from these bug fixes. But, to be honest, I wouldn't bother because libbpf will return BTF verification log with a very readable "No data" message in it.
Right, seems okay to me for this particular case given the user will be able to make some sense of it from the log.
Thanks,
Daniel
Quoted 8 lines · Andrii Nakryiko
Some modules don't declare any new types and end up with an empty BTF, containing only valid BTF header and no types or strings sections. This currently causes BTF validation error. There is nothing wrong with such BTF, so fix the issue by allowing module BTFs with no types or strings.
Reported-by Christopher William Snowhill <[email protected]>Fixes 36e68442d1af ("bpf: Load and verify kernel module BTFs")Signed-off-by Andrii Nakryiko <[email protected]>
This series was applied to bpf/bpf.git (refs/heads/master):
Hello:
This series was applied to bpf/bpf.git (refs/heads/master):
Quoted 10 lines · On Sat, 9 Jan 2021 23:03:40 -0800 you wrote:
Some modules don't declare any new types and end up with an empty BTF, containing only valid BTF header and no types or strings sections. This currently causes BTF validation error. There is nothing wrong with such BTF, so fix the issue by allowing module BTFs with no types or strings.
Reported-by Christopher William Snowhill <[email protected]>Fixes 36e68442d1af ("bpf: Load and verify kernel module BTFs")Signed-off-by Andrii Nakryiko <[email protected]>[...]
Here is the summary with links:
- [bpf,1/2] bpf: allow empty module BTFs
https://git.kernel.org/bpf/bpf/c/bcc5e6162d66 - [bpf,2/2] libbpf: allow loading empty BTFs https://git.kernel.org/bpf/bpf/c/b8d52264df85
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
When is this being applied to an actual kernel? 5.11 is still quite broken without these two patches. Unless you're not using a vfat EFI partition, I guess.
Quoted 29 lines · On Tue, Jan 12, 2021, at 12:20 PM, [email protected] wrote:
Hello:
This series was applied to bpf/bpf.git (refs/heads/master):
Quoted 10 lines · On Sat, 9 Jan 2021 23:03:40 -0800 you wrote:
Some modules don't declare any new types and end up with an empty BTF, containing only valid BTF header and no types or strings sections. This currently causes BTF validation error. There is nothing wrong with such BTF, so fix the issue by allowing module BTFs with no types or strings.
Reported-by Christopher William Snowhill <[email protected]>Fixes 36e68442d1af ("bpf: Load and verify kernel module BTFs")Signed-off-by Andrii Nakryiko <[email protected]>[...]
Here is the summary with links:
- [bpf,1/2] bpf: allow empty module BTFshttps://git.kernel.org/bpf/bpf/c/bcc5e6162d66 - [bpf,2/2] libbpf: allow loading empty BTFs https://git.kernel.org/bpf/bpf/c/b8d52264df85You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
Quoted 3 lines · Christopher William Snowhill
When is this being applied to an actual kernel? 5.11 is still quite broken without these two patches. Unless you're not using a vfat EFI partition, I guess.
It's in v5.11-rc5.
Quoted 30 lines · On Tue, Jan 12, 2021, at 12:20 PM, [email protected] wrote:
Quoted 29 lines
Hello:
This series was applied to bpf/bpf.git (refs/heads/master):
Quoted 10 lines · On Sat, 9 Jan 2021 23:03:40 -0800 you wrote:
Some modules don't declare any new types and end up with an empty BTF, containing only valid BTF header and no types or strings sections. This currently causes BTF validation error. There is nothing wrong with such BTF, so fix the issue by allowing module BTFs with no types or strings.
Reported-by Christopher William Snowhill <[email protected]>Fixes 36e68442d1af ("bpf: Load and verify kernel module BTFs")Signed-off-by Andrii Nakryiko <[email protected]>[...]
Here is the summary with links:
- [bpf,1/2] bpf: allow empty module BTFshttps://git.kernel.org/bpf/bpf/c/bcc5e6162d66 - [bpf,2/2] libbpf: allow loading empty BTFs https://git.kernel.org/bpf/bpf/c/b8d52264df85You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
Aha, that was just released. Nice. I'll report this to the issue tracker where I had lodged the bpf patches for QoL use when testing rc4 and older.
Quoted 39 lines · Andrii Nakryiko
Quoted 3 lines · Christopher William Snowhill
When is this being applied to an actual kernel? 5.11 is still quite broken without these two patches. Unless you're not using a vfat EFI partition, I guess.
It's in v5.11-rc5.
Quoted 30 lines · On Tue, Jan 12, 2021, at 12:20 PM, [email protected] wrote:
Quoted 29 lines
Hello:
This series was applied to bpf/bpf.git (refs/heads/master):
Quoted 10 lines · On Sat, 9 Jan 2021 23:03:40 -0800 you wrote:
Some modules don't declare any new types and end up with an empty BTF, containing only valid BTF header and no types or strings sections. This currently causes BTF validation error. There is nothing wrong with such BTF, so fix the issue by allowing module BTFs with no types or strings.
Reported-by Christopher William Snowhill <[email protected]>Fixes 36e68442d1af ("bpf: Load and verify kernel module BTFs")Signed-off-by Andrii Nakryiko <[email protected]>[...]
Here is the summary with links:
- [bpf,1/2] bpf: allow empty module BTFshttps://git.kernel.org/bpf/bpf/c/bcc5e6162d66 - [bpf,2/2] libbpf: allow loading empty BTFs https://git.kernel.org/bpf/bpf/c/b8d52264df85You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html