Skip to contentSkip to conversation

Calling kfuncs in modules - BTF mismatch?

DISCUSSION Toke Høiland-Jørgensen 6 participants 10 messages Nov 11, 2022 – Dec 1, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi everyone

There seems to be some issue with BTF mismatch when trying to run the bpf_ct_set_nat_info() kfunc from a module. I was under the impression that this is supposed to work, so is there some kind of BTF dedup issue here or something?

Steps to reproduce:

1. Compile kernel with nf_conntrack built-in and run selftests; './test_progs -a bpf_nf' works

2. Change the kernel config so nf_conntrack is build as a module

3. Start the test kernel and manually modprobe nf_conntrack and nf_nat

4. Run ./test_progs -a bpf_nf; this now fails with an error like:

kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init

Anyone has any ideas what's going on here, and how to fix it?

-Toke

#1 lore raw
Quoted 13 lines

Hi everyone

There seems to be some issue with BTF mismatch when trying to run the bpf_ct_set_nat_info() kfunc from a module. I was under the impression that this is supposed to work, so is there some kind of BTF dedup issue here or something?

Steps to reproduce:

1. Compile kernel with nf_conntrack built-in and run selftests; './test_progs -a bpf_nf' works

Quoted 7 lines

2. Change the kernel config so nf_conntrack is build as a module

3. Start the test kernel and manually modprobe nf_conntrack and nf_nat

4. Run ./test_progs -a bpf_nf; this now fails with an error like:

kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init

This week Kumar and I took a look at this issue and we ended up identifying a duplication of nf_conn___init structure. In particular:

[~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_conntrack.ko format raw | grep nf_conn__ [110941] STRUCT 'nf_conn___init' size=248 vlen=1 [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_nat.ko format raw | grep nf_conn__ [107488] STRUCT 'nf_conn___init' size=248 vlen=1

Is it the root cause of the problem?

Regards,
Lorenzo

Quoted 5 lines

Anyone has any ideas what's going on here, and how to fix it?

-Toke

#2 lore raw
Quoted 24 lines · Lorenzo Bianconi
Quoted 13 lines

Hi everyone

There seems to be some issue with BTF mismatch when trying to run the bpf_ct_set_nat_info() kfunc from a module. I was under the impression that this is supposed to work, so is there some kind of BTF dedup issue here or something?

Steps to reproduce:

1. Compile kernel with nf_conntrack built-in and run selftests; './test_progs -a bpf_nf' works

Quoted 7 lines

2. Change the kernel config so nf_conntrack is build as a module

3. Start the test kernel and manually modprobe nf_conntrack and nf_nat

4. Run ./test_progs -a bpf_nf; this now fails with an error like:

kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init

This week Kumar and I took a look at this issue and we ended up identifying a duplication of nf_conn___init structure. In particular:

Quoted 8 lines

[~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_conntrack.ko format raw | grep nf_conn__ [110941] STRUCT 'nf_conn___init' size=248 vlen=1 [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_nat.ko format raw | grep nf_conn__ [107488] STRUCT 'nf_conn___init' size=248 vlen=1

Is it the root cause of the problem?

It certainly seems to be related to it, at least. Amending the log message to include the BTF object IDs of the two versions shows that the register has a reference to nf_conn__init in nf_conntrack.ko, while the kernel expects it to point to nf_nat.ko.

Not sure what's the right fix for this? Should libbpf be smart enough to pull the kfunc arg ID from the same BTF ID as the function itself? Or should the kernel compare structs and allow things if they're identical? Andrii, WDYT?

-Toke

#3 lore raw
Quoted 35 lines · Toke Høiland-Jørgensen
Quoted 24 lines · Lorenzo Bianconi
Quoted 13 lines

Hi everyone

There seems to be some issue with BTF mismatch when trying to run the bpf_ct_set_nat_info() kfunc from a module. I was under the impression that this is supposed to work, so is there some kind of BTF dedup issue here or something?

Steps to reproduce:

1. Compile kernel with nf_conntrack built-in and run selftests; './test_progs -a bpf_nf' works

Quoted 7 lines

2. Change the kernel config so nf_conntrack is build as a module

3. Start the test kernel and manually modprobe nf_conntrack and nf_nat

4. Run ./test_progs -a bpf_nf; this now fails with an error like:

kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init

This week Kumar and I took a look at this issue and we ended up identifying a duplication of nf_conn___init structure. In particular:

Quoted 8 lines

[~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_conntrack.ko format raw | grep nf_conn__ [110941] STRUCT 'nf_conn___init' size=248 vlen=1 [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_nat.ko format raw | grep nf_conn__ [107488] STRUCT 'nf_conn___init' size=248 vlen=1

Is it the root cause of the problem?

Quoted 7 lines

It certainly seems to be related to it, at least. Amending the log message to include the BTF object IDs of the two versions shows that the register has a reference to nf_conn__init in nf_conntrack.ko, while the kernel expects it to point to nf_nat.ko.

Not sure what's the right fix for this? Should libbpf be smart enough to pull the kfunc arg ID from the same BTF ID as the function itself? Or

Verifier chose the ID based on where the variable was allocated, which is bpf_(skb|xdp)_ct_alloc() from nf_conntrack.ko. Assuming btf id based on usage location might be error prone.

Quoted 2 lines

should the kernel compare structs and allow things if they're identical? Andrii, WDYT?

This works but might make verifier slower.

kernel/bpf/verifier.c +3 -1
index 195d24316750..562d2c15906d 100644
@@ -24,6 +24,7 @@
 #include <linux/bpf_lsm.h>
 #include <linux/btf_ids.h>
 #include <linux/poison.h>
+#include "../tools/lib/bpf/relo_core.h"
 
 #include "disasm.h"
 
@@ -8236,7 +8237,8 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env,
 
        reg_ref_t = btf_type_skip_modifiers(reg_btf, reg_ref_id, &reg_ref_id);
        reg_ref_tname = btf_name_by_offset(reg_btf, reg_ref_t->name_off);
-       if (!btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match)) {
+       if (!btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match) && \
+                       !__bpf_core_types_match(reg_btf, reg_ref_id, meta->btf, ref_id, false, 10)) {
                verbose(env, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n",
                        meta->func_name, argno, btf_type_str(ref_t), ref_tname, argno + 1,
                        btf_type_str(reg_ref_t), reg_ref_tname);
Signature · 2 lines
Regards,
  Artem
#4 lore raw
Quoted 41 lines · Artem Savkov
Quoted 40 lines · Toke Høiland-Jørgensen
Quoted 24 lines · Lorenzo Bianconi
Quoted 13 lines

Hi everyone

There seems to be some issue with BTF mismatch when trying to run the bpf_ct_set_nat_info() kfunc from a module. I was under the impression that this is supposed to work, so is there some kind of BTF dedup issue here or something?

Steps to reproduce:

1. Compile kernel with nf_conntrack built-in and run selftests; './test_progs -a bpf_nf' works

Quoted 7 lines

2. Change the kernel config so nf_conntrack is build as a module

3. Start the test kernel and manually modprobe nf_conntrack and nf_nat

4. Run ./test_progs -a bpf_nf; this now fails with an error like:

kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init

This week Kumar and I took a look at this issue and we ended up identifying a duplication of nf_conn___init structure. In particular:

Quoted 8 lines

[~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_conntrack.ko format raw | grep nf_conn__ [110941] STRUCT 'nf_conn___init' size=248 vlen=1 [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_nat.ko format raw | grep nf_conn__ [107488] STRUCT 'nf_conn___init' size=248 vlen=1

Is it the root cause of the problem?

It certainly seems to be related to it, at least. Amending the log message to include the BTF object IDs of the two versions shows that the register has a reference to nf_conn__init in nf_conntrack.ko, while the kernel expects it to point to nf_nat.ko.

Quoted 6 lines
Quoted 2 lines

Not sure what's the right fix for this? Should libbpf be smart enough to pull the kfunc arg ID from the same BTF ID as the function itself? Or

Verifier chose the ID based on where the variable was allocated, which is bpf_(skb|xdp)_ct_alloc() from nf_conntrack.ko. Assuming btf id based on usage location might be error prone.

Yeah, I think we need something more robust.

Quoted 18 lines
Quoted 2 lines

should the kernel compare structs and allow things if they're identical? Andrii, WDYT?

This works but might make verifier slower.

kernel/bpf/verifier.c +1 -0
index 195d24316750..562d2c15906d 100644
@@ -24,6 +24,7 @@
 #include <linux/bpf_lsm.h>
 #include <linux/btf_ids.h>
 #include <linux/poison.h>
+#include "../tools/lib/bpf/relo_core.h"
 
 #include "disasm.h"
 

kernel/bpf/verifier.c static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · line 8244

Quoted 9 lines
kernel/bpf/verifier.c +2 -1
@@ -8236,7 +8237,8 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env,
 
        reg_ref_t = btf_type_skip_modifiers(reg_btf, reg_ref_id, &reg_ref_id);
        reg_ref_tname = btf_name_by_offset(reg_btf, reg_ref_t->name_off);
-       if (!btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match)) {
+       if (!btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match) && \
+                       !__bpf_core_types_match(reg_btf, reg_ref_id, meta->btf, ref_id, false, 10)) {
                verbose(env, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n",
                        meta->func_name, argno, btf_type_str(ref_t), ref_tname, argno + 1,
                        btf_type_str(reg_ref_t), reg_ref_tname);

Ah, cool! This is a lot of code to important into the kernel, though; do we really want to do that?

I guess an alternative could be to make sure that every data type used by a kfunc is always referenced in a built-in module somewhere, so modules will use the definition from vmlinux.

Andrii, Alexei, any opinions on which way to go with this?

-Toke

#5 lore raw
Quoted 39 lines · Toke Høiland-Jørgensen
Quoted 36 lines · Artem Savkov
Quoted 35 lines · Toke Høiland-Jørgensen
Quoted 32 lines · Lorenzo Bianconi
Quoted 13 lines

Hi everyone

There seems to be some issue with BTF mismatch when trying to run the bpf_ct_set_nat_info() kfunc from a module. I was under the impression that this is supposed to work, so is there some kind of BTF dedup issue here or something?

Steps to reproduce:

1. Compile kernel with nf_conntrack built-in and run selftests; './test_progs -a bpf_nf' works

Quoted 7 lines

2. Change the kernel config so nf_conntrack is build as a module

3. Start the test kernel and manually modprobe nf_conntrack and nf_nat

4. Run ./test_progs -a bpf_nf; this now fails with an error like:

kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init

This week Kumar and I took a look at this issue and we ended up identifying a duplication of nf_conn___init structure. In particular:

[~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_conntrack.ko format raw | grep nf_conn__ [110941] STRUCT 'nf_conn___init' size=248 vlen=1 [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_nat.ko format raw | grep nf_conn__ [107488] STRUCT 'nf_conn___init' size=248 vlen=1

Is it the root cause of the problem?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

It certainly seems to be related to it, at least. Amending the log message to include the BTF object IDs of the two versions shows that the register has a reference to nf_conn__init in nf_conntrack.ko, while the kernel expects it to point to nf_nat.ko.

Not sure what's the right fix for this? Should libbpf be smart enough to pull the kfunc arg ID from the same BTF ID as the function itself? Or

Libbpf is doing just that. Or rather this just happens automatically. Libbpf finds the FUNC type corresponding to a kfunc, and then all the types of all the arguments are consistent with that FUNC definition.

I think the problem is that test is getting struct nf_conn from bpf_xdp_ct_alloc() kfunc, which is defined in nf_conntrack module (and so specifies that it returns struct nf_conn coming from nf_conntrack's module BTF), while bpf_ct_set_nat_info() kfunc is defined in nf_nat module and specifies that it expects struct nf_conn defined in nf_nat's module BTF.

And those two types are two completely different types, with different BTF object ID and BTF type ID, as far as all the BTF stuff is concerned.

I don't know what the solution here is, but it's not on the libbpf side at all for sure. As Toke said, bringing BTF dedup into the kernel seems like an overkill. So some hacky "let's compare struct name and size" approach perhaps?

Quoted 46 lines
Quoted 4 lines

Verifier chose the ID based on where the variable was allocated, which is bpf_(skb|xdp)_ct_alloc() from nf_conntrack.ko. Assuming btf id based on usage location might be error prone.

Yeah, I think we need something more robust.

Quoted 18 lines
Quoted 2 lines

should the kernel compare structs and allow things if they're identical? Andrii, WDYT?

This works but might make verifier slower.

kernel/bpf/verifier.c +1 -0
index 195d24316750..562d2c15906d 100644
@@ -24,6 +24,7 @@
 #include <linux/bpf_lsm.h>
 #include <linux/btf_ids.h>
 #include <linux/poison.h>
+#include "../tools/lib/bpf/relo_core.h"
 
 #include "disasm.h"
 

kernel/bpf/verifier.c static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · line 8244

Quoted 9 lines
kernel/bpf/verifier.c +2 -1
@@ -8236,7 +8237,8 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env,
 
        reg_ref_t = btf_type_skip_modifiers(reg_btf, reg_ref_id, &reg_ref_id);
        reg_ref_tname = btf_name_by_offset(reg_btf, reg_ref_t->name_off);
-       if (!btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match)) {
+       if (!btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match) && \
+                       !__bpf_core_types_match(reg_btf, reg_ref_id, meta->btf, ref_id, false, 10)) {
                verbose(env, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n",
                        meta->func_name, argno, btf_type_str(ref_t), ref_tname, argno + 1,
                        btf_type_str(reg_ref_t), reg_ref_tname);

Ah, cool! This is a lot of code to important into the kernel, though; do we really want to do that?

I guess an alternative could be to make sure that every data type used by a kfunc is always referenced in a built-in module somewhere, so modules will use the definition from vmlinux.

Andrii, Alexei, any opinions on which way to go with this?

-Toke

#6 lore raw
Quoted 32 lines · Andrii Nakryiko
Quoted 12 lines · Toke Høiland-Jørgensen
Quoted 12 lines
Quoted 12 lines
Quoted 11 lines

This week Kumar and I took a look at this issue and we ended up identifying a duplication of nf_conn___init structure. In particular:

[~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_conntrack.ko format raw | grep nf_conn__ [110941] STRUCT 'nf_conn___init' size=248 vlen=1 [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_nat.ko format raw | grep nf_conn__ [107488] STRUCT 'nf_conn___init' size=248 vlen=1

Is it the root cause of the problem?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

It certainly seems to be related to it, at least. Amending the log message to include the BTF object IDs of the two versions shows that the register has a reference to nf_conn__init in nf_conntrack.ko, while the kernel expects it to point to nf_nat.ko.

Not sure what's the right fix for this? Should libbpf be smart enough to pull the kfunc arg ID from the same BTF ID as the function itself? Or

Libbpf is doing just that. Or rather this just happens automatically. Libbpf finds the FUNC type corresponding to a kfunc, and then all the types of all the arguments are consistent with that FUNC definition.

I think the problem is that test is getting struct nf_conn from bpf_xdp_ct_alloc() kfunc, which is defined in nf_conntrack module (and so specifies that it returns struct nf_conn coming from nf_conntrack's module BTF), while bpf_ct_set_nat_info() kfunc is defined in nf_nat module and specifies that it expects struct nf_conn defined in nf_nat's module BTF.

Quoted 8 lines

And those two types are two completely different types, with different BTF object ID and BTF type ID, as far as all the BTF stuff is concerned.

I don't know what the solution here is, but it's not on the libbpf side at all for sure. As Toke said, bringing BTF dedup into the kernel seems like an overkill. So some hacky "let's compare struct name and size" approach perhaps?

Wouldn't that be a bit too relaxed for a general case? I wonder how often can this issue come up. If this is relatively rare maybe known kfuncs that need this can be flagged with a new flag (KF_RELAXED_ARG_CHECK or similar) to allow this shortcut?

Signature · 2 lines
Regards,
  Artem
#7 lore raw
In reply to Toke Høiland-Jørgensen
Quoted 40 lines · Toke Høiland-Jørgensen
Quoted 24 lines · Lorenzo Bianconi
Quoted 13 lines

Hi everyone

There seems to be some issue with BTF mismatch when trying to run the bpf_ct_set_nat_info() kfunc from a module. I was under the impression that this is supposed to work, so is there some kind of BTF dedup issue here or something?

Steps to reproduce:

1. Compile kernel with nf_conntrack built-in and run selftests; './test_progs -a bpf_nf' works

Quoted 7 lines

2. Change the kernel config so nf_conntrack is build as a module

3. Start the test kernel and manually modprobe nf_conntrack and nf_nat

4. Run ./test_progs -a bpf_nf; this now fails with an error like:

kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init

This week Kumar and I took a look at this issue and we ended up identifying a duplication of nf_conn___init structure. In particular:

Quoted 8 lines

[~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_conntrack.ko format raw | grep nf_conn__ [110941] STRUCT 'nf_conn___init' size=248 vlen=1 [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_nat.ko format raw | grep nf_conn__ [107488] STRUCT 'nf_conn___init' size=248 vlen=1

Is it the root cause of the problem?

It certainly seems to be related to it, at least. Amending the log message to include the BTF object IDs of the two versions shows that the register has a reference to nf_conn__init in nf_conntrack.ko, while the kernel expects it to point to nf_nat.ko.

Quoted 5 lines

Not sure what's the right fix for this? Should libbpf be smart enough to pull the kfunc arg ID from the same BTF ID as the function itself? Or should the kernel compare structs and allow things if they're identical? Andrii, WDYT?

There were some dedup issues fixed recently in pahole and libbpf; since dwarves libbpf hasn't been synced with libbpf recently as far as I can see it won't have the fix for [1]; I suspect it may help with dedup-ing here. Would probably be worth trying rebuilding dwarves with a libbpf with [1] applied and seeing if the dedup issue goes away before we go any further. If it fixes the issue, would it be worth updating the libbpf that dwarves uses Arnaldo? I saw some pretty large improvements in removing redundant definitions.

Thanks!

Alan

https://lore.kernel.org/bpf/[email protected]/

Quoted 2 lines

-Toke

#8 lore raw

Em Tue, Nov 29, 2022 at 04:21:23PM +0000, Alan Maguire escreveu:

Quoted 47 lines · Toke Høiland-Jørgensen
Quoted 40 lines
Quoted 24 lines · Lorenzo Bianconi
Quoted 13 lines

Hi everyone

There seems to be some issue with BTF mismatch when trying to run the bpf_ct_set_nat_info() kfunc from a module. I was under the impression that this is supposed to work, so is there some kind of BTF dedup issue here or something?

Steps to reproduce:

1. Compile kernel with nf_conntrack built-in and run selftests; './test_progs -a bpf_nf' works

Quoted 7 lines

2. Change the kernel config so nf_conntrack is build as a module

3. Start the test kernel and manually modprobe nf_conntrack and nf_nat

4. Run ./test_progs -a bpf_nf; this now fails with an error like:

kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init

This week Kumar and I took a look at this issue and we ended up identifying a duplication of nf_conn___init structure. In particular:

Quoted 8 lines

[~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_conntrack.ko format raw | grep nf_conn__ [110941] STRUCT 'nf_conn___init' size=248 vlen=1 [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_nat.ko format raw | grep nf_conn__ [107488] STRUCT 'nf_conn___init' size=248 vlen=1

Is it the root cause of the problem?

It certainly seems to be related to it, at least. Amending the log message to include the BTF object IDs of the two versions shows that the register has a reference to nf_conn__init in nf_conntrack.ko, while the kernel expects it to point to nf_nat.ko.

Quoted 5 lines

Not sure what's the right fix for this? Should libbpf be smart enough to pull the kfunc arg ID from the same BTF ID as the function itself? Or should the kernel compare structs and allow things if they're identical? Andrii, WDYT?

Quoted 10 lines

There were some dedup issues fixed recently in pahole and libbpf; since dwarves libbpf hasn't been synced with libbpf recently as far as I can see it won't have the fix for [1]; I suspect it may help with dedup-ing here. Would probably be worth trying rebuilding dwarves with a libbpf with [1] applied and seeing if the dedup issue goes away before we go any further. If it fixes the issue, would it be worth updating the libbpf that dwarves uses Arnaldo? I saw some pretty large improvements in removing redundant definitions.

Feel free to send a patch updating the libbpf copy in pahole, so that I can go on testing it.

I'm working on supporting DW_TAG_imported_unit and split DWARF (.dwz) so as soon as I get that properly tested and make sure it doesn't regresses wrt BTF encoding I'll cut a new version.

- Arnaldo

#9 lore raw
In reply to Alan Maguire
Quoted 47 lines · Alan Maguire
Quoted 40 lines · Toke Høiland-Jørgensen
Quoted 24 lines · Lorenzo Bianconi
Quoted 13 lines

Hi everyone

There seems to be some issue with BTF mismatch when trying to run the bpf_ct_set_nat_info() kfunc from a module. I was under the impression that this is supposed to work, so is there some kind of BTF dedup issue here or something?

Steps to reproduce:

1. Compile kernel with nf_conntrack built-in and run selftests; './test_progs -a bpf_nf' works

Quoted 7 lines

2. Change the kernel config so nf_conntrack is build as a module

3. Start the test kernel and manually modprobe nf_conntrack and nf_nat

4. Run ./test_progs -a bpf_nf; this now fails with an error like:

kernel function bpf_ct_set_nat_info args#0 expected pointer to STRUCT nf_conn___init but R1 has a pointer to STRUCT nf_conn___init

This week Kumar and I took a look at this issue and we ended up identifying a duplication of nf_conn___init structure. In particular:

Quoted 8 lines

[~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_conntrack.ko format raw | grep nf_conn__ [110941] STRUCT 'nf_conn___init' size=248 vlen=1 [~/workspace/bpf-next]$ bpftool btf --base-btf vmlinux dump file net/netfilter/nf_nat.ko format raw | grep nf_conn__ [107488] STRUCT 'nf_conn___init' size=248 vlen=1

Is it the root cause of the problem?

It certainly seems to be related to it, at least. Amending the log message to include the BTF object IDs of the two versions shows that the register has a reference to nf_conn__init in nf_conntrack.ko, while the kernel expects it to point to nf_nat.ko.

Quoted 5 lines

Not sure what's the right fix for this? Should libbpf be smart enough to pull the kfunc arg ID from the same BTF ID as the function itself? Or should the kernel compare structs and allow things if they're identical? Andrii, WDYT?

Quoted 10 lines

There were some dedup issues fixed recently in pahole and libbpf; since dwarves libbpf hasn't been synced with libbpf recently as far as I can see it won't have the fix for [1]; I suspect it may help with dedup-ing here. Would probably be worth trying rebuilding dwarves with a libbpf with [1] applied and seeing if the dedup issue goes away before we go any further. If it fixes the issue, would it be worth updating the libbpf that dwarves uses Arnaldo? I saw some pretty large improvements in removing redundant definitions.

I don't think it's a deduplication issue; the type is simply not defined in vmlinux, so each module has its own "version" of it. And since each module's BTF is semantically independent of other modules the duplication is expected in this case.

-Toke