bpf: LLVM BTF inner map struct type def missing

Hello,

While writing a BPF prog using map of maps I bumped into this compiler bug that GitHub user thediveo and Isovalent colleague Timo Beckers already discussed in the cilium/ebpf discussions [^1].

The issue is that a struct only used in a inner map is not included in the program BTF, so it needs a dummy declaration elsewhere to work.

For example such program:

        #include "vmlinux.h"
        #include <bpf/bpf_helpers.h>

        struct missing_type {
                uint64_t foo;
        };

        // struct missing_type bar; // commented on purpose

        struct {
                __uint(type, BPF_MAP_TYPE_ARRAY_OF_MAPS);
                __type(key, uint32_t);
                __type(value, uint32_t);
                __uint(max_entries, 16);
                __array(
                        values, struct {
                                __uint(type, BPF_MAP_TYPE_HASH);
                                __type(key, uint64_t);
                                __type(value, struct missing_type);
                                __uint(max_entries, 32);
                        });
        } outer_map SEC(".maps");

Then do:

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h clang -target bpf -g -O2 -c prog.c -o prog.o
bpftool btf dump file prog.o

Will result in:

        [1] PTR '(anon)' type_id=3
        [2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
        [3] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=12
        [4] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none)
        [5] PTR '(anon)' type_id=6
        [6] TYPEDEF 'uint32_t' type_id=7
        [7] TYPEDEF 'u32' type_id=8
        [8] TYPEDEF '__u32' type_id=9
        [9] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)
        [10] PTR '(anon)' type_id=11
        [11] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=16
        [12] PTR '(anon)' type_id=13
        [13] STRUCT '(anon)' size=32 vlen=4
                'type' type_id=14 bits_offset=0
                'key' type_id=16 bits_offset=64
                'value' type_id=21 bits_offset=128
                'max_entries' type_id=22 bits_offset=192
        [14] PTR '(anon)' type_id=15
        [15] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=1
        [16] PTR '(anon)' type_id=17
        [17] TYPEDEF 'uint64_t' type_id=18
        [18] TYPEDEF 'u64' type_id=19
        [19] TYPEDEF '__u64' type_id=20
        [20] INT 'unsigned long long' size=8 bits_offset=0 nr_bits=64 encoding=(none)
        [21] PTR '(anon)' type_id=28
        [22] PTR '(anon)' type_id=23
        [23] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=32
        [24] ARRAY '(anon)' type_id=12 index_type_id=4 nr_elems=0
        [25] STRUCT '(anon)' size=32 vlen=5
                'type' type_id=1 bits_offset=0
                'key' type_id=5 bits_offset=64
                'value' type_id=5 bits_offset=128
                'max_entries' type_id=10 bits_offset=192
                'values' type_id=24 bits_offset=256
        [26] VAR 'outer_map' type_id=25, linkage=global
        [27] DATASEC '.maps' size=0 vlen=1
                type_id=26 offset=0 size=32 (VAR 'outer_map')
        [28] FWD 'missing_type' fwd_kind=struct

You can see that the outer map is [25], with values [24] with type to
[12] thus [13] and then the value of [13] is [21] which points to type [28]. And [28] is a forward declaration. Thus if we try to load this program (there's no program but the libbpf error msg is explicit):

bpftool prog load prog.o /sys/fs/bpf/prog

Output is

libbpf: map 'outer_map.inner': can't determine value size for type [28]: -22.

Now if you uncomment the commented line in the example (or use this type in a function as suggested by Timo), the BTF looks like this:

        [1] PTR '(anon)' type_id=3
        [2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
        [3] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=12
        [4] INT '__ARRAY_SIZE_TYPE__' size=4 bits_offset=0 nr_bits=32 encoding=(none)
        [5] PTR '(anon)' type_id=6
        [6] TYPEDEF 'uint32_t' type_id=7
        [7] TYPEDEF 'u32' type_id=8
        [8] TYPEDEF '__u32' type_id=9
        [9] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)
        [10] PTR '(anon)' type_id=11
        [11] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=16
        [12] PTR '(anon)' type_id=13
        [13] STRUCT '(anon)' size=32 vlen=4
                'type' type_id=14 bits_offset=0
                'key' type_id=16 bits_offset=64
                'value' type_id=21 bits_offset=128
                'max_entries' type_id=22 bits_offset=192
        [14] PTR '(anon)' type_id=15
        [15] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=1
        [16] PTR '(anon)' type_id=17
        [17] TYPEDEF 'uint64_t' type_id=18
        [18] TYPEDEF 'u64' type_id=19
        [19] TYPEDEF '__u64' type_id=20
        [20] INT 'unsigned long long' size=8 bits_offset=0 nr_bits=64 encoding=(none)
        [21] PTR '(anon)' type_id=27
        [22] PTR '(anon)' type_id=23
        [23] ARRAY '(anon)' type_id=2 index_type_id=4 nr_elems=32
        [24] ARRAY '(anon)' type_id=12 index_type_id=4 nr_elems=0
        [25] STRUCT '(anon)' size=32 vlen=5
                'type' type_id=1 bits_offset=0
                'key' type_id=5 bits_offset=64
                'value' type_id=5 bits_offset=128
                'max_entries' type_id=10 bits_offset=192
                'values' type_id=24 bits_offset=256
        [26] VAR 'outer_map' type_id=25, linkage=global
        [27] STRUCT 'missing_type' size=8 vlen=1
                'foo' type_id=17 bits_offset=0
        [28] VAR 'bar' type_id=27, linkage=global
        [29] DATASEC '.bss' size=0 vlen=1
                type_id=28 offset=0 size=8 (VAR 'bar')
        [30] DATASEC '.maps' size=0 vlen=1
                type_id=26 offset=0 size=32 (VAR 'outer_map')

And then the type [27] exists, loading can now proceed.

I tested it with latest LLVM-project head when writing this e789f8bdf369 ("[libc][math] Add Generic Comparison Operations for floating point types (#144983)").

If you think it's reasonable to fix, I would be interested looking into this.

[^1]: https://github.com/cilium/ebpf/discussions/1658#discussioncomment-12491339

Quoted 2 lines

If you think it's reasonable to fix, I would be interested looking into this.

Looks like this may be something we want to fix as people starts or already uses nested maps. So yes, please go ahead to look into this.

FYI, the llvm patch https://github.com/llvm/llvm-project/pull/141719/ tried to solve nested struct definition issue (similar to here). In that case, a type_tag is the indicator which allows further type generation. The patch listed llvm source locations you likely need to touch.

Quoted 3 lines

[^1]: https://github.com/cilium/ebpf/discussions/1658#discussioncomment-12491339

Quoted 11 lines

________________________________________
From: Mahe Tardy <[email protected]>
Sent: Friday, July 25, 2025 4:51 AM
To: Yonghong Song
Cc: [email protected]; Alexei Starovoitov; Daniel Borkmann
Subject: Re: bpf: LLVM BTF inner map struct type def missing

Quoted 3 lines · Yonghong Song

Quoted 2 lines

If you think it's reasonable to fix, I would be interested looking into this.

Quoted 7 lines

Quoted 2 lines

Looks like this may be something we want to fix as people starts or already uses nested maps. So yes, please go ahead to look into this.

Here is my proposal patch https://github.com/llvm/llvm-project/pull/150608 . The idea is to visit inner maps as we visit the global maps during BTF generation. Unfortunately, for that, I had to add a boolean arg to some functions, I hope it's okay.

Quoted 12 lines

Quoted 6 lines

FYI, the llvm patch https://github.com/llvm/llvm-project/pull/141719/ tried to solve nested struct definition issue (similar to here). In that case, a type_tag is the indicator which allows further type generation. The patch listed llvm source locations you likely need to touch.

Thank you, it was very useful to have this pointer!

Quoted 3 lines

Quoted 3 lines

[^1]: https://github.com/cilium/ebpf/discussions/1658#discussioncomment-12491339