Use uapi kernel headers with vmlinux.h
Subsystems (6)
Eduard Zingerman
maintainercover Use uapi kernel headers with vmlinux.h
Hi BPF community,
AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.
The basic idea
---
The goal of the patch set is to allow usage of header files from include/uapi alongside vmlinux.h as follows:
#include <uapi/linux/tcp.h> #include "vmlinux.h"
This goal is achieved by adding #ifndef ... #endif guards in vmlinux.h around definitions that originate from the include/uapi headers. The guards emitted match the guards used in the original headers. E.g. as follows:
include/uapi/linux/tcp.h:
#ifndef _UAPI_LINUX_TCP_H
#define _UAPI_LINUX_TCP_H
...
union tcp_word_hdr {
struct tcphdr hdr;
__be32 words[5]; };
...
#endif /* _UAPI_LINUX_TCP_H */
vmlinux.h:
... #ifndef _UAPI_LINUX_TCP_H
union tcp_word_hdr {
struct tcphdr hdr;
__be32 words[5];};
#endif /* _UAPI_LINUX_TCP_H */ ...
To get to this state the following steps are necessary:
- "header guard" name should be identified for each header file;
- the correspondence between data type and it's header guard has to be encoded in BTF;
- bpftool should be adjusted to emit #ifndef FOO_H ... #endif brackets.
It is not possible to identify header guard names for all uapi headers basing only on the file name. However a simple script could devised to identify the guards basing on the file name and it's content. Thus it is possible to obtain the list of header names with corresponding header guards.
The correspondence between type and it's declaration file (header) is available in DWARF as DW_AT_decl_file attribute. The DW_AT_decl_file can be matched with the list of header guards described above to obtain the header guard name for a specific type.
The pahole generates BTF using DWARF. It is possible to modify pahole to accept the header guards list as an additional parameter and to encode the header guard names in BTF.
Implementation details
---
Present patch-set implements these ideas as follows:
- A parameter --header_guards_db is added to pahole. If present it points to a file with a list of <header> <guard> records.
- pahole uses DWARF DW_AT_decl_file value to lookup the header guard for each type emitted to BTF. If header guard is present it is encoded alongside the type.
- Header guards are encoded in BTF as BTF_DECL_TAG records with a special prefix. The prefix "header_guard:" is added to a value of such tags. (Here BTF_DECL_TAG is used to avoid BTF binary format changes).
- A special script infer_header_guards.pl is added as a part of kbuild, it can infer header guard names for each UAPI header basing on the header content.
- This script is invoked from link-vmlinux.sh prior to BTF generation during kernel build. The output of the script is saved to a file, the file is passed to pahole as --header_guards_db parameter.
- libbpf is modified to aggregate BTF_DECL_TAG records for each type and to emit #ifndef FOO_H ... #endif brackets when "header_guard:" tag is present for a type.
Details for each patch in a set:
- libbpf: Deduplicate unambigous standalone forward declarations
- selftests/bpf: Tests for standalone forward BTF declarations deduplication
There is a small number (63 for defconfig) of forward declarations that are not de-duplicated with the main type declaration under certain conditions. This hinders the header guard brackets generation. This patch addresses this de-duplication issue.
- libbpf: Support for BTF_DECL_TAG dump in C format
- selftests/bpf: Tests for BTF_DECL_TAG dump in C format
Currently libbpf does not process BTF_DECL_TAG when btf is dumped in C format. This patch adds a hash table matching btf type ids with a list of decl tags to the struct btf_dump.
The btf_dump_emit_decl_tags is not necessary for the overall patch-set to function but simplifies testing a bit.
- libbpf: Header guards for selected data structures in vmlinux.h
- selftests/bpf: Tests for header guards printing in BTF dump
Adds option emit_header_guards to struct btf_dump_opts. When enabled the btf_dump__dump_type prints #ifndef ... #endif brackets around types for which header guard information is present in BTF.
- bpftool: Enable header guards generation
Unconditionally enables emit_header_guards for BTF dump in C format.
- kbuild: Script to infer header guard values for uapi headers
- kbuild: Header guards for types from include/uapi/*.h in kernel BTF
Adds scripts/infer_header_guards.pl and integrates it with link-vmlinux.sh.
- selftests/bpf: Script to verify uapi headers usage with vmlinux.h
Adds a script test_uapi_headers.py that tests header guards with vmlinux.h by compiling a simple C snippet. The snippet looks as follows:
#include <some_uapi_header.h>
#include "vmlinux.h"
__attribute__((section("tc"), used))
int syncookie_tc(struct __sk_buff *skb) { return 0; } The list of headers to test comes from tools/testing/selftests/bpf/good_uapi_headers.txt.
- selftests/bpf: Known good uapi headers for test_uapi_headers.py
The list of uapi headers that could be included alongside vmlinux.h. The headers are peeked from the following locations:
- <headers-export-dir>/linux/*.h
- <headers-export-dir>/linux/**/*.h
This choice of locations is somewhat arbitrary.
- selftests/bpf: script for infer_header_guards.pl testing
The test case for scripts/infer_header_guards.pl, verifies that header guards can be inferred for all uapi headers.
- There is also a patch for dwarves that adds --header_guards_db option (see [2]).
The test_uapi_headers.py is important as it demonstrates the the necessary compiler flags:
clang ... \
-D__x86_64__ \
-Xclang -fwchar-type=short \
-Xclang -fno-signed-wchar \
-I{exported_kernel_headers}/include/ \
...- -fwchar-type=short and -fno-signed-wchar had to be added because BPF target uses int for wchar_t by default and this differs from vmlinux.h definition of the type (at least for x86_64).
- __x86_64__ had to be added for uapi headers that include stddef.h (the one that is supplied my CLANG itself), in order to define correct sizes for size_t and ptrdiff_t.
- The {exported_kernel_headers} stands for exported kernel headers directory (the headers obtained by make headers_install or via distribution package).
When it works
---
The mechanics described above works for a significant number of UAPI headers. For example, for the test case above I chose the headers from the following locations:
- linux/*.h
- linux/**/*.h
There are 759 such headers and for 677 of them the test described above passes.
I excluded the headers from the following sub-directories as potentially not interesting:
asm rdma video xen asm-generic misc scsi drm mtd sound
Thus saving some time for both discussion and CI but the choice is somewhat arbitrary. If I run test_uapi_headers.py --test '*' (all headers) test passes for 834 out of 972 headers.
When it breaks
---
There several scenarios when this mechanics breaks. Specifically I found the following cases:
- When uapi header includes some system header that conflicts with vmlinux.h.
- When uapi header itself conflicts with vmlinux.h.
Below are examples for both cases.
Conflict with system headers
----
The following uapi headers:
- linux/atmbr2684.h
- linux/bpfilter.h
- linux/gsmmux.h
- linux/icmp.h
- linux/if.h
- linux/if_arp.h
- linux/if_bonding.h
- linux/if_pppox.h
- linux/if_tunnel.h
- linux/ip6_tunnel.h
- linux/llc.h
- linux/mctp.h
- linux/mptcp.h
- linux/netdevice.h
- linux/netfilter/xt_RATEEST.h
- linux/netfilter/xt_hashlimit.h
- linux/netfilter/xt_physdev.h
- linux/netfilter/xt_rateest.h
- linux/netfilter_arp/arp_tables.h
- linux/netfilter_arp/arpt_mangle.h
- linux/netfilter_bridge.h
- linux/netfilter_bridge/ebtables.h
- linux/netfilter_ipv4/ip_tables.h
- linux/netfilter_ipv6/ip6_tables.h
- linux/route.h
- linux/wireless.h
Include the following system header:
- /usr/include/sys/socket.h (all via linux/if.h)
The sys/socket.h conflicts with vmlinux.h in:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants: SOCK_STREAM, SOCK_DGRAM, ...
However, only two types are actually used:
- struct sockaddr
- struct sockaddr_storage (used only in linux/mptcp.h)
In 'vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.
The only workaround that I see is to:
- define a stub sys/socket.h as follows:
#ifndef __BPF_SOCKADDR__
#define __BPF_SOCKADDR__
/* For __kernel_sa_family_t */
#include <linux/socket.h>
struct sockaddr {
__kernel_sa_family_t sa_family;
char sa_data[14];
};
#endif- hardcode generation of __BPF_SOCKADDR__ bracket for 'struct sockaddr' in vmlinux.h.
Another possibility is to move the definition of 'struct sockaddr' from 'kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.
Conflict with vmlinux.h
----
Uapi header:
- linux/signal.h
Conflict with vmlinux.h in definition of 'struct sigaction'. Defined in:
- vmlinux.h: kernel/include/linux/signal_types.h
- uapi: kernel/arch/x86/include/asm/signal.h
Uapi headers:
- linux/tipc_sockets_diag.h
- linux/sock_diag.h
Conflict with vmlinux.h in definition of 'SOCK_DESTROY'. Defined in:
- vmlinux.h: kernel/include/net/sock.h
- uapi: kernel/include/uapi/linux/sock_diag.h
Constants seem to be unrelated.
And so on... I have details for many other headers but omit those for brevity.
In conclusion
---
Except from the general feasibility I have a few questions:
- What UAPI headers are the candidates for such use? If there are some interesting headers currently not working with this patch-set some hacks have to be added (e.g. like with linux/if.h).
- Is it ok to encode header guards as special BTF_DECL_TAG or should I change the BTF format a bit to save some bytes.
Thanks,
Eduard
[1] https://reviews.llvm.org/D111307
[clang] __attribute__ bpf_dominating_decl
[2] https://lore.kernel.org/dwarves/[email protected]/T/
[RFC dwarves] pahole: Save header guard names when
--header_guards_db is passed
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing
scripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-) create mode 100755 scripts/infer_header_guards.pl create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRFC 1/12 libbpf: Deduplicate unambigous standalone forward declarations
Deduplicate forward declarations that don't take part in type graphs comparisons if declaration name is unambiguous. Example:
CU #1:
struct foo; // standalone forward declaration
struct foo *some_global;
CU #2:
struct foo { int x; };
struct foo *another_global;
The struct foo from CU #1 is not a part of any definition that is compared against another definition while btf_dedup_struct_types processes structural types. The the BTF after btf_dedup_struct_types the BTF looks as follows:
[1] STRUCT 'foo' size=4 vlen=1 ...
[2] INT 'int' size=4 ...
[3] PTR '(anon)' type_id=1
[4] FWD 'foo' fwd_kind=struct
[5] PTR '(anon)' type_id=4
This commit adds a new pass btf_dedup_standalone_fwds, that maps such forward declarations to structs or unions with identical name in case if the name is not ambiguous.
The pass is positioned before btf_dedup_ref_types so that types
[3] and [5] could be merged as a same type after [1] and [4] are merged. The final result for the example above looks as follows:
[1] STRUCT 'foo' size=4 vlen=1
'x' type_id=2 bits_offset=0
[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[3] PTR '(anon)' type_id=1
For defconfig kernel with BTF enabled this removes 63 forward declarations. Examples of removed declarations: pt_regs, in6_addr. The running time of btf__dedup function is increased by about 3%.
--- tools/lib/bpf/btf.c | 178 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 174 insertions(+), 4 deletions(-)
tools/lib/bpf/btf.c +174 -4
@@ -2881,6 +2881,7 @@ static int btf_dedup_strings(struct btf_dedup *d);static int btf_dedup_prim_types (struct btf_dedup *d);static int btf_dedup_struct_types (struct btf_dedup *d);static int btf_dedup_ref_types (struct btf_dedup *d); +static int btf_dedup_standalone_fwds (struct btf_dedup *d);static int btf_dedup_compact_types (struct btf_dedup *d);static int btf_dedup_remap_types (struct btf_dedup *d);
@@ -2988,15 +2989,16 @@ static int btf_dedup_remap_types(struct btf_dedup *d);* Algorithm summary * ================= * -* Algorithm completes its work in 6 separate passes: +* Algorithm completes its work in 7 separate passes: * * 1. Strings deduplication. * 2. Primitive types deduplication (int, enum, fwd). * 3. Struct/union types deduplication. -* 4. Reference types deduplication (pointers, typedefs, arrays, funcs, func +* 4. Standalone fwd declarations deduplication. +* 5. Reference types deduplication (pointers, typedefs, arrays, funcs, func * protos, and const/volatile/restrict modifiers). -* 5. Types compaction. -* 6. Types remapping. +* 6. Types compaction. +* 7. Types remapping. * * Algorithm determines canonical type descriptor, which is a single * representative type for each truly unique type. This canonical type is the
@@ -3060,6 +3062,11 @@ int btf__dedup(struct btf *btf, const struct btf_dedup_opts *opts)pr_debug ("btf_dedup_struct_types failed:%d\n" , err); goto done; } + err =btf_dedup_standalone_fwds (d); +if (err <0 ) { +pr_debug ("btf_dedup_standalone_fwd failed:%d\n" , err); + goto done; + } err =btf_dedup_ref_types (d);if (err <0 ) {pr_debug ("btf_dedup_ref_types failed:%d\n" , err);
@@ -4525,6 +4532,169 @@ static int btf_dedup_ref_types(struct btf_dedup *d)return 0 ; } +/* +* `name_off_map` maps name offsets to type ids (essentially __u32 -> __u32). +* +* The __u32 key/value representations are cast to `void *` before passing +* to `hashmap__*` functions. These pseudo-pointers are never dereferenced. +* +*/ +static struct hashmap *name_off_map__new (void ) +{ +return hashmap__new (btf_dedup_identity_hash_fn, + btf_dedup_equal_fn, +NULL ); +} + +static int name_off_map__find (struct hashmap *map,__u32 name_off,__u32 *type_id) +{ +/* This has to be sizeof(void *) in order to be passed to hashmap__find */ +void *tmp; +int found =hashmap__find (map, (void *)(ptrdiff_t )name_off, &tmp); +/* +* __u64 cast is necessary to avoid pointer to integer conversion size warning. +* It is fine to get rid of this warning as `void *` is used as an integer value. +*/ +if (found) + *type_id = (__u64 )tmp; +return found; +} + +static int name_off_map__set (struct hashmap *map,__u32 name_off,__u32 type_id) +{ +return hashmap__set (map, (void *)(size_t )name_off, (void *)(size_t )type_id, +NULL ,NULL ); +} + +/* +* Collect a `name_off_map` that maps type names to type ids for all +* canonical structs and unions. If the same name is shared by several +* canonical types use a special value 0 to indicate this fact. +*/ +static int btf_dedup_fill_unique_names_map (struct btf_dedup *d,struct hashmap *names_map) +{ +int i, err =0 ; +__u32 type_id, collision_id; +__u16 kind; +struct btf_type *t; + +for (i =0 ; i < d->btf->nr_types; i++) { + type_id = d->btf->start_id + i; + t =btf_type_by_id (d->btf, type_id); + kind =btf_kind (t); + +if (kind !=BTF_KIND_STRUCT && kind !=BTF_KIND_UNION ) +continue ; + +/* Skip non-canonical types */ +if (type_id != d->map[type_id]) +continue ; + + err =0 ; +if (name_off_map__find (names_map, t->name_off, &collision_id)) { +/* Mark non-unique names with 0 */ +if (collision_id !=0 && collision_id != type_id) + err =name_off_map__set (names_map, t->name_off,0 ); + }else { + err =name_off_map__set (names_map, t->name_off, type_id); + } + +if (err <0 ) +return err; + } + +return 0 ; +} + +static int btf_dedup_standalone_fwd (struct btf_dedup *d, +struct hashmap *names_map, +__u32 type_id) +{ +struct btf_type *t =btf_type_by_id (d->btf, type_id); +__u16 kind =btf_kind (t); +enum btf_fwd_kind fwd_kind =BTF_INFO_KFLAG (t->info); + +struct btf_type *cand_t; +__u16 cand_kind; +__u32 cand_id =0 ; + +if (kind !=BTF_KIND_FWD ) +return 0 ; + +/* Skip if this FWD already has a mapping */ +if (type_id != d->map[type_id]) +return 0 ; + +name_off_map__find (names_map, t->name_off, &cand_id); +if (!cand_id) +return 0 ; + + cand_t =btf_type_by_id (d->btf, cand_id); + cand_kind =btf_kind (cand_t); +if (!(cand_kind ==BTF_KIND_STRUCT && fwd_kind ==BTF_FWD_STRUCT ) && + !(cand_kind ==BTF_KIND_UNION && fwd_kind ==BTF_FWD_UNION )) +return 0 ; + + d->map[type_id] = cand_id; + +return 0 ; +} + +/* +* Standalone fwd declarations deduplication. +* +* The lion's share of all FWD declarations is resolved during +* `btf_dedup_struct_types` phase when different type graphs are +* compared against each other. However, if in some compilation unit a +* FWD declaration is not a part of a type graph compared against +* another type graph that declaration's canonical type would not be +* changed. Example: +* +* CU #1: +* +* struct foo; +* struct foo *some_global; +* +* CU #2: +* +* struct foo { int u; }; +* struct foo *another_global; +* +* After `btf_dedup_struct_types` the BTF looks as follows: +* +* [1] STRUCT 'foo' size=4 vlen=1 ... +* [2] INT 'int' size=4 ... +* [3] PTR '(anon)' type_id=1 +* [4] FWD 'foo' fwd_kind=struct +* [5] PTR '(anon)' type_id=4 +* +* This pass assumes that such FWD declarations should be mapped to +* structs or unions with identical name in case if the name is not +* ambiguous. +*/ +static int btf_dedup_standalone_fwds (struct btf_dedup *d) +{ +int i, err; +struct hashmap *names_map =name_off_map__new (); + +if (!names_map) +return -ENOMEM ; + + err =btf_dedup_fill_unique_names_map (d, names_map); +if (err <0 ) + goto exit; + +for (i =0 ; i < d->btf->nr_types; i++) { + err =btf_dedup_standalone_fwd (d, names_map, d->btf->start_id + i); +if (err <0 ) + goto exit; + } + +exit: +hashmap__free (names_map); +return err; +} +/* * Compact types. *
Signature · 1 line
2.34.1
Re: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations
Quoted 33 lines · Eduard Zingerman
Deduplicate forward declarations that don't take part in type graphs comparisons if declaration name is unambiguous. Example:
CU #1:
struct foo; // standalone forward declaration
struct foo*some_global;CU #2:
struct foo { int x; };
struct foo*another_global;The
struct foofrom CU #1 is not a part of any definition that is compared against another definition whilebtf_dedup_struct_typesprocesses structural types. The the BTF afterbtf_dedup_struct_typesthe BTF looks as follows:[1] STRUCT 'foo' size=4 vlen=1 ...
[2] INT 'int' size=4 ...
[3] PTR '(anon)' type_id=1
[4] FWD 'foo' fwd_kind=struct
[5] PTR '(anon)' type_id=4This commit adds a new pass
btf_dedup_standalone_fwds, that maps such forward declarations to structs or unions with identical name in case if the name is not ambiguous.The pass is positioned before
btf_dedup_ref_typesso that types
[3] and [5] could be merged as a same type after [1] and [4] are merged. The final result for the example above looks as follows:
Quoted 9 lines
[1] STRUCT 'foo' size=4 vlen=1
'x' type_id=2 bits_offset=0[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[3] PTR '(anon)' type_id=1For defconfig kernel with BTF enabled this removes 63 forward declarations. Examples of removed declarations:
pt_regs,in6_addr. The running time ofbtf__dedupfunction is increased by about 3%.
What about modules, can you share stats for module BTFs?
Also cc Alan as he was looking at BTF dedup improvements for kernel module BTF dedup.
Quoted 19 lines
Signed-off-by Eduard Zingerman <[email protected]>--- tools/lib/bpf/btf.c | 178 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 174 insertions(+), 4 deletions(-)tools/lib/bpf/btf.c +1
-0@@ -2881,6 +2881,7 @@ static int btf_dedup_strings(struct btf_dedup *d);static int btf_dedup_prim_types (struct btf_dedup *d);static int btf_dedup_struct_types (struct btf_dedup *d);static int btf_dedup_ref_types (struct btf_dedup *d); +static int btf_dedup_standalone_fwds (struct btf_dedup *d);static int btf_dedup_compact_types (struct btf_dedup *d);static int btf_dedup_remap_types (struct btf_dedup *d);@@ -2988,15 +2989,16 @@ static int btf_dedup_remap_types(struct btf_dedup *d);* Algorithm summary
tools/lib/bpf/btf.c static int btf_dedup_remap_types(struct btf_dedup *d); · line 2997
Quoted 10 lines
tools/lib/bpf/btf.c +2
-2* ================= * -* Algorithm completes its work in 6 separate passes: +* Algorithm completes its work in 7 separate passes: * * 1. Strings deduplication. * 2. Primitive types deduplication (int, enum, fwd). * 3. Struct/union types deduplication. -* 4. Reference types deduplication (pointers, typedefs, arrays, funcs, func +* 4. Standalone fwd declarations deduplication.
Let's call this "Resolve unambiguous forward declarations", we don't really deduplicate anything. And call the function btf_dedup_resolve_fwds()?
Quoted 28 lines
tools/lib/bpf/btf.c +11
-2+* 5. Reference types deduplication (pointers, typedefs, arrays, funcs, func * protos, and const/volatile/restrict modifiers). -* 5. Types compaction. -* 6. Types remapping. +* 6. Types compaction. +* 7. Types remapping. * * Algorithm determines canonical type descriptor, which is a single * representative type for each truly unique type. This canonical type is the @@ -3060,6 +3062,11 @@ int btf__dedup(struct btf *btf, const struct btf_dedup_opts *opts)pr_debug ("btf_dedup_struct_types failed:%d\n" , err); goto done; } + err =btf_dedup_standalone_fwds (d); +if (err <0 ) { +pr_debug ("btf_dedup_standalone_fwd failed:%d\n" , err); + goto done; + } err =btf_dedup_ref_types (d);if (err <0 ) {pr_debug ("btf_dedup_ref_types failed:%d\n" , err);@@ -4525,6 +4532,169 @@ static int btf_dedup_ref_types(struct btf_dedup *d)return 0 ; } +/* +* `name_off_map` maps name offsets to type ids (essentially __u32 -> __u32). +*
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4547
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+* The __u32 key/value representations are cast to `void *` before passing +* to `hashmap__*` functions. These pseudo-pointers are never dereferenced. +* +*/ +static struct hashmap *name_off_map__new (void ) +{ +return hashmap__new (btf_dedup_identity_hash_fn, + btf_dedup_equal_fn, +NULL ); +}
is there a point in name_off_map__new and name_off_map__find wrappers except to add one extra function to jump through when reading the code? If you look at other uses of hashmaps in this file, we use the directly. Let's drop those.
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4553
Quoted 6 lines
tools/lib/bpf/btf.c +6
-0+ +static int name_off_map__find (struct hashmap *map,__u32 name_off,__u32 *type_id) +{ +/* This has to be sizeof(void *) in order to be passed to hashmap__find */ +void *tmp; +int found =hashmap__find (map, (void *)(ptrdiff_t )name_off, &tmp);
but this (void *) casting everything was an error in API design, mea culpa. I've been wanting to switch hashmap to use long as key/value type for a long while, maybe let's do it now, as we are adding even more code that looks weird? It seems like accepting long will make hashmap API usage cleaner in most cases. There are not a lot of places where we use hashmap APIs in libbpf, but we'll also need to fix up bpftool usage, and I believe perf copy/pasted hashmap.h (cc Arnaldo), so we'd need to make sure to not break all that. But good thing it's all in the same repo and we can convert them at the same time with no breakage.
WDYT?
Quoted 4 lines
tools/lib/bpf/btf.c +4
-0+/* +* __u64 cast is necessary to avoid pointer to integer conversion size warning. +* It is fine to get rid of this warning as `void *` is used as an integer value. +*/
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4567
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+if (found) + *type_id = (__u64 )tmp; +return found; +} + +static int name_off_map__set (struct hashmap *map,__u32 name_off,__u32 type_id) +{ +return hashmap__set (map, (void *)(size_t )name_off, (void *)(size_t )type_id, +NULL ,NULL ); +}
this function will also be completely unnecessary with longs
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+ +/* +* Collect a `name_off_map` that maps type names to type ids for all +* canonical structs and unions. If the same name is shared by several +* canonical types use a special value 0 to indicate this fact. +*/ +static int btf_dedup_fill_unique_names_map (struct btf_dedup *d,struct hashmap *names_map) +{ +int i, err =0 ; +__u32 type_id, collision_id;
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+__u16 kind; +struct btf_type *t; + +for (i =0 ; i < d->btf->nr_types; i++) { + type_id = d->btf->start_id + i; + t =btf_type_by_id (d->btf, type_id); + kind =btf_kind (t); + +if (kind !=BTF_KIND_STRUCT && kind !=BTF_KIND_UNION ) +continue ;
let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4600
Quoted 13 lines
tools/lib/bpf/btf.c +13
-0+ +/* Skip non-canonical types */ +if (type_id != d->map[type_id]) +continue ; + + err =0 ; +if (name_off_map__find (names_map, t->name_off, &collision_id)) { +/* Mark non-unique names with 0 */ +if (collision_id !=0 && collision_id != type_id) + err =name_off_map__set (names_map, t->name_off,0 ); + }else { + err =name_off_map__set (names_map, t->name_off, type_id); + }
err = hashmap__add(..., t->name_off, type_id);
if (err == -EEXISTS) {
hashmap__set(..., 0);
return 0;
}see comment for hashmap_insert_strategy in hashmap.h
Quoted 6 lines
tools/lib/bpf/btf.c +6
-0+ +if (err <0 ) +return err; + } + +return 0 ;
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4616
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+} + +static int btf_dedup_standalone_fwd (struct btf_dedup *d, +struct hashmap *names_map, +__u32 type_id) +{ +struct btf_type *t =btf_type_by_id (d->btf, type_id); +__u16 kind =btf_kind (t); +enum btf_fwd_kind fwd_kind =BTF_INFO_KFLAG (t->info); +
nit: don't break variables block in two parts, there shouldn't be empty lines
also please use btf_kflag(t)
Quoted 11 lines
tools/lib/bpf/btf.c +11
-0+struct btf_type *cand_t; +__u16 cand_kind; +__u32 cand_id =0 ; + +if (kind !=BTF_KIND_FWD ) +return 0 ; + +/* Skip if this FWD already has a mapping */ +if (type_id != d->map[type_id]) +return 0 ; +
[...]
Eduard Zingerman
maintainerRe: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations
Quoted 145 lines · Andrii Nakryiko
Quoted 33 lines · Eduard Zingerman
Deduplicate forward declarations that don't take part in type graphs comparisons if declaration name is unambiguous. Example:
CU #1:
struct foo; // standalone forward declaration
struct foo*some_global;CU #2:
struct foo { int x; };
struct foo*another_global;The
struct foofrom CU #1 is not a part of any definition that is compared against another definition whilebtf_dedup_struct_typesprocesses structural types. The the BTF afterbtf_dedup_struct_typesthe BTF looks as follows:[1] STRUCT 'foo' size=4 vlen=1 ...
[2] INT 'int' size=4 ...
[3] PTR '(anon)' type_id=1
[4] FWD 'foo' fwd_kind=struct
[5] PTR '(anon)' type_id=4This commit adds a new pass
btf_dedup_standalone_fwds, that maps such forward declarations to structs or unions with identical name in case if the name is not ambiguous.The pass is positioned before
btf_dedup_ref_typesso that types
[3] and [5] could be merged as a same type after [1] and [4] are merged. The final result for the example above looks as follows:Quoted 9 lines
[1] STRUCT 'foo' size=4 vlen=1
'x' type_id=2 bits_offset=0[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[3] PTR '(anon)' type_id=1For defconfig kernel with BTF enabled this removes 63 forward declarations. Examples of removed declarations:
pt_regs,in6_addr. The running time ofbtf__dedupfunction is increased by about 3%.What about modules, can you share stats for module BTFs?
Also cc Alan as he was looking at BTF dedup improvements for kernel module BTF dedup.
Quoted 19 lines
Signed-off-by Eduard Zingerman <[email protected]>--- tools/lib/bpf/btf.c | 178 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 174 insertions(+), 4 deletions(-)tools/lib/bpf/btf.c +1
-0@@ -2881,6 +2881,7 @@ static int btf_dedup_strings(struct btf_dedup *d);static int btf_dedup_prim_types (struct btf_dedup *d);static int btf_dedup_struct_types (struct btf_dedup *d);static int btf_dedup_ref_types (struct btf_dedup *d); +static int btf_dedup_standalone_fwds (struct btf_dedup *d);static int btf_dedup_compact_types (struct btf_dedup *d);static int btf_dedup_remap_types (struct btf_dedup *d);@@ -2988,15 +2989,16 @@ static int btf_dedup_remap_types(struct btf_dedup *d);* Algorithm summary tools/lib/bpf/btf.c static int btf_dedup_remap_types(struct btf_dedup *d); · line 2997
Quoted 10 lines
tools/lib/bpf/btf.c +2
-2* ================= * -* Algorithm completes its work in 6 separate passes: +* Algorithm completes its work in 7 separate passes: * * 1. Strings deduplication. * 2. Primitive types deduplication (int, enum, fwd). * 3. Struct/union types deduplication. -* 4. Reference types deduplication (pointers, typedefs, arrays, funcs, func +* 4. Standalone fwd declarations deduplication. Let's call this "Resolve unambiguous forward declarations", we don't really deduplicate anything. And call the function
btf_dedup_resolve_fwds()?Quoted 28 lines
tools/lib/bpf/btf.c +11
-2+* 5. Reference types deduplication (pointers, typedefs, arrays, funcs, func * protos, and const/volatile/restrict modifiers). -* 5. Types compaction. -* 6. Types remapping. +* 6. Types compaction. +* 7. Types remapping. * * Algorithm determines canonical type descriptor, which is a single * representative type for each truly unique type. This canonical type is the @@ -3060,6 +3062,11 @@ int btf__dedup(struct btf *btf, const struct btf_dedup_opts *opts)pr_debug ("btf_dedup_struct_types failed:%d\n" , err); goto done; } + err =btf_dedup_standalone_fwds (d); +if (err <0 ) { +pr_debug ("btf_dedup_standalone_fwd failed:%d\n" , err); + goto done; + } err =btf_dedup_ref_types (d);if (err <0 ) {pr_debug ("btf_dedup_ref_types failed:%d\n" , err);@@ -4525,6 +4532,169 @@ static int btf_dedup_ref_types(struct btf_dedup *d)return 0 ; } +/* +* `name_off_map` maps name offsets to type ids (essentially __u32 -> __u32). +* tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4547
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+* The __u32 key/value representations are cast to `void *` before passing +* to `hashmap__*` functions. These pseudo-pointers are never dereferenced. +* +*/ +static struct hashmap *name_off_map__new (void ) +{ +return hashmap__new (btf_dedup_identity_hash_fn, + btf_dedup_equal_fn, +NULL ); +}is there a point in
name_off_map__newandname_off_map__findwrappers except to add one extra function to jump through when reading the code? If you look at other uses of hashmaps in this file, we use the directly. Let's drop those.tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4553
Quoted 6 lines
tools/lib/bpf/btf.c +6
-0+ +static int name_off_map__find (struct hashmap *map,__u32 name_off,__u32 *type_id) +{ +/* This has to be sizeof(void *) in order to be passed to hashmap__find */ +void *tmp; +int found =hashmap__find (map, (void *)(ptrdiff_t )name_off, &tmp);but this (void *) casting everything was an error in API design, mea culpa. I've been wanting to switch hashmap to use long as key/value type for a long while, maybe let's do it now, as we are adding even more code that looks weird? It seems like accepting long will make hashmap API usage cleaner in most cases. There are not a lot of places where we use hashmap APIs in libbpf, but we'll also need to fix up bpftool usage, and I believe perf copy/pasted
hashmap.h(cc Arnaldo), so we'd need to make sure to not break all that. But good thing it's all in the same repo and we can convert them at the same time with no breakage.
Quoted 1 line
WDYT?
Well, I did the change, excluding tests it amounts to:
- 15 files changed, 114 insertions(+), 137 deletions(-);
- 45 casts removed;
- 30 casts added.
TBH, it seems like I should just use "u32_as_hash_field" and be done with it. In any case I'll post this as a part of v1 series for "libbpf: Resolve unambigous forward declarations".
To account for a case when map has to store pointers and pointers are 32 bit I chose to update the map interface to be "uintptr_t -> uintptr_t". Had it been "u64 -> u64" the additional temporary variable would be necessary for "old" values, e.g. while working with hashmap__insert. (Contrary to what we discussed on Friday).
Quoted 98 lines
Quoted 5 lines
tools/lib/bpf/btf.c +4
-0+/* +* __u64 cast is necessary to avoid pointer to integer conversion size warning. +* It is fine to get rid of this warning as `void *` is used as an integer value. +*/ tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4567
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+if (found) + *type_id = (__u64 )tmp; +return found; +} + +static int name_off_map__set (struct hashmap *map,__u32 name_off,__u32 type_id) +{ +return hashmap__set (map, (void *)(size_t )name_off, (void *)(size_t )type_id, +NULL ,NULL ); +}this function will also be completely unnecessary with longs
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+ +/* +* Collect a `name_off_map` that maps type names to type ids for all +* canonical structs and unions. If the same name is shared by several +* canonical types use a special value 0 to indicate this fact. +*/ +static int btf_dedup_fill_unique_names_map (struct btf_dedup *d,struct hashmap *names_map) +{ +int i, err =0 ; +__u32 type_id, collision_id;tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+__u16 kind; +struct btf_type *t; + +for (i =0 ; i < d->btf->nr_types; i++) { + type_id = d->btf->start_id + i; + t =btf_type_by_id (d->btf, type_id); + kind =btf_kind (t); + +if (kind !=BTF_KIND_STRUCT && kind !=BTF_KIND_UNION ) +continue ;let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4600
Quoted 13 lines
tools/lib/bpf/btf.c +13
-0+ +/* Skip non-canonical types */ +if (type_id != d->map[type_id]) +continue ; + + err =0 ; +if (name_off_map__find (names_map, t->name_off, &collision_id)) { +/* Mark non-unique names with 0 */ +if (collision_id !=0 && collision_id != type_id) + err =name_off_map__set (names_map, t->name_off,0 ); + }else { + err =name_off_map__set (names_map, t->name_off, type_id); + }err =
hashmap__add(..., t->name_off, type_id);if (err == -EEXISTS) { hashmap__set(..., 0); return 0; }see comment for
hashmap_insert_strategyinhashmap.hQuoted 6 lines
tools/lib/bpf/btf.c +6
-0+ +if (err <0 ) +return err; + } + +return 0 ;tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4616
Quoted 10 lines
tools/lib/bpf/btf.c +10
-0+} + +static int btf_dedup_standalone_fwd (struct btf_dedup *d, +struct hashmap *names_map, +__u32 type_id) +{ +struct btf_type *t =btf_type_by_id (d->btf, type_id); +__u16 kind =btf_kind (t); +enum btf_fwd_kind fwd_kind =BTF_INFO_KFLAG (t->info); +nit: don't break variables block in two parts, there shouldn't be empty lines
also please use
btf_kflag(t)Quoted 11 lines
tools/lib/bpf/btf.c +11
-0+struct btf_type *cand_t; +__u16 cand_kind; +__u32 cand_id =0 ; + +if (kind !=BTF_KIND_FWD ) +return 0 ; + +/* Skip if this FWD already has a mapping */ +if (type_id != d->map[type_id]) +return 0 ; +[...]
Eduard Zingerman
maintainerRe: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations
In reply to Andrii NakryikoQuoted 1 line · Andrii Nakryiko
On Tue, Oct 25, 2022 at 3:28 PM Eduard Zingerman <[email protected]> wrote:
[...]
Quoted 14 lines
Quoted 14 lines
tools/lib/bpf/btf.c +14
-0+ +/* +* Collect a `name_off_map` that maps type names to type ids for all +* canonical structs and unions. If the same name is shared by several +* canonical types use a special value 0 to indicate this fact. +*/ +static int btf_dedup_fill_unique_names_map (struct btf_dedup *d,struct hashmap *names_map) +{ +int i, err =0 ; +__u32 type_id, collision_id; +__u16 kind; +struct btf_type *t; + +for (i =0 ; i < d->btf->nr_types; i++) {
Quoted 8 lines
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587
Quoted 6 lines
tools/lib/bpf/btf.c +6
-0+ type_id = d->btf->start_id + i; + t =btf_type_by_id (d->btf, type_id); + kind =btf_kind (t); + +if (kind !=BTF_KIND_STRUCT && kind !=BTF_KIND_UNION ) +continue ;let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0
Interestingly this is necessary only for mixed enum / enum64 case. Forward enum declarations are resolved by bpf/btf.c:btf_dedup_prim_type:
case BTF_KIND_ENUM:
h = btf_hash_enum(t);
for_each_dedup_cand(d, hash_entry, h) {
cand_id = (__u32)(long)hash_entry->value;
cand = btf_type_by_id(d->btf, cand_id);
if (btf_equal_enum(t, cand)) {
new_id = cand_id;
break;
}
if (btf_compat_enum(t, cand)) {
if (btf_is_enum_fwd(t)) {
/* resolve fwd to full enum */
new_id = cand_id;
break;
}
/* resolve canonical enum fwd to full enum */
d->map[cand_id] = type_id;
}
}
break;
// ... similar logic for ENUM64 ...- btf_hash_enum ignores vlen when hashing;
- btf_compat_enum compares only names and sizes.
So, if forward and main declaration kinds match (either BTF_KIND_ENUM or BTF_KIND_ENUM64) the forward declaration would be removed. But if the kinds are different the forward declaration would remain. E.g.:
CU #1:
enum foo;
enum foo *a;
CU #2:
enum foo { x = 0xfffffffff };
enum foo *b;
BTF:
[1] ENUM64 'foo' encoding=UNSIGNED size=8 vlen=1
'x' val=68719476735ULL
[2] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
[3] PTR '(anon)' type_id=1
[4] ENUM 'foo' encoding=UNSIGNED size=4 vlen=0
[5] PTR '(anon)' type_id=4
BTF_KIND_FWDs are unified during btf_dedup_struct_types but enum forward declarations are not. So it would be incorrect to add enum forward declaration unification logic to btf_dedup_resolve_fwds, because the following case would not be covered:
CU #1:
enum foo;
struct s { enum foo *a; } *a;
CU #2:
enum foo { x = 0xfffffffff };
struct s { enum foo *a; } *b;
Currently STRUCTs 's' are not de-duplicated.
I think that btf_dedup_prim_type should be adjusted to handle this case.
[...]
Re: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations
Quoted 26 lines · Eduard Zingerman
Quoted 1 line · Andrii Nakryiko
On Tue, Oct 25, 2022 at 3:28 PM Eduard Zingerman <[email protected]> wrote:
[...]
Quoted 14 lines
Quoted 14 lines
tools/lib/bpf/btf.c +14
-0+ +/* +* Collect a `name_off_map` that maps type names to type ids for all +* canonical structs and unions. If the same name is shared by several +* canonical types use a special value 0 to indicate this fact. +*/ +static int btf_dedup_fill_unique_names_map (struct btf_dedup *d,struct hashmap *names_map) +{ +int i, err =0 ; +__u32 type_id, collision_id; +__u16 kind; +struct btf_type *t; + +for (i =0 ; i < d->btf->nr_types; i++) {Quoted 8 lines
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587
Quoted 6 lines
tools/lib/bpf/btf.c +6
-0+ type_id = d->btf->start_id + i; + t =btf_type_by_id (d->btf, type_id); + kind =btf_kind (t); + +if (kind !=BTF_KIND_STRUCT && kind !=BTF_KIND_UNION ) +continue ;let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0
Quoted 3 lines
Interestingly this is necessary only for mixed enum / enum64 case. Forward enum declarations are resolved by bpf/btf.c:btf_dedup_prim_type:
Ah, great catch! A forward can look like an enum to one CU but another CU can specify values that make it an enum64.
Quoted 54 lines
case BTF_KIND_ENUM: h = btf_hash_enum(t); for_each_dedup_cand(d, hash_entry, h) { cand_id = (__u32)(long)hash_entry->value; cand = btf_type_by_id(d->btf, cand_id); if (btf_equal_enum(t, cand)) { new_id = cand_id; break; } if (btf_compat_enum(t, cand)) { if (btf_is_enum_fwd(t)) { /* resolve fwd to full enum */ new_id = cand_id; break; } /* resolve canonical enum fwd to full enum */ d->map[cand_id] = type_id; } } break; // ... similar logic for ENUM64 ...-
btf_hash_enumignores vlen when hashing;
-btf_compat_enumcompares only names and sizes.So, if forward and main declaration kinds match (either
BTF_KIND_ENUMorBTF_KIND_ENUM64) the forward declaration would be removed. But if the kinds are different the forward declaration would remain. E.g.:CU #1:
enum foo;
enum foo *a;CU #2:
enum foo { x =0xfffffffff};
enum foo *b;BTF:
[1] ENUM64 'foo' encoding=UNSIGNED size=8 vlen=1'x' val=68719476735ULL[2] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
[3] PTR '(anon)' type_id=1
[4] ENUM 'foo' encoding=UNSIGNED size=4 vlen=0
[5] PTR '(anon)' type_id=4
BTF_KIND_FWDsare unified duringbtf_dedup_struct_typesbut enum forward declarations are not. So it would be incorrect to add enum forward declaration unification logic tobtf_dedup_resolve_fwds, because the following case would not be covered:CU #1:
enum foo;
struct s { enum foo *a; } *a;
Quoted 6 lines
CU #2:
enum foo { x =0xfffffffff};
struct s { enum foo *a; } *b;Currently STRUCTs 's' are not de-duplicated.
What if CU#1 is in base BTF and CU#2 in split module BTF? I think we'd explicitly want to avoid deduping "struct s" then since we can't be sure that it is the same enum they are pointing at. That's the logic we employ for structs at least, based upon the rationale that we can't feed back knowledge of types from module to kernel BTF since the latter is now fixed (Andrii, do correct me if I have this wrong). In such a case the enum is no longer standalone; it serves the purpose of allowing us to define a pointer to a module-specific type. We recently found some examples of this sort of thing with structs, where the struct was defined in module BTF, making dedup fail for some core kernel data types, but the problem was restricted to modules which _did_ define the type so wasn't a major driver of dedup failures. Not sure if there's many (any?) enum cases of this in practice.
I suppose if we could guarantee the dedup happened within the same object (kernel or module) we could relax this constraint though?
Alan
Eduard Zingerman
maintainerRe: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations
Quoted 95 lines · Alan Maguire
Quoted 26 lines · Eduard Zingerman
Quoted 1 line · Andrii Nakryiko
On Tue, Oct 25, 2022 at 3:28 PM Eduard Zingerman <[email protected]> wrote:
[...]
Quoted 14 lines
Quoted 14 lines
tools/lib/bpf/btf.c +14
-0+ +/* +* Collect a `name_off_map` that maps type names to type ids for all +* canonical structs and unions. If the same name is shared by several +* canonical types use a special value 0 to indicate this fact. +*/ +static int btf_dedup_fill_unique_names_map (struct btf_dedup *d,struct hashmap *names_map) +{ +int i, err =0 ; +__u32 type_id, collision_id; +__u16 kind; +struct btf_type *t; + +for (i =0 ; i < d->btf->nr_types; i++) {Quoted 8 lines
tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587
Quoted 6 lines
tools/lib/bpf/btf.c +6
-0+ type_id = d->btf->start_id + i; + t =btf_type_by_id (d->btf, type_id); + kind =btf_kind (t); + +if (kind !=BTF_KIND_STRUCT && kind !=BTF_KIND_UNION ) +continue ;let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0
Quoted 3 lines
Interestingly this is necessary only for mixed enum / enum64 case. Forward enum declarations are resolved by bpf/btf.c:btf_dedup_prim_type:
Ah, great catch! A forward can look like an enum to one CU but another CU can specify values that make it an enum64.
Quoted 54 lines
case BTF_KIND_ENUM: h = btf_hash_enum(t); for_each_dedup_cand(d, hash_entry, h) { cand_id = (__u32)(long)hash_entry->value; cand = btf_type_by_id(d->btf, cand_id); if (btf_equal_enum(t, cand)) { new_id = cand_id; break; } if (btf_compat_enum(t, cand)) { if (btf_is_enum_fwd(t)) { /* resolve fwd to full enum */ new_id = cand_id; break; } /* resolve canonical enum fwd to full enum */ d->map[cand_id] = type_id; } } break; // ... similar logic for ENUM64 ...-
btf_hash_enumignores vlen when hashing;
-btf_compat_enumcompares only names and sizes.So, if forward and main declaration kinds match (either
BTF_KIND_ENUMorBTF_KIND_ENUM64) the forward declaration would be removed. But if the kinds are different the forward declaration would remain. E.g.:CU #1:
enum foo;
enum foo *a;CU #2:
enum foo { x =0xfffffffff};
enum foo *b;BTF:
[1] ENUM64 'foo' encoding=UNSIGNED size=8 vlen=1'x' val=68719476735ULL[2] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
[3] PTR '(anon)' type_id=1
[4] ENUM 'foo' encoding=UNSIGNED size=4 vlen=0
[5] PTR '(anon)' type_id=4
BTF_KIND_FWDsare unified duringbtf_dedup_struct_typesbut enum forward declarations are not. So it would be incorrect to add enum forward declaration unification logic tobtf_dedup_resolve_fwds, because the following case would not be covered:CU #1:
enum foo;
struct s { enum foo *a; } *a;Quoted 6 lines
CU #2:
enum foo { x =0xfffffffff};
struct s { enum foo *a; } *b;Currently STRUCTs 's' are not de-duplicated.
Quoted 12 lines
What if CU#1 is in base BTF and CU#2 in split module BTF? I think we'd explicitly want to avoid deduping "struct s" then since we can't be sure that it is the same enum they are pointing at. That's the logic we employ for structs at least, based upon the rationale that we can't feed back knowledge of types from module to kernel BTF since the latter is now fixed (Andrii, do correct me if I have this wrong). In such a case the enum is no longer standalone; it serves the purpose of allowing us to define a pointer to a module-specific type. We recently found some examples of this sort of thing with structs, where the struct was defined in module BTF, making dedup fail for some core kernel data types, but the problem was restricted to modules which _did_ define the type so wasn't a major driver of dedup failures. Not sure if there's many (any?) enum cases of this in practice.
Hi Alan,
As far as I understand the loop in btf_dedup_prim_types guarantees that only ids from the split module would be remapped:
struct btf {
...
/* BTF type ID of the first type in this BTF instance:
* - for base BTF it's equal to 1;
* - for split BTF it's equal to biggest type ID of base BTF plus 1.
*/
int start_id;
...
}
...
for (i = 0; i < d->btf->nr_types; i++) {
err = btf_dedup_prim_type(d, d->btf->start_id + i);
if (err)
return err;
}Thus CU1:foo won't be updated to be CU2:foo and CU1:s will not be the same as CU2:s. Is that right or am I confused?
Thanks,
Eduard
Quoted 5 lines
I suppose if we could guarantee the dedup happened within the same object (kernel or module) we could relax this constraint though?
Alan
Eduard Zingerman
maintainerRFC 2/12 selftests/bpf: Tests for standalone forward BTF declarations deduplication
In reply to Eduard ZingermanTests to verify the following behavior of btf_dedup_standalone_fwds:
- remapping for struct forward declarations;
- remapping for union forward declarations;
- no remapping if forward declaration kind does not match similarly named struct or union declaration;
- no remapping if forward declaration name is ambiguous.
--- tools/testing/selftests/bpf/prog_tests/btf.c | 152 +++++++++++++++++++ 1 file changed, 152 insertions(+)
tools/testing/selftests/bpf/prog_tests/btf.c +152 -0
@@ -7598,6 +7598,158 @@ static struct btf_dedup_test dedup_tests[] = { BTF_STR_SEC("\0e1\0e1_val"), }, }, +{ + .descr = "dedup: standalone fwd declaration struct", +/* +* // CU 1: +* struct foo { int x; }; +* struct foo *a; +* +* // CU 2: +* struct foo; +* struct foo *b; +*/ + .input = { + .raw_types = { +BTF_STRUCT_ENC (NAME_NTH (1 ),1 ,4 ),/* [1] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_TYPE_INT_ENC (0 ,BTF_INT_SIGNED ,0 ,32 ,4 ),/* [2] */ +BTF_PTR_ENC (1 ),/* [3] */ +BTF_FWD_ENC (NAME_TBD ,0 ),/* [4] */ +BTF_PTR_ENC (4 ),/* [5] */ +BTF_END_RAW , + }, + BTF_STR_SEC("\0foo\0x"), + }, + .expect = { + .raw_types = { +BTF_STRUCT_ENC (NAME_NTH (1 ),1 ,4 ),/* [1] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_TYPE_INT_ENC (0 ,BTF_INT_SIGNED ,0 ,32 ,4 ),/* [2] */ +BTF_PTR_ENC (1 ),/* [3] */ +BTF_END_RAW , + }, + BTF_STR_SEC("\0foo\0x"), + }, +}, +{ + .descr ="dedup: standalone fwd declaration union" , +/* +* // CU 1: +* union foo { int x; }; +* union foo *another_global; +* +* // CU 2: +* union foo; +* union foo *some_global; +*/ + .input = { + .raw_types = { +BTF_UNION_ENC (NAME_NTH (1 ),1 ,4 ),/* [1] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_TYPE_INT_ENC (0 ,BTF_INT_SIGNED ,0 ,32 ,4 ),/* [2] */ +BTF_PTR_ENC (1 ),/* [3] */ +BTF_FWD_ENC (NAME_TBD ,1 ),/* [4] */ +BTF_PTR_ENC (4 ),/* [5] */ +BTF_END_RAW , + }, + BTF_STR_SEC("\0foo\0x"), + }, + .expect = { + .raw_types = { +BTF_UNION_ENC (NAME_NTH (1 ),1 ,4 ),/* [1] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_TYPE_INT_ENC (0 ,BTF_INT_SIGNED ,0 ,32 ,4 ),/* [2] */ +BTF_PTR_ENC (1 ),/* [3] */ +BTF_END_RAW , + }, + BTF_STR_SEC("\0foo\0x"), + }, +}, +{ + .descr ="dedup: standalone fwd declaration wrong kind" , +/* +* // CU 1: +* struct foo { int x; }; +* struct foo *b; +* +* // CU 2: +* union foo; +* union foo *a; +*/ + .input = { + .raw_types = { +BTF_STRUCT_ENC (NAME_NTH (1 ),1 ,4 ),/* [1] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_TYPE_INT_ENC (0 ,BTF_INT_SIGNED ,0 ,32 ,4 ),/* [2] */ +BTF_PTR_ENC (1 ),/* [3] */ +BTF_FWD_ENC (NAME_TBD ,1 ),/* [4] */ +BTF_PTR_ENC (4 ),/* [5] */ +BTF_END_RAW , + }, + BTF_STR_SEC("\0foo\0x"), + }, + .expect = { + .raw_types = { +BTF_STRUCT_ENC (NAME_NTH (1 ),1 ,4 ),/* [1] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_TYPE_INT_ENC (0 ,BTF_INT_SIGNED ,0 ,32 ,4 ),/* [2] */ +BTF_PTR_ENC (1 ),/* [3] */ +BTF_FWD_ENC (NAME_TBD ,1 ),/* [4] */ +BTF_PTR_ENC (4 ),/* [5] */ +BTF_END_RAW , + }, + BTF_STR_SEC("\0foo\0x"), + }, +}, +{ + .descr ="dedup: standalone fwd declaration name conflict" , +/* +* // CU 1: +* struct foo { int x; }; +* struct foo *a; +* +* // CU 2: +* struct foo; +* struct foo *b; +* +* // CU 3: +* struct foo { int x; int y; }; +* struct foo *c; +*/ + .input = { + .raw_types = { +BTF_STRUCT_ENC (NAME_NTH (1 ),1 ,4 ),/* [1] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_TYPE_INT_ENC (0 ,BTF_INT_SIGNED ,0 ,32 ,4 ),/* [2] */ +BTF_PTR_ENC (1 ),/* [3] */ +BTF_FWD_ENC (NAME_TBD ,0 ),/* [4] */ +BTF_PTR_ENC (4 ),/* [5] */ +BTF_STRUCT_ENC (NAME_NTH (1 ),2 ,8 ),/* [6] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_MEMBER_ENC (NAME_NTH (3 ),2 ,0 ), +BTF_PTR_ENC (6 ),/* [7] */ +BTF_END_RAW , + }, + BTF_STR_SEC("\0foo\0x\0y"), + }, + .expect = { + .raw_types = { +BTF_STRUCT_ENC (NAME_NTH (1 ),1 ,4 ),/* [1] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_TYPE_INT_ENC (0 ,BTF_INT_SIGNED ,0 ,32 ,4 ),/* [2] */ +BTF_PTR_ENC (1 ),/* [3] */ +BTF_FWD_ENC (NAME_TBD ,0 ),/* [4] */ +BTF_PTR_ENC (4 ),/* [5] */ +BTF_STRUCT_ENC (NAME_NTH (1 ),2 ,8 ),/* [6] */ +BTF_MEMBER_ENC (NAME_NTH (2 ),2 ,0 ), +BTF_MEMBER_ENC (NAME_NTH (3 ),2 ,0 ), +BTF_PTR_ENC (6 ),/* [7] */ +BTF_END_RAW , + }, + BTF_STR_SEC("\0foo\0x\0y"), + }, +}, };
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRFC 3/12 libbpf: Support for BTF_DECL_TAG dump in C format
In reply to Eduard ZingermanAt C level BTF_DECL_TAGs are represented as __attribute__ declarations, e.g.:
struct foo {
...;
} __attribute__((btf_decl_tag("bar")));This commit covers only decl tags attached to structs and unions.
BTF doc says that BTF_DECL_TAGs should follow a target type but this is not enforced and tests don't honor this restriction. This commit uses hash table to map types to the list of decl tags.
--- tools/lib/bpf/btf_dump.c | 143 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 142 insertions(+), 1 deletion(-)
tools/lib/bpf/btf_dump.c +142 -1
@@ -75,6 +75,15 @@ struct btf_dump_data {bool is_array_char; }; +/* +* An array of ids of BTF_DECL_TAG objects associated with a specific type. +*/ +struct decl_tag_array { +__u16 cnt; +__u16 cap; +__u32 tag_ids[0 ]; +}; +struct btf_dump {const struct btf *btf;btf_dump_printf_fn_t printf_fn;
@@ -111,6 +120,11 @@ struct btf_dump {* name occurrences */ struct hashmap *ident_names; +/* +* maps type id to decl_tag_array, assume that relatively small +* fraction of types has btf_decl_tag's attached +*/ +struct hashmap *decl_tags;/* * data for typed display; allocated if needed. */
@@ -127,6 +141,26 @@ static bool str_equal_fn(const void *a, const void *b, void *ctx)return strcmp (a, b) ==0 ; } +static size_t int_hash_fn (const void *key,void *ctx) +{ +int i; +size_t h =0 ; +char *bytes = (char *)key; + +for (i =0 ; i <4 ; ++i) + h = h *31 + bytes[i]; + +return h; +} + +static bool int_equal_fn (const void *a,const void *b,void *ctx) +{ +int *ia = (int *)a; +int *ib = (int *)b; + +return *ia == *ib; +} +static const char *btf_name_of (const struct btf_dump *d,__u32 name_off) {return btf__name_by_offset (d->btf, name_off);
@@ -143,6 +177,7 @@ static void btf_dump_printf(const struct btf_dump *d, const char *fmt, ...)static int btf_dump_mark_referenced (struct btf_dump *d);static int btf_dump_resize (struct btf_dump *d); +static int btf_dump_assign_decl_tags (struct btf_dump *d); struct btf_dump *btf_dump__new(const struct btf *btf, btf_dump_printf_fn_t printf_fn,
@@ -179,11 +214,24 @@ struct btf_dump *btf_dump__new(const struct btf *btf, d->ident_names =NULL ; goto err; } + d->decl_tags =hashmap__new (int_hash_fn, int_equal_fn,NULL ); +if (IS_ERR (d->decl_tags)) { + err =PTR_ERR (d->decl_tags); + d->decl_tags =NULL ; + goto err; + } err =btf_dump_resize (d);if (err) goto err; + err =btf_dump_assign_decl_tags (d); +if (err) + goto err; + +if (err) + goto err; +return d; err:btf_dump__free (d);
@@ -232,7 +280,8 @@ static void btf_dump_free_names(struct hashmap *map)void btf_dump__free (struct btf_dump *d) { -int i; +int i, bkt; +struct hashmap_entry *cur;if (IS_ERR_OR_NULL (d))return ;
@@ -250,6 +299,9 @@ void btf_dump__free(struct btf_dump *d)free (d->decl_stack);btf_dump_free_names (d->type_names);btf_dump_free_names (d->ident_names); +hashmap__for_each_entry (d->decl_tags, cur, bkt) +free (cur->value); +hashmap__free (d->decl_tags);free (d); }
@@ -373,6 +425,77 @@ static int btf_dump_mark_referenced(struct btf_dump *d)return 0 ; } +static struct decl_tag_array *btf_dump_find_decl_tags (struct btf_dump *d,__u32 id) +{ +struct decl_tag_array *decl_tags =NULL ; + +hashmap__find (d->decl_tags, &id, (void **)&decl_tags); + +return decl_tags; +} + +static struct decl_tag_array *realloc_decl_tags (struct decl_tag_array *tags,__u16 new_cap) +{ +size_t new_size =sizeof (struct decl_tag_array ) + new_cap *sizeof (__u32); +struct decl_tag_array *new_tags = (tags + ?realloc (tags, new_size) + :calloc (1 , new_size)); + +if (!new_tags) +return NULL ; + + new_tags->cap = new_cap; + +return new_tags; +} + +/* +* Scans all BTF objects looking for BTF_KIND_DECL_TAG entries. +* The id's of the entries are stored in the `btf_dump.decl_tags` table, +* grouped by a target type. +*/ +static int btf_dump_assign_decl_tags (struct btf_dump *d) +{ +int err; +__u32 id; +__u32 n =btf__type_cnt (d->btf); +__u32 new_capacity; +const struct btf_type *t; +struct decl_tag_array *decl_tags; + +for (id =0 ; id < n; id++) { + t =btf__type_by_id (d->btf, id); + +if (btf_kind (t) !=BTF_KIND_DECL_TAG ) +continue ; + + decl_tags =btf_dump_find_decl_tags (d, t->type); +if (!decl_tags) { + decl_tags =realloc_decl_tags (NULL ,1 ); +if (!decl_tags) +return -ENOMEM ; + err =hashmap__insert (d->decl_tags, &t->type, decl_tags, +HASHMAP_SET ,NULL ,NULL ); +if (err) +return err; + }else if (decl_tags->cnt == decl_tags->cap) { + new_capacity = decl_tags->cap *2 ; +if (new_capacity >0xffff ) +return -ERANGE ; + decl_tags =realloc_decl_tags (decl_tags, new_capacity); +if (!decl_tags) +return -ENOMEM ; + decl_tags->cap = new_capacity; + err =hashmap__update (d->decl_tags, &t->type, decl_tags,NULL ,NULL ); +if (err) +return err; + } + decl_tags->tag_ids[decl_tags->cnt++] = id; + } + +return 0 ; +} +static int btf_dump_add_emit_queue_id (struct btf_dump *d,__u32 id) {__u32 *new_queue;
@@ -899,6 +1022,23 @@ static void btf_dump_emit_bit_padding(const struct btf_dump *d, } } +static void btf_dump_emit_decl_tags (struct btf_dump *d,__u32 id) +{ +struct decl_tag_array *decl_tags =btf_dump_find_decl_tags (d, id); +struct btf_type *decl_tag_t; +const char *decl_tag_text; +__u32 i; + +if (!decl_tags) +return ; + +for (i =0 ; i < decl_tags->cnt; ++i) { + decl_tag_t =btf_type_by_id (d->btf, decl_tags->tag_ids[i]); + decl_tag_text =btf__name_by_offset (d->btf, decl_tag_t->name_off); +btf_dump_printf (d," __attribute__((btf_decl_tag(\"%s\")))" , decl_tag_text); + } +} +static void btf_dump_emit_struct_fwd (struct btf_dump *d,__u32 id,const struct btf_type *t) {
@@ -964,6 +1104,7 @@ static void btf_dump_emit_struct_def(struct btf_dump *d,btf_dump_printf (d,"%s}" ,pfx (lvl));if (packed)btf_dump_printf (d," __attribute__((packed))" ); +btf_dump_emit_decl_tags (d, id); }static const char *missing_base_types[][2 ] = {
Signature · 1 line
2.34.1
Re: [RFC bpf-next 03/12] libbpf: Support for BTF_DECL_TAG dump in C format
Quoted 56 lines · Eduard Zingerman
At C level
BTF_DECL_TAGsare represented as__attribute__declarations, e.g.:struct foo { ...; } __attribute__((btf_decl_tag("bar")));This commit covers only decl tags attached to structs and unions.
BTF doc says that
BTF_DECL_TAGsshould follow a target type but this is not enforced and tests don't honor this restriction. This commit uses hash table to map types to the list of decl tags.Signed-off-by Eduard Zingerman <[email protected]>--- tools/lib/bpf/btf_dump.c | 143 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 142 insertions(+), 1 deletion(-)tools/lib/bpf/btf_dump.c +15
-0@@ -75,6 +75,15 @@ struct btf_dump_data {bool is_array_char; }; +/* +* An array of ids of BTF_DECL_TAG objects associated with a specific type. +*/ +struct decl_tag_array { +__u16 cnt; +__u16 cap; +__u32 tag_ids[0 ]; +}; +struct btf_dump {const struct btf *btf;btf_dump_printf_fn_t printf_fn;@@ -111,6 +120,11 @@ struct btf_dump {* name occurrences */ struct hashmap *ident_names; +/* +* maps type id to decl_tag_array, assume that relatively small +* fraction of types has btf_decl_tag's attached +*/ +struct hashmap *decl_tags;/* * data for typed display; allocated if needed. */ @@ -127,6 +141,26 @@ static bool str_equal_fn(const void *a, const void *b, void *ctx)return strcmp (a, b) ==0 ; } +static size_t int_hash_fn (const void *key,void *ctx)
tools/lib/bpf/btf_dump.c static bool str_equal_fn(const void *a, const void *b, void *ctx) · line 154
Quoted 10 lines
tools/lib/bpf/btf_dump.c +10
-0+{ +int i; +size_t h =0 ; +char *bytes = (char *)key; + +for (i =0 ; i <4 ; ++i) + h = h *31 + bytes[i]; + +return h; +}
no need, you can just do what btf_dedup_identity_hash_fn() is doing and pass int/long/size_t as is, hashmap implementation does additional multiplicative hashing on top to find a bucket
tools/lib/bpf/btf_dump.c static bool str_equal_fn(const void *a, const void *b, void *ctx) · line 162
Quoted 8 lines
tools/lib/bpf/btf_dump.c +8
-0+ +static bool int_equal_fn (const void *a,const void *b,void *ctx) +{ +int *ia = (int *)a; +int *ib = (int *)b; + +return *ia == *ib; +}
see btf_dedup_equal_fn(), no need for casting, just return a == b;
Quoted 24 lines
tools/lib/bpf/btf_dump.c +8
-0+static const char *btf_name_of (const struct btf_dump *d,__u32 name_off) {return btf__name_by_offset (d->btf, name_off);@@ -143,6 +177,7 @@ static void btf_dump_printf(const struct btf_dump *d, const char *fmt, ...)static int btf_dump_mark_referenced (struct btf_dump *d);static int btf_dump_resize (struct btf_dump *d); +static int btf_dump_assign_decl_tags (struct btf_dump *d); struct btf_dump *btf_dump__new(const struct btf *btf, btf_dump_printf_fn_t printf_fn,@@ -179,11 +214,24 @@ struct btf_dump *btf_dump__new(const struct btf *btf, d->ident_names =NULL ; goto err; } + d->decl_tags =hashmap__new (int_hash_fn, int_equal_fn,NULL ); +if (IS_ERR (d->decl_tags)) { + err =PTR_ERR (d->decl_tags); + d->decl_tags =NULL ; + goto err; + } err =btf_dump_resize (d);
tools/lib/bpf/btf_dump.c struct btf_dump *btf_dump__new(const struct btf *btf, · line 234
Quoted 10 lines
tools/lib/bpf/btf_dump.c +7
-0if (err) goto err; + err =btf_dump_assign_decl_tags (d); +if (err) + goto err; + +if (err) + goto err; +
I like the bullet-proof error checking, but checking just once should be enough ;)
Quoted 24 lines
tools/lib/bpf/btf_dump.c +5
-1return d; err:btf_dump__free (d);@@ -232,7 +280,8 @@ static void btf_dump_free_names(struct hashmap *map)void btf_dump__free (struct btf_dump *d) { -int i; +int i, bkt; +struct hashmap_entry *cur;if (IS_ERR_OR_NULL (d))return ;@@ -250,6 +299,9 @@ void btf_dump__free(struct btf_dump *d)free (d->decl_stack);btf_dump_free_names (d->type_names);btf_dump_free_names (d->ident_names); +hashmap__for_each_entry (d->decl_tags, cur, bkt) +free (cur->value); +hashmap__free (d->decl_tags);free (d); }
tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 428
Quoted 4 lines
tools/lib/bpf/btf_dump.c +1
-0@@ -373,6 +425,77 @@ static int btf_dump_mark_referenced(struct btf_dump *d)return 0 ; } +static struct decl_tag_array *btf_dump_find_decl_tags (struct btf_dump *d,__u32 id)
do we really need this wrapper?
tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 432
Quoted 4 lines
tools/lib/bpf/btf_dump.c +4
-0+{ +struct decl_tag_array *decl_tags =NULL ; + +hashmap__find (d->decl_tags, &id, (void **)&decl_tags);
this &id also made me realize that this is all broken, you are remembering random pointers in hashmap (they point onto stack, which gets reused once this function returns; but hashmap remember it, so on next lookup or update we are going to be reading random values in int_equal_fn?)
you should be passing (void *)(long)id instead (and better yet let's refactor hashmap API as I suggested in previous patch)
either I'm missing something, or this works by accident, which suggests that tests could be improved maybe?..
tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 442
Quoted 10 lines
tools/lib/bpf/btf_dump.c +10
-0+ +return decl_tags; +} + +static struct decl_tag_array *realloc_decl_tags (struct decl_tag_array *tags,__u16 new_cap) +{ +size_t new_size =sizeof (struct decl_tag_array ) + new_cap *sizeof (__u32); +struct decl_tag_array *new_tags = (tags + ?realloc (tags, new_size) + :calloc (1 , new_size));
realloc allocates if passed NULL, so no need for calloc, assuming proper initialization
but let's use libbpf_reallocarray(), we'll waste few bytes on size_t, but given we expect few tags, it's not a big deal
Quoted 12 lines
tools/lib/bpf/btf_dump.c +12
-0+ +if (!new_tags) +return NULL ; + + new_tags->cap = new_cap; + +return new_tags; +} + +/* +* Scans all BTF objects looking for BTF_KIND_DECL_TAG entries. +* The id's of the entries are stored in the `btf_dump.decl_tags` table,
tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 464
Quoted 10 lines
tools/lib/bpf/btf_dump.c +10
-0+* grouped by a target type. +*/ +static int btf_dump_assign_decl_tags (struct btf_dump *d) +{ +int err; +__u32 id; +__u32 n =btf__type_cnt (d->btf); +__u32 new_capacity; +const struct btf_type *t; +struct decl_tag_array *decl_tags;
few nits: generally, for new code try to do reverse Christmas try style, where widest line is at the top, shortest at the bottom
but also here you can have id, new_capacity, and n on same line
and s/new_capacity/new_cap/
tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 466
Quoted 2 lines
tools/lib/bpf/btf_dump.c +2
-0+ +for (id =0 ; id < n; id++) {
0 is VOID, we never really need to process it, just start with id = 1
tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 470
Quoted 4 lines
tools/lib/bpf/btf_dump.c +4
-0+ t =btf__type_by_id (d->btf, id); + +if (btf_kind (t) !=BTF_KIND_DECL_TAG ) +continue ;
if (!btf_is_decl_tag(t))
continue;
Quoted 12 lines
tools/lib/bpf/btf_dump.c +12
-0+ + decl_tags =btf_dump_find_decl_tags (d, t->type); +if (!decl_tags) { + decl_tags =realloc_decl_tags (NULL ,1 ); +if (!decl_tags) +return -ENOMEM ; + err =hashmap__insert (d->decl_tags, &t->type, decl_tags, +HASHMAP_SET ,NULL ,NULL ); +if (err) +return err; + }else if (decl_tags->cnt == decl_tags->cap) { + new_capacity = decl_tags->cap *2 ;
tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 492
Quoted 10 lines
tools/lib/bpf/btf_dump.c +10
-0+if (new_capacity >0xffff ) +return -ERANGE ; + decl_tags =realloc_decl_tags (decl_tags, new_capacity); +if (!decl_tags) +return -ENOMEM ; + decl_tags->cap = new_capacity; + err =hashmap__update (d->decl_tags, &t->type, decl_tags,NULL ,NULL ); +if (err) +return err; + }
really, let's just use libbpf_reallocarray? I was going to suggest libbpf_ensure_mem, but it allocates at least 16 elements, which seems like an overkill. But also given we don't expect a lot of tags per type, realloc()'ing with + 1 (no * 2 strategy) seems reasonable. Modern allocators either way use differently sized buckets, so when realloc size increment is small, allocator basically will do nothing.
Quoted 19 lines
tools/lib/bpf/btf_dump.c +12
-0+ decl_tags->tag_ids[decl_tags->cnt++] = id; + } + +return 0 ; +} +static int btf_dump_add_emit_queue_id (struct btf_dump *d,__u32 id) {__u32 *new_queue;@@ -899,6 +1022,23 @@ static void btf_dump_emit_bit_padding(const struct btf_dump *d, } } +static void btf_dump_emit_decl_tags (struct btf_dump *d,__u32 id) +{ +struct decl_tag_array *decl_tags =btf_dump_find_decl_tags (d, id); +struct btf_type *decl_tag_t; +const char *decl_tag_text; +__u32 i;
tools/lib/bpf/btf_dump.c static void btf_dump_emit_bit_padding(const struct btf_dump *d, · line 1040
Quoted 10 lines
tools/lib/bpf/btf_dump.c +10
-0+ +if (!decl_tags) +return ; + +for (i =0 ; i < decl_tags->cnt; ++i) { + decl_tag_t =btf_type_by_id (d->btf, decl_tags->tag_ids[i]); + decl_tag_text =btf__name_by_offset (d->btf, decl_tag_t->name_off); +btf_dump_printf (d," __attribute__((btf_decl_tag(\"%s\")))" , decl_tag_text); + } +}
I'm wondering if we should anticipate that some compilers won't know about btf_decl_tag attribute? It seems a bit off for btf_dump to worry about this, but if we don't do something like:
#if __has_attribute(btf_decl_tag)
#define __btf_decl_tag(x) __attribute__((btf_decl_tag(#x))) #else
#define __btf_decl_tag(x)
#endif
.
.
.
struct my_struct {
...
} __btf_decl_tag(awesomeness);it will be hard for users to use resulting vmlinux.h with slightly older Clang?
Quoted 15 lines
tools/lib/bpf/btf_dump.c +2
-1+static void btf_dump_emit_struct_fwd (struct btf_dump *d,__u32 id,const struct btf_type *t) {@@ -964,6 +1104,7 @@ static void btf_dump_emit_struct_def(struct btf_dump *d,btf_dump_printf (d,"%s}" ,pfx (lvl));if (packed)btf_dump_printf (d," __attribute__((packed))" ); +btf_dump_emit_decl_tags (d, id); }static const char *missing_base_types[][2 ] = { --2.34.1
Eduard Zingerman
maintainerRFC 4/12 selftests/bpf: Tests for BTF_DECL_TAG dump in C format
In reply to Eduard ZingermanCovers the following cases:
- __atribute__((btf_decl_tag("..."))) could be applied to structs and unions;
- decl tag applied to an empty struct is printed on a single line;
- decl tags with the same name could be applied to several structs;
- several decl tags could be applied to the same struct;
- attribute packed works fine with decl tags (it is a separate branch in tools/lib/bpf/btf_dump.c:btf_dump_emit_attributes.
--- .../selftests/bpf/prog_tests/btf_dump.c | 1 + .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 +++++++++++++++++++ 2 files changed, 40 insertions(+) create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c
tools/testing/selftests/bpf/prog_tests/btf_dump.c +1 -0
@@ -21,6 +21,7 @@ static struct btf_dump_test_case { {"btf_dump: bitfields", "btf_dump_test_case_bitfields", true}, {"btf_dump: multidim", "btf_dump_test_case_multidim", false}, {"btf_dump: namespacing", "btf_dump_test_case_namespacing", false}, + {"btf_dump: decl_tag", "btf_dump_test_case_decl_tag", true}, };static int btf_dump_all_types (const struct btf *btf,void *ctx)
tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c new file +39 -0
@@ -0,0 +1,39 @@ +// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) + +/* +* BTF-to-C dumper test for __atribute__((btf_decl_tag("..."))). +*/ +/* ----- START-EXPECTED-OUTPUT ----- */ +struct empty_with_tag {} __attribute__((btf_decl_tag ("a" ))); + +struct one_tag { +int x; +} __attribute__((btf_decl_tag ("b" ))); + +struct same_tag { +int x; +} __attribute__((btf_decl_tag ("b" ))); + +struct two_tags { +int x; +} __attribute__((btf_decl_tag("a"))) __attribute__((btf_decl_tag("b"))); + +struct packed { +int x; +short y; +} __attribute__((packed)) __attribute__((btf_decl_tag("another_name"))); + +struct root_struct { +struct empty_with_tag a; +struct one_tag b; +struct same_tag c; +struct two_tags d; +struct packed e; +}; + +/* ------ END-EXPECTED-OUTPUT ------ */ + +int f (struct root_struct *s) +{ +return 0 ; +}
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRFC 5/12 libbpf: Header guards for selected data structures in vmlinux.h
In reply to Eduard ZingermanThe goal of the patch is to allow usage of header files from include/uapi alongside with vmlinux.h. E.g. as follows:
#include <uapi/linux/tcp.h> #include "vmlinux.h"
This goal is achieved by adding #ifndef / #endif guards in vmlinux.h around definitions that originate from the include/uapi headers. The guards emitted match the guards used in the original headers. E.g. as follows:
include/uapi/linux/tcp.h:
#ifndef _UAPI_LINUX_TCP_H
#define _UAPI_LINUX_TCP_H
...
union tcp_word_hdr {
struct tcphdr hdr;
__be32 words[5]; };
...
#endif /* _UAPI_LINUX_TCP_H */
vmlinux.h:
... #ifndef _UAPI_LINUX_TCP_H
union tcp_word_hdr {
struct tcphdr hdr;
__be32 words[5];};
#endif /* _UAPI_LINUX_TCP_H */ ...
The problem of identifying data structures from uapi and selecting proper guard names is delegated to pahole. When configured pahole generates fake BTF_DECL_TAG records with header guards information. The fake tag is distinguished from a real tag by a prefix "header_guard:" in its value. These tags could be present for unions, structures, enums and typedefs, e.g.:
[24139] STRUCT 'tcphdr' size=20 vlen=17
...
[24296] DECL_TAG 'header_guard:_UAPI_LINUX_TCP_H' type_id=24139 ...
This patch adds An option emit_header_guards to struct btf_dump_opts. When this option is present the function btf_dump__dump_type emits header guards for top-level declarations. The header guards are identified by inspecting fake BTF_DECL_TAG records described above.
--- tools/lib/bpf/btf.h | 7 +++- tools/lib/bpf/btf_dump.c | 89 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 94 insertions(+), 2 deletions(-)
tools/lib/bpf/btf.h +6 -1
@@ -235,8 +235,13 @@ struct btf_dump;struct btf_dump_opts {size_t sz; +/* +* if set to true #ifndef X ... #endif brackets would be printed +* for types marked by BTF_DECL_TAG with "header_guard:" prefix +*/ +bool emit_header_guards; }; -#define btf_dump_opts__last_field sz +#define btf_dump_opts__last_field emit_header_guardstypedef void (*btf_dump_printf_fn_t )(void *ctx,const char *fmt,va_list args);
tools/lib/bpf/btf_dump.c +88 -1
195 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [RFC bpf-next 05/12] libbpf: Header guards for selected data structures in vmlinux.h
Quoted 44 lines · Eduard Zingerman
The goal of the patch is to allow usage of header files from
include/uapialongside withvmlinux.h. E.g. as follows:#include <uapi/linux/tcp.h> #include "vmlinux.h"This goal is achieved by adding #ifndef / #endif guards in
vmlinux.haround definitions that originate from theinclude/uapiheaders. The guards emitted match the guards used in the original headers. E.g. as follows:
include/uapi/linux/tcp.h:#ifndef _UAPI_LINUX_TCP_H #define _UAPI_LINUX_TCP_H ... union tcp_word_hdr { struct tcphdr hdr; __be32 words[5]; }; ... #endif /* _UAPI_LINUX_TCP_H */
vmlinux.h:... #ifndef _UAPI_LINUX_TCP_H union tcp_word_hdr { struct tcphdr hdr; __be32 words[5]; }; #endif /* _UAPI_LINUX_TCP_H */ ...The problem of identifying data structures from uapi and selecting proper guard names is delegated to pahole. When configured pahole generates fake
BTF_DECL_TAGrecords with header guards information. The fake tag is distinguished from a real tag by a prefix "header_guard:" in its value. These tags could be present for unions, structures, enums and typedefs, e.g.:
Quoted 8 lines
[24139] STRUCT 'tcphdr' size=20 vlen=17
...
[24296]DECL_TAG'header_guard:_UAPI_LINUX_TCP_H' type_id=24139 ...This patch adds An option
emit_header_guardstostruct btf_dump_opts. When this option is present the functionbtf_dump__dump_typeemits header guards for top-level declarations. The header guards are identified by inspecting fakeBTF_DECL_TAGrecords described above.
This looks like a completely arbitrary convention that libbpf has no business knowing or caring about. I think bpftool should be emitting these guards when generating vmlinux.h. Let's solve this somehow differently.
Quoted 7 lines
Signed-off-by Eduard Zingerman <[email protected]>--- tools/lib/bpf/btf.h | 7 +++- tools/lib/bpf/btf_dump.c | 89 +++++++++++++++++++++++++++++++++++++++- 2 files changed, 94 insertions(+), 2 deletions(-)
[...]
Eduard Zingerman
maintainerRFC 6/12 selftests/bpf: Tests for header guards printing in BTF dump
In reply to Eduard ZingermanVerify that btf_dump__dump_type emits header guard brackets for various types when btf_dump_opts.emit_header_guards is set to true.
--- .../selftests/bpf/prog_tests/btf_dump.c | 10 +- .../progs/btf_dump_test_case_header_guards.c | 94 +++++++++++++++++++ 2 files changed, 101 insertions(+), 3 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c
tools/testing/selftests/bpf/prog_tests/btf_dump.c +7 -3
@@ -13,6 +13,7 @@ static struct btf_dump_test_case {const char *name;const char *file;bool known_ptr_sz; +bool emit_header_guards; } btf_dump_test_cases[] = { {"btf_dump: syntax", "btf_dump_test_case_syntax", true}, {"btf_dump: ordering", "btf_dump_test_case_ordering", false},
@@ -22,15 +23,18 @@ static struct btf_dump_test_case { {"btf_dump: multidim", "btf_dump_test_case_multidim", false}, {"btf_dump: namespacing", "btf_dump_test_case_namespacing", false}, {"btf_dump: decl_tag", "btf_dump_test_case_decl_tag", true}, + {"btf_dump: header guards", "btf_dump_test_case_header_guards", true, true}, }; -static int btf_dump_all_types (const struct btf *btf,void *ctx) +static int btf_dump_all_types (const struct btf *btf,void *ctx,struct btf_dump_test_case *t) {size_t type_cnt =btf__type_cnt (btf); +LIBBPF_OPTS (btf_dump_opts, opts);struct btf_dump *d;int err =0 , id; - d =btf_dump__new (btf, btf_dump_printf, ctx,NULL ); + opts.emit_header_guards = t->emit_header_guards; + d =btf_dump__new (btf, btf_dump_printf, ctx, &opts); err =libbpf_get_error (d);if (err)return err;
@@ -87,7 +91,7 @@ static int test_btf_dump_case(int n, struct btf_dump_test_case *t) goto done; } - err =btf_dump_all_types (btf, f); + err =btf_dump_all_types (btf, f, t);fclose (f);close (fd);if (CHECK (err,"btf_dump" ,"failure during C dumping: %d\n" , err)) {
tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c new file +94 -0
94 lines not on this page · raw message
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRFC 7/12 bpftool: Enable header guards generation
In reply to Eduard ZingermanEnables header guards generation for BTF dumps in C format, e.g. vmlinux.h would be printed as follows:
...
#ifndef _UAPI_LINUX_TCP_H
enum {
TCP_NO_QUEUE = 0,
TCP_RECV_QUEUE = 1,
TCP_SEND_QUEUE = 2,
TCP_QUEUES_NR = 3,
};
#endif /* _UAPI_LINUX_TCP_H */
...--- tools/bpf/bpftool/btf.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
tools/bpf/bpftool/btf.c +3 -1
@@ -466,7 +466,9 @@ static int dump_btf_c(const struct btf *btf,struct btf_dump *d;int err =0 , i; - d =btf_dump__new (btf, btf_dump_printf,NULL ,NULL ); +LIBBPF_OPTS (btf_dump_opts, opts); + opts.emit_header_guards = true; + d =btf_dump__new (btf, btf_dump_printf,NULL , &opts); err =libbpf_get_error (d);if (err)return err;
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRFC 8/12 kbuild: Script to infer header guard values for uapi headers
In reply to Eduard ZingermanThe script infers header guard defines in headers from include/uapi/**/*.h . E.g. header guard for the include/uapi/linux/tcp.h is _UAPI_LINUX_TCP_H:
include/uapi/linux/tcp.h:
#ifndef _UAPI_LINUX_TCP_H
#define _UAPI_LINUX_TCP_H
...
union tcp_word_hdr {
struct tcphdr hdr;
__be32 words[5];
};
...
#endif /* _UAPI_LINUX_TCP_H */The output of the script could be used as an input to pahole's --header_guards_db parameter. This information is necessary to repeat the same header guards in the vmlinux.h generated from BTF.
It is not possible to infer the guard names from header file names alone, the file content has to be analyzed. The following heuristic is used to infer guard for a specific file:
- All pairs #ifndef <candidate> / #define <candidate> are collected;
- If a unique candidate matching regex ${headername}.*_H(EADER)? it is selected;
- If a unique candidate matching regex _H(EADER)?_ it is selected;
- If a unique candidate matching regex _H(EADER)?$ it is selected;
There is also a small list of headers that can't be caught by the rules above, 15 in total. These headers and corresponding guard values are listed in the %OVERRIDES hash table.
--- scripts/infer_header_guards.pl | 191 +++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100755 scripts/infer_header_guards.pl
scripts/infer_header_guards.pl new file +191 -0
@@ -0,0 +1,191 @@ +#!/usr/bin/env perl +# SPDX-License-Identifier: GPL-2.0 + +# This script scans the passed directory for header files (files ending with ".h"). +# For each header file it tries to infer the name of a C pre-processor +# variable used as a double include guard (dubbed as "header guard"). +# For example: +# +# #ifdef __MY_HEADER__ // <-- "header guard" +# ... +# #endif +# +# The inferred guards are printed to stdout in the following format: +# +# <header-file> <header-guard> +# +# This is an expected format for pahole --header_guards_db parameter. +# Intended usage is to infer header guards for Linux UAPI headers. +# The collected information is further used in BTF embedded into kernel. +# +# The following inference logic is used for each file: +# - find all pairs `#ifndef <name> #define <name>` +# - if there is a unique <name> that matches a pattern - use this +# <name> as a header guard, (see subroutine `select_guard` for the +# list of the patterns); +# - files containing only #include directives are safe to ignore. +# +# There are a few UAPI header files that don't fit in such logic, +# header guards for these files are hard-coded in %OVERRIDES hash. +# +# The script reports inference only when --report-failures flag is +# passed. This flag is intended for BPF tests. +# +# See subroutine `help` for usage info. + +use strict; +use warnings; +use File::Basename; +use File::Find; +use Getopt::Long; + +sub help { + my $message = << "EOM"; +Usage: + $0 [--report-failures] directory-or-file... + $0 --help + +For a specific file or for each .h file in a directory infer the name +of a C pre-processor variable used as a double include guard. + +Options: + --report-failures Report inference errors to stderr, + exit with non-zero code if guards were not inferred + for some files. + --help Print this message and exit. +EOM + print $message; +} + +my %OVERRIDES = ( + # Header guards that don't follow common naming rules + "include/uapi/linux/cciss_ioctl.h" => "_UAPICCISS_IOCTLH", + "include/uapi/linux/hpet.h" => "_UAPI__HPET__", + "include/uapi/linux/if_ppp.h" => "_PPP_IOCTL_H", + "include/uapi/linux/netfilter/xt_NFLOG.h" => "_XT_NFLOG_TARGET", + "include/uapi/linux/netfilter_ipv6/ip6t_NPT.h" => "__NETFILTER_IP6T_NPT", + "include/uapi/linux/quota.h" => "_UAPI_LINUX_QUOTA_", + "include/uapi/linux/v4l2-common.h" => "__V4L2_COMMON__", + # Headers that should be ignored + "arch/x86/include/uapi/asm/hw_breakpoint.h" => undef, + "arch/x86/include/uapi/asm/posix_types.h" => undef, + "arch/x86/include/uapi/asm/setup.h" => undef, + "include/generated/uapi/linux/version.h" => undef, + "include/uapi/asm-generic/bitsperlong.h" => undef, + "include/uapi/asm-generic/kvm_para.h" => undef, + "include/uapi/asm-generic/unistd.h" => undef, + "include/uapi/linux/irqnr.h" => undef, + "include/uapi/linux/zorro_ids.h" => undef, + ); + +sub get_basename { + my ($filename) = @_; + my $basename = fileparse($filename, qr/\.[^.]*/); + return $basename; +} + +sub find_bracket_candidates { + my ($filename) = @_; + my @candidates = (); + my $guard_candidate = undef; + my $safe_to_ignore = 1; + + open my $file, $filename or die "Can't open file $filename: $!"; + while (my $line = <$file>) { + if (not($line =~ "^#include")) { + $safe_to_ignore = 0; + } + if ($line =~ "^#ifndef[ \t]+([a-zA-Z0-9_]+)") { + $guard_candidate = $1; + } elsif ($guard_candidate && $line =~ "^#define[ \t]+${guard_candidate}") { + push(@candidates, $guard_candidate); + $guard_candidate = undef; + } + } + close $file; + + return ($safe_to_ignore, @candidates); +} + +sub select_guard { + my ($filename, @candidates) = @_; + my $basename = get_basename($filename); + my @regexes = ("$basename.*_H(EADER)?", + "_H(EADER)?_", + "_H(EADER)?\$"); + foreach my $re (@regexes) { + my @filtered = grep(/$re/i, @candidates); + if (scalar(@filtered) == 1) { + return $filtered[0]; + } + } + + return undef; +} + +sub collect_headers { + my ($dir) = @_; + my @headers = (); + + find(sub { /\.h$/ && push(@headers, $File::Find::name); }, $dir); + + return @headers; +} + +my $report_failures = 0; +my $options_parsed = GetOptions( + "report-failures" => \$report_failures, + "help" => sub { help(); exit 0; }, + ); + +if (!$options_parsed || scalar @ARGV == 0) { + help(); + exit 1; +} + +my @headers; + +foreach my $dir_or_file (@ARGV) { + if (-f $dir_or_file) { + push(@headers, $dir_or_file); + } elsif (-d $dir_or_file) { + push(@headers, collect_headers($dir_or_file)); + } else { + print("'$dir_or_file' is not a file or directory.\n"); + help(); + exit 1; + } +} + +my $rc = 0; + +foreach my $header (@headers) { + my $basename = get_basename($header); + my $guard; + + if (exists $OVERRIDES{$header}) { + $guard = $OVERRIDES{$basename}; + } else { + my ($safe_to_ignore, @candidates) = find_bracket_candidates($header); + $guard = select_guard($header, @candidates); + if ((not $guard) && (not $safe_to_ignore) && $report_failures) { + print STDERR "Can't select guard for $header, candidates:\n"; + print STDERR " "; + if (scalar(@candidates)) { + print STDERR join(", ", @candidates); + } else { + print STDERR "<no candidates>" + } + print STDERR "\n"; + $rc = 1; + } + } + if ($guard) { + # Remove the _UAPI prefix/suffix the same way + # scripts/headers_install.sh does it. + $guard =~ s/_UAPI//; + print("$header $guard\n"); + } +} + +exit $rc;
Signature · 1 line
2.34.1
Re: [RFC bpf-next 08/12] kbuild: Script to infer header guard values for uapi headers
Quoted 30 lines · Eduard Zingerman
The script infers header guard defines in headers from
include/uapi/**/*.h. E.g. header guard for theinclude/uapi/linux/tcp.his_UAPI_LINUX_TCP_H:include/uapi/linux/tcp.h: #ifndef _UAPI_LINUX_TCP_H #define _UAPI_LINUX_TCP_H ... union tcp_word_hdr { struct tcphdr hdr; __be32 words[5]; }; ... #endif /* _UAPI_LINUX_TCP_H */The output of the script could be used as an input to pahole's
--header_guards_dbparameter. This information is necessary to repeat the same header guards in thevmlinux.hgenerated from BTF.It is not possible to infer the guard names from header file names alone, the file content has to be analyzed. The following heuristic is used to infer guard for a specific file:
- All pairs#ifndef <candidate>/#define <candidate>are collected;
- If a unique candidate matching regex${headername}.*_H(EADER)?it is selected;
- If a unique candidate matching regex_H(EADER)?_it is selected;
- If a unique candidate matching regex_H(EADER)?$it is selected;
Quoted 4 lines
There is also a small list of headers that can't be caught by the rules above, 15 in total. These headers and corresponding guard values are listed in the
%OVERRIDEShash table.
Instead of expecting naming pattern, why can't we just expect
/* some comments here */
#ifndef XXX
#define XXX
....
#endif
and extract XXX from such a pattern?
The harder part is skipping comments (but awk might help do this easier), or we can just ignore all the lines before the first #ifndef.
WDYT?
Quoted 6 lines
Signed-off-by Eduard Zingerman <[email protected]>--- scripts/infer_header_guards.pl | 191 +++++++++++++++++++++++++++++++++ 1 file changed, 191 insertions(+) create mode 100755 scripts/infer_header_guards.pl
[...]
Eduard Zingerman
maintainerRFC 9/12 kbuild: Header guards for types from include/uapi/*.h in kernel BTF
In reply to Eduard ZingermanUse pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the scripts/infer_header_guards.pl.
The encoded header guard information could be used to restore the original guards in the vmlinux.h, e.g.:
include/uapi/linux/tcp.h:
#ifndef _UAPI_LINUX_TCP_H
#define _UAPI_LINUX_TCP_H
...
union tcp_word_hdr {
struct tcphdr hdr;
__be32 words[5];
};
...
#endif /* _UAPI_LINUX_TCP_H */
vmlinux.h:
...
#ifndef _UAPI_LINUX_TCP_H
union tcp_word_hdr {
struct tcphdr hdr;
__be32 words[5];
};
#endif /* _UAPI_LINUX_TCP_H */
...--- scripts/link-vmlinux.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)
scripts/link-vmlinux.sh +12 -1
@@ -110,6 +110,7 @@ vmlinux_link() gen_btf() { local pahole_ver + local extra_flags if ! [ -x "$(command -v ${PAHOLE})" ]; then echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"
@@ -122,10 +123,20 @@ gen_btf() return 1 fi + if [ "${pahole_ver}" -ge "124" ]; then + scripts/infer_header_guards.pl \ + include/uapi \ + include/generated/uapi \ + arch/${SRCARCH}/include/uapi \ + arch/${SRCARCH}/include/generated/uapi \ + > .btf.uapi_header_guards || return 1; + extra_flags="--header_guards_db .btf.uapi_header_guards" + fi + vmlinux_link ${1} info "BTF" ${2} - LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} + LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1} # Create ${2} which contains just .BTF section but no symbols. Add # SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
Signature · 1 line
2.34.1
Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF
Quoted 50 lines · Eduard Zingerman
Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the
scripts/infer_header_guards.pl.The encoded header guard information could be used to restore the original guards in the
vmlinux.h, e.g.:include/uapi/linux/tcp.h: #ifndef _UAPI_LINUX_TCP_H #define _UAPI_LINUX_TCP_H ... union tcp_word_hdr { struct tcphdr hdr; __be32 words[5]; }; ... #endif /* _UAPI_LINUX_TCP_H */ vmlinux.h: ... #ifndef _UAPI_LINUX_TCP_H union tcp_word_hdr { struct tcphdr hdr; __be32 words[5]; }; #endif /* _UAPI_LINUX_TCP_H */ ...Signed-off-by Eduard Zingerman <[email protected]>--- scripts/link-vmlinux.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)scripts/link-vmlinux.sh +1
-0@@ -110,6 +110,7 @@ vmlinux_link() gen_btf() { local pahole_ver + local extra_flags if ! [ -x "$(command -v ${PAHOLE})" ]; then echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"
scripts/link-vmlinux.sh gen_btf() · line 127
Quoted 5 lines
scripts/link-vmlinux.sh +2
-0@@ -122,10 +123,20 @@ gen_btf() return 1 fi + if [ "${pahole_ver}" -ge "124" ]; then + scripts/infer_header_guards.pl \
We should have full path like
${srctree}/scripts/infer_header_guards.plso it can work if build directory is different from source directory.
Quoted 16 lines
scripts/link-vmlinux.sh +9
-1+ include/uapi \ + include/generated/uapi \ + arch/${SRCARCH}/include/uapi \ + arch/${SRCARCH}/include/generated/uapi \ + > .btf.uapi_header_guards || return 1; + extra_flags="--header_guards_db .btf.uapi_header_guards" + fi + vmlinux_link ${1} info "BTF" ${2} - LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} + LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1} # Create ${2} which contains just .BTF section but no symbols. Add # SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF
Quoted 52 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman
Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the
scripts/infer_header_guards.pl.The encoded header guard information could be used to restore the original guards in the
vmlinux.h, e.g.:
include/uapi/linux/tcp.h:#ifndef
_UAPI_LINUX_TCP_H
#define_UAPI_LINUX_TCP_H
...
uniontcp_word_hdr{
struct tcphdr hdr;
__be32 words[5];
};
...
#endif /*_UAPI_LINUX_TCP_H*/
vmlinux.h:...
#ifndef_UAPI_LINUX_TCP_Hunion
tcp_word_hdr{
struct tcphdr hdr;
__be32words[5];
};#endif /*
_UAPI_LINUX_TCP_H*/
...Signed-off-by Eduard Zingerman <[email protected]>---
scripts/link-vmlinux.sh| 13 ++++++++++++-1 file changed, 12 insertions(+), 1 deletion(-)scripts/link-vmlinux.sh +0
-0@@ -110,6 +110,7 @@ vmlinux_link() gen_btf() {local
pahole_ver
+ localextra_flags
if ! [ -x "$(command -v ${PAHOLE})" ]; then
echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"scripts/link-vmlinux.sh +0
-0@@ -122,10 +123,20 @@ gen_btf()
Quoted 8 lines
Quoted 4 lines
return 1
fi
+ if [ "${pahole_ver}" -ge "124" ]; then
+scripts/infer_header_guards.pl\We should have full path like
${srctree}/scripts/infer_header_guards.pl
so it can work if build directory is different from source directory.
handling arguments for infer_header_guards.pl should also take care of full file path.
+ /home/yhs/work/bpf-next/scripts/infer_header_guards.pl include/uapi include/generated/uapi arch/x86/include/uapi arch/x86/include/generated/uapi
+ return 1
Quoted 17 lines
Quoted 17 lines
+
include/uapi\
+include/generated/uapi\
+arch/${SRCARCH}/include/uapi\
+arch/${SRCARCH}/include/generated/uapi\
+ > .btf.uapi_header_guards || return 1;
+ extra_flags="--header_guards_db .btf.uapi_header_guards" + fi
+
vmlinux_link${1}
info "BTF" ${2}
- LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} + LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1}
# Create ${2} which contains just .BTF section but no symbols. Add #SHF_ALLOCbecause .BTF will be part of the vmlinux image. --strip-all
Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF
Quoted 64 lines · Yonghong Song
Quoted 52 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman
Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the
scripts/infer_header_guards.pl.The encoded header guard information could be used to restore the original guards in the
vmlinux.h, e.g.:
include/uapi/linux/tcp.h:#ifndef
_UAPI_LINUX_TCP_H
#define_UAPI_LINUX_TCP_H
...
uniontcp_word_hdr{
struct tcphdr hdr;
__be32 words[5];
};
...
#endif /*_UAPI_LINUX_TCP_H*/
vmlinux.h:...
#ifndef_UAPI_LINUX_TCP_Hunion
tcp_word_hdr{
struct tcphdr hdr;
__be32words[5];
};#endif /*
_UAPI_LINUX_TCP_H*/
...Signed-off-by Eduard Zingerman <[email protected]>---
scripts/link-vmlinux.sh| 13 ++++++++++++-1 file changed, 12 insertions(+), 1 deletion(-)scripts/link-vmlinux.sh +0
-0@@ -110,6 +110,7 @@ vmlinux_link() gen_btf() {local
pahole_ver
+ localextra_flags
if ! [ -x "$(command -v ${PAHOLE})" ]; then
echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"scripts/link-vmlinux.sh +0
-0@@ -122,10 +123,20 @@ gen_btf()Quoted 8 lines
Quoted 4 lines
return 1
fi
+ if [ "${pahole_ver}" -ge "124" ]; then
+scripts/infer_header_guards.pl\We should have full path like
${srctree}/scripts/infer_header_guards.pl
so it can work if build directory is different from source directory.
Quoted 6 lines
handling arguments for infer_header_guards.pl should also take care of full file path.
+
/home/yhs/work/bpf-next/scripts/infer_header_guards.plinclude/uapiinclude/generated/uapiarch/x86/include/uapiarch/x86/include/generated/uapi
+ return 1
Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.
Quoted 17 lines
Quoted 17 lines
Quoted 17 lines
+
include/uapi\
+include/generated/uapi\
+arch/${SRCARCH}/include/uapi\
+arch/${SRCARCH}/include/generated/uapi\
+ > .btf.uapi_header_guards || return 1;
+ extra_flags="--header_guards_db .btf.uapi_header_guards" + fi
+
vmlinux_link${1}
info "BTF" ${2}
- LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} + LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1}
# Create ${2} which contains just .BTF section but no symbols. Add #SHF_ALLOCbecause .BTF will be part of the vmlinux image. --strip-all
Eduard Zingerman
maintainerRe: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF
Quoted 69 lines · Yonghong Song
Quoted 67 lines · Yonghong Song
Quoted 52 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman
Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the
scripts/infer_header_guards.pl.The encoded header guard information could be used to restore the original guards in the
vmlinux.h, e.g.:
include/uapi/linux/tcp.h:#ifndef
_UAPI_LINUX_TCP_H
#define_UAPI_LINUX_TCP_H
...
uniontcp_word_hdr{
struct tcphdr hdr;
__be32 words[5];
};
...
#endif /*_UAPI_LINUX_TCP_H*/
vmlinux.h:...
#ifndef_UAPI_LINUX_TCP_Hunion
tcp_word_hdr{
struct tcphdr hdr;
__be32words[5];
};#endif /*
_UAPI_LINUX_TCP_H*/
...Signed-off-by Eduard Zingerman <[email protected]>---
scripts/link-vmlinux.sh| 13 ++++++++++++-1 file changed, 12 insertions(+), 1 deletion(-)scripts/link-vmlinux.sh +0
-0@@ -110,6 +110,7 @@ vmlinux_link() gen_btf() {local
pahole_ver
+ localextra_flags
if ! [ -x "$(command -v ${PAHOLE})" ]; then
echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"scripts/link-vmlinux.sh +0
-0@@ -122,10 +123,20 @@ gen_btf()Quoted 8 lines
Quoted 4 lines
return 1
fi
+ if [ "${pahole_ver}" -ge "124" ]; then
+scripts/infer_header_guards.pl\We should have full path like
${srctree}/scripts/infer_header_guards.pl
so it can work if build directory is different from source directory.handling arguments for infer_header_guards.pl should also take care of full file path.
Quoted 7 lines
Quoted 4 lines
+
/home/yhs/work/bpf-next/scripts/infer_header_guards.plinclude/uapiinclude/generated/uapiarch/x86/include/uapiarch/x86/include/generated/uapi
+ return 1Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.
Hi Yonghong,
Could you please copy-paste some of the error reports? I just re-run the selftests locally and have test_maps, test_verifier, test_progs and test_progs-no_alu32 passing.
Thanks,
Eduard
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
+
include/uapi\
+include/generated/uapi\
+arch/${SRCARCH}/include/uapi\
+arch/${SRCARCH}/include/generated/uapi\
+ > .btf.uapi_header_guards || return 1;
+ extra_flags="--header_guards_db .btf.uapi_header_guards" + fi
+
vmlinux_link${1}
info "BTF" ${2}
- LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} + LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1}
# Create ${2} which contains just .BTF section but no symbols. Add #SHF_ALLOCbecause .BTF will be part of the vmlinux image. --strip-all
Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF
Yonghong,
build will be failing without merged pahole changes.
Quoted 107 lines · Eduard Zingerman
Quoted 69 lines · Yonghong Song
Quoted 67 lines · Yonghong Song
Quoted 58 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman
Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the
scripts/infer_header_guards.pl.The encoded header guard information could be used to restore the original guards in the
vmlinux.h, e.g.:include/uapi/linux/tcp.h: #ifndef _UAPI_LINUX_TCP_H #define _UAPI_LINUX_TCP_H ... union tcp_word_hdr { struct tcphdr hdr; __be32 words[5]; }; ... #endif /* _UAPI_LINUX_TCP_H */ vmlinux.h: ... #ifndef _UAPI_LINUX_TCP_H union tcp_word_hdr { struct tcphdr hdr; __be32 words[5]; }; #endif /* _UAPI_LINUX_TCP_H */ ...Signed-off-by Eduard Zingerman <[email protected]>--- scripts/link-vmlinux.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)scripts/link-vmlinux.sh +1
-0@@ -110,6 +110,7 @@ vmlinux_link() gen_btf() { local pahole_ver + local extra_flags if ! [ -x "$(command -v ${PAHOLE})" ]; then echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"scripts/link-vmlinux.sh gen_btf() · line 126
Quoted 4 lines
scripts/link-vmlinux.sh +2
-0@@ -122,10 +123,20 @@ gen_btf() return 1 fi + if [ "${pahole_ver}" -ge "124" ]; then + scripts/infer_header_guards.pl \We should have full path like
Quoted 2 lines
${srctree}/scripts/infer_header_guards.plso it can work if build directory is different from source directory.
handling arguments for infer_header_guards.pl should also take care of full file path.
Quoted 7 lines
Quoted 4 lines
+
/home/yhs/work/bpf-next/scripts/infer_header_guards.plinclude/uapiinclude/generated/uapiarch/x86/include/uapiarch/x86/include/generated/uapi
+ return 1Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.
Hi Yonghong,
Could you please copy-paste some of the error reports? I just re-run the selftests locally and have
test_maps,test_verifier,test_progsand test_progs-no_alu32 passing.Thanks,
EduardQuoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
scripts/link-vmlinux.sh +9
-2+ include/uapi \ + include/generated/uapi \ + arch/${SRCARCH}/include/uapi \ + arch/${SRCARCH}/include/generated/uapi \ + > .btf.uapi_header_guards || return 1; + extra_flags="--header_guards_db .btf.uapi_header_guards" + fi + vmlinux_link ${1} info "BTF" ${2} - LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} + LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1} # Create ${2} which contains just .BTF section but no symbols. Add # SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF
Quoted 3 lines · Mykola Lysenko
Yonghong,
build will be failing without merged pahole changes.
Yes, I do have pahole changes. My failure is due to building in-tree vmlinux (KBUILD_OUTPUT= ) since current patch set doesn't support out-of-tree build, and out-of-tree selftest (KBUILD_OUTPUT=<path>). Using the same in-tree build fixed the problem.
Quoted 109 lines
Quoted 108 lines · Eduard Zingerman
Quoted 69 lines · Yonghong Song
Quoted 67 lines · Yonghong Song
Quoted 58 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman
Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the
scripts/infer_header_guards.pl.The encoded header guard information could be used to restore the original guards in the
vmlinux.h, e.g.:include/uapi/linux/tcp.h: #ifndef _UAPI_LINUX_TCP_H #define _UAPI_LINUX_TCP_H ... union tcp_word_hdr { struct tcphdr hdr; __be32 words[5]; }; ... #endif /* _UAPI_LINUX_TCP_H */ vmlinux.h: ... #ifndef _UAPI_LINUX_TCP_H union tcp_word_hdr { struct tcphdr hdr; __be32 words[5]; }; #endif /* _UAPI_LINUX_TCP_H */ ...Signed-off-by Eduard Zingerman <[email protected]>--- scripts/link-vmlinux.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)scripts/link-vmlinux.sh +1
-0@@ -110,6 +110,7 @@ vmlinux_link() gen_btf() { local pahole_ver + local extra_flags if ! [ -x "$(command -v ${PAHOLE})" ]; then echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"scripts/link-vmlinux.sh gen_btf() · line 126
Quoted 4 lines
scripts/link-vmlinux.sh +2
-0@@ -122,10 +123,20 @@ gen_btf() return 1 fi + if [ "${pahole_ver}" -ge "124" ]; then + scripts/infer_header_guards.pl \We should have full path like
Quoted 2 lines
${srctree}/scripts/infer_header_guards.plso it can work if build directory is different from source directory.
handling arguments for infer_header_guards.pl should also take care of full file path.
Quoted 7 lines
Quoted 4 lines
+
/home/yhs/work/bpf-next/scripts/infer_header_guards.plinclude/uapiinclude/generated/uapiarch/x86/include/uapiarch/x86/include/generated/uapi
+ return 1Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.
Hi Yonghong,
Could you please copy-paste some of the error reports? I just re-run the selftests locally and have
test_maps,test_verifier,test_progsand test_progs-no_alu32 passing.Thanks,
EduardQuoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
scripts/link-vmlinux.sh +9
-2+ include/uapi \ + include/generated/uapi \ + arch/${SRCARCH}/include/uapi \ + arch/${SRCARCH}/include/generated/uapi \ + > .btf.uapi_header_guards || return 1; + extra_flags="--header_guards_db .btf.uapi_header_guards" + fi + vmlinux_link ${1} info "BTF" ${2} - LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} + LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1} # Create ${2} which contains just .BTF section but no symbols. Add # SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF
In reply to Eduard ZingermanQuoted 75 lines · Eduard Zingerman
Quoted 74 lines · Yonghong Song
Quoted 71 lines · Yonghong Song
Quoted 52 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman
Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the
scripts/infer_header_guards.pl.The encoded header guard information could be used to restore the original guards in the
vmlinux.h, e.g.:
include/uapi/linux/tcp.h:#ifndef
_UAPI_LINUX_TCP_H
#define_UAPI_LINUX_TCP_H
...
uniontcp_word_hdr{
struct tcphdr hdr;
__be32 words[5];
};
...
#endif /*_UAPI_LINUX_TCP_H*/
vmlinux.h:...
#ifndef_UAPI_LINUX_TCP_Hunion
tcp_word_hdr{
struct tcphdr hdr;
__be32words[5];
};#endif /*
_UAPI_LINUX_TCP_H*/
...Signed-off-by Eduard Zingerman <[email protected]>--- scripts/link-vmlinux.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-)scripts/link-vmlinux.sh +1
-0@@ -110,6 +110,7 @@ vmlinux_link() gen_btf() { local pahole_ver + local extra_flags if ! [ -x "$(command -v ${PAHOLE})" ]; then echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"@@ -122,10 +123,20 @@ gen_btf()Quoted 8 lines
scripts/link-vmlinux.sh gen_btf() · line 127
Quoted 4 lines
scripts/link-vmlinux.sh +2
-0return 1 fi + if [ "${pahole_ver}" -ge "124" ]; then + scripts/infer_header_guards.pl \We should have full path like
${srctree}/scripts/infer_header_guards.pl
so it can work if build directory is different from source directory.handling arguments for infer_header_guards.pl should also take care of full file path.
+
/home/yhs/work/bpf-next/scripts/infer_header_guards.plinclude/uapiinclude/generated/uapiarch/x86/include/uapiarch/x86/include/generated/uapi
+ return 1
Quoted 8 lines
Quoted 2 lines
Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.
Hi Yonghong,
Could you please copy-paste some of the error reports? I just re-run the selftests locally and have
test_maps,test_verifier,test_progsand test_progs-no_alu32 passing.
Sorry about the noise. It is my fault. My default build is out of source tree with KBUILD_OUTPUT=<path>. Since the current patch set won't work with it, so I build a in-tree one for vmlinux but forgot to adjust selftest build which still has KBUILD_OUTPUT=<path> and it caused some selftest failures. Consistently doing in-tree build for vmlinux and selftest results in the same Success/Failure rate with and without this patch set.
Quoted 23 lines
Thanks,
EduardQuoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
scripts/link-vmlinux.sh +9
-2+ include/uapi \ + include/generated/uapi \ + arch/${SRCARCH}/include/uapi \ + arch/${SRCARCH}/include/generated/uapi \ + > .btf.uapi_header_guards || return 1; + extra_flags="--header_guards_db .btf.uapi_header_guards" + fi + vmlinux_link ${1} info "BTF" ${2} - LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} + LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1} # Create ${2} which contains just .BTF section but no symbols. Add # SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
Eduard Zingerman
maintainerRFC 10/12 selftests/bpf: Script to verify uapi headers usage with vmlinux.h
In reply to Eduard ZingermanA script to test header guards in vmlinux.h by compiling a simple C snippet for a set of selected UAPI headers. The snippet being compiled looks as follows:
#include <some_uapi_header.h> #include "vmlinux.h"
__attribute__((section("tc"), used))
int syncookie_tc(struct __sk_buff *skb) { return 0; }
If header guards are placed correctly in vmlinux.h the snippet should compile w/o errors.
The list of known good headers is supposed to be located in tools/testing/selftests/bpf/good_uapi_headers.txt added as a separate commit.
--- .../selftests/bpf/test_uapi_headers.py | 197 ++++++++++++++++++ 1 file changed, 197 insertions(+) create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
tools/testing/selftests/bpf/test_uapi_headers.py new file +197 -0
@@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) + +# A script to test header guards in vmlinux.h by compiling a simple C +# snippet for a set of selected UAPI headers. The snippet being +# compiled looks as follows: +# +# #include <some_uapi_header.h> +# #include "vmlinux.h" +# +# __attribute__((section("tc"), used)) +# int syncookie_tc(struct __sk_buff *skb) { return 0; } +# +# If header guards are placed correctly in vmlinux.h the snippet +# should compile w/o errors. +# +# The script could be used in two modes: +# - interactive BPF testing and CI; +# - debug mode. +# +# * Interactive BPF testing and CI +# +# Run script as follows: +# +# ./test_uapi_headers.py +# +# In this mode the following actions are performed: +# - kernel headers are installed to a temporary directory; +# - a list of known good uapi headers is read from ./good_uapi_headers.txt; +# - the snippet above is compiled by clang using BPF target for each header; +# - if shell is interactive the progress / ETA are reported during execution; +# - pass / fail statistics is reported in the end; +# - headers temporary directory is deleted; +# - script exit code is 0 if snippet could be compiled for all headers. +# +# The vmlinux.h processing time is significant (~700ms using Intel i7-4710HQ), +# thus the headers are processed in parallel. +# +# * Debug mode +# +# The following parameters are available for debugging: +# +# test_uapi_headers.py \ +# [-h] [--kheaders KHEADERS] [--vmlinuxh VMLINUXH] [--test TEST] +# +# options: +# -h, --help show this help message and exit +# --kheaders KHEADERS path to exported kernel headers +# --vmlinuxh VMLINUXH path to vmlinux.h +# --test TEST name of the header -or- +# file with header names -or- +# special value '*' +# +# When --kheaders is specified the temporary directory is not created +# and KHEADERS is used instead. It is assumed that headers are already +# installed to KHEADERS. +# +# When TEST names a header (e.g. 'linux/tcp.h') it is the to test. +# When TEST names a file this file should contain a list of +# headers to test one per line. +# When TEST is '*' all exported headers are tested. +# +# The simplest way to debug an issue with a single header is: +# +# ./test_uapi_headers.py --test linux/tcp.h + +import subprocess +import concurrent.futures +import pathlib +import time +import os +import sys +import argparse +import tempfile +import shutil +import atexit +from dataclasses import dataclass + +@dataclass +class Result: + header: pathlib.Path + returncode: int + stderr: str + +def run_one(header, kheaders, vmlinuxh): + code=f''' +#include <{header}> +#include "{vmlinuxh}" + +__attribute__((section("tc"), used)) +int syncookie_tc(struct __sk_buff *skb) +{{ + return 0; +}} + ''' + command = f''' +{os.getenv('CLANG', 'clang')} \ + -g -Werror -mlittle-endian \ + -D__x86_64__ \ + -Xclang -fwchar-type=short \ + -Xclang -fno-signed-wchar \ + -I{kheaders}/include/ \ + -Wno-compare-distinct-pointer-types \ + -mcpu=v3 \ + -O2 \ + -target bpf \ + -x c \ + -o /dev/null \ + -fsyntax-only \ + - +''' + proc = subprocess.run(command, input=code, capture_output=True, + shell=True, encoding='utf8') + return Result(header=header, + returncode=proc.returncode, + stderr=proc.stderr) + +def run_all(headers, kheaders, vmlinuxh): + start_time = time.time() + ok = 0 + fail = 0 + failures = [] + remain = len(headers) + print_progress = sys.stdout.isatty() + print(f'Processing {remain} headers.') + with concurrent.futures.ThreadPoolExecutor(max_workers=os.cpu_count()) as executor: + for result in executor.map(lambda header: run_one(header, kheaders, vmlinuxh), + headers): + if result.returncode == 0: + print(f"{result.header:<60} ok") + ok += 1 + else: + print(f"{result.header:<60} fail") + fail += 1 + failures.append(result) + remain -= 1 + if print_progress: + elapsed = time.time() - start_time + processed = ok + fail + time_per_header = elapsed / processed + eta = int(remain * time_per_header) + # keep this shorter than header ok/fail line + line = f"Ok {ok: >4} Fail {fail: >4} Remain {remain: >4} ETA {eta: >4}s" + print(line, end="\r") + if print_progress: + print('') + elapsed = int(time.time() - start_time) + if fail == 0: + print(f"Done in {elapsed}s, all {len(headers)} ok.") + else: + print('----- Failure details -----') + for result in failures: + print(f'{result.header}: rc = {result.returncode}') + for line in result.stderr.split('\n'): + print(f"{result.header}: {line}") + print(f"Done in {elapsed}s, {fail} out of {len(headers)} failed.") + return fail == 0 + +def main(argv): + bpf_test_dir = pathlib.Path(__file__).resolve().parent + default_vmlinuxh = bpf_test_dir / './tools/include/vmlinux.h' + parser = argparse.ArgumentParser() + parser.add_argument("--kheaders", type=str, help='path to exported kernel headers') + parser.add_argument("--vmlinuxh", type=str, default=default_vmlinuxh, + help='path to vmlinux.h') + parser.add_argument("--test", type=str, + default='./good_uapi_headers.txt', + help="name of the header | file with header names | special value '*'") + args = parser.parse_args(argv) + + if args.kheaders is None: + kheaders = tempfile.mkdtemp(prefix='kheaders') + atexit.register(lambda: shutil.rmtree(kheaders)) + kernel_dir = bpf_test_dir / '../../../../' + # Capture both stdout and stderr as stdout to simplify CI logging + subprocess.run(f'make -C {kernel_dir} INSTALL_HDR_PATH={kheaders} headers_install', + stdout=sys.stdout, stderr=sys.stdout, + check=True, shell=True) + else: + kheaders = args.kheaders + + if os.path.exists(args.test): + with open(args.test, 'r') as list_file: + headers = [line.strip() for line in list_file] + elif args.test == '*': + headers = [p.relative_to(f'{kheaders}/include').as_posix() + for p in pathlib.Path(kheaders).rglob("*.h")] + else: + headers = [args.test] + + if run_all(headers, kheaders, args.vmlinuxh): + sys.exit(0) + else: + sys.exit(1) + +if __name__ == '__main__': + main(sys.argv[1:])
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRFC 11/12 selftests/bpf: Known good uapi headers for test_uapi_headers.py
In reply to Eduard ZingermanThe list of uapi headers that could be included alongside vmlinux.h. The headers are peeked from the following locations:
- <headers-export-dir>/linux/*.h
- <headers-export-dir>/linux/**/*.h
--- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ 1 file changed, 677 insertions(+) create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt
tools/testing/selftests/bpf/good_uapi_headers.txt new file +677 -0
@@ -0,0 +1,677 @@ +linux/a.out.h +linux/acct.h +linux/acrn.h +linux/adb.h +linux/adfs_fs.h +linux/affs_hardblocks.h +linux/aio_abi.h +linux/amt.h +linux/android/binder.h +linux/android/binderfs.h +linux/apm_bios.h +linux/arcfb.h +linux/arm_sdei.h +linux/aspeed-lpc-ctrl.h +linux/aspeed-p2a-ctrl.h +linux/atalk.h +linux/atm.h +linux/atm_eni.h +linux/atm_he.h +linux/atm_idt77105.h +linux/atm_nicstar.h +linux/atm_tcp.h +linux/atm_zatm.h +linux/atmapi.h +linux/atmarp.h +linux/atmclip.h +linux/atmdev.h +linux/atmioc.h +linux/atmlec.h +linux/atmmpc.h +linux/atmppp.h +linux/atmsap.h +linux/atmsvc.h +linux/audit.h +linux/auxvec.h +linux/ax25.h +linux/batadv_packet.h +linux/batman_adv.h +linux/baycom.h +linux/bcm933xx_hcs.h +linux/bfs_fs.h +linux/binfmts.h +linux/blkpg.h +linux/blktrace_api.h +linux/blkzoned.h +linux/bpf.h +linux/bpf_common.h +linux/bpqether.h +linux/bsg.h +linux/bt-bmc.h +linux/btf.h +linux/btrfs.h +linux/btrfs_tree.h +linux/byteorder/big_endian.h +linux/byteorder/little_endian.h +linux/cachefiles.h +linux/caif/caif_socket.h +linux/caif/if_caif.h +linux/can.h +linux/can/bcm.h +linux/can/error.h +linux/can/gw.h +linux/can/isotp.h +linux/can/j1939.h +linux/can/netlink.h +linux/can/raw.h +linux/can/vxcan.h +linux/capability.h +linux/capi.h +linux/cciss_defs.h +linux/cciss_ioctl.h +linux/ccs.h +linux/cdrom.h +linux/cfm_bridge.h +linux/cgroupstats.h +linux/chio.h +linux/cifs/cifs_mount.h +linux/cifs/cifs_netlink.h +linux/close_range.h +linux/cm4000_cs.h +linux/cn_proc.h +linux/coff.h +linux/comedi.h +linux/connector.h +linux/const.h +linux/coresight-stm.h +linux/counter.h +linux/cramfs_fs.h +linux/cryptouser.h +linux/cuda.h +linux/cxl_mem.h +linux/cycx_cfm.h +linux/dcbnl.h +linux/dccp.h +linux/devlink.h +linux/dlm.h +linux/dlm_device.h +linux/dlm_netlink.h +linux/dlm_plock.h +linux/dlmconstants.h +linux/dm-ioctl.h +linux/dm-log-userspace.h +linux/dma-heap.h +linux/dns_resolver.h +linux/dqblk_xfs.h +linux/dvb/audio.h +linux/dvb/ca.h +linux/dvb/frontend.h +linux/dvb/net.h +linux/dvb/osd.h +linux/dvb/version.h +linux/dw100.h +linux/edd.h +linux/efs_fs_sb.h +linux/elf-em.h +linux/elf-fdpic.h +linux/elf.h +linux/errno.h +linux/erspan.h +linux/eventpoll.h +linux/f2fs.h +linux/fadvise.h +linux/falloc.h +linux/fanotify.h +linux/fb.h +linux/fcntl.h +linux/fd.h +linux/fdreg.h +linux/fib_rules.h +linux/fiemap.h +linux/filter.h +linux/firewire-cdev.h +linux/firewire-constants.h +linux/fou.h +linux/fpga-dfl.h +linux/fs.h +linux/fscrypt.h +linux/fsi.h +linux/fsl_hypervisor.h +linux/fsl_mc.h +linux/fsmap.h +linux/fsverity.h +linux/futex.h +linux/gameport.h +linux/gen_stats.h +linux/genetlink.h +linux/genwqe/genwqe_card.h +linux/gfs2_ondisk.h +linux/gpio.h +linux/gtp.h +linux/hash_info.h +linux/hdlc.h +linux/hdlcdrv.h +linux/hdreg.h +linux/hid.h +linux/hiddev.h +linux/hidraw.h +linux/hsi/cs-protocol.h +linux/hsi/hsi_char.h +linux/hsr_netlink.h +linux/hw_breakpoint.h +linux/hyperv.h +linux/i2c-dev.h +linux/i2c.h +linux/i2o-dev.h +linux/i8k.h +linux/icmpv6.h +linux/if_addr.h +linux/if_addrlabel.h +linux/if_alg.h +linux/if_arcnet.h +linux/if_bridge.h +linux/if_cablemodem.h +linux/if_eql.h +linux/if_ether.h +linux/if_fc.h +linux/if_fddi.h +linux/if_hippi.h +linux/if_infiniband.h +linux/if_link.h +linux/if_ltalk.h +linux/if_macsec.h +linux/if_packet.h +linux/if_phonet.h +linux/if_plip.h +linux/if_ppp.h +linux/if_pppol2tp.h +linux/if_slip.h +linux/if_team.h +linux/if_tun.h +linux/if_vlan.h +linux/if_x25.h +linux/if_xdp.h +linux/ife.h +linux/igmp.h +linux/iio/buffer.h +linux/iio/events.h +linux/iio/types.h +linux/ila.h +linux/in.h +linux/in6.h +linux/in_route.h +linux/inet_diag.h +linux/inotify.h +linux/input-event-codes.h +linux/io_uring.h +linux/ioam6.h +linux/ioam6_genl.h +linux/ioam6_iptunnel.h +linux/ioctl.h +linux/iommu.h +linux/ioprio.h +linux/ip.h +linux/ip_vs.h +linux/ipc.h +linux/ipmi.h +linux/ipmi_bmc.h +linux/ipmi_msgdefs.h +linux/ipsec.h +linux/ipv6.h +linux/ipv6_route.h +linux/irqnr.h +linux/isdn/capicmd.h +linux/iso_fs.h +linux/isst_if.h +linux/ivtvfb.h +linux/jffs2.h +linux/kcm.h +linux/kcmp.h +linux/kcov.h +linux/kd.h +linux/kdev_t.h +linux/kernel-page-flags.h +linux/kernel.h +linux/kernelcapi.h +linux/keyboard.h +linux/keyctl.h +linux/kfd_sysfs.h +linux/kvm_para.h +linux/l2tp.h +linux/landlock.h +linux/libc-compat.h +linux/limits.h +linux/lirc.h +linux/loadpin.h +linux/loop.h +linux/lp.h +linux/lwtunnel.h +linux/magic.h +linux/major.h +linux/map_to_14segment.h +linux/map_to_7segment.h +linux/max2175.h +linux/media-bus-format.h +linux/media.h +linux/mei.h +linux/membarrier.h +linux/memfd.h +linux/mempolicy.h +linux/meye.h +linux/minix_fs.h +linux/misc/bcm_vk.h +linux/mman.h +linux/mmc/ioctl.h +linux/mmtimer.h +linux/module.h +linux/mount.h +linux/mpls.h +linux/mpls_iptunnel.h +linux/mqueue.h +linux/mroute.h +linux/mroute6.h +linux/mrp_bridge.h +linux/msdos_fs.h +linux/msg.h +linux/mtio.h +linux/nbd-netlink.h +linux/nbd.h +linux/ncsi.h +linux/ndctl.h +linux/neighbour.h +linux/net.h +linux/net_dropmon.h +linux/net_namespace.h +linux/net_tstamp.h +linux/netconf.h +linux/netfilter.h +linux/netfilter/ipset/ip_set.h +linux/netfilter/ipset/ip_set_bitmap.h +linux/netfilter/ipset/ip_set_hash.h +linux/netfilter/ipset/ip_set_list.h +linux/netfilter/nf_conntrack_common.h +linux/netfilter/nf_conntrack_ftp.h +linux/netfilter/nf_conntrack_sctp.h +linux/netfilter/nf_conntrack_tcp.h +linux/netfilter/nf_conntrack_tuple_common.h +linux/netfilter/nf_log.h +linux/netfilter/nf_nat.h +linux/netfilter/nf_synproxy.h +linux/netfilter/nf_tables.h +linux/netfilter/nf_tables_compat.h +linux/netfilter/nfnetlink.h +linux/netfilter/nfnetlink_acct.h +linux/netfilter/nfnetlink_compat.h +linux/netfilter/nfnetlink_conntrack.h +linux/netfilter/nfnetlink_cthelper.h +linux/netfilter/nfnetlink_cttimeout.h +linux/netfilter/nfnetlink_hook.h +linux/netfilter/nfnetlink_log.h +linux/netfilter/nfnetlink_osf.h +linux/netfilter/nfnetlink_queue.h +linux/netfilter/x_tables.h +linux/netfilter/xt_AUDIT.h +linux/netfilter/xt_CHECKSUM.h +linux/netfilter/xt_CLASSIFY.h +linux/netfilter/xt_CONNMARK.h +linux/netfilter/xt_CONNSECMARK.h +linux/netfilter/xt_CT.h +linux/netfilter/xt_DSCP.h +linux/netfilter/xt_HMARK.h +linux/netfilter/xt_IDLETIMER.h +linux/netfilter/xt_LED.h +linux/netfilter/xt_LOG.h +linux/netfilter/xt_MARK.h +linux/netfilter/xt_NFQUEUE.h +linux/netfilter/xt_SECMARK.h +linux/netfilter/xt_SYNPROXY.h +linux/netfilter/xt_TCPMSS.h +linux/netfilter/xt_TCPOPTSTRIP.h +linux/netfilter/xt_TEE.h +linux/netfilter/xt_TPROXY.h +linux/netfilter/xt_addrtype.h +linux/netfilter/xt_bpf.h +linux/netfilter/xt_cgroup.h +linux/netfilter/xt_cluster.h +linux/netfilter/xt_comment.h +linux/netfilter/xt_connbytes.h +linux/netfilter/xt_connlabel.h +linux/netfilter/xt_connlimit.h +linux/netfilter/xt_connmark.h +linux/netfilter/xt_conntrack.h +linux/netfilter/xt_cpu.h +linux/netfilter/xt_dccp.h +linux/netfilter/xt_devgroup.h +linux/netfilter/xt_dscp.h +linux/netfilter/xt_ecn.h +linux/netfilter/xt_esp.h +linux/netfilter/xt_helper.h +linux/netfilter/xt_ipcomp.h +linux/netfilter/xt_iprange.h +linux/netfilter/xt_ipvs.h +linux/netfilter/xt_l2tp.h +linux/netfilter/xt_length.h +linux/netfilter/xt_limit.h +linux/netfilter/xt_mac.h +linux/netfilter/xt_mark.h +linux/netfilter/xt_multiport.h +linux/netfilter/xt_nfacct.h +linux/netfilter/xt_osf.h +linux/netfilter/xt_owner.h +linux/netfilter/xt_pkttype.h +linux/netfilter/xt_policy.h +linux/netfilter/xt_quota.h +linux/netfilter/xt_realm.h +linux/netfilter/xt_recent.h +linux/netfilter/xt_rpfilter.h +linux/netfilter/xt_sctp.h +linux/netfilter/xt_set.h +linux/netfilter/xt_socket.h +linux/netfilter/xt_state.h +linux/netfilter/xt_statistic.h +linux/netfilter/xt_string.h +linux/netfilter/xt_tcpmss.h +linux/netfilter/xt_tcpudp.h +linux/netfilter/xt_time.h +linux/netfilter/xt_u32.h +linux/netfilter_arp.h +linux/netfilter_bridge/ebt_among.h +linux/netfilter_bridge/ebt_arp.h +linux/netfilter_bridge/ebt_arpreply.h +linux/netfilter_bridge/ebt_ip.h +linux/netfilter_bridge/ebt_ip6.h +linux/netfilter_bridge/ebt_limit.h +linux/netfilter_bridge/ebt_log.h +linux/netfilter_bridge/ebt_mark_m.h +linux/netfilter_bridge/ebt_mark_t.h +linux/netfilter_bridge/ebt_nat.h +linux/netfilter_bridge/ebt_nflog.h +linux/netfilter_bridge/ebt_pkttype.h +linux/netfilter_bridge/ebt_redirect.h +linux/netfilter_bridge/ebt_stp.h +linux/netfilter_bridge/ebt_vlan.h +linux/netfilter_ipv4.h +linux/netfilter_ipv4/ipt_CLUSTERIP.h +linux/netfilter_ipv4/ipt_ECN.h +linux/netfilter_ipv4/ipt_LOG.h +linux/netfilter_ipv4/ipt_REJECT.h +linux/netfilter_ipv4/ipt_TTL.h +linux/netfilter_ipv4/ipt_ah.h +linux/netfilter_ipv4/ipt_ecn.h +linux/netfilter_ipv4/ipt_ttl.h +linux/netfilter_ipv6.h +linux/netfilter_ipv6/ip6t_HL.h +linux/netfilter_ipv6/ip6t_LOG.h +linux/netfilter_ipv6/ip6t_NPT.h +linux/netfilter_ipv6/ip6t_REJECT.h +linux/netfilter_ipv6/ip6t_ah.h +linux/netfilter_ipv6/ip6t_frag.h +linux/netfilter_ipv6/ip6t_hl.h +linux/netfilter_ipv6/ip6t_ipv6header.h +linux/netfilter_ipv6/ip6t_mh.h +linux/netfilter_ipv6/ip6t_opts.h +linux/netfilter_ipv6/ip6t_rt.h +linux/netfilter_ipv6/ip6t_srh.h +linux/netlink.h +linux/netlink_diag.h +linux/netrom.h +linux/nexthop.h +linux/nfc.h +linux/nfs.h +linux/nfs2.h +linux/nfs3.h +linux/nfs4.h +linux/nfs4_mount.h +linux/nfs_fs.h +linux/nfs_idmap.h +linux/nfs_mount.h +linux/nfsacl.h +linux/nfsd/cld.h +linux/nfsd/debug.h +linux/nfsd/export.h +linux/nfsd/stats.h +linux/nilfs2_api.h +linux/nilfs2_ondisk.h +linux/nitro_enclaves.h +linux/nl80211-vnd-intel.h +linux/nl80211.h +linux/nsfs.h +linux/nubus.h +linux/nvme_ioctl.h +linux/nvram.h +linux/oom.h +linux/openat2.h +linux/openvswitch.h +linux/packet_diag.h +linux/param.h +linux/parport.h +linux/pci.h +linux/pci_regs.h +linux/pcitest.h +linux/perf_event.h +linux/personality.h +linux/pfkeyv2.h +linux/pfrut.h +linux/pg.h +linux/phantom.h +linux/pidfd.h +linux/pkt_cls.h +linux/pkt_sched.h +linux/pktcdvd.h +linux/pmu.h +linux/poll.h +linux/posix_acl.h +linux/posix_types.h +linux/ppdev.h +linux/ppp-comp.h +linux/ppp-ioctl.h +linux/ppp_defs.h +linux/pps.h +linux/pr.h +linux/prctl.h +linux/psample.h +linux/psci.h +linux/psp-sev.h +linux/ptp_clock.h +linux/qemu_fw_cfg.h +linux/qnx4_fs.h +linux/qnxtypes.h +linux/qrtr.h +linux/radeonfb.h +linux/raid/md_p.h +linux/raid/md_u.h +linux/random.h +linux/rds.h +linux/reboot.h +linux/reiserfs_fs.h +linux/reiserfs_xattr.h +linux/remoteproc_cdev.h +linux/resource.h +linux/rfkill.h +linux/rio_cm_cdev.h +linux/rio_mport_cdev.h +linux/rkisp1-config.h +linux/romfs_fs.h +linux/rose.h +linux/rpl.h +linux/rpl_iptunnel.h +linux/rpmsg.h +linux/rpmsg_types.h +linux/rseq.h +linux/rtc.h +linux/rtnetlink.h +linux/rxrpc.h +linux/scc.h +linux/sched.h +linux/sched/types.h +linux/scif_ioctl.h +linux/screen_info.h +linux/seccomp.h +linux/securebits.h +linux/sed-opal.h +linux/seg6.h +linux/seg6_genl.h +linux/seg6_hmac.h +linux/seg6_iptunnel.h +linux/seg6_local.h +linux/selinux_netlink.h +linux/sem.h +linux/serial_reg.h +linux/serio.h +linux/sev-guest.h +linux/shm.h +linux/signalfd.h +linux/smc.h +linux/smc_diag.h +linux/smiapp.h +linux/snmp.h +linux/socket.h +linux/sockios.h +linux/sonet.h +linux/sonypi.h +linux/sound.h +linux/soundcard.h +linux/spi/spi.h +linux/spi/spidev.h +linux/stat.h +linux/stddef.h +linux/stm.h +linux/string.h +linux/sunrpc/debug.h +linux/surface_aggregator/cdev.h +linux/surface_aggregator/dtx.h +linux/suspend_ioctls.h +linux/swab.h +linux/switchtec_ioctl.h +linux/sync_file.h +linux/sysinfo.h +linux/target_core_user.h +linux/taskstats.h +linux/tc_act/tc_bpf.h +linux/tc_act/tc_csum.h +linux/tc_act/tc_ct.h +linux/tc_act/tc_defact.h +linux/tc_act/tc_gact.h +linux/tc_act/tc_gate.h +linux/tc_act/tc_ife.h +linux/tc_act/tc_ipt.h +linux/tc_act/tc_mirred.h +linux/tc_act/tc_mpls.h +linux/tc_act/tc_nat.h +linux/tc_act/tc_pedit.h +linux/tc_act/tc_sample.h +linux/tc_act/tc_skbedit.h +linux/tc_act/tc_skbmod.h +linux/tc_act/tc_tunnel_key.h +linux/tc_act/tc_vlan.h +linux/tc_ematch/tc_em_cmp.h +linux/tc_ematch/tc_em_ipt.h +linux/tc_ematch/tc_em_meta.h +linux/tc_ematch/tc_em_nbyte.h +linux/tc_ematch/tc_em_text.h +linux/tcp.h +linux/tcp_metrics.h +linux/tee.h +linux/termios.h +linux/thermal.h +linux/time.h +linux/time_types.h +linux/timerfd.h +linux/times.h +linux/timex.h +linux/tiocl.h +linux/tipc.h +linux/tipc_config.h +linux/tipc_netlink.h +linux/tls.h +linux/toshiba.h +linux/tty.h +linux/tty_flags.h +linux/types.h +linux/ublk_cmd.h +linux/udf_fs_i.h +linux/udmabuf.h +linux/udp.h +linux/uio.h +linux/uleds.h +linux/ultrasound.h +linux/um_timetravel.h +linux/un.h +linux/unistd.h +linux/unix_diag.h +linux/usb/cdc-wdm.h +linux/usb/ch11.h +linux/usb/ch9.h +linux/usb/charger.h +linux/usb/functionfs.h +linux/usb/g_printer.h +linux/usb/g_uvc.h +linux/usb/gadgetfs.h +linux/usb/midi.h +linux/usb/raw_gadget.h +linux/usb/tmc.h +linux/usb/video.h +linux/usbdevice_fs.h +linux/usbip.h +linux/userfaultfd.h +linux/userio.h +linux/utime.h +linux/utsname.h +linux/uuid.h +linux/uvcvideo.h +linux/v4l2-common.h +linux/v4l2-controls.h +linux/v4l2-dv-timings.h +linux/vbox_err.h +linux/vbox_vmmdev_types.h +linux/vboxguest.h +linux/vdpa.h +linux/vduse.h +linux/veth.h +linux/vfio.h +linux/vfio_ccw.h +linux/vfio_zdev.h +linux/virtio_9p.h +linux/virtio_balloon.h +linux/virtio_blk.h +linux/virtio_bt.h +linux/virtio_config.h +linux/virtio_console.h +linux/virtio_crypto.h +linux/virtio_fs.h +linux/virtio_gpio.h +linux/virtio_gpu.h +linux/virtio_i2c.h +linux/virtio_ids.h +linux/virtio_input.h +linux/virtio_iommu.h +linux/virtio_mem.h +linux/virtio_mmio.h +linux/virtio_net.h +linux/virtio_pci.h +linux/virtio_pcidev.h +linux/virtio_pmem.h +linux/virtio_rng.h +linux/virtio_scmi.h +linux/virtio_scsi.h +linux/virtio_snd.h +linux/virtio_types.h +linux/virtio_vsock.h +linux/vm_sockets_diag.h +linux/vmcore.h +linux/vsockmon.h +linux/vt.h +linux/vtpm_proxy.h +linux/wait.h +linux/watch_queue.h +linux/watchdog.h +linux/wireguard.h +linux/wmi.h +linux/wwan.h +linux/x25.h +linux/xattr.h +linux/xdp_diag.h +linux/xfrm.h +linux/xilinx-v4l2-controls.h +linux/zorro.h +linux/zorro_ids.h
Signature · 1 line
2.34.1
Eduard Zingerman
maintainerRFC 12/12 selftests/bpf: script for infer_header_guards.pl testing
In reply to Eduard ZingermanThis script verifies that patterns for header guard inference specified in scripts/infer_header_guards.pl cover all uapi headers. To achieve this the infer_header_guards.pl is invoked the same way it is invoked from link-vmlinux.sh but with --report-failures flag.
--- .../bpf/test_uapi_header_guards_infer.sh | 33 +++++++++++++++++++ 1 file changed, 33 insertions(+) create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh
tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh new file +33 -0
@@ -0,0 +1,33 @@ +#!/bin/bash +# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause) + +# This script verifies that patterns for header guard inference +# specified in scripts/infer_header_guards.pl cover all uapi headers. +# To achieve this the infer_header_guards.pl is invoked the same way +# it is invoked from link-vmlinux.sh but with --report-failures flag. + +kernel_dir=$(dirname $0)/../../../../ + +# The SRCARCH is defined in tools/scripts/Makefile.arch, thus use a +# temporary makefile to get access to this variable. +fake_makefile=$(cat <<EOF +include tools/scripts/Makefile.arch +default: + scripts/infer_header_guards.pl --report-failures \ + include/uapi \ + include/generated/uapi \ + arch/\$(SRCARCH)/include/uapi \ + arch/\$(SRCARCH)/include/generated/uapi 1>/dev/null +EOF +) + +# The infer_header_guards.pl script prints inferred guards to stdout, +# redirecting stdout to /dev/null to see only error messages. +echo "$fake_makefile" | make -C $kernel_dir -f - 1>/dev/null +if [ "$?" == "0" ]; then + echo "all good" + exit 0 +fi + +# Failures are already reported by infer_header_guards.pl +exit 1
Signature · 1 line
2.34.1
Quoted 31 lines · Eduard Zingerman
Include the following system header:
-/usr/include/sys/socket.h(all vialinux/if.h)The
sys/socket.hconflicts withvmlinux.hin:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants:SOCK_STREAM,SOCK_DGRAM, ...However, only two types are actually used:
- struct sockaddr
- structsockaddr_storage(used only inlinux/mptcp.h)In '
vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.The only workaround that I see is to:
- define a stubsys/socket.has follows:#ifndef __BPF_SOCKADDR__ #define __BPF_SOCKADDR__ /* For __kernel_sa_family_t */ #include <linux/socket.h> struct sockaddr { __kernel_sa_family_t sa_family; char sa_data[14]; }; #endif
Quoted 2 lines
- hardcode generation of
__BPF_SOCKADDR__bracket for 'struct sockaddr' invmlinux.h.
we don't need to hack sys/socket.h and can hardcode #ifdef _SYS_SOCKET_H as header guard for sockaddr instead, right? bits/socket.h has this:
#ifndef _SYS_SOCKET_H
# error "Never include <bits/socket.h> directly; use <sys/socket.h> instead."
So that ifdef is kinda stable.
Quoted 16 lines
Another possibility is to move the definition of 'struct sockaddr' from '
kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.Conflict with
vmlinux.h
----Uapi header:
-linux/signal.hConflict with
vmlinux.hin definition of 'struct sigaction'. Defined in:
-vmlinux.h:kernel/include/linux/signal_types.h
- uapi:kernel/arch/x86/include/asm/signal.h
Quoted 5 lines
Uapi headers:
-linux/tipc_sockets_diag.h
-linux/sock_diag.hConflict with
vmlinux.hin definition of 'SOCK_DESTROY'.
Interesting one!
I think we can hard code '#undef SOCK_DESTROY' in vmlinux.h
The goal is not to be able to mix arbitrary uapi header with vmlinux.h, but only those that could be useful out of bpf progs. Currently it's tcp.h and few other network related headers because they have #define-s in them that are useful inside bpf progs. As long as the solution covers this small subset we're fine.
Eduard Zingerman
maintainerQuoted 35 lines · Alexei Starovoitov
Quoted 31 lines · Eduard Zingerman
Include the following system header:
-/usr/include/sys/socket.h(all vialinux/if.h)The
sys/socket.hconflicts withvmlinux.hin:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants:SOCK_STREAM,SOCK_DGRAM, ...However, only two types are actually used:
- struct sockaddr
- structsockaddr_storage(used only inlinux/mptcp.h)In '
vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.The only workaround that I see is to:
- define a stubsys/socket.has follows:#ifndef __BPF_SOCKADDR__ #define __BPF_SOCKADDR__ /* For __kernel_sa_family_t */ #include <linux/socket.h> struct sockaddr { __kernel_sa_family_t sa_family; char sa_data[14]; }; #endifQuoted 2 lines
- hardcode generation of
__BPF_SOCKADDR__bracket for 'struct sockaddr' invmlinux.h.
Quoted 7 lines
we don't need to hack
sys/socket.hand can hardcode #ifdef_SYS_SOCKET_Has header guard for sockaddr instead, right?bits/socket.hhas this:
#ifndef_SYS_SOCKET_H# error "Never include <bits/socket.h> directly; use <sys/socket.h> instead."So that ifdef is kinda stable.
The if.h only uses two types from sys/socket.h, namely:
- struct sockaddr
- struct sockaddr_storage
However sys/socket.h itself defines more types, here is a complete list of types from sys/socket.h that conflict with vmlinux.h (generated for my x86_64 laptop):
Type name System header iovec /usr/include/bits/types/struct_iovec.h loff_t /usr/include/sys/types.h dev_t /usr/include/sys/types.h nlink_t /usr/include/sys/types.h timer_t /usr/include/bits/types/timer_t.h int16_t /usr/include/bits/stdint-intn.h int32_t /usr/include/bits/stdint-intn.h int64_t /usr/include/bits/stdint-intn.h u_int64_t /usr/include/sys/types.h sigset_t /usr/include/bits/types/sigset_t.h fd_set /usr/include/sys/select.h blkcnt_t /usr/include/sys/types.h SOCK_STREAM /usr/include/bits/socket_type.h SOCK_DGRAM /usr/include/bits/socket_type.h SOCK_RAW /usr/include/bits/socket_type.h SOCK_RDM /usr/include/bits/socket_type.h SOCK_SEQPACKET /usr/include/bits/socket_type.h SOCK_DCCP /usr/include/bits/socket_type.h SOCK_PACKET /usr/include/bits/socket_type.h sockaddr /usr/include/bits/socket.h msghdr /usr/include/bits/socket.h cmsghdr /usr/include/bits/socket.h linger /usr/include/bits/socket.h SHUT_RD /usr/include/sys/socket.h SHUT_WR /usr/include/sys/socket.h SHUT_RDWR /usr/include/sys/socket.h
It would be safe to wrap the corresponding types in the vmlinux.h with _SYS_SOCKET_H / _SYS_TYPES guards if the definitions above match between libc and kernel. To my surprise not all of them match. Here is the list of genuine conflicts (for typedefs I skip the intermediate definitions and print the last typedef in the chain):
Type: dev_t
typedef unsigned int __u32 vmlinux.h
typedef unsigned long int __dev_t /usr/include/bits/types.h
Type: nlink_t
typedef unsigned int __u32 vmlinux.h
typedef unsigned long int __nlink_t /usr/include/bits/types.h
Type: timer_t
typedef int __kernel_timer _t vmlinux.h
typedef void *__timer_t /usr/include/bits/types.h
Type: sigset_t
typedef struct vmlinux.h
{
long unsigned int sig[1];
} sigset_t
typedef struct /usr/include/bits/types/__sigset_t.h
{
unsigned long int __val[1024 / (8 * (sizeof(unsigned long int)))];
} __sigset_t
Type: fd_set
typedef struct vmlinux.h
{
long unsigned int fds_bits[16];
} __kernel_fd_set
typedef struct /usr/include/sys/select.h
{
__fd_mask __fds_bits[1024 / (8 * ((int) (sizeof(__fd_mask))))];
} fd_set
Type: sigset_t
typedef struct vmlinux.h
{
long unsigned int sig[1];
} sigset_t
typedef struct /usr/include/bits/types/__sigset_t.h
{
unsigned long int __val[1024 / (8 * (sizeof(unsigned long int)))];
} __sigset_t
Type: msghdr
struct msghdr vmlinux.h
{
void *msg_name;
int msg_namelen;
int msg_inq;
struct iov_iter msg_iter;
union
{
void *msg_control;
void *msg_control_user; };
bool msg_control_is_user : 1;
bool msg_get_inq : 1;
unsigned int msg_flags;
__kernel_size_t msg_controllen;
struct kiocb *msg_iocb;
struct ubuf_info *msg_ubuf;
struct sock *, struct sk_buff *, struct iov_iter *, size_tint;
}
struct msghdr /usr/include/bits/socket.h
{
void *msg_name;
socklen_t msg_namelen;
struct iovec *msg_iov;
size_t msg_iovlen;
void *msg_control;
size_t msg_controllen;
int msg_flags;
}
Quoted 23 lines
Quoted 17 lines
Another possibility is to move the definition of 'struct sockaddr' from '
kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.Conflict with
vmlinux.h
----Uapi header:
-linux/signal.hConflict with
vmlinux.hin definition of 'struct sigaction'. Defined in:
-vmlinux.h:kernel/include/linux/signal_types.h
- uapi:kernel/arch/x86/include/asm/signal.hQuoted 5 lines
Uapi headers:
-linux/tipc_sockets_diag.h
-linux/sock_diag.hConflict with
vmlinux.hin definition of 'SOCK_DESTROY'.
Quoted 8 lines
Interesting one!
I think we can hard code '#undefSOCK_DESTROY' invmlinux.hThe goal is not to be able to mix arbitrary uapi header with
vmlinux.h, but only those that could be useful out of bpf progs. Currently it'stcp.hand few other network related headers because they have #define-s in them that are useful inside bpf progs. As long as the solution covers this small subset we're fine.
Well, tcp.h works :) It would be great if someone could list the interesting headers.
Thanks,
Eduard
Quoted 8 lines · Eduard Zingerman
Hi BPF community,
AFAIK there is a long standing feature request to use kernel headers alongside
vmlinux.hgenerated bybpftool. For example significant effort was put to add an attributebpf_dominating_decl(see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.
Thanks for the details! It's a tricky problem to solve.
Before diving into this, can I ask if there's another way round this; is there no way we could teach vmlinux.h generation which types to skip via some kind of bootstrapping process?
For a bpf object foo.bpf.c that wants to include linux/tcp.h and vmlinux.h, something like this seems possible;
1. run the preprocessor (gcc -E) over the BPF program to generate
a bootstrap header foo.bpf.exclude_types.h consisting of all the types mentioned in it and associated includes, but not in vmlinux.h It would need to -D__VMLINUX_H__ to avoid including vmlinux.h definitions and -D__BPF_EXCLUDE_TYPE_BOOTSTRAP__ to skip the programs themselves, which would need a guard around them I think:
#include <stddef.h>
#include <stdbool.h>
#include <vmlinux.h>
#include <linux/tcp.h>
#ifndef __BPF_EXCLUDE_TYPE_BOOTSTRAP__
//rest of program here
#endif
So as a result of this, we now have a single header file that contains all the types that non-vmlinux.h include files define.
2. now to generate vmlinux.h, pass foo.bpf.exclude_types.h into "bpftool btf dump" as an exclude header:
bpftool btf dump --exclude /tmp/foo.bpf.types.h file /sys/kernel/btf/vmlinux format c > vmlinux.h
bpftool would have to parse the exclude header for actual type definitions, spotting struct, enum and typedef definitions. This is likely made easier by running the preprocessor at least since formatting is probably quite uniform. vmlinux.h could simply emit forward declarations for types described both in vmlinux BTF and in the exclude file.
So the build process would be
- start with empty vmlinux.h
- bootstrap a header consisting of the set of types to exclude via c preprocessor
- generate new vmlinux.h based on above
- build bpf program
Build processes for BPF objects already has bootstrapping elements like this for generating vmlinux.h and skeletons, so while it's not perfect it might be a simpler approach. There may be problems with this I'm not seeing though?
Thanks!
Alan
Quoted 359 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.The basic idea
---The goal of the patch set is to allow usage of header files from
include/uapialongsidevmlinux.has follows:#include <uapi/linux/tcp.h> #include "vmlinux.h"This goal is achieved by adding
#ifndef ... #endifguards invmlinux.haround definitions that originate from theinclude/uapiheaders. The guards emitted match the guards used in the original headers. E.g. as follows:
include/uapi/linux/tcp.h:#ifndef
_UAPI_LINUX_TCP_H
#define_UAPI_LINUX_TCP_H
...union tcp_word_hdr { struct tcphdr hdr; __be32 words[5];};
...
#endif /*_UAPI_LINUX_TCP_H*/
vmlinux.h:... #ifndef _UAPI_LINUX_TCP_Hunion tcp_word_hdr { struct tcphdr hdr; __be32 words[5];};
#endif /* _UAPI_LINUX_TCP_H */ ...To get to this state the following steps are necessary:
- "header guard" name should be identified for each header file;
- the correspondence between data type and it's header guard has to be encoded in BTF;
-bpftoolshould be adjusted to emit#ifndef FOO_H ... #endifbrackets.It is not possible to identify header guard names for all uapi headers basing only on the file name. However a simple script could devised to identify the guards basing on the file name and it's content. Thus it is possible to obtain the list of header names with corresponding header guards.
The correspondence between type and it's declaration file (header) is available in DWARF as
DW_AT_decl_fileattribute. TheDW_AT_decl_filecan be matched with the list of header guards described above to obtain the header guard name for a specific type.The
paholegenerates BTF using DWARF. It is possible to modifypaholeto accept the header guards list as an additional parameter and to encode the header guard names in BTF.Implementation details
---Present patch-set implements these ideas as follows:
- A parameter--header_guards_dbis added topahole. If present it points to a file with a list of<header> <guard>records.
-paholeuses DWARFDW_AT_decl_filevalue to lookup the header guard for each type emitted to BTF. If header guard is present it is encoded alongside the type.
- Header guards are encoded in BTF asBTF_DECL_TAGrecords with a special prefix. The prefix "header_guard:" is added to a value of such tags. (HereBTF_DECL_TAGis used to avoid BTF binary format changes).
- A special scriptinfer_header_guards.plis added as a part of kbuild, it can infer header guard names for each UAPI header basing on the header content.
- This script is invoked fromlink-vmlinux.shprior to BTF generation during kernel build. The output of the script is saved to a file, the file is passed topaholeas--header_guards_dbparameter.
-libbpfis modified to aggregateBTF_DECL_TAGrecords for each type and to emit#ifndef FOO_H ... #endifbrackets when "header_guard:" tag is present for a type.Details for each patch in a set:
- libbpf: Deduplicate unambigous standalone forward declarations
- selftests/bpf: Tests for standalone forward BTF declarations deduplicationThere is a small number (63 for defconfig) of forward declarations that are not de-duplicated with the main type declaration under certain conditions. This hinders the header guard brackets generation. This patch addresses this de-duplication issue.
- libbpf: Support for
BTF_DECL_TAGdump in C format
- selftests/bpf: Tests forBTF_DECL_TAGdump in C formatCurrently libbpf does not process
BTF_DECL_TAGwhen btf is dumped in C format. This patch adds a hash table matching btf type ids with a list of decl tags to the structbtf_dump.
Thebtf_dump_emit_decl_tagsis not necessary for the overall patch-set to function but simplifies testing a bit.- libbpf: Header guards for selected data structures in
vmlinux.h
- selftests/bpf: Tests for header guards printing in BTF dumpAdds option
emit_header_guardstostruct btf_dump_opts. When enabled thebtf_dump__dump_typeprints#ifndef ... #endifbrackets around types for which header guard information is present in BTF.- bpftool: Enable header guards generation
Unconditionally enables
emit_header_guardsfor BTF dump in C format.- kbuild: Script to infer header guard values for uapi headers
- kbuild: Header guards for types frominclude/uapi/*.hin kernel BTFAdds
scripts/infer_header_guards.pland integrates it withlink-vmlinux.sh.- selftests/bpf: Script to verify uapi headers usage with
vmlinux.hAdds a script
test_uapi_headers.pythat tests header guards withvmlinux.hby compiling a simple C snippet. The snippet looks as follows:#include <some_uapi_header.h> #include "vmlinux.h" __attribute__((section("tc"), used)) int syncookie_tc(struct __sk_buff *skb) { return 0; }The list of headers to test comes from
tools/testing/selftests/bpf/good_uapi_headers.txt.- selftests/bpf: Known good uapi headers for
test_uapi_headers.pyThe list of uapi headers that could be included alongside
vmlinux.h. The headers are peeked from the following locations:
- <headers-export-dir>/linux/*.h
- <headers-export-dir>/linux/**/*.h
This choice of locations is somewhat arbitrary.- selftests/bpf: script for infer_header_guards.pl testing
The test case for
scripts/infer_header_guards.pl, verifies that header guards can be inferred for all uapi headers.- There is also a patch for dwarves that adds
--header_guards_dboption (see [2]).The
test_uapi_headers.pyis important as it demonstrates the the necessary compiler flags:clang ... \ -D__x86_64__ \ -Xclang -fwchar-type=short \ -Xclang -fno-signed-wchar \ -I{exported_kernel_headers}/include/ \ ...-
-fwchar-type=shortand-fno-signed-wcharhad to be added because BPF target usesintforwchar_tby default and this differs fromvmlinux.hdefinition of the type (at least forx86_64).
-__x86_64__had to be added for uapi headers that includestddef.h(the one that is supplied my CLANG itself), in order to define correct sizes forsize_tandptrdiff_t.
- The{exported_kernel_headers}stands for exported kernel headers directory (the headers obtained bymake headers_installor via distribution package).When it works
---The mechanics described above works for a significant number of UAPI headers. For example, for the test case above I chose the headers from the following locations:
- linux/*.h
- linux/**/*.h
There are 759 such headers and for 677 of them the test described above passes.I excluded the headers from the following sub-directories as potentially not interesting:
asm rdma video xen asm-generic misc scsi drm mtd soundThus saving some time for both discussion and CI but the choice is somewhat arbitrary. If I run
test_uapi_headers.py --test '*'(all headers) test passes for 834 out of 972 headers.When it breaks
---There several scenarios when this mechanics breaks. Specifically I found the following cases:
- When uapi header includes some system header that conflicts withvmlinux.h.
- When uapi header itself conflicts withvmlinux.h.Below are examples for both cases.
Conflict with system headers
----The following uapi headers:
-linux/atmbr2684.h
-linux/bpfilter.h
-linux/gsmmux.h
-linux/icmp.h
-linux/if.h
-linux/if_arp.h
-linux/if_bonding.h
-linux/if_pppox.h
-linux/if_tunnel.h
-linux/ip6_tunnel.h
-linux/llc.h
-linux/mctp.h
-linux/mptcp.h
-linux/netdevice.h
-linux/netfilter/xt_RATEEST.h
-linux/netfilter/xt_hashlimit.h
-linux/netfilter/xt_physdev.h
-linux/netfilter/xt_rateest.h
-linux/netfilter_arp/arp_tables.h
-linux/netfilter_arp/arpt_mangle.h
-linux/netfilter_bridge.h
-linux/netfilter_bridge/ebtables.h
-linux/netfilter_ipv4/ip_tables.h
-linux/netfilter_ipv6/ip6_tables.h
-linux/route.h
-linux/wireless.hInclude the following system header:
-/usr/include/sys/socket.h(all vialinux/if.h)The
sys/socket.hconflicts withvmlinux.hin:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants:SOCK_STREAM,SOCK_DGRAM, ...However, only two types are actually used:
- struct sockaddr
- structsockaddr_storage(used only inlinux/mptcp.h)In '
vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.The only workaround that I see is to:
- define a stubsys/socket.has follows:#ifndef __BPF_SOCKADDR__ #define __BPF_SOCKADDR__ /* For __kernel_sa_family_t */ #include <linux/socket.h> struct sockaddr { __kernel_sa_family_t sa_family; char sa_data[14]; }; #endif- hardcode generation of
__BPF_SOCKADDR__bracket for 'struct sockaddr' invmlinux.h.Another possibility is to move the definition of 'struct sockaddr' from '
kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.Conflict with
vmlinux.h
----Uapi header:
-linux/signal.hConflict with
vmlinux.hin definition of 'struct sigaction'. Defined in:
-vmlinux.h:kernel/include/linux/signal_types.h
- uapi:kernel/arch/x86/include/asm/signal.hUapi headers:
-linux/tipc_sockets_diag.h
-linux/sock_diag.hConflict with
vmlinux.hin definition of 'SOCK_DESTROY'. Defined in:
-vmlinux.h:kernel/include/net/sock.h
- uapi:kernel/include/uapi/linux/sock_diag.h
Constants seem to be unrelated.And so on... I have details for many other headers but omit those for brevity.
In conclusion
---Except from the general feasibility I have a few questions:
- What UAPI headers are the candidates for such use? If there are some interesting headers currently not working with this patch-set some hacks have to be added (e.g. like withlinux/if.h).
- Is it ok to encode header guards as specialBTF_DECL_TAGor should I change the BTF format a bit to save some bytes.Thanks,
Eduard[1] https://reviews.llvm.org/D111307
[clang]__attribute__bpf_dominating_decl
[2] https://lore.kernel.org/dwarves/[email protected]/T/
[RFC dwarves] pahole: Save header guard names when
--header_guards_db is passedEduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testingscripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-) create mode 100755 scripts/infer_header_guards.pl create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
Eduard Zingerman
maintainerQuoted 49 lines · Alan Maguire
Quoted 8 lines · Eduard Zingerman
Hi BPF community,
AFAIK there is a long standing feature request to use kernel headers alongside
vmlinux.hgenerated bybpftool. For example significant effort was put to add an attributebpf_dominating_decl(see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.Thanks for the details! It's a tricky problem to solve.
Before diving into this, can I ask if there's another way round this; is there no way we could teach
vmlinux.hgeneration which types to skip via some kind of bootstrapping process?For a bpf object
foo.bpf.cthat wants to includelinux/tcp.handvmlinux.h, something like this seems possible;1. run the preprocessor (gcc -E) over the BPF program to generate
a bootstrap headerfoo.bpf.exclude_types.hconsisting of all the types mentioned in it and associated includes, but not invmlinux.hIt would need to -D__VMLINUX_H__ to avoid includingvmlinux.hdefinitions and -D__BPF_EXCLUDE_TYPE_BOOTSTRAP__ to skip the programs themselves, which would need a guard around them I think:#include <
stddef.h>
#include <stdbool.h>
#include <vmlinux.h>
#include <linux/tcp.h>#ifndef
__BPF_EXCLUDE_TYPE_BOOTSTRAP__
//rest of program here
#endifSo as a result of this, we now have a single header file that contains all the types that
non-vmlinux.hinclude files define.2. now to generate
vmlinux.h, passfoo.bpf.exclude_types.hinto "bpftool btf dump" as an exclude header:bpftool btf dump --exclude
/tmp/foo.bpf.types.hfile/sys/kernel/btf/vmlinuxformat c >vmlinux.hbpftool would have to parse the exclude header for actual type definitions, spotting struct, enum and typedef definitions. This is likely made easier by running the preprocessor at least since formatting is probably quite uniform.
vmlinux.hcould simply emit forward declarations for types described both in vmlinux BTF and in the exclude file.So the build process would be
Quoted 8 lines
- start with empty
vmlinux.h
- bootstrap a header consisting of the set of types to exclude via c preprocessor
- generate newvmlinux.hbased on above
- build bpf programBuild processes for BPF objects already has bootstrapping elements like this for generating
vmlinux.hand skeletons, so while it's not perfect it might be a simpler approach. There may be problems with this I'm not seeing though?
I like the tool based approach more but I heard there were some reservations about separate tool complicating the build process.
- the tool does not require changes to the kbuild;
- the tool is not limited to uapi headers;
- the tool could imitate the dominating declarations attribute and address the issue with definitions miss-match between vmlinux.h and system headers (see my reply to Alexei in the parallel sub-thread). To support this point it would have to work in a somewhat different order:
- read/pre-process the input file;
- remove from it the definitions that are also present in kernel BTF;
- prepend vmlinux.h to the beginning of the file.
On the other hand I think that implementation would need a C parser / type analysis step. Thus it would be harder to implement it as a part of bpftool. But the prototype using something like [1] should be simple.
Thanks,
Eduard
[1] https://github.com/inducer/pycparserext
Quoted 364 lines
Thanks!
Alan
Quoted 359 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.The basic idea
---The goal of the patch set is to allow usage of header files from
include/uapialongsidevmlinux.has follows:#include <uapi/linux/tcp.h> #include "vmlinux.h"This goal is achieved by adding
#ifndef ... #endifguards invmlinux.haround definitions that originate from theinclude/uapiheaders. The guards emitted match the guards used in the original headers. E.g. as follows:
include/uapi/linux/tcp.h:#ifndef
_UAPI_LINUX_TCP_H
#define_UAPI_LINUX_TCP_H
...union tcp_word_hdr { struct tcphdr hdr; __be32 words[5];};
...
#endif /*_UAPI_LINUX_TCP_H*/
vmlinux.h:... #ifndef _UAPI_LINUX_TCP_Hunion tcp_word_hdr { struct tcphdr hdr; __be32 words[5];};
#endif /* _UAPI_LINUX_TCP_H */ ...To get to this state the following steps are necessary:
- "header guard" name should be identified for each header file;
- the correspondence between data type and it's header guard has to be encoded in BTF;
-bpftoolshould be adjusted to emit#ifndef FOO_H ... #endifbrackets.It is not possible to identify header guard names for all uapi headers basing only on the file name. However a simple script could devised to identify the guards basing on the file name and it's content. Thus it is possible to obtain the list of header names with corresponding header guards.
The correspondence between type and it's declaration file (header) is available in DWARF as
DW_AT_decl_fileattribute. TheDW_AT_decl_filecan be matched with the list of header guards described above to obtain the header guard name for a specific type.The
paholegenerates BTF using DWARF. It is possible to modifypaholeto accept the header guards list as an additional parameter and to encode the header guard names in BTF.Implementation details
---Present patch-set implements these ideas as follows:
- A parameter--header_guards_dbis added topahole. If present it points to a file with a list of<header> <guard>records.
-paholeuses DWARFDW_AT_decl_filevalue to lookup the header guard for each type emitted to BTF. If header guard is present it is encoded alongside the type.
- Header guards are encoded in BTF asBTF_DECL_TAGrecords with a special prefix. The prefix "header_guard:" is added to a value of such tags. (HereBTF_DECL_TAGis used to avoid BTF binary format changes).
- A special scriptinfer_header_guards.plis added as a part of kbuild, it can infer header guard names for each UAPI header basing on the header content.
- This script is invoked fromlink-vmlinux.shprior to BTF generation during kernel build. The output of the script is saved to a file, the file is passed topaholeas--header_guards_dbparameter.
-libbpfis modified to aggregateBTF_DECL_TAGrecords for each type and to emit#ifndef FOO_H ... #endifbrackets when "header_guard:" tag is present for a type.Details for each patch in a set:
- libbpf: Deduplicate unambigous standalone forward declarations
- selftests/bpf: Tests for standalone forward BTF declarations deduplicationThere is a small number (63 for defconfig) of forward declarations that are not de-duplicated with the main type declaration under certain conditions. This hinders the header guard brackets generation. This patch addresses this de-duplication issue.
- libbpf: Support for
BTF_DECL_TAGdump in C format
- selftests/bpf: Tests forBTF_DECL_TAGdump in C formatCurrently libbpf does not process
BTF_DECL_TAGwhen btf is dumped in C format. This patch adds a hash table matching btf type ids with a list of decl tags to the structbtf_dump.
Thebtf_dump_emit_decl_tagsis not necessary for the overall patch-set to function but simplifies testing a bit.- libbpf: Header guards for selected data structures in
vmlinux.h
- selftests/bpf: Tests for header guards printing in BTF dumpAdds option
emit_header_guardstostruct btf_dump_opts. When enabled thebtf_dump__dump_typeprints#ifndef ... #endifbrackets around types for which header guard information is present in BTF.- bpftool: Enable header guards generation
Unconditionally enables
emit_header_guardsfor BTF dump in C format.- kbuild: Script to infer header guard values for uapi headers
- kbuild: Header guards for types frominclude/uapi/*.hin kernel BTFAdds
scripts/infer_header_guards.pland integrates it withlink-vmlinux.sh.- selftests/bpf: Script to verify uapi headers usage with
vmlinux.hAdds a script
test_uapi_headers.pythat tests header guards withvmlinux.hby compiling a simple C snippet. The snippet looks as follows:#include <some_uapi_header.h> #include "vmlinux.h" __attribute__((section("tc"), used)) int syncookie_tc(struct __sk_buff *skb) { return 0; }The list of headers to test comes from
tools/testing/selftests/bpf/good_uapi_headers.txt.- selftests/bpf: Known good uapi headers for
test_uapi_headers.pyThe list of uapi headers that could be included alongside
vmlinux.h. The headers are peeked from the following locations:
- <headers-export-dir>/linux/*.h
- <headers-export-dir>/linux/**/*.h
This choice of locations is somewhat arbitrary.- selftests/bpf: script for infer_header_guards.pl testing
The test case for
scripts/infer_header_guards.pl, verifies that header guards can be inferred for all uapi headers.- There is also a patch for dwarves that adds
--header_guards_dboption (see [2]).The
test_uapi_headers.pyis important as it demonstrates the the necessary compiler flags:clang ... \ -D__x86_64__ \ -Xclang -fwchar-type=short \ -Xclang -fno-signed-wchar \ -I{exported_kernel_headers}/include/ \ ...-
-fwchar-type=shortand-fno-signed-wcharhad to be added because BPF target usesintforwchar_tby default and this differs fromvmlinux.hdefinition of the type (at least forx86_64).
-__x86_64__had to be added for uapi headers that includestddef.h(the one that is supplied my CLANG itself), in order to define correct sizes forsize_tandptrdiff_t.
- The{exported_kernel_headers}stands for exported kernel headers directory (the headers obtained bymake headers_installor via distribution package).When it works
---The mechanics described above works for a significant number of UAPI headers. For example, for the test case above I chose the headers from the following locations:
- linux/*.h
- linux/**/*.h
There are 759 such headers and for 677 of them the test described above passes.I excluded the headers from the following sub-directories as potentially not interesting:
asm rdma video xen asm-generic misc scsi drm mtd soundThus saving some time for both discussion and CI but the choice is somewhat arbitrary. If I run
test_uapi_headers.py --test '*'(all headers) test passes for 834 out of 972 headers.When it breaks
---There several scenarios when this mechanics breaks. Specifically I found the following cases:
- When uapi header includes some system header that conflicts withvmlinux.h.
- When uapi header itself conflicts withvmlinux.h.Below are examples for both cases.
Conflict with system headers
----The following uapi headers:
-linux/atmbr2684.h
-linux/bpfilter.h
-linux/gsmmux.h
-linux/icmp.h
-linux/if.h
-linux/if_arp.h
-linux/if_bonding.h
-linux/if_pppox.h
-linux/if_tunnel.h
-linux/ip6_tunnel.h
-linux/llc.h
-linux/mctp.h
-linux/mptcp.h
-linux/netdevice.h
-linux/netfilter/xt_RATEEST.h
-linux/netfilter/xt_hashlimit.h
-linux/netfilter/xt_physdev.h
-linux/netfilter/xt_rateest.h
-linux/netfilter_arp/arp_tables.h
-linux/netfilter_arp/arpt_mangle.h
-linux/netfilter_bridge.h
-linux/netfilter_bridge/ebtables.h
-linux/netfilter_ipv4/ip_tables.h
-linux/netfilter_ipv6/ip6_tables.h
-linux/route.h
-linux/wireless.hInclude the following system header:
-/usr/include/sys/socket.h(all vialinux/if.h)The
sys/socket.hconflicts withvmlinux.hin:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants:SOCK_STREAM,SOCK_DGRAM, ...However, only two types are actually used:
- struct sockaddr
- structsockaddr_storage(used only inlinux/mptcp.h)In '
vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.The only workaround that I see is to:
- define a stubsys/socket.has follows:#ifndef __BPF_SOCKADDR__ #define __BPF_SOCKADDR__ /* For __kernel_sa_family_t */ #include <linux/socket.h> struct sockaddr { __kernel_sa_family_t sa_family; char sa_data[14]; }; #endif- hardcode generation of
__BPF_SOCKADDR__bracket for 'struct sockaddr' invmlinux.h.Another possibility is to move the definition of 'struct sockaddr' from '
kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.Conflict with
vmlinux.h
----Uapi header:
-linux/signal.hConflict with
vmlinux.hin definition of 'struct sigaction'. Defined in:
-vmlinux.h:kernel/include/linux/signal_types.h
- uapi:kernel/arch/x86/include/asm/signal.hUapi headers:
-linux/tipc_sockets_diag.h
-linux/sock_diag.hConflict with
vmlinux.hin definition of 'SOCK_DESTROY'. Defined in:
-vmlinux.h:kernel/include/net/sock.h
- uapi:kernel/include/uapi/linux/sock_diag.h
Constants seem to be unrelated.And so on... I have details for many other headers but omit those for brevity.
In conclusion
---Except from the general feasibility I have a few questions:
- What UAPI headers are the candidates for such use? If there are some interesting headers currently not working with this patch-set some hacks have to be added (e.g. like withlinux/if.h).
- Is it ok to encode header guards as specialBTF_DECL_TAGor should I change the BTF format a bit to save some bytes.Thanks,
Eduard[1] https://reviews.llvm.org/D111307
[clang]__attribute__bpf_dominating_decl
[2] https://lore.kernel.org/dwarves/[email protected]/T/
[RFC dwarves] pahole: Save header guard names when
--header_guards_db is passedEduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testingscripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-) create mode 100755 scripts/infer_header_guards.pl create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
Quoted 9 lines · Eduard Zingerman
Hi BPF community,
AFAIK there is a long standing feature request to use kernel headers alongside
vmlinux.hgenerated bybpftool. For example significant effort was put to add an attributebpf_dominating_decl(see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.
Maybe we should make another attempt to implement bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?
BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.
Quoted 5 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.
[...]
Quoted 39 lines
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testingscripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-) create mode 100755 scripts/infer_header_guards.pl create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py--
2.34.1
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman
Hi BPF community,
AFAIK there is a long standing feature request to use kernel headers alongside
vmlinux.hgenerated bybpftool. For example significant effort was put to add an attributebpf_dominating_decl(see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.
Quoted 3 lines
Maybe we should make another attempt to implement
bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?
I would say it would be very difficult for upstream to agree with bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.
Quoted 53 lines
BTW, I suggest splitting libbpf
btf_dedupandbtf_dumpchanges into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.Quoted 5 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.[...]
Quoted 39 lines
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testingscripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-)create mode 100755
scripts/infer_header_guards.pl
create mode 100644tools/testing/selftests/bpf/good_uapi_headers.txtcreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.ccreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.ccreate mode 100755tools/testing/selftests/bpf/test_uapi_header_guards_infer.shcreate mode 100755tools/testing/selftests/bpf/test_uapi_headers.py--
2.34.1
Quoted 19 lines · Yonghong Song
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman
Hi BPF community,
AFAIK there is a long standing feature request to use kernel headers alongside
vmlinux.hgenerated bybpftool. For example significant effort was put to add an attributebpf_dominating_decl(see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.Quoted 3 lines
Maybe we should make another attempt to implement
bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?
Quoted 7 lines
I would say it would be very difficult for upstream to agree with
bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.
Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like
#if !__is_type_defined(struct abc)
struct abc {
};
#endif
I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?
Quoted 54 lines
Quoted 54 lines
BTW, I suggest splitting libbpf
btf_dedupandbtf_dumpchanges into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.Quoted 5 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.[...]
Quoted 39 lines
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testingscripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-)create mode 100755
scripts/infer_header_guards.pl
create mode 100644tools/testing/selftests/bpf/good_uapi_headers.txtcreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.ccreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.ccreate mode 100755tools/testing/selftests/bpf/test_uapi_header_guards_infer.shcreate mode 100755tools/testing/selftests/bpf/test_uapi_headers.py--
2.34.1
Quoted 36 lines · Andrii Nakryiko
Quoted 19 lines · Yonghong Song
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman
Hi BPF community,
AFAIK there is a long standing feature request to use kernel headers alongside
vmlinux.hgenerated bybpftool. For example significant effort was put to add an attributebpf_dominating_decl(see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.Quoted 3 lines
Maybe we should make another attempt to implement
bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?Quoted 7 lines
I would say it would be very difficult for upstream to agree with
bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like
#if !__is_type_defined(struct abc)
struct abc {
};
#endif
Quoted 5 lines
I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
Let us see whether we can get some upstream agreement or not.
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
BTW, I suggest splitting libbpf
btf_dedupandbtf_dumpchanges into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.Quoted 5 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.[...]
Quoted 39 lines
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testingscripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-)create mode 100755
scripts/infer_header_guards.pl
create mode 100644tools/testing/selftests/bpf/good_uapi_headers.txtcreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.ccreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.ccreate mode 100755tools/testing/selftests/bpf/test_uapi_header_guards_infer.shcreate mode 100755tools/testing/selftests/bpf/test_uapi_headers.py--
2.34.1
Quoted 46 lines · Yonghong Song
Quoted 36 lines · Andrii Nakryiko
Quoted 19 lines · Yonghong Song
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman
Hi BPF community,
AFAIK there is a long standing feature request to use kernel headers alongside
vmlinux.hgenerated bybpftool. For example significant effort was put to add an attributebpf_dominating_decl(see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.Quoted 3 lines
Maybe we should make another attempt to implement
bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?Quoted 7 lines
I would say it would be very difficult for upstream to agree with
bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like
#if !__is_type_defined(struct abc)
struct abc {
};
#endifQuoted 5 lines
I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?
Quoted 7 lines
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
Let us see whether we can get some upstream agreement or not.
Thanks for starting the conversation! I'll be following along.
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
BTW, I suggest splitting libbpf
btf_dedupandbtf_dumpchanges into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.Quoted 5 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.[...]
Quoted 39 lines
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testingscripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-)create mode 100755
scripts/infer_header_guards.pl
create mode 100644tools/testing/selftests/bpf/good_uapi_headers.txtcreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.ccreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.ccreate mode 100755tools/testing/selftests/bpf/test_uapi_header_guards_infer.shcreate mode 100755tools/testing/selftests/bpf/test_uapi_headers.py--
2.34.1
Quoted 50 lines · Andrii Nakryiko
Quoted 49 lines · Yonghong Song
Quoted 36 lines · Andrii Nakryiko
Quoted 19 lines · Yonghong Song
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman
Hi BPF community,
AFAIK there is a long standing feature request to use kernel headers alongside
vmlinux.hgenerated bybpftool. For example significant effort was put to add an attributebpf_dominating_decl(see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.Quoted 3 lines
Maybe we should make another attempt to implement
bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?Quoted 7 lines
I would say it would be very difficult for upstream to agree with
bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like
#if !__is_type_defined(struct abc)
struct abc {
};
#endifQuoted 5 lines
I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
Quoted 7 lines
Quoted 4 lines
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
Let us see whether we can get some upstream agreement or not.
Thanks for starting the conversation! I'll be following along.
I think this sort of approach assumes that vmlinux.h is included after any uapi headers, and would guard type definitions with
#if type_is_defined(foo)
struct foo {
};
#endif
...is that right? My concern is that the vmlinux.h definitions have the CO-RE attributes. From a BPF perspective, would we like the vmlinux.h definitions to dominate over UAPI definitions do you think, or does it matter?
I was wondering if there might be yet another way to crack this; if we did want the vmlinux.h type definitions to be authoritative because they have the preserve access index attribute, and because bpftool knows all vmlinux types, it could use that info to selectively redefine those type names such that we avoid name clashes when later including UAPI headers. Something like
#ifdef __VMLINUX_H__
//usual vmlinux.h type definitions
#endif /* __VMLINUX_H__ */
#ifdef __VMLINUX_ALIAS__
if !defined(timespec64)
#define timespec64 __VMLINUX_ALIAS__timespec64
#endif
// rest of the types define aliases here
#undef __VMLINUX_ALIAS__
#else /* unalias */
#if defined(timespec64)
#undef timespec64
#endif
// rest of types undef aliases here
#endif /* __VMLINUX_ALIAS__ */
Then the consumer does this:
#define __VMLINUX_ALIAS__
#include "vmlinux.h"
// include uapi headers
#include "vmlinux.h"
(the latter include of vmlinux.h is needed to undef all the type aliases)
I tried hacking up bpftool to support this aliasing scheme and while it is kind of hacky it does seem to work, aside from some issues with IPPROTO_* definitions - for the enumerated IPPROTO_ values linux/in.h does this:
enum {
IPPROTO_IP = 0, /* Dummy protocol for TCP */ #define IPPROTO_IP IPPROTO_IP IPPROTO_ICMP = 1, /* Internet Control Message Protocol */ #define IPPROTO_ICMP IPPROTO_ICMP
...so our enum value definitions for IPPROTO_ values clash with the above definitions. These could be individually ifdef-guarded if needed though I think.
I can send the proof-of-concept patch if it would help, I just wanted to check in case that might be a workable path too, since it just requires changes to bpftool (and changes to in.h).
Thanks!
Alan
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
BTW, I suggest splitting libbpf
btf_dedupandbtf_dumpchanges into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.Quoted 5 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.[...]
Quoted 39 lines
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testingscripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-)create mode 100755
scripts/infer_header_guards.pl
create mode 100644tools/testing/selftests/bpf/good_uapi_headers.txtcreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.ccreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.ccreate mode 100755tools/testing/selftests/bpf/test_uapi_header_guards_infer.shcreate mode 100755tools/testing/selftests/bpf/test_uapi_headers.py--
2.34.1
Quoted 5 lines · Alan Maguire
Quoted 5 lines
Quoted 5 lines
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
That would be a great clang feature.
Thanks Yonghong!
Quoted 14 lines
Quoted 6 lines
Quoted 3 lines
Let us see whether we can get some upstream agreement or not.
Thanks for starting the conversation! I'll be following along.
I think this sort of approach assumes that
vmlinux.his included after any uapi headers, and would guard type definitions with#if
type_is_defined(foo)
struct foo {
Quoted 7 lines
};
#endif...is that right? My concern is that the
vmlinux.hdefinitions have the CO-RE attributes. From a BPF perspective, would we like thevmlinux.hdefinitions to dominate over UAPI definitions do you think, or does it matter?
I think it's totally fine to require #include "vmlinux.h" to be last. The attr(preserve_access_index) is only useful for kernel internal structs. uapi structs don't need it.
Quoted 26 lines
I was wondering if there might be yet another way to crack this; if we did want the
vmlinux.htype definitions to be authoritative because they have the preserve access index attribute, and because bpftool knows all vmlinux types, it could use that info to selectively redefine those type names such that we avoid name clashes when later including UAPI headers. Something like#ifdef
__VMLINUX_H__
//usualvmlinux.htype definitions
#endif /*__VMLINUX_H__*/#ifdef
__VMLINUX_ALIAS__
if !defined(timespec64)
#define timespec64__VMLINUX_ALIAS__timespec64
#endif
// rest of the types define aliases here
#undef__VMLINUX_ALIAS__
#else /* unalias */
#if defined(timespec64)
#undef timespec64
#endif
// rest of types undef aliases here
#endif /*__VMLINUX_ALIAS__*/
Quoted 8 lines
Then the consumer does this:
#define
__VMLINUX_ALIAS__
#include "vmlinux.h"
// include uapi headers
#include "vmlinux.h"(the latter include of
vmlinux.his needed to undef all the type aliases)
Sounds like a bunch of complexity for the use case that is not clear to me.
Quoted 7 lines
I tried hacking up bpftool to support this aliasing scheme and while it is kind of hacky it does seem to work, aside from some issues with IPPROTO_* definitions - for the enumerated
IPPROTO_valueslinux/in.hdoes this:enum {
Quoted 8 lines
IPPROTO_IP = 0, /* Dummy protocol for TCP */ #define IPPROTO_IP IPPROTO_IP IPPROTO_ICMP = 1, /* Internet Control Message Protocol */ #define IPPROTO_ICMP IPPROTO_ICMP...so our enum value definitions for
IPPROTO_values clash with the above definitions. These could be individually ifdef-guarded if needed though I think.
Including vmlinux.h last won't have this enum conflicts, right?
Quoted 3 lines
I can send the proof-of-concept patch if it would help, I just wanted to check in case that might be a workable path too, since it just requires changes to bpftool (and changes to
in.h).
I think changing the uapi header like in.h is no-go. Touching anything in uapi is too much of a risk.
Eduard Zingerman
maintainerQuoted 69 lines · Alexei Starovoitov
Quoted 5 lines · Alan Maguire
Quoted 5 lines
Quoted 5 lines
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
That would be a great clang feature.
Thanks Yonghong!Quoted 14 lines
Quoted 6 lines
Quoted 3 lines
Let us see whether we can get some upstream agreement or not.
Thanks for starting the conversation! I'll be following along.
I think this sort of approach assumes that
vmlinux.his included after any uapi headers, and would guard type definitions with#if
type_is_defined(foo)
struct foo {Quoted 7 lines
};
#endif...is that right? My concern is that the
vmlinux.hdefinitions have the CO-RE attributes. From a BPF perspective, would we like thevmlinux.hdefinitions to dominate over UAPI definitions do you think, or does it matter?I think it's totally fine to require #include "
vmlinux.h" to be last. Theattr(preserve_access_index)is only useful for kernel internal structs. uapi structs don't need it.Quoted 33 lines
I was wondering if there might be yet another way to crack this; if we did want the
vmlinux.htype definitions to be authoritative because they have the preserve access index attribute, and because bpftool knows all vmlinux types, it could use that info to selectively redefine those type names such that we avoid name clashes when later including UAPI headers. Something like#ifdef
__VMLINUX_H__
//usualvmlinux.htype definitions
#endif /*__VMLINUX_H__*/#ifdef
__VMLINUX_ALIAS__
if !defined(timespec64)
#define timespec64__VMLINUX_ALIAS__timespec64
#endif
// rest of the types define aliases here
#undef__VMLINUX_ALIAS__
#else /* unalias */
#if defined(timespec64)
#undef timespec64
#endif
// rest of types undef aliases here
#endif /*__VMLINUX_ALIAS__*/Then the consumer does this:
#define
__VMLINUX_ALIAS__
#include "vmlinux.h"
// include uapi headers
#include "vmlinux.h"
Quoted 4 lines
Quoted 1 line
(the latter include of
vmlinux.his needed to undef all the type aliases)Sounds like a bunch of complexity for the use case that is not clear to me.
Well, my RFC is not shy of complexity :)
What Alan suggests should solve the confilicts described in [1] or any other confilicts of such kind.
[1] https://lore.kernel.org/bpf/[email protected]/
Quoted 24 lines
Quoted 7 lines
I tried hacking up bpftool to support this aliasing scheme and while it is kind of hacky it does seem to work, aside from some issues with IPPROTO_* definitions - for the enumerated
IPPROTO_valueslinux/in.hdoes this:enum {
Quoted 8 lines
IPPROTO_IP = 0, /* Dummy protocol for TCP */ #define IPPROTO_IP IPPROTO_IP IPPROTO_ICMP = 1, /* Internet Control Message Protocol */ #define IPPROTO_ICMP IPPROTO_ICMP...so our enum value definitions for
IPPROTO_values clash with the above definitions. These could be individually ifdef-guarded if needed though I think.Including
vmlinux.hlast won't have this enum conflicts, right?Quoted 3 lines
I can send the proof-of-concept patch if it would help, I just wanted to check in case that might be a workable path too, since it just requires changes to bpftool (and changes to
in.h).I think changing the uapi header like
in.his no-go. Touching anything in uapi is too much of a risk.
Quoted 73 lines · Eduard Zingerman
Quoted 71 lines · Alexei Starovoitov
Quoted 5 lines · Alan Maguire
Quoted 5 lines
Quoted 5 lines
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
That would be a great clang feature.
Thanks Yonghong!Quoted 14 lines
Quoted 6 lines
Quoted 3 lines
Let us see whether we can get some upstream agreement or not.
Thanks for starting the conversation! I'll be following along.
I think this sort of approach assumes that
vmlinux.his included after any uapi headers, and would guard type definitions with#if
type_is_defined(foo)
struct foo {Quoted 7 lines
};
#endif...is that right? My concern is that the
vmlinux.hdefinitions have the CO-RE attributes. From a BPF perspective, would we like thevmlinux.hdefinitions to dominate over UAPI definitions do you think, or does it matter?I think it's totally fine to require #include "
vmlinux.h" to be last. Theattr(preserve_access_index)is only useful for kernel internal structs. uapi structs don't need it.Quoted 34 lines
I was wondering if there might be yet another way to crack this; if we did want the
vmlinux.htype definitions to be authoritative because they have the preserve access index attribute, and because bpftool knows all vmlinux types, it could use that info to selectively redefine those type names such that we avoid name clashes when later including UAPI headers. Something like#ifdef
__VMLINUX_H__
//usualvmlinux.htype definitions
#endif /*__VMLINUX_H__*/#ifdef
__VMLINUX_ALIAS__
if !defined(timespec64)
#define timespec64__VMLINUX_ALIAS__timespec64
#endif
// rest of the types define aliases here
#undef__VMLINUX_ALIAS__
#else /* unalias */
#if defined(timespec64)
#undef timespec64
#endif
// rest of types undef aliases here
#endif /*__VMLINUX_ALIAS__*/Then the consumer does this:
#define
__VMLINUX_ALIAS__
#include "vmlinux.h"
// include uapi headers
#include "vmlinux.h"(the latter include of
vmlinux.his needed to undef all the type aliases)
Quoted 8 lines
Quoted 2 lines
Sounds like a bunch of complexity for the use case that is not clear to me.
Well, my RFC is not shy of complexity :)
What Alan suggests should solve the confilicts described in [1] or any other confilicts of such kind.
I don't quite see how renaming all types in the 1st vmlinux.h will help with name collisions inside enum {}.
The enums will conflict with 2nd vmlinux.h too. Unless the proposal is to rename them as well,
but then what's the point?
Eduard Zingerman
maintainerQuoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line
[...]
Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like
#if !__is_type_defined(struct abc)
struct abc {
};
#endifQuoted 5 lines
I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?
Quoted 6 lines
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
Let us see whether we can get some upstream agreement or not.
I did a small investigation of this feature.
The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.
The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of __is_type_defined construct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.
However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.
This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:
$ cat test.c
struct foo { int x; };
#if __is_type_defined(foo)
const int x = 1;
#else
const int x = 2;
#endif$ clang -cc1 -ast-print test.c -o -
struct foo {
int x;
};
const int x = 1;$ clang -E test.c -o -
# ... some line directives ...
struct foo { int x; };
const int x = 2;Note that __is_type_defined is computed to different value in the first and second calls. This is so because semantic analysis (AST, symbol table) is not done for -E.
It also breaks that C11 standard which clearly separates pre-processing and semantic analysis phases, see [2] 5.1.1.2.
So, my conclusion is as follows: this is technically possible in clang but has no chance to reach llvm upstream.
Thanks,
Eduard
[1] https://github.com/llvm/llvm-project/compare/main...eddyz87:llvm-project:is-type-defined-experiment
[2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
BTW, I suggest splitting libbpf
btf_dedupandbtf_dumpchanges into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.Quoted 5 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.[...]
Quoted 39 lines
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testingscripts/infer_header_guards.pl | 191 +++++ scripts/link-vmlinux.sh | 13 +- tools/bpf/bpftool/btf.c | 4 +- tools/lib/bpf/btf.c | 178 ++++- tools/lib/bpf/btf.h | 7 +- tools/lib/bpf/btf_dump.c | 232 +++++- .../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++ tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++ .../selftests/bpf/prog_tests/btf_dump.c | 11 +- .../bpf/progs/btf_dump_test_case_decl_tag.c | 39 + .../progs/btf_dump_test_case_header_guards.c | 94 +++ .../bpf/test_uapi_header_guards_infer.sh | 33 + .../selftests/bpf/test_uapi_headers.py | 197 +++++ 13 files changed, 1816 insertions(+), 12 deletions(-)create mode 100755
scripts/infer_header_guards.pl
create mode 100644tools/testing/selftests/bpf/good_uapi_headers.txtcreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.ccreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.ccreate mode 100755tools/testing/selftests/bpf/test_uapi_header_guards_infer.shcreate mode 100755tools/testing/selftests/bpf/test_uapi_headers.py--
2.34.1
Quoted 64 lines · Eduard Zingerman
Quoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line
[...]
Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like
#if !__is_type_defined(struct abc)
struct abc {
};
#endifQuoted 5 lines
I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?
Quoted 6 lines
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
Let us see whether we can get some upstream agreement or not.
I did a small investigation of this feature.
The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.
The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of__is_type_definedconstruct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.
This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:
$ cat test.cstruct foo { int x; }; #if __is_type_defined(foo) const int x = 1; #else const int x = 2; #endif$ clang -cc1 -ast-print test.c -o -struct foo { int x; }; const int x = 1;
Quoted 5 lines
$ clang -E test.c -o -# ... some line directives ... struct foo { int x; }; const int x = 2;
Is it any chance '-E' could output the same one as '-cc1 -ast-print'? That is, even with -E we could do some semantics analysis as well, using either current clang semantics analysis or creating an minimal version of sema analysis in preprocessor itself?
Quoted 75 lines
Note that
__is_type_definedis computed to different value in the first and second calls. This is so because semantic analysis (AST, symbol table) is not done for -E.It also breaks that C11 standard which clearly separates pre-processing and semantic analysis phases, see [2] 5.1.1.2.
So, my conclusion is as follows: this is technically possible in clang but has no chance to reach llvm upstream.
Thanks,
Eduard[1] https://github.com/llvm/llvm-project/compare/main...eddyz87:llvm-project:is-type-defined-experiment
[2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdfQuoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
BTW, I suggest splitting libbpf
btf_dedupandbtf_dumpchanges into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.Quoted 5 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.[...]
Quoted 39 lines
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testing
scripts/infer_header_guards.pl| 191 +++++
scripts/link-vmlinux.sh| 13 +-
tools/bpf/bpftool/btf.c| 4 +-
tools/lib/bpf/btf.c| 178 ++++-
tools/lib/bpf/btf.h| 7 +-
tools/lib/bpf/btf_dump.c| 232 +++++-
.../selftests/bpf/good_uapi_headers.txt| 677 ++++++++++++++++++
tools/testing/selftests/bpf/prog_tests/btf.c| 152 ++++
.../selftests/bpf/prog_tests/btf_dump.c| 11 +-
.../bpf/progs/btf_dump_test_case_decl_tag.c| 39 +
.../progs/btf_dump_test_case_header_guards.c| 94 +++
.../bpf/test_uapi_header_guards_infer.sh| 33 +
.../selftests/bpf/test_uapi_headers.py| 197 +++++
13 files changed, 1816 insertions(+), 12 deletions(-)
create mode 100755scripts/infer_header_guards.pl
create mode 100644tools/testing/selftests/bpf/good_uapi_headers.txtcreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.ccreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.ccreate mode 100755tools/testing/selftests/bpf/test_uapi_header_guards_infer.shcreate mode 100755tools/testing/selftests/bpf/test_uapi_headers.py--
2.34.1
Eduard Zingerman
maintainerQuoted 68 lines · Yonghong Song
Quoted 66 lines · Eduard Zingerman
Quoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line
[...]
Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like
#if !__is_type_defined(struct abc)
struct abc {
};
#endifQuoted 5 lines
I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?
Quoted 6 lines
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
Let us see whether we can get some upstream agreement or not.
I did a small investigation of this feature.
The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.
The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of__is_type_definedconstruct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.
This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:
$ cat test.cstruct foo { int x; }; #if __is_type_defined(foo) const int x = 1; #else const int x = 2; #endif$ clang -cc1 -ast-print test.c -o -struct foo { int x; }; const int x = 1;$ clang -E test.c -o -
Quoted 8 lines
Quoted 3 lines
# ... some line directives ... struct foo { int x; }; const int x = 2;Is it any chance '-E' could output the same one as '-cc1 -ast-print'? That is, even with -E we could do some semantics analysis as well, using either current clang semantics analysis or creating an minimal version of sema analysis in preprocessor itself?
Sema drives consumption of tokens from Preprocessor. Calls to Preprocessor are done on a parsing recursive descent. Extracting a stream of tokens would require an incremental parser instead.
A minimal version of such parser is possible to implement for C. It might be the case that matching open / closing braces and identifiers following 'struct' / 'union' / 'enum' keywords might be almost sufficient but I need to try to be sure (e.g. it is more complex for 'typedef').
I can work on it but I don't think there is a chance to upstream this work.
Thanks,
Eduard
Quoted 76 lines
Quoted 76 lines
Note that
__is_type_definedis computed to different value in the first and second calls. This is so because semantic analysis (AST, symbol table) is not done for -E.It also breaks that C11 standard which clearly separates pre-processing and semantic analysis phases, see [2] 5.1.1.2.
So, my conclusion is as follows: this is technically possible in clang but has no chance to reach llvm upstream.
Thanks,
Eduard[1] https://github.com/llvm/llvm-project/compare/main...eddyz87:llvm-project:is-type-defined-experiment
[2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdfQuoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
BTW, I suggest splitting libbpf
btf_dedupandbtf_dumpchanges into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.Quoted 5 lines
After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding
#ifndef FOO_H ... #endifguards in the generatedvmlinux.h.[...]
Quoted 39 lines
Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support forBTF_DECL_TAGdump in C format selftests/bpf: Tests forBTF_DECL_TAGdump in C format
libbpf: Header guards for selected data structures invmlinux.hselftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types frominclude/uapi/*.hin kernel BTF selftests/bpf: Script to verify uapi headers usage withvmlinux.hselftests/bpf: Known good uapi headers fortest_uapi_headers.pyselftests/bpf: script for infer_header_guards.pl testing
scripts/infer_header_guards.pl| 191 +++++
scripts/link-vmlinux.sh| 13 +-
tools/bpf/bpftool/btf.c| 4 +-
tools/lib/bpf/btf.c| 178 ++++-
tools/lib/bpf/btf.h| 7 +-
tools/lib/bpf/btf_dump.c| 232 +++++-
.../selftests/bpf/good_uapi_headers.txt| 677 ++++++++++++++++++
tools/testing/selftests/bpf/prog_tests/btf.c| 152 ++++
.../selftests/bpf/prog_tests/btf_dump.c| 11 +-
.../bpf/progs/btf_dump_test_case_decl_tag.c| 39 +
.../progs/btf_dump_test_case_header_guards.c| 94 +++
.../bpf/test_uapi_header_guards_infer.sh| 33 +
.../selftests/bpf/test_uapi_headers.py| 197 +++++
13 files changed, 1816 insertions(+), 12 deletions(-)
create mode 100755scripts/infer_header_guards.pl
create mode 100644tools/testing/selftests/bpf/good_uapi_headers.txtcreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.ccreate mode 100644tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.ccreate mode 100755tools/testing/selftests/bpf/test_uapi_header_guards_infer.shcreate mode 100755tools/testing/selftests/bpf/test_uapi_headers.py--
2.34.1
Quoted 83 lines · Eduard Zingerman
Quoted 68 lines · Yonghong Song
Quoted 66 lines · Eduard Zingerman
Quoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line
[...]
Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like
#if !__is_type_defined(struct abc)
struct abc {
};
#endifQuoted 5 lines
I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?
Quoted 6 lines
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
Let us see whether we can get some upstream agreement or not.
I did a small investigation of this feature.
The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.
The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of__is_type_definedconstruct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.
This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:
$ cat test.cstruct foo { int x; }; #if __is_type_defined(foo) const int x = 1; #else const int x = 2; #endif$ clang -cc1 -ast-print test.c -o -struct foo { int x; }; const int x = 1;$ clang -E test.c -o -Quoted 8 lines
Quoted 3 lines
# ... some line directives ... struct foo { int x; }; const int x = 2;Is it any chance '-E' could output the same one as '-cc1 -ast-print'? That is, even with -E we could do some semantics analysis as well, using either current clang semantics analysis or creating an minimal version of sema analysis in preprocessor itself?
Sema drives consumption of tokens from Preprocessor. Calls to Preprocessor are done on a parsing recursive descent. Extracting a stream of tokens would require an incremental parser instead.
Quoted 7 lines
A minimal version of such parser is possible to implement for C. It might be the case that matching open / closing braces and identifiers following 'struct' / 'union' / 'enum' keywords might be almost sufficient but I need to try to be sure (e.g. it is more complex for 'typedef').
I can work on it but I don't think there is a chance to upstream this work.
Right. It's going to be C only.
C++ with namespaces and nested class decls won't work with simple type parser.
On the other side if we're asking preprocessor to look for 'struct foo' and remember that 'foo' is a type maybe we can add a regex-search instead?
It would be a bit more generic and will work for basic union/struct foo definition?
Something like instead of:
#if __is_type_defined(foo)
use:
#if regex(struct[\t]+foo)
enums are harder in this approach, but higher chance to land?
regex() would mean "search for this pattern in the file until this line.
Or some other preprocessor "language" tricks?
For example:
The preprocessor would grep for 'struct *' in a single line while processing a file and emit #define __secret_prefix_##$1 where $1 would be a capture from "single line regex". Then later in the same file instead of:
#if __is_type_defined(foo)
use:
#ifdef __secret_prefix_foo
This "single line regex" may look like:
#if regex_in_any_later_line(struct[\t]+[a-zA-Z_]+) define __secret_prefix_$2
Eduard Zingerman
maintainerQuoted 106 lines · Alexei Starovoitov
Quoted 83 lines · Eduard Zingerman
Quoted 68 lines · Yonghong Song
Quoted 66 lines · Eduard Zingerman
Quoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line
[...]
Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like
#if !__is_type_defined(struct abc)
struct abc {
};
#endifQuoted 5 lines
I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?
Quoted 6 lines
Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse
https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268
Let us see whether we can get some upstream agreement or not.
I did a small investigation of this feature.
The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.
The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of__is_type_definedconstruct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.
This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:
$ cat test.cstruct foo { int x; }; #if __is_type_defined(foo) const int x = 1; #else const int x = 2; #endif$ clang -cc1 -ast-print test.c -o -struct foo { int x; }; const int x = 1;$ clang -E test.c -o -Quoted 8 lines
Quoted 3 lines
# ... some line directives ... struct foo { int x; }; const int x = 2;Is it any chance '-E' could output the same one as '-cc1 -ast-print'? That is, even with -E we could do some semantics analysis as well, using either current clang semantics analysis or creating an minimal version of sema analysis in preprocessor itself?
Sema drives consumption of tokens from Preprocessor. Calls to Preprocessor are done on a parsing recursive descent. Extracting a stream of tokens would require an incremental parser instead.
Quoted 7 lines
A minimal version of such parser is possible to implement for C. It might be the case that matching open / closing braces and identifiers following 'struct' / 'union' / 'enum' keywords might be almost sufficient but I need to try to be sure (e.g. it is more complex for 'typedef').
I can work on it but I don't think there is a chance to upstream this work.
Right. It's going to be C only.
C++ with namespaces and nested class decls won't work with simple type parser.On the other side if we're asking preprocessor to look for 'struct foo' and remember that 'foo' is a type maybe we can add a regex-search instead?
It would be a bit more generic and will work for basic union/struct foo definition?
Something like instead of:
#if__is_type_defined(foo)
use:
#ifregex(struct[\t]+foo)
Quoted 6 lines
enums are harder in this approach, but higher chance to land?
regex()would mean "search for this pattern in the file until this line.Or some other preprocessor "language" tricks?
I talked to Yonhong today and he suggests to investigate whether pre-processor changes could be made BPF target specific. E.g. there are extension points in the clang pre-processor right now but those for tooling. There might be a way to extend this mechanism to allow target specific pre-processor behavior. I'll take a look and write another email here.
Quoted 11 lines
For example:
The preprocessor would grep for 'struct *' in a single line while processing a file and emit #define __secret_prefix_##$1 where $1 would be a capture from "single line regex". Then later in the same file instead of:
#if__is_type_defined(foo)
use:
#ifdef__secret_prefix_fooThis "single line regex" may look like:
#ifregex_in_any_later_line(struct[\t]+[a-zA-Z_]+)define __secret_prefix_$2