Skip to contentSkip to conversation

Use uapi kernel headers with vmlinux.h

RFC In review Eduard Zingerman 6 participants 46 messages Oct 25, 2022 – Nov 16, 2022

Subsystems (6)
Kbuild Kselftest BPF BPF selftests +2 more subsystems
lore
Share & tools
lore

Eduard Zingerman

author maintainer
#0 lore raw

cover Use uapi kernel headers with vmlinux.h

Hi BPF community,

AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

The basic idea
---

The goal of the patch set is to allow usage of header files from include/uapi alongside vmlinux.h as follows:

  #include <uapi/linux/tcp.h>
  #include "vmlinux.h"

This goal is achieved by adding #ifndef ... #endif guards in vmlinux.h around definitions that originate from the include/uapi headers. The guards emitted match the guards used in the original headers. E.g. as follows:

include/uapi/linux/tcp.h:

#ifndef _UAPI_LINUX_TCP_H
#define _UAPI_LINUX_TCP_H
...

  union tcp_word_hdr {
	struct tcphdr hdr;
	__be32        words[5];

};
...
#endif /* _UAPI_LINUX_TCP_H */

vmlinux.h:

  ...
  #ifndef _UAPI_LINUX_TCP_H
  union tcp_word_hdr {
	struct tcphdr hdr;
	__be32 words[5];

};

  #endif /* _UAPI_LINUX_TCP_H */
  ...

To get to this state the following steps are necessary:
- "header guard" name should be identified for each header file;
- the correspondence between data type and it's header guard has to be encoded in BTF;
- bpftool should be adjusted to emit #ifndef FOO_H ... #endif brackets.

It is not possible to identify header guard names for all uapi headers basing only on the file name. However a simple script could devised to identify the guards basing on the file name and it's content. Thus it is possible to obtain the list of header names with corresponding header guards.

The correspondence between type and it's declaration file (header) is available in DWARF as DW_AT_decl_file attribute. The DW_AT_decl_file can be matched with the list of header guards described above to obtain the header guard name for a specific type.

The pahole generates BTF using DWARF. It is possible to modify pahole to accept the header guards list as an additional parameter and to encode the header guard names in BTF.

Implementation details
---

Present patch-set implements these ideas as follows:
- A parameter --header_guards_db is added to pahole. If present it points to a file with a list of <header> <guard> records.
- pahole uses DWARF DW_AT_decl_file value to lookup the header guard for each type emitted to BTF. If header guard is present it is encoded alongside the type.
- Header guards are encoded in BTF as BTF_DECL_TAG records with a special prefix. The prefix "header_guard:" is added to a value of such tags. (Here BTF_DECL_TAG is used to avoid BTF binary format changes).
- A special script infer_header_guards.pl is added as a part of kbuild, it can infer header guard names for each UAPI header basing on the header content.
- This script is invoked from link-vmlinux.sh prior to BTF generation during kernel build. The output of the script is saved to a file, the file is passed to pahole as --header_guards_db parameter.
- libbpf is modified to aggregate BTF_DECL_TAG records for each type and to emit #ifndef FOO_H ... #endif brackets when "header_guard:" tag is present for a type.

Details for each patch in a set:
- libbpf: Deduplicate unambigous standalone forward declarations
- selftests/bpf: Tests for standalone forward BTF declarations deduplication

There is a small number (63 for defconfig) of forward declarations that are not de-duplicated with the main type declaration under certain conditions. This hinders the header guard brackets generation. This patch addresses this de-duplication issue.

- libbpf: Support for BTF_DECL_TAG dump in C format
- selftests/bpf: Tests for BTF_DECL_TAG dump in C format

Currently libbpf does not process BTF_DECL_TAG when btf is dumped in C format. This patch adds a hash table matching btf type ids with a list of decl tags to the struct btf_dump.
The btf_dump_emit_decl_tags is not necessary for the overall patch-set to function but simplifies testing a bit.

- libbpf: Header guards for selected data structures in vmlinux.h
- selftests/bpf: Tests for header guards printing in BTF dump

Adds option emit_header_guards to struct btf_dump_opts. When enabled the btf_dump__dump_type prints #ifndef ... #endif brackets around types for which header guard information is present in BTF.

- bpftool: Enable header guards generation

Unconditionally enables emit_header_guards for BTF dump in C format.

- kbuild: Script to infer header guard values for uapi headers
- kbuild: Header guards for types from include/uapi/*.h in kernel BTF

Adds scripts/infer_header_guards.pl and integrates it with link-vmlinux.sh.

- selftests/bpf: Script to verify uapi headers usage with vmlinux.h

Adds a script test_uapi_headers.py that tests header guards with vmlinux.h by compiling a simple C snippet. The snippet looks as follows:

    #include <some_uapi_header.h>
    #include "vmlinux.h"
  
    __attribute__((section("tc"), used))
    int syncookie_tc(struct __sk_buff *skb) { return 0; }

The list of headers to test comes from tools/testing/selftests/bpf/good_uapi_headers.txt.

- selftests/bpf: Known good uapi headers for test_uapi_headers.py

The list of uapi headers that could be included alongside vmlinux.h. The headers are peeked from the following locations:
- <headers-export-dir>/linux/*.h
- <headers-export-dir>/linux/**/*.h
This choice of locations is somewhat arbitrary.

- selftests/bpf: script for infer_header_guards.pl testing

The test case for scripts/infer_header_guards.pl, verifies that header guards can be inferred for all uapi headers.

- There is also a patch for dwarves that adds --header_guards_db option (see [2]).

The test_uapi_headers.py is important as it demonstrates the the necessary compiler flags:

clang ...                                  \
      -D__x86_64__                         \
      -Xclang -fwchar-type=short           \
      -Xclang -fno-signed-wchar            \
      -I{exported_kernel_headers}/include/ \
      ...

- -fwchar-type=short and -fno-signed-wchar had to be added because BPF target uses int for wchar_t by default and this differs from vmlinux.h definition of the type (at least for x86_64).
- __x86_64__ had to be added for uapi headers that include stddef.h (the one that is supplied my CLANG itself), in order to define correct sizes for size_t and ptrdiff_t.
- The {exported_kernel_headers} stands for exported kernel headers directory (the headers obtained by make headers_install or via distribution package).

When it works
---

The mechanics described above works for a significant number of UAPI headers. For example, for the test case above I chose the headers from the following locations:
- linux/*.h
- linux/**/*.h
There are 759 such headers and for 677 of them the test described above passes.

I excluded the headers from the following sub-directories as potentially not interesting:

  asm          rdma   video xen
  asm-generic  misc   scsi
  drm          mtd    sound

Thus saving some time for both discussion and CI but the choice is somewhat arbitrary. If I run test_uapi_headers.py --test '*' (all headers) test passes for 834 out of 972 headers.

When it breaks
---

There several scenarios when this mechanics breaks. Specifically I found the following cases:
- When uapi header includes some system header that conflicts with vmlinux.h.
- When uapi header itself conflicts with vmlinux.h.

Below are examples for both cases.

Conflict with system headers
----

The following uapi headers:
- linux/atmbr2684.h
- linux/bpfilter.h
- linux/gsmmux.h
- linux/icmp.h
- linux/if.h
- linux/if_arp.h
- linux/if_bonding.h
- linux/if_pppox.h
- linux/if_tunnel.h
- linux/ip6_tunnel.h
- linux/llc.h
- linux/mctp.h
- linux/mptcp.h
- linux/netdevice.h
- linux/netfilter/xt_RATEEST.h
- linux/netfilter/xt_hashlimit.h
- linux/netfilter/xt_physdev.h
- linux/netfilter/xt_rateest.h
- linux/netfilter_arp/arp_tables.h
- linux/netfilter_arp/arpt_mangle.h
- linux/netfilter_bridge.h
- linux/netfilter_bridge/ebtables.h
- linux/netfilter_ipv4/ip_tables.h
- linux/netfilter_ipv6/ip6_tables.h
- linux/route.h
- linux/wireless.h

Include the following system header:
- /usr/include/sys/socket.h (all via linux/if.h)

The sys/socket.h conflicts with vmlinux.h in:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants: SOCK_STREAM, SOCK_DGRAM, ...

However, only two types are actually used:
- struct sockaddr
- struct sockaddr_storage (used only in linux/mptcp.h)

In 'vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.

The only workaround that I see is to:
- define a stub sys/socket.h as follows:

    #ifndef __BPF_SOCKADDR__
    #define __BPF_SOCKADDR__
    
    /* For __kernel_sa_family_t */
    #include <linux/socket.h>
    
    struct sockaddr {
        __kernel_sa_family_t sa_family;
        char sa_data[14];
    };
    
    #endif

- hardcode generation of __BPF_SOCKADDR__ bracket for 'struct sockaddr' in vmlinux.h.

Another possibility is to move the definition of 'struct sockaddr' from 'kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.

Conflict with vmlinux.h
----

Uapi header:
- linux/signal.h

Conflict with vmlinux.h in definition of 'struct sigaction'. Defined in:
- vmlinux.h: kernel/include/linux/signal_types.h
- uapi: kernel/arch/x86/include/asm/signal.h

Uapi headers:
- linux/tipc_sockets_diag.h
- linux/sock_diag.h

Conflict with vmlinux.h in definition of 'SOCK_DESTROY'. Defined in:
- vmlinux.h: kernel/include/net/sock.h
- uapi: kernel/include/uapi/linux/sock_diag.h
Constants seem to be unrelated.

And so on... I have details for many other headers but omit those for brevity.

In conclusion
---

Except from the general feasibility I have a few questions:
- What UAPI headers are the candidates for such use? If there are some interesting headers currently not working with this patch-set some hacks have to be added (e.g. like with linux/if.h).
- Is it ok to encode header guards as special BTF_DECL_TAG or should I change the BTF format a bit to save some bytes.

Thanks,
Eduard

[1] https://reviews.llvm.org/D111307
[clang] __attribute__ bpf_dominating_decl
[2] https://lore.kernel.org/dwarves/[email protected]/T/
[RFC dwarves] pahole: Save header guard names when
--header_guards_db is passed

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

 scripts/infer_header_guards.pl                | 191 +++++
 scripts/link-vmlinux.sh                       |  13 +-
 tools/bpf/bpftool/btf.c                       |   4 +-
 tools/lib/bpf/btf.c                           | 178 ++++-
 tools/lib/bpf/btf.h                           |   7 +-
 tools/lib/bpf/btf_dump.c                      | 232 +++++-
 .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
 tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
 .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
 .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
 .../progs/btf_dump_test_case_header_guards.c  |  94 +++
 .../bpf/test_uapi_header_guards_infer.sh      |  33 +
 .../selftests/bpf/test_uapi_headers.py        | 197 +++++
 13 files changed, 1816 insertions(+), 12 deletions(-)
 create mode 100755 scripts/infer_header_guards.pl
 create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c
 create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh
 create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
Signature · 1 line
2.34.1
author maintainer
#1 lore raw

RFC 1/12 libbpf: Deduplicate unambigous standalone forward declarations

Deduplicate forward declarations that don't take part in type graphs comparisons if declaration name is unambiguous. Example:

CU #1:

struct foo; // standalone forward declaration
struct foo *some_global;

CU #2:

struct foo { int x; };
struct foo *another_global;

The struct foo from CU #1 is not a part of any definition that is compared against another definition while btf_dedup_struct_types processes structural types. The the BTF after btf_dedup_struct_types the BTF looks as follows:

[1] STRUCT 'foo' size=4 vlen=1 ...
[2] INT 'int' size=4 ...
[3] PTR '(anon)' type_id=1
[4] FWD 'foo' fwd_kind=struct
[5] PTR '(anon)' type_id=4

This commit adds a new pass btf_dedup_standalone_fwds, that maps such forward declarations to structs or unions with identical name in case if the name is not ambiguous.

The pass is positioned before btf_dedup_ref_types so that types
[3] and [5] could be merged as a same type after [1] and [4] are merged. The final result for the example above looks as follows:

[1] STRUCT 'foo' size=4 vlen=1

	'x' type_id=2 bits_offset=0

[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[3] PTR '(anon)' type_id=1

For defconfig kernel with BTF enabled this removes 63 forward declarations. Examples of removed declarations: pt_regs, in6_addr. The running time of btf__dedup function is increased by about 3%.

---
 tools/lib/bpf/btf.c | 178 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 174 insertions(+), 4 deletions(-)
tools/lib/bpf/btf.c +174 -4
index d88647da2c7f..c34c68d8e8a0 100644
@@ -2881,6 +2881,7 @@ static int btf_dedup_strings(struct btf_dedup *d);
 static int btf_dedup_prim_types(struct btf_dedup *d);
 static int btf_dedup_struct_types(struct btf_dedup *d);
 static int btf_dedup_ref_types(struct btf_dedup *d);
+static int btf_dedup_standalone_fwds(struct btf_dedup *d);
 static int btf_dedup_compact_types(struct btf_dedup *d);
 static int btf_dedup_remap_types(struct btf_dedup *d);
 
@@ -2988,15 +2989,16 @@ static int btf_dedup_remap_types(struct btf_dedup *d);
  * Algorithm summary
  * =================
  *
- * Algorithm completes its work in 6 separate passes:
+ * Algorithm completes its work in 7 separate passes:
  *
  * 1. Strings deduplication.
  * 2. Primitive types deduplication (int, enum, fwd).
  * 3. Struct/union types deduplication.
- * 4. Reference types deduplication (pointers, typedefs, arrays, funcs, func
+ * 4. Standalone fwd declarations deduplication.
+ * 5. Reference types deduplication (pointers, typedefs, arrays, funcs, func
  *    protos, and const/volatile/restrict modifiers).
- * 5. Types compaction.
- * 6. Types remapping.
+ * 6. Types compaction.
+ * 7. Types remapping.
  *
  * Algorithm determines canonical type descriptor, which is a single
  * representative type for each truly unique type. This canonical type is the
@@ -3060,6 +3062,11 @@ int btf__dedup(struct btf *btf, const struct btf_dedup_opts *opts)
 		pr_debug("btf_dedup_struct_types failed:%d\n", err);
 		goto done;
 	}
+	err = btf_dedup_standalone_fwds(d);
+	if (err < 0) {
+		pr_debug("btf_dedup_standalone_fwd failed:%d\n", err);
+		goto done;
+	}
 	err = btf_dedup_ref_types(d);
 	if (err < 0) {
 		pr_debug("btf_dedup_ref_types failed:%d\n", err);
@@ -4525,6 +4532,169 @@ static int btf_dedup_ref_types(struct btf_dedup *d)
 	return 0;
 }
 
+/*
+ * `name_off_map` maps name offsets to type ids (essentially __u32 -> __u32).
+ *
+ * The __u32 key/value representations are cast to `void *` before passing
+ * to `hashmap__*` functions. These pseudo-pointers are never dereferenced.
+ *
+ */
+static struct hashmap *name_off_map__new(void)
+{
+	return hashmap__new(btf_dedup_identity_hash_fn,
+			    btf_dedup_equal_fn,
+			    NULL);
+}
+
+static int name_off_map__find(struct hashmap *map, __u32 name_off, __u32 *type_id)
+{
+	/* This has to be sizeof(void *) in order to be passed to hashmap__find */
+	void *tmp;
+	int found = hashmap__find(map, (void *)(ptrdiff_t)name_off, &tmp);
+	/*
+	 * __u64 cast is necessary to avoid pointer to integer conversion size warning.
+	 * It is fine to get rid of this warning as `void *` is used as an integer value.
+	 */
+	if (found)
+		*type_id = (__u64)tmp;
+	return found;
+}
+
+static int name_off_map__set(struct hashmap *map, __u32 name_off, __u32 type_id)
+{
+	return hashmap__set(map, (void *)(size_t)name_off, (void *)(size_t)type_id,
+			    NULL, NULL);
+}
+
+/*
+ * Collect a `name_off_map` that maps type names to type ids for all
+ * canonical structs and unions. If the same name is shared by several
+ * canonical types use a special value 0 to indicate this fact.
+ */
+static int btf_dedup_fill_unique_names_map(struct btf_dedup *d, struct hashmap *names_map)
+{
+	int i, err = 0;
+	__u32 type_id, collision_id;
+	__u16 kind;
+	struct btf_type *t;
+
+	for (i = 0; i < d->btf->nr_types; i++) {
+		type_id = d->btf->start_id + i;
+		t = btf_type_by_id(d->btf, type_id);
+		kind = btf_kind(t);
+
+		if (kind != BTF_KIND_STRUCT && kind != BTF_KIND_UNION)
+			continue;
+
+		/* Skip non-canonical types */
+		if (type_id != d->map[type_id])
+			continue;
+
+		err = 0;
+		if (name_off_map__find(names_map, t->name_off, &collision_id)) {
+			/* Mark non-unique names with 0 */
+			if (collision_id != 0 && collision_id != type_id)
+				err = name_off_map__set(names_map, t->name_off, 0);
+		} else {
+			err = name_off_map__set(names_map, t->name_off, type_id);
+		}
+
+		if (err < 0)
+			return err;
+	}
+
+	return 0;
+}
+
+static int btf_dedup_standalone_fwd(struct btf_dedup *d,
+				    struct hashmap *names_map,
+				    __u32 type_id)
+{
+	struct btf_type *t = btf_type_by_id(d->btf, type_id);
+	__u16 kind = btf_kind(t);
+	enum btf_fwd_kind fwd_kind = BTF_INFO_KFLAG(t->info);
+
+	struct btf_type *cand_t;
+	__u16 cand_kind;
+	__u32 cand_id = 0;
+
+	if (kind != BTF_KIND_FWD)
+		return 0;
+
+	/* Skip if this FWD already has a mapping */
+	if (type_id != d->map[type_id])
+		return 0;
+
+	name_off_map__find(names_map, t->name_off, &cand_id);
+	if (!cand_id)
+		return 0;
+
+	cand_t = btf_type_by_id(d->btf, cand_id);
+	cand_kind = btf_kind(cand_t);
+	if (!(cand_kind == BTF_KIND_STRUCT && fwd_kind == BTF_FWD_STRUCT) &&
+	    !(cand_kind == BTF_KIND_UNION && fwd_kind == BTF_FWD_UNION))
+		return 0;
+
+	d->map[type_id] = cand_id;
+
+	return 0;
+}
+
+/*
+ * Standalone fwd declarations deduplication.
+ *
+ * The lion's share of all FWD declarations is resolved during
+ * `btf_dedup_struct_types` phase when different type graphs are
+ * compared against each other. However, if in some compilation unit a
+ * FWD declaration is not a part of a type graph compared against
+ * another type graph that declaration's canonical type would not be
+ * changed. Example:
+ *
+ * CU #1:
+ *
+ * struct foo;
+ * struct foo *some_global;
+ *
+ * CU #2:
+ *
+ * struct foo { int u; };
+ * struct foo *another_global;
+ *
+ * After `btf_dedup_struct_types` the BTF looks as follows:
+ *
+ * [1] STRUCT 'foo' size=4 vlen=1 ...
+ * [2] INT 'int' size=4 ...
+ * [3] PTR '(anon)' type_id=1
+ * [4] FWD 'foo' fwd_kind=struct
+ * [5] PTR '(anon)' type_id=4
+ *
+ * This pass assumes that such FWD declarations should be mapped to
+ * structs or unions with identical name in case if the name is not
+ * ambiguous.
+ */
+static int btf_dedup_standalone_fwds(struct btf_dedup *d)
+{
+	int i, err;
+	struct hashmap *names_map = name_off_map__new();
+
+	if (!names_map)
+		return -ENOMEM;
+
+	err = btf_dedup_fill_unique_names_map(d, names_map);
+	if (err < 0)
+		goto exit;
+
+	for (i = 0; i < d->btf->nr_types; i++) {
+		err = btf_dedup_standalone_fwd(d, names_map, d->btf->start_id + i);
+		if (err < 0)
+			goto exit;
+	}
+
+exit:
+	hashmap__free(names_map);
+	return err;
+}
+
 /*
  * Compact types.
  *
Signature · 1 line
2.34.1
author maintainer
#7 lore raw

RFC 2/12 selftests/bpf: Tests for standalone forward BTF declarations deduplication

In reply to Eduard Zingerman

Tests to verify the following behavior of btf_dedup_standalone_fwds:
- remapping for struct forward declarations;
- remapping for union forward declarations;
- no remapping if forward declaration kind does not match similarly named struct or union declaration;
- no remapping if forward declaration name is ambiguous.

---
 tools/testing/selftests/bpf/prog_tests/btf.c | 152 +++++++++++++++++++
 1 file changed, 152 insertions(+)
tools/testing/selftests/bpf/prog_tests/btf.c +152 -0
index 127b8caa3dc1..f14020d51ab9 100644
@@ -7598,6 +7598,158 @@ static struct btf_dedup_test dedup_tests[] = {
 		BTF_STR_SEC("\0e1\0e1_val"),
 	},
 },
+{
+	.descr = "dedup: standalone fwd declaration struct",
+	/*
+	 * // CU 1:
+	 * struct foo { int x; };
+	 * struct foo *a;
+	 *
+	 * // CU 2:
+	 * struct foo;
+	 * struct foo *b;
+	 */
+	.input = {
+		.raw_types = {
+			BTF_STRUCT_ENC(NAME_NTH(1), 1, 4),             /* [1] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [2] */
+			BTF_PTR_ENC(1),                                /* [3] */
+			BTF_FWD_ENC(NAME_TBD, 0),                      /* [4] */
+			BTF_PTR_ENC(4),                                /* [5] */
+			BTF_END_RAW,
+		},
+		BTF_STR_SEC("\0foo\0x"),
+	},
+	.expect = {
+		.raw_types = {
+			BTF_STRUCT_ENC(NAME_NTH(1), 1, 4),             /* [1] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [2] */
+			BTF_PTR_ENC(1),                                /* [3] */
+			BTF_END_RAW,
+		},
+		BTF_STR_SEC("\0foo\0x"),
+	},
+},
+{
+	.descr = "dedup: standalone fwd declaration union",
+	/*
+	 * // CU 1:
+	 * union foo { int x; };
+	 * union foo *another_global;
+	 *
+	 * // CU 2:
+	 * union foo;
+	 * union foo *some_global;
+	 */
+	.input = {
+		.raw_types = {
+			BTF_UNION_ENC(NAME_NTH(1), 1, 4),              /* [1] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [2] */
+			BTF_PTR_ENC(1),                                /* [3] */
+			BTF_FWD_ENC(NAME_TBD, 1),                      /* [4] */
+			BTF_PTR_ENC(4),                                /* [5] */
+			BTF_END_RAW,
+		},
+		BTF_STR_SEC("\0foo\0x"),
+	},
+	.expect = {
+		.raw_types = {
+			BTF_UNION_ENC(NAME_NTH(1), 1, 4),              /* [1] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [2] */
+			BTF_PTR_ENC(1),                                /* [3] */
+			BTF_END_RAW,
+		},
+		BTF_STR_SEC("\0foo\0x"),
+	},
+},
+{
+	.descr = "dedup: standalone fwd declaration wrong kind",
+	/*
+	 * // CU 1:
+	 * struct foo { int x; };
+	 * struct foo *b;
+	 *
+	 * // CU 2:
+	 * union foo;
+	 * union foo *a;
+	 */
+	.input = {
+		.raw_types = {
+			BTF_STRUCT_ENC(NAME_NTH(1), 1, 4),             /* [1] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [2] */
+			BTF_PTR_ENC(1),                                /* [3] */
+			BTF_FWD_ENC(NAME_TBD, 1),                      /* [4] */
+			BTF_PTR_ENC(4),                                /* [5] */
+			BTF_END_RAW,
+		},
+		BTF_STR_SEC("\0foo\0x"),
+	},
+	.expect = {
+		.raw_types = {
+			BTF_STRUCT_ENC(NAME_NTH(1), 1, 4),             /* [1] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [2] */
+			BTF_PTR_ENC(1),                                /* [3] */
+			BTF_FWD_ENC(NAME_TBD, 1),                      /* [4] */
+			BTF_PTR_ENC(4),                                /* [5] */
+			BTF_END_RAW,
+		},
+		BTF_STR_SEC("\0foo\0x"),
+	},
+},
+{
+	.descr = "dedup: standalone fwd declaration name conflict",
+	/*
+	 * // CU 1:
+	 * struct foo { int x; };
+	 * struct foo *a;
+	 *
+	 * // CU 2:
+	 * struct foo;
+	 * struct foo *b;
+	 *
+	 * // CU 3:
+	 * struct foo { int x; int y; };
+	 * struct foo *c;
+	 */
+	.input = {
+		.raw_types = {
+			BTF_STRUCT_ENC(NAME_NTH(1), 1, 4),             /* [1] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [2] */
+			BTF_PTR_ENC(1),                                /* [3] */
+			BTF_FWD_ENC(NAME_TBD, 0),                      /* [4] */
+			BTF_PTR_ENC(4),                                /* [5] */
+			BTF_STRUCT_ENC(NAME_NTH(1), 2, 8),             /* [6] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_MEMBER_ENC(NAME_NTH(3), 2, 0),
+			BTF_PTR_ENC(6),                                /* [7] */
+			BTF_END_RAW,
+		},
+		BTF_STR_SEC("\0foo\0x\0y"),
+	},
+	.expect = {
+		.raw_types = {
+			BTF_STRUCT_ENC(NAME_NTH(1), 1, 4),             /* [1] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_TYPE_INT_ENC(0, BTF_INT_SIGNED, 0, 32, 4), /* [2] */
+			BTF_PTR_ENC(1),                                /* [3] */
+			BTF_FWD_ENC(NAME_TBD, 0),                      /* [4] */
+			BTF_PTR_ENC(4),                                /* [5] */
+			BTF_STRUCT_ENC(NAME_NTH(1), 2, 8),             /* [6] */
+			BTF_MEMBER_ENC(NAME_NTH(2), 2, 0),
+			BTF_MEMBER_ENC(NAME_NTH(3), 2, 0),
+			BTF_PTR_ENC(6),                                /* [7] */
+			BTF_END_RAW,
+		},
+		BTF_STR_SEC("\0foo\0x\0y"),
+	},
+},
 
 };
 
Signature · 1 line
2.34.1
author maintainer
#8 lore raw

RFC 3/12 libbpf: Support for BTF_DECL_TAG dump in C format

In reply to Eduard Zingerman

At C level BTF_DECL_TAGs are represented as __attribute__ declarations, e.g.:

struct foo {
	...;
} __attribute__((btf_decl_tag("bar")));

This commit covers only decl tags attached to structs and unions.

BTF doc says that BTF_DECL_TAGs should follow a target type but this is not enforced and tests don't honor this restriction. This commit uses hash table to map types to the list of decl tags.

---
 tools/lib/bpf/btf_dump.c | 143 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 142 insertions(+), 1 deletion(-)
tools/lib/bpf/btf_dump.c +142 -1
index bf0cc0e986dd..9bfe2a4ae277 100644
@@ -75,6 +75,15 @@ struct btf_dump_data {
 	bool is_array_char;
 };
 
+/*
+ * An array of ids of BTF_DECL_TAG objects associated with a specific type.
+ */
+struct decl_tag_array {
+	__u16 cnt;
+	__u16 cap;
+	__u32 tag_ids[0];
+};
+
 struct btf_dump {
 	const struct btf *btf;
 	btf_dump_printf_fn_t printf_fn;
@@ -111,6 +120,11 @@ struct btf_dump {
 	 * name occurrences
 	 */
 	struct hashmap *ident_names;
+	/*
+	 * maps type id to decl_tag_array, assume that relatively small
+	 * fraction of types has btf_decl_tag's attached
+	 */
+	struct hashmap *decl_tags;
 	/*
 	 * data for typed display; allocated if needed.
 	 */
@@ -127,6 +141,26 @@ static bool str_equal_fn(const void *a, const void *b, void *ctx)
 	return strcmp(a, b) == 0;
 }
 
+static size_t int_hash_fn(const void *key, void *ctx)
+{
+	int i;
+	size_t h = 0;
+	char *bytes = (char *)key;
+
+	for (i = 0; i < 4; ++i)
+		h = h * 31 + bytes[i];
+
+	return h;
+}
+
+static bool int_equal_fn(const void *a, const void *b, void *ctx)
+{
+	int *ia = (int *)a;
+	int *ib = (int *)b;
+
+	return *ia == *ib;
+}
+
 static const char *btf_name_of(const struct btf_dump *d, __u32 name_off)
 {
 	return btf__name_by_offset(d->btf, name_off);
@@ -143,6 +177,7 @@ static void btf_dump_printf(const struct btf_dump *d, const char *fmt, ...)
 
 static int btf_dump_mark_referenced(struct btf_dump *d);
 static int btf_dump_resize(struct btf_dump *d);
+static int btf_dump_assign_decl_tags(struct btf_dump *d);
 
 struct btf_dump *btf_dump__new(const struct btf *btf,
 			       btf_dump_printf_fn_t printf_fn,
@@ -179,11 +214,24 @@ struct btf_dump *btf_dump__new(const struct btf *btf,
 		d->ident_names = NULL;
 		goto err;
 	}
+	d->decl_tags = hashmap__new(int_hash_fn, int_equal_fn, NULL);
+	if (IS_ERR(d->decl_tags)) {
+		err = PTR_ERR(d->decl_tags);
+		d->decl_tags = NULL;
+		goto err;
+	}
 
 	err = btf_dump_resize(d);
 	if (err)
 		goto err;
 
+	err = btf_dump_assign_decl_tags(d);
+	if (err)
+		goto err;
+
+	if (err)
+		goto err;
+
 	return d;
 err:
 	btf_dump__free(d);
@@ -232,7 +280,8 @@ static void btf_dump_free_names(struct hashmap *map)
 
 void btf_dump__free(struct btf_dump *d)
 {
-	int i;
+	int i, bkt;
+	struct hashmap_entry *cur;
 
 	if (IS_ERR_OR_NULL(d))
 		return;
@@ -250,6 +299,9 @@ void btf_dump__free(struct btf_dump *d)
 	free(d->decl_stack);
 	btf_dump_free_names(d->type_names);
 	btf_dump_free_names(d->ident_names);
+	hashmap__for_each_entry(d->decl_tags, cur, bkt)
+		free(cur->value);
+	hashmap__free(d->decl_tags);
 
 	free(d);
 }
@@ -373,6 +425,77 @@ static int btf_dump_mark_referenced(struct btf_dump *d)
 	return 0;
 }
 
+static struct decl_tag_array *btf_dump_find_decl_tags(struct btf_dump *d, __u32 id)
+{
+	struct decl_tag_array *decl_tags = NULL;
+
+	hashmap__find(d->decl_tags, &id, (void **)&decl_tags);
+
+	return decl_tags;
+}
+
+static struct decl_tag_array *realloc_decl_tags(struct decl_tag_array *tags, __u16 new_cap)
+{
+	size_t new_size = sizeof(struct decl_tag_array) + new_cap * sizeof(__u32);
+	struct decl_tag_array *new_tags = (tags
+					   ? realloc(tags, new_size)
+					   : calloc(1, new_size));
+
+	if (!new_tags)
+		return NULL;
+
+	new_tags->cap = new_cap;
+
+	return new_tags;
+}
+
+/*
+ * Scans all BTF objects looking for BTF_KIND_DECL_TAG entries.
+ * The id's of the entries are stored in the `btf_dump.decl_tags` table,
+ * grouped by a target type.
+ */
+static int btf_dump_assign_decl_tags(struct btf_dump *d)
+{
+	int err;
+	__u32 id;
+	__u32 n = btf__type_cnt(d->btf);
+	__u32 new_capacity;
+	const struct btf_type *t;
+	struct decl_tag_array *decl_tags;
+
+	for (id = 0; id < n; id++) {
+		t = btf__type_by_id(d->btf, id);
+
+		if (btf_kind(t) != BTF_KIND_DECL_TAG)
+			continue;
+
+		decl_tags = btf_dump_find_decl_tags(d, t->type);
+		if (!decl_tags) {
+			decl_tags = realloc_decl_tags(NULL, 1);
+			if (!decl_tags)
+				return -ENOMEM;
+			err = hashmap__insert(d->decl_tags, &t->type, decl_tags,
+					      HASHMAP_SET, NULL, NULL);
+			if (err)
+				return err;
+		} else if (decl_tags->cnt == decl_tags->cap) {
+			new_capacity = decl_tags->cap * 2;
+			if (new_capacity > 0xffff)
+				return -ERANGE;
+			decl_tags = realloc_decl_tags(decl_tags, new_capacity);
+			if (!decl_tags)
+				return -ENOMEM;
+			decl_tags->cap = new_capacity;
+			err = hashmap__update(d->decl_tags, &t->type, decl_tags, NULL, NULL);
+			if (err)
+				return err;
+		}
+		decl_tags->tag_ids[decl_tags->cnt++] = id;
+	}
+
+	return 0;
+}
+
 static int btf_dump_add_emit_queue_id(struct btf_dump *d, __u32 id)
 {
 	__u32 *new_queue;
@@ -899,6 +1022,23 @@ static void btf_dump_emit_bit_padding(const struct btf_dump *d,
 	}
 }
 
+static void btf_dump_emit_decl_tags(struct btf_dump *d, __u32 id)
+{
+	struct decl_tag_array *decl_tags = btf_dump_find_decl_tags(d, id);
+	struct btf_type *decl_tag_t;
+	const char *decl_tag_text;
+	__u32 i;
+
+	if (!decl_tags)
+		return;
+
+	for (i = 0; i < decl_tags->cnt; ++i) {
+		decl_tag_t = btf_type_by_id(d->btf, decl_tags->tag_ids[i]);
+		decl_tag_text = btf__name_by_offset(d->btf, decl_tag_t->name_off);
+		btf_dump_printf(d, " __attribute__((btf_decl_tag(\"%s\")))", decl_tag_text);
+	}
+}
+
 static void btf_dump_emit_struct_fwd(struct btf_dump *d, __u32 id,
 				     const struct btf_type *t)
 {
@@ -964,6 +1104,7 @@ static void btf_dump_emit_struct_def(struct btf_dump *d,
 	btf_dump_printf(d, "%s}", pfx(lvl));
 	if (packed)
 		btf_dump_printf(d, " __attribute__((packed))");
+	btf_dump_emit_decl_tags(d, id);
 }
 
 static const char *missing_base_types[][2] = {
Signature · 1 line
2.34.1
author maintainer
#10 lore raw

RFC 4/12 selftests/bpf: Tests for BTF_DECL_TAG dump in C format

In reply to Eduard Zingerman

Covers the following cases:
- __atribute__((btf_decl_tag("..."))) could be applied to structs and unions;
- decl tag applied to an empty struct is printed on a single line;
- decl tags with the same name could be applied to several structs;
- several decl tags could be applied to the same struct;
- attribute packed works fine with decl tags (it is a separate branch in tools/lib/bpf/btf_dump.c:btf_dump_emit_attributes.

---
 .../selftests/bpf/prog_tests/btf_dump.c       |  1 +
 .../bpf/progs/btf_dump_test_case_decl_tag.c   | 39 +++++++++++++++++++
 2 files changed, 40 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c
tools/testing/selftests/bpf/prog_tests/btf_dump.c +1 -0
index 24da335482d4..5f6ce7f1a801 100644
@@ -21,6 +21,7 @@ static struct btf_dump_test_case {
 	{"btf_dump: bitfields", "btf_dump_test_case_bitfields", true},
 	{"btf_dump: multidim", "btf_dump_test_case_multidim", false},
 	{"btf_dump: namespacing", "btf_dump_test_case_namespacing", false},
+	{"btf_dump: decl_tag", "btf_dump_test_case_decl_tag", true},
 };
 
 static int btf_dump_all_types(const struct btf *btf, void *ctx)
tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c new file +39 -0
new file mode 100644
index 000000000000..470bbbb530dc
@@ -0,0 +1,39 @@
+// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+/*
+ * BTF-to-C dumper test for __atribute__((btf_decl_tag("..."))).
+ */
+/* ----- START-EXPECTED-OUTPUT ----- */
+struct empty_with_tag {} __attribute__((btf_decl_tag("a")));
+
+struct one_tag {
+	int x;
+} __attribute__((btf_decl_tag("b")));
+
+struct same_tag {
+	int x;
+} __attribute__((btf_decl_tag("b")));
+
+struct two_tags {
+	int x;
+} __attribute__((btf_decl_tag("a"))) __attribute__((btf_decl_tag("b")));
+
+struct packed {
+	int x;
+	short y;
+} __attribute__((packed)) __attribute__((btf_decl_tag("another_name")));
+
+struct root_struct {
+	struct empty_with_tag a;
+	struct one_tag b;
+	struct same_tag c;
+	struct two_tags d;
+	struct packed e;
+};
+
+/* ------ END-EXPECTED-OUTPUT ------ */
+
+int f(struct root_struct *s)
+{
+	return 0;
+}
Signature · 1 line
2.34.1
author maintainer
#11 lore raw

RFC 5/12 libbpf: Header guards for selected data structures in vmlinux.h

In reply to Eduard Zingerman

The goal of the patch is to allow usage of header files from include/uapi alongside with vmlinux.h. E.g. as follows:

  #include <uapi/linux/tcp.h>
  #include "vmlinux.h"

This goal is achieved by adding #ifndef / #endif guards in vmlinux.h around definitions that originate from the include/uapi headers. The guards emitted match the guards used in the original headers. E.g. as follows:

include/uapi/linux/tcp.h:

#ifndef _UAPI_LINUX_TCP_H
#define _UAPI_LINUX_TCP_H
...

  union tcp_word_hdr {
	struct tcphdr hdr;
	__be32        words[5];

};
...
#endif /* _UAPI_LINUX_TCP_H */

vmlinux.h:

  ...
  #ifndef _UAPI_LINUX_TCP_H
  union tcp_word_hdr {
	struct tcphdr hdr;
	__be32 words[5];

};

  #endif /* _UAPI_LINUX_TCP_H */
  ...

The problem of identifying data structures from uapi and selecting proper guard names is delegated to pahole. When configured pahole generates fake BTF_DECL_TAG records with header guards information. The fake tag is distinguished from a real tag by a prefix "header_guard:" in its value. These tags could be present for unions, structures, enums and typedefs, e.g.:

[24139] STRUCT 'tcphdr' size=20 vlen=17
...
[24296] DECL_TAG 'header_guard:_UAPI_LINUX_TCP_H' type_id=24139 ...

This patch adds An option emit_header_guards to struct btf_dump_opts. When this option is present the function btf_dump__dump_type emits header guards for top-level declarations. The header guards are identified by inspecting fake BTF_DECL_TAG records described above.

---
 tools/lib/bpf/btf.h      |  7 +++-
 tools/lib/bpf/btf_dump.c | 89 +++++++++++++++++++++++++++++++++++++++-
 2 files changed, 94 insertions(+), 2 deletions(-)
tools/lib/bpf/btf.h +6 -1
index 8e6880d91c84..dcfb3f168750 100644
@@ -235,8 +235,13 @@ struct btf_dump;
 
 struct btf_dump_opts {
 	size_t sz;
+	/*
+	 * if set to true #ifndef X ... #endif brackets would be printed
+	 * for types marked by BTF_DECL_TAG with "header_guard:" prefix
+	 */
+	bool emit_header_guards;
 };
-#define btf_dump_opts__last_field sz
+#define btf_dump_opts__last_field emit_header_guards
 
 typedef void (*btf_dump_printf_fn_t)(void *ctx, const char *fmt, va_list args);
 
tools/lib/bpf/btf_dump.c +88 -1

195 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#13 lore raw

RFC 6/12 selftests/bpf: Tests for header guards printing in BTF dump

In reply to Eduard Zingerman

Verify that btf_dump__dump_type emits header guard brackets for various types when btf_dump_opts.emit_header_guards is set to true.

---
 .../selftests/bpf/prog_tests/btf_dump.c       | 10 +-
 .../progs/btf_dump_test_case_header_guards.c  | 94 +++++++++++++++++++
 2 files changed, 101 insertions(+), 3 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c
tools/testing/selftests/bpf/prog_tests/btf_dump.c +7 -3
index 5f6ce7f1a801..a3db352e61c7 100644
@@ -13,6 +13,7 @@ static struct btf_dump_test_case {
 	const char *name;
 	const char *file;
 	bool known_ptr_sz;
+	bool emit_header_guards;
 } btf_dump_test_cases[] = {
 	{"btf_dump: syntax", "btf_dump_test_case_syntax", true},
 	{"btf_dump: ordering", "btf_dump_test_case_ordering", false},
@@ -22,15 +23,18 @@ static struct btf_dump_test_case {
 	{"btf_dump: multidim", "btf_dump_test_case_multidim", false},
 	{"btf_dump: namespacing", "btf_dump_test_case_namespacing", false},
 	{"btf_dump: decl_tag", "btf_dump_test_case_decl_tag", true},
+	{"btf_dump: header guards", "btf_dump_test_case_header_guards", true, true},
 };
 
-static int btf_dump_all_types(const struct btf *btf, void *ctx)
+static int btf_dump_all_types(const struct btf *btf, void *ctx, struct btf_dump_test_case *t)
 {
 	size_t type_cnt = btf__type_cnt(btf);
+	LIBBPF_OPTS(btf_dump_opts, opts);
 	struct btf_dump *d;
 	int err = 0, id;
 
-	d = btf_dump__new(btf, btf_dump_printf, ctx, NULL);
+	opts.emit_header_guards = t->emit_header_guards;
+	d = btf_dump__new(btf, btf_dump_printf, ctx, &opts);
 	err = libbpf_get_error(d);
 	if (err)
 		return err;
@@ -87,7 +91,7 @@ static int test_btf_dump_case(int n, struct btf_dump_test_case *t)
 		goto done;
 	}
 
-	err = btf_dump_all_types(btf, f);
+	err = btf_dump_all_types(btf, f, t);
 	fclose(f);
 	close(fd);
 	if (CHECK(err, "btf_dump", "failure during C dumping: %d\n", err)) {
tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
2.34.1
author maintainer
#14 lore raw

RFC 7/12 bpftool: Enable header guards generation

In reply to Eduard Zingerman

Enables header guards generation for BTF dumps in C format, e.g. vmlinux.h would be printed as follows:

  ...
  #ifndef _UAPI_LINUX_TCP_H

  enum {
    TCP_NO_QUEUE = 0,
    TCP_RECV_QUEUE = 1,
    TCP_SEND_QUEUE = 2,
    TCP_QUEUES_NR = 3,
  };

  #endif /* _UAPI_LINUX_TCP_H */
  ...
---
 tools/bpf/bpftool/btf.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
tools/bpf/bpftool/btf.c +3 -1
index 68a70ac03c80..f8e8946b61a7 100644
@@ -466,7 +466,9 @@ static int dump_btf_c(const struct btf *btf,
 	struct btf_dump *d;
 	int err = 0, i;
 
-	d = btf_dump__new(btf, btf_dump_printf, NULL, NULL);
+	LIBBPF_OPTS(btf_dump_opts, opts);
+	opts.emit_header_guards = true;
+	d = btf_dump__new(btf, btf_dump_printf, NULL, &opts);
 	err = libbpf_get_error(d);
 	if (err)
 		return err;
Signature · 1 line
2.34.1
author maintainer
#15 lore raw

RFC 8/12 kbuild: Script to infer header guard values for uapi headers

In reply to Eduard Zingerman

The script infers header guard defines in headers from include/uapi/**/*.h . E.g. header guard for the include/uapi/linux/tcp.h is _UAPI_LINUX_TCP_H:

    include/uapi/linux/tcp.h:

      #ifndef _UAPI_LINUX_TCP_H
      #define _UAPI_LINUX_TCP_H
      ...
      union tcp_word_hdr {
            struct tcphdr hdr;
            __be32        words[5];
      };
      ...
      #endif /* _UAPI_LINUX_TCP_H */

The output of the script could be used as an input to pahole's --header_guards_db parameter. This information is necessary to repeat the same header guards in the vmlinux.h generated from BTF.

It is not possible to infer the guard names from header file names alone, the file content has to be analyzed. The following heuristic is used to infer guard for a specific file:
- All pairs #ifndef <candidate> / #define <candidate> are collected;
- If a unique candidate matching regex ${headername}.*_H(EADER)? it is selected;
- If a unique candidate matching regex _H(EADER)?_ it is selected;
- If a unique candidate matching regex _H(EADER)?$ it is selected;

There is also a small list of headers that can't be caught by the rules above, 15 in total. These headers and corresponding guard values are listed in the %OVERRIDES hash table.

---
 scripts/infer_header_guards.pl | 191 +++++++++++++++++++++++++++++++++
 1 file changed, 191 insertions(+)
 create mode 100755 scripts/infer_header_guards.pl
scripts/infer_header_guards.pl new file +191 -0
new file mode 100755
index 000000000000..201008ac83f3
@@ -0,0 +1,191 @@
+#!/usr/bin/env perl
+# SPDX-License-Identifier: GPL-2.0
+
+# This script scans the passed directory for header files (files ending with ".h").
+# For each header file it tries to infer the name of a C pre-processor
+# variable used as a double include guard (dubbed as "header guard").
+# For example:
+#
+#   #ifdef __MY_HEADER__  // <-- "header guard"
+#     ...
+#   #endif
+#
+# The inferred guards are printed to stdout in the following format:
+#
+#   <header-file> <header-guard>
+#
+# This is an expected format for pahole --header_guards_db parameter.
+# Intended usage is to infer header guards for Linux UAPI headers.
+# The collected information is further used in BTF embedded into kernel.
+#
+# The following inference logic is used for each file:
+# - find all pairs `#ifndef <name> #define <name>`
+# - if there is a unique <name> that matches a pattern - use this
+#   <name> as a header guard, (see subroutine `select_guard` for the
+#   list of the patterns);
+# - files containing only #include directives are safe to ignore.
+#
+# There are a few UAPI header files that don't fit in such logic,
+# header guards for these files are hard-coded in %OVERRIDES hash.
+#
+# The script reports inference only when --report-failures flag is
+# passed. This flag is intended for BPF tests.
+#
+# See subroutine `help` for usage info.
+
+use strict;
+use warnings;
+use File::Basename;
+use File::Find;
+use Getopt::Long;
+
+sub help {
+	my $message = << "EOM";
+Usage:
+  $0 [--report-failures] directory-or-file...
+  $0 --help
+
+For a specific file or for each .h file in a directory infer the name
+of a C pre-processor variable used as a double include guard.
+
+Options:
+  --report-failures   Report inference errors to stderr,
+                      exit with non-zero code if guards were not inferred
+                      for some files.
+  --help              Print this message and exit.
+EOM
+	print $message;
+}
+
+my %OVERRIDES = (
+	# Header guards that don't follow common naming rules
+	"include/uapi/linux/cciss_ioctl.h" => "_UAPICCISS_IOCTLH",
+	"include/uapi/linux/hpet.h" => "_UAPI__HPET__",
+	"include/uapi/linux/if_ppp.h" => "_PPP_IOCTL_H",
+	"include/uapi/linux/netfilter/xt_NFLOG.h" => "_XT_NFLOG_TARGET",
+	"include/uapi/linux/netfilter_ipv6/ip6t_NPT.h" => "__NETFILTER_IP6T_NPT",
+	"include/uapi/linux/quota.h" => "_UAPI_LINUX_QUOTA_",
+	"include/uapi/linux/v4l2-common.h" => "__V4L2_COMMON__",
+	# Headers that should be ignored
+	"arch/x86/include/uapi/asm/hw_breakpoint.h" => undef,
+	"arch/x86/include/uapi/asm/posix_types.h" => undef,
+	"arch/x86/include/uapi/asm/setup.h" => undef,
+	"include/generated/uapi/linux/version.h" => undef,
+	"include/uapi/asm-generic/bitsperlong.h" => undef,
+	"include/uapi/asm-generic/kvm_para.h" => undef,
+	"include/uapi/asm-generic/unistd.h" => undef,
+	"include/uapi/linux/irqnr.h" => undef,
+	"include/uapi/linux/zorro_ids.h" => undef,
+	);
+
+sub get_basename {
+	my ($filename) = @_;
+	my $basename = fileparse($filename, qr/\.[^.]*/);
+	return $basename;
+}
+
+sub find_bracket_candidates {
+	my ($filename) = @_;
+	my @candidates = ();
+	my $guard_candidate = undef;
+	my $safe_to_ignore = 1;
+
+	open my $file, $filename or die "Can't open file $filename: $!";
+	while (my $line = <$file>) {
+		if (not($line =~ "^#include")) {
+			$safe_to_ignore = 0;
+		}
+		if ($line =~ "^#ifndef[ \t]+([a-zA-Z0-9_]+)") {
+			$guard_candidate = $1;
+		} elsif ($guard_candidate && $line =~ "^#define[ \t]+${guard_candidate}") {
+			push(@candidates, $guard_candidate);
+			$guard_candidate = undef;
+		}
+	}
+	close $file;
+
+	return ($safe_to_ignore, @candidates);
+}
+
+sub select_guard {
+	my ($filename, @candidates) = @_;
+	my $basename = get_basename($filename);
+	my @regexes = ("$basename.*_H(EADER)?",
+		       "_H(EADER)?_",
+		       "_H(EADER)?\$");
+	foreach my $re (@regexes) {
+		my @filtered = grep(/$re/i, @candidates);
+		if (scalar(@filtered) == 1) {
+			return $filtered[0];
+		}
+	}
+
+	return undef;
+}
+
+sub collect_headers {
+	my ($dir) = @_;
+	my @headers = ();
+
+	find(sub { /\.h$/ && push(@headers, $File::Find::name); }, $dir);
+
+	return @headers;
+}
+
+my $report_failures = 0;
+my $options_parsed = GetOptions(
+	"report-failures" => \$report_failures,
+	"help" => sub { help(); exit 0; },
+    );
+
+if (!$options_parsed || scalar @ARGV == 0) {
+	help();
+	exit 1;
+}
+
+my @headers;
+
+foreach my $dir_or_file (@ARGV) {
+	if (-f $dir_or_file) {
+		push(@headers, $dir_or_file);
+	} elsif (-d $dir_or_file) {
+		push(@headers, collect_headers($dir_or_file));
+	} else {
+		print("'$dir_or_file' is not a file or directory.\n");
+		help();
+		exit 1;
+	}
+}
+
+my $rc = 0;
+
+foreach my $header (@headers) {
+	my $basename = get_basename($header);
+	my $guard;
+
+	if (exists $OVERRIDES{$header}) {
+		$guard = $OVERRIDES{$basename};
+	} else {
+		my ($safe_to_ignore, @candidates) = find_bracket_candidates($header);
+		$guard = select_guard($header, @candidates);
+		if ((not $guard) && (not $safe_to_ignore) && $report_failures) {
+			print STDERR "Can't select guard for $header, candidates:\n";
+			print STDERR "  ";
+			if (scalar(@candidates)) {
+				print STDERR join(", ", @candidates);
+			} else {
+				print STDERR "<no candidates>"
+			}
+			print STDERR "\n";
+			$rc = 1;
+		}
+	}
+	if ($guard) {
+		# Remove the _UAPI prefix/suffix the same way
+		# scripts/headers_install.sh does it.
+		$guard =~ s/_UAPI//;
+		print("$header $guard\n");
+	}
+}
+
+exit $rc;
Signature · 1 line
2.34.1
author maintainer
#17 lore raw

RFC 9/12 kbuild: Header guards for types from include/uapi/*.h in kernel BTF

In reply to Eduard Zingerman

Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the scripts/infer_header_guards.pl.

The encoded header guard information could be used to restore the original guards in the vmlinux.h, e.g.:

    include/uapi/linux/tcp.h:

      #ifndef _UAPI_LINUX_TCP_H
      #define _UAPI_LINUX_TCP_H
      ...
      union tcp_word_hdr {
    	struct tcphdr hdr;
    	__be32        words[5];
      };
      ...
      #endif /* _UAPI_LINUX_TCP_H */

    vmlinux.h:

      ...
      #ifndef _UAPI_LINUX_TCP_H

      union tcp_word_hdr {
    	struct tcphdr hdr;
    	__be32 words[5];
      };

      #endif /* _UAPI_LINUX_TCP_H */
      ...
---
 scripts/link-vmlinux.sh | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)
scripts/link-vmlinux.sh +12 -1
index 918470d768e9..f57f621eda1f 100755
@@ -110,6 +110,7 @@ vmlinux_link()
 gen_btf()
 {
 	local pahole_ver
+	local extra_flags
 
 	if ! [ -x "$(command -v ${PAHOLE})" ]; then
 		echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"
@@ -122,10 +123,20 @@ gen_btf()
 		return 1
 	fi
 
+	if [ "${pahole_ver}" -ge "124" ]; then
+		scripts/infer_header_guards.pl \
+			include/uapi \
+			include/generated/uapi \
+			arch/${SRCARCH}/include/uapi \
+			arch/${SRCARCH}/include/generated/uapi \
+			> .btf.uapi_header_guards || return 1;
+		extra_flags="--header_guards_db .btf.uapi_header_guards"
+	fi
+
 	vmlinux_link ${1}
 
 	info "BTF" ${2}
-	LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1}
+	LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1}
 
 	# Create ${2} which contains just .BTF section but no symbols. Add
 	# SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
Signature · 1 line
2.34.1
author maintainer
#25 lore raw

RFC 10/12 selftests/bpf: Script to verify uapi headers usage with vmlinux.h

In reply to Eduard Zingerman

A script to test header guards in vmlinux.h by compiling a simple C snippet for a set of selected UAPI headers. The snippet being compiled looks as follows:

  #include <some_uapi_header.h>
  #include "vmlinux.h"

__attribute__((section("tc"), used))
int syncookie_tc(struct __sk_buff *skb) { return 0; }

If header guards are placed correctly in vmlinux.h the snippet should compile w/o errors.

The list of known good headers is supposed to be located in tools/testing/selftests/bpf/good_uapi_headers.txt added as a separate commit.

---
 .../selftests/bpf/test_uapi_headers.py        | 197 ++++++++++++++++++
 1 file changed, 197 insertions(+)
 create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
tools/testing/selftests/bpf/test_uapi_headers.py new file +197 -0
new file mode 100755
index 000000000000..1740c4fe0625
@@ -0,0 +1,197 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+
+# A script to test header guards in vmlinux.h by compiling a simple C
+# snippet for a set of selected UAPI headers. The snippet being
+# compiled looks as follows:
+#
+#   #include <some_uapi_header.h>
+#   #include "vmlinux.h"
+#
+#   __attribute__((section("tc"), used))
+#   int syncookie_tc(struct __sk_buff *skb) { return 0; }
+#
+# If header guards are placed correctly in vmlinux.h the snippet
+# should compile w/o errors.
+#
+# The script could be used in two modes:
+# - interactive BPF testing and CI;
+# - debug mode.
+#
+# * Interactive BPF testing and CI
+#
+# Run script as follows:
+#
+#   ./test_uapi_headers.py
+#
+# In this mode the following actions are performed:
+# - kernel headers are installed to a temporary directory;
+# - a list of known good uapi headers is read from ./good_uapi_headers.txt;
+# - the snippet above is compiled by clang using BPF target for each header;
+# - if shell is interactive the progress / ETA are reported during execution;
+# - pass / fail statistics is reported in the end;
+# - headers temporary directory is deleted;
+# - script exit code is 0 if snippet could be compiled for all headers.
+#
+# The vmlinux.h processing time is significant (~700ms using Intel i7-4710HQ),
+# thus the headers are processed in parallel.
+#
+# * Debug mode
+#
+# The following parameters are available for debugging:
+#
+#   test_uapi_headers.py \
+#            [-h] [--kheaders KHEADERS] [--vmlinuxh VMLINUXH] [--test TEST]
+#
+#   options:
+#     -h, --help           show this help message and exit
+#     --kheaders KHEADERS  path to exported kernel headers
+#     --vmlinuxh VMLINUXH  path to vmlinux.h
+#     --test TEST          name of the header -or-
+#                          file with header names -or-
+#                          special value '*'
+#
+# When --kheaders is specified the temporary directory is not created
+# and KHEADERS is used instead. It is assumed that headers are already
+# installed to KHEADERS.
+#
+# When TEST names a header (e.g. 'linux/tcp.h') it is the to test.
+# When TEST names a file this file should contain a list of
+# headers to test one per line.
+# When TEST is '*' all exported headers are tested.
+#
+# The simplest way to debug an issue with a single header is:
+#
+#   ./test_uapi_headers.py --test linux/tcp.h
+
+import subprocess
+import concurrent.futures
+import pathlib
+import time
+import os
+import sys
+import argparse
+import tempfile
+import shutil
+import atexit
+from dataclasses import dataclass
+
+@dataclass
+class Result:
+    header: pathlib.Path
+    returncode: int
+    stderr: str
+
+def run_one(header, kheaders, vmlinuxh):
+    code=f'''
+#include <{header}>
+#include "{vmlinuxh}"
+
+__attribute__((section("tc"), used))
+int syncookie_tc(struct __sk_buff *skb)
+{{
+    return 0;
+}}
+    '''
+    command = f'''
+{os.getenv('CLANG', 'clang')} \
+    -g -Werror -mlittle-endian \
+    -D__x86_64__ \
+    -Xclang -fwchar-type=short \
+    -Xclang -fno-signed-wchar \
+    -I{kheaders}/include/ \
+    -Wno-compare-distinct-pointer-types \
+    -mcpu=v3 \
+    -O2 \
+    -target bpf \
+    -x c \
+    -o /dev/null \
+    -fsyntax-only \
+    -
+'''
+    proc = subprocess.run(command, input=code, capture_output=True,
+                          shell=True, encoding='utf8')
+    return Result(header=header,
+                  returncode=proc.returncode,
+                  stderr=proc.stderr)
+
+def run_all(headers, kheaders, vmlinuxh):
+    start_time = time.time()
+    ok = 0
+    fail = 0
+    failures = []
+    remain = len(headers)
+    print_progress = sys.stdout.isatty()
+    print(f'Processing {remain} headers.')
+    with concurrent.futures.ThreadPoolExecutor(max_workers=os.cpu_count()) as executor:
+        for result in executor.map(lambda header: run_one(header, kheaders, vmlinuxh),
+                                   headers):
+            if result.returncode == 0:
+                print(f"{result.header:<60}   ok")
+                ok += 1
+            else:
+                print(f"{result.header:<60} fail")
+                fail += 1
+                failures.append(result)
+            remain -= 1
+            if print_progress:
+                elapsed = time.time() - start_time
+                processed = ok + fail
+                time_per_header = elapsed / processed
+                eta = int(remain * time_per_header)
+                # keep this shorter than header ok/fail line
+                line = f"Ok {ok: >4} Fail {fail: >4} Remain {remain: >4} ETA {eta: >4}s"
+                print(line, end="\r")
+    if print_progress:
+        print('')
+    elapsed = int(time.time() - start_time)
+    if fail == 0:
+        print(f"Done in {elapsed}s, all {len(headers)} ok.")
+    else:
+        print('----- Failure details -----')
+        for result in failures:
+            print(f'{result.header}: rc = {result.returncode}')
+            for line in result.stderr.split('\n'):
+                print(f"{result.header}: {line}")
+        print(f"Done in {elapsed}s, {fail} out of {len(headers)} failed.")
+    return fail == 0
+
+def main(argv):
+    bpf_test_dir = pathlib.Path(__file__).resolve().parent
+    default_vmlinuxh = bpf_test_dir / './tools/include/vmlinux.h'
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--kheaders", type=str, help='path to exported kernel headers')
+    parser.add_argument("--vmlinuxh", type=str, default=default_vmlinuxh,
+                        help='path to vmlinux.h')
+    parser.add_argument("--test", type=str,
+                        default='./good_uapi_headers.txt',
+                        help="name of the header | file with header names | special value '*'")
+    args = parser.parse_args(argv)
+
+    if args.kheaders is None:
+        kheaders = tempfile.mkdtemp(prefix='kheaders')
+        atexit.register(lambda: shutil.rmtree(kheaders))
+        kernel_dir = bpf_test_dir / '../../../../'
+        # Capture both stdout and stderr as stdout to simplify CI logging
+        subprocess.run(f'make -C {kernel_dir} INSTALL_HDR_PATH={kheaders} headers_install',
+                       stdout=sys.stdout, stderr=sys.stdout,
+                       check=True, shell=True)
+    else:
+        kheaders = args.kheaders
+
+    if os.path.exists(args.test):
+        with open(args.test, 'r') as list_file:
+            headers = [line.strip() for line in list_file]
+    elif args.test == '*':
+        headers = [p.relative_to(f'{kheaders}/include').as_posix()
+                   for p in pathlib.Path(kheaders).rglob("*.h")]
+    else:
+        headers = [args.test]
+
+    if run_all(headers, kheaders, args.vmlinuxh):
+        sys.exit(0)
+    else:
+        sys.exit(1)
+
+if __name__ == '__main__':
+    main(sys.argv[1:])
Signature · 1 line
2.34.1
author maintainer
#26 lore raw

RFC 11/12 selftests/bpf: Known good uapi headers for test_uapi_headers.py

In reply to Eduard Zingerman

The list of uapi headers that could be included alongside vmlinux.h. The headers are peeked from the following locations:
- <headers-export-dir>/linux/*.h
- <headers-export-dir>/linux/**/*.h

---
 .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
 1 file changed, 677 insertions(+)
 create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt
tools/testing/selftests/bpf/good_uapi_headers.txt new file +677 -0
new file mode 100644
index 000000000000..d2a7a6e3de74
@@ -0,0 +1,677 @@
+linux/a.out.h
+linux/acct.h
+linux/acrn.h
+linux/adb.h
+linux/adfs_fs.h
+linux/affs_hardblocks.h
+linux/aio_abi.h
+linux/amt.h
+linux/android/binder.h
+linux/android/binderfs.h
+linux/apm_bios.h
+linux/arcfb.h
+linux/arm_sdei.h
+linux/aspeed-lpc-ctrl.h
+linux/aspeed-p2a-ctrl.h
+linux/atalk.h
+linux/atm.h
+linux/atm_eni.h
+linux/atm_he.h
+linux/atm_idt77105.h
+linux/atm_nicstar.h
+linux/atm_tcp.h
+linux/atm_zatm.h
+linux/atmapi.h
+linux/atmarp.h
+linux/atmclip.h
+linux/atmdev.h
+linux/atmioc.h
+linux/atmlec.h
+linux/atmmpc.h
+linux/atmppp.h
+linux/atmsap.h
+linux/atmsvc.h
+linux/audit.h
+linux/auxvec.h
+linux/ax25.h
+linux/batadv_packet.h
+linux/batman_adv.h
+linux/baycom.h
+linux/bcm933xx_hcs.h
+linux/bfs_fs.h
+linux/binfmts.h
+linux/blkpg.h
+linux/blktrace_api.h
+linux/blkzoned.h
+linux/bpf.h
+linux/bpf_common.h
+linux/bpqether.h
+linux/bsg.h
+linux/bt-bmc.h
+linux/btf.h
+linux/btrfs.h
+linux/btrfs_tree.h
+linux/byteorder/big_endian.h
+linux/byteorder/little_endian.h
+linux/cachefiles.h
+linux/caif/caif_socket.h
+linux/caif/if_caif.h
+linux/can.h
+linux/can/bcm.h
+linux/can/error.h
+linux/can/gw.h
+linux/can/isotp.h
+linux/can/j1939.h
+linux/can/netlink.h
+linux/can/raw.h
+linux/can/vxcan.h
+linux/capability.h
+linux/capi.h
+linux/cciss_defs.h
+linux/cciss_ioctl.h
+linux/ccs.h
+linux/cdrom.h
+linux/cfm_bridge.h
+linux/cgroupstats.h
+linux/chio.h
+linux/cifs/cifs_mount.h
+linux/cifs/cifs_netlink.h
+linux/close_range.h
+linux/cm4000_cs.h
+linux/cn_proc.h
+linux/coff.h
+linux/comedi.h
+linux/connector.h
+linux/const.h
+linux/coresight-stm.h
+linux/counter.h
+linux/cramfs_fs.h
+linux/cryptouser.h
+linux/cuda.h
+linux/cxl_mem.h
+linux/cycx_cfm.h
+linux/dcbnl.h
+linux/dccp.h
+linux/devlink.h
+linux/dlm.h
+linux/dlm_device.h
+linux/dlm_netlink.h
+linux/dlm_plock.h
+linux/dlmconstants.h
+linux/dm-ioctl.h
+linux/dm-log-userspace.h
+linux/dma-heap.h
+linux/dns_resolver.h
+linux/dqblk_xfs.h
+linux/dvb/audio.h
+linux/dvb/ca.h
+linux/dvb/frontend.h
+linux/dvb/net.h
+linux/dvb/osd.h
+linux/dvb/version.h
+linux/dw100.h
+linux/edd.h
+linux/efs_fs_sb.h
+linux/elf-em.h
+linux/elf-fdpic.h
+linux/elf.h
+linux/errno.h
+linux/erspan.h
+linux/eventpoll.h
+linux/f2fs.h
+linux/fadvise.h
+linux/falloc.h
+linux/fanotify.h
+linux/fb.h
+linux/fcntl.h
+linux/fd.h
+linux/fdreg.h
+linux/fib_rules.h
+linux/fiemap.h
+linux/filter.h
+linux/firewire-cdev.h
+linux/firewire-constants.h
+linux/fou.h
+linux/fpga-dfl.h
+linux/fs.h
+linux/fscrypt.h
+linux/fsi.h
+linux/fsl_hypervisor.h
+linux/fsl_mc.h
+linux/fsmap.h
+linux/fsverity.h
+linux/futex.h
+linux/gameport.h
+linux/gen_stats.h
+linux/genetlink.h
+linux/genwqe/genwqe_card.h
+linux/gfs2_ondisk.h
+linux/gpio.h
+linux/gtp.h
+linux/hash_info.h
+linux/hdlc.h
+linux/hdlcdrv.h
+linux/hdreg.h
+linux/hid.h
+linux/hiddev.h
+linux/hidraw.h
+linux/hsi/cs-protocol.h
+linux/hsi/hsi_char.h
+linux/hsr_netlink.h
+linux/hw_breakpoint.h
+linux/hyperv.h
+linux/i2c-dev.h
+linux/i2c.h
+linux/i2o-dev.h
+linux/i8k.h
+linux/icmpv6.h
+linux/if_addr.h
+linux/if_addrlabel.h
+linux/if_alg.h
+linux/if_arcnet.h
+linux/if_bridge.h
+linux/if_cablemodem.h
+linux/if_eql.h
+linux/if_ether.h
+linux/if_fc.h
+linux/if_fddi.h
+linux/if_hippi.h
+linux/if_infiniband.h
+linux/if_link.h
+linux/if_ltalk.h
+linux/if_macsec.h
+linux/if_packet.h
+linux/if_phonet.h
+linux/if_plip.h
+linux/if_ppp.h
+linux/if_pppol2tp.h
+linux/if_slip.h
+linux/if_team.h
+linux/if_tun.h
+linux/if_vlan.h
+linux/if_x25.h
+linux/if_xdp.h
+linux/ife.h
+linux/igmp.h
+linux/iio/buffer.h
+linux/iio/events.h
+linux/iio/types.h
+linux/ila.h
+linux/in.h
+linux/in6.h
+linux/in_route.h
+linux/inet_diag.h
+linux/inotify.h
+linux/input-event-codes.h
+linux/io_uring.h
+linux/ioam6.h
+linux/ioam6_genl.h
+linux/ioam6_iptunnel.h
+linux/ioctl.h
+linux/iommu.h
+linux/ioprio.h
+linux/ip.h
+linux/ip_vs.h
+linux/ipc.h
+linux/ipmi.h
+linux/ipmi_bmc.h
+linux/ipmi_msgdefs.h
+linux/ipsec.h
+linux/ipv6.h
+linux/ipv6_route.h
+linux/irqnr.h
+linux/isdn/capicmd.h
+linux/iso_fs.h
+linux/isst_if.h
+linux/ivtvfb.h
+linux/jffs2.h
+linux/kcm.h
+linux/kcmp.h
+linux/kcov.h
+linux/kd.h
+linux/kdev_t.h
+linux/kernel-page-flags.h
+linux/kernel.h
+linux/kernelcapi.h
+linux/keyboard.h
+linux/keyctl.h
+linux/kfd_sysfs.h
+linux/kvm_para.h
+linux/l2tp.h
+linux/landlock.h
+linux/libc-compat.h
+linux/limits.h
+linux/lirc.h
+linux/loadpin.h
+linux/loop.h
+linux/lp.h
+linux/lwtunnel.h
+linux/magic.h
+linux/major.h
+linux/map_to_14segment.h
+linux/map_to_7segment.h
+linux/max2175.h
+linux/media-bus-format.h
+linux/media.h
+linux/mei.h
+linux/membarrier.h
+linux/memfd.h
+linux/mempolicy.h
+linux/meye.h
+linux/minix_fs.h
+linux/misc/bcm_vk.h
+linux/mman.h
+linux/mmc/ioctl.h
+linux/mmtimer.h
+linux/module.h
+linux/mount.h
+linux/mpls.h
+linux/mpls_iptunnel.h
+linux/mqueue.h
+linux/mroute.h
+linux/mroute6.h
+linux/mrp_bridge.h
+linux/msdos_fs.h
+linux/msg.h
+linux/mtio.h
+linux/nbd-netlink.h
+linux/nbd.h
+linux/ncsi.h
+linux/ndctl.h
+linux/neighbour.h
+linux/net.h
+linux/net_dropmon.h
+linux/net_namespace.h
+linux/net_tstamp.h
+linux/netconf.h
+linux/netfilter.h
+linux/netfilter/ipset/ip_set.h
+linux/netfilter/ipset/ip_set_bitmap.h
+linux/netfilter/ipset/ip_set_hash.h
+linux/netfilter/ipset/ip_set_list.h
+linux/netfilter/nf_conntrack_common.h
+linux/netfilter/nf_conntrack_ftp.h
+linux/netfilter/nf_conntrack_sctp.h
+linux/netfilter/nf_conntrack_tcp.h
+linux/netfilter/nf_conntrack_tuple_common.h
+linux/netfilter/nf_log.h
+linux/netfilter/nf_nat.h
+linux/netfilter/nf_synproxy.h
+linux/netfilter/nf_tables.h
+linux/netfilter/nf_tables_compat.h
+linux/netfilter/nfnetlink.h
+linux/netfilter/nfnetlink_acct.h
+linux/netfilter/nfnetlink_compat.h
+linux/netfilter/nfnetlink_conntrack.h
+linux/netfilter/nfnetlink_cthelper.h
+linux/netfilter/nfnetlink_cttimeout.h
+linux/netfilter/nfnetlink_hook.h
+linux/netfilter/nfnetlink_log.h
+linux/netfilter/nfnetlink_osf.h
+linux/netfilter/nfnetlink_queue.h
+linux/netfilter/x_tables.h
+linux/netfilter/xt_AUDIT.h
+linux/netfilter/xt_CHECKSUM.h
+linux/netfilter/xt_CLASSIFY.h
+linux/netfilter/xt_CONNMARK.h
+linux/netfilter/xt_CONNSECMARK.h
+linux/netfilter/xt_CT.h
+linux/netfilter/xt_DSCP.h
+linux/netfilter/xt_HMARK.h
+linux/netfilter/xt_IDLETIMER.h
+linux/netfilter/xt_LED.h
+linux/netfilter/xt_LOG.h
+linux/netfilter/xt_MARK.h
+linux/netfilter/xt_NFQUEUE.h
+linux/netfilter/xt_SECMARK.h
+linux/netfilter/xt_SYNPROXY.h
+linux/netfilter/xt_TCPMSS.h
+linux/netfilter/xt_TCPOPTSTRIP.h
+linux/netfilter/xt_TEE.h
+linux/netfilter/xt_TPROXY.h
+linux/netfilter/xt_addrtype.h
+linux/netfilter/xt_bpf.h
+linux/netfilter/xt_cgroup.h
+linux/netfilter/xt_cluster.h
+linux/netfilter/xt_comment.h
+linux/netfilter/xt_connbytes.h
+linux/netfilter/xt_connlabel.h
+linux/netfilter/xt_connlimit.h
+linux/netfilter/xt_connmark.h
+linux/netfilter/xt_conntrack.h
+linux/netfilter/xt_cpu.h
+linux/netfilter/xt_dccp.h
+linux/netfilter/xt_devgroup.h
+linux/netfilter/xt_dscp.h
+linux/netfilter/xt_ecn.h
+linux/netfilter/xt_esp.h
+linux/netfilter/xt_helper.h
+linux/netfilter/xt_ipcomp.h
+linux/netfilter/xt_iprange.h
+linux/netfilter/xt_ipvs.h
+linux/netfilter/xt_l2tp.h
+linux/netfilter/xt_length.h
+linux/netfilter/xt_limit.h
+linux/netfilter/xt_mac.h
+linux/netfilter/xt_mark.h
+linux/netfilter/xt_multiport.h
+linux/netfilter/xt_nfacct.h
+linux/netfilter/xt_osf.h
+linux/netfilter/xt_owner.h
+linux/netfilter/xt_pkttype.h
+linux/netfilter/xt_policy.h
+linux/netfilter/xt_quota.h
+linux/netfilter/xt_realm.h
+linux/netfilter/xt_recent.h
+linux/netfilter/xt_rpfilter.h
+linux/netfilter/xt_sctp.h
+linux/netfilter/xt_set.h
+linux/netfilter/xt_socket.h
+linux/netfilter/xt_state.h
+linux/netfilter/xt_statistic.h
+linux/netfilter/xt_string.h
+linux/netfilter/xt_tcpmss.h
+linux/netfilter/xt_tcpudp.h
+linux/netfilter/xt_time.h
+linux/netfilter/xt_u32.h
+linux/netfilter_arp.h
+linux/netfilter_bridge/ebt_among.h
+linux/netfilter_bridge/ebt_arp.h
+linux/netfilter_bridge/ebt_arpreply.h
+linux/netfilter_bridge/ebt_ip.h
+linux/netfilter_bridge/ebt_ip6.h
+linux/netfilter_bridge/ebt_limit.h
+linux/netfilter_bridge/ebt_log.h
+linux/netfilter_bridge/ebt_mark_m.h
+linux/netfilter_bridge/ebt_mark_t.h
+linux/netfilter_bridge/ebt_nat.h
+linux/netfilter_bridge/ebt_nflog.h
+linux/netfilter_bridge/ebt_pkttype.h
+linux/netfilter_bridge/ebt_redirect.h
+linux/netfilter_bridge/ebt_stp.h
+linux/netfilter_bridge/ebt_vlan.h
+linux/netfilter_ipv4.h
+linux/netfilter_ipv4/ipt_CLUSTERIP.h
+linux/netfilter_ipv4/ipt_ECN.h
+linux/netfilter_ipv4/ipt_LOG.h
+linux/netfilter_ipv4/ipt_REJECT.h
+linux/netfilter_ipv4/ipt_TTL.h
+linux/netfilter_ipv4/ipt_ah.h
+linux/netfilter_ipv4/ipt_ecn.h
+linux/netfilter_ipv4/ipt_ttl.h
+linux/netfilter_ipv6.h
+linux/netfilter_ipv6/ip6t_HL.h
+linux/netfilter_ipv6/ip6t_LOG.h
+linux/netfilter_ipv6/ip6t_NPT.h
+linux/netfilter_ipv6/ip6t_REJECT.h
+linux/netfilter_ipv6/ip6t_ah.h
+linux/netfilter_ipv6/ip6t_frag.h
+linux/netfilter_ipv6/ip6t_hl.h
+linux/netfilter_ipv6/ip6t_ipv6header.h
+linux/netfilter_ipv6/ip6t_mh.h
+linux/netfilter_ipv6/ip6t_opts.h
+linux/netfilter_ipv6/ip6t_rt.h
+linux/netfilter_ipv6/ip6t_srh.h
+linux/netlink.h
+linux/netlink_diag.h
+linux/netrom.h
+linux/nexthop.h
+linux/nfc.h
+linux/nfs.h
+linux/nfs2.h
+linux/nfs3.h
+linux/nfs4.h
+linux/nfs4_mount.h
+linux/nfs_fs.h
+linux/nfs_idmap.h
+linux/nfs_mount.h
+linux/nfsacl.h
+linux/nfsd/cld.h
+linux/nfsd/debug.h
+linux/nfsd/export.h
+linux/nfsd/stats.h
+linux/nilfs2_api.h
+linux/nilfs2_ondisk.h
+linux/nitro_enclaves.h
+linux/nl80211-vnd-intel.h
+linux/nl80211.h
+linux/nsfs.h
+linux/nubus.h
+linux/nvme_ioctl.h
+linux/nvram.h
+linux/oom.h
+linux/openat2.h
+linux/openvswitch.h
+linux/packet_diag.h
+linux/param.h
+linux/parport.h
+linux/pci.h
+linux/pci_regs.h
+linux/pcitest.h
+linux/perf_event.h
+linux/personality.h
+linux/pfkeyv2.h
+linux/pfrut.h
+linux/pg.h
+linux/phantom.h
+linux/pidfd.h
+linux/pkt_cls.h
+linux/pkt_sched.h
+linux/pktcdvd.h
+linux/pmu.h
+linux/poll.h
+linux/posix_acl.h
+linux/posix_types.h
+linux/ppdev.h
+linux/ppp-comp.h
+linux/ppp-ioctl.h
+linux/ppp_defs.h
+linux/pps.h
+linux/pr.h
+linux/prctl.h
+linux/psample.h
+linux/psci.h
+linux/psp-sev.h
+linux/ptp_clock.h
+linux/qemu_fw_cfg.h
+linux/qnx4_fs.h
+linux/qnxtypes.h
+linux/qrtr.h
+linux/radeonfb.h
+linux/raid/md_p.h
+linux/raid/md_u.h
+linux/random.h
+linux/rds.h
+linux/reboot.h
+linux/reiserfs_fs.h
+linux/reiserfs_xattr.h
+linux/remoteproc_cdev.h
+linux/resource.h
+linux/rfkill.h
+linux/rio_cm_cdev.h
+linux/rio_mport_cdev.h
+linux/rkisp1-config.h
+linux/romfs_fs.h
+linux/rose.h
+linux/rpl.h
+linux/rpl_iptunnel.h
+linux/rpmsg.h
+linux/rpmsg_types.h
+linux/rseq.h
+linux/rtc.h
+linux/rtnetlink.h
+linux/rxrpc.h
+linux/scc.h
+linux/sched.h
+linux/sched/types.h
+linux/scif_ioctl.h
+linux/screen_info.h
+linux/seccomp.h
+linux/securebits.h
+linux/sed-opal.h
+linux/seg6.h
+linux/seg6_genl.h
+linux/seg6_hmac.h
+linux/seg6_iptunnel.h
+linux/seg6_local.h
+linux/selinux_netlink.h
+linux/sem.h
+linux/serial_reg.h
+linux/serio.h
+linux/sev-guest.h
+linux/shm.h
+linux/signalfd.h
+linux/smc.h
+linux/smc_diag.h
+linux/smiapp.h
+linux/snmp.h
+linux/socket.h
+linux/sockios.h
+linux/sonet.h
+linux/sonypi.h
+linux/sound.h
+linux/soundcard.h
+linux/spi/spi.h
+linux/spi/spidev.h
+linux/stat.h
+linux/stddef.h
+linux/stm.h
+linux/string.h
+linux/sunrpc/debug.h
+linux/surface_aggregator/cdev.h
+linux/surface_aggregator/dtx.h
+linux/suspend_ioctls.h
+linux/swab.h
+linux/switchtec_ioctl.h
+linux/sync_file.h
+linux/sysinfo.h
+linux/target_core_user.h
+linux/taskstats.h
+linux/tc_act/tc_bpf.h
+linux/tc_act/tc_csum.h
+linux/tc_act/tc_ct.h
+linux/tc_act/tc_defact.h
+linux/tc_act/tc_gact.h
+linux/tc_act/tc_gate.h
+linux/tc_act/tc_ife.h
+linux/tc_act/tc_ipt.h
+linux/tc_act/tc_mirred.h
+linux/tc_act/tc_mpls.h
+linux/tc_act/tc_nat.h
+linux/tc_act/tc_pedit.h
+linux/tc_act/tc_sample.h
+linux/tc_act/tc_skbedit.h
+linux/tc_act/tc_skbmod.h
+linux/tc_act/tc_tunnel_key.h
+linux/tc_act/tc_vlan.h
+linux/tc_ematch/tc_em_cmp.h
+linux/tc_ematch/tc_em_ipt.h
+linux/tc_ematch/tc_em_meta.h
+linux/tc_ematch/tc_em_nbyte.h
+linux/tc_ematch/tc_em_text.h
+linux/tcp.h
+linux/tcp_metrics.h
+linux/tee.h
+linux/termios.h
+linux/thermal.h
+linux/time.h
+linux/time_types.h
+linux/timerfd.h
+linux/times.h
+linux/timex.h
+linux/tiocl.h
+linux/tipc.h
+linux/tipc_config.h
+linux/tipc_netlink.h
+linux/tls.h
+linux/toshiba.h
+linux/tty.h
+linux/tty_flags.h
+linux/types.h
+linux/ublk_cmd.h
+linux/udf_fs_i.h
+linux/udmabuf.h
+linux/udp.h
+linux/uio.h
+linux/uleds.h
+linux/ultrasound.h
+linux/um_timetravel.h
+linux/un.h
+linux/unistd.h
+linux/unix_diag.h
+linux/usb/cdc-wdm.h
+linux/usb/ch11.h
+linux/usb/ch9.h
+linux/usb/charger.h
+linux/usb/functionfs.h
+linux/usb/g_printer.h
+linux/usb/g_uvc.h
+linux/usb/gadgetfs.h
+linux/usb/midi.h
+linux/usb/raw_gadget.h
+linux/usb/tmc.h
+linux/usb/video.h
+linux/usbdevice_fs.h
+linux/usbip.h
+linux/userfaultfd.h
+linux/userio.h
+linux/utime.h
+linux/utsname.h
+linux/uuid.h
+linux/uvcvideo.h
+linux/v4l2-common.h
+linux/v4l2-controls.h
+linux/v4l2-dv-timings.h
+linux/vbox_err.h
+linux/vbox_vmmdev_types.h
+linux/vboxguest.h
+linux/vdpa.h
+linux/vduse.h
+linux/veth.h
+linux/vfio.h
+linux/vfio_ccw.h
+linux/vfio_zdev.h
+linux/virtio_9p.h
+linux/virtio_balloon.h
+linux/virtio_blk.h
+linux/virtio_bt.h
+linux/virtio_config.h
+linux/virtio_console.h
+linux/virtio_crypto.h
+linux/virtio_fs.h
+linux/virtio_gpio.h
+linux/virtio_gpu.h
+linux/virtio_i2c.h
+linux/virtio_ids.h
+linux/virtio_input.h
+linux/virtio_iommu.h
+linux/virtio_mem.h
+linux/virtio_mmio.h
+linux/virtio_net.h
+linux/virtio_pci.h
+linux/virtio_pcidev.h
+linux/virtio_pmem.h
+linux/virtio_rng.h
+linux/virtio_scmi.h
+linux/virtio_scsi.h
+linux/virtio_snd.h
+linux/virtio_types.h
+linux/virtio_vsock.h
+linux/vm_sockets_diag.h
+linux/vmcore.h
+linux/vsockmon.h
+linux/vt.h
+linux/vtpm_proxy.h
+linux/wait.h
+linux/watch_queue.h
+linux/watchdog.h
+linux/wireguard.h
+linux/wmi.h
+linux/wwan.h
+linux/x25.h
+linux/xattr.h
+linux/xdp_diag.h
+linux/xfrm.h
+linux/xilinx-v4l2-controls.h
+linux/zorro.h
+linux/zorro_ids.h
Signature · 1 line
2.34.1
author maintainer
#27 lore raw

RFC 12/12 selftests/bpf: script for infer_header_guards.pl testing

In reply to Eduard Zingerman

This script verifies that patterns for header guard inference specified in scripts/infer_header_guards.pl cover all uapi headers. To achieve this the infer_header_guards.pl is invoked the same way it is invoked from link-vmlinux.sh but with --report-failures flag.

---
 .../bpf/test_uapi_header_guards_infer.sh      | 33 +++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh
tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh new file +33 -0
new file mode 100755
index 000000000000..bd332db100f3
@@ -0,0 +1,33 @@
+#!/bin/bash
+# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
+
+# This script verifies that patterns for header guard inference
+# specified in scripts/infer_header_guards.pl cover all uapi headers.
+# To achieve this the infer_header_guards.pl is invoked the same way
+# it is invoked from link-vmlinux.sh but with --report-failures flag.
+
+kernel_dir=$(dirname $0)/../../../../
+
+# The SRCARCH is defined in tools/scripts/Makefile.arch, thus use a
+# temporary makefile to get access to this variable.
+fake_makefile=$(cat <<EOF
+include tools/scripts/Makefile.arch
+default:
+	scripts/infer_header_guards.pl --report-failures \
+		include/uapi \
+		include/generated/uapi \
+		arch/\$(SRCARCH)/include/uapi \
+		arch/\$(SRCARCH)/include/generated/uapi 1>/dev/null
+EOF
+)
+
+# The infer_header_guards.pl script prints inferred guards to stdout,
+# redirecting stdout to /dev/null to see only error messages.
+echo "$fake_makefile" | make -C $kernel_dir -f - 1>/dev/null
+if [ "$?" == "0" ]; then
+	echo "all good"
+	exit 0
+fi
+
+# Failures are already reported by infer_header_guards.pl
+exit 1
Signature · 1 line
2.34.1
#28 lore raw
In reply to Eduard Zingerman
Quoted 31 lines · Eduard Zingerman

Include the following system header:
- /usr/include/sys/socket.h (all via linux/if.h)

The sys/socket.h conflicts with vmlinux.h in:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants: SOCK_STREAM, SOCK_DGRAM, ...

However, only two types are actually used:
- struct sockaddr
- struct sockaddr_storage (used only in linux/mptcp.h)

In 'vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.

The only workaround that I see is to:
- define a stub sys/socket.h as follows:

    #ifndef __BPF_SOCKADDR__
    #define __BPF_SOCKADDR__
    
    /* For __kernel_sa_family_t */
    #include <linux/socket.h>
    
    struct sockaddr {
        __kernel_sa_family_t sa_family;
        char sa_data[14];
    };
    
    #endif
Quoted 2 lines

- hardcode generation of __BPF_SOCKADDR__ bracket for 'struct sockaddr' in vmlinux.h.

we don't need to hack sys/socket.h and can hardcode #ifdef _SYS_SOCKET_H as header guard for sockaddr instead, right? bits/socket.h has this:
#ifndef _SYS_SOCKET_H

# error "Never include <bits/socket.h> directly; use <sys/socket.h> instead."

So that ifdef is kinda stable.

Quoted 16 lines

Another possibility is to move the definition of 'struct sockaddr' from 'kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.

Conflict with vmlinux.h
----

Uapi header:
- linux/signal.h

Conflict with vmlinux.h in definition of 'struct sigaction'. Defined in:
- vmlinux.h: kernel/include/linux/signal_types.h
- uapi: kernel/arch/x86/include/asm/signal.h

Quoted 5 lines

Uapi headers:
- linux/tipc_sockets_diag.h
- linux/sock_diag.h

Conflict with vmlinux.h in definition of 'SOCK_DESTROY'.

Interesting one!
I think we can hard code '#undef SOCK_DESTROY' in vmlinux.h

The goal is not to be able to mix arbitrary uapi header with vmlinux.h, but only those that could be useful out of bpf progs. Currently it's tcp.h and few other network related headers because they have #define-s in them that are useful inside bpf progs. As long as the solution covers this small subset we're fine.

#30 lore raw
In reply to Eduard Zingerman
Quoted 8 lines · Eduard Zingerman

Hi BPF community,

AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.

Thanks for the details! It's a tricky problem to solve.

Before diving into this, can I ask if there's another way round this; is there no way we could teach vmlinux.h generation which types to skip via some kind of bootstrapping process?

For a bpf object foo.bpf.c that wants to include linux/tcp.h and vmlinux.h, something like this seems possible;

1. run the preprocessor (gcc -E) over the BPF program to generate
a bootstrap header foo.bpf.exclude_types.h consisting of all the types mentioned in it and associated includes, but not in vmlinux.h It would need to -D__VMLINUX_H__ to avoid including vmlinux.h definitions and -D__BPF_EXCLUDE_TYPE_BOOTSTRAP__ to skip the programs themselves, which would need a guard around them I think:

#include <stddef.h>
#include <stdbool.h>
#include <vmlinux.h>
#include <linux/tcp.h>

#ifndef __BPF_EXCLUDE_TYPE_BOOTSTRAP__
//rest of program here
#endif

So as a result of this, we now have a single header file that contains all the types that non-vmlinux.h include files define.

2. now to generate vmlinux.h, pass foo.bpf.exclude_types.h into "bpftool btf dump" as an exclude header:

bpftool btf dump --exclude /tmp/foo.bpf.types.h file /sys/kernel/btf/vmlinux format c > vmlinux.h

bpftool would have to parse the exclude header for actual type definitions, spotting struct, enum and typedef definitions. This is likely made easier by running the preprocessor at least since formatting is probably quite uniform. vmlinux.h could simply emit forward declarations for types described both in vmlinux BTF and in the exclude file.

So the build process would be

- start with empty vmlinux.h
- bootstrap a header consisting of the set of types to exclude via c preprocessor
- generate new vmlinux.h based on above
- build bpf program

Build processes for BPF objects already has bootstrapping elements like this for generating vmlinux.h and skeletons, so while it's not perfect it might be a simpler approach. There may be problems with this I'm not seeing though?

Thanks!

Alan

Quoted 359 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

The basic idea
---

The goal of the patch set is to allow usage of header files from include/uapi alongside vmlinux.h as follows:

  #include <uapi/linux/tcp.h>
  #include "vmlinux.h"

This goal is achieved by adding #ifndef ... #endif guards in vmlinux.h around definitions that originate from the include/uapi headers. The guards emitted match the guards used in the original headers. E.g. as follows:

include/uapi/linux/tcp.h:

#ifndef _UAPI_LINUX_TCP_H
#define _UAPI_LINUX_TCP_H
...

  union tcp_word_hdr {
	struct tcphdr hdr;
	__be32        words[5];

};
...
#endif /* _UAPI_LINUX_TCP_H */

vmlinux.h:

  ...
  #ifndef _UAPI_LINUX_TCP_H
  union tcp_word_hdr {
	struct tcphdr hdr;
	__be32 words[5];

};

  #endif /* _UAPI_LINUX_TCP_H */
  ...

To get to this state the following steps are necessary:
- "header guard" name should be identified for each header file;
- the correspondence between data type and it's header guard has to be encoded in BTF;
- bpftool should be adjusted to emit #ifndef FOO_H ... #endif brackets.

It is not possible to identify header guard names for all uapi headers basing only on the file name. However a simple script could devised to identify the guards basing on the file name and it's content. Thus it is possible to obtain the list of header names with corresponding header guards.

The correspondence between type and it's declaration file (header) is available in DWARF as DW_AT_decl_file attribute. The DW_AT_decl_file can be matched with the list of header guards described above to obtain the header guard name for a specific type.

The pahole generates BTF using DWARF. It is possible to modify pahole to accept the header guards list as an additional parameter and to encode the header guard names in BTF.

Implementation details
---

Present patch-set implements these ideas as follows:
- A parameter --header_guards_db is added to pahole. If present it points to a file with a list of <header> <guard> records.
- pahole uses DWARF DW_AT_decl_file value to lookup the header guard for each type emitted to BTF. If header guard is present it is encoded alongside the type.
- Header guards are encoded in BTF as BTF_DECL_TAG records with a special prefix. The prefix "header_guard:" is added to a value of such tags. (Here BTF_DECL_TAG is used to avoid BTF binary format changes).
- A special script infer_header_guards.pl is added as a part of kbuild, it can infer header guard names for each UAPI header basing on the header content.
- This script is invoked from link-vmlinux.sh prior to BTF generation during kernel build. The output of the script is saved to a file, the file is passed to pahole as --header_guards_db parameter.
- libbpf is modified to aggregate BTF_DECL_TAG records for each type and to emit #ifndef FOO_H ... #endif brackets when "header_guard:" tag is present for a type.

Details for each patch in a set:
- libbpf: Deduplicate unambigous standalone forward declarations
- selftests/bpf: Tests for standalone forward BTF declarations deduplication

There is a small number (63 for defconfig) of forward declarations that are not de-duplicated with the main type declaration under certain conditions. This hinders the header guard brackets generation. This patch addresses this de-duplication issue.

- libbpf: Support for BTF_DECL_TAG dump in C format
- selftests/bpf: Tests for BTF_DECL_TAG dump in C format

Currently libbpf does not process BTF_DECL_TAG when btf is dumped in C format. This patch adds a hash table matching btf type ids with a list of decl tags to the struct btf_dump.
The btf_dump_emit_decl_tags is not necessary for the overall patch-set to function but simplifies testing a bit.

- libbpf: Header guards for selected data structures in vmlinux.h
- selftests/bpf: Tests for header guards printing in BTF dump

Adds option emit_header_guards to struct btf_dump_opts. When enabled the btf_dump__dump_type prints #ifndef ... #endif brackets around types for which header guard information is present in BTF.

- bpftool: Enable header guards generation

Unconditionally enables emit_header_guards for BTF dump in C format.

- kbuild: Script to infer header guard values for uapi headers
- kbuild: Header guards for types from include/uapi/*.h in kernel BTF

Adds scripts/infer_header_guards.pl and integrates it with link-vmlinux.sh.

- selftests/bpf: Script to verify uapi headers usage with vmlinux.h

Adds a script test_uapi_headers.py that tests header guards with vmlinux.h by compiling a simple C snippet. The snippet looks as follows:

    #include <some_uapi_header.h>
    #include "vmlinux.h"
  
    __attribute__((section("tc"), used))
    int syncookie_tc(struct __sk_buff *skb) { return 0; }

The list of headers to test comes from tools/testing/selftests/bpf/good_uapi_headers.txt.

- selftests/bpf: Known good uapi headers for test_uapi_headers.py

The list of uapi headers that could be included alongside vmlinux.h. The headers are peeked from the following locations:
- <headers-export-dir>/linux/*.h
- <headers-export-dir>/linux/**/*.h
This choice of locations is somewhat arbitrary.

- selftests/bpf: script for infer_header_guards.pl testing

The test case for scripts/infer_header_guards.pl, verifies that header guards can be inferred for all uapi headers.

- There is also a patch for dwarves that adds --header_guards_db option (see [2]).

The test_uapi_headers.py is important as it demonstrates the the necessary compiler flags:

clang ...                                  \
      -D__x86_64__                         \
      -Xclang -fwchar-type=short           \
      -Xclang -fno-signed-wchar            \
      -I{exported_kernel_headers}/include/ \
      ...

- -fwchar-type=short and -fno-signed-wchar had to be added because BPF target uses int for wchar_t by default and this differs from vmlinux.h definition of the type (at least for x86_64).
- __x86_64__ had to be added for uapi headers that include stddef.h (the one that is supplied my CLANG itself), in order to define correct sizes for size_t and ptrdiff_t.
- The {exported_kernel_headers} stands for exported kernel headers directory (the headers obtained by make headers_install or via distribution package).

When it works
---

The mechanics described above works for a significant number of UAPI headers. For example, for the test case above I chose the headers from the following locations:
- linux/*.h
- linux/**/*.h
There are 759 such headers and for 677 of them the test described above passes.

I excluded the headers from the following sub-directories as potentially not interesting:

  asm          rdma   video xen
  asm-generic  misc   scsi
  drm          mtd    sound

Thus saving some time for both discussion and CI but the choice is somewhat arbitrary. If I run test_uapi_headers.py --test '*' (all headers) test passes for 834 out of 972 headers.

When it breaks
---

There several scenarios when this mechanics breaks. Specifically I found the following cases:
- When uapi header includes some system header that conflicts with vmlinux.h.
- When uapi header itself conflicts with vmlinux.h.

Below are examples for both cases.

Conflict with system headers
----

The following uapi headers:
- linux/atmbr2684.h
- linux/bpfilter.h
- linux/gsmmux.h
- linux/icmp.h
- linux/if.h
- linux/if_arp.h
- linux/if_bonding.h
- linux/if_pppox.h
- linux/if_tunnel.h
- linux/ip6_tunnel.h
- linux/llc.h
- linux/mctp.h
- linux/mptcp.h
- linux/netdevice.h
- linux/netfilter/xt_RATEEST.h
- linux/netfilter/xt_hashlimit.h
- linux/netfilter/xt_physdev.h
- linux/netfilter/xt_rateest.h
- linux/netfilter_arp/arp_tables.h
- linux/netfilter_arp/arpt_mangle.h
- linux/netfilter_bridge.h
- linux/netfilter_bridge/ebtables.h
- linux/netfilter_ipv4/ip_tables.h
- linux/netfilter_ipv6/ip6_tables.h
- linux/route.h
- linux/wireless.h

Include the following system header:
- /usr/include/sys/socket.h (all via linux/if.h)

The sys/socket.h conflicts with vmlinux.h in:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants: SOCK_STREAM, SOCK_DGRAM, ...

However, only two types are actually used:
- struct sockaddr
- struct sockaddr_storage (used only in linux/mptcp.h)

In 'vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.

The only workaround that I see is to:
- define a stub sys/socket.h as follows:

    #ifndef __BPF_SOCKADDR__
    #define __BPF_SOCKADDR__
    
    /* For __kernel_sa_family_t */
    #include <linux/socket.h>
    
    struct sockaddr {
        __kernel_sa_family_t sa_family;
        char sa_data[14];
    };
    
    #endif

- hardcode generation of __BPF_SOCKADDR__ bracket for 'struct sockaddr' in vmlinux.h.

Another possibility is to move the definition of 'struct sockaddr' from 'kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.

Conflict with vmlinux.h
----

Uapi header:
- linux/signal.h

Conflict with vmlinux.h in definition of 'struct sigaction'. Defined in:
- vmlinux.h: kernel/include/linux/signal_types.h
- uapi: kernel/arch/x86/include/asm/signal.h

Uapi headers:
- linux/tipc_sockets_diag.h
- linux/sock_diag.h

Conflict with vmlinux.h in definition of 'SOCK_DESTROY'. Defined in:
- vmlinux.h: kernel/include/net/sock.h
- uapi: kernel/include/uapi/linux/sock_diag.h
Constants seem to be unrelated.

And so on... I have details for many other headers but omit those for brevity.

In conclusion
---

Except from the general feasibility I have a few questions:
- What UAPI headers are the candidates for such use? If there are some interesting headers currently not working with this patch-set some hacks have to be added (e.g. like with linux/if.h).
- Is it ok to encode header guards as special BTF_DECL_TAG or should I change the BTF format a bit to save some bytes.

Thanks,
Eduard

[1] https://reviews.llvm.org/D111307
[clang] __attribute__ bpf_dominating_decl
[2] https://lore.kernel.org/dwarves/[email protected]/T/
[RFC dwarves] pahole: Save header guard names when
--header_guards_db is passed

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

 scripts/infer_header_guards.pl                | 191 +++++
 scripts/link-vmlinux.sh                       |  13 +-
 tools/bpf/bpftool/btf.c                       |   4 +-
 tools/lib/bpf/btf.c                           | 178 ++++-
 tools/lib/bpf/btf.h                           |   7 +-
 tools/lib/bpf/btf_dump.c                      | 232 +++++-
 .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
 tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
 .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
 .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
 .../progs/btf_dump_test_case_header_guards.c  |  94 +++
 .../bpf/test_uapi_header_guards_infer.sh      |  33 +
 .../selftests/bpf/test_uapi_headers.py        | 197 +++++
 13 files changed, 1816 insertions(+), 12 deletions(-)
 create mode 100755 scripts/infer_header_guards.pl
 create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c
 create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh
 create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
author maintainer
#29 lore raw
In reply to Alexei Starovoitov
Quoted 35 lines · Alexei Starovoitov
Quoted 31 lines · Eduard Zingerman

Include the following system header:
- /usr/include/sys/socket.h (all via linux/if.h)

The sys/socket.h conflicts with vmlinux.h in:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants: SOCK_STREAM, SOCK_DGRAM, ...

However, only two types are actually used:
- struct sockaddr
- struct sockaddr_storage (used only in linux/mptcp.h)

In 'vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.

The only workaround that I see is to:
- define a stub sys/socket.h as follows:

    #ifndef __BPF_SOCKADDR__
    #define __BPF_SOCKADDR__
    
    /* For __kernel_sa_family_t */
    #include <linux/socket.h>
    
    struct sockaddr {
        __kernel_sa_family_t sa_family;
        char sa_data[14];
    };
    
    #endif
Quoted 2 lines

- hardcode generation of __BPF_SOCKADDR__ bracket for 'struct sockaddr' in vmlinux.h.

Quoted 7 lines

we don't need to hack sys/socket.h and can hardcode #ifdef _SYS_SOCKET_H as header guard for sockaddr instead, right? bits/socket.h has this:
#ifndef _SYS_SOCKET_H

# error "Never include <bits/socket.h> directly; use <sys/socket.h> instead."

So that ifdef is kinda stable.

The if.h only uses two types from sys/socket.h, namely:
- struct sockaddr
- struct sockaddr_storage

However sys/socket.h itself defines more types, here is a complete list of types from sys/socket.h that conflict with vmlinux.h (generated for my x86_64 laptop):

Type name       System header
iovec           /usr/include/bits/types/struct_iovec.h
loff_t          /usr/include/sys/types.h
dev_t           /usr/include/sys/types.h
nlink_t         /usr/include/sys/types.h
timer_t         /usr/include/bits/types/timer_t.h
int16_t         /usr/include/bits/stdint-intn.h
int32_t         /usr/include/bits/stdint-intn.h
int64_t         /usr/include/bits/stdint-intn.h
u_int64_t       /usr/include/sys/types.h
sigset_t        /usr/include/bits/types/sigset_t.h
fd_set          /usr/include/sys/select.h
blkcnt_t        /usr/include/sys/types.h
SOCK_STREAM     /usr/include/bits/socket_type.h
SOCK_DGRAM      /usr/include/bits/socket_type.h
SOCK_RAW        /usr/include/bits/socket_type.h
SOCK_RDM        /usr/include/bits/socket_type.h
SOCK_SEQPACKET  /usr/include/bits/socket_type.h
SOCK_DCCP       /usr/include/bits/socket_type.h
SOCK_PACKET     /usr/include/bits/socket_type.h
sockaddr        /usr/include/bits/socket.h
msghdr          /usr/include/bits/socket.h
cmsghdr         /usr/include/bits/socket.h
linger          /usr/include/bits/socket.h
SHUT_RD         /usr/include/sys/socket.h
SHUT_WR         /usr/include/sys/socket.h
SHUT_RDWR       /usr/include/sys/socket.h

It would be safe to wrap the corresponding types in the vmlinux.h with _SYS_SOCKET_H / _SYS_TYPES guards if the definitions above match between libc and kernel. To my surprise not all of them match. Here is the list of genuine conflicts (for typedefs I skip the intermediate definitions and print the last typedef in the chain):

Type: dev_t
typedef unsigned int __u32 vmlinux.h
typedef unsigned long int __dev_t /usr/include/bits/types.h

Type: nlink_t
typedef unsigned int __u32 vmlinux.h
typedef unsigned long int __nlink_t /usr/include/bits/types.h

Type: timer_t
typedef int __kernel_timer _t vmlinux.h
typedef void *__timer_t /usr/include/bits/types.h

Type: sigset_t
typedef struct vmlinux.h
{
long unsigned int sig[1];
} sigset_t

typedef struct /usr/include/bits/types/__sigset_t.h
{
unsigned long int __val[1024 / (8 * (sizeof(unsigned long int)))];
} __sigset_t

Type: fd_set
typedef struct vmlinux.h
{
long unsigned int fds_bits[16];
} __kernel_fd_set

typedef struct /usr/include/sys/select.h
{
__fd_mask __fds_bits[1024 / (8 * ((int) (sizeof(__fd_mask))))];
} fd_set

Type: sigset_t
typedef struct vmlinux.h
{
long unsigned int sig[1];
} sigset_t

typedef struct /usr/include/bits/types/__sigset_t.h
{
unsigned long int __val[1024 / (8 * (sizeof(unsigned long int)))];
} __sigset_t

Type: msghdr
struct msghdr vmlinux.h
{
void *msg_name;
int msg_namelen;
int msg_inq;
struct iov_iter msg_iter;
union

  {
    void *msg_control;
    void *msg_control_user;

};
bool msg_control_is_user : 1;
bool msg_get_inq : 1;
unsigned int msg_flags;
__kernel_size_t msg_controllen;
struct kiocb *msg_iocb;
struct ubuf_info *msg_ubuf;
struct sock *, struct sk_buff *, struct iov_iter *, size_tint;
}

struct msghdr /usr/include/bits/socket.h
{
void *msg_name;
socklen_t msg_namelen;
struct iovec *msg_iov;
size_t msg_iovlen;
void *msg_control;
size_t msg_controllen;
int msg_flags;
}

Quoted 23 lines
Quoted 17 lines

Another possibility is to move the definition of 'struct sockaddr' from 'kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.

Conflict with vmlinux.h
----

Uapi header:
- linux/signal.h

Conflict with vmlinux.h in definition of 'struct sigaction'. Defined in:
- vmlinux.h: kernel/include/linux/signal_types.h
- uapi: kernel/arch/x86/include/asm/signal.h

Quoted 5 lines

Uapi headers:
- linux/tipc_sockets_diag.h
- linux/sock_diag.h

Conflict with vmlinux.h in definition of 'SOCK_DESTROY'.

Quoted 8 lines

Interesting one!
I think we can hard code '#undef SOCK_DESTROY' in vmlinux.h

The goal is not to be able to mix arbitrary uapi header with vmlinux.h, but only those that could be useful out of bpf progs. Currently it's tcp.h and few other network related headers because they have #define-s in them that are useful inside bpf progs. As long as the solution covers this small subset we're fine.

Well, tcp.h works :) It would be great if someone could list the interesting headers.

Thanks,
Eduard

author maintainer
#31 lore raw
In reply to Alan Maguire
Quoted 49 lines · Alan Maguire
Quoted 8 lines · Eduard Zingerman

Hi BPF community,

AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.

Thanks for the details! It's a tricky problem to solve.

Before diving into this, can I ask if there's another way round this; is there no way we could teach vmlinux.h generation which types to skip via some kind of bootstrapping process?

For a bpf object foo.bpf.c that wants to include linux/tcp.h and vmlinux.h, something like this seems possible;

1. run the preprocessor (gcc -E) over the BPF program to generate
a bootstrap header foo.bpf.exclude_types.h consisting of all the types mentioned in it and associated includes, but not in vmlinux.h It would need to -D__VMLINUX_H__ to avoid including vmlinux.h definitions and -D__BPF_EXCLUDE_TYPE_BOOTSTRAP__ to skip the programs themselves, which would need a guard around them I think:

#include <stddef.h>
#include <stdbool.h>
#include <vmlinux.h>
#include <linux/tcp.h>

#ifndef __BPF_EXCLUDE_TYPE_BOOTSTRAP__
//rest of program here
#endif

So as a result of this, we now have a single header file that contains all the types that non-vmlinux.h include files define.

2. now to generate vmlinux.h, pass foo.bpf.exclude_types.h into "bpftool btf dump" as an exclude header:

bpftool btf dump --exclude /tmp/foo.bpf.types.h file /sys/kernel/btf/vmlinux format c > vmlinux.h

bpftool would have to parse the exclude header for actual type definitions, spotting struct, enum and typedef definitions. This is likely made easier by running the preprocessor at least since formatting is probably quite uniform. vmlinux.h could simply emit forward declarations for types described both in vmlinux BTF and in the exclude file.

So the build process would be

Quoted 8 lines

- start with empty vmlinux.h
- bootstrap a header consisting of the set of types to exclude via c preprocessor
- generate new vmlinux.h based on above
- build bpf program

Build processes for BPF objects already has bootstrapping elements like this for generating vmlinux.h and skeletons, so while it's not perfect it might be a simpler approach. There may be problems with this I'm not seeing though?

I like the tool based approach more but I heard there were some reservations about separate tool complicating the build process.
- the tool does not require changes to the kbuild;
- the tool is not limited to uapi headers;
- the tool could imitate the dominating declarations attribute and address the issue with definitions miss-match between vmlinux.h and system headers (see my reply to Alexei in the parallel sub-thread). To support this point it would have to work in a somewhat different order:
- read/pre-process the input file;
- remove from it the definitions that are also present in kernel BTF;
- prepend vmlinux.h to the beginning of the file.

On the other hand I think that implementation would need a C parser / type analysis step. Thus it would be harder to implement it as a part of bpftool. But the prototype using something like [1] should be simple.

Thanks,
Eduard

[1] https://github.com/inducer/pycparserext

Quoted 364 lines

Thanks!

Alan

Quoted 359 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

The basic idea
---

The goal of the patch set is to allow usage of header files from include/uapi alongside vmlinux.h as follows:

  #include <uapi/linux/tcp.h>
  #include "vmlinux.h"

This goal is achieved by adding #ifndef ... #endif guards in vmlinux.h around definitions that originate from the include/uapi headers. The guards emitted match the guards used in the original headers. E.g. as follows:

include/uapi/linux/tcp.h:

#ifndef _UAPI_LINUX_TCP_H
#define _UAPI_LINUX_TCP_H
...

  union tcp_word_hdr {
	struct tcphdr hdr;
	__be32        words[5];

};
...
#endif /* _UAPI_LINUX_TCP_H */

vmlinux.h:

  ...
  #ifndef _UAPI_LINUX_TCP_H
  union tcp_word_hdr {
	struct tcphdr hdr;
	__be32 words[5];

};

  #endif /* _UAPI_LINUX_TCP_H */
  ...

To get to this state the following steps are necessary:
- "header guard" name should be identified for each header file;
- the correspondence between data type and it's header guard has to be encoded in BTF;
- bpftool should be adjusted to emit #ifndef FOO_H ... #endif brackets.

It is not possible to identify header guard names for all uapi headers basing only on the file name. However a simple script could devised to identify the guards basing on the file name and it's content. Thus it is possible to obtain the list of header names with corresponding header guards.

The correspondence between type and it's declaration file (header) is available in DWARF as DW_AT_decl_file attribute. The DW_AT_decl_file can be matched with the list of header guards described above to obtain the header guard name for a specific type.

The pahole generates BTF using DWARF. It is possible to modify pahole to accept the header guards list as an additional parameter and to encode the header guard names in BTF.

Implementation details
---

Present patch-set implements these ideas as follows:
- A parameter --header_guards_db is added to pahole. If present it points to a file with a list of <header> <guard> records.
- pahole uses DWARF DW_AT_decl_file value to lookup the header guard for each type emitted to BTF. If header guard is present it is encoded alongside the type.
- Header guards are encoded in BTF as BTF_DECL_TAG records with a special prefix. The prefix "header_guard:" is added to a value of such tags. (Here BTF_DECL_TAG is used to avoid BTF binary format changes).
- A special script infer_header_guards.pl is added as a part of kbuild, it can infer header guard names for each UAPI header basing on the header content.
- This script is invoked from link-vmlinux.sh prior to BTF generation during kernel build. The output of the script is saved to a file, the file is passed to pahole as --header_guards_db parameter.
- libbpf is modified to aggregate BTF_DECL_TAG records for each type and to emit #ifndef FOO_H ... #endif brackets when "header_guard:" tag is present for a type.

Details for each patch in a set:
- libbpf: Deduplicate unambigous standalone forward declarations
- selftests/bpf: Tests for standalone forward BTF declarations deduplication

There is a small number (63 for defconfig) of forward declarations that are not de-duplicated with the main type declaration under certain conditions. This hinders the header guard brackets generation. This patch addresses this de-duplication issue.

- libbpf: Support for BTF_DECL_TAG dump in C format
- selftests/bpf: Tests for BTF_DECL_TAG dump in C format

Currently libbpf does not process BTF_DECL_TAG when btf is dumped in C format. This patch adds a hash table matching btf type ids with a list of decl tags to the struct btf_dump.
The btf_dump_emit_decl_tags is not necessary for the overall patch-set to function but simplifies testing a bit.

- libbpf: Header guards for selected data structures in vmlinux.h
- selftests/bpf: Tests for header guards printing in BTF dump

Adds option emit_header_guards to struct btf_dump_opts. When enabled the btf_dump__dump_type prints #ifndef ... #endif brackets around types for which header guard information is present in BTF.

- bpftool: Enable header guards generation

Unconditionally enables emit_header_guards for BTF dump in C format.

- kbuild: Script to infer header guard values for uapi headers
- kbuild: Header guards for types from include/uapi/*.h in kernel BTF

Adds scripts/infer_header_guards.pl and integrates it with link-vmlinux.sh.

- selftests/bpf: Script to verify uapi headers usage with vmlinux.h

Adds a script test_uapi_headers.py that tests header guards with vmlinux.h by compiling a simple C snippet. The snippet looks as follows:

    #include <some_uapi_header.h>
    #include "vmlinux.h"
  
    __attribute__((section("tc"), used))
    int syncookie_tc(struct __sk_buff *skb) { return 0; }

The list of headers to test comes from tools/testing/selftests/bpf/good_uapi_headers.txt.

- selftests/bpf: Known good uapi headers for test_uapi_headers.py

The list of uapi headers that could be included alongside vmlinux.h. The headers are peeked from the following locations:
- <headers-export-dir>/linux/*.h
- <headers-export-dir>/linux/**/*.h
This choice of locations is somewhat arbitrary.

- selftests/bpf: script for infer_header_guards.pl testing

The test case for scripts/infer_header_guards.pl, verifies that header guards can be inferred for all uapi headers.

- There is also a patch for dwarves that adds --header_guards_db option (see [2]).

The test_uapi_headers.py is important as it demonstrates the the necessary compiler flags:

clang ...                                  \
      -D__x86_64__                         \
      -Xclang -fwchar-type=short           \
      -Xclang -fno-signed-wchar            \
      -I{exported_kernel_headers}/include/ \
      ...

- -fwchar-type=short and -fno-signed-wchar had to be added because BPF target uses int for wchar_t by default and this differs from vmlinux.h definition of the type (at least for x86_64).
- __x86_64__ had to be added for uapi headers that include stddef.h (the one that is supplied my CLANG itself), in order to define correct sizes for size_t and ptrdiff_t.
- The {exported_kernel_headers} stands for exported kernel headers directory (the headers obtained by make headers_install or via distribution package).

When it works
---

The mechanics described above works for a significant number of UAPI headers. For example, for the test case above I chose the headers from the following locations:
- linux/*.h
- linux/**/*.h
There are 759 such headers and for 677 of them the test described above passes.

I excluded the headers from the following sub-directories as potentially not interesting:

  asm          rdma   video xen
  asm-generic  misc   scsi
  drm          mtd    sound

Thus saving some time for both discussion and CI but the choice is somewhat arbitrary. If I run test_uapi_headers.py --test '*' (all headers) test passes for 834 out of 972 headers.

When it breaks
---

There several scenarios when this mechanics breaks. Specifically I found the following cases:
- When uapi header includes some system header that conflicts with vmlinux.h.
- When uapi header itself conflicts with vmlinux.h.

Below are examples for both cases.

Conflict with system headers
----

The following uapi headers:
- linux/atmbr2684.h
- linux/bpfilter.h
- linux/gsmmux.h
- linux/icmp.h
- linux/if.h
- linux/if_arp.h
- linux/if_bonding.h
- linux/if_pppox.h
- linux/if_tunnel.h
- linux/ip6_tunnel.h
- linux/llc.h
- linux/mctp.h
- linux/mptcp.h
- linux/netdevice.h
- linux/netfilter/xt_RATEEST.h
- linux/netfilter/xt_hashlimit.h
- linux/netfilter/xt_physdev.h
- linux/netfilter/xt_rateest.h
- linux/netfilter_arp/arp_tables.h
- linux/netfilter_arp/arpt_mangle.h
- linux/netfilter_bridge.h
- linux/netfilter_bridge/ebtables.h
- linux/netfilter_ipv4/ip_tables.h
- linux/netfilter_ipv6/ip6_tables.h
- linux/route.h
- linux/wireless.h

Include the following system header:
- /usr/include/sys/socket.h (all via linux/if.h)

The sys/socket.h conflicts with vmlinux.h in:
- types: struct iovec, struct sockaddr, struct msghdr, ...
- constants: SOCK_STREAM, SOCK_DGRAM, ...

However, only two types are actually used:
- struct sockaddr
- struct sockaddr_storage (used only in linux/mptcp.h)

In 'vmlinux.h' this type originates from 'kernel/include/socket.h' (non UAPI header), thus does not have a header guard.

The only workaround that I see is to:
- define a stub sys/socket.h as follows:

    #ifndef __BPF_SOCKADDR__
    #define __BPF_SOCKADDR__
    
    /* For __kernel_sa_family_t */
    #include <linux/socket.h>
    
    struct sockaddr {
        __kernel_sa_family_t sa_family;
        char sa_data[14];
    };
    
    #endif

- hardcode generation of __BPF_SOCKADDR__ bracket for 'struct sockaddr' in vmlinux.h.

Another possibility is to move the definition of 'struct sockaddr' from 'kernel/include/socket.h' to 'kernel/include/uapi/linux/socket.h', but I expect that this won't fly with the mainline as it might break the programs that include both 'linux/socket.h' and 'sys/socket.h'.

Conflict with vmlinux.h
----

Uapi header:
- linux/signal.h

Conflict with vmlinux.h in definition of 'struct sigaction'. Defined in:
- vmlinux.h: kernel/include/linux/signal_types.h
- uapi: kernel/arch/x86/include/asm/signal.h

Uapi headers:
- linux/tipc_sockets_diag.h
- linux/sock_diag.h

Conflict with vmlinux.h in definition of 'SOCK_DESTROY'. Defined in:
- vmlinux.h: kernel/include/net/sock.h
- uapi: kernel/include/uapi/linux/sock_diag.h
Constants seem to be unrelated.

And so on... I have details for many other headers but omit those for brevity.

In conclusion
---

Except from the general feasibility I have a few questions:
- What UAPI headers are the candidates for such use? If there are some interesting headers currently not working with this patch-set some hacks have to be added (e.g. like with linux/if.h).
- Is it ok to encode header guards as special BTF_DECL_TAG or should I change the BTF format a bit to save some bytes.

Thanks,
Eduard

[1] https://reviews.llvm.org/D111307
[clang] __attribute__ bpf_dominating_decl
[2] https://lore.kernel.org/dwarves/[email protected]/T/
[RFC dwarves] pahole: Save header guard names when
--header_guards_db is passed

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

 scripts/infer_header_guards.pl                | 191 +++++
 scripts/link-vmlinux.sh                       |  13 +-
 tools/bpf/bpftool/btf.c                       |   4 +-
 tools/lib/bpf/btf.c                           | 178 ++++-
 tools/lib/bpf/btf.h                           |   7 +-
 tools/lib/bpf/btf_dump.c                      | 232 +++++-
 .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
 tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
 .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
 .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
 .../progs/btf_dump_test_case_header_guards.c  |  94 +++
 .../bpf/test_uapi_header_guards_infer.sh      |  33 +
 .../selftests/bpf/test_uapi_headers.py        | 197 +++++
 13 files changed, 1816 insertions(+), 12 deletions(-)
 create mode 100755 scripts/infer_header_guards.pl
 create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c
 create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh
 create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py
#18 lore raw

Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF

In reply to Eduard Zingerman
Quoted 50 lines · Eduard Zingerman

Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the scripts/infer_header_guards.pl.

The encoded header guard information could be used to restore the original guards in the vmlinux.h, e.g.:

     include/uapi/linux/tcp.h:

       #ifndef _UAPI_LINUX_TCP_H
       #define _UAPI_LINUX_TCP_H
       ...
       union tcp_word_hdr {
     	struct tcphdr hdr;
     	__be32        words[5];
       };
       ...
       #endif /* _UAPI_LINUX_TCP_H */

     vmlinux.h:

       ...
       #ifndef _UAPI_LINUX_TCP_H

       union tcp_word_hdr {
     	struct tcphdr hdr;
     	__be32 words[5];
       };

       #endif /* _UAPI_LINUX_TCP_H */
       ...
---
  scripts/link-vmlinux.sh | 13 ++++++++++++-
  1 file changed, 12 insertions(+), 1 deletion(-)
scripts/link-vmlinux.sh +1 -0
index 918470d768e9..f57f621eda1f 100755
@@ -110,6 +110,7 @@ vmlinux_link()
  gen_btf()
  {
  	local pahole_ver
+	local extra_flags
 
  	if ! [ -x "$(command -v ${PAHOLE})" ]; then
  		echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"

scripts/link-vmlinux.sh gen_btf() · line 127

Quoted 5 lines
scripts/link-vmlinux.sh +2 -0
@@ -122,10 +123,20 @@ gen_btf()
  		return 1
  	fi
 
+	if [ "${pahole_ver}" -ge "124" ]; then
+		scripts/infer_header_guards.pl \

We should have full path like

	${srctree}/scripts/infer_header_guards.pl

so it can work if build directory is different from source directory.

Quoted 16 lines
scripts/link-vmlinux.sh +9 -1
+			include/uapi \
+			include/generated/uapi \
+			arch/${SRCARCH}/include/uapi \
+			arch/${SRCARCH}/include/generated/uapi \
+			> .btf.uapi_header_guards || return 1;
+		extra_flags="--header_guards_db .btf.uapi_header_guards"
+	fi
+
  	vmlinux_link ${1}
 
  	info "BTF" ${2}
-	LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1}
+	LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1}
 
  	# Create ${2} which contains just .BTF section but no symbols. Add
  	# SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
#19 lore raw

Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF

Quoted 52 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman

Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the scripts/infer_header_guards.pl.

The encoded header guard information could be used to restore the original guards in the vmlinux.h, e.g.:

     include/uapi/linux/tcp.h:

       #ifndef _UAPI_LINUX_TCP_H
       #define _UAPI_LINUX_TCP_H
       ...
       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32        words[5];
       };
       ...
       #endif /* _UAPI_LINUX_TCP_H */

     vmlinux.h:

       ...
       #ifndef _UAPI_LINUX_TCP_H

       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32 words[5];
       };

       #endif /* _UAPI_LINUX_TCP_H */
       ...

---
  scripts/link-vmlinux.sh | 13 ++++++++++++-

  1 file changed, 12 insertions(+), 1 deletion(-)
scripts/link-vmlinux.sh +0 -0
index 918470d768e9..f57f621eda1f 100755
@@ -110,6 +110,7 @@ vmlinux_link()
   gen_btf()
   {

      local pahole_ver
+    local extra_flags
      if ! [ -x "$(command -v ${PAHOLE})" ]; then
          echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"

scripts/link-vmlinux.sh +0 -0
@@ -122,10 +123,20 @@ gen_btf()
Quoted 8 lines
Quoted 4 lines

          return 1
      fi
+    if [ "${pahole_ver}" -ge "124" ]; then
+        scripts/infer_header_guards.pl \

We should have full path like
    ${srctree}/scripts/infer_header_guards.pl
so it can work if build directory is different from source directory.

handling arguments for infer_header_guards.pl should also take care of full file path.

+ /home/yhs/work/bpf-next/scripts/infer_header_guards.pl include/uapi include/generated/uapi arch/x86/include/uapi arch/x86/include/generated/uapi
+ return 1

Quoted 17 lines
Quoted 17 lines

+            include/uapi \
+            include/generated/uapi \
+            arch/${SRCARCH}/include/uapi \
+            arch/${SRCARCH}/include/generated/uapi \
+            > .btf.uapi_header_guards || return 1;
+        extra_flags="--header_guards_db .btf.uapi_header_guards" +    fi
+
      vmlinux_link ${1}
      info "BTF" ${2}
-    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} +    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1}
      # Create ${2} which contains just .BTF section but no symbols. Add # SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all

#2 lore raw

Re: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations

In reply to Eduard Zingerman
Quoted 33 lines · Eduard Zingerman

Deduplicate forward declarations that don't take part in type graphs comparisons if declaration name is unambiguous. Example:

CU #1:

struct foo; // standalone forward declaration
struct foo *some_global;

CU #2:

struct foo { int x; };
struct foo *another_global;

The struct foo from CU #1 is not a part of any definition that is compared against another definition while btf_dedup_struct_types processes structural types. The the BTF after btf_dedup_struct_types the BTF looks as follows:

[1] STRUCT 'foo' size=4 vlen=1 ...
[2] INT 'int' size=4 ...
[3] PTR '(anon)' type_id=1
[4] FWD 'foo' fwd_kind=struct
[5] PTR '(anon)' type_id=4

This commit adds a new pass btf_dedup_standalone_fwds, that maps such forward declarations to structs or unions with identical name in case if the name is not ambiguous.

The pass is positioned before btf_dedup_ref_types so that types
[3] and [5] could be merged as a same type after [1] and [4] are merged. The final result for the example above looks as follows:

Quoted 9 lines

[1] STRUCT 'foo' size=4 vlen=1

        'x' type_id=2 bits_offset=0

[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[3] PTR '(anon)' type_id=1

For defconfig kernel with BTF enabled this removes 63 forward declarations. Examples of removed declarations: pt_regs, in6_addr. The running time of btf__dedup function is increased by about 3%.

What about modules, can you share stats for module BTFs?

Also cc Alan as he was looking at BTF dedup improvements for kernel module BTF dedup.

Quoted 19 lines
---
 tools/lib/bpf/btf.c | 178 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 174 insertions(+), 4 deletions(-)
tools/lib/bpf/btf.c +1 -0
index d88647da2c7f..c34c68d8e8a0 100644
@@ -2881,6 +2881,7 @@ static int btf_dedup_strings(struct btf_dedup *d);
 static int btf_dedup_prim_types(struct btf_dedup *d);
 static int btf_dedup_struct_types(struct btf_dedup *d);
 static int btf_dedup_ref_types(struct btf_dedup *d);
+static int btf_dedup_standalone_fwds(struct btf_dedup *d);
 static int btf_dedup_compact_types(struct btf_dedup *d);
 static int btf_dedup_remap_types(struct btf_dedup *d);
 
@@ -2988,15 +2989,16 @@ static int btf_dedup_remap_types(struct btf_dedup *d);
  * Algorithm summary

tools/lib/bpf/btf.c static int btf_dedup_remap_types(struct btf_dedup *d); · line 2997

Quoted 10 lines
tools/lib/bpf/btf.c +2 -2
  * =================
  *
- * Algorithm completes its work in 6 separate passes:
+ * Algorithm completes its work in 7 separate passes:
  *
  * 1. Strings deduplication.
  * 2. Primitive types deduplication (int, enum, fwd).
  * 3. Struct/union types deduplication.
- * 4. Reference types deduplication (pointers, typedefs, arrays, funcs, func
+ * 4. Standalone fwd declarations deduplication.

Let's call this "Resolve unambiguous forward declarations", we don't really deduplicate anything. And call the function btf_dedup_resolve_fwds()?

Quoted 28 lines
tools/lib/bpf/btf.c +11 -2
+ * 5. Reference types deduplication (pointers, typedefs, arrays, funcs, func
  *    protos, and const/volatile/restrict modifiers).
- * 5. Types compaction.
- * 6. Types remapping.
+ * 6. Types compaction.
+ * 7. Types remapping.
  *
  * Algorithm determines canonical type descriptor, which is a single
  * representative type for each truly unique type. This canonical type is the
@@ -3060,6 +3062,11 @@ int btf__dedup(struct btf *btf, const struct btf_dedup_opts *opts)
                pr_debug("btf_dedup_struct_types failed:%d\n", err);
                goto done;
        }
+       err = btf_dedup_standalone_fwds(d);
+       if (err < 0) {
+               pr_debug("btf_dedup_standalone_fwd failed:%d\n", err);
+               goto done;
+       }
        err = btf_dedup_ref_types(d);
        if (err < 0) {
                pr_debug("btf_dedup_ref_types failed:%d\n", err);
@@ -4525,6 +4532,169 @@ static int btf_dedup_ref_types(struct btf_dedup *d)
        return 0;
 }
 
+/*
+ * `name_off_map` maps name offsets to type ids (essentially __u32 -> __u32).
+ *

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4547

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+ * The __u32 key/value representations are cast to `void *` before passing
+ * to `hashmap__*` functions. These pseudo-pointers are never dereferenced.
+ *
+ */
+static struct hashmap *name_off_map__new(void)
+{
+       return hashmap__new(btf_dedup_identity_hash_fn,
+                           btf_dedup_equal_fn,
+                           NULL);
+}

is there a point in name_off_map__new and name_off_map__find wrappers except to add one extra function to jump through when reading the code? If you look at other uses of hashmaps in this file, we use the directly. Let's drop those.

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4553

Quoted 6 lines
tools/lib/bpf/btf.c +6 -0
+
+static int name_off_map__find(struct hashmap *map, __u32 name_off, __u32 *type_id)
+{
+       /* This has to be sizeof(void *) in order to be passed to hashmap__find */
+       void *tmp;
+       int found = hashmap__find(map, (void *)(ptrdiff_t)name_off, &tmp);

but this (void *) casting everything was an error in API design, mea culpa. I've been wanting to switch hashmap to use long as key/value type for a long while, maybe let's do it now, as we are adding even more code that looks weird? It seems like accepting long will make hashmap API usage cleaner in most cases. There are not a lot of places where we use hashmap APIs in libbpf, but we'll also need to fix up bpftool usage, and I believe perf copy/pasted hashmap.h (cc Arnaldo), so we'd need to make sure to not break all that. But good thing it's all in the same repo and we can convert them at the same time with no breakage.

WDYT?

Quoted 4 lines
tools/lib/bpf/btf.c +4 -0
+       /*
+        * __u64 cast is necessary to avoid pointer to integer conversion size warning.
+        * It is fine to get rid of this warning as `void *` is used as an integer value.
+        */

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4567

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+       if (found)
+               *type_id = (__u64)tmp;
+       return found;
+}
+
+static int name_off_map__set(struct hashmap *map, __u32 name_off, __u32 type_id)
+{
+       return hashmap__set(map, (void *)(size_t)name_off, (void *)(size_t)type_id,
+                           NULL, NULL);
+}

this function will also be completely unnecessary with longs

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+
+/*
+ * Collect a `name_off_map` that maps type names to type ids for all
+ * canonical structs and unions. If the same name is shared by several
+ * canonical types use a special value 0 to indicate this fact.
+ */
+static int btf_dedup_fill_unique_names_map(struct btf_dedup *d, struct hashmap *names_map)
+{
+       int i, err = 0;
+       __u32 type_id, collision_id;

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+       __u16 kind;
+       struct btf_type *t;
+
+       for (i = 0; i < d->btf->nr_types; i++) {
+               type_id = d->btf->start_id + i;
+               t = btf_type_by_id(d->btf, type_id);
+               kind = btf_kind(t);
+
+               if (kind != BTF_KIND_STRUCT && kind != BTF_KIND_UNION)
+                       continue;

let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4600

Quoted 13 lines
tools/lib/bpf/btf.c +13 -0
+
+               /* Skip non-canonical types */
+               if (type_id != d->map[type_id])
+                       continue;
+
+               err = 0;
+               if (name_off_map__find(names_map, t->name_off, &collision_id)) {
+                       /* Mark non-unique names with 0 */
+                       if (collision_id != 0 && collision_id != type_id)
+                               err = name_off_map__set(names_map, t->name_off, 0);
+               } else {
+                       err = name_off_map__set(names_map, t->name_off, type_id);
+               }

err = hashmap__add(..., t->name_off, type_id);

if (err == -EEXISTS) {
    hashmap__set(..., 0);
    return 0;
}

see comment for hashmap_insert_strategy in hashmap.h

Quoted 6 lines
tools/lib/bpf/btf.c +6 -0
+
+               if (err < 0)
+                       return err;
+       }
+
+       return 0;

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4616

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+}
+
+static int btf_dedup_standalone_fwd(struct btf_dedup *d,
+                                   struct hashmap *names_map,
+                                   __u32 type_id)
+{
+       struct btf_type *t = btf_type_by_id(d->btf, type_id);
+       __u16 kind = btf_kind(t);
+       enum btf_fwd_kind fwd_kind = BTF_INFO_KFLAG(t->info);
+

nit: don't break variables block in two parts, there shouldn't be empty lines

also please use btf_kflag(t)

Quoted 11 lines
tools/lib/bpf/btf.c +11 -0
+       struct btf_type *cand_t;
+       __u16 cand_kind;
+       __u32 cand_id = 0;
+
+       if (kind != BTF_KIND_FWD)
+               return 0;
+
+       /* Skip if this FWD already has a mapping */
+       if (type_id != d->map[type_id])
+               return 0;
+

[...]

#9 lore raw

Re: [RFC bpf-next 03/12] libbpf: Support for BTF_DECL_TAG dump in C format

In reply to Eduard Zingerman
Quoted 56 lines · Eduard Zingerman

At C level BTF_DECL_TAGs are represented as __attribute__ declarations, e.g.:

struct foo {
        ...;
} __attribute__((btf_decl_tag("bar")));

This commit covers only decl tags attached to structs and unions.

BTF doc says that BTF_DECL_TAGs should follow a target type but this is not enforced and tests don't honor this restriction. This commit uses hash table to map types to the list of decl tags.

---
 tools/lib/bpf/btf_dump.c | 143 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 142 insertions(+), 1 deletion(-)
tools/lib/bpf/btf_dump.c +15 -0
index bf0cc0e986dd..9bfe2a4ae277 100644
@@ -75,6 +75,15 @@ struct btf_dump_data {
        bool is_array_char;
 };
 
+/*
+ * An array of ids of BTF_DECL_TAG objects associated with a specific type.
+ */
+struct decl_tag_array {
+       __u16 cnt;
+       __u16 cap;
+       __u32 tag_ids[0];
+};
+
 struct btf_dump {
        const struct btf *btf;
        btf_dump_printf_fn_t printf_fn;
@@ -111,6 +120,11 @@ struct btf_dump {
         * name occurrences
         */
        struct hashmap *ident_names;
+       /*
+        * maps type id to decl_tag_array, assume that relatively small
+        * fraction of types has btf_decl_tag's attached
+        */
+       struct hashmap *decl_tags;
        /*
         * data for typed display; allocated if needed.
         */
@@ -127,6 +141,26 @@ static bool str_equal_fn(const void *a, const void *b, void *ctx)
        return strcmp(a, b) == 0;
 }
 
+static size_t int_hash_fn(const void *key, void *ctx)

tools/lib/bpf/btf_dump.c static bool str_equal_fn(const void *a, const void *b, void *ctx) · line 154

Quoted 10 lines
tools/lib/bpf/btf_dump.c +10 -0
+{
+       int i;
+       size_t h = 0;
+       char *bytes = (char *)key;
+
+       for (i = 0; i < 4; ++i)
+               h = h * 31 + bytes[i];
+
+       return h;
+}

no need, you can just do what btf_dedup_identity_hash_fn() is doing and pass int/long/size_t as is, hashmap implementation does additional multiplicative hashing on top to find a bucket

tools/lib/bpf/btf_dump.c static bool str_equal_fn(const void *a, const void *b, void *ctx) · line 162

Quoted 8 lines
tools/lib/bpf/btf_dump.c +8 -0
+
+static bool int_equal_fn(const void *a, const void *b, void *ctx)
+{
+       int *ia = (int *)a;
+       int *ib = (int *)b;
+
+       return *ia == *ib;
+}

see btf_dedup_equal_fn(), no need for casting, just return a == b;

Quoted 24 lines
tools/lib/bpf/btf_dump.c +8 -0
+
 static const char *btf_name_of(const struct btf_dump *d, __u32 name_off)
 {
        return btf__name_by_offset(d->btf, name_off);
@@ -143,6 +177,7 @@ static void btf_dump_printf(const struct btf_dump *d, const char *fmt, ...)
 
 static int btf_dump_mark_referenced(struct btf_dump *d);
 static int btf_dump_resize(struct btf_dump *d);
+static int btf_dump_assign_decl_tags(struct btf_dump *d);
 
 struct btf_dump *btf_dump__new(const struct btf *btf,
                               btf_dump_printf_fn_t printf_fn,
@@ -179,11 +214,24 @@ struct btf_dump *btf_dump__new(const struct btf *btf,
                d->ident_names = NULL;
                goto err;
        }
+       d->decl_tags = hashmap__new(int_hash_fn, int_equal_fn, NULL);
+       if (IS_ERR(d->decl_tags)) {
+               err = PTR_ERR(d->decl_tags);
+               d->decl_tags = NULL;
+               goto err;
+       }
 
        err = btf_dump_resize(d);

tools/lib/bpf/btf_dump.c struct btf_dump *btf_dump__new(const struct btf *btf, · line 234

Quoted 10 lines
tools/lib/bpf/btf_dump.c +7 -0
        if (err)
                goto err;
 
+       err = btf_dump_assign_decl_tags(d);
+       if (err)
+               goto err;
+
+       if (err)
+               goto err;
+

I like the bullet-proof error checking, but checking just once should be enough ;)

Quoted 24 lines
tools/lib/bpf/btf_dump.c +5 -1
        return d;
 err:
        btf_dump__free(d);
@@ -232,7 +280,8 @@ static void btf_dump_free_names(struct hashmap *map)
 
 void btf_dump__free(struct btf_dump *d)
 {
-       int i;
+       int i, bkt;
+       struct hashmap_entry *cur;
 
        if (IS_ERR_OR_NULL(d))
                return;
@@ -250,6 +299,9 @@ void btf_dump__free(struct btf_dump *d)
        free(d->decl_stack);
        btf_dump_free_names(d->type_names);
        btf_dump_free_names(d->ident_names);
+       hashmap__for_each_entry(d->decl_tags, cur, bkt)
+               free(cur->value);
+       hashmap__free(d->decl_tags);
 
        free(d);
 }

tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 428

Quoted 4 lines
tools/lib/bpf/btf_dump.c +1 -0
@@ -373,6 +425,77 @@ static int btf_dump_mark_referenced(struct btf_dump *d)
        return 0;
 }
 
+static struct decl_tag_array *btf_dump_find_decl_tags(struct btf_dump *d, __u32 id)

do we really need this wrapper?

tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 432

Quoted 4 lines
tools/lib/bpf/btf_dump.c +4 -0
+{
+       struct decl_tag_array *decl_tags = NULL;
+
+       hashmap__find(d->decl_tags, &id, (void **)&decl_tags);

this &id also made me realize that this is all broken, you are remembering random pointers in hashmap (they point onto stack, which gets reused once this function returns; but hashmap remember it, so on next lookup or update we are going to be reading random values in int_equal_fn?)

you should be passing (void *)(long)id instead (and better yet let's refactor hashmap API as I suggested in previous patch)

either I'm missing something, or this works by accident, which suggests that tests could be improved maybe?..

tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 442

Quoted 10 lines
tools/lib/bpf/btf_dump.c +10 -0
+
+       return decl_tags;
+}
+
+static struct decl_tag_array *realloc_decl_tags(struct decl_tag_array *tags, __u16 new_cap)
+{
+       size_t new_size = sizeof(struct decl_tag_array) + new_cap * sizeof(__u32);
+       struct decl_tag_array *new_tags = (tags
+                                          ? realloc(tags, new_size)
+                                          : calloc(1, new_size));

realloc allocates if passed NULL, so no need for calloc, assuming proper initialization

but let's use libbpf_reallocarray(), we'll waste few bytes on size_t, but given we expect few tags, it's not a big deal

Quoted 12 lines
tools/lib/bpf/btf_dump.c +12 -0
+
+       if (!new_tags)
+               return NULL;
+
+       new_tags->cap = new_cap;
+
+       return new_tags;
+}
+
+/*
+ * Scans all BTF objects looking for BTF_KIND_DECL_TAG entries.
+ * The id's of the entries are stored in the `btf_dump.decl_tags` table,

tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 464

Quoted 10 lines
tools/lib/bpf/btf_dump.c +10 -0
+ * grouped by a target type.
+ */
+static int btf_dump_assign_decl_tags(struct btf_dump *d)
+{
+       int err;
+       __u32 id;
+       __u32 n = btf__type_cnt(d->btf);
+       __u32 new_capacity;
+       const struct btf_type *t;
+       struct decl_tag_array *decl_tags;

few nits: generally, for new code try to do reverse Christmas try style, where widest line is at the top, shortest at the bottom

but also here you can have id, new_capacity, and n on same line

and s/new_capacity/new_cap/

tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 466

Quoted 2 lines
tools/lib/bpf/btf_dump.c +2 -0
+
+       for (id = 0; id < n; id++) {

0 is VOID, we never really need to process it, just start with id = 1

tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 470

Quoted 4 lines
tools/lib/bpf/btf_dump.c +4 -0
+               t = btf__type_by_id(d->btf, id);
+
+               if (btf_kind(t) != BTF_KIND_DECL_TAG)
+                       continue;

if (!btf_is_decl_tag(t))

    continue;
Quoted 12 lines
tools/lib/bpf/btf_dump.c +12 -0
+
+               decl_tags = btf_dump_find_decl_tags(d, t->type);
+               if (!decl_tags) {
+                       decl_tags = realloc_decl_tags(NULL, 1);
+                       if (!decl_tags)
+                               return -ENOMEM;
+                       err = hashmap__insert(d->decl_tags, &t->type, decl_tags,
+                                             HASHMAP_SET, NULL, NULL);
+                       if (err)
+                               return err;
+               } else if (decl_tags->cnt == decl_tags->cap) {
+                       new_capacity = decl_tags->cap * 2;

tools/lib/bpf/btf_dump.c static int btf_dump_mark_referenced(struct btf_dump *d) · line 492

Quoted 10 lines
tools/lib/bpf/btf_dump.c +10 -0
+                       if (new_capacity > 0xffff)
+                               return -ERANGE;
+                       decl_tags = realloc_decl_tags(decl_tags, new_capacity);
+                       if (!decl_tags)
+                               return -ENOMEM;
+                       decl_tags->cap = new_capacity;
+                       err = hashmap__update(d->decl_tags, &t->type, decl_tags, NULL, NULL);
+                       if (err)
+                               return err;
+               }

really, let's just use libbpf_reallocarray? I was going to suggest libbpf_ensure_mem, but it allocates at least 16 elements, which seems like an overkill. But also given we don't expect a lot of tags per type, realloc()'ing with + 1 (no * 2 strategy) seems reasonable. Modern allocators either way use differently sized buckets, so when realloc size increment is small, allocator basically will do nothing.

Quoted 19 lines
tools/lib/bpf/btf_dump.c +12 -0
+               decl_tags->tag_ids[decl_tags->cnt++] = id;
+       }
+
+       return 0;
+}
+
 static int btf_dump_add_emit_queue_id(struct btf_dump *d, __u32 id)
 {
        __u32 *new_queue;
@@ -899,6 +1022,23 @@ static void btf_dump_emit_bit_padding(const struct btf_dump *d,
        }
 }
 
+static void btf_dump_emit_decl_tags(struct btf_dump *d, __u32 id)
+{
+       struct decl_tag_array *decl_tags = btf_dump_find_decl_tags(d, id);
+       struct btf_type *decl_tag_t;
+       const char *decl_tag_text;
+       __u32 i;

tools/lib/bpf/btf_dump.c static void btf_dump_emit_bit_padding(const struct btf_dump *d, · line 1040

Quoted 10 lines
tools/lib/bpf/btf_dump.c +10 -0
+
+       if (!decl_tags)
+               return;
+
+       for (i = 0; i < decl_tags->cnt; ++i) {
+               decl_tag_t = btf_type_by_id(d->btf, decl_tags->tag_ids[i]);
+               decl_tag_text = btf__name_by_offset(d->btf, decl_tag_t->name_off);
+               btf_dump_printf(d, " __attribute__((btf_decl_tag(\"%s\")))", decl_tag_text);
+       }
+}

I'm wondering if we should anticipate that some compilers won't know about btf_decl_tag attribute? It seems a bit off for btf_dump to worry about this, but if we don't do something like:

#if __has_attribute(btf_decl_tag)
#define __btf_decl_tag(x) __attribute__((btf_decl_tag(#x))) #else
#define __btf_decl_tag(x)
#endif

.
.
.

struct my_struct {
     ...
} __btf_decl_tag(awesomeness);

it will be hard for users to use resulting vmlinux.h with slightly older Clang?

Quoted 15 lines
tools/lib/bpf/btf_dump.c +2 -1
+
 static void btf_dump_emit_struct_fwd(struct btf_dump *d, __u32 id,
                                     const struct btf_type *t)
 {
@@ -964,6 +1104,7 @@ static void btf_dump_emit_struct_def(struct btf_dump *d,
        btf_dump_printf(d, "%s}", pfx(lvl));
        if (packed)
                btf_dump_printf(d, " __attribute__((packed))");
+       btf_dump_emit_decl_tags(d, id);
 }
 
 static const char *missing_base_types[][2] = {
--

2.34.1

#20 lore raw

Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF

In reply to Yonghong Song
Quoted 64 lines · Yonghong Song
Quoted 52 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman

Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the scripts/infer_header_guards.pl.

The encoded header guard information could be used to restore the original guards in the vmlinux.h, e.g.:

     include/uapi/linux/tcp.h:

       #ifndef _UAPI_LINUX_TCP_H
       #define _UAPI_LINUX_TCP_H
       ...
       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32        words[5];
       };
       ...
       #endif /* _UAPI_LINUX_TCP_H */

     vmlinux.h:

       ...
       #ifndef _UAPI_LINUX_TCP_H

       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32 words[5];
       };

       #endif /* _UAPI_LINUX_TCP_H */
       ...

---
  scripts/link-vmlinux.sh | 13 ++++++++++++-

  1 file changed, 12 insertions(+), 1 deletion(-)
scripts/link-vmlinux.sh +0 -0
index 918470d768e9..f57f621eda1f 100755
@@ -110,6 +110,7 @@ vmlinux_link()
   gen_btf()
   {

      local pahole_ver
+    local extra_flags
      if ! [ -x "$(command -v ${PAHOLE})" ]; then
          echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"

scripts/link-vmlinux.sh +0 -0
@@ -122,10 +123,20 @@ gen_btf()
Quoted 8 lines
Quoted 4 lines

          return 1
      fi
+    if [ "${pahole_ver}" -ge "124" ]; then
+        scripts/infer_header_guards.pl \

We should have full path like ${srctree}/scripts/infer_header_guards.pl
so it can work if build directory is different from source directory.

Quoted 6 lines

handling arguments for infer_header_guards.pl should also take care of full file path.

+ /home/yhs/work/bpf-next/scripts/infer_header_guards.pl include/uapi include/generated/uapi arch/x86/include/uapi arch/x86/include/generated/uapi
+ return 1

Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.

Quoted 17 lines
Quoted 17 lines
Quoted 17 lines

+            include/uapi \
+            include/generated/uapi \
+            arch/${SRCARCH}/include/uapi \
+            arch/${SRCARCH}/include/generated/uapi \
+            > .btf.uapi_header_guards || return 1;
+        extra_flags="--header_guards_db .btf.uapi_header_guards" +    fi
+
      vmlinux_link ${1}
      info "BTF" ${2}
-    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} +    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1}
      # Create ${2} which contains just .BTF section but no symbols. Add # SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all

#12 lore raw

Re: [RFC bpf-next 05/12] libbpf: Header guards for selected data structures in vmlinux.h

In reply to Eduard Zingerman
Quoted 44 lines · Eduard Zingerman

The goal of the patch is to allow usage of header files from include/uapi alongside with vmlinux.h. E.g. as follows:

  #include <uapi/linux/tcp.h>
  #include "vmlinux.h"

This goal is achieved by adding #ifndef / #endif guards in vmlinux.h around definitions that originate from the include/uapi headers. The guards emitted match the guards used in the original headers. E.g. as follows:

include/uapi/linux/tcp.h:

  #ifndef _UAPI_LINUX_TCP_H
  #define _UAPI_LINUX_TCP_H
  ...
  union tcp_word_hdr {
        struct tcphdr hdr;
        __be32        words[5];
  };
  ...
  #endif /* _UAPI_LINUX_TCP_H */

vmlinux.h:

  ...
  #ifndef _UAPI_LINUX_TCP_H

  union tcp_word_hdr {
        struct tcphdr hdr;
        __be32 words[5];
  };

  #endif /* _UAPI_LINUX_TCP_H */
  ...

The problem of identifying data structures from uapi and selecting proper guard names is delegated to pahole. When configured pahole generates fake BTF_DECL_TAG records with header guards information. The fake tag is distinguished from a real tag by a prefix "header_guard:" in its value. These tags could be present for unions, structures, enums and typedefs, e.g.:

Quoted 8 lines

[24139] STRUCT 'tcphdr' size=20 vlen=17
...
[24296] DECL_TAG 'header_guard:_UAPI_LINUX_TCP_H' type_id=24139 ...

This patch adds An option emit_header_guards to struct btf_dump_opts. When this option is present the function btf_dump__dump_type emits header guards for top-level declarations. The header guards are identified by inspecting fake BTF_DECL_TAG records described above.

This looks like a completely arbitrary convention that libbpf has no business knowing or caring about. I think bpftool should be emitting these guards when generating vmlinux.h. Let's solve this somehow differently.

Quoted 7 lines
---
 tools/lib/bpf/btf.h      |  7 +++-
 tools/lib/bpf/btf_dump.c | 89 +++++++++++++++++++++++++++++++++++++++-
 2 files changed, 94 insertions(+), 2 deletions(-)

[...]

#16 lore raw

Re: [RFC bpf-next 08/12] kbuild: Script to infer header guard values for uapi headers

In reply to Eduard Zingerman
Quoted 30 lines · Eduard Zingerman

The script infers header guard defines in headers from include/uapi/**/*.h . E.g. header guard for the include/uapi/linux/tcp.h is _UAPI_LINUX_TCP_H:

    include/uapi/linux/tcp.h:

      #ifndef _UAPI_LINUX_TCP_H
      #define _UAPI_LINUX_TCP_H
      ...
      union tcp_word_hdr {
            struct tcphdr hdr;
            __be32        words[5];
      };
      ...
      #endif /* _UAPI_LINUX_TCP_H */

The output of the script could be used as an input to pahole's --header_guards_db parameter. This information is necessary to repeat the same header guards in the vmlinux.h generated from BTF.

It is not possible to infer the guard names from header file names alone, the file content has to be analyzed. The following heuristic is used to infer guard for a specific file:
- All pairs #ifndef <candidate> / #define <candidate> are collected;
- If a unique candidate matching regex ${headername}.*_H(EADER)? it is selected;
- If a unique candidate matching regex _H(EADER)?_ it is selected;
- If a unique candidate matching regex _H(EADER)?$ it is selected;

Quoted 4 lines

There is also a small list of headers that can't be caught by the rules above, 15 in total. These headers and corresponding guard values are listed in the %OVERRIDES hash table.

Instead of expecting naming pattern, why can't we just expect

/* some comments here */

#ifndef XXX
#define XXX
....
#endif

and extract XXX from such a pattern?

The harder part is skipping comments (but awk might help do this easier), or we can just ignore all the lines before the first #ifndef.

WDYT?

Quoted 6 lines
---
 scripts/infer_header_guards.pl | 191 +++++++++++++++++++++++++++++++++
 1 file changed, 191 insertions(+)
 create mode 100755 scripts/infer_header_guards.pl

[...]

#32 lore raw
In reply to Eduard Zingerman
Quoted 9 lines · Eduard Zingerman

Hi BPF community,

AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.

Maybe we should make another attempt to implement bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?

BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.

Quoted 5 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

[...]

Quoted 39 lines

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

 scripts/infer_header_guards.pl                | 191 +++++
 scripts/link-vmlinux.sh                       |  13 +-
 tools/bpf/bpftool/btf.c                       |   4 +-
 tools/lib/bpf/btf.c                           | 178 ++++-
 tools/lib/bpf/btf.h                           |   7 +-
 tools/lib/bpf/btf_dump.c                      | 232 +++++-
 .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
 tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
 .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
 .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
 .../progs/btf_dump_test_case_header_guards.c  |  94 +++
 .../bpf/test_uapi_header_guards_infer.sh      |  33 +
 .../selftests/bpf/test_uapi_headers.py        | 197 +++++
 13 files changed, 1816 insertions(+), 12 deletions(-)
 create mode 100755 scripts/infer_header_guards.pl
 create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c
 create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh
 create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py

--
2.34.1

author maintainer
#21 lore raw

Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF

In reply to Yonghong Song
Quoted 69 lines · Yonghong Song
Quoted 67 lines · Yonghong Song
Quoted 52 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman

Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the scripts/infer_header_guards.pl.

The encoded header guard information could be used to restore the original guards in the vmlinux.h, e.g.:

     include/uapi/linux/tcp.h:

       #ifndef _UAPI_LINUX_TCP_H
       #define _UAPI_LINUX_TCP_H
       ...
       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32        words[5];
       };
       ...
       #endif /* _UAPI_LINUX_TCP_H */

     vmlinux.h:

       ...
       #ifndef _UAPI_LINUX_TCP_H

       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32 words[5];
       };

       #endif /* _UAPI_LINUX_TCP_H */
       ...

---
  scripts/link-vmlinux.sh | 13 ++++++++++++-

  1 file changed, 12 insertions(+), 1 deletion(-)
scripts/link-vmlinux.sh +0 -0
index 918470d768e9..f57f621eda1f 100755
@@ -110,6 +110,7 @@ vmlinux_link()
   gen_btf()
   {

      local pahole_ver
+    local extra_flags
      if ! [ -x "$(command -v ${PAHOLE})" ]; then
          echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"

scripts/link-vmlinux.sh +0 -0
@@ -122,10 +123,20 @@ gen_btf()
Quoted 8 lines
Quoted 4 lines

          return 1
      fi
+    if [ "${pahole_ver}" -ge "124" ]; then
+        scripts/infer_header_guards.pl \

We should have full path like ${srctree}/scripts/infer_header_guards.pl
so it can work if build directory is different from source directory.

handling arguments for infer_header_guards.pl should also take care of full file path.

Quoted 7 lines
Quoted 4 lines

+ /home/yhs/work/bpf-next/scripts/infer_header_guards.pl include/uapi include/generated/uapi arch/x86/include/uapi arch/x86/include/generated/uapi
+ return 1

Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.

Hi Yonghong,

Could you please copy-paste some of the error reports? I just re-run the selftests locally and have test_maps, test_verifier, test_progs and test_progs-no_alu32 passing.

Thanks,
Eduard

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines

+            include/uapi \
+            include/generated/uapi \
+            arch/${SRCARCH}/include/uapi \
+            arch/${SRCARCH}/include/generated/uapi \
+            > .btf.uapi_header_guards || return 1;
+        extra_flags="--header_guards_db .btf.uapi_header_guards" +    fi
+
      vmlinux_link ${1}
      info "BTF" ${2}
-    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1} +    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${extra_flags} ${1}
      # Create ${2} which contains just .BTF section but no symbols. Add # SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all

#22 lore raw

Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF

Yonghong,

build will be failing without merged pahole changes.

Quoted 107 lines · Eduard Zingerman
Quoted 69 lines · Yonghong Song
Quoted 67 lines · Yonghong Song
Quoted 58 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman

Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the scripts/infer_header_guards.pl.

The encoded header guard information could be used to restore the original guards in the vmlinux.h, e.g.:

     include/uapi/linux/tcp.h:

       #ifndef _UAPI_LINUX_TCP_H
       #define _UAPI_LINUX_TCP_H
       ...
       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32        words[5];
       };
       ...
       #endif /* _UAPI_LINUX_TCP_H */

     vmlinux.h:

       ...
       #ifndef _UAPI_LINUX_TCP_H

       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32 words[5];
       };

       #endif /* _UAPI_LINUX_TCP_H */
       ...
---
  scripts/link-vmlinux.sh | 13 ++++++++++++-
  1 file changed, 12 insertions(+), 1 deletion(-)
scripts/link-vmlinux.sh +1 -0
index 918470d768e9..f57f621eda1f 100755
@@ -110,6 +110,7 @@ vmlinux_link()
  gen_btf()
  {
      local pahole_ver
+    local extra_flags
      if ! [ -x "$(command -v ${PAHOLE})" ]; then
          echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"

scripts/link-vmlinux.sh gen_btf() · line 126

Quoted 4 lines
scripts/link-vmlinux.sh +2 -0
@@ -122,10 +123,20 @@ gen_btf()
          return 1
      fi
+    if [ "${pahole_ver}" -ge "124" ]; then
+        scripts/infer_header_guards.pl \

We should have full path like

Quoted 2 lines
     ${srctree}/scripts/infer_header_guards.pl

so it can work if build directory is different from source directory.

handling arguments for infer_header_guards.pl should also take care of full file path.

Quoted 7 lines
Quoted 4 lines

+ /home/yhs/work/bpf-next/scripts/infer_header_guards.pl include/uapi include/generated/uapi arch/x86/include/uapi arch/x86/include/generated/uapi
+ return 1

Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.

Hi Yonghong,

Could you please copy-paste some of the error reports? I just re-run the selftests locally and have test_maps, test_verifier, test_progs and test_progs-no_alu32 passing.

Thanks,
Eduard

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
scripts/link-vmlinux.sh +9 -2
+            include/uapi \
+            include/generated/uapi \
+            arch/${SRCARCH}/include/uapi \
+            arch/${SRCARCH}/include/generated/uapi \
+            > .btf.uapi_header_guards || return 1;
+        extra_flags="--header_guards_db .btf.uapi_header_guards"
+    fi
+
      vmlinux_link ${1}
      info "BTF" ${2}
-    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1}
+    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} 
 ${extra_flags} ${1}
      # Create ${2} which contains just .BTF section but no symbols. Add
      # SHF_ALLOC because .BTF will be part of the vmlinux image. 
--strip-all
#24 lore raw

Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF

In reply to Eduard Zingerman
Quoted 75 lines · Eduard Zingerman
Quoted 74 lines · Yonghong Song
Quoted 71 lines · Yonghong Song
Quoted 52 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman

Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the scripts/infer_header_guards.pl.

The encoded header guard information could be used to restore the original guards in the vmlinux.h, e.g.:

     include/uapi/linux/tcp.h:

       #ifndef _UAPI_LINUX_TCP_H
       #define _UAPI_LINUX_TCP_H
       ...
       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32        words[5];
       };
       ...
       #endif /* _UAPI_LINUX_TCP_H */

     vmlinux.h:

       ...
       #ifndef _UAPI_LINUX_TCP_H

       union tcp_word_hdr {
         struct tcphdr hdr;
         __be32 words[5];
       };

       #endif /* _UAPI_LINUX_TCP_H */
       ...

---
   scripts/link-vmlinux.sh | 13 ++++++++++++-
   1 file changed, 12 insertions(+), 1 deletion(-)
scripts/link-vmlinux.sh +1 -0
index 918470d768e9..f57f621eda1f 100755
@@ -110,6 +110,7 @@ vmlinux_link()
   gen_btf()
   {
       local pahole_ver
+    local extra_flags
       if ! [ -x "$(command -v ${PAHOLE})" ]; then
           echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"
@@ -122,10 +123,20 @@ gen_btf()
Quoted 8 lines

scripts/link-vmlinux.sh gen_btf() · line 127

Quoted 4 lines
scripts/link-vmlinux.sh +2 -0
           return 1
       fi
+    if [ "${pahole_ver}" -ge "124" ]; then
+        scripts/infer_header_guards.pl \

We should have full path like
     ${srctree}/scripts/infer_header_guards.pl
so it can work if build directory is different from source directory.

handling arguments for infer_header_guards.pl should also take care of full file path.

+ /home/yhs/work/bpf-next/scripts/infer_header_guards.pl include/uapi include/generated/uapi arch/x86/include/uapi arch/x86/include/generated/uapi
+ return 1

Quoted 8 lines
Quoted 2 lines

Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.

Hi Yonghong,

Could you please copy-paste some of the error reports? I just re-run the selftests locally and have test_maps, test_verifier, test_progs and test_progs-no_alu32 passing.

Sorry about the noise. It is my fault. My default build is out of source tree with KBUILD_OUTPUT=<path>. Since the current patch set won't work with it, so I build a in-tree one for vmlinux but forgot to adjust selftest build which still has KBUILD_OUTPUT=<path> and it caused some selftest failures. Consistently doing in-tree build for vmlinux and selftest results in the same Success/Failure rate with and without this patch set.

Quoted 23 lines

Thanks,
Eduard

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
scripts/link-vmlinux.sh +9 -2
+            include/uapi \
+            include/generated/uapi \
+            arch/${SRCARCH}/include/uapi \
+            arch/${SRCARCH}/include/generated/uapi \
+            > .btf.uapi_header_guards || return 1;
+        extra_flags="--header_guards_db .btf.uapi_header_guards"
+    fi
+
       vmlinux_link ${1}
       info "BTF" ${2}
-    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1}
+    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS}
 ${extra_flags} ${1}
       # Create ${2} which contains just .BTF section but no symbols. Add
       # SHF_ALLOC because .BTF will be part of the vmlinux image.
--strip-all
#23 lore raw

Re: [RFC bpf-next 09/12] kbuild: Header guards for types from include/uapi/*.h in kernel BTF

In reply to Mykola Lysenko
Quoted 3 lines · Mykola Lysenko

Yonghong,

build will be failing without merged pahole changes.

Yes, I do have pahole changes. My failure is due to building in-tree vmlinux (KBUILD_OUTPUT= ) since current patch set doesn't support out-of-tree build, and out-of-tree selftest (KBUILD_OUTPUT=<path>). Using the same in-tree build fixed the problem.

Quoted 109 lines
Quoted 108 lines · Eduard Zingerman
Quoted 69 lines · Yonghong Song
Quoted 67 lines · Yonghong Song
Quoted 58 lines · Yonghong Song
Quoted 49 lines · Eduard Zingerman

Use pahole --header_guards_db flag to enable encoding of header guard information in kernel BTF. The actual correspondence between header file and guard string is computed by the scripts/infer_header_guards.pl.

The encoded header guard information could be used to restore the original guards in the vmlinux.h, e.g.:

      include/uapi/linux/tcp.h:

        #ifndef _UAPI_LINUX_TCP_H
        #define _UAPI_LINUX_TCP_H
        ...
        union tcp_word_hdr {
          struct tcphdr hdr;
          __be32        words[5];
        };
        ...
        #endif /* _UAPI_LINUX_TCP_H */

      vmlinux.h:

        ...
        #ifndef _UAPI_LINUX_TCP_H

        union tcp_word_hdr {
          struct tcphdr hdr;
          __be32 words[5];
        };

        #endif /* _UAPI_LINUX_TCP_H */
        ...
---
   scripts/link-vmlinux.sh | 13 ++++++++++++-
   1 file changed, 12 insertions(+), 1 deletion(-)
scripts/link-vmlinux.sh +1 -0
index 918470d768e9..f57f621eda1f 100755
@@ -110,6 +110,7 @@ vmlinux_link()
   gen_btf()
   {
       local pahole_ver
+    local extra_flags
       if ! [ -x "$(command -v ${PAHOLE})" ]; then
           echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"

scripts/link-vmlinux.sh gen_btf() · line 126

Quoted 4 lines
scripts/link-vmlinux.sh +2 -0
@@ -122,10 +123,20 @@ gen_btf()
           return 1
       fi
+    if [ "${pahole_ver}" -ge "124" ]; then
+        scripts/infer_header_guards.pl \

We should have full path like

Quoted 2 lines
      ${srctree}/scripts/infer_header_guards.pl

so it can work if build directory is different from source directory.

handling arguments for infer_header_guards.pl should also take care of full file path.

Quoted 7 lines
Quoted 4 lines

+ /home/yhs/work/bpf-next/scripts/infer_header_guards.pl include/uapi include/generated/uapi arch/x86/include/uapi arch/x86/include/generated/uapi
+ return 1

Also, please pay attention to bpf selftest result. I see quite a few selftest failures with this patch set.

Hi Yonghong,

Could you please copy-paste some of the error reports? I just re-run the selftests locally and have test_maps, test_verifier, test_progs and test_progs-no_alu32 passing.

Thanks,
Eduard

Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
Quoted 18 lines
scripts/link-vmlinux.sh +9 -2
+            include/uapi \
+            include/generated/uapi \
+            arch/${SRCARCH}/include/uapi \
+            arch/${SRCARCH}/include/generated/uapi \
+            > .btf.uapi_header_guards || return 1;
+        extra_flags="--header_guards_db .btf.uapi_header_guards"
+    fi
+
       vmlinux_link ${1}
       info "BTF" ${2}
-    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS} ${1}
+    LLVM_OBJCOPY="${OBJCOPY}" ${PAHOLE} -J ${PAHOLE_FLAGS}
 ${extra_flags} ${1}
       # Create ${2} which contains just .BTF section but no symbols. Add
       # SHF_ALLOC because .BTF will be part of the vmlinux image.
--strip-all
#33 lore raw
In reply to Andrii Nakryiko
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman

Hi BPF community,

AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.

Quoted 3 lines

Maybe we should make another attempt to implement bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?

I would say it would be very difficult for upstream to agree with bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.

Quoted 53 lines

BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.

Quoted 5 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

[...]

Quoted 39 lines

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

  scripts/infer_header_guards.pl                | 191 +++++
  scripts/link-vmlinux.sh                       |  13 +-
  tools/bpf/bpftool/btf.c                       |   4 +-
  tools/lib/bpf/btf.c                           | 178 ++++-
  tools/lib/bpf/btf.h                           |   7 +-
  tools/lib/bpf/btf_dump.c                      | 232 +++++-
  .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
  tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
  .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
  .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
  .../progs/btf_dump_test_case_header_guards.c  |  94 +++
  .../bpf/test_uapi_header_guards_infer.sh      |  33 +
  .../selftests/bpf/test_uapi_headers.py        | 197 +++++
  13 files changed, 1816 insertions(+), 12 deletions(-)

create mode 100755 scripts/infer_header_guards.pl
create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py

--
2.34.1

#34 lore raw
Quoted 19 lines · Yonghong Song
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman

Hi BPF community,

AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.

Quoted 3 lines

Maybe we should make another attempt to implement bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?

Quoted 7 lines

I would say it would be very difficult for upstream to agree with bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.

Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like

#if !__is_type_defined(struct abc)
struct abc {
};
#endif

I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?

Quoted 54 lines
Quoted 54 lines

BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.

Quoted 5 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

[...]

Quoted 39 lines

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

  scripts/infer_header_guards.pl                | 191 +++++
  scripts/link-vmlinux.sh                       |  13 +-
  tools/bpf/bpftool/btf.c                       |   4 +-
  tools/lib/bpf/btf.c                           | 178 ++++-
  tools/lib/bpf/btf.h                           |   7 +-
  tools/lib/bpf/btf_dump.c                      | 232 +++++-
  .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
  tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
  .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
  .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
  .../progs/btf_dump_test_case_header_guards.c  |  94 +++
  .../bpf/test_uapi_header_guards_infer.sh      |  33 +
  .../selftests/bpf/test_uapi_headers.py        | 197 +++++
  13 files changed, 1816 insertions(+), 12 deletions(-)

create mode 100755 scripts/infer_header_guards.pl
create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py

--
2.34.1

#35 lore raw
Quoted 36 lines · Andrii Nakryiko
Quoted 19 lines · Yonghong Song
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman

Hi BPF community,

AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.

Quoted 3 lines

Maybe we should make another attempt to implement bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?

Quoted 7 lines

I would say it would be very difficult for upstream to agree with bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.

Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like

#if !__is_type_defined(struct abc)
struct abc {
};
#endif

Quoted 5 lines

I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

Let us see whether we can get some upstream agreement or not.

Quoted 55 lines
Quoted 55 lines
Quoted 55 lines

BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.

Quoted 5 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

[...]

Quoted 39 lines

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

   scripts/infer_header_guards.pl                | 191 +++++
   scripts/link-vmlinux.sh                       |  13 +-
   tools/bpf/bpftool/btf.c                       |   4 +-
   tools/lib/bpf/btf.c                           | 178 ++++-
   tools/lib/bpf/btf.h                           |   7 +-
   tools/lib/bpf/btf_dump.c                      | 232 +++++-
   .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
   tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
   .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
   .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
   .../progs/btf_dump_test_case_header_guards.c  |  94 +++
   .../bpf/test_uapi_header_guards_infer.sh      |  33 +
   .../selftests/bpf/test_uapi_headers.py        | 197 +++++
   13 files changed, 1816 insertions(+), 12 deletions(-)

create mode 100755 scripts/infer_header_guards.pl
create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py

--
2.34.1

#36 lore raw
Quoted 46 lines · Yonghong Song
Quoted 36 lines · Andrii Nakryiko
Quoted 19 lines · Yonghong Song
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman

Hi BPF community,

AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.

Quoted 3 lines

Maybe we should make another attempt to implement bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?

Quoted 7 lines

I would say it would be very difficult for upstream to agree with bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.

Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like

#if !__is_type_defined(struct abc)
struct abc {
};
#endif

Quoted 5 lines

I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?

Quoted 7 lines

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

Let us see whether we can get some upstream agreement or not.

Thanks for starting the conversation! I'll be following along.

Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines

BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.

Quoted 5 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

[...]

Quoted 39 lines

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

   scripts/infer_header_guards.pl                | 191 +++++
   scripts/link-vmlinux.sh                       |  13 +-
   tools/bpf/bpftool/btf.c                       |   4 +-
   tools/lib/bpf/btf.c                           | 178 ++++-
   tools/lib/bpf/btf.h                           |   7 +-
   tools/lib/bpf/btf_dump.c                      | 232 +++++-
   .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
   tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
   .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
   .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
   .../progs/btf_dump_test_case_header_guards.c  |  94 +++
   .../bpf/test_uapi_header_guards_infer.sh      |  33 +
   .../selftests/bpf/test_uapi_headers.py        | 197 +++++
   13 files changed, 1816 insertions(+), 12 deletions(-)

create mode 100755 scripts/infer_header_guards.pl
create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py

--
2.34.1

author maintainer
#3 lore raw

Re: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations

In reply to Andrii Nakryiko
Quoted 145 lines · Andrii Nakryiko
Quoted 33 lines · Eduard Zingerman

Deduplicate forward declarations that don't take part in type graphs comparisons if declaration name is unambiguous. Example:

CU #1:

struct foo; // standalone forward declaration
struct foo *some_global;

CU #2:

struct foo { int x; };
struct foo *another_global;

The struct foo from CU #1 is not a part of any definition that is compared against another definition while btf_dedup_struct_types processes structural types. The the BTF after btf_dedup_struct_types the BTF looks as follows:

[1] STRUCT 'foo' size=4 vlen=1 ...
[2] INT 'int' size=4 ...
[3] PTR '(anon)' type_id=1
[4] FWD 'foo' fwd_kind=struct
[5] PTR '(anon)' type_id=4

This commit adds a new pass btf_dedup_standalone_fwds, that maps such forward declarations to structs or unions with identical name in case if the name is not ambiguous.

The pass is positioned before btf_dedup_ref_types so that types
[3] and [5] could be merged as a same type after [1] and [4] are merged. The final result for the example above looks as follows:

Quoted 9 lines

[1] STRUCT 'foo' size=4 vlen=1

        'x' type_id=2 bits_offset=0

[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED
[3] PTR '(anon)' type_id=1

For defconfig kernel with BTF enabled this removes 63 forward declarations. Examples of removed declarations: pt_regs, in6_addr. The running time of btf__dedup function is increased by about 3%.

What about modules, can you share stats for module BTFs?

Also cc Alan as he was looking at BTF dedup improvements for kernel module BTF dedup.

Quoted 19 lines
---
 tools/lib/bpf/btf.c | 178 +++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 174 insertions(+), 4 deletions(-)
tools/lib/bpf/btf.c +1 -0
index d88647da2c7f..c34c68d8e8a0 100644
@@ -2881,6 +2881,7 @@ static int btf_dedup_strings(struct btf_dedup *d);
 static int btf_dedup_prim_types(struct btf_dedup *d);
 static int btf_dedup_struct_types(struct btf_dedup *d);
 static int btf_dedup_ref_types(struct btf_dedup *d);
+static int btf_dedup_standalone_fwds(struct btf_dedup *d);
 static int btf_dedup_compact_types(struct btf_dedup *d);
 static int btf_dedup_remap_types(struct btf_dedup *d);
 
@@ -2988,15 +2989,16 @@ static int btf_dedup_remap_types(struct btf_dedup *d);
  * Algorithm summary

tools/lib/bpf/btf.c static int btf_dedup_remap_types(struct btf_dedup *d); · line 2997

Quoted 10 lines
tools/lib/bpf/btf.c +2 -2
  * =================
  *
- * Algorithm completes its work in 6 separate passes:
+ * Algorithm completes its work in 7 separate passes:
  *
  * 1. Strings deduplication.
  * 2. Primitive types deduplication (int, enum, fwd).
  * 3. Struct/union types deduplication.
- * 4. Reference types deduplication (pointers, typedefs, arrays, funcs, func
+ * 4. Standalone fwd declarations deduplication.

Let's call this "Resolve unambiguous forward declarations", we don't really deduplicate anything. And call the function btf_dedup_resolve_fwds()?

Quoted 28 lines
tools/lib/bpf/btf.c +11 -2
+ * 5. Reference types deduplication (pointers, typedefs, arrays, funcs, func
  *    protos, and const/volatile/restrict modifiers).
- * 5. Types compaction.
- * 6. Types remapping.
+ * 6. Types compaction.
+ * 7. Types remapping.
  *
  * Algorithm determines canonical type descriptor, which is a single
  * representative type for each truly unique type. This canonical type is the
@@ -3060,6 +3062,11 @@ int btf__dedup(struct btf *btf, const struct btf_dedup_opts *opts)
                pr_debug("btf_dedup_struct_types failed:%d\n", err);
                goto done;
        }
+       err = btf_dedup_standalone_fwds(d);
+       if (err < 0) {
+               pr_debug("btf_dedup_standalone_fwd failed:%d\n", err);
+               goto done;
+       }
        err = btf_dedup_ref_types(d);
        if (err < 0) {
                pr_debug("btf_dedup_ref_types failed:%d\n", err);
@@ -4525,6 +4532,169 @@ static int btf_dedup_ref_types(struct btf_dedup *d)
        return 0;
 }
 
+/*
+ * `name_off_map` maps name offsets to type ids (essentially __u32 -> __u32).
+ *

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4547

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+ * The __u32 key/value representations are cast to `void *` before passing
+ * to `hashmap__*` functions. These pseudo-pointers are never dereferenced.
+ *
+ */
+static struct hashmap *name_off_map__new(void)
+{
+       return hashmap__new(btf_dedup_identity_hash_fn,
+                           btf_dedup_equal_fn,
+                           NULL);
+}

is there a point in name_off_map__new and name_off_map__find wrappers except to add one extra function to jump through when reading the code? If you look at other uses of hashmaps in this file, we use the directly. Let's drop those.

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4553

Quoted 6 lines
tools/lib/bpf/btf.c +6 -0
+
+static int name_off_map__find(struct hashmap *map, __u32 name_off, __u32 *type_id)
+{
+       /* This has to be sizeof(void *) in order to be passed to hashmap__find */
+       void *tmp;
+       int found = hashmap__find(map, (void *)(ptrdiff_t)name_off, &tmp);

but this (void *) casting everything was an error in API design, mea culpa. I've been wanting to switch hashmap to use long as key/value type for a long while, maybe let's do it now, as we are adding even more code that looks weird? It seems like accepting long will make hashmap API usage cleaner in most cases. There are not a lot of places where we use hashmap APIs in libbpf, but we'll also need to fix up bpftool usage, and I believe perf copy/pasted hashmap.h (cc Arnaldo), so we'd need to make sure to not break all that. But good thing it's all in the same repo and we can convert them at the same time with no breakage.

Quoted 1 line

WDYT?

Well, I did the change, excluding tests it amounts to:
- 15 files changed, 114 insertions(+), 137 deletions(-);
- 45 casts removed;
- 30 casts added.

TBH, it seems like I should just use "u32_as_hash_field" and be done with it. In any case I'll post this as a part of v1 series for "libbpf: Resolve unambigous forward declarations".

To account for a case when map has to store pointers and pointers are 32 bit I chose to update the map interface to be "uintptr_t -> uintptr_t". Had it been "u64 -> u64" the additional temporary variable would be necessary for "old" values, e.g. while working with hashmap__insert. (Contrary to what we discussed on Friday).

Quoted 98 lines
Quoted 5 lines
tools/lib/bpf/btf.c +4 -0
+       /*
+        * __u64 cast is necessary to avoid pointer to integer conversion size warning.
+        * It is fine to get rid of this warning as `void *` is used as an integer value.
+        */

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4567

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+       if (found)
+               *type_id = (__u64)tmp;
+       return found;
+}
+
+static int name_off_map__set(struct hashmap *map, __u32 name_off, __u32 type_id)
+{
+       return hashmap__set(map, (void *)(size_t)name_off, (void *)(size_t)type_id,
+                           NULL, NULL);
+}

this function will also be completely unnecessary with longs

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+
+/*
+ * Collect a `name_off_map` that maps type names to type ids for all
+ * canonical structs and unions. If the same name is shared by several
+ * canonical types use a special value 0 to indicate this fact.
+ */
+static int btf_dedup_fill_unique_names_map(struct btf_dedup *d, struct hashmap *names_map)
+{
+       int i, err = 0;
+       __u32 type_id, collision_id;

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+       __u16 kind;
+       struct btf_type *t;
+
+       for (i = 0; i < d->btf->nr_types; i++) {
+               type_id = d->btf->start_id + i;
+               t = btf_type_by_id(d->btf, type_id);
+               kind = btf_kind(t);
+
+               if (kind != BTF_KIND_STRUCT && kind != BTF_KIND_UNION)
+                       continue;

let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4600

Quoted 13 lines
tools/lib/bpf/btf.c +13 -0
+
+               /* Skip non-canonical types */
+               if (type_id != d->map[type_id])
+                       continue;
+
+               err = 0;
+               if (name_off_map__find(names_map, t->name_off, &collision_id)) {
+                       /* Mark non-unique names with 0 */
+                       if (collision_id != 0 && collision_id != type_id)
+                               err = name_off_map__set(names_map, t->name_off, 0);
+               } else {
+                       err = name_off_map__set(names_map, t->name_off, type_id);
+               }

err = hashmap__add(..., t->name_off, type_id);

if (err == -EEXISTS) {
    hashmap__set(..., 0);
    return 0;
}

see comment for hashmap_insert_strategy in hashmap.h

Quoted 6 lines
tools/lib/bpf/btf.c +6 -0
+
+               if (err < 0)
+                       return err;
+       }
+
+       return 0;

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4616

Quoted 10 lines
tools/lib/bpf/btf.c +10 -0
+}
+
+static int btf_dedup_standalone_fwd(struct btf_dedup *d,
+                                   struct hashmap *names_map,
+                                   __u32 type_id)
+{
+       struct btf_type *t = btf_type_by_id(d->btf, type_id);
+       __u16 kind = btf_kind(t);
+       enum btf_fwd_kind fwd_kind = BTF_INFO_KFLAG(t->info);
+

nit: don't break variables block in two parts, there shouldn't be empty lines

also please use btf_kflag(t)

Quoted 11 lines
tools/lib/bpf/btf.c +11 -0
+       struct btf_type *cand_t;
+       __u16 cand_kind;
+       __u32 cand_id = 0;
+
+       if (kind != BTF_KIND_FWD)
+               return 0;
+
+       /* Skip if this FWD already has a mapping */
+       if (type_id != d->map[type_id])
+               return 0;
+

[...]

author maintainer
#4 lore raw

Re: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations

In reply to Andrii Nakryiko
Quoted 1 line · Andrii Nakryiko

On Tue, Oct 25, 2022 at 3:28 PM Eduard Zingerman <[email protected]> wrote:

[...]

Quoted 14 lines
Quoted 14 lines
tools/lib/bpf/btf.c +14 -0
+
+/*
+ * Collect a `name_off_map` that maps type names to type ids for all
+ * canonical structs and unions. If the same name is shared by several
+ * canonical types use a special value 0 to indicate this fact.
+ */
+static int btf_dedup_fill_unique_names_map(struct btf_dedup *d, struct hashmap *names_map)
+{
+       int i, err = 0;
+       __u32 type_id, collision_id;
+       __u16 kind;
+       struct btf_type *t;
+
+       for (i = 0; i < d->btf->nr_types; i++) {
Quoted 8 lines

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587

Quoted 6 lines
tools/lib/bpf/btf.c +6 -0
+               type_id = d->btf->start_id + i;
+               t = btf_type_by_id(d->btf, type_id);
+               kind = btf_kind(t);
+
+               if (kind != BTF_KIND_STRUCT && kind != BTF_KIND_UNION)
+                       continue;

let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0

Interestingly this is necessary only for mixed enum / enum64 case. Forward enum declarations are resolved by bpf/btf.c:btf_dedup_prim_type:

	case BTF_KIND_ENUM:
		h = btf_hash_enum(t);
		for_each_dedup_cand(d, hash_entry, h) {
			cand_id = (__u32)(long)hash_entry->value;
			cand = btf_type_by_id(d->btf, cand_id);
			if (btf_equal_enum(t, cand)) {
				new_id = cand_id;
				break;
			}
			if (btf_compat_enum(t, cand)) {
				if (btf_is_enum_fwd(t)) {
					/* resolve fwd to full enum */
					new_id = cand_id;
					break;
				}
				/* resolve canonical enum fwd to full enum */
				d->map[cand_id] = type_id;
			}
		}
		break;
    // ... similar logic for ENUM64 ...

- btf_hash_enum ignores vlen when hashing;
- btf_compat_enum compares only names and sizes.

So, if forward and main declaration kinds match (either BTF_KIND_ENUM or BTF_KIND_ENUM64) the forward declaration would be removed. But if the kinds are different the forward declaration would remain. E.g.:

CU #1:
enum foo;
enum foo *a;

CU #2:
enum foo { x = 0xfffffffff };
enum foo *b;

BTF:
[1] ENUM64 'foo' encoding=UNSIGNED size=8 vlen=1

	'x' val=68719476735ULL

[2] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
[3] PTR '(anon)' type_id=1
[4] ENUM 'foo' encoding=UNSIGNED size=4 vlen=0
[5] PTR '(anon)' type_id=4

BTF_KIND_FWDs are unified during btf_dedup_struct_types but enum forward declarations are not. So it would be incorrect to add enum forward declaration unification logic to btf_dedup_resolve_fwds, because the following case would not be covered:

CU #1:
enum foo;
struct s { enum foo *a; } *a;

CU #2:
enum foo { x = 0xfffffffff };
struct s { enum foo *a; } *b;

Currently STRUCTs 's' are not de-duplicated.

I think that btf_dedup_prim_type should be adjusted to handle this case.

[...]

#37 lore raw
In reply to Andrii Nakryiko
Quoted 50 lines · Andrii Nakryiko
Quoted 49 lines · Yonghong Song
Quoted 36 lines · Andrii Nakryiko
Quoted 19 lines · Yonghong Song
Quoted 11 lines · Andrii Nakryiko
Quoted 9 lines · Eduard Zingerman

Hi BPF community,

AFAIK there is a long standing feature request to use kernel headers alongside vmlinux.h generated by bpftool. For example significant effort was put to add an attribute bpf_dominating_decl (see [1]) to clang, unfortunately this effort was stuck due to concerns regarding C language semantics.

Quoted 3 lines

Maybe we should make another attempt to implement bpf_dominating_decl? That seems like a more elegant solution than any other implemented or discussed alternative. Yonghong, WDYT?

Quoted 7 lines

I would say it would be very difficult for upstream to agree with bpf_dominating_decl. We already have lots of discussions and we likely won't be able to satisfy Aaron who wants us to emit adequate diagnostics which will involve lots of other work and he also thinks this is too far away from C standard and he wants us to implement in a llvm/clang tool which is not what we want.

Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like

#if !__is_type_defined(struct abc)
struct abc {
};
#endif

Quoted 5 lines

I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

Quoted 7 lines
Quoted 4 lines

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

Let us see whether we can get some upstream agreement or not.

Thanks for starting the conversation! I'll be following along.

I think this sort of approach assumes that vmlinux.h is included after any uapi headers, and would guard type definitions with

#if type_is_defined(foo)
struct foo {

};
#endif

...is that right? My concern is that the vmlinux.h definitions have the CO-RE attributes. From a BPF perspective, would we like the vmlinux.h definitions to dominate over UAPI definitions do you think, or does it matter?

I was wondering if there might be yet another way to crack this; if we did want the vmlinux.h type definitions to be authoritative because they have the preserve access index attribute, and because bpftool knows all vmlinux types, it could use that info to selectively redefine those type names such that we avoid name clashes when later including UAPI headers. Something like

#ifdef __VMLINUX_H__
//usual vmlinux.h type definitions
#endif /* __VMLINUX_H__ */

#ifdef __VMLINUX_ALIAS__
if !defined(timespec64)
#define timespec64 __VMLINUX_ALIAS__timespec64
#endif
// rest of the types define aliases here
#undef __VMLINUX_ALIAS__
#else /* unalias */
#if defined(timespec64)
#undef timespec64
#endif
// rest of types undef aliases here
#endif /* __VMLINUX_ALIAS__ */

Then the consumer does this:

#define __VMLINUX_ALIAS__
#include "vmlinux.h"
// include uapi headers
#include "vmlinux.h"

(the latter include of vmlinux.h is needed to undef all the type aliases)

I tried hacking up bpftool to support this aliasing scheme and while it is kind of hacky it does seem to work, aside from some issues with IPPROTO_* definitions - for the enumerated IPPROTO_ values linux/in.h does this:

enum {

  IPPROTO_IP = 0,               /* Dummy protocol for TCP               */
#define IPPROTO_IP              IPPROTO_IP
  IPPROTO_ICMP = 1,             /* Internet Control Message Protocol    */
#define IPPROTO_ICMP            IPPROTO_ICMP

...so our enum value definitions for IPPROTO_ values clash with the above definitions. These could be individually ifdef-guarded if needed though I think.

I can send the proof-of-concept patch if it would help, I just wanted to check in case that might be a workable path too, since it just requires changes to bpftool (and changes to in.h).

Thanks!

Alan

Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines
Quoted 55 lines

BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.

Quoted 5 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

[...]

Quoted 39 lines

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

   scripts/infer_header_guards.pl                | 191 +++++
   scripts/link-vmlinux.sh                       |  13 +-
   tools/bpf/bpftool/btf.c                       |   4 +-
   tools/lib/bpf/btf.c                           | 178 ++++-
   tools/lib/bpf/btf.h                           |   7 +-
   tools/lib/bpf/btf_dump.c                      | 232 +++++-
   .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
   tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
   .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
   .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
   .../progs/btf_dump_test_case_header_guards.c  |  94 +++
   .../bpf/test_uapi_header_guards_infer.sh      |  33 +
   .../selftests/bpf/test_uapi_headers.py        | 197 +++++
   13 files changed, 1816 insertions(+), 12 deletions(-)

create mode 100755 scripts/infer_header_guards.pl
create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py

--
2.34.1

#5 lore raw

Re: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations

In reply to Eduard Zingerman
Quoted 26 lines · Eduard Zingerman
Quoted 1 line · Andrii Nakryiko

On Tue, Oct 25, 2022 at 3:28 PM Eduard Zingerman <[email protected]> wrote:

[...]

Quoted 14 lines
Quoted 14 lines
tools/lib/bpf/btf.c +14 -0
+
+/*
+ * Collect a `name_off_map` that maps type names to type ids for all
+ * canonical structs and unions. If the same name is shared by several
+ * canonical types use a special value 0 to indicate this fact.
+ */
+static int btf_dedup_fill_unique_names_map(struct btf_dedup *d, struct hashmap *names_map)
+{
+       int i, err = 0;
+       __u32 type_id, collision_id;
+       __u16 kind;
+       struct btf_type *t;
+
+       for (i = 0; i < d->btf->nr_types; i++) {
Quoted 8 lines

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587

Quoted 6 lines
tools/lib/bpf/btf.c +6 -0
+               type_id = d->btf->start_id + i;
+               t = btf_type_by_id(d->btf, type_id);
+               kind = btf_kind(t);
+
+               if (kind != BTF_KIND_STRUCT && kind != BTF_KIND_UNION)
+                       continue;

let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0

Quoted 3 lines

Interestingly this is necessary only for mixed enum / enum64 case. Forward enum declarations are resolved by bpf/btf.c:btf_dedup_prim_type:

Ah, great catch! A forward can look like an enum to one CU but another CU can specify values that make it an enum64.

Quoted 54 lines
	case BTF_KIND_ENUM:
		h = btf_hash_enum(t);
		for_each_dedup_cand(d, hash_entry, h) {
			cand_id = (__u32)(long)hash_entry->value;
			cand = btf_type_by_id(d->btf, cand_id);
			if (btf_equal_enum(t, cand)) {
				new_id = cand_id;
				break;
			}
			if (btf_compat_enum(t, cand)) {
				if (btf_is_enum_fwd(t)) {
					/* resolve fwd to full enum */
					new_id = cand_id;
					break;
				}
				/* resolve canonical enum fwd to full enum */
				d->map[cand_id] = type_id;
			}
		}
		break;
    // ... similar logic for ENUM64 ...

- btf_hash_enum ignores vlen when hashing;
- btf_compat_enum compares only names and sizes.

So, if forward and main declaration kinds match (either BTF_KIND_ENUM or BTF_KIND_ENUM64) the forward declaration would be removed. But if the kinds are different the forward declaration would remain. E.g.:

CU #1:
enum foo;
enum foo *a;

CU #2:
enum foo { x = 0xfffffffff };
enum foo *b;

BTF:
[1] ENUM64 'foo' encoding=UNSIGNED size=8 vlen=1

	'x' val=68719476735ULL

[2] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
[3] PTR '(anon)' type_id=1
[4] ENUM 'foo' encoding=UNSIGNED size=4 vlen=0
[5] PTR '(anon)' type_id=4

BTF_KIND_FWDs are unified during btf_dedup_struct_types but enum forward declarations are not. So it would be incorrect to add enum forward declaration unification logic to btf_dedup_resolve_fwds, because the following case would not be covered:

CU #1:
enum foo;
struct s { enum foo *a; } *a;

Quoted 6 lines

CU #2:
enum foo { x = 0xfffffffff };
struct s { enum foo *a; } *b;

Currently STRUCTs 's' are not de-duplicated.

What if CU#1 is in base BTF and CU#2 in split module BTF? I think we'd explicitly want to avoid deduping "struct s" then since we can't be sure that it is the same enum they are pointing at. That's the logic we employ for structs at least, based upon the rationale that we can't feed back knowledge of types from module to kernel BTF since the latter is now fixed (Andrii, do correct me if I have this wrong). In such a case the enum is no longer standalone; it serves the purpose of allowing us to define a pointer to a module-specific type. We recently found some examples of this sort of thing with structs, where the struct was defined in module BTF, making dedup fail for some core kernel data types, but the problem was restricted to modules which _did_ define the type so wasn't a major driver of dedup failures. Not sure if there's many (any?) enum cases of this in practice.

I suppose if we could guarantee the dedup happened within the same object (kernel or module) we could relax this constraint though?

Alan

author maintainer
#6 lore raw

Re: [RFC bpf-next 01/12] libbpf: Deduplicate unambigous standalone forward declarations

Quoted 95 lines · Alan Maguire
Quoted 26 lines · Eduard Zingerman
Quoted 1 line · Andrii Nakryiko

On Tue, Oct 25, 2022 at 3:28 PM Eduard Zingerman <[email protected]> wrote:

[...]

Quoted 14 lines
Quoted 14 lines
tools/lib/bpf/btf.c +14 -0
+
+/*
+ * Collect a `name_off_map` that maps type names to type ids for all
+ * canonical structs and unions. If the same name is shared by several
+ * canonical types use a special value 0 to indicate this fact.
+ */
+static int btf_dedup_fill_unique_names_map(struct btf_dedup *d, struct hashmap *names_map)
+{
+       int i, err = 0;
+       __u32 type_id, collision_id;
+       __u16 kind;
+       struct btf_type *t;
+
+       for (i = 0; i < d->btf->nr_types; i++) {
Quoted 8 lines

tools/lib/bpf/btf.c static int btf_dedup_ref_types(struct btf_dedup *d) · line 4587

Quoted 6 lines
tools/lib/bpf/btf.c +6 -0
+               type_id = d->btf->start_id + i;
+               t = btf_type_by_id(d->btf, type_id);
+               kind = btf_kind(t);
+
+               if (kind != BTF_KIND_STRUCT && kind != BTF_KIND_UNION)
+                       continue;

let's also do ENUM FWD resolution. ENUM FWD is just ENUM with vlen=0

Quoted 3 lines

Interestingly this is necessary only for mixed enum / enum64 case. Forward enum declarations are resolved by bpf/btf.c:btf_dedup_prim_type:

Ah, great catch! A forward can look like an enum to one CU but another CU can specify values that make it an enum64.

Quoted 54 lines
	case BTF_KIND_ENUM:
		h = btf_hash_enum(t);
		for_each_dedup_cand(d, hash_entry, h) {
			cand_id = (__u32)(long)hash_entry->value;
			cand = btf_type_by_id(d->btf, cand_id);
			if (btf_equal_enum(t, cand)) {
				new_id = cand_id;
				break;
			}
			if (btf_compat_enum(t, cand)) {
				if (btf_is_enum_fwd(t)) {
					/* resolve fwd to full enum */
					new_id = cand_id;
					break;
				}
				/* resolve canonical enum fwd to full enum */
				d->map[cand_id] = type_id;
			}
		}
		break;
    // ... similar logic for ENUM64 ...

- btf_hash_enum ignores vlen when hashing;
- btf_compat_enum compares only names and sizes.

So, if forward and main declaration kinds match (either BTF_KIND_ENUM or BTF_KIND_ENUM64) the forward declaration would be removed. But if the kinds are different the forward declaration would remain. E.g.:

CU #1:
enum foo;
enum foo *a;

CU #2:
enum foo { x = 0xfffffffff };
enum foo *b;

BTF:
[1] ENUM64 'foo' encoding=UNSIGNED size=8 vlen=1

	'x' val=68719476735ULL

[2] INT 'long unsigned int' size=8 bits_offset=0 nr_bits=64 encoding=(none)
[3] PTR '(anon)' type_id=1
[4] ENUM 'foo' encoding=UNSIGNED size=4 vlen=0
[5] PTR '(anon)' type_id=4

BTF_KIND_FWDs are unified during btf_dedup_struct_types but enum forward declarations are not. So it would be incorrect to add enum forward declaration unification logic to btf_dedup_resolve_fwds, because the following case would not be covered:

CU #1:
enum foo;
struct s { enum foo *a; } *a;

Quoted 6 lines

CU #2:
enum foo { x = 0xfffffffff };
struct s { enum foo *a; } *b;

Currently STRUCTs 's' are not de-duplicated.

Quoted 12 lines

What if CU#1 is in base BTF and CU#2 in split module BTF? I think we'd explicitly want to avoid deduping "struct s" then since we can't be sure that it is the same enum they are pointing at. That's the logic we employ for structs at least, based upon the rationale that we can't feed back knowledge of types from module to kernel BTF since the latter is now fixed (Andrii, do correct me if I have this wrong). In such a case the enum is no longer standalone; it serves the purpose of allowing us to define a pointer to a module-specific type. We recently found some examples of this sort of thing with structs, where the struct was defined in module BTF, making dedup fail for some core kernel data types, but the problem was restricted to modules which _did_ define the type so wasn't a major driver of dedup failures. Not sure if there's many (any?) enum cases of this in practice.

Hi Alan,

As far as I understand the loop in btf_dedup_prim_types guarantees that only ids from the split module would be remapped:

	struct btf {
    	...
		/* BTF type ID of the first type in this BTF instance:
		 *   - for base BTF it's equal to 1;
		 *   - for split BTF it's equal to biggest type ID of base BTF plus 1.
		 */
		int start_id;
    	...
	}

    ...
	for (i = 0; i < d->btf->nr_types; i++) {
		err = btf_dedup_prim_type(d, d->btf->start_id + i);
		if (err)
			return err;
	}

Thus CU1:foo won't be updated to be CU2:foo and CU1:s will not be the same as CU2:s. Is that right or am I confused?

Thanks,
Eduard

Quoted 5 lines

I suppose if we could guarantee the dedup happened within the same object (kernel or module) we could relax this constraint though?

Alan

#38 lore raw
In reply to Alan Maguire
Quoted 5 lines · Alan Maguire
Quoted 5 lines
Quoted 5 lines

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

That would be a great clang feature.
Thanks Yonghong!

Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

Let us see whether we can get some upstream agreement or not.

Thanks for starting the conversation! I'll be following along.

I think this sort of approach assumes that vmlinux.h is included after any uapi headers, and would guard type definitions with

#if type_is_defined(foo)
struct foo {

Quoted 7 lines

};
#endif

...is that right? My concern is that the vmlinux.h definitions have the CO-RE attributes. From a BPF perspective, would we like the vmlinux.h definitions to dominate over UAPI definitions do you think, or does it matter?

I think it's totally fine to require #include "vmlinux.h" to be last. The attr(preserve_access_index) is only useful for kernel internal structs. uapi structs don't need it.

Quoted 26 lines

I was wondering if there might be yet another way to crack this; if we did want the vmlinux.h type definitions to be authoritative because they have the preserve access index attribute, and because bpftool knows all vmlinux types, it could use that info to selectively redefine those type names such that we avoid name clashes when later including UAPI headers. Something like

#ifdef __VMLINUX_H__
//usual vmlinux.h type definitions
#endif /* __VMLINUX_H__ */

#ifdef __VMLINUX_ALIAS__
if !defined(timespec64)
#define timespec64 __VMLINUX_ALIAS__timespec64
#endif
// rest of the types define aliases here
#undef __VMLINUX_ALIAS__
#else /* unalias */
#if defined(timespec64)
#undef timespec64
#endif
// rest of types undef aliases here
#endif /* __VMLINUX_ALIAS__ */

Quoted 8 lines

Then the consumer does this:

#define __VMLINUX_ALIAS__
#include "vmlinux.h"
// include uapi headers
#include "vmlinux.h"

(the latter include of vmlinux.h is needed to undef all the type aliases)

Sounds like a bunch of complexity for the use case that is not clear to me.

Quoted 7 lines

I tried hacking up bpftool to support this aliasing scheme and while it is kind of hacky it does seem to work, aside from some issues with IPPROTO_* definitions - for the enumerated IPPROTO_ values linux/in.h does this:

enum {

Quoted 8 lines
  IPPROTO_IP = 0,               /* Dummy protocol for TCP               */
#define IPPROTO_IP              IPPROTO_IP
  IPPROTO_ICMP = 1,             /* Internet Control Message Protocol    */
#define IPPROTO_ICMP            IPPROTO_ICMP

...so our enum value definitions for IPPROTO_ values clash with the above definitions. These could be individually ifdef-guarded if needed though I think.

Including vmlinux.h last won't have this enum conflicts, right?

Quoted 3 lines

I can send the proof-of-concept patch if it would help, I just wanted to check in case that might be a workable path too, since it just requires changes to bpftool (and changes to in.h).

I think changing the uapi header like in.h is no-go. Touching anything in uapi is too much of a risk.

author maintainer
#39 lore raw
Quoted 69 lines · Alexei Starovoitov
Quoted 5 lines · Alan Maguire
Quoted 5 lines
Quoted 5 lines

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

That would be a great clang feature.
Thanks Yonghong!

Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

Let us see whether we can get some upstream agreement or not.

Thanks for starting the conversation! I'll be following along.

I think this sort of approach assumes that vmlinux.h is included after any uapi headers, and would guard type definitions with

#if type_is_defined(foo)
struct foo {

Quoted 7 lines

};
#endif

...is that right? My concern is that the vmlinux.h definitions have the CO-RE attributes. From a BPF perspective, would we like the vmlinux.h definitions to dominate over UAPI definitions do you think, or does it matter?

I think it's totally fine to require #include "vmlinux.h" to be last. The attr(preserve_access_index) is only useful for kernel internal structs. uapi structs don't need it.

Quoted 33 lines

I was wondering if there might be yet another way to crack this; if we did want the vmlinux.h type definitions to be authoritative because they have the preserve access index attribute, and because bpftool knows all vmlinux types, it could use that info to selectively redefine those type names such that we avoid name clashes when later including UAPI headers. Something like

#ifdef __VMLINUX_H__
//usual vmlinux.h type definitions
#endif /* __VMLINUX_H__ */

#ifdef __VMLINUX_ALIAS__
if !defined(timespec64)
#define timespec64 __VMLINUX_ALIAS__timespec64
#endif
// rest of the types define aliases here
#undef __VMLINUX_ALIAS__
#else /* unalias */
#if defined(timespec64)
#undef timespec64
#endif
// rest of types undef aliases here
#endif /* __VMLINUX_ALIAS__ */

Then the consumer does this:

#define __VMLINUX_ALIAS__
#include "vmlinux.h"
// include uapi headers
#include "vmlinux.h"

Quoted 4 lines
Quoted 1 line

(the latter include of vmlinux.h is needed to undef all the type aliases)

Sounds like a bunch of complexity for the use case that is not clear to me.

Well, my RFC is not shy of complexity :)
What Alan suggests should solve the confilicts described in [1] or any other confilicts of such kind.

[1] https://lore.kernel.org/bpf/[email protected]/

Quoted 24 lines
Quoted 7 lines

I tried hacking up bpftool to support this aliasing scheme and while it is kind of hacky it does seem to work, aside from some issues with IPPROTO_* definitions - for the enumerated IPPROTO_ values linux/in.h does this:

enum {

Quoted 8 lines
  IPPROTO_IP = 0,               /* Dummy protocol for TCP               */
#define IPPROTO_IP              IPPROTO_IP
  IPPROTO_ICMP = 1,             /* Internet Control Message Protocol    */
#define IPPROTO_ICMP            IPPROTO_ICMP

...so our enum value definitions for IPPROTO_ values clash with the above definitions. These could be individually ifdef-guarded if needed though I think.

Including vmlinux.h last won't have this enum conflicts, right?

Quoted 3 lines

I can send the proof-of-concept patch if it would help, I just wanted to check in case that might be a workable path too, since it just requires changes to bpftool (and changes to in.h).

I think changing the uapi header like in.h is no-go. Touching anything in uapi is too much of a risk.

#40 lore raw
Quoted 73 lines · Eduard Zingerman
Quoted 71 lines · Alexei Starovoitov
Quoted 5 lines · Alan Maguire
Quoted 5 lines
Quoted 5 lines

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

That would be a great clang feature.
Thanks Yonghong!

Quoted 14 lines
Quoted 6 lines
Quoted 3 lines

Let us see whether we can get some upstream agreement or not.

Thanks for starting the conversation! I'll be following along.

I think this sort of approach assumes that vmlinux.h is included after any uapi headers, and would guard type definitions with

#if type_is_defined(foo)
struct foo {

Quoted 7 lines

};
#endif

...is that right? My concern is that the vmlinux.h definitions have the CO-RE attributes. From a BPF perspective, would we like the vmlinux.h definitions to dominate over UAPI definitions do you think, or does it matter?

I think it's totally fine to require #include "vmlinux.h" to be last. The attr(preserve_access_index) is only useful for kernel internal structs. uapi structs don't need it.

Quoted 34 lines

I was wondering if there might be yet another way to crack this; if we did want the vmlinux.h type definitions to be authoritative because they have the preserve access index attribute, and because bpftool knows all vmlinux types, it could use that info to selectively redefine those type names such that we avoid name clashes when later including UAPI headers. Something like

#ifdef __VMLINUX_H__
//usual vmlinux.h type definitions
#endif /* __VMLINUX_H__ */

#ifdef __VMLINUX_ALIAS__
if !defined(timespec64)
#define timespec64 __VMLINUX_ALIAS__timespec64
#endif
// rest of the types define aliases here
#undef __VMLINUX_ALIAS__
#else /* unalias */
#if defined(timespec64)
#undef timespec64
#endif
// rest of types undef aliases here
#endif /* __VMLINUX_ALIAS__ */

Then the consumer does this:

#define __VMLINUX_ALIAS__
#include "vmlinux.h"
// include uapi headers
#include "vmlinux.h"

(the latter include of vmlinux.h is needed to undef all the type aliases)

Quoted 8 lines
Quoted 2 lines

Sounds like a bunch of complexity for the use case that is not clear to me.

Well, my RFC is not shy of complexity :)
What Alan suggests should solve the confilicts described in [1] or any other confilicts of such kind.

[1] https://lore.kernel.org/bpf/[email protected]/

I don't quite see how renaming all types in the 1st vmlinux.h will help with name collisions inside enum {}.
The enums will conflict with 2nd vmlinux.h too. Unless the proposal is to rename them as well,
but then what's the point?

author maintainer
#41 lore raw
In reply to Yonghong Song
Quoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line

[...]

Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like

#if !__is_type_defined(struct abc)
struct abc {
};
#endif

Quoted 5 lines

I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?

Quoted 6 lines

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

Let us see whether we can get some upstream agreement or not.

I did a small investigation of this feature.

The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.

The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of __is_type_defined construct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.

However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.

This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:

$ cat test.c

struct foo { int x; };

  #if __is_type_defined(foo)
    const int x = 1;
  #else
    const int x = 2;
  #endif
$ clang -cc1 -ast-print test.c -o -
  struct foo {
      int x;
  };
  const int x = 1;
$ clang -E test.c -o -
  # ... some line directives ...
  struct foo { int x; };
  const int x = 2;

Note that __is_type_defined is computed to different value in the first and second calls. This is so because semantic analysis (AST, symbol table) is not done for -E.

It also breaks that C11 standard which clearly separates pre-processing and semantic analysis phases, see [2] 5.1.1.2.

So, my conclusion is as follows: this is technically possible in clang but has no chance to reach llvm upstream.

Thanks,
Eduard

[1] https://github.com/llvm/llvm-project/compare/main...eddyz87:llvm-project:is-type-defined-experiment
[2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf

Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines

BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.

Quoted 5 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

[...]

Quoted 39 lines

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

   scripts/infer_header_guards.pl                | 191 +++++
   scripts/link-vmlinux.sh                       |  13 +-
   tools/bpf/bpftool/btf.c                       |   4 +-
   tools/lib/bpf/btf.c                           | 178 ++++-
   tools/lib/bpf/btf.h                           |   7 +-
   tools/lib/bpf/btf_dump.c                      | 232 +++++-
   .../selftests/bpf/good_uapi_headers.txt       | 677 ++++++++++++++++++
   tools/testing/selftests/bpf/prog_tests/btf.c  | 152 ++++
   .../selftests/bpf/prog_tests/btf_dump.c       |  11 +-
   .../bpf/progs/btf_dump_test_case_decl_tag.c   |  39 +
   .../progs/btf_dump_test_case_header_guards.c  |  94 +++
   .../bpf/test_uapi_header_guards_infer.sh      |  33 +
   .../selftests/bpf/test_uapi_headers.py        | 197 +++++
   13 files changed, 1816 insertions(+), 12 deletions(-)

create mode 100755 scripts/infer_header_guards.pl
create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py

--
2.34.1

#42 lore raw
Quoted 64 lines · Eduard Zingerman
Quoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line

[...]

Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like

#if !__is_type_defined(struct abc)
struct abc {
};
#endif

Quoted 5 lines

I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?

Quoted 6 lines

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

Let us see whether we can get some upstream agreement or not.

I did a small investigation of this feature.

The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.

The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of __is_type_defined construct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.

However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.

This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:

$ cat test.c
   struct foo { int x; };
   
   #if __is_type_defined(foo)
     const int x = 1;
   #else
     const int x = 2;
   #endif
$ clang -cc1 -ast-print test.c -o -
   struct foo {
       int x;
   };
   const int x = 1;
Quoted 5 lines
$ clang -E test.c -o -
   # ... some line directives ...
   struct foo { int x; };
   const int x = 2;

Is it any chance '-E' could output the same one as '-cc1 -ast-print'? That is, even with -E we could do some semantics analysis as well, using either current clang semantics analysis or creating an minimal version of sema analysis in preprocessor itself?

Quoted 75 lines

Note that __is_type_defined is computed to different value in the first and second calls. This is so because semantic analysis (AST, symbol table) is not done for -E.

It also breaks that C11 standard which clearly separates pre-processing and semantic analysis phases, see [2] 5.1.1.2.

So, my conclusion is as follows: this is technically possible in clang but has no chance to reach llvm upstream.

Thanks,
Eduard

[1] https://github.com/llvm/llvm-project/compare/main...eddyz87:llvm-project:is-type-defined-experiment
[2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf

Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines

BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.

Quoted 5 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

[...]

Quoted 39 lines

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

scripts/infer_header_guards.pl | 191 +++++
scripts/link-vmlinux.sh | 13 +-
tools/bpf/bpftool/btf.c | 4 +-
tools/lib/bpf/btf.c | 178 ++++-
tools/lib/bpf/btf.h | 7 +-
tools/lib/bpf/btf_dump.c | 232 +++++-
.../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++
tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++
.../selftests/bpf/prog_tests/btf_dump.c | 11 +-
.../bpf/progs/btf_dump_test_case_decl_tag.c | 39 +
.../progs/btf_dump_test_case_header_guards.c | 94 +++
.../bpf/test_uapi_header_guards_infer.sh | 33 +
.../selftests/bpf/test_uapi_headers.py | 197 +++++
13 files changed, 1816 insertions(+), 12 deletions(-)
create mode 100755 scripts/infer_header_guards.pl
create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py

--
2.34.1

author maintainer
#43 lore raw
Quoted 68 lines · Yonghong Song
Quoted 66 lines · Eduard Zingerman
Quoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line

[...]

Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like

#if !__is_type_defined(struct abc)
struct abc {
};
#endif

Quoted 5 lines

I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?

Quoted 6 lines

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

Let us see whether we can get some upstream agreement or not.

I did a small investigation of this feature.

The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.

The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of __is_type_defined construct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.

However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.

This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:

$ cat test.c
   struct foo { int x; };
   
   #if __is_type_defined(foo)
     const int x = 1;
   #else
     const int x = 2;
   #endif
$ clang -cc1 -ast-print test.c -o -
   struct foo {
       int x;
   };
   const int x = 1;
$ clang -E test.c -o -
Quoted 8 lines
Quoted 3 lines
   # ... some line directives ...
   struct foo { int x; };
   const int x = 2;

Is it any chance '-E' could output the same one as '-cc1 -ast-print'? That is, even with -E we could do some semantics analysis as well, using either current clang semantics analysis or creating an minimal version of sema analysis in preprocessor itself?

Sema drives consumption of tokens from Preprocessor. Calls to Preprocessor are done on a parsing recursive descent. Extracting a stream of tokens would require an incremental parser instead.

A minimal version of such parser is possible to implement for C. It might be the case that matching open / closing braces and identifiers following 'struct' / 'union' / 'enum' keywords might be almost sufficient but I need to try to be sure (e.g. it is more complex for 'typedef').

I can work on it but I don't think there is a chance to upstream this work.

Thanks,
Eduard

Quoted 76 lines
Quoted 76 lines

Note that __is_type_defined is computed to different value in the first and second calls. This is so because semantic analysis (AST, symbol table) is not done for -E.

It also breaks that C11 standard which clearly separates pre-processing and semantic analysis phases, see [2] 5.1.1.2.

So, my conclusion is as follows: this is technically possible in clang but has no chance to reach llvm upstream.

Thanks,
Eduard

[1] https://github.com/llvm/llvm-project/compare/main...eddyz87:llvm-project:is-type-defined-experiment
[2] https://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf

Quoted 56 lines
Quoted 56 lines
Quoted 56 lines
Quoted 56 lines

BTW, I suggest splitting libbpf btf_dedup and btf_dump changes into a separate series and sending them as non-RFC sooner. Those improvements are independent of all the header guards stuff, let's get them landed sooner.

Quoted 5 lines

After some discussion with Alexei and Yonghong I'd like to request your comments regarding a somewhat brittle and partial solution to this issue that relies on adding #ifndef FOO_H ... #endif guards in the generated vmlinux.h.

[...]

Quoted 39 lines

Eduard Zingerman (12):
libbpf: Deduplicate unambigous standalone forward declarations selftests/bpf: Tests for standalone forward BTF declarations
deduplication
libbpf: Support for BTF_DECL_TAG dump in C format selftests/bpf: Tests for BTF_DECL_TAG dump in C format
libbpf: Header guards for selected data structures in vmlinux.h selftests/bpf: Tests for header guards printing in BTF dump
bpftool: Enable header guards generation
kbuild: Script to infer header guard values for uapi headers
kbuild: Header guards for types from include/uapi/*.h in kernel BTF selftests/bpf: Script to verify uapi headers usage with vmlinux.h selftests/bpf: Known good uapi headers for test_uapi_headers.py selftests/bpf: script for infer_header_guards.pl testing

scripts/infer_header_guards.pl | 191 +++++
scripts/link-vmlinux.sh | 13 +-
tools/bpf/bpftool/btf.c | 4 +-
tools/lib/bpf/btf.c | 178 ++++-
tools/lib/bpf/btf.h | 7 +-
tools/lib/bpf/btf_dump.c | 232 +++++-
.../selftests/bpf/good_uapi_headers.txt | 677 ++++++++++++++++++
tools/testing/selftests/bpf/prog_tests/btf.c | 152 ++++
.../selftests/bpf/prog_tests/btf_dump.c | 11 +-
.../bpf/progs/btf_dump_test_case_decl_tag.c | 39 +
.../progs/btf_dump_test_case_header_guards.c | 94 +++
.../bpf/test_uapi_header_guards_infer.sh | 33 +
.../selftests/bpf/test_uapi_headers.py | 197 +++++
13 files changed, 1816 insertions(+), 12 deletions(-)
create mode 100755 scripts/infer_header_guards.pl
create mode 100644 tools/testing/selftests/bpf/good_uapi_headers.txt create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_decl_tag.c create mode 100644 tools/testing/selftests/bpf/progs/btf_dump_test_case_header_guards.c create mode 100755 tools/testing/selftests/bpf/test_uapi_header_guards_infer.sh create mode 100755 tools/testing/selftests/bpf/test_uapi_headers.py

--
2.34.1

#44 lore raw
Quoted 83 lines · Eduard Zingerman
Quoted 68 lines · Yonghong Song
Quoted 66 lines · Eduard Zingerman
Quoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line

[...]

Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like

#if !__is_type_defined(struct abc)
struct abc {
};
#endif

Quoted 5 lines

I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?

Quoted 6 lines

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

Let us see whether we can get some upstream agreement or not.

I did a small investigation of this feature.

The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.

The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of __is_type_defined construct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.

However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.

This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:

$ cat test.c
   struct foo { int x; };

   #if __is_type_defined(foo)
     const int x = 1;
   #else
     const int x = 2;
   #endif
$ clang -cc1 -ast-print test.c -o -
   struct foo {
       int x;
   };
   const int x = 1;
$ clang -E test.c -o -
Quoted 8 lines
Quoted 3 lines
   # ... some line directives ...
   struct foo { int x; };
   const int x = 2;

Is it any chance '-E' could output the same one as '-cc1 -ast-print'? That is, even with -E we could do some semantics analysis as well, using either current clang semantics analysis or creating an minimal version of sema analysis in preprocessor itself?

Sema drives consumption of tokens from Preprocessor. Calls to Preprocessor are done on a parsing recursive descent. Extracting a stream of tokens would require an incremental parser instead.

Quoted 7 lines

A minimal version of such parser is possible to implement for C. It might be the case that matching open / closing braces and identifiers following 'struct' / 'union' / 'enum' keywords might be almost sufficient but I need to try to be sure (e.g. it is more complex for 'typedef').

I can work on it but I don't think there is a chance to upstream this work.

Right. It's going to be C only.
C++ with namespaces and nested class decls won't work with simple type parser.

On the other side if we're asking preprocessor to look for 'struct foo' and remember that 'foo' is a type maybe we can add a regex-search instead?
It would be a bit more generic and will work for basic union/struct foo definition?
Something like instead of:
#if __is_type_defined(foo)
use:
#if regex(struct[\t]+foo)

enums are harder in this approach, but higher chance to land?

regex() would mean "search for this pattern in the file until this line.

Or some other preprocessor "language" tricks?

For example:
The preprocessor would grep for 'struct *' in a single line while processing a file and emit #define __secret_prefix_##$1 where $1 would be a capture from "single line regex". Then later in the same file instead of:
#if __is_type_defined(foo)
use:
#ifdef __secret_prefix_foo

This "single line regex" may look like:
#if regex_in_any_later_line(struct[\t]+[a-zA-Z_]+) define __secret_prefix_$2

author maintainer
#45 lore raw
Quoted 106 lines · Alexei Starovoitov
Quoted 83 lines · Eduard Zingerman
Quoted 68 lines · Yonghong Song
Quoted 66 lines · Eduard Zingerman
Quoted 16 lines · Yonghong Song
Quoted 10 lines
Quoted 1 line

[...]

Ok, could we change the problem to detecting if some type is defined. Would it be possible to have something like

#if !__is_type_defined(struct abc)
struct abc {
};
#endif

Quoted 5 lines

I think we talked about this and there were problems with this approach, but I don't remember details and how insurmountable the problem is. Having a way to check whether some type is defined would be very useful even outside of -target bpf parlance, though, so maybe it's the problem worth attacking?

Quoted 6 lines

Yes, we discussed this before. This will need to add additional work in preprocessor. I just made a discussion topic in llvm discourse

https://discourse.llvm.org/t/add-a-type-checking-macro-is-type-defined-type/66268

Let us see whether we can get some upstream agreement or not.

I did a small investigation of this feature.

The main pre-requirement is construction of the symbol table during source code pre-processing, which implies necessity to parse the source code at the same time. It is technically possible in clang, as lexing, pre-processing and AST construction happens at the same time when in compilation mode.

The prototype is available here [1], it includes:
- Change in the pre-processor that adds an optional callback "IsTypeDefinedFn" & necessary parsing of __is_type_defined construct.
- Change in Sema module (responsible for parsing/AST & symbol table) that installs the appropriate "IsTypeDefinedFn" in the pre-processor instance.

However, this prototype builds a backward dependency between pre-processor and semantic analysis. There are currently no such dependencies in the clang code base.

This makes it impossible to do pre-processing and compilation separately, e.g. consider the following example:

$ cat test.c
   struct foo { int x; };

   #if __is_type_defined(foo)
     const int x = 1;
   #else
     const int x = 2;
   #endif
$ clang -cc1 -ast-print test.c -o -
   struct foo {
       int x;
   };
   const int x = 1;
$ clang -E test.c -o -
Quoted 8 lines
Quoted 3 lines
   # ... some line directives ...
   struct foo { int x; };
   const int x = 2;

Is it any chance '-E' could output the same one as '-cc1 -ast-print'? That is, even with -E we could do some semantics analysis as well, using either current clang semantics analysis or creating an minimal version of sema analysis in preprocessor itself?

Sema drives consumption of tokens from Preprocessor. Calls to Preprocessor are done on a parsing recursive descent. Extracting a stream of tokens would require an incremental parser instead.

Quoted 7 lines

A minimal version of such parser is possible to implement for C. It might be the case that matching open / closing braces and identifiers following 'struct' / 'union' / 'enum' keywords might be almost sufficient but I need to try to be sure (e.g. it is more complex for 'typedef').

I can work on it but I don't think there is a chance to upstream this work.

Right. It's going to be C only.
C++ with namespaces and nested class decls won't work with simple type parser.

On the other side if we're asking preprocessor to look for 'struct foo' and remember that 'foo' is a type maybe we can add a regex-search instead?
It would be a bit more generic and will work for basic union/struct foo definition?
Something like instead of:
#if __is_type_defined(foo)
use:
#if regex(struct[\t]+foo)

Quoted 6 lines

enums are harder in this approach, but higher chance to land?

regex() would mean "search for this pattern in the file until this line.

Or some other preprocessor "language" tricks?

I talked to Yonhong today and he suggests to investigate whether pre-processor changes could be made BPF target specific. E.g. there are extension points in the clang pre-processor right now but those for tooling. There might be a way to extend this mechanism to allow target specific pre-processor behavior. I'll take a look and write another email here.

Quoted 11 lines

For example:
The preprocessor would grep for 'struct *' in a single line while processing a file and emit #define __secret_prefix_##$1 where $1 would be a capture from "single line regex". Then later in the same file instead of:
#if __is_type_defined(foo)
use:
#ifdef __secret_prefix_foo

This "single line regex" may look like:
#if regex_in_any_later_line(struct[\t]+[a-zA-Z_]+) define __secret_prefix_$2