Skip to contentSkip to conversation

resolve_btfids: Support for BTF modifications

PATCH Superseded Ihor Solodrai 6 participants 28 messages Dec 5, 2025 – Dec 16, 2025

Subsystems (6)
Kbuild Kselftest Library code BPF +2 more subsystems

v3 of 7 Superseded Latest is v7 →

lore
Share & tools
lore

Ihor Solodrai

author reviewer
#0 lore raw

v3 cover resolve_btfids: Support for BTF modifications

This series changes resolve_btfids and kernel build scripts to enable BTF transformations in resolve_btfids. Main motivation for enhancing resolve_btfids is to reduce dependency of the kernel build on pahole capabilities [1] and enable BTF features and optimizations [2][3] particular to the kernel.

Patches #1-#3 in the series are non-functional refactoring in resolve_btfids.

Patch #4 changes minimum version of pahole required for CONFIG_DEBUG_INFO_BTF to v1.22

Patch #5 makes a small prep change in selftests/bpf build.

The last patch (#6) makes significant changes in resolve_btfids and introduces scripts/gen-btf.sh. See implementation details in the patch description.

Successful CI run: https://github.com/kernel-patches/bpf/actions/runs/19976024062?pr=10438

[1] https://lore.kernel.org/dwarves/[email protected]/
[2] https://lore.kernel.org/bpf/[email protected]/
[3] https://lore.kernel.org/bpf/[email protected]/

---

v2->v3:
- add patch #4 bumping minimum pahole version (Andrii, Alan)
- add patch #5 pre-fixing resolve_btfids test (Donglin)
- add GEN_BTF var and assemble RESOLVE_BTFIDS_FLAGS in Makefile.btf (Alan)
- implement --distill_base flag in resolve_btfids, set it depending on KBUILD_EXTMOD in Makefile.btf (Eduard)
- various implementation nits, see the v2 thread for details (Andrii, Eduard)

v2: https://lore.kernel.org/bpf/[email protected]/

v1->v2:
- gen-btf.sh and other shell script fixes (Donglin)
- update selftests build (Donglin)
- generate .BTF.base only when KBUILD_EXTMOD is set (Alan)
- proper endianness handling for cross-compilation
- change elf_begin mode from ELF_C_RDWR_MMAP to ELF_C_READ_MMAP_PRIVATE
- remove compressed_section_fix()
- nit NULL check in patch #3 (suggested by AI)

v1: https://lore.kernel.org/bpf/[email protected]/

Ihor Solodrai (6):
resolve_btfids: Rename object btf field to btf_path
resolve_btfids: Factor out load_btf()
resolve_btfids: Introduce enum btf_id_kind lib/Kconfig.debug: Set the minimum required pahole version to v1.22 selftests/bpf: Run resolve_btfids only for relevant .test.o objects
resolve_btfids: change in-place update with raw binary output

 MAINTAINERS                                   |   1 +
 lib/Kconfig.debug                             |  13 +-
 scripts/Makefile.btf                          |  26 +-
 scripts/Makefile.modfinal                     |   5 +-
 scripts/Makefile.vmlinux                      |   2 +-
 scripts/gen-btf.sh                            | 157 ++++++++
 scripts/link-vmlinux.sh                       |  46 +--
 tools/bpf/resolve_btfids/main.c               | 355 ++++++++++++------
 tools/sched_ext/README.md                     |   1 -
 tools/testing/selftests/bpf/.gitignore        |   3 +
 tools/testing/selftests/bpf/Makefile          |  11 +-
 .../selftests/bpf/prog_tests/resolve_btfids.c |   4 +-
 12 files changed, 434 insertions(+), 190 deletions(-)
 create mode 100755 scripts/gen-btf.sh
Signature · 1 line
2.52.0
author reviewer
#1 lore raw

PATCH v3 1/6 resolve_btfids: Rename object btf field to btf_path

Rename the member of struct object holding the path to BTF data if provided via --btf arg. btf_path is less ambiguous.

---
 tools/bpf/resolve_btfids/main.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
tools/bpf/resolve_btfids/main.c +5 -5
index d47191c6e55e..164f0c941f04 100644
@@ -113,7 +113,7 @@ struct btf_id {
 
 struct object {
 	const char *path;
-	const char *btf;
+	const char *btf_path;
 	const char *base_btf_path;
 
 	struct {
@@ -550,11 +550,11 @@ static int symbols_resolve(struct object *obj)
 		}
 	}
 
-	btf = btf__parse_split(obj->btf ?: obj->path, base_btf);
+	btf = btf__parse_split(obj->btf_path ?: obj->path, base_btf);
 	err = libbpf_get_error(btf);
 	if (err) {
 		pr_err("FAILED: load BTF from %s: %s\n",
-			obj->btf ?: obj->path, strerror(-err));
+			obj->btf_path ?: obj->path, strerror(-err));
 		goto out;
 	}
 
@@ -790,8 +790,8 @@ int main(int argc, const char **argv)
 	struct option btfid_options[] = {
 		OPT_INCR('v', "verbose", &verbose,
 			 "be more verbose (show errors, etc)"),
-		OPT_STRING(0, "btf", &obj.btf, "BTF data",
-			   "BTF data"),
+		OPT_STRING(0, "btf", &obj.btf_path, "file",
+			   "path to a file with input BTF data"),
 		OPT_STRING('b', "btf_base", &obj.base_btf_path, "file",
 			   "path of file providing base BTF"),
 		OPT_BOOLEAN(0, "fatal_warnings", &fatal_warnings,
Signature · 1 line
2.52.0
author reviewer
#3 lore raw

PATCH v3 2/6 resolve_btfids: Factor out load_btf()

In reply to Ihor Solodrai

Increase the lifetime of parsed BTF in resolve_btfids by factoring load_btf() routine out of symbols_resolve() and storing the base_btf and btf pointers in the struct object.

---
 tools/bpf/resolve_btfids/main.c | 47 ++++++++++++++++++++++++---------
 1 file changed, 34 insertions(+), 13 deletions(-)
tools/bpf/resolve_btfids/main.c +34 -13
index 164f0c941f04..b4caae1170dd 100644
@@ -116,6 +116,9 @@ struct object {
 	const char *btf_path;
 	const char *base_btf_path;
 
+	struct btf *btf;
+	struct btf *base_btf;
+
 	struct {
 		int		 fd;
 		Elf		*elf;
@@ -529,16 +532,10 @@ static int symbols_collect(struct object *obj)
 	return 0;
 }
 
-static int symbols_resolve(struct object *obj)
+static int load_btf(struct object *obj)
 {
-	int nr_typedefs = obj->nr_typedefs;
-	int nr_structs  = obj->nr_structs;
-	int nr_unions   = obj->nr_unions;
-	int nr_funcs    = obj->nr_funcs;
-	struct btf *base_btf = NULL;
-	int err, type_id;
-	struct btf *btf;
-	__u32 nr_types;
+	struct btf *base_btf = NULL, *btf = NULL;
+	int err;
 
 	if (obj->base_btf_path) {
 		base_btf = btf__parse(obj->base_btf_path, NULL);
@@ -546,7 +543,7 @@ static int symbols_resolve(struct object *obj)
 		if (err) {
 			pr_err("FAILED: load base BTF from %s: %s\n",
 			       obj->base_btf_path, strerror(-err));
-			return -1;
+			goto out_err;
 		}
 	}
 
@@ -555,9 +552,30 @@ static int symbols_resolve(struct object *obj)
 	if (err) {
 		pr_err("FAILED: load BTF from %s: %s\n",
 			obj->btf_path ?: obj->path, strerror(-err));
-		goto out;
+		goto out_err;
 	}
 
+	obj->base_btf = base_btf;
+	obj->btf = btf;
+
+	return 0;
+
+out_err:
+	btf__free(base_btf);
+	btf__free(btf);
+	return err;
+}
+
+static int symbols_resolve(struct object *obj)
+{
+	int nr_typedefs = obj->nr_typedefs;
+	int nr_structs  = obj->nr_structs;
+	int nr_unions   = obj->nr_unions;
+	int nr_funcs    = obj->nr_funcs;
+	struct btf *btf = obj->btf;
+	int err, type_id;
+	__u32 nr_types;
+
 	err = -1;
 	nr_types = btf__type_cnt(btf);
 
@@ -615,8 +633,6 @@ static int symbols_resolve(struct object *obj)
 
 	err = 0;
 out:
-	btf__free(base_btf);
-	btf__free(btf);
 	return err;
 }
 
@@ -824,6 +840,9 @@ int main(int argc, const char **argv)
 	if (symbols_collect(&obj))
 		goto out;
 
+	if (load_btf(&obj))
+		goto out;
+
 	if (symbols_resolve(&obj))
 		goto out;
 
@@ -833,6 +852,8 @@ int main(int argc, const char **argv)
 	if (!(fatal_warnings && warnings))
 		err = 0;
 out:
+	btf__free(obj.base_btf);
+	btf__free(obj.btf);
 	if (obj.efile.elf) {
 		elf_end(obj.efile.elf);
 		close(obj.efile.fd);
Signature · 1 line
2.52.0
#4 lore raw

Re: [PATCH bpf-next v3 2/6] resolve_btfids: Factor out load_btf()

Can btf__free() safely handle ERR_PTR values here?
Quoted 4 lines
tools/bpf/resolve_btfids/main.c +0 -0
index 164f0c941f04..b4caae1170dd 100644

[ ... ]

Quoted 37 lines
tools/bpf/resolve_btfids/main.c +3 -9
@@ -529,16 +532,10 @@ static int symbols_collect(struct object *obj)
 	return 0;
 }
 
-static int symbols_resolve(struct object *obj)
+static int load_btf(struct object *obj)
 {
-	int nr_typedefs = obj->nr_typedefs;
-	int nr_structs  = obj->nr_structs;
-	int nr_unions   = obj->nr_unions;
-	int nr_funcs    = obj->nr_funcs;
-	struct btf *base_btf = NULL;
-	int err, type_id;
-	struct btf *btf;
-	__u32 nr_types;
+	struct btf *base_btf = NULL, *btf = NULL;
+	int err;
 
 	if (obj->base_btf_path) {
 		base_btf = btf__parse(obj->base_btf_path, NULL);
tools/bpf/resolve_btfids/main.c +2 -2
 		err = libbpf_get_error(base_btf);
 		if (err) {
 			pr_err("FAILED: load base BTF from %s: %s\n",
 			       obj->base_btf_path, strerror(-err));
-			return -1;
+			goto out_err;
 		}
 	}
 
 	btf = btf__parse_split(obj->btf_path ?: obj->path, base_btf);
 	err = libbpf_get_error(btf);
 	if (err) {
 		pr_err("FAILED: load BTF from %s: %s\n",
 			obj->btf_path ?: obj->path, strerror(-err));
-		goto out;
+		goto out_err;
 	}

tools/bpf/resolve_btfids/main.c static int symbols_resolve(struct object *obj) · line 566

Quoted 10 lines
tools/bpf/resolve_btfids/main.c +9 -0
 
+	obj->base_btf = base_btf;
+	obj->btf = btf;
+
+	return 0;
+
+out_err:
+	btf__free(base_btf);
+	btf__free(btf);
+	return err;

Can btf__free() safely handle ERR_PTR values here?

When btf__parse() fails on line 541, it returns ERR_PTR(-err), and libbpf_get_error() extracts the error code, but base_btf still holds the ERR_PTR value. The error path then calls btf__free(base_btf) where base_btf is ERR_PTR rather than NULL or a valid pointer.

The same issue occurs if btf__parse_split() fails on line 550, where btf contains ERR_PTR when btf__free(btf) is called.

In the old code at symbols_resolve(), if base_btf was ERR_PTR, the function would return -1 immediately without calling btf__free(), avoiding this issue. The new error handling with goto out_err changes this behavior.

Quoted 4 lines
tools/bpf/resolve_btfids/main.c +4 -0
+}
+
+static int symbols_resolve(struct object *obj)
+{

tools/bpf/resolve_btfids/main.c static int symbols_resolve(struct object *obj) · line 580

Quoted 10 lines
tools/bpf/resolve_btfids/main.c +8 -0
+	int nr_typedefs = obj->nr_typedefs;
+	int nr_structs  = obj->nr_structs;
+	int nr_unions   = obj->nr_unions;
+	int nr_funcs    = obj->nr_funcs;
+	struct btf *btf = obj->btf;
+	int err, type_id;
+	__u32 nr_types;
+
 	err = -1;
 	nr_types = btf__type_cnt(btf);

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19978081551

author reviewer
#5 lore raw

Re: [PATCH bpf-next v3 2/6] resolve_btfids: Factor out load_btf()

Quoted 62 lines · On 12/5/25 2:57 PM, [email protected] wrote:
Quoted 4 lines
tools/bpf/resolve_btfids/main.c +0 -0
index 164f0c941f04..b4caae1170dd 100644

[ ... ]

Quoted 37 lines
tools/bpf/resolve_btfids/main.c +3 -9
@@ -529,16 +532,10 @@ static int symbols_collect(struct object *obj)
 	return 0;
 }
 
-static int symbols_resolve(struct object *obj)
+static int load_btf(struct object *obj)
 {
-	int nr_typedefs = obj->nr_typedefs;
-	int nr_structs  = obj->nr_structs;
-	int nr_unions   = obj->nr_unions;
-	int nr_funcs    = obj->nr_funcs;
-	struct btf *base_btf = NULL;
-	int err, type_id;
-	struct btf *btf;
-	__u32 nr_types;
+	struct btf *base_btf = NULL, *btf = NULL;
+	int err;
 
 	if (obj->base_btf_path) {
 		base_btf = btf__parse(obj->base_btf_path, NULL);
tools/bpf/resolve_btfids/main.c +2 -2
 		err = libbpf_get_error(base_btf);
 		if (err) {
 			pr_err("FAILED: load base BTF from %s: %s\n",
 			       obj->base_btf_path, strerror(-err));
-			return -1;
+			goto out_err;
 		}
 	}
 
 	btf = btf__parse_split(obj->btf_path ?: obj->path, base_btf);
 	err = libbpf_get_error(btf);
 	if (err) {
 		pr_err("FAILED: load BTF from %s: %s\n",
 			obj->btf_path ?: obj->path, strerror(-err));
-		goto out;
+		goto out_err;
 	}

tools/bpf/resolve_btfids/main.c static int symbols_resolve(struct object *obj) · line 566

Quoted 10 lines
tools/bpf/resolve_btfids/main.c +9 -0
 
+	obj->base_btf = base_btf;
+	obj->btf = btf;
+
+	return 0;
+
+out_err:
+	btf__free(base_btf);
+	btf__free(btf);
+	return err;

Can btf__free() safely handle ERR_PTR values here?

When btf__parse() fails on line 541, it returns ERR_PTR(-err), and libbpf_get_error() extracts the error code, but base_btf still holds the ERR_PTR value. The error path then calls btf__free(base_btf) where base_btf is ERR_PTR rather than NULL or a valid pointer.

Quoted 7 lines

The same issue occurs if btf__parse_split() fails on line 550, where btf contains ERR_PTR when btf__free(btf) is called.

In the old code at symbols_resolve(), if base_btf was ERR_PTR, the function would return -1 immediately without calling btf__free(), avoiding this issue. The new error handling with goto out_err changes this behavior.

That's a good observation. Interesting that AI noticed this only on third try, probably due to changes in prompts.

btf__free() checks for error values [1], so this is not an issue.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/bpf/btf.c?h=v6.18#n968

Quoted 24 lines
Quoted 5 lines
tools/bpf/resolve_btfids/main.c +4 -0
+}
+
+static int symbols_resolve(struct object *obj)
+{

tools/bpf/resolve_btfids/main.c static int symbols_resolve(struct object *obj) · line 580

Quoted 10 lines
tools/bpf/resolve_btfids/main.c +8 -0
+	int nr_typedefs = obj->nr_typedefs;
+	int nr_structs  = obj->nr_structs;
+	int nr_unions   = obj->nr_unions;
+	int nr_funcs    = obj->nr_funcs;
+	struct btf *btf = obj->btf;
+	int err, type_id;
+	__u32 nr_types;
+
 	err = -1;
 	nr_types = btf__type_cnt(btf);

[ ... ]

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19978081551

#6 lore raw

Re: [PATCH bpf-next v3 2/6] resolve_btfids: Factor out load_btf()

Quoted 71 lines · Ihor Solodrai
Quoted 62 lines · On 12/5/25 2:57 PM, [email protected] wrote:
Quoted 4 lines
tools/bpf/resolve_btfids/main.c +0 -0
index 164f0c941f04..b4caae1170dd 100644

[ ... ]

Quoted 37 lines
tools/bpf/resolve_btfids/main.c +3 -9
@@ -529,16 +532,10 @@ static int symbols_collect(struct object *obj)
 	return 0;
 }
 
-static int symbols_resolve(struct object *obj)
+static int load_btf(struct object *obj)
 {
-	int nr_typedefs = obj->nr_typedefs;
-	int nr_structs  = obj->nr_structs;
-	int nr_unions   = obj->nr_unions;
-	int nr_funcs    = obj->nr_funcs;
-	struct btf *base_btf = NULL;
-	int err, type_id;
-	struct btf *btf;
-	__u32 nr_types;
+	struct btf *base_btf = NULL, *btf = NULL;
+	int err;
 
 	if (obj->base_btf_path) {
 		base_btf = btf__parse(obj->base_btf_path, NULL);
tools/bpf/resolve_btfids/main.c +2 -2
 		err = libbpf_get_error(base_btf);
 		if (err) {
 			pr_err("FAILED: load base BTF from %s: %s\n",
 			       obj->base_btf_path, strerror(-err));
-			return -1;
+			goto out_err;
 		}
 	}
 
 	btf = btf__parse_split(obj->btf_path ?: obj->path, base_btf);
 	err = libbpf_get_error(btf);
 	if (err) {
 		pr_err("FAILED: load BTF from %s: %s\n",
 			obj->btf_path ?: obj->path, strerror(-err));
-		goto out;
+		goto out_err;
 	}

tools/bpf/resolve_btfids/main.c static int symbols_resolve(struct object *obj) · line 566

Quoted 10 lines
tools/bpf/resolve_btfids/main.c +9 -0
 
+	obj->base_btf = base_btf;
+	obj->btf = btf;
+
+	return 0;
+
+out_err:
+	btf__free(base_btf);
+	btf__free(btf);
+	return err;

Can btf__free() safely handle ERR_PTR values here?

When btf__parse() fails on line 541, it returns ERR_PTR(-err), and libbpf_get_error() extracts the error code, but base_btf still holds the ERR_PTR value. The error path then calls btf__free(base_btf) where base_btf is ERR_PTR rather than NULL or a valid pointer.

Quoted 7 lines

The same issue occurs if btf__parse_split() fails on line 550, where btf contains ERR_PTR when btf__free(btf) is called.

In the old code at symbols_resolve(), if base_btf was ERR_PTR, the function would return -1 immediately without calling btf__free(), avoiding this issue. The new error handling with goto out_err changes this behavior.

Quoted 7 lines

That's a good observation. Interesting that AI noticed this only on third try, probably due to changes in prompts.

btf__free() checks for error values [1], so this is not an issue.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/bpf/btf.c?h=v6.18#n968

Hmm, it should have read btf__free() to answer this question on its own.
I'll check a look.

-chris

author reviewer
#8 lore raw

PATCH v3 3/6 resolve_btfids: Introduce enum btf_id_kind

In reply to Ihor Solodrai

Instead of using multiple flags, make struct btf_id tagged with an enum value indicating its kind in the context of resolve_btfids.

---
 tools/bpf/resolve_btfids/main.c | 76 ++++++++++++++++++++++-----------
 1 file changed, 52 insertions(+), 24 deletions(-)
tools/bpf/resolve_btfids/main.c +52 -24
index b4caae1170dd..cb1e69eb0bd7 100644
@@ -98,6 +98,13 @@
 # error "Unknown machine endianness!"
 #endif
 
+enum btf_id_kind {
+	BTF_ID_KIND_NONE,
+	BTF_ID_KIND_SYM,
+	BTF_ID_KIND_SET,
+	BTF_ID_KIND_SET8
+};
+
 struct btf_id {
 	struct rb_node	 rb_node;
 	char		*name;
@@ -105,9 +112,8 @@ struct btf_id {
 		int	 id;
 		int	 cnt;
 	};
+	enum btf_id_kind kind;
 	int		 addr_cnt;
-	bool		 is_set;
-	bool		 is_set8;
 	Elf64_Addr	 addr[ADDR_CNT];
 };
 
@@ -197,8 +203,7 @@ static struct btf_id *btf_id__find(struct rb_root *root, const char *name)
 	return NULL;
 }
 
-static struct btf_id *
-btf_id__add(struct rb_root *root, char *name, bool unique)
+static struct btf_id *btf_id__add(struct rb_root *root, char *name, enum btf_id_kind kind)
 {
 	struct rb_node **p = &root->rb_node;
 	struct rb_node *parent = NULL;
@@ -213,14 +218,19 @@ btf_id__add(struct rb_root *root, char *name, bool unique)
 			p = &(*p)->rb_left;
 		else if (cmp > 0)
 			p = &(*p)->rb_right;
-		else
-			return unique ? NULL : id;
+		else if (kind == BTF_ID_KIND_SYM && id->kind == BTF_ID_KIND_SYM)
+			return id;
+		else {
+			pr_err("Unexpected duplicate symbol %s of kind %d\n", name, id->kind);
+			return NULL;
+		}
 	}
 
 	id = zalloc(sizeof(*id));
 	if (id) {
 		pr_debug("adding symbol %s\n", name);
 		id->name = name;
+		id->kind = kind;
 		rb_link_node(&id->rb_node, parent, p);
 		rb_insert_color(&id->rb_node, root);
 	}
@@ -260,22 +270,36 @@ static char *get_id(const char *prefix_end)
 	return id;
 }
 
-static struct btf_id *add_set(struct object *obj, char *name, bool is_set8)
+static struct btf_id *add_set(struct object *obj, char *name, enum btf_id_kind kind)
 {
+	int len = strlen(name);
+	int prefixlen;
+	char *id;
+
 	/*
 	 * __BTF_ID__set__name
 	 * name =    ^
 	 * id   =         ^
 	 */
-	char *id = name + (is_set8 ? sizeof(BTF_SET8 "__") : sizeof(BTF_SET "__")) - 1;
-	int len = strlen(name);
+	switch (kind) {
+	case BTF_ID_KIND_SET:
+		prefixlen = sizeof(BTF_SET "__") - 1;
+		break;
+	case BTF_ID_KIND_SET8:
+		prefixlen = sizeof(BTF_SET8 "__") - 1;
+		break;
+	default:
+		pr_err("Unexpected kind %d passed to %s() for symbol %s\n", kind, __func__, name);
+		return NULL;
+	}
 
+	id = name + prefixlen - 1;
 	if (id >= name + len) {
 		pr_err("FAILED to parse set name: %s\n", name);
 		return NULL;
 	}
 
-	return btf_id__add(&obj->sets, id, true);
+	return btf_id__add(&obj->sets, id, kind);
 }
 
 static struct btf_id *add_symbol(struct rb_root *root, char *name, size_t size)
@@ -288,7 +312,7 @@ static struct btf_id *add_symbol(struct rb_root *root, char *name, size_t size)
 		return NULL;
 	}
 
-	return btf_id__add(root, id, false);
+	return btf_id__add(root, id, BTF_ID_KIND_SYM);
 }
 
 /* Older libelf.h and glibc elf.h might not yet define the ELF compression types. */
@@ -491,28 +515,24 @@ static int symbols_collect(struct object *obj)
 			id = add_symbol(&obj->funcs, prefix, sizeof(BTF_FUNC) - 1);
 		/* set8 */
 		} else if (!strncmp(prefix, BTF_SET8, sizeof(BTF_SET8) - 1)) {
-			id = add_set(obj, prefix, true);
+			id = add_set(obj, prefix, BTF_ID_KIND_SET8);
 			/*
 			 * SET8 objects store list's count, which is encoded
 			 * in symbol's size, together with 'cnt' field hence
 			 * that - 1.
 			 */
-			if (id) {
+			if (id)
 				id->cnt = sym.st_size / sizeof(uint64_t) - 1;
-				id->is_set8 = true;
-			}
 		/* set */
 		} else if (!strncmp(prefix, BTF_SET, sizeof(BTF_SET) - 1)) {
-			id = add_set(obj, prefix, false);
+			id = add_set(obj, prefix, BTF_ID_KIND_SET);
 			/*
 			 * SET objects store list's count, which is encoded
 			 * in symbol's size, together with 'cnt' field hence
 			 * that - 1.
 			 */
-			if (id) {
+			if (id)
 				id->cnt = sym.st_size / sizeof(int) - 1;
-				id->is_set = true;
-			}
 		} else {
 			pr_err("FAILED unsupported prefix %s\n", prefix);
 			return -1;
@@ -643,7 +663,7 @@ static int id_patch(struct object *obj, struct btf_id *id)
 	int i;
 
 	/* For set, set8, id->id may be 0 */
-	if (!id->id && !id->is_set && !id->is_set8) {
+	if (!id->id && id->kind != BTF_ID_KIND_SET && id->kind != BTF_ID_KIND_SET8) {
 		pr_err("WARN: resolve_btfids: unresolved symbol %s\n", id->name);
 		warnings++;
 	}
@@ -696,6 +716,7 @@ static int sets_patch(struct object *obj)
 {
 	Elf_Data *data = obj->efile.idlist;
 	struct rb_node *next;
+	int cnt;
 
 	next = rb_first(&obj->sets);
 	while (next) {
@@ -715,11 +736,15 @@ static int sets_patch(struct object *obj)
 			return -1;
 		}
 
-		if (id->is_set) {
+		switch (id->kind) {
+		case BTF_ID_KIND_SET:
 			set = data->d_buf + off;
+			cnt = set->cnt;
 			qsort(set->ids, set->cnt, sizeof(set->ids[0]), cmp_id);
-		} else {
+			break;
+		case BTF_ID_KIND_SET8:
 			set8 = data->d_buf + off;
+			cnt = set8->cnt;
 			/*
 			 * Make sure id is at the beginning of the pairs
 			 * struct, otherwise the below qsort would not work.
@@ -744,10 +769,13 @@ static int sets_patch(struct object *obj)
 						bswap_32(set8->pairs[i].flags);
 				}
 			}
+			break;
+		default:
+			pr_err("Unexpected btf_id_kind %d for set '%s'\n", id->kind, id->name);
+			return -1;
 		}
 
-		pr_debug("sorting  addr %5lu: cnt %6d [%s]\n",
-			 off, id->is_set ? set->cnt : set8->cnt, id->name);
+		pr_debug("sorting  addr %5lu: cnt %6d [%s]\n", off, cnt, id->name);
 
 		next = rb_next(next);
 	}
Signature · 1 line
2.52.0
maintainer Acked-by on v3 3/6
#9 lore raw

Re: [PATCH bpf-next v3 3/6] resolve_btfids: Introduce enum btf_id_kind

Quoted 5 lines · Ihor Solodrai

Instead of using multiple flags, make struct btf_id tagged with an enum value indicating its kind in the context of resolve_btfids.

---

(But see a question below).

tools/bpf/resolve_btfids/main.c btf_id__add(struct rb_root *root, char *name, bool unique) · line 221

Quoted 7 lines
tools/bpf/resolve_btfids/main.c +1 -2
@@ -213,14 +218,19 @@ btf_id__add(struct rb_root *root, char *name, bool unique)
 			p = &(*p)->rb_left;
 		else if (cmp > 0)
 			p = &(*p)->rb_right;
-		else
-			return unique ? NULL : id;
+		else if (kind == BTF_ID_KIND_SYM && id->kind == BTF_ID_KIND_SYM)

Nit: I'd keep the 'unique' parameter alongside 'kind' and resolve this
condition on the function callsite.

Quoted 5 lines
tools/bpf/resolve_btfids/main.c +5 -0
+			return id;
+		else {
+			pr_err("Unexpected duplicate symbol %s of kind %d\n", name, id->kind);
+			return NULL;
+		}

[...]

Quoted 17 lines
tools/bpf/resolve_btfids/main.c +2 -4
@@ -491,28 +515,24 @@ static int symbols_collect(struct object *obj)
 			id = add_symbol(&obj->funcs, prefix, sizeof(BTF_FUNC) - 1);
 		/* set8 */
 		} else if (!strncmp(prefix, BTF_SET8, sizeof(BTF_SET8) - 1)) {
-			id = add_set(obj, prefix, true);
+			id = add_set(obj, prefix, BTF_ID_KIND_SET8);
 			/*
 			 * SET8 objects store list's count, which is encoded
 			 * in symbol's size, together with 'cnt' field hence
 			 * that - 1.
 			 */
-			if (id) {
+			if (id)
 				id->cnt = sym.st_size / sizeof(uint64_t) - 1;
-				id->is_set8 = true;
-			}
 		/* set */

tools/bpf/resolve_btfids/main.c static int symbols_collect(struct object *obj) · line 534

Quoted 10 lines
tools/bpf/resolve_btfids/main.c +2 -2
 		} else if (!strncmp(prefix, BTF_SET, sizeof(BTF_SET) - 1)) {
-			id = add_set(obj, prefix, false);
+			id = add_set(obj, prefix, BTF_ID_KIND_SET);
 			/*
 			 * SET objects store list's count, which is encoded
 			 * in symbol's size, together with 'cnt' field hence
 			 * that - 1.
 			 */
-			if (id) {
+			if (id)

Current patch is not a culprit, but shouldn't resolve_btfids fail if id cannot be added? (here and in a hunk above).

Quoted 6 lines
tools/bpf/resolve_btfids/main.c +0 -2
 				id->cnt = sym.st_size / sizeof(int) - 1;
-				id->is_set = true;
-			}
 		} else {
 			pr_err("FAILED unsupported prefix %s\n", prefix);
 			return -1;

[...]

author reviewer
#10 lore raw

Re: [PATCH bpf-next v3 3/6] resolve_btfids: Introduce enum btf_id_kind

Quoted 11 lines · Eduard Zingerman
Quoted 5 lines · Ihor Solodrai

Instead of using multiple flags, make struct btf_id tagged with an enum value indicating its kind in the context of resolve_btfids.

---

(But see a question below).

Quoted 10 lines

tools/bpf/resolve_btfids/main.c btf_id__add(struct rb_root *root, char *name, bool unique) · line 221

Quoted 7 lines
tools/bpf/resolve_btfids/main.c +1 -2
@@ -213,14 +218,19 @@ btf_id__add(struct rb_root *root, char *name, bool unique)
 			p = &(*p)->rb_left;
 		else if (cmp > 0)
 			p = &(*p)->rb_right;
-		else
-			return unique ? NULL : id;
+		else if (kind == BTF_ID_KIND_SYM && id->kind == BTF_ID_KIND_SYM)

Nit: I'd keep the 'unique' parameter alongside 'kind' and resolve this
condition on the function callsite.

I don't like the boolean args, they're always opaque on the callsite.

We want to allow duplicates for _KIND_SYM and forbid for other kinds. Since we are passing the kind from outside, I think it makes sense to check for this inside the function. It makes the usage simpler.

Quoted 31 lines
Quoted 6 lines
tools/bpf/resolve_btfids/main.c +5 -0
+			return id;
+		else {
+			pr_err("Unexpected duplicate symbol %s of kind %d\n", name, id->kind);
+			return NULL;
+		}

[...]

Quoted 22 lines
tools/bpf/resolve_btfids/main.c +3 -5
@@ -491,28 +515,24 @@ static int symbols_collect(struct object *obj)
 			id = add_symbol(&obj->funcs, prefix, sizeof(BTF_FUNC) - 1);
 		/* set8 */
 		} else if (!strncmp(prefix, BTF_SET8, sizeof(BTF_SET8) - 1)) {
-			id = add_set(obj, prefix, true);
+			id = add_set(obj, prefix, BTF_ID_KIND_SET8);
 			/*
 			 * SET8 objects store list's count, which is encoded
 			 * in symbol's size, together with 'cnt' field hence
 			 * that - 1.
 			 */
-			if (id) {
+			if (id)
 				id->cnt = sym.st_size / sizeof(uint64_t) - 1;
-				id->is_set8 = true;
-			}
 		/* set */
 		} else if (!strncmp(prefix, BTF_SET, sizeof(BTF_SET) - 1)) {
-			id = add_set(obj, prefix, false);
+			id = add_set(obj, prefix, BTF_ID_KIND_SET);
 			/*
 			 * SET objects store list's count, which is encoded
Quoted 8 lines

tools/bpf/resolve_btfids/main.c static int symbols_collect(struct object *obj) · line 524

Quoted 5 lines
tools/bpf/resolve_btfids/main.c +1 -1
 			 * in symbol's size, together with 'cnt' field hence
 			 * that - 1.
 			 */
-			if (id) {
+			if (id)

Current patch is not a culprit, but shouldn't resolve_btfids fail if id cannot be added? (here and in a hunk above).

By the existing design, resolve_btfids generally fails if CONFIG_WERROR is set and warnings > 0.

And in this particular place it would fails with -ENOMEM a bit below:

       [...]
		} else if (!strncmp(prefix, BTF_SET, sizeof(BTF_SET) - 1)) {
			id = add_set(obj, prefix, BTF_ID_KIND_SET);
			/*
			 * SET objects store list's count, which is encoded
			 * in symbol's size, together with 'cnt' field hence
			 * that - 1.
			 */
			if (id)
				id->cnt = sym.st_size / sizeof(int) - 1;
		} else {
			pr_err("FAILED unsupported prefix %s\n", prefix);
			return -1;
		}

/* --> */ if (!id)

			return -ENOMEM;

So I think an error code change may be appropriate, and that's about it.

Quoted 10 lines
Quoted 7 lines
tools/bpf/resolve_btfids/main.c +0 -2
 				id->cnt = sym.st_size / sizeof(int) - 1;
-				id->is_set = true;
-			}
 		} else {
 			pr_err("FAILED unsupported prefix %s\n", prefix);
 			return -1;

[...]

maintainer
#11 lore raw

Re: [PATCH bpf-next v3 3/6] resolve_btfids: Introduce enum btf_id_kind

Quoted 23 lines · Ihor Solodrai
Quoted 11 lines · Eduard Zingerman
Quoted 5 lines · Ihor Solodrai

Instead of using multiple flags, make struct btf_id tagged with an enum value indicating its kind in the context of resolve_btfids.

---

(But see a question below).

Quoted 10 lines

tools/bpf/resolve_btfids/main.c btf_id__add(struct rb_root *root, char *name, bool unique) · line 221

Quoted 7 lines
tools/bpf/resolve_btfids/main.c +1 -2
@@ -213,14 +218,19 @@ btf_id__add(struct rb_root *root, char *name, bool unique)
 			p = &(*p)->rb_left;
 		else if (cmp > 0)
 			p = &(*p)->rb_right;
-		else
-			return unique ? NULL : id;
+		else if (kind == BTF_ID_KIND_SYM && id->kind == BTF_ID_KIND_SYM)

Nit: I'd keep the 'unique' parameter alongside 'kind' and resolve this
condition on the function callsite.

Quoted 5 lines

I don't like the boolean args, they're always opaque on the callsite.

We want to allow duplicates for _KIND_SYM and forbid for other kinds. Since we are passing the kind from outside, I think it makes sense to check for this inside the function. It makes the usage simpler.

On the contrary, the callsite knows exactly what it wants:
unique or non-unique entries. Here you need additional logic to figure out the intent.

Arguably the uniqueness is associated not with entry type, but with a particular tree the entry is added to. And that is a property of the callsite.

Quoted 59 lines
Quoted 30 lines
Quoted 5 lines
tools/bpf/resolve_btfids/main.c +5 -0
+			return id;
+		else {
+			pr_err("Unexpected duplicate symbol %s of kind %d\n", name, id->kind);
+			return NULL;
+		}

[...]

Quoted 22 lines
tools/bpf/resolve_btfids/main.c +3 -5
@@ -491,28 +515,24 @@ static int symbols_collect(struct object *obj)
 			id = add_symbol(&obj->funcs, prefix, sizeof(BTF_FUNC) - 1);
 		/* set8 */
 		} else if (!strncmp(prefix, BTF_SET8, sizeof(BTF_SET8) - 1)) {
-			id = add_set(obj, prefix, true);
+			id = add_set(obj, prefix, BTF_ID_KIND_SET8);
 			/*
 			 * SET8 objects store list's count, which is encoded
 			 * in symbol's size, together with 'cnt' field hence
 			 * that - 1.
 			 */
-			if (id) {
+			if (id)
 				id->cnt = sym.st_size / sizeof(uint64_t) - 1;
-				id->is_set8 = true;
-			}
 		/* set */
 		} else if (!strncmp(prefix, BTF_SET, sizeof(BTF_SET) - 1)) {
-			id = add_set(obj, prefix, false);
+			id = add_set(obj, prefix, BTF_ID_KIND_SET);
 			/*
 			 * SET objects store list's count, which is encoded
Quoted 8 lines

tools/bpf/resolve_btfids/main.c static int symbols_collect(struct object *obj) · line 524

Quoted 5 lines
tools/bpf/resolve_btfids/main.c +1 -1
 			 * in symbol's size, together with 'cnt' field hence
 			 * that - 1.
 			 */
-			if (id) {
+			if (id)

Current patch is not a culprit, but shouldn't resolve_btfids fail if id cannot be added? (here and in a hunk above).

By the existing design, resolve_btfids generally fails if CONFIG_WERROR is set and warnings > 0.

And in this particular place it would fails with -ENOMEM a bit below:

       [...]
		} else if (!strncmp(prefix, BTF_SET, sizeof(BTF_SET) - 1)) {
			id = add_set(obj, prefix, BTF_ID_KIND_SET);
			/*
			 * SET objects store list's count, which is encoded
			 * in symbol's size, together with 'cnt' field hence
			 * that - 1.
			 */
			if (id)
				id->cnt = sym.st_size / sizeof(int) - 1;
		} else {
			pr_err("FAILED unsupported prefix %s\n", prefix);
			return -1;
		}
Quoted 4 lines

/* --> */ if (!id)

			return -ENOMEM;

So I think an error code change may be appropriate, and that's about it.

Oh, ok, sorry, didn't notice that.

Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
tools/bpf/resolve_btfids/main.c +0 -2
 				id->cnt = sym.st_size / sizeof(int) - 1;
-				id->is_set = true;
-			}
 		} else {
 			pr_err("FAILED unsupported prefix %s\n", prefix);
 			return -1;

[...]

author reviewer
#12 lore raw

Re: [PATCH bpf-next v3 3/6] resolve_btfids: Introduce enum btf_id_kind

Quoted 30 lines · Eduard Zingerman
Quoted 23 lines · Ihor Solodrai
Quoted 11 lines · Eduard Zingerman
Quoted 5 lines · Ihor Solodrai

Instead of using multiple flags, make struct btf_id tagged with an enum value indicating its kind in the context of resolve_btfids.

---

(But see a question below).

Quoted 10 lines

tools/bpf/resolve_btfids/main.c btf_id__add(struct rb_root *root, char *name, bool unique) · line 221

Quoted 7 lines
tools/bpf/resolve_btfids/main.c +1 -2
@@ -213,14 +218,19 @@ btf_id__add(struct rb_root *root, char *name, bool unique)
 			p = &(*p)->rb_left;
 		else if (cmp > 0)
 			p = &(*p)->rb_right;
-		else
-			return unique ? NULL : id;
+		else if (kind == BTF_ID_KIND_SYM && id->kind == BTF_ID_KIND_SYM)

Nit: I'd keep the 'unique' parameter alongside 'kind' and resolve this
condition on the function callsite.

Quoted 5 lines

I don't like the boolean args, they're always opaque on the callsite.

We want to allow duplicates for _KIND_SYM and forbid for other kinds. Since we are passing the kind from outside, I think it makes sense to check for this inside the function. It makes the usage simpler.

Quoted 7 lines

On the contrary, the callsite knows exactly what it wants:
unique or non-unique entries. Here you need additional logic to figure out the intent.

Arguably the uniqueness is associated not with entry type, but with a particular tree the entry is added to. And that is a property of the callsite.

You're right that the uniqueness is associated with a tree. This means we could even check the kind of the root...

I'm thinking maybe it's cleaner to have btf_id__add() and btf_id__add_unique(). It can even be a wrapper around btf_id__add() with a boolean. wdyt?

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

[...]

maintainer
#13 lore raw

Re: [PATCH bpf-next v3 3/6] resolve_btfids: Introduce enum btf_id_kind

Quoted 39 lines · Ihor Solodrai
Quoted 30 lines · Eduard Zingerman
Quoted 23 lines · Ihor Solodrai
Quoted 11 lines · Eduard Zingerman
Quoted 5 lines · Ihor Solodrai

Instead of using multiple flags, make struct btf_id tagged with an enum value indicating its kind in the context of resolve_btfids.

---

(But see a question below).

Quoted 10 lines

tools/bpf/resolve_btfids/main.c btf_id__add(struct rb_root *root, char *name, bool unique) · line 221

Quoted 7 lines
tools/bpf/resolve_btfids/main.c +1 -2
@@ -213,14 +218,19 @@ btf_id__add(struct rb_root *root, char *name, bool unique)
 			p = &(*p)->rb_left;
 		else if (cmp > 0)
 			p = &(*p)->rb_right;
-		else
-			return unique ? NULL : id;
+		else if (kind == BTF_ID_KIND_SYM && id->kind == BTF_ID_KIND_SYM)

Nit: I'd keep the 'unique' parameter alongside 'kind' and resolve this
condition on the function callsite.

Quoted 5 lines

I don't like the boolean args, they're always opaque on the callsite.

We want to allow duplicates for _KIND_SYM and forbid for other kinds. Since we are passing the kind from outside, I think it makes sense to check for this inside the function. It makes the usage simpler.

Quoted 7 lines

On the contrary, the callsite knows exactly what it wants:
unique or non-unique entries. Here you need additional logic to figure out the intent.

Arguably the uniqueness is associated not with entry type, but with a particular tree the entry is added to. And that is a property of the callsite.

Quoted 6 lines

You're right that the uniqueness is associated with a tree. This means we could even check the kind of the root...

I'm thinking maybe it's cleaner to have btf_id__add() and btf_id__add_unique(). It can even be a wrapper around btf_id__add() with a boolean. wdyt?

Well, sure, that would be a bit cleaner on the callsite. Up to you, given the number of the callsites I wouldn't bother.

author reviewer
#14 lore raw

PATCH v3 4/6 lib/Kconfig.debug: Set the minimum required pahole version to v1.22

In reply to Ihor Solodrai

Subsequent patches in the series change vmlinux linking scripts to unconditionally pass --btf_encode_detached to pahole, which was introduced in v1.22 [1][2].

This change allows to remove PAHOLE_HAS_SPLIT_BTF Kconfig option and other checks of older pahole versions.

[1] https://github.com/acmel/dwarves/releases/tag/v1.22
[2] https://lore.kernel.org/bpf/[email protected]/

---
 lib/Kconfig.debug         | 13 ++++---------
 scripts/Makefile.btf      |  9 +--------
 tools/sched_ext/README.md |  1 -
 3 files changed, 5 insertions(+), 18 deletions(-)
lib/Kconfig.debug +4 -9
index 742b23ef0d8b..3abf3ae554b6 100644
@@ -389,18 +389,13 @@ config DEBUG_INFO_BTF
 	depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
 	depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
 	depends on BPF_SYSCALL
-	depends on PAHOLE_VERSION >= 116
-	depends on DEBUG_INFO_DWARF4 || PAHOLE_VERSION >= 121
+	depends on PAHOLE_VERSION >= 122
 	# pahole uses elfutils, which does not have support for Hexagon relocations
 	depends on !HEXAGON
 	help
 	  Generate deduplicated BTF type information from DWARF debug info.
-	  Turning this on requires pahole v1.16 or later (v1.21 or later to
-	  support DWARF 5), which will convert DWARF type info into equivalent
-	  deduplicated BTF type info.
-
-config PAHOLE_HAS_SPLIT_BTF
-	def_bool PAHOLE_VERSION >= 119
+	  Turning this on requires pahole v1.22 or later, which will convert
+	  DWARF type info into equivalent deduplicated BTF type info.
 
 config PAHOLE_HAS_BTF_TAG
 	def_bool PAHOLE_VERSION >= 123
@@ -422,7 +417,7 @@ config PAHOLE_HAS_LANG_EXCLUDE
 config DEBUG_INFO_BTF_MODULES
 	bool "Generate BTF type information for kernel modules"
 	default y
-	depends on DEBUG_INFO_BTF && MODULES && PAHOLE_HAS_SPLIT_BTF
+	depends on DEBUG_INFO_BTF && MODULES
 	help
 	  Generate compact split BTF type information for kernel modules.
 
scripts/Makefile.btf +1 -8
index db76335dd917..7c1cd6c2ff75 100644
@@ -7,14 +7,7 @@ JOBS := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))
 
 ifeq ($(call test-le, $(pahole-ver), 125),y)
 
-# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
-ifeq ($(call test-le, $(pahole-ver), 121),y)
-pahole-flags-$(call test-ge, $(pahole-ver), 118)	+= --skip_encoding_btf_vars
-endif
-
-pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
-
-pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j$(JOBS)
+pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= --btf_gen_floats -j$(JOBS)
 
 pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
 
tools/sched_ext/README.md +0 -1
index 16a42e4060f6..56a9d1557ac4 100644
@@ -65,7 +65,6 @@ It's also recommended that you also include the following Kconfig options:
 ```
 CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_BPF_JIT_DEFAULT_ON=y
-CONFIG_PAHOLE_HAS_SPLIT_BTF=y
 CONFIG_PAHOLE_HAS_BTF_TAG=y
 ```
 
Signature · 1 line
2.52.0
#15 lore raw

Re: [PATCH bpf-next v3 4/6] lib/Kconfig.debug: Set the minimum required pahole version to v1.22

The PAHOLE_HAS_SPLIT_BTF config option was removed here.
Quoted 4 lines
lib/Kconfig.debug +0 -0
index 742b23ef0d8b..3abf3ae554b6 100644

[ ... ]

lib/Kconfig.debug config DEBUG_INFO_BTF · line 403

Quoted 2 lines
lib/Kconfig.debug +0 -2
-config PAHOLE_HAS_SPLIT_BTF
-	def_bool PAHOLE_VERSION >= 119

The PAHOLE_HAS_SPLIT_BTF config option was removed here.

[ ... ]

tools/sched_ext/README.md It's also recommended that you also include the following Kconfig options: · line 69

Quoted 11 lines
tools/sched_ext/README.md +0 -1
index 16a42e4060f6..56a9d1557ac4 100644
@@ -65,7 +65,6 @@ It's also recommended that you also include the following Kconfig options:
 ```
 CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_BPF_JIT_DEFAULT_ON=y
-CONFIG_PAHOLE_HAS_SPLIT_BTF=y
 CONFIG_PAHOLE_HAS_BTF_TAG=y
 ```

The PAHOLE_HAS_SPLIT_BTF reference was removed from tools/sched_ext/README.md, but Documentation/scheduler/sched-ext.rst still contains this config option at line 46. Should that documentation file also be updated to remove this obsolete config option?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19978081551

author reviewer
#16 lore raw

Re: [PATCH bpf-next v3 4/6] lib/Kconfig.debug: Set the minimum required pahole version to v1.22

Quoted 14 lines · On 12/5/25 2:49 PM, [email protected] wrote:
Quoted 4 lines
lib/Kconfig.debug +0 -0
index 742b23ef0d8b..3abf3ae554b6 100644

[ ... ]

lib/Kconfig.debug config DEBUG_INFO_BTF · line 403

Quoted 2 lines
lib/Kconfig.debug +0 -2
-config PAHOLE_HAS_SPLIT_BTF
-	def_bool PAHOLE_VERSION >= 119

The PAHOLE_HAS_SPLIT_BTF config option was removed here.

[ ... ]

Quoted 16 lines

tools/sched_ext/README.md It's also recommended that you also include the following Kconfig options: · line 69

Quoted 11 lines
tools/sched_ext/README.md +0 -1
index 16a42e4060f6..56a9d1557ac4 100644
@@ -65,7 +65,6 @@ It's also recommended that you also include the following Kconfig options:
 ```
 CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_BPF_JIT_DEFAULT_ON=y
-CONFIG_PAHOLE_HAS_SPLIT_BTF=y
 CONFIG_PAHOLE_HAS_BTF_TAG=y
 ```

The PAHOLE_HAS_SPLIT_BTF reference was removed from tools/sched_ext/README.md, but Documentation/scheduler/sched-ext.rst still contains this config option at line 46. Should that documentation file also be updated to remove this obsolete config option?

Yes. I thought I removed it, but the change got lost in translation.

Quoted 7 lines

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/19978081551

#17 lore raw

Re: [PATCH bpf-next v3 4/6] lib/Kconfig.debug: Set the minimum required pahole version to v1.22

In reply to Ihor Solodrai
Quoted 60 lines · Ihor Solodrai

Subsequent patches in the series change vmlinux linking scripts to unconditionally pass --btf_encode_detached to pahole, which was introduced in v1.22 [1][2].

This change allows to remove PAHOLE_HAS_SPLIT_BTF Kconfig option and other checks of older pahole versions.

[1] https://github.com/acmel/dwarves/releases/tag/v1.22
[2] https://lore.kernel.org/bpf/[email protected]/

---
 lib/Kconfig.debug         | 13 ++++---------
 scripts/Makefile.btf      |  9 +--------
 tools/sched_ext/README.md |  1 -
 3 files changed, 5 insertions(+), 18 deletions(-)
lib/Kconfig.debug +4 -9
index 742b23ef0d8b..3abf3ae554b6 100644
@@ -389,18 +389,13 @@ config DEBUG_INFO_BTF
        depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
        depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
        depends on BPF_SYSCALL
-       depends on PAHOLE_VERSION >= 116
-       depends on DEBUG_INFO_DWARF4 || PAHOLE_VERSION >= 121
+       depends on PAHOLE_VERSION >= 122
        # pahole uses elfutils, which does not have support for Hexagon relocations
        depends on !HEXAGON
        help
          Generate deduplicated BTF type information from DWARF debug info.
-         Turning this on requires pahole v1.16 or later (v1.21 or later to
-         support DWARF 5), which will convert DWARF type info into equivalent
-         deduplicated BTF type info.
-
-config PAHOLE_HAS_SPLIT_BTF
-       def_bool PAHOLE_VERSION >= 119
+         Turning this on requires pahole v1.22 or later, which will convert
+         DWARF type info into equivalent deduplicated BTF type info.
 
 config PAHOLE_HAS_BTF_TAG
        def_bool PAHOLE_VERSION >= 123
@@ -422,7 +417,7 @@ config PAHOLE_HAS_LANG_EXCLUDE
 config DEBUG_INFO_BTF_MODULES
        bool "Generate BTF type information for kernel modules"
        default y
-       depends on DEBUG_INFO_BTF && MODULES && PAHOLE_HAS_SPLIT_BTF
+       depends on DEBUG_INFO_BTF && MODULES
        help
          Generate compact split BTF type information for kernel modules.
 
scripts/Makefile.btf +0 -0
index db76335dd917..7c1cd6c2ff75 100644
@@ -7,14 +7,7 @@ JOBS := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))
 
 ifeq ($(call test-le, $(pahole-ver), 125),y)

scripts/Makefile.btf JOBS := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS))) · line 10

Quoted 10 lines
scripts/Makefile.btf +1 -8
 
-# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
-ifeq ($(call test-le, $(pahole-ver), 121),y)
-pahole-flags-$(call test-ge, $(pahole-ver), 118)       += --skip_encoding_btf_vars
-endif
-
-pahole-flags-$(call test-ge, $(pahole-ver), 121)       += --btf_gen_floats
-
-pahole-flags-$(call test-ge, $(pahole-ver), 122)       += -j$(JOBS)
+pahole-flags-$(call test-ge, $(pahole-ver), 122)       += --btf_gen_floats -j$(JOBS)

this should be unconditional given we expect at least 1.22, no?

Quoted 18 lines
scripts/Makefile.btf +0 -0
 pahole-flags-$(call test-ge, $(pahole-ver), 125)       += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
 
tools/sched_ext/README.md +0 -2
index 16a42e4060f6..56a9d1557ac4 100644
@@ -65,7 +65,6 @@ It's also recommended that you also include the following Kconfig options:
 ```
 CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_BPF_JIT_DEFAULT_ON=y
-CONFIG_PAHOLE_HAS_SPLIT_BTF=y
 CONFIG_PAHOLE_HAS_BTF_TAG=y
 ```
 
--

2.52.0

author reviewer
#18 lore raw

Re: [PATCH bpf-next v3 4/6] lib/Kconfig.debug: Set the minimum required pahole version to v1.22

Quoted 62 lines · Andrii Nakryiko
Quoted 61 lines · Ihor Solodrai

Subsequent patches in the series change vmlinux linking scripts to unconditionally pass --btf_encode_detached to pahole, which was introduced in v1.22 [1][2].

This change allows to remove PAHOLE_HAS_SPLIT_BTF Kconfig option and other checks of older pahole versions.

[1] https://github.com/acmel/dwarves/releases/tag/v1.22
[2] https://lore.kernel.org/bpf/[email protected]/

---
 lib/Kconfig.debug         | 13 ++++---------
 scripts/Makefile.btf      |  9 +--------
 tools/sched_ext/README.md |  1 -
 3 files changed, 5 insertions(+), 18 deletions(-)
lib/Kconfig.debug +4 -9
index 742b23ef0d8b..3abf3ae554b6 100644
@@ -389,18 +389,13 @@ config DEBUG_INFO_BTF
        depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
        depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
        depends on BPF_SYSCALL
-       depends on PAHOLE_VERSION >= 116
-       depends on DEBUG_INFO_DWARF4 || PAHOLE_VERSION >= 121
+       depends on PAHOLE_VERSION >= 122
        # pahole uses elfutils, which does not have support for Hexagon relocations
        depends on !HEXAGON
        help
          Generate deduplicated BTF type information from DWARF debug info.
-         Turning this on requires pahole v1.16 or later (v1.21 or later to
-         support DWARF 5), which will convert DWARF type info into equivalent
-         deduplicated BTF type info.
-
-config PAHOLE_HAS_SPLIT_BTF
-       def_bool PAHOLE_VERSION >= 119
+         Turning this on requires pahole v1.22 or later, which will convert
+         DWARF type info into equivalent deduplicated BTF type info.
 
 config PAHOLE_HAS_BTF_TAG
        def_bool PAHOLE_VERSION >= 123
@@ -422,7 +417,7 @@ config PAHOLE_HAS_LANG_EXCLUDE
 config DEBUG_INFO_BTF_MODULES
        bool "Generate BTF type information for kernel modules"
        default y
-       depends on DEBUG_INFO_BTF && MODULES && PAHOLE_HAS_SPLIT_BTF
+       depends on DEBUG_INFO_BTF && MODULES
        help
          Generate compact split BTF type information for kernel modules.
 
scripts/Makefile.btf +0 -0
index db76335dd917..7c1cd6c2ff75 100644
@@ -7,14 +7,7 @@ JOBS := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))
 
 ifeq ($(call test-le, $(pahole-ver), 125),y)
 
Quoted 11 lines

scripts/Makefile.btf JOBS := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS))) · line 10

Quoted 9 lines
scripts/Makefile.btf +1 -8
-# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
-ifeq ($(call test-le, $(pahole-ver), 121),y)
-pahole-flags-$(call test-ge, $(pahole-ver), 118)       += --skip_encoding_btf_vars
-endif
-
-pahole-flags-$(call test-ge, $(pahole-ver), 121)       += --btf_gen_floats
-
-pahole-flags-$(call test-ge, $(pahole-ver), 122)       += -j$(JOBS)
+pahole-flags-$(call test-ge, $(pahole-ver), 122)       += --btf_gen_floats -j$(JOBS)

this should be unconditional given we expect at least 1.22, no?

Yes, it can be unconditional, but still under if ver < 125.

Quoted 19 lines
Quoted 19 lines
scripts/Makefile.btf +0 -0
 pahole-flags-$(call test-ge, $(pahole-ver), 125)       += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
 
tools/sched_ext/README.md +0 -2
index 16a42e4060f6..56a9d1557ac4 100644
@@ -65,7 +65,6 @@ It's also recommended that you also include the following Kconfig options:
 ```
 CONFIG_BPF_JIT_ALWAYS_ON=y
 CONFIG_BPF_JIT_DEFAULT_ON=y
-CONFIG_PAHOLE_HAS_SPLIT_BTF=y
 CONFIG_PAHOLE_HAS_BTF_TAG=y
 ```
 
--

2.52.0

maintainer Acked-by on v3 4/6
#19 lore raw

Re: [PATCH bpf-next v3 4/6] lib/Kconfig.debug: Set the minimum required pahole version to v1.22

In reply to Ihor Solodrai
Quoted 4 lines · Ihor Solodrai

Subsequent patches in the series change vmlinux linking scripts to unconditionally pass --btf_encode_detached to pahole, which was introduced in v1.22 [1][2].

Quoted 8 lines

This change allows to remove PAHOLE_HAS_SPLIT_BTF Kconfig option and other checks of older pahole versions.

[1] https://github.com/acmel/dwarves/releases/tag/v1.22
[2] https://lore.kernel.org/bpf/[email protected]/

---

[...]

#20 lore raw

Re: [PATCH bpf-next v3 4/6] lib/Kconfig.debug: Set the minimum required pahole version to v1.22

In reply to Ihor Solodrai
Quoted 57 lines · Ihor Solodrai

Subsequent patches in the series change vmlinux linking scripts to unconditionally pass --btf_encode_detached to pahole, which was introduced in v1.22 [1][2].

This change allows to remove PAHOLE_HAS_SPLIT_BTF Kconfig option and other checks of older pahole versions.

[1] https://github.com/acmel/dwarves/releases/tag/v1.22
[2] https://lore.kernel.org/bpf/[email protected]/

---
 lib/Kconfig.debug         | 13 ++++---------
 scripts/Makefile.btf      |  9 +--------
 tools/sched_ext/README.md |  1 -
 3 files changed, 5 insertions(+), 18 deletions(-)
lib/Kconfig.debug +4 -9
index 742b23ef0d8b..3abf3ae554b6 100644
@@ -389,18 +389,13 @@ config DEBUG_INFO_BTF
 	depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
 	depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
 	depends on BPF_SYSCALL
-	depends on PAHOLE_VERSION >= 116
-	depends on DEBUG_INFO_DWARF4 || PAHOLE_VERSION >= 121
+	depends on PAHOLE_VERSION >= 122
 	# pahole uses elfutils, which does not have support for Hexagon relocations
 	depends on !HEXAGON
 	help
 	  Generate deduplicated BTF type information from DWARF debug info.
-	  Turning this on requires pahole v1.16 or later (v1.21 or later to
-	  support DWARF 5), which will convert DWARF type info into equivalent
-	  deduplicated BTF type info.
-
-config PAHOLE_HAS_SPLIT_BTF
-	def_bool PAHOLE_VERSION >= 119
+	  Turning this on requires pahole v1.22 or later, which will convert
+	  DWARF type info into equivalent deduplicated BTF type info.
 
 config PAHOLE_HAS_BTF_TAG
 	def_bool PAHOLE_VERSION >= 123
@@ -422,7 +417,7 @@ config PAHOLE_HAS_LANG_EXCLUDE
 config DEBUG_INFO_BTF_MODULES
 	bool "Generate BTF type information for kernel modules"
 	default y
-	depends on DEBUG_INFO_BTF && MODULES && PAHOLE_HAS_SPLIT_BTF
+	depends on DEBUG_INFO_BTF && MODULES
 	help
 	  Generate compact split BTF type information for kernel modules.
 
scripts/Makefile.btf +0 -0
index db76335dd917..7c1cd6c2ff75 100644

scripts/Makefile.btf JOBS := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS))) · line 7

Quoted 1 line
scripts/Makefile.btf +0 -0
@@ -7,14 +7,7 @@ JOBS := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))
 

hi Ihor, a small suggestion here, and it is orthogonal to what you're doing here, so just for consideration if you're planning a v4 since you're touching this file.

We've had problems in the past because we get pahole version from .config in Makefile.btf

pahole-ver := $(CONFIG_PAHOLE_VERSION)

and it can be outdated.

Specifically the problem is that if "make oldconfig" is not run after updating pahole we don't get the actual pahole version during builds and options can be missing. See [1] for an example, but perhaps we should do

pahole-ver := $(shell $(srctree)/scripts/pahole-version.sh)

in Makefile.btf to ensure the value reflects latest pahole and that then determines which options we use? Andrii suggested an approach like CC_VERSION_TEXT might be worth pursuing; AFAICT that recomputes the CC_VERSION and warns the user if there is a version difference. Given that the CONFIG pahole version requirements are all pretty modest - it might simply be enough to recompute it in Makefile.btf and perhaps ensure it's not less than CONFIG_PAHOLE_VERSION. Just a thought anyway. Thanks!

Alan

[1] https://lore.kernel.org/bpf/CAEf4BzYi1xX3p_bY3j9dEuPvtCW3H7z=p2vdn-2GY0OOenxQAg@mail.gmail.com/

author reviewer
#21 lore raw

Re: [PATCH bpf-next v3 4/6] lib/Kconfig.debug: Set the minimum required pahole version to v1.22

Quoted 53 lines · Alan Maguire
Quoted 52 lines · Ihor Solodrai

Subsequent patches in the series change vmlinux linking scripts to unconditionally pass --btf_encode_detached to pahole, which was introduced in v1.22 [1][2].

This change allows to remove PAHOLE_HAS_SPLIT_BTF Kconfig option and other checks of older pahole versions.

[1] https://github.com/acmel/dwarves/releases/tag/v1.22
[2] https://lore.kernel.org/bpf/[email protected]/

---
 lib/Kconfig.debug         | 13 ++++---------
 scripts/Makefile.btf      |  9 +--------
 tools/sched_ext/README.md |  1 -
 3 files changed, 5 insertions(+), 18 deletions(-)
lib/Kconfig.debug +4 -9
index 742b23ef0d8b..3abf3ae554b6 100644
@@ -389,18 +389,13 @@ config DEBUG_INFO_BTF
 	depends on !DEBUG_INFO_SPLIT && !DEBUG_INFO_REDUCED
 	depends on !GCC_PLUGIN_RANDSTRUCT || COMPILE_TEST
 	depends on BPF_SYSCALL
-	depends on PAHOLE_VERSION >= 116
-	depends on DEBUG_INFO_DWARF4 || PAHOLE_VERSION >= 121
+	depends on PAHOLE_VERSION >= 122
 	# pahole uses elfutils, which does not have support for Hexagon relocations
 	depends on !HEXAGON
 	help
 	  Generate deduplicated BTF type information from DWARF debug info.
-	  Turning this on requires pahole v1.16 or later (v1.21 or later to
-	  support DWARF 5), which will convert DWARF type info into equivalent
-	  deduplicated BTF type info.
-
-config PAHOLE_HAS_SPLIT_BTF
-	def_bool PAHOLE_VERSION >= 119
+	  Turning this on requires pahole v1.22 or later, which will convert
+	  DWARF type info into equivalent deduplicated BTF type info.
 
 config PAHOLE_HAS_BTF_TAG
 	def_bool PAHOLE_VERSION >= 123
@@ -422,7 +417,7 @@ config PAHOLE_HAS_LANG_EXCLUDE
 config DEBUG_INFO_BTF_MODULES
 	bool "Generate BTF type information for kernel modules"
 	default y
-	depends on DEBUG_INFO_BTF && MODULES && PAHOLE_HAS_SPLIT_BTF
+	depends on DEBUG_INFO_BTF && MODULES
 	help
 	  Generate compact split BTF type information for kernel modules.
 
Quoted 10 lines

scripts/Makefile.btf JOBS := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS))) · line 7

Quoted 6 lines
scripts/Makefile.btf +0 -0
index db76335dd917..7c1cd6c2ff75 100644
@@ -7,14 +7,7 @@ JOBS := $(patsubst -j%,%,$(filter -j%,$(MAKEFLAGS)))
 

hi Ihor, a small suggestion here, and it is orthogonal to what you're doing here, so just for consideration if you're planning a v4 since you're touching this file.

Hi Alan. v4 for sure, and maybe even v5, we'll see.

Quoted 15 lines

We've had problems in the past because we get pahole version from .config in Makefile.btf

pahole-ver := $(CONFIG_PAHOLE_VERSION)

and it can be outdated.

Specifically the problem is that if "make oldconfig" is not run after updating pahole we don't get the actual pahole version during builds and options can be missing. See [1] for an example, but perhaps we should do

pahole-ver := $(shell $(srctree)/scripts/pahole-version.sh)

Quoted 7 lines

in Makefile.btf to ensure the value reflects latest pahole and that then determines which options we use? Andrii suggested an approach like CC_VERSION_TEXT might be worth pursuing; AFAICT that recomputes the CC_VERSION and warns the user if there is a version difference. Given that the CONFIG pahole version requirements are all pretty modest - it might simply be enough to recompute it in Makefile.btf and perhaps ensure it's not less than CONFIG_PAHOLE_VERSION. Just a thought anyway. Thanks!

Yeah, I am aware of the issue.

I am not sure version refresh in Makefile.btf would be enough, since there are config dependencies in Kconfig.debug. So we either need to trigger re-config, and maybe even force full kernel re-build, or somehow get rid of the version checks in the kconfig, which may be a challenge.

I think the simplest thing we could is to check if the version has changed and fail the build. That's a "panic!" approach though.

I'll look into how compiler versions are checked, maybe it's not that hard to add similar behavior for pahole.

Quoted 5 lines

Alan

[1] https://lore.kernel.org/bpf/CAEf4BzYi1xX3p_bY3j9dEuPvtCW3H7z=p2vdn-2GY0OOenxQAg@mail.gmail.com/

author reviewer
#22 lore raw

PATCH v3 5/6 selftests/bpf: Run resolve_btfids only for relevant .test.o objects

In reply to Ihor Solodrai

A selftest targeting resolve_btfids functionality relies on a resolved .BTF_ids section to be available in the TRUNNER_BINARY. The underlying BTF data is taken from a special BPF program (btf_data.c), and so resolve_btfids is executed as a part of a TRUNNER_BINARY build recipe on the final binary.

Subsequent patches in this series allow resolve_btfids to modify BTF before resolving the symbols, which means that the test needs access to that modified BTF [1]. Currently the test simply reads in btf_data.bpf.o on the assumption that BTF hasn't changed.

Implement resolve_btfids call only for particular test objects (just resolve_btfids.test.o for now). The test objects are linked into the TRUNNER_BINARY, and so .BTF_ids section will be available there.

This will make it trivial for the resolve_btfids test to access BTF modified by resolve_btfids.

[1] https://lore.kernel.org/bpf/CAErzpmvsgSDe-QcWH8SFFErL6y3p3zrqNri5-UHJ9iK2ChyiBw@mail.gmail.com/

---
 tools/testing/selftests/bpf/Makefile | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/Makefile +6 -2
index 4aa60e83ff19..ffd0a4c354c7 100644
@@ -643,6 +643,9 @@ $(TRUNNER_TESTS_HDR): $(TRUNNER_TESTS_DIR)/*.c
 		 ) > $$@)
 endif
 
+$(TRUNNER_OUTPUT)/resolve_btfids.test.o: $(RESOLVE_BTFIDS) $(TRUNNER_OUTPUT)/btf_data.bpf.o
+$(TRUNNER_OUTPUT)/resolve_btfids.test.o: private TEST_NEEDS_BTFIDS = 1
+
 # compile individual test files
 # Note: we cd into output directory to ensure embedded BPF object is found
 $(TRUNNER_TEST_OBJS): $(TRUNNER_OUTPUT)/%.test.o:			\
@@ -650,6 +653,9 @@ $(TRUNNER_TEST_OBJS): $(TRUNNER_OUTPUT)/%.test.o:			\
 		      | $(TRUNNER_OUTPUT)/%.test.d
 	$$(call msg,TEST-OBJ,$(TRUNNER_BINARY),$$@)
 	$(Q)cd $$(@D) && $$(CC) -I. $$(CFLAGS) -MMD -MT $$@ -c $(CURDIR)/$$< $$(LDLIBS) -o $$(@F)
+	$$(if $$(TEST_NEEDS_BTFIDS),					\
+		$$(call msg,BTFIDS,$(TRUNNER_BINARY),$$@)		\
+		$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.bpf.o $$@)
 
 $(TRUNNER_TEST_OBJS:.o=.d): $(TRUNNER_OUTPUT)/%.test.d:			\
 			    $(TRUNNER_TESTS_DIR)/%.c			\
@@ -695,13 +701,11 @@ $(OUTPUT)/$(TRUNNER_BINARY): | $(TRUNNER_BPF_OBJS)
 $(OUTPUT)/$(TRUNNER_BINARY): $(TRUNNER_TEST_OBJS)			\
 			     $(TRUNNER_EXTRA_OBJS) $$(BPFOBJ)		\
 			     $(TRUNNER_LIB_OBJS)			\
-			     $(RESOLVE_BTFIDS)				\
 			     $(TRUNNER_BPFTOOL)				\
 			     $(OUTPUT)/veristat				\
 			     | $(TRUNNER_BINARY)-extras
 	$$(call msg,BINARY,,$$@)
 	$(Q)$$(CC) $$(CFLAGS) $$(filter %.a %.o,$$^) $$(LDLIBS) $$(LLVM_LDLIBS) $$(LDFLAGS) $$(LLVM_LDFLAGS) -o $$@
-	$(Q)$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.bpf.o $$@
 	$(Q)ln -sf $(if $2,..,.)/tools/build/bpftool/$(USE_BOOTSTRAP)bpftool \
 		   $(OUTPUT)/$(if $2,$2/)bpftool
 
Signature · 1 line
2.52.0
author reviewer
#23 lore raw

PATCH v3 6/6 resolve_btfids: change in-place update with raw binary output

Currently resolve_btfids updates .BTF_ids section of an ELF file in-place, based on the contents of provided BTF, usually within the same input file, and optionally a BTF base.

Change resolve_btfids behavior to enable BTF transformations as part of its main operation. To achieve this, in-place ELF write in resolve_btfids is replaced with generation of the following binaries:
* ${1}.BTF with .BTF section data
* ${1}.BTF_ids with .BTF_ids section data if it existed in ${1}
* ${1}.BTF.base with .BTF.base section data for out-of-tree modules

The execution of resolve_btfids and consumption of its output is orchestrated by scripts/gen-btf.sh introduced in this patch.

The motivation for emitting binary data is that it allows simplifying resolve_btfids implementation by delegating ELF update to the $OBJCOPY tool [1], which is already widely used across the codebase.

There are two distinct paths for BTF generation and resolve_btfids application in the kernel build: for vmlinux and for kernel modules.

For the vmlinux binary a .BTF section is added in a roundabout way to ensure correct linking. The patch doesn't change this approach, only the implementation is a little different.

Before this patch it worked as follows:

* pahole consumed .tmp_vmlinux1 [2] and added .BTF section with llvm-objcopy [3] to it
* then everything except the .BTF section was stripped from .tmp_vmlinux1 into a .tmp_vmlinux1.bpf.o object [2], later linked into vmlinux
* resolve_btfids was executed later on vmlinux.unstripped [4], updating it in-place

After this patch gen-btf.sh implements the following:

* pahole consumes .tmp_vmlinux1 and produces a *detached* file with raw BTF data
* resolve_btfids consumes .tmp_vmlinux1 and detached BTF to produce (potentially modified) .BTF, and .BTF_ids sections data
* a .tmp_vmlinux1.bpf.o object is then produced with objcopy copying BTF output of resolve_btfids
* .BTF_ids data gets embedded into vmlinux.unstripped in

    link-vmlinux.sh by objcopy --update-section

For kernel modules, creating a special .bpf.o file is not necessary, and so embedding of sections data produced by resolve_btfids is straightforward with objcopy.

With this patch an ELF file becomes effectively read-only within resolve_btfids, which allows deleting elf_update() call and satellite code (like compressed_section_fix [5]).

Endianness handling of .BTF_ids data is also changed. Previously the "flags" part of the section was bswapped in sets_patch() [6], and then Elf_Type was modified before elf_update() to signal to libelf that bswap may be necessary. With this patch we explicitly bswap entire data buffer on load and on dump.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/link-vmlinux.sh?h=v6.18#n110
[3] https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/btf_encoder.c?h=v1.31#n1803
[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/link-vmlinux.sh?h=v6.18#n284
[5] https://lore.kernel.org/bpf/[email protected]/
[6] https://lore.kernel.org/bpf/[email protected]/

---
 MAINTAINERS                                   |   1 +
 scripts/Makefile.btf                          |  17 +-
 scripts/Makefile.modfinal                     |   5 +-
 scripts/Makefile.vmlinux                      |   2 +-
 scripts/gen-btf.sh                            | 157 ++++++++++++
 scripts/link-vmlinux.sh                       |  46 +---
 tools/bpf/resolve_btfids/main.c               | 228 +++++++++++-------
 tools/testing/selftests/bpf/.gitignore        |   3 +
 tools/testing/selftests/bpf/Makefile          |   9 +-
 .../selftests/bpf/prog_tests/resolve_btfids.c |   4 +-
 10 files changed, 338 insertions(+), 134 deletions(-)
 create mode 100755 scripts/gen-btf.sh
MAINTAINERS +1 -0
index e36689cd7cc7..fe6141c69708 100644
@@ -4673,6 +4673,7 @@ F:	net/sched/act_bpf.c
 F:	net/sched/cls_bpf.c
 F:	samples/bpf/
 F:	scripts/bpf_doc.py
+F:	scripts/gen-btf.sh
 F:	scripts/Makefile.btf
 F:	scripts/pahole-version.sh
 F:	tools/bpf/
scripts/Makefile.btf +12 -5
index 7c1cd6c2ff75..d067e91049cb 100644
@@ -1,5 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
 
+gen-btf-y				=
+gen-btf-$(CONFIG_DEBUG_INFO_BTF)	= $(srctree)/scripts/gen-btf.sh
+
+export GEN_BTF := $(gen-btf-y)
+
 pahole-ver := $(CONFIG_PAHOLE_VERSION)
 pahole-flags-y :=
 
@@ -18,13 +23,15 @@ pahole-flags-$(call test-ge, $(pahole-ver), 126)  = -j$(JOBS) --btf_features=enc
 
 pahole-flags-$(call test-ge, $(pahole-ver), 130) += --btf_features=attributes
 
-ifneq ($(KBUILD_EXTMOD),)
-module-pahole-flags-$(call test-ge, $(pahole-ver), 128) += --btf_features=distilled_base
-endif
-
 endif
 
 pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE)		+= --lang_exclude=rust
 
 export PAHOLE_FLAGS := $(pahole-flags-y)
-export MODULE_PAHOLE_FLAGS := $(module-pahole-flags-y)
+
+resolve-btfids-flags-y :=
+resolve-btfids-flags-$(CONFIG_WERROR) += --fatal_warnings
+resolve-btfids-flags-$(if $(KBUILD_EXTMOD),y) += --distill_base
+resolve-btfids-flags-$(if $(KBUILD_VERBOSE),y) += --verbose
+
+export RESOLVE_BTFIDS_FLAGS := $(resolve-btfids-flags-y)
scripts/Makefile.modfinal +2 -3
index 542ba462ed3e..9e4722e55efa 100644
@@ -38,9 +38,8 @@ quiet_cmd_btf_ko = BTF [M] $@
       cmd_btf_ko = 							\
 	if [ ! -f $(objtree)/vmlinux ]; then				\
 		printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
-	else								\
-		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) $(MODULE_PAHOLE_FLAGS) --btf_base $(objtree)/vmlinux $@; \
-		$(RESOLVE_BTFIDS) -b $(objtree)/vmlinux $@;		\
+	else	\
+		$(GEN_BTF) --btf_base $(objtree)/vmlinux $@; \
 	fi;
 
 # Same as newer-prereqs, but allows to exclude specified extra dependencies
scripts/Makefile.vmlinux +1 -1
index cd788cac9d91..33e9999724cc 100644
@@ -71,7 +71,7 @@ targets += vmlinux.unstripped .vmlinux.export.o
 vmlinux.unstripped: scripts/link-vmlinux.sh vmlinux.o .vmlinux.export.o $(KBUILD_LDS) FORCE
 	+$(call if_changed_dep,link_vmlinux)
 ifdef CONFIG_DEBUG_INFO_BTF
-vmlinux.unstripped: $(RESOLVE_BTFIDS)
+vmlinux.unstripped: $(RESOLVE_BTFIDS) $(GEN_BTF)
 endif
 
 ifdef CONFIG_BUILDTIME_TABLE_SORT
scripts/gen-btf.sh new file +157 -0

157 lines not on this page · raw message

scripts/link-vmlinux.sh +8 -38

74 lines not on this page · raw message

tools/bpf/resolve_btfids/main.c +146 -82

339 lines not on this page · raw message

tools/testing/selftests/bpf/.gitignore +3 -0
index 19c1638e312a..b8bf51b7a0b0 100644
@@ -45,3 +45,6 @@ xdp_synproxy
 xdp_hw_metadata
 xdp_features
 verification_cert.h
+*.BTF
+*.BTF_ids
+*.BTF.base
tools/testing/selftests/bpf/Makefile +6 -3
index ffd0a4c354c7..f28a32b16ff0 100644
@@ -4,6 +4,7 @@ include ../../../scripts/Makefile.arch
 include ../../../scripts/Makefile.include
 
 CXX ?= $(CROSS_COMPILE)g++
+OBJCOPY ?= $(CROSS_COMPILE)objcopy
 
 CURDIR := $(abspath .)
 TOOLSDIR := $(abspath ../../..)
@@ -653,9 +654,10 @@ $(TRUNNER_TEST_OBJS): $(TRUNNER_OUTPUT)/%.test.o:			\
 		      | $(TRUNNER_OUTPUT)/%.test.d
 	$$(call msg,TEST-OBJ,$(TRUNNER_BINARY),$$@)
 	$(Q)cd $$(@D) && $$(CC) -I. $$(CFLAGS) -MMD -MT $$@ -c $(CURDIR)/$$< $$(LDLIBS) -o $$(@F)
-	$$(if $$(TEST_NEEDS_BTFIDS),					\
-		$$(call msg,BTFIDS,$(TRUNNER_BINARY),$$@)		\
-		$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.bpf.o $$@)
+	$$(if $$(TEST_NEEDS_BTFIDS),						\
+		$$(call msg,BTFIDS,$(TRUNNER_BINARY),$$@)			\
+		$(RESOLVE_BTFIDS) --btf $(TRUNNER_OUTPUT)/btf_data.bpf.o $$@;	\
+		$(OBJCOPY) --update-section [email protected]_ids $$@)
 
 $(TRUNNER_TEST_OBJS:.o=.d): $(TRUNNER_OUTPUT)/%.test.d:			\
 			    $(TRUNNER_TESTS_DIR)/%.c			\
@@ -894,6 +896,7 @@ EXTRA_CLEAN := $(SCRATCH_DIR) $(HOST_SCRATCH_DIR)			\
 	prog_tests/tests.h map_tests/tests.h verifier/tests.h		\
 	feature bpftool $(TEST_KMOD_TARGETS)				\
 	$(addprefix $(OUTPUT)/,*.o *.d *.skel.h *.lskel.h *.subskel.h	\
+			       *.BTF *.BTF_ids *.BTF.base		\
 			       no_alu32 cpuv4 bpf_gcc			\
 			       liburandom_read.so)			\
 	$(OUTPUT)/FEATURE-DUMP.selftests				\
tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +2 -2
index 51544372f52e..41dfaaabb73f 100644
@@ -101,9 +101,9 @@ static int resolve_symbols(void)
 	int type_id;
 	__u32 nr;
 
-	btf = btf__parse_elf("btf_data.bpf.o", NULL);
+	btf = btf__parse_raw("resolve_btfids.test.o.BTF");
 	if (CHECK(libbpf_get_error(btf), "resolve",
-		  "Failed to load BTF from btf_data.bpf.o\n"))
+		  "Failed to load BTF from resolve_btfids.test.o.BTF\n"))
 		return -1;
 
 	nr = btf__type_cnt(btf);
Signature · 1 line
2.52.0
#24 lore raw

Re: [PATCH bpf-next v3 6/6] resolve_btfids: change in-place update with raw binary output

Quoted 68 lines · Ihor Solodrai

Currently resolve_btfids updates .BTF_ids section of an ELF file in-place, based on the contents of provided BTF, usually within the same input file, and optionally a BTF base.

Change resolve_btfids behavior to enable BTF transformations as part of its main operation. To achieve this, in-place ELF write in resolve_btfids is replaced with generation of the following binaries:
* ${1}.BTF with .BTF section data
* ${1}.BTF_ids with .BTF_ids section data if it existed in ${1}
* ${1}.BTF.base with .BTF.base section data for out-of-tree modules

The execution of resolve_btfids and consumption of its output is orchestrated by scripts/gen-btf.sh introduced in this patch.

The motivation for emitting binary data is that it allows simplifying resolve_btfids implementation by delegating ELF update to the $OBJCOPY tool [1], which is already widely used across the codebase.

There are two distinct paths for BTF generation and resolve_btfids application in the kernel build: for vmlinux and for kernel modules.

For the vmlinux binary a .BTF section is added in a roundabout way to ensure correct linking. The patch doesn't change this approach, only the implementation is a little different.

Before this patch it worked as follows:

* pahole consumed .tmp_vmlinux1 [2] and added .BTF section with llvm-objcopy [3] to it
* then everything except the .BTF section was stripped from .tmp_vmlinux1 into a .tmp_vmlinux1.bpf.o object [2], later linked into vmlinux
* resolve_btfids was executed later on vmlinux.unstripped [4], updating it in-place

After this patch gen-btf.sh implements the following:

* pahole consumes .tmp_vmlinux1 and produces a *detached* file with raw BTF data
* resolve_btfids consumes .tmp_vmlinux1 and detached BTF to produce (potentially modified) .BTF, and .BTF_ids sections data
* a .tmp_vmlinux1.bpf.o object is then produced with objcopy copying BTF output of resolve_btfids
* .BTF_ids data gets embedded into vmlinux.unstripped in

    link-vmlinux.sh by objcopy --update-section

For kernel modules, creating a special .bpf.o file is not necessary, and so embedding of sections data produced by resolve_btfids is straightforward with objcopy.

With this patch an ELF file becomes effectively read-only within resolve_btfids, which allows deleting elf_update() call and satellite code (like compressed_section_fix [5]).

Endianness handling of .BTF_ids data is also changed. Previously the "flags" part of the section was bswapped in sets_patch() [6], and then Elf_Type was modified before elf_update() to signal to libelf that bswap may be necessary. With this patch we explicitly bswap entire data buffer on load and on dump.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/link-vmlinux.sh?h=v6.18#n110
[3] https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/btf_encoder.c?h=v1.31#n1803
[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/link-vmlinux.sh?h=v6.18#n284
[5] https://lore.kernel.org/bpf/[email protected]/
[6] https://lore.kernel.org/bpf/[email protected]/

Quoted 14 lines
---
 MAINTAINERS                                   |   1 +
 scripts/Makefile.btf                          |  17 +-
 scripts/Makefile.modfinal                     |   5 +-
 scripts/Makefile.vmlinux                      |   2 +-
 scripts/gen-btf.sh                            | 157 ++++++++++++
 scripts/link-vmlinux.sh                       |  46 +---
 tools/bpf/resolve_btfids/main.c               | 228 +++++++++++-------
 tools/testing/selftests/bpf/.gitignore        |   3 +
 tools/testing/selftests/bpf/Makefile          |   9 +-
 .../selftests/bpf/prog_tests/resolve_btfids.c |   4 +-
 10 files changed, 338 insertions(+), 134 deletions(-)
 create mode 100755 scripts/gen-btf.sh

Overall it looks good, but I'd like another pair of eyes on this :) See some more minore nits below as well.

pw-bot: cr

Quoted 17 lines
MAINTAINERS +1 -0
index e36689cd7cc7..fe6141c69708 100644
@@ -4673,6 +4673,7 @@ F:        net/sched/act_bpf.c
 F:     net/sched/cls_bpf.c
 F:     samples/bpf/
 F:     scripts/bpf_doc.py
+F:     scripts/gen-btf.sh
 F:     scripts/Makefile.btf
 F:     scripts/pahole-version.sh
 F:     tools/bpf/
scripts/Makefile.btf +0 -0
index 7c1cd6c2ff75..d067e91049cb 100644

scripts/Makefile.btf line 7

Quoted 7 lines
scripts/Makefile.btf +5 -0
@@ -1,5 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
 
+gen-btf-y                              =
+gen-btf-$(CONFIG_DEBUG_INFO_BTF)       = $(srctree)/scripts/gen-btf.sh
+
+export GEN_BTF := $(gen-btf-y)
+

What's the point of GEN_BTF? It's just so that you don't have to have $(srctree)/scripts/gen-btf.sh specified in three places? Between obscure $(GEN_BTF) (and having to understand where it is set and how it's exported) and explicit $(srctree)/scripts/gen-btf.sh in a few places, I'd prefer the latter, as it is way more greppable and it's not like we are going to rename or move this script frequently

Quoted 3 lines
scripts/Makefile.btf +0 -0
 pahole-ver := $(CONFIG_PAHOLE_VERSION)
 pahole-flags-y :=
 

[...]

Quoted 10 lines
scripts/Makefile.btf +1 -1
@@ -371,7 +348,7 @@ static int elf_collect(struct object *obj)
 
        elf_version(EV_CURRENT);
 
-       elf = elf_begin(fd, ELF_C_RDWR_MMAP, NULL);
+       elf = elf_begin(fd, ELF_C_READ_MMAP_PRIVATE, NULL);
        if (!elf) {
                close(fd);
                pr_err("FAILED cannot create ELF descriptor: %s\n",

scripts/Makefile.btf static int elf_collect(struct object *obj) · line 418

Quoted 8 lines
scripts/Makefile.btf +5 -0
@@ -434,21 +411,20 @@ static int elf_collect(struct object *obj)
                        obj->efile.symbols_shndx = idx;
                        obj->efile.strtabidx     = sh.sh_link;
                } else if (!strcmp(name, BTF_IDS_SECTION)) {
+                       /*
+                        * If target endianness differs from host, we need to bswap32
+                        * the .BTF_ids section data on load, because .BTF_ids has
+                        * Elf_Type = ELF_T_BYTE, and so libelf returns data buffer in
+                        * the target endiannes. We repeat this on dump.

gmail screams at me for "endianness"

scripts/Makefile.btf static int elf_collect(struct object *obj) · line 422

Quoted 4 lines
scripts/Makefile.btf +4 -0
+                        */
+                       if (obj->efile.encoding != ELFDATANATIVE) {
+                               pr_debug("bswap_32 .BTF_ids data from target to host endianness\n");
+                               bswap_32_data(data->d_buf, data->d_size);

this looks like a violation of ELF_C_READ_MMAP_PRIVATE promise, no?... would it be too create a copy here? for simplicity we can just always malloc() a copy, regardless of bswap(), it can never be a huge amount of data

Quoted 20 lines
scripts/Makefile.btf +1 -11
+                       }
                        obj->efile.idlist       = data;
                        obj->efile.idlist_shndx = idx;
                        obj->efile.idlist_addr  = sh.sh_addr;
-               } else if (!strcmp(name, BTF_BASE_ELF_SEC)) {
-                       /* If a .BTF.base section is found, do not resolve
-                        * BTF ids relative to vmlinux; resolve relative
-                        * to the .BTF.base section instead.  btf__parse_split()
-                        * will take care of this once the base BTF it is
-                        * passed is NULL.
-                        */
-                       obj->base_btf_path = NULL;
                }
-
-               if (compressed_section_fix(elf, scn, &sh))
-                       return -1;
        }
 
        return 0;

scripts/Makefile.btf static int symbols_collect(struct object *obj) · line 537

Quoted 10 lines
scripts/Makefile.btf +7 -0
@@ -552,6 +528,13 @@ static int symbols_collect(struct object *obj)
        return 0;
 }
 
+static inline bool is_envvar_set(const char *var_name)
+{
+       const char *value = getenv(var_name);
+
+       return value && value[0] != '\0';
+}
+

leftovers?

Quoted 23 lines
scripts/Makefile.btf +13 -0
 static int load_btf(struct object *obj)
 {
        struct btf *base_btf = NULL, *btf = NULL;
@@ -578,6 +561,19 @@ static int load_btf(struct object *obj)
        obj->base_btf = base_btf;
        obj->btf = btf;
 
+       if (obj->base_btf && obj->distill_base) {
+               err = btf__distill_base(obj->btf, &base_btf, &btf);
+               if (err) {
+                       pr_err("FAILED to distill base BTF: %s\n", strerror(errno));
+                       goto out_err;
+               }
+
+               btf__free(obj->btf);
+               btf__free(obj->base_btf);
+               obj->btf = btf;
+               obj->base_btf = base_btf;
+       }
+
        return 0;
 
 out_err:

[...]

Quoted 8 lines
scripts/Makefile.btf +8 -0
+static int dump_raw_btf_ids(struct object *obj, const char *out_path)
+{
+       Elf_Data *data = obj->efile.idlist;
+       int fd, err;
+
+       if (!data || !data->d_buf) {
+               pr_debug("%s has no BTF_ids data to dump\n", obj->path);
+               return 0;

scripts/Makefile.btf

Quoted 10 lines
scripts/Makefile.btf +10 -0
+       }
+
+       /*
+        * If target endianness differs from host, we need to bswap32 the
+        * .BTF_ids section data before dumping so that the output is in
+        * target endianness.
+        */
+       if (obj->efile.encoding != ELFDATANATIVE) {
+               pr_debug("bswap_32 .BTF_ids data from host to target endianness\n");
+               bswap_32_data(data->d_buf, data->d_size);

same about modifying ELF data in-place for what is supposed to be read-only use

Quoted 10 lines
scripts/Makefile.btf +10 -0
+       }
+
+       err = dump_raw_data(out_path, data->d_buf, data->d_size);
+       if (err)
+               return -1;
+
+       return 0;
+}
+
+static int dump_raw_btf(struct btf *btf, const char *out_path)

scripts/Makefile.btf

Quoted 10 lines
scripts/Makefile.btf +10 -0
+{
+       const void *raw_btf_data;
+       u32 raw_btf_size;
+       int fd, err;
+
+       raw_btf_data = btf__raw_data(btf, &raw_btf_size);
+       if (!raw_btf_data) {
+               pr_err("btf__raw_data() failed\n");
+               return -1;
+       }

did you check that libbpf does proper byte swap as well?

scripts/Makefile.btf

Quoted 11 lines
scripts/Makefile.btf +11 -0
+
+       err = dump_raw_data(out_path, raw_btf_data, raw_btf_size);
+       if (err)
+               return -1;
+
+       return 0;
+}
+
+static inline int make_out_path(char *buf, const char *in_path, const char *suffix)
+{
+       int len = snprintf(buf, PATH_MAX, "%s%s", in_path, suffix);

nit: normally you pass buffer and its size as input arguments instead of assuming and hard-coding common PATH_MAX constant in two separate places

Quoted 13 lines
scripts/Makefile.btf +5 -3
+
+       if (len < 0 || len >= PATH_MAX) {
+               pr_err("Output path is too long: %s%s\n", in_path, suffix);
+               return -E2BIG;
        }
 
-       pr_debug("update %s for %s\n",
-                err >= 0 ? "ok" : "failed", obj->path);
-       return err < 0 ? -1 : 0;
+       return 0;
 }
 
 static const char * const resolve_btfids_usage[] = {

[...]

author reviewer
#25 lore raw

Re: [PATCH bpf-next v3 6/6] resolve_btfids: change in-place update with raw binary output

Quoted 6 lines · Andrii Nakryiko
Quoted 3 lines · Ihor Solodrai

[...]

Overall it looks good, but I'd like another pair of eyes on this :)

I don't think you need to worry about another pair of eyes :)

Quoted 24 lines

See some more minore nits below as well.

pw-bot: cr

Quoted 19 lines
MAINTAINERS +1 -0
index e36689cd7cc7..fe6141c69708 100644
@@ -4673,6 +4673,7 @@ F:        net/sched/act_bpf.c
 F:     net/sched/cls_bpf.c
 F:     samples/bpf/
 F:     scripts/bpf_doc.py
+F:     scripts/gen-btf.sh
 F:     scripts/Makefile.btf
 F:     scripts/pahole-version.sh
 F:     tools/bpf/
scripts/Makefile.btf +0 -0
index 7c1cd6c2ff75..d067e91049cb 100644
@@ -1,5 +1,10 @@
 # SPDX-License-Identifier: GPL-2.0
 
Quoted 12 lines

scripts/Makefile.btf line 7

Quoted 5 lines
scripts/Makefile.btf +5 -0
+gen-btf-y                              =
+gen-btf-$(CONFIG_DEBUG_INFO_BTF)       = $(srctree)/scripts/gen-btf.sh
+
+export GEN_BTF := $(gen-btf-y)
+

What's the point of GEN_BTF? It's just so that you don't have to have $(srctree)/scripts/gen-btf.sh specified in three places? Between obscure $(GEN_BTF) (and having to understand where it is set and how it's exported) and explicit $(srctree)/scripts/gen-btf.sh in a few places, I'd prefer the latter, as it is way more greppable and it's not like we are going to rename or move this script frequently

Yeah... It seems fine for me either way, tbh.

I think a slight difference is that in link-vmlinux.sh checking for a config value is a grep on include/config/auto.conf, which is not necessary if we have a dedicated env variable.

Quoted 12 lines
Quoted 5 lines
scripts/Makefile.btf +0 -0
 pahole-ver := $(CONFIG_PAHOLE_VERSION)
 pahole-flags-y :=
 

[...]

Quoted 4 lines
scripts/Makefile.btf +0 -0
@@ -371,7 +348,7 @@ static int elf_collect(struct object *obj)
 
        elf_version(EV_CURRENT);
 
Quoted 16 lines
Quoted 6 lines
scripts/Makefile.btf +1 -1
-       elf = elf_begin(fd, ELF_C_RDWR_MMAP, NULL);
+       elf = elf_begin(fd, ELF_C_READ_MMAP_PRIVATE, NULL);
        if (!elf) {
                close(fd);
                pr_err("FAILED cannot create ELF descriptor: %s\n",

scripts/Makefile.btf static int elf_collect(struct object *obj) · line 418

Quoted 8 lines
scripts/Makefile.btf +5 -0
@@ -434,21 +411,20 @@ static int elf_collect(struct object *obj)
                        obj->efile.symbols_shndx = idx;
                        obj->efile.strtabidx     = sh.sh_link;
                } else if (!strcmp(name, BTF_IDS_SECTION)) {
+                       /*
+                        * If target endianness differs from host, we need to bswap32
+                        * the .BTF_ids section data on load, because .BTF_ids has
+                        * Elf_Type = ELF_T_BYTE, and so libelf returns data buffer in
+                        * the target endiannes. We repeat this on dump.

gmail screams at me for "endianness"

That's how you know I am not AI.

Quoted 10 lines

scripts/Makefile.btf static int elf_collect(struct object *obj) · line 422

Quoted 5 lines
scripts/Makefile.btf +4 -0
+                        */
+                       if (obj->efile.encoding != ELFDATANATIVE) {
+                               pr_debug("bswap_32 .BTF_ids data from target to host endianness\n");
+                               bswap_32_data(data->d_buf, data->d_size);

this looks like a violation of ELF_C_READ_MMAP_PRIVATE promise, no?... would it be too create a copy here? for simplicity we can just always malloc() a copy, regardless of bswap(), it can never be a huge amount of data

No, it's not a violation. From libelf.h:

ELF_C_READ_MMAP_PRIVATE, /* Read, but memory is writable, results are
not written to the file. */

I haven't checked how it works under the hood, but from testing so far the comment seems to be telling the truth.

Quoted 24 lines
Quoted 24 lines
scripts/Makefile.btf +1 -11
+                       }
                        obj->efile.idlist       = data;
                        obj->efile.idlist_shndx = idx;
                        obj->efile.idlist_addr  = sh.sh_addr;
-               } else if (!strcmp(name, BTF_BASE_ELF_SEC)) {
-                       /* If a .BTF.base section is found, do not resolve
-                        * BTF ids relative to vmlinux; resolve relative
-                        * to the .BTF.base section instead.  btf__parse_split()
-                        * will take care of this once the base BTF it is
-                        * passed is NULL.
-                        */
-                       obj->base_btf_path = NULL;
                }
-
-               if (compressed_section_fix(elf, scn, &sh))
-                       return -1;
        }
 
        return 0;
@@ -552,6 +528,13 @@ static int symbols_collect(struct object *obj)
        return 0;
 }
 
Quoted 9 lines

scripts/Makefile.btf static int symbols_collect(struct object *obj) · line 537

Quoted 7 lines
scripts/Makefile.btf +7 -0
+static inline bool is_envvar_set(const char *var_name)
+{
+       const char *value = getenv(var_name);
+
+       return value && value[0] != '\0';
+}
+

leftovers?

Yes. I think I should add "fail build on warnings" for resolve_btfids.

Quoted 39 lines
Quoted 24 lines
scripts/Makefile.btf +13 -0
 static int load_btf(struct object *obj)
 {
        struct btf *base_btf = NULL, *btf = NULL;
@@ -578,6 +561,19 @@ static int load_btf(struct object *obj)
        obj->base_btf = base_btf;
        obj->btf = btf;
 
+       if (obj->base_btf && obj->distill_base) {
+               err = btf__distill_base(obj->btf, &base_btf, &btf);
+               if (err) {
+                       pr_err("FAILED to distill base BTF: %s\n", strerror(errno));
+                       goto out_err;
+               }
+
+               btf__free(obj->btf);
+               btf__free(obj->base_btf);
+               obj->btf = btf;
+               obj->base_btf = base_btf;
+       }
+
        return 0;
 
 out_err:

[...]

Quoted 12 lines
scripts/Makefile.btf +12 -0
+static int dump_raw_btf_ids(struct object *obj, const char *out_path)
+{
+       Elf_Data *data = obj->efile.idlist;
+       int fd, err;
+
+       if (!data || !data->d_buf) {
+               pr_debug("%s has no BTF_ids data to dump\n", obj->path);
+               return 0;
+       }
+
+       /*
+        * If target endianness differs from host, we need to bswap32 the
Quoted 8 lines

scripts/Makefile.btf

Quoted 6 lines
scripts/Makefile.btf +6 -0
+        * .BTF_ids section data before dumping so that the output is in
+        * target endianness.
+        */
+       if (obj->efile.encoding != ELFDATANATIVE) {
+               pr_debug("bswap_32 .BTF_ids data from host to target endianness\n");
+               bswap_32_data(data->d_buf, data->d_size);

same about modifying ELF data in-place for what is supposed to be read-only use

See above.

Quoted 15 lines
Quoted 15 lines
scripts/Makefile.btf +14 -0
+       }
+
+       err = dump_raw_data(out_path, data->d_buf, data->d_size);
+       if (err)
+               return -1;
+
+       return 0;
+}
+
+static int dump_raw_btf(struct btf *btf, const char *out_path)
+{
+       const void *raw_btf_data;
+       u32 raw_btf_size;
+       int fd, err;
Quoted 8 lines

scripts/Makefile.btf

Quoted 6 lines
scripts/Makefile.btf +6 -0
+
+       raw_btf_data = btf__raw_data(btf, &raw_btf_size);
+       if (!raw_btf_data) {
+               pr_err("btf__raw_data() failed\n");
+               return -1;
+       }

did you check that libbpf does proper byte swap as well?

libbpf is tracking BTF endianness internally, we don't have to worry about it here. If it wasn't, s390x selftests would have almost certainly failed (and they did when I messed up).

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/lib/bpf/btf.c?h=v6.18#n227

Quoted 16 lines

scripts/Makefile.btf

Quoted 12 lines
scripts/Makefile.btf +11 -0
+
+       err = dump_raw_data(out_path, raw_btf_data, raw_btf_size);
+       if (err)
+               return -1;
+
+       return 0;
+}
+
+static inline int make_out_path(char *buf, const char *in_path, const char *suffix)
+{
+       int len = snprintf(buf, PATH_MAX, "%s%s", in_path, suffix);

nit: normally you pass buffer and its size as input arguments instead of assuming and hard-coding common PATH_MAX constant in two separate places

acked

Quoted 16 lines
Quoted 14 lines
scripts/Makefile.btf +5 -3
+
+       if (len < 0 || len >= PATH_MAX) {
+               pr_err("Output path is too long: %s%s\n", in_path, suffix);
+               return -E2BIG;
        }
 
-       pr_debug("update %s for %s\n",
-                err >= 0 ? "ok" : "failed", obj->path);
-       return err < 0 ? -1 : 0;
+       return 0;
 }
 
 static const char * const resolve_btfids_usage[] = {

[...]

maintainer Acked-by on v3 6/6
#26 lore raw

Re: [PATCH bpf-next v3 6/6] resolve_btfids: change in-place update with raw binary output

In reply to Ihor Solodrai
Quoted 66 lines · Ihor Solodrai

Currently resolve_btfids updates .BTF_ids section of an ELF file in-place, based on the contents of provided BTF, usually within the same input file, and optionally a BTF base.

Change resolve_btfids behavior to enable BTF transformations as part of its main operation. To achieve this, in-place ELF write in resolve_btfids is replaced with generation of the following binaries:
* ${1}.BTF with .BTF section data
* ${1}.BTF_ids with .BTF_ids section data if it existed in ${1}
* ${1}.BTF.base with .BTF.base section data for out-of-tree modules

The execution of resolve_btfids and consumption of its output is orchestrated by scripts/gen-btf.sh introduced in this patch.

The motivation for emitting binary data is that it allows simplifying resolve_btfids implementation by delegating ELF update to the $OBJCOPY tool [1], which is already widely used across the codebase.

There are two distinct paths for BTF generation and resolve_btfids application in the kernel build: for vmlinux and for kernel modules.

For the vmlinux binary a .BTF section is added in a roundabout way to ensure correct linking. The patch doesn't change this approach, only the implementation is a little different.

Before this patch it worked as follows:

* pahole consumed .tmp_vmlinux1 [2] and added .BTF section with llvm-objcopy [3] to it
* then everything except the .BTF section was stripped from .tmp_vmlinux1 into a .tmp_vmlinux1.bpf.o object [2], later linked into vmlinux
* resolve_btfids was executed later on vmlinux.unstripped [4], updating it in-place

After this patch gen-btf.sh implements the following:

* pahole consumes .tmp_vmlinux1 and produces a *detached* file with raw BTF data
* resolve_btfids consumes .tmp_vmlinux1 and detached BTF to produce (potentially modified) .BTF, and .BTF_ids sections data
* a .tmp_vmlinux1.bpf.o object is then produced with objcopy copying BTF output of resolve_btfids
* .BTF_ids data gets embedded into vmlinux.unstripped in

    link-vmlinux.sh by objcopy --update-section

For kernel modules, creating a special .bpf.o file is not necessary, and so embedding of sections data produced by resolve_btfids is straightforward with objcopy.

With this patch an ELF file becomes effectively read-only within resolve_btfids, which allows deleting elf_update() call and satellite code (like compressed_section_fix [5]).

Endianness handling of .BTF_ids data is also changed. Previously the "flags" part of the section was bswapped in sets_patch() [6], and then Elf_Type was modified before elf_update() to signal to libelf that bswap may be necessary. With this patch we explicitly bswap entire data buffer on load and on dump.

[1] https://lore.kernel.org/bpf/[email protected]/
[2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/link-vmlinux.sh?h=v6.18#n110
[3] https://git.kernel.org/pub/scm/devel/pahole/pahole.git/tree/btf_encoder.c?h=v1.31#n1803
[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/scripts/link-vmlinux.sh?h=v6.18#n284
[5] https://lore.kernel.org/bpf/[email protected]/
[6] https://lore.kernel.org/bpf/[email protected]/

Quoted 2 lines

---

tools/bpf/resolve_btfids/main.c static int symbols_collect(struct object *obj) · line 537

Quoted 11 lines
tools/bpf/resolve_btfids/main.c +7 -0
@@ -552,6 +528,13 @@ static int symbols_collect(struct object *obj)
 	return 0;
 }
 
+static inline bool is_envvar_set(const char *var_name)
+{
+	const char *value = getenv(var_name);
+
+	return value && value[0] != '\0';
+}
+

This is a leftover, not used anywhere.

[...]

Quoted 20 lines
tools/bpf/resolve_btfids/main.c +2 -6
@@ -860,23 +913,34 @@ int main(int argc, const char **argv)
 	 */
 	if (obj.efile.idlist_shndx == -1 ||
 	    obj.efile.symbols_shndx == -1) {
-		pr_debug("Cannot find .BTF_ids or symbols sections, nothing to do\n");
-		err = 0;
-		goto out;
+		pr_debug("Cannot find .BTF_ids or symbols sections, skip symbols resolution\n");
+		goto dump_btf;
 	}
 
 	if (symbols_collect(&obj))
 		goto out;
 
-	if (load_btf(&obj))
-		goto out;
-
 	if (symbols_resolve(&obj))
 		goto out;
 

tools/bpf/resolve_btfids/main.c int main(int argc, const char **argv) · line 935

Quoted 10 lines
tools/bpf/resolve_btfids/main.c +7 -0
 	if (symbols_patch(&obj))
 		goto out;
 
+	err = make_out_path(out_path, obj.path, BTF_IDS_SECTION);
+	if (err || dump_raw_btf_ids(&obj, out_path))
+		goto out;
+
+dump_btf:
+	err = make_out_path(out_path, obj.path, BTF_ELF_SEC);
+	if (err || dump_raw_btf(obj.btf, out_path))

Nit: 'err' is not set if dump_raw_btf() errors out.

     Maybe use:

     	   err = make_out_path(out_path, obj.path, BTF_ELF_SEC);
     	   err = err ?: dump_raw_btf(obj.btf, out_path);
	   if (err)
	      goto out;
     ?
Quoted 11 lines
tools/bpf/resolve_btfids/main.c +8 -0
+		goto out;
+
+	if (obj.base_btf && obj.distill_base) {
+		err = make_out_path(out_path, obj.path, BTF_BASE_ELF_SEC);
+		if (err || dump_raw_btf(obj.base_btf, out_path))
+			goto out;
+	}
+
 	if (!(fatal_warnings && warnings))
 		err = 0;
 out:

[...]

author reviewer
#27 lore raw

Re: [PATCH bpf-next v3 6/6] resolve_btfids: change in-place update with raw binary output

Quoted 36 lines · Eduard Zingerman

On Fri, 2025-12-05 at 14:35 -0800, Ihor Solodrai wrote:

[...]

Quoted 20 lines
tools/bpf/resolve_btfids/main.c +2 -6
@@ -860,23 +913,34 @@ int main(int argc, const char **argv)
 	 */
 	if (obj.efile.idlist_shndx == -1 ||
 	    obj.efile.symbols_shndx == -1) {
-		pr_debug("Cannot find .BTF_ids or symbols sections, nothing to do\n");
-		err = 0;
-		goto out;
+		pr_debug("Cannot find .BTF_ids or symbols sections, skip symbols resolution\n");
+		goto dump_btf;
 	}
 
 	if (symbols_collect(&obj))
 		goto out;
 
-	if (load_btf(&obj))
-		goto out;
-
 	if (symbols_resolve(&obj))
 		goto out;
 

tools/bpf/resolve_btfids/main.c int main(int argc, const char **argv) · line 935

Quoted 10 lines
tools/bpf/resolve_btfids/main.c +7 -0
 	if (symbols_patch(&obj))
 		goto out;
 
+	err = make_out_path(out_path, obj.path, BTF_IDS_SECTION);
+	if (err || dump_raw_btf_ids(&obj, out_path))
+		goto out;
+
+dump_btf:
+	err = make_out_path(out_path, obj.path, BTF_ELF_SEC);
+	if (err || dump_raw_btf(obj.btf, out_path))

Nit: 'err' is not set if dump_raw_btf() errors out.

Quoted 7 lines
     Maybe use:

     	   err = make_out_path(out_path, obj.path, BTF_ELF_SEC);
     	   err = err ?: dump_raw_btf(obj.btf, out_path);
	   if (err)
	      goto out;
     ?

Good observation. I'll fix this, thanks.

Quoted 14 lines
Quoted 12 lines
tools/bpf/resolve_btfids/main.c +8 -0
+		goto out;
+
+	if (obj.base_btf && obj.distill_base) {
+		err = make_out_path(out_path, obj.path, BTF_BASE_ELF_SEC);
+		if (err || dump_raw_btf(obj.base_btf, out_path))
+			goto out;
+	}
+
 	if (!(fatal_warnings && warnings))
 		err = 0;
 out:

[...]