Skip to contentSkip to conversation

Add diagnostics for kfunc registration failures and jit subtest skips

PATCH In review Wang Yibo 3 participants 7 messages Oct 22 – Oct 25

Subsystems (4)
lore
Share & tools
lore

Wang Yibo

author
#0 lore raw

cover Add diagnostics for kfunc registration failures and jit subtest skips

While testing the unwind_goto series on arm64 I ran into two places where a failure is reported without any information about what went wrong, and each took a while to track down. These three patches add the diagnostics that would have saved that time.

Patch 1: when a kfunc in a registered set has no proper FUNC entry in the BTF of the registering object, register_btf_kfunc_id_set() returns -EINVAL and nothing is printed. With bpf_testmod built by pahole 1.30 the module fails to load with -22 and dmesg stays empty; the only hint is a resolve_btfids warning buried in the build log. Log the offending id, the owner of the BTF and the error code.

Patch 2: bpf_testmod_init() chains twelve registrations with "ret = ret ?:", so when one of them fails there is no way to tell which call it was without instrumenting the module. Print the failing expression.

Patch 3: when test_progs is built without LLVM development libraries, every subtest that checks jited output skips with a message that is only visible under -v and does not say what to do about it. Point at the fix, and add a missing newline in the verbose-only fallback message.

One observation from building both architectures: with CONFIG_WERROR=y (the x86_64 defconfig default) resolve_btfids runs with --fatal_warnings, so the stale-pahole case fails the module build outright instead of producing a module that fails at load time. Patches 1 and 2 only help the WERROR=n path, which is what we run in our arm64 CI.

Verified on arm64 (KVM guest) and x86_64 (TCG guest) against bpf-next master b71326388a: with the patched kernel a pahole-1.30-built bpf_testmod still fails to load, but dmesg now says which kfunc and which registration failed; with pahole 1.32 it loads cleanly, and the unwind selftests still pass 42/42.

Wang Yibo (3):
bpf: report kfunc sets that fail BTF sanity check selftests/bpf: name the failing registration in bpf_testmod_init selftests/bpf: say how to fix the missing LLVM skip for jited checks

 kernel/bpf/btf.c                              |  6 ++-
 .../selftests/bpf/jit_disasm_helpers.c        |  2 +-
 .../selftests/bpf/test_kmods/bpf_testmod.c    | 41 +++++++++++--------
 tools/testing/selftests/bpf/test_loader.c     |  2 +-
 4 files changed, 32 insertions(+), 19 deletions(-)
Signature · 1 line
2.51.0
author
#1 lore raw

PATCH 1/3 bpf: report kfunc sets that fail BTF sanity check

register_btf_kfunc_id_set() returns -EINVAL when a kfunc in the set has no valid FUNC/FUNC_PROTO pair in the BTF of the registering object, but nothing is logged: the caller (typically a module init function) propagates the error, insmod fails with -EINVAL and dmesg stays silent. The only hint is a warning from resolve_btfids at build time, which is easy to miss.

This happened with bpf_testmod built by pahole 1.30: that version does not encode __int128, so the three i128 test kfuncs had no BTF func entries, their ids in the set stayed 0 and loading the module failed with -22, with nothing printed to dmesg.

Log the offending kfunc id, the object that owns the BTF and the error code when the sanity check fails.

---
 kernel/bpf/btf.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
kernel/bpf/btf.c +5 -1
index 538129ed2..af07206f9 100644
@@ -10128,8 +10128,12 @@ static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,
 	for (i = 0; i < kset->set->cnt; i++) {
 		ret = btf_check_kfunc_protos(btf, btf_relocate_id(btf, kset->set->pairs[i].id),
 					     kset->set->pairs[i].flags);
-		if (ret)
+		if (ret) {
+			pr_err("register_btf_kfunc_id_set: kfunc id %u in BTF of %s failed sanity check, ret %d\n",
+			       kset->set->pairs[i].id,
+			       kset->owner ? kset->owner->name : "vmlinux", ret);
 			goto err_out;
+		}
 	}
 
 	ret = btf_populate_kfunc_set(btf, hook, kset);
Signature · 1 line
2.51.0
#2 lore raw

Re: [PATCH bpf-next 1/3] bpf: report kfunc sets that fail BTF sanity check

TL;DR: Commit message only: it says "three i128 test kfuncs" but bpf_testmod appears to register about six; suggests fixing the count or dropping it. No code issues raised.
Quoted 1 line

bpf: report kfunc sets that fail BTF sanity check

TL;DR: Commit message only: it says "three i128 test kfuncs" but bpf_testmod appears to register about six; suggests fixing the count or dropping it. No code issues raised.

Quoted 14 lines

bpf: report kfunc sets that fail BTF sanity check

register_btf_kfunc_id_set() returns -EINVAL when a kfunc in the set has no valid FUNC/FUNC_PROTO pair in the BTF of the registering object, but nothing is logged: the caller (typically a module init function) propagates the error, insmod fails with -EINVAL and dmesg stays silent. The only hint is a warning from resolve_btfids at build time, which is easy to miss.

This happened with bpf_testmod built by pahole 1.30: that version does not encode __int128, so the three i128 test kfuncs had no BTF func entries, their ids in the set stayed 0 and loading the module failed with -22, with nothing printed to dmesg.

Quoted 4 lines

Log the offending kfunc id, the object that owns the BTF and the error code when the sanity check fails.

The commit message says "the three i128 test kfuncs", but is that count accurate?

bpf_testmod registers six i128 test kfuncs in its kfunc set:

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c:

    BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128)
    ...
    BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg)
    BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_odd)
    BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_odd_many)
    BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_stack_hole)
    BTF_ID_FLAGS(func, bpf_kfunc_call_test_i128_arg_pad)

At least five of them take or return a bare __int128, and bpf_kfunc_call_test_i128_arg_stack_hole passes it through struct prog_test_i128_arg.

Could the message either state the correct number or say "the i128 test kfuncs" instead?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39699006936

author
#3 lore raw

PATCH 2/3 selftests/bpf: name the failing registration in bpf_testmod_init

In reply to Wang Yibo

bpf_testmod_init() chains its registrations with ret = ret ?: ...; when one of them fails the module only reports the final error code, and there is no way to tell which call failed without instrumenting the code. Wrap the calls in a macro that prints the failing expression and its return value instead.

---
 .../selftests/bpf/test_kmods/bpf_testmod.c    | 41 +++++++++++--------
 1 file changed, 25 insertions(+), 16 deletions(-)
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +25 -16
index 6380f52f4..625b6dc3b 100644
@@ -2593,22 +2593,31 @@ static int bpf_testmod_init(void)
 	void **tramp;
 	int ret;
 
-	ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &bpf_testmod_common_kfunc_set);
-	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_testmod_kfunc_set);
-	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_testmod_kfunc_set);
-	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_testmod_kfunc_set);
-	ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &bpf_testmod_kfunc_set);
-	ret = ret ?: register_btf_fmodret_id_set(&bpf_testmod_trampoline_count_fmodret_set);
-	ret = ret ?: register_bpf_struct_ops(&bpf_bpf_testmod_ops, bpf_testmod_ops);
-	ret = ret ?: register_bpf_struct_ops(&bpf_testmod_ops2, bpf_testmod_ops2);
-	ret = ret ?: register_bpf_struct_ops(&bpf_testmod_ops3, bpf_testmod_ops3);
-	ret = ret ?: register_bpf_struct_ops(&testmod_st_ops, bpf_testmod_st_ops);
-	ret = ret ?: register_bpf_struct_ops(&testmod_multi_st_ops, bpf_testmod_multi_st_ops);
-	ret = ret ?: register_btf_id_dtor_kfuncs(bpf_testmod_dtors,
-						 ARRAY_SIZE(bpf_testmod_dtors),
-						 THIS_MODULE);
-	if (ret < 0)
-		return ret;
+#define TRY(expr)							\
+	do {								\
+		ret = (expr);						\
+		if (ret < 0) {						\
+			pr_err("%s: %s failed, ret %d\n",		\
+			       __func__, #expr, ret);			\
+			return ret;					\
+		}							\
+	} while (0)
+
+	TRY(register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &bpf_testmod_common_kfunc_set));
+	TRY(register_btf_kfunc_id_set(BPF_PROG_TYPE_SCHED_CLS, &bpf_testmod_kfunc_set));
+	TRY(register_btf_kfunc_id_set(BPF_PROG_TYPE_TRACING, &bpf_testmod_kfunc_set));
+	TRY(register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_testmod_kfunc_set));
+	TRY(register_btf_kfunc_id_set(BPF_PROG_TYPE_STRUCT_OPS, &bpf_testmod_kfunc_set));
+	TRY(register_btf_fmodret_id_set(&bpf_testmod_trampoline_count_fmodret_set));
+	TRY(register_bpf_struct_ops(&bpf_bpf_testmod_ops, bpf_testmod_ops));
+	TRY(register_bpf_struct_ops(&bpf_testmod_ops2, bpf_testmod_ops2));
+	TRY(register_bpf_struct_ops(&bpf_testmod_ops3, bpf_testmod_ops3));
+	TRY(register_bpf_struct_ops(&testmod_st_ops, bpf_testmod_st_ops));
+	TRY(register_bpf_struct_ops(&testmod_multi_st_ops, bpf_testmod_multi_st_ops));
+	TRY(register_btf_id_dtor_kfuncs(bpf_testmod_dtors,
+					ARRAY_SIZE(bpf_testmod_dtors),
+					THIS_MODULE));
+#undef TRY
 	if (bpf_fentry_test1(0) < 0)
 		return -EINVAL;
 	sock = NULL;
Signature · 1 line
2.51.0
#4 lore raw

Re: [PATCH bpf-next 2/3] selftests/bpf: name the failing registration in bpf_testmod_init

tools/testing/selftests/bpf/test_kmods/bpf_testmod.c static int bpf_testmod_init(void) · line 2604

Quoted 9 lines · Wang Yibo
tools/testing/selftests/bpf/test_kmods/bpf_testmod.c +9 -0
+#define TRY(expr)							\
+	do {								\
+		ret = (expr);						\
+		if (ret < 0) {						\
+			pr_err("%s: %s failed, ret %d\n",		\
+			       __func__, #expr, ret);			\
+			return ret;					\
+		}							\
+	} while (0)

pls no macros with hidden 'return'.

I also don't feel that the whole thing (patch 1 and the rest) is necessary.

pw-bot: cr

author
#5 lore raw

PATCH 3/3 selftests/bpf: say how to fix the missing LLVM skip for jited checks

In reply to Wang Yibo

When test_progs is built without LLVM development libraries, every subtest that checks jited output is skipped with only "tests are built w/o LLVM development libs" visible under -v. The message does not say that installing the libraries and rebuilding is all that is needed, and the verbose-only message in jit_disasm_helpers.c is missing a trailing newline, so it runs together with the following output.

Mention the fix in the skip message and add the newline.

---
 tools/testing/selftests/bpf/jit_disasm_helpers.c | 2 +-
 tools/testing/selftests/bpf/test_loader.c        | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/jit_disasm_helpers.c +1 -1
index 3558fe10e..6162a06b8 100644
@@ -247,7 +247,7 @@ int get_jited_program_text(int fd, char *text, size_t text_sz)
 int get_jited_program_text(int fd, char *text, size_t text_sz)
 {
 	if (env.verbosity >= VERBOSE_VERY)
-		printf("compiled w/o llvm development libraries, can't dis-assembly binary code");
+		printf("compiled w/o llvm development libraries, can't disassemble binary code\n");
 	return -EOPNOTSUPP;
 }
 
tools/testing/selftests/bpf/test_loader.c +1 -1
index 824e5912e..a45e6c820 100644
@@ -1506,7 +1506,7 @@ void run_subtest(struct test_loader *tester,
 					     tester->log_buf, tester->log_buf_sz);
 		if (err == -EOPNOTSUPP) {
 			printf("%s:SKIP: jited programs disassembly is not supported,\n", __func__);
-			printf("%s:SKIP: tests are built w/o LLVM development libs\n", __func__);
+			printf("%s:SKIP: rebuild with LLVM development libs installed\n", __func__);
 			test__skip();
 			goto tobj_cleanup;
 		}
Signature · 1 line
2.51.0
#6 lore raw

Re: [PATCH bpf-next 3/3] selftests/bpf: say how to fix the missing LLVM skip for jited checks

TL;DR: The new skip message says installing LLVM libs and rebuilding suffices, but a stale stub jit_disasm_helpers.o may be kept, so the skip could persist; a clean rebuild or SKIP_LLVM may need…
Quoted 1 line

selftests/bpf: say how to fix the missing LLVM skip for jited checks

TL;DR: The new skip message says installing LLVM libs and rebuilding suffices, but a stale stub jit_disasm_helpers.o may be kept, so the skip could persist; a clean rebuild or SKIP_LLVM may need mentioning.

tools/testing/selftests/bpf/test_loader.c void run_subtest(struct test_loader *tester, · line 1509

Quoted 10 lines
tools/testing/selftests/bpf/test_loader.c +1 -1
index 824e5912ebcb..a45e6c8202eb 100644
@@ -1506,7 +1506,7 @@ void run_subtest(struct test_loader *tester,
 					     tester->log_buf, tester->log_buf_sz);
 		if (err == -EOPNOTSUPP) {
 			printf("%s:SKIP: jited programs disassembly is not supported,\n", __func__);
-			printf("%s:SKIP: tests are built w/o LLVM development libs\n", __func__);
+			printf("%s:SKIP: rebuild with LLVM development libs installed\n", __func__);

Is "rebuild with LLVM development libs installed" enough to get the jited checks running?

The commit message frames the fix the same way, saying the old message "does not say that installing the libraries and rebuilding is all that is needed".

The stub get_jited_program_text() is chosen at compile time by -DHAVE_LLVM_SUPPORT, which the Makefile adds only through a target-specific variable:

    $(OUTPUT)/jit_disasm_helpers.o: CFLAGS += $(LLVM_CFLAGS)

jit_disasm_helpers.o is built by the generic rule:

    $(OUTPUT)/%.o: %.c

and that rule has only the .c file as a prerequisite. Nothing ties the object to the LLVM feature detection result or to the CFLAGS used.

If a user sees this message, installs the LLVM development packages and runs make again, would the existing jit_disasm_helpers.o with the stub be kept, so that the same -EOPNOTSUPP skip comes back?

A build with SKIP_LLVM=1 would also need that variable dropped, which the message does not mention.

Could the text say that a clean rebuild may be needed (removing jit_disasm_helpers.o or running make clean), or name SKIP_LLVM, instead of implying that installing the libraries and rebuilding is sufficient?

---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md

CI run summary: https://github.com/kernel-patches/bpf/actions/runs/39699006936