Skip to contentSkip to conversation

selftests/bpf: Fix prog_tests framework self-tests

PATCH Applied KaFai Wan 2 participants 5 messages Oct 12 – Oct 16

Subsystems (3)
lore
Share & tools
lore

KaFai Wan

author
#0 lore raw

cover selftests/bpf: Fix prog_tests framework self-tests

Three prog_tests framework self-tests fail depending on how test_progs is invoked:

- ./test_progs -t prog_tests_framework/test_fail -v test_prog_tests_framework:FAIL:skip_cnt_check unexpected skip_cnt_check: actual 0 != expected 2

- ./test_progs -t prog_tests_framework_expected_msgs -v Caught signal #11!
Stack trace:
./test_progs(crash_handler+0x38)[0x55620006fbb0]

- ./test_progs -t prog_tests_framework_compare_text -v test_prog_tests_framework_compare_text:PASS:mismatch_rc 0 nsec Caught signal #11!
Stack trace:
./test_progs(crash_handler+0x38)[0x564ddbfcfbb0]

Patch #1 counts the subtests that actually ran, so the skip and error count checks no longer fail when -t or -n filters out some subtests.

Patches #2 and #3 skip the subtests that inspect the log buffer when the output is not captured, such as with -v in non-parallel mode, instead of crashing on a NULL log_buf.

---
KaFai Wan (3):
selftests/bpf: Fix prog_tests framework subtest count checks selftests/bpf: Skip expected_msgs subtests when stdout is not saved selftests/bpf: Skip compare_text subtest when stdout is not saved

 .../bpf/prog_tests/prog_tests_framework.c     | 34 ++++++++++++++-----
 1 file changed, 25 insertions(+), 9 deletions(-)
Signature · 1 line
2.43.0
author
#1 lore raw

PATCH 1/3 selftests/bpf: Fix prog_tests framework subtest count checks

The hardcoded expected skip_cnt/error_cnt in test_prog_tests_framework() mismatch the real count when the binary is run with -t or -n filters, since some subtests are not selected. Count the executed subtests and compare against that value instead.

Fixes 2324257dbd68 ("selftests/bpf: Refactor prog_tests logging and test execution")
---
 .../selftests/bpf/prog_tests/prog_tests_framework.c    | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
tools/testing/selftests/bpf/prog_tests/prog_tests_framework.c +8 -2
index d111fe105447..53347f379924 100644
@@ -13,6 +13,7 @@ static void clear_test_state(struct test_state *state)
 void test_prog_tests_framework(void)
 {
 	struct test_state *state = env.test_state;
+	int subtest_cnt;
 
 	/* in all the ASSERT calls below we need to return on the first
 	 * error due to the fact that we are cleaning the test state after
@@ -30,25 +31,30 @@ void test_prog_tests_framework(void)
 		return;
 	clear_test_state(state);
 
+	subtest_cnt = 0;
 	if (test__start_subtest("test_skip_subtest")) {
 		test__skip();
 		test__end_subtest();
+		subtest_cnt++;
 	}
 	if (test__start_subtest("test_skip_subtest")) {
 		test__skip();
 		test__end_subtest();
+		subtest_cnt++;
 	}
-	if (!ASSERT_EQ(state->skip_cnt, 2, "skip_cnt_check"))
+	if (!ASSERT_EQ(state->skip_cnt, subtest_cnt, "skip_cnt_check"))
 		return;
 	if (!ASSERT_EQ(state->subtest_num, 3, "subtest_num_check"))
 		return;
 	clear_test_state(state);
 
+	subtest_cnt = 0;
 	if (test__start_subtest("test_fail_subtest")) {
 		test__fail();
 		test__end_subtest();
+		subtest_cnt++;
 	}
-	if (!ASSERT_EQ(state->error_cnt, 1, "error_cnt_check"))
+	if (!ASSERT_EQ(state->error_cnt, subtest_cnt, "error_cnt_check"))
 		return;
 	if (!ASSERT_EQ(state->subtest_num, 4, "subtest_num_check"))
 		return;
Signature · 1 line
2.43.0
author
#2 lore raw

PATCH 2/3 selftests/bpf: Skip expected_msgs subtests when stdout is not saved

In reply to KaFai Wan

stdio_hijack_init() does not set subtest_state->stdout_saved when running with -v in non-parallel mode, since output goes straight to stdout. In that case subtest_state->log_buf is left NULL, so the expected-messages subtests crash when inspecting it. Skip them when stdout was not saved.

Fixes 34c513be3dad ("selftests/bpf: __not_msg() tag for test_loader framework")
---
 .../testing/selftests/bpf/prog_tests/prog_tests_framework.c | 6 ++++++
 1 file changed, 6 insertions(+)
tools/testing/selftests/bpf/prog_tests/prog_tests_framework.c +6 -0
index 53347f379924..b4cd7c7ff439 100644
@@ -169,6 +169,12 @@ void test_prog_tests_framework_expected_msgs(void)
 
 	for (i = 0; i < ARRAY_SIZE(cases); i++) {
 		if (test__start_subtest(cases[i].name)) {
+			if (!env.subtest_state->stdout_saved) {
+				test__skip();
+				test__end_subtest();
+				continue;
+			}
+
 			error_cnt = env.subtest_state->error_cnt;
 			msgs.patterns = cases[i].pats;
 			msgs.cnt = 0;
Signature · 1 line
2.43.0
author
#3 lore raw

PATCH 3/3 selftests/bpf: Skip compare_text subtest when stdout is not saved

In reply to KaFai Wan

stdio_hijack_init() does not set subtest_state->stdout_saved when running with -v in non-parallel mode, since output goes straight to stdout. In that case subtest_state->log_buf is left NULL, so the compare_text_mismatch subtest crashes when inspecting it. Skip it when stdout was not saved.

Fixes bd71904697c6 ("selftests/bpf: Add compare_text_to_expected() helper")
---
 .../bpf/prog_tests/prog_tests_framework.c      | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)
tools/testing/selftests/bpf/prog_tests/prog_tests_framework.c +11 -7
index b4cd7c7ff439..ddb6cdc1579f 100644
@@ -203,14 +203,18 @@ void test_prog_tests_framework_compare_text(void)
 	}
 
 	if (test__start_subtest("compare_text_mismatch")) {
-		err = compare_text_to_expected("line two\n", "line one\n");
-		fflush(stdout);
+		if (!env.subtest_state->stdout_saved) {
+			test__skip();
+		} else {
+			err = compare_text_to_expected("line two\n", "line one\n");
+			fflush(stdout);
 
-		ASSERT_EQ(err, -1, "mismatch_rc");
-		ASSERT_HAS_SUBSTR(env.subtest_state->log_buf, "-line one",
-				  "diff_has_expected");
-		ASSERT_HAS_SUBSTR(env.subtest_state->log_buf, "+line two",
-				  "diff_has_actual");
+			ASSERT_EQ(err, -1, "mismatch_rc");
+			ASSERT_HAS_SUBSTR(env.subtest_state->log_buf, "-line one",
+					  "diff_has_expected");
+			ASSERT_HAS_SUBSTR(env.subtest_state->log_buf, "+line two",
+					  "diff_has_actual");
+		}
 		test__end_subtest();
 	}
 }
Signature · 1 line
2.43.0
#4 lore raw
In reply to KaFai Wan
This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <[email protected]>:

Quoted 8 lines · On Mon, 12 Oct 2026 22:01:58 +0800 you wrote:

Three prog_tests framework self-tests fail depending on how test_progs is invoked:

- ./test_progs -t prog_tests_framework/test_fail -v test_prog_tests_framework:FAIL:skip_cnt_check unexpected skip_cnt_check: actual 0 != expected 2

[...]

Here is the summary with links:
- [bpf-next,1/3] selftests/bpf: Fix prog_tests framework subtest count checks

    https://git.kernel.org/bpf/bpf-next/c/c87652656a55
  - [bpf-next,2/3] selftests/bpf: Skip expected_msgs subtests when stdout is not saved
    https://git.kernel.org/bpf/bpf-next/c/cc4b1c161d92
  - [bpf-next,3/3] selftests/bpf: Skip compare_text subtest when stdout is not saved
    https://git.kernel.org/bpf/bpf-next/c/7ebf808caa76

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html