Skip to contentSkip to conversation

selftests/bpf: test_progs can read test lists from file

PATCH Applied Stephen Veiss 3 participants 6 messages Apr 27, 2023 – May 1, 2023

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v2 cover selftests/bpf: test_progs can read test lists from file

BPF selftests have ALLOWLIST and DENYLIST files, used to control which tests are run in CI. These files are currently parsed by a shell script. [1]

This patchset allows those files to be specified directly on the test_progs command line (eg, as -a @ALLOWLIST).

This also fixes a bug in the existing test filter code causing unnecessary duplicate top-level test filter entries to be created.

[1] https://github.com/kernel-patches/vmtest/blob/57feb460047b69f891cf4afe3cc860794a2ced17/ci/vmtest/run_selftests.sh#L21-L27

---

v2:
- error handling style changes per reviewer comments
- fdopen return value checking in test_parse_test_list_file

v1:
https://lore.kernel.org/bpf/[email protected]/

Stephen Veiss (2):
selftests/bpf: extract insert_test from parse_test_list selftests/bpf: test_progs can read test lists from file

 .../selftests/bpf/prog_tests/arg_parsing.c    |  68 ++++++
 tools/testing/selftests/bpf/test_progs.c      |  37 ++-
 tools/testing/selftests/bpf/testing_helpers.c | 225 ++++++++++++------
 tools/testing/selftests/bpf/testing_helpers.h |   3 +
 4 files changed, 249 insertions(+), 84 deletions(-)

base-commit: b580dac290822cfcbe741c0aa59c41c3fda398a8

Signature · 1 line
2.34.1
#1 lore raw

PATCH v2 1/2 selftests/bpf: extract insert_test from parse_test_list

Split the logic to insert new tests into test filter sets out from parse_test_list.

Fix the subtest insertion logic to reuse an existing top-level test filter, which prevents the creation of duplicate top-level test filters each with a single subtest.

---
 .../selftests/bpf/prog_tests/arg_parsing.c    |  13 ++
 tools/testing/selftests/bpf/testing_helpers.c | 178 ++++++++++--------
 2 files changed, 117 insertions(+), 74 deletions(-)
tools/testing/selftests/bpf/prog_tests/arg_parsing.c +13 -0
index b17bfa0e0aac..3754cd5f8c0a 100644
@@ -96,6 +96,19 @@ static void test_parse_test_list(void)
 		goto error;
 	ASSERT_OK(strcmp("*bpf_cookie*", set.tests[0].name), "test name");
 	ASSERT_OK(strcmp("*trace*", set.tests[0].subtests[0]), "subtest name");
+	free_test_filter_set(&set);
+
+	ASSERT_OK(parse_test_list("t/subtest1,t/subtest2", &set, true),
+		  "parsing");
+	if (!ASSERT_EQ(set.cnt, 1, "count of test filters"))
+		goto error;
+	if (!ASSERT_OK_PTR(set.tests, "test filters initialized"))
+		goto error;
+	if (!ASSERT_EQ(set.tests[0].subtest_cnt, 2, "subtest filters count"))
+		goto error;
+	ASSERT_OK(strcmp("t", set.tests[0].name), "test name");
+	ASSERT_OK(strcmp("subtest1", set.tests[0].subtests[0]), "subtest name");
+	ASSERT_OK(strcmp("subtest2", set.tests[0].subtests[1]), "subtest name");
 error:
 	free_test_filter_set(&set);
 }
tools/testing/selftests/bpf/testing_helpers.c +104 -74

196 lines not on this page · raw message

Signature · 1 line
2.34.1
Acked-by on v2 1/2
#2 lore raw

Re: [PATCH bpf-next v2 1/2] selftests/bpf: extract insert_test from parse_test_list

Quoted 8 lines · Stephen Veiss

Split the logic to insert new tests into test filter sets out from parse_test_list.

Fix the subtest insertion logic to reuse an existing top-level test filter, which prevents the creation of duplicate top-level test filters each with a single subtest.

#3 lore raw

PATCH v2 2/2 selftests/bpf: test_progs can read test lists from file

In reply to Stephen Veiss

Improve test selection logic when using -a/-b/-d/-t options. The list of tests to include or exclude can now be read from a file, specified as @<filename>.

The file contains one name (or wildcard pattern) per line, and comments beginning with # are ignored.

These options can be passed multiple times to read more than one file.

---

Notes:
There was a suggestion to replace remove() with close() in test_parse_test_list_file. mkstemp() doesn't unlink the temp file it creates, so this would have left a file behind. I did try to clean up the error logic here slightly instead.

 .../selftests/bpf/prog_tests/arg_parsing.c    | 55 +++++++++++++++++++
 tools/testing/selftests/bpf/test_progs.c      | 37 +++++++++----
 tools/testing/selftests/bpf/testing_helpers.c | 47 ++++++++++++++++
 tools/testing/selftests/bpf/testing_helpers.h |  3 +
 4 files changed, 132 insertions(+), 10 deletions(-)
tools/testing/selftests/bpf/prog_tests/arg_parsing.c +55 -0
index 3754cd5f8c0a..bb143de68875 100644
@@ -113,8 +113,63 @@ static void test_parse_test_list(void)
 	free_test_filter_set(&set);
 }
 
+static void test_parse_test_list_file(void)
+{
+	struct test_filter_set set;
+	char tmpfile[80];
+	FILE *fp;
+	int fd;
+
+	snprintf(tmpfile, sizeof(tmpfile), "/tmp/bpf_arg_parsing_test.XXXXXX");
+	fd = mkstemp(tmpfile);
+	if (!ASSERT_GE(fd, 0, "create tmp"))
+		return;
+
+	fp = fdopen(fd, "w");
+	if (!ASSERT_NEQ(fp, NULL, "fdopen tmp")) {
+		close(fd);
+		goto out_remove;
+	}
+
+	fprintf(fp, "# comment\n");
+	fprintf(fp, "  test_with_spaces    \n");
+	fprintf(fp, "testA/subtest    # comment\n");
+	fprintf(fp, "testB#comment with no space\n");
+	fprintf(fp, "testB # duplicate\n");
+	fprintf(fp, "testA/subtest # subtest duplicate\n");
+	fprintf(fp, "testA/subtest2\n");
+	fprintf(fp, "testC_no_eof_newline");
+	fflush(fp);
+
+	if (!ASSERT_OK(ferror(fp), "prepare tmp"))
+		goto out_fclose;
+
+	init_test_filter_set(&set);
+
+	ASSERT_OK(parse_test_list_file(tmpfile, &set, true), "parse file");
+
+	ASSERT_EQ(set.cnt, 4, "test  count");
+	ASSERT_OK(strcmp("test_with_spaces", set.tests[0].name), "test 0 name");
+	ASSERT_EQ(set.tests[0].subtest_cnt, 0, "test 0 subtest count");
+	ASSERT_OK(strcmp("testA", set.tests[1].name), "test 1 name");
+	ASSERT_EQ(set.tests[1].subtest_cnt, 2, "test 1 subtest count");
+	ASSERT_OK(strcmp("subtest", set.tests[1].subtests[0]), "test 1 subtest 0");
+	ASSERT_OK(strcmp("subtest2", set.tests[1].subtests[1]), "test 1 subtest 1");
+	ASSERT_OK(strcmp("testB", set.tests[2].name), "test 2 name");
+	ASSERT_OK(strcmp("testC_no_eof_newline", set.tests[3].name), "test 3 name");
+
+	free_test_filter_set(&set);
+
+out_fclose:
+	fclose(fp);
+out_remove:
+	remove(tmpfile);
+}
+
 void test_arg_parsing(void)
 {
 	if (test__start_subtest("test_parse_test_list"))
 		test_parse_test_list();
+	if (test__start_subtest("test_parse_test_list_file"))
+		test_parse_test_list_file();
 }
tools/testing/selftests/bpf/test_progs.c +27 -10
index ea82921110da..793689dcc170 100644
@@ -714,7 +714,13 @@ static struct test_state test_states[ARRAY_SIZE(prog_test_defs)];
 
 const char *argp_program_version = "test_progs 0.1";
 const char *argp_program_bug_address = "<[email protected]>";
-static const char argp_program_doc[] = "BPF selftests test runner";
+static const char argp_program_doc[] =
+"BPF selftests test runner\v"
+"Options accepting the NAMES parameter take either a comma-separated list\n"
+"of test names, or a filename prefixed with @. The file contains one name\n"
+"(or wildcard pattern) per line, and comments beginning with # are ignored.\n"
+"\n"
+"These options can be passed repeatedly to read multiple files.\n";
 
 enum ARG_KEYS {
 	ARG_TEST_NUM = 'n',
@@ -797,6 +803,7 @@ extern int extra_prog_load_log_flags;
 static error_t parse_arg(int key, char *arg, struct argp_state *state)
 {
 	struct test_env *env = state->input;
+	int err = 0;
 
 	switch (key) {
 	case ARG_TEST_NUM: {
@@ -821,18 +828,28 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state)
 	}
 	case ARG_TEST_NAME_GLOB_ALLOWLIST:
 	case ARG_TEST_NAME: {
-		if (parse_test_list(arg,
-				    &env->test_selector.whitelist,
-				    key == ARG_TEST_NAME_GLOB_ALLOWLIST))
-			return -ENOMEM;
+		if (arg[0] == '@')
+			err = parse_test_list_file(arg + 1,
+						   &env->test_selector.whitelist,
+						   key == ARG_TEST_NAME_GLOB_ALLOWLIST);
+		else
+			err = parse_test_list(arg,
+					      &env->test_selector.whitelist,
+					      key == ARG_TEST_NAME_GLOB_ALLOWLIST);
+
 		break;
 	}
 	case ARG_TEST_NAME_GLOB_DENYLIST:
 	case ARG_TEST_NAME_BLACKLIST: {
-		if (parse_test_list(arg,
-				    &env->test_selector.blacklist,
-				    key == ARG_TEST_NAME_GLOB_DENYLIST))
-			return -ENOMEM;
+		if (arg[0] == '@')
+			err = parse_test_list_file(arg + 1,
+						   &env->test_selector.blacklist,
+						   key == ARG_TEST_NAME_GLOB_DENYLIST);
+		else
+			err = parse_test_list(arg,
+					      &env->test_selector.blacklist,
+					      key == ARG_TEST_NAME_GLOB_DENYLIST);
+
 		break;
 	}
 	case ARG_VERIFIER_STATS:
@@ -900,7 +917,7 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state)
 	default:
 		return ARGP_ERR_UNKNOWN;
 	}
-	return 0;
+	return err;
 }
 
 /*
tools/testing/selftests/bpf/testing_helpers.c +47 -0

59 lines not on this page · raw message

tools/testing/selftests/bpf/testing_helpers.h +3 -0
index eb8790f928e4..98f09bbae86f 100644
@@ -20,5 +20,8 @@ struct test_filter_set;
 int parse_test_list(const char *s,
 		    struct test_filter_set *test_set,
 		    bool is_glob_pattern);
+int parse_test_list_file(const char *path,
+			 struct test_filter_set *test_set,
+			 bool is_glob_pattern);
 
 __u64 read_perf_max_sample_freq(void);
Signature · 1 line
2.34.1
Acked-by on v2 2/2
#4 lore raw
Quoted 10 lines · Stephen Veiss

Improve test selection logic when using -a/-b/-d/-t options. The list of tests to include or exclude can now be read from a file, specified as @<filename>.

The file contains one name (or wildcard pattern) per line, and comments beginning with # are ignored.

These options can be passed multiple times to read more than one file.

#5 lore raw
In reply to Stephen Veiss
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:

Hello:

This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:

Quoted 8 lines · On Thu, 27 Apr 2023 15:53:31 -0700 you wrote:

BPF selftests have ALLOWLIST and DENYLIST files, used to control which tests are run in CI. These files are currently parsed by a shell script. [1]

This patchset allows those files to be specified directly on the test_progs command line (eg, as -a @ALLOWLIST).

[...]

Here is the summary with links:
- [bpf-next,v2,1/2] selftests/bpf: extract insert_test from parse_test_list

    https://git.kernel.org/bpf/bpf-next/c/0a5c0de8b66f
  - [bpf-next,v2,2/2] selftests/bpf: test_progs can read test lists from file
    https://git.kernel.org/bpf/bpf-next/c/64276f01dce8

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html