selftests/bpf: test_progs can read test lists from file
Subsystems (3)
v2 cover selftests/bpf: test_progs can read test lists from file
BPF selftests have ALLOWLIST and DENYLIST files, used to control which tests are run in CI. These files are currently parsed by a shell script. [1]
This patchset allows those files to be specified directly on the test_progs command line (eg, as -a @ALLOWLIST).
This also fixes a bug in the existing test filter code causing unnecessary duplicate top-level test filter entries to be created.
---
v2:
- error handling style changes per reviewer comments
- fdopen return value checking in test_parse_test_list_file
v1:
https://lore.kernel.org/bpf/[email protected]/
Stephen Veiss (2):
selftests/bpf: extract insert_test from parse_test_list selftests/bpf: test_progs can read test lists from file
.../selftests/bpf/prog_tests/arg_parsing.c | 68 ++++++ tools/testing/selftests/bpf/test_progs.c | 37 ++- tools/testing/selftests/bpf/testing_helpers.c | 225 ++++++++++++------ tools/testing/selftests/bpf/testing_helpers.h | 3 + 4 files changed, 249 insertions(+), 84 deletions(-)
base-commit: b580dac290822cfcbe741c0aa59c41c3fda398a8
Signature · 1 line
2.34.1
PATCH v2 1/2 selftests/bpf: extract insert_test from parse_test_list
Split the logic to insert new tests into test filter sets out from parse_test_list.
Fix the subtest insertion logic to reuse an existing top-level test filter, which prevents the creation of duplicate top-level test filters each with a single subtest.
--- .../selftests/bpf/prog_tests/arg_parsing.c | 13 ++ tools/testing/selftests/bpf/testing_helpers.c | 178 ++++++++++-------- 2 files changed, 117 insertions(+), 74 deletions(-)
tools/testing/selftests/bpf/prog_tests/arg_parsing.c +13 -0
@@ -96,6 +96,19 @@ static void test_parse_test_list(void) goto error;ASSERT_OK (strcmp ("*bpf_cookie*" , set.tests[0 ].name),"test name" );ASSERT_OK (strcmp ("*trace*" , set.tests[0 ].subtests[0 ]),"subtest name" ); +free_test_filter_set (&set); + +ASSERT_OK (parse_test_list ("t/subtest1,t/subtest2" , &set, true), +"parsing" ); +if (!ASSERT_EQ (set.cnt,1 ,"count of test filters" )) + goto error; +if (!ASSERT_OK_PTR (set.tests,"test filters initialized" )) + goto error; +if (!ASSERT_EQ (set.tests[0 ].subtest_cnt,2 ,"subtest filters count" )) + goto error; +ASSERT_OK (strcmp ("t" , set.tests[0 ].name),"test name" ); +ASSERT_OK (strcmp ("subtest1" , set.tests[0 ].subtests[0 ]),"subtest name" ); +ASSERT_OK (strcmp ("subtest2" , set.tests[0 ].subtests[1 ]),"subtest name" ); error:free_test_filter_set (&set); }
tools/testing/selftests/bpf/testing_helpers.c +104 -74
196 lines not on this page · raw message
Signature · 1 line
2.34.1
Re: [PATCH bpf-next v2 1/2] selftests/bpf: extract insert_test from parse_test_list
Quoted 8 lines · Stephen Veiss
Split the logic to insert new tests into test filter sets out from
parse_test_list.Fix the subtest insertion logic to reuse an existing top-level test filter, which prevents the creation of duplicate top-level test filters each with a single subtest.
Signed-off-by Stephen Veiss <[email protected]>
PATCH v2 2/2 selftests/bpf: test_progs can read test lists from file
In reply to Stephen VeissImprove test selection logic when using -a/-b/-d/-t options. The list of tests to include or exclude can now be read from a file, specified as @<filename>.
The file contains one name (or wildcard pattern) per line, and comments beginning with # are ignored.
These options can be passed multiple times to read more than one file.
---
Notes:
There was a suggestion to replace remove() with close() in test_parse_test_list_file. mkstemp() doesn't unlink the temp file it creates, so this would have left a file behind. I did try to clean up the error logic here slightly instead.
.../selftests/bpf/prog_tests/arg_parsing.c | 55 +++++++++++++++++++ tools/testing/selftests/bpf/test_progs.c | 37 +++++++++---- tools/testing/selftests/bpf/testing_helpers.c | 47 ++++++++++++++++ tools/testing/selftests/bpf/testing_helpers.h | 3 + 4 files changed, 132 insertions(+), 10 deletions(-)
tools/testing/selftests/bpf/prog_tests/arg_parsing.c +55 -0
@@ -113,8 +113,63 @@ static void test_parse_test_list(void)free_test_filter_set (&set); } +static void test_parse_test_list_file (void ) +{ +struct test_filter_set set; +char tmpfile[80 ]; +FILE *fp; +int fd; + +snprintf (tmpfile,sizeof (tmpfile),"/tmp/bpf_arg_parsing_test.XXXXXX" ); + fd =mkstemp (tmpfile); +if (!ASSERT_GE (fd,0 ,"create tmp" )) +return ; + + fp =fdopen (fd,"w" ); +if (!ASSERT_NEQ (fp,NULL ,"fdopen tmp" )) { +close (fd); + goto out_remove; + } + +fprintf (fp,"# comment\n" ); +fprintf (fp," test_with_spaces \n" ); +fprintf (fp,"testA/subtest # comment\n" ); +fprintf (fp,"testB#comment with no space\n" ); +fprintf (fp,"testB # duplicate\n" ); +fprintf (fp,"testA/subtest # subtest duplicate\n" ); +fprintf (fp,"testA/subtest2\n" ); +fprintf (fp,"testC_no_eof_newline" ); +fflush (fp); + +if (!ASSERT_OK (ferror (fp),"prepare tmp" )) + goto out_fclose; + +init_test_filter_set (&set); + +ASSERT_OK (parse_test_list_file (tmpfile, &set, true),"parse file" ); + +ASSERT_EQ (set.cnt,4 ,"test count" ); +ASSERT_OK (strcmp ("test_with_spaces" , set.tests[0 ].name),"test 0 name" ); +ASSERT_EQ (set.tests[0 ].subtest_cnt,0 ,"test 0 subtest count" ); +ASSERT_OK (strcmp ("testA" , set.tests[1 ].name),"test 1 name" ); +ASSERT_EQ (set.tests[1 ].subtest_cnt,2 ,"test 1 subtest count" ); +ASSERT_OK (strcmp ("subtest" , set.tests[1 ].subtests[0 ]),"test 1 subtest 0" ); +ASSERT_OK (strcmp ("subtest2" , set.tests[1 ].subtests[1 ]),"test 1 subtest 1" ); +ASSERT_OK (strcmp ("testB" , set.tests[2 ].name),"test 2 name" ); +ASSERT_OK (strcmp ("testC_no_eof_newline" , set.tests[3 ].name),"test 3 name" ); + +free_test_filter_set (&set); + +out_fclose: +fclose (fp); +out_remove: +remove (tmpfile); +} +void test_arg_parsing (void ) {if (test__start_subtest ("test_parse_test_list" ))test_parse_test_list (); +if (test__start_subtest ("test_parse_test_list_file" )) +test_parse_test_list_file (); }
tools/testing/selftests/bpf/test_progs.c +27 -10
@@ -714,7 +714,13 @@ static struct test_state test_states[ARRAY_SIZE(prog_test_defs)];const char *argp_program_version ="test_progs 0.1" ;const char *argp_program_bug_address ="<[email protected]>" ; -static const char argp_program_doc[] ="BPF selftests test runner" ; +static const char argp_program_doc[] = +"BPF selftests test runner\v" +"Options accepting the NAMES parameter take either a comma-separated list\n" +"of test names, or a filename prefixed with @. The file contains one name\n" +"(or wildcard pattern) per line, and comments beginning with # are ignored.\n" +"\n" +"These options can be passed repeatedly to read multiple files.\n" ;enum ARG_KEYS {ARG_TEST_NUM ='n' ,
@@ -797,6 +803,7 @@ extern int extra_prog_load_log_flags;static error_t parse_arg (int key,char *arg,struct argp_state *state) {struct test_env *env = state->input; +int err =0 ;switch (key) {case ARG_TEST_NUM : {
@@ -821,18 +828,28 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state) }case ARG_TEST_NAME_GLOB_ALLOWLIST :case ARG_TEST_NAME : { -if (parse_test_list (arg, - &env->test_selector.whitelist, - key ==ARG_TEST_NAME_GLOB_ALLOWLIST )) -return -ENOMEM ; +if (arg[0 ] =='@' ) + err =parse_test_list_file (arg +1 , + &env->test_selector.whitelist, + key ==ARG_TEST_NAME_GLOB_ALLOWLIST ); +else + err =parse_test_list (arg, + &env->test_selector.whitelist, + key ==ARG_TEST_NAME_GLOB_ALLOWLIST ); +break ; }case ARG_TEST_NAME_GLOB_DENYLIST :case ARG_TEST_NAME_BLACKLIST : { -if (parse_test_list (arg, - &env->test_selector.blacklist, - key ==ARG_TEST_NAME_GLOB_DENYLIST )) -return -ENOMEM ; +if (arg[0 ] =='@' ) + err =parse_test_list_file (arg +1 , + &env->test_selector.blacklist, + key ==ARG_TEST_NAME_GLOB_DENYLIST ); +else + err =parse_test_list (arg, + &env->test_selector.blacklist, + key ==ARG_TEST_NAME_GLOB_DENYLIST ); +break ; }case ARG_VERIFIER_STATS :
@@ -900,7 +917,7 @@ static error_t parse_arg(int key, char *arg, struct argp_state *state)default :return ARGP_ERR_UNKNOWN ; } -return 0 ; +return err; }/*
tools/testing/selftests/bpf/testing_helpers.c +47 -0
59 lines not on this page · raw message
tools/testing/selftests/bpf/testing_helpers.h +3 -0
@@ -20,5 +20,8 @@ struct test_filter_set;int parse_test_list (const char *s,struct test_filter_set *test_set,bool is_glob_pattern); +int parse_test_list_file (const char *path, +struct test_filter_set *test_set, +bool is_glob_pattern);__u64 read_perf_max_sample_freq (void );
Signature · 1 line
2.34.1
Quoted 10 lines · Stephen Veiss
Improve test selection logic when using -a/-b/-d/-t options. The list of tests to include or exclude can now be read from a file, specified as @<filename>.
The file contains one name (or wildcard pattern) per line, and comments beginning with # are ignored.
These options can be passed multiple times to read more than one file.
Signed-off-by Stephen Veiss <[email protected]>
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Hello:
This series was applied to bpf/bpf-next.git (master) by Andrii Nakryiko <[email protected]>:
Quoted 8 lines · On Thu, 27 Apr 2023 15:53:31 -0700 you wrote:
BPF selftests have ALLOWLIST and DENYLIST files, used to control which tests are run in CI. These files are currently parsed by a shell script. [1]
This patchset allows those files to be specified directly on the
test_progscommand line (eg, as -a @ALLOWLIST).[...]
Here is the summary with links:
- [bpf-next,v2,1/2] selftests/bpf: extract insert_test from parse_test_list
https://git.kernel.org/bpf/bpf-next/c/0a5c0de8b66f - [bpf-next,v2,2/2] selftests/bpf: test_progs can read test lists from file https://git.kernel.org/bpf/bpf-next/c/64276f01dce8
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html