Skip to contentSkip to conversation

Parallelize verif_scale selftests

PATCH In review Andrii Nakryiko 4 participants 11 messages Oct 22, 2021 – Oct 26, 2021

Subsystems (4)
lore
Share & tools
lore

Andrii Nakryiko

author maintainer
#0 lore raw

cover Parallelize verif_scale selftests

Reduce amount of waiting time when running test_progs in parallel mode (-j) by splitting bpf_verif_scale selftests into multiple tests. Previously it was structured as a test with multiple subtests, but subtests are not easily parallelizable with test_progs' infra. Also in practice each scale subtest is really an independent test with nothing shared across all substest.

This patch set changes how test_progs test discovery works. Now it is possible to define multiple tests within a single source code file. One of the patches also marks tc_redirect selftests as serial, because it's extremely harmful to the test system when run in parallel mode.

Andrii Nakryiko (4):
selftests/bpf: normalize selftest entry points selftests/bpf: support multiple tests per file selftests/bpf: mark tc_redirect selftest as serial selftests/bpf: split out bpf_verif_scale selftests into multiple tests

 tools/testing/selftests/bpf/Makefile          |   7 +-
 .../bpf/prog_tests/bpf_verif_scale.c          | 220 ++++++++++++------
 .../selftests/bpf/prog_tests/btf_dump.c       |   2 +-
 .../selftests/bpf/prog_tests/resolve_btfids.c |  10 +-
 .../selftests/bpf/prog_tests/signal_pending.c |   2 +-
 .../selftests/bpf/prog_tests/snprintf.c       |   4 +-
 .../selftests/bpf/prog_tests/tc_redirect.c    |   2 +-
 .../bpf/prog_tests/xdp_adjust_tail.c          |   6 +-
 .../bpf/prog_tests/xdp_devmap_attach.c        |   4 +-
 9 files changed, 169 insertions(+), 88 deletions(-)
Signature · 1 line
2.30.2
author maintainer
#1 lore raw

PATCH 1/4 selftests/bpf: normalize selftest entry points

Ensure that all test entry points are global void functions with no input arguments. Mark few subtest entry points as static.

---
 tools/testing/selftests/bpf/prog_tests/btf_dump.c      |  2 +-
 .../testing/selftests/bpf/prog_tests/resolve_btfids.c  | 10 ++++------
 .../testing/selftests/bpf/prog_tests/signal_pending.c  |  2 +-
 tools/testing/selftests/bpf/prog_tests/snprintf.c      |  4 ++--
 .../testing/selftests/bpf/prog_tests/xdp_adjust_tail.c |  6 +++---
 .../selftests/bpf/prog_tests/xdp_devmap_attach.c       |  4 ++--
 6 files changed, 13 insertions(+), 15 deletions(-)
tools/testing/selftests/bpf/prog_tests/btf_dump.c +1 -1
index 12f457b6786d..d8961054c05b 100644
@@ -133,7 +133,7 @@ static char *dump_buf;
 static size_t dump_buf_sz;
 static FILE *dump_buf_file;
 
-void test_btf_dump_incremental(void)
+static void test_btf_dump_incremental(void)
 {
 	struct btf *btf = NULL;
 	struct btf_dump *d = NULL;
tools/testing/selftests/bpf/prog_tests/resolve_btfids.c +4 -6
index f62361306f6d..c7cb25334d78 100644
@@ -117,14 +117,14 @@ static int resolve_symbols(void)
 	return 0;
 }
 
-int test_resolve_btfids(void)
+void test_resolve_btfids(void)
 {
 	__u32 *test_list, *test_lists[] = { test_list_local, test_list_global };
 	unsigned int i, j;
 	int ret = 0;
 
 	if (resolve_symbols())
-		return -1;
+		return;
 
 	/* Check BTF_ID_LIST(test_list_local) and
 	 * BTF_ID_LIST_GLOBAL(test_list_global) IDs
@@ -138,7 +138,7 @@ int test_resolve_btfids(void)
 				    test_symbols[i].name,
 				    test_list[i], test_symbols[i].id);
 			if (ret)
-				return ret;
+				return;
 		}
 	}
 
@@ -161,9 +161,7 @@ int test_resolve_btfids(void)
 
 		if (i > 0) {
 			if (!ASSERT_LE(test_set.ids[i - 1], test_set.ids[i], "sort_check"))
-				return -1;
+				return;
 		}
 	}
-
-	return ret;
 }
tools/testing/selftests/bpf/prog_tests/signal_pending.c +1 -1
index dfcbddcbe4d3..fdfdcff6cbef 100644
@@ -42,7 +42,7 @@ static void test_signal_pending_by_type(enum bpf_prog_type prog_type)
 	signal(SIGALRM, SIG_DFL);
 }
 
-void test_signal_pending(enum bpf_prog_type prog_type)
+void test_signal_pending(void)
 {
 	test_signal_pending_by_type(BPF_PROG_TYPE_SOCKET_FILTER);
 	test_signal_pending_by_type(BPF_PROG_TYPE_FLOW_DISSECTOR);
tools/testing/selftests/bpf/prog_tests/snprintf.c +2 -2
index 8fd1b4b29a0e..394ebfc3bbf3 100644
@@ -33,7 +33,7 @@
 
 #define EXP_NO_BUF_RET 29
 
-void test_snprintf_positive(void)
+static void test_snprintf_positive(void)
 {
 	char exp_addr_out[] = EXP_ADDR_OUT;
 	char exp_sym_out[]  = EXP_SYM_OUT;
@@ -103,7 +103,7 @@ static int load_single_snprintf(char *fmt)
 	return ret;
 }
 
-void test_snprintf_negative(void)
+static void test_snprintf_negative(void)
 {
 	ASSERT_OK(load_single_snprintf("valid %d"), "valid usage");
 
tools/testing/selftests/bpf/prog_tests/xdp_adjust_tail.c +3 -3
index d5c98f2cb12f..f529e3c923ae 100644
@@ -2,7 +2,7 @@
 #include <test_progs.h>
 #include <network_helpers.h>
 
-void test_xdp_adjust_tail_shrink(void)
+static void test_xdp_adjust_tail_shrink(void)
 {
 	const char *file = "./test_xdp_adjust_tail_shrink.o";
 	__u32 duration, retval, size, expect_sz;
@@ -30,7 +30,7 @@ void test_xdp_adjust_tail_shrink(void)
 	bpf_object__close(obj);
 }
 
-void test_xdp_adjust_tail_grow(void)
+static void test_xdp_adjust_tail_grow(void)
 {
 	const char *file = "./test_xdp_adjust_tail_grow.o";
 	struct bpf_object *obj;
@@ -58,7 +58,7 @@ void test_xdp_adjust_tail_grow(void)
 	bpf_object__close(obj);
 }
 
-void test_xdp_adjust_tail_grow2(void)
+static void test_xdp_adjust_tail_grow2(void)
 {
 	const char *file = "./test_xdp_adjust_tail_grow.o";
 	char buf[4096]; /* avoid segfault: large buf to hold grow results */
tools/testing/selftests/bpf/prog_tests/xdp_devmap_attach.c +2 -2
index d4e9a9972a67..3079d5568f8f 100644
@@ -8,7 +8,7 @@
 
 #define IFINDEX_LO 1
 
-void test_xdp_with_devmap_helpers(void)
+static void test_xdp_with_devmap_helpers(void)
 {
 	struct test_xdp_with_devmap_helpers *skel;
 	struct bpf_prog_info info = {};
@@ -60,7 +60,7 @@ void test_xdp_with_devmap_helpers(void)
 	test_xdp_with_devmap_helpers__destroy(skel);
 }
 
-void test_neg_xdp_devmap_helpers(void)
+static void test_neg_xdp_devmap_helpers(void)
 {
 	struct test_xdp_devmap_helpers *skel;
 
Signature · 1 line
2.30.2
author maintainer
#2 lore raw

PATCH 2/4 selftests/bpf: support multiple tests per file

In reply to Andrii Nakryiko

Revamp how test discovery works for test_progs and allow multiple test entries per file. Any global void function with no arguments and serial_test_ or test_ prefix is considered a test.

---
 tools/testing/selftests/bpf/Makefile | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/Makefile +3 -4
index 498222543c37..ac47cf9760fc 100644
@@ -421,10 +421,9 @@ ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),)
 $(TRUNNER_TESTS_DIR)-tests-hdr := y
 $(TRUNNER_TESTS_HDR): $(TRUNNER_TESTS_DIR)/*.c
 	$$(call msg,TEST-HDR,$(TRUNNER_BINARY),$$@)
-	$$(shell ( cd $(TRUNNER_TESTS_DIR);				\
-		  echo '/* Generated header, do not edit */';		\
-		  ls *.c 2> /dev/null |					\
-			sed -e 's@\([^\.]*\)\.c@DEFINE_TEST(\1)@';	\
+	$$(shell (echo '/* Generated header, do not edit */';					\
+		  sed -n -E 's/^void (serial_)?test_([a-zA-Z0-9_]+)\((void)?\).*/DEFINE_TEST(\2)/p'	\
+			$(TRUNNER_TESTS_DIR)/*.c | sort ;	\
 		 ) > $$@)
 endif
 
Signature · 1 line
2.30.2
#3 lore raw

Re: [PATCH bpf-next 2/4] selftests/bpf: support multiple tests per file

Quoted 15 lines · Andrii Nakryiko

Revamp how test discovery works for test_progs and allow multiple test entries per file. Any global void function with no arguments and serial_test_ or test_ prefix is considered a test.

---
 tools/testing/selftests/bpf/Makefile | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/Makefile +0 -0
index 498222543c37..ac47cf9760fc 100644

tools/testing/selftests/bpf/Makefile ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),) · line 425

Quoted 9 lines
tools/testing/selftests/bpf/Makefile +2 -4
@@ -421,10 +421,9 @@ ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),)
 $(TRUNNER_TESTS_DIR)-tests-hdr := y
 $(TRUNNER_TESTS_HDR): $(TRUNNER_TESTS_DIR)/*.c
        $$(call msg,TEST-HDR,$(TRUNNER_BINARY),$$@)
-       $$(shell ( cd $(TRUNNER_TESTS_DIR);                             \
-                 echo '/* Generated header, do not edit */';           \
-                 ls *.c 2> /dev/null |                                 \
-                       sed -e 's@\([^\.]*\)\.c@DEFINE_TEST(\1)@';      \
+       $$(shell (echo '/* Generated header, do not edit */';                                   \
+                 sed -n -E 's/^void (serial_)?test_([a-zA-Z0-9_]+)\((void)?\).*/DEFINE_TEST(\2)/p'     \

probably not that important : allow \s* before void and after void. Or, maybe we can just (?!static) instead of anchoring to line start.

tools/testing/selftests/bpf/Makefile ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),) · line 426

Quoted 1 line
tools/testing/selftests/bpf/Makefile +1 -0
+                       $(TRUNNER_TESTS_DIR)/*.c | sort ;       \

to be super safe : maybe add a check here to ensure each file contains at least one test function.

Quoted 6 lines
tools/testing/selftests/bpf/Makefile +0 -1
                 ) > $$@)
 endif
 
--

2.30.2

#4 lore raw

Re: [PATCH bpf-next 2/4] selftests/bpf: support multiple tests per file

Quoted 22 lines · On Mon, Oct 25, 2021 at 1:13 PM [email protected] <[email protected]> wrote:
Quoted 20 lines · Andrii Nakryiko

Revamp how test discovery works for test_progs and allow multiple test entries per file. Any global void function with no arguments and serial_test_ or test_ prefix is considered a test.

---
 tools/testing/selftests/bpf/Makefile | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/Makefile +0 -2
index 498222543c37..ac47cf9760fc 100644
@@ -421,10 +421,9 @@ ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),)
 $(TRUNNER_TESTS_DIR)-tests-hdr := y
 $(TRUNNER_TESTS_HDR): $(TRUNNER_TESTS_DIR)/*.c
        $$(call msg,TEST-HDR,$(TRUNNER_BINARY),$$@)
-       $$(shell ( cd $(TRUNNER_TESTS_DIR);                             \
-                 echo '/* Generated header, do not edit */';           \
Quoted 8 lines

tools/testing/selftests/bpf/Makefile ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),) · line 425

Quoted 4 lines
tools/testing/selftests/bpf/Makefile +2 -2
-                 ls *.c 2> /dev/null |                                 \
-                       sed -e 's@\([^\.]*\)\.c@DEFINE_TEST(\1)@';      \
+       $$(shell (echo '/* Generated header, do not edit */';                                   \
+                 sed -n -E 's/^void (serial_)?test_([a-zA-Z0-9_]+)\((void)?\).*/DEFINE_TEST(\2)/p'     \

probably not that important : allow \s* before void and after void. Or, maybe we can just (?!static) instead of anchoring to line start.

Selftests source code is pretty strict with formatting, so I don't think we'll deviate from the strict ^void <name> pattern (and we certainly don't want to deviate). So I didn't want to overcomplicate regexes unnecessarily.

Quoted 5 lines

tools/testing/selftests/bpf/Makefile ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),) · line 426

Quoted 2 lines
tools/testing/selftests/bpf/Makefile +1 -0
+                       $(TRUNNER_TESTS_DIR)/*.c | sort ;       \

to be super safe : maybe add a check here to ensure each file contains at least one test function.

It's actually a useful property to have .c files that don't have tests. This can be used for adding various shared helpers. Currently all *_helpers.c are in selftests/bpf/ directory and have to be explicitly wired in Makefile, which is a bit annoying. With this setup we can just put a new .c file in the selftests/bpf/prog_tests/ and it will be automatically compiled and linked.

It also will significantly hurt readability to add some sort of per-file check in there, do you think it's worth it?

Quoted 7 lines
Quoted 7 lines
tools/testing/selftests/bpf/Makefile +0 -1
                 ) > $$@)
 endif
 
--

2.30.2

#5 lore raw

Re: [PATCH bpf-next 2/4] selftests/bpf: support multiple tests per file

Quoted 51 lines · Andrii Nakryiko
Quoted 22 lines · On Mon, Oct 25, 2021 at 1:13 PM [email protected] <[email protected]> wrote:
Quoted 20 lines · Andrii Nakryiko

Revamp how test discovery works for test_progs and allow multiple test entries per file. Any global void function with no arguments and serial_test_ or test_ prefix is considered a test.

---
 tools/testing/selftests/bpf/Makefile | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/Makefile +0 -2
index 498222543c37..ac47cf9760fc 100644
@@ -421,10 +421,9 @@ ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),)
 $(TRUNNER_TESTS_DIR)-tests-hdr := y
 $(TRUNNER_TESTS_HDR): $(TRUNNER_TESTS_DIR)/*.c
        $$(call msg,TEST-HDR,$(TRUNNER_BINARY),$$@)
-       $$(shell ( cd $(TRUNNER_TESTS_DIR);                             \
-                 echo '/* Generated header, do not edit */';           \
Quoted 8 lines

tools/testing/selftests/bpf/Makefile ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),) · line 425

Quoted 4 lines
tools/testing/selftests/bpf/Makefile +2 -2
-                 ls *.c 2> /dev/null |                                 \
-                       sed -e 's@\([^\.]*\)\.c@DEFINE_TEST(\1)@';      \
+       $$(shell (echo '/* Generated header, do not edit */';                                   \
+                 sed -n -E 's/^void (serial_)?test_([a-zA-Z0-9_]+)\((void)?\).*/DEFINE_TEST(\2)/p'     \

probably not that important : allow \s* before void and after void. Or, maybe we can just (?!static) instead of anchoring to line start.

Selftests source code is pretty strict with formatting, so I don't think we'll deviate from the strict ^void <name> pattern (and we certainly don't want to deviate). So I didn't want to overcomplicate regexes unnecessarily.

Quoted 5 lines

tools/testing/selftests/bpf/Makefile ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),) · line 426

Quoted 2 lines
tools/testing/selftests/bpf/Makefile +1 -0
+                       $(TRUNNER_TESTS_DIR)/*.c | sort ;       \

to be super safe : maybe add a check here to ensure each file contains at least one test function.

It's actually a useful property to have .c files that don't have tests. This can be used for adding various shared helpers. Currently all *_helpers.c are in selftests/bpf/ directory and have to be explicitly wired in Makefile, which is a bit annoying. With this setup we can just put a new .c file in the selftests/bpf/prog_tests/ and it will be automatically compiled and linked.

Quoted 2 lines

It also will significantly hurt readability to add some sort of per-file check in there, do you think it's worth it?

You are right, probably not really worth it. we just have to watch the total test numbers, it should always goes up :-D

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
tools/testing/selftests/bpf/Makefile +0 -1
                 ) > $$@)
 endif
 
--

2.30.2

#6 lore raw

Re: [PATCH bpf-next 2/4] selftests/bpf: support multiple tests per file

Quoted 54 lines · On Mon, Oct 25, 2021 at 1:56 PM [email protected] <[email protected]> wrote:
Quoted 51 lines · Andrii Nakryiko
Quoted 22 lines · On Mon, Oct 25, 2021 at 1:13 PM [email protected] <[email protected]> wrote:
Quoted 20 lines · Andrii Nakryiko

Revamp how test discovery works for test_progs and allow multiple test entries per file. Any global void function with no arguments and serial_test_ or test_ prefix is considered a test.

---
 tools/testing/selftests/bpf/Makefile | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
tools/testing/selftests/bpf/Makefile +0 -2
index 498222543c37..ac47cf9760fc 100644
@@ -421,10 +421,9 @@ ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),)
 $(TRUNNER_TESTS_DIR)-tests-hdr := y
 $(TRUNNER_TESTS_HDR): $(TRUNNER_TESTS_DIR)/*.c
        $$(call msg,TEST-HDR,$(TRUNNER_BINARY),$$@)
-       $$(shell ( cd $(TRUNNER_TESTS_DIR);                             \
-                 echo '/* Generated header, do not edit */';           \
Quoted 8 lines

tools/testing/selftests/bpf/Makefile ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),) · line 425

Quoted 4 lines
tools/testing/selftests/bpf/Makefile +2 -2
-                 ls *.c 2> /dev/null |                                 \
-                       sed -e 's@\([^\.]*\)\.c@DEFINE_TEST(\1)@';      \
+       $$(shell (echo '/* Generated header, do not edit */';                                   \
+                 sed -n -E 's/^void (serial_)?test_([a-zA-Z0-9_]+)\((void)?\).*/DEFINE_TEST(\2)/p'     \

probably not that important : allow \s* before void and after void. Or, maybe we can just (?!static) instead of anchoring to line start.

Selftests source code is pretty strict with formatting, so I don't think we'll deviate from the strict ^void <name> pattern (and we certainly don't want to deviate). So I didn't want to overcomplicate regexes unnecessarily.

Quoted 5 lines

tools/testing/selftests/bpf/Makefile ifeq ($($(TRUNNER_TESTS_DIR)-tests-hdr),) · line 426

Quoted 2 lines
tools/testing/selftests/bpf/Makefile +1 -0
+                       $(TRUNNER_TESTS_DIR)/*.c | sort ;       \

to be super safe : maybe add a check here to ensure each file contains at least one test function.

It's actually a useful property to have .c files that don't have tests. This can be used for adding various shared helpers. Currently all *_helpers.c are in selftests/bpf/ directory and have to be explicitly wired in Makefile, which is a bit annoying. With this setup we can just put a new .c file in the selftests/bpf/prog_tests/ and it will be automatically compiled and linked.

Quoted 5 lines
Quoted 2 lines

It also will significantly hurt readability to add some sort of per-file check in there, do you think it's worth it?

You are right, probably not really worth it. we just have to watch the total test numbers, it should always goes up :-D

Yep. We should be able to automate this once we have some sort of baseline comparison functionality in BPF CI.

Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
Quoted 9 lines
tools/testing/selftests/bpf/Makefile +0 -1
                 ) > $$@)
 endif
 
--

2.30.2

author maintainer
#7 lore raw

PATCH 3/4 selftests/bpf: mark tc_redirect selftest as serial

In reply to Andrii Nakryiko

It seems to cause a lot of harm to kprobe/tracepoint selftests. Yucong mentioned before that it does manipulate sysfs, which might be the reason. So let's mark it as serial, though ideally it would be less intrusive on the system at test.

---
 tools/testing/selftests/bpf/prog_tests/tc_redirect.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
tools/testing/selftests/bpf/prog_tests/tc_redirect.c +1 -1
index e87bc4466d9a..53672634bc52 100644
@@ -769,7 +769,7 @@ static void *test_tc_redirect_run_tests(void *arg)
 	return NULL;
 }
 
-void test_tc_redirect(void)
+void serial_test_tc_redirect(void)
 {
 	pthread_t test_thread;
 	int err;
Signature · 1 line
2.30.2
author maintainer
#8 lore raw

PATCH 4/4 selftests/bpf: split out bpf_verif_scale selftests into multiple tests

In reply to Andrii Nakryiko

Instead of using subtests in bpf_verif_scale selftest, turn each scale sub-test into its own test. Each subtest is compltely independent and just reuses a bit of common test running logic, so the conversion is trivial. For convenience, keep all of BPF verifier scale tests in one file.

This conversion shaves off a significant amount of time when running test_progs in parallel mode. E.g., just running scale tests (-t verif_scale):

BEFORE
======
Summary: 24/0 PASSED, 0 SKIPPED, 0 FAILED

real    0m22.894s
user    0m0.012s
sys     0m22.797s

AFTER
=====
Summary: 24/0 PASSED, 0 SKIPPED, 0 FAILED

real    0m12.044s
user    0m0.024s
sys     0m27.869s

Ten second saving right there. test_progs -j is not yet ready to be turned on by default, unfortunately, and some tests fail almost every time, but this is a good improvement nevertheless. Ignoring few failures, here is sequential vs parallel run times when running all tests now:

SEQUENTIAL
==========
Summary: 206/953 PASSED, 4 SKIPPED, 0 FAILED

real    1m5.625s
user    0m4.211s
sys     0m31.650s

PARALLEL
========
Summary: 204/952 PASSED, 4 SKIPPED, 2 FAILED

real    0m35.550s
user    0m4.998s
sys     0m39.890s
---
 .../bpf/prog_tests/bpf_verif_scale.c          | 220 ++++++++++++------
 1 file changed, 152 insertions(+), 68 deletions(-)
tools/testing/selftests/bpf/prog_tests/bpf_verif_scale.c +152 -68
index 3d002c245d2b..867349e4ed9e 100644
@@ -39,82 +39,166 @@ struct scale_test_def {
 	bool fails;
 };
 
-void test_bpf_verif_scale(void)
-{
-	struct scale_test_def tests[] = {
-		{ "loop3.o", BPF_PROG_TYPE_RAW_TRACEPOINT, true /* fails */ },
-
-		{ "test_verif_scale1.o", BPF_PROG_TYPE_SCHED_CLS },
-		{ "test_verif_scale2.o", BPF_PROG_TYPE_SCHED_CLS },
-		{ "test_verif_scale3.o", BPF_PROG_TYPE_SCHED_CLS },
-
-		{ "pyperf_global.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-		{ "pyperf_subprogs.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-
-		/* full unroll by llvm */
-		{ "pyperf50.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-		{ "pyperf100.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-		{ "pyperf180.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-
-		/* partial unroll. llvm will unroll loop ~150 times.
-		 * C loop count -> 600.
-		 * Asm loop count -> 4.
-		 * 16k insns in loop body.
-		 * Total of 5 such loops. Total program size ~82k insns.
-		 */
-		{ "pyperf600.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-
-		/* no unroll at all.
-		 * C loop count -> 600.
-		 * ASM loop count -> 600.
-		 * ~110 insns in loop body.
-		 * Total of 5 such loops. Total program size ~1500 insns.
-		 */
-		{ "pyperf600_nounroll.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-
-		{ "loop1.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-		{ "loop2.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-		{ "loop4.o", BPF_PROG_TYPE_SCHED_CLS },
-		{ "loop5.o", BPF_PROG_TYPE_SCHED_CLS },
-		{ "loop6.o", BPF_PROG_TYPE_KPROBE },
-
-		/* partial unroll. 19k insn in a loop.
-		 * Total program size 20.8k insn.
-		 * ~350k processed_insns
-		 */
-		{ "strobemeta.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-
-		/* no unroll, tiny loops */
-		{ "strobemeta_nounroll1.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-		{ "strobemeta_nounroll2.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-
-		/* non-inlined subprogs */
-		{ "strobemeta_subprogs.o", BPF_PROG_TYPE_RAW_TRACEPOINT },
-
-		{ "test_sysctl_loop1.o", BPF_PROG_TYPE_CGROUP_SYSCTL },
-		{ "test_sysctl_loop2.o", BPF_PROG_TYPE_CGROUP_SYSCTL },
-
-		{ "test_xdp_loop.o", BPF_PROG_TYPE_XDP },
-		{ "test_seg6_loop.o", BPF_PROG_TYPE_LWT_SEG6LOCAL },
-	};
+static void scale_test(const char *file,
+		       enum bpf_prog_type attach_type,
+		       bool should_fail)
+{
 	libbpf_print_fn_t old_print_fn = NULL;
-	int err, i;
+	int err;
 
 	if (env.verifier_stats) {
 		test__force_log();
 		old_print_fn = libbpf_set_print(libbpf_debug_print);
 	}
 
-	for (i = 0; i < ARRAY_SIZE(tests); i++) {
-		const struct scale_test_def *test = &tests[i];
-
-		if (!test__start_subtest(test->file))
-			continue;
-
-		err = check_load(test->file, test->attach_type);
-		CHECK_FAIL(err && !test->fails);
-	}
+	err = check_load(file, attach_type);
+	if (should_fail)
+		ASSERT_ERR(err, "expect_error");
+	else
+		ASSERT_OK(err, "expect_success");
 
 	if (env.verifier_stats)
 		libbpf_set_print(old_print_fn);
 }
+
+void test_verif_scale1()
+{
+	scale_test("test_verif_scale1.o", BPF_PROG_TYPE_SCHED_CLS, false);
+}
+
+void test_verif_scale2()
+{
+	scale_test("test_verif_scale2.o", BPF_PROG_TYPE_SCHED_CLS, false);
+}
+
+void test_verif_scale3()
+{
+	scale_test("test_verif_scale3.o", BPF_PROG_TYPE_SCHED_CLS, false);
+}
+
+void test_verif_scale_pyperf_global()
+{
+	scale_test("pyperf_global.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_pyperf_subprogs()
+{
+	scale_test("pyperf_subprogs.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_pyperf50()
+{
+	/* full unroll by llvm */
+	scale_test("pyperf50.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_pyperf100()
+{
+	/* full unroll by llvm */
+	scale_test("pyperf100.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_pyperf180()
+{
+	/* full unroll by llvm */
+	scale_test("pyperf180.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_pyperf600()
+{
+	/* partial unroll. llvm will unroll loop ~150 times.
+	 * C loop count -> 600.
+	 * Asm loop count -> 4.
+	 * 16k insns in loop body.
+	 * Total of 5 such loops. Total program size ~82k insns.
+	 */
+	scale_test("pyperf600.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_pyperf600_nounroll()
+{
+	/* no unroll at all.
+	 * C loop count -> 600.
+	 * ASM loop count -> 600.
+	 * ~110 insns in loop body.
+	 * Total of 5 such loops. Total program size ~1500 insns.
+	 */
+	scale_test("pyperf600_nounroll.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_loop1()
+{
+	scale_test("loop1.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_loop2()
+{
+	scale_test("loop2.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_loop3_fail()
+{
+	scale_test("loop3.o", BPF_PROG_TYPE_RAW_TRACEPOINT, true /* fails */);
+}
+
+void test_verif_scale_loop4()
+{
+	scale_test("loop4.o", BPF_PROG_TYPE_SCHED_CLS, false);
+}
+
+void test_verif_scale_loop5()
+{
+	scale_test("loop5.o", BPF_PROG_TYPE_SCHED_CLS, false);
+}
+
+void test_verif_scale_loop6()
+{
+	scale_test("loop6.o", BPF_PROG_TYPE_KPROBE, false);
+}
+
+void test_verif_scale_strobemeta()
+{
+	/* partial unroll. 19k insn in a loop.
+	 * Total program size 20.8k insn.
+	 * ~350k processed_insns
+	 */
+	scale_test("strobemeta.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_strobemeta_nounroll1()
+{
+	/* no unroll, tiny loops */
+	scale_test("strobemeta_nounroll1.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_strobemeta_nounroll2()
+{
+	/* no unroll, tiny loops */
+	scale_test("strobemeta_nounroll2.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_strobemeta_subprogs()
+{
+	/* non-inlined subprogs */
+	scale_test("strobemeta_subprogs.o", BPF_PROG_TYPE_RAW_TRACEPOINT, false);
+}
+
+void test_verif_scale_sysctl_loop1()
+{
+	scale_test("test_sysctl_loop1.o", BPF_PROG_TYPE_CGROUP_SYSCTL, false);
+}
+
+void test_verif_scale_sysctl_loop2()
+{
+	scale_test("test_sysctl_loop2.o", BPF_PROG_TYPE_CGROUP_SYSCTL, false);
+}
+
+void test_verif_scale_xdp_loop()
+{
+	scale_test("test_xdp_loop.o", BPF_PROG_TYPE_XDP, false);
+}
+
+void test_verif_scale_seg6_loop()
+{
+	scale_test("test_seg6_loop.o", BPF_PROG_TYPE_LWT_SEG6LOCAL, false);
+}
Signature · 1 line
2.30.2
Acked-by on the v1 series
#9 lore raw
In reply to Andrii Nakryiko

Thanks, this patch is awesome!

Quoted 32 lines · Andrii Nakryiko

Reduce amount of waiting time when running test_progs in parallel mode (-j) by splitting bpf_verif_scale selftests into multiple tests. Previously it was structured as a test with multiple subtests, but subtests are not easily parallelizable with test_progs' infra. Also in practice each scale subtest is really an independent test with nothing shared across all substest.

This patch set changes how test_progs test discovery works. Now it is possible to define multiple tests within a single source code file. One of the patches also marks tc_redirect selftests as serial, because it's extremely harmful to the test system when run in parallel mode.

Andrii Nakryiko (4):
selftests/bpf: normalize selftest entry points selftests/bpf: support multiple tests per file selftests/bpf: mark tc_redirect selftest as serial selftests/bpf: split out bpf_verif_scale selftests into multiple tests

 tools/testing/selftests/bpf/Makefile          |   7 +-
 .../bpf/prog_tests/bpf_verif_scale.c          | 220 ++++++++++++------
 .../selftests/bpf/prog_tests/btf_dump.c       |   2 +-
 .../selftests/bpf/prog_tests/resolve_btfids.c |  10 +-
 .../selftests/bpf/prog_tests/signal_pending.c |   2 +-
 .../selftests/bpf/prog_tests/snprintf.c       |   4 +-
 .../selftests/bpf/prog_tests/tc_redirect.c    |   2 +-
 .../bpf/prog_tests/xdp_adjust_tail.c          |   6 +-
 .../bpf/prog_tests/xdp_devmap_attach.c        |   4 +-
 9 files changed, 169 insertions(+), 88 deletions(-)

--
2.30.2

#10 lore raw
Quoted 18 lines · On Mon, Oct 25, 2021 at 1:15 PM [email protected] <[email protected]> wrote:

Thanks, this patch is awesome!

Quoted 11 lines · Andrii Nakryiko

Reduce amount of waiting time when running test_progs in parallel mode (-j) by splitting bpf_verif_scale selftests into multiple tests. Previously it was structured as a test with multiple subtests, but subtests are not easily parallelizable with test_progs' infra. Also in practice each scale subtest is really an independent test with nothing shared across all substest.

This patch set changes how test_progs test discovery works. Now it is possible to define multiple tests within a single source code file. One of the patches also marks tc_redirect selftests as serial, because it's extremely harmful to the test system when run in parallel mode.

The patch set was applied.