Skip to contentSkip to conversation

run tests in parallel

PATCH Applied Dylan Yudaken 2 participants 9 messages Apr 22, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover run tests in parallel

This series allows tests to be run in parallel, which speeds up iterating. Rather than build this functionality into the shell scripts, it seemed much easier to use make's parallel execution to do this.

My bash/make skills are not top notch, so I might have missed something obvious, however it does seem to work locally very nicely.

Patch #1 is a bug that seems to trigger quite often when running in parallel
Patch #2-5 fix bugs that prevent tests running in parallel
Patch #6 adds a make target that depends on running each test
Patch #7 Is not related to parallel tests. It is a prep for a later series to use nop to test IOPOLL.

v2:
- fix runtests-quiet.sh to return the correct error code
- add runtests-parallel to the main target

Dylan Yudaken (7):
test: handle mmap return failures in pollfree test
test: use unique path for socket
test: use unique ports
test: use unique filenames
test: mkdir -p output folder
test: add make targets for each test
test: use remove_buffers instead of nop to generate error codes

 Makefile               |  2 ++
 test/232c93d07b74.c    |  2 +-
 test/Makefile          | 10 +++++++++-
 test/accept-test.c     |  2 +-
 test/defer.c           | 28 ++++++++++++++++++++--------
 test/link.c            |  6 +++---
 test/openat2.c         |  6 +++---
 test/pollfree.c        |  4 ++--
 test/recv-msgall.c     |  2 +-
 test/runtests-quiet.sh | 11 +++++++++++
 test/runtests.sh       |  2 +-
 test/send_recv.c       |  2 +-
 test/send_recvmsg.c    |  2 +-
 test/sq-poll-dup.c     |  2 +-
 test/sq-poll-share.c   |  2 +-
 15 files changed, 58 insertions(+), 25 deletions(-)
 create mode 100755 test/runtests-quiet.sh

base-commit: b7d8dd8bbf5b8550c8a0c1ed70431cd8050709f0

Signature · 1 line
2.30.2
#1 lore raw

PATCH v2 1/7 test: handle mmap return failures in pollfree test

Sometimes these mmap's fail, and it causes SEGFAULTS.
I assume this was accidentally left off when this was originally landed.

---
 test/pollfree.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
test/pollfree.c +2 -2
index e2511df..d753ffe 100644
@@ -406,10 +406,10 @@ int main(int argc, char *argv[])
   ret = mmap((void *)0x1ffff000ul, 0x1000ul, 0ul, 0x32ul, -1, 0ul);
   if (ret == MAP_FAILED)
     return 0;
-  mmap((void *)0x20000000ul, 0x1000000ul, 7ul, 0x32ul, -1, 0ul);
+  ret = mmap((void *)0x20000000ul, 0x1000000ul, 7ul, 0x32ul, -1, 0ul);
   if (ret == MAP_FAILED)
     return 0;
-  mmap((void *)0x21000000ul, 0x1000ul, 0ul, 0x32ul, -1, 0ul);
+  ret = mmap((void *)0x21000000ul, 0x1000ul, 0ul, 0x32ul, -1, 0ul);
   if (ret == MAP_FAILED)
     return 0;
   loop();
Signature · 1 line
2.30.2
#2 lore raw

PATCH v2 2/7 test: use unique path for socket

In reply to Dylan Yudaken

Do not collide if multiple tests are running at once.

---
 test/accept-test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
test/accept-test.c +1 -1
index 71d9d80..4a904e4 100644
@@ -35,7 +35,7 @@ int main(int argc, char *argv[])
 
 	memset(&addr, 0, sizeof(addr));
 	addr.sun_family = AF_UNIX;
-	memcpy(addr.sun_path, "\0sock", 6);
+	memcpy(addr.sun_path, "\0sock2", 7);
 
 	ret = bind(fd, (struct sockaddr *)&addr, addrlen);
 	assert(ret != -1);
Signature · 1 line
2.30.2
#3 lore raw

PATCH v2 3/7 test: use unique ports

In reply to Dylan Yudaken

In preparation for running tests in parallel remove some collisions in ports. In the future this should probably use ephemeral ports, but for now there are not too many places to change.

---
 test/232c93d07b74.c | 2 +-
 test/recv-msgall.c  | 2 +-
 test/send_recv.c    | 2 +-
 test/send_recvmsg.c | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)
test/232c93d07b74.c +1 -1
index 4153aef..8a7810b 100644
@@ -26,7 +26,7 @@
 #define RECV_BUFF_SIZE 2
 #define SEND_BUFF_SIZE 3
 
-#define PORT	0x1235
+#define PORT	0x1234
 
 struct params {
 	int tcp;
test/recv-msgall.c +1 -1
index 5f202b4..a6f7cfc 100644
@@ -17,7 +17,7 @@
 
 #define MAX_MSG	128
 
-#define PORT	10200
+#define PORT	10201
 #define HOST	"127.0.0.1"
 
 static int recv_prep(struct io_uring *ring, struct iovec *iov, int *sock,
test/send_recv.c +1 -1
index ad8ea0e..a7b001a 100644
@@ -19,7 +19,7 @@ static char str[] = "This is a test of send and recv over io_uring!";
 
 #define MAX_MSG	128
 
-#define PORT	10200
+#define PORT	10202
 #define HOST	"127.0.0.1"
 
 static int recv_prep(struct io_uring *ring, struct iovec *iov, int *sock,
test/send_recvmsg.c +1 -1
index 43121f0..f607b5c 100644
@@ -19,7 +19,7 @@ static char str[] = "This is a test of sendmsg and recvmsg over io_uring!";
 
 #define MAX_MSG	128
 
-#define PORT	10200
+#define PORT	10203
 #define HOST	"127.0.0.1"
 
 #define BUF_BGID	10
Signature · 1 line
2.30.2
#4 lore raw

PATCH v2 4/7 test: use unique filenames

In reply to Dylan Yudaken

Allow tests to be run in parallel

---
 test/openat2.c       | 6 +++---
 test/sq-poll-dup.c   | 2 +-
 test/sq-poll-share.c | 2 +-
 3 files changed, 5 insertions(+), 5 deletions(-)
test/openat2.c +3 -3
index 379c61e..34c0f85 100644
@@ -246,12 +246,12 @@ int main(int argc, char *argv[])
 	}
 
 	if (argc > 1) {
-		path = "/tmp/.open.close";
+		path = "/tmp/.open.at2";
 		path_rel = argv[1];
 		do_unlink = 0;
 	} else {
-		path = "/tmp/.open.close";
-		path_rel = ".open.close";
+		path = "/tmp/.open.at2";
+		path_rel = ".open.at2";
 		do_unlink = 1;
 	}
 
test/sq-poll-dup.c +1 -1
index 0076a31..6a72b82 100644
@@ -164,7 +164,7 @@ int main(int argc, char *argv[])
 	if (argc > 1) {
 		fname = argv[1];
 	} else {
-		fname = ".basic-rw";
+		fname = ".basic-rw-poll-dup";
 		t_create_file(fname, FILE_SIZE);
 	}
 
test/sq-poll-share.c +1 -1
index 2f1c1dd..7bb7626 100644
@@ -82,7 +82,7 @@ int main(int argc, char *argv[])
 	if (argc > 1) {
 		fname = argv[1];
 	} else {
-		fname = ".basic-rw";
+		fname = ".basic-rw-poll-share";
 		t_create_file(fname, FILE_SIZE);
 	}
 
Signature · 1 line
2.30.2
#5 lore raw

PATCH v2 5/7 test: mkdir -p output folder

In reply to Dylan Yudaken

In case multiple mkdir are running at once, do not log an error if it exists due to a race condition.

---
 test/runtests.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
test/runtests.sh +1 -1
index 122e482..6d8f7af 100755
@@ -139,7 +139,7 @@ run_test()
 # Run all specified tests
 for tst in "${TESTS[@]}"; do
 	if [ ! -d output ]; then
-		mkdir output
+		mkdir -p output
 	fi
 	if [ -z "${TEST_MAP[$tst]}" ]; then
 		run_test "$tst"
Signature · 1 line
2.30.2
#6 lore raw

PATCH v2 6/7 test: add make targets for each test

In reply to Dylan Yudaken

Add a make target runtests-parallel which can run tests in parallel. This is very useful to quickly run all the tests locally with

  $ make -j runtests-parallel
---
 Makefile               |  2 ++
 test/Makefile          | 10 +++++++++-
 test/runtests-quiet.sh | 11 +++++++++++
 3 files changed, 22 insertions(+), 1 deletion(-)
 create mode 100755 test/runtests-quiet.sh
Makefile +2 -0
index 28c0fd8..d54551e 100644
@@ -21,6 +21,8 @@ runtests: all
 	@$(MAKE) -C test runtests
 runtests-loop:
 	@$(MAKE) -C test runtests-loop
+runtests-parallel:
+	@$(MAKE) -C test runtests-parallel
 
 config-host.mak: configure
 	@if [ ! -e "$@" ]; then					\
test/Makefile +9 -1
index cb7e15e..fe35ff9 100644
@@ -185,6 +185,7 @@ all_targets += sq-full-cpp.t
 
 test_targets := $(patsubst %.c,%,$(test_srcs))
 test_targets := $(patsubst %.cc,%,$(test_targets))
+run_test_targets := $(patsubst %,%.run_test,$(test_targets))
 test_targets := $(patsubst %,%.t,$(test_targets))
 all_targets += $(test_targets)
 
@@ -229,4 +230,11 @@ runtests: all
 runtests-loop: all
 	@./runtests-loop.sh $(test_targets)
 
-.PHONY: all install clean runtests runtests-loop
+%.run_test: %.t
+	@./runtests-quiet.sh $<
+
+runtests-parallel: $(run_test_targets)
+	@echo "All tests passed"
+
+.PHONY: all install clean runtests runtests-loop runtests-parallel
+.PHONY += $(run_test_targets)
test/runtests-quiet.sh new file +11 -0
new file mode 100755
index 0000000..438a00a
@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+
+TESTS=("$@")
+RESULT_FILE=$(mktemp)
+./runtests.sh "${TESTS[@]}" 2>&1 > $RESULT_FILE
+RET="$?"
+if [ "${RET}" -ne 0 ]; then
+    cat $RESULT_FILE
+fi
+rm $RESULT_FILE
+exit $RET
Signature · 1 line
2.30.2
#7 lore raw

PATCH v2 7/7 test: use remove_buffers instead of nop to generate error codes

In reply to Dylan Yudaken

This is in prep for allwoing NOP to be used in IOPOLL mode. remove_buffers will consistently return ENOENT if asked to remove buffers from a nonexistent group, and so this is a suitable replacement. Other opcodes return -EINVAL in IOPOLL from the prep stage, which has slightly different behaviour.

---
 test/defer.c | 28 ++++++++++++++++++++--------
 test/link.c  |  6 +++---
 2 files changed, 23 insertions(+), 11 deletions(-)
test/defer.c +20 -8
index 825b69f..68ee4b4 100644
@@ -12,6 +12,10 @@
 #include "liburing.h"
 
 #define RING_SIZE 128
+enum {
+	OP_NOP,
+	OP_REMOVE_BUFFERS
+};
 
 struct test_context {
 	struct io_uring *ring;
@@ -27,7 +31,8 @@ static void free_context(struct test_context *ctx)
 	memset(ctx, 0, sizeof(*ctx));
 }
 
-static int init_context(struct test_context *ctx, struct io_uring *ring, int nr)
+static int init_context(struct test_context *ctx, struct io_uring *ring, int nr,
+			int op)
 {
 	struct io_uring_sqe *sqe;
 	int i;
@@ -45,7 +50,14 @@ static int init_context(struct test_context *ctx, struct io_uring *ring, int nr)
 		sqe = io_uring_get_sqe(ring);
 		if (!sqe)
 			goto err;
-		io_uring_prep_nop(sqe);
+		switch (op) {
+		case OP_NOP:
+			io_uring_prep_nop(sqe);
+			break;
+		case OP_REMOVE_BUFFERS:
+			io_uring_prep_remove_buffers(sqe, 10, 1);
+			break;
+		};
 		sqe->user_data = i;
 		ctx->sqes[i] = sqe;
 	}
@@ -81,7 +93,7 @@ static int test_cancelled_userdata(struct io_uring *ring)
 	struct test_context ctx;
 	int ret, i, nr = 100;
 
-	if (init_context(&ctx, ring, nr))
+	if (init_context(&ctx, ring, nr, OP_NOP))
 		return 1;
 
 	for (i = 0; i < nr; i++)
@@ -115,7 +127,7 @@ static int test_thread_link_cancel(struct io_uring *ring)
 	struct test_context ctx;
 	int ret, i, nr = 100;
 
-	if (init_context(&ctx, ring, nr))
+	if (init_context(&ctx, ring, nr, OP_REMOVE_BUFFERS))
 		return 1;
 
 	for (i = 0; i < nr; i++)
@@ -134,12 +146,12 @@ static int test_thread_link_cancel(struct io_uring *ring)
 		bool fail = false;
 
 		if (i == 0)
-			fail = (ctx.cqes[i].res != -EINVAL);
+			fail = (ctx.cqes[i].res != -ENOENT);
 		else
 			fail = (ctx.cqes[i].res != -ECANCELED);
 
 		if (fail) {
-			printf("invalid status\n");
+			printf("invalid status %d\n", ctx.cqes[i].res);
 			goto err;
 		}
 	}
@@ -158,7 +170,7 @@ static int test_drain_with_linked_timeout(struct io_uring *ring)
 	struct test_context ctx;
 	int ret, i;
 
-	if (init_context(&ctx, ring, nr * 2))
+	if (init_context(&ctx, ring, nr * 2, OP_NOP))
 		return 1;
 
 	for (i = 0; i < nr; i++) {
@@ -188,7 +200,7 @@ static int run_drained(struct io_uring *ring, int nr)
 	struct test_context ctx;
 	int ret, i;
 
-	if (init_context(&ctx, ring, nr))
+	if (init_context(&ctx, ring, nr, OP_NOP))
 		return 1;
 
 	for (i = 0; i < nr; i++)
test/link.c +3 -3
index c89d6b2..41d3899 100644
@@ -178,7 +178,7 @@ static int test_single_link_fail(struct io_uring *ring)
 		goto err;
 	}
 
-	io_uring_prep_nop(sqe);
+	io_uring_prep_remove_buffers(sqe, 10, 1);
 	sqe->flags |= IOSQE_IO_LINK;
 
 	sqe = io_uring_get_sqe(ring);
@@ -205,8 +205,8 @@ static int test_single_link_fail(struct io_uring *ring)
 			printf("failed to get cqe\n");
 			goto err;
 		}
-		if (i == 0 && cqe->res != -EINVAL) {
-			printf("sqe0 failed with %d, wanted -EINVAL\n", cqe->res);
+		if (i == 0 && cqe->res != -ENOENT) {
+			printf("sqe0 failed with %d, wanted -ENOENT\n", cqe->res);
 			goto err;
 		}
 		if (i == 1 && cqe->res != -ECANCELED) {
Signature · 1 line
2.30.2
maintainer Applied
#8 lore raw
In reply to Dylan Yudaken
Quoted 8 lines · Dylan Yudaken

This series allows tests to be run in parallel, which speeds up iterating. Rather than build this functionality into the shell scripts, it seemed much easier to use make's parallel execution to do this.

My bash/make skills are not top notch, so I might have missed something obvious, however it does seem to work locally very nicely.

[...]

Applied, thanks!

[1/7] test: handle mmap return failures in pollfree test

      commit: 861c121c931d33dae4fbeeab3eac86544d856d9a
[2/7] test: use unique path for socket
      commit: 44673763e7467d49a1ef4bba8a641f9dd4c6ff70
[3/7] test: use unique ports
      commit: 736667aa9e31cecded9474d28f5666fa900d0b77
[4/7] test: use unique filenames
      commit: 16d9366b9f9ca44d2a4da7320663a06fb9df7a28
[5/7] test: mkdir -p output folder
      commit: fddf8e6fd0ec06ca84a1d6a769dbd891e7cdaf08
[6/7] test: add make targets for each test
      commit: 6480f692d62afbebb088febc369b30a63dbc2ea7
[7/7] test: use remove_buffers instead of nop to generate error codes
      commit: 473e16399327ff1a05cb58b256c9bc86b47561c3

Best regards,

Signature · 1 line
Jens Axboe