Skip to contentSkip to conversation

Dynamic subtests

PATCH In review Petri Latvala 4 participants 14 messages Jun 19, 2019 – Aug 2, 2019

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover Dynamic subtests

Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?

Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?

Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?

If so, this is your lucky day, for dynamic subtests are here!

While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:

With normal subtests:
for_each_pipe_static(pipe)

  igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
    int fd = drm_open_driver(DRIVER_ANY);
    igt_require(actually_have_this_pipe(fd, pipe));
    igt_assert(do_stuff(fd, pipe));

}

With dynamic subtests:
igt_subtest_container("fudge")
int fd = drm_open_driver(DRIVER_ANY);

  for_each_pipe(fd, pipe) {
    igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
      igt_assert(do_stuff(fd, pipe));
    }

}

Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.

Semantics
=========

Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it igt_subtest_container().

A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
- igt_assert not allowed
- igt_skip / igt_require is allowed though

Any failing dynamic subtest will result in the container reporting a failure.

Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using igt_require in an igt_dynamic_subtest, just don't enter it for such an occasion. In other words:

Do not:
for_each_pipe(fd, pipe)

  igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
    igt_require(pipe_has_things(pipe));
    ...

}

Instead do:
for_each_pipe(fd, pipe) {
if (!pipe_has_things(pipe))

    continue;
  igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
    ...

}
}

That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.

Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.

Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of igt_fixture usage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.

Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:

igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.

igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...

Existing tests that are good candidates for conversion to dynamic subtests:

1) Anything using for_each_pipe_static

2) Tests that have one subtest per i915 engine

3) Kernel selftests

Petri Latvala (6):
lib: Introduce dynamic subtests lib/tests: Unit tests for dynamic subtests runner/resultgen: Refactor output parsing runner/json_tests: Adapt to better output parsing
runner: Parse dynamic subtest outputs and results runner/json_tests: Test dynamic subtests

 lib/igt_core.c                                | 119 +++-
 lib/igt_core.h                                |  89 +++
 lib/tests/igt_dynamic_subtests.c              | 186 ++++++
 lib/tests/meson.build                         |   1 +
 runner/job_list.c                             |  11 +
 runner/job_list.h                             |   3 +
 .../aborted-after-a-test/reference.json       |   2 +-
 .../dmesg-escapes/reference.json              |   2 +-
 .../dmesg-results/reference.json              |   8 +-
 .../reference.json                            |   4 +-
 .../reference.json                            |   4 +-
 .../dmesg-warn-level/reference.json           |   4 +-
 .../dynamic-subtests/0/dmesg.txt              |   7 +
 .../dynamic-subtests/0/err.txt                |  36 ++
 .../dynamic-subtests/0/journal.txt            |   2 +
 .../dynamic-subtests/0/out.txt                |  19 +
 .../dynamic-subtests/1/dmesg.txt              |   5 +
 .../dynamic-subtests/1/err.txt                |   2 +
 .../dynamic-subtests/1/journal.txt            |   2 +
 .../dynamic-subtests/1/out.txt                |   5 +
 .../dynamic-subtests/2/dmesg.txt              |  10 +
 .../dynamic-subtests/2/err.txt                |   6 +
 .../dynamic-subtests/2/journal.txt            |   4 +
 .../dynamic-subtests/2/out.txt                |   8 +
 .../dynamic-subtests/README.txt               |   2 +
 .../dynamic-subtests/endtime.txt              |   1 +
 .../dynamic-subtests/joblist.txt              |   3 +
 .../dynamic-subtests/metadata.txt             |  12 +
 .../dynamic-subtests/reference.json           | 156 +++++
 .../dynamic-subtests/starttime.txt            |   1 +
 .../dynamic-subtests/uname.txt                |   1 +
 .../json_tests_data/normal-run/reference.json |   8 +-
 .../reference.json                            |   2 +-
 .../notrun-results/reference.json             |   2 +-
 .../piglit-style-dmesg/reference.json         |   8 +-
 .../warnings-with-dmesg-warns/reference.json  |   8 +-
 .../json_tests_data/warnings/reference.json   |   8 +-
 runner/output_strings.h                       |  29 +-
 runner/resultgen.c                            | 589 ++++++++++++------
 runner/runner_json_tests.c                    |   3 +-
 40 files changed, 1148 insertions(+), 224 deletions(-)
 create mode 100644 lib/tests/igt_dynamic_subtests.c
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/README.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/endtime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/joblist.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/metadata.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/reference.json
 create mode 100644 runner/json_tests_data/dynamic-subtests/starttime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/uname.txt
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#1 lore raw

PATCH 1/6 lib: Introduce dynamic subtests

Dynamic subtests, or subtests of subtests, are individual pieces of tests that are not statically available all the time.

A good example of a need for a dynamic subtest is i915 engine listing:
A normal subtest for each engine class ("bsd"), and a dynamic subtest for each instance ("bsd0", "bsd2", etc). Or a normal subtest for an operation with a dynamic subtest for every engine there is.

Another example is a dynamic subtest for pipes: Instead of using foreach_pipe_static, make one subtest and use foreach_pipe with dynamic subtests for each pipe.

---
 lib/igt_core.c | 119 ++++++++++++++++++++++++++++++++++++++++++++-----
 lib/igt_core.h |  89 ++++++++++++++++++++++++++++++++++++
 2 files changed, 197 insertions(+), 11 deletions(-)
lib/igt_core.c +108 -11
index 6b9f0425..937c2490 100644
@@ -260,9 +260,12 @@ const char *igt_interactive_debug;
 /* subtests helpers */
 static bool list_subtests = false;
 static char *run_single_subtest = NULL;
+static char *run_single_dynamic_subtest = NULL;
 static bool run_single_subtest_found = false;
 static const char *in_subtest = NULL;
+static const char *in_dynamic_subtest = NULL;
 static struct timespec subtest_time;
+static struct timespec dynamic_subtest_time;
 static clockid_t igt_clock = (clockid_t)-1;
 static bool in_fixture = false;
 static bool test_with_subtests = false;
@@ -286,6 +289,7 @@ enum {
 	 */
 	OPT_LIST_SUBTESTS = 500,
 	OPT_RUN_SUBTEST,
+	OPT_RUN_DYNAMIC_SUBTEST,
 	OPT_DESCRIPTION,
 	OPT_DEBUG,
 	OPT_INTERACTIVE_DEBUG,
@@ -308,6 +312,8 @@ char *igt_frame_dump_path;
 
 static bool stderr_needs_sentinel = false;
 
+static int _igt_dynamic_tests_executed = -1;
+
 const char *igt_test_name(void)
 {
 	return command_str;
@@ -339,7 +345,9 @@ static void _igt_log_buffer_dump(void)
 {
 	uint8_t i;
 
-	if (in_subtest)
+	if (in_dynamic_subtest)
+		fprintf(stderr, "Dynamic subtest %s failed.\n", in_dynamic_subtest);
+	else if (in_subtest)
 		fprintf(stderr, "Subtest %s failed.\n", in_subtest);
 	else
 		fprintf(stderr, "Test %s failed.\n", command_str);
@@ -555,6 +563,7 @@ static void print_usage(const char *help_str, bool output_on_stderr)
 	fprintf(f, "Usage: %s [OPTIONS]\n", command_str);
 	fprintf(f, "  --list-subtests\n"
 		   "  --run-subtest <pattern>\n"
+		   "  --dynamic-subtest <pattern>\n"
 		   "  --debug[=log-domain]\n"
 		   "  --interactive-debug[=domain]\n"
 		   "  --help-description\n"
@@ -672,6 +681,7 @@ static int common_init(int *argc, char **argv,
 	static struct option long_options[] = {
 		{"list-subtests",     no_argument,       NULL, OPT_LIST_SUBTESTS},
 		{"run-subtest",       required_argument, NULL, OPT_RUN_SUBTEST},
+		{"dynamic-subtest",   required_argument, NULL, OPT_RUN_DYNAMIC_SUBTEST},
 		{"help-description",  no_argument,       NULL, OPT_DESCRIPTION},
 		{"debug",             optional_argument, NULL, OPT_DEBUG},
 		{"interactive-debug", optional_argument, NULL, OPT_INTERACTIVE_DEBUG},
@@ -782,6 +792,11 @@ static int common_init(int *argc, char **argv,
 			if (!list_subtests)
 				run_single_subtest = strdup(optarg);
 			break;
+		case OPT_RUN_DYNAMIC_SUBTEST:
+			assert(optarg);
+			if (!list_subtests)
+				run_single_dynamic_subtest = strdup(optarg);
+			break;
 		case OPT_DESCRIPTION:
 			print_test_description();
 			ret = -1;
@@ -992,6 +1007,41 @@ bool __igt_run_subtest(const char *subtest_name)
 	return (in_subtest = subtest_name);
 }
 
+bool __igt_run_dynamic_subtest(const char *dynamic_subtest_name)
+{
+	int i;
+
+	assert(in_subtest);
+	assert(_igt_dynamic_tests_executed >= 0);
+
+	/* check the dynamic_subtest name only contains a-z, A-Z, 0-9, '-' and '_' */
+	for (i = 0; dynamic_subtest_name[i] != '\0'; i++)
+		if (dynamic_subtest_name[i] != '_' && dynamic_subtest_name[i] != '-'
+		    && !isalnum(dynamic_subtest_name[i])) {
+			igt_critical("Invalid dynamic subtest name \"%s\".\n",
+				     dynamic_subtest_name);
+			igt_exit();
+		}
+
+	if (run_single_dynamic_subtest &&
+	    uwildmat(dynamic_subtest_name, run_single_dynamic_subtest) == 0)
+		return false;
+
+	igt_kmsg(KMSG_INFO "%s: starting dynamic subtest %s\n",
+		 command_str, dynamic_subtest_name);
+	igt_info("Starting dynamic subtest: %s\n", dynamic_subtest_name);
+	fflush(stdout);
+	if (stderr_needs_sentinel)
+		fprintf(stderr, "Starting dynamic subtest: %s\n", dynamic_subtest_name);
+
+	_igt_log_buffer_reset();
+
+	_igt_dynamic_tests_executed++;
+
+	igt_gettime(&dynamic_subtest_time);
+	return (in_dynamic_subtest = dynamic_subtest_name);
+}
+
 /**
  * igt_subtest_name:
  *
@@ -1029,26 +1079,50 @@ void __igt_subtest_group_restore(int save)
 static bool skipped_one = false;
 static bool succeeded_one = false;
 static bool failed_one = false;
+static bool dynamic_failed_one = false;
+
+bool __igt_enter_dynamic_container(void)
+{
+	_igt_dynamic_tests_executed = 0;
+	dynamic_failed_one = false;
+
+	return true;
+}
 
 static void exit_subtest(const char *) __attribute__((noreturn));
 static void exit_subtest(const char *result)
 {
 	struct timespec now;
+	const char *subtest_text = in_dynamic_subtest ? "Dynamic subtest" : "Subtest";
+	const char **subtest_name = in_dynamic_subtest ? &in_dynamic_subtest : &in_subtest;
+	struct timespec *thentime = in_dynamic_subtest ? &dynamic_subtest_time : &subtest_time;
+	jmp_buf *jmptarget = in_dynamic_subtest ? &igt_dynamic_subtest_jmpbuf : &igt_subtest_jmpbuf;
 
 	igt_gettime(&now);
-	igt_info("%sSubtest %s: %s (%.3fs)%s\n",
+
+	igt_info("%s%s %s: %s (%.3fs)%s\n",
 		 (!__igt_plain_output) ? "\x1b[1m" : "",
-		 in_subtest, result, igt_time_elapsed(&subtest_time, &now),
+		 subtest_text, *subtest_name, result,
+		 igt_time_elapsed(thentime, &now),
 		 (!__igt_plain_output) ? "\x1b[0m" : "");
 	fflush(stdout);
 	if (stderr_needs_sentinel)
-		fprintf(stderr, "Subtest %s: %s (%.3fs)\n",
-			in_subtest, result, igt_time_elapsed(&subtest_time, &now));
+		fprintf(stderr, "%s %s: %s (%.3fs)\n",
+			subtest_text, *subtest_name,
+			result, igt_time_elapsed(thentime, &now));
 
 	igt_terminate_spins();
 
-	in_subtest = NULL;
-	siglongjmp(igt_subtest_jmpbuf, 1);
+	if (!in_dynamic_subtest)
+		_igt_dynamic_tests_executed = -1;
+
+	/* Don't keep the above text in the log, the container would print it again otherwise */
+	if (in_dynamic_subtest)
+		_igt_log_buffer_reset();
+
+	*subtest_name = NULL;
+
+	siglongjmp(*jmptarget, 1);
 }
 
 /**
@@ -1079,6 +1153,7 @@ void igt_skip(const char *f, ...)
 	}
 
 	if (in_subtest) {
+		/* Doing the same even if inside a dynamic subtest */
 		exit_subtest("SKIP");
 	} else if (test_with_subtests) {
 		skip_subtests_henceforth = SKIP;
@@ -1135,7 +1210,22 @@ void __igt_skip_check(const char *file, const int line,
  */
 void igt_success(void)
 {
-	succeeded_one = true;
+	if (in_subtest && !in_dynamic_subtest && _igt_dynamic_tests_executed >= 0) {
+		/*
+		 * We're exiting a dynamic container, yield a result
+		 * according to the dynamic tests that got
+		 * executed.
+		 */
+		if (dynamic_failed_one)
+			igt_fail(IGT_EXIT_FAILURE);
+
+		if (_igt_dynamic_tests_executed == 0)
+			igt_skip("No dynamic tests executed.\n");
+	}
+
+	if (!in_dynamic_subtest)
+		succeeded_one = true;
+
 	if (in_subtest)
 		exit_subtest("SUCCESS");
 }
@@ -1166,10 +1256,17 @@ void igt_fail(int exitcode)
 	if (in_atexit_handler)
 		_exit(IGT_EXIT_FAILURE);
 
-	if (!failed_one)
-		igt_exitcode = exitcode;
+	if (in_dynamic_subtest) {
+		dynamic_failed_one = true;
+	} else {
+		/* Dynamic subtest containers must not fail explicitly */
+		assert(_igt_dynamic_tests_executed < 0 || dynamic_failed_one);
+
+		if (!failed_one)
+			igt_exitcode = exitcode;
 
-	failed_one = true;
+		failed_one = true;
+	}
 
 	/* Silent exit, parent will do the yelling. */
 	if (test_child)
lib/igt_core.h +89 -0

107 lines not on this page · raw message

Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#2 lore raw

PATCH 2/6 lib/tests: Unit tests for dynamic subtests

In reply to Petri Latvala
---
 lib/tests/igt_dynamic_subtests.c | 186 +++++++++++++++++++++++++++++++
 lib/tests/meson.build            |   1 +
 2 files changed, 187 insertions(+)
 create mode 100644 lib/tests/igt_dynamic_subtests.c
lib/tests/igt_dynamic_subtests.c new file +186 -0
new file mode 100644
index 00000000..0e693744
@@ -0,0 +1,186 @@
+/*
+ * Copyright © 2019 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#include <errno.h>
+#include <sys/wait.h>
+
+#include "igt_core.h"
+#include "drmtest.h"
+
+#include "igt_tests_common.h"
+
+static int do_fork(void (*test_to_run)(void))
+{
+	int pid, status;
+
+	switch (pid = fork()) {
+	case -1:
+		internal_assert(0);
+	case 0:
+		test_to_run();
+	default:
+		while (waitpid(pid, &status, 0) == -1 &&
+		       errno == EINTR)
+			;
+
+		return status;
+	}
+}
+
+static void dynamic_subtest_in_normal_subtest(void)
+{
+	char prog[] = "igt_no_exit";
+	char *fake_argv[] = {prog};
+	int fake_argc = ARRAY_SIZE(fake_argv);
+
+	igt_subtest_init(fake_argc, fake_argv);
+
+	igt_subtest("normal-subtest") {
+		igt_dynamic_subtest("dynamic") {
+			igt_info("Dynamic subtest in normal subtest\n");
+		}
+	}
+
+	igt_exit();
+}
+
+static void invalid_dynamic_subtest_name(void)
+{
+	char prog[] = "igt_no_exit";
+	char *fake_argv[] = {prog};
+	int fake_argc = ARRAY_SIZE(fake_argv);
+
+	igt_subtest_init(fake_argc, fake_argv);
+
+	igt_dynamic_subtest_container("subtest") {
+		igt_dynamic_subtest("# invalid name !") {
+			igt_info("Invalid dynamic subtest name test\n");
+		}
+	}
+
+	igt_exit();
+}
+
+static void dynamic_subtest_in_toplevel(void)
+{
+	char prog[] = "igt_no_exit";
+	char *fake_argv[] = {prog};
+	int fake_argc = ARRAY_SIZE(fake_argv);
+
+	igt_subtest_init(fake_argc, fake_argv);
+
+	igt_dynamic_subtest("dynamic-subtest-in-toplevel") {
+		igt_info("Dynamic subtests need to be in a subtest\n");
+	}
+
+	igt_exit();
+}
+
+static void subtest_itself_failing(void)
+{
+	char prog[] = "igt_no_exit";
+	char *fake_argv[] = {prog};
+	int fake_argc = ARRAY_SIZE(fake_argv);
+
+	igt_subtest_init(fake_argc, fake_argv);
+
+	igt_dynamic_subtest_container("subtest") {
+		igt_assert(false);
+	}
+
+	igt_exit();
+}
+
+static void subtest_itself_skipping(void)
+{
+	char prog[] = "igt_no_exit";
+	char *fake_argv[] = {prog};
+	int fake_argc = ARRAY_SIZE(fake_argv);
+
+	igt_subtest_init(fake_argc, fake_argv);
+
+	igt_dynamic_subtest_container("subtest") {
+		igt_skip("Skipping\n");
+	}
+
+	igt_exit();
+}
+
+static void dynamic_subtest_failure_leads_to_fail(void)
+{
+	char prog[] = "igt_no_exit";
+	char *fake_argv[] = {prog};
+	int fake_argc = ARRAY_SIZE(fake_argv);
+
+	igt_subtest_init(fake_argc, fake_argv);
+
+	igt_dynamic_subtest_container("subtest") {
+		igt_dynamic_subtest("dynamic") {
+			igt_assert(false);
+		}
+	}
+
+	igt_exit();
+}
+
+static void no_dynamic_subtests_entered_leads_to_skip(void)
+{
+	char prog[] = "igt_no_exit";
+	char *fake_argv[] = {prog};
+	int fake_argc = ARRAY_SIZE(fake_argv);
+
+	igt_subtest_init(fake_argc, fake_argv);
+
+	igt_dynamic_subtest_container("subtest") {
+	}
+
+	igt_exit();
+}
+
+int main(int argc, char **argv)
+{
+	int ret;
+
+	ret = do_fork(dynamic_subtest_in_normal_subtest);
+	internal_assert_wsignaled(ret, SIGABRT);
+
+	ret = do_fork(invalid_dynamic_subtest_name);
+	internal_assert_wsignaled(ret, SIGABRT);
+
+	ret = do_fork(dynamic_subtest_in_toplevel);
+	internal_assert_wsignaled(ret, SIGABRT);
+
+	ret = do_fork(subtest_itself_failing);
+	internal_assert_wsignaled(ret, SIGABRT);
+
+	ret = do_fork(subtest_itself_skipping);
+	internal_assert_wexited(ret, IGT_EXIT_SKIP);
+
+	ret = do_fork(dynamic_subtest_failure_leads_to_fail);
+	internal_assert_wexited(ret, IGT_EXIT_FAILURE);
+
+	ret = do_fork(no_dynamic_subtests_entered_leads_to_skip);
+	internal_assert_wexited(ret, IGT_EXIT_SKIP);
+
+	return 0;
+}
lib/tests/meson.build +1 -0
index b930ee6e..4223bffc 100644
@@ -3,6 +3,7 @@ lib_tests = [
 	'igt_can_fail',
 	'igt_can_fail_simple',
 	'igt_conflicting_args',
+	'igt_dynamic_subtests',
 	'igt_edid',
 	'igt_exit_handler',
 	'igt_fork',
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#3 lore raw

PATCH 3/6 runner/resultgen: Refactor output parsing

In reply to Petri Latvala

Instead of searching back and forth for proper lines, first find all lines that we could be interested in with one pass through the output, and use the positions of found lines to delimit the extracted outputs.

---
 runner/resultgen.c | 259 +++++++++++++++++++++++----------------------
 1 file changed, 134 insertions(+), 125 deletions(-)
runner/resultgen.c +134 -125
index 7b4cd519..badf6502 100644
@@ -10,6 +10,7 @@
 
 #include <json.h>
 
+#include "igt_aux.h"
 #include "igt_core.h"
 #include "resultgen.h"
 #include "settings.h"
@@ -59,28 +60,7 @@ static char *find_line_starting_with(char *haystack, const char *needle, char *e
 	return NULL;
 }
 
-static char *find_line_starting_with_either(char *haystack,
-					    const char *needle1,
-					    const char *needle2,
-					    char *end)
-{
-	while (haystack < end) {
-		char *line_end = memchr(haystack, '\n', end - haystack);
-		size_t linelen = line_end != NULL ? line_end - haystack : end - haystack;
-		if ((linelen >= strlen(needle1) && !memcmp(haystack, needle1, strlen(needle1))) ||
-		    (linelen >= strlen(needle2) && !memcmp(haystack, needle2, strlen(needle2))))
-			return haystack;
-
-		if (line_end == NULL)
-			return NULL;
-
-		haystack = line_end + 1;
-	}
-
-	return NULL;
-}
-
-static char *next_line(char *line, char *bufend)
+static const char *next_line(const char *line, const char *bufend)
 {
 	char *ret;
 
@@ -97,45 +77,6 @@ static char *next_line(char *line, char *bufend)
 		return NULL;
 }
 
-static char *find_line_after_last(char *begin,
-				  const char *needle1,
-				  const char *needle2,
-				  char *end)
-{
-	char *one, *two;
-	char *current_pos = begin;
-	char *needle1_newline = malloc(strlen(needle1) + 2);
-	char *needle2_newline = malloc(strlen(needle2) + 2);
-
-	needle1_newline[0] = needle2_newline[0] = '\n';
-	strcpy(needle1_newline + 1, needle1);
-	strcpy(needle2_newline + 1, needle2);
-
-	while (true) {
-		one = memmem(current_pos, end - current_pos, needle1_newline, strlen(needle1_newline));
-		two = memmem(current_pos, end - current_pos, needle2_newline, strlen(needle2_newline));
-		if (one == NULL && two == NULL)
-			break;
-
-		if (one != NULL && current_pos < one)
-			current_pos = one;
-		if (two != NULL && current_pos < two)
-			current_pos = two;
-
-		one = next_line(current_pos, end);
-		if (one != NULL)
-			current_pos = one;
-	}
-	free(needle1_newline);
-	free(needle2_newline);
-
-	one = memchr(current_pos, '\n', end - current_pos);
-	if (one != NULL)
-		return ++one;
-
-	return current_pos;
-}
-
 static size_t count_lines(const char *buf, const char *bufend)
 {
 	size_t ret = 0;
@@ -166,7 +107,7 @@ static const struct {
 	{ "CRASH", "crash" },
 	{ "TIMEOUT", "timeout" },
 };
-static void parse_result_string(char *resultstring, size_t len, const char **result, double *time)
+static void parse_result_string(const char *resultstring, size_t len, const char **result, double *time)
 {
 	size_t i;
 	size_t wordlen = 0;
@@ -206,19 +147,20 @@ static void parse_result_string(char *resultstring, size_t len, const char **res
 	}
 }
 
-static void parse_subtest_result(char *subtest, const char **result, double *time, char *buf, char *bufend)
+static void parse_subtest_result(const char *subtest,
+				 const char **result,
+				 double *time,
+				 const char *line,
+				 const char *bufend)
 {
-	char *line;
-	char *line_end;
-	char *resultstring;
-	size_t linelen;
+	const char *resultstring;
 	size_t subtestlen = strlen(subtest);
+	const char *line_end;
+	size_t linelen;
 
-	*result = NULL;
+	*result = "incomplete";
 	*time = 0.0;
 
-	if (!buf) return;
-
 	/*
 	 * The result line structure is:
 	 *
@@ -235,18 +177,17 @@ static void parse_subtest_result(char *subtest, const char **result, double *tim
 	 * Subtest subtestname: PASS (0.003s)
 	 */
 
-	line = find_line_starting_with(buf, SUBTEST_RESULT, bufend);
-	if (!line) {
-		*result = "incomplete";
+	if (!line)
 		return;
-	}
 
 	line_end = memchr(line, '\n', bufend - line);
 	linelen = line_end != NULL ? line_end - line : bufend - line;
 
 	if (strlen(SUBTEST_RESULT) + subtestlen + strlen(": ") > linelen ||
-	    strncmp(line + strlen(SUBTEST_RESULT), subtest, subtestlen))
-		return parse_subtest_result(subtest, result, time, line + linelen, bufend);
+	    strncmp(line + strlen(SUBTEST_RESULT), subtest, subtestlen)) {
+		/* This is not the correct result line */
+		return;
+	}
 
 	resultstring = line + strlen(SUBTEST_RESULT) + subtestlen + strlen(": ");
 	parse_result_string(resultstring, linelen - (resultstring - line), result, time);
@@ -308,6 +249,59 @@ static void set_runtime(struct json_object *obj, double time)
 			       json_object_new_double(time));
 }
 
+struct match_item
+{
+	const char *where;
+	const char *what;
+};
+
+struct matches
+{
+	struct match_item *items;
+	size_t size;
+};
+
+static void match_add(struct matches *matches, const char *where, const char *what)
+{
+	struct match_item newitem = { where, what };
+
+	matches->size++;
+	matches->items = realloc(matches->items, matches->size * sizeof(*matches->items));
+	matches->items[matches->size - 1] = newitem;
+}
+
+static struct matches find_matches(const char *buf, const char *bufend,
+				   const char **needles)
+{
+	struct matches ret = {};
+
+	while (buf < bufend) {
+		const char **needle;
+
+		for (needle = needles; *needle; needle++) {
+			if (bufend - buf < strlen(*needle))
+				continue;
+
+			if (!memcmp(buf, *needle, strlen(*needle))) {
+				match_add(&ret, buf, *needle);
+				goto end_find;
+			}
+		}
+
+	end_find:
+		buf = next_line(buf, bufend);
+		if (!buf)
+			break;
+	}
+
+	return ret;
+}
+
+static void free_matches(struct matches *matches)
+{
+	free(matches->items);
+}
+
 static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtests *subtests,
 			     struct json_object *tests)
@@ -318,7 +312,13 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 	char *igt_version = NULL;
 	size_t igt_version_len = 0;
 	struct json_object *current_test = NULL;
-	size_t i;
+	const char *needles[] = {
+		STARTING_SUBTEST,
+		SUBTEST_RESULT,
+		NULL
+	};
+	struct matches matches = {};
+	size_t i, k;
 
 	if (fstat(fd, &statbuf))
 		return false;
@@ -363,10 +363,13 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		return true;
 	}
 
+	matches = find_matches(buf, bufend, needles);
+
 	for (i = 0; i < subtests->size; i++) {
 		char *this_sub_begin, *this_sub_result;
+		int begin_idx = -1, result_idx = -1;
 		const char *resulttext;
-		char *beg, *end, *startline;
+		const char *beg, *end;
 		double time;
 		int begin_len;
 		int result_len;
@@ -382,66 +385,68 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 			return false;
 		}
 
-		beg = find_line_starting_with(buf, this_sub_begin, bufend);
-		end = find_line_starting_with(buf, this_sub_result, bufend);
-		startline = beg;
+		for (k = 0; k < matches.size; ++k) {
+			if (matches.items[k].what == STARTING_SUBTEST &&
+			    !memcmp(matches.items[k].where,
+				    this_sub_begin,
+				    min(begin_len, bufend - matches.items[k].where))) {
+				beg = matches.items[k].where;
+				begin_idx = k;
+			}
+			if (matches.items[k].what == SUBTEST_RESULT &&
+			    !memcmp(matches.items[k].where,
+				    this_sub_result,
+				    min(result_len, bufend - matches.items[k].where))) {
+				end = matches.items[k].where;
+				result_idx = k;
+			}
+		}
 
 		free(this_sub_begin);
 		free(this_sub_result);
 
-		if (beg == NULL && end == NULL) {
+		if (begin_idx < 0 && result_idx < 0) {
 			/* No output at all */
 			beg = bufend;
 			end = bufend;
-		}
-
-		if (beg == NULL) {
+		} else if (begin_idx < 0) {
 			/*
-			 * Subtest didn't start, probably skipped from
-			 * fixture already. Start from the result
-			 * line, it gets adjusted below.
+			 * Subtest didn't start, but we have the
+			 * result. Probably because an igt_fixture
+			 * made it fail/skip.
+			 *
+			 * Start the output right after the previous
+			 * subtest result/start.
 			 */
-			beg = end;
+			if (result_idx > 0)
+				beg = next_line(matches.items[result_idx - 1].where, bufend);
+			else
+				beg = buf;
+		} else {
+			/* Stretch output beginning backwards */
+			if (begin_idx == 0)
+				beg = buf;
+			else
+				beg = next_line(matches.items[begin_idx - 1].where, bufend);
 		}
 
-		/* Include the output after the previous subtest output */
-		beg = find_line_after_last(buf,
-					   STARTING_SUBTEST,
-					   SUBTEST_RESULT,
-					   beg);
-
-		if (end == NULL) {
-			/* Incomplete result. Find the next starting subtest or result. */
-			end = next_line(startline, bufend);
-			if (end != NULL) {
-				end = find_line_starting_with_either(end,
-								     STARTING_SUBTEST,
-								     SUBTEST_RESULT,
-								     bufend);
-			}
-			if (end == NULL) {
-				end = bufend;
-			}
-		} else {
+		if (result_idx < 0 && begin_idx < 0) {
+			/* No output at all: Already handled above */
+		} else if (result_idx < 0) {
 			/*
-			 * Now pointing to the line where this sub's
-			 * result is. We need to include that of
-			 * course.
+			 * Incomplete result. Include output up to the
+			 * next starting subtest or result.
 			 */
-			char *nexttest = next_line(end, bufend);
-
-			/* Stretch onwards until the next subtest begins or ends */
-			if (nexttest != NULL) {
-				nexttest = find_line_starting_with_either(nexttest,
-									  STARTING_SUBTEST,
-									  SUBTEST_RESULT,
-									  bufend);
-			}
-			if (nexttest != NULL) {
-				end = nexttest;
-			} else {
+			if (begin_idx < matches.size - 1)
+				end = matches.items[begin_idx + 1].where;
+			else
+				end = bufend;
+		} else {
+			/* Stretch output to the next starting subtest or result. */
+			if (result_idx < matches.size - 1)
+				end = matches.items[result_idx + 1].where;
+			else
 				end = bufend;
-			}
 		}
 
 		json_object_object_add(current_test, key,
@@ -454,12 +459,16 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		}
 
 		if (!json_object_object_get_ex(current_test, "result", NULL)) {
-			parse_subtest_result(subtests->names[i], &resulttext, &time, beg, end);
+			parse_subtest_result(subtests->names[i],
+					     &resulttext, &time,
+					     result_idx < 0 ? NULL : matches.items[result_idx].where,
+					     end);
 			set_result(current_test, resulttext);
 			set_runtime(current_test, time);
 		}
 	}
 
+	free_matches(&matches);
 	return true;
 }
 
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#4 lore raw

PATCH 4/6 runner/json_tests: Adapt to better output parsing

In reply to Petri Latvala

Since we actually include the output before the subtest begins now, add it to the reference.jsons where applicable.

---
 .../json_tests_data/aborted-after-a-test/reference.json   | 2 +-
 runner/json_tests_data/dmesg-escapes/reference.json       | 2 +-
 runner/json_tests_data/dmesg-results/reference.json       | 8 ++++----
 .../dmesg-warn-level-one-piglit-style/reference.json      | 4 ++--
 .../dmesg-warn-level-piglit-style/reference.json          | 4 ++--
 runner/json_tests_data/dmesg-warn-level/reference.json    | 4 ++--
 runner/json_tests_data/normal-run/reference.json          | 8 ++++----
 .../notrun-results-multiple-mode/reference.json           | 2 +-
 runner/json_tests_data/notrun-results/reference.json      | 2 +-
 runner/json_tests_data/piglit-style-dmesg/reference.json  | 8 ++++----
 .../warnings-with-dmesg-warns/reference.json              | 8 ++++----
 runner/json_tests_data/warnings/reference.json            | 8 ++++----
 12 files changed, 30 insertions(+), 30 deletions(-)
runner/json_tests_data/aborted-after-a-test/reference.json +1 -1
index 06a8bff6..583242c7 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
runner/json_tests_data/dmesg-escapes/reference.json +1 -1
index a6a70109..70d6b366 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
runner/json_tests_data/dmesg-results/reference.json +4 -4
index 445cd0a0..0edbae93 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"dmesg-warn",
       "time":{
@@ -23,7 +23,7 @@
 	"dmesg-warnings":"<3> [3216186.101159] Warning from kernel\n"
     },
     "igt@successtest@second-subtest":{
-      "out":"Starting subtest: second-subtest\nSubtest second-subtest: FAIL (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: FAIL (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"dmesg-fail",
       "time":{
@@ -49,7 +49,7 @@
 	"dmesg-warnings":"<3> [3216186.101159] Warning from kernel\n"
     },
     "igt@skippers@skip-one":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
@@ -61,7 +61,7 @@
       "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<3> [3216186.101159] Warning from kernel\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@skippers@skip-two":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
runner/json_tests_data/dmesg-warn-level-one-piglit-style/reference.json +2 -2
index 520edf08..4ccb18ae 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"dmesg-warn",
       "time":{
@@ -70,4 +70,4 @@
       }
     }
   }
-}
\ No newline at end of file
+}
runner/json_tests_data/dmesg-warn-level-piglit-style/reference.json +2 -2
index 8cd2eeee..d706ee4c 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"dmesg-warn",
       "time":{
@@ -70,4 +70,4 @@
       }
     }
   }
-}
\ No newline at end of file
+}
runner/json_tests_data/dmesg-warn-level/reference.json +2 -2
index 93d9c6ec..11cc39d9 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"dmesg-warn",
       "time":{
@@ -70,4 +70,4 @@
       }
     }
   }
-}
\ No newline at end of file
+}
runner/json_tests_data/normal-run/reference.json +4 -4
index 5e62579b..982038f9 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
@@ -22,7 +22,7 @@
       "dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n<6> [3216186.101115] [IGT] successtest: starting subtest first-subtest\n<6> [3216186.101160] [IGT] successtest: exiting, ret=0\n<6> [3216186.101299] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@successtest@second-subtest":{
-      "out":"Starting subtest: second-subtest\nSubtest second-subtest: FAIL (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: FAIL (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"fail",
       "time":{
@@ -46,7 +46,7 @@
       "dmesg":"<6> [3216186.123400] Console: switching to colour dummy device 80x25\n<6> [3216186.123414] [IGT] no-subtests: executing\n<6> [3216186.125204] [IGT] no-subtests: exiting, ret=0\n<6> [3216186.125374] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@skippers@skip-one":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
@@ -58,7 +58,7 @@
       "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@skippers@skip-two":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
runner/json_tests_data/notrun-results-multiple-mode/reference.json +1 -1
index de1c3c31..492c0a9e 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
runner/json_tests_data/notrun-results/reference.json +1 -1
index 6b5ff69b..49a2f693 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
runner/json_tests_data/piglit-style-dmesg/reference.json +4 -4
index 59f46595..45d6108e 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"warn",
       "time":{
@@ -22,7 +22,7 @@
 	"dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n<6> [3216186.101115] [IGT] successtest: starting subtest first-subtest\n<3> [3216186.101159] Warning from kernel\n<6> [3216186.101160] [IGT] successtest: exiting, ret=0\n<6> [3216186.101299] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@successtest@second-subtest":{
-      "out":"Starting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
@@ -46,7 +46,7 @@
       "dmesg":"<6> [3216186.123400] Console: switching to colour dummy device 80x25\n<6> [3216186.123414] [IGT] no-subtests: executing\n<6> [3216186.125204] [IGT] no-subtests: exiting, ret=0\n<6> [3216186.125374] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@skippers@skip-one":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
@@ -58,7 +58,7 @@
       "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@skippers@skip-two":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
runner/json_tests_data/warnings-with-dmesg-warns/reference.json +4 -4
index f70990e8..fa571703 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"dmesg-warn",
       "time":{
@@ -23,7 +23,7 @@
 	"dmesg-warnings":"<3> [3216186.101159] Warning from kernel\n"
     },
     "igt@successtest@second-subtest":{
-      "out":"Starting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
@@ -47,7 +47,7 @@
       "dmesg":"<6> [3216186.123400] Console: switching to colour dummy device 80x25\n<6> [3216186.123414] [IGT] no-subtests: executing\n<6> [3216186.125204] [IGT] no-subtests: exiting, ret=0\n<6> [3216186.125374] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@skippers@skip-one":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
@@ -59,7 +59,7 @@
       "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@skippers@skip-two":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
runner/json_tests_data/warnings/reference.json +4 -4
index dade0439..53e0c3c7 100644
@@ -10,7 +10,7 @@
   },
   "tests":{
     "igt@successtest@first-subtest":{
-      "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"warn",
       "time":{
@@ -22,7 +22,7 @@
       "dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n<6> [3216186.101115] [IGT] successtest: starting subtest first-subtest\n<6> [3216186.101160] [IGT] successtest: exiting, ret=0\n<6> [3216186.101299] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@successtest@second-subtest":{
-      "out":"Starting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
@@ -46,7 +46,7 @@
       "dmesg":"<6> [3216186.123400] Console: switching to colour dummy device 80x25\n<6> [3216186.123414] [IGT] no-subtests: executing\n<6> [3216186.125204] [IGT] no-subtests: exiting, ret=0\n<6> [3216186.125374] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@skippers@skip-one":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
@@ -58,7 +58,7 @@
       "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@skippers@skip-two":{
-      "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
+      "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n",
       "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"skip",
       "time":{
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#5 lore raw

PATCH 5/6 runner: Parse dynamic subtest outputs and results

In reply to Petri Latvala

If binary 'bin' has a subtest 'sub', which has dynamic subtests 'foo' and 'bar', results.json will now have "subtests" by the names igt@bin@sub@foo and igt@bin@sub@bar, with data as expected of normal subtests.

---
 runner/job_list.c       |  11 ++
 runner/job_list.h       |   3 +
 runner/output_strings.h |  29 +++-
 runner/resultgen.c      | 340 +++++++++++++++++++++++++++++++++-------
 4 files changed, 321 insertions(+), 62 deletions(-)
runner/job_list.c +11 -0
index 5283fd72..93cede75 100644
@@ -357,6 +357,17 @@ void generate_piglit_name(const char *binary, const char *subtest,
 	free(lc_subtest);
 }
 
+void generate_piglit_name_for_dynamic(const char *base_piglit_name,
+				      const char *dynamic_subtest,
+				      char *namebuf, size_t namebuf_size)
+{
+	char *lc_dynamic = lowercase(dynamic_subtest);
+
+	snprintf(namebuf, namebuf_size, "%s@%s", base_piglit_name, lc_dynamic);
+
+	free(lc_dynamic);
+}
+
 void init_job_list(struct job_list *job_list)
 {
 	memset(job_list, 0, sizeof(*job_list));
runner/job_list.h +3 -0
index 39c9b863..30b4a252 100644
@@ -29,6 +29,9 @@ struct job_list
 
 void generate_piglit_name(const char *binary, const char *subtest,
 			  char *namebuf, size_t namebuf_size);
+void generate_piglit_name_for_dynamic(const char *base_piglit_name,
+				      const char *dynamic_subtest,
+				      char *namebuf, size_t namebuf_size);
 
 void init_job_list(struct job_list *job_list);
 void free_job_list(struct job_list *job_list);
runner/output_strings.h +28 -1
index 1e52c2ce..892895ea 100644
@@ -15,10 +15,28 @@ static const char STARTING_SUBTEST[] = "Starting subtest: ";
  *
  * Examples:
  * Subtest subtestname: SKIP
- * Subtest subtestname: PASS (0.003s)
+ * Subtest subtestname: SUCCESS (0.003s)
  */
 static const char SUBTEST_RESULT[] = "Subtest ";
 
+/*
+ * Output when a dynamic subtest has begun. Is followed by the subtest name.
+ *
+ * Example:
+ * Starting dynamic subtest: subtestname
+ */
+static const char STARTING_DYNAMIC_SUBTEST[] = "Starting dynamic subtest: ";
+
+/*
+ * Output when a dynamic subtest has ended. Is followed by the subtest name
+ * and optionally its runtime.
+ *
+ * Examples:
+ * Dynamic subtest subtestname: SKIP
+ * Dynamic subtest subtestname: SUCCESS (0.003s)
+ */
+static const char DYNAMIC_SUBTEST_RESULT[] = "Dynamic subtest ";
+
 /*
  * Output in dmesg when a subtest has begin. Is followed by the subtest name.
  *
@@ -27,6 +45,15 @@ static const char SUBTEST_RESULT[] = "Subtest ";
  */
 static const char STARTING_SUBTEST_DMESG[] = ": starting subtest ";
 
+/*
+ * Output in dmesg when a dynamic subtest has begin. Is followed by
+ * the subtest name.
+ *
+ * Example:
+ * [IGT] test-binary-name: starting dynamic subtest subtestname
+ */
+static const char STARTING_DYNAMIC_SUBTEST_DMESG[] = ": starting dynamic subtest ";
+
 /*
  * Output when a test process is executed.
  *
runner/resultgen.c +279 -61

561 lines not on this page · raw message

Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#6 lore raw

PATCH 6/6 runner/json_tests: Test dynamic subtests

In reply to Petri Latvala
---
 .../dynamic-subtests/0/dmesg.txt              |   7 +
 .../dynamic-subtests/0/err.txt                |  36 ++++
 .../dynamic-subtests/0/journal.txt            |   2 +
 .../dynamic-subtests/0/out.txt                |  19 +++
 .../dynamic-subtests/1/dmesg.txt              |   5 +
 .../dynamic-subtests/1/err.txt                |   2 +
 .../dynamic-subtests/1/journal.txt            |   2 +
 .../dynamic-subtests/1/out.txt                |   5 +
 .../dynamic-subtests/2/dmesg.txt              |  10 ++
 .../dynamic-subtests/2/err.txt                |   6 +
 .../dynamic-subtests/2/journal.txt            |   4 +
 .../dynamic-subtests/2/out.txt                |   8 +
 .../dynamic-subtests/README.txt               |   2 +
 .../dynamic-subtests/endtime.txt              |   1 +
 .../dynamic-subtests/joblist.txt              |   3 +
 .../dynamic-subtests/metadata.txt             |  12 ++
 .../dynamic-subtests/reference.json           | 156 ++++++++++++++++++
 .../dynamic-subtests/starttime.txt            |   1 +
 .../dynamic-subtests/uname.txt                |   1 +
 runner/runner_json_tests.c                    |   3 +-
 20 files changed, 284 insertions(+), 1 deletion(-)
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/README.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/endtime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/joblist.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/metadata.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/reference.json
 create mode 100644 runner/json_tests_data/dynamic-subtests/starttime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/uname.txt
runner/json_tests_data/dynamic-subtests/0/dmesg.txt new file +7 -0
new file mode 100644
index 00000000..c87b4271
@@ -0,0 +1,7 @@
+6,1157,23426155175691,-;Console: switching to colour dummy device 80x25
+14,1158,23426155175708,-;[IGT] dynamic: executing
+14,1159,23426155184875,-;[IGT] dynamic: starting subtest debug-log-checking
+14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-is-dynamic-1
+14,1161,23426155240164,-;[IGT] dynamic: starting dynamic subtest this-is-dynamic-2
+14,1162,23426155293846,-;[IGT] dynamic: exiting, ret=98
+6,1163,23426155294003,-;Console: switching to colour frame buffer device 240x75
runner/json_tests_data/dynamic-subtests/0/err.txt new file +36 -0
new file mode 100644
index 00000000..3c280788
@@ -0,0 +1,36 @@
+Starting subtest: debug-log-checking
+Starting dynamic subtest: this-is-dynamic-1
+(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ../runner/testdata/dynamic.c:8:
+(dynamic:20904) CRITICAL: Failed assertion: false
+Dynamic subtest this-is-dynamic-1 failed.
+**** DEBUG ****
+(dynamic:20904) DEBUG: This print is from 1
+(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ../runner/testdata/dynamic.c:8:
+(dynamic:20904) CRITICAL: Failed assertion: false
+(dynamic:20904) igt_core-INFO: Stack trace:
+(dynamic:20904) igt_core-INFO:   #0 ../lib/igt_core.c:1607 __igt_fail_assert()
+(dynamic:20904) igt_core-INFO:   #1 ../runner/testdata/dynamic.c:11 __real_main3()
+(dynamic:20904) igt_core-INFO:   #2 ../runner/testdata/dynamic.c:3 main()
+(dynamic:20904) igt_core-INFO:   #3 ../csu/libc-start.c:342 __libc_start_main()
+(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]
+****  END  ****
+Dynamic subtest this-is-dynamic-1: FAIL (0.055s)
+Starting dynamic subtest: this-is-dynamic-2
+(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ../runner/testdata/dynamic.c:13:
+(dynamic:20904) CRITICAL: Failed assertion: false
+Dynamic subtest this-is-dynamic-2 failed.
+**** DEBUG ****
+(dynamic:20904) DEBUG: This print is from 2
+(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ../runner/testdata/dynamic.c:13:
+(dynamic:20904) CRITICAL: Failed assertion: false
+(dynamic:20904) igt_core-INFO: Stack trace:
+(dynamic:20904) igt_core-INFO:   #0 ../lib/igt_core.c:1607 __igt_fail_assert()
+(dynamic:20904) igt_core-INFO:   #1 ../runner/testdata/dynamic.c:5 __real_main3()
+(dynamic:20904) igt_core-INFO:   #2 ../runner/testdata/dynamic.c:3 main()
+(dynamic:20904) igt_core-INFO:   #3 ../csu/libc-start.c:342 __libc_start_main()
+(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]
+****  END  ****
+Dynamic subtest this-is-dynamic-2: FAIL (0.054s)
+Subtest debug-log-checking failed.
+No log.
+Subtest debug-log-checking: FAIL (0.109s)
runner/json_tests_data/dynamic-subtests/0/journal.txt new file +2 -0
new file mode 100644
index 00000000..aa8cef0c
@@ -0,0 +1,2 @@
+debug-log-checking
+exit:98 (0.130s)
runner/json_tests_data/dynamic-subtests/0/out.txt new file +19 -0
new file mode 100644
index 00000000..286da3f3
@@ -0,0 +1,19 @@
+IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)
+Starting subtest: debug-log-checking
+Starting dynamic subtest: this-is-dynamic-1
+Stack trace:
+  #0 ../lib/igt_core.c:1607 __igt_fail_assert()
+  #1 ../runner/testdata/dynamic.c:11 __real_main3()
+  #2 ../runner/testdata/dynamic.c:3 main()
+  #3 ../csu/libc-start.c:342 __libc_start_main()
+  #4 [_start+0x2a]
+Dynamic subtest this-is-dynamic-1: FAIL (0.055s)
+Starting dynamic subtest: this-is-dynamic-2
+Stack trace:
+  #0 ../lib/igt_core.c:1607 __igt_fail_assert()
+  #1 ../runner/testdata/dynamic.c:5 __real_main3()
+  #2 ../runner/testdata/dynamic.c:3 main()
+  #3 ../csu/libc-start.c:342 __libc_start_main()
+  #4 [_start+0x2a]
+Dynamic subtest this-is-dynamic-2: FAIL (0.054s)
+Subtest debug-log-checking: FAIL (0.109s)
runner/json_tests_data/dynamic-subtests/1/dmesg.txt new file +5 -0
new file mode 100644
index 00000000..d6e16a46
@@ -0,0 +1,5 @@
+6,1164,23426155304955,-;Console: switching to colour dummy device 80x25
+14,1165,23426155304968,-;[IGT] dynamic: executing
+14,1166,23426155308644,-;[IGT] dynamic: starting subtest empty-container
+14,1167,23426155308671,-;[IGT] dynamic: exiting, ret=77
+6,1168,23426155308822,-;Console: switching to colour frame buffer device 240x75
runner/json_tests_data/dynamic-subtests/1/err.txt new file +2 -0
new file mode 100644
index 00000000..6247e714
@@ -0,0 +1,2 @@
+Starting subtest: empty-container
+Subtest empty-container: SKIP (0.000s)
runner/json_tests_data/dynamic-subtests/1/journal.txt new file +2 -0
new file mode 100644
index 00000000..3e4ce5c4
@@ -0,0 +1,2 @@
+empty-container
+exit:77 (0.014s)
runner/json_tests_data/dynamic-subtests/1/out.txt new file +5 -0
new file mode 100644
index 00000000..704734dc
@@ -0,0 +1,5 @@
+IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)
+Starting subtest: empty-container
+This should skip
+No dynamic tests executed.
+Subtest empty-container: SKIP (0.000s)
runner/json_tests_data/dynamic-subtests/2/dmesg.txt new file +10 -0
new file mode 100644
index 00000000..042ea911
@@ -0,0 +1,10 @@
+6,1157,23426155175691,-;Console: switching to colour dummy device 80x25
+14,1158,23426155175708,-;[IGT] dynamic: executing
+14,1159,23426155184875,-;[IGT] dynamic: starting subtest normal
+6,1160,23426155184895,-;Dmesg output for normal
+14,1159,23426155184875,-;[IGT] dynamic: starting subtest incomplete
+14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-is-incomplete
+6,1160,23426155184895,-;Dmesg output for incomplete
+14,1161,23426155240164,-;[IGT] dynamic: starting subtest resume
+14,1162,23426155293846,-;[IGT] dynamic: exiting, ret=0
+6,1163,23426155294003,-;Console: switching to colour frame buffer device 240x75
runner/json_tests_data/dynamic-subtests/2/err.txt new file +6 -0
new file mode 100644
index 00000000..b955af2f
@@ -0,0 +1,6 @@
+Starting subtest: normal
+Subtest normal: SUCCESS (0.100s)
+Starting subtest: incomplete
+Starting dynamic subtest: this-is-incomplete
+Starting subtest: resume
+Subtest resume: SUCCESS (0.109s)
runner/json_tests_data/dynamic-subtests/2/journal.txt new file +4 -0
new file mode 100644
index 00000000..9469e227
@@ -0,0 +1,4 @@
+normal
+incomplete
+resume
+exit:0 (0.130s)
runner/json_tests_data/dynamic-subtests/2/out.txt new file +8 -0
new file mode 100644
index 00000000..0ebc9c38
@@ -0,0 +1,8 @@
+IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)
+Starting subtest: normal
+Subtest normal: SUCCESS (0.100s)
+Starting subtest: incomplete
+Starting dynamic subtest: this-is-incomplete
+This is some output
+Starting subtest: resume
+Subtest resume: SUCCESS (0.109s)
runner/json_tests_data/dynamic-subtests/README.txt new file +2 -0
new file mode 100644
index 00000000..3a63ab17
@@ -0,0 +1,2 @@
+A test with dynamic subtests should generate separate subresults for
+the dynamic tests.
runner/json_tests_data/dynamic-subtests/endtime.txt new file +1 -0
new file mode 100644
index 00000000..5c7608b5
@@ -0,0 +1 @@
+1560163492.410489
runner/json_tests_data/dynamic-subtests/joblist.txt new file +3 -0
new file mode 100644
index 00000000..4fdcc5f1
@@ -0,0 +1,3 @@
+dynamic debug-log-checking
+dynamic empty-container
+dynamic normal,incomplete,resume
runner/json_tests_data/dynamic-subtests/metadata.txt new file +12 -0
new file mode 100644
index 00000000..8c77de01
@@ -0,0 +1,12 @@
+abort_mask : 0
+name : dynamic-subtests
+dry_run : 0
+sync : 0
+log_level : 0
+overwrite : 0
+multiple_mode : 1
+inactivity_timeout : 0
+use_watchdog : 0
+piglit_style_dmesg : 0
+test_root : /path/does/not/exist
+results_path : /path/does/not/exist
runner/json_tests_data/dynamic-subtests/reference.json new file +156 -0

156 lines not on this page · raw message

runner/json_tests_data/dynamic-subtests/starttime.txt new file +1 -0
new file mode 100644
index 00000000..b30b32ee
@@ -0,0 +1 @@
+1560163492.266377
runner/json_tests_data/dynamic-subtests/uname.txt new file +1 -0
new file mode 100644
index 00000000..a7aef6f7
@@ -0,0 +1 @@
+Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64
runner/runner_json_tests.c +2 -1
index 7c9540d1..3443bd2b 100644
@@ -161,7 +161,8 @@ static const char *dirnames[] = {
 	"notrun-results-multiple-mode",
 	"dmesg-warn-level",
 	"dmesg-warn-level-piglit-style",
-	"dmesg-warn-level-one-piglit-style"
+	"dmesg-warn-level-one-piglit-style",
+	"dynamic-subtests",
 };
 
 igt_main
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#7 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for Dynamic subtests

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: Dynamic subtests
URL : https://patchwork.freedesktop.org/series/62384/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_6307 -> IGTPW_3176
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/62384/revisions/1/mbox/

Known issues
------------

Here are the changes found in IGTPW_3176 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_ctx_create@basic-files:
- fi-icl-u3: [PASS][1] -> [INCOMPLETE][2] ([fdo#107713] / [fdo#109100])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-icl-u3/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-icl-u3/igt@[email protected]

* igt@gem_exec_fence@basic-busy-default:
- fi-icl-u2: [PASS][3] -> [INCOMPLETE][4] ([fdo#107713])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-icl-u2/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-icl-u2/igt@[email protected]

* igt@i915_selftest@live_contexts:
- fi-bdw-gvtdvm: [PASS][5] -> [DMESG-FAIL][6] ([fdo#110235])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-bdw-gvtdvm/igt@i915_selftest@live_contexts.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-bdw-gvtdvm/igt@i915_selftest@live_contexts.html

#### Possible fixes ####

* igt@gem_exec_suspend@basic-s4-devices:
- fi-blb-e6850: [INCOMPLETE][7] ([fdo#107718]) -> [PASS][8]
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-blb-e6850/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-blb-e6850/igt@[email protected]

* igt@i915_selftest@live_contexts:
- fi-skl-gvtdvm: [DMESG-FAIL][9] ([fdo#110235]) -> [PASS][10]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-skl-gvtdvm/igt@i915_selftest@live_contexts.html
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-skl-gvtdvm/igt@i915_selftest@live_contexts.html

* igt@i915_selftest@live_evict:
- fi-kbl-8809g: [INCOMPLETE][11] ([fdo#110938]) -> [PASS][12]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-kbl-8809g/igt@i915_selftest@live_evict.html
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-kbl-8809g/igt@i915_selftest@live_evict.html

* igt@kms_chamelium@hdmi-hpd-fast:
- fi-kbl-7500u: [FAIL][13] ([fdo#109485]) -> [PASS][14]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-kbl-7500u/igt@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-kbl-7500u/igt@[email protected]

[fdo#107713]: https://bugs.freedesktop.org/show_bug.cgi?id=107713
[fdo#107718]: https://bugs.freedesktop.org/show_bug.cgi?id=107718
[fdo#109100]: https://bugs.freedesktop.org/show_bug.cgi?id=109100
[fdo#109485]: https://bugs.freedesktop.org/show_bug.cgi?id=109485
[fdo#110235]: https://bugs.freedesktop.org/show_bug.cgi?id=110235
[fdo#110938]: https://bugs.freedesktop.org/show_bug.cgi?id=110938

Participating hosts (52 -> 43)
------------------------------

Additional (2): fi-byt-j1900 fi-pnv-d510
Missing (11): fi-kbl-soraka fi-ilk-m540 fi-hsw-4200u fi-skl-guc fi-byt-squawks fi-bsw-cyan fi-byt-clapper fi-icl-y fi-icl-dsi fi-bdw-samus fi-cml-u

Build changes
-------------

* IGT: IGT_5061 -> IGTPW_3176

CI_DRM_6307: bbb61954b61a5411cd6d7c0e98a6b30635d8cf4b @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3176: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/
IGT_5061: c88ced79a7b71aec58f1d9c5c599ac2f431bcf7a @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#8 lore raw
In reply to Petri Latvala
Quoted 26 lines · Petri Latvala

Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?

Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?

Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?

If so, this is your lucky day, for dynamic subtests are here!

While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:

With normal subtests:
for_each_pipe_static(pipe)

   igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
     int fd = drm_open_driver(DRIVER_ANY);
     igt_require(actually_have_this_pipe(fd, pipe));
     igt_assert(do_stuff(fd, pipe));

}

Quoted 9 lines

With dynamic subtests:
igt_subtest_container("fudge")

   int fd = drm_open_driver(DRIVER_ANY);
   for_each_pipe(fd, pipe) {
     igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
       igt_assert(do_stuff(fd, pipe));
     }
   }

How would the below be converted:

int fd = drm_open_driver(DRIVER_ANY);

for_each_pipe_static(pipe) {

  igt_subtest_group {
    igt_fixture {
      igt_require(actually_have_this_pipe(fd, pipe));
    }	

    igt_subtest_f("fudge-with-pipe-%s", pipe_name(pipe))
      fudge(fd, pipe);

    igt_subtest_f("kludge-with-pipe-%s", pipe_name(pipe))
      kludge(fd, pipe);

}
}

?

Quoted 10 lines

Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.

Semantics
=========

Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it igt_subtest_container().

igt_dynamic_subtest_group?

Quoted 4 lines

A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
- igt_assert not allowed
- igt_skip / igt_require is allowed though

Can it do fixtures?

Quoted 44 lines

Any failing dynamic subtest will result in the container reporting a failure.

Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using igt_require in an igt_dynamic_subtest, just don't enter it for such an occasion. In other words:

Do not:
for_each_pipe(fd, pipe)

   igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
     igt_require(pipe_has_things(pipe));
     ...

}

Instead do:

for_each_pipe(fd, pipe) {
   if (!pipe_has_things(pipe))
     continue;
   igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
     ...
   }
}

That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.

Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.

Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of igt_fixture usage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.

Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:

Quoted 7 lines

igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.

igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...

Did not get this part. Is it the first or second option? Or both? If it is the second then CI has to handle dynamic subtests so why we even need the complication to start with?

Regards,

Tvrtko

Quoted 76 lines

Existing tests that are good candidates for conversion to dynamic subtests:

1) Anything using for_each_pipe_static

2) Tests that have one subtest per i915 engine

3) Kernel selftests

Petri Latvala (6):
lib: Introduce dynamic subtests lib/tests: Unit tests for dynamic subtests runner/resultgen: Refactor output parsing runner/json_tests: Adapt to better output parsing
runner: Parse dynamic subtest outputs and results runner/json_tests: Test dynamic subtests

  lib/igt_core.c                                | 119 +++-
  lib/igt_core.h                                |  89 +++
  lib/tests/igt_dynamic_subtests.c              | 186 ++++++
  lib/tests/meson.build                         |   1 +
  runner/job_list.c                             |  11 +
  runner/job_list.h                             |   3 +
  .../aborted-after-a-test/reference.json       |   2 +-
  .../dmesg-escapes/reference.json              |   2 +-
  .../dmesg-results/reference.json              |   8 +-
  .../reference.json                            |   4 +-
  .../reference.json                            |   4 +-
  .../dmesg-warn-level/reference.json           |   4 +-
  .../dynamic-subtests/0/dmesg.txt              |   7 +
  .../dynamic-subtests/0/err.txt                |  36 ++
  .../dynamic-subtests/0/journal.txt            |   2 +
  .../dynamic-subtests/0/out.txt                |  19 +
  .../dynamic-subtests/1/dmesg.txt              |   5 +
  .../dynamic-subtests/1/err.txt                |   2 +
  .../dynamic-subtests/1/journal.txt            |   2 +
  .../dynamic-subtests/1/out.txt                |   5 +
  .../dynamic-subtests/2/dmesg.txt              |  10 +
  .../dynamic-subtests/2/err.txt                |   6 +
  .../dynamic-subtests/2/journal.txt            |   4 +
  .../dynamic-subtests/2/out.txt                |   8 +
  .../dynamic-subtests/README.txt               |   2 +
  .../dynamic-subtests/endtime.txt              |   1 +
  .../dynamic-subtests/joblist.txt              |   3 +
  .../dynamic-subtests/metadata.txt             |  12 +
  .../dynamic-subtests/reference.json           | 156 +++++
  .../dynamic-subtests/starttime.txt            |   1 +
  .../dynamic-subtests/uname.txt                |   1 +
  .../json_tests_data/normal-run/reference.json |   8 +-
  .../reference.json                            |   2 +-
  .../notrun-results/reference.json             |   2 +-
  .../piglit-style-dmesg/reference.json         |   8 +-
  .../warnings-with-dmesg-warns/reference.json  |   8 +-
  .../json_tests_data/warnings/reference.json   |   8 +-
  runner/output_strings.h                       |  29 +-
  runner/resultgen.c                            | 589 ++++++++++++------
  runner/runner_json_tests.c                    |   3 +-
  40 files changed, 1148 insertions(+), 224 deletions(-)

create mode 100644 lib/tests/igt_dynamic_subtests.c
create mode 100644 runner/json_tests_data/dynamic-subtests/0/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/journal.txt

Quoted 9 lines

create mode 100644 runner/json_tests_data/dynamic-subtests/2/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/README.txt create mode 100644 runner/json_tests_data/dynamic-subtests/endtime.txt create mode 100644 runner/json_tests_data/dynamic-subtests/joblist.txt create mode 100644 runner/json_tests_data/dynamic-subtests/metadata.txt create mode 100644 runner/json_tests_data/dynamic-subtests/reference.json create mode 100644 runner/json_tests_data/dynamic-subtests/starttime.txt create mode 100644 runner/json_tests_data/dynamic-subtests/uname.txt

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#9 lore raw
Quoted 53 lines · Tvrtko Ursulin
Quoted 26 lines · Petri Latvala

Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?

Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?

Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?

If so, this is your lucky day, for dynamic subtests are here!

While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:

With normal subtests:
for_each_pipe_static(pipe)

   igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
     int fd = drm_open_driver(DRIVER_ANY);
     igt_require(actually_have_this_pipe(fd, pipe));
     igt_assert(do_stuff(fd, pipe));

}

Quoted 9 lines

With dynamic subtests:
igt_subtest_container("fudge")

   int fd = drm_open_driver(DRIVER_ANY);
   for_each_pipe(fd, pipe) {
     igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
       igt_assert(do_stuff(fd, pipe));
     }
   }

How would the below be converted:

int fd = drm_open_driver(DRIVER_ANY);

for_each_pipe_static(pipe) {

  igt_subtest_group {
    igt_fixture {
      igt_require(actually_have_this_pipe(fd, pipe));
    }	

    igt_subtest_f("fudge-with-pipe-%s", pipe_name(pipe))
      fudge(fd, pipe);

    igt_subtest_f("kludge-with-pipe-%s", pipe_name(pipe))
      kludge(fd, pipe);
Quoted 3 lines

}
}

igt_dynamic_subtest_container("fudge")
for_each_pipe(fd, pipe)

    igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
      fudge(fd, pipe);

igt_dynamic_subtest_container("kludge")
for_each_pipe(fd, pipe)

    igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
      kludge(fd, pipe);

Then you get results for "subtests" called igt@binaryname@fudge and igt@binaryname@fudge@pipe-a and igt@binaryname@fudge@pipe-b, along with igt@binaryname@kludge and igt@binaryname@kludge@pipe-a and igt@binaryname@kludge@pipe-b

Quoted 13 lines

?

Quoted 10 lines

Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.

Semantics
=========

Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it igt_subtest_container().

Quoted 8 lines

igt_dynamic_subtest_group?

Quoted 4 lines

A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
- igt_assert not allowed
- igt_skip / igt_require is allowed though

Can it do fixtures?

No.

For shared initialization, do

igt_dynamic_subtest_container("hello-init") {
/* right here */
int fd = drm_open_driver(DRIVER_ANY);
init_stuffs(fd);
/* init done */

  for (......) {
    igt_dynamic_subtest(...)
}
Quoted 48 lines
Quoted 48 lines

Any failing dynamic subtest will result in the container reporting a failure.

Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using igt_require in an igt_dynamic_subtest, just don't enter it for such an occasion. In other words:

Do not:
for_each_pipe(fd, pipe)

   igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
     igt_require(pipe_has_things(pipe));
     ...

}

Instead do:

for_each_pipe(fd, pipe) {
   if (!pipe_has_things(pipe))
     continue;
   igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
     ...
   }
}

That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.

Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.

Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of igt_fixture usage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.

Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:

igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.

Quoted 8 lines
Quoted 4 lines

igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...

Did not get this part. Is it the first or second option? Or both? If it is the second then CI has to handle dynamic subtests so why we even need the complication to start with?

It's both. The container will report a result (based on what the dynamic tests did) and there's also a result for each dynamic subtest.

The complication is needed because with dynamic subtests the execution entry and results reporting are two different sets, reported subtests being a superset of the entries.

CI will handle (knocking on wood here) dynamic results fine in its current state, because dynamic subtest results walk like a duck, smell like a duck, and look like normal subtest results.

Signature · 5 lines
Petri Latvala
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#10 lore raw
Quoted 176 lines · Petri Latvala
Quoted 42 lines · Tvrtko Ursulin
Quoted 26 lines · Petri Latvala

Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?

Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?

Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?

If so, this is your lucky day, for dynamic subtests are here!

While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:

With normal subtests:
for_each_pipe_static(pipe)

    igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
      int fd = drm_open_driver(DRIVER_ANY);
      igt_require(actually_have_this_pipe(fd, pipe));
      igt_assert(do_stuff(fd, pipe));
    }
Quoted 9 lines

With dynamic subtests:
igt_subtest_container("fudge")

    int fd = drm_open_driver(DRIVER_ANY);
    for_each_pipe(fd, pipe) {
      igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
        igt_assert(do_stuff(fd, pipe));
      }
    }

How would the below be converted:

int fd = drm_open_driver(DRIVER_ANY);

Quoted 14 lines
for_each_pipe_static(pipe) {
   igt_subtest_group {
     igt_fixture {
       igt_require(actually_have_this_pipe(fd, pipe));
     }	

     igt_subtest_f("fudge-with-pipe-%s", pipe_name(pipe))
       fudge(fd, pipe);

     igt_subtest_f("kludge-with-pipe-%s", pipe_name(pipe))
       kludge(fd, pipe);
   }
}

igt_dynamic_subtest_container("fudge")
for_each_pipe(fd, pipe)

     igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
       fudge(fd, pipe);

igt_dynamic_subtest_container("kludge")
for_each_pipe(fd, pipe)

     igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
       kludge(fd, pipe);

Then you get results for "subtests" called igt@binaryname@fudge and igt@binaryname@fudge@pipe-a and igt@binaryname@fudge@pipe-b, along with igt@binaryname@kludge and igt@binaryname@kludge@pipe-a and igt@binaryname@kludge@pipe-b

Quoted 13 lines

?

Quoted 10 lines

Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.

Semantics
=========

Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it igt_subtest_container().

Quoted 8 lines

igt_dynamic_subtest_group?

Quoted 4 lines

A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
- igt_assert not allowed
- igt_skip / igt_require is allowed though

Can it do fixtures?

No.

For shared initialization, do

igt_dynamic_subtest_container("hello-init") {
   /* right here */
   int fd = drm_open_driver(DRIVER_ANY);
   init_stuffs(fd);
   /* init done */

   for (......) {
     igt_dynamic_subtest(...)
}
Quoted 48 lines
Quoted 48 lines

Any failing dynamic subtest will result in the container reporting a failure.

Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using igt_require in an igt_dynamic_subtest, just don't enter it for such an occasion. In other words:

Do not:
for_each_pipe(fd, pipe)

    igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
      igt_require(pipe_has_things(pipe));
      ...
    }

Instead do:

for_each_pipe(fd, pipe) {
    if (!pipe_has_things(pipe))
      continue;
    igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
      ...
    }
}

That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.

Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.

Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of igt_fixture usage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.

Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:

igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.

Quoted 8 lines
Quoted 4 lines

igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...

Did not get this part. Is it the first or second option? Or both? If it is the second then CI has to handle dynamic subtests so why we even need the complication to start with?

It's both. The container will report a result (based on what the dynamic tests did) and there's also a result for each dynamic subtest.

Quoted 7 lines

The complication is needed because with dynamic subtests the execution entry and results reporting are two different sets, reported subtests being a superset of the entries.

CI will handle (knocking on wood here) dynamic results fine in its current state, because dynamic subtest results walk like a duck, smell like a duck, and look like normal subtest results.

I am confused by all this. Could we then achieve the same result by just adding the ability to hide subtests from test enumeration?

igt_subtest_group {
	igt_subtest("a");
}
igt_dynamic_subtest_group("group") {
	igt_subtest("b");
}
# ./test --list-subtests
a
group
# .test --list-all-subtests
a
group-b
# ./test --r group
PASS
# ./test --r group-b
PASS

Would the above satisfy CI requirements? Sounds like it would be easier to implement wrt needed code base changes in individual tests and possibly the IGT core code would not be difficult either? Just needs to track in what kind of subgroup it is to decide what to do when listing/running tests.

Regards,

Tvrtko

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#11 lore raw
Quoted 22 lines · Tvrtko Ursulin

I am confused by all this. Could we then achieve the same result by just adding the ability to hide subtests from test enumeration?

igt_subtest_group {
	igt_subtest("a");
}
igt_dynamic_subtest_group("group") {
	igt_subtest("b");
}
# ./test --list-subtests
a
group
# .test --list-all-subtests
a
group-b
# ./test --r group
PASS
Quoted 7 lines
# ./test --r group-b
PASS

Would the above satisfy CI requirements? Sounds like it would be easier to implement wrt needed code base changes in individual tests and possibly the IGT core code would not be difficult either? Just needs to track in what kind of subgroup it is to decide what to do when listing/running tests.

With sufficient fiddling of names, this is exactly what this series does.

igt_subtest_group {
	igt_subtest("a");
}
igt_dynamic_subtest_container("group") {
	igt_dynamic_subtest("b");
}
# ./test --list-subtests
a
group
# .test --list-all-subtests
error: nope
# ./test --r group
Dynamic "b": PASS
PASS
# ./test --r group --dynamic-subtest b
Dynamic "b": PASS
PASS

The listing of dynamic subtests is not implemented as that is definitely not trivial.

Signature · 5 lines
Petri Latvala
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#12 lore raw
Quoted 54 lines · Petri Latvala
Quoted 22 lines · Tvrtko Ursulin

I am confused by all this. Could we then achieve the same result by just adding the ability to hide subtests from test enumeration?

igt_subtest_group {
	igt_subtest("a");
}
igt_dynamic_subtest_group("group") {
	igt_subtest("b");
}
# ./test --list-subtests
a
group
# .test --list-all-subtests
a
group-b
# ./test --r group
PASS
Quoted 7 lines
# ./test --r group-b
PASS

Would the above satisfy CI requirements? Sounds like it would be easier to implement wrt needed code base changes in individual tests and possibly the IGT core code would not be difficult either? Just needs to track in what kind of subgroup it is to decide what to do when listing/running tests.

With sufficient fiddling of names, this is exactly what this series does.

igt_subtest_group {
	igt_subtest("a");
}
igt_dynamic_subtest_container("group") {
	igt_dynamic_subtest("b");
}
# ./test --list-subtests
a
group
# .test --list-all-subtests
error: nope
# ./test --r group
Dynamic "b": PASS
PASS
Quoted 8 lines
# ./test --r group --dynamic-subtest b
Dynamic "b": PASS
PASS

The listing of dynamic subtests is not implemented as that is definitely not trivial.

I think trying to list dynamic subtests without running them defeats the entire point here. At least it would get really tricky to tell "hw/driver acccess necessary to enumerate tests" from "hw/driver access that actually tests something". E.g. with atomic TEST_ONLY you might have to actually construct all the possible configs to figure out which ones are supported. -Daniel

Quoted 8 lines
Signature · 6 lines
-- 
Petri Latvala
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#13 lore raw

[igt-dev] ✓ Fi.CI.IGT: success for Dynamic subtests

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: Dynamic subtests
URL : https://patchwork.freedesktop.org/series/62384/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_6307_full -> IGTPW_3176_full
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/62384/revisions/1/mbox/

Known issues
------------

Here are the changes found in IGTPW_3176_full that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_eio@in-flight-contexts-10ms:
- shard-kbl: [PASS][1] -> [DMESG-WARN][2] ([fdo#110913 ]) +2 similar issues
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl1/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl3/igt@[email protected]
- shard-snb: [PASS][3] -> [DMESG-WARN][4] ([fdo#110789] / [fdo#110913 ]) +1 similar issue
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb5/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb4/igt@[email protected]

* igt@gem_persistent_relocs@forked-interruptible-faulting-reloc-thrashing:
- shard-apl: [PASS][5] -> [DMESG-WARN][6] ([fdo#110913 ]) +4 similar issues
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-apl2/igt@gem_persistent_relocs@forked-interruptible-faulting-reloc-thrashing.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-apl6/igt@gem_persistent_relocs@forked-interruptible-faulting-reloc-thrashing.html

* igt@gem_userptr_blits@map-fixed-invalidate-overlap-busy-gup:
- shard-snb: [PASS][7] -> [DMESG-WARN][8] ([fdo#110913 ])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb5/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb2/igt@[email protected]

* igt@i915_pm_rc6_residency@rc6-accuracy:
- shard-snb: [PASS][9] -> [SKIP][10] ([fdo#109271])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb1/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb4/igt@[email protected]

* igt@i915_selftest@live_evict:
- shard-glk: [PASS][11] -> [INCOMPLETE][12] ([fdo#103359] / [fdo#110938] / [k.org#198133])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-glk8/igt@i915_selftest@live_evict.html
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-glk7/igt@i915_selftest@live_evict.html

* igt@i915_selftest@live_hangcheck:
- shard-snb: [PASS][13] -> [INCOMPLETE][14] ([fdo#105411])
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb2/igt@i915_selftest@live_hangcheck.html
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb2/igt@i915_selftest@live_hangcheck.html

* igt@i915_suspend@debugfs-reader:
- shard-apl: [PASS][15] -> [DMESG-WARN][16] ([fdo#108566]) +1 similar issue
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-apl3/igt@[email protected]
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-apl4/igt@[email protected]

* igt@kms_cursor_legacy@cursor-vs-flip-varying-size:
- shard-hsw: [PASS][17] -> [INCOMPLETE][18] ([fdo#103540])
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-hsw2/igt@[email protected]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-hsw8/igt@[email protected]

* igt@kms_flip@flip-vs-suspend-interruptible:
- shard-glk: [PASS][19] -> [INCOMPLETE][20] ([fdo#103359] / [k.org#198133])
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-glk9/igt@[email protected]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-glk2/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render:
- shard-hsw: [PASS][21] -> [SKIP][22] ([fdo#109271]) +19 similar issues
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-hsw7/igt@kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render.html
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-hsw1/igt@kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render.html

* igt@kms_vblank@pipe-b-query-forked-busy-hang:
- shard-kbl: [PASS][23] -> [INCOMPLETE][24] ([fdo#103665])
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl1/igt@[email protected]
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl4/igt@[email protected]

#### Possible fixes ####

* igt@gem_eio@in-flight-contexts-1us:
- shard-kbl: [DMESG-WARN][25] ([fdo#110913 ]) -> [PASS][26] +4 similar issues
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl4/igt@[email protected]
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl2/igt@[email protected]

* igt@gem_exec_create@forked:
- shard-glk: [FAIL][27] ([fdo#110946]) -> [PASS][28]
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-glk9/igt@[email protected]
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-glk9/igt@[email protected]

* igt@gem_persistent_relocs@forked-faulting-reloc-thrashing:
- shard-snb: [DMESG-WARN][29] ([fdo#110789] / [fdo#110913 ]) -> [PASS][30]
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb6/igt@[email protected]
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb4/igt@[email protected]

* igt@gem_userptr_blits@map-fixed-invalidate-busy-gup:
- shard-apl: [DMESG-WARN][31] ([fdo#110913 ]) -> [PASS][32] +5 similar issues
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-apl8/igt@[email protected]
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-apl5/igt@[email protected]

* igt@gem_userptr_blits@sync-unmap-cycles:
- shard-snb: [DMESG-WARN][33] ([fdo#110913 ]) -> [PASS][34] +2 similar issues
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb2/igt@[email protected]
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb5/igt@[email protected]

* igt@i915_selftest@live_evict:
- shard-kbl: [INCOMPLETE][35] ([fdo#103665] / [fdo#110938]) -> [PASS][36]
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl3/igt@i915_selftest@live_evict.html
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl2/igt@i915_selftest@live_evict.html

* igt@i915_suspend@sysfs-reader:
- shard-apl: [DMESG-WARN][37] ([fdo#108566]) -> [PASS][38] +5 similar issues
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-apl1/igt@[email protected]
[38]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-apl2/igt@[email protected]

* igt@kms_cursor_legacy@2x-long-flip-vs-cursor-atomic:
- shard-hsw: [SKIP][39] ([fdo#109271]) -> [PASS][40] +20 similar issues
[39]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-hsw1/igt@[email protected]
[40]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-hsw4/igt@[email protected]

* igt@kms_flip@flip-vs-suspend:
- shard-glk: [INCOMPLETE][41] ([fdo#103359] / [k.org#198133]) -> [PASS][42]
[41]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-glk4/igt@[email protected]
[42]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-glk1/igt@[email protected]

* igt@kms_setmode@basic:
- shard-hsw: [FAIL][43] ([fdo#99912]) -> [PASS][44]
[43]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-hsw7/igt@[email protected]
[44]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-hsw6/igt@[email protected]
- shard-kbl: [FAIL][45] ([fdo#99912]) -> [PASS][46]
[45]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl7/igt@[email protected]
[46]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl3/igt@[email protected]

[fdo#103359]: https://bugs.freedesktop.org/show_bug.cgi?id=103359
[fdo#103540]: https://bugs.freedesktop.org/show_bug.cgi?id=103540
[fdo#103665]: https://bugs.freedesktop.org/show_bug.cgi?id=103665
[fdo#105411]: https://bugs.freedesktop.org/show_bug.cgi?id=105411
[fdo#108566]: https://bugs.freedesktop.org/show_bug.cgi?id=108566
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[fdo#110789]: https://bugs.freedesktop.org/show_bug.cgi?id=110789
[fdo#110913 ]: https://bugs.freedesktop.org/show_bug.cgi?id=110913
[fdo#110938]: https://bugs.freedesktop.org/show_bug.cgi?id=110938
[fdo#110946]: https://bugs.freedesktop.org/show_bug.cgi?id=110946
[fdo#99912]: https://bugs.freedesktop.org/show_bug.cgi?id=99912
[k.org#198133]: https://bugzilla.kernel.org/show_bug.cgi?id=198133

Participating hosts (10 -> 5)
------------------------------

Missing (5): shard-skl pig-hsw-4770r pig-glk-j5005 shard-iclb pig-skl-6260u

Build changes
-------------

* IGT: IGT_5061 -> IGTPW_3176
* Piglit: piglit_4509 -> None

CI_DRM_6307: bbb61954b61a5411cd6d7c0e98a6b30635d8cf4b @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3176: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/
IGT_5061: c88ced79a7b71aec58f1d9c5c599ac2f431bcf7a @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev