Skip to contentSkip to conversation

runner: Don't report subtests that have dynamic subtests

PATCH In review Petri Latvala 3 participants 16 messages Dec 2, 2019 – Dec 11, 2019

lore
Share & tools
lore
#0 lore raw

cover runner: Don't report subtests that have dynamic subtests

A feature requested by Martin before full-on conversion to the use of dynamic subtests can begin from cibuglog's perspective.

If we have dynamic subtests to report, don't also report the subtest that contains them.

For example, if we have test "some-test" with subtest "some-subtest" with dynamic subtests "foo" and "bar", we only report igt@some-test@some-subtest@foo and igt@some-test@some-subtest@bar, avoiding igt@some-test@some-subtest altogether.

Parsing dynamic subtests has been changed so that the first dynamic subtest includes all output from the beginning of the subtest so we get the setup messages *somewhere*. Same for the last dynamic subtest, we include all output to the end of the subtest output.

Note that this only changes what gets into results.json; Running the test yourself still gives you all the output.

Petri Latvala (11):
runner/resultgen: Extract igt-version field handling to a helper runner/resultgen: Extract finding begin/end lines for a subtest to a
helper
runner/resultgen: Extract finding begin/end pointers for test output
to helpers
runner/resultgen: Hoist handling of dynamic subtest output to a helper runner/resultgen: Extrude dynamic subtest result texts runner/resultgen: Add support for extra validation hook in
find_matches()
runner/resultgen: Make subtest result line finding more robust runner/json_tests: Adapt to dynamic subtest result parsing runner/resultgen: Don't report subtest result if it has dynamic
subtests
runner/json_tests: Adapt to no longer reporting subtests with dynamic
subtests
runner/json_tests: Add test for parsing dynamic subtests with same
name

 .../0/dmesg.txt                               |  10 +
 .../0/err.txt                                 |  10 +
 .../0/journal.txt                             |   3 +
 .../0/out.txt                                 |  11 +
 .../README.txt                                |   2 +
 .../endtime.txt                               |   1 +
 .../joblist.txt                               |   1 +
 .../metadata.txt                              |  12 +
 .../reference.json                            |  84 +++
 .../starttime.txt                             |   1 +
 .../uname.txt                                 |   1 +
 .../dynamic-subtests/reference.json           |  70 +--
 runner/resultgen.c                            | 528 +++++++++++-------
 runner/runner_json_tests.c                    |   1 +
 14 files changed, 475 insertions(+), 260 deletions(-)
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#1 lore raw

PATCH 1/11 runner/resultgen: Extract igt-version field handling to a helper

---
 runner/resultgen.c | 27 ++++++++++++++-------------
 1 file changed, 14 insertions(+), 13 deletions(-)
runner/resultgen.c +14 -13
index aa438000..31bc0bb1 100644
@@ -365,6 +365,17 @@ static void free_matches(struct matches *matches)
 	free(matches->items);
 }
 
+static void add_igt_version(struct json_object *testobj,
+			    const char *igt_version,
+			    size_t igt_version_len)
+{
+	if (igt_version)
+		json_object_object_add(testobj, "igt-version",
+				       json_object_new_string_len(igt_version,
+								  igt_version_len));
+
+}
+
 static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtest_list *subtests,
 			     struct json_object *tests)
@@ -420,10 +431,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 
 		json_object_object_add(current_test, key,
 				       json_object_new_string_len(buf, statbuf.st_size));
-		if (igt_version)
-			json_object_object_add(current_test, "igt-version",
-					       json_object_new_string_len(igt_version,
-									  igt_version_len));
+		add_igt_version(current_test, igt_version, igt_version_len);
 
 		return true;
 	}
@@ -522,11 +530,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		json_object_object_add(current_test, key,
 				       json_object_new_string_len(beg, end - beg));
 
-		if (igt_version) {
-			json_object_object_add(current_test, "igt-version",
-					       json_object_new_string_len(igt_version,
-									  igt_version_len));
-		}
+		add_igt_version(current_test, igt_version, igt_version_len);
 
 		if (!json_object_object_get_ex(current_test, "result", NULL)) {
 			parse_subtest_result(subtests->subs[i].name,
@@ -608,10 +612,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 
 				json_object_object_add(current_dynamic_test, key,
 						       json_object_new_string_len(dynbeg, dynend - dynbeg));
-				if (igt_version)
-					json_object_object_add(current_dynamic_test, "igt-version",
-							       json_object_new_string_len(igt_version,
-											  igt_version_len));
+				add_igt_version(current_dynamic_test, igt_version, igt_version_len);
 
 				if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
 					const char *dynresulttext;
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#2 lore raw

PATCH 2/11 runner/resultgen: Extract finding begin/end lines for a subtest to a helper

In reply to Petri Latvala
---
 runner/resultgen.c | 99 ++++++++++++++++++++++++----------------------
 1 file changed, 51 insertions(+), 48 deletions(-)
runner/resultgen.c +51 -48
index 31bc0bb1..be058327 100644
@@ -376,6 +376,54 @@ static void add_igt_version(struct json_object *testobj,
 
 }
 
+static int find_subtest_idx_limited(struct matches matches,
+				    const char *bufend,
+				    const char *linekey,
+				    const char *pattern,
+				    const char *subtest_name,
+				    int first,
+				    int last)
+{
+	char *full_line;
+	int line_len;
+	int k;
+
+	/*
+	 * The pattern is a string literal in all call-sites, and we
+	 * don't want to disable this warning globally
+	 */
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wformat-nonliteral"
+	line_len = asprintf(&full_line, pattern, linekey, subtest_name);
+#pragma GCC diagnostic pop
+
+	if (line_len < 0)
+		return -1;
+
+	for (k = first; k < last; k++)
+		if (matches.items[k].what == linekey &&
+		    !memcmp(matches.items[k].where,
+			    full_line,
+			    min(line_len, bufend - matches.items[k].where)))
+			break;
+
+	free(full_line);
+
+	if (k == last)
+		k = -1;
+
+	return k;
+}
+
+static int find_subtest_idx(struct matches matches,
+			    const char *bufend,
+			    const char *linekey,
+			    const char *pattern,
+			    const char *subtest_name)
+{
+	return find_subtest_idx_limited(matches, bufend, linekey, pattern, subtest_name, 0, matches.size);
+}
+
 static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtest_list *subtests,
 			     struct json_object *tests)
@@ -439,44 +487,16 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 	matches = find_matches(buf, bufend, needles);
 
 	for (i = 0; i < subtests->size; i++) {
-		char *this_sub_begin, *this_sub_result;
 		int begin_idx = -1, result_idx = -1;
 		const char *resulttext;
 		const char *beg, *end;
 		double time;
-		int begin_len;
-		int result_len;
 
 		generate_piglit_name(binary, subtests->subs[i].name, piglit_name, sizeof(piglit_name));
 		current_test = get_or_create_json_object(tests, piglit_name);
 
-		begin_len = asprintf(&this_sub_begin, "%s%s\n", STARTING_SUBTEST, subtests->subs[i].name);
-		result_len = asprintf(&this_sub_result, "%s%s: ", SUBTEST_RESULT, subtests->subs[i].name);
-
-		if (begin_len < 0 || result_len < 0) {
-			fprintf(stderr, "Failure generating strings\n");
-			return false;
-		}
-
-		for (k = 0; k < matches.size; k++) {
-			if (matches.items[k].what == STARTING_SUBTEST &&
-			    !memcmp(matches.items[k].where,
-				    this_sub_begin,
-				    min(begin_len, bufend - matches.items[k].where))) {
-				beg = matches.items[k].where;
-				begin_idx = k;
-			}
-			if (matches.items[k].what == SUBTEST_RESULT &&
-			    !memcmp(matches.items[k].where,
-				    this_sub_result,
-				    min(result_len, bufend - matches.items[k].where))) {
-				end = matches.items[k].where;
-				result_idx = k;
-			}
-		}
-
-		free(this_sub_begin);
-		free(this_sub_result);
+		begin_idx = find_subtest_idx(matches, bufend, STARTING_SUBTEST, "%s%s\n", subtests->subs[i].name);
+		result_idx = find_subtest_idx(matches, bufend, SUBTEST_RESULT, "%s%s: ", subtests->subs[i].name);
 
 		if (begin_idx < 0 && result_idx < 0) {
 			/* No output at all */
@@ -562,31 +582,14 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 				int dyn_result_idx = -1;
 				char dynamic_name[256];
 				char dynamic_piglit_name[256];
-				char *this_dyn_result;
-				int dyn_result_len;
 				const char *dynbeg, *dynend;
-				int n;
 
 				if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
 					/* Cannot parse name, just ignore this one */
 					continue;
 				}
 
-				dyn_result_len = asprintf(&this_dyn_result, "%s%s: ", DYNAMIC_SUBTEST_RESULT, dynamic_name);
-				if (dyn_result_len < 0)
-					continue;
-
-				for (n = k + 1; n < result_idx; n++) {
-					if (matches.items[n].what == DYNAMIC_SUBTEST_RESULT &&
-					    !memcmp(matches.items[n].where,
-						    this_dyn_result,
-						    min(dyn_result_len, bufend - matches.items[n].where))) {
-						dyn_result_idx = n;
-						break;
-					}
-				}
-
-				free(this_dyn_result);
+				dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
 
 				if (k == 0)
 					dynbeg = beg;
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#3 lore raw

Re: [igt-dev] [PATCH i-g-t 02/11] runner/resultgen: Extract finding begin/end lines for a subtest to a helper

Quoted 23 lines · Petri Latvala
---
 runner/resultgen.c | 99 ++++++++++++++++++++++++----------------------
 1 file changed, 51 insertions(+), 48 deletions(-)
runner/resultgen.c +10 -0
index 31bc0bb1..be058327 100644
@@ -376,6 +376,54 @@ static void add_igt_version(struct json_object *testobj,
 
 }
 
+static int find_subtest_idx_limited(struct matches matches,
+				    const char *bufend,
+				    const char *linekey,
+				    const char *pattern,
+				    const char *subtest_name,
+				    int first,
+				    int last)
+{
+	char *full_line;
+	int line_len;

runner/resultgen.c static void add_igt_version(struct json_object *testobj, · line 398

Quoted 10 lines
runner/resultgen.c +10 -0
+	int k;
+
+	/*
+	 * The pattern is a string literal in all call-sites, and we
+	 * don't want to disable this warning globally
+	 */
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wformat-nonliteral"
+	line_len = asprintf(&full_line, pattern, linekey, subtest_name);
+#pragma GCC diagnostic pop

I guess that you didn't go for:

__attribute__((format(printf, 5, 6)))
static int find_subtest_idx_limited(struct matches matches,
int first,
int last, const char *bufend, const char *pattern, ...)

because linekey is needed for comparison few lines below this and extracting it with va_start()/va_arg()/va_end() is equally ugly.

How about using an enum for selecting the desired one from a set of locally defined patterns instead of passing it as char* with all this pragma wizardry.

Signature · 6 lines
Cheers,
Arek
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Reviewed-by
#4 lore raw

Re: [igt-dev] [PATCH i-g-t 02/11] runner/resultgen: Extract finding begin/end lines for a subtest to a helper

Quoted 45 lines · Arkadiusz Hiler
Quoted 23 lines · Petri Latvala
---
 runner/resultgen.c | 99 ++++++++++++++++++++++++----------------------
 1 file changed, 51 insertions(+), 48 deletions(-)
runner/resultgen.c +10 -0
index 31bc0bb1..be058327 100644
@@ -376,6 +376,54 @@ static void add_igt_version(struct json_object *testobj,
 
 }
 
+static int find_subtest_idx_limited(struct matches matches,
+				    const char *bufend,
+				    const char *linekey,
+				    const char *pattern,
+				    const char *subtest_name,
+				    int first,
+				    int last)
+{
+	char *full_line;
+	int line_len;

runner/resultgen.c static void add_igt_version(struct json_object *testobj, · line 398

Quoted 10 lines
runner/resultgen.c +10 -0
+	int k;
+
+	/*
+	 * The pattern is a string literal in all call-sites, and we
+	 * don't want to disable this warning globally
+	 */
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wformat-nonliteral"
+	line_len = asprintf(&full_line, pattern, linekey, subtest_name);
+#pragma GCC diagnostic pop

I guess that you didn't go for:

__attribute__((format(printf, 5, 6)))
static int find_subtest_idx_limited(struct matches matches,
int first,
int last, const char *bufend, const char *pattern, ...)

Quoted 6 lines

because linekey is needed for comparison few lines below this and extracting it with va_start()/va_arg()/va_end() is equally ugly.

How about using an enum for selecting the desired one from a set of locally defined patterns instead of passing it as char* with all this pragma wizardry.

Other than the comment above the series is:

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#5 lore raw

PATCH 3/11 runner/resultgen: Extract finding begin/end pointers for test output to helpers

In reply to Petri Latvala
---
 runner/resultgen.c | 211 +++++++++++++++++++++++----------------------
 1 file changed, 109 insertions(+), 102 deletions(-)
runner/resultgen.c +109 -102
index be058327..23bb3331 100644
@@ -424,6 +424,71 @@ static int find_subtest_idx(struct matches matches,
 	return find_subtest_idx_limited(matches, bufend, linekey, pattern, subtest_name, 0, matches.size);
 }
 
+static const char *find_subtest_begin_limit(struct matches matches,
+					    int begin_idx,
+					    int result_idx,
+					    const char *buf,
+					    const char *bufend)
+{
+	/* No matching output at all, include everything */
+	if (begin_idx < 0 && result_idx < 0)
+		return buf;
+
+	if (begin_idx < 0) {
+		/*
+		 * Subtest didn't start, but we have the
+		 * result. Probably because an igt_fixture
+		 * made it fail/skip.
+		 *
+		 * We go backwards one match from the result match,
+		 * and start from the next line.
+		 */
+		if (result_idx > 0)
+			return next_line(matches.items[result_idx - 1].where, bufend);
+		else
+			return buf;
+	}
+
+	/* Include all non-special output before the beginning line. */
+	if (begin_idx == 0)
+		return buf;
+
+	return next_line(matches.items[begin_idx - 1].where, bufend);
+}
+
+static const char *find_subtest_end_limit(struct matches matches,
+					  int begin_idx,
+					  int result_idx,
+					  const char *buf,
+					  const char *bufend)
+{
+	int k;
+
+	/* No matching output at all, include everything */
+	if (begin_idx < 0 && result_idx < 0)
+		return bufend;
+
+	if (result_idx < 0) {
+		/*
+		 * Incomplete result. Include all output up to the
+		 * next starting subtest, or the result of one.
+		 */
+		for (k = begin_idx + 1; k < matches.size; k++) {
+			if (matches.items[k].what == STARTING_SUBTEST ||
+			    matches.items[k].what == SUBTEST_RESULT)
+				return matches.items[k].where;
+		}
+
+		return bufend;
+	}
+
+	/* Include all non-special output to the next match, whatever it is. */
+	if (result_idx < matches.size - 1)
+		return matches.items[result_idx + 1].where;
+
+	return bufend;
+}
+
 static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtest_list *subtests,
 			     struct json_object *tests)
@@ -498,54 +563,8 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		begin_idx = find_subtest_idx(matches, bufend, STARTING_SUBTEST, "%s%s\n", subtests->subs[i].name);
 		result_idx = find_subtest_idx(matches, bufend, SUBTEST_RESULT, "%s%s: ", subtests->subs[i].name);
 
-		if (begin_idx < 0 && result_idx < 0) {
-			/* No output at all */
-			beg = buf;
-			end = bufend;
-		} else if (begin_idx < 0) {
-			/*
-			 * Subtest didn't start, but we have the
-			 * result. Probably because an igt_fixture
-			 * made it fail/skip.
-			 *
-			 * Start the output right after the previous
-			 * subtest result/start.
-			 */
-			if (result_idx > 0)
-				beg = next_line(matches.items[result_idx - 1].where, bufend);
-			else
-				beg = buf;
-		} else {
-			/* Stretch output beginning backwards */
-			if (begin_idx == 0)
-				beg = buf;
-			else
-				beg = next_line(matches.items[begin_idx - 1].where, bufend);
-		}
-
-		if (result_idx < 0 && begin_idx < 0) {
-			/* No output at all: Already handled above */
-		} else if (result_idx < 0) {
-			/*
-			 * Incomplete result. Include output up to the
-			 * next starting subtest or result.
-			 */
-			for (k = begin_idx + 1; k < matches.size; k++) {
-				if (matches.items[k].what == STARTING_SUBTEST ||
-				    matches.items[k].what == SUBTEST_RESULT)
-					break;
-			}
-			if (k < matches.size)
-				end = matches.items[k].where;
-			else
-				end = bufend;
-		} else {
-			/* Stretch output to the next starting subtest or result. */
-			if (result_idx < matches.size - 1)
-				end = matches.items[result_idx + 1].where;
-			else
-				end = bufend;
-		}
+		beg = find_subtest_begin_limit(matches, begin_idx, result_idx, buf, bufend);
+		end = find_subtest_end_limit(matches, begin_idx, result_idx, buf, bufend);
 
 		json_object_object_add(current_test, key,
 				       json_object_new_string_len(beg, end - beg));
@@ -567,68 +586,56 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		 * the subtest output.
 		 */
 		if (result_idx < 0) {
-			/* If the subtest itself is incomplete, stop at the next start of a subtest */
+			/* If the subtest itself is incomplete, stop at the next start/end of a subtest */
 			for (result_idx = begin_idx + 1;
 			     result_idx < matches.size;
 			     result_idx++) {
-				if (matches.items[result_idx].what == STARTING_SUBTEST)
+				if (matches.items[result_idx].what == STARTING_SUBTEST ||
+				    matches.items[result_idx].what == SUBTEST_RESULT)
 					break;
 			}
 		}
 
 		for (k = begin_idx + 1; k < result_idx; k++) {
-			if (matches.items[k].what == STARTING_DYNAMIC_SUBTEST) {
-				struct json_object *current_dynamic_test = NULL;
-				int dyn_result_idx = -1;
-				char dynamic_name[256];
-				char dynamic_piglit_name[256];
-				const char *dynbeg, *dynend;
-
-				if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
-					/* Cannot parse name, just ignore this one */
-					continue;
-				}
-
-				dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
-
-				if (k == 0)
-					dynbeg = beg;
-				else
-					dynbeg = next_line(matches.items[k - 1].where, end);
-
-				if (dyn_result_idx < 0) {
-					if (k < matches.size - 1)
-						dynend = matches.items[k + 1].where;
-					else
-						dynend = end;
-				} else {
-					if (dyn_result_idx < matches.size - 1)
-						dynend = matches.items[dyn_result_idx + 1].where;
-					else
-						dynend = end;
-				}
-
-				generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
-
-				add_dynamic_subtest(&subtests->subs[i], strdup(dynamic_name));
-				current_dynamic_test = get_or_create_json_object(tests, dynamic_piglit_name);
-
-				json_object_object_add(current_dynamic_test, key,
-						       json_object_new_string_len(dynbeg, dynend - dynbeg));
-				add_igt_version(current_dynamic_test, igt_version, igt_version_len);
-
-				if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
-					const char *dynresulttext;
-					double dyntime;
-
-					parse_subtest_result(dynamic_name,
-							     DYNAMIC_SUBTEST_RESULT,
-							     &dynresulttext, &dyntime,
-							     dyn_result_idx < 0 ? NULL : matches.items[dyn_result_idx].where,
-							     dynend);
-					set_result(current_dynamic_test, dynresulttext);
-					set_runtime(current_dynamic_test, dyntime);
-				}
+			struct json_object *current_dynamic_test = NULL;
+			int dyn_result_idx = -1;
+			char dynamic_name[256];
+			char dynamic_piglit_name[256];
+			const char *dynbeg, *dynend;
+
+			if (matches.items[k].what != STARTING_DYNAMIC_SUBTEST)
+				continue;
+
+			if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
+				/* Cannot parse name, just ignore this one */
+				continue;
+			}
+
+			dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
+
+			dynbeg = find_subtest_begin_limit(matches, k, dyn_result_idx, beg, end);
+			dynend = find_subtest_end_limit(matches, k, dyn_result_idx, beg, end);
+
+			generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
+
+			add_dynamic_subtest(&subtests->subs[i], strdup(dynamic_name));
+			current_dynamic_test = get_or_create_json_object(tests, dynamic_piglit_name);
+
+			json_object_object_add(current_dynamic_test, key,
+					       json_object_new_string_len(dynbeg, dynend - dynbeg));
+			add_igt_version(current_dynamic_test, igt_version, igt_version_len);
+
+			if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
+				const char *dynresulttext;
+				double dyntime;
+
+				parse_subtest_result(dynamic_name,
+						     DYNAMIC_SUBTEST_RESULT,
+						     &dynresulttext, &dyntime,
+						     dyn_result_idx < 0 ? NULL : matches.items[dyn_result_idx].where,
+						     dynend);
+				set_result(current_dynamic_test, dynresulttext);
+				set_runtime(current_dynamic_test, dyntime);
 			}
 		}
 	}
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#6 lore raw

PATCH 4/11 runner/resultgen: Hoist handling of dynamic subtest output to a helper

In reply to Petri Latvala
---
 runner/resultgen.c | 136 ++++++++++++++++++++++++++-------------------
 1 file changed, 78 insertions(+), 58 deletions(-)
runner/resultgen.c +78 -58
index 23bb3331..8f4bbe98 100644
@@ -489,6 +489,75 @@ static const char *find_subtest_end_limit(struct matches matches,
 	return bufend;
 }
 
+static void process_dynamic_subtest_output(const char *piglit_name,
+					   const char *igt_version,
+					   size_t igt_version_len,
+					   struct matches matches,
+					   int begin_idx,
+					   int result_idx,
+					   const char *beg,
+					   const char *end,
+					   const char *key,
+					   struct json_object *tests,
+					   struct subtest *subtest)
+{
+	size_t k;
+
+	if (result_idx < 0) {
+		/* If the subtest itself is incomplete, stop at the next start/end of a subtest */
+		for (result_idx = begin_idx + 1;
+		     result_idx < matches.size;
+		     result_idx++) {
+			if (matches.items[result_idx].what == STARTING_SUBTEST ||
+			    matches.items[result_idx].what == SUBTEST_RESULT)
+				break;
+		}
+	}
+
+	for (k = begin_idx + 1; k < result_idx; k++) {
+		struct json_object *current_dynamic_test = NULL;
+		int dyn_result_idx = -1;
+		char dynamic_name[256];
+		char dynamic_piglit_name[256];
+		const char *dynbeg, *dynend;
+
+		if (matches.items[k].what != STARTING_DYNAMIC_SUBTEST)
+			continue;
+
+		if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
+			/* Cannot parse name, just ignore this one */
+			continue;
+		}
+
+		dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
+
+		dynbeg = find_subtest_begin_limit(matches, k, dyn_result_idx, beg, end);
+		dynend = find_subtest_end_limit(matches, k, dyn_result_idx, beg, end);
+
+		generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
+
+		add_dynamic_subtest(subtest, strdup(dynamic_name));
+		current_dynamic_test = get_or_create_json_object(tests, dynamic_piglit_name);
+
+		json_object_object_add(current_dynamic_test, key,
+				       json_object_new_string_len(dynbeg, dynend - dynbeg));
+		add_igt_version(current_dynamic_test, igt_version, igt_version_len);
+
+		if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
+			const char *dynresulttext;
+			double dyntime;
+
+			parse_subtest_result(dynamic_name,
+					     DYNAMIC_SUBTEST_RESULT,
+					     &dynresulttext, &dyntime,
+					     dyn_result_idx < 0 ? NULL : matches.items[dyn_result_idx].where,
+					     dynend);
+			set_result(current_dynamic_test, dynresulttext);
+			set_runtime(current_dynamic_test, dyntime);
+		}
+	}
+}
+
 static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtest_list *subtests,
 			     struct json_object *tests)
@@ -507,7 +576,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		NULL
 	};
 	struct matches matches = {};
-	size_t i, k;
+	size_t i;
 
 	if (fstat(fd, &statbuf))
 		return false;
@@ -581,63 +650,14 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 			set_runtime(current_test, time);
 		}
 
-		/*
-		 * Look for dynamic subtests: If any, they are within
-		 * the subtest output.
-		 */
-		if (result_idx < 0) {
-			/* If the subtest itself is incomplete, stop at the next start/end of a subtest */
-			for (result_idx = begin_idx + 1;
-			     result_idx < matches.size;
-			     result_idx++) {
-				if (matches.items[result_idx].what == STARTING_SUBTEST ||
-				    matches.items[result_idx].what == SUBTEST_RESULT)
-					break;
-			}
-		}
-
-		for (k = begin_idx + 1; k < result_idx; k++) {
-			struct json_object *current_dynamic_test = NULL;
-			int dyn_result_idx = -1;
-			char dynamic_name[256];
-			char dynamic_piglit_name[256];
-			const char *dynbeg, *dynend;
-
-			if (matches.items[k].what != STARTING_DYNAMIC_SUBTEST)
-				continue;
-
-			if (sscanf(matches.items[k].where + strlen(STARTING_DYNAMIC_SUBTEST), "%s", dynamic_name) != 1) {
-				/* Cannot parse name, just ignore this one */
-				continue;
-			}
-
-			dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
-
-			dynbeg = find_subtest_begin_limit(matches, k, dyn_result_idx, beg, end);
-			dynend = find_subtest_end_limit(matches, k, dyn_result_idx, beg, end);
-
-			generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
-
-			add_dynamic_subtest(&subtests->subs[i], strdup(dynamic_name));
-			current_dynamic_test = get_or_create_json_object(tests, dynamic_piglit_name);
-
-			json_object_object_add(current_dynamic_test, key,
-					       json_object_new_string_len(dynbeg, dynend - dynbeg));
-			add_igt_version(current_dynamic_test, igt_version, igt_version_len);
-
-			if (!json_object_object_get_ex(current_dynamic_test, "result", NULL)) {
-				const char *dynresulttext;
-				double dyntime;
-
-				parse_subtest_result(dynamic_name,
-						     DYNAMIC_SUBTEST_RESULT,
-						     &dynresulttext, &dyntime,
-						     dyn_result_idx < 0 ? NULL : matches.items[dyn_result_idx].where,
-						     dynend);
-				set_result(current_dynamic_test, dynresulttext);
-				set_runtime(current_dynamic_test, dyntime);
-			}
-		}
+		process_dynamic_subtest_output(piglit_name,
+					       igt_version, igt_version_len,
+					       matches,
+					       begin_idx, result_idx,
+					       beg, end,
+					       key,
+					       tests,
+					       &subtests->subs[i]);
 	}
 
 	free_matches(&matches);
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#7 lore raw

PATCH 5/11 runner/resultgen: Extrude dynamic subtest result texts

In reply to Petri Latvala

The first dynamic part now starts its reported output at the beginning of its subtest's output, and the last dynamic part ends its reported output at the end of its subtest's output.

---
 runner/resultgen.c | 64 ++++++++++++++++++++++++++++++----------------
 1 file changed, 42 insertions(+), 22 deletions(-)
runner/resultgen.c +42 -22
index 8f4bbe98..e139e3fc 100644
@@ -424,17 +424,18 @@ static int find_subtest_idx(struct matches matches,
 	return find_subtest_idx_limited(matches, bufend, linekey, pattern, subtest_name, 0, matches.size);
 }
 
-static const char *find_subtest_begin_limit(struct matches matches,
-					    int begin_idx,
-					    int result_idx,
-					    const char *buf,
-					    const char *bufend)
+static const char *find_subtest_begin_limit_limited(struct matches matches,
+						    int begin_idx,
+						    int result_idx,
+						    const char *buf,
+						    const char *bufend,
+						    int first_idx)
 {
 	/* No matching output at all, include everything */
-	if (begin_idx < 0 && result_idx < 0)
+	if (begin_idx < first_idx && result_idx < first_idx)
 		return buf;
 
-	if (begin_idx < 0) {
+	if (begin_idx < first_idx) {
 		/*
 		 * Subtest didn't start, but we have the
 		 * result. Probably because an igt_fixture
@@ -443,37 +444,48 @@ static const char *find_subtest_begin_limit(struct matches matches,
 		 * We go backwards one match from the result match,
 		 * and start from the next line.
 		 */
-		if (result_idx > 0)
+		if (result_idx > first_idx)
 			return next_line(matches.items[result_idx - 1].where, bufend);
 		else
 			return buf;
 	}
 
 	/* Include all non-special output before the beginning line. */
-	if (begin_idx == 0)
+	if (begin_idx <= first_idx)
 		return buf;
 
 	return next_line(matches.items[begin_idx - 1].where, bufend);
 }
 
-static const char *find_subtest_end_limit(struct matches matches,
-					  int begin_idx,
-					  int result_idx,
-					  const char *buf,
-					  const char *bufend)
+static const char *find_subtest_begin_limit(struct matches matches,
+					    int begin_idx,
+					    int result_idx,
+					    const char *buf,
+					    const char *bufend)
+{
+	return find_subtest_begin_limit_limited(matches, begin_idx, result_idx, buf, bufend, 0);
+}
+
+static const char *find_subtest_end_limit_limited(struct matches matches,
+						  int begin_idx,
+						  int result_idx,
+						  const char *buf,
+						  const char *bufend,
+						  int first_idx,
+						  int last_idx)
 {
 	int k;
 
 	/* No matching output at all, include everything */
-	if (begin_idx < 0 && result_idx < 0)
+	if (begin_idx < first_idx && result_idx < first_idx)
 		return bufend;
 
-	if (result_idx < 0) {
+	if (result_idx < first_idx) {
 		/*
 		 * Incomplete result. Include all output up to the
 		 * next starting subtest, or the result of one.
 		 */
-		for (k = begin_idx + 1; k < matches.size; k++) {
+		for (k = begin_idx + 1; k < last_idx; k++) {
 			if (matches.items[k].what == STARTING_SUBTEST ||
 			    matches.items[k].what == SUBTEST_RESULT)
 				return matches.items[k].where;
@@ -483,12 +495,21 @@ static const char *find_subtest_end_limit(struct matches matches,
 	}
 
 	/* Include all non-special output to the next match, whatever it is. */
-	if (result_idx < matches.size - 1)
+	if (result_idx < last_idx - 1)
 		return matches.items[result_idx + 1].where;
 
 	return bufend;
 }
 
+static const char *find_subtest_end_limit(struct matches matches,
+					  int begin_idx,
+					  int result_idx,
+					  const char *buf,
+					  const char *bufend)
+{
+	return find_subtest_end_limit_limited(matches, begin_idx, result_idx, buf, bufend, 0, matches.size);
+}
+
 static void process_dynamic_subtest_output(const char *piglit_name,
 					   const char *igt_version,
 					   size_t igt_version_len,
@@ -531,8 +552,8 @@ static void process_dynamic_subtest_output(const char *piglit_name,
 
 		dyn_result_idx = find_subtest_idx_limited(matches, end, DYNAMIC_SUBTEST_RESULT, "%s%s: ", dynamic_name, k, result_idx);
 
-		dynbeg = find_subtest_begin_limit(matches, k, dyn_result_idx, beg, end);
-		dynend = find_subtest_end_limit(matches, k, dyn_result_idx, beg, end);
+		dynbeg = find_subtest_begin_limit_limited(matches, k, dyn_result_idx, beg, end, begin_idx + 1);
+		dynend = find_subtest_end_limit_limited(matches, k, dyn_result_idx, beg, end, begin_idx + 1, result_idx);
 
 		generate_piglit_name_for_dynamic(piglit_name, dynamic_name, dynamic_piglit_name, sizeof(dynamic_piglit_name));
 
@@ -931,8 +952,7 @@ static bool fill_from_dmesg(int fd,
 			}
 		}
 		append_line(&dmesg, &dmesglen, formatted);
-		if (current_test != NULL)
-			append_line(&dynamic_dmesg, &dynamic_dmesg_len, formatted);
+		append_line(&dynamic_dmesg, &dynamic_dmesg_len, formatted);
 		free(formatted);
 	}
 	free(line);
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#8 lore raw

PATCH 6/11 runner/resultgen: Add support for extra validation hook in find_matches()

In reply to Petri Latvala
---
 runner/resultgen.c | 43 +++++++++++++++++++++++++------------------
 1 file changed, 25 insertions(+), 18 deletions(-)
runner/resultgen.c +25 -18
index e139e3fc..b01e0a48 100644
@@ -324,6 +324,12 @@ struct matches
 	size_t size;
 };
 
+struct match_needle
+{
+	const char *str;
+	bool (*validate)(const char *needle, const char *line, const char *bufend);
+};
+
 static void match_add(struct matches *matches, const char *where, const char *what)
 {
 	struct match_item newitem = { where, what };
@@ -334,19 +340,20 @@ static void match_add(struct matches *matches, const char *where, const char *wh
 }
 
 static struct matches find_matches(const char *buf, const char *bufend,
-				   const char **needles)
+				   const struct match_needle *needles)
 {
 	struct matches ret = {};
 
 	while (buf < bufend) {
-		const char **needle;
+		const struct match_needle *needle;
 
-		for (needle = needles; *needle; needle++) {
-			if (bufend - buf < strlen(*needle))
+		for (needle = needles; needle->str; needle++) {
+			if (bufend - buf < strlen(needle->str))
 				continue;
 
-			if (!memcmp(buf, *needle, strlen(*needle))) {
-				match_add(&ret, buf, *needle);
+			if (!memcmp(buf, needle->str, strlen(needle->str)) &&
+			    (!needle->validate || needle->validate(needle->str, buf, bufend))) {
+				match_add(&ret, buf, needle->str);
 				goto end_find;
 			}
 		}
@@ -589,12 +596,12 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 	char *igt_version = NULL;
 	size_t igt_version_len = 0;
 	struct json_object *current_test = NULL;
-	const char *needles[] = {
-		STARTING_SUBTEST,
-		SUBTEST_RESULT,
-		STARTING_DYNAMIC_SUBTEST,
-		DYNAMIC_SUBTEST_RESULT,
-		NULL
+	struct match_needle needles[] = {
+		{ STARTING_SUBTEST, NULL },
+		{ SUBTEST_RESULT, NULL },
+		{ STARTING_DYNAMIC_SUBTEST, NULL },
+		{ DYNAMIC_SUBTEST_RESULT, NULL },
+		{ NULL, NULL },
 	};
 	struct matches matches = {};
 	size_t i;
@@ -1110,12 +1117,12 @@ static void fill_from_journal(int fd,
 
 static bool stderr_contains_warnings(const char *beg, const char *end)
 {
-	const char *needles[] = {
-		STARTING_SUBTEST,
-		SUBTEST_RESULT,
-		STARTING_DYNAMIC_SUBTEST,
-		DYNAMIC_SUBTEST_RESULT,
-		NULL
+	struct match_needle needles[] = {
+		{ STARTING_SUBTEST, NULL },
+		{ SUBTEST_RESULT, NULL },
+		{ STARTING_DYNAMIC_SUBTEST, NULL },
+		{ DYNAMIC_SUBTEST_RESULT, NULL },
+		{ NULL, NULL },
 	};
 	struct matches matches;
 	size_t i = 0;
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#9 lore raw

PATCH 7/11 runner/resultgen: Make subtest result line finding more robust

In reply to Petri Latvala

On an assertion failure, the string "Subtest xyz failed" is printed. Make sure we don't match that for SUBTEST_RESULT, or the equivalent for dynamic subtests.

Parsing the results already explicitly searched for the proper result line, the difference is when we delimit output up to "the next line of interest".

---
 runner/resultgen.c | 41 +++++++++++++++++++++++++++++++++++++----
 1 file changed, 37 insertions(+), 4 deletions(-)
runner/resultgen.c +37 -4
index b01e0a48..64b899ca 100644
@@ -367,6 +367,39 @@ static struct matches find_matches(const char *buf, const char *bufend,
 	return ret;
 }
 
+static bool valid_char_for_subtest_name(char x)
+{
+	return x == '-' || x == '_' || isalnum(x);
+}
+
+static bool is_subtest_result_line(const char *needle, const char *line, const char *bufend)
+{
+	line += strlen(needle);
+
+	/*
+	 * At this point of the string we're expecting:
+	 * - The subtest name (one or more of a-z, A-Z, 0-9, '-' and '_')
+	 * - The characters ':' and ' '
+	 *
+	 * If we find all those, allow parsing this line as [dynamic]
+	 * subtest result.
+	 */
+
+	if (!valid_char_for_subtest_name(*line))
+		return false;
+
+	while (line < bufend && valid_char_for_subtest_name(*line))
+		line++;
+
+	if (line >= bufend || *line++ != ':')
+		return false;
+
+	if (line >= bufend || *line++ != ' ')
+		return false;
+
+	return true;
+}
+
 static void free_matches(struct matches *matches)
 {
 	free(matches->items);
@@ -598,9 +631,9 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 	struct json_object *current_test = NULL;
 	struct match_needle needles[] = {
 		{ STARTING_SUBTEST, NULL },
-		{ SUBTEST_RESULT, NULL },
+		{ SUBTEST_RESULT, is_subtest_result_line },
 		{ STARTING_DYNAMIC_SUBTEST, NULL },
-		{ DYNAMIC_SUBTEST_RESULT, NULL },
+		{ DYNAMIC_SUBTEST_RESULT, is_subtest_result_line },
 		{ NULL, NULL },
 	};
 	struct matches matches = {};
@@ -1119,9 +1152,9 @@ static bool stderr_contains_warnings(const char *beg, const char *end)
 {
 	struct match_needle needles[] = {
 		{ STARTING_SUBTEST, NULL },
-		{ SUBTEST_RESULT, NULL },
+		{ SUBTEST_RESULT, is_subtest_result_line },
 		{ STARTING_DYNAMIC_SUBTEST, NULL },
-		{ DYNAMIC_SUBTEST_RESULT, NULL },
+		{ DYNAMIC_SUBTEST_RESULT, is_subtest_result_line },
 		{ NULL, NULL },
 	};
 	struct matches matches;
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#10 lore raw

PATCH 8/11 runner/json_tests: Adapt to dynamic subtest result parsing

In reply to Petri Latvala

Dynamic subtests now include more output, change the dynamic subtest parsing test accordingly.

---
 .../dynamic-subtests/reference.json           | 20 +++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)
runner/json_tests_data/dynamic-subtests/reference.json +10 -10
index 274173f7..24a1fe8c 100644
@@ -22,7 +22,7 @@
       "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest debug-log-checking\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-dynamic-1\n<6> [23426155.240164] [IGT] dynamic: starting dynamic subtest this-is-dynamic-2\n<6> [23426155.293846] [IGT] dynamic: exiting, ret=98\n<6> [23426155.294003] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@dynamic@debug-log-checking@this-is-dynamic-1":{
-      "out":"Starting dynamic subtest: this-is-dynamic-1\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
+      "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"fail",
       "time":{
@@ -30,11 +30,11 @@
         "start":0,
         "end":0.055
       },
-      "err":"Starting dynamic subtest: this-is-dynamic-1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-1 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
-      "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest debug-log-checking\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-dynamic-1\n"
+      "err":"Starting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-1 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
+      "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest debug-log-checking\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-dynamic-1\n"
     },
     "igt@dynamic@debug-log-checking@this-is-dynamic-2":{
-      "out":"Starting dynamic subtest: this-is-dynamic-2\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\n",
+      "out":"Starting dynamic subtest: this-is-dynamic-2\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\nSubtest debug-log-checking: FAIL (0.109s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"fail",
       "time":{
@@ -42,7 +42,7 @@
         "start":0,
         "end":0.054
       },
-      "err":"Starting dynamic subtest: this-is-dynamic-2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-2 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\n",
+      "err":"Starting dynamic subtest: this-is-dynamic-2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-2 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\nSubtest debug-log-checking failed.\nNo log.\nSubtest debug-log-checking: FAIL (0.109s)\n",
       "dmesg":"<6> [23426155.240164] [IGT] dynamic: starting dynamic subtest this-is-dynamic-2\n<6> [23426155.293846] [IGT] dynamic: exiting, ret=98\n<6> [23426155.294003] Console: switching to colour frame buffer device 240x75\n"
     },
     "igt@dynamic@empty-container":{
@@ -70,7 +70,7 @@
       "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
     },
     "igt@dynamic@normal@normal-dynamic-subtest":{
-      "out":"Starting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\n",
+      "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"pass",
       "time":{
@@ -78,8 +78,8 @@
         "start":0,
         "end":0.055
       },
-      "err":"Starting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\n",
-      "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
+      "err":"Starting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
+      "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
     },
     "igt@dynamic@incomplete":{
       "out":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\nThis is some output\n",
@@ -94,7 +94,7 @@
       "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest incomplete\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-incomplete\n<6> [23426155.184895] Dmesg output for incomplete\n"
     },
     "igt@dynamic@incomplete@this-is-incomplete":{
-      "out":"Starting dynamic subtest: this-is-incomplete\nThis is some output\n",
+      "out":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\nThis is some output\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
       "result":"incomplete",
       "time":{
@@ -102,7 +102,7 @@
         "start":0,
         "end":0
       },
-      "err":"Starting dynamic subtest: this-is-incomplete\n",
+      "err":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\n",
       "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest incomplete\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-incomplete\n<6> [23426155.184895] Dmesg output for incomplete\n"
     },
     "igt@dynamic@resume":{
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#11 lore raw

PATCH 9/11 runner/resultgen: Don't report subtest result if it has dynamic subtests

In reply to Petri Latvala
---
 runner/resultgen.c | 43 +++++++++++++++++++++++++++++++++----------
 1 file changed, 33 insertions(+), 10 deletions(-)
runner/resultgen.c +33 -10
index 64b899ca..2e15331e 100644
@@ -1148,6 +1148,22 @@ static void fill_from_journal(int fd,
 	fclose(f);
 }
 
+static void prune_subtests_with_dynamic_subtests(const char *binary,
+						 struct subtest_list *subtests,
+						 struct json_object *tests)
+{
+	char piglit_name[256];
+	size_t i;
+
+	for (i = 0; i < subtests->size; i++) {
+		if (subtests->subs[i].dynamic_size) {
+			generate_piglit_name(binary, subtests->subs[i].name, piglit_name, sizeof(piglit_name));
+			json_object_object_del(tests, piglit_name);
+		}
+
+	}
+}
+
 static bool stderr_contains_warnings(const char *beg, const char *end)
 {
 	struct match_needle needles[] = {
@@ -1218,8 +1234,10 @@ static void override_results(char *binary,
 
 	for (i = 0; i < subtests->size; i++) {
 		generate_piglit_name(binary, subtests->subs[i].name, piglit_name, sizeof(piglit_name));
-		obj = get_or_create_json_object(tests, piglit_name);
-		override_result_single(obj);
+		if (subtests->subs[i].dynamic_size == 0) {
+			obj = get_or_create_json_object(tests, piglit_name);
+			override_result_single(obj);
+		}
 
 		for (k = 0; k < subtests->subs[i].dynamic_size; k++) {
 			generate_piglit_name_for_dynamic(piglit_name, subtests->subs[i].dynamic_names[k],
@@ -1300,15 +1318,18 @@ static void add_to_totals(const char *binary,
 
 	for (i = 0; i < subtests->size; i++) {
 		generate_piglit_name(binary, subtests->subs[i].name, piglit_name, sizeof(piglit_name));
-		test = get_or_create_json_object(results->tests, piglit_name);
-		if (!json_object_object_get_ex(test, "result", &resultobj)) {
-			fprintf(stderr, "Warning: No results set for %s\n", piglit_name);
-			return;
+
+		if (subtests->subs[i].dynamic_size == 0) {
+			test = get_or_create_json_object(results->tests, piglit_name);
+			if (!json_object_object_get_ex(test, "result", &resultobj)) {
+				fprintf(stderr, "Warning: No results set for %s\n", piglit_name);
+				return;
+			}
+			result = json_object_get_string(resultobj);
+			add_result_to_totals(emptystrtotal, result);
+			add_result_to_totals(roottotal, result);
+			add_result_to_totals(binarytotal, result);
 		}
-		result = json_object_get_string(resultobj);
-		add_result_to_totals(emptystrtotal, result);
-		add_result_to_totals(roottotal, result);
-		add_result_to_totals(binarytotal, result);
 
 		for (k = 0; k < subtests->subs[i].dynamic_size; k++) {
 			generate_piglit_name_for_dynamic(piglit_name, subtests->subs[i].dynamic_names[k],
@@ -1355,6 +1376,8 @@ static bool parse_test_directory(int dirfd,
 		goto parse_output_end;
 	}
 
+	prune_subtests_with_dynamic_subtests(entry->binary, &subtests, results->tests);
+
 	override_results(entry->binary, &subtests, results->tests);
 	add_to_totals(entry->binary, &subtests, results);
 
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#12 lore raw

PATCH 10/11 runner/json_tests: Adapt to no longer reporting subtests with dynamic subtests

In reply to Petri Latvala
---
 .../dynamic-subtests/reference.json           | 54 ++++---------------
 1 file changed, 9 insertions(+), 45 deletions(-)
runner/json_tests_data/dynamic-subtests/reference.json +9 -45
index 24a1fe8c..ca8f6cd7 100644
@@ -9,18 +9,6 @@
     "end":1560163492.4104891
   },
   "tests":{
-    "igt@dynamic@debug-log-checking":{
-      "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\nStarting dynamic subtest: this-is-dynamic-2\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\nSubtest debug-log-checking: FAIL (0.109s)\n",
-      "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
-      "result":"fail",
-      "time":{
-        "__type__":"TimeAttribute",
-        "start":0,
-        "end":0.109
-      },
-      "err":"Starting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-1 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 1\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:8:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\nStarting dynamic subtest: this-is-dynamic-2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\nDynamic subtest this-is-dynamic-2 failed.\n**** DEBUG ****\n(dynamic:20904) DEBUG: This print is from 2\n(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ..\/runner\/testdata\/dynamic.c:13:\n(dynamic:20904) CRITICAL: Failed assertion: false\n(dynamic:20904) igt_core-INFO: Stack trace:\n(dynamic:20904) igt_core-INFO:   #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n(dynamic:20904) igt_core-INFO:   #1 ..\/runner\/testdata\/dynamic.c:5 __real_main3()\n(dynamic:20904) igt_core-INFO:   #2 ..\/runner\/testdata\/dynamic.c:3 main()\n(dynamic:20904) igt_core-INFO:   #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n(dynamic:20904) igt_core-INFO:   #4 [_start+0x2a]\n****  END  ****\nDynamic subtest this-is-dynamic-2: FAIL (0.054s)\nSubtest debug-log-checking failed.\nNo log.\nSubtest debug-log-checking: FAIL (0.109s)\n",
-      "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest debug-log-checking\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-dynamic-1\n<6> [23426155.240164] [IGT] dynamic: starting dynamic subtest this-is-dynamic-2\n<6> [23426155.293846] [IGT] dynamic: exiting, ret=98\n<6> [23426155.294003] Console: switching to colour frame buffer device 240x75\n"
-    },
     "igt@dynamic@debug-log-checking@this-is-dynamic-1":{
       "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: debug-log-checking\nStarting dynamic subtest: this-is-dynamic-1\nStack trace:\n  #0 ..\/lib\/igt_core.c:1607 __igt_fail_assert()\n  #1 ..\/runner\/testdata\/dynamic.c:11 __real_main3()\n  #2 ..\/runner\/testdata\/dynamic.c:3 main()\n  #3 ..\/csu\/libc-start.c:342 __libc_start_main()\n  #4 [_start+0x2a]\nDynamic subtest this-is-dynamic-1: FAIL (0.055s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
@@ -57,18 +45,6 @@
       "err":"Starting subtest: empty-container\nSubtest empty-container: SKIP (0.000s)\n",
       "dmesg":"<6> [23426155.304955] Console: switching to colour dummy device 80x25\n<6> [23426155.304968] [IGT] dynamic: executing\n<6> [23426155.308644] [IGT] dynamic: starting subtest empty-container\n<6> [23426155.308671] [IGT] dynamic: exiting, ret=77\n<6> [23426155.308822] Console: switching to colour frame buffer device 240x75\n"
     },
-    "igt@dynamic@normal":{
-      "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
-      "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
-      "result":"pass",
-      "time":{
-        "__type__":"TimeAttribute",
-        "start":0,
-        "end":0.100
-      },
-      "err":"Starting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
-      "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
-    },
     "igt@dynamic@normal@normal-dynamic-subtest":{
       "out":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
@@ -81,18 +57,6 @@
       "err":"Starting subtest: normal\nStarting dynamic subtest: normal-dynamic-subtest\nDynamic subtest normal-dynamic-subtest: SUCCESS (0.055s)\nSubtest normal: SUCCESS (0.100s)\n",
       "dmesg":"<6> [23426155.175691] Console: switching to colour dummy device 80x25\n<6> [23426155.175708] [IGT] dynamic: executing\n<6> [23426155.184875] [IGT] dynamic: starting subtest normal\n<6> [23426155.184895] Dmesg output for normal\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest normal-dynamic-subtest\n"
     },
-    "igt@dynamic@incomplete":{
-      "out":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\nThis is some output\n",
-      "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
-      "result":"incomplete",
-      "time":{
-        "__type__":"TimeAttribute",
-        "start":0,
-        "end":0
-      },
-      "err":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\n",
-      "dmesg":"<6> [23426155.184875] [IGT] dynamic: starting subtest incomplete\n<6> [23426155.184895] [IGT] dynamic: starting dynamic subtest this-is-incomplete\n<6> [23426155.184895] Dmesg output for incomplete\n"
-    },
     "igt@dynamic@incomplete@this-is-incomplete":{
       "out":"Starting subtest: incomplete\nStarting dynamic subtest: this-is-incomplete\nThis is some output\n",
       "igt-version":"IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)",
@@ -121,38 +85,38 @@
   "totals":{
     "":{
       "crash":0,
-      "pass":3,
+      "pass":2,
       "dmesg-fail":0,
       "dmesg-warn":0,
       "skip":1,
-      "incomplete":2,
+      "incomplete":1,
       "timeout":0,
       "notrun":0,
-      "fail":3,
+      "fail":2,
       "warn":0
     },
     "root":{
       "crash":0,
-      "pass":3,
+      "pass":2,
       "dmesg-fail":0,
       "dmesg-warn":0,
       "skip":1,
-      "incomplete":2,
+      "incomplete":1,
       "timeout":0,
       "notrun":0,
-      "fail":3,
+      "fail":2,
       "warn":0
     },
     "igt@dynamic":{
       "crash":0,
-      "pass":3,
+      "pass":2,
       "dmesg-fail":0,
       "dmesg-warn":0,
       "skip":1,
-      "incomplete":2,
+      "incomplete":1,
       "timeout":0,
       "notrun":0,
-      "fail":3,
+      "fail":2,
       "warn":0
     }
   },
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#13 lore raw

PATCH 11/11 runner/json_tests: Add test for parsing dynamic subtests with same name

In reply to Petri Latvala

Multiple different subtests can have a dynamic subtest with the same name. Add a test to make sure we correctly delimit the output parsing.

---
 .../0/dmesg.txt                               | 10 +++
 .../0/err.txt                                 | 10 +++
 .../0/journal.txt                             |  3 +
 .../0/out.txt                                 | 11 +++
 .../README.txt                                |  2 +
 .../endtime.txt                               |  1 +
 .../joblist.txt                               |  1 +
 .../metadata.txt                              | 12 +++
 .../reference.json                            | 84 +++++++++++++++++++
 .../starttime.txt                             |  1 +
 .../uname.txt                                 |  1 +
 runner/runner_json_tests.c                    |  1 +
 12 files changed, 137 insertions(+)
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/dmesg.txt new file +10 -0
new file mode 100644
index 00000000..59741e5f
@@ -0,0 +1,10 @@
+6,1157,23426155175691,-;Console: switching to colour dummy device 80x25
+14,1158,23426155175708,-;[IGT] dynamic: executing
+14,1159,23426155184875,-;[IGT] dynamic: starting subtest subtest-one
+14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-name-is-shared
+14,1160,23426155184895,-;This dmesg output is from subtest-one/this-name-is-shared
+14,1159,23426155184875,-;[IGT] dynamic: starting subtest subtest-two
+14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-name-is-shared
+14,1160,23426155184895,-;This dmesg output is from subtest-two/this-name-is-shared
+14,1162,23426155293846,-;[IGT] dynamic: exiting, ret=0
+6,1163,23426155294003,-;Console: switching to colour frame buffer device 240x75
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/err.txt new file +10 -0
new file mode 100644
index 00000000..3d551819
@@ -0,0 +1,10 @@
+Starting subtest: subtest-one
+Starting dynamic subtest: this-name-is-shared
+This stderr output is from subtest-one/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-one: SUCCESS (0.050s)
+Starting subtest: subtest-two
+Starting dynamic subtest: this-name-is-shared
+This stderr output is from subtest-two/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-two: SUCCESS (0.050s)
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/journal.txt new file +3 -0
new file mode 100644
index 00000000..3af2b007
@@ -0,0 +1,3 @@
+subtest-one
+subtest-two
+exit:0 (0.100s)
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/0/out.txt new file +11 -0
new file mode 100644
index 00000000..e025c096
@@ -0,0 +1,11 @@
+IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)
+Starting subtest: subtest-one
+Starting dynamic subtest: this-name-is-shared
+This stdout output is from subtest-one/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-one: SUCCESS (0.050s)
+Starting subtest: subtest-two
+Starting dynamic subtest: this-name-is-shared
+This stdout output is from subtest-two/this-name-is-shared
+Dynamic subtest this-name-is-shared: SUCCESS (0.050s)
+Subtest subtest-two: SUCCESS (0.050s)
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/README.txt new file +2 -0
new file mode 100644
index 00000000..e141077d
@@ -0,0 +1,2 @@
+Parsing of dynamic subtest output is tricky if multiple subtests use
+the same name for a dynamic subtest.
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/endtime.txt new file +1 -0
new file mode 100644
index 00000000..5c7608b5
@@ -0,0 +1 @@
+1560163492.410489
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/joblist.txt new file +1 -0
new file mode 100644
index 00000000..9296b9ed
@@ -0,0 +1 @@
+dynamic subtest-one,subtest-two
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/metadata.txt new file +12 -0
new file mode 100644
index 00000000..8da1aee2
@@ -0,0 +1,12 @@
+abort_mask : 0
+name : dynamic-subtest-name-in-multiple-subtests
+dry_run : 0
+sync : 0
+log_level : 0
+overwrite : 0
+multiple_mode : 1
+inactivity_timeout : 0
+use_watchdog : 0
+piglit_style_dmesg : 0
+test_root : /path/does/not/exist
+results_path : /path/does/not/exist
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/reference.json new file +84 -0

84 lines not on this page · raw message

runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/starttime.txt new file +1 -0
new file mode 100644
index 00000000..b30b32ee
@@ -0,0 +1 @@
+1560163492.266377
runner/json_tests_data/dynamic-subtest-name-in-multiple-subtests/uname.txt new file +1 -0
new file mode 100644
index 00000000..a7aef6f7
@@ -0,0 +1 @@
+Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64
runner/runner_json_tests.c +1 -0
index 3443bd2b..21e87a02 100644
@@ -163,6 +163,7 @@ static const char *dirnames[] = {
 	"dmesg-warn-level-piglit-style",
 	"dmesg-warn-level-one-piglit-style",
 	"dynamic-subtests",
+	"dynamic-subtest-name-in-multiple-subtests",
 };
 
 igt_main
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#14 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for runner: Don't report subtests that have dynamic subtests

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: runner: Don't report subtests that have dynamic subtests
URL : https://patchwork.freedesktop.org/series/70286/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_7460 -> IGTPW_3794
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html

Known issues
------------

Here are the changes found in IGTPW_3794 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@i915_pm_rpm@module-reload:
- fi-skl-6770hq: [PASS][1] -> [FAIL][2] ([i915#178])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-skl-6770hq/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-skl-6770hq/igt@[email protected]

* igt@i915_selftest@live_blt:
- fi-hsw-4770: [PASS][3] -> [DMESG-FAIL][4] ([i915#563])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-hsw-4770/igt@i915_selftest@live_blt.html
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-hsw-4770/igt@i915_selftest@live_blt.html

#### Possible fixes ####

* igt@i915_module_load@reload-with-fault-injection:
- {fi-kbl-7560u}: [INCOMPLETE][5] ([i915#243] / [i915#609]) -> [PASS][6]
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-kbl-7560u/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-kbl-7560u/igt@[email protected]

* igt@kms_frontbuffer_tracking@basic:
- fi-icl-u2: [FAIL][7] ([i915#49]) -> [PASS][8]
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-icl-u2/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-icl-u2/igt@[email protected]

#### Warnings ####

* igt@kms_cursor_legacy@basic-busy-flip-before-cursor-legacy:
- fi-kbl-x1275: [DMESG-WARN][9] ([i915#62] / [i915#92]) -> [DMESG-WARN][10] ([i915#62] / [i915#92] / [i915#95]) +4 similar issues
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-kbl-x1275/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-kbl-x1275/igt@[email protected]

* igt@kms_cursor_legacy@basic-flip-after-cursor-atomic:
- fi-kbl-x1275: [DMESG-WARN][11] ([i915#62] / [i915#92] / [i915#95]) -> [DMESG-WARN][12] ([i915#62] / [i915#92]) +2 similar issues
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/fi-kbl-x1275/igt@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/fi-kbl-x1275/igt@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[i915#178]: https://gitlab.freedesktop.org/drm/intel/issues/178
[i915#243]: https://gitlab.freedesktop.org/drm/intel/issues/243
[i915#49]: https://gitlab.freedesktop.org/drm/intel/issues/49
[i915#563]: https://gitlab.freedesktop.org/drm/intel/issues/563
[i915#609]: https://gitlab.freedesktop.org/drm/intel/issues/609
[i915#62]: https://gitlab.freedesktop.org/drm/intel/issues/62
[i915#92]: https://gitlab.freedesktop.org/drm/intel/issues/92
[i915#95]: https://gitlab.freedesktop.org/drm/intel/issues/95

Participating hosts (50 -> 46)
------------------------------

Additional (1): fi-tgl-y
Missing (5): fi-ilk-m540 fi-hsw-4200u fi-bsw-cyan fi-ctg-p8600 fi-byt-clapper

Build changes
-------------

* CI: CI-20190529 -> None
* IGT: IGT_5320 -> IGTPW_3794

CI-20190529: 20190529
CI_DRM_7460: 9cd639929b6ad1efb35e90ee862cf37a9b6f6348 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3794: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html
IGT_5320: 06833ffeaf7621c3fef097166539bbe999a33e9c @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#15 lore raw

[igt-dev] ✓ Fi.CI.IGT: success for runner: Don't report subtests that have dynamic subtests

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: runner: Don't report subtests that have dynamic subtests
URL : https://patchwork.freedesktop.org/series/70286/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_7460_full -> IGTPW_3794_full
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html

Known issues
------------

Here are the changes found in IGTPW_3794_full that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_ctx_persistence@vcs1-cleanup:
- shard-iclb: [PASS][1] -> [SKIP][2] ([fdo#109276] / [fdo#112080]) +1 similar issue
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb2/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb3/igt@[email protected]

* igt@gem_exec_parallel@bcs0-contexts:
- shard-hsw: [PASS][3] -> [FAIL][4] ([i915#676]) +1 similar issue
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw5/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw4/igt@[email protected]

* igt@gem_exec_schedule@independent-bsd1:
- shard-iclb: [PASS][5] -> [SKIP][6] ([fdo#109276]) +1 similar issue
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb4/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb5/igt@[email protected]

* igt@gem_exec_schedule@preempt-queue-contexts-chain-bsd:
- shard-iclb: [PASS][7] -> [SKIP][8] ([fdo#112146])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb2/igt@[email protected]

* igt@gem_exec_schedule@smoketest-all:
- shard-tglb: [PASS][9] -> [INCOMPLETE][10] ([i915#463])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb1/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb4/igt@[email protected]

* igt@gem_userptr_blits@map-fixed-invalidate-busy:
- shard-snb: [PASS][11] -> [DMESG-WARN][12] ([fdo#111870])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-snb6/igt@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-snb7/igt@[email protected]

* igt@i915_pm_dc@dc6-psr:
- shard-iclb: [PASS][13] -> [FAIL][14] ([i915#454])
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb2/igt@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb3/igt@[email protected]

* igt@kms_cursor_crc@pipe-a-cursor-256x85-onscreen:
- shard-hsw: [PASS][15] -> [DMESG-WARN][16] ([IGT#6])
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw4/igt@[email protected]
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw6/igt@[email protected]

* igt@kms_cursor_crc@pipe-b-cursor-suspend:
- shard-apl: [PASS][17] -> [DMESG-FAIL][18] ([i915#180] / [i915#54])
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-apl1/igt@[email protected]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-apl6/igt@[email protected]
- shard-glk: [PASS][19] -> [FAIL][20] ([i915#54])
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-glk4/igt@[email protected]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-glk6/igt@[email protected]

* igt@kms_draw_crc@draw-method-rgb565-render-xtiled:
- shard-iclb: [PASS][21] -> [INCOMPLETE][22] ([i915#140])
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@[email protected]
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb4/igt@[email protected]

* igt@kms_flip@2x-flip-vs-expired-vblank:
- shard-glk: [PASS][23] -> [FAIL][24] ([i915#79])
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-glk6/igt@[email protected]
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-glk4/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-1p-offscren-pri-shrfb-draw-blt:
- shard-iclb: [PASS][25] -> [FAIL][26] ([i915#49])
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb2/igt@[email protected]
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb8/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-1p-primscrn-indfb-plflip-blt:
- shard-tglb: [PASS][27] -> [FAIL][28] ([i915#49])
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb7/igt@[email protected]
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb9/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-1p-rte:
- shard-iclb: [PASS][29] -> [FAIL][30] ([fdo#110378] / [i915#49])
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@[email protected]
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb5/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-wc:
- shard-tglb: [PASS][31] -> [INCOMPLETE][32] ([i915#474])
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb2/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-wc.html
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb3/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-spr-indfb-draw-mmap-wc.html

* igt@kms_plane@plane-panning-bottom-right-suspend-pipe-a-planes:
- shard-tglb: [PASS][33] -> [INCOMPLETE][34] ([i915#456] / [i915#460])
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb6/igt@[email protected]
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb1/igt@[email protected]

* igt@kms_vblank@pipe-a-ts-continuation-suspend:
- shard-kbl: [PASS][35] -> [DMESG-WARN][36] ([i915#180]) +7 similar issues
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl4/igt@[email protected]
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl2/igt@[email protected]

#### Possible fixes ####

* igt@gem_ctx_isolation@rcs0-s3:
- shard-kbl: [DMESG-WARN][37] ([i915#180]) -> [PASS][38] +2 similar issues
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl4/igt@[email protected]
[38]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl6/igt@[email protected]

* igt@gem_ctx_isolation@vcs1-s3:
- shard-iclb: [SKIP][39] ([fdo#109276] / [fdo#112080]) -> [PASS][40]
[39]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@[email protected]
[40]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb2/igt@[email protected]

* igt@gem_exec_parallel@vcs0-fds:
- shard-hsw: [FAIL][41] ([i915#676]) -> [PASS][42]
[41]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw4/igt@[email protected]
[42]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw7/igt@[email protected]

* igt@gem_exec_schedule@in-order-bsd2:
- shard-iclb: [SKIP][43] ([fdo#109276]) -> [PASS][44] +4 similar issues
[43]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb6/igt@[email protected]
[44]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb4/igt@[email protected]

* igt@gem_exec_schedule@reorder-wide-bsd:
- shard-iclb: [SKIP][45] ([fdo#112146]) -> [PASS][46] +1 similar issue
[45]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb4/igt@[email protected]
[46]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb5/igt@[email protected]

* igt@gem_ppgtt@flink-and-close-vma-leak:
- shard-apl: [FAIL][47] ([i915#644]) -> [PASS][48]
[47]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-apl6/igt@[email protected]
[48]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-apl4/igt@[email protected]

* igt@gem_userptr_blits@sync-unmap-cycles:
- shard-snb: [DMESG-WARN][49] ([fdo#111870]) -> [PASS][50] +1 similar issue
[49]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-snb4/igt@[email protected]
[50]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-snb6/igt@[email protected]

* {igt@i915_pm_dc@dc3co-vpb-simulation}:
- shard-tglb: [SKIP][51] ([i915#657]) -> [PASS][52]
[51]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb2/igt@[email protected]
[52]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb1/igt@[email protected]

* igt@i915_selftest@live_blt:
- shard-hsw: [DMESG-FAIL][53] ([i915#683]) -> [PASS][54]
[53]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw7/igt@i915_selftest@live_blt.html
[54]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw4/igt@i915_selftest@live_blt.html

* igt@kms_color@pipe-a-ctm-green-to-red:
- shard-kbl: [FAIL][55] ([i915#129]) -> [PASS][56]
[55]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl2/igt@[email protected]
[56]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl4/igt@[email protected]
- shard-apl: [FAIL][57] ([i915#129]) -> [PASS][58]
[57]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-apl2/igt@[email protected]
[58]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-apl6/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbcpsr-indfb-scaledprimary:
- shard-tglb: [FAIL][59] ([i915#49]) -> [PASS][60]
[59]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-tglb2/igt@[email protected]
[60]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-tglb2/igt@[email protected]

* igt@kms_plane@pixel-format-pipe-a-planes:
- shard-kbl: [INCOMPLETE][61] ([fdo#103665]) -> [PASS][62] +1 similar issue
[61]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl3/igt@[email protected]
[62]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl2/igt@[email protected]

* igt@kms_psr@psr2_dpms:
- shard-iclb: [SKIP][63] ([fdo#109441]) -> [PASS][64] +1 similar issue
[63]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-iclb5/igt@kms_psr@psr2_dpms.html
[64]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-iclb2/igt@kms_psr@psr2_dpms.html

* igt@kms_vblank@pipe-b-ts-continuation-suspend:
- shard-apl: [DMESG-WARN][65] ([i915#180]) -> [PASS][66] +1 similar issue
[65]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-apl4/igt@[email protected]
[66]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-apl6/igt@[email protected]

* {igt@perf_pmu@frequency-idle}:
- shard-hsw: [FAIL][67] ([i915#675]) -> [PASS][68]
[67]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw4/igt@[email protected]
[68]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw5/igt@[email protected]

#### Warnings ####

* igt@gem_exec_parallel@contexts:
- shard-hsw: [FAIL][69] ([i915#676]) -> [TIMEOUT][70] ([i915#676])
[69]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-hsw1/igt@[email protected]
[70]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-hsw5/igt@[email protected]

* igt@gem_workarounds@suspend-resume-context:
- shard-kbl: [INCOMPLETE][71] ([fdo#103665]) -> [DMESG-WARN][72] ([i915#180])
[71]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl3/igt@[email protected]
[72]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl6/igt@[email protected]

* igt@kms_cursor_crc@pipe-b-cursor-suspend:
- shard-kbl: [DMESG-WARN][73] ([i915#180]) -> [FAIL][74] ([i915#54])
[73]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl2/igt@[email protected]
[74]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl7/igt@[email protected]

* igt@kms_flip@flip-vs-suspend-interruptible:
- shard-kbl: [DMESG-WARN][75] ([i915#180] / [i915#391]) -> [DMESG-WARN][76] ([i915#180])
[75]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_7460/shard-kbl7/igt@[email protected]
[76]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/shard-kbl1/igt@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[IGT#6]: https://gitlab.freedesktop.org/drm/igt-gpu-tools/issues/6
[fdo#103665]: https://bugs.freedesktop.org/show_bug.cgi?id=103665
[fdo#109276]: https://bugs.freedesktop.org/show_bug.cgi?id=109276
[fdo#109441]: https://bugs.freedesktop.org/show_bug.cgi?id=109441
[fdo#110378]: https://bugs.freedesktop.org/show_bug.cgi?id=110378
[fdo#111870]: https://bugs.freedesktop.org/show_bug.cgi?id=111870
[fdo#112080]: https://bugs.freedesktop.org/show_bug.cgi?id=112080
[fdo#112146]: https://bugs.freedesktop.org/show_bug.cgi?id=112146
[i915#129]: https://gitlab.freedesktop.org/drm/intel/issues/129
[i915#140]: https://gitlab.freedesktop.org/drm/intel/issues/140
[i915#180]: https://gitlab.freedesktop.org/drm/intel/issues/180
[i915#391]: https://gitlab.freedesktop.org/drm/intel/issues/391
[i915#454]: https://gitlab.freedesktop.org/drm/intel/issues/454
[i915#456]: https://gitlab.freedesktop.org/drm/intel/issues/456
[i915#460]: https://gitlab.freedesktop.org/drm/intel/issues/460
[i915#463]: https://gitlab.freedesktop.org/drm/intel/issues/463
[i915#474]: https://gitlab.freedesktop.org/drm/intel/issues/474
[i915#49]: https://gitlab.freedesktop.org/drm/intel/issues/49
[i915#54]: https://gitlab.freedesktop.org/drm/intel/issues/54
[i915#644]: https://gitlab.freedesktop.org/drm/intel/issues/644
[i915#657]: https://gitlab.freedesktop.org/drm/intel/issues/657
[i915#675]: https://gitlab.freedesktop.org/drm/intel/issues/675
[i915#676]: https://gitlab.freedesktop.org/drm/intel/issues/676
[i915#683]: https://gitlab.freedesktop.org/drm/intel/issues/683
[i915#79]: https://gitlab.freedesktop.org/drm/intel/issues/79

Participating hosts (11 -> 8)
------------------------------

Missing (3): pig-skl-6260u pig-glk-j5005 pig-hsw-4770r

Build changes
-------------

* CI: CI-20190529 -> None
* IGT: IGT_5320 -> IGTPW_3794
* Piglit: piglit_4509 -> None

CI-20190529: 20190529
CI_DRM_7460: 9cd639929b6ad1efb35e90ee862cf37a9b6f6348 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3794: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html
IGT_5320: 06833ffeaf7621c3fef097166539bbe999a33e9c @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3794/index.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev