Skip to contentSkip to conversation

igt: Test tagging support

Intel-gfx Tvrtko Ursulin 4 participants 7 messages Sep 7, 2018 – Sep 12, 2018 Library code

lore
Share & tools
lore
#0 lore raw

From: Tvrtko Ursulin <[email protected]>

Proposal to add test tags as a replacement for separate test list which can be difficult to maintain and get out of date.

Putting this maintanenace inline with tests makes it easier to remember to update the (now implicit) lists, and also enables richer test selection possibilities for the test runner.

Current method of implying tags from test/subtest names has a couple of problems one of which is where some names can use the same token for different meanings. (One example is the "default" token.) It also creates a name clash between naming and tagging.

Test tags are strings of tokens separated by spaces, meaning of which we decide by creating a convetion and helped by the suitable helper macros.

For example tags can be things like: gem, kms, basic, guc, flip, semaphore, bz12345678, gt4e, reset, etc..

At runtime this would look something like this (abbreviated for readability):

  @ tests/gem_sync --list-subtests-with-tags
  ...
  forked-each TAGS="gem "
  forked-store-each TAGS="gem "
  basic-all TAGS="gem basic "
  basic-store-all TAGS="gem basic "

  @ tests/gem_concurrent_blit --list-subtests-with-tags
  ...
  16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress "
  16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress "
  16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress "

  @ tests/kms_flip --list-subtests-with-tags | grep basic
  basic-plain-flip TAGS="kms basic "
  basic-flip-vs-dpms TAGS="kms basic "

Test runner could then enable usages like:

  ./run-tests --include gem --exclude stress

Or on simulation for example:

  ./run-tests --exclude stress,reset,rc6,rps,vblank

Which I think could really be useful and more flexible than name based selection.

The simulation example could also replace the sprinkle of igt_skip_on_simulation which often lacks context, which tags could provide.

This RFC implementation automatically adds various tags based on both binary and subtest names. This is to enable easy transition to the tag based system.

Idea is that all current tests get some useful tags with minimum amount of work. Unfortunately this doesn't work for test binaries which at the moment do not support the subtests enumeration so those are the ones which would need identifying and converting.

Source code usage, for tests which would want to override the default tag generation would look something like:

	igt_gem_subtest("basic", "each")
		...

	igt_gem_subtest("manual", "special-setup-test")
		...

        igt_gem_stress_subtest_f("tag", "name-%s", ...) {
	        ...

We can of course polish the details of the API if the idea will be deemed interesting.

v2: More default tags for untagged tests for easier migration.

---
 lib/igt_core.c             | 115 +++++++++++++++++++++++++++++++++++--
 lib/igt_core.h             |  29 +++++++++-
 tests/gem_concurrent_all.c |  34 +++++------
 3 files changed, 152 insertions(+), 26 deletions(-)
lib/igt_core.c +109 -6
index 23bb858fd886..daae1a3e4b56 100644
@@ -258,10 +258,12 @@ static unsigned int exit_handler_count;
 const char *igt_interactive_debug;
 
 /* subtests helpers */
-static bool list_subtests = false;
-static char *run_single_subtest = NULL;
+static bool list_subtests, list_subtests_tags;
+static char *run_single_subtest;
 static bool run_single_subtest_found = false;
-static const char *in_subtest = NULL;
+static const char *in_subtest;
+static char *subtest_tags;
+static bool subtest_tags_allocated;
 static struct timespec subtest_time;
 static clockid_t igt_clock = (clockid_t)-1;
 static bool in_fixture = false;
@@ -281,6 +283,7 @@ bool test_child;
 
 enum {
  OPT_LIST_SUBTESTS,
+ OPT_LIST_SUBTESTS_TAGS,
  OPT_RUN_SUBTEST,
  OPT_DESCRIPTION,
  OPT_DEBUG,
@@ -549,6 +552,7 @@ static void print_usage(const char *help_str, bool output_on_stderr)
 
 	fprintf(f, "Usage: %s [OPTIONS]\n", command_str);
 	fprintf(f, "  --list-subtests\n"
+		   "  --list-subtests-with-tags\n"
 		   "  --run-subtest <pattern>\n"
 		   "  --debug[=log-domain]\n"
 		   "  --interactive-debug[=domain]\n"
@@ -659,6 +663,7 @@ static int common_init(int *argc, char **argv,
 	int c, option_index = 0, i, x;
 	static struct option long_options[] = {
 		{"list-subtests", 0, 0, OPT_LIST_SUBTESTS},
+		{"list-subtests-with-tags", 0, 0, OPT_LIST_SUBTESTS_TAGS},
 		{"run-subtest", 1, 0, OPT_RUN_SUBTEST},
 		{"help-description", 0, 0, OPT_DESCRIPTION},
 		{"debug", optional_argument, 0, OPT_DEBUG},
@@ -753,6 +758,12 @@ static int common_init(int *argc, char **argv,
 			if (!run_single_subtest)
 				list_subtests = true;
 			break;
+		case OPT_LIST_SUBTESTS_TAGS:
+			if (!run_single_subtest) {
+				list_subtests = true;
+				list_subtests_tags = true;
+			}
+			break;
 		case OPT_RUN_SUBTEST:
 			if (!list_subtests)
 				run_single_subtest = strdup(optarg);
@@ -884,19 +895,100 @@ void igt_simple_init_parse_opts(int *argc, char **argv,
 		    extra_opt_handler, handler_data);
 }
 
+#define __IGT_TAGS_MAX (4096)
+
+static char * __get_tag_str(char *tags)
+{
+	if (tags)
+		return tags;
+
+	tags = malloc(__IGT_TAGS_MAX);
+	igt_assert(tags);
+
+	memset(tags, 0, __IGT_TAGS_MAX);
+
+	return tags;
+}
+
+static char * __add_tag(char *tags, unsigned int *pos, const char *tag)
+{
+	size_t left = __IGT_TAGS_MAX - *pos;
+	char _tag[128];
+	int ret;
+
+	if (!tag)
+		return tags;
+
+	tags = __get_tag_str(tags);
+
+	sprintf(_tag, "%s ", tag);
+	if (!strncmp(tags, _tag, strlen(_tag)))
+		return tags;
+
+	sprintf(_tag, " %s ", tag);
+	if (strstr(tags, _tag))
+		return tags;
+
+	ret = snprintf(tags + *pos, left, "%s ", tag);
+	igt_assert(ret > 0 && ret < left);
+
+	*pos += ret;
+
+	return tags;
+}
+
 /*
  * Note: Testcases which use these helpers MUST NOT output anything to stdout
  * outside of places protected by igt_run_subtest checks - the piglit
  * runner adds every line to the subtest list.
  */
-bool __igt_run_subtest(const char *subtest_name)
+bool __igt_run_subtest(const char *subtest_name, const char *tags)
 {
+	unsigned int tag_idx = 0;
+	char *_tags = NULL;
+	const char *tag;
 	int i;
 
 	assert(!in_subtest);
+	assert(!subtest_tags);
 	assert(!in_fixture);
 	assert(test_with_subtests);
 
+	if (!tags) {
+		if (!strncmp(command_str, "gem_", 4))
+			tag = "gem";
+		else if (!strncmp(command_str, "prime_", 4))
+			tag = "gem";
+		else if (!strncmp(command_str, "kms_", 4))
+			tag = "kms";
+		else if (!strncmp(command_str, "pm_", 3))
+			tag = "pm";
+		else if (!strncmp(command_str, "drv_", 3))
+			tag = "drv";
+		else if (!strncmp(command_str, "debugfs_", 3))
+			tag = "drv";
+		else if (!strncmp(command_str, "core_", 4))
+			tag = "drm";
+		else if (!strncmp(command_str, "drm_", 4))
+			tag = "drm";
+		else if (!strncmp(command_str, "sw_", 4))
+			tag = "external";
+		else
+			tag = NULL;
+
+		_tags = __add_tag(_tags, &tag_idx, tag);
+
+		if (!strncmp(subtest_name, "basic-", 6))
+			tag = "basic";
+		else
+			tag = NULL;
+
+		_tags = __add_tag(_tags, &tag_idx, tag);
+
+		tags = _tags;
+		subtest_tags_allocated = _tags;
+	}
+
 	/* check the subtest name only contains a-z, A-Z, 0-9, '-' and '_' */
 	for (i = 0; subtest_name[i] != '\0'; i++)
 		if (subtest_name[i] != '_' && subtest_name[i] != '-'
@@ -907,7 +999,10 @@ bool __igt_run_subtest(const char *subtest_name)
 		}
 
 	if (list_subtests) {
-		printf("%s\n", subtest_name);
+		if (list_subtests_tags && tags)
+			printf("%s TAGS=\"%s\"\n", subtest_name, tags);
+		else
+			printf("%s\n", subtest_name);
 		return false;
 	}
 
@@ -940,7 +1035,10 @@ bool __igt_run_subtest(const char *subtest_name)
 
 	_igt_log_buffer_reset();
 
+	subtest_tags = (char *)tags;
+
 	igt_gettime(&subtest_time);
+
 	return (in_subtest = subtest_name);
 }
 
@@ -997,7 +1095,12 @@ static void exit_subtest(const char *result)
 
 	igt_terminate_spin_batches();
 
-	in_subtest = NULL;
+	if (subtest_tags_allocated)
+		free(subtest_tags);
+
+	in_subtest = subtest_tags = NULL;
+	subtest_tags_allocated = false;
+
 	siglongjmp(igt_subtest_jmpbuf, 1);
 }
 
lib/igt_core.h +26 -3
index b80e1702a16f..300ff739d97b 100644
@@ -145,7 +145,7 @@ int igt_subtest_init_parse_opts(int *argc, char **argv,
 #define igt_subtest_init(argc, argv) \
 	igt_subtest_init_parse_opts(&argc, argv, NULL, NULL, NULL, NULL, NULL);
 
-bool __igt_run_subtest(const char *subtest_name);
+bool __igt_run_subtest(const char *subtest_name, const char *subtest_tags);
 #define __igt_tokencat2(x, y) x ## y
 
 /**
@@ -158,6 +158,29 @@ bool __igt_run_subtest(const char *subtest_name);
  */
 #define igt_tokencat(x, y) __igt_tokencat2(x, y)
 
+#define igt_tagged_subtest(tags, name) \
+	for (; __igt_run_subtest((name), (tags)) && \
+	       (sigsetjmp(igt_subtest_jmpbuf, 1) == 0); \
+	     igt_success())
+
+#define igt_gem_subtest(tags, name) igt_tagged_subtest("gem "tags, name)
+
+#define __igt_tagged_subtest_f(tags, tmp, format...) \
+	for (char tmp [256]; \
+	     snprintf( tmp , sizeof( tmp ), format), \
+	     __igt_run_subtest(tmp, tags) && \
+	     (sigsetjmp(igt_subtest_jmpbuf, 1) == 0); \
+	     igt_success())
+
+#define igt_tagged_subtest_f(tags, f...) \
+	__igt_tagged_subtest_f(tags, igt_tokencat(__tmpchar, __LINE__), f)
+
+#define igt_gem_subtest_f(tags, f...) \
+	igt_tagged_subtest_f("gem "tags, f)
+
+#define igt_gem_stress_subtest_f(tags, f...) \
+	igt_tagged_subtest_f("gem stress "tags, f)
+
 /**
  * igt_subtest:
  * @name: name of the subtest
@@ -169,14 +192,14 @@ bool __igt_run_subtest(const char *subtest_name);
  *
  * This is a simpler version of igt_subtest_f()
  */
-#define igt_subtest(name) for (; __igt_run_subtest((name)) && \
+#define igt_subtest(name) for (; __igt_run_subtest((name), NULL) && \
 				   (sigsetjmp(igt_subtest_jmpbuf, 1) == 0); \
 				   igt_success())
 #define __igt_subtest_f(tmp, format...) \
 	for (char tmp [256]; \
 	     snprintf( tmp , sizeof( tmp ), \
 		      format), \
-	     __igt_run_subtest( tmp ) && \
+	     __igt_run_subtest(tmp, NULL) && \
 	     (sigsetjmp(igt_subtest_jmpbuf, 1) == 0); \
 	     igt_success())
 
tests/gem_concurrent_all.c +17 -17
index 4ac08c1b1c90..c1d29421b8ba 100644
@@ -1481,47 +1481,47 @@ run_mode(const char *prefix,
 			igt_subtest_group  {
 				igt_fixture p->require();
 
-				igt_subtest_f("%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-sanitycheck0%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers, do_basic0,
 							p->copy, h->hang);
 				}
 
-				igt_subtest_f("%s-%s-%s-sanitycheck1%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-sanitycheck1%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers, do_basic1,
 							p->copy, h->hang);
 				}
 
-				igt_subtest_f("%s-%s-%s-sanitycheckN%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-sanitycheckN%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers, do_basicN,
 							p->copy, h->hang);
 				}
 
 				/* try to overwrite the source values */
-				igt_subtest_f("%s-%s-%s-overwrite-source-one%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source-one%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_overwrite_source__one,
 							p->copy, h->hang);
 				}
 
-				igt_subtest_f("%s-%s-%s-overwrite-source%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_overwrite_source,
 							p->copy, h->hang);
 				}
 
-				igt_subtest_f("%s-%s-%s-overwrite-source-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_overwrite_source_read_bcs,
 							p->copy, h->hang);
 				}
 
-				igt_subtest_f("%s-%s-%s-overwrite-source-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					igt_require(rendercopy);
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
@@ -1529,7 +1529,7 @@ run_mode(const char *prefix,
 							p->copy, h->hang);
 				}
 
-				igt_subtest_f("%s-%s-%s-overwrite-source-rev%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-overwrite-source-rev%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_overwrite_source__rev,
@@ -1537,21 +1537,21 @@ run_mode(const char *prefix,
 				}
 
 				/* try to intermix copies with GPU copies*/
-				igt_subtest_f("%s-%s-%s-intermix-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-intermix-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					igt_require(rendercopy);
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_intermix_rcs,
 							p->copy, h->hang);
 				}
-				igt_subtest_f("%s-%s-%s-intermix-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-intermix-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					igt_require(rendercopy);
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_intermix_bcs,
 							p->copy, h->hang);
 				}
-				igt_subtest_f("%s-%s-%s-intermix-both%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-intermix-both%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					igt_require(rendercopy);
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
@@ -1560,7 +1560,7 @@ run_mode(const char *prefix,
 				}
 
 				/* try to read the results before the copy completes */
-				igt_subtest_f("%s-%s-%s-early-read%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-early-read%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_early_read,
@@ -1568,13 +1568,13 @@ run_mode(const char *prefix,
 				}
 
 				/* concurrent reads */
-				igt_subtest_f("%s-%s-%s-read-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-read-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_read_read_bcs,
 							p->copy, h->hang);
 				}
-				igt_subtest_f("%s-%s-%s-read-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-read-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					igt_require(rendercopy);
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
@@ -1583,13 +1583,13 @@ run_mode(const char *prefix,
 				}
 
 				/* split copying between rings */
-				igt_subtest_f("%s-%s-%s-write-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-write-read-bcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_write_read_bcs,
 							p->copy, h->hang);
 				}
-				igt_subtest_f("%s-%s-%s-write-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-write-read-rcs%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					igt_require(rendercopy);
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
@@ -1598,7 +1598,7 @@ run_mode(const char *prefix,
 				}
 
 				/* and finally try to trick the kernel into loosing the pending write */
-				igt_subtest_f("%s-%s-%s-gpu-read-after-write%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
+				igt_gem_stress_subtest_f("", "%s-%s-%s-gpu-read-after-write%s%s", prefix, mode->name, p->prefix, suffix, h->suffix) {
 					buffers_create(&buffers);
 					run_wrap_func(&buffers,
 							do_gpu_read_after_write,
Signature · 6 lines
2.17.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for igt: Test tagging support

== Series Details ==

== Series Details ==

Series: igt: Test tagging support
URL : https://patchwork.freedesktop.org/series/49338/
State : success

== Summary ==

= CI Bug Log - changes from CI_DRM_4782 -> IGTPW_1812 =

== Summary - SUCCESS ==

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/49338/revisions/1/mbox/

== Known issues ==

Here are the changes found in IGTPW_1812 that come from known issues:

=== IGT changes ===

    ==== Issues hit ====

    igt@debugfs_test@read_all_entries:
      fi-snb-2520m:       PASS -> INCOMPLETE (fdo#103713)

    igt@kms_frontbuffer_tracking@basic:
      fi-byt-clapper:     PASS -> FAIL (fdo#103167)

    igt@kms_pipe_crc_basic@suspend-read-crc-pipe-c:
      fi-bxt-dsi:         PASS -> INCOMPLETE (fdo#103927)

    
    ==== Possible fixes ====

    igt@amdgpu/amd_cs_nop@sync-fork-gfx0:
      fi-kbl-8809g:       DMESG-WARN (fdo#107762) -> PASS

    igt@kms_psr@primary_page_flip:
      fi-kbl-7560u:       FAIL (fdo#107336) -> PASS

    
    ==== Warnings ====

    igt@amdgpu/amd_prime@amd-to-i915:
      fi-kbl-8809g:       DMESG-FAIL (fdo#107762) -> FAIL (fdo#107341)

fdo#103167 https://bugs.freedesktop.org/show_bug.cgi?id=103167
fdo#103713 https://bugs.freedesktop.org/show_bug.cgi?id=103713
fdo#103927 https://bugs.freedesktop.org/show_bug.cgi?id=103927
fdo#107336 https://bugs.freedesktop.org/show_bug.cgi?id=107336
fdo#107341 https://bugs.freedesktop.org/show_bug.cgi?id=107341
fdo#107762 https://bugs.freedesktop.org/show_bug.cgi?id=107762

== Participating hosts (52 -> 47) ==

Additional (2): fi-byt-j1900 fi-gdg-551
Missing (7): fi-hsw-4770r fi-ilk-m540 fi-hsw-4200u fi-hsw-peppy fi-byt-squawks fi-bsw-cyan fi-ctg-p8600

== Build changes ==

* IGT: IGT_4632 -> IGTPW_1812

CI_DRM_4782: 60edf94611d2374821fbe2a824cebcb425ce7b0d @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1812: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1812/
IGT_4632: 94b4e204473a7d9f49e536c8877a4a5636e0d1b2 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1812/issues.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#2 lore raw
In reply to Tvrtko Ursulin

Quoting Tvrtko Ursulin (2018-09-07 12:14:20)

Quoted 42 lines

From: Tvrtko Ursulin <[email protected]>

Proposal to add test tags as a replacement for separate test list which can be difficult to maintain and get out of date.

Putting this maintanenace inline with tests makes it easier to remember to update the (now implicit) lists, and also enables richer test selection possibilities for the test runner.

Current method of implying tags from test/subtest names has a couple of problems one of which is where some names can use the same token for different meanings. (One example is the "default" token.) It also creates a name clash between naming and tagging.

Test tags are strings of tokens separated by spaces, meaning of which we decide by creating a convetion and helped by the suitable helper macros.

For example tags can be things like: gem, kms, basic, guc, flip, semaphore, bz12345678, gt4e, reset, etc..

At runtime this would look something like this (abbreviated for readability):

  @ tests/gem_sync --list-subtests-with-tags
  ...
  forked-each TAGS="gem "
  forked-store-each TAGS="gem "
  basic-all TAGS="gem basic "
  basic-store-all TAGS="gem basic "

  @ tests/gem_concurrent_blit --list-subtests-with-tags
  ...
  16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress "
  16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress "
  16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress "

  @ tests/kms_flip --list-subtests-with-tags | grep basic
  basic-plain-flip TAGS="kms basic "
  basic-flip-vs-dpms TAGS="kms basic "
Quoted 3 lines

Test runner could then enable usages like:

  ./run-tests --include gem --exclude stress

Minor comment, I like some basic boolean algebra here --include "gem AND smoketest NOT stress"
:)

I'd probably tag everything according to which ioctls they use. I feel my endgoal is still to list tests by their kernel functions (which can and should be automatically derived), and their hw function which is the open problem.
-Chris
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#6 lore raw

[igt-dev] ✓ Fi.CI.IGT: success for igt: Test tagging support

In reply to Tvrtko Ursulin
== Series Details ==

== Series Details ==

Series: igt: Test tagging support
URL : https://patchwork.freedesktop.org/series/49338/
State : success

== Summary ==

= CI Bug Log - changes from IGT_4632_full -> IGTPW_1812_full =

== Summary - SUCCESS ==

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/49338/revisions/1/mbox/

== Known issues ==

Here are the changes found in IGTPW_1812_full that come from known issues:

=== IGT changes ===

    ==== Issues hit ====

    igt@drv_suspend@shrink:
      shard-glk:          PASS -> FAIL (fdo#106886)

    igt@kms_cursor_legacy@2x-long-cursor-vs-flip-atomic:
      shard-hsw:          PASS -> FAIL (fdo#105767)

    igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-pwrite:
      shard-glk:          PASS -> FAIL (fdo#103167)

    igt@kms_sysfs_edid_timing:
      shard-glk:          NOTRUN -> WARN (fdo#100047)

    igt@kms_vblank@pipe-a-ts-continuation-suspend:
      shard-kbl:          PASS -> INCOMPLETE (fdo#103665, fdo#107556)

    igt@kms_vblank@pipe-b-ts-continuation-suspend:
      shard-apl:          PASS -> INCOMPLETE (fdo#103927)

    igt@kms_vblank@pipe-b-wait-forked-busy-hang:
      shard-snb:          PASS -> INCOMPLETE (fdo#105411)

    
    ==== Possible fixes ====

    igt@kms_cursor_legacy@cursor-vs-flip-toggle:
      shard-hsw:          FAIL (fdo#103355) -> PASS +1

    igt@kms_frontbuffer_tracking@fbc-1p-primscrn-cur-indfb-draw-blt:
      shard-snb:          INCOMPLETE (fdo#105411) -> PASS

    igt@kms_frontbuffer_tracking@fbc-2p-primscrn-spr-indfb-draw-mmap-cpu:
      shard-glk:          FAIL (fdo#103167) -> PASS

    igt@kms_plane@pixel-format-pipe-a-planes:
      shard-snb:          FAIL (fdo#107749) -> PASS

    igt@kms_rotation_crc@primary-rotation-180:
      shard-snb:          FAIL (fdo#103925) -> PASS

fdo#100047 https://bugs.freedesktop.org/show_bug.cgi?id=100047
fdo#103167 https://bugs.freedesktop.org/show_bug.cgi?id=103167
fdo#103355 https://bugs.freedesktop.org/show_bug.cgi?id=103355
fdo#103665 https://bugs.freedesktop.org/show_bug.cgi?id=103665
fdo#103925 https://bugs.freedesktop.org/show_bug.cgi?id=103925
fdo#103927 https://bugs.freedesktop.org/show_bug.cgi?id=103927
fdo#105411 https://bugs.freedesktop.org/show_bug.cgi?id=105411
fdo#105767 https://bugs.freedesktop.org/show_bug.cgi?id=105767
fdo#106886 https://bugs.freedesktop.org/show_bug.cgi?id=106886
fdo#107556 https://bugs.freedesktop.org/show_bug.cgi?id=107556
fdo#107749 https://bugs.freedesktop.org/show_bug.cgi?id=107749

== Participating hosts (5 -> 5) ==

No changes in participating hosts

== Build changes ==

* IGT: IGT_4632 -> IGTPW_1812
* Linux: CI_DRM_4775 -> CI_DRM_4782

CI_DRM_4775: 1a2bb6c061217718b972b3f4a74b96b61cf19d0c @ git://anongit.freedesktop.org/gfx-ci/linux
CI_DRM_4782: 60edf94611d2374821fbe2a824cebcb425ce7b0d @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1812: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1812/
IGT_4632: 94b4e204473a7d9f49e536c8877a4a5636e0d1b2 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1812/shards.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#3 lore raw
In reply to Chris Wilson
Quoted 43 lines · Chris Wilson

Quoting Tvrtko Ursulin (2018-09-07 12:14:20)

Quoted 42 lines

From: Tvrtko Ursulin <[email protected]>

Proposal to add test tags as a replacement for separate test list which can be difficult to maintain and get out of date.

Putting this maintanenace inline with tests makes it easier to remember to update the (now implicit) lists, and also enables richer test selection possibilities for the test runner.

Current method of implying tags from test/subtest names has a couple of problems one of which is where some names can use the same token for different meanings. (One example is the "default" token.) It also creates a name clash between naming and tagging.

Test tags are strings of tokens separated by spaces, meaning of which we decide by creating a convetion and helped by the suitable helper macros.

For example tags can be things like: gem, kms, basic, guc, flip, semaphore, bz12345678, gt4e, reset, etc..

At runtime this would look something like this (abbreviated for readability):

   @ tests/gem_sync --list-subtests-with-tags
   ...
   forked-each TAGS="gem "
   forked-store-each TAGS="gem "
   basic-all TAGS="gem basic "
   basic-store-all TAGS="gem basic "

   @ tests/gem_concurrent_blit --list-subtests-with-tags
   ...
   16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress "
   16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress "
   16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress "

   @ tests/kms_flip --list-subtests-with-tags | grep basic
   basic-plain-flip TAGS="kms basic "
   basic-flip-vs-dpms TAGS="kms basic "
Quoted 7 lines
Quoted 3 lines

Test runner could then enable usages like:

   ./run-tests --include gem --exclude stress

Minor comment, I like some basic boolean algebra here --include "gem AND smoketest NOT stress"
:)

That's what my hypothetical examples showed just with a different syntax!

Quoted 4 lines

I'd probably tag everything according to which ioctls they use. I feel my endgoal is still to list tests by their kernel functions (which can and should be automatically derived), and their hw function which is the open problem.

If we want to do that automatically then tagging in this flavour probably doesn't make sense and it should instead be an external database.

If we go on the ioctl granularity it can probably mostly have one version, and it should be static enough to be able to live in the tree, but if we go more granular, like something derived from kcov, then that's out of the window. Since it will be tied both to a platform and kernel version.

So I think tagging as proposed here is not appropriate for either, but only if we wanted to tag on coarser functional areas and such.

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#4 lore raw

Quoting Tvrtko Ursulin (2018-09-12 09:48:00)

Quoted 63 lines · Chris Wilson
Quoted 43 lines

Quoting Tvrtko Ursulin (2018-09-07 12:14:20)

Quoted 42 lines

From: Tvrtko Ursulin <[email protected]>

Proposal to add test tags as a replacement for separate test list which can be difficult to maintain and get out of date.

Putting this maintanenace inline with tests makes it easier to remember to update the (now implicit) lists, and also enables richer test selection possibilities for the test runner.

Current method of implying tags from test/subtest names has a couple of problems one of which is where some names can use the same token for different meanings. (One example is the "default" token.) It also creates a name clash between naming and tagging.

Test tags are strings of tokens separated by spaces, meaning of which we decide by creating a convetion and helped by the suitable helper macros.

For example tags can be things like: gem, kms, basic, guc, flip, semaphore, bz12345678, gt4e, reset, etc..

At runtime this would look something like this (abbreviated for readability):

   @ tests/gem_sync --list-subtests-with-tags
   ...
   forked-each TAGS="gem "
   forked-store-each TAGS="gem "
   basic-all TAGS="gem basic "
   basic-store-all TAGS="gem basic "

   @ tests/gem_concurrent_blit --list-subtests-with-tags
   ...
   16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress "
   16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress "
   16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress "

   @ tests/kms_flip --list-subtests-with-tags | grep basic
   basic-plain-flip TAGS="kms basic "
   basic-flip-vs-dpms TAGS="kms basic "
Quoted 7 lines
Quoted 3 lines

Test runner could then enable usages like:

   ./run-tests --include gem --exclude stress

Minor comment, I like some basic boolean algebra here --include "gem AND smoketest NOT stress"
:)

That's what my hypothetical examples showed just with a different syntax!

Quoted 4 lines

I'd probably tag everything according to which ioctls they use. I feel my endgoal is still to list tests by their kernel functions (which can and should be automatically derived), and their hw function which is the open problem.

If we want to do that automatically then tagging in this flavour probably doesn't make sense and it should instead be an external database.

Quoted 8 lines

If we go on the ioctl granularity it can probably mostly have one version, and it should be static enough to be able to live in the tree, but if we go more granular, like something derived from kcov, then that's out of the window. Since it will be tied both to a platform and kernel version.

So I think tagging as proposed here is not appropriate for either, but only if we wanted to tag on coarser functional areas and such.

Yeah, I think the same as well, that tags should be "smoke", "stress". (But one man's stress is another's minimal workload :(

But both of those are in essence a duration parameter, and I'd prefer if we expressed those as configurable parameters. At which point there is a meta level of test scripts to tag ;)

I feel that "gem", "kms" is better expressed in lower level terms, so what's left to tag? Clients? "display", "opencl", "media", "opengl" ? Even using client specs for features (e.g. EGL_IMG_context_priority)?

Who and why would use those? From a selfish perspective, I want lcov with tests sorted in order of increasing "stress" (start with the precise X does Y test, finish with can it survive pathological client behaviour for 48 hours).
-Chris
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#5 lore raw
Quoted 76 lines · Chris Wilson

Quoting Tvrtko Ursulin (2018-09-12 09:48:00)

Quoted 63 lines · Chris Wilson
Quoted 43 lines

Quoting Tvrtko Ursulin (2018-09-07 12:14:20)

Quoted 42 lines

From: Tvrtko Ursulin <[email protected]>

Proposal to add test tags as a replacement for separate test list which can be difficult to maintain and get out of date.

Putting this maintanenace inline with tests makes it easier to remember to update the (now implicit) lists, and also enables richer test selection possibilities for the test runner.

Current method of implying tags from test/subtest names has a couple of problems one of which is where some names can use the same token for different meanings. (One example is the "default" token.) It also creates a name clash between naming and tagging.

Test tags are strings of tokens separated by spaces, meaning of which we decide by creating a convetion and helped by the suitable helper macros.

For example tags can be things like: gem, kms, basic, guc, flip, semaphore, bz12345678, gt4e, reset, etc..

At runtime this would look something like this (abbreviated for readability):

@ tests/gem_sync --list-subtests-with-tags
...
forked-each TAGS="gem "
forked-store-each TAGS="gem "
basic-all TAGS="gem basic "
basic-store-all TAGS="gem basic "

@ tests/gem_concurrent_blit --list-subtests-with-tags
...
16MiB-swap-gpuX-render-write-read-bcs-bomb TAGS="gem stress " 16MiB-swap-gpuX-render-write-read-rcs-bomb TAGS="gem stress " 16MiB-swap-gpuX-render-gpu-read-after-write-bomb TAGS="gem stress "

@ tests/kms_flip --list-subtests-with-tags | grep basic basic-plain-flip TAGS="kms basic " basic-flip-vs-dpms TAGS="kms basic "

Quoted 7 lines
Quoted 3 lines

Test runner could then enable usages like:

    ./run-tests --include gem --exclude stress

Minor comment, I like some basic boolean algebra here --include "gem AND smoketest NOT stress"
:)

That's what my hypothetical examples showed just with a different syntax!

Quoted 4 lines

I'd probably tag everything according to which ioctls they use. I feel my endgoal is still to list tests by their kernel functions (which can and should be automatically derived), and their hw function which is the open problem.

If we want to do that automatically then tagging in this flavour probably doesn't make sense and it should instead be an external database.

Quoted 8 lines

If we go on the ioctl granularity it can probably mostly have one version, and it should be static enough to be able to live in the tree, but if we go more granular, like something derived from kcov, then that's out of the window. Since it will be tied both to a platform and kernel version.

So I think tagging as proposed here is not appropriate for either, but only if we wanted to tag on coarser functional areas and such.

Yeah, I think the same as well, that tags should be "smoke", "stress". (But one man's stress is another's minimal workload :(

Quoted 7 lines

But both of those are in essence a duration parameter, and I'd prefer if we expressed those as configurable parameters. At which point there is a meta level of test scripts to tag ;)

I feel that "gem", "kms" is better expressed in lower level terms, so what's left to tag? Clients? "display", "opencl", "media", "opengl" ? Even using client specs for features (e.g. EGL_IMG_context_priority)?

If we overcomplicate and try to change too much at once it is deemed to fail. GEM, KMS, etc, have this advantage of being very established.

So I was thinking these basic keywords and then something high level like reset, rps, stolen, and similar. So for instance we can remove the igt_skip_on_simulation, which can be time consuming to answer why it is there, and replace it with --exclude reset,... when running under the simulator.

True it is manual work to keep the tags up to date. Is it more work than the current scheme is TBD.

Quoted 4 lines

Who and why would use those? From a selfish perspective, I want lcov with tests sorted in order of increasing "stress" (start with the precise X does Y test, finish with can it survive pathological client behaviour for 48 hours).

Who and why would use which ones? Tags in general, tags in the style the patch proposes, or tags in the flavour you described?

My initial idea was that it would make easy for developers to run an approximate sweep when touching a feature. But with good CI and trybot maybe no one would use tags and just rely on "outsourced" test runs? Could be.. but hey, you convinced me to re-send this. Or was it Joonas? :)

Regards,

Tvrtko
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev