Skip to contentSkip to conversation

tests: Separate tests with lots of subtests

PATCH Superseded Gabriel Feceoru 2 participants 5 messages Apr 18, 2016 – May 17, 2016

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tests: Separate tests with lots of subtests

Currently, when trying to run i-g-t tests with piglit, it takes up to 20s until the first test is executed.
The main reason is that gem_concurrent_all has ~500k subtests, overkilling piglit.

This patch separates gem_concurrent_* tests from the rest of the tests, creating two files test-list.txt and test-list-full.txt.
The piglit can now enumerate the i-g-t tests within a decent few seconds. The second list could be used when trying to execute specific these tests.

---
 tests/Makefile.am      |  6 ++++++
 tests/Makefile.sources | 22 ++++++++++++++++++++--
 2 files changed, 26 insertions(+), 2 deletions(-)
tests/Makefile.am +6 -0
index 771e9ee..185962b 100644
@@ -14,6 +14,12 @@ test-list.txt: Makefile.sources
 	@echo ${single_kernel_tests} ${multi_kernel_tests} >> $@
 	@echo END TESTLIST >> $@
 
+test-list-full.txt: Makefile.sources
+	@echo TESTLIST > $@
+	@echo ${single_kernel_tests} ${multi_kernel_tests} ${extra_multi_kernel_tests} >> $@
+	@echo END TESTLIST >> $@
+
+
 noinst_PROGRAMS = \
 	$(HANG) \
 	$(TESTS_testsuite) \
tests/Makefile.sources +20 -2
index c52be2c..ca72243 100644
@@ -20,8 +20,6 @@ TESTS_progs_M = \
 	gem_busy \
 	gem_caching \
 	gem_close_race \
-	gem_concurrent_blit \
-	gem_concurrent_all \
 	gem_create \
 	gem_cs_tlb \
 	gem_ctx_param_basic \
@@ -117,6 +115,11 @@ TESTS_progs_M = \
 	template \
 	$(NULL)
 
+TESTS_progs_XM = \
+    gem_concurrent_blit \
+    gem_concurrent_all \
+    $(NULL)
+
 TESTS_progs = \
 	core_auth \
 	core_getclient \
@@ -219,6 +222,15 @@ multi_kernel_tests = \
 	$(TESTS_scripts_M) \
 	$(NULL)
 
+
+# This target contains testcases with lots of subtests.
+# It was split from multi_kernel_tests because of the overhead of listing all
+# the subtests in piglit
+extra_multi_kernel_tests = \
+	$(TESTS_progs_XM) \
+	$(NULL)
+
+
 # This target is for simple testcase which don't expose any subtest.
 #
 # Your testcase should probably use igt_simple_main, but none of the other magic
@@ -233,6 +245,12 @@ kernel_tests = \
 	$(multi_kernel_tests) \
 	$(NULL)
 
+kernel_tests_full = \
+    $(single_kernel_tests) \
+    $(multi_kernel_tests) \
+	$(extra_multi_kernel_tests) \
+	$(NULL)
+
 # Test that exercise specific asserts in the test framework library and are
 # hence expected to fail.
 XFAIL_TESTS = \
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw
Quoted 11 lines · Gabriel Feceoru

Currently, when trying to run i-g-t tests with piglit, it takes up to 20s until the first test is executed.
The main reason is that gem_concurrent_all has ~500k subtests, overkilling piglit.

This patch separates gem_concurrent_* tests from the rest of the tests, creating two files test-list.txt and test-list-full.txt.
The piglit can now enumerate the i-g-t tests within a decent few seconds. The second list could be used when trying to execute specific these tests.

Imo just shuffle gem_concurrent_all into the EXTRA binaries target. With 500k tests it's way past the "just a few regression tests" phase fully into "full blown stress test suite that takes centuries to run".

With that change: Acked-by: Daniel Vetter <[email protected]>

So feel free to ask Marius to push right away.
-Daniel

Quoted 79 lines
---
 tests/Makefile.am      |  6 ++++++
 tests/Makefile.sources | 22 ++++++++++++++++++++--
 2 files changed, 26 insertions(+), 2 deletions(-)
tests/Makefile.am +6 -0
index 771e9ee..185962b 100644
@@ -14,6 +14,12 @@ test-list.txt: Makefile.sources
 	@echo ${single_kernel_tests} ${multi_kernel_tests} >> $@
 	@echo END TESTLIST >> $@
 
+test-list-full.txt: Makefile.sources
+	@echo TESTLIST > $@
+	@echo ${single_kernel_tests} ${multi_kernel_tests} ${extra_multi_kernel_tests} >> $@
+	@echo END TESTLIST >> $@
+
+
 noinst_PROGRAMS = \
 	$(HANG) \
 	$(TESTS_testsuite) \
tests/Makefile.sources +20 -2
index c52be2c..ca72243 100644
@@ -20,8 +20,6 @@ TESTS_progs_M = \
 	gem_busy \
 	gem_caching \
 	gem_close_race \
-	gem_concurrent_blit \
-	gem_concurrent_all \
 	gem_create \
 	gem_cs_tlb \
 	gem_ctx_param_basic \
@@ -117,6 +115,11 @@ TESTS_progs_M = \
 	template \
 	$(NULL)
 
+TESTS_progs_XM = \
+    gem_concurrent_blit \
+    gem_concurrent_all \
+    $(NULL)
+
 TESTS_progs = \
 	core_auth \
 	core_getclient \
@@ -219,6 +222,15 @@ multi_kernel_tests = \
 	$(TESTS_scripts_M) \
 	$(NULL)
 
+
+# This target contains testcases with lots of subtests.
+# It was split from multi_kernel_tests because of the overhead of listing all
+# the subtests in piglit
+extra_multi_kernel_tests = \
+	$(TESTS_progs_XM) \
+	$(NULL)
+
+
 # This target is for simple testcase which don't expose any subtest.
 #
 # Your testcase should probably use igt_simple_main, but none of the other magic
@@ -233,6 +245,12 @@ kernel_tests = \
 	$(multi_kernel_tests) \
 	$(NULL)
 
+kernel_tests_full = \
+    $(single_kernel_tests) \
+    $(multi_kernel_tests) \
+	$(extra_multi_kernel_tests) \
+	$(NULL)
+
 # Test that exercise specific asserts in the test framework library and are
 # hence expected to fail.
 XFAIL_TESTS = \
Signature · 1 line
1.9.1
Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

PATCH v2 1/1 tests: Separate tests with lots of subtests

In reply to Gabriel Feceoru

Currently, when trying to run i-g-t tests with piglit, it takes up to 20s until the first test is executed.
The main reason is that gem_concurrent_all has ~500k subtests, overkilling piglit.

This patch separates gem_concurrent_* tests from the rest of the tests, creating two files test-list.txt and test-list-full.txt.
The piglit can now enumerate the i-g-t tests within a decent few seconds. The second list could be used when trying to execute specific these tests.

v2: Rebased

---
 tests/Makefile.am      | 12 +++++++++---
 tests/Makefile.sources | 22 ++++++++++++++++++++--
 2 files changed, 29 insertions(+), 5 deletions(-)
tests/Makefile.am +9 -3
index 771e9ee..0ed2a07 100644
@@ -14,6 +14,12 @@ test-list.txt: Makefile.sources
 	@echo ${single_kernel_tests} ${multi_kernel_tests} >> $@
 	@echo END TESTLIST >> $@
 
+test-list-full.txt: Makefile.sources
+	@echo TESTLIST > $@
+	@echo ${single_kernel_tests} ${multi_kernel_tests} ${extra_multi_kernel_tests} >> $@
+	@echo END TESTLIST >> $@
+
+
 noinst_PROGRAMS = \
 	$(HANG) \
 	$(TESTS_testsuite) \
@@ -38,14 +44,14 @@ dist_pkgdata_DATA = \
 
 all-local: .gitignore
 .gitignore: Makefile.sources
-	@echo "$(pkglibexec_PROGRAMS) $(HANG) test-list.txt .gitignore" | sed 's/\s\+/\n/g' | sort > $@
+	@echo "$(pkglibexec_PROGRAMS) $(HANG) test-list.txt test-list-full.txt .gitignore" | sed 's/\s\+/\n/g' | sort > $@
 
-pkgdata_DATA = test-list.txt
+pkgdata_DATA = test-list.txt test-list-full.txt
 
 EXTRA_PROGRAMS = $(HANG)
 EXTRA_DIST = $(common_files)
 
-CLEANFILES = $(EXTRA_PROGRAMS) test-list.txt .gitignore
+CLEANFILES = $(EXTRA_PROGRAMS) test-list.txt test-list-full.txt .gitignore
 
 AM_CFLAGS = $(DRM_CFLAGS) $(CWARNFLAGS) $(DEBUG_CFLAGS)\
 	-I$(srcdir)/.. \
tests/Makefile.sources +20 -2
index 41eca0d..abcf325 100644
@@ -20,8 +20,6 @@ TESTS_progs_M = \
 	gem_busy \
 	gem_caching \
 	gem_close_race \
-	gem_concurrent_blit \
-	gem_concurrent_all \
 	gem_create \
 	gem_cs_tlb \
 	gem_ctx_bad_exec \
@@ -117,6 +115,11 @@ TESTS_progs_M = \
 	template \
 	$(NULL)
 
+TESTS_progs_XM = \
+    gem_concurrent_blit \
+    gem_concurrent_all \
+    $(NULL)
+
 TESTS_progs = \
 	core_auth \
 	core_getclient \
@@ -219,6 +222,15 @@ multi_kernel_tests = \
 	$(TESTS_scripts_M) \
 	$(NULL)
 
+
+# This target contains testcases with lots of subtests.
+# It was split from multi_kernel_tests because of the overhead of listing all
+# the subtests in piglit
+extra_multi_kernel_tests = \
+	$(TESTS_progs_XM) \
+	$(NULL)
+
+
 # This target is for simple testcase which don't expose any subtest.
 #
 # Your testcase should probably use igt_simple_main, but none of the other magic
@@ -233,6 +245,12 @@ kernel_tests = \
 	$(multi_kernel_tests) \
 	$(NULL)
 
+kernel_tests_full = \
+    $(single_kernel_tests) \
+    $(multi_kernel_tests) \
+	$(extra_multi_kernel_tests) \
+	$(NULL)
+
 # Test that exercise specific asserts in the test framework library and are
 # hence expected to fail.
 XFAIL_TESTS = \
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#3 lore raw
Quoted 5 lines · Gabriel Feceoru

Currently, when trying to run i-g-t tests with piglit, it takes up to 20s until the first test is executed.
The main reason is that gem_concurrent_all has ~500k subtests, overkilling piglit.

Quoted 8 lines

This patch separates gem_concurrent_* tests from the rest of the tests, creating two files test-list.txt and test-list-full.txt.
The piglit can now enumerate the i-g-t tests within a decent few seconds. The second list could be used when trying to execute specific these tests.

v2: Rebased

You didn't take my review into account but still just pushed this patch. Please fix up.

Also I think we should keep gem_concurrent_blit in the normal set of testcases.
-Daniel

Quoted 101 lines
---
 tests/Makefile.am      | 12 +++++++++---
 tests/Makefile.sources | 22 ++++++++++++++++++++--
 2 files changed, 29 insertions(+), 5 deletions(-)
tests/Makefile.am +9 -3
index 771e9ee..0ed2a07 100644
@@ -14,6 +14,12 @@ test-list.txt: Makefile.sources
 	@echo ${single_kernel_tests} ${multi_kernel_tests} >> $@
 	@echo END TESTLIST >> $@
 
+test-list-full.txt: Makefile.sources
+	@echo TESTLIST > $@
+	@echo ${single_kernel_tests} ${multi_kernel_tests} ${extra_multi_kernel_tests} >> $@
+	@echo END TESTLIST >> $@
+
+
 noinst_PROGRAMS = \
 	$(HANG) \
 	$(TESTS_testsuite) \
@@ -38,14 +44,14 @@ dist_pkgdata_DATA = \
 
 all-local: .gitignore
 .gitignore: Makefile.sources
-	@echo "$(pkglibexec_PROGRAMS) $(HANG) test-list.txt .gitignore" | sed 's/\s\+/\n/g' | sort > $@
+	@echo "$(pkglibexec_PROGRAMS) $(HANG) test-list.txt test-list-full.txt .gitignore" | sed 's/\s\+/\n/g' | sort > $@
 
-pkgdata_DATA = test-list.txt
+pkgdata_DATA = test-list.txt test-list-full.txt
 
 EXTRA_PROGRAMS = $(HANG)
 EXTRA_DIST = $(common_files)
 
-CLEANFILES = $(EXTRA_PROGRAMS) test-list.txt .gitignore
+CLEANFILES = $(EXTRA_PROGRAMS) test-list.txt test-list-full.txt .gitignore
 
 AM_CFLAGS = $(DRM_CFLAGS) $(CWARNFLAGS) $(DEBUG_CFLAGS)\
 	-I$(srcdir)/.. \
tests/Makefile.sources +20 -2
index 41eca0d..abcf325 100644
@@ -20,8 +20,6 @@ TESTS_progs_M = \
 	gem_busy \
 	gem_caching \
 	gem_close_race \
-	gem_concurrent_blit \
-	gem_concurrent_all \
 	gem_create \
 	gem_cs_tlb \
 	gem_ctx_bad_exec \
@@ -117,6 +115,11 @@ TESTS_progs_M = \
 	template \
 	$(NULL)
 
+TESTS_progs_XM = \
+    gem_concurrent_blit \
+    gem_concurrent_all \
+    $(NULL)
+
 TESTS_progs = \
 	core_auth \
 	core_getclient \
@@ -219,6 +222,15 @@ multi_kernel_tests = \
 	$(TESTS_scripts_M) \
 	$(NULL)
 
+
+# This target contains testcases with lots of subtests.
+# It was split from multi_kernel_tests because of the overhead of listing all
+# the subtests in piglit
+extra_multi_kernel_tests = \
+	$(TESTS_progs_XM) \
+	$(NULL)
+
+
 # This target is for simple testcase which don't expose any subtest.
 #
 # Your testcase should probably use igt_simple_main, but none of the other magic
@@ -233,6 +245,12 @@ kernel_tests = \
 	$(multi_kernel_tests) \
 	$(NULL)
 
+kernel_tests_full = \
+    $(single_kernel_tests) \
+    $(multi_kernel_tests) \
+	$(extra_multi_kernel_tests) \
+	$(NULL)
+
 # Test that exercise specific asserts in the test framework library and are
 # hence expected to fail.
 XFAIL_TESTS = \
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#4 lore raw
Quoted 11 lines · Daniel Vetter
Quoted 10 lines · Gabriel Feceoru

Currently, when trying to run i-g-t tests with piglit, it takes up to 20s until the first test is executed.
The main reason is that gem_concurrent_all has ~500k subtests, overkilling piglit.

This patch separates gem_concurrent_* tests from the rest of the tests, creating two files test-list.txt and test-list-full.txt.
The piglit can now enumerate the i-g-t tests within a decent few seconds. The second list could be used when trying to execute specific these tests.

Quoted 6 lines
Quoted 3 lines

v2: Rebased

You didn't take my review into account but still just pushed this patch. Please fix up.

My apologies, I didn't ignore it but didn't read it carefully.

Regards,
Gabriel

Quoted 102 lines

Also I think we should keep gem_concurrent_blit in the normal set of testcases.
-Daniel

Quoted 97 lines
---
  tests/Makefile.am      | 12 +++++++++---
  tests/Makefile.sources | 22 ++++++++++++++++++++--
  2 files changed, 29 insertions(+), 5 deletions(-)
tests/Makefile.am +9 -3
index 771e9ee..0ed2a07 100644
@@ -14,6 +14,12 @@ test-list.txt: Makefile.sources
  	@echo ${single_kernel_tests} ${multi_kernel_tests} >> $@
  	@echo END TESTLIST >> $@
 
+test-list-full.txt: Makefile.sources
+	@echo TESTLIST > $@
+	@echo ${single_kernel_tests} ${multi_kernel_tests} ${extra_multi_kernel_tests} >> $@
+	@echo END TESTLIST >> $@
+
+
  noinst_PROGRAMS = \
  	$(HANG) \
  	$(TESTS_testsuite) \
@@ -38,14 +44,14 @@ dist_pkgdata_DATA = \
 
  all-local: .gitignore
  .gitignore: Makefile.sources
-	@echo "$(pkglibexec_PROGRAMS) $(HANG) test-list.txt .gitignore" | sed 's/\s\+/\n/g' | sort > $@
+	@echo "$(pkglibexec_PROGRAMS) $(HANG) test-list.txt test-list-full.txt .gitignore" | sed 's/\s\+/\n/g' | sort > $@
 
-pkgdata_DATA = test-list.txt
+pkgdata_DATA = test-list.txt test-list-full.txt
 
  EXTRA_PROGRAMS = $(HANG)
  EXTRA_DIST = $(common_files)
 
-CLEANFILES = $(EXTRA_PROGRAMS) test-list.txt .gitignore
+CLEANFILES = $(EXTRA_PROGRAMS) test-list.txt test-list-full.txt .gitignore
 
  AM_CFLAGS = $(DRM_CFLAGS) $(CWARNFLAGS) $(DEBUG_CFLAGS)\
  	-I$(srcdir)/.. \
tests/Makefile.sources +20 -3
index 41eca0d..abcf325 100644
@@ -20,8 +20,6 @@ TESTS_progs_M = \
  	gem_busy \
  	gem_caching \
  	gem_close_race \
-	gem_concurrent_blit \
-	gem_concurrent_all \
  	gem_create \
  	gem_cs_tlb \
  	gem_ctx_bad_exec \
@@ -117,6 +115,11 @@ TESTS_progs_M = \
  	template \
  	$(NULL)
 
+TESTS_progs_XM = \
+    gem_concurrent_blit \
+    gem_concurrent_all \
+    $(NULL)
+
  TESTS_progs = \
  	core_auth \
  	core_getclient \
@@ -219,6 +222,15 @@ multi_kernel_tests = \
  	$(TESTS_scripts_M) \
  	$(NULL)
 
+
+# This target contains testcases with lots of subtests.
+# It was split from multi_kernel_tests because of the overhead of listing all
+# the subtests in piglit
+extra_multi_kernel_tests = \
+	$(TESTS_progs_XM) \
+	$(NULL)
+
+
  # This target is for simple testcase which don't expose any subtest.
  #
  # Your testcase should probably use igt_simple_main, but none of the other magic
@@ -233,6 +245,12 @@ kernel_tests = \
  	$(multi_kernel_tests) \
  	$(NULL)
 
+kernel_tests_full = \
+    $(single_kernel_tests) \
+    $(multi_kernel_tests) \
+	$(extra_multi_kernel_tests) \
+	$(NULL)
+
  # Test that exercise specific asserts in the test framework library and are
  # hence expected to fail.
  XFAIL_TESTS = \
--

1.9.1

Quoted 5 lines
Quoted 4 lines

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx