Skip to contentSkip to conversation

scripts: add quick-testlist.py

PATCH Superseded Thomas Wood 5 participants 10 messages Sep 15, 2015 – Sep 23, 2015 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 scripts: add quick-testlist.py

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

---
 scripts/quick-testlist.py | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100755 scripts/quick-testlist.py
scripts/quick-testlist.py new file +44 -0
new file mode 100755
index 0000000..2cd95b4
@@ -0,0 +1,44 @@
+#!/usr/bin/env python
+#
+# Copyright 2015 Intel Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+from __future__ import print_function
+import json
+import sys
+
+def filter_results(filename):
+    with open(filename) as data:
+        json_data = json.load(data)
+
+    for test_name in json_data["tests"]:
+        if json_data["tests"][test_name]["time"] < 60:
+            print(test_name)
+
+
+if len(sys.argv) < 2:
+    print("Usage: quick-testlist.py [RESULTS]")
+    print("Read piglit results from RESULTS and print the tests that executed"
+          " in under 60 seconds. The list can be used by the --test-list option"
+          " of piglit.")
+    sys.exit(1)
+
+filter_results(sys.argv[1])
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw

PATCH v2 1/1 scripts: add quick-testlist.py

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

v2: exclude incomplete tests

---
 scripts/quick-testlist.py | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 scripts/quick-testlist.py
scripts/quick-testlist.py new file +46 -0
new file mode 100755
index 0000000..0dd4c69
@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+#
+# Copyright 2015 Intel Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+from __future__ import print_function
+import json
+import sys
+
+def filter_results(filename):
+    with open(filename) as data:
+        json_data = json.load(data)
+
+    for test_name in json_data["tests"]:
+        if json_data["tests"][test_name]["result"] == "incomplete":
+            continue
+        if json_data["tests"][test_name]["time"] < 60:
+            print(test_name)
+
+
+if len(sys.argv) < 2:
+    print("Usage: quick-testlist.py RESULTS")
+    print("Read piglit results from RESULTS and print the tests that executed"
+          " in under 60 seconds, excluding any incomplete tests. The list can"
+          " be used by the --test-list option of piglit.")
+    sys.exit(1)
+
+filter_results(sys.argv[1])
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw
Quoted 3 lines · Thomas Wood

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

And passed?
-Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#3 lore raw
Quoted 6 lines · Chris Wilson
Quoted 3 lines · Thomas Wood

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

And passed?

Any status except "incomplete", which normally means a reboot was required. It might also be worth noting that tests skipped on one platform may not skip on another, so the output only really applies to the platform the results were initially produced on.

Quoted 8 lines

-Chris

--
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#4 lore raw
Quoted 6 lines · Thomas Wood
Quoted 5 lines · Chris Wilson
Quoted 3 lines · Thomas Wood

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

Quoted 6 lines
Quoted 1 line

And passed?

Any status except "incomplete", which normally means a reboot was required. It might also be worth noting that tests skipped on one platform may not skip on another, so the output only really applies to the platform the results were initially produced on.

My point is that the timing can only be expected to be consistent for a pass. A failure may be quick, but success may take a few hours. -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
#5 lore raw

This script measures the duration of the subtests. But in case of gem_concurrent_all, for instance, there are 3780 subtests for which, although many shorter than 60s, the total duration will not qualify this for a quick test (~2h).

Thank you,
Gabriel
________________________________________
From: Intel-gfx [[email protected]] on behalf of Chris Wilson [[email protected]]
Sent: Thursday, September 17, 2015 3:43 PM
To: Wood, Thomas
Cc: Intel Graphics Development
Subject: Re: [Intel-gfx] [PATCH i-g-t v2] scripts: add quick-testlist.py

Quoted 6 lines · Thomas Wood
Quoted 5 lines · Chris Wilson
Quoted 3 lines · Thomas Wood

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

Quoted 6 lines
Quoted 1 line

And passed?

Any status except "incomplete", which normally means a reboot was required. It might also be worth noting that tests skipped on one platform may not skip on another, so the output only really applies to the platform the results were initially produced on.

My point is that the timing can only be expected to be consistent for a pass. A failure may be quick, but success may take a few hours. -Chris

--
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#6 lore raw
Quoted 3 lines · Gabriel

This script measures the duration of the subtests. But in case of gem_concurrent_all, for instance, there are 3780 subtests for which, although many shorter than 60s, the total duration will not qualify this for a quick test (~2h).

For this specific case, there has been some discussion about hiding some of the combinatorial subtests from the default testing list. kms_frontbuffer_tracking already has an option to show hidden subtests and pm_rpm has an option to reduce the amount of work in the stress tests. A common approach to hiding or reducing the number of stress tests for quick testing would be useful.

Quoted 28 lines

Thank you,
Gabriel
________________________________________
From: Intel-gfx [[email protected]] on behalf of Chris Wilson [[email protected]]
Sent: Thursday, September 17, 2015 3:43 PM
To: Wood, Thomas
Cc: Intel Graphics Development
Subject: Re: [Intel-gfx] [PATCH i-g-t v2] scripts: add quick-testlist.py

Quoted 6 lines · Thomas Wood
Quoted 5 lines · Chris Wilson
Quoted 3 lines · Thomas Wood

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

Quoted 6 lines
Quoted 1 line

And passed?

Any status except "incomplete", which normally means a reboot was required. It might also be worth noting that tests skipped on one platform may not skip on another, so the output only really applies to the platform the results were initially produced on.

My point is that the timing can only be expected to be consistent for a pass. A failure may be quick, but success may take a few hours. -Chris

Quoted 6 lines

--
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#7 lore raw
In reply to Chris Wilson
Quoted 9 lines · Chris Wilson
Quoted 8 lines · Thomas Wood
Quoted 6 lines · Chris Wilson
Quoted 3 lines · Thomas Wood

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

And passed?

Quoted 7 lines
Quoted 4 lines

Any status except "incomplete", which normally means a reboot was required. It might also be worth noting that tests skipped on one platform may not skip on another, so the output only really applies to the platform the results were initially produced on.

My point is that the timing can only be expected to be consistent for a pass. A failure may be quick, but success may take a few hours.

I think including failed tests depends on whether the "quick" run should be able to detect fixes from previous runs. Perhaps an option to exclude anything other than successful tests might be useful. There is also the timeout mechanism in piglit/igt that will prevent any test from running for more than 10 minutes.

Quoted 4 lines

-Chris

--
Chris Wilson, Intel Open Source Technology Centre

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

#8 lore raw
Quoted 15 lines · Thomas Wood
Quoted 14 lines · Chris Wilson
Quoted 12 lines · Thomas Wood
Quoted 6 lines · Chris Wilson
Quoted 3 lines · Thomas Wood

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

And passed?

Any status except "incomplete", which normally means a reboot was required. It might also be worth noting that tests skipped on one platform may not skip on another, so the output only really applies to the platform the results were initially produced on.

Quoted 8 lines
Quoted 2 lines

My point is that the timing can only be expected to be consistent for a pass. A failure may be quick, but success may take a few hours.

I think including failed tests depends on whether the "quick" run should be able to detect fixes from previous runs. Perhaps an option to exclude anything other than successful tests might be useful. There is also the timeout mechanism in piglit/igt that will prevent any test from running for more than 10 minutes.

Yeah, for the fast run we also need to reduce the piglit timeout to 60s (and kill anything that timed out ofc too). Otherwise there's indeed a gap where a testcase that takes too long could sneak in.
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx

A message that is not in the archive (sent off-list or not yet received): 1442490164-16728-1-git-send-email-thomas.wood-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org

#10 lore raw
Quoted 7 lines · Thomas Wood

Add a script to take a piglit results file and create a list of tests that ran in under 60 seconds. This list can be used by the --test-list option of piglit.

v2: exclude incomplete tests

I think this might be generally useful for piglit uses, maybe in contrib/?

Anyway adding piglit m-l.
-Daniel

Quoted 64 lines
---
 scripts/quick-testlist.py | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100755 scripts/quick-testlist.py
scripts/quick-testlist.py new file +46 -0
new file mode 100755
index 0000000..0dd4c69
@@ -0,0 +1,46 @@
+#!/usr/bin/env python
+#
+# Copyright 2015 Intel Corporation
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice (including the next
+# paragraph) shall be included in all copies or substantial portions of the
+# Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+# IN THE SOFTWARE.
+
+from __future__ import print_function
+import json
+import sys
+
+def filter_results(filename):
+    with open(filename) as data:
+        json_data = json.load(data)
+
+    for test_name in json_data["tests"]:
+        if json_data["tests"][test_name]["result"] == "incomplete":
+            continue
+        if json_data["tests"][test_name]["time"] < 60:
+            print(test_name)
+
+
+if len(sys.argv) < 2:
+    print("Usage: quick-testlist.py RESULTS")
+    print("Read piglit results from RESULTS and print the tests that executed"
+          " in under 60 seconds, excluding any incomplete tests. The list can"
+          " be used by the --test-list option of piglit.")
+    sys.exit(1)
+
+filter_results(sys.argv[1])
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Piglit mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/piglit