Skip to contentSkip to conversation

./scripts/igt_doc.py: speedup check logic

PATCH Superseded Mauro Carvalho Chehab 2 participants 5 messages Apr 12, 2023 Arm/Amlogic Meson SoC support

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover ./scripts/igt_doc.py: speedup check logic

From: Mauro Carvalho Chehab <[email protected]>

The logic which checks for missing/obsolete documentation is too slow. Currently, on my notebook it takes 12 seconds:

	$ time ./scripts/igt_doc.py --check --config tests/xe/xe_*json
	real	0m12.085s
	user	0m10.468s
	sys	0m1.457s

Implent it on a different way, in order to speedup the build process.

After the change, it now takes less than 500ms:

	$ time ./scripts/igt_doc.py --check --config tests/xe/xe_*json
	real	0m0.460s
	user	0m0.357s
	sys	0m0.102s

It should be noticed that it is possible to speed it up even further by running:

	 build/tests/* --list-subtests

tasks in parallel. I did some tests: the improvements weren't so relevant on my notebook with 8 CPUs, and, on my tests with python 3.11 it seemed a little too fragile when using multithreading, as it was not hard to mess with it, not having threads or subprocess forks not happening, and without triggering an error.

As the benefit of such optimization was marginal, let's avoid the additional complexity of multithreading.

---

v2:
- remove a now uneeded loop

Mauro Carvalho Chehab (3):
scripts/igt_doc.py: cleanup some pylint warnings scripts/igt_doc.py: don't depend on igt_runner anymore scripts/test_list.py: use a compiled regex for check

Mauro Carvalho Chehab (3):
scripts/igt_doc.py: cleanup some pylint warnings scripts/igt_doc.py: don't depend on igt_runner anymore scripts/test_list.py: use a compiled regex for check

 docs/testplan/meson.build |  7 ++--
 scripts/igt_doc.py        | 11 +++---
 scripts/test_list.py      | 73 ++++++++++++++++++++++++++-------------
 3 files changed, 55 insertions(+), 36 deletions(-)
Signature · 1 line
2.39.2
#1 lore raw

PATCH v2 1/3 scripts/igt_doc.py: cleanup some pylint warnings

From: Mauro Carvalho Chehab <[email protected]>

Ensure that pylint won't report any issues on igt_doc.py.

---
 scripts/igt_doc.py   | 1 +
 scripts/test_list.py | 4 ++--
 2 files changed, 3 insertions(+), 2 deletions(-)
scripts/igt_doc.py +1 -0
index 547cb81bce02..8fa5af15033e 100755
@@ -11,6 +11,7 @@
 """Maintain test plan and test implementation documentation on IGT."""
 
 import argparse
+import sys
 
 from test_list import TestList
 
scripts/test_list.py +2 -2
index 6af1914822a3..d1b9a2794967 100755
@@ -1,5 +1,5 @@
 #!/usr/bin/env python3
-# pylint: disable=C0301,R0902,R0914,R0912,R0915,R1702,C0302
+# pylint: disable=C0301,R0902,R0914,R0912,R0913,R0915,R1702,C0302
 # SPDX-License-Identifier: (GPL-2.0 OR MIT)
 
 ## Copyright (C) 2023    Intel Corporation                 ##
@@ -394,7 +394,7 @@ class TestList:
         for subtest in self.doc[test]["subtest"].keys():
             summary = test_name
             if self.doc[test]["subtest"][subtest]["Summary"] != '':
-                 summary += '@' + self.doc[test]["subtest"][subtest]["Summary"]
+                summary += '@' + self.doc[test]["subtest"][subtest]["Summary"]
             if not summary:
                 continue
 
Signature · 1 line
2.39.2
#2 lore raw

PATCH v2 2/3 scripts/igt_doc.py: don't depend on igt_runner anymore

In reply to Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <[email protected]>

There's no need to actually call IGT runner to get test lists.

Remove such dependency, in order to speedup --check.

---
 docs/testplan/meson.build |  7 ++----
 scripts/igt_doc.py        | 10 +++-----
 scripts/test_list.py      | 53 ++++++++++++++++++++++++++-------------
 3 files changed, 41 insertions(+), 29 deletions(-)
docs/testplan/meson.build +2 -5
index ea9f6eeb7038..3347f61876ef 100644
@@ -11,11 +11,8 @@ xe_test_config = join_paths(source_root, 'tests', 'xe', 'xe_test_config.json')
 check_testlist = []
 if build_tests
 	doc_dependencies = test_executables
-	if jsonc.found()
-		# Check if documentation matches the actual tests
-		check_testlist = [ '--check-testlist', '--igt-build-path', build_root ]
-		doc_dependencies += runner
-	endif
+	# Check if documentation matches the actual tests
+	check_testlist = [ '--check-testlist', '--igt-build-path', build_root ]
 else
 	doc_dependencies = []
 endif
scripts/igt_doc.py +3 -7
index 8fa5af15033e..01df35f98ace 100755
@@ -16,7 +16,6 @@ import sys
 from test_list import TestList
 
 IGT_BUILD_PATH = 'build'
-IGT_RUNNER = 'runner/igt_runner'
 
 parser = argparse.ArgumentParser(description = "Print formatted kernel documentation to stdout.",
                                  formatter_class = argparse.ArgumentDefaultsHelpFormatter,
@@ -36,24 +35,21 @@ parser.add_argument("--sort-field",
 parser.add_argument("--filter-field",
                     help="modify --show-subtests to filter output based a regex given by FILTER_FIELD=~'regex'")
 parser.add_argument("--check-testlist", action="store_true",
-                    help="Compare documentation against IGT runner testlist.")
+                    help="Compare documentation against IGT built tests.")
 parser.add_argument("--include-plan", action="store_true",
                     help="Include test plans, if any.")
 parser.add_argument("--igt-build-path",
-                    help="Path where the IGT runner is sitting. Used by --check-testlist.",
+                    help="Path to the IGT build directory. Used by --check-testlist.",
                     default=IGT_BUILD_PATH)
 parser.add_argument("--gen-testlist",
                     help="Generate documentation at the GEN_TESTLIST directory, using SORT_FIELD to split the tests. Requires --sort-field.")
-parser.add_argument("--igt-runner",
-                    help="Path where the IGT runner is sitting. Used by --check-testlist.",
-                    default=IGT_RUNNER)
 parser.add_argument('--files', nargs='+',
                     help="File name(s) to be processed")
 
 parse_args = parser.parse_args()
 
 tests = TestList(parse_args.config, parse_args.include_plan, parse_args.files,
-                 parse_args.igt_build_path, parse_args.igt_runner)
+                 parse_args.igt_build_path)
 
 RUN = 0
 if parse_args.show_subtests:
scripts/test_list.py +36 -17

84 lines not on this page · raw message

Signature · 1 line
2.39.2
#3 lore raw

PATCH v2 3/3 scripts/test_list.py: use a compiled regex for check

In reply to Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <[email protected]>

Speed up even further the logic for --check-testlist by pre-compiling the regex that will be used to validate the results.

---
 scripts/test_list.py | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)
scripts/test_list.py +11 -5
index 287351e717b3..96d8883db122 100755
@@ -820,10 +820,18 @@ class TestList:
         run_missing = []
         doc_uneeded = []
 
+        test_regex = r""
+        for doc_test in doc_subtests:
+            if test_regex != r"":
+                test_regex += r"|"
+            test_regex += r'^' + doc_test + r'$'
+
+        test_regex = re.compile(test_regex)
+
         for doc_test in doc_subtests:
             found = False
             for run_test in run_subtests:
-                if re.match(r'^' + doc_test + r'$', run_test):
+                if re.match(test_regex, run_test):
                     found = True
                     break
             if not found:
@@ -831,10 +839,8 @@ class TestList:
 
         for run_test in run_subtests:
             found = False
-            for doc_test in doc_subtests:
-                if re.match(r'^' + doc_test + r'$', run_test):
-                    found = True
-                    break
+            if re.match(test_regex, run_test):
+                found = True
             if not found:
                 run_missing.append(run_test)
 
Signature · 1 line
2.39.2
#4 lore raw

[igt-dev] ✗ Fi.CI.BAT: failure for ./scripts/igt_doc.py: speedup check logic (rev2)

In reply to Mauro Carvalho Chehab
== Series Details ==

== Series Details ==

Series: ./scripts/igt_doc.py: speedup check logic (rev2)
URL : https://patchwork.freedesktop.org/series/116379/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_12995 -> IGTPW_8790
====================================================

Summary
-------

**FAILURE**

Serious unknown changes coming with IGTPW_8790 absolutely need to be verified manually.

If you think the reported changes have nothing to do with the changes introduced in IGTPW_8790, please notify your bug team to allow them to document this new failure mode, which will reduce false positives in CI.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/index.html

Participating hosts (37 -> 35)
------------------------------

Missing (2): fi-kbl-soraka fi-snb-2520m

Possible new issues
-------------------

Here are the unknown changes that may have been introduced in IGTPW_8790:

### IGT changes ###

#### Possible regressions ####

* igt@i915_selftest@live@hangcheck:
- fi-hsw-4770: [PASS][1] -> [DMESG-WARN][2]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/fi-hsw-4770/igt@i915_selftest@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/fi-hsw-4770/igt@i915_selftest@[email protected]

Known issues
------------

Here are the changes found in IGTPW_8790 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@i915_pm_rps@basic-api:
- bat-dg2-11: [PASS][3] -> [FAIL][4] ([i915#8308])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-dg2-11/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-dg2-11/igt@[email protected]

* igt@i915_selftest@live@gt_heartbeat:
- fi-glk-j4005: [PASS][5] -> [DMESG-FAIL][6] ([i915#5334])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/fi-glk-j4005/igt@i915_selftest@live@gt_heartbeat.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/fi-glk-j4005/igt@i915_selftest@live@gt_heartbeat.html
- fi-skl-guc: [PASS][7] -> [DMESG-FAIL][8] ([i915#5334])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/fi-skl-guc/igt@i915_selftest@live@gt_heartbeat.html
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/fi-skl-guc/igt@i915_selftest@live@gt_heartbeat.html

* igt@i915_selftest@live@hangcheck:
- bat-dg2-11: [PASS][9] -> [ABORT][10] ([i915#7913] / [i915#7979])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-dg2-11/igt@i915_selftest@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-dg2-11/igt@i915_selftest@[email protected]

* igt@i915_selftest@live@migrate:
- bat-dg2-11: [PASS][11] -> [DMESG-WARN][12] ([i915#7699])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-dg2-11/igt@i915_selftest@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-dg2-11/igt@i915_selftest@[email protected]

* igt@i915_selftest@live@reset:
- bat-rpls-2: [PASS][13] -> [ABORT][14] ([i915#4983] / [i915#7913] / [i915#7981])
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-rpls-2/igt@i915_selftest@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-2/igt@i915_selftest@[email protected]

* igt@i915_selftest@live@slpc:
- bat-rpls-1: [PASS][15] -> [DMESG-FAIL][16] ([i915#6367] / [i915#7996])
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-rpls-1/igt@i915_selftest@[email protected]
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-1/igt@i915_selftest@[email protected]
- bat-adln-1: NOTRUN -> [DMESG-FAIL][17] ([i915#6997])
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-adln-1/igt@i915_selftest@[email protected]

* igt@kms_chamelium_hpd@common-hpd-after-suspend:
- bat-adln-1: NOTRUN -> [SKIP][18] ([i915#7828])
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-adln-1/igt@[email protected]
- bat-rpls-1: NOTRUN -> [SKIP][19] ([i915#7828])
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-1/igt@[email protected]

* igt@kms_pipe_crc_basic@read-crc:
- bat-adlp-9: NOTRUN -> [SKIP][20] ([i915#3546]) +1 similar issue
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-adlp-9/igt@[email protected]

* igt@kms_pipe_crc_basic@suspend-read-crc:
- bat-rpls-1: NOTRUN -> [SKIP][21] ([i915#1845])
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-1/igt@[email protected]

#### Possible fixes ####

* igt@gem_exec_suspend@basic-s3@smem:
- bat-rpls-1: [ABORT][22] ([i915#6687] / [i915#7978]) -> [PASS][23]
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-rpls-1/igt@gem_exec_suspend@[email protected]
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-rpls-1/igt@gem_exec_suspend@[email protected]

* igt@i915_module_load@load:
- fi-elk-e7500: [DMESG-WARN][24] ([i915#4391]) -> [PASS][25]
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/fi-elk-e7500/igt@[email protected]
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/fi-elk-e7500/igt@[email protected]

* igt@i915_selftest@live@gt_lrc:
- bat-adln-1: [INCOMPLETE][26] ([i915#7609]) -> [PASS][27]
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-adln-1/igt@i915_selftest@live@gt_lrc.html
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-adln-1/igt@i915_selftest@live@gt_lrc.html

* igt@kms_pipe_crc_basic@nonblocking-crc-frame-sequence@pipe-d-dp-1:
- bat-dg2-8: [FAIL][28] ([i915#7932]) -> [PASS][29]
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12995/bat-dg2-8/igt@kms_pipe_crc_basic@[email protected]
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/bat-dg2-8/igt@kms_pipe_crc_basic@[email protected]

[i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
[i915#3546]: https://gitlab.freedesktop.org/drm/intel/issues/3546
[i915#4391]: https://gitlab.freedesktop.org/drm/intel/issues/4391
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#5334]: https://gitlab.freedesktop.org/drm/intel/issues/5334
[i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
[i915#6687]: https://gitlab.freedesktop.org/drm/intel/issues/6687
[i915#6997]: https://gitlab.freedesktop.org/drm/intel/issues/6997
[i915#7609]: https://gitlab.freedesktop.org/drm/intel/issues/7609
[i915#7699]: https://gitlab.freedesktop.org/drm/intel/issues/7699
[i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828
[i915#7913]: https://gitlab.freedesktop.org/drm/intel/issues/7913
[i915#7932]: https://gitlab.freedesktop.org/drm/intel/issues/7932
[i915#7978]: https://gitlab.freedesktop.org/drm/intel/issues/7978
[i915#7979]: https://gitlab.freedesktop.org/drm/intel/issues/7979
[i915#7981]: https://gitlab.freedesktop.org/drm/intel/issues/7981
[i915#7996]: https://gitlab.freedesktop.org/drm/intel/issues/7996
[i915#8308]: https://gitlab.freedesktop.org/drm/intel/issues/8308

Build changes
-------------

* CI: CI-20190529 -> None
* IGT: IGT_7251 -> IGTPW_8790

CI-20190529: 20190529
CI_DRM_12995: 0c5be3027d7ac4a8b107bf1113867744aa3bb440 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_8790: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/index.html
IGT_7251: 55fa959aad79b3771350a801c1c2dbd4e5034102 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_8790/index.html