Dynamic subtests
Get series with b4
b4 am [email protected]Versions newest first
- v1 Latest6 patches 5 repliesIn review
Patches in v1 No applies recorded yet
| # | Subject and files | Size | Replies | Applied |
|---|---|---|---|---|
| 1/6 | lib: Introduce dynamic subtestslib/ lib/ | +197 | 0 replies | |
| 2/6 | lib/tests: Unit tests for dynamic subtestslib/ lib/ | +187 | 0 replies | |
| 3/6 | runner/resultgen: Refactor output parsingrunner/ | +134 | 0 replies | |
| 4/6 | runner/json_tests: Adapt to better output parsingrunner/ runner/ runner/ +9 more | +30 | 0 replies | |
| 5/6 | runner: Parse dynamic subtest outputs and resultsrunner/ runner/ runner/ +1 more | +321 | 0 replies | |
| 6/6 | runner/json_tests: Test dynamic subtestsrunner/ runner/ runner/ +17 more | +284 | 0 replies |
Reviews trailers per patch
No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.
Cover letter [igt-dev] [PATCH i-g-t 0/6] Dynamic subtests
Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?
Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?
Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?
If so, this is your lucky day, for dynamic subtests are here!
While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:
With normal subtests:for_each_pipe_static(pipe)
igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
int fd = drm_open_driver(DRIVER_ANY);
igt_require(actually_have_this_pipe(fd, pipe));
igt_assert(do_stuff(fd, pipe));}
With dynamic subtests:igt_subtest_container("fudge")
int fd = drm_open_driver(DRIVER_ANY);
for_each_pipe(fd, pipe) {
igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
igt_assert(do_stuff(fd, pipe));
}}
Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.
Semantics
=========
Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it igt_subtest_container().
A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
- igt_assert not allowed
- igt_skip / igt_require is allowed though
Any failing dynamic subtest will result in the container reporting a failure.
Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using igt_require in an igt_dynamic_subtest, just don't enter it for such an occasion. In other words:
Do not:for_each_pipe(fd, pipe)
igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
igt_require(pipe_has_things(pipe));
...}
Instead do:for_each_pipe(fd, pipe) {
if (!pipe_has_things(pipe))
continue;
igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
... }
}
That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.
Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.
Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of igt_fixture usage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.
Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:
igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.
igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...
Existing tests that are good candidates for conversion to dynamic subtests:
1) Anything using for_each_pipe_static
2) Tests that have one subtest per i915 engine
3) Kernel selftests
Petri Latvala (6):
lib: Introduce dynamic subtests lib/tests: Unit tests for dynamic subtests runner/resultgen: Refactor output parsing runner/json_tests: Adapt to better output parsing
runner: Parse dynamic subtest outputs and results runner/json_tests: Test dynamic subtests
lib/igt_core.c | 119 +++- lib/igt_core.h | 89 +++ lib/tests/igt_dynamic_subtests.c | 186 ++++++ lib/tests/meson.build | 1 + runner/job_list.c | 11 + runner/job_list.h | 3 + .../aborted-after-a-test/reference.json | 2 +- .../dmesg-escapes/reference.json | 2 +- .../dmesg-results/reference.json | 8 +- .../reference.json | 4 +- .../reference.json | 4 +- .../dmesg-warn-level/reference.json | 4 +- .../dynamic-subtests/0/dmesg.txt | 7 + .../dynamic-subtests/0/err.txt | 36 ++ .../dynamic-subtests/0/journal.txt | 2 + .../dynamic-subtests/0/out.txt | 19 + .../dynamic-subtests/1/dmesg.txt | 5 + .../dynamic-subtests/1/err.txt | 2 + .../dynamic-subtests/1/journal.txt | 2 + .../dynamic-subtests/1/out.txt | 5 + .../dynamic-subtests/2/dmesg.txt | 10 + .../dynamic-subtests/2/err.txt | 6 + .../dynamic-subtests/2/journal.txt | 4 + .../dynamic-subtests/2/out.txt | 8 + .../dynamic-subtests/README.txt | 2 + .../dynamic-subtests/endtime.txt | 1 + .../dynamic-subtests/joblist.txt | 3 + .../dynamic-subtests/metadata.txt | 12 + .../dynamic-subtests/reference.json | 156 +++++ .../dynamic-subtests/starttime.txt | 1 + .../dynamic-subtests/uname.txt | 1 + .../json_tests_data/normal-run/reference.json | 8 +- .../reference.json | 2 +- .../notrun-results/reference.json | 2 +- .../piglit-style-dmesg/reference.json | 8 +- .../warnings-with-dmesg-warns/reference.json | 8 +- .../json_tests_data/warnings/reference.json | 8 +- runner/output_strings.h | 29 +- runner/resultgen.c | 589 ++++++++++++------ runner/runner_json_tests.c | 3 +- 40 files changed, 1148 insertions(+), 224 deletions(-) create mode 100644 lib/tests/igt_dynamic_subtests.c create mode 100644 runner/json_tests_data/dynamic-subtests/0/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/README.txt create mode 100644 runner/json_tests_data/dynamic-subtests/endtime.txt create mode 100644 runner/json_tests_data/dynamic-subtests/joblist.txt create mode 100644 runner/json_tests_data/dynamic-subtests/metadata.txt create mode 100644 runner/json_tests_data/dynamic-subtests/reference.json create mode 100644 runner/json_tests_data/dynamic-subtests/starttime.txt create mode 100644 runner/json_tests_data/dynamic-subtests/uname.txt
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev