Skip to content

Dynamic subtests

Petri Latvala igt-dev 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest6 patches 5 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/6lib: Introduce dynamic subtestslib/igt_core.c lib/igt_core.h+197 -110 replies
2/6lib/tests: Unit tests for dynamic subtestslib/tests/igt_dynamic_subtests.c lib/tests/meson.build+187 -00 replies
3/6runner/resultgen: Refactor output parsingrunner/resultgen.c+134 -1250 replies
4/6runner/json_tests: Adapt to better output parsingrunner/json_tests_data/aborted-after-a-test/reference.json runner/json_tests_data/dmesg-escapes/reference.json runner/json_tests_data/dmesg-results/reference.json +9 more+30 -300 replies
5/6runner: Parse dynamic subtest outputs and resultsrunner/job_list.c runner/job_list.h runner/output_strings.h +1 more+321 -620 replies
6/6runner/json_tests: Test dynamic subtestsrunner/json_tests_data/dynamic-subtests/0/dmesg.txt runner/json_tests_data/dynamic-subtests/0/err.txt runner/json_tests_data/dynamic-subtests/0/journal.txt +17 more+284 -10 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [igt-dev] [PATCH i-g-t 0/6] Dynamic subtests

Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?

Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?

Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?

If so, this is your lucky day, for dynamic subtests are here!

While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:

With normal subtests:
for_each_pipe_static(pipe)

  igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
    int fd = drm_open_driver(DRIVER_ANY);
    igt_require(actually_have_this_pipe(fd, pipe));
    igt_assert(do_stuff(fd, pipe));

}

With dynamic subtests:
igt_subtest_container("fudge")
int fd = drm_open_driver(DRIVER_ANY);

  for_each_pipe(fd, pipe) {
    igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
      igt_assert(do_stuff(fd, pipe));
    }

}

Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.

Semantics
=========

Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it igt_subtest_container().

A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
- igt_assert not allowed
- igt_skip / igt_require is allowed though

Any failing dynamic subtest will result in the container reporting a failure.

Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using igt_require in an igt_dynamic_subtest, just don't enter it for such an occasion. In other words:

Do not:
for_each_pipe(fd, pipe)

  igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
    igt_require(pipe_has_things(pipe));
    ...

}

Instead do:
for_each_pipe(fd, pipe) {
if (!pipe_has_things(pipe))

    continue;
  igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
    ...

}
}

That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.

Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.

Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of igt_fixture usage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.

Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:

igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.

igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...

Existing tests that are good candidates for conversion to dynamic subtests:

1) Anything using for_each_pipe_static

2) Tests that have one subtest per i915 engine

3) Kernel selftests

Petri Latvala (6):
lib: Introduce dynamic subtests lib/tests: Unit tests for dynamic subtests runner/resultgen: Refactor output parsing runner/json_tests: Adapt to better output parsing
runner: Parse dynamic subtest outputs and results runner/json_tests: Test dynamic subtests

 lib/igt_core.c                                | 119 +++-
 lib/igt_core.h                                |  89 +++
 lib/tests/igt_dynamic_subtests.c              | 186 ++++++
 lib/tests/meson.build                         |   1 +
 runner/job_list.c                             |  11 +
 runner/job_list.h                             |   3 +
 .../aborted-after-a-test/reference.json       |   2 +-
 .../dmesg-escapes/reference.json              |   2 +-
 .../dmesg-results/reference.json              |   8 +-
 .../reference.json                            |   4 +-
 .../reference.json                            |   4 +-
 .../dmesg-warn-level/reference.json           |   4 +-
 .../dynamic-subtests/0/dmesg.txt              |   7 +
 .../dynamic-subtests/0/err.txt                |  36 ++
 .../dynamic-subtests/0/journal.txt            |   2 +
 .../dynamic-subtests/0/out.txt                |  19 +
 .../dynamic-subtests/1/dmesg.txt              |   5 +
 .../dynamic-subtests/1/err.txt                |   2 +
 .../dynamic-subtests/1/journal.txt            |   2 +
 .../dynamic-subtests/1/out.txt                |   5 +
 .../dynamic-subtests/2/dmesg.txt              |  10 +
 .../dynamic-subtests/2/err.txt                |   6 +
 .../dynamic-subtests/2/journal.txt            |   4 +
 .../dynamic-subtests/2/out.txt                |   8 +
 .../dynamic-subtests/README.txt               |   2 +
 .../dynamic-subtests/endtime.txt              |   1 +
 .../dynamic-subtests/joblist.txt              |   3 +
 .../dynamic-subtests/metadata.txt             |  12 +
 .../dynamic-subtests/reference.json           | 156 +++++
 .../dynamic-subtests/starttime.txt            |   1 +
 .../dynamic-subtests/uname.txt                |   1 +
 .../json_tests_data/normal-run/reference.json |   8 +-
 .../reference.json                            |   2 +-
 .../notrun-results/reference.json             |   2 +-
 .../piglit-style-dmesg/reference.json         |   8 +-
 .../warnings-with-dmesg-warns/reference.json  |   8 +-
 .../json_tests_data/warnings/reference.json   |   8 +-
 runner/output_strings.h                       |  29 +-
 runner/resultgen.c                            | 589 ++++++++++++------
 runner/runner_json_tests.c                    |   3 +-
 40 files changed, 1148 insertions(+), 224 deletions(-)
 create mode 100644 lib/tests/igt_dynamic_subtests.c
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/0/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/1/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/dmesg.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/err.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/journal.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/2/out.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/README.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/endtime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/joblist.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/metadata.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/reference.json
 create mode 100644 runner/json_tests_data/dynamic-subtests/starttime.txt
 create mode 100644 runner/json_tests_data/dynamic-subtests/uname.txt
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev