Dynamic subtests
Subsystems (2)
cover Dynamic subtests
Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?
Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?
Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?
If so, this is your lucky day, for dynamic subtests are here!
While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:
With normal subtests:for_each_pipe_static(pipe)
igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) {
int fd = drm_open_driver(DRIVER_ANY);
igt_require(actually_have_this_pipe(fd, pipe));
igt_assert(do_stuff(fd, pipe));}
With dynamic subtests:igt_subtest_container("fudge")
int fd = drm_open_driver(DRIVER_ANY);
for_each_pipe(fd, pipe) {
igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) {
igt_assert(do_stuff(fd, pipe));
}}
Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.
Semantics
=========
Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it igt_subtest_container().
A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
- igt_assert not allowed
- igt_skip / igt_require is allowed though
Any failing dynamic subtest will result in the container reporting a failure.
Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using igt_require in an igt_dynamic_subtest, just don't enter it for such an occasion. In other words:
Do not:for_each_pipe(fd, pipe)
igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
igt_require(pipe_has_things(pipe));
...}
Instead do:for_each_pipe(fd, pipe) {
if (!pipe_has_things(pipe))
continue;
igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) {
... }
}
That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.
Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.
Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of igt_fixture usage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.
Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:
igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.
igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...
Existing tests that are good candidates for conversion to dynamic subtests:
1) Anything using for_each_pipe_static
2) Tests that have one subtest per i915 engine
3) Kernel selftests
Petri Latvala (6):
lib: Introduce dynamic subtests lib/tests: Unit tests for dynamic subtests runner/resultgen: Refactor output parsing runner/json_tests: Adapt to better output parsing
runner: Parse dynamic subtest outputs and results runner/json_tests: Test dynamic subtests
lib/igt_core.c | 119 +++- lib/igt_core.h | 89 +++ lib/tests/igt_dynamic_subtests.c | 186 ++++++ lib/tests/meson.build | 1 + runner/job_list.c | 11 + runner/job_list.h | 3 + .../aborted-after-a-test/reference.json | 2 +- .../dmesg-escapes/reference.json | 2 +- .../dmesg-results/reference.json | 8 +- .../reference.json | 4 +- .../reference.json | 4 +- .../dmesg-warn-level/reference.json | 4 +- .../dynamic-subtests/0/dmesg.txt | 7 + .../dynamic-subtests/0/err.txt | 36 ++ .../dynamic-subtests/0/journal.txt | 2 + .../dynamic-subtests/0/out.txt | 19 + .../dynamic-subtests/1/dmesg.txt | 5 + .../dynamic-subtests/1/err.txt | 2 + .../dynamic-subtests/1/journal.txt | 2 + .../dynamic-subtests/1/out.txt | 5 + .../dynamic-subtests/2/dmesg.txt | 10 + .../dynamic-subtests/2/err.txt | 6 + .../dynamic-subtests/2/journal.txt | 4 + .../dynamic-subtests/2/out.txt | 8 + .../dynamic-subtests/README.txt | 2 + .../dynamic-subtests/endtime.txt | 1 + .../dynamic-subtests/joblist.txt | 3 + .../dynamic-subtests/metadata.txt | 12 + .../dynamic-subtests/reference.json | 156 +++++ .../dynamic-subtests/starttime.txt | 1 + .../dynamic-subtests/uname.txt | 1 + .../json_tests_data/normal-run/reference.json | 8 +- .../reference.json | 2 +- .../notrun-results/reference.json | 2 +- .../piglit-style-dmesg/reference.json | 8 +- .../warnings-with-dmesg-warns/reference.json | 8 +- .../json_tests_data/warnings/reference.json | 8 +- runner/output_strings.h | 29 +- runner/resultgen.c | 589 ++++++++++++------ runner/runner_json_tests.c | 3 +- 40 files changed, 1148 insertions(+), 224 deletions(-) create mode 100644 lib/tests/igt_dynamic_subtests.c create mode 100644 runner/json_tests_data/dynamic-subtests/0/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/README.txt create mode 100644 runner/json_tests_data/dynamic-subtests/endtime.txt create mode 100644 runner/json_tests_data/dynamic-subtests/joblist.txt create mode 100644 runner/json_tests_data/dynamic-subtests/metadata.txt create mode 100644 runner/json_tests_data/dynamic-subtests/reference.json create mode 100644 runner/json_tests_data/dynamic-subtests/starttime.txt create mode 100644 runner/json_tests_data/dynamic-subtests/uname.txt
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
PATCH 1/6 lib: Introduce dynamic subtests
Dynamic subtests, or subtests of subtests, are individual pieces of tests that are not statically available all the time.
A good example of a need for a dynamic subtest is i915 engine listing:
A normal subtest for each engine class ("bsd"), and a dynamic subtest for each instance ("bsd0", "bsd2", etc). Or a normal subtest for an operation with a dynamic subtest for every engine there is.
Another example is a dynamic subtest for pipes: Instead of using foreach_pipe_static, make one subtest and use foreach_pipe with dynamic subtests for each pipe.
--- lib/igt_core.c | 119 ++++++++++++++++++++++++++++++++++++++++++++----- lib/igt_core.h | 89 ++++++++++++++++++++++++++++++++++++ 2 files changed, 197 insertions(+), 11 deletions(-)
lib/igt_core.c +108 -11
@@ -260,9 +260,12 @@ const char *igt_interactive_debug;/* subtests helpers */ static bool list_subtests = false;static char *run_single_subtest =NULL ; +static char *run_single_dynamic_subtest =NULL ;static bool run_single_subtest_found = false;static const char *in_subtest =NULL ; +static const char *in_dynamic_subtest =NULL ;static struct timespec subtest_time; +static struct timespec dynamic_subtest_time;static clockid_t igt_clock = (clockid_t)-1 ;static bool in_fixture = false;static bool test_with_subtests = false;
@@ -286,6 +289,7 @@ enum {*/ OPT_LIST_SUBTESTS =500 ,OPT_RUN_SUBTEST , +OPT_RUN_DYNAMIC_SUBTEST ,OPT_DESCRIPTION ,OPT_DEBUG , OPT_INTERACTIVE_DEBUG,
@@ -308,6 +312,8 @@ char *igt_frame_dump_path;static bool stderr_needs_sentinel = false; +static int _igt_dynamic_tests_executed =-1 ; +const char *igt_test_name (void ) {return command_str;
@@ -339,7 +345,9 @@ static void _igt_log_buffer_dump(void) {uint8_t i; -if (in_subtest) +if (in_dynamic_subtest) +fprintf (stderr,"Dynamic subtest %s failed.\n" , in_dynamic_subtest); +else if (in_subtest)fprintf (stderr,"Subtest %s failed.\n" , in_subtest);else fprintf (stderr,"Test %s failed.\n" , command_str);
@@ -555,6 +563,7 @@ static void print_usage(const char *help_str, bool output_on_stderr)fprintf (f,"Usage: %s [OPTIONS]\n" , command_str);fprintf (f," --list-subtests\n" " --run-subtest <pattern>\n" +" --dynamic-subtest <pattern>\n" " --debug[=log-domain]\n" " --interactive-debug[=domain]\n" " --help-description\n"
@@ -672,6 +681,7 @@ static int common_init(int *argc, char **argv,static struct option long_options[] = { {"list-subtests" , no_argument,NULL ,OPT_LIST_SUBTESTS }, {"run-subtest" , required_argument,NULL ,OPT_RUN_SUBTEST }, + {"dynamic-subtest" , required_argument,NULL ,OPT_RUN_DYNAMIC_SUBTEST }, {"help-description" , no_argument,NULL ,OPT_DESCRIPTION }, {"debug" , optional_argument,NULL ,OPT_DEBUG }, {"interactive-debug" , optional_argument,NULL ,OPT_INTERACTIVE_DEBUG },
@@ -782,6 +792,11 @@ static int common_init(int *argc, char **argv,if (!list_subtests) run_single_subtest =strdup (optarg);break ; +case OPT_RUN_DYNAMIC_SUBTEST : +assert (optarg); +if (!list_subtests) + run_single_dynamic_subtest =strdup (optarg); +break ;case OPT_DESCRIPTION :print_test_description (); ret =-1 ;
@@ -992,6 +1007,41 @@ bool __igt_run_subtest(const char *subtest_name)return (in_subtest = subtest_name); } +bool __igt_run_dynamic_subtest (const char *dynamic_subtest_name) +{ +int i; + +assert (in_subtest); +assert (_igt_dynamic_tests_executed >=0 ); + +/* check the dynamic_subtest name only contains a-z, A-Z, 0-9, '-' and '_' */ +for (i =0 ; dynamic_subtest_name[i] !='\0' ; i++) +if (dynamic_subtest_name[i] !='_' && dynamic_subtest_name[i] !='-' + && !isalnum (dynamic_subtest_name[i])) { +igt_critical ("Invalid dynamic subtest name \"%s\".\n" , + dynamic_subtest_name); +igt_exit (); + } + +if (run_single_dynamic_subtest && +uwildmat (dynamic_subtest_name, run_single_dynamic_subtest) ==0 ) +return false; + +igt_kmsg (KMSG_INFO "%s: starting dynamic subtest %s\n" , + command_str, dynamic_subtest_name); +igt_info ("Starting dynamic subtest: %s\n" , dynamic_subtest_name); +fflush (stdout); +if (stderr_needs_sentinel) +fprintf (stderr,"Starting dynamic subtest: %s\n" , dynamic_subtest_name); + +_igt_log_buffer_reset (); + + _igt_dynamic_tests_executed++; + +igt_gettime (&dynamic_subtest_time); +return (in_dynamic_subtest = dynamic_subtest_name); +} +/** * igt_subtest_name: *
@@ -1029,26 +1079,50 @@ void __igt_subtest_group_restore(int save)static bool skipped_one = false;static bool succeeded_one = false;static bool failed_one = false; +static bool dynamic_failed_one = false; + +bool __igt_enter_dynamic_container (void ) +{ + _igt_dynamic_tests_executed =0 ; + dynamic_failed_one = false; + +return true; +}static void exit_subtest (const char *) __attribute__((noreturn));static void exit_subtest (const char *result) {struct timespec now; +const char *subtest_text = in_dynamic_subtest ?"Dynamic subtest" :"Subtest" ; +const char **subtest_name = in_dynamic_subtest ? &in_dynamic_subtest : &in_subtest; +struct timespec *thentime = in_dynamic_subtest ? &dynamic_subtest_time : &subtest_time; +jmp_buf *jmptarget = in_dynamic_subtest ? &igt_dynamic_subtest_jmpbuf : &igt_subtest_jmpbuf;igt_gettime (&now); -igt_info ("%sSubtest %s: %s (%.3fs)%s\n" , + +igt_info ("%s%s %s: %s (%.3fs)%s\n" , (!__igt_plain_output) ?"\x1b[1m" :"" , - in_subtest, result,igt_time_elapsed (&subtest_time, &now), + subtest_text, *subtest_name, result, +igt_time_elapsed (thentime, &now), (!__igt_plain_output) ?"\x1b[0m" :"" );fflush (stdout);if (stderr_needs_sentinel) -fprintf (stderr,"Subtest %s: %s (%.3fs)\n" , - in_subtest, result,igt_time_elapsed (&subtest_time, &now)); +fprintf (stderr,"%s %s: %s (%.3fs)\n" , + subtest_text, *subtest_name, + result,igt_time_elapsed (thentime, &now));igt_terminate_spins (); - in_subtest =NULL ; -siglongjmp (igt_subtest_jmpbuf,1 ); +if (!in_dynamic_subtest) + _igt_dynamic_tests_executed =-1 ; + +/* Don't keep the above text in the log, the container would print it again otherwise */ +if (in_dynamic_subtest) +_igt_log_buffer_reset (); + + *subtest_name =NULL ; + +siglongjmp (*jmptarget,1 ); }/**
@@ -1079,6 +1153,7 @@ void igt_skip(const char *f, ...) }if (in_subtest) { +/* Doing the same even if inside a dynamic subtest */ exit_subtest ("SKIP" ); }else if (test_with_subtests) { skip_subtests_henceforth =SKIP ;
@@ -1135,7 +1210,22 @@ void __igt_skip_check(const char *file, const int line,*/ void igt_success (void ) { - succeeded_one = true; +if (in_subtest && !in_dynamic_subtest && _igt_dynamic_tests_executed >=0 ) { +/* +* We're exiting a dynamic container, yield a result +* according to the dynamic tests that got +* executed. +*/ +if (dynamic_failed_one) +igt_fail (IGT_EXIT_FAILURE ); + +if (_igt_dynamic_tests_executed ==0 ) +igt_skip ("No dynamic tests executed.\n" ); + } + +if (!in_dynamic_subtest) + succeeded_one = true; +if (in_subtest)exit_subtest ("SUCCESS" ); }
@@ -1166,10 +1256,17 @@ void igt_fail(int exitcode)if (in_atexit_handler)_exit (IGT_EXIT_FAILURE ); -if (!failed_one) - igt_exitcode = exitcode; +if (in_dynamic_subtest) { + dynamic_failed_one = true; + }else { +/* Dynamic subtest containers must not fail explicitly */ +assert (_igt_dynamic_tests_executed <0 || dynamic_failed_one); + +if (!failed_one) + igt_exitcode = exitcode; - failed_one = true; + failed_one = true; + }/* Silent exit, parent will do the yelling. */ if (test_child)
lib/igt_core.h +89 -0
107 lines not on this page · raw message
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
PATCH 2/6 lib/tests: Unit tests for dynamic subtests
In reply to Petri Latvala--- lib/tests/igt_dynamic_subtests.c | 186 +++++++++++++++++++++++++++++++ lib/tests/meson.build | 1 + 2 files changed, 187 insertions(+) create mode 100644 lib/tests/igt_dynamic_subtests.c
lib/tests/igt_dynamic_subtests.c new file +186 -0
@@ -0,0 +1,186 @@ +/* +* Copyright © 2019 Intel Corporation +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice (including the next +* paragraph) shall be included in all copies or substantial portions of the +* Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +* IN THE SOFTWARE. +*/ + +#include <errno.h> +#include <sys/wait.h> + +#include "igt_core.h" +#include "drmtest.h" + +#include "igt_tests_common.h" + +static int do_fork (void (*test_to_run)(void )) +{ +int pid, status; + +switch (pid =fork ()) { +case -1 : +internal_assert (0 ); +case 0 : +test_to_run (); +default : +while (waitpid (pid, &status,0 ) ==-1 && + errno ==EINTR ) + ; + +return status; + } +} + +static void dynamic_subtest_in_normal_subtest (void ) +{ +char prog[] ="igt_no_exit" ; +char *fake_argv[] = {prog}; +int fake_argc =ARRAY_SIZE (fake_argv); + +igt_subtest_init (fake_argc, fake_argv); + + igt_subtest("normal-subtest") { + igt_dynamic_subtest("dynamic") { +igt_info ("Dynamic subtest in normal subtest\n" ); + } + } + +igt_exit (); +} + +static void invalid_dynamic_subtest_name (void ) +{ +char prog[] ="igt_no_exit" ; +char *fake_argv[] = {prog}; +int fake_argc =ARRAY_SIZE (fake_argv); + +igt_subtest_init (fake_argc, fake_argv); + + igt_dynamic_subtest_container("subtest") { + igt_dynamic_subtest("# invalid name !") { +igt_info ("Invalid dynamic subtest name test\n" ); + } + } + +igt_exit (); +} + +static void dynamic_subtest_in_toplevel (void ) +{ +char prog[] ="igt_no_exit" ; +char *fake_argv[] = {prog}; +int fake_argc =ARRAY_SIZE (fake_argv); + +igt_subtest_init (fake_argc, fake_argv); + + igt_dynamic_subtest("dynamic-subtest-in-toplevel") { +igt_info ("Dynamic subtests need to be in a subtest\n" ); + } + +igt_exit (); +} + +static void subtest_itself_failing (void ) +{ +char prog[] ="igt_no_exit" ; +char *fake_argv[] = {prog}; +int fake_argc =ARRAY_SIZE (fake_argv); + +igt_subtest_init (fake_argc, fake_argv); + + igt_dynamic_subtest_container("subtest") { +igt_assert (false); + } + +igt_exit (); +} + +static void subtest_itself_skipping (void ) +{ +char prog[] ="igt_no_exit" ; +char *fake_argv[] = {prog}; +int fake_argc =ARRAY_SIZE (fake_argv); + +igt_subtest_init (fake_argc, fake_argv); + + igt_dynamic_subtest_container("subtest") { +igt_skip ("Skipping\n" ); + } + +igt_exit (); +} + +static void dynamic_subtest_failure_leads_to_fail (void ) +{ +char prog[] ="igt_no_exit" ; +char *fake_argv[] = {prog}; +int fake_argc =ARRAY_SIZE (fake_argv); + +igt_subtest_init (fake_argc, fake_argv); + + igt_dynamic_subtest_container("subtest") { + igt_dynamic_subtest("dynamic") { +igt_assert (false); + } + } + +igt_exit (); +} + +static void no_dynamic_subtests_entered_leads_to_skip (void ) +{ +char prog[] ="igt_no_exit" ; +char *fake_argv[] = {prog}; +int fake_argc =ARRAY_SIZE (fake_argv); + +igt_subtest_init (fake_argc, fake_argv); + + igt_dynamic_subtest_container("subtest") { + } + +igt_exit (); +} + +int main (int argc,char **argv) +{ +int ret; + + ret =do_fork (dynamic_subtest_in_normal_subtest); +internal_assert_wsignaled (ret,SIGABRT ); + + ret =do_fork (invalid_dynamic_subtest_name); +internal_assert_wsignaled (ret,SIGABRT ); + + ret =do_fork (dynamic_subtest_in_toplevel); +internal_assert_wsignaled (ret,SIGABRT ); + + ret =do_fork (subtest_itself_failing); +internal_assert_wsignaled (ret,SIGABRT ); + + ret =do_fork (subtest_itself_skipping); +internal_assert_wexited (ret,IGT_EXIT_SKIP ); + + ret =do_fork (dynamic_subtest_failure_leads_to_fail); +internal_assert_wexited (ret,IGT_EXIT_FAILURE ); + + ret =do_fork (no_dynamic_subtests_entered_leads_to_skip); +internal_assert_wexited (ret,IGT_EXIT_SKIP ); + +return 0 ; +}
lib/tests/meson.build +1 -0
@@ -3,6 +3,7 @@ lib_tests = [ 'igt_can_fail', 'igt_can_fail_simple', 'igt_conflicting_args', + 'igt_dynamic_subtests', 'igt_edid', 'igt_exit_handler', 'igt_fork',
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
PATCH 3/6 runner/resultgen: Refactor output parsing
In reply to Petri LatvalaInstead of searching back and forth for proper lines, first find all lines that we could be interested in with one pass through the output, and use the positions of found lines to delimit the extracted outputs.
--- runner/resultgen.c | 259 +++++++++++++++++++++++---------------------- 1 file changed, 134 insertions(+), 125 deletions(-)
runner/resultgen.c +134 -125
@@ -10,6 +10,7 @@#include <json.h> +#include "igt_aux.h" #include "igt_core.h" #include "resultgen.h" #include "settings.h"
@@ -59,28 +60,7 @@ static char *find_line_starting_with(char *haystack, const char *needle, char *ereturn NULL ; } -static char *find_line_starting_with_either (char *haystack, -const char *needle1, -const char *needle2, -char *end) -{ -while (haystack < end) { -char *line_end =memchr (haystack,'\n' , end - haystack); -size_t linelen = line_end !=NULL ? line_end - haystack : end - haystack; -if ((linelen >=strlen (needle1) && !memcmp (haystack, needle1,strlen (needle1))) || - (linelen >=strlen (needle2) && !memcmp (haystack, needle2,strlen (needle2)))) -return haystack; - -if (line_end ==NULL ) -return NULL ; - - haystack = line_end +1 ; - } - -return NULL ; -} - -static char *next_line (char *line,char *bufend) +static const char *next_line (const char *line,const char *bufend) {char *ret;
@@ -97,45 +77,6 @@ static char *next_line(char *line, char *bufend)return NULL ; } -static char *find_line_after_last (char *begin, -const char *needle1, -const char *needle2, -char *end) -{ -char *one, *two; -char *current_pos = begin; -char *needle1_newline =malloc (strlen (needle1) +2 ); -char *needle2_newline =malloc (strlen (needle2) +2 ); - - needle1_newline[0 ] = needle2_newline[0 ] ='\n' ; -strcpy (needle1_newline +1 , needle1); -strcpy (needle2_newline +1 , needle2); - -while (true) { - one =memmem (current_pos, end - current_pos, needle1_newline,strlen (needle1_newline)); - two =memmem (current_pos, end - current_pos, needle2_newline,strlen (needle2_newline)); -if (one ==NULL && two ==NULL ) -break ; - -if (one !=NULL && current_pos < one) - current_pos = one; -if (two !=NULL && current_pos < two) - current_pos = two; - - one =next_line (current_pos, end); -if (one !=NULL ) - current_pos = one; - } -free (needle1_newline); -free (needle2_newline); - - one =memchr (current_pos,'\n' , end - current_pos); -if (one !=NULL ) -return ++one; - -return current_pos; -} -static size_t count_lines (const char *buf,const char *bufend) {size_t ret =0 ;
@@ -166,7 +107,7 @@ static const struct { { "CRASH", "crash" }, { "TIMEOUT", "timeout" }, }; -static void parse_result_string (char *resultstring,size_t len,const char **result,double *time) +static void parse_result_string (const char *resultstring,size_t len,const char **result,double *time) {size_t i;size_t wordlen =0 ;
@@ -206,19 +147,20 @@ static void parse_result_string(char *resultstring, size_t len, const char **res } } -static void parse_subtest_result (char *subtest,const char **result,double *time,char *buf,char *bufend) +static void parse_subtest_result (const char *subtest, +const char **result, +double *time, +const char *line, +const char *bufend) { -char *line; -char *line_end; -char *resultstring; -size_t linelen; +const char *resultstring;size_t subtestlen =strlen (subtest); +const char *line_end; +size_t linelen; - *result =NULL ; + *result ="incomplete" ; *time =0.0 ; -if (!buf)return ; -/* * The result line structure is: *
@@ -235,18 +177,17 @@ static void parse_subtest_result(char *subtest, const char **result, double *tim* Subtest subtestname: PASS (0.003s) */ - line =find_line_starting_with (buf,SUBTEST_RESULT , bufend); -if (!line) { - *result ="incomplete" ; +if (!line)return ; - } line_end =memchr (line,'\n' , bufend - line); linelen = line_end !=NULL ? line_end - line : bufend - line;if (strlen (SUBTEST_RESULT ) + subtestlen +strlen (": " ) > linelen || -strncmp (line +strlen (SUBTEST_RESULT ), subtest, subtestlen)) -return parse_subtest_result (subtest, result, time, line + linelen, bufend); +strncmp (line +strlen (SUBTEST_RESULT ), subtest, subtestlen)) { +/* This is not the correct result line */ +return ; + } resultstring = line +strlen (SUBTEST_RESULT ) + subtestlen +strlen (": " );parse_result_string (resultstring, linelen - (resultstring - line), result, time);
@@ -308,6 +249,59 @@ static void set_runtime(struct json_object *obj, double time) json_object_new_double(time)); } +struct match_item +{ +const char *where; +const char *what; +}; + +struct matches +{ +struct match_item *items; +size_t size; +}; + +static void match_add (struct matches *matches,const char *where,const char *what) +{ +struct match_item newitem = { where, what }; + + matches->size++; + matches->items =realloc (matches->items, matches->size *sizeof (*matches->items)); + matches->items[matches->size -1 ] = newitem; +} + +static struct matches find_matches (const char *buf,const char *bufend, +const char **needles) +{ +struct matches ret = {}; + +while (buf < bufend) { +const char **needle; + +for (needle = needles; *needle; needle++) { +if (bufend - buf <strlen (*needle)) +continue ; + +if (!memcmp (buf, *needle,strlen (*needle))) { +match_add (&ret, buf, *needle); + goto end_find; + } + } + + end_find: + buf =next_line (buf, bufend); +if (!buf) +break ; + } + +return ret; +} + +static void free_matches (struct matches *matches) +{ +free (matches->items); +} +static bool fill_from_output (int fd,const char *binary,const char *key,struct subtests *subtests,struct json_object *tests)
@@ -318,7 +312,13 @@ static bool fill_from_output(int fd, const char *binary, const char *key,char *igt_version =NULL ;size_t igt_version_len =0 ;struct json_object *current_test =NULL ; -size_t i; +const char *needles[] = { +STARTING_SUBTEST , +SUBTEST_RESULT , +NULL + }; +struct matches matches = {}; +size_t i, k;if (fstat (fd, &statbuf))return false;
@@ -363,10 +363,13 @@ static bool fill_from_output(int fd, const char *binary, const char *key,return true; } + matches =find_matches (buf, bufend, needles); +for (i =0 ; i < subtests->size; i++) {char *this_sub_begin, *this_sub_result; +int begin_idx =-1 , result_idx =-1 ;const char *resulttext; -char *beg, *end, *startline; +const char *beg, *end;double time;int begin_len;int result_len;
@@ -382,66 +385,68 @@ static bool fill_from_output(int fd, const char *binary, const char *key,return false; } - beg =find_line_starting_with (buf, this_sub_begin, bufend); - end =find_line_starting_with (buf, this_sub_result, bufend); - startline = beg; +for (k =0 ; k < matches.size; ++k) { +if (matches.items[k].what ==STARTING_SUBTEST && + !memcmp (matches.items[k].where, + this_sub_begin, +min (begin_len, bufend - matches.items[k].where))) { + beg = matches.items[k].where; + begin_idx = k; + } +if (matches.items[k].what ==SUBTEST_RESULT && + !memcmp (matches.items[k].where, + this_sub_result, +min (result_len, bufend - matches.items[k].where))) { + end = matches.items[k].where; + result_idx = k; + } + }free (this_sub_begin);free (this_sub_result); -if (beg ==NULL && end ==NULL ) { +if (begin_idx <0 && result_idx <0 ) {/* No output at all */ beg = bufend; end = bufend; - } - -if (beg ==NULL ) { + }else if (begin_idx <0 ) {/* -* Subtest didn't start, probably skipped from -* fixture already. Start from the result -* line, it gets adjusted below. +* Subtest didn't start, but we have the +* result. Probably because an igt_fixture +* made it fail/skip. +* +* Start the output right after the previous +* subtest result/start. */ - beg = end; +if (result_idx >0 ) + beg =next_line (matches.items[result_idx -1 ].where, bufend); +else + beg = buf; + }else { +/* Stretch output beginning backwards */ +if (begin_idx ==0 ) + beg = buf; +else + beg =next_line (matches.items[begin_idx -1 ].where, bufend); } -/* Include the output after the previous subtest output */ - beg =find_line_after_last (buf, -STARTING_SUBTEST , -SUBTEST_RESULT , - beg); - -if (end ==NULL ) { -/* Incomplete result. Find the next starting subtest or result. */ - end =next_line (startline, bufend); -if (end !=NULL ) { - end =find_line_starting_with_either (end, -STARTING_SUBTEST , -SUBTEST_RESULT , - bufend); - } -if (end ==NULL ) { - end = bufend; - } - }else { +if (result_idx <0 && begin_idx <0 ) { +/* No output at all: Already handled above */ + }else if (result_idx <0 ) {/* -* Now pointing to the line where this sub's -* result is. We need to include that of -* course. +* Incomplete result. Include output up to the +* next starting subtest or result. */ -char *nexttest =next_line (end, bufend); - -/* Stretch onwards until the next subtest begins or ends */ -if (nexttest !=NULL ) { - nexttest =find_line_starting_with_either (nexttest, -STARTING_SUBTEST , -SUBTEST_RESULT , - bufend); - } -if (nexttest !=NULL ) { - end = nexttest; - }else { +if (begin_idx < matches.size -1 ) + end = matches.items[begin_idx +1 ].where; +else + end = bufend; + }else { +/* Stretch output to the next starting subtest or result. */ +if (result_idx < matches.size -1 ) + end = matches.items[result_idx +1 ].where; +else end = bufend; - } } json_object_object_add(current_test, key,
@@ -454,12 +459,16 @@ static bool fill_from_output(int fd, const char *binary, const char *key, }if (!json_object_object_get_ex (current_test,"result" ,NULL )) { -parse_subtest_result (subtests->names[i], &resulttext, &time, beg, end); +parse_subtest_result (subtests->names[i], + &resulttext, &time, + result_idx <0 ?NULL : matches.items[result_idx].where, + end);set_result (current_test, resulttext);set_runtime (current_test, time); } } +free_matches (&matches);return true; }
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
PATCH 4/6 runner/json_tests: Adapt to better output parsing
In reply to Petri LatvalaSince we actually include the output before the subtest begins now, add it to the reference.jsons where applicable.
--- .../json_tests_data/aborted-after-a-test/reference.json | 2 +- runner/json_tests_data/dmesg-escapes/reference.json | 2 +- runner/json_tests_data/dmesg-results/reference.json | 8 ++++---- .../dmesg-warn-level-one-piglit-style/reference.json | 4 ++-- .../dmesg-warn-level-piglit-style/reference.json | 4 ++-- runner/json_tests_data/dmesg-warn-level/reference.json | 4 ++-- runner/json_tests_data/normal-run/reference.json | 8 ++++---- .../notrun-results-multiple-mode/reference.json | 2 +- runner/json_tests_data/notrun-results/reference.json | 2 +- runner/json_tests_data/piglit-style-dmesg/reference.json | 8 ++++---- .../warnings-with-dmesg-warns/reference.json | 8 ++++---- runner/json_tests_data/warnings/reference.json | 8 ++++---- 12 files changed, 30 insertions(+), 30 deletions(-)
runner/json_tests_data/aborted-after-a-test/reference.json +1 -1
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"pass", "time":{
runner/json_tests_data/dmesg-escapes/reference.json +1 -1
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"pass", "time":{
runner/json_tests_data/dmesg-results/reference.json +4 -4
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"dmesg-warn", "time":{
@@ -23,7 +23,7 @@ "dmesg-warnings":"<3> [3216186.101159] Warning from kernel\n" }, "igt@successtest@second-subtest":{ - "out":"Starting subtest: second-subtest\nSubtest second-subtest: FAIL (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: FAIL (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"dmesg-fail", "time":{
@@ -49,7 +49,7 @@ "dmesg-warnings":"<3> [3216186.101159] Warning from kernel\n" }, "igt@skippers@skip-one":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
@@ -61,7 +61,7 @@ "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<3> [3216186.101159] Warning from kernel\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n" }, "igt@skippers@skip-two":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
runner/json_tests_data/dmesg-warn-level-one-piglit-style/reference.json +2 -2
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"dmesg-warn", "time":{
@@ -70,4 +70,4 @@ } } } -} +}
runner/json_tests_data/dmesg-warn-level-piglit-style/reference.json +2 -2
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"dmesg-warn", "time":{
@@ -70,4 +70,4 @@ } } } -} +}
runner/json_tests_data/dmesg-warn-level/reference.json +2 -2
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"dmesg-warn", "time":{
@@ -70,4 +70,4 @@ } } } -} +}
runner/json_tests_data/normal-run/reference.json +4 -4
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"pass", "time":{
@@ -22,7 +22,7 @@ "dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n<6> [3216186.101115] [IGT] successtest: starting subtest first-subtest\n<6> [3216186.101160] [IGT] successtest: exiting, ret=0\n<6> [3216186.101299] Console: switching to colour frame buffer device 240x75\n" }, "igt@successtest@second-subtest":{ - "out":"Starting subtest: second-subtest\nSubtest second-subtest: FAIL (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: FAIL (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"fail", "time":{
@@ -46,7 +46,7 @@ "dmesg":"<6> [3216186.123400] Console: switching to colour dummy device 80x25\n<6> [3216186.123414] [IGT] no-subtests: executing\n<6> [3216186.125204] [IGT] no-subtests: exiting, ret=0\n<6> [3216186.125374] Console: switching to colour frame buffer device 240x75\n" }, "igt@skippers@skip-one":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
@@ -58,7 +58,7 @@ "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n" }, "igt@skippers@skip-two":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
runner/json_tests_data/notrun-results-multiple-mode/reference.json +1 -1
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"pass", "time":{
runner/json_tests_data/notrun-results/reference.json +1 -1
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"pass", "time":{
runner/json_tests_data/piglit-style-dmesg/reference.json +4 -4
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"warn", "time":{
@@ -22,7 +22,7 @@ "dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n<6> [3216186.101115] [IGT] successtest: starting subtest first-subtest\n<3> [3216186.101159] Warning from kernel\n<6> [3216186.101160] [IGT] successtest: exiting, ret=0\n<6> [3216186.101299] Console: switching to colour frame buffer device 240x75\n" }, "igt@successtest@second-subtest":{ - "out":"Starting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"pass", "time":{
@@ -46,7 +46,7 @@ "dmesg":"<6> [3216186.123400] Console: switching to colour dummy device 80x25\n<6> [3216186.123414] [IGT] no-subtests: executing\n<6> [3216186.125204] [IGT] no-subtests: exiting, ret=0\n<6> [3216186.125374] Console: switching to colour frame buffer device 240x75\n" }, "igt@skippers@skip-one":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
@@ -58,7 +58,7 @@ "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n" }, "igt@skippers@skip-two":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
runner/json_tests_data/warnings-with-dmesg-warns/reference.json +4 -4
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"dmesg-warn", "time":{
@@ -23,7 +23,7 @@ "dmesg-warnings":"<3> [3216186.101159] Warning from kernel\n" }, "igt@successtest@second-subtest":{ - "out":"Starting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"pass", "time":{
@@ -47,7 +47,7 @@ "dmesg":"<6> [3216186.123400] Console: switching to colour dummy device 80x25\n<6> [3216186.123414] [IGT] no-subtests: executing\n<6> [3216186.125204] [IGT] no-subtests: exiting, ret=0\n<6> [3216186.125374] Console: switching to colour frame buffer device 240x75\n" }, "igt@skippers@skip-one":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
@@ -59,7 +59,7 @@ "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n" }, "igt@skippers@skip-two":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
runner/json_tests_data/warnings/reference.json +4 -4
@@ -10,7 +10,7 @@ }, "tests":{ "igt@successtest@first-subtest":{ - "out":"Starting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: first-subtest\nSubtest first-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"warn", "time":{
@@ -22,7 +22,7 @@ "dmesg":"<6> [3216186.095083] Console: switching to colour dummy device 80x25\n<6> [3216186.095097] [IGT] successtest: executing\n<6> [3216186.101115] [IGT] successtest: starting subtest first-subtest\n<6> [3216186.101160] [IGT] successtest: exiting, ret=0\n<6> [3216186.101299] Console: switching to colour frame buffer device 240x75\n" }, "igt@successtest@second-subtest":{ - "out":"Starting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nStarting subtest: second-subtest\nSubtest second-subtest: SUCCESS (0.000s)\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"pass", "time":{
@@ -46,7 +46,7 @@ "dmesg":"<6> [3216186.123400] Console: switching to colour dummy device 80x25\n<6> [3216186.123414] [IGT] no-subtests: executing\n<6> [3216186.125204] [IGT] no-subtests: exiting, ret=0\n<6> [3216186.125374] Console: switching to colour frame buffer device 240x75\n" }, "igt@skippers@skip-one":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-one: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
@@ -58,7 +58,7 @@ "dmesg":"<6> [3216186.135188] Console: switching to colour dummy device 80x25\n<6> [3216186.135212] [IGT] skippers: executing\n<6> [3216186.137075] [IGT] skippers: exiting, ret=77\n<6> [3216186.137206] Console: switching to colour frame buffer device 240x75\n" }, "igt@skippers@skip-two":{ - "out":"Test requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", + "out":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)\nTest requirement not met in function __real_main3, file ..\/runner\/testdata\/skippers.c:6:\nTest requirement: false\nSkipping from fixture\nLast errno: 2, No such file or directory\nSubtest skip-two: SKIP\n", "igt-version":"IGT-Version: 1.23-g0c763bfd (x86_64) (Linux: 4.18.0-1-amd64 x86_64)", "result":"skip", "time":{
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
PATCH 5/6 runner: Parse dynamic subtest outputs and results
In reply to Petri LatvalaIf binary 'bin' has a subtest 'sub', which has dynamic subtests 'foo' and 'bar', results.json will now have "subtests" by the names igt@bin@sub@foo and igt@bin@sub@bar, with data as expected of normal subtests.
--- runner/job_list.c | 11 ++ runner/job_list.h | 3 + runner/output_strings.h | 29 +++- runner/resultgen.c | 340 +++++++++++++++++++++++++++++++++------- 4 files changed, 321 insertions(+), 62 deletions(-)
runner/job_list.c +11 -0
@@ -357,6 +357,17 @@ void generate_piglit_name(const char *binary, const char *subtest,free (lc_subtest); } +void generate_piglit_name_for_dynamic (const char *base_piglit_name, +const char *dynamic_subtest, +char *namebuf,size_t namebuf_size) +{ +char *lc_dynamic =lowercase (dynamic_subtest); + +snprintf (namebuf, namebuf_size,"%s@%s" , base_piglit_name, lc_dynamic); + +free (lc_dynamic); +} +void init_job_list (struct job_list *job_list) {memset (job_list,0 ,sizeof (*job_list));
runner/job_list.h +3 -0
@@ -29,6 +29,9 @@ struct job_listvoid generate_piglit_name (const char *binary,const char *subtest,char *namebuf,size_t namebuf_size); +void generate_piglit_name_for_dynamic (const char *base_piglit_name, +const char *dynamic_subtest, +char *namebuf,size_t namebuf_size);void init_job_list (struct job_list *job_list);void free_job_list (struct job_list *job_list);
runner/output_strings.h +28 -1
@@ -15,10 +15,28 @@ static const char STARTING_SUBTEST[] = "Starting subtest: ";* * Examples: * Subtest subtestname: SKIP -* Subtest subtestname: PASS (0.003s) +* Subtest subtestname: SUCCESS (0.003s) */ static const char SUBTEST_RESULT [] ="Subtest " ; +/* +* Output when a dynamic subtest has begun. Is followed by the subtest name. +* +* Example: +* Starting dynamic subtest: subtestname +*/ +static const char STARTING_DYNAMIC_SUBTEST [] ="Starting dynamic subtest: " ; + +/* +* Output when a dynamic subtest has ended. Is followed by the subtest name +* and optionally its runtime. +* +* Examples: +* Dynamic subtest subtestname: SKIP +* Dynamic subtest subtestname: SUCCESS (0.003s) +*/ +static const char DYNAMIC_SUBTEST_RESULT [] ="Dynamic subtest " ; +/* * Output in dmesg when a subtest has begin. Is followed by the subtest name. *
@@ -27,6 +45,15 @@ static const char SUBTEST_RESULT[] = "Subtest ";*/ static const char STARTING_SUBTEST_DMESG [] =": starting subtest " ; +/* +* Output in dmesg when a dynamic subtest has begin. Is followed by +* the subtest name. +* +* Example: +* [IGT] test-binary-name: starting dynamic subtest subtestname +*/ +static const char STARTING_DYNAMIC_SUBTEST_DMESG [] =": starting dynamic subtest " ; +/* * Output when a test process is executed. *
runner/resultgen.c +279 -61
561 lines not on this page · raw message
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
PATCH 6/6 runner/json_tests: Test dynamic subtests
In reply to Petri Latvala--- .../dynamic-subtests/0/dmesg.txt | 7 + .../dynamic-subtests/0/err.txt | 36 ++++ .../dynamic-subtests/0/journal.txt | 2 + .../dynamic-subtests/0/out.txt | 19 +++ .../dynamic-subtests/1/dmesg.txt | 5 + .../dynamic-subtests/1/err.txt | 2 + .../dynamic-subtests/1/journal.txt | 2 + .../dynamic-subtests/1/out.txt | 5 + .../dynamic-subtests/2/dmesg.txt | 10 ++ .../dynamic-subtests/2/err.txt | 6 + .../dynamic-subtests/2/journal.txt | 4 + .../dynamic-subtests/2/out.txt | 8 + .../dynamic-subtests/README.txt | 2 + .../dynamic-subtests/endtime.txt | 1 + .../dynamic-subtests/joblist.txt | 3 + .../dynamic-subtests/metadata.txt | 12 ++ .../dynamic-subtests/reference.json | 156 ++++++++++++++++++ .../dynamic-subtests/starttime.txt | 1 + .../dynamic-subtests/uname.txt | 1 + runner/runner_json_tests.c | 3 +- 20 files changed, 284 insertions(+), 1 deletion(-) create mode 100644 runner/json_tests_data/dynamic-subtests/0/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/0/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/1/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/dmesg.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/err.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/journal.txt create mode 100644 runner/json_tests_data/dynamic-subtests/2/out.txt create mode 100644 runner/json_tests_data/dynamic-subtests/README.txt create mode 100644 runner/json_tests_data/dynamic-subtests/endtime.txt create mode 100644 runner/json_tests_data/dynamic-subtests/joblist.txt create mode 100644 runner/json_tests_data/dynamic-subtests/metadata.txt create mode 100644 runner/json_tests_data/dynamic-subtests/reference.json create mode 100644 runner/json_tests_data/dynamic-subtests/starttime.txt create mode 100644 runner/json_tests_data/dynamic-subtests/uname.txt
runner/json_tests_data/dynamic-subtests/0/dmesg.txt new file +7 -0
@@ -0,0 +1,7 @@ +6,1157,23426155175691,-;Console: switching to colour dummy device 80x25 +14,1158,23426155175708,-;[IGT] dynamic: executing +14,1159,23426155184875,-;[IGT] dynamic: starting subtest debug-log-checking +14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-is-dynamic-1 +14,1161,23426155240164,-;[IGT] dynamic: starting dynamic subtest this-is-dynamic-2 +14,1162,23426155293846,-;[IGT] dynamic: exiting, ret=98 +6,1163,23426155294003,-;Console: switching to colour frame buffer device 240x75
runner/json_tests_data/dynamic-subtests/0/err.txt new file +36 -0
@@ -0,0 +1,36 @@ +Starting subtest: debug-log-checking +Starting dynamic subtest: this-is-dynamic-1 +(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ../runner/testdata/dynamic.c:8: +(dynamic:20904) CRITICAL: Failed assertion: false +Dynamic subtest this-is-dynamic-1 failed. +**** DEBUG **** +(dynamic:20904) DEBUG: This print is from 1 +(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ../runner/testdata/dynamic.c:8: +(dynamic:20904) CRITICAL: Failed assertion: false +(dynamic:20904) igt_core-INFO: Stack trace: +(dynamic:20904) igt_core-INFO: #0 ../lib/igt_core.c:1607 __igt_fail_assert() +(dynamic:20904) igt_core-INFO: #1 ../runner/testdata/dynamic.c:11 __real_main3() +(dynamic:20904) igt_core-INFO: #2 ../runner/testdata/dynamic.c:3 main() +(dynamic:20904) igt_core-INFO: #3 ../csu/libc-start.c:342 __libc_start_main() +(dynamic:20904) igt_core-INFO: #4 [_start+0x2a] +**** END **** +Dynamic subtest this-is-dynamic-1: FAIL (0.055s) +Starting dynamic subtest: this-is-dynamic-2 +(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ../runner/testdata/dynamic.c:13: +(dynamic:20904) CRITICAL: Failed assertion: false +Dynamic subtest this-is-dynamic-2 failed. +**** DEBUG **** +(dynamic:20904) DEBUG: This print is from 2 +(dynamic:20904) CRITICAL: Test assertion failure function __real_main3, file ../runner/testdata/dynamic.c:13: +(dynamic:20904) CRITICAL: Failed assertion: false +(dynamic:20904) igt_core-INFO: Stack trace: +(dynamic:20904) igt_core-INFO: #0 ../lib/igt_core.c:1607 __igt_fail_assert() +(dynamic:20904) igt_core-INFO: #1 ../runner/testdata/dynamic.c:5 __real_main3() +(dynamic:20904) igt_core-INFO: #2 ../runner/testdata/dynamic.c:3 main() +(dynamic:20904) igt_core-INFO: #3 ../csu/libc-start.c:342 __libc_start_main() +(dynamic:20904) igt_core-INFO: #4 [_start+0x2a] +**** END **** +Dynamic subtest this-is-dynamic-2: FAIL (0.054s) +Subtest debug-log-checking failed. +No log. +Subtest debug-log-checking: FAIL (0.109s)
runner/json_tests_data/dynamic-subtests/0/journal.txt new file +2 -0
@@ -0,0 +1,2 @@ +debug-log-checking +exit:98 (0.130s)
runner/json_tests_data/dynamic-subtests/0/out.txt new file +19 -0
@@ -0,0 +1,19 @@ +IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64) +Starting subtest: debug-log-checking +Starting dynamic subtest: this-is-dynamic-1 +Stack trace: + #0 ../lib/igt_core.c:1607 __igt_fail_assert() + #1 ../runner/testdata/dynamic.c:11 __real_main3() + #2 ../runner/testdata/dynamic.c:3 main() + #3 ../csu/libc-start.c:342 __libc_start_main() + #4 [_start+0x2a] +Dynamic subtest this-is-dynamic-1: FAIL (0.055s) +Starting dynamic subtest: this-is-dynamic-2 +Stack trace: + #0 ../lib/igt_core.c:1607 __igt_fail_assert() + #1 ../runner/testdata/dynamic.c:5 __real_main3() + #2 ../runner/testdata/dynamic.c:3 main() + #3 ../csu/libc-start.c:342 __libc_start_main() + #4 [_start+0x2a] +Dynamic subtest this-is-dynamic-2: FAIL (0.054s) +Subtest debug-log-checking: FAIL (0.109s)
runner/json_tests_data/dynamic-subtests/1/dmesg.txt new file +5 -0
@@ -0,0 +1,5 @@ +6,1164,23426155304955,-;Console: switching to colour dummy device 80x25 +14,1165,23426155304968,-;[IGT] dynamic: executing +14,1166,23426155308644,-;[IGT] dynamic: starting subtest empty-container +14,1167,23426155308671,-;[IGT] dynamic: exiting, ret=77 +6,1168,23426155308822,-;Console: switching to colour frame buffer device 240x75
runner/json_tests_data/dynamic-subtests/1/err.txt new file +2 -0
@@ -0,0 +1,2 @@ +Starting subtest: empty-container +Subtest empty-container: SKIP (0.000s)
runner/json_tests_data/dynamic-subtests/1/journal.txt new file +2 -0
@@ -0,0 +1,2 @@ +empty-container +exit:77 (0.014s)
runner/json_tests_data/dynamic-subtests/1/out.txt new file +5 -0
@@ -0,0 +1,5 @@ +IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64) +Starting subtest: empty-container +This should skip +No dynamic tests executed. +Subtest empty-container: SKIP (0.000s)
runner/json_tests_data/dynamic-subtests/2/dmesg.txt new file +10 -0
@@ -0,0 +1,10 @@ +6,1157,23426155175691,-;Console: switching to colour dummy device 80x25 +14,1158,23426155175708,-;[IGT] dynamic: executing +14,1159,23426155184875,-;[IGT] dynamic: starting subtest normal +6,1160,23426155184895,-;Dmesg output for normal +14,1159,23426155184875,-;[IGT] dynamic: starting subtest incomplete +14,1160,23426155184895,-;[IGT] dynamic: starting dynamic subtest this-is-incomplete +6,1160,23426155184895,-;Dmesg output for incomplete +14,1161,23426155240164,-;[IGT] dynamic: starting subtest resume +14,1162,23426155293846,-;[IGT] dynamic: exiting, ret=0 +6,1163,23426155294003,-;Console: switching to colour frame buffer device 240x75
runner/json_tests_data/dynamic-subtests/2/err.txt new file +6 -0
@@ -0,0 +1,6 @@ +Starting subtest: normal +Subtest normal: SUCCESS (0.100s) +Starting subtest: incomplete +Starting dynamic subtest: this-is-incomplete +Starting subtest: resume +Subtest resume: SUCCESS (0.109s)
runner/json_tests_data/dynamic-subtests/2/journal.txt new file +4 -0
@@ -0,0 +1,4 @@ +normal +incomplete +resume +exit:0 (0.130s)
runner/json_tests_data/dynamic-subtests/2/out.txt new file +8 -0
@@ -0,0 +1,8 @@ +IGT-Version: 1.23-g9e957acd (x86_64) (Linux: 4.18.0-1-amd64 x86_64) +Starting subtest: normal +Subtest normal: SUCCESS (0.100s) +Starting subtest: incomplete +Starting dynamic subtest: this-is-incomplete +This is some output +Starting subtest: resume +Subtest resume: SUCCESS (0.109s)
runner/json_tests_data/dynamic-subtests/README.txt new file +2 -0
@@ -0,0 +1,2 @@ +A test with dynamic subtests should generate separate subresults for +the dynamic tests.
runner/json_tests_data/dynamic-subtests/endtime.txt new file +1 -0
@@ -0,0 +1 @@ +1560163492.410489
runner/json_tests_data/dynamic-subtests/joblist.txt new file +3 -0
@@ -0,0 +1,3 @@ +dynamic debug-log-checking +dynamic empty-container +dynamic normal,incomplete,resume
runner/json_tests_data/dynamic-subtests/metadata.txt new file +12 -0
@@ -0,0 +1,12 @@ +abort_mask : 0 +name : dynamic-subtests +dry_run : 0 +sync : 0 +log_level : 0 +overwrite : 0 +multiple_mode : 1 +inactivity_timeout : 0 +use_watchdog : 0 +piglit_style_dmesg : 0 +test_root : /path/does/not/exist +results_path : /path/does/not/exist
runner/json_tests_data/dynamic-subtests/reference.json new file +156 -0
156 lines not on this page · raw message
runner/json_tests_data/dynamic-subtests/starttime.txt new file +1 -0
@@ -0,0 +1 @@ +1560163492.266377
runner/json_tests_data/dynamic-subtests/uname.txt new file +1 -0
@@ -0,0 +1 @@ +Linux hostname 4.18.0-1-amd64 #1 SMP Debian 4.18.6-1 (2018-09-06) x86_64
runner/runner_json_tests.c +2 -1
@@ -161,7 +161,8 @@ static const char *dirnames[] = {"notrun-results-multiple-mode" ,"dmesg-warn-level" ,"dmesg-warn-level-piglit-style" , - "dmesg-warn-level-one-piglit-style" +"dmesg-warn-level-one-piglit-style" , +"dynamic-subtests" , }; igt_main
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
Patchwork
CI[igt-dev] ✓ Fi.CI.BAT: success for Dynamic subtests
In reply to Petri Latvala== Series Details ==
== Series Details ==
Series: Dynamic subtests
URL : https://patchwork.freedesktop.org/series/62384/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_6307 -> IGTPW_3176
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://patchwork.freedesktop.org/api/1.0/series/62384/revisions/1/mbox/
Known issues
------------
Here are the changes found in IGTPW_3176 that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@gem_ctx_create@basic-files:
- fi-icl-u3: [PASS][1] -> [INCOMPLETE][2] ([fdo#107713] / [fdo#109100])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-icl-u3/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-icl-u3/igt@[email protected]
* igt@gem_exec_fence@basic-busy-default:
- fi-icl-u2: [PASS][3] -> [INCOMPLETE][4] ([fdo#107713])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-icl-u2/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-icl-u2/igt@[email protected]
* igt@i915_selftest@live_contexts:
- fi-bdw-gvtdvm: [PASS][5] -> [DMESG-FAIL][6] ([fdo#110235])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-bdw-gvtdvm/igt@i915_selftest@live_contexts.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-bdw-gvtdvm/igt@i915_selftest@live_contexts.html
#### Possible fixes ####
* igt@gem_exec_suspend@basic-s4-devices:
- fi-blb-e6850: [INCOMPLETE][7] ([fdo#107718]) -> [PASS][8]
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-blb-e6850/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-blb-e6850/igt@[email protected]
* igt@i915_selftest@live_contexts:
- fi-skl-gvtdvm: [DMESG-FAIL][9] ([fdo#110235]) -> [PASS][10]
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-skl-gvtdvm/igt@i915_selftest@live_contexts.html
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-skl-gvtdvm/igt@i915_selftest@live_contexts.html
* igt@i915_selftest@live_evict:
- fi-kbl-8809g: [INCOMPLETE][11] ([fdo#110938]) -> [PASS][12]
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-kbl-8809g/igt@i915_selftest@live_evict.html
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-kbl-8809g/igt@i915_selftest@live_evict.html
* igt@kms_chamelium@hdmi-hpd-fast:
- fi-kbl-7500u: [FAIL][13] ([fdo#109485]) -> [PASS][14]
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/fi-kbl-7500u/igt@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/fi-kbl-7500u/igt@[email protected]
[fdo#107713]: https://bugs.freedesktop.org/show_bug.cgi?id=107713
[fdo#107718]: https://bugs.freedesktop.org/show_bug.cgi?id=107718
[fdo#109100]: https://bugs.freedesktop.org/show_bug.cgi?id=109100
[fdo#109485]: https://bugs.freedesktop.org/show_bug.cgi?id=109485
[fdo#110235]: https://bugs.freedesktop.org/show_bug.cgi?id=110235
[fdo#110938]: https://bugs.freedesktop.org/show_bug.cgi?id=110938
Participating hosts (52 -> 43)
------------------------------
Additional (2): fi-byt-j1900 fi-pnv-d510
Missing (11): fi-kbl-soraka fi-ilk-m540 fi-hsw-4200u fi-skl-guc fi-byt-squawks fi-bsw-cyan fi-byt-clapper fi-icl-y fi-icl-dsi fi-bdw-samus fi-cml-u
Build changes
-------------
* IGT: IGT_5061 -> IGTPW_3176
CI_DRM_6307: bbb61954b61a5411cd6d7c0e98a6b30635d8cf4b @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3176: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/
IGT_5061: c88ced79a7b71aec58f1d9c5c599ac2f431bcf7a @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Quoted 26 lines · Petri Latvala
Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?
Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?
Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?
If so, this is your lucky day, for dynamic subtests are here!
While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:
With normal subtests:
for_each_pipe_static(pipe)igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) { int fd = drm_open_driver(DRIVER_ANY); igt_require(actually_have_this_pipe(fd, pipe)); igt_assert(do_stuff(fd, pipe));}
Quoted 9 lines
With dynamic subtests:
igt_subtest_container("fudge")int fd = drm_open_driver(DRIVER_ANY); for_each_pipe(fd, pipe) { igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) { igt_assert(do_stuff(fd, pipe)); } }
How would the below be converted:
int fd = drm_open_driver(DRIVER_ANY);
for_each_pipe_static(pipe) {
igt_subtest_group {
igt_fixture {
igt_require(actually_have_this_pipe(fd, pipe));
}
igt_subtest_f("fudge-with-pipe-%s", pipe_name(pipe))
fudge(fd, pipe);
igt_subtest_f("kludge-with-pipe-%s", pipe_name(pipe))
kludge(fd, pipe); }
}
?
Quoted 10 lines
Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.
Semantics
=========Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it
igt_subtest_container().
igt_dynamic_subtest_group?
Quoted 4 lines
A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
-igt_assertnot allowed
-igt_skip/igt_requireis allowed though
Can it do fixtures?
Quoted 44 lines
Any failing dynamic subtest will result in the container reporting a failure.
Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using
igt_requirein anigt_dynamic_subtest, just don't enter it for such an occasion. In other words:Do not:
for_each_pipe(fd, pipe)igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) { igt_require(pipe_has_things(pipe)); ...}
Instead do:
for_each_pipe(fd, pipe) { if (!pipe_has_things(pipe)) continue; igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) { ... } }That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.
Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.
Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of
igt_fixtureusage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:
Quoted 7 lines
igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.
igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...
Did not get this part. Is it the first or second option? Or both? If it is the second then CI has to handle dynamic subtests so why we even need the complication to start with?
Regards,
Tvrtko
Quoted 76 lines
Existing tests that are good candidates for conversion to dynamic subtests:
1) Anything using
for_each_pipe_static2) Tests that have one subtest per i915 engine
3) Kernel selftests
Petri Latvala (6):
lib: Introduce dynamic subtests lib/tests: Unit tests for dynamic subtests runner/resultgen: Refactor output parsing runner/json_tests: Adapt to better output parsing
runner: Parse dynamic subtest outputs and results runner/json_tests: Test dynamic subtestslib/igt_core.c | 119 +++- lib/igt_core.h | 89 +++ lib/tests/igt_dynamic_subtests.c | 186 ++++++ lib/tests/meson.build | 1 + runner/job_list.c | 11 + runner/job_list.h | 3 + .../aborted-after-a-test/reference.json | 2 +- .../dmesg-escapes/reference.json | 2 +- .../dmesg-results/reference.json | 8 +- .../reference.json | 4 +- .../reference.json | 4 +- .../dmesg-warn-level/reference.json | 4 +- .../dynamic-subtests/0/dmesg.txt | 7 + .../dynamic-subtests/0/err.txt | 36 ++ .../dynamic-subtests/0/journal.txt | 2 + .../dynamic-subtests/0/out.txt | 19 + .../dynamic-subtests/1/dmesg.txt | 5 + .../dynamic-subtests/1/err.txt | 2 + .../dynamic-subtests/1/journal.txt | 2 + .../dynamic-subtests/1/out.txt | 5 + .../dynamic-subtests/2/dmesg.txt | 10 + .../dynamic-subtests/2/err.txt | 6 + .../dynamic-subtests/2/journal.txt | 4 + .../dynamic-subtests/2/out.txt | 8 + .../dynamic-subtests/README.txt | 2 + .../dynamic-subtests/endtime.txt | 1 + .../dynamic-subtests/joblist.txt | 3 + .../dynamic-subtests/metadata.txt | 12 + .../dynamic-subtests/reference.json | 156 +++++ .../dynamic-subtests/starttime.txt | 1 + .../dynamic-subtests/uname.txt | 1 + .../json_tests_data/normal-run/reference.json | 8 +- .../reference.json | 2 +- .../notrun-results/reference.json | 2 +- .../piglit-style-dmesg/reference.json | 8 +- .../warnings-with-dmesg-warns/reference.json | 8 +- .../json_tests_data/warnings/reference.json | 8 +- runner/output_strings.h | 29 +- runner/resultgen.c | 589 ++++++++++++------ runner/runner_json_tests.c | 3 +- 40 files changed, 1148 insertions(+), 224 deletions(-)create mode 100644
lib/tests/igt_dynamic_subtests.c
create mode 100644runner/json_tests_data/dynamic-subtests/0/dmesg.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/0/err.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/0/journal.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/0/out.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/1/dmesg.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/1/err.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/1/journal.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/1/out.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/2/dmesg.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/2/err.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/2/journal.txt
Quoted 9 lines
create mode 100644
runner/json_tests_data/dynamic-subtests/2/out.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/README.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/endtime.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/joblist.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/metadata.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/reference.jsoncreate mode 100644runner/json_tests_data/dynamic-subtests/starttime.txtcreate mode 100644runner/json_tests_data/dynamic-subtests/uname.txt
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Quoted 53 lines · Tvrtko Ursulin
Quoted 26 lines · Petri Latvala
Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?
Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?
Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?
If so, this is your lucky day, for dynamic subtests are here!
While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:
With normal subtests:
for_each_pipe_static(pipe)igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) { int fd = drm_open_driver(DRIVER_ANY); igt_require(actually_have_this_pipe(fd, pipe)); igt_assert(do_stuff(fd, pipe));}
Quoted 9 lines
With dynamic subtests:
igt_subtest_container("fudge")int fd = drm_open_driver(DRIVER_ANY); for_each_pipe(fd, pipe) { igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) { igt_assert(do_stuff(fd, pipe)); } }How would the below be converted:
int fd =
drm_open_driver(DRIVER_ANY);
for_each_pipe_static(pipe){igt_subtest_group { igt_fixture { igt_require(actually_have_this_pipe(fd, pipe)); } igt_subtest_f("fudge-with-pipe-%s", pipe_name(pipe)) fudge(fd, pipe); igt_subtest_f("kludge-with-pipe-%s", pipe_name(pipe)) kludge(fd, pipe);
Quoted 3 lines
}
}
igt_dynamic_subtest_container("fudge")
for_each_pipe(fd, pipe)
igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
fudge(fd, pipe);igt_dynamic_subtest_container("kludge")
for_each_pipe(fd, pipe)
igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe))
kludge(fd, pipe);Then you get results for "subtests" called igt@binaryname@fudge and igt@binaryname@fudge@pipe-a and igt@binaryname@fudge@pipe-b, along with igt@binaryname@kludge and igt@binaryname@kludge@pipe-a and igt@binaryname@kludge@pipe-b
Quoted 13 lines
?
Quoted 10 lines
Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.
Semantics
=========Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it
igt_subtest_container().
Quoted 8 lines
igt_dynamic_subtest_group?Quoted 4 lines
A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
-igt_assertnot allowed
-igt_skip/igt_requireis allowed thoughCan it do fixtures?
No.
For shared initialization, do
igt_dynamic_subtest_container("hello-init") {
/* right here */
int fd = drm_open_driver(DRIVER_ANY);
init_stuffs(fd);
/* init done */
for (......) {
igt_dynamic_subtest(...)
}Quoted 48 lines
Quoted 48 lines
Any failing dynamic subtest will result in the container reporting a failure.
Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using
igt_requirein anigt_dynamic_subtest, just don't enter it for such an occasion. In other words:Do not:
for_each_pipe(fd, pipe)igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) { igt_require(pipe_has_things(pipe)); ...}
Instead do:
for_each_pipe(fd, pipe) { if (!pipe_has_things(pipe)) continue; igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) { ... } }That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.
Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.
Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of
igt_fixtureusage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:
igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.
Quoted 8 lines
Quoted 4 lines
igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...
Did not get this part. Is it the first or second option? Or both? If it is the second then CI has to handle dynamic subtests so why we even need the complication to start with?
It's both. The container will report a result (based on what the dynamic tests did) and there's also a result for each dynamic subtest.
The complication is needed because with dynamic subtests the execution entry and results reporting are two different sets, reported subtests being a superset of the entries.
CI will handle (knocking on wood here) dynamic results fine in its current state, because dynamic subtest results walk like a duck, smell like a duck, and look like normal subtest results.
Signature · 5 lines
Petri Latvala _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
Quoted 176 lines · Petri Latvala
Quoted 42 lines · Tvrtko Ursulin
Quoted 26 lines · Petri Latvala
Have you ever had a massive list of named things that you'd quite like to test with separate subtests but maintaining a compile-time exhaustive list of those things seems impossible, stupid, or both?
Have you ever wished you could just go over a list of things at runtime and create subtests on the fly?
Have you ever looked at a long list of SKIP-results and wondered if it makes sense to keep plopping those in CI results?
If so, this is your lucky day, for dynamic subtests are here!
While the restriction remains that "execution entry points" must still be statically available no matter what the runtime environment and configuration is, dynamic subtest containers are special subtest-like entry points that can contain dynamic subtests. For example:
With normal subtests:
for_each_pipe_static(pipe)igt_subtest_f("fudge-with-pipe-%s", kmstest_pipe_name(pipe)) { int fd = drm_open_driver(DRIVER_ANY); igt_require(actually_have_this_pipe(fd, pipe)); igt_assert(do_stuff(fd, pipe)); }Quoted 9 lines
With dynamic subtests:
igt_subtest_container("fudge")int fd = drm_open_driver(DRIVER_ANY); for_each_pipe(fd, pipe) { igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) { igt_assert(do_stuff(fd, pipe)); } }How would the below be converted:
int fd =
drm_open_driver(DRIVER_ANY);Quoted 14 lines
for_each_pipe_static(pipe) { igt_subtest_group { igt_fixture { igt_require(actually_have_this_pipe(fd, pipe)); } igt_subtest_f("fudge-with-pipe-%s", pipe_name(pipe)) fudge(fd, pipe); igt_subtest_f("kludge-with-pipe-%s", pipe_name(pipe)) kludge(fd, pipe); } }
igt_dynamic_subtest_container("fudge")
for_each_pipe(fd, pipe)igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) fudge(fd, pipe);
igt_dynamic_subtest_container("kludge")
for_each_pipe(fd, pipe)igt_dynamic_subtest_f("pipe-%s", kmstest_pipe_name(pipe)) kludge(fd, pipe);Then you get results for "subtests" called igt@binaryname@fudge and igt@binaryname@fudge@pipe-a and igt@binaryname@fudge@pipe-b, along with igt@binaryname@kludge and igt@binaryname@kludge@pipe-a and igt@binaryname@kludge@pipe-b
Quoted 13 lines
?
Quoted 10 lines
Running on hardware that only has pipes A-C? pipe-d doesn't even get attempted.
Semantics
=========Dynamic subtests can only appear in a subtest container. The name is up to debate: This series calls it
igt_subtest_container().Quoted 8 lines
igt_dynamic_subtest_group?Quoted 4 lines
A dynamic subtest container can only contain dynamic subtests, it cannot do failures on its own. In other words:
-igt_assertnot allowed
-igt_skip/igt_requireis allowed thoughCan it do fixtures?
No.
For shared initialization, do
igt_dynamic_subtest_container("hello-init") { /* right here */ int fd = drm_open_driver(DRIVER_ANY); init_stuffs(fd); /* init done */ for (......) { igt_dynamic_subtest(...) }Quoted 48 lines
Quoted 48 lines
Any failing dynamic subtest will result in the container reporting a failure.
Not executing any dynamic subtests from a container will result in the container reporting a SKIP automatically. Best practices: If possible, instead of using
igt_requirein anigt_dynamic_subtest, just don't enter it for such an occasion. In other words:Do not:
for_each_pipe(fd, pipe)igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) { igt_require(pipe_has_things(pipe)); ... }Instead do:
for_each_pipe(fd, pipe) { if (!pipe_has_things(pipe)) continue; igt_dynamic_subtest("%s", kmstest_pipe_name(pipe)) { ... } }That way, tests that currently carefully track the number of, say, connected displays of type $x, to properly skip when their amount is 0, will get their SKIP automatically instead.
Dynamic subtests are filterable: Just like --run-subtest, there's --dynamic-subtest that takes glob expressions. This is for debugging only, CI will not use them.
Dynamic subtests are _NOT_ listable. While it might be useful, implementing listing requires another layer of
igt_fixtureusage inside dynamic subtest containers and I'd rather have the code be simple than have listing. The default of running all dynamic subtests should make sense for most cases, and special cases (like debugging) should be able to know already what they want to run.Results in CI: CI will show results for both the container, and the dynamic subtests within it. The naming is:
igt@binary@subcontainer - the container, has as its output the whole shebang that dynamic subtests printed.
Quoted 8 lines
Quoted 4 lines
igt@binary@subcontainer@dynamicname - a dynamic subtest appears as a separate name, WITHOUT grouping or nesting. Any relation to the container will not be linked at this point. Possibly in the future when a usability wizard figures out the best way to browse results...
Did not get this part. Is it the first or second option? Or both? If it is the second then CI has to handle dynamic subtests so why we even need the complication to start with?
It's both. The container will report a result (based on what the dynamic tests did) and there's also a result for each dynamic subtest.
Quoted 7 lines
The complication is needed because with dynamic subtests the execution entry and results reporting are two different sets, reported subtests being a superset of the entries.
CI will handle (knocking on wood here) dynamic results fine in its current state, because dynamic subtest results walk like a duck, smell like a duck, and look like normal subtest results.
I am confused by all this. Could we then achieve the same result by just adding the ability to hide subtests from test enumeration?
igt_subtest_group {
igt_subtest("a");
}igt_dynamic_subtest_group("group") {
igt_subtest("b");
}# ./test --list-subtests a group
# .test --list-all-subtests a group-b
# ./test --r group PASS
# ./test --r group-b PASS
Would the above satisfy CI requirements? Sounds like it would be easier to implement wrt needed code base changes in individual tests and possibly the IGT core code would not be difficult either? Just needs to track in what kind of subgroup it is to decide what to do when listing/running tests.
Regards,
Tvrtko
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Quoted 22 lines · Tvrtko Ursulin
I am confused by all this. Could we then achieve the same result by just adding the ability to hide subtests from test enumeration?
igt_subtest_group { igt_subtest("a"); }igt_dynamic_subtest_group("group") { igt_subtest("b"); }# ./test --list-subtests a group# .test --list-all-subtests a group-b# ./test --r group PASS
Quoted 7 lines
# ./test --r group-b PASSWould the above satisfy CI requirements? Sounds like it would be easier to implement wrt needed code base changes in individual tests and possibly the IGT core code would not be difficult either? Just needs to track in what kind of subgroup it is to decide what to do when listing/running tests.
With sufficient fiddling of names, this is exactly what this series does.
igt_subtest_group {
igt_subtest("a");
}igt_dynamic_subtest_container("group") {
igt_dynamic_subtest("b");
}# ./test --list-subtests a group
# .test --list-all-subtests error: nope
# ./test --r group Dynamic "b": PASS PASS
# ./test --r group --dynamic-subtest b Dynamic "b": PASS PASS
The listing of dynamic subtests is not implemented as that is definitely not trivial.
Signature · 5 lines
Petri Latvala _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
Quoted 54 lines · Petri Latvala
Quoted 22 lines · Tvrtko Ursulin
I am confused by all this. Could we then achieve the same result by just adding the ability to hide subtests from test enumeration?
igt_subtest_group { igt_subtest("a"); }igt_dynamic_subtest_group("group") { igt_subtest("b"); }# ./test --list-subtests a group# .test --list-all-subtests a group-b# ./test --r group PASSQuoted 7 lines
# ./test --r group-b PASSWould the above satisfy CI requirements? Sounds like it would be easier to implement wrt needed code base changes in individual tests and possibly the IGT core code would not be difficult either? Just needs to track in what kind of subgroup it is to decide what to do when listing/running tests.
With sufficient fiddling of names, this is exactly what this series does.
igt_subtest_group { igt_subtest("a"); }igt_dynamic_subtest_container("group") { igt_dynamic_subtest("b"); }# ./test --list-subtests a group# .test --list-all-subtests error: nope# ./test --r group Dynamic "b": PASS PASS
Quoted 8 lines
# ./test --r group --dynamic-subtest b Dynamic "b": PASS PASSThe listing of dynamic subtests is not implemented as that is definitely not trivial.
I think trying to list dynamic subtests without running them defeats the entire point here. At least it would get really tricky to tell "hw/driver acccess necessary to enumerate tests" from "hw/driver access that actually tests something". E.g. with atomic TEST_ONLY you might have to actually construct all the possible configs to figure out which ones are supported. -Daniel
Quoted 8 lines
Signature · 6 lines
-- Petri Latvala _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
Patchwork
CI[igt-dev] ✓ Fi.CI.IGT: success for Dynamic subtests
In reply to Petri Latvala== Series Details ==
== Series Details ==
Series: Dynamic subtests
URL : https://patchwork.freedesktop.org/series/62384/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_6307_full -> IGTPW_3176_full
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://patchwork.freedesktop.org/api/1.0/series/62384/revisions/1/mbox/
Known issues
------------
Here are the changes found in IGTPW_3176_full that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@gem_eio@in-flight-contexts-10ms:
- shard-kbl: [PASS][1] -> [DMESG-WARN][2] ([fdo#110913 ]) +2 similar issues
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl1/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl3/igt@[email protected]
- shard-snb: [PASS][3] -> [DMESG-WARN][4] ([fdo#110789] / [fdo#110913 ]) +1 similar issue
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb5/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb4/igt@[email protected]
* igt@gem_persistent_relocs@forked-interruptible-faulting-reloc-thrashing:
- shard-apl: [PASS][5] -> [DMESG-WARN][6] ([fdo#110913 ]) +4 similar issues
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-apl2/igt@gem_persistent_relocs@forked-interruptible-faulting-reloc-thrashing.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-apl6/igt@gem_persistent_relocs@forked-interruptible-faulting-reloc-thrashing.html
* igt@gem_userptr_blits@map-fixed-invalidate-overlap-busy-gup:
- shard-snb: [PASS][7] -> [DMESG-WARN][8] ([fdo#110913 ])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb5/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb2/igt@[email protected]
* igt@i915_pm_rc6_residency@rc6-accuracy:
- shard-snb: [PASS][9] -> [SKIP][10] ([fdo#109271])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb1/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb4/igt@[email protected]
* igt@i915_selftest@live_evict:
- shard-glk: [PASS][11] -> [INCOMPLETE][12] ([fdo#103359] / [fdo#110938] / [k.org#198133])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-glk8/igt@i915_selftest@live_evict.html
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-glk7/igt@i915_selftest@live_evict.html
* igt@i915_selftest@live_hangcheck:
- shard-snb: [PASS][13] -> [INCOMPLETE][14] ([fdo#105411])
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb2/igt@i915_selftest@live_hangcheck.html
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb2/igt@i915_selftest@live_hangcheck.html
* igt@i915_suspend@debugfs-reader:
- shard-apl: [PASS][15] -> [DMESG-WARN][16] ([fdo#108566]) +1 similar issue
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-apl3/igt@[email protected]
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-apl4/igt@[email protected]
* igt@kms_cursor_legacy@cursor-vs-flip-varying-size:
- shard-hsw: [PASS][17] -> [INCOMPLETE][18] ([fdo#103540])
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-hsw2/igt@[email protected]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-hsw8/igt@[email protected]
* igt@kms_flip@flip-vs-suspend-interruptible:
- shard-glk: [PASS][19] -> [INCOMPLETE][20] ([fdo#103359] / [k.org#198133])
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-glk9/igt@[email protected]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-glk2/igt@[email protected]
* igt@kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render:
- shard-hsw: [PASS][21] -> [SKIP][22] ([fdo#109271]) +19 similar issues
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-hsw7/igt@kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render.html
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-hsw1/igt@kms_frontbuffer_tracking@fbc-2p-primscrn-cur-indfb-draw-render.html
* igt@kms_vblank@pipe-b-query-forked-busy-hang:
- shard-kbl: [PASS][23] -> [INCOMPLETE][24] ([fdo#103665])
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl1/igt@[email protected]
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl4/igt@[email protected]
#### Possible fixes ####
* igt@gem_eio@in-flight-contexts-1us:
- shard-kbl: [DMESG-WARN][25] ([fdo#110913 ]) -> [PASS][26] +4 similar issues
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl4/igt@[email protected]
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl2/igt@[email protected]
* igt@gem_exec_create@forked:
- shard-glk: [FAIL][27] ([fdo#110946]) -> [PASS][28]
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-glk9/igt@[email protected]
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-glk9/igt@[email protected]
* igt@gem_persistent_relocs@forked-faulting-reloc-thrashing:
- shard-snb: [DMESG-WARN][29] ([fdo#110789] / [fdo#110913 ]) -> [PASS][30]
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb6/igt@[email protected]
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb4/igt@[email protected]
* igt@gem_userptr_blits@map-fixed-invalidate-busy-gup:
- shard-apl: [DMESG-WARN][31] ([fdo#110913 ]) -> [PASS][32] +5 similar issues
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-apl8/igt@[email protected]
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-apl5/igt@[email protected]
* igt@gem_userptr_blits@sync-unmap-cycles:
- shard-snb: [DMESG-WARN][33] ([fdo#110913 ]) -> [PASS][34] +2 similar issues
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-snb2/igt@[email protected]
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-snb5/igt@[email protected]
* igt@i915_selftest@live_evict:
- shard-kbl: [INCOMPLETE][35] ([fdo#103665] / [fdo#110938]) -> [PASS][36]
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl3/igt@i915_selftest@live_evict.html
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl2/igt@i915_selftest@live_evict.html
* igt@i915_suspend@sysfs-reader:
- shard-apl: [DMESG-WARN][37] ([fdo#108566]) -> [PASS][38] +5 similar issues
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-apl1/igt@[email protected]
[38]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-apl2/igt@[email protected]
* igt@kms_cursor_legacy@2x-long-flip-vs-cursor-atomic:
- shard-hsw: [SKIP][39] ([fdo#109271]) -> [PASS][40] +20 similar issues
[39]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-hsw1/igt@[email protected]
[40]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-hsw4/igt@[email protected]
* igt@kms_flip@flip-vs-suspend:
- shard-glk: [INCOMPLETE][41] ([fdo#103359] / [k.org#198133]) -> [PASS][42]
[41]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-glk4/igt@[email protected]
[42]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-glk1/igt@[email protected]
* igt@kms_setmode@basic:
- shard-hsw: [FAIL][43] ([fdo#99912]) -> [PASS][44]
[43]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-hsw7/igt@[email protected]
[44]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-hsw6/igt@[email protected]
- shard-kbl: [FAIL][45] ([fdo#99912]) -> [PASS][46]
[45]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6307/shard-kbl7/igt@[email protected]
[46]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/shard-kbl3/igt@[email protected]
[fdo#103359]: https://bugs.freedesktop.org/show_bug.cgi?id=103359
[fdo#103540]: https://bugs.freedesktop.org/show_bug.cgi?id=103540
[fdo#103665]: https://bugs.freedesktop.org/show_bug.cgi?id=103665
[fdo#105411]: https://bugs.freedesktop.org/show_bug.cgi?id=105411
[fdo#108566]: https://bugs.freedesktop.org/show_bug.cgi?id=108566
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[fdo#110789]: https://bugs.freedesktop.org/show_bug.cgi?id=110789
[fdo#110913 ]: https://bugs.freedesktop.org/show_bug.cgi?id=110913
[fdo#110938]: https://bugs.freedesktop.org/show_bug.cgi?id=110938
[fdo#110946]: https://bugs.freedesktop.org/show_bug.cgi?id=110946
[fdo#99912]: https://bugs.freedesktop.org/show_bug.cgi?id=99912
[k.org#198133]: https://bugzilla.kernel.org/show_bug.cgi?id=198133
Participating hosts (10 -> 5)
------------------------------
Missing (5): shard-skl pig-hsw-4770r pig-glk-j5005 shard-iclb pig-skl-6260u
Build changes
-------------
* IGT: IGT_5061 -> IGTPW_3176
* Piglit: piglit_4509 -> None
CI_DRM_6307: bbb61954b61a5411cd6d7c0e98a6b30635d8cf4b @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3176: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/
IGT_5061: c88ced79a7b71aec58f1d9c5c599ac2f431bcf7a @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3176/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev