runner: Add support for aborting on network failure
PATCH v3 1/2 runner: Add support for aborting on network failure
If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.
A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.
v2:
- Use a higher timeout
- Allow hostname configuration from environment
v3:
- Use runner_c_args for holding c args for runner
- Handle runner's meson options in runner/meson.build
- Instead of one ping with 20 second timeout, ping with 1 second timeout for a duration of 20 seconds
--- meson_options.txt | 6 ++ runner/executor.c | 150 +++++++++++++++++++++++++++++++++++++++++++++ runner/meson.build | 14 ++++- runner/settings.c | 4 ++ runner/settings.h | 5 +- 5 files changed, 176 insertions(+), 3 deletions(-)
meson_options.txt +6 -0
@@ -58,6 +58,12 @@ option('build_runner', choices : ['auto', 'true', 'false'], description : 'Build test runner') +option('with_oping', + type : 'combo', + value : 'auto', + choices : ['auto', 'true', 'false'], + description : 'Build igt_runner with liboping') + option('use_rpath', type : 'boolean', value : false,
runner/executor.c +150 -0
180 lines not on this page · raw message
runner/meson.build +13 -1
@@ -1,4 +1,15 @@ jsonc = dependency('json-c', required: _runner_required) +runner_deps = [jsonc, glib] +runner_c_args = [] + +with_oping = get_option('with_oping') +if with_oping != 'false' + oping = dependency('liboping', required: with_oping == 'true') + if oping.found() + runner_deps += oping + runner_c_args += '-DHAVE_OPING=1' + endif +endif runnerlib_sources = [ 'settings.c', 'job_list.c',
@@ -21,7 +32,8 @@ if _build_runner and _build_tests and jsonc.found() runnerlib = static_library('igt_runner', runnerlib_sources, include_directories : inc, - dependencies : [jsonc, glib]) + c_args : runner_c_args, + dependencies : runner_deps) runner = executable('igt_runner', runner_sources, link_with : runnerlib,
runner/settings.c +4 -0
@@ -48,6 +48,7 @@ static struct { } abort_conditions[] = { { ABORT_TAINT, "taint" }, { ABORT_LOCKDEP, "lockdep" }, + { ABORT_PING, "ping" }, { ABORT_ALL, "all" }, { 0, 0 }, };
@@ -136,6 +137,9 @@ static const char *usage_str =" Possible conditions:\n" " lockdep - abort when kernel lockdep has been angered.\n" " taint - abort when kernel becomes fatally tainted.\n" +" ping - abort when a host configured in .igtrc or\n" +" environment variable IGT_PING_HOSTNAME does\n" +" not respond to ping.\n" " all - abort for all of the above.\n" " -s, --sync Sync results to disk after every test\n" " -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
runner/settings.h +3 -2
@@ -15,9 +15,10 @@ enum {#define ABORT_TAINT (1 << 0)#define ABORT_LOCKDEP (1 << 1) -#define ABORT_ALL (ABORT_TAINT | ABORT_LOCKDEP) +#define ABORT_PING (1 << 2) +#define ABORT_ALL (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING) -_Static_assert (ABORT_ALL == (ABORT_TAINT |ABORT_LOCKDEP ),"ABORT_ALL must be all conditions bitwise or'd" ); +_Static_assert (ABORT_ALL == (ABORT_TAINT |ABORT_LOCKDEP |ABORT_PING ),"ABORT_ALL must be all conditions bitwise or'd" );struct regex_list {char **regex_strings;
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
PATCH 2/2 HAX: Check all conditions to abort
--- runner/executor.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
runner/executor.c +1 -1
@@ -354,7 +354,7 @@ static char *need_to_abort(const struct settings* settings)for (it = abort_handlers; it->condition; it++) {char *abort; -if (!(settings->abort_mask & it->condition)) +if (false && !(settings->abort_mask & it->condition))continue ; abort = it->handler ();
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
PATCH 2/2 HAX: Check all conditions to abort
... and remember to HAX the ping setup as well
--- runner/executor.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
runner/executor.c +2 -2
@@ -354,7 +354,7 @@ static char *need_to_abort(const struct settings* settings)for (it = abort_handlers; it->condition; it++) {char *abort; -if (!(settings->abort_mask & it->condition)) +if (false && !(settings->abort_mask & it->condition))continue ; abort = it->handler ();
@@ -1435,7 +1435,7 @@ bool execute(struct execute_state *state,init_watchdogs (settings); -if (settings->abort_mask &ABORT_PING ) +if (true && settings->abort_mask &ABORT_PING )ping_config ();if (!uname (&unamebuf)) {
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
PATCH 2/2 HAX: Check all conditions to abort
... and remember to HAX the ping setup as well
... correctly
--- runner/executor.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
runner/executor.c +2 -2
@@ -354,7 +354,7 @@ static char *need_to_abort(const struct settings* settings)for (it = abort_handlers; it->condition; it++) {char *abort; -if (!(settings->abort_mask & it->condition)) +if (false && !(settings->abort_mask & it->condition))continue ; abort = it->handler ();
@@ -1435,7 +1435,7 @@ bool execute(struct execute_state *state,init_watchdogs (settings); -if (settings->abort_mask &ABORT_PING ) +if (true || settings->abort_mask &ABORT_PING )ping_config ();if (!uname (&unamebuf)) {
Signature · 6 lines
2.19.1 _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
Patchwork
CI[igt-dev] ✓ Fi.CI.BAT: success for series starting with [i-g-t,v3,1/2] runner: Add support for aborting on network failure (rev2)
In reply to Petri Latvala== Series Details ==
== Series Details ==
Series: series starting with [i-g-t,v3,1/2] runner: Add support for aborting on network failure (rev2)
URL : https://patchwork.freedesktop.org/series/60854/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_6099 -> IGTPW_3007
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://patchwork.freedesktop.org/api/1.0/series/60854/revisions/2/mbox/
Known issues
------------
Here are the changes found in IGTPW_3007 that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@prime_vgem@basic-fence-flip:
- fi-ilk-650: [PASS][1] -> [DMESG-WARN][2] ([fdo#106387]) +1 similar issue
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6099/fi-ilk-650/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3007/fi-ilk-650/igt@[email protected]
#### Possible fixes ####
* igt@gem_exec_suspend@basic-s4-devices:
- fi-kbl-7500u: [DMESG-WARN][3] ([fdo#105128] / [fdo#107139]) -> [PASS][4]
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6099/fi-kbl-7500u/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3007/fi-kbl-7500u/igt@[email protected]
[fdo#105128]: https://bugs.freedesktop.org/show_bug.cgi?id=105128
[fdo#106387]: https://bugs.freedesktop.org/show_bug.cgi?id=106387
[fdo#107139]: https://bugs.freedesktop.org/show_bug.cgi?id=107139
Participating hosts (52 -> 43)
------------------------------
Missing (9): fi-kbl-soraka fi-ilk-m540 fi-bxt-dsi fi-hsw-4200u fi-byt-squawks fi-bsw-cyan fi-apl-guc fi-byt-clapper fi-bdw-samus
Build changes
-------------
* IGT: IGT_4997 -> IGTPW_3007
CI_DRM_6099: 3cef0eacdd3d3448dfefb7011f1e66821c5fd41d @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3007: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3007/
IGT_4997: eff5d0db3248734845b78fcc2e2772dd4012e5af @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3007/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Patchwork
CI[igt-dev] ✓ Fi.CI.BAT: success for series starting with [i-g-t,v3,1/2] runner: Add support for aborting on network failure (rev3)
In reply to Petri Latvala== Series Details ==
== Series Details ==
Series: series starting with [i-g-t,v3,1/2] runner: Add support for aborting on network failure (rev3)
URL : https://patchwork.freedesktop.org/series/60854/
State : success
== Summary ==
CI Bug Log - changes from CI_DRM_6100 -> IGTPW_3009
====================================================
Summary
-------
**SUCCESS**
No regressions found.
External URL: https://patchwork.freedesktop.org/api/1.0/series/60854/revisions/3/mbox/
Known issues
------------
Here are the changes found in IGTPW_3009 that come from known issues:
### IGT changes ###
#### Issues hit ####
* igt@i915_selftest@live_hangcheck:
- fi-skl-iommu: [PASS][1] -> [INCOMPLETE][2] ([fdo#108602] / [fdo#108744])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6100/fi-skl-iommu/igt@i915_selftest@live_hangcheck.html
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/fi-skl-iommu/igt@i915_selftest@live_hangcheck.html
* igt@kms_frontbuffer_tracking@basic:
- fi-hsw-peppy: [PASS][3] -> [DMESG-WARN][4] ([fdo#102614])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6100/fi-hsw-peppy/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/fi-hsw-peppy/igt@[email protected]
#### Possible fixes ####
* igt@i915_selftest@live_contexts:
- fi-skl-gvtdvm: [DMESG-FAIL][5] ([fdo#110235]) -> [PASS][6]
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6100/fi-skl-gvtdvm/igt@i915_selftest@live_contexts.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/fi-skl-gvtdvm/igt@i915_selftest@live_contexts.html
[fdo#102614]: https://bugs.freedesktop.org/show_bug.cgi?id=102614
[fdo#108602]: https://bugs.freedesktop.org/show_bug.cgi?id=108602
[fdo#108744]: https://bugs.freedesktop.org/show_bug.cgi?id=108744
[fdo#110235]: https://bugs.freedesktop.org/show_bug.cgi?id=110235
Participating hosts (50 -> 43)
------------------------------
Additional (1): fi-icl-y
Missing (8): fi-kbl-soraka fi-ilk-m540 fi-hsw-4200u fi-byt-squawks fi-bsw-cyan fi-apl-guc fi-pnv-d510 fi-byt-clapper
Build changes
-------------
* IGT: IGT_4997 -> IGTPW_3009
CI_DRM_6100: 6cd6c39683ceb3e48b98871fd25aea47f291f2b0 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3009: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/
IGT_4997: eff5d0db3248734845b78fcc2e2772dd4012e5af @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools
== Logs ==
For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Quoted 98 lines · Petri Latvala
If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.
A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.
v2:
- Use a higher timeout
- Allow hostname configuration from environment
v3:
- Userunner_c_argsfor holding c args for runner
- Handle runner's meson options in runner/meson.build
- Instead of one ping with 20 second timeout, ping with 1 second timeout for a duration of 20 secondsSigned-off-by Petri Latvala <[email protected]>Cc Arkadiusz Hiler <[email protected]>Cc Martin Peres <[email protected]>Cc Tomi Sarvela <[email protected]>Cc Daniel Vetter <[email protected]>--- meson_options.txt | 6 ++ runner/executor.c | 150 +++++++++++++++++++++++++++++++++++++++++++++ runner/meson.build | 14 ++++- runner/settings.c | 4 ++ runner/settings.h | 5 +- 5 files changed, 176 insertions(+), 3 deletions(-)meson_options.txt +6
-0@@ -58,6 +58,12 @@ option('build_runner', choices : ['auto', 'true', 'false'], description : 'Build test runner') +option('with_oping', + type : 'combo', + value : 'auto', + choices : ['auto', 'true', 'false'], + description : 'Build igt_runner with liboping') + option('use_rpath', type : 'boolean', value : false,runner/executor.c +28
-0@@ -1,6 +1,10 @@#include <errno.h> #include <fcntl.h> +#include <glib.h> #include <linux/watchdog.h> +#if HAVE_OPING +#include <oping.h> +#endif #include <signal.h> #include <stdio.h> #include <stdlib.h> @@ -16,6 +20,7 @@#include <time.h> #include <unistd.h> +#include "igt_aux.h" #include "igt_core.h" #include "executor.h" #include "output_strings.h" @@ -108,6 +113,147 @@ static void ping_watchdogs(void) } } +#if HAVE_OPING +static pingobj_t *pingobj =NULL ; + +static bool load_ping_config_from_file (void ) +{ +char *key_file_env =NULL ; +char *key_file_loc =NULL ; +GError *error =NULL ; +GKeyFile *key_file =NULL ; +const char *ping_hostname; +int ret; + +/* Determine igt config path */ + key_file_env =getenv ("IGT_CONFIG_PATH" ); +if (key_file_env) { + key_file_loc =strdup (key_file_env); + }else { +asprintf (&key_file_loc,"%s/.igtrc" ,g_get_home_dir ()); + } + +/* Load igt config file */ + key_file =g_key_file_new (); + ret = g_key_file_load_from_file(key_file, key_file_loc,
runner/executor.c static void ping_watchdogs(void) · line 148
Quoted 10 lines
runner/executor.c +10
-0+ G_KEY_FILE_NONE, &error); +free (key_file_loc); + +if (!ret) { +g_error_free (error); +g_key_file_free (key_file); +return false; + } + +g_clear_error (&error);
put that in
GKeyFile *igt_open_igtrc();
and use it in igt_core, so we have only one implementation
Quoted 36 lines
runner/executor.c +36
-0+ + ping_hostname = +g_key_file_get_string (key_file,"DUT" , +"PingHostName" , &error); + +g_clear_error (&error); +g_key_file_free (key_file); + +if (!ping_hostname) +return false; + +if (ping_host_add (pingobj, ping_hostname)) { +fprintf (stderr, +"abort on ping: Cannot use hostname from config file\n" ); +return false; + } + +return true; +} + +static bool load_ping_config_from_env (void ) +{ +const char *ping_hostname; + + ping_hostname =getenv ("IGT_PING_HOSTNAME" ); +if (!ping_hostname) +return false; + +if (ping_host_add (pingobj, ping_hostname)) { +fprintf (stderr, +"abort on ping: Cannot use hostname from environment\n" ); +return false; + } + +return true; +}
runner/executor.c static void ping_watchdogs(void) · line 194
Quoted 10 lines
runner/executor.c +10
-0+ +static bool can_ping (void ) +{ +/* +* On some hosts, getting network back up after suspend takes +* upwards of 10 seconds. 20 seconds should be enough to see +* if network comes back at all, and hopefully not too long to +* make external monitoring freak out. +*/ + igt_until_timeout(20) {
make 20 a #define
Quoted 18 lines
runner/executor.c +18
-0+pingobj_iter_t *iter; + +ping_send (pingobj); + +for (iter =ping_iterator_get (pingobj); + iter !=NULL ; + iter =ping_iterator_next (iter)) { +double latency; +size_t len =sizeof (latency); + +ping_iterator_get_info (iter, +PING_INFO_LATENCY , + &latency, + &len); +if (latency >=0.0 ) +return true; + } + }
runner/executor.c static void ping_watchdogs(void) · line 222
Quoted 10 lines
runner/executor.c +10
-0+ + return false; +} + +#endif + +static void ping_config (void ) +{ +#if HAVE_OPING +double timeout =1.0 ;
single_attempt_timeout
Quoted 38 lines
runner/executor.c +34
-0+ +if (pingobj) +return ; + + pingobj =ping_construct (); + +/* Try env first, then config file */ +if (!load_ping_config_from_env () && !load_ping_config_from_file ()) { +fprintf (stderr, +"abort on ping: No host to ping configured\n" ); +ping_destroy (pingobj); + pingobj =NULL ; +return ; + } + +ping_setopt (pingobj,PING_OPT_TIMEOUT , &timeout); +#endif +} + +static char *handle_ping (void ) +{ +#if HAVE_OPING +if (pingobj && !can_ping ()) { +char *reason; + +asprintf (&reason, +"Ping host did not respond to ping, network down" ); +return reason; + } +#endif + +return NULL ; +} +static char *handle_lockdep (void ) {const char *header ="Lockdep not active\n\n/proc/lockdep_stats contents:\n" ;
runner/executor.c static const struct { · line 348
Quoted 6 lines
runner/executor.c +1
-0@@ -197,6 +343,7 @@ static const struct { } abort_handlers[] = { { ABORT_LOCKDEP, handle_lockdep }, { ABORT_TAINT, handle_taint }, + { ABORT_PING, handle_ping }, { 0, 0 }, };
So the handlers are used need_to_abort():
* when we start execution
* after each tests
What worries me here are the suspend tests. You have mentioned that getting network up can take up to 17s, so potentially we are adding 17s * numer_of_executed_suspend_tests to the total execution time.
Have you checked the actual impact?
Also this does not cover any network issues that happen when executing longer tests - we still can get killed by any external watchdogs.
What is the maximum ping timeout on those external watchdogs? Maybe we should get strict on tests' time budget for BAT/shards first...
- Arek
Quoted 70 lines
runner/executor.c +3
-0@@ -1288,6 +1435,9 @@ bool execute(struct execute_state *state,init_watchdogs (settings); +if (settings->abort_mask &ABORT_PING ) +ping_config (); +if (!uname (&unamebuf)) { dprintf(unamefd, "%s %s %s %s %s\n", unamebuf.sysname,runner/meson.build +13
-1@@ -1,4 +1,15 @@ jsonc = dependency('json-c', required: _runner_required) +runner_deps = [jsonc, glib] +runner_c_args = [] + +with_oping = get_option('with_oping') +if with_oping != 'false' + oping = dependency('liboping', required: with_oping == 'true') + if oping.found() + runner_deps += oping + runner_c_args += '-DHAVE_OPING=1' + endif +endif runnerlib_sources = [ 'settings.c', 'job_list.c',@@ -21,7 +32,8 @@ if _build_runner and _build_tests and jsonc.found() runnerlib = static_library('igt_runner', runnerlib_sources, include_directories : inc, - dependencies : [jsonc, glib]) + c_args : runner_c_args, + dependencies : runner_deps) runner = executable('igt_runner', runner_sources, link_with : runnerlib,runner/settings.c +4
-0@@ -48,6 +48,7 @@ static struct { } abort_conditions[] = { { ABORT_TAINT, "taint" }, { ABORT_LOCKDEP, "lockdep" }, + { ABORT_PING, "ping" }, { ABORT_ALL, "all" }, { 0, 0 }, };@@ -136,6 +137,9 @@ static const char *usage_str =" Possible conditions:\n" " lockdep - abort when kernel lockdep has been angered.\n" " taint - abort when kernel becomes fatally tainted.\n" +" ping - abort when a host configured in .igtrc or\n" +" environment variable IGT_PING_HOSTNAME does\n" +" not respond to ping.\n" " all - abort for all of the above.\n" " -s, --sync Sync results to disk after every test\n" " -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n" runner/settings.h +0
-0@@ -15,9 +15,10 @@ enum {#define ABORT_TAINT (1 << 0)
runner/settings.h enum { · line 24
Quoted 10 lines
runner/settings.h +3
-2#define ABORT_LOCKDEP (1 << 1) -#define ABORT_ALL (ABORT_TAINT | ABORT_LOCKDEP) +#define ABORT_PING (1 << 2) +#define ABORT_ALL (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING) -_Static_assert (ABORT_ALL == (ABORT_TAINT |ABORT_LOCKDEP ),"ABORT_ALL must be all conditions bitwise or'd" ); +_Static_assert (ABORT_ALL == (ABORT_TAINT |ABORT_LOCKDEP |ABORT_PING ),"ABORT_ALL must be all conditions bitwise or'd" );struct regex_list {char **regex_strings;
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Quoted 245 lines · Arkadiusz Hiler
Quoted 98 lines · Petri Latvala
If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.
A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.
v2:
- Use a higher timeout
- Allow hostname configuration from environment
v3:
- Userunner_c_argsfor holding c args for runner
- Handle runner's meson options in runner/meson.build
- Instead of one ping with 20 second timeout, ping with 1 second timeout for a duration of 20 secondsSigned-off-by Petri Latvala <[email protected]>Cc Arkadiusz Hiler <[email protected]>Cc Martin Peres <[email protected]>Cc Tomi Sarvela <[email protected]>Cc Daniel Vetter <[email protected]>--- meson_options.txt | 6 ++ runner/executor.c | 150 +++++++++++++++++++++++++++++++++++++++++++++ runner/meson.build | 14 ++++- runner/settings.c | 4 ++ runner/settings.h | 5 +- 5 files changed, 176 insertions(+), 3 deletions(-)meson_options.txt +6
-0@@ -58,6 +58,12 @@ option('build_runner', choices : ['auto', 'true', 'false'], description : 'Build test runner') +option('with_oping', + type : 'combo', + value : 'auto', + choices : ['auto', 'true', 'false'], + description : 'Build igt_runner with liboping') + option('use_rpath', type : 'boolean', value : false,runner/executor.c +28
-0@@ -1,6 +1,10 @@#include <errno.h> #include <fcntl.h> +#include <glib.h> #include <linux/watchdog.h> +#if HAVE_OPING +#include <oping.h> +#endif #include <signal.h> #include <stdio.h> #include <stdlib.h> @@ -16,6 +20,7 @@#include <time.h> #include <unistd.h> +#include "igt_aux.h" #include "igt_core.h" #include "executor.h" #include "output_strings.h" @@ -108,6 +113,147 @@ static void ping_watchdogs(void) } } +#if HAVE_OPING +static pingobj_t *pingobj =NULL ; + +static bool load_ping_config_from_file (void ) +{ +char *key_file_env =NULL ; +char *key_file_loc =NULL ; +GError *error =NULL ; +GKeyFile *key_file =NULL ; +const char *ping_hostname; +int ret; + +/* Determine igt config path */ + key_file_env =getenv ("IGT_CONFIG_PATH" ); +if (key_file_env) { + key_file_loc =strdup (key_file_env); + }else { +asprintf (&key_file_loc,"%s/.igtrc" ,g_get_home_dir ()); + } + +/* Load igt config file */ + key_file =g_key_file_new (); + ret = g_key_file_load_from_file(key_file, key_file_loc,runner/executor.c static void ping_watchdogs(void) · line 148
Quoted 10 lines
runner/executor.c +10
-0+ G_KEY_FILE_NONE, &error); +free (key_file_loc); + +if (!ret) { +g_error_free (error); +g_key_file_free (key_file); +return false; + } + +g_clear_error (&error);put that in
GKeyFile
*igt_open_igtrc();and use it in
igt_core, so we have only one implementationQuoted 36 lines
runner/executor.c +36
-0+ + ping_hostname = +g_key_file_get_string (key_file,"DUT" , +"PingHostName" , &error); + +g_clear_error (&error); +g_key_file_free (key_file); + +if (!ping_hostname) +return false; + +if (ping_host_add (pingobj, ping_hostname)) { +fprintf (stderr, +"abort on ping: Cannot use hostname from config file\n" ); +return false; + } + +return true; +} + +static bool load_ping_config_from_env (void ) +{ +const char *ping_hostname; + + ping_hostname =getenv ("IGT_PING_HOSTNAME" ); +if (!ping_hostname) +return false; + +if (ping_host_add (pingobj, ping_hostname)) { +fprintf (stderr, +"abort on ping: Cannot use hostname from environment\n" ); +return false; + } + +return true; +}runner/executor.c static void ping_watchdogs(void) · line 194
Quoted 10 lines
runner/executor.c +10
-0+ +static bool can_ping (void ) +{ +/* +* On some hosts, getting network back up after suspend takes +* upwards of 10 seconds. 20 seconds should be enough to see +* if network comes back at all, and hopefully not too long to +* make external monitoring freak out. +*/ + igt_until_timeout(20) {make 20 a #define
Quoted 18 lines
runner/executor.c +18
-0+pingobj_iter_t *iter; + +ping_send (pingobj); + +for (iter =ping_iterator_get (pingobj); + iter !=NULL ; + iter =ping_iterator_next (iter)) { +double latency; +size_t len =sizeof (latency); + +ping_iterator_get_info (iter, +PING_INFO_LATENCY , + &latency, + &len); +if (latency >=0.0 ) +return true; + } + }runner/executor.c static void ping_watchdogs(void) · line 222
Quoted 10 lines
runner/executor.c +10
-0+ + return false; +} + +#endif + +static void ping_config (void ) +{ +#if HAVE_OPING +double timeout =1.0 ;
single_attempt_timeoutQuoted 38 lines
runner/executor.c +34
-0+ +if (pingobj) +return ; + + pingobj =ping_construct (); + +/* Try env first, then config file */ +if (!load_ping_config_from_env () && !load_ping_config_from_file ()) { +fprintf (stderr, +"abort on ping: No host to ping configured\n" ); +ping_destroy (pingobj); + pingobj =NULL ; +return ; + } + +ping_setopt (pingobj,PING_OPT_TIMEOUT , &timeout); +#endif +} + +static char *handle_ping (void ) +{ +#if HAVE_OPING +if (pingobj && !can_ping ()) { +char *reason; + +asprintf (&reason, +"Ping host did not respond to ping, network down" ); +return reason; + } +#endif + +return NULL ; +} +static char *handle_lockdep (void ) {const char *header ="Lockdep not active\n\n/proc/lockdep_stats contents:\n" ;runner/executor.c static const struct { · line 348
Quoted 6 lines
runner/executor.c +1
-0@@ -197,6 +343,7 @@ static const struct { } abort_handlers[] = { { ABORT_LOCKDEP, handle_lockdep }, { ABORT_TAINT, handle_taint }, + { ABORT_PING, handle_ping }, { 0, 0 }, };So the handlers are used
need_to_abort():
* when we start execution
* after each tests
Quoted 5 lines
What worries me here are the suspend tests. You have mentioned that getting network up can take up to 17s, so potentially we are adding 17s *
numer_of_executed_suspend_teststo the total execution time.Have you checked the actual impact?
No, let's take a look at that after I send v4.
Even with that extra overhead, an argument can be made that getting incompletes without clear indicators why they happen is worse.
Quoted 2 lines
Also this does not cover any network issues that happen when executing longer tests - we still can get killed by any external watchdogs.
Yeah, and we don't get easily machine-parseable timing data on how long those running tests were fine for. Note to self: Make runner dump periodical still-alive marks to journal. Unrelated to this patch though...
Needs to be said that this patch won't make much of a difference if you only look at the passrates or amount of incompletes. What this does is allowing us to close that one silly semi-catchall cibuglog filter for incompletes without good data, and thus forcing proper issue binning.
Quoted 3 lines
What is the maximum ping timeout on those external watchdogs? Maybe we should get strict on tests' time budget for BAT/shards first...
Don't know actually. Tomi?
Signature · 5 lines
Petri Latvala _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev
Quoted 273 lines · Petri Latvala
Quoted 245 lines · Arkadiusz Hiler
Quoted 98 lines · Petri Latvala
If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.
A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.
v2:
- Use a higher timeout
- Allow hostname configuration from environment
v3:
- Userunner_c_argsfor holding c args for runner
- Handle runner's meson options in runner/meson.build
- Instead of one ping with 20 second timeout, ping with 1 second timeout for a duration of 20 secondsSigned-off-by Petri Latvala <[email protected]>Cc Arkadiusz Hiler <[email protected]>Cc Martin Peres <[email protected]>Cc Tomi Sarvela <[email protected]>Cc Daniel Vetter <[email protected]>--- meson_options.txt | 6 ++ runner/executor.c | 150 +++++++++++++++++++++++++++++++++++++++++++++ runner/meson.build | 14 ++++- runner/settings.c | 4 ++ runner/settings.h | 5 +- 5 files changed, 176 insertions(+), 3 deletions(-)meson_options.txt +6
-0@@ -58,6 +58,12 @@ option('build_runner', choices : ['auto', 'true', 'false'], description : 'Build test runner') +option('with_oping', + type : 'combo', + value : 'auto', + choices : ['auto', 'true', 'false'], + description : 'Build igt_runner with liboping') + option('use_rpath', type : 'boolean', value : false,runner/executor.c +28
-0@@ -1,6 +1,10 @@#include <errno.h> #include <fcntl.h> +#include <glib.h> #include <linux/watchdog.h> +#if HAVE_OPING +#include <oping.h> +#endif #include <signal.h> #include <stdio.h> #include <stdlib.h> @@ -16,6 +20,7 @@#include <time.h> #include <unistd.h> +#include "igt_aux.h" #include "igt_core.h" #include "executor.h" #include "output_strings.h" @@ -108,6 +113,147 @@ static void ping_watchdogs(void) } } +#if HAVE_OPING +static pingobj_t *pingobj =NULL ; + +static bool load_ping_config_from_file (void ) +{ +char *key_file_env =NULL ; +char *key_file_loc =NULL ; +GError *error =NULL ; +GKeyFile *key_file =NULL ; +const char *ping_hostname; +int ret; + +/* Determine igt config path */ + key_file_env =getenv ("IGT_CONFIG_PATH" ); +if (key_file_env) { + key_file_loc =strdup (key_file_env); + }else { +asprintf (&key_file_loc,"%s/.igtrc" ,g_get_home_dir ()); + } + +/* Load igt config file */ + key_file =g_key_file_new (); + ret = g_key_file_load_from_file(key_file, key_file_loc,runner/executor.c static void ping_watchdogs(void) · line 148
Quoted 10 lines
runner/executor.c +10
-0+ G_KEY_FILE_NONE, &error); +free (key_file_loc); + +if (!ret) { +g_error_free (error); +g_key_file_free (key_file); +return false; + } + +g_clear_error (&error);put that in
GKeyFile
*igt_open_igtrc();and use it in
igt_core, so we have only one implementationQuoted 36 lines
runner/executor.c +36
-0+ + ping_hostname = +g_key_file_get_string (key_file,"DUT" , +"PingHostName" , &error); + +g_clear_error (&error); +g_key_file_free (key_file); + +if (!ping_hostname) +return false; + +if (ping_host_add (pingobj, ping_hostname)) { +fprintf (stderr, +"abort on ping: Cannot use hostname from config file\n" ); +return false; + } + +return true; +} + +static bool load_ping_config_from_env (void ) +{ +const char *ping_hostname; + + ping_hostname =getenv ("IGT_PING_HOSTNAME" ); +if (!ping_hostname) +return false; + +if (ping_host_add (pingobj, ping_hostname)) { +fprintf (stderr, +"abort on ping: Cannot use hostname from environment\n" ); +return false; + } + +return true; +}runner/executor.c static void ping_watchdogs(void) · line 194
Quoted 10 lines
runner/executor.c +10
-0+ +static bool can_ping (void ) +{ +/* +* On some hosts, getting network back up after suspend takes +* upwards of 10 seconds. 20 seconds should be enough to see +* if network comes back at all, and hopefully not too long to +* make external monitoring freak out. +*/ + igt_until_timeout(20) {make 20 a #define
Quoted 18 lines
runner/executor.c +18
-0+pingobj_iter_t *iter; + +ping_send (pingobj); + +for (iter =ping_iterator_get (pingobj); + iter !=NULL ; + iter =ping_iterator_next (iter)) { +double latency; +size_t len =sizeof (latency); + +ping_iterator_get_info (iter, +PING_INFO_LATENCY , + &latency, + &len); +if (latency >=0.0 ) +return true; + } + }runner/executor.c static void ping_watchdogs(void) · line 222
Quoted 10 lines
runner/executor.c +10
-0+ + return false; +} + +#endif + +static void ping_config (void ) +{ +#if HAVE_OPING +double timeout =1.0 ;
single_attempt_timeoutQuoted 38 lines
runner/executor.c +34
-0+ +if (pingobj) +return ; + + pingobj =ping_construct (); + +/* Try env first, then config file */ +if (!load_ping_config_from_env () && !load_ping_config_from_file ()) { +fprintf (stderr, +"abort on ping: No host to ping configured\n" ); +ping_destroy (pingobj); + pingobj =NULL ; +return ; + } + +ping_setopt (pingobj,PING_OPT_TIMEOUT , &timeout); +#endif +} + +static char *handle_ping (void ) +{ +#if HAVE_OPING +if (pingobj && !can_ping ()) { +char *reason; + +asprintf (&reason, +"Ping host did not respond to ping, network down" ); +return reason; + } +#endif + +return NULL ; +} +static char *handle_lockdep (void ) {const char *header ="Lockdep not active\n\n/proc/lockdep_stats contents:\n" ;runner/executor.c static const struct { · line 348
Quoted 6 lines
runner/executor.c +1
-0@@ -197,6 +343,7 @@ static const struct { } abort_handlers[] = { { ABORT_LOCKDEP, handle_lockdep }, { ABORT_TAINT, handle_taint }, + { ABORT_PING, handle_ping }, { 0, 0 }, };So the handlers are used
need_to_abort():
* when we start execution
* after each testsQuoted 5 lines
What worries me here are the suspend tests. You have mentioned that getting network up can take up to 17s, so potentially we are adding 17s *
numer_of_executed_suspend_teststo the total execution time.Have you checked the actual impact?
No, let's take a look at that after I send v4.
Even with that extra overhead, an argument can be made that getting incompletes without clear indicators why they happen is worse.
Quoted 2 lines
Also this does not cover any network issues that happen when executing longer tests - we still can get killed by any external watchdogs.
Yeah, and we don't get easily machine-parseable timing data on how long those running tests were fine for. Note to self: Make runner dump periodical still-alive marks to journal. Unrelated to this patch though...
Needs to be said that this patch won't make much of a difference if you only look at the passrates or amount of incompletes. What this does is allowing us to close that one silly semi-catchall cibuglog filter for incompletes without good data, and thus forcing proper issue binning.
Quoted 1 line
Quoted 4 lines
Quoted 2 lines
What is the maximum ping timeout on those external watchdogs? Maybe we should get strict on tests' time budget for BAT/shards first...
Don't know actually. Tomi?
SSH timeout (TCP timeout) is first, ~2 minutes. To make sure that the external watchdog doesn't mess up live system, it only activates when the target is offline 6 minutes (pinged several times during this time).
Tomi
Signature · 5 lines
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo _______________________________________________ igt-dev mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/igt-dev