Skip to contentSkip to conversation

runner: Add support for aborting on network failure

PATCH In review Petri Latvala 4 participants 9 messages May 20, 2019 – Sep 10, 2019 Arm/Amlogic Meson SoC support

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/2 runner: Add support for aborting on network failure

If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.

A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.

v2:
- Use a higher timeout
- Allow hostname configuration from environment
v3:
- Use runner_c_args for holding c args for runner
- Handle runner's meson options in runner/meson.build
- Instead of one ping with 20 second timeout, ping with 1 second timeout for a duration of 20 seconds

Cc Arkadiusz Hiler <[email protected]>
Cc Martin Peres <[email protected]>
Cc Tomi Sarvela <[email protected]>
Cc Daniel Vetter <[email protected]>
---
 meson_options.txt  |   6 ++
 runner/executor.c  | 150 +++++++++++++++++++++++++++++++++++++++++++++
 runner/meson.build |  14 ++++-
 runner/settings.c  |   4 ++
 runner/settings.h  |   5 +-
 5 files changed, 176 insertions(+), 3 deletions(-)
meson_options.txt +6 -0
index 888efe56..935f883e 100644
@@ -58,6 +58,12 @@ option('build_runner',
        choices : ['auto', 'true', 'false'],
        description : 'Build test runner')
 
+option('with_oping',
+       type : 'combo',
+       value : 'auto',
+       choices : ['auto', 'true', 'false'],
+       description : 'Build igt_runner with liboping')
+
 option('use_rpath',
        type : 'boolean',
        value : false,
runner/executor.c +150 -0

180 lines not on this page · raw message

runner/meson.build +13 -1
index b658f1d2..6d60c530 100644
@@ -1,4 +1,15 @@
 jsonc = dependency('json-c', required: _runner_required)
+runner_deps = [jsonc, glib]
+runner_c_args = []
+
+with_oping = get_option('with_oping')
+if with_oping != 'false'
+	oping = dependency('liboping', required: with_oping == 'true')
+	if oping.found()
+		runner_deps += oping
+		runner_c_args += '-DHAVE_OPING=1'
+	endif
+endif
 
 runnerlib_sources = [ 'settings.c',
 		      'job_list.c',
@@ -21,7 +32,8 @@ if _build_runner and _build_tests and jsonc.found()
 
 	runnerlib = static_library('igt_runner', runnerlib_sources,
 				   include_directories : inc,
-				   dependencies : [jsonc, glib])
+				   c_args : runner_c_args,
+				   dependencies : runner_deps)
 
 	runner = executable('igt_runner', runner_sources,
 			    link_with : runnerlib,
runner/settings.c +4 -0
index ad38ae8d..b57d1a6a 100644
@@ -48,6 +48,7 @@ static struct {
 } abort_conditions[] = {
 	{ ABORT_TAINT, "taint" },
 	{ ABORT_LOCKDEP, "lockdep" },
+	{ ABORT_PING, "ping" },
 	{ ABORT_ALL, "all" },
 	{ 0, 0 },
 };
@@ -136,6 +137,9 @@ static const char *usage_str =
 	"                        Possible conditions:\n"
 	"                         lockdep - abort when kernel lockdep has been angered.\n"
 	"                         taint   - abort when kernel becomes fatally tainted.\n"
+	"                         ping    - abort when a host configured in .igtrc or\n"
+	"                                   environment variable IGT_PING_HOSTNAME does\n"
+	"                                   not respond to ping.\n"
 	"                         all     - abort for all of the above.\n"
 	"  -s, --sync            Sync results to disk after every test\n"
 	"  -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
runner/settings.h +3 -2
index 0a1ee08d..2b6e65d0 100644
@@ -15,9 +15,10 @@ enum {
 
 #define ABORT_TAINT   (1 << 0)
 #define ABORT_LOCKDEP (1 << 1)
-#define ABORT_ALL     (ABORT_TAINT | ABORT_LOCKDEP)
+#define ABORT_PING    (1 << 2)
+#define ABORT_ALL     (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING)
 
-_Static_assert(ABORT_ALL == (ABORT_TAINT | ABORT_LOCKDEP), "ABORT_ALL must be all conditions bitwise or'd");
+_Static_assert(ABORT_ALL == (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING), "ABORT_ALL must be all conditions bitwise or'd");
 
 struct regex_list {
 	char **regex_strings;
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#1 lore raw

PATCH 2/2 HAX: Check all conditions to abort

---
 runner/executor.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
runner/executor.c +1 -1
index c07e53fa..4a44c633 100644
@@ -354,7 +354,7 @@ static char *need_to_abort(const struct settings* settings)
 	for (it = abort_handlers; it->condition; it++) {
 		char *abort;
 
-		if (!(settings->abort_mask & it->condition))
+		if (false && !(settings->abort_mask & it->condition))
 			continue;
 
 		abort = it->handler();
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#2 lore raw

PATCH 2/2 HAX: Check all conditions to abort

... and remember to HAX the ping setup as well

---
 runner/executor.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
runner/executor.c +2 -2
index c07e53fa..af87e312 100644
@@ -354,7 +354,7 @@ static char *need_to_abort(const struct settings* settings)
 	for (it = abort_handlers; it->condition; it++) {
 		char *abort;
 
-		if (!(settings->abort_mask & it->condition))
+		if (false && !(settings->abort_mask & it->condition))
 			continue;
 
 		abort = it->handler();
@@ -1435,7 +1435,7 @@ bool execute(struct execute_state *state,
 
 	init_watchdogs(settings);
 
-	if (settings->abort_mask & ABORT_PING)
+	if (true && settings->abort_mask & ABORT_PING)
 		ping_config();
 
 	if (!uname(&unamebuf)) {
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#3 lore raw

PATCH 2/2 HAX: Check all conditions to abort

... and remember to HAX the ping setup as well

... correctly

---
 runner/executor.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
runner/executor.c +2 -2
index c07e53fa..8cb09dda 100644
@@ -354,7 +354,7 @@ static char *need_to_abort(const struct settings* settings)
 	for (it = abort_handlers; it->condition; it++) {
 		char *abort;
 
-		if (!(settings->abort_mask & it->condition))
+		if (false && !(settings->abort_mask & it->condition))
 			continue;
 
 		abort = it->handler();
@@ -1435,7 +1435,7 @@ bool execute(struct execute_state *state,
 
 	init_watchdogs(settings);
 
-	if (settings->abort_mask & ABORT_PING)
+	if (true || settings->abort_mask & ABORT_PING)
 		ping_config();
 
 	if (!uname(&unamebuf)) {
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#4 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for series starting with [i-g-t,v3,1/2] runner: Add support for aborting on network failure (rev2)

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,v3,1/2] runner: Add support for aborting on network failure (rev2)
URL : https://patchwork.freedesktop.org/series/60854/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_6099 -> IGTPW_3007
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/60854/revisions/2/mbox/

Known issues
------------

Here are the changes found in IGTPW_3007 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@prime_vgem@basic-fence-flip:
- fi-ilk-650: [PASS][1] -> [DMESG-WARN][2] ([fdo#106387]) +1 similar issue
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6099/fi-ilk-650/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3007/fi-ilk-650/igt@[email protected]

#### Possible fixes ####

* igt@gem_exec_suspend@basic-s4-devices:
- fi-kbl-7500u: [DMESG-WARN][3] ([fdo#105128] / [fdo#107139]) -> [PASS][4]
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6099/fi-kbl-7500u/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3007/fi-kbl-7500u/igt@[email protected]

[fdo#105128]: https://bugs.freedesktop.org/show_bug.cgi?id=105128
[fdo#106387]: https://bugs.freedesktop.org/show_bug.cgi?id=106387
[fdo#107139]: https://bugs.freedesktop.org/show_bug.cgi?id=107139

Participating hosts (52 -> 43)
------------------------------

Missing (9): fi-kbl-soraka fi-ilk-m540 fi-bxt-dsi fi-hsw-4200u fi-byt-squawks fi-bsw-cyan fi-apl-guc fi-byt-clapper fi-bdw-samus

Build changes
-------------

* IGT: IGT_4997 -> IGTPW_3007

CI_DRM_6099: 3cef0eacdd3d3448dfefb7011f1e66821c5fd41d @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3007: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3007/
IGT_4997: eff5d0db3248734845b78fcc2e2772dd4012e5af @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3007/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#5 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for series starting with [i-g-t,v3,1/2] runner: Add support for aborting on network failure (rev3)

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,v3,1/2] runner: Add support for aborting on network failure (rev3)
URL : https://patchwork.freedesktop.org/series/60854/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_6100 -> IGTPW_3009
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/60854/revisions/3/mbox/

Known issues
------------

Here are the changes found in IGTPW_3009 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@i915_selftest@live_hangcheck:
- fi-skl-iommu: [PASS][1] -> [INCOMPLETE][2] ([fdo#108602] / [fdo#108744])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6100/fi-skl-iommu/igt@i915_selftest@live_hangcheck.html
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/fi-skl-iommu/igt@i915_selftest@live_hangcheck.html

* igt@kms_frontbuffer_tracking@basic:
- fi-hsw-peppy: [PASS][3] -> [DMESG-WARN][4] ([fdo#102614])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6100/fi-hsw-peppy/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/fi-hsw-peppy/igt@[email protected]

#### Possible fixes ####

* igt@i915_selftest@live_contexts:
- fi-skl-gvtdvm: [DMESG-FAIL][5] ([fdo#110235]) -> [PASS][6]
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_6100/fi-skl-gvtdvm/igt@i915_selftest@live_contexts.html
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/fi-skl-gvtdvm/igt@i915_selftest@live_contexts.html

[fdo#102614]: https://bugs.freedesktop.org/show_bug.cgi?id=102614
[fdo#108602]: https://bugs.freedesktop.org/show_bug.cgi?id=108602
[fdo#108744]: https://bugs.freedesktop.org/show_bug.cgi?id=108744
[fdo#110235]: https://bugs.freedesktop.org/show_bug.cgi?id=110235

Participating hosts (50 -> 43)
------------------------------

Additional (1): fi-icl-y
Missing (8): fi-kbl-soraka fi-ilk-m540 fi-hsw-4200u fi-byt-squawks fi-bsw-cyan fi-apl-guc fi-pnv-d510 fi-byt-clapper

Build changes
-------------

* IGT: IGT_4997 -> IGTPW_3009

CI_DRM_6100: 6cd6c39683ceb3e48b98871fd25aea47f291f2b0 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_3009: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/
IGT_4997: eff5d0db3248734845b78fcc2e2772dd4012e5af @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_3009/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#6 lore raw
In reply to Petri Latvala
Quoted 98 lines · Petri Latvala

If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.

A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.

v2:
- Use a higher timeout
- Allow hostname configuration from environment
v3:
- Use runner_c_args for holding c args for runner
- Handle runner's meson options in runner/meson.build
- Instead of one ping with 20 second timeout, ping with 1 second timeout for a duration of 20 seconds

Cc Arkadiusz Hiler <[email protected]>
Cc Martin Peres <[email protected]>
Cc Tomi Sarvela <[email protected]>
Cc Daniel Vetter <[email protected]>
---
 meson_options.txt  |   6 ++
 runner/executor.c  | 150 +++++++++++++++++++++++++++++++++++++++++++++
 runner/meson.build |  14 ++++-
 runner/settings.c  |   4 ++
 runner/settings.h  |   5 +-
 5 files changed, 176 insertions(+), 3 deletions(-)
meson_options.txt +6 -0
index 888efe56..935f883e 100644
@@ -58,6 +58,12 @@ option('build_runner',
        choices : ['auto', 'true', 'false'],
        description : 'Build test runner')
 
+option('with_oping',
+       type : 'combo',
+       value : 'auto',
+       choices : ['auto', 'true', 'false'],
+       description : 'Build igt_runner with liboping')
+
 option('use_rpath',
        type : 'boolean',
        value : false,
runner/executor.c +28 -0
index 7e5fbe8f..c07e53fa 100644
@@ -1,6 +1,10 @@
 #include <errno.h>
 #include <fcntl.h>
+#include <glib.h>
 #include <linux/watchdog.h>
+#if HAVE_OPING
+#include <oping.h>
+#endif
 #include <signal.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -16,6 +20,7 @@
 #include <time.h>
 #include <unistd.h>
 
+#include "igt_aux.h"
 #include "igt_core.h"
 #include "executor.h"
 #include "output_strings.h"
@@ -108,6 +113,147 @@ static void ping_watchdogs(void)
 	}
 }
 
+#if HAVE_OPING
+static pingobj_t *pingobj = NULL;
+
+static bool load_ping_config_from_file(void)
+{
+	char *key_file_env = NULL;
+	char *key_file_loc = NULL;
+	GError *error = NULL;
+	GKeyFile *key_file = NULL;
+	const char *ping_hostname;
+	int ret;
+
+	/* Determine igt config path */
+	key_file_env = getenv("IGT_CONFIG_PATH");
+	if (key_file_env) {
+		key_file_loc = strdup(key_file_env);
+	} else {
+		asprintf(&key_file_loc, "%s/.igtrc", g_get_home_dir());
+	}
+
+	/* Load igt config file */
+	key_file = g_key_file_new();
+	ret = g_key_file_load_from_file(key_file, key_file_loc,

runner/executor.c static void ping_watchdogs(void) · line 148

Quoted 10 lines
runner/executor.c +10 -0
+					G_KEY_FILE_NONE, &error);
+	free(key_file_loc);
+
+	if (!ret) {
+		g_error_free(error);
+		g_key_file_free(key_file);
+		return false;
+	}
+
+	g_clear_error(&error);

put that in

GKeyFile *igt_open_igtrc();

and use it in igt_core, so we have only one implementation

Quoted 36 lines
runner/executor.c +36 -0
+
+	ping_hostname =
+		g_key_file_get_string(key_file, "DUT",
+				      "PingHostName", &error);
+
+	g_clear_error(&error);
+	g_key_file_free(key_file);
+
+	if (!ping_hostname)
+		return false;
+
+	if (ping_host_add(pingobj, ping_hostname)) {
+		fprintf(stderr,
+			"abort on ping: Cannot use hostname from config file\n");
+		return false;
+	}
+
+	return true;
+}
+
+static bool load_ping_config_from_env(void)
+{
+	const char *ping_hostname;
+
+	ping_hostname = getenv("IGT_PING_HOSTNAME");
+	if (!ping_hostname)
+		return false;
+
+	if (ping_host_add(pingobj, ping_hostname)) {
+		fprintf(stderr,
+			"abort on ping: Cannot use hostname from environment\n");
+		return false;
+	}
+
+	return true;
+}

runner/executor.c static void ping_watchdogs(void) · line 194

Quoted 10 lines
runner/executor.c +10 -0
+
+static bool can_ping(void)
+{
+	/*
+	 * On some hosts, getting network back up after suspend takes
+	 * upwards of 10 seconds. 20 seconds should be enough to see
+	 * if network comes back at all, and hopefully not too long to
+	 * make external monitoring freak out.
+	 */
+	igt_until_timeout(20) {

make 20 a #define

Quoted 18 lines
runner/executor.c +18 -0
+		pingobj_iter_t *iter;
+
+		ping_send(pingobj);
+
+		for (iter = ping_iterator_get(pingobj);
+		     iter != NULL;
+		     iter = ping_iterator_next(iter)) {
+			double latency;
+			size_t len = sizeof(latency);
+
+			ping_iterator_get_info(iter,
+					       PING_INFO_LATENCY,
+					       &latency,
+					       &len);
+			if (latency >= 0.0)
+				return true;
+		}
+	}

runner/executor.c static void ping_watchdogs(void) · line 222

Quoted 10 lines
runner/executor.c +10 -0
+
+	return false;
+}
+
+#endif
+
+static void ping_config(void)
+{
+#if HAVE_OPING
+	double timeout = 1.0;

single_attempt_timeout

Quoted 38 lines
runner/executor.c +34 -0
+
+	if (pingobj)
+		return;
+
+	pingobj = ping_construct();
+
+	/* Try env first, then config file */
+	if (!load_ping_config_from_env() && !load_ping_config_from_file()) {
+		fprintf(stderr,
+			"abort on ping: No host to ping configured\n");
+		ping_destroy(pingobj);
+		pingobj = NULL;
+		return;
+	}
+
+	ping_setopt(pingobj, PING_OPT_TIMEOUT, &timeout);
+#endif
+}
+
+static char *handle_ping(void)
+{
+#if HAVE_OPING
+	if (pingobj && !can_ping()) {
+		char *reason;
+
+		asprintf(&reason,
+			 "Ping host did not respond to ping, network down");
+		return reason;
+	}
+#endif
+
+	return NULL;
+}
+
 static char *handle_lockdep(void)
 {
 	const char *header = "Lockdep not active\n\n/proc/lockdep_stats contents:\n";

runner/executor.c static const struct { · line 348

Quoted 6 lines
runner/executor.c +1 -0
@@ -197,6 +343,7 @@ static const struct {
 } abort_handlers[] = {
 	{ ABORT_LOCKDEP, handle_lockdep },
 	{ ABORT_TAINT, handle_taint },
+	{ ABORT_PING, handle_ping },
 	{ 0, 0 },
 };

So the handlers are used need_to_abort():
* when we start execution
* after each tests

What worries me here are the suspend tests. You have mentioned that getting network up can take up to 17s, so potentially we are adding 17s * numer_of_executed_suspend_tests to the total execution time.

Have you checked the actual impact?

Also this does not cover any network issues that happen when executing longer tests - we still can get killed by any external watchdogs.

What is the maximum ping timeout on those external watchdogs? Maybe we should get strict on tests' time budget for BAT/shards first...

- Arek

Quoted 70 lines
runner/executor.c +3 -0
@@ -1288,6 +1435,9 @@ bool execute(struct execute_state *state,
 
 	init_watchdogs(settings);
 
+	if (settings->abort_mask & ABORT_PING)
+		ping_config();
+
 	if (!uname(&unamebuf)) {
 		dprintf(unamefd, "%s %s %s %s %s\n",
 			unamebuf.sysname,
runner/meson.build +13 -1
index b658f1d2..6d60c530 100644
@@ -1,4 +1,15 @@
 jsonc = dependency('json-c', required: _runner_required)
+runner_deps = [jsonc, glib]
+runner_c_args = []
+
+with_oping = get_option('with_oping')
+if with_oping != 'false'
+	oping = dependency('liboping', required: with_oping == 'true')
+	if oping.found()
+		runner_deps += oping
+		runner_c_args += '-DHAVE_OPING=1'
+	endif
+endif
 
 runnerlib_sources = [ 'settings.c',
 		      'job_list.c',
@@ -21,7 +32,8 @@ if _build_runner and _build_tests and jsonc.found()
 
 	runnerlib = static_library('igt_runner', runnerlib_sources,
 				   include_directories : inc,
-				   dependencies : [jsonc, glib])
+				   c_args : runner_c_args,
+				   dependencies : runner_deps)
 
 	runner = executable('igt_runner', runner_sources,
 			    link_with : runnerlib,
runner/settings.c +4 -0
index ad38ae8d..b57d1a6a 100644
@@ -48,6 +48,7 @@ static struct {
 } abort_conditions[] = {
 	{ ABORT_TAINT, "taint" },
 	{ ABORT_LOCKDEP, "lockdep" },
+	{ ABORT_PING, "ping" },
 	{ ABORT_ALL, "all" },
 	{ 0, 0 },
 };
@@ -136,6 +137,9 @@ static const char *usage_str =
 	"                        Possible conditions:\n"
 	"                         lockdep - abort when kernel lockdep has been angered.\n"
 	"                         taint   - abort when kernel becomes fatally tainted.\n"
+	"                         ping    - abort when a host configured in .igtrc or\n"
+	"                                   environment variable IGT_PING_HOSTNAME does\n"
+	"                                   not respond to ping.\n"
 	"                         all     - abort for all of the above.\n"
 	"  -s, --sync            Sync results to disk after every test\n"
 	"  -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
runner/settings.h +0 -0
index 0a1ee08d..2b6e65d0 100644
@@ -15,9 +15,10 @@ enum {
 
 #define ABORT_TAINT   (1 << 0)

runner/settings.h enum { · line 24

Quoted 10 lines
runner/settings.h +3 -2
 #define ABORT_LOCKDEP (1 << 1)
-#define ABORT_ALL     (ABORT_TAINT | ABORT_LOCKDEP)
+#define ABORT_PING    (1 << 2)
+#define ABORT_ALL     (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING)
 
-_Static_assert(ABORT_ALL == (ABORT_TAINT | ABORT_LOCKDEP), "ABORT_ALL must be all conditions bitwise or'd");
+_Static_assert(ABORT_ALL == (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING), "ABORT_ALL must be all conditions bitwise or'd");
 
 struct regex_list {
 	char **regex_strings;

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#7 lore raw
Quoted 245 lines · Arkadiusz Hiler
Quoted 98 lines · Petri Latvala

If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.

A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.

v2:
- Use a higher timeout
- Allow hostname configuration from environment
v3:
- Use runner_c_args for holding c args for runner
- Handle runner's meson options in runner/meson.build
- Instead of one ping with 20 second timeout, ping with 1 second timeout for a duration of 20 seconds

Cc Arkadiusz Hiler <[email protected]>
Cc Martin Peres <[email protected]>
Cc Tomi Sarvela <[email protected]>
Cc Daniel Vetter <[email protected]>
---
 meson_options.txt  |   6 ++
 runner/executor.c  | 150 +++++++++++++++++++++++++++++++++++++++++++++
 runner/meson.build |  14 ++++-
 runner/settings.c  |   4 ++
 runner/settings.h  |   5 +-
 5 files changed, 176 insertions(+), 3 deletions(-)
meson_options.txt +6 -0
index 888efe56..935f883e 100644
@@ -58,6 +58,12 @@ option('build_runner',
        choices : ['auto', 'true', 'false'],
        description : 'Build test runner')
 
+option('with_oping',
+       type : 'combo',
+       value : 'auto',
+       choices : ['auto', 'true', 'false'],
+       description : 'Build igt_runner with liboping')
+
 option('use_rpath',
        type : 'boolean',
        value : false,
runner/executor.c +28 -0
index 7e5fbe8f..c07e53fa 100644
@@ -1,6 +1,10 @@
 #include <errno.h>
 #include <fcntl.h>
+#include <glib.h>
 #include <linux/watchdog.h>
+#if HAVE_OPING
+#include <oping.h>
+#endif
 #include <signal.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -16,6 +20,7 @@
 #include <time.h>
 #include <unistd.h>
 
+#include "igt_aux.h"
 #include "igt_core.h"
 #include "executor.h"
 #include "output_strings.h"
@@ -108,6 +113,147 @@ static void ping_watchdogs(void)
 	}
 }
 
+#if HAVE_OPING
+static pingobj_t *pingobj = NULL;
+
+static bool load_ping_config_from_file(void)
+{
+	char *key_file_env = NULL;
+	char *key_file_loc = NULL;
+	GError *error = NULL;
+	GKeyFile *key_file = NULL;
+	const char *ping_hostname;
+	int ret;
+
+	/* Determine igt config path */
+	key_file_env = getenv("IGT_CONFIG_PATH");
+	if (key_file_env) {
+		key_file_loc = strdup(key_file_env);
+	} else {
+		asprintf(&key_file_loc, "%s/.igtrc", g_get_home_dir());
+	}
+
+	/* Load igt config file */
+	key_file = g_key_file_new();
+	ret = g_key_file_load_from_file(key_file, key_file_loc,

runner/executor.c static void ping_watchdogs(void) · line 148

Quoted 10 lines
runner/executor.c +10 -0
+					G_KEY_FILE_NONE, &error);
+	free(key_file_loc);
+
+	if (!ret) {
+		g_error_free(error);
+		g_key_file_free(key_file);
+		return false;
+	}
+
+	g_clear_error(&error);

put that in

GKeyFile *igt_open_igtrc();

and use it in igt_core, so we have only one implementation

Quoted 36 lines
runner/executor.c +36 -0
+
+	ping_hostname =
+		g_key_file_get_string(key_file, "DUT",
+				      "PingHostName", &error);
+
+	g_clear_error(&error);
+	g_key_file_free(key_file);
+
+	if (!ping_hostname)
+		return false;
+
+	if (ping_host_add(pingobj, ping_hostname)) {
+		fprintf(stderr,
+			"abort on ping: Cannot use hostname from config file\n");
+		return false;
+	}
+
+	return true;
+}
+
+static bool load_ping_config_from_env(void)
+{
+	const char *ping_hostname;
+
+	ping_hostname = getenv("IGT_PING_HOSTNAME");
+	if (!ping_hostname)
+		return false;
+
+	if (ping_host_add(pingobj, ping_hostname)) {
+		fprintf(stderr,
+			"abort on ping: Cannot use hostname from environment\n");
+		return false;
+	}
+
+	return true;
+}

runner/executor.c static void ping_watchdogs(void) · line 194

Quoted 10 lines
runner/executor.c +10 -0
+
+static bool can_ping(void)
+{
+	/*
+	 * On some hosts, getting network back up after suspend takes
+	 * upwards of 10 seconds. 20 seconds should be enough to see
+	 * if network comes back at all, and hopefully not too long to
+	 * make external monitoring freak out.
+	 */
+	igt_until_timeout(20) {

make 20 a #define

Quoted 18 lines
runner/executor.c +18 -0
+		pingobj_iter_t *iter;
+
+		ping_send(pingobj);
+
+		for (iter = ping_iterator_get(pingobj);
+		     iter != NULL;
+		     iter = ping_iterator_next(iter)) {
+			double latency;
+			size_t len = sizeof(latency);
+
+			ping_iterator_get_info(iter,
+					       PING_INFO_LATENCY,
+					       &latency,
+					       &len);
+			if (latency >= 0.0)
+				return true;
+		}
+	}

runner/executor.c static void ping_watchdogs(void) · line 222

Quoted 10 lines
runner/executor.c +10 -0
+
+	return false;
+}
+
+#endif
+
+static void ping_config(void)
+{
+#if HAVE_OPING
+	double timeout = 1.0;

single_attempt_timeout

Quoted 38 lines
runner/executor.c +34 -0
+
+	if (pingobj)
+		return;
+
+	pingobj = ping_construct();
+
+	/* Try env first, then config file */
+	if (!load_ping_config_from_env() && !load_ping_config_from_file()) {
+		fprintf(stderr,
+			"abort on ping: No host to ping configured\n");
+		ping_destroy(pingobj);
+		pingobj = NULL;
+		return;
+	}
+
+	ping_setopt(pingobj, PING_OPT_TIMEOUT, &timeout);
+#endif
+}
+
+static char *handle_ping(void)
+{
+#if HAVE_OPING
+	if (pingobj && !can_ping()) {
+		char *reason;
+
+		asprintf(&reason,
+			 "Ping host did not respond to ping, network down");
+		return reason;
+	}
+#endif
+
+	return NULL;
+}
+
 static char *handle_lockdep(void)
 {
 	const char *header = "Lockdep not active\n\n/proc/lockdep_stats contents:\n";

runner/executor.c static const struct { · line 348

Quoted 6 lines
runner/executor.c +1 -0
@@ -197,6 +343,7 @@ static const struct {
 } abort_handlers[] = {
 	{ ABORT_LOCKDEP, handle_lockdep },
 	{ ABORT_TAINT, handle_taint },
+	{ ABORT_PING, handle_ping },
 	{ 0, 0 },
 };

So the handlers are used need_to_abort():
* when we start execution
* after each tests

Quoted 5 lines

What worries me here are the suspend tests. You have mentioned that getting network up can take up to 17s, so potentially we are adding 17s * numer_of_executed_suspend_tests to the total execution time.

Have you checked the actual impact?

No, let's take a look at that after I send v4.

Even with that extra overhead, an argument can be made that getting incompletes without clear indicators why they happen is worse.

Quoted 2 lines

Also this does not cover any network issues that happen when executing longer tests - we still can get killed by any external watchdogs.

Yeah, and we don't get easily machine-parseable timing data on how long those running tests were fine for. Note to self: Make runner dump periodical still-alive marks to journal. Unrelated to this patch though...

Needs to be said that this patch won't make much of a difference if you only look at the passrates or amount of incompletes. What this does is allowing us to close that one silly semi-catchall cibuglog filter for incompletes without good data, and thus forcing proper issue binning.

Quoted 3 lines

What is the maximum ping timeout on those external watchdogs? Maybe we should get strict on tests' time budget for BAT/shards first...

Don't know actually. Tomi?

Signature · 5 lines
Petri Latvala
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#8 lore raw
Quoted 273 lines · Petri Latvala
Quoted 245 lines · Arkadiusz Hiler
Quoted 98 lines · Petri Latvala

If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.

A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.

v2:
- Use a higher timeout
- Allow hostname configuration from environment
v3:
- Use runner_c_args for holding c args for runner
- Handle runner's meson options in runner/meson.build
- Instead of one ping with 20 second timeout, ping with 1 second timeout for a duration of 20 seconds

Cc Arkadiusz Hiler <[email protected]>
Cc Martin Peres <[email protected]>
Cc Tomi Sarvela <[email protected]>
Cc Daniel Vetter <[email protected]>
---
  meson_options.txt  |   6 ++
  runner/executor.c  | 150 +++++++++++++++++++++++++++++++++++++++++++++
  runner/meson.build |  14 ++++-
  runner/settings.c  |   4 ++
  runner/settings.h  |   5 +-
  5 files changed, 176 insertions(+), 3 deletions(-)
meson_options.txt +6 -0
index 888efe56..935f883e 100644
@@ -58,6 +58,12 @@ option('build_runner',
         choices : ['auto', 'true', 'false'],
         description : 'Build test runner')
 
+option('with_oping',
+       type : 'combo',
+       value : 'auto',
+       choices : ['auto', 'true', 'false'],
+       description : 'Build igt_runner with liboping')
+
  option('use_rpath',
         type : 'boolean',
         value : false,
runner/executor.c +28 -0
index 7e5fbe8f..c07e53fa 100644
@@ -1,6 +1,10 @@
  #include <errno.h>
  #include <fcntl.h>
+#include <glib.h>
  #include <linux/watchdog.h>
+#if HAVE_OPING
+#include <oping.h>
+#endif
  #include <signal.h>
  #include <stdio.h>
  #include <stdlib.h>
@@ -16,6 +20,7 @@
  #include <time.h>
  #include <unistd.h>
 
+#include "igt_aux.h"
  #include "igt_core.h"
  #include "executor.h"
  #include "output_strings.h"
@@ -108,6 +113,147 @@ static void ping_watchdogs(void)
  	}
  }
 
+#if HAVE_OPING
+static pingobj_t *pingobj = NULL;
+
+static bool load_ping_config_from_file(void)
+{
+	char *key_file_env = NULL;
+	char *key_file_loc = NULL;
+	GError *error = NULL;
+	GKeyFile *key_file = NULL;
+	const char *ping_hostname;
+	int ret;
+
+	/* Determine igt config path */
+	key_file_env = getenv("IGT_CONFIG_PATH");
+	if (key_file_env) {
+		key_file_loc = strdup(key_file_env);
+	} else {
+		asprintf(&key_file_loc, "%s/.igtrc", g_get_home_dir());
+	}
+
+	/* Load igt config file */
+	key_file = g_key_file_new();
+	ret = g_key_file_load_from_file(key_file, key_file_loc,

runner/executor.c static void ping_watchdogs(void) · line 148

Quoted 10 lines
runner/executor.c +10 -0
+					G_KEY_FILE_NONE, &error);
+	free(key_file_loc);
+
+	if (!ret) {
+		g_error_free(error);
+		g_key_file_free(key_file);
+		return false;
+	}
+
+	g_clear_error(&error);

put that in

GKeyFile *igt_open_igtrc();

and use it in igt_core, so we have only one implementation

Quoted 36 lines
runner/executor.c +36 -0
+
+	ping_hostname =
+		g_key_file_get_string(key_file, "DUT",
+				      "PingHostName", &error);
+
+	g_clear_error(&error);
+	g_key_file_free(key_file);
+
+	if (!ping_hostname)
+		return false;
+
+	if (ping_host_add(pingobj, ping_hostname)) {
+		fprintf(stderr,
+			"abort on ping: Cannot use hostname from config file\n");
+		return false;
+	}
+
+	return true;
+}
+
+static bool load_ping_config_from_env(void)
+{
+	const char *ping_hostname;
+
+	ping_hostname = getenv("IGT_PING_HOSTNAME");
+	if (!ping_hostname)
+		return false;
+
+	if (ping_host_add(pingobj, ping_hostname)) {
+		fprintf(stderr,
+			"abort on ping: Cannot use hostname from environment\n");
+		return false;
+	}
+
+	return true;
+}

runner/executor.c static void ping_watchdogs(void) · line 194

Quoted 10 lines
runner/executor.c +10 -0
+
+static bool can_ping(void)
+{
+	/*
+	 * On some hosts, getting network back up after suspend takes
+	 * upwards of 10 seconds. 20 seconds should be enough to see
+	 * if network comes back at all, and hopefully not too long to
+	 * make external monitoring freak out.
+	 */
+	igt_until_timeout(20) {

make 20 a #define

Quoted 18 lines
runner/executor.c +18 -0
+		pingobj_iter_t *iter;
+
+		ping_send(pingobj);
+
+		for (iter = ping_iterator_get(pingobj);
+		     iter != NULL;
+		     iter = ping_iterator_next(iter)) {
+			double latency;
+			size_t len = sizeof(latency);
+
+			ping_iterator_get_info(iter,
+					       PING_INFO_LATENCY,
+					       &latency,
+					       &len);
+			if (latency >= 0.0)
+				return true;
+		}
+	}

runner/executor.c static void ping_watchdogs(void) · line 222

Quoted 10 lines
runner/executor.c +10 -0
+
+	return false;
+}
+
+#endif
+
+static void ping_config(void)
+{
+#if HAVE_OPING
+	double timeout = 1.0;

single_attempt_timeout

Quoted 38 lines
runner/executor.c +34 -0
+
+	if (pingobj)
+		return;
+
+	pingobj = ping_construct();
+
+	/* Try env first, then config file */
+	if (!load_ping_config_from_env() && !load_ping_config_from_file()) {
+		fprintf(stderr,
+			"abort on ping: No host to ping configured\n");
+		ping_destroy(pingobj);
+		pingobj = NULL;
+		return;
+	}
+
+	ping_setopt(pingobj, PING_OPT_TIMEOUT, &timeout);
+#endif
+}
+
+static char *handle_ping(void)
+{
+#if HAVE_OPING
+	if (pingobj && !can_ping()) {
+		char *reason;
+
+		asprintf(&reason,
+			 "Ping host did not respond to ping, network down");
+		return reason;
+	}
+#endif
+
+	return NULL;
+}
+
  static char *handle_lockdep(void)
  {
  	const char *header = "Lockdep not active\n\n/proc/lockdep_stats contents:\n";

runner/executor.c static const struct { · line 348

Quoted 6 lines
runner/executor.c +1 -0
@@ -197,6 +343,7 @@ static const struct {
  } abort_handlers[] = {
  	{ ABORT_LOCKDEP, handle_lockdep },
  	{ ABORT_TAINT, handle_taint },
+	{ ABORT_PING, handle_ping },
  	{ 0, 0 },
  };

So the handlers are used need_to_abort():
* when we start execution
* after each tests

Quoted 5 lines

What worries me here are the suspend tests. You have mentioned that getting network up can take up to 17s, so potentially we are adding 17s * numer_of_executed_suspend_tests to the total execution time.

Have you checked the actual impact?

No, let's take a look at that after I send v4.

Even with that extra overhead, an argument can be made that getting incompletes without clear indicators why they happen is worse.

Quoted 2 lines

Also this does not cover any network issues that happen when executing longer tests - we still can get killed by any external watchdogs.

Yeah, and we don't get easily machine-parseable timing data on how long those running tests were fine for. Note to self: Make runner dump periodical still-alive marks to journal. Unrelated to this patch though...

Needs to be said that this patch won't make much of a difference if you only look at the passrates or amount of incompletes. What this does is allowing us to close that one silly semi-catchall cibuglog filter for incompletes without good data, and thus forcing proper issue binning.

Quoted 1 line
Quoted 4 lines
Quoted 2 lines

What is the maximum ping timeout on those external watchdogs? Maybe we should get strict on tests' time budget for BAT/shards first...

Don't know actually. Tomi?

SSH timeout (TCP timeout) is first, ~2 minutes. To make sure that the external watchdog doesn't mess up live system, it only activates when the target is offline 6 minutes (pinged several times during this time).

Tomi

Signature · 5 lines
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev