Skip to contentSkip to conversation

runner: Add support for aborting on network failure

PATCH Superseded Petri Latvala 5 participants 8 messages May 16, 2019 – May 20, 2019 Arm/Amlogic Meson SoC support

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/2 runner: Add support for aborting on network failure

If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.

A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.

v2:
- Use a higher timeout
- Allow hostname configuration from environment

Cc Arkadiusz Hiler <[email protected]>
Cc Martin Peres <[email protected]>
Cc Tomi Sarvela <[email protected]>
---
 meson.build        |   1 +
 meson_options.txt  |   6 ++
 runner/executor.c  | 135 +++++++++++++++++++++++++++++++++++++++++++++
 runner/meson.build |  12 +++-
 runner/settings.c  |   4 ++
 runner/settings.h  |   5 +-
 6 files changed, 160 insertions(+), 3 deletions(-)
meson.build +1 -0
index a05e912c..aac67f1a 100644
@@ -100,6 +100,7 @@ build_tests = get_option('build_tests')
 with_libdrm = get_option('with_libdrm')
 with_libunwind = get_option('with_libunwind')
 build_runner = get_option('build_runner')
+with_oping = get_option('with_oping')
 
 _build_overlay = build_overlay != 'false'
 _overlay_required = build_overlay == 'true'
meson_options.txt +6 -0
index 888efe56..935f883e 100644
@@ -58,6 +58,12 @@ option('build_runner',
        choices : ['auto', 'true', 'false'],
        description : 'Build test runner')
 
+option('with_oping',
+       type : 'combo',
+       value : 'auto',
+       choices : ['auto', 'true', 'false'],
+       description : 'Build igt_runner with liboping')
+
 option('use_rpath',
        type : 'boolean',
        value : false,
runner/executor.c +135 -0

159 lines not on this page · raw message

runner/meson.build +11 -1
index b658f1d2..a54aaab4 100644
@@ -1,4 +1,13 @@
 jsonc = dependency('json-c', required: _runner_required)
+runner_deps = [jsonc, glib]
+have_oping = []
+if with_oping != 'false'
+	oping = dependency('liboping', required: with_oping == 'true')
+	if oping.found()
+		runner_deps += oping
+		have_oping = '-DHAVE_OPING=1'
+	endif
+endif
 
 runnerlib_sources = [ 'settings.c',
 		      'job_list.c',
@@ -21,7 +30,8 @@ if _build_runner and _build_tests and jsonc.found()
 
 	runnerlib = static_library('igt_runner', runnerlib_sources,
 				   include_directories : inc,
-				   dependencies : [jsonc, glib])
+				   c_args : have_oping,
+				   dependencies : runner_deps)
 
 	runner = executable('igt_runner', runner_sources,
 			    link_with : runnerlib,
runner/settings.c +4 -0
index ad38ae8d..b57d1a6a 100644
@@ -48,6 +48,7 @@ static struct {
 } abort_conditions[] = {
 	{ ABORT_TAINT, "taint" },
 	{ ABORT_LOCKDEP, "lockdep" },
+	{ ABORT_PING, "ping" },
 	{ ABORT_ALL, "all" },
 	{ 0, 0 },
 };
@@ -136,6 +137,9 @@ static const char *usage_str =
 	"                        Possible conditions:\n"
 	"                         lockdep - abort when kernel lockdep has been angered.\n"
 	"                         taint   - abort when kernel becomes fatally tainted.\n"
+	"                         ping    - abort when a host configured in .igtrc or\n"
+	"                                   environment variable IGT_PING_HOSTNAME does\n"
+	"                                   not respond to ping.\n"
 	"                         all     - abort for all of the above.\n"
 	"  -s, --sync            Sync results to disk after every test\n"
 	"  -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
runner/settings.h +3 -2
index 0a1ee08d..2b6e65d0 100644
@@ -15,9 +15,10 @@ enum {
 
 #define ABORT_TAINT   (1 << 0)
 #define ABORT_LOCKDEP (1 << 1)
-#define ABORT_ALL     (ABORT_TAINT | ABORT_LOCKDEP)
+#define ABORT_PING    (1 << 2)
+#define ABORT_ALL     (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING)
 
-_Static_assert(ABORT_ALL == (ABORT_TAINT | ABORT_LOCKDEP), "ABORT_ALL must be all conditions bitwise or'd");
+_Static_assert(ABORT_ALL == (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING), "ABORT_ALL must be all conditions bitwise or'd");
 
 struct regex_list {
 	char **regex_strings;
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#1 lore raw

PATCH 2/2 HAX: Check all conditions to abort

---
 runner/executor.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
runner/executor.c +1 -1
index be78474c..72f2f020 100644
@@ -339,7 +339,7 @@ static char *need_to_abort(const struct settings* settings)
 	for (it = abort_handlers; it->condition; it++) {
 		char *abort;
 
-		if (!(settings->abort_mask & it->condition))
+		if (false && !(settings->abort_mask & it->condition))
 			continue;
 
 		abort = it->handler();
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#2 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for series starting with [i-g-t,1/2,v2] runner: Add support for aborting on network failure

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/2,v2] runner: Add support for aborting on network failure
URL : https://patchwork.freedesktop.org/series/60711/
State : success

== Summary ==

CI Bug Log - changes from IGT_4993 -> IGTPW_2988
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/60711/revisions/1/mbox/

Known issues
------------

Here are the changes found in IGTPW_2988 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@i915_selftest@live_contexts:
- fi-bdw-gvtdvm: [PASS][1] -> [DMESG-FAIL][2] ([fdo#110235])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/fi-bdw-gvtdvm/igt@i915_selftest@live_contexts.html
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/fi-bdw-gvtdvm/igt@i915_selftest@live_contexts.html

* igt@kms_frontbuffer_tracking@basic:
- fi-icl-u2: [PASS][3] -> [FAIL][4] ([fdo#103167])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/fi-icl-u2/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/fi-icl-u2/igt@[email protected]

* igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a:
- fi-blb-e6850: [PASS][5] -> [INCOMPLETE][6] ([fdo#107718])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/fi-blb-e6850/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/fi-blb-e6850/igt@[email protected]

#### Warnings ####

* igt@i915_selftest@live_hangcheck:
- fi-apl-guc: [INCOMPLETE][7] ([fdo#103927] / [fdo#110624]) -> [DMESG-FAIL][8] ([fdo#110620])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/fi-apl-guc/igt@i915_selftest@live_hangcheck.html
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/fi-apl-guc/igt@i915_selftest@live_hangcheck.html

* igt@runner@aborted:
- fi-apl-guc: [FAIL][9] ([fdo#110624]) -> [FAIL][10] ([fdo#110622])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/fi-apl-guc/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/fi-apl-guc/igt@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[fdo#103167]: https://bugs.freedesktop.org/show_bug.cgi?id=103167
[fdo#103927]: https://bugs.freedesktop.org/show_bug.cgi?id=103927
[fdo#107713]: https://bugs.freedesktop.org/show_bug.cgi?id=107713
[fdo#107718]: https://bugs.freedesktop.org/show_bug.cgi?id=107718
[fdo#108569]: https://bugs.freedesktop.org/show_bug.cgi?id=108569
[fdo#110235]: https://bugs.freedesktop.org/show_bug.cgi?id=110235
[fdo#110620]: https://bugs.freedesktop.org/show_bug.cgi?id=110620
[fdo#110622]: https://bugs.freedesktop.org/show_bug.cgi?id=110622
[fdo#110624]: https://bugs.freedesktop.org/show_bug.cgi?id=110624

Participating hosts (52 -> 45)
------------------------------

Missing (7): fi-kbl-soraka fi-ilk-m540 fi-hsw-4200u fi-byt-squawks fi-bsw-cyan fi-byt-clapper fi-bdw-samus

Build changes
-------------

* IGT: IGT_4993 -> IGTPW_2988

CI_DRM_6090: df27ebbc9ec0d8ae42e8cf3d7a3b29fd6baf4940 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_2988: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/
IGT_4993: 2d41b51980257a7d0a3de30a9b3ea0e95b13df91 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#3 lore raw
In reply to Petri Latvala
Quoted 166 lines · Petri Latvala

If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.

A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.

v2:
- Use a higher timeout
- Allow hostname configuration from environment

Cc Arkadiusz Hiler <[email protected]>
Cc Martin Peres <[email protected]>
Cc Tomi Sarvela <[email protected]>
---
 meson.build        |   1 +
 meson_options.txt  |   6 ++
 runner/executor.c  | 135 +++++++++++++++++++++++++++++++++++++++++++++
 runner/meson.build |  12 +++-
 runner/settings.c  |   4 ++
 runner/settings.h  |   5 +-
 6 files changed, 160 insertions(+), 3 deletions(-)
meson.build +1 -0
index a05e912c..aac67f1a 100644
@@ -100,6 +100,7 @@ build_tests = get_option('build_tests')
 with_libdrm = get_option('with_libdrm')
 with_libunwind = get_option('with_libunwind')
 build_runner = get_option('build_runner')
+with_oping = get_option('with_oping')
 
 _build_overlay = build_overlay != 'false'
 _overlay_required = build_overlay == 'true'
meson_options.txt +6 -0
index 888efe56..935f883e 100644
@@ -58,6 +58,12 @@ option('build_runner',
        choices : ['auto', 'true', 'false'],
        description : 'Build test runner')
 
+option('with_oping',
+       type : 'combo',
+       value : 'auto',
+       choices : ['auto', 'true', 'false'],
+       description : 'Build igt_runner with liboping')
+
 option('use_rpath',
        type : 'boolean',
        value : false,
runner/executor.c +96 -0
index 7e5fbe8f..be78474c 100644
@@ -1,6 +1,10 @@
 #include <errno.h>
 #include <fcntl.h>
+#include <glib.h>
 #include <linux/watchdog.h>
+#if HAVE_OPING
+#include <oping.h>
+#endif
 #include <signal.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -108,6 +112,133 @@ static void ping_watchdogs(void)
 	}
 }
 
+#if HAVE_OPING
+static pingobj_t *pingobj = NULL;
+
+static bool load_ping_config_from_file(void)
+{
+	char *key_file_env = NULL;
+	char *key_file_loc = NULL;
+	GError *error = NULL;
+	GKeyFile *key_file = NULL;
+	const char *ping_hostname;
+	int ret;
+
+	/* Determine igt config path */
+	key_file_env = getenv("IGT_CONFIG_PATH");
+	if (key_file_env) {
+		key_file_loc = strdup(key_file_env);
+	} else {
+		asprintf(&key_file_loc, "%s/.igtrc", g_get_home_dir());
+	}
+
+	/* Load igt config file */
+	key_file = g_key_file_new();
+	ret = g_key_file_load_from_file(key_file, key_file_loc,
+					G_KEY_FILE_NONE, &error);
+	free(key_file_loc);
+
+	if (!ret) {
+		g_error_free(error);
+		g_key_file_free(key_file);
+		return false;
+	}
+
+	g_clear_error(&error);
+
+	ping_hostname =
+		g_key_file_get_string(key_file, "DUT",
+				      "PingHostName", &error);
+
+	g_clear_error(&error);
+	g_key_file_free(key_file);
+
+	if (!ping_hostname)
+		return false;
+
+	if (ping_host_add(pingobj, ping_hostname)) {
+		fprintf(stderr,
+			"abort on ping: Cannot use hostname from config file\n");
+		return false;
+	}
+
+	return true;
+}
+
+static bool load_ping_config_from_env(void)
+{
+	const char *ping_hostname;
+
+	ping_hostname = getenv("IGT_PING_HOSTNAME");
+	if (!ping_hostname)
+		return false;
+
+	if (ping_host_add(pingobj, ping_hostname)) {
+		fprintf(stderr,
+			"abort on ping: Cannot use hostname from environment\n");
+		return false;
+	}
+
+	return true;
+}
+
+#endif
+
+static void ping_config(void)
+{
+#if HAVE_OPING
+	double timeout = 20.0; /* Fair dice roll */
+
+	if (pingobj)
+		return;
+
+	pingobj = ping_construct();
+
+	/* Try env first, then config file */
+	if (!load_ping_config_from_env() && !load_ping_config_from_file()) {
+		fprintf(stderr,
+			"abort on ping: No host to ping configured\n");
+		ping_destroy(pingobj);
+		pingobj = NULL;
+		return;
+	}
+
+	ping_setopt(pingobj, PING_OPT_TIMEOUT, &timeout);

runner/executor.c static void ping_watchdogs(void) · line 216

Quoted 10 lines
runner/executor.c +10 -0
+#endif
+}
+
+static char *handle_ping(void)
+{
+#if HAVE_OPING
+	if (pingobj) {
+		pingobj_iter_t *iter;
+
+		ping_send(pingobj);

I think we should try to ping a few times, in case our ping (or the reply got lost). Networks aren't 100% reliable, even when they work. Throwing out 3 pings or so should be good enough, maybe try that twice.

As long as we die quicker than jenkins declares us dead we should be good enough.

btw, could we also somehow check the ssh connection itself that we're running under? I think that would more directly make sure that Jenkins still knows we're doing well ...
-Daniel

Quoted 120 lines
runner/executor.c +29 -0
+
+		for (iter = ping_iterator_get(pingobj);
+		     iter != NULL;
+		     iter = ping_iterator_next(iter)) {
+			double latency;
+			size_t len = sizeof(latency);
+
+			ping_iterator_get_info(iter,
+					       PING_INFO_LATENCY,
+					       &latency,
+					       &len);
+			if (latency < 0.0) {
+				char *reason;
+
+				asprintf(&reason,
+					 "Ping host did not respond to ping, network down");
+				return reason;
+			}
+		}
+	}
+#endif
+
+	return NULL;
+}
+
 static char *handle_lockdep(void)
 {
 	const char *header = "Lockdep not active\n\n/proc/lockdep_stats contents:\n";
@@ -197,6 +328,7 @@ static const struct {
 } abort_handlers[] = {
 	{ ABORT_LOCKDEP, handle_lockdep },
 	{ ABORT_TAINT, handle_taint },
+	{ ABORT_PING, handle_ping },
 	{ 0, 0 },
 };
 
@@ -1288,6 +1420,9 @@ bool execute(struct execute_state *state,
 
 	init_watchdogs(settings);
 
+	if (settings->abort_mask & ABORT_PING)
+		ping_config();
+
 	if (!uname(&unamebuf)) {
 		dprintf(unamefd, "%s %s %s %s %s\n",
 			unamebuf.sysname,
runner/meson.build +11 -1
index b658f1d2..a54aaab4 100644
@@ -1,4 +1,13 @@
 jsonc = dependency('json-c', required: _runner_required)
+runner_deps = [jsonc, glib]
+have_oping = []
+if with_oping != 'false'
+	oping = dependency('liboping', required: with_oping == 'true')
+	if oping.found()
+		runner_deps += oping
+		have_oping = '-DHAVE_OPING=1'
+	endif
+endif
 
 runnerlib_sources = [ 'settings.c',
 		      'job_list.c',
@@ -21,7 +30,8 @@ if _build_runner and _build_tests and jsonc.found()
 
 	runnerlib = static_library('igt_runner', runnerlib_sources,
 				   include_directories : inc,
-				   dependencies : [jsonc, glib])
+				   c_args : have_oping,
+				   dependencies : runner_deps)
 
 	runner = executable('igt_runner', runner_sources,
 			    link_with : runnerlib,
runner/settings.c +4 -0
index ad38ae8d..b57d1a6a 100644
@@ -48,6 +48,7 @@ static struct {
 } abort_conditions[] = {
 	{ ABORT_TAINT, "taint" },
 	{ ABORT_LOCKDEP, "lockdep" },
+	{ ABORT_PING, "ping" },
 	{ ABORT_ALL, "all" },
 	{ 0, 0 },
 };
@@ -136,6 +137,9 @@ static const char *usage_str =
 	"                        Possible conditions:\n"
 	"                         lockdep - abort when kernel lockdep has been angered.\n"
 	"                         taint   - abort when kernel becomes fatally tainted.\n"
+	"                         ping    - abort when a host configured in .igtrc or\n"
+	"                                   environment variable IGT_PING_HOSTNAME does\n"
+	"                                   not respond to ping.\n"
 	"                         all     - abort for all of the above.\n"
 	"  -s, --sync            Sync results to disk after every test\n"
 	"  -l {quiet,verbose,dummy}, --log-level {quiet,verbose,dummy}\n"
runner/settings.h +3 -2
index 0a1ee08d..2b6e65d0 100644
@@ -15,9 +15,10 @@ enum {
 
 #define ABORT_TAINT   (1 << 0)
 #define ABORT_LOCKDEP (1 << 1)
-#define ABORT_ALL     (ABORT_TAINT | ABORT_LOCKDEP)
+#define ABORT_PING    (1 << 2)
+#define ABORT_ALL     (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING)
 
-_Static_assert(ABORT_ALL == (ABORT_TAINT | ABORT_LOCKDEP), "ABORT_ALL must be all conditions bitwise or'd");
+_Static_assert(ABORT_ALL == (ABORT_TAINT | ABORT_LOCKDEP | ABORT_PING), "ABORT_ALL must be all conditions bitwise or'd");
 
 struct regex_list {
 	char **regex_strings;
Signature · 6 lines
2.19.1

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#4 lore raw

[igt-dev] ✓ Fi.CI.IGT: success for series starting with [i-g-t,1/2,v2] runner: Add support for aborting on network failure

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/2,v2] runner: Add support for aborting on network failure
URL : https://patchwork.freedesktop.org/series/60711/
State : success

== Summary ==

CI Bug Log - changes from IGT_4993_full -> IGTPW_2988_full
====================================================

Summary
-------

**SUCCESS**

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/60711/revisions/1/mbox/

Known issues
------------

Here are the changes found in IGTPW_2988_full that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_eio@reset-stress:
- shard-snb: [PASS][1] -> [FAIL][2] ([fdo#109661])
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-snb5/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-snb6/igt@[email protected]

* igt@gem_tiled_swapping@non-threaded:
- shard-iclb: [PASS][3] -> [FAIL][4] ([fdo#108686])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb5/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb7/igt@[email protected]
- shard-hsw: [PASS][5] -> [FAIL][6] ([fdo#108686])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-hsw8/igt@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-hsw1/igt@[email protected]

* igt@i915_pm_rpm@fences:
- shard-kbl: [PASS][7] -> [DMESG-WARN][8] ([fdo#103558] / [fdo#105602]) +20 similar issues
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-kbl1/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-kbl4/igt@[email protected]

* igt@i915_pm_rpm@i2c:
- shard-iclb: [PASS][9] -> [FAIL][10] ([fdo#104097])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb5/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb6/igt@[email protected]

* igt@i915_suspend@fence-restore-tiled2untiled:
- shard-apl: [PASS][11] -> [DMESG-WARN][12] ([fdo#108566]) +4 similar issues
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-apl8/igt@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-apl8/igt@[email protected]

* igt@kms_cursor_legacy@2x-nonblocking-modeset-vs-cursor-atomic:
- shard-glk: [PASS][13] -> [FAIL][14] ([fdo#107409])
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-glk5/igt@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-glk9/igt@[email protected]

* igt@kms_fence_pin_leak:
- shard-hsw: [PASS][15] -> [DMESG-WARN][16] ([fdo#102614]) +2 similar issues
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-hsw2/igt@kms_fence_pin_leak.html
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-hsw5/igt@kms_fence_pin_leak.html

* igt@kms_flip@flip-vs-suspend:
- shard-kbl: [PASS][17] -> [INCOMPLETE][18] ([fdo#103665])
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-kbl7/igt@[email protected]
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-kbl6/igt@[email protected]

* igt@kms_flip@modeset-vs-vblank-race-interruptible:
- shard-glk: [PASS][19] -> [FAIL][20] ([fdo#103060])
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-glk4/igt@[email protected]
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-glk6/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-render:
- shard-hsw: [PASS][21] -> [DMESG-FAIL][22] ([fdo#102614] / [fdo#103167])
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-hsw6/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-render.html
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-hsw5/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-render.html

* igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-render:
- shard-iclb: [PASS][23] -> [FAIL][24] ([fdo#103167]) +1 similar issue
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb4/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-render.html
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb5/igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-pri-shrfb-draw-render.html

* igt@kms_plane_lowres@pipe-a-tiling-y:
- shard-iclb: [PASS][25] -> [FAIL][26] ([fdo#103166])
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb2/igt@[email protected]
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb2/igt@[email protected]

* igt@kms_psr@psr2_cursor_plane_onoff:
- shard-iclb: [PASS][27] -> [SKIP][28] ([fdo#109441]) +5 similar issues
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb2/igt@kms_psr@psr2_cursor_plane_onoff.html
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb6/igt@kms_psr@psr2_cursor_plane_onoff.html

* igt@kms_vblank@pipe-c-ts-continuation-modeset-hang:
- shard-hsw: [PASS][29] -> [INCOMPLETE][30] ([fdo#103540])
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-hsw5/igt@[email protected]
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-hsw4/igt@[email protected]

#### Possible fixes ####

* igt@gem_mocs_settings@mocs-settings-render:
- shard-iclb: [INCOMPLETE][31] ([fdo#107713]) -> [PASS][32]
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb7/igt@[email protected]
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb5/igt@[email protected]

* {igt@kms_cursor_crc@pipe-a-cursor-128x128-onscreen}:
- shard-kbl: [FAIL][33] ([fdo#103232]) -> [PASS][34] +1 similar issue
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-kbl2/igt@[email protected]
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-kbl1/igt@[email protected]

* {igt@kms_cursor_crc@pipe-c-cursor-256x85-onscreen}:
- shard-apl: [FAIL][35] ([fdo#103232]) -> [PASS][36] +1 similar issue
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-apl4/igt@[email protected]
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-apl2/igt@[email protected]

* igt@kms_cursor_legacy@2x-long-flip-vs-cursor-atomic:
- shard-glk: [FAIL][37] ([fdo#104873]) -> [PASS][38]
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-glk2/igt@[email protected]
[38]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-glk5/igt@[email protected]

* igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-render:
- shard-iclb: [FAIL][39] ([fdo#103167]) -> [PASS][40] +1 similar issue
[39]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb4/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-render.html
[40]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb5/igt@kms_frontbuffer_tracking@fbc-1p-primscrn-pri-shrfb-draw-render.html

* igt@kms_frontbuffer_tracking@fbcpsr-1p-primscrn-cur-indfb-onoff:
- shard-iclb: [INCOMPLETE][41] ([fdo#106978] / [fdo#107713]) -> [PASS][42]
[41]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb7/igt@[email protected]
[42]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb3/igt@[email protected]

* igt@kms_plane_lowres@pipe-a-tiling-x:
- shard-iclb: [FAIL][43] ([fdo#103166]) -> [PASS][44]
[43]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb6/igt@[email protected]
[44]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb2/igt@[email protected]

* igt@kms_psr@psr2_sprite_render:
- shard-iclb: [SKIP][45] ([fdo#109441]) -> [PASS][46] +1 similar issue
[45]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb7/igt@kms_psr@psr2_sprite_render.html
[46]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb2/igt@kms_psr@psr2_sprite_render.html

* igt@kms_setmode@basic:
- shard-apl: [FAIL][47] ([fdo#99912]) -> [PASS][48]
[47]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-apl2/igt@[email protected]
[48]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-apl1/igt@[email protected]

* igt@kms_vblank@pipe-c-ts-continuation-suspend:
- shard-apl: [DMESG-WARN][49] ([fdo#108566]) -> [PASS][50] +3 similar issues
[49]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-apl1/igt@[email protected]
[50]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-apl3/igt@[email protected]

#### Warnings ####

* igt@gem_mmap_gtt@forked-big-copy-xy:
- shard-iclb: [TIMEOUT][51] ([fdo#109673]) -> [INCOMPLETE][52] ([fdo#107713] / [fdo#109100])
[51]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-iclb7/igt@[email protected]
[52]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-iclb4/igt@[email protected]

* igt@kms_busy@extended-modeset-hang-newfb-render-d:
- shard-kbl: [SKIP][53] ([fdo#109271] / [fdo#109278]) -> [SKIP][54] ([fdo#105602] / [fdo#109271] / [fdo#109278]) +2 similar issues
[53]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-kbl1/igt@[email protected]
[54]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-kbl4/igt@[email protected]

* igt@kms_panel_fitting@legacy:
- shard-kbl: [SKIP][55] ([fdo#109271]) -> [SKIP][56] ([fdo#105602] / [fdo#109271]) +15 similar issues
[55]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_4993/shard-kbl3/igt@[email protected]
[56]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/shard-kbl4/igt@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[fdo#102614]: https://bugs.freedesktop.org/show_bug.cgi?id=102614
[fdo#103060]: https://bugs.freedesktop.org/show_bug.cgi?id=103060
[fdo#103166]: https://bugs.freedesktop.org/show_bug.cgi?id=103166
[fdo#103167]: https://bugs.freedesktop.org/show_bug.cgi?id=103167
[fdo#103232]: https://bugs.freedesktop.org/show_bug.cgi?id=103232
[fdo#103540]: https://bugs.freedesktop.org/show_bug.cgi?id=103540
[fdo#103558]: https://bugs.freedesktop.org/show_bug.cgi?id=103558
[fdo#103665]: https://bugs.freedesktop.org/show_bug.cgi?id=103665
[fdo#104097]: https://bugs.freedesktop.org/show_bug.cgi?id=104097
[fdo#104873]: https://bugs.freedesktop.org/show_bug.cgi?id=104873
[fdo#105602]: https://bugs.freedesktop.org/show_bug.cgi?id=105602
[fdo#106978]: https://bugs.freedesktop.org/show_bug.cgi?id=106978
[fdo#107409]: https://bugs.freedesktop.org/show_bug.cgi?id=107409
[fdo#107713]: https://bugs.freedesktop.org/show_bug.cgi?id=107713
[fdo#108566]: https://bugs.freedesktop.org/show_bug.cgi?id=108566
[fdo#108686]: https://bugs.freedesktop.org/show_bug.cgi?id=108686
[fdo#109100]: https://bugs.freedesktop.org/show_bug.cgi?id=109100
[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[fdo#109278]: https://bugs.freedesktop.org/show_bug.cgi?id=109278
[fdo#109441]: https://bugs.freedesktop.org/show_bug.cgi?id=109441
[fdo#109661]: https://bugs.freedesktop.org/show_bug.cgi?id=109661
[fdo#109673]: https://bugs.freedesktop.org/show_bug.cgi?id=109673
[fdo#99912]: https://bugs.freedesktop.org/show_bug.cgi?id=99912

Participating hosts (7 -> 6)
------------------------------

Missing (1): shard-skl

Build changes
-------------

* IGT: IGT_4993 -> IGTPW_2988

CI_DRM_6090: df27ebbc9ec0d8ae42e8cf3d7a3b29fd6baf4940 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_2988: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/
IGT_4993: 2d41b51980257a7d0a3de30a9b3ea0e95b13df91 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_2988/
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#5 lore raw
In reply to Petri Latvala
Quoted 31 lines · Petri Latvala

If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.

A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.

v2:
- Use a higher timeout
- Allow hostname configuration from environment

Cc Arkadiusz Hiler <[email protected]>
Cc Martin Peres <[email protected]>
Cc Tomi Sarvela <[email protected]>
---
 meson.build        |   1 +
 meson_options.txt  |   6 ++
 runner/executor.c  | 135 +++++++++++++++++++++++++++++++++++++++++++++
 runner/meson.build |  12 +++-
 runner/settings.c  |   4 ++
 runner/settings.h  |   5 +-
 6 files changed, 160 insertions(+), 3 deletions(-)
meson.build +0 -0
index a05e912c..aac67f1a 100644

meson.build build_tests = get_option('build_tests') · line 103

Quoted 4 lines
meson.build +1 -0
@@ -100,6 +100,7 @@ build_tests = get_option('build_tests')
 with_libdrm = get_option('with_libdrm')
 with_libunwind = get_option('with_libunwind')
 build_runner = get_option('build_runner')
+with_oping = get_option('with_oping')

I find it a bit distracting that we have this bit here, whereas everything else is in runner/meson.build. Is there any reason for that?

Quoted 2 lines
meson.build +0 -0
 _build_overlay = build_overlay != 'false'
 _overlay_required = build_overlay == 'true'

<SNIP>

Quoted 5 lines
runner/meson.build +0 -0
index b658f1d2..a54aaab4 100644

runner/meson.build line 10

Quoted 10 lines
runner/meson.build +9 -0
@@ -1,4 +1,13 @@
 jsonc = dependency('json-c', required: _runner_required)
+runner_deps = [jsonc, glib]
+have_oping = []
+if with_oping != 'false'
+	oping = dependency('liboping', required: with_oping == 'true')
+	if oping.found()
+		runner_deps += oping
+		have_oping = '-DHAVE_OPING=1'
+	endif
+endif

I think it is about time to:
https://mesonbuild.com/Release-notes-for-0-47-0.html#new-type-of-build-option-for-features

I'll check meson versions in various popular distros Today and see how the patch would look like.

runner/meson.build if _build_runner and _build_tests and jsonc.found() · line 33

Quoted 8 lines
runner/meson.build +1 -1
 runnerlib_sources = [ 'settings.c',
 		      'job_list.c',
@@ -21,7 +30,8 @@ if _build_runner and _build_tests and jsonc.found()
 
 	runnerlib = static_library('igt_runner', runnerlib_sources,
 				   include_directories : inc,
-				   dependencies : [jsonc, glib])
+				   c_args : have_oping,

I know that we have just a single dep here, but can we name this runner_lib_c_args or something?

runner/meson.build if _build_runner and _build_tests and jsonc.found() · line 37

Quoted 4 lines
runner/meson.build +1 -0
+				   dependencies : runner_deps)
 
 	runner = executable('igt_runner', runner_sources,
 			    link_with : runnerlib,

Other than that the C part looks ok. I share Daniel's sentiment on network and NIC possibly being be a bit too "best-efforty" for aborting on a single dropped ping.

It also seems like the feature haven't been properly tested, as the HAX does not specify any IP address.

Signature · 6 lines
Cheers,
Arek
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#6 lore raw
Quoted 62 lines · Arkadiusz Hiler
Quoted 31 lines · Petri Latvala

If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.

A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.

v2:
- Use a higher timeout
- Allow hostname configuration from environment

Cc Arkadiusz Hiler <[email protected]>
Cc Martin Peres <[email protected]>
Cc Tomi Sarvela <[email protected]>
---
 meson.build        |   1 +
 meson_options.txt  |   6 ++
 runner/executor.c  | 135 +++++++++++++++++++++++++++++++++++++++++++++
 runner/meson.build |  12 +++-
 runner/settings.c  |   4 ++
 runner/settings.h  |   5 +-
 6 files changed, 160 insertions(+), 3 deletions(-)
meson.build +0 -0
index a05e912c..aac67f1a 100644

meson.build build_tests = get_option('build_tests') · line 103

Quoted 4 lines
meson.build +1 -0
@@ -100,6 +100,7 @@ build_tests = get_option('build_tests')
 with_libdrm = get_option('with_libdrm')
 with_libunwind = get_option('with_libunwind')
 build_runner = get_option('build_runner')
+with_oping = get_option('with_oping')

I find it a bit distracting that we have this bit here, whereas everything else is in runner/meson.build. Is there any reason for that?

Quoted 2 lines
meson.build +0 -0
 _build_overlay = build_overlay != 'false'
 _overlay_required = build_overlay == 'true'

<SNIP>

Quoted 5 lines
runner/meson.build +0 -0
index b658f1d2..a54aaab4 100644

runner/meson.build line 10

Quoted 10 lines
runner/meson.build +9 -0
@@ -1,4 +1,13 @@
 jsonc = dependency('json-c', required: _runner_required)
+runner_deps = [jsonc, glib]
+have_oping = []
+if with_oping != 'false'
+	oping = dependency('liboping', required: with_oping == 'true')
+	if oping.found()
+		runner_deps += oping
+		have_oping = '-DHAVE_OPING=1'
+	endif
+endif

I think it is about time to:
https://mesonbuild.com/Release-notes-for-0-47-0.html#new-type-of-build-option-for-features

Quoted 2 lines

I'll check meson versions in various popular distros Today and see how the patch would look like.

Oh, +1 from me!

I'd really like to replace some complicated logic with feature options.

Quoted 21 lines

runner/meson.build if _build_runner and _build_tests and jsonc.found() · line 33

Quoted 8 lines
runner/meson.build +1 -1
 runnerlib_sources = [ 'settings.c',
 		      'job_list.c',
@@ -21,7 +30,8 @@ if _build_runner and _build_tests and jsonc.found()
 
 	runnerlib = static_library('igt_runner', runnerlib_sources,
 				   include_directories : inc,
-				   dependencies : [jsonc, glib])
+				   c_args : have_oping,

I know that we have just a single dep here, but can we name this runner_lib_c_args or something?

runner/meson.build if _build_runner and _build_tests and jsonc.found() · line 37

Quoted 4 lines
runner/meson.build +1 -0
+				   dependencies : runner_deps)
 
 	runner = executable('igt_runner', runner_sources,
 			    link_with : runnerlib,

Other than that the C part looks ok. I share Daniel's sentiment on network and NIC possibly being be a bit too "best-efforty" for aborting on a single dropped ping.

Quoted 3 lines

It also seems like the feature haven't been properly tested, as the HAX does not specify any IP address.

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#7 lore raw
In reply to Arkadiusz Hiler
Quoted 27 lines · Arkadiusz Hiler
Quoted 26 lines · Petri Latvala

If the network goes down while testing, CI tends to interpret that as the device being down, cutting its power after a while. This causes an incomplete to an innocent test, increasing noise in the results.

A new flag to --abort-on-monitored-error, "ping", uses liboping to ping a host configured in .igtrc with one ping after each test execution and aborts the run if there is no reply in a hardcoded amount of time.

v2:
- Use a higher timeout
- Allow hostname configuration from environment

Cc Arkadiusz Hiler <[email protected]>
Cc Martin Peres <[email protected]>
Cc Tomi Sarvela <[email protected]>
---
 meson.build        |   1 +
 meson_options.txt  |   6 ++
 runner/executor.c  | 135 +++++++++++++++++++++++++++++++++++++++++++++
 runner/meson.build |  12 +++-
 runner/settings.c  |   4 ++
 runner/settings.h  |   5 +-
 6 files changed, 160 insertions(+), 3 deletions(-)
Quoted 12 lines

meson.build build_tests = get_option('build_tests') · line 103

Quoted 9 lines
meson.build +1 -0
index a05e912c..aac67f1a 100644
@@ -100,6 +100,7 @@ build_tests = get_option('build_tests')
 with_libdrm = get_option('with_libdrm')
 with_libunwind = get_option('with_libunwind')
 build_runner = get_option('build_runner')
+with_oping = get_option('with_oping')

I find it a bit distracting that we have this bit here, whereas everything else is in runner/meson.build. Is there any reason for that?

Oh, I thought we had all option fetches in toplevel. Will move that to runner/.

Quoted 23 lines
Quoted 3 lines
meson.build +0 -0
 _build_overlay = build_overlay != 'false'
 _overlay_required = build_overlay == 'true'

<SNIP>

Quoted 5 lines
runner/meson.build +0 -0
index b658f1d2..a54aaab4 100644

runner/meson.build line 10

Quoted 10 lines
runner/meson.build +9 -0
@@ -1,4 +1,13 @@
 jsonc = dependency('json-c', required: _runner_required)
+runner_deps = [jsonc, glib]
+have_oping = []
+if with_oping != 'false'
+	oping = dependency('liboping', required: with_oping == 'true')
+	if oping.found()
+		runner_deps += oping
+		have_oping = '-DHAVE_OPING=1'
+	endif
+endif

I think it is about time to:
https://mesonbuild.com/Release-notes-for-0-47-0.html#new-type-of-build-option-for-features

Quoted 3 lines

I'll check meson versions in various popular distros Today and see how the patch would look like.

Oh yes please!

Quoted 11 lines

runner/meson.build if _build_runner and _build_tests and jsonc.found() · line 33

Quoted 8 lines
runner/meson.build +1 -1
 runnerlib_sources = [ 'settings.c',
 		      'job_list.c',
@@ -21,7 +30,8 @@ if _build_runner and _build_tests and jsonc.found()
 
 	runnerlib = static_library('igt_runner', runnerlib_sources,
 				   include_directories : inc,
-				   dependencies : [jsonc, glib])
+				   c_args : have_oping,

I know that we have just a single dep here, but can we name this runner_lib_c_args or something?

Sure.

Quoted 9 lines

runner/meson.build if _build_runner and _build_tests and jsonc.found() · line 37

Quoted 5 lines
runner/meson.build +1 -0
+				   dependencies : runner_deps)
 
 	runner = executable('igt_runner', runner_sources,
 			    link_with : runnerlib,

Other than that the C part looks ok. I share Daniel's sentiment on network and NIC possibly being be a bit too "best-efforty" for aborting on a single dropped ping.

Yes, I'll make it retry a few times in the next version.

Quoted 2 lines

It also seems like the feature haven't been properly tested, as the HAX does not specify any IP address.

Why would it need a HAX? The .igtrc configuration from the time of v1 was still in place. See run.log, "abort on ping" doesn't appear on the machines I tested, and build log (not publicly available) says liboping was used.

Signature · 5 lines
Petri Latvala
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev