Skip to contentSkip to conversation

kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

PATCH In review Imre Deak 6 participants 11 messages Oct 18, 2017 – Oct 20, 2017

lore
Share & tools
lore

Imre Deak

author
#0 lore raw

PATCH 1/1 kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

The CI software watchdog (owatch) will timeout if the test doesn't output anything for a long time on standard out or error. At least the plane-all-modeset-transition and plane-all-modeset-transition-fences subtests run without any output longer than the watchdog timeout, so output some more progress info.

Bugzilla: https://bugs.freedesktop.org/show_bug.cgi?id=103334

---
 tests/kms_atomic_transition.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)
tests/kms_atomic_transition.c +13 -0
index 2ae75f2d..db43d2c9 100644
@@ -525,6 +525,9 @@ run_transition_test(igt_display_t *display, enum pipe pipe, igt_output_t *output
 	}
 
 	for (i = 0; i < iter_max; i++) {
+		igt_info(".");
+		fflush(stdout);
+
 		igt_output_set_pipe(output, pipe);
 
 		wm_setup_plane(display, pipe, i, parms, fencing);
@@ -544,6 +547,9 @@ run_transition_test(igt_display_t *display, enum pipe pipe, igt_output_t *output
 
 			/* i -> i+1 will be done when i increases, can be skipped here */
 			for (j = iter_max - 1; j > i + 1; j--) {
+				igt_info(".");
+				fflush(stdout);
+
 				wm_setup_plane(display, pipe, j, parms, fencing);
 
 				if (type == TRANSITION_MODESET)
@@ -562,6 +568,8 @@ run_transition_test(igt_display_t *display, enum pipe pipe, igt_output_t *output
 		}
 	}
 
+	igt_info("\n");
+
 cleanup:
 	if (fencing)
 		unprepare_fencing(display, pipe);
@@ -780,6 +788,9 @@ static void run_modeset_tests(igt_display_t *display, int howmany, bool nonblock
 			if (!event_mask)
 				continue;
 
+			igt_info(".");
+			fflush(stdout);
+
 			commit_display(display, event_mask, nonblocking);
 
 			collect_crcs_mask(pipe_crcs, j, crcs[1]);
@@ -814,6 +825,8 @@ static void run_modeset_tests(igt_display_t *display, int howmany, bool nonblock
 		}
 	}
 
+	igt_info("\n");
+
 cleanup:
 	set_combinations(display, 0, NULL);
 	igt_display_commit2(display, COMMIT_ATOMIC);
Signature · 6 lines
2.11.0

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw

Quoting Imre Deak (2017-10-18 12:29:33)

Quoted 5 lines

The CI software watchdog (owatch) will timeout if the test doesn't output anything for a long time on standard out or error. At least the plane-all-modeset-transition and plane-all-modeset-transition-fences subtests run without any output longer than the watchdog timeout, so output some more progress info.

Then why are we using owatch? We have a test suite that is quite capable of supporting /dev/watchdog and providing a ping function. -Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#2 lore raw
In reply to Imre Deak
Quoted 5 lines · Imre Deak

The CI software watchdog (owatch) will timeout if the test doesn't output anything for a long time on standard out or error. At least the plane-all-modeset-transition and plane-all-modeset-transition-fences subtests run without any output longer than the watchdog timeout, so output some more progress info.

No, owatch is wrapping piglit, and pings the watchdog if _piglit_ prints anything. Which it does on start/exit of a test.

Signature · 5 lines
Petri Latvala
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
author
#3 lore raw
Quoted 9 lines · Petri Latvala
Quoted 5 lines · Imre Deak

The CI software watchdog (owatch) will timeout if the test doesn't output anything for a long time on standard out or error. At least the plane-all-modeset-transition and plane-all-modeset-transition-fences subtests run without any output longer than the watchdog timeout, so output some more progress info.

No, owatch is wrapping piglit, and pings the watchdog if _piglit_ prints anything. Which it does on start/exit of a test.

Yes, but the subtest itself takes more time than owatch's timeout, during which there is no output. So owatch will trigger a reboot before the subtest ends, even though it's making progress.

--Imre

Quoted 4 lines
Signature · 2 lines
-- 
Petri Latvala

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#4 lore raw
Quoted 8 lines · Imre Deak
Quoted 7 lines · Petri Latvala
Quoted 5 lines · Imre Deak

The CI software watchdog (owatch) will timeout if the test doesn't output anything for a long time on standard out or error. At least the plane-all-modeset-transition and plane-all-modeset-transition-fences subtests run without any output longer than the watchdog timeout, so output some more progress info.

Quoted 6 lines
Quoted 2 lines

No, owatch is wrapping piglit, and pings the watchdog if _piglit_ prints anything. Which it does on start/exit of a test.

Yes, but the subtest itself takes more time than owatch's timeout, during which there is no output. So owatch will trigger a reboot before the subtest ends, even though it's making progress.

Or you mean piglit doesn't forward the test's stdout, stderr? In that case this won't work.

--Imre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#5 lore raw
Quoted 8 lines · Imre Deak
Quoted 7 lines · Imre Deak
Quoted 6 lines · Petri Latvala
Quoted 5 lines · Imre Deak

The CI software watchdog (owatch) will timeout if the test doesn't output anything for a long time on standard out or error. At least the plane-all-modeset-transition and plane-all-modeset-transition-fences subtests run without any output longer than the watchdog timeout, so output some more progress info.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

No, owatch is wrapping piglit, and pings the watchdog if _piglit_ prints anything. Which it does on start/exit of a test.

Yes, but the subtest itself takes more time than owatch's timeout, during which there is no output. So owatch will trigger a reboot before the subtest ends, even though it's making progress.

Or you mean piglit doesn't forward the test's stdout, stderr? In that case this won't work.

Yeah, sorry I was too terse. Piglit redirects the test output (out and err) to its own files. Printing from the test does not trigger owatch's heartbeat.

Signature · 6 lines
Petri Latvala

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#6 lore raw
In reply to Petri Latvala
Quoted 9 lines · Petri Latvala
Quoted 5 lines · Imre Deak

The CI software watchdog (owatch) will timeout if the test doesn't output anything for a long time on standard out or error. At least the plane-all-modeset-transition and plane-all-modeset-transition-fences subtests run without any output longer than the watchdog timeout, so output some more progress info.

No, owatch is wrapping piglit, and pings the watchdog if _piglit_ prints anything. Which it does on start/exit of a test.

tbh this sounds like owatch being dense and it shouldn't try to reboot this quickly. What's the current owatch timeout?

Aside: What exactly does owatch give us? I thought jenkins also watches machines and reboots them using the ac switch ... And owatch provides spurious reboots?
-Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#7 lore raw
Quoted 23 lines

-----Original Message-----
From: Intel-gfx [mailto:[email protected]] On Behalf Of Daniel Vetter
Sent: Wednesday, October 18, 2017 5:36 PM
To: Latvala, Petri <[email protected]>
Cc: [email protected]
Subject: Re: [Intel-gfx] [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

Quoted 10 lines · Petri Latvala
Quoted 6 lines · Imre Deak

The CI software watchdog (owatch) will timeout if the test doesn't output anything for a long time on standard out or error. At least the plane-all-modeset-transition and plane-all-modeset-transition-fences
subtests run without any output longer than the watchdog timeout, so output some more progress info.

No, owatch is wrapping piglit, and pings the watchdog if _piglit_ prints anything. Which it does on start/exit of a test.

Quoted 6 lines

tbh this sounds like owatch being dense and it shouldn't try to reboot this quickly. What's the current owatch timeout?

Aside: What exactly does owatch give us? I thought jenkins also watches machines and reboots them using the ac switch ... And owatch provides spurious reboots?

Daniel,
Owatch gives us the knowledge that is was a test that took too long. I.e. we will know that it was not a system hang.
We also know that the NMI watchdog didn't trigger.
I believe this is extremely useful information when you are starting to debug the issue

Imre if you believe that owatch is preventing you from getting information to debug why these test are taking so extremely long time, it would be easy to increase the timeout or even do runs without it being enabled.

/Marta

Quoted 9 lines

-Daniel
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

author
#8 lore raw
Quoted 34 lines · Marta
Quoted 23 lines

-----Original Message-----
From: Intel-gfx [mailto:[email protected]] On Behalf Of Daniel Vetter
Sent: Wednesday, October 18, 2017 5:36 PM
To: Latvala, Petri <[email protected]>
Cc: [email protected]
Subject: Re: [Intel-gfx] [PATCH i-g-t] kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

Quoted 10 lines · Petri Latvala
Quoted 6 lines · Imre Deak

The CI software watchdog (owatch) will timeout if the test doesn't output anything for a long time on standard out or error. At least the plane-all-modeset-transition and plane-all-modeset-transition-fences
subtests run without any output longer than the watchdog timeout, so output some more progress info.

No, owatch is wrapping piglit, and pings the watchdog if _piglit_ prints anything. Which it does on start/exit of a test.

Quoted 6 lines

tbh this sounds like owatch being dense and it shouldn't try to reboot this quickly. What's the current owatch timeout?

Aside: What exactly does owatch give us? I thought jenkins also watches machines and reboots them using the ac switch ... And owatch provides spurious reboots?

Daniel,
Owatch gives us the knowledge that is was a test that took too long. I.e. we will know that it was not a system hang.
We also know that the NMI watchdog didn't trigger.
I believe this is extremely useful information when you are starting to debug the issue

Quoted 4 lines

Imre if you believe that owatch is preventing you from getting information to debug why these test are taking so extremely long time, it would be easy to increase the timeout or even do runs without it being enabled.

It would be useful to have a clear indication if the test really hang or it just took too long. Right now this isn't obvious without going through the logs.

--Imre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#9 lore raw

✓ Fi.CI.BAT: success for kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

In reply to Imre Deak
== Series Details ==

== Series Details ==

Series: kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout
URL : https://patchwork.freedesktop.org/series/32206/
State : success

== Summary ==

IGT patchset tested on top of latest successful build 9a3f6f59d4c208219f7d3178106bdef628ff68d0 tests/intel-ci: Remove fast-feedback-simulation.testlist

with latest DRM-Tip kernel build CI_DRM_3258
8b18ad103908 drm-tip: 2017y-10m-18d-08h-27m-31s UTC integration manifest

No testlist changes.

Test chamelium:

        Subgroup dp-crc-fast:
                fail       -> PASS       (fi-kbl-7500u) fdo#102514

fdo#102514 https://bugs.freedesktop.org/show_bug.cgi?id=102514

fi-bdw-5557u     total:289  pass:268  dwarn:0   dfail:0   fail:0   skip:21  time:442s
fi-blb-e6850     total:289  pass:223  dwarn:1   dfail:0   fail:0   skip:65  time:371s
fi-bsw-n3050     total:289  pass:243  dwarn:0   dfail:0   fail:0   skip:46  time:531s
fi-bwr-2160      total:289  pass:183  dwarn:0   dfail:0   fail:0   skip:106 time:270s
fi-bxt-dsi       total:289  pass:259  dwarn:0   dfail:0   fail:0   skip:30  time:504s
fi-bxt-j4205     total:289  pass:260  dwarn:0   dfail:0   fail:0   skip:29  time:498s
fi-byt-j1900     total:289  pass:253  dwarn:1   dfail:0   fail:0   skip:35  time:499s
fi-byt-n2820     total:289  pass:249  dwarn:1   dfail:0   fail:0   skip:39  time:483s
fi-elk-e7500     total:289  pass:229  dwarn:0   dfail:0   fail:0   skip:60  time:424s
fi-gdg-551       total:289  pass:178  dwarn:1   dfail:0   fail:1   skip:109 time:252s
fi-glk-1         total:289  pass:261  dwarn:0   dfail:0   fail:0   skip:28  time:582s
fi-hsw-4770      total:289  pass:262  dwarn:0   dfail:0   fail:0   skip:27  time:445s
fi-hsw-4770r     total:289  pass:262  dwarn:0   dfail:0   fail:0   skip:27  time:429s
fi-ilk-650       total:289  pass:228  dwarn:0   dfail:0   fail:0   skip:61  time:438s
fi-ivb-3520m     total:289  pass:260  dwarn:0   dfail:0   fail:0   skip:29  time:487s
fi-ivb-3770      total:289  pass:260  dwarn:0   dfail:0   fail:0   skip:29  time:463s
fi-kbl-7500u     total:289  pass:264  dwarn:1   dfail:0   fail:0   skip:24  time:491s
fi-kbl-7560u     total:289  pass:270  dwarn:0   dfail:0   fail:0   skip:19  time:577s
fi-kbl-7567u     total:289  pass:269  dwarn:0   dfail:0   fail:0   skip:20  time:477s
fi-kbl-r         total:289  pass:262  dwarn:0   dfail:0   fail:0   skip:27  time:584s
fi-pnv-d510      total:289  pass:222  dwarn:1   dfail:0   fail:0   skip:66  time:551s
fi-skl-6260u     total:289  pass:269  dwarn:0   dfail:0   fail:0   skip:20  time:451s
fi-skl-6700hq    total:289  pass:263  dwarn:0   dfail:0   fail:0   skip:26  time:643s
fi-skl-6700k     total:289  pass:265  dwarn:0   dfail:0   fail:0   skip:24  time:515s
fi-skl-6770hq    total:289  pass:269  dwarn:0   dfail:0   fail:0   skip:20  time:504s
fi-skl-gvtdvm    total:289  pass:266  dwarn:0   dfail:0   fail:0   skip:23  time:459s
fi-snb-2520m     total:289  pass:250  dwarn:0   dfail:0   fail:0   skip:39  time:571s
fi-snb-2600      total:289  pass:249  dwarn:0   dfail:0   fail:0   skip:40  time:427s

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_375/
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#10 lore raw

✓ Fi.CI.IGT: success for kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout

In reply to Imre Deak
== Series Details ==

== Series Details ==

Series: kms_atomic_transition: Output more finegrained progress info to avoid CI watchdog timeout
URL : https://patchwork.freedesktop.org/series/32206/
State : success

== Summary ==

Test kms_setmode:

        Subgroup basic:
                pass       -> FAIL       (shard-hsw) fdo#99912
Test perf:
        Subgroup polling:
                fail       -> PASS       (shard-hsw) fdo#102252

fdo#99912 https://bugs.freedesktop.org/show_bug.cgi?id=99912
fdo#102252 https://bugs.freedesktop.org/show_bug.cgi?id=102252

shard-hsw total:2540 pass:1430 dwarn:0 dfail:0 fail:9 skip:1101 time:9226s

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_375/shards.html
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx