Skip to contentSkip to conversation

runner: Add time_elapsed field to results

PATCH In review Petri Latvala 3 participants 5 messages Sep 27, 2018 – Oct 3, 2018

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 runner: Add time_elapsed field to results

When starting a test run, drop a timestamp file. Do the same when ending a run. Slap those timestamps directly into the time_elapsed field in results.json.

Using timestamps instead of measuring actual elapsed time goes against the naming of the field, but the name is chosen by piglit. Even though piglit itself uses timestamps.

Corner cases:

On incomplete test runs, the end timestamp will be missing. The time_elapsed field will only have the start timestamp. This matches piglit behaviour exactly.

On incomplete but resumed test runs, the end timestamp will be the time when the resume finishes. Piglit doesn't do this, and instead leaves the end timestamp missing. Discussing which behaviour is better is left as an exercise to the readers.

Cc Arkadiusz Hiler <[email protected]>
Cc Tomi Sarvela <[email protected]>
---
 runner/executor.c  | 33 ++++++++++++++++++++++++++++++---
 runner/resultgen.c | 27 +++++++++++++++++++++------
 2 files changed, 51 insertions(+), 9 deletions(-)
runner/executor.c +30 -3
index d0539aa1..4552b02c 100644
@@ -9,6 +9,7 @@
 #include <sys/select.h>
 #include <sys/signalfd.h>
 #include <sys/stat.h>
+#include <sys/time.h>
 #include <sys/types.h>
 #include <sys/utsname.h>
 #include <sys/wait.h>
@@ -860,7 +861,9 @@ static bool clear_old_results(char *path)
 		return false;
 	}
 
-	if (unlinkat(dirfd, "uname.txt", 0) && errno != ENOENT) {
+	if (remove_file(dirfd, "uname.txt") ||
+	    remove_file(dirfd, "starttime.txt") ||
+	    remove_file(dirfd, "endtime.txt")) {
 		close(dirfd);
 		fprintf(stderr, "Error clearing old results: %s\n", strerror(errno));
 		return false;
@@ -892,6 +895,15 @@ static bool clear_old_results(char *path)
 	return true;
 }
 
+static double timeofday_double()
+{
+	struct timeval tv;
+
+	if (!gettimeofday(&tv, NULL))
+		return tv.tv_sec + tv.tv_usec / 1000000.0;
+	return 0.0;
+}
+
 bool initialize_execute_state_from_resume(int dirfd,
 					  struct execute_state *state,
 					  struct settings *settings,
@@ -973,7 +985,7 @@ bool execute(struct execute_state *state,
 	     struct job_list *job_list)
 {
 	struct utsname unamebuf;
-	int resdirfd, testdirfd, unamefd;
+	int resdirfd, testdirfd, unamefd, timefd;
 
 	if ((resdirfd = open(settings->results_path, O_DIRECTORY | O_RDONLY)) < 0) {
 		/* Initialize state should have done this */
@@ -991,13 +1003,23 @@ bool execute(struct execute_state *state,
 
 	/* TODO: On resume, don't rewrite, verify that content matches current instead */
 	if ((unamefd = openat(resdirfd, "uname.txt", O_CREAT | O_WRONLY | O_TRUNC, 0666)) < 0) {
-		fprintf(stderr, "Error: Failure creating opening uname.txt: %s\n",
+		fprintf(stderr, "Error: Failure opening uname.txt: %s\n",
 			strerror(errno));
 		close(testdirfd);
 		close(resdirfd);
 		return false;
 	}
 
+	if ((timefd = openat(resdirfd, "starttime.txt", O_CREAT | O_WRONLY | O_EXCL, 0666)) >= 0) {
+		/*
+		 * Ignore failure to open. If this is a resume, we
+		 * don't want to overwrite. For other errors, we
+		 * ignore the start time.
+		 */
+		dprintf(timefd, "%f\n", timeofday_double());
+		close(timefd);
+	}
+
 	init_watchdogs(settings);
 
 	if (!uname(&unamebuf)) {
@@ -1031,6 +1053,11 @@ bool execute(struct execute_state *state,
 		}
 	}
 
+	if ((timefd = openat(resdirfd, "endtime.txt", O_CREAT | O_WRONLY | O_EXCL, 0666)) >= 0) {
+		dprintf(timefd, "%f\n", timeofday_double());
+		close(timefd);
+	}
+
 	close(testdirfd);
 	close(resdirfd);
 	close_watchdogs(settings);
runner/resultgen.c +21 -6
index e8a60083..11eff9d3 100644
@@ -987,9 +987,9 @@ bool generate_results(int dirfd)
 {
 	struct settings settings;
 	struct job_list job_list;
-	struct json_object *obj;
+	struct json_object *obj, *elapsed;
 	struct results results;
-	int resultsfd, testdirfd, unamefd;
+	int resultsfd, testdirfd, fd;
 	const char *json_string;
 	size_t i;
 
@@ -1020,17 +1020,33 @@ bool generate_results(int dirfd)
 			       json_object_new_string(settings.name) :
 			       json_object_new_string(""));
 
-	if ((unamefd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0) {
+	if ((fd = openat(dirfd, "uname.txt", O_RDONLY)) >= 0) {
 		char buf[128];
-		ssize_t r = read(unamefd, buf, 128);
+		ssize_t r = read(fd, buf, sizeof(buf));
 
 		if (r > 0 && buf[r - 1] == '\n')
 			r--;
 
 		json_object_object_add(obj, "uname",
 				       json_object_new_string_len(buf, r));
-		close(unamefd);
+		close(fd);
+	}
+
+	elapsed = json_object_new_object();
+	json_object_object_add(elapsed, "__type__", json_object_new_string("TimeAttribute"));
+	if ((fd = openat(dirfd, "starttime.txt", O_RDONLY)) >= 0) {
+		char buf[128] = {};
+		read(fd, buf, sizeof(buf));
+		json_object_object_add(elapsed, "start", json_object_new_double(atof(buf)));
+		close(fd);
+	}
+	if ((fd = openat(dirfd, "endtime.txt", O_RDONLY)) >= 0) {
+		char buf[128] = {};
+		read(fd, buf, sizeof(buf));
+		json_object_object_add(elapsed, "end", json_object_new_double(atof(buf)));
+		close(fd);
 	}
+	json_object_object_add(obj, "time_elapsed", elapsed);
 
 	create_result_root_nodes(obj, &results);
 
@@ -1045,7 +1061,6 @@ bool generate_results(int dirfd)
 	 *
 	 * - lspci
 	 * - options
-	 * - time_elapsed
 	 */
 
 	for (i = 0; i < job_list.size; i++) {
Signature · 6 lines
2.18.0

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#1 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for series starting with [i-g-t,1/1] runner: Add time_elapsed field to results

== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/1] runner: Add time_elapsed field to results
URL : https://patchwork.freedesktop.org/series/50274/
State : success

== Summary ==

= CI Bug Log - changes from CI_DRM_4899 -> IGTPW_1878 =

== Summary - SUCCESS ==

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/50274/revisions/1/mbox/

== Known issues ==

Here are the changes found in IGTPW_1878 that come from known issues:

=== IGT changes ===

    ==== Issues hit ====

    igt@gem_exec_suspend@basic-s3:
      fi-bdw-samus:       PASS -> INCOMPLETE (fdo#107773)

    igt@kms_frontbuffer_tracking@basic:
      fi-byt-clapper:     PASS -> FAIL (fdo#103167)

    
    ==== Possible fixes ====

    igt@kms_psr@primary_page_flip:
      fi-icl-u:           FAIL (fdo#107336) -> PASS
      fi-whl-u:           FAIL (fdo#107336) -> PASS
      fi-kbl-r:           FAIL (fdo#107336) -> PASS

fdo#103167 https://bugs.freedesktop.org/show_bug.cgi?id=103167
fdo#107336 https://bugs.freedesktop.org/show_bug.cgi?id=107336
fdo#107773 https://bugs.freedesktop.org/show_bug.cgi?id=107773

== Participating hosts (45 -> 39) ==

Additional (1): fi-skl-6700hq
Missing (7): fi-ilk-m540 fi-byt-squawks fi-icl-u2 fi-bsw-cyan fi-ctg-p8600 fi-pnv-d510 fi-kbl-7560u

== Build changes ==

* IGT: IGT_4654 -> IGTPW_1878

CI_DRM_4899: fd4ca44f6b056b12a76ebea32b09bac8501dc9a2 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1878: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1878/
IGT_4654: 036b60a388685dcd38c33dc4b7ba55130a2d200d @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1878/issues.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#2 lore raw

[igt-dev] ✓ Fi.CI.IGT: success for series starting with [i-g-t,1/1] runner: Add time_elapsed field to results

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/1] runner: Add time_elapsed field to results
URL : https://patchwork.freedesktop.org/series/50274/
State : success

== Summary ==

= CI Bug Log - changes from IGT_4654_full -> IGTPW_1878_full =

== Summary - SUCCESS ==

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/50274/revisions/1/mbox/

== Known issues ==

Here are the changes found in IGTPW_1878_full that come from known issues:

=== IGT changes ===

    ==== Issues hit ====

    igt@gem_ppgtt@blt-vs-render-ctxn:
      shard-kbl:          PASS -> INCOMPLETE (fdo#106023, fdo#103665)

    igt@kms_available_modes_crc@available_mode_test_crc:
      shard-snb:          NOTRUN -> FAIL (fdo#106641)

    igt@kms_busy@extended-modeset-hang-newfb-with-reset-render-a:
      shard-snb:          NOTRUN -> DMESG-WARN (fdo#107956)

    igt@kms_cursor_legacy@cursorb-vs-flipb-toggle:
      shard-glk:          PASS -> DMESG-WARN (fdo#106538, fdo#105763)

    igt@kms_draw_crc@draw-method-rgb565-mmap-gtt-xtiled:
      shard-glk:          PASS -> FAIL (fdo#103184) +1

    igt@kms_flip@single-buffer-flip-vs-dpms-off-vs-modeset:
      shard-glk:          PASS -> INCOMPLETE (fdo#103359, k.org#198133)

    igt@kms_frontbuffer_tracking@fbc-1p-primscrn-spr-indfb-draw-mmap-gtt:
      shard-glk:          PASS -> FAIL (fdo#103167) +1

    igt@kms_pipe_crc_basic@nonblocking-crc-pipe-c:
      shard-kbl:          NOTRUN -> DMESG-WARN (fdo#105602, fdo#103558)

    igt@kms_setmode@basic:
      shard-kbl:          PASS -> FAIL (fdo#99912)

    igt@kms_vblank@pipe-c-wait-busy:
      shard-kbl:          PASS -> DMESG-WARN (fdo#105602, fdo#103558) +7

    
    ==== Possible fixes ====

    igt@gem_userptr_blits@readonly-unsync:
      shard-glk:          INCOMPLETE (fdo#103359, k.org#198133) -> PASS

    igt@kms_cursor_legacy@cursora-vs-flipa-toggle:
      shard-glk:          DMESG-WARN (fdo#106538, fdo#105763) -> PASS

    igt@kms_flip@flip-vs-expired-vblank-interruptible:
      shard-glk:          FAIL (fdo#105363) -> PASS

    igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-gtt:
      shard-glk:          FAIL (fdo#103167) -> PASS +1

    igt@pm_rps@reset:
      shard-glk:          FAIL (fdo#102250) -> PASS

fdo#102250 https://bugs.freedesktop.org/show_bug.cgi?id=102250
fdo#103167 https://bugs.freedesktop.org/show_bug.cgi?id=103167
fdo#103184 https://bugs.freedesktop.org/show_bug.cgi?id=103184
fdo#103359 https://bugs.freedesktop.org/show_bug.cgi?id=103359
fdo#103558 https://bugs.freedesktop.org/show_bug.cgi?id=103558
fdo#103665 https://bugs.freedesktop.org/show_bug.cgi?id=103665
fdo#105363 https://bugs.freedesktop.org/show_bug.cgi?id=105363
fdo#105602 https://bugs.freedesktop.org/show_bug.cgi?id=105602
fdo#105763 https://bugs.freedesktop.org/show_bug.cgi?id=105763
fdo#106023 https://bugs.freedesktop.org/show_bug.cgi?id=106023
fdo#106538 https://bugs.freedesktop.org/show_bug.cgi?id=106538
fdo#106641 https://bugs.freedesktop.org/show_bug.cgi?id=106641
fdo#107956 https://bugs.freedesktop.org/show_bug.cgi?id=107956
fdo#99912 https://bugs.freedesktop.org/show_bug.cgi?id=99912
k.org#198133 https://bugzilla.kernel.org/show_bug.cgi?id=198133

== Participating hosts (6 -> 5) ==

Missing (1): shard-skl

== Build changes ==

* IGT: IGT_4654 -> IGTPW_1878
* Linux: CI_DRM_4897 -> CI_DRM_4899

CI_DRM_4897: e9c4ad6fd03bd0e28e4534b841d62407ac6003af @ git://anongit.freedesktop.org/gfx-ci/linux
CI_DRM_4899: fd4ca44f6b056b12a76ebea32b09bac8501dc9a2 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1878: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1878/
IGT_4654: 036b60a388685dcd38c33dc4b7ba55130a2d200d @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1878/shards.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Reviewed-by on v1 1/1
#3 lore raw
In reply to Petri Latvala
Quoted 14 lines · Petri Latvala

When starting a test run, drop a timestamp file. Do the same when ending a run. Slap those timestamps directly into the time_elapsed field in results.json.

Using timestamps instead of measuring actual elapsed time goes against the naming of the field, but the name is chosen by piglit. Even though piglit itself uses timestamps.

Corner cases:

On incomplete test runs, the end timestamp will be missing. The time_elapsed field will only have the start timestamp. This matches piglit behaviour exactly.

Quoted 8 lines

On incomplete but resumed test runs, the end timestamp will be the time when the resume finishes. Piglit doesn't do this, and instead leaves the end timestamp missing. Discussing which behaviour is better is left as an exercise to the readers.

Cc Arkadiusz Hiler <[email protected]>
Cc Tomi Sarvela <[email protected]>

Once this gets merged we will have to update IGT on the the CI host.

Signature · 6 lines
Cheers,
Arek
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#4 lore raw
Quoted 20 lines · Arkadiusz Hiler
Quoted 19 lines · Petri Latvala

When starting a test run, drop a timestamp file. Do the same when ending a run. Slap those timestamps directly into the time_elapsed field in results.json.

Using timestamps instead of measuring actual elapsed time goes against the naming of the field, but the name is chosen by piglit. Even though piglit itself uses timestamps.

Corner cases:

On incomplete test runs, the end timestamp will be missing. The time_elapsed field will only have the start timestamp. This matches piglit behaviour exactly.

On incomplete but resumed test runs, the end timestamp will be the time when the resume finishes. Piglit doesn't do this, and instead leaves the end timestamp missing. Discussing which behaviour is better is left as an exercise to the readers.

Quoted 7 lines
Quoted 3 lines
Cc Arkadiusz Hiler <[email protected]>
Cc Tomi Sarvela <[email protected]>

Once this gets merged we will have to update IGT on the the CI host.

Merged now, thanks.

Signature · 5 lines
Petri Latvala
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev