Skip to contentSkip to conversation

runner: Avoid null characters in results.json

PATCH In review Petri Latvala 4 participants 5 messages Sep 26, 2018 – Sep 27, 2018

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 runner: Avoid null characters in results.json

CI pipeline (namely, cibuglog) doesn't cope well with strings that have \0 in them. If null characters appear in output files, pretend the output stops at the first such character. Well behaving tests should not print them anyway.

The case in CI happened due to some hang/crash/explosion/solar flare that corrupted the output of a test.

Cc Arkadiusz Hiler <[email protected]>
Cc Tomi Sarvela <[email protected]>
Cc Martin Peres <[email protected]>
---
 runner/resultgen.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
runner/resultgen.c +9 -1
index d34b52db..e8a60083 100644
@@ -313,7 +313,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtests *subtests,
 			     struct json_object *tests)
 {
-	char *buf, *bufend;
+	char *buf, *bufend, *nullchr;
 	struct stat statbuf;
 	char piglit_name[256];
 	char *igt_version = NULL;
@@ -332,6 +332,14 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		buf = NULL;
 	}
 
+	/*
+	 * Avoid null characters: Just pretend the output stops at the
+	 * first such character, if any.
+	 */
+	if ((nullchr = memchr(buf, '\0', statbuf.st_size)) != NULL) {
+		statbuf.st_size = nullchr - buf;
+	}
+
 	bufend = buf + statbuf.st_size;
 
 	igt_version = find_line_starting_with(buf, IGT_VERSIONSTRING, bufend);
Signature · 6 lines
2.18.0

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
Acked-by on v1 1/1
#1 lore raw
Quoted 5 lines · Petri Latvala

CI pipeline (namely, cibuglog) doesn't cope well with strings that have \0 in them. If null characters appear in output files, pretend the output stops at the first such character. Well behaving tests should not print them anyway.

Quoted 7 lines

The case in CI happened due to some hang/crash/explosion/solar flare that corrupted the output of a test.

Cc Arkadiusz Hiler <[email protected]>
Cc Tomi Sarvela <[email protected]>
Cc Martin Peres <[email protected]>
Quoted 23 lines
---
 runner/resultgen.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
runner/resultgen.c +2 -1
index d34b52db..e8a60083 100644
@@ -313,7 +313,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 			     struct subtests *subtests,
 			     struct json_object *tests)
 {
-	char *buf, *bufend;
+	char *buf, *bufend, *nullchr;
 	struct stat statbuf;
 	char piglit_name[256];
 	char *igt_version = NULL;
@@ -332,6 +332,14 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
 		buf = NULL;
 	}
 
+	/*

runner/resultgen.c static bool fill_from_output(int fd, const char *binary, const char *key, · line 345

Quoted 10 lines
runner/resultgen.c +7 -0
+	 * Avoid null characters: Just pretend the output stops at the
+	 * first such character, if any.
+	 */
+	if ((nullchr = memchr(buf, '\0', statbuf.st_size)) != NULL) {
+		statbuf.st_size = nullchr - buf;
+	}
+
 	bufend = buf + statbuf.st_size;
 
 	igt_version = find_line_starting_with(buf, IGT_VERSIONSTRING, bufend);

_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

Acked-by
#2 lore raw

Re: [igt-dev] (no subject)

Quoted 9 lines · Martin Peres
Quoted 8 lines · Petri Latvala

CI pipeline (namely, cibuglog) doesn't cope well with strings that have \0 in them. If null characters appear in output files, pretend the output stops at the first such character. Well behaving tests should not print them anyway.

The case in CI happened due to some hang/crash/explosion/solar flare that corrupted the output of a test.

Quoted 6 lines
Quoted 4 lines
Cc Arkadiusz Hiler <[email protected]>
Cc Tomi Sarvela <[email protected]>
Cc Martin Peres <[email protected]>

The original reason was very probably a hang/panic/reboot before flushed fd was written to the disk. Result was about one lines worth of NULLs (which, in itself, shows that the logwriter is significantly more careful than piglits one).

If the expectation is that the out/err files are text, I think runner can safely discard the rest from any indication of data corruption.

Quoted 32 lines
Quoted 32 lines
---
  runner/resultgen.c | 10 +++++++++-
  1 file changed, 9 insertions(+), 1 deletion(-)
runner/resultgen.c +9 -1
index d34b52db..e8a60083 100644
@@ -313,7 +313,7 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
  			     struct subtests *subtests,
  			     struct json_object *tests)
  {
-	char *buf, *bufend;
+	char *buf, *bufend, *nullchr;
  	struct stat statbuf;
  	char piglit_name[256];
  	char *igt_version = NULL;
@@ -332,6 +332,14 @@ static bool fill_from_output(int fd, const char *binary, const char *key,
  		buf = NULL;
  	}
 
+	/*
+	 * Avoid null characters: Just pretend the output stops at the
+	 * first such character, if any.
+	 */
+	if ((nullchr = memchr(buf, '\0', statbuf.st_size)) != NULL) {
+		statbuf.st_size = nullchr - buf;
+	}
+
  	bufend = buf + statbuf.st_size;
 
Quoted 3 lines

runner/resultgen.c static bool fill_from_output(int fd, const char *binary, const char *key, · line 345

Quoted 2 lines
runner/resultgen.c +0 -0
  	igt_version = find_line_starting_with(buf, IGT_VERSIONSTRING, bufend);

Tomi

Signature · 5 lines
Intel Finland Oy - BIC 0357606-4 - Westendinkatu 7, 02160 Espoo
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev
#3 lore raw

[igt-dev] ✓ Fi.CI.BAT: success for series starting with [i-g-t,1/1] runner: Avoid null characters in results.json

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/1] runner: Avoid null characters in results.json
URL : https://patchwork.freedesktop.org/series/50214/
State : success

== Summary ==

= CI Bug Log - changes from CI_DRM_4874 -> IGTPW_1875 =

== Summary - SUCCESS ==

No regressions found.

External URL: https://patchwork.freedesktop.org/api/1.0/series/50214/revisions/1/mbox/

== Known issues ==

Here are the changes found in IGTPW_1875 that come from known issues:

=== IGT changes ===

    ==== Issues hit ====

    igt@debugfs_test@read_all_entries:
      fi-icl-u:           PASS -> DMESG-WARN (fdo#107724, fdo#108070)

    igt@drv_selftest@live_evict:
      fi-bsw-kefka:       PASS -> DMESG-WARN (fdo#107709)

    igt@gem_exec_suspend@basic-s3:
      fi-bdw-samus:       PASS -> INCOMPLETE (fdo#107773)

    igt@kms_flip@basic-flip-vs-dpms:
      fi-skl-6700hq:      NOTRUN -> DMESG-WARN (fdo#105998)

    igt@kms_pipe_crc_basic@nonblocking-crc-pipe-a:
      fi-byt-clapper:     PASS -> FAIL (fdo#107362)

    igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a:
      fi-byt-clapper:     PASS -> FAIL (fdo#103191, fdo#107362)

    igt@kms_psr@primary_page_flip:
      fi-kbl-7560u:       PASS -> FAIL (fdo#107336)
      fi-kbl-r:           PASS -> FAIL (fdo#107336)

    
    ==== Possible fixes ====

    igt@drv_module_reload@basic-reload:
      fi-blb-e6850:       INCOMPLETE (fdo#107718) -> PASS

    igt@kms_frontbuffer_tracking@basic:
      fi-byt-clapper:     FAIL (fdo#103167) -> PASS

    igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b:
      fi-cfl-8109u:       INCOMPLETE (fdo#106070) -> PASS

    igt@kms_psr@primary_page_flip:
      fi-cnl-u:           FAIL (fdo#107336) -> PASS

    
    ==== Warnings ====

    igt@kms_pipe_crc_basic@suspend-read-crc-pipe-b:
      fi-icl-u:           DMESG-WARN (fdo#108070) -> INCOMPLETE (fdo#107713)

fdo#103167 https://bugs.freedesktop.org/show_bug.cgi?id=103167
fdo#103191 https://bugs.freedesktop.org/show_bug.cgi?id=103191
fdo#105998 https://bugs.freedesktop.org/show_bug.cgi?id=105998
fdo#106070 https://bugs.freedesktop.org/show_bug.cgi?id=106070
fdo#107336 https://bugs.freedesktop.org/show_bug.cgi?id=107336
fdo#107362 https://bugs.freedesktop.org/show_bug.cgi?id=107362
fdo#107709 https://bugs.freedesktop.org/show_bug.cgi?id=107709
fdo#107713 https://bugs.freedesktop.org/show_bug.cgi?id=107713
fdo#107718 https://bugs.freedesktop.org/show_bug.cgi?id=107718
fdo#107724 https://bugs.freedesktop.org/show_bug.cgi?id=107724
fdo#107773 https://bugs.freedesktop.org/show_bug.cgi?id=107773
fdo#108070 https://bugs.freedesktop.org/show_bug.cgi?id=108070

== Participating hosts (48 -> 42) ==

Additional (1): fi-skl-6700hq
Missing (7): fi-ilk-m540 fi-hsw-4200u fi-bdw-gvtdvm fi-byt-squawks fi-icl-u2 fi-bsw-cyan fi-ctg-p8600

== Build changes ==

* IGT: IGT_4649 -> IGTPW_1875

CI_DRM_4874: 7bb42432ef1cc63b4a6a223e19455f6c4f9a1538 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1875: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1875/
IGT_4649: 19b0c74d20d9b53d4c82be14af0909a3b6846010 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1875/issues.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev

#4 lore raw

[igt-dev] ✓ Fi.CI.IGT: success for series starting with [i-g-t,1/1] runner: Avoid null characters in results.json

In reply to Petri Latvala
== Series Details ==

== Series Details ==

Series: series starting with [i-g-t,1/1] runner: Avoid null characters in results.json
URL : https://patchwork.freedesktop.org/series/50214/
State : success

== Summary ==

= CI Bug Log - changes from IGT_4649_full -> IGTPW_1875_full =

== Summary - WARNING ==

Minor unknown changes coming with IGTPW_1875_full need to be verified manually.

If you think the reported changes have nothing to do with the changes introduced in IGTPW_1875_full, please notify your bug team to allow them to document this new failure mode, which will reduce false positives in CI.

External URL: https://patchwork.freedesktop.org/api/1.0/series/50214/revisions/1/mbox/

== Possible new issues ==

Here are the unknown changes that may have been introduced in IGTPW_1875_full:

=== IGT changes ===

    ==== Warnings ====

    igt@perf_pmu@rc6:
      shard-kbl:          PASS -> SKIP

== Known issues ==

Here are the changes found in IGTPW_1875_full that come from known issues:

=== IGT changes ===

    ==== Issues hit ====

    igt@kms_frontbuffer_tracking@fbc-2p-scndscrn-spr-indfb-draw-mmap-gtt:
      shard-glk:          PASS -> FAIL (fdo#103167)

    igt@kms_plane@plane-panning-bottom-right-suspend-pipe-c-planes:
      shard-kbl:          PASS -> INCOMPLETE (fdo#103665)

    igt@kms_setmode@basic:
      shard-kbl:          PASS -> FAIL (fdo#99912)

    
    ==== Possible fixes ====

    igt@kms_cursor_crc@cursor-256x256-suspend:
      shard-kbl:          INCOMPLETE (fdo#103665) -> PASS

    igt@kms_cursor_legacy@pipe-c-forked-move:
      shard-apl:          INCOMPLETE (fdo#103927) -> PASS

    igt@kms_universal_plane@universal-plane-gen9-features-pipe-a:
      shard-kbl:          DMESG-WARN (fdo#105602, fdo#103558) -> PASS +14

fdo#103167 https://bugs.freedesktop.org/show_bug.cgi?id=103167
fdo#103558 https://bugs.freedesktop.org/show_bug.cgi?id=103558
fdo#103665 https://bugs.freedesktop.org/show_bug.cgi?id=103665
fdo#103927 https://bugs.freedesktop.org/show_bug.cgi?id=103927
fdo#105602 https://bugs.freedesktop.org/show_bug.cgi?id=105602
fdo#99912 https://bugs.freedesktop.org/show_bug.cgi?id=99912

== Participating hosts (5 -> 5) ==

No changes in participating hosts

== Build changes ==

* IGT: IGT_4649 -> IGTPW_1875
* Linux: CI_DRM_4859 -> CI_DRM_4874

CI_DRM_4859: 841654aee134e50b21bf8f2aab17da8d0afd14c4 @ git://anongit.freedesktop.org/gfx-ci/linux
CI_DRM_4874: 7bb42432ef1cc63b4a6a223e19455f6c4f9a1538 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_1875: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1875/
IGT_4649: 19b0c74d20d9b53d4c82be14af0909a3b6846010 @ git://anongit.freedesktop.org/xorg/app/intel-gpu-tools

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_1875/shards.html
_______________________________________________
igt-dev mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/igt-dev