Skip to contentSkip to conversation

benchmarks/gem_blt: fix baseline estimation

PATCH Superseded Mauro Carvalho Chehab 3 participants 3 messages May 19, 2022

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 benchmarks/gem_blt: fix baseline estimation

From: Mauro Carvalho Chehab <[email protected]>

This test is expected to run under a certain time defined by a command line parameter (by default, 2s).

In order to achive the specified time, the baseline() function tries to estimate the value of a counter that will get the minimal amount of interaction to wait for > 0.1s.

However, currently, the baseline estimation is broken, returning a too high number, as it is measuring the memcpy() time, instead of actually measuring each loop's interaction.

Due to that, a default test without passing any command line parameter would take more than 1.5 years to output the first benchmark data!

Fix it by using the same logic that it is inside run() at the baseline time estimation.

On ADL-P, before this patch, baseline would return 399242693. After it, it now return 20 - meaning an interval of about 120 ms, which should be good enough to ensure that the tests won't take too long.

---
 benchmarks/gem_blt.c | 18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)
benchmarks/gem_blt.c +5 -13
index 424ce8e75913..bd8264b4e896 100644
@@ -57,29 +57,21 @@ elapsed(const struct timespec *start, const struct timespec *end)
 	return (end->tv_sec - start->tv_sec) + 1e-9*(end->tv_nsec - start->tv_nsec);
 }
 
-static int baseline(uint64_t bytes, int milliseconds)
+static int baseline(int fd, struct drm_i915_gem_execbuffer2 execbuf, int milliseconds)
 {
 	struct timespec start, end;
-	const int size = 64*1024*1024;
 	int count = 0;
-	void *mem;
-
-	mem = malloc(size);
-	if (mem == NULL)
-		return 1;
 
 	clock_gettime(CLOCK_MONOTONIC, &start);
 	do {
-		memset(mem, count, size);
+		gem_execbuf(fd, &execbuf);
 		count++;
 		clock_gettime(CLOCK_MONOTONIC, &end);
-		if (elapsed(&start, &end) > 0.1)
+		if (elapsed(&start, &end) > (milliseconds / 1000.))
 			break;
 	} while (1);
 
-	free(mem);
-
-	return ceil(1e-3*milliseconds/elapsed(&start, &end) * count * size / bytes);
+	return count;
 }
 
 static int gem_linear_blt(int fd,
@@ -260,7 +252,7 @@ static int run(int object, int batch, int time, int reps, int ncpus, unsigned fl
 		execbuf.flags |= I915_EXEC_NO_RELOC;
 
 	/* Guess how many loops we need for 0.1s */
-	count = baseline((uint64_t)object * batch, 100) / ncpus;
+	count = baseline(fd, execbuf, 100) / ncpus;
 	if (flags & SYNC) {
 		time *= count / 2;
 		count = 1;
Signature · 1 line
2.36.1
#1 lore raw

[igt-dev] ✗ Fi.CI.BAT: failure for benchmarks/gem_blt: fix baseline estimation

== Series Details ==

== Series Details ==

Series: benchmarks/gem_blt: fix baseline estimation
URL : https://patchwork.freedesktop.org/series/104163/
State : failure

== Summary ==

CI Bug Log - changes from IGT_6481 -> IGTPW_7141
====================================================

Summary
-------

**FAILURE**

Serious unknown changes coming with IGTPW_7141 absolutely need to be verified manually.

If you think the reported changes have nothing to do with the changes introduced in IGTPW_7141, please notify your bug team to allow them to document this new failure mode, which will reduce false positives in CI.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/index.html

Participating hosts (45 -> 45)
------------------------------

Additional (2): fi-icl-u2 fi-pnv-d510
Missing (2): bat-dg2-8 bat-dg2-9

Possible new issues
-------------------

Here are the unknown changes that may have been introduced in IGTPW_7141:

### IGT changes ###

#### Possible regressions ####

* igt@i915_selftest@live@workarounds:
- fi-bdw-5557u: [PASS][1] -> [INCOMPLETE][2]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-bdw-5557u/igt@i915_selftest@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-bdw-5557u/igt@i915_selftest@[email protected]

* igt@i915_suspend@basic-s3-without-i915:
- fi-cfl-8109u: [PASS][3] -> [INCOMPLETE][4]
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-cfl-8109u/igt@[email protected]
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-cfl-8109u/igt@[email protected]

Known issues
------------

Here are the changes found in IGTPW_7141 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_huc_copy@huc-copy:
- fi-icl-u2: NOTRUN -> [SKIP][5] ([i915#2190])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-icl-u2/igt@[email protected]

* igt@gem_lmem_swapping@parallel-random-engines:
- fi-icl-u2: NOTRUN -> [SKIP][6] ([i915#4613]) +3 similar issues
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-icl-u2/igt@[email protected]

* igt@i915_pm_rpm@basic-rte:
- fi-cfl-8109u: [PASS][7] -> [DMESG-WARN][8] ([i915#1888] / [i915#62])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-cfl-8109u/igt@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-cfl-8109u/igt@[email protected]

* igt@i915_pm_rpm@module-reload:
- fi-cfl-8109u: [PASS][9] -> [DMESG-FAIL][10] ([i915#62]) +1 similar issue
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-cfl-8109u/igt@[email protected]
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-cfl-8109u/igt@[email protected]

* igt@i915_pm_rps@basic-api:
- fi-glk-j4005: [PASS][11] -> [DMESG-WARN][12] ([i915#118])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-glk-j4005/igt@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-glk-j4005/igt@[email protected]

* igt@i915_selftest@live@hangcheck:
- fi-hsw-g3258: [PASS][13] -> [INCOMPLETE][14] ([i915#4785])
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-hsw-g3258/igt@i915_selftest@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-hsw-g3258/igt@i915_selftest@[email protected]

* igt@i915_selftest@live@requests:
- fi-pnv-d510: NOTRUN -> [DMESG-FAIL][15] ([i915#4528])
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-pnv-d510/igt@i915_selftest@[email protected]

* igt@i915_suspend@basic-s2idle-without-i915:
- bat-dg1-5: NOTRUN -> [INCOMPLETE][16] ([i915#6011])
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/bat-dg1-5/igt@[email protected]

* igt@i915_suspend@basic-s3-without-i915:
- fi-icl-u2: NOTRUN -> [SKIP][17] ([i915#5903])
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-icl-u2/igt@[email protected]

* igt@kms_busy@basic@flip:
- fi-tgl-u2: [PASS][18] -> [DMESG-WARN][19] ([i915#402]) +2 similar issues
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-tgl-u2/igt@kms_busy@[email protected]
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-tgl-u2/igt@kms_busy@[email protected]

* igt@kms_chamelium@common-hpd-after-suspend:
- fi-bsw-nick: NOTRUN -> [SKIP][20] ([fdo#109271] / [fdo#111827])
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-bsw-nick/igt@[email protected]

* igt@kms_chamelium@hdmi-hpd-fast:
- fi-icl-u2: NOTRUN -> [SKIP][21] ([fdo#111827]) +8 similar issues
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-icl-u2/igt@[email protected]

* igt@kms_cursor_legacy@basic-busy-flip-before-cursor-legacy:
- fi-icl-u2: NOTRUN -> [SKIP][22] ([fdo#109278]) +2 similar issues
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-icl-u2/igt@[email protected]

* igt@kms_flip@basic-flip-vs-modeset@b-edp1:
- bat-adlp-4: [PASS][23] -> [DMESG-WARN][24] ([i915#3576])
[23]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/bat-adlp-4/igt@kms_flip@[email protected]
[24]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/bat-adlp-4/igt@kms_flip@[email protected]

* igt@kms_force_connector_basic@force-load-detect:
- fi-icl-u2: NOTRUN -> [SKIP][25] ([fdo#109285])
[25]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-icl-u2/igt@[email protected]

* igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-b:
- fi-cfl-8109u: [PASS][26] -> [DMESG-WARN][27] ([i915#62]) +12 similar issues
[26]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-cfl-8109u/igt@[email protected]
[27]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-cfl-8109u/igt@[email protected]

* igt@kms_pipe_crc_basic@suspend-read-crc-pipe-a:
- fi-bsw-nick: NOTRUN -> [SKIP][28] ([fdo#109271])
[28]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-bsw-nick/igt@[email protected]

* igt@kms_setmode@basic-clone-single-crtc:
- fi-icl-u2: NOTRUN -> [SKIP][29] ([i915#3555])
[29]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-icl-u2/igt@[email protected]

* igt@prime_vgem@basic-userptr:
- fi-pnv-d510: NOTRUN -> [SKIP][30] ([fdo#109271]) +39 similar issues
[30]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-pnv-d510/igt@[email protected]
- fi-icl-u2: NOTRUN -> [SKIP][31] ([i915#3301])
[31]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-icl-u2/igt@[email protected]

* igt@runner@aborted:
- fi-pnv-d510: NOTRUN -> [FAIL][32] ([fdo#109271] / [i915#2403] / [i915#4312])
[32]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-pnv-d510/igt@[email protected]
- fi-cfl-8109u: NOTRUN -> [FAIL][33] ([i915#4312])
[33]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-cfl-8109u/igt@[email protected]
- fi-hsw-g3258: NOTRUN -> [FAIL][34] ([fdo#109271] / [i915#4312])
[34]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-hsw-g3258/igt@[email protected]

#### Possible fixes ####

* igt@i915_selftest@live@execlists:
- fi-bsw-nick: [INCOMPLETE][35] ([i915#5801]) -> [PASS][36]
[35]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-bsw-nick/igt@i915_selftest@[email protected]
[36]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-bsw-nick/igt@i915_selftest@[email protected]

* igt@i915_selftest@live@gt_engines:
- bat-dg1-5: [INCOMPLETE][37] ([i915#4418]) -> [PASS][38]
[37]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/bat-dg1-5/igt@i915_selftest@live@gt_engines.html
[38]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/bat-dg1-5/igt@i915_selftest@live@gt_engines.html

* igt@kms_flip@basic-flip-vs-modeset@a-edp1:
- {bat-adlp-6}: [DMESG-WARN][39] ([i915#3576]) -> [PASS][40] +1 similar issue
[39]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/bat-adlp-6/igt@kms_flip@[email protected]
[40]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/bat-adlp-6/igt@kms_flip@[email protected]

* igt@kms_pipe_crc_basic@compare-crc-sanitycheck-pipe-d:
- fi-tgl-u2: [DMESG-WARN][41] ([i915#402]) -> [PASS][42]
[41]: https://intel-gfx-ci.01.org/tree/drm-tip/IGT_6481/fi-tgl-u2/igt@[email protected]
[42]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/fi-tgl-u2/igt@[email protected]

{name}: This element is suppressed. This means it is ignored when computing
the status of the difference (SUCCESS, WARNING, or FAILURE).

[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[fdo#109278]: https://bugs.freedesktop.org/show_bug.cgi?id=109278
[fdo#109285]: https://bugs.freedesktop.org/show_bug.cgi?id=109285
[fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
[i915#118]: https://gitlab.freedesktop.org/drm/intel/issues/118
[i915#1888]: https://gitlab.freedesktop.org/drm/intel/issues/1888
[i915#1982]: https://gitlab.freedesktop.org/drm/intel/issues/1982
[i915#2190]: https://gitlab.freedesktop.org/drm/intel/issues/2190
[i915#2403]: https://gitlab.freedesktop.org/drm/intel/issues/2403
[i915#2411]: https://gitlab.freedesktop.org/drm/intel/issues/2411
[i915#3301]: https://gitlab.freedesktop.org/drm/intel/issues/3301
[i915#3555]: https://gitlab.freedesktop.org/drm/intel/issues/3555
[i915#3576]: https://gitlab.freedesktop.org/drm/intel/issues/3576
[i915#402]: https://gitlab.freedesktop.org/drm/intel/issues/402
[i915#4312]: https://gitlab.freedesktop.org/drm/intel/issues/4312
[i915#4391]: https://gitlab.freedesktop.org/drm/intel/issues/4391
[i915#4418]: https://gitlab.freedesktop.org/drm/intel/issues/4418
[i915#4528]: https://gitlab.freedesktop.org/drm/intel/issues/4528
[i915#4613]: https://gitlab.freedesktop.org/drm/intel/issues/4613
[i915#4785]: https://gitlab.freedesktop.org/drm/intel/issues/4785
[i915#5801]: https://gitlab.freedesktop.org/drm/intel/issues/5801
[i915#5903]: https://gitlab.freedesktop.org/drm/intel/issues/5903
[i915#5950]: https://gitlab.freedesktop.org/drm/intel/issues/5950
[i915#6011]: https://gitlab.freedesktop.org/drm/intel/issues/6011
[i915#62]: https://gitlab.freedesktop.org/drm/intel/issues/62

Build changes
-------------

* CI: CI-20190529 -> None
* IGT: IGT_6481 -> IGTPW_7141

CI-20190529: 20190529
CI_DRM_11675: c31582c636f4c6ac8c0f822c39b250db2c5cd437 @ git://anongit.freedesktop.org/gfx-ci/linux
IGTPW_7141: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/index.html
IGT_6481: f2a9c2e6f6f7aa97e5d92274f20aa698087359c4 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7141/index.html

#2 lore raw
In reply to Mauro Carvalho Chehab

Hi Mauro,

small nitpicks on commit message.

Quoted 10 lines · Mauro Carvalho Chehab

From: Mauro Carvalho Chehab <[email protected]>

This test is expected to run under a certain time defined by a command line parameter (by default, 2s).

In order to achive the specified time, the baseline() function tries to estimate the value of a counter that will get the minimal amount of interaction to wait for > 0.1s.

However, currently, the baseline estimation is broken, returning a too high

------------------------------------------------------------------------ ^ s/high/big/

Quoted 2 lines

number, as it is measuring the memcpy() time, instead of actually measuring each loop's interaction.

/each loop's interaction./gem_execbuf()./

Quoted 8 lines

Due to that, a default test without passing any command line parameter would take more than 1.5 years to output the first benchmark data!

Fix it by using the same logic that it is inside run() at the baseline time estimation.

On ADL-P, before this patch, baseline would return 399242693. After

s/On ADL-P, before/Before/
s/would/could/

Quoted 1 line

it, it now return 20 - meaning an interval of about 120 ms, which

Well, both numbers means the same interval _if_ used for the same syscall.

With that fixed you can add my r-b tag.

--
Kamil

Quoted 57 lines

should be good enough to ensure that the tests won't take too long.

---
 benchmarks/gem_blt.c | 18 +++++-------------
 1 file changed, 5 insertions(+), 13 deletions(-)
benchmarks/gem_blt.c +5 -13
index 424ce8e75913..bd8264b4e896 100644
@@ -57,29 +57,21 @@ elapsed(const struct timespec *start, const struct timespec *end)
 	return (end->tv_sec - start->tv_sec) + 1e-9*(end->tv_nsec - start->tv_nsec);
 }
 
-static int baseline(uint64_t bytes, int milliseconds)
+static int baseline(int fd, struct drm_i915_gem_execbuffer2 execbuf, int milliseconds)
 {
 	struct timespec start, end;
-	const int size = 64*1024*1024;
 	int count = 0;
-	void *mem;
-
-	mem = malloc(size);
-	if (mem == NULL)
-		return 1;
 
 	clock_gettime(CLOCK_MONOTONIC, &start);
 	do {
-		memset(mem, count, size);
+		gem_execbuf(fd, &execbuf);
 		count++;
 		clock_gettime(CLOCK_MONOTONIC, &end);
-		if (elapsed(&start, &end) > 0.1)
+		if (elapsed(&start, &end) > (milliseconds / 1000.))
 			break;
 	} while (1);
 
-	free(mem);
-
-	return ceil(1e-3*milliseconds/elapsed(&start, &end) * count * size / bytes);
+	return count;
 }
 
 static int gem_linear_blt(int fd,
@@ -260,7 +252,7 @@ static int run(int object, int batch, int time, int reps, int ncpus, unsigned fl
 		execbuf.flags |= I915_EXEC_NO_RELOC;
 
 	/* Guess how many loops we need for 0.1s */
-	count = baseline((uint64_t)object * batch, 100) / ncpus;
+	count = baseline(fd, execbuf, 100) / ncpus;
 	if (flags & SYNC) {
 		time *= count / 2;
 		count = 1;
Signature · 1 line
2.36.1