Skip to contentSkip to conversation

Updates to gem_exec_nop parallel execution test

PATCH In review Dave Gordon 4 participants 14 messages Aug 3, 2016 – Aug 22, 2016

lore
Share & tools
lore
#0 lore raw

cover Updates to gem_exec_nop parallel execution test

The parallel execution test in gem_exec_nop is unrealistic, and in fact chooses a pessimal distribution of work to multiple engines. The first patch in this sequence adapts it to send multiple batches (currently 64) to each engine, in a round-robin fashion, thus keeping the engines fed adequately with workstreams.

The second patch simply changes the output of this same test to make it more obvious what measurements are made and what results can be calculated from them.

Dave Gordon (2):
igt/gem_exec_nop: add burst submission to parallel execution test igt/gem_exec_nop: clarify & extend output from parallel execution test

 tests/gem_exec_nop.c | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw

PATCH 1/2 igt/gem_exec_nop: add burst submission to parallel execution test

The parallel execution test in gem_exec_nop chooses a pessimal distribution of work to multiple engines; specifically, it round-robins one batch to each engine in turn. As the workloads are trivial (NOPs), this results in each engine becoming idle between batches. Hence parallel submission is seen to take LONGER than the same number of batches executed sequentially.

If on the other hand we send enough work to each engine to keep it busy until the next time we add to its queue, (i.e. round-robin some larger number of batches to each engine in turn) then we can get true parallel execution and should find that it is FASTER than sequential execuion.

By experiment, burst sizes of between 8 and 256 are sufficient to keep multiple engines loaded, with the optimum (for this trivial workload) being around 64. This is expected to be lower (possibly as low as one) for more realistic (heavier) workloads.

---
 tests/gem_exec_nop.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
tests/gem_exec_nop.c +5 -2
index 9b89260..c2bd472 100644
@@ -166,14 +166,17 @@ static void all(int fd, uint32_t handle, int timeout)
 	gem_sync(fd, handle);
 	intel_detect_and_clear_missed_interrupts(fd);
 
+#define	BURST	64
+
 	count = 0;
 	clock_gettime(CLOCK_MONOTONIC, &start);
 	do {
-		for (int loop = 0; loop < 1024; loop++) {
+		for (int loop = 0; loop < 1024/BURST; loop++) {
 			for (int n = 0; n < nengine; n++) {
 				execbuf.flags &= ~ENGINE_FLAGS;
 				execbuf.flags |= engines[n];
-				gem_execbuf(fd, &execbuf);
+				for (int b = 0; b < BURST; ++b)
+					gem_execbuf(fd, &execbuf);
 			}
 		}
 		count += nengine * 1024;
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

Re: [PATCH 1/2] igt/gem_exec_nop: add burst submission to parallel execution test

Quoted 13 lines · Dave Gordon

The parallel execution test in gem_exec_nop chooses a pessimal distribution of work to multiple engines; specifically, it round-robins one batch to each engine in turn. As the workloads are trivial (NOPs), this results in each engine becoming idle between batches. Hence parallel submission is seen to take LONGER than the same number of batches executed sequentially.

If on the other hand we send enough work to each engine to keep it busy until the next time we add to its queue, (i.e. round-robin some larger number of batches to each engine in turn) then we can get true parallel execution and should find that it is FASTER than sequential execuion.

Quoted 4 lines

By experiment, burst sizes of between 8 and 256 are sufficient to keep multiple engines loaded, with the optimum (for this trivial workload) being around 64. This is expected to be lower (possibly as low as one) for more realistic (heavier) workloads.

Quite funny. The driver submission overhead of A...A vs ABAB... engines is nearly identical, at least as far as the analysis presented here. -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#3 lore raw

Re: [PATCH 1/2] igt/gem_exec_nop: add burst submission to parallel execution test

Quoted 14 lines · Chris Wilson
Quoted 13 lines · Dave Gordon

The parallel execution test in gem_exec_nop chooses a pessimal distribution of work to multiple engines; specifically, it round-robins one batch to each engine in turn. As the workloads are trivial (NOPs), this results in each engine becoming idle between batches. Hence parallel submission is seen to take LONGER than the same number of batches executed sequentially.

If on the other hand we send enough work to each engine to keep it busy until the next time we add to its queue, (i.e. round-robin some larger number of batches to each engine in turn) then we can get true parallel execution and should find that it is FASTER than sequential execuion.

Quoted 8 lines
Quoted 4 lines

By experiment, burst sizes of between 8 and 256 are sufficient to keep multiple engines loaded, with the optimum (for this trivial workload) being around 64. This is expected to be lower (possibly as low as one) for more realistic (heavier) workloads.

Quite funny. The driver submission overhead of A...A vs ABAB... engines is nearly identical, at least as far as the analysis presented here. -Chris

Correct; but because the workloads are so trivial, if we hand out jobs one at a time to each engine, the first will have finished the one batch it's been given before we get round to giving at a second one (even in execlist mode). If there are N engines, submitting a single batch takes S seconds, and the workload takes W seconds to execute, then if W < N*S the engine will be idle between batches. For example, if N is 4, W is 2us, and S is 1us, then the engine will be idle some 50% of the time.

This wouldn't be an issue for more realistic workloads, where W >> S. It only looks problematic because of the trivial nature of the work.

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#4 lore raw

Re: [PATCH 1/2] igt/gem_exec_nop: add burst submission to parallel execution test

Quoted 20 lines · Dave Gordon
Quoted 19 lines · Chris Wilson
Quoted 17 lines · Dave Gordon

The parallel execution test in gem_exec_nop chooses a pessimal distribution of work to multiple engines; specifically, it round-robins one batch to each engine in turn. As the workloads are trivial (NOPs), this results in each engine becoming idle between batches. Hence parallel submission is seen to take LONGER than the same number of batches executed sequentially.

If on the other hand we send enough work to each engine to keep it busy until the next time we add to its queue, (i.e. round-robin some larger number of batches to each engine in turn) then we can get true parallel execution and should find that it is FASTER than sequential execuion.

By experiment, burst sizes of between 8 and 256 are sufficient to keep multiple engines loaded, with the optimum (for this trivial workload) being around 64. This is expected to be lower (possibly as low as one) for more realistic (heavier) workloads.

Quoted 8 lines
Quoted 3 lines

Quite funny. The driver submission overhead of A...A vs ABAB... engines is nearly identical, at least as far as the analysis presented here. -Chris

Correct; but because the workloads are so trivial, if we hand out jobs one at a time to each engine, the first will have finished the one batch it's been given before we get round to giving at a second one (even in execlist mode). If there are N engines, submitting a single batch takes S seconds, and the workload takes W seconds to execute, then if W < N*S the engine will be idle between batches. For example, if N is 4, W is 2us, and S is 1us, then the engine will be idle some 50% of the time.

This wouldn't be an issue for more realistic workloads, where W >> S. It only looks problematic because of the trivial nature of the work.

Can you post the numbers that you get?

I seem to get massive variability on my BDW. The render ring always gives me around 2.9us/batch but the other rings sometimes give me region of 1.2us and sometimes 7-8us.

Quoted 6 lines

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#5 lore raw

Re: [PATCH 1/2] igt/gem_exec_nop: add burst submission to parallel execution test

Quoted 34 lines · John Harrison
Quoted 33 lines · Dave Gordon
Quoted 14 lines · Chris Wilson
Quoted 13 lines · Dave Gordon

The parallel execution test in gem_exec_nop chooses a pessimal distribution of work to multiple engines; specifically, it round-robins one batch to each engine in turn. As the workloads are trivial (NOPs), this results in each engine becoming idle between batches. Hence parallel submission is seen to take LONGER than the same number of batches executed sequentially.

If on the other hand we send enough work to each engine to keep it busy until the next time we add to its queue, (i.e. round-robin some larger number of batches to each engine in turn) then we can get true parallel execution and should find that it is FASTER than sequential execuion.

Quoted 8 lines
Quoted 4 lines

By experiment, burst sizes of between 8 and 256 are sufficient to keep multiple engines loaded, with the optimum (for this trivial workload) being around 64. This is expected to be lower (possibly as low as one) for more realistic (heavier) workloads.

Quite funny. The driver submission overhead of A...A vs ABAB... engines is nearly identical, at least as far as the analysis presented here. -Chris

Correct; but because the workloads are so trivial, if we hand out jobs one at a time to each engine, the first will have finished the one batch it's been given before we get round to giving at a second one (even in execlist mode). If there are N engines, submitting a single batch takes S seconds, and the workload takes W seconds to execute, then if W < N*S the engine will be idle between batches. For example, if N is 4, W is 2us, and S is 1us, then the engine will be idle some 50% of the time.

Quoted 8 lines
Quoted 2 lines

This wouldn't be an issue for more realistic workloads, where W >> S. It only looks problematic because of the trivial nature of the work.

Can you post the numbers that you get?

I seem to get massive variability on my BDW. The render ring always gives me around 2.9us/batch but the other rings sometimes give me region of 1.2us and sometimes 7-8us.

skylake# ./intel-gpu-tools/tests/gem_exec_nop --run-subtest basic
IGT-Version: 1.15-gd09ad86 (x86_64) (Linux: 4.8.0-rc1-dsg-10839-g5e5a29c-z-tvrtko-fwname x86_64) Using GuC submission
render: 594,944 cycles: 3.366us/batch
bsd: 737,280 cycles: 2.715us/batch
blt: 833,536 cycles: 2.400us/batch
vebox: 710,656 cycles: 2.818us/batch
Slowest engine was render, 3.366us/batch
Total for all 4 engines is 11.300us per cycle, average 2.825us/batch
All 4 engines (parallel/64): 5,324,800 cycles, average 1.878us/batch, overlap 90.1% Subtest basic: SUCCESS (18.013s)

These are the results of running the modified test on SKL with GuC submission.

If the GPU could execute a trivial batch in less time than it takes the CPU to submit one, then CPU/driver/GuC performance would become the determining factor -- every batch would be completed before the next one was submitted to the GPU even when they're going to the same engine.

If the GPU takes longer to execute a batch than N times the time taken for the driver to submit it (where N is the number of engines), then the GPU performance would become the limiting factor; the CPU would be able to hand out one batch to each engine, and by the time it returned to the first, that engine would still not be idle.

But in crossover territory, where the batch takes longer to execute than the time to submit it, but less than N times as long, the round-robin burst size (number of batches sent to each engine before moving to the next) can make a big difference, primarily because the submission mechanism gets the opportunity to use dual submission and/or lite restore, effectively reducing the number of separate writes to the ELSP and hence the s/w overhead per batch.

Note that SKL GuC firmware 6.1 didn't support dual submission or lite restore, whereas the next version (8.11) does. Therefore, with that firmware we don't see the same slowdown when going to 1-at-a-time round-robin. I have a different (new) test that shows this more clearly.

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#6 lore raw

Re: [PATCH 1/2] igt/gem_exec_nop: add burst submission to parallel execution test

On 18/08/16 16:27, Dave Gordon wrote:

[snip]

Quoted 4 lines

Note that SKL GuC firmware 6.1 didn't support dual submission or lite restore, whereas the next version (8.11) does. Therefore, with that firmware we don't see the same slowdown when going to 1-at-a-time round-robin. I have a different (new) test that shows this more clearly.

This is with GuC version 6.1:

skylake# ./intel-gpu-tools/tests/gem_exec_paranop | fgrep -v SUCCESS

Time to exec 8-byte batch:	  3.428µs (ring=render)
Time to exec 8-byte batch:	  2.444µs (ring=bsd)
Time to exec 8-byte batch:	  2.394µs (ring=blt)
Time to exec 8-byte batch:	  2.615µs (ring=vebox)
Time to exec 8-byte batch:	  2.625µs (ring=all, sequential)
Time to exec 8-byte batch:	 12.701µs (ring=all, parallel/1) ***
Time to exec 8-byte batch:	  7.259µs (ring=all, parallel/2)
Time to exec 8-byte batch:	  4.336µs (ring=all, parallel/4)
Time to exec 8-byte batch:	  2.937µs (ring=all, parallel/8)
Time to exec 8-byte batch:	  2.661µs (ring=all, parallel/16)
Time to exec 8-byte batch:	  2.245µs (ring=all, parallel/32)
Time to exec 8-byte batch:	  1.626µs (ring=all, parallel/64)
Time to exec 8-byte batch:	  2.170µs (ring=all, parallel/128)
Time to exec 8-byte batch:	  1.804µs (ring=all, parallel/256)
Time to exec 8-byte batch:	  2.602µs (ring=all, parallel/512)
Time to exec 8-byte batch:	  2.602µs (ring=all, parallel/1024)
Time to exec 8-byte batch:	  2.607µs (ring=all, parallel/2048)

Time to exec 4Kbyte batch: 14.835µs (ring=render)
Time to exec 4Kbyte batch: 11.787µs (ring=bsd)
Time to exec 4Kbyte batch: 11.533µs (ring=blt)
Time to exec 4Kbyte batch: 11.991µs (ring=vebox)
Time to exec 4Kbyte batch: 12.444µs (ring=all, sequential)
Time to exec 4Kbyte batch: 16.211µs (ring=all, parallel/1)
Time to exec 4Kbyte batch: 13.943µs (ring=all, parallel/2)
Time to exec 4Kbyte batch: 13.878µs (ring=all, parallel/4)
Time to exec 4Kbyte batch: 13.841µs (ring=all, parallel/8)
Time to exec 4Kbyte batch: 14.188µs (ring=all, parallel/16)
Time to exec 4Kbyte batch: 13.747µs (ring=all, parallel/32)
Time to exec 4Kbyte batch: 13.734µs (ring=all, parallel/64)
Time to exec 4Kbyte batch: 13.727µs (ring=all, parallel/128)
Time to exec 4Kbyte batch: 13.947µs (ring=all, parallel/256)
Time to exec 4Kbyte batch: 12.230µs (ring=all, parallel/512)
Time to exec 4Kbyte batch: 12.147µs (ring=all, parallel/1024)
Time to exec 4Kbyte batch: 12.617µs (ring=all, parallel/2048)

What this shows is that the submission overhead is ~3us which is comparable with the execution time of a trivial (8-byte) batch, but insignificant compared with the time to execute the 4Kbyte batch. The burst size therefore makes very little difference to the larger batches.

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#7 lore raw

Re: [PATCH 1/2] igt/gem_exec_nop: add burst submission to parallel execution test

Quoted 22 lines · Dave Gordon

On 18/08/16 16:27, Dave Gordon wrote:

[snip]

Quoted 4 lines

Note that SKL GuC firmware 6.1 didn't support dual submission or lite restore, whereas the next version (8.11) does. Therefore, with that firmware we don't see the same slowdown when going to 1-at-a-time round-robin. I have a different (new) test that shows this more clearly.

This is with GuC version 6.1:

skylake# ./intel-gpu-tools/tests/gem_exec_paranop | fgrep -v SUCCESS

Time to exec 8-byte batch:      3.428µs (ring=render)
Time to exec 8-byte batch:      2.444µs (ring=bsd)
Time to exec 8-byte batch:      2.394µs (ring=blt)
Time to exec 8-byte batch:      2.615µs (ring=vebox)
Time to exec 8-byte batch:      2.625µs (ring=all, sequential)
Time to exec 8-byte batch:     12.701µs (ring=all, parallel/1) ***
Time to exec 8-byte batch:      7.259µs (ring=all, parallel/2)
Time to exec 8-byte batch:      4.336µs (ring=all, parallel/4)
Time to exec 8-byte batch:      2.937µs (ring=all, parallel/8)
Quoted 8 lines
Time to exec 8-byte batch:      2.661µs (ring=all, parallel/16)
Time to exec 8-byte batch:      2.245µs (ring=all, parallel/32)
Time to exec 8-byte batch:      1.626µs (ring=all, parallel/64)
Time to exec 8-byte batch:      2.170µs (ring=all, parallel/128)
Time to exec 8-byte batch:      1.804µs (ring=all, parallel/256)
Time to exec 8-byte batch:      2.602µs (ring=all, parallel/512)
Time to exec 8-byte batch:      2.602µs (ring=all, parallel/1024)
Time to exec 8-byte batch:      2.607µs (ring=all, parallel/2048)

And for comparison, here are the figures with v8.11:

# ./intel-gpu-tools/tests/gem_exec_paranop | fgrep -v SUCCESS
Time to exec 8-byte batch:	  3.458µs (ring=render)
Time to exec 8-byte batch:	  2.154µs (ring=bsd)
Time to exec 8-byte batch:	  2.156µs (ring=blt)
Time to exec 8-byte batch:	  2.156µs (ring=vebox)
Time to exec 8-byte batch:	  2.388µs (ring=all, sequential)
Time to exec 8-byte batch:	  5.897µs (ring=all, parallel/1)
Time to exec 8-byte batch:	  4.669µs (ring=all, parallel/2)
Time to exec 8-byte batch:	  4.278µs (ring=all, parallel/4)
Time to exec 8-byte batch:	  2.410µs (ring=all, parallel/8)
Time to exec 8-byte batch:	  2.165µs (ring=all, parallel/16)
Time to exec 8-byte batch:	  2.158µs (ring=all, parallel/32)
Time to exec 8-byte batch:	  1.594µs (ring=all, parallel/64)
Time to exec 8-byte batch:	  1.583µs (ring=all, parallel/128)
Time to exec 8-byte batch:	  2.473µs (ring=all, parallel/256)
Time to exec 8-byte batch:	  2.264µs (ring=all, parallel/512)
Time to exec 8-byte batch:	  2.357µs (ring=all, parallel/1024)
Time to exec 8-byte batch:	  2.382µs (ring=all, parallel/2048)

All generally slightly faster, but parallel/1 is approximately twice as fast, while parallel/64 is virtually unchanged, as are all the timings for large batches.

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#8 lore raw

Re: [PATCH 1/2] igt/gem_exec_nop: add burst submission to parallel execution test

In reply to Dave Gordon
Quoted 1 line · Dave Gordon

On 18/08/16 13:01, John Harrison wrote:

[snip]

Quoted 6 lines
Quoted 5 lines

Can you post the numbers that you get?

I seem to get massive variability on my BDW. The render ring always gives me around 2.9us/batch but the other rings sometimes give me region of 1.2us and sometimes 7-8us.

Quoted 13 lines

skylake# ./intel-gpu-tools/tests/gem_exec_nop --run-subtest basic
IGT-Version: 1.15-gd09ad86 (x86_64) (Linux:
4.8.0-rc1-dsg-10839-g5e5a29c-z-tvrtko-fwname x86_64)
Using GuC submission
render: 594,944 cycles: 3.366us/batch
bsd: 737,280 cycles: 2.715us/batch
blt: 833,536 cycles: 2.400us/batch
vebox: 710,656 cycles: 2.818us/batch
Slowest engine was render, 3.366us/batch
Total for all 4 engines is 11.300us per cycle, average 2.825us/batch All 4 engines (parallel/64): 5,324,800 cycles, average 1.878us/batch, overlap 90.1%
Subtest basic: SUCCESS (18.013s)

That was GuC f/w 6.1, here's the results from 8.11:

skylake# sudo ./intel-gpu-tools/tests/gem_exec_nop --run-subtest basic
IGT-Version: 1.15-gd09ad86 (x86_64) (Linux: 4.8.0-rc2-dsg-11313-g7430e5f-dsg-work-101 x86_64) Using GuC submission
render: 585,728 cycles: 3.418us/batch
bsd: 930,816 cycles: 2.151us/batch
blt: 930,816 cycles: 2.150us/batch
vebox: 930,816 cycles: 2.150us/batch
Slowest engine was render, 3.418us/batch
Total for all 4 engines is 9.869us per cycle, average 2.467us/batch
All 4 engines (parallel/64): 5,668,864 cycles, average 1.765us/batch, overlap 89.9% Subtest basic: SUCCESS (18.016s)

... showing minor improvements generally, especially the non-render engines.

.Dave.
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Reviewed-by on v1 1/2
#9 lore raw

Re: [PATCH 1/2] igt/gem_exec_nop: add burst submission to parallel execution test

In reply to Dave Gordon
Quoted 37 lines · Dave Gordon

The parallel execution test in gem_exec_nop chooses a pessimal distribution of work to multiple engines; specifically, it round-robins one batch to each engine in turn. As the workloads are trivial (NOPs), this results in each engine becoming idle between batches. Hence parallel submission is seen to take LONGER than the same number of batches executed sequentially.

If on the other hand we send enough work to each engine to keep it busy until the next time we add to its queue, (i.e. round-robin some larger number of batches to each engine in turn) then we can get true parallel execution and should find that it is FASTER than sequential execuion.

By experiment, burst sizes of between 8 and 256 are sufficient to keep multiple engines loaded, with the optimum (for this trivial workload) being around 64. This is expected to be lower (possibly as low as one) for more realistic (heavier) workloads.

---
  tests/gem_exec_nop.c | 7 +++++--
  1 file changed, 5 insertions(+), 2 deletions(-)
tests/gem_exec_nop.c +2 -1
index 9b89260..c2bd472 100644
@@ -166,14 +166,17 @@ static void all(int fd, uint32_t handle, int timeout)
  	gem_sync(fd, handle);
  	intel_detect_and_clear_missed_interrupts(fd);
 
+#define	BURST	64
+
  	count = 0;
  	clock_gettime(CLOCK_MONOTONIC, &start);
  	do {
-		for (int loop = 0; loop < 1024; loop++) {

tests/gem_exec_nop.c static void all(int fd, uint32_t handle, int timeout) · line 182

Quoted 10 lines
tests/gem_exec_nop.c +3 -1
+		for (int loop = 0; loop < 1024/BURST; loop++) {
  			for (int n = 0; n < nengine; n++) {
  				execbuf.flags &= ~ENGINE_FLAGS;
  				execbuf.flags |= engines[n];
-				gem_execbuf(fd, &execbuf);
+				for (int b = 0; b < BURST; ++b)
+					gem_execbuf(fd, &execbuf);
  			}
  		}
  		count += nengine * 1024;

Would be nice to have the burst size configurable but either way...

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#10 lore raw

PATCH 2/2 igt/gem_exec_nop: clarify & extend output from parallel execution test

In reply to Dave Gordon

To make sense of the output of the parallel execution test (preferably without reading the source!), we need to see the various measurements that it makes, specifically: time/batch on each engine separately, total time across all engines sequentially, and the time/batch when the work is distributed over all engines in parallel.

Since we know the per-batch time on the slowest engine (which will determine the minimum possible execution time of any equal-split parallel test), we can also calculate a new figure representing the degree to which work on the faster engines is overlapped with that on the slowest engine, and therefore does not contribute to the total time. Here we choose to present it as a percentage, with parallel-time==serial time giving 0% overlap, up to parallel-time==slowest-engine- time/n_engines being 100%. Note that negative values are possible; values greater than 100% may also be possible, although less likely.

---
 tests/gem_exec_nop.c | 15 ++++++++++-----
 1 file changed, 10 insertions(+), 5 deletions(-)
tests/gem_exec_nop.c +10 -5
index c2bd472..05aa383 100644
@@ -137,7 +137,9 @@ static void all(int fd, uint32_t handle, int timeout)
 		if (ignore_engine(fd, engine))
 			continue;
 
-		time = nop_on_ring(fd, handle, engine, 1, &count) / count;
+		time = nop_on_ring(fd, handle, engine, 2, &count) / count;
+		igt_info("%s: %'lu cycles: %.3fus/batch\n",
+			 e__->name, count, time*1e6);
 		if (time > max) {
 			name = e__->name;
 			max = time;
@@ -148,8 +150,9 @@ static void all(int fd, uint32_t handle, int timeout)
 		engines[nengine++] = engine;
 	}
 	igt_require(nengine);
-	igt_info("Maximum execution latency on %s, %.3fus, total %.3fus per cycle\n",
-		 name, max*1e6, sum*1e6);
+	igt_info("Slowest engine was %s, %.3fus/batch\n", name, max*1e6);
+	igt_info("Total for all %d engines is %.3fus per cycle, average %.3fus/batch\n",
+		 nengine, sum*1e6, sum*1e6/nengine);
 
 	memset(&obj, 0, sizeof(obj));
 	obj.handle = handle;
@@ -187,8 +190,10 @@ static void all(int fd, uint32_t handle, int timeout)
 	igt_assert_eq(intel_detect_and_clear_missed_interrupts(fd), 0);
 
 	time = elapsed(&start, &now) / count;
-	igt_info("All (%d engines): %'lu cycles, average %.3fus per cycle\n",
-		 nengine, count, 1e6*time);
+	igt_info("All %d engines (parallel/%d): %'lu cycles, "
+		 "average %.3fus/batch, overlap %.1f%\n",
+		 nengine, BURST, count,
+		 1e6*time, 100*(sum-time)/(sum-(max/nengine)));
 
 	/* The rate limiting step is how fast the slowest engine can
 	 * its queue of requests, if we wait upon a full ring all dispatch
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Reviewed-by on v1 2/2
#11 lore raw

Re: [PATCH 2/2] igt/gem_exec_nop: clarify & extend output from parallel execution test

Quoted 51 lines · Dave Gordon

To make sense of the output of the parallel execution test (preferably without reading the source!), we need to see the various measurements that it makes, specifically: time/batch on each engine separately, total time across all engines sequentially, and the time/batch when the work is distributed over all engines in parallel.

Since we know the per-batch time on the slowest engine (which will determine the minimum possible execution time of any equal-split parallel test), we can also calculate a new figure representing the degree to which work on the faster engines is overlapped with that on the slowest engine, and therefore does not contribute to the total time. Here we choose to present it as a percentage, with parallel-time==serial time giving 0% overlap, up to parallel-time==slowest-engine- time/n_engines being 100%. Note that negative values are possible; values greater than 100% may also be possible, although less likely.

---
  tests/gem_exec_nop.c | 15 ++++++++++-----
  1 file changed, 10 insertions(+), 5 deletions(-)
tests/gem_exec_nop.c +6 -3
index c2bd472..05aa383 100644
@@ -137,7 +137,9 @@ static void all(int fd, uint32_t handle, int timeout)
  		if (ignore_engine(fd, engine))
  			continue;
 
-		time = nop_on_ring(fd, handle, engine, 1, &count) / count;
+		time = nop_on_ring(fd, handle, engine, 2, &count) / count;
+		igt_info("%s: %'lu cycles: %.3fus/batch\n",
+			 e__->name, count, time*1e6);
  		if (time > max) {
  			name = e__->name;
  			max = time;
@@ -148,8 +150,9 @@ static void all(int fd, uint32_t handle, int timeout)
  		engines[nengine++] = engine;
  	}
  	igt_require(nengine);
-	igt_info("Maximum execution latency on %s, %.3fus, total %.3fus per cycle\n",
-		 name, max*1e6, sum*1e6);
+	igt_info("Slowest engine was %s, %.3fus/batch\n", name, max*1e6);
+	igt_info("Total for all %d engines is %.3fus per cycle, average %.3fus/batch\n",
+		 nengine, sum*1e6, sum*1e6/nengine);
 
  	memset(&obj, 0, sizeof(obj));
  	obj.handle = handle;
@@ -187,8 +190,10 @@ static void all(int fd, uint32_t handle, int timeout)
  	igt_assert_eq(intel_detect_and_clear_missed_interrupts(fd), 0);
 

tests/gem_exec_nop.c static void all(int fd, uint32_t handle, int timeout) · line 199

Quoted 10 lines
tests/gem_exec_nop.c +4 -2
  	time = elapsed(&start, &now) / count;
-	igt_info("All (%d engines): %'lu cycles, average %.3fus per cycle\n",
-		 nengine, count, 1e6*time);
+	igt_info("All %d engines (parallel/%d): %'lu cycles, "
+		 "average %.3fus/batch, overlap %.1f%\n",
+		 nengine, BURST, count,
+		 1e6*time, 100*(sum-time)/(sum-(max/nengine)));
 
  	/* The rate limiting step is how fast the slowest engine can
  	 * its queue of requests, if we wait upon a full ring all dispatch

I'm not entirely convinced about the overlap calculation. The other info is definitely useful though.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#12 lore raw

Re: [PATCH 2/2] igt/gem_exec_nop: clarify & extend output from parallel execution test

Quoted 52 lines · John Harrison
Quoted 51 lines · Dave Gordon

To make sense of the output of the parallel execution test (preferably without reading the source!), we need to see the various measurements that it makes, specifically: time/batch on each engine separately, total time across all engines sequentially, and the time/batch when the work is distributed over all engines in parallel.

Since we know the per-batch time on the slowest engine (which will determine the minimum possible execution time of any equal-split parallel test), we can also calculate a new figure representing the degree to which work on the faster engines is overlapped with that on the slowest engine, and therefore does not contribute to the total time. Here we choose to present it as a percentage, with parallel-time==serial time giving 0% overlap, up to parallel-time==slowest-engine- time/n_engines being 100%. Note that negative values are possible; values greater than 100% may also be possible, although less likely.

---
  tests/gem_exec_nop.c | 15 ++++++++++-----
  1 file changed, 10 insertions(+), 5 deletions(-)
tests/gem_exec_nop.c +6 -3
index c2bd472..05aa383 100644
@@ -137,7 +137,9 @@ static void all(int fd, uint32_t handle, int timeout)
  		if (ignore_engine(fd, engine))
  			continue;
 
-		time = nop_on_ring(fd, handle, engine, 1, &count) / count;
+		time = nop_on_ring(fd, handle, engine, 2, &count) / count;
+		igt_info("%s: %'lu cycles: %.3fus/batch\n",
+			 e__->name, count, time*1e6);
  		if (time > max) {
  			name = e__->name;
  			max = time;
@@ -148,8 +150,9 @@ static void all(int fd, uint32_t handle, int timeout)
  		engines[nengine++] = engine;
  	}
  	igt_require(nengine);
-	igt_info("Maximum execution latency on %s, %.3fus, total %.3fus per cycle\n",
-		 name, max*1e6, sum*1e6);
+	igt_info("Slowest engine was %s, %.3fus/batch\n", name, max*1e6);
+	igt_info("Total for all %d engines is %.3fus per cycle, average %.3fus/batch\n",
+		 nengine, sum*1e6, sum*1e6/nengine);
 
  	memset(&obj, 0, sizeof(obj));
  	obj.handle = handle;
@@ -187,8 +190,10 @@ static void all(int fd, uint32_t handle, int timeout)
  	igt_assert_eq(intel_detect_and_clear_missed_interrupts(fd), 0);
 
Quoted 5 lines

tests/gem_exec_nop.c static void all(int fd, uint32_t handle, int timeout) · line 194

Quoted 5 lines
tests/gem_exec_nop.c +2 -2
  	time = elapsed(&start, &now) / count;
-	igt_info("All (%d engines): %'lu cycles, average %.3fus per cycle\n",
-		 nengine, count, 1e6*time);
+	igt_info("All %d engines (parallel/%d): %'lu cycles, "
+		 "average %.3fus/batch, overlap %.1f%\n",

PS: As mentioned in person, the above format string is should end with '%%\n' not '%\n'. But with that fixed, review-by as below.

Quoted 10 lines

tests/gem_exec_nop.c static void all(int fd, uint32_t handle, int timeout) · line 199

Quoted 5 lines
tests/gem_exec_nop.c +2 -0
+		 nengine, BURST, count,
+		 1e6*time, 100*(sum-time)/(sum-(max/nengine)));
 
  	/* The rate limiting step is how fast the slowest engine can
  	 * its queue of requests, if we wait upon a full ring all dispatch

I'm not entirely convinced about the overlap calculation. The other info is definitely useful though.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#13 lore raw

✗ Ro.CI.BAT: failure for Updates to gem_exec_nop parallel execution test

In reply to Dave Gordon
== Series Details ==

== Series Details ==

Series: Updates to gem_exec_nop parallel execution test
URL : https://patchwork.freedesktop.org/series/10603/
State : failure

== Summary ==

Applying: igt/gem_exec_nop: add burst submission to parallel execution test
fatal: sha1 information is lacking or useless (tests/gem_exec_nop.c).
error: could not build fake ancestor
Patch failed at 0001 igt/gem_exec_nop: add burst submission to parallel execution test The copy of the patch that failed is found in: .git/rebase-apply/patch When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx