Workload simulation and tracing
Subsystems (2)
cover Workload simulation and tracing
From: Tvrtko Ursulin <[email protected]>
Another WIP posting with some interopability improvements this time.
Example usage:
root@scnuc:~/intel-gpu-tools# benchmarks/gem_wsim Calibrating nop delay with 1% tolerance... Nop calibration for 1000us delay is 438660. root@scnuc:~/intel-gpu-tools# scripts/trace.pl --trace benchmarks/gem_wsim -n 438660 -r 600 -w benchmarks/wsim/workload1 -c 2 -x Using 438660 nop calibration for 1000us delay. 2 clients. Swapping VCS rings between clients. 0: 10.222053s elapsed (58.696622 workloads/s) 1: 10.225807s elapsed (58.675078 workloads/s) 10.226307s elapsed (117.344411 workloads/s) [ perf record: Woken up 16 times to write data ] [ perf record: Captured and wrote 3.929 MB perf.data (44688 samples) ] root@scnuc:~/intel-gpu-tools# perf script | scripts/trace.pl -i 1 -i 4 Ring0: 4832 batches, 4329.07 (4348.11) avg batch us, 0.15% idle, 138.73% busy, 23.88% runnable, 79.66% queued, 0.01% wait) Ring2: 1808 batches, 1715.68 (1719.99) avg batch us, 69.68% idle, 30.38% busy, 0.12% runnable, 112.04% queued, 99.32% wait) Ring3: 1810 batches, 1727.59 (1739.31) avg batch us, 69.28% idle, 30.75% busy, 0.11% runnable, 105.37% queued, 99.46% wait)
Most interesting metric here is the engine idle time which will come into play as I start adding the load balancing options to gem_wsim.
Secondary mode here would be:
root@scnuc:~/intel-gpu-tools# perf script | scripts/trace.pl -i 1 -i 4 --html >graph.html
This would enable a timeline of GPU request execution to be viewed with a little bit of local setup as described in the trace.pl help text.
Tvrtko Ursulin (2):
benchmarks/gem_wsim: Command submission workload simulator igt/scripts: trace.pl to parse the i915 tracepoints
benchmarks/Makefile.sources | 1 + benchmarks/gem_wsim.c | 593 +++++++++++++++++++++++++++ benchmarks/wsim/workload1 | 7 + scripts/Makefile.am | 2 +- scripts/trace.pl | 946 ++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 1548 insertions(+), 1 deletion(-) create mode 100644 benchmarks/gem_wsim.c create mode 100644 benchmarks/wsim/workload1 create mode 100755 scripts/trace.pl
Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 1/2 benchmarks/gem_wsim: Command submission workload simulator
From: Tvrtko Ursulin <[email protected]>
Tool which emits batch buffers to engines with configurable sequences, durations, contexts, dependencies and userspace waits.
Unfinished but shows promise so sending out for early feedback.
v2:
* Load workload descriptors from files. (also -w)
* Help text.
* Calibration control if needed. (-t)
* NORELOC | LUT to eb flags.
* Added sample workload to wsim/workload1.
TODO list:
* Better error handling.
* Multi-context support for individual clients.
* Random/variable batch length.
* Load balancing plug-in.
* ... ?
gem_wsim updates
--- benchmarks/Makefile.sources | 1 + benchmarks/gem_wsim.c | 593 ++++++++++++++++++++++++++++++++++++++++++++ benchmarks/wsim/workload1 | 7 + 3 files changed, 601 insertions(+) create mode 100644 benchmarks/gem_wsim.c create mode 100644 benchmarks/wsim/workload1
benchmarks/Makefile.sources +1 -0
@@ -14,6 +14,7 @@ benchmarks_prog_list = \ gem_prw \ gem_set_domain \ gem_syslatency \ + gem_wsim \ kms_vblank \ prime_lookup \ vgem_mmap \
benchmarks/gem_wsim.c new file +593 -0
593 lines not on this page · raw message
benchmarks/wsim/workload1 new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS1.3000.0.1 +1.RCS.1000.-1.0 +1.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS2.2300.-2.0 +1.RCS.4700.-1.0 +1.VCS2.600.-1.1
Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t 1/2] benchmarks/gem_wsim: Command submission workload simulator
Quoted 14 lines · Tvrtko Ursulin
From: Tvrtko Ursulin <[email protected]>
Tool which emits batch buffers to engines with configurable sequences, durations, contexts, dependencies and userspace waits.
Unfinished but shows promise so sending out for early feedback.
v2:
* Load workload descriptors from files. (also -w)
* Help text.
* Calibration control if needed. (-t)
* NORELOC | LUT to eb flags.
* Added sample workload to wsim/workload1.
Quoted 4 lines
TODO list:
* Better error handling.
* Multi-context support for individual clients.
I think that will also wants multiple dependencies.
Quoted 3 lines
* Random/variable batch length.
* Load balancing plug-in.
* ... ?
Waits and delayed execution cycles.
Multiple clients (as threads).
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v3 1/1 benchmarks/gem_wsim: Command submission workload simulator
In reply to Tvrtko UrsulinFrom: Tvrtko Ursulin <[email protected]>
Tool which emits batch buffers to engines with configurable sequences, durations, contexts, dependencies and userspace waits.
Unfinished but shows promise so sending out for early feedback.
v2:
* Load workload descriptors from files. (also -w)
* Help text.
* Calibration control if needed. (-t)
* NORELOC | LUT to eb flags.
* Added sample workload to wsim/workload1.
v3:
* Multiple parallel different workloads (-w -w ...).
* Multi-context workloads.
* Variable (random) batch length.
* Load balancing (round robin and queue depth estimation).
* Workloads delays and explicit sync steps.
* Workload frequency (period) control.
TODO list:
* Fence support.
* Move majority of help text to README.
* Better error handling.
* Less 1980's workload parsing.
* Proper workloads.
* Explicit waits?
* Threads?
* ... ?
--- benchmarks/Makefile.sources | 1 + benchmarks/gem_wsim.c | 1053 +++++++++++++++++++++++++++++++++++++++++++ benchmarks/wsim/workload1 | 7 + benchmarks/wsim/workload2 | 7 + benchmarks/wsim/workload3 | 7 + benchmarks/wsim/workload4 | 8 + benchmarks/wsim/workload5 | 8 + benchmarks/wsim/workload6 | 8 + 8 files changed, 1099 insertions(+) create mode 100644 benchmarks/gem_wsim.c create mode 100644 benchmarks/wsim/workload1 create mode 100644 benchmarks/wsim/workload2 create mode 100644 benchmarks/wsim/workload3 create mode 100644 benchmarks/wsim/workload4 create mode 100644 benchmarks/wsim/workload5 create mode 100644 benchmarks/wsim/workload6
benchmarks/Makefile.sources +1 -0
@@ -14,6 +14,7 @@ benchmarks_prog_list = \ gem_prw \ gem_set_domain \ gem_syslatency \ + gem_wsim \ kms_vblank \ prime_lookup \ vgem_mmap \
benchmarks/gem_wsim.c new file +1053 -0
1,053 lines not on this page · raw message
benchmarks/wsim/workload1 new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS1.3000.0.1 +1.RCS.1000.-1.0 +1.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS2.2300.-2.0 +1.RCS.4700.-1.0 +1.VCS2.600.-1.1
benchmarks/wsim/workload2 new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS.3000.0.1 +1.RCS.1000.-1.0 +1.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS.2300.-2.0 +1.RCS.4700.-1.0 +1.VCS.600.-1.1
benchmarks/wsim/workload3 new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS.3000.0.0 +1.RCS.500-1500.-1.0 +0.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS.2300.-2.0 +2.RCS.4700.-1.0 +1.VCS.600.-1.0
benchmarks/wsim/workload4 new file +8 -0
@@ -0,0 +1,8 @@ +1.VCS.3000.0.0 +1.RCS.500-1500.-1.0 +d.1000 +0.RCS.3700.0.0 +1.RCS.1000.-3.0 +1.VCS.2300.-2.0 +2.RCS.4700.-1.0 +1.VCS.600.-1.0
benchmarks/wsim/workload5 new file +8 -0
@@ -0,0 +1,8 @@ +1.VCS.3000.0.0 +1.RCS.500-1500.-1.0 +0.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS.2300.-2.0 +2.RCS.4700.-1.0 +1.VCS.600.-1.0 +p.16000
benchmarks/wsim/workload6 new file +8 -0
@@ -0,0 +1,8 @@ +1.VCS.3000.0.0 +1.RCS.500-1500.-1.0 +s.-1 +0.RCS.3700.0.0 +1.RCS.1000.-3.0 +1.VCS.2300.-2.0 +2.RCS.4700.-1.0 +1.VCS.600.-1.0
Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v3] benchmarks/gem_wsim: Command submission workload simulator
Quoted 50 lines · Tvrtko Ursulin
benchmarks/gem_wsim.c new file +50
-0+static void +__emit_bb_end (struct w_step *w,bool terminate,bool seqnos,uint32_t seqno) +{ +const uint32_t bbe =0xa <<23 ; +unsigned long bb_sz =get_bb_sz (&w->duration); +unsigned long mmap_start, cmd_offset, mmap_len; +uint32_t *ptr, *cs; + + mmap_len = (seqnos ?5 :1 ) *sizeof (uint32_t ); + cmd_offset = bb_sz - mmap_len; + mmap_start =rounddown (cmd_offset,PAGE_SIZE ); + mmap_len += cmd_offset - mmap_start; + +gem_set_domain (fd, w->bb_handle, +I915_GEM_DOMAIN_CPU ,I915_GEM_DOMAIN_CPU ); + + ptr =gem_mmap__cpu (fd, w->bb_handle, mmap_start, mmap_len,PROT_WRITE ); + cs = (uint32_t *)((char *)ptr + cmd_offset - mmap_start); + +if (seqnos) { +const int gen =intel_gen (intel_get_drm_devid (fd)); + +igt_assert (gen >=8 ); + + w->reloc.offset = bb_sz -4 *sizeof (uint32_t ); + w->seqno_offset = bb_sz -2 *sizeof (uint32_t ); + + *cs++ = terminate ?MI_STORE_DWORD_IMM :0 ; + *cs++ =0 ; + *cs++ =0 ; + *cs++ = seqno; + } + + *cs = terminate ? bbe :0 ; + +munmap (ptr, mmap_len); +} + +static void terminate_bb (struct w_step *w,bool seqnos,uint32_t seqno) +{ +__emit_bb_end (w, true, seqnos, seqno); +} + +static void unterminate_bb (struct w_step *w,bool seqnos) +{ +__emit_bb_end (w, false, seqnos,0 ); +} + +static void +prepare_workload (struct workload *wrk,bool swap_vcs,bool seqnos)
benchmarks/gem_wsim.c line 444
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+{ +int max_ctx =-1 ; +struct w_step *w; +int i; + +if (seqnos) { +const unsigned int status_sz =sizeof (uint32_t ); + +for (i =0 ; i <NUM_ENGINES ; i++) { + wrk->status_page_handle[i] =gem_create (fd, status_sz);
Need to set_cache_level(CACHED) for llc.
You can use one page for all engines. Just use a different cacheline for each, for safety.
Quoted 33 lines
benchmarks/gem_wsim.c new file +33
-0+ wrk->status_page[i] = +gem_mmap__cpu (fd, wrk->status_page_handle[i], +0 , status_sz,PROT_READ ); + } + } + +for (i =0 , w = wrk->steps; i < wrk->nr_steps; i++, w++) { +if ((int )w->context > max_ctx) { +int delta = w->context +1 - wrk->nr_ctxs; + + wrk->nr_ctxs += delta; + wrk->ctx_id =realloc (wrk->ctx_id, + wrk->nr_ctxs *sizeof (uint32_t )); +memset (&wrk->ctx_id[wrk->nr_ctxs - delta],0 , + delta *sizeof (uint32_t )); + + max_ctx = w->context; + } + +if (!wrk->ctx_id[w->context]) { +struct drm_i915_gem_context_create arg = {}; + +drmIoctl (fd,DRM_IOCTL_I915_GEM_CONTEXT_CREATE , &arg); +igt_assert (arg.ctx_id); + + wrk->ctx_id[w->context] = arg.ctx_id; + } + } + +for (i =0 , w = wrk->steps; i < wrk->nr_steps; i++, w++) { +enum intel_engine_id engine = w->engine; +unsigned int bb_i, j =0 ; +
benchmarks/gem_wsim.c line 487
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+if (w->type !=BATCH ) +continue ; + + w->obj[j].handle =gem_create (fd,4096 ); + w->obj[j].flags =EXEC_OBJECT_WRITE ; + j++; + +if (seqnos) { + w->obj[j].handle = wrk->status_page_handle[engine]; + w->obj[j].flags =EXEC_OBJECT_WRITE ;
The trick for sharing between engines is to not mark this as a WRITE. Fun little lies.
benchmarks/gem_wsim.c line 500
Quoted 13 lines
benchmarks/gem_wsim.c new file +13
-0+ j++; + } + + bb_i = j++; + w->duration.cur = w->duration.max; + w->bb_sz =get_bb_sz (&w->duration); + w->bb_handle = w->obj[bb_i].handle =gem_create (fd, w->bb_sz); +terminate_bb (w, seqnos,0 ); +if (seqnos) { + w->reloc.presumed_offset =-1 ; + w->reloc.target_handle =1 ; + w->reloc.read_domains =I915_GEM_DOMAIN_INSTRUCTION ; + w->reloc.write_domain =I915_GEM_DOMAIN_INSTRUCTION ;
Ugh. That's a magic w/a value for pipecontrols. Fortunately we don't want to set write_domain here anyway.
Quoted 23 lines
benchmarks/gem_wsim.c new file +23
-0+ } + +igt_assert (w->dependency <=0 ); +if (w->dependency) { +int dep_idx = i + w->dependency; + +igt_assert (dep_idx >=0 && dep_idx < wrk->nr_steps); +igt_assert (wrk->steps[dep_idx].type ==BATCH ); + + w->obj[j].handle = w->obj[bb_i].handle; + bb_i = j; + w->obj[j -1 ].handle = + wrk->steps[dep_idx].obj[0 ].handle; + j++; + } + +if (seqnos) { + w->obj[bb_i].relocs_ptr =to_user_pointer (&w->reloc); + w->obj[bb_i].relocation_count =1 ; + } + + w->eb.buffers_ptr =to_user_pointer (w->obj); + w->eb.buffer_count = j;
benchmarks/gem_wsim.c line 533
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ w->eb.rsvd1 = wrk->ctx_id[w->context]; + +if (swap_vcs && engine ==VCS1 ) + engine =VCS2 ; +else if (swap_vcs && engine ==VCS2 ) + engine =VCS1 ; + w->eb.flags = eb_engine_map[engine]; + w->eb.flags |=I915_EXEC_HANDLE_LUT ; +if (!seqnos) + w->eb.flags |=I915_EXEC_NO_RELOC ;
Doesn't look too hard to get the relocation right. Forcing relocations between batches is probably a good one to check (just to say don't do that)
Quoted 8 lines
benchmarks/gem_wsim.c new file +8
-0+#ifdef DEBUG +printf ("%u: %u:%x|%x|%x|%x %10lu flags=%llx bb=%x[%u] ctx[%u]=%u\n" , + i, w->eb.buffer_count, w->obj[0 ].handle, + w->obj[1 ].handle, w->obj[2 ].handle, w->obj[3 ].handle, + w->bb_sz, w->eb.flags, w->bb_handle, bb_i, + w->context, wrk->ctx_id[w->context]); +#endif + }
benchmarks/gem_wsim.c line 551
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+} + +static double elapsed (const struct timespec *start,const struct timespec *end) +{ +return (end->tv_sec - start->tv_sec) + + (end->tv_nsec - start->tv_nsec) /1e9 ; +} + +static int elapsed_us (const struct timespec *start,const struct timespec *end) +{
return 1e6 * elapsed(); might as well use gcc for something!
Quoted 39 lines
benchmarks/gem_wsim.c new file +39
-0+return (1e9 * (end->tv_sec - start->tv_sec) + + (end->tv_nsec - start->tv_nsec)) /1e3 ; +} + +static enum intel_engine_id +rr_balance (struct workload *wrk,struct w_step *w) +{ +unsigned int engine; + +if (wrk->vcs_rr) + engine =VCS2 ; +else + engine =VCS1 ; + + wrk->vcs_rr ^=1 ; + +return engine; +} + +static enum intel_engine_id +qd_balance (struct workload *wrk,struct w_step *w) +{ +unsigned long qd[NUM_ENGINES ]; +enum intel_engine_id engine = w->engine; + +igt_assert (engine ==VCS ); + + qd[VCS1 ] = wrk->seqno[VCS1 ] - wrk->status_page[VCS1 ][0 ]; + wrk->qd_sum[VCS1 ] += qd[VCS1 ]; + + qd[VCS2 ] = wrk->seqno[VCS2 ] - wrk->status_page[VCS2 ][0 ]; + wrk->qd_sum[VCS2 ] += qd[VCS2 ]; + +if (qd[VCS1 ] < qd[VCS2 ]) { + engine =VCS1 ; + wrk->vcs_rr =0 ; + }else if (qd[VCS2 ] < qd[VCS1 ]) { + engine =VCS2 ; + wrk->vcs_rr =1 ;
benchmarks/gem_wsim.c line 600
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ }else { +unsigned int vcs = wrk->vcs_rr ^1 ; + + wrk->vcs_rr = vcs; + +if (vcs ==0 ) + engine =VCS1 ; +else + engine =VCS2 ; + }
Hmm. Just thinking we don't even need hw to simulate a load-balancer, but that would be boring!
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+// printf("qd_balance: 1:%lu 2:%lu rr:%u = %u\n", qd[VCS1], qd[VCS2], wrk->vcs_rr, engine); + +return engine; +} + +static void update_bb_seqno (struct w_step *w,uint32_t seqno) +{ +unsigned long mmap_start, mmap_offset, mmap_len; +void *ptr; +
benchmarks/gem_wsim.c line 621
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ mmap_start =rounddown (w->seqno_offset,PAGE_SIZE ); + mmap_offset = w->seqno_offset - mmap_start; + mmap_len =sizeof (uint32_t ) + mmap_offset; + +gem_set_domain (fd, w->bb_handle, +I915_GEM_DOMAIN_CPU ,I915_GEM_DOMAIN_CPU ); + + ptr =gem_mmap__cpu (fd, w->bb_handle, mmap_start, mmap_len,PROT_WRITE ); + + *(uint32_t *)((char *)ptr + mmap_offset) = seqno;
Uh oh. I hope this isn't called inside any loop. Note this is unsynchronized to the gpu so I wonder what this is for.
Quoted 67 lines
benchmarks/gem_wsim.c new file +67
-0+ +munmap (ptr, mmap_len); +} + +static void +run_workload (unsigned int id,struct workload *wrk,unsigned int repeat, +enum intel_engine_id (*balance)(struct workload *wrk, +struct w_step *w),bool seqnos) +{ +struct timespec t_start, t_end; +struct w_step *w; +double t; +int i, j; + +clock_gettime (CLOCK_MONOTONIC , &t_start); + +srand (t_start.tv_nsec); + +for (j =0 ; j < repeat; j++) { +for (i =0 , w = wrk->steps; i < wrk->nr_steps; i++, w++) { +enum intel_engine_id engine = w->engine; +uint32_t seqno; +bool seqno_updated = false; +int do_sleep =0 ; + +if (i ==0 ) +clock_gettime (CLOCK_MONOTONIC , + &wrk->repeat_start); + +if (w->type ==DELAY ) { + do_sleep = w->wait; + }else if (w->type ==PERIOD ) { +struct timespec now; + +clock_gettime (CLOCK_MONOTONIC , &now); + do_sleep = w->wait - +elapsed_us (&wrk->repeat_start, &now); +if (do_sleep <0 ) { +if (!quiet) { +printf ("%u: Dropped period @ %u/%u (%dus late)!\n" , + id, j, i, do_sleep); +continue ; + } + } + }else if (w->type ==SYNC ) { +unsigned int s_idx = i + w->wait; + +igt_assert (i >0 && i < wrk->nr_steps); +igt_assert (wrk->steps[s_idx].type ==BATCH ); +gem_sync (fd, wrk->steps[s_idx].obj[0 ].handle); +continue ; + } + +if (do_sleep) { +usleep (do_sleep); +continue ; + } + + wrk->nr_bb[engine]++; + +if (engine ==VCS && balance) { + engine =balance (wrk, w); + wrk->nr_bb[engine]++; + + w->obj[1 ].handle = wrk->status_page_handle[engine]; + + w->eb.flags = eb_engine_map[engine];
benchmarks/gem_wsim.c line 698
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ w->eb.flags |=I915_EXEC_HANDLE_LUT ; + } + + seqno = ++wrk->seqno[engine]; + +if (w->duration.min != w->duration.max) { +unsigned int cur =get_duration (&w->duration); + +if (cur != w->duration.cur) { +unterminate_bb (w, seqnos);
Ah, you said this was for adjusting runlength of the batches. I suggest using batch_start_offset to change the number of nops rather than rewrite the batch.
I need to study this a bit more...
-Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v3] benchmarks/gem_wsim: Command submission workload simulator
Quoted 105 lines · Chris Wilson
Quoted 50 lines · Tvrtko Ursulin
benchmarks/gem_wsim.c new file +50
-0+static void +__emit_bb_end (struct w_step *w,bool terminate,bool seqnos,uint32_t seqno) +{ +const uint32_t bbe =0xa <<23 ; +unsigned long bb_sz =get_bb_sz (&w->duration); +unsigned long mmap_start, cmd_offset, mmap_len; +uint32_t *ptr, *cs; + + mmap_len = (seqnos ?5 :1 ) *sizeof (uint32_t ); + cmd_offset = bb_sz - mmap_len; + mmap_start =rounddown (cmd_offset,PAGE_SIZE ); + mmap_len += cmd_offset - mmap_start; + +gem_set_domain (fd, w->bb_handle, +I915_GEM_DOMAIN_CPU ,I915_GEM_DOMAIN_CPU ); + + ptr =gem_mmap__cpu (fd, w->bb_handle, mmap_start, mmap_len,PROT_WRITE ); + cs = (uint32_t *)((char *)ptr + cmd_offset - mmap_start); + +if (seqnos) { +const int gen =intel_gen (intel_get_drm_devid (fd)); + +igt_assert (gen >=8 ); + + w->reloc.offset = bb_sz -4 *sizeof (uint32_t ); + w->seqno_offset = bb_sz -2 *sizeof (uint32_t ); + + *cs++ = terminate ?MI_STORE_DWORD_IMM :0 ; + *cs++ =0 ; + *cs++ =0 ; + *cs++ = seqno; + } + + *cs = terminate ? bbe :0 ; + +munmap (ptr, mmap_len); +} + +static void terminate_bb (struct w_step *w,bool seqnos,uint32_t seqno) +{ +__emit_bb_end (w, true, seqnos, seqno); +} + +static void unterminate_bb (struct w_step *w,bool seqnos) +{ +__emit_bb_end (w, false, seqnos,0 ); +} + +static void +prepare_workload (struct workload *wrk,bool swap_vcs,bool seqnos)benchmarks/gem_wsim.c line 444
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+{ +int max_ctx =-1 ; +struct w_step *w; +int i; + +if (seqnos) { +const unsigned int status_sz =sizeof (uint32_t ); + +for (i =0 ; i <NUM_ENGINES ; i++) { + wrk->status_page_handle[i] =gem_create (fd, status_sz);Need to
set_cache_level(CACHED)for llc.You can use one page for all engines. Just use a different cacheline for each, for safety.
Quoted 38 lines
benchmarks/gem_wsim.c new file +38
-0+ wrk->status_page[i] = +gem_mmap__cpu (fd, wrk->status_page_handle[i], +0 , status_sz,PROT_READ ); + } + } + +for (i =0 , w = wrk->steps; i < wrk->nr_steps; i++, w++) { +if ((int )w->context > max_ctx) { +int delta = w->context +1 - wrk->nr_ctxs; + + wrk->nr_ctxs += delta; + wrk->ctx_id =realloc (wrk->ctx_id, + wrk->nr_ctxs *sizeof (uint32_t )); +memset (&wrk->ctx_id[wrk->nr_ctxs - delta],0 , + delta *sizeof (uint32_t )); + + max_ctx = w->context; + } + +if (!wrk->ctx_id[w->context]) { +struct drm_i915_gem_context_create arg = {}; + +drmIoctl (fd,DRM_IOCTL_I915_GEM_CONTEXT_CREATE , &arg); +igt_assert (arg.ctx_id); + + wrk->ctx_id[w->context] = arg.ctx_id; + } + } + +for (i =0 , w = wrk->steps; i < wrk->nr_steps; i++, w++) { +enum intel_engine_id engine = w->engine; +unsigned int bb_i, j =0 ; + +if (w->type !=BATCH ) +continue ; + + w->obj[j].handle =gem_create (fd,4096 ); + w->obj[j].flags =EXEC_OBJECT_WRITE ;
Quoted 8 lines
benchmarks/gem_wsim.c line 487
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+ j++; + +if (seqnos) { + w->obj[j].handle = wrk->status_page_handle[engine]; + w->obj[j].flags =EXEC_OBJECT_WRITE ;The trick for sharing between engines is to not mark this as a WRITE. Fun little lies.
Yeah thats why I have per-engine objects. Which I don't mind since it is not like they are wasting any resources compared to everything else. But not admitting the write sounds still interesting. What would the repercussions of that be - limit us to llc platforms or something?
Quoted 8 lines
Quoted 8 lines
benchmarks/gem_wsim.c new file +8
-0+ j++; + } + + bb_i = j++; + w->duration.cur = w->duration.max; + w->bb_sz =get_bb_sz (&w->duration); + w->bb_handle = w->obj[bb_i].handle =gem_create (fd, w->bb_sz); +terminate_bb (w, seqnos,0 );
Quoted 8 lines
benchmarks/gem_wsim.c line 500
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+if (seqnos) { + w->reloc.presumed_offset =-1 ; + w->reloc.target_handle =1 ; + w->reloc.read_domains =I915_GEM_DOMAIN_INSTRUCTION ; + w->reloc.write_domain =I915_GEM_DOMAIN_INSTRUCTION ;Ugh. That's a magic w/a value for pipecontrols. Fortunately we don't want to set
write_domainhere anyway.
I think I copy-pasted this from another IGT. So you say cheat here as well and set zero for both domains?
Quoted 30 lines
Quoted 30 lines
benchmarks/gem_wsim.c new file +29
-0+ } + +igt_assert (w->dependency <=0 ); +if (w->dependency) { +int dep_idx = i + w->dependency; + +igt_assert (dep_idx >=0 && dep_idx < wrk->nr_steps); +igt_assert (wrk->steps[dep_idx].type ==BATCH ); + + w->obj[j].handle = w->obj[bb_i].handle; + bb_i = j; + w->obj[j -1 ].handle = + wrk->steps[dep_idx].obj[0 ].handle; + j++; + } + +if (seqnos) { + w->obj[bb_i].relocs_ptr =to_user_pointer (&w->reloc); + w->obj[bb_i].relocation_count =1 ; + } + + w->eb.buffers_ptr =to_user_pointer (w->obj); + w->eb.buffer_count = j; + w->eb.rsvd1 = wrk->ctx_id[w->context]; + +if (swap_vcs && engine ==VCS1 ) + engine =VCS2 ; +else if (swap_vcs && engine ==VCS2 ) + engine =VCS1 ;
Quoted 8 lines
benchmarks/gem_wsim.c line 533
Quoted 4 lines
benchmarks/gem_wsim.c new file +4
-0+ w->eb.flags = eb_engine_map[engine]; + w->eb.flags |=I915_EXEC_HANDLE_LUT ; +if (!seqnos) + w->eb.flags |=I915_EXEC_NO_RELOC ;Doesn't look too hard to get the relocation right. Forcing relocations between batches is probably a good one to check (just to say don't do that)
I am not following here? You are saying don't do relocations at all? How do I make sure things stay fixed and even how to find out where they are in the first pass?
Quoted 65 lines
Quoted 8 lines
benchmarks/gem_wsim.c new file +8
-0+#ifdef DEBUG +printf ("%u: %u:%x|%x|%x|%x %10lu flags=%llx bb=%x[%u] ctx[%u]=%u\n" , + i, w->eb.buffer_count, w->obj[0 ].handle, + w->obj[1 ].handle, w->obj[2 ].handle, w->obj[3 ].handle, + w->bb_sz, w->eb.flags, w->bb_handle, bb_i, + w->context, wrk->ctx_id[w->context]); +#endif + }benchmarks/gem_wsim.c line 551
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+} + +static double elapsed (const struct timespec *start,const struct timespec *end) +{ +return (end->tv_sec - start->tv_sec) + + (end->tv_nsec - start->tv_nsec) /1e9 ; +} + +static int elapsed_us (const struct timespec *start,const struct timespec *end) +{return 1e6 *
elapsed(); might as well use gcc for something!Quoted 44 lines
benchmarks/gem_wsim.c new file +44
-0+return (1e9 * (end->tv_sec - start->tv_sec) + + (end->tv_nsec - start->tv_nsec)) /1e3 ; +} + +static enum intel_engine_id +rr_balance (struct workload *wrk,struct w_step *w) +{ +unsigned int engine; + +if (wrk->vcs_rr) + engine =VCS2 ; +else + engine =VCS1 ; + + wrk->vcs_rr ^=1 ; + +return engine; +} + +static enum intel_engine_id +qd_balance (struct workload *wrk,struct w_step *w) +{ +unsigned long qd[NUM_ENGINES ]; +enum intel_engine_id engine = w->engine; + +igt_assert (engine ==VCS ); + + qd[VCS1 ] = wrk->seqno[VCS1 ] - wrk->status_page[VCS1 ][0 ]; + wrk->qd_sum[VCS1 ] += qd[VCS1 ]; + + qd[VCS2 ] = wrk->seqno[VCS2 ] - wrk->status_page[VCS2 ][0 ]; + wrk->qd_sum[VCS2 ] += qd[VCS2 ]; + +if (qd[VCS1 ] < qd[VCS2 ]) { + engine =VCS1 ; + wrk->vcs_rr =0 ; + }else if (qd[VCS2 ] < qd[VCS1 ]) { + engine =VCS2 ; + wrk->vcs_rr =1 ; + }else { +unsigned int vcs = wrk->vcs_rr ^1 ; + + wrk->vcs_rr = vcs; +
Quoted 8 lines
benchmarks/gem_wsim.c line 600
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+if (vcs ==0 ) + engine =VCS1 ; +else + engine =VCS2 ; + }Hmm. Just thinking we don't even need hw to simulate a load-balancer, but that would be boring!
Definitely not as exciting. :) Perhaps there will be other benefits from this tool than the load balancing, but we'll see.
Quoted 15 lines
Quoted 15 lines
benchmarks/gem_wsim.c new file +15
-0+// printf("qd_balance: 1:%lu 2:%lu rr:%u = %u\n", qd[VCS1], qd[VCS2], wrk->vcs_rr, engine); + +return engine; +} + +static void update_bb_seqno (struct w_step *w,uint32_t seqno) +{ +unsigned long mmap_start, mmap_offset, mmap_len; +void *ptr; + + mmap_start =rounddown (w->seqno_offset,PAGE_SIZE ); + mmap_offset = w->seqno_offset - mmap_start; + mmap_len =sizeof (uint32_t ) + mmap_offset; + + gem_set_domain(fd, w->bb_handle,
Quoted 8 lines
benchmarks/gem_wsim.c line 621
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+ I915_GEM_DOMAIN_CPU, I915_GEM_DOMAIN_CPU); + + ptr =gem_mmap__cpu (fd, w->bb_handle, mmap_start, mmap_len,PROT_WRITE ); + + *(uint32_t *)((char *)ptr + mmap_offset) = seqno;Uh oh. I hope this isn't called inside any loop. Note this is unsynchronized to the gpu so I wonder what this is for.
To update the seqno inside the store_dword_imm. It is called every time before a batch is executed so I was thinking whether a gem_sync should be preceding it. But then I was thinking it is problematic in general if we queue up multiple same batches before they get executed. :( Sounds like I would need a separate batch for every iteration for this to work correctly. But that sounds too costly. So I don't know at the moment.
Quoted 73 lines
Quoted 73 lines
benchmarks/gem_wsim.c new file +73
-0+ +munmap (ptr, mmap_len); +} + +static void +run_workload (unsigned int id,struct workload *wrk,unsigned int repeat, +enum intel_engine_id (*balance)(struct workload *wrk, +struct w_step *w),bool seqnos) +{ +struct timespec t_start, t_end; +struct w_step *w; +double t; +int i, j; + +clock_gettime (CLOCK_MONOTONIC , &t_start); + +srand (t_start.tv_nsec); + +for (j =0 ; j < repeat; j++) { +for (i =0 , w = wrk->steps; i < wrk->nr_steps; i++, w++) { +enum intel_engine_id engine = w->engine; +uint32_t seqno; +bool seqno_updated = false; +int do_sleep =0 ; + +if (i ==0 ) +clock_gettime (CLOCK_MONOTONIC , + &wrk->repeat_start); + +if (w->type ==DELAY ) { + do_sleep = w->wait; + }else if (w->type ==PERIOD ) { +struct timespec now; + +clock_gettime (CLOCK_MONOTONIC , &now); + do_sleep = w->wait - +elapsed_us (&wrk->repeat_start, &now); +if (do_sleep <0 ) { +if (!quiet) { +printf ("%u: Dropped period @ %u/%u (%dus late)!\n" , + id, j, i, do_sleep); +continue ; + } + } + }else if (w->type ==SYNC ) { +unsigned int s_idx = i + w->wait; + +igt_assert (i >0 && i < wrk->nr_steps); +igt_assert (wrk->steps[s_idx].type ==BATCH ); +gem_sync (fd, wrk->steps[s_idx].obj[0 ].handle); +continue ; + } + +if (do_sleep) { +usleep (do_sleep); +continue ; + } + + wrk->nr_bb[engine]++; + +if (engine ==VCS && balance) { + engine =balance (wrk, w); + wrk->nr_bb[engine]++; + + w->obj[1 ].handle = wrk->status_page_handle[engine]; + + w->eb.flags = eb_engine_map[engine]; + w->eb.flags |=I915_EXEC_HANDLE_LUT ; + } + + seqno = ++wrk->seqno[engine]; + +if (w->duration.min != w->duration.max) {
Quoted 8 lines
benchmarks/gem_wsim.c line 698
Quoted 4 lines
benchmarks/gem_wsim.c new file +4
-0+unsigned int cur =get_duration (&w->duration); + +if (cur != w->duration.cur) { +unterminate_bb (w, seqnos);Ah, you said this was for adjusting runlength of the batches. I suggest using
batch_start_offsetto change the number of nops rather than rewrite the batch.
Yeah thanks for that suggestion, I did not think of that.
Quoted 1 line
I need to study this a bit more...
Yes please, especially the bit about how to get accurate seqnos written out in each step without needing separate execbuf batches.
I've heard recursive batches mentioned in the past so maybe each iteration could have it's own small batch which would jump to the nop/delay one (shared between all iterations) and write the unique seqno. No idea if that is possible/supported at the moment - I'll go and dig a bit.
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v3] benchmarks/gem_wsim: Command submission workload simulator
Quoted 116 lines · Tvrtko Ursulin
Quoted 105 lines · Chris Wilson
Quoted 50 lines · Tvrtko Ursulin
benchmarks/gem_wsim.c new file +50
-0+static void +__emit_bb_end (struct w_step *w,bool terminate,bool seqnos,uint32_t seqno) +{ +const uint32_t bbe =0xa <<23 ; +unsigned long bb_sz =get_bb_sz (&w->duration); +unsigned long mmap_start, cmd_offset, mmap_len; +uint32_t *ptr, *cs; + + mmap_len = (seqnos ?5 :1 ) *sizeof (uint32_t ); + cmd_offset = bb_sz - mmap_len; + mmap_start =rounddown (cmd_offset,PAGE_SIZE ); + mmap_len += cmd_offset - mmap_start; + +gem_set_domain (fd, w->bb_handle, +I915_GEM_DOMAIN_CPU ,I915_GEM_DOMAIN_CPU ); + + ptr =gem_mmap__cpu (fd, w->bb_handle, mmap_start, mmap_len,PROT_WRITE ); + cs = (uint32_t *)((char *)ptr + cmd_offset - mmap_start); + +if (seqnos) { +const int gen =intel_gen (intel_get_drm_devid (fd)); + +igt_assert (gen >=8 ); + + w->reloc.offset = bb_sz -4 *sizeof (uint32_t ); + w->seqno_offset = bb_sz -2 *sizeof (uint32_t ); + + *cs++ = terminate ?MI_STORE_DWORD_IMM :0 ; + *cs++ =0 ; + *cs++ =0 ; + *cs++ = seqno; + } + + *cs = terminate ? bbe :0 ; + +munmap (ptr, mmap_len); +} + +static void terminate_bb (struct w_step *w,bool seqnos,uint32_t seqno) +{ +__emit_bb_end (w, true, seqnos, seqno); +} + +static void unterminate_bb (struct w_step *w,bool seqnos) +{ +__emit_bb_end (w, false, seqnos,0 ); +} + +static void +prepare_workload (struct workload *wrk,bool swap_vcs,bool seqnos)benchmarks/gem_wsim.c line 444
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+{ +int max_ctx =-1 ; +struct w_step *w; +int i; + +if (seqnos) { +const unsigned int status_sz =sizeof (uint32_t ); + +for (i =0 ; i <NUM_ENGINES ; i++) { + wrk->status_page_handle[i] =gem_create (fd, status_sz);Need to
set_cache_level(CACHED)for llc.You can use one page for all engines. Just use a different cacheline for each, for safety.
Quoted 38 lines
benchmarks/gem_wsim.c new file +38
-0+ wrk->status_page[i] = +gem_mmap__cpu (fd, wrk->status_page_handle[i], +0 , status_sz,PROT_READ ); + } + } + +for (i =0 , w = wrk->steps; i < wrk->nr_steps; i++, w++) { +if ((int )w->context > max_ctx) { +int delta = w->context +1 - wrk->nr_ctxs; + + wrk->nr_ctxs += delta; + wrk->ctx_id =realloc (wrk->ctx_id, + wrk->nr_ctxs *sizeof (uint32_t )); +memset (&wrk->ctx_id[wrk->nr_ctxs - delta],0 , + delta *sizeof (uint32_t )); + + max_ctx = w->context; + } + +if (!wrk->ctx_id[w->context]) { +struct drm_i915_gem_context_create arg = {}; + +drmIoctl (fd,DRM_IOCTL_I915_GEM_CONTEXT_CREATE , &arg); +igt_assert (arg.ctx_id); + + wrk->ctx_id[w->context] = arg.ctx_id; + } + } + +for (i =0 , w = wrk->steps; i < wrk->nr_steps; i++, w++) { +enum intel_engine_id engine = w->engine; +unsigned int bb_i, j =0 ; + +if (w->type !=BATCH ) +continue ; + + w->obj[j].handle =gem_create (fd,4096 ); + w->obj[j].flags =EXEC_OBJECT_WRITE ;Quoted 8 lines
benchmarks/gem_wsim.c line 487
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+ j++; + +if (seqnos) { + w->obj[j].handle = wrk->status_page_handle[engine]; + w->obj[j].flags =EXEC_OBJECT_WRITE ;The trick for sharing between engines is to not mark this as a WRITE. Fun little lies.
Quoted 5 lines
Yeah thats why I have per-engine objects. Which I don't mind since it is not like they are wasting any resources compared to everything else. But not admitting the write sounds still interesting. What would the repercussions of that be - limit us to llc platforms or something?
It used to be that if we evicted the object (e.g. mempressure/suspend), then we would not save the contents since it was never marked as dirty. However, between libva being buggy and Vk deliberately eskewing write hazards, we had to always mark GPU usage as dirtying the buffers. So nowadays, EXEC_OBJECT_WRITE only means to track the implicit write hazard.
Quoted 18 lines
Quoted 17 lines
Quoted 4 lines
benchmarks/gem_wsim.c new file +3
-0+ j++; + } +benchmarks/gem_wsim.c line 500
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ bb_i = j++; + w->duration.cur = w->duration.max; + w->bb_sz =get_bb_sz (&w->duration); + w->bb_handle = w->obj[bb_i].handle =gem_create (fd, w->bb_sz); +terminate_bb (w, seqnos,0 ); +if (seqnos) { + w->reloc.presumed_offset =-1 ; + w->reloc.target_handle =1 ; + w->reloc.read_domains =I915_GEM_DOMAIN_INSTRUCTION ; + w->reloc.write_domain =I915_GEM_DOMAIN_INSTRUCTION ;Ugh. That's a magic w/a value for pipecontrols. Fortunately we don't want to set
write_domainhere anyway.
Quoted 2 lines
I think I copy-pasted this from another IGT. So you say cheat here as well and set zero for both domains?
Technically the MI is outside of all the GPU cache domains we have :) Which you pick is immaterial, aside from understanding that (INSTRUCTION, INSTRUCTION) is special ;)
If you were to drop EXEC_OBJECT_WRITE, you would also drop reloc.write_domain.
Quoted 38 lines
Quoted 29 lines
Quoted 29 lines
benchmarks/gem_wsim.c new file +29
-0+ } + +igt_assert (w->dependency <=0 ); +if (w->dependency) { +int dep_idx = i + w->dependency; + +igt_assert (dep_idx >=0 && dep_idx < wrk->nr_steps); +igt_assert (wrk->steps[dep_idx].type ==BATCH ); + + w->obj[j].handle = w->obj[bb_i].handle; + bb_i = j; + w->obj[j -1 ].handle = + wrk->steps[dep_idx].obj[0 ].handle; + j++; + } + +if (seqnos) { + w->obj[bb_i].relocs_ptr =to_user_pointer (&w->reloc); + w->obj[bb_i].relocation_count =1 ; + } + + w->eb.buffers_ptr =to_user_pointer (w->obj); + w->eb.buffer_count = j; + w->eb.rsvd1 = wrk->ctx_id[w->context]; + +if (swap_vcs && engine ==VCS1 ) + engine =VCS2 ; +else if (swap_vcs && engine ==VCS2 ) + engine =VCS1 ;Quoted 8 lines
benchmarks/gem_wsim.c line 533
Quoted 4 lines
benchmarks/gem_wsim.c new file +4
-0+ w->eb.flags = eb_engine_map[engine]; + w->eb.flags |=I915_EXEC_HANDLE_LUT ; +if (!seqnos) + w->eb.flags |=I915_EXEC_NO_RELOC ;Doesn't look too hard to get the relocation right. Forcing relocations between batches is probably a good one to check (just to say don't do that)
Quoted 3 lines
I am not following here? You are saying don't do relocations at all? How do I make sure things stay fixed and even how to find out where they are in the first pass?
Depending on the workload, it may be informative to also do comparisons between NORELOC and always RELOC. Personally I would make sure we were using NORELOC as this should be a simulator/example.
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+static void update_bb_seqno (struct w_step *w,uint32_t seqno) +{ +unsigned long mmap_start, mmap_offset, mmap_len; +void *ptr; + + mmap_start =rounddown (w->seqno_offset,PAGE_SIZE ); + mmap_offset = w->seqno_offset - mmap_start; + mmap_len =sizeof (uint32_t ) + mmap_offset; + + gem_set_domain(fd, w->bb_handle,Quoted 8 lines
benchmarks/gem_wsim.c line 621
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+ I915_GEM_DOMAIN_CPU, I915_GEM_DOMAIN_CPU); + + ptr =gem_mmap__cpu (fd, w->bb_handle, mmap_start, mmap_len,PROT_WRITE ); + + *(uint32_t *)((char *)ptr + mmap_offset) = seqno;Uh oh. I hope this isn't called inside any loop. Note this is unsynchronized to the gpu so I wonder what this is for.
Quoted 7 lines
To update the seqno inside the
store_dword_imm. It is called every time before a batch is executed so I was thinking whether agem_syncshould be preceding it. But then I was thinking it is problematic in general if we queue up multiple same batches before they get executed. :( Sounds like I would need a separate batch for every iteration for this to work correctly. But that sounds too costly. So I don't know at the moment.
mmap/munmap, especially munmap, is not free. The munmap will do a tlb_flush across all cores -- though maybe that's batched and the munmaps I do all tend to be large enough to trigger every time.
Since you are using a CPU write, on !llc this will be clflushing everytime. I would suggest stashing the gem_mmap__wc for updating the seqno between repeats.
[snip]
Quoted 10 lines
Quoted 1 line
I need to study this a bit more...
Yes please, especially the bit about how to get accurate seqnos written out in each step without needing separate execbuf batches.
I've heard recursive batches mentioned in the past so maybe each iteration could have it's own small batch which would jump to the nop/delay one (shared between all iterations) and write the unique seqno. No idea if that is possible/supported at the moment - I'll go and dig a bit.
You end up with the same problem of having the reloc change and need to update every cycle. You could use a fresh batch to rewrite the seqno values... However, now that you explained what you want, just keep the WC mmap.
-Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v3] benchmarks/gem_wsim: Command submission workload simulator
Quoted 1 line · Chris Wilson
On Thu, Apr 06, 2017 at 09:18:36AM +0100, Tvrtko Ursulin wrote:
[snip]
Quoted 20 lines
Quoted 17 lines
Quoted 8 lines
Quoted 8 lines
benchmarks/gem_wsim.c new file +8
-0+ j++; + } + + bb_i = j++; + w->duration.cur = w->duration.max; + w->bb_sz =get_bb_sz (&w->duration); + w->bb_handle = w->obj[bb_i].handle =gem_create (fd, w->bb_sz); +terminate_bb (w, seqnos,0 );Quoted 8 lines
benchmarks/gem_wsim.c line 500
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+if (seqnos) { + w->reloc.presumed_offset =-1 ; + w->reloc.target_handle =1 ; + w->reloc.read_domains =I915_GEM_DOMAIN_INSTRUCTION ; + w->reloc.write_domain =I915_GEM_DOMAIN_INSTRUCTION ;Ugh. That's a magic w/a value for pipecontrols. Fortunately we don't want to set
write_domainhere anyway.Quoted 2 lines
I think I copy-pasted this from another IGT. So you say cheat here as well and set zero for both domains?
Quoted 6 lines
Technically the MI is outside of all the GPU cache domains we have :) Which you pick is immaterial, aside from understanding that (INSTRUCTION, INSTRUCTION) is special ;)
If you were to drop
EXEC_OBJECT_WRITE, you would also drop reloc.write_domain.
Okay, I will try the cheating approach then.
Quoted 38 lines
Quoted 38 lines
Quoted 37 lines
Quoted 23 lines
benchmarks/gem_wsim.c new file +23
-0+ } + +igt_assert (w->dependency <=0 ); +if (w->dependency) { +int dep_idx = i + w->dependency; + +igt_assert (dep_idx >=0 && dep_idx < wrk->nr_steps); +igt_assert (wrk->steps[dep_idx].type ==BATCH ); + + w->obj[j].handle = w->obj[bb_i].handle; + bb_i = j; + w->obj[j -1 ].handle = + wrk->steps[dep_idx].obj[0 ].handle; + j++; + } + +if (seqnos) { + w->obj[bb_i].relocs_ptr =to_user_pointer (&w->reloc); + w->obj[bb_i].relocation_count =1 ; + } + + w->eb.buffers_ptr =to_user_pointer (w->obj); + w->eb.buffer_count = j;benchmarks/gem_wsim.c line 533
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ w->eb.rsvd1 = wrk->ctx_id[w->context]; + +if (swap_vcs && engine ==VCS1 ) + engine =VCS2 ; +else if (swap_vcs && engine ==VCS2 ) + engine =VCS1 ; + w->eb.flags = eb_engine_map[engine]; + w->eb.flags |=I915_EXEC_HANDLE_LUT ; +if (!seqnos) + w->eb.flags |=I915_EXEC_NO_RELOC ;Doesn't look too hard to get the relocation right. Forcing relocations between batches is probably a good one to check (just to say don't do that)
Quoted 7 lines
Quoted 3 lines
I am not following here? You are saying don't do relocations at all? How do I make sure things stay fixed and even how to find out where they are in the first pass?
Depending on the workload, it may be informative to also do comparisons between NORELOC and always RELOC. Personally I would make sure we were using NORELOC as this should be a simulator/example.
How do I use NORELOC? I mean, I have to know where to objects will be pinned, or be able to pin them first and know they will remain put. What am I not understanding here?
Quoted 27 lines
Quoted 19 lines
Quoted 10 lines
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+static void update_bb_seqno (struct w_step *w,uint32_t seqno) +{ +unsigned long mmap_start, mmap_offset, mmap_len; +void *ptr; + + mmap_start =rounddown (w->seqno_offset,PAGE_SIZE ); + mmap_offset = w->seqno_offset - mmap_start; + mmap_len =sizeof (uint32_t ) + mmap_offset; + + gem_set_domain(fd, w->bb_handle,Quoted 8 lines
benchmarks/gem_wsim.c line 621
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+ I915_GEM_DOMAIN_CPU, I915_GEM_DOMAIN_CPU); + + ptr =gem_mmap__cpu (fd, w->bb_handle, mmap_start, mmap_len,PROT_WRITE ); + + *(uint32_t *)((char *)ptr + mmap_offset) = seqno;Uh oh. I hope this isn't called inside any loop. Note this is unsynchronized to the gpu so I wonder what this is for.
Quoted 7 lines
To update the seqno inside the
store_dword_imm. It is called every time before a batch is executed so I was thinking whether agem_syncshould be preceding it. But then I was thinking it is problematic in general if we queue up multiple same batches before they get executed. :( Sounds like I would need a separate batch for every iteration for this to work correctly. But that sounds too costly. So I don't know at the moment.
Quoted 7 lines
mmap/munmap, especially munmap, is not free. The munmap will do a
tlb_flushacross all cores -- though maybe that's batched and the munmaps I do all tend to be large enough to trigger every time.Since you are using a CPU write, on !llc this will be clflushing everytime. I would suggest stashing the
gem_mmap__wcfor updating the seqno between repeats.
Ok, I can try that approach.
Quoted 13 lines
[snip]
Quoted 10 lines
Quoted 1 line
I need to study this a bit more...
Yes please, especially the bit about how to get accurate seqnos written out in each step without needing separate execbuf batches.
I've heard recursive batches mentioned in the past so maybe each iteration could have it's own small batch which would jump to the nop/delay one (shared between all iterations) and write the unique seqno. No idea if that is possible/supported at the moment - I'll go and dig a bit.
Quoted 4 lines
You end up with the same problem of having the reloc change and need to update every cycle. You could use a fresh batch to rewrite the seqno values... However, now that you explained what you want, just keep the WC mmap.
Hm not sure without researching that approach first.
But in general is this correctly implementing your idea for queue depth estimation?
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v3] benchmarks/gem_wsim: Command submission workload simulator
Quoted 5 lines · Tvrtko Ursulin
Quoted 1 line · Chris Wilson
On Thu, Apr 06, 2017 at 09:18:36AM +0100, Tvrtko Ursulin wrote:
[snip]
[snip]
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
benchmarks/gem_wsim.c new file +4
-0+if (swap_vcs && engine ==VCS1 ) + engine =VCS2 ; +else if (swap_vcs && engine ==VCS2 ) + engine =VCS1 ;Quoted 8 lines
benchmarks/gem_wsim.c line 533
Quoted 4 lines
benchmarks/gem_wsim.c new file +4
-0+ w->eb.flags = eb_engine_map[engine]; + w->eb.flags |=I915_EXEC_HANDLE_LUT ; +if (!seqnos) + w->eb.flags |=I915_EXEC_NO_RELOC ;Doesn't look too hard to get the relocation right. Forcing relocations between batches is probably a good one to check (just to say don't do that)
I am not following here? You are saying don't do relocations at all? How do I make sure things stay fixed and even how to find out where they are in the first pass?
Quoted 7 lines
Quoted 3 lines
Depending on the workload, it may be informative to also do comparisons between NORELOC and always RELOC. Personally I would make sure we were using NORELOC as this should be a simulator/example.
How do I use NORELOC? I mean, I have to know where to objects will be pinned, or be able to pin them first and know they will remain put. What am I not understanding here?
It will be assigned an address on first execution. Can I quote the spiel I wrote for i915_gem_execbuffer.c and see if that answers how to use NORELOC:
* Reserving resources for the execbuf is the most complicated phase. We
* neither want to have to migrate the object in the address space, nor do
* we want to have to update any relocations pointing to this object. Ideally,
* we want to leave the object where it is and for all the existing relocations
* to match. If the object is given a new address, or if userspace thinks the
* object is elsewhere, we have to parse all the relocation entries and update
* the addresses. Userspace can set the I915_EXEC_NORELOC flag to hint that
* all the target addresses in all of its objects match the value in the
* relocation entries and that they all match the presumed offsets given by the
* list of execbuffer objects. Using this knowledge, we know that if we haven't
* moved any buffers, all the relocation entries are valid and we can skip
* the update. (If userspace is wrong, the likely outcome is an impromptu GPU
* hang.) The requirement for using I915_EXEC_NO_RELOC are:
*
* The addresses written in the objects must match the corresponding * reloc.presumed_offset which in turn must match the corresponding * execobject.offset.
*
* Any render targets written to in the batch must be flagged with
* EXEC_OBJECT_WRITE.
*
* To avoid stalling, execobject.offset should match the current
* address of that object within the active context.
*
Does that make sense? How questions remain unanswered?
Hmm, I usually sum it up as
batch[reloc.offset] == reloc.presumed_offset + reloc.delta;
and
execobj.offset == reloc.presumed_offset
must be true at the time of execbuf. Note that upon relocation, batch[reloc.offset], reloc.presumed_offset and execobj.offset are updated. This is important to remember if you are prerecording the reloc/execobj arrays, and not feeding back the results of execbuf between phases.
Quoted 2 lines
But in general is this correctly implementing your idea for queue depth estimation?
From my rough checklist:
* writes engine->next_seqno++ after each op (in this case end of batch)
* qlen[engine] = engine->next_seqno - *engine->current_seqno;
Design looks right. Implementation requires checking... I'll be back. -Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 1/1 benchmarks/gem_wsim: Command submission workload simulator
In reply to Tvrtko UrsulinFrom: Tvrtko Ursulin <[email protected]>
Tool which emits batch buffers to engines with configurable sequences, durations, contexts, dependencies and userspace waits.
Unfinished but shows promise so sending out for early feedback.
v2:
* Load workload descriptors from files. (also -w)
* Help text.
* Calibration control if needed. (-t)
* NORELOC | LUT to eb flags.
* Added sample workload to wsim/workload1.
v3:
* Multiple parallel different workloads (-w -w ...).
* Multi-context workloads.
* Variable (random) batch length.
* Load balancing (round robin and queue depth estimation).
* Workloads delays and explicit sync steps.
* Workload frequency (period) control.
v4:
* Fixed queue-depth estimation by creating separate batches per engine when qd load balancing is on.
* Dropped separate -s cmd line option. It can turn itself on automatically when needed.
* Keep a single status page and lie about the write hazard as suggested by Chris.
* Use batch_start_offset for controlling the batch duration. (Chris)
* Set status page object cache level. (Chris)
* Moved workload description to a README.
* Tidied example workloads.
* Some other cleanups and refactorings.
TODO list:
* Fence support.
* Better error handling.
* Less 1980's workload parsing.
* Proper workloads.
* Threads?
* ... ?
---
Comparing some test workloads under load balancing it seems that it is starting to work, but it still needs more thorough verification. For example, round- robin balancing:
# benchmarks/gem_wsim -n 585341 \ -w benchmarks/wsim/vcs1.wsim \ -w benchmarks/wsim/vcs_balanced.wsim \ -r 100 -b 0 Using 585341 nop calibration for 1000us delay. 2 clients. 1: 3.008s elapsed (33.243 workloads/s). 2500 (1250 + 1250) total VCS batches. 0: 4.455s elapsed (22.449 workloads/s). 0 (2500 + 0) total VCS batches. 4.455s elapsed (44.889 workloads/s)
Versus the queue-depth estimation:
# benchmarks/gem_wsim -n 585341 \ -w benchmarks/wsim/vcs1.wsim \ -w benchmarks/wsim/vcs_balanced.wsim \ -r 100 -b 1 Using 585341 nop calibration for 1000us delay. 2 clients. 1: 2.239s elapsed (44.659 workloads/s). 2500 (837 + 1663) total VCS batches. Average queue depths 27.575, 19.285. 0: 4.012s elapsed (24.928 workloads/s). 0 (2500 + 0) total VCS batches. Average queue depths -nan, -nan. 4.012s elapsed (49.845 workloads/s)
In both cases we run two workloads, one which only submits to VCS1 and one which can be load-balanced. The latter gets a ~33% boost with queue-depth estimation, and the non-balancing workload ~10%.
--- benchmarks/Makefile.sources | 1 + benchmarks/gem_wsim.c | 1014 ++++++++++++++++++++++++++ benchmarks/wsim/README | 54 ++ benchmarks/wsim/media_17i7.wsim | 7 + benchmarks/wsim/media_load_balance_17i7.wsim | 7 + benchmarks/wsim/vcs1.wsim | 25 + benchmarks/wsim/vcs_balanced.wsim | 25 + 7 files changed, 1133 insertions(+) create mode 100644 benchmarks/gem_wsim.c create mode 100644 benchmarks/wsim/README create mode 100644 benchmarks/wsim/media_17i7.wsim create mode 100644 benchmarks/wsim/media_load_balance_17i7.wsim create mode 100644 benchmarks/wsim/vcs1.wsim create mode 100644 benchmarks/wsim/vcs_balanced.wsim
benchmarks/Makefile.sources +1 -0
@@ -14,6 +14,7 @@ benchmarks_prog_list = \ gem_prw \ gem_set_domain \ gem_syslatency \ + gem_wsim \ kms_vblank \ prime_lookup \ vgem_mmap \
benchmarks/gem_wsim.c new file +1014 -0
1,014 lines not on this page · raw message
benchmarks/wsim/README new file +54 -0
54 lines not on this page · raw message
benchmarks/wsim/media_17i7.wsim new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS1.3000.0.1 +1.RCS.1000.-1.0 +1.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS2.2300.-2.0 +1.RCS.4700.-1.0 +1.VCS2.600.-1.1
benchmarks/wsim/media_load_balance_17i7.wsim new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS.3000.0.1 +1.RCS.1000.-1.0 +1.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS.2300.-2.0 +1.RCS.4700.-1.0 +1.VCS.600.-1.1
benchmarks/wsim/vcs1.wsim new file +25 -0
@@ -0,0 +1,25 @@ +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0
benchmarks/wsim/vcs_balanced.wsim new file +25 -0
@@ -0,0 +1,25 @@ +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0
Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v4] benchmarks/gem_wsim: Command submission workload simulator
Quoted 33 lines · Tvrtko Ursulin
benchmarks/gem_wsim.c new file +33
-0+static void +alloc_step_batch (struct workload *wrk,struct w_step *w,struct w_step_eb *b, +enum intel_engine_id engine,unsigned int flags) +{ +unsigned int bb_i, j =0 ; + + b->obj[j].handle =gem_create (fd,4096 ); + b->obj[j].flags =EXEC_OBJECT_WRITE ; + j++; + +if (flags &SEQNO ) { + b->obj[j].handle = wrk->status_page_handle; + j++; + } + + bb_i = j++; + b->bb_sz =get_bb_sz (w->duration.max); + b->bb_handle = b->obj[bb_i].handle =gem_create (fd, b->bb_sz); +terminate_bb (w, b, engine, flags); + +igt_assert (w->dependency <=0 ); +if (w->dependency) { +int dep_idx = w->idx + w->dependency; + +igt_assert (dep_idx >=0 && dep_idx < wrk->nr_steps); +igt_assert (wrk->steps[dep_idx].type ==BATCH ); + + b->obj[j].handle = b->obj[bb_i].handle; + bb_i = j; + b->obj[j -1 ].handle = wrk->steps[dep_idx].b[0 ].obj[0 ].handle; + j++; + +if (wrk->steps[dep_idx].b[1 ].obj[0 ].handle) {
benchmarks/gem_wsim.c line 472
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ b->obj[j].handle = b->obj[bb_i].handle; + bb_i = j; + b->obj[j -1 ].handle = + wrk->steps[dep_idx].b[1 ].obj[0 ].handle; + j++; + } + } + +if (flags &SEQNO ) { + b->reloc.presumed_offset =-1 ;
So as I understand it, you are caching the execbuf/obj/reloc for the workload and then may reissue later with different seqno on different rings? In which case we have a problem as the kernel will write back the updated offsets to b->reloc.presumed_offset and b->obj[].offset and in future passes they will match and the seqno write will go into the wrong slot (if it swaps rings).
You either want to reset presumed_offset=-1 each time, or better for all concerned write the correct address alongside the seqno (which also enables NORELOC).
Delta incoming.
-Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v4] benchmarks/gem_wsim: Command submission workload simulator
Quoted 5 lines · Chris Wilson
You either want to reset presumed_offset=-1 each time, or better for all concerned write the correct address alongside the seqno (which also enables NORELOC).
Delta incoming.
See attached.
Next concern is that I have full rings which implies that we are not waiting on each batch before resubmitting with a new seqno?
If I throw a assert(!busy(batch_bo)) before the *b->mapped_seqno am I going to be upset?
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Quoted 1 line
From
5bf424c2719e81f926b74f4136610cbdfd26a4d8Mon Sep 17 00:00:00 2001
From: Chris Wilson <[email protected]>
Date: Thu, 20 Apr 2017 15:30:07 +0100
Subject: [PATCH] seqno-reloc
--- benchmarks/gem_wsim.c | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-)
benchmarks/gem_wsim.c +11 -6
@@ -45,6 +45,8 @@#include "drmtest.h" #include "intel_io.h" +#define LOCAL_I915_GEM_DOMAIN_WC 0x80 +enum intel_engine_id {RCS ,BCS ,
@@ -86,7 +88,9 @@ struct w_stepstruct drm_i915_gem_relocation_entry reloc;unsigned long bb_sz;uint32_t bb_handle; -uint32_t *mapped_batch, *mapped_seqno; +uint32_t *mapped_batch; +uint64_t *mapped_address; +uint32_t *mapped_seqno;unsigned int mapped_len; } b[2 ];/* One for each VCS when load balancing */ };
@@ -405,7 +409,8 @@ terminate_bb(struct w_step *w, struct w_step_eb *b, enum intel_engine_id engine, mmap_len += cmd_offset - mmap_start;gem_set_domain (fd, b->bb_handle, -I915_GEM_DOMAIN_CPU ,I915_GEM_DOMAIN_CPU ); +LOCAL_I915_GEM_DOMAIN_WC , +LOCAL_I915_GEM_DOMAIN_WC ); ptr =gem_mmap__wc (fd, b->bb_handle, mmap_start, mmap_len,PROT_WRITE ); cs = (uint32_t *)((char *)ptr + cmd_offset - mmap_start);
@@ -415,6 +420,7 @@ terminate_bb(struct w_step *w, struct w_step_eb *b, enum intel_engine_id engine, b->reloc.delta = (engine -VCS1 ) *sizeof (uint32_t ); *cs++ =MI_STORE_DWORD_IMM ; + b->mapped_address = (uint64_t *)cs; *cs++ =0 ; *cs++ =0 ; b->mapped_seqno = cs;
@@ -469,7 +475,6 @@ alloc_step_batch(struct workload *wrk, struct w_step *w, struct w_step_eb *b, }if (flags &SEQNO ) { - b->reloc.presumed_offset =-1 ; b->reloc.target_handle =1 ; b->obj[bb_i].relocs_ptr =to_user_pointer (&b->reloc); b->obj[bb_i].relocation_count =1 ;
@@ -485,8 +490,7 @@ alloc_step_batch(struct workload *wrk, struct w_step *w, struct w_step_eb *b, engine =VCS1 ; b->eb.flags = eb_engine_map[engine]; b->eb.flags |=I915_EXEC_HANDLE_LUT ; -if (!(flags &SEQNO )) - b->eb.flags |=I915_EXEC_NO_RELOC ; + b->eb.flags |=I915_EXEC_NO_RELOC ;#ifdef DEBUG printf("%u: %u:%x|%x|%x|%x %10lu flags=%llx bb=%x[%u] ctx[%u]=%u\n", w->idx, b->eb.buffer_count, b->obj[0].handle,
@@ -628,8 +632,9 @@ static void update_bb_seqno(struct w_step_eb *b, enum intel_engine_id engine, uint32_t seqno) { - *b->mapped_seqno = seqno; b->reloc.delta = (engine - VCS1) * sizeof(uint32_t); + *b->mapped_address = b->reloc.presumed_offset + b->reloc.delta; + *b->mapped_seqno = seqno; } static void
Signature · 1 line
2.11.0
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v4] benchmarks/gem_wsim: Command submission workload simulator
Quoted 7 lines · Chris Wilson
Quoted 5 lines · Chris Wilson
You either want to reset presumed_offset=-1 each time, or better for all concerned write the correct address alongside the seqno (which also enables NORELOC).
Delta incoming.
Quoted 7 lines
See attached.
Next concern is that I have full rings which implies that we are not waiting on each batch before resubmitting with a new seqno?
If I throw a assert(!busy(batch_bo)) before the
*b->mapped_seqnoam I going to be upset?
Yes you would. :) I had a sync (as a move to cpu domain) before seqno update in the last version but it disappeared as I was fixing the whole area of seqno tracking. So the balancing results in the patch are bogus since the seqno can jump to latest ahead of the time...
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v4] benchmarks/gem_wsim: Command submission workload simulator
In reply to Chris WilsonQuoted 53 lines · Chris Wilson
Quoted 33 lines · Tvrtko Ursulin
benchmarks/gem_wsim.c new file +33
-0+static void +alloc_step_batch (struct workload *wrk,struct w_step *w,struct w_step_eb *b, +enum intel_engine_id engine,unsigned int flags) +{ +unsigned int bb_i, j =0 ; + + b->obj[j].handle =gem_create (fd,4096 ); + b->obj[j].flags =EXEC_OBJECT_WRITE ; + j++; + +if (flags &SEQNO ) { + b->obj[j].handle = wrk->status_page_handle; + j++; + } + + bb_i = j++; + b->bb_sz =get_bb_sz (w->duration.max); + b->bb_handle = b->obj[bb_i].handle =gem_create (fd, b->bb_sz); +terminate_bb (w, b, engine, flags); + +igt_assert (w->dependency <=0 ); +if (w->dependency) { +int dep_idx = w->idx + w->dependency; + +igt_assert (dep_idx >=0 && dep_idx < wrk->nr_steps); +igt_assert (wrk->steps[dep_idx].type ==BATCH ); + + b->obj[j].handle = b->obj[bb_i].handle; + bb_i = j; + b->obj[j -1 ].handle = wrk->steps[dep_idx].b[0 ].obj[0 ].handle; + j++; + +if (wrk->steps[dep_idx].b[1 ].obj[0 ].handle) {benchmarks/gem_wsim.c line 472
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ b->obj[j].handle = b->obj[bb_i].handle; + bb_i = j; + b->obj[j -1 ].handle = + wrk->steps[dep_idx].b[1 ].obj[0 ].handle; + j++; + } + } + +if (flags &SEQNO ) { + b->reloc.presumed_offset =-1 ;So as I understand it, you are caching the execbuf/obj/reloc for the workload and then may reissue later with different seqno on different rings? In which case we have a problem as the kernel will write back the >
updated offsets to
b->reloc.presumed_offsetand b->obj[].offset and in future passes they will match and the seqno write will go into the wrong slot (if it swaps rings).
Quoted 5 lines
You either want to reset presumed_offset=-1 each time, or better for all concerned write the correct address alongside the seqno (which also enables NORELOC).
Delta incoming.
Only the seqno changes, but each engine has its own eb/obj/reloc. So I think there is no problem. Or is there still?
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v4] benchmarks/gem_wsim: Command submission workload simulator
Quoted 55 lines · Tvrtko Ursulin
Quoted 53 lines · Chris Wilson
Quoted 33 lines · Tvrtko Ursulin
benchmarks/gem_wsim.c new file +33
-0+static void +alloc_step_batch (struct workload *wrk,struct w_step *w,struct w_step_eb *b, +enum intel_engine_id engine,unsigned int flags) +{ +unsigned int bb_i, j =0 ; + + b->obj[j].handle =gem_create (fd,4096 ); + b->obj[j].flags =EXEC_OBJECT_WRITE ; + j++; + +if (flags &SEQNO ) { + b->obj[j].handle = wrk->status_page_handle; + j++; + } + + bb_i = j++; + b->bb_sz =get_bb_sz (w->duration.max); + b->bb_handle = b->obj[bb_i].handle =gem_create (fd, b->bb_sz); +terminate_bb (w, b, engine, flags); + +igt_assert (w->dependency <=0 ); +if (w->dependency) { +int dep_idx = w->idx + w->dependency; + +igt_assert (dep_idx >=0 && dep_idx < wrk->nr_steps); +igt_assert (wrk->steps[dep_idx].type ==BATCH ); + + b->obj[j].handle = b->obj[bb_i].handle; + bb_i = j; + b->obj[j -1 ].handle = wrk->steps[dep_idx].b[0 ].obj[0 ].handle; + j++; + +if (wrk->steps[dep_idx].b[1 ].obj[0 ].handle) {benchmarks/gem_wsim.c line 472
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ b->obj[j].handle = b->obj[bb_i].handle; + bb_i = j; + b->obj[j -1 ].handle = + wrk->steps[dep_idx].b[1 ].obj[0 ].handle; + j++; + } + } + +if (flags &SEQNO ) { + b->reloc.presumed_offset =-1 ;So as I understand it, you are caching the execbuf/obj/reloc for the workload and then may reissue later with different seqno on different rings? In which case we have a problem as the kernel will write back the >
updated offsets to
b->reloc.presumed_offsetand b->obj[].offset and in future passes they will match and the seqno write will go into the wrong slot (if it swaps rings).
Quoted 8 lines
Quoted 5 lines
You either want to reset presumed_offset=-1 each time, or better for all concerned write the correct address alongside the seqno (which also enables NORELOC).
Delta incoming.
Only the seqno changes, but each engine has its own eb/obj/reloc. So I think there is no problem. Or is there still?
bb_handle is per engine as well. Ugh. No, that seems like you are self-consistent, you just need to remove the -1 and your code is NORELOC correct.
I wouldn't go so far as having entirely separate batches for each engine though :)
-Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v4] benchmarks/gem_wsim: Command submission workload simulator
In reply to Tvrtko UrsulinQuoted 12 lines · Tvrtko Ursulin
benchmarks/gem_wsim.c new file +12
-0+ wrk->nr_bb[engine]++; + +if (engine ==VCS && balance) { + engine =balance (wrk, w); + wrk->nr_bb[engine]++; + b = &w->b[engine -VCS1 ]; + +if (flags &SEQNO ) +update_bb_seqno (b, engine, + ++wrk->seqno[engine]); + } +
benchmarks/gem_wsim.c line 712
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+if (w->duration.min != w->duration.max) { +unsigned int d =get_duration (&w->duration); +unsigned long offset; + + offset =ALIGN (b->bb_sz -get_bb_sz (d), +2 *sizeof (uint32_t )); + b->eb.batch_start_offset = offset; + } + +gem_execbuf (fd, &b->eb);
Likely double counting wrk->nr_bb. I suggest placing it next to the gem_execbuf().
-Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v4] benchmarks/gem_wsim: Command submission workload simulator
Quoted 18 lines · Chris Wilson
Quoted 17 lines · Tvrtko Ursulin
benchmarks/gem_wsim.c new file +17
-0+ wrk->nr_bb[engine]++; + +if (engine ==VCS && balance) { + engine =balance (wrk, w); + wrk->nr_bb[engine]++; + b = &w->b[engine -VCS1 ]; + +if (flags &SEQNO ) +update_bb_seqno (b, engine, + ++wrk->seqno[engine]); + } + +if (w->duration.min != w->duration.max) { +unsigned int d =get_duration (&w->duration); +unsigned long offset; + + offset = ALIGN(b->bb_sz - get_bb_sz(d),
Quoted 8 lines
benchmarks/gem_wsim.c line 712
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+ 2 * sizeof(uint32_t)); + b->eb.batch_start_offset = offset; + } + +gem_execbuf (fd, &b->eb);Likely double counting
wrk->nr_bb. I suggest placing it next to thegem_execbuf().
Just convenience in balancing mode so that nr(VCS) = nr(VCS1) + nr(VCS2). Also from a different angle, if the sum does not hold, that means workload had auto-balancing and explicit VCS1/2 batches. It's only used to print out the stats at the end of the run.
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v4] benchmarks/gem_wsim: Command submission workload simulator
In reply to Tvrtko UrsulinQuoted 5 lines · Tvrtko Ursulin
From: Tvrtko Ursulin <[email protected]>
benchmarks/gem_wsim.c new file +4
-0+static void +run_workload(unsigned int id, struct workload *wrk, unsigned int repeat, + enum intel_engine_id (*balance)(struct workload *wrk, + struct w_step *w),
benchmarks/gem_wsim.c line 648
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+ unsigned int flags) +{ +struct timespec t_start, t_end; +struct w_step *w; +double t; +int i, j; + +clock_gettime (CLOCK_MONOTONIC , &t_start); + +srand (t_start.tv_nsec);
Let's supply the seed with the workload specification. And use a portable prng so we can be sure we can reproduce results from one system to the next.
-Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v5 1/1 benchmarks/gem_wsim: Command submission workload simulator
In reply to Tvrtko UrsulinFrom: Tvrtko Ursulin <[email protected]>
Tool which emits batch buffers to engines with configurable sequences, durations, contexts, dependencies and userspace waits.
Unfinished but shows promise so sending out for early feedback.
v2:
* Load workload descriptors from files. (also -w)
* Help text.
* Calibration control if needed. (-t)
* NORELOC | LUT to eb flags.
* Added sample workload to wsim/workload1.
v3:
* Multiple parallel different workloads (-w -w ...).
* Multi-context workloads.
* Variable (random) batch length.
* Load balancing (round robin and queue depth estimation).
* Workloads delays and explicit sync steps.
* Workload frequency (period) control.
v4:
* Fixed queue-depth estimation by creating separate batches per engine when qd load balancing is on.
* Dropped separate -s cmd line option. It can turn itself on automatically when needed.
* Keep a single status page and lie about the write hazard as suggested by Chris.
* Use batch_start_offset for controlling the batch duration. (Chris)
* Set status page object cache level. (Chris)
* Moved workload description to a README.
* Tidied example workloads.
* Some other cleanups and refactorings.
v5:
* Master and background workloads (-W / -w).
* Single batch per step is enough even when balancing. (Chris)
* Use hars_petruska_f54_1_random IGT functions and see to zero at start. (Chris)
* Use WC cache domain when WC mapping. (Chris)
* Keep seqnos 64-bytes apart in the status page. (Chris)
* Add workload throttling and queue-depth throttling commands. (Chris)
TODO list:
* Fence support.
* Better error handling.
* Less 1980's workload parsing.
* Proper workloads.
* Threads?
* ... ?
--- benchmarks/Makefile.sources | 1 + benchmarks/gem_wsim.c | 1189 ++++++++++++++++++++++++++ benchmarks/wsim/README | 56 ++ benchmarks/wsim/media_17i7.wsim | 7 + benchmarks/wsim/media_load_balance_17i7.wsim | 7 + benchmarks/wsim/vcs1.wsim | 26 + benchmarks/wsim/vcs_balanced.wsim | 26 + lib/igt_core.c | 26 + lib/igt_core.h | 1 + 9 files changed, 1339 insertions(+) create mode 100644 benchmarks/gem_wsim.c create mode 100644 benchmarks/wsim/README create mode 100644 benchmarks/wsim/media_17i7.wsim create mode 100644 benchmarks/wsim/media_load_balance_17i7.wsim create mode 100644 benchmarks/wsim/vcs1.wsim create mode 100644 benchmarks/wsim/vcs_balanced.wsim
benchmarks/Makefile.sources +1 -0
@@ -14,6 +14,7 @@ benchmarks_prog_list = \ gem_prw \ gem_set_domain \ gem_syslatency \ + gem_wsim \ kms_vblank \ prime_lookup \ vgem_mmap \
benchmarks/gem_wsim.c new file +1189 -0
1,189 lines not on this page · raw message
benchmarks/wsim/README new file +56 -0
56 lines not on this page · raw message
benchmarks/wsim/media_17i7.wsim new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS1.3000.0.1 +1.RCS.1000.-1.0 +1.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS2.2300.-2.0 +1.RCS.4700.-1.0 +1.VCS2.600.-1.1
benchmarks/wsim/media_load_balance_17i7.wsim new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS.3000.0.1 +1.RCS.1000.-1.0 +1.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS.2300.-2.0 +1.RCS.4700.-1.0 +1.VCS.600.-1.1
benchmarks/wsim/vcs1.wsim new file +26 -0
@@ -0,0 +1,26 @@ +t.5 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0
benchmarks/wsim/vcs_balanced.wsim new file +26 -0
@@ -0,0 +1,26 @@ +q.5 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0
lib/igt_core.c +26 -0
@@ -1558,6 +1558,32 @@ bool __igt_fork(void) }/** +* igt_child_done: +* +* Lets the IGT core know that one of the children has exited. +*/ +void igt_child_done (pid_t pid) +{ +int i =0 ; +int found =-1 ; + +igt_assert (num_test_children >1 ); + +for (i =0 ; i < num_test_children; i++) { +if (pid == test_children[i]) { + found = i; +break ; + } + } + +igt_assert (found >=0 ); + + num_test_children--; +for (i = found; i < num_test_children; i++) + test_children[i] = test_children[i +1 ]; +} + +/** * igt_waitchildren: * * Wait for all children forked with igt_fork.
lib/igt_core.h +1 -0
@@ -688,6 +688,7 @@ bool __igt_fork(void);#define igt_fork (child, num_children) \ for (int child = 0; child < (num_children); child++) \ for (; __igt_fork(); exit(0)) +void igt_child_done (pid_t pid);void igt_waitchildren (void );void igt_waitchildren_timeout (int seconds,const char *reason);
Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v6 1/1 benchmarks/gem_wsim: Command submission workload simulator
From: Tvrtko Ursulin <[email protected]>
Tool which emits batch buffers to engines with configurable sequences, durations, contexts, dependencies and userspace waits.
Unfinished but shows promise so sending out for early feedback.
v2:
* Load workload descriptors from files. (also -w)
* Help text.
* Calibration control if needed. (-t)
* NORELOC | LUT to eb flags.
* Added sample workload to wsim/workload1.
v3:
* Multiple parallel different workloads (-w -w ...).
* Multi-context workloads.
* Variable (random) batch length.
* Load balancing (round robin and queue depth estimation).
* Workloads delays and explicit sync steps.
* Workload frequency (period) control.
v4:
* Fixed queue-depth estimation by creating separate batches per engine when qd load balancing is on.
* Dropped separate -s cmd line option. It can turn itself on automatically when needed.
* Keep a single status page and lie about the write hazard as suggested by Chris.
* Use batch_start_offset for controlling the batch duration. (Chris)
* Set status page object cache level. (Chris)
* Moved workload description to a README.
* Tidied example workloads.
* Some other cleanups and refactorings.
v5:
* Master and background workloads (-W / -w).
* Single batch per step is enough even when balancing. (Chris)
* Use hars_petruska_f54_1_random IGT functions and see to zero at start. (Chris)
* Use WC cache domain when WC mapping. (Chris)
* Keep seqnos 64-bytes apart in the status page. (Chris)
* Add workload throttling and queue-depth throttling commands. (Chris)
v6:
* Added two more workloads.
* Merged RT balancer from Chris.
TODO list:
* Fence support.
* Better error handling.
* Less 1980's workload parsing.
* More workloads.
* Threads?
* ... ?
--- benchmarks/Makefile.sources | 1 + benchmarks/gem_wsim.c | 1307 ++++++++++++++++++++++++++ benchmarks/wsim/README | 56 ++ benchmarks/wsim/media_17i7.wsim | 7 + benchmarks/wsim/media_19.wsim | 10 + benchmarks/wsim/media_load_balance_17i7.wsim | 7 + benchmarks/wsim/media_load_balance_19.wsim | 10 + benchmarks/wsim/vcs1.wsim | 26 + benchmarks/wsim/vcs_balanced.wsim | 26 + lib/igt_core.c | 26 + lib/igt_core.h | 1 + 11 files changed, 1477 insertions(+) create mode 100644 benchmarks/gem_wsim.c create mode 100644 benchmarks/wsim/README create mode 100644 benchmarks/wsim/media_17i7.wsim create mode 100644 benchmarks/wsim/media_19.wsim create mode 100644 benchmarks/wsim/media_load_balance_17i7.wsim create mode 100644 benchmarks/wsim/media_load_balance_19.wsim create mode 100644 benchmarks/wsim/vcs1.wsim create mode 100644 benchmarks/wsim/vcs_balanced.wsim
benchmarks/Makefile.sources +1 -0
@@ -14,6 +14,7 @@ benchmarks_prog_list = \ gem_prw \ gem_set_domain \ gem_syslatency \ + gem_wsim \ kms_vblank \ prime_lookup \ vgem_mmap \
benchmarks/gem_wsim.c new file +1307 -0
1,307 lines not on this page · raw message
benchmarks/wsim/README new file +56 -0
56 lines not on this page · raw message
benchmarks/wsim/media_17i7.wsim new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS1.3000.0.1 +1.RCS.1000.-1.0 +1.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS2.2300.-2.0 +1.RCS.4700.-1.0 +1.VCS2.600.-1.1
benchmarks/wsim/media_19.wsim new file +10 -0
@@ -0,0 +1,10 @@ +0.VECS.1400-1500.0.0 +0.RCS.1000-1500.-1.0 +s.-2 +2.VCS2.50-350.0.1 +1.VCS1.1300-1400.0.1 +0.VECS.1400-1500.0.0 +0.RCS.100-300.-1.1 +2.RCS.1300-1500.0.0 +2.VCS2.100-300.-1.1 +1.VCS1.900-1400.0.1
benchmarks/wsim/media_load_balance_17i7.wsim new file +7 -0
@@ -0,0 +1,7 @@ +1.VCS.3000.0.1 +1.RCS.1000.-1.0 +1.RCS.3700.0.0 +1.RCS.1000.-2.0 +1.VCS.2300.-2.0 +1.RCS.4700.-1.0 +1.VCS.600.-1.1
benchmarks/wsim/media_load_balance_19.wsim new file +10 -0
@@ -0,0 +1,10 @@ +0.VECS.1400-1500.0.0 +0.RCS.1000-1500.-1.0 +s.-2 +1.VCS.50-350.0.1 +1.VCS.1300-1400.0.1 +0.VECS.1400-1500.0.0 +0.RCS.100-300.-1.1 +1.RCS.1300-1500.0.0 +1.VCS.100-300.-1.1 +1.VCS.900-1400.0.1
benchmarks/wsim/vcs1.wsim new file +26 -0
@@ -0,0 +1,26 @@ +t.5 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0 +0.VCS1.500-2000.0.0
benchmarks/wsim/vcs_balanced.wsim new file +26 -0
@@ -0,0 +1,26 @@ +q.5 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0 +0.VCS.500-2000.0.0
lib/igt_core.c +26 -0
@@ -1558,6 +1558,32 @@ bool __igt_fork(void) }/** +* igt_child_done: +* +* Lets the IGT core know that one of the children has exited. +*/ +void igt_child_done (pid_t pid) +{ +int i =0 ; +int found =-1 ; + +igt_assert (num_test_children >1 ); + +for (i =0 ; i < num_test_children; i++) { +if (pid == test_children[i]) { + found = i; +break ; + } + } + +igt_assert (found >=0 ); + + num_test_children--; +for (i = found; i < num_test_children; i++) + test_children[i] = test_children[i +1 ]; +} + +/** * igt_waitchildren: * * Wait for all children forked with igt_fork.
lib/igt_core.h +1 -0
@@ -688,6 +688,7 @@ bool __igt_fork(void);#define igt_fork (child, num_children) \ for (int child = 0; child < (num_children); child++) \ for (; __igt_fork(); exit(0)) +void igt_child_done (pid_t pid);void igt_waitchildren (void );void igt_waitchildren_timeout (int seconds,const char *reason);
Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v6] benchmarks/gem_wsim: Command submission workload simulator
Quoted 46 lines · Tvrtko Ursulin
From: Tvrtko Ursulin <[email protected]>
Tool which emits batch buffers to engines with configurable sequences, durations, contexts, dependencies and userspace waits.
Unfinished but shows promise so sending out for early feedback.
v2:
* Load workload descriptors from files. (also -w)
* Help text.
* Calibration control if needed. (-t)
* NORELOC | LUT to eb flags.
* Added sample workload to wsim/workload1.v3:
* Multiple parallel different workloads (-w -w ...).
* Multi-context workloads.
* Variable (random) batch length.
* Load balancing (round robin and queue depth estimation).
* Workloads delays and explicit sync steps.
* Workload frequency (period) control.v4:
* Fixed queue-depth estimation by creating separate batches per engine when qd load balancing is on.
* Dropped separate -s cmd line option. It can turn itself on automatically when needed.
* Keep a single status page and lie about the write hazard as suggested by Chris.
* Usebatch_start_offsetfor controlling the batch duration. (Chris)
* Set status page object cache level. (Chris)
* Moved workload description to a README.
* Tidied example workloads.
* Some other cleanups and refactorings.v5:
* Master and background workloads (-W / -w).
* Single batch per step is enough even when balancing. (Chris)
* Usehars_petruska_f54_1_randomIGT functions and see to zero at start. (Chris)
* Use WC cache domain when WC mapping. (Chris)
* Keep seqnos 64-bytes apart in the status page. (Chris)
* Add workload throttling and queue-depth throttling commands. (Chris)
Quoted 5 lines
v6:
* Added two more workloads.
* Merged RT balancer from Chris.TODO list:
* No reloc!
* bb caching/reuse
Quoted 18 lines
* Fence support.
* Better error handling.
* Less 1980's workload parsing.
* More workloads.
* Threads?
* ... ?Signed-off-by Tvrtko Ursulin <[email protected]>Cc Chris Wilson <[email protected]>Cc "Rogozhkin, Dmitry V" <[email protected]>---
benchmarks/gem_wsim.c new file +6
-0+static enum intel_engine_id +rt_balance (const struct workload_balancer *balancer, +struct workload *wrk,struct w_step *w) +{ +enum intel_engine_id engine; +long qd[NUM_ENGINES ];
benchmarks/gem_wsim.c line 734
Quoted 10 lines
benchmarks/gem_wsim.c new file +10
-0+unsigned int n; + +igt_assert (w->engine ==VCS ); + +/* Estimate the "speed" of the most recent batch +* (finish time - submit time) +* and use that as an approximate for the total remaining time for +* all batches on that engine. We try to keep the total remaining +* balanced between the engines. +*/
Next steps for this would be to move from an instantaneous speed, to an average. I'm thinking something like a exponential decay moving average just to make the estimation more robust.
benchmarks/gem_wsim.c line 925
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+if (qd_throttle >0 && balancer && balancer->get_qd) { +unsigned int target; + +for (target = wrk->nr_steps -1 ; target >0 ; + target--) {
I think this should skip other engines.
if (target->engine != engine)
continue;
Quoted 6 lines
benchmarks/gem_wsim.c new file +6
-0+if (balancer->get_qd (balancer, wrk, + engine) < + qd_throttle) +break ; +w_sync_to (wrk, w, i - target); + }
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v6] benchmarks/gem_wsim: Command submission workload simulator
Quoted 47 lines · Chris Wilson
Quoted 46 lines · Tvrtko Ursulin
From: Tvrtko Ursulin <[email protected]>
Tool which emits batch buffers to engines with configurable sequences, durations, contexts, dependencies and userspace waits.
Unfinished but shows promise so sending out for early feedback.
v2:
* Load workload descriptors from files. (also -w)
* Help text.
* Calibration control if needed. (-t)
* NORELOC | LUT to eb flags.
* Added sample workload to wsim/workload1.v3:
* Multiple parallel different workloads (-w -w ...).
* Multi-context workloads.
* Variable (random) batch length.
* Load balancing (round robin and queue depth estimation).
* Workloads delays and explicit sync steps.
* Workload frequency (period) control.v4:
* Fixed queue-depth estimation by creating separate batches per engine when qd load balancing is on.
* Dropped separate -s cmd line option. It can turn itself on automatically when needed.
* Keep a single status page and lie about the write hazard as suggested by Chris.
* Usebatch_start_offsetfor controlling the batch duration. (Chris)
* Set status page object cache level. (Chris)
* Moved workload description to a README.
* Tidied example workloads.
* Some other cleanups and refactorings.v5:
* Master and background workloads (-W / -w).
* Single batch per step is enough even when balancing. (Chris)
* Usehars_petruska_f54_1_randomIGT functions and see to zero at start. (Chris)
* Use WC cache domain when WC mapping. (Chris)
* Keep seqnos 64-bytes apart in the status page. (Chris)
* Add workload throttling and queue-depth throttling commands. (Chris)
Quoted 8 lines
Quoted 5 lines
v6:
* Added two more workloads.
* Merged RT balancer from Chris.TODO list:
* No reloc!
* bb caching/reuse
Yeah I know, but have to progress the overall case as well and I am thinking it is getting close to good enough now. So now is the time to think of interesting workloads, and workload combinations.
Quoted 24 lines
Quoted 24 lines
* Fence support.
* Better error handling.
* Less 1980's workload parsing.
* More workloads.
* Threads?
* ... ?Signed-off-by Tvrtko Ursulin <[email protected]>Cc Chris Wilson <[email protected]>Cc "Rogozhkin, Dmitry V" <[email protected]>---
benchmarks/gem_wsim.c new file +12
-0+static enum intel_engine_id +rt_balance (const struct workload_balancer *balancer, +struct workload *wrk,struct w_step *w) +{ +enum intel_engine_id engine; +long qd[NUM_ENGINES ]; +unsigned int n; + +igt_assert (w->engine ==VCS ); + +/* Estimate the "speed" of the most recent batch +* (finish time - submit time)
Quoted 8 lines
benchmarks/gem_wsim.c line 734
Quoted 4 lines
benchmarks/gem_wsim.c new file +4
-0+* and use that as an approximate for the total remaining time for +* all batches on that engine. We try to keep the total remaining +* balanced between the engines. +*/ Next steps for this would be to move from an instantaneous speed, to an average. I'm thinking something like a exponential decay moving average just to make the estimation more robust.
Do you think it would be OK to merge these two tools at this point and continue improving them in place?
Your balancer already looks a solid step up from the queue-depth one. I checked today myself and, what looks like a worst case of a VCS1 hog and a balancing workloads running together, it gets the VCS2 utilisation to impressive 85%.
As mentioned before those stats can now be collected easily with:
trace.pl --trace gem_wsim ...; perf script | trace.pl
I need to start pining the relevant people for help with creating relevant workloads and am also entertaining the idea of trying balancing via exporting the stats from i915 directly. Just to see if true vs estimated numbers would make a difference here.
Quoted 10 lines
benchmarks/gem_wsim.c line 925
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+if (qd_throttle >0 && balancer && balancer->get_qd) { +unsigned int target; + +for (target = wrk->nr_steps -1 ; target >0 ; + target--) {I think this should skip other engines.
if (
target->engine!= engine)continue;
If you say so. I don't have an opinion on it. Would it be useful to perhaps have both options - to throttle globally and per-engine? I could easily add two different workload commands for that.
Regards,
Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v6] benchmarks/gem_wsim: Command submission workload simulator
Quoted 4 lines · Tvrtko Ursulin
Quoted 2 lines · Chris Wilson
Quoted 1 line · Tvrtko Ursulin
From: Tvrtko Ursulin <[email protected]>
[snip]
Quoted 21 lines
Quoted 12 lines
Quoted 12 lines
benchmarks/gem_wsim.c new file +12
-0+static enum intel_engine_id +rt_balance (const struct workload_balancer *balancer, +struct workload *wrk,struct w_step *w) +{ +enum intel_engine_id engine; +long qd[NUM_ENGINES ]; +unsigned int n; + +igt_assert (w->engine ==VCS ); + +/* Estimate the "speed" of the most recent batch +* (finish time - submit time) Quoted 8 lines
benchmarks/gem_wsim.c line 734
Quoted 4 lines
benchmarks/gem_wsim.c new file +4
-0+* and use that as an approximate for the total remaining time for +* all batches on that engine. We try to keep the total remaining +* balanced between the engines. +*/ Next steps for this would be to move from an instantaneous speed, to an average. I'm thinking something like a exponential decay moving average just to make the estimation more robust.
Quoted 2 lines
Do you think it would be OK to merge these two tools at this point and continue improving them in place?
Yes. Although there's no excuse no to make this NO_RELOC from the start, especially if we want to demonstrate how it should be done! Hopefully attached the delta.
Quoted 4 lines
Your balancer already looks a solid step up from the queue-depth one. I checked today myself and, what looks like a worst case of a VCS1 hog and a balancing workloads running together, it gets the VCS2 utilisation to impressive 85%.
Yup. Just thinking of the danger in using an instantaneous value as our estimate and how to improve.
Quoted 8 lines
As mentioned before those stats can now be collected easily with:
trace.pl --trace gem_wsim ...; perf script | trace.plI need to start pining the relevant people for help with creating relevant workloads and am also entertaining the idea of trying balancing via exporting the stats from i915 directly. Just to see if true vs estimated numbers would make a difference here.
Aye.
Quoted 8 lines
Quoted 8 lines
benchmarks/gem_wsim.c line 925
Quoted 5 lines
benchmarks/gem_wsim.c new file +5
-0+if (qd_throttle >0 && balancer && balancer->get_qd) { +unsigned int target; + +for (target = wrk->nr_steps -1 ; target >0 ; + target--) {I think this should skip other engines.
Quoted 6 lines
Quoted 2 lines
if (
target->engine!= engine)continue;If you say so. I don't have an opinion on it. Would it be useful to perhaps have both options - to throttle globally and per-engine? I could easily add two different workload commands for that.
I'm leaning towards making it a balancer->callback(). -Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH i-g-t v6] benchmarks/gem_wsim: Command submission workload simulator
Quoted 27 lines · Chris Wilson
Quoted 4 lines · Tvrtko Ursulin
Quoted 2 lines · Chris Wilson
Quoted 1 line · Tvrtko Ursulin
From: Tvrtko Ursulin <[email protected]>
[snip]
Quoted 21 lines
Quoted 20 lines
Quoted 6 lines
(quoted diff) +6
-0+static enum intel_engine_id +rt_balance(const struct workload_balancer *balancer, + struct workload *wrk, struct w_step *w) +{ + enum intel_engine_id engine; + long qd[NUM_ENGINES];Review
Quoted 10 lines
(quoted diff) +10
-0+ unsigned int n; + + igt_assert(w->engine == VCS); + + /* Estimate the "speed" of the most recent batch + * (finish time - submit time) + * and use that as an approximate for the total remaining time for + * all batches on that engine. We try to keep the total remaining + * balanced between the engines. + */Next steps for this would be to move from an instantaneous speed, to an average. I'm thinking something like a exponential decay moving average just to make the estimation more robust.
Quoted 6 lines
Quoted 2 lines
Do you think it would be OK to merge these two tools at this point and continue improving them in place?
Yes. Although there's no excuse no to make this
NO_RELOCfrom the start, especially if we want to demonstrate how it should be done! Hopefully attached the delta.
Which I forgot. Let's try again...
-Chris
Signature · 1 line
Chris Wilson, Intel Open Source Technology Centre
Quoted 1 line
From
985f873f1c9cdaec396c5410738910da04e8f95bMon Sep 17 00:00:00 2001
From: Chris Wilson <[email protected]>
Date: Tue, 25 Apr 2017 13:22:39 +0100
Subject: [PATCH] no-reloc
--- benchmarks/gem_wsim.c | 45 +++++++++++++++++++++++++++++---------------- 1 file changed, 29 insertions(+), 16 deletions(-)
benchmarks/gem_wsim.c +29 -16
@@ -90,9 +90,13 @@ struct w_stepstruct drm_i915_gem_relocation_entry reloc[3 ];unsigned long bb_sz;uint32_t bb_handle; -uint32_t *mapped_batch, *mapped_seqno; -unsigned int mapped_len; +uint32_t *mapped_batch; +uint32_t *seqno_value; +uint32_t *seqno_address;uint32_t *rt0_value; +uint32_t *rt0_address; +uint32_t *rt1_address; +unsigned int mapped_len; };struct workload
@@ -463,9 +467,10 @@ terminate_bb(struct w_step *w, unsigned int flags) batch_start +=4 *sizeof (uint32_t ); *cs++ =MI_STORE_DWORD_IMM ; + w->seqno_address = cs; *cs++ =0 ; *cs++ =0 ; - w->mapped_seqno = cs; + w->seqno_value = cs; *cs++ =0 ; }
@@ -474,6 +479,7 @@ terminate_bb(struct w_step *w, unsigned int flags) batch_start +=4 *sizeof (uint32_t ); *cs++ =MI_STORE_DWORD_IMM ; + w->rt0_address = cs; *cs++ =0 ; *cs++ =0 ; w->rt0_value = cs;
@@ -484,6 +490,7 @@ terminate_bb(struct w_step *w, unsigned int flags) *cs++ =0x24 <<23 |2 ;/* MI_STORE_REG_MEM */ *cs++ =RCS_TIMESTAMP ; + w->rt1_address = cs; *cs++ =0 ; *cs++ =0 ; }
@@ -500,8 +507,7 @@ eb_update_flags(struct w_step *w, enum intel_engine_id engine, { w->eb.flags = eb_engine_map[engine]; w->eb.flags |=I915_EXEC_HANDLE_LUT ; -if (!(flags &SEQNO )) - w->eb.flags |=I915_EXEC_NO_RELOC ; + w->eb.flags |=I915_EXEC_NO_RELOC ; } static void
@@ -543,10 +549,8 @@ alloc_step_batch(struct workload *wrk, struct w_step *w, unsigned int flags) w->obj[bb_i].relocation_count =3 ; else w->obj[bb_i].relocation_count = 1; -for (int i =0 ; i < w->obj[bb_i].relocation_count; i++) { - w->reloc[i].presumed_offset =-1 ; +for (int i =0 ; i < w->obj[bb_i].relocation_count; i++) w->reloc[i].target_handle =1 ; - } } w->eb.buffers_ptr =to_user_pointer (w->obj);
@@ -782,10 +786,14 @@ update_bb_seqno(struct w_step *w, enum intel_engine_id engine, uint32_t seqno)gem_set_domain (fd, w->bb_handle,I915_GEM_DOMAIN_WC ,I915_GEM_DOMAIN_WC ); - *w->mapped_seqno = seqno; - - w->reloc[0 ].presumed_offset =-1 ; w->reloc[0 ].delta =VCS_SEQNO_OFFSET (engine); + + *w->seqno_value = seqno; + *w->seqno_address = w->reloc[0 ].presumed_offset + w->reloc[0 ].delta; + +/* If not using NO_RELOC, force the relocations */ +if ((w->eb.flags &I915_EXEC_NO_RELOC )) + w->reloc[0 ].presumed_offset =-1 ; } static void
@@ -796,13 +804,18 @@ update_bb_rt(struct w_step *w, enum intel_engine_id engine)gem_set_domain (fd, w->bb_handle,I915_GEM_DOMAIN_WC ,I915_GEM_DOMAIN_WC ); - *w->rt0_value = *REG (RCS_TIMESTAMP ); - - w->reloc[1 ].presumed_offset =-1 ; w->reloc[1 ].delta =VCS_SEQNO_OFFSET (engine) +sizeof (uint32_t ); - - w->reloc[2 ].presumed_offset =-1 ; w->reloc[2 ].delta =VCS_SEQNO_OFFSET (engine) +2 *sizeof (uint32_t ); + + *w->rt0_value = *REG (RCS_TIMESTAMP ); + *w->rt0_address = w->reloc[1 ].presumed_offset + w->reloc[1 ].delta; + *w->rt1_address = w->reloc[1 ].presumed_offset + w->reloc[1 ].delta; + +/* If not using NO_RELOC, force the relocations */ +if ((w->eb.flags &I915_EXEC_NO_RELOC )) { + w->reloc[1 ].presumed_offset =-1 ; + w->reloc[2 ].presumed_offset =-1 ; + } }static void w_sync_to (struct workload *wrk,struct w_step *w,int target)
Signature · 1 line
1.9.1
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 2/2 igt/scripts: trace.pl to parse the i915 tracepoints
In reply to Tvrtko UrsulinFrom: Tvrtko Ursulin <[email protected]>
Given a log file created via perf with some interesting trace events enabled, this tool can generate the timeline graph of requests getting queued, their dependencies resolved, sent to the GPU for executing and finally completed.
This can be useful when analyzing certain classes of performance issues. More help is available in the tool itself.
The tool will also calculate some overall per engine statistics, like total time engine was idle and similar.
v2:
* Address missing git add.
* Make html output optional (--html switch) and by default just output aggregated per engine stats to stdout.
v3:
* Added --trace option which invokes perf with the correct options automatically.
* Added --avg-delay-stats which prints averages for things like waiting on ready, waiting on GPU and context save duration.
* Fix warnings when no waits on an engine.
* Correct help text.
--- scripts/Makefile.am | 2 +- scripts/trace.pl | 946 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 947 insertions(+), 1 deletion(-) create mode 100755 scripts/trace.pl
scripts/Makefile.am +1 -1
@@ -1,2 +1,2 @@ -dist_noinst_SCRIPTS = intel-gfx-trybot who.sh run-tests.sh +dist_noinst_SCRIPTS = intel-gfx-trybot who.sh run-tests.sh trace.pl noinst_PYTHON = throttle.py
scripts/trace.pl new file +946 -0
946 lines not on this page · raw message
Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 1/1 igt/scripts: trace.pl to parse the i915 tracepoints
In reply to Tvrtko UrsulinFrom: Tvrtko Ursulin <[email protected]>
Given a log file created via perf with some interesting trace events enabled, this tool can generate the timeline graph of requests getting queued, their dependencies resolved, sent to the GPU for executing and finally completed.
This can be useful when analyzing certain classes of performance issues. More help is available in the tool itself.
The tool will also calculate some overall per engine statistics, like total time engine was idle and similar.
v2:
* Address missing git add.
* Make html output optional (--html switch) and by default just output aggregated per engine stats to stdout.
v3:
* Added --trace option which invokes perf with the correct options automatically.
* Added --avg-delay-stats which prints averages for things like waiting on ready, waiting on GPU and context save duration.
* Fix warnings when no waits on an engine.
* Correct help text.
v4:
* Add --squash-ctx-id to substract engine id from ctx id when parsing to make it easier to identify which context is which with new i915 ctx id allocation scheme.
* Reconstruct request_out events where they are missing.
--- scripts/Makefile.am | 2 +- scripts/trace.pl | 990 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 991 insertions(+), 1 deletion(-) create mode 100755 scripts/trace.pl
scripts/Makefile.am +1 -1
@@ -1,2 +1,2 @@ -dist_noinst_SCRIPTS = intel-gfx-trybot who.sh run-tests.sh +dist_noinst_SCRIPTS = intel-gfx-trybot who.sh run-tests.sh trace.pl noinst_PYTHON = throttle.py
scripts/trace.pl new file +990 -0
990 lines not on this page · raw message
Signature · 6 lines
2.9.3 _______________________________________________ Intel-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/intel-gfx