GPU scheduler for i915 driver
Subsystems (4)
cover GPU scheduler for i915 driver
From: John Harrison <[email protected]>
Implemented a batch buffer submission scheduler for the i915 DRM driver.
The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code assigns a unique seqno value and then packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.
The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.
As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, the seqno values of the individual buffers are not necessary incrementing as the scheduler may have re-ordered their submission. However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.
A deferred work queue is also poked by the interrupt handler. When this wakes up it can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.
When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.
If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate.
The scheduler also allows high priority batch buffers (e.g. from a desktop compositor) to jump ahead of whatever is already running if the underlying hardware supports pre-emption. In this situation, any work that was pre-empted is returned to the queued list ready to be resubmitted when no more high priority work is outstanding.
Various IGT tests are in progress to test the scheduler's operation and will follow.
v2: Updated for changes in struct fence patch series and other changes to underlying tree (e.g. removal of cliprects). Also changed priority levels to be signed +/-1023 range and reduced mutex lock usage.
[Patches against drm-intel-nightly tree fetched 17/11/2015 with struct fence conversion patches applied]
Dave Gordon (2):
drm/i915: Updating assorted register and status page definitions drm/i915: Add scheduling priority to per-context parameters
John Harrison (37):
drm/i915: Add total count to context status debugfs output drm/i915: Explicit power enable during deferred context initialisation drm/i915: Prelude to splitting i915_gem_do_execbuffer in two drm/i915: Split i915_dem_do_execbuffer() in half
drm/i915: Re-instate request->uniq because it is extremely useful drm/i915: Start of GPU scheduler
drm/i915: Prepare retire_requests to handle out-of-order seqnos drm/i915: Disable hardware semaphores when GPU scheduler is enabled drm/i915: Force MMIO flips when scheduler enabled
drm/i915: Added scheduler hook when closing DRM file handles drm/i915: Added scheduler hook into i915_gem_request_notify() drm/i915: Added deferred work handler for scheduler
drm/i915: Redirect execbuffer_final() via scheduler
drm/i915: Keep the reserved space mechanism happy
drm/i915: Added tracking/locking of batch buffer objects drm/i915: Hook scheduler node clean up into retire requests drm/i915: Added scheduler support to __wait_request() calls drm/i915: Added scheduler support to page fault handler drm/i915: Added scheduler flush calls to ring throttle and idle functions drm/i915: Added a module parameter for allowing scheduler overrides drm/i915: Support for 'unflushed' ring idle
drm/i915: Defer seqno allocation until actual hardware submission time drm/i915: Added immediate submission override to scheduler drm/i915: Add sync wait support to scheduler
drm/i915: Connecting execbuff fences to scheduler
drm/i915: Added trace points to scheduler
drm/i915: Added scheduler queue throttling by DRM file handle drm/i915: Added debugfs interface to scheduler tuning parameters drm/i915: Added debug state dump facilities to scheduler drm/i915: Add early exit to execbuff_final() if insufficient ring space drm/i915: Added scheduler statistic reporting to debugfs drm/i915: Added seqno values to scheduler status dump drm/i915: Add scheduler support functions for TDR
drm/i915: GPU priority bumping to prevent starvation drm/i915: Scheduler state dump via debugfs
drm/i915: Enable GPU scheduler by default
drm/i915: Allow scheduler to manage inter-ring object synchronisation
drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/i915_debugfs.c | 283 +++++ drivers/gpu/drm/i915/i915_dma.c | 6 + drivers/gpu/drm/i915/i915_drv.c | 9 + drivers/gpu/drm/i915/i915_drv.h | 56 +- drivers/gpu/drm/i915/i915_gem.c | 174 ++- drivers/gpu/drm/i915/i915_gem_context.c | 24 + drivers/gpu/drm/i915/i915_gem_execbuffer.c | 341 ++++-- drivers/gpu/drm/i915/i915_params.c | 4 + drivers/gpu/drm/i915/i915_reg.h | 30 +- drivers/gpu/drm/i915/i915_scheduler.c | 1697 ++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 176 +++ drivers/gpu/drm/i915/i915_trace.h | 215 +++- drivers/gpu/drm/i915/intel_display.c | 10 +- drivers/gpu/drm/i915/intel_lrc.c | 162 ++- drivers/gpu/drm/i915/intel_lrc.h | 1 + drivers/gpu/drm/i915/intel_ringbuffer.c | 47 +- drivers/gpu/drm/i915/intel_ringbuffer.h | 35 +- include/uapi/drm/i915_drm.h | 1 + 19 files changed, 3110 insertions(+), 162 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 1/39 drm/i915: Add total count to context status debugfs output
From: John Harrison <[email protected]>
When there are lots and lots and even more lots of contexts (e.g. when running with execlists) it is useful to be able to immediately see what the total context count is.
Change-Id: If9726d4df86567100ecf53867b43f4753f08bf84
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_debugfs.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/gpu/drm/i915/i915_debugfs.c +4 -0
@@ -1953,6 +1953,7 @@ static int i915_context_status(struct seq_file *m, void *unused)struct drm_i915_private *dev_priv = dev->dev_private;struct intel_engine_cs *ring;struct intel_context *ctx; +uint32_t count =0 ;int ret, i; ret =mutex_lock_interruptible (&dev->struct_mutex);
@@ -1966,6 +1967,7 @@ static int i915_context_status(struct seq_file *m, void *unused)seq_puts (m,"HW context " );describe_ctx (m, ctx); + count++;for_each_ring (ring, dev_priv, i) {if (ring->default_context == ctx) seq_printf(m, "(default context %s) ",
@@ -1994,6 +1996,8 @@ static int i915_context_status(struct seq_file *m, void *unused)seq_putc (m,'\n' ); } +seq_printf (m,"Total: %d contexts\n" , count); +mutex_unlock (&dev->struct_mutex);return 0 ;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 01/39] drm/i915: Add total count to context status debugfs output
Quoted 10 lines · On ma, 2015-11-23 at 11:38 +0000, [email protected] wrote:
From: John Harrison <[email protected]>
When there are lots and lots and even more lots of contexts (e.g. when running
with execlists) it is useful to be able to immediately see what the total
context count is.Change-Id: If9726d4df86567100ecf53867b43f4753f08bf84
For: VIZ-1587
With below notes;
Quoted 11 lines
Signed-off-by John Harrison <[email protected]>--- drivers/gpu/drm/i915/i915_debugfs.c | 4 ++++ 1 file changed, 4 insertions(+)drivers/gpu/drm/i915/i915_debugfs.c +0
-0
drivers/gpu/drm/i915/i915_debugfs.c static int i915_context_status(struct seq_file · line 1957
Quoted 5 lines
drivers/gpu/drm/i915/i915_debugfs.c +1
-0@@ -1953,6 +1953,7 @@ static int i915_context_status(struct seq_file *m, void *unused)struct drm_i915_private *dev_priv = dev->dev_private;struct intel_engine_cs *ring;struct intel_context *ctx; +uint32_t count =0 ;
Should really be just of type 'int'. Overall the driver code use 'int' for counting.
Quoted 14 lines
drivers/gpu/drm/i915/i915_debugfs.c +0
-0int ret, i;drivers/gpu/drm/i915/i915_debugfs.c +1
-0ret =mutex_lock_interruptible (&dev->struct_mutex);@@ -1966,6 +1967,7 @@ static int i915_context_status(struct seq_file *m, void *unused)seq_puts (m,"HW context " );describe_ctx (m, ctx); + count++;for_each_ring (ring, dev_priv, i) {if (ring->default_context == ctx)seq_printf(m, "(default context %s)
",
drivers/gpu/drm/i915/i915_debugfs.c static int i915_context_status(struct seq_file · line 2001
Quoted 6 lines
drivers/gpu/drm/i915/i915_debugfs.c +2
-0@@ -1994,6 +1996,8 @@ static int i915_context_status(struct seq_file *m, void *unused)seq_putc (m,'\n' ); } +seq_printf (m,"Total: %d contexts\n" , count); +
If the above variable is not changed, %d is wrong for uint32_t and unsi gned int, should be %u.
Quoted 3 lines
drivers/gpu/drm/i915/i915_debugfs.c +0
-0mutex_unlock (&dev->struct_mutex);return 0;
Signature · 8 lines
Joonas Lahtinen Open Source Technology Center Intel Corporation _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 2/39 drm/i915: Updating assorted register and status page definitions
In reply to [email protected]From: Dave Gordon <[email protected]>
Added various definitions that will be useful for the scheduler in general and pre-emptive context switching in particular.
Change-Id: Ica805b94160426def51f5d520f5ce51c60864a98
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_reg.h | 30 +++++++++++++++++++++++++++++- drivers/gpu/drm/i915/intel_ringbuffer.h | 32 +++++++++++++++++++++++++++++--- 2 files changed, 58 insertions(+), 4 deletions(-)
drivers/gpu/drm/i915/i915_reg.h +29 -1
@@ -259,6 +259,10 @@#define MI_GLOBAL_GTT (1<<22)#define MI_NOOP MI_INSTR(0, 0) +#define MI_NOOP_WRITE_ID (1<<22) +#define MI_NOOP_ID_MASK ((1<<22) - 1) +#define MI_NOOP_MID (id) ((id) & MI_NOOP_ID_MASK) +#define MI_NOOP_WITH_ID (id) MI_INSTR(0, MI_NOOP_WRITE_ID|MI_NOOP_MID(id))#define MI_USER_INTERRUPT MI_INSTR(0x02, 0)#define MI_WAIT_FOR_EVENT MI_INSTR(0x03, 0)#define MI_WAIT_FOR_OVERLAY_FLIP (1<<16)
@@ -276,6 +280,7 @@#define MI_ARB_ON_OFF MI_INSTR(0x08, 0)#define MI_ARB_ENABLE (1<<0)#define MI_ARB_DISABLE (0<<0) +#define MI_ARB_CHECK MI_INSTR(0x05, 0)#define MI_BATCH_BUFFER_END MI_INSTR(0x0a, 0)#define MI_SUSPEND_FLUSH MI_INSTR(0x0b, 0)#define MI_SUSPEND_FLUSH_EN (1<<0)
@@ -325,6 +330,8 @@#define MI_SEMAPHORE_SYNC_INVALID (3<<16)#define MI_SEMAPHORE_SYNC_MASK (3<<16)#define MI_SET_CONTEXT MI_INSTR(0x18, 0) +#define MI_CONTEXT_ADDR_MASK ((~0)<<12) +#define MI_SET_CONTEXT_FLAG_MASK ((1<<12)-1)#define MI_MM_SPACE_GTT (1<<8)#define MI_MM_SPACE_PHYSICAL (0<<8)#define MI_SAVE_EXT_STATE_EN (1<<3)
@@ -344,6 +351,10 @@#define MI_USE_GGTT (1 << 22)/* g4x+ */ #define MI_STORE_DWORD_INDEX MI_INSTR(0x21, 1)#define MI_STORE_DWORD_INDEX_SHIFT 2 +#define MI_STORE_REG_MEM MI_INSTR(0x24, 1) +#define MI_STORE_REG_MEM_GTT (1 << 22) +#define MI_STORE_REG_MEM_PREDICATE (1 << 21) +/* Official intel docs are somewhat sloppy concerning MI_LOAD_REGISTER_IMM: * - Always issue a MI_NOOP _before_ the MI_LOAD_REGISTER_IMM - otherwise hw * simply ignores the register load under certain conditions.
@@ -358,7 +369,10 @@#define MI_FLUSH_DW MI_INSTR(0x26, 1)/* for GEN6 */ #define MI_FLUSH_DW_STORE_INDEX (1<<21)#define MI_INVALIDATE_TLB (1<<18) +#define MI_FLUSH_DW_OP_NONE (0<<14)#define MI_FLUSH_DW_OP_STOREDW (1<<14) +#define MI_FLUSH_DW_OP_RSVD (2<<14) +#define MI_FLUSH_DW_OP_STAMP (3<<14)#define MI_FLUSH_DW_OP_MASK (3<<14)#define MI_FLUSH_DW_NOTIFY (1<<8)#define MI_INVALIDATE_BSD (1<<7)
@@ -1533,6 +1547,19 @@ enum skl_disp_power_wells {#define HSW_GTT_CACHE_EN 0x4024#define GTT_CACHE_EN_ALL 0xF0007FFF + +/* +* Premption-related registers +*/ +#define RING_UHPTR (base) ((base)+0x134) +#define UHPTR_GFX_ADDR_ALIGN (0x7) +#define UHPTR_VALID (0x1) +#define RING_PREEMPT_ADDR 0x0214c +#define PREEMPT_BATCH_LEVEL_MASK (0x3) +#define BB_PREEMPT_ADDR 0x02148 +#define SBB_PREEMPT_ADDR 0x0213c +#define RS_PREEMPT_STATUS 0x0215c +#define GEN7_WR_WATERMARK 0x4028#define GEN7_GFX_PRIO_CTRL 0x402C#define ARB_MODE 0x4030
@@ -6736,7 +6763,8 @@ enum skl_disp_power_wells {#define VLV_SPAREG2H 0xA194#define GTFIFODBG 0x120000 -#define GT_FIFO_SBDROPERR (1<<6) +#define GT_FIFO_CPU_ERROR_MASK 0xf +#define GT_FIFO_SDDROPERR (1<<6)#define GT_FIFO_BLOBDROPERR (1<<5)#define GT_FIFO_SB_READ_ABORTERR (1<<4)#define GT_FIFO_DROPERR (1<<3)
drivers/gpu/drm/i915/intel_ringbuffer.h +29 -3
@@ -49,6 +49,12 @@ struct intel_hw_status_page {#define I915_READ_MODE (ring) I915_READ(RING_MI_MODE((ring)->mmio_base))#define I915_WRITE_MODE (ring, val) I915_WRITE(RING_MI_MODE((ring)->mmio_base), val) +#define I915_READ_UHPTR (ring) \ + I915_READ(RING_UHPTR((ring)->mmio_base)) +#define I915_WRITE_UHPTR (ring, val) \ + I915_WRITE(RING_UHPTR((ring)->mmio_base), val) +#define I915_READ_NOPID (ring) I915_READ(RING_NOPID((ring)->mmio_base)) +/* seqno size is actually only a uint32, but since we plan to use MI_FLUSH_DW to * do the writes, and that must have qw aligned offsets, simply pretend it's 8b. */
@@ -426,10 +432,30 @@ intel_write_status_page(struct intel_engine_cs *ring,* 0x20-0x2f: Reserved (Gen6+) * * The area from dword 0x30 to 0x3ff is available for driver usage. +* +* Note: in general the allocation of these indices is arbitrary, as long +* as they are all unique. But a few of them are used with instructions that +* have specific alignment requirements, those particular indices must be +* chosen carefully to meet those requirements. The list below shows the +* currently-known alignment requirements: +* +* I915_GEM_SCRATCH_INDEX must be EVEN +*/ + +/* +* Tracking; these are updated by the GPU at the beginning and/or end of every +* batch. One pair for regular buffers, the other for preemptive ones. */ -#define I915_GEM_HWS_INDEX 0x30 -#define I915_GEM_HWS_SCRATCH_INDEX 0x40 -#define I915_GEM_HWS_SCRATCH_ADDR (I915_GEM_HWS_SCRATCH_INDEX << MI_STORE_DWORD_INDEX_SHIFT) +#define I915_BATCH_DONE_SEQNO 0x30/* Completed batch seqno */ +#define I915_BATCH_ACTIVE_SEQNO 0x31/* In progress batch seqno */ +#define I915_PREEMPTIVE_DONE_SEQNO 0x32/* Completed preemptive batch */ +#define I915_PREEMPTIVE_ACTIVE_SEQNO 0x33/* In progress preemptive batch */ + +#define I915_GEM_HWS_INDEX I915_BATCH_DONE_SEQNO/* alias */ +#define I915_GEM_ACTIVE_SEQNO_INDEX I915_BATCH_ACTIVE_SEQNO/* alias */ + +#define I915_GEM_HWS_SCRATCH_INDEX 0x38/* QWord */ +#define I915_GEM_HWS_SCRATCH_ADDR (I915_GEM_HWS_SCRATCH_INDEX << MI_STORE_DWORD_INDEX_SHIFT)struct intel_ringbuffer *intel_engine_create_ringbuffer (struct intel_engine_cs *engine,int size);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Joonas Lahtinen
maintainerRe: [PATCH 02/39] drm/i915: Updating assorted register and status page definitions
Hi,
Comments below.
Quoted 22 lines · On ma, 2015-11-23 at 11:38 +0000, [email protected] wrote:
From: Dave Gordon <[email protected]>
Added various definitions that will be useful for the scheduler in general and
pre-emptive context switching in particular.Change-Id: Ica805b94160426def51f5d520f5ce51c60864a98
For: VIZ-1587Signed-off-by Dave Gordon <[email protected]>--- drivers/gpu/drm/i915/i915_reg.h | 30 +++++++++++++++++++++++++++++- drivers/gpu/drm/i915/intel_ringbuffer.h | 32 +++++++++++++++++++++++++++++--- 2 files changed, 58 insertions(+), 4 deletions(-)drivers/gpu/drm/i915/i915_reg.h +0
-0
drivers/gpu/drm/i915/i915_reg.h line 264
Quoted 6 lines
drivers/gpu/drm/i915/i915_reg.h +3
-0@@ -259,6 +259,10 @@#define MI_GLOBAL_GTT (1<<22)#define MI_NOOP MI_INSTR(0, 0) +#define MI_NOOP_WRITE_ID (1<<22) +#define MI_NOOP_ID_MASK ((1<<22) - 1) +#define MI_NOOP_MID (id) ((id) & MI_NOOP_ID_MASK)
I'd stay consistent with the style of other _MASK/_SHIFT pairs elsewhere in this header. I find this rather unintuitive compared to 0xff notation especially if it would be with nonzero shift. MI_NOOP_MID is extra, too.
Most of these new definitions are not used anywhere in the series? Partially due to that I would prefer these to be introduced in smaller patches prior to first use, at least to an extent that one patch per series, with only stuff being used. Would make reviewing much easier to know they have been actually tested too.
Daniel, how do you want to see the register updates? As much added at once as possible, or only needed?
Quoted 149 lines
drivers/gpu/drm/i915/i915_reg.h +1
-0+#define MI_NOOP_WITH_ID (id) MI_INSTR(0, MI_NOOP_WRITE_ID|MI_NOOP_MID(id))#define MI_USER_INTERRUPT MI_INSTR(0x02, 0)#define MI_WAIT_FOR_EVENT MI_INSTR(0x03, 0)#define
MI_WAIT_FOR_OVERLAY_FLIP(1<<16)drivers/gpu/drm/i915/i915_reg.h +7
-0@@ -276,6 +280,7 @@#define MI_ARB_ON_OFF MI_INSTR(0x08, 0)#define MI_ARB_ENABLE (1<<0)#define MI_ARB_DISABLE (0<<0) +#define MI_ARB_CHECK MI_INSTR(0x05, 0)#define MI_BATCH_BUFFER_END MI_INSTR(0x0a, 0)#define MI_SUSPEND_FLUSH MI_INSTR(0x0b, 0)#define MI_SUSPEND_FLUSH_EN (1<<0)@@ -325,6 +330,8 @@#define MI_SEMAPHORE_SYNC_INVALID (3<<16)#define MI_SEMAPHORE_SYNC_MASK (3<<16)#define MI_SET_CONTEXT MI_INSTR(0x18, 0) +#define MI_CONTEXT_ADDR_MASK ((~0)<<12) +#define MI_SET_CONTEXT_FLAG_MASK ((1<<12)-1)#define MI_MM_SPACE_GTT (1<<8)#define MI_MM_SPACE_PHYSICAL (0<<8)#define MI_SAVE_EXT_STATE_EN (1<<3)@@ -344,6 +351,10 @@#define MI_USE_GGTT (1 << 22)/* g4x+ */ #define MI_STORE_DWORD_INDEX MI_INSTR(0x21, 1)#define MI_STORE_DWORD_INDEX_SHIFT 2 +#define MI_STORE_REG_MEM MI_INSTR(0x24, 1) +#define MI_STORE_REG_MEM_GTT (1 << 22) +#define MI_STORE_REG_MEM_PREDICATE (1 << 21) +/* Official intel docs are somewhat sloppy concerning
MI_LOAD_REGISTER_IMM:
* - Always issue aMI_NOOP_before_ theMI_LOAD_REGISTER_IMM-
otherwise hw
* simply ignores the register load under certain conditions.drivers/gpu/drm/i915/i915_reg.h +18
-1@@ -358,7 +369,10 @@#define MI_FLUSH_DW MI_INSTR(0x26, 1)/* for GEN6 */ #define MI_FLUSH_DW_STORE_INDEX (1<<21)#define MI_INVALIDATE_TLB (1<<18) +#define MI_FLUSH_DW_OP_NONE (0<<14)#define MI_FLUSH_DW_OP_STOREDW (1<<14) +#define MI_FLUSH_DW_OP_RSVD (2<<14) +#define MI_FLUSH_DW_OP_STAMP (3<<14)#define MI_FLUSH_DW_OP_MASK (3<<14)#define MI_FLUSH_DW_NOTIFY (1<<8)#define MI_INVALIDATE_BSD (1<<7)@@ -1533,6 +1547,19 @@ enum skl_disp_power_wells {#define HSW_GTT_CACHE_EN 0x4024#define GTT_CACHE_EN_ALL 0xF0007FFF + +/* +* Premption-related registers +*/ +#define RING_UHPTR (base) ((base)+0x134) +#define UHPTR_GFX_ADDR_ALIGN (0x7) +#define UHPTR_VALID (0x1) +#define RING_PREEMPT_ADDR 0x0214c +#define PREEMPT_BATCH_LEVEL_MASK (0x3) +#define BB_PREEMPT_ADDR 0x02148 +#define SBB_PREEMPT_ADDR 0x0213c +#define RS_PREEMPT_STATUS 0x0215c +#define GEN7_WR_WATERMARK 0x4028#define GEN7_GFX_PRIO_CTRL 0x402C#define ARB_MODE 0x4030@@ -6736,7 +6763,8 @@ enum skl_disp_power_wells {#define VLV_SPAREG2H 0xA194#define GTFIFODBG 0x120000 -#define GT_FIFO_SBDROPERR (1<<6) +#define GT_FIFO_CPU_ERROR_MASK 0xf +#define GT_FIFO_SDDROPERR (1<<6)#define GT_FIFO_BLOBDROPERR (1<<5)#define GT_FIFO_SB_READ_ABORTERR (1<<4)#define GT_FIFO_DROPERR (1<<3)drivers/gpu/drm/i915/intel_ringbuffer.h +6
-3@@ -49,6 +49,12 @@ struct intel_hw_status_page {#define I915_READ_MODE (ring) I915_READ(RING_MI_MODE((ring) ->mmio_base))#define I915_WRITE_MODE (ring, val) I915_WRITE(RING_MI_MODE((ring) ->mmio_base), val) +#define I915_READ_UHPTR (ring) \ + I915_READ(RING_UHPTR((ring)->mmio_base)) +#define I915_WRITE_UHPTR (ring, val) \ + I915_WRITE(RING_UHPTR((ring)->mmio_base), val) +#define I915_READ_NOPID (ring) I915_READ(RING_NOPID((ring) ->mmio_base)) +/* seqno size is actually only a uint32, but since we plan to use
MI_FLUSH_DWto
* do the writes, and that must have qw aligned offsets, simply
pretend it's 8b.
*/drivers/gpu/drm/i915/intel_ringbuffer.h +2
-0@@ -426,10 +432,30 @@ intel_write_status_page(struct intel_engine_cs *ring, * 0x20-0x2f: Reserved (Gen6+) * * The area from dword 0x30 to 0x3ff is available for driver usage. + * + * Note: in general the allocation of these indices is arbitrary, aslong
+ * as they are all unique. But a few of them are used with instructions that
+ * have specific alignment requirements, those particular indices must be
+ * chosen carefully to meet those requirements. The list below shows the
+ * currently-known alignment requirements:
+ *
+ *I915_GEM_SCRATCH_INDEXmust be EVEN
+ */drivers/gpu/drm/i915/intel_ringbuffer.h +3
-0+ +/* +* Tracking; these are updated by the GPU at the beginning and/or end of every
+ * batch. One pair for regular buffers, the other for preemptive ones.
*/drivers/gpu/drm/i915/intel_ringbuffer.h +0
-3-#define I915_GEM_HWS_INDEX 0x30 -#define I915_GEM_HWS_SCRATCH_INDEX 0x40 -#define I915_GEM_HWS_SCRATCH_ADDR (I915_GEM_HWS_SCRATCH_INDEX <<
MI_STORE_DWORD_INDEX_SHIFT)
+#defineI915_BATCH_DONE_SEQNO0x30/* Completed
batch seqno */
+#defineI915_BATCH_ACTIVE_SEQNO0x31/* In progress
batch seqno */
+#defineI915_PREEMPTIVE_DONE_SEQNO0x32/* Completed
preemptive batch */
+#defineI915_PREEMPTIVE_ACTIVE_SEQNO0x33/* In progress
preemptive batch */drivers/gpu/drm/i915/intel_ringbuffer.h +2
-0+ +#define I915_GEM_HWS_INDEX I915_BATCH_DONE_SEQNO/* alias */
+#defineI915_GEM_ACTIVE_SEQNO_INDEXI915_BATCH_ACTIVE_SEQNO
/* alias */drivers/gpu/drm/i915/intel_ringbuffer.h +3
-0+ +#define I915_GEM_HWS_SCRATCH_INDEX 0x38/* QWord */ +#define I915_GEM_HWS_SCRATCH_ADDR (I915_GEM_HWS_SCRATCH_INDEX<<
MI_STORE_DWORD_INDEX_SHIFT)struct
intel_ringbuffer* intel_engine_create_ringbuffer(structintel_engine_cs*engine, int
size);
Signature · 8 lines
Joonas Lahtinen Open Source Technology Center Intel Corporation _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 02/39] drm/i915: Updating assorted register and status page definitions
Quoted 39 lines · Joonas Lahtinen
Hi,
Comments below.
Quoted 22 lines · On ma, 2015-11-23 at 11:38 +0000, [email protected] wrote:
From: Dave Gordon <[email protected]>
Added various definitions that will be useful for the scheduler in general and
pre-emptive context switching in particular.Change-Id: Ica805b94160426def51f5d520f5ce51c60864a98
For: VIZ-1587Signed-off-by Dave Gordon <[email protected]>--- drivers/gpu/drm/i915/i915_reg.h | 30 +++++++++++++++++++++++++++++- drivers/gpu/drm/i915/intel_ringbuffer.h | 32 +++++++++++++++++++++++++++++--- 2 files changed, 58 insertions(+), 4 deletions(-)drivers/gpu/drm/i915/i915_reg.h +0
-0drivers/gpu/drm/i915/i915_reg.h line 264
Quoted 6 lines
drivers/gpu/drm/i915/i915_reg.h +3
-0@@ -259,6 +259,10 @@#define MI_GLOBAL_GTT (1<<22)#define MI_NOOP MI_INSTR(0, 0) +#define MI_NOOP_WRITE_ID (1<<22) +#define MI_NOOP_ID_MASK ((1<<22) - 1) +#define MI_NOOP_MID (id) ((id) & MI_NOOP_ID_MASK)I'd stay consistent with the style of other _MASK/_SHIFT pairs elsewhere in this header. I find this rather unintuitive compared to
0xffnotation especially if it would be with nonzero shift.MI_NOOP_MIDis extra, too.
Quoted 7 lines
Most of these new definitions are not used anywhere in the series? Partially due to that I would prefer these to be introduced in smaller patches prior to first use, at least to an extent that one patch per series, with only stuff being used. Would make reviewing much easier to know they have been actually tested too.
Daniel, how do you want to see the register updates? As much added at once as possible, or only needed?
I'm fine with a separate patch to just update them all. We do that regularly for feature work.
-Daniel
Quoted 155 lines
Quoted 150 lines
drivers/gpu/drm/i915/i915_reg.h +1
-0+#define MI_NOOP_WITH_ID (id) MI_INSTR(0, MI_NOOP_WRITE_ID|MI_NOOP_MID(id))#define MI_USER_INTERRUPT MI_INSTR(0x02, 0)#define MI_WAIT_FOR_EVENT MI_INSTR(0x03, 0)#define
MI_WAIT_FOR_OVERLAY_FLIP(1<<16)drivers/gpu/drm/i915/i915_reg.h +7
-0@@ -276,6 +280,7 @@#define MI_ARB_ON_OFF MI_INSTR(0x08, 0)#define MI_ARB_ENABLE (1<<0)#define MI_ARB_DISABLE (0<<0) +#define MI_ARB_CHECK MI_INSTR(0x05, 0)#define MI_BATCH_BUFFER_END MI_INSTR(0x0a, 0)#define MI_SUSPEND_FLUSH MI_INSTR(0x0b, 0)#define MI_SUSPEND_FLUSH_EN (1<<0)@@ -325,6 +330,8 @@#define MI_SEMAPHORE_SYNC_INVALID (3<<16)#define MI_SEMAPHORE_SYNC_MASK (3<<16)#define MI_SET_CONTEXT MI_INSTR(0x18, 0) +#define MI_CONTEXT_ADDR_MASK ((~0)<<12) +#define MI_SET_CONTEXT_FLAG_MASK ((1<<12)-1)#define MI_MM_SPACE_GTT (1<<8)#define MI_MM_SPACE_PHYSICAL (0<<8)#define MI_SAVE_EXT_STATE_EN (1<<3)@@ -344,6 +351,10 @@#define MI_USE_GGTT (1 << 22)/* g4x+ */ #define MI_STORE_DWORD_INDEX MI_INSTR(0x21, 1)#define MI_STORE_DWORD_INDEX_SHIFT 2 +#define MI_STORE_REG_MEM MI_INSTR(0x24, 1) +#define MI_STORE_REG_MEM_GTT (1 << 22) +#define MI_STORE_REG_MEM_PREDICATE (1 << 21) +/* Official intel docs are somewhat sloppy concerning
MI_LOAD_REGISTER_IMM:
* - Always issue aMI_NOOP_before_ theMI_LOAD_REGISTER_IMM-
otherwise hw
* simply ignores the register load under certain conditions.drivers/gpu/drm/i915/i915_reg.h +18
-1@@ -358,7 +369,10 @@#define MI_FLUSH_DW MI_INSTR(0x26, 1)/* for GEN6 */ #define MI_FLUSH_DW_STORE_INDEX (1<<21)#define MI_INVALIDATE_TLB (1<<18) +#define MI_FLUSH_DW_OP_NONE (0<<14)#define MI_FLUSH_DW_OP_STOREDW (1<<14) +#define MI_FLUSH_DW_OP_RSVD (2<<14) +#define MI_FLUSH_DW_OP_STAMP (3<<14)#define MI_FLUSH_DW_OP_MASK (3<<14)#define MI_FLUSH_DW_NOTIFY (1<<8)#define MI_INVALIDATE_BSD (1<<7)@@ -1533,6 +1547,19 @@ enum skl_disp_power_wells {#define HSW_GTT_CACHE_EN 0x4024#define GTT_CACHE_EN_ALL 0xF0007FFF + +/* +* Premption-related registers +*/ +#define RING_UHPTR (base) ((base)+0x134) +#define UHPTR_GFX_ADDR_ALIGN (0x7) +#define UHPTR_VALID (0x1) +#define RING_PREEMPT_ADDR 0x0214c +#define PREEMPT_BATCH_LEVEL_MASK (0x3) +#define BB_PREEMPT_ADDR 0x02148 +#define SBB_PREEMPT_ADDR 0x0213c +#define RS_PREEMPT_STATUS 0x0215c +#define GEN7_WR_WATERMARK 0x4028#define GEN7_GFX_PRIO_CTRL 0x402C#define ARB_MODE 0x4030@@ -6736,7 +6763,8 @@ enum skl_disp_power_wells {#define VLV_SPAREG2H 0xA194#define GTFIFODBG 0x120000 -#define GT_FIFO_SBDROPERR (1<<6) +#define GT_FIFO_CPU_ERROR_MASK 0xf +#define GT_FIFO_SDDROPERR (1<<6)#define GT_FIFO_BLOBDROPERR (1<<5)#define GT_FIFO_SB_READ_ABORTERR (1<<4)#define GT_FIFO_DROPERR (1<<3)drivers/gpu/drm/i915/intel_ringbuffer.h +6
-3@@ -49,6 +49,12 @@ struct intel_hw_status_page {#define I915_READ_MODE (ring) I915_READ(RING_MI_MODE((ring) ->mmio_base))#define I915_WRITE_MODE (ring, val) I915_WRITE(RING_MI_MODE((ring) ->mmio_base), val) +#define I915_READ_UHPTR (ring) \ + I915_READ(RING_UHPTR((ring)->mmio_base)) +#define I915_WRITE_UHPTR (ring, val) \ + I915_WRITE(RING_UHPTR((ring)->mmio_base), val) +#define I915_READ_NOPID (ring) I915_READ(RING_NOPID((ring) ->mmio_base)) +/* seqno size is actually only a uint32, but since we plan to use
MI_FLUSH_DWto
* do the writes, and that must have qw aligned offsets, simply
pretend it's 8b.
*/drivers/gpu/drm/i915/intel_ringbuffer.h +2
-0@@ -426,10 +432,30 @@ intel_write_status_page(struct intel_engine_cs *ring, * 0x20-0x2f: Reserved (Gen6+) * * The area from dword 0x30 to 0x3ff is available for driver usage. + * + * Note: in general the allocation of these indices is arbitrary, aslong
+ * as they are all unique. But a few of them are used with instructions that
+ * have specific alignment requirements, those particular indices must be
+ * chosen carefully to meet those requirements. The list below shows the
+ * currently-known alignment requirements:
+ *
+ *I915_GEM_SCRATCH_INDEXmust be EVEN
+ */drivers/gpu/drm/i915/intel_ringbuffer.h +3
-0+ +/* +* Tracking; these are updated by the GPU at the beginning and/or end of every
+ * batch. One pair for regular buffers, the other for preemptive ones.
*/drivers/gpu/drm/i915/intel_ringbuffer.h +0
-3-#define I915_GEM_HWS_INDEX 0x30 -#define I915_GEM_HWS_SCRATCH_INDEX 0x40 -#define I915_GEM_HWS_SCRATCH_ADDR (I915_GEM_HWS_SCRATCH_INDEX <<
MI_STORE_DWORD_INDEX_SHIFT)
+#defineI915_BATCH_DONE_SEQNO0x30/* Completed
batch seqno */
+#defineI915_BATCH_ACTIVE_SEQNO0x31/* In progress
batch seqno */
+#defineI915_PREEMPTIVE_DONE_SEQNO0x32/* Completed
preemptive batch */
+#defineI915_PREEMPTIVE_ACTIVE_SEQNO0x33/* In progress
preemptive batch */drivers/gpu/drm/i915/intel_ringbuffer.h +2
-0+ +#define I915_GEM_HWS_INDEX I915_BATCH_DONE_SEQNO/* alias */
+#defineI915_GEM_ACTIVE_SEQNO_INDEXI915_BATCH_ACTIVE_SEQNO
/* alias */drivers/gpu/drm/i915/intel_ringbuffer.h +3
-0+ +#define I915_GEM_HWS_SCRATCH_INDEX 0x38/* QWord */ +#define I915_GEM_HWS_SCRATCH_ADDR (I915_GEM_HWS_SCRATCH_INDEX<<
MI_STORE_DWORD_INDEX_SHIFT)struct
intel_ringbuffer* intel_engine_create_ringbuffer(structintel_engine_cs*engine, int
size);Signature · 3 lines
Joonas Lahtinen Open Source Technology Center Intel Corporation
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 3/39 drm/i915: Explicit power enable during deferred context initialisation
In reply to [email protected]From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware submission part in the back half.
While that is a good thing in general, it causes a problem with the deferred context initialisation. That is done quite early on in the execbuf code path - it happens at context validation time rather than context switch time. Some of the deferred work requires the power to be enabled. Hence this patch adds an explicit power reference count to the deferred initialisation code itself.
Change-Id: Id7b1535dfd8809a2bd5546272de2bbec39da2868
Issue: GMINL-5159
--- drivers/gpu/drm/i915/intel_lrc.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/intel_lrc.c +8 -1
@@ -2470,6 +2470,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx,WARN_ON (ctx->legacy_hw_ctx.rcs_state !=NULL );WARN_ON (ctx->engine[ring->id].state); +intel_runtime_pm_get (dev->dev_private); + context_size =round_up (get_lr_context_size (ring),4096 );/* One extra page as the sharing data between driver and GuC */
@@ -2478,7 +2480,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx, ctx_obj =i915_gem_alloc_object (dev, context_size);if (!ctx_obj) {DRM_DEBUG_DRIVER ("Alloc LRC backing obj failed.\n" ); -return -ENOMEM ; + ret = -ENOMEM ; + goto error_pm; } ringbuf =intel_engine_create_ringbuffer (ring,4 *PAGE_SIZE );
@@ -2524,6 +2527,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx, }i915_add_request_no_flush (req); } + +intel_runtime_pm_put (dev->dev_private);return 0 ; error_ringbuf:
@@ -2532,6 +2537,8 @@ error_deref_obj:drm_gem_object_unreference (&ctx_obj->base); ctx->engine[ring->id].ringbuf =NULL ; ctx->engine[ring->id].state =NULL ; +error_pm: +intel_runtime_pm_put (dev->dev_private);return ret; }
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH 03/39] drm/i915: Explicit power enable during deferred context initialisation
Quoted 17 lines · On ma, 2015-11-23 at 11:38 +0000, [email protected] wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware
submission part in the back half.While that is a good thing in general, it causes a problem with the deferred context initialisation. That is done quite early on in the execbuf code path - it happens at context validation time rather than context switch time. Some of the deferred work requires the power to be enabled. Hence this patch adds an explicit power reference count to
the deferred initialisation code itself.
Quoted 2 lines
Change-Id: Id7b1535dfd8809a2bd5546272de2bbec39da2868
Issue: GMINL-5159
Quoted 52 lines
Signed-off-by John Harrison <[email protected]>--- drivers/gpu/drm/i915/intel_lrc.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)drivers/gpu/drm/i915/intel_lrc.c +2
-0@@ -2470,6 +2470,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx,WARN_ON (ctx->legacy_hw_ctx.rcs_state !=NULL );WARN_ON (ctx->engine[ring->id].state); +intel_runtime_pm_get (dev->dev_private); + context_size =round_up (get_lr_context_size (ring),4096 );/* One extra page as the sharing data between driver and GuC
*/drivers/gpu/drm/i915/intel_lrc.c +2
-1@@ -2478,7 +2480,8 @@ int intel_lr_context_deferred_alloc(structintel_context *ctx, ctx_obj =i915_gem_alloc_object (dev, context_size);if (!ctx_obj) {DRM_DEBUG_DRIVER ("Alloc LRC backing obj failed.\n" ); -return -ENOMEM ; + ret = -ENOMEM ; + goto error_pm; }ringbuf = intel_engine_create_ringbuffer(ring, 4 *
PAGE_SIZE);drivers/gpu/drm/i915/intel_lrc.c +2
-0@@ -2524,6 +2527,8 @@ int intel_lr_context_deferred_alloc(structintel_context *ctx, }i915_add_request_no_flush (req ); } + +intel_runtime_pm_put (dev->dev_private);return 0 ;
error_ringbuf:drivers/gpu/drm/i915/intel_lrc.c +2
-0@@ -2532,6 +2537,8 @@ error_deref_obj:drm_gem_object_unreference (&ctx_obj->base); ctx->engine[ring->id].ringbuf =NULL ; ctx->engine[ring->id].state =NULL ; +error_pm: +intel_runtime_pm_put (dev->dev_private);return ret; }
Signature · 8 lines
Joonas Lahtinen Open Source Technology Center Intel Corporation _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 4/39 drm/i915: Prelude to splitting i915_gem_do_execbuffer in two
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler decouples the submission of batch buffers to the driver with their submission to the hardware. This basically means splitting the execbuffer() function in half. This change rearranges some code ready for the split to occur.
Change-Id: Icc9c8afaac18821f3eb8a151a49f918f90c068a3
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 64 ++++++++++++++++++------------ drivers/gpu/drm/i915/intel_lrc.c | 18 ++++++--- 2 files changed, 51 insertions(+), 31 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +38 -26
@@ -935,10 +935,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,if (flush_domains &I915_GEM_DOMAIN_GTT )wmb (); -/* Unconditionally invalidate gpu caches and ensure that we do flush -* any residual writes from the previous batch. -*/ -return intel_ring_invalidate_all_caches (req); +return 0 ; } static bool
@@ -1191,17 +1188,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,u32 instp_mask;int ret; - ret =i915_gem_execbuffer_move_to_gpu (params->request, vmas); -if (ret) -return ret; - - ret =i915_switch_context (params->request); -if (ret) -return ret; - -WARN (params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1 <<ring->id), -"%s didn't clear reload\n" , ring->name); - instp_mode = args->flags &I915_EXEC_CONSTANTS_MASK ; instp_mask =I915_EXEC_CONSTANTS_MASK ;switch (instp_mode) {
@@ -1235,11 +1221,37 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,return -EINVAL ; } + ret =i915_gem_execbuffer_move_to_gpu (params->request, vmas); +if (ret) +return ret; + +i915_gem_execbuffer_move_to_active (vmas, params->request); + +/* To be split into two functions here... */ + +intel_runtime_pm_get (dev_priv); + +/* +* Unconditionally invalidate gpu caches and ensure that we do flush +* any residual writes from the previous batch. +*/ + ret =intel_ring_invalidate_all_caches (params->request); +if (ret) + goto error; + +/* Switch to the correct context for the batch */ + ret =i915_switch_context (params->request); +if (ret) + goto error; + +WARN (params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1 <<ring->id), +"%s didn't clear reload\n" , ring->name); +if (ring == &dev_priv->ring[RCS ] && instp_mode != dev_priv->relative_constants_mode) { ret =intel_ring_begin (params->request,4 );if (ret) -return ret; + goto error;intel_ring_emit (ring,MI_NOOP );intel_ring_emit (ring,MI_LOAD_REGISTER_IMM (1 ));
@@ -1253,7 +1265,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,if (args->flags &I915_EXEC_GEN7_SOL_RESET ) { ret =i915_reset_gen7_sol_offsets (dev, params->request);if (ret) -return ret; + goto error; } exec_len = args->batch_len;
@@ -1264,14 +1276,20 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params, exec_start, exec_len, params->dispatch_flags);if (ret) -return ret; + goto error;trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); -i915_gem_execbuffer_move_to_active (vmas, params->request);i915_gem_execbuffer_retire_commands (params); -return 0 ; +error: +/* +* intel_gpu_busy should also get a ref, so it will free when the device +* is really idle. +*/ +intel_runtime_pm_put (dev_priv); + +return ret; }/**
@@ -1480,8 +1498,6 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, args->flags &= ~I915_EXEC_WAIT_FENCE ; } -intel_runtime_pm_get (dev_priv); - ret =i915_mutex_lock_interruptible (dev);if (ret) goto pre_mutex_err;
@@ -1688,10 +1704,6 @@ err:mutex_unlock (&dev->struct_mutex); pre_mutex_err: -/* intel_gpu_busy should also get a ref, so it will free when the device -* is really idle. */ -intel_runtime_pm_put (dev_priv); -return ret; }
drivers/gpu/drm/i915/intel_lrc.c +13 -5
@@ -647,10 +647,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,if (flush_domains &I915_GEM_DOMAIN_GTT )wmb (); -/* Unconditionally invalidate gpu caches and ensure that we do flush -* any residual writes from the previous batch. -*/ -return logical_ring_invalidate_all_caches (req); +return 0 ; }int intel_logical_ring_alloc_request_extras (struct drm_i915_gem_request *request)
@@ -913,6 +910,18 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,if (ret)return ret; +i915_gem_execbuffer_move_to_active (vmas, params->request); + +/* To be split into two functions here... */ + +/* +* Unconditionally invalidate gpu caches and ensure that we do flush +* any residual writes from the previous batch. +*/ + ret =logical_ring_invalidate_all_caches (params->request); +if (ret) +return ret; +if (ring == &dev_priv->ring[RCS ] && instp_mode != dev_priv->relative_constants_mode) { ret =intel_logical_ring_begin (params->request,4 );
@@ -937,7 +946,6 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); -i915_gem_execbuffer_move_to_active (vmas, params->request);i915_gem_execbuffer_retire_commands (params);return 0 ;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 5/39 drm/i915: Split i915_dem_do_execbuffer() in half
In reply to [email protected]From: John Harrison <[email protected]>
Split the execbuffer() function in half. The first half collects and validates all the information requried to process the batch buffer. It also does all the object pinning, relocations, active list management, etc - basically anything that must be done upfront before the IOCTL returns and allows the user land side to start changing/freeing things. The second half does the actual ring submission.
This change implements the split but leaves the back half being called directly from the end of the front half.
v2: Updated due to changes in underlying tree - addition of sync fence support and removal of cliprects.
Change-Id: I5e1c77639ce526ab2401b0323186c518bf13da0a
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 11 +++ drivers/gpu/drm/i915/i915_gem.c | 2 + drivers/gpu/drm/i915/i915_gem_execbuffer.c | 130 ++++++++++++++++++++--------- drivers/gpu/drm/i915/intel_lrc.c | 58 +++++++++---- drivers/gpu/drm/i915/intel_lrc.h | 1 + 5 files changed, 146 insertions(+), 56 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +11 -0
@@ -1679,10 +1679,18 @@ struct i915_execbuffer_params {struct drm_device *dev;struct drm_file *file;uint32_t dispatch_flags; +uint32_t args_flags;uint32_t args_batch_start_offset; +uint32_t args_batch_len; +uint32_t args_num_cliprects; +uint32_t args_DR1; +uint32_t args_DR4;uint64_t batch_obj_vm_offset;struct intel_engine_cs *ring;struct drm_i915_gem_object *batch_obj; +struct drm_clip_rect *cliprects; +uint32_t instp_mask; +int instp_mode;struct intel_context *ctx;struct drm_i915_gem_request *request; };
@@ -1944,6 +1952,7 @@ struct drm_i915_private {int (*execbuf_submit)(struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int (*execbuf_final)(struct i915_execbuffer_params *params);int (*init_rings)(struct drm_device *dev);void (*cleanup_ring)(struct intel_engine_cs *ring);void (*stop_ring)(struct intel_engine_cs *ring);
@@ -2798,9 +2807,11 @@ int i915_gem_sw_finish_ioctl(struct drm_device *dev, void *data,void i915_gem_execbuffer_move_to_active (struct list_head *vmas,struct drm_i915_gem_request *req);void i915_gem_execbuffer_retire_commands (struct i915_execbuffer_params *params); +void i915_gem_execbuff_release_batch_obj (struct drm_i915_gem_object *batch_obj);int i915_gem_ringbuffer_submission (struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int i915_gem_ringbuffer_submission_final (struct i915_execbuffer_params *params);int i915_gem_execbuffer (struct drm_device *dev,void *data,struct drm_file *file_priv); int i915_gem_execbuffer2(struct drm_device *dev, void *data,
drivers/gpu/drm/i915/i915_gem.c +2 -0
@@ -5256,11 +5256,13 @@ int i915_gem_init(struct drm_device *dev)if (!i915.enable_execlists) { dev_priv->gt.execbuf_submit = i915_gem_ringbuffer_submission; + dev_priv->gt.execbuf_final = i915_gem_ringbuffer_submission_final; dev_priv->gt.init_rings = i915_gem_init_rings; dev_priv->gt.cleanup_ring = intel_cleanup_ring_buffer; dev_priv->gt.stop_ring = intel_stop_ring_buffer; }else { dev_priv->gt.execbuf_submit = intel_execlists_submission; + dev_priv->gt.execbuf_final = intel_execlists_submission_final; dev_priv->gt.init_rings = intel_logical_rings_init; dev_priv->gt.cleanup_ring = intel_logical_ring_cleanup; dev_priv->gt.stop_ring = intel_logical_ring_stop;
drivers/gpu/drm/i915/i915_gem_execbuffer.c +91 -39
214 lines not on this page · raw message
drivers/gpu/drm/i915/intel_lrc.c +41 -17
104 lines not on this page · raw message
drivers/gpu/drm/i915/intel_lrc.h +1 -0
@@ -93,6 +93,7 @@ struct i915_execbuffer_params;int intel_execlists_submission (struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int intel_execlists_submission_final (struct i915_execbuffer_params *params);u32 intel_execlists_ctx_id (struct drm_i915_gem_object *ctx_obj);void intel_lrc_irq_handler (struct intel_engine_cs *ring);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 5/40 drm/i915: Split i915_dem_do_execbuffer() in half
From: John Harrison <[email protected]>
Split the execbuffer() function in half. The first half collects and validates all the information requried to process the batch buffer. It also does all the object pinning, relocations, active list management, etc - basically anything that must be done upfront before the IOCTL returns and allows the user land side to start changing/freeing things. The second half does the actual ring submission.
This change implements the split but leaves the back half being called directly from the end of the front half.
v2: Updated due to changes in underlying tree - addition of sync fence support and removal of cliprects.
v3: Moved local 'ringbuf' variable to make later patches in the series a bit neater.
Change-Id: I5e1c77639ce526ab2401b0323186c518bf13da0a
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 11 +++ drivers/gpu/drm/i915/i915_gem.c | 2 + drivers/gpu/drm/i915/i915_gem_execbuffer.c | 130 ++++++++++++++++++++--------- drivers/gpu/drm/i915/intel_lrc.c | 57 +++++++++---- drivers/gpu/drm/i915/intel_lrc.h | 1 + 5 files changed, 145 insertions(+), 56 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +11 -0
@@ -1679,10 +1679,18 @@ struct i915_execbuffer_params {struct drm_device *dev;struct drm_file *file;uint32_t dispatch_flags; +uint32_t args_flags;uint32_t args_batch_start_offset; +uint32_t args_batch_len; +uint32_t args_num_cliprects; +uint32_t args_DR1; +uint32_t args_DR4;uint64_t batch_obj_vm_offset;struct intel_engine_cs *ring;struct drm_i915_gem_object *batch_obj; +struct drm_clip_rect *cliprects; +uint32_t instp_mask; +int instp_mode;struct intel_context *ctx;struct drm_i915_gem_request *request; };
@@ -1944,6 +1952,7 @@ struct drm_i915_private {int (*execbuf_submit)(struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int (*execbuf_final)(struct i915_execbuffer_params *params);int (*init_rings)(struct drm_device *dev);void (*cleanup_ring)(struct intel_engine_cs *ring);void (*stop_ring)(struct intel_engine_cs *ring);
@@ -2798,9 +2807,11 @@ int i915_gem_sw_finish_ioctl(struct drm_device *dev, void *data,void i915_gem_execbuffer_move_to_active (struct list_head *vmas,struct drm_i915_gem_request *req);void i915_gem_execbuffer_retire_commands (struct i915_execbuffer_params *params); +void i915_gem_execbuff_release_batch_obj (struct drm_i915_gem_object *batch_obj);int i915_gem_ringbuffer_submission (struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int i915_gem_ringbuffer_submission_final (struct i915_execbuffer_params *params);int i915_gem_execbuffer (struct drm_device *dev,void *data,struct drm_file *file_priv); int i915_gem_execbuffer2(struct drm_device *dev, void *data,
drivers/gpu/drm/i915/i915_gem.c +2 -0
@@ -5257,11 +5257,13 @@ int i915_gem_init(struct drm_device *dev)if (!i915.enable_execlists) { dev_priv->gt.execbuf_submit = i915_gem_ringbuffer_submission; + dev_priv->gt.execbuf_final = i915_gem_ringbuffer_submission_final; dev_priv->gt.init_rings = i915_gem_init_rings; dev_priv->gt.cleanup_ring = intel_cleanup_ring_buffer; dev_priv->gt.stop_ring = intel_stop_ring_buffer; }else { dev_priv->gt.execbuf_submit = intel_execlists_submission; + dev_priv->gt.execbuf_final = intel_execlists_submission_final; dev_priv->gt.init_rings = intel_logical_rings_init; dev_priv->gt.cleanup_ring = intel_logical_ring_cleanup; dev_priv->gt.stop_ring = intel_logical_ring_stop;
drivers/gpu/drm/i915/i915_gem_execbuffer.c +91 -39
214 lines not on this page · raw message
drivers/gpu/drm/i915/intel_lrc.c +40 -17
103 lines not on this page · raw message
drivers/gpu/drm/i915/intel_lrc.h +1 -0
@@ -93,6 +93,7 @@ struct i915_execbuffer_params;int intel_execlists_submission (struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int intel_execlists_submission_final (struct i915_execbuffer_params *params);u32 intel_execlists_ctx_id (struct drm_i915_gem_object *ctx_obj);void intel_lrc_irq_handler (struct intel_engine_cs *ring);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 6/39 drm/i915: Re-instate request->uniq because it is extremely useful
In reply to [email protected]From: John Harrison <[email protected]>
The seqno value cannot always be used when debugging issues via trace points. This is because it can be reset back to start, especially during TDR type tests. Also, when the scheduler arrives the seqno is only valid while a given request is executing on the hardware. While the request is simply queued waiting for submission, it's seqno value will be zero (meaning invalid).
For: VIZ-5115
--- drivers/gpu/drm/i915/i915_drv.h | 5 +++++ drivers/gpu/drm/i915/i915_gem.c | 3 ++- drivers/gpu/drm/i915/i915_trace.h | 25 +++++++++++++++++-------- 3 files changed, 24 insertions(+), 9 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +5 -0
@@ -1963,6 +1963,8 @@ struct drm_i915_private {/* perform PHY state sanity checks? */ bool chv_phy_assert[2 ]; +uint32_t request_uniq; +/* * NOTE: This is the dri1/ums dungeon, don't add stuff here. Your patch * will be rejected. Instead look for a better place.
@@ -2210,6 +2212,9 @@ struct drm_i915_gem_request {/** GEM sequence number associated with this request. */ uint32_t seqno; +/* Unique identifier which can be used for trace points & debug */ +uint32_t uniq; +/** Position in the ringbuffer of the start of the request */ u32 head;
drivers/gpu/drm/i915/i915_gem.c +2 -1
@@ -2862,7 +2862,7 @@ static void i915_gem_request_fence_value_str(struct fence *req_fence, char *str, req =container_of (req_fence,typeof (*req), fence); -snprintf (str, size,"%d [%d]" , req->fence.seqno, req->seqno); +snprintf (str, size,"%d [%d:%d]" , req->fence.seqno, req->uniq, req->seqno); }static const struct fence_ops i915_gem_request_fops = {
@@ -3015,6 +3015,7 @@ int i915_gem_request_alloc(struct intel_engine_cs *ring, req->i915 = dev_priv; req->ring = ring; + req->uniq = dev_priv->request_uniq++; req->ctx = ctx;i915_gem_context_reference (req->ctx);
drivers/gpu/drm/i915/i915_trace.h +17 -8
@@ -469,6 +469,7 @@ TRACE_EVENT(i915_gem_ring_sync_to, __field(u32, dev) __field(u32, sync_from) __field(u32, sync_to) + __field(u32, uniq_to)__field (u32, seqno) ),
@@ -476,13 +477,14 @@ TRACE_EVENT(i915_gem_ring_sync_to, __entry->dev = from->dev->primary->index; __entry->sync_from = from->id; __entry->sync_to = to_req->ring->id; + __entry->uniq_to = to_req->uniq; __entry->seqno =i915_gem_request_get_seqno (req); ), -TP_printk ("dev=%u, sync-from=%u, sync-to=%u, seqno=%u" , +TP_printk ("dev=%u, sync-from=%u, sync-to=%u, seqno=%u, to_uniq=%u" , __entry->dev, __entry->sync_from, __entry->sync_to, - __entry->seqno) + __entry->seqno, __entry->uniq_to) ); TRACE_EVENT(i915_gem_ring_dispatch,
@@ -517,6 +519,7 @@ TRACE_EVENT(i915_gem_ring_flush,TP_STRUCT__entry (__field (u32 ,dev )__field (u32, ring) +__field (u32, uniq)__field (u32, invalidate)__field (u32, flush) ),
@@ -524,12 +527,13 @@ TRACE_EVENT(i915_gem_ring_flush,TP_fast_assign ( __entry->dev = req->ring->dev->primary->index; __entry->ring = req->ring->id; + __entry->uniq = req->uniq; __entry->invalidate = invalidate; __entry->flush = flush; ), -TP_printk ("dev=%u, ring=%x, invalidate=%04x, flush=%04x" , - __entry->dev, __entry->ring, +TP_printk ("dev=%u, ring=%x, request=%u, invalidate=%04x, flush=%04x" , + __entry->dev, __entry->ring, __entry->uniq, __entry->invalidate, __entry->flush) );
@@ -540,6 +544,7 @@ DECLARE_EVENT_CLASS(i915_gem_request,TP_STRUCT__entry (__field (u32 ,dev )__field (u32, ring) +__field (u32, uniq)__field (u32, seqno) ),
@@ -548,11 +553,13 @@ DECLARE_EVENT_CLASS(i915_gem_request,i915_gem_request_get_ring (req); __entry->dev = ring->dev->primary->index; __entry->ring = ring->id; + __entry->uniq = req ? req->uniq :0 ; __entry->seqno =i915_gem_request_get_seqno (req); ), -TP_printk ("dev=%u, ring=%u, seqno=%u" , - __entry->dev, __entry->ring, __entry->seqno) +TP_printk ("dev=%u, ring=%u, uniq=%u, seqno=%u" , + __entry->dev, __entry->ring, __entry->uniq, + __entry->seqno) ); DEFINE_EVENT(i915_gem_request, i915_gem_request_add,
@@ -600,6 +607,7 @@ TRACE_EVENT(i915_gem_request_wait_begin,TP_STRUCT__entry (__field (u32 ,dev )__field (u32, ring) +__field (u32, uniq)__field (u32, seqno)__field (bool, blocking) ),
@@ -615,13 +623,14 @@ TRACE_EVENT(i915_gem_request_wait_begin,i915_gem_request_get_ring (req); __entry->dev = ring->dev->primary->index; __entry->ring = ring->id; + __entry->uniq = req ? req->uniq :0 ; __entry->seqno =i915_gem_request_get_seqno (req); __entry->blocking =mutex_is_locked (&ring->dev->struct_mutex); ), -TP_printk ("dev=%u, ring=%u, seqno=%u, blocking=%s" , - __entry->dev, __entry->ring, +TP_printk ("dev=%u, ring=%u, uniq=%u, seqno=%u, blocking=%s" , + __entry->dev, __entry->ring, __entry->uniq, __entry->seqno, __entry->blocking ?"yes (NB)" :"no" ) );
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 7/39 drm/i915: Start of GPU scheduler
In reply to [email protected]From: John Harrison <[email protected]>
Initial creation of scheduler source files. Note that this patch implements most of the scheduler functionality but does not hook it in to the driver yet. It also leaves the scheduler code in 'pass through' mode so that even when it is hooked in, it will not actually do very much. This allows the hooks to be added one at a time in byte size chunks and only when the scheduler is finally enabled at the end does anything start happening.
The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code assigns a unique request and then packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.
The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.
As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, it is possible for the seqno values to be un-ordered (particularly once pre-emption is enabled). However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.
A deferred work queue is also poked by the interrupt handler. When this wakes up it can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.
When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.
If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate.
Note that this patch does not implement pre-emptive scheduling. Only basic scheduling by re-ordering batch buffer submission is currently implemented.
v2: Changed priority levels to +/-1023 due to feedback from Chris Wilson.
Removed redundant index from scheduler node.
Changed time stamps to use jiffies instead of raw monotonic. This provides lower resolution but improved compatibility with other i915 code.
Major re-write of completion tracking code due to struct fence conversion. The scheduler no longer has it's own private IRQ handler but just lets the existing request code handle completion events. Instead, the scheduler now hooks into the request notify code to be told when a request has completed.
Reduced driver mutex locking scope. Removal of scheduler nodes no longer grabs the mutex lock.
Change-Id: I1e08f59e650a3c2bbaaa9de7627da33849b06106
For: VIZ-1587
--- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/i915_drv.h | 4 + drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_scheduler.c | 740 ++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 90 +++++ 5 files changed, 840 insertions(+) create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
drivers/gpu/drm/i915/Makefile +1 -0
@@ -10,6 +10,7 @@ ccflags-y := -Werror i915-y := i915_drv.o \ i915_irq.o \ i915_params.o \ + i915_scheduler.o \ i915_suspend.o \ i915_sysfs.o \ intel_csr.o \
drivers/gpu/drm/i915/i915_drv.h +4 -0
@@ -1695,6 +1695,8 @@ struct i915_execbuffer_params {struct drm_i915_gem_request *request; }; +struct i915_scheduler ; +/* used in computing the new watermarks state */ struct intel_wm_config {unsigned int num_pipes_active;
@@ -1947,6 +1949,8 @@ struct drm_i915_private {struct i915_runtime_pm pm; +struct i915_scheduler *scheduler; +/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */ struct { int (*execbuf_submit)(struct i915_execbuffer_params *params,
drivers/gpu/drm/i915/i915_gem.c +5 -0
@@ -38,6 +38,7 @@#include <linux/pci.h> #include <linux/dma-buf.h> #include <../drivers/android/sync.h> +#include "i915_scheduler.h" #define RQ_BUG_ON (expr)
@@ -5277,6 +5278,10 @@ int i915_gem_init(struct drm_device *dev)*/ intel_uncore_forcewake_get (dev_priv,FORCEWAKE_ALL ); + ret =i915_scheduler_init (dev); +if (ret) + goto out_unlock; + ret =i915_gem_init_userptr (dev);if (ret) goto out_unlock;
drivers/gpu/drm/i915/i915_scheduler.c new file +740 -0
740 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h new file +90 -0
90 lines not on this page · raw message
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 8/40 drm/i915: Start of GPU scheduler
From: John Harrison <[email protected]>
Initial creation of scheduler source files. Note that this patch implements most of the scheduler functionality but does not hook it in to the driver yet. It also leaves the scheduler code in 'pass through' mode so that even when it is hooked in, it will not actually do very much. This allows the hooks to be added one at a time in byte size chunks and only when the scheduler is finally enabled at the end does anything start happening.
The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code assigns a unique request and then packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.
The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.
As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, it is possible for the seqno values to be un-ordered (particularly once pre-emption is enabled). However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.
A deferred work queue is also poked by the interrupt handler. When this wakes up it can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.
When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.
If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate.
Note that this patch does not implement pre-emptive scheduling. Only basic scheduling by re-ordering batch buffer submission is currently implemented.
v2: Changed priority levels to +/-1023 due to feedback from Chris Wilson.
Removed redundant index from scheduler node.
Changed time stamps to use jiffies instead of raw monotonic. This provides lower resolution but improved compatibility with other i915 code.
Major re-write of completion tracking code due to struct fence conversion. The scheduler no longer has it's own private IRQ handler but just lets the existing request code handle completion events. Instead, the scheduler now hooks into the request notify code to be told when a request has completed.
Reduced driver mutex locking scope. Removal of scheduler nodes no longer grabs the mutex lock.
v3: Refactor of dependency generation to make the code more readable. Also added in read-read optimisation support - i.e., don't treat a shared read-only buffer as being a dependency.
Allowed the killing of queued nodes rather than only flying ones.
Change-Id: I1e08f59e650a3c2bbaaa9de7627da33849b06106
For: VIZ-1587
--- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/i915_drv.h | 4 + drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_scheduler.c | 763 ++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 91 ++++ 5 files changed, 864 insertions(+) create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
drivers/gpu/drm/i915/Makefile +1 -0
@@ -10,6 +10,7 @@ ccflags-y := -Werror i915-y := i915_drv.o \ i915_irq.o \ i915_params.o \ + i915_scheduler.o \ i915_suspend.o \ i915_sysfs.o \ intel_csr.o \
drivers/gpu/drm/i915/i915_drv.h +4 -0
@@ -1695,6 +1695,8 @@ struct i915_execbuffer_params {struct drm_i915_gem_request *request; }; +struct i915_scheduler ; +/* used in computing the new watermarks state */ struct intel_wm_config {unsigned int num_pipes_active;
@@ -1947,6 +1949,8 @@ struct drm_i915_private {struct i915_runtime_pm pm; +struct i915_scheduler *scheduler; +/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */ struct { int (*execbuf_submit)(struct i915_execbuffer_params *params,
drivers/gpu/drm/i915/i915_gem.c +5 -0
@@ -38,6 +38,7 @@#include <linux/pci.h> #include <linux/dma-buf.h> #include <../drivers/android/sync.h> +#include "i915_scheduler.h" #define RQ_BUG_ON (expr)
@@ -5278,6 +5279,10 @@ int i915_gem_init(struct drm_device *dev)*/ intel_uncore_forcewake_get (dev_priv,FORCEWAKE_ALL ); + ret =i915_scheduler_init (dev); +if (ret) + goto out_unlock; + ret =i915_gem_init_userptr (dev);if (ret) goto out_unlock;
drivers/gpu/drm/i915/i915_scheduler.c new file +763 -0
763 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h new file +91 -0
91 lines not on this page · raw message
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 8/39 drm/i915: Prepare retire_requests to handle out-of-order seqnos
In reply to [email protected]From: John Harrison <[email protected]>
A major point of the GPU scheduler is that it re-orders batch buffers after they have been submitted to the driver. This leads to requests completing out of order. In turn, this means that the retire processing can no longer assume that all completed entries are at the front of the list. Rather than attempting to re-order the request list on a regular basis, it is better to simply scan the entire list.
v2: Removed deferred free code as no longer necessary due to request handling updates.
Change-Id: I7eb6793581d9d28eb832e0e94c116b7202fa1b26
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem.c | 31 +++++++++++++------------------ 1 file changed, 13 insertions(+), 18 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +13 -18
@@ -3218,6 +3218,7 @@ void i915_gem_reset(struct drm_device *dev)void i915_gem_retire_requests_ring (struct intel_engine_cs *ring) { +struct drm_i915_gem_object *obj, *obj_next;struct drm_i915_gem_request *req, *req_next;LIST_HEAD (list_head);unsigned long flags;
@@ -3231,37 +3232,31 @@ i915_gem_retire_requests_ring(struct intel_engine_cs *ring)*/ i915_gem_request_notify (ring, false); +/* +* Note that request entries might be out of order due to rescheduling +* and pre-emption. Thus both lists must be processed in their entirety +* rather than stopping at the first non-complete entry. +*/ +/* Retire requests first as we use it above for the early return. * If we retire requests last, we may use a later seqno and so clear * the requests lists without clearing the active list, leading to * confusion. */ -while (!list_empty (&ring->request_list)) { -struct drm_i915_gem_request *request; - - request =list_first_entry (&ring->request_list, - struct drm_i915_gem_request, - list); - -if (!i915_gem_request_completed (request)) -break ; +list_for_each_entry_safe (req, req_next, &ring->request_list, list) { +if (!i915_gem_request_completed (req)) +continue ; -i915_gem_request_retire (request); +i915_gem_request_retire (req); }/* Move any buffers on the active list that are no longer referenced * by the ringbuffer to the flushing/inactive lists as appropriate, * before we free the context associated with the requests. */ -while (!list_empty (&ring->active_list)) { -struct drm_i915_gem_object *obj; - - obj =list_first_entry (&ring->active_list, - struct drm_i915_gem_object, - ring_list[ring->id]); - +list_for_each_entry_safe (obj, obj_next, &ring->active_list, ring_list[ring->id]) {if (!list_empty (&obj->last_read_req[ring->id]->list)) -break ; +continue ;i915_gem_object_retire__read (obj, ring->id); }
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 9/39 drm/i915: Disable hardware semaphores when GPU scheduler is enabled
In reply to [email protected]From: John Harrison <[email protected]>
Hardware sempahores require seqno values to be continuously incrementing. However, the scheduler's reordering of batch buffers means that the seqno values going through the hardware could be out of order. Thus semaphores can not be used.
On the other hand, the scheduler superceeds the need for hardware semaphores anyway. Having one ring stall waiting for something to complete on another ring is inefficient if that ring could be working on some other, independent task. This is what the scheduler is meant to do - keep the hardware as busy as possible by reordering batch buffers to avoid dependency stalls.
Change-Id: I95d1fceacd370455a9720d7dca55cfd0a1f6beaa
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.c | 9 +++++++++ drivers/gpu/drm/i915/i915_scheduler.c | 7 +++++++ drivers/gpu/drm/i915/i915_scheduler.h | 1 + drivers/gpu/drm/i915/intel_ringbuffer.c | 4 ++++ 4 files changed, 21 insertions(+)
drivers/gpu/drm/i915/i915_drv.c +9 -0
@@ -34,6 +34,7 @@#include "i915_drv.h" #include "i915_trace.h" #include "intel_drv.h" +#include "i915_scheduler.h" #include <linux/console.h> #include <linux/module.h>
@@ -581,6 +582,14 @@ void intel_detect_pch(struct drm_device *dev)bool i915_semaphore_is_enabled (struct drm_device *dev) { +/* Hardware semaphores are not compatible with the scheduler due to the +* seqno values being potentially out of order. However, semaphores are +* also not required as the scheduler will handle interring dependencies +* and try do so in a way that does not cause dead time on the hardware. +*/ +if (i915_scheduler_is_enabled (dev)) +return false; +if (INTEL_INFO (dev)->gen <6 )return false;
drivers/gpu/drm/i915/i915_scheduler.c +7 -0
@@ -38,6 +38,13 @@ static int i915_scheduler_priority_bump(struct i915_scheduler *schedulerstruct i915_scheduler_queue_entry *target, uint32_t bump); +bool i915_scheduler_is_enabled (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; + +return dev_priv->scheduler !=NULL ; +} +int i915_scheduler_init (struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private;
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -83,6 +83,7 @@ enum { i915_sf_submitting = (1 << 1), }; +bool i915_scheduler_is_enabled (struct drm_device *dev);int i915_scheduler_init (struct drm_device *dev);int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe);bool i915_scheduler_notify_request (struct drm_i915_gem_request *req);
drivers/gpu/drm/i915/intel_ringbuffer.c +4 -0
@@ -32,6 +32,7 @@#include <drm/i915_drm.h> #include "i915_trace.h" #include "intel_drv.h" +#include "i915_scheduler.h" bool intel_ring_initialized (struct intel_engine_cs *ring)
@@ -1411,6 +1412,9 @@ gen6_ring_sync(struct drm_i915_gem_request *waiter_req,u32 wait_mbox = signaller->semaphore.mbox.wait[waiter->id];int ret; +/* Arithmetic on sequence numbers is unreliable with a scheduler. */ +BUG_ON (i915_scheduler_is_enabled (signaller->dev)); +/* Throughout all of the GEM code, seqno passed implies our current * seqno is >= the last seqno executed. However for hardware the * comparison is strictly greater than.
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 10/39 drm/i915: Force MMIO flips when scheduler enabled
In reply to [email protected]From: John Harrison <[email protected]>
Change-Id: Ice071af6d88306b0d1c53bdb651a1a3e20bdc1af
For: VIZ-1587
--- drivers/gpu/drm/i915/intel_display.c | 3 +++ 1 file changed, 3 insertions(+)
drivers/gpu/drm/i915/intel_display.c +3 -0
@@ -44,6 +44,7 @@#include <drm/drm_plane_helper.h> #include <drm/drm_rect.h> #include <linux/dma_remapping.h> +#include "i915_scheduler.h" /* Primary plane formats for gen <= 3 */ static const uint32_t i8xx_primary_formats[] = {
@@ -11135,6 +11136,8 @@ static bool use_mmio_flip(struct intel_engine_cs *ring,return true; else if (i915.enable_execlists) return true; + else if (i915_scheduler_is_enabled(ring->dev)) + return true; else return ring != i915_gem_request_get_ring(obj->last_write_req); }
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 11/39 drm/i915: Added scheduler hook when closing DRM file handles
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler decouples the submission of batch buffers to the driver with submission of batch buffers to the hardware. Thus it is possible for an application to submit work, then close the DRM handle and free up all the resources that piece of work wishes to use before the work has even been submitted to the hardware. To prevent this, the scheduler needs to be informed of the DRM close event so that it can force through any outstanding work attributed to that file handle.
Change-Id: I24ac056c062b075ff1cc5e2ed2d3fa8e17e85951
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_dma.c | 3 ++ drivers/gpu/drm/i915/i915_scheduler.c | 66 +++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 2 ++ 3 files changed, 71 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +3 -0
@@ -46,6 +46,7 @@#include <linux/vga_switcheroo.h> #include <linux/slab.h> #include <acpi/video.h> +#include "i915_scheduler.h" #include <linux/pm.h> #include <linux/pm_runtime.h> #include <linux/oom.h>
@@ -1250,6 +1251,8 @@ void i915_driver_lastclose(struct drm_device *dev)void i915_driver_preclose (struct drm_device *dev,struct drm_file *file) { +i915_scheduler_closefile (dev, file); +mutex_lock (&dev->struct_mutex);i915_gem_context_close (dev, file);i915_gem_release (dev, file);
drivers/gpu/drm/i915/i915_scheduler.c +66 -0
69 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h +2 -0
@@ -85,6 +85,8 @@ enum {bool i915_scheduler_is_enabled (struct drm_device *dev);int i915_scheduler_init (struct drm_device *dev); +int i915_scheduler_closefile (struct drm_device *dev, +struct drm_file *file);int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe);bool i915_scheduler_notify_request (struct drm_i915_gem_request *req);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 12/40 drm/i915: Added scheduler hook when closing DRM file handles
From: John Harrison <[email protected]>
The scheduler decouples the submission of batch buffers to the driver with submission of batch buffers to the hardware. Thus it is possible for an application to close its DRM file handle while there is still work outstanding. That means the scheduler needs to know about file close events so it can remove the file pointer from such orphaned batch buffers and not attempt to dereference it later.
v3: Updated to not wait for outstanding work to complete but merely remove the file handle reference. The wait was getting excessively complicated with inter-ring dependencies, pre-emption, and other such issues.
Change-Id: I24ac056c062b075ff1cc5e2ed2d3fa8e17e85951
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_dma.c | 3 +++ drivers/gpu/drm/i915/i915_scheduler.c | 35 +++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 2 ++ 3 files changed, 40 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +3 -0
@@ -46,6 +46,7 @@#include <linux/vga_switcheroo.h> #include <linux/slab.h> #include <acpi/video.h> +#include "i915_scheduler.h" #include <linux/pm.h> #include <linux/pm_runtime.h> #include <linux/oom.h>
@@ -1250,6 +1251,8 @@ void i915_driver_lastclose(struct drm_device *dev)void i915_driver_preclose (struct drm_device *dev,struct drm_file *file) { +i915_scheduler_closefile (dev, file); +mutex_lock (&dev->struct_mutex);i915_gem_context_close (dev, file);i915_gem_release (dev, file);
drivers/gpu/drm/i915/i915_scheduler.c +35 -0
@@ -768,3 +768,38 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,return 0 ; } + +int i915_scheduler_closefile (struct drm_device *dev,struct drm_file *file) +{ +struct i915_scheduler_queue_entry *node; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct intel_engine_cs *ring; +int i; +unsigned long flags; + +if (!scheduler) +return 0 ; + +spin_lock_irqsave (&scheduler->lock, flags); + +for_each_ring (ring, dev_priv, i) { +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (node->params.file != file) +continue ; + +if (!I915_SQS_IS_COMPLETE (node)) +DRM_DEBUG_DRIVER ("Closing file handle with outstanding work: %d:%d/%d on %s\n" , + node->params.request->uniq, + node->params.request->seqno, + node->status, + ring->name); + + node->params.file =NULL ; + } + } + +spin_unlock_irqrestore (&scheduler->lock, flags); + +return 0 ; +}
drivers/gpu/drm/i915/i915_scheduler.h +2 -0
@@ -86,6 +86,8 @@ enum {bool i915_scheduler_is_enabled (struct drm_device *dev);int i915_scheduler_init (struct drm_device *dev); +int i915_scheduler_closefile (struct drm_device *dev, +struct drm_file *file);int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe);bool i915_scheduler_notify_request (struct drm_i915_gem_request *req);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 12/39 drm/i915: Added scheduler hook into i915_gem_request_notify()
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler needs to know when requests have completed so that it can keep its own internal state up to date and can submit new requests to the hardware from its queue.
v2: Updated due to changes in request handling. The operation is now reversed from before. Rather than the scheduler being in control of completion events, it is now the request code itself. The scheduler merely receives a notification event. It can then optionally request it's worker thread be woken up after all completion processing is complete.
Change-Id: I149250a8f9382586514ca324aba1c53063b83e19
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 2 ++ drivers/gpu/drm/i915/i915_gem.c | 16 ++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.c | 28 ++++++++++++++++++---------- drivers/gpu/drm/i915/i915_scheduler.h | 1 + 4 files changed, 37 insertions(+), 10 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -0
@@ -2262,6 +2262,8 @@ struct drm_i915_gem_request {/** process identifier submitting this request */ struct pid *pid; +struct i915_scheduler_queue_entry *scheduler_qe; +/** * The ELSP only accepts two elements at a time, so we queue * context/tail pairs on a given queue (ring->execlist_queue) until the
drivers/gpu/drm/i915/i915_gem.c +16 -0
@@ -2785,6 +2785,7 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked) {struct drm_i915_gem_request *req, *req_next;unsigned long flags; +bool wake_sched = false;u32 seqno;if (list_empty (&ring->fence_signal_list)) {
@@ -2813,6 +2814,14 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)*/ list_del_init (&req->signal_link); +/* +* NB: Must notify the scheduler before signalling +* the node. Otherwise the node can get retired first +* and call scheduler_clean() while the scheduler +* thinks it is still active. +*/ + wake_sched |=i915_scheduler_notify_request (req); +if (!req->cancelled) {fence_signal_locked (&req->fence);trace_i915_gem_request_complete (req);
@@ -2829,6 +2838,13 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)if (!fence_locked)spin_unlock_irqrestore (&ring->fence_lock, flags); + +/* Necessary? Or does the fence_signal() call do an implicit wakeup? */ +wake_up_all (&ring->irq_queue); + +/* Final scheduler processing after all individual updates are done. */ +if (wake_sched) +i915_scheduler_wakeup (ring->dev); }static const char *i915_gem_request_get_driver_name (struct fence *req_fence)
drivers/gpu/drm/i915/i915_scheduler.c +18 -10
@@ -120,6 +120,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe) node->stamp = jiffies;i915_gem_request_reference (node->params.request); +BUG_ON (node->params.request->scheduler_qe); + node->params.request->scheduler_qe = node; +/* Need to determine the number of incomplete entries in the list as * that will be the maximum size of the dependency list. *
@@ -319,14 +322,16 @@ static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)* code has mapped it back to a request and will mark that request complete. * It also calls this function to notify the scheduler about the completion * so the scheduler's node can be updated appropriately. -* Returns true if the request is scheduler managed, false if not. +* Returns true if the request is scheduler managed, false if not. The return +* value is combined for all freshly completed requests and if any were true +* then i915_scheduler_wakeup() is called so the scheduler can do further +* processing (submit more work) at the end. */ bool i915_scheduler_notify_request (struct drm_i915_gem_request *req) {struct drm_i915_private *dev_priv =to_i915 (req->ring->dev);struct i915_scheduler *scheduler = dev_priv->scheduler; -/* XXX: Need to map back from request to node */ -struct i915_scheduler_queue_entry *node =NULL ; +struct i915_scheduler_queue_entry *node = req->scheduler_qe;unsigned long flags;if (!node)
@@ -344,16 +349,18 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)spin_unlock_irqrestore (&scheduler->lock, flags); -/* -* XXX: If the in-flight list is now empty then new work should be -* submitted. However, this function is called from interrupt context -* and thus cannot acquire mutex locks and other such things that are -* necessary for fresh submission. -*/ -return true; } +/* +* Called at the end of seqno interrupt processing if any request has +* completed that corresponds to a scheduler node. +*/ +void i915_scheduler_wakeup (struct drm_device *dev) +{ +/* XXX: Need to call i915_scheduler_remove() via work handler. */ +} +int i915_scheduler_remove (struct intel_engine_cs *ring) {struct drm_i915_private *dev_priv = ring->dev->dev_private;
@@ -468,6 +475,7 @@ int i915_scheduler_remove(struct intel_engine_cs *ring)i915_gem_execbuff_release_batch_obj (node->params.batch_obj);/* Free everything that is owned by the node: */ + node->params.request->scheduler_qe =NULL ;i915_gem_request_unreference (node->params.request);kfree (node->params.cliprects);kfree (node->dep_list);
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -89,5 +89,6 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file);int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe);bool i915_scheduler_notify_request (struct drm_i915_gem_request *req); +void i915_scheduler_wakeup (struct drm_device *dev);#endif /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 13/39 drm/i915: Added deferred work handler for scheduler
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler needs to do interrupt triggered work that is too complex to do in the interrupt handler. Thus it requires a deferred work handler to process such tasks asynchronously.
v2: Updated to reduce mutex lock usage. The lock is now only held for the minimum time within the remove function rather than for the whole of the worker thread's operation.
Change-Id: I0f7cc2b6f034a50bf8f7e368b60ad8bafd00f993
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_dma.c | 3 +++ drivers/gpu/drm/i915/i915_drv.h | 10 ++++++++++ drivers/gpu/drm/i915/i915_gem.c | 2 ++ drivers/gpu/drm/i915/i915_scheduler.c | 21 +++++++++++++++++++-- drivers/gpu/drm/i915/i915_scheduler.h | 1 + 5 files changed, 35 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_dma.c +3 -0
@@ -1149,6 +1149,9 @@ int i915_driver_unload(struct drm_device *dev)WARN_ON (unregister_oom_notifier (&dev_priv->mm.oom_notifier));unregister_shrinker (&dev_priv->mm.shrinker); +/* Cancel the scheduler work handler, which should be idle now. */ +cancel_work_sync (&dev_priv->mm.scheduler_work); +io_mapping_free (dev_priv->gtt.mappable);arch_phys_wc_del (dev_priv->gtt.mtrr);
drivers/gpu/drm/i915/i915_drv.h +10 -0
@@ -1285,6 +1285,16 @@ struct i915_gem_mm {struct delayed_work retire_work;/** +* New scheme is to get an interrupt after every work packet +* in order to allow the low latency scheduling of pending +* packets. The idea behind adding new packets to a pending +* queue rather than directly into the hardware ring buffer +* is to allow high priority packets to over take low priority +* ones. +*/ +struct work_struct scheduler_work; + +/** * When we detect an idle GPU, we want to turn on * powersaving features. So once we see that there * are no more requests outstanding and no more
drivers/gpu/drm/i915/i915_gem.c +2 -0
@@ -5389,6 +5389,8 @@ i915_gem_load(struct drm_device *dev) i915_gem_retire_work_handler);INIT_DELAYED_WORK (&dev_priv->mm.idle_work, i915_gem_idle_work_handler); +INIT_WORK (&dev_priv->mm.scheduler_work, + i915_gem_scheduler_work_handler);init_waitqueue_head (&dev_priv->gpu_error.reset_queue); dev_priv->relative_constants_mode =I915_EXEC_CONSTANTS_REL_GENERAL ;
drivers/gpu/drm/i915/i915_scheduler.c +19 -2
@@ -358,10 +358,12 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)*/ void i915_scheduler_wakeup (struct drm_device *dev) { -/* XXX: Need to call i915_scheduler_remove() via work handler. */ +struct drm_i915_private *dev_priv =to_i915 (dev); + +queue_work (dev_priv->wq, &dev_priv->mm.scheduler_work); } -int i915_scheduler_remove (struct intel_engine_cs *ring) +static int i915_scheduler_remove (struct intel_engine_cs *ring) {struct drm_i915_private *dev_priv = ring->dev->dev_private;struct i915_scheduler *scheduler = dev_priv->scheduler;
@@ -487,6 +489,21 @@ int i915_scheduler_remove(struct intel_engine_cs *ring)return ret; } +void i915_gem_scheduler_work_handler (struct work_struct *work) +{ +struct intel_engine_cs *ring; +struct drm_i915_private *dev_priv; +struct drm_device *dev; +int i; + + dev_priv = container_of(work, struct drm_i915_private, mm.scheduler_work); + dev = dev_priv->dev; + +for_each_ring (ring, dev_priv, i) { +i915_scheduler_remove (ring); + } +} +static void i915_scheduler_priority_bump_clear (struct i915_scheduler *scheduler) {struct i915_scheduler_queue_entry *node;
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -90,5 +90,6 @@ int i915_scheduler_closefile(struct drm_device *dev,int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe);bool i915_scheduler_notify_request (struct drm_i915_gem_request *req);void i915_scheduler_wakeup (struct drm_device *dev); +void i915_gem_scheduler_work_handler (struct work_struct *work);#endif /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 14/39 drm/i915: Redirect execbuffer_final() via scheduler
In reply to [email protected]From: John Harrison <[email protected]>
Updated the execbuffer() code to pass the packaged up batch buffer information to the scheduler rather than calling execbuffer_final() directly. The scheduler queue() code is currently a stub which simply chains on to _final() immediately.
Change-Id: I2a19062a9e66845f2e886332fc4b5fc7ac992864
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 18 +++++++----------- drivers/gpu/drm/i915/intel_lrc.c | 12 ++++-------- 2 files changed, 11 insertions(+), 19 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +7 -11
@@ -35,6 +35,7 @@#include <linux/dma_remapping.h> #include <linux/uaccess.h> #include <../drivers/android/sync.h> +#include "i915_scheduler.h" #define __EXEC_OBJECT_HAS_PIN (1<<31)#define __EXEC_OBJECT_HAS_FENCE (1<<30)
@@ -1180,6 +1181,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args, struct list_head *vmas) { + struct i915_scheduler_queue_entry *qe;struct drm_device *dev = params->dev;struct intel_engine_cs *ring = params->ring;struct drm_i915_private *dev_priv = dev->dev_private;
@@ -1224,17 +1226,11 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,i915_gem_execbuffer_move_to_active (vmas, params->request); - ret = dev_priv->gt.execbuf_final (params); + qe =container_of (params,typeof (*qe), params); + ret =i915_scheduler_queue_execbuffer (qe);if (ret)return ret; -/* -* Free everything that was stored in the QE structure (until the -* scheduler arrives and does it instead): -*/ -if (params->dispatch_flags &I915_DISPATCH_SECURE ) -i915_gem_execbuff_release_batch_obj (params->batch_obj); -return 0 ; }
@@ -1410,8 +1406,8 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,struct intel_engine_cs *ring;struct intel_context *ctx;struct i915_address_space *vm; -struct i915_execbuffer_params params_master;/* XXX: will be removed later */ -struct i915_execbuffer_params *params = ¶ms_master; +struct i915_scheduler_queue_entry qe; +struct i915_execbuffer_params *params = &qe.params;const u32 ctx_id =i915_execbuffer2_get_context_id (*args);u32 dispatch_flags;int ret;
@@ -1539,7 +1535,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,else vm = &dev_priv->gtt.base; -memset (¶ms_master,0x00 ,sizeof (params_master)); +memset (&qe,0x00 ,sizeof (qe)); eb =eb_create (args);if (eb ==NULL ) {
drivers/gpu/drm/i915/intel_lrc.c +4 -8
@@ -136,6 +136,7 @@#include <drm/i915_drm.h> #include "i915_drv.h" #include "intel_mocs.h" +#include "i915_scheduler.h" #define GEN9_LR_CONTEXT_RENDER_SIZE (22 * PAGE_SIZE)#define GEN8_LR_CONTEXT_RENDER_SIZE (20 * PAGE_SIZE)
@@ -866,6 +867,7 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args, struct list_head *vmas) { + struct i915_scheduler_queue_entry *qe;struct drm_device *dev = params->dev;struct intel_engine_cs *ring = params->ring;struct drm_i915_private *dev_priv = dev->dev_private;
@@ -908,17 +910,11 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,i915_gem_execbuffer_move_to_active (vmas, params->request); - ret = dev_priv->gt.execbuf_final (params); + qe =container_of (params,typeof (*qe), params); + ret =i915_scheduler_queue_execbuffer (qe);if (ret)return ret; -/* -* Free everything that was stored in the QE structure (until the -* scheduler arrives and does it instead): -*/ -if (params->dispatch_flags &I915_DISPATCH_SECURE ) -i915_gem_execbuff_release_batch_obj (params->batch_obj); -return 0 ; }
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 15/39 drm/i915: Keep the reserved space mechanism happy
In reply to [email protected]From: John Harrison <[email protected]>
Ring space is reserved when constructing a request to ensure that the subsequent 'add_request()' call cannot fail due to waiting for space on a busy or broken GPU. However, the scheduler jumps in to the middle of the execbuffer process between request creation and request submission. Thus it needs to cancel the reserved space when the request is simply added to the scheduler's queue and not yet submitted. Similarly, it needs to re-reserve the space when it finally does want to send the batch buffer to the hardware.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 7 +++++++ drivers/gpu/drm/i915/i915_scheduler.c | 4 ++++ drivers/gpu/drm/i915/intel_lrc.c | 13 +++++++++++-- 3 files changed, 22 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +7 -0
@@ -1248,6 +1248,10 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex)); + ret =intel_ring_reserve_space (params->request); +if (ret) +return ret; +intel_runtime_pm_get (dev_priv);/*
@@ -1308,6 +1312,9 @@ error:*/ intel_runtime_pm_put (dev_priv); +if (ret) +intel_ring_reserved_space_cancel (params->request->ringbuf); +return ret; }
drivers/gpu/drm/i915/i915_scheduler.c +4 -0
@@ -91,6 +91,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)if (1 /*i915.scheduler_override & i915_so_direct_submit*/ ) {int ret; +intel_ring_reserved_space_cancel (qe->params.request->ringbuf); + scheduler->flags[qe->params.ring->id] |= i915_sf_submitting; ret = dev_priv->gt.execbuf_final (&qe->params); scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
@@ -120,6 +122,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe) node->stamp = jiffies;i915_gem_request_reference (node->params.request); +intel_ring_reserved_space_cancel (node->params.request->ringbuf); +BUG_ON (node->params.request->scheduler_qe); node->params.request->scheduler_qe = node;
drivers/gpu/drm/i915/intel_lrc.c +11 -2
@@ -932,13 +932,17 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex)); + ret =intel_logical_ring_reserve_space (params->request); +if (ret) +return ret; +/* * Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch. */ ret =logical_ring_invalidate_all_caches (params->request);if (ret) -return ret; + goto err;if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) {
@@ -962,13 +966,18 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params) ret = ring->emit_bb_start (params->request, exec_start, params->dispatch_flags);if (ret) -return ret; + goto err;trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags);i915_gem_execbuffer_retire_commands (params);return 0 ; + +err: +intel_ring_reserved_space_cancel (params->request->ringbuf); + +return ret; }void intel_execlists_retire_requests (struct intel_engine_cs *ring)
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 16/40 drm/i915: Keep the reserved space mechanism happy
From: John Harrison <[email protected]>
Ring space is reserved when constructing a request to ensure that the subsequent 'add_request()' call cannot fail due to waiting for space on a busy or broken GPU. However, the scheduler jumps in to the middle of the execbuffer process between request creation and request submission. Thus it needs to cancel the reserved space when the request is simply added to the scheduler's queue and not yet submitted. Similarly, it needs to re-reserve the space when it finally does want to send the batch buffer to the hardware.
v3: Updated to use locally cached request pointer.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 7 +++++++ drivers/gpu/drm/i915/i915_scheduler.c | 4 ++++ drivers/gpu/drm/i915/intel_lrc.c | 13 +++++++++++-- 3 files changed, 22 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +7 -0
@@ -1249,6 +1249,10 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex)); + ret =intel_ring_reserve_space (req); +if (ret) +return ret; +intel_runtime_pm_get (dev_priv);/*
@@ -1309,6 +1313,9 @@ error:*/ intel_runtime_pm_put (dev_priv); +if (ret) +intel_ring_reserved_space_cancel (req->ringbuf); +return ret; }
drivers/gpu/drm/i915/i915_scheduler.c +4 -0
@@ -145,6 +145,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)if (1 /*i915.scheduler_override & i915_so_direct_submit*/ ) {int ret; +intel_ring_reserved_space_cancel (qe->params.request->ringbuf); + scheduler->flags[qe->params.ring->id] |= i915_sf_submitting; ret = dev_priv->gt.execbuf_final (&qe->params); scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting;
@@ -174,6 +176,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe) node->stamp = jiffies;i915_gem_request_reference (node->params.request); +intel_ring_reserved_space_cancel (node->params.request->ringbuf); +BUG_ON (node->params.request->scheduler_qe); node->params.request->scheduler_qe = node;
drivers/gpu/drm/i915/intel_lrc.c +11 -2
@@ -934,13 +934,17 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex)); + ret =intel_logical_ring_reserve_space (req); +if (ret) +return ret; +/* * Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch. */ ret =logical_ring_invalidate_all_caches (req);if (ret) -return ret; + goto err;if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) {
@@ -962,13 +966,18 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params) ret = ring->emit_bb_start (req, exec_start, params->dispatch_flags);if (ret) -return ret; + goto err;trace_i915_gem_ring_dispatch (req, params->dispatch_flags);i915_gem_execbuffer_retire_commands (params);return 0 ; + +err: +intel_ring_reserved_space_cancel (params->request->ringbuf); + +return ret; }void intel_execlists_retire_requests (struct intel_engine_cs *ring)
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 16/39 drm/i915: Added tracking/locking of batch buffer objects
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler needs to track interdependencies between batch buffers. These are calculated by analysing the object lists of the buffers and looking for commonality. The scheduler also needs to keep those buffers locked long after the initial IOCTL call has returned to user land.
Change-Id: I31e3677ecfc2c9b5a908bda6acc4850432d55f1e
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 48 ++++++++++++++++++++++++++++-- drivers/gpu/drm/i915/i915_scheduler.c | 33 ++++++++++++++++++-- 2 files changed, 76 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +46 -2
@@ -1417,7 +1417,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,struct i915_execbuffer_params *params = &qe.params;const u32 ctx_id =i915_execbuffer2_get_context_id (*args);u32 dispatch_flags; -int ret; +int ret, i;bool need_relocs;int fd_fence_complete =-1 ;int fd_fence_wait =lower_32_bits (args->rsvd2);
@@ -1552,6 +1552,14 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, goto pre_mutex_err; } + qe.saved_objects =kzalloc ( +sizeof (*qe.saved_objects) * args->buffer_count, +GFP_KERNEL ); +if (!qe.saved_objects) { + ret = -ENOMEM ; + goto err; + } +/* Look up object handles */ ret =eb_lookup_vmas (eb, exec, args, vm, file);if (ret)
@@ -1672,7 +1680,26 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, params->args_DR1 = args->DR1; params->args_DR4 = args->DR4; params->batch_obj = batch_obj; - params->ctx = ctx; + +/* +* Save away the list of objects used by this batch buffer for the +* purpose of tracking inter-buffer dependencies. +*/ +for (i =0 ; i < args->buffer_count; i++) { +/* +* NB: 'drm_gem_object_lookup()' increments the object's +* reference count and so must be matched by a +* 'drm_gem_object_unreference' call. +*/ + qe.saved_objects[i].obj = +to_intel_bo (drm_gem_object_lookup (dev, file, + exec[i].handle)); + } + qe.num_objs = i; + +/* Lock and save the context object as well. */ +i915_gem_context_reference (ctx); + params->ctx = ctx;if (args->flags &I915_EXEC_CREATE_FENCE ) {/*
@@ -1737,6 +1764,23 @@ err:i915_gem_context_unreference (ctx);eb_destroy (eb); +if (qe.saved_objects) { +/* Need to release the objects: */ +for (i =0 ; i < qe.num_objs; i++) { +if (!qe.saved_objects[i].obj) +continue ; + +drm_gem_object_unreference ( + &qe.saved_objects[i].obj->base); + } + +kfree (qe.saved_objects); + +/* Context too */ +if (params->ctx) +i915_gem_context_unreference (params->ctx); + } +/* * If the request was created but not successfully submitted then it * must be freed again. If it was submitted then it is being tracked
drivers/gpu/drm/i915/i915_scheduler.c +30 -3
@@ -104,7 +104,23 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)if (ret)return ret; -/* Free everything that is owned by the QE structure: */ +/* Need to release the objects: */ +for (i =0 ; i < qe->num_objs; i++) { +if (!qe->saved_objects[i].obj) +continue ; + +drm_gem_object_unreference (&qe->saved_objects[i].obj->base); + } + +kfree (qe->saved_objects); + qe->saved_objects =NULL ; + qe->num_objs =0 ; + +/* Free the context object too: */ +if (qe->params.ctx) +i915_gem_context_unreference (qe->params.ctx); + +/* And anything else owned by the QE structure: */ kfree (qe->params.cliprects);if (qe->params.dispatch_flags &I915_DISPATCH_SECURE )i915_gem_execbuff_release_batch_obj (qe->params.batch_obj);
@@ -376,7 +392,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)int flying =0 , queued =0 ;int ret =0 ;bool do_submit; -uint32_t min_seqno; +uint32_t i, min_seqno;struct list_head remove;if (list_empty (&scheduler->node_queue[ring->id]))
@@ -480,7 +496,18 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)if (node->params.dispatch_flags &I915_DISPATCH_SECURE )i915_gem_execbuff_release_batch_obj (node->params.batch_obj); -/* Free everything that is owned by the node: */ +/* Release the locked buffers: */ +for (i =0 ; i < node->num_objs; i++) { +drm_gem_object_unreference ( + &node->saved_objects[i].obj->base); + } +kfree (node->saved_objects); + +/* Context too: */ +if (node->params.ctx) +i915_gem_context_unreference (node->params.ctx); + +/* And anything else owned by the node: */ node->params.request->scheduler_qe =NULL ;i915_gem_request_unreference (node->params.request);kfree (node->params.cliprects);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 17/40 drm/i915: Added tracking/locking of batch buffer objects
From: John Harrison <[email protected]>
The scheduler needs to track interdependencies between batch buffers. These are calculated by analysing the object lists of the buffers and looking for commonality. The scheduler also needs to keep those buffers locked long after the initial IOCTL call has returned to user land.
v3: Updated to support read-read optimisation.
Change-Id: I31e3677ecfc2c9b5a908bda6acc4850432d55f1e
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 52 ++++++++++++++++++++++++++++-- drivers/gpu/drm/i915/i915_scheduler.c | 33 +++++++++++++++++-- 2 files changed, 80 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +50 -2
@@ -1418,7 +1418,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,struct i915_execbuffer_params *params = &qe.params;const u32 ctx_id =i915_execbuffer2_get_context_id (*args);u32 dispatch_flags; -int ret; +int ret, i;bool need_relocs;int fd_fence_complete =-1 ;int fd_fence_wait =lower_32_bits (args->rsvd2);
@@ -1553,6 +1553,14 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, goto pre_mutex_err; } + qe.saved_objects =kzalloc ( +sizeof (*qe.saved_objects) * args->buffer_count, +GFP_KERNEL ); +if (!qe.saved_objects) { + ret = -ENOMEM ; + goto err; + } +/* Look up object handles */ ret =eb_lookup_vmas (eb, exec, args, vm, file);if (ret)
@@ -1673,7 +1681,30 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, params->args_DR1 = args->DR1; params->args_DR4 = args->DR4; params->batch_obj = batch_obj; - params->ctx = ctx; + +/* +* Save away the list of objects used by this batch buffer for the +* purpose of tracking inter-buffer dependencies. +*/ +for (i =0 ; i < args->buffer_count; i++) { +struct drm_i915_gem_object *obj; + +/* +* NB: 'drm_gem_object_lookup()' increments the object's +* reference count and so must be matched by a +* 'drm_gem_object_unreference' call. +*/ + obj =to_intel_bo (drm_gem_object_lookup (dev, file, + exec[i].handle)); + qe.saved_objects[i].obj = obj; + qe.saved_objects[i].read_only = obj->base.pending_write_domain ==0 ; + + } + qe.num_objs = i; + +/* Lock and save the context object as well. */ +i915_gem_context_reference (ctx); + params->ctx = ctx;if (args->flags &I915_EXEC_CREATE_FENCE ) {/*
@@ -1738,6 +1769,23 @@ err:i915_gem_context_unreference (ctx);eb_destroy (eb); +if (qe.saved_objects) { +/* Need to release the objects: */ +for (i =0 ; i < qe.num_objs; i++) { +if (!qe.saved_objects[i].obj) +continue ; + +drm_gem_object_unreference ( + &qe.saved_objects[i].obj->base); + } + +kfree (qe.saved_objects); + +/* Context too */ +if (params->ctx) +i915_gem_context_unreference (params->ctx); + } +/* * If the request was created but not successfully submitted then it * must be freed again. If it was submitted then it is being tracked
drivers/gpu/drm/i915/i915_scheduler.c +30 -3
@@ -158,7 +158,23 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)if (ret)return ret; -/* Free everything that is owned by the QE structure: */ +/* Need to release the objects: */ +for (i =0 ; i < qe->num_objs; i++) { +if (!qe->saved_objects[i].obj) +continue ; + +drm_gem_object_unreference (&qe->saved_objects[i].obj->base); + } + +kfree (qe->saved_objects); + qe->saved_objects =NULL ; + qe->num_objs =0 ; + +/* Free the context object too: */ +if (qe->params.ctx) +i915_gem_context_unreference (qe->params.ctx); + +/* And anything else owned by the QE structure: */ kfree (qe->params.cliprects);if (qe->params.dispatch_flags &I915_DISPATCH_SECURE )i915_gem_execbuff_release_batch_obj (qe->params.batch_obj);
@@ -399,7 +415,7 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)int flying =0 , queued =0 ;int ret =0 ;bool do_submit; -uint32_t min_seqno; +uint32_t i, min_seqno;struct list_head remove;if (list_empty (&scheduler->node_queue[ring->id]))
@@ -503,7 +519,18 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)if (node->params.dispatch_flags &I915_DISPATCH_SECURE )i915_gem_execbuff_release_batch_obj (node->params.batch_obj); -/* Free everything that is owned by the node: */ +/* Release the locked buffers: */ +for (i =0 ; i < node->num_objs; i++) { +drm_gem_object_unreference ( + &node->saved_objects[i].obj->base); + } +kfree (node->saved_objects); + +/* Context too: */ +if (node->params.ctx) +i915_gem_context_unreference (node->params.ctx); + +/* And anything else owned by the node: */ node->params.request->scheduler_qe =NULL ;i915_gem_request_unreference (node->params.request);kfree (node->params.cliprects);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 17/39 drm/i915: Hook scheduler node clean up into retire requests
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler keeps its own lock on various DRM objects in order to guarantee safe access long after the original execbuff IOCTL has completed. This is especially important when pre-emption is enabled as the batch buffer might need to be submitted to the hardware multiple times. This patch hooks the clean up of these locks into the request retire function. The request can only be retired after it has completed on the hardware and thus is no longer eligible for re-submission. Thus there is no point holding on to the locks beyond that time.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem.c | 3 +++ drivers/gpu/drm/i915/i915_scheduler.c | 51 ++++++++++++++++++++++++----------- drivers/gpu/drm/i915/i915_scheduler.h | 1 + 3 files changed, 39 insertions(+), 16 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +3 -0
@@ -1402,6 +1402,9 @@ static void i915_gem_request_retire(struct drm_i915_gem_request *request)fence_signal_locked (&request->fence); } +if (request->scheduler_qe) +i915_gem_scheduler_clean_node (request->scheduler_qe); +i915_gem_request_unreference (request); }
drivers/gpu/drm/i915/i915_scheduler.c +35 -16
69 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -87,6 +87,7 @@ bool i915_scheduler_is_enabled(struct drm_device *dev);int i915_scheduler_init (struct drm_device *dev);int i915_scheduler_closefile (struct drm_device *dev,struct drm_file *file); +void i915_gem_scheduler_clean_node (struct i915_scheduler_queue_entry *node);int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe);bool i915_scheduler_notify_request (struct drm_i915_gem_request *req);void i915_scheduler_wakeup (struct drm_device *dev);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 18/40 drm/i915: Hook scheduler node clean up into retire requests
From: John Harrison <[email protected]>
The scheduler keeps its own lock on various DRM objects in order to guarantee safe access long after the original execbuff IOCTL has completed. This is especially important when pre-emption is enabled as the batch buffer might need to be submitted to the hardware multiple times. This patch hooks the clean up of these locks into the request retire function. The request can only be retired after it has completed on the hardware and thus is no longer eligible for re-submission. Thus there is no point holding on to the locks beyond that time.
v3: Updated to not WARN when cleaning a node that is being cancelled. The clean will happen later so skipping it at the point of cancellation is fine.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem.c | 3 ++ drivers/gpu/drm/i915/i915_scheduler.c | 54 ++++++++++++++++++++++++----------- drivers/gpu/drm/i915/i915_scheduler.h | 1 + 3 files changed, 42 insertions(+), 16 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +3 -0
@@ -1402,6 +1402,9 @@ static void i915_gem_request_retire(struct drm_i915_gem_request *request)fence_signal_locked (&request->fence); } +if (request->scheduler_qe) +i915_gem_scheduler_clean_node (request->scheduler_qe); +i915_gem_request_unreference (request); }
drivers/gpu/drm/i915/i915_scheduler.c +38 -16
72 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -88,6 +88,7 @@ bool i915_scheduler_is_enabled(struct drm_device *dev);int i915_scheduler_init (struct drm_device *dev);int i915_scheduler_closefile (struct drm_device *dev,struct drm_file *file); +void i915_gem_scheduler_clean_node (struct i915_scheduler_queue_entry *node);int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe);bool i915_scheduler_notify_request (struct drm_i915_gem_request *req);void i915_scheduler_wakeup (struct drm_device *dev);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 18/39 drm/i915: Added scheduler support to __wait_request() calls
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler can cause batch buffers, and hence requests, to be submitted to the ring out of order and asynchronously to their submission to the driver. Thus at the point of waiting for the completion of a given request, it is not even guaranteed that the request has actually been sent to the hardware yet. Even it is has been sent, it is possible that it could be pre-empted and thus 'unsent'.
This means that it is necessary to be able to submit requests to the hardware during the wait call itself. Unfortunately, while some callers of __wait_request() release the mutex lock first, others do not (and apparently can not). Hence there is the ability to deadlock as the wait stalls for submission but the asynchronous submission is stalled for the mutex lock.
This change hooks the scheduler in to the __wait_request() code to ensure correct behaviour. That is, flush the target batch buffer through to the hardware and do not deadlock waiting for something that cannot currently be submitted. Instead, the wait call must return EAGAIN at least as far back as necessary to release the mutex lock and allow the scheduler's asynchronous processing to get in and handle the pre-emption operation and eventually (re-)submit the work.
Change-Id: I31fe6bc7e38f6ffdd843fcae16e7cc8b1e52a931
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 3 +- drivers/gpu/drm/i915/i915_gem.c | 37 +++++++++-- drivers/gpu/drm/i915/i915_scheduler.c | 111 ++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 4 ++ drivers/gpu/drm/i915/intel_display.c | 5 +- drivers/gpu/drm/i915/intel_ringbuffer.c | 2 +- 6 files changed, 151 insertions(+), 11 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -1
@@ -3029,7 +3029,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,unsigned reset_counter, bool interruptible, s64 *timeout, - struct intel_rps_client *rps); + struct intel_rps_client *rps, + bool is_locked);int __must_check i915_wait_request (struct drm_i915_gem_request *req);int i915_gem_fault (struct vm_area_struct *vma,struct vm_fault *vmf);int __must_check
drivers/gpu/drm/i915/i915_gem.c +30 -7
@@ -1207,7 +1207,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,unsigned reset_counter, bool interruptible, s64 *timeout, - struct intel_rps_client *rps) + struct intel_rps_client *rps, + bool is_locked) { struct intel_engine_cs *ring = i915_gem_request_get_ring(req);struct drm_device *dev = ring->dev;
@@ -1217,8 +1218,10 @@ int __i915_wait_request(struct drm_i915_gem_request *req,DEFINE_WAIT (wait);unsigned long timeout_expire;s64 before, now; -int ret; +int ret =0 ; +bool busy; +might_sleep ();WARN (!intel_irqs_enabled (dev_priv),"IRQs disabled" );if (i915_gem_request_completed (req))
@@ -1269,6 +1272,22 @@ int __i915_wait_request(struct drm_i915_gem_request *req,break ; } +if (is_locked) { +/* If this request is being processed by the scheduler +* then it is unsafe to sleep with the mutex lock held +* as the scheduler may require the lock in order to +* progress the request. */ +if (i915_scheduler_is_request_tracked (req,NULL , &busy)) { +if (busy) { + ret = -EAGAIN ; +break ; + } + } + +/* If the request is not tracked by the scheduler then the +* regular test can be done. */ + } +if (i915_gem_request_completed (req)) { ret =0 ;break ;
@@ -1449,13 +1468,17 @@ i915_wait_request(struct drm_i915_gem_request *req)BUG_ON (!mutex_is_locked (&dev->struct_mutex)); + ret =i915_scheduler_flush_request (req, true); +if (ret <0 ) +return ret; + ret =i915_gem_check_wedge (&dev_priv->gpu_error, interruptible);if (ret)return ret; ret =__i915_wait_request (req,atomic_read (&dev_priv->gpu_error.reset_counter), - interruptible,NULL ,NULL ); + interruptible,NULL ,NULL , true);if (ret)return ret;
@@ -1568,7 +1591,7 @@ i915_gem_object_wait_rendering__nonblocking(struct drm_i915_gem_object *obj,mutex_unlock (&dev->struct_mutex);for (i =0 ; ret ==0 && i < n; i++) ret =__i915_wait_request (requests[i], reset_counter, true, -NULL , rps); +NULL , rps, false);mutex_lock (&dev->struct_mutex);for (i =0 ; i < n; i++) {
@@ -3493,7 +3516,7 @@ i915_gem_wait_ioctl(struct drm_device *dev, void *data, struct drm_file *file)if (ret ==0 ) ret =__i915_wait_request (req[i], reset_counter, true, args->timeout_ns >0 ? &args->timeout_ns :NULL , - file->driver_priv); + file->driver_priv, false);i915_gem_request_unreference (req[i]); }return ret;
@@ -3526,7 +3549,7 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,atomic_read (&i915->gpu_error.reset_counter), i915->mm.interruptible,NULL , - &i915->rps.semaphores); + &i915->rps.semaphores, true);if (ret)return ret;
@@ -4485,7 +4508,7 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)if (target ==NULL )return 0 ; - ret =__i915_wait_request (target, reset_counter, true,NULL ,NULL ); + ret =__i915_wait_request (target, reset_counter, true,NULL ,NULL , false);if (ret ==0 )queue_delayed_work (dev_priv->wq, &dev_priv->mm.retire_work,0 );
drivers/gpu/drm/i915/i915_scheduler.c +111 -0
135 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h +4 -0
@@ -92,5 +92,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qbool i915_scheduler_notify_request (struct drm_i915_gem_request *req);void i915_scheduler_wakeup (struct drm_device *dev);void i915_gem_scheduler_work_handler (struct work_struct *work); +int i915_scheduler_flush_request (struct drm_i915_gem_request *req, +bool is_locked); +bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req, +bool *completed,bool *busy);#endif /* _I915_SCHEDULER_H_ */
drivers/gpu/drm/i915/intel_display.c +3 -2
@@ -11258,7 +11258,8 @@ static void intel_mmio_flip_work_func(struct work_struct *work)WARN_ON (__i915_wait_request (mmio_flip->req, mmio_flip->crtc->reset_counter, false,NULL , - &mmio_flip->i915->rps.mmioflips)); + &mmio_flip->i915->rps.mmioflips, + false));i915_gem_request_unreference (mmio_flip->req); }
@@ -13258,7 +13259,7 @@ static int intel_atomic_prepare_commit(struct drm_device *dev, ret =__i915_wait_request (intel_plane_state->wait_req, reset_counter, true, -NULL ,NULL ); +NULL ,NULL , false);/* Swallow -EIO errors to allow updates during hw lockup. */ if (ret == -EIO )
drivers/gpu/drm/i915/intel_ringbuffer.c +1 -1
@@ -2294,7 +2294,7 @@ int intel_ring_idle(struct intel_engine_cs *ring)return __i915_wait_request (req,atomic_read (&to_i915 (ring->dev)->gpu_error.reset_counter),to_i915 (ring->dev)->mm.interruptible, -NULL ,NULL ); +NULL ,NULL , true); }int intel_ring_alloc_request_extras (struct drm_i915_gem_request *request)
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 19/40 drm/i915: Added scheduler support to __wait_request() calls
From: John Harrison <[email protected]>
The scheduler can cause batch buffers, and hence requests, to be submitted to the ring out of order and asynchronously to their submission to the driver. Thus at the point of waiting for the completion of a given request, it is not even guaranteed that the request has actually been sent to the hardware yet. Even it is has been sent, it is possible that it could be pre-empted and thus 'unsent'.
This means that it is necessary to be able to submit requests to the hardware during the wait call itself. Unfortunately, while some callers of __wait_request() release the mutex lock first, others do not (and apparently can not). Hence there is the ability to deadlock as the wait stalls for submission but the asynchronous submission is stalled for the mutex lock.
This change hooks the scheduler in to the __wait_request() code to ensure correct behaviour. That is, flush the target batch buffer through to the hardware and do not deadlock waiting for something that cannot currently be submitted. Instead, the wait call must return EAGAIN at least as far back as necessary to release the mutex lock and allow the scheduler's asynchronous processing to get in and handle the pre-emption operation and eventually (re-)submit the work.
v3: Removed the explicit scheduler flush from i915_wait_request(). This is no longer necessary and was causing unintended changes to the scheduler priority level which broke a validation team test.
Change-Id: I31fe6bc7e38f6ffdd843fcae16e7cc8b1e52a931
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 3 ++- drivers/gpu/drm/i915/i915_gem.c | 33 ++++++++++++++++++++++++++------- drivers/gpu/drm/i915/i915_scheduler.c | 20 ++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 2 ++ drivers/gpu/drm/i915/intel_display.c | 5 +++-- drivers/gpu/drm/i915/intel_ringbuffer.c | 2 +- 6 files changed, 54 insertions(+), 11 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +2 -1
@@ -3029,7 +3029,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,unsigned reset_counter, bool interruptible, s64 *timeout, - struct intel_rps_client *rps); + struct intel_rps_client *rps, + bool is_locked);int __must_check i915_wait_request (struct drm_i915_gem_request *req);int i915_gem_fault (struct vm_area_struct *vma,struct vm_fault *vmf);int __must_check
drivers/gpu/drm/i915/i915_gem.c +26 -7
@@ -1207,7 +1207,8 @@ int __i915_wait_request(struct drm_i915_gem_request *req,unsigned reset_counter, bool interruptible, s64 *timeout, - struct intel_rps_client *rps) + struct intel_rps_client *rps, + bool is_locked) { struct intel_engine_cs *ring = i915_gem_request_get_ring(req);struct drm_device *dev = ring->dev;
@@ -1217,8 +1218,10 @@ int __i915_wait_request(struct drm_i915_gem_request *req,DEFINE_WAIT (wait);unsigned long timeout_expire;s64 before, now; -int ret; +int ret =0 ; +bool busy; +might_sleep ();WARN (!intel_irqs_enabled (dev_priv),"IRQs disabled" );if (i915_gem_request_completed (req))
@@ -1269,6 +1272,22 @@ int __i915_wait_request(struct drm_i915_gem_request *req,break ; } +if (is_locked) { +/* If this request is being processed by the scheduler +* then it is unsafe to sleep with the mutex lock held +* as the scheduler may require the lock in order to +* progress the request. */ +if (i915_scheduler_is_request_tracked (req,NULL , &busy)) { +if (busy) { + ret = -EAGAIN ; +break ; + } + } + +/* If the request is not tracked by the scheduler then the +* regular test can be done. */ + } +if (i915_gem_request_completed (req)) { ret =0 ;break ;
@@ -1455,7 +1474,7 @@ i915_wait_request(struct drm_i915_gem_request *req) ret =__i915_wait_request (req,atomic_read (&dev_priv->gpu_error.reset_counter), - interruptible,NULL ,NULL ); + interruptible,NULL ,NULL , true);if (ret)return ret;
@@ -1568,7 +1587,7 @@ i915_gem_object_wait_rendering__nonblocking(struct drm_i915_gem_object *obj,mutex_unlock (&dev->struct_mutex);for (i =0 ; ret ==0 && i < n; i++) ret =__i915_wait_request (requests[i], reset_counter, true, -NULL , rps); +NULL , rps, false);mutex_lock (&dev->struct_mutex);for (i =0 ; i < n; i++) {
@@ -3494,7 +3513,7 @@ i915_gem_wait_ioctl(struct drm_device *dev, void *data, struct drm_file *file)if (ret ==0 ) ret =__i915_wait_request (req[i], reset_counter, true, args->timeout_ns >0 ? &args->timeout_ns :NULL , - file->driver_priv); + file->driver_priv, false);i915_gem_request_unreference (req[i]); }return ret;
@@ -3527,7 +3546,7 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,atomic_read (&i915->gpu_error.reset_counter), i915->mm.interruptible,NULL , - &i915->rps.semaphores); + &i915->rps.semaphores, true);if (ret)return ret;
@@ -4486,7 +4505,7 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)if (target ==NULL )return 0 ; - ret =__i915_wait_request (target, reset_counter, true,NULL ,NULL ); + ret =__i915_wait_request (target, reset_counter, true,NULL ,NULL , false);if (ret ==0 )queue_delayed_work (dev_priv->wq, &dev_priv->mm.retire_work,0 );
drivers/gpu/drm/i915/i915_scheduler.c +20 -0
@@ -847,6 +847,26 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,return 0 ; } +bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req, +bool *completed,bool *busy) +{ +struct drm_i915_private *dev_priv = req->ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + +if (!scheduler) +return false; + +if (req->scheduler_qe ==NULL ) +return false; + +if (completed) + *completed =I915_SQS_IS_COMPLETE (req->scheduler_qe); +if (busy) + *busy =I915_SQS_IS_QUEUED (req->scheduler_qe); + +return true; +} +int i915_scheduler_closefile (struct drm_device *dev,struct drm_file *file) {struct i915_scheduler_queue_entry *node;
drivers/gpu/drm/i915/i915_scheduler.h +2 -0
@@ -93,5 +93,7 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qbool i915_scheduler_notify_request (struct drm_i915_gem_request *req);void i915_scheduler_wakeup (struct drm_device *dev);void i915_gem_scheduler_work_handler (struct work_struct *work); +bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req, +bool *completed,bool *busy);#endif /* _I915_SCHEDULER_H_ */
drivers/gpu/drm/i915/intel_display.c +3 -2
@@ -11258,7 +11258,8 @@ static void intel_mmio_flip_work_func(struct work_struct *work)WARN_ON (__i915_wait_request (mmio_flip->req, mmio_flip->crtc->reset_counter, false,NULL , - &mmio_flip->i915->rps.mmioflips)); + &mmio_flip->i915->rps.mmioflips, + false));i915_gem_request_unreference (mmio_flip->req); }
@@ -13258,7 +13259,7 @@ static int intel_atomic_prepare_commit(struct drm_device *dev, ret =__i915_wait_request (intel_plane_state->wait_req, reset_counter, true, -NULL ,NULL ); +NULL ,NULL , false);/* Swallow -EIO errors to allow updates during hw lockup. */ if (ret == -EIO )
drivers/gpu/drm/i915/intel_ringbuffer.c +1 -1
@@ -2294,7 +2294,7 @@ int intel_ring_idle(struct intel_engine_cs *ring)return __i915_wait_request (req,atomic_read (&to_i915 (ring->dev)->gpu_error.reset_counter),to_i915 (ring->dev)->mm.interruptible, -NULL ,NULL ); +NULL ,NULL , true); }int intel_ring_alloc_request_extras (struct drm_i915_gem_request *request)
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 19/39 drm/i915: Added scheduler support to page fault handler
In reply to [email protected]From: John Harrison <[email protected]>
GPU page faults can now require scheduler operation in order to complete. For example, in order to free up sufficient memory to handle the fault the handler must wait for a batch buffer to complete that has not even been sent to the hardware yet. Thus EAGAIN no longer means a GPU hang, it can occur under normal operation.
Change-Id: Iff6bd2744ef12bb7405fbcd6b43c286caad4141f
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +7 -2
@@ -1916,10 +1916,15 @@ out: }case -EAGAIN :/* -* EAGAIN means the gpu is hung and we'll wait for the error -* handler to reset everything when re-faulting in +* EAGAIN can mean the gpu is hung and we'll have to wait for +* the error handler to reset everything when re-faulting in * i915_mutex_lock_interruptible. +* +* It can also indicate various other nonfatal errors for which +* the best response is to give other threads a chance to run, +* and then retry the failing operation in its entirety. */ +/*FALLTHRU*/ case 0 :case -ERESTARTSYS :case -EINTR :
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 20/39 drm/i915: Added scheduler flush calls to ring throttle and idle functions
In reply to [email protected]From: John Harrison <[email protected]>
When requesting that all GPU work is completed, it is now necessary to get the scheduler involved in order to flush out work that queued and not yet submitted.
v2: Updated to add support for flushing the scheduler queue by time stamp rather than just doing a blanket flush.
Change-Id: I95dcc2a2ee5c1a844748621c333994ddd6cf6a66
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem.c | 24 ++++++++- drivers/gpu/drm/i915/i915_scheduler.c | 92 +++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 3 ++ 3 files changed, 118 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_gem.c +23 -1
@@ -3768,6 +3768,10 @@ int i915_gpu_idle(struct drm_device *dev)/* Flush everything onto the inactive list. */ for_each_ring (ring, dev_priv, i) { + ret =i915_scheduler_flush (ring, true); +if (ret <0 ) +return ret; +if (!i915.enable_execlists) {struct drm_i915_gem_request *req;
@@ -4481,7 +4485,8 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)unsigned long recent_enough = jiffies -DRM_I915_THROTTLE_JIFFIES ;struct drm_i915_gem_request *request, *target =NULL ;unsigned reset_counter; -int ret; +int i, ret; +struct intel_engine_cs *ring; ret =i915_gem_wait_for_error (&dev_priv->gpu_error);if (ret)
@@ -4491,6 +4496,23 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)if (ret)return ret; +for_each_ring (ring, dev_priv, i) { +/* +* Flush out scheduler entries that are getting 'stale'. Note +* that the following recent_enough test will only check +* against the time at which the request was submitted to the +* hardware (i.e. when it left the scheduler) not the time it +* was submitted to the driver. +* +* Also, there is not much point worring about busy return +* codes from the scheduler flush call. Even if more work +* cannot be submitted right now for whatever reason, we +* still want to throttle against stale work that has already +* been submitted. +*/ +i915_scheduler_flush_stamp (ring, recent_enough, false); + } +spin_lock (&file_priv->mm.lock);list_for_each_entry (request, &file_priv->mm.request_list, client_list) {if (time_after_eq (request->emitted_jiffies, recent_enough))
drivers/gpu/drm/i915/i915_scheduler.c +92 -0
98 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h +3 -0
@@ -92,6 +92,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qbool i915_scheduler_notify_request (struct drm_i915_gem_request *req);void i915_scheduler_wakeup (struct drm_device *dev);void i915_gem_scheduler_work_handler (struct work_struct *work); +int i915_scheduler_flush (struct intel_engine_cs *ring,bool is_locked); +int i915_scheduler_flush_stamp (struct intel_engine_cs *ring, +unsigned long stamp,bool is_locked);int i915_scheduler_flush_request (struct drm_i915_gem_request *req,bool is_locked); bool i915_scheduler_is_request_tracked(struct drm_i915_gem_request *req,
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 21/40 drm/i915: Added scheduler flush calls to ring throttle and idle functions
From: John Harrison <[email protected]>
When requesting that all GPU work is completed, it is now necessary to get the scheduler involved in order to flush out work that queued and not yet submitted.
v2: Updated to add support for flushing the scheduler queue by time stamp rather than just doing a blanket flush.
v3: Moved submit_max_priority() to this patch from an earlier patch is it is no longer required in the other.
Change-Id: I95dcc2a2ee5c1a844748621c333994ddd6cf6a66
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem.c | 24 ++++++- drivers/gpu/drm/i915/i915_scheduler.c | 132 ++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 3 + 3 files changed, 158 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_gem.c +23 -1
@@ -3765,6 +3765,10 @@ int i915_gpu_idle(struct drm_device *dev)/* Flush everything onto the inactive list. */ for_each_ring (ring, dev_priv, i) { + ret =i915_scheduler_flush (ring, true); +if (ret <0 ) +return ret; +if (!i915.enable_execlists) {struct drm_i915_gem_request *req;
@@ -4478,7 +4482,8 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)unsigned long recent_enough = jiffies -DRM_I915_THROTTLE_JIFFIES ;struct drm_i915_gem_request *request, *target =NULL ;unsigned reset_counter; -int ret; +int i, ret; +struct intel_engine_cs *ring; ret =i915_gem_wait_for_error (&dev_priv->gpu_error);if (ret)
@@ -4488,6 +4493,23 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)if (ret)return ret; +for_each_ring (ring, dev_priv, i) { +/* +* Flush out scheduler entries that are getting 'stale'. Note +* that the following recent_enough test will only check +* against the time at which the request was submitted to the +* hardware (i.e. when it left the scheduler) not the time it +* was submitted to the driver. +* +* Also, there is not much point worring about busy return +* codes from the scheduler flush call. Even if more work +* cannot be submitted right now for whatever reason, we +* still want to throttle against stale work that has already +* been submitted. +*/ +i915_scheduler_flush_stamp (ring, recent_enough, false); + } +spin_lock (&file_priv->mm.lock);list_for_each_entry (request, &file_priv->mm.request_list, client_list) {if (time_after_eq (request->emitted_jiffies, recent_enough))
drivers/gpu/drm/i915/i915_scheduler.c +132 -0
150 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h +3 -0
@@ -93,6 +93,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qbool i915_scheduler_notify_request (struct drm_i915_gem_request *req);void i915_scheduler_wakeup (struct drm_device *dev);void i915_gem_scheduler_work_handler (struct work_struct *work); +int i915_scheduler_flush (struct intel_engine_cs *ring,bool is_locked); +int i915_scheduler_flush_stamp (struct intel_engine_cs *ring, +unsigned long stamp,bool is_locked);bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req,bool *completed,bool *busy);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 21/39 drm/i915: Added a module parameter for allowing scheduler overrides
In reply to [email protected]From: John Harrison <[email protected]>
It can be useful to be able to disable certain features (e.g. the entire scheduler) via a module parameter for debugging purposes. A parameter has the advantage of not being a compile time switch but without implying that it can be changed dynamically at runtime.
Change-Id: I92f4c832be88f5b34b49b90d6a9903fac68f7004
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 1 + drivers/gpu/drm/i915/i915_params.c | 4 ++++ drivers/gpu/drm/i915/i915_scheduler.c | 5 ++++- drivers/gpu/drm/i915/i915_scheduler.h | 5 +++++ 4 files changed, 14 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_drv.h +1 -0
@@ -2717,6 +2717,7 @@ struct i915_params {bool verbose_state_checks;bool nuclear_pageflip;int edp_vswing; +int scheduler_override; };extern struct i915_params i915__read_mostly ;
drivers/gpu/drm/i915/i915_params.c +4 -0
@@ -54,6 +54,7 @@ struct i915_params i915 __read_mostly = { .edp_vswing = 0, .enable_guc_submission = true, .guc_log_level = -1, + .scheduler_override = 1, };module_param_named (modeset, i915.modeset, int,0400 );
@@ -191,3 +192,6 @@ MODULE_PARM_DESC(enable_guc_submission, "Enable GuC submission (default:false)")module_param_named (guc_log_level, i915.guc_log_level, int,0400 );MODULE_PARM_DESC (guc_log_level,"GuC firmware logging level (-1:disabled (default), 0-3:enabled)" ); + +module_param_named (scheduler_override, i915.scheduler_override, int,0600 ); +MODULE_PARM_DESC (scheduler_override,"Scheduler override mask (0 = none, 1 = direct submission [default])" );
drivers/gpu/drm/i915/i915_scheduler.c +4 -1
@@ -44,6 +44,9 @@ bool i915_scheduler_is_enabled(struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private; +if (i915.scheduler_override & i915_so_direct_submit) +return false; +return dev_priv->scheduler !=NULL ; }
@@ -90,7 +93,7 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)BUG_ON (!scheduler); -if (1 /*i915.scheduler_override & i915_so_direct_submit*/ ) { +if (i915.scheduler_override & i915_so_direct_submit) {int ret;intel_ring_reserved_space_cancel (qe->params.request->ringbuf);
drivers/gpu/drm/i915/i915_scheduler.h +5 -0
@@ -83,6 +83,11 @@ enum { i915_sf_submitting = (1 << 1), }; +/* Options for 'scheduler_override' module parameter: */ +enum { + i915_so_direct_submit = (1 <<0 ), +}; +bool i915_scheduler_is_enabled (struct drm_device *dev);int i915_scheduler_init (struct drm_device *dev); int i915_scheduler_closefile(struct drm_device *dev,
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 22/39 drm/i915: Support for 'unflushed' ring idle
In reply to [email protected]From: John Harrison <[email protected]>
When the seqno wraps around zero, the entire GPU is forced to be idle for some reason (possibly only to work around issues with hardware semaphores but no-one seems too sure!). This causes a problem if the force idle occurs at an inopportune moment such as in the middle of submitting a batch buffer. Specifically, it would lead to recursive submits - submitting work requires a new seqno, the new seqno requires idling the ring, idling the ring requires submitting work, submitting work requires a new seqno...
This change adds a 'flush' parameter to the idle function call which specifies whether the scheduler queues should be flushed out. I.e. is the call intended to just idle the ring as it is right now (no flush) or is it intended to force all outstanding work out of the system (with flush).
In the seqno wrap case, pending work is not an issue because the next operation will be to submit it. However, in other cases, the intention is to make sure everything that could be done has been done.
Change-Id: I182e9a5853666c64ecc9e84d8a8b820a7f8e8836
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem.c | 4 ++-- drivers/gpu/drm/i915/intel_lrc.c | 2 +- drivers/gpu/drm/i915/intel_ringbuffer.c | 17 +++++++++++++++-- drivers/gpu/drm/i915/intel_ringbuffer.h | 2 +- 4 files changed, 19 insertions(+), 6 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +2 -2
@@ -2477,7 +2477,7 @@ i915_gem_init_seqno(struct drm_device *dev, u32 seqno)/* Carefully retire all requests without writing to the rings */ for_each_ring (ring, dev_priv, i) { - ret =intel_ring_idle (ring); + ret =intel_ring_idle (ring, false);if (ret)return ret; }
@@ -3788,7 +3788,7 @@ int i915_gpu_idle(struct drm_device *dev)i915_add_request_no_flush (req); } - ret =intel_ring_idle (ring); + ret =intel_ring_idle (ring, true);if (ret)return ret; }
drivers/gpu/drm/i915/intel_lrc.c +1 -1
@@ -1014,7 +1014,7 @@ void intel_logical_ring_stop(struct intel_engine_cs *ring)if (!intel_ring_initialized (ring))return ; - ret =intel_ring_idle (ring); + ret =intel_ring_idle (ring, true);if (ret && !i915_reset_in_progress (&to_i915 (ring->dev)->gpu_error))DRM_ERROR ("failed to quiesce %s whilst cleaning up: %d\n" , ring->name, ret);
drivers/gpu/drm/i915/intel_ringbuffer.c +15 -2
@@ -2278,9 +2278,22 @@ static void __wrap_ring_buffer(struct intel_ringbuffer *ringbuf)intel_ring_update_space (ringbuf); } -int intel_ring_idle (struct intel_engine_cs *ring) +int intel_ring_idle (struct intel_engine_cs *ring,bool flush) {struct drm_i915_gem_request *req; +int ret; + +/* +* NB: Must not flush the scheduler if this idle request is from +* within an execbuff submission (i.e. due to 'get_seqno' calling +* 'wrap_seqno' calling 'idle'). As that would lead to recursive +* flushes! +*/ +if (flush) { + ret =i915_scheduler_flush (ring, true); +if (ret) +return ret; + }/* Wait upon the last request to be completed */ if (list_empty (&ring->request_list))
@@ -3085,7 +3098,7 @@ intel_stop_ring_buffer(struct intel_engine_cs *ring)if (!intel_ring_initialized (ring))return ; - ret =intel_ring_idle (ring); + ret =intel_ring_idle (ring, true);if (ret && !i915_reset_in_progress (&to_i915 (ring->dev)->gpu_error))DRM_ERROR ("failed to quiesce %s whilst cleaning up: %d\n" , ring->name, ret);
drivers/gpu/drm/i915/intel_ringbuffer.h +1 -1
@@ -488,7 +488,7 @@ void intel_ring_update_space(struct intel_ringbuffer *ringbuf);int intel_ring_space (struct intel_ringbuffer *ringbuf);bool intel_ring_stopped (struct intel_engine_cs *ring); -int __must_check intel_ring_idle (struct intel_engine_cs *ring); +int __must_check intel_ring_idle (struct intel_engine_cs *ring,bool flush);void intel_ring_init_seqno (struct intel_engine_cs *ring,u32 seqno);int intel_ring_flush_all_caches (struct drm_i915_gem_request *req);int intel_ring_invalidate_all_caches (struct drm_i915_gem_request *req);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 23/39 drm/i915: Defer seqno allocation until actual hardware submission time
In reply to [email protected]From: John Harrison <[email protected]>
The seqno value is now only used for the final test for completion of a request. It is no longer used to track the request through the software stack. Thus it is no longer necessary to allocate the seqno immediately with the request. Instead, it can be done lazily and left until the request is actually sent to the hardware. This is particular advantageous with a GPU scheduler as the requests can then be re-ordered between their creation and their hardware submission without having out of order seqnos.
v2: i915_add_request() can't fail!
v3: combine with 'drm/i915: Assign seqno at start of exec_final()' Various bits of code during the execbuf code path need a seqno value to be assigned to the request. This change makes this assignment explicit at the start of submission_final() rather than relying on an auto-generated seqno to have happened already. This is in preparation for a future patch which changes seqno values to be assigned lazily (during add_request).
Change-Id: I0d922b84c517611a79fa6c2b9e730d4fe3671d6a
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 1 + drivers/gpu/drm/i915/i915_gem.c | 21 ++++++++++++++++++++- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++++++++ drivers/gpu/drm/i915/intel_lrc.c | 13 +++++++++++++ 4 files changed, 47 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_drv.h +1 -0
@@ -2225,6 +2225,7 @@ struct drm_i915_gem_request {/** GEM sequence number associated with this request. */ uint32_t seqno; +uint32_t reserved_seqno;/* Unique identifier which can be used for trace points & debug */ uint32_t uniq;
drivers/gpu/drm/i915/i915_gem.c +20 -1
@@ -2529,6 +2529,9 @@ i915_gem_get_seqno(struct drm_device *dev, u32 *seqno)/* reserve 0 for non-seqno */ if (dev_priv->next_seqno ==0 ) { +/* Why is the full re-initialisation required? Is it only for +* hardware semaphores? If so, could skip it in the case where +* semaphores are disabled? */ int ret =i915_gem_init_seqno (dev,0 );if (ret)return ret;
@@ -2586,6 +2589,12 @@ void __i915_add_request(struct drm_i915_gem_request *request,WARN (ret,"*_ring_flush_all_caches failed: %d!\n" , ret); } +/* Make the request's seqno 'live': */ +if (!request->seqno) { + request->seqno = request->reserved_seqno; +WARN_ON (request->seqno != dev_priv->last_seqno); + } +/* Record the position of the start of the request so that * should we detect the updated seqno part-way through the * GPU processing the request, we never over-estimate the
@@ -2835,6 +2844,9 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)list_for_each_entry_safe (req, req_next, &ring->fence_signal_list, signal_link) {if (!req->cancelled) { +/* How can this happen? */ +WARN_ON (req->seqno ==0 ); +if (!i915_seqno_passed (seqno, req->seqno))break ; }
@@ -3057,7 +3069,14 @@ int i915_gem_request_alloc(struct intel_engine_cs *ring,if (req ==NULL )return -ENOMEM ; - ret =i915_gem_get_seqno (ring->dev, &req->seqno); +/* +* Assign an identifier to track this request through the hardware +* but don't make it live yet. It could change in the future if this +* request gets overtaken. However, it still needs to be allocated +* in advance because the point of submission must not fail and seqno +* allocation can fail. +*/ + ret =i915_gem_get_seqno (ring->dev, &req->reserved_seqno);if (ret) goto err;
drivers/gpu/drm/i915/i915_gem_execbuffer.c +13 -0
@@ -1248,6 +1248,19 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex)); +/* Make sure the request's seqno is the latest and greatest: */ +if (params->request->reserved_seqno != dev_priv->last_seqno) { + ret =i915_gem_get_seqno (ring->dev, ¶ms->request->reserved_seqno); +if (ret) +return ret; + } +/* +* And make it live because some of the execbuff submission code +* requires the seqno to be available up front. */ +WARN_ON (params->request->seqno); + params->request->seqno = params->request->reserved_seqno; +WARN_ON (params->request->seqno != dev_priv->last_seqno); + ret =intel_ring_reserve_space (params->request);if (ret)return ret;
drivers/gpu/drm/i915/intel_lrc.c +13 -0
@@ -932,6 +932,19 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex)); +/* Make sure the request's seqno is the latest and greatest: */ +if (params->request->reserved_seqno != dev_priv->last_seqno) { + ret =i915_gem_get_seqno (ring->dev, ¶ms->request->reserved_seqno); +if (ret) +return ret; + } +/* +* And make it live because some of the execbuff submission code +* requires the seqno to be available up front. */ +WARN_ON (params->request->seqno); + params->request->seqno = params->request->reserved_seqno; +WARN_ON (params->request->seqno != dev_priv->last_seqno); + ret =intel_logical_ring_reserve_space (params->request);if (ret)return ret;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 24/40 drm/i915: Defer seqno allocation until actual hardware submission time
From: John Harrison <[email protected]>
The seqno value is now only used for the final test for completion of a request. It is no longer used to track the request through the software stack. Thus it is no longer necessary to allocate the seqno immediately with the request. Instead, it can be done lazily and left until the request is actually sent to the hardware. This is particular advantageous with a GPU scheduler as the requests can then be re-ordered between their creation and their hardware submission without having out of order seqnos.
v2: i915_add_request() can't fail!
Combine with 'drm/i915: Assign seqno at start of exec_final()' Various bits of code during the execbuf code path need a seqno value to be assigned to the request. This change makes this assignment explicit at the start of submission_final() rather than relying on an auto-generated seqno to have happened already. This is in preparation for a future patch which changes seqno values to be assigned lazily (during add_request).
v3: Updated to use locally cached request pointer.
Change-Id: I0d922b84c517611a79fa6c2b9e730d4fe3671d6a
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 1 + drivers/gpu/drm/i915/i915_gem.c | 21 ++++++++++++++++++++- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++++++++ drivers/gpu/drm/i915/intel_lrc.c | 13 +++++++++++++ 4 files changed, 47 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_drv.h +1 -0
@@ -2225,6 +2225,7 @@ struct drm_i915_gem_request {/** GEM sequence number associated with this request. */ uint32_t seqno; +uint32_t reserved_seqno;/* Unique identifier which can be used for trace points & debug */ uint32_t uniq;
drivers/gpu/drm/i915/i915_gem.c +20 -1
@@ -2525,6 +2525,9 @@ i915_gem_get_seqno(struct drm_device *dev, u32 *seqno)/* reserve 0 for non-seqno */ if (dev_priv->next_seqno ==0 ) { +/* Why is the full re-initialisation required? Is it only for +* hardware semaphores? If so, could skip it in the case where +* semaphores are disabled? */ int ret =i915_gem_init_seqno (dev,0 );if (ret)return ret;
@@ -2582,6 +2585,12 @@ void __i915_add_request(struct drm_i915_gem_request *request,WARN (ret,"*_ring_flush_all_caches failed: %d!\n" , ret); } +/* Make the request's seqno 'live': */ +if (!request->seqno) { + request->seqno = request->reserved_seqno; +WARN_ON (request->seqno != dev_priv->last_seqno); + } +/* Record the position of the start of the request so that * should we detect the updated seqno part-way through the * GPU processing the request, we never over-estimate the
@@ -2830,6 +2839,9 @@ void i915_gem_request_notify(struct intel_engine_cs *ring, bool fence_locked)list_for_each_entry_safe (req, req_next, &ring->fence_signal_list, signal_link) {if (!req->cancelled) { +/* How can this happen? */ +WARN_ON (req->seqno ==0 ); +if (!i915_seqno_passed (seqno, req->seqno))break ; }
@@ -3054,7 +3066,14 @@ int i915_gem_request_alloc(struct intel_engine_cs *ring,if (req ==NULL )return -ENOMEM ; - ret =i915_gem_get_seqno (ring->dev, &req->seqno); +/* +* Assign an identifier to track this request through the hardware +* but don't make it live yet. It could change in the future if this +* request gets overtaken. However, it still needs to be allocated +* in advance because the point of submission must not fail and seqno +* allocation can fail. +*/ + ret =i915_gem_get_seqno (ring->dev, &req->reserved_seqno);if (ret) goto err;
drivers/gpu/drm/i915/i915_gem_execbuffer.c +13 -0
@@ -1249,6 +1249,19 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex)); +/* Make sure the request's seqno is the latest and greatest: */ +if (req->reserved_seqno != dev_priv->last_seqno) { + ret =i915_gem_get_seqno (ring->dev, &req->reserved_seqno); +if (ret) +return ret; + } +/* +* And make it live because some of the execbuff submission code +* requires the seqno to be available up front. */ +WARN_ON (req->seqno); + req->seqno = req->reserved_seqno; +WARN_ON (req->seqno != dev_priv->last_seqno); + ret =intel_ring_reserve_space (req);if (ret)return ret;
drivers/gpu/drm/i915/intel_lrc.c +13 -0
@@ -934,6 +934,19 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex)); +/* Make sure the request's seqno is the latest and greatest: */ +if (req->reserved_seqno != dev_priv->last_seqno) { + ret =i915_gem_get_seqno (ring->dev, &req->reserved_seqno); +if (ret) +return ret; + } +/* +* And make it live because some of the execbuff submission code +* requires the seqno to be available up front. */ +WARN_ON (req->seqno); + req->seqno = req->reserved_seqno; +WARN_ON (req->seqno != dev_priv->last_seqno); + ret =intel_logical_ring_reserve_space (req);if (ret)return ret;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 24/39 drm/i915: Added immediate submission override to scheduler
In reply to [email protected]From: John Harrison <[email protected]>
To aid with debugging issues related to the scheduler, it can be useful to ensure that all batch buffers are submitted immediately rather than queued until later. This change adds an override flag via the module parameter to force instant submission.
Change-Id: I7652df53e2d3c3d77d78bebcf99856e2c53f2801
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_scheduler.c | 7 +++++-- drivers/gpu/drm/i915/i915_scheduler.h | 1 + 2 files changed, 6 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_scheduler.c +5 -2
@@ -235,8 +235,11 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)list_add_tail (&node->link, &scheduler->node_queue[ring->id]); - not_flying =i915_scheduler_count_flying (scheduler, ring) < - scheduler->min_flying; +if (i915.scheduler_override & i915_so_submit_on_queue) + not_flying = true; +else + not_flying =i915_scheduler_count_flying (scheduler, ring) < + scheduler->min_flying;spin_unlock_irqrestore (&scheduler->lock, flags);
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -86,6 +86,7 @@ enum {/* Options for 'scheduler_override' module parameter: */ enum { i915_so_direct_submit = (1 <<0 ), + i915_so_submit_on_queue = (1 <<1 ), };bool i915_scheduler_is_enabled (struct drm_device *dev);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 25/39 drm/i915: Add sync wait support to scheduler
In reply to [email protected]From: John Harrison <[email protected]>
There is a sync framework to allow work for multiple independent systems to be synchronised with each other but without stalling the CPU whether in the application or the driver. This patch adds support for this framework to the GPU scheduler.
Batch buffers can now have sync framework fence objects associated with them. The scheduler will look at this fence when deciding what to submit next to the hardware. If the fence is outstanding then that batch buffer will be passed over in preference of one that is ready to run. If no other batches are ready then the scheduler will queue an asynchronous callback to be woken up when the fence has been signalled. The callback will wake the scheduler and submit the now ready batch buffer.
v2: Updated to use the new sync_fence_is_signaled() API rather than having to know about the internals of a fence object.
Also removed the spin lock from the fence wait callback function. This was used to clear the 'waiting' flag. However, it causes a problem with the TDR code. Specifically, when cancelling work packets due to a TDR there is a code path where the fence can be signalled while the spinlock is already held. It is not really necessary to clear the flag anyway as it's purpose is solely to prevent multiple waits being issued. Once the fence has been signalled, no further waits will be attempted so it doesn't matter whether the fence is marked as having an outstanding wait or not.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 1 + drivers/gpu/drm/i915/i915_scheduler.c | 144 ++++++++++++++++++++++++++++++++-- drivers/gpu/drm/i915/i915_scheduler.h | 6 ++ 3 files changed, 146 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +1 -0
@@ -1698,6 +1698,7 @@ struct i915_execbuffer_params {uint64_t batch_obj_vm_offset;struct intel_engine_cs *ring;struct drm_i915_gem_object *batch_obj; +struct sync_fence *fence_wait;struct drm_clip_rect *cliprects;uint32_t instp_mask;int instp_mode;
drivers/gpu/drm/i915/i915_scheduler.c +139 -5
216 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h +6 -0
@@ -51,6 +51,11 @@ struct i915_scheduler_obj_entry {struct drm_i915_gem_object *obj; }; +enum i915_scheduler_queue_entry_flags { +/* Fence is being waited on */ + i915_qef_fence_waiting = (1 <<0 ), +}; +struct i915_scheduler_queue_entry {struct i915_execbuffer_params params;/* -1023 = lowest priority, 0 = default, 1023 = highest */
@@ -63,6 +68,7 @@ struct i915_scheduler_queue_entry {enum i915_scheduler_queue_status status;unsigned long stamp;struct list_head link; +uint32_t flags; };struct i915_scheduler {
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 26/39 drm/i915: Connecting execbuff fences to scheduler
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler now supports sync framework fences being associated with batch buffers. The execbuff IOCTL allows such fences to be passed in from user land. This patch wires the two together so that the IOCTL no longer needs to stall on the fence immediately. Instead the stall is now swallowed by the scheduler's scheduling algorithm.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 19 ++++++++++++++++++- drivers/gpu/drm/i915/i915_scheduler.c | 3 +++ 2 files changed, 21 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +18 -1
@@ -1529,7 +1529,9 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,/* * Without a GPU scheduler, any fence waits must be done up front. */ -if (args->flags &I915_EXEC_WAIT_FENCE ) { +if ((args->flags &I915_EXEC_WAIT_FENCE ) && + (i915.scheduler_override & i915_so_direct_submit)) + { ret =i915_early_fence_wait (ring, fd_fence_wait);if (ret <0 )return ret;
@@ -1714,6 +1716,18 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,i915_gem_context_reference (ctx); params->ctx = ctx; +if (args->flags &I915_EXEC_WAIT_FENCE ) { +if (fd_fence_wait <0 ) { +DRM_ERROR ("Wait fence for ring %d has invalid id %d\n" , + (int ) ring->id, fd_fence_wait); + }else { + params->fence_wait =sync_fence_fdget (fd_fence_wait); +if (params->fence_wait ==NULL ) +DRM_ERROR ("Invalid wait fence %d\n" , + fd_fence_wait); + } + } +if (args->flags &I915_EXEC_CREATE_FENCE ) {/* * Caller has requested a sync fence.
@@ -1794,6 +1808,9 @@ err:i915_gem_context_unreference (params->ctx); } +if (params->fence_wait) +sync_fence_put (params->fence_wait); +/* * If the request was created but not successfully submitted then it * must be freed again. If it was submitted then it is being tracked
drivers/gpu/drm/i915/i915_scheduler.c +3 -0
@@ -915,6 +915,9 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,else signalled = true; +if (!signalled) + signalled =i915_safe_to_ignore_fence (ring, node->params.fence_wait); + has_local = false; has_remote = false;for (i =0 ; i < node->num_deps; i++) {
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 27/39 drm/i915: Added trace points to scheduler
In reply to [email protected]From: John Harrison <[email protected]>
Added trace points to the scheduler to track all the various events, node state transitions and other interesting things that occur.
v2: Updated for new request completion tracking implementation.
Change-Id: I9886390cfc7897bc1faf50a104bc651d8baed8a5
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 2 + drivers/gpu/drm/i915/i915_scheduler.c | 24 ++++ drivers/gpu/drm/i915/i915_trace.h | 190 +++++++++++++++++++++++++++++ drivers/gpu/drm/i915/intel_lrc.c | 2 + 4 files changed, 218 insertions(+)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +2 -0
@@ -1226,6 +1226,8 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,i915_gem_execbuffer_move_to_active (vmas, params->request); +trace_i915_gem_ring_queue (ring, params); + qe =container_of (params,typeof (*qe), params); ret =i915_scheduler_queue_execbuffer (qe);if (ret)
drivers/gpu/drm/i915/i915_scheduler.c +24 -0
@@ -97,6 +97,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)if (i915.scheduler_override & i915_so_direct_submit) {int ret; +trace_i915_scheduler_queue (qe->params.ring, qe); +WARN_ON (qe->params.fence_wait && (!sync_fence_is_signaled (qe->params.fence_wait)));
@@ -248,6 +250,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe) not_flying =i915_scheduler_count_flying (scheduler, ring) < scheduler->min_flying; +trace_i915_scheduler_queue (ring, node); +trace_i915_scheduler_node_state_change (ring, node); +spin_unlock_irqrestore (&scheduler->lock, flags);if (not_flying)
@@ -275,6 +280,9 @@ static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node) node->status = i915_sqs_flying; +trace_i915_scheduler_fly (ring, node); +trace_i915_scheduler_node_state_change (ring, node); +if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {bool success = true;
@@ -340,6 +348,8 @@ static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node) node->status = i915_sqs_queued; node->params.request->seqno =0 ; +trace_i915_scheduler_unfly (node->params.ring, node); +trace_i915_scheduler_node_state_change (node->params.ring, node); }/* Give up on a popped node completely. For example, because it is causing the
@@ -350,6 +360,8 @@ static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)BUG_ON (!I915_SQS_IS_FLYING (node)); node->status = i915_sqs_dead; +trace_i915_scheduler_unfly (node->params.ring, node); +trace_i915_scheduler_node_state_change (node->params.ring, node); }/*
@@ -369,6 +381,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)struct i915_scheduler_queue_entry *node = req->scheduler_qe;unsigned long flags; +trace_i915_scheduler_landing (req); +if (!node)return false;
@@ -382,6 +396,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)else node->status = i915_sqs_complete; +trace_i915_scheduler_node_state_change (req->ring, node); +spin_unlock_irqrestore (&scheduler->lock, flags);return true;
@@ -524,6 +540,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)/* Launch more packets now? */ do_submit = (queued >0 ) && (flying < scheduler->min_flying); +trace_i915_scheduler_remove (ring, min_seqno, do_submit); +spin_unlock_irqrestore (&scheduler->lock, flags);if (!do_submit &&list_empty (&remove))
@@ -538,6 +556,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring) node =list_first_entry (&remove,typeof (*node), link);list_del (&node->link); +trace_i915_scheduler_destroy (ring, node); +if (node->params.fence_wait)sync_fence_put (node->params.fence_wait);
@@ -952,6 +972,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,INIT_LIST_HEAD (&best->link); best->status = i915_sqs_popped; +trace_i915_scheduler_node_state_change (ring, best); + ret =0 ; }else {/* Can only get here if:
@@ -1005,6 +1027,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,spin_lock_irqsave (&scheduler->lock, *flags); } +trace_i915_scheduler_pop_from_queue (ring, best); + *pop_node = best;return ret; }
drivers/gpu/drm/i915/i915_trace.h +190 -0
202 lines not on this page · raw message
drivers/gpu/drm/i915/intel_lrc.c +2 -0
@@ -910,6 +910,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,i915_gem_execbuffer_move_to_active (vmas, params->request); +trace_i915_gem_ring_queue (ring, params); + qe =container_of (params,typeof (*qe), params); ret =i915_scheduler_queue_execbuffer (qe);if (ret)
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 28/40 drm/i915: Added trace points to scheduler
From: John Harrison <[email protected]>
Added trace points to the scheduler to track all the various events, node state transitions and other interesting things that occur.
v2: Updated for new request completion tracking implementation.
v3: Updated for changes to node kill code.
Change-Id: I9886390cfc7897bc1faf50a104bc651d8baed8a5
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 2 + drivers/gpu/drm/i915/i915_scheduler.c | 26 ++++ drivers/gpu/drm/i915/i915_trace.h | 190 +++++++++++++++++++++++++++++ drivers/gpu/drm/i915/intel_lrc.c | 2 + 4 files changed, 220 insertions(+)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +2 -0
@@ -1226,6 +1226,8 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,i915_gem_execbuffer_move_to_active (vmas, params->request); +trace_i915_gem_ring_queue (ring, params); + qe =container_of (params,typeof (*qe), params); ret =i915_scheduler_queue_execbuffer (qe);if (ret)
drivers/gpu/drm/i915/i915_scheduler.c +26 -0
@@ -151,6 +151,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)if (i915.scheduler_override & i915_so_direct_submit) {int ret; +trace_i915_scheduler_queue (qe->params.ring, qe); +WARN_ON (qe->params.fence_wait && (!sync_fence_is_signaled (qe->params.fence_wait)));
@@ -271,6 +273,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe) not_flying =i915_scheduler_count_flying (scheduler, ring) < scheduler->min_flying; +trace_i915_scheduler_queue (ring, node); +trace_i915_scheduler_node_state_change (ring, node); +spin_unlock_irqrestore (&scheduler->lock, flags);if (not_flying)
@@ -298,6 +303,9 @@ static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node) node->status = i915_sqs_flying; +trace_i915_scheduler_fly (ring, node); +trace_i915_scheduler_node_state_change (ring, node); +if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) {bool success = true;
@@ -363,6 +371,8 @@ static void i915_scheduler_node_requeue(struct i915_scheduler_queue_entry *node) node->status = i915_sqs_queued; node->params.request->seqno =0 ; +trace_i915_scheduler_unfly (node->params.ring, node); +trace_i915_scheduler_node_state_change (node->params.ring, node); }/* Give up on a node completely. For example, because it is causing the
@@ -372,7 +382,11 @@ static void i915_scheduler_node_kill(struct i915_scheduler_queue_entry *node)BUG_ON (!node);BUG_ON (I915_SQS_IS_COMPLETE (node)); +if (I915_SQS_IS_FLYING (node)) +trace_i915_scheduler_unfly (node->params.ring, node); + node->status = i915_sqs_dead; +trace_i915_scheduler_node_state_change (node->params.ring, node); }/*
@@ -392,6 +406,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)struct i915_scheduler_queue_entry *node = req->scheduler_qe;unsigned long flags; +trace_i915_scheduler_landing (req); +if (!node)return false;
@@ -405,6 +421,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)else node->status = i915_sqs_complete; +trace_i915_scheduler_node_state_change (req->ring, node); +spin_unlock_irqrestore (&scheduler->lock, flags);return true;
@@ -550,6 +568,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)/* Launch more packets now? */ do_submit = (queued >0 ) && (flying < scheduler->min_flying); +trace_i915_scheduler_remove (ring, min_seqno, do_submit); +spin_unlock_irqrestore (&scheduler->lock, flags);if (!do_submit &&list_empty (&remove))
@@ -564,6 +584,8 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring) node =list_first_entry (&remove,typeof (*node), link);list_del (&node->link); +trace_i915_scheduler_destroy (ring, node); +if (node->params.fence_wait)sync_fence_put (node->params.fence_wait);
@@ -927,6 +949,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,INIT_LIST_HEAD (&best->link); best->status = i915_sqs_popped; +trace_i915_scheduler_node_state_change (ring, best); + ret =0 ; }else {/* Can only get here if:
@@ -980,6 +1004,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *ring,spin_lock_irqsave (&scheduler->lock, *flags); } +trace_i915_scheduler_pop_from_queue (ring, best); + *pop_node = best;return ret; }
drivers/gpu/drm/i915/i915_trace.h +190 -0
202 lines not on this page · raw message
drivers/gpu/drm/i915/intel_lrc.c +2 -0
@@ -910,6 +910,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,i915_gem_execbuffer_move_to_active (vmas, params->request); +trace_i915_gem_ring_queue (ring, params); + qe =container_of (params,typeof (*qe), params); ret =i915_scheduler_queue_execbuffer (qe);if (ret)
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 28/39 drm/i915: Added scheduler queue throttling by DRM file handle
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler decouples the submission of batch buffers to the driver from their subsequent submission to the hardware. This means that an application which is continuously submitting buffers as fast as it can could potentialy flood the driver. To prevent this, the driver now tracks how many buffers are in progress (queued in software or executing in hardware) and limits this to a given (tunable) number. If this number is exceeded then the queue to the driver will return EAGAIN and thus prevent the scheduler's queue becoming arbitrarily large.
Change-Id: I83258240aec7c810db08c006a3062d46aa91363f
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 2 ++ drivers/gpu/drm/i915/i915_gem_execbuffer.c | 8 +++++++ drivers/gpu/drm/i915/i915_scheduler.c | 34 ++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 2 ++ 4 files changed, 46 insertions(+)
drivers/gpu/drm/i915/i915_drv.h +2 -0
@@ -342,6 +342,8 @@ struct drm_i915_file_private { } rps;struct intel_engine_cs *bsd_ring; + +u32 scheduler_queue_length; };enum intel_dpll_id {
drivers/gpu/drm/i915/i915_gem_execbuffer.c +8 -0
@@ -1857,6 +1857,10 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,return -EINVAL ; } +/* Throttle batch requests per device file */ +if (i915_scheduler_file_queue_is_full (file)) +return -EAGAIN ; +/* Copy in the exec list from userland */ exec_list =drm_malloc_ab (sizeof (*exec_list), args->buffer_count); exec2_list =drm_malloc_ab (sizeof (*exec2_list), args->buffer_count);
@@ -1940,6 +1944,10 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,return -EINVAL ; } +/* Throttle batch requests per device file */ +if (i915_scheduler_file_queue_is_full (file)) +return -EAGAIN ; + exec2_list =kmalloc (sizeof (*exec2_list)*args->buffer_count,GFP_TEMPORARY | __GFP_NOWARN | __GFP_NORETRY);if (exec2_list ==NULL )
drivers/gpu/drm/i915/i915_scheduler.c +34 -0
@@ -40,6 +40,8 @@ static void i915_scheduler_priority_bump_clear(struct i915_scheduler *schstatic int i915_scheduler_priority_bump (struct i915_scheduler *scheduler,struct i915_scheduler_queue_entry *target,uint32_t bump); +static void i915_scheduler_file_queue_inc (struct drm_file *file); +static void i915_scheduler_file_queue_dec (struct drm_file *file);bool i915_scheduler_is_enabled (struct drm_device *dev) {
@@ -74,6 +76,7 @@ int i915_scheduler_init(struct drm_device *dev) scheduler->priority_level_max =1023 ; scheduler->priority_level_preempt =900 ; scheduler->min_flying =2 ; + scheduler->file_queue_max =64 ; dev_priv->scheduler = scheduler;
@@ -244,6 +247,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)list_add_tail (&node->link, &scheduler->node_queue[ring->id]); +i915_scheduler_file_queue_inc (node->params.file); +if (i915.scheduler_override & i915_so_submit_on_queue) not_flying = true;else
@@ -523,6 +528,12 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)/* Strip the dependency info while the mutex is still locked */ i915_scheduler_remove_dependent (scheduler, node); +/* Likewise clean up the file descriptor before it might disappear. */ +if (node->params.file) { +i915_scheduler_file_queue_dec (node->params.file); + node->params.file =NULL ; + } +continue ; }
@@ -1256,3 +1267,26 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)return 0 ; } + +bool i915_scheduler_file_queue_is_full (struct drm_file *file) +{ +struct drm_i915_file_private *file_priv = file->driver_priv; +struct drm_i915_private *dev_priv = file_priv->dev_priv; +struct i915_scheduler *scheduler = dev_priv->scheduler; + +return file_priv->scheduler_queue_length >= scheduler->file_queue_max; +} + +static void i915_scheduler_file_queue_inc (struct drm_file *file) +{ +struct drm_i915_file_private *file_priv = file->driver_priv; + + file_priv->scheduler_queue_length++; +} + +static void i915_scheduler_file_queue_dec (struct drm_file *file) +{ +struct drm_i915_file_private *file_priv = file->driver_priv; + + file_priv->scheduler_queue_length--; +}
drivers/gpu/drm/i915/i915_scheduler.h +2 -0
@@ -81,6 +81,7 @@ struct i915_scheduler {int32_t priority_level_max;int32_t priority_level_preempt;uint32_t min_flying; +uint32_t file_queue_max; };/* Flag bits for i915_scheduler::flags */
@@ -111,5 +112,6 @@ int i915_scheduler_flush_request(struct drm_i915_gem_request *req, bool is_locked);bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req,bool *completed,bool *busy); +bool i915_scheduler_file_queue_is_full (struct drm_file *file);#endif /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 29/40 drm/i915: Added scheduler queue throttling by DRM file handle
From: John Harrison <[email protected]>
The scheduler decouples the submission of batch buffers to the driver from their subsequent submission to the hardware. This means that an application which is continuously submitting buffers as fast as it can could potentialy flood the driver. To prevent this, the driver now tracks how many buffers are in progress (queued in software or executing in hardware) and limits this to a given (tunable) number. If this number is exceeded then the queue to the driver will return EAGAIN and thus prevent the scheduler's queue becoming arbitrarily large.
v3: Added a missing decrement of the file queue counter.
Change-Id: I83258240aec7c810db08c006a3062d46aa91363f
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 2 ++ drivers/gpu/drm/i915/i915_gem_execbuffer.c | 8 +++++++ drivers/gpu/drm/i915/i915_scheduler.c | 35 ++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 2 ++ 4 files changed, 47 insertions(+)
drivers/gpu/drm/i915/i915_drv.h +2 -0
@@ -342,6 +342,8 @@ struct drm_i915_file_private { } rps;struct intel_engine_cs *bsd_ring; + +u32 scheduler_queue_length; };enum intel_dpll_id {
drivers/gpu/drm/i915/i915_gem_execbuffer.c +8 -0
@@ -1862,6 +1862,10 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,return -EINVAL ; } +/* Throttle batch requests per device file */ +if (i915_scheduler_file_queue_is_full (file)) +return -EAGAIN ; +/* Copy in the exec list from userland */ exec_list =drm_malloc_ab (sizeof (*exec_list), args->buffer_count); exec2_list =drm_malloc_ab (sizeof (*exec2_list), args->buffer_count);
@@ -1945,6 +1949,10 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,return -EINVAL ; } +/* Throttle batch requests per device file */ +if (i915_scheduler_file_queue_is_full (file)) +return -EAGAIN ; + exec2_list =kmalloc (sizeof (*exec2_list)*args->buffer_count,GFP_TEMPORARY | __GFP_NOWARN | __GFP_NORETRY);if (exec2_list ==NULL )
drivers/gpu/drm/i915/i915_scheduler.c +35 -0
@@ -40,6 +40,8 @@ static void i915_scheduler_priority_bump_clear(struct i915_scheduler *schstatic int i915_scheduler_priority_bump (struct i915_scheduler *scheduler,struct i915_scheduler_queue_entry *target,uint32_t bump); +static void i915_scheduler_file_queue_inc (struct drm_file *file); +static void i915_scheduler_file_queue_dec (struct drm_file *file);bool i915_scheduler_is_enabled (struct drm_device *dev) {
@@ -74,6 +76,7 @@ int i915_scheduler_init(struct drm_device *dev) scheduler->priority_level_max =1023 ; scheduler->priority_level_preempt =900 ; scheduler->min_flying =2 ; + scheduler->file_queue_max =64 ; dev_priv->scheduler = scheduler;
@@ -267,6 +270,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)list_add_tail (&node->link, &scheduler->node_queue[ring->id]); +i915_scheduler_file_queue_inc (node->params.file); +if (i915.scheduler_override & i915_so_submit_on_queue) not_flying = true;else
@@ -551,6 +556,12 @@ static int i915_scheduler_remove(struct intel_engine_cs *ring)/* Strip the dependency info while the mutex is still locked */ i915_scheduler_remove_dependent (scheduler, node); +/* Likewise clean up the file descriptor before it might disappear. */ +if (node->params.file) { +i915_scheduler_file_queue_dec (node->params.file); + node->params.file =NULL ; + } +continue ; }
@@ -1194,6 +1205,7 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file) node->status, ring->name); +i915_scheduler_file_queue_dec (node->params.file); node->params.file =NULL ; } }
@@ -1202,3 +1214,26 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)return 0 ; } + +bool i915_scheduler_file_queue_is_full (struct drm_file *file) +{ +struct drm_i915_file_private *file_priv = file->driver_priv; +struct drm_i915_private *dev_priv = file_priv->dev_priv; +struct i915_scheduler *scheduler = dev_priv->scheduler; + +return file_priv->scheduler_queue_length >= scheduler->file_queue_max; +} + +static void i915_scheduler_file_queue_inc (struct drm_file *file) +{ +struct drm_i915_file_private *file_priv = file->driver_priv; + + file_priv->scheduler_queue_length++; +} + +static void i915_scheduler_file_queue_dec (struct drm_file *file) +{ +struct drm_i915_file_private *file_priv = file->driver_priv; + + file_priv->scheduler_queue_length--; +}
drivers/gpu/drm/i915/i915_scheduler.h +2 -0
@@ -82,6 +82,7 @@ struct i915_scheduler {int32_t priority_level_max;int32_t priority_level_preempt;uint32_t min_flying; +uint32_t file_queue_max; };/* Flag bits for i915_scheduler::flags */
@@ -110,5 +111,6 @@ int i915_scheduler_flush_stamp(struct intel_engine_cs *ring, unsigned long stamp, bool is_locked);bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req,bool *completed,bool *busy); +bool i915_scheduler_file_queue_is_full (struct drm_file *file);#endif /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 29/39 drm/i915: Added debugfs interface to scheduler tuning parameters
In reply to [email protected]From: John Harrison <[email protected]>
There are various parameters within the scheduler which can be tuned to improve performance, reduce memory footprint, etc. This change adds support for altering these via debugfs.
v2: Updated for priorities now being signed values.
Change-Id: I6c26765269ae7173ff4d3a5c20921eaaca7c36ed
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_debugfs.c | 141 ++++++++++++++++++++++++++++++++++++ 1 file changed, 141 insertions(+)
drivers/gpu/drm/i915/i915_debugfs.c +141 -0
@@ -39,6 +39,7 @@#include "intel_ringbuffer.h" #include <drm/i915_drm.h> #include "i915_drv.h" +#include "i915_scheduler.h" enum {ACTIVE_LIST ,
@@ -1122,6 +1123,141 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_next_seqno_fops, i915_next_seqno_get, i915_next_seqno_set, "0x%llx\n"); +static int +i915_scheduler_priority_min_get (void *data,u64 *val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + *val = (u64 ) scheduler->priority_level_min; +return 0 ; +} + +static int +i915_scheduler_priority_min_set (void *data,u64 val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + scheduler->priority_level_min = (int32_t ) val; +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (i915_scheduler_priority_min_fops, + i915_scheduler_priority_min_get, + i915_scheduler_priority_min_set, +"%lld\n" ); + +static int +i915_scheduler_priority_max_get (void *data,u64 *val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + *val = (u64 ) scheduler->priority_level_max; +return 0 ; +} + +static int +i915_scheduler_priority_max_set (void *data,u64 val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + scheduler->priority_level_max = (int32_t ) val; +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (i915_scheduler_priority_max_fops, + i915_scheduler_priority_max_get, + i915_scheduler_priority_max_set, +"%lld\n" ); + +static int +i915_scheduler_priority_preempt_get (void *data,u64 *val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + *val = (u64 ) scheduler->priority_level_preempt; +return 0 ; +} + +static int +i915_scheduler_priority_preempt_set (void *data,u64 val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + scheduler->priority_level_preempt = (u32 ) val; +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (i915_scheduler_priority_preempt_fops, + i915_scheduler_priority_preempt_get, + i915_scheduler_priority_preempt_set, +"%lld\n" ); + +static int +i915_scheduler_min_flying_get (void *data,u64 *val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + *val = (u64 ) scheduler->min_flying; +return 0 ; +} + +static int +i915_scheduler_min_flying_set (void *data,u64 val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + scheduler->min_flying = (u32 ) val; +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (i915_scheduler_min_flying_fops, + i915_scheduler_min_flying_get, + i915_scheduler_min_flying_set, +"%llu\n" ); + +static int +i915_scheduler_file_queue_max_get (void *data,u64 *val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + *val = (u64 ) scheduler->file_queue_max; +return 0 ; +} + +static int +i915_scheduler_file_queue_max_set (void *data,u64 val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + scheduler->file_queue_max = (u32 ) val; +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (i915_scheduler_file_queue_max_fops, + i915_scheduler_file_queue_max_get, + i915_scheduler_file_queue_max_set, +"%llu\n" ); +static int i915_frequency_info (struct seq_file *m,void *unused) {struct drm_info_node *node = m->private;
@@ -5457,6 +5593,11 @@ static const struct i915_debugfs_files { {"i915_gem_drop_caches", &i915_drop_caches_fops}, {"i915_error_state", &i915_error_state_fops}, {"i915_next_seqno", &i915_next_seqno_fops}, + {"i915_scheduler_priority_min", &i915_scheduler_priority_min_fops}, + {"i915_scheduler_priority_max", &i915_scheduler_priority_max_fops}, + {"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops}, + {"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops}, + {"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops}, {"i915_display_crc_ctl", &i915_display_crc_ctl_fops}, {"i915_pri_wm_latency", &i915_pri_wm_latency_fops}, {"i915_spr_wm_latency", &i915_spr_wm_latency_fops},
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 30/39 drm/i915: Added debug state dump facilities to scheduler
In reply to [email protected]From: John Harrison <[email protected]>
When debugging batch buffer submission issues, it is useful to be able to see what the current state of the scheduler is. This change adds functions for decoding the internal scheduler state and reporting it.
Change-Id: I0634168e3f3465ff023f5a673165c90b07e535b6
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_scheduler.c | 276 ++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 14 ++ 2 files changed, 290 insertions(+)
drivers/gpu/drm/i915/i915_scheduler.c +276 -0
@@ -36,6 +36,9 @@ static int i915_scheduler_submit_max_priority(struct intel_engine_cs *ri bool is_locked);static uint32_t i915_scheduler_count_flying (struct i915_scheduler *scheduler,struct intel_engine_cs *ring); +static int i915_scheduler_dump_locked (struct intel_engine_cs *ring, +const char *msg); +static int i915_scheduler_dump_all_locked (struct drm_device *dev,const char *msg);static void i915_scheduler_priority_bump_clear (struct i915_scheduler *scheduler); static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler, struct i915_scheduler_queue_entry *target,
@@ -53,6 +56,116 @@ bool i915_scheduler_is_enabled(struct drm_device *dev)return dev_priv->scheduler !=NULL ; } +const char *i915_qe_state_str (struct i915_scheduler_queue_entry *node) +{ +static char str[50 ]; +char *ptr = str; + + *(ptr++) = node->bumped ?'B' :'-' , + *(ptr++) =i915_gem_request_completed (node->params.request) ?'C' :'-' ; + + *ptr =0 ; + +return str; +} + +char i915_scheduler_queue_status_chr (enum i915_scheduler_queue_status status) +{ +switch (status) { +case i915_sqs_none: +return 'N' ; + +case i915_sqs_queued: +return 'Q' ; + +case i915_sqs_popped: +return 'X' ; + +case i915_sqs_flying: +return 'F' ; + +case i915_sqs_complete: +return 'C' ; + +case i915_sqs_dead: +return 'D' ; + +default : +break ; + } + +return '?' ; +} + +const char *i915_scheduler_queue_status_str ( +enum i915_scheduler_queue_status status) +{ +static char str[50 ]; + +switch (status) { +case i915_sqs_none: +return "None" ; + +case i915_sqs_queued: +return "Queued" ; + +case i915_sqs_popped: +return "Popped" ; + +case i915_sqs_flying: +return "Flying" ; + +case i915_sqs_complete: +return "Complete" ; + +case i915_sqs_dead: +return "Dead" ; + +default : +break ; + } + +sprintf (str,"[Unknown_%d!]" , status); +return str; +} + +const char *i915_scheduler_flag_str (uint32_t flags) +{ +static char str[100 ]; +char *ptr = str; + + *ptr =0 ; + +#define TEST_FLAG (flag, msg) \ + do { \ + if (flags & (flag)) { \ + strcpy(ptr, msg); \ + ptr += strlen(ptr); \ + flags &= ~(flag); \ + } \ + } while (0) + +TEST_FLAG (i915_sf_interrupts_enabled,"IntOn|" ); +TEST_FLAG (i915_sf_submitting,"Submitting|" ); +TEST_FLAG (i915_sf_dump_force,"DumpForce|" ); +TEST_FLAG (i915_sf_dump_details,"DumpDetails|" ); +TEST_FLAG (i915_sf_dump_dependencies,"DumpDeps|" ); + +#undef TEST_FLAG + +if (flags) { +sprintf (ptr,"Unknown_0x%X!" , flags); + ptr +=strlen (ptr); + } + +if (ptr == str) +strcpy (str,"-" ); +else + ptr[-1 ] =0 ; + +return str; +}; +int i915_scheduler_init (struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private;
@@ -603,6 +716,169 @@ void i915_gem_scheduler_work_handler(struct work_struct *work) } } +int i915_scheduler_dump_all (struct drm_device *dev,const char *msg) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +unsigned long flags; +int ret; + +spin_lock_irqsave (&scheduler->lock, flags); + ret =i915_scheduler_dump_all_locked (dev, msg); +spin_unlock_irqrestore (&scheduler->lock, flags); + +return ret; +} + +static int i915_scheduler_dump_all_locked (struct drm_device *dev,const char *msg) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct intel_engine_cs *ring; +int i, r, ret =0 ; + +for_each_ring (ring, dev_priv, i) { + scheduler->flags[ring->id] |= i915_sf_dump_force | + i915_sf_dump_details | + i915_sf_dump_dependencies; + r =i915_scheduler_dump_locked (ring, msg); +if (ret ==0 ) + ret = r; + } + +return ret; +} + +int i915_scheduler_dump (struct intel_engine_cs *ring,const char *msg) +{ +struct drm_i915_private *dev_priv = ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +unsigned long flags; +int ret; + +spin_lock_irqsave (&scheduler->lock, flags); + ret =i915_scheduler_dump_locked (ring, msg); +spin_unlock_irqrestore (&scheduler->lock, flags); + +return ret; +} + +static int i915_scheduler_dump_locked (struct intel_engine_cs *ring,const char *msg) +{ +struct drm_i915_private *dev_priv = ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct i915_scheduler_queue_entry *node; +int flying =0 , queued =0 , complete =0 , other =0 ; +static int old_flying =-1 , old_queued =-1 , old_complete =-1 ; +bool b_dump; +char brkt[2 ] = {'<' ,'>' }; + +if (!ring) +return -EINVAL ; + +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (I915_SQS_IS_QUEUED (node)) + queued++; +else if (I915_SQS_IS_FLYING (node)) + flying++; +else if (I915_SQS_IS_COMPLETE (node)) + complete++; +else + other++; + } + + b_dump = (flying != old_flying) || + (queued != old_queued) || + (complete != old_complete); +if (scheduler->flags[ring->id] & i915_sf_dump_force) { +if (!b_dump) { + b_dump = true; + brkt[0 ] ='{' ; + brkt[1 ] ='}' ; + } + + scheduler->flags[ring->id] &= ~i915_sf_dump_force; + } + +if (b_dump) { + old_flying = flying; + old_queued = queued; + old_complete = complete; +DRM_DEBUG_DRIVER ("<%s> Q:%02d, F:%02d, C:%02d, O:%02d, " +"Flags = %s, Next = %d:%d %c%s%c\n" , + ring->name, queued, flying, complete, other, +i915_scheduler_flag_str (scheduler->flags[ring->id]), + dev_priv->request_uniq, dev_priv->next_seqno, + brkt[0 ], msg, brkt[1 ]); + }else { +/*DRM_DEBUG_DRIVER("<%s> Q:%02d, F:%02d, C:%02d, O:%02d" +", Flags = %s, Next = %d:%d [%s]\n", +ring->name, +queued, flying, complete, other, +i915_scheduler_flag_str(scheduler->flags[ring->id]), +dev_priv->request_uniq, dev_priv->next_seqno, msg); */ + +return 0 ; + } + +if (scheduler->flags[ring->id] & i915_sf_dump_details) { +int i, deps; +uint32_t count, counts[i915_sqs_MAX]; + +memset (counts,0x00 ,sizeof (counts)); + +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (node->status < i915_sqs_MAX) { + count = counts[node->status]++; + }else { +DRM_DEBUG_DRIVER ("<%s> Unknown status: %d!\n" , + ring->name, node->status); + count =-1 ; + } + + deps =0 ; +for (i =0 ; i < node->num_deps; i++) +if (i915_scheduler_is_dependency_valid (node, i)) + deps++; + +DRM_DEBUG_DRIVER ("<%s> %c:%02d> uniq = %d, seqno" +" = %d/%s, deps = %d / %d, fence = %p/%d, %s [pri = " +"%4d]\n" , ring->name, +i915_scheduler_queue_status_chr (node->status), + count, + node->params.request->uniq, + node->params.request->seqno, + node->params.ring->name, + deps, node->num_deps, + node->params.fence_wait, + node->params.fence_wait ?sync_fence_is_signaled (node->params.fence_wait) :0 , +i915_qe_state_str (node), + node->priority); + +if ((scheduler->flags[ring->id] & i915_sf_dump_dependencies) + ==0 ) +continue ; + +for (i =0 ; i < node->num_deps; i++) +if (node->dep_list[i]) +DRM_DEBUG_DRIVER ("<%s> |-%c:" +"%02d%c uniq = %d, seqno = %d/%s, %s [pri = %4d]\n" , + ring->name, +i915_scheduler_queue_status_chr (node->dep_list[i]->status), + i, +i915_scheduler_is_dependency_valid (node, i) + ?'>' :'#' , + node->dep_list[i]->params.request->uniq, + node->dep_list[i]->params.request->seqno, + node->dep_list[i]->params.ring->name, +i915_qe_state_str (node->dep_list[i]), + node->dep_list[i]->priority); + } + } + +return 0 ; +} +int i915_scheduler_flush_request (struct drm_i915_gem_request *req,bool is_locked) {
drivers/gpu/drm/i915/i915_scheduler.h +14 -0
@@ -41,6 +41,9 @@ enum i915_scheduler_queue_status {/* Limit value for use with arrays/loops */ i915_sqs_MAX }; +char i915_scheduler_queue_status_chr (enum i915_scheduler_queue_status status); +const char *i915_scheduler_queue_status_str ( +enum i915_scheduler_queue_status status);#define I915_SQS_IS_QUEUED (node) (((node)->status == i915_sqs_queued))#define I915_SQS_IS_FLYING (node) (((node)->status == i915_sqs_flying))
@@ -70,6 +73,7 @@ struct i915_scheduler_queue_entry {struct list_head link;uint32_t flags; }; +const char *i915_qe_state_str (struct i915_scheduler_queue_entry *node);struct i915_scheduler {struct list_head node_queue[I915_NUM_RINGS ];
@@ -86,9 +90,16 @@ struct i915_scheduler {/* Flag bits for i915_scheduler::flags */ enum { +/* Internal state */ i915_sf_interrupts_enabled = (1 <<0 ), i915_sf_submitting = (1 <<1 ), + +/* Dump/debug flags */ + i915_sf_dump_force = (1 <<8 ), + i915_sf_dump_details = (1 <<9 ), + i915_sf_dump_dependencies = (1 <<10 ), }; +const char *i915_scheduler_flag_str (uint32_t flags);/* Options for 'scheduler_override' module parameter: */ enum {
@@ -110,6 +121,9 @@ int i915_scheduler_flush_stamp(struct intel_engine_cs *ring, unsigned long stamp, bool is_locked);int i915_scheduler_flush_request (struct drm_i915_gem_request *req,bool is_locked); +int i915_scheduler_dump (struct intel_engine_cs *ring, +const char *msg); +int i915_scheduler_dump_all (struct drm_device *dev,const char *msg);bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req,bool *completed,bool *busy);bool i915_scheduler_file_queue_is_full (struct drm_file *file);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 31/40 drm/i915: Added debug state dump facilities to scheduler
From: John Harrison <[email protected]>
When debugging batch buffer submission issues, it is useful to be able to see what the current state of the scheduler is. This change adds functions for decoding the internal scheduler state and reporting it.
v3: Updated a debug message with the new state_str() function.
Change-Id: I0634168e3f3465ff023f5a673165c90b07e535b6
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_scheduler.c | 280 +++++++++++++++++++++++++++++++++- drivers/gpu/drm/i915/i915_scheduler.h | 14 ++ 2 files changed, 292 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_scheduler.c +278 -2
@@ -36,6 +36,9 @@ static int i915_scheduler_submit_max_priority(struct intel_engine_cs *ri bool is_locked);static uint32_t i915_scheduler_count_flying (struct i915_scheduler *scheduler,struct intel_engine_cs *ring); +static int i915_scheduler_dump_locked (struct intel_engine_cs *ring, +const char *msg); +static int i915_scheduler_dump_all_locked (struct drm_device *dev,const char *msg);static void i915_scheduler_priority_bump_clear (struct i915_scheduler *scheduler); static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler, struct i915_scheduler_queue_entry *target,
@@ -53,6 +56,116 @@ bool i915_scheduler_is_enabled(struct drm_device *dev)return dev_priv->scheduler !=NULL ; } +const char *i915_qe_state_str (struct i915_scheduler_queue_entry *node) +{ +static char str[50 ]; +char *ptr = str; + + *(ptr++) = node->bumped ?'B' :'-' , + *(ptr++) =i915_gem_request_completed (node->params.request) ?'C' :'-' ; + + *ptr =0 ; + +return str; +} + +char i915_scheduler_queue_status_chr (enum i915_scheduler_queue_status status) +{ +switch (status) { +case i915_sqs_none: +return 'N' ; + +case i915_sqs_queued: +return 'Q' ; + +case i915_sqs_popped: +return 'X' ; + +case i915_sqs_flying: +return 'F' ; + +case i915_sqs_complete: +return 'C' ; + +case i915_sqs_dead: +return 'D' ; + +default : +break ; + } + +return '?' ; +} + +const char *i915_scheduler_queue_status_str ( +enum i915_scheduler_queue_status status) +{ +static char str[50 ]; + +switch (status) { +case i915_sqs_none: +return "None" ; + +case i915_sqs_queued: +return "Queued" ; + +case i915_sqs_popped: +return "Popped" ; + +case i915_sqs_flying: +return "Flying" ; + +case i915_sqs_complete: +return "Complete" ; + +case i915_sqs_dead: +return "Dead" ; + +default : +break ; + } + +sprintf (str,"[Unknown_%d!]" , status); +return str; +} + +const char *i915_scheduler_flag_str (uint32_t flags) +{ +static char str[100 ]; +char *ptr = str; + + *ptr =0 ; + +#define TEST_FLAG (flag, msg) \ + do { \ + if (flags & (flag)) { \ + strcpy(ptr, msg); \ + ptr += strlen(ptr); \ + flags &= ~(flag); \ + } \ + } while (0) + +TEST_FLAG (i915_sf_interrupts_enabled,"IntOn|" ); +TEST_FLAG (i915_sf_submitting,"Submitting|" ); +TEST_FLAG (i915_sf_dump_force,"DumpForce|" ); +TEST_FLAG (i915_sf_dump_details,"DumpDetails|" ); +TEST_FLAG (i915_sf_dump_dependencies,"DumpDeps|" ); + +#undef TEST_FLAG + +if (flags) { +sprintf (ptr,"Unknown_0x%X!" , flags); + ptr +=strlen (ptr); + } + +if (ptr == str) +strcpy (str,"-" ); +else + ptr[-1 ] =0 ; + +return str; +}; +int i915_scheduler_init (struct drm_device *dev) {struct drm_i915_private *dev_priv = dev->dev_private;
@@ -631,6 +744,169 @@ void i915_gem_scheduler_work_handler(struct work_struct *work) } } +int i915_scheduler_dump_all (struct drm_device *dev,const char *msg) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +unsigned long flags; +int ret; + +spin_lock_irqsave (&scheduler->lock, flags); + ret =i915_scheduler_dump_all_locked (dev, msg); +spin_unlock_irqrestore (&scheduler->lock, flags); + +return ret; +} + +static int i915_scheduler_dump_all_locked (struct drm_device *dev,const char *msg) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct intel_engine_cs *ring; +int i, r, ret =0 ; + +for_each_ring (ring, dev_priv, i) { + scheduler->flags[ring->id] |= i915_sf_dump_force | + i915_sf_dump_details | + i915_sf_dump_dependencies; + r =i915_scheduler_dump_locked (ring, msg); +if (ret ==0 ) + ret = r; + } + +return ret; +} + +int i915_scheduler_dump (struct intel_engine_cs *ring,const char *msg) +{ +struct drm_i915_private *dev_priv = ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +unsigned long flags; +int ret; + +spin_lock_irqsave (&scheduler->lock, flags); + ret =i915_scheduler_dump_locked (ring, msg); +spin_unlock_irqrestore (&scheduler->lock, flags); + +return ret; +} + +static int i915_scheduler_dump_locked (struct intel_engine_cs *ring,const char *msg) +{ +struct drm_i915_private *dev_priv = ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct i915_scheduler_queue_entry *node; +int flying =0 , queued =0 , complete =0 , other =0 ; +static int old_flying =-1 , old_queued =-1 , old_complete =-1 ; +bool b_dump; +char brkt[2 ] = {'<' ,'>' }; + +if (!ring) +return -EINVAL ; + +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (I915_SQS_IS_QUEUED (node)) + queued++; +else if (I915_SQS_IS_FLYING (node)) + flying++; +else if (I915_SQS_IS_COMPLETE (node)) + complete++; +else + other++; + } + + b_dump = (flying != old_flying) || + (queued != old_queued) || + (complete != old_complete); +if (scheduler->flags[ring->id] & i915_sf_dump_force) { +if (!b_dump) { + b_dump = true; + brkt[0 ] ='{' ; + brkt[1 ] ='}' ; + } + + scheduler->flags[ring->id] &= ~i915_sf_dump_force; + } + +if (b_dump) { + old_flying = flying; + old_queued = queued; + old_complete = complete; +DRM_DEBUG_DRIVER ("<%s> Q:%02d, F:%02d, C:%02d, O:%02d, " +"Flags = %s, Next = %d:%d %c%s%c\n" , + ring->name, queued, flying, complete, other, +i915_scheduler_flag_str (scheduler->flags[ring->id]), + dev_priv->request_uniq, dev_priv->next_seqno, + brkt[0 ], msg, brkt[1 ]); + }else { +/*DRM_DEBUG_DRIVER("<%s> Q:%02d, F:%02d, C:%02d, O:%02d" +", Flags = %s, Next = %d:%d [%s]\n", +ring->name, +queued, flying, complete, other, +i915_scheduler_flag_str(scheduler->flags[ring->id]), +dev_priv->request_uniq, dev_priv->next_seqno, msg); */ + +return 0 ; + } + +if (scheduler->flags[ring->id] & i915_sf_dump_details) { +int i, deps; +uint32_t count, counts[i915_sqs_MAX]; + +memset (counts,0x00 ,sizeof (counts)); + +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (node->status < i915_sqs_MAX) { + count = counts[node->status]++; + }else { +DRM_DEBUG_DRIVER ("<%s> Unknown status: %d!\n" , + ring->name, node->status); + count =-1 ; + } + + deps =0 ; +for (i =0 ; i < node->num_deps; i++) +if (i915_scheduler_is_dependency_valid (node, i)) + deps++; + +DRM_DEBUG_DRIVER ("<%s> %c:%02d> uniq = %d, seqno" +" = %d/%s, deps = %d / %d, fence = %p/%d, %s [pri = " +"%4d]\n" , ring->name, +i915_scheduler_queue_status_chr (node->status), + count, + node->params.request->uniq, + node->params.request->seqno, + node->params.ring->name, + deps, node->num_deps, + node->params.fence_wait, + node->params.fence_wait ?sync_fence_is_signaled (node->params.fence_wait) :0 , +i915_qe_state_str (node), + node->priority); + +if ((scheduler->flags[ring->id] & i915_sf_dump_dependencies) + ==0 ) +continue ; + +for (i =0 ; i < node->num_deps; i++) +if (node->dep_list[i]) +DRM_DEBUG_DRIVER ("<%s> |-%c:" +"%02d%c uniq = %d, seqno = %d/%s, %s [pri = %4d]\n" , + ring->name, +i915_scheduler_queue_status_chr (node->dep_list[i]->status), + i, +i915_scheduler_is_dependency_valid (node, i) + ?'>' :'#' , + node->dep_list[i]->params.request->uniq, + node->dep_list[i]->params.request->seqno, + node->dep_list[i]->params.ring->name, +i915_qe_state_str (node->dep_list[i]), + node->dep_list[i]->priority); + } + } + +return 0 ; +} +int i915_scheduler_flush_stamp (struct intel_engine_cs *ring,unsigned long target,bool is_locked)
@@ -1199,10 +1475,10 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)continue ;if (!I915_SQS_IS_COMPLETE (node)) -DRM_DEBUG_DRIVER ("Closing file handle with outstanding work: %d:%d/%d on %s\n" , +DRM_DEBUG_DRIVER ("Closing file handle with outstanding work: %d:%d/%s on %s\n" , node->params.request->uniq, node->params.request->seqno, - node->status, +i915_qe_state_str (node), ring->name);i915_scheduler_file_queue_dec (node->params.file);
drivers/gpu/drm/i915/i915_scheduler.h +14 -0
@@ -41,6 +41,9 @@ enum i915_scheduler_queue_status {/* Limit value for use with arrays/loops */ i915_sqs_MAX }; +char i915_scheduler_queue_status_chr (enum i915_scheduler_queue_status status); +const char *i915_scheduler_queue_status_str ( +enum i915_scheduler_queue_status status);#define I915_SQS_IS_QUEUED (node) (((node)->status == i915_sqs_queued))#define I915_SQS_IS_FLYING (node) (((node)->status == i915_sqs_flying))
@@ -71,6 +74,7 @@ struct i915_scheduler_queue_entry {struct list_head link;uint32_t flags; }; +const char *i915_qe_state_str (struct i915_scheduler_queue_entry *node);struct i915_scheduler {struct list_head node_queue[I915_NUM_RINGS ];
@@ -87,9 +91,16 @@ struct i915_scheduler {/* Flag bits for i915_scheduler::flags */ enum { +/* Internal state */ i915_sf_interrupts_enabled = (1 <<0 ), i915_sf_submitting = (1 <<1 ), + +/* Dump/debug flags */ + i915_sf_dump_force = (1 <<8 ), + i915_sf_dump_details = (1 <<9 ), + i915_sf_dump_dependencies = (1 <<10 ), }; +const char *i915_scheduler_flag_str (uint32_t flags);/* Options for 'scheduler_override' module parameter: */ enum {
@@ -109,6 +120,9 @@ void i915_gem_scheduler_work_handler(struct work_struct *work);int i915_scheduler_flush (struct intel_engine_cs *ring,bool is_locked);int i915_scheduler_flush_stamp (struct intel_engine_cs *ring,unsigned long stamp,bool is_locked); +int i915_scheduler_dump (struct intel_engine_cs *ring, +const char *msg); +int i915_scheduler_dump_all (struct drm_device *dev,const char *msg);bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req,bool *completed,bool *busy);bool i915_scheduler_file_queue_is_full (struct drm_file *file);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 31/39 drm/i915: Add early exit to execbuff_final() if insufficient ring space
In reply to [email protected]From: John Harrison <[email protected]>
One of the major purposes of the GPU scheduler is to avoid stalling the CPU when the GPU is busy and unable to accept more work. This change adds support to the ring submission code to allow a ring space check to be performed before attempting to submit a batch buffer to the hardware. If insufficient space is available then the scheduler can go away and come back later, letting the CPU get on with other work, rather than stalling and waiting for the hardware to catch up.
Change-Id: I267159ce1150cb6714d34a49b841bcbe4bf66326
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 42 ++++++++++++++++------ drivers/gpu/drm/i915/intel_lrc.c | 57 +++++++++++++++++++++++++++--- drivers/gpu/drm/i915/intel_ringbuffer.c | 24 +++++++++++++ drivers/gpu/drm/i915/intel_ringbuffer.h | 1 + 4 files changed, 109 insertions(+), 15 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +31 -11
@@ -1101,25 +1101,19 @@ i915_reset_gen7_sol_offsets(struct drm_device *dev, {struct intel_engine_cs *ring = req->ring;struct drm_i915_private *dev_priv = dev->dev_private; -int ret, i; +int i;if (!IS_GEN7 (dev) || ring != &dev_priv->ring[RCS ]) {DRM_DEBUG ("sol reset is gen7/rcs only\n" );return -EINVAL ; } - ret =intel_ring_begin (req,4 *3 ); -if (ret) -return ret; -for (i =0 ; i <4 ; i++) {intel_ring_emit (ring,MI_LOAD_REGISTER_IMM (1 ));intel_ring_emit (ring,GEN7_SO_WRITE_OFFSET (i));intel_ring_emit (ring,0 ); } -intel_ring_advance (ring); -return 0 ; }
@@ -1246,6 +1240,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)struct intel_engine_cs *ring = params->ring;u64 exec_start, exec_len;int ret; +uint32_t min_space;/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex));
@@ -1267,8 +1262,36 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)if (ret)return ret; +/* +* It would be a bad idea to run out of space while writing commands +* to the ring. One of the major aims of the scheduler is to not stall +* at any point for any reason. However, doing an early exit half way +* through submission could result in a partial sequence being written +* which would leave the engine in an unknown state. Therefore, check in +* advance that there will be enough space for the entire submission +* whether emitted by the code below OR by any other functions that may +* be executed before the end of final(). +* +* NB: This test deliberately overestimates, because that's easier than +* tracing every potential path that could be taken! +* +* Current measurements suggest that we may need to emit up to 744 bytes +* (186 dwords), so this is rounded up to 256 dwords here. Then we double +* that to get the free space requirement, because the block isn't allowed +* to span the transition from the end to the beginning of the ring. +*/ +#define I915_BATCH_EXEC_MAX_LEN 256/* max dwords emitted here */ + min_space =I915_BATCH_EXEC_MAX_LEN *2 *sizeof (uint32_t ); + ret =intel_ring_test_space (params->request->ringbuf, min_space); +if (ret) + goto early_error; +intel_runtime_pm_get (dev_priv); + ret =intel_ring_begin (params->request,I915_BATCH_EXEC_MAX_LEN ); +if (ret) + goto error; +/* * Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch.
@@ -1287,10 +1310,6 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) { - ret =intel_ring_begin (params->request,4 ); -if (ret) - goto error; -intel_ring_emit (ring,MI_NOOP );intel_ring_emit (ring,MI_LOAD_REGISTER_IMM (1 ));intel_ring_emit (ring,INSTPM );
@@ -1327,6 +1346,7 @@ error:*/ intel_runtime_pm_put (dev_priv); +early_error:if (ret)intel_ring_reserved_space_cancel (params->request->ringbuf);
drivers/gpu/drm/i915/intel_lrc.c +53 -4
81 lines not on this page · raw message
drivers/gpu/drm/i915/intel_ringbuffer.c +24 -0
@@ -2472,6 +2472,30 @@ int intel_ring_cacheline_align(struct drm_i915_gem_request *req)return 0 ; } +/* Test to see if the ring has sufficient space to submit a given piece of work +* without causing a stall */ +int intel_ring_test_space (struct intel_ringbuffer *ringbuf,int min_space) +{ +struct drm_i915_private *dev_priv = ringbuf->ring->dev->dev_private; + +/* There is a separate LRC version of this code. */ +BUG_ON (i915.enable_execlists); + +if (ringbuf->space < min_space) { +/* Need to update the actual ring space. Otherwise, the system +* hangs forever testing a software copy of the space value that +* never changes! +*/ + ringbuf->head =I915_READ_HEAD (ringbuf->ring); + ringbuf->space =intel_ring_space (ringbuf); + +if (ringbuf->space < min_space) +return -EAGAIN ; + } + +return 0 ; +} +void intel_ring_init_seqno (struct intel_engine_cs *ring,u32 seqno) {struct drm_device *dev = ring->dev;
drivers/gpu/drm/i915/intel_ringbuffer.h +1 -0
@@ -469,6 +469,7 @@ void intel_cleanup_ring_buffer(struct intel_engine_cs *ring);int intel_ring_alloc_request_extras (struct drm_i915_gem_request *request); +int intel_ring_test_space (struct intel_ringbuffer *ringbuf,int min_space);int __must_check intel_ring_begin (struct drm_i915_gem_request *req,int n);int __must_check intel_ring_cacheline_align (struct drm_i915_gem_request *req); static inline void intel_ring_emit(struct intel_engine_cs *ring,
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 32/40 drm/i915: Add early exit to execbuff_final() if insufficient ring space
From: John Harrison <[email protected]>
One of the major purposes of the GPU scheduler is to avoid stalling the CPU when the GPU is busy and unable to accept more work. This change adds support to the ring submission code to allow a ring space check to be performed before attempting to submit a batch buffer to the hardware. If insufficient space is available then the scheduler can go away and come back later, letting the CPU get on with other work, rather than stalling and waiting for the hardware to catch up.
v3: Updated to use locally cached request pointer.
Change-Id: I267159ce1150cb6714d34a49b841bcbe4bf66326
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 42 ++++++++++++++++------ drivers/gpu/drm/i915/intel_lrc.c | 57 +++++++++++++++++++++++++++--- drivers/gpu/drm/i915/intel_ringbuffer.c | 24 +++++++++++++ drivers/gpu/drm/i915/intel_ringbuffer.h | 1 + 4 files changed, 109 insertions(+), 15 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +31 -11
@@ -1101,25 +1101,19 @@ i915_reset_gen7_sol_offsets(struct drm_device *dev, {struct intel_engine_cs *ring = req->ring;struct drm_i915_private *dev_priv = dev->dev_private; -int ret, i; +int i;if (!IS_GEN7 (dev) || ring != &dev_priv->ring[RCS ]) {DRM_DEBUG ("sol reset is gen7/rcs only\n" );return -EINVAL ; } - ret =intel_ring_begin (req,4 *3 ); -if (ret) -return ret; -for (i =0 ; i <4 ; i++) {intel_ring_emit (ring,MI_LOAD_REGISTER_IMM (1 ));intel_ring_emit (ring,GEN7_SO_WRITE_OFFSET (i));intel_ring_emit (ring,0 ); } -intel_ring_advance (ring); -return 0 ; }
@@ -1247,6 +1241,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)struct intel_engine_cs *ring = params->ring;u64 exec_start, exec_len;int ret; +uint32_t min_space;/* The mutex must be acquired before calling this function */ BUG_ON (!mutex_is_locked (¶ms->dev->struct_mutex));
@@ -1268,8 +1263,36 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)if (ret)return ret; +/* +* It would be a bad idea to run out of space while writing commands +* to the ring. One of the major aims of the scheduler is to not stall +* at any point for any reason. However, doing an early exit half way +* through submission could result in a partial sequence being written +* which would leave the engine in an unknown state. Therefore, check in +* advance that there will be enough space for the entire submission +* whether emitted by the code below OR by any other functions that may +* be executed before the end of final(). +* +* NB: This test deliberately overestimates, because that's easier than +* tracing every potential path that could be taken! +* +* Current measurements suggest that we may need to emit up to 744 bytes +* (186 dwords), so this is rounded up to 256 dwords here. Then we double +* that to get the free space requirement, because the block isn't allowed +* to span the transition from the end to the beginning of the ring. +*/ +#define I915_BATCH_EXEC_MAX_LEN 256/* max dwords emitted here */ + min_space =I915_BATCH_EXEC_MAX_LEN *2 *sizeof (uint32_t ); + ret =intel_ring_test_space (req->ringbuf, min_space); +if (ret) + goto early_error; +intel_runtime_pm_get (dev_priv); + ret =intel_ring_begin (req,I915_BATCH_EXEC_MAX_LEN ); +if (ret) + goto error; +/* * Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch.
@@ -1288,10 +1311,6 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) { - ret =intel_ring_begin (req,4 ); -if (ret) - goto error; -intel_ring_emit (ring,MI_NOOP );intel_ring_emit (ring,MI_LOAD_REGISTER_IMM (1 ));intel_ring_emit (ring,INSTPM );
@@ -1328,6 +1347,7 @@ error:*/ intel_runtime_pm_put (dev_priv); +early_error:if (ret)intel_ring_reserved_space_cancel (req->ringbuf);
drivers/gpu/drm/i915/intel_lrc.c +53 -4
81 lines not on this page · raw message
drivers/gpu/drm/i915/intel_ringbuffer.c +24 -0
@@ -2472,6 +2472,30 @@ int intel_ring_cacheline_align(struct drm_i915_gem_request *req)return 0 ; } +/* Test to see if the ring has sufficient space to submit a given piece of work +* without causing a stall */ +int intel_ring_test_space (struct intel_ringbuffer *ringbuf,int min_space) +{ +struct drm_i915_private *dev_priv = ringbuf->ring->dev->dev_private; + +/* There is a separate LRC version of this code. */ +BUG_ON (i915.enable_execlists); + +if (ringbuf->space < min_space) { +/* Need to update the actual ring space. Otherwise, the system +* hangs forever testing a software copy of the space value that +* never changes! +*/ + ringbuf->head =I915_READ_HEAD (ringbuf->ring); + ringbuf->space =intel_ring_space (ringbuf); + +if (ringbuf->space < min_space) +return -EAGAIN ; + } + +return 0 ; +} +void intel_ring_init_seqno (struct intel_engine_cs *ring,u32 seqno) {struct drm_device *dev = ring->dev;
drivers/gpu/drm/i915/intel_ringbuffer.h +1 -0
@@ -469,6 +469,7 @@ void intel_cleanup_ring_buffer(struct intel_engine_cs *ring);int intel_ring_alloc_request_extras (struct drm_i915_gem_request *request); +int intel_ring_test_space (struct intel_ringbuffer *ringbuf,int min_space);int __must_check intel_ring_begin (struct drm_i915_gem_request *req,int n);int __must_check intel_ring_cacheline_align (struct drm_i915_gem_request *req); static inline void intel_ring_emit(struct intel_engine_cs *ring,
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 32/39 drm/i915: Added scheduler statistic reporting to debugfs
In reply to [email protected]From: John Harrison <[email protected]>
It is useful for know what the scheduler is doing for both debugging and performance analysis purposes. This change adds a bunch of counters and such that keep track of various scheduler operations (batches submitted, completed, flush requests, etc.). The data can then be read in userland via the debugfs mechanism.
v2: Updated to match changes to scheduler implementation.
Change-Id: I3266c631cd70c9eeb2c235f88f493e60462f85d7
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_debugfs.c | 77 ++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_gem_execbuffer.c | 11 +++- drivers/gpu/drm/i915/i915_scheduler.c | 87 +++++++++++++++++++++++++++--- drivers/gpu/drm/i915/i915_scheduler.h | 36 +++++++++++++ 4 files changed, 203 insertions(+), 8 deletions(-)
drivers/gpu/drm/i915/i915_debugfs.c +77 -0
@@ -3603,6 +3603,82 @@ static int i915_drrs_status(struct seq_file *m, void *unused)return 0 ; } +static int i915_scheduler_info (struct seq_file *m,void *unused) +{ +struct drm_info_node *node = (struct drm_info_node *) m->private; +struct drm_device *dev = node->minor->dev; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct i915_scheduler_stats *stats = scheduler->stats; +struct i915_scheduler_stats_nodes node_stats[I915_NUM_RINGS ]; +struct intel_engine_cs *ring; +char str[50 * (I915_NUM_RINGS +1 )], name[50 ], *ptr; +int ret, i, r; + + ret =mutex_lock_interruptible (&dev->mode_config.mutex); +if (ret) +return ret; + +#define PRINT_VAR (name, fmt, var) \ + do { \ + sprintf(str, "%-22s", name); \ + ptr = str + strlen(str); \ + for_each_ring(ring, dev_priv, r) { \ + sprintf(ptr, " %10" fmt, var); \ + ptr += strlen(ptr); \ + } \ + seq_printf(m, "%s\n", str); \ + } while (0) + +PRINT_VAR ("Ring name:" ,"s" , dev_priv->ring[r].name); +PRINT_VAR (" Ring seqno" ,"d" , ring->get_seqno (ring, false)); +seq_putc (m,'\n' ); + +seq_puts (m,"Batch submissions:\n" ); +PRINT_VAR (" Queued" ,"u" , stats[r].queued); +PRINT_VAR (" Submitted" ,"u" , stats[r].submitted); +PRINT_VAR (" Completed" ,"u" , stats[r].completed); +PRINT_VAR (" Expired" ,"u" , stats[r].expired); +seq_putc (m,'\n' ); + +seq_puts (m,"Flush counts:\n" ); +PRINT_VAR (" By object" ,"u" , stats[r].flush_obj); +PRINT_VAR (" By request" ,"u" , stats[r].flush_req); +PRINT_VAR (" By stamp" ,"u" , stats[r].flush_stamp); +PRINT_VAR (" Blanket" ,"u" , stats[r].flush_all); +PRINT_VAR (" Entries bumped" ,"u" , stats[r].flush_bump); +PRINT_VAR (" Entries submitted" ,"u" , stats[r].flush_submit); +seq_putc (m,'\n' ); + +seq_puts (m,"Miscellaneous:\n" ); +PRINT_VAR (" ExecEarly retry" ,"u" , stats[r].exec_early); +PRINT_VAR (" ExecFinal requeue" ,"u" , stats[r].exec_again); +PRINT_VAR (" ExecFinal killed" ,"u" , stats[r].exec_dead); +PRINT_VAR (" Fence wait" ,"u" , stats[r].fence_wait); +PRINT_VAR (" Fence wait again" ,"u" , stats[r].fence_again); +PRINT_VAR (" Fence wait ignore" ,"u" , stats[r].fence_ignore); +PRINT_VAR (" Fence supplied" ,"u" , stats[r].fence_got); +PRINT_VAR (" Hung flying" ,"u" , stats[r].kill_flying); +PRINT_VAR (" Hung queued" ,"u" , stats[r].kill_queued); +seq_putc (m,'\n' ); + +seq_puts (m,"Queue contents:\n" ); +for_each_ring (ring, dev_priv, i) +i915_scheduler_query_stats (ring, node_stats + ring->id); + +for (i =0 ; i < (i915_sqs_MAX +1 ); i++) { +sprintf (name," %s" ,i915_scheduler_queue_status_str (i)); +PRINT_VAR (name,"d" , node_stats[r].counts[i]); + } +seq_putc (m,'\n' ); + +#undef PRINT_VAR + +mutex_unlock (&dev->mode_config.mutex); + +return 0 ; +} +struct pipe_crc_info {const char *name;struct drm_device *dev;
@@ -5571,6 +5647,7 @@ static const struct drm_info_list i915_debugfs_list[] = { {"i915_semaphore_status", i915_semaphore_status, 0}, {"i915_shared_dplls_info", i915_shared_dplls_info, 0}, {"i915_dp_mst_info", i915_dp_mst_info, 0}, + {"i915_scheduler_info", i915_scheduler_info, 0}, {"i915_wa_registers", i915_wa_registers, 0}, {"i915_ddb_info", i915_ddb_info, 0}, {"i915_sseu_status", i915_sseu_status, 0},
drivers/gpu/drm/i915/i915_gem_execbuffer.c +10 -1
@@ -1422,13 +1422,20 @@ static int i915_early_fence_wait(struct intel_engine_cs *ring, int fence_fd) }if (!sync_fence_is_signaled (fence)) { +struct drm_i915_private *dev_priv = ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +/* * Wait forever for the fence to be signalled. This is safe * because the the mutex lock has not yet been acquired and * the wait is interruptible. */ -if (!i915_safe_to_ignore_fence (ring, fence)) +if (i915_safe_to_ignore_fence (ring, fence)) + scheduler->stats[ring->id].fence_ignore++; +else { + scheduler->stats[ring->id].fence_wait++; ret =sync_fence_wait (fence,-1 ); + } }sync_fence_put (fence);
@@ -1844,6 +1851,8 @@ err:mutex_unlock (&dev->struct_mutex); pre_mutex_err: + dev_priv->scheduler->stats[ring->id].exec_early++; +return ret; }
drivers/gpu/drm/i915/i915_scheduler.c +80 -7
248 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h +36 -0
@@ -75,6 +75,37 @@ struct i915_scheduler_queue_entry { };const char *i915_qe_state_str (struct i915_scheduler_queue_entry *node); +struct i915_scheduler_stats_nodes { +uint32_t counts[i915_sqs_MAX +1 ]; +}; + +struct i915_scheduler_stats { +/* Batch buffer counts: */ +uint32_t queued; +uint32_t submitted; +uint32_t completed; +uint32_t expired; + +/* Other stuff: */ +uint32_t flush_obj; +uint32_t flush_req; +uint32_t flush_stamp; +uint32_t flush_all; +uint32_t flush_bump; +uint32_t flush_submit; + +uint32_t exec_early; +uint32_t exec_again; +uint32_t exec_dead; +uint32_t kill_flying; +uint32_t kill_queued; + +uint32_t fence_wait; +uint32_t fence_again; +uint32_t fence_ignore; +uint32_t fence_got; +}; +struct i915_scheduler {struct list_head node_queue[I915_NUM_RINGS ];uint32_t flags[I915_NUM_RINGS ];
@@ -86,6 +117,9 @@ struct i915_scheduler {int32_t priority_level_preempt;uint32_t min_flying;uint32_t file_queue_max; + +/* Statistics: */ +struct i915_scheduler_stats stats[I915_NUM_RINGS ]; };/* Flag bits for i915_scheduler::flags */
@@ -126,6 +160,8 @@ int i915_scheduler_dump(struct intel_engine_cs *ring,int i915_scheduler_dump_all (struct drm_device *dev,const char *msg);bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req,bool *completed,bool *busy); +int i915_scheduler_query_stats (struct intel_engine_cs *ring, +struct i915_scheduler_stats_nodes *stats);bool i915_scheduler_file_queue_is_full (struct drm_file *file);#endif /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 33/40 drm/i915: Added scheduler statistic reporting to debugfs
From: John Harrison <[email protected]>
It is useful for know what the scheduler is doing for both debugging and performance analysis purposes. This change adds a bunch of counters and such that keep track of various scheduler operations (batches submitted, completed, flush requests, etc.). The data can then be read in userland via the debugfs mechanism.
v2: Updated to match changes to scheduler implementation.
v3: Updated for changes to kill code and flush code.
Change-Id: I3266c631cd70c9eeb2c235f88f493e60462f85d7
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_debugfs.c | 77 +++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_gem_execbuffer.c | 11 +++- drivers/gpu/drm/i915/i915_scheduler.c | 85 +++++++++++++++++++++++++++--- drivers/gpu/drm/i915/i915_scheduler.h | 36 +++++++++++++ 4 files changed, 200 insertions(+), 9 deletions(-)
drivers/gpu/drm/i915/i915_debugfs.c +77 -0
@@ -3603,6 +3603,82 @@ static int i915_drrs_status(struct seq_file *m, void *unused)return 0 ; } +static int i915_scheduler_info (struct seq_file *m,void *unused) +{ +struct drm_info_node *node = (struct drm_info_node *) m->private; +struct drm_device *dev = node->minor->dev; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct i915_scheduler_stats *stats = scheduler->stats; +struct i915_scheduler_stats_nodes node_stats[I915_NUM_RINGS ]; +struct intel_engine_cs *ring; +char str[50 * (I915_NUM_RINGS +1 )], name[50 ], *ptr; +int ret, i, r; + + ret =mutex_lock_interruptible (&dev->mode_config.mutex); +if (ret) +return ret; + +#define PRINT_VAR (name, fmt, var) \ + do { \ + sprintf(str, "%-22s", name); \ + ptr = str + strlen(str); \ + for_each_ring(ring, dev_priv, r) { \ + sprintf(ptr, " %10" fmt, var); \ + ptr += strlen(ptr); \ + } \ + seq_printf(m, "%s\n", str); \ + } while (0) + +PRINT_VAR ("Ring name:" ,"s" , dev_priv->ring[r].name); +PRINT_VAR (" Ring seqno" ,"d" , ring->get_seqno (ring, false)); +seq_putc (m,'\n' ); + +seq_puts (m,"Batch submissions:\n" ); +PRINT_VAR (" Queued" ,"u" , stats[r].queued); +PRINT_VAR (" Submitted" ,"u" , stats[r].submitted); +PRINT_VAR (" Completed" ,"u" , stats[r].completed); +PRINT_VAR (" Expired" ,"u" , stats[r].expired); +seq_putc (m,'\n' ); + +seq_puts (m,"Flush counts:\n" ); +PRINT_VAR (" By object" ,"u" , stats[r].flush_obj); +PRINT_VAR (" By request" ,"u" , stats[r].flush_req); +PRINT_VAR (" By stamp" ,"u" , stats[r].flush_stamp); +PRINT_VAR (" Blanket" ,"u" , stats[r].flush_all); +PRINT_VAR (" Entries bumped" ,"u" , stats[r].flush_bump); +PRINT_VAR (" Entries submitted" ,"u" , stats[r].flush_submit); +seq_putc (m,'\n' ); + +seq_puts (m,"Miscellaneous:\n" ); +PRINT_VAR (" ExecEarly retry" ,"u" , stats[r].exec_early); +PRINT_VAR (" ExecFinal requeue" ,"u" , stats[r].exec_again); +PRINT_VAR (" ExecFinal killed" ,"u" , stats[r].exec_dead); +PRINT_VAR (" Fence wait" ,"u" , stats[r].fence_wait); +PRINT_VAR (" Fence wait again" ,"u" , stats[r].fence_again); +PRINT_VAR (" Fence wait ignore" ,"u" , stats[r].fence_ignore); +PRINT_VAR (" Fence supplied" ,"u" , stats[r].fence_got); +PRINT_VAR (" Hung flying" ,"u" , stats[r].kill_flying); +PRINT_VAR (" Hung queued" ,"u" , stats[r].kill_queued); +seq_putc (m,'\n' ); + +seq_puts (m,"Queue contents:\n" ); +for_each_ring (ring, dev_priv, i) +i915_scheduler_query_stats (ring, node_stats + ring->id); + +for (i =0 ; i < (i915_sqs_MAX +1 ); i++) { +sprintf (name," %s" ,i915_scheduler_queue_status_str (i)); +PRINT_VAR (name,"d" , node_stats[r].counts[i]); + } +seq_putc (m,'\n' ); + +#undef PRINT_VAR + +mutex_unlock (&dev->mode_config.mutex); + +return 0 ; +} +struct pipe_crc_info {const char *name;struct drm_device *dev;
@@ -5571,6 +5647,7 @@ static const struct drm_info_list i915_debugfs_list[] = { {"i915_semaphore_status", i915_semaphore_status, 0}, {"i915_shared_dplls_info", i915_shared_dplls_info, 0}, {"i915_dp_mst_info", i915_dp_mst_info, 0}, + {"i915_scheduler_info", i915_scheduler_info, 0}, {"i915_wa_registers", i915_wa_registers, 0}, {"i915_ddb_info", i915_ddb_info, 0}, {"i915_sseu_status", i915_sseu_status, 0},
drivers/gpu/drm/i915/i915_gem_execbuffer.c +10 -1
@@ -1423,13 +1423,20 @@ static int i915_early_fence_wait(struct intel_engine_cs *ring, int fence_fd) }if (!sync_fence_is_signaled (fence)) { +struct drm_i915_private *dev_priv = ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +/* * Wait forever for the fence to be signalled. This is safe * because the the mutex lock has not yet been acquired and * the wait is interruptible. */ -if (!i915_safe_to_ignore_fence (ring, fence)) +if (i915_safe_to_ignore_fence (ring, fence)) + scheduler->stats[ring->id].fence_ignore++; +else { + scheduler->stats[ring->id].fence_wait++; ret =sync_fence_wait (fence,-1 ); + } }sync_fence_put (fence);
@@ -1849,6 +1856,8 @@ err:mutex_unlock (&dev->struct_mutex); pre_mutex_err: + dev_priv->scheduler->stats[ring->id].exec_early++; +return ret; }
drivers/gpu/drm/i915/i915_scheduler.c +77 -8
229 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h +36 -0
@@ -76,6 +76,37 @@ struct i915_scheduler_queue_entry { };const char *i915_qe_state_str (struct i915_scheduler_queue_entry *node); +struct i915_scheduler_stats_nodes { +uint32_t counts[i915_sqs_MAX +1 ]; +}; + +struct i915_scheduler_stats { +/* Batch buffer counts: */ +uint32_t queued; +uint32_t submitted; +uint32_t completed; +uint32_t expired; + +/* Other stuff: */ +uint32_t flush_obj; +uint32_t flush_req; +uint32_t flush_stamp; +uint32_t flush_all; +uint32_t flush_bump; +uint32_t flush_submit; + +uint32_t exec_early; +uint32_t exec_again; +uint32_t exec_dead; +uint32_t kill_flying; +uint32_t kill_queued; + +uint32_t fence_wait; +uint32_t fence_again; +uint32_t fence_ignore; +uint32_t fence_got; +}; +struct i915_scheduler {struct list_head node_queue[I915_NUM_RINGS ];uint32_t flags[I915_NUM_RINGS ];
@@ -87,6 +118,9 @@ struct i915_scheduler {int32_t priority_level_preempt;uint32_t min_flying;uint32_t file_queue_max; + +/* Statistics: */ +struct i915_scheduler_stats stats[I915_NUM_RINGS ]; };/* Flag bits for i915_scheduler::flags */
@@ -125,6 +159,8 @@ int i915_scheduler_dump(struct intel_engine_cs *ring,int i915_scheduler_dump_all (struct drm_device *dev,const char *msg);bool i915_scheduler_is_request_tracked (struct drm_i915_gem_request *req,bool *completed,bool *busy); +int i915_scheduler_query_stats (struct intel_engine_cs *ring, +struct i915_scheduler_stats_nodes *stats);bool i915_scheduler_file_queue_is_full (struct drm_file *file);#endif /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 33/39 drm/i915: Added seqno values to scheduler status dump
In reply to [email protected]From: John Harrison <[email protected]>
It is useful to be able to see what seqnos have actually popped out of the hardware when viewing the scheduler status.
Change-Id: Ie93e51c64328be2606b8b43440f6344d5f225426
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_scheduler.c | 10 ++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 1 + 2 files changed, 11 insertions(+)
drivers/gpu/drm/i915/i915_scheduler.c +10 -0
@@ -153,6 +153,7 @@ const char *i915_scheduler_flag_str(uint32_t flags)TEST_FLAG (i915_sf_dump_force,"DumpForce|" );TEST_FLAG (i915_sf_dump_details,"DumpDetails|" );TEST_FLAG (i915_sf_dump_dependencies,"DumpDeps|" ); +TEST_FLAG (i915_sf_dump_seqno,"DumpSeqno|" );#undef TEST_FLAG
@@ -760,6 +761,7 @@ static int i915_scheduler_dump_all_locked(struct drm_device *dev, const char *msfor_each_ring (ring, dev_priv, i) { scheduler->flags[ring->id] |= i915_sf_dump_force | i915_sf_dump_details | + i915_sf_dump_seqno | i915_sf_dump_dependencies; r =i915_scheduler_dump_locked (ring, msg);if (ret ==0 )
@@ -841,6 +843,14 @@ static int i915_scheduler_dump_locked(struct intel_engine_cs *ring, const char *return 0 ; } +if (scheduler->flags[ring->id] & i915_sf_dump_seqno) { +uint32_t seqno; + + seqno = ring->get_seqno (ring, true); + +DRM_DEBUG_DRIVER ("<%s> Seqno = %d\n" , ring->name, seqno); + } +if (scheduler->flags[ring->id] & i915_sf_dump_details) {int i, deps;uint32_t count, counts[i915_sqs_MAX];
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -132,6 +132,7 @@ enum { i915_sf_dump_force = (1 <<8 ), i915_sf_dump_details = (1 <<9 ), i915_sf_dump_dependencies = (1 <<10 ), + i915_sf_dump_seqno = (1 << 11), };const char *i915_scheduler_flag_str (uint32_t flags);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 34/39 drm/i915: Add scheduler support functions for TDR
In reply to [email protected]From: John Harrison <[email protected]>
Change-Id: I720463f01c4edd3579ce52e315a85e4d7874d7e5
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_scheduler.c | 31 +++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 1 + 2 files changed, 32 insertions(+)
drivers/gpu/drm/i915/i915_scheduler.c +31 -0
@@ -1627,6 +1627,37 @@ int i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)return 0 ; } +/* +* Used by TDR to distinguish hung rings (not moving but with work to do) +* from idle rings (not moving because there is nothing to do). +*/ +bool i915_scheduler_is_ring_flying (struct intel_engine_cs *ring) +{ +struct drm_i915_private *dev_priv = ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct i915_scheduler_queue_entry *node; +unsigned long flags; +bool found = false; + +/* With the scheduler in bypass mode, no information can be returned. */ +if (i915.scheduler_override & i915_so_direct_submit) { +return true; + } + +spin_lock_irqsave (&scheduler->lock, flags); + +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (I915_SQS_IS_FLYING (node)) { + found = true; +break ; + } + } + +spin_unlock_irqrestore (&scheduler->lock, flags); + +return found; +} +bool i915_scheduler_file_queue_is_full (struct drm_file *file) {struct drm_i915_file_private *file_priv = file->driver_priv;
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -150,6 +150,7 @@ void i915_gem_scheduler_clean_node(struct i915_scheduler_queue_entry *nodint i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe);bool i915_scheduler_notify_request (struct drm_i915_gem_request *req);void i915_scheduler_wakeup (struct drm_device *dev); +bool i915_scheduler_is_ring_flying (struct intel_engine_cs *ring);void i915_gem_scheduler_work_handler (struct work_struct *work);int i915_scheduler_flush (struct intel_engine_cs *ring,bool is_locked); int i915_scheduler_flush_stamp(struct intel_engine_cs *ring,
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 35/39 drm/i915: GPU priority bumping to prevent starvation
In reply to [email protected]From: John Harrison <[email protected]>
If a high priority task was to continuously submit batch buffers to the driver, it could starve out any lower priority task from getting any GPU time at all. To prevent this, the priority of a queued batch buffer is bumped each time it does not get submitted to the hardware.
v2: Updated for signed priority values.
Change-Id: I0319c7d2f306c61a283f03edda9b5d09a6d3b621
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_debugfs.c | 28 ++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.c | 14 ++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 1 + 3 files changed, 43 insertions(+)
drivers/gpu/drm/i915/i915_debugfs.c +28 -0
@@ -1178,6 +1178,33 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_max_fops, "%lld\n");static int +i915_scheduler_priority_bump_get (void *data,u64 *val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + *val = (u64 ) scheduler->priority_level_bump; +return 0 ; +} + +static int +i915_scheduler_priority_bump_set (void *data,u64 val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + scheduler->priority_level_bump = (u32 ) val; +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (i915_scheduler_priority_bump_fops, + i915_scheduler_priority_bump_get, + i915_scheduler_priority_bump_set, +"%lld\n" ); + +static int i915_scheduler_priority_preempt_get (void *data,u64 *val) {struct drm_device *dev = data;
@@ -5672,6 +5699,7 @@ static const struct i915_debugfs_files { {"i915_next_seqno", &i915_next_seqno_fops}, {"i915_scheduler_priority_min", &i915_scheduler_priority_min_fops}, {"i915_scheduler_priority_max", &i915_scheduler_priority_max_fops}, + {"i915_scheduler_priority_bump", &i915_scheduler_priority_bump_fops}, {"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops}, {"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops}, {"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops},
drivers/gpu/drm/i915/i915_scheduler.c +14 -0
@@ -191,6 +191,7 @@ int i915_scheduler_init(struct drm_device *dev)/* Default tuning values: */ scheduler->priority_level_min =-1023 ; scheduler->priority_level_max =1023 ; + scheduler->priority_level_bump =50 ; scheduler->priority_level_preempt =900 ; scheduler->min_flying =2 ; scheduler->file_queue_max =64 ;
@@ -1502,6 +1503,19 @@ static int i915_scheduler_submit(struct intel_engine_cs *ring, bool was_locked) ret =i915_scheduler_pop_from_queue_locked (ring, &node, &flags); }while (ret ==0 ); +/* +* Bump the priority of everything that was not submitted to prevent +* starvation of low priority tasks by a spamming high priority task. +*/ +i915_scheduler_priority_bump_clear (scheduler); +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (!I915_SQS_IS_QUEUED (node)) +continue ; + +i915_scheduler_priority_bump (scheduler, node, + scheduler->priority_level_bump); + } +spin_unlock_irqrestore (&scheduler->lock, flags);if (!was_locked)
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -114,6 +114,7 @@ struct i915_scheduler {/* Tuning parameters: */ int32_t priority_level_min;int32_t priority_level_max; +int32_t priority_level_bump;int32_t priority_level_preempt;uint32_t min_flying;uint32_t file_queue_max;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 36/39 drm/i915: Scheduler state dump via debugfs
In reply to [email protected]From: John Harrison <[email protected]>
Added a facility for triggering the scheduler state dump via a debugfs entry.
v2: New patch in series.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_debugfs.c | 33 +++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.c | 9 +++++---- drivers/gpu/drm/i915/i915_scheduler.h | 6 ++++++ 3 files changed, 44 insertions(+), 4 deletions(-)
drivers/gpu/drm/i915/i915_debugfs.c +33 -0
@@ -1285,6 +1285,38 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_file_queue_max_fops, i915_scheduler_file_queue_max_set, "%llu\n"); +static int +i915_scheduler_dump_flags_get (void *data,u64 *val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + *val = scheduler->dump_flags; + +return 0 ; +} + +static int +i915_scheduler_dump_flags_set (void *data,u64 val) +{ +struct drm_device *dev = data; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + + scheduler->dump_flags =lower_32_bits (val) & i915_sf_dump_mask; + +if (val &1 ) +i915_scheduler_dump_all (dev,"DebugFS" ); + +return 0 ; +} + +DEFINE_SIMPLE_ATTRIBUTE (i915_scheduler_dump_flags_fops, + i915_scheduler_dump_flags_get, + i915_scheduler_dump_flags_set, +"0x%llx\n" ); +static int i915_frequency_info (struct seq_file *m,void *unused) {struct drm_info_node *node = m->private;
@@ -5703,6 +5735,7 @@ static const struct i915_debugfs_files { {"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops}, {"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops}, {"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops}, + {"i915_scheduler_dump_flags", &i915_scheduler_dump_flags_fops}, {"i915_display_crc_ctl", &i915_display_crc_ctl_fops}, {"i915_pri_wm_latency", &i915_pri_wm_latency_fops}, {"i915_spr_wm_latency", &i915_spr_wm_latency_fops},
drivers/gpu/drm/i915/i915_scheduler.c +5 -4
@@ -195,6 +195,10 @@ int i915_scheduler_init(struct drm_device *dev) scheduler->priority_level_preempt =900 ; scheduler->min_flying =2 ; scheduler->file_queue_max =64 ; + scheduler->dump_flags = i915_sf_dump_force | + i915_sf_dump_details | + i915_sf_dump_seqno | + i915_sf_dump_dependencies; dev_priv->scheduler = scheduler;
@@ -760,10 +764,7 @@ static int i915_scheduler_dump_all_locked(struct drm_device *dev, const char *msint i, r, ret =0 ;for_each_ring (ring, dev_priv, i) { - scheduler->flags[ring->id] |= i915_sf_dump_force | - i915_sf_dump_details | - i915_sf_dump_seqno | - i915_sf_dump_dependencies; + scheduler->flags[ring->id] |= scheduler->dump_flags & i915_sf_dump_mask; r =i915_scheduler_dump_locked (ring, msg);if (ret ==0 ) ret = r;
drivers/gpu/drm/i915/i915_scheduler.h +6 -0
@@ -118,6 +118,7 @@ struct i915_scheduler {int32_t priority_level_preempt;uint32_t min_flying;uint32_t file_queue_max; +uint32_t dump_flags;/* Statistics: */ struct i915_scheduler_stats stats[I915_NUM_RINGS ];
@@ -134,6 +135,11 @@ enum { i915_sf_dump_details = (1 <<9 ), i915_sf_dump_dependencies = (1 <<10 ), i915_sf_dump_seqno = (1 <<11 ), + + i915_sf_dump_mask = i915_sf_dump_force | + i915_sf_dump_details | + i915_sf_dump_dependencies | + i915_sf_dump_seqno, };const char *i915_scheduler_flag_str (uint32_t flags);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 37/39 drm/i915: Enable GPU scheduler by default
In reply to [email protected]From: John Harrison <[email protected]>
Now that all the scheduler patches have been applied, it is safe to enable.
Change-Id: I128042e85a30fca765ce1eb46c837c62dee66089
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_params.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_params.c +2 -2
@@ -54,7 +54,7 @@ struct i915_params i915 __read_mostly = { .edp_vswing = 0, .enable_guc_submission = true, .guc_log_level = -1, - .scheduler_override = 1, + .scheduler_override = 0, };module_param_named (modeset, i915.modeset, int,0400 );
@@ -194,4 +194,4 @@ MODULE_PARM_DESC(guc_log_level, "GuC firmware logging level (-1:disabled (default), 0-3:enabled)");module_param_named (scheduler_override, i915.scheduler_override, int,0600 ); -MODULE_PARM_DESC (scheduler_override,"Scheduler override mask (0 = none, 1 = direct submission [default])" ); +MODULE_PARM_DESC (scheduler_override,"Scheduler override mask (default: 0)" );
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 38/39 drm/i915: Add scheduling priority to per-context parameters
In reply to [email protected]From: Dave Gordon <[email protected]>
Added an interface for user land applications/libraries/services to set their GPU scheduler priority. This extends the existing context parameter IOCTL interface to add a scheduler priority parameter. The range is +/-1023 with +ve numbers meaning higher priority. Only system processes may set a higher priority than the default (zero), normal applications may only lower theirs.
v2: New patch in series.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 14 ++++++++++++++ drivers/gpu/drm/i915/i915_gem_context.c | 24 ++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_gem_execbuffer.c | 3 +++ include/uapi/drm/i915_drm.h | 1 + 4 files changed, 42 insertions(+)
drivers/gpu/drm/i915/i915_drv.h +14 -0
@@ -843,6 +843,19 @@ struct i915_ctx_hang_stats {bool banned; }; +/* +* User-settable GFX scheduler priorities are on a scale of -1023 (I don't +* care about running) to +1023 (I'm the most important thing in existence) +* with zero being the default. Any process may decrease its scheduling +* priority, but only a sufficiently privileged process may increase it +* beyond zero. +*/ + +struct i915_ctx_sched_info { +/* Scheduling priority */ +int32_t priority; +}; +struct i915_fence_timeline {char name[32 ];unsigned fence_context;
@@ -883,6 +896,7 @@ struct intel_context {int flags;struct drm_i915_file_private *file_priv;struct i915_ctx_hang_stats hang_stats; +struct i915_ctx_sched_info sched_info;struct i915_hw_ppgtt *ppgtt;/* Legacy ring buffer submission */
drivers/gpu/drm/i915/i915_gem_context.c +24 -0
@@ -946,6 +946,9 @@ int i915_gem_context_getparam_ioctl(struct drm_device *dev, void *data,else args->value =to_i915 (dev)->gtt.base.total;break ; +case I915_CONTEXT_PARAM_PRIORITY : + args->value = (__u64 ) ctx->sched_info.priority; +break ;default : ret = -EINVAL ;break ;
@@ -983,6 +986,7 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,else ctx->hang_stats.ban_period_seconds = args->value;break ; +case I915_CONTEXT_PARAM_NO_ZEROMAP :if (args->size) { ret = -EINVAL ;
@@ -991,6 +995,26 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data, ctx->flags |= args->value ?CONTEXT_NO_ZEROMAP :0 ; }break ; + +case I915_CONTEXT_PARAM_PRIORITY : + { +int32_t priority = (int32_t ) args->value; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; + +if (args->size) + ret = -EINVAL ; +else if ((priority > scheduler->priority_level_max) || + (priority < scheduler->priority_level_min)) + ret = -EINVAL ; +else if ((priority >0 ) && + !capable (CAP_SYS_ADMIN )) + ret = -EPERM ; +else + ctx->sched_info.priority = priority; +break ; + } +default : ret = -EINVAL ;break ;
drivers/gpu/drm/i915/i915_gem_execbuffer.c +3 -0
@@ -1725,6 +1725,9 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, params->args_DR4 = args->DR4; params->batch_obj = batch_obj; +/* Start with the context's priority level */ + qe.priority = ctx->sched_info.priority; +/* * Save away the list of objects used by this batch buffer for the * purpose of tracking inter-buffer dependencies.
include/uapi/drm/i915_drm.h +1 -0
@@ -1138,6 +1138,7 @@ struct drm_i915_gem_context_param {#define I915_CONTEXT_PARAM_BAN_PERIOD 0x1#define I915_CONTEXT_PARAM_NO_ZEROMAP 0x2#define I915_CONTEXT_PARAM_GTT_SIZE 0x3 +#define I915_CONTEXT_PARAM_PRIORITY 0x4__u64 value; };
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 39/39 drm/i915: Allow scheduler to manage inter-ring object synchronisation
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler has always tracked batch buffer dependencies based on DRM object usage. This means that it will not submit a batch on one ring that has outstanding dependencies still executing on other rings. This is exactly the same synchronisation performed by i915_gem_object_sync() using hardware semaphores where available and CPU stalls where not (e.g. in execlist mode and/or on Gen8 hardware).
Unfortunately, when a batch buffer is submitted to the driver the _object_sync() call happens first. Thus in case where hardware semaphores are disabled, the driver has already stalled until the dependency has been resolved.
This patch adds an optimisation to _object_sync() to ignore the synchronisation in the case where it will subsequently be handled by the scheduler. This removes the driver stall and (in the single application case) provides near hardware semaphore performance even when hardware semaphores are disabled. In a busy system where there is other work that can be executed on the stalling ring, it provides better than hardware semaphore performance as it removes the stall from both the driver and from the hardware. There is also a theory that this method should improve power usage as hardware semaphores are apparently not very power efficient - the stalled ring does not go into as low a power a state as when it is genuinely idle.
The optimisation is to check whether both ends of the synchronisation are batch buffer requests. If they are, then the scheduler will have the inter-dependency tracked and managed. If one or other end is not a batch buffer request (e.g. a page flip) then the code falls back to the CPU stall or hardware semaphore as appropriate.
To check whether the existing usage is a batch buffer, the code simply calls the 'are you tracking this request' function of the scheduler on the object's last_read_req member. To check whether the new usage is a batch buffer, a flag is passed in from the caller.
Change-Id: Idc16e19b5a4dc8b3782ce9db44dd3df445f396c1
Issue: VIZ-5566
--- drivers/gpu/drm/i915/i915_drv.h | 2 +- drivers/gpu/drm/i915/i915_gem.c | 17 ++++++++++++++--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 2 +- drivers/gpu/drm/i915/intel_display.c | 2 +- drivers/gpu/drm/i915/intel_lrc.c | 2 +- 5 files changed, 18 insertions(+), 7 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +1 -1
@@ -2971,7 +2971,7 @@ int __must_check i915_mutex_lock_interruptible(struct drm_device *dev);#endif int i915_gem_object_sync (struct drm_i915_gem_object *obj,struct intel_engine_cs *to, -struct drm_i915_gem_request **to_req); +struct drm_i915_gem_request **to_req,bool to_batch);void i915_vma_move_to_active (struct i915_vma *vma,struct drm_i915_gem_request *req); int i915_gem_dumb_create(struct drm_file *file_priv,
drivers/gpu/drm/i915/i915_gem.c +14 -3
@@ -3555,7 +3555,7 @@ static int __i915_gem_object_sync(struct drm_i915_gem_object *obj, struct intel_engine_cs *to, struct drm_i915_gem_request *from_req, - struct drm_i915_gem_request **to_req) + struct drm_i915_gem_request **to_req, bool to_batch) { struct intel_engine_cs *from;int ret;
@@ -3567,6 +3567,15 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,if (i915_gem_request_completed (from_req))return 0 ; +/* +* The scheduler will manage inter-ring object dependencies +* as long as both to and from requests are scheduler managed +* (i.e. batch buffers). +*/ +if (to_batch && +i915_scheduler_is_request_tracked (from_req,NULL ,NULL )) +return 0 ; +if (!i915_semaphore_is_enabled (obj->base.dev)) {struct drm_i915_private *i915 =to_i915 (obj->base.dev); ret = __i915_wait_request(from_req,
@@ -3617,6 +3626,8 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,* @to_req: request we wish to use the object for. See below. * This will be allocated and returned if a request is * required but not passed in. +* @to_batch: is the sync request on behalf of batch buffer submission? +* If so then the scheduler can (potentially) manage the synchronisation. * * This code is meant to abstract object synchronization with the GPU. * Calling with NULL implies synchronizing the object with the CPU
@@ -3647,7 +3658,7 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,int i915_gem_object_sync (struct drm_i915_gem_object *obj,struct intel_engine_cs *to, -struct drm_i915_gem_request **to_req) +struct drm_i915_gem_request **to_req,bool to_batch) {const bool readonly = obj->base.pending_write_domain ==0 ;struct drm_i915_gem_request *req[I915_NUM_RINGS ];
@@ -3669,7 +3680,7 @@ i915_gem_object_sync(struct drm_i915_gem_object *obj, req[n++] = obj->last_read_req[i]; }for (i =0 ; i < n; i++) { - ret =__i915_gem_object_sync (obj, to, req[i], to_req); + ret =__i915_gem_object_sync (obj, to, req[i], to_req, to_batch);if (ret)return ret; }
drivers/gpu/drm/i915/i915_gem_execbuffer.c +1 -1
@@ -919,7 +919,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,struct drm_i915_gem_object *obj = vma->obj;if (obj->active & other_rings) { - ret =i915_gem_object_sync (obj, req->ring, &req); + ret =i915_gem_object_sync (obj, req->ring, &req, true);if (ret)return ret; }
drivers/gpu/drm/i915/intel_display.c +1 -1
@@ -11487,7 +11487,7 @@ static int intel_crtc_page_flip(struct drm_crtc *crtc,* into the display plane and skip any waits. */ if (!mmio_flip) { - ret =i915_gem_object_sync (obj, ring, &request); + ret =i915_gem_object_sync (obj, ring, &request, false);if (ret) goto cleanup_pending; }
drivers/gpu/drm/i915/intel_lrc.c +1 -1
@@ -658,7 +658,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,struct drm_i915_gem_object *obj = vma->obj;if (obj->active & other_rings) { - ret =i915_gem_object_sync (obj, req->ring, &req); + ret =i915_gem_object_sync (obj, req->ring, &req, true);if (ret)return ret; }
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH 6/40 drm/i915: Cache request pointer in *_submission_final()
In reply to [email protected]From: Dave Gordon <[email protected]>
Keep a local copy of the request pointer in the _final() functions rather than dereferencing the params block repeatedly.
v3: New patch in series.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++------ drivers/gpu/drm/i915/intel_lrc.c | 11 ++++++----- 2 files changed, 13 insertions(+), 11 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +7 -6
@@ -1245,6 +1245,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,int i915_gem_ringbuffer_submission_final (struct i915_execbuffer_params *params) {struct drm_i915_private *dev_priv = params->dev->dev_private; +struct drm_i915_gem_request *req = params->request;struct intel_engine_cs *ring = params->ring;u64 exec_start, exec_len;int ret;
@@ -1258,12 +1259,12 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)* Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch. */ - ret =intel_ring_invalidate_all_caches (params->request); + ret =intel_ring_invalidate_all_caches (req);if (ret) goto error;/* Switch to the correct context for the batch */ - ret =i915_switch_context (params->request); + ret =i915_switch_context (req);if (ret) goto error;
@@ -1272,7 +1273,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) { - ret =intel_ring_begin (params->request,4 ); + ret =intel_ring_begin (req,4 );if (ret) goto error;
@@ -1286,7 +1287,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params) }if (params->args_flags &I915_EXEC_GEN7_SOL_RESET ) { - ret =i915_reset_gen7_sol_offsets (params->dev, params->request); + ret =i915_reset_gen7_sol_offsets (params->dev, req);if (ret) goto error; }
@@ -1295,13 +1296,13 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params) exec_start = params->batch_obj_vm_offset + params->args_batch_start_offset; - ret = ring->dispatch_execbuffer (params->request, + ret = ring->dispatch_execbuffer (req, exec_start, exec_len, params->dispatch_flags);if (ret) goto error; -trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); +trace_i915_gem_ring_dispatch (req, params->dispatch_flags);i915_gem_execbuffer_retire_commands (params);
drivers/gpu/drm/i915/intel_lrc.c +6 -5
@@ -929,7 +929,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,int intel_execlists_submission_final (struct i915_execbuffer_params *params) {struct drm_i915_private *dev_priv = params->dev->dev_private; -struct intel_ringbuffer *ringbuf = params->request->ringbuf; +struct drm_i915_gem_request *req = params->request; +struct intel_ringbuffer *ringbuf = req->ringbuf;struct intel_engine_cs *ring = params->ring;u64 exec_start;int ret;
@@ -941,13 +942,13 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)* Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch. */ - ret =logical_ring_invalidate_all_caches (params->request); + ret =logical_ring_invalidate_all_caches (req);if (ret)return ret;if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) { - ret =intel_logical_ring_begin (params->request,4 ); + ret =intel_logical_ring_begin (req,4 );if (ret)return ret;
@@ -963,11 +964,11 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params) exec_start = params->batch_obj_vm_offset + params->args_batch_start_offset; - ret = ring->emit_bb_start (params->request, exec_start, params->dispatch_flags); + ret = ring->emit_bb_start (req, exec_start, params->dispatch_flags);if (ret)return ret; -trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); +trace_i915_gem_ring_dispatch (req, params->dispatch_flags);i915_gem_execbuffer_retire_commands (params);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
cover GPU scheduler for i915 driver
In reply to [email protected]From: John Harrison <[email protected]>
Implemented a batch buffer submission scheduler for the i915 DRM driver.
The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code assigns a unique seqno value and then packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.
The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.
As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, the seqno values of the individual buffers are not necessary incrementing as the scheduler may have re-ordered their submission. However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.
A deferred work queue is also poked by the interrupt handler. When this wakes up it can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.
When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.
If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate.
The scheduler also allows high priority batch buffers (e.g. from a desktop compositor) to jump ahead of whatever is already running if the underlying hardware supports pre-emption. In this situation, any work that was pre-empted is returned to the queued list ready to be resubmitted when no more high priority work is outstanding.
Various IGT tests are in progress to test the scheduler's operation and will follow.
v2: Updated for changes in struct fence patch series and other changes to underlying tree (e.g. removal of cliprects). Also changed priority levels to be signed +/-1023 range and reduced mutex lock usage.
v3: More reuse of cached pointers rather than repeated dereferencing (David Gordon).
Moved the dependency generation code out to a seperate function for easier readability. Also added in support for the read-read optimisation.
Major simplification of the DRM file close handler.
Fixed up an overzealous WARN.
Removed unnecessary flushing of the scheduler queue when waiting for a request.
[Patches against drm-intel-nightly tree fetched 17/11/2015 with struct fence conversion patches applied]
Dave Gordon (3):
drm/i915: Updating assorted register and status page definitions drm/i915: Cache request pointer in *_submission_final() drm/i915: Add scheduling priority to per-context parameters
John Harrison (37):
drm/i915: Add total count to context status debugfs output drm/i915: Explicit power enable during deferred context initialisation drm/i915: Prelude to splitting i915_gem_do_execbuffer in two drm/i915: Split i915_dem_do_execbuffer() in half
drm/i915: Re-instate request->uniq because it is extremely useful drm/i915: Start of GPU scheduler
drm/i915: Prepare retire_requests to handle out-of-order seqnos drm/i915: Disable hardware semaphores when GPU scheduler is enabled drm/i915: Force MMIO flips when scheduler enabled
drm/i915: Added scheduler hook when closing DRM file handles drm/i915: Added scheduler hook into i915_gem_request_notify() drm/i915: Added deferred work handler for scheduler
drm/i915: Redirect execbuffer_final() via scheduler
drm/i915: Keep the reserved space mechanism happy
drm/i915: Added tracking/locking of batch buffer objects drm/i915: Hook scheduler node clean up into retire requests drm/i915: Added scheduler support to __wait_request() calls drm/i915: Added scheduler support to page fault handler drm/i915: Added scheduler flush calls to ring throttle and idle functions drm/i915: Added a module parameter for allowing scheduler overrides drm/i915: Support for 'unflushed' ring idle
drm/i915: Defer seqno allocation until actual hardware submission time drm/i915: Added immediate submission override to scheduler drm/i915: Add sync wait support to scheduler
drm/i915: Connecting execbuff fences to scheduler
drm/i915: Added trace points to scheduler
drm/i915: Added scheduler queue throttling by DRM file handle drm/i915: Added debugfs interface to scheduler tuning parameters drm/i915: Added debug state dump facilities to scheduler drm/i915: Add early exit to execbuff_final() if insufficient ring space drm/i915: Added scheduler statistic reporting to debugfs drm/i915: Added seqno values to scheduler status dump drm/i915: Add scheduler support functions for TDR
drm/i915: GPU priority bumping to prevent starvation drm/i915: Scheduler state dump via debugfs
drm/i915: Enable GPU scheduler by default
drm/i915: Allow scheduler to manage inter-ring object synchronisation
drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/i915_debugfs.c | 283 +++++ drivers/gpu/drm/i915/i915_dma.c | 6 + drivers/gpu/drm/i915/i915_drv.c | 9 + drivers/gpu/drm/i915/i915_drv.h | 56 +- drivers/gpu/drm/i915/i915_gem.c | 170 ++- drivers/gpu/drm/i915/i915_gem_context.c | 24 + drivers/gpu/drm/i915/i915_gem_execbuffer.c | 350 ++++-- drivers/gpu/drm/i915/i915_params.c | 4 + drivers/gpu/drm/i915/i915_reg.h | 30 +- drivers/gpu/drm/i915/i915_scheduler.c | 1640 ++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 175 +++ drivers/gpu/drm/i915/i915_trace.h | 215 +++- drivers/gpu/drm/i915/intel_display.c | 10 +- drivers/gpu/drm/i915/intel_lrc.c | 166 ++- drivers/gpu/drm/i915/intel_lrc.h | 1 + drivers/gpu/drm/i915/intel_ringbuffer.c | 47 +- drivers/gpu/drm/i915/intel_ringbuffer.h | 35 +- include/uapi/drm/i915_drm.h | 1 + 19 files changed, 3057 insertions(+), 166 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
v4 cover GPU scheduler for i915 driver
In reply to [email protected]From: John Harrison <[email protected]>
Implemented a batch buffer submission scheduler for the i915 DRM driver.
The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code assigns a unique seqno value and then packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.
The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.
As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, the seqno values of the individual buffers are not necessary incrementing as the scheduler may have re-ordered their submission. However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.
A deferred work queue is also poked by the interrupt handler. When this wakes up it can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.
When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.
If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate.
The scheduler also allows high priority batch buffers (e.g. from a desktop compositor) to jump ahead of whatever is already running if the underlying hardware supports pre-emption. In this situation, any work that was pre-empted is returned to the queued list ready to be resubmitted when no more high priority work is outstanding.
Various IGT tests are in progress to test the scheduler's operation and will follow.
v2: Updated for changes in struct fence patch series and other changes to underlying tree (e.g. removal of cliprects). Also changed priority levels to be signed +/-1023 range and reduced mutex lock usage.
v3: More reuse of cached pointers rather than repeated dereferencing (David Gordon).
Moved the dependency generation code out to a seperate function for easier readability. Also added in support for the read-read optimisation.
Major simplification of the DRM file close handler.
Fixed up an overzealous WARN.
Removed unnecessary flushing of the scheduler queue when waiting for a request.
v4: Removed user land fence/sync integration as this is dependent upon the de-staging of the Android sync code. That de-staging is now being done by someone else. The sync support will be added back in to the scheduler in a separate patch series which must wait for the de-staging to be landed first.
Added support for killing batches from contexts that were banned after the batches were submitted to the scheduler.
Changed various comments to fix typos, update to reflect changes to the code, correct formatting and line wrapping, etc. Also wrapped various long lines and twiddled white space to keep the style checker happy.
Changed a bunch of BUG_ONs to WARN_ONs as apparently the latter are preferred.
Used the correct array memory allocator function (kmalloc_array instead of kmalloc).
Fixed a variable type (review comment by Joonas).
Fixed a WARN_ON firing incorrectly when removing killed nodes from the scheduler's queue.
Dropped register definition update patch from this series. The changes are all for pre-emption so it makes more sense for it to be part of that series instead.
[Patches against drm-intel-nightly tree fetched 17/11/2015 with struct fence conversion patches applied]
Dave Gordon (2):
drm/i915: Cache request pointer in *_submission_final() drm/i915: Add scheduling priority to per-context parameters
John Harrison (36):
drm/i915: Add total count to context status debugfs output drm/i915: Explicit power enable during deferred context initialisation drm/i915: Prelude to splitting i915_gem_do_execbuffer in two drm/i915: Split i915_dem_do_execbuffer() in half
drm/i915: Re-instate request->uniq because it is extremely useful drm/i915: Start of GPU scheduler
drm/i915: Prepare retire_requests to handle out-of-order seqnos drm/i915: Disable hardware semaphores when GPU scheduler is enabled drm/i915: Force MMIO flips when scheduler enabled
drm/i915: Added scheduler hook when closing DRM file handles drm/i915: Added scheduler hook into i915_gem_request_notify() drm/i915: Added deferred work handler for scheduler
drm/i915: Redirect execbuffer_final() via scheduler
drm/i915: Keep the reserved space mechanism happy
drm/i915: Added tracking/locking of batch buffer objects drm/i915: Hook scheduler node clean up into retire requests drm/i915: Added scheduler support to __wait_request() calls drm/i915: Added scheduler support to page fault handler drm/i915: Added scheduler flush calls to ring throttle and idle functions drm/i915: Added a module parameter for allowing scheduler overrides drm/i915: Support for 'unflushed' ring idle
drm/i915: Defer seqno allocation until actual hardware submission time drm/i915: Added immediate submission override to scheduler drm/i915: Added trace points to scheduler
drm/i915: Added scheduler queue throttling by DRM file handle drm/i915: Added debugfs interface to scheduler tuning parameters drm/i915: Added debug state dump facilities to scheduler drm/i915: Add early exit to execbuff_final() if insufficient ring space drm/i915: Added scheduler statistic reporting to debugfs drm/i915: Added seqno values to scheduler status dump drm/i915: Add scheduler support functions for TDR
drm/i915: GPU priority bumping to prevent starvation drm/i915: Scheduler state dump via debugfs
drm/i915: Enable GPU scheduler by default
drm/i915: Add support for retro-actively banning batch buffers drm/i915: Allow scheduler to manage inter-ring object synchronisation
drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/i915_debugfs.c | 280 ++++- drivers/gpu/drm/i915/i915_dma.c | 6 + drivers/gpu/drm/i915/i915_drv.c | 9 + drivers/gpu/drm/i915/i915_drv.h | 55 +- drivers/gpu/drm/i915/i915_gem.c | 177 +++- drivers/gpu/drm/i915/i915_gem_context.c | 24 + drivers/gpu/drm/i915/i915_gem_execbuffer.c | 301 ++++-- drivers/gpu/drm/i915/i915_params.c | 4 + drivers/gpu/drm/i915/i915_scheduler.c | 1530 ++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 164 +++ drivers/gpu/drm/i915/i915_trace.h | 218 +++- drivers/gpu/drm/i915/intel_display.c | 10 +- drivers/gpu/drm/i915/intel_lrc.c | 170 +++- drivers/gpu/drm/i915/intel_lrc.h | 1 + drivers/gpu/drm/i915/intel_ringbuffer.c | 49 +- drivers/gpu/drm/i915/intel_ringbuffer.h | 3 +- include/uapi/drm/i915_drm.h | 1 + 18 files changed, 2856 insertions(+), 147 deletions(-) create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 1/38 drm/i915: Add total count to context status debugfs output
From: John Harrison <[email protected]>
When there are lots and lots and even more lots of contexts (e.g. when running with execlists) it is useful to be able to immediately see what the total context count is.
v4: Re-typed a variable (review feedback from Joonas)
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_debugfs.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_debugfs.c +4 -1
@@ -1953,7 +1953,7 @@ static int i915_context_status(struct seq_file *m, void *unused)struct drm_i915_private *dev_priv = dev->dev_private;struct intel_engine_cs *ring;struct intel_context *ctx; -int ret, i; +int ret, i, count =0 ; ret =mutex_lock_interruptible (&dev->struct_mutex);if (ret)
@@ -1966,6 +1966,7 @@ static int i915_context_status(struct seq_file *m, void *unused)seq_puts (m,"HW context " );describe_ctx (m, ctx); + count++;for_each_ring (ring, dev_priv, i) {if (ring->default_context == ctx) seq_printf(m, "(default context %s) ",
@@ -1994,6 +1995,8 @@ static int i915_context_status(struct seq_file *m, void *unused)seq_putc (m,'\n' ); } +seq_printf (m,"Total: %d contexts\n" , count); +mutex_unlock (&dev->struct_mutex);return 0 ;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 2/38 drm/i915: Explicit power enable during deferred context initialisation
In reply to [email protected]From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware submission part in the back half.
While that is a good thing in general, it causes a problem with the deferred context initialisation. That is done quite early on in the execbuf code path - it happens at context validation time rather than context switch time. Some of the deferred work requires the power to be enabled. Hence this patch adds an explicit power reference count to the deferred initialisation code itself.
Issue: GMINL-5159
--- drivers/gpu/drm/i915/intel_lrc.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/intel_lrc.c +8 -1
@@ -2470,6 +2470,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx,WARN_ON (ctx->legacy_hw_ctx.rcs_state !=NULL );WARN_ON (ctx->engine[ring->id].state); +intel_runtime_pm_get (dev->dev_private); + context_size =round_up (get_lr_context_size (ring),4096 );/* One extra page as the sharing data between driver and GuC */
@@ -2478,7 +2480,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx, ctx_obj =i915_gem_alloc_object (dev, context_size);if (!ctx_obj) {DRM_DEBUG_DRIVER ("Alloc LRC backing obj failed.\n" ); -return -ENOMEM ; + ret = -ENOMEM ; + goto error_pm; } ringbuf =intel_engine_create_ringbuffer (ring,4 *PAGE_SIZE );
@@ -2524,6 +2527,8 @@ int intel_lr_context_deferred_alloc(struct intel_context *ctx, }i915_add_request_no_flush (req); } + +intel_runtime_pm_put (dev->dev_private);return 0 ; error_ringbuf:
@@ -2532,6 +2537,8 @@ error_deref_obj:drm_gem_object_unreference (&ctx_obj->base); ctx->engine[ring->id].ringbuf =NULL ; ctx->engine[ring->id].state =NULL ; +error_pm: +intel_runtime_pm_put (dev->dev_private);return ret; }
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 7 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware submission part in the back half.
However, as you haven't fixed the ordering issue that requires rpm_get before struct_mutex, this is broken. When we have rpm fixed, you don't need this patch as we can take the wakeref around the GSM access itself. We still want the prolonged rpm wakeref for the GPU activity, ofc. -Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 10 lines · Chris Wilson
Quoted 7 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware submission part in the back half.
However, as you haven't fixed the ordering issue that requires
rpm_getbeforestruct_mutex, this is broken.
Why does 'intel_runtime_pm_get' require the struct mutex to be held? It has certainly not complained at me about trying to do stuff without it.
Quoted 5 lines
When we have rpm fixed, you don't
need this patch as we can take the wakeref around the GSM access itself. We still want the prolonged rpm wakeref for the GPU activity, ofc. -Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 9 lines · John Harrison
Quoted 8 lines · Chris Wilson
Quoted 7 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware submission part in the back half.
Quoted 5 lines
Quoted 2 lines
However, as you haven't fixed the ordering issue that requires
rpm_getbeforestruct_mutex, this is broken.Why does '
intel_runtime_pm_get' require the struct mutex to be held? It has certainly not complained at me about trying to do stuff without it.
Because it depends upon the struct_mutex and rpm doesn't have sufficient lockdep integration to be able to warn about using rpm from the incorrect contexts.
-Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 10 lines · Chris Wilson
Quoted 9 lines · John Harrison
Quoted 8 lines · Chris Wilson
Quoted 7 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware submission part in the back half.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
However, as you haven't fixed the ordering issue that requires
rpm_getbeforestruct_mutex, this is broken.Why does '
intel_runtime_pm_get' require the struct mutex to be held? It has certainly not complained at me about trying to do stuff without it.Because it depends upon the
struct_mutexand rpm doesn't have sufficient lockdep integration to be able to warn about using rpm from the incorrect contexts.
Where? What does the 'pm_runtime_get_sync' call turn into? There are already other places in the driver which call intel_runtime_pm_get() immediately after grabbing the mutex lock. Also, the description comment for _pm_get() does not mention anything about mutexes at all.
Quoted 2 lines
-Chris
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 16 lines · John Harrison
Quoted 15 lines · Chris Wilson
Quoted 14 lines · John Harrison
Quoted 10 lines · Chris Wilson
Quoted 7 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware submission part in the back half.
However, as you haven't fixed the ordering issue that requires
rpm_getbeforestruct_mutex, this is broken.Why does '
intel_runtime_pm_get' require the struct mutex to be held? It has certainly not complained at me about trying to do stuff without it.
Quoted 7 lines
Quoted 3 lines
Because it depends upon the
struct_mutexand rpm doesn't have sufficient lockdep integration to be able to warn about using rpm from the incorrect contexts.Where? What does the '
pm_runtime_get_sync' call turn into? There are already other places in the driver which callintel_runtime_pm_get()immediately after grabbing the mutex lock. Also, the description comment for_pm_get()does not mention anything about mutexes at all.
If you nest rpm_get within dev->struct_mutex that's a bug and could deadlock. Where does this happen? And for any such place we need a new subtest in pm_rpm.
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 20 lines · Daniel Vetter
Quoted 19 lines · John Harrison
Quoted 18 lines · Chris Wilson
Quoted 9 lines · John Harrison
Quoted 8 lines · Chris Wilson
Quoted 7 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware submission part in the back half.
Quoted 5 lines
Quoted 2 lines
However, as you haven't fixed the ordering issue that requires
rpm_getbeforestruct_mutex, this is broken.Why does '
intel_runtime_pm_get' require the struct mutex to be held? It has certainly not complained at me about trying to do stuff without it.Because it depends upon the
struct_mutexand rpm doesn't have sufficient lockdep integration to be able to warn about using rpm from the incorrect contexts.
Quoted 7 lines
Quoted 4 lines
Where? What does the '
pm_runtime_get_sync' call turn into? There are already other places in the driver which callintel_runtime_pm_get()immediately after grabbing the mutex lock. Also, the description comment for_pm_get()does not mention anything about mutexes at all.If you nest
rpm_getwithindev->struct_mutexthat's a bug and could deadlock. Where does this happen? And for any such place we need a new subtest inpm_rpm.
The first two hits when grepping the driver are in 'i915_gem_seqno_info()' and 'i915_interrupt_info()' in i915_debugfs.c. Both say:
ret = mutex_lock_interruptible(&dev->struct_mutex);
if (ret)
return ret;
intel_runtime_pm_get(dev_priv);Quoted 1 line
-Daniel
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 29 lines · John Harrison
Quoted 20 lines · Daniel Vetter
Quoted 19 lines · John Harrison
Quoted 18 lines · Chris Wilson
Quoted 9 lines · John Harrison
Quoted 8 lines · Chris Wilson
Quoted 7 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission code. Part of that is to reduce the scope of a pm_get/put pair. Specifically, they previously wrapped the entire submission path from the very start to the very end, now they only wrap the actual hardware submission part in the back half.
Quoted 5 lines
Quoted 2 lines
However, as you haven't fixed the ordering issue that requires
rpm_getbeforestruct_mutex, this is broken.Why does '
intel_runtime_pm_get' require the struct mutex to be held? It has certainly not complained at me about trying to do stuff without it.Because it depends upon the
struct_mutexand rpm doesn't have sufficient lockdep integration to be able to warn about using rpm from the incorrect contexts.Quoted 7 lines
Quoted 4 lines
Where? What does the '
pm_runtime_get_sync' call turn into? There are already other places in the driver which callintel_runtime_pm_get()immediately after grabbing the mutex lock. Also, the description comment for_pm_get()does not mention anything about mutexes at all.If you nest
rpm_getwithindev->struct_mutexthat's a bug and could deadlock. Where does this happen? And for any such place we need a new subtest inpm_rpm.
Quoted 6 lines
The first two hits when grepping the driver are in '
i915_gem_seqno_info()' and 'i915_interrupt_info()' ini915_debugfs.c. Both say:ret = mutex_lock_interruptible(&dev->struct_mutex); if (ret) return ret; intel_runtime_pm_get(dev_priv);
Yeah that's totally bonkers and will deadlock if the device is actually suspend.
/me goes and files JIRAs
Thanks, Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 54 lines · Daniel Vetter
Quoted 45 lines · John Harrison
Quoted 38 lines · Daniel Vetter
Quoted 30 lines · John Harrison
Quoted 24 lines · Chris Wilson
Quoted 14 lines · John Harrison
Quoted 13 lines · Chris Wilson
Quoted 11 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison @Intel.com wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission
code. Part of that is to reduce the scope of a pm_get/put pair.
Specifically, they previously wrapped the entire submission path from
the very start to the very end, now they only wrap the actual hardware
submission part in the back half.Quoted 8 lines
Quoted 3 lines
However, as you haven't fixed the ordering issue that requires
rpm_get
beforestruct_mutex, this is broken.Why does '
intel_runtime_pm_get' require the struct mutex to be held?
It has certainly not complained at me about trying to do stuff
without it.Quoted 5 lines
Because it depends upon the
struct_mutexand rpm doesn't have sufficient
lockdep integration to be able to warn about using rpm from the
incorrect contexts.Quoted 7 lines
Where? What does the '
pm_runtime_get_sync' call turn into? There are already
other places in the driver which callintel_runtime_pm_get()immediately
after grabbing the mutex lock. Also, the description comment for_pm_get()
does not mention anything about mutexes at all.Quoted 5 lines
If you nest
rpm_getwithindev->struct_mutexthat's a bug and could
deadlock. Where does this happen? And for any such place we need a new
subtest inpm_rpm.Quoted 7 lines
The first two hits when grepping the driver are in '
i915_gem_seqno_info()'
and 'i915_interrupt_info()' ini915_debugfs.c. Both say:
ret =mutex_lock_interruptible(&dev->struct_mutex); if (ret)
return ret;intel_runtime_pm_get(dev_priv);
Quoted 3 lines
Yeah that's totally bonkers and will deadlock if the device is actually
suspend.
It won't deadlock, runtime resume doesn't need struct mutex. Runtime suspend needs it, but we return -EAGAIN from there if it's already held. That in turn will delay the suspend.
--Imre
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 55 lines · Imre Deak
Quoted 54 lines · Daniel Vetter
Quoted 52 lines · John Harrison
Quoted 38 lines · Daniel Vetter
Quoted 30 lines · John Harrison
Quoted 24 lines · Chris Wilson
Quoted 14 lines · John Harrison
Quoted 13 lines · Chris Wilson
Quoted 11 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison @Intel.com wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission
code. Part of that is to reduce the scope of a pm_get/put pair.
Specifically, they previously wrapped the entire submission path from
the very start to the very end, now they only wrap the actual hardware
submission part in the back half.Quoted 8 lines
Quoted 3 lines
However, as you haven't fixed the ordering issue that requires
rpm_get
beforestruct_mutex, this is broken.Why does '
intel_runtime_pm_get' require the struct mutex to be held?
It has certainly not complained at me about trying to do stuff
without it.Quoted 5 lines
Because it depends upon the
struct_mutexand rpm doesn't have sufficient
lockdep integration to be able to warn about using rpm from the
incorrect contexts.Quoted 7 lines
Where? What does the '
pm_runtime_get_sync' call turn into? There are already
other places in the driver which callintel_runtime_pm_get()immediately
after grabbing the mutex lock. Also, the description comment for_pm_get()
does not mention anything about mutexes at all.Quoted 5 lines
If you nest
rpm_getwithindev->struct_mutexthat's a bug and could
deadlock. Where does this happen? And for any such place we need a new
subtest inpm_rpm.The first two hits when grepping the driver are in '
i915_gem_seqno_info()'
and 'i915_interrupt_info()' ini915_debugfs.c. Both say:
ret =mutex_lock_interruptible(&dev->struct_mutex); if (ret)
return ret;intel_runtime_pm_get(dev_priv);
Quoted 7 lines
Quoted 3 lines
Yeah that's totally bonkers and will deadlock if the device is actually
suspend.It won't deadlock, runtime resume doesn't need struct mutex. Runtime suspend needs it, but we return
-EAGAINfrom there if it's already held. That in turn will delay the suspend.
You're right, why do I always mix this up. But this just got me thinking about possible lockdep annotations. We could place a might_lock(rpm_lockdep_key) within rpm_get. And fake-acquire that around the actual ->resume callback. We could even do this in the rpm core ...
That would put all my worries on this topic at easy I think, and I could finally stop confusing everyone ;-)
Or did I again miss the obvious?
-Daniel
Signature · 7 lines
Daniel Vetter Software Engineer, Intel Corporation http://blog.ffwll.ch _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 02/38] drm/i915: Explicit power enable during deferred context initialisation
Quoted 60 lines · Daniel Vetter
Quoted 59 lines · Imre Deak
Quoted 57 lines · Daniel Vetter
Quoted 45 lines · John Harrison
Quoted 38 lines · Daniel Vetter
Quoted 30 lines · John Harrison
Quoted 24 lines · Chris Wilson
Quoted 14 lines · John Harrison
Quoted 13 lines · Chris Wilson
Quoted 11 lines · On Mon, Jan 11, 2016 at 06:42:31PM +0000, John.C.Harrison @Intel.com wrote:
From: John Harrison <[email protected]>
A later patch in this series re-organises the batch buffer submission
code. Part of that is to reduce the scope of a pm_get/put pair.
Specifically, they previously wrapped the entire submission path from
the very start to the very end, now they only wrap the actual hardware
submission part in the back half.Quoted 8 lines
Quoted 3 lines
However, as you haven't fixed the ordering issue that requires
rpm_get
beforestruct_mutex, this is broken.Why does '
intel_runtime_pm_get' require the struct mutex to be held?
It has certainly not complained at me about trying to do stuff
without it.Quoted 5 lines
Because it depends upon the
struct_mutexand rpm doesn't have sufficient
lockdep integration to be able to warn about using rpm from the
incorrect contexts.Quoted 7 lines
Where? What does the '
pm_runtime_get_sync' call turn into? There are already
other places in the driver which callintel_runtime_pm_get()immediately
after grabbing the mutex lock. Also, the description comment for_pm_get()
does not mention anything about mutexes at all.Quoted 5 lines
If you nest
rpm_getwithindev->struct_mutexthat's a bug and could
deadlock. Where does this happen? And for any such place we need a new
subtest inpm_rpm.Quoted 7 lines
The first two hits when grepping the driver are in '
i915_gem_seqno_info()'
and 'i915_interrupt_info()' ini915_debugfs.c. Both say:
ret =mutex_lock_interruptible(&dev->struct_mutex); if (ret)
return ret;intel_runtime_pm_get(dev_priv);Yeah that's totally bonkers and will deadlock if the device is actually
suspend.
Quoted 8 lines
Quoted 3 lines
It won't deadlock, runtime resume doesn't need struct mutex. Runtime suspend needs it, but we return
-EAGAINfrom there if it's already held. That in turn will delay the suspend.You're right, why do I always mix this up. But this just got me thinking about possible lockdep annotations. We could place a
might_lock(rpm_lockdep_key)withinrpm_get. And fake-acquire that around the actual ->resume callback. We could even do this in the rpm core ...
Or we could make progress on eliminating the need for struct_mutex in rpm resume/suspend/whenevner :)
-Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 3/38 drm/i915: Prelude to splitting i915_gem_do_execbuffer in two
In reply to [email protected]From: John Harrison <[email protected]>
The scheduler decouples the submission of batch buffers to the driver with their submission to the hardware. This basically means splitting the execbuffer() function in half. This change rearranges some code ready for the split to occur.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 63 ++++++++++++++++++------------ drivers/gpu/drm/i915/intel_lrc.c | 18 ++++++--- 2 files changed, 51 insertions(+), 30 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +38 -25
@@ -933,10 +933,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,if (flush_domains &I915_GEM_DOMAIN_GTT )wmb (); -/* Unconditionally invalidate gpu caches and ensure that we do flush -* any residual writes from the previous batch. -*/ -return intel_ring_invalidate_all_caches (req); +return 0 ; } static bool
@@ -1189,17 +1186,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,u32 instp_mask;int ret; - ret =i915_gem_execbuffer_move_to_gpu (params->request, vmas); -if (ret) -return ret; - - ret =i915_switch_context (params->request); -if (ret) -return ret; - -WARN (params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1 <<ring->id), -"%s didn't clear reload\n" , ring->name); - instp_mode = args->flags &I915_EXEC_CONSTANTS_MASK ; instp_mask =I915_EXEC_CONSTANTS_MASK ;switch (instp_mode) {
@@ -1233,11 +1219,37 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,return -EINVAL ; } + ret =i915_gem_execbuffer_move_to_gpu (params->request, vmas); +if (ret) +return ret; + +i915_gem_execbuffer_move_to_active (vmas, params->request); + +/* To be split into two functions here... */ + +intel_runtime_pm_get (dev_priv); + +/* +* Unconditionally invalidate gpu caches and ensure that we do flush +* any residual writes from the previous batch. +*/ + ret =intel_ring_invalidate_all_caches (params->request); +if (ret) + goto error; + +/* Switch to the correct context for the batch */ + ret =i915_switch_context (params->request); +if (ret) + goto error; + +WARN (params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1 <<ring->id), +"%s didn't clear reload\n" , ring->name); +if (ring == &dev_priv->ring[RCS ] && instp_mode != dev_priv->relative_constants_mode) { ret =intel_ring_begin (params->request,4 );if (ret) -return ret; + goto error;intel_ring_emit (ring,MI_NOOP );intel_ring_emit (ring,MI_LOAD_REGISTER_IMM (1 ));
@@ -1251,7 +1263,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,if (args->flags &I915_EXEC_GEN7_SOL_RESET ) { ret =i915_reset_gen7_sol_offsets (dev, params->request);if (ret) -return ret; + goto error; } exec_len = args->batch_len;
@@ -1262,14 +1274,20 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params, exec_start, exec_len, params->dispatch_flags);if (ret) -return ret; + goto error;trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); -i915_gem_execbuffer_move_to_active (vmas, params->request);i915_gem_execbuffer_retire_commands (params); -return 0 ; +error: +/* +* intel_gpu_busy should also get a ref, so it will free when the device +* is really idle. +*/ +intel_runtime_pm_put (dev_priv); + +return ret; }/**
@@ -1424,8 +1442,6 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, dispatch_flags |=I915_DISPATCH_RS ; } -intel_runtime_pm_get (dev_priv); - ret =i915_mutex_lock_interruptible (dev);if (ret) goto pre_mutex_err;
@@ -1599,9 +1615,6 @@ err:mutex_unlock (&dev->struct_mutex); pre_mutex_err: -/* intel_gpu_busy should also get a ref, so it will free when the device -* is really idle. */ -intel_runtime_pm_put (dev_priv);return ret; }
drivers/gpu/drm/i915/intel_lrc.c +13 -5
@@ -647,10 +647,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,if (flush_domains &I915_GEM_DOMAIN_GTT )wmb (); -/* Unconditionally invalidate gpu caches and ensure that we do flush -* any residual writes from the previous batch. -*/ -return logical_ring_invalidate_all_caches (req); +return 0 ; }int intel_logical_ring_alloc_request_extras (struct drm_i915_gem_request *request)
@@ -913,6 +910,18 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,if (ret)return ret; +i915_gem_execbuffer_move_to_active (vmas, params->request); + +/* To be split into two functions here... */ + +/* +* Unconditionally invalidate gpu caches and ensure that we do flush +* any residual writes from the previous batch. +*/ + ret =logical_ring_invalidate_all_caches (params->request); +if (ret) +return ret; +if (ring == &dev_priv->ring[RCS ] && instp_mode != dev_priv->relative_constants_mode) { ret =intel_logical_ring_begin (params->request,4 );
@@ -937,7 +946,6 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); -i915_gem_execbuffer_move_to_active (vmas, params->request);i915_gem_execbuffer_retire_commands (params);return 0 ;
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 03/38] drm/i915: Prelude to splitting i915_gem_do_execbuffer in two
Quoted 176 lines · On 01/11/2016 10:42 AM, [email protected] wrote:
From: John Harrison <[email protected]>
The scheduler decouples the submission of batch buffers to the driver with their submission to the hardware. This basically means splitting the
execbuffer()function in half. This change rearranges some code ready for the split to occur.For: VIZ-1587
Signed-off-by John Harrison <[email protected]>--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 63 ++++++++++++++++++------------ drivers/gpu/drm/i915/intel_lrc.c | 18 ++++++--- 2 files changed, 51 insertions(+), 30 deletions(-)drivers/gpu/drm/i915/i915_gem_execbuffer.c +38
-25@@ -933,10 +933,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,if (flush_domains &I915_GEM_DOMAIN_GTT )wmb (); -/* Unconditionally invalidate gpu caches and ensure that we do flush -* any residual writes from the previous batch. -*/ -return intel_ring_invalidate_all_caches (req); +return 0 ; } static bool@@ -1189,17 +1186,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,u32 instp_mask;int ret; - ret =i915_gem_execbuffer_move_to_gpu (params->request, vmas); -if (ret) -return ret; - - ret =i915_switch_context (params->request); -if (ret) -return ret; - -WARN (params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1 <<ring->id), -"%s didn't clear reload\n" , ring->name); - instp_mode = args->flags &I915_EXEC_CONSTANTS_MASK ; instp_mask =I915_EXEC_CONSTANTS_MASK ;switch (instp_mode) {@@ -1233,11 +1219,37 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,return -EINVAL ; } + ret =i915_gem_execbuffer_move_to_gpu (params->request, vmas); +if (ret) +return ret; + +i915_gem_execbuffer_move_to_active (vmas, params->request); + +/* To be split into two functions here... */ + +intel_runtime_pm_get (dev_priv); + +/* +* Unconditionally invalidate gpu caches and ensure that we do flush +* any residual writes from the previous batch. +*/ + ret =intel_ring_invalidate_all_caches (params->request); +if (ret) + goto error; + +/* Switch to the correct context for the batch */ + ret =i915_switch_context (params->request); +if (ret) + goto error; + +WARN (params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1 <<ring->id), +"%s didn't clear reload\n" , ring->name); +if (ring == &dev_priv->ring[RCS ] && instp_mode != dev_priv->relative_constants_mode) { ret =intel_ring_begin (params->request,4 );if (ret) -return ret; + goto error;intel_ring_emit (ring,MI_NOOP );intel_ring_emit (ring,MI_LOAD_REGISTER_IMM (1 ));@@ -1251,7 +1263,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,if (args->flags &I915_EXEC_GEN7_SOL_RESET ) { ret =i915_reset_gen7_sol_offsets (dev, params->request);if (ret) -return ret; + goto error; } exec_len = args->batch_len;@@ -1262,14 +1274,20 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params, exec_start, exec_len, params->dispatch_flags);if (ret) -return ret; + goto error;trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); -i915_gem_execbuffer_move_to_active (vmas, params->request);i915_gem_execbuffer_retire_commands (params); -return 0 ; +error: +/* +* intel_gpu_busy should also get a ref, so it will free when the device +* is really idle. +*/ +intel_runtime_pm_put (dev_priv); + +return ret; }/** @@ -1424,8 +1442,6 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, dispatch_flags |=I915_DISPATCH_RS ; } -intel_runtime_pm_get (dev_priv); - ret =i915_mutex_lock_interruptible (dev);if (ret) goto pre_mutex_err;@@ -1599,9 +1615,6 @@ err:mutex_unlock (&dev->struct_mutex); pre_mutex_err: -/* intel_gpu_busy should also get a ref, so it will free when the device -* is really idle. */ -intel_runtime_pm_put (dev_priv);return ret; }drivers/gpu/drm/i915/intel_lrc.c +13
-4@@ -647,10 +647,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,if (flush_domains &I915_GEM_DOMAIN_GTT )wmb (); -/* Unconditionally invalidate gpu caches and ensure that we do flush -* any residual writes from the previous batch. -*/ -return logical_ring_invalidate_all_caches (req); +return 0 ; }int intel_logical_ring_alloc_request_extras (struct drm_i915_gem_request *request)@@ -913,6 +910,18 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,if (ret)return ret; +i915_gem_execbuffer_move_to_active (vmas, params->request); + +/* To be split into two functions here... */ + +/* +* Unconditionally invalidate gpu caches and ensure that we do flush +* any residual writes from the previous batch. +*/ + ret =logical_ring_invalidate_all_caches (params->request); +if (ret) +return ret; +if (ring == &dev_priv->ring[RCS ] && instp_mode != dev_priv->relative_constants_mode) { ret =intel_logical_ring_begin (params->request,4 );
drivers/gpu/drm/i915/intel_lrc.c int intel_execlists_submission(struct i915_execbuffer_params *params, · line 951
Quoted 7 lines
drivers/gpu/drm/i915/intel_lrc.c +0
-1@@ -937,7 +946,6 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); -i915_gem_execbuffer_move_to_active (vmas, params->request);i915_gem_execbuffer_retire_commands (params);return 0 ;
Do we need to do anything if the cache invalidation fails like move the buffers back off the active list? The order changed here, so I'm wondering.
If that's not a problem, then:
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 03/38] drm/i915: Prelude to splitting i915_gem_do_execbuffer in two
Quoted 177 lines · Jesse Barnes
Quoted 176 lines · On 01/11/2016 10:42 AM, [email protected] wrote:
From: John Harrison <[email protected]>
The scheduler decouples the submission of batch buffers to the driver with their submission to the hardware. This basically means splitting the
execbuffer()function in half. This change rearranges some code ready for the split to occur.For: VIZ-1587
Signed-off-by John Harrison <[email protected]>--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 63 ++++++++++++++++++------------ drivers/gpu/drm/i915/intel_lrc.c | 18 ++++++--- 2 files changed, 51 insertions(+), 30 deletions(-)drivers/gpu/drm/i915/i915_gem_execbuffer.c +38
-25@@ -933,10 +933,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,if (flush_domains &I915_GEM_DOMAIN_GTT )wmb (); -/* Unconditionally invalidate gpu caches and ensure that we do flush -* any residual writes from the previous batch. -*/ -return intel_ring_invalidate_all_caches (req); +return 0 ; } static bool@@ -1189,17 +1186,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,u32 instp_mask;int ret; - ret =i915_gem_execbuffer_move_to_gpu (params->request, vmas); -if (ret) -return ret; - - ret =i915_switch_context (params->request); -if (ret) -return ret; - -WARN (params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1 <<ring->id), -"%s didn't clear reload\n" , ring->name); - instp_mode = args->flags &I915_EXEC_CONSTANTS_MASK ; instp_mask =I915_EXEC_CONSTANTS_MASK ;switch (instp_mode) {@@ -1233,11 +1219,37 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,return -EINVAL ; } + ret =i915_gem_execbuffer_move_to_gpu (params->request, vmas); +if (ret) +return ret; + +i915_gem_execbuffer_move_to_active (vmas, params->request); + +/* To be split into two functions here... */ + +intel_runtime_pm_get (dev_priv); + +/* +* Unconditionally invalidate gpu caches and ensure that we do flush +* any residual writes from the previous batch. +*/ + ret =intel_ring_invalidate_all_caches (params->request); +if (ret) + goto error; + +/* Switch to the correct context for the batch */ + ret =i915_switch_context (params->request); +if (ret) + goto error; + +WARN (params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1 <<ring->id), +"%s didn't clear reload\n" , ring->name); +if (ring == &dev_priv->ring[RCS ] && instp_mode != dev_priv->relative_constants_mode) { ret =intel_ring_begin (params->request,4 );if (ret) -return ret; + goto error;intel_ring_emit (ring,MI_NOOP );intel_ring_emit (ring,MI_LOAD_REGISTER_IMM (1 ));@@ -1251,7 +1263,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,if (args->flags &I915_EXEC_GEN7_SOL_RESET ) { ret =i915_reset_gen7_sol_offsets (dev, params->request);if (ret) -return ret; + goto error; } exec_len = args->batch_len;@@ -1262,14 +1274,20 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params, exec_start, exec_len, params->dispatch_flags);if (ret) -return ret; + goto error;trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); -i915_gem_execbuffer_move_to_active (vmas, params->request);i915_gem_execbuffer_retire_commands (params); -return 0 ; +error: +/* +* intel_gpu_busy should also get a ref, so it will free when the device +* is really idle. +*/ +intel_runtime_pm_put (dev_priv); + +return ret; }/** @@ -1424,8 +1442,6 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, dispatch_flags |=I915_DISPATCH_RS ; } -intel_runtime_pm_get (dev_priv); - ret =i915_mutex_lock_interruptible (dev);if (ret) goto pre_mutex_err;@@ -1599,9 +1615,6 @@ err:mutex_unlock (&dev->struct_mutex); pre_mutex_err: -/* intel_gpu_busy should also get a ref, so it will free when the device -* is really idle. */ -intel_runtime_pm_put (dev_priv);return ret; }drivers/gpu/drm/i915/intel_lrc.c +13
-4@@ -647,10 +647,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,if (flush_domains &I915_GEM_DOMAIN_GTT )wmb (); -/* Unconditionally invalidate gpu caches and ensure that we do flush -* any residual writes from the previous batch. -*/ -return logical_ring_invalidate_all_caches (req); +return 0 ; }int intel_logical_ring_alloc_request_extras (struct drm_i915_gem_request *request)@@ -913,6 +910,18 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,if (ret)return ret; +i915_gem_execbuffer_move_to_active (vmas, params->request); + +/* To be split into two functions here... */ + +/* +* Unconditionally invalidate gpu caches and ensure that we do flush +* any residual writes from the previous batch. +*/ + ret =logical_ring_invalidate_all_caches (params->request); +if (ret) +return ret; +if (ring == &dev_priv->ring[RCS ] && instp_mode != dev_priv->relative_constants_mode) { ret =intel_logical_ring_begin (params->request,4 );@@ -937,7 +946,6 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
Quoted 8 lines
drivers/gpu/drm/i915/intel_lrc.c int intel_execlists_submission(struct i915_execbuffer_params *params, · line 951
Quoted 7 lines
drivers/gpu/drm/i915/intel_lrc.c +0
-1trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); -i915_gem_execbuffer_move_to_active (vmas, params->request);i915_gem_execbuffer_retire_commands (params);return 0 ;Do we need to do anything if the cache invalidation fails like move the buffers back off the active list? The order changed here, so I'm wondering.
I don't believe so. The objects are all tracked by request. The request itself is cancelled rather than added to the request list. So the next run of i915_gem_retire_requests_ring() will do all the necessary clean up. Potentially we could do something explicit here to speed the process up. However, this is only an intermediate step and there is no nice and simple move_to_inactive function to call. Once the scheduler arrives, a failed backend submission would be retried by the scheduler. So objects must not be cleaned up as that would break the re-submission.
Quoted 2 lines
If that's not a problem, then:
Reviewed-by Jesse Barnes <[email protected]>
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 4/38 drm/i915: Split i915_dem_do_execbuffer() in half
In reply to [email protected]From: John Harrison <[email protected]>
Split the execbuffer() function in half. The first half collects and validates all the information required to process the batch buffer. It also does all the object pinning, relocations, active list management, etc - basically anything that must be done upfront before the IOCTL returns and allows the user land side to start changing/freeing things. The second half does the actual ring submission.
This change implements the split but leaves the back half being called directly from the end of the front half.
v2: Updated due to changes in underlying tree - addition of sync fence support and removal of cliprects.
v3: Moved local 'ringbuf' variable to make later patches in the series a bit neater.
v4: Corrected a typo in the commit message and downgraded a BUG_ON to a WARN_ON as the latter is preferred. Also removed all the external sync/fence support as that will now be a separate patch series.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_drv.h | 11 ++++ drivers/gpu/drm/i915/i915_gem.c | 2 + drivers/gpu/drm/i915/i915_gem_execbuffer.c | 101 ++++++++++++++++++++++------- drivers/gpu/drm/i915/intel_lrc.c | 57 +++++++++++----- drivers/gpu/drm/i915/intel_lrc.h | 1 + 5 files changed, 130 insertions(+), 42 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +11 -0
@@ -1679,10 +1679,18 @@ struct i915_execbuffer_params {struct drm_device *dev;struct drm_file *file;uint32_t dispatch_flags; +uint32_t args_flags;uint32_t args_batch_start_offset; +uint32_t args_batch_len; +uint32_t args_num_cliprects; +uint32_t args_DR1; +uint32_t args_DR4;uint64_t batch_obj_vm_offset;struct intel_engine_cs *ring;struct drm_i915_gem_object *batch_obj; +struct drm_clip_rect *cliprects; +uint32_t instp_mask; +int instp_mode;struct intel_context *ctx;struct drm_i915_gem_request *request; };
@@ -1944,6 +1952,7 @@ struct drm_i915_private {int (*execbuf_submit)(struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int (*execbuf_final)(struct i915_execbuffer_params *params);int (*init_rings)(struct drm_device *dev);void (*cleanup_ring)(struct intel_engine_cs *ring);void (*stop_ring)(struct intel_engine_cs *ring);
@@ -2790,9 +2799,11 @@ int i915_gem_sw_finish_ioctl(struct drm_device *dev, void *data,void i915_gem_execbuffer_move_to_active (struct list_head *vmas,struct drm_i915_gem_request *req);void i915_gem_execbuffer_retire_commands (struct i915_execbuffer_params *params); +void i915_gem_execbuff_release_batch_obj (struct drm_i915_gem_object *batch_obj);int i915_gem_ringbuffer_submission (struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int i915_gem_ringbuffer_submission_final (struct i915_execbuffer_params *params);int i915_gem_execbuffer (struct drm_device *dev,void *data,struct drm_file *file_priv); int i915_gem_execbuffer2(struct drm_device *dev, void *data,
drivers/gpu/drm/i915/i915_gem.c +2 -0
@@ -5220,11 +5220,13 @@ int i915_gem_init(struct drm_device *dev)if (!i915.enable_execlists) { dev_priv->gt.execbuf_submit = i915_gem_ringbuffer_submission; + dev_priv->gt.execbuf_final = i915_gem_ringbuffer_submission_final; dev_priv->gt.init_rings = i915_gem_init_rings; dev_priv->gt.cleanup_ring = intel_cleanup_ring_buffer; dev_priv->gt.stop_ring = intel_stop_ring_buffer; }else { dev_priv->gt.execbuf_submit = intel_execlists_submission; + dev_priv->gt.execbuf_final = intel_execlists_submission_final; dev_priv->gt.init_rings = intel_logical_rings_init; dev_priv->gt.cleanup_ring = intel_logical_ring_cleanup; dev_priv->gt.stop_ring = intel_logical_ring_stop;
drivers/gpu/drm/i915/i915_gem_execbuffer.c +76 -25
179 lines not on this page · raw message
drivers/gpu/drm/i915/intel_lrc.c +40 -17
103 lines not on this page · raw message
drivers/gpu/drm/i915/intel_lrc.h +1 -0
@@ -93,6 +93,7 @@ struct i915_execbuffer_params;int intel_execlists_submission (struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int intel_execlists_submission_final (struct i915_execbuffer_params *params);u32 intel_execlists_ctx_id (struct drm_i915_gem_object *ctx_obj);void intel_lrc_irq_handler (struct intel_engine_cs *ring);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 04/38] drm/i915: Split i915_dem_do_execbuffer() in half
Quoted 8 lines · On Mon, Jan 11, 2016 at 06:42:33PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
Split the
execbuffer()function in half. The first half collects and validates all the information required to process the batch buffer. It also does all the object pinning, relocations, active list management, etc - basically anything that must be done upfront before the IOCTL returns and allows the user land side to start changing/freeing things. The second half does the actual ring submission.
I don't get this at all. The point of requests is that GEM constructed a request, which could be used to pass along all the implicit GEM synchronisation points and the explict ones, along with the ringbuffer to execute, to an engine that could then submit it. For legacy, the request was inline and so added immediately to the ring (but that is an implementation detail, there is nothing stopping us from using a chained batch to implement a ring per context), for execlists the request is queued for future execution. A scheduler was meant to sit in the middle and determine the order in which requests were executed, but that should be almost transparent to the high level code tracking the requests. -Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 04/38] drm/i915: Split i915_dem_do_execbuffer() in half
In reply to [email protected]Quoted 398 lines · On 01/11/2016 10:42 AM, [email protected] wrote:
From: John Harrison <[email protected]>
Split the
execbuffer()function in half. The first half collects and validates all the information required to process the batch buffer. It also does all the object pinning, relocations, active list management, etc - basically anything that must be done upfront before the IOCTL returns and allows the user land side to start changing/freeing things. The second half does the actual ring submission.This change implements the split but leaves the back half being called directly from the end of the front half.
v2: Updated due to changes in underlying tree - addition of sync fence support and removal of cliprects.
v3: Moved local 'ringbuf' variable to make later patches in the series a bit neater.
v4: Corrected a typo in the commit message and downgraded a
BUG_ONto aWARN_ONas the latter is preferred. Also removed all the external sync/fence support as that will now be a separate patch series.For: VIZ-1587
Signed-off-by John Harrison <[email protected]>--- drivers/gpu/drm/i915/i915_drv.h | 11 ++++ drivers/gpu/drm/i915/i915_gem.c | 2 + drivers/gpu/drm/i915/i915_gem_execbuffer.c | 101 ++++++++++++++++++++++------- drivers/gpu/drm/i915/intel_lrc.c | 57 +++++++++++----- drivers/gpu/drm/i915/intel_lrc.h | 1 + 5 files changed, 130 insertions(+), 42 deletions(-)drivers/gpu/drm/i915/i915_drv.h +11
-0@@ -1679,10 +1679,18 @@ struct i915_execbuffer_params {struct drm_device *dev;struct drm_file *file;uint32_t dispatch_flags; +uint32_t args_flags;uint32_t args_batch_start_offset; +uint32_t args_batch_len; +uint32_t args_num_cliprects; +uint32_t args_DR1; +uint32_t args_DR4;uint64_t batch_obj_vm_offset;struct intel_engine_cs *ring;struct drm_i915_gem_object *batch_obj; +struct drm_clip_rect *cliprects; +uint32_t instp_mask; +int instp_mode;struct intel_context *ctx;struct drm_i915_gem_request *request; };@@ -1944,6 +1952,7 @@ struct drm_i915_private {int (*execbuf_submit)(struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int (*execbuf_final)(struct i915_execbuffer_params *params);int (*init_rings)(struct drm_device *dev);void (*cleanup_ring)(struct intel_engine_cs *ring);void (*stop_ring)(struct intel_engine_cs *ring);@@ -2790,9 +2799,11 @@ int i915_gem_sw_finish_ioctl(struct drm_device *dev, void *data,void i915_gem_execbuffer_move_to_active (struct list_head *vmas,struct drm_i915_gem_request *req);void i915_gem_execbuffer_retire_commands (struct i915_execbuffer_params *params); +void i915_gem_execbuff_release_batch_obj (struct drm_i915_gem_object *batch_obj);int i915_gem_ringbuffer_submission (struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int i915_gem_ringbuffer_submission_final (struct i915_execbuffer_params *params);int i915_gem_execbuffer (struct drm_device *dev,void *data,struct drm_file *file_priv); int i915_gem_execbuffer2(struct drm_device *dev, void *data,drivers/gpu/drm/i915/i915_gem.c +2
-0@@ -5220,11 +5220,13 @@ int i915_gem_init(struct drm_device *dev)if (!i915.enable_execlists) { dev_priv->gt.execbuf_submit = i915_gem_ringbuffer_submission; + dev_priv->gt.execbuf_final = i915_gem_ringbuffer_submission_final; dev_priv->gt.init_rings = i915_gem_init_rings; dev_priv->gt.cleanup_ring = intel_cleanup_ring_buffer; dev_priv->gt.stop_ring = intel_stop_ring_buffer; }else { dev_priv->gt.execbuf_submit = intel_execlists_submission; + dev_priv->gt.execbuf_final = intel_execlists_submission_final; dev_priv->gt.init_rings = intel_logical_rings_init; dev_priv->gt.cleanup_ring = intel_logical_ring_cleanup; dev_priv->gt.stop_ring = intel_logical_ring_stop;drivers/gpu/drm/i915/i915_gem_execbuffer.c +76
-25@@ -1181,41 +1181,38 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,struct drm_device *dev = params->dev;struct intel_engine_cs *ring = params->ring;struct drm_i915_private *dev_priv = dev->dev_private; -u64 exec_start, exec_len; -int instp_mode; -u32 instp_mask;int ret; - instp_mode = args->flags &I915_EXEC_CONSTANTS_MASK ; - instp_mask =I915_EXEC_CONSTANTS_MASK ; -switch (instp_mode) { + params->instp_mode = args->flags &I915_EXEC_CONSTANTS_MASK ; + params->instp_mask =I915_EXEC_CONSTANTS_MASK ; +switch (params->instp_mode) {case I915_EXEC_CONSTANTS_REL_GENERAL :case I915_EXEC_CONSTANTS_ABSOLUTE :case I915_EXEC_CONSTANTS_REL_SURFACE : -if (instp_mode !=0 && ring != &dev_priv->ring[RCS ]) { +if (params->instp_mode !=0 && ring != &dev_priv->ring[RCS ]) {DRM_DEBUG ("non-0 rel constants mode on non-RCS\n" );return -EINVAL ; } -if (instp_mode != dev_priv->relative_constants_mode) { +if (params->instp_mode != dev_priv->relative_constants_mode) {if (INTEL_INFO (dev)->gen <4 ) {DRM_DEBUG ("no rel constants on pre-gen4\n" );return -EINVAL ; }if (INTEL_INFO (dev)->gen >5 && - instp_mode ==I915_EXEC_CONSTANTS_REL_SURFACE ) { + params->instp_mode ==I915_EXEC_CONSTANTS_REL_SURFACE ) {DRM_DEBUG ("rel surface constants mode invalid on gen5+\n" );return -EINVAL ; }/* The HW changed the meaning on this bit on gen6 */ if (INTEL_INFO (dev)->gen >=6 ) - instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE ; + params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE ; }break ;default : -DRM_DEBUG ("execbuf with unknown constants: %d\n" , instp_mode); +DRM_DEBUG ("execbuf with unknown constants: %d\n" , params->instp_mode);return -EINVAL ; }@@ -1225,7 +1222,33 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,i915_gem_execbuffer_move_to_active (vmas, params->request); -/* To be split into two functions here... */ + ret = dev_priv->gt.execbuf_final (params); +if (ret) +return ret; + +/* +* Free everything that was stored in the QE structure (until the +* scheduler arrives and does it instead): +*/ +if (params->dispatch_flags &I915_DISPATCH_SECURE ) +i915_gem_execbuff_release_batch_obj (params->batch_obj); + +return 0 ; +} + +/* +* This is the main function for adding a batch to the ring. +* It is called from the scheduler, with the struct_mutex already held. +*/ +int i915_gem_ringbuffer_submission_final (struct i915_execbuffer_params *params) +{ +struct drm_i915_private *dev_priv = params->dev->dev_private; +struct intel_engine_cs *ring = params->ring; +u64 exec_start, exec_len; +int ret; + +/* The mutex must be acquired before calling this function */ +WARN_ON (!mutex_is_locked (¶ms->dev->struct_mutex));intel_runtime_pm_get (dev_priv);@@ -1246,7 +1269,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params, "%s didn't clear reload\n", ring->name);if (ring == &dev_priv->ring[RCS ] && - instp_mode != dev_priv->relative_constants_mode) { + params->instp_mode != dev_priv->relative_constants_mode) { ret =intel_ring_begin (params->request,4 );if (ret) goto error;@@ -1254,19 +1277,19 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,intel_ring_emit (ring,MI_NOOP );intel_ring_emit (ring,MI_LOAD_REGISTER_IMM (1 ));intel_ring_emit (ring,INSTPM ); -intel_ring_emit (ring, instp_mask <<16 | instp_mode); +intel_ring_emit (ring, params->instp_mask <<16 | params->instp_mode);intel_ring_advance (ring); - dev_priv->relative_constants_mode = instp_mode; + dev_priv->relative_constants_mode = params->instp_mode; } -if (args->flags &I915_EXEC_GEN7_SOL_RESET ) { - ret =i915_reset_gen7_sol_offsets (dev, params->request); +if (params->args_flags &I915_EXEC_GEN7_SOL_RESET ) { + ret =i915_reset_gen7_sol_offsets (params->dev, params->request);if (ret) goto error; } - exec_len = args->batch_len; + exec_len = params->args_batch_len; exec_start = params->batch_obj_vm_offset + params->args_batch_start_offset;@@ -1584,23 +1607,40 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data, params->file = file; params->ring = ring; params->dispatch_flags = dispatch_flags; + params->args_flags = args->flags; + params->args_batch_len = args->batch_len; + params->args_num_cliprects = args->num_cliprects; + params->args_DR1 = args->DR1; + params->args_DR4 = args->DR4; params->batch_obj = batch_obj; params->ctx = ctx; ret = dev_priv->gt.execbuf_submit (params, args, &eb->vmas); +if (ret) + goto err_batch_unpin; + +/* the request owns the ref now */ +i915_gem_context_unreference (ctx); -err_batch_unpin:/* -* FIXME: We crucially rely upon the active tracking for the (ppgtt) -* batch vma for correctness. For less ugly and less fragility this -* needs to be adjusted to also track the ggtt batch vma properly as -* active. +* The eb list is no longer required. The scheduler has extracted all +* the information than needs to persist. */ +eb_destroy (eb); + +/* +* Don't clean up everything that is now saved away in the queue. +* Just unlock and return immediately. +*/ +mutex_unlock (&dev->struct_mutex); + +return 0 ; + +err_batch_unpin:if (dispatch_flags &I915_DISPATCH_SECURE ) -i915_gem_object_ggtt_unpin (batch_obj); +i915_gem_execbuff_release_batch_obj (batch_obj); err: -/* the request owns the ref now */ i915_gem_context_unreference (ctx);eb_destroy (eb);@@ -1618,6 +1658,17 @@ pre_mutex_err:return ret; } +void i915_gem_execbuff_release_batch_obj (struct drm_i915_gem_object *batch_obj) +{ +/* +* FIXME: We crucially rely upon the active tracking for the (ppgtt) +* batch vma for correctness. For less ugly and less fragility this +* needs to be adjusted to also track the ggtt batch vma properly as +* active. +*/ +i915_gem_object_ggtt_unpin (batch_obj); +} +/* * Legacy execbuffer just creates an exec2 list from the original exec object * list array and passes it to the real function. drivers/gpu/drm/i915/intel_lrc.c +40
-17@@ -869,35 +869,31 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,struct drm_device *dev = params->dev;struct intel_engine_cs *ring = params->ring;struct drm_i915_private *dev_priv = dev->dev_private; -struct intel_ringbuffer *ringbuf = params->ctx->engine[ring->id].ringbuf; -u64 exec_start; -int instp_mode; -u32 instp_mask;int ret; - instp_mode = args->flags &I915_EXEC_CONSTANTS_MASK ; - instp_mask =I915_EXEC_CONSTANTS_MASK ; -switch (instp_mode) { + params->instp_mode = args->flags &I915_EXEC_CONSTANTS_MASK ; + params->instp_mask =I915_EXEC_CONSTANTS_MASK ; +switch (params->instp_mode) {case I915_EXEC_CONSTANTS_REL_GENERAL :case I915_EXEC_CONSTANTS_ABSOLUTE :case I915_EXEC_CONSTANTS_REL_SURFACE : -if (instp_mode !=0 && ring != &dev_priv->ring[RCS ]) { +if (params->instp_mode !=0 && ring != &dev_priv->ring[RCS ]) {DRM_DEBUG ("non-0 rel constants mode on non-RCS\n" );return -EINVAL ; } -if (instp_mode != dev_priv->relative_constants_mode) { -if (instp_mode ==I915_EXEC_CONSTANTS_REL_SURFACE ) { +if (params->instp_mode != dev_priv->relative_constants_mode) { +if (params->instp_mode ==I915_EXEC_CONSTANTS_REL_SURFACE ) {DRM_DEBUG ("rel surface constants mode invalid on gen5+\n" );return -EINVAL ; }/* The HW changed the meaning on this bit on gen6 */ - instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE ; + params->instp_mask &= ~I915_EXEC_CONSTANTS_REL_SURFACE ; }break ;default : -DRM_DEBUG ("execbuf with unknown constants: %d\n" , instp_mode); +DRM_DEBUG ("execbuf with unknown constants: %d\n" , params->instp_mode);return -EINVAL ; }@@ -912,7 +908,34 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,i915_gem_execbuffer_move_to_active (vmas, params->request); -/* To be split into two functions here... */ + ret = dev_priv->gt.execbuf_final (params); +if (ret) +return ret; + +/* +* Free everything that was stored in the QE structure (until the +* scheduler arrives and does it instead): +*/ +if (params->dispatch_flags &I915_DISPATCH_SECURE ) +i915_gem_execbuff_release_batch_obj (params->batch_obj); + +return 0 ; +} + +/* +* This is the main function for adding a batch to the ring. +* It is called from the scheduler, with the struct_mutex already held. +*/ +int intel_execlists_submission_final (struct i915_execbuffer_params *params) +{ +struct drm_i915_private *dev_priv = params->dev->dev_private; +struct intel_ringbuffer *ringbuf = params->request->ringbuf; +struct intel_engine_cs *ring = params->ring; +u64 exec_start; +int ret; + +/* The mutex must be acquired before calling this function */ +WARN_ON (!mutex_is_locked (¶ms->dev->struct_mutex));/* * Unconditionally invalidate gpu caches and ensure that we do flush @@ -923,7 +946,7 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,return ret;if (ring == &dev_priv->ring[RCS ] && - instp_mode != dev_priv->relative_constants_mode) { + params->instp_mode != dev_priv->relative_constants_mode) { ret =intel_logical_ring_begin (params->request,4 );if (ret)return ret;@@ -931,14 +954,14 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,intel_logical_ring_emit (ringbuf,MI_NOOP );intel_logical_ring_emit (ringbuf,MI_LOAD_REGISTER_IMM (1 ));intel_logical_ring_emit (ringbuf,INSTPM ); -intel_logical_ring_emit (ringbuf, instp_mask <<16 | instp_mode); +intel_logical_ring_emit (ringbuf, params->instp_mask <<16 | params->instp_mode);intel_logical_ring_advance (ringbuf); - dev_priv->relative_constants_mode = instp_mode; + dev_priv->relative_constants_mode = params->instp_mode; } exec_start = params->batch_obj_vm_offset + - args->batch_start_offset; + params->args_batch_start_offset; ret = ring->emit_bb_start (params->request, exec_start, params->dispatch_flags);if (ret)drivers/gpu/drm/i915/intel_lrc.h +0
-0
drivers/gpu/drm/i915/intel_lrc.h struct i915_execbuffer_params; · line 99
Quoted 7 lines
drivers/gpu/drm/i915/intel_lrc.h +1
-0@@ -93,6 +93,7 @@ struct i915_execbuffer_params;int intel_execlists_submission (struct i915_execbuffer_params *params,struct drm_i915_gem_execbuffer2 *args,struct list_head *vmas); +int intel_execlists_submission_final (struct i915_execbuffer_params *params);u32 intel_execlists_ctx_id (struct drm_i915_gem_object *ctx_obj);void intel_lrc_irq_handler (struct intel_engine_cs *ring);
Just a nitpick on naming; I think prepare/commit might be better than submit/submit_final. But you don't have to respin just for that.
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 5/38 drm/i915: Cache request pointer in *_submission_final()
In reply to [email protected]From: Dave Gordon <[email protected]>
Keep a local copy of the request pointer in the _final() functions rather than dereferencing the params block repeatedly.
v3: New patch in series.
For: VIZ-1587
--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++------ drivers/gpu/drm/i915/intel_lrc.c | 11 ++++++----- 2 files changed, 13 insertions(+), 11 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +7 -6
@@ -1243,6 +1243,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,int i915_gem_ringbuffer_submission_final (struct i915_execbuffer_params *params) {struct drm_i915_private *dev_priv = params->dev->dev_private; +struct drm_i915_gem_request *req = params->request;struct intel_engine_cs *ring = params->ring;u64 exec_start, exec_len;int ret;
@@ -1256,12 +1257,12 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)* Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch. */ - ret =intel_ring_invalidate_all_caches (params->request); + ret =intel_ring_invalidate_all_caches (req);if (ret) goto error;/* Switch to the correct context for the batch */ - ret =i915_switch_context (params->request); + ret =i915_switch_context (req);if (ret) goto error;
@@ -1270,7 +1271,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) { - ret =intel_ring_begin (params->request,4 ); + ret =intel_ring_begin (req,4 );if (ret) goto error;
@@ -1284,7 +1285,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params) }if (params->args_flags &I915_EXEC_GEN7_SOL_RESET ) { - ret =i915_reset_gen7_sol_offsets (params->dev, params->request); + ret =i915_reset_gen7_sol_offsets (params->dev, req);if (ret) goto error; }
@@ -1293,13 +1294,13 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params) exec_start = params->batch_obj_vm_offset + params->args_batch_start_offset; - ret = ring->dispatch_execbuffer (params->request, + ret = ring->dispatch_execbuffer (req, exec_start, exec_len, params->dispatch_flags);if (ret) goto error; -trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); +trace_i915_gem_ring_dispatch (req, params->dispatch_flags);i915_gem_execbuffer_retire_commands (params);
drivers/gpu/drm/i915/intel_lrc.c +6 -5
@@ -929,7 +929,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,int intel_execlists_submission_final (struct i915_execbuffer_params *params) {struct drm_i915_private *dev_priv = params->dev->dev_private; -struct intel_ringbuffer *ringbuf = params->request->ringbuf; +struct drm_i915_gem_request *req = params->request; +struct intel_ringbuffer *ringbuf = req->ringbuf;struct intel_engine_cs *ring = params->ring;u64 exec_start;int ret;
@@ -941,13 +942,13 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)* Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch. */ - ret =logical_ring_invalidate_all_caches (params->request); + ret =logical_ring_invalidate_all_caches (req);if (ret)return ret;if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) { - ret =intel_logical_ring_begin (params->request,4 ); + ret =intel_logical_ring_begin (req,4 );if (ret)return ret;
@@ -963,11 +964,11 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params) exec_start = params->batch_obj_vm_offset + params->args_batch_start_offset; - ret = ring->emit_bb_start (params->request, exec_start, params->dispatch_flags); + ret = ring->emit_bb_start (req, exec_start, params->dispatch_flags);if (ret)return ret; -trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); +trace_i915_gem_ring_dispatch (req, params->dispatch_flags);i915_gem_execbuffer_retire_commands (params);
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 05/38] drm/i915: Cache request pointer in *_submission_final()
Quoted 111 lines · On 01/11/2016 10:42 AM, [email protected] wrote:
From: Dave Gordon <[email protected]>
Keep a local copy of the request pointer in the
_final()functions rather than dereferencing the params block repeatedly.v3: New patch in series.
For: VIZ-1587
Signed-off-by Dave Gordon <[email protected]>Signed-off-by John Harrison <[email protected]>--- drivers/gpu/drm/i915/i915_gem_execbuffer.c | 13 +++++++------ drivers/gpu/drm/i915/intel_lrc.c | 11 ++++++----- 2 files changed, 13 insertions(+), 11 deletions(-)drivers/gpu/drm/i915/i915_gem_execbuffer.c +7
-6@@ -1243,6 +1243,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,int i915_gem_ringbuffer_submission_final (struct i915_execbuffer_params *params) {struct drm_i915_private *dev_priv = params->dev->dev_private; +struct drm_i915_gem_request *req = params->request;struct intel_engine_cs *ring = params->ring;u64 exec_start, exec_len;int ret;@@ -1256,12 +1257,12 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)* Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch. */ - ret =intel_ring_invalidate_all_caches (params->request); + ret =intel_ring_invalidate_all_caches (req);if (ret) goto error;/* Switch to the correct context for the batch */ - ret =i915_switch_context (params->request); + ret =i915_switch_context (req);if (ret) goto error;@@ -1270,7 +1271,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) { - ret =intel_ring_begin (params->request,4 ); + ret =intel_ring_begin (req,4 );if (ret) goto error;@@ -1284,7 +1285,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params) }if (params->args_flags &I915_EXEC_GEN7_SOL_RESET ) { - ret =i915_reset_gen7_sol_offsets (params->dev, params->request); + ret =i915_reset_gen7_sol_offsets (params->dev, req);if (ret) goto error; }@@ -1293,13 +1294,13 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params) exec_start = params->batch_obj_vm_offset + params->args_batch_start_offset; - ret = ring->dispatch_execbuffer (params->request, + ret = ring->dispatch_execbuffer (req, exec_start, exec_len, params->dispatch_flags);if (ret) goto error; -trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); +trace_i915_gem_ring_dispatch (req, params->dispatch_flags);i915_gem_execbuffer_retire_commands (params);drivers/gpu/drm/i915/intel_lrc.c +4
-3@@ -929,7 +929,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,int intel_execlists_submission_final (struct i915_execbuffer_params *params) {struct drm_i915_private *dev_priv = params->dev->dev_private; -struct intel_ringbuffer *ringbuf = params->request->ringbuf; +struct drm_i915_gem_request *req = params->request; +struct intel_ringbuffer *ringbuf = req->ringbuf;struct intel_engine_cs *ring = params->ring;u64 exec_start;int ret;@@ -941,13 +942,13 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)* Unconditionally invalidate gpu caches and ensure that we do flush * any residual writes from the previous batch. */ - ret =logical_ring_invalidate_all_caches (params->request); + ret =logical_ring_invalidate_all_caches (req);if (ret)return ret;if (ring == &dev_priv->ring[RCS ] && params->instp_mode != dev_priv->relative_constants_mode) { - ret =intel_logical_ring_begin (params->request,4 ); + ret =intel_logical_ring_begin (req,4 );if (ret)return ret;@@ -963,11 +964,11 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params) exec_start = params->batch_obj_vm_offset + params->args_batch_start_offset;
drivers/gpu/drm/i915/intel_lrc.c int intel_execlists_submission_final(struct i915_execbuffer_params *params) · line 974
Quoted 10 lines
drivers/gpu/drm/i915/intel_lrc.c +2
-2- ret = ring->emit_bb_start (params->request, exec_start, params->dispatch_flags); + ret = ring->emit_bb_start (req, exec_start, params->dispatch_flags);if (ret)return ret; -trace_i915_gem_ring_dispatch (params->request, params->dispatch_flags); +trace_i915_gem_ring_dispatch (req, params->dispatch_flags);i915_gem_execbuffer_retire_commands (params);
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 6/38 drm/i915: Re-instate request->uniq because it is extremely useful
In reply to [email protected]From: John Harrison <[email protected]>
The seqno value cannot always be used when debugging issues via trace points. This is because it can be reset back to start, especially during TDR type tests. Also, when the scheduler arrives the seqno is only valid while a given request is executing on the hardware. While the request is simply queued waiting for submission, it's seqno value will be zero (meaning invalid).
v4: Wrapped a long line to keep the style checker happy.
For: VIZ-5115
--- drivers/gpu/drm/i915/i915_drv.h | 5 +++++ drivers/gpu/drm/i915/i915_gem.c | 4 +++- drivers/gpu/drm/i915/i915_trace.h | 25 +++++++++++++++++-------- 3 files changed, 25 insertions(+), 9 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +5 -0
@@ -1963,6 +1963,8 @@ struct drm_i915_private {/* perform PHY state sanity checks? */ bool chv_phy_assert[2 ]; +uint32_t request_uniq; +/* * NOTE: This is the dri1/ums dungeon, don't add stuff here. Your patch * will be rejected. Instead look for a better place.
@@ -2210,6 +2212,9 @@ struct drm_i915_gem_request {/** GEM sequence number associated with this request. */ uint32_t seqno; +/* Unique identifier which can be used for trace points & debug */ +uint32_t uniq; +/** Position in the ringbuffer of the start of the request */ u32 head;
drivers/gpu/drm/i915/i915_gem.c +3 -1
@@ -2904,7 +2904,8 @@ static void i915_gem_request_fence_value_str(struct fence *req_fence, req =container_of (req_fence,typeof (*req), fence); -snprintf (str, size,"%d [%d]" , req->fence.seqno, req->seqno); +snprintf (str, size,"%d [%d:%d]" , req->fence.seqno, req->uniq, + req->seqno); }static const struct fence_ops i915_gem_request_fops = {
@@ -2980,6 +2981,7 @@ int i915_gem_request_alloc(struct intel_engine_cs *ring, req->i915 = dev_priv; req->ring = ring; + req->uniq = dev_priv->request_uniq++; req->ctx = ctx;i915_gem_context_reference (req->ctx);
drivers/gpu/drm/i915/i915_trace.h +17 -8
@@ -469,6 +469,7 @@ TRACE_EVENT(i915_gem_ring_sync_to, __field(u32, dev) __field(u32, sync_from) __field(u32, sync_to) + __field(u32, uniq_to)__field (u32, seqno) ),
@@ -476,13 +477,14 @@ TRACE_EVENT(i915_gem_ring_sync_to, __entry->dev = from->dev->primary->index; __entry->sync_from = from->id; __entry->sync_to = to_req->ring->id; + __entry->uniq_to = to_req->uniq; __entry->seqno =i915_gem_request_get_seqno (req); ), -TP_printk ("dev=%u, sync-from=%u, sync-to=%u, seqno=%u" , +TP_printk ("dev=%u, sync-from=%u, sync-to=%u, seqno=%u, to_uniq=%u" , __entry->dev, __entry->sync_from, __entry->sync_to, - __entry->seqno) + __entry->seqno, __entry->uniq_to) ); TRACE_EVENT(i915_gem_ring_dispatch,
@@ -517,6 +519,7 @@ TRACE_EVENT(i915_gem_ring_flush,TP_STRUCT__entry (__field (u32 ,dev )__field (u32, ring) +__field (u32, uniq)__field (u32, invalidate)__field (u32, flush) ),
@@ -524,12 +527,13 @@ TRACE_EVENT(i915_gem_ring_flush,TP_fast_assign ( __entry->dev = req->ring->dev->primary->index; __entry->ring = req->ring->id; + __entry->uniq = req->uniq; __entry->invalidate = invalidate; __entry->flush = flush; ), -TP_printk ("dev=%u, ring=%x, invalidate=%04x, flush=%04x" , - __entry->dev, __entry->ring, +TP_printk ("dev=%u, ring=%x, request=%u, invalidate=%04x, flush=%04x" , + __entry->dev, __entry->ring, __entry->uniq, __entry->invalidate, __entry->flush) );
@@ -540,6 +544,7 @@ DECLARE_EVENT_CLASS(i915_gem_request,TP_STRUCT__entry (__field (u32 ,dev )__field (u32, ring) +__field (u32, uniq)__field (u32, seqno) ),
@@ -548,11 +553,13 @@ DECLARE_EVENT_CLASS(i915_gem_request,i915_gem_request_get_ring (req); __entry->dev = ring->dev->primary->index; __entry->ring = ring->id; + __entry->uniq = req ? req->uniq :0 ; __entry->seqno =i915_gem_request_get_seqno (req); ), -TP_printk ("dev=%u, ring=%u, seqno=%u" , - __entry->dev, __entry->ring, __entry->seqno) +TP_printk ("dev=%u, ring=%u, uniq=%u, seqno=%u" , + __entry->dev, __entry->ring, __entry->uniq, + __entry->seqno) ); DEFINE_EVENT(i915_gem_request, i915_gem_request_add,
@@ -601,6 +608,7 @@ TRACE_EVENT(i915_gem_request_wait_begin,TP_STRUCT__entry (__field (u32 ,dev )__field (u32, ring) +__field (u32, uniq)__field (u32, seqno)__field (bool, blocking) ),
@@ -616,13 +624,14 @@ TRACE_EVENT(i915_gem_request_wait_begin,i915_gem_request_get_ring (req); __entry->dev = ring->dev->primary->index; __entry->ring = ring->id; + __entry->uniq = req ? req->uniq :0 ; __entry->seqno =i915_gem_request_get_seqno (req); __entry->blocking =mutex_is_locked (&ring->dev->struct_mutex); ), -TP_printk ("dev=%u, ring=%u, seqno=%u, blocking=%s" , - __entry->dev, __entry->ring, +TP_printk ("dev=%u, ring=%u, uniq=%u, seqno=%u, blocking=%s" , + __entry->dev, __entry->ring, __entry->uniq, __entry->seqno, __entry->blocking ?"yes (NB)" :"no" ) );
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 06/38] drm/i915: Re-instate request->uniq because it is extremely useful
Quoted 8 lines · On Mon, Jan 11, 2016 at 06:42:35PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
The seqno value cannot always be used when debugging issues via trace points. This is because it can be reset back to start, especially during TDR type tests. Also, when the scheduler arrives the seqno is only valid while a given request is executing on the hardware. While the request is simply queued waiting for submission, it's seqno value will be zero (meaning invalid).
Even with per-context seqno that can be assigned before execution as we know that requests within a context cannot be reordered? -Chris
Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 06/38] drm/i915: Re-instate request->uniq because it is extremely useful
Quoted 9 lines · Chris Wilson
Quoted 8 lines · On Mon, Jan 11, 2016 at 06:42:35PM +0000, [email protected] wrote:
From: John Harrison <[email protected]>
The seqno value cannot always be used when debugging issues via trace points. This is because it can be reset back to start, especially during TDR type tests. Also, when the scheduler arrives the seqno is only valid while a given request is executing on the hardware. While the request is simply queued waiting for submission, it's seqno value will be zero (meaning invalid).
Quoted 4 lines
Even with per-context seqno that can be assigned before execution as we know that requests within a context cannot be reordered? -Chris
Firstly, we do not have per-context seqno values at the moment. However, even if we did that would make a per request unique value even more useful as the only way to identify a given request then would be through a context pointer and seqno pair which would make scanning through debug traces even worse.
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
PATCH v4 7/38 drm/i915: Start of GPU scheduler
In reply to [email protected]From: John Harrison <[email protected]>
Initial creation of scheduler source files. Note that this patch implements most of the scheduler functionality but does not hook it in to the driver yet. It also leaves the scheduler code in 'pass through' mode so that even when it is hooked in, it will not actually do very much. This allows the hooks to be added one at a time in bite size chunks and only when the scheduler is finally enabled at the end does anything start happening.
The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.
The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.
The scheduler is notified when each batch buffer completes and updates its internal tracking accordingly. At the end of the completion interrupt processing, if any scheduler tracked batches were processed, the scheduler's deferred worker thread is woken up. This can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.
When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.
If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate.
Note that this patch does not implement pre-emptive scheduling. Only basic scheduling by re-ordering batch buffer submission is currently implemented. Pre-emption of actively executing batch buffers comes in the next patch series.
v2: Changed priority levels to +/-1023 due to feedback from Chris Wilson.
Removed redundant index from scheduler node.
Changed time stamps to use jiffies instead of raw monotonic. This provides lower resolution but improved compatibility with other i915 code.
Major re-write of completion tracking code due to struct fence conversion. The scheduler no longer has it's own private IRQ handler but just lets the existing request code handle completion events. Instead, the scheduler now hooks into the request notify code to be told when a request has completed.
Reduced driver mutex locking scope. Removal of scheduler nodes no longer grabs the mutex lock.
v3: Refactor of dependency generation to make the code more readable. Also added in read-read optimisation support - i.e., don't treat a shared read-only buffer as being a dependency.
Allowed the killing of queued nodes rather than only flying ones.
v4: Updated the commit message to better reflect the current state of the code. Downgraded some BUG_ONs to WARN_ONs. Used the correct array memory allocator function (kmalloc_array instead of kmalloc). Corrected the format of some comments. Wrapped some lines differently to keep the style checker happy.
Fixed a WARN_ON when killing nodes. The dependency removal code checks that nodes being destroyed do not have any oustanding dependencies (which would imply they should not have been executed yet). In the case of nodes being destroyed, e.g. due to context banning, then this might well be the case - they have not been executed and do indeed have outstanding dependencies.
Re-instated the code to disble interrupts when not in use. The underlying problem causing broken IRQ reference counts seems to have been fixed now.
For: VIZ-1587
--- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/i915_drv.h | 4 + drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_scheduler.c | 797 ++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 91 ++++ 5 files changed, 898 insertions(+) create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
drivers/gpu/drm/i915/Makefile +1 -0
@@ -10,6 +10,7 @@ ccflags-y := -Werror i915-y := i915_drv.o \ i915_irq.o \ i915_params.o \ + i915_scheduler.o \ i915_suspend.o \ i915_sysfs.o \ intel_csr.o \
drivers/gpu/drm/i915/i915_drv.h +4 -0
@@ -1695,6 +1695,8 @@ struct i915_execbuffer_params {struct drm_i915_gem_request *request; }; +struct i915_scheduler ; +/* used in computing the new watermarks state */ struct intel_wm_config {unsigned int num_pipes_active;
@@ -1947,6 +1949,8 @@ struct drm_i915_private {struct i915_runtime_pm pm; +struct i915_scheduler *scheduler; +/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */ struct { int (*execbuf_submit)(struct i915_execbuffer_params *params,
drivers/gpu/drm/i915/i915_gem.c +5 -0
@@ -37,6 +37,7 @@#include <linux/swap.h> #include <linux/pci.h> #include <linux/dma-buf.h> +#include "i915_scheduler.h" #define RQ_BUG_ON (expr)
@@ -5242,6 +5243,10 @@ int i915_gem_init(struct drm_device *dev)*/ intel_uncore_forcewake_get (dev_priv,FORCEWAKE_ALL ); + ret =i915_scheduler_init (dev); +if (ret) + goto out_unlock; + ret =i915_gem_init_userptr (dev);if (ret) goto out_unlock;
drivers/gpu/drm/i915/i915_scheduler.c new file +797 -0
797 lines not on this page · raw message
drivers/gpu/drm/i915/i915_scheduler.h new file +91 -0
91 lines not on this page · raw message
Signature · 6 lines
1.9.1 _______________________________________________ Intel-gfx mailing list [email protected] http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Joonas Lahtinen
maintainerRe: [PATCH v4 07/38] drm/i915: Start of GPU scheduler
Hi,
Comments below this pre text.
Many of the comments are related to the indent and style of the code. That stuff is important to fix for future maintainability. In order for the future review to be more effective, I'd like to next see a v5 of the series where the code quality concerns have been addressed, patches squashed to be actual reviewable chunks and appropriate kerneldoc being added.
To give an idea of proper slicing of patches, first produce a no-op scheduler, adding the extra function calls where needed and still keeping the scheduling completely linear. Second patch could introduce out of order submitting, third one priority bumping, fourth pre-empting and so on. That way, each patch extends the functionality and is itself already mergeable. That way I've been able to go through and understand the existing code, and I can actually review (other than just nag about indent and coding style) if the changes are appropriate to bring in the functionality desired.
In the current split, for me or anyone who did not participate writing the code, it is otherwise too confusing to try to guess what future changes might make each piece of code make sense, and which will be redundant in the future too. There is no value in splitting code to chunks that are not itself functional.
Regards, Joonas
Quoted 140 lines · On Mon, 2016-01-11 at 18:42 +0000, [email protected] wrote:
From: John Harrison <[email protected]>
Initial creation of scheduler source files. Note that this patch implements most of the scheduler functionality but does not hook it in
to the driver yet. It also leaves the scheduler code in 'pass through'
mode so that even when it is hooked in, it will not actually do very much. This allows the hooks to be added one at a time in bite size chunks and only when the scheduler is finally enabled at the end does anything start happening.The general theory of operation is that when batch buffers are submitted to the driver, the
execbuffer()code packages up all the information required to execute the batch buffer at a later time. This
package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in
use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed
straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.The scheduler is notified when each batch buffer completes and updates
its internal tracking accordingly. At the end of the completion interrupt processing, if any scheduler tracked batches were processed,
the scheduler's deferred worker thread is woken up. This can do more involved processing such as actually removing completed nodes from the
queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.
If a suitable node is found then it is sent to
execbuff_final()for submission to the hardware. The in flight count is then re-checked and
a new node popped from the list if appropriate.Note that this patch does not implement pre-emptive scheduling. Only basic scheduling by re-ordering batch buffer submission is currently implemented. Pre-emption of actively executing batch buffers comes in the next patch series.
v2: Changed priority levels to +/-1023 due to feedback from Chris Wilson.
Removed redundant index from scheduler node.
Changed time stamps to use jiffies instead of raw monotonic. This provides lower resolution but improved compatibility with other i915 code.
Major re-write of completion tracking code due to struct fence conversion. The scheduler no longer has it's own private IRQ handler but just lets the existing request code handle completion events. Instead, the scheduler now hooks into the request notify code to be told when a request has completed.
Reduced driver mutex locking scope. Removal of scheduler nodes no longer grabs the mutex lock.
v3: Refactor of dependency generation to make the code more readable. Also added in read-read optimisation support - i.e., don't treat a shared read-only buffer as being a dependency.
Allowed the killing of queued nodes rather than only flying ones.
v4: Updated the commit message to better reflect the current state of the code. Downgraded some
BUG_ONstoWARN_ONs. Used the correct array memory allocator function (kmalloc_arrayinstead of kmalloc). Corrected the format of some comments. Wrapped some lines differently to keep the style checker happy.Fixed a
WARN_ONwhen killing nodes. The dependency removal code checks
that nodes being destroyed do not have any oustanding dependencies (which would imply they should not have been executed yet). In the case of nodes being destroyed, e.g. due to context banning, then this might well be the case - they have not been executed and do indeed have outstanding dependencies.Re-instated the code to disble interrupts when not in use. The underlying problem causing broken IRQ reference counts seems to have been fixed now.
For: VIZ-1587
Signed-off-by John Harrison <[email protected]>---
drivers/gpu/drm/i915/Makefile | 1 +
drivers/gpu/drm/i915/i915_drv.h | 4 +
drivers/gpu/drm/i915/i915_gem.c | 5 +
drivers/gpu/drm/i915/i915_scheduler.c | 797
++++++++++++++++++++++++++++++++++
drivers/gpu/drm/i915/i915_scheduler.h | 91 ++++5 files changed, 898 insertions(+)create mode 100644
drivers/gpu/drm/i915/i915_scheduler.ccreate mode 100644drivers/gpu/drm/i915/i915_scheduler.hdrivers/gpu/drm/i915/Makefile +1
-0@@ -10,6 +10,7 @@ ccflags-y := -Werror i915-y := i915_drv.o \ i915_irq.o \ i915_params.o \ + i915_scheduler.o \ i915_suspend.o \ i915_sysfs.o \ intel_csr.o \drivers/gpu/drm/i915/i915_drv.h +0
-0@@ -1695,6 +1695,8 @@ struct i915_execbuffer_params { struct drm_i915_gem_request *request; };
drivers/gpu/drm/i915/i915_drv.h struct i915_execbuffer_params { · line 1699
Quoted 2 lines
drivers/gpu/drm/i915/i915_drv.h +2
-0+struct i915_scheduler ; +
Rather add "i915_scheduler.h" include at the top and eliminate circular include dependencies. This is needed for the next comment.
drivers/gpu/drm/i915/i915_drv.h struct drm_i915_private { · line 1953
Quoted 9 lines
/* used in computing the new watermarks state */
structintel_wm_config{
unsigned intnum_pipes_active;drivers/gpu/drm/i915/i915_drv.h +0
-0@@ -1947,6 +1949,8 @@ struct drm_i915_private {
structi915_runtime_pmpm;drivers/gpu/drm/i915/i915_drv.h +2
-0+struct i915_scheduler *scheduler; +
As the scheduler is going to be enabled on all platforms to an extent, no point in making it a pointer. Just making it member like "pm" is the best, this also requires the above include change.
Quoted 12 lines
/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */ struct {
int (*execbuf_submit)(structi915_execbuffer_params*params,drivers/gpu/drm/i915/i915_gem.c +1
-0@@ -37,6 +37,7 @@ #include #include #include +#include "i915_scheduler.h"
Quoted 1 line
This should go before any <linux/...> not to mask missing includes from the header itself, so correct place would be right after #include "i915_trace.h"
Quoted 36 lines
#define
RQ_BUG_ON(expr)drivers/gpu/drm/i915/i915_gem.c +0
-0@@ -5242,6 +5243,10 @@ int i915_gem_init(struct drm_device *dev)*/ intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);drivers/gpu/drm/i915/i915_gem.c +4
-0+ ret =i915_scheduler_init (dev); +if (ret) + goto out_unlock; +ret =
i915_gem_init_userptr(dev);
if (ret)
gotoout_unlock;drivers/gpu/drm/i915/i915_scheduler.c new file +17
-0@@ -0,0 +1,797 @@ +/* +* Copyright (c) 2014 Intel Corporation +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice (including the next +* paragraph) shall be included in all copies or substantial portions of the +* Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
drivers/gpu/drm/i915/i915_scheduler.c line 27
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +* IN THE SOFTWARE. +* +*/ + +#include "i915_drv.h" +#include "intel_drv.h" +#include "i915_scheduler.h"
Again, this include should be the top one, should not require any includes before it.
drivers/gpu/drm/i915/i915_scheduler.c line 39
Quoted 12 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +12
-0+ +static int i915_scheduler_fly_node(struct i915_scheduler_queue_entry *node); +static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler, + struct i915_scheduler_queue_entry *remove); +static int i915_scheduler_submit(struct intel_engine_cs *ring, + bool is_locked); +static uint32_t i915_scheduler_count_flying(struct i915_scheduler *scheduler, + struct intel_engine_cs *ring); +static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler); +static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler, +struct i915_scheduler_queue_entry *target, +uint32_t bump);
Do not indent the function names like this, it becomes unmaintainable and messy very fast if somebody adds a new function with a more complex return type, which is a very likely thing to happen.
What I would do is move all the helper functions here at the top and order them so that the forward declarations are not needed, at least i915_scheduler_fly_node is not used before its definition.
drivers/gpu/drm/i915/i915_scheduler.c line 45
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +6
-0+ +int i915_scheduler_init (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; + struct i915_scheduler *scheduler = dev_priv->scheduler; + int r;
Again, this indent is a no-go. I'll not mention it on further functions, assume it to be fixed for next revision.
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +8
-0+ +if (scheduler) +return 0 ; + + scheduler =kzalloc (sizeof (*scheduler),GFP_KERNEL ); +if (!scheduler) +return -ENOMEM ; +
drivers/gpu/drm/i915/i915_scheduler.c line 63
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+spin_lock_init (&scheduler->lock); + +for (r =0 ; r <I915_NUM_RINGS ; r++) +INIT_LIST_HEAD (&scheduler->node_queue[r]); + +/* Default tuning values: */ + scheduler->priority_level_min = -1023; + scheduler->priority_level_max = 1023; + scheduler->priority_level_preempt =900 ; + scheduler->min_flying = 2;
This kind of indent is tolerable because it is a contained code block, but not needed either.
Quoted 19 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +19
-0+ + dev_priv->scheduler = scheduler; + +return 0 ; +} + +static void i915_generate_dependencies (struct i915_scheduler *scheduler, + struct i915_scheduler_queue_entry *node, + uint32_t ring) +{ +struct i915_scheduler_obj_entry *this, *that; + struct i915_scheduler_queue_entry *test; + int i, j; + bool found; + +list_for_each_entry (test, &scheduler->node_queue[ring], link) { +if (I915_SQS_IS_COMPLETE (test)) +continue ; +
drivers/gpu/drm/i915/i915_scheduler.c line 92
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+/* +* Batches on the same ring for the same +* context must be kept in order. +*/ + found = (node->params.ctx == test->params.ctx) && + (node->params.ring == test->params.ring); + +if (found) + goto depends; +
This is not needed.
drivers/gpu/drm/i915/i915_scheduler.c line 97
Quoted 5 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +5
-0+/* +* Batches working on the same objects must +* be kept in order. +*/ +for (i =0 ; (i < node->num_objs) && !found; i++) {
As the test is here already ---^
drivers/gpu/drm/i915/i915_scheduler.c line 104
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+ this = node->saved_objects + i; + +for (j =0 ; j < test->num_objs; j++) { + that = test->saved_objects + j; + +if (this->obj != that->obj) +continue ;
How about VMAs? There might be multiple mappings to an object, isn't it enough to depend on the required VMA instead of the whole object?
drivers/gpu/drm/i915/i915_scheduler.c line 114
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ +/* Only need to worry about writes */ +if (this->read_only && that->read_only) +continue ; + + found = true; +break ; + } + } +
The following block is not needed.
drivers/gpu/drm/i915/i915_scheduler.c line 118
Quoted 4 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +4
-0+if (!found) +continue ; + +depends:
Rather like this, in order to avoid a goto label;
if (found) {
...
drivers/gpu/drm/i915/i915_scheduler.c line 123
Quoted 5 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +5
-0+ node->dep_list[node->num_deps] = test; + node->num_deps++; + } +} +
Please add a brief kerneldoc above each function in the header, it's required. Adding it to non-trivial inline helper functions too will make reviewing much easier.
Quoted 4 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +4
-0+int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe) +{ +struct drm_i915_private *dev_priv = qe->params.dev->dev_private; + struct i915_scheduler *scheduler = dev_priv->scheduler;
drivers/gpu/drm/i915/i915_scheduler.c line 137
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ struct intel_engine_cs *ring = qe->params.ring; + struct i915_scheduler_queue_entry *node; + struct i915_scheduler_queue_entry *test; + unsigned long flags; + bool not_flying; + int i, r; + int incomplete = 0; + +WARN_ON (!scheduler); +
This kind of situations should have a be a BUG_ON, because scheduler being zero is literally going to cause an OOPS in the next dereference which is going to happen unconditionally. WARN + OOPS is kind of what BUG_ON should be used avoid. But this should be removed anyway after scheduler is made a data member of dev_priv.
drivers/gpu/drm/i915/i915_scheduler.c line 138
Quoted 1 line
drivers/gpu/drm/i915/i915_scheduler.c new file +1
-0+if (1 /*i915.scheduler_override & i915_so_direct_submit*/ ) {
I assume this is going to be addressed in a future commit. Could have been introduced in this patch, too.
drivers/gpu/drm/i915/i915_scheduler.c line 144
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +6
-0+int ret; + + scheduler->flags[qe->params.ring->id] |= i915_sf_submitting; + ret = dev_priv->gt.execbuf_final (&qe->params); + scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting; +
The kerneldoc should mention locking requirements of this function.
drivers/gpu/drm/i915/i915_scheduler.c line 157
Quoted 13 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +13
-0+/* +* Don't do any clean up on failure because the caller will +* do it all anyway. +*/ +if (ret) +return ret; + +/* Free everything that is owned by the QE structure: */ +kfree (qe->params.cliprects); +if (qe->params.dispatch_flags &I915_DISPATCH_SECURE ) +i915_gem_execbuff_release_batch_obj (qe->params.batch_obj); + +return 0 ;
Above piece of code looks like its own function, so it should probably be one.
drivers/gpu/drm/i915/i915_scheduler.c line 164
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+ } + + node =kmalloc (sizeof (*node),GFP_KERNEL ); +if (!node) +return -ENOMEM ; + + *node = *qe;
Any reason we can't simply move ownership of qe? If not, I'd rather make a clone function
Quoted 16 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +16
-0+INIT_LIST_HEAD (&node->link); + node->status = i915_sqs_queued; + node->stamp = jiffies; +i915_gem_request_reference (node->params.request); + +/* Need to determine the number of incomplete entries in the list as +* that will be the maximum size of the dependency list. +* +* Note that the allocation must not be made with the spinlock acquired +* as kmalloc can sleep. However, the unlock/relock is safe because no +* new entries can be queued up during the unlock as the i915 driver +* mutex is still held. Entries could be removed from the list but that +* just means the dep_list will be over-allocated which is fine. +*/ +spin_lock_irqsave (&scheduler->lock, flags); +for (r =0 ; r <I915_NUM_RINGS ; r++) {
drivers/gpu/drm/i915/i915_scheduler.c line 190
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+list_for_each_entry (test, &scheduler->node_queue[r], link) { +if (I915_SQS_IS_COMPLETE (test)) +continue ; + + incomplete++; + } + } + +/* Temporarily unlock to allocate memory: */ +spin_unlock_irqrestore (&scheduler->lock, flags);
I'd make the above piece of code a helper, these stats are to be counted for debugfs anyway, too?
Quoted 9 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +9
-0+if (incomplete) { + node->dep_list =kmalloc_array (incomplete, + sizeof(*node->dep_list), + GFP_KERNEL); +if (!node->dep_list) { +kfree (node); +return -ENOMEM ; + } + }else
drivers/gpu/drm/i915/i915_scheduler.c line 209
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ node->dep_list =NULL ; + +spin_lock_irqsave (&scheduler->lock, flags); + node->num_deps =0 ; + +if (node->dep_list) { +for (r =0 ; r <I915_NUM_RINGS ; r++) +i915_generate_dependencies (scheduler, node, r); + +WARN_ON (node->num_deps > incomplete);
Wouldn't this condition again lead to a crash? If so, should be BUG_ON to cause that crash as early as possible. WARN_ON is only good if there is a way of coping with the situation and no imminent system crash is bound to happen.
drivers/gpu/drm/i915/i915_scheduler.c line 216
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+ } + +if (node->priority > scheduler->priority_level_max) + node->priority = scheduler->priority_level_max; +else if (node->priority < scheduler->priority_level_min) + node->priority = scheduler->priority_level_min; +
There is clamp_val macro in linux/kernel.h .
Quoted 20 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +20
-0+if ((node->priority >0 ) && node->num_deps) { +i915_scheduler_priority_bump_clear (scheduler); + +for (i =0 ; i < node->num_deps; i++) +i915_scheduler_priority_bump (scheduler, + node->dep_list[i], node->priority); + } + +list_add_tail (&node->link, &scheduler->node_queue[ring->id]); + + not_flying =i915_scheduler_count_flying (scheduler, ring) < + scheduler->min_flying; + +spin_unlock_irqrestore (&scheduler->lock, flags); + +if (not_flying) +i915_scheduler_submit (ring, true); + +return 0 ; +}
drivers/gpu/drm/i915/i915_scheduler.c line 246
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ +static int i915_scheduler_fly_node (struct i915_scheduler_queue_entry *node) +{ +struct drm_i915_private *dev_priv = node->params.dev->dev_private; + struct i915_scheduler *scheduler = dev_priv->scheduler; + struct intel_engine_cs *ring; + +WARN_ON (!scheduler); +WARN_ON (!node); +WARN_ON (node->status != i915_sqs_popped);
Other states had their I915_SQS_IS_* macro, why some don't?
Quoted 51 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +51
-0+ + ring = node->params.ring; + +/* +* Add the node (which should currently be in state popped) to the +* front of the queue. This ensure that flying nodes are always held +* in hardware submission order. +*/ +list_add (&node->link, &scheduler->node_queue[ring->id]); + + node->status = i915_sqs_flying; + +if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) { + bool success = true; + + success = ring->irq_get (ring); +if (success) + scheduler->flags[ring->id] |= i915_sf_interrupts_enabled; +else +return -EINVAL ; + } + +return 0 ; +} + +/* +* Nodes are considered valid dependencies if they are queued on any ring or +* if they are in flight on a different ring. In flight on the same ring is no +* longer interesting for non-premptive nodes as the ring serialises execution. +* For pre-empting nodes, all in flight dependencies are valid as they must not +* be jumped by the act of pre-empting. +* +* Anything that is neither queued nor flying is uninteresting. +*/ +static inline bool i915_scheduler_is_dependency_valid ( +struct i915_scheduler_queue_entry *node,uint32_t idx) +{ +struct i915_scheduler_queue_entry *dep; + + dep = node->dep_list[idx]; +if (!dep) +return false; + +if (I915_SQS_IS_QUEUED (dep)) +return true; + +if (I915_SQS_IS_FLYING (dep)) { +if (node->params.ring != dep->params.ring) +return true; + } +
drivers/gpu/drm/i915/i915_scheduler.c line 307
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+return false; +} + +static uint32_t i915_scheduler_count_flying (struct i915_scheduler *scheduler, + struct intel_engine_cs *ring) +{ +struct i915_scheduler_queue_entry *node; + uint32_t flying = 0; + +list_for_each_entry (node, &scheduler->node_queue[ring->id], link)
This loop keeps popping up, it could use a define similar to the ones in i915_drv.h ;
#define for_each_hpd_pin(__pin) \
Quoted 4 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +4
-0+if (I915_SQS_IS_FLYING (node)) + flying++; + +return flying;
drivers/gpu/drm/i915/i915_scheduler.c line 321
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+} + +/* +* Add a popped node back in to the queue. For example, because the ring was +* hung when execfinal() was called and thus the ring submission needs to be +* retried later. +*/ +static void i915_scheduler_node_requeue (struct i915_scheduler_queue_entry *node) +{ +WARN_ON (!node);
if (WARN_ON(!node)) return;
Or rather no check at all, it's going to crash anyway even in the calling function if there's NULL, and it's internal function. It's relevant to check if the userspace.
Quoted 26 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +26
-0+WARN_ON (!I915_SQS_IS_FLYING (node)); + +/* Seqno will be reassigned on relaunch */ + node->params.request->seqno =0 ; + node->status = i915_sqs_queued; +} + +/* +* Give up on a node completely. For example, because it is causing the +* ring to hang or is using some resource that no longer exists. +*/ +static void i915_scheduler_node_kill (struct i915_scheduler_queue_entry *node) +{ +WARN_ON (!node); +WARN_ON (I915_SQS_IS_COMPLETE (node)); + + node->status = i915_sqs_dead; +} + +/* +* A sequence number has popped out of the hardware and the request handling +* code has mapped it back to a request and will mark that request complete. +* It also calls this function to notify the scheduler about the completion +* so the scheduler's node can be updated appropriately. +* Returns true if the request is scheduler managed, false if not. +*/
drivers/gpu/drm/i915/i915_scheduler.c line 357
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+bool i915_scheduler_notify_request (struct drm_i915_gem_request *req) +{ + struct drm_i915_private *dev_priv = to_i915(req->ring->dev); + struct i915_scheduler *scheduler = dev_priv->scheduler; +/* XXX: Need to map back from request to node */ +struct i915_scheduler_queue_entry *node =NULL ; + unsigned long flags; + +if (!node) +return false;
Not so sure if slicing the series down to an extent that functions are impossible to review, was a good idea. Idea is to slice things down, to reviewable pieces. It's hard to predict or keep looking forward the series what is going to come.
Quoted 32 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +32
-0+ +spin_lock_irqsave (&scheduler->lock, flags); + +WARN_ON (!I915_SQS_IS_FLYING (node)); + +/* Node was in flight so mark it as complete. */ +if (req->cancelled) + node->status = i915_sqs_dead; +else + node->status = i915_sqs_complete; + +spin_unlock_irqrestore (&scheduler->lock, flags); + +/* +* XXX: If the in-flight list is now empty then new work should be +* submitted. However, this function is called from interrupt context +* and thus cannot acquire mutex locks and other such things that are +* necessary for fresh submission. +*/ + +return true; +} + +int i915_scheduler_remove (struct intel_engine_cs *ring) +{ +struct drm_i915_private *dev_priv = ring->dev->dev_private; + struct i915_scheduler *scheduler = dev_priv->scheduler; + struct i915_scheduler_queue_entry *node, *node_next; + unsigned long flags; + int flying = 0, queued = 0; + int ret = 0; + bool do_submit;
drivers/gpu/drm/i915/i915_scheduler.c line 399
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ uint32_t min_seqno; + struct list_head remove; + +if (list_empty (&scheduler->node_queue[ring->id])) +return 0 ; + +spin_lock_irqsave (&scheduler->lock, flags); + +/* /i915_scheduler_dump_locked(ring, "remove/pre");/ */ +
This should not be here at all.
Quoted 18 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +18
-0+/* +* In the case where the system is idle, starting 'min_seqno' from a big +* number will cause all nodes to be removed as they are now back to +* being in-order. However, this will be a problem if the last one to +* complete was actually out-of-order as the ring seqno value will be +* lower than one or more completed buffers. Thus code looking for the +* completion of said buffers will wait forever. +* Instead, use the hardware seqno as the starting point. This means +* that some buffers might be kept around even in a completely idle +* system but it should guarantee that no-one ever gets confused when +* waiting for buffer completion. +*/ + min_seqno = ring->get_seqno (ring, true); + +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (I915_SQS_IS_QUEUED (node)) + queued++; +else if (I915_SQS_IS_FLYING (node))
drivers/gpu/drm/i915/i915_scheduler.c line 427
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ flying++; + else if (I915_SQS_IS_COMPLETE(node)) + continue; + +if (node->params.request->seqno ==0 ) +continue ; + +if (!i915_seqno_passed (node->params.request->seqno, min_seqno)) + min_seqno = node->params.request->seqno; + }
Couldn't these values be kept cached, instead of counting them at each function?
Quoted 45 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +45
-0+ +INIT_LIST_HEAD (&remove); +list_for_each_entry_safe (node, node_next, &scheduler->node_queue[ring->id], link) { +/* +* Only remove completed nodes which have a lower seqno than +* all pending nodes. While there is the possibility of the +* ring's seqno counting backwards, all higher buffers must +* be remembered so that the 'i915_seqno_passed()' test can +* report that they have in fact passed. +* +* NB: This is not true for 'dead' nodes. The GPU reset causes +* the software seqno to restart from its initial value. Thus +* the dead nodes must be removed even though their seqno values +* are potentially vastly greater than the current ring seqno. +*/ +if (!I915_SQS_IS_COMPLETE (node)) +continue ; + +if (node->status != i915_sqs_dead) { +if (i915_seqno_passed (node->params.request->seqno, min_seqno) && + (node->params.request->seqno != min_seqno)) +continue ; + } + +list_del (&node->link); +list_add (&node->link, &remove); + +/* Strip the dependency info while the mutex is still locked */ +i915_scheduler_remove_dependent (scheduler, node); + +continue ; + } + +/* +* Release the interrupt reference count if there are no longer any +* nodes to worry about. +*/ +if (!flying && !queued && + (scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) { + ring->irq_put (ring); + scheduler->flags[ring->id] &= ~i915_sf_interrupts_enabled; + } + +/* Launch more packets now? */ + do_submit = (queued >0 ) && (flying < scheduler->min_flying);
drivers/gpu/drm/i915/i915_scheduler.c line 482
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ +spin_unlock_irqrestore (&scheduler->lock, flags); + +if (!do_submit &&list_empty (&remove)) +return ret; + +mutex_lock (&ring->dev->struct_mutex); + +if (do_submit) + ret =i915_scheduler_submit (ring, true);
Confusing to have this at a remove function. Function naming needs to be reconsidered or moved out from here.
Quoted 86 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +86
-0+ +while (!list_empty (&remove)) { + node =list_first_entry (&remove,typeof (*node), link); +list_del (&node->link); + +/* +* The batch buffer must be unpinned before it is unreferenced +* otherwise the unpin fails with a missing vma!? + */priority_bump_clear + if (node->params.dispatch_flags & I915_DISPATCH_SECURE) + i915_gem_execbuff_release_batch_obj(node->params.batch_obj); + +/* Free everything that is owned by the node: */ +i915_gem_request_unreference (node->params.request); +kfree (node->params.cliprects); +kfree (node->dep_list); +kfree (node); + } + +mutex_unlock (&ring->dev->struct_mutex); + +return ret; +} + +static void i915_scheduler_priority_bump_clear (struct i915_scheduler *scheduler) +{ +struct i915_scheduler_queue_entry *node; +int i; + +/* +* Ensure circular dependencies don't cause problems and that a bump +* by object usage only bumps each using buffer once: +*/ +for (i =0 ; i <I915_NUM_RINGS ; i++) { +list_for_each_entry (node, &scheduler->node_queue[i], link) + node->bumped = false; + } +} + +static int i915_scheduler_priority_bump (struct i915_scheduler *scheduler, +struct i915_scheduler_queue_entry *target, +uint32_t bump) +{ +uint32_t new_priority; + int i, count; + +if (target->priority >= scheduler->priority_level_max) +return 1 ; + +if (target->bumped) +return 0 ; + + new_priority = target->priority + bump; +if ((new_priority <= target->priority) || + (new_priority > scheduler->priority_level_max)) + target->priority = scheduler->priority_level_max; +else + target->priority = new_priority; + + count =1 ; + target->bumped = true; + +for (i =0 ; i < target->num_deps; i++) { +if (!target->dep_list[i]) +continue ; + +if (target->dep_list[i]->bumped) +continue ; + + count +=i915_scheduler_priority_bump (scheduler, + target->dep_list[i], + bump); + } + +return count; +} + +static int i915_scheduler_pop_from_queue_locked (struct intel_engine_cs *ring, +struct i915_scheduler_queue_entry **pop_node, +unsigned long *flags) +{ + struct drm_i915_private *dev_priv = ring->dev->dev_private; + struct i915_scheduler *scheduler = dev_priv->scheduler; + struct i915_scheduler_queue_entry *best; + struct i915_scheduler_queue_entry *node; + int ret;
drivers/gpu/drm/i915/i915_scheduler.c line 578
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ int i; +bool any_queued; +bool has_local, has_remote, only_remote; + + *pop_node =NULL ; + ret = -ENODATA ; + + any_queued = false; + only_remote = false; + best =NULL ;
These should just be initialized in-place. But looking at the code forward.
Quoted 62 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +62
-0+ +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (!I915_SQS_IS_QUEUED (node)) +continue ; + any_queued = true; + + has_local = false; + has_remote = false; +for (i =0 ; i < node->num_deps; i++) { +if (!i915_scheduler_is_dependency_valid (node, i)) +continue ; + +if (node->dep_list[i]->params.ring == node->params.ring) + has_local = true; +else + has_remote = true; + } + +if (has_remote && !has_local) + only_remote = true; + +if (!has_local && !has_remote) { +if (!best || + (node->priority > best->priority)) + best = node; + } + } + +if (best) { +list_del (&best->link); + +INIT_LIST_HEAD (&best->link); + best->status = i915_sqs_popped; + + ret =0 ; + }else { +/* Can only get here if: +* (a) there are no buffers in the queue +* (b) all queued buffers are dependent on other buffers +* e.g. on a buffer that is in flight on a different ring +*/ +if (only_remote) { +/* The only dependent buffers are on another ring. */ + ret = -EAGAIN ; + }else if (any_queued) { +/* It seems that something has gone horribly wrong! */ +DRM_ERROR ("Broken dependency tracking on ring %d!\n" , + (int) ring->id); + } + } + +/* i915_scheduler_dump_queue_pop(ring, best); */ + + *pop_node = best; +return ret; +} + +static int i915_scheduler_submit (struct intel_engine_cs *ring,bool was_locked) +{ + struct drm_device *dev = ring->dev; +struct drm_i915_private *dev_priv = dev->dev_private; + struct i915_scheduler *scheduler = dev_priv->scheduler;
drivers/gpu/drm/i915/i915_scheduler.c line 650
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ struct i915_scheduler_queue_entry *node; + unsigned long flags; + int ret = 0, count = 0; + +if (!was_locked) { + ret =i915_mutex_lock_interruptible (dev); +if (ret) +return ret; + } +
I don't really fancy this construct. Should be moved outside of this function for proper lockdep tracking.
Quoted 5 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +5
-0+WARN_ON (!mutex_is_locked (&dev->struct_mutex)); + +spin_lock_irqsave (&scheduler->lock, flags); + +/* First time around, complain if anything unexpected occurs: */
drivers/gpu/drm/i915/i915_scheduler.c line 665
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ ret =i915_scheduler_pop_from_queue_locked (ring, &node, &flags); +if (ret) { +spin_unlock_irqrestore (&scheduler->lock, flags); + +if (!was_locked) +mutex_unlock (&dev->struct_mutex); + +return ret; + } +
Dropping the was_locked stuff, this should become a proper goto error label. e.g. out_unlock
drivers/gpu/drm/i915/i915_scheduler.c line 676
Quoted 11 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +11
-0+ do { + WARN_ON(!node); + WARN_ON(node->params.ring != ring); + WARN_ON(node->status != i915_sqs_popped); + count++; + + /* + * The call to pop above will have removed the node from the + * list. So add it back in and mark it as in flight. + */ + i915_scheduler_fly_node(node);
Why do we want to pull an object out of the list inside spin lock and push it back immediately in our critical code path? Seems like a waste for no obvious gain at this point. Why do not we rather just select an entry and modify it in-place, if it's going to stay in the same queue anyway.
drivers/gpu/drm/i915/i915_scheduler.c line 685
Quoted 9 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +9
-0+ + scheduler->flags[ring->id] |= i915_sf_submitting; +spin_unlock_irqrestore (&scheduler->lock, flags); + ret = dev_priv->gt.execbuf_final (&node->params); +spin_lock_irqsave (&scheduler->lock, flags); + scheduler->flags[ring->id] &= ~i915_sf_submitting; + +if (ret) { +int requeue =1 ;
Multipurpose variable, not really a good idea. And as commented further, should not exist at all.
drivers/gpu/drm/i915/i915_scheduler.c line 698
Quoted 13 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +13
-0+ +/* +* Oh dear! Either the node is broken or the ring is +* busy. So need to kill the node or requeue it and try +* again later as appropriate. +*/ + +switch (-ret) { +case ENODEV : +case ENOENT : +/* Fatal errors. Kill the node. */ + requeue =-1 ; +break ;
"break" indent is wrong.
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+ +case EAGAIN : +case EBUSY : +case EIO : +case ENOMEM : +case ERESTARTSYS : +case EINTR :
drivers/gpu/drm/i915/i915_scheduler.c line 715
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+/* Supposedly recoverable errors. */ +break ; + +default : +/* +* Assume the error is recoverable and hope +* for the best. +*/ +DRM_DEBUG_DRIVER ("<%s> Got unexpected error from execfinal(): %d!\n" , + ring->name, ret);
There's MISSING_CASE macro, should use it.
drivers/gpu/drm/i915/i915_scheduler.c line 114
Quoted 3 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +3
-0+break ; + } +
Just move the code below this point to the switch, no point having a switch to categorize your options and then doing bunch of ifs to execute code that could be in switch.
Quoted 9 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +9
-0+/* +* Check that the watchdog/reset code has not nuked +* the node while we weren't looking: +*/ +if (node->status == i915_sqs_dead) + requeue =0 ; + +if (requeue ==1 ) { +i915_scheduler_node_requeue (node);
drivers/gpu/drm/i915/i915_scheduler.c line 737
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+/* +* No point spinning if the ring is currently +* unavailable so just give up and come back +* later. +*/ +break ; + }else if (requeue ==-1 ) +i915_scheduler_node_kill (node); + } +
Ending here, this actual submission of a single node could go to its own helper function, these functions now become too long to follow, although they really are not doing anything complicated.
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +8
-0+/* Keep launching until the sky is sufficiently full. */ +if (i915_scheduler_count_flying (scheduler, ring) >= + scheduler->min_flying) +break ; + + ret =i915_scheduler_pop_from_queue_locked (ring, &node, &flags); + }while (ret ==0 ); +
drivers/gpu/drm/i915/i915_scheduler.c line 755
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+spin_unlock_irqrestore (&scheduler->lock, flags); + +if (!was_locked) +mutex_unlock (&dev->struct_mutex); + +/* Don't complain about not being able to submit extra entries */ +if (ret == -ENODATA ) + ret =0 ; + +return (ret <0 ) ? ret : count;
This is a combined error and success path, keeping to the convention of kernel drivers is preferred;
if (ret != -ENODATA) goto out_foo; return count;
out_unlock:
spin_unlock(...); out_foo: return ret;
Quoted 81 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +42
-0+} + +static int i915_scheduler_remove_dependent (struct i915_scheduler *scheduler, +struct i915_scheduler_queue_entry *remove) +{ + struct i915_scheduler_queue_entry *node; + int i, r; + int count = 0; + +/* +* Ensure that a node is not being removed which is still dependent +* upon other (not completed) work. If that happens, it implies +* something has gone very wrong with the dependency tracking! Note +* that there is no need to worry if this node has been explicitly +* killed for some reason - it might be being killed before it got +* sent to the hardware. +*/ +if (remove->status != i915_sqs_dead) { +for (i =0 ; i < remove->num_deps; i++) +if ((remove->dep_list[i]) && + (!I915_SQS_IS_COMPLETE(remove->dep_list[i]))) + count++; +WARN_ON (count); + } + +/* +* Remove this node from the dependency lists of any other node which +* might be waiting on it. +*/ +for (r =0 ; r <I915_NUM_RINGS ; r++) { +list_for_each_entry (node, &scheduler->node_queue[r], link) { +for (i =0 ; i < node->num_deps; i++) { +if (node->dep_list[i] != remove) +continue ; + + node->dep_list[i] =NULL ; + } + } + } + +return 0 ; +}drivers/gpu/drm/i915/i915_scheduler.h new file +33
-0@@ -0,0 +1,91 @@ +/* +* Copyright (c) 2014 Intel Corporation +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice (including the next +* paragraph) shall be included in all copies or substantial portions of the +* Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +* IN THE SOFTWARE. +* +*/ + +#ifndef _I915_SCHEDULER_H_ +#define _I915_SCHEDULER_H_ + +enum i915_scheduler_queue_status { +/* Limbo: */ + i915_sqs_none =0 , +/* Not yet submitted to hardware: */ + i915_sqs_queued, + /* Popped from queue, ready to fly: */
drivers/gpu/drm/i915/i915_scheduler.h line 43
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +10
-0+ i915_sqs_popped, +/* Sent to hardware for processing: */ + i915_sqs_flying, +/* Finished processing on the hardware: */ + i915_sqs_complete, +/* Killed by watchdog or catastrophic submission failure: */ + i915_sqs_dead, +/* Limit value for use with arrays/loops */ + i915_sqs_MAX +};
These should be UPPERCASE_FOR_ENUM_VALUES . See i915_drv.h for samples.
drivers/gpu/drm/i915/i915_scheduler.h line 49
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +6
-0+ +#define I915_SQS_IS_QUEUED (node) (((node)->status == i915_sqs_queued)) +#define I915_SQS_IS_FLYING (node) (((node)->status == i915_sqs_flying)) +#define I915_SQS_IS_COMPLETE (node) (((node)->status == i915_sqs_complete) || \ + ((node)->status == i915_sqs_dead)) +
Might be slightly confusing that name is IS_COMPLETE and there is actual COMPLETE value. Rather have the test like IS_DONE and then test for COMPLETE or DEAD, no confusion. Also, some states have their IS_* macro and others don't, is there going to be more?
Quoted 25 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +25
-0+struct i915_scheduler_obj_entry { + struct drm_i915_gem_object *obj; + bool read_only; +}; + +struct i915_scheduler_queue_entry { + struct i915_execbuffer_params params; +/* -1023 = lowest priority, 0 = default, 1023 = highest */ + int32_t priority; + struct i915_scheduler_obj_entry *saved_objects; + int num_objs; + bool bumped; + struct i915_scheduler_queue_entry **dep_list; + int num_deps; + enum i915_scheduler_queue_status status; + unsigned long stamp; + struct list_head link; +}; + +struct i915_scheduler { + struct list_head node_queue[I915_NUM_RINGS]; + uint32_t flags[I915_NUM_RINGS]; + spinlock_t lock; + +/* Tuning parameters: */
drivers/gpu/drm/i915/i915_scheduler.h line 84
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +10
-0+ int32_t priority_level_min; + int32_t priority_level_max; + int32_t priority_level_preempt; + uint32_t min_flying; +}; + +/* Flag bits for i915_scheduler::flags */ +enum { + i915_sf_interrupts_enabled = (1 << 0), + i915_sf_submitting = (1 << 1),
Again, should be uppercase. Also, enums to the beginning of file.
drivers/gpu/drm/i915/i915_scheduler.h line 91
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +7
-0+}; + +int i915_scheduler_init(struct drm_device *dev); +int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe); +bool i915_scheduler_notify_request(struct drm_i915_gem_request *req); + +#endif /* _I915_SCHEDULER_H_ */
_______________________________________________
Intel-gfx mailing list
[email protected]
http://lists.freedesktop.org/mailman/listinfo/intel-gfx
Re: [PATCH v4 07/38] drm/i915: Start of GPU scheduler
Quoted 326 lines · Joonas Lahtinen
Hi,
Comments below this pre text.
Many of the comments are related to the indent and style of the code. That stuff is important to fix for future maintainability. In order for the future review to be more effective, I'd like to next see a v5 of the series where the code quality concerns have been addressed, patches squashed to be actual reviewable chunks and appropriate kerneldoc being added.
To give an idea of proper slicing of patches, first produce a no-op scheduler, adding the extra function calls where needed and still keeping the scheduling completely linear. Second patch could introduce out of order submitting, third one priority bumping, fourth pre-empting and so on. That way, each patch extends the functionality and is itself already mergeable. That way I've been able to go through and understand the existing code, and I can actually review (other than just nag about indent and coding style) if the changes are appropriate to bring in the functionality desired.
In the current split, for me or anyone who did not participate writing the code, it is otherwise too confusing to try to guess what future changes might make each piece of code make sense, and which will be redundant in the future too. There is no value in splitting code to chunks that are not itself functional.
Regards, Joonas
Quoted 137 lines · On Mon, 2016-01-11 at 18:42 +0000, [email protected] wrote:
From: John Harrison <[email protected]>
Initial creation of scheduler source files. Note that this patch implements most of the scheduler functionality but does not hook it in
to the driver yet. It also leaves the scheduler code in 'pass through'
mode so that even when it is hooked in, it will not actually do very much. This allows the hooks to be added one at a time in bite size chunks and only when the scheduler is finally enabled at the end does anything start happening.The general theory of operation is that when batch buffers are submitted to the driver, the
execbuffer()code packages up all the information required to execute the batch buffer at a later time. This
package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in
use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed
straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.The scheduler is notified when each batch buffer completes and updates
its internal tracking accordingly. At the end of the completion interrupt processing, if any scheduler tracked batches were processed,
the scheduler's deferred worker thread is woken up. This can do more involved processing such as actually removing completed nodes from the
queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.
If a suitable node is found then it is sent to
execbuff_final()for submission to the hardware. The in flight count is then re-checked and
a new node popped from the list if appropriate.Note that this patch does not implement pre-emptive scheduling. Only basic scheduling by re-ordering batch buffer submission is currently implemented. Pre-emption of actively executing batch buffers comes in the next patch series.
v2: Changed priority levels to +/-1023 due to feedback from Chris Wilson.
Removed redundant index from scheduler node.
Changed time stamps to use jiffies instead of raw monotonic. This provides lower resolution but improved compatibility with other i915 code.
Major re-write of completion tracking code due to struct fence conversion. The scheduler no longer has it's own private IRQ handler but just lets the existing request code handle completion events. Instead, the scheduler now hooks into the request notify code to be told when a request has completed.
Reduced driver mutex locking scope. Removal of scheduler nodes no longer grabs the mutex lock.
v3: Refactor of dependency generation to make the code more readable. Also added in read-read optimisation support - i.e., don't treat a shared read-only buffer as being a dependency.
Allowed the killing of queued nodes rather than only flying ones.
v4: Updated the commit message to better reflect the current state of the code. Downgraded some
BUG_ONstoWARN_ONs. Used the correct array memory allocator function (kmalloc_arrayinstead of kmalloc). Corrected the format of some comments. Wrapped some lines differently to keep the style checker happy.Fixed a
WARN_ONwhen killing nodes. The dependency removal code checks
that nodes being destroyed do not have any oustanding dependencies (which would imply they should not have been executed yet). In the case of nodes being destroyed, e.g. due to context banning, then this might well be the case - they have not been executed and do indeed have outstanding dependencies.Re-instated the code to disble interrupts when not in use. The underlying problem causing broken IRQ reference counts seems to have been fixed now.
For: VIZ-1587
Signed-off-by John Harrison <[email protected]>--- drivers/gpu/drm/i915/Makefile | 1 + drivers/gpu/drm/i915/i915_drv.h | 4 + drivers/gpu/drm/i915/i915_gem.c | 5 + drivers/gpu/drm/i915/i915_scheduler.c | 797 ++++++++++++++++++++++++++++++++++ drivers/gpu/drm/i915/i915_scheduler.h | 91 ++++ 5 files changed, 898 insertions(+)create mode 100644
drivers/gpu/drm/i915/i915_scheduler.ccreate mode 100644drivers/gpu/drm/i915/i915_scheduler.hdrivers/gpu/drm/i915/Makefile +1
-0@@ -10,6 +10,7 @@ ccflags-y := -Werror i915-y := i915_drv.o \ i915_irq.o \ i915_params.o \ + i915_scheduler.o \ i915_suspend.o \ i915_sysfs.o \ intel_csr.o \drivers/gpu/drm/i915/i915_drv.h +0
-0drivers/gpu/drm/i915/i915_drv.h struct i915_execbuffer_params { · line 1699
Quoted 5 lines
drivers/gpu/drm/i915/i915_drv.h +2
-0@@ -1695,6 +1695,8 @@ struct i915_execbuffer_params {struct drm_i915_gem_request *request; }; +struct i915_scheduler ; +Rather add "
i915_scheduler.h" include at the top and eliminate circular include dependencies. This is needed for the next comment.drivers/gpu/drm/i915/i915_drv.h struct drm_i915_private { · line 1953
Quoted 9 lines
drivers/gpu/drm/i915/i915_drv.h +2
-0/* used in computing the new watermarks state */ struct intel_wm_config {unsigned int num_pipes_active;@@ -1947,6 +1949,8 @@ struct drm_i915_private {struct i915_runtime_pm pm; +struct i915_scheduler *scheduler; +As the scheduler is going to be enabled on all platforms to an extent, no point in making it a pointer. Just making it member like "pm" is the best, this also requires the above include change.
drivers/gpu/drm/i915/i915_gem.c line 41
Quoted 13 lines
drivers/gpu/drm/i915/i915_drv.h +0
-0/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */ struct { int (*execbuf_submit)(struct i915_execbuffer_params *params,drivers/gpu/drm/i915/i915_gem.c +1
-0@@ -37,6 +37,7 @@#include #include #include +#include "i915_scheduler.h"This should go before any <linux/...> not to mask missing includes from the header itself, so correct place would be right after #include "
i915_trace.h"Quoted 36 lines
drivers/gpu/drm/i915/i915_gem.c +4
-0#define RQ_BUG_ON (expr)@@ -5242,6 +5243,10 @@ int i915_gem_init(struct drm_device *dev)*/ intel_uncore_forcewake_get (dev_priv,FORCEWAKE_ALL ); + ret =i915_scheduler_init (dev); +if (ret) + goto out_unlock; + ret =i915_gem_init_userptr (dev);if (ret) goto out_unlock;drivers/gpu/drm/i915/i915_scheduler.c new file +17
-0@@ -0,0 +1,797 @@ +/* +* Copyright (c) 2014 Intel Corporation +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice (including the next +* paragraph) shall be included in all copies or substantial portions of the +* Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL drivers/gpu/drm/i915/i915_scheduler.c line 27
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +* IN THE SOFTWARE. +* +*/ + +#include "i915_drv.h" +#include "intel_drv.h" +#include "i915_scheduler.h" Again, this include should be the top one, should not require any includes before it.
drivers/gpu/drm/i915/i915_scheduler.c line 39
Quoted 12 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +12
-0+ +static int i915_scheduler_fly_node (struct i915_scheduler_queue_entry *node); +static int i915_scheduler_remove_dependent (struct i915_scheduler *scheduler, +struct i915_scheduler_queue_entry *remove); +static int i915_scheduler_submit (struct intel_engine_cs *ring, +bool is_locked); +static uint32_t i915_scheduler_count_flying (struct i915_scheduler *scheduler, +struct intel_engine_cs *ring); +static void i915_scheduler_priority_bump_clear (struct i915_scheduler *scheduler); +static int i915_scheduler_priority_bump (struct i915_scheduler *scheduler, +struct i915_scheduler_queue_entry *target, +uint32_t bump);Do not indent the function names like this, it becomes unmaintainable and messy very fast if somebody adds a new function with a more complex return type, which is a very likely thing to happen.
What I would do is move all the helper functions here at the top and order them so that the forward declarations are not needed, at least
i915_scheduler_fly_nodeis not used before its definition.drivers/gpu/drm/i915/i915_scheduler.c line 45
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +6
-0+ +int i915_scheduler_init (struct drm_device *dev) +{ +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +int r;Again, this indent is a no-go. I'll not mention it on further functions, assume it to be fixed for next revision.
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +8
-0+ +if (scheduler) +return 0 ; + + scheduler =kzalloc (sizeof (*scheduler),GFP_KERNEL ); +if (!scheduler) +return -ENOMEM ; +drivers/gpu/drm/i915/i915_scheduler.c line 63
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+spin_lock_init (&scheduler->lock); + +for (r =0 ; r <I915_NUM_RINGS ; r++) +INIT_LIST_HEAD (&scheduler->node_queue[r]); + +/* Default tuning values: */ + scheduler->priority_level_min =-1023 ; + scheduler->priority_level_max =1023 ; + scheduler->priority_level_preempt =900 ; + scheduler->min_flying =2 ;This kind of indent is tolerable because it is a contained code block, but not needed either.
Quoted 22 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +22
-0+ + dev_priv->scheduler = scheduler; + +return 0 ; +} + +static void i915_generate_dependencies (struct i915_scheduler *scheduler, +struct i915_scheduler_queue_entry *node, +uint32_t ring) +{ +struct i915_scheduler_obj_entry *this, *that; +struct i915_scheduler_queue_entry *test; +int i, j; +bool found; + +list_for_each_entry (test, &scheduler->node_queue[ring], link) { +if (I915_SQS_IS_COMPLETE (test)) +continue ; + +/* +* Batches on the same ring for the same +* context must be kept in order.
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c line 92
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+*/ + found = (node->params.ctx == test->params.ctx) && + (node->params.ring == test->params.ring); + +if (found) + goto depends; +This is not needed.
Yeah, this code block has been re-worked a number of times with things being added and/or removed. I guess it had gotten a bit messy by now.
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c line 97
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +5
-0+/* +* Batches working on the same objects must +* be kept in order. +*/ +for (i =0 ; (i < node->num_objs) && !found; i++) {As the test is here already ---^
Quoted 9 lines
drivers/gpu/drm/i915/i915_scheduler.c line 104
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+ this = node->saved_objects + i; + +for (j =0 ; j < test->num_objs; j++) { + that = test->saved_objects + j; + +if (this->obj != that->obj) +continue ;How about VMAs? There might be multiple mappings to an object, isn't it enough to depend on the required VMA instead of the whole object?
The object is what we get coming in from user land through the IOCTL. So why make things more complicated? If there are multiple VMAs referring to the same object then we can't just track an individual VMA as that would loose the dependency on all the other VMAs. Just because the object is mapped to someone else's address space doesn't mean that this batch buffer can't overwrite data they are reading.
Quoted 42 lines
drivers/gpu/drm/i915/i915_scheduler.c line 114
Quoted 11 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ +/* Only need to worry about writes */ +if (this->read_only && that->read_only) +continue ; + + found = true; +break ; + } + } +The following block is not needed.
drivers/gpu/drm/i915/i915_scheduler.c line 118
Quoted 4 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +4
-0+if (!found) +continue ; + +depends:Rather like this, in order to avoid a goto label;
if (found) {
...drivers/gpu/drm/i915/i915_scheduler.c line 123
Quoted 5 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +5
-0+ node->dep_list[node->num_deps] = test; + node->num_deps++; + } +} +Please add a brief kerneldoc above each function in the header, it's required. Adding it to non-trivial inline helper functions too will make reviewing much easier.
Quoted 11 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +11
-0+int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe) +{ +struct drm_i915_private *dev_priv = qe->params.dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct intel_engine_cs *ring = qe->params.ring; +struct i915_scheduler_queue_entry *node; +struct i915_scheduler_queue_entry *test; +unsigned long flags; +bool not_flying; +int i, r; +int incomplete =0 ;
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c line 137
Quoted 3 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +3
-0+ +WARN_ON (!scheduler); +This kind of situations should have a be a
BUG_ON, because scheduler being zero is literally going to cause an OOPS in the next dereference which is going to happen unconditionally. WARN + OOPS is kind of whatBUG_ONshould be used avoid. But this should be removed anyway after scheduler is made a data member ofdev_priv.
The WARNs were originally BUGs but Daniel Vetter had the opposite demand. His view was the driver should never BUG under any circumstances. A WARN followed by an oops is better than a BUG because maybe it won't actually oops.
Quoted 29 lines
drivers/gpu/drm/i915/i915_scheduler.c line 138
Quoted 2 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +1
-0+if (1 /*i915.scheduler_override & i915_so_direct_submit*/ ) {I assume this is going to be addressed in a future commit. Could have been introduced in this patch, too.
drivers/gpu/drm/i915/i915_scheduler.c line 144
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +6
-0+int ret; + + scheduler->flags[qe->params.ring->id] |= i915_sf_submitting; + ret = dev_priv->gt.execbuf_final (&qe->params); + scheduler->flags[qe->params.ring->id] &= ~i915_sf_submitting; +The kerneldoc should mention locking requirements of this function.
drivers/gpu/drm/i915/i915_scheduler.c line 157
Quoted 13 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +13
-0+/* +* Don't do any clean up on failure because the caller will +* do it all anyway. +*/ +if (ret) +return ret; + +/* Free everything that is owned by the QE structure: */ +kfree (qe->params.cliprects); +if (qe->params.dispatch_flags &I915_DISPATCH_SECURE ) +i915_gem_execbuff_release_batch_obj (qe->params.batch_obj); + +return 0 ;Above piece of code looks like its own function, so it should probably be one.
Quoted 9 lines
drivers/gpu/drm/i915/i915_scheduler.c line 164
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+ } + + node =kmalloc (sizeof (*node),GFP_KERNEL ); +if (!node) +return -ENOMEM ; + + *node = *qe;Any reason we can't simply move ownership of qe? If not, I'd rather make a clone function
The qe pointer passed in is a reference to a stack local object in the execbuff code path. Thus ownership cannot be transferred. Doing it this way keeps the execbuff code nice and simple and all the dynamic memory management and list tracking is self contained within the scheduler.
Quoted 21 lines
Quoted 21 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +20
-0+INIT_LIST_HEAD (&node->link); + node->status = i915_sqs_queued; + node->stamp = jiffies; +i915_gem_request_reference (node->params.request); + +/* Need to determine the number of incomplete entries in the list as +* that will be the maximum size of the dependency list. +* +* Note that the allocation must not be made with the spinlock acquired +* as kmalloc can sleep. However, the unlock/relock is safe because no +* new entries can be queued up during the unlock as the i915 driver +* mutex is still held. Entries could be removed from the list but that +* just means the dep_list will be over-allocated which is fine. +*/ +spin_lock_irqsave (&scheduler->lock, flags); +for (r =0 ; r <I915_NUM_RINGS ; r++) { +list_for_each_entry (test, &scheduler->node_queue[r], link) { +if (I915_SQS_IS_COMPLETE (test)) +continue ; +
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c line 190
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +6
-0+ incomplete++; + } + } + +/* Temporarily unlock to allocate memory: */ +spin_unlock_irqrestore (&scheduler->lock, flags);I'd make the above piece of code a helper, these stats are to be counted for debugfs anyway, too?
The debugfs code counts all states, not just complete, and requires a scheduler stats object. Also, the above explanation about locking is not applicable in the debugfs case, only here. And it makes much more sense to me to keep the comment and the code all together in one very obvious and understandable block than splitting it up.
Quoted 57 lines
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +9
-0+if (incomplete) { + node->dep_list =kmalloc_array (incomplete, +sizeof (*node->dep_list), +GFP_KERNEL ); +if (!node->dep_list) { +kfree (node); +return -ENOMEM ; + } + }else drivers/gpu/drm/i915/i915_scheduler.c line 209
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ node->dep_list =NULL ; + +spin_lock_irqsave (&scheduler->lock, flags); + node->num_deps =0 ; + +if (node->dep_list) { +for (r =0 ; r <I915_NUM_RINGS ; r++) +i915_generate_dependencies (scheduler, node, r); + +WARN_ON (node->num_deps > incomplete);Wouldn't this condition again lead to a crash? If so, should be
BUG_ONto cause that crash as early as possible.WARN_ONis only good if there is a way of coping with the situation and no imminent system crash is bound to happen.drivers/gpu/drm/i915/i915_scheduler.c line 216
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+ } + +if (node->priority > scheduler->priority_level_max) + node->priority = scheduler->priority_level_max; +else if (node->priority < scheduler->priority_level_min) + node->priority = scheduler->priority_level_min; +There is
clamp_valmacro inlinux/kernel.h.Quoted 23 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +23
-0+if ((node->priority >0 ) && node->num_deps) { +i915_scheduler_priority_bump_clear (scheduler); + +for (i =0 ; i < node->num_deps; i++) +i915_scheduler_priority_bump (scheduler, + node->dep_list[i], node->priority); + } + +list_add_tail (&node->link, &scheduler->node_queue[ring->id]); + + not_flying =i915_scheduler_count_flying (scheduler, ring) < + scheduler->min_flying; + +spin_unlock_irqrestore (&scheduler->lock, flags); + +if (not_flying) +i915_scheduler_submit (ring, true); + +return 0 ; +} + +static int i915_scheduler_fly_node (struct i915_scheduler_queue_entry *node) +{
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c line 246
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+struct drm_i915_private *dev_priv = node->params.dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct intel_engine_cs *ring; + +WARN_ON (!scheduler); +WARN_ON (!node); +WARN_ON (node->status != i915_sqs_popped);Other states had their I915_SQS_IS_* macro, why some don't?
The purpose of the macro is to allow the combining of individual states into classes. E.g. dead and complete can both be considered complete for the majority of cases. Only in certain situations do you need to know that it really was dead. Hence most places that don't really care just use the merged macros, whereas places like this that do care use the explicit enum value.
Quoted 164 lines
Quoted 52 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +51
-0+ + ring = node->params.ring; + +/* +* Add the node (which should currently be in state popped) to the +* front of the queue. This ensure that flying nodes are always held +* in hardware submission order. +*/ +list_add (&node->link, &scheduler->node_queue[ring->id]); + + node->status = i915_sqs_flying; + +if (!(scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) { +bool success = true; + + success = ring->irq_get (ring); +if (success) + scheduler->flags[ring->id] |= i915_sf_interrupts_enabled; +else +return -EINVAL ; + } + +return 0 ; +} + +/* +* Nodes are considered valid dependencies if they are queued on any ring or +* if they are in flight on a different ring. In flight on the same ring is no +* longer interesting for non-premptive nodes as the ring serialises execution. +* For pre-empting nodes, all in flight dependencies are valid as they must not +* be jumped by the act of pre-empting. +* +* Anything that is neither queued nor flying is uninteresting. +*/ +static inline bool i915_scheduler_is_dependency_valid ( +struct i915_scheduler_queue_entry *node,uint32_t idx) +{ +struct i915_scheduler_queue_entry *dep; + + dep = node->dep_list[idx]; +if (!dep) +return false; + +if (I915_SQS_IS_QUEUED (dep)) +return true; + +if (I915_SQS_IS_FLYING (dep)) { +if (node->params.ring != dep->params.ring) +return true; + } +drivers/gpu/drm/i915/i915_scheduler.c line 307
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+return false; +} + +static uint32_t i915_scheduler_count_flying (struct i915_scheduler *scheduler, +struct intel_engine_cs *ring) +{ +struct i915_scheduler_queue_entry *node; +uint32_t flying =0 ; + +list_for_each_entry (node, &scheduler->node_queue[ring->id], link)This loop keeps popping up, it could use a define similar to the ones in
i915_drv.h;#define
for_each_hpd_pin(__pin)\Quoted 4 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +4
-0+if (I915_SQS_IS_FLYING (node)) + flying++; + +return flying;drivers/gpu/drm/i915/i915_scheduler.c line 321
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+} + +/* +* Add a popped node back in to the queue. For example, because the ring was +* hung when execfinal() was called and thus the ring submission needs to be +* retried later. +*/ +static void i915_scheduler_node_requeue (struct i915_scheduler_queue_entry *node) +{ +WARN_ON (!node);if (WARN_ON(!node)) return;Or rather no check at all, it's going to crash anyway even in the calling function if there's NULL, and it's internal function. It's relevant to check if the userspace.
Quoted 26 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +26
-0+WARN_ON (!I915_SQS_IS_FLYING (node)); + +/* Seqno will be reassigned on relaunch */ + node->params.request->seqno =0 ; + node->status = i915_sqs_queued; +} + +/* +* Give up on a node completely. For example, because it is causing the +* ring to hang or is using some resource that no longer exists. +*/ +static void i915_scheduler_node_kill (struct i915_scheduler_queue_entry *node) +{ +WARN_ON (!node); +WARN_ON (I915_SQS_IS_COMPLETE (node)); + + node->status = i915_sqs_dead; +} + +/* +* A sequence number has popped out of the hardware and the request handling +* code has mapped it back to a request and will mark that request complete. +* It also calls this function to notify the scheduler about the completion +* so the scheduler's node can be updated appropriately. +* Returns true if the request is scheduler managed, false if not. +*/ drivers/gpu/drm/i915/i915_scheduler.c line 357
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+bool i915_scheduler_notify_request (struct drm_i915_gem_request *req) +{ +struct drm_i915_private *dev_priv =to_i915 (req->ring->dev); +struct i915_scheduler *scheduler = dev_priv->scheduler; +/* XXX: Need to map back from request to node */ +struct i915_scheduler_queue_entry *node =NULL ; +unsigned long flags; + +if (!node) +return false;Not so sure if slicing the series down to an extent that functions are impossible to review, was a good idea. Idea is to slice things down, to reviewable pieces. It's hard to predict or keep looking forward the series what is going to come.
Quoted 35 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +35
-0+ +spin_lock_irqsave (&scheduler->lock, flags); + +WARN_ON (!I915_SQS_IS_FLYING (node)); + +/* Node was in flight so mark it as complete. */ +if (req->cancelled) + node->status = i915_sqs_dead; +else + node->status = i915_sqs_complete; + +spin_unlock_irqrestore (&scheduler->lock, flags); + +/* +* XXX: If the in-flight list is now empty then new work should be +* submitted. However, this function is called from interrupt context +* and thus cannot acquire mutex locks and other such things that are +* necessary for fresh submission. +*/ + +return true; +} + +int i915_scheduler_remove (struct intel_engine_cs *ring) +{ +struct drm_i915_private *dev_priv = ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct i915_scheduler_queue_entry *node, *node_next; +unsigned long flags; +int flying =0 , queued =0 ; +int ret =0 ; +bool do_submit; +uint32_t min_seqno; +struct list_head remove; +
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c line 399
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+if (list_empty (&scheduler->node_queue[ring->id])) +return 0 ; + +spin_lock_irqsave (&scheduler->lock, flags); + +/* /i915_scheduler_dump_locked(ring, "remove/pre");/ */ +This should not be here at all.
Oops. Old debug code. Hadn't spotted it was still there.
Quoted 23 lines
Quoted 23 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +22
-0+/* +* In the case where the system is idle, starting 'min_seqno' from a big +* number will cause all nodes to be removed as they are now back to +* being in-order. However, this will be a problem if the last one to +* complete was actually out-of-order as the ring seqno value will be +* lower than one or more completed buffers. Thus code looking for the +* completion of said buffers will wait forever. +* Instead, use the hardware seqno as the starting point. This means +* that some buffers might be kept around even in a completely idle +* system but it should guarantee that no-one ever gets confused when +* waiting for buffer completion. +*/ + min_seqno = ring->get_seqno (ring, true); + +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (I915_SQS_IS_QUEUED (node)) + queued++; +else if (I915_SQS_IS_FLYING (node)) + flying++; +else if (I915_SQS_IS_COMPLETE (node)) +continue ; +
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c line 427
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +6
-0+if (node->params.request->seqno ==0 ) +continue ; + +if (!i915_seqno_passed (node->params.request->seqno, min_seqno)) + min_seqno = node->params.request->seqno; + }Couldn't these values be kept cached, instead of counting them at each function?
The 'queued' and flying totals could be kept cached but min_seqno is dependent upon the state of the hardware so needs to be recalculated. In which case calculating the totals here is trivial and avoids having extra code elsewhere to keep them up to date.
Quoted 251 lines
Quoted 46 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +45
-0+ +INIT_LIST_HEAD (&remove); +list_for_each_entry_safe (node, node_next, &scheduler->node_queue[ring->id], link) { +/* +* Only remove completed nodes which have a lower seqno than +* all pending nodes. While there is the possibility of the +* ring's seqno counting backwards, all higher buffers must +* be remembered so that the 'i915_seqno_passed()' test can +* report that they have in fact passed. +* +* NB: This is not true for 'dead' nodes. The GPU reset causes +* the software seqno to restart from its initial value. Thus +* the dead nodes must be removed even though their seqno values +* are potentially vastly greater than the current ring seqno. +*/ +if (!I915_SQS_IS_COMPLETE (node)) +continue ; + +if (node->status != i915_sqs_dead) { +if (i915_seqno_passed (node->params.request->seqno, min_seqno) && + (node->params.request->seqno != min_seqno)) +continue ; + } + +list_del (&node->link); +list_add (&node->link, &remove); + +/* Strip the dependency info while the mutex is still locked */ +i915_scheduler_remove_dependent (scheduler, node); + +continue ; + } + +/* +* Release the interrupt reference count if there are no longer any +* nodes to worry about. +*/ +if (!flying && !queued && + (scheduler->flags[ring->id] & i915_sf_interrupts_enabled)) { + ring->irq_put (ring); + scheduler->flags[ring->id] &= ~i915_sf_interrupts_enabled; + } + +/* Launch more packets now? */ + do_submit = (queued >0 ) && (flying < scheduler->min_flying);drivers/gpu/drm/i915/i915_scheduler.c line 482
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ +spin_unlock_irqrestore (&scheduler->lock, flags); + +if (!do_submit &&list_empty (&remove)) +return ret; + +mutex_lock (&ring->dev->struct_mutex); + +if (do_submit) + ret =i915_scheduler_submit (ring, true);Confusing to have this at a remove function. Function naming needs to be reconsidered or moved out from here.
Quoted 86 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +86
-0+ +while (!list_empty (&remove)) { + node =list_first_entry (&remove,typeof (*node), link); +list_del (&node->link); + +/* +* The batch buffer must be unpinned before it is unreferenced +* otherwise the unpin fails with a missing vma!? + */priority_bump_clear + if (node->params.dispatch_flags & I915_DISPATCH_SECURE) + i915_gem_execbuff_release_batch_obj(node->params.batch_obj); + +/* Free everything that is owned by the node: */ +i915_gem_request_unreference (node->params.request); +kfree (node->params.cliprects); +kfree (node->dep_list); +kfree (node); + } + +mutex_unlock (&ring->dev->struct_mutex); + +return ret; +} + +static void i915_scheduler_priority_bump_clear (struct i915_scheduler *scheduler) +{ +struct i915_scheduler_queue_entry *node; +int i; + +/* +* Ensure circular dependencies don't cause problems and that a bump +* by object usage only bumps each using buffer once: +*/ +for (i =0 ; i <I915_NUM_RINGS ; i++) { +list_for_each_entry (node, &scheduler->node_queue[i], link) + node->bumped = false; + } +} + +static int i915_scheduler_priority_bump (struct i915_scheduler *scheduler, +struct i915_scheduler_queue_entry *target, +uint32_t bump) +{ +uint32_t new_priority; +int i, count; + +if (target->priority >= scheduler->priority_level_max) +return 1 ; + +if (target->bumped) +return 0 ; + + new_priority = target->priority + bump; +if ((new_priority <= target->priority) || + (new_priority > scheduler->priority_level_max)) + target->priority = scheduler->priority_level_max; +else + target->priority = new_priority; + + count =1 ; + target->bumped = true; + +for (i =0 ; i < target->num_deps; i++) { +if (!target->dep_list[i]) +continue ; + +if (target->dep_list[i]->bumped) +continue ; + + count +=i915_scheduler_priority_bump (scheduler, + target->dep_list[i], + bump); + } + +return count; +} + +static int i915_scheduler_pop_from_queue_locked (struct intel_engine_cs *ring, +struct i915_scheduler_queue_entry **pop_node, +unsigned long *flags) +{ +struct drm_i915_private *dev_priv = ring->dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler; +struct i915_scheduler_queue_entry *best; +struct i915_scheduler_queue_entry *node; +int ret;drivers/gpu/drm/i915/i915_scheduler.c line 578
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+int i; +bool any_queued; +bool has_local, has_remote, only_remote; + + *pop_node =NULL ; + ret = -ENODATA ; + + any_queued = false; + only_remote = false; + best =NULL ;These should just be initialized in-place. But looking at the code forward.
Quoted 62 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +62
-0+ +list_for_each_entry (node, &scheduler->node_queue[ring->id], link) { +if (!I915_SQS_IS_QUEUED (node)) +continue ; + any_queued = true; + + has_local = false; + has_remote = false; +for (i =0 ; i < node->num_deps; i++) { +if (!i915_scheduler_is_dependency_valid (node, i)) +continue ; + +if (node->dep_list[i]->params.ring == node->params.ring) + has_local = true; +else + has_remote = true; + } + +if (has_remote && !has_local) + only_remote = true; + +if (!has_local && !has_remote) { +if (!best || + (node->priority > best->priority)) + best = node; + } + } + +if (best) { +list_del (&best->link); + +INIT_LIST_HEAD (&best->link); + best->status = i915_sqs_popped; + + ret =0 ; + }else { +/* Can only get here if: +* (a) there are no buffers in the queue +* (b) all queued buffers are dependent on other buffers +* e.g. on a buffer that is in flight on a different ring +*/ +if (only_remote) { +/* The only dependent buffers are on another ring. */ + ret = -EAGAIN ; + }else if (any_queued) { +/* It seems that something has gone horribly wrong! */ +DRM_ERROR ("Broken dependency tracking on ring %d!\n" , + (int ) ring->id); + } + } + +/* i915_scheduler_dump_queue_pop(ring, best); */ + + *pop_node = best; +return ret; +} + +static int i915_scheduler_submit (struct intel_engine_cs *ring,bool was_locked) +{ +struct drm_device *dev = ring->dev; +struct drm_i915_private *dev_priv = dev->dev_private; +struct i915_scheduler *scheduler = dev_priv->scheduler;drivers/gpu/drm/i915/i915_scheduler.c line 650
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+struct i915_scheduler_queue_entry *node; +unsigned long flags; +int ret =0 , count =0 ; + +if (!was_locked) { + ret =i915_mutex_lock_interruptible (dev); +if (ret) +return ret; + } +I don't really fancy this construct. Should be moved outside of this function for proper lockdep tracking.
Quoted 5 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +5
-0+WARN_ON (!mutex_is_locked (&dev->struct_mutex)); + +spin_lock_irqsave (&scheduler->lock, flags); + +/* First time around, complain if anything unexpected occurs: */ drivers/gpu/drm/i915/i915_scheduler.c line 665
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+ ret =i915_scheduler_pop_from_queue_locked (ring, &node, &flags); +if (ret) { +spin_unlock_irqrestore (&scheduler->lock, flags); + +if (!was_locked) +mutex_unlock (&dev->struct_mutex); + +return ret; + } +Dropping the
was_lockedstuff, this should become a proper goto error label. e.g.out_unlock
Quoted 16 lines
drivers/gpu/drm/i915/i915_scheduler.c line 676
Quoted 11 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +11
-0+ do { + WARN_ON(!node); + WARN_ON(node->params.ring != ring); + WARN_ON(node->status != i915_sqs_popped); + count++; + + /* + * The call to pop above will have removed the node from the + * list. So add it back in and mark it as in flight. + */ + i915_scheduler_fly_node(node);Why do we want to pull an object out of the list inside spin lock and push it back immediately in our critical code path? Seems like a waste for no obvious gain at this point. Why do not we rather just select an entry and modify it in-place, if it's going to stay in the same queue anyway.
The list order is significant. The element must be moved to the front to keep the submitted items in submission order. Doing it this way also keeps the code nicely partitioned and easier to understand/maintain. Plus, there is a plan to optimise the code by splitting the one single list into three separate ones - queued, flying, complete. If/when that happens, the element will have to be removed from one list and added to another.
Quoted 47 lines
drivers/gpu/drm/i915/i915_scheduler.c line 685
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +9
-0+ + scheduler->flags[ring->id] |= i915_sf_submitting; +spin_unlock_irqrestore (&scheduler->lock, flags); + ret = dev_priv->gt.execbuf_final (&node->params); +spin_lock_irqsave (&scheduler->lock, flags); + scheduler->flags[ring->id] &= ~i915_sf_submitting; + +if (ret) { +int requeue =1 ;Multipurpose variable, not really a good idea. And as commented further, should not exist at all.
drivers/gpu/drm/i915/i915_scheduler.c line 698
Quoted 13 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +13
-0+ +/* +* Oh dear! Either the node is broken or the ring is +* busy. So need to kill the node or requeue it and try +* again later as appropriate. +*/ + +switch (-ret) { +case ENODEV : +case ENOENT : +/* Fatal errors. Kill the node. */ + requeue =-1 ; +break ;"break" indent is wrong.
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7
-0+ +case EAGAIN : +case EBUSY : +case EIO : +case ENOMEM : +case ERESTARTSYS : +case EINTR :drivers/gpu/drm/i915/i915_scheduler.c line 715
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+/* Supposedly recoverable errors. */ +break ; + +default : +/* +* Assume the error is recoverable and hope +* for the best. +*/ +DRM_DEBUG_DRIVER ("<%s> Got unexpected error from execfinal(): %d!\n" , + ring->name, ret);There's
MISSING_CASEmacro, should use it.
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.c line 114
Quoted 3 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +3
-0+break ; + } +Just move the code below this point to the switch, no point having a switch to categorize your options and then doing bunch of ifs to execute code that could be in switch.
One of the 'if' paths is to break out of the while loop. Can't do that from inside the switch.
Quoted 197 lines
Quoted 9 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +9
-0+/* +* Check that the watchdog/reset code has not nuked +* the node while we weren't looking: +*/ +if (node->status == i915_sqs_dead) + requeue =0 ; + +if (requeue ==1 ) { +i915_scheduler_node_requeue (node);drivers/gpu/drm/i915/i915_scheduler.c line 737
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+/* +* No point spinning if the ring is currently +* unavailable so just give up and come back +* later. +*/ +break ; + }else if (requeue ==-1 ) +i915_scheduler_node_kill (node); + } +Ending here, this actual submission of a single node could go to its own helper function, these functions now become too long to follow, although they really are not doing anything complicated.
Quoted 8 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +8
-0+/* Keep launching until the sky is sufficiently full. */ +if (i915_scheduler_count_flying (scheduler, ring) >= + scheduler->min_flying) +break ; + + ret =i915_scheduler_pop_from_queue_locked (ring, &node, &flags); + }while (ret ==0 ); +drivers/gpu/drm/i915/i915_scheduler.c line 755
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10
-0+spin_unlock_irqrestore (&scheduler->lock, flags); + +if (!was_locked) +mutex_unlock (&dev->struct_mutex); + +/* Don't complain about not being able to submit extra entries */ +if (ret == -ENODATA ) + ret =0 ; + +return (ret <0 ) ? ret : count;This is a combined error and success path, keeping to the convention of kernel drivers is preferred;
if (ret != -ENODATA) goto out_foo; return count;
out_unlock:spin_unlock(...); out_foo: return ret;Quoted 81 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +42
-0+} + +static int i915_scheduler_remove_dependent (struct i915_scheduler *scheduler, +struct i915_scheduler_queue_entry *remove) +{ +struct i915_scheduler_queue_entry *node; +int i, r; +int count =0 ; + +/* +* Ensure that a node is not being removed which is still dependent +* upon other (not completed) work. If that happens, it implies +* something has gone very wrong with the dependency tracking! Note +* that there is no need to worry if this node has been explicitly +* killed for some reason - it might be being killed before it got +* sent to the hardware. +*/ +if (remove->status != i915_sqs_dead) { +for (i =0 ; i < remove->num_deps; i++) +if ((remove->dep_list[i]) && + (!I915_SQS_IS_COMPLETE (remove->dep_list[i]))) + count++; +WARN_ON (count); + } + +/* +* Remove this node from the dependency lists of any other node which +* might be waiting on it. +*/ +for (r =0 ; r <I915_NUM_RINGS ; r++) { +list_for_each_entry (node, &scheduler->node_queue[r], link) { +for (i =0 ; i < node->num_deps; i++) { +if (node->dep_list[i] != remove) +continue ; + + node->dep_list[i] =NULL ; + } + } + } + +return 0 ; +}drivers/gpu/drm/i915/i915_scheduler.h new file +33
-0@@ -0,0 +1,91 @@ +/* +* Copyright (c) 2014 Intel Corporation +* +* Permission is hereby granted, free of charge, to any person obtaining a +* copy of this software and associated documentation files (the "Software"), +* to deal in the Software without restriction, including without limitation +* the rights to use, copy, modify, merge, publish, distribute, sublicense, +* and/or sell copies of the Software, and to permit persons to whom the +* Software is furnished to do so, subject to the following conditions: +* +* The above copyright notice and this permission notice (including the next +* paragraph) shall be included in all copies or substantial portions of the +* Software. +* +* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL +* THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +* FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS +* IN THE SOFTWARE. +* +*/ + +#ifndef _I915_SCHEDULER_H_ +#define _I915_SCHEDULER_H_ + +enum i915_scheduler_queue_status { +/* Limbo: */ + i915_sqs_none =0 , +/* Not yet submitted to hardware: */ + i915_sqs_queued, + /* Popped from queue, ready to fly: */drivers/gpu/drm/i915/i915_scheduler.h line 43
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +10
-0+ i915_sqs_popped, +/* Sent to hardware for processing: */ + i915_sqs_flying, +/* Finished processing on the hardware: */ + i915_sqs_complete, +/* Killed by watchdog or catastrophic submission failure: */ + i915_sqs_dead, +/* Limit value for use with arrays/loops */ + i915_sqs_MAX +};These should be
UPPERCASE_FOR_ENUM_VALUES. Seei915_drv.hfor samples.drivers/gpu/drm/i915/i915_scheduler.h line 49
Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +6
-0+ +#define I915_SQS_IS_QUEUED (node) (((node)->status == i915_sqs_queued)) +#define I915_SQS_IS_FLYING (node) (((node)->status == i915_sqs_flying)) +#define I915_SQS_IS_COMPLETE (node) (((node)->status == i915_sqs_complete) || \ + ((node)->status == i915_sqs_dead)) +Might be slightly confusing that name is
IS_COMPLETEand there is actual COMPLETE value. Rather have the test likeIS_DONEand then test for COMPLETE or DEAD, no confusion. Also, some states have their IS_* macro and others don't, is there going to be more?Quoted 25 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +25
-0+struct i915_scheduler_obj_entry { +struct drm_i915_gem_object *obj; +bool read_only; +}; + +struct i915_scheduler_queue_entry { +struct i915_execbuffer_params params; +/* -1023 = lowest priority, 0 = default, 1023 = highest */ +int32_t priority; +struct i915_scheduler_obj_entry *saved_objects; +int num_objs; +bool bumped; +struct i915_scheduler_queue_entry **dep_list; +int num_deps; +enum i915_scheduler_queue_status status; +unsigned long stamp; +struct list_head link; +}; + +struct i915_scheduler { +struct list_head node_queue[I915_NUM_RINGS ]; +uint32_t flags[I915_NUM_RINGS ]; +spinlock_t lock; + +/* Tuning parameters: */ drivers/gpu/drm/i915/i915_scheduler.h line 84
Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +10
-0+int32_t priority_level_min; +int32_t priority_level_max; +int32_t priority_level_preempt; +uint32_t min_flying; +}; + +/* Flag bits for i915_scheduler::flags */ +enum { + i915_sf_interrupts_enabled = (1 <<0 ), + i915_sf_submitting = (1 <<1 ),Again, should be uppercase. Also, enums to the beginning of file.
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.h line 91
Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +7
-0+}; + +int i915_scheduler_init (struct drm_device *dev); +int i915_scheduler_queue_execbuffer (struct i915_scheduler_queue_entry *qe); +bool i915_scheduler_notify_request (struct drm_i915_gem_request *req); + +#endif /* _I915_SCHEDULER_H_ */
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx