Skip to contentSkip to conversation

GPU scheduler for i915 driver

PATCH In review [email protected] 5 participants 50 messages Apr 20, 2016 – Jun 10, 2016

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v6 cover GPU scheduler for i915 driver

From: John Harrison <[email protected]>

Implemented a batch buffer submission scheduler for the i915 DRM driver.

The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code assigns a unique seqno value and then packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.

As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, the seqno values of the individual buffers are not necessary incrementing as the scheduler may have re-ordered their submission. However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.

A deferred work queue is also poked by the interrupt handler. When this wakes up it can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.

When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate.

The scheduler also allows high priority batch buffers (e.g. from a desktop compositor) to jump ahead of whatever is already running if the underlying hardware supports pre-emption. In this situation, any work that was pre-empted is returned to the queued list ready to be resubmitted when no more high priority work is outstanding.

Various IGT tests are in progress to test the scheduler's operation and will follow.

v2: Updated for changes in struct fence patch series and other changes to underlying tree (e.g. removal of cliprects). Also changed priority levels to be signed +/-1023 range and reduced mutex lock usage.

v3: More reuse of cached pointers rather than repeated dereferencing (David Gordon).

Moved the dependency generation code out to a seperate function for easier readability. Also added in support for the read-read optimisation.

Major simplification of the DRM file close handler.

Fixed up an overzealous WARN.

Removed unnecessary flushing of the scheduler queue when waiting for a request.

v4: Removed user land fence/sync integration as this is dependent upon the de-staging of the Android sync code. That de-staging is now being done by someone else. The sync support will be added back in to the scheduler in a separate patch series which must wait for the de-staging to be landed first.

Added support for killing batches from contexts that were banned after the batches were submitted to the scheduler.

Changed various comments to fix typos, update to reflect changes to the code, correct formatting and line wrapping, etc. Also wrapped various long lines and twiddled white space to keep the style checker happy.

Changed a bunch of BUG_ONs to WARN_ONs as apparently the latter are preferred.

Used the correct array memory allocator function (kmalloc_array instead of kmalloc).

Fixed a variable type (review comment by Joonas).

Fixed a WARN_ON firing incorrectly when removing killed nodes from the scheduler's queue.

Dropped register definition update patch from this series. The changes are all for pre-emption so it makes more sense for it to be part of that series instead.

v5: Reverted power management changes as they apparently conflict with mutex acquisition. Converted the override mask module parameter to a set of boolean enable flags (just one in this patch set, but others are added later for controlling pre-emption). [Chris Wilson]

Removed lots of whitespace from i915_scheduler.c and re-ordered it to remove all forward declarations. Squashed down the i915_scheduler.c sections of various patches into the initial 'start of scheduler' patch. Thus the later patches simply hook in existing code into various parts of the driver rather than adding the code as well. Added documentation to various functions. Re-worked the submit function in terms of mutex locking, error handling and exit paths. Split the delayed work handler function in half. Made use of the kernel 'clamp' macro. [Joonas Lahtinen]

Dropped the 'immediate submission' override option. This was a half way house between full scheduler and direct submission and was only really useful during early debug.

Added a re-install of the scheduler's interrupt hook around GPU reset. [Zhipeng Gong]

Used lighter weight spinlocks.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' and 'assert_scheduler_lock_held()' helper macros. Renamed 'i915_gem_execbuff_release_batch_obj' to 'i915_gem_execbuf_release_batch_obj'. Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. Removed various unnecessary WARNs. Renamed 'saved_objects' to just 'objs'. More code refactoring. Removed even more white space. Added an i915_scheduler_destroy() function instead of doing explicit clean up of scheduler internals from i915_driver_unload(). Changed extra boolean i915_wait_request() parameter to a flags word and consumed the original boolean parameter too. Also, replaced the i915_scheduler_is_request_tracked() function with i915_scheduler_is_mutex_required() and i915_scheduler_is_request_batch_buffer() as the need for the former has gone away and it was really being used to ask the latter two questions in a convoluted manner. Wrapped boolean 'flush' parameter to intel_engine_idle() with an _flush() macro.
[review feedback from Joonas Lahtinen]

Moved scheduler modue parameter declaration to correct place in i915_params struct. [review feedback from Matt Roper]

Added an admin only check when setting the tuning parameters via debugfs to prevent rogue user code trying to break the system with strange settings. [review feedback from Jesse Barnes]

Added kerneldoc for intel_engine_idle().

Added running totals of 'flying' and 'queued' nodes rather than re-calculating each time as a minor CPU performance optimisation.

Removed support for out of order seqno completion. All the prep work patch series (seqno to request conversion, late seqno assignment, etc.) that has now been done means that the scheduler no longer generates out of order seqno completions. Thus all the complex code for coping with such is no longer required and can be removed.

Fixed a bug in scheduler bypass mode introduced in the clean up code refactoring of v5. The clean up function was seeing the node in the wrong state and thus refusing to process it.

Improved the throttle by file handle feature by chaning from a simple 'return to userland when full' scheme with a 'sleep on request' scheme. The former could lead to the busy polling and wasting lots of CPU time as user land continuously retried the execbuf IOCTL in a tight loop. Now the driver will sleep (without holding the mutex lock) on the oldest request outstanding for that file and then automatically retry. This is closer to the pre-scheduler behaviour of stalling on a full ring buffer.

[Patches against drm-intel-nightly tree fetched 13/04/2016 with struct fence conversion patches applied]

Dave Gordon (2):
drm/i915: Cache request pointer in *_submission_final() drm/i915: Add scheduling priority to per-context parameters

John Harrison (32):
drm/i915: Add total count to context status debugfs output drm/i915: Prelude to splitting i915_gem_do_execbuffer in two drm/i915: Split i915_dem_do_execbuffer() in half
drm/i915: Re-instate request->uniq because it is extremely useful drm/i915: Start of GPU scheduler
drm/i915: Disable hardware semaphores when GPU scheduler is enabled drm/i915: Force MMIO flips when scheduler enabled
drm/i915: Added scheduler hook when closing DRM file handles drm/i915: Added scheduler hook into i915_gem_request_notify() drm/i915: Added deferred work handler for scheduler
drm/i915: Redirect execbuffer_final() via scheduler
drm/i915: Keep the reserved space mechanism happy
drm/i915: Added tracking/locking of batch buffer objects drm/i915: Hook scheduler node clean up into retire requests drm/i915: Added scheduler support to __wait_request() calls drm/i915: Added scheduler support to page fault handler drm/i915: Added scheduler flush calls to ring throttle and idle functions drm/i915: Add scheduler hook to GPU reset
drm/i915: Added a module parameter to allow the scheduler to be disabled drm/i915: Support for 'unflushed' ring idle
drm/i915: Defer seqno allocation until actual hardware submission time drm/i915: Added trace points to scheduler
drm/i915: Added scheduler queue throttling by DRM file handle drm/i915: Added debugfs interface to scheduler tuning parameters drm/i915: Add early exit to execbuff_final() if insufficient ring space drm/i915: Added scheduler statistic reporting to debugfs drm/i915: Add scheduler support functions for TDR
drm/i915: Enable GPU scheduler by default
drm/i915: Add support for retro-actively banning batch buffers drm/i915: Allow scheduler to manage inter-ring object synchronisation drm/i915: Added debug state dump facilities to scheduler drm/i915: Scheduler state dump via debugfs

 drivers/gpu/drm/i915/Makefile              |    1 +
 drivers/gpu/drm/i915/i915_debugfs.c        |  336 +++++-
 drivers/gpu/drm/i915/i915_dma.c            |    5 +
 drivers/gpu/drm/i915/i915_drv.c            |    9 +
 drivers/gpu/drm/i915/i915_drv.h            |   58 +-
 drivers/gpu/drm/i915/i915_gem.c            |  156 ++-
 drivers/gpu/drm/i915/i915_gem_context.c    |   24 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  297 +++--
 drivers/gpu/drm/i915/i915_params.c         |    4 +
 drivers/gpu/drm/i915/i915_params.h         |    1 +
 drivers/gpu/drm/i915/i915_scheduler.c      | 1709 ++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |  180 +++
 drivers/gpu/drm/i915/i915_trace.h          |  225 +++-
 drivers/gpu/drm/i915/intel_display.c       |   10 +-
 drivers/gpu/drm/i915/intel_lrc.c           |  161 ++-
 drivers/gpu/drm/i915/intel_lrc.h           |    1 +
 drivers/gpu/drm/i915/intel_ringbuffer.c    |   69 +-
 drivers/gpu/drm/i915/intel_ringbuffer.h    |    5 +-
 include/uapi/drm/i915_drm.h                |    1 +
 19 files changed, 3118 insertions(+), 134 deletions(-)
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#1 lore raw

PATCH v6 1/34 drm/i915: Add total count to context status debugfs output

From: John Harrison <[email protected]>

When there are lots and lots and even more lots of contexts (e.g. when running with execlists) it is useful to be able to immediately see what the total context count is.

v4: Re-typed a variable (review feedback from Joonas)

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_debugfs.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_debugfs.c +4 -1
index 29ff1b3..e89781a 100644
@@ -1961,7 +1961,7 @@ static int i915_context_status(struct seq_file *m, void *unused)
 	struct intel_engine_cs *engine;
 	struct intel_context *ctx;
 	enum intel_engine_id id;
-	int ret;
+	int ret, count = 0;
 
 	ret = mutex_lock_interruptible(&dev->struct_mutex);
 	if (ret)
@@ -1976,6 +1976,7 @@ static int i915_context_status(struct seq_file *m, void *unused)
 		describe_ctx(m, ctx);
 		if (ctx == dev_priv->kernel_context)
 			seq_printf(m, "(kernel context) ");
+		count++;
 
 		if (i915.enable_execlists) {
 			seq_putc(m, '\n');
@@ -1999,6 +2000,8 @@ static int i915_context_status(struct seq_file *m, void *unused)
 		seq_putc(m, '\n');
 	}
 
+	seq_printf(m, "Total: %d contexts\n", count);
+
 	mutex_unlock(&dev->struct_mutex);
 
 	return 0;
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#2 lore raw

PATCH v6 2/34 drm/i915: Prelude to splitting i915_gem_do_execbuffer in two

In reply to [email protected]

From: John Harrison <[email protected]>

The scheduler decouples the submission of batch buffers to the driver with their submission to the hardware. This basically means splitting the execbuffer() function in half. This change rearranges some code ready for the split to occur.

v5: Dropped runtime PM calls as they conflict with the mutex lock. Instead of being done at the lowest submission level, they are now left right at the top driver entry level. [feedback from Chris Wilson]

v6: Updated to newer nightly (lots of ring -> engine renaming).

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 41 ++++++++++++++++++------------
 drivers/gpu/drm/i915/intel_lrc.c           | 18 +++++++++----
 2 files changed, 38 insertions(+), 21 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +25 -16
index 40d333c..f25fd2c 100644
@@ -970,10 +970,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,
 	if (flush_domains & I915_GEM_DOMAIN_GTT)
 		wmb();
 
-	/* Unconditionally invalidate gpu caches and ensure that we do flush
-	 * any residual writes from the previous batch.
-	 */
-	return intel_ring_invalidate_all_caches(req);
+	return 0;
 }
 
 static bool
@@ -1238,17 +1235,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 	u32 instp_mask;
 	int ret;
 
-	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
-	if (ret)
-		return ret;
-
-	ret = i915_switch_context(params->request);
-	if (ret)
-		return ret;
-
-	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<engine->id),
-	     "%s didn't clear reload\n", engine->name);
-
 	instp_mode = args->flags & I915_EXEC_CONSTANTS_MASK;
 	instp_mask = I915_EXEC_CONSTANTS_MASK;
 	switch (instp_mode) {
@@ -1282,6 +1268,30 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 		return -EINVAL;
 	}
 
+	ret = i915_gem_execbuffer_move_to_gpu(params->request, vmas);
+	if (ret)
+		return ret;
+
+	i915_gem_execbuffer_move_to_active(vmas, params->request);
+
+	/* To be split into two functions here... */
+
+	/*
+	 * Unconditionally invalidate gpu caches and ensure that we do flush
+	 * any residual writes from the previous batch.
+	 */
+	ret = intel_ring_invalidate_all_caches(params->request);
+	if (ret)
+		return ret;
+
+	/* Switch to the correct context for the batch */
+	ret = i915_switch_context(params->request);
+	if (ret)
+		return ret;
+
+	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<engine->id),
+	     "%s didn't clear reload\n", engine->name);
+
 	if (engine == &dev_priv->engine[RCS] &&
 	    instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_ring_begin(params->request, 4);
@@ -1318,7 +1328,6 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
 
-	i915_gem_execbuffer_move_to_active(vmas, params->request);
 	i915_gem_execbuffer_retire_commands(params);
 
 	return 0;
drivers/gpu/drm/i915/intel_lrc.c +13 -5
index a351a0e..5866342 100644
@@ -690,10 +690,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,
 	if (flush_domains & I915_GEM_DOMAIN_GTT)
 		wmb();
 
-	/* Unconditionally invalidate gpu caches and ensure that we do flush
-	 * any residual writes from the previous batch.
-	 */
-	return logical_ring_invalidate_all_caches(req);
+	return 0;
 }
 
 int intel_logical_ring_alloc_request_extras(struct drm_i915_gem_request *request)
@@ -989,6 +986,18 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 	if (ret)
 		return ret;
 
+	i915_gem_execbuffer_move_to_active(vmas, params->request);
+
+	/* To be split into two functions here... */
+
+	/*
+	 * Unconditionally invalidate gpu caches and ensure that we do flush
+	 * any residual writes from the previous batch.
+	 */
+	ret = logical_ring_invalidate_all_caches(params->request);
+	if (ret)
+		return ret;
+
 	if (engine == &dev_priv->engine[RCS] &&
 	    instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_logical_ring_begin(params->request, 4);
@@ -1013,7 +1022,6 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
 
-	i915_gem_execbuffer_move_to_active(vmas, params->request);
 	i915_gem_execbuffer_retire_commands(params);
 
 	return 0;
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#3 lore raw

PATCH v6 3/34 drm/i915: Split i915_dem_do_execbuffer() in half

In reply to [email protected]

From: John Harrison <[email protected]>

Split the execbuffer() function in half. The first half collects and validates all the information required to process the batch buffer. It also does all the object pinning, relocations, active list management, etc - basically anything that must be done upfront before the IOCTL returns and allows the user land side to start changing/freeing things. The second half does the actual ring submission.

This change implements the split but leaves the back half being called directly from the end of the front half.

v2: Updated due to changes in underlying tree - addition of sync fence support and removal of cliprects.

v3: Moved local 'ringbuf' variable to make later patches in the series a bit neater.

v4: Corrected a typo in the commit message and downgraded a BUG_ON to a WARN_ON as the latter is preferred. Also removed all the external sync/fence support as that will now be a separate patch series.

v5: Updated for runtime PM changes.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Renamed 'i915_gem_execbuff_release_batch_obj' to 'i915_gem_execbuf_release_batch_obj' and updated to use 'to_i915()' instead of dev_private. [review feedback from Joonas Lahtinen]

Added an explicit remove_from_client() call to the failure path to fix a race condition with invalid requests the client list.

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_drv.h            |  11 +++
 drivers/gpu/drm/i915/i915_gem.c            |   2 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 113 +++++++++++++++++++++--------
 drivers/gpu/drm/i915/intel_lrc.c           |  57 ++++++++++-----
 drivers/gpu/drm/i915/intel_lrc.h           |   1 +
 5 files changed, 137 insertions(+), 47 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +11 -0
index 9519b11..f978539 100644
@@ -1701,10 +1701,18 @@ struct i915_execbuffer_params {
 	struct drm_device               *dev;
 	struct drm_file                 *file;
 	uint32_t                        dispatch_flags;
+	uint32_t                        args_flags;
 	uint32_t                        args_batch_start_offset;
+	uint32_t                        args_batch_len;
+	uint32_t                        args_num_cliprects;
+	uint32_t                        args_DR1;
+	uint32_t                        args_DR4;
 	uint64_t                        batch_obj_vm_offset;
 	struct intel_engine_cs *engine;
 	struct drm_i915_gem_object      *batch_obj;
+	struct drm_clip_rect            *cliprects;
+	uint32_t                        instp_mask;
+	int                             instp_mode;
 	struct intel_context            *ctx;
 	struct drm_i915_gem_request     *request;
 };
@@ -1991,6 +1999,7 @@ struct drm_i915_private {
 		int (*execbuf_submit)(struct i915_execbuffer_params *params,
 				      struct drm_i915_gem_execbuffer2 *args,
 				      struct list_head *vmas);
+		int (*execbuf_final)(struct i915_execbuffer_params *params);
 		int (*init_engines)(struct drm_device *dev);
 		void (*cleanup_engine)(struct intel_engine_cs *engine);
 		void (*stop_engine)(struct intel_engine_cs *engine);
@@ -2896,9 +2905,11 @@ int i915_gem_sw_finish_ioctl(struct drm_device *dev, void *data,
 void i915_gem_execbuffer_move_to_active(struct list_head *vmas,
 					struct drm_i915_gem_request *req);
 void i915_gem_execbuffer_retire_commands(struct i915_execbuffer_params *params);
+void i915_gem_execbuf_release_batch_obj(struct drm_i915_gem_object *batch_obj);
 int i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 				   struct drm_i915_gem_execbuffer2 *args,
 				   struct list_head *vmas);
+int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params);
 int i915_gem_execbuffer(struct drm_device *dev, void *data,
 			struct drm_file *file_priv);
 int i915_gem_execbuffer2(struct drm_device *dev, void *data,
drivers/gpu/drm/i915/i915_gem.c +2 -0
index d1f8c47..21fce67 100644
@@ -5383,11 +5383,13 @@ int i915_gem_init(struct drm_device *dev)
 
 	if (!i915.enable_execlists) {
 		dev_priv->gt.execbuf_submit = i915_gem_ringbuffer_submission;
+		dev_priv->gt.execbuf_final = i915_gem_ringbuffer_submission_final;
 		dev_priv->gt.init_engines = i915_gem_init_engines;
 		dev_priv->gt.cleanup_engine = intel_cleanup_engine;
 		dev_priv->gt.stop_engine = intel_stop_engine;
 	} else {
 		dev_priv->gt.execbuf_submit = intel_execlists_submission;
+		dev_priv->gt.execbuf_final = intel_execlists_submission_final;
 		dev_priv->gt.init_engines = intel_logical_rings_init;
 		dev_priv->gt.cleanup_engine = intel_logical_ring_cleanup;
 		dev_priv->gt.stop_engine = intel_logical_ring_stop;
drivers/gpu/drm/i915/i915_gem_execbuffer.c +83 -30

199 lines not on this page · raw message

drivers/gpu/drm/i915/intel_lrc.c +40 -17

103 lines not on this page · raw message

drivers/gpu/drm/i915/intel_lrc.h +1 -0
index 9affda2..3d0a4c9 100644
@@ -121,6 +121,7 @@ struct i915_execbuffer_params;
 int intel_execlists_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas);
+int intel_execlists_submission_final(struct i915_execbuffer_params *params);
 
 void intel_execlists_retire_requests(struct intel_engine_cs *engine);
 
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#4 lore raw

PATCH v6 4/34 drm/i915: Cache request pointer in *_submission_final()

In reply to [email protected]

From: Dave Gordon <[email protected]>

Keep a local copy of the request pointer in the _final() functions rather than dereferencing the params block repeatedly.

v3: New patch in series.

v6: Updated to newer nightly (lots of ring -> engine renaming).

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 17 +++++++++--------
 drivers/gpu/drm/i915/intel_lrc.c           | 11 ++++++-----
 2 files changed, 15 insertions(+), 13 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +9 -8
index 6e500bf..2b902e3 100644
@@ -1292,6 +1292,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 {
 	struct drm_i915_private *dev_priv = to_i915(params->dev);
+	struct drm_i915_gem_request *req = params->request;
 	struct intel_engine_cs  *engine = params->engine;
 	u64 exec_start, exec_len;
 	int ret;
@@ -1303,12 +1304,12 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
-	ret = intel_ring_invalidate_all_caches(params->request);
+	ret = intel_ring_invalidate_all_caches(req);
 	if (ret)
 		return ret;
 
 	/* Switch to the correct context for the batch */
-	ret = i915_switch_context(params->request);
+	ret = i915_switch_context(req);
 	if (ret)
 		return ret;
 
@@ -1317,7 +1318,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 
 	if (engine == &dev_priv->engine[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_ring_begin(params->request, 4);
+		ret = intel_ring_begin(req, 4);
 		if (ret)
 			return ret;
 
@@ -1331,7 +1332,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	}
 
 	if (params->args_flags & I915_EXEC_GEN7_SOL_RESET) {
-		ret = i915_reset_gen7_sol_offsets(params->dev, params->request);
+		ret = i915_reset_gen7_sol_offsets(params->dev, req);
 		if (ret)
 			return ret;
 	}
@@ -1343,13 +1344,13 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	if (exec_len == 0)
 		exec_len = params->batch_obj->base.size;
 
-	ret = engine->dispatch_execbuffer(params->request,
-					exec_start, exec_len,
-					params->dispatch_flags);
+	ret = engine->dispatch_execbuffer(req,
+					  exec_start, exec_len,
+					  params->dispatch_flags);
 	if (ret)
 		return ret;
 
-	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
+	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
drivers/gpu/drm/i915/intel_lrc.c +6 -5
index 6ac063d..69df948 100644
@@ -1005,7 +1005,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 {
 	struct drm_i915_private *dev_priv = to_i915(params->dev);
-	struct intel_ringbuffer *ringbuf = params->request->ringbuf;
+	struct drm_i915_gem_request *req = params->request;
+	struct intel_ringbuffer *ringbuf = req->ringbuf;
 	struct intel_engine_cs *engine = params->engine;
 	u64 exec_start;
 	int ret;
@@ -1017,13 +1018,13 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
-	ret = logical_ring_invalidate_all_caches(params->request);
+	ret = logical_ring_invalidate_all_caches(req);
 	if (ret)
 		return ret;
 
 	if (engine == &dev_priv->engine[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_logical_ring_begin(params->request, 4);
+		ret = intel_logical_ring_begin(req, 4);
 		if (ret)
 			return ret;
 
@@ -1039,11 +1040,11 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	exec_start = params->batch_obj_vm_offset +
 		     params->args_batch_start_offset;
 
-	ret = engine->emit_bb_start(params->request, exec_start, params->dispatch_flags);
+	ret = engine->emit_bb_start(req, exec_start, params->dispatch_flags);
 	if (ret)
 		return ret;
 
-	trace_i915_gem_ring_dispatch(params->request, params->dispatch_flags);
+	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#5 lore raw

PATCH v6 5/34 drm/i915: Re-instate request->uniq because it is extremely useful

In reply to [email protected]

From: John Harrison <[email protected]>

The seqno value cannot always be used when debugging issues via trace points. This is because it can be reset back to start, especially during TDR type tests. Also, when the scheduler arrives the seqno is only valid while a given request is executing on the hardware. While the request is simply queued waiting for submission, it's seqno value will be zero (meaning invalid).

v4: Wrapped a long line to keep the style checker happy.

v5: Added uniq to the dispatch trace point [Svetlana Kukanova]

For: VIZ-5115

---
 drivers/gpu/drm/i915/i915_drv.h   |  5 +++++
 drivers/gpu/drm/i915/i915_gem.c   |  4 +++-
 drivers/gpu/drm/i915/i915_trace.h | 32 ++++++++++++++++++++++----------
 3 files changed, 30 insertions(+), 11 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +5 -0
index f978539..7492ce7 100644
@@ -2012,6 +2012,8 @@ struct drm_i915_private {
 
 	struct intel_encoder *dig_port_map[I915_MAX_PORTS];
 
+	uint32_t request_uniq;
+
 	/*
 	 * NOTE: This is the dri1/ums dungeon, don't add stuff here. Your patch
 	 * will be rejected. Instead look for a better place.
@@ -2287,6 +2289,9 @@ struct drm_i915_gem_request {
 	  */
 	u32 seqno;
 
+	/* Unique identifier which can be used for trace points & debug */
+	uint32_t uniq;
+
 	/** Position in the ringbuffer of the start of the request */
 	u32 head;
 
drivers/gpu/drm/i915/i915_gem.c +3 -1
index 21fce67..a632276 100644
@@ -3008,7 +3008,8 @@ static void i915_gem_request_fence_value_str(struct fence *req_fence,
 
 	req = container_of(req_fence, typeof(*req), fence);
 
-	snprintf(str, size, "%d [%d]", req->fence.seqno, req->seqno);
+	snprintf(str, size, "%d [%d:%d]", req->fence.seqno, req->uniq,
+		 req->seqno);
 }
 
 static const struct fence_ops i915_gem_request_fops = {
@@ -3085,6 +3086,7 @@ __i915_gem_request_alloc(struct intel_engine_cs *engine,
 
 	req->i915 = dev_priv;
 	req->engine = engine;
+	req->uniq = dev_priv->request_uniq++;
 	req->ctx  = ctx;
 	i915_gem_context_reference(req->ctx);
 
drivers/gpu/drm/i915/i915_trace.h +22 -10
index b7c7031..59a6266 100644
@@ -458,6 +458,7 @@ TRACE_EVENT(i915_gem_ring_sync_to,
 			     __field(u32, dev)
 			     __field(u32, sync_from)
 			     __field(u32, sync_to)
+			     __field(u32, uniq_to)
 			     __field(u32, seqno)
 			     ),
 
@@ -465,13 +466,14 @@ TRACE_EVENT(i915_gem_ring_sync_to,
 			   __entry->dev = from->dev->primary->index;
 			   __entry->sync_from = from->id;
 			   __entry->sync_to = to_req->engine->id;
+			   __entry->uniq_to = to_req->uniq;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   ),
 
-	    TP_printk("dev=%u, sync-from=%u, sync-to=%u, seqno=%u",
+	    TP_printk("dev=%u, sync-from=%u, sync-to=%u, seqno=%u, to_uniq=%u",
 		      __entry->dev,
 		      __entry->sync_from, __entry->sync_to,
-		      __entry->seqno)
+		      __entry->seqno, __entry->uniq_to)
 );
 
 TRACE_EVENT(i915_gem_ring_dispatch,
@@ -481,6 +483,7 @@ TRACE_EVENT(i915_gem_ring_dispatch,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, seqno)
 			     __field(u32, flags)
 			     ),
@@ -490,13 +493,15 @@ TRACE_EVENT(i915_gem_ring_dispatch,
 						i915_gem_request_get_engine(req);
 			   __entry->dev = engine->dev->primary->index;
 			   __entry->ring = engine->id;
+			   __entry->uniq = req->uniq;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   __entry->flags = flags;
 			   i915_trace_irq_get(engine, req);
 			   ),
 
-	    TP_printk("dev=%u, ring=%u, seqno=%u, flags=%x",
-		      __entry->dev, __entry->ring, __entry->seqno, __entry->flags)
+	    TP_printk("dev=%u, ring=%u, uniq=%u, seqno=%u, flags=%x",
+		      __entry->dev, __entry->ring, __entry->uniq,
+		      __entry->seqno, __entry->flags)
 );
 
 TRACE_EVENT(i915_gem_ring_flush,
@@ -506,6 +511,7 @@ TRACE_EVENT(i915_gem_ring_flush,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, invalidate)
 			     __field(u32, flush)
 			     ),
@@ -513,12 +519,13 @@ TRACE_EVENT(i915_gem_ring_flush,
 	    TP_fast_assign(
 			   __entry->dev = req->engine->dev->primary->index;
 			   __entry->ring = req->engine->id;
+			   __entry->uniq = req->uniq;
 			   __entry->invalidate = invalidate;
 			   __entry->flush = flush;
 			   ),
 
-	    TP_printk("dev=%u, ring=%x, invalidate=%04x, flush=%04x",
-		      __entry->dev, __entry->ring,
+	    TP_printk("dev=%u, ring=%x, request=%u, invalidate=%04x, flush=%04x",
+		      __entry->dev, __entry->ring, __entry->uniq,
 		      __entry->invalidate, __entry->flush)
 );
 
@@ -529,6 +536,7 @@ DECLARE_EVENT_CLASS(i915_gem_request,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, seqno)
 			     ),
 
@@ -537,11 +545,13 @@ DECLARE_EVENT_CLASS(i915_gem_request,
 						i915_gem_request_get_engine(req);
 			   __entry->dev = engine->dev->primary->index;
 			   __entry->ring = engine->id;
+			   __entry->uniq = req ? req->uniq : 0;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   ),
 
-	    TP_printk("dev=%u, ring=%u, seqno=%u",
-		      __entry->dev, __entry->ring, __entry->seqno)
+	    TP_printk("dev=%u, ring=%u, uniq=%u, seqno=%u",
+		      __entry->dev, __entry->ring, __entry->uniq,
+		      __entry->seqno)
 );
 
 DEFINE_EVENT(i915_gem_request, i915_gem_request_add,
@@ -590,6 +600,7 @@ TRACE_EVENT(i915_gem_request_wait_begin,
 	    TP_STRUCT__entry(
 			     __field(u32, dev)
 			     __field(u32, ring)
+			     __field(u32, uniq)
 			     __field(u32, seqno)
 			     __field(bool, blocking)
 			     ),
@@ -605,13 +616,14 @@ TRACE_EVENT(i915_gem_request_wait_begin,
 						i915_gem_request_get_engine(req);
 			   __entry->dev = engine->dev->primary->index;
 			   __entry->ring = engine->id;
+			   __entry->uniq = req ? req->uniq : 0;
 			   __entry->seqno = i915_gem_request_get_seqno(req);
 			   __entry->blocking =
 				     mutex_is_locked(&engine->dev->struct_mutex);
 			   ),
 
-	    TP_printk("dev=%u, ring=%u, seqno=%u, blocking=%s",
-		      __entry->dev, __entry->ring,
+	    TP_printk("dev=%u, ring=%u, uniq=%u, seqno=%u, blocking=%s",
+		      __entry->dev, __entry->ring, __entry->uniq,
 		      __entry->seqno, __entry->blocking ?  "yes (NB)" : "no")
 );
 
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#6 lore raw

PATCH v6 6/34 drm/i915: Start of GPU scheduler

In reply to [email protected]

From: John Harrison <[email protected]>

Initial creation of scheduler source files. Note that this patch implements most of the scheduler functionality but does not hook it in to the driver yet. It also leaves the scheduler code in 'pass through' mode so that even when it is hooked in, it will not actually do very much. This allows the hooks to be added one at a time in bite size chunks and only when the scheduler is finally enabled at the end does anything start happening.

The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.

The scheduler is notified when each batch buffer completes and updates its internal tracking accordingly. At the end of the completion interrupt processing, if any scheduler tracked batches were processed, the scheduler's deferred worker thread is woken up. This can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.

When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate. All nodes that are not submitted have their priority bumped. This ensures that low priority tasks do not get starved out by busy higher priority ones - everything will eventually get its turn to run.

Note that this patch does not implement pre-emptive scheduling. Only basic scheduling by re-ordering batch buffer submission is currently implemented. Pre-emption of actively executing batch buffers comes in the next patch series.

v2: Changed priority levels to +/-1023 due to feedback from Chris Wilson.

Removed redundant index from scheduler node.

Changed time stamps to use jiffies instead of raw monotonic. This provides lower resolution but improved compatibility with other i915 code.

Major re-write of completion tracking code due to struct fence conversion. The scheduler no longer has it's own private IRQ handler but just lets the existing request code handle completion events. Instead, the scheduler now hooks into the request notify code to be told when a request has completed.

Reduced driver mutex locking scope. Removal of scheduler nodes no longer grabs the mutex lock.

v3: Refactor of dependency generation to make the code more readable. Also added in read-read optimisation support - i.e., don't treat a shared read-only buffer as being a dependency.

Allowed the killing of queued nodes rather than only flying ones.

v4: Updated the commit message to better reflect the current state of the code. Downgraded some BUG_ONs to WARN_ONs. Used the correct array memory allocator function (kmalloc_array instead of kmalloc). Corrected the format of some comments. Wrapped some lines differently to keep the style checker happy.

Fixed a WARN_ON when killing nodes. The dependency removal code checks that nodes being destroyed do not have any oustanding dependencies (which would imply they should not have been executed yet). In the case of nodes being destroyed, e.g. due to context banning, then this might well be the case - they have not been executed and do indeed have outstanding dependencies.

Re-instated the code to disble interrupts when not in use. The underlying problem causing broken IRQ reference counts seems to have been fixed now.

v5: Shuffled various functions around to remove forward declarations as apparently these are frowned upon. Removed lots of white space as apparently having easy to read code is also frowned upon. Split the direct submission scheduler bypass code out into a separate function. Squashed down the i915_scheduler.c sections of various patches into this patch. Thus the later patches simply hook in existing code into various parts of the driver rather than adding the code as well. Added documentation to various functions. Re-worked the submit function in terms of mutex locking, error handling and exit paths. Split the delayed work handler function in half. Made use of the kernel 'clamp' macro. [Joonas Lahtinen]

Added runtime PM calls as these must be done at the top level before acquiring the driver mutex lock. [Chris Wilson]

Removed some obsolete debug code that had been forgotten about.

Moved more clean up code into the 'i915_gem_scheduler_clean_node()' function rather than replicating it in mutliple places.

Used lighter weight spinlocks.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' and 'assert_scheduler_lock_held()' helper macros. Renamed 'i915_gem_execbuff_release_batch_obj' to 'i915_gem_execbuf_release_batch_obj'. Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. Removed various unnecessary WARNs. Renamed 'saved_objects' to just 'objs'. Split code for counting incomplete nodes out into a separate function. Removed even more white space. Added a destroy() function. [review feedback from Joonas Lahtinen]

Added running totals of 'flying' and 'queued' nodes rather than re-calculating each time as a minor CPU performance optimisation.

Removed support for out of order seqno completion. All the prep work patch series (seqno to request conversion, late seqno assignment, etc.) that has now been done means that the scheduler no longer generates out of order seqno completions. Thus all the complex code for coping with such is no longer required and can be removed.

Fixed a bug in scheduler bypass mode introduced in the clean up code refactoring of v5. The clean up function was seeing the node in the wrong state and thus refusing to process it.

For: VIZ-1587

Cc Joonas Lahtinen <[email protected]>
---
 drivers/gpu/drm/i915/Makefile         |   1 +
 drivers/gpu/drm/i915/i915_dma.c       |   3 +
 drivers/gpu/drm/i915/i915_drv.h       |   6 +
 drivers/gpu/drm/i915/i915_gem.c       |   5 +
 drivers/gpu/drm/i915/i915_scheduler.c | 867 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h | 113 +++++
 6 files changed, 995 insertions(+)
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c
 create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h
drivers/gpu/drm/i915/Makefile +1 -0
index e9cdeb5..289fa73 100644
@@ -10,6 +10,7 @@ ccflags-y := -Werror
 i915-y := i915_drv.o \
 	  i915_irq.o \
 	  i915_params.o \
+	  i915_scheduler.o \
           i915_suspend.o \
 	  i915_sysfs.o \
 	  intel_csr.o \
drivers/gpu/drm/i915/i915_dma.c +3 -0
index b377753..2ad4071 100644
@@ -37,6 +37,7 @@
 #include "i915_drv.h"
 #include "i915_vgpu.h"
 #include "i915_trace.h"
+#include "i915_scheduler.h"
 #include <linux/pci.h>
 #include <linux/console.h>
 #include <linux/vt.h>
@@ -1448,6 +1449,8 @@ int i915_driver_unload(struct drm_device *dev)
 
 	intel_csr_ucode_fini(dev_priv);
 
+	i915_scheduler_destroy(dev_priv);
+
 	/* Free error state after interrupts are fully disabled. */
 	cancel_delayed_work_sync(&dev_priv->gpu_error.hangcheck_work);
 	i915_destroy_error_state(dev);
drivers/gpu/drm/i915/i915_drv.h +6 -0
index 7492ce7..7b62e2c 100644
@@ -1717,6 +1717,8 @@ struct i915_execbuffer_params {
 	struct drm_i915_gem_request     *request;
 };
 
+struct i915_scheduler;
+
 /* used in computing the new watermarks state */
 struct intel_wm_config {
 	unsigned int num_pipes_active;
@@ -1994,6 +1996,8 @@ struct drm_i915_private {
 
 	struct i915_runtime_pm pm;
 
+	struct i915_scheduler *scheduler;
+
 	/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */
 	struct {
 		int (*execbuf_submit)(struct i915_execbuffer_params *params,
@@ -2335,6 +2339,8 @@ struct drm_i915_gem_request {
 	/** process identifier submitting this request */
 	struct pid *pid;
 
+	struct i915_scheduler_queue_entry *scheduler_qe;
+
 	/**
 	 * The ELSP only accepts two elements at a time, so we queue
 	 * context/tail pairs on a given queue (ring->execlist_queue) until the
drivers/gpu/drm/i915/i915_gem.c +5 -0
index a632276..b7466cb 100644
@@ -32,6 +32,7 @@
 #include "i915_vgpu.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
+#include "i915_scheduler.h"
 #include <linux/shmem_fs.h>
 #include <linux/slab.h>
 #include <linux/swap.h>
@@ -5405,6 +5406,10 @@ int i915_gem_init(struct drm_device *dev)
 	 */
 	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
+	ret = i915_scheduler_init(dev);
+	if (ret)
+		goto out_unlock;
+
 	ret = i915_gem_init_userptr(dev);
 	if (ret)
 		goto out_unlock;
drivers/gpu/drm/i915/i915_scheduler.c new file +867 -0

867 lines not on this page · raw message

drivers/gpu/drm/i915/i915_scheduler.h new file +113 -0

113 lines not on this page · raw message

Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#7 lore raw

Re: [PATCH v6 06/34] drm/i915: Start of GPU scheduler

Hi,

Just a few random comments/questions. (not a full review!)

Quoted 303 lines · On 20/04/16 18:13, [email protected] wrote:

From: John Harrison <[email protected]>

Initial creation of scheduler source files. Note that this patch implements most of the scheduler functionality but does not hook it in to the driver yet. It also leaves the scheduler code in 'pass through' mode so that even when it is hooked in, it will not actually do very much. This allows the hooks to be added one at a time in bite size chunks and only when the scheduler is finally enabled at the end does anything start happening.

The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.

The scheduler is notified when each batch buffer completes and updates its internal tracking accordingly. At the end of the completion interrupt processing, if any scheduler tracked batches were processed, the scheduler's deferred worker thread is woken up. This can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.

When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate. All nodes that are not submitted have their priority bumped. This ensures that low priority tasks do not get starved out by busy higher priority ones - everything will eventually get its turn to run.

Note that this patch does not implement pre-emptive scheduling. Only basic scheduling by re-ordering batch buffer submission is currently implemented. Pre-emption of actively executing batch buffers comes in the next patch series.

v2: Changed priority levels to +/-1023 due to feedback from Chris Wilson.

Removed redundant index from scheduler node.

Changed time stamps to use jiffies instead of raw monotonic. This provides lower resolution but improved compatibility with other i915 code.

Major re-write of completion tracking code due to struct fence conversion. The scheduler no longer has it's own private IRQ handler but just lets the existing request code handle completion events. Instead, the scheduler now hooks into the request notify code to be told when a request has completed.

Reduced driver mutex locking scope. Removal of scheduler nodes no longer grabs the mutex lock.

v3: Refactor of dependency generation to make the code more readable. Also added in read-read optimisation support - i.e., don't treat a shared read-only buffer as being a dependency.

Allowed the killing of queued nodes rather than only flying ones.

v4: Updated the commit message to better reflect the current state of the code. Downgraded some BUG_ONs to WARN_ONs. Used the correct array memory allocator function (kmalloc_array instead of kmalloc). Corrected the format of some comments. Wrapped some lines differently to keep the style checker happy.

Fixed a WARN_ON when killing nodes. The dependency removal code checks that nodes being destroyed do not have any oustanding dependencies (which would imply they should not have been executed yet). In the case of nodes being destroyed, e.g. due to context banning, then this might well be the case - they have not been executed and do indeed have outstanding dependencies.

Re-instated the code to disble interrupts when not in use. The underlying problem causing broken IRQ reference counts seems to have been fixed now.

v5: Shuffled various functions around to remove forward declarations as apparently these are frowned upon. Removed lots of white space as apparently having easy to read code is also frowned upon. Split the direct submission scheduler bypass code out into a separate function. Squashed down the i915_scheduler.c sections of various patches into this patch. Thus the later patches simply hook in existing code into various parts of the driver rather than adding the code as well. Added documentation to various functions. Re-worked the submit function in terms of mutex locking, error handling and exit paths. Split the delayed work handler function in half. Made use of the kernel 'clamp' macro. [Joonas Lahtinen]

Added runtime PM calls as these must be done at the top level before acquiring the driver mutex lock. [Chris Wilson]

Removed some obsolete debug code that had been forgotten about.

Moved more clean up code into the 'i915_gem_scheduler_clean_node()' function rather than replicating it in mutliple places.

Used lighter weight spinlocks.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' and 'assert_scheduler_lock_held()' helper macros. Renamed 'i915_gem_execbuff_release_batch_obj' to 'i915_gem_execbuf_release_batch_obj'. Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. Removed various unnecessary WARNs. Renamed 'saved_objects' to just 'objs'. Split code for counting incomplete nodes out into a separate function. Removed even more white space. Added a destroy() function. [review feedback from Joonas Lahtinen]

Added running totals of 'flying' and 'queued' nodes rather than re-calculating each time as a minor CPU performance optimisation.

Removed support for out of order seqno completion. All the prep work patch series (seqno to request conversion, late seqno assignment, etc.) that has now been done means that the scheduler no longer generates out of order seqno completions. Thus all the complex code for coping with such is no longer required and can be removed.

Fixed a bug in scheduler bypass mode introduced in the clean up code refactoring of v5. The clean up function was seeing the node in the wrong state and thus refusing to process it.

For: VIZ-1587

Cc Joonas Lahtinen <[email protected]>
---
  drivers/gpu/drm/i915/Makefile         |   1 +
  drivers/gpu/drm/i915/i915_dma.c       |   3 +
  drivers/gpu/drm/i915/i915_drv.h       |   6 +
  drivers/gpu/drm/i915/i915_gem.c       |   5 +
  drivers/gpu/drm/i915/i915_scheduler.c | 867 ++++++++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/i915_scheduler.h | 113 +++++
  6 files changed, 995 insertions(+)

create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h

drivers/gpu/drm/i915/Makefile +1 -0
index e9cdeb5..289fa73 100644
@@ -10,6 +10,7 @@ ccflags-y := -Werror
  i915-y := i915_drv.o \
  	  i915_irq.o \
  	  i915_params.o \
+	  i915_scheduler.o \
            i915_suspend.o \
  	  i915_sysfs.o \
  	  intel_csr.o \
drivers/gpu/drm/i915/i915_dma.c +3 -0
index b377753..2ad4071 100644
@@ -37,6 +37,7 @@
  #include "i915_drv.h"
  #include "i915_vgpu.h"
  #include "i915_trace.h"
+#include "i915_scheduler.h"
  #include <linux/pci.h>
  #include <linux/console.h>
  #include <linux/vt.h>
@@ -1448,6 +1449,8 @@ int i915_driver_unload(struct drm_device *dev)
 
  	intel_csr_ucode_fini(dev_priv);
 
+	i915_scheduler_destroy(dev_priv);
+
  	/* Free error state after interrupts are fully disabled. */
  	cancel_delayed_work_sync(&dev_priv->gpu_error.hangcheck_work);
  	i915_destroy_error_state(dev);
drivers/gpu/drm/i915/i915_drv.h +6 -0
index 7492ce7..7b62e2c 100644
@@ -1717,6 +1717,8 @@ struct i915_execbuffer_params {
  	struct drm_i915_gem_request     *request;
  };
 
+struct i915_scheduler;
+
  /* used in computing the new watermarks state */
  struct intel_wm_config {
  	unsigned int num_pipes_active;
@@ -1994,6 +1996,8 @@ struct drm_i915_private {
 
  	struct i915_runtime_pm pm;
 
+	struct i915_scheduler *scheduler;
+
  	/* Abstract the submission mechanism (legacy ringbuffer or execlists) away */
  	struct {
  		int (*execbuf_submit)(struct i915_execbuffer_params *params,
@@ -2335,6 +2339,8 @@ struct drm_i915_gem_request {
  	/** process identifier submitting this request */
  	struct pid *pid;
 
+	struct i915_scheduler_queue_entry *scheduler_qe;
+
  	/**
  	 * The ELSP only accepts two elements at a time, so we queue
  	 * context/tail pairs on a given queue (ring->execlist_queue) until the
drivers/gpu/drm/i915/i915_gem.c +5 -0
index a632276..b7466cb 100644
@@ -32,6 +32,7 @@
  #include "i915_vgpu.h"
  #include "i915_trace.h"
  #include "intel_drv.h"
+#include "i915_scheduler.h"
  #include <linux/shmem_fs.h>
  #include <linux/slab.h>
  #include <linux/swap.h>
@@ -5405,6 +5406,10 @@ int i915_gem_init(struct drm_device *dev)
  	 */
  	intel_uncore_forcewake_get(dev_priv, FORCEWAKE_ALL);
 
+	ret = i915_scheduler_init(dev);
+	if (ret)
+		goto out_unlock;
+
  	ret = i915_gem_init_userptr(dev);
  	if (ret)
  		goto out_unlock;
drivers/gpu/drm/i915/i915_scheduler.c new file +50 -0
new file mode 100644
index 0000000..9d628b9
@@ -0,0 +1,867 @@
+/*
+ * Copyright (c) 2014 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include "i915_drv.h"
+#include "intel_drv.h"
+#include "i915_scheduler.h"
+
+#define for_each_scheduler_node(node, id)				\
+	list_for_each_entry((node), &scheduler->node_queue[(id)], link)
+
+#define assert_scheduler_lock_held(scheduler)				\
+	do {								\
+		WARN_ONCE(!spin_is_locked(&(scheduler)->lock), "Spinlock not locked!");	\
+	} while(0)
+
+/**
+ * i915_scheduler_is_enabled - Returns true if the scheduler is enabled.
+ * @dev: DRM device
+ */
+bool i915_scheduler_is_enabled(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = to_i915(dev);
+
+	return dev_priv->scheduler != NULL;
+}
+
+/**
+ * i915_scheduler_init - Initialise the scheduler.
+ * @dev: DRM device

drivers/gpu/drm/i915/i915_scheduler.c line 60

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10 -0
+ * Returns zero on success or -ENOMEM if memory allocations fail.
+ */
+int i915_scheduler_init(struct drm_device *dev)
+{
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+	int e;
+
+	if (scheduler)
+		return 0;

Probably a GEM_BUG_ON or something.

Quoted 108 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +108 -0
+
+	scheduler = kzalloc(sizeof(*scheduler), GFP_KERNEL);
+	if (!scheduler)
+		return -ENOMEM;
+
+	spin_lock_init(&scheduler->lock);
+
+	for (e = 0; e < I915_NUM_ENGINES; e++) {
+		INIT_LIST_HEAD(&scheduler->node_queue[e]);
+		scheduler->counts[e].flying = 0;
+		scheduler->counts[e].queued = 0;
+	}
+
+	/* Default tuning values: */
+	scheduler->priority_level_min     = -1023;
+	scheduler->priority_level_max     = 1023;
+	scheduler->priority_level_bump    = 50;
+	scheduler->priority_level_preempt = 900;
+	scheduler->min_flying             = 2;
+
+	dev_priv->scheduler = scheduler;
+
+	return 0;
+}
+
+/**
+ * i915_scheduler_destroy - Get rid of the scheduler.
+ * @dev: DRM device
+ */
+void i915_scheduler_destroy(struct drm_i915_private *dev_priv)
+{
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+	int e;
+
+	if (!scheduler)
+		return;
+
+	for (e = 0; e < I915_NUM_ENGINES; e++)
+		WARN(!list_empty(&scheduler->node_queue[e]), "Destroying with list entries on engine %d!", e);
+
+	kfree(scheduler);
+	dev_priv->scheduler = NULL;
+}
+
+/*
+ * Add a popped node back in to the queue. For example, because the engine
+ * was hung when execfinal() was called and thus the engine submission needs
+ * to be retried later.
+ */
+static void i915_scheduler_node_requeue(struct i915_scheduler *scheduler,
+					struct i915_scheduler_queue_entry *node)
+{
+	assert_scheduler_lock_held(scheduler);
+
+	WARN_ON(!I915_SQS_IS_FLYING(node));
+
+	/* Seqno will be reassigned on relaunch */
+	node->params.request->seqno = 0;
+	node->status = I915_SQS_QUEUED;
+	scheduler->counts[node->params.engine->id].flying--;
+	scheduler->counts[node->params.engine->id].queued++;
+}
+
+/*
+ * Give up on a node completely. For example, because it is causing the
+ * engine to hang or is using some resource that no longer exists.
+ */
+static void i915_scheduler_node_kill(struct i915_scheduler *scheduler,
+				     struct i915_scheduler_queue_entry *node)
+{
+	assert_scheduler_lock_held(scheduler);
+
+	WARN_ON(I915_SQS_IS_COMPLETE(node));
+
+	if (I915_SQS_IS_FLYING(node))
+		scheduler->counts[node->params.engine->id].flying--;
+	else
+		scheduler->counts[node->params.engine->id].queued--;
+
+	node->status = I915_SQS_DEAD;
+}
+
+/* Mark a node as in flight on the hardware. */
+static void i915_scheduler_node_fly(struct i915_scheduler_queue_entry *node)
+{
+	struct drm_i915_private *dev_priv = to_i915(node->params.dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs *engine = node->params.engine;
+
+	assert_scheduler_lock_held(scheduler);
+
+	WARN_ON(node->status != I915_SQS_POPPED);
+
+	/*
+	 * Add the node (which should currently be in state popped) to the
+	 * front of the queue. This ensure that flying nodes are always held
+	 * in hardware submission order.
+	 */
+	list_add(&node->link, &scheduler->node_queue[engine->id]);
+
+	node->status = I915_SQS_FLYING;
+
+	scheduler->counts[engine->id].flying++;
+
+	if (!(scheduler->flags[engine->id] & I915_SF_INTERRUPTS_ENABLED)) {
+		bool success = true;
+
+		success = engine->irq_get(engine);

drivers/gpu/drm/i915/i915_scheduler.c line 178

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10 -0
+		if (success)
+			scheduler->flags[engine->id] |= I915_SF_INTERRUPTS_ENABLED;
+	}
+}
+
+static inline uint32_t i915_scheduler_count_flying(struct i915_scheduler *scheduler,
+					    struct intel_engine_cs *engine)
+{
+	return scheduler->counts[engine->id].flying;
+}

Maybe num_flying would be a more obvious name, or at least less worry inducing.

Quoted 90 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +90 -0
+
+static void i915_scheduler_priority_bump_clear(struct i915_scheduler *scheduler)
+{
+	struct i915_scheduler_queue_entry *node;
+	int i;
+
+	assert_scheduler_lock_held(scheduler);
+
+	/*
+	 * Ensure circular dependencies don't cause problems and that a bump
+	 * by object usage only bumps each using buffer once:
+	 */
+	for (i = 0; i < I915_NUM_ENGINES; i++) {
+		for_each_scheduler_node(node, i)
+			node->bumped = false;
+	}
+}
+
+static int i915_scheduler_priority_bump(struct i915_scheduler *scheduler,
+				struct i915_scheduler_queue_entry *target,
+				uint32_t bump)
+{
+	uint32_t new_priority;
+	int i, count;
+
+	if (target->priority >= scheduler->priority_level_max)
+		return 1;
+
+	if (target->bumped)
+		return 0;
+
+	new_priority = target->priority + bump;
+	if ((new_priority <= target->priority) ||
+	    (new_priority > scheduler->priority_level_max))
+		target->priority = scheduler->priority_level_max;
+	else
+		target->priority = new_priority;
+
+	count = 1;
+	target->bumped = true;
+
+	for (i = 0; i < target->num_deps; i++) {
+		if (!target->dep_list[i])
+			continue;
+
+		if (target->dep_list[i]->bumped)
+			continue;
+
+		count += i915_scheduler_priority_bump(scheduler,
+						      target->dep_list[i],
+						      bump);
+	}
+
+	return count;
+}
+
+/*
+ * Nodes are considered valid dependencies if they are queued on any engine
+ * or if they are in flight on a different engine. In flight on the same
+ * engine is no longer interesting for non-premptive nodes as the engine
+ * serialises execution. For pre-empting nodes, all in flight dependencies
+ * are valid as they must not be jumped by the act of pre-empting.
+ *
+ * Anything that is neither queued nor flying is uninteresting.
+ */
+static inline bool i915_scheduler_is_dependency_valid(
+			struct i915_scheduler_queue_entry *node, uint32_t idx)
+{
+	struct i915_scheduler_queue_entry *dep;
+
+	dep = node->dep_list[idx];
+	if (!dep)
+		return false;
+
+	if (I915_SQS_IS_QUEUED(dep))
+		return true;
+
+	if (I915_SQS_IS_FLYING(dep)) {
+		if (node->params.engine != dep->params.engine)
+			return true;
+	}
+
+	return false;
+}
+
+static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *engine,
+				struct i915_scheduler_queue_entry **pop_node)
+{
+	struct drm_i915_private *dev_priv = to_i915(engine->dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;

drivers/gpu/drm/i915/i915_scheduler.c line 278

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10 -0
+	struct i915_scheduler_queue_entry *best = NULL;
+	struct i915_scheduler_queue_entry *node;
+	int ret;
+	int i;
+	bool any_queued = false;
+	bool has_local, has_remote, only_remote = false;
+
+	assert_scheduler_lock_held(scheduler);
+
+	*pop_node = NULL;

Looks to be not needed since there is *pop_node = best below.

Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +7 -0
+	ret = -ENODATA;
+
+	for_each_scheduler_node(node, engine->id) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+		any_queued = true;
+

drivers/gpu/drm/i915/i915_scheduler.c line 295

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10 -0
+		has_local  = false;
+		has_remote = false;
+		for (i = 0; i < node->num_deps; i++) {
+			if (!i915_scheduler_is_dependency_valid(node, i))
+				continue;
+
+			if (node->dep_list[i]->params.engine == node->params.engine)
+				has_local = true;
+			else
+				has_remote = true;

Would it be worth breaking early from this loop if has_local && has_remote?

Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +6 -0
+		}
+
+		if (has_remote && !has_local)
+			only_remote = true;
+
+		if (!has_local && !has_remote) {

drivers/gpu/drm/i915/i915_scheduler.c line 311

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10 -0
+			if (!best ||
+			    (node->priority > best->priority))
+				best = node;
+		}
+	}
+
+	if (best) {
+		list_del(&best->link);
+
+		INIT_LIST_HEAD(&best->link);

list_del_init can replace the two lines above.

Quoted 132 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +132 -0
+		best->status = I915_SQS_POPPED;
+
+		scheduler->counts[engine->id].queued--;
+
+		ret = 0;
+	} else {
+		/* Can only get here if:
+		 * (a) there are no buffers in the queue
+		 * (b) all queued buffers are dependent on other buffers
+		 *     e.g. on a buffer that is in flight on a different engine
+		 */
+		if (only_remote) {
+			/* The only dependent buffers are on another engine. */
+			ret = -EAGAIN;
+		} else if (any_queued) {
+			/* It seems that something has gone horribly wrong! */
+			WARN_ONCE(true, "Broken dependency tracking on engine %d!\n",
+				  (int) engine->id);
+		}
+	}
+
+	*pop_node = best;
+	return ret;
+}
+
+/*
+ * NB: The driver mutex lock must be held before calling this function. It is
+ * only really required during the actual back end submission call. However,
+ * attempting to acquire a mutex while holding a spin lock is a Bad Idea.
+ * And releasing the one before acquiring the other leads to other code
+ * being run and interfering.
+ */
+static int i915_scheduler_submit(struct intel_engine_cs *engine)
+{
+	struct drm_i915_private *dev_priv = to_i915(engine->dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry *node;
+	int ret, count = 0, flying;
+
+	WARN_ON(!mutex_is_locked(&engine->dev->struct_mutex));
+
+	spin_lock_irq(&scheduler->lock);
+
+	WARN_ON(scheduler->flags[engine->id] & I915_SF_SUBMITTING);
+	scheduler->flags[engine->id] |= I915_SF_SUBMITTING;
+
+	/* First time around, complain if anything unexpected occurs: */
+	ret = i915_scheduler_pop_from_queue_locked(engine, &node);
+	if (ret)
+		goto error;
+
+	do {
+		WARN_ON(node->params.engine != engine);
+		WARN_ON(node->status != I915_SQS_POPPED);
+		count++;
+
+		/*
+		 * The call to pop above will have removed the node from the
+		 * list. So add it back in and mark it as in flight.
+		 */
+		i915_scheduler_node_fly(node);
+
+		spin_unlock_irq(&scheduler->lock);
+		ret = dev_priv->gt.execbuf_final(&node->params);
+		spin_lock_irq(&scheduler->lock);
+
+		/*
+		 * Handle failed submission but first check that the
+		 * watchdog/reset code has not nuked the node while we
+		 * weren't looking:
+		 */
+		if (ret && (node->status != I915_SQS_DEAD)) {
+			bool requeue = true;
+
+			/*
+			 * Oh dear! Either the node is broken or the engine is
+			 * busy. So need to kill the node or requeue it and try
+			 * again later as appropriate.
+			 */
+
+			switch (-ret) {
+			case ENODEV:
+			case ENOENT:
+				/* Fatal errors. Kill the node. */
+				requeue = false;
+				i915_scheduler_node_kill(scheduler, node);
+				break;
+
+			case EAGAIN:
+			case EBUSY:
+			case EIO:
+			case ENOMEM:
+			case ERESTARTSYS:
+			case EINTR:
+				/* Supposedly recoverable errors. */
+				break;
+
+			default:
+				/*
+				 * Assume the error is recoverable and hope
+				 * for the best.
+				 */
+				MISSING_CASE(-ret);
+				break;
+			}
+
+			if (requeue) {
+				i915_scheduler_node_requeue(scheduler, node);
+				/*
+				 * No point spinning if the engine is currently
+				 * unavailable so just give up and come back
+				 * later.
+				 */
+				break;
+			}
+		}
+
+		/* Keep launching until the sky is sufficiently full. */
+		flying = i915_scheduler_count_flying(scheduler, engine);
+		if (flying >= scheduler->min_flying)
+			break;
+
+		/* Grab another node and go round again... */
+		ret = i915_scheduler_pop_from_queue_locked(engine, &node);
+	} while (ret == 0);
+
+	/* Don't complain about not being able to submit extra entries */
+	if (ret == -ENODATA)
+		ret = 0;
+
+	/*
+	 * Bump the priority of everything that was not submitted to prevent

drivers/gpu/drm/i915/i915_scheduler.c line 453

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10 -0
+	 * starvation of low priority tasks by a spamming high priority task.
+	 */
+	i915_scheduler_priority_bump_clear(scheduler);
+	for_each_scheduler_node(node, engine->id) {
+		if (!I915_SQS_IS_QUEUED(node))
+			continue;
+
+		i915_scheduler_priority_bump(scheduler, node,
+					     scheduler->priority_level_bump);
+	}

bump_clear will iterate queues for all engines and then you iterate it again. Would this be equivalent:

       for (i = 0; i < I915_NUM_ENGINES; i++) {
               for_each_scheduler_node(node, i) {
                       node->bumped = false;
                       if (i == engine->id && I915_SQS_IS_QUEUED(node))
		i915_scheduler_priority_bump(scheduler, 		node, 
scheduler->priority_level_bump);
               }
        }

Advantage is one loop fewer.

Quoted 262 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +262 -0
+
+	/* On success, return the number of buffers submitted. */
+	if (ret == 0)
+		ret = count;
+
+error:
+	scheduler->flags[engine->id] &= ~I915_SF_SUBMITTING;
+	spin_unlock_irq(&scheduler->lock);
+	return ret;
+}
+
+static void i915_generate_dependencies(struct i915_scheduler *scheduler,
+				       struct i915_scheduler_queue_entry *node,
+				       uint32_t engine)
+{
+	struct i915_scheduler_obj_entry *this, *that;
+	struct i915_scheduler_queue_entry *test;
+	int i, j;
+	bool found;
+
+	for_each_scheduler_node(test, engine) {
+		if (I915_SQS_IS_COMPLETE(test))
+			continue;
+
+		/*
+		 * Batches on the same engine for the same
+		 * context must be kept in order.
+		 */
+		found = (node->params.ctx == test->params.ctx) &&
+			(node->params.engine == test->params.engine);
+
+		/*
+		 * Batches working on the same objects must
+		 * be kept in order.
+		 */
+		for (i = 0; (i < node->num_objs) && !found; i++) {
+			this = node->objs + i;
+
+			for (j = 0; j < test->num_objs; j++) {
+				that = test->objs + j;
+
+				if (this->obj != that->obj)
+					continue;
+
+				/* Only need to worry about writes */
+				if (this->read_only && that->read_only)
+					continue;
+
+				found = true;
+				break;
+			}
+		}
+
+		if (found) {
+			node->dep_list[node->num_deps] = test;
+			node->num_deps++;
+		}
+	}
+}
+
+static int i915_scheduler_queue_execbuffer_bypass(struct i915_scheduler_queue_entry *qe)
+{
+	struct drm_i915_private *dev_priv = to_i915(qe->params.dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+	int ret;
+
+	scheduler->flags[qe->params.engine->id] |= I915_SF_SUBMITTING;
+	ret = dev_priv->gt.execbuf_final(&qe->params);
+	scheduler->flags[qe->params.engine->id] &= ~I915_SF_SUBMITTING;
+
+	/*
+	 * Don't do any clean up on failure because the caller will
+	 * do it all anyway.
+	 */
+	if (ret)
+		return ret;
+
+	/* Need to release any resources held by the node: */
+	qe->status = I915_SQS_COMPLETE;
+	i915_scheduler_clean_node(qe);
+
+	return 0;
+}
+
+static inline uint32_t i915_scheduler_count_incomplete(struct i915_scheduler *scheduler)
+{
+	int e, incomplete = 0;
+
+	for (e = 0; e < I915_NUM_ENGINES; e++)
+		incomplete += scheduler->counts[e].queued + scheduler->counts[e].flying;
+
+	return incomplete;
+}
+
+/**
+ * i915_scheduler_queue_execbuffer - Submit a batch buffer request to the
+ * scheduler.
+ * @qe: The batch buffer request to be queued.
+ * The expectation is the qe passed in is a local stack variable. This
+ * function will copy its contents into a freshly allocated list node. The
+ * new node takes ownership of said contents so the original qe should simply
+ * be discarded and not cleaned up (i.e. don't free memory it points to or
+ * dereference objects it holds). The node is added to the scheduler's queue
+ * and the batch buffer will be submitted to the hardware at some future
+ * point in time (which may be immediately, before returning or may be quite
+ * a lot later).
+ */
+int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
+{
+	struct drm_i915_private *dev_priv = to_i915(qe->params.dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+	struct intel_engine_cs *engine = qe->params.engine;
+	struct i915_scheduler_queue_entry *node;
+	bool not_flying;
+	int i, e;
+	int incomplete;
+
+	/* Bypass the scheduler and send the buffer immediately? */
+	if (1/*!i915.enable_scheduler*/)
+		return i915_scheduler_queue_execbuffer_bypass(qe);
+
+	node = kmalloc(sizeof(*node), GFP_KERNEL);
+	if (!node)
+		return -ENOMEM;
+
+	*node = *qe;
+	INIT_LIST_HEAD(&node->link);
+	node->status = I915_SQS_QUEUED;
+	node->stamp  = jiffies;
+	i915_gem_request_reference(node->params.request);
+
+	WARN_ON(node->params.request->scheduler_qe);
+	node->params.request->scheduler_qe = node;
+
+	/*
+	 * Need to determine the number of incomplete entries in the list as
+	 * that will be the maximum size of the dependency list.
+	 *
+	 * Note that the allocation must not be made with the spinlock acquired
+	 * as kmalloc can sleep. However, the unlock/relock is safe because no
+	 * new entries can be queued up during the unlock as the i915 driver
+	 * mutex is still held. Entries could be removed from the list but that
+	 * just means the dep_list will be over-allocated which is fine.
+	 */
+	spin_lock_irq(&scheduler->lock);
+	incomplete = i915_scheduler_count_incomplete(scheduler);
+
+	/* Temporarily unlock to allocate memory: */
+	spin_unlock_irq(&scheduler->lock);
+	if (incomplete) {
+		node->dep_list = kmalloc_array(incomplete,
+					       sizeof(*node->dep_list),
+					       GFP_KERNEL);
+		if (!node->dep_list) {
+			kfree(node);
+			return -ENOMEM;
+		}
+	} else
+		node->dep_list = NULL;
+
+	spin_lock_irq(&scheduler->lock);
+	node->num_deps = 0;
+
+	if (node->dep_list) {
+		for (e = 0; e < I915_NUM_ENGINES; e++)
+			i915_generate_dependencies(scheduler, node, e);
+
+		WARN_ON(node->num_deps > incomplete);
+	}
+
+	node->priority = clamp(node->priority,
+			       scheduler->priority_level_min,
+			       scheduler->priority_level_max);
+
+	if ((node->priority > 0) && node->num_deps) {
+		i915_scheduler_priority_bump_clear(scheduler);
+
+		for (i = 0; i < node->num_deps; i++)
+			i915_scheduler_priority_bump(scheduler,
+					node->dep_list[i], node->priority);
+	}
+
+	list_add_tail(&node->link, &scheduler->node_queue[engine->id]);
+
+	not_flying = i915_scheduler_count_flying(scheduler, engine) <
+						 scheduler->min_flying;
+
+	scheduler->counts[engine->id].queued++;
+
+	spin_unlock_irq(&scheduler->lock);
+
+	if (not_flying)
+		i915_scheduler_submit(engine);
+
+	return 0;
+}
+
+/**
+ * i915_scheduler_notify_request - Notify the scheduler that the given
+ * request has completed on the hardware.
+ * @req: Request structure which has completed
+ * @preempt: Did it complete pre-emptively?
+ * A sequence number has popped out of the hardware and the request handling
+ * code has mapped it back to a request and will mark that request complete.
+ * It also calls this function to notify the scheduler about the completion
+ * so the scheduler's node can be updated appropriately.
+ * Returns true if the request is scheduler managed, false if not. The return
+ * value is combined for all freshly completed requests and if any were true
+ * then i915_scheduler_wakeup() is called so the scheduler can do further
+ * processing (submit more work) at the end.
+ */
+bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
+{
+	struct drm_i915_private *dev_priv = to_i915(req->engine->dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry *node = req->scheduler_qe;
+	unsigned long flags;
+
+	if (!node)
+		return false;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	WARN_ON(!I915_SQS_IS_FLYING(node));
+
+	/* Node was in flight so mark it as complete. */
+	if (req->cancelled)
+		node->status = I915_SQS_DEAD;
+	else
+		node->status = I915_SQS_COMPLETE;
+
+	scheduler->counts[req->engine->id].flying--;
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return true;
+}
+
+static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
+				struct i915_scheduler_queue_entry *remove)
+{
+	struct i915_scheduler_queue_entry *node;
+	int i, r;
+	int count = 0;
+
+	/*
+	 * Ensure that a node is not being removed which is still dependent
+	 * upon other (not completed) work. If that happens, it implies
+	 * something has gone very wrong with the dependency tracking! Note
+	 * that there is no need to worry if this node has been explicitly
+	 * killed for some reason - it might be being killed before it got
+	 * sent to the hardware.
+	 */
+	if (remove->status != I915_SQS_DEAD) {
+		for (i = 0; i < remove->num_deps; i++)
+			if ((remove->dep_list[i]) &&
+			    (!I915_SQS_IS_COMPLETE(remove->dep_list[i])))
+				count++;
+		WARN_ON(count);
+	}
+
+	/*

drivers/gpu/drm/i915/i915_scheduler.c line 725

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10 -0
+	 * Remove this node from the dependency lists of any other node which
+	 * might be waiting on it.
+	 */
+	for (r = 0; r < I915_NUM_ENGINES; r++) {
+		for_each_scheduler_node(node, r) {
+			for (i = 0; i < node->num_deps; i++) {
+				if (node->dep_list[i] != remove)
+					continue;
+
+				node->dep_list[i] = NULL;

Can the same node be listed in other's node dependency list multiple times? If not you could break after clearing it and not iterate the rest of the list.

Quoted 61 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +61 -0
+			}
+		}
+	}
+
+	return 0;
+}
+
+/**
+ * i915_scheduler_wakeup - wake the scheduler's worker thread
+ * @dev: DRM device
+ * Called at the end of seqno interrupt processing if any request has
+ * completed that corresponds to a scheduler node.
+ */
+void i915_scheduler_wakeup(struct drm_device *dev)
+{
+	/* XXX: Need to call i915_scheduler_remove() via work handler. */
+}
+
+/**
+ * i915_scheduler_clean_node - free up any allocations/references
+ * associated with the given scheduler queue entry.
+ * @node: Queue entry structure which is complete
+ * After a give batch buffer completes on the hardware, all the information
+ * required to resubmit it is no longer required. However, the node entry
+ * itself might still be required for tracking purposes for a while longer.
+ * This function should be called as soon as the node is known to be complete
+ * so that these resources may be freed even though the node itself might
+ * hang around.
+ */
+void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node)
+{
+	if (!I915_SQS_IS_COMPLETE(node)) {
+		WARN(!node->params.request->cancelled,
+		     "Cleaning active node: %d!\n", node->status);
+		return;
+	}
+
+	if (node->params.batch_obj) {
+		/*
+		 * The batch buffer must be unpinned before it is unreferenced
+		 * otherwise the unpin fails with a missing vma!?
+		 */
+		if (node->params.dispatch_flags & I915_DISPATCH_SECURE)
+			i915_gem_execbuf_release_batch_obj(node->params.batch_obj);
+
+		node->params.batch_obj = NULL;
+	}
+
+	/* And anything else owned by the node: */
+	if (node->params.cliprects) {
+		kfree(node->params.cliprects);
+		node->params.cliprects = NULL;
+	}
+}
+
+static bool i915_scheduler_remove(struct i915_scheduler *scheduler,
+				  struct intel_engine_cs *engine,
+				  struct list_head *remove)
+{
+	struct i915_scheduler_queue_entry *node, *node_next;
+	bool do_submit;

drivers/gpu/drm/i915/i915_scheduler.c line 796

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10 -0
+
+	spin_lock_irq(&scheduler->lock);
+
+	INIT_LIST_HEAD(remove);
+	list_for_each_entry_safe(node, node_next, &scheduler->node_queue[engine->id], link) {
+		if (!I915_SQS_IS_COMPLETE(node))
+			break;
+
+		list_del(&node->link);
+		list_add(&node->link, remove);

list_move

drivers/gpu/drm/i915/i915_scheduler.c line 801

Quoted 5 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +5 -0
+
+		/* Strip the dependency info while the mutex is still locked */
+		i915_scheduler_remove_dependent(scheduler, node);
+
+		continue;

Could kill the continue. :)

Quoted 18 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +18 -0
+	}
+
+	/*
+	 * Release the interrupt reference count if there are no longer any
+	 * nodes to worry about.
+	 */
+	if (list_empty(&scheduler->node_queue[engine->id]) &&
+	    (scheduler->flags[engine->id] & I915_SF_INTERRUPTS_ENABLED)) {
+		engine->irq_put(engine);
+		scheduler->flags[engine->id] &= ~I915_SF_INTERRUPTS_ENABLED;
+	}
+
+	/* Launch more packets now? */
+	do_submit = (scheduler->counts[engine->id].queued > 0) &&
+		    (scheduler->counts[engine->id].flying < scheduler->min_flying);
+
+	spin_unlock_irq(&scheduler->lock);
+

drivers/gpu/drm/i915/i915_scheduler.c line 829

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +10 -0
+	return do_submit;
+}
+
+void i915_scheduler_process_work(struct intel_engine_cs *engine)
+{
+	struct drm_i915_private *dev_priv = to_i915(engine->dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry *node;
+	bool do_submit;
+	struct list_head remove;

Could do LIST_HEAD(remove); to declare an empty list and then wouldn't have to do INIT_LIST_HEAD in i915_scheduler_remove. Would probably be better to group the logic together like that.

Quoted 148 lines
drivers/gpu/drm/i915/i915_scheduler.c new file +38 -0
+
+	if (list_empty(&scheduler->node_queue[engine->id]))
+		return;
+
+	/* Remove completed nodes. */
+	do_submit = i915_scheduler_remove(scheduler, engine, &remove);
+
+	if (!do_submit && list_empty(&remove))
+		return;
+
+	/* Need to grab the pm lock outside of the mutex lock */
+	if (do_submit)
+		intel_runtime_pm_get(dev_priv);
+
+	mutex_lock(&engine->dev->struct_mutex);
+
+	if (do_submit)
+		i915_scheduler_submit(engine);
+
+	while (!list_empty(&remove)) {
+		node = list_first_entry(&remove, typeof(*node), link);
+		list_del(&node->link);
+
+		/* Free up all the DRM references */
+		i915_scheduler_clean_node(node);
+
+		/* And anything else owned by the node: */
+		node->params.request->scheduler_qe = NULL;
+		i915_gem_request_unreference(node->params.request);
+		kfree(node->dep_list);
+		kfree(node);
+	}
+
+	mutex_unlock(&engine->dev->struct_mutex);
+
+	if (do_submit)
+		intel_runtime_pm_put(dev_priv);
+}
drivers/gpu/drm/i915/i915_scheduler.h new file +103 -0
new file mode 100644
index 0000000..c895c4c
@@ -0,0 +1,113 @@
+/*
+ * Copyright (c) 2014 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#ifndef _I915_SCHEDULER_H_
+#define _I915_SCHEDULER_H_
+
+enum i915_scheduler_queue_status {
+	/* Limbo: */
+	I915_SQS_NONE = 0,
+	/* Not yet submitted to hardware: */
+	I915_SQS_QUEUED,
+	/* Popped from queue, ready to fly: */
+	I915_SQS_POPPED,
+	/* Sent to hardware for processing: */
+	I915_SQS_FLYING,
+	/* Finished processing on the hardware: */
+	I915_SQS_COMPLETE,
+	/* Killed by watchdog or catastrophic submission failure: */
+	I915_SQS_DEAD,
+	/* Limit value for use with arrays/loops */
+	I915_SQS_MAX
+};
+
+#define I915_SQS_IS_QUEUED(node)	(((node)->status == I915_SQS_QUEUED))
+#define I915_SQS_IS_FLYING(node)	(((node)->status == I915_SQS_FLYING))
+#define I915_SQS_IS_COMPLETE(node)	(((node)->status == I915_SQS_COMPLETE) || \
+					 ((node)->status == I915_SQS_DEAD))
+
+struct i915_scheduler_obj_entry {
+	struct drm_i915_gem_object *obj;
+	bool read_only;
+};
+
+struct i915_scheduler_queue_entry {
+	/* Any information required to submit this batch buffer to the hardware */
+	struct i915_execbuffer_params params;
+
+	/* -1023 = lowest priority, 0 = default, 1023 = highest */
+	int32_t priority;
+	bool bumped;
+
+	/* Objects referenced by this batch buffer */
+	struct i915_scheduler_obj_entry *objs;
+	int num_objs;
+
+	/* Batch buffers this one is dependent upon */
+	struct i915_scheduler_queue_entry **dep_list;
+	int num_deps;
+
+	enum i915_scheduler_queue_status status;
+	unsigned long stamp;
+
+	/* List of all scheduler queue entry nodes */
+	struct list_head link;
+};
+
+struct i915_scheduler_node_states {
+	uint32_t flying;
+	uint32_t queued;
+};
+
+struct i915_scheduler {
+	struct list_head node_queue[I915_NUM_ENGINES];
+	uint32_t flags[I915_NUM_ENGINES];
+	spinlock_t lock;
+
+	/* Node counts: */
+	struct i915_scheduler_node_states counts[I915_NUM_ENGINES];
+
+	/* Tuning parameters: */
+	int32_t priority_level_min;
+	int32_t priority_level_max;
+	int32_t priority_level_bump;
+	int32_t priority_level_preempt;
+	uint32_t min_flying;
+};
+
+/* Flag bits for i915_scheduler::flags */
+enum {
+	I915_SF_INTERRUPTS_ENABLED  = (1 << 0),
+	I915_SF_SUBMITTING          = (1 << 1),
+};

drivers/gpu/drm/i915/i915_scheduler.h line 113

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.h new file +10 -0
+
+bool i915_scheduler_is_enabled(struct drm_device *dev);
+int i915_scheduler_init(struct drm_device *dev);
+void i915_scheduler_destroy(struct drm_i915_private *dev_priv);
+void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node);
+int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
+bool i915_scheduler_notify_request(struct drm_i915_gem_request *req);
+void i915_scheduler_wakeup(struct drm_device *dev);
+
+#endif  /* _I915_SCHEDULER_H_ */

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#8 lore raw

PATCH v6 7/34 drm/i915: Disable hardware semaphores when GPU scheduler is enabled

In reply to [email protected]

From: John Harrison <[email protected]>

Hardware sempahores require seqno values to be continuously incrementing. However, the scheduler's reordering of batch buffers means that the seqno values going through the hardware could be out of order. Thus semaphores can not be used.

On the other hand, the scheduler superceeds the need for hardware semaphores anyway. Having one ring stall waiting for something to complete on another ring is inefficient if that ring could be working on some other, independent task. This is what the scheduler is meant to do - keep the hardware as busy as possible by reordering batch buffers to avoid dependency stalls.

v4: Downgraded a BUG_ON to WARN_ON as the latter is preferred.

v5: Squashed the i915_scheduler.c portions down into the 'start of scheduler' patch. [Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_drv.c         | 9 +++++++++
 drivers/gpu/drm/i915/intel_ringbuffer.c | 4 ++++
 2 files changed, 13 insertions(+)
drivers/gpu/drm/i915/i915_drv.c +9 -0
index 29b4e79..1c5bdd0 100644
@@ -34,6 +34,7 @@
 #include "i915_drv.h"
 #include "i915_trace.h"
 #include "intel_drv.h"
+#include "i915_scheduler.h"
 
 #include <linux/apple-gmux.h>
 #include <linux/console.h>
@@ -532,6 +533,14 @@ void intel_detect_pch(struct drm_device *dev)
 
 bool i915_semaphore_is_enabled(struct drm_device *dev)
 {
+	/* Hardware semaphores are not compatible with the scheduler due to the
+	 * seqno values being potentially out of order. However, semaphores are
+	 * also not required as the scheduler will handle interring dependencies
+	 * and try do so in a way that does not cause dead time on the hardware.
+	 */
+	if (i915_scheduler_is_enabled(dev))
+		return false;
+
 	if (INTEL_INFO(dev)->gen < 6)
 		return false;
 
drivers/gpu/drm/i915/intel_ringbuffer.c +4 -0
index 5f209ba..e3f2237 100644
@@ -33,6 +33,7 @@
 #include <drm/i915_drm.h>
 #include "i915_trace.h"
 #include "intel_drv.h"
+#include "i915_scheduler.h"
 
 int __intel_ring_space(int head, int tail, int size)
 {
@@ -1478,6 +1479,9 @@ gen6_ring_sync(struct drm_i915_gem_request *waiter_req,
 	u32 wait_mbox = signaller->semaphore.mbox.wait[waiter->id];
 	int ret;
 
+	/* Arithmetic on sequence numbers is unreliable with a scheduler. */
+	WARN_ON(i915_scheduler_is_enabled(signaller->dev));
+
 	/* Throughout all of the GEM code, seqno passed implies our current
 	 * seqno is >= the last seqno executed. However for hardware the
 	 * comparison is strictly greater than.
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#9 lore raw

PATCH v6 8/34 drm/i915: Force MMIO flips when scheduler enabled

In reply to [email protected]

From: John Harrison <[email protected]>

MMIO flips are the preferred mechanism now but more importantly, pipe based flips cause issues for the scheduler. Specifically, submitting work to the rings around the side of the scheduler could cause that work to be lost if the scheduler generates a pre-emption event on that ring.

For: VIZ-1587

---
 drivers/gpu/drm/i915/intel_display.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/i915/intel_display.c +3 -0
index e550e5b..9407934 100644
@@ -47,6 +47,7 @@
 #include <linux/dma_remapping.h>
 #include <linux/reservation.h>
 #include <linux/dma-buf.h>
+#include "i915_scheduler.h"
 
 /* Primary plane formats for gen <= 3 */
 static const uint32_t i8xx_primary_formats[] = {
@@ -11244,6 +11245,8 @@ static bool use_mmio_flip(struct intel_engine_cs *engine,
 		return true;
 	else if (i915.enable_execlists)
 		return true;
+	else if (i915_scheduler_is_enabled(engine->dev))
+		return true;
 	else if (obj->base.dma_buf &&
 		 !reservation_object_test_signaled_rcu(obj->base.dma_buf->resv,
 						       false))
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#10 lore raw

PATCH v6 9/34 drm/i915: Added scheduler hook when closing DRM file handles

In reply to [email protected]

From: John Harrison <[email protected]>

The scheduler decouples the submission of batch buffers to the driver with submission of batch buffers to the hardware. Thus it is possible for an application to close its DRM file handle while there is still work outstanding. That means the scheduler needs to know about file close events so it can remove the file pointer from such orphaned batch buffers and not attempt to dereference it later.

v3: Updated to not wait for outstanding work to complete but merely remove the file handle reference. The wait was getting excessively complicated with inter-ring dependencies, pre-emption, and other such issues.

v4: Changed some white space to keep the style checker happy.

v5: Added function documentation and removed apparently objectionable white space. [Joonas Lahtinen]

Used lighter weight spinlocks.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' helper macro. Updated to use 'to_i915()' instead of dev_private. Removed the return value from i915_scheduler_closefile() as it is not used for anything. [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_dma.c       |  2 ++
 drivers/gpu/drm/i915/i915_scheduler.c | 45 +++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 3 files changed, 48 insertions(+)
drivers/gpu/drm/i915/i915_dma.c +2 -0
index 2ad4071..75dc313 100644
@@ -1509,6 +1509,8 @@ void i915_driver_lastclose(struct drm_device *dev)
 
 void i915_driver_preclose(struct drm_device *dev, struct drm_file *file)
 {
+	i915_scheduler_closefile(dev, file);
+
 	mutex_lock(&dev->struct_mutex);
 	i915_gem_context_close(dev, file);
 	i915_gem_release(dev, file);
drivers/gpu/drm/i915/i915_scheduler.c +45 -0

48 lines not on this page · raw message

drivers/gpu/drm/i915/i915_scheduler.h +1 -0
index c895c4c..40398bb 100644
@@ -105,6 +105,7 @@ enum {
 bool i915_scheduler_is_enabled(struct drm_device *dev);
 int i915_scheduler_init(struct drm_device *dev);
 void i915_scheduler_destroy(struct drm_i915_private *dev_priv);
+void i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file);
 void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node);
 int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool i915_scheduler_notify_request(struct drm_i915_gem_request *req);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#11 lore raw

PATCH v6 10/34 drm/i915: Added scheduler hook into i915_gem_request_notify()

In reply to [email protected]

From: John Harrison <[email protected]>

The scheduler needs to know when requests have completed so that it can keep its own internal state up to date and can submit new requests to the hardware from its queue.

v2: Updated due to changes in request handling. The operation is now reversed from before. Rather than the scheduler being in control of completion events, it is now the request code itself. The scheduler merely receives a notification event. It can then optionally request it's worker thread be woken up after all completion processing is complete.

v4: Downgraded a BUG_ON to a WARN_ON as the latter is preferred.

v5: Squashed the i915_scheduler.c portions down into the 'start of scheduler' patch. [Joonas Lahtinen]

v6: Updated to newer nightly (lots of ring -> engine renaming).

Changed an '|=' to an 'if() ='. [review feedback from Joonas Lahtinen]

For: VIZ-1587

Cc Joonas Lahtinen <[email protected]>
---
 drivers/gpu/drm/i915/i915_gem.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +17 -0
index b7466cb..14dc641 100644
@@ -2918,6 +2918,7 @@ void i915_gem_request_notify(struct intel_engine_cs *engine, bool fence_locked)
 {
 	struct drm_i915_gem_request *req, *req_next;
 	unsigned long flags;
+	bool wake_sched = false;
 	u32 seqno;
 
 	if (list_empty(&engine->fence_signal_list)) {
@@ -2956,6 +2957,15 @@ void i915_gem_request_notify(struct intel_engine_cs *engine, bool fence_locked)
 		 */
 		list_del_init(&req->signal_link);
 
+		/*
+		 * NB: Must notify the scheduler before signalling
+		 * the node. Otherwise the node can get retired first
+		 * and call scheduler_clean() while the scheduler
+		 * thinks it is still active.
+		 */
+		if (i915_scheduler_notify_request(req))
+			wake_sched = true;
+
 		if (!req->cancelled) {
 			fence_signal_locked(&req->fence);
 			trace_i915_gem_request_complete(req);
@@ -2972,6 +2982,13 @@ void i915_gem_request_notify(struct intel_engine_cs *engine, bool fence_locked)
 
 	if (!fence_locked)
 		spin_unlock_irqrestore(&engine->fence_lock, flags);
+
+	/* Necessary? Or does the fence_signal() call do an implicit wakeup? */
+	wake_up_all(&engine->irq_queue);
+
+	/* Final scheduler processing after all individual updates are done. */
+	if (wake_sched)
+		i915_scheduler_wakeup(engine->dev);
 }
 
 static const char *i915_gem_request_get_driver_name(struct fence *req_fence)
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#12 lore raw

PATCH v6 11/34 drm/i915: Added deferred work handler for scheduler

In reply to [email protected]

From: John Harrison <[email protected]>

The scheduler needs to do interrupt triggered work that is too complex to do in the interrupt handler. Thus it requires a deferred work handler to process such tasks asynchronously.

v2: Updated to reduce mutex lock usage. The lock is now only held for the minimum time within the remove function rather than for the whole of the worker thread's operation.

v5: Removed objectionable white space and added some documentation. [Joonas Lahtinen]

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added an i915_scheduler_destroy() function instead of doing explicit clean up of scheduler internals from i915_driver_unload(). [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_drv.h       | 10 ++++++++++
 drivers/gpu/drm/i915/i915_gem.c       |  2 ++
 drivers/gpu/drm/i915/i915_scheduler.c | 28 ++++++++++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 4 files changed, 39 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +10 -0
index 7b62e2c..ed9d829 100644
@@ -1296,6 +1296,16 @@ struct i915_gem_mm {
 	struct delayed_work retire_work;
 
 	/**
+	 * New scheme is to get an interrupt after every work packet
+	 * in order to allow the low latency scheduling of pending
+	 * packets. The idea behind adding new packets to a pending
+	 * queue rather than directly into the hardware ring buffer
+	 * is to allow high priority packets to over take low priority
+	 * ones.
+	 */
+	struct work_struct scheduler_work;
+
+	/**
 	 * When we detect an idle GPU, we want to turn on
 	 * powersaving features. So once we see that there
 	 * are no more requests outstanding and no more
drivers/gpu/drm/i915/i915_gem.c +2 -0
index 14dc641..50c45f3 100644
@@ -5546,6 +5546,8 @@ i915_gem_load_init(struct drm_device *dev)
 			  i915_gem_retire_work_handler);
 	INIT_DELAYED_WORK(&dev_priv->mm.idle_work,
 			  i915_gem_idle_work_handler);
+	INIT_WORK(&dev_priv->mm.scheduler_work,
+				i915_scheduler_work_handler);
 	init_waitqueue_head(&dev_priv->gpu_error.reset_queue);
 
 	dev_priv->relative_constants_mode = I915_EXEC_CONSTANTS_REL_GENERAL;
drivers/gpu/drm/i915/i915_scheduler.c +26 -2
index 6dd9838..2dc5597 100644
@@ -95,6 +95,8 @@ void i915_scheduler_destroy(struct drm_i915_private *dev_priv)
 	if (!scheduler)
 		return;
 
+	cancel_work_sync(&dev_priv->mm.scheduler_work);
+
 	for (e = 0; e < I915_NUM_ENGINES; e++)
 		WARN(!list_empty(&scheduler->node_queue[e]), "Destroying with list entries on engine %d!", e);
 
@@ -738,7 +740,9 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
  */
 void i915_scheduler_wakeup(struct drm_device *dev)
 {
-	/* XXX: Need to call i915_scheduler_remove() via work handler. */
+	struct drm_i915_private *dev_priv = to_i915(dev);
+
+	queue_work(dev_priv->wq, &dev_priv->mm.scheduler_work);
 }
 
 /**
@@ -820,7 +824,7 @@ static bool i915_scheduler_remove(struct i915_scheduler *scheduler,
 	return do_submit;
 }
 
-void i915_scheduler_process_work(struct intel_engine_cs *engine)
+static void i915_scheduler_process_work(struct intel_engine_cs *engine)
 {
 	struct drm_i915_private *dev_priv = to_i915(engine->dev);
 	struct i915_scheduler *scheduler = dev_priv->scheduler;
@@ -867,6 +871,26 @@ void i915_scheduler_process_work(struct intel_engine_cs *engine)
 }
 
 /**
+ * i915_scheduler_work_handler - scheduler's work handler callback.
+ * @work: Work structure
+ * A lot of the scheduler's work must be done asynchronously in response to
+ * an interrupt or other event. However, that work cannot be done at
+ * interrupt time or in the context of the event signaller (which might in
+ * fact be an interrupt). Thus a worker thread is required. This function
+ * will cause the thread to wake up and do its processing.
+ */
+void i915_scheduler_work_handler(struct work_struct *work)
+{
+	struct intel_engine_cs *engine;
+	struct drm_i915_private *dev_priv;
+
+	dev_priv = container_of(work, struct drm_i915_private, mm.scheduler_work);
+
+	for_each_engine(engine, dev_priv)
+		i915_scheduler_process_work(engine);
+}
+
+/**
  * i915_scheduler_closefile - notify the scheduler that a DRM file handle
  * has been closed.
  * @dev: DRM device
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
index 40398bb..b8d4a343 100644
@@ -110,5 +110,6 @@ void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node);
 int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void i915_scheduler_wakeup(struct drm_device *dev);
+void i915_scheduler_work_handler(struct work_struct *work);
 
 #endif  /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#13 lore raw

Re: [PATCH v6 11/34] drm/i915: Added deferred work handler for scheduler

Hi,

More random comments.

Quoted 78 lines · On 20/04/16 18:13, [email protected] wrote:

From: John Harrison <[email protected]>

The scheduler needs to do interrupt triggered work that is too complex to do in the interrupt handler. Thus it requires a deferred work handler to process such tasks asynchronously.

v2: Updated to reduce mutex lock usage. The lock is now only held for the minimum time within the remove function rather than for the whole of the worker thread's operation.

v5: Removed objectionable white space and added some documentation. [Joonas Lahtinen]

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added an i915_scheduler_destroy() function instead of doing explicit clean up of scheduler internals from i915_driver_unload(). [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
  drivers/gpu/drm/i915/i915_drv.h       | 10 ++++++++++
  drivers/gpu/drm/i915/i915_gem.c       |  2 ++
  drivers/gpu/drm/i915/i915_scheduler.c | 28 ++++++++++++++++++++++++++--
  drivers/gpu/drm/i915/i915_scheduler.h |  1 +
  4 files changed, 39 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +10 -0
index 7b62e2c..ed9d829 100644
@@ -1296,6 +1296,16 @@ struct i915_gem_mm {
  	struct delayed_work retire_work;
 
  	/**
+	 * New scheme is to get an interrupt after every work packet
+	 * in order to allow the low latency scheduling of pending
+	 * packets. The idea behind adding new packets to a pending
+	 * queue rather than directly into the hardware ring buffer
+	 * is to allow high priority packets to over take low priority
+	 * ones.
+	 */
+	struct work_struct scheduler_work;
+
+	/**
  	 * When we detect an idle GPU, we want to turn on
  	 * powersaving features. So once we see that there
  	 * are no more requests outstanding and no more
drivers/gpu/drm/i915/i915_gem.c +2 -0
index 14dc641..50c45f3 100644
@@ -5546,6 +5546,8 @@ i915_gem_load_init(struct drm_device *dev)
  			  i915_gem_retire_work_handler);
  	INIT_DELAYED_WORK(&dev_priv->mm.idle_work,
  			  i915_gem_idle_work_handler);
+	INIT_WORK(&dev_priv->mm.scheduler_work,
+				i915_scheduler_work_handler);
  	init_waitqueue_head(&dev_priv->gpu_error.reset_queue);
 
  	dev_priv->relative_constants_mode = I915_EXEC_CONSTANTS_REL_GENERAL;
drivers/gpu/drm/i915/i915_scheduler.c +2 -0
index 6dd9838..2dc5597 100644
@@ -95,6 +95,8 @@ void i915_scheduler_destroy(struct drm_i915_private *dev_priv)
  	if (!scheduler)
  		return;
 
+	cancel_work_sync(&dev_priv->mm.scheduler_work);
+
  	for (e = 0; e < I915_NUM_ENGINES; e++)
  		WARN(!list_empty(&scheduler->node_queue[e]), "Destroying with list entries on engine %d!", e);
 

drivers/gpu/drm/i915/i915_scheduler.c static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler, · line 745

Quoted 7 lines
drivers/gpu/drm/i915/i915_scheduler.c +3 -1
@@ -738,7 +740,9 @@ static int i915_scheduler_remove_dependent(struct i915_scheduler *scheduler,
   */
  void i915_scheduler_wakeup(struct drm_device *dev)
  {
-	/* XXX: Need to call i915_scheduler_remove() via work handler. */
+	struct drm_i915_private *dev_priv = to_i915(dev);
+
+	queue_work(dev_priv->wq, &dev_priv->mm.scheduler_work);

As I commented in person, sharing this wq with the rest of the driver could introduce scheduling latency since it is an ordered (one work item at a time) queue.

It would probably be good to create a dedicated wq for the scheduler, maybe even WQ_HIGHPRI one.

Quoted 23 lines
drivers/gpu/drm/i915/i915_scheduler.c +8 -1
  }
 
  /**
@@ -820,7 +824,7 @@ static bool i915_scheduler_remove(struct i915_scheduler *scheduler,
  	return do_submit;
  }
 
-void i915_scheduler_process_work(struct intel_engine_cs *engine)
+static void i915_scheduler_process_work(struct intel_engine_cs *engine)
  {
  	struct drm_i915_private *dev_priv = to_i915(engine->dev);
  	struct i915_scheduler *scheduler = dev_priv->scheduler;
@@ -867,6 +871,26 @@ void i915_scheduler_process_work(struct intel_engine_cs *engine)
  }
 
  /**
+ * i915_scheduler_work_handler - scheduler's work handler callback.
+ * @work: Work structure
+ * A lot of the scheduler's work must be done asynchronously in response to
+ * an interrupt or other event. However, that work cannot be done at
+ * interrupt time or in the context of the event signaller (which might in
+ * fact be an interrupt). Thus a worker thread is required. This function
+ * will cause the thread to wake up and do its processing.

drivers/gpu/drm/i915/i915_scheduler.c void i915_scheduler_process_work(struct intel_engine_cs *engine) · line 890

Quoted 10 lines
drivers/gpu/drm/i915/i915_scheduler.c +10 -0
+ */
+void i915_scheduler_work_handler(struct work_struct *work)
+{
+	struct intel_engine_cs *engine;
+	struct drm_i915_private *dev_priv;
+
+	dev_priv = container_of(work, struct drm_i915_private, mm.scheduler_work);
+
+	for_each_engine(engine, dev_priv)
+		i915_scheduler_process_work(engine);

I wonder how easy or hard it would be to refactor things a bit so that the i915_scheduler_process_work does not have to grab and drop the global scheduler->lock multiple times. Maybe splitting i915_scheduler_process_work in some stages could help, like the remove bit and submit or something.

Quoted 12 lines
drivers/gpu/drm/i915/i915_scheduler.c +3 -0
+}
+
+/**
   * i915_scheduler_closefile - notify the scheduler that a DRM file handle
   * has been closed.
   * @dev: DRM device
drivers/gpu/drm/i915/i915_scheduler.h +0 -0
index 40398bb..b8d4a343 100644

drivers/gpu/drm/i915/i915_scheduler.h void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node); · line 115

Quoted 6 lines
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
@@ -110,5 +110,6 @@ void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node);
  int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
  bool i915_scheduler_notify_request(struct drm_i915_gem_request *req);
  void i915_scheduler_wakeup(struct drm_device *dev);
+void i915_scheduler_work_handler(struct work_struct *work);
 
  #endif  /* _I915_SCHEDULER_H_ */

Regards,

Tvrtko
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#14 lore raw

PATCH v6 12/34 drm/i915: Redirect execbuffer_final() via scheduler

In reply to [email protected]

From: John Harrison <[email protected]>

Updated the execbuffer() code to pass the packaged up batch buffer information to the scheduler rather than calling execbuffer_final() directly. The scheduler queue() code is currently a stub which simply chains on to _final() immediately.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Renamed 'i915_gem_execbuff_release_batch_obj' to 'i915_gem_execbuf_release_batch_obj'. [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 18 +++++++-----------
 drivers/gpu/drm/i915/intel_lrc.c           | 12 ++++--------
 2 files changed, 11 insertions(+), 19 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +7 -11
index 2b902e3..f7f3eb0 100644
@@ -33,6 +33,7 @@
 #include "intel_drv.h"
 #include <linux/dma_remapping.h>
 #include <linux/uaccess.h>
+#include "i915_scheduler.h"
 
 #define  __EXEC_OBJECT_HAS_PIN (1<<31)
 #define  __EXEC_OBJECT_HAS_FENCE (1<<30)
@@ -1227,6 +1228,7 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas)
 {
+	struct i915_scheduler_queue_entry *qe;
 	struct drm_device *dev = params->dev;
 	struct intel_engine_cs *engine = params->engine;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -1271,17 +1273,11 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	ret = dev_priv->gt.execbuf_final(params);
+	qe = container_of(params, typeof(*qe), params);
+	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
 		return ret;
 
-	/*
-	 * Free everything that was stored in the QE structure (until the
-	 * scheduler arrives and does it instead):
-	 */
-	if (params->dispatch_flags & I915_DISPATCH_SECURE)
-		i915_gem_execbuf_release_batch_obj(params->batch_obj);
-
 	return 0;
 }
 
@@ -1471,8 +1467,8 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	struct intel_engine_cs *engine;
 	struct intel_context *ctx;
 	struct i915_address_space *vm;
-	struct i915_execbuffer_params params_master; /* XXX: will be removed later */
-	struct i915_execbuffer_params *params = &params_master;
+	struct i915_scheduler_queue_entry qe;
+	struct i915_execbuffer_params *params = &qe.params;
 	const u32 ctx_id = i915_execbuffer2_get_context_id(*args);
 	u32 dispatch_flags;
 	int ret;
@@ -1538,7 +1534,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	else
 		vm = &ggtt->base;
 
-	memset(&params_master, 0x00, sizeof(params_master));
+	memset(&qe, 0x00, sizeof(qe));
 
 	eb = eb_create(args);
 	if (eb == NULL) {
drivers/gpu/drm/i915/intel_lrc.c +4 -8
index 69df948..6bf0e3f 100644
@@ -137,6 +137,7 @@
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
 #include "intel_mocs.h"
+#include "i915_scheduler.h"
 
 #define GEN9_LR_CONTEXT_RENDER_SIZE (22 * PAGE_SIZE)
 #define GEN8_LR_CONTEXT_RENDER_SIZE (20 * PAGE_SIZE)
@@ -942,6 +943,7 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 			       struct drm_i915_gem_execbuffer2 *args,
 			       struct list_head *vmas)
 {
+	struct i915_scheduler_queue_entry *qe;
 	struct drm_device       *dev = params->dev;
 	struct intel_engine_cs *engine = params->engine;
 	struct drm_i915_private *dev_priv = dev->dev_private;
@@ -984,17 +986,11 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
-	ret = dev_priv->gt.execbuf_final(params);
+	qe = container_of(params, typeof(*qe), params);
+	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
 		return ret;
 
-	/*
-	 * Free everything that was stored in the QE structure (until the
-	 * scheduler arrives and does it instead):
-	 */
-	if (params->dispatch_flags & I915_DISPATCH_SECURE)
-		i915_gem_execbuf_release_batch_obj(params->batch_obj);
-
 	return 0;
 }
 
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#15 lore raw

PATCH v6 13/34 drm/i915: Keep the reserved space mechanism happy

In reply to [email protected]

From: John Harrison <[email protected]>

Ring space is reserved when constructing a request to ensure that the subsequent 'add_request()' call cannot fail due to waiting for space on a busy or broken GPU. However, the scheduler jumps in to the middle of the execbuffer process between request creation and request submission. Thus it needs to cancel the reserved space when the request is simply added to the scheduler's queue and not yet submitted. Similarly, it needs to re-reserve the space when it finally does want to send the batch buffer to the hardware.

v3: Updated to use locally cached request pointer.

v5: Updated due to changes to earlier patches in series - for runtime PM calls and splitting bypass mode into a separate function.

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 20 ++++++++++++++------
 drivers/gpu/drm/i915/i915_scheduler.c      |  4 ++++
 drivers/gpu/drm/i915/intel_lrc.c           | 13 +++++++++++--
 3 files changed, 29 insertions(+), 8 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +14 -6
index f7f3eb0..5df7ae2 100644
@@ -1296,18 +1296,22 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	ret = intel_ring_reserve_space(req);
+	if (ret)
+		goto error;
+
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
 	ret = intel_ring_invalidate_all_caches(req);
 	if (ret)
-		return ret;
+		goto error;
 
 	/* Switch to the correct context for the batch */
 	ret = i915_switch_context(req);
 	if (ret)
-		return ret;
+		goto error;
 
 	WARN(params->ctx->ppgtt && params->ctx->ppgtt->pd_dirty_rings & (1<<engine->id),
 	     "%s didn't clear reload\n", engine->name);
@@ -1316,7 +1320,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	    params->instp_mode != dev_priv->relative_constants_mode) {
 		ret = intel_ring_begin(req, 4);
 		if (ret)
-			return ret;
+			goto error;
 
 		intel_ring_emit(engine, MI_NOOP);
 		intel_ring_emit(engine, MI_LOAD_REGISTER_IMM(1));
@@ -1330,7 +1334,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	if (params->args_flags & I915_EXEC_GEN7_SOL_RESET) {
 		ret = i915_reset_gen7_sol_offsets(params->dev, req);
 		if (ret)
-			return ret;
+			goto error;
 	}
 
 	exec_len   = params->args_batch_len;
@@ -1344,13 +1348,17 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 					  exec_start, exec_len,
 					  params->dispatch_flags);
 	if (ret)
-		return ret;
+		goto error;
 
 	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
-	return 0;
+error:
+	if (ret)
+		intel_ring_reserved_space_cancel(req->ringbuf);
+
+	return ret;
 }
 
 /**
drivers/gpu/drm/i915/i915_scheduler.c +4 -0
index 2dc5597..2fb3f52 100644
@@ -519,6 +519,8 @@ static int i915_scheduler_queue_execbuffer_bypass(struct i915_scheduler_queue_en
 	struct i915_scheduler *scheduler = dev_priv->scheduler;
 	int ret;
 
+	intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
+
 	scheduler->flags[qe->params.engine->id] |= I915_SF_SUBMITTING;
 	ret = dev_priv->gt.execbuf_final(&qe->params);
 	scheduler->flags[qe->params.engine->id] &= ~I915_SF_SUBMITTING;
@@ -584,6 +586,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	node->stamp  = jiffies;
 	i915_gem_request_reference(node->params.request);
 
+	intel_ring_reserved_space_cancel(node->params.request->ringbuf);
+
 	WARN_ON(node->params.request->scheduler_qe);
 	node->params.request->scheduler_qe = node;
 
drivers/gpu/drm/i915/intel_lrc.c +11 -2
index 6bf0e3f..0a4ef61 100644
@@ -1010,13 +1010,17 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	ret = intel_logical_ring_reserve_space(req);
+	if (ret)
+		goto err;
+
 	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
 	ret = logical_ring_invalidate_all_caches(req);
 	if (ret)
-		return ret;
+		goto err;
 
 	if (engine == &dev_priv->engine[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
@@ -1038,13 +1042,18 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 
 	ret = engine->emit_bb_start(req, exec_start, params->dispatch_flags);
 	if (ret)
-		return ret;
+		goto err;
 
 	trace_i915_gem_ring_dispatch(req, params->dispatch_flags);
 
 	i915_gem_execbuffer_retire_commands(params);
 
 	return 0;
+
+err:
+	intel_ring_reserved_space_cancel(params->request->ringbuf);
+
+	return ret;
 }
 
 void intel_execlists_retire_requests(struct intel_engine_cs *engine)
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#16 lore raw

PATCH v6 14/34 drm/i915: Added tracking/locking of batch buffer objects

In reply to [email protected]

From: John Harrison <[email protected]>

The scheduler needs to track interdependencies between batch buffers. These are calculated by analysing the object lists of the buffers and looking for commonality. The scheduler also needs to keep those buffers locked long after the initial IOCTL call has returned to user land.

v3: Updated to support read-read optimisation.

v5: Updated due to changes to earlier patches in series for splitting bypass mode into a separate function and consoliding the clean up code.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Renamed 'saved_objects' to just 'objs'. [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 45 ++++++++++++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.c      | 15 ++++++++++
 2 files changed, 58 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +43 -2
index 5df7ae2..e105aae 100644
@@ -1479,7 +1479,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	struct i915_execbuffer_params *params = &qe.params;
 	const u32 ctx_id = i915_execbuffer2_get_context_id(*args);
 	u32 dispatch_flags;
-	int ret;
+	int ret, i;
 	bool need_relocs;
 
 	if (!i915_gem_check_execbuffer(args))
@@ -1552,6 +1552,12 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		goto pre_mutex_err;
 	}
 
+	qe.objs = kzalloc(sizeof(*qe.objs) * args->buffer_count, GFP_KERNEL);
+	if (!qe.objs) {
+		ret = -ENOMEM;
+		goto err;
+	}
+
 	/* Look up object handles */
 	ret = eb_lookup_vmas(eb, exec, args, vm, file);
 	if (ret)
@@ -1676,9 +1682,32 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->args_DR1                = args->DR1;
 	params->args_DR4                = args->DR4;
 	params->batch_obj               = batch_obj;
-	params->ctx                     = ctx;
 	params->request                 = req;
 
+	/*
+	 * Save away the list of objects used by this batch buffer for the
+	 * purpose of tracking inter-buffer dependencies.
+	 */
+	for (i = 0; i < args->buffer_count; i++) {
+		struct drm_i915_gem_object *obj;
+
+		/*
+		 * NB: 'drm_gem_object_lookup()' increments the object's
+		 * reference count and so must be matched by a
+		 * 'drm_gem_object_unreference' call.
+		 */
+		obj = to_intel_bo(drm_gem_object_lookup(dev, file,
+							  exec[i].handle));
+		qe.objs[i].obj       = obj;
+		qe.objs[i].read_only = obj->base.pending_write_domain == 0;
+
+	}
+	qe.num_objs = i;
+
+	/* Lock and save the context object as well. */
+	i915_gem_context_reference(ctx);
+	params->ctx = ctx;
+
 	ret = dev_priv->gt.execbuf_submit(params, args, &eb->vmas);
 	if (ret)
 		goto err_client;
@@ -1714,6 +1743,18 @@ err:
 	i915_gem_context_unreference(ctx);
 	eb_destroy(eb);
 
+	/* Need to release the objects: */
+	if (qe.objs) {
+		for (i = 0; i < qe.num_objs; i++)
+			drm_gem_object_unreference(&qe.objs[i].obj->base);
+
+		kfree(qe.objs);
+	}
+
+	/* Context too */
+	if (params->ctx)
+		i915_gem_context_unreference(params->ctx);
+
 	/*
 	 * If the request was created but not successfully submitted then it
 	 * must be freed again. If it was submitted then it is being tracked
drivers/gpu/drm/i915/i915_scheduler.c +15 -0
index 2fb3f52..cdb86da 100644
@@ -762,6 +762,8 @@ void i915_scheduler_wakeup(struct drm_device *dev)
  */
 void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node)
 {
+	int i;
+
 	if (!I915_SQS_IS_COMPLETE(node)) {
 		WARN(!node->params.request->cancelled,
 		     "Cleaning active node: %d!\n", node->status);
@@ -779,6 +781,19 @@ void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node)
 		node->params.batch_obj = NULL;
 	}
 
+	/* Release the locked buffers: */
+	for (i = 0; i < node->num_objs; i++)
+		drm_gem_object_unreference(&node->objs[i].obj->base);
+	kfree(node->objs);
+	node->objs = NULL;
+	node->num_objs = 0;
+
+	/* Context too: */
+	if (node->params.ctx) {
+		i915_gem_context_unreference(node->params.ctx);
+		node->params.ctx = NULL;
+	}
+
 	/* And anything else owned by the node: */
 	if (node->params.cliprects) {
 		kfree(node->params.cliprects);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#17 lore raw

PATCH v6 15/34 drm/i915: Hook scheduler node clean up into retire requests

In reply to [email protected]

From: John Harrison <[email protected]>

The scheduler keeps its own lock on various DRM objects in order to guarantee safe access long after the original execbuff IOCTL has completed. This is especially important when pre-emption is enabled as the batch buffer might need to be submitted to the hardware multiple times. This patch hooks the clean up of these locks into the request retire function. The request can only be retired after it has completed on the hardware and thus is no longer eligible for re-submission. Thus there is no point holding on to the locks beyond that time.

v3: Updated to not WARN when cleaning a node that is being cancelled. The clean will happen later so skipping it at the point of cancellation is fine.

v5: Squashed the i915_scheduler.c portions down into the 'start of scheduler' patch. [Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +3 -0
index 50c45f3..cf0316d 100644
@@ -1483,6 +1483,9 @@ static void i915_gem_request_retire(struct drm_i915_gem_request *request)
 		fence_signal_locked(&request->fence);
 	}
 
+	if (request->scheduler_qe)
+		i915_scheduler_clean_node(request->scheduler_qe);
+
 	i915_gem_request_unreference(request);
 }
 
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#18 lore raw

PATCH v6 16/34 drm/i915: Added scheduler support to __wait_request() calls

In reply to [email protected]

From: John Harrison <[email protected]>

The scheduler can cause batch buffers, and hence requests, to be submitted to the ring out of order and asynchronously to their submission to the driver. Thus at the point of waiting for the completion of a given request, it is not even guaranteed that the request has actually been sent to the hardware yet. Even it is has been sent, it is possible that it could be pre-empted and thus 'unsent'.

This means that it is necessary to be able to submit requests to the hardware during the wait call itself. Unfortunately, while some callers of __wait_request() release the mutex lock first, others do not (and apparently can not). Hence there is the ability to deadlock as the wait stalls for submission but the asynchronous submission is stalled for the mutex lock.

This change hooks the scheduler in to the __wait_request() code to ensure correct behaviour. That is, flush the target batch buffer through to the hardware and do not deadlock waiting for something that cannot currently be submitted. Instead, the wait call must return EAGAIN at least as far back as necessary to release the mutex lock and allow the scheduler's asynchronous processing to get in and handle the pre-emption operation and eventually (re-)submit the work.

v3: Removed the explicit scheduler flush from i915_wait_request(). This is no longer necessary and was causing unintended changes to the scheduler priority level which broke a validation team test.

v4: Corrected the format of some comments to keep the style checker happy.

v5: Added function description. [Joonas Lahtinen]

v6: Updated to newer nightly (lots of ring -> engine renaming).

Updated to use 'to_i915()' instead of dev_private. Changed extra boolean wait_request() parameter to a flags word and consumed the original boolean parameter too. Also, replaced the i915_scheduler_is_request_tracked() function with i915_scheduler_is_mutex_required() as the need for the former has gone away and it was really being used to ask the latter question in a convoluted manner. [review feedback from Joonas Lahtinen]

For: VIZ-1587

Cc Joonas Lahtinen <[email protected]>
---
 drivers/gpu/drm/i915/i915_drv.h         |  7 ++++-
 drivers/gpu/drm/i915/i915_gem.c         | 49 ++++++++++++++++++++++++++-------
 drivers/gpu/drm/i915/i915_scheduler.c   | 26 +++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h   |  1 +
 drivers/gpu/drm/i915/intel_display.c    |  5 ++--
 drivers/gpu/drm/i915/intel_ringbuffer.c |  8 ++++--
 6 files changed, 81 insertions(+), 15 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +6 -1
index ed9d829..6fa2541 100644
@@ -3159,9 +3159,14 @@ void __i915_add_request(struct drm_i915_gem_request *req,
 	__i915_add_request(req, NULL, false)
 int __i915_wait_request(struct drm_i915_gem_request *req,
 			unsigned reset_counter,
-			bool interruptible,
+			uint32_t flags,
 			s64 *timeout,
 			struct intel_rps_client *rps);
+
+/* flags used by users of __i915_wait_request */
+#define I915_WAIT_REQUEST_INTERRUPTIBLE  (1 << 0)
+#define I915_WAIT_REQUEST_LOCKED         (1 << 1)
+
 int __must_check i915_wait_request(struct drm_i915_gem_request *req);
 int i915_gem_fault(struct vm_area_struct *vma, struct vm_fault *vmf);
 int __must_check
drivers/gpu/drm/i915/i915_gem.c +39 -10

119 lines not on this page · raw message

drivers/gpu/drm/i915/i915_scheduler.c +26 -0
index cdb86da..cf8b50a 100644
@@ -910,6 +910,32 @@ void i915_scheduler_work_handler(struct work_struct *work)
 }
 
 /**
+ * i915_scheduler_is_mutex_required - query if it is safe to hold the mutex
+ * lock while waiting for the given request.
+ * @req: request to be queried
+ *
+ * Looks up the given request in the scheduler's internal queue and reports
+ * on whether the scheduler will need to acquire the driver's mutex lock in
+ * order for the that request to complete.
+ */
+bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req)
+{
+	struct drm_i915_private *dev_priv = to_i915(req->engine->dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	if (!scheduler)
+		return false;
+
+	if (req->scheduler_qe == NULL)
+		return false;
+
+	if (I915_SQS_IS_QUEUED(req->scheduler_qe))
+		return true;
+
+	return false;
+}
+
+/**
  * i915_scheduler_closefile - notify the scheduler that a DRM file handle
  * has been closed.
  * @dev: DRM device
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
index b8d4a343..38e860d 100644
@@ -111,5 +111,6 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void i915_scheduler_wakeup(struct drm_device *dev);
 void i915_scheduler_work_handler(struct work_struct *work);
+bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req);
 
 #endif  /* _I915_SCHEDULER_H_ */
drivers/gpu/drm/i915/intel_display.c +3 -2
index 9407934..f1d730a 100644
@@ -11371,7 +11371,7 @@ static void intel_mmio_flip_work_func(struct work_struct *work)
 	if (mmio_flip->req) {
 		WARN_ON(__i915_wait_request(mmio_flip->req,
 					    mmio_flip->crtc->reset_counter,
-					    false, NULL,
+					    0, NULL,
 					    &mmio_flip->i915->rps.mmioflips));
 		i915_gem_request_unreference(mmio_flip->req);
 	}
@@ -13439,7 +13439,8 @@ static int intel_atomic_prepare_commit(struct drm_device *dev,
 				continue;
 
 			ret = __i915_wait_request(intel_plane_state->wait_req,
-						  reset_counter, true,
+						  reset_counter,
+						  I915_WAIT_REQUEST_INTERRUPTIBLE,
 						  NULL, NULL);
 
 			/* Swallow -EIO errors to allow updates during hw lockup. */
drivers/gpu/drm/i915/intel_ringbuffer.c +6 -2
index e3f2237..6021655 100644
@@ -2362,6 +2362,7 @@ static void __wrap_ring_buffer(struct intel_ringbuffer *ringbuf)
 int intel_engine_idle(struct intel_engine_cs *engine)
 {
 	struct drm_i915_gem_request *req;
+	uint32_t flags;
 
 	/* Wait upon the last request to be completed */
 	if (list_empty(&engine->request_list))
@@ -2371,11 +2372,14 @@ int intel_engine_idle(struct intel_engine_cs *engine)
 			 struct drm_i915_gem_request,
 			 list);
 
+	flags = I915_WAIT_REQUEST_LOCKED;
+	if (to_i915(engine->dev)->mm.interruptible)
+		flags |= I915_WAIT_REQUEST_INTERRUPTIBLE;
+
 	/* Make sure we do not trigger any retires */
 	return __i915_wait_request(req,
 				   atomic_read(&to_i915(engine->dev)->gpu_error.reset_counter),
-				   to_i915(engine->dev)->mm.interruptible,
-				   NULL, NULL);
+				   flags, NULL, NULL);
 }
 
 int intel_ring_alloc_request_extras(struct drm_i915_gem_request *request)
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#19 lore raw

PATCH v6 17/34 drm/i915: Added scheduler support to page fault handler

In reply to [email protected]

From: John Harrison <[email protected]>

GPU page faults can now require scheduler operation in order to complete. For example, in order to free up sufficient memory to handle the fault the handler must wait for a batch buffer to complete that has not even been sent to the hardware yet. Thus EAGAIN no longer means a GPU hang, it can occur under normal operation.

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem.c | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +7 -2
index 8f7bf6f..5fb4472 100644
@@ -1998,10 +1998,15 @@ out:
 		}
 	case -EAGAIN:
 		/*
-		 * EAGAIN means the gpu is hung and we'll wait for the error
-		 * handler to reset everything when re-faulting in
+		 * EAGAIN can mean the gpu is hung and we'll have to wait for
+		 * the error handler to reset everything when re-faulting in
 		 * i915_mutex_lock_interruptible.
+		 *
+		 * It can also indicate various other nonfatal errors for which
+		 * the best response is to give other threads a chance to run,
+		 * and then retry the failing operation in its entirety.
 		 */
+		/*FALLTHRU*/
 	case 0:
 	case -ERESTARTSYS:
 	case -EINTR:
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#20 lore raw

PATCH v6 18/34 drm/i915: Added scheduler flush calls to ring throttle and idle functions

In reply to [email protected]

From: John Harrison <[email protected]>

When requesting that all GPU work is completed, it is now necessary to get the scheduler involved in order to flush out work that queued and not yet submitted.

v2: Updated to add support for flushing the scheduler queue by time stamp rather than just doing a blanket flush.

v3: Moved submit_max_priority() to this patch from an earlier patch is it is no longer required in the other.

v4: Corrected the format of a comment to keep the style checker happy. Downgraded a BUG_ON to a WARN_ON as the latter is preferred.

v5: Shuffled functions around to remove forward prototypes, removed similarly offensive white space and added documentation. Re-worked the mutex locking around the submit function. [Joonas Lahtinen]

Used lighter weight spinlocks.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' helper macro. Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. [review feedback from Joonas Lahtinen]

For: VIZ-1587

Cc Joonas Lahtinen <[email protected]>
---
 drivers/gpu/drm/i915/i915_gem.c       |  22 +++++
 drivers/gpu/drm/i915/i915_scheduler.c | 178 ++++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |   3 +
 3 files changed, 203 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +22 -0
index 5fb4472..3ee0283 100644
@@ -3874,6 +3874,10 @@ int i915_gpu_idle(struct drm_device *dev)
 
 	/* Flush everything onto the inactive list. */
 	for_each_engine(engine, dev_priv) {
+		ret = i915_scheduler_flush(engine, true);
+		if (ret < 0)
+			return ret;
+
 		if (!i915.enable_execlists) {
 			struct drm_i915_gem_request *req;
 
@@ -4612,6 +4616,7 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	struct drm_i915_gem_request *request, *target = NULL;
 	unsigned reset_counter;
 	int ret;
+	struct intel_engine_cs *engine;
 
 	ret = i915_gem_wait_for_error(&dev_priv->gpu_error);
 	if (ret)
@@ -4621,6 +4626,23 @@ i915_gem_ring_throttle(struct drm_device *dev, struct drm_file *file)
 	if (ret)
 		return ret;
 
+	for_each_engine(engine, dev_priv) {
+		/*
+		 * Flush out scheduler entries that are getting 'stale'. Note
+		 * that the following recent_enough test will only check
+		 * against the time at which the request was submitted to the
+		 * hardware (i.e. when it left the scheduler) not the time it
+		 * was submitted to the driver.
+		 *
+		 * Also, there is not much point worring about busy return
+		 * codes from the scheduler flush call. Even if more work
+		 * cannot be submitted right now for whatever reason, we
+		 * still want to throttle against stale work that has already
+		 * been submitted.
+		 */
+		i915_scheduler_flush_stamp(engine, recent_enough, false);
+	}
+
 	spin_lock(&file_priv->mm.lock);
 	list_for_each_entry(request, &file_priv->mm.request_list, client_list) {
 		if (time_after_eq(request->emitted_jiffies, recent_enough))
drivers/gpu/drm/i915/i915_scheduler.c +178 -0

196 lines not on this page · raw message

drivers/gpu/drm/i915/i915_scheduler.h +3 -0
index 38e860d..92dd5df 100644
@@ -111,6 +111,9 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void i915_scheduler_wakeup(struct drm_device *dev);
 void i915_scheduler_work_handler(struct work_struct *work);
+int i915_scheduler_flush(struct intel_engine_cs *engine, bool is_locked);
+int i915_scheduler_flush_stamp(struct intel_engine_cs *engine,
+			       unsigned long stamp, bool is_locked);
 bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req);
 
 #endif  /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#21 lore raw

PATCH v6 19/34 drm/i915: Add scheduler hook to GPU reset

In reply to [email protected]

From: John Harrison <[email protected]>

When the watchdog resets the GPU, all interrupts get disabled despite the reference count remaining. As the scheduler probably had interrupts enabled during the reset (it would have been waiting for the bad batch to complete), it must be poked to tell it that the interrupt has been disabled.

v5: New patch in series.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem.c       |  2 ++
 drivers/gpu/drm/i915/i915_scheduler.c | 11 +++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 3 files changed, 14 insertions(+)
drivers/gpu/drm/i915/i915_gem.c +2 -0
index 3ee0283..ebc7f0a 100644
@@ -3327,6 +3327,8 @@ static void i915_gem_reset_engine_cleanup(struct drm_i915_private *dev_priv,
 	}
 
 	intel_ring_init_seqno(engine, engine->last_submitted_seqno);
+
+	i915_scheduler_reset_cleanup(engine);
 }
 
 void i915_gem_reset(struct drm_device *dev)
drivers/gpu/drm/i915/i915_scheduler.c +11 -0
index 580b9d2..9c3150a 100644
@@ -821,6 +821,17 @@ void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node)
 	}
 }
 
+void i915_scheduler_reset_cleanup(struct intel_engine_cs *engine)
+{
+	struct drm_i915_private *dev_priv = to_i915(engine->dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	if (scheduler->flags[engine->id] & I915_SF_INTERRUPTS_ENABLED) {
+		engine->irq_put(engine);
+		scheduler->flags[engine->id] &= ~I915_SF_INTERRUPTS_ENABLED;
+	}
+}
+
 static bool i915_scheduler_remove(struct i915_scheduler *scheduler,
 				  struct intel_engine_cs *engine,
 				  struct list_head *remove)
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
index 92dd5df..4e7c0a7 100644
@@ -106,6 +106,7 @@ bool i915_scheduler_is_enabled(struct drm_device *dev);
 int i915_scheduler_init(struct drm_device *dev);
 void i915_scheduler_destroy(struct drm_i915_private *dev_priv);
 void i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file);
+void i915_scheduler_reset_cleanup(struct intel_engine_cs *engine);
 void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node);
 int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool i915_scheduler_notify_request(struct drm_i915_gem_request *req);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#22 lore raw

PATCH v6 20/34 drm/i915: Added a module parameter to allow the scheduler to be disabled

In reply to [email protected]

From: John Harrison <[email protected]>

It can be useful to be able to disable the GPU scheduler via a module parameter for debugging purposes.

v5: Converted from a multi-feature 'overrides' mask to a single 'enable' boolean. Further features (e.g. pre-emption) will now be separate 'enable' booleans added later. [Chris Wilson]

v6: Moved scheduler parameter declaration to correct place in i915_params struct. [review feedback from Matt Roper]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_params.c    | 4 ++++
 drivers/gpu/drm/i915/i915_params.h    | 1 +
 drivers/gpu/drm/i915/i915_scheduler.c | 5 ++++-
 3 files changed, 9 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_params.c +4 -0
index 1285063..45288a2 100644
@@ -58,6 +58,7 @@ struct i915_params i915 __read_mostly = {
 	.guc_log_level = -1,
 	.enable_dp_mst = true,
 	.inject_load_failure = 0,
+	.enable_scheduler = 0,
 };
 
 module_param_named(modeset, i915.modeset, int, 0400);
@@ -210,3 +211,6 @@ MODULE_PARM_DESC(enable_dp_mst,
 module_param_named_unsafe(inject_load_failure, i915.inject_load_failure, uint, 0400);
 MODULE_PARM_DESC(inject_load_failure,
 	"Force an error after a number of failure check points (0:disabled (default), N:force failure at the Nth failure check point)");
+
+module_param_named_unsafe(enable_scheduler, i915.enable_scheduler, int, 0600);
+MODULE_PARM_DESC(enable_scheduler, "Enable scheduler (0 = disable [default], 1 = enable)");
drivers/gpu/drm/i915/i915_params.h +1 -0
index 02bc278..44b08b3 100644
@@ -50,6 +50,7 @@ struct i915_params {
 	int mmio_debug;
 	int edp_vswing;
 	unsigned int inject_load_failure;
+	int enable_scheduler;
 	/* leave bools at the end to not create holes */
 	bool enable_hangcheck;
 	bool fastboot;
drivers/gpu/drm/i915/i915_scheduler.c +4 -1
index 9c3150a..13084fb 100644
@@ -42,6 +42,9 @@ bool i915_scheduler_is_enabled(struct drm_device *dev)
 {
 	struct drm_i915_private *dev_priv = to_i915(dev);
 
+	if (!i915.enable_scheduler)
+		return false;
+
 	return dev_priv->scheduler != NULL;
 }
 
@@ -593,7 +596,7 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 	int incomplete;
 
 	/* Bypass the scheduler and send the buffer immediately? */
-	if (1/*!i915.enable_scheduler*/)
+	if (!i915.enable_scheduler)
 		return i915_scheduler_queue_execbuffer_bypass(qe);
 
 	node = kmalloc(sizeof(*node), GFP_KERNEL);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#23 lore raw

PATCH v6 21/34 drm/i915: Support for 'unflushed' ring idle

In reply to [email protected]

From: John Harrison <[email protected]>

When the seqno wraps around zero, the entire GPU is forced to be idle for some reason (possibly only to work around issues with hardware semaphores but no-one seems too sure!). This causes a problem if the force idle occurs at an inopportune moment such as in the middle of submitting a batch buffer. Specifically, it would lead to recursive submits - submitting work requires a new seqno, the new seqno requires idling the ring, idling the ring requires submitting work, submitting work requires a new seqno...

This change adds a 'flush' parameter to the idle function call which specifies whether the scheduler queues should be flushed out. I.e. is the call intended to just idle the ring as it is right now (no flush) or is it intended to force all outstanding work out of the system (with flush).

In the seqno wrap case, pending work is not an issue because the next operation will be to submit it. However, in other cases, the intention is to make sure everything that could be done has been done.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added kerneldoc for intel_engine_idle().

Wrapped boolean 'flush' parameter with an _flush() macro. [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem.c         |  2 +-
 drivers/gpu/drm/i915/intel_lrc.c        |  2 +-
 drivers/gpu/drm/i915/intel_ringbuffer.c | 31 +++++++++++++++++++++++++++++--
 drivers/gpu/drm/i915/intel_ringbuffer.h |  4 +++-
 4 files changed, 34 insertions(+), 5 deletions(-)
drivers/gpu/drm/i915/i915_gem.c +1 -1
index ebc7f0a..1737f13 100644
@@ -3896,7 +3896,7 @@ int i915_gpu_idle(struct drm_device *dev)
 			i915_add_request_no_flush(req);
 		}
 
-		ret = intel_engine_idle(engine);
+		ret = intel_engine_idle_flush(engine);
 		if (ret)
 			return ret;
 	}
drivers/gpu/drm/i915/intel_lrc.c +1 -1
index 0a4ef61..d67b08b 100644
@@ -1091,7 +1091,7 @@ void intel_logical_ring_stop(struct intel_engine_cs *engine)
 	if (!intel_engine_initialized(engine))
 		return;
 
-	ret = intel_engine_idle(engine);
+	ret = intel_engine_idle_flush(engine);
 	if (ret && !i915_reset_in_progress(&to_i915(engine->dev)->gpu_error))
 		DRM_ERROR("failed to quiesce %s whilst cleaning up: %d\n",
 			  engine->name, ret);
drivers/gpu/drm/i915/intel_ringbuffer.c +29 -2
index 6021655..f5bcd24 100644
@@ -2359,10 +2359,37 @@ static void __wrap_ring_buffer(struct intel_ringbuffer *ringbuf)
 	intel_ring_update_space(ringbuf);
 }
 
-int intel_engine_idle(struct intel_engine_cs *engine)
+/**
+ * __intel_engine_idle - Force the engine to be idle.
+ * @engine: Engine to be idled
+ * @flush: Should queued scheduler work also be flushed
+ * Waits for all outstanding requests that have been sent to the given engine
+ * to complete. Can optionally also force all unsent requests that are queued
+ * in the scheduler to be sent first.
+ * Returns zero on success otherwise a negative error code.
+ *
+ * NB: Flushing can lead to recursion if called at the wrong time. E.g. flush
+ * causes the scheduler to submit requests to the hardware, submitting
+ * requests requires allocating a new seqno, when the seqno wraps around it
+ * idles the engine, idling with flush causes the scheduler to submit requests...
+ */
+int __intel_engine_idle(struct intel_engine_cs *engine, bool flush)
 {
 	struct drm_i915_gem_request *req;
 	uint32_t flags;
+	int ret;
+
+	/*
+	 * NB: Must not flush the scheduler if this idle request is from
+	 * within an execbuff submission (i.e. due to 'get_seqno' calling
+	 * 'wrap_seqno' calling 'idle'). As that would lead to recursive
+	 * flushes!
+	 */
+	if (flush) {
+		ret = i915_scheduler_flush(engine, true);
+		if (ret)
+			return ret;
+	}
 
 	/* Wait upon the last request to be completed */
 	if (list_empty(&engine->request_list))
@@ -3202,7 +3229,7 @@ intel_stop_engine(struct intel_engine_cs *engine)
 	if (!intel_engine_initialized(engine))
 		return;
 
-	ret = intel_engine_idle(engine);
+	ret = intel_engine_idle_flush(engine);
 	if (ret && !i915_reset_in_progress(&to_i915(engine->dev)->gpu_error))
 		DRM_ERROR("failed to quiesce %s whilst cleaning up: %d\n",
 			  engine->name, ret);
drivers/gpu/drm/i915/intel_ringbuffer.h +3 -1
index c75c5e1..2e7daef 100644
@@ -474,7 +474,9 @@ void intel_ring_update_space(struct intel_ringbuffer *ringbuf);
 int intel_ring_space(struct intel_ringbuffer *ringbuf);
 bool intel_engine_stopped(struct intel_engine_cs *engine);
 
-int __must_check intel_engine_idle(struct intel_engine_cs *engine);
+#define intel_engine_idle(engine)           __intel_engine_idle((engine), false)
+#define intel_engine_idle_flush(engine)     __intel_engine_idle((engine), true)
+int __must_check __intel_engine_idle(struct intel_engine_cs *engine, bool flush);
 void intel_ring_init_seqno(struct intel_engine_cs *engine, u32 seqno);
 int intel_ring_flush_all_caches(struct drm_i915_gem_request *req);
 int intel_ring_invalidate_all_caches(struct drm_i915_gem_request *req);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#24 lore raw

PATCH v6 22/34 drm/i915: Defer seqno allocation until actual hardware submission time

In reply to [email protected]

From: John Harrison <[email protected]>

The seqno value is now only used for the final test for completion of a request. It is no longer used to track the request through the software stack. Thus it is no longer necessary to allocate the seqno immediately with the request. Instead, it can be done lazily and left until the request is actually sent to the hardware. This is particular advantageous with a GPU scheduler as the requests can then be re-ordered between their creation and their hardware submission without having out of order seqnos.

v2: i915_add_request() can't fail!

Combine with 'drm/i915: Assign seqno at start of exec_final()' Various bits of code during the execbuf code path need a seqno value to be assigned to the request. This change makes this assignment explicit at the start of submission_final() rather than relying on an auto-generated seqno to have happened already. This is in preparation for a future patch which changes seqno values to be assigned lazily (during add_request).

v3: Updated to use locally cached request pointer.

v4: Changed some white space and comment formatting to keep the style checker happy.

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_drv.h            |  1 +
 drivers/gpu/drm/i915/i915_gem.c            | 23 ++++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 14 ++++++++++++++
 drivers/gpu/drm/i915/intel_lrc.c           | 14 ++++++++++++++
 4 files changed, 51 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_drv.h +1 -0
index 6fa2541..e9aaacc 100644
@@ -2302,6 +2302,7 @@ struct drm_i915_gem_request {
 	  * has finished processing this request.
 	  */
 	u32 seqno;
+	u32 reserved_seqno;
 
 	/* Unique identifier which can be used for trace points & debug */
 	uint32_t uniq;
drivers/gpu/drm/i915/i915_gem.c +22 -1
index 1737f13..467d7da 100644
@@ -2659,6 +2659,11 @@ i915_gem_get_seqno(struct drm_device *dev, u32 *seqno)
 
 	/* reserve 0 for non-seqno */
 	if (dev_priv->next_seqno == 0) {
+		/*
+		 * Why is the full re-initialisation required? Is it only for
+		 * hardware semaphores? If so, could skip it in the case where
+		 * semaphores are disabled?
+		 */
 		int ret = i915_gem_init_seqno(dev, 0);
 		if (ret)
 			return ret;
@@ -2716,6 +2721,12 @@ void __i915_add_request(struct drm_i915_gem_request *request,
 		WARN(ret, "*_ring_flush_all_caches failed: %d!\n", ret);
 	}
 
+	/* Make the request's seqno 'live': */
+	if (!request->seqno) {
+		request->seqno = request->reserved_seqno;
+		WARN_ON(request->seqno != dev_priv->last_seqno);
+	}
+
 	trace_i915_gem_request_add(request);
 
 	request->head = request_start;
@@ -2978,6 +2989,9 @@ void i915_gem_request_notify(struct intel_engine_cs *engine, bool fence_locked)
 
 	list_for_each_entry_safe(req, req_next, &engine->fence_signal_list, signal_link) {
 		if (!req->cancelled) {
+			/* How can this happen? */
+			WARN_ON(req->seqno == 0);
+
 			if (!i915_seqno_passed(seqno, req->seqno))
 				break;
 		}
@@ -3129,7 +3143,14 @@ __i915_gem_request_alloc(struct intel_engine_cs *engine,
 	if (req == NULL)
 		return -ENOMEM;
 
-	ret = i915_gem_get_seqno(engine->dev, &req->seqno);
+	/*
+	 * Assign an identifier to track this request through the hardware
+	 * but don't make it live yet. It could change in the future if this
+	 * request gets overtaken. However, it still needs to be allocated
+	 * in advance because the point of submission must not fail and seqno
+	 * allocation can fail.
+	 */
+	ret = i915_gem_get_seqno(engine->dev, &req->reserved_seqno);
 	if (ret)
 		goto err;
 
drivers/gpu/drm/i915/i915_gem_execbuffer.c +14 -0
index e105aae..5450219 100644
@@ -1296,6 +1296,20 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Make sure the request's seqno is the latest and greatest: */
+	if (req->reserved_seqno != dev_priv->last_seqno) {
+		ret = i915_gem_get_seqno(engine->dev, &req->reserved_seqno);
+		if (ret)
+			return ret;
+	}
+	/*
+	 * And make it live because some of the execbuff submission code
+	 * requires the seqno to be available up front.
+	 */
+	WARN_ON(req->seqno);
+	req->seqno = req->reserved_seqno;
+	WARN_ON(req->seqno != dev_priv->last_seqno);
+
 	ret = intel_ring_reserve_space(req);
 	if (ret)
 		goto error;
drivers/gpu/drm/i915/intel_lrc.c +14 -0
index d67b08b..b01571e 100644
@@ -1010,6 +1010,20 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Make sure the request's seqno is the latest and greatest: */
+	if (req->reserved_seqno != dev_priv->last_seqno) {
+		ret = i915_gem_get_seqno(engine->dev, &req->reserved_seqno);
+		if (ret)
+			return ret;
+	}
+	/*
+	 * And make it live because some of the execbuff submission code
+	 * requires the seqno to be available up front.
+	 */
+	WARN_ON(req->seqno);
+	req->seqno = req->reserved_seqno;
+	WARN_ON(req->seqno != dev_priv->last_seqno);
+
 	ret = intel_logical_ring_reserve_space(req);
 	if (ret)
 		goto err;
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#25 lore raw

PATCH v6 23/34 drm/i915: Added trace points to scheduler

In reply to [email protected]

From: John Harrison <[email protected]>

Added trace points to the scheduler to track all the various events, node state transitions and other interesting things that occur.

v2: Updated for new request completion tracking implementation.

v3: Updated for changes to node kill code.

v4: Wrapped some long lines to keep the style checker happy.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Dropped 'min_seqno' value from 'i915_scheduler_remove' tracepoint as it has also been removed from the code.

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |   2 +
 drivers/gpu/drm/i915/i915_scheduler.c      |  24 +++-
 drivers/gpu/drm/i915/i915_trace.h          | 193 +++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/intel_lrc.c           |   2 +
 4 files changed, 219 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +2 -0
index 5450219..a08638a 100644
@@ -1273,6 +1273,8 @@ i915_gem_ringbuffer_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
+	trace_i915_gem_ring_queue(engine, params);
+
 	qe = container_of(params, typeof(*qe), params);
 	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
drivers/gpu/drm/i915/i915_scheduler.c +22 -2
index 13084fb..a3a7a82 100644
@@ -124,6 +124,8 @@ static void i915_scheduler_node_requeue(struct i915_scheduler *scheduler,
 	node->status = I915_SQS_QUEUED;
 	scheduler->counts[node->params.engine->id].flying--;
 	scheduler->counts[node->params.engine->id].queued++;
+	trace_i915_scheduler_unfly(node->params.engine, node);
+	trace_i915_scheduler_node_state_change(node->params.engine, node);
 }
 
 /*
@@ -137,12 +139,14 @@ static void i915_scheduler_node_kill(struct i915_scheduler *scheduler,
 
 	WARN_ON(I915_SQS_IS_COMPLETE(node));
 
-	if (I915_SQS_IS_FLYING(node))
+	if (I915_SQS_IS_FLYING(node)) {
 		scheduler->counts[node->params.engine->id].flying--;
-	else
+		trace_i915_scheduler_unfly(node->params.engine, node);
+	} else
 		scheduler->counts[node->params.engine->id].queued--;
 
 	node->status = I915_SQS_DEAD;
+	trace_i915_scheduler_node_state_change(node->params.engine, node);
 }
 
 /* Mark a node as in flight on the hardware. */
@@ -166,6 +170,8 @@ static void i915_scheduler_node_fly(struct i915_scheduler_queue_entry *node)
 	node->status = I915_SQS_FLYING;
 
 	scheduler->counts[engine->id].flying++;
+	trace_i915_scheduler_fly(engine, node);
+	trace_i915_scheduler_node_state_change(engine, node);
 
 	if (!(scheduler->flags[engine->id] & I915_SF_INTERRUPTS_ENABLED)) {
 		bool success = true;
@@ -317,6 +323,7 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *engine,
 		best->status = I915_SQS_POPPED;
 
 		scheduler->counts[engine->id].queued--;
+		trace_i915_scheduler_node_state_change(engine, best);
 
 		ret = 0;
 	} else {
@@ -335,6 +342,8 @@ static int i915_scheduler_pop_from_queue_locked(struct intel_engine_cs *engine,
 		}
 	}
 
+	trace_i915_scheduler_pop_from_queue(engine, best);
+
 	*pop_node = best;
 	return ret;
 }
@@ -542,6 +551,8 @@ static int i915_scheduler_queue_execbuffer_bypass(struct i915_scheduler_queue_en
 	struct i915_scheduler *scheduler = dev_priv->scheduler;
 	int ret;
 
+	trace_i915_scheduler_queue(qe->params.engine, qe);
+
 	intel_ring_reserved_space_cancel(qe->params.request->ringbuf);
 
 	scheduler->flags[qe->params.engine->id] |= I915_SF_SUBMITTING;
@@ -668,6 +679,8 @@ int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe)
 						 scheduler->min_flying;
 
 	scheduler->counts[engine->id].queued++;
+	trace_i915_scheduler_queue(engine, node);
+	trace_i915_scheduler_node_state_change(engine, node);
 
 	spin_unlock_irq(&scheduler->lock);
 
@@ -698,6 +711,8 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 	struct i915_scheduler_queue_entry *node = req->scheduler_qe;
 	unsigned long flags;
 
+	trace_i915_scheduler_landing(req);
+
 	if (!node)
 		return false;
 
@@ -712,6 +727,7 @@ bool i915_scheduler_notify_request(struct drm_i915_gem_request *req)
 		node->status = I915_SQS_COMPLETE;
 
 	scheduler->counts[req->engine->id].flying--;
+	trace_i915_scheduler_node_state_change(req->engine, node);
 
 	spin_unlock_irqrestore(&scheduler->lock, flags);
 
@@ -872,6 +888,8 @@ static bool i915_scheduler_remove(struct i915_scheduler *scheduler,
 	do_submit = (scheduler->counts[engine->id].queued > 0) &&
 		    (scheduler->counts[engine->id].flying < scheduler->min_flying);
 
+	trace_i915_scheduler_remove(engine, do_submit);
+
 	spin_unlock_irq(&scheduler->lock);
 
 	return do_submit;
@@ -907,6 +925,8 @@ static void i915_scheduler_process_work(struct intel_engine_cs *engine)
 		node = list_first_entry(&remove, typeof(*node), link);
 		list_del(&node->link);
 
+		trace_i915_scheduler_destroy(engine, node);
+
 		/* Free up all the DRM references */
 		i915_scheduler_clean_node(node);
 
drivers/gpu/drm/i915/i915_trace.h +193 -0

205 lines not on this page · raw message

drivers/gpu/drm/i915/intel_lrc.c +2 -0
index b01571e..252fc24 100644
@@ -986,6 +986,8 @@ int intel_execlists_submission(struct i915_execbuffer_params *params,
 
 	i915_gem_execbuffer_move_to_active(vmas, params->request);
 
+	trace_i915_gem_ring_queue(engine, params);
+
 	qe = container_of(params, typeof(*qe), params);
 	ret = i915_scheduler_queue_execbuffer(qe);
 	if (ret)
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#26 lore raw

PATCH v6 24/34 drm/i915: Added scheduler queue throttling by DRM file handle

In reply to [email protected]

From: John Harrison <[email protected]>

The scheduler decouples the submission of batch buffers to the driver from their subsequent submission to the hardware. This means that an application which is continuously submitting buffers as fast as it can could potentialy flood the driver. To prevent this, the driver now tracks how many buffers are in progress (queued in software or executing in hardware) and limits this to a given (tunable) number. If this number is exceeded then the queue to the driver will return EAGAIN and thus prevent the scheduler's queue becoming arbitrarily large.

v3: Added a missing decrement of the file queue counter.

v4: Updated a comment.

v5: Updated due to changes to earlier patches in series - removing forward declarations and white space. Also added some documentation. [Joonas Lahtinen]

v6: Updated to newer nightly (lots of ring -> engine renaming).

Replace the simple 'return to userland when full' scheme with a 'sleep on request' scheme. The former could lead to the busy polling and wasting lots of CPU time as user land continuously retried the execbuf IOCTL in a tight loop. Now the driver will sleep (without holding the mutex lock) on the oldest request outstanding for that file and then automatically retry. This is closer to the pre-scheduler behaviour of stalling on a full ring buffer.

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_drv.h            |   2 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |   8 ++
 drivers/gpu/drm/i915/i915_scheduler.c      | 118 +++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |   2 +
 4 files changed, 130 insertions(+)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index e9aaacc..25b8fd6 100644
@@ -376,6 +376,8 @@ struct drm_i915_file_private {
 	} rps;
 
 	unsigned int bsd_ring;
+
+	u32 scheduler_queue_length;
 };
 
 /* Used by dp and fdi links */
drivers/gpu/drm/i915/i915_gem_execbuffer.c +8 -0
index a08638a..1f8486e 100644
@@ -1818,6 +1818,10 @@ i915_gem_execbuffer(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	/* Throttle batch requests per device file */
+	if (i915_scheduler_file_queue_wait(file))
+		return -EAGAIN;
+
 	/* Copy in the exec list from userland */
 	exec_list = drm_malloc_ab(sizeof(*exec_list), args->buffer_count);
 	exec2_list = drm_malloc_ab(sizeof(*exec2_list), args->buffer_count);
@@ -1908,6 +1912,10 @@ i915_gem_execbuffer2(struct drm_device *dev, void *data,
 		return -EINVAL;
 	}
 
+	/* Throttle batch requests per device file */
+	if (i915_scheduler_file_queue_wait(file))
+		return -EAGAIN;
+
 	exec2_list = drm_malloc_gfp(args->buffer_count,
 				    sizeof(*exec2_list),
 				    GFP_TEMPORARY);
drivers/gpu/drm/i915/i915_scheduler.c +118 -0

154 lines not on this page · raw message

drivers/gpu/drm/i915/i915_scheduler.h +2 -0
index 4e7c0a7..5c33c83 100644
@@ -94,6 +94,7 @@ struct i915_scheduler {
 	int32_t priority_level_bump;
 	int32_t priority_level_preempt;
 	uint32_t min_flying;
+	uint32_t file_queue_max;
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -116,5 +117,6 @@ int i915_scheduler_flush(struct intel_engine_cs *engine, bool is_locked);
 int i915_scheduler_flush_stamp(struct intel_engine_cs *engine,
 			       unsigned long stamp, bool is_locked);
 bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req);
+bool i915_scheduler_file_queue_wait(struct drm_file *file);
 
 #endif  /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#27 lore raw

PATCH v6 24/34 drm/i915: Added scheduler queue throttling by DRM file handle

From: John Harrison <[email protected]>

The scheduler decouples the submission of batch buffers to the driver from their subsequent submission to the hardware. This means that an application which is continuously submitting buffers as fast as it can could potentialy flood the driver. To prevent this, the driver now tracks how many buffers are in progress (queued in software or executing in hardware) and limits this to a given (tunable) number. If this number is exceeded then the queue to the driver will return EAGAIN and thus prevent the scheduler's queue becoming arbitrarily large.

v3: Added a missing decrement of the file queue counter.

v4: Updated a comment.

v5: Updated due to changes to earlier patches in series - removing forward declarations and white space. Also added some documentation. [Joonas Lahtinen]

v6: Updated to newer nightly (lots of ring -> engine renaming).

Replace the simple 'return to userland when full' scheme with a 'sleep on request' scheme. The former could lead to the busy polling and wasting lots of CPU time as user land continuously retried the execbuf IOCTL in a tight loop. Now the driver will sleep (without holding the mutex lock) on the oldest request outstanding for that file and then automatically retry. This is closer to the pre-scheduler behaviour of stalling on a full ring buffer.

v6.1: Moved throttle point to later common location. Required for a subsequent patch that needs the engine to have been determined already.

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_drv.h            |   2 +
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |   4 +
 drivers/gpu/drm/i915/i915_scheduler.c      | 118 +++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |   2 +
 4 files changed, 126 insertions(+)
drivers/gpu/drm/i915/i915_drv.h +2 -0
index e9aaacc..25b8fd6 100644
@@ -376,6 +376,8 @@ struct drm_i915_file_private {
 	} rps;
 
 	unsigned int bsd_ring;
+
+	u32 scheduler_queue_length;
 };
 
 /* Used by dp and fdi links */
drivers/gpu/drm/i915/i915_gem_execbuffer.c +4 -0
index a08638a..5a674ad 100644
@@ -1538,6 +1538,10 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 		dispatch_flags |= I915_DISPATCH_RS;
 	}
 
+	/* Throttle batch requests per device file */
+	if (i915_scheduler_file_queue_wait(file))
+		return -EAGAIN;
+
 	intel_runtime_pm_get(dev_priv);
 
 	ret = i915_mutex_lock_interruptible(dev);
drivers/gpu/drm/i915/i915_scheduler.c +118 -0

154 lines not on this page · raw message

drivers/gpu/drm/i915/i915_scheduler.h +2 -0
index 4e7c0a7..5c33c83 100644
@@ -94,6 +94,7 @@ struct i915_scheduler {
 	int32_t priority_level_bump;
 	int32_t priority_level_preempt;
 	uint32_t min_flying;
+	uint32_t file_queue_max;
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -116,5 +117,6 @@ int i915_scheduler_flush(struct intel_engine_cs *engine, bool is_locked);
 int i915_scheduler_flush_stamp(struct intel_engine_cs *engine,
 			       unsigned long stamp, bool is_locked);
 bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req);
+bool i915_scheduler_file_queue_wait(struct drm_file *file);
 
 #endif  /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#28 lore raw

PATCH v6 25/34 drm/i915: Added debugfs interface to scheduler tuning parameters

In reply to [email protected]

From: John Harrison <[email protected]>

There are various parameters within the scheduler which can be tuned to improve performance, reduce memory footprint, etc. This change adds support for altering these via debugfs.

v2: Updated for priorities now being signed values.

v5: Squashed priority bumping entries into this patch rather than a separate patch all of their own.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Updated to use 'to_i915()' instead of dev_private. [review feedback from Joonas Lahtinen]

Added an admin only check when setting the parameters to prevent rogue user code trying to break the system with strange settings. [review feedback from Jesse Barnes]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_debugfs.c | 187 ++++++++++++++++++++++++++++++++++++
 1 file changed, 187 insertions(+)
drivers/gpu/drm/i915/i915_debugfs.c +187 -0
index e89781a..980bb20 100644
@@ -39,6 +39,7 @@
 #include "intel_ringbuffer.h"
 #include <drm/i915_drm.h>
 #include "i915_drv.h"
+#include "i915_scheduler.h"
 
 enum {
 	ACTIVE_LIST,
@@ -1125,6 +1126,186 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_next_seqno_fops,
 			i915_next_seqno_get, i915_next_seqno_set,
 			"0x%llx\n");
 
+static int
+i915_scheduler_priority_min_get(void *data, u64 *val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_min;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_min_set(void *data, u64 val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	scheduler->priority_level_min = (int32_t) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_min_fops,
+			i915_scheduler_priority_min_get,
+			i915_scheduler_priority_min_set,
+			"%lld\n");
+
+static int
+i915_scheduler_priority_max_get(void *data, u64 *val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_max;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_max_set(void *data, u64 val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	scheduler->priority_level_max = (int32_t) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_max_fops,
+			i915_scheduler_priority_max_get,
+			i915_scheduler_priority_max_set,
+			"%lld\n");
+
+static int
+i915_scheduler_priority_bump_get(void *data, u64 *val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_bump;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_bump_set(void *data, u64 val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	scheduler->priority_level_bump = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_bump_fops,
+			i915_scheduler_priority_bump_get,
+			i915_scheduler_priority_bump_set,
+			"%lld\n");
+
+static int
+i915_scheduler_priority_preempt_get(void *data, u64 *val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->priority_level_preempt;
+	return 0;
+}
+
+static int
+i915_scheduler_priority_preempt_set(void *data, u64 val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	scheduler->priority_level_preempt = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_priority_preempt_fops,
+			i915_scheduler_priority_preempt_get,
+			i915_scheduler_priority_preempt_set,
+			"%lld\n");
+
+static int
+i915_scheduler_min_flying_get(void *data, u64 *val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->min_flying;
+	return 0;
+}
+
+static int
+i915_scheduler_min_flying_set(void *data, u64 val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	scheduler->min_flying = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_min_flying_fops,
+			i915_scheduler_min_flying_get,
+			i915_scheduler_min_flying_set,
+			"%llu\n");
+
+static int
+i915_scheduler_file_queue_max_get(void *data, u64 *val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	*val = (u64) scheduler->file_queue_max;
+	return 0;
+}
+
+static int
+i915_scheduler_file_queue_max_set(void *data, u64 val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	if (!capable(CAP_SYS_ADMIN))
+		return -EPERM;
+
+	scheduler->file_queue_max = (u32) val;
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_file_queue_max_fops,
+			i915_scheduler_file_queue_max_get,
+			i915_scheduler_file_queue_max_set,
+			"%llu\n");
+
 static int i915_frequency_info(struct seq_file *m, void *unused)
 {
 	struct drm_info_node *node = m->private;
@@ -5426,6 +5607,12 @@ static const struct i915_debugfs_files {
 	{"i915_gem_drop_caches", &i915_drop_caches_fops},
 	{"i915_error_state", &i915_error_state_fops},
 	{"i915_next_seqno", &i915_next_seqno_fops},
+	{"i915_scheduler_priority_min", &i915_scheduler_priority_min_fops},
+	{"i915_scheduler_priority_max", &i915_scheduler_priority_max_fops},
+	{"i915_scheduler_priority_bump", &i915_scheduler_priority_bump_fops},
+	{"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops},
+	{"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops},
+	{"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops},
 	{"i915_display_crc_ctl", &i915_display_crc_ctl_fops},
 	{"i915_pri_wm_latency", &i915_pri_wm_latency_fops},
 	{"i915_spr_wm_latency", &i915_spr_wm_latency_fops},
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#29 lore raw

PATCH v6 26/34 drm/i915: Add early exit to execbuff_final() if insufficient ring space

In reply to [email protected]

From: John Harrison <[email protected]>

One of the major purposes of the GPU scheduler is to avoid stalling the CPU when the GPU is busy and unable to accept more work. This change adds support to the ring submission code to allow a ring space check to be performed before attempting to submit a batch buffer to the hardware. If insufficient space is available then the scheduler can go away and come back later, letting the CPU get on with other work, rather than stalling and waiting for the hardware to catch up.

v3: Updated to use locally cached request pointer.

v4: Line wrapped some comments differently to keep the style checker happy. Downgraded a BUG_ON to a WARN_ON as the latter is preferred.

Removed some obsolete, commented out code.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Updated to use 'to_i915()' instead of dev_private. [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 41 +++++++++++++++++------
 drivers/gpu/drm/i915/intel_lrc.c           | 54 +++++++++++++++++++++++++++---
 drivers/gpu/drm/i915/intel_ringbuffer.c    | 26 ++++++++++++++
 drivers/gpu/drm/i915/intel_ringbuffer.h    |  1 +
 4 files changed, 107 insertions(+), 15 deletions(-)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +30 -11
index 1f8486e..bacee5c 100644
@@ -1148,25 +1148,19 @@ i915_reset_gen7_sol_offsets(struct drm_device *dev,
 {
 	struct intel_engine_cs *engine = req->engine;
 	struct drm_i915_private *dev_priv = dev->dev_private;
-	int ret, i;
+	int i;
 
 	if (!IS_GEN7(dev) || engine != &dev_priv->engine[RCS]) {
 		DRM_DEBUG("sol reset is gen7/rcs only\n");
 		return -EINVAL;
 	}
 
-	ret = intel_ring_begin(req, 4 * 3);
-	if (ret)
-		return ret;
-
 	for (i = 0; i < 4; i++) {
 		intel_ring_emit(engine, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit_reg(engine, GEN7_SO_WRITE_OFFSET(i));
 		intel_ring_emit(engine, 0);
 	}
 
-	intel_ring_advance(engine);
-
 	return 0;
 }
 
@@ -1294,6 +1288,7 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	struct intel_engine_cs  *engine = params->engine;
 	u64 exec_start, exec_len;
 	int ret;
+	uint32_t min_space;
 
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
@@ -1317,6 +1312,34 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 		goto error;
 
 	/*
+	 * It would be a bad idea to run out of space while writing commands
+	 * to the ring. One of the major aims of the scheduler is to not
+	 * stall at any point for any reason. However, doing an early exit
+	 * half way through submission could result in a partial sequence
+	 * being written which would leave the engine in an unknown state.
+	 * Therefore, check in advance that there will be enough space for
+	 * the entire submission whether emitted by the code below OR by any
+	 * other functions that may be executed before the end of final().
+	 *
+	 * NB: This test deliberately overestimates, because that's easier
+	 * than tracing every potential path that could be taken!
+	 *
+	 * Current measurements suggest that we may need to emit up to 186
+	 * dwords, so this is rounded up to 256 here. Then double that to get
+	 * the free space requirement, because the block is not allowed to
+	 * span the transition from the end to the beginning of the ring.
+	 */
+#define I915_BATCH_EXEC_MAX_LEN         256	/* max dwords emitted here */
+	min_space = I915_BATCH_EXEC_MAX_LEN * 2 * sizeof(uint32_t);
+	ret = intel_ring_test_space(req->ringbuf, min_space);
+	if (ret)
+		goto error;
+
+	ret = intel_ring_begin(req, I915_BATCH_EXEC_MAX_LEN);
+	if (ret)
+		goto error;
+
+	/*
 	 * Unconditionally invalidate gpu caches and ensure that we do flush
 	 * any residual writes from the previous batch.
 	 */
@@ -1334,10 +1357,6 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 
 	if (engine == &dev_priv->engine[RCS] &&
 	    params->instp_mode != dev_priv->relative_constants_mode) {
-		ret = intel_ring_begin(req, 4);
-		if (ret)
-			goto error;
-
 		intel_ring_emit(engine, MI_NOOP);
 		intel_ring_emit(engine, MI_LOAD_REGISTER_IMM(1));
 		intel_ring_emit_reg(engine, INSTPM);
drivers/gpu/drm/i915/intel_lrc.c +50 -4

78 lines not on this page · raw message

drivers/gpu/drm/i915/intel_ringbuffer.c +26 -0
index f5bcd24..6ea27c6 100644
@@ -2571,6 +2571,32 @@ int intel_ring_cacheline_align(struct drm_i915_gem_request *req)
 	return 0;
 }
 
+/*
+ * Test to see if the ring has sufficient space to submit a given piece
+ * of work without causing a stall
+ */
+int intel_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space)
+{
+	struct drm_i915_private *dev_priv = to_i915(ringbuf->engine->dev);
+
+	/* There is a separate LRC version of this code. */
+	WARN_ON(i915.enable_execlists);
+
+	if (ringbuf->space < min_space) {
+		/* Need to update the actual ring space. Otherwise, the system
+		 * hangs forever testing a software copy of the space value that
+		 * never changes!
+		 */
+		ringbuf->head  = I915_READ_HEAD(ringbuf->engine);
+		ringbuf->space = intel_ring_space(ringbuf);
+
+		if (ringbuf->space < min_space)
+			return -EAGAIN;
+	}
+
+	return 0;
+}
+
 void intel_ring_init_seqno(struct intel_engine_cs *engine, u32 seqno)
 {
 	struct drm_i915_private *dev_priv = to_i915(engine->dev);
drivers/gpu/drm/i915/intel_ringbuffer.h +1 -0
index 2e7daef..067d635 100644
@@ -450,6 +450,7 @@ void intel_cleanup_engine(struct intel_engine_cs *engine);
 
 int intel_ring_alloc_request_extras(struct drm_i915_gem_request *request);
 
+int intel_ring_test_space(struct intel_ringbuffer *ringbuf, int min_space);
 int __must_check intel_ring_begin(struct drm_i915_gem_request *req, int n);
 int __must_check intel_ring_cacheline_align(struct drm_i915_gem_request *req);
 static inline void intel_ring_emit(struct intel_engine_cs *engine,
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#30 lore raw

PATCH v6 27/34 drm/i915: Added scheduler statistic reporting to debugfs

In reply to [email protected]

From: John Harrison <[email protected]>

It is useful for know what the scheduler is doing for both debugging and performance analysis purposes. This change adds a bunch of counters and such that keep track of various scheduler operations (batches submitted, completed, flush requests, etc.). The data can then be read in userland via the debugfs mechanism.

v2: Updated to match changes to scheduler implementation.

v3: Updated for changes to kill code and flush code.

v4: Removed the fence/sync code as that will be part of a separate patch series. Wrapped a long line to keep the style checker happy.

v5: Updated to remove forward declarations and white space. Added documentation. [Joonas Lahtinen]

Used lighter weight spinlocks.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' and 'assert_scheduler_lock_held()' helper macros. Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. Removed even more white space. Moved the enum to string conversion function to debugfs.c rather than scheduler.c [review feedback from Joonas Lahtinen]

Added running totals of 'flying' and 'queued' nodes rather than re-calculating each time as a minor CPU performance optimisation.

Added stats to the new file queue wait implementation.

For: VIZ-1587

Cc Joonas Lahtinen <[email protected]>
---
 drivers/gpu/drm/i915/i915_debugfs.c        | 111 +++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |   3 +
 drivers/gpu/drm/i915/i915_scheduler.c      |  85 +++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_scheduler.h      |  35 +++++++++
 4 files changed, 231 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_debugfs.c +111 -0
index 980bb20..1d04cde 100644
@@ -3609,6 +3609,116 @@ static int i915_drrs_status(struct seq_file *m, void *unused)
 	return 0;
 }
 
+static const char *i915_scheduler_queue_status_str(
+				enum i915_scheduler_queue_status status)
+{
+	static char	str[50];
+
+	switch (status) {
+	case I915_SQS_NONE:
+	return "None";
+
+	case I915_SQS_QUEUED:
+	return "Queued";
+
+	case I915_SQS_POPPED:
+	return "Popped";
+
+	case I915_SQS_FLYING:
+	return "Flying";
+
+	case I915_SQS_COMPLETE:
+	return "Complete";
+
+	case I915_SQS_DEAD:
+	return "Dead";
+
+	case I915_SQS_MAX:
+	return "Invalid";
+
+	default:
+	break;
+	}
+
+	sprintf(str, "[Unknown_%d!]", status);
+	return str;
+}
+
+static int i915_scheduler_info(struct seq_file *m, void *unused)
+{
+	struct drm_info_node *node = (struct drm_info_node *) m->private;
+	struct drm_device *dev = node->minor->dev;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_stats *stats = scheduler->stats;
+	struct i915_scheduler_stats_nodes node_stats[I915_NUM_ENGINES];
+	struct intel_engine_cs *engine;
+	char   str[50 * (I915_NUM_ENGINES + 1)], name[50], *ptr;
+	int ret, i, e;
+
+	ret = mutex_lock_interruptible(&dev->mode_config.mutex);
+	if (ret)
+		return ret;
+
+#define PRINT_VAR(name, fmt, var)					\
+	do {								\
+		sprintf(str, "%-22s", name);				\
+		ptr = str + strlen(str);				\
+		for_each_engine_id(engine, dev_priv, e) {		\
+			sprintf(ptr, " %10" fmt, var);			\
+			ptr += strlen(ptr);				\
+		}							\
+		seq_printf(m, "%s\n", str);				\
+	} while (0)
+
+	PRINT_VAR("Engine name:",           "s", dev_priv->engine[e].name);
+	PRINT_VAR("  Engine seqno",         "d", engine->get_seqno(engine));
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Batch submissions:\n");
+	PRINT_VAR("  Queued",               "u", stats[e].queued);
+	PRINT_VAR("  Submitted",            "u", stats[e].submitted);
+	PRINT_VAR("  Completed",            "u", stats[e].completed);
+	PRINT_VAR("  Expired",              "u", stats[e].expired);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Flush counts:\n");
+	PRINT_VAR("  By object",            "u", stats[e].flush_obj);
+	PRINT_VAR("  By request",           "u", stats[e].flush_req);
+	PRINT_VAR("  By stamp",             "u", stats[e].flush_stamp);
+	PRINT_VAR("  Blanket",              "u", stats[e].flush_all);
+	PRINT_VAR("  Entries bumped",       "u", stats[e].flush_bump);
+	PRINT_VAR("  Entries submitted",    "u", stats[e].flush_submit);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Miscellaneous:\n");
+	PRINT_VAR("  ExecEarly retry",      "u", stats[e].exec_early);
+	PRINT_VAR("  ExecFinal requeue",    "u", stats[e].exec_again);
+	PRINT_VAR("  ExecFinal killed",     "u", stats[e].exec_dead);
+	PRINT_VAR("  Hung flying",          "u", stats[e].kill_flying);
+	PRINT_VAR("  Hung queued",          "u", stats[e].kill_queued);
+	PRINT_VAR("  File queue wait",      "u", stats[e].file_wait);
+	PRINT_VAR("  File queue stall",     "u", stats[e].file_stall);
+	PRINT_VAR("  File queue lost",      "u", stats[e].file_lost);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Queue contents:\n");
+	for_each_engine(engine, dev_priv)
+		i915_scheduler_query_stats(engine, node_stats + engine->id);
+
+	for (i = 0; i < (I915_SQS_MAX + 1); i++) {
+		sprintf(name, "  %s", i915_scheduler_queue_status_str(i));
+		PRINT_VAR(name, "d", node_stats[e].counts[i]);
+	}
+	seq_putc(m, '\n');
+
+#undef PRINT_VAR
+
+	mutex_unlock(&dev->mode_config.mutex);
+
+	return 0;
+}
+
 struct pipe_crc_info {
 	const char *name;
 	struct drm_device *dev;
@@ -5585,6 +5695,7 @@ static const struct drm_info_list i915_debugfs_list[] = {
 	{"i915_semaphore_status", i915_semaphore_status, 0},
 	{"i915_shared_dplls_info", i915_shared_dplls_info, 0},
 	{"i915_dp_mst_info", i915_dp_mst_info, 0},
+	{"i915_scheduler_info", i915_scheduler_info, 0},
 	{"i915_wa_registers", i915_wa_registers, 0},
 	{"i915_ddb_info", i915_ddb_info, 0},
 	{"i915_sseu_status", i915_sseu_status, 0},
drivers/gpu/drm/i915/i915_gem_execbuffer.c +3 -0
index bacee5c..823283d 100644
@@ -1804,6 +1804,9 @@ pre_mutex_err:
 	/* intel_gpu_busy should also get a ref, so it will free when the device
 	 * is really idle. */
 	intel_runtime_pm_put(dev_priv);
+
+	dev_priv->scheduler->stats[engine->id].exec_early++;
+
 	return ret;
 }
 
drivers/gpu/drm/i915/i915_scheduler.c +82 -3

206 lines not on this page · raw message

drivers/gpu/drm/i915/i915_scheduler.h +35 -0
index 5c33c83..75d7962 100644
@@ -80,6 +80,36 @@ struct i915_scheduler_node_states {
 	uint32_t queued;
 };
 
+struct i915_scheduler_stats {
+	/* Batch buffer counts: */
+	uint32_t queued;
+	uint32_t submitted;
+	uint32_t completed;
+	uint32_t expired;
+
+	/* Other stuff: */
+	uint32_t flush_obj;
+	uint32_t flush_req;
+	uint32_t flush_stamp;
+	uint32_t flush_all;
+	uint32_t flush_bump;
+	uint32_t flush_submit;
+
+	uint32_t exec_early;
+	uint32_t exec_again;
+	uint32_t exec_dead;
+	uint32_t kill_flying;
+	uint32_t kill_queued;
+
+	uint32_t file_wait;
+	uint32_t file_stall;
+	uint32_t file_lost;
+};
+
+struct i915_scheduler_stats_nodes {
+	uint32_t counts[I915_SQS_MAX + 1];
+};
+
 struct i915_scheduler {
 	struct list_head node_queue[I915_NUM_ENGINES];
 	uint32_t flags[I915_NUM_ENGINES];
@@ -95,6 +125,9 @@ struct i915_scheduler {
 	int32_t priority_level_preempt;
 	uint32_t min_flying;
 	uint32_t file_queue_max;
+
+	/* Statistics: */
+	struct i915_scheduler_stats stats[I915_NUM_ENGINES];
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -117,6 +150,8 @@ int i915_scheduler_flush(struct intel_engine_cs *engine, bool is_locked);
 int i915_scheduler_flush_stamp(struct intel_engine_cs *engine,
 			       unsigned long stamp, bool is_locked);
 bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req);
+int i915_scheduler_query_stats(struct intel_engine_cs *engine,
+			       struct i915_scheduler_stats_nodes *stats);
 bool i915_scheduler_file_queue_wait(struct drm_file *file);
 
 #endif  /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#31 lore raw

PATCH v6 27/34 drm/i915: Added scheduler statistic reporting to debugfs

From: John Harrison <[email protected]>

It is useful for know what the scheduler is doing for both debugging and performance analysis purposes. This change adds a bunch of counters and such that keep track of various scheduler operations (batches submitted, completed, flush requests, etc.). The data can then be read in userland via the debugfs mechanism.

v2: Updated to match changes to scheduler implementation.

v3: Updated for changes to kill code and flush code.

v4: Removed the fence/sync code as that will be part of a separate patch series. Wrapped a long line to keep the style checker happy.

v5: Updated to remove forward declarations and white space. Added documentation. [Joonas Lahtinen]

Used lighter weight spinlocks.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' and 'assert_scheduler_lock_held()' helper macros. Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. Removed even more white space. Moved the enum to string conversion function to debugfs.c rather than scheduler.c [review feedback from Joonas Lahtinen]

Added running totals of 'flying' and 'queued' nodes rather than re-calculating each time as a minor CPU performance optimisation.

Added stats to the new file queue wait implementation.

v6.1: Added a target engine parameter to the throttle function. This is used for tracking per engine throttle stats introduced in the v6 update. This fixes a random memory corruption bug caused by using an invalid engine pointer.

For: VIZ-1587

Cc Joonas Lahtinen <[email protected]>
---
 drivers/gpu/drm/i915/i915_debugfs.c        | 111 +++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |   5 +-
 drivers/gpu/drm/i915/i915_scheduler.c      |  89 +++++++++++++++++++++--
 drivers/gpu/drm/i915/i915_scheduler.h      |  38 +++++++++-
 4 files changed, 237 insertions(+), 6 deletions(-)
drivers/gpu/drm/i915/i915_debugfs.c +111 -0
index 980bb20..1d04cde 100644
@@ -3609,6 +3609,116 @@ static int i915_drrs_status(struct seq_file *m, void *unused)
 	return 0;
 }
 
+static const char *i915_scheduler_queue_status_str(
+				enum i915_scheduler_queue_status status)
+{
+	static char	str[50];
+
+	switch (status) {
+	case I915_SQS_NONE:
+	return "None";
+
+	case I915_SQS_QUEUED:
+	return "Queued";
+
+	case I915_SQS_POPPED:
+	return "Popped";
+
+	case I915_SQS_FLYING:
+	return "Flying";
+
+	case I915_SQS_COMPLETE:
+	return "Complete";
+
+	case I915_SQS_DEAD:
+	return "Dead";
+
+	case I915_SQS_MAX:
+	return "Invalid";
+
+	default:
+	break;
+	}
+
+	sprintf(str, "[Unknown_%d!]", status);
+	return str;
+}
+
+static int i915_scheduler_info(struct seq_file *m, void *unused)
+{
+	struct drm_info_node *node = (struct drm_info_node *) m->private;
+	struct drm_device *dev = node->minor->dev;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler   *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_stats *stats = scheduler->stats;
+	struct i915_scheduler_stats_nodes node_stats[I915_NUM_ENGINES];
+	struct intel_engine_cs *engine;
+	char   str[50 * (I915_NUM_ENGINES + 1)], name[50], *ptr;
+	int ret, i, e;
+
+	ret = mutex_lock_interruptible(&dev->mode_config.mutex);
+	if (ret)
+		return ret;
+
+#define PRINT_VAR(name, fmt, var)					\
+	do {								\
+		sprintf(str, "%-22s", name);				\
+		ptr = str + strlen(str);				\
+		for_each_engine_id(engine, dev_priv, e) {		\
+			sprintf(ptr, " %10" fmt, var);			\
+			ptr += strlen(ptr);				\
+		}							\
+		seq_printf(m, "%s\n", str);				\
+	} while (0)
+
+	PRINT_VAR("Engine name:",           "s", dev_priv->engine[e].name);
+	PRINT_VAR("  Engine seqno",         "d", engine->get_seqno(engine));
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Batch submissions:\n");
+	PRINT_VAR("  Queued",               "u", stats[e].queued);
+	PRINT_VAR("  Submitted",            "u", stats[e].submitted);
+	PRINT_VAR("  Completed",            "u", stats[e].completed);
+	PRINT_VAR("  Expired",              "u", stats[e].expired);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Flush counts:\n");
+	PRINT_VAR("  By object",            "u", stats[e].flush_obj);
+	PRINT_VAR("  By request",           "u", stats[e].flush_req);
+	PRINT_VAR("  By stamp",             "u", stats[e].flush_stamp);
+	PRINT_VAR("  Blanket",              "u", stats[e].flush_all);
+	PRINT_VAR("  Entries bumped",       "u", stats[e].flush_bump);
+	PRINT_VAR("  Entries submitted",    "u", stats[e].flush_submit);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Miscellaneous:\n");
+	PRINT_VAR("  ExecEarly retry",      "u", stats[e].exec_early);
+	PRINT_VAR("  ExecFinal requeue",    "u", stats[e].exec_again);
+	PRINT_VAR("  ExecFinal killed",     "u", stats[e].exec_dead);
+	PRINT_VAR("  Hung flying",          "u", stats[e].kill_flying);
+	PRINT_VAR("  Hung queued",          "u", stats[e].kill_queued);
+	PRINT_VAR("  File queue wait",      "u", stats[e].file_wait);
+	PRINT_VAR("  File queue stall",     "u", stats[e].file_stall);
+	PRINT_VAR("  File queue lost",      "u", stats[e].file_lost);
+	seq_putc(m, '\n');
+
+	seq_puts(m, "Queue contents:\n");
+	for_each_engine(engine, dev_priv)
+		i915_scheduler_query_stats(engine, node_stats + engine->id);
+
+	for (i = 0; i < (I915_SQS_MAX + 1); i++) {
+		sprintf(name, "  %s", i915_scheduler_queue_status_str(i));
+		PRINT_VAR(name, "d", node_stats[e].counts[i]);
+	}
+	seq_putc(m, '\n');
+
+#undef PRINT_VAR
+
+	mutex_unlock(&dev->mode_config.mutex);
+
+	return 0;
+}
+
 struct pipe_crc_info {
 	const char *name;
 	struct drm_device *dev;
@@ -5585,6 +5695,7 @@ static const struct drm_info_list i915_debugfs_list[] = {
 	{"i915_semaphore_status", i915_semaphore_status, 0},
 	{"i915_shared_dplls_info", i915_shared_dplls_info, 0},
 	{"i915_dp_mst_info", i915_dp_mst_info, 0},
+	{"i915_scheduler_info", i915_scheduler_info, 0},
 	{"i915_wa_registers", i915_wa_registers, 0},
 	{"i915_ddb_info", i915_ddb_info, 0},
 	{"i915_sseu_status", i915_sseu_status, 0},
drivers/gpu/drm/i915/i915_gem_execbuffer.c +4 -1
index caf7e00..150b699 100644
@@ -1558,7 +1558,7 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	}
 
 	/* Throttle batch requests per device file */
-	if (i915_scheduler_file_queue_wait(file))
+	if (i915_scheduler_file_queue_wait(file, engine))
 		return -EAGAIN;
 
 	intel_runtime_pm_get(dev_priv);
@@ -1808,6 +1808,9 @@ pre_mutex_err:
 	/* intel_gpu_busy should also get a ref, so it will free when the device
 	 * is really idle. */
 	intel_runtime_pm_put(dev_priv);
+
+	dev_priv->scheduler->stats[engine->id].exec_early++;
+
 	return ret;
 }
 
drivers/gpu/drm/i915/i915_scheduler.c +85 -4

222 lines not on this page · raw message

drivers/gpu/drm/i915/i915_scheduler.h +37 -1
index 5c33c83..464c051 100644
@@ -80,6 +80,36 @@ struct i915_scheduler_node_states {
 	uint32_t queued;
 };
 
+struct i915_scheduler_stats {
+	/* Batch buffer counts: */
+	uint32_t queued;
+	uint32_t submitted;
+	uint32_t completed;
+	uint32_t expired;
+
+	/* Other stuff: */
+	uint32_t flush_obj;
+	uint32_t flush_req;
+	uint32_t flush_stamp;
+	uint32_t flush_all;
+	uint32_t flush_bump;
+	uint32_t flush_submit;
+
+	uint32_t exec_early;
+	uint32_t exec_again;
+	uint32_t exec_dead;
+	uint32_t kill_flying;
+	uint32_t kill_queued;
+
+	uint32_t file_wait;
+	uint32_t file_stall;
+	uint32_t file_lost;
+};
+
+struct i915_scheduler_stats_nodes {
+	uint32_t counts[I915_SQS_MAX + 1];
+};
+
 struct i915_scheduler {
 	struct list_head node_queue[I915_NUM_ENGINES];
 	uint32_t flags[I915_NUM_ENGINES];
@@ -95,6 +125,9 @@ struct i915_scheduler {
 	int32_t priority_level_preempt;
 	uint32_t min_flying;
 	uint32_t file_queue_max;
+
+	/* Statistics: */
+	struct i915_scheduler_stats stats[I915_NUM_ENGINES];
 };
 
 /* Flag bits for i915_scheduler::flags */
@@ -117,6 +150,9 @@ int i915_scheduler_flush(struct intel_engine_cs *engine, bool is_locked);
 int i915_scheduler_flush_stamp(struct intel_engine_cs *engine,
 			       unsigned long stamp, bool is_locked);
 bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req);
-bool i915_scheduler_file_queue_wait(struct drm_file *file);
+int i915_scheduler_query_stats(struct intel_engine_cs *engine,
+			       struct i915_scheduler_stats_nodes *stats);
+bool i915_scheduler_file_queue_wait(struct drm_file *file,
+				    struct intel_engine_cs *engine);
 
 #endif  /* _I915_SCHEDULER_H_ */
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#32 lore raw

PATCH v6 28/34 drm/i915: Add scheduler support functions for TDR

In reply to [email protected]

From: John Harrison <[email protected]>

The TDR code needs to know what the scheduler is up to in order to work out whether a ring is really hung or not.

v4: Removed some unnecessary braces to keep the style checker happy.

v5: Removed white space and added documentation. [Joonas Lahtinen]

Also updated for new module parameter.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' helper macro. Updated to use 'to_i915()' instead of dev_private. [review feedback from Joonas Lahtinen]

For: VIZ-1587

Cc Joonas Lahtinen <[email protected]>
---
 drivers/gpu/drm/i915/i915_scheduler.c | 33 +++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h |  1 +
 2 files changed, 34 insertions(+)
drivers/gpu/drm/i915/i915_scheduler.c +33 -0
index 74b33f9..e7377bb 100644
@@ -1388,3 +1388,36 @@ void i915_scheduler_closefile(struct drm_device *dev, struct drm_file *file)
 
 	spin_unlock_irq(&scheduler->lock);
 }
+
+/**
+ * i915_scheduler_is_engine_flying - does the given engine have in flight batches?
+ * @engine: Engine to query
+ * Used by TDR to distinguish hung engines (not moving but with work to do)
+ * from idle engines (not moving because there is nothing to do). Returns true
+ * if the given engine has batches currently executing on the hardware.
+ */
+bool i915_scheduler_is_engine_flying(struct intel_engine_cs *engine)
+{
+	struct drm_i915_private *dev_priv = to_i915(engine->dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+	struct i915_scheduler_queue_entry *node;
+	unsigned long flags;
+	bool found = false;
+
+	/* With the scheduler in bypass mode, no information can be returned. */
+	if (!i915.enable_scheduler)
+		return true;
+
+	spin_lock_irqsave(&scheduler->lock, flags);
+
+	for_each_scheduler_node(node, engine->id) {
+		if (I915_SQS_IS_FLYING(node)) {
+			found = true;
+			break;
+		}
+	}
+
+	spin_unlock_irqrestore(&scheduler->lock, flags);
+
+	return found;
+}
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
index 75d7962..bdb9403 100644
@@ -145,6 +145,7 @@ void i915_scheduler_clean_node(struct i915_scheduler_queue_entry *node);
 int i915_scheduler_queue_execbuffer(struct i915_scheduler_queue_entry *qe);
 bool i915_scheduler_notify_request(struct drm_i915_gem_request *req);
 void i915_scheduler_wakeup(struct drm_device *dev);
+bool i915_scheduler_is_engine_flying(struct intel_engine_cs *engine);
 void i915_scheduler_work_handler(struct work_struct *work);
 int i915_scheduler_flush(struct intel_engine_cs *engine, bool is_locked);
 int i915_scheduler_flush_stamp(struct intel_engine_cs *engine,
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#33 lore raw

PATCH v6 29/34 drm/i915: Enable GPU scheduler by default

In reply to [email protected]

From: John Harrison <[email protected]>

Now that all the scheduler patches have been applied, it is safe to enable.

v5: Updated for new module parameter.

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_params.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_params.c +2 -2
index 45288a2..067f13a 100644
@@ -58,7 +58,7 @@ struct i915_params i915 __read_mostly = {
 	.guc_log_level = -1,
 	.enable_dp_mst = true,
 	.inject_load_failure = 0,
-	.enable_scheduler = 0,
+	.enable_scheduler = 1,
 };
 
 module_param_named(modeset, i915.modeset, int, 0400);
@@ -213,4 +213,4 @@ MODULE_PARM_DESC(inject_load_failure,
 	"Force an error after a number of failure check points (0:disabled (default), N:force failure at the Nth failure check point)");
 
 module_param_named_unsafe(enable_scheduler, i915.enable_scheduler, int, 0600);
-MODULE_PARM_DESC(enable_scheduler, "Enable scheduler (0 = disable [default], 1 = enable)");
+MODULE_PARM_DESC(enable_scheduler, "Enable scheduler (0 = disable, 1 = enable [default])");
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#34 lore raw

PATCH v6 30/34 drm/i915: Add scheduling priority to per-context parameters

In reply to [email protected]

From: Dave Gordon <[email protected]>

Added an interface for user land applications/libraries/services to set their GPU scheduler priority. This extends the existing context parameter IOCTL interface to add a scheduler priority parameter. The range is +/-1023 with +ve numbers meaning higher priority. Only system processes may set a higher priority than the default (zero), normal applications may only lower theirs.

v2: New patch in series.

v6: Updated to use 'to_i915()' instead of dev_private. [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_drv.h            | 14 ++++++++++++++
 drivers/gpu/drm/i915/i915_gem_context.c    | 24 ++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  3 +++
 include/uapi/drm/i915_drm.h                |  1 +
 4 files changed, 42 insertions(+)
drivers/gpu/drm/i915/i915_drv.h +14 -0
index 25b8fd6..4d5d059 100644
@@ -815,6 +815,19 @@ struct i915_ctx_hang_stats {
 	bool banned;
 };
 
+/*
+ * User-settable GFX scheduler priorities are on a scale of -1023 (I don't
+ * care about running) to +1023 (I'm the most important thing in existence)
+ * with zero being the default. Any process may decrease its scheduling
+ * priority, but only a sufficiently privileged process may increase it
+ * beyond zero.
+ */
+
+struct i915_ctx_sched_info {
+	/* Scheduling priority */
+	int32_t priority;
+};
+
 struct i915_fence_timeline {
 	char        name[32];
 	unsigned    fence_context;
@@ -855,6 +868,7 @@ struct intel_context {
 	int flags;
 	struct drm_i915_file_private *file_priv;
 	struct i915_ctx_hang_stats hang_stats;
+	struct i915_ctx_sched_info sched_info;
 	struct i915_hw_ppgtt *ppgtt;
 
 	/* Legacy ring buffer submission */
drivers/gpu/drm/i915/i915_gem_context.c +24 -0
index 6b30de9..6330b39 100644
@@ -961,6 +961,9 @@ int i915_gem_context_getparam_ioctl(struct drm_device *dev, void *data,
 		else
 			args->value = to_i915(dev)->ggtt.base.total;
 		break;
+	case I915_CONTEXT_PARAM_PRIORITY:
+		args->value = (__u64) ctx->sched_info.priority;
+		break;
 	default:
 		ret = -EINVAL;
 		break;
@@ -998,6 +1001,7 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,
 		else
 			ctx->hang_stats.ban_period_seconds = args->value;
 		break;
+
 	case I915_CONTEXT_PARAM_NO_ZEROMAP:
 		if (args->size) {
 			ret = -EINVAL;
@@ -1006,6 +1010,26 @@ int i915_gem_context_setparam_ioctl(struct drm_device *dev, void *data,
 			ctx->flags |= args->value ? CONTEXT_NO_ZEROMAP : 0;
 		}
 		break;
+
+	case I915_CONTEXT_PARAM_PRIORITY:
+	{
+		int32_t	priority = (int32_t) args->value;
+		struct drm_i915_private *dev_priv  = to_i915(dev);
+		struct i915_scheduler   *scheduler = dev_priv->scheduler;
+
+		if (args->size)
+			ret = -EINVAL;
+		else if ((priority > scheduler->priority_level_max) ||
+			 (priority < scheduler->priority_level_min))
+			ret = -EINVAL;
+		else if ((priority > 0) &&
+			 !capable(CAP_SYS_ADMIN))
+			ret = -EPERM;
+		else
+			ctx->sched_info.priority = priority;
+		break;
+	}
+
 	default:
 		ret = -EINVAL;
 		break;
drivers/gpu/drm/i915/i915_gem_execbuffer.c +3 -0
index 823283d..fc663fa 100644
@@ -1719,6 +1719,9 @@ i915_gem_do_execbuffer(struct drm_device *dev, void *data,
 	params->batch_obj               = batch_obj;
 	params->request                 = req;
 
+	/* Start with the context's priority level */
+	qe.priority = ctx->sched_info.priority;
+
 	/*
 	 * Save away the list of objects used by this batch buffer for the
 	 * purpose of tracking inter-buffer dependencies.
include/uapi/drm/i915_drm.h +1 -0
index a5524cc..0e87551 100644
@@ -1167,6 +1167,7 @@ struct drm_i915_gem_context_param {
 #define I915_CONTEXT_PARAM_BAN_PERIOD	0x1
 #define I915_CONTEXT_PARAM_NO_ZEROMAP	0x2
 #define I915_CONTEXT_PARAM_GTT_SIZE	0x3
+#define I915_CONTEXT_PARAM_PRIORITY	0x4
 	__u64 value;
 };
 
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#35 lore raw

PATCH v6 31/34 drm/i915: Add support for retro-actively banning batch buffers

In reply to [email protected]

From: John Harrison <[email protected]>

If a given context submits too many hanging batch buffers then it will be banned and no further batch buffers will be accepted for it. However, it is possible that a large number of buffers may already have been accepted and are sat in the scheduler waiting to be executed. This patch adds a late ban check to ensure that these will also be discarded.

v4: New patch in series.

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c | 6 ++++++
 drivers/gpu/drm/i915/intel_lrc.c           | 6 ++++++
 2 files changed, 12 insertions(+)
drivers/gpu/drm/i915/i915_gem_execbuffer.c +6 -0
index fc663fa..789f668 100644
@@ -1293,6 +1293,12 @@ int i915_gem_ringbuffer_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Check the context wasn't banned between submission and execution: */
+	if (params->ctx->hang_stats.banned) {
+		DRM_DEBUG("Trying to execute for banned context!\n");
+		return -ENOENT;
+	}
+
 	/* Make sure the request's seqno is the latest and greatest: */
 	if (req->reserved_seqno != dev_priv->last_seqno) {
 		ret = i915_gem_get_seqno(engine->dev, &req->reserved_seqno);
drivers/gpu/drm/i915/intel_lrc.c +6 -0
index b9258ee..02808f7 100644
@@ -1034,6 +1034,12 @@ int intel_execlists_submission_final(struct i915_execbuffer_params *params)
 	/* The mutex must be acquired before calling this function */
 	WARN_ON(!mutex_is_locked(&params->dev->struct_mutex));
 
+	/* Check the context wasn't banned between submission and execution: */
+	if (params->ctx->hang_stats.banned) {
+		DRM_DEBUG("Trying to execute for banned context!\n");
+		return -ENOENT;
+	}
+
 	/* Make sure the request's seqno is the latest and greatest: */
 	if (req->reserved_seqno != dev_priv->last_seqno) {
 		ret = i915_gem_get_seqno(engine->dev, &req->reserved_seqno);
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#36 lore raw

PATCH v6 32/34 drm/i915: Allow scheduler to manage inter-ring object synchronisation

In reply to [email protected]

From: John Harrison <[email protected]>

The scheduler has always tracked batch buffer dependencies based on DRM object usage. This means that it will not submit a batch on one ring that has outstanding dependencies still executing on other rings. This is exactly the same synchronisation performed by i915_gem_object_sync() using hardware semaphores where available and CPU stalls where not (e.g. in execlist mode and/or on Gen8 hardware).

Unfortunately, when a batch buffer is submitted to the driver the _object_sync() call happens first. Thus in case where hardware semaphores are disabled, the driver has already stalled until the dependency has been resolved.

This patch adds an optimisation to _object_sync() to ignore the synchronisation in the case where it will subsequently be handled by the scheduler. This removes the driver stall and (in the single application case) provides near hardware semaphore performance even when hardware semaphores are disabled. In a busy system where there is other work that can be executed on the stalling ring, it provides better than hardware semaphore performance as it removes the stall from both the driver and from the hardware. There is also a theory that this method should improve power usage as hardware semaphores are apparently not very power efficient - the stalled ring does not go into as low a power a state as when it is genuinely idle.

The optimisation is to check whether both ends of the synchronisation are batch buffer requests. If they are, then the scheduler will have the inter-dependency tracked and managed. If one or other end is not a batch buffer request (e.g. a page flip) then the code falls back to the CPU stall or hardware semaphore as appropriate.

To check whether the existing usage is a batch buffer, the code simply calls the 'are you tracking this request' function of the scheduler on the object's last_read_req member. To check whether the new usage is a batch buffer, a flag is passed in from the caller.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Replaced the i915_scheduler_is_request_tracked() function with i915_scheduler_is_request_batch_buffer() as the need for the former has gone away and it was really being used to ask the latter question in a convoluted manner. [review feedback from Joonas Lahtinen]

Issue: VIZ-5566

---
 drivers/gpu/drm/i915/i915_drv.h            |  2 +-
 drivers/gpu/drm/i915/i915_gem.c            | 16 +++++++++++++---
 drivers/gpu/drm/i915/i915_gem_execbuffer.c |  2 +-
 drivers/gpu/drm/i915/i915_scheduler.c      | 19 +++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.h      |  1 +
 drivers/gpu/drm/i915/intel_display.c       |  2 +-
 drivers/gpu/drm/i915/intel_lrc.c           |  2 +-
 7 files changed, 37 insertions(+), 7 deletions(-)
drivers/gpu/drm/i915/i915_drv.h +1 -1
index 4d5d059..cb346ab 100644
@@ -3101,7 +3101,7 @@ static inline void i915_gem_object_unpin_map(struct drm_i915_gem_object *obj)
 int __must_check i915_mutex_lock_interruptible(struct drm_device *dev);
 int i915_gem_object_sync(struct drm_i915_gem_object *obj,
 			 struct intel_engine_cs *to,
-			 struct drm_i915_gem_request **to_req);
+			 struct drm_i915_gem_request **to_req, bool to_batch);
 void i915_vma_move_to_active(struct i915_vma *vma,
 			     struct drm_i915_gem_request *req);
 int i915_gem_dumb_create(struct drm_file *file_priv,
drivers/gpu/drm/i915/i915_gem.c +13 -3
index 467d7da..818113d 100644
@@ -3658,7 +3658,7 @@ static int
 __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 		       struct intel_engine_cs *to,
 		       struct drm_i915_gem_request *from_req,
-		       struct drm_i915_gem_request **to_req)
+		       struct drm_i915_gem_request **to_req, bool to_batch)
 {
 	struct intel_engine_cs *from;
 	int ret;
@@ -3670,6 +3670,14 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 	if (i915_gem_request_completed(from_req))
 		return 0;
 
+	/*
+	 * The scheduler will manage inter-ring object dependencies
+	 * as long as both to and from requests are scheduler managed
+	 * (i.e. batch buffers).
+	 */
+	if (to_batch && i915_scheduler_is_request_batch_buffer(from_req))
+		return 0;
+
 	if (!i915_semaphore_is_enabled(obj->base.dev)) {
 		struct drm_i915_private *i915 = to_i915(obj->base.dev);
 		uint32_t flags;
@@ -3728,6 +3736,8 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
  * @to_req: request we wish to use the object for. See below.
  *          This will be allocated and returned if a request is
  *          required but not passed in.
+ * @to_batch: is the sync request on behalf of batch buffer submission?
+ * If so then the scheduler can (potentially) manage the synchronisation.
  *
  * This code is meant to abstract object synchronization with the GPU.
  * Calling with NULL implies synchronizing the object with the CPU
@@ -3758,7 +3768,7 @@ __i915_gem_object_sync(struct drm_i915_gem_object *obj,
 int
 i915_gem_object_sync(struct drm_i915_gem_object *obj,
 		     struct intel_engine_cs *to,
-		     struct drm_i915_gem_request **to_req)
+		     struct drm_i915_gem_request **to_req, bool to_batch)
 {
 	const bool readonly = obj->base.pending_write_domain == 0;
 	struct drm_i915_gem_request *req[I915_NUM_ENGINES];
@@ -3780,7 +3790,7 @@ i915_gem_object_sync(struct drm_i915_gem_object *obj,
 				req[n++] = obj->last_read_req[i];
 	}
 	for (i = 0; i < n; i++) {
-		ret = __i915_gem_object_sync(obj, to, req[i], to_req);
+		ret = __i915_gem_object_sync(obj, to, req[i], to_req, to_batch);
 		if (ret)
 			return ret;
 	}
drivers/gpu/drm/i915/i915_gem_execbuffer.c +1 -1
index 789f668..70ad67c 100644
@@ -954,7 +954,7 @@ i915_gem_execbuffer_move_to_gpu(struct drm_i915_gem_request *req,
 		struct drm_i915_gem_object *obj = vma->obj;
 
 		if (obj->active & other_rings) {
-			ret = i915_gem_object_sync(obj, req->engine, &req);
+			ret = i915_gem_object_sync(obj, req->engine, &req, true);
 			if (ret)
 				return ret;
 		}
drivers/gpu/drm/i915/i915_scheduler.c +19 -0
index e7377bb..fd53833 100644
@@ -1344,6 +1344,25 @@ bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req)
 }
 
 /**
+ * i915_scheduler_is_request_batch_buffer - is this request related to a
+ * batch buffer object?
+ * @req: request to be queried
+ *
+ * Returns true if the given request is for a batch buffer. False means it
+ * is for something else - page flip, context initialisation, etc.
+ */
+bool i915_scheduler_is_request_batch_buffer(struct drm_i915_gem_request *req)
+{
+	if (req->scheduler_qe == NULL)
+		return false;
+
+	if (req->scheduler_qe->params.batch_obj == NULL)
+		return false;
+
+	return true;
+}
+
+/**
  * i915_scheduler_closefile - notify the scheduler that a DRM file handle
  * has been closed.
  * @dev: DRM device
drivers/gpu/drm/i915/i915_scheduler.h +1 -0
index bdb9403..0df16e7 100644
@@ -151,6 +151,7 @@ int i915_scheduler_flush(struct intel_engine_cs *engine, bool is_locked);
 int i915_scheduler_flush_stamp(struct intel_engine_cs *engine,
 			       unsigned long stamp, bool is_locked);
 bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req);
+bool i915_scheduler_is_request_batch_buffer(struct drm_i915_gem_request *req);
 int i915_scheduler_query_stats(struct intel_engine_cs *engine,
 			       struct i915_scheduler_stats_nodes *stats);
 bool i915_scheduler_file_queue_wait(struct drm_file *file);
drivers/gpu/drm/i915/intel_display.c +1 -1
index f1d730a..0e5da38 100644
@@ -11607,7 +11607,7 @@ static int intel_crtc_page_flip(struct drm_crtc *crtc,
 	 * into the display plane and skip any waits.
 	 */
 	if (!mmio_flip) {
-		ret = i915_gem_object_sync(obj, engine, &request);
+		ret = i915_gem_object_sync(obj, engine, &request, false);
 		if (ret)
 			goto cleanup_pending;
 	}
drivers/gpu/drm/i915/intel_lrc.c +1 -1
index 02808f7..c1263e6 100644
@@ -698,7 +698,7 @@ static int execlists_move_to_gpu(struct drm_i915_gem_request *req,
 		struct drm_i915_gem_object *obj = vma->obj;
 
 		if (obj->active & other_rings) {
-			ret = i915_gem_object_sync(obj, req->engine, &req);
+			ret = i915_gem_object_sync(obj, req->engine, &req, true);
 			if (ret)
 				return ret;
 		}
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#37 lore raw

PATCH v6 33/34 drm/i915: Added debug state dump facilities to scheduler

In reply to [email protected]

From: John Harrison <[email protected]>

When debugging batch buffer submission issues, it is useful to be able to see what the current state of the scheduler is. This change adds functions for decoding the internal scheduler state and reporting it.

v3: Updated a debug message with the new state_str() function.

v4: Wrapped some long lines to keep the style checker happy. Removed the fence/sync code as that will now be part of a separate patch series.

v5: Removed forward declarations and white space. Added documentation. [Joonas Lahtinen]

Also squashed in later patch to add seqno information from the start. It was only being added in a separate patch due to historical reasons which have since gone away.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' helper macro. Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. Moved the enum to string conversion function to debugfs.c rather than scheduler.c [review feedback from Joonas Lahtinen]

For: VIZ-1587

Cc Joonas Lahtinen <[email protected]>
---
 drivers/gpu/drm/i915/i915_debugfs.c   |   2 +-
 drivers/gpu/drm/i915/i915_scheduler.c | 270 +++++++++++++++++++++++++++++++++-
 drivers/gpu/drm/i915/i915_scheduler.h |  15 ++
 3 files changed, 284 insertions(+), 3 deletions(-)
drivers/gpu/drm/i915/i915_debugfs.c +1 -1
index 1d04cde..9ac486f 100644
@@ -3609,7 +3609,7 @@ static int i915_drrs_status(struct seq_file *m, void *unused)
 	return 0;
 }
 
-static const char *i915_scheduler_queue_status_str(
+const char *i915_scheduler_queue_status_str(
 				enum i915_scheduler_queue_status status)
 {
 	static char	str[50];
drivers/gpu/drm/i915/i915_scheduler.c +268 -2

290 lines not on this page · raw message

drivers/gpu/drm/i915/i915_scheduler.h +15 -0
index 0df16e7..7c0edf5 100644
@@ -41,6 +41,9 @@ enum i915_scheduler_queue_status {
 	/* Limit value for use with arrays/loops */
 	I915_SQS_MAX
 };
+char i915_scheduler_queue_status_chr(enum i915_scheduler_queue_status status);
+const char *i915_scheduler_queue_status_str(
+				enum i915_scheduler_queue_status status);
 
 #define I915_SQS_IS_QUEUED(node)	(((node)->status == I915_SQS_QUEUED))
 #define I915_SQS_IS_FLYING(node)	(((node)->status == I915_SQS_FLYING))
@@ -74,6 +77,7 @@ struct i915_scheduler_queue_entry {
 	/* List of all scheduler queue entry nodes */
 	struct list_head link;
 };
+const char *i915_qe_state_str(struct i915_scheduler_queue_entry *node);
 
 struct i915_scheduler_node_states {
 	uint32_t flying;
@@ -132,9 +136,17 @@ struct i915_scheduler {
 
 /* Flag bits for i915_scheduler::flags */
 enum {
+	/* Internal state */
 	I915_SF_INTERRUPTS_ENABLED  = (1 << 0),
 	I915_SF_SUBMITTING          = (1 << 1),
+
+	/* Dump/debug flags */
+	I915_SF_DUMP_FORCE          = (1 << 8),
+	I915_SF_DUMP_DETAILS        = (1 << 9),
+	I915_SF_DUMP_DEPENDENCIES   = (1 << 10),
+	I915_SF_DUMP_SEQNO          = (1 << 11),
 };
+const char *i915_scheduler_flag_str(uint32_t flags);
 
 bool i915_scheduler_is_enabled(struct drm_device *dev);
 int i915_scheduler_init(struct drm_device *dev);
@@ -150,6 +162,9 @@ void i915_scheduler_work_handler(struct work_struct *work);
 int i915_scheduler_flush(struct intel_engine_cs *engine, bool is_locked);
 int i915_scheduler_flush_stamp(struct intel_engine_cs *engine,
 			       unsigned long stamp, bool is_locked);
+int i915_scheduler_dump(struct intel_engine_cs *engine,
+			const char *msg);
+int i915_scheduler_dump_all(struct drm_device *dev, const char *msg);
 bool i915_scheduler_is_mutex_required(struct drm_i915_gem_request *req);
 bool i915_scheduler_is_request_batch_buffer(struct drm_i915_gem_request *req);
 int i915_scheduler_query_stats(struct intel_engine_cs *engine,
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#38 lore raw

PATCH v6 34/34 drm/i915: Scheduler state dump via debugfs

In reply to [email protected]

From: John Harrison <[email protected]>

Added a facility for triggering the scheduler state dump via a debugfs entry.

v2: New patch in series.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. [review feedback from Joonas Lahtinen]

For: VIZ-1587

---
 drivers/gpu/drm/i915/i915_debugfs.c   | 33 +++++++++++++++++++++++++++++++++
 drivers/gpu/drm/i915/i915_scheduler.c |  9 +++++----
 drivers/gpu/drm/i915/i915_scheduler.h |  6 ++++++
 3 files changed, 44 insertions(+), 4 deletions(-)
drivers/gpu/drm/i915/i915_debugfs.c +33 -0
index 9ac486f..5328dc1 100644
@@ -1306,6 +1306,38 @@ DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_file_queue_max_fops,
 			i915_scheduler_file_queue_max_set,
 			"%llu\n");
 
+static int
+i915_scheduler_dump_flags_get(void *data, u64 *val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	*val = scheduler->dump_flags;
+
+	return 0;
+}
+
+static int
+i915_scheduler_dump_flags_set(void *data, u64 val)
+{
+	struct drm_device *dev = data;
+	struct drm_i915_private *dev_priv = to_i915(dev);
+	struct i915_scheduler *scheduler = dev_priv->scheduler;
+
+	scheduler->dump_flags = lower_32_bits(val) & I915_SF_DUMP_MASK;
+
+	if (val & 1)
+		i915_scheduler_dump_all(dev, "DebugFS");
+
+	return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(i915_scheduler_dump_flags_fops,
+			i915_scheduler_dump_flags_get,
+			i915_scheduler_dump_flags_set,
+			"0x%llx\n");
+
 static int i915_frequency_info(struct seq_file *m, void *unused)
 {
 	struct drm_info_node *node = m->private;
@@ -5724,6 +5756,7 @@ static const struct i915_debugfs_files {
 	{"i915_scheduler_priority_preempt", &i915_scheduler_priority_preempt_fops},
 	{"i915_scheduler_min_flying", &i915_scheduler_min_flying_fops},
 	{"i915_scheduler_file_queue_max", &i915_scheduler_file_queue_max_fops},
+	{"i915_scheduler_dump_flags", &i915_scheduler_dump_flags_fops},
 	{"i915_display_crc_ctl", &i915_display_crc_ctl_fops},
 	{"i915_pri_wm_latency", &i915_pri_wm_latency_fops},
 	{"i915_spr_wm_latency", &i915_spr_wm_latency_fops},
drivers/gpu/drm/i915/i915_scheduler.c +5 -4
index 92ca786..7afb528 100644
@@ -160,6 +160,10 @@ int i915_scheduler_init(struct drm_device *dev)
 	scheduler->priority_level_preempt = 900;
 	scheduler->min_flying             = 2;
 	scheduler->file_queue_max         = 64;
+	scheduler->dump_flags             = I915_SF_DUMP_FORCE   |
+					    I915_SF_DUMP_DETAILS |
+					    I915_SF_DUMP_SEQNO   |
+					    I915_SF_DUMP_DEPENDENCIES;
 
 	dev_priv->scheduler = scheduler;
 
@@ -1339,10 +1343,7 @@ static int i915_scheduler_dump_all_locked(struct drm_device *dev,
 	int r, ret = 0;
 
 	for_each_engine(engine, dev_priv) {
-		scheduler->flags[engine->id] |= I915_SF_DUMP_FORCE   |
-						I915_SF_DUMP_DETAILS |
-						I915_SF_DUMP_SEQNO   |
-						I915_SF_DUMP_DEPENDENCIES;
+		scheduler->flags[engine->id] |= scheduler->dump_flags & I915_SF_DUMP_MASK;
 		r = i915_scheduler_dump_locked(engine, msg);
 		if (ret == 0)
 			ret = r;
drivers/gpu/drm/i915/i915_scheduler.h +6 -0
index 7c0edf5..1840536 100644
@@ -129,6 +129,7 @@ struct i915_scheduler {
 	int32_t priority_level_preempt;
 	uint32_t min_flying;
 	uint32_t file_queue_max;
+	uint32_t dump_flags;
 
 	/* Statistics: */
 	struct i915_scheduler_stats stats[I915_NUM_ENGINES];
@@ -145,6 +146,11 @@ enum {
 	I915_SF_DUMP_DETAILS        = (1 << 9),
 	I915_SF_DUMP_DEPENDENCIES   = (1 << 10),
 	I915_SF_DUMP_SEQNO          = (1 << 11),
+
+	I915_SF_DUMP_MASK           = I915_SF_DUMP_FORCE        |
+				      I915_SF_DUMP_DETAILS      |
+				      I915_SF_DUMP_DEPENDENCIES |
+				      I915_SF_DUMP_SEQNO,
 };
 const char *i915_scheduler_flag_str(uint32_t flags);
 
Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#39 lore raw

PATCH 1/1 drm/i915: Add wrapper for context priority interface

In reply to [email protected]

From: John Harrison <[email protected]>

There is an EGL extension to set execution priority per context. This can be implemented via the i915 per context priority parameter. This patch adds a wrapper to connect the two together in a way that can be updated as necessary without breaking one side or the other.

---
 include/drm/i915_drm.h   |  1 +
 intel/intel_bufmgr.h     |  5 ++++
 intel/intel_bufmgr_gem.c | 70 ++++++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 76 insertions(+)
include/drm/i915_drm.h +1 -0
index c4ce6b2..8d9f063 100644
@@ -1167,6 +1167,7 @@ struct drm_i915_gem_context_param {
 #define I915_CONTEXT_PARAM_BAN_PERIOD	0x1
 #define I915_CONTEXT_PARAM_NO_ZEROMAP	0x2
 #define I915_CONTEXT_PARAM_GTT_SIZE	0x3
+#define I915_CONTEXT_PARAM_PRIORITY	0x4
 	__u64 value;
 };
 
intel/intel_bufmgr.h +5 -0
index a1abbcd..49d4125 100644
@@ -209,6 +209,11 @@ int drm_intel_gem_bo_wait(drm_intel_bo *bo, int64_t timeout_ns);
 
 drm_intel_context *drm_intel_gem_context_create(drm_intel_bufmgr *bufmgr);
 void drm_intel_gem_context_destroy(drm_intel_context *ctx);
+int drm_intel_gem_context_get_priority(drm_intel_context *ctx, int *priority);
+#define DRM_INTEL_CTX_SET_PRIORITY_EGL_LOW		(0x10001)
+#define DRM_INTEL_CTX_SET_PRIORITY_EGL_MEDIUM		(0x10002)
+#define DRM_INTEL_CTX_SET_PRIORITY_EGL_HIGH		(0x10003)
+int drm_intel_gem_context_set_priority(drm_intel_context *ctx, int priority);
 int drm_intel_gem_bo_context_exec(drm_intel_bo *bo, drm_intel_context *ctx,
 				  int used, unsigned int flags);
 
intel/intel_bufmgr_gem.c +70 -0

76 lines not on this page · raw message

Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#40 lore raw

PATCH 1/2 igt/gem_ctx_param_basic: Updated to support scheduler priority interface

In reply to [email protected]

From: John Harrison <[email protected]>

The GPU scheduler has added an execution priority level to the context object. There is an IOCTL interface to allow user apps/libraries to set this priority. This patch updates the context paramter IOCTL test to include the new interface.

For: VIZ-1587

---
 lib/ioctl_wrappers.h        |   1 +
 tests/gem_ctx_param_basic.c | 108 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 108 insertions(+), 1 deletion(-)
lib/ioctl_wrappers.h +1 -0
index 8fe35b0..00cf324 100644
@@ -109,6 +109,7 @@ struct local_i915_gem_context_param {
 #define LOCAL_CONTEXT_PARAM_BAN_PERIOD	0x1
 #define LOCAL_CONTEXT_PARAM_NO_ZEROMAP	0x2
 #define LOCAL_CONTEXT_PARAM_GTT_SIZE	0x3
+#define LOCAL_CONTEXT_PARAM_PRIORITY	0x4
 	uint64_t value;
 };
 void gem_context_require_ban_period(int fd);
tests/gem_ctx_param_basic.c +107 -1

117 lines not on this page · raw message

Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#41 lore raw

PATCH 2/2 igt/gem_scheduler: Add gem_scheduler test

In reply to [email protected]

From: Derek Morton <[email protected]>

---
 lib/Makefile.sources   |   2 +
 lib/igt.h              |   1 +
 lib/igt_bb_factory.c   | 391 +++++++++++++++++++++++++++++++++++++++++++++
 lib/igt_bb_factory.h   |  47 ++++++
 tests/Makefile.sources |   1 +
 tests/gem_scheduler.c  | 421 +++++++++++++++++++++++++++++++++++++++++++++++++
 6 files changed, 863 insertions(+)
 create mode 100644 lib/igt_bb_factory.c
 create mode 100644 lib/igt_bb_factory.h
 create mode 100644 tests/gem_scheduler.c
lib/Makefile.sources +2 -0
index 1316fd2..c450db2 100644
@@ -7,6 +7,8 @@ libintel_tools_la_SOURCES = 	\
 	i915_reg.h		\
 	i915_pciids.h		\
 	igt.h			\
+	igt_bb_factory.c	\
+	igt_bb_factory.h	\
 	igt_debugfs.c		\
 	igt_debugfs.h		\
 	igt_aux.c		\
lib/igt.h +1 -0
index d751f24..be87915 100644
@@ -37,6 +37,7 @@
 #include "igt_kms.h"
 #include "igt_pm.h"
 #include "igt_stats.h"
+#include "igt_bb_factory.h"
 #include "instdone.h"
 #include "intel_batchbuffer.h"
 #include "intel_chipset.h"
lib/igt_bb_factory.c new file +391 -0

391 lines not on this page · raw message

lib/igt_bb_factory.h new file +47 -0

47 lines not on this page · raw message

tests/Makefile.sources +1 -0
index c52be2c..6acf2f6 100644
@@ -69,6 +69,7 @@ TESTS_progs_M = \
 	gem_request_retire \
 	gem_reset_stats \
 	gem_ringfill \
+	gem_scheduler \
 	gem_set_tiling_vs_blt \
 	gem_shrink \
 	gem_softpin \
tests/gem_scheduler.c new file +421 -0

421 lines not on this page · raw message

Signature · 6 lines
1.9.1

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#42 lore raw

✓ Fi.CI.BAT: success for GPU scheduler for i915 driver (rev2)

In reply to [email protected]
== Series Details ==

== Series Details ==

Series: GPU scheduler for i915 driver (rev2)
URL : https://patchwork.freedesktop.org/series/3585/
State : success

== Summary ==

Series 3585v2 GPU scheduler for i915 driver
http://patchwork.freedesktop.org/api/1.0/series/3585/revisions/2/mbox/

bdw-ultra        total:194  pass:170  dwarn:0   dfail:0   fail:1   skip:23 
bsw-nuc-2        total:193  pass:154  dwarn:0   dfail:0   fail:0   skip:39 
byt-nuc          total:193  pass:155  dwarn:0   dfail:0   fail:0   skip:38 
ilk-hp8440p      total:194  pass:137  dwarn:0   dfail:0   fail:0   skip:57 
ivb-t430s        total:194  pass:166  dwarn:0   dfail:0   fail:0   skip:28 
skl-i7k-2        total:194  pass:168  dwarn:0   dfail:0   fail:1   skip:25 
skl-nuci5        total:194  pass:183  dwarn:0   dfail:0   fail:0   skip:11 

bdw-nuci7 failed to connect after reboot
snb-x220t failed to collect. IGT log at Patchwork_1965/snb-x220t/igt.log

Results at /archive/results/CI_IGT_test/Patchwork_1965/

a09f2af0c7698d8b10f95f12bd0aa245dd53017e drm-intel-nightly: 2016y-04m-21d-07h-18m-40s UTC integration manifest 227a282 drm/i915: Add total count to context status debugfs output

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#43 lore raw
In reply to [email protected]

This patch series can be pulled from 'ssh://people.freedesktop.org/~johnharr/scheduler' as the 'scheduler' branch.

Quoted 244 lines · On 20/04/2016 18:13, [email protected] wrote:

From: John Harrison <[email protected]>

Implemented a batch buffer submission scheduler for the i915 DRM driver.

The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code assigns a unique seqno value and then packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list. The scheduler also scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list. If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.

The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.

As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, the seqno values of the individual buffers are not necessary incrementing as the scheduler may have re-ordered their submission. However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.

A deferred work queue is also poked by the interrupt handler. When this wakes up it can do more involved processing such as actually removing completed nodes from the queue and freeing up the resources associated with them (internal memory allocations, DRM object references, context reference, etc.). The work handler also checks the in flight count and calls the submission code if a new slot has appeared.

When the scheduler's submit code is called, it scans the queued node list for the highest priority node that has no unmet dependencies. Note that the dependency calculation is complex as it must take inter-ring dependencies and potential preemptions into account. Note also that in the future this will be extended to include external dependencies such as the Android Native Sync file descriptors and/or the linux dma-buff synchronisation scheme.

If a suitable node is found then it is sent to execbuff_final() for submission to the hardware. The in flight count is then re-checked and a new node popped from the list if appropriate.

The scheduler also allows high priority batch buffers (e.g. from a desktop compositor) to jump ahead of whatever is already running if the underlying hardware supports pre-emption. In this situation, any work that was pre-empted is returned to the queued list ready to be resubmitted when no more high priority work is outstanding.

Various IGT tests are in progress to test the scheduler's operation and will follow.

v2: Updated for changes in struct fence patch series and other changes to underlying tree (e.g. removal of cliprects). Also changed priority levels to be signed +/-1023 range and reduced mutex lock usage.

v3: More reuse of cached pointers rather than repeated dereferencing (David Gordon).

Moved the dependency generation code out to a seperate function for easier readability. Also added in support for the read-read optimisation.

Major simplification of the DRM file close handler.

Fixed up an overzealous WARN.

Removed unnecessary flushing of the scheduler queue when waiting for a request.

v4: Removed user land fence/sync integration as this is dependent upon the de-staging of the Android sync code. That de-staging is now being done by someone else. The sync support will be added back in to the scheduler in a separate patch series which must wait for the de-staging to be landed first.

Added support for killing batches from contexts that were banned after the batches were submitted to the scheduler.

Changed various comments to fix typos, update to reflect changes to the code, correct formatting and line wrapping, etc. Also wrapped various long lines and twiddled white space to keep the style checker happy.

Changed a bunch of BUG_ONs to WARN_ONs as apparently the latter are preferred.

Used the correct array memory allocator function (kmalloc_array instead of kmalloc).

Fixed a variable type (review comment by Joonas).

Fixed a WARN_ON firing incorrectly when removing killed nodes from the scheduler's queue.

Dropped register definition update patch from this series. The changes are all for pre-emption so it makes more sense for it to be part of that series instead.

v5: Reverted power management changes as they apparently conflict with mutex acquisition. Converted the override mask module parameter to a set of boolean enable flags (just one in this patch set, but others are added later for controlling pre-emption). [Chris Wilson]

Removed lots of whitespace from i915_scheduler.c and re-ordered it to remove all forward declarations. Squashed down the i915_scheduler.c sections of various patches into the initial 'start of scheduler' patch. Thus the later patches simply hook in existing code into various parts of the driver rather than adding the code as well. Added documentation to various functions. Re-worked the submit function in terms of mutex locking, error handling and exit paths. Split the delayed work handler function in half. Made use of the kernel 'clamp' macro. [Joonas Lahtinen]

Dropped the 'immediate submission' override option. This was a half way house between full scheduler and direct submission and was only really useful during early debug.

Added a re-install of the scheduler's interrupt hook around GPU reset. [Zhipeng Gong]

Used lighter weight spinlocks.

v6: Updated to newer nightly (lots of ring -> engine renaming).

Added 'for_each_scheduler_node()' and 'assert_scheduler_lock_held()' helper macros. Renamed 'i915_gem_execbuff_release_batch_obj' to 'i915_gem_execbuf_release_batch_obj'. Updated to use 'to_i915()' instead of dev_private. Converted all enum labels to uppercase. Removed various unnecessary WARNs. Renamed 'saved_objects' to just 'objs'. More code refactoring. Removed even more white space. Added an i915_scheduler_destroy() function instead of doing explicit clean up of scheduler internals from i915_driver_unload(). Changed extra boolean i915_wait_request() parameter to a flags word and consumed the original boolean parameter too. Also, replaced the i915_scheduler_is_request_tracked() function with i915_scheduler_is_mutex_required() and i915_scheduler_is_request_batch_buffer() as the need for the former has gone away and it was really being used to ask the latter two questions in a convoluted manner. Wrapped boolean 'flush' parameter to intel_engine_idle() with an _flush() macro.
[review feedback from Joonas Lahtinen]

Moved scheduler modue parameter declaration to correct place in i915_params struct. [review feedback from Matt Roper]

Added an admin only check when setting the tuning parameters via debugfs to prevent rogue user code trying to break the system with strange settings. [review feedback from Jesse Barnes]

Added kerneldoc for intel_engine_idle().

Added running totals of 'flying' and 'queued' nodes rather than re-calculating each time as a minor CPU performance optimisation.

Removed support for out of order seqno completion. All the prep work patch series (seqno to request conversion, late seqno assignment, etc.) that has now been done means that the scheduler no longer generates out of order seqno completions. Thus all the complex code for coping with such is no longer required and can be removed.

Fixed a bug in scheduler bypass mode introduced in the clean up code refactoring of v5. The clean up function was seeing the node in the wrong state and thus refusing to process it.

Improved the throttle by file handle feature by chaning from a simple 'return to userland when full' scheme with a 'sleep on request' scheme. The former could lead to the busy polling and wasting lots of CPU time as user land continuously retried the execbuf IOCTL in a tight loop. Now the driver will sleep (without holding the mutex lock) on the oldest request outstanding for that file and then automatically retry. This is closer to the pre-scheduler behaviour of stalling on a full ring buffer.

[Patches against drm-intel-nightly tree fetched 13/04/2016 with struct fence conversion patches applied]

Dave Gordon (2):
drm/i915: Cache request pointer in *_submission_final() drm/i915: Add scheduling priority to per-context parameters

John Harrison (32):
drm/i915: Add total count to context status debugfs output drm/i915: Prelude to splitting i915_gem_do_execbuffer in two drm/i915: Split i915_dem_do_execbuffer() in half
drm/i915: Re-instate request->uniq because it is extremely useful drm/i915: Start of GPU scheduler
drm/i915: Disable hardware semaphores when GPU scheduler is enabled drm/i915: Force MMIO flips when scheduler enabled
drm/i915: Added scheduler hook when closing DRM file handles drm/i915: Added scheduler hook into i915_gem_request_notify() drm/i915: Added deferred work handler for scheduler
drm/i915: Redirect execbuffer_final() via scheduler
drm/i915: Keep the reserved space mechanism happy
drm/i915: Added tracking/locking of batch buffer objects drm/i915: Hook scheduler node clean up into retire requests drm/i915: Added scheduler support to __wait_request() calls drm/i915: Added scheduler support to page fault handler drm/i915: Added scheduler flush calls to ring throttle and idle functions drm/i915: Add scheduler hook to GPU reset
drm/i915: Added a module parameter to allow the scheduler to be disabled drm/i915: Support for 'unflushed' ring idle
drm/i915: Defer seqno allocation until actual hardware submission time drm/i915: Added trace points to scheduler
drm/i915: Added scheduler queue throttling by DRM file handle drm/i915: Added debugfs interface to scheduler tuning parameters drm/i915: Add early exit to execbuff_final() if insufficient ring space drm/i915: Added scheduler statistic reporting to debugfs drm/i915: Add scheduler support functions for TDR
drm/i915: Enable GPU scheduler by default
drm/i915: Add support for retro-actively banning batch buffers drm/i915: Allow scheduler to manage inter-ring object synchronisation drm/i915: Added debug state dump facilities to scheduler drm/i915: Scheduler state dump via debugfs

  drivers/gpu/drm/i915/Makefile              |    1 +
  drivers/gpu/drm/i915/i915_debugfs.c        |  336 +++++-
  drivers/gpu/drm/i915/i915_dma.c            |    5 +
  drivers/gpu/drm/i915/i915_drv.c            |    9 +
  drivers/gpu/drm/i915/i915_drv.h            |   58 +-
  drivers/gpu/drm/i915/i915_gem.c            |  156 ++-
  drivers/gpu/drm/i915/i915_gem_context.c    |   24 +
  drivers/gpu/drm/i915/i915_gem_execbuffer.c |  297 +++--
  drivers/gpu/drm/i915/i915_params.c         |    4 +
  drivers/gpu/drm/i915/i915_params.h         |    1 +
  drivers/gpu/drm/i915/i915_scheduler.c      | 1709 ++++++++++++++++++++++++++++
  drivers/gpu/drm/i915/i915_scheduler.h      |  180 +++
  drivers/gpu/drm/i915/i915_trace.h          |  225 +++-
  drivers/gpu/drm/i915/intel_display.c       |   10 +-
  drivers/gpu/drm/i915/intel_lrc.c           |  161 ++-
  drivers/gpu/drm/i915/intel_lrc.h           |    1 +
  drivers/gpu/drm/i915/intel_ringbuffer.c    |   69 +-
  drivers/gpu/drm/i915/intel_ringbuffer.h    |    5 +-
  include/uapi/drm/i915_drm.h                |    1 +
  19 files changed, 3118 insertions(+), 134 deletions(-)
Quoted 3 lines

create mode 100644 drivers/gpu/drm/i915/i915_scheduler.c create mode 100644 drivers/gpu/drm/i915/i915_scheduler.h

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#44 lore raw

✗ Fi.CI.BAT: failure for GPU scheduler for i915 driver (rev2)

In reply to [email protected]
== Series Details ==

== Series Details ==

Series: GPU scheduler for i915 driver (rev2)
URL : https://patchwork.freedesktop.org/series/3585/
State : failure

== Summary ==

Series 3585v2 GPU scheduler for i915 driver
http://patchwork.freedesktop.org/api/1.0/series/3585/revisions/2/mbox/

Test kms_flip:

        Subgroup basic-flip-vs-wf_vblank:
                pass       -> FAIL       (snb-x220t)
Test kms_pipe_crc_basic:
        Subgroup hang-read-crc-pipe-c:
                skip       -> INCOMPLETE (snb-dellxps)
bdw-nuci7        total:193  pass:181  dwarn:0   dfail:0   fail:0   skip:12 
bdw-ultra        total:193  pass:170  dwarn:0   dfail:0   fail:0   skip:23 
bsw-nuc-2        total:192  pass:153  dwarn:0   dfail:0   fail:0   skip:39 
byt-nuc          total:192  pass:154  dwarn:0   dfail:0   fail:0   skip:38 
ilk-hp8440p      total:193  pass:136  dwarn:0   dfail:0   fail:0   skip:57 
ivb-t430s        total:193  pass:165  dwarn:0   dfail:0   fail:0   skip:28 
skl-i7k-2        total:193  pass:168  dwarn:0   dfail:0   fail:0   skip:25 
skl-nuci5        total:193  pass:182  dwarn:0   dfail:0   fail:0   skip:11 
snb-dellxps      total:45   pass:34   dwarn:0   dfail:0   fail:0   skip:10 
snb-x220t        total:193  pass:154  dwarn:0   dfail:0   fail:2   skip:37 

Results at /archive/results/CI_IGT_test/Patchwork_2009/

340c485ad98d0ec0369a3b18d4a09938f3f5537d drm-intel-nightly: 2016y-04m-22d-17h-32m-25s UTC integration manifest 0d53aec drm/i915: Add total count to context status debugfs output

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#45 lore raw
In reply to [email protected]
Quoted 9 lines · On Wed, Apr 20, 2016 at 06:13:18PM +0100, [email protected] wrote:

From: John Harrison <[email protected]>

Implemented a batch buffer submission scheduler for the i915 DRM driver.

The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code assigns a unique seqno value and then packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list.

That is called the request. Please use it properly. We pass a request to the engine. The engine can hook up the scheduler in which case it may defer execution of that request until the scheduler says it is ready. But that is a conversation inside the engine, not intermingled with GEM.

GEM needs to only care about its unique seqno along the timeline. We can leave space for the scheduler to inject a second seqno if that is convenient, but since the scheduler can keep a global list of inflight requests, its retirement is also globally ordered even if we only inspect each request wrt to its own timeline.

Quoted 4 lines

The scheduler also
scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list.

This part moves the policy of the GEM API and enshrines it into the scheduler. Let GEM convert the implicit rules it has into explicit dependencies, on to which we can add the explicit dependencies passed in by the user. GEM has the information it needs to maintain its API, let's not let that silliness spread.

Quoted 5 lines

If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.

Sounds like a rough cut at PI avoidance. This is easily abusable. It sounds like you want a prio/nice split. Is that a good idea to introduce in the first series?

Quoted 5 lines

The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.

Keep it in the scheduler, I see you decided to add EAGAIN to GEM execbuf when currently execbuf *blocks* in this situation. Don't use EAGAIN here since you just make userspace busyspin over a very, very heavyweight ioctl, and you have the power here to throttle userspace without blocking.

Quoted 8 lines

As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, the seqno values of the individual buffers are not necessary incrementing as the scheduler may have re-ordered their submission. However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.

No. You haven't built your timelines correctly. The idea behind the timeline is that a request can wait upon its seqno and check it against its own timeline, which is ordered only with other requests on its timeline. Because of this, it is independent of whatever order the scheduler executes the timelines in, each timeline is ordered.

A request can simply wait for its timeline to advance, completely ignorant of the scheduler. (Request signaling may be driven by the scheduler, but that is a lowlevel, not GEM or dma-buf/fence, implementation detail. And only if the scheduler is coupled into the user-interupt, but on execlists it will be using the context-switch interrupt to driver itself, and for ringbuffer mode we have a choice of user-interrupt or using pipe-control/dw-notify to keep the paths separate.)

Quoted 1 line

A deferred work queue is also poked by the interrupt handler.

Wait you did all that in the interrupt handler. NO.

Quoted 2 lines

If a suitable node is found then it is sent to execbuff_final() for submission to the hardware.

That is an absolutely atrocious implementation. Don't pass it back up the layers when you already have all the information you need packaged in the request to submit it to hardware.

execbuf -> engine->add_request -> [scheduler] -> engine->submit_request

(conceptually, since what should happen is execlists gets it context-switch interrupt and then asks what set of requests to submit next. In a ringbuffer mode, a kthread/ktasklet would run off the interrupt ask which request to execute next and do the write into the ring for the switch into the new context and execution of the next batch, and then write the RING_TAIL.)

At the fundamental level it looks like you have not introduced timelines correctly or introduced the scheduler as a separate entity for deciding which request to execute next (or if this request should preempt execution). -Chris

Signature · 5 lines
Chris Wilson, Intel Open Source Technology Centre
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#46 lore raw
Quoted 10 lines · Chris Wilson
Quoted 9 lines · On Wed, Apr 20, 2016 at 06:13:18PM +0100, [email protected] wrote:

From: John Harrison <[email protected]>

Implemented a batch buffer submission scheduler for the i915 DRM driver.

The general theory of operation is that when batch buffers are submitted to the driver, the execbuffer() code assigns a unique seqno value and then packages up all the information required to execute the batch buffer at a later time. This package is given over to the scheduler which adds it to an internal node list.

Quoted 4 lines

That is called the request. Please use it properly. We pass a request to the engine. The engine can hook up the scheduler in which case it may defer execution of that request until the scheduler says it is ready. But that is a conversation inside the engine, not intermingled with GEM.

It looks like I forgot to update that part of the description. The scheduler and its cover letter were written before the conversion of the driver from using seqnos everywhere to using requests. Indeed, that conversion was implemented as prep-work for the scheduler. So, yes the description should be updated to match the code and say 'assigns a request structure' not a seqno.

Quoted 5 lines

GEM needs to only care about its unique seqno along the timeline. We can leave space for the scheduler to inject a second seqno if that is convenient, but since the scheduler can keep a global list of inflight requests, its retirement is also globally ordered even if we only inspect each request wrt to its own timeline.

Currently, there is no 'own timeline'. The seqno timeline is global to the entire driver and not per engine, per context, or whatever. The conversion to more useful timelines still needs to be implemented. Until then, the scheduler must cope with the single global one.

Quoted 10 lines
Quoted 5 lines

The scheduler also
scans the list of objects associated with the batch buffer and compares them against the objects already in use by other buffers in the node list.

This part moves the policy of the GEM API and enshrines it into the scheduler. Let GEM convert the implicit rules it has into explicit dependencies, on to which we can add the explicit dependencies passed in by the user. GEM has the information it needs to maintain its API, let's not let that silliness spread.

So how should this be done? The scheduler needs some mechanism for ensuring that requests are only submitted to the hardware in a suitable order. And that information needs to persist for as long as the request still exists in an uncompleted state.

Quoted 9 lines
Quoted 6 lines

If matches are found then the new batch buffer node is marked as being dependent upon the matching node. The same is done for the context object. The scheduler also bumps up the priority of such matching nodes on the grounds that the more dependencies a given batch buffer has the more important it is likely to be.

Sounds like a rough cut at PI avoidance. This is easily abusable. It sounds like you want a prio/nice split. Is that a good idea to introduce in the first series?

Not sure how you mean that it can be abused?

Quoted 10 lines
Quoted 5 lines

The scheduler aims to have a given (tuneable) number of batch buffers in flight on the hardware at any given time. If fewer than this are currently executing when a new node is queued, then the node is passed straight through to the submit function. Otherwise it is simply added to the queue and the driver returns back to user land.

Keep it in the scheduler, I see you decided to add EAGAIN to GEM execbuf when currently execbuf *blocks* in this situation. Don't use EAGAIN here since you just make userspace busyspin over a very, very heavyweight ioctl, and you have the power here to throttle userspace without blocking.

Not sure where you are looking. The above description does not talk about returning EAGAIN at any point. It says the batch buffer is either sent directly to the hardware (if the case of idle hardware) or added to the scheduler's queue (in the case of busy hardware). Either way, the execbuf IOCTL returns success to the user. There is a later patch in the series which adds throttling to prevent a user from submitting arbitrarily large numbers of slow batches to the scheduler and effectively consuming unbounded amounts of memory due to the scheduler's queue not being limited by the size of a ring buffer. However, the throttling is done by waiting on an outstanding request so as to block until the queue depth has been reduced. It only returns an error to the user in the case where the wait could not be done for some reason (e.g. wedged GPU).

Quoted 14 lines
Quoted 8 lines

As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, the seqno values of the individual buffers are not necessary incrementing as the scheduler may have re-ordered their submission. However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.

No. You haven't built your timelines correctly. The idea behind the timeline is that a request can wait upon its seqno and check it against its own timeline, which is ordered only with other requests on its timeline. Because of this, it is independent of whatever order the scheduler executes the timelines in, each timeline is ordered.

Quoted 8 lines

A request can simply wait for its timeline to advance, completely ignorant of the scheduler. (Request signaling may be driven by the scheduler, but that is a lowlevel, not GEM or dma-buf/fence, implementation detail. And only if the scheduler is coupled into the user-interupt, but on execlists it will be using the context-switch interrupt to driver itself, and for ringbuffer mode we have a choice of user-interrupt or using pipe-control/dw-notify to keep the paths separate.)

Again, you are assuming per context timelines which the driver does not yet have. That is planned work but is not yet implemented.

Quoted 3 lines
Quoted 2 lines

A deferred work queue is also poked by the interrupt handler.

Wait you did all that in the interrupt handler. NO.

Actually, the comment is again out of date. Since the seqno -> request conversion series, the scheduler no longer has to worry about out of order seqno values from the hardware. So all of the above IRQ mess no longer occurs.

Quoted 16 lines
Quoted 3 lines

If a suitable node is found then it is sent to execbuff_final() for submission to the hardware.

That is an absolutely atrocious implementation. Don't pass it back up the layers when you already have all the information you need packaged in the request to submit it to hardware.

execbuf -> engine->add_request -> [scheduler] -> engine->submit_request

(conceptually, since what should happen is execlists gets it context-switch interrupt and then asks what set of requests to submit next. In a ringbuffer mode, a kthread/ktasklet would run off the interrupt ask which request to execute next and do the write into the ring for the switch into the new context and execution of the next batch, and then write the RING_TAIL.)

Quoted 3 lines

At the fundamental level it looks like you have not introduced timelines correctly or introduced the scheduler as a separate entity for deciding which request to execute next (or if this request should preempt execution).

When this code was first written, there was not even the possibility of implementing per context timelines. Since that first version, there has been a lot of complex prep work (anti-OLR series, seqno to request conversion, struct fence conversion) which all makes the scheduler series a lot simpler. During that process, there has been amply opportunity to comment on the direction it was taking and to explain what could be improved and how.

The agreement on per context timelines is that while it would make things simpler still, it is yet more delay and could be left until after the current scheduler series has landed. Otherwise it is a never ending task to simply keep rebasing the scheduler patches onto a rapidly changing target let alone write entire extra patch series and rework the scheduler accordingly. Plus, we have multiple customers that require the scheduler functionality (and the pre-emption support that builds on top) so we need to get something shipping now rather than in another X years time.

Quoted 2 lines

-Chris

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#47 lore raw
In reply to Chris Wilson
Quoted 15 lines · Chris Wilson
Quoted 8 lines

As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, the seqno values of the individual buffers are not necessary incrementing as the scheduler may have re-ordered their submission. However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.

No. You haven't built your timelines correctly. The idea behind the timeline is that a request can wait upon its seqno and check it against its own timeline, which is ordered only with other requests on its timeline. Because of this, it is independent of whatever order the scheduler executes the timelines in, each timeline is ordered.

Quoted 8 lines

A request can simply wait for its timeline to advance, completely ignorant of the scheduler. (Request signaling may be driven by the scheduler, but that is a lowlevel, not GEM or dma-buf/fence, implementation detail. And only if the scheduler is coupled into the user-interupt, but on execlists it will be using the context-switch interrupt to driver itself, and for ringbuffer mode we have a choice of user-interrupt or using pipe-control/dw-notify to keep the paths separate.)

This is rather crucial, since that expectations that other drivers can rely on fence->seqno being ordered correctly within one timeline. And e.g. amdgpu does what Chris describes and collapses fences on one timeline to just one.

We do have to fix this before we can enable the scheduler.

The related issues with using struct fence (request) more is that we need that also for android integration. On mutli-gpu desktops we already have different kinds of fences, but real soon we'll also have different kinds of fences (gem requests and kms vblank/flip complete events) on just one gpu, and on android.

[more cut]

Quoted 3 lines

At the fundamental level it looks like you have not introduced timelines correctly or introduced the scheduler as a separate entity for deciding which request to execute next (or if this request should preempt execution).

Reworking the scheduler to take request and in-fences, and correctly use timelines is definitely the way to go.

/me out

Cheers, Daniel

Signature · 7 lines
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx
#48 lore raw
Quoted 23 lines · Daniel Vetter
Quoted 14 lines · Chris Wilson
Quoted 8 lines

As each batch buffer completes, it raises an interrupt which wakes up the scheduler. Note that it is possible for multiple buffers to complete before the IRQ handler gets to run. Further, the seqno values of the individual buffers are not necessary incrementing as the scheduler may have re-ordered their submission. However, the scheduler keeps the list of executing buffers in order of hardware submission. Thus it can scan through the list until a matching seqno is found and then mark all in flight nodes from that point on as completed.

No. You haven't built your timelines correctly. The idea behind the timeline is that a request can wait upon its seqno and check it against its own timeline, which is ordered only with other requests on its timeline. Because of this, it is independent of whatever order the scheduler executes the timelines in, each timeline is ordered.

Quoted 8 lines

A request can simply wait for its timeline to advance, completely ignorant of the scheduler. (Request signaling may be driven by the scheduler, but that is a lowlevel, not GEM or dma-buf/fence, implementation detail. And only if the scheduler is coupled into the user-interupt, but on execlists it will be using the context-switch interrupt to driver itself, and for ringbuffer mode we have a choice of user-interrupt or using pipe-control/dw-notify to keep the paths separate.)

Quoted 6 lines

This is rather crucial, since that expectations that other drivers can rely on fence->seqno being ordered correctly within one timeline. And e.g. amdgpu does what Chris describes and collapses fences on one timeline to just one.

We do have to fix this before we can enable the scheduler.

As I have stated several times, this is not broken. The fence series might not introduce per context seqnos for the entire driver but it does introduce a per context timeline specifically for the fence. Thus it is perfectly safe to collapse fences that exist on the same timeline as their internal seqno values are guaranteed to be properly ordered. The fact that the fence's software seqno is not the same value as the driver's hardware seqno is irrelevant to any external view and will not cause a problem.

When the driver has been updated to support per context hardware seqnos, it would be trivial to drop the fence's software timeline and switch to using the hardware one. That has always been the plan.

Quoted 6 lines

The related issues with using struct fence (request) more is that we need that also for android integration. On mutli-gpu desktops we already have different kinds of fences, but real soon we'll also have different kinds of fences (gem requests and kms vblank/flip complete events) on just one gpu, and on android.

Quoted 7 lines

[more cut]

Quoted 3 lines

At the fundamental level it looks like you have not introduced timelines correctly or introduced the scheduler as a separate entity for deciding which request to execute next (or if this request should preempt execution).

Reworking the scheduler to take request and in-fences, and correctly use timelines is definitely the way to go.

The scheduler has been taking 'in-fences' since it's start on Android. That was one of the reasons the whole piece of work was first begun. Before the de-staging patches were dropped from the struct fence series (due to being picked up by external contributors) this scheduler patch series included adding support for passing an external fence in to the execbuf code path to have the driver wait on before allowing that work to begin execution. Once the external work has fully landed and the scheduler patches have been rebased ontop, that support will be back in and is actually really quite trivial. Any resource which has a 'am I ready yet' test can be used as a dependency check on a request when the scheduler is decided what to execute next. Adding in new internal or external dependencies is pretty easy.

Quoted 4 lines

/me out

Cheers, Daniel

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

#49 lore raw

✗ Fi.CI.BAT: warning for GPU scheduler for i915 driver (rev4)

In reply to [email protected]
== Series Details ==

== Series Details ==

Series: GPU scheduler for i915 driver (rev4)
URL : https://patchwork.freedesktop.org/series/3585/
State : warning

== Summary ==

Series 3585v4 GPU scheduler for i915 driver
http://patchwork.freedesktop.org/api/1.0/series/3585/revisions/4/mbox/

Test kms_pipe_crc_basic:

        Subgroup hang-read-crc-pipe-a:
                incomplete -> PASS       (snb-dellxps)
        Subgroup nonblocking-crc-pipe-b:
                pass       -> SKIP       (hsw-brixbox)
        Subgroup suspend-read-crc-pipe-c:
                skip       -> PASS       (hsw-brixbox)
bdw-nuci7-2      total:219  pass:206  dwarn:0   dfail:0   fail:0   skip:13 
bdw-ultra        total:219  pass:193  dwarn:0   dfail:0   fail:0   skip:26 
bsw-nuc-2        total:218  pass:174  dwarn:0   dfail:0   fail:2   skip:42 
byt-nuc          total:218  pass:174  dwarn:0   dfail:0   fail:3   skip:41 
hsw-brixbox      total:219  pass:192  dwarn:0   dfail:0   fail:0   skip:27 
hsw-gt2          total:219  pass:197  dwarn:0   dfail:0   fail:1   skip:21 
ilk-hp8440p      total:219  pass:155  dwarn:0   dfail:0   fail:1   skip:63 
ivb-t430s        total:219  pass:188  dwarn:0   dfail:0   fail:0   skip:31 
skl-i7k-2        total:219  pass:191  dwarn:0   dfail:0   fail:0   skip:28 
skl-nuci5        total:219  pass:207  dwarn:0   dfail:0   fail:0   skip:12 
snb-dellxps      total:219  pass:176  dwarn:0   dfail:0   fail:0   skip:43 
snb-x220t        total:219  pass:176  dwarn:0   dfail:0   fail:1   skip:42 

Results at /archive/results/CI_IGT_test/Patchwork_2147/

4cf0071a8ecbb693f9364c66503fe5b4a7d9aa22 drm-intel-nightly: 2016y-05m-09d-08h-37m-34s UTC integration manifest 28f00ae drm/i915: Add total count to context status debugfs output

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx