Skip to contentSkip to conversation

Waitboost drm syncobj waits

RFC New Tvrtko Ursulin 2 participants 14 messages Mar 7, 2023

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

v4 cover Waitboost drm syncobj waits

From: Tvrtko Ursulin <[email protected]>

In i915 we have this concept of "wait boosting" where we give a priority boost for instance to fences which are actively waited upon from userspace. This has it's pros and cons and can certainly be discussed at lenght. However fact is some workloads really like it.

Problem is that with the arrival of drm syncobj and a new userspace waiting entry point it added, the waitboost mechanism was bypassed. AFAIU this mostly happens with all Vulkan based userspaces. Hence I cooked up this mini series to see if discussion about restoring the waitboost can be had.

The series adds a concept of "wait count" to dma fence which is intended to represent explicit userspace waits. It is therefore incremented for every explicit dma_fence_enable_sw_signaling and dma_fence_add_wait_callback (like dma_fence_add_callback but from explicit/userspace wait paths). Individual drivers can then inspect this via dma_fence_wait_count() and decide to wait boost the waits on such fences.

Patch has been slightly tested for performance impact by Google using some clvk workloads and shows a good improvement (frame time improved from 16ms to 13ms).

It is also important to mention that although benefits of waitboosting may not only be with workloads related to frame presentation time, but also with serialized computations which constantly move between the CPU and GPU, there are also workloads which do not improve on the performance but degrade on efficiency. Therefore extending this approach needs to be carefully evaluated.

*)
https://gitlab.freedesktop.org/drm/intel/-/issues/8014

v2:
* Small fixups based on CI feedback:
* Handle decrement correctly for already signalled case while adding callback.
* Remove i915 assert which was making sure struct i915_request does not grow.
* Split out the i915 patch into three separate functional changes.

v3:
* Handle drivers which open-code callback additions.

v4:
* Handle dma-fence-array and dma-fence-chain.

Tvrtko Ursulin (12):
dma-fence: Move i915 helpers into common
dma-fence: Add callback initialization helper drm/i915: Use fence callback initialization helper drm/vmwgfx: Use fence callback initialization helper
dma-fence: Track explicit waiters
dma-fence: Export __dma_fence_add_callback
dma-fence-array: Propagate wait status to contained fences
dma-fence-chain: Propagate wait status to contained fences drm/syncobj: Mark syncobj waits as external waiters drm/i915: Waitboost external waits
drm/i915: Mark waits as explicit
drm/i915: Wait boost requests waited upon by others

 drivers/dma-buf/dma-fence-array.c           |   5 +-
 drivers/dma-buf/dma-fence-chain.c           |  22 +++-
 drivers/dma-buf/dma-fence.c                 | 138 ++++++++++++++------
 drivers/gpu/drm/drm_syncobj.c               |   6 +-
 drivers/gpu/drm/i915/gt/intel_breadcrumbs.c |  22 ----
 drivers/gpu/drm/i915/gt/intel_engine_pm.c   |   1 -
 drivers/gpu/drm/i915/i915_active.c          |   2 +-
 drivers/gpu/drm/i915/i915_active.h          |   2 +-
 drivers/gpu/drm/i915/i915_request.c         |  13 +-
 drivers/gpu/drm/vmwgfx/vmwgfx_fence.c       |   2 +-
 include/linux/dma-fence-chain.h             |   1 +
 include/linux/dma-fence.h                   |  29 ++++
 12 files changed, 165 insertions(+), 78 deletions(-)
Signature · 1 line
2.37.2
#1 lore raw

RFC v4 1/12 dma-fence: Move i915 helpers into common

From: Tvrtko Ursulin <[email protected]>

Unhide some i915 helpers which are used for splitting the signalled check vs notification stages during en masse fence processing.

---
 drivers/dma-buf/dma-fence.c                 | 35 +++++++++++++++------
 drivers/gpu/drm/i915/gt/intel_breadcrumbs.c | 22 -------------
 include/linux/dma-fence.h                   |  4 +++
 3 files changed, 30 insertions(+), 31 deletions(-)
drivers/dma-buf/dma-fence.c +26 -9
index 0de0482cd36e..41da0da17eba 100644
@@ -344,6 +344,30 @@ void __dma_fence_might_wait(void)
 }
 #endif
 
+void __dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp)
+{
+	lockdep_assert_held(fence->lock);
+
+	fence->timestamp = timestamp;
+	set_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags);
+	trace_dma_fence_signaled(fence);
+}
+EXPORT_SYMBOL(__dma_fence_signal__timestamp);
+
+void
+__dma_fence_signal__notify(struct dma_fence *fence,
+			   const struct list_head *list)
+{
+	struct dma_fence_cb *cur, *tmp;
+
+	lockdep_assert_held(fence->lock);
+
+	list_for_each_entry_safe(cur, tmp, list, node) {
+		INIT_LIST_HEAD(&cur->node);
+		cur->func(fence, cur);
+	}
+}
+EXPORT_SYMBOL(__dma_fence_signal__notify);
 
 /**
  * dma_fence_signal_timestamp_locked - signal completion of a fence
@@ -366,7 +390,6 @@ void __dma_fence_might_wait(void)
 int dma_fence_signal_timestamp_locked(struct dma_fence *fence,
 				      ktime_t timestamp)
 {
-	struct dma_fence_cb *cur, *tmp;
 	struct list_head cb_list;
 
 	lockdep_assert_held(fence->lock);
@@ -378,14 +401,8 @@ int dma_fence_signal_timestamp_locked(struct dma_fence *fence,
 	/* Stash the cb_list before replacing it with the timestamp */
 	list_replace(&fence->cb_list, &cb_list);
 
-	fence->timestamp = timestamp;
-	set_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags);
-	trace_dma_fence_signaled(fence);
-
-	list_for_each_entry_safe(cur, tmp, &cb_list, node) {
-		INIT_LIST_HEAD(&cur->node);
-		cur->func(fence, cur);
-	}
+	__dma_fence_signal__timestamp(fence, timestamp);
+	__dma_fence_signal__notify(fence, &cb_list);
 
 	return 0;
 }
drivers/gpu/drm/i915/gt/intel_breadcrumbs.c +0 -22
index ecc990ec1b95..26b6b777c479 100644
@@ -127,28 +127,6 @@ __dma_fence_signal(struct dma_fence *fence)
 	return !test_and_set_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags);
 }
 
-static void
-__dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp)
-{
-	fence->timestamp = timestamp;
-	set_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags);
-	trace_dma_fence_signaled(fence);
-}
-
-static void
-__dma_fence_signal__notify(struct dma_fence *fence,
-			   const struct list_head *list)
-{
-	struct dma_fence_cb *cur, *tmp;
-
-	lockdep_assert_held(fence->lock);
-
-	list_for_each_entry_safe(cur, tmp, list, node) {
-		INIT_LIST_HEAD(&cur->node);
-		cur->func(fence, cur);
-	}
-}
-
 static void add_retire(struct intel_breadcrumbs *b, struct intel_timeline *tl)
 {
 	if (b->irq_engine)
include/linux/dma-fence.h +4 -0
index 775cdc0b4f24..2af328a89468 100644
@@ -367,6 +367,10 @@ static inline void dma_fence_end_signalling(bool cookie) {}
 static inline void __dma_fence_might_wait(void) {}
 #endif
 
+void __dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp);
+void __dma_fence_signal__notify(struct dma_fence *fence,
+				const struct list_head *list);
+
 int dma_fence_signal(struct dma_fence *fence);
 int dma_fence_signal_locked(struct dma_fence *fence);
 int dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp);
Signature · 1 line
2.37.2
#2 lore raw

RFC v4 2/12 dma-fence: Add callback initialization helper

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

In preparation of adding a new field to struct dma_fence_cb we will need an initialization helper for those callers who add callbacks by open- coding. That will ensure they initialize all the fields so common code does not get confused by potential garbage in some fields.

---
 drivers/dma-buf/dma-fence.c | 10 ++++------
 include/linux/dma-fence.h   |  7 +++++++
 2 files changed, 11 insertions(+), 6 deletions(-)
drivers/dma-buf/dma-fence.c +4 -6
index 41da0da17eba..ea4a1f82c9bf 100644
@@ -662,21 +662,19 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 	unsigned long flags;
 	int ret = 0;
 
+	__dma_fence_cb_init(cb, func);
+
 	if (WARN_ON(!fence || !func))
 		return -EINVAL;
 
-	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)) {
-		INIT_LIST_HEAD(&cb->node);
+	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
 		return -ENOENT;
-	}
 
 	spin_lock_irqsave(fence->lock, flags);
 
 	if (__dma_fence_enable_signaling(fence)) {
-		cb->func = func;
 		list_add_tail(&cb->node, &fence->cb_list);
 	} else {
-		INIT_LIST_HEAD(&cb->node);
 		ret = -ENOENT;
 	}
 
@@ -795,7 +793,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)
 		goto out;
 	}
 
-	cb.base.func = dma_fence_default_wait_cb;
+	__dma_fence_cb_init(&cb.base, dma_fence_default_wait_cb);
 	cb.task = current;
 	list_add(&cb.base.node, &fence->cb_list);
 
include/linux/dma-fence.h +7 -0
index 2af328a89468..35933e0ae62c 100644
@@ -629,4 +629,11 @@ static inline bool dma_fence_is_container(struct dma_fence *fence)
 	return dma_fence_is_array(fence) || dma_fence_is_chain(fence);
 }
 
+static inline void __dma_fence_cb_init(struct dma_fence_cb *cb,
+				       dma_fence_func_t func)
+{
+	INIT_LIST_HEAD(&cb->node);
+	cb->func = func;
+}
+
 #endif /* __LINUX_DMA_FENCE_H */
Signature · 1 line
2.37.2
#3 lore raw

RFC v4 3/12 drm/i915: Use fence callback initialization helper

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

Use the previously added initialization helper to ensure correct operation of the common code.

---
 drivers/gpu/drm/i915/i915_active.c | 2 +-
 drivers/gpu/drm/i915/i915_active.h | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
drivers/gpu/drm/i915/i915_active.c +1 -1
index a9fea115f2d2..a3ad64583520 100644
@@ -890,7 +890,7 @@ int i915_active_acquire_preallocate_barrier(struct i915_active *ref,
 				goto unwind;
 
 			RCU_INIT_POINTER(node->base.fence, NULL);
-			node->base.cb.func = node_retire;
+			__dma_fence_cb_init(&node->base.cb, node_retire);
 			node->timeline = idx;
 			node->ref = ref;
 		}
drivers/gpu/drm/i915/i915_active.h +1 -1
index 7eb44132183a..da0c5b917cb1 100644
@@ -65,7 +65,7 @@ __i915_active_fence_init(struct i915_active_fence *active,
 			 dma_fence_func_t fn)
 {
 	RCU_INIT_POINTER(active->fence, fence);
-	active->cb.func = fn ?: i915_active_noop;
+	__dma_fence_cb_init(&active->cb, fn ?: i915_active_noop);
 }
 
 #define INIT_ACTIVE_FENCE(A) \
Signature · 1 line
2.37.2
#4 lore raw

RFC v4 4/12 drm/vmwgfx: Use fence callback initialization helper

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

Use the previously added initialization helper to ensure correct operation of the common code.

---
 drivers/gpu/drm/vmwgfx/vmwgfx_fence.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/gpu/drm/vmwgfx/vmwgfx_fence.c +1 -1
index 2a0cda324703..0306596765e5 100644
@@ -200,7 +200,7 @@ static long vmw_fence_wait(struct dma_fence *f, bool intr, signed long timeout)
 		goto out;
 	}
 
-	cb.base.func = vmwgfx_wait_cb;
+	__dma_fence_cb_init(&cb.base, vmwgfx_wait_cb);
 	cb.task = current;
 	list_add(&cb.base.node, &f->cb_list);
 
Signature · 1 line
2.37.2
#5 lore raw

RFC v4 5/12 dma-fence: Track explicit waiters

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

Track how many callers are explicity waiting on a fence to signal and allow querying that via new dma_fence_wait_count() API.

This provides infrastructure on top of which generic "waitboost" concepts can be implemented by individual drivers. Wait-boosting is any reactive activity, such as raising the GPU clocks, which happens while there are active external waiters.

---
 drivers/dma-buf/dma-fence.c               | 98 +++++++++++++++++------
 drivers/gpu/drm/i915/gt/intel_engine_pm.c |  1 -
 include/linux/dma-fence.h                 | 15 ++++
 3 files changed, 87 insertions(+), 27 deletions(-)
drivers/dma-buf/dma-fence.c +72 -26
index ea4a1f82c9bf..bdba5a8e21b1 100644
@@ -344,6 +344,25 @@ void __dma_fence_might_wait(void)
 }
 #endif
 
+static void incr_wait_count(struct dma_fence *fence, struct dma_fence_cb *cb)
+{
+	lockdep_assert_held(fence->lock);
+
+	__set_bit(DMA_FENCE_CB_FLAG_WAITCOUNT_BIT, &cb->flags);
+	fence->waitcount++;
+	WARN_ON_ONCE(!fence->waitcount);
+}
+
+static void decr_wait_count(struct dma_fence *fence, struct dma_fence_cb *cb)
+{
+	lockdep_assert_held(fence->lock);
+
+	if (__test_and_clear_bit(DMA_FENCE_CB_FLAG_WAITCOUNT_BIT, &cb->flags)) {
+		WARN_ON_ONCE(!fence->waitcount);
+		fence->waitcount--;
+	}
+}
+
 void __dma_fence_signal__timestamp(struct dma_fence *fence, ktime_t timestamp)
 {
 	lockdep_assert_held(fence->lock);
@@ -363,6 +382,7 @@ __dma_fence_signal__notify(struct dma_fence *fence,
 	lockdep_assert_held(fence->lock);
 
 	list_for_each_entry_safe(cur, tmp, list, node) {
+		decr_wait_count(fence, cur);
 		INIT_LIST_HEAD(&cur->node);
 		cur->func(fence, cur);
 	}
@@ -629,11 +649,44 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence)
 	unsigned long flags;
 
 	spin_lock_irqsave(fence->lock, flags);
+	fence->waitcount++;
+	WARN_ON_ONCE(!fence->waitcount);
 	__dma_fence_enable_signaling(fence);
 	spin_unlock_irqrestore(fence->lock, flags);
 }
 EXPORT_SYMBOL(dma_fence_enable_sw_signaling);
 
+static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
+			dma_fence_func_t func, bool wait)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	__dma_fence_cb_init(cb, func);
+
+	if (WARN_ON(!fence || !func))
+		return -EINVAL;
+
+	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
+		return -ENOENT;
+
+	spin_lock_irqsave(fence->lock, flags);
+
+	if (wait)
+		incr_wait_count(fence, cb);
+
+	if (__dma_fence_enable_signaling(fence)) {
+		list_add_tail(&cb->node, &fence->cb_list);
+	} else {
+		decr_wait_count(fence, cb);
+		ret = -ENOENT;
+	}
+
+	spin_unlock_irqrestore(fence->lock, flags);
+
+	return ret;
+}
+
 /**
  * dma_fence_add_callback - add a callback to be called when the fence
  * is signaled
@@ -659,31 +712,18 @@ EXPORT_SYMBOL(dma_fence_enable_sw_signaling);
 int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 			   dma_fence_func_t func)
 {
-	unsigned long flags;
-	int ret = 0;
-
-	__dma_fence_cb_init(cb, func);
-
-	if (WARN_ON(!fence || !func))
-		return -EINVAL;
-
-	if (test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags))
-		return -ENOENT;
-
-	spin_lock_irqsave(fence->lock, flags);
-
-	if (__dma_fence_enable_signaling(fence)) {
-		list_add_tail(&cb->node, &fence->cb_list);
-	} else {
-		ret = -ENOENT;
-	}
-
-	spin_unlock_irqrestore(fence->lock, flags);
-
-	return ret;
+	return add_callback(fence, cb, func, false);
 }
 EXPORT_SYMBOL(dma_fence_add_callback);
 
+int dma_fence_add_wait_callback(struct dma_fence *fence,
+				struct dma_fence_cb *cb,
+				dma_fence_func_t func)
+{
+	return add_callback(fence, cb, func, true);
+}
+EXPORT_SYMBOL(dma_fence_add_wait_callback);
+
 /**
  * dma_fence_get_status - returns the status upon completion
  * @fence: the dma_fence to query
@@ -736,8 +776,10 @@ dma_fence_remove_callback(struct dma_fence *fence, struct dma_fence_cb *cb)
 	spin_lock_irqsave(fence->lock, flags);
 
 	ret = !list_empty(&cb->node);
-	if (ret)
+	if (ret) {
+		decr_wait_count(fence, cb);
 		list_del_init(&cb->node);
+	}
 
 	spin_unlock_irqrestore(fence->lock, flags);
 
@@ -795,6 +837,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)
 
 	__dma_fence_cb_init(&cb.base, dma_fence_default_wait_cb);
 	cb.task = current;
+	incr_wait_count(fence, &cb.base);
 	list_add(&cb.base.node, &fence->cb_list);
 
 	while (!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags) && ret > 0) {
@@ -811,8 +854,10 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)
 			ret = -ERESTARTSYS;
 	}
 
-	if (!list_empty(&cb.base.node))
+	if (!list_empty(&cb.base.node)) {
+		decr_wait_count(fence, &cb.base);
 		list_del(&cb.base.node);
+	}
 	__set_current_state(TASK_RUNNING);
 
 out:
@@ -890,8 +935,8 @@ dma_fence_wait_any_timeout(struct dma_fence **fences, uint32_t count,
 		struct dma_fence *fence = fences[i];
 
 		cb[i].task = current;
-		if (dma_fence_add_callback(fence, &cb[i].base,
-					   dma_fence_default_wait_cb)) {
+		if (dma_fence_add_wait_callback(fence, &cb[i].base,
+						dma_fence_default_wait_cb)) {
 			/* This fence is already signaled */
 			if (idx)
 				*idx = i;
@@ -972,6 +1017,7 @@ dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,
 	fence->context = context;
 	fence->seqno = seqno;
 	fence->flags = 0UL;
+	fence->waitcount = 0;
 	fence->error = 0;
 
 	trace_dma_fence_init(fence);
drivers/gpu/drm/i915/gt/intel_engine_pm.c +0 -1
index e971b153fda9..2693a0151a6b 100644
@@ -218,7 +218,6 @@ static bool switch_to_kernel_context(struct intel_engine_cs *engine)
 		 * until the background request retirement running every
 		 * second or two).
 		 */
-		BUILD_BUG_ON(sizeof(rq->duration) > sizeof(rq->submitq));
 		dma_fence_add_callback(&rq->fence, &rq->duration.cb, duration);
 		rq->duration.emitted = ktime_get();
 	}
include/linux/dma-fence.h +15 -0
index 35933e0ae62c..2b696f9de276 100644
@@ -92,6 +92,7 @@ struct dma_fence {
 	u64 seqno;
 	unsigned long flags;
 	struct kref refcount;
+	unsigned int waitcount;
 	int error;
 };
 
@@ -116,6 +117,11 @@ typedef void (*dma_fence_func_t)(struct dma_fence *fence,
 struct dma_fence_cb {
 	struct list_head node;
 	dma_fence_func_t func;
+	unsigned long flags;
+};
+
+enum dma_fence_cb_flag_bits {
+	DMA_FENCE_CB_FLAG_WAITCOUNT_BIT,
 };
 
 /**
@@ -381,6 +387,9 @@ signed long dma_fence_default_wait(struct dma_fence *fence,
 int dma_fence_add_callback(struct dma_fence *fence,
 			   struct dma_fence_cb *cb,
 			   dma_fence_func_t func);
+int dma_fence_add_wait_callback(struct dma_fence *fence,
+				struct dma_fence_cb *cb,
+				dma_fence_func_t func);
 bool dma_fence_remove_callback(struct dma_fence *fence,
 			       struct dma_fence_cb *cb);
 void dma_fence_enable_sw_signaling(struct dma_fence *fence);
@@ -532,6 +541,11 @@ static inline int dma_fence_get_status_locked(struct dma_fence *fence)
 
 int dma_fence_get_status(struct dma_fence *fence);
 
+static inline unsigned int dma_fence_wait_count(struct dma_fence *fence)
+{
+	return fence->waitcount;
+}
+
 /**
  * dma_fence_set_error - flag an error condition on the fence
  * @fence: the dma_fence
@@ -634,6 +648,7 @@ static inline void __dma_fence_cb_init(struct dma_fence_cb *cb,
 {
 	INIT_LIST_HEAD(&cb->node);
 	cb->func = func;
+	cb->flags = 0;
 }
 
 #endif /* __LINUX_DMA_FENCE_H */
Signature · 1 line
2.37.2
#6 lore raw

RFC v4 6/12 dma-fence: Export __dma_fence_add_callback

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

...

---
 drivers/dma-buf/dma-fence.c | 9 +++++----
 include/linux/dma-fence.h   | 3 +++
 2 files changed, 8 insertions(+), 4 deletions(-)
drivers/dma-buf/dma-fence.c +5 -4
index bdba5a8e21b1..5607acdb6ccf 100644
@@ -656,8 +656,8 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence)
 }
 EXPORT_SYMBOL(dma_fence_enable_sw_signaling);
 
-static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
-			dma_fence_func_t func, bool wait)
+int __dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
+			     dma_fence_func_t func, bool wait)
 {
 	unsigned long flags;
 	int ret = 0;
@@ -686,6 +686,7 @@ static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 
 	return ret;
 }
+EXPORT_SYMBOL(__dma_fence_add_callback);
 
 /**
  * dma_fence_add_callback - add a callback to be called when the fence
@@ -712,7 +713,7 @@ static int add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,
 			   dma_fence_func_t func)
 {
-	return add_callback(fence, cb, func, false);
+	return __dma_fence_add_callback(fence, cb, func, false);
 }
 EXPORT_SYMBOL(dma_fence_add_callback);
 
@@ -720,7 +721,7 @@ int dma_fence_add_wait_callback(struct dma_fence *fence,
 				struct dma_fence_cb *cb,
 				dma_fence_func_t func)
 {
-	return add_callback(fence, cb, func, true);
+	return __dma_fence_add_callback(fence, cb, func, true);
 }
 EXPORT_SYMBOL(dma_fence_add_wait_callback);
 
include/linux/dma-fence.h +3 -0
index 2b696f9de276..1f41d60ce6f7 100644
@@ -384,6 +384,9 @@ int dma_fence_signal_timestamp_locked(struct dma_fence *fence,
 				      ktime_t timestamp);
 signed long dma_fence_default_wait(struct dma_fence *fence,
 				   bool intr, signed long timeout);
+int __dma_fence_add_callback(struct dma_fence *fence,
+			     struct dma_fence_cb *cb,
+			     dma_fence_func_t func, bool wait);
 int dma_fence_add_callback(struct dma_fence *fence,
 			   struct dma_fence_cb *cb,
 			   dma_fence_func_t func);
Signature · 1 line
2.37.2
#7 lore raw

RFC v4 7/12 dma-fence-array: Propagate wait status to contained fences

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

As signaling is enabled on the container fence we need to propagate any external waiting status to individual fences in order to enable owning drivers see it.

---
 drivers/dma-buf/dma-fence-array.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
drivers/dma-buf/dma-fence-array.c +3 -2
index 5c8a7084577b..e732adc230d1 100644
@@ -84,8 +84,9 @@ static bool dma_fence_array_enable_signaling(struct dma_fence *fence)
 		 * insufficient).
 		 */
 		dma_fence_get(&array->base);
-		if (dma_fence_add_callback(array->fences[i], &cb[i].cb,
-					   dma_fence_array_cb_func)) {
+		if (__dma_fence_add_callback(array->fences[i], &cb[i].cb,
+					     dma_fence_array_cb_func,
+					     fence->waitcount > 0)) {
 			int error = array->fences[i]->error;
 
 			dma_fence_array_set_pending_error(array, error);
Signature · 1 line
2.37.2
#8 lore raw

RFC v4 8/12 dma-fence-chain: Propagate wait status to contained fences

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

As signaling is enabled on the container fence we need to propagate any external waiting status to individual fences in order to enable owning drivers see it.

---
 drivers/dma-buf/dma-fence-chain.c | 22 ++++++++++++++++------
 include/linux/dma-fence-chain.h   |  1 +
 2 files changed, 17 insertions(+), 6 deletions(-)
drivers/dma-buf/dma-fence-chain.c +16 -6
index a0d920576ba6..ef6a5988c8aa 100644
@@ -145,20 +145,30 @@ static void dma_fence_chain_cb(struct dma_fence *f, struct dma_fence_cb *cb)
 static bool dma_fence_chain_enable_signaling(struct dma_fence *fence)
 {
 	struct dma_fence_chain *head = to_dma_fence_chain(fence);
+	struct dma_fence *callback = NULL;
 
 	dma_fence_get(&head->base);
 	dma_fence_chain_for_each(fence, &head->base) {
 		struct dma_fence *f = dma_fence_chain_contained(fence);
 
-		dma_fence_get(f);
-		if (!dma_fence_add_callback(f, &head->cb, dma_fence_chain_cb)) {
+		if (!callback) {
+			dma_fence_get(f);
+			if (!dma_fence_add_callback(f, &head->cb,
+						    dma_fence_chain_cb))
+				callback = f;
+			else
+				dma_fence_put(f);
+		} else if (head->base.waitcount && !head->waitcount) {
+			dma_fence_enable_sw_signaling(f);
+		} else {
 			dma_fence_put(fence);
-			return true;
+			break;
 		}
-		dma_fence_put(f);
 	}
-	dma_fence_put(&head->base);
-	return false;
+	head->waitcount = head->base.waitcount;
+	if (!callback)
+		dma_fence_put(&head->base);
+	return callback;
 }
 
 static bool dma_fence_chain_signaled(struct dma_fence *fence)
include/linux/dma-fence-chain.h +1 -0
index 4bdf0b96da28..349b882d31ea 100644
@@ -25,6 +25,7 @@
 struct dma_fence_chain {
 	struct dma_fence base;
 	struct dma_fence __rcu *prev;
+	bool waitcount;
 	u64 prev_seqno;
 	struct dma_fence *fence;
 	union {
Signature · 1 line
2.37.2
#9 lore raw

RFC v4 9/12 drm/syncobj: Mark syncobj waits as external waiters

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

Use the previously added dma-fence tracking of explicit waiters.

---
 drivers/gpu/drm/drm_syncobj.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/gpu/drm/drm_syncobj.c +3 -3
index 0c2be8360525..776b90774a64 100644
@@ -1065,9 +1065,9 @@ static signed long drm_syncobj_array_wait_timeout(struct drm_syncobj **syncobjs,
 			if ((flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_AVAILABLE) ||
 			    dma_fence_is_signaled(fence) ||
 			    (!entries[i].fence_cb.func &&
-			     dma_fence_add_callback(fence,
-						    &entries[i].fence_cb,
-						    syncobj_wait_fence_func))) {
+			     dma_fence_add_wait_callback(fence,
+							 &entries[i].fence_cb,
+							 syncobj_wait_fence_func))) {
 				/* The fence has been signaled */
 				if (flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_ALL) {
 					signaled_count++;
Signature · 1 line
2.37.2
#10 lore raw

RFC v4 10/12 drm/i915: Waitboost external waits

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

Userspace waits coming via the drm_syncobj route have so far been bypassing the waitboost mechanism.

Use the previously added dma-fence wait tracking API and apply the same waitboosting logic which applies to other entry points.

This should fix the perfomance regressions experience by clvk and similar userspace which relies on drm_syncobj.

---
 drivers/gpu/drm/i915/i915_request.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_request.c +6 -1
index 630a732aaecc..42b04cced6f0 100644
@@ -93,7 +93,12 @@ static bool i915_fence_signaled(struct dma_fence *fence)
 
 static bool i915_fence_enable_signaling(struct dma_fence *fence)
 {
-	return i915_request_enable_breadcrumb(to_request(fence));
+	struct i915_request *rq = to_request(fence);
+
+	if (dma_fence_wait_count(&rq->fence) && !i915_request_started(rq))
+		intel_rps_boost(rq);
+
+	return i915_request_enable_breadcrumb(rq);
 }
 
 static signed long i915_fence_wait(struct dma_fence *fence,
Signature · 1 line
2.37.2
#11 lore raw

RFC v4 11/12 drm/i915: Mark waits as explicit

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

Use the previously added dma-fence API to mark the direct i915 waits as explicit. This has no significant effect apart from following the new pattern.

---
 drivers/gpu/drm/i915/i915_request.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_request.c +2 -1
index 42b04cced6f0..dd8a9045086a 100644
@@ -2045,7 +2045,8 @@ long i915_request_wait_timeout(struct i915_request *rq,
 		intel_rps_boost(rq);
 
 	wait.tsk = current;
-	if (dma_fence_add_callback(&rq->fence, &wait.cb, request_wait_wake))
+	if (dma_fence_add_wait_callback(&rq->fence, &wait.cb,
+					request_wait_wake))
 		goto out;
 
 	/*
Signature · 1 line
2.37.2
#12 lore raw

RFC v4 12/12 drm/i915: Wait boost requests waited upon by others

In reply to Tvrtko Ursulin

From: Tvrtko Ursulin <[email protected]>

Use the newly added dma-fence API to apply waitboost not only requests which have been marked with I915_WAIT_PRIORITY by i915, but which may be waited upon by others (such as for instance buffer sharing in multi-GPU scenarios).

---
 drivers/gpu/drm/i915/i915_request.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/gpu/drm/i915/i915_request.c +2 -1
index dd8a9045086a..c4adf34016d0 100644
@@ -2041,7 +2041,8 @@ long i915_request_wait_timeout(struct i915_request *rq,
 	 * but at a cost of spending more power processing the workload
 	 * (bad for battery).
 	 */
-	if (flags & I915_WAIT_PRIORITY && !i915_request_started(rq))
+	if (((flags & I915_WAIT_PRIORITY) || dma_fence_wait_count(&rq->fence))
+	    && !i915_request_started(rq))
 		intel_rps_boost(rq);
 
 	wait.tsk = current;
Signature · 1 line
2.37.2
#13 lore raw

[Intel-gfx] ✗ Fi.CI.BAT: failure for Waitboost drm syncobj waits (rev4)

In reply to Tvrtko Ursulin
== Series Details ==

== Series Details ==

Series: Waitboost drm syncobj waits (rev4)
URL : https://patchwork.freedesktop.org/series/113846/
State : failure

== Summary ==

CI Bug Log - changes from CI_DRM_12821 -> Patchwork_113846v4
====================================================

Summary
-------

**FAILURE**

Serious unknown changes coming with Patchwork_113846v4 absolutely need to be verified manually.

If you think the reported changes have nothing to do with the changes introduced in Patchwork_113846v4, please notify your bug team to allow them to document this new failure mode, which will reduce false positives in CI.

External URL: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/index.html

Participating hosts (36 -> 36)
------------------------------

Additional (1): fi-kbl-soraka
Missing (1): fi-snb-2520m

Possible new issues
-------------------

Here are the unknown changes that may have been introduced in Patchwork_113846v4:

### IGT changes ###

#### Possible regressions ####

* igt@i915_suspend@basic-s2idle-without-i915:
- fi-hsw-4770: [PASS][1] -> [ABORT][2]
[1]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/fi-hsw-4770/igt@[email protected]
[2]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-hsw-4770/igt@[email protected]

Known issues
------------

Here are the changes found in Patchwork_113846v4 that come from known issues:

### IGT changes ###

#### Issues hit ####

* igt@gem_huc_copy@huc-copy:
- fi-kbl-soraka: NOTRUN -> [SKIP][3] ([fdo#109271] / [i915#2190])
[3]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@[email protected]

* igt@gem_lmem_swapping@basic:
- fi-kbl-soraka: NOTRUN -> [SKIP][4] ([fdo#109271] / [i915#4613]) +3 similar issues
[4]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@[email protected]

* igt@i915_selftest@live@execlists:
- fi-bsw-n3050: [PASS][5] -> [ABORT][6] ([i915#7911])
[5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/fi-bsw-n3050/igt@i915_selftest@[email protected]
[6]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-bsw-n3050/igt@i915_selftest@[email protected]
- fi-bsw-nick: [PASS][7] -> [ABORT][8] ([i915#7911] / [i915#7913])
[7]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/fi-bsw-nick/igt@i915_selftest@[email protected]
[8]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-bsw-nick/igt@i915_selftest@[email protected]

* igt@i915_selftest@live@gem_contexts:
- fi-kbl-soraka: NOTRUN -> [INCOMPLETE][9] ([i915#7913])
[9]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@i915_selftest@live@gem_contexts.html

* igt@i915_selftest@live@gt_pm:
- fi-kbl-soraka: NOTRUN -> [DMESG-FAIL][10] ([i915#1886])
[10]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@i915_selftest@live@gt_pm.html

* igt@i915_selftest@live@hangcheck:
- fi-skl-guc: [PASS][11] -> [DMESG-WARN][12] ([i915#8073])
[11]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/fi-skl-guc/igt@i915_selftest@[email protected]
[12]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-skl-guc/igt@i915_selftest@[email protected]

* igt@i915_selftest@live@reset:
- bat-rpls-2: [PASS][13] -> [ABORT][14] ([i915#4983] / [i915#7913])
[13]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/bat-rpls-2/igt@i915_selftest@[email protected]
[14]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-2/igt@i915_selftest@[email protected]

* igt@i915_selftest@live@slpc:
- bat-rpls-1: [PASS][15] -> [DMESG-FAIL][16] ([i915#6367])
[15]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/bat-rpls-1/igt@i915_selftest@[email protected]
[16]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-1/igt@i915_selftest@[email protected]

* igt@kms_chamelium_frames@hdmi-crc-fast:
- fi-kbl-soraka: NOTRUN -> [SKIP][17] ([fdo#109271]) +16 similar issues
[17]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/fi-kbl-soraka/igt@[email protected]

* igt@kms_chamelium_hpd@common-hpd-after-suspend:
- bat-rpls-1: NOTRUN -> [SKIP][18] ([i915#7828])
[18]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-1/igt@[email protected]

* igt@kms_pipe_crc_basic@read-crc:
- bat-adlp-9: NOTRUN -> [SKIP][19] ([i915#3546]) +1 similar issue
[19]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-adlp-9/igt@[email protected]

* igt@kms_pipe_crc_basic@suspend-read-crc:
- bat-rpls-1: NOTRUN -> [SKIP][20] ([i915#1845])
[20]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-1/igt@[email protected]

#### Possible fixes ####

* igt@gem_exec_suspend@basic-s3@smem:
- bat-rpls-1: [ABORT][21] ([i915#6687] / [i915#7978]) -> [PASS][22]
[21]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_12821/bat-rpls-1/igt@gem_exec_suspend@[email protected]
[22]: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/bat-rpls-1/igt@gem_exec_suspend@[email protected]

[fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
[i915#1845]: https://gitlab.freedesktop.org/drm/intel/issues/1845
[i915#1886]: https://gitlab.freedesktop.org/drm/intel/issues/1886
[i915#2190]: https://gitlab.freedesktop.org/drm/intel/issues/2190
[i915#3546]: https://gitlab.freedesktop.org/drm/intel/issues/3546
[i915#4613]: https://gitlab.freedesktop.org/drm/intel/issues/4613
[i915#4983]: https://gitlab.freedesktop.org/drm/intel/issues/4983
[i915#6367]: https://gitlab.freedesktop.org/drm/intel/issues/6367
[i915#6687]: https://gitlab.freedesktop.org/drm/intel/issues/6687
[i915#7828]: https://gitlab.freedesktop.org/drm/intel/issues/7828
[i915#7911]: https://gitlab.freedesktop.org/drm/intel/issues/7911
[i915#7913]: https://gitlab.freedesktop.org/drm/intel/issues/7913
[i915#7978]: https://gitlab.freedesktop.org/drm/intel/issues/7978
[i915#8073]: https://gitlab.freedesktop.org/drm/intel/issues/8073

Build changes
-------------

* Linux: CI_DRM_12821 -> Patchwork_113846v4

CI-20190529: 20190529
CI_DRM_12821: 24f94240c4bca70cadfd00528ffd56c3049e5f58 @ git://anongit.freedesktop.org/gfx-ci/linux
IGT_7182: b25ca07c7a75bfda3358c3450810bc023dd7cee9 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
Patchwork_113846v4: 24f94240c4bca70cadfd00528ffd56c3049e5f58 @ git://anongit.freedesktop.org/gfx-ci/linux

### Linux commits

23685a131d74 drm/i915: Wait boost requests waited upon by others 5b86272fb7f4 drm/i915: Mark waits as explicit
11b69ba30cf5 drm/i915: Waitboost external waits
e0a5f922af61 drm/syncobj: Mark syncobj waits as external waiters be104075d13d dma-fence-chain: Propagate wait status to contained fences 0a24463f803c dma-fence-array: Propagate wait status to contained fences f64808b8ce78 dma-fence: Export __dma_fence_add_callback c27b7561c45a dma-fence: Track explicit waiters
212098e25a8d drm/vmwgfx: Use fence callback initialization helper 5183a52e9ee2 drm/i915: Use fence callback initialization helper 3c92d9f13b31 dma-fence: Add callback initialization helper 9ba9823a5aae dma-fence: Move i915 helpers into common

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/Patchwork_113846v4/index.html