Skip to contentSkip to conversation

dma-fence: Fix timestamp memory ordering

PATCH In review Philipp Stanner 4 participants 10 messages Nov 19 – Nov 20

Subsystems (7)
lore
Share & tools
lore

Philipp Stanner

author maintainer
#0 lore raw

cover dma-fence: Fix timestamp memory ordering

Based on misc-fixes

The memory ordering for obtaining a fences timestamp seems to be broken. Moreover, there is no canonical way for users to obtain the timestamp, as proven by the broken ordering in drm_sched.

See also my patch for the signaled flag [1].

@Tvrtko, would you mind basing your patch [2] on this series? That way we'd solve both bugs.

[1] https://lore.kernel.org/dri-devel/[email protected]/
[2] https://lore.kernel.org/dri-devel/[email protected]/

Philipp Stanner (3):
dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()
dma-buf: dma-fence: Add dma_fence_overwrite_timestamp() drm/sched: Fix memory ordering with dma_fence_overwrite_timestamp()

 drivers/dma-buf/dma-fence.c            | 28 ++++++++++++++++++++++++++
 drivers/gpu/drm/scheduler/sched_main.c | 28 +++++++++++++++++++++-----
 include/linux/dma-fence.h              | 15 +++++++++-----
 3 files changed, 61 insertions(+), 10 deletions(-)

base-commit: 7488ee210542b918b9c5749a29fd1502c49a0665

Signature · 1 line
2.56.0
author maintainer
#1 lore raw

PATCH 1/3 dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()

dma_fence_timestamp() tests two fence flags with test_bit(), not providing any ordering.

The while loop + cpu_relax() seem to satisfy the desire to avoid locking, effectively rebuilding a locks spinning behavior in the rare case that a fence is signaling right now. Contrary to proper locking, however, correct memory ordering is not guaranteed.

Ensure correct ordering with locking.

Fixes b83ce9cb4a46 ("dma-buf: add dma_fence_timestamp helper")
---
 include/linux/dma-fence.h | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
include/linux/dma-fence.h +9 -5
index dd07d128adc3..34aefb63a7fe 100644
@@ -708,13 +708,17 @@ static inline void dma_fence_set_error(struct dma_fence *fence,
  */
 static inline ktime_t dma_fence_timestamp(struct dma_fence *fence)
 {
-	if (WARN_ON(!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)))
-		return ktime_get();
+	unsigned long flags;
+	ktime_t ret;
 
-	while (!test_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags))
-		cpu_relax();
+	dma_fence_lock_irqsave(fence, flags);
+	if (WARN_ON(!dma_fence_test_signaled_flag(fence)))
+		ret = ktime_get();
+	else
+		ret = fence->timestamp;
+	dma_fence_unlock_irqrestore(fence, flags);
 
-	return fence->timestamp;
+	return ret;
 }
 
 signed long dma_fence_wait_timeout(struct dma_fence *,
Signature · 1 line
2.56.0
#2 lore raw

Re: [PATCH 1/3] dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] dma-buf/dma-fence: dma_fence_timestamp() reads cb_list.next on i915 fences --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] dma-buf/dma-fence: dma_fence_timestamp() reads cb_list.next on i915 fences
--

commit 095b11917635c3969d6a6f93941ebe8376eacd96
Author: Philipp Stanner <[email protected]>

dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()

This commit replaces the lockless test_bit() and cpu_relax() loop in dma_fence_timestamp() with fence spinlock acquisition to ensure memory ordering when reading fence->timestamp.

Quoted 17 lines
include/linux/dma-fence.h +4 -4
index dd07d128adc3b..34aefb63a7fe8 100644
@@ -708,13 +708,17 @@ static inline void dma_fence_set_error(struct dma_fence *fence,
  */
 static inline ktime_t dma_fence_timestamp(struct dma_fence *fence)
 {
-	if (WARN_ON(!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)))
-		return ktime_get();
+	unsigned long flags;
+	ktime_t ret;
 
-	while (!test_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags))
-		cpu_relax();
+	dma_fence_lock_irqsave(fence, flags);
+	if (WARN_ON(!dma_fence_test_signaled_flag(fence)))

include/linux/dma-fence.h static inline void dma_fence_set_error(struct dma_fence *fence, · line 724

Quoted 10 lines
include/linux/dma-fence.h +5 -1
+		ret = ktime_get();
+	else
+		ret = fence->timestamp;
+	dma_fence_unlock_irqrestore(fence, flags);
 
-	return fence->timestamp;
+	return ret;
 }
 
 signed long dma_fence_wait_timeout(struct dma_fence *,

[Severity: Medium]
Can dma_fence_timestamp() read fence->cb_list.next instead of a valid timestamp when a fence signals in two phases?

In struct dma_fence, cb_list and timestamp share an anonymous union. i915 signals request fences in two phases in drivers/gpu/drm/i915/gt/intel_breadcrumbs.c: __dma_fence_signal() first sets DMA_FENCE_FLAG_SIGNALED_BIT without holding rq->lock, and signal_irq_work() only later acquires rq->lock, replaces cb_list, and calls __dma_fence_signal__timestamp() to write fence->timestamp and set DMA_FENCE_FLAG_TIMESTAMP_BIT:

CPU0 CPU1
irq_signal_request()
__dma_fence_signal(&rq->fence)

    test_and_set_bit(SIGNALED_BIT, ...)
  irq_work_queue(&b->irq_work)
                                        sync_file_ioctl_fence_info()
                                          sync_fill_fence_info()
                                            dma_fence_is_signaled() -> true
                                            dma_fence_timestamp()
                                              spin_lock(&rq->lock)
                                              ret = fence->timestamp
                                              <-- reads fence->cb_list.next
                                              spin_unlock(&rq->lock)

signal_irq_work()
spin_lock(&rq->lock)
list_replace(&rq->fence.cb_list, ...)
__dma_fence_signal__timestamp()

    fence->timestamp = timestamp
    set_bit(TIMESTAMP_BIT, ...)

spin_unlock(&rq->lock)

Because dma_fence_timestamp() no longer waits for DMA_FENCE_FLAG_TIMESTAMP_BIT, callers such as sync_fill_fence_info() can read the raw kernel pointer in fence->cb_list.next as the completion timestamp and copy it to userspace in info->timestamp_ns.

Also, the kernel-doc comment above dma_fence_timestamp() still states that this helper busy-waits for the timestamp to appear.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
maintainer
#3 lore raw

Re: [PATCH 1/3] dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()

In reply to Philipp Stanner
Quoted 9 lines · Philipp Stanner

dma_fence_timestamp() tests two fence flags with test_bit(), not providing any ordering.

The while loop + cpu_relax() seem to satisfy the desire to avoid locking, effectively rebuilding a locks spinning behavior in the rare case that a fence is signaling right now. Contrary to proper locking, however, correct memory ordering is not guaranteed.

Ensure correct ordering with locking.

My words when I tried to clean that up the first time: "Another alternative would be to grab the spinlock when that happens."

I don't remember the story why the busy loop was used and can't see any reasons why it wasn't done this way in the first place.

But my suggestion is to completely drop the DMA_FENCE_FLAG_TIMESTAMP_BIT as well. As far as I can tell that was only added as indicator that the timestamp is set, but we do that now for all fences anyway.

Regards,
Christian.

Quoted 34 lines
Fixes b83ce9cb4a46 ("dma-buf: add dma_fence_timestamp helper")
---
 include/linux/dma-fence.h | 14 +++++++++-----
 1 file changed, 9 insertions(+), 5 deletions(-)
include/linux/dma-fence.h +9 -5
index dd07d128adc3..34aefb63a7fe 100644
@@ -708,13 +708,17 @@ static inline void dma_fence_set_error(struct dma_fence *fence,
  */
 static inline ktime_t dma_fence_timestamp(struct dma_fence *fence)
 {
-	if (WARN_ON(!test_bit(DMA_FENCE_FLAG_SIGNALED_BIT, &fence->flags)))
-		return ktime_get();
+	unsigned long flags;
+	ktime_t ret;
 
-	while (!test_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, &fence->flags))
-		cpu_relax();
+	dma_fence_lock_irqsave(fence, flags);
+	if (WARN_ON(!dma_fence_test_signaled_flag(fence)))
+		ret = ktime_get();
+	else
+		ret = fence->timestamp;
+	dma_fence_unlock_irqrestore(fence, flags);
 
-	return fence->timestamp;
+	return ret;
 }
 
 signed long dma_fence_wait_timeout(struct dma_fence *,
#4 lore raw

Re: [PATCH 1/3] dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()

Quoted 9 lines · Christian König
Quoted 8 lines · Philipp Stanner

dma_fence_timestamp() tests two fence flags with test_bit(), not providing any ordering.

The while loop + cpu_relax() seem to satisfy the desire to avoid locking, effectively rebuilding a locks spinning behavior in the rare case that a fence is signaling right now. Contrary to proper locking, however, correct memory ordering is not guaranteed.

Quoted 4 lines
Quoted 1 line

Ensure correct ordering with locking.

My words when I tried to clean that up the first time: "Another alternative would be to grab the spinlock when that happens."

:)

Quoted 3 lines

I don't remember the story why the busy loop was used and can't see any reasons why it wasn't done this way in the first place.

I reached a point where I don't even want to know the story. Whatever the reason is, it will be something insane <.<

Quoted 5 lines

But my suggestion is to completely drop the DMA_FENCE_FLAG_TIMESTAMP_BIT as well. As far as I can tell that was only added as indicator that the timestamp is set, but we do that now for all fences anyway.

Agreed, but the API has been ossified by amdgpu who again violates layering with drm_sched and by i915, who does something extremely weird in intel_breadcrumbs.c. They seem to have built their own fence there.

amdgpu should be portable though.

P.

author maintainer
#5 lore raw

PATCH 2/3 dma-buf: dma-fence: Add dma_fence_overwrite_timestamp()

In reply to Philipp Stanner

A fence's timestamp is just an opportunistic data point for debugging information etc. drm_sched, for example, uses the timestamps for its scheduling algorithms.

In rare situations it can make sense to allow users to overwrite a timestamp with a more accurate value. This is relatively harmless since the timestamp has no meaning for the implementation itself.

Add dma_fence_overwrite_timestamp().

---
 drivers/dma-buf/dma-fence.c | 28 ++++++++++++++++++++++++++++
 include/linux/dma-fence.h   |  1 +
 2 files changed, 29 insertions(+)
drivers/dma-buf/dma-fence.c +28 -0
index bd58688b81a7..f1b7e90864bf 100644
@@ -412,6 +412,34 @@ void dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp)
 }
 EXPORT_SYMBOL(dma_fence_signal_timestamp);
 
+/**
+ * dma_fence_overwrite_timestamp - overwrite a fence's timestamp
+ * @fence: the fence whose timestamp shall be overwritten
+ * @timestamp: the timestamp
+ *
+ * A dma_fence's timestamp is merely an opportunistically evaluated value to
+ * obtain information such as scheduling granularity. In some situations it can
+ * make sense to overwrite with a custom value. This function provides said
+ * functionality with the correct memory ordering.
+ *
+ * Return: 0 on success, -EAGAIN if the fence was not yet signaled.
+ */
+int dma_fence_overwrite_timestamp(struct dma_fence *fence, ktime_t timestamp)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	dma_fence_lock_irqsave(fence, flags);
+	if (dma_fence_test_signaled_flag(fence))
+		fence->timestamp = timestamp;
+	else
+		ret = -EAGAIN;
+	dma_fence_unlock_irqrestore(fence, flags);
+
+	return ret;
+}
+EXPORT_SYMBOL(dma_fence_overwrite_timestamp);
+
 /**
  * dma_fence_signal_locked - signal completion of a fence
  * @fence: the fence to signal
include/linux/dma-fence.h +1 -0
index 34aefb63a7fe..815471b844f0 100644
@@ -447,6 +447,7 @@ bool dma_fence_check_and_signal_locked(struct dma_fence *fence);
 void dma_fence_signal_locked(struct dma_fence *fence);
 void dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp);
 void dma_fence_signal_timestamp_locked(struct dma_fence *fence, ktime_t timestamp);
+int dma_fence_overwrite_timestamp(struct dma_fence *fence, ktime_t timestamp);
 signed long dma_fence_default_wait(struct dma_fence *fence,
 				   bool intr, signed long timeout);
 int dma_fence_add_callback(struct dma_fence *fence,
Signature · 1 line
2.56.0
maintainer
#6 lore raw

Re: [PATCH 2/3] dma-buf: dma-fence: Add dma_fence_overwrite_timestamp()

Quoted 9 lines · Philipp Stanner

A fence's timestamp is just an opportunistic data point for debugging information etc. drm_sched, for example, uses the timestamps for its scheduling algorithms.

In rare situations it can make sense to allow users to overwrite a timestamp with a more accurate value. This is relatively harmless since the timestamp has no meaning for the implementation itself.

Add dma_fence_overwrite_timestamp().

I think that this is a very bad idea since it gives the impression that overwriting the timestamp on a fence is ok.

That was only done on the scheduled dma-fence because nobody cares about that time in the uAPI. For all other cases that can blow up.

I would rather come up with a clean approach for the scheduler to track the spend time per entity which would make that hack completely superfluous.

IIRC Tvrtko already created patches for that, I'm just not sure if they were ever committed.

Regards,
Christian.

Quoted 59 lines
---
 drivers/dma-buf/dma-fence.c | 28 ++++++++++++++++++++++++++++
 include/linux/dma-fence.h   |  1 +
 2 files changed, 29 insertions(+)
drivers/dma-buf/dma-fence.c +28 -0
index bd58688b81a7..f1b7e90864bf 100644
@@ -412,6 +412,34 @@ void dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp)
 }
 EXPORT_SYMBOL(dma_fence_signal_timestamp);
 
+/**
+ * dma_fence_overwrite_timestamp - overwrite a fence's timestamp
+ * @fence: the fence whose timestamp shall be overwritten
+ * @timestamp: the timestamp
+ *
+ * A dma_fence's timestamp is merely an opportunistically evaluated value to
+ * obtain information such as scheduling granularity. In some situations it can
+ * make sense to overwrite with a custom value. This function provides said
+ * functionality with the correct memory ordering.
+ *
+ * Return: 0 on success, -EAGAIN if the fence was not yet signaled.
+ */
+int dma_fence_overwrite_timestamp(struct dma_fence *fence, ktime_t timestamp)
+{
+	unsigned long flags;
+	int ret = 0;
+
+	dma_fence_lock_irqsave(fence, flags);
+	if (dma_fence_test_signaled_flag(fence))
+		fence->timestamp = timestamp;
+	else
+		ret = -EAGAIN;
+	dma_fence_unlock_irqrestore(fence, flags);
+
+	return ret;
+}
+EXPORT_SYMBOL(dma_fence_overwrite_timestamp);
+
 /**
  * dma_fence_signal_locked - signal completion of a fence
  * @fence: the fence to signal
include/linux/dma-fence.h +1 -0
index 34aefb63a7fe..815471b844f0 100644
@@ -447,6 +447,7 @@ bool dma_fence_check_and_signal_locked(struct dma_fence *fence);
 void dma_fence_signal_locked(struct dma_fence *fence);
 void dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp);
 void dma_fence_signal_timestamp_locked(struct dma_fence *fence, ktime_t timestamp);
+int dma_fence_overwrite_timestamp(struct dma_fence *fence, ktime_t timestamp);
 signed long dma_fence_default_wait(struct dma_fence *fence,
 				   bool intr, signed long timeout);
 int dma_fence_add_callback(struct dma_fence *fence,
#7 lore raw

Re: [PATCH 2/3] dma-buf: dma-fence: Add dma_fence_overwrite_timestamp()

Quoted 14 lines · Christian König
Quoted 9 lines · Philipp Stanner

A fence's timestamp is just an opportunistic data point for debugging information etc. drm_sched, for example, uses the timestamps for its scheduling algorithms.

In rare situations it can make sense to allow users to overwrite a timestamp with a more accurate value. This is relatively harmless since the timestamp has no meaning for the implementation itself.

Add dma_fence_overwrite_timestamp().

I think that this is a very bad idea since it gives the impression that overwriting the timestamp on a fence is ok.

Quoted 6 lines

That was only done on the scheduled dma-fence because nobody cares about that time in the uAPI. For all other cases that can blow up.

I would rather come up with a clean approach for the scheduler to track the spend time per entity which would make that hack completely superfluous.

Yes, I agree.

The *one* feature we would have needed in C is a public-private system. With structs in APIs it's often completely undefined what the user can and cannot touch.

That led to API ossification. Just look at drm_sched, where we now cannot trivially change API internals anymore because external parties conveniently use this or that.

Quoted 2 lines

IIRC Tvrtko already created patches for that, I'm just not sure if they were ever committed.

I suppose it should be somewhat easy to track that data in drm_sched_fence.

Maybe Tvrtko can point me to his ideas so I can take a look.

Thx
P.

author maintainer
#8 lore raw

PATCH 3/3 drm/sched: Fix memory ordering with dma_fence_overwrite_timestamp()

In reply to Philipp Stanner

There is a memory ordering bug in drm_sched_get_finished_job(). The manual timestamp flag check with test_bit() is not ordered.

The dma_fence framework now provides dma_fence_overwrite_timestamp(), a function to overwrite a timestamp in a safe manner with correct memory ordering.

Solve the bug with dma_fence_overwrite_timestamp().

Fixes f7fe64ad0f22 ("drm/sched: Split free_job into own work item")
---
 drivers/gpu/drm/scheduler/sched_main.c | 28 +++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +23 -5
index 99252674e266..a065a6d2eb1c 100644
@@ -66,6 +66,7 @@
  * This implies waiting for previously executed jobs.
  */
 
+#include <linux/dma-fence.h>
 #include <linux/export.h>
 #include <linux/wait.h>
 #include <linux/sched.h>
@@ -898,6 +899,22 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched)
 	return IS_ERR(entity) ? NULL : entity;
 }
 
+/**
+ * drm_sched_overwrite_scheduled_timestamp - make timestamp more accurate
+ *
+ * Overwrites the timestamp of the provided scheduled fence with the one from
+ * the finished fence, if the scheduled fence is already signaled. Does nothing
+ * otherwise.
+ */
+static void
+drm_sched_overwrite_scheduled_timestamp(struct dma_fence *finished,
+					struct dma_fence *scheduled)
+{
+	ktime_t timestamp = dma_fence_timestamp(finished);
+
+	dma_fence_overwrite_timestamp(scheduled, timestamp);
+}
+
 /**
  * drm_sched_get_finished_job - fetch the next finished job to be destroyed
  *
@@ -928,11 +945,12 @@ drm_sched_get_finished_job(struct drm_gpu_scheduler *sched)
 		next = list_first_entry_or_null(&sched->pending_list,
 						typeof(*next), list);
 		if (next) {
-			/* make the scheduled timestamp more accurate */
-			if (test_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT,
-				     &next->s_fence->scheduled.flags))
-				next->s_fence->scheduled.timestamp =
-					dma_fence_timestamp(&job->s_fence->finished);
+			/* Try to make the scheduled timestamp more accurate. */
+			drm_sched_overwrite_scheduled_timestamp(
+					&job->s_fence->finished,
+					&next->s_fence->scheduled
+					);
+
 
 			/* start TO timer for next job */
 			drm_sched_start_timeout(sched);
Signature · 1 line
2.56.0
#9 lore raw

Re: [PATCH 3/3] drm/sched: Fix memory ordering with dma_fence_overwrite_timestamp()

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] drm/sched: missing kernel-doc for drm_sched_overwrite_scheduled_timestamp() --

Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] drm/sched: missing kernel-doc for drm_sched_overwrite_scheduled_timestamp()
--

commit 4813d16b1085e58469356c4e77e1cbeb576347c5
Author: Philipp Stanner <[email protected]>

drm/sched: Fix memory ordering with dma_fence_overwrite_timestamp()

This commit replaces the manual timestamp flag check and assignment in drm_sched_get_finished_job() with dma_fence_overwrite_timestamp() via a new drm_sched_overwrite_scheduled_timestamp() helper.

Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0 -0
index 99252674e2668..a065a6d2eb1c3 100644

[ ... ]

Quoted 9 lines
drivers/gpu/drm/scheduler/sched_main.c +5 -0
@@ -898,6 +899,22 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched)
 	return IS_ERR(entity) ? NULL : entity;
 }
 
+/**
+ * drm_sched_overwrite_scheduled_timestamp - make timestamp more accurate
+ *
+ * Overwrites the timestamp of the provided scheduled fence with the one from
+ * the finished fence, if the scheduled fence is already signaled. Does nothing

drivers/gpu/drm/scheduler/sched_main.c drm_sched_select_entity(struct drm_gpu_scheduler *sched) · line 916

Quoted 10 lines
drivers/gpu/drm/scheduler/sched_main.c +10 -0
+ * otherwise.
+ */
+static void
+drm_sched_overwrite_scheduled_timestamp(struct dma_fence *finished,
+					struct dma_fence *scheduled)
+{
+	ktime_t timestamp = dma_fence_timestamp(finished);
+
+	dma_fence_overwrite_timestamp(scheduled, timestamp);
+}

[Severity: Low]
This kernel-doc comment omits @finished and @scheduled descriptions, which causes scripts/kernel-doc warnings when building with W=1 or generating documentation.

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3