dma-fence: Fix timestamp memory ordering
Subsystems (7)
Philipp Stanner
maintainercover dma-fence: Fix timestamp memory ordering
Based on misc-fixes
The memory ordering for obtaining a fences timestamp seems to be broken. Moreover, there is no canonical way for users to obtain the timestamp, as proven by the broken ordering in drm_sched.
See also my patch for the signaled flag [1].
@Tvrtko, would you mind basing your patch [2] on this series? That way we'd solve both bugs.
[1] https://lore.kernel.org/dri-devel/[email protected]/
[2] https://lore.kernel.org/dri-devel/[email protected]/
Philipp Stanner (3):
dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()
dma-buf: dma-fence: Add dma_fence_overwrite_timestamp() drm/sched: Fix memory ordering with dma_fence_overwrite_timestamp()
drivers/dma-buf/dma-fence.c | 28 ++++++++++++++++++++++++++ drivers/gpu/drm/scheduler/sched_main.c | 28 +++++++++++++++++++++----- include/linux/dma-fence.h | 15 +++++++++----- 3 files changed, 61 insertions(+), 10 deletions(-)
base-commit: 7488ee210542b918b9c5749a29fd1502c49a0665
Signature · 1 line
2.56.0
Philipp Stanner
maintainerPATCH 1/3 dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()
dma_fence_timestamp() tests two fence flags with test_bit(), not providing any ordering.
The while loop + cpu_relax() seem to satisfy the desire to avoid locking, effectively rebuilding a locks spinning behavior in the rare case that a fence is signaling right now. Contrary to proper locking, however, correct memory ordering is not guaranteed.
Ensure correct ordering with locking.
--- include/linux/dma-fence.h | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
include/linux/dma-fence.h +9 -5
@@ -708,13 +708,17 @@ static inline void dma_fence_set_error(struct dma_fence *fence,*/ static inline ktime_t dma_fence_timestamp (struct dma_fence *fence) { -if (WARN_ON (!test_bit (DMA_FENCE_FLAG_SIGNALED_BIT , &fence->flags))) -return ktime_get (); +unsigned long flags; +ktime_t ret; -while (!test_bit (DMA_FENCE_FLAG_TIMESTAMP_BIT , &fence->flags)) -cpu_relax (); +dma_fence_lock_irqsave (fence, flags); +if (WARN_ON (!dma_fence_test_signaled_flag (fence))) + ret =ktime_get (); +else + ret = fence->timestamp; +dma_fence_unlock_irqrestore (fence, flags); -return fence->timestamp; +return ret; } signed long dma_fence_wait_timeout(struct dma_fence *,
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH 1/3] dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Medium] dma-buf/dma-fence: dma_fence_timestamp() reads cb_list.next on i915 fences --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Medium] dma-buf/dma-fence: dma_fence_timestamp() reads cb_list.next on i915 fences
--
commit 095b11917635c3969d6a6f93941ebe8376eacd96
Author: Philipp Stanner <[email protected]>
dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()
This commit replaces the lockless test_bit() and cpu_relax() loop in dma_fence_timestamp() with fence spinlock acquisition to ensure memory ordering when reading fence->timestamp.
Quoted 17 lines
include/linux/dma-fence.h +4
-4@@ -708,13 +708,17 @@ static inline void dma_fence_set_error(struct dma_fence *fence,*/ static inline ktime_t dma_fence_timestamp (struct dma_fence *fence) { -if (WARN_ON (!test_bit (DMA_FENCE_FLAG_SIGNALED_BIT , &fence->flags))) -return ktime_get (); +unsigned long flags; +ktime_t ret; -while (!test_bit (DMA_FENCE_FLAG_TIMESTAMP_BIT , &fence->flags)) -cpu_relax (); +dma_fence_lock_irqsave (fence, flags); +if (WARN_ON (!dma_fence_test_signaled_flag (fence)))
include/linux/dma-fence.h static inline void dma_fence_set_error(struct dma_fence *fence, · line 724
Quoted 10 lines
include/linux/dma-fence.h +5
-1+ ret =ktime_get (); + else + ret = fence->timestamp; +dma_fence_unlock_irqrestore (fence, flags); -return fence->timestamp; +return ret; } signed long dma_fence_wait_timeout(struct dma_fence *,
[Severity: Medium]
Can dma_fence_timestamp() read fence->cb_list.next instead of a valid timestamp when a fence signals in two phases?
In struct dma_fence, cb_list and timestamp share an anonymous union. i915 signals request fences in two phases in drivers/gpu/drm/i915/gt/intel_breadcrumbs.c: __dma_fence_signal() first sets DMA_FENCE_FLAG_SIGNALED_BIT without holding rq->lock, and signal_irq_work() only later acquires rq->lock, replaces cb_list, and calls __dma_fence_signal__timestamp() to write fence->timestamp and set DMA_FENCE_FLAG_TIMESTAMP_BIT:
CPU0 CPU1irq_signal_request()
__dma_fence_signal(&rq->fence)
test_and_set_bit(SIGNALED_BIT, ...)
irq_work_queue(&b->irq_work)
sync_file_ioctl_fence_info()
sync_fill_fence_info()
dma_fence_is_signaled() -> true
dma_fence_timestamp()
spin_lock(&rq->lock)
ret = fence->timestamp
<-- reads fence->cb_list.next
spin_unlock(&rq->lock)signal_irq_work()
spin_lock(&rq->lock)
list_replace(&rq->fence.cb_list, ...)
__dma_fence_signal__timestamp()
fence->timestamp = timestamp
set_bit(TIMESTAMP_BIT, ...) spin_unlock(&rq->lock)
Because dma_fence_timestamp() no longer waits for DMA_FENCE_FLAG_TIMESTAMP_BIT, callers such as sync_fill_fence_info() can read the raw kernel pointer in fence->cb_list.next as the completion timestamp and copy it to userspace in info->timestamp_ns.
Also, the kernel-doc comment above dma_fence_timestamp() still states that this helper busy-waits for the timestamp to appear.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Christian König
maintainerRe: [PATCH 1/3] dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()
In reply to Philipp StannerQuoted 9 lines · Philipp Stanner
dma_fence_timestamp()tests two fence flags withtest_bit(), not providing any ordering.The while loop +
cpu_relax()seem to satisfy the desire to avoid locking, effectively rebuilding a locks spinning behavior in the rare case that a fence is signaling right now. Contrary to proper locking, however, correct memory ordering is not guaranteed.Ensure correct ordering with locking.
My words when I tried to clean that up the first time: "Another alternative would be to grab the spinlock when that happens."
I don't remember the story why the busy loop was used and can't see any reasons why it wasn't done this way in the first place.
But my suggestion is to completely drop the DMA_FENCE_FLAG_TIMESTAMP_BIT as well. As far as I can tell that was only added as indicator that the timestamp is set, but we do that now for all fences anyway.
Regards,
Christian.
Quoted 34 lines
Fixes b83ce9cb4a46 ("dma-buf: add dma_fence_timestamp helper")Signed-off-by Philipp Stanner <[email protected]>--- include/linux/dma-fence.h | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)include/linux/dma-fence.h +9
-5@@ -708,13 +708,17 @@ static inline void dma_fence_set_error(struct dma_fence *fence,*/ static inline ktime_t dma_fence_timestamp (struct dma_fence *fence) { -if (WARN_ON (!test_bit (DMA_FENCE_FLAG_SIGNALED_BIT , &fence->flags))) -return ktime_get (); +unsigned long flags; +ktime_t ret; -while (!test_bit (DMA_FENCE_FLAG_TIMESTAMP_BIT , &fence->flags)) -cpu_relax (); +dma_fence_lock_irqsave (fence, flags); +if (WARN_ON (!dma_fence_test_signaled_flag (fence))) + ret =ktime_get (); +else + ret = fence->timestamp; +dma_fence_unlock_irqrestore (fence, flags); -return fence->timestamp; +return ret; } signed long dma_fence_wait_timeout(struct dma_fence *,
Re: [PATCH 1/3] dma-buf/dma-fence: Ensure correct ordering in dma_fence_timestamp()
Quoted 9 lines · Christian König
Quoted 8 lines · Philipp Stanner
dma_fence_timestamp()tests two fence flags withtest_bit(), not providing any ordering.The while loop +
cpu_relax()seem to satisfy the desire to avoid locking, effectively rebuilding a locks spinning behavior in the rare case that a fence is signaling right now. Contrary to proper locking, however, correct memory ordering is not guaranteed.
Quoted 4 lines
Quoted 1 line
Ensure correct ordering with locking.
My words when I tried to clean that up the first time: "Another alternative would be to grab the spinlock when that happens."
:)
Quoted 3 lines
I don't remember the story why the busy loop was used and can't see any reasons why it wasn't done this way in the first place.
I reached a point where I don't even want to know the story. Whatever the reason is, it will be something insane <.<
Quoted 5 lines
But my suggestion is to completely drop the
DMA_FENCE_FLAG_TIMESTAMP_BITas well. As far as I can tell that was only added as indicator that the timestamp is set, but we do that now for all fences anyway.
Agreed, but the API has been ossified by amdgpu who again violates layering with drm_sched and by i915, who does something extremely weird in intel_breadcrumbs.c. They seem to have built their own fence there.
amdgpu should be portable though.
P.
Philipp Stanner
maintainerPATCH 2/3 dma-buf: dma-fence: Add dma_fence_overwrite_timestamp()
In reply to Philipp StannerA fence's timestamp is just an opportunistic data point for debugging information etc. drm_sched, for example, uses the timestamps for its scheduling algorithms.
In rare situations it can make sense to allow users to overwrite a timestamp with a more accurate value. This is relatively harmless since the timestamp has no meaning for the implementation itself.
Add dma_fence_overwrite_timestamp().
--- drivers/dma-buf/dma-fence.c | 28 ++++++++++++++++++++++++++++ include/linux/dma-fence.h | 1 + 2 files changed, 29 insertions(+)
drivers/dma-buf/dma-fence.c +28 -0
@@ -412,6 +412,34 @@ void dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp) }EXPORT_SYMBOL (dma_fence_signal_timestamp); +/** +* dma_fence_overwrite_timestamp - overwrite a fence's timestamp +* @fence: the fence whose timestamp shall be overwritten +* @timestamp: the timestamp +* +* A dma_fence's timestamp is merely an opportunistically evaluated value to +* obtain information such as scheduling granularity. In some situations it can +* make sense to overwrite with a custom value. This function provides said +* functionality with the correct memory ordering. +* +* Return: 0 on success, -EAGAIN if the fence was not yet signaled. +*/ +int dma_fence_overwrite_timestamp (struct dma_fence *fence,ktime_t timestamp) +{ +unsigned long flags; +int ret =0 ; + +dma_fence_lock_irqsave (fence, flags); +if (dma_fence_test_signaled_flag (fence)) + fence->timestamp = timestamp; +else + ret = -EAGAIN ; +dma_fence_unlock_irqrestore (fence, flags); + +return ret; +} +EXPORT_SYMBOL (dma_fence_overwrite_timestamp); +/** * dma_fence_signal_locked - signal completion of a fence * @fence: the fence to signal
include/linux/dma-fence.h +1 -0
@@ -447,6 +447,7 @@ bool dma_fence_check_and_signal_locked(struct dma_fence *fence);void dma_fence_signal_locked (struct dma_fence *fence);void dma_fence_signal_timestamp (struct dma_fence *fence,ktime_t timestamp);void dma_fence_signal_timestamp_locked (struct dma_fence *fence,ktime_t timestamp); +int dma_fence_overwrite_timestamp (struct dma_fence *fence,ktime_t timestamp);signed long dma_fence_default_wait (struct dma_fence *fence,bool intr,signed long timeout); int dma_fence_add_callback(struct dma_fence *fence,
Signature · 1 line
2.56.0
Christian König
maintainerRe: [PATCH 2/3] dma-buf: dma-fence: Add dma_fence_overwrite_timestamp()
Quoted 9 lines · Philipp Stanner
A fence's timestamp is just an opportunistic data point for debugging information etc.
drm_sched, for example, uses the timestamps for its scheduling algorithms.In rare situations it can make sense to allow users to overwrite a timestamp with a more accurate value. This is relatively harmless since the timestamp has no meaning for the implementation itself.
Add
dma_fence_overwrite_timestamp().
I think that this is a very bad idea since it gives the impression that overwriting the timestamp on a fence is ok.
That was only done on the scheduled dma-fence because nobody cares about that time in the uAPI. For all other cases that can blow up.
I would rather come up with a clean approach for the scheduler to track the spend time per entity which would make that hack completely superfluous.
IIRC Tvrtko already created patches for that, I'm just not sure if they were ever committed.
Regards,
Christian.
Quoted 59 lines
Signed-off-by Philipp Stanner <[email protected]>--- drivers/dma-buf/dma-fence.c | 28 ++++++++++++++++++++++++++++ include/linux/dma-fence.h | 1 + 2 files changed, 29 insertions(+)drivers/dma-buf/dma-fence.c +28
-0@@ -412,6 +412,34 @@ void dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp) }EXPORT_SYMBOL (dma_fence_signal_timestamp); +/** +* dma_fence_overwrite_timestamp - overwrite a fence's timestamp +* @fence: the fence whose timestamp shall be overwritten +* @timestamp: the timestamp +* +* A dma_fence's timestamp is merely an opportunistically evaluated value to +* obtain information such as scheduling granularity. In some situations it can +* make sense to overwrite with a custom value. This function provides said +* functionality with the correct memory ordering. +* +* Return: 0 on success, -EAGAIN if the fence was not yet signaled. +*/ +int dma_fence_overwrite_timestamp (struct dma_fence *fence,ktime_t timestamp) +{ +unsigned long flags; +int ret =0 ; + +dma_fence_lock_irqsave (fence, flags); +if (dma_fence_test_signaled_flag (fence)) + fence->timestamp = timestamp; +else + ret = -EAGAIN ; +dma_fence_unlock_irqrestore (fence, flags); + +return ret; +} +EXPORT_SYMBOL (dma_fence_overwrite_timestamp); +/** * dma_fence_signal_locked - signal completion of a fence * @fence: the fence to signal include/linux/dma-fence.h +1
-0@@ -447,6 +447,7 @@ bool dma_fence_check_and_signal_locked(struct dma_fence *fence);void dma_fence_signal_locked (struct dma_fence *fence);void dma_fence_signal_timestamp (struct dma_fence *fence,ktime_t timestamp);void dma_fence_signal_timestamp_locked (struct dma_fence *fence,ktime_t timestamp); +int dma_fence_overwrite_timestamp (struct dma_fence *fence,ktime_t timestamp);signed long dma_fence_default_wait (struct dma_fence *fence,bool intr,signed long timeout); int dma_fence_add_callback(struct dma_fence *fence,
Re: [PATCH 2/3] dma-buf: dma-fence: Add dma_fence_overwrite_timestamp()
Quoted 14 lines · Christian König
Quoted 9 lines · Philipp Stanner
A fence's timestamp is just an opportunistic data point for debugging information etc.
drm_sched, for example, uses the timestamps for its scheduling algorithms.In rare situations it can make sense to allow users to overwrite a timestamp with a more accurate value. This is relatively harmless since the timestamp has no meaning for the implementation itself.
Add
dma_fence_overwrite_timestamp().I think that this is a very bad idea since it gives the impression that overwriting the timestamp on a fence is ok.
Quoted 6 lines
That was only done on the scheduled dma-fence because nobody cares about that time in the uAPI. For all other cases that can blow up.
I would rather come up with a clean approach for the scheduler to track the spend time per entity which would make that hack completely superfluous.
Yes, I agree.
The *one* feature we would have needed in C is a public-private system. With structs in APIs it's often completely undefined what the user can and cannot touch.
That led to API ossification. Just look at drm_sched, where we now cannot trivially change API internals anymore because external parties conveniently use this or that.
Quoted 2 lines
IIRC Tvrtko already created patches for that, I'm just not sure if they were ever committed.
I suppose it should be somewhat easy to track that data in drm_sched_fence.
Maybe Tvrtko can point me to his ideas so I can take a look.
Thx
P.
Philipp Stanner
maintainerPATCH 3/3 drm/sched: Fix memory ordering with dma_fence_overwrite_timestamp()
In reply to Philipp StannerThere is a memory ordering bug in drm_sched_get_finished_job(). The manual timestamp flag check with test_bit() is not ordered.
The dma_fence framework now provides dma_fence_overwrite_timestamp(), a function to overwrite a timestamp in a safe manner with correct memory ordering.
Solve the bug with dma_fence_overwrite_timestamp().
--- drivers/gpu/drm/scheduler/sched_main.c | 28 +++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-)
drivers/gpu/drm/scheduler/sched_main.c +23 -5
@@ -66,6 +66,7 @@* This implies waiting for previously executed jobs. */ +#include <linux/dma-fence.h> #include <linux/export.h> #include <linux/wait.h> #include <linux/sched.h>
@@ -898,6 +899,22 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched)return IS_ERR (entity) ?NULL : entity; } +/** +* drm_sched_overwrite_scheduled_timestamp - make timestamp more accurate +* +* Overwrites the timestamp of the provided scheduled fence with the one from +* the finished fence, if the scheduled fence is already signaled. Does nothing +* otherwise. +*/ +static void +drm_sched_overwrite_scheduled_timestamp (struct dma_fence *finished, +struct dma_fence *scheduled) +{ +ktime_t timestamp =dma_fence_timestamp (finished); + +dma_fence_overwrite_timestamp (scheduled, timestamp); +} +/** * drm_sched_get_finished_job - fetch the next finished job to be destroyed *
@@ -928,11 +945,12 @@ drm_sched_get_finished_job(struct drm_gpu_scheduler *sched) next =list_first_entry_or_null (&sched->pending_list,typeof (*next), list);if (next) { -/* make the scheduled timestamp more accurate */ -if (test_bit (DMA_FENCE_FLAG_TIMESTAMP_BIT , - &next->s_fence->scheduled.flags)) - next->s_fence->scheduled.timestamp = -dma_fence_timestamp (&job->s_fence->finished); +/* Try to make the scheduled timestamp more accurate. */ +drm_sched_overwrite_scheduled_timestamp ( + &job->s_fence->finished, + &next->s_fence->scheduled + ); +/* start TO timer for next job */ drm_sched_start_timeout (sched);
Signature · 1 line
2.56.0
[email protected]
AI reviewRe: [PATCH 3/3] drm/sched: Fix memory ordering with dma_fence_overwrite_timestamp()
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider: - [Low] drm/sched: missing kernel-doc for drm_sched_overwrite_scheduled_timestamp() --
Thank you for your contribution! Sashiko AI review found 1 potential issue to consider:
- [Low] drm/sched: missing kernel-doc for drm_sched_overwrite_scheduled_timestamp()
--
commit 4813d16b1085e58469356c4e77e1cbeb576347c5
Author: Philipp Stanner <[email protected]>
drm/sched: Fix memory ordering with dma_fence_overwrite_timestamp()
This commit replaces the manual timestamp flag check and assignment in drm_sched_get_finished_job() with dma_fence_overwrite_timestamp() via a new drm_sched_overwrite_scheduled_timestamp() helper.
Quoted 4 lines
drivers/gpu/drm/scheduler/sched_main.c +0
-0
[ ... ]
Quoted 9 lines
drivers/gpu/drm/scheduler/sched_main.c +5
-0@@ -898,6 +899,22 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched)return IS_ERR (entity) ?NULL : entity; } +/** +* drm_sched_overwrite_scheduled_timestamp - make timestamp more accurate +* +* Overwrites the timestamp of the provided scheduled fence with the one from +* the finished fence, if the scheduled fence is already signaled. Does nothing
drivers/gpu/drm/scheduler/sched_main.c drm_sched_select_entity(struct drm_gpu_scheduler *sched) · line 916
Quoted 10 lines
drivers/gpu/drm/scheduler/sched_main.c +10
-0+* otherwise. +*/ +static void +drm_sched_overwrite_scheduled_timestamp (struct dma_fence *finished, +struct dma_fence *scheduled) +{ +ktime_t timestamp =dma_fence_timestamp (finished); + +dma_fence_overwrite_timestamp (scheduled, timestamp); +}
[Severity: Low]
This kernel-doc comment omits @finished and @scheduled descriptions, which causes scripts/kernel-doc warnings when building with W=1 or generating documentation.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3