Independence for dma_fences! v6
Subsystems (10)
Christian König
reviewerHi everyone,
dma_fences have ever lived under the tyranny dictated by the module lifetime of their issuer, leading to crashes should anybody still holding a reference to a dma_fence when the module of the issuer was unloaded.
The basic problem is that when buffer are shared between drivers dma_fence objects can leak into external drivers and stay there even after they are signaled. The dma_resv object for example only lazy releases dma_fences.
So what happens is that when the module who originally created the dma_fence unloads the dma_fence_ops function table becomes unavailable as well and so any attempt to release the fence crashes the system.
Previously various approaches have been discussed, including changing the locking semantics of the dma_fence callbacks (by me) as well as using the drm scheduler as intermediate layer (by Sima) to disconnect dma_fences from their actual users, but none of them are actually solving all problems.
Tvrtko did some really nice prerequisite work by protecting the returned strings of the dma_fence_ops by RCU. This way dma_fence creators where able to just wait for an RCU grace period after fence signaling before they could be save to free those data structures.
Now this patch set here goes a step further and protects the whole dma_fence_ops structure by RCU, so that after the fence signals the pointer to the dma_fence_ops is set to NULL when there is no wait nor release callback given. All functionality which use the dma_fence_ops reference are put inside an RCU critical section, except for the deprecated issuer specific wait and of course the optional release callback.
Additional to the RCU changes the lock protecting the dma_fence state previously had to be allocated external. This set here now changes the functionality to make that external lock optional and allows dma_fences to use an inline lock and be self contained.
v4:
Rebases the whole set on upstream changes, especially the cleanup from Philip in patch "drm/amdgpu: independence for the amdkfd_fence!".
Adding two patches which brings the DMA-fence self tests up to date. The first selftest changes removes the mock_wait and so actually starts testing the default behavior instead of some hacky implementation in the test. This one got upstreamed independent of this set.
The second drops the mock_fence as well and tests the new RCU and inline spinlock functionality.
v5:
Rebase on top of drm-misc-next instead of drm-tip, leave out all driver changes for now since those should go through the driver specific paths anyway.
Address a few more review comments, especially some rebase mess and typos. And finally fix one more bug found by AMDs CI system.
v6:
Minor style changes, re-ordered patch #1, dropped the scheduler fence change for now
Please review and comment,
Christian.
Christian König
reviewerPATCH 1/9 dma-buf: add dma_fence_was_initialized function v2
Some driver use fence->ops to test if a fence was initialized or not. The problem is that this utilizes internal behavior of the dma_fence implementation.
So better abstract that into a function.
v2: use a flag instead of testing fence->ops, rename the function, move
to the beginning of the patch set.
--- drivers/dma-buf/dma-fence.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 13 +++++++------ drivers/gpu/drm/qxl/qxl_release.c | 2 +- include/linux/dma-fence.h | 15 +++++++++++++++ 4 files changed, 24 insertions(+), 8 deletions(-)
drivers/dma-buf/dma-fence.c +1 -1
@@ -1054,7 +1054,7 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, fence->lock = lock; fence->context = context; fence->seqno = seqno; - fence->flags = flags; + fence->flags = flags |BIT (DMA_FENCE_FLAG_INITIALIZED_BIT ); fence->error =0 ;trace_dma_fence_init (fence);
drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +7 -6
@@ -278,9 +278,10 @@ void amdgpu_job_free_resources(struct amdgpu_job *job)unsigned i;/* Check if any fences were initialized */ -if (job->base.s_fence && job->base.s_fence->finished.ops) +if (job->base.s_fence && +dma_fence_was_initialized (&job->base.s_fence->finished)) f = &job->base.s_fence->finished; -else if (job->hw_fence && job->hw_fence->base.ops) +else if (dma_fence_was_initialized (&job->hw_fence->base)) f = &job->hw_fence->base;else f =NULL ;
@@ -297,11 +298,11 @@ static void amdgpu_job_free_cb(struct drm_sched_job *s_job)amdgpu_sync_free (&job->explicit_sync); -if (job->hw_fence->base.ops) +if (dma_fence_was_initialized (&job->hw_fence->base))dma_fence_put (&job->hw_fence->base);else kfree (job->hw_fence); -if (job->hw_vm_fence->base.ops) +if (dma_fence_was_initialized (&job->hw_vm_fence->base))dma_fence_put (&job->hw_vm_fence->base);else kfree (job->hw_vm_fence);
@@ -335,11 +336,11 @@ void amdgpu_job_free(struct amdgpu_job *job)if (job->gang_submit != &job->base.s_fence->scheduled)dma_fence_put (job->gang_submit); -if (job->hw_fence->base.ops) +if (dma_fence_was_initialized (&job->hw_fence->base))dma_fence_put (&job->hw_fence->base);else kfree (job->hw_fence); -if (job->hw_vm_fence->base.ops) +if (dma_fence_was_initialized (&job->hw_vm_fence->base))dma_fence_put (&job->hw_vm_fence->base);else kfree (job->hw_vm_fence);
drivers/gpu/drm/qxl/qxl_release.c +1 -1
@@ -146,7 +146,7 @@ qxl_release_free(struct qxl_device *qdev,idr_remove (&qdev->release_idr, release->id);spin_unlock (&qdev->release_idr_lock); -if (release->base.ops) { +if (dma_fence_was_initialized (&release->base)) {WARN_ON (list_empty (&release->bos));qxl_release_free_list (release);
include/linux/dma-fence.h +15 -0
@@ -48,6 +48,7 @@ struct seq_file;* atomic ops (bit_*), so taking the spinlock will not be needed most * of the time. * +* DMA_FENCE_FLAG_INITIALIZED_BIT - fence was initialized * DMA_FENCE_FLAG_SIGNALED_BIT - fence is already signaled * DMA_FENCE_FLAG_TIMESTAMP_BIT - timestamp recorded for fence signaling * DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT - enable_signaling might have been called
@@ -98,6 +99,7 @@ struct dma_fence { };enum dma_fence_flag_bits { +DMA_FENCE_FLAG_INITIALIZED_BIT ,DMA_FENCE_FLAG_SEQNO64_BIT ,DMA_FENCE_FLAG_SIGNALED_BIT ,DMA_FENCE_FLAG_TIMESTAMP_BIT ,
@@ -263,6 +265,19 @@ void dma_fence_release(struct kref *kref);void dma_fence_free (struct dma_fence *fence);void dma_fence_describe (struct dma_fence *fence,struct seq_file *seq); +/** +* dma_fence_was_initialized - test if fence was initialized +* @fence: fence to test +* +* Return: True if fence was ever initialized, false otherwise. Works correctly +* only when memory backing the fence structure is zero initialized on +* allocation. +*/ +static inline bool dma_fence_was_initialized (struct dma_fence *fence) +{ +return fence &&test_bit (DMA_FENCE_FLAG_INITIALIZED_BIT , &fence->flags); +} +/** * dma_fence_put - decreases refcount of the fence * @fence: fence to reduce refcount of
Signature · 1 line
2.43.0
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 118 lines · Christian König
Some driver use
fence->opsto test if a fence was initialized or not. The problem is that this utilizes internal behavior of thedma_fenceimplementation.So better abstract that into a function.
v2: use a flag instead of testing
fence->ops, rename the function, move
to the beginning of the patch set.Signed-off-by Christian König <[email protected]>--- drivers/dma-buf/dma-fence.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 13 +++++++------ drivers/gpu/drm/qxl/qxl_release.c | 2 +- include/linux/dma-fence.h | 15 +++++++++++++++ 4 files changed, 24 insertions(+), 8 deletions(-)drivers/dma-buf/dma-fence.c +1
-1@@ -1054,7 +1054,7 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, fence->lock = lock; fence->context = context; fence->seqno = seqno; - fence->flags = flags; + fence->flags = flags |BIT (DMA_FENCE_FLAG_INITIALIZED_BIT ); fence->error =0 ;trace_dma_fence_init (fence);drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +7
-6@@ -278,9 +278,10 @@ void amdgpu_job_free_resources(struct amdgpu_job *job)unsigned i;/* Check if any fences were initialized */ -if (job->base.s_fence && job->base.s_fence->finished.ops) +if (job->base.s_fence && +dma_fence_was_initialized (&job->base.s_fence->finished)) f = &job->base.s_fence->finished; -else if (job->hw_fence && job->hw_fence->base.ops) +else if (dma_fence_was_initialized (&job->hw_fence->base)) f = &job->hw_fence->base;else f =NULL ;@@ -297,11 +298,11 @@ static void amdgpu_job_free_cb(struct drm_sched_job *s_job)amdgpu_sync_free (&job->explicit_sync); -if (job->hw_fence->base.ops) +if (dma_fence_was_initialized (&job->hw_fence->base))dma_fence_put (&job->hw_fence->base);else kfree (job->hw_fence); -if (job->hw_vm_fence->base.ops) +if (dma_fence_was_initialized (&job->hw_vm_fence->base))dma_fence_put (&job->hw_vm_fence->base);else kfree (job->hw_vm_fence);@@ -335,11 +336,11 @@ void amdgpu_job_free(struct amdgpu_job *job)if (job->gang_submit != &job->base.s_fence->scheduled)dma_fence_put (job->gang_submit); -if (job->hw_fence->base.ops) +if (dma_fence_was_initialized (&job->hw_fence->base))dma_fence_put (&job->hw_fence->base);else kfree (job->hw_fence); -if (job->hw_vm_fence->base.ops) +if (dma_fence_was_initialized (&job->hw_vm_fence->base))dma_fence_put (&job->hw_vm_fence->base);else kfree (job->hw_vm_fence);drivers/gpu/drm/qxl/qxl_release.c +1
-1@@ -146,7 +146,7 @@ qxl_release_free(struct qxl_device *qdev,idr_remove (&qdev->release_idr, release->id);spin_unlock (&qdev->release_idr_lock); -if (release->base.ops) { +if (dma_fence_was_initialized (&release->base)) {WARN_ON (list_empty (&release->bos));qxl_release_free_list (release);include/linux/dma-fence.h +8
-0@@ -48,6 +48,7 @@ struct seq_file;* atomic ops (bit_*), so taking the spinlock will not be needed most * of the time. * +* DMA_FENCE_FLAG_INITIALIZED_BIT - fence was initialized * DMA_FENCE_FLAG_SIGNALED_BIT - fence is already signaled * DMA_FENCE_FLAG_TIMESTAMP_BIT - timestamp recorded for fence signaling * DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT - enable_signaling might have been called @@ -98,6 +99,7 @@ struct dma_fence { };enum dma_fence_flag_bits { +DMA_FENCE_FLAG_INITIALIZED_BIT ,DMA_FENCE_FLAG_SEQNO64_BIT ,DMA_FENCE_FLAG_SIGNALED_BIT ,DMA_FENCE_FLAG_TIMESTAMP_BIT ,@@ -263,6 +265,19 @@ void dma_fence_release(struct kref *kref);void dma_fence_free (struct dma_fence *fence);void dma_fence_describe (struct dma_fence *fence,struct seq_file *seq); +/** +* dma_fence_was_initialized - test if fence was initialized +* @fence: fence to test +* +* Return: True if fence was ever initialized, false otherwise. Works correctly +* only when memory backing the fence structure is zero initialized on
include/linux/dma-fence.h void dma_fence_release(struct kref *kref); · line 283
Quoted 10 lines
include/linux/dma-fence.h +7
-0+* allocation. +*/ +static inline bool dma_fence_was_initialized (struct dma_fence *fence) +{ +return fence &&test_bit (DMA_FENCE_FLAG_INITIALIZED_BIT , &fence->flags); +} +/** * dma_fence_put - decreases refcount of the fence * @fence: fence to reduce refcount of
Regards,
Tvrtko
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
In reply to Christian KönigHi,
Quoted 8 lines · Christian König
Some driver use
fence->opsto test if a fence was initialized or not. The problem is that this utilizes internal behavior of thedma_fenceimplementation.So better abstract that into a function.
v2: use a flag instead of testing
fence->ops, rename the function, move
to the beginning of the patch set.
...
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 152
Quoted 11 lines
drivers/gpu/drm/qxl/qxl_release.c +1
-1@@ -146,7 +146,7 @@ qxl_release_free(struct qxl_device *qdev,idr_remove (&qdev->release_idr, release->id);spin_unlock (&qdev->release_idr_lock); -if (release->base.ops) { +if (dma_fence_was_initialized (&release->base)) {WARN_ON (list_empty (&release->bos));qxl_release_free_list (release);
This likely breaks qxl:
https://bugzilla.suse.com/show_bug.cgi?id=1271081
Quoted 23 lines
refcount_t: underflow; use-after-free.WARNING: lib/refcount.c:28 at refcount_warn_saturate+0x59/0x90, CPU#0: kworker/0:0/1534 Modules linked in: af_packet nft_fib_inet ... CPU: 0 UID: 0 PID: 1534 Comm: kworker/0:0 Not tainted 7.1.3-1-default #1 PREEMPT(full) openSUSE Tumbleweed b041a6527f6e58424f4cd3de0fade8d408b378fd Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-2-gc13ff2cd-prebuilt.qemu.org 04/01/2014 Workqueue: events qxl_gc_work [qxl] RIP: 0010:refcount_warn_saturate+0x59/0x90 Code: 44 48 8d 3d a9 72 b7 01 67 48 0f b9 3a c3 cc cc cc cc 48 8d 3d a8 72 b7 01 67 48 0f b9 3a c3 cc cc cc cc 48 8d 3d a7 72 b7 01 <67> 48 0f b9 3a c3 cc cc cc cc 48 8d 3d a6 72 b7 01 67 48 0f b9 3a RSP: 0018:ffffce09019efe00 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8b5a56f819c0 RCX: 0000000000000017 RDX: 0000000000011696 RSI: 0000000000000003 RDI: ffffffffbc953b70 RBP: ffff8b5a4a314000 R08: 0000000000000001 R09: ffffffffc07d488a R10: fffffa6fc4787d80 R11: ffff8b5a40044000 R12: 0000000000000004 R13: 0000000000000003 R14: ffffce09003f1434 R15: ffff8b5a410c4640 FS: 0000000000000000(0000) GS:ffff8b5afe954000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ff463d65000 CR3: 000000012b476004 CR4: 0000000000772ef0 PKRU: 55555554 Call Trace: <TASK> qxl_release_free+0xee/0xf0 [qxl d93e9381353e619799d56790f5f8dda6cce491f6] qxl_garbage_collect+0xd1/0x1b0 [qxl d93e9381353e619799d56790f5f8dda6cce491f6] process_one_work+0x19e/0x3a0
Quoted 1 line
...
The function does now:
Quoted 5 lines
void
qxl_release_free(structqxl_device*qdev,
structqxl_release*release)
{
...
Quoted 10 lines
if (dma_fence_was_initialized(&release->base)) {
WARN_ON(list_empty(&release->bos));
qxl_release_free_list(release);
dma_fence_signal(&release->base);
dma_fence_put(&release->base); refcount_dec_and_test() in kref_put() in this ^^ crashes. Apparently, the reference count was not increased anywhere. QXL does not call dma_fence_get() -- who is supposed to increase the refcount. And why it did not fail before?
Quoted 11 lines
} else { qxl_release_free_list(release); kfree(release); } atomic_dec(&qdev->release_count);}
thanks,
Signature · 2 lines
js suse labs
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 12 lines · Jiri Slaby
Hi,
Quoted 8 lines · Christian König
Some driver use
fence->opsto test if a fence was initialized or not. The problem is that this utilizes internal behavior of thedma_fenceimplementation.So better abstract that into a function.
v2: use a flag instead of testing
fence->ops, rename the function, move to the beginning of the patch set....
Quoted 8 lines
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 149
Quoted 8 lines
drivers/gpu/drm/qxl/qxl_release.c +0
-0@@ -146,7 +146,7 @@ qxl_release_free(struct qxl_device *qdev, idr_remove(&qdev->release_idr, release->id); spin_unlock(&qdev->release_idr_lock);drivers/gpu/drm/qxl/qxl_release.c +1
-1-if (release->base.ops) { +if (dma_fence_was_initialized (&release->base)) {
Could you verify the cause with sth like
if (release->base.ops && dma_fence_was_initialized(…)) {
?
Quoted 57 lines
Quoted 3 lines
WARN_ON(list_empty(&release->bos));
qxl_release_free_list(release);This likely breaks qxl:
https://bugzilla.suse.com/show_bug.cgi?id=1271081>
refcount_t: underflow; use-after-free.
> WARNING:lib/refcount.c:28atrefcount_warn_saturate+0x59/0x90, CPU#0: kworker/0:0/1534
> Modules linked in:af_packetnft_fib_inet...
> CPU: 0 UID: 0 PID: 1534 Comm: kworker/0:0 Not tainted 7.1.3-1-default #1 PREEMPT(full) openSUSE Tumbleweedb041a6527f6e58424f4cd3de0fade8d408b378fd
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-2-gc13ff2cd-prebuilt.qemu.org 04/01/2014
> Workqueue: eventsqxl_gc_work[qxl]
> RIP: 0010:refcount_warn_saturate+0x59/0x90
> Code: 44 48 8d 3d a9 72 b7 01 67 48 0f b9 3a c3 cc cc cc cc 48 8d 3d a8 72 b7 01 67 48 0f b9 3a c3 cc cc cc cc 48 8d 3d a7 72 b7 01 <67> 48 0f b9 3a c3 cc cc cc cc 48 8d 3d a6 72 b7 01 67 48 0f b9 3a > RSP: 0018:ffffce09019efe00 EFLAGS: 00010246
> RAX: 0000000000000000 RBX:ffff8b5a56f819c0RCX: 0000000000000017 > RDX: 0000000000011696 RSI: 0000000000000003 RDI:ffffffffbc953b70> RBP:ffff8b5a4a314000R08: 0000000000000001 R09:ffffffffc07d488a> R10:fffffa6fc4787d80R11:ffff8b5a40044000R12: 0000000000000004 > R13: 0000000000000003 R14:ffffce09003f1434R15:ffff8b5a410c4640> FS: 0000000000000000(0000) GS:ffff8b5afe954000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2:00007ff463d65000CR3:000000012b476004CR4:0000000000772ef0> PKRU: 55555554
> Call Trace:
> <TASK>
>qxl_release_free+0xee/0xf0[qxld93e9381353e619799d56790f5f8dda6cce491f6]
>qxl_garbage_collect+0xd1/0x1b0[qxld93e9381353e619799d56790f5f8dda6cce491f6]
>process_one_work+0x19e/0x3a0
> ...The function does now:
> void
> qxl_release_free(structqxl_device*qdev,
> structqxl_release*release)
> {
> ...
> if (dma_fence_was_initialized(&release->base)) { > WARN_ON(list_empty(&release->bos));>
qxl_release_free_list(release);>
>
dma_fence_signal(&release->base);
Quoted 8 lines
>
dma_fence_put(&release->base);
refcount_dec_and_test()inkref_put()in this ^^ crashes. Apparently, the reference count was not increased anywhere. QXL does not calldma_fence_get()-- who is supposed to increase the refcount.
dma_fence_init() initializes the refcount to 1. So since there is only one place with dma_fence_put(), it probably aims at countering that.
Quoted 2 lines
And why it
did not fail before?
My guess would be that base->ops was used as a boolean to see whether qxl_release contains a valid fence. It seems qxl_release_free() is intended as some sort of universal function; it is called at many places. So I would suppose it is sometimes being called without the fence being initialized.
Should most certainly have been documented. Anyways, verifying the issue would probably be cool
Cheers,
P.
Quoted 15 lines
> } else {
>
qxl_release_free_list(release);> kfree(release);
> }
>
atomic_dec(&qdev->release_count);> }
thanks,
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 24 lines · Philipp Stanner
Quoted 12 lines · Jiri Slaby
Hi,
Quoted 8 lines · Christian König
Some driver use
fence->opsto test if a fence was initialized or not. The problem is that this utilizes internal behavior of thedma_fenceimplementation.So better abstract that into a function.
v2: use a flag instead of testing
fence->ops, rename the function, move to the beginning of the patch set....
Quoted 8 lines
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 149
Quoted 8 lines
drivers/gpu/drm/qxl/qxl_release.c +0
-0@@ -146,7 +146,7 @@ qxl_release_free(struct qxl_device *qdev, idr_remove(&qdev->release_idr, release->id); spin_unlock(&qdev->release_idr_lock);drivers/gpu/drm/qxl/qxl_release.c +1
-1-if (release->base.ops) { +if (dma_fence_was_initialized (&release->base)) {Could you verify the cause with sth like
Quoted 1 line
if (
release->base.ops&&dma_fence_was_initialized(…)) {
Forget about that, probably would not work or cause other issues because the NULL-setting on signal().
I'd then probably try to verify it with a separate boolean in struct release. Though I also don't get why the fence-initialized check does not do the trick.
P.
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 25 lines · Philipp Stanner
Quoted 24 lines · Philipp Stanner
Quoted 12 lines · Jiri Slaby
Hi,
Quoted 8 lines · Christian König
Some driver use
fence->opsto test if a fence was initialized or not. The problem is that this utilizes internal behavior of thedma_fenceimplementation.So better abstract that into a function.
v2: use a flag instead of testing
fence->ops, rename the function, move
to the beginning of the patch set....
Quoted 8 lines
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 149
Quoted 8 lines
drivers/gpu/drm/qxl/qxl_release.c +1
-1@@ -146,7 +146,7 @@ qxl_release_free(struct qxl_device *qdev, idr_remove(&qdev->release_idr, release->id); spin_unlock(&qdev->release_idr_lock); -if (release->base.ops) { +if (dma_fence_was_initialized (&release->base)) {Could you verify the cause with sth like
Quoted 8 lines
Quoted 1 line
if (
release->base.ops&&dma_fence_was_initialized(…)) {Forget about that, probably would not work or cause other issues because the NULL-setting on
signal().I'd then probably try to verify it with a separate boolean in struct release. Though I also don't get why the fence-initialized check does not do the trick.
Hi, have you come up with something yet?
thanks,
Signature · 2 lines
js suse labs
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 28 lines · Jiri Slaby
Quoted 27 lines · Philipp Stanner
Quoted 25 lines · Philipp Stanner
Quoted 12 lines · Jiri Slaby
Hi,
Quoted 8 lines · Christian König
Some driver use
fence->opsto test if a fence was initialized or not. The problem is that this utilizes internal behavior of thedma_fenceimplementation.So better abstract that into a function.
v2: use a flag instead of testing
fence->ops, rename the function, move to the beginning of the patch set....
Quoted 8 lines
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 149
Quoted 8 lines
drivers/gpu/drm/qxl/qxl_release.c +0
-0@@ -146,7 +146,7 @@ qxl_release_free(struct qxl_device *qdev, idr_remove(&qdev->release_idr, release->id); spin_unlock(&qdev->release_idr_lock);drivers/gpu/drm/qxl/qxl_release.c +1
-1-if (release->base.ops) { +if (dma_fence_was_initialized (&release->base)) {Could you verify the cause with sth like
if (
release->base.ops&&dma_fence_was_initialized(…)) {
Quoted 8 lines
Quoted 6 lines
Forget about that, probably would not work or cause other issues because the NULL-setting on
signal().I'd then probably try to verify it with a separate boolean in struct release. Though I also don't get why the fence-initialized check does not do the trick.
Hi, have you come up with something yet?
Well, I was more like suggesting this as a debug option [to you] :D
Regardless, looking at the code again, I would say that this might be a race, but I don't know enough about QXL to say for sure.
dma_fence_init() is (of course) not ordered:
static void
__dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,
spinlock_t *lock, u64 context, u64 seqno, unsigned long flags)
{
BUG_ON(!ops || !ops->get_driver_name || !ops->get_timeline_name);
kref_init(&fence->refcount);
/*
* While it is counter intuitive to protect a constant function pointer
* table by RCU it allows modules to wait for an RCU grace period
* before they unload, to make sure that nobody is executing their
* functions any more.
*/
RCU_INIT_POINTER(fence->ops, ops);
INIT_LIST_HEAD(&fence->cb_list);
fence->context = context;
fence->seqno = seqno;
fence->flags = flags | BIT(DMA_FENCE_FLAG_INITIALIZED_BIT);(Should this maybe be set_bit() btw?)
The fact that QXL could run into qxl_release_free() with an uninitialized fence hints at the fact that this might race, so DMA_FENCE_FLAG_INITIALIZED_BIT could be set / read before kref_init() ran.
Maybe one way to verify / debug that would be to move spin_unlock(&qdev->release_idr_lock) downwards so it also guards dma_fence_was_initialized(), and also lock the initialization of the fence (in qxl_release_fence_buffer_objects() ?) with said lock.
If that's possible. Just brainstorming a bit for ways how to debug.
QXL does a few tricky things with the release->base.ops pointer. qxl_release_alloc() sets it to NULL, and only qxl_release_fence_buffer_objects() then actually sets it. So this could be the race? Setting of the ops pointer got replaced by setting of the fence-flag.
P.
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
On Wed, 2026-08-19 at 14:33 +0200, Philipp Stanner wrote:
[…]
Quoted 42 lines
Regardless, looking at the code again, I would say that this might be a race, but I don't know enough about QXL to say for sure.
dma_fence_init()is (of course) not ordered:static void
__dma_fence_init(structdma_fence*fence, const structdma_fence_ops*ops,
spinlock_t*lock, u64 context, u64 seqno, unsigned long flags){ BUG_ON(!ops || !ops->get_driver_name || !ops->get_timeline_name); kref_init(&fence->refcount); /* * While it is counter intuitive to protect a constant function pointer * table by RCU it allows modules to wait for an RCU grace period * before they unload, to make sure that nobody is executing their * functions any more. */ RCU_INIT_POINTER(fence->ops, ops); INIT_LIST_HEAD(&fence->cb_list); fence->context = context; fence->seqno = seqno; fence->flags = flags | BIT(DMA_FENCE_FLAG_INITIALIZED_BIT);(Should this maybe be
set_bit()btw?)The fact that QXL could run into
qxl_release_free()with an uninitialized fence hints at the fact that this might race, soDMA_FENCE_FLAG_INITIALIZED_BITcould be set / read beforekref_init()ran.Maybe one way to verify / debug that would be to move
spin_unlock(&qdev->release_idr_lock)downwards so it also guardsdma_fence_was_initialized(), and also lock the initialization of the fence (inqxl_release_fence_buffer_objects()?) with said lock.If that's possible. Just brainstorming a bit for ways how to debug.
Quoted 8 lines
QXL does a few tricky things with the
release->base.opspointer.qxl_release_alloc()sets it to NULL, and onlyqxl_release_fence_buffer_objects()then actually sets it. So this could be the race? Setting of the ops pointer got replaced by setting of the fence-flag.P.
Could you test something like this? (not even compile-tested, just an idea)
drivers/dma-buf/dma-fence.c +2 -1
@@ -1075,7 +1075,6 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,INIT_LIST_HEAD (&fence->cb_list); fence->context = context; fence->seqno = seqno; - fence->flags = flags |BIT (DMA_FENCE_FLAG_INITIALIZED_BIT );if (lock) { fence->extern_lock = lock; }else {
@@ -1084,6 +1083,8 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, } fence->error =0 ; +smp_mb (); + fence->flags = flags |BIT (DMA_FENCE_FLAG_INITIALIZED_BIT );trace_dma_fence_init (fence); }
include/linux/dma-fence.h +6 -1
@@ -292,7 +292,12 @@ void dma_fence_describe(struct dma_fence *fence, struct seq_file *seq);*/ static inline bool dma_fence_was_initialized (struct dma_fence *fence) { -return fence &&test_bit (DMA_FENCE_FLAG_INITIALIZED_BIT , &fence->flags); +bool init; + + init = fence &&test_bit (DMA_FENCE_FLAG_INITIALIZED_BIT , &fence->flags); +smp_mb (); + +return init; }/**
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 54 lines · Philipp Stanner
On Wed, 2026-08-19 at 14:33 +0200, Philipp Stanner wrote:
[…]
Quoted 49 lines
Regardless, looking at the code again, I would say that this might be a race, but I don't know enough about QXL to say for sure.
dma_fence_init()is (of course) not ordered:static void
__dma_fence_init(structdma_fence*fence, const structdma_fence_ops*ops,
spinlock_t*lock, u64 context, u64 seqno, unsigned long flags){ BUG_ON(!ops || !ops->get_driver_name || !ops->get_timeline_name); kref_init(&fence->refcount); /* * While it is counter intuitive to protect a constant function pointer * table by RCU it allows modules to wait for an RCU grace period * before they unload, to make sure that nobody is executing their * functions any more. */ RCU_INIT_POINTER(fence->ops, ops); INIT_LIST_HEAD(&fence->cb_list); fence->context = context; fence->seqno = seqno; fence->flags = flags | BIT(DMA_FENCE_FLAG_INITIALIZED_BIT);(Should this maybe be
set_bit()btw?)The fact that QXL could run into
qxl_release_free()with an uninitialized fence hints at the fact that this might race, soDMA_FENCE_FLAG_INITIALIZED_BITcould be set / read beforekref_init()ran.Maybe one way to verify / debug that would be to move
spin_unlock(&qdev->release_idr_lock)downwards so it also guardsdma_fence_was_initialized(), and also lock the initialization of the fence (inqxl_release_fence_buffer_objects()?) with said lock.If that's possible. Just brainstorming a bit for ways how to debug.
QXL does a few tricky things with the
release->base.opspointer.qxl_release_alloc()sets it to NULL, and onlyqxl_release_fence_buffer_objects()then actually sets it. So this could be the race? Setting of the ops pointer got replaced by setting of the fence-flag.
Quoted 3 lines
Quoted 1 line
P.
Could you test something like this? (not even compile-tested, just an idea)
It makes the system dead during early boot :P.
drivers/dma-buf/dma-fence.c __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,
Quoted 13 lines
drivers/dma-buf/dma-fence.c +0
-1@@ -1075,7 +1075,6 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,INIT_LIST_HEAD (&fence->cb_list); fence->context = context; fence->seqno = seqno; - fence->flags = flags |BIT (DMA_FENCE_FLAG_INITIALIZED_BIT );if (lock) { fence->extern_lock = lock; }else {@@ -1084,6 +1083,8 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,
This missing piece was here:
- fence->flags |= BIT(DMA_FENCE_FLAG_INLINE_LOCK_BIT);
+ flags |= BIT(DMA_FENCE_FLAG_INLINE_LOCK_BIT);
drivers/dma-buf/dma-fence.c __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, · line 1089
Quoted 7 lines
drivers/dma-buf/dma-fence.c +2
-0} fence->error =0 ; +smp_mb (); + fence->flags = flags |BIT (DMA_FENCE_FLAG_INITIALIZED_BIT );trace_dma_fence_init (fence); }
But it does not help either...
Quoted 18 lines
include/linux/dma-fence.h +6
-1@@ -292,7 +292,12 @@ void dma_fence_describe(struct dma_fence *fence, struct seq_file *seq);*/ static inline bool dma_fence_was_initialized (struct dma_fence *fence) { -return fence &&test_bit (DMA_FENCE_FLAG_INITIALIZED_BIT , &fence->flags); +bool init; + + init = fence &&test_bit (DMA_FENCE_FLAG_INITIALIZED_BIT , &fence->flags); +smp_mb (); + +return init; }/**
Signature · 2 lines
js suse labs
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 82 lines · Jiri Slaby
Quoted 54 lines · Philipp Stanner
On Wed, 2026-08-19 at 14:33 +0200, Philipp Stanner wrote:
[…]
Quoted 49 lines
Regardless, looking at the code again, I would say that this might be a race, but I don't know enough about QXL to say for sure.
dma_fence_init()is (of course) not ordered:static void
__dma_fence_init(structdma_fence*fence, const structdma_fence_ops*ops,spinlock_t*lock, u64 context, u64 seqno, unsigned long flags) {
BUG_ON(!ops || !ops->get_driver_name || !ops->get_timeline_name);kref_init(&fence->refcount);
/*
* While it is counter intuitive to protect a constant function pointer
* table by RCU it allows modules to wait for an RCU grace period
* before they unload, to make sure that nobody is executing their
* functions any more.
*/
RCU_INIT_POINTER(fence->ops, ops);INIT_LIST_HEAD(&fence->cb_list);
fence->context = context;
fence->seqno = seqno;
fence->flags = flags |BIT(DMA_FENCE_FLAG_INITIALIZED_BIT);(Should this maybe be
set_bit()btw?)The fact that QXL could run into
qxl_release_free()with an uninitialized fence hints at the fact that this might race, soDMA_FENCE_FLAG_INITIALIZED_BITcould be set / read beforekref_init()ran.Maybe one way to verify / debug that would be to move
spin_unlock(&qdev->release_idr_lock)downwards so it also guardsdma_fence_was_initialized(), and also lock the initialization of the fence (inqxl_release_fence_buffer_objects()?) with said lock.If that's possible. Just brainstorming a bit for ways how to debug.
QXL does a few tricky things with the
release->base.opspointer.qxl_release_alloc()sets it to NULL, and onlyqxl_release_fence_buffer_objects()then actually sets it. So this could be the race? Setting of the ops pointer got replaced by setting of the fence-flag.Quoted 3 lines
Quoted 1 line
P.
Could you test something like this? (not even compile-tested, just an idea)
It makes the system dead during early boot :P.
Quoted 14 lines
drivers/dma-buf/dma-fence.c +0
-1@@ -1075,7 +1075,6 @@ __dma_fence_init(struct dma_fence *fence, conststruct dma_fence_ops *ops, INIT_LIST_HEAD(&fence->cb_list); fence->context = context; fence->seqno = seqno; - fence->flags = flags | BIT(DMA_FENCE_FLAG_INITIALIZED_BIT); if (lock) { fence->extern_lock = lock;} else {
drivers/dma-buf/dma-fence.c __dma_fence_init(struct dma_fence *fence, const · line 1083
Quoted 1 line
drivers/dma-buf/dma-fence.c +0
-0@@ -1084,6 +1083,8 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,This missing piece was here:
-fence->flags|=BIT(DMA_FENCE_FLAG_INLINE_LOCK_BIT); + flags |=BIT(DMA_FENCE_FLAG_INLINE_LOCK_BIT);
Quoted 8 lines
drivers/dma-buf/dma-fence.c __dma_fence_init(struct dma_fence *fence, const · line 1089
Quoted 6 lines
drivers/dma-buf/dma-fence.c +2
-0} fence->error = 0; + smp_mb(); + fence->flags = flags | BIT(DMA_FENCE_FLAG_INITIALIZED_BIT); trace_dma_fence_init(fence); }But it does not help either...
What helps is indeed the revert back to:
drivers/gpu/drm/qxl/qxl_release.c +1 -1
@@ -147,7 +147,7 @@ qxl_release_free(struct qxl_device *qdev,idr_remove (&qdev->release_idr, release->id);spin_unlock (&qdev->release_idr_lock); -if (dma_fence_was_initialized (&release->base)) { +if (release->base.ops) {WARN_ON (list_empty (&release->bos));qxl_release_free_list (release);
Or the bool flag appears to help too:
drivers/gpu/drm/qxl/qxl_drv.h +1 -0
@@ -144,6 +144,7 @@ enum {#define QXL_MAX_RES 96struct qxl_release {struct dma_fence base; +bool uses_fence;int id;int type;
drivers/gpu/drm/qxl/qxl_release.c +3 -1
@@ -97,6 +97,7 @@ qxl_release_alloc(struct qxl_device *qdev, int type,return -ENOMEM ; } release->base.ops =NULL ; + release->uses_fence = false; release->type = type; release->release_offset =0 ; release->surface_release_id =0 ;
@@ -147,7 +148,7 @@ qxl_release_free(struct qxl_device *qdev,idr_remove (&qdev->release_idr, release->id);spin_unlock (&qdev->release_idr_lock); -if (dma_fence_was_initialized (&release->base)) { +if (release->uses_fence) {WARN_ON (list_empty (&release->bos));qxl_release_free_list (release);
@@ -431,6 +432,7 @@ void qxl_release_fence_buffer_objects(struct qxl_release *release)*/ dma_fence_init (&release->base, &qxl_fence_ops, &qdev->release_lock, release->id |0xf0000000 , release->base.seqno); + release->uses_fence = true;trace_dma_fence_emit (&release->base);list_for_each_entry (entry, &release->bos, list) {
thanks,
Signature · 2 lines
js suse labs
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 150
Quoted 10 lines · Jiri Slaby
What helps is indeed the revert back to:
drivers/gpu/drm/qxl/qxl_release.c +1
-1@@ -147,7 +147,7 @@ qxl_release_free(struct qxl_device *qdev, idr_remove(&qdev->release_idr, release->id); spin_unlock(&qdev->release_idr_lock); - if (dma_fence_was_initialized(&release->base)) { + if (release->base.ops) {
Crap, the flags simple got garbage from previous freed buffer, so:
- release = kmalloc(size, GFP_KERNEL);
+ release = kzalloc(size, GFP_KERNEL);
fixes this of course. Patch coming.
Now, the ops and initialized are either both set or unset:qxl_release_free 1: release->base.ops=0x0 initialized=0 id=6 type=2 qxl_release_free 1: release->base.ops=qxl_fence_ops initialized=1 id=2 type=0
Unlike before.
FTR release->base.ops were NULLed after that kmalloc. Unlike release->base.flags.
thanks,
Signature · 2 lines
js suse labs
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 23 lines · Jiri Slaby
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 150
Quoted 10 lines · Jiri Slaby
What helps is indeed the revert back to:
drivers/gpu/drm/qxl/qxl_release.c +1
-1@@ -147,7 +147,7 @@ qxl_release_free(struct qxl_device *qdev, idr_remove(&qdev->release_idr, release->id); spin_unlock(&qdev->release_idr_lock); - if (dma_fence_was_initialized(&release->base)) { + if (release->base.ops) {Crap, the flags simple got garbage from previous freed buffer, so:
- release =kmalloc(size, GFP_KERNEL);
+ release =kzalloc(size, GFP_KERNEL);fixes this of course. Patch coming.
Now, the ops and initialized are either both set or unset:
qxl_release_free1: release->base.ops=0x0 initialized=0 id=6 type=2qxl_release_free1: release->base.ops=qxl_fence_ops initialized=1 id=2 type=0
Quoted 4 lines
Unlike before.
FTR
release->base.opswere NULLed after that kmalloc. Unlikerelease->base.flags.
You're saying that QXL is re-using that fence if it were uninitialized?
P.
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 24 lines · Philipp Stanner
Quoted 23 lines · Jiri Slaby
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 150
Quoted 10 lines · Jiri Slaby
What helps is indeed the revert back to:
drivers/gpu/drm/qxl/qxl_release.c +1
-1@@ -147,7 +147,7 @@ qxl_release_free(struct qxl_device *qdev, idr_remove(&qdev->release_idr, release->id); spin_unlock(&qdev->release_idr_lock); - if (dma_fence_was_initialized(&release->base)) { + if (release->base.ops) {Crap, the flags simple got garbage from previous freed buffer, so:
- release =kmalloc(size, GFP_KERNEL);
+ release =kzalloc(size, GFP_KERNEL);fixes this of course. Patch coming.
Now, the ops and initialized are either both set or unset:
qxl_release_free1: release->base.ops=0x0 initialized=0 id=6 type=2qxl_release_free1: release->base.ops=qxl_fence_ops initialized=1 id=2 type=0
Quoted 6 lines
Quoted 4 lines
Unlike before.
FTR
release->base.opswere NULLed after that kmalloc. Unlikerelease->base.flags.You're saying that QXL is re-using that fence if it were uninitialized?
Sort of. The fence was dropped and freed. Then a new memory was allocated. And garbage from that allocation was used -- it might be the old (freed) fence. Or anything else freed previously. Simply garbage.
thanks,
Signature · 2 lines
js suse labs
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 27 lines · Jiri Slaby
Quoted 26 lines · Philipp Stanner
Quoted 25 lines · Jiri Slaby
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 150
Quoted 10 lines · Jiri Slaby
What helps is indeed the revert back to:
drivers/gpu/drm/qxl/qxl_release.c +1
-1@@ -147,7 +147,7 @@ qxl_release_free(struct qxl_device *qdev, idr_remove(&qdev->release_idr, release->id); spin_unlock(&qdev->release_idr_lock); - if (dma_fence_was_initialized(&release->base)) { + if (release->base.ops) {Crap, the flags simple got garbage from previous freed buffer, so:
- release =kmalloc(size, GFP_KERNEL);
+ release =kzalloc(size, GFP_KERNEL);fixes this of course. Patch coming.
Now, the ops and initialized are either both set or unset:
qxl_release_free1: release->base.ops=0x0 initialized=0 id=6 type=2qxl_release_free1: release->base.ops=qxl_fence_ops initialized=1 id=2 type=0Unlike before.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
FTR
release->base.opswere NULLed after that kmalloc. Unlikerelease->base.flags.You're saying that QXL is re-using that fence if it were uninitialized?
Sort of. The fence was dropped and freed. Then a new memory was allocated. And garbage from that allocation was used -- it might be the old (freed) fence. Or anything else freed previously. Simply garbage.
Maybe even dma_fence_init() should zero out the memory for maximum robustness.
We do that in drm_sched at a few places.
Opinions? @Christian?
P.
Re: [PATCH 1/9] dma-buf: add dma_fence_was_initialized function v2
Quoted 33 lines · Philipp Stanner
Quoted 32 lines · Jiri Slaby
Quoted 30 lines · Philipp Stanner
Quoted 23 lines · Jiri Slaby
drivers/gpu/drm/qxl/qxl_release.c qxl_release_free(struct qxl_device *qdev, · line 150
Quoted 10 lines · Jiri Slaby
What helps is indeed the revert back to:
drivers/gpu/drm/qxl/qxl_release.c +1
-1@@ -147,7 +147,7 @@ qxl_release_free(struct qxl_device *qdev, idr_remove(&qdev->release_idr, release->id); spin_unlock(&qdev->release_idr_lock); - if (dma_fence_was_initialized(&release->base)) { + if (release->base.ops) {Crap, the flags simple got garbage from previous freed buffer, so:
- release =kmalloc(size, GFP_KERNEL);
+ release =kzalloc(size, GFP_KERNEL);fixes this of course. Patch coming.
Now, the ops and initialized are either both set or unset:
qxl_release_free1: release->base.ops=0x0 initialized=0 id=6 type=2qxl_release_free1: release->base.ops=qxl_fence_ops initialized=1 id=2 type=0Quoted 4 lines
Unlike before.
FTR
release->base.opswere NULLed after that kmalloc. Unlikerelease->base.flags.You're saying that QXL is re-using that fence if it were uninitialized?
Quoted 7 lines
Quoted 3 lines
Sort of. The fence was dropped and freed. Then a new memory was allocated. And garbage from that allocation was used -- it might be the old (freed) fence. Or anything else freed previously. Simply garbage.
Maybe even
dma_fence_init()should zero out the memory for maximum robustness.
That would not help as dma_fence_init() is called too late or not at all here...
dma_fence_init() is called late as the driver wants to call dma_fence_was_initialized() to distinguish the cases when the init was called and when not.
Quoted 3 lines
We do that in
drm_schedat a few places.Opinions? @Christian?
Signature · 2 lines
js suse labs
Christian König
reviewerPATCH 2/9 dma-buf: protected fence ops by RCU v5
In reply to Christian KönigAt first glance it is counter intuitive to protect a constant function pointer table by RCU, but this allows modules providing the function table to unload by waiting for an RCU grace period.
v2: make one the now duplicated lockdep warnings a comment instead.
v3: Add more documentation to ->wait and ->release callback.
v4: fix typo in documentation
v5: rebased on drm-tip
--- drivers/dma-buf/dma-fence.c | 69 +++++++++++++++++++++++++------------ include/linux/dma-fence.h | 29 ++++++++++++++-- 2 files changed, 73 insertions(+), 25 deletions(-)
drivers/dma-buf/dma-fence.c +47 -22
@@ -522,6 +522,7 @@ EXPORT_SYMBOL(dma_fence_signal);signed long dma_fence_wait_timeout (struct dma_fence *fence,bool intr,signed long timeout) { +const struct dma_fence_ops *ops;signed long ret;if (WARN_ON (timeout <0 ))
@@ -533,15 +534,21 @@ dma_fence_wait_timeout(struct dma_fence *fence, bool intr, signed long timeout)dma_fence_enable_sw_signaling (fence); -if (trace_dma_fence_wait_start_enabled ()) { -rcu_read_lock (); -trace_dma_fence_wait_start (fence); +rcu_read_lock (); + ops =rcu_dereference (fence->ops); +trace_dma_fence_wait_start (fence); +if (ops->wait) { +/* +* Implementing the wait ops is deprecated and not supported for +* issuer independent fences, so it is ok to use the ops outside +* the RCU protected section. +*/ +rcu_read_unlock (); + ret = ops->wait (fence, intr, timeout); + }else {rcu_read_unlock (); - } -if (fence->ops->wait) - ret = fence->ops->wait (fence, intr, timeout); -else ret =dma_fence_default_wait (fence, intr, timeout); + }if (trace_dma_fence_wait_end_enabled ()) {rcu_read_lock ();trace_dma_fence_wait_end (fence);
@@ -562,6 +569,7 @@ void dma_fence_release(struct kref *kref) {struct dma_fence *fence = container_of(kref, struct dma_fence, refcount); +const struct dma_fence_ops *ops;rcu_read_lock ();trace_dma_fence_destroy (fence);
@@ -593,12 +601,12 @@ void dma_fence_release(struct kref *kref)spin_unlock_irqrestore (fence->lock, flags); } -rcu_read_unlock (); - -if (fence->ops->release) - fence->ops->release (fence); + ops =rcu_dereference (fence->ops); +if (ops->release) + ops->release (fence);else dma_fence_free (fence); +rcu_read_unlock (); }EXPORT_SYMBOL (dma_fence_release);
@@ -617,6 +625,7 @@ EXPORT_SYMBOL(dma_fence_free);static bool __dma_fence_enable_signaling (struct dma_fence *fence) { +const struct dma_fence_ops *ops;bool was_set;lockdep_assert_held (fence->lock);
@@ -627,14 +636,18 @@ static bool __dma_fence_enable_signaling(struct dma_fence *fence)if (dma_fence_test_signaled_flag (fence))return false; -if (!was_set && fence->ops->enable_signaling) { +rcu_read_lock (); + ops =rcu_dereference (fence->ops); +if (!was_set && ops->enable_signaling) {trace_dma_fence_enable_signal (fence); -if (!fence->ops->enable_signaling (fence)) { +if (!ops->enable_signaling (fence)) { +rcu_read_unlock ();dma_fence_signal_locked (fence);return false; } } +rcu_read_unlock ();return true; }
@@ -1007,8 +1020,13 @@ EXPORT_SYMBOL(dma_fence_wait_any_timeout);*/ void dma_fence_set_deadline (struct dma_fence *fence,ktime_t deadline) { -if (fence->ops->set_deadline && !dma_fence_is_signaled (fence)) - fence->ops->set_deadline (fence, deadline); +const struct dma_fence_ops *ops; + +rcu_read_lock (); + ops =rcu_dereference (fence->ops); +if (ops->set_deadline && !dma_fence_is_signaled (fence)) + ops->set_deadline (fence, deadline); +rcu_read_unlock (); }EXPORT_SYMBOL (dma_fence_set_deadline);
@@ -1049,7 +1067,12 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,BUG_ON (!ops || !ops->get_driver_name || !ops->get_timeline_name);kref_init (&fence->refcount); - fence->ops = ops; +/* +* At first glance it is counter intuitive to protect a constant +* function pointer table by RCU, but this allows modules providing the +* function table to unload by waiting for an RCU grace period. +*/ +RCU_INIT_POINTER (fence->ops, ops);INIT_LIST_HEAD (&fence->cb_list); fence->lock = lock; fence->context = context;
@@ -1129,11 +1152,12 @@ EXPORT_SYMBOL(dma_fence_init64);*/ const char __rcu *dma_fence_driver_name (struct dma_fence *fence) { -RCU_LOCKDEP_WARN (!rcu_read_lock_held (), -"RCU protection is required for safe access to returned string" ); +const struct dma_fence_ops *ops; +/* RCU protection is required for safe access to returned string */ + ops =rcu_dereference (fence->ops);if (!dma_fence_test_signaled_flag (fence)) -return fence->ops->get_driver_name (fence); +return ops->get_driver_name (fence);else return "detached-driver" ; }
@@ -1161,11 +1185,12 @@ EXPORT_SYMBOL(dma_fence_driver_name);*/ const char __rcu *dma_fence_timeline_name (struct dma_fence *fence) { -RCU_LOCKDEP_WARN (!rcu_read_lock_held (), -"RCU protection is required for safe access to returned string" ); +const struct dma_fence_ops *ops; +/* RCU protection is required for safe access to returned string */ + ops =rcu_dereference (fence->ops);if (!dma_fence_test_signaled_flag (fence)) -return fence->ops->get_timeline_name (fence); +return ops->get_timeline_name (fence);else return "signaled-timeline" ; }
include/linux/dma-fence.h +26 -3
@@ -67,7 +67,7 @@ struct seq_file;*/ struct dma_fence {spinlock_t *lock; -const struct dma_fence_ops *ops; +const struct dma_fence_ops __rcu *ops;/* * We clear the callback list on kref_put so that by the time we * release the fence it is unused. No one should be adding to the
@@ -220,6 +220,10 @@ struct dma_fence_ops {* timed out. Can also return other error values on custom implementations, * which should be treated as if the fence is signaled. For example a hardware * lockup could be reported like that. +* +* Implementing this callback prevents the fence from detaching after +* signaling and so it is mandatory for the module providing the +* dma_fence_ops to stay loaded as long as the dma_fence exists. */ signed long (*wait)(struct dma_fence *fence,bool intr,signed long timeout);
@@ -231,6 +235,13 @@ struct dma_fence_ops {* Can be called from irq context. This callback is optional. If it is * NULL, then dma_fence_free() is instead called as the default * implementation. +* +* Implementing this callback prevents the fence from detaching after +* signaling and so it is mandatory for the module providing the +* dma_fence_ops to stay loaded as long as the dma_fence exists. +* +* If the callback is implemented the memory backing the dma_fence +* object must be freed RCU safe. */ void (*release)(struct dma_fence *fence);
@@ -454,13 +465,19 @@ dma_fence_test_signaled_flag(struct dma_fence *fence)static inline bool dma_fence_is_signaled_locked (struct dma_fence *fence) { +const struct dma_fence_ops *ops; +if (dma_fence_test_signaled_flag (fence))return true; -if (fence->ops->signaled && fence->ops->signaled (fence)) { +rcu_read_lock (); + ops =rcu_dereference (fence->ops); +if (ops->signaled && ops->signaled (fence)) { +rcu_read_unlock ();dma_fence_signal_locked (fence);return true; } +rcu_read_unlock ();return false; }
@@ -484,13 +501,19 @@ dma_fence_is_signaled_locked(struct dma_fence *fence)static inline bool dma_fence_is_signaled (struct dma_fence *fence) { +const struct dma_fence_ops *ops; +if (dma_fence_test_signaled_flag (fence))return true; -if (fence->ops->signaled && fence->ops->signaled (fence)) { +rcu_read_lock (); + ops =rcu_dereference (fence->ops); +if (ops->signaled && ops->signaled (fence)) { +rcu_read_unlock ();dma_fence_signal (fence);return true; } +rcu_read_unlock ();return false; }
Signature · 1 line
2.43.0
Christian König
reviewerPATCH 3/9 dma-buf: detach fence ops on signal v2
In reply to Christian KönigWhen neither a release nor a wait backend ops is specified it is possible to let the dma_fence live on independently of the module who issued it.
This makes it possible to unload drivers and only wait for all their fences to signal.
v2: fix typo in comment
--- drivers/dma-buf/dma-fence.c | 16 ++++++++++++---- include/linux/dma-fence.h | 4 ++-- 2 files changed, 14 insertions(+), 6 deletions(-)
drivers/dma-buf/dma-fence.c +12 -4
@@ -371,6 +371,14 @@ void dma_fence_signal_timestamp_locked(struct dma_fence *fence, &fence->flags)))return ; +/* +* When neither a release nor a wait operation is specified set the ops +* pointer to NULL to allow the fence structure to become independent +* from who originally issued it. +*/ +if (!fence->ops->release && !fence->ops->wait) +RCU_INIT_POINTER (fence->ops,NULL ); +/* Stash the cb_list before replacing it with the timestamp */ list_replace (&fence->cb_list, &cb_list);
@@ -537,7 +545,7 @@ dma_fence_wait_timeout(struct dma_fence *fence, bool intr, signed long timeout)rcu_read_lock (); ops =rcu_dereference (fence->ops);trace_dma_fence_wait_start (fence); -if (ops->wait) { +if (ops && ops->wait) {/* * Implementing the wait ops is deprecated and not supported for * issuer independent fences, so it is ok to use the ops outside
@@ -602,7 +610,7 @@ void dma_fence_release(struct kref *kref) } ops =rcu_dereference (fence->ops); -if (ops->release) +if (ops && ops->release) ops->release (fence);else dma_fence_free (fence);
@@ -638,7 +646,7 @@ static bool __dma_fence_enable_signaling(struct dma_fence *fence)rcu_read_lock (); ops =rcu_dereference (fence->ops); -if (!was_set && ops->enable_signaling) { +if (!was_set && ops && ops->enable_signaling) {trace_dma_fence_enable_signal (fence);if (!ops->enable_signaling (fence)) {
@@ -1024,7 +1032,7 @@ void dma_fence_set_deadline(struct dma_fence *fence, ktime_t deadline)rcu_read_lock (); ops =rcu_dereference (fence->ops); -if (ops->set_deadline && !dma_fence_is_signaled (fence)) +if (ops && ops->set_deadline && !dma_fence_is_signaled (fence)) ops->set_deadline (fence, deadline);rcu_read_unlock (); }
include/linux/dma-fence.h +2 -2
@@ -472,7 +472,7 @@ dma_fence_is_signaled_locked(struct dma_fence *fence)rcu_read_lock (); ops =rcu_dereference (fence->ops); -if (ops->signaled && ops->signaled (fence)) { +if (ops && ops->signaled && ops->signaled (fence)) {rcu_read_unlock ();dma_fence_signal_locked (fence);return true;
@@ -508,7 +508,7 @@ dma_fence_is_signaled(struct dma_fence *fence)rcu_read_lock (); ops =rcu_dereference (fence->ops); -if (ops->signaled && ops->signaled (fence)) { +if (ops && ops->signaled && ops->signaled (fence)) {rcu_read_unlock ();dma_fence_signal (fence);return true;
Signature · 1 line
2.43.0
Christian König
reviewerPATCH 4/9 dma-buf: abstract fence locking
In reply to Christian KönigAdd dma_fence_lock_irqsafe() and dma_fence_unlock_irqrestore() wrappers and mechanically apply them everywhere.
Just a pre-requisite cleanup for a follow up patch.
--- drivers/dma-buf/dma-fence.c | 44 +++++++++++------------- drivers/dma-buf/st-dma-fence.c | 6 ++-- drivers/dma-buf/sw_sync.c | 14 ++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c | 4 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 4 +-- drivers/gpu/drm/nouveau/nouveau_drm.c | 5 +-- drivers/gpu/drm/scheduler/sched_fence.c | 6 ++-- drivers/gpu/drm/xe/xe_sched_job.c | 4 +-- include/linux/dma-fence.h | 20 +++++++++++ 9 files changed, 63 insertions(+), 44 deletions(-)
drivers/dma-buf/dma-fence.c +20 -24
@@ -412,9 +412,9 @@ void dma_fence_signal_timestamp(struct dma_fence *fence, ktime_t timestamp)if (WARN_ON (!fence))return ; -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags);dma_fence_signal_timestamp_locked (fence, timestamp); -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags); }EXPORT_SYMBOL (dma_fence_signal_timestamp);
@@ -473,9 +473,9 @@ bool dma_fence_check_and_signal(struct dma_fence *fence)unsigned long flags;bool ret; -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags); ret =dma_fence_check_and_signal_locked (fence); -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags);return ret; }
@@ -501,9 +501,9 @@ void dma_fence_signal(struct dma_fence *fence) tmp =dma_fence_begin_signalling (); -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags);dma_fence_signal_timestamp_locked (fence,ktime_get ()); -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags);dma_fence_end_signalling (tmp); }
@@ -603,10 +603,10 @@ void dma_fence_release(struct kref *kref)* don't leave chains dangling. We set the error flag first * so that the callbacks know this signal is due to an error. */ -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags); fence->error = -EDEADLK ;dma_fence_signal_locked (fence); -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags); } ops =rcu_dereference (fence->ops);
@@ -672,9 +672,9 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence) {unsigned long flags; -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags);__dma_fence_enable_signaling (fence); -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags); }EXPORT_SYMBOL (dma_fence_enable_sw_signaling);
@@ -714,8 +714,7 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,return -ENOENT ; } -spin_lock_irqsave (fence->lock, flags); - +dma_fence_lock_irqsave (fence, flags);if (__dma_fence_enable_signaling (fence)) { cb->func = func;list_add_tail (&cb->node, &fence->cb_list);
@@ -723,8 +722,7 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb,INIT_LIST_HEAD (&cb->node); ret = -ENOENT ; } - -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags);return ret; }
@@ -747,9 +745,9 @@ int dma_fence_get_status(struct dma_fence *fence)unsigned long flags;int status; -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags); status =dma_fence_get_status_locked (fence); -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags);return status; }
@@ -779,13 +777,11 @@ dma_fence_remove_callback(struct dma_fence *fence, struct dma_fence_cb *cb)unsigned long flags;bool ret; -spin_lock_irqsave (fence->lock, flags); - +dma_fence_lock_irqsave (fence, flags); ret = !list_empty (&cb->node);if (ret)list_del_init (&cb->node); - -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags);return ret; }
@@ -824,7 +820,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)unsigned long flags;signed long ret = timeout ? timeout :1 ; -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags);if (dma_fence_test_signaled_flag (fence)) goto out;
@@ -848,11 +844,11 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)__set_current_state (TASK_INTERRUPTIBLE ); else__set_current_state (TASK_UNINTERRUPTIBLE ); -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags); ret =schedule_timeout (ret); -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags);if (ret >0 && intr &&signal_pending (current)) ret = -ERESTARTSYS ; }
@@ -862,7 +858,7 @@ dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout)__set_current_state (TASK_RUNNING ); out: -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags);return ret; }EXPORT_SYMBOL (dma_fence_default_wait);
drivers/dma-buf/st-dma-fence.c +4 -2
@@ -410,8 +410,10 @@ struct race_thread {static void __wait_for_callbacks (struct dma_fence *f) { -spin_lock_irq (f->lock); -spin_unlock_irq (f->lock); +unsigned long flags; + +dma_fence_lock_irqsave (f, flags); +dma_fence_unlock_irqrestore (f, flags); }static int thread_signal_callback (void *arg)
drivers/dma-buf/sw_sync.c +7 -7
@@ -156,12 +156,12 @@ static void timeline_fence_release(struct dma_fence *fence)struct sync_timeline *parent =dma_fence_parent (fence);unsigned long flags; -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags);if (!list_empty (&pt->link)) {list_del (&pt->link);rb_erase (&pt->node, &parent->pt_tree); } -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags);sync_timeline_put (parent);dma_fence_free (fence);
@@ -179,7 +179,7 @@ static void timeline_fence_set_deadline(struct dma_fence *fence, ktime_t deadlinstruct sync_pt *pt =dma_fence_to_sync_pt (fence);unsigned long flags; -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags);if (test_bit (SW_SYNC_HAS_DEADLINE_BIT , &fence->flags)) {if (ktime_before (deadline, pt->deadline)) pt->deadline = deadline;
@@ -187,7 +187,7 @@ static void timeline_fence_set_deadline(struct dma_fence *fence, ktime_t deadlin pt->deadline = deadline;__set_bit (SW_SYNC_HAS_DEADLINE_BIT , &fence->flags); } -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags); }static const struct dma_fence_ops timeline_fence_ops = {
@@ -431,13 +431,13 @@ static int sw_sync_ioctl_get_deadline(struct sync_timeline *obj, unsigned long a goto put_fence; } -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags);if (!test_bit (SW_SYNC_HAS_DEADLINE_BIT , &fence->flags)) { ret = -ENOENT ; goto unlock; } data.deadline_ns =ktime_to_ns (pt->deadline); -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags);dma_fence_put (fence);
@@ -450,7 +450,7 @@ static int sw_sync_ioctl_get_deadline(struct sync_timeline *obj, unsigned long areturn 0 ; unlock: -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags); put_fence:dma_fence_put (fence);
drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +2 -2
@@ -474,10 +474,10 @@ bool amdgpu_ring_soft_recovery(struct amdgpu_ring *ring, unsigned int vmid,if (amdgpu_sriov_vf (ring->adev) || !ring->funcs->soft_recovery || !fence)return false; -spin_lock_irqsave (fence->lock, flags); +dma_fence_lock_irqsave (fence, flags);if (!dma_fence_is_signaled_locked (fence))dma_fence_set_error (fence, -ENODATA ); -spin_unlock_irqrestore (fence->lock, flags); +dma_fence_unlock_irqrestore (fence, flags);while (!dma_fence_is_signaled (fence) &&ktime_to_ns (ktime_sub (deadline,ktime_get ())) >0 )
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +2 -2
@@ -2784,8 +2784,8 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)dma_fence_put (vm->last_unlocked);dma_fence_wait (vm->last_tlb_flush, false);/* Make sure that all fence callbacks have completed */ -spin_lock_irqsave (vm->last_tlb_flush->lock, flags); -spin_unlock_irqrestore (vm->last_tlb_flush->lock, flags); +dma_fence_lock_irqsave (vm->last_tlb_flush, flags); +dma_fence_unlock_irqrestore (vm->last_tlb_flush, flags);dma_fence_put (vm->last_tlb_flush);list_for_each_entry_safe (mapping, tmp, &vm->freed, list) {
drivers/gpu/drm/nouveau/nouveau_drm.c +3 -2
@@ -156,12 +156,13 @@ nouveau_name(struct drm_device *dev)static inline bool nouveau_cli_work_ready (struct dma_fence *fence) { +unsigned long flags;bool ret = true; -spin_lock_irq (fence->lock); +dma_fence_lock_irqsave (fence, flags);if (!dma_fence_is_signaled_locked (fence)) ret = false; -spin_unlock_irq (fence->lock); +dma_fence_unlock_irqrestore (fence, flags);if (ret == true)dma_fence_put (fence);
drivers/gpu/drm/scheduler/sched_fence.c +3 -3
@@ -156,19 +156,19 @@ static void drm_sched_fence_set_deadline_finished(struct dma_fence *f,struct dma_fence *parent;unsigned long flags; -spin_lock_irqsave (&fence->lock, flags); +dma_fence_lock_irqsave (f, flags);/* If we already have an earlier deadline, keep it: */ if (test_bit (DRM_SCHED_FENCE_FLAG_HAS_DEADLINE_BIT , &f->flags) &&ktime_before (fence->deadline, deadline)) { -spin_unlock_irqrestore (&fence->lock, flags); +dma_fence_unlock_irqrestore (f, flags);return ; } fence->deadline = deadline;set_bit (DRM_SCHED_FENCE_FLAG_HAS_DEADLINE_BIT , &f->flags); -spin_unlock_irqrestore (&fence->lock, flags); +dma_fence_unlock_irqrestore (f, flags);/* * smp_load_aquire() to ensure that if we are racing another
drivers/gpu/drm/xe/xe_sched_job.c +2 -2
@@ -189,11 +189,11 @@ static bool xe_fence_set_error(struct dma_fence *fence, int error)unsigned long irq_flags;bool signaled; -spin_lock_irqsave (fence->lock, irq_flags); +dma_fence_lock_irqsave (fence, irq_flags); signaled =test_bit (DMA_FENCE_FLAG_SIGNALED_BIT , &fence->flags);if (!signaled)dma_fence_set_error (fence, error); -spin_unlock_irqrestore (fence->lock, irq_flags); +dma_fence_unlock_irqrestore (fence, irq_flags);return signaled; }
include/linux/dma-fence.h +20 -0
@@ -377,6 +377,26 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep) }while (1 ); } +/** +* dma_fence_lock_irqsave - irqsave lock the fence +* @fence: the fence to lock +* @flags: where to store the CPU flags. +* +* Lock the fence, preventing it from changing to the signaled state. +*/ +#define dma_fence_lock_irqsave (fence, flags) \ + spin_lock_irqsave(fence->lock, flags) + +/** +* dma_fence_unlock_irqrestore - unlock the fence and irqrestore +* @fence: the fence to unlock +* @flags the CPU flags to restore +* +* Unlock the fence, allowing it to change it's state to signaled again. +*/ +#define dma_fence_unlock_irqrestore (fence, flags) \ + spin_unlock_irqrestore(fence->lock, flags) +#ifdef CONFIG_LOCKDEP bool dma_fence_begin_signalling (void );void dma_fence_end_signalling (bool cookie);
Signature · 1 line
2.43.0
Christian König
reviewerPATCH 5/9 dma-buf: inline spinlock for fence protection v4
In reply to Christian KönigImplement per-fence spinlocks, allowing implementations to not give an external spinlock to protect the fence internal statei. Instead a spinlock embedded into the fence structure itself is used in this case.
Shared spinlocks have the problem that implementations need to guarantee that the lock live at least as long all fences referencing them.
Using a per-fence spinlock allows completely decoupling spinlock producer and consumer life times, simplifying the handling in most use cases.
v2: improve naming, coverage and function documentation
v3: fix one additional locking in the selftests
v4: separate out some changes to make the patch smaller,
fix one amdgpu crash found by CI systems
--- drivers/dma-buf/dma-fence.c | 25 +++++++++++++++++------- drivers/dma-buf/sync_debug.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +- drivers/gpu/drm/drm_crtc.c | 2 +- drivers/gpu/drm/drm_writeback.c | 2 +- drivers/gpu/drm/nouveau/nouveau_fence.c | 3 ++- drivers/gpu/drm/qxl/qxl_release.c | 3 ++- drivers/gpu/drm/vmwgfx/vmwgfx_fence.c | 3 ++- drivers/gpu/drm/xe/xe_hw_fence.c | 3 ++- include/linux/dma-fence.h | 26 +++++++++++++++++++++---- 10 files changed, 52 insertions(+), 19 deletions(-)
drivers/dma-buf/dma-fence.c +18 -7
@@ -343,7 +343,6 @@ void __dma_fence_might_wait(void) }#endif -/** * dma_fence_signal_timestamp_locked - signal completion of a fence * @fence: the fence to signal
@@ -365,7 +364,7 @@ void dma_fence_signal_timestamp_locked(struct dma_fence *fence,struct dma_fence_cb *cur, *tmp;struct list_head cb_list; -lockdep_assert_held (fence->lock); +lockdep_assert_held (dma_fence_spinlock (fence));if (unlikely (test_and_set_bit (DMA_FENCE_FLAG_SIGNALED_BIT , &fence->flags)))
@@ -636,7 +635,7 @@ static bool __dma_fence_enable_signaling(struct dma_fence *fence)const struct dma_fence_ops *ops;bool was_set; -lockdep_assert_held (fence->lock); +lockdep_assert_held (dma_fence_spinlock (fence)); was_set =test_and_set_bit (DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT , &fence->flags);
@@ -1067,7 +1066,6 @@ static void __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, spinlock_t *lock, u64 context, u64 seqno, unsigned long flags) { - BUG_ON(!lock); BUG_ON(!ops || !ops->get_driver_name || !ops->get_timeline_name);kref_init (&fence->refcount);
@@ -1078,10 +1076,15 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,*/ RCU_INIT_POINTER (fence->ops, ops);INIT_LIST_HEAD (&fence->cb_list); - fence->lock = lock; fence->context = context; fence->seqno = seqno; fence->flags = flags |BIT (DMA_FENCE_FLAG_INITIALIZED_BIT ); +if (lock) { + fence->extern_lock = lock; + }else { +spin_lock_init (&fence->inline_lock); + fence->flags |=BIT (DMA_FENCE_FLAG_INLINE_LOCK_BIT ); + } fence->error =0 ;trace_dma_fence_init (fence);
@@ -1091,7 +1094,7 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,* dma_fence_init - Initialize a custom fence. * @fence: the fence to initialize * @ops: the dma_fence_ops for operations on this fence -* @lock: the irqsafe spinlock to use for locking this fence +* @lock: optional irqsafe spinlock to use for locking this fence * @context: the execution context this fence is run on * @seqno: a linear increasing sequence number for this context *
@@ -1101,6 +1104,10 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,* * context and seqno are used for easy comparison between fences, allowing * to check which fence is later by simply using dma_fence_later(). +* +* It is strongly discouraged to provide an external lock. This is only allowed +* for legacy use cases when multiple fences need to be prevented from +* signaling out of order. */ void dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,
@@ -1114,7 +1121,7 @@ EXPORT_SYMBOL(dma_fence_init);* dma_fence_init64 - Initialize a custom fence with 64-bit seqno support. * @fence: the fence to initialize * @ops: the dma_fence_ops for operations on this fence -* @lock: the irqsafe spinlock to use for locking this fence +* @lock: optional irqsafe spinlock to use for locking this fence * @context: the execution context this fence is run on * @seqno: a linear increasing sequence number for this context *
@@ -1124,6 +1131,10 @@ EXPORT_SYMBOL(dma_fence_init);* * Context and seqno are used for easy comparison between fences, allowing * to check which fence is later by simply using dma_fence_later(). +* +* It is strongly discouraged to provide an external lock. This is only allowed +* for legacy use cases when multiple fences need to be prevented from +* signaling out of order. */ void dma_fence_init64(struct dma_fence *fence, const struct dma_fence_ops *ops,
drivers/dma-buf/sync_debug.h +1 -1
@@ -47,7 +47,7 @@ struct sync_timeline {static inline struct sync_timeline *dma_fence_parent (struct dma_fence *fence) { - return container_of(fence->lock, struct sync_timeline, lock); + return container_of(fence->extern_lock, struct sync_timeline, lock); }/**
drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +1 -1
@@ -636,7 +636,7 @@ static inline uint64_t amdgpu_vm_tlb_seq(struct amdgpu_vm *vm)* sure that the dma_fence structure isn't freed up. */ rcu_read_lock (); - lock = vm->last_tlb_flush->lock; + lock =dma_fence_spinlock (vm->last_tlb_flush);rcu_read_unlock ();spin_lock_irqsave (lock, flags);
drivers/gpu/drm/drm_crtc.c +1 -1
@@ -159,7 +159,7 @@ static const struct dma_fence_ops drm_crtc_fence_ops;static struct drm_crtc *fence_to_crtc (struct dma_fence *fence) {BUG_ON (fence->ops != &drm_crtc_fence_ops); - return container_of(fence->lock, struct drm_crtc, fence_lock); + return container_of(fence->extern_lock, struct drm_crtc, fence_lock); }static const char *drm_crtc_fence_get_driver_name (struct dma_fence *fence)
drivers/gpu/drm/drm_writeback.c +1 -1
@@ -81,7 +81,7 @@* From userspace, this property will always read as zero. */ -#define fence_to_wb_connector (x) container_of(x->lock, \ +#define fence_to_wb_connector (x) container_of(x->extern_lock, \ struct drm_writeback_connector, \ fence_lock)
drivers/gpu/drm/nouveau/nouveau_fence.c +2 -1
@@ -41,7 +41,8 @@ static const struct dma_fence_ops nouveau_fence_ops_legacy;static inline struct nouveau_fence_chan *nouveau_fctx (struct nouveau_fence *fence) { - return container_of(fence->base.lock, struct nouveau_fence_chan, lock); + return container_of(fence->base.extern_lock, struct nouveau_fence_chan, + lock); } static bool
drivers/gpu/drm/qxl/qxl_release.c +2 -1
@@ -62,7 +62,8 @@ static long qxl_fence_wait(struct dma_fence *fence, bool intr,struct qxl_device *qdev;unsigned long cur, end = jiffies + timeout; - qdev = container_of(fence->lock, struct qxl_device, release_lock); + qdev = container_of(fence->extern_lock, struct qxl_device, + release_lock); if (!wait_event_timeout(qdev->release_event, (dma_fence_is_signaled(fence) ||
drivers/gpu/drm/vmwgfx/vmwgfx_fence.c +2 -1
@@ -47,7 +47,8 @@ struct vmw_event_fence_action {static struct vmw_fence_manager *fman_from_fence (struct vmw_fence_obj *fence) { - return container_of(fence->base.lock, struct vmw_fence_manager, lock); + return container_of(fence->base.extern_lock, struct vmw_fence_manager, + lock); }static void vmw_fence_obj_destroy (struct dma_fence *f)
drivers/gpu/drm/xe/xe_hw_fence.c +2 -1
@@ -142,7 +142,8 @@ static struct xe_hw_fence *to_xe_hw_fence(struct dma_fence *fence);static struct xe_hw_fence_irq *xe_hw_fence_irq (struct xe_hw_fence *fence) { - return container_of(fence->dma.lock, struct xe_hw_fence_irq, lock); + return container_of(fence->dma.extern_lock, struct xe_hw_fence_irq, + lock); }static const char *xe_hw_fence_get_driver_name (struct dma_fence *dma_fence)
include/linux/dma-fence.h +22 -4
@@ -34,7 +34,8 @@ struct seq_file;* @ops: dma_fence_ops associated with this fence * @rcu: used for releasing fence with kfree_rcu * @cb_list: list of all callbacks to call -* @lock: spin_lock_irqsave used for locking +* @extern_lock: external spin_lock_irqsave used for locking +* @inline_lock: alternative internal spin_lock_irqsave used for locking * @context: execution context this fence belongs to, returned by * dma_fence_context_alloc() * @seqno: the sequence number of this fence inside the execution context,
@@ -49,6 +50,7 @@ struct seq_file;* of the time. * * DMA_FENCE_FLAG_INITIALIZED_BIT - fence was initialized +* DMA_FENCE_FLAG_INLINE_LOCK_BIT - use inline spinlock instead of external one * DMA_FENCE_FLAG_SIGNALED_BIT - fence is already signaled * DMA_FENCE_FLAG_TIMESTAMP_BIT - timestamp recorded for fence signaling * DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT - enable_signaling might have been called
@@ -66,7 +68,10 @@ struct seq_file;* been completed, or never called at all. */ struct dma_fence { -spinlock_t *lock; +union { +spinlock_t *extern_lock; +spinlock_t inline_lock; + };const struct dma_fence_ops __rcu *ops;/* * We clear the callback list on kref_put so that by the time we
@@ -100,6 +105,7 @@ struct dma_fence {enum dma_fence_flag_bits {DMA_FENCE_FLAG_INITIALIZED_BIT , +DMA_FENCE_FLAG_INLINE_LOCK_BIT ,DMA_FENCE_FLAG_SEQNO64_BIT ,DMA_FENCE_FLAG_SIGNALED_BIT ,DMA_FENCE_FLAG_TIMESTAMP_BIT ,
@@ -377,6 +383,18 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep) }while (1 ); } +/** +* dma_fence_spinlock - return pointer to the spinlock protecting the fence +* @fence: the fence to get the lock from +* +* Return either the pointer to the embedded or the external spin lock. +*/ +static inline spinlock_t *dma_fence_spinlock (struct dma_fence *fence) +{ +return test_bit (DMA_FENCE_FLAG_INLINE_LOCK_BIT , &fence->flags) ? + &fence->inline_lock : fence->extern_lock; +} +/** * dma_fence_lock_irqsave - irqsave lock the fence * @fence: the fence to lock
@@ -385,7 +403,7 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep)* Lock the fence, preventing it from changing to the signaled state. */ #define dma_fence_lock_irqsave (fence, flags) \ - spin_lock_irqsave(fence->lock, flags) + spin_lock_irqsave(dma_fence_spinlock(fence), flags)/** * dma_fence_unlock_irqrestore - unlock the fence and irqrestore
@@ -395,7 +413,7 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep)* Unlock the fence, allowing it to change it's state to signaled again. */ #define dma_fence_unlock_irqrestore (fence, flags) \ - spin_unlock_irqrestore(fence->lock, flags) + spin_unlock_irqrestore(dma_fence_spinlock(fence), flags)#ifdef CONFIG_LOCKDEP bool dma_fence_begin_signalling (void );
Signature · 1 line
2.43.0
Re: [PATCH 5/9] dma-buf: inline spinlock for fence protection v4
Quoted 16 lines · Christian König
Implement per-fence spinlocks, allowing implementations to not give an external spinlock to protect the fence internal statei. Instead a spinlock embedded into the fence structure itself is used in this case.
Shared spinlocks have the problem that implementations need to guarantee that the lock live at least as long all fences referencing them.
Using a per-fence spinlock allows completely decoupling spinlock producer and consumer life times, simplifying the handling in most use cases.
v2: improve naming, coverage and function documentation
v3: fix one additional locking in the selftests
v4: separate out some changes to make the patch smaller,
fix one amdgpu crash found by CI systemsSigned-off-by Christian König <[email protected]>
Quoted 10 lines
--- drivers/dma-buf/dma-fence.c | 25 +++++++++++++++++------- drivers/dma-buf/sync_debug.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +- drivers/gpu/drm/drm_crtc.c | 2 +- drivers/gpu/drm/drm_writeback.c | 2 +- drivers/gpu/drm/nouveau/nouveau_fence.c | 3 ++- drivers/gpu/drm/qxl/qxl_release.c | 3 ++- drivers/gpu/drm/vmwgfx/vmwgfx_fence.c | 3 ++- drivers/gpu/drm/xe/xe_hw_fence.c | 3 ++-
i915 needed changes too, based on the kbuild report.
Have you seen my note about the RCU sparse warning as well?
Quoted 253 lines
include/linux/dma-fence.h | 26 +++++++++++++++++++++---- 10 files changed, 52 insertions(+), 19 deletions(-)drivers/dma-buf/dma-fence.c +18
-7@@ -343,7 +343,6 @@ void __dma_fence_might_wait(void) }#endif -/** * dma_fence_signal_timestamp_locked - signal completion of a fence * @fence: the fence to signal @@ -365,7 +364,7 @@ void dma_fence_signal_timestamp_locked(struct dma_fence *fence,struct dma_fence_cb *cur, *tmp;struct list_head cb_list; -lockdep_assert_held (fence->lock); +lockdep_assert_held (dma_fence_spinlock (fence));if (unlikely (test_and_set_bit (DMA_FENCE_FLAG_SIGNALED_BIT , &fence->flags)))@@ -636,7 +635,7 @@ static bool __dma_fence_enable_signaling(struct dma_fence *fence)const struct dma_fence_ops *ops;bool was_set; -lockdep_assert_held (fence->lock); +lockdep_assert_held (dma_fence_spinlock (fence)); was_set =test_and_set_bit (DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT , &fence->flags);@@ -1067,7 +1066,6 @@ static void __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, spinlock_t *lock, u64 context, u64 seqno, unsigned long flags) { - BUG_ON(!lock); BUG_ON(!ops || !ops->get_driver_name || !ops->get_timeline_name);kref_init (&fence->refcount);@@ -1078,10 +1076,15 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,*/ RCU_INIT_POINTER (fence->ops, ops);INIT_LIST_HEAD (&fence->cb_list); - fence->lock = lock; fence->context = context; fence->seqno = seqno; fence->flags = flags |BIT (DMA_FENCE_FLAG_INITIALIZED_BIT ); +if (lock) { + fence->extern_lock = lock; + }else { +spin_lock_init (&fence->inline_lock); + fence->flags |=BIT (DMA_FENCE_FLAG_INLINE_LOCK_BIT ); + } fence->error =0 ;trace_dma_fence_init (fence);@@ -1091,7 +1094,7 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,* dma_fence_init - Initialize a custom fence. * @fence: the fence to initialize * @ops: the dma_fence_ops for operations on this fence -* @lock: the irqsafe spinlock to use for locking this fence +* @lock: optional irqsafe spinlock to use for locking this fence * @context: the execution context this fence is run on * @seqno: a linear increasing sequence number for this context * @@ -1101,6 +1104,10 @@ __dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,* * context and seqno are used for easy comparison between fences, allowing * to check which fence is later by simply using dma_fence_later(). +* +* It is strongly discouraged to provide an external lock. This is only allowed +* for legacy use cases when multiple fences need to be prevented from +* signaling out of order. */ void dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops,@@ -1114,7 +1121,7 @@ EXPORT_SYMBOL(dma_fence_init);* dma_fence_init64 - Initialize a custom fence with 64-bit seqno support. * @fence: the fence to initialize * @ops: the dma_fence_ops for operations on this fence -* @lock: the irqsafe spinlock to use for locking this fence +* @lock: optional irqsafe spinlock to use for locking this fence * @context: the execution context this fence is run on * @seqno: a linear increasing sequence number for this context * @@ -1124,6 +1131,10 @@ EXPORT_SYMBOL(dma_fence_init);* * Context and seqno are used for easy comparison between fences, allowing * to check which fence is later by simply using dma_fence_later(). +* +* It is strongly discouraged to provide an external lock. This is only allowed +* for legacy use cases when multiple fences need to be prevented from +* signaling out of order. */ void dma_fence_init64(struct dma_fence *fence, const struct dma_fence_ops *ops,drivers/dma-buf/sync_debug.h +1
-1@@ -47,7 +47,7 @@ struct sync_timeline {static inline struct sync_timeline *dma_fence_parent (struct dma_fence *fence) { - return container_of(fence->lock, struct sync_timeline, lock); + return container_of(fence->extern_lock, struct sync_timeline, lock); }/** drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +1
-1@@ -636,7 +636,7 @@ static inline uint64_t amdgpu_vm_tlb_seq(struct amdgpu_vm *vm)* sure that the dma_fence structure isn't freed up. */ rcu_read_lock (); - lock = vm->last_tlb_flush->lock; + lock =dma_fence_spinlock (vm->last_tlb_flush);rcu_read_unlock ();spin_lock_irqsave (lock, flags);drivers/gpu/drm/drm_crtc.c +1
-1@@ -159,7 +159,7 @@ static const struct dma_fence_ops drm_crtc_fence_ops;static struct drm_crtc *fence_to_crtc (struct dma_fence *fence) {BUG_ON (fence->ops != &drm_crtc_fence_ops); - return container_of(fence->lock, struct drm_crtc, fence_lock); + return container_of(fence->extern_lock, struct drm_crtc, fence_lock); }static const char *drm_crtc_fence_get_driver_name (struct dma_fence *fence)drivers/gpu/drm/drm_writeback.c +1
-1@@ -81,7 +81,7 @@* From userspace, this property will always read as zero. */ -#define fence_to_wb_connector (x) container_of(x->lock, \ +#define fence_to_wb_connector (x) container_of(x->extern_lock, \ struct drm_writeback_connector, \ fence_lock)drivers/gpu/drm/nouveau/nouveau_fence.c +2
-1@@ -41,7 +41,8 @@ static const struct dma_fence_ops nouveau_fence_ops_legacy;static inline struct nouveau_fence_chan *nouveau_fctx (struct nouveau_fence *fence) { - return container_of(fence->base.lock, struct nouveau_fence_chan, lock); + return container_of(fence->base.extern_lock, struct nouveau_fence_chan, + lock); } static booldrivers/gpu/drm/qxl/qxl_release.c +2
-1@@ -62,7 +62,8 @@ static long qxl_fence_wait(struct dma_fence *fence, bool intr,struct qxl_device *qdev;unsigned long cur, end = jiffies + timeout; - qdev = container_of(fence->lock, struct qxl_device, release_lock); + qdev = container_of(fence->extern_lock, struct qxl_device, + release_lock); if (!wait_event_timeout(qdev->release_event, (dma_fence_is_signaled(fence) ||drivers/gpu/drm/vmwgfx/vmwgfx_fence.c +2
-1@@ -47,7 +47,8 @@ struct vmw_event_fence_action {static struct vmw_fence_manager *fman_from_fence (struct vmw_fence_obj *fence) { - return container_of(fence->base.lock, struct vmw_fence_manager, lock); + return container_of(fence->base.extern_lock, struct vmw_fence_manager, + lock); }static void vmw_fence_obj_destroy (struct dma_fence *f)drivers/gpu/drm/xe/xe_hw_fence.c +2
-1@@ -142,7 +142,8 @@ static struct xe_hw_fence *to_xe_hw_fence(struct dma_fence *fence);static struct xe_hw_fence_irq *xe_hw_fence_irq (struct xe_hw_fence *fence) { - return container_of(fence->dma.lock, struct xe_hw_fence_irq, lock); + return container_of(fence->dma.extern_lock, struct xe_hw_fence_irq, + lock); }static const char *xe_hw_fence_get_driver_name (struct dma_fence *dma_fence)include/linux/dma-fence.h +9
-2@@ -34,7 +34,8 @@ struct seq_file;* @ops: dma_fence_ops associated with this fence * @rcu: used for releasing fence with kfree_rcu * @cb_list: list of all callbacks to call -* @lock: spin_lock_irqsave used for locking +* @extern_lock: external spin_lock_irqsave used for locking +* @inline_lock: alternative internal spin_lock_irqsave used for locking * @context: execution context this fence belongs to, returned by * dma_fence_context_alloc() * @seqno: the sequence number of this fence inside the execution context, @@ -49,6 +50,7 @@ struct seq_file;* of the time. * * DMA_FENCE_FLAG_INITIALIZED_BIT - fence was initialized +* DMA_FENCE_FLAG_INLINE_LOCK_BIT - use inline spinlock instead of external one * DMA_FENCE_FLAG_SIGNALED_BIT - fence is already signaled * DMA_FENCE_FLAG_TIMESTAMP_BIT - timestamp recorded for fence signaling * DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT - enable_signaling might have been called @@ -66,7 +68,10 @@ struct seq_file;* been completed, or never called at all. */ struct dma_fence { -spinlock_t *lock; +union { +spinlock_t *extern_lock; +spinlock_t inline_lock; + };const struct dma_fence_ops __rcu *ops;/* * We clear the callback list on kref_put so that by the time we @@ -100,6 +105,7 @@ struct dma_fence {enum dma_fence_flag_bits {DMA_FENCE_FLAG_INITIALIZED_BIT , +DMA_FENCE_FLAG_INLINE_LOCK_BIT ,DMA_FENCE_FLAG_SEQNO64_BIT ,DMA_FENCE_FLAG_SIGNALED_BIT ,DMA_FENCE_FLAG_TIMESTAMP_BIT ,@@ -377,6 +383,18 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep) }while (1 ); } +/**
include/linux/dma-fence.h dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep) · line 396
Quoted 10 lines
include/linux/dma-fence.h +10
-0+* dma_fence_spinlock - return pointer to the spinlock protecting the fence +* @fence: the fence to get the lock from +* +* Return either the pointer to the embedded or the external spin lock. +*/ +static inline spinlock_t *dma_fence_spinlock (struct dma_fence *fence) +{ +return test_bit (DMA_FENCE_FLAG_INLINE_LOCK_BIT , &fence->flags) ? + &fence->inline_lock : fence->extern_lock; +}
You did not want to move this helper into "dma-buf: abstract fence locking" ? I think that would have been better to keep everything mechanical in one patch, and then this patch which changes behaviour does not touch any drivers but only dma-fence core.
Also, what about adding something like dma_fence_container_of() in that patch as well?
Regards,
Tvrtko
Quoted 22 lines
include/linux/dma-fence.h +3
-2+/** * dma_fence_lock_irqsave - irqsave lock the fence * @fence: the fence to lock @@ -385,7 +403,7 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep)* Lock the fence, preventing it from changing to the signaled state. */ #define dma_fence_lock_irqsave (fence, flags) \ - spin_lock_irqsave(fence->lock, flags) + spin_lock_irqsave(dma_fence_spinlock(fence), flags)/** * dma_fence_unlock_irqrestore - unlock the fence and irqrestore @@ -395,7 +413,7 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep)* Unlock the fence, allowing it to change it's state to signaled again. */ #define dma_fence_unlock_irqrestore (fence, flags) \ - spin_unlock_irqrestore(fence->lock, flags) + spin_unlock_irqrestore(dma_fence_spinlock(fence), flags)#ifdef CONFIG_LOCKDEP bool dma_fence_begin_signalling (void );
Christian König
maintainerRe: [PATCH 5/9] dma-buf: inline spinlock for fence protection v4
Quoted 29 lines · Tvrtko Ursulin
Quoted 16 lines · Christian König
Implement per-fence spinlocks, allowing implementations to not give an external spinlock to protect the fence internal statei. Instead a spinlock embedded into the fence structure itself is used in this case.
Shared spinlocks have the problem that implementations need to guarantee that the lock live at least as long all fences referencing them.
Using a per-fence spinlock allows completely decoupling spinlock producer and consumer life times, simplifying the handling in most use cases.
v2: improve naming, coverage and function documentation
v3: fix one additional locking in the selftests
v4: separate out some changes to make the patch smaller, fix one amdgpu crash found by CI systemsSigned-off-by Christian König <[email protected]>Quoted 10 lines
---
drivers/dma-buf/dma-fence.c | 25 +++++++++++++++++------- drivers/dma-buf/sync_debug.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +- drivers/gpu/drm/drm_crtc.c | 2 +- drivers/gpu/drm/drm_writeback.c | 2 +-drivers/gpu/drm/nouveau/nouveau_fence.c| 3 ++- drivers/gpu/drm/qxl/qxl_release.c | 3 ++- drivers/gpu/drm/vmwgfx/vmwgfx_fence.c | 3 ++- drivers/gpu/drm/xe/xe_hw_fence.c | 3 ++-
Quoted 1 line
i915 needed changes too, based on the kbuild report.
Going to take a look now.
Quoted 1 line
Have you seen my note about the RCU sparse warning as well?
Nope, I must have missed that mail.
...
Quoted 5 lines
Quoted 5 lines
+/**
+ *dma_fence_spinlock- return pointer to the spinlock protecting the fence
+ * @fence: the fence to get the lock from
+ *
+ * Return either the pointer to the embedded or the external spin lock.
Quoted 8 lines
include/linux/dma-fence.h dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep) · line 396
Quoted 6 lines
+ */
include/linux/dma-fence.h +5
-0+static inline spinlock_t *dma_fence_spinlock (struct dma_fence *fence) +{ + return test_bit(DMA_FENCE_FLAG_INLINE_LOCK_BIT, &fence->flags) ? + &fence->inline_lock : fence->extern_lock; +}You did not want to move this helper into "dma-buf: abstract fence locking" ?
I was avoiding that to keep the pre-requisite patch smaller, cause this change here seemed independent to that.
But thinking about it I could make a third patch which introduces dma_fence_spinlock() and changes all the container_of uses.
Quoted 3 lines
I think that would have been better to keep everything mechanical in one patch, and then this patch which changes behaviour does not touch any drivers but only dma-fence core.
Also, what about adding something like
dma_fence_container_of()in that patch as well?
I would rather like to avoid that. Using the spinlock pointer with container_of seemed to be a bit of a hack to me in the first place and I don't want to encourage people to do that in new code as well.
Regards,
Christian.
Quoted 26 lines
Regards,
Tvrtko
Quoted 20 lines
+
/**
*dma_fence_lock_irqsave- irqsave lock the fence
* @fence: the fence to lockinclude/linux/dma-fence.h +2
-2@@ -385,7 +403,7 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep)* Lock the fence, preventing it from changing to the signaled state. */ #define dma_fence_lock_irqsave(fence, flags) \ - spin_lock_irqsave(fence->lock, flags) + spin_lock_irqsave(dma_fence_spinlock(fence), flags) /*** dma_fence_unlock_irqrestore - unlock the fence and irqrestore @@ -395,7 +413,7 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep)* Unlock the fence, allowing it to change it's state to signaled again. */ #define dma_fence_unlock_irqrestore(fence, flags) \ - spin_unlock_irqrestore(fence->lock, flags) + spin_unlock_irqrestore(dma_fence_spinlock(fence), flags) #ifdef CONFIG_LOCKDEP bool dma_fence_begin_signalling(void);
Re: [PATCH 5/9] dma-buf: inline spinlock for fence protection v4
Quoted 30 lines · Christian König
Quoted 29 lines · Tvrtko Ursulin
Quoted 26 lines · Christian König
Implement per-fence spinlocks, allowing implementations to not give an external spinlock to protect the fence internal statei. Instead a spinlock embedded into the fence structure itself is used in this case.
Shared spinlocks have the problem that implementations need to guarantee that the lock live at least as long all fences referencing them.
Using a per-fence spinlock allows completely decoupling spinlock producer and consumer life times, simplifying the handling in most use cases.
v2: improve naming, coverage and function documentation
v3: fix one additional locking in the selftests
v4: separate out some changes to make the patch smaller,
fix one amdgpu crash found by CI systemsSigned-off-by Christian König <[email protected]>--- drivers/dma-buf/dma-fence.c | 25 +++++++++++++++++------- drivers/dma-buf/sync_debug.h | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 2 +- drivers/gpu/drm/drm_crtc.c | 2 +- drivers/gpu/drm/drm_writeback.c | 2 +- drivers/gpu/drm/nouveau/nouveau_fence.c | 3 ++- drivers/gpu/drm/qxl/qxl_release.c | 3 ++- drivers/gpu/drm/vmwgfx/vmwgfx_fence.c | 3 ++- drivers/gpu/drm/xe/xe_hw_fence.c | 3 ++-
Quoted 7 lines
Quoted 1 line
i915 needed changes too, based on the kbuild report.
Going to take a look now.
Quoted 1 line
Have you seen my note about the RCU sparse warning as well?
Nope, I must have missed that mail.
No worries, it was this one:
https://lore.kernel.org/dri-devel/[email protected]/
Quoted 17 lines
...
Quoted 5 lines
Quoted 5 lines
+/**
+ *dma_fence_spinlock- return pointer to the spinlock protecting the fence
+ * @fence: the fence to get the lock from
+ *
+ * Return either the pointer to the embedded or the external spin lock.Quoted 8 lines
include/linux/dma-fence.h dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep) · line 396
Quoted 6 lines
+ */
include/linux/dma-fence.h +5
-0+static inline spinlock_t *dma_fence_spinlock (struct dma_fence *fence) +{ + return test_bit(DMA_FENCE_FLAG_INLINE_LOCK_BIT, &fence->flags) ? + &fence->inline_lock : fence->extern_lock; +}You did not want to move this helper into "dma-buf: abstract fence locking" ?
I was avoiding that to keep the pre-requisite patch smaller, cause this change here seemed independent to that.
Quoted 7 lines
But thinking about it I could make a third patch which introduces
dma_fence_spinlock()and changes all thecontainer_ofuses.Quoted 3 lines
I think that would have been better to keep everything mechanical in one patch, and then this patch which changes behaviour does not touch any drivers but only dma-fence core.
Also, what about adding something like
dma_fence_container_of()in that patch as well?I would rather like to avoid that. Using the spinlock pointer with
container_ofseemed to be a bit of a hack to me in the first place and I don't want to encourage people to do that in new code as well.
I agree dma_fence_container_of would be a bit of questionable taste.
My thinking was from the angle, as you have dma_fence_spinlock() which is fence->lock accessor at runtime, while container_of is the same but needs to be compile time, and since there are the patches which touch a bunch of drivers purely mechanical, maybe wrap both. Then the patch which add the inline mode only changes dma-fence.h|c and so can be easily reverted should things go bad.
I don't however think this reasoning fully applies, since there would be no change in behaviour until each an every turns on the inline lock mode. So I guess for me I would be happy if dma_fence_spinlock() would be in the same patch as the dma_fence_lock_irqsave(). Logic being abstracting access to the lock can be justified to go together.
For container_of I don't know what to do. I don't see how it can be a separate patch if there is no accessor? In which case I guess just leave it as is.
Regards,
Tvrtko
Quoted 27 lines
Quoted 26 lines
Regards,
Tvrtko
Quoted 20 lines
include/linux/dma-fence.h +3
-2+ /*** dma_fence_lock_irqsave - irqsave lock the fence * @fence: the fence to lock @@ -385,7 +403,7 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep)* Lock the fence, preventing it from changing to the signaled state. */ #define dma_fence_lock_irqsave(fence, flags) \ - spin_lock_irqsave(fence->lock, flags) + spin_lock_irqsave(dma_fence_spinlock(fence), flags) /*** dma_fence_unlock_irqrestore - unlock the fence and irqrestore @@ -395,7 +413,7 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep)* Unlock the fence, allowing it to change it's state to signaled again. */ #define dma_fence_unlock_irqrestore(fence, flags) \ - spin_unlock_irqrestore(fence->lock, flags) + spin_unlock_irqrestore(dma_fence_spinlock(fence), flags) #ifdef CONFIG_LOCKDEP bool dma_fence_begin_signalling(void);
kernel test robot
build botRe: [PATCH 5/9] dma-buf: inline spinlock for fence protection v4
In reply to Christian Königkernel test robot noticed the following build errors:
Hi Christian,
kernel test robot noticed the following build errors:
[auto build test ERROR on drm-misc/drm-misc-next]
[also build test ERROR on drm-xe/drm-xe-next drm-tip/drm-tip next-20260126] [cannot apply to linus/master v6.19-rc7]
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Christian-K-nig/dma-buf-protected-fence-ops-by-RCU-v5/20260120-195535 base: https://gitlab.freedesktop.org/drm/misc/kernel.git drm-misc-next patch link: https://lore.kernel.org/r/20260120105655.7134-6-christian.koenig%40amd.com patch subject: [PATCH 5/9] dma-buf: inline spinlock for fence protection v4 config: x86_64-rhel-9.4 (https://download.01.org/0day-ci/archive/20260127/[email protected]/config) compiler: gcc-14 (Debian 14.2.0-19) 14.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260127/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from include/linux/spinlock.h:63,
from include/linux/sched.h:37,
from include/linux/kthread.h:6,
from drivers/gpu/drm/i915/gt/intel_breadcrumbs.c:6:
drivers/gpu/drm/i915/gt/intel_breadcrumbs.c: In function '__dma_fence_signal__notify':
>> drivers/gpu/drm/i915/gt/intel_breadcrumbs.c:151:34: error: 'struct dma_fence' has no member named 'lock'
151 | lockdep_assert_held(fence->lock);
| ^~ include/linux/lockdep.h:392:61: note: in definition of macro 'lockdep_assert_held'
392 | #define lockdep_assert_held(l) do { (void)(l); } while (0)
| ^
--
In file included from include/linux/debugobjects.h:6,
from drivers/gpu/drm/i915/i915_active.c:7:
drivers/gpu/drm/i915/i915_active.c: In function '__i915_active_fence_set':
>> drivers/gpu/drm/i915/i915_active.c:1048:32: error: 'struct dma_fence' has no member named 'lock'
1048 | spin_lock_irqsave(fence->lock, flags);
| ^~
include/linux/spinlock.h:244:48: note: in definition of macro 'raw_spin_lock_irqsave'
244 | flags = _raw_spin_lock_irqsave(lock); \
| ^~~~
drivers/gpu/drm/i915/i915_active.c:1048:9: note: in expansion of macro 'spin_lock_irqsave'
1048 | spin_lock_irqsave(fence->lock, flags);
| ^~~~~~~~~~~~~~~~~
drivers/gpu/drm/i915/i915_active.c:1050:38: error: 'struct dma_fence' has no member named 'lock'
1050 | spin_lock_nested(prev->lock, SINGLE_DEPTH_NESTING);
| ^~
include/linux/spinlock.h:235:44: note: in definition of macro 'raw_spin_lock_nested'
235 | _raw_spin_lock(((void)(subclass), (lock)))
| ^~~~
drivers/gpu/drm/i915/i915_active.c:1050:17: note: in expansion of macro 'spin_lock_nested'
1050 | spin_lock_nested(prev->lock, SINGLE_DEPTH_NESTING);
| ^~~~~~~~~~~~~~~~
drivers/gpu/drm/i915/i915_active.c:1064:41: error: 'struct dma_fence' has no member named 'lock'
1064 | spin_unlock(prev->lock);
| ^~
drivers/gpu/drm/i915/i915_active.c:1067:45: error: 'struct dma_fence' has no member named 'lock'
1067 | spin_unlock_irqrestore(fence->lock, flags);
| ^~
drivers/gpu/drm/i915/i915_active.c:1072:40: error: 'struct dma_fence' has no member named 'lock'
1072 | spin_lock_irqsave(fence->lock, flags);
| ^~
include/linux/spinlock.h:244:48: note: in definition of macro 'raw_spin_lock_irqsave'
244 | flags = _raw_spin_lock_irqsave(lock); \
| ^~~~
drivers/gpu/drm/i915/i915_active.c:1072:17: note: in expansion of macro 'spin_lock_irqsave'
1072 | spin_lock_irqsave(fence->lock, flags);
| ^~~~~~~~~~~~~~~~~
drivers/gpu/drm/i915/i915_active.c:1074:46: error: 'struct dma_fence' has no member named 'lock'
1074 | spin_lock_nested(prev->lock, SINGLE_DEPTH_NESTING);
| ^~
include/linux/spinlock.h:235:44: note: in definition of macro 'raw_spin_lock_nested'
235 | _raw_spin_lock(((void)(subclass), (lock)))
| ^~~~
drivers/gpu/drm/i915/i915_active.c:1074:25: note: in expansion of macro 'spin_lock_nested'
1074 | spin_lock_nested(prev->lock, SINGLE_DEPTH_NESTING);
| ^~~~~~~~~~~~~~~~
drivers/gpu/drm/i915/i915_active.c:1091:33: error: 'struct dma_fence' has no member named 'lock'
1091 | spin_unlock(prev->lock); /* serialise with prev->cb_list */
| ^~
drivers/gpu/drm/i915/i915_active.c:1094:37: error: 'struct dma_fence' has no member named 'lock'
1094 | spin_unlock_irqrestore(fence->lock, flags);
| ^~vim +151 drivers/gpu/drm/i915/gt/intel_breadcrumbs.c
0152b3b3f49b36 Chris Wilson 2019-05-08 144
0152b3b3f49b36 Chris Wilson 2019-05-08 145 static void
f2cb60e9a3881e Chris Wilson 2019-08-17 146 __dma_fence_signal__notify(struct dma_fence *fence,
f2cb60e9a3881e Chris Wilson 2019-08-17 147 const struct list_head *list)
0152b3b3f49b36 Chris Wilson 2019-05-08 148 {
0152b3b3f49b36 Chris Wilson 2019-05-08 149 struct dma_fence_cb *cur, *tmp;
0152b3b3f49b36 Chris Wilson 2019-05-08 150
0152b3b3f49b36 Chris Wilson 2019-05-08 @151 lockdep_assert_held(fence->lock);
0152b3b3f49b36 Chris Wilson 2019-05-08 152
f2cb60e9a3881e Chris Wilson 2019-08-17 153 list_for_each_entry_safe(cur, tmp, list, node) {
0152b3b3f49b36 Chris Wilson 2019-05-08 154 INIT_LIST_HEAD(&cur->node);
0152b3b3f49b36 Chris Wilson 2019-05-08 155 cur->func(fence, cur);
0152b3b3f49b36 Chris Wilson 2019-05-08 156 }
0152b3b3f49b36 Chris Wilson 2019-05-08 157 }
0152b3b3f49b36 Chris Wilson 2019-05-08 158 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
Christian König
reviewerPATCH 6/9 dma-buf/selftests: test RCU ops and inline lock v2
In reply to Christian KönigDrop the mock_fence and the kmem_cache, instead use the inline lock and test if the ops are properly dropped after signaling.
v2: move the RCU check to the end of the test
--- drivers/dma-buf/st-dma-fence.c | 44 ++++++++-------------------------- 1 file changed, 10 insertions(+), 34 deletions(-)
drivers/dma-buf/st-dma-fence.c +10 -34
@@ -14,43 +14,26 @@#include "selftest.h" -static struct kmem_cache *slab_fences; - -static struct mock_fence { -struct dma_fence base; -struct spinlock lock; -} *to_mock_fence (struct dma_fence *f) { - return container_of(f, struct mock_fence, base); -} -static const char *mock_name (struct dma_fence *f) {return "mock" ; } -static void mock_fence_release (struct dma_fence *f) -{ -kmem_cache_free (slab_fences,to_mock_fence (f)); -} -static const struct dma_fence_ops mock_ops = { .get_driver_name = mock_name, .get_timeline_name = mock_name, - .release = mock_fence_release, };static struct dma_fence *mock_fence (void ) { -struct mock_fence *f; +struct dma_fence *f; - f =kmem_cache_alloc (slab_fences,GFP_KERNEL ); + f =kmalloc (sizeof (*f),GFP_KERNEL );if (!f)return NULL ; -spin_lock_init (&f->lock); -dma_fence_init (&f->base, &mock_ops, &f->lock,0 ,0 ); - -return &f->base; +dma_fence_init (f, &mock_ops,NULL ,0 ,0 ); +return f; }static int sanitycheck (void *arg)
@@ -100,6 +83,11 @@ static int test_signaling(void *arg) goto err_free; } +if (rcu_dereference_protected (f->ops, true)) { +pr_err ("Fence ops not cleared on signal\n" ); + goto err_free; + } + err =0 ; err_free:dma_fence_put (f);
@@ -540,19 +528,7 @@ int dma_fence(void)SUBTEST (test_stub),SUBTEST (race_signal_callback), }; -int ret;pr_info ("sizeof(dma_fence)=%zu\n" ,sizeof (struct dma_fence )); - - slab_fences =KMEM_CACHE (mock_fence, -SLAB_TYPESAFE_BY_RCU | -SLAB_HWCACHE_ALIGN ); -if (!slab_fences) -return -ENOMEM ; - - ret =subtests (tests,NULL ); - -kmem_cache_destroy (slab_fences); - -return ret; +return subtests (tests,NULL ); }
Signature · 1 line
2.43.0
Christian König
reviewerPATCH 7/9 dma-buf: use inline lock for the stub fence v2
In reply to Christian KönigUsing the inline lock is now the recommended way for dma_fence implementations.
So use this approach for the framework's internal fences as well.
Also saves about 4 bytes for the external spinlock.
v2: drop unnecessary changes
--- drivers/dma-buf/dma-fence.c | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-)
drivers/dma-buf/dma-fence.c +2 -10
@@ -24,7 +24,6 @@ EXPORT_TRACEPOINT_SYMBOL(dma_fence_emit);EXPORT_TRACEPOINT_SYMBOL (dma_fence_enable_signal);EXPORT_TRACEPOINT_SYMBOL (dma_fence_signaled); -static DEFINE_SPINLOCK (dma_fence_stub_lock);static struct dma_fence dma_fence_stub;/*
@@ -123,12 +122,9 @@ static const struct dma_fence_ops dma_fence_stub_ops = {static int __init dma_fence_init_stub (void ) { -dma_fence_init (&dma_fence_stub, &dma_fence_stub_ops, - &dma_fence_stub_lock,0 ,0 ); - +dma_fence_init (&dma_fence_stub, &dma_fence_stub_ops,NULL ,0 ,0 );set_bit (DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT , &dma_fence_stub.flags); -dma_fence_signal (&dma_fence_stub);return 0 ; }
@@ -160,11 +156,7 @@ struct dma_fence *dma_fence_allocate_private_stub(ktime_t timestamp)if (fence ==NULL )return NULL ; -dma_fence_init (fence, - &dma_fence_stub_ops, - &dma_fence_stub_lock, -0 ,0 ); - +dma_fence_init (fence, &dma_fence_stub_ops,NULL ,0 ,0 );set_bit (DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT , &fence->flags);
Signature · 1 line
2.43.0
Re: [PATCH 7/9] dma-buf: use inline lock for the stub fence v2
Quoted 11 lines · Christian König
Using the inline lock is now the recommended way for
dma_fenceimplementations.So use this approach for the framework's internal fences as well.
Also saves about 4 bytes for the external spinlock.
v2: drop unnecessary changes
Signed-off-by Christian König <[email protected]>Reviewed-by Tvrtko Ursulin <[email protected]>
Quoted 43 lines
---
drivers/dma-buf/dma-fence.c | 12 ++----------1 file changed, 2 insertions(+), 10 deletions(-)drivers/dma-buf/dma-fence.c +0
-0@@ -24,7 +24,6 @@ EXPORT_TRACEPOINT_SYMBOL(dma_fence_emit); EXPORT_TRACEPOINT_SYMBOL(dma_fence_enable_signal); EXPORT_TRACEPOINT_SYMBOL(dma_fence_signaled);
-staticDEFINE_SPINLOCK(dma_fence_stub_lock);
static structdma_fencedma_fence_stub;
/*drivers/dma-buf/dma-fence.c +0
-0@@ -123,12 +122,9 @@ static const struct dma_fence_ops dma_fence_stub_ops = {
static int__initdma_fence_init_stub(void)
{drivers/dma-buf/dma-fence.c +1
-3-dma_fence_init (&dma_fence_stub, &dma_fence_stub_ops, - &dma_fence_stub_lock, 0, 0); - +dma_fence_init (&dma_fence_stub, &dma_fence_stub_ops,NULL ,0 ,0 );set_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT,
&dma_fence_stub.flags);
-
dma_fence_signal(&dma_fence_stub);
return 0;
}drivers/dma-buf/dma-fence.c +0
-0@@ -160,11 +156,7 @@ struct dma_fence *dma_fence_allocate_private_stub(ktime_t timestamp) if (fence == NULL) return NULL;drivers/dma-buf/dma-fence.c +1
-5-dma_fence_init (fence, - &dma_fence_stub_ops, - &dma_fence_stub_lock, - 0, 0); - +dma_fence_init (fence, &dma_fence_stub_ops,NULL ,0 ,0 );set_bit(DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT,
&fence->flags);
Christian König
reviewerPATCH 8/9 dma-buf: use inline lock for the dma-fence-array
In reply to Christian KönigUsing the inline lock is now the recommended way for dma_fence implementations.
So use this approach for the framework's internal fences as well.
Also saves about 4 bytes for the external spinlock.
--- drivers/dma-buf/dma-fence-array.c | 5 ++--- include/linux/dma-fence-array.h | 1 - 2 files changed, 2 insertions(+), 4 deletions(-)
drivers/dma-buf/dma-fence-array.c +2 -3
@@ -204,9 +204,8 @@ void dma_fence_array_init(struct dma_fence_array *array, array->num_fences = num_fences; -spin_lock_init (&array->lock); -dma_fence_init (&array->base, &dma_fence_array_ops, &array->lock, - context, seqno); +dma_fence_init (&array->base, &dma_fence_array_ops,NULL , context, + seqno);init_irq_work (&array->work, irq_dma_fence_array_work);atomic_set (&array->num_pending, signal_on_any ?1 : num_fences);
include/linux/dma-fence-array.h +0 -1
@@ -38,7 +38,6 @@ struct dma_fence_array_cb {struct dma_fence_array {struct dma_fence base; -spinlock_t lock;unsigned num_fences;atomic_t num_pending;struct dma_fence **fences;
Signature · 1 line
2.43.0
Christian König
reviewerPATCH 9/9 dma-buf: use inline lock for the dma-fence-chain
In reply to Christian KönigUsing the inline lock is now the recommended way for dma_fence implementations.
So use this approach for the framework's internal fences as well.
Also saves about 4 bytes for the external spinlock.
--- drivers/dma-buf/dma-fence-chain.c | 3 +-- include/linux/dma-fence-chain.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-)
drivers/dma-buf/dma-fence-chain.c +1 -2
@@ -245,7 +245,6 @@ void dma_fence_chain_init(struct dma_fence_chain *chain,struct dma_fence_chain *prev_chain =to_dma_fence_chain (prev);uint64_t context; -spin_lock_init (&chain->lock);rcu_assign_pointer (chain->prev, prev); chain->fence = fence; chain->prev_seqno =0 ;
@@ -261,7 +260,7 @@ void dma_fence_chain_init(struct dma_fence_chain *chain, seqno =max (prev->seqno, seqno); } -dma_fence_init64 (&chain->base, &dma_fence_chain_ops, &chain->lock, +dma_fence_init64 (&chain->base, &dma_fence_chain_ops,NULL , context, seqno);/*
include/linux/dma-fence-chain.h +0 -1
@@ -46,7 +46,6 @@ struct dma_fence_chain {*/ struct irq_work work; }; -spinlock_t lock; };
Signature · 1 line
2.43.0