Add dma_fence abstractions and DRM Jobqueue

impl_list_item_mod.rs calls container_of() without unsafe blocks at a couple of places. Since container_of() is an unsafe macro / function, the blocks are strictly necessary.

For unknown reasons, that problem was so far not visible and only gets visible once one utilizes the list implementation from within the core crate:

error[E0133]: call to unsafe function core::ptr::mut_ptr::<impl *mut T>::byte_sub is unsafe and requires unsafe block
--> rust/kernel/lib.rs:252:29

    |
252 |           let container_ptr = field_ptr.byte_sub(offset).cast::<$Container>();
    |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^ call to unsafe function
    |
   ::: rust/kernel/drm/jq.rs:98:1
    |
98  | / impl_list_item! {
99  | |     impl ListItem<0> for BasicItem { using ListLinks { self.links }; }
100 | | }
    | |_- in this macro invocation
    |
note: an unsafe function restricts its caller, but its body is safe by default
   --> rust/kernel/list/impl_list_item_mod.rs:216:13
    |
216 |               unsafe fn view_value(me: *mut $crate::list::ListLinks<$num>) -> *const Self {
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
   ::: rust/kernel/drm/jq.rs:98:1
    |
98  | / impl_list_item! {
99  | |     impl ListItem<0> for BasicItem { using ListLinks { self.links }; }
100 | | }
    | |_- in this macro invocation

= note: requested on the command line with -D unsafe-op-in-unsafe-fn = note: this error originates in the macro $crate::container_of which comes from the expansion of the macro impl_list_item

Add unsafe blocks to container_of to fix the issue.

Cc [email protected] # v6.17+

Fixes c77f85b347dd ("rust: list: remove OFFSET constants")

Suggested-by Alice Ryhl <[email protected]>

Signed-off-by Philipp Stanner <[email protected]>

---
 rust/kernel/list/impl_list_item_mod.rs | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

impl_list_item_mod.rs calls container_of() without unsafe blocks at a couple of places. Since container_of() is an unsafe macro / function, the blocks are strictly necessary.

For unknown reasons, that problem was so far not visible and only gets visible once one utilizes the list implementation from within the core crate:

error[E0133]: call to unsafe function core::ptr::mut_ptr::<impl *mut T>::byte_sub is unsafe and requires unsafe block
--> rust/kernel/lib.rs:252:29

    |
252 |           let container_ptr = field_ptr.byte_sub(offset).cast::<$Container>();
    |                               ^^^^^^^^^^^^^^^^^^^^^^^^^^ call to unsafe function
    |
   ::: rust/kernel/drm/jq.rs:98:1
    |
98  | / impl_list_item! {
99  | |     impl ListItem<0> for BasicItem { using ListLinks { self.links }; }
100 | | }
    | |_- in this macro invocation
    |
note: an unsafe function restricts its caller, but its body is safe by default
   --> rust/kernel/list/impl_list_item_mod.rs:216:13
    |
216 |               unsafe fn view_value(me: *mut $crate::list::ListLinks<$num>) -> *const Self {
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
   ::: rust/kernel/drm/jq.rs:98:1
    |
98  | / impl_list_item! {
99  | |     impl ListItem<0> for BasicItem { using ListLinks { self.links }; }
100 | | }
    | |_- in this macro invocation

= note: requested on the command line with -D unsafe-op-in-unsafe-fn = note: this error originates in the macro $crate::container_of which comes from the expansion of the macro impl_list_item

Add unsafe blocks to container_of to fix the issue.

Cc [email protected] # v6.17+

Fixes c77f85b347dd ("rust: list: remove OFFSET constants")

Suggested-by Alice Ryhl <[email protected]>

Signed-off-by Philipp Stanner <[email protected]>

---
 rust/kernel/list/impl_list_item_mod.rs | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

rust/kernel/list/impl_list_item_mod.rs +2 -2

index 202bc6f97c13..7052095efde5 100644
@@ -217,7 +217,7 @@ unsafe fn view_value(me: *mut $crate::list::ListLinks<$num>) -> *const Self {
                 // SAFETY: `me` originates from the most recent call to `prepare_to_insert`, so it
                 // points at the field `$field` in a value of type `Self`. Thus, reversing that
                 // operation is still in-bounds of the allocation.
-                $crate::container_of!(me, Self, $($field).*)
+                unsafe { $crate::container_of!(me, Self, $($field).*) }
             }
 
             // GUARANTEES:

@@ -242,7 +242,7 @@ unsafe fn post_remove(me: *mut $crate::list::ListLinks<$num>) -> *const Self {
                 // SAFETY: `me` originates from the most recent call to `prepare_to_insert`, so it
                 // points at the field `$field` in a value of type `Self`. Thus, reversing that
                 // operation is still in-bounds of the allocation.
-                $crate::container_of!(me, Self, $($field).*)
+                unsafe { $crate::container_of!(me, Self, $($field).*) }
             }
         }
     )*};

rust/kernel/sync/dma_fence.rs new file +4 -0

+
+    /// Mark the beginning of a DmaFence signalling critical section. Should be called once a fence
+    /// gets published.
+    ///

On Tue, 3 Feb 2026 09:14:01 +0100
Philipp Stanner <[email protected]> wrote:
Unfortunately, I don't know how to translate that in rust, but we need a way to check if any path code path does a DmaFence.signal(), go back to the entry point (for a WorkItem, that would be WorkItem::run() for instance), and make it a DmaFenceSignallingPath. Not only that, but we need to know all the deps that make it so this path can be called (if I take the WorkItem example, that would be the path that leads to the WorkItem being scheduled).

Quoted 9 lines · Boris Brezillon

On Tue,  3 Feb 2026 09:14:01 +0100
Philipp Stanner <[email protected]> wrote:
Unfortunately, I don't know how to translate that in rust, but we need a way to check if any path code path does a DmaFence.signal(), go back to the entry point (for a WorkItem, that would be WorkItem::run() for instance), and make it a DmaFenceSignallingPath. Not only that, but we need to know all the deps that make it so this path can be called (if I take the WorkItem example, that would be the path that leads to the WorkItem being scheduled).

I think we need a guard object for this that is not Send, just like for any other lock.

Internally, those markers rely on lockdep, i.e. they just acquire and release a "fake" lock.

Quoted 11 lines · Danilo Krummrich

Quoted 9 lines · Boris Brezillon

On Tue,  3 Feb 2026 09:14:01 +0100
Philipp Stanner <[email protected]> wrote:
Unfortunately, I don't know how to translate that in rust, but we need a way to check if any path code path does a DmaFence.signal(), go back to the entry point (for a WorkItem, that would be WorkItem::run() for instance), and make it a DmaFenceSignallingPath. Not only that, but we need to know all the deps that make it so this path can be called (if I take the WorkItem example, that would be the path that leads to the WorkItem being scheduled).

Quoted 5 lines

I think we need a guard object for this that is not Send, just like for any other lock.

Internally, those markers rely on lockdep, i.e. they just acquire and release a "fake" lock.

The guard object would be created through fence.begin_signalling(), wouldn't it?

And when it drops you call dma_fence_end_signalling()?

How would that ensure that the driver actually marks the signalling region correctly?

Quoted 12 lines · On Mon, 09 Feb 2026 09:19:46 +0100 Philipp Stanner <[email protected]> wrote:

Quoted 11 lines · Danilo Krummrich

Quoted 9 lines · Boris Brezillon

On Tue,  3 Feb 2026 09:14:01 +0100
Philipp Stanner <[email protected]> wrote:
Unfortunately, I don't know how to translate that in rust, but we need a way to check if any path code path does a DmaFence.signal(), go back to the entry point (for a WorkItem, that would be WorkItem::run() for instance), and make it a DmaFenceSignallingPath. Not only that, but we need to know all the deps that make it so this path can be called (if I take the WorkItem example, that would be the path that leads to the WorkItem being scheduled).

Quoted 7 lines

Quoted 5 lines

I think we need a guard object for this that is not Send, just like for any other lock.

Internally, those markers rely on lockdep, i.e. they just acquire and release a "fake" lock.

The guard object would be created through fence.begin_signalling(), wouldn't it?

It shouldn't be a (&self)-method, because at the start of a DMA signaling path, you don't necessarily know which fence you're going to signal (you might actually signal several of them).

Quoted 1 line

And when it drops you call dma_fence_end_signalling()?

Yep, dma_fence_end_signalling() should be called when the guard is dropped.

Quoted 2 lines

How would that ensure that the driver actually marks the signalling region correctly?

Nothing, and that's a problem we have in C: you have no way of telling which code section is going to be a DMA-signaling path. I can't think of any way to make that safer in rust, unfortunately. The best I can think of would be to

- Have a special DmaFenceSignalWorkItem (wrapper a WorkItem with extra constraints) that's designed for DMA-fence signaling, and that takes the DmaSignaling guard around the ::run() call.
- We would then need to ensure that any code path scheduling this work item is also in a DMA-signaling path by taking a ref to the DmaSignalingGuard. This of course doesn't guarantee that the section is wide enough to prevent any non-authorized operations in any path leading to this WorkItem scheduling, but it would at least force the caller to consider the problem.

Quoted 40 lines · Boris Brezillon

Quoted 12 lines · On Mon, 09 Feb 2026 09:19:46 +0100 Philipp Stanner <[email protected]> wrote:

Quoted 11 lines · Danilo Krummrich

Quoted 9 lines · Boris Brezillon

On Tue,  3 Feb 2026 09:14:01 +0100
Philipp Stanner <[email protected]> wrote:
Unfortunately, I don't know how to translate that in rust, but we need a way to check if any path code path does a DmaFence.signal(), go back to the entry point (for a WorkItem, that would be WorkItem::run() for instance), and make it a DmaFenceSignallingPath. Not only that, but we need to know all the deps that make it so this path can be called (if I take the WorkItem example, that would be the path that leads to the WorkItem being scheduled).

Quoted 7 lines

Quoted 5 lines

I think we need a guard object for this that is not Send, just like for any other lock.

Internally, those markers rely on lockdep, i.e. they just acquire and release a "fake" lock.

The guard object would be created through fence.begin_signalling(), wouldn't it?

It shouldn't be a (&self)-method, because at the start of a DMA signaling path, you don't necessarily know which fence you're going to signal (you might actually signal several of them).

Quoted 1 line

And when it drops you call dma_fence_end_signalling()?

Yep, dma_fence_end_signalling() should be called when the guard is dropped.

Quoted 2 lines

How would that ensure that the driver actually marks the signalling region correctly?

Nothing, and that's a problem we have in C: you have no way of telling which code section is going to be a DMA-signaling path. I can't think of any way to make that safer in rust, unfortunately. The best I can think of would be to

Quoted 9 lines

- Have a special DmaFenceSignalWorkItem (wrapper a WorkItem with extra constraints) that's designed for DMA-fence signaling, and that takes the DmaSignaling guard around the ::run() call.
- We would then need to ensure that any code path scheduling this work item is also in a DMA-signaling path by taking a ref to the DmaSignalingGuard. This of course doesn't guarantee that the section is wide enough to prevent any non-authorized operations in any path leading to this WorkItem scheduling, but it would at least force the caller to consider the problem.

On the C side I have a patch set which does something very similar.

It's basically a WARN_ON_ONCE() which triggers as soon as you try to signal a DMA fence from an IOCTL, or more specific process context.

Signaling a DMA fence from interrupt context, a work item or kernel thread is still allowed, there is just the hole that you can schedule a work item from process context as well.

The major problem with that patch set is that we have tons of very hacky signaling paths in drivers already because we initially didn't knew how much trouble getting this wrong causes.

I'm strongly in favor of getting this right for the rust side from the beginning and enforcing strict rules for every code trying to implement a DMA fence.

Quoted 15 lines · Christian König

On the C side I have a patch set which does something very similar.

It's basically a WARN_ON_ONCE() which triggers as soon as you try to signal a DMA fence from an IOCTL, or more specific process context.

Signaling a DMA fence from interrupt context, a work item or kernel thread is still allowed, there is just the hole that you can schedule a work item from process context as well.

The major problem with that patch set is that we have tons of very hacky signaling paths in drivers already because we initially didn't knew how much trouble getting this wrong causes.

Quoted 3 lines

I'm strongly in favor of getting this right for the rust side from the beginning and enforcing strict rules for every code trying to implement a DMA fence.

Hmm. Could you say a bit more about what the rules are? I just re-read the comments in dma-fence.c, but I have some questions.

Quoted 16 lines · Alice Ryhl

Quoted 15 lines · Christian König

On the C side I have a patch set which does something very similar.

It's basically a WARN_ON_ONCE() which triggers as soon as you try to signal a DMA fence from an IOCTL, or more specific process context.

Signaling a DMA fence from interrupt context, a work item or kernel thread is still allowed, there is just the hole that you can schedule a work item from process context as well.

The major problem with that patch set is that we have tons of very hacky signaling paths in drivers already because we initially didn't knew how much trouble getting this wrong causes.

Quoted 6 lines

Quoted 3 lines

I'm strongly in favor of getting this right for the rust side from the beginning and enforcing strict rules for every code trying to implement a DMA fence.

Hmm. Could you say a bit more about what the rules are? I just re-read the comments in dma-fence.c, but I have some questions.

The rules need to be written down. Elaborately and in detail, once and for all.

We're having those discussions about the mysterious "dma fence rules" for years, but no one has ever seen them, despite knowing that they exist.

They seem to live only in the heads of a small number of GPU developers who figured out through long years of experience what works and what doesn't.

There are reasons why the state writes the laws on paper somewhere. So that you can read them and have a source of authority..

That would end much of our endless misunderstandings and repetitive discussions.

P.

Quoted 60 lines · Christian König

Quoted 40 lines · Boris Brezillon

Quoted 12 lines · On Mon, 09 Feb 2026 09:19:46 +0100 Philipp Stanner <[email protected]> wrote:

Quoted 11 lines · Danilo Krummrich

Quoted 9 lines · Boris Brezillon

On Tue,  3 Feb 2026 09:14:01 +0100
Philipp Stanner <[email protected]> wrote:
Unfortunately, I don't know how to translate that in rust, but we need a way to check if any path code path does a DmaFence.signal(), go back to the entry point (for a WorkItem, that would be WorkItem::run() for instance), and make it a DmaFenceSignallingPath. Not only that, but we need to know all the deps that make it so this path can be called (if I take the WorkItem example, that would be the path that leads to the WorkItem being scheduled).

Quoted 7 lines

Quoted 5 lines

I think we need a guard object for this that is not Send, just like for any other lock.

Internally, those markers rely on lockdep, i.e. they just acquire and release a "fake" lock.

The guard object would be created through fence.begin_signalling(), wouldn't it?

It shouldn't be a (&self)-method, because at the start of a DMA signaling path, you don't necessarily know which fence you're going to signal (you might actually signal several of them).

Quoted 1 line

And when it drops you call dma_fence_end_signalling()?

Yep, dma_fence_end_signalling() should be called when the guard is dropped.

Quoted 2 lines

How would that ensure that the driver actually marks the signalling region correctly?

Nothing, and that's a problem we have in C: you have no way of telling which code section is going to be a DMA-signaling path. I can't think of any way to make that safer in rust, unfortunately. The best I can think of would be to

Quoted 9 lines

- Have a special DmaFenceSignalWorkItem (wrapper a WorkItem with extra constraints) that's designed for DMA-fence signaling, and that takes the DmaSignaling guard around the ::run() call.
- We would then need to ensure that any code path scheduling this work item is also in a DMA-signaling path by taking a ref to the DmaSignalingGuard. This of course doesn't guarantee that the section is wide enough to prevent any non-authorized operations in any path leading to this WorkItem scheduling, but it would at least force the caller to consider the problem.

On the C side I have a patch set which does something very similar.

It's basically a WARN_ON_ONCE() which triggers as soon as you try to signal a DMA fence from an IOCTL, or more specific process context.

Signaling a DMA fence from interrupt context, a work item or kernel thread is still allowed, there is just the hole that you can schedule a work item from process context as well.

Quoted 7 lines

The major problem with that patch set is that we have tons of very hacky signaling paths in drivers already because we initially didn't knew how much trouble getting this wrong causes.

I'm strongly in favor of getting this right for the rust side from the beginning and enforcing strict rules for every code trying to implement a DMA fence.

Hmm. Could you say a bit more about what the rules are? I just re-read the comments in dma-fence.c, but I have some questions.

First, how does the signalling annotation work when the signalling path crosses thread boundaries?

For example, let's say I call an ioctl to
perform an async VM_BIND, then the dma fence signalling critical path starts in the ioctl, but then it moves into a workqueue and finishes there, right?

Second, it looks like we have the same challenge as with irq locks where you must properly nest dma_fence_begin_signalling() regions, and can't e.g. do this:

c1 = dma_fence_begin_signalling()
c2 = dma_fence_begin_signalling()
dma_fence_end_signalling(c1)
dma_fence_end_signalling(c2)

Quoted 69 lines · On Tue, 10 Feb 2026 08:38:00 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 60 lines · Christian König

Quoted 40 lines · Boris Brezillon

Quoted 12 lines · On Mon, 09 Feb 2026 09:19:46 +0100 Philipp Stanner <[email protected]> wrote:

Quoted 11 lines · Danilo Krummrich

Quoted 9 lines · Boris Brezillon

On Tue,  3 Feb 2026 09:14:01 +0100
Philipp Stanner <[email protected]> wrote:
Unfortunately, I don't know how to translate that in rust, but we need a way to check if any path code path does a DmaFence.signal(), go back to the entry point (for a WorkItem, that would be WorkItem::run() for instance), and make it a DmaFenceSignallingPath. Not only that, but we need to know all the deps that make it so this path can be called (if I take the WorkItem example, that would be the path that leads to the WorkItem being scheduled).

Quoted 7 lines

Quoted 5 lines

I think we need a guard object for this that is not Send, just like for any other lock.

Internally, those markers rely on lockdep, i.e. they just acquire and release a "fake" lock.

The guard object would be created through fence.begin_signalling(), wouldn't it?

It shouldn't be a (&self)-method, because at the start of a DMA signaling path, you don't necessarily know which fence you're going to signal (you might actually signal several of them).

Quoted 1 line

And when it drops you call dma_fence_end_signalling()?

Yep, dma_fence_end_signalling() should be called when the guard is dropped.

Quoted 2 lines

How would that ensure that the driver actually marks the signalling region correctly?

Nothing, and that's a problem we have in C: you have no way of telling which code section is going to be a DMA-signaling path. I can't think of any way to make that safer in rust, unfortunately. The best I can think of would be to

Quoted 9 lines

- Have a special DmaFenceSignalWorkItem (wrapper a WorkItem with extra constraints) that's designed for DMA-fence signaling, and that takes the DmaSignaling guard around the ::run() call.
- We would then need to ensure that any code path scheduling this work item is also in a DMA-signaling path by taking a ref to the DmaSignalingGuard. This of course doesn't guarantee that the section is wide enough to prevent any non-authorized operations in any path leading to this WorkItem scheduling, but it would at least force the caller to consider the problem.

On the C side I have a patch set which does something very similar.

It's basically a WARN_ON_ONCE() which triggers as soon as you try to signal a DMA fence from an IOCTL, or more specific process context.

Signaling a DMA fence from interrupt context, a work item or kernel thread is still allowed, there is just the hole that you can schedule a work item from process context as well.

Quoted 7 lines

The major problem with that patch set is that we have tons of very hacky signaling paths in drivers already because we initially didn't knew how much trouble getting this wrong causes.

I'm strongly in favor of getting this right for the rust side from the beginning and enforcing strict rules for every code trying to implement a DMA fence.

Quoted 5 lines

Hmm. Could you say a bit more about what the rules are? I just re-read the comments in dma-fence.c, but I have some questions.

First, how does the signalling annotation work when the signalling path crosses thread boundaries?

It's not supposed to cross the thread boundary at all. The annotation is per-thread, and it that sense, it matches the lock guard model perfectly.

Quoted 4 lines

For example, let's say I call an ioctl to
perform an async VM_BIND, then the dma fence signalling critical path starts in the ioctl, but then it moves into a workqueue and finishes there, right?

It's a bit trickier. The fence signalling path usually doesn't exist in the submitting ioctl until the submission becomes effective and the emitted fences are exposed to the outside world. That is, when:
- syncobjs are updated to point to this new fence
- fencefd pointing to this new fence is returned
- fence is added to the dma_resvs inside the gem/dma_buf objects
- ... (there might be other cases I forgot about)

In the submission path, what's important is that no blocking allocation is done between the moment the fence is exposed, and the moment it's queued. In practice what happens is that the job this fence is bound to is queued even before the fences are exposed, so if anything, what we should ensure is the ordering, and having a guarantee that a job being queued means it's going to be dequeued and executed soon enough.

The second DMA signaling path exists in the context of the workqueue/item dequeuing a job from the JobQueue (or drm_sched) and pushing it to the HW. Then there's the IRQ handler being called to inform the GPU is done executing this job, which might in some cases lead to another work item being queued for further processing from which the dma_fence is signaled. In other cases, the dma_fence is signaled directly from the IRQ handler. All of these contexts are considered being part of the DMA-signaling path. But it's not like the fence signaling annotation is passed around, because the cookies returned by dma_fence_begin_signalling() are only valid in a single thread context, IIRC.

Quoted 8 lines

Second, it looks like we have the same challenge as with irq locks where you must properly nest dma_fence_begin_signalling() regions, and can't e.g. do this:

c1 = dma_fence_begin_signalling()
c2 = dma_fence_begin_signalling()
dma_fence_end_signalling(c1)
dma_fence_end_signalling(c2)

I think that's the case yes, you have to end in reverse being order.

One way you can see this is by looking at what we require of the workqueue. For all this to work, it's pretty important that we never schedule anything on the workqueue that's not signalling safe, since otherwise you could have a deadlock where the workqueue is executes some random job calling kmalloc(GFP_KERNEL) and then blocks on our fence, meaning that the VM_BIND job never gets scheduled since the workqueue is never freed up. Deadlock.

Quoted 7 lines · Alice Ryhl

One way you can see this is by looking at what we require of the workqueue. For all this to work, it's pretty important that we never schedule anything on the workqueue that's not signalling safe, since otherwise you could have a deadlock where the workqueue is executes some random job calling kmalloc(GFP_KERNEL) and then blocks on our fence, meaning that the VM_BIND job never gets scheduled since the workqueue is never freed up. Deadlock.

Yes, I also pointed this out multiple times in the past in the context of C GPU scheduler discussions. It really depends on the workqueue and how it is used.

In the C GPU scheduler the driver can pass its own workqueue to the scheduler, which means that the driver has to ensure that at least one out of the wq->max_active works is free for the scheduler to make progress on the scheduler's run and free job work.

Or in other words, there must be no more than wq->max_active - 1 works that execute code violating the DMA fence signalling rules.

This is also why the JobQ needs its own workqueue and relying on the system WQ is unsound.

In case of an ordered workqueue, it is always a potential deadlock to schedule work that does non-atomic allocations or takes a lock that is used elsewhere for non-atomic allocations of course.

Quoted 12 lines · Danilo Krummrich

Quoted 7 lines · Alice Ryhl

One way you can see this is by looking at what we require of the workqueue. For all this to work, it's pretty important that we never schedule anything on the workqueue that's not signalling safe, since otherwise you could have a deadlock where the workqueue is executes some random job calling kmalloc(GFP_KERNEL) and then blocks on our fence, meaning that the VM_BIND job never gets scheduled since the workqueue is never freed up. Deadlock.

Yes, I also pointed this out multiple times in the past in the context of C GPU scheduler discussions. It really depends on the workqueue and how it is used.

Quoted 7 lines

In the C GPU scheduler the driver can pass its own workqueue to the scheduler, which means that the driver has to ensure that at least one out of the wq->max_active works is free for the scheduler to make progress on the scheduler's run and free job work.

Or in other words, there must be no more than wq->max_active - 1 works that execute code violating the DMA fence signalling rules.

*And* the workqueue must be created with WQ_MEM_RECLAIM so that work items can also start under memory pressure and not potentially cycle back into the memory management to wait for a dma_fence to signal.

But apart from that your explanation is perfectly correct, yes.

Quoted 18 lines · Christian König

Quoted 17 lines · Danilo Krummrich

Quoted 7 lines · Alice Ryhl

One way you can see this is by looking at what we require of the workqueue. For all this to work, it's pretty important that we never schedule anything on the workqueue that's not signalling safe, since otherwise you could have a deadlock where the workqueue is executes some random job calling kmalloc(GFP_KERNEL) and then blocks on our fence, meaning that the VM_BIND job never gets scheduled since the workqueue is never freed up. Deadlock.

Yes, I also pointed this out multiple times in the past in the context of C GPU scheduler discussions. It really depends on the workqueue and how it is used.

In the C GPU scheduler the driver can pass its own workqueue to the scheduler, which means that the driver has to ensure that at least one out of the wq->max_active works is free for the scheduler to make progress on the scheduler's run and free job work.

Quoted 2 lines

Quoted 2 lines

Or in other words, there must be no more than wq->max_active - 1 works that execute code violating the DMA fence signalling rules.

Ouch, is that really the best way to do that? Why not two workqueues?

Quoted 18 lines · Christian König

Quoted 17 lines · Danilo Krummrich

Quoted 7 lines · Alice Ryhl

One way you can see this is by looking at what we require of the workqueue. For all this to work, it's pretty important that we never schedule anything on the workqueue that's not signalling safe, since otherwise you could have a deadlock where the workqueue is executes some random job calling kmalloc(GFP_KERNEL) and then blocks on our fence, meaning that the VM_BIND job never gets scheduled since the workqueue is never freed up. Deadlock.

Yes, I also pointed this out multiple times in the past in the context of C GPU scheduler discussions. It really depends on the workqueue and how it is used.

In the C GPU scheduler the driver can pass its own workqueue to the scheduler, which means that the driver has to ensure that at least one out of the wq->max_active works is free for the scheduler to make progress on the scheduler's run and free job work.

Quoted 2 lines

Quoted 2 lines

Or in other words, there must be no more than wq->max_active - 1 works that execute code violating the DMA fence signalling rules.

Ouch, is that really the best way to do that? Why not two workqueues?

(Cc: Xe maintainers)

Quoted 4 lines · Alice Ryhl

Quoted 3 lines · Christian König

Quoted 2 lines · Danilo Krummrich

Quoted 1 line · Alice Ryhl

Quoted 4 lines

Quoted 3 lines

Quoted 3 lines

Or in other words, there must be no more than wq->max_active - 1 works that execute code violating the DMA fence signalling rules.

Quoted 1 line

Ouch, is that really the best way to do that? Why not two workqueues?

Most drivers making use of this re-use the same workqueue for multiple GPU scheduler instances in firmware scheduling mode (i.e. 1:1 relationship between scheduler and entity). This is equivalent to the JobQ use-case.

Note that we will have one JobQ instance per userspace queue, so sharing the workqueue between JobQ instances can make sense.

Quoted 7 lines · Danilo Krummrich

(Cc: Xe maintainers)

Quoted 4 lines · Alice Ryhl

Quoted 3 lines · Christian König

Quoted 2 lines · Danilo Krummrich

Quoted 1 line · Alice Ryhl

[…]

Quoted 6 lines

Quoted 5 lines

Quoted 3 lines

Quoted 3 lines

Or in other words, there must be no more than wq->max_active - 1 works that execute code violating the DMA fence signalling rules.

Ouch, is that really the best way to do that? Why not two workqueues?

Quoted 6 lines

Most drivers making use of this re-use the same workqueue for multiple GPU scheduler instances in firmware scheduling mode (i.e. 1:1 relationship between scheduler and entity). This is equivalent to the JobQ use-case.

Note that we will have one JobQ instance per userspace queue, so sharing the workqueue between JobQ instances can make sense.

Why, what for?

(Cc: Xe maintainers)

Quoted 22 lines · Alice Ryhl

Quoted 20 lines · Christian König

Quoted 19 lines · Danilo Krummrich

Quoted 7 lines · Alice Ryhl

One way you can see this is by looking at what we require of the workqueue. For all this to work, it's pretty important that we never schedule anything on the workqueue that's not signalling safe, since otherwise you could have a deadlock where the workqueue is executes some random job calling kmalloc(GFP_KERNEL) and then blocks on our fence, meaning that the VM_BIND job never gets scheduled since the workqueue is never freed up. Deadlock.

Yes, I also pointed this out multiple times in the past in the context of C GPU scheduler discussions. It really depends on the workqueue and how it is used.

In the C GPU scheduler the driver can pass its own workqueue to the scheduler, which means that the driver has to ensure that at least one out of the wq->max_active works is free for the scheduler to make progress on the scheduler's run and free job work.

Or in other words, there must be no more than wq->max_active - 1 works that execute code violating the DMA fence signalling rules.

Quoted 1 line

Ouch, is that really the best way to do that? Why not two workqueues?

Most drivers making use of this re-use the same workqueue for multiple GPU scheduler instances in firmware scheduling mode (i.e. 1:1 relationship between scheduler and entity). This is equivalent to the JobQ use-case.

Note that we will have one JobQ instance per userspace queue, so sharing the workqueue between JobQ instances can make sense.

Besides that, IIRC Xe was re-using the workqueue for something else, but that doesn't seem to be the case anymore. I can only find [1], which more seems like some custom GPU scheduler extention [2] to me...

[1] https://elixir.bootlin.com/linux/v6.18.6/source/drivers/gpu/drm/xe/xe_gpu_scheduler.c#L40
[2] https://elixir.bootlin.com/linux/v6.18.6/source/drivers/gpu/drm/xe/xe_gpu_scheduler_types.h#L28

Quoted 31 lines · On Wed, 11 Feb 2026 10:57:27 +0100 "Danilo Krummrich" <[email protected]> wrote:

(Cc: Xe maintainers)

Quoted 19 lines · Alice Ryhl

Quoted 18 lines · Christian König

Quoted 17 lines · Danilo Krummrich

Quoted 7 lines · Alice Ryhl

One way you can see this is by looking at what we require of the workqueue. For all this to work, it's pretty important that we never schedule anything on the workqueue that's not signalling safe, since otherwise you could have a deadlock where the workqueue is executes some random job calling kmalloc(GFP_KERNEL) and then blocks on our fence, meaning that the VM_BIND job never gets scheduled since the workqueue is never freed up. Deadlock.

Yes, I also pointed this out multiple times in the past in the context of C GPU scheduler discussions. It really depends on the workqueue and how it is used.

In the C GPU scheduler the driver can pass its own workqueue to the scheduler, which means that the driver has to ensure that at least one out of the wq->max_active works is free for the scheduler to make progress on the scheduler's run and free job work.

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

Or in other words, there must be no more than wq->max_active - 1 works that execute code violating the DMA fence signalling rules.

Ouch, is that really the best way to do that? Why not two workqueues?

Most drivers making use of this re-use the same workqueue for multiple GPU scheduler instances in firmware scheduling mode (i.e. 1:1 relationship between scheduler and entity). This is equivalent to the JobQ use-case.

Quoted 2 lines

Note that we will have one JobQ instance per userspace queue, so sharing the workqueue between JobQ instances can make sense.

Definitely, but I think that's orthogonal to allowing this common workqueue to be used for work items that don't comply with the dma-fence signalling rules, isn't it?

Quoted 3 lines

Besides that, IIRC Xe was re-using the workqueue for something else, but that doesn't seem to be the case anymore. I can only find [1], which more seems like some custom GPU scheduler extention [2] to me...

Yep, I think it can be the problematic case. It doesn't mean we can't schedule work items that don't signal fences, but I think it'd be simpler if we were forcing those to follow the same rules (no blocking alloc, no locks taken that are also taken in other paths were blocking allocs happen, etc) regardless of this wq->max_active value.

Quoted 3 lines

[1] https://elixir.bootlin.com/linux/v6.18.6/source/drivers/gpu/drm/xe/xe_gpu_scheduler.c#L40
[2] https://elixir.bootlin.com/linux/v6.18.6/source/drivers/gpu/drm/xe/xe_gpu_scheduler_types.h#L28

Quoted 34 lines · Boris Brezillon

Quoted 31 lines · On Wed, 11 Feb 2026 10:57:27 +0100 "Danilo Krummrich" <[email protected]> wrote:

(Cc: Xe maintainers)

Quoted 19 lines · Alice Ryhl

Quoted 18 lines · Christian König

Quoted 17 lines · Danilo Krummrich

Quoted 7 lines · Alice Ryhl

One way you can see this is by looking at what we require of the workqueue. For all this to work, it's pretty important that we never schedule anything on the workqueue that's not signalling safe, since otherwise you could have a deadlock where the workqueue is executes some random job calling kmalloc(GFP_KERNEL) and then blocks on our fence, meaning that the VM_BIND job never gets scheduled since the workqueue is never freed up. Deadlock.

Yes, I also pointed this out multiple times in the past in the context of C GPU scheduler discussions. It really depends on the workqueue and how it is used.

In the C GPU scheduler the driver can pass its own workqueue to the scheduler, which means that the driver has to ensure that at least one out of the wq->max_active works is free for the scheduler to make progress on the scheduler's run and free job work.

Quoted 4 lines

Quoted 2 lines

Quoted 2 lines

Or in other words, there must be no more than wq->max_active - 1 works that execute code violating the DMA fence signalling rules.

Ouch, is that really the best way to do that? Why not two workqueues?

Most drivers making use of this re-use the same workqueue for multiple GPU scheduler instances in firmware scheduling mode (i.e. 1:1 relationship between scheduler and entity). This is equivalent to the JobQ use-case.

Quoted 6 lines

Quoted 2 lines

Note that we will have one JobQ instance per userspace queue, so sharing the workqueue between JobQ instances can make sense.

Definitely, but I think that's orthogonal to allowing this common workqueue to be used for work items that don't comply with the dma-fence signalling rules, isn't it?

Yes and no. If we allow passing around shared WQs without a corresponding type abstraction we open the door for drivers to abuse it the schedule their own work.

I.e. sharing a workqueue between JobQs is fine, but we have to ensure they can't be used for anything else.

Quoted 14 lines

Quoted 4 lines

Besides that, IIRC Xe was re-using the workqueue for something else, but that doesn't seem to be the case anymore. I can only find [1], which more seems like some custom GPU scheduler extention [2] to me...

Yep, I think it can be the problematic case. It doesn't mean we can't schedule work items that don't signal fences, but I think it'd be simpler if we were forcing those to follow the same rules (no blocking alloc, no locks taken that are also taken in other paths were blocking allocs happen, etc) regardless of this wq->max_active value.

Quoted 3 lines

[1] https://elixir.bootlin.com/linux/v6.18.6/source/drivers/gpu/drm/xe/xe_gpu_scheduler.c#L40
[2] https://elixir.bootlin.com/linux/v6.18.6/source/drivers/gpu/drm/xe/xe_gpu_scheduler_types.h#L28

Quoted 2 lines · On Wed, 11 Feb 2026 12:00:30 +0100 "Danilo Krummrich" <[email protected]> wrote:

I.e. sharing a workqueue between JobQs is fine, but we have to ensure they can't be used for anything else.

Totally agree with that, and that's where I was going with this special DmaFenceWorkqueue wrapper/abstract, that would only accept scheduling MaySignalDmaFencesWorkItem objects.

Quoted 5 lines · Boris Brezillon

Quoted 2 lines · On Wed, 11 Feb 2026 12:00:30 +0100 "Danilo Krummrich" <[email protected]> wrote:

I.e. sharing a workqueue between JobQs is fine, but we have to ensure they can't be used for anything else.

Quoted 3 lines

Totally agree with that, and that's where I was going with this special DmaFenceWorkqueue wrapper/abstract, that would only accept scheduling MaySignalDmaFencesWorkItem objects.

Not sure if it has to be that complicated (for a first shot). At least for the JobQ it would probably be enough to have a helper to create a new, let's say, struct JobQueueWorker that encapsulates a (reference counted) workqueue, but does not give access to it outside of jobq.rs.

Quoted 6 lines · On Wed, 11 Feb 2026 15:38:32 +0100 "Danilo Krummrich" <[email protected]> wrote:

Quoted 5 lines · Boris Brezillon

Quoted 2 lines · On Wed, 11 Feb 2026 12:00:30 +0100 "Danilo Krummrich" <[email protected]> wrote:

I.e. sharing a workqueue between JobQs is fine, but we have to ensure they can't be used for anything else.

Quoted 8 lines

Quoted 3 lines

Totally agree with that, and that's where I was going with this special DmaFenceWorkqueue wrapper/abstract, that would only accept scheduling MaySignalDmaFencesWorkItem objects.

Not sure if it has to be that complicated (for a first shot). At least for the JobQ it would probably be enough to have a helper to create a new, let's say, struct JobQueueWorker that encapsulates a (reference counted) workqueue, but does not give access to it outside of jobq.rs.

Except we need to schedule some work items that are in the DMA-signaling path but not directly controlled by the jobq.rs implementation (see [1] for the post-execution work we schedule in panthor).

The two options I can think of are:

1. Add a an unsafe interface to schedule work items on the wq attached to JobQ. Safety requirements in that case being compliance with the DMA-fence signalling rules.
2. The thing I was describing before, where we add the concept of DmaFenceWorkqueue that can only take MaySignalDmaFencesWorkItem. We can then have a DmaFenceWorkqueue that's global, and pass it to the JobQueue so it can use it for its own work item.

We could start with option 1, sure, but since we're going to need to schedule post-execution work items that have to be considered part of the DMA-signalling path, I'd rather have these concepts clearly defined from the start.

Mind if I give this DmaFenceWorkqueue/MaySignalDmaFencesWorkItem a try to see what it looks like a get the discussion going from there (hopefully it's just a thin wrapper around a regular Workqueue/WorkItem, with an extra dma_fence_signalling annotation in the WorkItem::run() path), or are you completely against the idea?

[1]https://elixir.bootlin.com/linux/v6.19-rc5/source/drivers/gpu/drm/panthor/panthor_sched.c#L1913

Quoted 38 lines · Boris Brezillon

Quoted 6 lines · On Wed, 11 Feb 2026 15:38:32 +0100 "Danilo Krummrich" <[email protected]> wrote:

Quoted 5 lines · Boris Brezillon

Quoted 2 lines · On Wed, 11 Feb 2026 12:00:30 +0100 "Danilo Krummrich" <[email protected]> wrote:

I.e. sharing a workqueue between JobQs is fine, but we have to ensure they can't be used for anything else.

Quoted 8 lines

Quoted 3 lines

Totally agree with that, and that's where I was going with this special DmaFenceWorkqueue wrapper/abstract, that would only accept scheduling MaySignalDmaFencesWorkItem objects.

Not sure if it has to be that complicated (for a first shot). At least for the JobQ it would probably be enough to have a helper to create a new, let's say, struct JobQueueWorker that encapsulates a (reference counted) workqueue, but does not give access to it outside of jobq.rs.

Except we need to schedule some work items that are in the DMA-signaling path but not directly controlled by the jobq.rs implementation (see [1] for the post-execution work we schedule in panthor).

The two options I can think of are:

1. Add a an unsafe interface to schedule work items on the wq attached to JobQ. Safety requirements in that case being compliance with the DMA-fence signalling rules.
2. The thing I was describing before, where we add the concept of DmaFenceWorkqueue that can only take MaySignalDmaFencesWorkItem. We can then have a DmaFenceWorkqueue that's global, and pass it to the JobQueue so it can use it for its own work item.

We could start with option 1, sure, but since we're going to need to schedule post-execution work items that have to be considered part of the DMA-signalling path, I'd rather have these concepts clearly defined from the start.

Quoted 5 lines

Mind if I give this DmaFenceWorkqueue/MaySignalDmaFencesWorkItem a try to see what it looks like a get the discussion going from there (hopefully it's just a thin wrapper around a regular Workqueue/WorkItem, with an extra dma_fence_signalling annotation in the WorkItem::run() path), or are you completely against the idea?

Not at all, I think it's a good generalization.

But I'm very skeptical about the "we allow drivers to schedule arbitrary work on the (shared) JobQueue workqueue" part. I think drivers can just have a separate workqueue for such use-cases.

Okay, that would be one DmaFenceWorkqueue only used for the driver JobQueue instances (or one per-instance if the driver wants that) wrapped into some object that doesn't expose it as a generic workqueue, so only JobQueue instances can use it. And then drivers are free to instantiate their own DmaFenceWorkqueue for anything else that's still in the DMA-signalling path, but not directly related to JobQueues. I think I'd be fine with that.

Quoted 6 lines · On Wed, 11 Feb 2026 15:38:32 +0100 "Danilo Krummrich" <[email protected]> wrote:

Quoted 5 lines · Boris Brezillon

Quoted 2 lines · On Wed, 11 Feb 2026 12:00:30 +0100 "Danilo Krummrich" <[email protected]> wrote:

I.e. sharing a workqueue between JobQs is fine, but we have to ensure they can't be used for anything else.

Quoted 8 lines

Quoted 3 lines

Totally agree with that, and that's where I was going with this special DmaFenceWorkqueue wrapper/abstract, that would only accept scheduling MaySignalDmaFencesWorkItem objects.

Not sure if it has to be that complicated (for a first shot). At least for the JobQ it would probably be enough to have a helper to create a new, let's say, struct JobQueueWorker that encapsulates a (reference counted) workqueue, but does not give access to it outside of jobq.rs.

Except we need to schedule some work items that are in the DMA-signaling path but not directly controlled by the jobq.rs implementation (see [1] for the post-execution work we schedule in panthor).

The two options I can think of are:

1. Add a an unsafe interface to schedule work items on the wq attached to JobQ. Safety requirements in that case being compliance with the DMA-fence signalling rules.

2. The thing I was describing before, where we add the concept of DmaFenceWorkqueue that can only take MaySignalDmaFencesWorkItem. We can then have a DmaFenceWorkqueue that's global, and pass it to the JobQueue so it can use it for its own work item.

We could start with option 1, sure, but since we're going to need to schedule post-execution work items that have to be considered part of the DMA-signalling path, I'd rather have these concepts clearly defined from the start.

Mind if I give this DmaFenceWorkqueue/MaySignalDmaFencesWorkItem a try to see what it looks like a get the discussion going from there (hopefully it's just a thin wrapper around a regular Workqueue/WorkItem, with an extra dma_fence_signalling annotation in the WorkItem::run() path), or are you completely against the idea?

[1]https://elixir.bootlin.com/linux/v6.19-rc5/source/drivers/gpu/drm/panthor/panthor_sched.c#L1913

Quoted 102 lines · Boris Brezillon

Quoted 69 lines · On Tue, 10 Feb 2026 08:38:00 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 60 lines · Christian König

Quoted 40 lines · Boris Brezillon

Quoted 12 lines · On Mon, 09 Feb 2026 09:19:46 +0100 Philipp Stanner <[email protected]> wrote:

Quoted 11 lines · Danilo Krummrich

Quoted 9 lines · Boris Brezillon

On Tue,  3 Feb 2026 09:14:01 +0100
Philipp Stanner <[email protected]> wrote:
Unfortunately, I don't know how to translate that in rust, but we need a way to check if any path code path does a DmaFence.signal(), go back to the entry point (for a WorkItem, that would be WorkItem::run() for instance), and make it a DmaFenceSignallingPath. Not only that, but we need to know all the deps that make it so this path can be called (if I take the WorkItem example, that would be the path that leads to the WorkItem being scheduled).

Quoted 7 lines

Quoted 5 lines

I think we need a guard object for this that is not Send, just like for any other lock.

Internally, those markers rely on lockdep, i.e. they just acquire and release a "fake" lock.

The guard object would be created through fence.begin_signalling(), wouldn't it?

It shouldn't be a (&self)-method, because at the start of a DMA signaling path, you don't necessarily know which fence you're going to signal (you might actually signal several of them).

Quoted 1 line

And when it drops you call dma_fence_end_signalling()?

Yep, dma_fence_end_signalling() should be called when the guard is dropped.

Quoted 2 lines

How would that ensure that the driver actually marks the signalling region correctly?

Nothing, and that's a problem we have in C: you have no way of telling which code section is going to be a DMA-signaling path. I can't think of any way to make that safer in rust, unfortunately. The best I can think of would be to

Quoted 9 lines

- Have a special DmaFenceSignalWorkItem (wrapper a WorkItem with extra constraints) that's designed for DMA-fence signaling, and that takes the DmaSignaling guard around the ::run() call.
- We would then need to ensure that any code path scheduling this work item is also in a DMA-signaling path by taking a ref to the DmaSignalingGuard. This of course doesn't guarantee that the section is wide enough to prevent any non-authorized operations in any path leading to this WorkItem scheduling, but it would at least force the caller to consider the problem.

On the C side I have a patch set which does something very similar.

It's basically a WARN_ON_ONCE() which triggers as soon as you try to signal a DMA fence from an IOCTL, or more specific process context.

Signaling a DMA fence from interrupt context, a work item or kernel thread is still allowed, there is just the hole that you can schedule a work item from process context as well.

Quoted 7 lines

The major problem with that patch set is that we have tons of very hacky signaling paths in drivers already because we initially didn't knew how much trouble getting this wrong causes.

I'm strongly in favor of getting this right for the rust side from the beginning and enforcing strict rules for every code trying to implement a DMA fence.

Quoted 5 lines

Hmm. Could you say a bit more about what the rules are? I just re-read the comments in dma-fence.c, but I have some questions.

First, how does the signalling annotation work when the signalling path crosses thread boundaries?

It's not supposed to cross the thread boundary at all. The annotation is per-thread, and it that sense, it matches the lock guard model perfectly.

Quoted 4 lines

For example, let's say I call an ioctl to
perform an async VM_BIND, then the dma fence signalling critical path starts in the ioctl, but then it moves into a workqueue and finishes there, right?

It's a bit trickier. The fence signalling path usually doesn't exist in the submitting ioctl until the submission becomes effective and the emitted fences are exposed to the outside world. That is, when:
- syncobjs are updated to point to this new fence
- fencefd pointing to this new fence is returned
- fence is added to the dma_resvs inside the gem/dma_buf objects
- ... (there might be other cases I forgot about)

In the submission path, what's important is that no blocking allocation is done between the moment the fence is exposed, and the moment it's queued. In practice what happens is that the job this fence is bound to is queued even before the fences are exposed, so if anything, what we should ensure is the ordering, and having a guarantee that a job being queued means it's going to be dequeued and executed soon enough.

Quoted 11 lines

The second DMA signaling path exists in the context of the workqueue/item dequeuing a job from the JobQueue (or drm_sched) and pushing it to the HW. Then there's the IRQ handler being called to inform the GPU is done executing this job, which might in some cases lead to another work item being queued for further processing from which the dma_fence is signaled. In other cases, the dma_fence is signaled directly from the IRQ handler. All of these contexts are considered being part of the DMA-signaling path. But it's not like the fence signaling annotation is passed around, because the cookies returned by dma_fence_begin_signalling() are only valid in a single thread context, IIRC.

Ok I understand what's going on now.

You talk about it as-if there are two signalling paths, one in the ioctl and another in the workqueue. But that sounds like a workaround to make it work with how dma_fence_begin_signalling() is implemented, and not the true situation. (Added: looks like Christian confirms this.)

One way you can see this is by looking at what we require of the workqueue. For all this to work, it's pretty important that we never schedule anything on the workqueue that's not signalling safe, since otherwise you could have a deadlock where the workqueue is executes some random job calling kmalloc(GFP_KERNEL) and then blocks on our fence, meaning that the VM_BIND job never gets scheduled since the workqueue is never freed up. Deadlock.

And the correct way to model the above in lockdep is to have the DMA fence lockdep key be nested *outside* the lockdep key of the workqueue. Because there is a step in the signalling path where you wait for other jobs to complete before the signalling path is able to continue.

And the meaning of such a lockdep dependency is exactly that the critical region moves from one thread to another.

Perhaps we could have an API like this:

// Split the DriverDmaFence into two separate fence concepts:
struct PrivateFence { ... }
struct PublishedFence { ... }

/// The owner of this value must ensure that this fence is signalled. struct MustBeSignalled<'fence> { ... }
/// Proof value indicating that the fence has either already been /// signalled, or it will be. The lifetime ensures that you cannot mix /// up the proof value.
struct WillBeSignalled<'fence> { ... }

/// Create a PublishedFence by entering a region that promises to signal /// it.
///
/// The only way to return from the region closure it to construct a /// WillBeSignalled value, and the only way to do that (see below) is to /// signal it, or spawn a workqueue job that promises to signal it. /// (Since the only way for that workqueue job to exit is to construct a /// second WillBeSignalled value.)
fn dma_fence_begin_signalling(

    fence: PrivateFence,
    region: impl for<'f> FnOnce(MustBeSignalled<'f>) -> WillBeSignalled<'f>,
) -> PublishedFence {
    let cookie = bindings::dma_fence_begin_signalling();

    region(<create token here>);

    bindings::dma_fence_end_signalling(cookie);

    fence.i_promise_it_will_be_signalled();
}

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }
}

trait DmaFenceWorkItem {
    fn run<'f>(self, fence: MustBeSignalled<'f>) -> WillBeSignalled<'f>;
}

with this API, the ioctl can do this:

let published_fence = dma_fence_begin_signalling(|fence| {
fence.transfer_to_wq(my_wq, my_work_item)
}); somehow_publish_the_fence_to_userspace(published_fence);

And we're ensured that the fence is really signalled because the signature of the work item closure is such that it can only return from DmaFenceWorkItem::run() by signalling the fence (or spawning it to a second workqueue, which in turn promises to signal it).

Of course, the signature only enforces that the fence is (or will be) signalled if you return. One can still put an infinite loop inside dma_fence_begin_signalling() and avoid signalling it, but there's nothing we can do about that.

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,

        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }
}

Quoted 14 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,

Quoted 8 lines

        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

Sorry, I'm still trying to connect the dots here. I get that the intent is to ensure the pseudo-lock ordering is always:

   -> dma_fence_lockdep_map
      -> wq->lockdep_map

but how can this order be the same in the WorkItem execution path? My interpretation of process_one_work() makes me think we'll end up with

-> wq->lockdep_map

    -> work->run()
       -> WorkItem::run()
          -> dma_fence_lockdep_map
             -> DmaFenceSignalingWorkItem::run()
                ...

Am I missing something? Is there a way you can insert the dma_fence_lockdep_map acquisition before the wq->lockdep_map one in the execution path?

Quoted 42 lines · Boris Brezillon

Quoted 14 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,

Quoted 8 lines

        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);  

Sorry, I'm still trying to connect the dots here. I get that the intent is to ensure the pseudo-lock ordering is always:

   -> dma_fence_lockdep_map
      -> wq->lockdep_map  

but how can this order be the same in the WorkItem execution path? My interpretation of process_one_work() makes me think we'll end up with

-> wq->lockdep_map

    -> work->run()
       -> WorkItem::run()
          -> dma_fence_lockdep_map
             -> DmaFenceSignalingWorkItem::run()  
                ...

Quoted 3 lines

Am I missing something? Is there a way you can insert the dma_fence_lockdep_map acquisition before the wq->lockdep_map one in the execution path?

Conceptually, the dma_fence_lockdep_map is already taken by the time you get to WorkItem::run() because it was taken all the way back in the ioctl, so WorkItem::run() does not need to reacquire it.

Now, of course that does not translate cleanly to how lockdep does things, so in lockdep we do have to re-acquire it in WorkItem::run(). You can do that by setting the trylock bit when calling lock_acquire() on dma_fence_lockdep_map. This has the correct semantics because trylock does not create an edge from wq->lockdep_map to dma_fence_lockdep_map.

Quoted 51 lines · On Tue, 10 Feb 2026 11:45:36 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 42 lines · Boris Brezillon

Quoted 14 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,

Quoted 8 lines

        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);  

Sorry, I'm still trying to connect the dots here. I get that the intent is to ensure the pseudo-lock ordering is always:

   -> dma_fence_lockdep_map
      -> wq->lockdep_map  

but how can this order be the same in the WorkItem execution path? My interpretation of process_one_work() makes me think we'll end up with

-> wq->lockdep_map

    -> work->run()
       -> WorkItem::run()
          -> dma_fence_lockdep_map
             -> DmaFenceSignalingWorkItem::run()  
                ...

Quoted 3 lines

Am I missing something? Is there a way you can insert the dma_fence_lockdep_map acquisition before the wq->lockdep_map one in the execution path?

Conceptually, the dma_fence_lockdep_map is already taken by the time you get to WorkItem::run() because it was taken all the way back in the ioctl, so WorkItem::run() does not need to reacquire it.

Quoted 5 lines

Now, of course that does not translate cleanly to how lockdep does things, so in lockdep we do have to re-acquire it in WorkItem::run(). You can do that by setting the trylock bit when calling lock_acquire() on dma_fence_lockdep_map. This has the correct semantics because trylock does not create an edge from wq->lockdep_map to dma_fence_lockdep_map.

Ah, I never noticed dma_fence_begin_signalling() was recording a try_lock not a regular lock. I guess it would do then.

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }

}

Quoted 26 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

Quoted 4 lines

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }

I guess what's still missing is some sort of transfer_to_hw() function and way to flag the IRQ handler taking over the fence signaling token.

Quoted 29 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

Quoted 8 lines

Quoted 4 lines

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }  

I guess what's still missing is some sort of transfer_to_hw() function and way to flag the IRQ handler taking over the fence signaling token.

Yes, transfer to hardware needs to be another piece of logic similar to transfer to wq. And I imagine there are many ways such a transfer to hardware could work.

Unless you have a timeout on it, in which case the WillBeSignalled is satisfied by the fact you have a timeout alone, and the signalling that happens from the irq is just an opportunistic signal from outside the dma fence signalling critical path.

From dma-fence.c:

* * The only exception are fast paths and opportunistic signalling code, which

 *   calls dma_fence_signal() purely as an optimization, but is not required to
 *   guarantee completion of a &dma_fence. The usual example is a wait IOCTL
 *   which calls dma_fence_signal(), while the mandatory completion path goes
 *   through a hardware interrupt and possible job completion worker.

Well ... unless triggering timeouts can block on GFP_KERNEL allocations...

Quoted 39 lines · On Tue, 10 Feb 2026 13:15:31 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 29 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

Quoted 8 lines

Quoted 4 lines

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }  

I guess what's still missing is some sort of transfer_to_hw() function and way to flag the IRQ handler taking over the fence signaling token.

Quoted 8 lines

Yes, transfer to hardware needs to be another piece of logic similar to transfer to wq. And I imagine there are many ways such a transfer to hardware could work.

Unless you have a timeout on it, in which case the WillBeSignalled is satisfied by the fact you have a timeout alone, and the signalling that happens from the irq is just an opportunistic signal from outside the dma fence signalling critical path.

Yes and no. If it deadlocks in the completion WorkItem because of allocations (or any of the forbidden use cases), I think we want to catch that, because that's a sign fences are likely to end up with timeouts when they should have otherwise been signaled properly.

Quoted 2 lines

Well ... unless triggering timeouts can block on GFP_KERNEL allocations...

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

Quoted 59 lines · Boris Brezillon

Quoted 39 lines · On Tue, 10 Feb 2026 13:15:31 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 29 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

Quoted 8 lines

Quoted 4 lines

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }  

I guess what's still missing is some sort of transfer_to_hw() function and way to flag the IRQ handler taking over the fence signaling token.

Quoted 8 lines

Yes, transfer to hardware needs to be another piece of logic similar to transfer to wq. And I imagine there are many ways such a transfer to hardware could work.

Unless you have a timeout on it, in which case the WillBeSignalled is satisfied by the fact you have a timeout alone, and the signalling that happens from the irq is just an opportunistic signal from outside the dma fence signalling critical path.

Yes and no. If it deadlocks in the completion WorkItem because of allocations (or any of the forbidden use cases), I think we want to catch that, because that's a sign fences are likely to end up with timeouts when they should have otherwise been signaled properly.

Quoted 2 lines

Well ... unless triggering timeouts can block on GFP_KERNEL allocations...

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

1. A in signalling path, B is not
2. B in signalling path, A is not
3. A and B both in signalling path

But the downside of choosing (1.) or (2.) is that if you declare that event B is not in the signalling path, then B can kmalloc(GFP_KERNEL), which may deadlock on itself until event A happens, and if A is a timeout that could be a long time, so this scenario is undesirable even if technically it's not a deadlock because it eventually unblocks itself.

So we should choose option (3.) and declare that both timeout and hw irq codepaths are signalling paths.

Alice

Quoted 6 lines · Alice Ryhl

Quoted 5 lines · Boris Brezillon

On Tue, 10 Feb 2026 13:15:31 +0000
Alice Ryhl <[email protected]> wrote:

Quoted 6 lines

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

Quoted 2 lines

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

That doesn't happen in practice.

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

is_signaled()? Or are you talking about different reference holders

We used to have some exceptions to that, especially around aborting submissions, but those turned out to be a really bad idea as well.

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

Quoted 7 lines · Christian König

Quoted 6 lines · Alice Ryhl

Quoted 5 lines · Boris Brezillon

On Tue, 10 Feb 2026 13:15:31 +0000
Alice Ryhl <[email protected]> wrote:

[…]

Quoted 11 lines

Quoted 8 lines

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

That doesn't happen in practice.

Quoted 3 lines

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

Are you now referring to the fast-path callbacks like fence->ops-

Quoted 1 line

is_signaled()? Or are you talking about different reference holders

which might want to signal?

Quoted 4 lines

We used to have some exceptions to that, especially around aborting submissions, but those turned out to be a really bad idea as well.

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

An idea that is floating around is to move the entire fence signaling functionality into the dma fence context. That would have exclusive access, and could also finally guarantee that fences must be signaled in order.

P.

Quoted 26 lines · Philipp Stanner

Quoted 7 lines · Christian König

Quoted 6 lines · Alice Ryhl

Quoted 5 lines · Boris Brezillon

On Tue, 10 Feb 2026 13:15:31 +0000
Alice Ryhl <[email protected]> wrote:

[…]

Quoted 14 lines

Quoted 8 lines

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

That doesn't happen in practice.

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

Quoted 3 lines

Are you now referring to the fast-path callbacks like fence->ops-

Quoted 1 line

is_signaled()? Or are you talking about different reference holders

which might want to signal?

Yes, I'm referring to the is_signaled() callback.

When you have multiple reference holders which can all signal the fence by calling dma_fence_signal then there is clearly something going wrong.

Quoted 31 lines · Christian König

Quoted 29 lines · Philipp Stanner

Quoted 7 lines · Christian König

Quoted 6 lines · Alice Ryhl

Quoted 5 lines · Boris Brezillon

On Tue, 10 Feb 2026 13:15:31 +0000
Alice Ryhl <[email protected]> wrote:

[…]

Quoted 6 lines

Quoted 6 lines

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

Quoted 8 lines

Quoted 2 lines

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

That doesn't happen in practice.

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

Are you now referring to the fast-path callbacks like fence->ops-

Quoted 1 line

is_signaled()? Or are you talking about different reference holders

which might want to signal?

Quoted 3 lines

Yes, I'm referring to the is_signaled() callback.

When you have multiple reference holders which can all signal the fence by calling dma_fence_signal then there is clearly something going wrong.

From our previous discussions it always seemed to me that there is already something wrong when is_signaled() fastpaths are being utilized. Remember the mess we have in Nouveau because of that?

I agree that it sounds very sane to just have 1 party that can signal a fence.

However, that also implies that the fastpath is wrong because

a) a consumer of a fence can use that to poll on the fence and
b) because it breaks good naming pattern, where a check whether a fence is signaled can result in the fence being signaled.

So I would kill that fast path for good.

In the past you mentioned that you had users who were wondering why there fences are not signalling (like yeah, if you don't call dma_fence_signal(), then your fence will not signal). What ever the confusion was, this time we have the chance to set it straight.

P.

Quoted 60 lines · Alice Ryhl

Quoted 59 lines · Boris Brezillon

Quoted 39 lines · On Tue, 10 Feb 2026 13:15:31 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 29 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

Quoted 8 lines

Quoted 4 lines

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }  

I guess what's still missing is some sort of transfer_to_hw() function and way to flag the IRQ handler taking over the fence signaling token.

Quoted 8 lines

Yes, transfer to hardware needs to be another piece of logic similar to transfer to wq. And I imagine there are many ways such a transfer to hardware could work.

Unless you have a timeout on it, in which case the WillBeSignalled is satisfied by the fact you have a timeout alone, and the signalling that happens from the irq is just an opportunistic signal from outside the dma fence signalling critical path.

Yes and no. If it deadlocks in the completion WorkItem because of allocations (or any of the forbidden use cases), I think we want to catch that, because that's a sign fences are likely to end up with timeouts when they should have otherwise been signaled properly.

Quoted 2 lines

Well ... unless triggering timeouts can block on GFP_KERNEL allocations...

Quoted 8 lines

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

That doesn't happen in practice.

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

We used to have some exceptions to that, especially around aborting submissions, but those turned out to be a really bad idea as well.

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

Quoted 79 lines · Christian König

Quoted 60 lines · Alice Ryhl

Quoted 59 lines · Boris Brezillon

Quoted 39 lines · On Tue, 10 Feb 2026 13:15:31 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 29 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

Quoted 8 lines

Quoted 4 lines

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }  

I guess what's still missing is some sort of transfer_to_hw() function and way to flag the IRQ handler taking over the fence signaling token.

Quoted 8 lines

Yes, transfer to hardware needs to be another piece of logic similar to transfer to wq. And I imagine there are many ways such a transfer to hardware could work.

Unless you have a timeout on it, in which case the WillBeSignalled is satisfied by the fact you have a timeout alone, and the signalling that happens from the irq is just an opportunistic signal from outside the dma fence signalling critical path.

Yes and no. If it deadlocks in the completion WorkItem because of allocations (or any of the forbidden use cases), I think we want to catch that, because that's a sign fences are likely to end up with timeouts when they should have otherwise been signaled properly.

Quoted 2 lines

Well ... unless triggering timeouts can block on GFP_KERNEL allocations...

Quoted 8 lines

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

That doesn't happen in practice.

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

Quoted 5 lines

We used to have some exceptions to that, especially around aborting submissions, but those turned out to be a really bad idea as well.

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

I'm not really convinced by this.

First, the timeout path must be a fence signalling path because the reason you have a timeout in the first place is because the hw might never signal the fence. So if the timeout path deadlocks on a kmalloc(GFP_KERNEL) and the hw never comes around to wake you up, boom.

Second, for the reasons I mentioned you also want the signal-from-irq path to be a fence signalling critical path, because if we allow you to kmalloc(GFP_KERNEL) on the path from getting notification from hardware to signalling the fence, then you may deadlock until the timeout triggers ... even if the deadlock is only temporary, we should still avoid such cases IMO. Thus, the hw signal path should also be a fence signalling critical path.

Alice

Quoted 1 line · Alice Ryhl

Quoted 1 line

Quoted 15 lines

Quoted 9 lines

That doesn't happen in practice.

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

Quoted 5 lines

We used to have some exceptions to that, especially around aborting submissions, but those turned out to be a really bad idea as well.

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

Quoted 6 lines

I'm not really convinced by this.

First, the timeout path must be a fence signalling path because the reason you have a timeout in the first place is because the hw might never signal the fence. So if the timeout path deadlocks on a kmalloc(GFP_KERNEL) and the hw never comes around to wake you up, boom.

Mhm, good point. On the other hand the timeout handling should probably be considered part of the normal signaling path.

In other words the timeout handler either disables the normal signaling path (e.g. by disabling the interrupt) and then reset the HW or it tells the HW to force signal some work and observes the result.

So it can be that the timeout handler finishes only after the fence is signaled from the normal signaling paths.

Quoted 8 lines

Second, for the reasons I mentioned you also want the signal-from-irq path to be a fence signalling critical path, because if we allow you to kmalloc(GFP_KERNEL) on the path from getting notification from hardware to signalling the fence, then you may deadlock until the timeout triggers ... even if the deadlock is only temporary, we should still avoid such cases IMO. Thus, the hw signal path should also be a fence signalling critical path.

As far as I remember we didn't had any of such cases.

You can't call kmalloc(GFP_KERNEL) from an interrupt handler, so you would need something like irq->work item->kmalloc(GFP_KERNEL)->signaling and I think that's unlikely to be implemented this way.

But yeah, it is still something which should be prevented somehow.

Quoted 86 lines · Alice Ryhl

Quoted 84 lines · Christian König

Quoted 60 lines · Alice Ryhl

Quoted 59 lines · Boris Brezillon

Quoted 39 lines · On Tue, 10 Feb 2026 13:15:31 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 29 lines · Boris Brezillon

On Tue, 10 Feb 2026 10:15:04 +0000
Alice Ryhl <[email protected]> wrote:

Quoted 26 lines

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it. fn signal_fence(self) -> WillBeSignalled { self.fence.signal(); self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to // lockdep that the signalling path "blocks" on arbitrary jobs // from this wq completing bindings::lock_acquire(&wq->key); bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

Quoted 8 lines

Quoted 4 lines

        // The signature of DmaFenceWorkItem::run() promises to arrange // for it to be signalled. self.i_promise_it_will_be_signalled()
    }

I guess what's still missing is some sort of transfer_to_hw() function and way to flag the IRQ handler taking over the fence signaling token.

Quoted 8 lines

Yes, transfer to hardware needs to be another piece of logic similar to transfer to wq. And I imagine there are many ways such a transfer to hardware could work.

Unless you have a timeout on it, in which case the WillBeSignalled is satisfied by the fact you have a timeout alone, and the signalling that happens from the irq is just an opportunistic signal from outside the dma fence signalling critical path.

Yes and no. If it deadlocks in the completion WorkItem because of allocations (or any of the forbidden use cases), I think we want to catch that, because that's a sign fences are likely to end up with timeouts when they should have otherwise been signaled properly.

Quoted 2 lines

Well ... unless triggering timeouts can block on GFP_KERNEL allocations...

Quoted 8 lines

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

That doesn't happen in practice.

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

We used to have some exceptions to that, especially around aborting submissions, but those turned out to be a really bad idea as well.

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

Quoted 6 lines

I'm not really convinced by this.

First, the timeout path must be a fence signalling path because the reason you have a timeout in the first place is because the hw might never signal the fence. So if the timeout path deadlocks on a kmalloc(GFP_KERNEL) and the hw never comes around to wake you up, boom.

Mhm, good point. On the other hand the timeout handling should probably be considered part of the normal signaling path.

Quoted 87 lines · Christian König

Quoted 86 lines · Alice Ryhl

Quoted 84 lines · Christian König

Quoted 60 lines · Alice Ryhl

Quoted 59 lines · Boris Brezillon

Quoted 39 lines · On Tue, 10 Feb 2026 13:15:31 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 29 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

Quoted 8 lines

Quoted 4 lines

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }

I guess what's still missing is some sort of transfer_to_hw() function and way to flag the IRQ handler taking over the fence signaling token.

Quoted 8 lines

Yes, transfer to hardware needs to be another piece of logic similar to transfer to wq. And I imagine there are many ways such a transfer to hardware could work.

Unless you have a timeout on it, in which case the WillBeSignalled is satisfied by the fact you have a timeout alone, and the signalling that happens from the irq is just an opportunistic signal from outside the dma fence signalling critical path.

Yes and no. If it deadlocks in the completion WorkItem because of allocations (or any of the forbidden use cases), I think we want to catch that, because that's a sign fences are likely to end up with timeouts when they should have otherwise been signaled properly.

Quoted 2 lines

Well ... unless triggering timeouts can block on GFP_KERNEL allocations...

Quoted 8 lines

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

That doesn't happen in practice.

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

We used to have some exceptions to that, especially around aborting submissions, but those turned out to be a really bad idea as well.

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

Quoted 8 lines

Quoted 6 lines

I'm not really convinced by this.

First, the timeout path must be a fence signalling path because the reason you have a timeout in the first place is because the hw might never signal the fence. So if the timeout path deadlocks on a kmalloc(GFP_KERNEL) and the hw never comes around to wake you up, boom.

Mhm, good point. On the other hand the timeout handling should probably be considered part of the normal signaling path.

Why would anyone want to allocate in a timeout path in the first place – especially for jobqueue?

Timeout -> close the associated ring. Done. JobQueue will signal the done_fences with -ECANCELED.

What would the driver want to allocate in its timeout path, i.e.: timeout callback.

Quoted 101 lines · Philipp Stanner

Quoted 87 lines · Christian König

Quoted 86 lines · Alice Ryhl

Quoted 84 lines · Christian König

Quoted 60 lines · Alice Ryhl

Quoted 59 lines · Boris Brezillon

Quoted 39 lines · On Tue, 10 Feb 2026 13:15:31 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 29 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

impl MustBeSignalled<'_> {
    /// Drivers generally should not use this one.
    fn i_promise_it_will_be_signalled(self) -> WillBeSignalled { ... }

    /// One way to ensure the fence has been signalled is to signal it.
    fn signal_fence(self) -> WillBeSignalled {
        self.fence.signal();
        self.i_promise_it_will_be_signalled()
    }

    /// Another way to ensure the fence will be signalled is to spawn a
    /// workqueue item that promises to signal it.
    fn transfer_to_wq(
        self,
        wq: &Workqueue,
        item: impl DmaFenceWorkItem,
    ) -> WillBeSignalled {
        // briefly obtain the lock class of the wq to indicate to
        // lockdep that the signalling path "blocks" on arbitrary jobs
        // from this wq completing
        bindings::lock_acquire(&wq->key);
        bindings::lock_release(&wq->key);

        // enqueue the job
        wq.enqueue(item, wq);

Quoted 8 lines

Quoted 4 lines

        // The signature of DmaFenceWorkItem::run() promises to arrange
        // for it to be signalled.
        self.i_promise_it_will_be_signalled()
    }

I guess what's still missing is some sort of transfer_to_hw() function and way to flag the IRQ handler taking over the fence signaling token.

Quoted 8 lines

Yes, transfer to hardware needs to be another piece of logic similar to transfer to wq. And I imagine there are many ways such a transfer to hardware could work.

Unless you have a timeout on it, in which case the WillBeSignalled is satisfied by the fact you have a timeout alone, and the signalling that happens from the irq is just an opportunistic signal from outside the dma fence signalling critical path.

Yes and no. If it deadlocks in the completion WorkItem because of allocations (or any of the forbidden use cases), I think we want to catch that, because that's a sign fences are likely to end up with timeouts when they should have otherwise been signaled properly.

Quoted 2 lines

Well ... unless triggering timeouts can block on GFP_KERNEL allocations...

Quoted 8 lines

Quoted 2 lines

I mean, the timeout handler should also be considered a DMA-signalling path, and the same rules should apply to it.

I guess that's fair. Even with a timeout you want both to be signalling path.

I guess more generally, if a fence is signalled by mechanism A or B, whichever happens first, you have the choice between:

That doesn't happen in practice.

For each fence you only have one signaling path you need to guarantee forward progress for.

All other signaling paths are just opportunistically optimizations which *can* signal the fence, but there is no guarantee that they will.

We used to have some exceptions to that, especially around aborting submissions, but those turned out to be a really bad idea as well.

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

Quoted 8 lines

Quoted 6 lines

I'm not really convinced by this.

First, the timeout path must be a fence signalling path because the reason you have a timeout in the first place is because the hw might never signal the fence. So if the timeout path deadlocks on a kmalloc(GFP_KERNEL) and the hw never comes around to wake you up, boom.

Mhm, good point. On the other hand the timeout handling should probably be considered part of the normal signaling path.

Why would anyone want to allocate in a timeout path in the first place – especially for jobqueue?

Quoted 4 lines

Timeout -> close the associated ring. Done. JobQueue will signal the done_fences with -ECANCELED.

What would the driver want to allocate in its timeout path, i.e.: timeout callback.

Maybe you need an allocation to hold the struct delayed_work_struct field that you use to enqueue the timeout?

Alice

Quoted 3 lines · Alice Ryhl

Quoted 2 lines · Philipp Stanner

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

Quoted 18 lines

Quoted 18 lines

Quoted 11 lines

Quoted 9 lines

Quoted 2 lines

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

I'm not really convinced by this.

First, the timeout path must be a fence signalling path because the reason you have a timeout in the first place is because the hw might never signal the fence. So if the timeout path deadlocks on a kmalloc(GFP_KERNEL) and the hw never comes around to wake you up, boom.

Mhm, good point. On the other hand the timeout handling should probably be considered part of the normal signaling path.

Why would anyone want to allocate in a timeout path in the first place – especially for jobqueue?

Timeout -> close the associated ring. Done. JobQueue will signal the done_fences with -ECANCELED.

Quoted 4 lines

Quoted 1 line

What would the driver want to allocate in its timeout path, i.e.: timeout callback.

Maybe you need an allocation to hold the struct delayed_work_struct field that you use to enqueue the timeout?

And the workqueue were you schedule the delayed_work on must have the reclaim bit set.

Otherwise it can be that the workqueue finds all kthreads busy and tries to start a new one, e.g. allocating task structure......

You also potentially want device core dumps. Those usually use GFP_NOWAIT so that they can't cycle back and wait for some fence. The down side is that they can trivially fail under even light memory pressure.

OK, maybe I'm lost, but what delayed_work?

The jobqueue's delayed work item gets either created on JQ::new() or in jq.submit_job(). Why would anyone – that is: any driver – implement a delayed work in its timeout callback?

That doesn't make sense.

JQ notifies the driver from its delayed_work through timeout_callback(), and in that callback the driver closes the associated firmware ring.

And it drops the JQ. So it is gone. A new JQ will get a new timeout work item.

That's basically all the driver must ever do. Maybe some logging and stuff.

With firmware scheduling it should really be that simple.

And signalling / notifying userspace gets done by jobqueue.

Right?

Quoted 3 lines

You also potentially want device core dumps. Those usually use GFP_NOWAIT so that they can't cycle back and wait for some fence. The down side is that they can trivially fail under even light memory pressure.

Simply logging into dmesg should do the trick, shouldn't it?

Quoted 15 lines · Philipp Stanner

OK, maybe I'm lost, but what delayed_work?

The jobqueue's delayed work item gets either created on JQ::new() or in jq.submit_job(). Why would anyone – that is: any driver – implement a delayed work in its timeout callback?

That doesn't make sense.

JQ notifies the driver from its delayed_work through timeout_callback(), and in that callback the driver closes the associated firmware ring.

And it drops the JQ. So it is gone. A new JQ will get a new timeout work item.

Quoted 8 lines

That's basically all the driver must ever do. Maybe some logging and stuff.

With firmware scheduling it should really be that simple.

And signalling / notifying userspace gets done by jobqueue.

Right?

Well, the timeout path is part of the fence signaling critical section until all fences have been signaled.

But, if I, for instance, just kick off another work from the timeout handler and subsequently signal all fences by dropping the JQ, this other work must not play after DMA fence signaling rules anymore and is free to do whatever (maybe even take a device coredump without needing GFP_NOWAIT).

Xe does this with xe_devcoredump_deferred_snap_work for instance.

Quoted 5 lines

Quoted 3 lines

You also potentially want device core dumps. Those usually use GFP_NOWAIT so that they can't cycle back and wait for some fence. The down side is that they can trivially fail under even light memory pressure.

Simply logging into dmesg should do the trick, shouldn't it?

You can't "log" a device coredump into dmesg. :)

Quoted 4 lines · Christian König

Quoted 3 lines · Alice Ryhl

Quoted 2 lines · Philipp Stanner

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

[…]

Quoted 18 lines

Quoted 18 lines

Quoted 18 lines

Quoted 11 lines

Quoted 9 lines

Quoted 2 lines

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

I'm not really convinced by this.

First, the timeout path must be a fence signalling path because the reason you have a timeout in the first place is because the hw might never signal the fence. So if the timeout path deadlocks on a kmalloc(GFP_KERNEL) and the hw never comes around to wake you up, boom.

Mhm, good point. On the other hand the timeout handling should probably be considered part of the normal signaling path.

Why would anyone want to allocate in a timeout path in the first place – especially for jobqueue?

Timeout -> close the associated ring. Done. JobQueue will signal the done_fences with -ECANCELED.

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

What would the driver want to allocate in its timeout path, i.e.: timeout callback.

Maybe you need an allocation to hold the struct delayed_work_struct field that you use to enqueue the timeout?

And the workqueue were you schedule the delayed_work on must have the reclaim bit set.

Otherwise it can be that the workqueue finds all kthreads busy and tries to start a new one, e.g. allocating task structure......

OK, maybe I'm lost, but what delayed_work?

The jobqueue's delayed work item gets either created on JQ::new() or in jq.submit_job(). Why would anyone – that is: any driver – implement a delayed work in its timeout callback?

That doesn't make sense.

JQ notifies the driver from its delayed_work through timeout_callback(), and in that callback the driver closes the associated firmware ring.

And it drops the JQ. So it is gone. A new JQ will get a new timeout work item.

That's basically all the driver must ever do. Maybe some logging and stuff.

With firmware scheduling it should really be that simple.

And signalling / notifying userspace gets done by jobqueue.

Right?

Quoted 2 lines

You also potentially want device core dumps. Those usually use GFP_NOWAIT so that they can't cycle back and wait for some fence. The down side is that they can trivially fail under even light memory pressure.

Simply logging into dmesg should do the trick, shouldn't it?

P.

Quoted 4 lines · Christian König

Quoted 3 lines · Alice Ryhl

Quoted 2 lines · Philipp Stanner

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

[…]

Quoted 18 lines

Quoted 18 lines

Quoted 18 lines

Quoted 11 lines

Quoted 9 lines

Quoted 2 lines

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

I'm not really convinced by this.

First, the timeout path must be a fence signalling path because the reason you have a timeout in the first place is because the hw might never signal the fence. So if the timeout path deadlocks on a kmalloc(GFP_KERNEL) and the hw never comes around to wake you up, boom.

Mhm, good point. On the other hand the timeout handling should probably be considered part of the normal signaling path.

Why would anyone want to allocate in a timeout path in the first place – especially for jobqueue?

Timeout -> close the associated ring. Done. JobQueue will signal the done_fences with -ECANCELED.

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

What would the driver want to allocate in its timeout path, i.e.: timeout callback.

Maybe you need an allocation to hold the struct delayed_work_struct field that you use to enqueue the timeout?

And the workqueue were you schedule the delayed_work on must have the reclaim bit set.

Otherwise it can be that the workqueue finds all kthreads busy and tries to start a new one, e.g. allocating task structure......

OK, maybe I'm lost, but what delayed_work?

The jobqueue's delayed work item gets either created on JQ::new() or in jq.submit_job(). Why would anyone – that is: any driver – implement a delayed work in its timeout callback?

That doesn't make sense.

JQ notifies the driver from its delayed_work through timeout_callback(), and in that callback the driver closes the associated firmware ring.

And it drops the JQ. So it is gone. A new JQ will get a new timeout work item.

That's basically all the driver must ever do. Maybe some logging and stuff.

With firmware scheduling it should really be that simple.

And signalling / notifying userspace gets done by jobqueue.

Right?

Quoted 50 lines · Philipp Stanner

Quoted 4 lines · Christian König

Quoted 3 lines · Alice Ryhl

Quoted 2 lines · Philipp Stanner

Quoted 2 lines

Quoted 2 lines

Quoted 2 lines

[…]

Quoted 18 lines

Quoted 18 lines

Quoted 18 lines

Quoted 11 lines

Quoted 9 lines

Quoted 2 lines

Thinking more about it you should probably enforce that there is only one signaling path for each fence signaling.

I'm not really convinced by this.

First, the timeout path must be a fence signalling path because the reason you have a timeout in the first place is because the hw might never signal the fence. So if the timeout path deadlocks on a kmalloc(GFP_KERNEL) and the hw never comes around to wake you up, boom.

Mhm, good point. On the other hand the timeout handling should probably be considered part of the normal signaling path.

Why would anyone want to allocate in a timeout path in the first place – especially for jobqueue?

Timeout -> close the associated ring. Done. JobQueue will signal the done_fences with -ECANCELED.

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

What would the driver want to allocate in its timeout path, i.e.: timeout callback.

Maybe you need an allocation to hold the struct delayed_work_struct field that you use to enqueue the timeout?

And the workqueue were you schedule the delayed_work on must have the reclaim bit set.

Otherwise it can be that the workqueue finds all kthreads busy and tries to start a new one, e.g. allocating task structure......

OK, maybe I'm lost, but what delayed_work?

The jobqueue's delayed work item gets either created on JQ::new() or in jq.submit_job(). Why would anyone – that is: any driver – implement a delayed work in its timeout callback?

That doesn't make sense.

JQ notifies the driver from its delayed_work through timeout_callback(), and in that callback the driver closes the associated firmware ring.

And it drops the JQ. So it is gone. A new JQ will get a new timeout work item.

Quoted 8 lines

That's basically all the driver must ever do. Maybe some logging and stuff.

With firmware scheduling it should really be that simple.

And signalling / notifying userspace gets done by jobqueue.

Right?

What I'm getting at is that a driver author might attempt to implement their own timeout logic instead of using the job queue, and if they do, they might get it wrong in the way I described.

You're correct that they shouldn't do this. But you asked how a driver author might get the timeout wrong, and doing it the wrong way is one such way they might do it in the wrong way.

/// The owner of this value must ensure that this fence is signalled. struct MustBeSignalled<'fence> { ... }
/// Proof value indicating that the fence has either already been /// signalled, or it will be. The lifetime ensures that you cannot mix /// up the proof value.
struct WillBeSignalled<'fence> { ... }

Quoted 6 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

/// The owner of this value must ensure that this fence is signalled. struct MustBeSignalled<'fence> { ... }
/// Proof value indicating that the fence has either already been /// signalled, or it will be. The lifetime ensures that you cannot mix /// up the proof value.
struct WillBeSignalled<'fence> { ... }

Sorry, I have more questions, unfortunately. Seems that {Must,Will}BeSignalled are targeting specific fences (at least that's what the doc and 'fence lifetime says), but in practice, the WorkItem backing the scheduler can queue 0-N jobs (0 if no jobs have their deps met, and N > 1 if more than one job is ready). Similarly, an IRQ handler can signal 0-N fences (can be that the IRQ has nothing to do we job completion, or, it can be that multiple jobs have completed). How is this MustBeSignalled object going to be instantiated in practice if it's done before the DmaFenceWorkItem::run() function is called?

Quoted 6 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

/// The owner of this value must ensure that this fence is signalled. struct MustBeSignalled<'fence> { ... }
/// Proof value indicating that the fence has either already been /// signalled, or it will be. The lifetime ensures that you cannot mix /// up the proof value.
struct WillBeSignalled<'fence> { ... }

Sorry, I have more questions, unfortunately. Seems that {Must,Will}BeSignalled are targeting specific fences (at least that's what the doc and 'fence lifetime says), but in practice, the WorkItem backing the scheduler can queue 0-N jobs (0 if no jobs have their deps met, and N > 1 if more than one job is ready). Similarly, an IRQ handler can signal 0-N fences (can be that the IRQ has nothing to do we job completion, or, it can be that multiple jobs have completed). How is this MustBeSignalled object going to be instantiated in practice if it's done before the DmaFenceWorkItem::run() function is called?

Quoted 10 lines · Boris Brezillon

Quoted 6 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

/// The owner of this value must ensure that this fence is signalled. struct MustBeSignalled<'fence> { ... }
/// Proof value indicating that the fence has either already been /// signalled, or it will be. The lifetime ensures that you cannot mix /// up the proof value.
struct WillBeSignalled<'fence> { ... }

Quoted 9 lines

Sorry, I have more questions, unfortunately. Seems that {Must,Will}BeSignalled are targeting specific fences (at least that's what the doc and 'fence lifetime says), but in practice, the WorkItem backing the scheduler can queue 0-N jobs (0 if no jobs have their deps met, and N > 1 if more than one job is ready). Similarly, an IRQ handler can signal 0-N fences (can be that the IRQ has nothing to do we job completion, or, it can be that multiple jobs have completed). How is this MustBeSignalled object going to be instantiated in practice if it's done before the DmaFenceWorkItem::run() function is called?

The {Must,Will}BeSignalled closure pair needs to wrap the piece of code that ensures a specific fence is signalled. If you have code that manages a collection of fences and invokes code for specific fences depending on outside conditions, then that's a different matter.

After all, transfer_to_wq() has two components:
1. Logic to ensure any spawned workqueue job eventually gets to run.
2. Once the individual job runs, logic specific to the one fence ensures that this one fence gets signalled.

And {Must,Will}BeSignalled exists to help model part (2.). But what you described with the IRQ callback falls into (1.) instead, which is outside the scope of {Must,Will}BeSignalled (or at least requires more complex APIs).

Quoted 26 lines · On Tue, 10 Feb 2026 13:26:48 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 10 lines · Boris Brezillon

Quoted 6 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

/// The owner of this value must ensure that this fence is signalled. struct MustBeSignalled<'fence> { ... }
/// Proof value indicating that the fence has either already been /// signalled, or it will be. The lifetime ensures that you cannot mix /// up the proof value.
struct WillBeSignalled<'fence> { ... }

Quoted 9 lines

Sorry, I have more questions, unfortunately. Seems that {Must,Will}BeSignalled are targeting specific fences (at least that's what the doc and 'fence lifetime says), but in practice, the WorkItem backing the scheduler can queue 0-N jobs (0 if no jobs have their deps met, and N > 1 if more than one job is ready). Similarly, an IRQ handler can signal 0-N fences (can be that the IRQ has nothing to do we job completion, or, it can be that multiple jobs have completed). How is this MustBeSignalled object going to be instantiated in practice if it's done before the DmaFenceWorkItem::run() function is called?

The {Must,Will}BeSignalled closure pair needs to wrap the piece of code that ensures a specific fence is signalled. If you have code that manages a collection of fences and invokes code for specific fences depending on outside conditions, then that's a different matter.

Quoted 4 lines

After all, transfer_to_wq() has two components:
1. Logic to ensure any spawned workqueue job eventually gets to run.
2. Once the individual job runs, logic specific to the one fence ensures that this one fence gets signalled.

Okay, that's a change compared to how things are modeled in C (and in JobQueue) at the moment: the WorkItem is not embedded in a specific job, it's something that's attached to the JobQueue. The idea being that the WorkItem represents a task to be done on the queue itself (check if the first element in the queue is ready for execution), not on a particular job. Now, we could change that and have a per-job WorkItem, but ultimately, we'll have to make sure jobs are dequeued in order (deps on JobN can be met before deps on Job0, but we still want JobN to be submitted after Job0), and we'd pay the WorkItem overhead once per Job instead of once per JobQueue. Probably not the end of the world, but it's worth considering, still.

Quoted 4 lines

And {Must,Will}BeSignalled exists to help model part (2.). But what you described with the IRQ callback falls into (1.) instead, which is outside the scope of {Must,Will}BeSignalled (or at least requires more complex APIs).

For IRQ callbacks, it's not just about making sure they run, but also making sure nothing in there can lead to deadlocks, which is basically #2, except it's not scoped to a particular fence. It's just a "fences can be signaled from there" marker. We could restrict it to "fences of this particular implementation can be signaled from there" but not "this particular fence instance will be signaled next, if any", because that we don't know until we've walked some HW state to figure out which job is complete and thus which fence we need to signal (the interrupt we get is most likely multiplexing completion on multiple GPU contexts, so before we can even get to our per-context in-flight-jobs FIFO, we need to demux this thing).

Quoted 34 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 13:26:48 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 10 lines · Boris Brezillon

Quoted 6 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

/// The owner of this value must ensure that this fence is signalled. struct MustBeSignalled<'fence> { ... }
/// Proof value indicating that the fence has either already been /// signalled, or it will be. The lifetime ensures that you cannot mix /// up the proof value.
struct WillBeSignalled<'fence> { ... }

Quoted 9 lines

Sorry, I have more questions, unfortunately. Seems that {Must,Will}BeSignalled are targeting specific fences (at least that's what the doc and 'fence lifetime says), but in practice, the WorkItem backing the scheduler can queue 0-N jobs (0 if no jobs have their deps met, and N > 1 if more than one job is ready). Similarly, an IRQ handler can signal 0-N fences (can be that the IRQ has nothing to do we job completion, or, it can be that multiple jobs have completed). How is this MustBeSignalled object going to be instantiated in practice if it's done before the DmaFenceWorkItem::run() function is called?

The {Must,Will}BeSignalled closure pair needs to wrap the piece of code that ensures a specific fence is signalled. If you have code that manages a collection of fences and invokes code for specific fences depending on outside conditions, then that's a different matter.

Quoted 4 lines

After all, transfer_to_wq() has two components:
1. Logic to ensure any spawned workqueue job eventually gets to run.
2. Once the individual job runs, logic specific to the one fence ensures that this one fence gets signalled.

Quoted 11 lines

Okay, that's a change compared to how things are modeled in C (and in JobQueue) at the moment: the WorkItem is not embedded in a specific job, it's something that's attached to the JobQueue. The idea being that the WorkItem represents a task to be done on the queue itself (check if the first element in the queue is ready for execution), not on a particular job. Now, we could change that and have a per-job WorkItem, but ultimately, we'll have to make sure jobs are dequeued in order (deps on JobN can be met before deps on Job0, but we still want JobN to be submitted after Job0), and we'd pay the WorkItem overhead once per Job instead of once per JobQueue. Probably not the end of the world, but it's worth considering, still.

It sounds like the fix here is to have transfer_to_job_queue() instead of trying to do it at the workqueue level.

Quoted 6 lines

Quoted 5 lines

And {Must,Will}BeSignalled exists to help model part (2.). But what you described with the IRQ callback falls into (1.) instead, which is outside the scope of {Must,Will}BeSignalled (or at least requires more complex APIs).

Quoted 11 lines

For IRQ callbacks, it's not just about making sure they run, but also making sure nothing in there can lead to deadlocks, which is basically #2, except it's not scoped to a particular fence. It's just a "fences can be signaled from there" marker. We could restrict it to "fences of this particular implementation can be signaled from there" but not "this particular fence instance will be signaled next, if any", because that we don't know until we've walked some HW state to figure out which job is complete and thus which fence we need to signal (the interrupt we get is most likely multiplexing completion on multiple GPU contexts, so before we can even get to our per-context in-flight-jobs FIFO, we need to demux this thing).

All I can say is that this is a different use-case for the C api dma_fence_begin_signalling(). This different usage also seems useful, but it would be one that does not involve {Must,Will}BeSignalled arguments at all.

After all, dma_fence_begin_signalling() only requires those arguments if you want to convert a PrivateFence into a PublishedFence. (I guess a better name is PublishableFence.) If you're not trying to prove that a specific fence will be signalled, then you don't need the {Must,Will}BeSignalled arguments.

Quoted 47 lines · On Tue, 10 Feb 2026 14:11:12 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 34 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 13:26:48 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 10 lines · Boris Brezillon

Quoted 6 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

/// The owner of this value must ensure that this fence is signalled. struct MustBeSignalled<'fence> { ... }
/// Proof value indicating that the fence has either already been /// signalled, or it will be. The lifetime ensures that you cannot mix /// up the proof value.
struct WillBeSignalled<'fence> { ... }

Quoted 9 lines

Sorry, I have more questions, unfortunately. Seems that {Must,Will}BeSignalled are targeting specific fences (at least that's what the doc and 'fence lifetime says), but in practice, the WorkItem backing the scheduler can queue 0-N jobs (0 if no jobs have their deps met, and N > 1 if more than one job is ready). Similarly, an IRQ handler can signal 0-N fences (can be that the IRQ has nothing to do we job completion, or, it can be that multiple jobs have completed). How is this MustBeSignalled object going to be instantiated in practice if it's done before the DmaFenceWorkItem::run() function is called?

The {Must,Will}BeSignalled closure pair needs to wrap the piece of code that ensures a specific fence is signalled. If you have code that manages a collection of fences and invokes code for specific fences depending on outside conditions, then that's a different matter.

Quoted 4 lines

After all, transfer_to_wq() has two components:
1. Logic to ensure any spawned workqueue job eventually gets to run.
2. Once the individual job runs, logic specific to the one fence ensures that this one fence gets signalled.

Quoted 11 lines

Okay, that's a change compared to how things are modeled in C (and in JobQueue) at the moment: the WorkItem is not embedded in a specific job, it's something that's attached to the JobQueue. The idea being that the WorkItem represents a task to be done on the queue itself (check if the first element in the queue is ready for execution), not on a particular job. Now, we could change that and have a per-job WorkItem, but ultimately, we'll have to make sure jobs are dequeued in order (deps on JobN can be met before deps on Job0, but we still want JobN to be submitted after Job0), and we'd pay the WorkItem overhead once per Job instead of once per JobQueue. Probably not the end of the world, but it's worth considering, still.

Quoted 2 lines

It sounds like the fix here is to have transfer_to_job_queue() instead of trying to do it at the workqueue level.

Hm, so Job would be something like that (naming/trait-def are just suggestions to get the discussion going):

trait JobConsumer {
	type FenceType;
	type JobData;

	fn run(self: MustBeSignalled<T::FenceType>) -> Result<WillBeSignaled<Self::FenceType>>;
}

struct Job<T: JobConsumer> {
fence: MustBeSignalled<T::FenceType>,
data: T::JobData,
}

I guess that would do.

And then we need to flag the WorkItem that's exposed by the JobQueue as a DmaFenceWorkItem so that bindings::dma_fence_begin_signalling() is called before entry and lockdep can do its job and check that nothing forbidden happens in this WorkItem.

Quoted 22 lines

Quoted 5 lines

Quoted 4 lines

And {Must,Will}BeSignalled exists to help model part (2.). But what you described with the IRQ callback falls into (1.) instead, which is outside the scope of {Must,Will}BeSignalled (or at least requires more complex APIs).

Quoted 11 lines

For IRQ callbacks, it's not just about making sure they run, but also making sure nothing in there can lead to deadlocks, which is basically #2, except it's not scoped to a particular fence. It's just a "fences can be signaled from there" marker. We could restrict it to "fences of this particular implementation can be signaled from there" but not "this particular fence instance will be signaled next, if any", because that we don't know until we've walked some HW state to figure out which job is complete and thus which fence we need to signal (the interrupt we get is most likely multiplexing completion on multiple GPU contexts, so before we can even get to our per-context in-flight-jobs FIFO, we need to demux this thing).

All I can say is that this is a different use-case for the C api dma_fence_begin_signalling(). This different usage also seems useful, but it would be one that does not involve {Must,Will}BeSignalled arguments at all.

Quoted 5 lines

After all, dma_fence_begin_signalling() only requires those arguments if you want to convert a PrivateFence into a PublishedFence. (I guess a better name is PublishableFence.) If you're not trying to prove that a specific fence will be signalled, then you don't need the {Must,Will}BeSignalled arguments.

Okay, so that would be another function returning some sort of guard then? What I find confusing is the fact dma_fence::dma_fence_begin_signalling() matches the C function name which is not per-fence, but just this lock-guard model flagging a section from which any fence can be signalled, so maybe we should name your dma_fence_begin_signalling() proposal differently, dunno.

Quoted 63 lines · Boris Brezillon

Quoted 47 lines · On Tue, 10 Feb 2026 14:11:12 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 34 lines · Boris Brezillon

Quoted 26 lines · On Tue, 10 Feb 2026 13:26:48 +0000 Alice Ryhl <[email protected]> wrote:

Quoted 10 lines · Boris Brezillon

Quoted 6 lines · On Tue, 10 Feb 2026 10:15:04 +0000 Alice Ryhl <[email protected]> wrote:

/// The owner of this value must ensure that this fence is signalled. struct MustBeSignalled<'fence> { ... }
/// Proof value indicating that the fence has either already been /// signalled, or it will be. The lifetime ensures that you cannot mix /// up the proof value.
struct WillBeSignalled<'fence> { ... }

Quoted 9 lines

Sorry, I have more questions, unfortunately. Seems that {Must,Will}BeSignalled are targeting specific fences (at least that's what the doc and 'fence lifetime says), but in practice, the WorkItem backing the scheduler can queue 0-N jobs (0 if no jobs have their deps met, and N > 1 if more than one job is ready). Similarly, an IRQ handler can signal 0-N fences (can be that the IRQ has nothing to do we job completion, or, it can be that multiple jobs have completed). How is this MustBeSignalled object going to be instantiated in practice if it's done before the DmaFenceWorkItem::run() function is called?

The {Must,Will}BeSignalled closure pair needs to wrap the piece of code that ensures a specific fence is signalled. If you have code that manages a collection of fences and invokes code for specific fences depending on outside conditions, then that's a different matter.

Quoted 4 lines

After all, transfer_to_wq() has two components:
1. Logic to ensure any spawned workqueue job eventually gets to run.
2. Once the individual job runs, logic specific to the one fence ensures that this one fence gets signalled.

Quoted 11 lines

Okay, that's a change compared to how things are modeled in C (and in JobQueue) at the moment: the WorkItem is not embedded in a specific job, it's something that's attached to the JobQueue. The idea being that the WorkItem represents a task to be done on the queue itself (check if the first element in the queue is ready for execution), not on a particular job. Now, we could change that and have a per-job WorkItem, but ultimately, we'll have to make sure jobs are dequeued in order (deps on JobN can be met before deps on Job0, but we still want JobN to be submitted after Job0), and we'd pay the WorkItem overhead once per Job instead of once per JobQueue. Probably not the end of the world, but it's worth considering, still.

Quoted 2 lines

It sounds like the fix here is to have transfer_to_job_queue() instead of trying to do it at the workqueue level.

Hm, so Job would be something like that (naming/trait-def are just suggestions to get the discussion going):

trait JobConsumer {
	type FenceType;
	type JobData;

	fn run(self: MustBeSignalled<T::FenceType>) -> Result<WillBeSignaled<Self::FenceType>>;
}

Quoted 4 lines

struct Job<T: JobConsumer> {
fence: MustBeSignalled<T::FenceType>,
data: T::JobData,
}

The fence field of Job would be PublishedFence or PrivateFence (or just DriverDmaFence). The MustBeSignalled/WillBeSignaled types should only exist temporarily in a function scope.

Any time you transfer from one function scope to another (like our transfer_to_job_queue() or transfer_to_wq() examples), that results in finishing the MustBeSignalled/WillBeSignaled scope on one thread and creating a new MustBeSignalled/WillBeSignaled scope on another thread.

One could imagine a model where there is no lifetime and you can carry it around as you wish. That model works okay in most regards, but it gives up the ability to ensure that dma_fence_lockdep_map is properly configured to catch mistakes.

The lifetime prohibits you from using the normal ownership semantics to e.g. transfer the MustBeSignalled into a random workqueue, enforcing that you can only transfer it into a workqueue by using the provided methods, which sets up the lockdep dependencies correctly and ensures that dma_fence_lockdep_map is taken in the workqueue job too.

Quoted 7 lines

I guess that would do.

And then we need to flag the WorkItem that's exposed by the JobQueue as a DmaFenceWorkItem so that bindings::dma_fence_begin_signalling() is called before entry and lockdep can do its job and check that nothing forbidden happens in this WorkItem.

In the case of JobQueue, it may make sense to just have the job queue implementation do that manually. I do not think the workqueue-level API can fully enforce that the job queue can't make mistakes here.

Quoted 29 lines

Quoted 23 lines

Quoted 6 lines

Quoted 5 lines

And {Must,Will}BeSignalled exists to help model part (2.). But what you described with the IRQ callback falls into (1.) instead, which is outside the scope of {Must,Will}BeSignalled (or at least requires more complex APIs).

Quoted 11 lines

For IRQ callbacks, it's not just about making sure they run, but also making sure nothing in there can lead to deadlocks, which is basically #2, except it's not scoped to a particular fence. It's just a "fences can be signaled from there" marker. We could restrict it to "fences of this particular implementation can be signaled from there" but not "this particular fence instance will be signaled next, if any", because that we don't know until we've walked some HW state to figure out which job is complete and thus which fence we need to signal (the interrupt we get is most likely multiplexing completion on multiple GPU contexts, so before we can even get to our per-context in-flight-jobs FIFO, we need to demux this thing).

All I can say is that this is a different use-case for the C api dma_fence_begin_signalling(). This different usage also seems useful, but it would be one that does not involve {Must,Will}BeSignalled arguments at all.

Quoted 5 lines

After all, dma_fence_begin_signalling() only requires those arguments if you want to convert a PrivateFence into a PublishedFence. (I guess a better name is PublishableFence.) If you're not trying to prove that a specific fence will be signalled, then you don't need the {Must,Will}BeSignalled arguments.

Quoted 6 lines

Okay, so that would be another function returning some sort of guard then? What I find confusing is the fact dma_fence::dma_fence_begin_signalling() matches the C function name which is not per-fence, but just this lock-guard model flagging a section from which any fence can be signalled, so maybe we should name your dma_fence_begin_signalling() proposal differently, dunno.

Yes we would need multiple methods that call dma_fence_begin_signalling() depending on why you are calling it.

Quoted 64 lines · Christian König

Quoted 40 lines · Boris Brezillon

Quoted 12 lines · On Mon, 09 Feb 2026 09:19:46 +0100 Philipp Stanner <[email protected]> wrote:

Quoted 11 lines · Danilo Krummrich

Quoted 9 lines · Boris Brezillon

On Tue,  3 Feb 2026 09:14:01 +0100
Philipp Stanner <[email protected]> wrote:
Unfortunately, I don't know how to translate that in rust, but we need a way to check if any path code path does a DmaFence.signal(), go back to the entry point (for a WorkItem, that would be WorkItem::run() for instance), and make it a DmaFenceSignallingPath. Not only that, but we need to know all the deps that make it so this path can be called (if I take the WorkItem example, that would be the path that leads to the WorkItem being scheduled).

Quoted 7 lines

Quoted 5 lines

I think we need a guard object for this that is not Send, just like for any other lock.

Internally, those markers rely on lockdep, i.e. they just acquire and release a "fake" lock.

The guard object would be created through fence.begin_signalling(), wouldn't it?

It shouldn't be a (&self)-method, because at the start of a DMA signaling path, you don't necessarily know which fence you're going to signal (you might actually signal several of them).

Quoted 1 line

And when it drops you call dma_fence_end_signalling()?

Yep, dma_fence_end_signalling() should be called when the guard is dropped.

Quoted 2 lines

How would that ensure that the driver actually marks the signalling region correctly?

Nothing, and that's a problem we have in C: you have no way of telling which code section is going to be a DMA-signaling path. I can't think of any way to make that safer in rust, unfortunately. The best I can think of would be to

Quoted 9 lines

- Have a special DmaFenceSignalWorkItem (wrapper a WorkItem with extra constraints) that's designed for DMA-fence signaling, and that takes the DmaSignaling guard around the ::run() call.
- We would then need to ensure that any code path scheduling this work item is also in a DMA-signaling path by taking a ref to the DmaSignalingGuard. This of course doesn't guarantee that the section is wide enough to prevent any non-authorized operations in any path leading to this WorkItem scheduling, but it would at least force the caller to consider the problem.

On the C side I have a patch set which does something very similar.

It's basically a WARN_ON_ONCE() which triggers as soon as you try to signal a DMA fence from an IOCTL, or more specific process context.

Signaling a DMA fence from interrupt context, a work item or kernel thread is still allowed, there is just the hole that you can schedule a work item from process context as well.

The major problem with that patch set is that we have tons of very hacky signaling paths in drivers already because we initially didn't knew how much trouble getting this wrong causes.

Quoted 3 lines

I'm strongly in favor of getting this right for the rust side from the beginning and enforcing strict rules for every code trying to implement a DMA fence.

Hmm. Could you say a bit more about what the rules are? I just re-read the comments in dma-fence.c, but I have some questions.

First, how does the signalling annotation work when the signalling path crosses thread boundaries?

For example, let's say I call an ioctl to
perform an async VM_BIND, then the dma fence signalling critical path starts in the ioctl, but then it moves into a workqueue and finishes there, right?

Second, it looks like we have the same challenge as with irq locks where you must properly nest dma_fence_begin_signalling() regions, and can't e.g. do this:

c1 = dma_fence_begin_signalling()
c2 = dma_fence_begin_signalling()
dma_fence_end_signalling(c1)
dma_fence_end_signalling(c2)

Alice

rust/kernel/sync/dma_fence.rs new file +57 -0

+/// A synchronization primitive mainly for GPU drivers.
+///
+/// DmaFences are always reference counted. The typical use case is that one side registers
+/// callbacks on the fence which will perform a certain action (such as queueing work) once the
+/// other side signals the fence.
+///
+/// # Examples
+///
+/// ```
+/// use kernel::sync::{Arc, ArcBorrow, DmaFence, DmaFenceCtx, DmaFenceCb, DmaFenceCbFunc};
+/// use core::sync::atomic::{self, AtomicBool};
+///
+/// static mut CHECKER: AtomicBool = AtomicBool::new(false);
+///
+/// struct CallbackData {
+///     i: u32,
+/// }
+///
+/// impl CallbackData {
+///     fn new() -> Self {
+///         Self { i: 9 }
+///     }
+/// }
+///
+/// impl DmaFenceCbFunc for CallbackData {
+///     fn callback(cb: Pin<KBox<DmaFenceCb<Self>>>) where Self: Sized {
+///         assert_eq!(cb.data.i, 9);
+///         // SAFETY: Just to have an easy way for testing. This cannot race with the checker
+///         // because the fence signalling callbacks are executed synchronously.
+///         unsafe { CHECKER.store(true, atomic::Ordering::Relaxed); }
+///     }
+/// }
+///
+/// struct DriverData {
+///     i: u32,
+/// }
+///
+/// impl DriverData {
+///     fn new() -> Self {
+///         Self { i: 5 }
+///     }
+/// }
+///
+/// let data = DriverData::new();
+/// let fctx = DmaFenceCtx::new()?;
+///
+/// let mut fence = fctx.as_arc_borrow().new_fence(data)?;
+///
+/// let cb_data = CallbackData::new();
+/// fence.register_callback(cb_data);
+/// // fence.begin_signalling();
+/// fence.signal()?;
+/// // Now check wehether the callback was actually executed.
+/// // SAFETY: `fence.signal()` above works sequentially. We just check here whether the signalling
+/// // actually did set the boolean correctly.
+/// unsafe { assert_eq!(CHECKER.load(atomic::Ordering::Relaxed), true); }
+///

rust/kernel/sync/dma_fence.rs new file +8 -0

+    /// Marks whether the fence is currently in the signalling critical section.
+    signalling: bool,
+    /// A boolean needed for the C backend's lockdep guard.
+    signalling_cookie: bool,
+    /// A reference to the associated [`DmaFenceCtx`] so that it cannot be dropped while there are
+    /// still fences around.
+    fctx: Arc<DmaFenceCtx>,
+}

Quoted 57 lines · On Tue, 3 Feb 2026 09:14:01 +0100 Philipp Stanner <[email protected]> wrote:

rust/kernel/sync/dma_fence.rs new file +57 -0

+/// A synchronization primitive mainly for GPU drivers.
+///
+/// DmaFences are always reference counted. The typical use case is that one side registers
+/// callbacks on the fence which will perform a certain action (such as queueing work) once the
+/// other side signals the fence.
+///
+/// # Examples
+///
+/// ```
+/// use kernel::sync::{Arc, ArcBorrow, DmaFence, DmaFenceCtx, DmaFenceCb, DmaFenceCbFunc};
+/// use core::sync::atomic::{self, AtomicBool};
+///
+/// static mut CHECKER: AtomicBool = AtomicBool::new(false);
+///
+/// struct CallbackData {
+///     i: u32,
+/// }
+///
+/// impl CallbackData {
+///     fn new() -> Self {
+///         Self { i: 9 }
+///     }
+/// }
+///
+/// impl DmaFenceCbFunc for CallbackData {
+///     fn callback(cb: Pin<KBox<DmaFenceCb<Self>>>) where Self: Sized {
+///         assert_eq!(cb.data.i, 9);
+///         // SAFETY: Just to have an easy way for testing. This cannot race with the checker
+///         // because the fence signalling callbacks are executed synchronously.
+///         unsafe { CHECKER.store(true, atomic::Ordering::Relaxed); }
+///     }
+/// }
+///
+/// struct DriverData {
+///     i: u32,
+/// }
+///
+/// impl DriverData {
+///     fn new() -> Self {
+///         Self { i: 5 }
+///     }
+/// }
+///
+/// let data = DriverData::new();
+/// let fctx = DmaFenceCtx::new()?;
+///
+/// let mut fence = fctx.as_arc_borrow().new_fence(data)?;
+///
+/// let cb_data = CallbackData::new();
+/// fence.register_callback(cb_data);
+/// // fence.begin_signalling();
+/// fence.signal()?;
+/// // Now check wehether the callback was actually executed.
+/// // SAFETY: `fence.signal()` above works sequentially. We just check here whether the signalling
+/// // actually did set the boolean correctly.
+/// unsafe { assert_eq!(CHECKER.load(atomic::Ordering::Relaxed), true); }
+///

rust/kernel/sync/dma_fence.rs line 207

Quoted 10 lines

rust/kernel/sync/dma_fence.rs new file +10 -0

+/// Ok::<(), Error>(())
+/// ```
+#[pin_data]
+pub struct DmaFence<T> {
+    /// The actual dma_fence passed to C.
+    #[pin]
+    inner: Opaque<bindings::dma_fence>,
+    /// User data.
+    #[pin]
+    data: T,

A DmaFence is a cross-device synchronization mechanism that can (and will) cross the driver boundary (one driver can wait on a fence emitted by a different driver). As such, I don't think embedding a generic T in the DmaFence and considering it's the object being passed around is going to work, because, how can one driver know the T chosen by the driver that created the fence? If you want to have some fence emitter data attached to the DmaFence allocation, you'll need two kind of objects:

- one that's type agnostic and on which you can do the callback registration/unregistration, signalling checks, and generally all type-agnostic operations. That's basically just a wrapper around a bindings::dma_fence implementing AlwaysRefCounted.
- one that has the extra data and fctx, with a way to transmute from a generic fence to a implementer specific one in case the driver wants to do something special when waiting on its own fences (check done with the fence ops in C, I don't know how that translates in rust)

Quoted 9 lines

rust/kernel/sync/dma_fence.rs new file +8 -0

+    /// Marks whether the fence is currently in the signalling critical section.
+    signalling: bool,
+    /// A boolean needed for the C backend's lockdep guard.
+    signalling_cookie: bool,
+    /// A reference to the associated [`DmaFenceCtx`] so that it cannot be dropped while there are
+    /// still fences around.
+    fctx: Arc<DmaFenceCtx>,
+}

Quoted 4 lines · Boris Brezillon

Quoted 1 line · On Tue,  3 Feb 2026 09:14:01 +0100 Philipp Stanner <[email protected]> wrote:

Quoted 4 lines

Quoted 4 lines

rust/kernel/sync/dma_fence.rs new file +4 -0

+#[pin_data]
+pub struct DmaFence<T> {
+    /// The actual dma_fence passed to C.
+    #[pin]

Quoted 8 lines

Quoted 4 lines

+    inner: Opaque<bindings::dma_fence>,
+    /// User data.
+    #[pin]
+    data: T,

A DmaFence is a cross-device synchronization mechanism that can (and will)

Quoted 8 lines

cross the driver boundary (one driver can wait on a fence emitted
by a different driver). As such, I don't think embedding a generic T in the DmaFence and considering it's the object being passed around is going to work, because, how can one driver know the T chosen by the driver that created the fence? If you want to have some fence emitter data attached to the DmaFence allocation, you'll need two kind of objects:

Quoted 8 lines

- one that's type agnostic and on which you can do the callback registration/unregistration, signalling checks, and generally all type-agnostic operations. That's basically just a wrapper around a bindings::dma_fence implementing AlwaysRefCounted.
- one that has the extra data and fctx, with a way to transmute from a generic fence to a implementer specific one in case the driver wants to do something special when waiting on its own fences (check done with the fence ops in C, I don't know how that translates in rust)

If data is moved to the end of struct and DmaFence<T> changed to DmaFence<T: ?Sized>, you would also gain the ability to coerce DmaFence<T> to DmaFence<dyn Trait>, e.g. DmaFence<dyn Any>.

Best,
Gary

Quoted 9 lines

Quoted 9 lines

+    /// Marks whether the fence is currently in the signalling critical section.
+    signalling: bool,
+    /// A boolean needed for the C backend's lockdep guard.
+    signalling_cookie: bool,
+    /// A reference to the associated [DmaFenceCtx] so that it cannot be dropped while there are +    /// still fences around.
+    fctx: Arc<DmaFenceCtx>,
+}

Who needs fences from another driver?

I think we should go one step back here and question the general design.

I only included data: T because it was among the early feedback that this is how you do it in Rust.

I was never convinced that it's a good idea. Jobqueue doesn't need the 'data' field. Can anyone think of anyone who would need it?

What kind of data would be in there? It seems a driver would store its equivalent of C's

struct my_fence {
   struct dma_fence f;
   /* other driver data */
}

which is then accessed in C with container_of.

But that data is only ever needed by that very driver.

They are *not* a data transfer mechanism. It seems very wrong design- wise to transfer generic data T from one driver to another. That's not a fence's purpose. Another primitive should be used for that.

If another driver could touch / consume / see / use the emitter's data:
T, that would grossly decouple us from the original dma_fence design. It would be akin to doing a container_of to consume foreign driver data.

Quoted 1 line · Philipp Stanner

Who needs fences from another driver?

When you get VM_BIND and EXEC IOCTLs a driver takes a list of syncobjs the submitted job should wait for before execution.

The fences of those syncobjs can be from anywhere, including other DRM drivers.

Quoted 12 lines

I think we should go one step back here and question the general design.

I only included data: T because it was among the early feedback that this is how you do it in Rust.

I was never convinced that it's a good idea. Jobqueue doesn't need the 'data' field. Can anyone think of anyone who would need it?

What kind of data would be in there? It seems a driver would store its equivalent of C's

Quoted 6 lines

struct my_fence {
   struct dma_fence f;
   /* other driver data */
}

which is then accessed in C with container_of.

Your current struct is exactly this pattern:

	struct DmaFence<T> {
	    inner: Opaque<bindings::dma_fence>,
	    ...
	    data: T,
	}

So, in Rust you can just write DmaFence<MyData> rather than,

	struct my_dma_fence {
		struct dma_fence inner;
		struct my_data data;
	}

Quoted 1 line

But that data is only ever needed by that very driver.

Exactly, this is the "owned" type that is only ever used by this driver.

Quoted 8 lines

They are *not* a data transfer mechanism. It seems very wrong design- wise to transfer generic data T from one driver to another. That's not a fence's purpose. Another primitive should be used for that.

If another driver could touch / consume / see / use the emitter's data:
T, that would grossly decouple us from the original dma_fence design. It would be akin to doing a container_of to consume foreign driver data.

Correct, that's why the suggestion here was to have a second type that is only

	struct ForeignDmaFence {
	    inner: Opaque<bindings::dma_fence>,
	    ...,
	    /* No data. */
	}

i.e. it does not not provide access to the rest of the allocation, since it is private to the owning driver.

This type should also not have methods like signal(), since only the owner of the fence should be allowed to signal the fence.

Quoted 5 lines · Gary Guo

Quoted 4 lines · Boris Brezillon

Quoted 1 line · On Tue,  3 Feb 2026 09:14:01 +0100 Philipp Stanner <[email protected]> wrote:

[…]

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

rust/kernel/sync/dma_fence.rs new file +4 -0

+#[pin_data]
+pub struct DmaFence<T> {
+    /// The actual dma_fence passed to C.
+    #[pin]

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

+    inner: Opaque<bindings::dma_fence>,
+    /// User data.
+    #[pin]
+    data: T,

A DmaFence is a cross-device synchronization mechanism that can (and will)

I'm not questioning the truth behind this statement. They are designed to do that. But is that actually being done, currently? I recently found that the get_driver_name() callback intended to inform the consumer of a fence about who actually issued the fence is only ever used by i915.

Who actually uses that feature? Who needs fences from another driver?

Just out of curiousity

Quoted 17 lines

Quoted 8 lines

cross the driver boundary (one driver can wait on a fence emitted
by a different driver). As such, I don't think embedding a generic T in the DmaFence and considering it's the object being passed around is going to work, because, how can one driver know the T chosen by the driver that created the fence? If you want to have some fence emitter data attached to the DmaFence allocation, you'll need two kind of objects:

Quoted 8 lines

- one that's type agnostic and on which you can do the callback registration/unregistration, signalling checks, and generally all type-agnostic operations. That's basically just a wrapper around a bindings::dma_fence implementing AlwaysRefCounted.
- one that has the extra data and fctx, with a way to transmute from a generic fence to a implementer specific one in case the driver wants to do something special when waiting on its own fences (check done with the fence ops in C, I don't know how that translates in rust)

Quoted 3 lines

If data is moved to the end of struct and DmaFence<T> changed to DmaFence<T: ?Sized>, you would also gain the ability to coerce DmaFence<T> to DmaFence<dyn Trait>, e.g. DmaFence<dyn Any>.

I think we should go one step back here and question the general design.

I only included data: T because it was among the early feedback that this is how you do it in Rust.

I was never convinced that it's a good idea. Jobqueue doesn't need the 'data' field. Can anyone think of anyone who would need it?

What kind of data would be in there? It seems a driver would store its equivalent of C's

struct my_fence {
   struct dma_fence f;
   /* other driver data */
}

which is then accessed in C with container_of.

But that data is only ever needed by that very driver.

My main point here is:
dma_fence's are a synchronization primitive very similar to
completions: informing about that something is done, executing every registrants callbacks.

They are *not* a data transfer mechanism. It seems very wrong design- wise to transfer generic data T from one driver to another. That's not a fence's purpose. Another primitive should be used for that.

If another driver could touch / consume / see / use the emitter's data:
T, that would grossly decouple us from the original dma_fence design. It would be akin to doing a container_of to consume foreign driver data.

Like Xe doing a

struct nouveau_fence *f = container_of(generic_fence, …);

Why would that ever be done? Seems totally broken.

So I strongly think that we'd either want to drop data: T, or we should think about possibilities to hide it from other drivers.

I've got currently no idea how that could be addressed in Rust, though

On Fri, 06 Feb 2026 10:32:38 +0100
Philipp Stanner <[email protected]> wrote:

Quoted 5 lines

So I strongly think that we'd either want to drop data: T, or we should think about possibilities to hide it from other drivers.

I've got currently no idea how that could be addressed in Rust, though

So, as Danilo explained in his reply, there's two kind of users:

1. those that want to wait on fences (that'd be the JobQueue, for instance)
2. those that are emitting fences (AKA those implementing the fence_ops in C)

And each of them should be given different access to the underlying dma_fence, hence the proposal to have different objects to back those concepts.

Quoted 5 lines · Gary Guo

Quoted 4 lines · Boris Brezillon

Quoted 1 line · On Tue,  3 Feb 2026 09:14:01 +0100 Philipp Stanner <[email protected]> wrote:

[…]

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

rust/kernel/sync/dma_fence.rs new file +4 -0

+#[pin_data]
+pub struct DmaFence<T> {
+    /// The actual dma_fence passed to C.
+    #[pin]

Quoted 8 lines

Quoted 8 lines

Quoted 4 lines

+    inner: Opaque<bindings::dma_fence>,
+    /// User data.
+    #[pin]
+    data: T,

A DmaFence is a cross-device synchronization mechanism that can (and will)

I'm not questioning the truth behind this statement. They are designed to do that. But is that actually being done, currently? I recently found that the get_driver_name() callback intended to inform the consumer of a fence about who actually issued the fence is only ever used by i915.

rust/helpers/dma_fence.c new file +14 -0

+void rust_helper_dma_fence_get(struct dma_fence *f)
+{
+	dma_fence_get(f);
+}
+
+void rust_helper_dma_fence_put(struct dma_fence *f)
+{
+	dma_fence_put(f);
+}
+
+bool rust_helper_dma_fence_begin_signalling(void)
+{
+	return dma_fence_begin_signalling();
+}

rust/kernel/sync/dma_fence.rs new file +10 -0

+void rust_helper_spin_lock_init(spinlock_t *lock)
+{
+	spin_lock_init(lock);
+}
[..]
+#[pin_data]
+pub struct DmaFenceCtx {
+    /// An opaque spinlock. Only ever passed to the C backend, never used by Rust.
+    #[pin]
+    lock: Opaque<bindings::spinlock_t>,
[...]
+}

rust/kernel/sync/dma_fence.rs new file +5 -0

+impl<T: DmaFenceCbFunc + 'static> DmaFenceCb<T> {
+    unsafe extern "C" fn callback(
+        _fence_ptr: *mut bindings::dma_fence,
+        cb_ptr: *mut bindings::dma_fence_cb,
+    ) {

rust/kernel/sync/dma_fence.rs new file +10 -0

+    pub fn register_callback<U: DmaFenceCbFunc + 'static>(&self, data: impl PinInit<U>) -> Result {
+        let cb = DmaFenceCb::new(data)?;
+        let ptr = cb.into_foreign() as *mut DmaFenceCb<U>;
+        // SAFETY: `ptr` was created validly directly above.
+        let inner_cb = unsafe { (*ptr).inner.get() };
+
+        // SAFETY: `self.as_raw()` is valid because `self` is refcounted, `inner_cb` was created
+        // validly above and was turned into a ForeignOwnable, so it won't be dropped. `callback`
+        // has static life time.
+        let ret = unsafe {